In the world of NoSQL

I ran out of space on the server running namenode, hbase master, hbase regionserver and a datanode and during the subsequent restarts hbase master wouldn’t start.
During log splitting it died with the following error:

2013-07-02 19:52:12,269 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader$WALReaderFSDataInputStream.getPos(
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(
        at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(
        at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(
        at org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(
        at org.apache.hadoop.hbase.master.HMaster.finishInitialization(
2013-07-02 19:52:12,271 INFO org.apache.hadoop.hbase.master.HMaster: Aborting

I found two ways to get it to start up again, the first one I tried was to move away the log splitting directory in hdfs with the following command (strongly discouraged to do this):

$ hadoop fs -mv /hbase/.logs/,60020,1367325077343-splitting /user/hdfs

After some help from #hbase on I moved it back and tried starting hbase master with java assertions disabled, and that solved the issue.

To disable assertions in the JVM you make sure that the parameter -da (or -disableassertions) is passed to java when invoked.

I did this by editing /etc/hbase/conf/ and adding -da to the HBASE_MASTER_OPTS environment variable.