I ran out of space on the server running namenode, hbase master, hbase regionserver and a datanode and during the subsequent restarts hbase master wouldn’t start.
During log splitting it died with the following error:
2013-07-02 19:52:12,269 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown. java.lang.AssertionError at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader$WALReaderFSDataInputStream.getPos(SequenceFileLogReader.java:112) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1491) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:57) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:158) at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:648) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:834) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:750) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:283) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:202) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:275) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:205) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:408) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:301) 2013-07-02 19:52:12,271 INFO org.apache.hadoop.hbase.master.HMaster: Aborting |
I found two ways to get it to start up again, the first one I tried was to move away the log splitting directory in hdfs with the following command (strongly discouraged to do this):
$ hadoop fs -mv /hbase/.logs/hbase1.domain.com,60020,1367325077343-splitting /user/hdfs |
After some help from #hbase on irc.freenode.net I moved it back and tried starting hbase master with java assertions disabled, and that solved the issue.
To disable assertions in the JVM you make sure that the parameter -da (or -disableassertions) is passed to java when invoked.
I did this by editing /etc/hbase/conf/hbase-site.sh and adding -da to the HBASE_MASTER_OPTS environment variable.
