I ran out of space on the server running namenode, hbase master, hbase regionserver and a datanode and during the subsequent restarts hbase master wouldn’t start.
During log splitting it died with the following error:
2013-07-02 19:52:12,269 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
java.lang.AssertionError
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader$WALReaderFSDataInputStream.getPos(SequenceFileLogReader.java:112)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1491)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:57)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:158)
at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:648)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:834)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:750)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:283)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:202)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:275)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:205)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:408)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:301)
2013-07-02 19:52:12,271 INFO org.apache.hadoop.hbase.master.HMaster: Aborting |
2013-07-02 19:52:12,269 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
java.lang.AssertionError
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader$WALReaderFSDataInputStream.getPos(SequenceFileLogReader.java:112)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1491)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:57)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:158)
at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:648)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:834)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:750)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:283)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:202)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:275)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:205)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:408)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:301)
2013-07-02 19:52:12,271 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
I found two ways to get it to start up again, the first one I tried was to move away the log splitting directory in hdfs with the following command (strongly discouraged to do this):
$ hadoop fs -mv /hbase/.logs/hbase1.domain.com,60020,1367325077343-splitting /user/hdfs |
$ hadoop fs -mv /hbase/.logs/hbase1.domain.com,60020,1367325077343-splitting /user/hdfs
After some help from #hbase on irc.freenode.net I moved it back and tried starting hbase master with java assertions disabled, and that solved the issue.
To disable assertions in the JVM you make sure that the parameter -da (or -disableassertions) is passed to java when invoked.
I did this by editing /etc/hbase/conf/hbase-site.sh and adding -da to the HBASE_MASTER_OPTS environment variable.
Read more...