In the world of NoSQL


I have been getting the following exception in my HBase master logs:

14/08/15 16:51:53 WARN util.FSTableDescriptors: The following folder is in HBase's root
directory and doesn't contain a table descriptor, do consider deleting it: .logs2

The reason was because I a long time ago had some issues, where I had moved /hbase/.logs to /hbase/.logs2 solve them. Don’t remember what the issue was, but since then I had been getting the above error message.

Apparently HBase WARNs about every directory in /hbase/ that shouldn’t be there, the solution was, obviously enough, to remove /hbase/.logs2.

root@hbase1:~$ sudo -u hdfs hdfs dfs -rm -r -skipTrash /hbase/.logs2

I don’t recommend deleting stuff from /hbase/ unless you know exactly what it is and why you are doing so.

Read more...

§618 · augusti 17, 2014 · Uncategorized · (No comments) ·



Like the subject says, recently when decommissioning a node I needed to restart the NameNode.

After restarting it, it instantly crashed with the following error message:

2014-08-16 11:52:52,528 WARN org.apache.hadoop.ipc.Server: IPC Server handler 3176 on 8020, call org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.blockReport from 1.2.3.4:38596: error: java.lang.AssertionError:  SafeMode: Inconsistent filesystem state: SafeMode data: blockTotal=1044077 blockSafe=1044078; BlockManager data: active=1044088
java.lang.AssertionError:  SafeMode: Inconsistent filesystem state: SafeMode data: blockTotal=1044077 blockSafe=1044078; BlockManager data: active=1044088
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.doConsistencyCheck(FSNamesystem.java:4286)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.isOn(FSNamesystem.java:3960)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.checkMode(FSNamesystem.java:4091)
... cut ...
2014-08-16 11:52:52,749 FATAL org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: ReplicationMonitor thread received Runtime exception.
java.lang.AssertionError:  SafeMode: Inconsistent filesystem state: SafeMode data: blockTotal=1044077 blockSafe=1044078; BlockManager data: active=1044088
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.doConsistencyCheck(FSNamesystem.java:4286)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.isOn(FSNamesystem.java:3960)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.access$1400(FSNamesystem.java:3852)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isInSafeMode(FSNamesystem.java:4380)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3060)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3032)
        at java.lang.Thread.run(Thread.java:722)
2014-08-16 11:52:52,752 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1

I think the most interesting part of the above exception is:
Inconsistent filesystem state: SafeMode data: blockTotal=1044077 blockSafe=1044078; BlockManager data: active=1044088

blockSafe and BlockManager active are both higher than blockTotal, which looks suspicious.

My next step was to stop all DataNodes, and then try to start the cluster again. Same thing happened again every time I retried it.

After a few times I noticed that one DataNode was also crashing together with the NameNode.

2014-08-16 11:50:38,779 WARN org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
    Two block files with the same block id exist on disk:  /data/1/.../blk_1234 and /data/2/.../blk_1234

The solution was to start the NameNode and all DataNodes except the one that was crashing.

After waiting about 10-15 minutes I started the last DataNode, and it came back up without any problems. It seems that the NameNode required all the DataNodes that didn’t have duplicate blocks to settle before it was possible to start the DataNode with duplicate blocks.

Read more...

§604 · augusti 16, 2014 · Uncategorized · (No comments) ·



Not really NoSQL related, but kind of.  A lot of people uses Scribe for processing large amount of events. Since I had some problems compiling it and since it’s not prepackaged I figured I’d post here how to do it.

It’s extremely important that you install the dependencies before starting!

First you need to install thrift(depends on package: libevent-dev libglib2.0-dev automake libboost-dev libssl-dev) and fb303 (needed to communicate with Scribe):

$ wget https://dist.apache.org/repos/dist/release/thrift/0.9.0/thrift-0.9.0.tar.gz
$ tar -zxf thrift-0.9.0.tar.gz; cd thrift-0.9.0
$ ./configure CPPFLAGS="-DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H"
$ make 
$ sudo make install
$ cd contrib/fb303
$ ./bootstrap.sh
$ make
$ sudo make install

Then install scribe(depends on package: gawk libboost-system-dev libboost-filesystem-dev libtool):

$ git clone git://github.com/facebook/scribe.git
$ cd scribe
$ ./bootstrap.sh
$ ./configure CPPFLAGS="-DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -DBOOST_FILESYSTEM_VERSION=2"
$ make

Make failed while trying to link scribed, complaining about:

file.cpp:(.text.startup+0x23): undefined reference to `boost::system::generic_category()'
file.cpp:(.text.startup+0x2f): undefined reference to `boost::system::generic_category()'
file.cpp:(.text.startup+0x3b): undefined reference to `boost::system::system_category()'

The solution to this was to take the g++-row from just before the error:

g++  -Wall -O3 -L/usr/lib  -lboost_system-mt -lboost_filesystem-mt -o scribed store.o store_queue.o conf.o file.o conn_pool.o scribe_server.o network_dynamic_config.o dynamic_bucket_updater.o  env_default.o  -L/usr/local/lib -L/usr/local/lib -L/usr/local/lib -lfb303 -lthrift -lthriftnb -levent -lpthread  libscribe.a libdynamicbucketupdater.a

and change it a bit (move ”-lboost_system-mt -lboost_filesystem-mt” to the back of the line), and run it manually in the src directory:

$ cd src
$ g++  -Wall -O3 -L/usr/lib  -o scribed store.o store_queue.o conf.o file.o conn_pool.o scribe_server.o network_dynamic_config.o dynamic_bucket_updater.o  env_default.o  -L/usr/local/lib -L/usr/local/lib -L/usr/local/lib -lfb303 -lthrift -lthriftnb -levent -lpthread  libscribe.a libdynamicbucketupdater.a -lboost_system-mt -lboost_filesystem-mt

The ”-DBOOST_FILESYSTEM_VERSION=2” is only needed if you have libboost 1.46 or newer.

Now you have a compiled scribed binary in that directory, either run it directly from there or run make install to install it globally.

Here’s an example config file that just logs to ./logs:

port=1463
max_msg_per_second=2000000
check_interval=3
 
<store>
  category=default
 
  type=file
  fs_type=std
  file_path=./logs/
  base_filename=tmpdata
  max_size=1000000000
</store>

Put it in the scribe directory and name it to local.conf and run ./src/scribed local.conf to start scribe with it.

Read more...

§177 · oktober 18, 2012 · Uncategorized · Kommentarer inaktiverade för Tutorial: Installing Scribe on Debian/Ubuntu · Tags: , , ,



The title says it all. HBase doesn’t perform to well with lots of regions, and I have a table which is rarely used and contains a lot of data, so I needed to up the region size to 16 GB for only that table.

Here’s how to do it:

hbase(main):002:0> disable 'ia'
0 row(s) in 362.8930 seconds
 
hbase(main):003:0> alter 'ia', {METHOD=>'table_att',MAX_FILESIZE=>'17179869184'}
0 row(s) in 0.0380 seconds
 
hbase(main):005:0> enable 'ia'
0 row(s) in 423.0210 seconds

Read more...

§160 · juli 2, 2012 · Uncategorized · Kommentarer inaktiverade för Changing region sizes for a single HBase table ·