In the world of NoSQL


Not really NoSQL related, but kind of.  A lot of people uses Scribe for processing large amount of events. Since I had some problems compiling it and since it’s not prepackaged I figured I’d post here how to do it.

It’s extremely important that you install the dependencies before starting!

First you need to install thrift(depends on package: libevent-dev libglib2.0-dev automake libboost-dev libssl-dev) and fb303 (needed to communicate with Scribe):

$ wget https://dist.apache.org/repos/dist/release/thrift/0.9.0/thrift-0.9.0.tar.gz
$ tar -zxf thrift-0.9.0.tar.gz; cd thrift-0.9.0
$ ./configure CPPFLAGS="-DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H"
$ make 
$ sudo make install
$ cd contrib/fb303
$ ./bootstrap.sh
$ make
$ sudo make install

Then install scribe(depends on package: gawk libboost-system-dev libboost-filesystem-dev libtool):

$ git clone git://github.com/facebook/scribe.git
$ cd scribe
$ ./bootstrap.sh
$ ./configure CPPFLAGS="-DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -DBOOST_FILESYSTEM_VERSION=2"
$ make

Make failed while trying to link scribed, complaining about:

file.cpp:(.text.startup+0x23): undefined reference to `boost::system::generic_category()'
file.cpp:(.text.startup+0x2f): undefined reference to `boost::system::generic_category()'
file.cpp:(.text.startup+0x3b): undefined reference to `boost::system::system_category()'

The solution to this was to take the g++-row from just before the error:

g++  -Wall -O3 -L/usr/lib  -lboost_system-mt -lboost_filesystem-mt -o scribed store.o store_queue.o conf.o file.o conn_pool.o scribe_server.o network_dynamic_config.o dynamic_bucket_updater.o  env_default.o  -L/usr/local/lib -L/usr/local/lib -L/usr/local/lib -lfb303 -lthrift -lthriftnb -levent -lpthread  libscribe.a libdynamicbucketupdater.a

and change it a bit (move ”-lboost_system-mt -lboost_filesystem-mt” to the back of the line), and run it manually in the src directory:

$ cd src
$ g++  -Wall -O3 -L/usr/lib  -o scribed store.o store_queue.o conf.o file.o conn_pool.o scribe_server.o network_dynamic_config.o dynamic_bucket_updater.o  env_default.o  -L/usr/local/lib -L/usr/local/lib -L/usr/local/lib -lfb303 -lthrift -lthriftnb -levent -lpthread  libscribe.a libdynamicbucketupdater.a -lboost_system-mt -lboost_filesystem-mt

The ”-DBOOST_FILESYSTEM_VERSION=2” is only needed if you have libboost 1.46 or newer.

Now you have a compiled scribed binary in that directory, either run it directly from there or run make install to install it globally.

Here’s an example config file that just logs to ./logs:

port=1463
max_msg_per_second=2000000
check_interval=3
 
<store>
  category=default
 
  type=file
  fs_type=std
  file_path=./logs/
  base_filename=tmpdata
  max_size=1000000000
</store>

Put it in the scribe directory and name it to local.conf and run ./src/scribed local.conf to start scribe with it.

Read more...

§177 · oktober 18, 2012 · Uncategorized · Kommentarer inaktiverade för Tutorial: Installing Scribe on Debian/Ubuntu · Tags: , , ,



The title says it all. HBase doesn’t perform to well with lots of regions, and I have a table which is rarely used and contains a lot of data, so I needed to up the region size to 16 GB for only that table.

Here’s how to do it:

hbase(main):002:0> disable 'ia'
0 row(s) in 362.8930 seconds
 
hbase(main):003:0> alter 'ia', {METHOD=>'table_att',MAX_FILESIZE=>'17179869184'}
0 row(s) in 0.0380 seconds
 
hbase(main):005:0> enable 'ia'
0 row(s) in 423.0210 seconds

Read more...

§160 · juli 2, 2012 · Uncategorized · Kommentarer inaktiverade för Changing region sizes for a single HBase table ·



HBase crashed for me this night, due to the extra leap second inserted (2012-06-30 23:59:60).

 

When attempting to restart HBase, it just didn’t start. I found this resource for a tip that might work to get it up (although I found it after rebooting my servers, so I didn’t try it):
http://pedroalves-bi.blogspot.com/2012/07/java-leap-second-bug-how-to-fix-your.html

 

All java processes(including all HDFS-related) were using 100% CPU, together with ksoftirq. I turned off ntpd autostart(chkconfig ntpd off), and rebooted the servers, and then started my HBase cluster back up. This solved the issue.

Read more...

§163 · juli 1, 2012 · Hadoop, HBase · 1 comment · Tags: , ,



I found a neat trick to enable a history file for the HBase shell, put the following into ~/.irbrc:

require 'irb/ext/save-history'
 
IRB.conf[:SAVE_HISTORY] = 100
IRB.conf[:HISTORY_FILE] = "#{ENV['HOME']}/.irb-save-history"

This enabled history saving for me when running irb directly, but didn’t work in the HBase shell, so I also added the following to the end of ~/.irbrc:

Kernel.at_exit do
  IRB.conf[:AT_EXIT].each do |i|
    i.call
  end
end

In CentOS you also need to make sure that the package ruby-irb is installed, and in debian the package is named irb1.8.

Read more...

§150 · juni 28, 2012 · HBase · Kommentarer inaktiverade för History in the HBase shell · Tags: , , ,



This is an example on how to import data into hbase with importtsv and completebulkload:

Step 1, run the TSV file through importtsv to create the HFiles:

[root@hbase1 bulktest]# HADOOP_CLASSPATH=$(hbase classpath) sudo -u hdfs -E hadoop jar \
/usr/lib/hbase/hbase-0.90.4-cdh3u3.jar importtsv \
-Dimporttsv.bulk.output=/bulktest-hfiles \
-Dimporttsv.columns=HBASE_ROW_KEY,a:b,a:c bulktest /bulktest-tsv

This will generate HFiles from /bulktest-tsv and store in to /bulktest-hfiles.

I have three columns in the TSV files, first being the row key, second being what I want stored in columnfamily a with qualifier b, and third with qualifier c (this was controlled by importtsv.columns).

After that job is done, you need to change the permissions of /bulktest-hfiles so that the HBase user owns the HFiles, and then run completebulkload so HBase finds the HFiles:

[root@hbase1 bulktest]# sudo -u hdfs hadoop dfs -chown -R hbase /bulktest-hfiles
[root@hbase1 bulktest]# HADOOP_CLASSPATH=$(hbase classpath) sudo -u hdfs -E hadoop jar \
/usr/lib/hbase/hbase-0.90.4-cdh3u3.jar completebulkload /bulktest-hfiles bulktest

HBase should now see the new data. For usage help, run importtsv or completebulkload without any parameters.

References:
http://hbase.apache.org/bulk-loads.html

Read more...

§134 · maj 23, 2012 · HBase · Kommentarer inaktiverade för Bulk load in HBase with importtsv and completebulkload · Tags: , , , , ,



CentOS has a bad tendency to install msdos partition table instead of GPT, which causes problems if you want partitions larger than 2 TB.

This is how I converted from MBR to GPT. It’s worked for me several times, but I wont give out any guarantees, so make sure you’re backed up if you have important data on the disks.

The following is the excerpt from parted /dev/sda:

(parted) unit s
(parted) print
Model: LSI 9750-8i DISK (scsi)
Disk /dev/sda: 5859352576s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 2048s 102402047s 102400000s primary ext4 boot, raid
2 102402048s 122882047s 20480000s primary linux-swap(v1)
(parted) mklabel gpt
Warning: Partition(s) on /dev/sda are being used.
Ignore/Cancel? ignore
Warning: The existing disk label on /dev/sda will be destroyed and all data on this disk will be lost. Do
you want to continue?
Yes/No? yes
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy).
As a result, it may not reflect all of your changes until after reboot.
(parted) mkpart primary ext4 2048s 102402047s
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy).
As a result, it may not reflect all of your changes until after reboot.
(parted) mkpart primary linux-swap 102402048s 122882047s
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy).
As a result, it may not reflect all of your changes until after reboot.
(parted) print
Model: LSI 9750-8i DISK (scsi)
Disk /dev/sda: 5859352576s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 2048s 102402047s 102400000s ext4 primary
2 102402048s 122882047s 20480000s linux-swap(v1) primary


As you can see I just created the same partitions in the GPT table.

Step two was to run grub-install /dev/sda and then reboot the server.

Read more...

§126 · maj 10, 2012 · Filesystem · Kommentarer inaktiverade för Converting from MBR based partitioning to GPT (GUID Partition Table) while online · Tags: , ,



I’ve previously found a great addon to hadoop streaming called ”hadoop hbase streaming” which enables you to use a HBase table as input or output format for your hadoop streaming map reduce jobs, but it’s not been working since a recent API change.

The error it was saying was:

Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.RowResult

I just found a fork of it on github by David Maust that has been updated for newer versions of HBase.

You can find the fork here:
https://github.com/dmaust/hadoop-hbase-streaming
And the original branch here:
https://github.com/wanpark/hadoop-hbase-streaming

Read more...

§98 · maj 2, 2012 · Hadoop, HBase · Kommentarer inaktiverade för Fork of hadoop-hbase-streaming with support for CDH3u3 · Tags: , , ,



I got the following exceptions whenever running heavy map reduce jobs towards my HBase tables:

INFO mapred.JobClient: Task Id : attempt_201204240028_0048_m_000015_2, Status : FAILED
 org.apache.hadoop.hbase.regionserver.LeaseException:
 org.apache.hadoop.hbase.regionserver.LeaseException:
      lease '-8170712910260484725' does not exist
 at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230)
 at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1879)
 at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:601)
 ...

Most oftenly they were severe enough to cause the entire job to fail. This indicated that I needed to raise hbase.regionserver.lease.period, which says how long a scanner lives between calls to scanner.next(). This however didn’t help, apparently you also need to raise hbase.rpc.timeout (the Exception that indicated this was hidden in log level DEBUG, so took a while to realise that).

So, adding the following to hbase-site.xml solved it:

  <property>
    <name>hbase.regionserver.lease.period</name>
    <value>900000</value> <!-- 900 000, 15 minutes -->
  </property>
  <property>
    <name>hbase.rpc.timeout</name>
    <value>900000>/value> <!-- 15 minutes -->
  </property>

Read more...

§90 · maj 2, 2012 · HBase · Kommentarer inaktiverade för HBase scanner LeaseException · Tags: , , ,



tcollector is a nice subset of scripts to graph system performance and store it into OpenTSDB (which in turn stores the data in HBase).

 

Most people seam to install it via puppet, but this is how you do it otherwise. First of all you need to make sure that you have started TSD with –auto-metric so that you don’t have to run mkmetric for every metric that a collector tracks, then you need to run the following commands:

# cd /usr/local
# git clone git://github.com/OpenTSDB/tcollector.git
# cd tcollector
# sed -i 5iTSD_HOST=dns.name.of.tsd startstop
# ./startstop start

It will log into /var/log/tcollector.log. Dependencies (in Debian): git, python-json, python-mysqldb.

Read more...

§73 · februari 28, 2012 · HBase, OpenTSDB · Kommentarer inaktiverade för Installing and configuring tcollector · Tags: , , ,



Searches with leading wildcards stopped working for me after moving to a multicore configuration of Solr, and at the same time removing all the dead entries in my schema.xml.

 

I was getting the following exception whenever I searched with a leading wildcard:

1
2
3
4
5
6
7
Nov 4, 2011 12:14:44 AM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException:
 org.apache.lucene.queryParser.ParseException:
Cannot parse 'field1:*': '*' or '?' not allowed as first character in WildcardQuery
at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:105)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:173)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)

 

After diffing the configs, and schemas, I still couldn’t find anything that suggested why it didn’t work.  Finally I found a forum post which said that to perform leading wildcard searches, you have to have a field type defined using the ”ReversedWildcardFilterFactory”.  So I pasted the text_rev field from the default schema.xml:

<fieldType name="text_rev" class="solr.TextField" positionIncrementGap="100">
  <analyzer type="index">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="0"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.ReversedWildcardFilterFactory" withOriginal="true" maxPosAsterisk="3" maxPosQuestion="2" maxFractionAsterisk="0.33"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="0"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>

and restarted Solr, and it started to work.

 
And this is despite the fact that I’m not using the field type text_rev on any field.

Read more...

§55 · november 4, 2011 · Apache Solr / Lucene · Kommentarer inaktiverade för How to activate Solr / Lucene leading wildcards · Tags: , , ,