In the world of NoSQL


I increased the number of map tasks in hadoop to 64 per TaskTracker, and TaskTracker started to crash every time I launched a map reduce job.

 

Errors were:

java.lang.OutOfMemoryError: unable to create new native thread

And:

org.apache.hadoop.mapred.DefaultTaskController: Unexpected error launching task JVM java.io.IOException: Cannot run program ”bash” (in directory ”/data/1/mapred/local/taskTracker/hdfs/jobcache/job_201110201642_0001/attempt_201110201642_0001_m_000031_0/work”): error=11,  Resource temporarily unavailable.

 

Googling for this problem presented the following solutions:

  1. Increase the heap size for the TaskTracker, I did this by changing HADOOP_HEAPSIZE to 4096 in /etc/hadoop/conf/hadoop-env.sh to test.  This did not solve it.
  2. Increase the heap size for the spawned child.  Add -Xmx1024 in mapred-site.xml for mapred.map.child.java.opts.  This did not solve it.
  3. Make sure that the limit of open files is not reached, I had already done this by adding ”mapred – nofile 65536” in /etc/security/limits.conf.  This did not solve it.

I decided to sudo to the mapred user and check the ulimits again, what I noticed that was off was:

max user processes              (-u) 1024

 

Adding the following to /etc/security/limits.conf and restarting the TaskTracker solved it:

mapred – nproc 8192

 

Apparently CentOS limits the number of processes for regular users to 1024 by default.

§50 · oktober 20, 2011 · Hadoop · Tags: , , , · [Print]

Comments are closed.