I increased the number of map tasks in hadoop to 64 per TaskTracker, and TaskTracker started to crash every time I launched a map reduce job.
Errors were:
java.lang.OutOfMemoryError: unable to create new native thread |
And:
org.apache.hadoop.mapred.DefaultTaskController: Unexpected error launching task JVM java.io.IOException: Cannot run program ”bash” (in directory ”/data/1/mapred/local/taskTracker/hdfs/jobcache/job_201110201642_0001/attempt_201110201642_0001_m_000031_0/work”): error=11, Resource temporarily unavailable.
Googling for this problem presented the following solutions:
- Increase the heap size for the TaskTracker, I did this by changing HADOOP_HEAPSIZE to 4096 in /etc/hadoop/conf/hadoop-env.sh to test. This did not solve it.
- Increase the heap size for the spawned child. Add -Xmx1024 in mapred-site.xml for mapred.map.child.java.opts. This did not solve it.
- Make sure that the limit of open files is not reached, I had already done this by adding ”mapred – nofile 65536” in /etc/security/limits.conf. This did not solve it.
I decided to sudo to the mapred user and check the ulimits again, what I noticed that was off was:
max user processes (-u) 1024
Adding the following to /etc/security/limits.conf and restarting the TaskTracker solved it:
mapred – nproc 8192
Apparently CentOS limits the number of processes for regular users to 1024 by default.
