Skip to content
Advertisement

Unable to submit concurrent Hadoop jobs

I am running Hadoop 2.7 on my local machine, along with HBase 1.4 and Phoenix 4.15. I have written an application which submits map reduce jobs that delete data in HBase through Phoenix. Each job is run by an individual thread of a ThreadPoolExecutor and looks like this:

JavaScript

Everything is fine if there is only 1 thread in the ThreadPoolExecutor. If more than one such Hadoop jobs are submitted concurrently, nothing happens. As per the logs, the error looks like:

JavaScript

The tasks are submitted using ThreadPoolExecutor.submit() and their status is being checked using the returned future future.isDone().

Advertisement

Answer

The jobs were not being submitted to YARN, but instead running locally from Intellij. Adding the following to the job configuration solved the issue:

JavaScript
User contributions licensed under: CC BY-SA
8 People found this is helpful
Advertisement