Error while uploading to S3 from Hadoop using s3a

I am running a java application on kubernetes which upload multiple files from local container to s3 bucket using s3a, but I am getting the below exception in logs and files are not getting uploaded to s3.

Partial files are getting uploaded to s3

   2021-11-30 12:28:44,529 614982 [secor-threadpool-1] ERROR c.pinterest.secor.uploader.Uploader - Error while uploading to S3 : java.lang.RuntimeException: org.apache.hadoop.fs.s3a.AWSClientIOException: copyFromLocalFile(/tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate, s3a://prod-dataplatform-hive/orc_secor_test/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate) on /tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate: com.amazonaws.AmazonClientException: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V
    java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.hadoop.fs.s3a.AWSClientIOException: copyFromLocalFile(/tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate, s3a://prod-dataplatform-hive/orc_secor_test/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate) on /tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate: com.amazonaws.AmazonClientException: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at com.pinterest.secor.uploader.FutureHandle.get(FutureHandle.java:34)
        at com.pinterest.secor.uploader.Uploader.uploadFiles(Uploader.java:117)
        at com.pinterest.secor.uploader.Uploader.checkTopicPartition(Uploader.java:244)
        at com.pinterest.secor.uploader.Uploader.applyPolicy(Uploader.java:284)
        at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:141)
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:133)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.RuntimeException: org.apache.hadoop.fs.s3a.AWSClientIOException: copyFromLocalFile(/tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate, s3a://prod-dataplatform-hive/orc_secor_test/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate) on /tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate: com.amazonaws.AmazonClientException: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V
        at com.pinterest.secor.uploader.HadoopS3UploadManager$1.run(HadoopS3UploadManager.java:63)
        ... 5 more
    Caused by: org.apache.hadoop.fs.s3a.AWSClientIOException: copyFromLocalFile(/tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate, s3a://prod-dataplatform-hive/orc_secor_test/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate) on /tmp/secor_data/message_logs/backup/7_22/DAPI_UNICAST_BOOKING_ACTION/driverapi_transactional_ha/dapi_unicast_booking_action/dt=2021-11-29/hr=12/pk_1_1_00000000000000101414.deflate: com.amazonaws.AmazonClientException: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:144)
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:117)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFromLocalFile(S3AFileSystem.java:2039)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2320)
        at org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:2307)
        at com.pinterest.secor.util.FileUtil.moveToCloud(FileUtil.java:206)
        at com.pinterest.secor.uploader.HadoopS3UploadManager$1.run(HadoopS3UploadManager.java:61)
        ... 5 more
    Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V
        at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:286)
        at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:265)
        at com.amazonaws.services.s3.transfer.internal.UploadImpl.waitForUploadResult(UploadImpl.java:66)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerCopyFromLocalFile(S3AFileSystem.java:2099)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFromLocalFile(S3AFileSystem.java:2037)
        ... 9 more
    Caused by: java.lang.NoSuchMethodError: com.amazonaws.util.IOUtils.release(Ljava/io/Closeable;Lorg/apache/commons/logging/Log;)V
        at com.amazonaws.services.s3.model.S3DataSource$Utils.cleanupDataSource(S3DataSource.java:48)
        at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1863)
        at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1817)
        at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:169)
        at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:149)
        at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:115)
        at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:45)
        ... 4 more

Answer

Looks like this is a bug.

problems caused by AWS SDK classes from the localstack-utils-fat.jar overriding classes defined in the actual Lambda jar/zip itself.

Here’s the version you need with the fix. It sounds like there is a work around:

implemented a partial fix for this that moved the localstack-utils-fat.jar later on the classpath, but this fix only applied to lambdas being run using the docker executor.

Basically, it’s not your fault. It’s a code issue with dependencies overwriting each others function signatures. You need to use the latest localstack-utils-fat.jar and it should fix your issue.