Skip to content
Advertisement

Checkpoint with spark file streaming in java

I want to implement checkpoint with spark file streaming application to process all unprocessed files from hadoop if in any case my spark streaming application stop/terminates. I am following this : streaming programming guide, but not found JavaStreamingContextFactory. Please help me what should I do.

My Code is

JavaScript

Advertisement

Answer

You must use Checkpointing

For checkpointing use stateful transformations either updateStateByKey or reduceByKeyAndWindow. There are a plenty of examples in spark-examples provided along with prebuild spark and spark source in git-hub. For your specific, see JavaStatefulNetworkWordCount.java;

User contributions licensed under: CC BY-SA
9 People found this is helpful
Advertisement