How to avoid warning message when read BigQuery data to custom data type: Can’t verify serialized elements of type BoundedSource

I defined a custom data type reference the document here. https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java#L127 And read data from …

Dataflow writing a pCollection of GenericRecords to Parquet files

In apache beam step I have a PCollection of KV<String, Iterable<KV>>>. I want to write all the records in the iterable to the same parquet file. My code …

Dataflow: string to pubsub message

I’m trying to make unit testing in Dataflow. For that test, at the begging, I will start with a simple hardcoded string. The problem is that I would need to transform that string to a pubsub message….

Beam – Error while branching PCollections

I have a pipeline that reads data from kafka. It splits the incoming data into processing and rejected outputs. Data from Kafka is read into custom class MyData and output is produced as KV<byte[], …

No repackaged dependencies when building Apache Beam Cassandra JAR

Trying to compile and use the snapshot for Apache Beam Cassandra JAR. Seems like the build does not pack the Guava dependencies within the JAR. This causes compilation to fail when the JAR is used by …

How do I use MapElements and KV in together in Apache Beam?

I wanted to do something like: PCollection a = whatever; PCollection> b = a.apply( MapElements.into(TypeDescriptor.of(KV.class)) …