Skip to content
Advertisement

Tag: apache-beam-io

Beam PAssert messes up the Row

I am exploring testing with Beam and encountered a weird problem. My driver program works as expected, but its test is failing with an error like this: And here is my PAssert code: On the last step of my pipeline, I log the element in question. This is the expected result. When I debugged the test, the problem boiled down

Beam – Error while branching PCollections

I have a pipeline that reads data from kafka. It splits the incoming data into processing and rejected outputs. Data from Kafka is read into custom class MyData and output is produced as KV<byte[], byte[]> Define two TupleTags with MyData. InvalidDataDoFn has application logic that splits MyData data into processing and rejected OutputDoFn converts MyData into KV<byte[], byte[]>. While running

Advertisement