I would like to use Apache Beam Java with the recently published Firestore connector to add new documents to a Firestore collection. While I thought that this should be a relatively easy task, the need for creating com.google.firestore.v1.Document objects seem to make things a bit more difficult. I was using this blog post on Using Firestore and Apache Beam for
Tag: apache-beam-io
How to append new rows or perform union on tow PCollection
In the following CSV, I need to append new row values for it. ID date balance 01 31/01/2021 100 01 28/02/2021 200 01 31/03/2021 200 01 30/04/2021 200 01 31/05/2021 500 01 30/06/2021 600 Expected output: ID date balance 01 31/01/2021 100 01 28/02/2021 200 01 31/03/2021 200 01 30/04/2021 200 01 31/05/2021 500 01 30/06/2021 600 01 30/07/2021 999
Beam PAssert messes up the Row
I am exploring testing with Beam and encountered a weird problem. My driver program works as expected, but its test is failing with an error like this: And here is my PAssert code: On the last step of my pipeline, I log the element in question. This is the expected result. When I debugged the test, the problem boiled down
Beam – Error while branching PCollections
I have a pipeline that reads data from kafka. It splits the incoming data into processing and rejected outputs. Data from Kafka is read into custom class MyData and output is produced as KV<byte[], byte[]> Define two TupleTags with MyData. InvalidDataDoFn has application logic that splits MyData data into processing and rejected OutputDoFn converts MyData into KV<byte[], byte[]>. While running