Skip to content
Advertisement

Flink: does Flink support abstract operator which can process different data streams with common fields?

Suppose we have multiple data streams and they share some common features.

For example, we have a stream of Teacher and a stream of Student, and they both have an age field. If I want to find out the eldest student or teacher from the realtime stream, I can implement an operator as below.

JavaScript

To find out the eldest Teacher, we need to implement a similar operator as below

JavaScript

But actually these two operators have common process logic, so my idea is to define a parent class, such as People.

JavaScript

Then Student and Teacher can be defined as their child class, and also keep their own fields.

JavaScript
JavaScript

In this case, I can define an operator as below.

JavaScript

But when I try to use this operator to implement a Flink execution topology, it won’t work because of the unmatched data type.

JavaScript

And this is my question, is it possible to make an abstract operator for input streams having common fields?

Advertisement

Answer

This is more a Java than a Flink question:

What you want to do is to make MaxiumAgeFunc parameterized like this

JavaScript

and then use it like this

JavaScript

edit:

btw your function is not working with checkpointing (so will produce wrong results upon recovery from a checkpoint) and I’d rather go with an aggregation function over the global window.

JavaScript
User contributions licensed under: CC BY-SA
1 People found this is helpful
Advertisement