Operators transform Dataflow into a new one. Users can write a program to combine these operators ino a sophisticated dataflow topologies

MAP

Dataflow -> Dataflow

Takes one element and produce a new element.

Example

stream.map(v => v + "sss")

FLATMAP

Dataflow -> Dataflow

Takes one element and produces zero, one, or more elements

Example

// let stream: Dataflow<string> = ....
stream.flatMap(v => v.split("."))

FILTER

Dataflow -> Dataflow

Evaluates a boolean function for each element and retains those for which the function returns true

Example

// let stream: Dataflow<string> = ....
stream.filter(v => v.contains("foo"))

KEYBY

Dataflow -> KeyedDataflow

Logically partitions a stream into disjoint partitions. All records with the same key are assigned to the same partition.

For now, Lightflus does not support keyBy in parallelism.