Why do we use a streaming Data approach?

Sometimes the incoming data from a source has a massive amount of data that shouldn’t be read into memory all at once. Sometimes the data needs to be filtered or manipulated in some way.

Often the data comes from a paginated API endpoint that can take time to pull in.

When this is the case, imperatively looping through the data can be memory and time intensive. This is why the OAA framework used a streaming data approach in the early stages of the data manipulations.

We can hook into this stream of data and queue up all sorts of changes to be applied to the records in-flight.

We leverage a lightweight, yet robust streaming python library called petl to load source data.