How AI is applied across API Evangelist and APIs.io. Read my AI disclosure →
API Evangelist API Evangelist
Discovery
Learnings
Guidance
Toolbox
Alignment
API Evangelist LLC

Decoupling a Data Pipeline From the Tables It Was Never Supposed to Depend On

calendar_today November 3, 2026 person Rahul Narakula domain dzone

I built a pipeline framework on top of a convenience feature, and for about a year it looked like the best decision in the design. The framework generates analytics pipelines from declarative configuration: a user describes the events, stages, and metrics they care about, and the framework produces the Spark jobs, output tables, and dashboards. To make those pipelines incremental, I required every upstream source to be a table with change data feed enabled.

open_in_new Read original post