For years, data pipelines have been synonymous with ETL (Extract, Transform, Load): extract data from one place, transform it, and load it somewhere else. That model worked well when analytics workflows were mostly batch driven and isolated from the rest of the business.
But that world has changed.
Now, organizations use data platforms to power operational decisions, business processes, and Realtime actions—not just analytics dashboards. As a result, pipelines are evolving beyond traditional ETL into something more flexible, interactive, and human aware.
In Microsoft Fabric, we’re seeing this shift firsthand.
From data movement to end-to-end workflows
Modern pipelines don’t just move data—they coordinate work.
Customers increasingly use pipelines to:
- Trigger notebooks and data preparation jobs.
- Refresh semantic models and downstream assets.
- Call external systems and APIs.
- Notify stakeholders when something completes—or fails.
- Wait for a decision before continuing.
In other words, pipelines are becoming end-to-end workflows that span systems, teams, and decisions.
This shift reflects a broader trend: data engineering workflows are no longer isolated from the business. They’re deeply connected to how organizations operate day to day.
Why ETL alone is no longer enough
Traditional ETL pipelines assume a simple model:
- Everything is automated
- Everything runs quickly
- Failures are handled by engineers looking at logs
In practice, many real-world workflows don’t fit that mold.
Some steps require human input—for example, approving a data refresh, validating a business rule, or signing off on a downstream action. Other workflows are long running, spanning hours or days, and need to be visible to people who aren’t data engineers.
When pipelines lack these capabilities, teams often stitch together external tools, scripts, and notification systems. That approach works—but it comes with tradeoffs:
- Fragmented monitoring
- Lost lineage
- Harder debugging
- More operational overhead
Bringing these workflows into the pipeline helps teams keep orchestration, visibility, and governance in one place.
Pipelines that pause for human input
A clear sign of this shift is the growing demand for human-in-the-loop workflows.
Examples we hear from customers include:
- “Pause the pipeline until a data owner approves this step.”
- “Wait for sign‑off before refreshing a production model.”
- “Notify the business team and continue only after confirmation.”
These aren’t edge cases—they’re common patterns in regulated, business critical environments.
Supporting this directly in pipelines allows teams to:
- Gate execution safely
- Track approval status alongside technical steps
- Avoid fragile, external handoffs
- Maintain end-to-end observability
Instead of pipelines being purely automated assembly lines, they become collaborative workflows that blend automation with accountability.
Introducing Approval activity (Preview)
To support these scenarios natively, Fabric Data Factory is introducing an Approval activity that brings human decision points directly into pipeline execution.
Figure: Approval activity with settings for the Outlook 365 Email type of approval request.
Capabilities with the activity pipeline:
- Pause execution and wait for an explicit approval or rejection decision.
- Route execution dynamically based on the outcome (approved, rejected, or timed out).
- Send approval requests through familiar channels like Outlook or Teams.
- Enable reviewers to take action directly from the monitoring experience.
This transforms pipelines from purely technical orchestration into full business workflows—where data movement, validation, and human judgment are modeled together in a single, auditable process.
To learn more about the Approval activity, refer to the Approval activity documentation.
Lowering the barrier to business actions
Another important evolution is who can build and understand pipelines.
As pipelines take on more business responsibility, they need to be accessible to more than just API experts. Many workflows involve calling external systems—placing an order, submitting a request, triggering an operational action—but writing and maintaining raw REST calls can be intimidating and error-prone.
We’re seeing strong demand for low code abstractions that:
- Focus on the action, not the endpoint.
- Surface required parameters automatically.
- Reduce the need to understand API schemas.
- Make workflows easier to read and maintain.
This shift helps pipelines serve both technical and business users, without sacrificing control or transparency.
Observability for long running workflows
As pipelines evolve beyond short lived ETL jobs, observability becomes even more important.
When a pipeline runs for minutes or hours—or waits for days for an approval—users need clear answers to simple questions:
- What’s the status?
- What’s blocking progress?
- Who needs to take action?
- What happens next?
Traditional log-centric views aren’t enough, especially for stakeholders who aren’t focused on code or infrastructure. Monitoring experiences need to surface workflow state, not just execution details.
This is a key part of making pipelines usable for operational and business critical scenarios.
What this means for the future of Pipelines
The evolution beyond ETL isn’t about replacing data engineering—it’s about expanding what pipelines can do.
Pipelines are becoming:
- Orchestrators, not just movers of data
- Bridges between systems, teams, and decisions
- First-class workflow engines for analytics driven operations
In Microsoft Fabric, this evolution is shaping how we think about orchestration, activities, and monitoring—so pipelines can support the full lifecycle of modern data and business workflows.
As customers continue to push pipelines into new scenarios, one thing is clear: the future of pipelines isn’t just automated—it’s collaborative, observable, and deeply connected to how businesses run.
Learn more about Business Process Management (BPM) in pipelines, refer to the BPM documentation. We’d love your feedback in the comments section or suggest an idea at Fabric Ideas.