The High Cost of Mixing Operative Flows, ETLs, and Analytics

Operative data flows are triggered on demand to fulfill immediate business goals. They manage small amounts of data, and a human user may be actively waiting for their completion or be severely impacted by any delays. The objective is to initiate a change within the company or notify an external entity, such as notifying that something needs to be shipped, issuing a transaction receipt, or initiating a user onboarding process. For these operative workflows, you rely on event-driven runtimes like Hatchet, Temporal, and in some cases, n8n. These tools are designed to easily call external APIs and to get workflows up and running quickly, providing clear visibility into the steps involved and the systems they interact with.

ETLs, or traditional data pipelines, are processes designed to synchronize data between different systems. While this may be a somewhat controversial and simplistic view, their job is to retrieve data from various sources and prepare it for consumption by another system. For example, a data pipeline might run backups or extract data from CRM and product analytics platforms, such as Mixpanel or Google Analytics. It then cleans and formats this data to build market attribution and conversion funnel analytics in a separate system. These are batch processes that handle large data loads and have higher latency. They are managed by orchestrators like Dagster, Airflow, or Databricks (Spark), which focus on asset-centric, scheduled or event-triggered runs.

Data analytics should feed on the outputs of ETLs and Operative data flows. During operative data flows, you log data in a structured format, which is then loaded into the staging layer, also known as the bronze layer if following the medallion architecture. Alternatively, you can use an ETL to load these operative data logs into these same layers at scheduled intervals. After that, you use data warehouses and query engines like Snowflake, BigQuery, ClickHouse, or Databricks SQL Serverless to query that stored data. This allows you to use the outputs from ETLs, their metadata, or the operative logs to generate insights for human (or AI!) analysis and monitoring.

Contracts, recovery, and AI traceability

Regardless of the flow, you have to ensure maintainability, traceability, and error recovery:

Focus on data contracts: In all cases, data contracts are essential to keep the data from diverging, detect upstream changes, and ensure that data analytics remain accurate. This is vital because changes in the schema of an upstream dependency or modifications to property calculation formulas will lead to incorrect results in data analysis.
Track AI models and prompt versions: Equally important in modern stacks is tracking the model and version of prompts (or AI Agent if using OpenClaw or similar frameworks) that generate data. A change of version will most likely lead to a data distribution change, and it is vital to preserve such information to avoid extracting wrong conclusions during data analytics and insights.
Traceability and error recovery per domain: In operative workflows, error recovery requires deterministic retries, step timeouts, and rate-limit handling. In ETLs, it means backfill support, audit logs, and data quality checks. In analytics, it means monitoring pipeline metadata and freshness rather than running diagnostics ad hoc.

Why you should not mix them

With my approach, you end up with systems that should be disjoint and decoupled, thus avoiding a distributed monolith (the worst nightmare!). While it may be tempting to centralize the business logic for certain processes, it's crucial that both systems operate in isolation during execution. You don't want an Operative Flow being blocked because an ETL is running and updating a dataset (distributed locks and business operations delayed, ugh!). Additionally, creating shared contact points, such as database tables or datasets between both systems, can introduce synchronization issues or, even worse, race conditions.

Dedicated operative runtimes (e.g., Hatchet, Go/Rust micro-workers) handle stateful orchestration, rate limits, and external API side-effects with millisecond overhead, whereas distributed Spark clusters have fixed scheduling latency and higher base cost.

As always, you can try to use one tool for all, or squeeze it into another domain, but it will not work properly. These are the typical recipes for failure:

Using an ETL orchestrator for operative workflows: You end up with delays (slow flows), high costs if billed per invocation, or compute usage taken by very slim tasks. It's like using a bazooka to kill flies.
Using operative workflow tools for ETLs: You run into endless individual executions, a lack of backfill support, absent data insights forcing to build your own, challenges in tracking data quality, poor integration with data enrichment, and the absence of audit logs.
Using a data warehouse for ETLs and operative loops: You end up with high costs and create the Reverse ETL trap: In this scenario, teams load raw data, run SQL transformations in Snowflake or BigQuery, and then pay another tool (like Hightouch or Census) to push the results back to systems like Salesforce or Stripe. Instead, operative workflows should execute these side effects directly at the time of ingestion.

This separation also applies to live streams. Running data analytics on live flows can introduce noise and complicate the reasoning about current status versus past data, and it's rarely necessary. While Apache Flink offers this capability, most product teams do not need to build these use cases.

Tools like Flink or Kafka Streams are really good for complex sliding-window analytics, such as in ad-tech attribution, algorithmic trading, or fraud telemetry. However, most SaaS and B2B teams I see adopt streaming engines for the wrong reasons: they often only need to create reactive side effects, like triggering alerts, sending emails, or updating caches. This represents an operational workflow, not a stream analytics problem.

Running complex analytics on live streams forces heavy state management onto engineers, including handling watermarks, late-arriving data, and state checkpointing. Adopting data analytics as the process to run solely on clean stored data separates state recovery from business intelligence. Generally, data analytics are meant for human interpretation, and monitoring should be integrated into these processes with alerts and escalation protocols to ensure issues are detected early.

Keep raw data outside the warehouse

My recommendation is to use a medallion architecture, which allows you to keep raw and historical data in the staging layer (bronze layer) in case you ever need reprocessing. Then, the data is cleaned outside of the data warehouses with batch processing ETL tools. Only after normalizing the data, you load it into BigQuery or Snowflake, ensuring that only clean data is stored there with appropriate partitions, indexes, TTLs, etc. Dealing with raw, dirty data can be challenging and expensive to clean within a data warehouse. In contrast, processing it outside of BigQuery and Snowflake is much easier and more cost-effective.

This addresses the common modern data stack dogma: "Storage is cheap, just ELT everything into Snowflake first." Storing raw data on S3/GCS is fine given its low cost for volume data, but warehouse compute to repeatedly parse, deduplicate, and unnest raw JSON is not. This is why you do not want to do that work in data warehouses, but rather clean the data in S3/GCS and only then ingest it into warehouses.

The winning architecture is relying on Open Table Formats (Apache Iceberg / Delta Lake): write raw Bronze into object storage, run lightweight normalization workers outside the warehouse, write a partitioned (or clustered) silver layer Iceberg/Delta tables, and use Snowflake Polaris or BigQuery BigLake to query those tables directly without paying ingestion compute taxes.

Closing remarks

"The core rule is simple: do not make one runtime responsible for three different jobs."

Operative flows are part of the product or business operation. They call APIs, trigger business actions, retry failed steps, and often have a person or external system waiting for the result. ETLs have a different job: they move data between systems, normalize it, and make it reusable. Analytics comes after that. It should work from stored, cleaned, traceable data so people can understand what happened and make decisions from it.

When these concerns are mixed, the system becomes harder to reason about. A batch job can delay a business action. A workflow engine can be forced to process historical datasets it was never designed to manage, with weak backfills, poor auditability, and little visibility into data quality. A warehouse can become an expensive place to clean raw data and trigger business actions such as sending emails, updating CRMs, or calling payment systems. Separating the three does not make the architecture more complex. It makes each part easier to understand, cheaper to operate, and safer to change.