Alex TarverdyDATA ENGINEERING
Data correctness

A pipeline can be green and still be wrong.

Why successful execution is only one part of trusting a data pipeline.

Source data passes through transformation and validation before reporting.
Conceptual illustration; details and guarantee boundaries are explained below.

Execution is a technical signal

A green job tells you that execution satisfied its technical success conditions. It does not tell you whether the output represents the business question correctly.

Consider a payment table joined to an account table. If the account table contains multiple versions per account and the join ignores effective dates, each payment may appear multiple times. The job can finish successfully and still inflate the resulting total.

Start with the grain

Before writing the join, define what one output row represents. Is it a transaction, an account at a point in time, or a daily balance? Then ask whether each transformation preserves that grain or changes it intentionally.

  • Check expected key uniqueness at each stage.
  • Validate join cardinality and unmatched records.
  • Reconcile totals at meaningful business boundaries.
  • Test nulls, missing reference data and effective-date boundaries.

Make expectations executable

A useful validation states an expectation precisely enough to fail. “The data looks reasonable” is weaker than “this dataset has one row per account and reporting date.” Technical checks and domain checks should complement each other.

Domain checks need agreed definitions. A reconciliation can look convincing while comparing two outputs derived from the same incorrect assumption.

The practical takeaway

Give correctness a place in the design, implementation and review process. Ask what would make the output wrong even if every component ran successfully. That question often exposes the most consequential gaps.

This is a conceptual engineering note. The payment example is synthetic and describes no employer’s system.

Let’s work together

Have a data challenge
or a role in mind?

Let’s discuss the problem, the team and where I could contribute.

Start a conversation

What are you working on?

Tell me about your team, a role or a data challenge.

20–3,000 characters. Please leave out sensitive data and credentials.

Your message and reply address are sent to Alex by email. Cloudflare checks for spam; Resend delivers the email. Privacy details.

Prefer email? [email protected]