The question to answer
Can a technically successful Spark job detect business errors such as duplicated balances and incorrect point-in-time joins?
Proposed scope
Create synthetic accounts, payments and slowly changing reference data. Use an illustrative reporting definition rather than proprietary reporting rules or real customer data.
Constraints to make explicit
- Define the output grain and accounting meaning before transformations.
- Use decimal types and an explicit rounding policy.
- Keep expected results independent of the transformation being tested.
Your contribution
To complete: identify what you personally designed, implemented and verified, the project version, and any collaborators’ contributions.
Architecture & boundaries
Conceptual starting point, not an implemented architecture.
To complete: replace this overview with the implemented components, data flow, persistence boundaries and failure paths. Link the diagram source to the tested commit.
Engineering decisions
Point-in-time joins
Explain event time versus reporting date, valid-from/to boundaries and the handling of overlapping reference versions.
Decision record to add
Chosen approach · alternative considered · trade-off · supporting code or experiment.
Business validation
Separate schema checks, uniqueness checks and independent reconciliations. Explain which checks can share a common mistaken assumption.
Decision record to add
Chosen approach · alternative considered · trade-off · supporting code or experiment.
Failure policy
Choose rejection, quarantine or controlled correction for invalid records; preserve an auditable reason.
Decision record to add
Chosen approach · alternative considered · trade-off · supporting code or experiment.
How to verify it
Planned scenarios — no test outcomes recorded yet
| Scenario | Acceptance criterion | Evidence to attach |
|---|---|---|
| One account has multiple reference versions | Each payment joins only the applicable version. | Attach synthetic fixtures and expected join cardinalities. |
| Missing reference, null key or overlapping validity window | The documented rejection or quarantine rule is applied. | Show rejected rows and reason codes. |
| A join duplicates balance rows | Independent grain and total checks fail although the job completes. | Attach a failing example, the fix and passing assertions. |
Inspect the evidence
Repository
Not attached
Add the transformation, synthetic dataset generator and independent oracle.
Data contract
Not attached
Attach grain, keys, field definitions and effective-date semantics.
Test report
Not attached
Link a CI run covering duplication, nulls and time-boundary cases.
Results
Not attached
Show before/after output tables and independent reconciliation results.
Reproduction record to add: repository URL, exact commit, tool versions, setup commands, data-generation seed, expected output and the command used to run verification.
Results & measurements
No measurements have been published. Fill in the baseline, observed result, workload and method for each metric; attach raw evidence.
Reconciliation difference
Compare expected and actual totals at agreed business boundaries.
Baseline: not measured · Result: not measured · Evidence: not attached
Rule coverage
List each validation rule, its adversarial fixture and resulting test.
Baseline: not measured · Result: not measured · Evidence: not attached
Runtime and scale
Record input sizes, partitions, Spark configuration and observed runtime.
Baseline: not measured · Result: not measured · Evidence: not attached
Limitations & next steps
This is an engineering demonstration, not regulatory certification. Results must identify their synthetic scope and any financial assumptions requiring domain review.
To complete: list unsupported cases, remaining risks, what the results do not establish, and the next improvement you would make.
Related experience & writing
Read the professional experience summary →
Read the related engineering article →
Professional experience provides context; it is separate from this proposed public demonstration.