The problem
Understanding a reporting output often means following a column through several SQL aliases, joins and filters. This tool turns an ordered query specification into an inspectable lineage model.
Conceptual summary of the components visible in the source; not a deployment diagram.
What is inside the repository
A separate analysis engine
The lineage package loads query specifications, parses SQL, builds schemas and resolves source-to-sink paths. Column records carry expressions, intermediate hops and influences from filters, joins, grouping and ordering.
Command-line and browser interfaces
The CLI emits text or JSON. The Flask /analyze endpoint returns schemas, graph edges, derivations, ambiguities and sink-column mappings for the browser inspector.
Explicit ambiguity handling
Unqualified references in multi-source queries can produce ambiguity records. A resolution mapping lets a caller choose a qualifier rather than silently treating every interpretation as correct.
A visual inspection workflow
The web template includes zoom/pan, node SQL inspection, column path highlighting, a lineage matrix and PNG export controls. Preview, profile and log tabs remain placeholders.
What this project demonstrates
Data platform tooling
Making complex transformations inspectable rather than relying on screenshots or manual tracing.
Separation of concerns
One analysis model feeds a CLI and a web interface.
Correctness awareness
Preserving intermediate expressions and flagging ambiguous references makes interpretation limits visible.
These are observations about the code and its relevance to engineering work, not claims of production deployment, measured business impact or exclusive authorship.
Inspect the implementation
These links point to the reviewed commit, so the description can be checked against a stable source snapshot.
-
Input format, CLI usage and documented limitations
-
Schema construction and lineage traversal
-
SQLGlot path and heuristic fallback parser
-
Text/JSON output and resolution input
-
Analysis endpoint and graph payload
-
Browser inspection and graph controls
Verification and reproducibility
A local smoke check ran the bundled queries.json through the CLI using the fallback parser, without SQLGlot or Flask installed. It returned the retail_accounts_transactions sink, three output columns and source mappings. This establishes that one sample executes; it is not a SQL-dialect correctness test or a browser test.
Inspect the captured CLI smoke-check output →
Repository command to start with
python -m lineage.cli queries.json --json
Use the repository README for prerequisites and setup. A listed command is not a promise that an unprepared environment will pass its tests.
Limitations and useful next steps
- Static SQL analysis does not prove runtime row counts, business correctness or complete Spark SQL coverage.
- The fallback parser is heuristic; the smoke result exposed naming/aggregation details that need stronger fixtures.
- The inspected tree has no dedicated automated test suite. Web preview/profile/log panels are not implemented runtime instrumentation.
What would strengthen the evidence
- Add golden lineage fixtures for aliases, joins, stars, CTEs, ambiguous columns and Spark-specific expressions.
- Pin dependencies, run both parser paths in CI and add an end-to-end browser example.