The question to answer
Can an engineer inspect a large dataset and understand schema or data problems without loading the entire file into memory?
Proposed scope
Build a new public tool using generated datasets, or attach an existing implementation only if it can be published. Choose and document the implementation language and interface.
Constraints to make explicit
- Separate metadata inspection from full scans and previews.
- Bound preview size and document sampling behavior.
- Default to read-only inspection; any export or editing needs an explicit safety model.
Your contribution
To complete: identify what you personally designed, implemented and verified, the project version, and any collaborators’ contributions.
Architecture & boundaries
Conceptual starting point, not an implemented architecture.
To complete: replace this overview with the implemented components, data flow, persistence boundaries and failure paths. Link the diagram source to the tested commit.
Engineering decisions
Resource limits
Choose lazy reads, batching or row-group access. Describe where memory can still grow, including unusually large individual records.
Decision record to add
Chosen approach · alternative considered · trade-off · supporting code or experiment.
Correctness and usability
Define null/empty-string behavior, CSV dialect handling and type representation. Explain how truncation is made visible.
Decision record to add
Chosen approach · alternative considered · trade-off · supporting code or experiment.
Failure handling
Describe behavior for corrupt files, unsupported encodings and interrupted operations.
Decision record to add
Chosen approach · alternative considered · trade-off · supporting code or experiment.
How to verify it
Planned scenarios — no test outcomes recorded yet
| Scenario | Acceptance criterion | Evidence to attach |
|---|---|---|
| Dataset exceeds the preview memory budget | Preview remains bounded under the stated workload and limits. | Attach memory measurements and input-generation commands. |
| Quoted CSV fields, newlines, nulls and schema mismatch | The tool preserves or clearly reports the selected parsing semantics. | Attach golden fixtures and expected output. |
| Truncated Parquet file or interrupted operation | The tool reports a useful error and does not modify the input. | Attach failure tests and input integrity assertions. |
Inspect the evidence
Repository
Not attached
Add the tool, license, supported formats and runnable setup.
Demo
Not attached
Attach a screenshot or short recording using synthetic data.
Test report
Not attached
Link parser, corruption and resource-bound tests.
Results
Not attached
Publish the benchmark inputs, hardware, commands and raw measurements.
Reproduction record to add: repository URL, exact commit, tool versions, setup commands, data-generation seed, expected output and the command used to run verification.
Results & measurements
No measurements have been published. Fill in the baseline, observed result, workload and method for each metric; attach raw evidence.
Peak memory
Measure resident memory for several file sizes and document the measurement method.
Baseline: not measured · Result: not measured · Evidence: not attached
Time to first useful preview
Measure metadata and preview latency separately from any full scan.
Baseline: not measured · Result: not measured · Evidence: not attached
Debugging workflow
Describe the exact task and compare repeatable steps; do not invent productivity percentages.
Baseline: not measured · Result: not measured · Evidence: not attached
Limitations & next steps
No bounded-memory or performance claim is established until measured. Any relationship to earlier internal tooling must be described accurately without publishing restricted code.
To complete: list unsupported cases, remaining risks, what the results do not establish, and the next improvement you would make.
Related experience & writing
Read the professional experience summary →
Professional experience provides context; it is separate from this proposed public demonstration.