I built a Python package to diff large data files - here's why existing tools weren't enough

Every few months at work I run into the same problem. Two systems are supposed to produce identical data exports. A CSV from the old pipeline, a CSV from the new one. Simple enough to check - until the file has 500,000 rows, three sources to compare instead of two, and columns that were renamed somewhere along the way. The usual tools fall apart fast: - Excel - opens maybe 100k rows before giving up - diff / fc - order-dependent, one mismatch per line, useless for structured data - pandas - fine for two files that fit in memory, painful for anything larger, no built-in N-way support - Custom scripts - I've written three. None of them were reusable. So I built duckdiff. What it does duckdiff is a Python package for N-way, order-independent comparison of large structured files - CSV, TSV, and Parquet. It's powered by DuckDB, which means comparisons stream off disk and aren't bounded by RAM. Install it: pip install duckdiff Compare two files from the command line: duckdiff compare old=export_v1.csv new=export_v2.csv --key transaction_id Output: Sources: old: 547,823 rows, 14 columns new: 547,823 rows, 14 columns Matched: 541,200 Mismatched: 6,123 Only in old: 500 Only in new: 0 Not sure which columns to use as --key ? There's a subcommand for that: duckdiff keys a=export_v1.csv It scans the file and tells you which column combinations uniquely identify each row - printing results as it goes so you're not staring at a blank screen: Key column suggestions for 'a': ✓ transaction_id (unique) Suggested: duckdiff compare ... --key "transaction_id" A few things that make it different N-way comparison. Compare 2, 3, or 20 sources in one pass. Not N pairwise diffs. Order-independent. Rows don't need to be sorted. DuckDB handles it. Fuzzy column mapping. If the new pipeline renamed cust_id to customer_id , duckdiff can suggest a mapping - but never applies one silently. You opt in explicitly. Schema flexibility. If sources don't share all columns, --auto-intersect compares only the shared ones and tells you what was dropped. Pre-flight dry-run. --dry-run checks schema compatibility and file sizes without scanning a single row. Useful before running a comparison on a large file. Python API The CLI is a thin wrapper around a clean Python API: from duckdiff import ComparisonSession, ComparisonConfig config = ComparisonConfig(key_columns=["transaction_id"]) with ComparisonSession(config) as session: session.add_source("old", "export_v1.csv") session.add_source("new", "export_v2.csv") result = session.compare() print(result.matched_row_count) print(result.mismatched_row_count) print(result.only_in) Where to find it It's at v0.1.0 - early, but tested (161 tests) and usable. Feedback and contributions are welcome - feel free to open an issue or star the repo on GitHub Top comments (0)