gomarc: MARC21 for Go, 4x-11x faster than pymarc

If you work with library data, you work with MARC21 - the length-prefixed binary record format catalogues have run on since the 1960s, complete with a directory of field offsets, subfield delimiters, and a pre-Unicode character encoding called MARC-8 that needs a lookup table with thousands of entries to decode. In Python that problem is solved: pymarc is mature, complete, and pleasant to use. In Go it wasn't. gomarc is a port of pymarc to Go. It covers the binary MARC21 transmission format, MARC-8 to Unicode conversion, MARCXML, and MARC-in-JSON - and on real catalogue exports it runs 4x to 11x faster than the library it was ported from. go get github.com/beyto1974/gomarc@v0.1.0 It reads like pymarc If you know pymarc, you already know this API. Iterate records, pull the fields you want: reader := marc.NewReader(f) for { record, err := reader.Next() if errors.Is(err, io.EOF) { break } if err != nil { log.Println(err) // permissive: bad records are skipped, not fatal continue } title, _ := record.Title() fmt.Println(title) } Title , Author , ISBN , ISSN , Subjects , Publisher , PubYear and more are there as methods. For anything else, go at the tag and subfield directly: value, ok := record.Get("245").Subfield("a") for _, f := range record.GetFields("650") { fmt.Println(f) } Build records, modify them, write them back: record.Get("245").SetSubfield("a", "The Zombie Programmer : ") writer := marc.NewWriter(out) writer.Write(record) And convert to the formats the rest of your stack can actually read - both use UTF-8 throughout instead of MARC-8, so standard tooling works: s, err := record.AsJSON() // MARC-in-JSON records, err := marc.ParseXML(r) // MARCXML Large MARCXML files stream one record at a time via marc.NewXMLReader rather than loading into memory. The numbers Two real catalogue exports - 138,076 records, 166 MB. AMD Ryzen 5 3600, Go 1.25.12, CPython 3.13.5, gomarc v0.1.0, pymarc 5.4.0, single-threaded, fastest of 3 repetitions. 50,000 records, 58 MB | scenario | gomarc | pymarc | speedup | gomarc rec/s | pymarc rec/s | |---|---|---|---|---|---| | parse (MARC-8 to Unicode) | 1.96 s | 21.54 s | 11.0x | 25,545 | 2,321 | | parse (force UTF-8) | 0.82 s | 5.10 s | 6.2x | 61,096 | 9,813 | | parse + field access | 2.02 s | 23.01 s | 11.4x | 24,758 | 2,173 | | parse + write MARC21 | 4.69 s | 26.01 s | 5.5x | 10,661 | 1,922 | | parse + write MARCXML | 3.23 s | 36.78 s | 11.4x | 15,481 | 1,360 | | parse + write MARC-in-JSON | 8.18 s | 32.22 s | 3.9x | 6,110 | 1,552 | 88,076 records, 108 MB | scenario | gomarc | pymarc | speedup | gomarc rec/s | pymarc rec/s | |---|---|---|---|---|---| | parse (MARC-8 to Unicode) | 3.33 s | 35.74 s | 10.7x | 26,459 | 2,464 | | parse (force UTF-8) | 1.53 s | 7.12 s | 4.7x | 57,662 | 12,370 | | parse + field access | 3.72 s | 34.55 s | 9.3x | 23,663 | 2,550 | | parse + write MARC21 | 5.48 s | 38.45 s | 7.0x | 16,059 | 2,290 | | parse + write MARCXML | 5.62 s | 53.66 s | 9.5x | 15,674 | 1,641 | | parse + write MARC-in-JSON | 7.09 s | 45.67 s | 6.4x | 12,430 | 1,929 | In practical terms: a full MARC-8 parse of 88,076 records drops from 36 seconds to 3.3. A catalogue-to-MARCXML conversion drops from 54 seconds to 5.6. Peak memory stays between 11 and 23 MB - gomarc streams, so file size doesn't drive memory. Where the win is biggest is the MARC-8 decode path, which is where real catalogue data spends most of its time. Forcing UTF-8 instead narrows the gap to roughly 5x, because the per-character table lookups that dominate the Python run mostly disappear. Why you can trust those numbers Speed claims about a port are cheap. A benchmark of two libraries is really a benchmark of two programs someone wrote, and if one parses lazily while the other eagerly materialises everything, the ratio means nothing. So the benchmark suite proves equivalence before it reports a single timing: Identical record acceptance. Both libraries run permissively and report the same counts - 50,000 and 88,076 records, zero errors - so they agree on exactly which records are well-formed. Identical decoded text. Summing the codepoint length of every extracted title, author, ISBN and subject gives the same total, 671,999, from both libraries. MARC-8 decoding agrees character for character. Byte-identical output. Read a file with each library, write every record back out as binary MARC21, and compare: $ cmp py.marc go.marc && echo BYTE_IDENTICAL BYTE_IDENTICAL Every leader byte, every directory offset, every field terminator. That last one is the practical point: if you're swapping gomarc into a pipeline that pymarc currently feeds, the bytes coming out the far end don't change. The harnesses, runner, raw per-run JSON and generated tables are all published alongside the library, so you can re-run the whole thing: REPS=3 ./run.sh python report.py results.jsonl The report includes a spread column - slowest repetition over fastest - because a benchmark that publishes an estimator without its noise isn't worth much. Where it stands today gomarc is at v0.1.0. Two things worth knowing before you adopt it: - to_unicode=false (pymarc'sRawField mode) isn't implemented yet. If you need raw undecoded bytes rather than Unicode strings, that path returns an error. - MARC-in-JSON encoding is the least optimised corner. At 3.9x-6.4x it's still comfortably ahead of pymarc, but it allocates far more than the other paths and is the obvious next thing to tune. Everything else - binary MARC21 read and write, MARC-8 conversion, MARCXML read/write/stream, MARC-in-JSON read/write, field and subfield manipulation - is covered and tested. Try it go get github.com/beyto1974/gomarc@v0.1.0 Repository, docs and full benchmark suite: github.com/beyto1974/gomarc Issues and pull requests welcome - particularly if you have MARC data that breaks it. Top comments (0)