Hacker News

Show HN: Mojibake - a low-level Unicode library written in C

Usage

You don't need to install anything. There are two files (mojibake.c, mojibake.h) to add to your C/C++ project. Download it here: mojibake-amalgamation-027.zip

Examples of normalization, characters count and NFKC casefold

#include <stdio.h>
#include <string.h>
#include "mojibake.h"

void print_string(const char *input, size_t length);

int main(int argc, char *const argv[]) {
    const char *input = "Cafe\xCC\x81";
    size_t length = strlen(input);
    mjb_result result;

    // Normalize example: in NFC e + β—ŒΜ -> Γ© (U+00E9)
    if(mjb_normalize(input, length, MJB_ENC_UTF_8, MJB_NORMALIZATION_NFC, MJB_ENC_UTF_8, &result) != MJB_STATUS_OK) {
        return 1;
    }
    // Cafe + β—ŒΜ (U+0301, COMBINING ACUTE ACCENT) -> CafΓ©
    print_string(input, length);
    // Caf + Γ© (U+00E9, LATIN SMALL LETTER E WITH ACUTE) -> CafΓ©
    print_string(result.output, result.output_size);

    const char *mojibake = "ζ–‡ε­—εŒ–γ‘";
    length = strlen(mojibake);
    // String length example: mjb_string_length counts the number of characters in a
    // string, not the number of bytes.
    printf("\"%s\" encoded in UTF-8 is %zu bytes long, and %zu characters long\n",
           mojibake, length, mjb_string_length(mojibake, length, MJB_ENC_UTF_8));
    mjb_result_free(&result);

    const char *case_input = "Straße";
    // NFKC casefold example: in NFKC casefold, ß -> ss
    if(mjb_nfkc_casefold(case_input, strlen(case_input), MJB_ENC_UTF_8, MJB_ENC_UTF_8, &result) != MJB_STATUS_OK) {
        return 1;
    }
    printf("%s -> %.*s\n", case_input, (int)result.output_size, result.output);
    mjb_result_free(&result);

    return 0;
}

void print_string(const char *input, size_t length) {
    for(size_t i = 0; i < length; i++) {
        unsigned char byte = (unsigned char)input[i];
        if(byte >= 0x21 && byte <= 0x7E) {
            printf("%c", byte);
        } else {
            printf("\\x%02X", byte);
        }
    }
    printf("\n");
}

This outputs:

Cafe\xCC\x81
Caf\xC3\xA9
"ζ–‡ε­—εŒ–γ‘" encoded in UTF-8 is 12 bytes long, and 4 characters long
Straße -> strasse

Design goals

Mojibake aims to be:

  • Small
  • Easy to use
  • Fast
  • Self-contained

Mojibake does:

  • Run in all modern OSes (Linux, macOS, FreeBSD, OpenBSD, NetBSD, Windows 10/11)
  • Pass the official Unicode test suites for supported algorithms
  • Implement all Unicode standard algorithms
  • Satisfy all Unicode Conformance Requirements

Feature highlights

All the C files, together with the Unicode data tables, are concatenated into a single large file and header: mojibake.c and mojibake.h. Zero dependencies.

Text transformation

  • Normalization: NFC/NFD/NFKC/NFKD (mjb_normalize), identifier-oriented NFKC case folding (mjb_nfkc_casefold), plus a fast quick-check (mjb_string_is_normalized) (UAX #15, Unicode 17.0.0)
  • Case conversion: uppercase, lowercase, titlecase, and case folding with full special-casing and conditional mappings (mjb_case)
  • Filtering: strip controls, spaces, or numeric characters while normalizing (mjb_string_filter)

Text analysis

  • Character database: every Unicode Character Database property: category, script and Script_Extensions, block, plane, numeric value, name (mjb_codepoint_character, mjb_codepoint_script_extensions)
  • Segmentation: grapheme clusters, words, sentences, and line-break opportunities (UAX #29, Unicode 17.0.0, UAX #14, Unicode 17.0.0)
  • Bidirectional text: full Unicode Bidirectional Algorithm: paragraph resolution, line reordering, runs (UAX #9, Unicode 17.0.0)
  • Emoji: codepoint properties, sequence analysis, RGI emoji detection
  • Display width: East Asian width and terminal display width, with width-aware truncation (mjb_display_width, mjb_truncate_width)

Sorting and comparison

  • Collation: Unicode Collation Algorithm string comparison and sort keys, in shifted and non-ignorable modes (mjb_string_compare, mjb_collation_key, UTS #10, Unicode 17.0.0)

Security

  • Confusable detection: generate reusable skeletons and check if strings are visually confusable (mjb_confusable_skeleton, mjb_string_is_confusable, UTS #39, Unicode 17.0.0)
  • Identifier validation: XID/ID checks for parser and compiler authors (mjb_string_is_identifier, UAX #31, Unicode 17.0.0)

Integration

  • Encodings: the API accepts and outputs UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE strings, with encoding detection and conversion (mjb_string_encoding, mjb_string_convert_encoding)
  • Parsing and string functions: character-by-character iteration (mjb_next_character) and standard string.h-style helpers (mjb_string_length, and others)
  • Locales: strict BCP 47 language tag parsing (mjb_locale_parse)
  • Embeddable: custom allocators (mjb_set_memory_functions), build-time feature flags to trim table size, a C++17 wrapper (src/cpp/mojibake.hpp), a CLI tool (src/shell), and a WASM + TypeScript API (src/api)
  • Tested: Mojibake uses Attractor as test suite and runs 1.5M+ assertions including the official Unicode conformance suites for supported algorithms
  • Fuzz: Mojibake is fuzzed with libFuzzer over untrusted byte input, AddressSanitizer and UBSan clean

Build-time features

Mojibake can compile out optional feature tables to reduce binary size. Feature macros default to enabled.

#define MJB_FEATURE_CHARACTER_NAMES controls the Unicode character-name tables used by mjb_codepoint_character(...) to fill mjb_character.name. When disabled, the tables are not compiled and mjb_character.name is reported as Codepoint U+XXXX. This will reduce the output of ~30%.

With CMake:

cmake -S . -B build-no-name -DMJB_FEATURE_CHARACTER_NAMES=OFF
cmake --build build-no-name

With the provided Makefile:

make build BUILD_DIR=build-no-name FEATURE_CHARACTER_NAMES=OFF
make test-no-names

API documentation

See API.md or the site for the detailed documentation.

CLI

The src/shell directory builds the mojibake CLI used to test the library. Example usage:

# This outputs "NFC: CafΓ©", e + β—ŒΜ -> Γ©
mojibake nfc $'Cafe\u0301'

# The output an emoji sequence [1] Basic, [2] Fully-qualified of two characters U+263A U+FE0F
mojibake emoji "☺️"

Building from source and contributing

See CONTRIBUTING.md for instructions.

Licenses

Mojibake is released under the MIT License (see LICENSE).

Legalese

Here you can find the very detailed and boring information needed to have this library conformant to the Unicode standard, or at least what I got, at CONFORMANCE_REQUIREMENTS.md.

Thanks

Mojibake is built using the work of extraordinary individuals and teams.

  • Unicode Character Database - Copyright Β© 1991-2026 Unicode, Inc. (see license.txt)
  • Unicode CLDR Project - Copyright Β© 2004-2026 Unicode, Inc. (see LICENSE)
Read on Hacker News ↗ ← Back to News

Comments

No comments yet. Start the discussion.