yasbd-lib vs PySBD: two philosophies of sentence boundary detection

Sentence boundary detection sounds boring. Split on . ? ! , done, right? Anyone who has tried knows otherwise. Abbreviations, decimals, URLs, nested quotes, ellipsis, legal citations, biomedical jargon-each one turns "split text" into a language-specific puzzle.

Two Python libraries tackle this problem with different philosophies. pysbd has been the go-to since 2020 with 22 languages, ported from Ruby’s pragmatic segmenter [1]. yasbd-lib is newer, covers 39 languages, and takes a different architectural approach.

This is the difference between protecting boundaries and finding them.

Architecture: Mutation vs. Pointers

To understand how these engines behave on large datasets, we have to look at how they treat input strings.

PySBD: The Transformation Pipeline

PySBD operates as a multi-stage transformation pipeline [1]. It treats text as a mutable object that must be modified before it can be split [1]. To prevent punctuation within abbreviations, numbers, or URLs from triggering false splits, PySBD applies regular expressions to replace characters with placeholder tokens [1].

flowchart TD
    A[Raw Input Text] --> B["Replace . with {} / Mask URLs"]
    B --> C[Run Rule Engine]
    C --> D[Split on Splitting Marks]
    D --> E[Reverse Replacement / Restore Text]
    E --> F[Extract Text Segments]

The structural consequence: the original string layout is transformed during processing. Because the text is modified mid-flight, calculating exact character offsets (spans) relative to the original uncleaned text requires a post-processing reconstruction step [1]. If you enable text cleaning (clean=True), PySBD raises an error when requesting character spans because it cannot guarantee coordinate matching after modification [1].

yasbd-lib: The Query Planning Approach

yasbd-lib treats text as immutable [2]. It does not modify the raw string [2]. Instead, its architecture resembles a database query planner-generating candidate coordinate arrays and using language-specific filters to narrow down boundary slices [2].

flowchart LR
    A[Raw Input String] --> B[Pass 1: Aggressive Candidate Identification]
    B --> C[Pass 2: Modular Filter Elimination]
    C --> D[Project Slices / Return Index Pointers]

By evolving integer pointers rather than altering text strings, yasbd-lib maintains context of the source layout throughout processing [2]. Token spans are tracked as a first-class structural signal during parsing rather than reconstructed afterward [2].

Deep-Dive Feature Breakdown

1. Spans and Character Offsets

Because PySBD does not natively track indices during its transformation phase, calculating character offsets requires a post-processing step that searches the original document to locate each sentence [1].

The PySBD Reconstruction Step

PySBD reconstructs spans by scanning the original text for each sentence (source):

def sentences_with_char_spans(self, sentences):
    sent_spans = []
    prior_end_char_idx = 0
    for sent in sentences:
        for match in re.finditer(
            ' {0}\s* '.format(re.escape(sent)), self.original_text
        ):
            match_str = match.group()
            match_start_idx, match_end_idx = match.span()
            if match_end_idx > prior_end_char_idx:
                sent_spans.append(
                    TextSpan(match_str, match_start_idx, match_end_idx)
                )
                prior_end_char_idx = match_end_idx
                break
    return sent_spans

The Trade-off: This reconstruction performs repeated searches over the original document. In pathological cases-such as documents with many repeated sentences-this can approach quadratic behavior. For typical use cases, the overhead is manageable, but it does add runtime cost on longer texts.

The yasbd-lib Approach

In yasbd-lib, spans are produced natively during boundary detection [2]. It emits boundary allocations dynamically, avoiding lookback overhead [2]. The library also provides an adapter layer for migrating from PySBD [2]:

from yasbd.utils.pysbd_adapter import Segmenter

seg = Segmenter(language="ja")
res = seg.segment(
    '田中さんは「準備は完了しました」そう言って部屋を出た。U.S.A.の経済政策 is complex.'
)
print(res)
# ['田中さんは「準備は完了しました」そう言って部屋を出た。', 'U.S.A.の経済政策 is complex.']

2. Memory and Streaming

PySBD processes text as complete string buffers [1]. yasbd-lib provides abstractions for memory-constrained environments through lazy evaluation via ParagraphStream and StreamCleaner [2]:

from yasbd.utils.cleaner import StreamCleaner
from yasbd import BoundaryDetector

cleaner = StreamCleaner("Helloworld.   This ismessy.")
detector = BoundaryDetector(lang="en")
sentences = list(detector.segment(cleaner))
print(sentences)
# ['Hello world.', 'This is messy.']

3. Resource Management

Under the hood of the BoundaryDetector pipeline, yasbd-lib manages execution rules using a 5-entry LRU cache (_MAX_CACHED_RULES = 5) [2]. When using automatic language identification (lang="auto"), if confidence drops below the threshold (_MIN_CONFIDENCE = 0.8), the module logs an informational message rather than masking the failure [2]:

# From boundary_detector.py
if lang == "auto":
    lang, confidence = classify_language(snippet)
    if confidence < _MIN_CONFIDENCE:
        log_info(
            self.verbose,
            "Low confidence ({:.2f}) for detected lang {!r} in auto mode",
            confidence,
            lang,
        )

Additionally, yasbd-lib supports preserving token boundaries inside parentheses or brackets via preserve_quote_and_paren=True [2].

Maintenance Status: A Critical Consideration

The architectural differences matter, but there’s another factor: PySBD is effectively unmaintained. As of December 2025, an open issue (#135) [9] notes that the repository has seen no recent updates, with multiple open PRs from contributors and a maintainer who has seemingly abandoned the project. The issue author explicitly requested archiving the project to signal to downstream users that they should no longer incorporate it [9].

The maintenance situation has real consequences. Consider these unresolved issues:

• [Issue #79] [10] - Infinite Loop (October 2020):

  segmenter = pysbd.Segmenter(language="en", clean=False)
  text = "..[111 111 111 111 111 111 111 111 111 111]"
  segmenter.segment(text)  # Hangs indefinitely
  

  The problem is catastrophic backtracking in NUMBERED_REFERENCE_REGEX. The maintainer acknowledged it in February 2021, saying "Need to dug into details" [10]. Over four years later, it remains unresolved.

• [Issue #92] [11] - Catastrophic Backtracking in HTMLTagRule (February 2021):

  HTMLTagRule = Rule(
      r"<\/?\w+((\s+\w+(\s*=\s*(?:\".*?\"|'.*?'|[^'\" >\s]+))?)+\s*|\s*)\/?>",
      '',
  )
  

  When processing unfinished HTML attributes, this regex can cause the segmenter to hang indefinitely [11]. A simplified fix was proposed in the same issue, but it remains unreviewed and unmerged.

Both issues stem from the same root cause: regex patterns with nested quantifiers in the transformation pipeline [10, 11]. The project has no active maintainer to review or merge fixes [9].

yasbd-lib was built in response to this situation, offering a drop-in adapter for PySBD to fix edge cases without heavy refactoring [2, 9].

Benchmark Comparisons

The architectural differences influence accuracy and performance.

Benchmark Note: The English Golden Score is measured on the project’s expanded golden corpus of 92 evaluation cases [2]. The original PySBD corpus contained 48 cases; the expanded set removes ambiguous examples and adds coverage for abbreviation chains, contiguous terminators, and other edge cases. Full methodology and test cases are available in the benchmarks directory.

Edge Case Behavior

Consider how both engines handle challenging inputs:

• Input with numbered references (Issue #79): "..[111 111 111 111 111 111 111 111 111 111]" [10]

  • PySBD: Can enter an infinite loop due to catastrophic backtracking in NUMBERED_REFERENCE_REGEX. This was reported in October 2020 and remains unresolved [10].
  • yasbd-lib: The two-pass boundary detection approach avoids complex regex substitutions, preventing this class of issue [2].

• Input with unfinished HTML (Issue #92): <iframe width="100%" ... src="url Lorem ipsum... [11]

  • PySBD: The HTML cleaning regex can cause catastrophic backtracking, hanging the segmenter indefinitely [11]. Reported in February 2021, still unresolved.
  • yasbd-lib: Uses a StreamCleaner with configurable cleaning steps, including optional HTML unwrapping that avoids nested quantifiers [2].

Extensibility: Configuration Approaches

What happens when you need to handle custom abbreviations like "Com." or "Adm."?

PySBD: Internal Rule Modification

Because PySBD’s rules operate on a shared transformation timeline, they are interdependent [1]. Adding exceptions requires modifying the internal mutation flow [1]. As documented in [Issue #108] [3]:

"Unfortunately, there is no specific documentation about modifying rules as there are so many and each rule is associated with some form of transformation... All those operations need to be performed in that sequence as they are interrelated... Best way is to use python debugger and see how your input text goes through different transformations."

Adding rules without understanding the full pipeline can break downstream regex patterns [1]. With the project unmaintained, there is no clear path for getting such fixes merged upstream [9].

yasbd-lib: Declarative Language Profiles

yasbd-lib decouples matching mechanics from language-specific data [2]. It exposes structured hooks for customization [2]:

The base Rules class defines sets for:

• TITLE_ABBRVS: Honorifics that should not split sentences
• REFERENCE_ABBRVS: Citation abbreviations (fig, pág)
• INLINE_ONLY_ABBRVS: Abbreviations that don’t end sentences (blvd)
• DATE_ABBRVS: Month and weekday abbreviations
• DOTTED_GEOPOL_ABBRVS: Geo abbreviations like U.S., E.U.
• TERMINATORS: Extra sentence-ending punctuation
• COMMON_SENT_STARTERS: Boundary hints for languages without spaces
• POST_QUOTATIVE_PARTICLES and REPORTING_WORDS: For dialogue attribution

To add a new language, you create a file like fr.py, subclass Rules as FrRules, and override only the sets your language needs. The [language template][6] provides the structure.

External Language Packs

yasbd-lib supports loading custom language modules at runtime via register_lang_packs() [2]:

from yasbd.rules import register_lang_packs
from yasbd import BoundaryDetector

register_lang_packs(["clinical_yasbd_pack"])
detector = BoundaryDetector(lang="clinical")

Changes to a language-specific profile do not affect the core engine’s boundary detection [2].

Language Profile Policy

As documented in [Issue #198] [5], yasbd-lib has frozen its built-in language set at 39 profiles for the v1.x series to maintain API stability. Additional languages must be loaded externally via register_lang_packs() using community-maintained packages like yasbd-extras or yasbd-community.

Which Library Should You Choose?

graph TD
    A[Which SBD to choose?] --> B{Using legacy spaCy v2?}
    B -- Yes --> C[Consider PySBD with caution]
    B -- No --> D{Need active maintenance?}
    D -- Yes --> E[Use yasbd-lib]
    D -- No --> F[Use yasbd-lib for accuracy gains]

Consider PySBD only if:

• Absolute legacy lock-in: You are maintaining an existing pipeline tied to spaCy v2 or older deployments that cannot be migrated. Be aware that the project is unmaintained and has known unresolved issues, including infinite loops with numbered references and catastrophic backtracking with certain HTML inputs [9, 10, 11].

Use yasbd-lib if:

• Active maintenance: The project is actively maintained with a clear contribution path [2].
• Performance at scale: Benchmark results on the Sherlock Holmes text (594k characters) show yasbd completing in approximately 1.6 seconds compared to 13.3 seconds for PySBD on the same hardware [2]. (These results are from the project’s benchmark suite; your mileage may vary based on hardware and Python version.)
• Character span accuracy: Native span tracking may be preferable for downstream tasks like NER training or RAG indexing [2].
• Non-standard inputs: The modular design handles raw markdown, chat logs, and multilingual text [2].
• Custom language rules: The declarative profile system simplifies adding new languages or domain-specific abbreviations [2].
• Migration path: The included PySBD adapter allows incremental migration without rewriting your entire pipeline [2].
• No catastrophic backtracking: The pointer-based architecture avoids the regex issues that plague PySBD’s transformation pipeline [2, 10, 11].

References

[1] PySBD GitHub Repository
[2] yasbd-lib GitHub Repository
[3] PySBD Issue #108: Examples of modifying sentence segmentation rules
[4] yasbd-lib Issue #20: Help Add More Languages to Yasbd
[5] yasbd-lib Issue #198: Core Languages Locked at 39
[6] yasbd-lib Language Template
[7] yasbd-lib Benchmarks
[8] PySBD Paper: arXiv:2010.09657
[9] PySBD Issue #135: Archive Project
[10] PySBD Issue #79: Infinite loop?
[11] PySBD Issue #92: Catastrophic backtracking in HTMLTagRule