Can recurring LLM traces be synthesized into deterministic pipelines of typed ML and NLP operators? [D]

We are investigating whether recurring LLM workloads can be replaced, where appropriate, by automatically constructed pipelines of regexes, deterministic parsers, traditional ML and NLP models.

As an example, suppose an application repeatedly asks a frontier model to read an annual report and return all customer-supplier relationships as structured records containing a customer, supplier, and supporting evidence. A possible replacement pipeline might run named-entity recognition, entity normalization, candidate generation, entity linking, relation extraction, and schema validation.

NER → entity normalization → candidate generation → entity linking → relation extraction → schema validation

A calibrated uncertainty or out-of-distribution gate would use the pipeline for inputs inside its validated domain and escalate other cases to the original frontier model.

Current action space

Our current action space is a taxonomy of 41 atomic task types spanning:

• classification
• token and span labeling
• structured extraction
• retrieval and entity resolution
• similarity
• normalization
• reshaping
• deterministic computation

The idea is that we would first cluster repeated traces into workload families and induce an end-to-end typed contract for each family. We would then generate candidate DAGs using the 41 task types as building blocks, instantiate each node with an appropriate implementation, and optimize the composition for quality, cost, and latency.

Candidate pipelines would be tested on time-separated and group-separated holdouts before being deployed behind abstention and fallback.

Open concerns

The problem is quite likely undetermined based on just the input and output contracts alone even if inferred correctly. The intermediate graph is therefore not a recovered latent reasoning trace. It is a synthesized program hypothesized to be behaviorally equivalent over a bounded input distribution.

A fixed task taxonomy may help by constraining the search space and supplying type signatures, candidate implementations, and task-specific evaluators.

We are thinking about this problem as a form of program synthesis and formal verification for now, but wondering if this is the right approach and if there is a better way.

Looking to speak with people who have worked in this problem space and/or the program synthesis domain for insights.

TL;DR: We want to synthesize executable DAGs composed of regexes, deterministic parsers and ML/NLP models from LLM traces for appropriate tasks. Does this seem feasible and what might be some good approaches?