Building a Research-Driven, Evidence-Based AI Application

Typewright is a compiler and lightweight runtime for typed AI decisions. You declare the inputs an application has, the decisions it needs (pick one label, answer true or false, rate on an ordered scale), and a set of labeled examples. Typewright searches for better question wording within that fixed contract, fits review thresholds on separate data, and freezes the result into a checksummed JSON program that an application loads and calls. The application never needs the optimizer stack at runtime.

I built it because most “we improved the prompt” claims I see are not backed by anything a second person could check. I wanted the opposite: a pipeline where every number in a report traces back to a declared protocol, a specific data split, and a hashed artifact, and where a negative result is a normal outcome instead of something to hide. The project is written by CompleteTech LLC and lives in its Jev-Engineering organization on GitHub under the MIT license.

Status: alpha 0.1. The offline test suite and no-key demo run and pass. A few small live measurements are recorded in the repository, but no full live study has been completed, and no accuracy gain from optimization is claimed.

Solution

Typewright targets TypeSafe’s Jev model, which answers typed questions natively instead of returning free text. Three output types matter:

• choice: one label from a fixed set, with a probability for every label.
• noul: a binary probability, P(true), not a text answer.
• score: an ordered, zero-indexed scale, such as 0 to 2.

A declaration is plain YAML. This is the shipped support-triage example, trimmed:

# examples/support_triage/usecase.yaml
name: support_triage
description: Route support tickets and expose uncertainty; never perform account actions.
model: jev-1.13.0
decisions:
  department:
    type: choice
    goal: Which support department should handle the main request in `message`?
    criteria:
      billing: Charges, invoices, payments, refunds, billing errors, or subscription cancellation.
      technical: Broken features, login errors, crashes, integrations, or service outages.
  urgent:
    type: noul
    goal: Does `message` report an ongoing blocking outage or an explicit immediate deadline?
  frustration:
    type: score
    goal: How frustrated is the customer based only on the wording of `message`?

Criteria are treated as definitions, not hand-tuned prompts. During compilation, an optional DSPy “teacher” model and a standalone GEPA adapter may revise question wording and rubrics. The labels, output types, scale lengths, declared input fields, and pinned model version are not search parameters. The compiler rebuilds and validates every candidate against the original declaration, and proposals are JSON data, never generated code.

Architecture

How evidence is captured and traced

Every frozen program carries a provenance block: the hash of the source declaration, a hash and row count for each split, the architect and optimizer used, the seed, installed package versions, the teacher model if any, and a status of synthetic or measured. It also carries deployment_approved: false, always. “Measured” means a non-synthetic backend produced the evaluation. It does not mean anyone approved the program.

The study runner, typewright-study, goes further for multi-stage graphs. Each step writes digests that the next step checks:

Two details show how much of the evidence design is about not hiding uncertainty. If a request was started but no result was recorded, the runner marks it uncertain and refuses to retry automatically, because a silent retry could hide a charged call. A human can record a reconciliation, but the docs say the command “proves nothing”; it is stored as an unverified claim and listed in the report. For live studies, a reviewed manifest binds every run parameter to one digest, and select regenerates the manifest and refuses to run if anything has drifted.

Implementation

How research shaped the design

Three kinds of research went into the design, and the repository keeps each one written down.

Primary sources for every integration. docs/SOURCES.md lists the exact upstream documents used to build each adapter: TypeSafe’s pages on primitives, score ranges, the response schema, and the difference between probability and confidence; the DSPy 3.4.0 engine and client source files; and GEPA’s adapter protocol. It records an inspection date and says plainly that “a source link is not a live execution result.” One concrete design decision came straight out of this reading: Jev’s Noul answer exposes P(true), not a confidence field, so Typewright never treats a probability as vendor confidence. The two are stored in separate fields, and the review gate records which one it used.

Published evaluation methodology. docs/RESEARCH_BENCHMARKS.md defines a five-benchmark study (BoolQ, SST-5, CLINC150, GoEmotions, ANLI) on top of an earlier BANKING77 protocol. Each dataset is cited to its paper and pinned by commit or file hash. The statistical plan cites Dror et al. (2018) on significance testing in NLP and Holm (1979) on multiple comparisons, and it fixes the primary metrics, seeds (7, 17, 29), the family of ten primary tests, 5,000 paired cluster-bootstrap replicates, and 10,000 label-swap randomizations before any model is called.

The project’s own experiment log. docs/EXPERIMENTS.md records what was actually run, including results that contradicted earlier assumptions. That log changed the design more than anything else, so I cover it in its own section below.

A design point that shows up in every protocol is the control arm most prompt-optimization comparisons leave out. Besides the authored baseline (A), a one-shot rewrite (B), and GEPA search (C), there is arm D: independent rewrites selected on validation with the same ceiling. Comparing C against D asks whether reflective search beats plain best-of-N, which is a different and harder question than whether it beats the baseline.

Four splits and no leakage

Compilation requires four disjoint labeled splits, each with one job:

The loader enforces the separation rather than trusting the user to get it right:

# src/typewright/data.py
def assert_disjoint(splits: dict[str, list[Example]], source: UseCase) -> None:
    """Reject reused IDs, model-visible states, and cross-split group leakage."""
    ...
            digest = fingerprint(project_state(source.state, row.state))
            if digest in seen_states:
                raise DataError(f"Duplicate model-visible state in {split} and {seen_states[digest]}.")
            seen_states[digest] = split
            if row.group:
                previous = seen_groups.get(row.group)
                if previous is not None and previous != split:
                    raise DataError(f"Group leakage detected between {previous} and {split}.")

The duplicate check runs on the projected state, meaning only the fields the model will actually see. It cannot catch semantic near-duplicates, temporal leaks, or mislabeled records, so meaningful group IDs are the user’s job.

The compiler then runs the test split exactly once, after everything is frozen. The comment in the code states the rule:

# src/typewright/compiler.py
        # The test split is first executed here, AFTER every prompt and policy is frozen.
        # Neither test predictions nor labels are provided to the teacher or optimizer.
        baseline_test, _ = evaluate(baseline, test, self.backend)
        compiled_test, _ = evaluate(best, test, self.backend)

Typed outputs

The runtime returns a typed decision per output, not prose. This is from the no-key demo I ran, so the values are synthetic:

# runs/first-demo/sample_prediction.json (generated by typewright demo, synthetic)
"urgent": {
  "type": "noul",
  "value": false,
  "probabilities": {"false": 0.5, "true": 0.5},
  "p_true": 0.5,
  "vendor_confidence": null,
  "gate_score": 0.5,
  "gate_basis": "selected_binary_outcome_probability",
  "review_required": true
}

Each decision names its gate basis. For Noul, the gate is the probability of the selected outcome, so a false decision at P(true) = 0.6 with a 0.8 threshold has gate 0.4, not 0.6. For weighted Score composites, the gate is the minimum component gate, and the architecture document calls that “explicitly a heuristic.” Review gates are fitted on calibration data to maximize accepted coverage subject to a minimum accepted count and an empirical error limit; if nothing qualifies, the policy sets force_review=true. Typewright returns decisions. It does not move files, merge pull requests, or take any external action; the application owns that.

This is the same idea I wrote about in Building Auditable Agents With Typed AI Decisions: a decision with a type, a probability, and a review flag is something you can test, log and audit. Typewright is the part that gets such a decision from labeled data to a frozen, measured artifact.

Evidence

The offline runs, the demo report and the test results in this section were captured on 2026-10-21 against commit 6532cc8. The live numbers come from the repository’s experiment log at that commit.

The offline study chain

I ran the shipped offline check end to end. register returned registered_unexecuted with a total ceiling of 774 native attempts. select returned frozen_unreviewed with test_executed: false. test and report completed on 6 paired roots in 6 groups. The resulting directory has one record per arm and root, plus, for each graph run, an events.jsonl, an attempts.json ledger, and a manifest.json that pins the input hash, graph hash, implementation hash, model, retry policy, and the split, group and root ID the record came from. Each stage event stores the typed answer, the decision derived from it, the gate score and its basis, and whether review is required.

The report step replays every graph from those stored answers and frozen policies without making a new request, and refuses to report on incomplete evidence. The report itself is labeled “SYNTHETIC SOFTWARE CHECK – NOT JEV QUALITY EVIDENCE” and lists its own limitations, including that equal call ceilings do not imply matched spend. On this tiny fixture, the paired comparisons came out at a mean root-utility delta of -0.167 and 0.0, both with Holm-adjusted p = 1.0. On six synthetic rows, that is the correct non-result.

What the experiments actually showed

The experiment log is where “evidence-based” stopped being a slogan. Five recorded findings changed what the project claims and how it is built.

The mock fixture was blind to half the program. An ablation (E1) swapped one layer at a time between the template and a compiled program. Compiled instructions changed the mock objective by exactly zero; compiled criteria accounted for the whole +0.035. The lexical mock scores criteria text only. A follow-up (E3) found 6 of 12 GEPA components completely inert under the mock, which means roughly half the search budget landed on text that could not change the score.

The mock overstated the gain by about 26 times. In E4, both programs were evaluated on twelve test rows against the real jev-1.13.0 model, with operator authorization. The mock delta was +0.035. The Jev delta was +0.00135, with a 95% interval of [-0.0019, +0.0046] that includes zero, while the compiled program used 47% more input tokens. The log calls this compile a net negative on cost per quality. The template already scored 0.9746 on Jev, so there was almost no headroom to find.

The mock ranks hard examples, but in the wrong regime. E5 compared 24 paired per-example scores and found a Spearman correlation of about 0.53. The rule the log draws from it: use the fixture for plumbing and hard-example triage, never to choose between candidate programs.

A live pilot found a real blocker. A 200-row validation pilot on BANKING77 failed on its first request. Jev returns probabilities rounded to two decimals, so 77 rounded labels summed to 0.99, far outside the original fixed tolerance of 1e-4. The fix derives the tolerance from the decimal grid the provider actually returned:

# src/typewright/runtime.py
MAX_SUM_DRIFT = 0.05

def sum_tolerance(values) -> float:
    """Worst-case rounding drift for this many values on this grid, capped."""
    step = quantization_step(values)
    return min(MAX_SUM_DRIFT, max(1e-4, len(values) * step / 2))

Full-precision responses keep the strict 1e-4 check, and the 0.05 cap still rejects malformed distributions. The project’s rules forbid weakening checks to make a benchmark pass, so the fix went in only as a derived worst-case rounding bound, with regression tests for the failing 77-label shape and for a malformed distribution that must still fail.

BANKING77 has real headroom. After the fix, the baseline arm scored 0.81 accuracy (95% CI [0.756, 0.864]) on those 200 validation rows, an objective of 0.847 against support-triage’s 0.975. That makes it a task where optimization could be measured at all. A single-seed full run is estimated at 32,664 requests and about 100 million tokens, and has not been authorized.

These live numbers are small samples from the repository’s experiment log. I did not rerun them.

Tests and offline verification

I cloned the repository at commit 6532cc8, installed it into a fresh virtual environment with only the base and dev extras (no DSPy, GEPA, or TypeSafe SDK, and no keys), and ran the suite on Windows 11 with Python 3.14.3:

• pytest: 602 passed, 82 skipped, 1 failed. The skips are optional-package tests that need numpy, DSPy or the SDK. The one failure, in test_hierarchy_evidence.py, expects an IsADirectoryError where Windows raised a PermissionError; the code still refused to create the lock and raised its configuration error, so this looks like a platform assumption in the test rather than a behavior bug, but I have not confirmed it with the maintainers.
• ruff check src tests examples: all checks passed.
• typewright demo, flat and hierarchy starters: both completed with network_calls: 0.
• The synthetic typewright-study register, select, test and report chain: completed as described above.

The repository’s own build report records fuller runs, including 685 passed on DSPy 3.4.0 with the optional packages installed, and CI configuration covering Python 3.11 to 3.13 plus an optional-contract job for DSPy 3.3.1 and 3.4.0. The workflow uses no API secrets.

Limitations

• Alpha software. Version 0.1.0. Not production-certified, not a hosted service.
• No completed live study. The BANKING77 and five-benchmark protocols, and the flat-versus-hierarchy study, are implemented and exercised offline. Their live phases have not run. The live numbers in this article come from 48 support-triage requests and a 200-row pilot.
• No measured gain. The only completed live A/B found a negligible difference at higher token cost. Structural search and hierarchies are implemented but not shown to improve accuracy.
• Synthetic fixtures are limited. The bundled mock reads criteria only and its Score question is degenerate. Demo and study metrics are pipeline checks.
• Integrity, not authenticity. SHA-256 checksums detect accidental change, but they are not signatures; anyone can recompute them. Preregistration is local and checksummed, not an independent public registry.
• Calibration is empirical. Review gates are fitted thresholds, not posterior calibration or a guaranteed risk bound. No probability recalibrator is implemented.
• Privacy is your job. Input projection is an allowlist of top-level fields, not PII detection. Live compilation can send training examples to a second provider and requires a separate consent flag.
• Dependencies. Pins are integration targets, not a resolved lock. Native compile-time evaluation requires exactly DSPy 3.4.0 with TypeSafe SDK 0.7.0, and DSPy 3.5 is excluded because a hook the teacher meter relies on is deprecated.
• Costs are not estimated. Budgets cap request counts, not dollars, and dollar_cost is reported as null rather than guessed.

If you are turning an AI idea into something you can measure, or building a prototype that needs an honest evaluation loop from day one, that is the kind of work I do through Idea to Solution and Prototyping and MVP Development.

Resources

• Jev-Engineering/TypeWright – MIT License – commit 6532cc8

Typewright was written by Timothy Wayne Gregg of CompleteTech LLC and is published under CompleteTech’s Jev-Engineering organization on GitHub. TypeSafe is the third-party provider of the Jev model it targets.