Finding Where AI Decisions Actually Improve Existing Software

Most “add AI to it” conversations start from the model and look for somewhere to put it. I wanted a tool that starts from the code instead. The JEV Integration Evaluator is a Python CLI and agent skill I built at CompleteTech that scans an existing repository, finds the places where the software is already making a small judgment call (which handler gets this request, is this passage relevant, did that action actually succeed), scores those places, and then walks you through proving whether a bounded AI decision does better than what is already there.

The key word is “proving.” The tool is built to say no. Of the six frozen studies committed to the repository so far, five ended in reject or disable, and the one that passed its gates is explicitly not authorized for live use. That outcome is the point of the project, not a failure of it.

Status: experimental, version 1.3.0.dev12. Every validation in the repository uses synthetic fixtures and mocked HTTP. No live-model benefit has been measured.

Problem

What a “decision point” actually is

JEV, the model the tool is designed around, answers small, structured questions: “which of these five known options fits this situation?” It is served by TypeSafe’s API; my evaluator is an independent integration tool and the repository states that no TypeSafe affiliation or sponsorship is claimed. The design rule the README puts at the top is the one I would apply to any model:

LLMs generate. JEV classifies, selects, and evaluates. Deterministic code enforces. Instrumentation measures. Experiments decide whether JEV stays.

So the useful places are not “everywhere there is text.” They are spots where existing code makes a bounded classification with an imperfect rule: a keyword router, a retry heuristic, an “HTTP 200 means done” check, a count-based memory trim. The repository encodes thirteen of these as placement patterns, labeled A through M, in jev_integration_evaluator/patterns.py. Each one has a fixed question, a closed set of answers that always includes an abstention, and a list of checks that must stay in deterministic code.

Pattern C (tool or function routing), for example, asks “Which available handler has the declared capability needed for this request?” and allows only deterministic_handler, specialist, general_model, human, or uncertain. Registered handlers, argument types, handler availability, and permissions stay host-owned. The model never gets to decide whether it is allowed to act.

Solution

Avoiding unnecessary AI integrations

If I had to reduce the project to one practical habit, it would be this: before adding a model to a decision, write down what the decision is, what the closed set of answers is, what deterministic code must still enforce, and what a better non-AI version would look like. Then compare all three on the same frozen cases. The repository’s references/anti-patterns.md lists the usual traps, including authorization by model, truth by confidence, and automatic merge or deploy from a score.

Often the answer will be “the deterministic version is good enough.” That is a successful result, and far cheaper to learn offline than from a permanent model call on a path that did not need one.

This is the same discipline I apply in client work. Through our AI-Driven Analytics service I help teams identify where a model can measurably improve an existing decision, and through Idea to Solution I help turn that kind of candidate into a scoped, testable prototype, including recommending against AI where the evidence does not support it.

Architecture

The tool runs as a staged workflow, from scan to decision:

Implementation

Static inspection without running anything

The scanner in jev_integration_evaluator/scanner.py parses source; it does not import or execute it. Python goes through the standard ast module. JavaScript and TypeScript go through a trusted TypeScript parser helper when Node and TypeScript are installed. Every other language falls back to an explicitly labeled lexical pass that produces review leads, not recommendations. Sensitive-looking files and symlinks are skipped, and git is queried read-only with hooks disabled.

For each function it records calls, branches, loops, exception handling, string literals, and simple dataflow, then assigns call “roles” by matching call names: model, tool, retrieve, write, verify, deterministic, and so on. Patterns are inferred from combinations of roles and structure, never from the function name alone. A function named choose_route with no supporting structure becomes a “name-only review lead” and nothing more.

The part I care about most is the rule for when not to suggest a model:

# jev_integration_evaluator/scanner.py
    # Pure operations / exact checks: no model, side effects, retrieval, or unknown calls.
    pure_calls = (not calls or all(any(c is m or c["name"] == m["name"] for m in roles.get("deterministic", [])) for c in calls))
    if pure_calls and (f["operators"] or has("deterministic")) and not (has("model") or has("write") or has("retrieve")):
        reason = "Exact computation or state validation has a preferred deterministic implementation."
        pats = ["NONE"]

Arithmetic, sorting, parsing, schema validation, hashes, permission checks, and date comparisons belong in exact code. The scanner also flags the reverse mistake: if code is using JEV for open-ended generation (SQL, shell commands, summaries), it marks the spot NONE with the note “keep generation separate from assessment.”

Scoring, and the gate a score cannot override

Each candidate is scored on 16 dimensions normalized to [0,1], documented in references/scoring-model.md. Positive weights reward things like semantic uncertainty (+1.60), decision boundedness (+1.50), downstream consequence (+1.60), and testability (+1.00). Negative weights penalize model use: quality of the deterministic alternative (-2.40, the heaviest weight in the model), latency sensitivity (-1.20), implementation complexity (-0.80), and cost sensitivity (-0.70). The formula is clip(sum(weight × dimension) / sum(positive weights), 0, 1).

Unknown dimensions sit at a neutral 0.5 with [0,1] bounds, which is why the sample scores in the Evidence section carry wide ranges. The docs are explicit that this midpoint “is not a measurement” and the range is a sensitivity envelope, not a confidence interval.

More important than the arithmetic is the order of operations. Hard gates come before thresholds:

# jev_integration_evaluator/scoring.py
    rejected = c["deterministic_alternative"] in ("preferred", "mandatory") or c["pattern"] == "NONE"
    ...
    reviewed = review.get("approved") is True and review.get("source_sha256") == c["source"]["source_sha256"] and bool(review.get("reason")) and bool(review.get("reviewer"))
    if rejected: tier = 0
    elif not reviewed: tier = 1

A placement cannot climb above Tier 1 until a named reviewer approves it against the exact SHA-256 of the function’s source. If the source changes, the review goes stale. Tier 3 additionally requires at least 30 source-matched runtime observations explicitly marked as observed rather than synthetic, and the docs note that 30 is an evidence-presence floor, not proof of statistical power. Every tier still carries deployment_status: not_validated.

Typed judgments instead of free text

When a placement is eventually connected to the real API, the response is not parsed as prose. jev_integration_evaluator/client.py checks that every answer matches the question type, that the label set is exactly the requested set, that probabilities are finite and sum to one, and that the chosen label is actually the maximum-probability label:

# jev_integration_evaluator/client.py
        if set(probs)!=labels: raise InputError("Response labels differ from request")
        for p in probs.values(): finite(p,"probability",0,1)
        if abs(sum(probs.values())-1)>1e-5: raise InputError("Probabilities do not sum to one")
        if q["type"]=="choice":
            if a.get("choice") not in labels: raise InputError("Invalid selected action")
            if probs[a["choice"]]+1e-6<max(probs.values()): raise InputError("Choice is not a maximum-probability label")

The same client refuses to construct without allow_network=True, refuses HTTP redirects so the API key is never forwarded elsewhere, and rejects a response from a model other than the pinned one. Live replay also requires explicit call and cost budgets. I did not call the live API for this article.

Even a well-formed answer is advisory. Generated adapters ship default-off behind flags, fall back to the existing baseline on timeout, invalid output, or low confidence, and the host re-checks permissions and arguments at the execution boundary. In the bundled coding-agent fixture, the router can only propose a tool name; the host’s admissible() check verifies registry membership, permission, and exact argument names and types before anything runs.

Approval gates for code changes

The tool can generate an integration, but it will not apply one on its own. A patch plan contains every file change with old and new SHA-256 hashes and a unified diff, plus a fixed authorization scope: “Apply exactly these local text changes; no install, test execution, git push, merge, or deployment.” The plan is hashed, and applying it requires passing that exact digest back:

# jev_integration_evaluator/implementation.py
    body={k:v for k,v in plan.items() if k!="plan_digest"}
    if digest(body)!=plan.get("plan_digest") or approval!=plan.get("plan_digest"):
        raise InputError("Exact plan digest approval is required")
    if plan.get("repository_identity")!=digest(str(root)): raise InputError("Plan belongs to a different repository path")

Creating a worktree and running the host project’s tests are separate approvals as well, because running tests executes repository code. The editing engine itself is intentionally narrow: the original recipes support one documented Python seam shape (a top-level function that returns a call to an existing baseline function), plus a separately versioned method/async adaptation path and one JavaScript/TypeScript shape. Anything else is analysis-only and the tool says so.

Benchmark design: frozen, paired, and allowed to say no

Finding a candidate is the cheap part. The experimental methodology in references/experimental-methodology.md asks you to freeze the insertion point, the outcome measure, the minimum useful effect, model and rubric versions, task hashes, random seeds, a calibration split, an untouched holdout, labels, safety and resource limits, and a stop rule, all before the holdout runs. Results are counted as paired outcomes: both pass, both fail, rescues (baseline fails, JEV passes), and regressions. Independent pairs use an exact McNemar test; clustered pairs use a paired cluster bootstrap instead.

The comparison also includes the best deterministic alternative you can write. A brittle regex is not a reason to add a model if a better regex fixes it.

Evidence

Everything in this section was captured on 2026-10-21 at commit dd386d2.

What I ran for this article

I cloned the repository at commit dd386d2, installed it in a local Python 3.14.3 virtual environment, and ran the offline scan on my sample and on the bundled examples/coding-agent fixture (44 candidates, 7 deterministic or anti-pattern rejections, 0 strong candidates). I also ran 11 of the repository’s test files (scanner, study fixtures, review gates, statistics, runtime client): 225 passed, 1 skipped because Node/TypeScript was not installed, and 1 failed because my Windows account lacks the privilege to create symlinks, which that scanner test needs. I did not run the full suite, which includes package build, install, and platform-specific jobs.

Running it on a small sample repo

To see the scanner on something other than the bundled fixtures, I wrote a deliberately tiny synthetic “helpdesk” module for this article with four functions: a keyword ticket router, a retrieve-then-generate answer function, a ticket close that checks status_code == 200 before a database update, and an invoice total. This is my own throwaway sample, not code from the repository or from any real system:

# sample/helpdesk/helpdesk.py (synthetic sample written for this article)
def route_ticket(ticket, handlers):
    """Keyword router: the kind of brittle judgment call the scanner looks for."""
    text = ticket["body"].lower()
    if "refund" in text or "charge" in text:
        category = "billing"
    elif "password" in text or "login" in text:
        category = "account"
    else:
        category = "general"
    return handlers[category](ticket)

I ran python -m jev_integration_evaluator scan --repo sample/helpdesk --out run/helpdesk in a local virtual environment. No API key was set, and the scan makes no model calls. It wrote thirteen report, JSON, and CSV files. The ranked table below is rendered directly from its jev-opportunities.json.

A few things worth noticing in that output:

• invoice_total was classified NONE and put in Tier 0. Rounding a sum is exact computation; no model belongs there.
• close_ticket produced two candidates. Pattern E (did the close actually accomplish the goal, beyond HTTP 200) stays experimental. Pattern B (risk assessment before an irreversible write) was rejected, because the function has no model, planner, or agent role and the scanner marks the deterministic alternative as “preferred.”
• route_ticket was found as a pattern C routing candidate but ranked last; static analysis cannot know how often that keyword list misroutes.
• The results report says “No experiment has been run for this repository by this report command,” with disposition needs_more_evidence. The scan does not pretend to know whether any of these will help.

What the committed studies found

The reports/ directory summarizes six pre-registered, offline studies (repository issues #44 through #49) run against the bundled example systems. All six are synthetic: model responses are authored or recorded fixtures, labels are the case author’s frozen adjudication, and costs and latencies are predeclared assumptions rather than provider measurements. They test the evaluation machinery and the decision logic; they do not measure real-world model quality.

The routing study (issue #48) is the clearest illustration of why the deterministic arm matters. The numbers below come straight from validation/issue48-routing-report.json:

Adding the host guard alone took unsafe fake dispatches from 5 to 0. Both the deterministic cue router and the synthetic JEV arm got all 12 routes right. The predeclared gate required JEV to beat the deterministic router by at least 2 correct cases; it beat it by 0, while costing 0.024 synthetic units against zero. The recorded decision is disable_for_fixture_no_incremental_evidence. A perfect score was not enough, because deterministic code matched it.

The other studies went similarly. Graph identity (#44) and the two RAG studies (#45, #47) rejected the JEV arm. The completion study (#46) disabled it for that fixture after false completions and a safety violation. Only context retention (#49) met its synthetic gates: 12 of 20 later-task successes versus 8 for the current count-budget function and 7 for a deterministic pinned-plus-recent policy, with zero pin loss. Even there, JEV retained 137 scorer-classified unnecessary tokens versus 92 and 15, and the report records live_adoption: not_authorized_no_real_history_or_provider. A synthetic pass is the entry ticket to a real study, nothing more.

Limitations

• Experimental. Version 1.3.0.dev12. Not production-ready, and the README says no live-model benefit has been measured for this build.
• All validation is synthetic. The six frozen studies use authored fixtures, scorer-only labels written by the case author, and predeclared synthetic costs and latencies. They show the tooling and gates work; they do not show a live model helps.
• Static analysis is incomplete by design. Call edges are conservative approximations. Dynamic dispatch, reflection, framework callbacks, and generated files need traces or human review. A partial scan is not proof a repository has no opportunities.
• Role detection is name-based. Roles come from matching call names such as retrieve or dispatch, so unconventional naming can hide or invent candidates. That is why every candidate starts unreviewed.
• Languages: Python and JavaScript/TypeScript get AST analysis; other languages get lexical review leads only.
• Code changes are narrow. Edits are supported for a small set of documented Python shapes and one JS/TS shape. Connected async, canary, and active modes, and a general rewriter, are not qualified.
• Live operation is pending. Real provider operation, canary/active qualification, and measured benefit remain open items throughout the documentation.
• Not a sandbox. Evaluate untrusted projects in an isolated environment.
• Single maintainer so far. The project is written by me for CompleteTech LLC and published under CompleteTech’s Jev-Engineering GitHub organization.

Resources

• Jev-Engineering/jev-integration-evaluator – MIT License – commit dd386d2