Evolving Code Through Reproducible Experiments

LibreEvolve is a small, local workbench for code-evolution experiments: a language model proposes changes to a seed program, an evaluator scores each candidate, and a population archive decides what survives. The project lives under its own LibreEvolve GitHub organization and is MIT licensed. What I wanted from it was not another claim that “the model found a better algorithm”, but a run directory I could open afterwards and check: which candidates were proposed, how each one scored, why each was kept or rejected, and whether the one I picked still passes when I run it again from scratch.

The public checkout is deliberately narrow. It ships one example problem (integer bin packing), one documented model lane (a Codex CLI login), and a set of commands for inspecting, reporting on and exporting a run. Everything in this article comes from the code at commit 129b1fa and from runs I did on my own machine with a local stand-in for the model.

Status: experimental engineering preview (package version 0.1.0, classified “Alpha” in its own metadata). It does not promise improvement, and it is not a security sandbox for the code it runs.

Problem

Where this sits: evolutionary code search

The general idea is not new. Google DeepMind’s AlphaEvolve paper and open-source projects such as OpenEvolve describe loops in which a language model mutates programs, an automated evaluator scores them, and an archive of diverse, high-scoring programs feeds the next round of prompts. LibreEvolve borrows vocabulary from that family. Its SEARCH/REPLACE diff parser says in its docstring that it is “inspired by AlphaEvolve (arXiv:2506.13131)”, and its population layer combines MAP-Elites grids with an island model.

I mention those projects only as context. LibreEvolve is not a reimplementation of either, I have not compared it against them, and the code itself sets "alphaevolve_section_2_4_claim_ready": False in its benchmark-readiness records. The part I care about here is narrower: making each experiment reproducible and auditable on a single machine.

Solution

A LibreEvolve experiment is a task directory plus a run directory. libreevolve alpha init copies the bundled task into a new folder, and every libreevolve run writes to a new runs/<name>/ folder. The CLI refuses to overwrite an existing task, run, report or export path; you pick a fresh name instead.

Architecture

Archives

Evaluated candidates go into a population of MAP-Elites grids spread across islands (five by default, with ring migration). Each admission is written to archive_events.jsonl with the cell it landed in, the reason it was admitted or rejected, the normalized metric vector and the descriptor values. The rest of the run directory follows the same pattern: history.jsonl for evaluated candidates, failure_history.jsonl for rejected proposals, llm_calls.jsonl for each model call, controller_budget_events.jsonl for ordered counters, and manifest.json tying it together.

Implementation

Seeds

The word “seed” means two things in this project, and both matter. The first is the seed program: the starting point that the model is asked to improve. For bin packing it is an intentionally weak next-fit heuristic that only ever looks at the most recently opened bin.

# libreevolve/problems/examples/bin_packing/initial_programs/seed.py
def pack(items: list[int], capacity: int) -> list[list[int]]:
    """Valid next-fit baseline: only consider the most recently opened bin."""
    bins = []
    load = 0
    for item in items:
        if not bins or load + item > capacity:
            bins.append([])
            load = 0
        bins[-1].append(item)
        load += item
    return bins

The second is the random seed. The config defaults to seed: 42, and the run loop builds one random.Random(config.seed) that drives the framework’s own choices: explore-versus-exploit parent selection, prompt template sampling, database, island and MAP-Elites sampling.

Configs

The generated config.yaml is short and explicit. The initialized defaults allow three generations, four evaluations (the seed plus three candidates), three model calls, a 900-second run limit and a 10-second evaluator timeout, with retries and fallback turned off. The config validator refuses to start a preview run without explicit evaluation, runtime and call limits, and it refuses token or dollar caps outright because the Codex lane cannot enforce them. The docs repeat that point: these are bounds on local work, not on what a subscription is charged.

Evaluators

The evaluator is a plain Python file kept outside the candidate’s workspace. It builds its test cases from fixed tuples and a local Random(1729), so the training corpus (55 cases) is identical on every call and does not touch global random state. A separate held-out corpus (30 cases) uses a different seed, different capacities and different item distributions. Only evaluate is wired into the search; evaluate_holdout is a reporting call you make after the selection is frozen.

# libreevolve/problems/examples/bin_packing/validate.py
def evaluate(code: str) -> dict:
    """Search entry point: training only, with no holdout metrics."""
    return _evaluate(code, training_cases())


def evaluate_holdout(code: str) -> dict:
    """Explicit post-search reporting on a frozen candidate; never search fitness."""
    return _evaluate(code, holdout_cases())

The score is the corpus sum of volume lower bounds divided by total bins used, so it sits between 0 and 1 and goes up when the candidate uses fewer bins. The validator compares exact types and item multiplicities against the original tuple, so a candidate that drops, duplicates or converts items is invalid and the whole evaluation scores zero. The metric declaration lives next to it:

# libreevolve/problems/examples/bin_packing/metrics.yaml
metrics:
  - name: quality
    direction: maximize
    primary: true
    bounds: [0.0, 1.0]
  - name: total_bins
    direction: minimize
    primary: false
    bounds: [0.0, 10000.0]

Candidates and the validator run in timed local subprocesses. The documentation is direct that this is an execution boundary, not isolation: candidate code has host access.

Reproducibility mechanisms

Several features work together here, and it helps to be clear about what each one does and does not cover.

• Provenance hashes. The manifest records SHA-256 hashes of the config file, of the LibreEvolve source tree and of the problem directory, plus the Python version and platform. Local root paths are replaced with labels such as <problem-root>.
• RNG snapshots. At the end of a run the manifest stores the full Mersenne Twister state of the framework RNG, its hash, the seed, and an explicit list of which surfaces that RNG owns.
• Fresh re-evaluation. alpha report and alpha export do not trust stored scores. They re-run the baseline and the selected candidate in new subprocesses on both training and held-out cases.
• Identity on export. The export writes solution.py and verification.json with the candidate’s SHA-256, the validator’s SHA-256, which source was used (saved history or on-disk workspace), and whether they matched.
• Separate dimensions. Run status, validity, training quality, held-out quality, archive retention and usage are reported separately, so a completed run is never mistaken for an improvement.

The RNG snapshot is the most honest part of the design, because it spells out its own limits. The model’s outputs are listed as an external surface that the framework RNG does not replay:

# libreevolve/core/loop.py
_RNG_REPLAY_EXTERNAL_SURFACES = [
    {
        "surface": "llm_provider_outputs",
        "restored_by_runtime_rng": False,
        "boundary": "remote_or_sdk_provider_nondeterminism",
    },
    {
        "surface": "validator_subprocess_randomness",
        "restored_by_runtime_rng": False,
        "boundary": "stage_rng_seeds_are_recorded_per_evaluation",
    },
]

In practice that means two runs with the same seed, config, problem and model responses should make the same framework decisions. Two runs against a real hosted model will almost certainly not get the same responses, so the search itself is not reproducible end to end. What you can reproduce is the verdict on any candidate: the evaluator is deterministic, and the export re-checks the exact bytes.

Evidence

I did not make any live model calls for this article. I ran the documented offline path and then a full loop against a local stand-in for the model. Everything in this section was run against commit 129b1fa and captured on 2026-10-21, including the report screenshots and the test results.

Seed-only check

Following the quickstart on Windows 11 with Python 3.12.10, I ran alpha init, alpha doctor --offline (it reported “Task and evaluator contract: passed” and skipped credential checks), then a run with --max-generations 0, which evaluates only the seed and makes no model request. The next-fit seed used 1,457 bins against a lower bound of 1,168 on the 55 training cases, a quality of 0.8016. The export re-checked it on the held-out split at 0.8122 (1,118 bins against a bound of 908).

Full loop with a mock model

LibreEvolve’s only built-in backend runs the codex exec command it finds on PATH. To run the loop without a provider, I wrote a 60-line Python script, outside the repository, that behaves like that command and returns three fixed programs in order: a plain first-fit, a deliberately invalid program that de-duplicates items, and first-fit decreasing. I ran it with a PATH limited to that script, the virtual environment and the Windows system folders, so the real CLI could not be reached. This is a synthetic test of the plumbing, not evidence that any model can improve the heuristic.

The run stopped with stop_reason: max_evaluations after four evaluations and three calls. The invalid candidate appeared in failure_history.jsonl with the error domain_invalid rather than vanishing. The third candidate’s parent was the seed, not the invalid program, because parent selection is drawn from the archive.

alpha report then re-ran the baseline and the selected workspace in fresh subprocesses. The selected candidate scored 0.9782 on training and 0.9806 on held-out (926 bins against a bound of 908). The repository’s own tests already use first-fit decreasing as a calibration case, described as showing that “an established heuristic improves on the seed”, not as discovery, and that is all this result shows too.

Did the same seed give the same run?

I reset the mock and ran the same task again under a new run name. Both runs produced the same four candidate IDs, the same parent links, the same archive placements (islands 0, 1 and 2, with the invalid candidate rejected) and the same final RNG state hash. The config and problem hashes in the two manifests matched as well. With fixed model responses, the framework’s decisions repeated exactly. That is the strongest reproducibility statement I can make from this test, and it depends on the responses being fixed.

Tests

The repository includes a pytest suite and a CI workflow that runs it on Ubuntu, Windows and macOS with Python 3.11 and 3.12, plus a detect-secrets scan. On my Windows machine the full suite gave 1,613 passed, 19 skipped, 1 failed. The failure was test_symlinks_and_traversal_rejected, which raised WinError 1314 because my account cannot create symbolic links without Developer Mode or admin rights. It is an environment limitation on my side, not something I traced to a defect in the code.

Limitations

• Engineering preview. Version 0.1.0, “Alpha” in the package metadata. Installation is from a source checkout only; there is no package-registry release.
• One problem, one model lane. The public checkout covers bounded Python bin packing through a Codex CLI login. Other providers, plugins and optimizing arbitrary repositories are explicitly out of scope.
• No live result here. Every number above comes from the seed-only path or from my mock. I have not shown that any model improves the seed, and the project’s own roadmap lists a reviewed, reproducible example as planned work, not something it ships.
• Not end-to-end reproducible with a real model. The framework RNG and evaluator are deterministic; hosted model outputs are not replayed.
• No sandbox. Candidate and validator code run in timed subprocesses with host access. Use trusted tasks on an isolated machine.
• Usage is not billing. Call and runtime limits bound local work. Token or dollar caps are not supported, and missing usage data is treated as unknown, not zero. The token counts in my mock run’s manifest are made-up numbers from my stand-in script.
• Platform coverage. The documented reference is Python 3.12 on Linux or WSL. Native Windows Codex execution is listed as unvalidated, and one symlink test failed on my unprivileged Windows account.
• Small corpora. The training and held-out sets are 55 and 30 public, handcrafted or seeded cases. The README is explicit that the held-out split is a reporting view, not a secret benchmark.

All 60 commits in the history I cloned are authored by Timothy Wayne Gregg (me), under two committer identities. The license is held by Timothy Gregg / CompleteTech, and the package metadata lists me as author. The project is published under its own LibreEvolve organization on GitHub. Experiments like this are close to the evaluation work I do in AI-driven analytics: deciding what a model output has to pass before anyone relies on it, and keeping the evidence so the decision can be checked later.

Resources

• LibreEvolve/libreevolve – MIT license – commit 129b1fa

LibreEvolve is published by the LibreEvolve organization on GitHub; the commit history at the time of writing lists Timothy Wayne Gregg as the only contributor.