AutoHarness Deep Dive: How "Small Model + Harness" Beats "Large Model" — an Open-Source Rust Library That Automatically Synthesizes Code Harnesses for LLM Agents

Core idea: Wrapping a code guardrail around your LLM beats swapping in a bigger model. AutoHarness turns the paper's idea into a runnable Rust library and CLI: it uses Tree Search + Thompson Sampling to automatically generate and iteratively optimize a piece of "harness code" — code that filters, verifies, proposes, or enforces policy — constraining the agent's action space so it only takes "legal actions." It reproduces the paper's core finding: "Small model + harness > Large model without harness", converging to a 100% legal action rate in an average of just 14.5 iterations across 145 TextArena games. It isn't meant to replace the LLM; it's an explainable, verifiable layer of code that squeezes the most out of the model.

1. Project Overview

1.1 What Is It?

AutoHarness is a library + CLI tool written in Rust that automatically synthesizes and optimizes code harnesses for LLM agents. It implements the approach described in the AutoHarness paper (arXiv:2603.03329) by Xinghua Lou et al.: using tree search with Thompson sampling to iteratively refine harness code.

One sentence to know it by: Automatically synthesize code harnesses for LLM agents.

1.2 Key Facts

Repository: https://github.com/gyc567/AutoHarness
Stars: 8 (early-stage project, primarily a single maintainer)
Forks: 1
Language: Rust (Tokio async, Serde serialization, Clap CLI)
Created: 2026-03-21
Last push: 2026-03-29
License: MIT
Commits: 18
Version: autoharness = "0.1.0"
Dual form: an installable CLI (autoharness synthesize/evaluate/run/benchmark/config) and a Cargo dependency you can embed in your own project

1.3 What Problem Does It Solve?

When LLM agents carry out tasks in real environments, one of the biggest pain points is too much freedom: a model's output actions may be illegal, out of bounds, inefficient, or out of line with business policy. The traditional fixes — repeatedly nagging via prompts, or swapping in a bigger model as brute force — are costly and unreliable.

AutoHarness's answer: generate a strongly-constraining harness code for the agent. It acts like a diligent supervisor that, before any model action lands, performs filtering, verification, proposing, and policy alignment. And the crucial part — this step is automatic: a human doesn't hand-write the harness; an algorithm searches it out, refines it, and optimizes it to the end.

2. Core Ideas

2.1 A Remarkable Empirical Finding

Small model + harness > Large model without harness.

This is the core claim AutoHarness aims to prove — and validates across 145 TextArena games. It directly challenges the naive intuition that "a stronger agent means a bigger model," showing that a guardrail (harness) is often worth more than raw parameter count.

2.2 Three Pillars

The whole design rests on three pillars:

Tree Search: modeling "search for a better harness" as climbing a tree of code variants — start from the root, keep spawning candidate nodes, converge toward nodes that make the LLM take legal actions; when evaluation looks bad, backtrack, fork, and switch to another branch.
Thompson Sampling: among many candidate harness variants, strike an intelligent exploration vs. exploitation balance — focus fire on what works without stubbornly missing stronger mutations, using Bayesian reasoning to aim at each branch's expected success rate under uncertainty.
Sandboxed Execution: every candidate harness runs in an isolated environment with configurable resource limits (memory / time / file descriptors / output size / network toggle) — letting the search be aggressively trial-and-error without letting malicious or runaway code harm the host.

2.3 A Mental Model Shift

The overall mindset that emerges: the LLM provides intent; the harness provides guardrails. Intent gets to be wildly creative; the harness translates ideas into legal, safe, actionable moves. Their combination outperforms relying on a single bigger model alone.

3. Architecture (Modules & Data Structures)

3.1 Source Tree

AutoHarness/
├── src/lib.rs         # exports core, engine, memory, sandbox, templates
├── benches/            # benchmarks
├── examples/           # example code
├── install/            # install.sh + platform binaries (darwin-x86_64, linux-x86_64)
├── memory/             # MemoryStore for persistent harness storage
├── tests/              # integration tests
├── Cargo.toml
├── autoharness.toml    # default config
├── README.md / README_zh-CN.md
└── TUTORIAL.md / TUTORIAL_zh-CN.md

3.2 Core Modules

core: defines the State, Action, Harness traits plus the HarnessType enum
engine: CodeSynthesisEngine, SynthesisConfig, the Evaluator trait, tree search
sandbox: SandboxExecutor, SandboxConfig, resource limits
memory: MemoryStore, MemoryConfig (persistent storage)
templates: FilterTemplate, VerifierTemplate, PolicyTemplate, EnsembleTemplate

3.3 The Three Core Traits

pub trait State: Serialize + Clone + Send + Sync {
    fn to_prompt(&self) -> String;   // turn state into a prompt for the LLM
    fn validate(&self) -> Result<()>;  // validate whether the state is legal
}

pub trait Action: Serialize + Clone + Send + Sync + PartialEq {
    fn to_string(&self) -> String;         // string representation of the action
    fn from_string(s: &str) -> Result<Self>; // parse an action from a string
}

pub trait Harness<S: State, A: Action>: Send + Sync {
    fn harness_type(&self) -> HarnessType;   // Filter / Verifier / Policy
    fn evaluate(&self, state: &S, action: &A) -> Result<bool>; // is the action legal?
    fn propose_actions(&self, state: &S) -> Result<Vec<A>>;      // propose candidate actions
}

3.4 Synthesis Engine Config (Defaults)

SynthesisConfig is the search algorithm's control panel; its defaults reveal the convergence target:

max_iterations: 50 (maximum iterations)
convergence_threshold: 0.95 (stop once 95% legality is reached)
max_depth: 10 (max tree-search depth)
mutations_per_node: 3 (max 3 mutations per node)
exploration_constant: 1.414 (Thompson sampling exploration constant)
adaptive_sampling: true (adaptively tune sampling)
target_iterations: 20 (target iteration count)
min_improvement: 0.01 (minimum acceptable improvement)
max_nodes: 1000 (max nodes)

3.5 Sandbox Config (Defaults)

SandboxConfig defines the safety boundary for trial-running candidate code:

memory_limit_mb: 256 (256 MB memory cap)
time_limit_ms: 5000 (5-second timeout per execution)
max_file_descriptors: 64 (max open file descriptors)
max_output_size: 10MB (max output)
enable_network: false (network off by default)

4. Design Philosophy

4.1 Guardrail First, Not Scale First

No arms race to "swap in a bigger model" — the guardrail is a first-class citizen. A harness is readable, verifiable, auditable code; it turns "does the model's behavior match expectations?" into a deterministic check, reducing blind trust in the LLM black box.

4.2 Grow a Tree, Not a Single Blade of Grass

No grid search, no random patching. Tree search + sampling does directed hill-climbing over a space of variants — avoiding the crudeness of hand-writing and the exponential waste of blind trial and error, compressing the complexity into a bounded, tunable search space (max_nodes=1000, max_depth=10).

4.3 Trial and Error, Inside a Cage

Synthesizing a harness means repeatedly trial-running code — and that code may be untrusted. So "optimize boldly" and "sandbox limits" go together: resource limits / enforced timeouts / syscall filtering / input validation make automated search safe enough to hand over to the machine for self-iterating.

4.4 A Tool-First Philosophy

It isn't just a paper reproduction — it's a tool you can plug into AI coding agents (OpenCode/CloudCode). The README offers a "one-sentence quick start": hand a single prompt to an AI coding agent and it kicks off the whole harness-synthesis flow. That's a developer-tooling product orientation, not pure research.

5. Step-by-Step Tutorial

5.1 Install the CLI (One Command)

curl -fsSL https://raw.githubusercontent.com/gyc567/AutoHarness/main/install/install.sh | bash

Or via the jsDelivr CDN:

curl -fsSL https://cdn.jsdelivr.net/gh/gyc567/AutoHarness@main/install/install.sh | bash

Installs to ~/.local/bin/autoharness; verify:

autoharness --version
# autoharness 0.1.0

Platform support: macOS Intel ✅, macOS Apple Silicon (runs the x86_64 binary), Linux x86_64 (build from source), Windows x86_64 (build from source).

5.2 Use It as a Cargo Library

Add to Cargo.toml:

[dependencies]
autoharness = "0.1.0"

5.3 Three-Step CLI Workflow

# 1) Synthesize: auto-synthesize and optimize a harness with tree search
autoharness synthesize --file my_harness.py --max-iterations 20 --stats

# 2) Evaluate: score how good the harness is
autoharness evaluate --file my_harness.py --detailed

# 3) Run in the sandbox
autoharness run --file my_harness.py --input "test_state"

5.4 Write a Minimal Harness in Rust

Define state and actions, implement the Harness trait, then drive synthesis with CodeSynthesisEngine:

use autoharness::{core::{State, Action, Harness, HarnessType}, engine::CodeSynthesisEngine};

// 1. Define the game state
#[derive(Serialize, Clone)]
struct GameState {
    board: Vec<char>,  // the board
    turn: usize,       // whose turn it is
}
impl State for GameState {
    fn to_prompt(&self) -> String { format!("board={:?} turn={}", self.board, self.turn) }
    fn validate(&self) -> Result<()> { Ok(()) }
}

// 2. Define the action
#[derive(Clone, PartialEq, Deserialize)]
struct Move { cell: usize }
impl Action for Move {
    fn to_string(&self) -> String { format!("move {}", self.cell) }
    fn from_string(s: &str) -> Result<Self> {
        Ok(Move { cell: s.trim_start_matches("move ").parse()? })
    }
}

// 3. Define how good a harness is
struct GameEvaluator;   // judges whether an action / board position is legal

// 4. Let the synthesis engine find a better harness
let engine = CodeSynthesisEngine::new(Default::default());
// engine.synthesize::<GameState, Move>(&game, &harness) → returns a better harness

5.5 One-Sentence Kickoff

(The README's "one-sentence quick start": hand a single prompt to an AI coding agent such as OpenCode / CloudCode and it triggers the whole flow.)

5.6 Run the Tests

cargo test
# includes test_synthesis / test_sandbox integration tests

6. Feature List

Three harness modes: Filter (filter actions) / Verifier (verify conditions) / Policy (align with policy)
Tree search + Thompson sampling: efficient exploration of the code-variant space
Sandboxed execution: run with configurable resource boundaries (memory / time / output / network)
Adaptive optimization: dynamically balances exploration vs. exploitation
High performance: converges in an average of 14.5 iterations
Five CLI commands: synthesize / evaluate / run / benchmark / config
Cargo library API: autoharness = "0.1.0"
Cross-platform installer: one-line curl | bash for macOS/Linux
Config file: autoharness.toml
Memory system: MemoryStore persists harnesses
Harness templates: FilterTemplate / VerifierTemplate / PolicyTemplate / EnsembleTemplate
Security hardening: syscall filtering / timeout enforcement / input validation

7. Key Takeaways (Observations & Conclusions)

Looking at the project and the paper together, here are the points worth thinking about:

"Guardrail beats scale" holds, at least in controllable settings. AutoHarness's measurements (145 TextArena games, 100% legal-action rate) show that for tasks with a bounded action space, a reliable harness lets a small model reach the level of a large one — at excellent cost-effectiveness.
Tree search is the "upgrade shortcut" for harness engineering. Rather than hand-writing a harness (crude, easy to miss edge cases), let tree search enumerate, Thompson sampling select, and the sandbox catch the failures — that's turning "writing code" itself into an optimizable objective.
Security and automation are not mutually exclusive. Search needs to trial-run untrusted code, so it must isolate the trial-and-error — AutoHarness binds the two as the default posture (enable_network:false, 5s timeout), an engineering taste worth learning.
It's more a "pattern" than an endpoint. Model foundations churn quickly, but the ideas of "constrained by a guardrail, verified by code, protected by a sandbox" are slow variables that will outlast any single model.
It also reminds us harnesses have a cost. A harness itself needs synthesis and ongoing maintenance; the compute behind max_nodes=1000 and adaptive sampling grows with task complexity — so this is the sweet spot for tasks with small action spaces and well-defined constraints.

References

Repository: https://github.com/gyc567/AutoHarness
Paper: arXiv:2603.03329 (Xinghua Lou et al., AutoHarness)
TextArena benchmark: google-deepmind/arena (145 game environments)
Thompson sampling: the classic exploration vs. exploitation method
Install script: https://raw.githubusercontent.com/gyc567/AutoHarness/main/install/install.sh
Default config: autoharness.toml
Cargo dependency: autoharness = "0.1.0"