Anvil

What it is

A focused engineering artifact, not a SaaS. The shipped pieces:

• Dataset pipeline - generated 4,000 synthetic NDAs, MSAs, and licenses via GPT-4o with strict JSON-schema enforcement; curated 3,707 accepted records; split them into 2,966 train, 371 validation, and 370 test examples.
• Training - QLoRA (NF4 + LoRA, rank 16, alpha 32, all_linear targets) on Llama 3.1 8B via Unsloth on a single RunPod A40. The paid run completed 558 optimizer steps in 8,259 seconds, about 2h 18m.
• Eval - base, fine-tuned, and GPT-4o predictors behind one protocol, run over the same 370-example held-out test split and scored on JSON validity plus per-field extraction quality.
• Cost model - measured training run cost (synthesis API, GPU time, GPT-4o eval) plus a self-hosted $/1M-tokens model and breakeven volume against continuing to call GPT-4o.
• CI and rehearsal - the full pipeline can be rehearsed on an M1 against fixtures before renting a GPU; CI covers parsing, schema, cost, charts, preflight, training config, and eval utilities.

Why I built it

To prove I can fine-tune and evaluate an open model end to end, with a defensible cost story, rather than only calling a hosted API. The useful question is whether the fine-tune closes the gap to a frontier model on a narrow structured-output task, and how long it takes for the one-time training cost to pay for itself.

Anvil answers that with numbers tied to committed artifacts: the dataset counts, training log, evaluation JSON, chart images, and cost report are all reproducible from the repo workflow.

Measured result

The base model failed the strict schema gate: 0% valid JSON on the 370-case held-out test split. The fine-tuned adapter produced 100% valid JSON, slightly above GPT-4o’s 99.73% validity on the same cases.

On per-field extraction score, the fine-tuned adapter reached a 0.899 macro score versus GPT-4o’s 0.881. The task is narrow and synthetic, so I treat that as evidence that the adapter learned the target schema and field distribution, not as a broad claim that it beats GPT-4o.

How it works

Data synthesis with an anti-contamination guard

The training set is synthetic contracts generated under a strict JSON schema, so every sample has a known ground-truth extraction. Before training, a guard hashes the normalized contract text of every sample and aborts the run if any document appears in more than one split. That removes the most common way a fine-tuning benchmark quietly inflates its own scores.

QLoRA on a single A40, rehearsed on M1

The same orchestrator that runs the paid RunPod job runs in a dry-run mode on a base-M1 MacBook against in-repo fixtures, end to end (data-smoke -> train-smoke -> eval-smoke -> cost) with zero API spend. That rehearsal must pass before a paid GPU is rented.

The paid run used a RunPod A40 at $0.44/hr, trained for 3 epochs, and ended at train loss 0.346 and eval loss 0.328.

Three-way eval gated on JSON validity

Base, fine-tuned, and GPT-4o predictors answer the same prompts. Validity is the gate: an output that does not parse against the schema earns no field credit, because an F1 number on unparsable JSON is meaningless. This is where the base model collapses and the fine-tune becomes useful.

Cost and breakeven

The full run cost $58.46: $54.00 for synthesis, $1.76 for A40 GPU time, and $2.70 for GPT-4o eval. The self-hosted inference line is intentionally labeled as a configured A40 serving assumption, not a measured serving benchmark from this project.

At those assumptions, the one-time run cost breaks even against GPT-4o at 0.80M tokens per month over a 12-month horizon.

Status

Done. The core Anvil milestone is complete: paid data generation, training, evaluation, cost analysis, charts, and documentation are landed. I am intentionally not treating Hugging Face publishing or a demo Space as part of the scope.