Pressure-testing my own AutoML benchmark

I tried to write a benchmark for an agentic AutoML platform. The result was negative: the obvious metric was wrong, and there is no unified benchmark runner yet. These are the design notes that talked me out of shipping a number I could not defend, from a ~137k line TypeScript codebase with ~2,208 tests.

The metric that looked right and was not

The first idea was to score the preprocessing agent by how closely its output matched a human-written reference, using set overlap of the chosen transformations. The natural measure is Jaccard similarity over the two sets of steps $A$ (agent) and $H$ (human):

J(A, H) = \frac{|A \cap H|}{|A \cup H|}

It is clean, it is a single number between 0 and 1, and it is the wrong thing to optimize. Two preprocessing pipelines can disagree on every step and produce an equally good model: drop a column or impute it, one-hot or target-encode, standardize or leave it to a tree. Jaccard scores those as failures because the strings differ, while the thing I actually care about, the quality of the model that comes out the far end, is unchanged. A high Jaccard score just means the agent copied one human's habits. It says nothing about whether the pipeline is any good.

What the metric should have been

The right target is outcome, not process: held-out model quality under a fixed budget, with the human pipeline as one baseline rather than the ground truth. Something closer to

\text{score}(A) = \text{quality}\big(\text{model}(A)\big) - \text{quality}\big(\text{model}(H)\big)

measured on a holdout the agent never saw, across several datasets, at a capped time-to-model so a slow agent cannot buy quality with compute. The notes reach that conclusion, then hit a harder fact: that runner does not exist in the repo yet. The quality gate in the benchmark catalog is literally set to tbd, and no run artifacts are committed. So I do not publish a quality number, because there is not one I can stand behind.

What I can measure, and do

The benchmark story is unfinished. The system around it is not. These are the numbers I report, every one counted from the code:

test suite: ~2,208 test cases across about 246 files (1,229 backend, 908 frontend, 71 landing), counted from the it() and test() declarations.
source size: ~137k lines of TypeScript across backend and frontend, excluding tests.
MCP registry: 20+ tools exposed to the agent over the official Model Context Protocol SDK, with human approval gates between phases.
migrations: 23 sequential SQL migrations recording the schema as it grew (auth, notebooks, workflows, experiments, pgvector embeddings, deployments).

What the tests actually prove

Agent-generated Python never runs in-process. Every cell executes inside a hardened Docker container, and the flag set is in one auditable function covered by tests rather than scattered across the codebase:

--network none          // no egress by default
--read-only             // read-only root filesystem
--user sandbox          // non-root
--memory, --cpus        // resource caps
--tmpfs /tmp:nosuid     // no setuid binaries in scratch
-v datasets:ro          // datasets mounted read-only
--add-host ...          // SSRF block

backend/src/services/container/dockerBuilder.ts: every sandbox flag, in one tested place

dockerBuilder.test.ts asserts it. Network off, read-only root, non-root user, memory and CPU caps, a nosuid tmpfs, read-only dataset mounts, and an SSRF block. Every one of those flags has a passing test behind it. The benchmark numbers don't have that yet, which is why none are published.

What this discipline looks like

The expo notes record a rejected design before a misleading number shipped. Until that runner exists, I'm reporting the test counts and the sandbox guarantees instead of a benchmark score I cannot back up.

The architecture, the sandbox, and the design notes are at github.com/ShreeChaturvedi/auto-ml.