Ask a language model to fix a bug and it will happily hand you a diff. It looks plausible. It compiles in your head. And a distressing fraction of the time it’s wrong: a wrong import, an API that changed two versions ago, a test that was never run. The model had no way to know, because it never executed anything. It was pattern-matching over text.
Our position is simple: an AI teammate that proposes code should be held to the same bar as an engineer who proposes code. It should clone the repo, make the change, install the dependencies, run the linter and the type-checker and the tests, and only then open a pull request. To do that, the agent needs a real computer, and giving it one safely is most of the work.
The easy version, and why it fails
The first instinct is a “code interpreter”: a stripped-down Python jail that executes a snippet and returns stdout. It demos well. It falls apart the moment the task is “fix this failing service,” because real work needs a real toolchain: a specific Go version, npm install, a linter config, a headless browser to reproduce a UI bug. It also needs the customer’s actual repository at a specific commit, not a paste.
How the sandbox actually works
Every agent turn gets an ephemeral Linux container on Modal booted from a version-pinned image with some of the toolchain baked in at build time: Go, Node, Bun, Python etc. Baking it in means per-turn startup is fast enough to feel interactive, instead of reinstalling packages on every call. Inside, the agent works against a temporary workspace and gets four tools:
// the tools the model can call inside the sandbox
read_file // cat -n with offset / limit
search_code // grep + find, excludes node_modules / .git
execute_code // arbitrary bash / python: installs, linters, tests
create_artifact // upload a workspace file → public link
// NB: there is no `git` in the sandbox. Branch / commit / PR
// all happen host-side, over the GitHub API. (see below)
The load-bearing decision is the split on the right of that diagram. The agent edits the filesystem freely inside the box, but the mutation of your repository never happens in that box. Cloning isn’t git clone; it’s a tarball pulled from the GitHub REST API and untarred, with owner, repo, and ref regex-validated and the token shell-escaped before it’s ever interpolated. When the agent is done, we don’t trust its account of what changed: on clone we wrote a sha256 manifest of every file, and now we re-hash, diff host-side, and push the exact changeset through the GitHub git API, createBlob → createTree → createCommit → createRef → open PR, bounded by hard caps of 100 files, 1 MB per file, 10 MB total.
That decoupling buys three things at once. Your credentials never enter a shell the model can influence. The PR reflects exactly what the tested edits produced, computed by a trusted host-side hash, not whatever the model claims. And the blast radius is bounded by construction. The PR-creation tool goes further: it’s a long, battle-tested policy that forces the agent to search for a duplicate open PR (using verbatim distinctive tokens, not a paraphrase) and to honour the repo’s PR template before it opens anything.
Reasoning over data bigger than the context window
Deep work produces enormous tool outputs, a log query returns megabytes, not sentences. Rather than stuffing that into the model’s context, oversized tool results are spooled to JSON files under the workspace, and the reference marker instead returned to the LLM with just a preview. The agent is then told to analyze them the way an engineer would: write one Python script that globs the files and computes the answer. This is something a “suggest a diff” tool structurally cannot do: the analysis happens in the sandbox, over data far larger than any prompt.
The parts that hurt
We originally ran this on AWS Bedrock AgentCore, then migrated the entire backend to Modal while it was live, without breaking in-flight durable sessions. The two backends checkpoint completely differently: Bedrock saves the image of the workspace to S3 (capped at 2 GB, with some exclusions similar to what you have added to your Gitignore file); Modal takes a native filesystem snapshot and gives back the image reference, which we store it for future reference with a TTL, so the next turn cold-boots straight from it. An agent can resume a repo it already cloned and built without re-cloning. These are abstracted both behind single createCheckpoint/restore interface and made existing sessions default to the old backend, so nothing in flight shattered on deploy.
Then there’s the artifact-upload path, which took four separate fixes to get right. Letting the agent upload “any file that exists” is a data-exfiltration hole, so we jail uploads by resolving symlinks with realpath and refusing anything that doesn’t land under temporary workspace. Getting the parser for that guard correct then meant handling the CRLF line endings the sandbox shell emits, and, one fix later, un-fixing an over-eager .trim() that was corrupting file paths which legitimately end in whitespace. None of that is in a tutorial. You find it in production, and each fix is one specific, non-obvious failure mode.
Build-it-yourself reality check
A weekend “code interpreter” is a few hundred lines. A sandbox you’d trust to open PRs against a customer’s repo is a different animal: a pinned polyglot image, per-second cost metering with an idle-reclaim timeout, crash-safe checkpoint/restore, symlink-jailed artifact I/O, and a server-side mutation path that never trusts the model’s account of what it changed.
We rebuilt the execution backend once already and it barely touched the agent code, because the abstraction was in the right place. That’s the tell: the moat isn’t “we call an LLM,” it’s everything between the model and your
mainbranch, and most of it only reveals itself after something goes wrong at 2am.
Offer a code change without running it and it usually fails. So we run it, in a metered, throwaway Linux box, before a human ever sees the PR.
