AWS Inspector scans our ECR repositories continuously, and the list of CRITICAL and HIGH findings it maintains never stays empty for long. Our platform builds around 40 container images: 21 Go services, 9 Bun services for the AI team, and a long tail of custom bases such as a Playwright image for synthetic monitoring, a Python MCP proxy, and a Milvus-backed memory service. Each one carries a base image, a language runtime, and its own dependency tree. Across the fleet that is about 4,000 third-party packages pinned in lockfiles and vendor directories, plus a few hundred OS packages per image, and each of them ships a fix on its own schedule. There are weeks when the list is short and weeks when a Go point release or a glibc advisory touches most of the fleet at once.
We have always fixed them, and that is the point of this post. Keeping that list at zero was a standing effort shared by three groups: security engineers who triaged the findings and chased the ones that stayed open, DevOps engineers who owned the Dockerfiles and the build pipeline, and the developers of each service who had to confirm that a bumped dependency or a new base image did not break anything. The work was real and it was done well, but it was expensive. A bump that should take ten minutes often took half a day, because the person doing it had to rediscover how that particular image was built, wait for a build, read a scanner report, and go around again. Multiply that by 40 images and a weekly cadence of new advisories, and container patching was consuming a meaningful share of three teams’ time.
In Introducing Loops we described this category of work as peacetime operating: it makes the next incident less likely, and it competes with feature work for the same people every week. This post is about the loops we now run to carry that work, the CI gate that keeps them honest, and the things we had to teach them along the way. The numbers and pull requests are from our own production repository.
Three loops, split by how the fix is made
The first version was one loop with the goal “no fixable CRITICAL or HIGH findings on master images.” It discovered images well enough, but the fixes it proposed were all over the place, because a Go stdlib CVE, a vulnerable npm package in a Bun image, and a stale apt package in a Playwright base are three different jobs. So we split it by language and build type, since each one has its own challenges: Go, Bun, and custom bases.
Each loop runs the same discovery step. It queries Inspector for findings on master images pushed in the last seven days, reads the Dockerfile of every affected image, and classifies it as Go, Bun, or custom base. It then keeps only its own class and hands the rest off by writing them into its notes for the sibling loops.
| Loop | Owns | Typical fix |
|---|---|---|
| Go services | 21 images built from cmd/<service> | Go toolchain pin, module bumps through go mod tidy && go mod vendor, Alpine apk upgrade |
| Bun runtime | 9 AI services sharing a bun base | Bun version, image-local lockfile, dependency bumps |
| Custom bases | Everything else | New base tag, multi-stage rebuild, tool rebuilt from source |
Inspector itself scans continuously; we chose to run the loops once a week, on Monday mornings at 09:00, 09:30 and 10:00 UTC on the standard tier, because one well-reviewed PR per image class per week is what the reviewers on our side can absorb. Each run opens one pull request on a security/ branch, with the images it touched named in the branch and a Scan-Images: trailer in the commit so the gate knows what to rebuild. If a previous run’s PR is still open, the loop pushes to that branch instead of opening a second one.
The gate: rebuild, rescan, and only then trust the diff
A loop that edits Dockerfiles needs something to check its work that is not the loop. For us that is a CircleCI job called security_image_scan. It runs only on security/ branches, builds every affected image with the same scripts the release pipeline uses, and runs Docker Scout against the local build. If any fixable CRITICAL or HIGH finding remains, the job is red and the loop reads the per-image gate log on its next run.
Two details turned out to matter more than the scanner itself.
The first is the exceptions file. Some findings cannot be fixed by anyone but upstream. The grpc CVE whose fix exists only as an unreleased pseudo-version, or the aws-iam-authenticator binary we download into two images, which embeds whatever Go stdlib its maintainers built it with. A scanner that is red forever teaches everyone to ignore it, so we keep a small scan-exceptions.json in the repository. Every entry carries the CVE, the package, an expiry no more than 90 days out, and a recheck block that says what would make it obsolete: a released version, a minimum Go version, or a minimum version of an embedded dependency. A nightly job re-evaluates every entry and fails when one is stale. When the authenticator maintainers shipped a build on a newer Go, the nightly went red, and the next Go run bumped the binary and deleted 21 exceptions in one commit.
The second is the healthcheck. A CVE-free image that does not start is worse than the one it replaced, and the loops are willing to make large changes: new base images, a Node major, a server bundled by a different toolchain. So after the scan passes, the job starts the container. An image opts in with a healthcheck.env next to its Dockerfile, holding the port, the path, the expected body, and whatever environment the service needs to come up. The job assumes a development AWS role and forwards those credentials into the container, so a service that reads a secret from Secrets Manager at startup can do it. Services with dependencies get a small setup hook: a throwaway Redis for the Bun services, a standalone Milvus for the memory service, a NATS JetStream with the expected stream for the worker. Sixteen images opt in today, and the loops’ prompts say in plain words that a failing healthcheck means the image is wrong, and that the healthcheck.env is never the thing to edit.
# build/Dockerfiles/ai-memory/healthcheck.env
HEALTHCHECK_PORT=3010
HEALTHCHECK_PATH=/health
HEALTHCHECK_EXPECT="status":"healthy"
HEALTHCHECK_TIMEOUT=180
HEALTHCHECK_SETUP=build/Dockerfiles/ai-memory/healthcheck-setup.sh
MODE=staging
MILVUS_HOST=milvus
MILVUS_PORT=19530What the loops have actually shipped
Over the first three weeks the loops opened or updated eleven pull requests. The merged ones cover most of our fleet. A few are worth describing, because they show the range of work an unattended run can do when the gate is strict.
The Go dependency PR. Thirty-plus module bumps across five commits on one branch, each pushed from a different Monday’s snapshot. The hard part was goxmldsig, our SAML signature library, which had a fix only in an incompatible fork. The loop migrated crewjam/saml to the fork, ran the auth tests, and left a note in the PR body that this one needs the auth owners to sign off. It also found that the fixed external-secrets release could not be consumed as a root module because its go.mod references unpublished submodules, so it imported the apis package at the release commit instead. Forty-five files, and a human merged it after a staging login check.
The MCP proxy. A Python image with a Node sidecar. The loop moved it to a two-stage build on python:3.13-slim, upgraded Node from 22 to 26 to pick up a patched OpenSSL, and rebuilt mcp-grafana from source with the current Go toolchain because the published binary carried old stdlib findings. The first build failed because Node 26 wants libatomic in the build stage. The loop’s next run read the gate log and added it.
The inference runner. This Bun service had 143 fixable findings on master, almost all of them in npm packages pulled in by a test-only dependency tree. The loop’s fix was to bundle the server with bun build to a Node ESM file, externalize the sixteen packages that cannot be bundled into an image-local lockfile, and ship it on node:22-slim with npm and yarn stripped. Zero findings, and the healthcheck answered on the first try.
Synthetic monitoring. The Playwright base image ships a binary called pebble with an old Go stdlib. The loop’s first fix added RUN rm -f /usr/bin/pebble, and the scan stayed red. Docker Scout attributes a package to the layer that introduced it and ignores later deletions, which we confirmed by hand with a two-line Dockerfile. The loop’s third commit copies the finished runtime into a FROM scratch final stage, which squashes the filesystem into one layer so only surviving files are visible. The scan went green, and the rule now lives in all three prompts.
What we had to teach them
The prompts have a section called DISCIPLINE that did not exist on day one. Every bullet in it is a scar.
- Change only the lines the finding needs, and verify every version exists. The first Bun run on the cheapest model tier rewrote
EXPOSEports it had no reason to touch and pinned a Python package to a version that was never published. We moved the loops to the standard tier and wrote this rule the same afternoon. - One new pull request per run. Proposals are executed from the sandbox’s final working tree after the run ends, so three fixes made in one run produce three PRs with the same diff. The loop can push to as many existing branches as it likes, because pushes carry exact commits.
- A clean
git statusbefore proposing. Build steps leave artifacts behind. A run once shipped seventeen generated husky hook files alongside a Dockerfile change. - Merge master, never rebase. When master moved the Go toolchain under an open PR, the loop had to resolve
go.modby taking the higher version of each module and re-vendoring. A conflicting PR counts as not delivered. - Restore the continuation bundle first. A run that times out saves its working tree as an artifact. The next run starts by restoring it, so a half-finished module bump is not redone from scratch.
- Never touch the gate. When a loop found itself behind master, it re-implemented part of the gate script to make its own PR pass. The fix is to merge master, and the prompt now says the gate, the affected-images list and the CI config are off limits.
The guardrails did the rest. Our GitHub policy lets the loops open pull requests on their own and requires a person to approve a push to an existing branch. In practice I approve the pushes from my phone after reading the run’s report, and a human still reviews and merges every PR. The loops also comment on one weekly Jira issue, shared by all three, so the security review has a single place to look.
What this bought us
The same three groups still own container security. What changed is where their time goes. Security engineers review a short weekly Jira issue with the findings and the PRs that address them, instead of triaging a scanner console. DevOps engineers review Dockerfile diffs that arrive with a green scan and a container that answered its health probe, instead of writing them. Service owners get a pull request they can read in a few minutes, with the test run and the reasoning in the body, instead of a request to go figure out whether a bump is safe. The findings that remain on our master images are the ones with a written reason in the exceptions file and a date on which that reason expires.
Image changes that used to be a cautious afternoon for a person, such as a Node major or a SAML library migration, now arrive as reviewed diffs. In the first three weeks the loops opened or updated 11 pull requests covering 35 of the 40 images, and pushed 34 commits onto them; a person’s share of each was reading the diff and clicking approve. Of the findings that remain on master, every one is listed in the exceptions file with a reason and an expiry. The hours we used to spend on the mechanical part of patching have gone back to the work only people can do: deciding what is safe to merge, and building the product.
The part I did not expect is how much of the work turned out to be building the loop’s environment rather than the loop. The gate, the exceptions file, the healthcheck and its setup hooks are ordinary engineering, and all of them would be useful with no AI in the picture. The loops are what make them get used every week.
If you run Edge Delta, the Loops page under AI Team is where this starts. Describe the goal in a sentence, give it a CI job that can say no, and let it carry the work.
