You let the plumber into your house, but you take your wallet off the counter first. Letting an AI teammate investigate your production systems is letting a very fast contractor into every room at once, and telemetry is full of wallets. Bearer tokens sit in request logs, AWS keys in stack traces, connection strings in crash dumps, card numbers in payment-service payloads. An agent that reads your telemetry to find a root cause is going to encounter all of it.
So the question every buyer should ask an AI observability vendor is a precise one: when your agent calls a model, what exactly is in the prompt? Our answer is a redaction layer that sits in front of every model call our agents make. In this post, we’ll walk through where it sits, how a redacted secret can stay useful to the investigation, and what happens on the day redaction itself breaks.
One checkpoint, every model call
We run more than one LLM client stack, and we route across multiple model providers, so redacting at individual call sites was never an option. Somebody would forget one. Instead, redaction is a layer both stacks pass through on the way out: whichever provider a request is routed to, the prompt has already been scrubbed by the time it leaves.
Coverage means the whole conversation, and the whole conversation is more than the user’s message. System prompts, the agent’s own replies, its reasoning, the arguments it passes to tools, and the results those tools return all get walked, including a recursive pass through structured JSON, because a connection string buried three levels deep in a tool result is still a connection string.
Redacted, but still recognizable
A detected secret does not become ****. It becomes a placeholder like [REDACTED:aws_access_key_id:a1b2c3d4], where the tail is the first eight hex characters of the secret’s SHA-256 hash. That one design choice is what keeps redaction from ruining investigations.
The same secret produces the same placeholder everywhere it appears. So the model can still reason the way an engineer would: this token shows up in both services’ configs, this key appears in the deploy log and the error message, these two incidents involve the same credential. The fingerprint gives the model identity without giving it the value, and eight hex characters of a SHA-256 are not something you reverse.
Two smaller choices support it. Patterns replace only the secret portion of a match, so postgresql://checkout:<placeholder>@db.internal keeps the scheme, user, and host the agent actually needs. And placeholders are recognized on later passes, so the history can be re-scanned every turn without wrapping yesterday’s redactions in today’s.
You can watch the boundary work from the outside. Paste a credential-shaped string to one of our agents and ask about it:
The model is not being coy. It genuinely received a placeholder, and it tells you so.
There is one tradeoff here we made deliberately, and it occasionally bites. If an agent once passed a credential to a tool, that argument comes back as a placeholder in its own conversation history, and an agent that re-sends it verbatim will send the placeholder and the call will fail. We accepted that failure mode, because the alternative is leaking the credential to a model provider, and an agent that needs a secret should be reading it from a connector at call time, never from its own transcript.
What secrets actually look like
Detection is a table of patterns ordered most-specific first: private key blocks, certificates, JWTs, cloud access keys, vendor tokens for a dozen services, bearer and basic auth headers, database connection strings, card numbers, social security numbers, and labeled key-value secrets like api_key=... at the end. Specific rules claim their text before loose ones get a look.
The craft is almost entirely in the false-positive fight. A card-number rule that trusts its regex alone would eat order IDs all day, so a match has to clear an issuer prefix and length table and then pass a Luhn check before it redacts:
Values that only look like secrets are let through on purpose. Documentation placeholders like changeme or your_api_key are rejected, because redacting an example teaches nobody anything. More interesting is token: sa-token-prod. That is a Kubernetes secret’s name, and the name is the one identifier an agent needs to tell you which secret is misconfigured. Lowercase, hyphen-joined, digit-free words pass through as references, but only up to a size limit, because a word-list passphrase has exactly the same shape and only its length separates the two. The direction of that tradeoff was chosen consciously: over-redacting a long name costs some legibility, while letting a passphrase through costs the secret.
Pagination cursors got the same treatment. Fields like next_token front continuation cursors, never credentials, and a rule that redacted them would quietly break every paginated tool the agent uses. Meanwhile the genuinely scary inputs get bounded: a private key block only matches up to a size cap, so a truncated BEGIN header with no END in a megabyte of logs cannot send a regex engine on an expedition.
The deliberate blind spot
One absence is worth defending in public. Names, email addresses, phone numbers, and IP addresses are not on the category list. They are the connective tissue of incident triage: who deployed, who got paged, which host misbehaved. Matching them reliably also needs context a pattern table does not have, and a redactor that mangles every hostname-looking string helps nobody. The categories cover secrets and identity numbers, the things that are dangerous to ship to a third party, and each of the thirteen can be toggled per organization.
When redaction itself fails
Every pattern runs inside its own guard. A pathological input that breaks one rule degrades that one category and reports it, instead of blocking a model call in the middle of an incident.
The bigger decision sits above that: the layer fails open. If redaction throws entirely, the prompt goes out unredacted and OnCall AI keeps working, because a data-protection bug should degrade data protection, and a control that fails closed turns every redaction bug into a full outage of the agent.
The policy lookup follows the same philosophy in the other direction. Whether redaction is on, and which categories, is resolved per organization and cached, and the cached answer deliberately outlives its source: if redaction settings are not reachable, we keep serving the last known state for up to a day rather than letting a third-party outage switch a data-protection control off. Stale-on costs a little over-redaction. That is the cheap direction to be wrong in.
The boundary is observable
“Trust us” is not an answer that survives a security review, so the boundary reports on itself. Every pass emits metrics, counting redactions by category and timing the scan. Every run that redacted something emits an event carrying which agent, which channel, which model, and how many hits per category, so you can backtrack exactly where a secret was caught. And the Data Boundary panel in Guardrails shows the running tally, with per-category toggles and an audit trail of who changed what:
That number in the corner, secrets kept out of model context over the last two weeks, is the whole feature summarized for your CISO.
The Data Boundary panel ships in the product today. Start a free trial, paste a fake key into a chat with your agent, and watch it tell you what it never saw.
