You Cannot Defend What You Cannot Inspect

The first AI governance committee meeting I sat in, somebody asked the product team a question nobody had a real answer to. It wasn't a gotcha. It was the most basic question a security leader can ask: when this model makes a bad decision, how will we know which part of the model made it?

The answer was silence, followed by a slide that said "human in the loop." The loop was a reviewer approving or rejecting outputs with no visibility into how they were produced. The model was a third-party API. The vendor questionnaire had a row that said "model is proprietary." We had a risk tier, a registry, a vendor red team report, a SOC 2 Type II, and no idea what was inside the box.

That is the state of AI governance in most enterprises today. We have built extensive paperwork around the outside of a black box and declared the box governed. The paperwork is real work. The box is still a box.

I am releasing CIRCUIT as an open-source framework to close that gap, and I am looking for contributors from across the security community to build it out. This post explains the gap, what CIRCUIT does about it, and why now.

What our existing controls actually do

If you have stood up an AI program in the last two years, you probably have a risk tiering scheme, a model or agent registry, output monitoring with guardrails and DLP, a vendor questionnaire, a red team program, and a mapping to NIST AI RMF. Good. Do not throw any of that away.

But look at what those controls actually do. Risk tiers classify intended use. Registries track where a model is deployed. Output monitoring fires on patterns in what the model says. Vendor attestations are assertions about the vendor's process, not the vendor's model. Red teaming probes behavior from the outside. Every one of these controls treats the model as an opaque function from input to output.

When an AI-driven SOC triage tool buries a real intrusion as a false positive, those controls tell you the model was "high risk, deployed in prod, monitored, vendor attested." None of them tell you which feature the model weighted wrong, so your incident responders can't tell whether it was a one-off miss or a blind spot an attacker can reproduce on demand. When a coding assistant slips a subtle backdoor into a pull request, none of them tell you whether it came from a training artifact, a poisoned dependency, or a prompt injection pathway in the repo — the difference between a code review finding and a supply-chain compromise. When an autonomous response agent quarantines the wrong host or pushes a bad firewall rule mid-incident, none of them tell you why it made that call, leaving the blue team to reverse-engineer their own tooling while the clock is running.

Seven things a CISO is now accountable for — and can't answer

1. Explainability under incident pressure. When the post-incident review asks why a detection model missed an intrusion or flagged the wrong activity, regulators, auditors, and your own board want the reasoning — not a log line proving the output was recorded.
2. AI-augmented adversaries. Attackers are using generative models for phishing, voice clones, and synthetic identities at scale. Your defensive models are the counter — but defending with them requires understanding what they key on, because the adversary will probe for that and evade it.
3. Data and model poisoning. Training-time compromise of weights and fine-tune corpora is a supply-chain attack on your detection stack. It is detectable only by inspecting internal representations, not by watching outputs.
4. Failures in high-consequence response workflows. When an AI tool drives containment, enrichment, or alert triage and confabulates, remediation requires knowing which part of the reasoning went off the rails — not just that the SOC acted on bad information.
5. Inherited vendor AI in your defensive tooling. It is already in your SIEM, your EDR, your code host, and your collaboration stack — Atlassian, Slack, GitHub, Salesforce, Microsoft 365 — whether you adopted it deliberately or not. You own the blast radius. The vendor owns the weights.
6. Prompt injection against security agents. MITRE ATLAS AML.T0051. An AI agent reading attacker-controlled content — a malicious log entry, a poisoned ticket, a crafted email in an inbox it monitors — can be turned against you. Defenses that don't understand the model's internal response to adversarial context are guessing.
7. Third-party AI dependencies. The AI supply chain is now the supply chain. Shadow AI is a subset of shadow IT, and almost every SaaS renewal in 2026 is quietly an AI renewal — expanding the attack surface your blue team has to defend without expanding their visibility into it.

Why now: the research caught up, and so did the clock

For years the excuse was that frontier models are too large and too complex to inspect. That excuse stopped being valid about eighteen months ago.

• Anthropic (2025) traced multi-step reasoning, planning, and refusal mechanisms on a production frontier model down to feature-level causal graphs.
• Anthropic's Sleeper Agents work reported detection rates above 99% on deceptive behaviors using internal-state probes. A backdoor that survives safety training can be caught by reading the model's own activations. That is a security control.
• OpenAI (November 2025) showed weight-sparse transformers produce circuits roughly sixteen times smaller than dense baselines, with neurons mapping to nameable concepts — proof that interpretability is a training-time design choice, not a post-hoc art.
• DeepMind's Gemma Scope shipped open sparse autoencoders across every layer of its models — a microscope, free to researchers.
• A commercial ecosystem now sells circuit-level tooling with funded go-to-market.

Meanwhile the regulatory clock is running. The EU AI Act's high-risk obligations enter enforcement in August 2026, requiring transparency, human oversight, and robustness evidence — not policy, evidence. NIST AI RMF names interpretability as a trustworthiness characteristic. SR 11-7 requires conceptual soundness in banking models. ISO/IEC 42001 is showing up in procurement RFPs. Everybody has a framework. Nobody has a meter. That is the gap.

What CIRCUIT is

CIRCUIT — Circuit-Informed Risk & Control, Understanding, Inventory & Transparency — is deliberately small. It is three things, and only three things:

• A Score you can brief to a board. The Interpretability Maturity Score (IMS) runs 0 to 5 as an evidence ratchet — from fully opaque (0) through post-hoc explainability and feature-level inspection to continuous, CI/CD-integrated circuit analysis (5). You don't declare a level; you produce the artifacts that prove it.
• A Registry your auditor can read. An eight-section YAML schema, one entry per model or system, machine-readable and diffable in Git, that extends your existing agent inventory rather than replacing it.
• A Control your pipeline can enforce. The Circuit Risk Score (CRS) combines risk tier, interpretability maturity, and decision consequence into a single number that maps to four action bands — from standard approval to not-deployable — backed by ten binding hard rules.

It accounts honestly for what you can actually inspect. Open-weight models you host can reach the full maturity ceiling. API and foundation models are capped lower. Embedded vendor AI is capped lower still — you own the blast radius, they own the weights. The ceilings aren't a judgment on vendors; they're an honest accounting of access. CIRCUIT also crosswalks to NIST AI RMF, the EU AI Act, ISO 42001, SR 11-7, SOC 2, and MITRE ATLAS, and ships with a 29-question "Show Me Your Circuits" vendor questionnaire so a security team can make the transparency gap visible on a single page.

Why open source, and why I need contributors

Vendors respond to pressure, not to individual requests. One security team, one researcher, or one incident responder asking a vendor to prove its model is inspectable is easy to ignore. Hundreds asking the same question, in the same format, is impossible to ignore. That coordinated demand is the entire strategy — and it only works if the standard is open.

CIRCUIT is released under Apache 2.0 so that no single organization owns it and any team can adopt, extend, and govern it collectively. It builds on the open work of NIST, MITRE, OWASP, and CSA — and the goal is to pay that forward. Jumpmind is the initial adopter, not the owner.

This is where the security community comes in. CIRCUIT needs practitioners to pressure-test the maturity rubric against real deployments, contribute regulatory crosswalks for jurisdictions beyond the EU and US, refine the vendor questionnaire, and report where the framework breaks. The repository is open now — issues, discussions, and an adopters list included. If you run AI controls you cannot currently inspect, you are exactly who this is for.