AI Interpretability as a Security Control

Recent research on weight-sparse transformers offers the first serious path toward AI systems with inspectable internal logic. This is not theoretical — it is a concrete methodology that produces models where we can point to the exact circuit implementing a specific decision. For security leaders navigating the "use AI everywhere" mandate while remaining accountable when things go wrong, this research deserves your attention.

The Problem Every CISO Recognizes

In the last twelve months, you have probably approved the deployment of at least one AI system you cannot explain. Not because you were careless. Because everyone who asked "when this model makes a bad decision, how will we know which part of the model made it?" received the same answer: silence, followed by a slide that said "human in the loop."

The compliance stack around that black box grew. Risk tiers were assigned. An agent registry was created. Red team reports were collected from vendors. SOC 2 Type II certificates were filed. Output monitors were tuned. And if something goes sideways today, none of that paper trail will tell you why the model did what it did.

This is the governance gap. We have built extensive paperwork around the outside of a black box and declared the box governed.

The technical excuse for accepting this situation — that neural networks are fundamentally opaque, that interpretability is a research frontier too immature for operational use — is no longer available.

What Changed in November 2025

Gao et al. at OpenAI published a paper on weight-sparse transformers. ^[1] The core result: training a transformer with explicit sparsity constraints on weight matrices produces a model with circuits that are an order of magnitude smaller than dense-baseline equivalents, while maintaining comparable task performance. The published circuits are human-readable. A reviewer with domain expertise can examine the circuit for a given behavior and evaluate whether it is implementing the intended computation.

To be concrete: a weight-sparse model trained for a classification task produces, for that task, a circuit with on the order of dozens to low hundreds of features and a few hundred feature-to-feature connections. A dense baseline model performing the same task implements the same behavior using thousands of interacting components. The sparse model is not less capable — it is restructured so that its capabilities are expressed in a form that can be audited.

This matters for security for a specific reason. The threat models that keep me up at night are not the ones where the model produces obviously wrong output. They are the ones where the model produces subtly wrong output that looks correct until it has caused material harm. Sleeper-agent behaviors. Gradual behavioral drift under distribution shift. Prompt injection attacks that produce contextually coherent but adversarially directed completions. These failures are invisible to output monitoring by design. Circuit-level inspection is the only tool that can see them before they surface as incidents.

Mechanistic Interpretability Is Not New — What Is New Is Scale and Tooling

The mechanistic interpretability research program — reverse-engineering the algorithms implemented by neural networks at the level of weights, circuits, and features — has been active since at least 2021, when Elhage et al. published the mathematical framework for transformer circuits. ^[2] The problem has been scaling these methods to production models.

Three developments in 2025 changed the scaling picture:

Anthropic's attribution graphs on Claude 3.5 Haiku (March 2025). Ameisen, Lindsey et al. published circuit-tracing methodology and open-sourced tooling capable of reverse-engineering multi-step reasoning inside a production language model. Attribution graphs were produced for complex behaviors including multi-step factual reasoning, safety refusal circuits, and tool-use gating. The tooling was made publicly available. This is the first time a commercially deployed model had its internal computational graph documented at the circuit level with the documentation made publicly available. ^[3]

OpenAI's weight-sparse transformers (November 2025). The architecture innovation described above. Not just a research result — a practical training methodology that produces interpretable models without sacrificing performance. ^[1]

DeepMind's Gemma Scope 2 (December 2025). A comprehensive suite of pre-trained sparse autoencoders covering all layers of the Gemma 3 model family. Organizations deploying Gemma 3 variants can begin feature-level monitoring using published dictionaries without building their own SAE training infrastructure. ^[4]

Alongside these, a commercial tooling ecosystem has emerged: Goodfire Ember provides API-accessible SAE feature editing and monitoring for supported model families. Neuronpedia maintains a public feature database across model families. Transluce specializes in continuous interpretability monitoring. The methods have been productized. This is no longer a research capability that requires a team of ML researchers to access.

What Circuit-Level Visibility Actually Enables

Let me be specific about what security functions gain from circuit-level interpretability, because "we can see inside the model" is too abstract to be actionable.

Post-incident attribution. When a model produces a harmful or anomalous output, circuit-level analysis can identify which specific internal components were active during that inference. This is the equivalent of a stack trace for neural network behavior. Instead of "the model made a bad decision and we don't know why," you get "feature 4,847 — which the SAE labels as 'escalate authority claim' — activated in this context when it should not have, and here is the attribution graph showing how that feature's activation propagated to the output." This changes incident response from guesswork to evidence-based triage.

Adversarial perturbation detection. Prompt injection attacks that manipulate model behavior through adversarial inputs produce characteristic activation patterns in safety-relevant circuits before they produce anomalous outputs. A production monitoring system watching the relevant features can detect the attack at the circuit level rather than waiting for it to surface as a policy violation in the output. For automated agentic workflows where the output triggers irreversible downstream actions, this time advantage is material.

Stability verification across model updates. When a vendor updates the model serving your production traffic, you want to know whether the safety-relevant circuits changed. With circuit-level inspection at IMS 4 or above, you can compare the attribution graphs before and after the update and quantify circuit stability. A stability score below 75% means the monitoring configuration built on the previous model version may no longer correspond to the current model's actual computation — your monitoring is looking for circuits that have moved, merged, or disappeared.

Vendor accountability. The question "can you show us the circuits implementing your model's safety behaviors?" is now answerable by vendors with genuine interpretability programs. A vendor who cannot answer it is either unwilling or unable — both of which are risk signals that belong in your procurement record and your registry.

The Category Problem

Not all deployed AI systems are equally amenable to circuit-level inspection. This is the honest constraint that any governance framework has to acknowledge.

For open-weights models — Llama, Gemma, Qwen, Mistral, DeepSeek — you have access to the weights, and therefore full circuit-level inspection is achievable. The Gemma Scope 2 release means that SAE feature dictionaries are available for Gemma 3 without any additional training work.

For API-served models — OpenAI, Anthropic, Gemini, Cohere, Bedrock — you do not have weight access. Your circuit-level visibility depends entirely on what evidence the vendor provides. Anthropic has published attribution graphs and open-sourced circuit-tracing tools for Claude 3.5 Haiku. If your vendor has done equivalent work, you can reach IMS 3 (feature-level inspection) using vendor-provided artifacts. If they have not, you cannot reach above IMS 2 regardless of your internal tooling investments.

For embedded vendor AI — M365 Copilot, Atlassian Intelligence, Slack AI, GitHub Copilot, Salesforce Einstein — you have no activation access. You can monitor inputs and outputs. You cannot see inside. The maximum achievable interpretability maturity for these systems is IMS 2, and it depends entirely on what interpretability evidence the vendor is willing to provide.

This creates a governance asymmetry that most enterprise AI programs are not yet pricing into their deployment decisions. A Critical-tier workflow running on embedded vendor AI operates under a structural interpretability ceiling that no amount of internal investment can raise. The decision to deploy that workflow in an automated or irreversible consequence context is a decision to accept a risk that cannot be reduced through your own controls.

What a Governance Framework Needs to Handle This

An AI governance framework that accounts for interpretability as a security control needs to do three things that most current frameworks do not do:

First, it needs a meter. A computed score that takes interpretability evidence into account and produces a deployment authorization decision. Not a framework with categories and criteria. A number that drives a gate.

Second, it needs evidence requirements, not assertion requirements. Claiming that a model is "interpretable" or "explainable" means nothing without named artifacts on file that prove it. An IMS claim requires specific deliverables: a sparse autoencoder feature dictionary for IMS 3, attribution graphs with named circuits and ablation study evidence for IMS 4, CI/CD circuit diffs for IMS 5. The governance program needs to verify what is on file, not what the vendor claims.

Third, it needs vendor transparency obligations with teeth. A vendor who cannot answer the interpretability questions in your questionnaire, and who does not respond within a documented window, needs to face a computable risk consequence — not just a note in a procurement file. If a vendor's silence translates directly into IMS 0 for CRS computation, and IMS 0 on a Critical-tier system pushes CRS into territory that blocks deployment, then the vendor's transparency posture is load-bearing in the governance program.

These three requirements are what CIRCUIT is designed to deliver. The Interpretability Maturity Score (IMS 0–5) is the meter. The evidence artifacts required per level are the verification requirement. Rule 8's 60-day vendor response window with IMS-0 consequence is the vendor transparency obligation with teeth.

The Timeline

The EU AI Act's high-risk system obligations hit enforcement in August 2026. Article 15 requires demonstrable accuracy, robustness, and cybersecurity — not vendor attestation, but evidence. NIST AI RMF MEASURE 2.9 explicitly asks for interpretability methods. SR 11-7 requires conceptual soundness validation that summary statistics alone can no longer satisfy for models in regulated banking workflows.

The regulatory clock is running. The research capability is available. The tooling ecosystem is commercial. The question is whether your governance program has a mechanism for counting interpretability evidence as a security control — or whether it is still treating AI systems as black boxes and hoping the paperwork around the outside is enough.

It is not enough. It has never been enough. The difference now is that there is something better available, and the regulators know it.

The CIRCUIT framework is the governance instrument built to close that gap. The three-part series that follows — The Governance Gap, What CIRCUIT Is, and How to Adopt It — walks through the full specification and implementation path. This post is the before picture. The series is the after.

References

1. Gao, L., et al. (2025). Weight sparse transformers have interpretable circuits. arXiv:2511.13653 [cs.LG]. November 2025.
2. Elhage, N., et al. (2021). A Mathematical Framework for Transformer Circuits. transformer-circuits.pub.
3. Ameisen, E., Lindsey, J., et al. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models. transformer-circuits.pub. March 2025.
4. Google DeepMind. (2025). Gemma Scope 2: Open Sparse Autoencoders Across All Layers of Gemma 3. December 2025.
5. Templeton, A., et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. transformer-circuits.pub.
6. Hubinger, E., et al. (2024). Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training. arXiv:2401.05566.