In January, Anthropic CEO Dario Amodei published “The Adolescence of Technology,” a sober and carefully argued essay about where AI is heading and what it will demand of us. At its centre is a striking image. Amodei describes a near-term AI system capable of acting as a “country of geniuses in a datacenter”: millions of instances, each smarter than a Nobel Prize winner across virtually every domain, operating at ten to one hundred times human speed, able to take on multi-week tasks autonomously and collaborate or divide labor as needed. His argument is not simply that this capability is coming. It is that capability alone is not enough. For that potential to produce genuine benefit rather than harm, the institutional and governance infrastructure around AI must develop alongside the technology itself.
That gap between raw capability and deployable benefit is not abstract. We are living it, right now, in financial services.
Agentic AI is arriving in financial services. Not as a concept deck or a vendor demo, but as production architecture. Banks are building systems where multiple AI agents collaborate to process loan applications, triage fraud alerts, monitor transactions, and draft regulatory filings. The vision is compelling: autonomous workflows that operate at speeds and scales no human team can match.
But something keeps happening between pilot and production. The workflows stall. Not because the technology doesn’t work (it does, often impressively), but because nobody can answer the question that matters most in a regulated environment: on what basis did the AI make that decision?
This is the deployment gap. Billions are being invested in enterprise AI. High-stakes pilots are frozen not for the lack of capability, but for lack of explanation. And no amount of prompt engineering, guardrails, or post-hoc rationalisation will close it.
We believe the solution is architectural: a new kind of agent, purpose-built to sit at the decision point within an agentic workflow and provide mechanistic explanations of every material judgment it makes. We call it an Interpretable Decision Agent.
The Agentic Reality in Financial Services
The agentic AI paradigm is a genuine step change. Traditional AI systems were single-model, single-task: a fraud classifier, a credit scorer, a document extractor. Each could be validated in relative isolation. Agentic systems are different. They compose multiple models and tools into workflows where agents research, reason, plan, and act, often making dozens of intermediate judgments before producing a final output.
In a fraud triage workflow, for example, one agent might gather transaction history, another might assess customer risk profile, a third might cross-reference external data sources, and a coordinating agent might synthesise everything into a disposition decision: escalate, investigate, or auto-close.
The challenge is real and well documented. When the system auto-closes a case that later turns out to be material fraud, the question from compliance isn’t what the system decided. It’s why. Which factors did the model actually weigh? What did it consider, and what did it overlook? Was the decision driven by the data, or by an artefact of the model’s training?
For most agentic AI systems deployed today, those questions cannot be answered faithfully. And that gap has a measurable cost.
The financial services industry is investing at scale. BIS data shows AI spending in the sector rising from $35 billion in 2023 to a projected $97 billion by 2027. Large firms are averaging $22 million in annual AI investment, with the top decile spending over $100 million, according to Bain. Where deployment succeeds, the returns are real: McKinsey projects 15 to 20 percent net cost reductions across banking as AI scales, and institutions like HSBC have reported a 20 percent reduction in false positives while processing over a billion transactions monthly. But most firms are not yet there. The IIF-EY 2025 report found that explainability is the single most commonly raised issue in regulator-institution AI engagements. The gap between AI’s potential in financial services and what institutions can actually deploy is not a capability problem. It is a governance and explainability problem, and the cost of it is growing every quarter.
Why Current Approaches Fall Short
The industry has tried several approaches to this problem, none of which are adequate for the regulatory reality of financial services.
Prompt-based explanations ask the LLM to explain its own reasoning. But as the interpretability research community has established, LLMs do not introspect their own weights and activations when generating these explanations. They generate plausible narratives about their reasoning, narratives that may bear little relationship to the actual computation that produced the output. In regulatory terms, a fabricated explanation is worse than no explanation at all, because it creates a false record.
Guardrails and output filtering can constrain what a model says, but they cannot reveal why it said it. They operate on the surface of the output, not on the internals of the decision. A guardrail that prevents a harmful output does nothing to explain the reasoning behind a permitted one.
Traditional evals and observability platforms tell you how the model performed in aggregate: accuracy, latency, drift. They do not tell you what drove any individual decision. For a regulator reviewing a specific disposition, aggregate metrics are insufficient.
Post-hoc explanation tools like LIME and SHAP were genuinely transformative for tabular models and traditional machine learning. They decompose a model’s output into feature contributions, showing that a credit decision was driven 40% by debt-to-income ratio, 25% by employment tenure, and so on. This kind of factor-level attribution is exactly what compliance teams need.
But LIME and SHAP were designed for an era of structured inputs and relatively simple model architectures. They do not extend faithfully to large language models processing unstructured text, images, and multi-step reasoning chains. The interpretability community recognised this gap years ago. What has been missing is a production-grade alternative.
What Mechanistic Interpretability Is, and Why It Matters Here
Before describing the architectural pattern, it helps to be precise about what kind of explanation we mean, because not all explanations are equal.
Most AI explanation approaches work from the outside in. They observe the inputs and outputs of a model and try to infer what the model was responding to. This is the logic behind LIME and SHAP: given what went in and what came out, what does a statistical analysis suggest was influential?
Mechanistic interpretability is different. It works from the inside out. Rather than inferring the model’s reasoning from its outputs, it attempts to identify the actual internal computations responsible for a given decision. This means examining the model’s learned representations, circuits, and feature activations directly, tracing causal pathways from input to output through the model’s internals rather than around them.
The distinction matters for regulatory purposes. An outside-in explanation is, at best, a well-calibrated approximation; it may be accurate, or it may be a plausible rationalization that happens to fit the output. An inside-out explanation is, to the extent the methods work faithfully, an account of what actually happened computationally. For a compliance officer or a model risk validator, that difference is significant.
The field is still developing. No current method provides complete, verified mechanistic transparency for large language models at production scale, and we want to be clear about that. But the progress is real, and specific techniques are now mature enough to be operationalized in well-scoped settings.
The Architectural Insight: Delegate the Decision
The critical insight is that not every step in an agentic workflow needs to be explainable. Most agent actions (information retrieval, data formatting, summarisation, tool calling) are operational. They can be validated through conventional software testing and observability.
What does need to be explainable is the material decision: the moment when the system commits to a judgment that has regulatory, financial, or customer impact. The fraud disposition. The credit recommendation. The compliance determination. The collections strategy.
The architectural pattern is straightforward: within your agentic ecosystem, delegate each material decision to a single, purpose-built Interpretable Decision Agent. This agent is implemented with technology that provides mechanistic explanations of what factors were considered and how they influenced the outcome, analogous to the feature attribution that LIME and SHAP provide for tabular models, but built on methods that work faithfully for the current generation of AI architectures.
The surrounding agents continue to do what they do well: gather information, orchestrate workflows, interact with tools and data sources. But when the workflow reaches a decision point that carries material consequences, it calls the Interpretable Decision Agent. That agent makes the decision and produces an audit trail that can be inspected, challenged, and presented to a regulator.
This is not a bolt-on. It is a deliberate architectural choice that separates the operational complexity of agentic workflows from the governance requirements of regulated decision-making.
What Makes a Decision Agent Explainable
An Explainable Decision Agent must do more than produce a decision and a narrative. It must provide mechanistic attribution: a faithful account of which internal features and representations drove the decision, derived from the model’s actual computation, not from a separate explanation model, and not from the model’s own self-report.
This is where the science matters. Two methods, used in combination, make this possible today.
Cross-Layer Transcoders (CLTs) decompose a model’s neural activations into interpretable features. Rather than treating the model as a monolithic black box, CLTs reveal the specific concepts and patterns the model activates when processing an input. They answer the question: what did the model actually respond to?
Topological Data Analysis (TDA) maps the relationships between those features: how they cluster, how they combine, and how they form the higher-level concepts that drive decisions. TDA reveals the structure of model behaviour, surfacing patterns, transitions, and failure modes that no amount of output-level testing would uncover.
Together, these methods produce decisions whose reasoning is structurally transparent. The explanation is not a post-hoc narrative generated by the model. It is a decomposition of the actual computation, the AI equivalent of showing your working.
For a fraud triage decision, this might look like: the model identified features corresponding to transaction velocity, geographic anomaly, and account age; the interaction between the velocity and geographic features activated a pattern associated with synthetic identity fraud; and the account age feature modulated the confidence level downward. This is the kind of factor-level attribution that compliance and model risk teams can engage with, and that regulators expect.
Why This Matters Now
Three forces are converging to make this urgent.
Regulatory pressure is intensifying. SR 11-7 already defines any quantitative method used for decision-making as a model requiring validation and governance. As practitioners in the model risk management community are beginning to argue, this definition extends to any LLM used in a decision-making capacity, including LLMs used to evaluate the outputs of other LLMs. The governance framework exists and is being applied.
Agentic architectures are becoming the default. The industry is moving from single-model deployments to multi-agent systems. This multiplies the number of decision points that require transparency. The longer organisations wait to address interpretability architecturally, the more deeply embedded the governance debt becomes.
The economics are hard to ignore. Manual review cannot keep pace with the volume of AI-generated decisions. The alternative to mechanistic interpretability is not “no interpretability,” it is “no production deployment.” Every month that a high-value AI use case sits in pilot purgatory because it cannot satisfy model risk requirements is a month of unrealised ROI. The 15 to 20 percent net cost reduction that McKinsey projects across banking is only reachable by institutions that can actually deploy.
From Research to Implementation
The concept of an Interpretable Decision Agent is not speculative. The underlying research is mature, the methods are validated.
We work in close collaboration with teams in banking, financial services, and insurance, the institutions where the reasoning behind a decision carries the same weight as the decision itself.
We built BluelightAI because we saw this problem coming: an industry investing aggressively in AI capabilities, while the interpretability infrastructure lagged behind. Agentic AI makes the gap wider. Explainable Decision Agents close it.
If you are deploying agentic AI in a regulated environment, or planning to, we should talk. Not about whether interpretability matters (that question is settled), but about how to architect it from the beginning.
Sachin Khanna is the CEO of BluelightAI. Gunnar Carlsson is the Founder and CTO of BluelightAI, and Professor Emeritus of Mathematics at Stanford University. Nicholas Lewins is an advisor to BluelightAI.
To learn more about Interpretable Decision Agents and the Cobalt platform, visit bluelightai.com or contact us at hello@bluelightai.com.