Stop Rubber-Stamping AI: A Contestable AI Framework for Regulated Industries

Most AI deployed in regulated industries is neither explainable nor contestable. In those systems, AI may return a recommendation or a confidence score with no way for you to contest the results. This creates a barrier to deploying AI in regulated industries such as healthcare and manufacturing because the people responsible for outcomes have no meaningful way to participate in the decision.

For example, in a family medical clinic, a general practitioner (GP) may get insights from a black box multi-agent system about a patient care plan. If they do not agree with that intel, there is no way to contest or understand why the system suggested what it did. Some systems provide an audit trail, but that comes in the form of a verbose log file, readable only by developers. In many cases, the audit trail shows only the output, not the reasoning that produced it.

This post makes the case for contestable AI as an implementable architectural pattern, not a governance aspiration, and walks through what that looks like in two regulated-industry scenarios: a clinical care plan and a manufacturing sensor classification. The pattern is domain-agnostic, but making it work depends on the data layer underneath it. MongoDB's document model stores the AI's arguments, the human's overrides, and the resulting decision together, so the full reasoning and intervention history is retrievable in a single query rather than reconstructed after the fact.

Contestable AI in healthcare

Here is how this human-agent debate works in practice, using a patient care scenario:

Figure 1. Patient care contestable AI workflow: patient analyzer agent, specialist debate, GP review, care plan generation.

A patient analyzer agent scores clinical complexity and retrieves similar historical cases via vector search. A team of specialist agents representing endocrinology, geriatrics, sleep medicine, and cardiology then argues for and against each care option using structured, scored arguments. The GP reviews the full argument set, challenges anything that conflicts with their clinical judgment, and the system re-scores in real time. The final care plan carries both the AI analysis and the clinician's interventions, and the full decision lineage is retrievable in a single query from MongoDB.

Contestable AI in manufacturing

The same pattern applies in manufacturing. On a medical device assembly line, an AI vision system flags a weld as out-of-spec. A quality assurance engineer reviews the argument, overrides it with context the model lacked—for instance, a known fixture offset that shifts that measurement by 0.3mm—and the override is stored alongside the original flag, ready for the next audit.
The same framework holds up in the field, too. For example, on an IoT platform managing HVAC units, where one of the hardest problems is classifying sensors on a newly onboarded unit.

Modern industrial equipment uses proprietary internal sensor labels that are meaningless without the original documentation. In the case of HVAC systems, when a new unit is onboarded, every sensor point needs to be correctly classified before it can feed into building management or analytics systems. If the setup is done incorrectly, downstream systems end up reading the wrong data. Manual classification requires more time and ontology expertise, neither of which is not scalable.

To solve this, set up a contestable AI workflow. A site reader agent first ingests the equipment point list and runs a vector search against previously verified sensor mappings to ground the debate in documentation and precedent. Two specialist agents then analyze the sensor label and the equipment manual, debate multiple possible classifications against a standard equipment ontology (for example, Brick Schema), and surface a recommendation. A domain expert reviews every argument, overrides the ones they disagree with, and the system re-scores in real time, producing a verified sensor record that is stored in MongoDB.

Figure 2. HVAC sensor classification contestable AI workflow: site reader agent, multi-agent argumentation, human contestation, equipment record generation.

This approach scales naturally. As more sensors are classified and verified, their classification gets stored in the database, enriching the vector search library that Phase 1 in Figure 2 draws from. Each new HVAC unit onboarded makes the next one faster and more accurate because the system is reasoning from a growing base of verified precedent rather than starting from scratch each time.

The structure is identical to the clinical scenario described above. Here we have a different domain and a different number of agents (the healthcare use case used six), but the same overall workflow. A physician contesting a care recommendation and an engineer contesting a sensor classification are structurally the same act in practice. A qualified human provides context that the LLM/agents did not have access to, and that intervention is stored as part of the final decision record. This is what makes contestable AI a pattern rather than a product. The framework does not change, no matter the domain.

Why MongoDB for contestable AI

Contestable AI only works if the data underneath it holds together, which is MongoDB’s role. IoT telemetry from a factory floor and structured argument scores are different shapes of data, and the document model stores both without forcing either into a rigid schema. Time series collections handle the high-speed sensor ingestion directly, with automatic bucketing and window functions built in rather than bolted on. Vector search sits inside the same database, so unstructured data—such as patient history summaries and equipment manuals—gets embedded and retrieved for RAG without standing up a separate vector store. The argumentation structure itself—arguments, support, and attack relations, and scores—maps naturally onto nested documents. Then every human edit gets embedded as a subdocument that becomes part of agent's memory for future use. The final output, the arguments behind it, the human edits, and the confidence scores all live together in one rich document, so a single query reconstructs the full decision lineage instead of joining data across five different systems.

Regulatory bodies are moving toward requiring this kind of documented human oversight by design. Building it into the data and reasoning layer now means you meet that requirement with evidence rather than assertions. The framework does not change with the domain, but the data layer decides whether it holds together.