A language model can write a flawless email and then invent a legal citation that doesn’t exist—all in the same conversation. Assessing the reliability of artificial intelligence means looking beyond whether its answers are true and examining how the design of these systems can produce results that are sometimes useful and sometimes misleading.
AI reliability in testing versus real-world conditions: an underestimated gap
Most articles about the limitations of AI focus on visible errors: a wrong date, an invented figure, a nonexistent source. The NIST analysis framework (AI 600-1, a profile dedicated to generative AI) highlights a more structural problem: performance measured in a lab does not predict behavior after deployment.
A model can score highly on a test dataset, then produce degraded results when faced with differently worded queries or in a context its training data did not cover. For this reason, NIST recommends continuously monitoring systems in operation, rather than relying solely on initial validation.
This distinction has real-world consequences. A company that validates an AI tool using a controlled sample, then deploys it at scale without a monitoring mechanism, risks a silent decline in results. To better understand why AI is unreliable, we need to look beyond simple, one-off accuracy.
Hallucinations, bias and privacy: an overview of risks identified by NIST
NIST’s AI 600-1 profile goes beyond factual errors. It identifies several categories of risk that affect the reliability of a generative AI system, even when its answer appears correct on the surface.
| Type of risk | Description | Impact on the user |
|---|---|---|
| Fabricated information (confabulation) | The model generates nonexistent facts, quotes or references | Decisions based on false information |
| Bias inherited from training data | Results reproduce imbalances present in the dataset | Discrimination in hiring, credit and healthcare |
| Privacy breach | The model may regurgitate personal data absorbed during training | Privacy violation without the user’s knowledge |
| Inability to trace provenance | No reliable way to verify where generated information came from | Inability to audit or challenge a result |
A plausible answer can still be problematic even when there is no obvious factual error if it uses personal data or cannot be traced back to its source. This issue remains largely overlooked in general-audience guides to the limitations of AI.
European AI Act: reliability treated as an obligation, not a recommendation
The European Union’s AI Act entered into force on August 1, 2024. It is being implemented in stages: AI literacy obligations and certain prohibitions have applied since February 2, 2025, while the rules for general-purpose AI models will take effect in August 2025.
This risk-based approach changes the landscape. An AI tool used to screen job applications or assign credit scores is subject to a different regime from a customer service chatbot. Transparency, documentation and human oversight obligations increase with the level of risk assigned to a system.
What the regulatory framework requires in practice
- High-risk systems (healthcare, recruitment, justice) must undergo a conformity assessment before being placed on the market, including documentation of training data and measures to address bias
- Providers of general-purpose AI models will have to meet transparency obligations regarding the data used and the system’s known limitations
- Professional users of high-risk systems must put human oversight in place and report serious malfunctions
The regulation therefore treats reliability as a matter of governance and accountability, not merely a technical question of model performance.
Business adoption: the gap between having a tool and using it effectively
In France, only 10% of companies with 10 or more employees used at least one AI technology in 2024, up from 6% the previous year. This rapid growth masks a problem: having a tool does not mean knowing how to use it reliably.
An individual trial by a curious employee does not create a process. Without a protocol for checking results, training on the model’s limitations, or clear rules about what data can and cannot be submitted to a third-party tool, adoption remains fragile. The risk is then not just a wrong answer, but misplaced confidence in unchecked results.
Checking results: what is often missing
NIST recommends systematically checking the sources and citations produced by a generative model. In practice, this step is rarely built into business processes. An employee using an AI assistant to draft a summary or prepare a decision may not always think to check every claim—or have the time to do so.
The problem is more serious in fields where errors have significant consequences: healthcare, law and finance. In these sectors, human oversight is not a luxury but a requirement for safe operation.
The reliability of artificial intelligence cannot be reduced to an error rate on a benchmark. It depends on the quality of the training data, the deployment context, the level of human oversight and the regulatory framework governing its use. By imposing obligations proportionate to risk, the European regulation formalizes what real-world experience confirms: a model that performs well under controlled conditions can become a source of faulty decisions without appropriate governance.
