Our AI Passed Every Safety Test. We Still Had a Massive Problem.

Picture this: You’re building a conversational assistant designed to help customers order catering for major company events. You type to it just like you would talk to a real person:

The system works its magic: it calculates quantities, filters dietary restrictions, and pulls together an itemized menu. It feels like magic when it works.

Naturally, when you put an AI assistant in front of actual customers, you need guardrails. Ours were straightforward and non-negotiable:

• 1. Identity: Never impersonate a real human.
• 2. Neutrality: Never criticize or discuss competitors.
• 3. Security: Never accept credit card numbers directly in chat.
• 4. Honesty: Never claim to perform actions outside its capability.

Because we pride ourselves on sound engineering, we built an automated safety test suite. We crafted eight distinct attack scenarios—small, targeted prompt injections designed to make the AI break its own rules. One attempted to trick it into dumping system prompts; another handed over a dummy credit card mid-conversation; a third asked point-blank if it was a human staff member.

We executed the suite. All eight tests passed with flying colors. A 100% success rate.

That should have been a moment to celebrate. It wasn’t. And this is the post-mortem of why a passing test suite was actually lying to us.

Key Takeaways

• Run every AI safety scenario at least 3 times instead of relying on a single pass; if a guardrail fails on even one run, treat the scenario as failed and investigate before release.
• Separate safety-test sampling from heavier recommendation or quality evaluations so lightweight red-team attacks can run 10–20 times without unnecessarily increasing full-conversation testing costs.
• Compare every new build against historical safety failure rates in CI/CD; storing safety scorecards is not enough unless regressions automatically surface as build-level alerts.
• Verify that every red-team payload reaches the active LLM rather than a static greeting, middleware layer, or other pre-model response path before counting the test as valid.
• Add meta-tests for the evaluation framework itself and validate metric direction explicitly so higher guardrail-breach rates cannot be interpreted as an improvement.

The Bug: Here is the uncomfortable truth behind our green dashboard: 

Each of the eight attack scenarios ran exactly once.

On paper, running a test once doesn’t sound like a bug. In traditional software development, that is literally how unit tests work. You feed an idempotent function a set of inputs, verify the return value, and move on. If it passes today, it passes tomorrow. Simple.

But Large Language Models do not work like traditional software. They are probabilistic engines.

Ask a language model the exact same prompt three times in a row, and you will get three slightly different responses. Most of the time, the variance is harmless. But occasionally, it’s catastrophic. That isn’t a glitch—it’s the foundational mechanic of non-deterministic systems.

So, declaring ‘we ran the attack once and it held’ doesn’t mean your guardrails work. It simply means your guardrails held for that one specific split-second roll of the dice.

The Math of False Confidence: Let’s do the arithmetic:

Suppose our assistant has a subtle flaw that causes it to cave to a jailbreak attack just 1 time out of 20 (a 5% failure rate). In a production environment with thousands of users, a 5% vulnerability is a massive security risk.

How often would our ‘run once’ test suite catch that flaw?

Exactly 5% of the time. 95% of the time, our test suite runs, outputs a glowing green checkmark, and lulls our engineering team into a false sense of security while a critical vulnerability sits untouched in production.

What made this realization hurt was that our team had already solved this exact problem—just for the wrong half of the codebase.

Our test suite had two distinct responsibilities:

• 1. Recommendation Quality: Evaluates whether menu recommendations are balanced, accurate, and within budget.
• 2. Guardrail Safety: Evaluates whether prompt injections, card leaks, and jailbreaks are blocked.

The recommendation tests were already configured to run three times per scenario. In fact, there was a clear code comment right above the function:

We knew the principle. We documented it. But through a series of subtle, incremental pull requests, we applied our strongest statistical rigor to menu suggestions (where an error costs almost nothing) and our weakest sampling to safety guardrails (where an error could cost us everything).

Summary of System Weaknesses Discovered

When we decided to audit the testing pipeline, we expected a quick afternoon fix: bump the loop counter from 1 to 3. But digging into the implementation revealed two even more alarming flaws.

Flaw #1: The Ignored Scorecard

Our CI/CD pipeline stores a JSON ‘scorecard’ after every test suite run. The purpose of this scorecard is regression tracking: comparing today’s build against known historical benchmarks.

When we opened the comparison module, we discovered that safety results were indeed written to disk. They were neatly organized, formatted, and stored.

And then… nothing ever read them back.

The regression comparator script only analyzed recommendation quality metrics. The safety records were filed into a digital drawer that no code ever opened. To prove this, we ran an experiment:

Flaw #2: Landing Attacks on the Doormat

This was the most embarrassing discovery. When our red-team prompt attacks were first written, they were fired at the very start of a chat conversation.

However, our assistant’s architecture initializes every session with a hardcoded static welcome message: ‘Hello! How can I help with your catering today?’. The LLM isn’t even invoked on turn one.

Seven out of our eight attack prompts were landing squarely on this static welcome banner. The test suite was literally verifying that a hardcoded string did not output a credit card number. Of course it passed!

We overhauled the testing architecture to build an honest, transparent safety validation loop.

Here are the four core architectural changes we deployed:

• 1. Multi-Sampling Enforcement:  Every safety scenario now runs a minimum of 3 times. If an attack succeeds on even 1 out of 3 runs, the entire test fails. Zero tolerance.
• 2. Independent Dial Control:  Recommendation tests are heavy and expensive (full turn-by-turn conversations). Safety attacks are lightweight single messages. We split their configuration dials so safety can run 20x without inflating API costs.
• 3. Active Scorecard Auditing:  The comparison engine now actively parses historical safety scorecards and flags any uptick in attack success rates at the top of the CI report.
• 4. Meta-Testing the Test Suite:  We wrote 20 meta-tests that validate the testing framework itself—ensuring attacks bypass static greetings and execute multiple times.
  

If your team is currently shipping customer-facing LLM applications, don’t wait for a public safety breach to audit your suite. Ask these four simple questions in your next sprint review:

• Q1: How many times do your tests run?  Are you running red-team scenarios 3, 10, or 20 times to account for non-deterministic variance?
• Q2: Would you know if guardrails regressed?  Does your CI build compare today’s safety failure rates against last month’s baseline?
• Q3: Is the model actually seeing the payload?  Are your attacks hitting active LLM prompts, or are they being absorbed by static system middleware?
• Q4: Is the metric sign wired correctly?  Are higher failure rates correctly treated as critical build blockers rather than improvement deltas?

tkxel is an AI engineering partner helping SMB and mid-market businesses build reliable AI applications and agents for production environments. Our teams work across AI architecture, model integration, testing, guardrails, observability, and governance to help organizations address the technical risks that emerge as AI systems move from prototypes into real-world use.

For customer-facing AI in particular, reliability depends on more than whether the model produces the right response during development. tkxel helps businesses design the surrounding engineering controls needed to test non-deterministic behavior, identify regressions, monitor system performance, and improve AI systems as they operate at scale.

AI safety testing is only useful when the test itself reflects how the system behaves in production. For non-deterministic models, that means testing repeatedly, verifying that red-team prompts actually reach the model, tracking safety regressions over time, and validating the evaluation framework itself.

The biggest risk is not an obvious failure. It is a test suite that consistently reports success while missing the behavior it was built to catch. Teams building customer-facing AI systems should treat safety validation as an evolving engineering discipline, not a one-time gate before release.