Does the guardrail evaluate risk or superficial signals?

A control can block a dangerous trajectory in English and change its decision when the same intent appears in another language, with different marking, or through an equivalent shortcut. Two preprints from the same group investigate this sensitivity without showing that guardrails are useless.

Status: Evidence Brief approved for publication. Editorial series: Papers we read for you. This text groups *Language Is a Security Surface for Agent Guards* and *Shortcut Sensitivity in Agent-Safety Evaluations* because they share authors and part of the data. They are complementary evidence from the same research program, not two independent reproductions.

Language is also part of the security surface

The first study assembled 1,220 steps—868 safe and 352 unsafe—derived from 589 AgentDojo trajectories and 58 unique instructions. Instructions were translated into five languages and rewritten in six English forms, plus a placebo condition. Nineteen guards produced 489,000 verdicts.

For smaller guards, changing language increased the placebo-adjusted flip from 1.5 percentage points in French to 4.4 points in Amharic. False positives rose by up to 5.0 points. Statistically significant recall loss appeared only in models below 10 billion parameters, reaching 7.4 points in the reported slice.

The result does not say that a low-resource language always defeats protection. It says the observed policy also depends on linguistic representation and guard capacity, requiring testing in each language actually used.

Equivalent shortcuts also changed false positives

The second study analyzed 932 public LinuxArena rows and focused its main analysis on 576 single-action shell trajectories—176 attacks and 400 honest cases. It also used 1,220 AgentDojo steps and 731 AgentHarm steps.

In a paired cohort of 64 bases derived from 58 interactions, the authors ran 960 calls per condition. In one guard, rewriting the marker as a shortcut reduced blocking by 8.33 percentage points; an opaque alias increased blocking by 11.46 points. The effect did not reproduce in the same way in two other guards. In the cohort, all unsafe actions remained blocked; the observed changes occurred in false positives.

This distinction prevents an alarmist conclusion: the experiment shows surface sensitivity, but it does not demonstrate a universal bypass or loss of security across every system.

The guard needs an invariance contract

Before promoting a control, declare which transformations should preserve the verdict:

1. translations in the languages actually supported;
2. equivalent paraphrases, order, and punctuation;
3. semantically equivalent aliases and paths;
4. legitimate variations in context, identity, and tool;
5. ambiguous cases that should escalate to review instead of being silently blocked.

The goal is not absolute invariance. A language or shortcut may change meaning. The contract must separate equivalent transformation, semantic change, and inconclusive cases.

Minimum evaluation harness

• maintain paired sets of safe and unsafe examples by language and domain;
• measure recall, false positives, paired flips, and stability by transformation;
• preserve the original example, variant, verdict, rationale, and guard version;
• evaluate small and large guards against the same operational risk without transferring averages;
• combine a classifier model with deterministic policy and capability limits;
• avoid auto-approval for irreversible actions when the guard is the only barrier;
• run regression tests whenever the prompt, tokenizer, model, tool, or language changes.

Limits and conflicts of interest

Translations in the first study were produced by Gemini 3.8 Flash, which also appears among the guards. No author reports fluency in Swahili or Amharic, and some partner guards are not optimized for low-resource languages. Although there are 589 trajectories, there are 58 instruction clusters, which may make bootstrap intervals narrower than the actual semantic diversity. Guards originally designed for other formats were adapted to step-level classification.

In the shortcut study, the main paired cohort has 64 bases and effects varied by guard. One model's result should not be promoted to a general property. LinuxArena, AgentDojo, and AgentHarm are benchmarks; they do not cover every integration, policy, or production effect.

The authors are affiliated with Kontext, a commercial agent-security vendor. We found no explicit funding or conflict-of-interest statement in the consulted texts. By the editorial cutoff, we found no complete independent reproduction or specific public responses from the evaluated vendors.

Architecture rule

Forge must treat language, formatting, and aliases as dimensions of the guard test matrix. A control can support a decision only when its contract declares coverage, known sensitivity, and fallback behavior. Security is not a benchmark's average score; it is the composition of classification, granted capability, executable policy, observability, and review proportional to impact. [R2-C3]

Direct sources

1. Language Is a Security Surface for Agent Guards — arXiv. Preprint, sample, method, results, and limitations.
2. Language Is a Security Surface for Agent Guards — full text.
3. Shortcut Sensitivity in Agent-Safety Evaluations — arXiv. Preprint, cohorts, method, and paired results.
4. Shortcut Sensitivity in Agent-Safety Evaluations — full text.

Editorial and responsibility note

This text combines facts attributed to sources with the author's analysis and technical proposal. Personal and professional views are not proven facts; data, denominators, limitations, affiliations, and the absence of independent reproduction are stated when available. The content is informational and does not replace technical, legal, regulatory, or security assessment. Authors, benchmarks, models, products, and institutions mentioned belong to their respective owners; there is no sponsorship or endorsement. Tech Human and Trustyu operate commercially in digital-product development and governance. Research, structure, and writing were AI-assisted; factual review, authorial approval, and publication were authorized by Fernando Parreiras on October 16, 2026, without additional independent human review.

Research cutoff: October 16, 2026. Editorial status: special Evidence Brief authorized for October 19, 2026 at 09:00 BRT.