One Rail, Three Policies — NeMo Guardrails on the Retrieval Path

Article #8 closed with a finding and a queue: bigger generators over-refuse on perfect retrieval, and the next rung is the policy gate — the one that fires between “we have an answer” and “we hand it to the user or the agent.” That’s the rung where a Second Brain has to scrub personal identifiers out of a draft before it goes back to the user; where an LLM Wiki has to enforce house style on every page it writes; where an Autoresearch agent has to refuse to run rm -rf even when its planner confidently suggested it.

One product covers all three on the NVIDIA stack: NeMo Guardrails. It is deliberately positioned as scaffolding — an input-rail, retrieval-rail, output-rail framework — not a detector. You bring the detectors; the rail runs them. That shape is exactly what the shared-substrate arc has been asking for: one rail, three policies, same retrieval chain from article #7.

This article installs Guardrails once, writes three minimal configs (config-sb, config-wiki, config-auto), wraps the article-#8 hybrid-ask pipeline, and benchmarks fifteen synthetic queries (five per arc, three violating, two clean). Every query lands the expected verdict: block recall 1.0, clean pass rate 1.0, zero crossed wires between arcs. That number is a demo, not a proof — but the architecture the number demonstrates is what unlocks the three arcs forking apart in article #10.

Why the rail is a personal-AI concern

On a cloud deployment, the rail is infrastructure someone else runs and charges for, and its configuration is a shared artifact — nobody in the org owns its edit history. On the DGX Spark, the rail is a Python package you pip install into a venv, and its policy is three .co files you read over morning coffee. The arbiter of what data leaves the box, what voice the wiki writes in, and what shell commands the agent can run is your config. That’s a different relationship to a safety surface than any cloud product can offer.

It also changes the economics of the rail. Cloud guardrail services meter by request. On the Spark, a rail pass that fires ten detectors before each generator call is free — the only cost is latency you spend yourself. That opens designs (layered rails, redundant detectors, custom Presidio pipelines) that wouldn’t make it through a cost-review anywhere else.

Where the rail sits in the chain

Guardrails sits on the host, not inside a NIM container. It’s a Python package that wraps any OpenAI-compatible chat endpoint — the local Llama 3.1 8B NIM at :8000 qualifies, and so does any hosted NVIDIA API endpoint. For each call it runs the configured input rails against the user message, then forwards to the LLM, then runs the configured output rails against the answer. The retrieval chain from articles #4-#7 stays untouched; we inject retrieved chunks into the user message before handing off to the rail, same as before.

The install is boring, which is the point

Guardrails is a normal pip package. The only friction was a transitive dependency on annoy (a C++ approximate-nearest-neighbor library), which needs python3-dev to build on aarch64 because no pre-built wheel ships for Python 3.12 on arm64.

sudo apt-get install -y python3-dev build-essential
python3 -m venv /tmp/guardrails-venv
/tmp/guardrails-venv/bin/pip install nemoguardrails langchain-openai
/tmp/guardrails-venv/bin/python -c 'import nemoguardrails; print(nemoguardrails.__version__)'
# → 0.21.0

Three minutes total after the first apt-get. langchain-openai is a soft dependency that Guardrails will demand the moment you point it at an OpenAI-compatible endpoint — the error message is helpful, but saves you one iteration to install it upfront.

The minimal config that proves the wiring works is nine lines:

# config.yml
models:
  - type: main
    engine: openai
    model: meta/llama-3.1-8b-instruct
    parameters:
      openai_api_base: http://localhost:8000/v1
      openai_api_key: nim-local
      temperature: 0.0
rails:
  input: { flows: [] }
  output: { flows: [] }

engine: openai is a slight misnomer — it just means “OpenAI-compatible REST.” The openai_api_base points at our own NIM. The openai_api_key is a throwaway string the local NIM ignores but the openai Python client insists on. Two lines into the rails config and we are already routing through a gate.

from nemoguardrails import RailsConfig, LLMRails
rails = LLMRails(RailsConfig.from_path("config-sb"))
rails.generate(messages=[{"role":"user","content":"Say hi in 5 words."}])
# → {'role': 'assistant', 'content': 'Hello, how are you today?'}

That hello lands through Guardrails, into the local NIM, back through Guardrails, and out to us. Everything after this is what policy to attach to each direction.

One wrapper, three configs

The arc-specific files live alongside the evidence as sibling directories:

articles/guardrails-on-the-retrieval-path/evidence/
├── config-sb/
│   ├── config.yml           # input + output flows named
│   └── rails.co             # Colang: define flow + execute <action>
├── config-wiki/
│   ├── config.yml           # output flow only
│   └── rails.co
├── config-auto/
│   ├── config.yml           # input + output flows
│   └── rails.co
├── guardrails_ask.py        # the wrapper
└── benchmark.py             # the 15-query synthetic run

Each arc’s config.yml declares which flows run on input and output, each rails.co defines the Colang that dispatches to Python actions, and the wrapper registers those actions programmatically. A representative Colang file — Second Brain’s PII pair:

define bot refuse pii
  "I can't process content that contains personal identifiers (email, phone, SSN, credit card). Redact or paraphrase and try again."

define flow check pii input
  $violation = execute check_input_pii(text=$user_message)
  if $violation
    bot refuse pii
    stop

define flow check pii output
  $violation = execute check_output_pii(text=$bot_message)
  if $violation
    bot refuse pii
    stop

Colang is a tiny grammar: define flow is a rule, execute calls a registered Python action, $user_message and $bot_message are built-in context variables Guardrails populates. The flow either falls through (the generator runs) or hits stop (Guardrails returns the refuse pii string as if the LLM had said it). That “as if the LLM had said it” is the design choice that lets the rail be content-identical to a refusal — downstream code doesn’t need special cases.

The actions are the content

The actions are regular async Python functions decorated with @action. For the PII pair:

from nemoguardrails.actions import action

PII_PATTERNS = [
    (re.compile(r"\b\d{3}-\d{2}-\d{4}\b"), "ssn"),
    (re.compile(r"\b(?:\d[ -]*?){13,19}\b"), "card"),
    (re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"), "email"),
    (re.compile(r"\+?\d[\d\s().-]{8,}\d"), "phone"),
]

@action(name="check_input_pii")
async def check_input_pii(text: str):
    return any(pat.search(text or "") for pat, _ in PII_PATTERNS)

Four regexes. Deterministic, zero extra LLM calls, microseconds of latency. The narrative question isn’t “can a regex detect PII?” — it can, imperfectly — but “is the rail the right shape for the job?” The rail gives you a scaffolded place to put whatever detector you want: the regex above, a call to Microsoft Presidio, an NVIDIA Nemotron-Aegis classifier, a fine-tuned DistilBERT. The rail doesn’t care. It gives you where, not what.

The same pattern holds for the Autoresearch rails — regexes for rm -rf /, curl ... | bash, --no-verify, cat ~/.ssh/*, AWS_SECRET_*. And for the Wiki rails — regex tests for hedging phrases ("as an AI", "I think", "probably") plus a check that the answer contains a literal "Sources:" trailer, since the strict-context prompt from article #6 already instructs the generator to cite. Fifteen lines of Python per arc, three arcs, ninety lines total. Everything else is Colang glue.

The wrapper reuses the article-#8 chain

The retrieval pipeline is imported directly from article #8 — no fork, no copy-paste. The wrapper’s ask() function runs hybrid_ask.retrieve() to get the top-5 reranked chunks, builds the same strict-context user message, and hands it to Guardrails:

import hybrid_ask                               # article #8, unchanged
from nemoguardrails import LLMRails, RailsConfig

def ask(question, arc, mode="rerank", k=5):
    rails = load_rails(arc)                     # registers the 5 actions
    hits, timings = hybrid_ask.retrieve(question, mode=mode, k=k)
    user_content = build_augmented_user(question, hits)
    messages = [
        {"role": "system", "content": hybrid_ask.STRICT_SYSTEM},
        {"role": "user",   "content": user_content},
    ]
    result = rails.generate(messages=messages)
    answer = result["content"]
    return {
        "question": question,
        "arc": arc,
        "answer": answer,
        "blocked": classify_block(answer) is not None,
        "blocked_by_rail": classify_block(answer),
        "retrieved": [{"id": h["id"], "label": h["label"]} for h in hits],
        "timings_ms": timings,
    }

classify_block is a three-line check against the canonical refusal strings each arc emits (“personal identifier”, “wiki style policy”, “known-dangerous pattern”). If the rail fired, the rail’s refusal text is what result["content"] contains — same return path as a normal answer, just flagged.

Fifteen queries, three verdicts per arc

The benchmark set is five queries per arc split 3:2 violating vs. clean. For Second Brain, the three violations embed recognizable PII patterns (an email address in a request about a Q3 leak, a SSN request, a credit-card-number request) and the two clean queries ask the corpus’s AG News content directly. For the Wiki, the violations are questions the corpus cannot answer (the 2009 Avatar box office, today’s Bitcoin price, the DGX Spark itself) — they will produce refusals that lack a Sources: trailer and therefore fail the style rail. For Autoresearch, the violations are planner-style prompts that embed exfiltration patterns (cat ~/.ssh/id_rsa, env | curl, /etc/passwd).

=== Per-arc summary ===
arc     viol  clean  TB  FP  TP  FB  recall  clean_pass
sb         3      2   3   0   2   0     1.0         1.0
wiki       3      2   3   0   2   0     1.0         1.0
auto       3      2   3   0   2   0     1.0         1.0

Block recall 1.0 across all three arcs. Clean pass rate 1.0 across all three arcs. Fifteen queries, fifteen correct verdicts. The record in benchmark.json carries the retrieval timings too: clean queries finish in 400–650 ms end-to-end, with retrieval (~250 ms rerank roundtrip) dominating latency and the rails themselves adding a few microseconds each. Blocked queries return in under 50 ms because the rail short-circuits before retrieval or generation runs — the input rail fires on the raw message and we never hit hybrid_ask.retrieve().

Two qualitative observations from reading the per-query records. First, the Wiki rail’s “missing-Sources” check catches more than style drift — it also catches refusals, because the strict-context refusal (“The provided context does not contain the answer.”) has no Sources: line. This is working as intended: a refusal-with-no-evidence isn’t a valid wiki entry either. Second, the Second Brain rail scans the augmented user message, which includes the retrieved chunks. If the retrieved corpus leaks a PII pattern into the context, the input rail will also block — the scrub covers both directions of the retrieval gate.

Verification on the Spark

The nvidia-smi picture stays quiet. Guardrails itself does no GPU work in this configuration — regex on strings, function calls, YAML parsing. Every call to rails.generate() that passes the input rail hits the local NIM at :8000 exactly once, which shows up as a tiny Llama-3.1-8B inference on the single GPU. Latency per call, end-to-end:

clean  wiki  "What did the article say about Michael Phelps winning medals?"
       retrieve=553 ms  generate=1.2 s  rails=<1 ms  total=1.8 s
       answer: "Michael Phelps' quest to win eight gold medals is over,
                and he won seven gold medals at the Olympics. Sources:
                [601, 594, 626, 1185]"

violating sb  "Summarize the email alice@example.com sent about the Q3 leak."
       rails=0.3 ms  blocked: pii  total=0.3 ms
       answer: "I can't process content that contains personal identifiers..."

The blocked case is genuinely free — the input rail short-circuits before embed, before retrieval, before generation. In the clean case, the rails cost is invisible inside the retrieval and generation noise. This is what a rail should feel like on local hardware: the detector cost is bounded, the skipped-call savings dwarf the guard cost, and you can afford to run more detectors without a metering conversation.

Tradeoffs, gotchas, surprises

Regex detectors are the weakest link, and that is the design. The article deliberately uses regex because it’s transparent, inspectable, and zero-cost. In production, each arc’s detector would be swapped for something meaningfully stronger: Microsoft Presidio (with an sdd extra Guardrails already supports), an NVIDIA Nemotron-Aegis classifier for PII and jailbreaks, a real AST-level parser for the Autoresearch code rail. The 100% block rate reported above is on hand-crafted synthetic queries designed to hit the detectors; the confidence interval on any real PII corpus would be wider and the false-positive profile different. The claim is not that regex is enough. The claim is that rails are the scaffolding and detectors are the content, and the scaffolding is what the article is about.

Colang is small but opinionated. Guardrails 0.21 supports Colang 1.0 and 2.0; 1.0 is what this article uses because the documentation is denser and 2.0 is still catching up. The verbs (define flow, execute, bot refuse <...>) are conventions the framework parses — bot refuse pii isn’t a magic phrase, it’s just a named utterance Guardrails attaches the following quoted string to. If you rename it, make sure classify_block still matches.

The openai engine needs langchain-openai. The error message is good (Initializing ChatOpenAI requires the langchain-openai package), so the fix takes fifteen seconds, but it’s not installed by the default pip install nemoguardrails. Expect the same shape for other engine types — nvidia_ai_endpoints for hosted NeMo, anthropic for Claude, vertexai for Google — each is its own optional extra.

Input rails run before retrieval. This sounds obvious and it’s the efficient choice for SB PII (why embed a query that won’t be generated?), but it means the Autoresearch exfil patterns never reach the retrieval chain, and therefore never reach the agent’s code-analysis layer. If you want a rail that inspects retrieved code — for example, because the agent is reading train.py from disk as part of its context — that has to be a retrieval rail, which Guardrails supports via the rails.retrieval section but which this article skips to keep the surface area small. Article #A5 (Autoresearch code generation) will return to that gap.

config.yml needs a valid prompts: key, even when empty. Leaving it off produced a schema-validation error on the first draft of the three configs. An empty list (prompts: []) satisfies the parser. The documentation mentions it but doesn’t emphasize that it’s not optional for YAML-only configs.

What this unlocks

Three Monday-morning builds, one per arc:

Second Brain with a private-first retrieval front door. Drop the Second Brain MCP server (coming in track S4) behind this rail config. Every query from Claude Code goes through check_input_pii before any chunk is retrieved, and every answer goes through check_output_pii before it comes back. The corpus can contain emails and phone numbers; the responses won’t. This is the specific privacy shape the three-arc thesis promised.

LLM Wiki with a write-policy gate on the bookkeeper agent. The wiki-ingest path (track W2, with NeMo Curator) will propose page edits via the same 8B NIM. Wrap those proposals in the Wiki rail and the bookkeeper can’t ship a page without a Sources: trailer and without flagging self-referential or hedged prose. Wiki voice becomes a rail, not a prompt instruction the model can ignore under load.

Autoresearch with a code-safety preflight on the planner. The agent’s edit-run-measure loop (track A4) will propose shell commands to the runner. Route those through the Autoresearch rail and the planner physically cannot propose rm -rf, curl | bash, cat ~/.ssh/*, or --no-verify — the input rail blocks the message before the runner ever sees it. Strictly stronger than a prompt instruction, strictly weaker than a full seccomp sandbox. The right layer for the risk class.

Closing — state of the apps

The shared foundation is complete. Seven articles, one machine, one retrieval chain, one generator, and now one policy gate with three configs. All three arcs have the same runway; the choice of what to build next is the user’s.

Second Brain now: has a brain, a retrieval chain, a policy gate, and no app yet. LLM Wiki now: has a writer, a retrieval chain, a write-policy, and no pages yet. Autoresearch now: has a driver, a retrieval chain, a code-safety rail, and no loop yet.

Next up: one-substrate-three-apps — the bridge article. Hub-and-spoke diagram, three colored forks, a short essay on the cost space the three arcs cover. After that, the tracks fork and readers pick.