LLM security testing becomes useful when it is tied to a specific system state: model, prompt, retrieval source, tool permissions, guardrails, and release date. Without that context, a scan result is easy to overstate or dismiss.
NVIDIA garak, short for Generative AI Red-teaming and Assessment Kit, is an open-source LLM vulnerability scanner for models and dialogue systems. For a production team, the value depends on whether the test run matches the way the application can actually fail.
For teams building AI features in Singapore, including startups and SMEs, that difference matters. Enterprise customers, procurement teams, and internal risk owners increasingly want evidence that AI behaviour has been tested before deployment. A garak run can support that evidence, but only when the scope and interpretation are handled carefully.
LLM Testing Has Different Failure Modes
The garak paper frames LLM security as a moving target: models are updated, outputs vary, and a weakness in one context may be irrelevant in another. That is the practical reason to treat each scan as a measurement of a configured system, not a general statement about a model family.
For a production feature, the risk rarely sits in a single prompt. It may come from retrieved documents, user-controlled context, hidden instructions in web pages, tool calls, memory, logging, or policy boundaries that the model is expected to respect. A good garak plan starts by deciding which of those surfaces should be stressed.
The minimum run metadata should include the target endpoint, model or deployment name, system prompt, retrieval configuration if relevant, guardrail version, selected probes, detectors, generation count, seed if used, and run date. Without that metadata, the report has limited value for release review because nobody can tell what changed between two results.
Where NVIDIA garak Helps
garak is built around plugins. In garak terminology, generators, probes, detectors, evaluators, harnesses, and buffs define the scan. The generator wraps the target model or service. Probes send adversarial prompts. Detectors score responses. Evaluators and harnesses organise the run. Buffs can transform prompts, for example by applying encodings or other perturbations.
The plugin model matters because a scanner run is a test specification: which generator represents the real target, which probe modules match the risk, which detectors are acceptable for the finding, and how many generations per prompt are needed to make the result meaningful.
The CLI reference exposes many of those choices directly: --probes, --probe_tags, --detectors, --extended_detectors, --buffs, --generations, --config, --report_prefix, and --taxonomy. For custom applications, the configuration docs are more useful than the basic README because they show how to define plugin behaviour, target settings, REST endpoints, reporting options, and confidence interval settings.
That matters for product testing. If the deployed AI feature is behind an application endpoint, the target should usually be that endpoint, not a raw foundation model. The REST generator can be configured with an endpoint URI, authentication environment variable, headers, response field, and timeout. That lets the scan exercise the same prompt wrapper, guardrails, and application response path that users see.
Running Focused Probes
Probe selection should follow the application architecture.
For a RAG assistant, prompt injection, leak replay, context manipulation, and encoded instruction probes are more relevant than a broad default run. The scan should target the application path that includes retrieval and any output filtering, because testing the base model alone will not tell the team whether private documents can be coerced into the response.
For a coding assistant, package hallucination, insecure code generation, malware-related output, XSS-oriented output, and unsafe command patterns deserve attention. The result should be reviewed against the product's actual behaviour: whether the assistant only explains code, writes pull requests, suggests dependencies, or can trigger build and deployment workflows.
For a customer-facing chatbot, jailbreaks, toxic output, misinformation, escalation bypass, and policy refusal failures are usually more important. If the chatbot handles support tickets or customer data, leakage tests should be included even when the model is not directly connected to a sensitive database.
For agentic systems, garak can test language behaviour, but tool abuse still needs application-layer review. If the agent can send email, update records, query tenant data, or call internal tools, the test plan should include permission checks and simulated misuse paths outside the scanner.
Reading The Results
For repeatable testing, use a config file rather than relying on an ad hoc command. This keeps probe choices, generation count, target settings, and reporting options visible to the team.
run:
generations: 5
plugins:
target_type: rest
probe_spec: promptinject,leakreplay,encoding
reporting:
report_prefix: garak-rag-release-2026-07-01
taxonomy: owasp
The exact target configuration will depend on the endpoint, but the principle is the same: keep the run narrow enough to interpret, and store the configuration beside the release evidence.
garak writes machine-readable and human-readable artefacts. The reporting documentation describes the JSONL report, HTML report, and JSONL hit log. The separate AVID report reference is useful when teams need to convert garak output for downstream vulnerability sharing or analysis.
The hit log is usually the fastest place to start because it contains prompt and response pairs that were scored as target failures. From there, the reviewer should check which detector fired, whether the response is reproducible, whether the finding maps to a real product risk, and whether the issue points to the model, prompt, guardrail, retrieval layer, or application permissions.
Percentages need care. A high pass rate can still hide a serious failure if the successful attack path affects sensitive data or an irreversible action. A low pass rate can still be acceptable if the response is harmless in the product context. Treat the report as a triage input, then convert confirmed issues into engineering tickets with the prompt, response, detector, target version, and expected control change.
Limits Of Scanner-Based Testing
Detector quality is one of the main constraints. Some detectors are straightforward string or pattern checks. Others rely on classifiers or more subjective judgement. A 2024 comparative analysis of open-source LLM vulnerability scanners found reliability issues in detecting successful attacks, which is a useful reminder that scanner output should be reviewed rather than accepted mechanically.
Business policy has to be encoded into the test plan. A response that is acceptable for an internal research assistant may be unacceptable for a regulated customer workflow. A detector may flag an output that is technically undesirable but commercially irrelevant, or miss a response that violates the company's own policy.
Application integration is the other limit. Serious failures often involve retrieval scope, tool permissions, tenant isolation, audit logging, memory, human approval paths, or workflow side effects. garak can provide pressure against the language interface, but those surrounding controls need direct testing.
Guardrail testing should be comparative. NVIDIA's NeMo Guardrails vulnerability scanning example compares a bare LLM, general instructions, dialogue rails, and moderation rails across garak vulnerability categories. That is a useful pattern: scan before and after control changes, then pair the security results with positive tests to check whether legitimate user tasks still work.
Using garak In A Release Process
A practical release workflow is simple enough for a startup and still useful for a larger organisation.
Define the AI feature and risk surface. Select probes that map to that risk. Run against the same endpoint, prompt, retrieval configuration, and guardrails intended for release. Store the config, JSONL report, hit log, HTML output, and reviewer notes. Re-run the same test when the model, system prompt, retrieval source, guardrail, or tool permission changes.
For Singapore companies, this creates evidence that can support customer security reviews, internal AI governance, vendor due diligence, and discussions about privacy or data-handling risk. It will not replace a full AI security assessment, but it gives teams a concrete way to show that known LLM failure modes were considered before release.
The review should end with decisions, not just a report. Some findings may require prompt changes. Others point to retrieval filtering, output validation, stricter tool permissions, extra human approval, or a product decision to remove a risky capability.
Palisade Support For LLM Security Testing
Palisade helps teams turn LLM scanning into an assessment workflow. For garak specifically, we can help scope the probe set, configure scans against real application endpoints, review hit logs, validate detector results, and convert findings into practical remediation work.
We also look beyond the scan: prompt-injection paths, RAG data exposure, guardrail design, tool permissions, logging, approval flows, model-change controls, and release evidence. That wider review is especially important for teams deploying customer-facing chatbots, internal assistants, coding tools, and agentic workflows.
To discuss LLM red teaming, AI application security, or garak-based testing, book a free consultation.
