Vibe Coding Agent Security And Evaluation

Sokratis Kartakis, Aron Eidelman, Wafae Bakkali, Meltem Subasioglu · pdf · 2026-06-08 Reader 🖥️

Agent Harness Agent Security Agentic Engineering Evals MCP (Model Context Protocol)Sandboxing Software Supply-Chain Security Vibe Coding

Operationalizing vibe coding in the enterprise requires redefining trust across two distinct, jointly necessary axes — a defence-in-depth 'Security' architecture (seven pillars: infrastructure, data, model, application/runtime, identity, observability/SecOps, governance) that proves an agent stayed inside its boundary, and a seven-dimension 'Evaluation' framework that proves what happened inside that boundary was actually worth shipping — because a vibe-coded agent can pass every security check while still misreading intent, and no single check (deterministic or AI-judged) covers both questions at once.

practical · expository — A Google whitepaper arguing organizations should adopt two specific frameworks (a 7-pillar security architecture and a seven-dimension evaluation framework) for operationalizing vibe-coding agents. Practical throughout: every section ends in 'therefore do X,' and the conclusion is explicitly prescriptive.

Structure

Part	Role
Introduction: reframes the core shift as 'from writing code to expressing intent' and splits enterprise trust into Security (stayed inside the boundary) and Evaluation (worth shipping).	thesis statement
The 7-Pillar Security Architecture (Infrastructure, Data, Model, Application/Runtime, IAM, Observability & SecOps, Governance): a layered defence-in-depth baseline, each pillar mapped to a named owner role and concrete controls.	framework / evidence
Deep-dives into specific vibe-coding threat vectors within the pillars — sandboxing and supply-chain defence, frontend/backend application vulnerabilities, MCP spoofing, identity and the Confused Deputy problem, Red/Blue/Green agentic SecOps, and observability (Vibe Trajectory, AgBOM, Trust Decay, circuit breakers).	elaboration / worked mechanism
The Evaluation framework: why vibe-coding evaluation differs (underspecification gap, user-can't-validate gap, iterative-session-as-state), seven evaluation dimensions, and the methods (automated tests, LLM-as-judge, browser testing, trajectory inspection, human review, online eval, standardized benchmarks) mapped to each dimension.	second framework / evidence
Applied tips with code snippets (session-prefix-as-rubric, judging the rendered artifact, session convergence, mining corrections) and a conclusion restating that generation is solved while verification/security/judgment are the new craft.	worked examples / practical close

Key terms

Effective Trust
A continuous, dynamically-enforced metric evaluated across an agent's supply chain, identity, runtime behaviour, and contextual associations — never a one-time gate an agent passes through at deployment.
Context-as-a-Perimeter
A security model that replaces static Identity-as-a-Perimeter (RBAC-style 'if you hold the token, you're trusted') with dynamic, contextual evaluation of whether a given tool call matches the developer's actual intent at that moment.
Vibe Trajectory
A unified, chronological OpenTelemetry-based trace of an agent's cognitive steps — API calls, tool inputs/outputs, RAG retrievals, token latency — from the user's initial prompt to the compiled output, used to answer 'why did the agent do that?'
Runtime Agent Bill of Materials (AgBOM)
A living, millisecond-fresh inventory of the tools, models, and data sources an agent is actively using — contrasted with a static SBOM, which the paper argues goes stale the moment an agent dynamically pulls in a new tool.
Underspecification Gap
The absence of a complete, rigid specification to test against in vibe coding — the user's natural-language prompt ('make the dashboard load faster') is not a test case, so the agent must reconstruct the unstated spec itself, and evaluation's first job is to judge whether it reconstructed the right one.

Propositions

explicitA raw AI model is not an agent; it only becomes one when wrapped in a harness — the scaffolding that provides state, tool execution, feedback loops, and enforceable constraints — so securing vibe coding means securing the harness, not the model's code syntax.conf: high
supportsThe New SDLC With Vibe Coding: From ad-hoc prompting to Agentic Engineering · p7 — Both state the same claim in near-identical terms: a model is only one input into a working agent, and the harness (prompts, tools, context, sandboxes) dominates observed behaviour — independent convergence from a security-framework angle and a developer-practice angle.
explicitSecurity and Evaluation are two distinct, jointly necessary axes of trust: Security proves an agent stayed inside its boundary; Evaluation proves what happened inside that boundary was actually worth shipping. A vibe-coded agent can pass every security check and still misread intent, ignore conventions, or degrade user experience.conf: high
explicitEveryday agent failures most often trace back to a missing tool, a vague rule, or an absent guardrail — i.e., harness configuration gaps — which motivates replacing static Identity-as-a-Perimeter models with a dynamic 'Context-as-a-Perimeter' model.conf: high
supportsThe New SDLC With Vibe Coding: From ad-hoc prompting to Agentic Engineering · p8 — Restates, almost verbatim, the sibling document's claim that most agent failures trace to a missing tool, a vague rule, or an absent guardrail — i.e. harness configuration gaps rather than model limitations.
explicitcontestedAn agent must authenticate using a dedicated identity explicitly tagged as 'agentic' rather than operating under the human user's delegated credentials, because delegated credentials grant the agent dangerous ambient access and leave it exposed to the Confused Deputy problem (an over-privileged agent tricked by a prompt injection into executing an unauthorised command).conf: high
contradictsCloudflare WriteGuard Brings Fine-Grained Security Controls for MCP Servers · p7 — Kartakis et al. insist an agent must authenticate via a dedicated, explicitly-tagged 'agentic' identity rather than the user's delegated credentials, to avoid the Confused Deputy problem. WriteGuard's design takes the opposite position: reusing the human's existing OAuth identity plus contextual tagging (client, session) is sufficient for accountable, auditable agent write actions, and a dedicated agent-identity system is unnecessary.
explicitZero Ambient Authority requires Just-In-Time (JIT) token downscoping: an agent executing a task should receive fresh, hyper-restricted credentials scoped to exactly that task's data sources, expiring the moment the task concludes, rather than inheriting a parent agent's or a developer's broad permissions.conf: high
explicitLarge language models frequently hallucinate software package names, and attackers actively monitor for these hallucinations to publish malware under the fabricated names ('slopsquatting'), so agents must source dependencies only from vetted internal registries with cryptographic version pinning, gated by SBOM/signature verification in CI/CD.conf: high
explicitA simple domain allowlist cannot secure an agent against indirect prompt injection hidden in third-party web content, so agents should be restricted to non-interactive internet access via offline caches or pre-sanitised crawling services rather than direct browsing.conf: high
explicitHigh-stakes agent actions (production database changes, financial transfers, IAM edits) should require structured, context-aware elicitation — hardware MFA plus a 'Vibe Diff' that translates the generated code into a plain-English summary — because simple approve/deny buttons cause confirmation fatigue and lead developers to authorise code they don't understand.conf: high
explicitSecurity operations must themselves become agentic (a continuous Red/Blue/Green triad) because the attack surface in vibe coding is non-deterministic, natural-language-driven, and changes faster than manual security review can track; traditional UEBA is ineffective against non-deterministic AI behaviour.conf: high
explicitWhen a compromised agent is detected, a 'Stateful Quarantine' (gracefully revoking tool access while preserving short-term memory for forensics) is safer than killing the host container, because terminating an agent mid-thought can leave connected APIs in a corrupted state.conf: high
explicitA 'successful' status code (e.g. HTTP 200) can mask an agent that has quietly cascaded into a hallucination loop; this creates a distinct security risk category, 'Denial of Wallet' (DoW), where an adversary deliberately triggers infinite, expensive API loops to bankrupt an organisation's cloud/LLM billing.conf: high
explicitVibe-coding evaluation differs from evaluating deterministic software or other agent categories along three axes: there is no complete spec to test against (underspecification gap), the gap between 'the agent thinks it succeeded' and 'the code is actually correct' is unusually wide because users and even experienced engineers cannot fully validate the output, and each session is iterative over a stateful, living codebase where bad early decisions compound.conf: high
explicitVibe-coding evaluation breaks into seven dimensions — intent satisfaction, functional correctness, visual/behavioural correctness, cost and efficiency, code quality/convention matching, trajectory quality, and self-repair behaviour — with safety and responsible AI cutting across all seven rather than being a separate gate.conf: high
explicitNo single evaluation method covers all seven dimensions; production pipelines must combine automated functional testing, security/safety scanning, LLM-as-judge/agent-as-judge, browser-based testing, trajectory inspection, human review, online sampling, and standardised benchmarks, each mapped to the dimensions it is actually suited for.conf: high
explicitStandardised benchmarks (SWE-bench Verified, LiveCodeBench, Vibe Code Bench, Kaggle Standardised Agent Exams) provide a useful, contamination-resistant empirical baseline for cognitive calibration, but overfitting to them is a real risk: a high SWE-bench score proves an agent can navigate a structured repository, but gives zero guarantee it has the aesthetic judgment to vibe-code a consumer-facing application.conf: high
explicitThe relevant success unit for a vibe-coding session is convergence — did the user end up with something they wanted — not turn-level accuracy; sessions abandoned mid-flow are far more informative failure signals than individual incorrect turns.conf: high

Arguments

Trust in an agentic system cannot be a static, pre-deployment gate (as in deterministic software); it must be a continuously re-evaluated metric across supply chain, identity, runtime behaviour, and context ('Effective Trust'). strength: high

Explicit premises:

In deterministic software, trust is binary: code compiles, tests pass, credentials are valid.
An autonomous agent possesses ambient agency to execute generated code, access internal APIs, and modify production environments on an ongoing, unpredictable basis.
Static identity (holding a valid token) is a poor perimeter because an agent can hold a valid token while operating with misaligned intent.

Implicit premises:

Pre-deployment checks (tests, static scans) cannot anticipate the behaviour of a non-deterministic system across all future runtime contexts.
An organisation that treats agent trust as a one-time gate will systematically under-detect in-session drift.

Inferences:

Every security control the paper proposes (sandboxing, JIT downscoping, circuit breakers, Red/Blue/Green teaming) is really an instance of the same underlying move: replacing a one-time check with a continuously-monitored, revocable one.
Observability (Pillar 6) is not an optional add-on to this architecture but a logical prerequisite for any of the other pillars to function as continuous rather than static controls.

Narratives

A simple prompt to 'optimise the database query' drifts into the agent attempting to download a new, unauthorised indexing library — offered as the illustrative example of Intent Drift triggering Trust Decay.
→ Makes the abstract Trust Decay / circuit-breaker mechanism concrete: shows exactly the kind of small, plausible-looking deviation the AgBOM-monitoring and automated rollback machinery is built to catch before it compounds.
Wiz Research's finding that attackers monitor for LLM-hallucinated package names and proactively publish malware under those exact fabricated names ('slopsquatting'), so that agents autonomously altering dependency graphs pull malware straight into the build.
→ Grounds the supply-chain-defence pillar in a documented, named attack pattern rather than a hypothetical, and justifies the specific prescription (vetted registries, cryptographic version pinning, SBOM/signature gates) as a response to a real adversarial behaviour, not a generic best practice.

Questions addressed

✅ How should an enterprise secure a non-human workforce (coding agents) that has ambient agency to execute code, access internal APIs, and modify production environments autonomously?
The 7-pillar defence-in-depth architecture (infrastructure/sandboxing, data, model, application/runtime, IAM, observability/SecOps, governance), organized around continuously-earned 'Effective Trust' rather than a one-time deployment gate.
✅ Given that security alone cannot tell you whether an agent's output is actually good, how should you evaluate whether a vibe-coded agent delivered what the user meant?
A seven-dimension evaluation framework (intent satisfaction, functional correctness, visual/behavioural correctness, cost/efficiency, code quality, trajectory quality, self-repair), each mapped to the specific method(s) capable of measuring it, with session-convergence rather than turn-level accuracy as the key unit of analysis.
⭕ What should happen to the feedback loop from user corrections back into improving the model, harness, or eval suite over time?
Explicitly named as out of scope ('both warranting dedicated treatment'), alongside subjective evaluation of non-verifiable, preference-defined outputs.

Gaps

incompleteEvery control in the 7-pillar architecture is justified by first-principles reasoning about non-determinism and by citing named threat patterns (slopsquatting, invisible payloads, Confused Deputy), but no incident data, overhead measurements, or cost-benefit comparison is offered for any proposed control — e.g., no data on how much the 'Vibe Diff' + hardware MFA step actually reduces blind-approval rates versus a simpler gate, or on the operational cost of running a continuous Red/Blue/Green triad alongside every development workflow.
Read next: This corpus's own extract-lulla-loop-engineering-building-blocks-adoption-impact documents, for a structurally similar case (loop engineering's prescribed building blocks), that almost none of a comparably detailed practitioner-prescribed architecture shows up as committed, verified practice in real repositories — worth reading against this whitepaper's unverified 7-pillar prescriptions for the same pattern.
incompleteThe paper prescribes a dedicated, non-delegated 'agentic identity' (SPIFFE IDs, JIT downscoping) as the resolution to the Confused Deputy problem, but never engages with the operational cost or adoption barrier of standing up a full non-human identity system — issuing, rotating, and revoking per-agent cryptographic identities at the granularity the paper implies (per sub-agent, per task) is a significant IAM-infrastructure undertaking the paper treats as a given rather than a tradeoff.
Read next: Compare against this corpus's treatment of centralized MCP authorization control planes (extract-mcp-centralised-auth-enterprise), which raises exactly this concentration-of-risk question for a comparable identity-centralization design.
incompleteThe evaluation framework explicitly scopes out 'the feedback loop from user corrections back into the model, the harness, or the eval suite' as future work — but the applied tip 'Mine user corrections as labeled failure data' in the same document is precisely that feedback loop's first step (clustering corrections into failure modes), left unconnected to how those clusters would actually drive harness or eval changes.
Read next: The corpus's context-engineering and agent-config-decay material on how harness/context changes get fed back from observed failures.

Closes gaps elsewhere

AI Model Context Protocol Adds Centralised Auth for Enterprise · g2 — Supplies the specific non-human identity standard that gap asked for by name: unique cryptographic SPIFFE IDs per agent, combined with ABAC and JIT token downscoping, as the mechanism for verifying 'who' is acting without relying on a human's delegated token — though it does not analyze the EMA/ID-JAG control-plane's own concentration-of-risk question specifically.