Research

A Planner-Directed Deep Agent with Skill-Equipped Specialists for Autonomous Capture-the-Flag Solving

Project Details

Abstract

Large language model (LLM) agents are increasingly applied to autonomous penetration testing, yet “autonomous” conflates two distinct capabilities: generating correct commands in isolation, and sustaining coherent, stateful interaction over a long horizon. We first formalize a stateful single agent that maintains a persistent session and emits exactly one structured action per turn. On sequential cybersecurity wargames, the agent performs substantially better when levels are solved independently than when they must be completed as a continuous sequence, revealing long-horizon state management as a major bottleneck. Existing systems often address this loss of coherence by distributing planning, execution, and memory across a fixed team of specialized agents, but such designs invoke every role on every cycle and therefore incur a constant coordination cost. We instead introduce a planner-directed Deep Agent: a single non-executing planner that delegates on demand to transient specialists, each equipped with retrievable procedural skills and adaptive escalation to a stronger model. On the NYU CTF Benchmark test split, using only mini-tier backbones (gpt-4o-mini with gpt-5-mini escalation), it attains a leak-audited genuine solve rate of 13.5% (27 of 200 challenges), compared with a raw accepted-flag rate of 19.5% before excluding environment-leak shortcuts, exceeding the benchmark’s original ≤5% baselines and matching prior agentic systems built on substantially larger frontier models. A planner-directed Deep Agent is thus a capable and cost-efficient design for autonomous penetration testing, reaching competitive performance with only mini-tier backbones.

2025-2026-research-grad-ece-amine.jpg