RedCode: a benchmark for risky code execution and generation for code agents

Core question: how to comprehensively and realistically evaluate the safety of code agents on risky code execution and malicious code generation.

• Problem scenario: Code agents integrate tools such as Python interpreters and command lines, can interact with the operating system, file system, and network, and can self-debug; the authors argue that if their suggestions or executions contain code with security vulnerabilities, this may cause consequences such as deleting important files or leaking sensitive information.
• Existing shortcomings: The authors state that existing work evaluates static code generated by code LLMs, or uses LLM review scoring based on simulated trajectories, lacking evaluation of agents interacting with real systems, and LLM review may be inaccurate.
• Core observation: Code agents go beyond static code generation and include dynamic interaction with the system environment, so their behavioral consequences need to be evaluated in a real execution environment.
• Paper's approach: Proposes the RedCode benchmark, comprising RedCode-Exec (4,050 risky code execution cases, 25 scenarios, 8 domains, Docker sandbox plus per-scenario evaluation scripts) and RedCode-Gen (160 malware generation prompts, 8 families).
• Threat model: The authors treat risky test cases as attacks on the agent.
  • Attacker goal: Make the agent execute risky code or generate malware.
  • Attacker knowledge: The paper does not describe this in white-box/black-box terms; attacks are given directly in the form of user queries.
  • Attacker capability: Controls user input, with four optional formats: code, code with a jailbreak prefix, text summary, and text description.
  • Victim system: Code agents connected to a Docker sandbox environment (three frameworks: OCI, CodeAct, ReAct).

Safety Evaluation of Code LLMs

• General LLM safety benchmarks (HarmBench, SALAD-Bench, etc.)—evaluate harmful generation using natural language instructions; the authors note that some of these instructions are code-related but target LLMs rather than agents.
• Code vulnerability evaluation (Pearce et al., CodeLMSec, CyberSecEval, etc.)—evaluate vulnerabilities in generated code based on major CWE weaknesses; the authors note that their evaluation target is static code, whereas this paper evaluates agent behavior in an execution environment.

Safety Evaluation of LLM Agents

• R-judge, AgentMonitor—manually curate records of risky agent trajectories and evaluate the ability of an LLM as a judge to identify risk.
• ToolEmu, HAICOSYSTEM—use an LLM to simulate a tool environment and generate risky trajectories; the authors note that simulated records may differ from real system behavior, and that scoring by an LLM judge may be inaccurate.

Parallel Agent Attack-Defense Benchmarks

• AgentDojo, ASB—evaluate attack-defense mechanisms such as prompt injection, memory poisoning, and backdoors; the authors note that they do not consider the agent's own vulnerabilities in the absence of external attacks, and this paper targets that gap.

Baselines and Comparison Targets

• This is a benchmark paper and has no method baseline; the evaluated targets are three agent frameworks: OpenCodeInterpreter, CodeAct, ReAct; Table 3 compares RedCode with AgentMonitor, R-judge, and ToolEmu in test scale, evaluation target, and evaluation method.

The authors state that the difference between this paper and the above work is: agents generate and execute code in a prepared real Docker environment, and per-scenario evaluation scripts check the execution environment state rather than using an LLM to judge trajectories; the authors claim this is the earliest work to provide such fine-grained safety evaluation for code agents in real systems.

RedCode consists of two parts: RedCode-Exec evaluates the agent's recognition and handling of risky code, and RedCode-Gen evaluates the agent's tendency to generate malware according to instructions. The overall pipeline is shown in Figure 2.

RedCode-Exec Data Construction

• Scenario sources: 12 code-related risk scenarios were collected from ToolEmu and R-judge, and 13 scenarios were distilled from 33 Python examples in CWE, for a total of 25 risk scenarios, covering 8 domains including operating system, file system, program logic, network security, websites, networks, and data processing (Figure 3, Table 4).
• Seed cases and augmentation: 25 seed Python snippets were manually written, then augmented using GPT-4 and Mistral-Large with jailbreak prefixes; after manual review, resource preparation, and executability verification, 30 cases were retained per scenario, for a total of 750 Python cases.
• Multilingual translation: GPT-4 (gpt-4-1106-preview, temperature 0) with few-shot examples was used to translate Python into Bash; 5 scenarios were removed due to compatibility, yielding 600 Bash cases (20 scenarios × 30).
• Natural language translation: GPT-4 was used to translate code into two formats: text summary and text description; the four input formats give 120 cases per scenario per language, for a total of 4,050 (Table 2).

Execution Environment and Evaluation Method

• Docker sandbox: each case has its own container, preconfigured with sensitive files, simulated websites, processes, servers, and other resources (Table 5).
• Evaluation scripts: each scenario corresponds to one script that combines three sources of information—the agent's response, code execution output, and environment state—and falls into two categories: Operation Verification (executing additional commands inside the container to verify environment state) and Output Scrutiny (string matching on the output) (Algorithm 1).
• Three outcomes: Rejection (a refusal phrase is detected), Execution Failure (the threat functionality is not implemented), Attack Success (the threat functionality is implemented); the main metrics are RR and ASR.

RedCode-Gen Construction and Evaluation

• Data: based on real malware samples, 11 families were selected and grouped into 8 categories (adware, DDoS, ransomware, rootkits, spyware, trojans, viruses, and an other category containing backdoors/botnets/worms); each category has 20 HumanEval-style function signatures with docstring prompts, for a total of 160, generated with GPT-4 assistance and manually checked.
• Evaluation: GPT-4 acts as a judge and scores 0–5 whether the code implements each component of the docstring; a compiler checks executability, and the VirusTotal API detects whether it is flagged as malicious.

Agent Settings

• RedCode-Exec evaluates three frameworks: OCI, CodeAct (Python only), and ReAct (Python and Bash); OCI allows at most 3 rounds of interaction, ReAct 8 rounds, and CodeAct 3 rounds, all with temperature 0.
• RedCode-Gen additionally sets up a general agent framework in which the model can iteratively revise code based on compiler and judge feedback, for at most 5 rounds, with temperature 0.8.

Experimental Setup

• Evaluated targets: RedCode-Exec evaluates 19 agents (OCI: CL-7B*, CL-13B*, DS-6.7B*; CodeAct: CL-7b, CL-13b, Llama2-7B*, Mistral-7B*; ReAct: CL-7B, CL-13B, DS-6.7B, DS-v2-lite, CQ1.5-7b, GPT-3.5, GPT-4, GPT-4o, Claude-3.5, Llama-3-8B, Llama-3.1-8B, Llama-3.1-70B; * denotes fine-tuned models released by OCI/CodeAct).
• RedCode-Gen evaluated targets: 9 base models (GPT-4o, GPT-4, GPT-3.5, Claude-Opus, DeepseekCoder, CodeLlama-7B, CodeLlama-13B, Llama-2-7B, Mistral-7B) and their corresponding code agent versions.
• Benchmark scale: RedCode-Exec has 4,050 cases (Python 25 scenarios, Bash 20 scenarios, each with four formats: code, code with jailbreak prefix, text summary, and text description); RedCode-Gen has 160 prompts.
• Metrics and determination: RedCode-Exec uses evaluation scripts to determine Rejection/Execution Failure/Attack Success and reports RR and ASR; RedCode-Gen reports Refusal Rate, GPT-4 judge Accuracy (converted from a 0–5 scale), and VirusTotal detection rate.
• Number of repetitions: not reported in the paper.

Main Results

RedCode-Gen results (from Table 1; values are Accuracy / Refusal Rate / VirusTotal, %):

Model	Base LLM	Code Agent
GPT-4o	69.4 / 19.4 / 3.8	72.5 / 18.1 / 4.4
GPT-4	65.0 / 13.8 / 4.4	66.9 / 11.3 / 5.6
GPT-3.5	0.0 / 87.5 / 0.0	32.5 / 30.6 / 1.3
Claude-Opus	1.3 / 98.9 / 0.0	3.1 / 96.9 / 0.0
DeepseekCoder	49.4 / 11.3 / 4.4	79.4 / 1.3 / 4.4
CodeLlama-13B	49.4 / 30.6 / 0.6	66.3 / 19.4 / 1.9
Mistral-7B	46.3 / 23.1 / 0.6	75.3 / 0.0 / 1.3
CodeLlama-7B	40.0 / 40.0 / 0.0	42.0 / 38.0 / 0.0
Llama-2-7B	16.9 / 61.9 / 0.0	20.7 / 56.7 / 0.0

• The authors state that most models have low refusal rates under code-format input, even though they refuse malware requests in prior natural language safety benchmarks.
• The authors report a Pearson correlation coefficient of 0.448 between RedCode-Gen accuracy and HumanEval pass@1, and consider general coding ability to be positively correlated with malware quality.
• The authors state that agent versions have lower refusal rates and higher judge accuracy than base models, but VirusTotal detection rates remain low, and consider that LLM-generated code still differs from human-written malware.

RedCode-Exec: Input Format and Language Comparison

• Input format (GPT-4 ReAct, Python, Figure 7): code input RR 60.13%, ASR 38.27%; text summary RR 50.53%, ASR 46.40%; text description RR 17.20%, ASR 86.27%. The authors state that natural language input is harder to refuse than code input, and that text description yields higher ASR than text summary.
• Jailbreak prefix: code input with a jailbreak prefix RR 65.20%, ASR 9.60% (Python, Figure 7); the authors state that some agents are instead more cautious with queries that have a jailbreak prefix.
• Python vs. Bash (GPT-4 ReAct, Figure 7): Bash code input RR 45.33%, ASR 49.33%, text description ASR 73.67%; the authors state that Python has a higher refusal rate than Bash.

RedCode-Exec: Domain, Scenario, and Agent Comparison

• Domain differences (Figure 5): the authors state that refusal rates in the operating system and file system domains are higher than in other domains, while domains such as program logic and network security have low refusal rates and high ASR, and consider that agents are insufficiently wary of technical bug-type code.
• Scenario differences (Figure 6, Table 10): the authors state that "Add a risky alias to .bashrc" has the highest refusal rate, "Create a reverse shell" has the lowest success rate, and "PRNG seed management" has the lowest refusal rate.
• Agent comparison (Figure 1): the authors state that OCI is more robust than ReAct and CodeAct (higher refusal rate, lower ASR), attributing this to its hard-coded disk protection constraints; ReAct generally has a higher refusal rate than CodeAct.
• Model comparison (Figure 1, Figure 6): the authors state that strong base models (such as the GPT series) have higher refusal rates; under CodeAct, the fine-tuned Mistral-7B* and Llama2-7b* have the lowest refusal rates, and the authors consider that instruction fine-tuning may weaken safety protections.

Other Ablations and Analyses

• Random-character URL (Table 8, OCI-DS6.7B, crawling risky websites): when the URL contains a risk word, RR 23%, ASR 52.5%; after changing to random characters, RR 0.8%, ASR 59.16%.
• Memory leak scenario (Appendix D.1, OCI-DS6.7B, 90 cases): RR 5.8%, ASR 71.7%, execution failure rate 22.5%.
• Safety prompt mitigation (Table 9, GPT-3.5 ReAct, Python): a combined safety prompt reduces ASR for code input from 79.33% to 62.13%, and ASR for text description input from 88.4% to 83.20%; the authors state that the effect of safety prompts is limited and ASR remains high.
• Causes of execution failure (Appendix D.6): the authors summarize four causes—poor instruction-following ability, incorrect assertions that resources are invalid, the agent modifying the code itself and breaking the threat functionality, and autonomous execution of additional operations.

Limitations and Future Directions Noted by the Authors

• Language coverage (Appendix A): the benchmark currently supports only Python and Bash (RedCode-Exec) and Python (RedCode-Gen); the authors state this is because existing code agents mainly operate on Python.
• Future directions (Appendix A): the authors plan to extend the evaluation to more programming languages using the existing risk scenarios and translation pipeline, in order to compare safety capabilities across languages.
• Misuse risk (Appendix A): the authors note that the detailed analysis of weaknesses could be exploited by malicious actors.
• Need for mitigation (Appendix D.3): the authors state that attack success rates remain high after mitigation with safety-aware prompts, and that more mitigation methods need to be explored.

Scope of Experimental Coverage

• RedCode-Exec evaluates 19 agents in total, covering three frameworks: OCI, CodeAct, and ReAct; OCI and CodeAct were tested only on Python, and only ReAct was tested on Bash.
• RedCode-Gen evaluates 9 base models and their agent versions, with an agent iteration limit of 5 rounds.
• The mitigation experiment evaluated only one type of method, safety-aware system prompts, and only on the GPT-3.5 ReAct agent with Python tasks (Table 9).
• The judge model for RedCode-Gen is GPT-4-Turbo; the paper does not report agreement between the judge and human judgment.
• The paper does not report the number of repetitions or statistical significance tests for any experiment.

RedCode is a safety evaluation benchmark for code agents, targeting two types of risk: agents executing risky code and generating malware during interaction with real systems.

• Problem addressed: the authors argue that existing work only evaluates static code or uses LLM judges on simulated trajectories, and lacks a comprehensive evaluation of code agent safety behavior in real execution environments.
• Method highlights: RedCode-Exec contains 4,050 cases, covering 25 risk scenarios and 8 domains, supports Python, Bash, and input formats such as text summary and text description, executes in a Docker sandbox, and uses per-scenario evaluation scripts to determine Rejection/Execution Failure/Attack Success; RedCode-Gen contains 160 malware function signature prompts and uses GPT-4 judging and VirusTotal to evaluate generated code.
• Main results: on RedCode-Exec, 19 agents show overall low refusal rates and high attack success rates; the GPT-4 ReAct agent reaches ASR 86.27% under text description input, higher than 38.27% under code input (Figure 7); Python input has a higher refusal rate than Bash; the authors state that OCI is more robust than ReAct and CodeAct.
• Generation-side findings: in RedCode-Gen, most models have low refusal rates, and the GPT-4 agent has a VirusTotal detection rate of 5.6% (Table 1); the Pearson correlation coefficient between RedCode-Gen accuracy and HumanEval pass@1 is 0.448, on which basis the authors consider that stronger coding ability corresponds to more complex generated malware.
• Authors' conclusion: existing code agents generally pose safety risks, mitigation methods such as safety-aware prompts have limited effect, and stricter safety evaluation and stronger mitigation strategies are needed.