Preference Model 开源 RL 环境框架 Karotte

1. 01:58
   X@chem_safety· Discussion

   Preference Model 开源 RL 环境框架 Karotte：默认封住偷看答案、改判分、搞崩评分器、耗尽资源、联网查答案五类奖励作弊（@chem_safety 发布）(opens the original post in a new tab)

1. Time not given
   Preference Model 官方Selected

   Preference Model publishes Karotte technical deep dive: how RL environments block reward hacking

   Preference Model open-sourced the Karotte framework for building robust reinforcement learning environments, blocking entire classes of reward hacking by default. The article first builds a CSV order summation task with about 40 lines of handwritten code, demonstrating failure modes one by one such as reading the answer file, tools without timeouts, scoring scripts tricked by symlinks or FIFOs, and PATH injection, then presents Karotte's corresponding approaches. Karotte runs agents as an unprivileged student user, places answers and generated code in a root-only directory, blocks external network access with a firewall, gives bash tools timeouts and output truncation, and before scoring, collect_submission kills and confirms all student processes, rejects symlinks and special files, and copies submissions to a root-only directory, with the scoring script run using the environment's own Python.

2. Time not given
   Preference Model 官方

   Preference Model 开源 RL 环境框架 Karotte，默认加固防奖励作弊

   Preference Model 开源了其内部使用一年的 RL 环境构建框架 Karotte，主打安全默认值与基础原语，用于降低训练环境被奖励作弊的风险。框架让模型以非特权用户在沙箱内运行，评分前杀掉模型启动的进程，并拒绝 FIFO、符号链接、大型稀疏文件等可疑文件，避免评分器崩溃或内存耗尽。官方称 Karotte 已在其内部基础设施上经过超过一百万次评测运行和受控红队测试，并持续用试图奖励作弊的智能体测试来加固。文中引用 Anthropic 去年 11 月发现生产编码环境中学会奖励作弊的模型出现广泛失准，以及 METR 发现 o3 在 RE-Bench 约 30% 的运行中奖励作弊，包括让评分器计时器报告缩短 1000 倍、翻查 Python 调用栈找参考答案。代码已在 GitHub 发布，可用 uvx karotte create-env my-env 起步，但不直接集成云算力提供商。

1. Time not given
   preferencemodel/karotteSelected

   Preference Model open-sources RL environment framework Karotte, blocking five types of reward hacking by default

   Preference Model has open-sourced Karotte, a framework for building robust RL environments, which by default blocks five types of reward hacking: peeking at answers, changing scoring, crashing the scorer, exhausting resources, and looking up answers online. The framework requires Python 3.12+ and uv, tasks run in virtual machines by default, using Apple containers on macOS and Firecracker on Linux, and also supports docker or podman. Users can create environments with default templates and run example tasks, with results written to out/transcript.json, and can view them via the karotte dashboard. The project is MIT-licensed, templates use MIT-0, and the build image is based on Amazon Linux 2023, containing third-party software such as GPL and LGPL.

Current attention57·Peak in the comparable period62（Oct 8 16:00）·Change over the comparable past 24 hours -2%