Preference Model open-sources RL environment framework Karotte, blocking five types of reward hacking by default
Preference Model 开源 RL 环境框架 Karotte：默认封住偷看答案、改判分、搞崩评分器、耗尽资源、联网查答案五类奖励作弊（GitHub 仓库）
Preference Model has open-sourced Karotte, a framework for building robust RL environments, which by default blocks five types of reward hacking: peeking at answers, changing scoring, crashing the scorer, exhausting resources, and looking up answers online. The framework requires Python 3.12+ and uv, tasks run in virtual machines by default, using Apple containers on macOS and Firecracker on Linux, and also supports docker or podman. Users can create environments with default templates and run example tasks, with results written to out/transcript.json, and can view them via the karotte dashboard. The project is MIT-licensed, templates use MIT-0, and the build image is based on Amazon Linux 2023, containing third-party software such as GPL and LGPL.
The framework blocks five categories of reward hacking paths by default; teams doing RL environments and agent evaluation can reference its isolation and runtime design.