Preference Model 官方, preferencemodel/karottePreference Model 官方 · 2 sources · 3 reports · Preference Model open-sources RL environment framework Karotte
Preference Model has open-sourced Karotte, an RL environment construction framework it has used internally for a year, focused on secure defaults and basic primitives to reduce the risk of reward hacking in training environments. The framework runs models as unprivileged users (student) in a sandbox, places answers and generated code in root-only directories, blocks external network access via firewall, kills processes started by the model before scoring, and rejects suspicious files such as FIFOs, symbolic links, and large sparse files to prevent the scorer from crashing or running out of memory. The company says Karotte has undergone over one million evaluation runs and controlled red team testing on its internal infrastructure, and is continuously hardened by testing with agents attempting reward hacking. The technical explainer builds a CSV order summation task with about 40 lines of handwritten code, demonstrating failure modes such as reading answer files, tools without timeouts, scoring scripts tricked by symbolic links or FIFOs, and PATH injection, then shows Karotte's corresponding approaches. The framework blocks five categories of reward hacking by default: peeking at answers, changing scores, crashing the scorer, exhausting resources, and querying the network for answers. It requires Python 3.12+ and uv, and tasks run in virtual machines by default—Apple containers on macOS and Firecracker on Linux—with support for docker or podman.