Preference Model publishes Karotte technical deep dive: how RL environments block reward hacking

Preference Model open-sourced the Karotte framework for building robust reinforcement learning environments, blocking entire classes of reward hacking by default. The article first builds a CSV order summation task with about 40 lines of handwritten code, demonstrating failure modes one by one such as reading the answer file, tools without timeouts, scoring scripts tricked by symlinks or FIFOs, and PATH injection, then presents Karotte's corresponding approaches. Karotte runs agents as an unprivileged student user, places answers and generated code in a root-only directory, blocks external network access with a firewall, gives bash tools timeouts and output truncation, and before scoring, collect_submission kills and confirms all student processes, rejects symlinks and special files, and copies submissions to a root-only directory, with the scoring script run using the environment's own Python.