UK AISI open-sources Transect, turning large-scale agent evaluation records into verifiable reports.

UK AISI releases the open-source Python package Transect, built on Inspect Scout, which places activity labels, token usage, and recorded events from agent evaluation logs onto a single turn-indexed timeline, generating an interactive report that reviewers can use to jump to corresponding log segments to verify the basis of automated analysis. Users need to provide evaluation logs, task context, and the activity categories they wish to distinguish. Transect uses user-selected LLM-as-judges to label activity segments and can classify sub-agents based on delegation instructions. In an open-ended AI research evaluation, AISI used Transect to track the read/write flow of four documents—research plan, baseline code, experiment draft, and blind review—across multiple agents, showing that sub-agents first collaborate on the research plan and codebase, and after the experiment begins, focus collaboration on experiment design and data. Transect saves activity labels and individual model judgments, allowing analysis to be reopened without calling the model again; when judgments are repeated or multiple judge models are used, reviewers can see where judgments diverge.