Training AI to solve CAPTCHAs

A study built a CAPTCHA-solving system composed of a fine-tuned YOLOv8 detector and an open-weight VLM, routed by confidence, achieving 85.4% overall accuracy and 84.2% macro-average across 16 categories. The system runs entirely on screenshots and operating system input events, without browser automation or DOM access. Each answer given by the VLM also serves as a training label, and the detector typically absorbs previously untrained categories after one or two encounters, without manual annotation. CAPTCHA operators can apply perturbations to the public detector, driving its accuracy to 0%, but the perturbations do not affect the VLM, and the labels the VLM provides allow the detector to recover; in a one-year simulated attack-defense in which the operator remakes the perturbations monthly, the solver recovers every round, and a cheap open-weight teacher model at about 70% accuracy is as effective as a perfect oracle. Bruce Schneier called it an interesting study.