CAPTCHAs in the agent era: a solver that learns from each encounter

Can a visual CAPTCHA solver get stronger with use, rather than inferring from scratch each time, or forever staying at the classes it was trained on.

• Setting: The authors state that reCAPTCHA v2 is still widely deployed, yet GUI agents are often blocked by CAPTCHA. Specialized detectors cover only the training classes, and VLMs pay high latency every time; neither improves with exposure.
• Existing shortcomings: The authors state that specialized solvers rely on browser automation to read the DOM, are hard to integrate into screenshot-only agents, and leave fingerprints that can be checked. VLMs cost orders of magnitude more than discriminative models, and fine-grained localization is also difficult.
• Observation: The authors believe that once the pattern is understood, the challenge is deterministic recognition, so expensive inference can be distilled into a lightweight detector. The authors state that VLMs can already handle multiple CAPTCHA families without task-specific training.
• Proposal: A hybrid solver driven by screenshots and OS input events: YOLOv8 performs localization and handles familiar classes, open-weight Qwen-7B-VL infers when uncertain, and its answers simultaneously serve as training labels without human annotation.
• Threat model:
  • Goal: The solver automatically passes visual CAPTCHA. The defender is the CAPTCHA operator, who wants to prevent automation. The authors note that this role is reversed relative to common convention.
  • Knowledge: The defender knows the solver uses the published YOLO classifier and can apply adversarial perturbations to it; the defender does not know whether a VLM is included, nor the model identity, prompt template, or predictions.
  • Gray-box: The defender knows the solver will retrain on adversarial examples and can reproduce the published model, training data, and training procedure; what cannot be reproduced is the batch shuffling order and the VLM labels.
  • Input and system: The solver only reads screenshots and operates via OS-level mouse and keyboard events, without accessing the DOM or browser automation. The target is the classification and segmentation puzzles of visual reCAPTCHA.

Early Text and Image Solving

• von Ahn et al. (2003) proposed CAPTCHA. Bursztein et al. (2014) and Yan and Ahmad (2008) used OCR and segmentation to handle text CAPTCHAs; the authors state this drove image systems.
• Sivakorn et al. (2016) attacked reCAPTCHA v2, with a 70.78% success rate on image challenges and 83.5% on Facebook image CAPTCHA.

Specialized Detectors

• Hossen et al. (2020) used YOLOv3 to solve image reCAPTCHA v2, with an 83.25% success rate and a mean of 19.93 s, and accessed the DOM with puppeteer-firefox. The authors state that it reached 73% detection after adversarial training against its own perturbations, but did not consider gradient-based attacks.
• Weng et al. (2019) used CNN and Fast-RCNN to attack selection, sliding, and click image CAPTCHAs, with 79–90% success on the selection type.
• Plesner et al. (2024) fine-tuned YOLOv8 on 14,000 images across 13 classes. The authors state that it reaches near-perfect on classification puzzles, with 0% coverage for unseen classes, and relies on Selenium; they also note that DOM dependence is incompatible with pure-vision agents and leaves fingerprints such as navigator.webdriver.

Vision Agents and VLMs

• OmniParser uses a pretrained YOLO to localize small icons; the authors state that it does not handle CAPTCHA layouts. Zhang et al.'s UFO instead uses the Windows UI Automation API, but still requires accessibility hooks and is not pure screenshot.
• Deng et al. (2025)'s OEDIPUS uses a DSL to decompose reasoning CAPTCHAs, with a mean success rate of 63.5% and latency over 100 s. The authors state that this paper changes the VLM from solving independently each time into a teacher, to address the cost and speed bottlenecks therein.
• Teoh et al. (2025)'s Halligan uses GPT-4o for search, 60.7% across 26 families, 68.0% on reCAPTCHA v2, with a median of 21.8 s and $2.40 per 100 attempts. The authors state that the solve rate drops from 15.4% to 49.6% under visual distraction, and that it assumes challenges can be cleanly cropped from the interface.

Baselines and Data

• The fast path uses the YOLO classifier released by Plesner et al. Evaluation uses the public reCAPTCHA dataset and samples collected in the experiments. Table I compares Weng, Hossen, Plesner, OEDIPUS, and Halligan; Hossen's 60.74% and Plesner's 75.69% macro accuracy were computed by the authors from the per-class success rates in the original figures.

The authors position this paper as a teacher–student hybrid solver without DOM access, and state that the gain in Table I relative to the same released YOLO comes from the hybrid design, not from a stronger detector.

The system is a hybrid solver: screenshots in, OS events out. Familiar and high-confidence cells go through YOLO; uncertain or unsupported classes go through Qwen-7B-VL; the VLM's hard labels are then written back to the detector via experience replay.

Interface localization and control

• YOLOv8 is fine-tuned from OmniParser's icon detector to detect the captcha area, cell, robot checkbox, reload button, and submit button. Input is 512×512; the optimizer is AdamW, with multi-scale training and early stopping.
• Training images are synthetic: 220 CAPTCHA source images are overlaid onto web page backgrounds, yielding 1,200 images, to avoid accessing online services during development (Appendix B). Appendix C states training converges at epoch 148.
• A finite state machine covers initialization, analysis, solving, verification, and recovery, with transitions triggered by visual events. Static puzzles are solved once and then verified; dynamic puzzles refresh after a click and are repeatedly analyzed until no targets remain. Appendix Algorithm 1 gives the procedure.
• Detection boxes are mapped from screenshot pixels to screen coordinates, then mouse and keyboard events are sent. Figure 1 states that the browser is not programmatically accessed.

Classification, segmentation, and routing

• EasyOCR reads the target in the instruction and distinguishes 3×3 classification from 4×4 segmentation by cell count. For classification, the 9 cells are predicted independently; for segmentation, the 16 cells are assembled into one image, and a predicted region is selected if it overlaps the cell area by at least 1%.
• The VLM is Qwen-7B-VL fine-tuned for GUI interaction. For classification, a numbered prompt constrains it to return only one class index; for segmentation, it is required to output target boxes. Appendix H gives the full prompts.
• Supported classes go through YOLO first, and are accepted only if the top-1 softmax confidence is at least τ=0.70; otherwise it falls back to the VLM. Boat, Taxi, and Tractor are not in YOLO's supported set and are sent directly to the VLM. Figure 1 states: if the engine is YOLO and the target is unsupported, the puzzle is reloaded.
• The authors do not compare the two via top-k: YOLO has softmax, while the VLM only outputs a class index; when asked to output probabilities, the authors state the values are unnormalized, inconsistent, or hallucinated. Both sides are therefore evaluated only on a single top-1.

Teacher–student distillation

• For classification, VLM outputs are treated as hard labels and supervise YOLO with cross-entropy, rather than matching soft distributions. The authors state that on segmentation the 7B model as teacher is not precise enough and distillation would degrade, so teacher–student and adversarial experiments are done only for classification.
• Experience Replay combines N images of the new class with N samples of each old class. The first 5 backbone layers are frozen, training runs for 50 epochs with learning rate 5×10−4, with color jitter, rotation, scaling, horizontal flip, and random erasing.
• Each setting of N uses 10 seeds, with 5 held out for validation. The authors state that VLM mislabels are not filtered, because the autonomous pipeline has no ground truth. The main text maps N=3–5 to roughly one 3×3 puzzle and N=7–10 to roughly two.

Adversarial protocol and success criteria

• The defender applies untargeted PGD to the published YOLO: ε=4/255, step size α=1/255, 10 steps, random initialization. Gray-box is 12 rounds, one per month; both sides use label smoothing 0.1 in training.
• The two teachers are ground-truth labels and the 7B VLM that the paper states is ∼70% accurate on the attacked training images. The authors use ground truth to represent the upper bound an expensive model would approach; the experiments themselves did not run the GPT-5 or Claude examples mentioned in the text. Another factor is whether the defender can reproduce the batch order.
• Static classification is judged by the top-1 of the ground-truth dominant class, reporting both frequency-weighted overall accuracy and class-equal-weighted macro accuracy. The main segmentation metric is F1. An online session counts as passed from the checkbox to the green check; the authors state that puzzles are not skipped via reload before attempting.

Experimental Setup

• Models evaluated: the released YOLO classifier, the Qwen-7B-VL GUI agent, a YOLOv8 interface detector fine-tuned from OmniParser, and the YOLOv8n/m/x segmentation models. Hybrid threshold τ=0.70. GPT-4, Gemini, and GPT-4o in Table I are literature comparisons and were not run in this paper.
• Data: 15,172 images in total, 16 classes. The classification holdout set is the validation split plus additionally collected Boat, Stairs, Taxi, and Tractor; the appendix says it was filtered to 893 images. The majority class has 74 validation images; Boat 11, Mountain 2, Chimney 17. Segmentation has 204 puzzles and 3,264 cells. Online is 100 sessions and 206 puzzles on the demo page.
• Comparisons: YOLO-only and VLM-only on the same images. The authors state that the classifier was trained on the training split of a public dataset, so only holdout data unseen by both sides is evaluated.
• Metrics: classification reports overall and macro accuracy, plus per-class precision, recall, and F1. Segmentation is primarily F1, with accuracy also reported. Latency, adversarial accuracy, session pass rate, and first-attempt rate are also reported.
• Judging: static comparisons use dataset ground truth, not an LLM judge. Online, a session ends with a green check on the screenshot. The paper reports neither human agreement nor significance tests.
• Repetition: distillation uses N=10 seeds per setting, and the gray-box reports the mean of 12 rounds. For the main classification table and the online evaluation, the paper reports no additional repetition counts.

Main Results

The accuracies in Table II with clear column alignment are as follows. The 12 classes exclude Boat, Stairs, Taxi, and Tractor.

Swipe horizontally to read the table

Configuration	Overall (16 classes)	Macro (16 classes)	Overall (12 classes)
YOLO	0.7548	0.5999	0.8314
Qwen-7B-VL	0.8052	0.8101	0.7827
Hybrid	0.8544	0.8417	0.8498

• Relative to single models: the authors state that Hybrid is not an interpolation: relative to YOLO, overall is 10.0 percentage points higher and macro 24.2 points higher; relative to VLM, 4.9 and 3.2 points. The conclusion writes the latter as macro recall; the Table II column name is Macro Acc.. The abstract writes these two as 85.4% and 84.2%.
• 12-class subset: the authors state that the gain remains after removing the four additionally collected classes: overall 85.0% vs. 83.1% and 78.3%, macro 81.6% vs. 75.2% and 76.6%. The authors consider YOLO's 83.1% close to the 82.4% reported by Plesner et al., with the difference possibly from unreleased samples.
• Speed and weak classes: the main text writes an average of 71.9 ms/cell, a 3.1× reduction relative to VLM-only; the introduction writes 3×, Table II writes 71 ms, and VLM invocation is 30.0%. YOLO is 0% on Boat, Taxi, and Tractor; VLM's Crosswalk recall is 41.9% and Other is 21.6% (Table III). The authors interpret the latter as object-centric bias and semantic over-interpretation.

New-Class Distillation and Forgetting

• Setup: using VLM hard labels without manual correction, Boat, Taxi, and Tractor are each expanded into a 14th class, with Experience Replay, 10 seeds per setting (Section VI).
• New-class recall: Taxi is 96%±8% at N=10 and 98%±6% at N=23; Tractor is 88%±10% at N=15 (Table V). Boat is 48%±18% at N=6 in Figure 3; in Table V, 48±18 at N=5 and 28±18 at N=3.
• Forgetting: at N=5, old-class accuracy relative to the 83.1% baseline: Taxi 76.3%±1.4 (6.8% forgetting), Tractor 75.9%±1.2 (7.2%), Boat 65.9%±1.5 (17.2%) (Table VI). The authors state that within-class visual differences determine difficulty, and they retain Taxi's 5/28 mislabels.

Adversarial Recovery and Gray-Box Escalation

• Single-round attack: PGD-10, ε=4/255, α=1/255 drives YOLO from 83.1% to 0.0%. On the same validation images, VLM drops from 78.7% to 75.0%, which the authors state is a 3.7 percentage point drop. Table VII writes the baselines as 83.0% and 0.0%.
• Retraining: adv acc with VLM labels goes from 27.8% at 442 images to 65.0% at 10,790 images; with ground-truth labels to 71.0% (Table VII). The authors state that VLM labels are 6–10 percentage points lower, and clean acc falls to the ∼72–73% written in the paper. Stairs is not among the 12 attacked classes because its training data was not released.
• 12 rounds: mean pre-recovery: perfect-teacher coupling 28.4%, independent batch 52.9%; VLM-teacher coupling 58.2%, independent 58.9%. After retraining, 60.1%, 64.2%, 62.3%, 62.3% (Table VIII). The authors state that both noisy labels and hidden batch order can reduce transfer, and combining the two adds almost nothing.

Online reCAPTCHA Evaluation

• Protocol: Google demo page, regular Chrome, 1920×1080, straight-line mouse, no cookies, no VPN, τ=0.70. The authors state that they do not reload before an attempt to skip unsupported classes.
• Results: all 100 sessions passed. Of 206 puzzles, 140 static and 66 dynamic; 181 succeeded on the first attempt, which the authors give as 87.9%. 80 sessions required no reload, 18 one, and 2 required 3 and 4 respectively. Median 17.1 s/puzzle, 35.2 s/session.
• Authors' comparisons: VLM invocation rate 37.1%, ranging from 0% to 75% within a session (standard deviation 15.5%). The authors state that Halligan is 68% on reCAPTCHA v2 with a median of 21.8 s, and that 17.1 s is 21% faster. The authors report that Plesner et al. were blocked after 20 runs once the VPN was removed.

Other Ablations and Analyses

• Segmentation Hybrid: accuracy 88.6%, F1=0.850; YOLOv8x-mask 86.7%, F1=0.814; VLM F1=0.769 (Table IX).
• In segmentation, YOLO F1 for Crosswalk, Stairs, and Taxi is 0.000 each; VLM is 0.685, 0.752, 0.857 (Table X).
• Median latency: interface detection 1.1 ms; YOLO classification 4.4 ms per cell; VLM classification 225 ms; VLM segmentation 2251 ms (Table IV).
• Without label smoothing, 28% of predictions are exactly 1.0, and only 4% of attacked cells escalate; with smoothing 0.1, escalation is 29–46%, and the VLM teacher's pre-recovery goes from 58.2% to 70.2% (Section VII-B).
• Last-round gradient alignment relative to the surrogate: coupled perfect teacher 0.42, independent batch order 0.07, same-order noisy labels 0.02 (Section VII-B).
• Interface detector validation: Precision 0.999, Recall 1.000, [email protected] 0.995, [email protected]:0.95 0.990 (Appendix C).

Limitations and future directions noted by the authors

The paper does not specifically discuss limitations. There is no Limitations or Future Work subsection in the full text; the conclusion does not frame undone items as future directions.

Scope of experimental coverage

• The main classification results compare the published YOLO, Qwen-7B-VL, and Hybrid, 16 classes, τ=0.70 (Section V, Table II).
• Segmentation evaluation covers YOLOv8n/m/x boxes and masks, VLM, and Hybrid, totaling 204 puzzles and 3,264 cells (Appendix D).
• Distillation adds only Boat, Taxi, Tractor, with 10 seeds per setting. The authors state that on segmentation the VLM as teacher is not precise enough, so no segmentation distillation is done (Section VI).
• Adversarial evaluation uses untargeted PGD against YOLO and 12 rounds of gray-box; the teacher is either ground truth or that 7B VLM. The GPT-5 and Claude examples mentioned in the text were not run (Section VII).
• Online evaluation is 100 sessions and 206 puzzles on the demo page (Section VIII). The paper does not report the number of repetitions for the main classification results, nor significance tests.

The hybrid solver folds the reasoning of an open-weight VLM into YOLO to solve visual reCAPTCHA on interfaces using only screenshots, without DOM access.

• Problem: the agent still re-reasons every time on familiar puzzles; a specialized detector is fast but only recognizes training classes and does not improve with use.
• Method: fine-tune YOLOv8 to localize the interface. The released YOLO answers directly when top-1 confidence is at least 0.70; otherwise, or when the class is unsupported, Qwen-7B-VL is invoked. VLM hard labels are written into the detector via experience replay.
• Classification: on the 16-class holdout set, Hybrid's overall is 0.8544 and macro is 0.8417 (Table II). The main text states this is higher than either component alone, and that the VLM handles only 30.0% of images. YOLO is 0% on Boat, Taxi, and Tractor.
• Expansion and recovery: Taxi's mean recall at N=10 is 96%±8% (Table V). PGD-10, ε=4/255 drives YOLO from 83.1% to 0.0%; after retraining on 10,790 VLM labels, adv acc is 65.0% (Table VII). Across 12 rounds, the mean pre-recovery for the coupled perfect teacher is 28.4% and for the VLM teacher 58.2%; after retraining, the four cells in Table VIII are 60.1% to 64.2%.
• Online and authors' conclusions: all 100 sessions passed, the first-attempt rate on 206 puzzles is 87.9%, and the median is 17.1 s/puzzle (Section VIII). The authors state that monthly-regenerated perturbations also cannot stop the solver from recovering each round, and that the ∼70%-accurate teacher written in the paper is comparable to perfect ground truth on hardening, because noisy labels reduce gradient alignment with the surrogate model.