Evaluating social engineering risk in AI interaction using biometric features and game experiments

The researchers release AIriskEval-gaming, an open platform and dataset for evaluating social engineering risk in LLM-mediated multimodal interaction through controllable games, supporting three settings: human-human, human-AI, and AI-AI, and integrating configurable game templates, role-conditioned LLM agents, psychological participant profiles, structured interaction trees, as well as synchronized collection, filtering, and deep-learning-based feature extraction of behavioral and biometric features. The dataset AIriskEval-gaming-db comes from interactions between 15 participants and role-conditioned GPT-5.4 agents in two connected games, an adapted Prisoner's Dilemma and the Ultimatum Game, covering six streams of raw and processed data totaling 340GB—interaction logs, video, screen recordings, gaze logs, smartwatch signals, and game/questionnaire metadata—and is accompanied by descriptive analyses of interaction paths, written justifications, psychological profiles, subjective feedback, perception of the opponent's identity, game outcomes, and derived behavioral, facial, and gaze features. The dataset and related code are open on GitHub.