AgentSafeLabs evaluates security differences across seven agent frameworks with 9,360 trials(original, opens in a new tab)
AgentSafeLabs released AgentPort-Bench, which, after verifying attack payloads byte-for-byte consistency, conducted 9,360 controlled trials on a direct API baseline and seven agent frameworks (including LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI Agents SDK, Google ADK, Semantic Kernel), covering six models and five attack categories mapped to the OWASP Agentic Security Initiative threat taxonomy. Results show that attack category had the greatest impact on outcomes, model choice was second, and framework choice had an impact roughly two orders of magnitude smaller than attack category and one order of magnitude smaller than model choice; all 28 pairwise comparisons among the eight conditions were judged equivalent under the preset equivalence bounds. The evaluation also recorded one model exhausting its reasoning token budget and returning empty responses in all six trials under a specific prompt, and that Google ADK and Semantic Kernel use different accounting methods for reasoning tokens.
