Authors - Marcos Paulo Jeronimo Francisco, Carlos Hideo Arima, Napoleao Verardi Galegale, Joshua Onome Imoniana Abstract - The increasing complexity of digital systems and the growing sophistication of cyber threats have intensified the need for proactive security assessment methods. Threat Modeling is a structured practice for identifying potential vulnerabilities, attack paths and mitigation strategies during the software development lifecycle. However, its manual application is often time-consuming, subjective and dependent on scarce cybersecurity expertise. In this context, Large Language Models (LLMs) may support security teams by generating threat hypotheses, classifying risks and recommending controls. This study evaluates the effectiveness of three LLM-based tools — ChatGPT, Gemini and Manus — in cybersecurity threat modeling for a real-world backend information system. A controlled computational experiment was conducted using standardized prompts applied to the three models, with three independent executions per prompt. The evaluation considered five dimensions: threat identification coverage, technical depth of analysis, quality of risk classification, assertiveness of control recommendations and result consistency. To consolidate the comparison, a Final Effectiveness Metric (FEM) was proposed. The results show different performance profiles among the evaluated models. Manus achieved the highest FEM score, with stronger threat coverage, technical depth, control recommendations and consistency. ChatGPT presented intermediate performance, with structured and detailed analyses, while Gemini showed lower threat coverage, but satisfactory technical reasoning in specific tasks. The findings also indicate that LLMs can enhance threat modeling activities by expanding analytical capacity and supporting DevSecOps practices. Nevertheless, their outputs require human validation, especially regarding risk classification, framework alignment and false positive analysis.
Open Zoom