An Evaluate Automated Anomaly Detection Methods for Efficiently Analyses Large-Scale Workflow System Logs, Overcoming the Limitations of Manual Inspection and Basic Statistical Techniques
| International Journal of Electronics and Communication Engineering |
| © 2025 by SSRG - IJECE Journal |
| Volume 12 Issue 10 |
| Year of Publication : 2025 |
| Authors : Arun Kumar Bandlamudi, Sunitha Pachala |
10.14445/23488549/IJECE-V12I10P110
How to Cite?
Arun Kumar Bandlamudi, Sunitha Pachala, "An Evaluate Automated Anomaly Detection Methods for Efficiently Analyses Large-Scale Workflow System Logs, Overcoming the Limitations of Manual Inspection and Basic Statistical Techniques," SSRG International Journal of Electronics and Communication Engineering, vol. 12, no. 10, pp. 104-115, 2025. Crossref, https://doi.org/10.14445/23488549/IJECE-V12I10P110
Abstract:
Traditional manual inspection and simple statistical analysis of them, though, have proved unrealistic due to the growing complexity and size of workflow system logs in an enterprise and cloud environment, and thus anomaly detection has become complex and time-consuming. In the present paper, automated anomaly detection methods that will eliminate these limitations will be assessed. We analyze the machine learning-based, deep learning-based, and hybrid models regarding the detection of anomalies in large and heterogeneous log data. We evaluate the efficiency, scalability, and detection accuracy of all methods by carrying out experiments on real-life workflow logs of cloud systems. Our results illustrate that automated techniques, especially those taking advantage of unsupervised deep learning models such as auto-encoders and LSTM-based networks, are much more successful than manual and statistical approaches in identifying the small and potentially new anomalies. This experimental design gives a comparative framework, which may guide the adoption of scalable and intelligent anomaly detection systems in large workflow settings in the future.
Keywords:
Anomaly detection, Workflow logs, Machine Learning, Deep Learning, Autoencoders, Log Analysis, Scalability, System monitoring, LSTM, Unsupervised learning.
References:
[1] Łukasz Korzeniowski, and Krzysztof Goczyła, “Landscape of Automated Log Analysis: A Systematic Literature Review and Mapping Study,” IEEE Access, vol. 10, pp. 21892-21913, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[2] Junchen Ma et al., “Automatic Parsing and Utilization of System Log Features in Log Analysis: A Survey,” Applied Sciences, vol. 13, no. 8, pp. 1-21, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[3] Scott Lupton et al., “Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods,” IEEE Access, vol. 12, pp. 78193-78218, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[4] Shampa Banik et al., “Anomaly Detection Techniques in Smart Grid Systems: A Review,” 2023 IEEE World AI IoT Congress (AIIoT), Seattle, WA, USA, pp. 331-337, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[5] Hamzeh Alimohammadi, and Shengnan Nancy Chen, “Performance Evaluation of Outlier Detection Techniques in Production Timeseries: A Systematic Review and Meta-Analysis,” Expert Systems with Applications, vol. 191, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[6] Jiang Zhaoxue et al., “A Survey on Log Research of AIOPs: Methods and Trends,” Mobile Networks and Applications, vol. 26, pp. 2353-2364, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[7] Mériem Ghali et al., “Threats Modeling and Anomaly Detection in the Behaviour of a System - A Review of Some Approaches,” Transactions on Large-Scale Data- and Knowledge-Centered Systems LI, pp. 1-27, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[8] Oluwambo Tolulope Olowe et al., “Enhancing Cybersecurity Through Advanced Fraud and Anomaly Detection Techniques: A Systematic Review,” 2024 International Conference on Science, Engineering and Business for Driving Sustainable Development Goals (SEB4SDG), Omu-Aran, Nigeria, pp. 1-12, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[9] Vishwanath D. Chavan, and Pratibha S. Yalagi, “A Review of Machine Learning Tools and Techniques for Anomaly Detection,” ICT for Intelligent Systems, pp. 395-406, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[10] Yaa Takyiwaa Acquaah, and Roy Kaushik, “Normal-only Anomaly Detection in Environmental Sensors in CPS: A Comprehensive Review,” IEEE Access, vol. 12, pp. 191086-191107, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[11] Mamdouh Alenezi, and Mohammed Akour, “AI-Driven Innovations in Software Engineering: A Review of Current Practices and Future Directions,” Applied Sciences, vol. 15, no. 3, pp. 1-26, 2025.
[CrossRef] [Google Scholar] [Publisher Link]
[12] Nachaat Mohamed, “Artificial Intelligence and Machine Learning in Cybersecurity: A Deep Dive into State-of-the-art Techniques and Future Paradigms,” Knowledge and Information Systems, vol. 67, pp. 6969-7055, 2025.
[CrossRef] [Google Scholar] [Publisher Link]
[13] Munish Rathee, Boris Bačić, and Maryam Doborjeh, “Automated Road Defect and Anomaly Detection for Traffic Safety: A Systematic Review,” Sensors, vol. 23, no. 12, pp. 1-34, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[14] Christos Cholevas et al., “Anomaly Detection in Blockchain Networks using Unsupervised Learning: A Survey,” Algorithms, vol. 17, no. 5, pp. 1-41, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[15] Fatima Rashed Alzaabi, and Abid Mehmood, “A Review of Recent Advances, Challenges, and Opportunities in Malicious Insider Threat Detection using Machine Learning Methods,” IEEE Access, vol. 12, pp. 30907-30927, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[16] Jie Zhang et al., “When LLMs Meet Cybersecurity: A Systematic Literature Review,” Cybersecurity, vol. 8, pp. 1-41, 2025.
[CrossRef] [Google Scholar] [Publisher Link]
[17] Michał Bałdyga et al., “Anomaly Detection in Railway Sensor Data Environments: State-of-the-Art Methods and Empirical Performance Evaluation,” Sensors, vol. 24, no. 8, pp. 1-32, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[18] Metehan Gelgi et al., “Systematic Literature Review of IoT Botnet DDOS Attacks and Evaluation of Detection Techniques,” Sensors, vol. 24, no. 11, pp. 1-37, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[19] Lorenzo Diana, Pierpaolo Dini, and Davide Paolini, “Overview on Intrusion Detection Systems for Computers Networking Security,” Computers, vol. 14, no. 3, pp. 1-44, 2025.
[CrossRef] [Google Scholar] [Publisher Link]
[20] Kinzah Noor et al., “A Review of Machine Learning and Transfer Learning Strategies for Intrusion Detection Systems in 5G and Beyond,” Mathematics, vol. 13, no. 7, pp. 1-63, 2025.
[CrossRef] [Google Scholar] [Publisher Link]
[21] Farid Binbeshr et al., “The Rise of Cognitive SOCs: A Systematic Literature Review on AI Approaches,” IEEE Open Journal of the Computer Society, vol. 6, pp. 360-379, 2025.
[CrossRef] [Google Scholar] [Publisher Link]
[22] Mohamad Khayat et al., “Empowering Security Operation Center with Artificial Intelligence and Machine Learning – A Systematic Literature Review,” IEEE Access, vol. 13, pp. 19162-19197, 2025.
[CrossRef] [Google Scholar] [Publisher Link]
[23] Ching-Nam Hang et al., “Large Language Models Meet Next-Generation Networking Technologies: A Review,” Future Internet, vol. 16, no. 10, pp. 1-29, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[24] Nour Moustafa et al., “Explainable Intrusion Detection for Cyber Defences in the Internet of Things: Opportunities and Solutions,” IEEE Communications Surveys & Tutorials, vol. 25, no. 3, pp. 1775-1807, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[25] Mohammed Alshomrani et al., “Survey of Transformer-Based Malicious Software Detection Systems,” Electronics, vol. 13, no. 23, pp. 1-34, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[26] Huiyao Dong, and Igor Kotenko, “Cybersecurity in the AI Era: Analyzing the Impact of Machine Learning on Intrusion Detection,” Knowledge and Information Systems, vol. 67, pp. 3915-3966, 2025.
[CrossRef] [Google Scholar] [Publisher Link]
[27] Hany F. Atlam, “LLMs in Cyber Security: Bridging Practice and Education,” Big Data and Cognitive Computing, vol. 9, no. 7, pp. 1-53, 2025.
[CrossRef] [Google Scholar] [Publisher Link]