Analysis of RSA algorithm cryptographic resilience using artificial intelligence methods

Abstract

This study investigates the cryptographic robustness of the RSA algorithm by applying machine learning techniques to assess its vulnerability, particularly in the context of modulus factorization, where n=p×q. The primary goal was to enhance the generation of random numbers within RSA systems to reduce the feasibility of factorization-based attacks. Four machine learning models were examined: Random Forest Classifier, Decision Tree, XGBoost, and a Sequential Model (neural network). These models were trained and evaluated using data relevant to RSA key generation and threat detection scenarios. A comparative performance analysis was conducted based on key classification metrics, including accuracy, precision, recall, F1-score, and ROC AUC. The Random Forest Classifier demonstrated superior overall performance, offering balanced detection across classes and high generalization capability. In contrast, the Sequential Model, despite high accuracy on paper, failed to identify minority class instances, limiting its reliability. The results suggest that integrating artificial intelligence, particularly ensemble learning methods, into cryptographic systems can improve the security of RSA against classical threats such as factorization. These findings highlight the potential of machine learning to support future developments in adaptive and intelligent cryptographic defense mechanisms.