Asian Journal of Probability and Statistics

Detecting fraud in financial systems is challenging due to highly imbalanced datasets, where fraudulent transactions make up less than 1%. Misclassifying these rare cases can be costly. This study examines how activation function choice affects the performance of artificial neural networks (ANNs) in fraud detection using the Financial Fraud Alert Review (FiFAR) dataset of one million transactions. A consistent ANN with two hidden layers was tested across seven activation functions: Sigmoid, Tanh, ReLU, Leaky ReLU, ELU, GELU, and Swish. Model performance was evaluated using accuracy, recall, F1 score, ROC-AUC, PR-AUC, and false positive rate (FPR). The Sigmoid function achieved the highest recall (0.6976), ROC-AUC (0.8182), and PR-AUC (0.0871), indicating superior fraud detection, though it had a higher FPR (0.2110) and low precision (0.0356). Compared to Logistic Regression, which had higher accuracy (0.1188) but low recall (0.1799), and Support Vector Machines (recall = 0.5238; FPR = 0.16395), the Sigmoid-based ANN performed better at identifying fraudulent cases. The results show that ANNs with suitable activation functions can capture complex patterns in imbalanced datasets more effectively than traditional models. In conclusion, selecting an appropriate activation function is crucial in fraud detection. Sigmoid offers optimal recall, while Tanh, Swish, and GELU provide better trade-offs between recall and false positives. The study offers practical guidance for optimizing neural network designs in fraud detection systems.