Success Story
Client Introduction
A leading bank in the United States, which offers a range of services, including personal and business banking, commercial banking, wealth management, and more.
Problem Statement
The bank uses a combination of vendor and proprietary FINCRIME models for fraud detection and prevention, identity and document verification, Know Your Customer (KYC), transaction monitoring, money laundering, and other critical business needs.
The Model Risk Management (MRM) team at the bank had mandated rigorous testing and validation of these models to determine the underlying risks and vulnerabilities. Fulfilling this mandate was critical for internal policy compliance and business continuity.
The key challenges were:
- Lack of technical expertise to test and validate the functioning of models, which used convolutional neural networks (CNN), decision trees, XGboost, and other advanced techniques.
- The scope of model validation spanned investigating, reviewing, and reporting the findings for multiple complex models.
Solution Offered
The Anaptyss team collaborated with the bank’s Model Risk Management department to classify risks, set monitoring controls, and track performance.
The team conducted thorough due diligence and independent testing of the models using quantitative and qualitative techniques.
It delivered a comprehensive report with findings and actionable recommendations to ensure each model meets high standards for accuracy, robustness, and regulatory compliance.
Key Solution Delivered
Qualitative Validation
- Tested the models' ability to detect and confirm the authenticity of real living users using deep learning techniques such as Convolutional Neural Networks (CNN).
- Validated limitations of models, such as susceptibility to advanced spoofing, demographic bias, device constraints, environmental interference, and advanced fraud techniques.
- Tested the models' compliance with BSA for AML, Regulation (EU) 2016/679, eIDAS Regulations (EN319 401), ISO/IEC 30107-3, WCAG 2.1 AA, Section 508 of the US Rehabilitation Act, and the FIDO Face Verification Program.
Quantitative Validation
- Evaluated the models' decision-making process and robustness against fraud. The complexity of CNN-based feature extraction hindered the validation of features that drive outcomes, assess demographic fairness, and diagnose performance issues.
- Used grid search, random search, and Bayesian optimization to test and determine the optimal values for hyperparameter tuning and improve CNN’s performance on data validation.
Business Outcomes
- 100% quantification of model performance based on accuracy, precision, recall, F1 score, and LIME and Shapley values.
- Increased the robustness of models with EMMS-compliant Model Development Report for each FINCRIME model.
- Ensured the long-term effectiveness of the models by providing a customized performance monitoring plan comprising the required, ad-hoc, and recommended metrics with tolerance bands.
Want to learn more or need a solution?
Write to us: info@anaptyss.com