Hybrid Deep Learning for AML: Unsupervised Detection of Emerging Schemes via Feature Fusion and Explainable Artificial Intelligence

From rules to signals: hybrid deep learning for practical AML detection

Traditional rule-based anti-money laundering (AML) systems remain the default in many institutions, but they generate overwhelming volumes of false-positive alerts and struggle with complex, multi-step laundering schemes. A recent study on 54,258 SWIFT cross-border transactions from an East African commercial bank presents a practical, high-performing hybrid solution that fuses rule-based heuristics with deep learning embeddings – variational autoencoders (VAEs) for behavioural signals and graph neural networks (GNNs) for relational network structure – and uses an optimised One-Class Support Vector Machine (OCSVM) for semi-supervised anomaly detection. The result is a system that reduces compliance noise, surfaces novel suspicious behaviour, and remains explainable for audit and regulatory review.

Why hybrid fusion matters – the core idea

Pure rule-based systems are transparent but rigid, and pure deep learning models can be powerful but opaque. The study combines the strengths of both: seven carefully engineered heuristic flags provide regulatory grounding; an 8-dimensional VAE latent space compresses non-linear behavioural patterns (for example, sudden changes in a customer’s transaction variance); and GNN embeddings capture network topologies such as layering, structuring, and unusually high distribution of funds to multiple beneficiaries. After feature selection and mutual-information filtering, a concise nine-feature fused representation is used to characterise each transaction. The detection model is trained only on non-flagged transactions – learning the manifold of legitimate behaviour – and then used to detect novelty in the operational stream.

Operational performance you can act on

Benchmarked against Local Outlier Factor and Isolation Forest , the OCSVM trained on fused features achieved the best operational metrics that matter for compliance triage. The model reached a precision of 99.63% in the top 5% of prioritised alerts, meaning almost every alert in that high-priority queue was a true anomaly according to the study’s proxy labels. Average precision (AP) and AUC were also strong for the tuned OCSVM. The system processed 300,000 transactions in under five minutes on standard hardware (Intel Core i7, 16 GB RAM), equivalent to roughly 1,000 transactions per second, demonstrating that the inclusion of deep learning-derived features need not create operational bottlenecks in production environments.

Bastian Schwind-Wagner _Follow

"The hybrid AML framework combines rule-based heuristics with VAE and GNN embeddings to reduce false positives while uncovering novel suspicious behaviours that static rules miss. Its semi-supervised One-Class SVM trains on non-flagged data to learn legitimate transaction patterns, improving prioritisation for compliance teams.

Operational validation shows the system can process high transaction volumes on standard hardware and offers SHAP-based explanations for auditability. Institutions should pair the model with human-in-the-loop review and continuous feedback to confirm findings and adapt to evolving laundering techniques."

Detecting what rules miss – discovery of novel typologies

Because the model is trained on “normal” (non-flagged) data, it does not merely replicate the rule logic. The study reports that the hybrid OCSVM rejected 1,275 transactions that rules had flagged as suspicious, effectively reducing false-positive noise, and discovered 536 novel anomalies that the rules did not catch. These findings indicate the model can both filter spurious rule alerts that are consistent with a customer’s latent behaviour and surface emerging risk patterns orthogonal to static heuristics, such as subtle structuring or complex multi-hop fund flows.

Explainability and feature contributions

Explainability was handled through SHAP (Shapley additive explanations) . The analysis showed that the final detection decision draws on a balanced mix of signals: log-transformed transaction amount and deviation from a customer’s average transaction amount remain strong global predictors, but certain VAE and GNN embeddings provide indispensable context for specific typologies. For example, particular GNN embeddings correlate highly with transaction counts and volume variability – helpful for spotting layering or accounts suddenly distributing funds to many beneficiaries – while VAE embeddings capture latent customer volatility not captured by fixed thresholds. This explainability is essential for audit trails and to meet regulatory expectations around transparency and contested decisions.

Why One-Class SVM worked better here

In the semi-supervised setting used, where the model learns only from non-flagged transactions, the decision-boundary approach of OCSVM with an RBF kernel proved superior to partition- or density-based methods . The OCSVM effectively constructed a tight non-linear hypersphere around legitimate behaviour, yielding high precision on the top-ranked alerts and reducing false positives. Isolation Forest, which isolates anomalies via random partitioning, struggled to create an equally tight boundary in this fusion space. LOF attained a slightly higher ROC-AUC in some tests, reflecting its sensitivity to local density anomalies, but OCSVM produced superior operational triage metrics that matter most for compliance teams.

Practical deployment notes

The study validated the model in a real-world institutional setting. Its deployment used a simple concatenation fusion strategy , standardisation of inputs , and mutual-information-based feature filtering to prevent the curse of dimensionality . The system runs without specialised GPUs, integrates into a Django-based web interface for batch uploads and alert triage, and preserves an audit trail for compliance officers. The dataset and code supporting the paper are publicly available, enabling reproducibility and further experimentation.

Limitations and important caveats

A critical limitation is the absence of confirmed Suspicious Activity Reports (SARs) for ground-truth labels . The study used a rule-based proxy flag as a test target and mitigated circularity by training the detection model only on non-flagged transactions so the model learned normal behaviour rather than simply reproducing rules. While this reduces dependence on proxy labels and enabled discovery of novel anomalies, it does not substitute for full SAR-backed evaluation. Institutions considering adoption should plan for controlled human-in-the-loop validation and continuous feedback to refine thresholds and confirm typologies.

Regulatory alignment and next steps

This hybrid approach aligns with the FATF’s updated emphasis on advanced, agile monitoring for cross-border payments and the June 2025 update to Recommendation 16 . By combining rule-based explainability with deep-learning-derived behaviour and network features, the system addresses key FATF requirements – improved detection of complex typologies, operational speed, and interpretability. For financial institutions, the next steps are pragmatic: integrate the hybrid model into existing alert pipelines, adopt iterative validation with compliance investigators, and set up processes to retrain embeddings and recalibrate decision boundaries as customer behaviour and threat patterns evolve.

Bottom line

Fusing rule-based heuristics with VAE and GNN embeddings, and applying a semi-supervised, boundary-focused anomaly detector, can yield a detection pipeline that reduces false positives, surfaces novel threats, and keeps explanations auditable. For resource-constrained institutions aiming to comply with updated regulatory expectations while improving compliance efficiency, this hybrid architecture presents a pragmatic, deployable path forward.

Dive deeper

• Research ¦ Cosmas Ochieng Kungu, Kennedy Senagi, Evans Omondi, Hybrid deep learning for anti-money laundering: Unsupervised detection of emerging schemes via feature fusion and explainable artificial intelligence, Machine Learning with Applications, Volume 23, 2026, 100856, ISSN 2666-8270, https://doi.org/10.1016/j.mlwa.2026.100856. ¦ Link ¦ licensed under the following terms, with no changes made: CC BY-NC-ND 4.0