Journal Article
Machine Learning

Machine Learning Security: Threats, Countermeasures, and Evaluations

Mingfu Xue(Nanjing University of Aeronautics and Astronautics), Chengxiang Yuan(Nanjing University of Aeronautics and Astronautics), Heyi Wu, Yushu Zhang(Nanjing University of Aeronautics and Astronautics), Weiqiang Liu(Nanjing University of Aeronautics and Astronautics)
January 1, 2020IEEE Access191 citations
PDFDOI

191

Citations

7

Influential Citations

IEEE Access

Venue

2020

Year

Abstract

Machine learning has been pervasively used in a wide range of applications due to its technical breakthroughs in recent years. It has demonstrated significant success in dealing with various complex problems, and shows capabilities close to humans or even beyond humans. However, recent studies show that machine learning models are vulnerable to various attacks, which will compromise the security of the models themselves and the application systems. Moreover, such attacks are stealthy due to the unexplained nature of the deep learning models. In this survey, we systematically analyze the security issues of machine learning, focusing on existing attacks on machine learning systems, corresponding defenses or secure learning techniques, and security evaluation methods. Instead of focusing on one stage or one type of attack, this paper covers all the aspects of machine learning security from the training phase to the test phase. First, the machine learning model in the presence of adversaries is presented, and the reasons why machine learning can be attacked are analyzed. Then, the machine learning security-related issues are classified into five categories: training set poisoning; backdoors in the training set; adversarial example attacks; model theft; recovery of sensitive training data. The threat models, attack approaches, and defense techniques are analyzed systematically. To demonstrate that these threats are real concerns in the physical world, we also reviewed the attacks in real-world conditions. Several suggestions on security evaluations of machine learning systems are also provided. Last, future directions for machine learning security are also presented.

Analysis

Why This Paper Matters

Machine learning models are increasingly deployed in critical applications, yet their security vulnerabilities remain poorly understood by many practitioners. This survey, published in IEEE Access with 191 citations, provides a comprehensive and structured overview of the threat landscape, covering the entire ML pipeline from training to inference. It is particularly valuable because it consolidates disparate attack types—poisoning, backdoors, adversarial examples, model theft, and data recovery—into a single framework, making it easier for researchers and engineers to identify relevant risks.

The paper also emphasizes that these threats are not just theoretical; it reviews real-world physical attacks, such as adversarial patches on stop signs, demonstrating that security concerns are practical. By including evaluation suggestions, it offers actionable guidance for assessing model robustness, which is often missing in more narrowly focused surveys.

Technical Contributions

The paper's main innovation is its systematic categorization and analysis:

  • Five-category taxonomy: Training set poisoning, backdoors in training set, adversarial example attacks, model theft, and recovery of sensitive training data.
  • Threat model analysis: For each category, the paper describes the adversary's goal, knowledge, and capability, providing a clear framework for understanding attack surfaces.
  • Defense techniques: Covers both proactive (e.g., robust training, differential privacy) and reactive (e.g., detection, sanitization) defenses.
  • Real-world attacks: Reviews physical-world adversarial examples (e.g., stop sign attacks) and model extraction via APIs.
  • Evaluation methods: Suggests metrics and methodologies for security evaluation, such as attack success rate and robustness curves.

Results

As a survey, the paper does not present new experimental results. However, it synthesizes findings from prior work, noting that:

  • Adversarial examples can achieve over 90% success rate in evading classifiers with small perturbations.
  • Backdoor attacks can achieve high accuracy on clean data while causing targeted misclassification on triggered inputs.
  • Model theft attacks can replicate model functionality with high fidelity using limited queries.
  • Defenses often trade off accuracy for robustness, and no universal defense exists.

Significance

This survey has become a widely cited reference (191 citations) for ML security research. Its taxonomy and systematic analysis have influenced subsequent work on adversarial robustness, privacy-preserving ML, and secure learning. For practitioners, it provides a checklist of threats to consider when deploying models, and for researchers, it highlights open challenges such as defending against multiple attack types simultaneously and developing provable robustness guarantees. The paper's emphasis on real-world attacks bridges the gap between academic research and practical deployment concerns.

machine-learningsecurityadversarial-attackspoisoningbackdoorsmodel-theftprivacy

Related Papers