Journal Article
Large Language Models

From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy

Maanak Gupta(Tennessee Technological University), Charankumar Akiri(Tennessee Technological University), Kshitiz Aryal(Tennessee Technological University), Eli Parker(Tennessee Technological University), Lopamudra Praharaj(Tennessee Technological University)
January 1, 2023IEEE Access717 citations
arXivPDFDOI

717

Citations

46

Influential Citations

IEEE Access

Venue

2023

Year

Abstract

Undoubtedly, the evolution of Generative AI (GenAI) models has been the highlight of digital transformation in the year 2022. As the different GenAI models like ChatGPT and Google Bard continue to foster their complexity and capability, it’s critical to understand its consequences from a cybersecurity perspective. Several instances recently have demonstrated the use of GenAI tools in both the defensive and offensive side of cybersecurity, and focusing on the social, ethical and privacy implications this technology possesses. This research paper highlights the limitations, challenges, potential risks, and opportunities of GenAI in the domain of cybersecurity and privacy. The work presents the vulnerabilities of ChatGPT, which can be exploited by malicious users to exfiltrate malicious information bypassing the ethical constraints on the model. This paper demonstrates successful example attacks like Jailbreaks, reverse psychology, and prompt injection attacks on the ChatGPT. The paper also investigates how cyber offenders can use the GenAI tools in developing cyber attacks, and explore the scenarios where ChatGPT can be used by adversaries to create social engineering attacks, phishing attacks, automated hacking, attack payload generation, malware creation, and polymorphic malware. This paper then examines defense techniques and uses GenAI tools to improve security measures, including cyber defense automation, reporting, threat intelligence, secure code generation and detection, attack identification, developing ethical guidelines, incidence response plans, and malware detection. We will also discuss the social, legal, and ethical implications of ChatGPT. In conclusion, the paper highlights open challenges and future directions to make this GenAI secure, safe, trustworthy, and ethical as the community understands its cybersecurity impacts.

Analysis

Why This Paper Matters

This paper arrives at a critical juncture where generative AI models like ChatGPT are being rapidly adopted across industries, yet their security implications remain poorly understood. By systematically cataloging both offensive and defensive use cases, the authors provide a comprehensive overview that is valuable for practitioners, policymakers, and researchers. The demonstration of jailbreak and prompt injection attacks highlights the fragility of current safety guardrails, underscoring the urgency of developing more robust alignment techniques.

The paper's significance lies in its balanced perspective: it does not merely warn about risks but also explores how GenAI can strengthen cybersecurity defenses. This dual lens is essential for organizations seeking to leverage AI while mitigating threats. The inclusion of social, legal, and ethical dimensions further broadens the discussion beyond technical fixes, making it relevant for governance and compliance teams.

Technical Contributions

  • Attack Demonstrations: The paper shows concrete examples of jailbreak attacks (e.g., role-playing scenarios), reverse psychology prompts, and prompt injection that cause ChatGPT to output malicious code or phishing content.
  • Offensive Use Cases: It maps how adversaries can automate social engineering, generate polymorphic malware, and craft spear-phishing emails using GenAI, reducing the skill barrier for cybercrime.
  • Defensive Applications: The authors propose using GenAI for automated threat intelligence, secure code generation, attack pattern identification, and incident response plan creation.
  • Ethical Framework: The paper discusses the need for ethical guidelines and legal frameworks to govern GenAI use in cybersecurity.

Results

The paper does not present quantitative experimental results. Instead, it provides qualitative demonstrations of successful attacks (e.g., bypassing ChatGPT's content filters) and lists defensive scenarios. No accuracy, precision, or comparison metrics are reported. The main result is a taxonomy of attack and defense vectors, supported by illustrative examples.

Significance

This paper serves as an early comprehensive survey of GenAI's cybersecurity implications, influencing subsequent research on prompt injection defenses, adversarial robustness, and AI safety. Its impact is reflected in 717 citations, indicating widespread recognition among the AI security community. The work helps shape the conversation around responsible AI deployment and the need for proactive security measures in generative models.

generative-aicybersecurityprivacychatgptprompt-injectionjailbreak-attacksphishingmalware-generation
PromptingCode Generation

Related Papers