frangelbarrera/Awesome-Hacking-with-AI

<p align="center"> <img src="assets/images/logo1.png" alt="Awesome Hacking with AI Logo" width="700"> </p>

---

<div align="center">

    

</div>

---

Awesome Hacking with AI

A curated, evidence-led guide to AI for security and security for AI systems. It brings together authorized security testing, application security, AI agents, model adaptation, datasets, benchmarks, MCP security, defensive automation, and research.

This repository is for security practitioners, AI engineers, researchers, educators, and authorized red teams. It is deliberately broad: the aim is to explain what a resource does, why it matters, what evidence supports it, and where it belongs in a responsible workflow.

Responsible Use

All practical security work must be explicitly authorized, legally compliant, and proportionate to the agreed scope. This repository supports research, education, hardening, validation, detection, remediation, disclosure, and authorized testing. It does not provide instructions for unauthorized access, evading protections, deploying malware, operating command-and-control, impersonating people, or targeting third-party systems.

If you cannot demonstrate authorization for a target, do not test it. Use a lab, a benchmark, a capture-the-flag environment, or an intentionally vulnerable training application instead.

For coordinated disclosure guidance, consult the OWASP Vulnerability Disclosure Cheat Sheet and CERT/CC.

Contents

• How to Navigate
• Foundations, Governance, and Threat Taxonomies
• Security for AI Systems
• LLM, RAG, and Agent Security
• MCP Security
• AI Red Teaming and Evaluation
• AI for Security
• Application Security and Vulnerability Research
• Authorized Security Agents
• Threat Intelligence and Security Operations
• Models, Data, and Adaptation
• Benchmarks, Cyber Ranges, and Evaluation
• Practice Arenas, CTFs, and Learning Labs
• MCP Servers for Security Workflows
• Research and Architecture Case Studies
• Community Voices and Continuing Coverage
• Learning Path

How to Navigate

The collection follows two complementary paths. AI for security concerns how AI can improve code review, vulnerability management, detection engineering, threat intelligence, incident response, and authorized assessments. Security for AI systems concerns models, data, RAG, agents, memory, tools, MCP servers, and their software supply chain.

| Reader goal | Start with | Then continue with | |---|---|---| | Secure an LLM, RAG, or agentic application | LLM, RAG, and Agent Security | MCP Security and AI Security Guide | | Evaluate security agents responsibly | Authorized Security Agents | Benchmarks and Agentic Security Guide | | Specialize a model for a security task | Models, Data, and Adaptation | Models, Data, and Evaluation Guide | | Build an approved MCP workflow | MCP Security | MCP server directory and MCP Security Guide | | Learn systematically | Learning Path | Foundations, extended guides, benchmarks, and labs | | Practice AI-security skills safely | Practice Arenas, CTFs, and Learning Labs | Benchmarks and the Agentic Security Guide |

Foundations, Governance, and Threat Taxonomies

Security work involving AI benefits from a shared vocabulary, a lifecycle view, and an evidence model. The following resources help teams reason about risk before selecting a tool or model.

| Resource | Type | Why it matters | |---|---|---| | OWASP GenAI LLM Top 10 | Guidance | Current OWASP guidance for LLM and generative AI application risks; the former site is maintained as a historical entry point.[1] | | OWASP AI Exchange | Guidance | Security and privacy reference for AI systems — threat models, controls, and testing guidance; maintained as the successor to the OWASP AI Security and Privacy Guide. | | NIST AI RMF | Framework | Voluntary framework organized around Govern, Map, Measure, and Manage.[2] | | NCSC Secure AI System Development | Lifecycle guidance | Secure design, development, deployment, and operation/maintenance for AI systems.[3] | | OWASP AI Security Verification Standard (AISVS) | Verification standard | Versioned, testable security requirements across data, models, infrastructure, identities, agents, MCP, adversarial robustness, and monitoring.[20] | | MITRE ATLAS | Threat knowledge base | Living knowledge base of threats to AI-enabled systems, with techniques, mitigations, and case studies.[4] | | NIST Adversarial ML Taxonomy | Taxonomy | Shared terminology for adversarial ML threats and mitigations. | | AI Incident Database | Incident resource | Documented AI incidents to inform risk discovery and governance exercises. | | AVID Taxonomy | Risk taxonomy | Structured vocabulary for AI risk identification and documentation. | | MLSecOps Top 10 | Practice guide | Security perspective on the ML lifecycle and operational controls. | | Orca AI Incident Archive | Incident resource | Open database of AI agent security incidents (prompt injection, agent supply chain, MCP, rogue agent actions); each record links at least one primary source and labels confirmed harm and AI involvement. Classifications are the editors', not official findings. |

Security for AI Systems

LLM, RAG, and Agent Security

An AI application is more than a model. Prompts, retrieval sources, vector stores, identities, tool definitions, code execution paths, logs, memory, deployment infrastructure, providers, and users are all part of the security boundary. Secure design should consider the whole system.[2] [3]

| Resource | Category | Description | |---|---|---| | OWASP GenAI Security Project | Community guidance | Active home for LLM, agentic AI, red teaming, incident response, and data-security resources. | | Promptfoo | Testing framework | Declarative evaluation, red teaming, and regression testing for LLM applications, agents, and RAG systems. | | Garak | LLM probing | Open-source LLM vulnerability scanner for systematic model-behavior probing. | | PyRIT | AI risk assessment | Microsoft framework for identifying and mitigating generative-AI risks. | | Purple Llama | Safety resources | Meta tools and research artifacts for trust, safety, and LLM security evaluation. | | FuzzyAI | Fuzzing | Automated LLM fuzzing for resilience and jailbreak testing. | | Open Prompt Injection | Benchmark | Benchmark resources for studying prompt injection and defenses. | | Agentic Radar | Agent security | CLI scanner for issues in agentic workflows. | | NVIDIA SkillSpector | Skill and extension security | Scanner for agent skills and related supply-chain, permission, prompt-injection, and data-flow risks. Treat optional model-assisted analysis as an explicit data-handling decision.[21] | | Skill Scanner | Skills security | Cisco AI Defense scanner for agent-skill packages covering supply-chain and injection risks before installation; complements SkillSpector in pre-install review. | | Agent Threat Rules | Detection-rule format | Machine-readable agent-threat detection rules and reference implementations. The upstream format is a working draft; assess maturity and test coverage before enforcement.[22] | | OWASP Agent Memory Guard | Memory security | OWASP project for detecting and preventing agent memory poisoning. | | Semgrep AI Best Practices | Secure development | Security rules for code integrating LLM providers, MCP, assistants, and agent frameworks. | | Giskard | Evaluation | Open-source evaluation and testing for AI and LLM systems. | | | Red teaming | LLM red-team testing for engineering workflows. |

MCP Security

The Model Context Protocol (MCP) standardizes how hosts, clients, and servers expose resources, prompts, and tools to AI systems. That composability is powerful, but tools are security boundaries: they can access data or cause actions, and untrusted tool descriptions must be handled with caution.[5]

| Resource | Type | Description | |---|---|---| | MCP Specification | Protocol | Canonical concepts and trust-and-safety principles for resources, prompts, tools, authorization, and user interaction. Pin review to the deployed specification version.[5] | | MCP Security Best Practices | Technical guidance | Covers authorization, consent, token passthrough, SSRF, state handling, and local-server compromise.[6] | | MCP Security Bench (MSB) | Research benchmark | End-to-end evaluation of MCP-specific attacks across planning, invocation, and response handling. Use only in controlled evaluation and report both security and task-performance conditions.[23] | | OWASP MCP Top 10 | Threat model | OWASP project in beta covering token exposure, scope creep, tool poisoning, supply chain