Penetration testing and AI: secure code and responsible use

AI coding tools and generative AI have rapidly become part of everyday work within organisations. They accelerate development and lower the barrier to building software, but they also introduce risks that traditional security controls do not automatically address. The question is not whether your organisation uses AI, but whether you understand how it changes your attack surface.

Explanation

AI affects organisations in two ways. Developers increasingly write code with assistants such as GitHub Copilot or a language model, and generated code sometimes moves directly into production. At the same time, employees use AI tools for writing, analysis and automation, often outside the visibility of IT. Both require attention: the code produced by AI and the way AI is used throughout the organisation.

Risk

AI-generated code is not secure by default. Models may reproduce unsafe patterns from their training data and introduce vulnerabilities such as injection, weak input validation or insecure default settings. Models can also reference software packages that do not exist, allowing attackers to register those names with malicious code. AI applications themselves introduce risks including prompt injection and the disclosure of source code or sensitive information to external models, matching categories described in the OWASP Top 10 for LLM Applications.

Points to check

• AI-generated code reaching production without a security review.
• References to non-existent or compromised dependencies in the software supply chain.
• Prompt injection and insufficient isolation of AI functionality in applications.
• Shadow AI, where employees enter business data into public AI tools.
• Sensitive information and secrets leaking to external models.

Approach

Treat AI as part of the attack surface and test it accordingly. Subject AI-generated code to the same code review, secure coding requirements and scans as manually written code. Test AI functions and agents specifically for prompt injection, unintended data disclosure and guardrail bypasses. Define clear policies for approved AI tools and permitted data, and make the secure option the easiest one to use. A penetration test that includes AI applications and their underlying integrations reveals where the actual risks are located.

Conclusion

AI delivers speed and convenience, but moves risk into areas that traditional controls may miss. Organisations that review AI-generated code, test AI applications and govern their use within clear boundaries can benefit from AI without weakening resilience. Testing AI should therefore become a standard part of the security strategy.

Contact

Want to understand the impact in your own environment? Contact DeepBlue Security & Intelligence at info@deepbluesecurity.nl or +31 (0) 70 290 6 290.