Pentesting and AI: secure code

AI coding tools and generative AI have rapidly become part of everyday work within organisations. They accelerate development and lower the barrier to building software, but they also introduce risks that traditional security controls do not automatically address. The question is not whether your organisation uses AI, but whether you understand how it changes your attack surface.

Explanation

Generative AI can support penetration testers with code review, documentation, hypothesis development and the structuring of test results. A model is not deterministic, however, and may produce incorrect code, invented dependencies or incomplete security assumptions. The risk profile is also shaped by the selected service, prompt processing, retention, model-training terms, connected data sources and the privileges granted to agents. In a penetration-testing context, prompts may contain source code, architecture details, test accounts or vulnerability information. AI use is therefore not a standalone productivity choice; it combines information security, software assurance and supplier risk. Human validation remains necessary to determine whether a generated finding is technically reproducible and relevant within the agreed scope.

Risk

The principal risks are unintended disclosure of sensitive information, prompt injection, insecure generated code, vulnerable model or plug-in dependencies and excessive agent permissions. A model can produce convincing but incorrect exploit logic or omit an existing security control. If output is copied directly into tools, scripts or reports, false positives, missed attack paths or new vulnerabilities may result. Connected agents increase potential impact when they can independently access files, repositories or external services. Client information may also leave the agreed processing chain. Removing personal data alone does not eliminate this risk; technical details, tokens, configurations and distinctive error messages can also be confidential or attributable to a specific environment.

Points to check

• AI-generated code reaching production without a security review.
• References to non-existent or compromised dependencies in the software supply chain.
• Prompt injection and insufficient isolation of AI functionality in applications.
• Shadow AI, where employees enter business data into public AI tools.
• Sensitive information and secrets leaking to external models.

Approach

DeepBlue applies a controlled-use model centred on data classification, purpose limitation and technical validation. Approved models, storage locations, retention, training terms and integrations are assessed before use. Sensitive input is minimised or pseudonymised, and agents receive only the permissions required for the task. Generated code and testing hypotheses are reproduced in an isolated environment and reviewed through manual analysis, static and dynamic testing and, where relevant, dependency scanning. A finding is reported only after its cause, impact and reproducibility have been established. Logging and periodic evaluation show where AI adds value and where human control remains necessary. This allows AI to accelerate parts of the engagement without replacing professional accountability or the rules of engagement.

Conclusion

AI delivers speed and convenience, but moves risk into areas that traditional controls may miss. Organisations that review AI-generated code, test AI applications and govern their use within clear boundaries can benefit from AI without weakening resilience. Testing AI should therefore become a standard part of the security strategy.

Contact

Want to understand the impact in your own environment? Contact DeepBlue Security & Intelligence at info@deepbluesecurity.nl or +31 (0) 70 290 6 290.