The AI Double-Edged Sword: How Generative AI Is Fueling a New Wave of Cyberattacks

The rapid rise of generative AI (GenAI) has revolutionized industries by enabling new forms of content creation, automation, and digital interaction. But as Palo Alto Networks’ Unit 42 warns in its latest analysis, these same tools are being exploited by cybercriminals to craft realistic phishing campaigns, deepfake content, and fraudulent websites at unprecedented speed.

According to the report, “because of its growing prevalence, GenAI also opens new vectors for threat actors to misuse. Adversaries are increasingly leveraging GenAI platforms to create realistic phishing content, clone trusted brands and automate large-scale deployment using services like low-code site builders. The threats are getting harder to detect.”

Unit 42 telemetry shows that adoption of GenAI platforms has more than doubled within six months. The high-tech sector accounts for over 70% of usage, followed by education, telecommunications, and legal services. Most activity centers on text-generation tools, chatbots, and media-generation platforms, with about 16% focused on data processing and workflow automation.

The report highlights that “this overall trend of increased traffic to AI websites indicates a growing adoption of GenAI applications and services,” underscoring how quickly AI capabilities are embedding themselves into business operations.

Unit 42 stresses that while GenAI provides enormous benefits, it has also amplified the scale and sophistication of phishing attacks. Attackers are increasingly misusing AI tools to craft highly convincing lures:

• Website generators (40%) are being exploited to create realistic phishing landing pages.
• Writing assistants (30%) are being misused to produce professional-looking phishing messages.
• Chatbots (11%) are being adapted into malicious conversational tools to trick victims.

“Attackers can use multi-media AI tools — including media generators and website builders — to rapidly create realistic-looking but fraudulent websites, deepfake content and deceptive phishing pages to mimic trusted brands,” the report states.

One of the most concerning findings is the misuse of AI-assisted website builders, which can generate entire websites in under a minute from a single text prompt.

Unit 42 researchers demonstrated how easily these platforms can be abused: “As a test, we used a known AI-assisted website builder to create a fake page to appear as if it’s for Palo Alto Networks. The website builder only required a valid email address… to establish a trial account and publish a page impersonating our company.”

Real-world cases have already been observed. In May 2025, Unit 42 detected two phishing landing pages generated with AI-powered site builders, linking victims to credential-harvesting sites. The report cautions that while today’s AI-generated phishing sites may seem rudimentary, “in the medium to long term, we expect that these attacks will become more convincing as AI-powered website builders grow more powerful.”

Beyond websites, AI writing assistant platforms are also being exploited as free hosting environments for phishing pages. These attacks often mimic legitimate business notifications, such as “You have new documents — click the button to view.” Victims who click are redirected to fake Microsoft login pages or other credential-stealing sites.

Unit 42 explains: “While attackers might leverage the AI functionality of these platforms in more powerful ways in the future, they are currently using these platforms primarily as a hosting service for malicious content.”

The key takeaway for defenders is that traditional detection strategies may no longer be sufficient. As Unit 42 emphasizes, “collectively, these risks underscore the potential of GenAI to amplify phishing campaigns and other social engineering threats. Therefore, stronger safeguards and threat detection are necessary.”

Related Posts:

• Publicly Exposed GenAI Development Services Raise Serious Security Concerns
• North Korean Operatives Use GenAI to Infiltrate Global Tech Jobs, Okta Warns
• Deepfakes and Deception: The Rise of Synthetic Identities in Remote Work
• Beyond Phishing: How AI and Deepfakes Are Powering a New Generation of Scams
• Deepfake Scams on the Rise: CEOs, News Anchors, and Government Officials Impersonated