AI Uncovers GhostPenguin: Undetectable Linux Backdoor Used RC5-Encrypted UDP for Covert C2
Ddos 
In a demonstration of AI-powered defense, security researchers at Trend Micro have uncovered a previously unknown and undetectable Linux backdoor dubbed “GhostPenguin.” The malware, which had zero detections on VirusTotal for over four months, was identified using an advanced automated threat-hunting pipeline designed to sift through millions of samples to find needles in the haystack.
GhostPenguin was first submitted to VirusTotal on July 7, 2025, but remained completely invisible to traditional antivirus engines. It wasn’t until Trend Micro’s AI-driven hunting system flagged the file for deep analysis that its malicious nature was revealed.
“GhostPenguin was discovered using Trend™ Research’s AI-driven, automated threat hunting pipeline that collected and analyzed zero-detection Linux samples from VirusTotal,” the report states. By leveraging AI to profile code structures and extract artifacts, researchers were able to identify the backdoor despite its lack of known signatures.
Written in C++, GhostPenguin is a sophisticated, multi-threaded backdoor designed to give attackers total control over compromised Linux systems. It eschews noisy TCP connections in favor of a custom, encrypted UDP protocol.
“GhostPenguin is a multi-threaded Linux backdoor… that provides remote shell access and comprehensive file system operations over an RC5-encrypted UDP channel”.
The malware’s architecture is highly modular, using separate threads to handle registration, heartbeats, and data transmission simultaneously. This design ensures that even if one component hangs, the backdoor remains responsive.
Once executed, GhostPenguin performs a rigorous setup routine to ensure it runs smoothly without alerting the user:
- Singleton Check: It checks for a PID file (e.g., <home>/.temp) to ensure no other instance is running. If it finds a stale lock file, it verifies if the process is actually active using kill(pid, 0) before proceeding.
- Session Handshake: The malware initiates a handshake with its Command & Control (C2) server by sending a request for a Session ID. This ID is then used as the key for RC5 encryption, securing all subsequent traffic.
- Registration: It profiles the victim machine, collecting data such as the LAN IP, hostname, OS version (e.g., “Ubuntu 24.04.2 LTS”), and architecture, which it sends to the C2 server every second until acknowledged.
Interestingly, analysis suggests that GhostPenguin may still be under active development. Researchers found “debug artifacts and unused functions” within the code, such as a leftover debug configuration variable (g_szConfigDebug) containing test domains like www.iytest.com.
Additionally, the malware contains several spelling errors in its internal strings, such as “ImpPresistence” (Persistence) and “Userame” (Username), pointing to a developer who may be rushing or lacking in QA processes.
The discovery of GhostPenguin highlights the critical need for next-generation detection capabilities. Traditional signatures failed to catch this threat, but AI-driven profiling succeeded.
“This case study exemplifies the increasing complexity of modern malware and the critical need for security researchers to continuously evolve their threat hunting strategies,” Trend Micro concludes.
Donate