Malcolm v2.0.0-pre1 releases: powerful, easily deployable network traffic analysis tool
Malcolm is a powerful network traffic analysis tool suite designed with the following goals in mind:
- Easy to use – Malcolm accepts network traffic data in the form of full packet capture (PCAP) files and Zeek (formerly Bro) logs. These artifacts can be uploaded via a simple browser-based interface or captured live and forwarded to Malcolm using lightweight forwarders. In either case, the data is automatically normalized, enriched, and correlated for analysis.
- Powerful traffic analysis – Visibility into network communications is provided through two intuitive interfaces: Kibana, a flexible data visualization plugin with dozens of prebuilt dashboards providing an at-a-glance overview of network protocols; and Moloch, a powerful tool for finding and identifying the network sessions comprising suspected security incidents.
- Streamlined deployment – Malcolm operates as a cluster of Docker containers, isolated sandboxes which each serve a dedicated function of the system. This Docker-based deployment model, combined with a few simple scripts for setup and run-time management, makes Malcolm suitable to be deployed quickly across a variety of platforms and use cases, whether it be for long-term deployment on a Linux server in a security operations center (SOC) or for incident response on a Macbook for an individual engagement.
- Secure communications – All communications with Malcolm, both from the user interface and from remote log forwarders, are secured with industry-standard encryption protocols.
- Permissive license – Malcolm is comprised of several widely used open-source tools, making it an attractive alternative to security solutions requiring paid licenses.
- Expanding control systems visibility – While Malcolm is great for general-purpose network traffic analysis, its creators see a particular need in the community for tools providing insight into protocols used in industrial control systems (ICS) environments. Ongoing Malcolm development will aim to provide additional parsers for common ICS protocols.
Although all of the open-source tools which make up Malcolm are already available and in general use, it provides a framework of interconnectivity which makes it greater than the sum of its parts. And while there are many other network traffic analysis solutions out there, ranging from complete Linux distributions like Security Onion to licensed products like Splunk Enterprise Security, the creators of Malcolm feel its easy deployment and robust combination of tools fill a void in the network security space that will make network traffic analysis accessible to many in both the public and private sectors as well as individual enthusiasts.
In short, Malcolm provides an easily deployable network analysis tool suite for full packet capture artifacts (PCAP files) and Zeek logs. While Internet access is required to build it, it is not required at runtime.
Network map baseline comparison (issue #2)
Improving on Moloch’s connections view, you can now compare current logical network topology against a
previous time frame, using any of Malcolm’s 900+ fields as references for the graph’s source and destination nodes.
Network changes are easily visualized with icons for new ( ✨ ) and removed (🚫 ) nodes. The graph of connections can be switched on the fly between all nodes, actual nodes (i.e., nodes in the specified query time frame), baseline nodes (i.e., nodes in the specified and baseline query time frames), new nodes only and baseline nodes only.
This feature makes it easy to answer questions like:
• “Are there any hosts in my network this week that didn’t exist last month?”
• “Are hosts in my OT network making any new DNS queries compared to last quarter?”
• “Does my network contain any hardware from new vendors not accounted for the last time inventory was
This connections report can be accessed visually in the web browser (see screenshot) or programatically via REST API.
Security overview dashboards
Two new “security overview” Kibana dashboards have been created to bring potential network security issues to
the forefront for IT and OT networks:
• Security Overview (issue #108)
◦ Zeek notices by category
◦ AV signatures triggered by files carved from network traffic
◦ Clear-text transmission of passwords
◦ Outdated/insecure application protocols (e.g., TLSv1.0, SMBv1)
◦ Inbound external traffic by country (i.e., traffic where the source is a publicly routable IP and the
destination is an internal/private IP)
◦ Outbound internal traffic by country (i.e., traffic where the source is an internal/private IP and the
destination is a publicly routable IP)
◦ Summary of file types observed in file downloads/transfers
◦ External remote access over time (i.e., use of “remote access” protocols such as SSH, RDP, VNC, etc.
where either end of the connection is a publicly routable IP address)
◦ DNS queries by randomness (for identifying domain generation algorithms (DGA) used by some
• ICS/IoT Security Overview (issue #109)
◦ Log count by ICS/IoT protocol
◦ Traffic over time by ICS/IoT protocol
◦ ICS/IoT external traffic (i.e., any use of ICS/IoT protocols where either end of the connection is a publicly
routable IP address)
◦ ICS/IoT action summary
◦ Non-ICS/IoT protocols observed (for identifying IT protocols in OT networks)
◦ Source and destination IP summaries for ICS/IoT traffic
◦ File types by transport
Character frequency/entropy analysis (issue #107)
Malcolm can now optionally employ character frequency analysis to detect domain generation algorithm (DGA) hostnames often used by malware. Currently Malcolm employs this technique on DNS queries and SSL certificate servers. This makes it easier to find suspect domains (e.g., fqoxibdvbycnsappxc.nu) vs. common ones (e.g., example.org).
User interface for defining host and subnet name assignment
Track user access to Malcolm web interfaces
All access to Malcolm’s web interfaces (e.g., Moloch, Kibana, PCAP upload, etc.) requires authentication by a valid account. These accesses to Malcolm’s own interfaces can now be logged and viewed in Kibana dashboards built for that purpose.
ISO (live USB and installed) improvements
Both Malcolm and Hedgehog Linux can be installed using a standard ISO file image on systems supporting UEFI
boot. Hedgehog Linux can also be run in live USB mode, effectively turning any commodity hardware into an ad-
hoc network sensor. Improvements have been made to the base OS, including:
• improved hardening for both Malcolm and Hedgehog Linux
• installations should now detect virtual environments (VMWare and VirtualBox) and install the correct guest mode drivers for changing video resolution on the fly, shared folders, etc.
• many more minor fixes and improvements
Component version updates
Updated the following components to their latest stable released versions for security updates, bug fixes,
performance improvements and new features
• Elastic stack (Elasticsearch, Kibana, Logstash and Beats) 7.6.1
• Moloch 2.2.3
• Zeek 3.0.3
Miscellaneous fixes and improvements
• Fixed cross-platform compatibility of control scripts (#103)
• Fixed offline region maps (#112 and #84)
• Fixed intermittent failure when uploading very large PCAP files (#101)
• Fixed /upload URL incorrect redirect without trailing slash (#104)
• Fixed MANAGE_PCAP_FILES not working (#114)
• and more
Copyright 2019 Battelle Energy Alliance, LLC