Malcolm v24.01 releases: powerful, easily deployable network traffic analysis tool

Malcolm is a powerful network traffic analysis tool suite designed with the following goals in mind:

• Easy to use – Malcolm accepts network traffic data in the form of full packet capture (PCAP) files and Zeek (formerly Bro) logs. These artifacts can be uploaded via a simple browser-based interface or captured live and forwarded to Malcolm using lightweight forwarders. In either case, the data is automatically normalized, enriched, and correlated for analysis.
• Powerful traffic analysis – Visibility into network communications is provided through two intuitive interfaces: Kibana, a flexible data visualization plugin with dozens of prebuilt dashboards providing an at-a-glance overview of network protocols; and Moloch, a powerful tool for finding and identifying the network sessions comprising suspected security incidents.
• Streamlined deployment – Malcolm operates as a cluster of Docker containers, isolated sandboxes which each serve a dedicated function of the system. This Docker-based deployment model, combined with a few simple scripts for setup and run-time management, makes Malcolm suitable to be deployed quickly across a variety of platforms and use cases, whether it be for long-term deployment on a Linux server in a security operations center (SOC) or for incident response on a Macbook for an individual engagement.
• Secure communications – All communications with Malcolm, both from the user interface and from remote log forwarders, are secured with industry-standard encryption protocols.
• Permissive license – Malcolm is comprised of several widely used open-source tools, making it an attractive alternative to security solutions requiring paid licenses.
• Expanding control systems visibility – While Malcolm is great for general-purpose network traffic analysis, its creators see a particular need in the community for tools providing insight into protocols used in industrial control systems (ICS) environments. Ongoing Malcolm development will aim to provide additional parsers for common ICS protocols.

Although all of the open-source tools which make up Malcolm are already available and in general use, it provides a framework of interconnectivity which makes it greater than the sum of its parts. And while there are many other network traffic analysis solutions out there, ranging from complete Linux distributions like Security Onion to licensed products like Splunk Enterprise Security, the creators of Malcolm feel its easy deployment and robust combination of tools fill a void in the network security space that will make network traffic analysis accessible to many in both the public and private sectors as well as individual enthusiasts.

In short, Malcolm provides an easily deployable network analysis tool suite for full packet capture artifacts (PCAP files) and Zeek logs. While Internet access is required to build it, it is not required at runtime.

Changelog v24.01

• Features and enhancements
  • new Malcolm instance landing page (#252)
  • file carve download with password-protected .zip file (#288)
  • new “all files exept common plain text files” option for Malcolm’s file carving to match Hedgehog capability (#290)
  • allow customizing indexes for logs written to OpenSearch/Elasticsearch (#313)
  • more consistently differentiate between uploaded and live-captured traffic (#321)
  • make download extracted file context item from Arkime smarter (#330)
  • improve netbox device type library import by using “official” import script (#384)
• Component version updates
  • Alpine Linux to v3.19 as the base for some Docker images
  • Fluent Bit to v2.2.2
  • Beats to v8.11.4
  • LogStash to v8.11.4
• Bug fixes
  • Suricata Alerts dashboard “Alerts – Tags” visualization is useless (#314)
  • third party logs are not parsed correctly from fluentbit -> fluentd aggregator -> Malcolm (#318)
  • update document lookup APIs to search either network or host data (#322)
  • suricata rule update is broken (#323)
  • time sync from hedgehog to Malcolm opensearch instance not working (#324)
  • fix issue specifying database mode via command-line
  • have pruning of OpenSearch indices (based on size) include “other” Malcolm indices as well (e.g., nginx logs, system resources, third-party logs, etc.)
• Configuration changes (in environment variables in ./config/)
  • added the following variables with relation to #313
    • added ARKIME_ROTATE_INDEX to arkime.env with default value of daily (see Arkime docs on rotateIndex)
    • added the following variables and defaults to opensearch.env:

      # OpenSearch index patterns and timestamp fields
      # Index pattern for network traffic logs written via Logstash (e.g., Zeek logs, Suricata alerts)
      MALCOLM_NETWORK_INDEX_PATTERN=arkime_sessions3-*
      # Default time field to use for network traffic logs in Logstash and Dashboards
      MALCOLM_NETWORK_INDEX_TIME_FIELD=firstPacket
      # Suffix used to create index to which network traffic logs are written (supports Ruby strftime strings in %{})
      MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%d}
      # Index pattern for other logs written via Logstash (e.g., nginx, beats, fluent-bit, etc.)
      MALCOLM_OTHER_INDEX_PATTERN=malcolm_beats_*
      # Default time field to use for other logs in Logstash and Dashboards
      MALCOLM_OTHER_INDEX_TIME_FIELD=@timestamp
      # Suffix used to create index to which other logs are written (supports Ruby strftime strings in %{})
      MALCOLM_OTHER_INDEX_SUFFIX=%{%y%m%d}
      # Index pattern used specifically by Arkime (will probably match MALCOLM_NETWORK_INDEX_PATTERN, should probably be arkime_sessions3-*)
      ARKIME_NETWORK_INDEX_PATTERN=arkime_sessions3-*
      # Default time field used by for sessions in Arkime viewer
      ARKIME_NETWORK_INDEX_TIME_FIELD=firstPacket
  • changed default for EXTRACTED_FILE_HTTP_SERVER_KEY to infected in zeek-secret.env
  • added EXTRACTED_FILE_HTTP_SERVER_ZIP with default value of false in zeek.env, see (#288)

Install && Use

Copyright 2019 Battelle Energy Alliance, LLC