Arkime v5.0 releases: open source, large scale, full packet capturing, indexing, and database system

Arkime is an open-source, large-scale, full packet capturing, indexing, and a database system. Arkime augments your current security infrastructure to store and index network traffic in standard PCAP format, providing fast, indexed access. An intuitive and simple web interface is provided for PCAP browsing, searching, and exporting. Arkime exposes APIs which allow for PCAP data and JSON formatted session data to be downloaded and consumed directly. Arkime stores and exports all packets in standard PCAP format allow you to also use your favorite PCAP ingesting tools, such as Wireshark, during your analysis workflow.

Access to Arkime is protected by using HTTPS with digest passwords or by using an authentication providing webserver proxy. All PCAPs are stored on the sensors and are only accessed using the Arkime interface or API. Arkime is not meant to replace an IDS but instead work alongside them to store and index all the network traffic in standard PCAP format, providing fast access. Arkime is built to be deployed across many systems and can scale to handle tens of gigabits/sec of traffic. PCAP retention is based on available sensor disk space. Metadata retention is based on the Elasticsearch cluster scale. Both can be increased at any time and are under your complete control.

Architecture

Here are some sample deployments of Arkime for different network architectures. Most folks will probably run a hybrid of the following since no one solution fits all. The ability to scale capturing can be done horizontally by adding more capture machines, vertically by adding more CPUs/disk, or both. We usually recommend scaling horizontally unless physically space-constrained and using a network packet broker in front of multiple machines. However it is possible to use big machines, with lots of cpu/disk, and run Arkime-capture with more threads.

Legend/Info:

• A box represents a physical machine.
• It is possible to run multiple capture processes per machine or have a single capture process to listen to multiple interfaces – (FAQ Answer)
• Recommend “Big Data” style boxes for capture – (FAQ Answer)
• Run multiple Elasticsearch processes per machine since each ES node should be configured at most to 30G – (FAQ Answer)
• Except for single-host deployments, it is recommended/useful that all operator access flows through a single Apache/viewer combination that can provide better authentication, logging, and a single choke point – (FAQ Answer)

Security

• All ES instances should have iptables for port 9200-920N and 9300-930N, where N is the number of ES instances per machine, and only allow the other elasticsearch, capture and viewer machines to connect
• All viewer hosts, except the apache/viewer box, should have iptables for port 8005 and only allow other viewer machines to connect. The viewer must listen on OS interface if using multiple machines
• The shared viewer instances can listen on localhost since only apache talks to it

Single Host

Multiple Hosts Monitoring Multiple Network Segments

Multiple Hosts Monitoring High Traffic Networks

Notes:

• Using a Network Packet Broker (NPB) allows traffic to be load balanced and recombined. This is especially useful in HA or asymmetric routing cases
• By using an NPB, other security devices can see the same traffic Arkime sees
• When running multiple Arkime-captures on the same host make sure the IO doesn’t overwhelm the disk and other subsystems.
• Use a TAP with high traffic networks since many mirror ports drop traffic under heavy load
• Operators use an apache fronted viewer and don’t hit the other viewers directly. The apache provides authentication.
• Lockdown ES and Arkime viewer with iptables

Multiple Clusters

Notes:

• It is possible to use a single ES cluster using the prefix= ini configuration
• Operator uses apache fronted viewers and doesn’t hit the other viewers directly. The apache provides authentication. Can use virtual paths to route to different clusters.
• NPBs are recommended for high traffic networks

Changelog v5.0

BREAKING

• #2297 s3Compression/simpleCompression now defaults to zstd
• #2297 s3WriteGzip removed, use s3Compression=gzip for gzip instead of new zstd default
• #2297 s3GapPacketPos defaults to TRUE
• #2297 enablePacketDedup defaults to TRUE
• #2299 #2308 authMode defaults to digest now
• #2312 removed old v1 viewer APIs
• #2349 parliament password removed, must configure common auth via the UI before upgrading or manually in the config file see parliament and how do I upgrade to 5
• #2402 WISE/tagger must now use http.request.FIELD/http.response.FIELD when referencing header defined with headers-http-request/headers-http-response
• #2450 Centos 7 build no longers includes pfring support
• #2453 Increase simpleCompressionBlockSize default to 64000

Release

• #2448 zstd 1.5.5, nghttp2 1.57.0, maxmind 1.7.1, yara 4.2.3
• #2443 Centos 7, Ubuntu 18, Alpine use unofficial builds of node
• #2543 node v18.19.0
• #2447 support building on alpine
• #2549 use configure prefix more places (thanks @vpiserchia)
• #2584 AL2023 & Ubuntu22.04 ARM builds

All

• #2316 programs support same config file formats (ini/json/yaml) and retrieval (file, elasticsearch)
• #2419 json/yaml config file formats now allow arrays instead of comma/semi separated
• #2299 #2308 authMode setting added
• #2299 #2408 #2463 added authMode: basic, form, basic+form, basic+oidc, headerOnly, header+digest (same as header), header+basic
• #2387 notifiers for parliament and arkime merged conflicts mitigated by appending “Parliament” to parliament notifiers
• #2396 drop privileges is now AFTER http(s) list
• #2509 add optional login message for form auth
• #2511 new authOIDCScope setting
• #2482 new logoutUrl setting
• #2571 new scheme pcap reading
• #2618 better error message when can’t use OpenSearch/Elasticsearch on startup
• More…

Download && Tutorial

Copyright 2012-2017 AOL Inc