pmacct v1.7.8 releases: passive network monitoring tools

pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect infrastructure data via Streaming Telemetry. Each component works both as a standalone daemon and as a thread of execution for correlation purposes (ie. enrich NetFlow with BGP data). pmacct main features are:

  • Suitable to ISP, IXP, CDN, IP carrier, Cloud, DC and hot-spots environments and SDN solutions
  • Runs on Linux, BSDs, Solaris and embedded systems
  • Support for both IPv4 and IPv6
  • Collects data through libpcap, Netlink/NFLOG, NetFlow v1/v5/v7/v8/v9, sFlow v2/v4/v5 and IPFIX
  • Collects Streaming Telemetry data
  • Supports Cisco NEL for CGNAT scenarios and Cisco NSEL
  • Saves data to a number of backends including:
    • Relational databases: MySQL, PostgreSQL and SQLite
    • noSQL databases: MongoDB and BerkeleyDB
    • AMQP message exchanges: RabbitMQ
    • Kafka message brokers
    • memory tables
    • flat files
  • Exports data to remote collectors through IPFIX, NetFlow v5/v9 and sFlow v5
  • Replicates incoming IPFIX, NetFlow and sFlow packets to remote collectors
  • Flexible architecture to tag, filter, redirect, aggregate and split captured data
  • Comes with:
    • a BGP daemon/thread for efficient visibility into the inter-domain routing plane. Read more here.
      • Supports BGP/MPLS VPNs rfc4364, Label Unicast rfc3107
      • Supports BGP ADD-PATHs (draft-IETF-IDR-add-paths) for visibility of BGP multi-path routes
      • Can log live BGP messaging and/or dump BGP tables per peer at a regular time interval
    • a BMP daemon/thread to gain insight in BGP data, events and statistics
      • Supports draft-IETF-grow-bmp-loc-rib and draft-IETF-grow-bmp-adj-rib-out (from 1.7.1)
    • an IS-IS/IGP daemon/thread for visibility of internal routes
  • Packet classification via nDPI (from 1.7.0)
  • Inspection of tunneled traffic (ie. GTP)
  • GeoIP lookups leveraging Maxmind library
  • Pluggable architecture for easy integration of new capturing environments and data backends
  • Careful SQL support: data pre-processing, triggers, dynamic table naming
  • It’s free, open-source, developed and supported with passion and an open mind for more than 10 years

Changelog v1.7.8

  • Introduced support for eBPF for all daemons: if SO_REUSEPORT is
    supported by the OS and eBPF support is compiled in, this allows
    to load a custom load-balancer. To load-share, daemons have to
    be part of the same cluster_name and each be configured with a
    distinct cluster_id.
  • Introduced support for listening on VRF interfaces on Linux for
    all daemons. The feature can be enabled via nfacctd_interface,
    bgp_daemon_interface and equivalent knobs. Many thanks to
    Marcel Menzel ( @WRMSRwasTaken ) for this contribution.
  • pre_tag_map: introduced limited tagging / labelling support for
    BGP (pmbgpd), BMP (pmbmpd), Streaming Telemetry (pmtelemetryd)
    daemons. ip, set_tag, set_label keys being currently supported.
  • pre_tag_map: defined a new pre_tag_label_encode_as_map config
    knob to encode the output ‘label’ value as a map for JSON and
    Apache Avro encodings, ie. in JSON “label”: { “key1”: “value1”,
    “key2”: “value2” }. For keys and values to be correctly mapped,
    the ‘%’ delimiter is used when composing a pre_tag_map, ie.
    “set_label=key1%value1,key2%value2 ip=0.0.0.0/0”. Thanks to
    Salvatore Cuzzilla ( @scuzzilla ) for this contribution.
  • pre_tag_map: introduced support for IP prefixes for src_net
    and dst_net keys for indexed maps (maps_index set to true).
    Indexing being an hash map, this feature currently tests data
    against all defined IP prefix lenghts in the map for a match
    (first defined matching prefix wins).
  • pre_tag_map: introduced two new ‘is_nsel’, ‘is_nel’ keys to
    check for the presence of firewallEvent field (233) and
    natEvent field (230) in NetFlow/IPFIX respectively in order
    to infer whether data is NSEL / NEL. If set to ‘true’ this
    does match NSEL / NEL data, if set to ‘false’ it does match
    non NSEL / NEL data respectively.
  • Introduced a new mpls_label_stack primitive, encoded as a
    string and includes a comma-separated list of integers (label
    values). Thanks to Salvatore Cuzzilla ( @scuzzilla ) for this
    contribution.
  • Introduced a new fw_event primitive, to support NetFlow v9/
    IPFIX firewallEvent 233 Information Element.
  • Introduced a new tunnel_tcp_flags primitive for pmacctd and
    sfacctd to record TCP flags for the inner layer of a tunneled
    technology (ie. VXLAN). Also tunnel_dst_port decoding was
    fixed for sfacctd.
  • Introduced support for in/out VLAN support for sfacctd. To be
    savy, ‘in_vlan’ and ‘vlan’ were muxed onto the same primitive
    depending on the daemon being used. Thanks to Jim Westfall
    @jwestfall69 ) for this contribution.
  • Introduced a new mpls_label_stack_encode_as_array config knob
    to encode the MPLS label stack as an array for JSON and Apache
    Avro encodings, ie. in JSON “mpls_label_stack”: [ “0-label0”,
    “1-label1”, “2-label2”, “3-label3”, “4-label4”, “5-label5” ]
    and in Avro “name”: “mpls_label_stack”, “type”: { “type”:
    “array”, “items”: { “type”: “string” } }. Thanks to Salvatore
    Cuzzilla ( @scuzzilla ) for this contribution.
  • Introduced a new tcpflags_encode_as_array config knob to encode
    TCP flags as an array for JSON and Apache Avro, ie. in JSON
    “tcp_flags”: [ “URG”, “ACK”, “PSH”, “RST”, “SYN”, “FIN” ] and
    in Avro “name”: “tcp_flags”, “type”: { “type”: “array”,
    “items”: { “type”: “string” } }. Thanks to Salvatore Cuzzilla
    @scuzzilla ) for this contribution.
  • Introduced a new fwd_status_encode_as_string config knob to
    encode the ‘fwd_status’ primitive in human-readable format
    like described by RFC-7270 Section 4.12 when JSON or Avro
    formats are selected for output. Thanks to Salvatore Cuzzilla
    @scuzzilla ) for this contribution.
  • Introduced a new protos_file to define a list of (known/
    interesting/meaningful) IP protocols. Both protocol names, ie.
    “tcp”, and protocol numbers, ie. 1 (for icmp), are accepted.
    IANA reserved protocol value 255 is used to bucket as ‘others’
    those IP protocols not matching the ones defined in the list.
  • Introduced a new tos_file to define a list of (meaningful) IP
    ToS values; if tos_encode_as_dscp is set to true then DSCP
    values are expected as part of the file. The directive uses
    value 255 to bucket as ‘others’ those ToS/DSCP values not
    matching the ones defined in the list.
  • A new tos_encode_as_dscp config knob makes pmacct to honour
    only the 6 bits used by DSCP and report only on those.
  • BGP, BMP, Streaming Telemetry daemons: introduced a new
    dump_time_slots config knob to spread the load deriving by
    dumps over the configured refresh time interval. The interval
    is divided into time slots and nodes are assigned to such
    slots. The slot for each node is determined using its IP
    address. Thanks to Raphael Barazzutti ( @rbarazzutti ) for
    this contribution.
  • BGP, BMP daemons: End-of-RIB messages are now being exposed
    in the output feed in order to facilitate tracking their
    arrival (or not!).
  • pmtelemetryd: aligned daemon to the latest Unyte UDP-Notif API
    (0.6.1) and related standardization draft-ietf-netconf-udp-notif
  • RPKI daemon: added case for input “asn” value being integer (ie.
    “asn” : 2914) on top of the string case (ie. “asn” : “AS2914”).
  • Kafka, amqp plugins: introduced a new writer_id_string config
    knob to allow to customize the the “writer_id” field value. A
    few variables are supported along with static text definitions.
  • Added a new aggregate_unknown_etype config knob to account also
    frames with EtherTypes for which there is no decoding support
    and allow to aggregate them by the available Ethernet L2 fields
    (ie. ‘src_mac’, ‘dst_mac’, ‘vlan’, ‘cos’, ‘etype’). Thanks to
    @singularsyntax for this contribution.
  • Added a new bgp_daemon_add_path_ignore config knob to ignore
    (do not advertise back) the ADD-PATH capability advertised by
    remote BGP peers.
  • nfacctd, sfacctd: extended the possibility to run daemons from
    a user with non root privileges to these daemons.
  • nfacctd: if Information Element 90 (MPLS VPN RD) is present in
    NetFlow v9/IPFIX, make it available for BGP/BMP correlation.
  • pmacctd, sfacctd: introduced basic support for QinQ, 802.1AD.
  • [print|kafka|amqp]_preprocess: added suppport for ‘maxp’,
    ‘maxb’ and ‘maxf’ keys when preprocessing aggregates of non-
    SQL plugins. Thanks to Andrew R. Lake ( @arlake228 ) for this
    contribution.
  • nDPI: newer versions of the library (ie. >= 4.0) bring changes
    to the API. pmacct is now aligned to compile against these. At
    the same time support for nDPI 3.x was dropped.
  • fix, plugin_common.[ch]: when stitching feature was enabled,
    ie. nfacctd_stitching, timestamp_min was never reset. Also both
    timestamp_min and timestamp_max were clamped to sec granularity.
  • fix, BGP, BMP daemons: added a tmp_bgp_daemon_origin_type_int to
    print out BGP “origin” field as int (legacy behaviour) instead
    of string (current behaviour). In a future major release the
    legacy behaviour will be dropped.
  • fix, BGP, BMP daemons: MPLS labels are now encoded in both JSON
    and Apache Avro as ‘mpls_label’ instead of ‘label’. This is to
    align behaviour with pre_tag_map where ‘label’ has a different
    semantic.
  • fix, BGP, BMP daemons: resolved memory leak when encoding log
    messaging (logmsg) in Avro format with Schema Registry support.
  • fix, BGP daemon: improved handling of ADD-PATH capability,
    making it per-AF (as it is supposed to be) and not global.
  • fix, BMP daemon: now checking that ADD-PATH capability is
    enabled at both ends of the monitored session (check both BGP
    OPEN in a Peer Up message) in order to infer that the capability
    exchange was successful. Also some heuristics were added to
    conciliate BGP Open vs BGP Update 4-bytes ASN reality.
  • fix, nfacctd: improved parsing of NetFlow v9 Options data
    particularly when multiple IEs are packed as part of a flowset.
  • fix, nfacctd: corrected parsing of Information Element 351
    (layer2SegmentId).
  • fix, pmacctd: improved processing of pcap_interfaces_map for
    cases where the same interface is present multiple times (maybe
    with different directions). Also, if the map is empty then bail
    out at startup.
  • fix, pmacctd: SEGV when ICMP/ICMPv6 traffic was processed and
    ‘flows’ primitive was enabled.
  • fix, pmacctd: sampling_rate primitive value was not reported
    correctly when ‘sampling_rate’ config directive was specified.
  • fix, pmbgpd, pmpmbd, pmtelemtryd: changed SIGCHLD handler to
    prevent zombification of last spawned data dump writer.
  • fix, Kafka plugin: moved the schema registration from the dump
    writer to the plugin process in order to register the schemas
    only once at plugin startup and not on every start of a writer
    process. Thanks to Uwe Storbeck ( @ustorbeck ) for this
    contribution.
  • fix, Kafka plugin: a check for kafka_partition was missing,
    leading the plugin to always use the default partitioner
    instead of sending data to the configured fixed partition.
    Thanks to Martin Pels ( @rodecker ) for this contribution.
  • fix, nfprobe plugin: BGP data enrichment was not working due to
    a mistakenly moved pointer.
  • fix, sfprobe plugin: AS-PATH was being populated even when null;
    added a check to see if the destination AS is not zero in order
    to put the destination AS into the AS-PATH for sFlow packets.
    Thanks to Marcel Menzel ( @WRMSRwasTaken ) for this contribution.
  • fix, networks_file: remove_dupes() was making partial commits
    of valid rows hence creating data inconsistencies.
  • fix, pre_tag_map: resolved a potential string overflow that was
    being triggered in pretag_append_label() when data would be
    assigned more than one single label. Also now allow ‘,’ chars
    in set_label.

Download && Tutorial

pmacct is Copyright (C) 2003-2018 by Paolo Lucente

Source: https://github.com/pmacct/pmacct