vector v0.34 releases: High-Performance, Logs, Metrics, & Events Router
Vector
Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space.
Use cases
- Reduce total observability costs.
- Transition vendors without disrupting workflows.
- Enhance data quality and improve insights.
- Consolidate agents and eliminate agent fatigue.
- Improve overall observability performance and reliability.
Performance
| Test | Vector | Filebeat | FluentBit | FluentD | Logstash | SplunkUF | SplunkHF |
|---|---|---|---|---|---|---|---|
| TCP to Blackhole | 86mib/s | n/a | 64.4mib/s | 27.7mib/s | 40.6mib/s | n/a | n/a |
| File to TCP | 76.7mib/s | 7.8mib/s | 35mib/s | 26.1mib/s | 3.1mib/s | 40.1mib/s | 39mib/s |
| Regex Parsing | 13.2mib/s | n/a | 20.5mib/s | 2.6mib/s | 4.6mib/s | n/a | 7.8mib/s |
| TCP to HTTP | 26.7mib/s | n/a | 19.6mib/s | <1mib/s | 2.7mib/s | n/a | n/a |
| TCP to TCP | 69.9mib/s | 5mib/s | 67.1mib/s | 3.9mib/s | 10mib/s | 70.4mib/s | 7.6mib/s |
Changelog v0.34
7 enhancements
- Sinks now have additional options for
encoding.timestamp_format:unix_float: Represents the timestamp as a Unix timestamp in floating point.unix_ms: Represents the timestamp as a Unix timestamp in milliseconds.unix_ns: Represents the timestamp as a Unix timestamp in nanoseconds.unix_us: Represents the timestamp as a Unix timestamp in microseconds
Thanks to srstrickland for contributing this change!
- The
prometheus_remote_writesink now has the ability to disable aggregation by settingbatch.aggregatetofalse. - The
natssource has a newsubscriber_capacityconfiguration option to control how many messages the NATS subscriber buffers before incoming messages are dropped. - Sources and sinks that run a HTTP server now emit additional internal metrics:
http_server_requests_received_totalhttp_server_responses_sent_totalhttp_server_handler_duration_seconds
- Sources that run a gRPC server now emit additional internal metrics:
grpc_server_messages_received_totalgrpc_server_messages_sent_total- `grpc_server_handler_duration_seconds
- Event metadata, including secrets like Datadog API key or Splunk HEC token, are now persisted when writing events to a disk buffer so that sinks have access to it.
As part of this change, disk buffers created by Vector now have more restrictive file permissions on Unix platforms. Previously, they were world-readable, but are now only writable by the Vector process user (typically
vector) and readable by group.See the release highlight for details about this change and recommended practices to either secure the disk buffers or to avoid storing secrets in events altogether.
- The
journaldsource has a newemit_cursoroption that, when enabled, adds the__CURSORfield to emitted log records. Thanks to sproberts92 for contributing this change!
3 new features
- Sinks can now encode data as protobuf through support for a new
protobufencoder (configurable usingencoding.codec). Thanks to goakley for contributing this change! - Vector’s version of VRL was updated to 0.8.1, with the following changes:
- Added the
contains_allfunction from_unix_timestampnow accepts a new unit,microsecondsparse_nginx_logno longer fails ifupstream_response_length,upstream_response_time, andupstream_statusare missing- Added the
parse_floatfunction - Improved fallibility diagnostics
- Added the
- Vector now has the ability to start with an empty configuration when using
--allow-empty-config. This is useful if you want to start Vector before loading a configuration using--watch-configor when reloading.
11 bug fixes
- The
docker_logssource no longer incrementscomponent_errors_totalfor out-of-order logs since this is not an error. - The
native_jsoncodec no longer errors when encoding 64-bit float values that represent infinity. Instead, these are encoded as the stringsinfand-inf. This most commonly occurred when encoding histograms where the last bucket bound included infinity. - The
http_serversource no longer panics when handling metrics decoded using thenativeornative_jsoncodecs. This means that it can now be used in conjunction with thehttpsink to send data between Vector instances using thenativeornative_jsoncodecs; however, thevectorsource/sink pair is still the preferred route for Vector-to-Vector communication. - The
kafkasource now fully drains acknowledgements during consumer group rebalancing and when Vector is shutting down. This avoids situations where Vector would duplicate message processing. Thanks to jches for contributing this change! - The
gcp_stackdriver_metricssink now correctly handles configured batch sizes greater than the default of1. Previously it would only send the last event in each batch. - Sources that receive incoming TLS traffic now correctly apply any configured
tls.alpn_protocolsoptions. Previously, these were only applied for sources creating outgoing TLS connections. Thanks to anil-db for contributing this change! - Sources now correctly emit a log and increment
component_discarded_events_totalwhen incoming requests are cancelled before the events are pushed to downstream components. - The
amqpsink no longer panics when the channel is in an error state. Instead, Vector now emits an error event when this occurs. - The
datadog_agentsource now records the “interval” on any incoming metrics that have it set rather than justrate. This is useful as metrics can be interpreted as rates later when viewing the data in Datadog, where theintervalfield will be used. - The
blackholesink no longer reports events processed by default. Instead, this behavior can be opted into by settingprint_interval_secsto a non-zero number.See the upgrade guide for more details.
- A bug in the
clickhousesink heath check where it would add an extra/to the URI, resulting in failures, was fixed. This was a regression in v0.33.0.
Install && Use
Copyright 2019, Vector Authors. All rights reserved.
