vector v0.34 releases: High-Performance, Logs, Metrics, & Events Router
Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space.
- Reduce total observability costs.
- Transition vendors without disrupting workflows.
- Enhance data quality and improve insights.
- Consolidate agents and eliminate agent fatigue.
- Improve overall observability performance and reliability.
|TCP to Blackhole||86mib/s||n/a||64.4mib/s||27.7mib/s||40.6mib/s||n/a||n/a|
|File to TCP||76.7mib/s||7.8mib/s||35mib/s||26.1mib/s||3.1mib/s||40.1mib/s||39mib/s|
|TCP to HTTP||26.7mib/s||n/a||19.6mib/s||<1mib/s||2.7mib/s||n/a||n/a|
|TCP to TCP||69.9mib/s||5mib/s||67.1mib/s||3.9mib/s||10mib/s||70.4mib/s||7.6mib/s|
- Sinks now have additional options for
unix_float: Represents the timestamp as a Unix timestamp in floating point.
unix_ms: Represents the timestamp as a Unix timestamp in milliseconds.
unix_ns: Represents the timestamp as a Unix timestamp in nanoseconds.
unix_us: Represents the timestamp as a Unix timestamp in microseconds
Thanks to srstrickland for contributing this change!
prometheus_remote_writesink now has the ability to disable aggregation by setting
natssource has a new
subscriber_capacityconfiguration option to control how many messages the NATS subscriber buffers before incoming messages are dropped.
- Sources and sinks that run a HTTP server now emit additional internal metrics:
- Sources that run a gRPC server now emit additional internal metrics:
- Event metadata, including secrets like Datadog API key or Splunk HEC token, are now persisted when writing events to a disk buffer so that sinks have access to it.
As part of this change, disk buffers created by Vector now have more restrictive file permissions on Unix platforms. Previously, they were world-readable, but are now only writable by the Vector process user (typically
vector) and readable by group.
See the release highlight for details about this change and recommended practices to either secure the disk buffers or to avoid storing secrets in events altogether.
journaldsource has a new
emit_cursoroption that, when enabled, adds the
__CURSORfield to emitted log records. Thanks to sproberts92 for contributing this change!
3 new features
- Sinks can now encode data as protobuf through support for a new
protobufencoder (configurable using
encoding.codec). Thanks to goakley for contributing this change!
- Vector’s version of VRL was updated to 0.8.1, with the following changes:
- Added the
from_unix_timestampnow accepts a new unit,
parse_nginx_logno longer fails if
- Added the
- Improved fallibility diagnostics
- Added the
- Vector now has the ability to start with an empty configuration when using
--allow-empty-config. This is useful if you want to start Vector before loading a configuration using
--watch-configor when reloading.
11 bug fixes
docker_logssource no longer increments
component_errors_totalfor out-of-order logs since this is not an error.
native_jsoncodec no longer errors when encoding 64-bit float values that represent infinity. Instead, these are encoded as the strings
-inf. This most commonly occurred when encoding histograms where the last bucket bound included infinity.
http_serversource no longer panics when handling metrics decoded using the
native_jsoncodecs. This means that it can now be used in conjunction with the
httpsink to send data between Vector instances using the
native_jsoncodecs; however, the
vectorsource/sink pair is still the preferred route for Vector-to-Vector communication.
kafkasource now fully drains acknowledgements during consumer group rebalancing and when Vector is shutting down. This avoids situations where Vector would duplicate message processing. Thanks to jches for contributing this change!
gcp_stackdriver_metricssink now correctly handles configured batch sizes greater than the default of
1. Previously it would only send the last event in each batch.
- Sources that receive incoming TLS traffic now correctly apply any configured
tls.alpn_protocolsoptions. Previously, these were only applied for sources creating outgoing TLS connections. Thanks to anil-db for contributing this change!
- Sources now correctly emit a log and increment
component_discarded_events_totalwhen incoming requests are cancelled before the events are pushed to downstream components.
amqpsink no longer panics when the channel is in an error state. Instead, Vector now emits an error event when this occurs.
datadog_agentsource now records the “interval” on any incoming metrics that have it set rather than just
rate. This is useful as metrics can be interpreted as rates later when viewing the data in Datadog, where the
intervalfield will be used.
blackholesink no longer reports events processed by default. Instead, this behavior can be opted into by setting
print_interval_secsto a non-zero number.
See the upgrade guide for more details.
- A bug in the
clickhousesink heath check where it would add an extra
/to the URI, resulting in failures, was fixed. This was a regression in v0.33.0.
Copyright 2019, Vector Authors. All rights reserved.