vector v0.34 releases: High-Performance, Logs, Metrics, & Events Router
![Events Router](https://cdn-0.securityonline.info/wp-content/uploads/2019/07/Capture-2.png)
Vector
Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space.
Use cases
- Reduce total observability costs.
- Transition vendors without disrupting workflows.
- Enhance data quality and improve insights.
- Consolidate agents and eliminate agent fatigue.
- Improve overall observability performance and reliability.
Performance
Test | Vector | Filebeat | FluentBit | FluentD | Logstash | SplunkUF | SplunkHF |
---|---|---|---|---|---|---|---|
TCP to Blackhole | 86mib/s | n/a | 64.4mib/s | 27.7mib/s | 40.6mib/s | n/a | n/a |
File to TCP | 76.7mib/s | 7.8mib/s | 35mib/s | 26.1mib/s | 3.1mib/s | 40.1mib/s | 39mib/s |
Regex Parsing | 13.2mib/s | n/a | 20.5mib/s | 2.6mib/s | 4.6mib/s | n/a | 7.8mib/s |
TCP to HTTP | 26.7mib/s | n/a | 19.6mib/s | <1mib/s | 2.7mib/s | n/a | n/a |
TCP to TCP | 69.9mib/s | 5mib/s | 67.1mib/s | 3.9mib/s | 10mib/s | 70.4mib/s | 7.6mib/s |
Changelog v0.34
7 enhancements
- Sinks now have additional options for
encoding.timestamp_format
:unix_float
: Represents the timestamp as a Unix timestamp in floating point.unix_ms
: Represents the timestamp as a Unix timestamp in milliseconds.unix_ns
: Represents the timestamp as a Unix timestamp in nanoseconds.unix_us
: Represents the timestamp as a Unix timestamp in microseconds
Thanks to srstrickland for contributing this change!
- The
prometheus_remote_write
sink now has the ability to disable aggregation by settingbatch.aggregate
tofalse
. - The
nats
source has a newsubscriber_capacity
configuration option to control how many messages the NATS subscriber buffers before incoming messages are dropped. - Sources and sinks that run a HTTP server now emit additional internal metrics:
http_server_requests_received_total
http_server_responses_sent_total
http_server_handler_duration_seconds
- Sources that run a gRPC server now emit additional internal metrics:
grpc_server_messages_received_total
grpc_server_messages_sent_total
- `grpc_server_handler_duration_seconds
- Event metadata, including secrets like Datadog API key or Splunk HEC token, are now persisted when writing events to a disk buffer so that sinks have access to it.
As part of this change, disk buffers created by Vector now have more restrictive file permissions on Unix platforms. Previously, they were world-readable, but are now only writable by the Vector process user (typically
vector
) and readable by group.See the release highlight for details about this change and recommended practices to either secure the disk buffers or to avoid storing secrets in events altogether.
- The
journald
source has a newemit_cursor
option that, when enabled, adds the__CURSOR
field to emitted log records. Thanks to sproberts92 for contributing this change!
3 new features
- Sinks can now encode data as protobuf through support for a new
protobuf
encoder (configurable usingencoding.codec
). Thanks to goakley for contributing this change! - Vector’s version of VRL was updated to 0.8.1, with the following changes:
- Added the
contains_all
function from_unix_timestamp
now accepts a new unit,microseconds
parse_nginx_log
no longer fails ifupstream_response_length
,upstream_response_time
, andupstream_status
are missing- Added the
parse_float
function - Improved fallibility diagnostics
- Added the
- Vector now has the ability to start with an empty configuration when using
--allow-empty-config
. This is useful if you want to start Vector before loading a configuration using--watch-config
or when reloading.
11 bug fixes
- The
docker_logs
source no longer incrementscomponent_errors_total
for out-of-order logs since this is not an error. - The
native_json
codec no longer errors when encoding 64-bit float values that represent infinity. Instead, these are encoded as the stringsinf
and-inf
. This most commonly occurred when encoding histograms where the last bucket bound included infinity. - The
http_server
source no longer panics when handling metrics decoded using thenative
ornative_json
codecs. This means that it can now be used in conjunction with thehttp
sink to send data between Vector instances using thenative
ornative_json
codecs; however, thevector
source/sink pair is still the preferred route for Vector-to-Vector communication. - The
kafka
source now fully drains acknowledgements during consumer group rebalancing and when Vector is shutting down. This avoids situations where Vector would duplicate message processing. Thanks to jches for contributing this change! - The
gcp_stackdriver_metrics
sink now correctly handles configured batch sizes greater than the default of1
. Previously it would only send the last event in each batch. - Sources that receive incoming TLS traffic now correctly apply any configured
tls.alpn_protocols
options. Previously, these were only applied for sources creating outgoing TLS connections. Thanks to anil-db for contributing this change! - Sources now correctly emit a log and increment
component_discarded_events_total
when incoming requests are cancelled before the events are pushed to downstream components. - The
amqp
sink no longer panics when the channel is in an error state. Instead, Vector now emits an error event when this occurs. - The
datadog_agent
source now records the “interval” on any incoming metrics that have it set rather than justrate
. This is useful as metrics can be interpreted as rates later when viewing the data in Datadog, where theinterval
field will be used. - The
blackhole
sink no longer reports events processed by default. Instead, this behavior can be opted into by settingprint_interval_secs
to a non-zero number.See the upgrade guide for more details.
- A bug in the
clickhouse
sink heath check where it would add an extra/
to the URI, resulting in failures, was fixed. This was a regression in v0.33.0.
Install && Use
Copyright 2019, Vector Authors. All rights reserved.