vector v0.30 releases: High-Performance, Logs, Metrics, & Events Router
Vector
Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space.
Use cases
- Reduce total observability costs.
- Transition vendors without disrupting workflows.
- Enhance data quality and improve insights.
- Consolidate agents and eliminate agent fatigue.
- Improve overall observability performance and reliability.
Performance
Test | Vector | Filebeat | FluentBit | FluentD | Logstash | SplunkUF | SplunkHF |
---|---|---|---|---|---|---|---|
TCP to Blackhole | 86mib/s | n/a | 64.4mib/s | 27.7mib/s | 40.6mib/s | n/a | n/a |
File to TCP | 76.7mib/s | 7.8mib/s | 35mib/s | 26.1mib/s | 3.1mib/s | 40.1mib/s | 39mib/s |
Regex Parsing | 13.2mib/s | n/a | 20.5mib/s | 2.6mib/s | 4.6mib/s | n/a | 7.8mib/s |
TCP to HTTP | 26.7mib/s | n/a | 19.6mib/s | <1mib/s | 2.7mib/s | n/a | n/a |
TCP to TCP | 69.9mib/s | 5mib/s | 67.1mib/s | 3.9mib/s | 10mib/s | 70.4mib/s | 7.6mib/s |
Changelog v0.30
10 enhancements
- The
pulsar
sink supports a few new features:- Dynamic topics using a topic template
- Can receive both logs and metrics
- Dynamic message
properties
can be set viaproperties_key
This brings functionality in-line with that which is supported by the
kafka
sink.Thanks to addisonj for contributing this change!
- The
kubernetes_logs
source supports a newuse_apiserver_cache
option to have requests from Vector hit the Kubernetes API server cache rather than always hitting etcd. It can significantly reduce Kubernetes control plane memory pressure in exchange for a chance of receiving stale data. Thanks to nabokihms for contributing this change! - The
appsignal
sink now allows configuration of TLS options via thetls
config field. This brings it in-line with other sinks that support TLS. Thanks to tombruijn for contributing this change! - The
amqp
sink now allows configuration of thecontent_encoding
andcontent_type
message properties via the newproperties
configuration option. Thanks to arouene for contributing this change! - The
docker_logs
source now supports usage of thetcp://
scheme for thehost
option. The connection is the same as-if thehttp://
scheme was used. Thanks to OrangeFlag for contributing this change! - Vector’s distroless libc docker images (tags ending in
-distroless-libc
) are now based on Debian 11 rather than Debian 10. This matches Vector’s published Debian images (tags ending in-debian
). Thanks to SIPR-octo for contributing this change! - The
aws_s3
source andaws_s3
sink now have full support for codecs and can receive/send any event type allowingaws_s3
to be used as a transport layer between Vector instances. - The
tag_cardinality_limit
now includes themetric_name
field on logs it produces to more easily identify the metric that was limited. Thanks to nomonamo for contributing this change! - HTTP-based sinks now log the underlying error if an unexpected error condition is hit. This makes debugging easier.
- AWS components now allow configuring
auth.region
without any of the other authentication options so that a different region can be given to the default authentication provider chain than the region that the component is otherwise connecting to.
7 bug fixes
- Disk buffers now recover from partial writes that can occur during unclean shutdowns.
- The
influxdb_logs
sink now correctly encodes logs whentags
are present. Thanks to juvenn for contributing this change! - The
loki
sink now warns when addedlabels
collide via wildcard expansion. Thanks to hargut for contributing this change! - The
elasticsearch
sink now uses the correct API to automatically determine the version of the downstream Elasticsearch instance (whenapi_version = "auto"
). Thanks to syedriko for contributing this change! - The
gcp_stackdriver_metrics
sink now correctly refreshes the authentication token before it expires. - Vector’s internal logs were updated to use “suppress” rather than “rate-limit” in the hopes that it makes it clearer that it is only Vector’s log output that is being suppressed, rather than data processing being throttled.
- The
kafka
source now attempts to send any pending acknowledgements to the Kafka server before reading additional messages to process.
Install && Use
Copyright 2019, Vector Authors. All rights reserved.