vector v0.30 releases: High-Performance, Logs, Metrics, & Events Router
Vector
Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space.
Use cases
- Reduce total observability costs.
- Transition vendors without disrupting workflows.
- Enhance data quality and improve insights.
- Consolidate agents and eliminate agent fatigue.
- Improve overall observability performance and reliability.
Performance
| Test | Vector | Filebeat | FluentBit | FluentD | Logstash | SplunkUF | SplunkHF |
|---|---|---|---|---|---|---|---|
| TCP to Blackhole | 86mib/s | n/a | 64.4mib/s | 27.7mib/s | 40.6mib/s | n/a | n/a |
| File to TCP | 76.7mib/s | 7.8mib/s | 35mib/s | 26.1mib/s | 3.1mib/s | 40.1mib/s | 39mib/s |
| Regex Parsing | 13.2mib/s | n/a | 20.5mib/s | 2.6mib/s | 4.6mib/s | n/a | 7.8mib/s |
| TCP to HTTP | 26.7mib/s | n/a | 19.6mib/s | <1mib/s | 2.7mib/s | n/a | n/a |
| TCP to TCP | 69.9mib/s | 5mib/s | 67.1mib/s | 3.9mib/s | 10mib/s | 70.4mib/s | 7.6mib/s |
Changelog v0.30
10 enhancements
- The
pulsarsink supports a few new features:- Dynamic topics using a topic template
- Can receive both logs and metrics
- Dynamic message
propertiescan be set viaproperties_key
This brings functionality in-line with that which is supported by the
kafkasink.Thanks to addisonj for contributing this change!
- The
kubernetes_logssource supports a newuse_apiserver_cacheoption to have requests from Vector hit the Kubernetes API server cache rather than always hitting etcd. It can significantly reduce Kubernetes control plane memory pressure in exchange for a chance of receiving stale data. Thanks to nabokihms for contributing this change! - The
appsignalsink now allows configuration of TLS options via thetlsconfig field. This brings it in-line with other sinks that support TLS. Thanks to tombruijn for contributing this change! - The
amqpsink now allows configuration of thecontent_encodingandcontent_typemessage properties via the newpropertiesconfiguration option. Thanks to arouene for contributing this change! - The
docker_logssource now supports usage of thetcp://scheme for thehostoption. The connection is the same as-if thehttp://scheme was used. Thanks to OrangeFlag for contributing this change! - Vector’s distroless libc docker images (tags ending in
-distroless-libc) are now based on Debian 11 rather than Debian 10. This matches Vector’s published Debian images (tags ending in-debian). Thanks to SIPR-octo for contributing this change! - The
aws_s3source andaws_s3sink now have full support for codecs and can receive/send any event type allowingaws_s3to be used as a transport layer between Vector instances. - The
tag_cardinality_limitnow includes themetric_namefield on logs it produces to more easily identify the metric that was limited. Thanks to nomonamo for contributing this change! - HTTP-based sinks now log the underlying error if an unexpected error condition is hit. This makes debugging easier.
- AWS components now allow configuring
auth.regionwithout any of the other authentication options so that a different region can be given to the default authentication provider chain than the region that the component is otherwise connecting to.
7 bug fixes
- Disk buffers now recover from partial writes that can occur during unclean shutdowns.
- The
influxdb_logssink now correctly encodes logs whentagsare present. Thanks to juvenn for contributing this change! - The
lokisink now warns when addedlabelscollide via wildcard expansion. Thanks to hargut for contributing this change! - The
elasticsearchsink now uses the correct API to automatically determine the version of the downstream Elasticsearch instance (whenapi_version = "auto"). Thanks to syedriko for contributing this change! - The
gcp_stackdriver_metricssink now correctly refreshes the authentication token before it expires. - Vector’s internal logs were updated to use “suppress” rather than “rate-limit” in the hopes that it makes it clearer that it is only Vector’s log output that is being suppressed, rather than data processing being throttled.
- The
kafkasource now attempts to send any pending acknowledgements to the Kafka server before reading additional messages to process.
Install && Use
Copyright 2019, Vector Authors. All rights reserved.
