The objective of this article is to help clients migrate the default collector in Red Hat OpenShift 4 from Fluentd to Vector. Fluentd is deprecated in Logging 5.X versions. To utilize the latest features of Logging 6.0, it’s necessary to migrate to Vector from Fluentd. Test these changes in development and test environments and develop a plan for implementing these changes in production.
What is Vector?
Vector is a log collector and analyzer that streamlines the processing of distributed logs from any source to a single stream for real-time analysis.
For example, with Vector as a log collector, customers can send logs to an Amazon CloudWatch destination for further analysis. Additionally, customers can assemble log messages as a stack-trace in one single log entity using the JSON format.
Why migrate to Vector?
Vector draws inspiration from monitoring tools like StatsD and collectd, leveraging their instrumentation techniques to provide direct processing of JSON and StatsD formatted data without additional configuration or plugins.
In addition, Fluentd is deprecated in Logging 5.X versions. To utilize the latest features of Logging 6.0, it’s necessary to migrate to Vector from Fluentd. This article is intended to be a guide for users to test these changes in their dev and test environments and develop a plan for implementing these changes in production.
Migrate Fluentd to Vector in OpenShift
The following describes how to migrate the OpenShift Logging collection service from Fluentd to Vector. Starting with OpenShift Logging 5.6, Vector replaces Fluentd as the default Collector. In Logging 5.6, Fluentd is deprecated and is planned to be removed in a future release. Red Hat will provide bug fixes and support for this feature during the current release lifecycle, but this feature will no longer receive enhancements and will be removed.
Prerequisites:
- Installed Red Hat OpenShift Logging Operator (current stable v5.5+)
- Review features (supported and unsupported) comparison between Vector and Fluentd before performing a migration to Vector.
Note:
- To migrate default log store from Elasticsearch to Loki in Red Hat OpenShift Logging follow the article.
- To migrate the collector from Fluentd to Vector reducing the number of logs duplicated follow the article
Current stack
Assuming the current stack looks like the below that represents basic log collection only. The config might vary regarding resources/nodes/tolerations/selectors.
apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" namespace: "openshift-logging" spec: managementState: "Managed" collection: logs: type: "fluentd" fluentd: resources: limits: memory: 736Mi requests: cpu: 100m memory: 736Mi ----- Output Omitted -----Stack migrated to Vector
Edit the clusterlogging instance custom resource to use vector as log collection type. Allocated resources can be customized as shown in below config. To configure CPU and memory limits for Vector, follow this article.
$ oc edit clusterlogging instance -n openshift-logging apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" namespace: "openshift-logging" spec: managementState: "Managed" collection: type: vector ----- Output Omitted -----Summary
This article is a guide for users to test changes in their dev and test environments and develop a plan for implementing these changes in production.
