Argo Rollouts Traffic Manager for OpenShift Routes

Introduction

Argo Rollouts is a drop-in replacement for Deployments that supports advanced deployment strategies like blue-green and canaries versus the simpler strategies of replace and rolling that Deployment supports.

In a blue-green rollout, we deploy a new version of the application in a separate stack from the current version, with the two versions running in parallel for some time. This enables testing on the new version while users continue to access the current version of the application until a traffic cutover occurs.

Canary can be thought of as an advanced version of blue-green. Instead of an abrupt cut-over of live traffic between a current and new revision, traffic is instead gradually increased to the new version in increments, or steps, while simultaneously decreasing the current revision.

Both of these strategies require that Argo Rollouts manage the traffic between the blue/green or stable/canary versions of the application. The blue/green strategy does not require active traffic management since it is simply a cut-over between services, however the canary strategy does require more active involvement to match the service weighting specified in each step.

To manage this, Argo Rollouts includes a number of traffic managers out-of-the-box as per the documentation. While the Istio Traffic Manager works wonderfully with OpenShift Service Mesh, unfortunately there is not an included traffic manager for OpenShift Routes.

The good news though is Rollouts includes the ability to create traffic managers as plugins and OpenShift GitOps 1.13 and later provide a fully supported plugin to manage OpenShift Routes. This blog will provide a quick introduction to using this traffic manager however complete documentation is also available.

Using the OpenShift Routes Traffic Manager

An OpenShift Route supports multiple services, a primary and one or more alternates, with a weighting for each of the provided services. With Argo Rollouts, the OpenShift Routes Traffic Manager will automatically manage this weighting by adjusting it as needed to match the weighting of the current step in the canary Rollout. 

So for example if the desired weighting for a canary step is 80/20, the traffic manager will automatically adjust the weighting in the Route to 80/20 between services to match. 

To use the OpenShift Routes Traffic Manager all that is required is to specify it in the trafficRouting stanza under strategy and include the name of the Route that is to be managed. Here is an example:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: canary
spec:
  ...
  strategy:
    canary:
      canaryService: canary-canary
      stableService: canary-stable
      trafficRouting:
        plugins:
          argoproj-labs/openshift:
            routes:
              - canary
      steps:
      - setWeight: 20
      - pause: {}
      - setWeight: 40
      - pause: {duration: 10}
      - setWeight: 60
      - pause: {duration: 10}
      - setWeight: 80
      - pause: {duration: 10}
  ...

The canary route, canary, that is being managed is specified as follows:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: canary
  annotations:
    haproxy.router.openshift.io/disable_cookies: 'true'
    haproxy.router.openshift.io/balance: roundrobin
spec:
  port:
    targetPort: http
  tls:
    insecureEdgeTerminationPolicy: Redirect
    termination: edge
  to:
    kind: Service
    name: canary-stable
    weight: 100
  alternateBackends:
    - kind: Service
      name: canary-canary
      weight: 0

Note the stable service (canary-stable) is the primary and the canary service (canary-canary, yes my naming could be better 🙂) is the alternate service. The default weighting is set to 100 for the primary and 0 for the stable. Note also that sticky sessions are disabled to test the weighting independently of persistent sessions 

The demo application shows colored squares based on an image tag (:blue, :green, etc). Once you promote a new image with a new color tag in the Rollout it will progress through the specified steps. When the first step is executed it sets the weighting at 20% and then pauses indefinitely in the second step. Checking the stable route at this point shows the application splitting the canary between the blue (stable) and green (canary) colors in a 80/20 split:

Examining the route at this moment shows the weighting set to 80/20:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: canary
  ...
spec:
  ...
  to:
    kind: Service
    name: canary-stable
    weight: 80
  alternateBackends:
    - kind: Service
      name: canary-canary
      weight: 20

Rollouts Traffic Manager and Argo CD

When deploying the Route keep in mind that the traffic manager will be updating the Route dynamically. If the Route is being managed by Argo CD this will cause Argo CD to complain that it is out-of-sync, and worse if self heal is enabled, revert the changes.

This can be addressed by using the ignoreDifferences feature in Argo CD, also to prevent the traffic manager changes from being reverted on a manual sync the sync option RespectIgnoreDifferences should be used.

Here is an example Argo CD Application with these features enabled:
 

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  labels:
    app.kubernetes.io/name: rollouts-demo
  name: rollouts-demo-prod
  namespace: rollouts-demo-gitops
spec:
  destination:
    namespace: rollouts-demo-prod
    server: https://kubernetes.default.svc
  ignoreDifferences:
  - group: route.openshift.io
    jsonPointers:
    - /status
    - /spec/to/weight
    - /spec/alternateBackends
    kind: Route
  project: default
  source:
    path: environments/overlays/prod
    repoURL: https://github.com/gitops-examples/rollouts-demo.git
    targetRevision: main
  syncPolicy:
    automated:
      selfHeal: true
    syncOptions:
    - RespectIgnoreDifferences=true

Note if you are deploying multiple routes out of the same Application the ignoreDifferences capability can select resources based on a specific name enabling it to be applied only to the canary route managed by the traffic manager.

Conclusion

In this brief blog we reviewed how to use the OpenShift Routes Traffic Manager in OpenShift GitOps. A complete example of this feature is available in the gitops-example/rollouts-demo repository and again the documentation is available here.