Custom WebAssembly extensions in OpenShift Service Mesh

Red Hat OpenShift Service Mesh 2.1 requires using WebAssembly extensions instead of Istio Mixer to extend Service Mesh functionality. The 2.11 release of Red Hat 3scale API Management also supports using WebAssembly extensions. Thus, the latest release of the 3scale integration for Istio uses the WebAssembly proxy instead of the Istio Mixer component.

Developers can use WebAssembly extensions in OpenShift Service Mesh and 3scale to add features directly to the Envoy proxy, thereby moving common functionality out of applications and into the sidecar. Examples of useful extensions include:

• Manipulating headers for the request or response.
• Adding authorization or custom rate limiting to requests.
• Adding an additional data store or cache for side-channel request processing.

This article introduces the new WebAssembly proxy for OpenShift Service Mesh, with an example demonstrating how to configure an extension using 3scale.

Note: Red Hat developers working on this integration are speaking at Apidays LIVE Paris - APIs and the Future of Software, a virtual event happening December 7 through 9, 2021. See the end of the article for details and a video demonstration showing how to configure the new 3scale WebAssembly extension in Red Hat OpenShift.

What is WebAssembly?

WebAssembly (sometimes called Wasm) is a binary instruction format for stack-based virtual machines (VMs). The code is executed at near-native speed in a memory-safe sandbox, with clearly defined resource constraints and an API for communicating with the embedding host environment. Although the standards were developed to work for the web and to be executed in web browsers, WebAssembly can be embedded in non-web applications as well.

The Proxy-Wasm specification provides conventions to use between hosts and their extensions delivered as WebAssembly modules. Hosts are typically transport-layer (L4) or application-layer (L7) proxies but can be other host environments, as well. Although the specification was primarily developed for using WebAssembly in Envoy, it is proxy-agnostic and can be used with different proxies.

The specification defines the set of interfaces, commonly called application binary interfaces (ABIs), in order to facilitate WebAssembly extension development. Proxy-Wasm extensions can be written in any language targeting WebAssembly and adhering to the interface library in the languages provided by Envoy: C/C++ using Emscripten, Rust, Go, and TypeScript. The extensions are compiled to bytecode and run inside a virtual machine sandbox with a well-defined interface to external applications using an HTTP proxy.

WebAssembly in Istio and Service Mesh

WebAssembly has been included with Istio, the OpenShift Service Mesh upstream project, since version 1.5, and was introduced to OpenShift Service Mesh 2.0 as a tech preview. The Service Mesh 2.1 release includes Istio 1.9, which standardizes on using extensions from WebAssembly for Proxies (Proxy-Wasm). Proxy-Wasm uses HTTP filters provided by the Envoy proxy to extend common functionality.

Advantages of WebAssembly over Istio Mixer

Previously, Istio used a mixer component to handle the collection of telemetry data across the many Envoy proxies that make up a data plane (Figure 1). This central component also made it possible to add extensions for creating custom policies and metrics that apply across the mesh.

This design had multiple drawbacks. Most notably, the central point could become a bottleneck and a resource hog when there was a large amount of traffic. Having a central point also creates a single point of failure.

Mixer was deprecated in OpenShift Service Mesh 2.0, and has now been completely removed. You can't even upgrade to OpenShift Service Mesh 2.1 while the Istio Mixer component is active. Instead, OpenShift Service Mesh embraced WebAssembly extensions for extending Envoy. The new Telemetry v2 architecture, introduced previously, replaces the mixer component's telemetry functions. The Istio and Envoy developer communities have been building WebAssembly extensibility into Envoy since Istio 1.5.

WebAssembly extensions provide two significant benefits over Istio Mixer:

• WebAssembly extensions live at the Envoy proxy level, so there is no single point of contention.
• Extensions are deployed in a sandbox with resource constraints and a clearly defined API, providing better security and isolation.

Using the WebAssembly plugin with Envoy proxies

Figure 2 shows the typical communication between WebAssembly components.

WebAssembly, as a format for extending Envoy, has been incorporated into Istio using the ServiceMeshExtension API. This API specifies how to package and deploy a WebAssembly plugin to be used alongside Istio’s Envoy proxies. It is possible to deploy extensions to all proxies in the mesh, to specific namespaces, or to individual workloads using the workloadSelector field. Consider the following sample ServiceMeshExtension:

apiVersion: maistra.io/v1alpha1
kind: ServiceMeshExtension
metadata:
  name: header-append
spec:
  config:
    custom-header: test
  image: quay.io/maistra-dev/header-append-filter:2.1
  phase: PostAuthZ
  priority: 1000
  workloadSelector:
    labels:
      app: httpbin

Custom WebAssembly extensions

There are two steps to writing and deploying a custom WebAssembly extension to OpenShift Service Mesh:

1. Use the SDK and WebAssembly for the Proxies API to write the custom extension in a supported language.
2. Compile the WebAssembly extension into a .wasm file containing the bytecode and provide a Manifest.yaml in the container image's root folder.

The 3scale WebAssembly extension

The Proxy-WASM Authorization extension using 3scale (threescale-wasm-auth) module is a Proxy-Wasm extension that authorizes HTTP requests made to 3scale. The extension is part of OpenShift Service Mesh 2.1 and, unlike custom WebAssembly extensions, is completely supported by Red Hat. The threescale-wasm-auth module is designed to be fully compatible with all implementations of the Proxy-WASM ABI specification. The module has been thoroughly tested and is supported to work with the Envoy reverse proxy. Figure 3 shows typical communication through the extension.

The 3scale WebAssembly extension supports the following activities:

• Connect to a 3scale Software-as-a-Service (SaaS), Red Hat OpenShift API Management, or a 3scale self-managed tenant for a specific provider account.
• Configure one or many services to manage within the specified provider account.
• Provide mapping rules to apply for each service.
• Provide a security pattern to be used by the services. All supported 3scale patterns—API key, application identifier/key pair, and Open ID Connect (OIDC)—can be used.
• As with APIcast, authorize and report each request to 3scale's backend URL through an API request to the Service Management API's AuthRep operation.

How to configure a 3scale WebAssembly extension

The 3scale WebAssembly configuration is provided by the ServiceEntry and ServiceMeshExtension custom resources (CRs). In the next sections, I'll explain how to configure these custom resources.

The ServiceEntry custom resource

This custom resource lets you add entries to Istio’s internal service registry so that auto-discovered services in the mesh can access and route to manually specified services. For 3scale WebAssembly, two service entries are needed.

The first entry is used to connect to the 3scale provider account's admin URL, in order to access the service configurations. Here is an example configuration:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
 labels:
   app: ''
 name: system-entry
spec:
 hosts:
   - 3scale-admin.apps.cluster-qtrg2.qtrg2.sandbox961.opentlc.com
 location: MESH_EXTERNAL
 ports:
   - name: https
     number: 80
     protocol: HTTP
 resolution: DNS

The second entry is used to connect to the backend URL of the 3scale provider, in order to authorize each request. Here is an example configuration:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
 labels:
   app: ''
 name: system-entry
spec:
 hosts:
   - backend-3scale.apps.cluster-qtrg2.qtrg2.sandbox961.opentlc.com
 location: MESH_EXTERNAL
 ports:
   - name: https
     number: 80
     protocol: HTTP
 resolution: DNS

The ServiceMeshExtension custom resource

This custom resource configures the WebAssembly module containing the 3scale provider and access token, list of services, mapping rules, authentication methods, backend URL, and service token. An example of this configuration follows:

apiVersion: maistra.io/v1
kind: ServiceMeshExtension
metadata:
 labels:
   app: ''
 name: threescale-wasm-extension
 namespace: bookinfo
spec:
 config:
   api: v1
   backend:
     extensions:
       - no_body
     name: backend
     upstream:
       name: >-
         outbound|80||backend-3scale.apps.cluster-qtrg2.qtrg2.sandbox961.opentlc.com
       timeout: 5000
       url: 'http://backend-3scale.apps.cluster-qtrg2.qtrg2.sandbox961.opentlc.com'
   services:
     - authorities:
         - '*'
       credentials:
         app_id:
           - header:
               keys:
                 - app_id
           - query_string:
               keys:
                 - app_id
         app_key:
           - header:
               keys:
                 - app_key
           - query_string:
               keys:
                 - app_key
         user_key:
           - query_string:
               keys:
                 - user_key
           - header:
               keys:
                 - user_key
       id: '3'
       mapping_rules:
         - method: GET
           pattern: /
           usages:
             - delta: 1
               name: hits
       token: 43a5db7bf61bdf55a67ff54edbee7c579b6a2956146cfef2179486561476abe0
   system:
     name: system
     token: ose6aAdE777Dz0zb
     upstream:
       name: >-
         outbound|80||3scale-admin.apps.cluster-qtrg2.qtrg2.sandbox961.opentlc.com
       timeout: 5000
       url: 'http://3scale-admin.apps.cluster-qtrg2.qtrg2.sandbox961.opentlc.com'
 image: 'quay.io/3scale/threescale-wasm-auth:qe'
 phase: PostAuthZ
 priority: 100
 workloadSelector:
   labels:
     app: productpage

Note: The service configuration contains the service ID, which is the unique ID of the service in 3scale. Also, in this release, the service configuration is provided in the ServiceMeshExtension configuration. There are plans to fetch this configuration from the provider configuration in future releases.

Watch a video demonstration

The following video demonstrates how to use the Red Hat OpenShift user interface, commands, and configuration files to configure a 3scale WebAssembly extension.

Learn more at Apidays Paris (virtual event)

Apidays is the leading industry tech and business series of conferences about APIs and the programmable economy. From December 7 through 9, 2021, Apidays LIVE Paris - APIs and the Future of Software will host virtual workshops and roundtables with technical and business leaders who will explain how they create value with APIs, and how they aligned technical capabilities to make it happen.

Registration at Apidays Paris is free, and you can join both of the Red Hat sessions at the event:

• Tuesday, December 7, 2021 at 3:40 PM CET: Author Satya Jayanti will present on Istio and API management integration with 3scale WebAssembly extensions.
• Wednesday, December 8, 2021 at 2:25 pm CET: Author Hugo Guerrero Olivares will present on getting started with event-driven APIs.

Red Hat is a sponsor of the conference, and our experts are available in our sponsor booth throughout the event to answer questions on API management, integration, open banking, and cloud-native development at Red Hat.

Conclusion

This article introduced the benefits of WebAssembly and how to use it to create Envoy HTTP filter extensions for OpenShift Service Mesh. In particular, I introduced the 3scale WebAssembly authorization extension and showed you an example configuration.

To learn more about the 3scale WebAssembly extension, see the threescale-webassembly-module documentation. You might also want to visit the following resources for more detail and explanation:

• My video demonstration showing how to configure a 3scale WebAssembly extension in OpenShift.
• The article, Design an authorization cache for Envoy proxy using WebAssembly (November 2021).
• The article, How we implemented an authorization cache for Envoy proxy (November 2021).

The 3scale WebAssembly extension eases the integration of OpenShift Service Mesh and 3scale, and it provides a standard way to inject 3scale API Management configurations into OpenShift Service Mesh for execution in a single data plane. As one of the first WebAssembly extensions to API Management platforms, this work pioneers an efficient integration with OpenShift Service Mesh and paves the way for further Envoy proxy integrations to 3scale. We hope you will join us at Apidays Paris to learn more about it!