Skupper.io: Let your services communicate across Kubernetes clusters

In the past few years, the popularity and adoption of containers has skyrocketed, and the Kubernetes container orchestration platform has been largely adopted as well. With these changes, a new set of challenges has emerged when dealing with applications deployed on Kubernetes clusters in the real world. One challenge is how to deal with communication between multiple clusters that might be in different networks (even private ones), behind firewalls, and so on.

One possible solution to this problem is to use a Virtual Application Network (VAN), which is sometimes referred to as a Layer 7 network. In a nutshell, a VAN is a logical network that is deployed at the application level and introduces a new layer of addressing for fine-grained application components with no constraints on the network topology. For a much more in-depth explanation, please read this excellent article.

So, what is Skupper? In the project's own words:

Skupper is a layer seven service interconnect. It enables secure communication across Kubernetes clusters with no VPNs or special firewall rules.

Skupper is a VAN implementation. Its high-level architecture is depicted in Figure 1:

Once Skupper is installed in each Kubernetes cluster and connected with the others, Service A from cluster one can communicate with Service B from cluster two, and with Service C from cluster three (and the other way around). The clusters do not need to be all public or on the same infrastructure. They can be behind a firewall, or even on private networks not accessible from outside (providing that they can reach outside to connect themselves to the skupper VAN).

Skupper architecture

Given the overall architecture, how does the green Skupper component in Figure 1 work? Figure 2 lists the Skupper components for Kubernetes cluster one and describes the interactions between them:

The first thing to notice is that Skupper can be managed using the dedicated skupper CLI command. Next is that when you first install Skupper in a Kubernetes and OpenShift cluster, two components are deployed: skupper-router and skupper-proxy-controller. The former is an instance of Apache Qpid, an open source Advanced Message Queuing Protocol (AMQP) router that is responsible for creating the VAN. Qpid helps Kubernetes and OpenShift clusters communicate by tunneling HTTP and TCP traffic into AMQP.

If you are familiar with the Operator pattern, think of skupper-proxy-controller as an Operator: This tool watches for services annotated with skupper.io/proxy and instantiates, for each of them, a service-*-proxy pod. That pod tunnels the protocols spoken by Service A (i.e., HTTP or TCP) into AMQP and the skupper-router does the rest.

Use case: A local service exporter with Skupper

Another challenge that comes with developing cloud-native applications coding and testing effectively in a hybrid environment. Some of the services that compose your application might be, at development time, hosted locally on your laptop (or in your dev environment), while others might be deployed on a remote Kubernetes cluster. How would you easily make your local services accessible to the remote cluster without setting up a VPN or another "complicated" network configuration?

One option is to leverage the Skupper mechanism to export a local service running outside of Kubernetes to a remote Kubernetes cluster. Figure 3 sketches what we want to achieve—a simple TCP echo service running on our laptop that is usually connected to a firewalled private network:

We want to access this service from a pod running on the remote Kubernetes cluster that has no network access to the laptop network.
Note: A number of manual steps are needed at the moment because this use case is not yet implemented as a skupper CLI subcommand. There is an issue open regarding when that feature will be implemented. When it is, all of the following steps will be replaced by a single skupper CLI invocation.

Let us look at the steps you will need to follow (for now):

1. Start by downloading the latest version of the skupper CLI from https://github.com/skupperproject/skupper-cli/releases. Add it to your path and make it executable.
2. Log into your remote cluster with oc or kubectl.
3. Install Skupper on the remote cluster:

$ skupper init --id public

4. Make the skupper router accessible from outside the cluster:

$ oc apply -f - << EOF
kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: skupper-messaging
spec:
  to:
    kind: Service
    name: skupper-messaging
    weight: 100
  port:
    targetPort: amqps
  tls:
    termination: passthrough
    insecureEdgeTerminationPolicy: None
  wildcardPolicy: None
EOF

5. Create a service on the cluster representing the local echo service. This service will not have an implementation, it is just to help Skupper to correctly handle this feature:

$ oc apply -f - << EOF
kind: Service
apiVersion: v1
metadata:
  name: echo
  annotations:
    skupper.io/proxy: tcp
spec:
  ports:
    - protocol: TCP
      port: 2000
      targetPort: 2000
  selector:
    dummy: selector
EOF

6. Clone the skupper-proxy project:

$ git clone git@github.com:skupperproject/skupper-proxy.git

7. Go to that project's bin/ directory with:

$ cd skupper-proxy/bin

8. Extract certificates that are needed to mutually authenticate your connection:

$ oc extract secret/skupper

9. Modify connection.json to look like this:

{
    "scheme": "amqps",
    "host": "",
    "port": "443",
    "verify": false,
    "tls": {
        "ca": "ca.crt",
        "cert": "tls.crt",
        "key": "tls.key",
        "verify": false
    }
}

where the host can be obtained by:

$ oc get route skupper-messaging -o=jsonpath='{.spec.host}{"\n"}'

10. Run a simple Bash TCP echo service on port 2000 locally in a separate shell that will be kept running:

$ nc -l 2000 -k -c 'xargs -n1 echo'

11. Connect to the Skupper running on the remote cluster. Note that the port must match the one used by the echo service in step 10 (i.e., 2000) and echo must match the name of the service you created on the cluster earlier:

$ node ./simple.js 'amqp:echo=>tcp:2000'

Now. everything is set. You can go the remote cluster, use a pod that has the Netcat command nc installed like busybox:latest, and run:

$ nc localhost echo

Type a word, and you should see it echoed back as it passes through skupper-proxy-pod -> skupper routerrouter -> simple.js on your laptop -> the TCP echo service on your laptop, and back!

Conclusions and next steps

This project is still under heavy development and improves every day. Check out the progress on the official site or directly from the code on the GitHub project. In particular, here is the issue that tracks all of the work involved in supporting the local service exporter use case directly as a Skupper CLI command.

Last updated: June 29, 2020