Bringing Coolstore Microservices to the Service Mesh: Part 2–Manual Injection

Coolstore+Istio Logo

In the first part of this series we explored the Istio project and how Red Hat is committed to and actively involved in the project and working to integrate it into Kubernetes and OpenShift to bring the benefits of a service mesh to our customers and the wider communities involved. If you want to play with Istio, check out the service mesh tutorials on learn.openshift.com. If you want to install it, follow the Istio Kubernetes quickstart instructions and install it on OpenShift 3.7 or later. Also don’t miss Don Schenck’s series of blogs on Istio technology in general to learn more about it and what Red Hat is doing in this space.

In this post, we will deploy the existing Coolstore microservices demo as a service mesh and start to demonstrate the tangible value you can get out of the system without any major rewrite or rearchitecture of the existing app. We’ll also improve our project along the way to adhere to Istio (and general microservice) best practices. In the real world, your applications and developers often make bad assumptions or fail to implement best practices, so with this information you can learn something about your own projects. For Coolstore, many of these workarounds will eventually find their way into the source code of the demo.

Getting Started

Let’s assume you already have OpenShift 3.7+ installed (I am using OpenShift Origin 3.9.0.alpha3 because at press time, OpenShift Container Platform 3.9 has not yet been released). Let’s further assume you’ve installed Istio 0.6.0 or later, including the Prometheus, Servicegraph, Jaeger, and Grafana plug-ins  (see the shell script installer in the More Reading section below for instructions on how to install istio and its plug-ins). Verify Istio is installed and running in the istio-system namespace:

% oc get pods -n istio-system
NAME                                READY     STATUS    RESTARTS   AGE
grafana-89f97d9c-2wtxx              1/1       Running   0          7m
istio-ca-59f6dcb7d9-bs9hx           1/1       Running   0          7m
istio-ingress-779649ff5b-jw4hg      1/1       Running   0          7m
istio-mixer-7f4fd7dff-pct4j         3/3       Running   0          7m
istio-pilot-5f5f76ddc8-pvq4d        2/2       Running   0          7m
jaeger-deployment-559c8b9b8-klspv   1/1       Running   0          7m
prometheus-cf8456855-svpmb          1/1       Running   0          7m
servicegraph-59ff5dbbff-7zrfp       1/1       Running   0          7m

Let’s get started. Clone the Coolstore repo and then play along:

% git clone https://github.com/jbossdemocentral/coolstore-microservice

And make sure you are logged in as a cluster administrator, or you have cluster-admin privileges, since it’ll require you to make some policy and permission changes later on. (As mentioned in part 1, this will be fine-tuned in future Istio releases to not require so much privilege and permission.) You can do this by logging in as a cluster administrator, or if you have sudoer privileges, you can add --as=system:admin to all the oc commands below.

Manual Injection of Sidecars

In part 1, I showed that with sidecar auto-injection, your app’s pods are automatically festooned with Envoy proxies without ever having to change the application’s deployments. However, there are issues with it that currently prevent us from using it. For now, we’ll do manual injection. Manually injecting has the obvious drawback that you have to do the injection, but it has a big benefit as well: you need to do it only once (per release of Istio), and once it’s done you can check in the results to your source code management system—infrastructure as code and all that.

Let’s first set a few environment variables in our Linux shell so we can reference them later on. (Just copy and paste all of these commands into your Linux terminal. If you’re on Windows, there is hope for you.)

# Version of Istio we are using
ISTIO_VERSION=0.6.0

# The name of the OpenShift project into which you installed Istio. It should
# be istio-system as that's what the Istio nstaller creates for you
ISTIO_PROJECT=istio-system

# Name of project to house our coolstore service mesh
COOLSTORE_PROJECT=coolstore-mesh

# Location of istio binaries that you downloaded and installed from istio.io
ISTIO_HOME=${HOME}/istio-${ISTIO_VERSION}

# Location of Coolstore microservice demo repo
COOLSTORE_HOME=${HOME}/coolstore-microservice

The Coolstore Microservice demo uses a lot of JBoss middleware, so let’s install the necessary image stream definitions into the openshift namespace:

oc create -n openshift -f https://raw.githubusercontent.com/jboss-fuse/application-templates/master/fis-image-streams.json
oc create -n openshift -f https://raw.githubusercontent.com/jboss-openshift/application-templates/master/eap/eap70-image-stream.json
oc create -n openshift -f https://raw.githubusercontent.com/jboss-openshift/application-templates/master/webserver/jws31-tomcat8-image-stream.json
oc create -n openshift -f https://raw.githubusercontent.com/jboss-openshift/application-templates/master/openjdk/openjdk18-image-stream.json
oc create -n openshift -f https://raw.githubusercontent.com/jboss-openshift/application-templates/master/decisionserver/decisionserver64-image-stream.json

Next, create a project to house the mesh and give the default serviceaccount within the project the necessary permissions for Istio to do its thing:

oc new-project $COOLSTORE_PROJECT
oc adm policy add-scc-to-user privileged -z default
oc adm policy add-scc-to-user anyuid -z default

Now comes the big bang. The coolstore microservice demo comes with a giant OpenShift template that will create the microservices and associated databases in your new project. It will fire off a number of builds that should eventually succeed, but like many real-world projects, ours does not adhere fully to best practices for container-based microservices, so we’ll need to work around these issues. We will use oc process to convert the template into a list of Kubernetes objects, and then pass them through the Istio manual injection CLI, which will do the same thing as auto-injection, but do it outside of OpenShift itself. Finally, after passing through the manual injector (istioctl kube-inject), the results are sent to OpenShift via oc apply. You could instead capture the output and save it to your source code management system, but for demo purposes we’ll just send it directly to OpenShift. Right after deploying Coolstore, we’ll stop the deployments using oc rollout cancel to give us a chance to do some hacking before things are up and running:

oc process -f ${COOLSTORE_HOME}/openshift/coolstore-template.yaml | \
 ${ISTIO_HOME}/bin/istioctl kube-inject -f - | \
 oc apply -f -

for i in $(oc get dc -o name) ; do
  oc rollout cancel $i
  oc rollout pause $i
done

At this point, your builds should be progressing (and CPU contributing to the heat death of the universe):

% oc get builds
NAME             TYPE      FROM          STATUS    STARTED          DURATION
cart-1           Source    Git@f63f51d   Running   37 seconds ago
catalog-1        Source    Git@f63f51d   Running   38 seconds ago
coolstore-gw-1   Source    Git@f63f51d   Running   38 seconds ago
inventory-1      Source    Git@f63f51d   Running   38 seconds ago
pricing-1        Source    Git@f63f51d   Running   37 seconds ago
rating-1         Source    Git@f63f51d   Running   37 seconds ago
review-1         Source    Git@f63f51d   Running   37 seconds ago
web-ui-1         Source    Git@f63f51d   Running   38 seconds ago

You can keep running oc get builds until the STATUS column shows Complete, but you don’t have to wait for it in order to continue below.

While the builds are progressing (and deployments are cancelled), one of the first best practices this application does NOT follow is naming the service and container ports used. Istio currently requires that for services to participate in the service mesh, their exposed TCP ports must be named, and they must be named starting with http or https. Istio can only intelligently route and trace SNI or equivalent protocols such as HTTP and HTTPS that have parseable headers indicating the destination host, so by explicitly naming the service and container ports, you are confirming with Istio your service’s intention to participate. It just so happens that all the Coolstore services bind to port 8080, so let’s hack the demo and brute-force name all the service ports http for simplicity:

for i in $(oc get svc -o name) ; do
  PATCH=$(mktemp)
  cat <<EOF > $PATCH
spec:
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: http
EOF
  oc patch $i -p "$(cat $PATCH)"
  rm -f $PATCH
done

With the above code, all our services’ ports are now named (with the exception of database service ports, which you’ll see in the output are skipped). We need to do the same for our containers, and we also need to add some sleep time (see part 1 or consult your doctor for an explanation of the need for sleep). Istio’s intelligent routing can also operate on service versions, so you can do things like canary deployments or dark launches of different versions of a service. So we’ll add a version specifier of v1 for all services (later, we’ll do fun things with this). So here’s the magic hack to do all of that to our DeploymentConfigs:

for i in $(oc get dc -o name) ; do
 oc label $i version=v1
 DCNAME=$(echo $i | cut -d'/' -f 2)
 PATCH=$(mktemp)
 cat <<EOF > $PATCH
spec:
  strategy:
    customParams:
      command:
      - /bin/sh
      - '-c'
      - 'sleep 5; echo slept for 5; /usr/bin/openshift-deploy'
  template:
    metadata:
      labels:
        version: v1
    spec:
      containers:
      - name: $DCNAME
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
EOF
   oc patch $i -p "$(cat $PATCH)"
   rm -f $PATCH
done

Next, since this demo is often used on low-powered laptops, by default, we disable (scale to 0 pods) some services. But because we’re in big bang mode, let’s turn those services back on:

for i in rating rating-mongodb review review-postgresql pricing ; do
  oc scale --replicas=1 dc $i
done

We’re almost there. The next issue is with JBoss EAP. Out of the box, when the containerized JBoss EAP image starts up, it binds its Undertow listeners to a private IP address (the output of hostname -i). Due to the networking magic of Istio, it expects containers to bind to its public IP address or 0.0.0.0 (that is, all interfaces, and it’s not documented anywhere that I could find). Unfortunately for us, the bind address for JBoss EAP is fixed to always bind to a private IP address on an interface that Istio does not control or proxy, so we have to resort to another hack to work around this. This hack modifies the JBoss EAP S2I builder image, creating a derived builder image (using the container image format that shall not be named) that we then use to rebuild our JBoss EAP–based inventory microservice so that it binds to 0.0.0.0:

cat <<EOF | oc new-build --name inventory-builder -D -
  FROM registry.access.redhat.com/jboss-eap-7/eap70-openshift:1.6
  RUN sed -i 's/JBOSS_HA_ARGS="-b \${IP_ADDR}/JBOSS_HA_ARGS="-b 0.0.0.0/' /opt/eap/bin/launch/ha.sh
EOF

Wait for it to complete:

for i in {1..200}; do oc logs -f bc/inventory-builder && break || sleep 1; done

And then rebuild the inventory service using S2I:

oc new-build --name inventory-hack --to='inventory:latest' ${COOLSTORE_PROJECT}/inventory-builder~${COOLSTORE_HOME} --context-dir=inventory-service

And wait for it to complete:

for i in {1..200}; do oc logs -f bc/inventory-hack && break || sleep 1; done

One final hack to go. We are using JBoss Fuse to implement our Coolstore gateway (which fronts all of our microservices and provides aggregated data back to the UI at runtime). Unfortunately, some features of Camel (most notably the ones we are using to implement our AggregationStrategy) strip HTTP headers when making downstream calls to other services. This will interfere with proper tracing, causing the downstream calls to appear as individual traces with a single span rather than as a single trace containing aggregated spans representing the downstream calls. Proper tracing comes relatively free with the Istio+Prometheus+Jaeger combination so popular these days, and there’s a lot of value that can be extracted when tracing works, so let’s hack around this limitation by modifying the source code to the Coolstore gateway to preserve the headers.

This hack assumes you have Maven 3.3.9+ installed, because it will do an in-place edit of the source using sed, then do a local rebuild of the hacked source code with Maven, and finally kick off an OpenShift S2I binary rebuild of the service on your local machine:

sed -i.bak 's/return original;/original.getOut().setHeaders(original.getIn().getHeaders()); return original;/g' \
 $COOLSTORE_HOME/coolstore-gw/src/main/java/com/redhat/coolstore/api_gateway/ProductGateway.java

mvn -f $COOLSTORE_HOME/coolstore-gw clean package -DskipTests -Dfabric8.skip -e -B

oc new-build --name coolstore-gw-hack --to='coolstore-gw:latest' --image fis-java-openshift:2.0 --strategy source --binary

oc start-build coolstore-gw-hack --from-file=${COOLSTORE_HOME}/coolstore-gw/target/coolstore-gw.jar --follow

One other best practice that Coolstore fails to follow is to declare all of the exposed ports in containers that you wish to access. Istio’s proxy will proxy traffic only to named and declared ports, so if your containers are listening on undeclared or unnamed ports, you won’t be able to access them, even from within the running container, because Istio transparently intercepts all traffic and passes traffic only on named ports. So in this case, our Coolstore gateway has failed to declare port 8081 (the port on which its health probes are exposed), so the health checks will fail. So for now, let’s disable the Coolstore gateway health check (which in itself is a bad practice, and should be eventually fixed in Coolstore itself):

oc set probe dc/coolstore-gw --readiness --liveness --remove

The Final Step

Wow; that’s a lot of hacking. With our newly minted and Istio-ified project, it’s time to redeploy everything. We’ll use oc rollout to do this for all of our deployments:

for i in $(oc get dc -o name) ; do
  oc rollout resume $i
  oc rollout latest $i
done

Wait for the redeployments to complete:

for i in $(oc get dc -o name) ; do
  oc rollout status -w $i
done

If the above command reports timeouts, just rerun the for loop until all deployments report success.

Ordinarily at this point, once all the deployments complete, you could access the UI of Coolstore directly, but we will use the built-in Istio Ingress component so that we can fully control routing of the incoming requests and generate respectable distributed traces from the start of the requests:

cat <<EOF | oc create -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: coolstore-ingress
  annotations:
    kubernetes.io/ingress.class: "istio"
spec:
  backend:
    serviceName: web-ui
    servicePort: http
  rules:
  - http:
      paths:
      - path: /api/*
        backend:
          serviceName: coolstore-gw
          servicePort: http
EOF

This will set up Istio Ingress to route requests for /api/* to our Coolstore gateway, and all other requests will just go to the web UI front end. You will access the application through the Ingress route installed in the istio-system project (you did run oc expose svc/istio-ingress -n istio-system, right?)

After everything is rebuilt and all the deployments are successfully rolled out, you should be able to access the Coolstore UI through the Istio Ingress service URL in your browser, which you can generate with this command:

echo "CoolStore URL: http://$(oc get route istio-ingress -n ${ISTIO_PROJECT} --template='{{ .spec.host }}')"

CoolStore Web UI Screenshot

You can also access the web consoles of various services such as Prometheus and Grafana. Run these commands and then copy/paste the URLs into your browser to verify that everything is working:

echo "Primary web frontend URL: http://$(oc get route istio-ingress -n ${ISTIO_PROJECT} --template='{{ .spec.host }}')"
echo "D3 force layout service graph: http://$(oc get route servicegraph -n ${ISTIO_PROJECT} --template='{{ .spec.host }}')/force/forcegraph.html?time_horizon=5m&filter_empty=true"
echo "Example Prometheus query: http://$(oc get route prometheus -n ${ISTIO_PROJECT} --template='{{ .spec.host }}')/graph?g0.range_input=30m&g0.expr=istio_request_count&g0.tab=0"
echo "Grafana Istio Dashboard: http://$(oc get route grafana -n ${ISTIO_PROJECT} --template='{{ .spec.host }}')/d/1/istio-dashboard?refresh=5s&orgId=1"
echo "Jaeger Tracing Console: http://$(oc get route jaeger-query -n ${ISTIO_PROJECT} --template='{{ .spec.host }}')"

We will explore these in the next post, but feel free to play around, access the page, generate some load, and inspect the results.

For added fun and value, access the Web front-end URL a few times in your browser, and then check out the D3 Force Layout endpoint above. It should look something like this:

Service Graph of Coolstore screenshot

The diagram is pretty confusing, because some of the non-HTTP/S accesses (for example, to databases) aren’t properly linked to the services calling them because Istio cannot interpret these calls. But you can, for example, see that the coolstore-gw makes several downstream calls to other services (inventory, catalog, rating, cart) and these will also show up as proper traces and spans in Jaeger.

Summary and Observations

This is a quick way to see a visualization of the service dependencies and confirm that the gateway is indeed accessing the underlying microservices. Some observations:

  • You should name all of your container and service ports. You may have noticed in the hacks above we brute-force named all services on port 8080 as http, but non-HTTP services (most notably PostgreSQL and MongoDB, both of which use non-HTTP protocols) were skipped/ignored. These services cannot participate in the service mesh because Istio currently cannot route these services.
  • Containerized JBoss EAP has a limitation that we worked around.
  • JBoss Fuse has an unfortuate side effect for HTTP headers that we worked around.
  • Manual injection is most likely better than auto-injection in production because you can capture it in infrastructure source code.
    Auto-injection is great for demos (once this issue is resolved).
  • There is HUGE value in Istio+OpenShift for existing apps, whether they use microservices or not.

In the next part of this series, we’ll explore additional value you can get out of Istio+OpenShift for existing applications, and we’ll even detect and fix a few issues in our distributed microservice application using the power of Istio service mesh and Red Hat technology. Stay tuned!

More Reading

To learn more, visit our Linux containers or microservices Topic pages.

To learn more, visit our Join the Red Hat Developer Program (it’s free) and get access to related cheat sheets (e.g. containers), books (e.g. microservices), and product downloads that can help you with your microservices and/or container application development.

Share