Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Bringing Coolstore Microservices to the Service Mesh: Part 2--Manual Injection

 

April 12, 2018
James Falkner
Related topics:
KubernetesMicroservicesService Mesh
Related products:
Red Hat OpenShift

Share:

    Coolstore+Istio Logo

    In the first part of this series we explored the Istio project and how Red Hat is committed to and actively involved in the project and working to integrate it into Kubernetes and OpenShift to bring the benefits of a service mesh to our customers and the wider communities involved. If you want to play with Istio, check out the service mesh tutorials on learn.openshift.com. If you want to install it, follow the Istio Kubernetes quickstart instructions and install it on OpenShift 3.7 or later. Also don't miss Don Schenck's series of blogs on Istio technology in general to learn more about it and what Red Hat is doing in this space.

    In this post, we will deploy the existing Coolstore microservices demo as a service mesh and start to demonstrate the tangible value you can get out of the system without any major rewrite or rearchitecture of the existing app. We'll also improve our project along the way to adhere to Istio (and general microservice) best practices. In the real world, your applications and developers often make bad assumptions or fail to implement best practices, so with this information you can learn something about your own projects. For Coolstore, many of these workarounds will eventually find their way into the source code of the demo.

    Getting Started

    Let's assume you already have OpenShift 3.7+ installed (I am using OpenShift Origin 3.9.0.alpha3 because at press time, OpenShift Container Platform 3.9 has not yet been released). Let's further assume you've installed Istio 0.6.0 or later, including the Prometheus, Servicegraph, Jaeger, and Grafana plug-ins  (see the shell script installer in the More Reading section below for instructions on how to install istio and its plug-ins). Verify Istio is installed and running in the istio-system namespace:

    % oc get pods -n istio-system
    NAME                                READY     STATUS    RESTARTS   AGE
    grafana-89f97d9c-2wtxx              1/1       Running   0          7m
    istio-ca-59f6dcb7d9-bs9hx           1/1       Running   0          7m
    istio-ingress-779649ff5b-jw4hg      1/1       Running   0          7m
    istio-mixer-7f4fd7dff-pct4j         3/3       Running   0          7m
    istio-pilot-5f5f76ddc8-pvq4d        2/2       Running   0          7m
    jaeger-deployment-559c8b9b8-klspv   1/1       Running   0          7m
    prometheus-cf8456855-svpmb          1/1       Running   0          7m
    servicegraph-59ff5dbbff-7zrfp       1/1       Running   0          7m
    

    Let's get started. Clone the Coolstore repo and then play along:

    % git clone https://github.com/jbossdemocentral/coolstore-microservice

    And make sure you are logged in as a cluster administrator, or you have cluster-admin privileges, since it'll require you to make some policy and permission changes later on. (As mentioned in part 1, this will be fine-tuned in future Istio releases to not require so much privilege and permission.) You can do this by logging in as a cluster administrator, or if you have sudoer privileges, you can add --as=system:admin to all the oc commands below.

    Manual Injection of Sidecars

    In part 1, I showed that with sidecar auto-injection, your app's pods are automatically festooned with Envoy proxies without ever having to change the application's deployments. However, there are issues with it that currently prevent us from using it. For now, we'll do manual injection. Manually injecting has the obvious drawback that you have to do the injection, but it has a big benefit as well: you need to do it only once (per release of Istio), and once it's done you can check in the results to your source code management system—infrastructure as code and all that.

    Let's first set a few environment variables in our Linux shell so we can reference them later on. (Just copy and paste all of these commands into your Linux terminal. If you're on Windows, there is hope for you.)

    # Version of Istio we are using
    ISTIO_VERSION=0.6.0
    
    # The name of the OpenShift project into which you installed Istio. It should
    # be istio-system as that's what the Istio nstaller creates for you
    ISTIO_PROJECT=istio-system
    
    # Name of project to house our coolstore service mesh
    COOLSTORE_PROJECT=coolstore-mesh
    
    # Location of istio binaries that you downloaded and installed from istio.io
    ISTIO_HOME=${HOME}/istio-${ISTIO_VERSION}
    
    # Location of Coolstore microservice demo repo
    COOLSTORE_HOME=${HOME}/coolstore-microservice
    

    The Coolstore Microservice demo uses a lot of JBoss middleware, so let's install the necessary image stream definitions into the openshift namespace:

    oc create -n openshift -f https://raw.githubusercontent.com/jboss-fuse/application-templates/master/fis-image-streams.json
    oc create -n openshift -f https://raw.githubusercontent.com/jboss-openshift/application-templates/master/eap/eap70-image-stream.json
    oc create -n openshift -f https://raw.githubusercontent.com/jboss-openshift/application-templates/master/webserver/jws31-tomcat8-image-stream.json
    oc create -n openshift -f https://raw.githubusercontent.com/jboss-openshift/application-templates/master/openjdk/openjdk18-image-stream.json
    oc create -n openshift -f https://raw.githubusercontent.com/jboss-openshift/application-templates/master/decisionserver/decisionserver64-image-stream.json

    Next, create a project to house the mesh and give the default serviceaccount within the project the necessary permissions for Istio to do its thing:

    oc new-project $COOLSTORE_PROJECT
    oc adm policy add-scc-to-user privileged -z default
    oc adm policy add-scc-to-user anyuid -z default

    Now comes the big bang. The coolstore microservice demo comes with a giant OpenShift template that will create the microservices and associated databases in your new project. It will fire off a number of builds that should eventually succeed, but like many real-world projects, ours does not adhere fully to best practices for container-based microservices, so we'll need to work around these issues. We will use oc process to convert the template into a list of Kubernetes objects, and then pass them through the Istio manual injection CLI, which will do the same thing as auto-injection, but do it outside of OpenShift itself. Finally, after passing through the manual injector (istioctl kube-inject), the results are sent to OpenShift via oc apply. You could instead capture the output and save it to your source code management system, but for demo purposes we'll just send it directly to OpenShift. Right after deploying Coolstore, we'll stop the deployments using oc rollout cancel to give us a chance to do some hacking before things are up and running:

    oc process -f ${COOLSTORE_HOME}/openshift/coolstore-template.yaml | \
     ${ISTIO_HOME}/bin/istioctl kube-inject -f - | \
     oc apply -f -
    
    for i in $(oc get dc -o name) ; do
      oc rollout cancel $i
      oc rollout pause $i
    done
    

    At this point, your builds should be progressing (and CPU contributing to the heat death of the universe):

    % oc get builds
    NAME             TYPE      FROM          STATUS    STARTED          DURATION
    cart-1           Source    Git@f63f51d   Running   37 seconds ago
    catalog-1        Source    Git@f63f51d   Running   38 seconds ago
    coolstore-gw-1   Source    Git@f63f51d   Running   38 seconds ago
    inventory-1      Source    Git@f63f51d   Running   38 seconds ago
    pricing-1        Source    Git@f63f51d   Running   37 seconds ago
    rating-1         Source    Git@f63f51d   Running   37 seconds ago
    review-1         Source    Git@f63f51d   Running   37 seconds ago
    web-ui-1         Source    Git@f63f51d   Running   38 seconds ago
    

    You can keep running oc get builds until the STATUS column shows Complete, but you don't have to wait for it in order to continue below.

    While the builds are progressing (and deployments are cancelled), one of the first best practices this application does NOT follow is naming the service and container ports used. Istio currently requires that for services to participate in the service mesh, their exposed TCP ports must be named, and they must be named starting with http or https. Istio can only intelligently route and trace SNI or equivalent protocols such as HTTP and HTTPS that have parseable headers indicating the destination host, so by explicitly naming the service and container ports, you are confirming with Istio your service's intention to participate. It just so happens that all the Coolstore services bind to port 8080, so let's hack the demo and brute-force name all the service ports http for simplicity:

    for i in $(oc get svc -o name) ; do
      PATCH=$(mktemp)
      cat <<EOF > $PATCH
    spec:
      ports:
      - name: http
        port: 8080
        protocol: TCP
        targetPort: http
    EOF
      oc patch $i -p "$(cat $PATCH)"
      rm -f $PATCH
    done

    With the above code, all our services' ports are now named (with the exception of database service ports, which you'll see in the output are skipped). We need to do the same for our containers, and we also need to add some sleep time (see part 1 or consult your doctor for an explanation of the need for sleep). Istio's intelligent routing can also operate on service versions, so you can do things like canary deployments or dark launches of different versions of a service. So we'll add a version specifier of v1 for all services (later, we'll do fun things with this). So here's the magic hack to do all of that to our DeploymentConfigs:

    for i in $(oc get dc -o name) ; do
     oc label $i version=v1
     DCNAME=$(echo $i | cut -d'/' -f 2)
     PATCH=$(mktemp)
     cat <<EOF > $PATCH
    spec:
      strategy:
        customParams:
          command:
          - /bin/sh
          - '-c'
          - 'sleep 5; echo slept for 5; /usr/bin/openshift-deploy'
      template:
        metadata:
          labels:
            version: v1
        spec:
          containers:
          - name: $DCNAME
            ports:
            - containerPort: 8080
              name: http
              protocol: TCP
    EOF
       oc patch $i -p "$(cat $PATCH)"
       rm -f $PATCH
    done

    Next, since this demo is often used on low-powered laptops, by default, we disable (scale to 0 pods) some services. But because we're in big bang mode, let's turn those services back on:

    for i in rating rating-mongodb review review-postgresql pricing ; do
      oc scale --replicas=1 dc $i
    done

    We're almost there. The next issue is with JBoss EAP. Out of the box, when the containerized JBoss EAP image starts up, it binds its Undertow listeners to a private IP address (the output of hostname -i). Due to the networking magic of Istio, it expects containers to bind to its public IP address or 0.0.0.0 (that is, all interfaces, and it's not documented anywhere that I could find). Unfortunately for us, the bind address for JBoss EAP is fixed to always bind to a private IP address on an interface that Istio does not control or proxy, so we have to resort to another hack to work around this. This hack modifies the JBoss EAP S2I builder image, creating a derived builder image (using the container image format that shall not be named) that we then use to rebuild our JBoss EAP–based inventory microservice so that it binds to 0.0.0.0:

    cat <<EOF | oc new-build --name inventory-builder -D -
      FROM registry.access.redhat.com/jboss-eap-7/eap70-openshift:1.6
      RUN sed -i 's/JBOSS_HA_ARGS="-b \${IP_ADDR}/JBOSS_HA_ARGS="-b 0.0.0.0/' /opt/eap/bin/launch/ha.sh
    EOF

    Wait for it to complete:

    for i in {1..200}; do oc logs -f bc/inventory-builder && break || sleep 1; done

    And then rebuild the inventory service using S2I:

    oc new-build --name inventory-hack --to='inventory:latest' ${COOLSTORE_PROJECT}/inventory-builder~${COOLSTORE_HOME} --context-dir=inventory-service

    And wait for it to complete:

    for i in {1..200}; do oc logs -f bc/inventory-hack && break || sleep 1; done

    One final hack to go. We are using JBoss Fuse to implement our Coolstore gateway (which fronts all of our microservices and provides aggregated data back to the UI at runtime). Unfortunately, some features of Camel (most notably the ones we are using to implement our AggregationStrategy) strip HTTP headers when making downstream calls to other services. This will interfere with proper tracing, causing the downstream calls to appear as individual traces with a single span rather than as a single trace containing aggregated spans representing the downstream calls. Proper tracing comes relatively free with the Istio+Prometheus+Jaeger combination so popular these days, and there's a lot of value that can be extracted when tracing works, so let's hack around this limitation by modifying the source code to the Coolstore gateway to preserve the headers.

    This hack assumes you have Maven 3.3.9+ installed, because it will do an in-place edit of the source using sed, then do a local rebuild of the hacked source code with Maven, and finally kick off an OpenShift S2I binary rebuild of the service on your local machine:

    sed -i.bak 's/return original;/original.getOut().setHeaders(original.getIn().getHeaders()); return original;/g' \
     $COOLSTORE_HOME/coolstore-gw/src/main/java/com/redhat/coolstore/api_gateway/ProductGateway.java
    
    mvn -f $COOLSTORE_HOME/coolstore-gw clean package -DskipTests -Dfabric8.skip -e -B
    
    oc new-build --name coolstore-gw-hack --to='coolstore-gw:latest' --image fis-java-openshift:2.0 --strategy source --binary
    
    oc start-build coolstore-gw-hack --from-file=${COOLSTORE_HOME}/coolstore-gw/target/coolstore-gw.jar --follow
    

    One other best practice that Coolstore fails to follow is to declare all of the exposed ports in containers that you wish to access. Istio's proxy will proxy traffic only to named and declared ports, so if your containers are listening on undeclared or unnamed ports, you won't be able to access them, even from within the running container, because Istio transparently intercepts all traffic and passes traffic only on named ports. So in this case, our Coolstore gateway has failed to declare port 8081 (the port on which its health probes are exposed), so the health checks will fail. So for now, let's disable the Coolstore gateway health check (which in itself is a bad practice, and should be eventually fixed in Coolstore itself):

    oc set probe dc/coolstore-gw --readiness --liveness --remove

    The Final Step

    Wow; that's a lot of hacking. With our newly minted and Istio-ified project, it's time to redeploy everything. We'll use oc rollout to do this for all of our deployments:

    for i in $(oc get dc -o name) ; do
      oc rollout resume $i
      oc rollout latest $i
    done

    Wait for the redeployments to complete:

    for i in $(oc get dc -o name) ; do
      oc rollout status -w $i
    done

    If the above command reports timeouts, just rerun the for loop until all deployments report success.

    Ordinarily at this point, once all the deployments complete, you could access the UI of Coolstore directly, but we will use the built-in Istio Ingress component so that we can fully control routing of the incoming requests and generate respectable distributed traces from the start of the requests:

    cat <<EOF | oc create -f -
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: coolstore-ingress
      annotations:
        kubernetes.io/ingress.class: "istio"
    spec:
      backend:
        serviceName: web-ui
        servicePort: http
      rules:
      - http:
          paths:
          - path: /api/*
            backend:
              serviceName: coolstore-gw
              servicePort: http
    EOF

    This will set up Istio Ingress to route requests for /api/* to our Coolstore gateway, and all other requests will just go to the web UI front end. You will access the application through the Ingress route installed in the istio-system project (you did run oc expose svc/istio-ingress -n istio-system, right?)

    After everything is rebuilt and all the deployments are successfully rolled out, you should be able to access the Coolstore UI through the Istio Ingress service URL in your browser, which you can generate with this command:

    echo "CoolStore URL: http://$(oc get route istio-ingress -n ${ISTIO_PROJECT} --template='{{ .spec.host }}')"

    CoolStore Web UI Screenshot

    You can also access the web consoles of various services such as Prometheus and Grafana. Run these commands and then copy/paste the URLs into your browser to verify that everything is working:

    echo "Primary web frontend URL: http://$(oc get route istio-ingress -n ${ISTIO_PROJECT} --template='{{ .spec.host }}')"
    echo "D3 force layout service graph: http://$(oc get route servicegraph -n ${ISTIO_PROJECT} --template='{{ .spec.host }}')/force/forcegraph.html?time_horizon=5m&filter_empty=true"
    echo "Example Prometheus query: http://$(oc get route prometheus -n ${ISTIO_PROJECT} --template='{{ .spec.host }}')/graph?g0.range_input=30m&g0.expr=istio_request_count&g0.tab=0"
    echo "Grafana Istio Dashboard: http://$(oc get route grafana -n ${ISTIO_PROJECT} --template='{{ .spec.host }}')/d/1/istio-dashboard?refresh=5s&orgId=1"
    echo "Jaeger Tracing Console: http://$(oc get route jaeger-query -n ${ISTIO_PROJECT} --template='{{ .spec.host }}')"

    We will explore these in the next post, but feel free to play around, access the page, generate some load, and inspect the results.

    For added fun and value, access the Web front-end URL a few times in your browser, and then check out the D3 Force Layout endpoint above. It should look something like this:

    Service Graph of Coolstore screenshot

    The diagram is pretty confusing, because some of the non-HTTP/S accesses (for example, to databases) aren't properly linked to the services calling them because Istio cannot interpret these calls. But you can, for example, see that the coolstore-gw makes several downstream calls to other services (inventory, catalog, rating, cart) and these will also show up as proper traces and spans in Jaeger.

    Summary and Observations

    This is a quick way to see a visualization of the service dependencies and confirm that the gateway is indeed accessing the underlying microservices. Some observations:

    • You should name all of your container and service ports. You may have noticed in the hacks above we brute-force named all services on port 8080 as http, but non-HTTP services (most notably PostgreSQL and MongoDB, both of which use non-HTTP protocols) were skipped/ignored. These services cannot participate in the service mesh because Istio currently cannot route these services.
    • Containerized JBoss EAP has a limitation that we worked around.
    • JBoss Fuse has an unfortuate side effect for HTTP headers that we worked around.
    • Manual injection is most likely better than auto-injection in production because you can capture it in infrastructure source code.
      Auto-injection is great for demos (once this issue is resolved).
    • There is HUGE value in Istio+OpenShift for existing apps, whether they use microservices or not.

    In the next part of this series, we'll explore additional value you can get out of Istio+OpenShift for existing applications, and we'll even detect and fix a few issues in our distributed microservice application using the power of Istio service mesh and Red Hat technology. Stay tuned!

    More Reading

    • Bringing Coolstore microservices to the Service Mesh Part 1: Automatic Injection
    • A runnable shell script gist of all of the above hacks including Istio installation
    • Christian Posta on Istio
    • Introduction to Istio; It Makes A Mesh Of Things, by Don Schenck
    • Red Hat Developer Blogs on Istio
    Last updated: May 30, 2023

    Recent Posts

    • How to build a Model-as-a-Service platform

    • How Quarkus works with OpenTelemetry on OpenShift

    • Our top 10 articles of 2025 (so far)

    • The benefits of auto-merging GitHub and GitLab repositories

    • Supercharging AI isolation: microVMs with RamaLama & libkrun

    What’s up next?

     

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue