Part three of a four-part series on DevOps with Gene Kim and Red Hat IT.
- Part 1: Getting DevOps Off the Ground
- Part 2: The Importance of Partnership in DevOps
- Part 3: A DevOps Implementation Strategy
- Part 4: DevOps Successes and Failures
Panelists:
- Bill Montgomery: Manager, Red Hat IT
- Steve Milner: Engineer, Red Hat IT
- Jen Krieger: Product Owner, Red Hat IT
- Tim Bielawa: Engineer, Red Hat IT
- Chris Murphy: Engineer, Red Hat IT
Introduction: Gene Kim, award-winning CTO and co-author of “The Phoenix Project,” recently sat down with Red Hat IT’s Inception team to discuss their DevOps mission. Here are the highlights from the conversation.
On the Inception team’s DevOps and CI/CD implementation strategy:
Gene: Why did you decide to first focus on release automation?
Jen: We spent a few months doing research, trying to identify the best improvement that would benefit the most number of developer teams. At first, we spent a lot of time talking with people across IT about their woes in group and individual interviews. We tried to document and find correlations across teams, had hallway conversations, and attended other team’s stand‑ups.
Another thing our team did was read Jez Humble and David Farley's Continuous Delivery book. 
It reads like a Bible for everything we want to do, and in the very last chapter is their CD maturity model. We used that maturity model as a survey and asked IT, "Where do you think your team is in the maturity scale?" While not very scientific, the survey gave us the data we needed to confirm what we were hearing verbally. We are struggling in two theme areas; Environments and Deployment and Test Automation.
After establishing that pattern at the department level, we asked our partner team, “what do you want us to focus on?” They said, "Please help us deploy our code automatically," and that is essentially how we got to that decision.
Gene: So, why service buses? Why are they the bottleneck?
Bill: Say a Red Hat customer buys a RHEL subscription. The PO comes through the order system, the entitlement system activates it, there's another system so you can download the binaries, and yet another to handle support calls against your newly activated subscription. Those are all discrete systems, but all the communications flow through the ESB and SOA services tier. If you want to make a change to any of those systems, there's often going to be a change to the ESB or SOA services tier.
Steve, Engineer: And, if we can help automate something so core to our processes like our SOA tier and ESB, we should be able to help every other group within IT. If we were to start with a one‑off team, small application, that might be a nice proof of concept, but that doesn't really prove this is possible in the other teams.
Bill: Also, the SOA & ESB team is one of our more advanced dev teams as far as their cadence, agile practices, and testing methodologies. Plus, very simply, they reached out to us during the formation period of the team and said, "Hey, we're starting to get a sense for what your mission is, and we want to partner with you." That was a big part of it.
Gene: This is awesome. When do you think you'll make a material improvement on the change lead time for the SOA & ESB team?
Jen: Starting in July, when we roll out the first version of Release Engine. The plan is to target release automation to QA, and I suspect it will be easy to automate to Stage at that point. After that, we will focus on automating deployment to Dev, which we expect will expose additional technical debt that will need to be resolved. Then we’ll talk about how we're going to automate deployment to production.
The reason I'm suggesting that order is because our Dev environments are not always consistent with our other environments. I would personally feel more comfortable if we had a stable code deployment pipeline on the backend of the process before releasing to production. The challenge will be that we need to have conversations with developers on how they're going to work together to make they aren’t bumping into each other. I suspect there is work hidden in that process as well.
There is also a change management conversation that has to occur, and that has to be agreed upon by the management community here before we can automate code deployment to production; another very challenging thing to orchestrate. I suspect some teams will want to vary their process from others.
Bill: This intersects our existing ITIL change management processes. I'm more optimistic than Jen on how challenging it'll be to get agreement there.
Jen: I'm the organizational change management realist.
Bill: --and I'm the optimist and cheerleader.
[laughter]
Gene: This is actually fascinating. Can you approximate the expected change lead time improvement from before and after, your July milestone?
Jen: Right now, from the point that code arrives to QA to the point that it goes to production. It's about an average of 7.62‑something days.
Bill: --approximately. [laughs]
Jen: About a week. I would like to see that reduced, as in “a day.” Like I was saying earlier, I am guessing that when we have the Release Engine in place, we're going to discover that the majority of that change lead time delay is due to manual testing.
Next week, Gene asks the Inception team about their early wins and failures with DevOps at Red Hat.
Gene Kim is hosting the DevOps Enterprise Summit on October 21-23, where more stories will be told about DevOps transformations in large, complex organizations. Learn more about the summit and submit your own talk here!
Read more blogs from the Inception Team and follow us on Twitter.
Last updated: January 11, 2023