Back in December, I shared with you how Red Hat IT was beginning its DevOps transformation. That early post discussed our strategy, team composition, the importance of Open Source, and incremental change. It's been about three months and I'd like to share an update with some early lessons learned.
Overall impression so far? This is harder than we thought.
To recap, our approach to jump-starting a DevOps transformation in Red Hat IT was to form a small, temporary team inside Red Hat IT made up of six individuals from development, ops, security, and program management backgrounds. We'd have 18-24 months to influence change, build tools, coach teams, facilitate collaboration, and then go back to our "normal" jobs in Red Hat IT. Today, we're about four months in.
At the start, I thought, "awesome--we have a top-notch team of four engineers and a product owner, freed up to just fix stuff and connect teams as their day job. We will be off to the races faster than you can say 'DevOps' and have a string of major successes under our belt by Valentine's Day."
The reality, however, was that the six of us had never worked together and only half had significant experience working on an Agile team. And, we had a very broad mission to start with. It took three months to start performing anywhere near the team's collective potential. Chalk up my unrealistic expectations to only having led teams that I slowly grew or inherited. But, I'll share here some lessons learned that I think should help anyone approaching a similar mission of enterprise DevOps transformation.
1. It takes time to form a new team. In reflection, it took a lot of work just to get to the starting line. Our biggest challenge was the human one: to get over being polite and to begin to trust and then challenge one another. And, because of the unique nature of our mission, we had to develop outside relationships--as a team--with our stakeholders. Plus, what was our mission, and what do we need to do to accomplish it? We all had very different ideas on what "enable DevOps for Red Hat IT" actually meant. Last, we had all the "new team" logistics to sort out (meeting times, seating arrangements, electronic communications, et cetera ad nauseam). All of this took a deceptively high amount of time and effort.
2. "Transition our enterprise IT department to DevOps" is a huge problem set. It's a serious and ongoing effort to decompose that into actual work you can take on. At our start, our team members had wildly different opinions on what that work should be, ranging from "build a platform-agnostic version of Netflix OSS' Asgard to run on OpenStack, etc." to "fundamentally change the way departments like Marketing, Engineering, Sales, and IT collaborate with one another."
To begin to dissect the problem, we collected data. Lots of it. Mostly, we simply asked our few-hundred colleagues "what's not working for you in IT today?" across several channels: an in-person "complaint fest," a video conference session for our friends overseas, copious discussion threads on our intranet site, and simple in-person meetings with other teams and individuals. This generated mountains of input. It took weeks of analysis, debate, and iterative planning to distill that down to an initial "theme" that our team could all sufficiently agree to and rally behind.
(FWIW, that theme is to automate Red Hat IT's RPM-based software deployments to all environments--Dev, QA, Stage, Prod--focusing first on a single product from a single development team as our proof-of-concept. And, to lead by example and visibly demonstrate DevOps values like collaboration, openness, waste reduction, and engineering rigor while doing that. But that deserves a blog post of its own.)
While we worked to collect and distill all that input, we simultaneously focused on "quick wins"--building small software tools to help out other IT teams wherever we could. So, we still got some valuable, usable nuggets from this phase of our team's development: an open source tool for developers to self-service inspect the state of machines to which they can't login, jsonstats / talook, and the git-branch-blacklist utility, both huge time / interrupt / headache savers for several of our IT teams.
3. Language differences are an artifact of a siloed culture, and present a hurdle for DevOps. The Inception team in Red Hat IT is diverse by design, with members from development, operations, security, and program management backgrounds. We discovered early (but not right away) that even between our few team members, the same terms often had different meaning. In one grooming session, two team members--one sysadmin, one infosec wonk--spent 10 minutes talking past each other because to one of them, the term "full stack" included the load balancers; to the other, it didn't.
If we are having these communication issues with our small, focused, collocated team of native English speakers, then how much is lost in translation when teams send each other tickets at a rate of dozens or hundreds per week in a mid-sized IT shop? How many hours are lost, and customer-impacting errors are made, due to a lack of common language between teams?
With these language differences, how on Earth are we going to encourage better communication between teams that each work in their own domain, and span the globe? Red Hat has offices in 33 countries, and IT has a presence in many of these.
I think the answer is actually right there in the question: we encourage better communication between teams, and the semantic differences of words eventually resolve themselves. Maybe other organizations have solved this effectively with some master glossary (I'm skeptical) but I'm pretty sure this can only be solved by building stronger channels of communication between teams (though I'd love a counterexample--tweet me!)
In summary, if you're a leader--an architect, a tech lead, an operations manager, a CTO--embarking on a DevOps transformation project in your own enterprise IT shop, give yourself a little space to form your team, break down your mission, and get everyone speaking the same language. You'll be happier and more successful for it.
Last updated: January 10, 2023