Buzz Image

In Part 1, we talked a bit about this DevOps thing and why people won't stop talking about it. In Part 2, we'll talk about the areas where you can change your IT focus today to help benefit from DevOps.

A classic mistake is to focus primarily on the tools associated with successful DevOps shops. It's not as if you can bring up your own Deployinator and suddenly become as high-functioning an IT shop as Etsy. The tooling is important, but won't succeed if used for its own sake. The most relevant tooling is used to support a culture that can consume it effectively.

The culture really has to come first. And this isn't something that is going to get fixed just from the top down. Start reaching out to the people who are going to be impacted by the stuff you're working on. It can be as simple as a water cooler chat, or if some real truth needs to be dished out, buy your fellow nerd a beer and figure it out together. Get them plugged in to your planning work sessions so they can tell you how what you're doing will help or hinder them. Take time to burn off some steam with the people you work with. It could be a quick game of Mario Kart in the break room, or a round of libations after work. If you're not having fun together, you might want to fix that! So much goes wrong when you don't have mutual trust, so consciously work towards that!

People from three entirely different teams at Red Hat finding a few minutes in their busy day to have a good time together.

One thing to watch out for is hiding inside of your little silo, protecting your team, and forgetting about the whole point of being at the office. If you're a Sysadmin, the point isn't building servers or making sure backups are running. Yes, those things are important. But the reason you are there is to be part of putting a product or service in front of customers that will want to give your employer lots of money for the pleasure of doing business together. You're there for the same reason as the people on other teams, including HR, Legal, Finance, Change Management, QA, InfoSec, etc. If there is something that needs to be done for the company to succeed at this simple goal, and you've got the capability to help, don't worry about whether or not it's in your job description. Just do it!

As part of your change to a DevOps culture, consider certain mantras that have helped me out on the same journey. One thing to always keep in mind is that your infrastructure should be defined by code. Think of your servers as "cattle, not pets" (or also sometimes conveyed as "wheat, not houseplants"). For everybody to have a high confidence level that changes are going to work, your infrastructure has to be very consistent. Your test environments should look like production. And the only real way to do that on a sustainable basis is to define your infrastructure in software.

All of those manual things you do again and again should be automated. The processes that are the most painful should be done more often. Improve the process again and again until it's simple and routine. This will happen as the automation of the process gets more mature and robust.

How's your monitoring system working out? If you're like most shops, it's probably a pain to deal with the false alarms, and a pain to make many changes per day to what's being monitored. Are you just doing simple health checks and then spamming and/or paging too many people for every deviation from the expected norm? One of the things that good DevOps shops have in common is extensive measurement capabilities. Not just basic health checking, but recording metrics for every little thing. Record code pushes to environments in the same system. Start using these collected metrics intelligently to gain a better understanding of how your code is impacting its environment, or being impacted by its environment. Look for clever ways to notify on the non-linear changes to your metrics (those are the ones that more often can give you an early warning of the kinds of things you'd normally only catch much later in a failed healthcheck). While monitoring relentlessly is helpful, notifications should be sent only when really needed. Better yet, figure out how to allow self-service notification settings. Some people will want emails, some will want texts. Try to be reasonably accommodating.

Take those collected metrics and paint your walls with them. You'll see TV's up on the walls all over in DevOps shops. Team pods sometimes look like little NOCs, with two or more TV sets up on the wall cycling through everything from the team's Agile board to site metrics, cloud resource scaling, and recent Continuous Integration events. Keep it useful, and continue improving on it. Inception has been using RNOC. We've already successfully made use of small and fun bounties to encourage improvements.

Mature DevOps shops share their successes and their failures. And failures tend to happen in blame-free ways. Find time and opportunities to demonstrate the things you're working on, or what you've learned, to your colleagues, your customers, and to the broader community. This will really help to grow a collaborative culture. Consider continually extending invitations to others in your organization to take more of an interest in what you're working on. Often, they will welcome the opportunity help you to help them succeed at the goal you all share.

I'm just going to leave this here. All of this talk about culture, automation, measurement, and sharing is often presented in DevOps circles under the acronym CAMS.

You'll hear all of these manager types and six sigma blackbelts, project managers, and others laying down a torrent of Japanese words that are all different ways of saying the same thing: waste. Without diving into a study in creative use of Japanese language in English speaking countries, it suffices to say that one other aspect of mature DevOps cultures is allergy to waste in all of its forms. Pushing bad code downstream to the Sysadmins is waste. Long running feature branches are waste. Having way more infrastructure than you need because it's too cumbersome to scale it up and down on demand is waste. Technical debt tends to turn into waste. Don't be afraid to call it out, draw attention to it, and try to sell your colleagues on a path to redemption from it. That's all an aspect of continuous improvement (represented by yet another Japanese word which we'll not propagate here). It's actually okay to say "continuous improvement" in plain English.

Here at Red Hat IT, we're taking a very deliberate path towards building a DevOps culture. One of the things we had to do was figure out what DevOps means for us. We talked about it over many beers, and something really resonated with us: for all the talk about Agile and Lean and all that, DevOps can also claim a lot of its heritage comes from Open Source culture. The Open Source culture is what made us great. It's our place of strength. We know this. Saying it out loud was a game changer for us.

So in the office, amongst each other, we try to downplay the word "DevOps". We even say it mockingly to one another, like it's something to be evoked reverently, as it carries with it some dark and powerful magic. Really, what we're doing is rediscovering our Open Source roots within IT, and getting better in our practice of it. And it looks and smells and tastes like DevOps. So just between you and me, we'll call it DevOps. Don't tell anyone else I told you that.

Last updated: March 15, 2023