DevOps in Straight English – Part 2 of 2

In Part 1, we talked a bit about this DevOps thing and why people won’t stop talking about it. In Part 2, we’ll talk about the areas where you can change your IT focus today to help benefit from DevOps.

A classic mistake is to focus primarily on the tools associated with successful DevOps shops. It’s not as if you can bring up your own Deployinator and suddenly become as high-functioning an IT shop as Etsy. The tooling is important, but won’t succeed if used for its own sake. The most relevant tooling is used to support a culture that can consume it effectively.

The culture really has to come first. And this isn’t something that is going to get fixed just from the top down. Start reaching out to the people who are going to be impacted by the stuff you’re working on. It can be as simple as a water cooler chat, or if some real truth needs to be dished out, buy your fellow nerd a beer and figure it out together. Get them plugged in to your planning work sessions so they can tell you how what you’re doing will help or hinder them. Take time to burn off some steam with the people you work with. It could be a quick game of Mario Kart in the break room, or a round of libations after work. If you’re not having fun together, you might want to fix that! So much goes wrong when you don’t have mutual trust, so consciously work towards that!


People from three entirely different teams at Red Hat finding a few minutes in their busy day to have a good time together.

One thing to watch out for is hiding inside of your little silo, protecting your team, and forgetting about the whole point of being at the office. If you’re a Sysadmin, the point isn’t building servers or making sure backups are running. Yes, those things are important. But the reason you are there is to be part of putting a product or service in front of customers that will want to give your employer lots of money for the pleasure of doing business together. You’re there for the same reason as the people on other teams, including HR, Legal, Finance, Change Management, QA, InfoSec, etc. If there is something that needs to be done for the company to succeed at this simple goal, and you’ve got the capability to help, don’t worry about whether or not it’s in your job description. Just do it!

As part of your change to a DevOps culture, consider certain mantras that have helped me out on the same journey. One thing to always keep in mind is that your infrastructure should be defined by code. Think of your servers as “cattle, not pets” (or also sometimes conveyed as “wheat, not houseplants”). For everybody to have a high confidence level that changes are going to work, your infrastructure has to be very consistent. Your test environments should look like production. And the only real way to do that on a sustainable basis is to define your infrastructure in software.

All of those manual things you do again and again should be automated. The processes that are the most painful should be done more often. Improve the process again and again until it’s simple and routine. This will happen as the automation of the process gets more mature and robust.

How’s your monitoring system working out? If you’re like most shops, it’s probably a pain to deal with the false alarms, and a pain to make many changes per day to what’s being monitored. Are you just doing simple health checks and then spamming and/or paging too many people for every deviation from the expected norm? One of the things that good DevOps shops have in common is extensive measurement capabilities. Not just basic health checking, but recording metrics for every little thing. Record code pushes to environments in the same system. Start using these collected metrics intelligently to gain a better understanding of how your code is impacting its environment, or being impacted by its environment. Look for clever ways to notify on the non-linear changes to your metrics (those are the ones that more often can give you an early warning of the kinds of things you’d normally only catch much later in a failed healthcheck). While monitoring relentlessly is helpful, notifications should be sent only when really needed. Better yet, figure out how to allow self-service notification settings. Some people will want emails, some will want texts. Try to be reasonably accommodating.

Take those collected metrics and paint your walls with them. You’ll see TV’s up on the walls all over in DevOps shops. Team pods sometimes look like little NOCs, with two or more TV sets up on the wall cycling through everything from the team’s Agile board to site metrics, cloud resource scaling, and recent Continuous Integration events. Keep it useful, and continue improving on it. Inception has been using RNOC. We’ve already successfully made use of small and fun bounties to encourage improvements.

Mature DevOps shops share their successes and their failures. And failures tend to happen in blame-free ways. Find time and opportunities to demonstrate the things you’re working on, or what you’ve learned, to your colleagues, your customers, and to the broader community. This will really help to grow a collaborative culture. Consider continually extending invitations to others in your organization to take more of an interest in what you’re working on. Often, they will welcome the opportunity help you to help them succeed at the goal you all share.

I’m just going to leave this here. All of this talk about culture, automation, measurement, and sharing is often presented in DevOps circles under the acronym CAMS.

You’ll hear all of these manager types and six sigma blackbelts, project managers, and others laying down a torrent of Japanese words that are all different ways of saying the same thing: waste. Without diving into a study in creative use of Japanese language in English speaking countries, it suffices to say that one other aspect of mature DevOps cultures is allergy to waste in all of its forms. Pushing bad code downstream to the Sysadmins is waste. Long running feature branches are waste. Having way more infrastructure than you need because it’s too cumbersome to scale it up and down on demand is waste. Technical debt tends to turn into waste. Don’t be afraid to call it out, draw attention to it, and try to sell your colleagues on a path to redemption from it. That’s all an aspect of continuous improvement (represented by yet another Japanese word which we’ll not propagate here). It’s actually okay to say “continuous improvement” in plain English.

Here at Red Hat IT, we’re taking a very deliberate path towards building a DevOps culture. One of the things we had to do was figure out what DevOps means for us. We talked about it over many beers, and something really resonated with us: for all the talk about Agile and Lean and all that, DevOps can also claim a lot of its heritage comes from Open Source culture. The Open Source culture is what made us great. It’s our place of strength. We know this. Saying it out loud was a game changer for us.

So in the office, amongst each other, we try to downplay the word “DevOps”. We even say it mockingly to one another, like it’s something to be evoked reverently, as it carries with it some dark and powerful magic. Really, what we’re doing is rediscovering our Open Source roots within IT, and getting better in our practice of it. And it looks and smells and tastes like DevOps. So just between you and me, we’ll call it DevOps. Don’t tell anyone else I told you that.


Join the Red Hat Developer Program (it’s free) and get access to related cheat sheets, books, and product downloads.

 

Share

  1. This is a great piece Magnus. IT culture has to change. I agree it can’t be top down, but middle management needs to be invested in the shift to devops. The management team is responsible for rewarding behaviour and more importantly, is accountable for hiring people. I like the ground up approach, but accelerating the revolution will require the management team to be involved!

    1. Chris, great point, probably worthy of a whole blog post. I’ve also observed middle management and first-line managers as the most squeezed, and therefore challenged to adopt DevOps. (Disclaimer, I’m one of those managers).

      Senior managers / execs are responsible for a longer term view, so it’s easy for them to see where short-term investments in teams’ self-improvement will pay off. The men and women on the keyboards feel the pain of organizational silos and clunky processes directly, so they usually get the value of DevOps with little convincing.

      Middle managers, however, are squeezed between demanding customers with impossible deadlines and frustrated delivery teams. This is the group that most has to step up and fight for their teams to have the time to work on themselves and their inter-team relationships. This requires guts. As DevOps champions, we can help middle managers by providing talking points, success anecdotes, and real data for them to use in negotiations with their customers.

      1. I also think middle managers are more prone to wanting to protect their current real-estate. So if they manage developers or system admins they want to prevent their team being pulled apart into other teams. This could be well intentioned because they feel they need the staff or they want to preserve a certain dynamic. Or they could just not want to lose the number of direct reports they currently have.

        As they see walls coming down between different roles or teams they might (irrationally) fear that their value is lessened.

        1. Great point, Michael. You describe the same behaviors that created the silos DevOps attempts to tear down.

          My vision for DevOps in the enterprise is that everyone gets to keep their functional teams, they just have to evolve how they work and interact. But, many DevOps pundits advocate for a “no-ops” approach, where “DBA” or “Network Engineer” is a specialization within a team, not a functional group.

          While I think “no-ops” is a good approach in certain cases (e.g. for a web startup running on AWS), the association with DevOps can strike (very rational) fear in a functional manager in a more typical organization. Not helpful for DevOps in the enterprise.

  2. I really appreciate the insight here Magnus. We are going through this exercise now to define “what devops means to us”. The tendency I have seen early on is for folks to want to throw tools around and say “hey, now we are devops”, but I completely agree its a cultural shift that happens first, and along the way your automation and orchestration tools come together (or start to integrate better, because chances are you already have some pretty good ones in place). I think the libations and friendly chats over a ping pong table definitely help. Thanks for the article!

  3. Transparency. A search for truth. Team work and the whole community and the whole team being greater than any one of us or one company. Those are fundamental drives. It’s pretty amazing to see IT finally getting out of its own way to let the potential of today’s amazing technologies shine through. Great post.

  4. Pingback: What is DevOps?

Leave a Reply