In many organizations development teams are split up from systems administrators.  Both teams focus mainly on their roles and responsibilities, but this can lead to problems.  Tensions between the groups can hinder your organization's efforts in getting the best product to the consumers in the shortest amount of time. Below are quotes which are commonly come up between developers and systems administrators.

• “Why doesn't the development team have to be on call? It's their code that breaks the system.”
• “Why does the sysadmin get to dictate what tools/languages I can use to solve the problem?”
• “Add more hardware to the server”
• "Stop using so many resources on the system!"

You begin to easily see how this relationship is strained and broken, but does it have to be? Let’s talk about some more of the common stereotypes and discuss the ways to move past them.

Developers in Prod

A very dated, yet still common, idea is not allowing developers any access to production. Ever. Period. There are many different excuses cited by system administrators for  limiting developers access to production systems such as auditing, the concern that “developers will break prod”, and additional security concerns. Some companies feel allowing developers into production may allow for them to maliciously insert code that may compromise the production system.

We’d like to encourage a different way of thinking: Developers should be hired for their skills in development. If you do not trust the person and their skills then it does not make sense to hire them in the first place.

Speaking as a systems administrator it would be very difficult to parse through all lines of code to even check to ensure a developer was not doing something maliciously.  Giving a developer Production access also allows collaboration in the solving of problems that a systems administrator notices and then use that information to create a fix and promote it up to the production environment. This is not to say that developers get access to everything, but certainly access to the areas where they are the best people to be handling the issues.  At DOES14, Netflix’s Dianne Marsh and Roy Rapoport had similar thoughts on this topic.

This is also not a one way street. System administrators should feel welcome and even be encouraged to join in development design sessions and even during programming. After all, who better than those versed in infrastructure to lend a hand in designing a solution?

Problem Solving and Perspective

Once, Steve Milner and I were walking down the hall and I brought up a thought about the water temperature at the faucets. I asked him something along the lines of how the proper portions of hot and cold water were mixed to gain a nice warm, but not hot, water temperature on every use. After all, the warm and cold temperature wouldn’t always be the same and it would be tough to maintain the same mix. I thought the warm and cold water mixtures could be adjusted per faucet. His response was that the same water was likely heated in a single tank to the proper temperature and not mixed before dispense. Yeah, it’s a silly example but right there we had two different ways of thinking about a simple process and his view of how it works probably wouldn’t have crossed my mind!

Having developers and systems administrators work closely together allows for complex issues to be analyzed by people with vastly diverse backgrounds.  For example, as a systems administrator, I may feel like we could set up monitoring using Nagios and create custom checks to monitor a service, but a developer may know that the product we’re trying to monitor has an API designed specially for this purpose. Both solutions may work but the approach is dramatically different.

Developers Don't Do Deployments

In many organizations there is a hand off that takes place from the developers to a deployment team. Actually, the term “hand off” might be a little too reassuring, more often than not, it’s more like a relay race with a stick of butter baton on a hot day. The deployment team is usually comprised of one or more operations teams and, in many cases, are already slammed with other work on their plates.  The problem with that is if the queue for the operations teams is full then the developer is left waiting around or, even worse, another release from the development team may “stack on top” of previous release which can make it very difficult to troubleshoot issues that may arise from one of the deployments.

Final Thoughts

The Inception team is one of those unicorn teams where developers and systems administrators work together both locally and remotely. Working in a group designed like this is a dream come true.  We enabled the developers in our team to deploy code in any environment through the use of deployment scripts. As an Administrator, I am creating a process that not only better serves my developers, but also better services the company because it gets code out the door in a much more efficient manner. The admins and developers enable each other to be successful and that has delivered immeasurable value to our IT organization as well as to the home grown Winternewt tool.  Like the Osmonds, developers and systems administrators are little bit country and a little bit rock-and-roll but if you combine these two groups you can increase the speed of deployment from development to production and enable the teams to create the best version their product or application.  Oh, and both groups will be much happier with each other. Always a bonus.

Last updated: January 12, 2023