Today, we’ll share a small victory in our DevOps journey at Red Hat IT. This cross-team collaboration has saved our IT organization some headaches and wasted time. We open-sourced the code, hoping it can help you, too.
The Dev problem, from Sam Van Oort:
Old, pruned git branches are sometimes re-created by accident, making a mess for our developers.
Code repositories are the final resting place for code, acting as equal parts bank vault, museum, and graveyard. Unlike a vault, content is almost always added faster than it is removed, but much like a graveyard, there is a definite miasma when a repository becomes too full. That smell is developer frustration, as they have to search through dozens, or eventually, hundreds of branches to find the one they want.
We've had sporadic cases where branches did not get merged into masters (and sometimes fixes were overwritten in later releases) and have wasted collectively hundreds of developer hours on "which branch is this in?" exchanges.
This situation becomes worse when dealing with a configuration management system like Chef or Puppet. In Red Hat IT, Puppet environments are also defined by branch names, and there's a hierarchy in which branch they look to for code. We had a case where a branch used in testing new code was re-pushed by accident, breaking the ability to create new nodes in that dev environment, and greatly puzzling two developers.
The solution is to periodically prune branches and throw out content which is no longer needed. But, there’s a big problem: git provides one very dangerous command that can quickly undo all your careful pruning work: "git push." One developer that accidentally runs this (rather than “git push origin branchname”) can immediately recreate all the branches that were so carefully removed.
The Ops collaboration, from Steve Milner:
Hey, we have pre-receive hooks in git. We can solve this, and prevent yet another “did I do that?!” moment.
To address this challenge, the Inception team put together a free, open source git branch blacklisting tool that adds shell commands to “blacklist” branches, and has git hooks to prevent accidental re-pushes.
Two approaches originally came to mind while thinking about implementation of a blacklisting tool: an in-repo blacklist file and a server-side configuration blacklist. After mulling over the two approaches we decided that having an in-repo blacklist file may lead to developers accidentally merging or overwriting each others’ blacklisted branches and that would negate the tool’s purpose! Instead, the idea of using the built in git config system to house a master list of blacklisted branches on the git origin was chosen. Since all of our git repositories utilize ssh as the transport it became trivial to use ssh to execute git config on the server side allowing for adding, listing, and removing of blacklisted branches. The server side would also house a pre-receive hook which checks the blacklist before allowing the branch to be pushed.
It was important to make the tool as easy as possible so developers could use it without remembering special conventions. Forcing developers to remember the right remote commands to modify the blacklist was out of the question. The tool also needed to work in a similar fashion as other git tools. To that end, the client side of the tool, wrapping remote execution, was named git-branch-blacklist. By using this name the developer can call it directly or as git branch-blacklist. The tool also follows a simple convention of git branch-blacklist $COMMAND where $COMMAND is add, remove, or list.
Finally, adding the pre-receive hook needed to be as easy as possible. So, we wrote a hook installer named git-branch-blacklist-install. It simply verifies the repository is set up in such a way the blacklist system will work, and then copies the hook into the right location for said repository.
The implementation was not without its downsides. It assumes ssh is your transport, so if you use a different transport and disable ssh access, it doesn’t work. Also, the tool is written in shell script and requires common unix tools. If someone wanted to run this tool on Windows they would need to install Cygwin first. That said, the downsides were acceptable as they don’t apply to our use cases.
The DevOps success:
Today, Sam’s team has implemented the git-branch-blacklist tool. They’re at 180 and 65 branches (after initial pruning) in their two main repositories, and expect blacklisting to help reduce this over time, saving much developer time and frustration. Nobody has fessed up to git-branch-blacklist saving their hide just yet, but we suspect it’s just a matter of time.
Last updated: January 10, 2023