Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Protect secrets in Git with the clean/smudge filter

February 2, 2022
Tomer Figenblat
Related topics:
GitOpsSecure Coding
Related products:
Red Hat OpenShift

Share:

    When working on public Git repositories, you need to pay close attention so that you don't accidentally push secret information such as tokens, private server addresses, personal email addresses, and the like. One of the tools that can help you is Git's clean/smudge filter.

    Clean and smudge your Git repository 

    The clean/smudge filter is quite simple to use. Create a filter driver with two commands—clean and smudge—and then apply the filter per record in the .gitattributes file.

    This makes the filter stand between the working directory and the staging area for the specific .gitattributes record. When adding content from the working directory to the staging area with git add, files that match the .gitattributes record will go through the clean filter. When pulling content back into the working directory with git pull, those same files will go through the smudge filter.

    Create the filter driver

    Let's create a filter driver named cleanPass that uses sed expressions to replace the value secretpassword with the value hiddenpassword and vice versa:

    # mac users should use gsed instead of sed
    git config filter.cleanPass.clean "sed -e 's/secretpassword/hiddenpassword/g'"
    git config filter.cleanPass.smudge "sed -e 's/hiddenpassword/secretpassword/g'"
    

    Tip: Add --global to the config command to make the filter driver available globally to all repositories at ~/.gitconfig instead of locally at .git/config.

    As an example, let's apply the cleanPass filter driver to the JSON file type. In the repository's .gitattributes, look for the JSON file type record and modify it. The result should look something like this, depending on the original configuration:

    *.json text eol=lf filter=cleanPass
    

    Tip: To avoid pushing, this can be done in .git/info/attributes, which takes precedence over .gitattributes. Just be sure not to mess up any of the original record configurations like eol.

    From now on, every new or modified JSON file in the repository will go through the cleanPass filter. Figure 1 provides an animated look at what happens next.

    git-clean-smudge-filter
    Created by Tomer Figenblat, License under MIT License.
    Figure 1. The clean filter in action.

    One filter to rule them all

    That was cool, but using this filter can get quite cumbersome at times. For instance, on a recent task, I worked with multiple repositories using multiple (but identical) services from my team's Red Hat OpenShift lab. To add to the complexity, I usually work from two different computers—my desktop at my office and my laptop at home.

    That all means that I needed to create multiple filters (or one complex one) and apply them to multiple file types, for multiple repositories, on multiple computers. That gave me a headache, especially when the fact hit me that this workflow will probably reoccur in future tasks.

    Eventually, I decided to create a very simple script that handles multiple value replacements in both directions, save it on each of my computers, and configure it as a global filter.

    Configure a global filter

    Adding values to be replaced in this script is super easy, and attaching it to .gitattributes records, as seen above, is quite simple as well.

    First, create a script in a path available to all repositories. For instance, you could put it in your home directory at ~/scripts/git-smudge-clean-filter.sh:

    #!/bin/bash
    
    declare -A mapArr
    
    mapArr["my-work-private-server.mywork.com"]="<reducted-work-server>"
    mapArr["my-personal-private-server.myowndomain.org"]="<reducted-personal-server>"
    mapArr["A*&#QAADDA(77##F"]="super-secret-token"
    mapArr["oops@mypersonal.email"]="support@correct.email"
    
    # mac users should use gsed instead of sed
    sedcmd="sed"
    if [[ "$1" == "clean" ]]; then
      for key in ${!mapArr[@]}; do
        sedcmd+=" -e \"s/${key}/${mapArr[${key}]}/g\""
      done  
    elif [[ "$1" == "smudge" ]]; then
      for key in ${!mapArr[@]}; do
        sedcmd+=" -e \"s/${mapArr[${key}]}/${key}/g\""
      done  
    else  
      echo "use smudge/clean as the first argument"
      exit 1
    fi
    
    eval $sedcmd

    Add as many pairs as you like to mapArr.

    Note: Per the nature of Git's filter driver, you can't use the same key or value more than once.

    Next, create the filter driver globally (note the script argument):

    git config --global filter.reductScript.smudge "~/scripts/git-smudge-clean-filter.sh smudge"
    git config --global filter.reductScript.clean "~/scripts/git-smudge-clean-filter.sh clean"

    Now, for every .gitattributes record to which you apply the filter=reductScript configuration in any repository, every file matching this record will go through the script, and every value specified in mapArr will be reducted (or returned).

    Conclusion

    That's all there is to it. I hope you find this useful. We'll cover other Git tips in future articles.

    In the meantime, check out the following Git resources:

    • My GitHub gist
    • gitattributes documentation
    • git-config documentation
    • Git filter.<driver>.clean command
    • Git filter.<driver>.smudge command
    • Git Cheat Sheet
    Last updated: September 7, 2022

    Related Posts

    • How to ignore files in Git without .gitignore

    • Deploy self-hosted GitHub Actions runners for Red Hat OpenShift

    • Automate dependency analytics with GitHub Actions

    Recent Posts

    • Unleashing multimodal magic with RamaLama

    • Integrate Red Hat AI Inference Server & LangChain in agentic workflows

    • Streamline multi-cloud operations with Ansible and ServiceNow

    • Automate dynamic application security testing with RapiDAST

    • Assessing AI for OpenShift operations: Advanced configurations

    What’s up next?

    Path to GitOps cover card

    Read The Path to GitOps for a comprehensive look at the tools, workflows, and structures teams need to have in place in order to enable a complete GitOps workflow.

    Get the e-book
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue