since 1999

 

9 minutes estimated reading time.

Restrict Who Can Push to Matching Branches on Github

— (Last Updated: 10/23/2019)

An anonymous attacker has been compromising Git repositories and demanding ransom. This attacker stole the contents and used a force push to wipe the remote repository causing many to lose access to their critical source code assets. Use critical security tools available within the Git ecosystem to protect your company from this threat with:

The technique that appears to have been used by the anonymous attacker is not dependent on which Git repository hosting service provider, meaning the threat impacted github.com, bitbucket.org, and gitlab.com equally. It was reported by scmagazine.com that the attackers compromised the username and passwords of accounts from Git repositories via config files exposed on web servers that fetched content.

The Threat Scenario

What happened

A remote attacker was able to discover Git repository username and passwords in plaintext configuration files on an unknown number of servers at many different companies. The attack against private Git repositories was not limited to a particular host and the same attack pattern was used against many companies based on the fact that all that was necessary to access and compromise the account was a developer’s user credential that was used to authenticate with the Git host. Relying only on passwords is poor practice and storing those credentials in a config file for to support remote deployments is worse.

Once access was gained, the attacker was able to:

  1. Clone a full copy of the repository and all of its contents, including
    • files that were deleted from the repository
    • any database passwords, encryption keys, and server secrets that were stored within the repository
  2. Encrypt the contents of the repository and force push back to the remote host to overwrite the code
  3. Demand ransom in Bitcoin of the victims
  4. Publicly disclose the contents of the victim’s previously private repository and all its contents

Understanding the Distributed Nature of Git

Git is the proper name of a version-control system for tracking changes in computer files and coordinating work on those files among multiple authors. It is a high profile tool that was created by Linus Torvalds, the inventor of the Linux operating system. Junio Hamano, who currently works for Google, has maintained the system since 2005.

An important aspect to Git is that no central repository server is required! In fact, the original users of Git were the core members of the Linux kernel development team who shared commits as patches using e-mail! You can still do this work flow today, but many have taken to using a central source control repository that provides not only the services of Git itself but also a rich web-based user interface to manage the repository, track issues, and perform code review functions on pull requests.

Finally, the distributed nature has an important nuance for backups. Your central code repository is not the only copy of the data. Every time the repository is cloned - by a developer or by a deployment - a complete copy of the entire repository along with any file that has ever been part of the repository is fully copied to the developers' machine or to the server. This means there are fully copies of your repository on every system that works with it.

Depending on context, these full spare copies are a massive benefit and also a risk. On the positive side, every copy serves as a backup (potentially ad hoc) or they can be a source unintentional disclosure. This property is why you should never store database credentials and other server secrets in your Git repository but instead load those values from a separate, appropriate secrets store at deployment time.

How to Git Protection from This Threat

One one level, we can tell organizations to not mess up their configuration on their servers. That any config file that is accessible in the web root or via a runtime compromise is at risk. While this is true, there are good ways to have a defense in depth strategy.

Use Deploy Keys, not User Credentials, for Unattended Deployment Access

When a server needs read only access to a Git repository to clone and deploy code, one should not simply use a Git credential with read/write support. Instead create a read-only SSH deploy key on the server and add it to the repository.

You start by generating an OpenSSH keypair on the system that needs read-only access to the repository. It is usually okay to skip adding a password to this key because it may need to access Git unattended as a CI/CD or deploy process step. Once you have a keypair, you add the public key as a Deploy Key like shown below:

Example Deploy Key in GitHub

You should read up on your Git hosting provider’s instructions on managing deploy keys for more in-depth information:

Require Mandatory Two Factor Authentication for Developers

Passwords, especially those chosen by users, are the weakest link in your security posture. Many data breaches are triggered when your staff have their credentials compromised by a phish or when haphazardly stored in a config file like in the recent Git attacks. Weak credentials is a problem so prevalent that the U.S. National Institute for Science and Technology adopted the NIST 800–63b Federal Standard for password verifiers, which includes the recommendation to implement two factor authentication (2FA).

You should check out the details of how your Git hosting service provides 2FA, including:

At Rietta, we mandate 2FA for all of our developers and enforce this on our GitHub.com organization such that people cannot even connect to a repository to do work without 2FA enabled. Our team members use TOTP apps on their phones as well as company provided Yubikey Fido U2F tokens.

Implementing 2FA really increases your security more than any other single measure can. It is a good practice that is recommended by Federal standards.

Use Protected Branches to Mandate Pull Requests and Peer Code Reviews

Protected Branches are a feature of your Git repository host that “disables force-pushes to all matching branches and prevents them from being deleted.” The idea is that you define master and staging and other mainline branches of your choosing as protected. Then you can require that certain rules are satisfied before any changes can be made to the protected branch.

GitHub provides several options that can greatly enhance both your code quality and make it so that a developer’s Git credential cannot overwrite the repository. These options include:

We actually use this feature to manage this website. The config looks like this:

Example Protected Branch

Even though my company is currently using Github and thus I am most familiar with its tools, other Git hosting companies support it too. Read more about this feature on

Backup Your Git Repositories

The ransomware aspect of the recent attack goes to show that many companies do not have a comprehensive backup strategy for their Git resources. As covered in the section on the Distributed Nature of Git, one has an adhoc backup of many Git repositories by default with fully cloned copies being distributed to:

  1. every team member with a local cloned copy,
  2. as private forks in individual team member’s accounts on the Git hosting provider when your team practices a fork/pull request model rather than having developers commit to the master repo directly
  3. your CI/CD environment with cloned copies,
  4. and possibly even your production servers as well (look for hidden .git folders).

The challenge with ad hoc backups is knowing which team member has which project and if they have them all.

I am not aware of a Git-specific commercial backup service. However, you can implement your own 3-2-1 strategy by fully cloning each of your repos in your Git by creating a backup server that has deploy keys on each of your company’s repositories. A cronjob on that server would simply merge upstream changes from each of the repositories it is cloning on a regular basis. You would then setup a commercially-proper backup service on that Git backup server - by something like CrashPlan for Small Business, or your automatic virtual machine snapshots, etc. Then if your repository was overwritten by malice or accident, you would be able to restore your repositories to the previous state from your backups.

Going the Full Self Hosted Route

When I first started using Git professionally, I self hosted the repositories on my own FreeBSD server. My collaborators and I all had shell accounts and could push, pull, and merge from that shared server. Things have come a long way. You can get the full GitHub-like experience for your organization and self host. Your self hosted solution can then be backed up by your enterprise-class backup service that you have available.

If this route interests you, check out the the open-source GitLab Community Edition, paid GitLab self hosted, or GitHub Enterprise.

Conclusion

The attacks against Git repositories in the news have served as a reminder that organizations need a solid defense in depth strategy to protect their source code assets that are conveniently hosted on third party servers. The attack was possible because of credentials with full read/write permissions were compromised via phishing or because they were written to plaintext files that were exposed on servers. I’ve show you how you can significantly harden your Git repositories through the use of deploy keys, 2FA, and protected branches. I also discussed how you could implement a backup strategy that could survive even if your only master repo was overwritten by malice or accident.

Using protected branches and pull requests not only protects against this situation, but will set you up for an effective way to do peer-code reviews to build higher quality software. A double win for security and code quality!

If anyone in your organization is concerned about Git security, please show them this article. If I can be of any assistance, I’m happy to do so. I do not have any direct financial stake in any of the companies mentioned here except incidental holdings through mutual funds. I am a happy user who uses Git extensively for my company and for our clients.