GitBleed Tools: extracting data from mirrored git repositories
GitBleed Tools – for extracting data from mirrored git repositories
This repo contains shell scripts that can be used to download and analyze differences between cloned and mirror Git repositories. For more information about the underlying quirk in Git behavior, please visit read our blog post.
What Do These Scripts Do?
These scripts will clone a copy of the given Git repository, both as a regular clone and mirrored (“–mirror”) option. It will then create a delta between the two, seeking to find the parts of the repository that are only available in mirror mode. Last, gitleaks will be run to see if any secrets are present in the delta portion, and “git log” will be used to create a single file containing the bodies of the commits so they can be analyzed easier.
Please note that since this script creates three copies of the repository, it may consume a lot of disk space.
Example repositories
You can test these tools on the following two example repositories:
- gb_testrepo_delete – repository hiding secrets via deleted commits
- gb_testrepo_reset – repository hiding secrets via “git reset”
Requirements
You will need Git, Python 3. GitLeaks and git-filter-repo to be installed. Here is an example of installing these on MacOS:
brew install git python3 gitleaks git-filter-repo
How to Install and Run
You can run this against a repository as follows:
git clone https://github.com/nightwatchcybersecurity/gitbleed_tools.git
cd gitbleed_tools
./gitbleed.sh https://github.com/nightwatchcybersecurity/gitbleed_tools.git example
There are also some helper scripts that can be used to scan GitHub and GitLab repositories as follows:
./gitbleed_gh.sh nightwatchcybersecurity/gitbleed_tools
./gitbleed_gl.sh nwcs/junit_ui_bug
This will create an example folder containing three subfolders:
- clone – contains the cloned repository
- delta – contains the mirrored repository minus all of the commits in the “clone”
- mirror – contains the mirrored repository cloned with the “–mirror” option
There are also three files created:
- clone_hashes.done.txt – list of hashes in the cloned repository
- gitleaks.json – results from running gitleaks
- gitlog.txt – all commits from the delta folder concatenated into a single file
Copyright (C) 2022 nightwatchcyber
