gitlab watchman v2.0 releases: audit GitLab for sensitive data and credentials exposed internally

by do son · Published February 9, 2022 · Updated April 3, 2022

GitLab Watchman

GitLab Watchman is an application that uses the GitLab API to audit GitLab for sensitive data and credentials exposed internally.

Features

It searches GitLab for internally shared projects and looks at:

Code
Commits
Wiki pages
Issues
Merge requests
Milestones

For the following data:

GCP keys and service account files
AWS keys
Azure keys and service account files
Google API keys
Slack API tokens & webhooks
Private keys (SSH, PGP, any other misc private key)
Exposed tokens (Bearer tokens, access tokens, client_secret, etc.)
S3 config files
Tokens for services such as Heroku, PayPal, and more
Passwords in plaintext
and more

Time-based searching

You can run GitLab Watchman to look for results going back as far as:

24 hours
7 days
30 days
All-time

This means after one deep scan, you can schedule GitLab Watchman to run regularly and only return results from your chosen timeframe.

Rules

GitLab Watchman uses custom YAML rules to detect matches in GitLab.

They follow this format:

---

filename:

enabled: #[true|false]

meta:

  name:

  author:

  date:

  description: #what the search should find#

  severity: #rating out of 100#

scope: #what to search, any combination of the below#

- blobs

- commits

- milestones

- wiki_blobs

- issues

- merge_requests

test_cases:

  match_cases:

  - #test case that should match the regex#

  fail_cases:

  - #test case that should not match the regex#

strings:

- #search query to use in GitLab#

pattern: #Regex pattern to filter out false positives#

There are Python tests to ensure rules are formatted properly and that the Regex patterns work in the tests dir

More information about rules, and how you can add your own, is in the file docs/rules.md.

Logging

GitLab Watchman gives the following logging options:

Log file
Stdout
TCP stream

Results are output in JSON format, perfect for ingesting into a SIEM or other log analysis platform.

For file and TCP stream logging, configuration options need to be passed via .conf file or environment variable. See the file docs/logging.md for instructions on how to set it up.

If no logging option is given, GitLab Watchman defaults to Stdout logging.

Changelog v2.0

Added:

New scopes for finding exposed data in:
- notes
- snippets
Docker image now available from the Docker hub, or by building from source. (Credit @adioss for the inspiration)
Complete rewrite of the codebase to make searching faster and more efficient.
- More modern packaging and distribution.
Logs now include more data
Additional signatures added to find more leaked data
Updated logo to play nicely with dark mode displays

Removed:

Logging to file and TCP stream – logs to stdout like a true 12 factor app. Reroute stdout as you see fit. –output
.conf file for configuration options. Pass the environment variables GITLAB_WATCHMAN_TOKEN and GITLAB_WATCHMAN_URL

Breaking changes: