cve-search

cve-search is a tool to import CVE (Common Vulnerabilities and Exposures) and CPE (Common Platform Enumeration) into a MongoDB to facilitate search and processing of CVEs.

The main objective of the software is to avoid doing the direct and public lookup into the public CVE databases. This is usually faster to do local lookups and limits your sensitive queries via the Internet.

cve-search includes a back-end to store vulnerabilities and related information, an intuitive web interface for search and managing vulnerabilities, a series of tools to query the system, and a web API interface.

cve-search is used by many organizations including the public CVE services of CIRCL.

Changelog v3.0

• [travis] fix to use JSON NVD source + removed unsupported Python
  version. [Alexandre Dulaunoy]
• Merge pull request #451 from P-T-I/import_impr. [Alexandre Dulaunoy]

  Initial import restructure

• Final fix for missing field. [Paul Tikken Laptop]
• Fix for missing last-modified field in cve documents. [Paul Tikken
  Laptop]
• Fix for missing last-modified field in cve documents. [Paul Tikken
  Laptop]
• Minor adjustment travis.yml. [Paul Tikken Laptop]
• Merge from master. [Paul Tikken Laptop]
• Merge pull request #478 from P-T-I/unit_tests. [Alexandre Dulaunoy]

  Unit tests

• Added specific parser to BeautifulSoup. [Paul Tikken Laptop]
• Added build arguments to travis file. [Paul Tikken Laptop]
• Final travis file. [Paul Tikken Laptop]
• Working on tests. [Paul Tikken Laptop]
• Working on tests. [Paul Tikken Laptop]
• Working on tests. [Paul Tikken Laptop]
• Working on tests. [Paul Tikken Laptop]
• Added BeautifulSoup to requirements. [Paul Tikken Laptop]
• Working on unit tests. [Paul Tikken Laptop]
• Working on unit tests. [Paul Tikken Laptop]
• Added dict to xml requirement. [Paul Tikken Laptop]
• Fix search.py not returning xml. [Paul Tikken Laptop]
• Testing_travis. [Paul Tikken Laptop]
• Testing_travis. [Paul Tikken Laptop]
• Testing_travis. [Paul Tikken Laptop]
• Testing_travis. [Paul Tikken Laptop]
• Testing_travis. [Paul Tikken Laptop]
• Testing_travis. [Paul Tikken Laptop]
• Testing_travis. [Paul Tikken Laptop]
• Testing with travis. [Paul Tikken Laptop]
• Testing travis file. [Paul Tikken Laptop]
• Testing travis file. [Paul Tikken Laptop]
• Testing travis file. [Paul Tikken Laptop]
• Fixed -p switch travis file. [Paul Tikken Laptop]
• Change to unit_tests. [Paul Tikken Laptop]
• Change to unit_tests. [Paul Tikken Laptop]
• Change to unit_tests. [Paul Tikken Laptop]
• Change in unit_tests. [Paul Tikken Laptop]
• Change in unit_tests. [Paul Tikken Laptop]
• Change in unit_tests. [Paul Tikken Laptop]
• Requirements.txt fix. [Paul Tikken Laptop]
• Altered travis for pytest support. [Paul Tikken Laptop]
• Setup unit testing scripts. [Paul Tikken Laptop]
• Init files added when needed for unit_tests. [Paul Tikken Laptop]
• Black formatting. [Paul Tikken Laptop]
• Added nltk to requirements.txt as it was not covered. [Paul Tikken
  Laptop]
• Black formatting. [Paul Tikken Laptop]
• Config files added for testing. [Paul Tikken Laptop]
• Added pytest requirements. [Paul Tikken Laptop]
• Removed old testing file. [Paul Tikken Laptop]
• Black formatting. [Paul Tikken Laptop]
• Testing with travis. [Paul Tikken Laptop]
• Merge remote-tracking branch ‘origin/master’ [Paul Tikken Laptop]
• Create stale.yml. [PT]
• Merge branch ‘up_master’ into import_impr. [Paul Tikken Laptop]
• Merge pull request #470 from P-T-I/cve-search-469. [Alexandre
  Dulaunoy]

  cve-search-469; fix for not deplaying results

• Cve-search-469; fix for not deplaying results. [Paul Tikken Laptop]
• Merge branch ‘up_master’ into import_impr. [Paul Tikken Laptop]
• Merge pull request #468 from P-T-I/regex_options. [Alexandre Dulaunoy]

  fix #464; corrects bad fix from #465

• Fix #464; corrects bad fix from #465. [Paul Tikken Laptop]
• Merging. [Paul Tikken Laptop]
• Merge pull request #465 from P-T-I/regex_fail. [Alexandre Dulaunoy]

  Regex fail

• Alter .gitignore. [Paul Tikken]
• Fixes #464; double options (IGNORE_CASE) declaration for a regex
  search. [Paul Tikken]
• Altered .gitignore. [Paul Tikken]
• Fixed syntax warnings. [Paul Tikken Laptop]
• Added jsonpickle requirement. [Paul Tikken Laptop]
• Added auto creation of log dir. [Paul Tikken Laptop]
• Troubleshooting build error on feedformatter version. [Paul Tikken
  Laptop]
• Merge branch ‘master’ into import_impr. [Paul Tikken Laptop]
• Merge pull request #459 from P-T-I/docker_version. [Alexandre
  Dulaunoy]

  fix #205; official dockerized version of CVE-Search added

• Fix #205; official dockerized version of CVE-Search added. [Paul
  Tikken Laptop]
• Merge pull request #460 from P-T-I/cve_search_#395. [Alexandre
  Dulaunoy]

  fix #395; Fixed warning message Mongoclient create pre-fork

• Fix #395; Fixed warning message Mongoclient create pre-fork. [Paul
  Tikken Laptop]
• Added variable interval counter for debug logging. [Paul Tikken
  Laptop]
• Corrected update error. [Paul Tikken Laptop]
• Rebase. [Paul Tikken Laptop]
• Merge pull request #456 from P-T-I/syntax_warnings. [Alexandre
  Dulaunoy]

  Fixed Tornado’s syntax warnings

• Fixed Tornado’s syntax warnings. [Paul Tikken Laptop]
• Merge pull request #454 from P-T-I/cve_search-449. [Alexandre
  Dulaunoy]

  fix #449; Added stricter regex for matching CVE on CPE

• Fix #449; added stricter cpe regex when matching CVEs on CPEs. [Paul
  Tikken Laptop]
• Refactor. [Paul Tikken Laptop]
• Refactor. [Paul Tikken Laptop]
• Refactor. [Paul Tikken Laptop]
• Merge branch ‘master’ into cve_search-449. [Paul Tikken Laptop]
• Merge pull request #453 from P-T-I/doc_update. [Alexandre Dulaunoy]

  fix #452; Documentation update

• Fix #452; Documentation update to bring the docs in line with the
  readme.md in the root. [Paul Tikken Laptop]
• Cleanup. [Paul Tikken Laptop]
• Black formatting. [Paul Tikken Laptop]
• Missing sys import and black formatting. [Paul Tikken Laptop]
• Moved DatabaseIndexer to separate class in Sources_process.py. [Paul
  Tikken Laptop]
• Moved DatabaseIndexer to separate class in Sources_process.py. [Paul
  Tikken Laptop]
• Added additional log entries. [Paul Tikken Laptop]
• Changed logger name. [Paul Tikken Laptop]
• Added description to tqdm progressbar from CPERedisBrowser class.
  [Paul Tikken Laptop]
• Unified logging with updater and black formatting. [Paul Tikken
  Laptop]
• Import refactor and minor edit. [Paul Tikken Laptop]
• Set JSON file progress debug logging to every 5000 items. [Paul Tikken
  Laptop]
• Moved logic to process class. [Paul Tikken Laptop]
• Added CPERedisBrowser class. [Paul Tikken Laptop]
• Added logging and tqdm progressbar. [Paul Tikken Laptop]
• Added logging. [Paul Tikken Laptop]
• Fixed misspelled method (getCVEID instead of getCVEIDs) and black
  formatting. [Paul Tikken Laptop]
• Added debug counter from processing items from file every 1000 items.
  [Paul Tikken Laptop]
• Added debug counter from processing items from file. [Paul Tikken
  Laptop]
• Refactor and unified logging with process classes. [Paul Tikken
  Laptop]
• Refactor and unified logging with process classes. [Paul Tikken
  Laptop]
• Modified update doc versus insert doc. [Paul Tikken Laptop]
• Moved process classes to separate file. [Paul Tikken Laptop]
• Refactor. [Paul Tikken Laptop]
• Separate file for source process classes. [Paul Tikken Laptop]
• Separate file for xml Content Handlers. [Paul Tikken Laptop]
• Methods refactor. [Paul Tikken Laptop]
• Added process methods to class instead. [Paul Tikken Laptop]
• Changed process_item method. [Paul Tikken Laptop]
• Added process_item to DownloadHandler class. [Paul Tikken Laptop]
• Added method to retrieve the entire redis list. [Paul Tikken Laptop]
• Added process_item to XMLFileHandler class. [Paul Tikken Laptop]
• Added db (9) for redis queue. [Paul Tikken Laptop]
• Added RedisQueue. [Paul Tikken Laptop]
• Moved download_site method to DownloadHandler.py. [Paul Tikken Laptop]
• Added redis queue as a replacement of multiprocessing queue. [Paul
  Tikken Laptop]
• Added database action class. [Paul Tikken Laptop]
• Refactor. [Paul Tikken Laptop]
• Added additional logging. [Paul Tikken Laptop]
• Minor changes. [Paul Tikken Laptop]
• Reset insert to original. [Paul Tikken Laptop]
• Added different handlers. [Paul Tikken Laptop]
• Added different handlers. [Paul Tikken Laptop]
• Added different handlers. [Paul Tikken Laptop]
• Minor. [Paul Tikken Laptop]
• Minor. [Paul Tikken Laptop]
• Set debug print to every 10 cycles. [Paul Tikken Laptop]
• Added venv and .idea folders to ignore. [Paul Tikken Laptop]
• Set exit code on errors to 1. [Paul Tikken Laptop]
• Added VIADownloads class for update optimalization. [Paul Tikken
  Laptop]
• Moved updates of info collection to DownloadHandler. [Paul Tikken
  Laptop]
• Added requirements ijson and tqdm. [Paul Tikken Laptop]
• Added logging and file extension specific classes. [Paul Tikken
  Laptop]
• Added tqdm and ijson requirements. [Paul Tikken Laptop]
• Added queues and multiprocessing. [Paul Tikken Laptop]
• Added further multiprocessing. [Paul Tikken Laptop]
• Added speed improvements for initial import. [Paul Tikken Laptop]
• Black formatting. [Paul Tikken Laptop]
• Black formatting. [Paul Tikken Laptop]
• Speed improvements for initial import of data. [Paul Tikken Laptop]
• Merge pull request #450 from P-T-I/web_impr. [Alexandre Dulaunoy]

  minor admin page gui adjustments

• Minor admin page gui adjustments. [Paul Tikken Laptop]
• Merge pull request #448 from P-T-I/query_opt. [Alexandre Dulaunoy]

  small http query optimalization and black formatting

• Small http query optimalization and black formatting. [Paul Tikken
  Laptop]

Installation

git clone https://github.com/cve-search/cve-search.git
cd cve-search
sudo pip3 install -r requirements.txt
Install mongodb

Usage

For the initial run, you need to populate the CVE database by running:

./sbin/db_mgmt.py -p
./sbin/db_mgmt_cpe_dictionary.py
./sbin/db_updater.py -c

It will fetch all the existing XML files from the Common Vulnerabilities and Exposures database and the Common Platform Enumeration. The initial Common Platform Enumeration (CPE) import might take some time depending on your configuration.

If you want to add the cross-references from NIST, Red Hat and other vendors:

./sbin/db_mgmt_ref.py

Databases and collections

The MongoDB database is called cvedb and there are 11 collections:

• cves (Common Vulnerabilities and Exposure items) – source NVD NIST
• cpe (Common Platform Enumeration items) – source NVD NIST
• cwe (Common Weakness Enumeration items) – source NVD NIST
• capec (Common Attack Pattern Enumeration and Classification) – source NVD NIST
• ranking (ranking rules per group) – local cve-search
• d2sec (Exploitation reference from D2 Elliot Web Exploitation Framework) – source d2sec.com
• MITRE Reference Key/Maps – source MITRE reference Key/Maps
• ms – (Microsoft Bulletin (Security Vulnerabilities and Bulletin)) – source Microsoft
• exploitdb (Offensive Security – Exploit Database) – source offensive security
• info (metadata of each collection like last-modified) – local cve-search
• via4 VIA4CVE cross-references.

The Redis database has 3 databases:

• 10: The cpe (Common Platform Enumeration) cache – source MongoDB cvedb collection cpe
• 11: The notification database – source cve-search
• 12: The CVE reference database is a cross-reference database to CVE ids against various vendors ID – source NVD NIST/MITRE

The reference database has 3 additional sources:

• MITRE Reference Key/Maps.
• Red Hat RPM to CVE database.
• Red Hat RHSA Oval database.

Updating the database

./sbin/db_updater.py -v

Repopulating the database

To easily drop and re-populate all the databases

./sbin/db_updater.py -v -f

This will drop all the existing external sources and reimport everything. This operation can take some time and it’s usually only required when new attributes parsing are added in cve-search.

You can search the database using search.py

./bin/search.py -p cisco:ios:12.4
./bin/search.py -p cisco:ios:12.4 -o json
./bin/search.py -f nagios -n
./bin/search.py -p microsoft:windows_7 -o html

If you want to search all the WebEx vulnerabilities and only printing the official references from the supplier.

./bin/search.py -p webex: -o csv -v “cisco”

You can also dump the JSON for a specific CVE ID.

./bin/search.py -c CVE-2010-3333

Advanced Tutorial