ARTIF: advanced real time threat intelligence framework

ARTIF

ARTIF is a new advanced real time threat intelligence framework built that adds another abstraction layer on the top of MISP to identify threats and malicious web traffic on the basis of IP reputation and historical data. It also performs automatic enrichment and threat scoring by collecting, processing and correlating observables based on different factors.

Key features of ARTIF includes:-

• Scoring System: Enriches IP addresses with threat metadata including a threat score which can act as a threshold value for security teams to take action on.
• Containerized: ARTIF is deployed using containers and hence provides ease for deployment.
• Modular Architecture: The project is plugin-based and can be extended easily by just modifyng threat feeds in MISP. There would be no downtime to the actual service as these would be in-line update.
• Alerting: Extended feature providng seamless integration with slack for active alerting. It also provides better attack profiling and visualization.

Some use Cases:-

• Threat Detection
• Logging and Monitoring
• User profiling
• Alerting automation

Why use ARTIF? It is a real-time Threat Intel Framework that can help identify malicious IPs even though they are not present in the MISP. This helps organizations to set up the first layer of defense by providing transparency over malicious web traffic reaching their servers.

• By default 52 configured open source threat feeds with a database of 0.7M IP addresses.
• Has latency of ~ 180 ms > 10x faster than commercial products.
• Historical IPs are stored for analysis and used in scoring on the basis of past records and patterns.
• Adds score to each IP in addition to other metadata.

Technical Details

ARTIF being a threat framework, is highly useful for visibility inside organization traffic. It is completely written in python and collects intelligence on an IP from various feeds. It then sends this data to a correlation engine which generates a threat score where historical data is also one of the factors taken into account for threat score calculation.

Every time a new IP hits the service a celery worker is assigned the task to update its score in the database by coordinating with the correlation engine, which in turn gathers data from multiple sources. To ensure that the data isn’t stale (default ‘stale time’ value is 24hrs), we run a worker for each IP in the database which hasn’t been updated for the past ‘stale time’. As with all the other parameters this value is configurable too. Since the threat score calculation is the key to all of this, we’ve laid a lot of emphasis on it. Apart from the well-known threat feeds such as MISP, Cortex, Alien Vault, VirusTotal, and popular blacklist of IPs, we can also add custom feeds according to business and security use case, where customer loyalty can also be used as a parameter for threat score calculation.

For ease of configuration, ARTIF needs the following input to run itself:

• Feeds in MISP instance
• A MISP key for communicating with the MISP instance
• A MISP URL to connect to MISP instance using the MISP key

Upon running, the data is processed and stored on a MongoDB container. MongoDB container contains 3 important databases which store information about the IP in the feed as well as its metadata for eg. country/ASN, Org, etc. If the IP is not found in the database that means it’s a new IP reaching the server and whose information is not present in MISP. We then calculate a risk score for the particular IP using an algorithm and various other parameters like geolocation, ASN, and Org. More information about the scoring engine can be found here.

ARTIF has additional functionality to keep track of old feeds. The default configuration replenishes the latest feeds every 24 hrs. After 24 hrs the old feed is moved to another collection and the latest ones are added. For ease of operation, we have added a field called “historical” in the output which tells if the IP was historically bad. A false value indicates that IP has been recently added to the feeds while a True value suggests that IP was already present in older feeds and hence its a historically malicious IP.

All the historical IP will be removed from the database after 7 days by default.

Install & Use

Copyright 2021 Dreamplug Technologies Private Limited