SpiderFoot 3.0: OSINT reconnaissance tool

SpiderFoot is an OSINT automation tool for reconnaissance process, written in Python 3 and GPL-licensed.

Recently, Steve Micallef released on GitHub [1] a new version (3) of SpiderFoot, with a lot of interesting enhancements.

Web based UI or CLI
Over 170 modules (see below)
Python 3
CSV/JSON/GEXF export
API key export/import
SQLite back-end for custom querying
Highly configurable
Fully documented
Visualisations
TOR integration for dark web searching
Dockerfile for Docker-based deployments
Can call other tools like DNSTwist, Whatweb and CMSeeK

According to the release notes [2]:

Out of all the targets SpiderFoot supports, those new in 3.0 since 2.12 are in bold:
- IPv4 addresses
- IPv6 addresses
- Subnets
- Hostnames/sub-domains
- Domain names
- Phone numbers
- E-mail addresses
- Usernames
- Real names
- ASNs

When targeting names and usernames, it’s important to remember to place them in quotes, e.g. "Frank Smith" and "fsmith2000". Phone numbers must be in international format, prefixed with a + followed by the country code, e.g. +15550211221.

SpiderFoot integrates with just about every OSINT data source available, utilises a range of methods for data analysis and making that data easy to navigate using an embedded web-server for providing a clean and intuitive web-based interface:

Module	Description
abuse.ch	Check if a host/domain, IP or netblock is malicious according to abuse.ch.
AbuseIPDB	Check if a netblock or IP is malicious according to AbuseIPDB.com.
Accounts	Look for possible associated accounts on nearly 200 websites like Ebay, Slashdot, reddit, etc.
AdBlock Check	Check if linked pages would be blocked by AdBlock Plus.
Ahmia	Search Tor 'Ahmia' search engine for mentions of the target domain.
AlienVault IP Reputation	Check if an IP or netblock is malicious according to the AlienVault IP Reputation database.
AlienVault OTX	Obtain information from AlienVault Open Threat Exchange (OTX)
Apility	Search Apility API for IP address and domain reputation.
Archive.org	Identifies historic versions of interesting files/pages from the Wayback Machine.
ARIN	Queries ARIN registry for contact information.
Azure Blob Finder	Search for potential Azure blobs associated with the target and attempt to list their contents.
badips.com	Check if a domain or IP is malicious according to badips.com.
Bambenek C&C List	Check if a host/domain or IP appears on Bambenek Consulting's C&C tracker lists.
Base64	Identify Base64-encoded strings in any content and URLs, often revealing interesting hidden information.
BGPView	Obtain network information from BGPView API.
BinaryEdge	Obtain information from BinaryEdge.io's Internet scanning systems about breaches, vulerabilities, torrents and passive DNS.
Bing	Obtain information from bing to identify sub-domains and links.
Bing (Shared IPs)	Search Bing for hosts sharing the same IP.
Binary String Extractor	Attempt to identify strings in binary content.
Bitcoin Finder	Identify bitcoin addresses in scraped webpages.
Blockchain	Queries blockchain.info to find the balance of identified bitcoin wallet addresses.
blocklist.de	Check if a netblock or IP is malicious according to blocklist.de.
BotScout	Searches botscout.com's database of spam-bot IPs and e-mail addresses.
BuiltWith	Query BuiltWith.com's Domain API for information about your target's web technology stack, e-mail addresses and more.
CallerName	Lookup US phone number location and reputation information.
Censys	Obtain information from Censys.io
CINS Army List	Check if a netblock or IP is malicious according to cinsscore.com's Army List.
CIRCL.LU	Obtain information from CIRCL.LU's Passive DNS and Passive SSL databases.
Citadel Engine	Searches Leak-Lookup.com's database of breaches.
Cleanbrowsing.org	Check if a host would be blocked by Cleanbrowsing.org DNS
CleanTalk Spam List	Check if an IP is on CleanTalk.org's spam IP list.
Clearbit	Check for names, addresses, domains and more based on lookups of e-mail addresses on clearbit.com.
CoinBlocker Lists	Check if a host/domain or IP appears on CoinBlocker lists.
CommonCrawl	Searches for URLs found through CommonCrawl.org.
Comodo	Check if a host would be blocked by Comodo DNS
Company Names	Identify company names in any obtained data.
Cookies	Extract Cookies from HTTP headers.
Cross-Reference	Identify whether other domains are associated ('Affiliates') of the target.
Certificate Transparency	Gather hostnames from historical certificates in crt.sh.
Custom Threat Feed	Check if a host/domain, netblock, ASN or IP is malicious according to your custom feed.
cybercrime-tracker.net	Check if a host/domain or IP is malicious according to cybercrime-tracker.net.
Darksearch	Search the Darksearch.io Tor search engine for mentions of the target domain.
Digital Ocean Space Finder	Search for potential Digital Ocean Spaces associated with the target and attempt to list their contents.
DNS Brute-force	Attempts to identify hostnames through brute-forcing common names and iterations.
DNS Common SRV	Attempts to identify hostnames through common SRV.
DNS Look-aside	Attempt to reverse-resolve the IP addresses next to your target to see if they are related.
DNS Raw Records	Retrieves raw DNS records such as MX, TXT and others.
DNS Resolver	Resolves Hosts and IP Addresses identified, also extracted from raw content.
DNS Zone Transfer	Attempts to perform a full DNS zone transfer.
DroneBL	Query the DroneBL database for open relays, open proxies, vulnerable servers, etc.
DuckDuckGo	Query DuckDuckGo's API for descriptive information about your target.
EmailFormat	Look up e-mail addresses on email-format.com.
E-Mail	Identify e-mail addresses in any obtained data.
EmailRep	Search EmailRep.io for email address reputation.
Errors	Identify common error messages in content like SQL errors, etc.
Ethereum Finder	Identify ethereum addresses in scraped webpages.
File Metadata	Extracts meta data from documents and images.
Flickr	Look up e-mail addresses on Flickr.
Fortiguard.com	Check if an IP is malicious according to Fortiguard.com.
Fraudguard	Obtain threat information from Fraudguard.io
Fringe Project	Obtain network information from Fringe Project API.
F-Secure Riddler.io	Obtain network information from F-Secure Riddler.io API.
FullContact	Gather domain and e-mail information from fullcontact.com.
Github	Identify associated public code repositories on Github.
Google Maps	Identifies potential physical addresses and latitude/longitude coordinates.
Google	Obtain information from the Google Custom Search API to identify sub-domains and links.
Gravatar	Retrieve user information from Gravatar API.
Greynoise	Obtain information from Greynoise.io's Enterprise API.
HackerOne (Unofficial)	Check external vulnerability scanning/reporting service h1.nobbd.de to see if the target is listed.
HackerTarget.com	Search HackerTarget.com for hosts sharing the same IP.
HaveIBeenPwned	Check HaveIBeenPwned.com for hacked e-mail addresses identified in breaches.
Honeypot Checker	Query the projecthoneypot.org database for entries.
Hosting Providers	Find out if any IP addresses identified fall within known 3rd party hosting ranges, e.g. Amazon, Azure, etc.
hosts-file.net Malicious Hosts	Check if a host/domain is malicious according to hosts-file.net Malicious Hosts.
Hunter.io	Check for e-mail addresses and names on hunter.io.
Iknowwhatyoudownload.com	Check iknowwhatyoudownload.com for IP addresses that have been using BitTorrent.
Instagram	Gather information from Instagram profiles.
IntelligenceX	Obtain information from IntelligenceX about identified IP addresses, domains, e-mail addresses and phone numbers.
Interesting Files	Identifies potential files of interest, e.g. office documents, zip files.
IPInfo.io	Identifies the physical location of IP addresses identified using ipinfo.io.
ipstack	Identifies the physical location of IP addresses identified using ipstack.com.
Internet Storm Center	Check if an IP is malicious according to SANS ISC.
Junk Files	Looks for old/temporary and other similar files.
malwaredomainlist.com	Check if a host/domain, IP or netblock is malicious according to malwaredomainlist.com.
malwaredomains.com	Check if a host/domain is malicious according to malwaredomains.com.
MalwarePatrol	Searches malwarepatrol.net's database of malicious URLs/IPs.
MetaDefender	Search MetaDefender API for IP address and domain IP reputation.
Mnemonic PassiveDNS	Obtain Passive DNS information from PassiveDNS.mnemonic.no.
multiproxy.org Open Proxies	Check if an IP is an open proxy according to multiproxy.org' open proxy list.
MySpace	Gather username and location from MySpace.com profiles.
Name Extractor	Attempt to identify human names in fetched content.
NeutrinoAPI	Search NeutrinoAPI for IP address info and check IP reputation.
Norton ConnectSafe	Check if a host would be blocked by Norton ConnectSafe DNS
Nothink.org	Check if a host/domain, netblock or IP is malicious according to Nothink.org.
numpi	Lookup USA/Canada phone number location and carrier information from numpi.com.
numverify	Lookup phone number location and carrier information from numverify.com.
Onion.link	Search Tor 'Onion City' search engine for mentions of the target domain.
Onionsearchengine.com	Search Tor onionsearchengine.com for mentions of the target domain.
Open Bug Bounty	Check external vulnerability scanning/reporting service openbugbounty.org to see if the target is listed.
OpenCorporates	Look up company information from OpenCorporates.
OpenDNS	Check if a host would be blocked by OpenDNS DNS
OpenPhish	Check if a host/domain is malicious according to OpenPhish.com.
OpenStreetMap	Retrieves latitude/longitude coordinates for physical addresses from OpenStreetMap API.
Page Info	Obtain information about web pages (do they take passwords, do they contain forms, etc.)
PasteBin	PasteBin scraping (via Google) to identify related content.
PGP Key Look-up	Look up e-mail addresses in PGP public key servers.
PhishTank	Check if a host/domain is malicious according to PhishTank.
Phone Numbers	Identify phone numbers in scraped webpages.
Port Scanner - TCP	Scans for commonly open TCP ports on Internet-facing systems.
Psbdmp.com	Check psbdmp.cc (PasteBin Dump) for potentially hacked e-mails and domains.
Pulsedive	Obtain information from Pulsedive's API.
Quad9	Check if a host would be blocked by Quad9
RIPE	Queries the RIPE registry (includes ARIN data) to identify netblocks and other info.
RiskIQ	Obtain information from RiskIQ's (formerly PassiveTotal) Passive DNS and Passive SSL databases.
Robtex	Search Robtex.com for hosts sharing the same IP.
Amazon S3 Bucket Finder	Search for potential Amazon S3 buckets associated with the target and attempt to list their contents.
Scylla	Gather breach data from Scylla API.
SecurityTrails	Obtain Passive DNS and other information from SecurityTrails
SHODAN	Obtain information from SHODAN about identified IP addresses.
Similar Domains	Search various sources to identify similar looking domain names, for instance squatted domains.
Skymem	Look up e-mail addresses on Skymem.
SlideShare	Gather name and location from SlideShare profiles.
Social Media Profiles	Tries to discover the social media profiles for human names identified.
Social Networks	Identify presence on social media networks such as LinkedIn, Twitter and others.
SORBS	Query the SORBS database for open relays, open proxies, vulnerable servers, etc.
SpamCop	Query various spamcop databases for open relays, open proxies, vulnerable servers, etc.
Spamhaus	Query the Spamhaus databases for open relays, open proxies, vulnerable servers, etc.
Spider	Spidering of web-pages to extract content for searching.
SpyOnWeb	Search SpyOnWeb for hosts sharing the same IP address, Google Analytics code, or Google Adsense code.
SSL Certificates	Gather information about SSL certificates used by the target's HTTPS sites.
SSL Tools	Gather information about SSL certificates from SSLTools.com.
Storage	Stores scan results into the back-end SpiderFoot database. You will need this.
Command-line output	Dumps output to standard out. Used for when a SpiderFoot scan is run via the command-line.
Strange Headers	Obtain non-standard HTTP headers returned by web servers.
Talos Intelligence	Check if a netblock or IP is malicious according to talosintelligence.com.
ThreatCrowd	Obtain information from ThreatCrowd about identified IP addresses, domains and e-mail addresses.
ThreatExpert.com	Check if a host/domain or IP is malicious according to ThreatExpert.com.
ThreatMiner	Obtain information from ThreatMiner's database for passive DNS and threat intelligence.
TLD Search	Search all Internet TLDs for domains with the same name as the target (this can be very slow.)
Tool - CMSeeK	Identify what Content Management System (CMS) might be used.
Tool - DNSTwist	Identify bit-squatting, typo and other similar domains to the target using a local DNSTwist installation.
Tool - WhatWeb	Identify what software is in use on the specified website.
TORCH	Search Tor 'TORCH' search engine for mentions of the target domain.
TOR Exit Nodes	Check if an IP or netblock appears on the torproject.org exit node list.
TotalHash.com	Check if a host/domain or IP is malicious according to TotalHash.com.
Twitter	Gather name and location from Twitter profiles.
UCEPROTECT	Query the UCEPROTECT databases for open relays, open proxies, vulnerable servers, etc.
URLScan.io	Search URLScan.io cache for domain information.
Venmo	Gather user information from Venmo API.
ViewDNS.info	Reverse Whois lookups using ViewDNS.info.
VirusTotal	Obtain information from VirusTotal about identified IP addresses.
VoIPBL OpenPBX IPs	Check if an IP or netblock is an open PBX according to VoIPBL OpenPBX IPs.
VXVault.net	Check if a domain or IP is malicious according to VXVault.net.
Watchguard	Check if an IP is malicious according to Watchguard's reputationauthority.org.
Web Analytics	Identify web analytics IDs in scraped webpages and DNS TXT records.
Web Framework	Identify the usage of popular web frameworks like jQuery, YUI and others.
Web Server	Obtain web server banners to identify versions of web servers being used.
WhatCMS	Check web technology using WhatCMS.org API.
Whoisology	Reverse Whois lookups using Whoisology.com.
Whois	Perform a WHOIS look-up on domain names and owned netblocks.
Whoxy	Reverse Whois lookups using Whoxy.com.
Wigle.net	Query wigle.net to identify nearby WiFi access points.
Wikileaks	Search Wikileaks for mentions of domain names and e-mail addresses.
Wikipedia Edits	Identify edits to Wikipedia articles made from a given IP address or username.
XForce Exchange	Obtain information from IBM X-Force Exchange
Yandex DNS	Check if a host would be blocked by Yandex DNS
Zone-H Defacement Check	Check if a hostname/domain appears on the zone-h.org 'special defacements' RSS feed.

SpiderFoot can be also completely controlled via the command-line:

Previously, SpiderFoot was controlled exclusively through a web interface but it’s now possible to also orchestrate scans through sf.py itself via the command-line. This means you can do things like python3 ./sf.py -m sfp_haveibeenpwned -s support@spiderfoot.net to query HaveIBeenPwned? for an e-mail address.

References