Currently it is rare to see an internet user who does not have at least one account on social media sites. People use social media services to post all types of contents online such as photos, videos, text messages, but also geolocation data, mentioning their education, employment history, and the addresses where they live.



So, personal information such as social connections, places visited, habits, likes and dislikes, family members, spouse and more can all be found easily, although social networking sites allow their users to tighten their privacy controls to prevent others from seeing posted content: this large volume of accessible data about users is the essence of “social intelligence” (SOCINT).

Today, I’d like to share five tools useful for scrape informations from social platforms.

Social Mapper

Social Mapper is an Open Source Intelligence Tool that uses facial recognition to correlate social media profiles across different sites on a large scale. It takes an automated approach to search popular social media sites for targets' names and pictures to accurately detect and group a person’s presence, outputting the results into report that a human operator can quickly review.

Install and run

sudo apt install firefox-geckodriver
sudo apt-get install build-essential cmake
sudo apt-get install libgtk-3-dev
sudo apt-get install libboost-all-dev
git clone https://github.com/Greenwolf/social_mapper
cd social_mapper/setup
python3 -m pip install --no-cache-dir -r requirements.txt

Finally, open social_mapper.py and enter social media credentials into global variables at the top of the file.

Here are a couple of example runs to get started for differing use cases:

A quick run for Facebook and Twitter on some targets you have in an imagefolder, that you plan to manually review and don't mind some false positives:
python3 social_mapper.py -f imagefolder -i ./Input-Examples/imagefolder/ -m fast -fb -tw

The same as above but with the browser showing, and waiting enabled to allow a user to enter 2FA codes and manually rectify changed login processes:
python3 social_mapper.py -f imagefolder -i ./Input-Examples/imagefolder/ -m fast -fb -tw -s -w

An exhaustive run on a large company where false positives must be kept to a minimum:
python3 social_mapper.py -f company -i "Evil Corp LLC" -m accurate -a -t strict

A large run that needs to be split over multiple sessions due to time, the first run doing LinkedIn and Facebook, with the second resuming and filling in Twitter, Google Plus and Instagram:
python3 social_mapper.py -f company -i "Evil Corp LLC" -m accurate -li -fb
python3 social_mapper.py -f socialmapper -i ./Evil-Corp-LLC-social-mapper-linkedin-facebook.html -m accurate -tw -gp -ig

A quick run (~5min) without facial recognition to generate a CSV full of names, email addresses, profiles and photo links from up to 1000 people pulled out of a LinkedIn company, where the email format is known to be "firstname.lastname":
python3 social_mapper.py -f company -i "Evil Corp LLC" -m accurate -li -e "<first>.<last>@evilcorpllc.com"

Social Analyzer

Analyze & find profiles across +1000 social media/websites. It includes different analysis and detection modules, and you can choose which modules to use during the investigation process.

The detection modules utilize a rating mechanism based on different detection techniques, which produces a rate value that starts from 0 to 100 (No-Maybe-Yes).

Install and run

sudo apt-get update

#Depending on your Linux distro, you may or may not need these 2 lines 
sudo DEBIAN_FRONTEND=noninteractive 
apt-get install -y software-properties-common
 
sudo add-apt-repository ppa:mozillateam/ppa -y 
sudo apt-get install -y firefox-esr tesseract-ocr git 
nodejs npm git clone https://github.com/qeeqbox/social-analyzer.git 
cd social-analyzer 
npm ci 
npm start

Then, navigate http://0.0.0.0:9005/app.html

Sherlock

Sherlock, a powerful command line tool provided by Sherlock Project, can be used to find usernames across many social networks.

Install and run

# clone the repo
$ git clone https://github.com/sherlock-project/sherlock.git

# change the working directory to sherlock
$ cd sherlock

# install the requirements
$ python3 -m pip install -r requirements.txt

To search for only one user:

python3 sherlock user123

To search for more than one user:

python3 sherlock user1 user2 user3

Maigret

A fork of Sherlock.
Maigret collect a dossier on a person by username only, checking for accounts on a huge number of sites and gathering all the available information from web pages. No API keys required.

Install and run

# install from pypi
pip3 install maigret

# or clone and install manually
git clone https://github.com/soxoj/maigret && cd maigret
pip3 install .

# usage
maigret username

SocialRecon

An Open source intelligence tool which is used to gather an information about an person using single input like Image,name or an ip address. The tool works without any API, except virustotal.

Install and run

Currently, the official repository lacks of “requirements.txt”, so you can clone my own fork (or wait the approval of my pull request):

git clone https://github.com/andreafortuna/SocialRecon
cd SocialRecon
pip3 install -r requirements.txt
python3 socialrecon.py

References

  1. https://github.com/Greenwolf/social_mapper
  2. https://github.com/sherlock-project/sherlock
  3. https://github.com/Ramalingasamy012/SocialRecon
  4. https://github.com/soxoj/maigret
  5. https://github.com/qeeqbox/social-analyzer