Throughout our engagements, we have often used Nessus or nmap to scan targets to find open ports and the possible services listening on those ports. However, in regards to web services, Nessus does not list if the service is http or https, and neither tool gives much information about the possible web applications running on the discovered ports. When confronted with thousands of web listeners, it can be a soul draining task to manually inspect each service. I myself find it especially frustrating when the bulk of web listeners merely redirect to either a previously inspected host or a site that is out of scope.
To automate the task of inspecting the discovered web listeners, I put together a Python script to classify them. At the time of writing, the accepted input types are Nessus nbe files, nmap gnmap files, or text files containing a list of hosts and ports in "host:port" format. With this input, the script will attempt to connect to each of the web listeners and determine several details:
- http or https
- web service banner
- are there form elements
- is a login present
We have found this python script to be useful and hope the community does as well.
The script can be downloaded from here:
Version 3.1 Change notes:
- Removed QT. It was too buggy and unreliable for our uses. Instead, phantomjs is now used.
- Added -A, which analyzes the webbies and groups them according to similarity. This method generates graphs in the form of .ps files to be later converted into jpg,png, etc.
- Generates pickle files of webbies when given the Debug flag or Analyze option. These can be reloaded into ClassifyWebbies.py using the -P option to be reanalyzed or re-screenshot the hosts without crawling.
- Fixed a ton of random bugs.