Welcome!¶
Here we present the ptp (Pentesters’ Tools Parser) project and answer
the What is it? What does it do? Why does it do it? How does it do it?
questions.
The project has been developed during the Google Summer of Code 2014, 10th edition, in order to create an automated ranking system for the OWASP - OWTF project.
OWASP - OWTF in a word¶
The OWASP - OWTF project provides an efficient approach to combine the power of automation with the out-of-the-box thinking that only the user can provide.
It gathers a complete set of plugins and merges their results into an interactive report. The user has then the possibility to add notes, to change details and to add media like screenshots in order to have a complete report.
The goals aimed by ptp¶
The primary goal of ptp is to enhance OWASP - OWTF in order to provide
an automated ranking for each plugin. This will allow the user to focus
attention on the most likely weak areas of a web application or network first,
which will be valuable to efficiently use the remaining time in a penetration
assessment.
Instead of evaluating every plugins run by OWASP - OWTF and defining the
rankings for each of them, thanks to ptp, the user will be able to focus
on the ones that have been ranked with the highest risks. The user is then able
to confirm or override the automated rankings since we estimate that she/he is
the only one that can accurately detect the false positives.
When developing the automated ranking system, ptp’s main goal was joined
with a secondary one. Apart from its main feature which is ranking the
results from security tools reports, it also provides an unified way to
reuse these reports directly in your python code, without having to deal with
complex parsing.
Note
The long-term objective for ptp is to support all security tools and
tests. But ptp is in its early development phase and only supports
the main ones for now.
Warning
Since v0.4, PTP relies on the fact that the supported tools are following semantic version (except observed otherwise). In other words, as long as the tool doesn’t update its MAJOR version, PTP will assume that it can parse its report, reducing the maintenance cost on our side.
Installation¶
Using pip¶
The ptp library is available on PyPI
at the following address: https://pypi.python.org/pypi/ptp.
The easiest way to install it is using pip.
$ pip install ptp
Note
If an error occurs during the installation process, check your permissions. It might be required to run pip as root.
From scratch¶
It is also possible to install the library from its repository. You will then be able to use the latest possible version or even try the develop branch.
The first step is to clone the repository of the project:
$ git clone https://github.com/owtf/ptp.git
Then use the Makefile command:
$ make install
Basic usage¶
Auto-detection mode¶
The ptp module provides the ptp.PTP class that exposes the
public API of the library.
The simplest way to use ptp.PTP is with the auto-detection mode.
This mode tries to reduce as much as possible our work by auto-detecting which
tool has generated a given report and use the corresponding
ptp.libptp.parser.AbstractParser.
That way, we do not need to know if the report we want to parse has been generated by W3AF, DirBuster or even Skipfish.
Example:
>>> from ptp import PTP
>>> myptp = PTP()
>>> myptp.parse(pathname='my/directory', filename='my_report')
[{'ranking': 4}, ..., {'ranking': 3}, ..., {'ranking': 1}]
Note
In the example above, the filename could have been omitted. In that case,
ptp would have recursively walked into the directory pathname
until a file would have matched one supported tool.
For instance, we could have done:
>>> from ptp import PTP
>>> myptp = PTP()
>>> myptp.parse(pathname='my/directory')
[{'ranking': 4}, ..., {'ranking': 3}, ..., {'ranking': 1}]
Be careful though, when omitting the filename parameter, ptp will
stop as soon as a supported report file will be found! (i.e. ptp
will not parse all the files in the pathname directory.)
In order to force ptp to process each file that has been found, the
parameter first must be set to False like below:
>>> myptp = PTP()
>>> myptp.parse(pathname='my/directory', first=False)
If we are only looking for the highest risk that is listed in the report, we can use the following function:
>>> myptp.highest_ranking
4
>>> from libptp.constants import HIGH
>>> myptp.highest_ranking == HIGH
True
Note
To know the possible ranking values, please refer to the Constants section.
Explicit mode¶
If we already know which tool has generated the report, we can explicitly give
that information to ptp.PTP. That will even speed up the whole process
since it will not have to lookup for the right parser.
The list of the supported tools can be found like below:
>>> PTP.supported
{
'arachni': [<class 'libptp.tools.arachni.parser.ArachniXMLParser'>],
'dirbuster': [<class 'libptp.tools.dirbuster.parser.DirbusterParser'>],
'metasploit': [<class 'libptp.tools.metasploit.parser.MetasploitParser'>],
'nmap': [<class 'libptp.tools.nmap.parser.NmapXMLParser'>],
<class 'libptp.tools.wapiti.parser.WapitiXMLParser'>,
<class 'libptp.tools.wapiti.parser.Wapiti221XMLParser'>
],
'owasp-cm-008': [<class 'libptp.tools.owasp.cm008.parser.OWASPCM008Parser'>],
'robots': [<class 'libptp.tools.robots.parser.RobotsParser'>]
'skipfish': [<class 'libptp.tools.skipfish.parser.SkipfishJSParser'>],
'wapiti': [
'w3af': [<class 'libptp.tools.w3af.parser.W3AFXMLParser'>],
}
Warning
The current support to Nmap does not provide any ranking yet. Refer to the Nmap section for more information.
Example:
>>> myptp = PTP('skipfish')
>>> myptp.parse(pathname='my/other/directory')
[{'ranking': 2}, {'ranking': 2}, {'ranking': 1}]
Attributes¶
If we are interested in the name of the tool that generated the report, it is
stored in the ptp.PTP.tool_name attribute and can be retrieved like
below:
>>> print(myptp.tool_name)
arachni # In our case, it is Arachni that has generated our report.
We can also retrieve the list of the vulnerabilities thanks to the
ptp.PTP.vulns attribute:
>>> myptp.vulns
[{'ranking': 4}, ..., {'ranking': 3}, ..., {'ranking': 1}]
And the metadata thanks to the ptp.PTP.metadata attribute.
>>> myptp.metadata
{'version': 'a.b'}
Unit tests¶
The ptp module can be tested by running following command:
$ make check
Note
Make sure that make install has been successful before running the script. Plus, there are additional
dependencies for running the unit tests suites such as nosetest, coverage, mock and pyhamcrest
Example of running all tests:
make check
nosetests -v -d --cover-erase --with-coverage --cover-package=ptp
test_constants_high (tests.libptp.test_constants.TestLibptpConstants) ... ok
test_constants_info (tests.libptp.test_constants.TestLibptpConstants) ... ok
# [ omitted ]
test_ptp_parse_mock_parser (tests.test_ptp.TestPTP) ... ok
test_ptp_parse_no_tool (tests.test_ptp.TestPTP) ... ok
Name Stmts Miss Cover Missing
-------------------------------------------------------------------
ptp.py 1 0 100%
ptp/libptp.py 0 0 100%
ptp/libptp/constants.py 6 0 100%
ptp/libptp/exceptions.py 8 0 100%
ptp/libptp/parser.py 91 0 100%
ptp/ptp.py 64 0 100%
ptp/tools.py 0 0 100%
ptp/tools/arachni.py 0 0 100%
ptp/tools/arachni/parser.py 88 0 100%
ptp/tools/burpsuite.py 0 0 100%
ptp/tools/burpsuite/parser.py 45 3 93% 79, 88-89
ptp/tools/dirbuster.py 0 0 100%
ptp/tools/dirbuster/parser.py 57 0 100%
ptp/tools/dirbuster/signatures.py 3 0 100%
ptp/tools/hoppy.py 0 0 100%
ptp/tools/metasploit.py 0 0 100%
ptp/tools/metasploit/parser.py 22 12 45% 28-29, 46-48, 58, 67-76
ptp/tools/metasploit/signatures.py 2 0 100%
ptp/tools/nmap.py 0 0 100%
ptp/tools/nmap/parser.py 26 15 42% 38-46, 58-62, 76-77
ptp/tools/owasp.py 0 0 100%
ptp/tools/owasp/cm008.py 0 0 100%
ptp/tools/owasp/cm008/parser.py 20 0 100%
ptp/tools/owasp/cm008/signatures.py 2 0 100%
ptp/tools/robots.py 0 0 100%
ptp/tools/robots/parser.py 22 0 100%
ptp/tools/robots/signatures.py 2 0 100%
ptp/tools/skipfish.py 0 0 100%
ptp/tools/skipfish/parser.py 100 5 95% 148, 153-155, 206
ptp/tools/w3af.py 0 0 100%
ptp/tools/w3af/parser.py 68 0 100%
ptp/tools/wapiti.py 0 0 100%
ptp/tools/wapiti/parser.py 80 0 100%
ptp/tools/wapiti/signatures.py 2 0 100%
-------------------------------------------------------------------
TOTAL 709 35 95%
----------------------------------------------------------------------
Ran 137 tests in 3.562s
OK (SKIP=7)