======== Welcome! ======== Here we present the :mod:`ptp` (*Pentesters' Tools Parser*) project and answer the *What is it? What does it do? Why does it do it? How does it do it?* questions. The project has been developed during the `Google Summer of Code 2014, 10th edition `_, in order to create an `automated ranking system `_ for the `OWASP - OWTF project `_. OWASP - OWTF in a word ====================== The `OWASP - OWTF project `_ provides an efficient approach to combine the power of automation with the out-of-the-box thinking that only the user can provide. It gathers a complete set of plugins and merges their results into an interactive report. The user has then the possibility to add notes, to change details and to add media like screenshots in order to have a complete report. The goals aimed by :mod:`ptp` ============================= The primary goal of :mod:`ptp` is to enhance OWASP - OWTF in order to provide an automated ranking for each plugin. This will allow the user to focus attention on the most likely weak areas of a web application or network first, which will be valuable to efficiently use the remaining time in a penetration assessment. Instead of evaluating every plugins run by OWASP - OWTF and defining the rankings for each of them, thanks to :mod:`ptp`, the user will be able to focus on the ones that have been ranked with the highest risks. The user is then able to confirm or override the automated rankings since we estimate that she/he is the only one that can accurately detect the false positives. When developing the automated ranking system, :mod:`ptp`'s main goal was joined with a secondary one. Apart from its main feature which is **ranking the results from security tools reports**, it also provides an **unified way to reuse these reports directly in your python code**, without having to deal with complex parsing. .. note:: The long-term objective for :mod:`ptp` is to support all security tools and tests. But :mod:`ptp` is in its early development phase and only supports the main ones for now. .. warning:: Since v0.4, PTP relies on the fact that the supported tools are following `semantic version `_ (except observed otherwise). In other words, as long as the tool doesn't update its MAJOR version, PTP will assume that it can parse its report, reducing the maintenance cost on our side. ============ Installation ============ Using `pip `_ =================================================================== The :mod:`ptp` library is available on `PyPI `_ at the following address: `https://pypi.python.org/pypi/ptp `_. The easiest way to install it is using `pip `_. .. code-block:: bash $ pip install ptp .. note:: If an error occurs during the installation process, check your permissions. It might be required to run `pip `_ as root. From scratch ============ It is also possible to install the library from its repository. You will then be able to use the latest possible version or even try the `develop branch `_. The first step is to clone the repository of the project: .. code-block:: bash $ git clone https://github.com/owtf/ptp.git Then use the Makefile command: .. code-block:: bash $ make install =========== Basic usage =========== Auto-detection mode =================== The :mod:`ptp` module provides the :class:`ptp.PTP` class that exposes the public API of the library. The simplest way to use :class:`ptp.PTP` is with the **auto-detection mode**. This mode tries to reduce as much as possible our work by auto-detecting which tool has generated a given report and use the corresponding :class:`ptp.libptp.parser.AbstractParser`. That way, we do not need to know if the report we want to parse has been generated by `W3AF `_, `DirBuster `_ or even `Skipfish `_. Example: .. code-block:: pycon >>> from ptp import PTP >>> myptp = PTP() >>> myptp.parse(pathname='my/directory', filename='my_report') [{'ranking': 4}, ..., {'ranking': 3}, ..., {'ranking': 1}] .. note:: In the example above, the filename could have been omitted. In that case, :mod:`ptp` would have recursively walked into the directory `pathname` until a file would have matched one supported tool. For instance, we could have done: >>> from ptp import PTP >>> myptp = PTP() >>> myptp.parse(pathname='my/directory') [{'ranking': 4}, ..., {'ranking': 3}, ..., {'ranking': 1}] Be careful though, when omitting the `filename` parameter, :mod:`ptp` will stop as soon as a supported report file will be found! (i.e. :mod:`ptp` will not parse all the files in the `pathname` directory.) In order to force :mod:`ptp` to process each file that has been found, the parameter ``first`` must be set to ``False`` like below: >>> myptp = PTP() >>> myptp.parse(pathname='my/directory', first=False) If we are only looking for the highest risk that is listed in the report, we can use the following function: .. code-block:: pycon >>> myptp.highest_ranking 4 >>> from libptp.constants import HIGH >>> myptp.highest_ranking == HIGH True .. note:: To know the possible ranking values, please refer to the :doc:`libptp/constants` section. Explicit mode ============= If we already know which tool has generated the report, we can explicitly give that information to :class:`ptp.PTP`. That will even speed up the whole process since it will not have to lookup for the right parser. The list of the supported tools can be found like below: .. code-block:: pycon >>> PTP.supported { 'arachni': [], 'dirbuster': [], 'metasploit': [], 'nmap': [], , ], 'owasp-cm-008': [], 'robots': [] 'skipfish': [], 'wapiti': [ 'w3af': [], } .. warning:: The current support to Nmap does not provide any ranking yet. Refer to the :doc:`tools/nmap` section for more information. Example: .. code-block:: pycon >>> myptp = PTP('skipfish') >>> myptp.parse(pathname='my/other/directory') [{'ranking': 2}, {'ranking': 2}, {'ranking': 1}] Attributes ========== If we are interested in the name of the tool that generated the report, it is stored in the :attr:`ptp.PTP.tool_name` attribute and can be retrieved like below: .. code-block:: pycon >>> print(myptp.tool_name) arachni # In our case, it is Arachni that has generated our report. We can also retrieve the list of the vulnerabilities thanks to the :attr:`ptp.PTP.vulns` attribute: .. code-block:: pycon >>> myptp.vulns [{'ranking': 4}, ..., {'ranking': 3}, ..., {'ranking': 1}] And the metadata thanks to the :attr:`ptp.PTP.metadata` attribute. .. code-block:: pycon >>> myptp.metadata {'version': 'a.b'} ========== Unit tests ========== The :mod:`ptp` module can be tested by running following command: .. code-block:: bash $ make check .. note:: Make sure that ``make install`` has been successful before running the script. Plus, there are additional dependencies for running the unit tests suites such as `nosetest`, `coverage`, `mock` and `pyhamcrest` Example of running all tests: .. code-block:: bash make check nosetests -v -d --cover-erase --with-coverage --cover-package=ptp test_constants_high (tests.libptp.test_constants.TestLibptpConstants) ... ok test_constants_info (tests.libptp.test_constants.TestLibptpConstants) ... ok # [ omitted ] test_ptp_parse_mock_parser (tests.test_ptp.TestPTP) ... ok test_ptp_parse_no_tool (tests.test_ptp.TestPTP) ... ok Name Stmts Miss Cover Missing ------------------------------------------------------------------- ptp.py 1 0 100% ptp/libptp.py 0 0 100% ptp/libptp/constants.py 6 0 100% ptp/libptp/exceptions.py 8 0 100% ptp/libptp/parser.py 91 0 100% ptp/ptp.py 64 0 100% ptp/tools.py 0 0 100% ptp/tools/arachni.py 0 0 100% ptp/tools/arachni/parser.py 88 0 100% ptp/tools/burpsuite.py 0 0 100% ptp/tools/burpsuite/parser.py 45 3 93% 79, 88-89 ptp/tools/dirbuster.py 0 0 100% ptp/tools/dirbuster/parser.py 57 0 100% ptp/tools/dirbuster/signatures.py 3 0 100% ptp/tools/hoppy.py 0 0 100% ptp/tools/metasploit.py 0 0 100% ptp/tools/metasploit/parser.py 22 12 45% 28-29, 46-48, 58, 67-76 ptp/tools/metasploit/signatures.py 2 0 100% ptp/tools/nmap.py 0 0 100% ptp/tools/nmap/parser.py 26 15 42% 38-46, 58-62, 76-77 ptp/tools/owasp.py 0 0 100% ptp/tools/owasp/cm008.py 0 0 100% ptp/tools/owasp/cm008/parser.py 20 0 100% ptp/tools/owasp/cm008/signatures.py 2 0 100% ptp/tools/robots.py 0 0 100% ptp/tools/robots/parser.py 22 0 100% ptp/tools/robots/signatures.py 2 0 100% ptp/tools/skipfish.py 0 0 100% ptp/tools/skipfish/parser.py 100 5 95% 148, 153-155, 206 ptp/tools/w3af.py 0 0 100% ptp/tools/w3af/parser.py 68 0 100% ptp/tools/wapiti.py 0 0 100% ptp/tools/wapiti/parser.py 80 0 100% ptp/tools/wapiti/signatures.py 2 0 100% ------------------------------------------------------------------- TOTAL 709 35 95% ---------------------------------------------------------------------- Ran 137 tests in 3.562s OK (SKIP=7)