Security Tools Benchmarking

WAVSEP 2017/2018 - Evaluating DAST against PT/SDL Challenges

2017-11-10T05:59:00.001-08:00

The 2017/2018 WAVSEP DAST Benchmark:

Evaluation of Web Application Vulnerability Scanners in Modern Pentest/SSDLC Usage Scenarios

By Shay Chen

Information Security Analyst, Researcher, and Speaker

November 10th, 2017 / Updated 31/01/2018
Assessment Environments: WAVSEP 1.7, WAVSEP-EXT, Various SPA Apps
Multiple content contributions by Achiad Avivi and Blessen Thomas

1. Introduction

Two years of preparations, development and research had finally come to fruition, and the 2017 WAVSEP benchmark is finally here.

It includes extremely useful information for anyone planning to integrate DAST scanners into SDLC processes, compares numerous features of commercial and open-source solutions, and demonstrates how far these technologies advanced and matured over the last decade.

Like the benchmarks published in previous years, WAVSEP covers the main contenders in the DAST (dynamic application security testing) field, both open source and commercial.

We did a couple of things different this time.

We've been to the field, integrated, and implemented the automated-periodic execution of the various tested solutions in real-life enterprise SDLC processes to test their effectiveness - gaining some real-life experiences to better help us understand the field and the requirements -

And during at least 4-5 long-term implementations of DAST/SAST/IAST solutions in SSDLC processes for financial / hi-tech / telcom organizations,
we attempted to directly and indirectly handle modern technologies (SPA/angular/react/etc) and complex architectures to see if the various vendor-proclaimed features actually work,
and with solutions ranging from prominent open source projects to high-end enterprise commercial solutions, that processes yielded some interesting conclusions.

Some of these experiences led us to develop test cases aimed to inspect issues in proclaimed features that we noticed didn't work as expected in actual implementations, and some to the creation of comparison categories that are apparently crucial for real-world implementations.

The wavsep evaluation test-bed (now at version 1.7) was expanded with additional test cases, and a new wavsep-mirror project called wavsep-ext was created to host JSON/XML test case variants.

Before discussing the actual content, I'd like to extend my deep gratitude to the various volunteers that assisted in obtaining and compiling the vast amount of information, and even more important, in compiling it to a (relatively) readable format. I'd also like to thank the vendors, which assisted us in licensing, unrealistic response times, and by pushing us to move forward with the evaluation.

And for the curious - a simple explanation to an expected question -

Why did the publication delay so long ?

Each benchmark is typically more complicated than the previous one, primarly because of our goal to cover additional aspects in each publication, which requires us to expand the test-beds (via the development of new test cases or usage of new test-beds).

Some of these "expansions" require unproportional amount of effort -

For example -

Some of the newly implemented test cases required the scanner to both crawl javascript/jquery/ajax driven web pages, and eventually scan entry points that expect JSON/XML input being sent through those pages.

During the test, many scanners surprisingly did NOT manage to crawl the JSON/XML pages (due to lack of relevant crawling features, bugs, or perhaps our specific implementation).

So, prior to scanning, for over 250+ different JSON/XML test cases, we had to manually teach the various tested tools the structure of the XML/JSON requests and parameters, or when we got lucky and had a valid license to a scanner that could crawl these new tests cases - chain the scanner to our various tools that couldn't.

Since these licenses typically expired just when we needed them... most of that work was manual, and thus, setbacks during these assessments became common, and QA session became longer and more time-consuming.

And with that out of the way, we can start cover this year's content -

2. Benchmark Overview

2.1 List of Tested Web Application Scanners

Evaluated commercial web application scanners:

Appspider v6.14.060 (Rapid7 ltd, acquirer of NTO)
Netsparker v4.8 (Netsparker ltd.)
Acunetix v11.0.x build 171181742 (Acunetix ltd.)
Burpsuite v1.7.23 (Portswigger)
WebInspect v17.10XX (HPE)
WebCruiser v3.5.4 (Janusec)

Evaluated open source scanners:

Zed Attack Proxy (ZAP) 2.6.0
Arachni 1.5-0.5.11
IronWASP 0.9.8.6
WATOBO v0.9.22

Previously evaluated / upcoming evaluations of commercial web application scanners:

Appscan v9.0.0.999 build 466 - 2014/2015 (IBM)
Syhunt Dynamic v5.0.0.7 RC2 - 2014/2015 (Syhunt)
N-Stalker Enterprise v10.14.1.7 - 2014/2015 (N-Stalker)(Results for new benchmark tests for most of these products will be updated soon in STM)

Legacy/Inactive commercial web application scanners results were not included (but are still included in the various charts in STM):

ParosPro v1.9.12 (Milescan)
JSky v3.5.1-905 (NoSec)
Ammonite v1.2 (RyscCorp)

Previously evaluated / upcoming evaluations of open source web application scanners:

W3AF 1.6
Vega 1.0
Wapiti 2.3.0
Skipfish 2.1.0
sqlmap 1.0
XSSer 1.6.1

In regard to vendors with newer production versions that were NOT evaluated, we either initially got a license which expired mid test and then didn't manage to contact the vendor in time (the contacts we had were unreachable / no longer worked for the vendor), or the budget and deadline we had for the project required us to restrict our coverage to a limited set of vendors.

Some of the unevaluated newer product versions will be covered in the upcoming weeks, while new and previously uncovered contenders may refer to the join-wavsep section in STM.

2.2 The Evaluation Criteria

The 2017 evaluation focused on several previously uncovered aspects, in addition to "repeating" most of the tests performed on previous benchmarks:

Covering the prominent vendors in 2017, and the newly introduced technologies in the field, and their effect on various users (penetration testers, SSDLC users).
Assessing the detection ratio of previously tested vulnerabilities in modern JSON/XML input vectors - to verify the proclaimed support for the attack vector in the various categories (extremely interesting results that affect the overall scores). For that purpose, MANY test cases were re-implemented with JSON/XML inputs, posing a challenge for both scanning features (covered in the article) and crawling features (will be discussed in future article updates)
Assessing the detection of additional vulnerabilities (OS Command Injection, repurposing XSS-via-RFI test to be used for SSRF evaluations as well).
The release of a new version of wavsep evaluation test-bed, available in wavsep git-hub and source-forge repositories.

The article attempts to simplify the presentation of content, and as a result, various additional elements will only be presented and updated through the benchmark presentation platform residing at STM (full and extensive list of features, scan logs, etc).

Before we start with the raw stats, it's important to cover a few subjects related to the technologies being evaluated:

2.3 The False Positive Aspect in Penetration-Tests / SSDLC

Depending on the vulnerability assessment process, false positives will typically have the following effect:

An (arguably) minor time consuming effect in external penetration tests (in the context of a manual-driven pentest)
A major time consumer in the overall scale of periodic automated scans within SSDLC scenarios

Weeding out a reasonable amount of false positives during a pentest is not ideal, but could be performed with relative ease. However, thousands upon thousands of false positives in enterprise SSDLC periodic scan scenarios can take their toll.

Replacing DAST technologies ("scanners") in black box penetration tests, a widespread and widely consumed commodity these days, does not seem likely in the visible future.
The concept of a black-box penetration test prevents inherently "intrusive" technologies (IAST/SAST) from being used by the external assessing entity in the vast majority of scenarios.

However, in SSDLC driven assessments (secure software development life-cycle), such as periodic scanning / build-triggered scanning, DAST technologies are challenged by SAST / IAST / Hybrid technologies.

Several years of fierce competition and the adoption of additional technologies for vulnerability detection (IAST / SAST / OSS) hasn't been easy for the various vendors, and even prompted certain entities to proclaim that DAST technologies are obsolete, and/or superceded by SAST/IAST technologies.

In the long run however, competition tends to have positive impacts on technology (and quality), and in the case of DAST vendors, in addition to enhanced technology support and adaptation, the detection ratio of exposures and ratio of false positive exposures in maintained DAST solutions was improved drastically, due to enhancements and a new DAST-related technology -

2.4 Enter Out-of-Band-Security-Testing (OAST) - Overview

The technological response of DAST vendors to the enhanced accuracy of Active IAST capabilities, whether intentional or unrelated, includes (among additional enhancements) tests falling under the category of Out-of-Band security testing (sometimes coined OAST, to match the typically used naming convention), a previously understated and an unprecedentedly accurate method of identifying second order vulnerabilities ("blind"/"indirect") and reducing false positives, keeping DAST technologies well within the race of relevant technologies.

The concept of OAST tests is to inject exploitation "payloads" that correspond with verification-servers in the internet, which are able to identify and associate external access to a specific vulnerability test in a specific scan.

So, let's say for example that an Out-of-Band SQL/XSS injection payloads were used in a scan and stored in the database, they may only be "executed" at a later phase, such as when an administrator views logs of application activities that include these payloads, or when a processing script is running on database content. While "normal" scanner injection tests would have likely missed the exposure, out-of-band exploitation payloads "report" to to the external server when executed, enabling the scanner to (eventually) identify the vulnerability.

Although out-of-band payloads are not yet included in all of the relevant scan "plugins" of the various vendors, the support for these tests is becoming more extensive, at least for some of the commercial vendors.

Coupled with the other accuracy enhancement in DAST technologies (algorithm/attack-tree improvements), out-of-band-testing can provide a huge boost to both detection accuracy and to the type of vulnerabilities automated scanning solutions (DAST in this case) are able to identify.

And now, to the main point -

3. Benchmark Results

The following sections cover the results of the feature comparisons and accuracy benchmarks of the various tools assessed.

FAQ

1) Why do the results differ from previous benchmarks?

The results in the various charts represent an aggregated score of four input vectors (GET/POST/XML/JSON), as opposed to previous benchmark which only included two input vectors (GET/POST).

The newly implemented test cases are only covered for the relevant attack vectors (e.g. attacks that can be executed via JSON/XML POST requests) such as LFI/RFI/Etc., and not for (mostly) irrelevant attack vectors (XSS/Unvalidated-Redirect/Backup-Files).

3.1 Input Delivery Vector Support (Updated)

The term "input delivery vector", or rather input delivery format, refers to the structure of inputs being used in the client-server communication to deliver values from the browser/mobile/client-application to the web server.
Examples can include query-string embedded parameters (GET), HTTP body parameters (POST), JSON arrays in the HTTP body (JSON), and so on.

Since the ability to parse, analyze and simulate attacks in input delivery vectors is key to weather or not DAST scanners will be able to identify vulnerabilities relevant to the parameter, I still consider the scanner's support for the tested application input delivery method to be the single MOST significant aspect in the selection process of any scanner.

Although it's not necessary to support every possible input delivery vector, the scanner should be able to scan the prominent input vectors used in the application to be effective.

The following table presents a prominent vector-support comparison between commercial DAST vendors:

Click to Enlarge

* burp-suite requires changing the default configuration for effective AMF scan support
* burp-suite requires the "GWT Insertion Points" extension for effective GWT scan support

The following table presents a prominent vector-support comparison between open source DAST tools:

Click to Enlarge

* zap effective support of AMF scanning is unclear (a project was initiated) and requires the installation of an optional "AMF" plugin from the store
* w3af has open and active projects to develop support for REST API/JSON and AMF support. Unknown schedule/release date.
* Bugs in IronWASP JSON/XML support prevent it from effectively parsing and scanning JSON/XML inputs.May be related to improper content-types.

Although the interpretation of the results is left to the reader, it's important to note that lack of support for prominent input vectors limits the capabilities of scanners in relevant test scenarios, particularly in various payload injection tests.

The definition of prominent input vectors changes between applications, and require the tester to "profile" the input vectors in use in the application, to identify the input formats crucial for scanners to support.

It is well worth mentioning that burp-suite, zap, ironwasp and arachni (and in theory, other tools with fuzz testing capabilities) support custom input vectors (e.g. scanning ANY part of the protocol) - typically by configuring specific sections in HTTP requests (useful for limited testing of "unsupported" delivery methods). Furthermore, burp-suite and zap seem to support scanning/testing raw websockets (e.g. scanning non-HTTP protocols), which might be useful for certain assessments.

To my best knowledge currently only zap supports out-of-the-box scanning of odata-id vectors (with webinspect planning support it upcoming publications), while DOM related vectors were not evaluated in this article for any of the contenders.

Full charts will be available in the upcoming update to STM.

3.2 Support for Overcoming Modern Scan Barriers (Altered)

Ever tried to run a scanner against a website, and it "just didn't work" ?
Apart from the lack of support of relevant input vectors (JSON/XML/etc), or an ineffective crawling mechanism, there's additional "barriers" that can prevent a scanner from successfully testing a target.

Support for replaying CSRF parameters/headers, support for including multiple domains in the scope of a single scan (crucial for SPA micro service architecture), and similar key elements are required to successfully scan modern applications, particularly in the context of periodic BDD/TDD assessments.

The following table compares the scan barrier support of commercial DAST scanners:

Click to Enlarge

* burp-suite support for recording/re-performing login / in-session detection currently relies on the macro feature
* burp-suite has de-facto support of SPA with multiple domains, due to the testers ability to include any domain in scope
* burp-suite support anti-CSRF tokens via the CSurfer extension or the macro feature (Run a post-request macro)
* Acunetix support for multiple domains requires a set-up an additional Target and then adding the secondary Target as an "Allowed Host" to the first Target.
(Targets > TARGET_NAME > Advanced > Enabling "Allowed Hosts" and picking the other Target you want to include as part of the scan)
* The angular/react crawling support is based on vendor claims, and was not yet evaluated through a dedicated benchmark
* Some of the missing features can be "externally" supported, by forwarding traffic through burpsuite/zap/fiddler with auth/match-and-replace rules

The following table compares the scan barrier support of open-source DAST scanners:

Click to Enlarge

* ZAP supports re-performing authentication via built-in authentication features and zest scripts, as shown in the following article
* Subgraph vega supports authentication recording via macros
* Any scanner that has tree-view manual scan support can at least partially support scans of SPA with multiple domains
* Some of the missing features can be "externally" supported, by forwarding traffic through burpsuite/zap/fiddler with auth/match-and-replace rules
* ZAP/IronWASP angular/react crawling is possible only through browser based crawling (crawljax/etc). Requires configuration/dependencies.
* W3AF anti-CSRF plugin seems to be only partially implemented (false-negatives)

3.3 Support for Crucial SSDLC Integration Features (New!)

As opposed to manual security assessments, and performance in detection accuracy tests aside, to be able to efficiently use DAST tools in SSDLC, the scanner typically needs to support several key features -

Although NOT all of the features are required for each SSDLC integration, some can be useful or even necessary, depending on the process goals and requirements:

Defect Tracking Integration - support reporting "vulnerabilities" directly to defect tracking repositories such as JIRA/TFS/Bugzilla/Trac/etc.

Continuous Integration Support (BDD) - support for CLI/API/plugin-based scanning through external continues-integration / build-management software such as Jenkins. De-facto external support for scheduled scans.

Selenium Import/Integration (TDD) - importing crawling results or otherwise integrating with selenium scan scripts.

Periodic/Scheduled Scans - built-in scheduled scans (also possible through continues integration support through CLI/API/plugins)

Periodic Results Gap Analysis - analyze and presents results diff between scans, or otherwise compare periodic scans.

IAST Module Hybrid Analysis - although classified under different categories, some products have both DAST and IAST modules, and are further able to combine their results through scans in what is typically called Hybrid-Analysis, and integrate them in TDD/BDD scenarios.

SAST Module Hybrid Analysis - In a similar manner, either through collaboration or built-in features, some vendors (typically commercial) have both DAST and SAST modules, or are otherwise able to use "hybrid-analysis" with results of external SAST tools, potentially getting the "best" out of both worlds. DOM-focused SAST mechanisms were NOT considered full SAST modules for the purpose of this article.

Extensibility - the ability to extend the scanner with custom plugins, tests and scripts.

WAF Virtual Patch Generation - the ability to generate a virtual patch rule for a WAF out of scan results/vulnerabilities identified.

Enterprise Console Management Features - the ability to manage results in a graphical user interface - view charts/lists, mark false positives, search, import, export and classify results, etc. A full check-mark is awarded to products with homegrown on-premise solutions (to support finance/defense sector closed networks), while a half-check-mark is awarded to cloud solutions that can be used "indirectly" to scan internal solutions and presents results in an enterprise-like console, or to solutions with 3rd party enterprise console integration (e.g. threadfix, seccubus, etc).

The comparison tables attempted to present both the built-in support for features (full check-mark), and support through integration with 3rd party products (cross-check-mark).

The following table presents a comparison of built-in/external prominent SSDLC feature support in commercial DAST tools:

Click to Enlarge

Comparison notes:

* SAST modules are typically included in 3rd party products and require additional costs. IAST modules (for whatever reason, and lucky for us) are typically included in the pricing of the commercial DAST web application scanner.

* WAF virtual patching rule generation support is WAF specific and varies between vendors: Netsparker supports rule generation for ModSecurity, Acunetix supports F5/Fortinet/Imperva, AppSpider supports F5/Fortinet/Imperva/Akamai/DenyAll/ModSecurity/Barracuda, Appscan/Webinspect support virtual patch generation for various WAFs (missing list), and 3rd party interfaces such as Threadfix can create virtual patch rules for WAFs from the results of multiple supported scanners.

* Usage of external management frameworks (BDD-Security, Seccubus, ThreatFix, DefectDojo, Farady, Dradis, Jenkis + CLI) is able to "compensate" for MANY of the missing SSDLC features of supported scanners, by parsing scanning reports and converting issues to virtual patch rules/defect tracking entries, periodic result gap analysis, defining externally scheduling scans, etc. The table uses a check-mark to signify built-in feature support, and half-cross/half-check-mark for external support through 3rd-party software or plugin.

* Netsparker on-premise centrlized management frame is availble through an on-premise installation of netsparker cloud.

* In particular - burpsuite virtual patching rule generation is available through external mod-security scripts or through threatfix integration.The same applies for "indirect" defect tracking support, "enterprise-console" vulnerability management features, and scan scheduling scheduling, which is possible by combining jenkins/team-city/sonar-cube with any scanner CLI interface.

* articles describing methods of "automating" scans with burpsuite:
   https://www.securify.nl/blog/SFY20160901/burp-suite-security-automation-with-selenium-and-jenkins.html

* recent updates to burp CLI interface, external plugins makes it possible to use it in some CI scenarios - the extent is still verified by the author:
   http://releases.portswigger.net/2017/

* various external projects provide some interfaces to use burpsuite with selenium, such as: https://github.com/malerisch/burp-csj

* Methods for manual/external selenium integration with Netsparker / Burpsuite are officially documented:
   https://www.netsparker.com/blog/docs-and-faqs/selenium-netsparker-manual-crawling-web-applications-scanner/
   https://support.portswigger.net/customer/portal/articles/2669413-using-burp-with-selenium
   Similar methods could be used with webcruiser as well.

* enterprise console management features are integrated into appscan/webinspect enterprise versions. Other products may support the features either in cloud product variations or through integration with a 3rd party product (e.g. Threadfix, DefectDojo, etc).

* Webinspect/Appscan SAST modules hybrid-analysis features are actually more integrations with Fortify (webinspect) and Appscan-source. 3rd party products are also capable of performing hybrid analysis on seemingly unrelarted products.

The following table presents a comparison of built-in/external prominent SSDLC feature support in open source DAST tools:

Click to Enlarge

Comparison notes:

* Usage of external management frameworks (BDD-Security, Seccubus, ThreatFix, DefectDojo, Farady, Dradis, Jenkis + CLI) is able to "compensate" for MANY of the missing SSDLC features of supported scanners, by parsing scanning reports and converting issues to virtual patch rules/defect tracking entries, periodic result gap analysis, defining externally scheduling scans, etc. The table uses a check-mark to signify built-in feature support, and half-cross/half-check-mark for external support through 3rd-party software or plugin.

* In particular - zap / arachni / w3af / skipfish Virtual Patching rule generation is available through external mod-security scripts or through threatfix integration.The same applies for "indirect" defect tracking support, "enterprise-console" vulnerability management features, and scan scheduling scheduling, which is possible by combining jenkins/team-city/sonar-cube with any scanner CLI interface.

* arachni custom parsing of json is reported to being used to update jira defect tracking system:
https://www.newcontext.com/automate-web-app-security-scanning/

* selenium support is available/partially available for some of the tools using the following methods:
** arachni uses selenium webdrivers for login scenarios, support for importing selenium crawl results is unknown.
** Unofficial plugins for using w3af with selenium has been published: https://dumpz.org/16826/
** External IronWASP selenium Integration module: https://github.com/arorarajan/IronWaspSelenium/tree/V1
** Guides for using selenium through ZAP are available (while using various projects) - the process could theoritically be used for other scanners with proxy capabilities:
https://linkeshkannavelu.com/2015/01/08/security-test-automation-using-selenium-and-zap/
https://securify.nl/blog/SFY20160601/using_owasp_zap__selenium__and_jenkins_to_automate_your_security_tests_.html
https://www.coveros.com/running-selenium-tests-zap/

In general, although some of the tools contain built-in SSDLC related features, 3rd party software (ThreadFix / DefectDojo / Dradis / Seccubus / BDD-Security / Faraday / Jenkins) can enhance the scanner capabilities through external features and integration.

These solutions typically have either commercial and/or integration costs, but may be able to allow using most of the tools in SSDLC scenarios with some integration effort, while providing the various benefits of a commercial-scale managed platform.

3.4 The Detection Ratio of OS Command Injection (New!)

To expand the coverage of the benchmark evaluated features, dedicated test cases were implemented to simulate entry points vulnerable to OS command injection.

Unlike most of the previous benchmark evaluations, this year's benchmark included test cases with JSON/XML support (primarly implmented in the extension project wavsep-ext).
In total, the OS-command injection benchamrk included 224 NEW test cases, half of which were with JSON/XML inputs.

The following table presents the OS Command Injection detection / false-positive ratio of commercial DAST vendors:

OS Command Injection Benchmark - Commercial Vendors - Click to Enlarge

* missing results from 2-4 additional commercial vendors will be updated in the upcoming weeks.
* In order to get to a score of 100% with Netsparker, it's required to disable the "content optimization features", otherwise the score will be 45.98%.
Since WAVSEP test cases look very similar to each other (rare in real life applications), various products with scan optimization features may ignore some of the test cases since they will be categorized as "identical pages", and thus scanning may require similar configurations in other products as well.
To disable it Hold Down CTRL WHILE Clicking "Options"->Change "DisableContentOptimization" to TRUE, Save and Restart Netsparker.

The following table presents the OS Command Injection detection / false-positive ratio of open source DAST projects:

OS Command Injection Benchmark - Open Source Vendors - Click to Enlarge

3.5 The Detection Ratio of Remote File Inclusion/SSRF (Altered)

Although (XSS via) remote file inclusion (RFI) test cases were covered in previous benchmarks, in terms of exploitation, we didn't treat them as potential SSRF (server-side request forgery) vulnerable entry points, an exploitation method which is (arguably) more severe then XSS-via-RFI.

This year we included both RFI and SSRF plugins in the scans of the original "RFI" test cases, which might have affected the results.
Furthermore, as in the case of OS command injection, NEW test cases for JSON/XML inputs were implemented for SSRF/RFI, effectively doubling the number of tests from 108 to 216 valid test cases (108 NEW test cases), with the previous 6 false positive categories remaining intact.

However, at the moment, new "blind" SSRF test cases were not (YET) included in the benchmark, due to time-frame and licensing constraints, so evaluations of out-of-band SSRF detection mechanisms are still pending.

The following table presents the RFI / SSRF detection / false-positive ratio of commercial DAST vendors:

RFI / SSRF Benchmark - Commercial Vendors - Click to Enlarge

* results from previous benchmarks might be DRASTICALLY different due to the introduction of 108 NEW JSON/XML test cases.
* the GET/POST only results of Appscan, Syhunt and N-Stalker were intentionally not published as to avoid misinterpretation. The products updated results might be published in the upcoming weeks.

The following table presents the RFI / SSRF detection / false-positive ratio of open source DAST projects:

RFI / SSRF Benchmark - Open Source Vendors - Click to Enlarge

* results from previous benchmarks might be DRASTICALLY different due to the introduction of 108 NEW JSON/XML test cases.

3.6 The Detection Ratio of Path Traversal (Update)

The evaluation used the same Path-Traversal/LFI test-bed used in the previous benchmarks, which cover GET and POST input delivery vectors in 816 valid test cases, and 8 false positive categories. Due to some automation methods on our part, the interpretation of certain false-positive test cases might be more severe than in previous benchmarks.

The following table presents the Path Traversal detection / false-positive ratio of commercial DAST vendors:

Path Traversal Benchmark - Commercial Vendors - Click to Enlarge

* due to incomplete QA processes on our behalf, the results of Acunetix and Webinspect may require an update in the next few weeks.

The following table presents the Path Traversal detection / false-positive ratio of open source DAST vendors:

Path Traversal Benchmark - Open Source Vendors - Click to Enlarge

* due to incomplete QA processes on our behalf, the results of WATOBO and arachni may require an update in the next few weeks.

3.7 The Detection Ratio of SQL Injection (Update)

The evaluation used the same SQL injection test-bed used in the previous benchmarks, which cover GET and POST input delivery vectors in 136 valid test cases, and 10 false positive categories. Load related issues may have slightly affected results (a typical problem with sql injection scanning due to time based plugins / connection pool issues), but the overall results remain intact.

The following table presents the SQL Injection detection / false-positive ratio of commercial DAST vendors:

SQL Injection Benchmark - Commercial Vendors - Click to Enlarge

The following table presents the SQL Injection detection / false-positive ratio of open source DAST projects:

SQL Injection Benchmark - Open Source Vendors - Click to Enlarge

* ZAP vanilla installation gets about 75% detection, as opposed to the high result of previous benchmarks, and only yielded a result similar to previous benchmarks after installing the beta/alpha active scan plugins and configuring Low/Insane detection ratios. these additional plugins also seem to yield a significant amount of false positives.

3.8 The Detection Ratio of Reflected Cross Site Scripting (Update)

The evaluation used the same XSS test-bed used in the previous benchmarks, which cover GET and POST input delivery vectors in 66 valid test cases, and 7 false positive categories.

The following table presents the XSS detection / false-positive ratio of commercial DAST vendors:

Reflected XSS Benchmark - Commercial Vendors - Click to Enlarge

The following table presents the XSS detection / false-positive ratio of open source DAST projects:

Reflected XSS Benchmark - Open Source Vendors - Click to Enlarge

* arachni's "imperfect" score seems to be intentional - since the project removed support for VBscript related XSS tests due to their lack of relevance to modern browsers and modern websites. the only test cases currently missed by the project are VBScript XSS test cases.

3.9 The Detection Ratio of Unvalidated Redirect (Update)

The evaluation used the same unvalidated-redirect test-bed used in the previous benchmarks, and focused only on GET input delivery vectors in total of 30 valid test cases, and 9 false positive categories (vulnerable unvaliataed redirect POST entry points only contribute to phishing credibility in indirect session-storing/multi-phase scenarios, and these were not covered in the benchmark).

The following table presents the unvalidated redirect detection / false-positive ratio of commercial DAST vendors:

Unvalidated Redirect Benchmark - Commercial Vendors - Click to Enlarge

* The results of appscan DO NOT REFLECT the latest version of the product - which is still under evaluation. they have already proclaimed that they have drastically improved the result from the previous version.

The following table presents the unvalidated redirect detection / false-positive ratio of open source DAST projects:

Unvalidated Redirect Benchmark - Open Source Vendors - Click to Enlarge

3.10 The Detection Ratio of Backup/Hidden Files (Update)

The backup file results will be published in the upcoming weeks, due to vendor specific-bugs, licensing issues and time-frame constraints.

DAST vs. SAST vs. IAST - Modern SSLDC Guide - Part I

2017-05-01T09:26:00.000-07:00

Disclaimer

This article uses a relative ratio for the various charts, to emphasize the ups and downs of various technologies to the reader. It also reflects the current situation to date (which may change as technologies mature), and relies on generalization’s and estimations on capabilities of technologies, and so, must be read in the proper context.

Using a variety of vulnerability detection solutions have become widespread in software development projects, with the aim of detecting crucial vulnerabilities as early as possible.

The vast collection of technologies and tool-sets available to address the issue can dazzle even an expert, raising questions such as -

The engimous "Which technology is the best ?"

The intriguing "Do DAST/SAST/IAST complement or supersede each other ?"

And the inevitable "How to prioritize their usage and acquisition ?"

With the upcoming publication of the WAVSEP 2017 benchmark close at hand, I wanted to take the opportunity to provide my take on the role of DAST tools within the context of prominent technologies and trends in the field.

Where do they fit in? What do they excel in (compared to alternatives)? When should we use them?

With the introduction and maturity of new vulnerability-detection technologies (IAST/DAST/SAST/HYBRID/OSS), and the expected streamline of (understandably) conflicting vendor claims, users may find it hard to discern which technologies may fit their needs, how to PRIORITISE their acquisition/integration, and when’s the right time to engage each solution category.

In the following article, I will be covering a few of the key aspects in the integration of these toolsets into an SSDLC (secure software development life cycle) environment – the OVERALL EFFORT and the IDEAL TIMING of each solution category, and the benefit of SUPPORTED TECHNOLOGIES and CODE COVERAGE they provide under different cirucmstances.

In addition to being exposed to a wide variaty of tools-of-the-trade, the article can also help the reader answer some basic questions when evaluating any one of these tools.

For those of us who somehow managed to escape the terms currently in use – this article covers the following technologies:

1) DAST – Dynamic Application Security Testing – Generic and Known Web Application Vulnerability Scanners that analyze a live application instance for security vulnerabilities. To further clarify – this is the category of tools that was covered in all the previous WAVSEP benchmarks.

This article specifically focuses on DAST solutions which are actively maintained and/or SSDLC adapted, with the ability to verify potential vulnerabiliteis through some sort of Exploitation/Verification process (referred to as EV for the purposes of the article), either external or in built into the detection algorithm, as opposed to "fuzzer" like tools based primarly on algorithms that rely on identifying specific keywords in the response.

2) SAST – Static Application Security Testing – Generic & Known Application Vulnerability Code-Level Scanners that analyze source code and application configuration files for security vulnerabilities.

3) IAST – Interactive Application Security Testing – Generic and Known Application Vulnerability Debug/Memory Level Analysis Solutions that attempt to identify vulnerabilities on live application instances while also analyzing code structures in the memory and tracking the input flow throughout the application sections. This category is further divided into the following subcategories:

Passive IAST – IAST solutions that rely on traffic already being generated to identify potentially vulnerable sections, WITHOUT performing additional attack/exploit verifications (e.g. sending input with all the necessary exploitation characters, etc).
Active IAST – IAST solutions that verify potential vulnerability sinks/sources through the use of requests that verify the actual exploitability of the potential vulnerability (again, by issuing requests that contain input with all the necessary exploitation characters, or through similar means).

4) OSS - Open Source Security – the SAST equivalent of the mythological CGI-scanner – these solutions that were, for the purposes of this article, integrated into the category of SAST, due to similarity of the chart positioning and role, although these solutions operate in an entirely different manner, and focus only on the identification of “known” vulnerabilities in 3^rd party libraries.

*) The various aspects of hybrid analysis tools are NOT covered in the various article sections and charts, and the same goes for network vulnerability scanners with application level features but without SSDLC adaptation, or cloud security solutions without SSDLC integrations.

So Which Solution Category Is Most Important in SSDLC?

Technology Support vs. Code/Application Coverage

The most obvious differentiation between the various scanning solution categories is the amount of supported technologies - as in - which development languages are supported.

After comparing the various official and unoffical supported technologies proclaimed at the various vendor data-sheets, we'll quickly get to the following conclusion:

IAST solutions typically support only a handful of development technologies (2017 stats), SAST solutions can support a myrid of modern and legacy programming languages, and DAST solutions are rarely affected by the development technology -

Click to Enlarge

#

Supported Technologies

DAST

Any application with WEB/REST/WebService back-end.

Some exotic back-end listeners may be supported as well (web-sockets, DWR, AMF, etc).
The support also depends on compatibility with input delivery vectors, as well as compatible crawling OR session recording features.

SAST

Java, ASP.Net, C#.Net, VB.Net, PHP, Noje.js, Html/JS, SQL, Ruby, Pyhon, C, C++, JSP, ASP3, VB6, VBScript, Groovy, Scala, Perl, Apex, VisualForce, Android/iOS/WinMobile, Objective C, Swift, PhoneGap, Flex ActionScript, COBOL, ABAP, Coldfusion CFML

IAST

Java, .Net, PHP (few vendors), Node.js (few vendors), Ruby (few vendors), Python (experimental)

* data gathered from the proclaimed/documented features of the vast majority of existing products (May 2017)

The charts and tables signify the result of high end DAST/SAST/IAST in the industry, and obviously, some SAST and IAST solutions may support a much lower subset of technologies than the listed scope. Comparing the support for scanning non-web application variants (custom non-HTTP-based protocols) will drastically affect the chart as well.

It is also important to mention that DAST/Active IAST solutions in automated modes also need to be able to "crawl" the technology, or at the very least support the creation of recorded "sessions" of a manual crawling process, and also support sending attack payloads through the "input delivery vectors" used by the application (e.g. query-string / body / JSON/ XML / AMF / etc).

Although SAST / Passive IAST solutions also need to support "tracking" the input delivery vectors, they could, theoretically, identify hazardous code patterns without tracking the entire input-output flow, with the price of potential false positives being reported.

The difference in technology support in IAST solutions is partially related to the fact that IAST implementations are relatively new compared to DAST or SAST implementations, but also to the amount of effort required to "integrate" the IAST engine to each new technology, and furthermore, to maintain the implementation with the release of newer versions of the same supported technologies (adaptations may be required for major java JVM versions, newer .net framework versions, etc).

So, although this technology-support "GAP" may be smaller over time, the effort that will be required to maintain technology support will grow at a bigger pace, at least when compared to the pace of DAST and SAST technology compliance.

It is however, worth mentioning that most IAST vendors focus on widely used technologies that would cover as much ground as possible (Java / .Net / PHP / Node.js), and thus the actual importance of this "gap" will greatly vary for some organizations, and may even be insignificant for some.

And what of coverage ?

As it appears, being able to "support" a technology, does not necessarily allow the testing tool to automatically cover the larger portion of it's scope, which in turn, may dramatically affect the results.

The type of technologies evaluated, the method of evaluation, the code deployment format, and even the "legal" ownership of the source code libraries may affect the actual sections being covered by the tool.

The coverage criteria will be easier to understand in the form of a table, rather than a chart:

Coverage

DAST

SAST

Passive-IAST

Active-IAST

Out-Of-The-Box

Wide Coverage Min Effort

?

In Unauthenticated/ Form/Basic/NTLM

✔

Most Scenarios

?

In Tested/Used Instances

X

Depending on Implementation

End-To-End Coverage

Scan/Correlate Issues in All Client/FE/BE Layers

✔

In Client Triggered Sequences

X

Depending on Implementation

X

Depending on Implementation

?

Depending on Implementation

3^rd Party Code

Closed Source Libraries/Entry-Points

?

For “Visible” Methods

X

No DE-compilation

✔

Depending on Implementation

✔

Depending on Implementation

Dead/Blocked Code

Non-Web Executable

X

Depending on Implementation

✔

Most Scenarios

X

Depending on Implementation

X

Depending on Implementation

Conclusion:
DAST and SAST tools typically support more technologies, and as far as coverage is concerned -

DAST excels in end-to-end coverage (As in scanning the FULL CYCLE of front-end to backend) AND "visible" 3rd-party coverage, but may require manual configuration for complex applications, or at the very least, an effective crawling mechanism that supports the front-end GUI technology.

SAST excels in out-of-the-box coverage, but may lack in 3rd party software coverage (assuming it does not perform de-compilation of 3rd-party libraries), and may requires manual syncing to "identify" associated end-to-end layers. That being said, early in development, it's probably the most likely method of getting early feedback on potential vulnerabilities.

IAST will typically be positioned somewhere between the two in the various coverage categories - it will require agent distribution to support end-to-end detection (if it is supported at all), but will require less effort to achieve a wide coverage of application entry points (particularly in the case of Passive-IAST), and might have the advantage of potentially providing an in-depth coverage for CLOSED 3rd-party code/libraries.

Integration Effort vs. False Positives Effort

Throughout the development process, in both the early and later stages, the amount of effort invested in detecting vulnerabilities can, knowingly or not, play a key role in the success of the SSDLC process.

For every vulnerability detection solution and for every scenario, resources are required to integrate the chosen solutions, maintain the integration (not as easy as it sounds), and go over the results to filter high-impact and relevant issues.

Since for all the phases there’s a limited amount of human and IT resources, overly complex integrations can DELAY (or sometimes even PREVENT) the detection of security issues to a point that the benefit of detecting them early won’t apply, while complex and tedious result analysis processes can easily cause the developers to ignore identified critical issues due the sheer number of irrelevant results.

The overall effort of using each tool, is not always properly estimated by potential consumers, and for various tool categories, is focused on different areas.

Although the most obvious effort seems to be the initial integration of the vulnerability scanning process (for live instances, code, or combination thereof), the process of verifying which of the results is REAL and EXPLOITABLE, to justify mitigation effort, may be just as tedious and even more time consuming.

To put the upcoming results into proper perspective, it's crucial to understand that the relative ratio presented in the various charts is exactly that - relative, and that in fact - most modern solutions are FAR BETTER in terms of accuracy then previous generations of tools (early DAST / early SAST / fuzzers / parsers). To further emphasize the perspective, assume the accuracy of modern tools falls in the following context when compared to that of previous generation tools:

* some MODERN solutions may do much better than in the chart generalizations (and some obviously worse)

And now, when the relative scale has been clarified, we can begin to compare modern technologies against each other.

Ease of Integration and Maintenance

Integrating a vulnerability detection solution may require different steps, depending on the complexity of the application being assessed.

The requirements will often include defining the scope of the test, providing necessary scan data (credentials / etc), configuring a scan policy, deploying an agent, etc.

The following table describes and assigns various integration/maintenance prerequisites to the relevant solution categories (DAST / SAST / IAST):

	DAST	SAST	Passive IAST	Active IAST
Deploy Application Deploy application Instance or scan an existing instance Common	✔ Required	X Not Required	✔ Required	✔ Required
Configure App URL Explicitly define scan target in scan policy/configuration Common	✔ Define URL in Scan Policy	✔ Source Control Path	? Licensing Purposes	✔ Define URL in Scan Policy
Define Credentials Define credentials / login scenarios in the scan policy Common	✔ Define Credentials AND Login Sequence	? Define Source Control Credentials (Optional)	X Not Required	✔ Define Credentials AND Login Sequence
Install an Agent Install a “scan” agent on the tested system framework Common	X Not Required	X Not Required	✔ Required	✔ Required
Define Scan Exclusions Configure “forbidden” pages and parameters to scan (logout, delete, etc) Situational	✔ Possible, Policy Specific	X Not Required	X Not Required	✔ Possible, Policy Specific
Handle Scan Barriers Configure/flag anti-CSRF parameters, custom tokens, complex login scenarios (micro-services), or record multi-phase processes Situational	✔ Possible, Policy Specific	X Not Required	X Not Required	✔ Possible, Policy Specific
Create Test Policies Define test plugins, "record" traffic, set threads, custom pages, and input vectors Situational	✔ Possible, Policy Specific	X Not Required	? Coverage Optimization	✔ Possible, Policy Specific
Aggregate Sources Aggregate sources from key homegrown libraries (fragmented source projects) Situational	X Not Required	✔ Possible, Policy Specific	X Not Required	X Not Required
Maintain Policies Update scan policies in major changes in relatively complex applications Situational	✔ Possible, Policy Specific	? Source Code Path / Code Fragmentation	? Agent Re-install	✔ Possible, Policy Specific

To simplify the analysis of the data, we will present the integration / maintenance requirements in charts describing two main scenarios:
1) A typical scan of an unauthenticated OR common application with FORM/HTTP authentication,
2) A scan of a (relatively) complex application, with micro-service architecture, scan barriers (anti-CSRF mechanisms, multi-phase processes, etc), or similar complex prerequisites:

Click to Enlarge

From the point of view of integration, some tools are easier to integrate than others, some have very little or no effort required for maintenance, and some require a specific scan policy in order to maximize the result efficiency.

To summarize the topic :

SAST - In an environment without any live application instances, SAST solutions can still be used to scan source code repositories, either directly or through the upload of source code projects, simplifying the initial assessment process, and making the integration of SAST relatively simple compared to alternatives.
Passive IAST - In an environment with live application instances, IAST solutions can be integrated simply by deploying an agent to the assessed application baseline framework. Although the initial integration may be difficult (objection from developers/QA, dedicated servers, configuration, potential performance issues in shared environments, etc), once the solution is set up there's very little maintenance .
DAST / Active IAST - In an environment with live application instances, DAST solutions can be easily used to scan unauthenticated applications and will require minor configuration to scan applications with FORMS/HTTP authentication. More complex authentication methods, scan barriers (e.g. anti-CSRF mechanisms) or diverse architectures (micro-service architecture, REST, WS, etc) may require the creation of a dedicated policy and/or manual crawling session recording, while in the case of Active IAST solutions, in addition to requiring all the DAST prerequisites, will also require deployment of agents to the various tested layers.

Or in short, SAST solutions are probably easier to integrate, DAST and Passive IAST compete in terms of ease of integration, and Active IAST typically requires more effort to integrate than other solution categories (but provides adequately accurate results, as seen in the next section)

Effort of Weeding Out False Positives

The relative ratio of false positives derives from a number of methods that can be used by each technology to verify that identified vulnerabilities are not false positives, in addition to the individual solution efficiency:

Click to Enlarge

The justification for the chart diversity of the various solution categories stems from the verification methods that can be used by each solution category:

	DAST	SAST	Passive IAST	Active IAST
Execution URL Client-Driven Exploitation Entry-Point-To-Vuln-Code	✔ Exploit URL / Payload	x / ✔ Framework Dependent	✔ Exploit URL	✔ Exploit URL and Payload
Execution CLI Command Line Exploitation CLI-Param-To-Vuln-Code	X CLI Not Supported	✔ CLI Entry Point Detected	? Theoretically Possible	? Only via Passive
Flow/Taint Analysis Track Sequence of Methods to Activate Vulnerable Code	? Irrelevant for Technology	x / ✔ Key-Word Dependent	✔ In Effect	✔ In Effect
Input Effect on Sink Track Live Input Effect on the Vulnerable Code	? Through Binary Methods	X Not Performed	✔ Commonly Used	✔ Commonly Used
Modified Input Effect Track Modified Input Effect on the Vulnerable Code	✔ Payload Effect Analysis	X Not Performed	X Not Performed	✔ Payload Effect Analysis
Execution POC Time Delay, External Access, Browser Effect, Response Diff	✔ Commonly Used	X Not Performed	X Not Performed	✔ Commonly Used
Exploitation POC Full Scale Exploitation: Data Extraction, RCE, Shell Upload	✔ In Some Solutions	X Not Performed	X Not Performed	✔ In Some Solutions

Additional importance is given to the information the various tools provide to a HUMAN trying to discern the relevance of the issues reported, and tool-set (if any) provided to "reproduce" or manually "verify" the identified security issues.

Furthermore, the false positive factor will become EVER MORE IMPORTANT with the increase in volume of scanned applications. Weeding out false positive from actual issues will require time and effort from a security expert, and any misinterpretations will cost even more for developers to mitigate.

To complete the picture for the various technologies, let's present the RELATIVE integration/maintenance effort vs. the *typical* false-positive effort required to identify actual issues throughout the analysis cycle:

Click to Enlarge

So there you have it, technology and coverage, integration and false positive ratio for various *typical* modern technologies, pitted against each other.

And if the choice still seems complicated, that's ok, it's because it is.

There’s always the exception

A relative obsolete and un-maintained DAST vulnerability scanner, in which there is little or no effort to “verify” detected vulnerabilities, will fare no better, and probably much worse, than a typical SAST/Passive-IAST solution, in terms of the ratio of false positive identification.

On the other hand, a relatively immature or un-maintained SAST/Passive-IAST solution will fare much worse than presented in the charts - even in the effort required for integration and maintenance, especially compared to a modern DAST implementation.

And what of other aspects ?
The ability to detect common and exotic issues, the performance scale, the RISKS applied to using each technology in different environments, and the overall recommendation ?

Part II Coming Soon...

WAVSEP Updates, FAQ and the 2015 Benchmark Roadmap

2015-09-15T21:43:00.001-07:00

A couple of updates on the WAVSEP 2015 benchmark:

The 2015 benchmark is already ongoing, and I started testing scanners against a newer unpublished version of WAVSEP which will be published at the end of the benchmark.

I'll be focusing on the usual commercial and actively maintained open source contenders, but may include additional vulnerability scanner engines that match my criteria or join the comparison in one of the methods listed in SecToolMarket.

WAVSEP New Homepage

As of August 2015, WAVSEP has been official migrated to github, and the various installation instructions have been migrated to the relevant github wavsep wiki (installation / features).

The source code, builds and wiki will be maintained in github, but I'll be releasing builds to wavsep sourceforge repository as well.

Just to clarify - both repositories currently contain the latest public version of WAVSEP.

About the Upcoming Benchmark

The benchmark will cover all the previously covered aspects, as well as 2-3 additional attack vectors, and 2-3 new measurement concepts. Its the biggest one so far, but hopefully, I'll find smarter methods of assessing the products to speed up the process.

As mentioned before, to make the results useful earlier, I'll be publishing some of the results during the testing to SecToolMarket, and tweet when there's updates to the various engines, instead of waiting to the end of the benchmark.

Vulnerability Scanner Feature Mapping to RvR

The plan is to eventually associate the various features assessed in WAVSEP with a new project called RvR (relative vulnerability rating), currently hosted in the following address, aimed to define identical classifications of features for comparing security products.

The RvR list still includes 288 (!) attack vectors with videos, links, etc, but there's already 60+ additional attacks pending to be added, contributed by volunteers from around the globe.

Trying to Release an Initial WAFEP Benchmark

WAFEP (Web Application Firewall Evaluation Project), WAVSEP's evil WAF testing brother, is almost ready for initial release, with thousands of proven WAF bypass payloads ready.
However, I'm trying to release an initial benchmark with the framework, covering 2-5 WAF engines to make my point.

Its tricky to stuff these projects in the same timeframe, and WAVSEP is my current priority, but we'll see how it works - WAFEP is designed to take a lot less testing time.

In any event, I'll tweet about additional updates and whenever I update the results.

Cheers

RvR, WAFEP and WAVSEP results update

2015-01-18T15:06:00.000-08:00

Most of my time these days is spent on creating a dynamic interface for updating benchmark results, and on two major projects aimed at enhancing the WAVSEP evaluations and adding additional comparison content, in addition to accuracy, crawling and automation.

The first project, RvR (Relative Vulnerability Rating), is a project I already mentioned in the past which merges vulnerabilities from well known vulnerability classifications (WASC, CWE, CAPEC, OWASP, Blogs, Conferences, etc) into a list customized specifically for product feature evaluations.

The list, originally planned to include 233 attack vectors, already includes 284 (!!!) different attack vectors with unique classifications, links, repository mapping and videos,
A web site containing the content was published last week, and although all the content is very much usable, I'm still delaying the publication until I get some vendor feedback (expect an official publication soon).

The purpose of the project is not only to evaluate features of dynamic vulnerability scanners (DAST), but also to cover source code analysis tools (SAST), interactive application testing tools (IAST), and in contrast to the past - various software protection products, including application-level IDS/IPS mechanisms and web application firewalls (WAF).

Which leads me to the second project -

WAFEP - The Web Application Firewall Evaluation Project

WAFEP is an upcoming project aimed to serve a WAVSEP-like role for various application-level protection products.

Unlike WAVSEP, WAFEP is planned on being completely automated in terms of payload execution AND result calculation, and would enable the evaluation of web application firewalls in relatively short timeframes.

The "accuracy" aspect is implemented as attack vector specific payloads meant to simulate context-specific exploits that an IDS/IPS/WAF should identify and/or prevent, false positive scenarios that should not be identified, and in the future, evasion techniques that may circumvent the detection process.

The project already includes thousands of payloads imitating flavors of +-10 high-impact attack vectors, some of which were already published in an early alpha version uploaded to the project source forge repository last week.

The published alpha version is just a technology POC, and does not include most of the vector payloads or content, but in the upcoming weeks I'll make an effort to finish up some sections in the platform and release a v1.0 public version.
I'll also publish updated versions with relevant payloads in the meantime, at least until I reach the 1.0 goal.

http://sourceforge.net/projects/wafep/

WAVSEP Results Update

Finally, from time to time, I still try to squeeze in additional WAVSEP product assessments for additional vendors, the latest of which is Tinfoil Security, alongside certain version upgrades,

As always, the full list is found in SecToolMarket, and the following image summarizes the updates:

If all goes well, in the near future, the list will be updated with the results of a couple of more.

I didn't update the results of any of the open source products, and will try to find the time to do so in the near future, at least for some of the projects - a task that should be much easier once the dynamic interface is finally online.

EL 3.0/Lambda Injection: Hacker Friendly Java

2014-12-17T04:13:00.003-08:00

The following article explains the mechanics of a code injection attack called EL3 Injection in applications that make use of the relatively new EL3 processor in java.

New mechanics and operators introduced in EL3 make the discovery and exploitation of this exposure almost as easy and seamless as SQL Injection, and the impact of the vulnerability is severe, with potential impacts such as denial of service, information theft and even remote code execution.

Since the EL3 technology is relatively new it's probably not (YET) as common as other severe exposures, but at the very least, it will put a big wide THEY DID WHAAAAT!? smile on your face.

[Note –
The following article discusses a generic application-level coding flaw in modern Java applications, NOT a java 0-day.

Keep on reading – the juicier RCE payloads are presented at the end]

While trying to (and miserably failing at) create a training kit for EL Injection (or Spring EL Injection, JSR245, if you will), published by Stefano Di Paola and Arshan Dabirsiaghi, I spent some time trying to get a working build of the eclipse-based STS IDE version which supported the vulnerable Java Spring MVC versions (Spring 3.0.0-3.0.5).

Turns out that someone did a REALLY GOOD job eradicating every trace of the vulnerable builds, leaving only time consuming options of compiling the environment from scratch.

Luckily, at some point, I decided to take a short break, and read about the relatively new EL in Java (JSR341, not necessarily in Java Spring) – and found something VERY interesting.

Turns out that the newest java expression language version, EL 3.0 (published sometime in 2013), includes multiple enhancements, such as operators, security restrictions on class access, and so on.

A typical source code sample of using EL3 in a Servlet or JSP page would look something like:

<%@page import="javax.el.ELProcessor"%>

…

ELProcessor elp = new ELProcessor();

Object msg = elp.eval("'Welcome' + user.name");

out.println(msg.toString());

The ELProcessor dynamically evaluates the EL statement, and attempts to access the "name" fields of the Bean (or registered class) user.

After taking a couple of shots at "guessing" objects that might be accessible by default, I stumbled on one of the features that can be used to define access to classes in EL3, which includes the ELManager class methods importClass, importPackage and importStatic.

These methods could be used to "import" various classes and even packages into the scope of the expression language, so they could be referenced within expressions.

So in order to use classes in EL3 expressions, you'll need to include them using statements such as –

elp.getELManager().importClass("java.io.File");

This feature was implemented due to safety concerns (or in other words, security), to make sure that access to classes is presumably prevented for any class that was not also included in the page/project original EL imports AND application imports, so that even if developers will enable user input to affect the "importPackage" or "importClass" statements, the external effect will be limited to the classes already imported in the context.

However, since many interesting classes and packages are typically used in Servlets and JSP pages, an attacker can still abuse this feature in multiple scenarios –

(1) If the developer already imported a class that the attacker needs into the EL context, and an attacker controlled input is used within the expression evaluation:

Input1 = "File.listRoots()[0].getAbsolutePath()"

<%@page import="javax.el.ELProcessor"%>

<%@page import="javax.el.ELManager"%>

…

String input1 = request.getParameter("input1");

ELProcessor elp = new ELProcessor();

elp.getELManager().importClass("java.io.File");

Object path = elp.eval(input1);

out.println(path);

(2) If the developer enabled the user to control the importClass/Package statement (no limits to human stupidity, right?), and already has a wide enough scope imported in the page/application imports:

Input1 = "File.listRoots()[0].listFiles()[1].getAbsolutePath()"

Input2 = "java.io.File";

<%@page import="javax.el.ELProcessor"%>

<%@page import="javax.el.ELManager"%>

…

String input1 = request.getParameter("input1");

String input2 = request.getParameter("input2");

ELProcessor elp = new ELProcessor();

elp.getELManager().importClass(Input2);

Object path = elp.eval(input1);

out.println(path);

So, here you go.
A nice exploit that will probably affect a couple of desolate apps, with super insecure code. Hardly worth its own classification.

However, while trying to squeeze some more juice out of the potential attack vector, I stumbled upon the following video, which explains the features of EL3 in great details.

To make a long story short, watch the video and skip to 7:52.

It's well worth your time.

Turns out that despite the security restrictions that required developers to explicitly import classes and packages to be used in the EL3 scripts, the java.lang package was included by default, to enable the typical developer to gain access to static type object and methods such as Boolean.TRUE and Integer.numberOfTrailingZeros.

They enabled access by default to the static members of classes in JAVA.LANG, as in the java.lang package that includes java.lang.System and java.lang.Runtime!

JAVA.LANG!

Seems like somebody there confused "user friendly" with "hacker friendly" J

So, if for some reason, a user controlled input would stumble into an EL3 eval clause, which for some reason java is encouraging users to use in many platforms such as JSF, CDI, Avatar and many CMSs, than attackers could do a LOT more with no requirements on specific imports -

Input1 = "System.getProperties()"

<%@page import="javax.el.ELProcessor"%>

<%@page import="javax.el.ELManager"%>

…

String input1 = request.getParameter("input1");

ELProcessor elp = new ELProcessor();

Object sys = elp.eval(input1);

out.println(sys);

Also, Instead of using the System class, we can use the Runtime static class methods to execute shell commands. For example:

Input1 = "Runtime.getRuntime().exec('mkdir abcde').waitFor()"

<%@page import="javax.el.ELProcessor"%>

<%@page import="javax.el.ELManager"%>

…

String input1 = request.getParameter("input1");

ELProcessor elp = new ELProcessor();

Object sys = elp.eval(input1);

out.println(sys);

An impact similar to that of the Spring's counterpart of EL injection, only in mainstream Java.

Cool. Now we can shamelessly classify the attack and rest.

But there's more!

Although scenarios in which the user's input will get full control of the entire EL string are possible, they are much less common than scenarios in which user input might be integrated as a part of an EL string, in which case most of the previously mentioned payloads won't work.

However, EL 3.0 was kind enough to present us with NEW operators, one of which is the infamous semicolon (;).

As its SQL counterpart functionality suggests, the semicolon delimiter can be used in EL 3 to close one expression, and add additional expressions, with or without logical relations to each other.

Think adding multiple lines of code to a single attack payload. Think injecting payloads into the middle of expression, while using techniques similar to blind SQL injection.

Don't think. Here's a couple of examples:

Input1 = "; Runtime.getRuntime().exec('mkdir aaaaa12').waitFor()"

<%@page import="javax.el.ELProcessor"%>

<%@page import="javax.el.ELManager"%>

…

String input1 = request.getParameter("input1");

ELProcessor elp = new ELProcessor();

Object sys = elp.eval(("'Welcome' + input1);

out.println(sys);

Input1 = "1); Runtime.getRuntime().exec('mkdir jjjbc12').waitFor("

<%@page import="javax.el.ELProcessor"%>

<%@page import="javax.el.ELManager"%>

…

String input1 = request.getParameter("input1");

ELProcessor elp = new ELProcessor();

Object sys = elp.eval(("SomeClass.StaticMethod( + input1 + ")");

out.println(sys);

So due to the implementation of the semicolon operator, potential injections can now CLOSE PREVIOUS STATEMENTS and start new statements, making the potential injection almost as usable as SQL injection. Features such as EL variable declaration, value assignments and others (watch the video) just add more fuel to the fire.

So much for enhanced security features.

We already identified a few instances that affect real world applications (no instances in core products, so far), and are currently handling them infront of the relevant entities.

I'll probably invest some more time in the upcoming weeks to see if any prominent java projects are prone to this issue, but in the meantime, some practical notes:

Regardless of how common these issues are, these potential exposures could easily be identified in code reviews or by source code analysis tools that track the effect of input on the various methods of the ELProcessor class, and on similar EL related classes.

Generic blind injection payloads can be added as plugins for automated scanners, and we could go bug hunting to see if any more of these potential issues exists in the wild.

The mitigation is also simple, not embedding input into EL statements and validating input in case you do.

I'll update this post as the research progresses.

Cheers

Relative Vulnerability Rating (RvR.) and WAVSEP Results Update

2014-11-03T14:42:00.001-08:00

[For a list of the most updated open source, commercial & SAAS scan results click HERE]

The 2014 vulnerability scanner benchmark included a lot of content, but not nearly as much as I originally planned to publish.
My original aim was to add additional comparison aspects, and provide an initial formula for measuring the VALUE of vulnerability scanners, and infosec products in general.

Despite the help I got from volunteers and multiple kind souls, due to the fact that the new comparison aspects were progressing a bit slower than I hoped, I decided to release the content in February 2014, and process the rest of the data later.

Its been more than 8 month of development, and although I can't claim all the content is ready for release, a significant portion of it is -

A security-product-oriented vulnerability classification called RvR.

RvR - Relative Vulnerability Rating

RvR includes a comprehensive collection of GENERIC application-level attack vectors (e.g. sql injection, xss, etc), gathered from every prominent resource out there, and classified based on a DETECTION vs. PREVENTION approach.

The list currently includes the incomprehensible number of 233, or to be exact, 233 generic ways to hack applications, that security products may be able to perform or prevent.

Unlike OWASP Attacks & Vulnerabilities, CWE, CAPEC, and WASC Threat Classification,
The RvR threat classification aims to provide a common ground for measuring the value of security products, starting with the security products in the application security field.

Each generic exposure in the classification is mapped to other prominent vulnerability classification lists, informative resources, videos, as well as a relative severity, measured in relation to other RvR items, aimed to classify the importance of supporting features in each product.

To make things simpler - it will enable measuring how much a security product covers in comparison to the whole collection of possible generic hacking methods, and in comparison to other products, and drill down into various quality aspects.

I invested most of last few months implementing a framework for an online website infrastructure to present the RvR and WAVSEP data, and I'm pretty close to the finish line.

The list has already been distributed to vendors that asked to view it in advance, and will be published as soon as I make some adjustments to the initial website version.

In the meantime,
I managed to update some of the comparison results with a few updated product versions, and will try to find the time to update some more.

Vulnerability Scanner Stats (latest tested versions):

WAVSEP 2014 Results Update

2014-03-29T13:05:00.001-07:00

After the benchmark publication, several vendors contacted me with recommended configurations that could enhance their score, and with feature documentation corrections.
After testing the various provided configurations, I was able to update the various charts and data in the benchmark original post, as well as the various charts in sectoolmarket.

Update summary:
The WIVET score of Webinspect was slightly improved from 94% to 96% by selecting the "depth first" mode in the scan wizard (the default configuration still yields 94%), which makes it the official winner of the WIVET category.

The path traversal detection score of arachni was updated from 30.88% to 100% (!!!) by making use of the source code disclosure plugin (as suggested by the vendor, in addition to the path traversal and local file inclusion plugins), which makes it the co-winner in this category, alongside Appscan.
The LFI detection results of Webinspect were likewise improved from 72.06% to 91.18%, by using vendor recommended configuration that included the following plugins: 10287 – Local File Include, 10271 – Local File Inclusion/Reading Vulnerability, 10272 – Possible Local File Inclusion/Reading Vulnerability, 11327 – LFI Tomcat, 11332 – LFI IIS

Finally, the list of supported input vectors was updated after the Appscan team reported support for 4 more vectors, the ZAP project reported support for additional two input vectors, and the arachni project reported support for one additional vector. All updates represent support in the tested versions.

There may be some minor updates to the SQL injection results of one scanner - if the vendor provided configuration will work.

As mentioned earlier, the benchmark charts already reflect the changes, and summarizing content will be published soon.

WAVSEP Web Application Scanner Benchmark 2014

2014-02-05T22:24:00.000-08:00

WAVSEP 2013/2014 Score Chart:

The Web Application Vulnerability Scanners Benchmark

Commercial, SAAS & Open Source Scanners

An Accuracy, Coverage, Versatility, Adaptability, Feature and Price Comparison of 63 Black Box Web Application Vulnerability Scanners and SAAS Services

Part I

By Shay Chen

Information Security Researcher, Analyst, Tool Author and Speaker

Sponsored by

Multiple content contributions by

Ozhan Sisic and Sharath Unni

February 2014

Assessment Environments: WAVSEP 1.5, WIVET v3-rev148, ZAP-WAVE (WAVSEP integration), various undisclosed verification platforms

https://code.google.com/p/wavsep/ , http://www.sectoolmarket.com/

sectooladdict-{at}-gmail-{dot}-com

Table of Contents

1. Introduction

2. List of Tested Web Application Scanners

3. Benchmark Overview & Assessment Criteria

4. A Glimpse at the Results of the Benchmark

5. SURPRISE, SURPRISE!

6. How to Read and Use the Results - IMPORTANT

7. Test I - Scanner Versatility - Input Vector Support

8. Test II - WIVET - Coverage via Automated Crawling

9. Introduction to the Various Accuracy Assessments

10. Test III – The Detection Accuracy of Unvalidated Redirect (NEW!)

11. Test IV – The Detection Accuracy of Backup/Hidden Files (NEW!)

12. Test V – The Detection Accuracy of Path Traversal/LFI

13. Test VI – The Detection Accuracy of RFI (XSS via RFI)

14. Test VII – The Detection Accuracy of Reflected XSS

15. Test VIII – The Detection Accuracy of SQL Injection

16. Test IX – Attack Vector Support – Counting Audit Features

17. Test X – Scanner Adaptability - Crawling & Scan Barriers

18. Test XI – Authentication and Usability Feature Comparison

19. Test XII – The Crown Jewel - Results & Features vs. Pricing

20. Additional Comparisons, Built-in Products and Licenses

21. What Changed?

22. Initial Conclusions – Open Source vs. Commercial

23. Verifying The Benchmark Results

24. So What Now?

25. Recommended Reading List: Scanner Benchmarks

26. Acknowledgments

27. Appendix A – List of Tools Not Included In the Test

1. Introduction

Detailed Result Presentation at

http://www.sectoolmarket.com/

Tools, Features, Results, Statistics and Price Comparison

(Delete Cache Prior to Viewing)

A Step by Step Guide for Choosing the Right Web Application Vulnerability Scanner for *You*

infosec-island

It is fashionably late, but the time eventually came.

Months and months of research finally came to fruition with the publication of the yearly WAVSEP benchmark, the fourth one in the series.

It's been a very exciting year for the project… with many new things happening.

I'd like to share some of those, as they can put in perspective how the project is progressing:

I've noticed the project was included in many continues integration processes of various commercial vendors, and lately, even in similar processes of open source projects (for example – ZAP).

The same commercial vendors, as well as colleagues and people I met in conferences around the world, brought to my attention that various government institutes and agencies worldwide use the platform as an assessment platform for vulnerability scanners, often as the main one.

I got contacted by many organizations in the financial and technology sector that asked me to help them do the same, and found some time to enhance the platform for that purpose.

I also received source code contributions from multiple project and individuals, as well as support from volunteers, feedback, and plenty of inspiration.

I even began receiving phone calls, and on multiple occasions, from "angels", relevant companies and investors around the globe, that wanted to know whether or not to invest in vulnerability detection initiatives and products.

With all the support, contribution, and data collected in this research over the years, I believe that soon a subject that still remained obscure could finally be determined –

A simple process that will enable to evaluate the customized ROI per product:

The Return of Investment (ROI) from each product in the category

While were definitely not there yet, not in this article anyway, with each publication, there's less and less missing pieces, and the data collected while preparing for this publication closed a significant portion of the gap.

The assessment covers 12 different aspects of the tools (or 16, if you consider non competitive charts), including two new attack vectors they were not assessed in the past (!), and this time, they were all assigned with recommended priorities that readers can use for evaluation.

The research also managed to finally breach the traditional level 60 cap (the best metaphor a gamer could come up with at 5AM) and add three additional products to the assessment, to a total of 63 different web application vulnerability scanners, including some that were never assessed in the past, and with potential to add more in the near future.

These include a total of 14 commercial products and SAAS services, as well as 49 free and/or opensource projects.

Following its tradition, the research focused on the main module which is usually associated with term "web application vulnerability scanner", and this time, it is in our interest to define this module properly, as well as the difference between it and other modules that may be associated to the same title.

Although the term "web application scanner" meant different things over the years, I believe that dividing its various functionalities into modules, can help understand the focus of this research, as well as properly classify and evaluate the contribution of the various modules in the future.

Since I didn't find any dominant classification, I am going to use a descriptive one for the purpose of this research.

Black-Box web application scanners may contain any of the following modules:

(*) Generic Application-Level Vulnerability Detection Module: a collection of features that attempt to identify generic exposures in the application layer, without prior knowledge about the application and its structure, and while potentially overcoming barriers along the way. This module is the primary focus of this research.

(*) Known Application-Level / Web Server Vulnerability Detection Module: Commonly classified as a CGI scanner (a bit old school for my taste), or a web server scanner, but often using the same classification as the above module – the collection of features that falls under that category attempts to identify vulnerabilities that are known (and/or were published) in a shelf product. This module is NOT covered by this research

Additional modules may include "Generic Vulnerability Exploitation Module", "Known Vulnerability Exploitation Module", a "web site infection" detection module, and others. These too, are not covered by this research, and although many of the tested projects/products contain a couple of types, they are also often implemented in separate products.

So now that were done clarifying and classifying, as always, one last tip:

A lot of the information gathered in this research cannot be presented in graphs, so if you're seeking for the more significant content, you'll have dig in past the charts and graphs. If you're reading 3 graphs and can already declare a winner, you're missing some good stuff along the way.

Try the sections in the main menu with all the fancy words beside them… they usually do the trick.

Update:

During the assessment of Qualys it is highly likely that an optimization mechanism affected the scan results of POST test cases (compared to WAVSEP 2012 results). Although in the case of other vendors disabling similar mechanisms solved the problem, in the case of Qualys this optimization mechanism could not be disabled via the configuration interface. We are currently trying to find solutions to the problem.

2. List of Tested Web Application Scanners

The following commercial scanners were covered in the benchmark:

*Acunetix WVS* v9.0 build 20140113 (Acunetix)	*NTOSpider* v6.0 builds 773/778 (NT OBJECTives)	*Netsparker* v3.1.7.0 (Netsparker Ltd, p.k.a Mavituna Security)
*IBM AppScan* v9.0.0.999 & v8.8.0.0 build 466 (IBM)	*WebInspect* v10.1.177.0 SecureBase 4.11.00 (HP)	Syhunt Dynamic v5.0.0.7 RC2 (Syhunt)
*Burp Suite* v1.5.20 (Portswigger)	*N-Stalker Enterprise Edition* X, build 10.13.11.31 (N-Stalker)	*WebCruiser* v2.7.0 EE (Janus Security)

The following SAAS services were assessed in the benchmark:

Qualys WAS - during January 2014 (Qualys)

ScanToSecure - during January 2014 (Netsparker Ltd)

The previous results of the following commercial scanners were included in the benchmark, since they were not updated since the previous benchmark (website):

ParosPro v1.9.12 (Milescan)

JSky v3.5.1-905 (NoSec)

Ammonite v1.2 (RyscCorp)

The following commercial scanners will be updated soon:

Nessus (Tenable Network Security) - Web Scanning Features

The latest versions of following free/open source scanners were re-tested:

*Zed Attack Proxy (ZAP)* v2.2.2 (OWASP)	*IronWASP* v 0.9.7.4 (Lavakumar Kuppan)	*W3AF* v1.6 revision 5460aa0377 (The W3AF team)
*arachni* v 0.4.6 (Tasos Laskos)	*Skipfish* v2.10b (Google)	*WATOBO* v 0.9.19 (Andreas Schmidt)
*VEGA* v1.1 beta build 108 (Subgraph)	*Wapiti* v2.3.0 (Nicolas Surribas)	*XSSer* v1.6-1 (OWASP)
*Netsparker Community Edition* v 3.1.6.0 (Netsparker Ltd)	*N-Stalker 2012* *Free Edition* v 10.13.11.31 (N-Stalker)	*Syhunt Mini* v4.4.3.0 (Syhunt) *(p.k.a Sandcat Mini)*

New aspects of the following open source scanners were tested in the benchmark:

Andiparos v1.0.6
(Compass Security)

Paros Proxy v3.2.13

(Milescan)

The previous results of the following free scanners were included in the benchmark, since they were not updated since the previous benchmark (website):

*Acunetix Free Edition* v8.0-20120509 (Acunetix)	*N-Stalker 2009 Free Edition* v7.0.0.223 (N-Stalker)	*WebSecurify* v0.9 latest free edition (GNUCITIZEN)
*Sandcat Free Edition* v4.0.0.1 (Syhunt)	*WebCruiser* v2.4.2 FE	*JSKY Free Edition* v1.0.0
*Scrawler* v1.0 (HP)	*Safe3WVS* v10.1 FE (Safe3 Network Center)

The results of the following open source scanners were included but not re-verified:

sqlmap v1.0-Jul-5-2012 (Github) – already achieved mastery in its supported feature

DSSS (Damn Simple SQLi Scanner) v0.1h – 0.2h exists, will be tested in the future

aidSQL 02062011 – newer version released in 2013-05-27, will be tested in the future

The results were compared to those of unmaintained scanners tested in the past:

*ProxyStrike* v2.2	*Grendel Scan* v1.0	*PowerFuzzer* v1.0
*Oedipus* v1.8.1 (v1.8.3 is around somewhere)	*Xcobra* v0.2	*XSSploit* v0.5
*UWSS (Uber Web Security Scanner)* v0.0.2	*Grabber* v0.1	*WebScarab* v20100820
*Mini MySqlat0r* v0.5	*WSTool* v0.14001	*crawlfish* v0.92
*Gamja* v1.6	*iScan* v0.1	*LoverBoy* v1.0
*openAcunetix* v0.1	*ScreamingCSS* v1.02	*Secubat* v0.5
*SQID (SQL Injection Digger)* v0.3	*SQLiX* v1.0	*VulnDetector* v0.0.2
*Web Injection Scanner* (WIS) v0.4	*XSSS* v0.40	*Priamos* v1.0

For a full list of commercial & open source tools that were not tested in this benchmark, refer to the appendix.

3. Benchmark Overview & Assessment Criteria

The benchmark focused on testing commercial & open source tools that are able to detect (and not necessarily exploit) security vulnerabilities on a wide range of URLs, and thus, each tool tested was required to support the following features:

(*)The ability to detect Reflected XSS and/or SQL Injection and/or Path Traversal/Local File Inclusion/Remote File Inclusion vulnerabilities.

(*)The ability to scan multiple URLs at once (using either a crawler/spider feature, URL/Log file parsing feature or a built-in proxy).

(*)The ability to control and limit the scan to internal or external host (domain/IP).

The testing procedure of all the tools included the following phases:

Feature Documentation

The features of each scanner were documented and compared, according to documentation, configuration, plugins and information received from the vendor. The features were then divided into groups, which were used to compose various hierarchal charts.

Accuracy Assessment

The fact that a scanner supports a certain category of tests, does not say anything on HOW WELL it is able to detect the supported issues. The purpose of the accuracy assessment is to see how effective each scanner is in detecting a variety of vulnerabilities, and to see whether or not the detection logic "settles" for simple scenarios, or covers a collection of common and advanced scenarios.

The scanners were all tested against the latest version of WAVSEP (v1.5), a benchmarking platform designed to assess the detection accuracy of web application scanners, which was released alongside the publication of this benchmark.

The purpose of WAVSEP’s test cases is to provide a scale for understanding which detection barriers each scanning tool can bypass, and which common vulnerability variations can be detected by each tool.

WAVSEP 1.5 added includes test cases from ZAP-WAVE, code contributions from various volunteers and a collection of 250+ NEW test cases for two new exposures: unvalidated redirect and obsolete/hidden files.

The various scanners were tested against the following test cases (GET/POST):

60 test cases that were vulnerable to Phishing via Unvalidated Redirect.
184 test cases that included Hidden, Obsolete and Backup files.
816 test cases that were vulnerable to Path Traversal attacks.
108 test cases that were vulnerable to (XSS via) Remote File Inclusion attacks.
66 test cases that were vulnerable to Reflected Cross Site Scripting attacks.
80 test cases that contained Error Disclosing SQL Injection exposures.
46 test cases that contained Blind SQL Injection exposures.
10 test cases that were vulnerable to Time Based SQL Injection attacks.

The various scanners were also tested against a variety of false positive scenarios:

9 different categories of false positive Unvalidated Redirect vulnerabilities.
3 different categories of false positive Obsolete/Hidden/Backup files.
8 different categories of false positive Path Traversal / LFI vulnerabilities.
6 different categories of false positive Remote File Inclusion vulnerabilities.
7 different categories of false positive Reflected XSS vulnerabilities.
10 different categories of false positive SQL Injection vulnerabilities.

Overall, a collection of 1413 vulnerable test cases for 6 different attack vectors, each test case simulating a different and unique scenario that may exist in an application.

Although the testing platform included a variety of experimental test cases for similar and different vulnerabilities (DOM-XSS, information disclosure issues, etc), these were not included in the scope of the benchmark, and their results did not affect the final score.

Attack Surface Coverage Assessment

In order to assess the scanners attack surface coverage, the assessment included tests that measure the efficiency of the scanner's automated crawling mechanism (input vector extraction) , and feature comparisons meant to assess its support for various technologies and its ability to handle different scan barriers.

This section of the benchmark also included the WIVET test (Web Input Vector Extractor Teaser v3-rev148), in which scanners were executed against a dedicated application that can assess their crawling mechanism efficiency in the aspect of input vector extraction. The specific details of this assessment are provided in the relevant section.

Result Verification

In order to ensure the result consistency, the directory of each exposure sub category was individually scanned multiple times using various configurations (for the vast majority of tested products), usually using a single thread and using a scan policy that only included the relevant plugins.

In order to ensure that the detection features of each scanner were truly effective, most of the scanners were tested against an additional benchmarking application that was prone to the same vulnerable test cases as the WAVSEP platform, but had a different design, slightly different behavior and different entry point format, in order to verify that no signatures were used, and that any improvement was due to the enhancement of the scanner's attack tree.

Furthermore, in order to verify that all WIVET results were reliable, the vast majority of tools were also tested against an unpublished online version of WIVET that included additional enhancements that prevent pre-adaptation to the platform URLs (http://wivet.webscantest.com/).

Finally, since the test was performed with the aid of several volunteers, some results were verified by more than one person and on multiple environments.

Making the Results Useful to Vendors

In order to help vendors understand which scenarios were "missed" by their products, the list of identified test cases was documented in detail, for each class of vulnerabilities, and the list of test cases that were missed can be deducted from that list. Since WAVSEP contains detailed documentation on each and every test case, this information can help vendors identify their weaknesses and cover prominent scenarios.

Refer to the scan description section (click the version link) of each scanner in http://www.sectoolmarket.com to locate exactly which test cases were identified by each scanner.

Public tests vs. Obscure tests

In order to make the test as fair as possible, while still enabling the various vendors to show improvement, the benchmark was divided into tests that were publically announced, and tests that were obscure to all vendors:

(*)Publically announced tests: the various feature comparisons, the WIVET assessment and the detection accuracy assessment of the SQL Injection, Reflected Cross Site Scripting, Path Traversal/LFI and (XSS via) Remote File Inclusion were well known to all vendors, and already published as a part of WAVSEP v1.2 (which was available online for the last year and a half).

(*)Tests that were obscure to all vendors until the moment of the publication: the detection accuracy assessment of the Unvalidated Redirect and Obsolete/Hidden File Detection implemented as 256+ NEW test cases in WAVSEP 1.5 (a new version that was only published alongside this benchmark).

The results of the main test categories are presented within three graphs (commercial/SAAS graph, free & open source graph, unified graph), and the detailed information of each test is presented in a dedicated section in benchmark presentation platform at http://www.sectoolmarket.com.

4. A Glimpse to the Results of the Benchmark

This presentation of results in this benchmark, alongside the dedicated result presentation website (http://www.sectoolmarket.com/) and a series of supporting articles and methodologies, are all designed to help the reader to make a decision - to choose the proper product/s or tool/s for the task at hand, within the borders of the time or budget.

A summary of the most significant results can be seen in the following links, and filtered according to the product license (commercial/opensource):

Price & Feature Comparison of Commercial Scanners

http://sectoolmarket.com/price-and-feature-comparison-of-web-application-scanners-commercial-list.html

Price & Feature Comparison of a Unified List of Commercial, Free and Open Source Products

http://sectoolmarket.com/price-and-feature-comparison-of-web-application-scanners-unified-list.html

Some of the sections might not be clear to some of the readers at this phase, especially since many of them contain new conclusions and new results, which is why I advise both veterans and newcomers to read the rest of the article, prior to analyzing this summary.

5. SURPRISE SURPRISE

Although on a general basis – the vast majority of product improve their results from benchmark to benchmark, and this case is not different, this benchmark also has an above -average amount of conflicting results.

More than a few tools that got high results in the previous benchmarks categories, got lesser results in this one – in the same categories, although nothing in the test environment has changed.

Furthermore, some of the new tests were met with… surprising difficulty by the vast majority of the tools in the industry, leading me to believe that many products in the industry had grown to a size which may be challenging to maintain in the future years.

The overall problem is related to product testing and maintenance – the fact that software bugs may cause a variety of crucial features not to function for long periods of time, without anyone being aware of them.

The cost of the mitigating processes to the vendor (or lack of to the consumer!) may be very high, and to the fact that it's very difficult for the consumer to indentify such issues, especially on a periodic basis, can have a major effect.

It's hard to avoid it… all you need to do is take a look at a couple of the "new" charts, and even in some of the "traditional" WAVSEP charts to notice this issue, which I will discuss in details in some of the sections.

This phenomenon is something which I will probably analyze in future publication, and should be a reason to be concerned, especially since unless certain precautions will be taken, will probably become more severe with time.

6. How to Read and Use the Benchmark Results

The practical reader, the one who wants to make use of the information provided in this research to his advantage, can use the following guidelines for interpreting the results, and the following steps to get to practical decisions:

(*) Although it's tempting to look only at the tools at the top, it's important to remember that insignificant differences in results are just that – insignificant – and should be treated accordingly. The benchmark can never cover every single scenario, and a few percents don't always make a product better in a category (although plenty of percents probably do). I would therefore recommend the reader that evaluates a tool to figure out whether or not the tool has a good score in an assessment and in general, instead of falling to the 100% percent trap. That being said, a perfect score certainly isn't bad, so don't take it the other way around either.

(*) When trying to figure which tool you should use, try the following simple methodology:

1. Input Vector and Scan Barrier Support

Figure out if the input delivery method (Test I) used by the application or applications you are using is supported by the scanners you are evaluating. Do the same for the various security mechanisms, technologies and scan barriers that are used in the application (Text X). The scanner won't work at all, or will provide little value if it won't support those.

[Note: pentesters should probably go for a tool that supports enough of those, as the technological barriers they may encounter vary, while other organization may use tools that support only what they need]

2. Crawling & Input Vector Extraction

If you use scanners mainly in a point-and-shoot scenario, and prefer as much automation as possible, a high WIVET score will be the second most important feature you should follow.

[Note: for the most part, most pentesters can deal with a reasonable score as well, although a high one will certainly help, while organizations and QA/DEV departments really need a tool with a high score in this category – especially in 2014]

3. Vulnerability Detection Features and Accuracy

It's hard to say what's more important – so try and keep those in balance. The more accurate and the more feature rich – the better. Bear in mind that an accuracy difference of 1%, 5% or even 10% is NOT necessarily significant, although larger differences might be.

4. Price

No point in buying a product that can't run, isn't automated enough for you (in case you need it), isn't accurate at all (will only result in extra work for you), or doesn't have enough features to justify the price, but once all that out of the way, price is your next criterion. Bear in mind that you can usually negotiate, and that from time to time, prices changes.

5. All the rest

Some features may be special, such as platform specific capabilities, result documentation features, complementary features that can make your life easier, configure your WAF, generate reports for you manager or get you a free trip to mars.

Some of these features may even tip the scale on the expanse of other features, but in the long run, try to stick to that order.

Also note that these are general guidelines, and that if this choice is significant, you might want to consult with an expert to help you evaluate which tools match your needs.

7. Test I - Versatility - Input Vector Support

As I mentioned in previous posts from 2012, after investigating the field of DAST for the past five years, I consider the scanner's support for the tested application input delivery method to be the single MOST significant aspect in the selection process of any scanner.

Reasoning: the input delivery method (a.k.a the input vector) is the method used by the HTML/Flash/Applet/Silverlight application to deliver user-originating input from the client to the server.

These "formats" include common formats such as:

(*)Query String Parameters (URL?param1=value1&param2=value2)

(*)HTTP Body Parameters (param1=value1&param2=value2)

And "modern" formats such as:

(*) JSON Arrays ({"param1":"value1","param2":"value2"})

(*) XML Elements and Attributes (element value)

These methods may also include binary delivery methods for technology specific objects such as AMF, Java serialized objects and WCF, as well as many other input delivery methods.

Since the majority of attacks rely on malicious input being delivered through input parameters to the application, a scanner that is not able to deliver those values to most of the application server entry points WILL NOT be a good choice.

An automated tool can't detect vulnerabilities in a given parameter, if it can't scan the protocol or mimic the application's method of delivering the input.

In fact, lack of support for the dominant input vector used by the application can make the scanner NEARLY USELESS for that specific application (without demoting how useful it may be for other types of applications).

While organizations that stick with specific development technologies only need to verify that the scanner they use supports the input delivery method used by their applications, since in 2013/2014 there is a vast collection of different input delivery methods, versatility becomes a major issue for pentesters, and to some extent for organization that rapidly develop applications in different technologies.

Although the position in this section charts don't necessarily represent the most important score, it is the most important perquisite for the scanner to comply with when scanning a specific technology.

Therefore, the first assessment criterion of this benchmark is the number of input vectors each tool can scan (not just parse), which is a major component in the scanner versatility score.

Important Note

Although, it may seem logical that a scanner that supports an input delivery method will do so consistently, some scanners support for an input vector may be limited to SOME of the vulnerability detection plugins, while the rest may be supported only for basic input delivery methods.

I became aware of this condition after a thorough research, and unfortunately, at the moment there is no sure way to verify which detection capabilities of scanners are actually supported for each input vector, at least not on a large scale, and for the vast majority of scanners.

Since WAVSEP test cases are implemented with either query string or HTTP body parameters, only the support for these vectors was actually verified, and the rest of the information in this section derives from a thorough research that covered the vendor proclaimed results, source code (when possible) and feature documentation.

Future versions of WAVSEP may include test cases to verify the support of scanners for different input vectors.

Before viewing the charts that represent the versatility of different vulnerability scanners, it may be a good time to mention interesting features of two products which are related to this category.

This proclamation does not mean that the author takes a stand as to which product is "the best" (a conclusion that anyone who read my previous benchmarks knows very well not to expect), just that the approach these products take to classify attacks, manage scan scope and present the information to the user can be very beneficial in many situations.

The products I refer to are NTOSpider and Acunetix, and to some extent IronWASP, ZAP and Burp (and products with similar features, in case I forgot any), each taking an interesting approach to input vector support and scan scope management:

(*) NTOSpider enables the user to manage which input vectors should be tested for each attack, therefore presenting which vectors are supported for each attack, information which is very hard to obtain from documentation:

(*) Acunetix presents which attacks are performed per directory, schema, file, etc:

Other tools contained interesting features (with no attack-per-vector info) that provided control over which input vectors will be scanned:

IronWASP input delivery method scope selection in scan wizard:

OWASP ZAP input delivery method scope selection in the configuration window:

Similar features were verified in Burp Suite Pro, and may exist in other products as well.

The more vectors of input delivery that the scanner supports, the more versatile it is in scanning different technologies and applications (assuming it can handle the relevant scan barriers, supports necessary features such as authentication, or alternatively, contains features that can be used to work around the specific limitations).

The detailed comparison of the scanners support for various input delivery methods is documented in detail in the following section of sectoolmarket: http://www.sectoolmarket.com/input-vector-support-unified-list.html

The following charts shows how versatile each scanner is in scanning different input delivery vectors (and although not entirely comprehensive - different technologies):

Result Update (29/03/2014): Appscan, ZAP and arachni reported support for additional input vectors AFTER the original benchmark publication (in the same tested versions). The current charts include these updates, alongside others.

The Number of Input Vectors Supported – Commercial Tools & SAAS

The Number of Input Vectors Supported – Free & Open Source Tools

The Number of Input Vectors Supported – Unified List

Versatility of Open Source Scanners vs. Commercial Scanners in 2014

The vast majority of open source tools tested in 2012 (with the exception of IronWASP) did not support vectors besides the basic GET/POST/Header/Cookie vectors, making the task of using them against "modern" applications that rely on JSON/XML/etc impractical.

However, as the graph proves, certain open source vendors invested efforts in supporting additional input delivery methods in their vulnerability scanning features, and thus, these scanners can be used effectively against applications with "modern" input vectors and technologies.

Although this scenario is rare, and by no means representative, the careful inspector will even identify input delivery methods that are only supported by certain open source projects (for example, ZAPs support for GWT), although the same goes the other way around for many vectors supported by commercial vendors .

8. Test II - WIVET - Crawling Coverage

The second assessment criterion was focused on assessing crawling coverage features, which included the various discovery methods used to increase the attack surface of the tested application: to locate additional resources and input delivery methods to attack.

Although scanners can increase the attack surface in a number of ways, from detecting hidden files to exposing device-specific interfaces (mobile, tablet, etc), this assessment was focused at assessing the automated crawling capabilities and input vector extraction coverage (as opposed to input vector scanning support measured in the previous section) of the various scanners, and is primarily represented using the scanner's WIVET score.

This aspect of a scanner is extremely important in point-and-shoot scans, scans in which the user does not "train" the scanner to recognize the application structure, URLs and requests, either due to time/methodology restrictions, or when the user is not a security expert that knows how to properly use manual crawling with the scanner.

Although users that can afford "training" the scanner to recognize the URL and input sources in the application (by using it as a proxy, for example) don't necessarily require enhanced crawling coverage, organizations and individuals that prefer or require using the web application scanner in an automated manner (point-and-shoot) should consider the crawling coverage / input vector extraction to be of highest importance, second only to the support of the scanner for testing the necessary input delivery vectors.

As mentioned earlier, in order to evaluate these aspects in scanners, I used a project called WIVET (Web Input Vector Extractor Teaser); The WIVET project is a benchmarking project that was written by Bedirhan Urgun, and released under the GPL2 license.

The project is implemented as a web application which aims to "statistically analyze web link extractors", and measures the amount of input vectors extracted by each scanner scanning the WIVET website.

Plainly speaking, the project simply measures how well a scanner is able to crawl the application, and how well can it locate input vectors, by presenting a collection of challenges that contain links, parameters and input delivery methods that the crawling process should locate and extract.

In order for WIVET to work, the scanner must crawl the application while consistently using the same session identifier in its crawling requests, and while avoiding the 100.php logout page (which initializes the session, and thus the results).

The results can then be viewed by accessing the application index page, while using the session identifier used during the scan.

During the tests I used a variety of workarounds designed to "assist" scanners with missing proxy/cookie customization features to scan WIVET, usually by scanning a proxy that forwarded the communication to WIVET while adding consistent session identifiers and restricting the access to the logout page.

The scan configuration used with each scanner against WIVET was documented in detail in the scanners "scan log", and the comparison of the scanners' WIVET score is presented in the following section of sectoolmarket:

http://sectoolmarket.com/wivet-score-unified-list.html

Result Update (29/03/2014): the impressive 96% result of Webinspect can be achieved by selecting the "depth first" mode in the scan wizard. The default option in the wizard is slightly less efficient, but still yields a great result that competes with the best result of any other scanner (94%).

The WIVET Score of Web Application Scanners – Commercial Tools & SAAS

Due to technical difficulties and time constraints the WIVET results of ScanToSecure are not yet included, it can be assumed to have the same score of Netsparker, since this is the engine at its core.

The WIVET Score of Web Application Scanners – Free and Open Source Tools

The WIVET Score of Web Application Scanners – Unified List

Although the scan success rate was much higher than in previous years, still, some of the scanners were not able to scan this platform despite all my efforts. The score of these projects will be updated as soon as they enhance their crawling mechanisms enough to scan WIVET.

It's crucial to remind the reader that scanners with burp-log parsing features (such sqlmap and IronWASP) can effectively be assigned with the WIVET score of burp, and also that scanners with internal proxy features (such as ZAP, Burp, etc) can be used with the crawling mechanisms of other scanners (such as Netsparker CE).

Thus, any scanner that supports any of those features can be artificially "enhanced" and assigned the WIVET score of any other scanner in the possession of the tester.

9. Introduction to the Accuracy Assessments

The following sections presents the results of the detection accuracy assessments performed for *Unvalidated Redirect*, *Old, Backup and Unreferenced Files*, *Path Traversal / LFI*, *(XSS via) Remote File Inclusion*, *Reflected XSS* and *SQL Injection*, six of the most commonly supported features in web application scanners.

Since two of these assessments are *NEW* to this yearly benchmark (the backup files and unvalidated redirect accuracy assessments - which were not disclosed to the various vendors prior to the publication of this benchmark), two more were new in the 2012 benchmark (the path traversal/LFI and the remote file inclusion accuracy assessments), and two existed in the benchmark from day one (SQL injection and reflected XSS) – there's an interesting combination of results that can help assess the overall scanner's performance.

Sure - the detection accuracy of a specific exposure might not reflect the overall condition of the scanner on its own, but the careful reader can go back and analyze previous benchmarks to identify patterns, and as always, these results serve as a crucial indicator for how good a scanner is at detecting specific vulnerability instances.

The various assessments were performed against the various test cases of WAVSEP v1.5, which emulate different common test case scenarios for generic technologies.

Reasoning: a scanner that is not accurate enough will not be able to identify many exposures, and might classify non-vulnerable entry points as vulnerable. These tests aim to assess how good is each tool at detecting the vulnerabilities it claims to support, in a supported input vector, which is located in a known entry point, without any restrictions that can prevent the tool from operating properly.

These accuracy assessments were also performed under optimal conditions (or at least as optimal as we could create), since the purpose was to see how well the detection logic functions, with no interference from various barriers that can affect it in applications.

Such optimal conditions included scanning relatively small groups of URLs, using a limited amount of threads, defining optimal configuration entries (in some cases), and so on.

Therefore, to reproduce these results, it is necessary to follow the exact instructions listed in the various scan logs included in sectoolmarket.

10. Test III - Unvalidated Redirect Detection

The third assessment criterion was the detection accuracy of Unvalidated Redirect, a common exposure which is also a commonly implemented feature in web application scanners, and most importantly, a NEW TEST in WAVSEP which the vendors were not aware of prior to the publication of this article.

It's also included in OWASP TOP 10 2010 and in OWASP TOP 10 2013, and represents a continued effort to make WAVSEP as compliant as possible with the various OWASP TOP 10 lists.

This score chart is different from the rest because unlike the rest of the detection accuracy charts, it calculates the score only based on QueryString/GET test cases, and does not take into account the HTTP POST test cases.

The reason to include only GET test cases in the score calculation is related to the properties of an unvalidated redirect attack:

It's essentially a phishing enhancing attack which relies on web site redirection features that redirect the browser to user-controlled addresses sent in the input. These attacks eventually redirect the user to an attacker controlled website, while misleading even cautious users that verify the domain address prior to accessing a link.

For example:

Original URL -

http://domain:port/app/login.jsp?nextPage=internal.jsp

Abused URL -

http://domain:port/app/login.jsp?nextPage=http://hacker-domain:port/fake-login.jsp

A case could be made to state that since submitting malicious redirect values in POST parameters requires the user to first access an HTML form in an attacker controlled website, than there's no point in performing this attack at all, since the user already "trusted" the attackers website.

In fact, this statement is well ingrained in the perception of many tool authors, which usually don't submit any redirect payloads in POST parameters.

Several arguments can be made against that perception:

(*) Detecting persistent unvalidated redirect attacks (like persistent XSS attacks) in which the payload is "injected" into the database and affects other users, may very well justify sending redirect payloads in POST parameters.

(*) Detecting session-hosted unvalidated redirect attacks and pages in the actual website that embed externally supplied URLs in a form that will later be submitted using POST may justify performing POST tests as well.

Regardless of whether the argument is true or not, due to the lack of support for POST unvalidated redirect tests in most of the tested products, I decided not to include the POST test cases in this benchmark, despite the fact that they are already included in WAVSEP, and despite the various scenarios in which testing POST parameters with unvalidated redirect payloads may lead to valid vulnerabilities (persistent redirect, session redirect, reprinted redirect form, etc).

The POST test cases may however be included in the next benchmark, in one way or the other, and the full results are already included in the relevant scan logs of sectoolmarket.

In order to assess the detection accuracy of different unvalidated redirect instances, I used a total of 30-60 test cases (for 302 redirection, and even for JS redirection). I also used a bunch of false positive test cases, to see how permissive the detection process is.

The comparison of the scanners' unvalidated redirect detection accuracy is documented in detail in the following section of sectoolmarket:

http://sectoolmarket.com/unvalidated-redirect-detection-accuracy-unified-list.html

Result Chart Glossary

The Unvalidated Redirect Detection Accuracy of Commercial/SAAS Scanners

The Unvalidated Redirect Detection Accuracy of Opensource/Free Scanners

The Unvalidated Redirect Detection Accuracy of Scanners – Unified List

11. Test IV - Backup/Hidden File Detection

The fourth assessment criterion was the detection accuracy of Old, Backup and Unreferenced Files, a very common exposure, that may lead to source code and configuration theft, which is also a commonly implemented feature in web application scanners, and once again, a NEW TEST in WAVSEP which the vendors were not aware of prior to the publication of this article.

This is also the test in which the results are MOST SURPRISING.

To make it clear, this test assessed the capabilities of scanners to locate backup files with non-executable extensions, compressed versions of files and directories that developers may have forgotten, sequential files or copies of files and directories that are remnants of various development tests, and additional hazards that may lead to source code configuration disclosure.

For those of you that doubt the importance of this vector, it's an exposure that as a pen-tester I personally abused to download the entire source code of banks, e-commerce web sites, and credit card companies, obtained connection strings and hard-coded credentials from obsolete source code fragments and configuration files, as well as located numerous hidden entry points that were vulnerable to exposures that the rest of the application was not prone to.

What I'm trying to say is that while some instances of this exposure may yield insignificant results, some severe instance could mean the "game is over" for the application, and expose every server side vulnerability or hidden credential to the attacker.

Back in the old days, I used a collection of tools and lists to identify such issues;

I made heavy use of Sensepost's Wikto with customized lists of files and extensions; I used the backup/hidden file detection features of the earliest published version of W3AF to download the source code of several banks, and from time to time, even suffered through the false positives of the mythical Paros Proxy obsolete file detection features.

However, since then, many open source and commercial tools mastered those attacks, and tried to make the detection task easier.

But as the results obviously show, something bad happened along the way, which is not necessarily related to this specific vulnerability, as much as it is related to a major problem that affects the entire automated vulnerability detection industry.

Insufficient Implementation of

TDD

If there's any obvious conclusion that the reader can conclude from this benchmark, this is probably it:

The is a serious problem in (and therefore insufficient use of) implementations of TDD in the development of many web application vulnerability scanners:

Test Driven Development is a development process in which the software developers invest efforts in writing unit tests for code modules, often even prior to writing the modules themselves, and in which the build process of the product uses these tests to verify the code modules function properly, and that there aren't any unexpected behaviors.

TDD is usually very costly to implement, but in my opinion, pays in the long run – and in many aspects.

Now don't get me wrong, I'm certain that almost all vendors use TDD to some extent, however, after experiencing what I have in this benchmark, I'm also certain its probably insufficient (at least for some products).

And honestly, I find it very hard to blame the vendors.

Allow me to elaborate:

There is a lot of competition in this product category, and new features are often rushed to market as soon as possible. It also takes a major effort to write unit-tests that include network communication and scanning, and to review the results, even for a single vulnerability detection plugin.

Although it makes sense that the same outcome could be accomplished using traditional QA processes, which may very well be true for small-mid scale projects, one need only to look at the insane number of plugins and features in products like Qualys, Appscan, Webinspect and W3AF, to understand the futility of leaving all the testing to humans.

Imagine how much effort it would take to manually test that 200 generic detection plugins function properly… Implementing unit-tests for all those modules isn't a small investment as well.

And what about 50000 signature-based product specific vulnerabilities? How long will it take to manually test that (or develop unit-tests to verify) those features work?

During the testing process, I have seen plugins in several tools which were actually named after the various extensions of obsolete files I was trying to detect in WAVSEP, and still, scanning the platform with some or all of them did not yield results for many tools.

My assumption is that the same problem is also responsible for the results of tools that got 100% in previous benchmarks, and got different results in this bulk of tests, even though the testing framework (WAVSEP/WIVET) did not include any changes in the test cases scanned.

My Assumption:

The various plugins and features are based on a scan engine, and changes made to the engine (or plugins) may cause some of them to malfunction.

Since there wasn't a unit test (or other pre/post build test method) for those plugins, newer versions were released while those plugins were not functioning, maybe even for years, and without anybody knowing about it.

Not so scary when considering , let's say - small scale projects,

But VERY scary when you consider a product update that causes many plugins to malfunction in a scanner with 50000 plugins, which is released after the organization tested it successfully and used it for years, and while the official recommendation of the vendor was to install the update.

The vendor may never know, and the customer/user may only discover the issue after vulnerabilities that the product was suppose to identify will be exploited.

Customers that are currently not aware of a problem, vendors that may never be, and entities that can abuse that problem are a terrible combination… No malice intended.

In order to assess the detection accuracy of different old/backup/hidden file instances, I used a total of 184 test cases (many of them simulating files created in windows XP / windows 7 developer stations, as well as in common Linux flavors such as Ubuntu, Debian and Fedora). I also used three main groups of false positive behaviors - each representing real life scenarios that vulnerability scanners can experience.

The comparison of the scanners' old, backup and unreferenced files detection accuracy is documented in detail in the following section of sectoolmarket:

http://sectoolmarket.com/old-backup-and-unreferenced-files-detection-accuracy-unified-list.html

Note: as mentioned earlier, I saw various features in several of the tested tools that were supposed to identify additional results, but for some reason did not function. My current assumption (and that's all that is – my assumption) is that the reason is related to bugs in the engine or the module of those tools.

As luck (or lack of) would have it, the same problem seemed to persist for many vendors in that specific category of tests.

Disclaimer:

The results of OWASP ZAP in the obsolete file detection test were obtained using an external ZAP extension called Good-Old-Files (GoF - included in ZAP built-in marketplace).

The extension was written by a colleague of mine by the name of Michal Goldstein, and was originally inspired (to the previous extension authors) by various modules in W3AF.

She was not aware of the benchmark, or to the fact I was assessing her project, and when I built the testing platform, I used input from a collection of tools and sources to build the benchmark test-bed, including GoF/W3AF.

Those of you that believe that might have affected the testing process may feel free to ignore the results of that tool.

Result Chart Glossary

The Old/Backup/Hidden File Detection Accuracy of Commercial/SAAS Scanners

The Old/Backup/Hidden File Detection Accuracy of Opensource/Free Scanners

The Old/Backup/Hidden File Detection Accuracy of Scanners – Unified List

12. Test V - Path Traversal / LFI Detection

The fifth assessment criterion is identical to the previous benchmark - the detection accuracy of Path Traversal (a.k.a Directory Traversal), an assessment feature that was implemented in WAVSEP v1.2, and tested in the 2012 benchmark for the first time.

It's also the third most commonly implemented attack vector in web application scanners, and a significant attack vector in its own right.

Many scanners had a difficult time locating a variety of traversal test cases in 2012, but this time, the results show a significant improvement in the results of many of the tools, proving that many vendors invested major efforts in improving their products.

Path Traversal vs. Local File Inclusion – Reminder

As I explained in the past, the reason Path Traversal was tagged along with Local File Inclusion (LFI) is simple - many scanners don't make the differentiation between inclusion and traversal, and furthermore, a few online vulnerability documentation sources do. In addition, the results obtained from the tests performed on the vast majority of tools lead to the same conclusion - many plugins listed under the name LFI detected the path traversal test cases.

While implementing the path traversal test cases in 2012 and consuming nearly every relevant piece of documentation I could find on the subject, I decided to take the current path, in spite of some acute differences some of the documentation sources suggested (although I did implemented an infrastructure in WAVSEP for "true" inclusion exposures).

The point is not to get into a discussion of whether or not path traversal, directory traversal and local file inclusion should be classified as the same vulnerability, but simply to explain why in spite of the differences some organizations / classification methods have for these exposures, they were listed under the same name.

The evaluation was performed on a WAVSEP v1.2 instance that was hosted on a windows XP VM, and although there are specific test cases meant to emulate servers that are running with a low privileged OS user accounts (using the servlet context file access method), many of the test cases emulate web servers that are running with administrative user accounts.

[Note - in addition to the wavsep installation, to produce identical results to those of this benchmark, a file by the name of content.ini must be placed in the root installation directory of the tomcat server- which is different than the root directory of the web server.

It’s also crucial to install WAVSEP on windows, and run the tomcat server with administrative privileges, as some of the test cases rely on windows-specific paths or require access to directories outside of the web server scope]

In order to assess the detection accuracy of different path traversal instances, I used a total of 816 path traversal test cases, and a bunch of false positive test cases as well.

The comparison of the scanners' path traversal detection accuracy is documented in detail in the following section of sectoolmarket:

http://sectoolmarket.com/path-traversal-local-file-inclusion-detection-accuracy-unified-list.html

Note:

During the testing of the development version of W3AF (the latest stable I could get was 1.2 which was tested in 2012, and the current development version was 1.6+) I experienced several bugs, specifically bugs that prevented the scanner from scanning HTML forms submitted using HTTP POST (or in short, POST parameters).

One of these bugs was related to the LFI/Path Traversal detection plugin, which caused the scan to crash whenever it was used, after detecting only a few vulnerable test cases.

I tried various methods to overcome that bug artificially, but failed to do so, so I was not able to obtain the actual results of the latest version of W3AF, and thus, decided to use the results from the previous benchmark to represent it's score.

The bugs were reported to the project leader, and hopefully, will be fixed in the future.

I had similar issues trying to use the various LFI/RFI plugins of Qualys, and unfortunately, wasn't able to overcome them and get an actual score by the publication of this benchmark (which is why Qualys is absent from the LFI/RFI charts). I'm currently not sure if the reason is a bug in product or in the configuration used during my testing process.

Result Chart Glossary

Note that the GREEN bar represents the vulnerable test case detection accuracy, while the RED bar represents false positive categories detected by the tool (which may result in more instances then what the bar actually presents, when compared to the detection accuracy bar).

Result Update (29/03/2014): The results of arachni were improved from 30.88% to 100% (!!!) according to vendor recommendations provided AFTER the original benchmark publication, by using the source code disclosure plugin, in addition to the local file inclusion and path traversal plugins, after verifying that the plugin behavior is relevant to the exposure (the name may deceive), and while using the same version.

The result of Webinspect were likewise improved from 72.06% to 91.18% by using a custom configuration provided by the vendor AFTER the original benchmark publication, using the same tested version, which included the following plugins:
i. 10287 – Local File Include
ii. 10271 – Local File Inclusion/Reading Vulnerability
iii. 10272 – Possible Local File Inclusion/Reading Vulnerability
iv. 11327 – LFI Tomcat
v. 11332 – LFI IIS

The Path Traversal / LFI Detection Accuracy of Commercial /SAAS Scanners

The Path Traversal / LFI Detection Accuracy of Opensource /Free Scanners

The Path Traversal / LFI Detection Accuracy of Scanners – Unified List

13. Test VI - (XSS via) RFI Detection

The sixth assessment criterion was again, identical to the 2012 benchmark - the detection accuracy of Remote File Inclusion (or more accurately, vectors of RFI that can result in XSS or Phishing - and currently, not necessarily in server code execution), an assessment suite implemented in WAVSEP v1.2, which was tested in the 2012 benchmark for the first time, with interesting results indeed.

A reminder - although in the 2012 benchmark several products identified the vulnerable test cases properly, some products with RFI detection features ignored it completely.

Obviously, 1.5 years after the 2012 publication, that's no longer the case for the vast majority of vendors; the detection accuracy and support for (XSS via) RFI was dramatically improved in many tools, and we – the users – can reap the rewards in penetration tests.

In order to assess the detection accuracy of different remote file inclusion exposures, I used a total of 108 (xss via) remote file inclusion test cases, and as always, a bunch of false positive cases that represent common scenarios.

The comparison of the scanners' (xss via) remote file inclusion detection accuracy is documented in detail in the following section of sectoolmarket:

http://sectoolmarket.com/remote-file-inclusion-detection-accuracy-unified-list.html

Result Chart Glossary

The (XSS via) RFI Detection Accuracy of Opensource/Free Scanners

The (XSS via) RFI Detection Accuracy of Scanners -Unified List

14. Test VII - Reflected XSS Detection

The seventh assessment criterion has been a part of the yearly WAVSEP assessment for four years now (!), and the results of the various vendors that maintain their tools emphasize that well.

As the title suggests, this section deals with the detection accuracy of Reflected Cross Site Scripting, a very common exposure which is the 2nd most commonly implemented feature in web application vulnerability scanners.

The assessment was performed using 66 different Reflected XSS test cases and a bunch of false positive test cases, and while ignoring the results of the various experimental RXSS test cases included in WAVSEP 1.5 (although the "experimental" results are included in most of the individual tools scan logs in sectoolmarket).

There's not much to say in this section that wasn't already said in previous articles and benchmarks, except to present the current (and generally IMPRESSIVE) results of the various maintained products / projects.

The comparison of the scanners' reflected cross site scripting detection accuracy is documented in detail in the following section of sectoolmarket:

http://sectoolmarket.com/reflected-cross-site-scripting-detection-accuracy-unified-list.html

Note

Bugs in certain products seemed to affect their detection accuracy for Reflected XSS, since in the past, these products obtained higher results (notably arachni/W3AF).

Result Chart Glossary

Note:

The Reflected XSS Detection Accuracy of Commercial/SAAS Scanners

The Reflected XSS Detection Accuracy of Opensource/Free Scanners

The Reflected XSS Detection Accuracy of Scanners - Unified List

15. Test VIII – SQL Injection Detection Accuracy

15. Test VIII - SQL Injection Detection

The eight assessment criterion was the good old SQL Injection detection accuracy, another assessment suite that's been with us for the last four years (!) of WAVSEP benchmarks.

As one of the most famous exposures (and powerful attacks) and the most commonly implemented attack vector in web application scanners, it's also one of the aspects in which maintained projects showed the greatest improvement over the years.

Although the release of WAVSEP 1.5 includes optional vulnerable SQL injection test cases that were adjusted to support other databases (such as MSSQL, ORACLE, etc – contributed due to the endless generosity of the ZAP team members), due to time constraints, the evaluation was only performed on an application that used MySQL 5.5.x as its data repository, and thus, can only reflect the detection accuracy of the tool when scanning an application that uses similar data repositories.

My assumption however, is that the detection results of error-based test cases and behavior based test cases will be nearly identical if the underlying database will be different, but that there will be a difference for some of the tested tools in test cases that require time-based detection methods (in which some scanners may not support using the appropriate database-specific time delaying function).

The comparison of the scanners' SQL injection detection accuracy is documented in detail in the following section of sectoolmarket:

http://sectoolmarket.com/sql-injection-detection-accuracy-unified-list.html

Result Chart Glossary

Note:

During the assessment of Qualys it is highly likely that an optimization mechanism affected the scan results of POST test cases (compared to WAVSEP 2012 results). Although in the case of other vendors disabling similar mechanisms solved the problem, in the case of Qualys this optimization mechanism could not be disabled via the configuration interface. We are currently trying to find solutions to the problem.

The SQL Injection Detection Accuracy of Commercial/SAAS Scanners

The SQL Injection Detection Accuracy of Opensource/Free Scanners

The SQL Injection Detection Accuracy of Scanners – Unified List

16. Test IX - Attack Vector Support

The ninth assessment criterion is the number of audit features each tool supports.

For the purpose of the benchmark, an audit feature was defined as a common generic application-level scanning feature, supporting the detection of exposures which could be used to attack the tested web application, gain access to sensitive assets or attack legitimate clients.

Reasoning: An automated tool can't detect an exposure without a code module designed to identify the issue, and therefore, the number of audit features will affect the type (and amount) of exposures that the tool will be able to detect (assuming the audit features are implemented properly, that vulnerable entry points will be detected, that the tool will be able to handle the relevant scan barriers and scanning perquisites, and that the tool will manage to scan the vulnerable input vectors).

Although I typically place the assessment of supported audit features in a position of higher importance in the benchmark, my current research led me to make some changes.

I still consider the amount of supported generic vulnerability detection features (a.k.a audit plugins) to be a very significant aspect, probably more than I ever did.

Unfortunately, I came to the conclusion the current list that the WAVSEP project documents is like a drop in the ocean.

WAVSEP currently contains information on which scanners are relatively more audit-feature rich – relative, as in relation to other projects, not to the actual variety of attacks out there.

Although "relative" may still be very useful to the consumer, in my opinion, it's not as useful to the industry as I had hoped.

Originally, when I created the list of supported audit plugins which is currently used (and covers 32 attack categories at the moment), I composed it from the list of plugins that were commonly supported by scanners at the time (2009-2010).

Although the list was somehow limited, and by no means representative to the overall list of attacks that scanners should detect (and hopefully would one day be able to detect), it was enough to represent the differences between the products.

Five years passed – and many things changed.

Numerous new generic application-level attacks were invented, published or re-classified.

Projects like CWE, CAPEC, OWASP Testing Guide, Attacks and Vulnerabilities, WASC and others added more and more attack classifications, and that's without taking into account the numerous vectors that were published in blogs, conferences and competitions, which often didn't get the attention they deserved.

While the commonly implemented scanning features in scanners were usually derived from feature demands, attack vectors receiving higher levels of "popularity" and publicity, vulnerabilities that the vendors (and to some extent the users) perceived to be the most common or severe, and sometimes some vendor-specific "exotic" vectors, there was never any roadmap that will classify to consumers what was MORE important for vendors to support.

So after figuring that out, prior to the benchmark, I decided to expand my list of attacks and vulnerabilities, so I could properly map the contribution of the various tools against the overall risk map, and during the research stages that preceded this publication, I started researching which vectors that scanners can potentially identify actually exist, and which of those are supported by the individual scanners.

Well, it went pretty well…

In fact, it went so well that so far I classified 227 distinct application attacks;

227 attacks, not including multipliers due to persistent/session/indirect states, and I'm not even done mapping and classifying them.

Needless to say, that's a lot of mapping tasks for each individual product.

In fact, the effort of classifying and prioritizing those vectors while verifying which products supported them was so high, that I had to postpone their publication, or else the research you are currently reading might not have been published any time soon.

So, at the moment, this section describes the relative support for various audit features, and the rest of the content collected during the research will have to wait for another publication.

The detailed comparison of the scanners support for various audit features is documented in detail in the following section of sectoolmarket:

http://sectoolmarket.com/audit-features-comparison-unified-list.html

Note

The audit-feature count results of Webinspect may change in the coming days due additional verification processes I'm currently conducting. If eventually there are any changes, I will announce them using the comparison dedicated twitter account: @sectoolmarket

The Number of Audit Features in Scanners – Commercial/SAAS Tools

The Number of Audit Features in Scanners – Opensource/Free Tools

The Number of Audit Features in Scanners – Unified List

17. Test X - Adaptability - Scan Barriers

Applications may contain a variety of mechanisms and technologies that could be pose a barrier to a scanner – and in fact, effectively prevent it from being effective when scanning the application.

Scan barriers such as Anti-CSRF tokens, CAPTCHA mechanisms, platform specific tokens (such as required viewstate values) or account lock mechanisms have already become an integral part of many applications. Complicated RIA client technologies such as Flash, Applets and Silverlight are certainly not rare.

Although not necessarily a measurable quality, the ability of the scanner to handle different technologies and scan barriers is an important perquisite, and in a sense, almost as important as being able to scan the input delivery method.

Reasoning: An automated tool can't detect a vulnerability in a point and shoot scenario if it is can't locate and scan the vulnerable location due to the lack of support in a certain a browser add-on, the lack of support for extracting data from certain non-standard vectors, or the lack of support in overcoming a specific barrier, such as a required token or challenge. The more barriers the scanner is able to handle, the more useful it is when scanning complex applications that employ the use of various technologies and scan barriers.

The detailed comparison of the scanners support for various barriers is documented in detail in the following of sectoolmarket:

http://sectoolmarket.com/coverage-features-comparison-unified-list.html

The following charts show how many types of barriers each product claims to be able to handle (note that many of these features were not verified, and the information currently relies on documentation, research and vendor supplied information):

The Adaptability Score of Commercial/SAAS Scanners

The Adaptability Score of Opensource/Free Scanners

The Adaptability Score of Web Application Scanners – Unified List

18. Test XI - Authentication/Usability

Although supporting the authentication method required by the application seems like a crucial quality (and certainly is a convenient feature), in reality, certain scanner proxy chaining features can make-up for the lack of support in most of the authentication methods, by employing the use of a 3rd party proxy to authenticate on the scanner's behalf.

For example, if we wanted to use a scanner that does not support NTLM authentication (but does support an upstream proxy), we could have defined the relevant credentials in Burpsuite FE, and define it as an upstream proxy for the scanner we intend to use.

However, chaining the scanner to an external tool that supports the authentication still has some disadvantages, some of them major, such as reduced performance, potential stability issues, thread limitation and general inconvenience.

The following comparison table shows which authentication methods and features are supported by the various assessed scanners:

http://sectoolmarket.com/authentication-features-comparison-unified-list.html

19. Test XII - Results/Features vs. Pricing

The following assessment is in fact a summary of the important results, in comparison to the product price and features.

This section will probably be the most useful section for anyone looking to purchase a commercial or SAAS solution, or is debating whether or not to use open source products instead.

As I mentioned in the introduction, since web application scanners might actually be a bundle of several semi-independent products (generic vulnerability scanner, known vulnerability scanner, infection scanner, etc), it's very important to notice which modules are included in each offer, especially in relation to commercial scanner pricing.

WAVSEP currently focuses on assessing the generic vulnerability scanning module of web application scanners, and whatever it is you're paying might be relative to the rest of the modules the product contains (or does not contain), in case you actually need those.

In short, the scanner price might (or might not) reflect a set of products that could probably have been priced separately as independent products.

For your convenience, I invested some effort in mapping which of these products contain additional modules, although some classification of modules might still be missing.

The mapped modules include generic web-app scanning modules, generic web service scanning modules, flash application scanning modules and CGI scanning modules (a.k.a web server scanning modules or known vulnerability scanning modules).

The mapped categories don't yet include SAST and IAST scanning modules, Applet/Silverlight scanning modules, website infection scanning modules and additional categories which may be mapped in the future.

Another important issue to pay attention to is the type of license acquired.

In general, I did not cover non commercial prices in this comparison, and in addition, did not include any vendor specific bundles, sales, discounts and sales pitches.

I presented the base prices listed in the vendor website or provided to me by the vendors, according to a total of 6 predefined categories, which are in fact, combinations of the following concepts:

Consultant Licenses: although there isn't a commonly accepted term, I defined "Consultant" licenses as licenses that fit the common requirements of a consulting firm - scanning an unrestricted amount of IP addresses, without any boundaries or limitations.

Limited Enterprise Licenses: Any license that allowed scanning an unlimited but restricted set of addresses (for example - internal network addresses or organization-specific assets) was defined as an enterprise license, which might not be suited for a consultant, but will usually suffice for an organization interested in assessing its own applications.

Website/Year - a license to install the software on a single station and use it for one year against a single IP address (the exception to this rule is Netsparker, in which the price per website reflects 3 Websites).

Seat/Year - a license to install the software on a single station and use it for a single year.

Perpetual Licenses - pay once, and it's yours (might still be limited by seat, website, enterprise or consultant restrictions). The vendor's website usually includes additional prices for optional support and product updates.

The various prices can be viewed in the dedicated comparison in sectoolmarket, available in the following address:

http://www.sectoolmarket.com/price-and-feature-comparison-of-web-application-scanners-unified-list.html

It is important to remember that these prices might change, vary or be affected by numerous variables, from special discounts and sales to a strategic conscious decision of vendors to invest in you as a customer or as a beta testing site.

20. Additional Comparisons

The following section contains additional information on the tested tools that was documented throughout the research, and may be of use to the reader.

List of Tools

The list of tools tested in this benchmark, and in the previous benchmarks, can be accessed through the following link:

(*)List of Tested Scanners and Their Licenses, Notes, Vendor, Source Repository and Latest Update

Additional Features

Complementary scan features that were not evaluated or included in the benchmark:

(*)Complementary Scan Features

(*)General Scanner Features

In order to clarify what each column in the report table means, use the following glossary table:

Title	Possible Values
Configuration and Usage Scale	Very Simple - GUI + Wizard Simple - GUI with simple options, Command line with scan configuration file or simple options Complex - GUI with numerous options, Command line with multiple options Very Complex - Manual scanning feature dependencies, multiple configuration requirements
Stability Scale	Very Stable - Rarely crashes, Never gets stuck Stable - Rarely crashes, Gets stuck only in extreme scenarios Unstable - Crashes every once in a while, Freezes on a consistent basis Fragile – Freezes or Crashes on a consistent basis, Fails performing the operation in many cases
Performance Scale	Very Fast - Fast implementation with limited amount of scanning tasks Fast - Fast implementation with plenty of scanning tasks Slow - Slow implementation with limited amount of scanning tasks Very Slow - Slow implementation with plenty of scanning tasks

Scan Logs

In order to access the scan logs and detailed scan results of each scanner, simply access the scan-specific information for that scanner, by clicking on the scanner version in the various comparison charts: http://sectoolmarket.com/

21. What Changed?

Since the latest benchmark, many open source and commercial tools added new features and improved their detection accuracy.

The following list presents a summary of changes in the detection accuracy and coverage of Commercial tools that were tested in the previous benchmark (+new):

(*) NTOSpider – NTOSpider last assessment took place in 2011, and since then there has been a significant improvement in all the test categories, as well as new results for tests not performed in 2011. It also came out FIRST in the WIVET category (along with 3 other products) and the XSS category (along with 10 others), and got high scores in many others. The rankings it got for the new tests (redirect/backup) were mixed.

(*) N-Stalker (Commerical Edition) – The commercial edition of N-Stalker was assessed in this benchmark for the first time. The only comparable result was to the XSS result of the free version tested in 2012, and in that case, there was a significant improvement. The rest of the results got it various ranks.

(*) Qualys – Qualys was first tested in 2012, and since then. The WIVET score didn't change (still one of the highest), and there are some new test results as well, but the SQL Injection and Reflected XSS results are actually worse, due to what I currently attribute to temporary bugs, either in the product or (less likely) my testing procedure.

(*) ScanToSecure – Another new SAAS service which is assessed for the first time, and got results that were almost identical to those of Netsparker.

(*) Netsparker (Commercial Edition) – Netsparker results were improved in almost every single category. The WIVET score was slightly improved (one of the highest), it came out FIRST in the Reflected XSS (along with 10 others) and Remote File Inclusion (along with 4 others) categories, dramatically improved the previous Local File Inclusion results (one of the highest results), and got a great results in many other tests. Like the vast majority of the products in the industry, its results were somehow mixed in the new tests (backup/redirect).

(*)WebInspect – Webinspect significantly improved its scores in various categories: It was the only winner in the client/barrier coverage feature comparison, came out FIRST in the WIVET assessment (along with 3 others), the Remote File Inclusion (along with 4 others), Reflected XSS (along with 10 others) and SQL Injection (along with 4 others) categories, got surprisingly high score in the new (and secret) Unvalidated Redirect category (highest among commercial), and plenty of other high scores in different categories, but didn't get a good score in the backup/hidden file detection assessment.

(*) AppScan – AppScan too significantly improved its scores in various categories: It was the only winner in the Local File Inclusion and Supported Audit Features categories, got one of the highest WIVET scores, came out FIRST in the SQL Injection (along with 4 others), Reflected XSS (along with 10 others) and Remote File Inclusion (along with 4 others) categories, got plenty of high scores in other categories, but got mixed results in the new tests (backup/hidden files and unvalidated redirect).

(*) Acunetix WVS (Commercial Edition) – Acunetix slightly improved the results from the previous benchmarks, and got some very interesting new results: it got the BEST SCORE in the NEW Backup/Hidden Files category among commercial scanners (and some would argue, in total), came out FIRST in WIVET (along with 3 others), SQL Injection (along with 4 others), Reflected XSS (along with 10 others), got great results in many other categories, but didn't get a good score in the new unvalidated redirect category.

(*) Syhunt Dynamic – Syhunt dramatically improved their WIVET score (came out FIRST, along with 3 others), and slightly improved other scores as well (LFI, etc). They got a mixed result when scanning backup/hidden files, and didn't have a plugin to scan unvalidated redirect test cases (at least as far as I could tell).

(*) Burp Suite Pro – Burp is the undisputed winner of the (overall) versatility category, was the only winner in the input vector support category (followed closely by NTO, and less closely by Appscan, Webinspect and IronWASP), got one of the highest scores in detecting Backup/Hidden Files (relative), and decent scores in many other categories. It also came out FIRST in the SQL Injection (along with 4 others) and Reflected XSS (along with 10 others) categories, and dramatically improved its RFI score, but alas, didn't get a good score in the WIVET test (same as last year).

(*) WebCruiser – No significant changes compared to previous versions in the tested categories.

(*) ParosPro – was not retested, since no updates were released, so it has identical results.

(*) JSky – was not retested, since no updates were released, so it has identical results.

(*) Ammonite – was not retested, since no updates were released, so it has identical results.

The following list presents a summary of changes in the detection accuracy and coverage of Opensource/Free tools that were tested in the previous benchmark:

(*) ZAP – ZAP significantly improved almost all of its results. It implemented a new AJAX crawling feature that dramatically improved its WIVET score (highest among opensource) – but this feature optional and requires time to use. It came out FIRST in the Reflected XSS category (along with 10 others), got one of the highest scores in SQL Injection, Remote File Inclusion and Local File Inclusion, as well as a decent result in many others categories. If you take into account the external GoF plugin, ZAP is also the winner of the Backup/Hidden file detection category, although I'm leaving that interpretation to the reader. ZAP however, didn't get a good score when tested against unvalidated redirect test cases.

(*) IronWASP – Although IronWASP too had a new AJAX crawling feature, it was released too late for me to test it properly, and in my opinion, required a little more polishing (although rumors say it gets an insane WIVET score). It did however, make a clean (and unexpected) take away by being the only winner in the new and hidden Unvalidated Redirect category, with an impressive score that detected test cases that no other tool has. It also co-won the Reflected XSS category (along with 10 others), and got some great results in many other tests. Due to technical difficulties, I still don't a WIVET score for it, but hopefully will have soon.

(*) Skipfish – Skipfish is back in the game. Although previous version were relatively buggy, the currently tested version had very impressive results, notable result consistency (which unfortunately I did not measure), and a dramatic improvement in almost every test category I used it in. It got very impressive results in many categories, and also a relatively very high results in the unvalidated redirect category.

(*) Vega – Vega was definitely a surprising player in this benchmark. It came out FIRST in both Reflected XSS (along with 10 others) and Remote File Inclusion (along with 4 others). It got a fantastic WIVET result for an open source tool (the best opensource result without using a visible browser – something no other opensource tool with good result did – worth reusing for other java tools), and got very impressive results in both the Local File Inclusion (although with lots of false positives) and SQL Injection. Sadly enough, it didn't have plugins for unvalidated redirect or backup/hidden files that I could test.

(*) Arachni – although anyone that will install and use the latest version of arachni will immediately notice a significant improvement in usability – a very impressive improvement if I might add, and probably the most consistent behavior I saw – and unfortunately did not measure (the idea behind the "AutoThrottle" feature is very interesting – and probably responsible for some of the consistent results – since it got the same results regardless of how many URLs it scanned – very rare in this industry), a bug in the XSS plugins seemed to reduce its score in that category in comparison to the previous assessment, and another bug caused the backup/hidden file detection plugin not to function at all. It still came out FIRST in the Remote File Inclusion test (along with 4 others), improved some other results, and got the third best score in the NEW Unvalidated Redirect category (along with Webinspect), and also got me thinking on how easy it is to start a new SAAS business just by using it.

(*) W3AF – The development version of W3AF had several bugs that affected its score, and in fact, some results were actually worse than the last benchmark (bugs were reported to the vendor). It did however, still manage to surprise and get the best score for an opensource tool in the Unvalidated Redirect category (and second best score in that category in total), a relatively good result in the Backup/Hidden File detection category, and a couple of other results that were impressive, especially in the context of the open-source industry (wivet, features).

(*) WATOBO – WATOBO significantly improved both its SQL Injection and Reflected XSS scores, got the same scores in LFI, and got above average (relative) results in backup/hidden file detection (which were generally bad to mediocre for most tools), but at the time of the test did not have any RFI or Unvalidated Redirect features I could test.

(*) WAPITI – those who recall this tool which got surprisingly high scores in previous benchmarks, would be delighted to know that the project has been recently revived and that a new version was released. It got relatively good results (impressive WIVET for an opensource tool), as well as improvement in almost every category. It did however, have a hard time with the Backup/Hidden file category in which it got a low score.

(*) N-Stalker 2012 FE Significantly improved its Reflected XSS Score compared to the previous benchmark.

(*) Netsparker Community/Free Edition got some slight improvements in some of its scores, and still has one of the best WIVET scores for a free tool, but in the overall, there were no major changes compared to the previous benchmark.

(*) SQLMap, WebSecurify, Acunetix FE and a couple of other projects were not retested, and most of the features of Syhunt Mini, AndiParos and Paros were not retested (although the latter three got some new results for unvalidated redirect and backup/hidden files).

22. Opensource vs. Commercial - Insights?

The conclusions I have this year in relation to the open source vs. commercial tools enigma are not as decisive compared to the previous year.

Part of that is because I didn't yet completed all the analysis processes I planned, and part of it because there really was a significant improvement in the open source industry (and without taking lightly the significant improvements that took place in the commercial section).

Projects such as ZAP and IronWASP started supporting scanning input delivery methods of modern web applications, including JSON/AJAX, XML, and even nearly unique vectors such as OData and GWT, that even most commercial vendors don't support.

Projects like W3AF have long ago been almost as feature rich as Webinspect and Appscan (although they still lack stability), Vega is coming closer to having a crawling mechanism that can produce similar results to that of a commercial vendor, and if I were Qualys (or any other cloud vendor), I would watch the Improvement of the Arachni project CLOSELY. Seriously – Install it and give it a shot… The results don't emphasize the maturity level it got to.

However, in sheer numbers, as an overall solution, most open source tools still lag a bit behind some of the major commercial players, at least if you take into account all the categories… although I admit that I don't say that with the same confidence as I did before, and I believe that further analysis is required to get to a practical conclusion.

23. Verifying the Benchmark Results

The results of the benchmark can be verified by replicating the scan methods described in the scan log of each scanner (accessible in sectoolmarket through the version link of each product), and by testing the scanner against WAVSEP v1.5 (obtained from the sourceforge WAVSEP repository) and WIVET v3-revision148.

The same methodology can be used to assess vulnerability scanners that were not included in the benchmark.

24. So What's Next?

During this research, which I have been conducting for the past 18 months or so (7 of those just to gather the results you are currently seeing), I gathered a ton of information.

Due to my consistently tight schedule, too many adventurous endeavors and the fact that I didn't want to delay the publication any longer, I didn't publish A LOT of content that was gathered, so in the next couple of weeks I'm going to try and wrap it up so it could come to fruition ASAP…in my opinion, the conclusions from the unpublished content can be very interesting for the technological trends in this industry.

The benchmark was branded as part I for a reason, and although I might add the results of additional products soon, in the upcoming weeks, I plan to focus on trying to see how much effort will be required to release part II, which will have a very different result format compared to the typical WAVSEP benchmark.

25. Recommended Read-List: Benchmarks

The following resources include additional information on previous benchmarks, comparisons and assessments in the field of web application vulnerability scanners:

(*) "HackMiami Web Application Scanner 2013 PwnOff", by James Ball, Alexander Heid, Rod Soto (a comparison of 5 web application scanners published at the HackMiami 2013 conference).

(*) "Top 10: The Web Application Vulnerability Scanners Benchmark, 2012", one of the predecessors of the current benchmark, by Shay Chen (a comparison of 60 commercial and open source scanners, July 2012)

(*)"Enemy of the State: A State-Aware Black-Box Web Vulnerability Scanner", by Adam Doup´e, Ludovico Cavedon, Christopher Kruegel, and Giovanni Vigna (a comparison of 3 scanners published in 2012).

(*)"SQL Injection through HTTP Headers", by Yasser Aboukir (an analysis and enhancement of the 2011 60 scanners benchmark, with a different approach for interpreting the results, March 2012)

(*)"The Scanning Legion: Web Application Scanners Accuracy Assessment & Feature Comparison", one of the predecessors of the current benchmark, by Shay Chen (a comparison of 60 commercial and open source scanners, August 2011)

(*)"Building a Benchmark for SQL Injection Scanners", by Andrew Petukhov (a commercial and opensource scanner SQL injection benchmark with a generator that produces 27680 (!!!) test cases, August 2011)

(*)"Webapp Scanner Review: Acunetix versus Netsparker", by Mark Baldwin (commercial scanner comparison, April 2011)

(*)"Effectiveness of Automated Application Penetration Testing Tools", by Alexandre Miguel Ferreira and Harald Kleppe (commercial and freeware scanner comparison, February 2011)

(*)"Web Application Scanners Accuracy Assessment", one of the predecessors of the current benchmark, by Shay Chen (a comparison of 43 free and open source scanners, December 2010)

(*)"State of the Art: Automated Black-Box Web Application Vulnerability Testing" (Original Paper), by Jason Bau, Elie Bursztein, Divij Gupta, John Mitchell (May 2010) – original paper

(*)"Analyzing the Accuracy and Time Costs of Web Application Security Scanners", by Larry Suto (commercial scanners comparison, February 2010)

(*)"Why Johnny Can’t Pentest: An Analysis of Black-box Web Vulnerability Scanners", by Adam Doup´e, Marco Cova, Giovanni Vigna (commercial and open source scanner comparison, 2010)

(*)"Web Vulnerability Scanner Evaluation", by AnantaSec (commercial scanner comparison, January 2009)

(*)"Analyzing the Effectiveness and Coverage of Web Application Security Scanners", by Larry Suto (commercial scanners comparison, October 2007)

(*)"Rolling Review: Web App Scanners Still Have Trouble with Ajax", by Jordan Wiens (commercial scanners comparison, October 2007)

(*)"Web Application Vulnerability Scanners – a Benchmark" , by Andreas Wiegenstein, Frederik Weidemann, Dr. Markus Schumacher, Sebastian Schinzel (Anonymous scanners comparison, October 2006)

26. Acknowledgements

While performing the research described in this article, I have received help from plenty of individuals and resources, and I’d like to take the opportunity to acknowledge them all.

To the researchers Ozhan Sisic and Sharath Unni which contributed content and results to the assessment, and did so at the expense of their own time, in dense timeframes, and often in unreasonable hours and timeframes.

To the various additional volunteers that did their best to assist me whenever they could, especially to the ones that chose to stay anonymous.

To the various members at Denim Group, and especially Dan Cornel, which assisted throughout the project, adapted their excellent platform Threadfix to fit my needs, and enabled me to handle nearly unreadable results and share information with volunteers that participated in the tests around the world.

For the various entities and projects that contributed code to WAVSEP, including (but not limited to) the various authors of the ZAP project and Lavakumar Kuppan from the IronWASP project.

To Dan Kuÿkendall from NTOBJECTives who permitted me to use their online enhanced adaptation of WIVET as an additional verification mechanism for the local WIVET results.

For all the open source tool authors that assisted me in testing the various tools in unreasonable late night hours and bothered to adjust their tools for me, discuss their various features and invest their time in explaining how I can optimize their use,

To the CEO's, Product Managers, Marketing Executives, QA engineers, Support Personal and Development teams of commercial vendors, which saved me tons of time, supported me throughout the process, helped me overcome obstacles and made my experience a pleasant one.

To the various information sources that helped me gather the list of scanners over the years, spread the news about the previous benchmarks, and gain knowledge, ideas, and insights, including (but not limited to) information security sources such as Security Sh3ll (http://security-sh3ll.blogspot.com/), PenTestIT (http://www.pentestit.com/), The Hacker News (http://thehackernews.com/), Toolswatch (http://www.vulnerabilitydatabase.com/toolswatch/), Darknet (http://www.darknet.org.uk/), Packet Storm (http://packetstormsecurity.org/), Google (of course), Twitter (and the never-ending list of favorites I keep there) and many others great sources that I have used over the years to gather the list of tools.

I can't thank you all enough, and wish you all the best.

27. Appendix A: Tools That Were Not Included

The following commercial web application vulnerability scanners were not included in the benchmark, due to deadlines and time restrictions from my part:

Commercial Scanners not included in this benchmark

(*)Websure

(*)Hailstorm (Cenzic)

(*)McAfee Vulnerability Manager (McAfee / Foundstone)

(*)NeXpose Enterprise Edition Web Application Scanning Features (Rapid7)

(*)Retina Web Application Scanner (eEye Digital Security)

(*)SAINT Scanner Web Application Scanning Features (SAINT co.)

(*)WebApp360 (NCircle)

(*)Core Impact Pro Web Application Scanning Features (Core Impact)

(*)Parasoft Web Application Scanning Features (a.k.a WebKing, by Parasoft)

(*)MatriXay Web Application Scanner (DBAppSecurity)

(*)Falcove (BuyServers ltd, currently Unmaintained)

(*)Safe3WVS 13.1 Commercial Edition (Safe3 Network Center)

The following open source web application vulnerability scanners were not included in the benchmark, mainly due to time restrictions, but might be included in future benchmarks:

Open Source Scanners not included in this benchmark

(*)Vanguard

(*)WebVulScan

(*)SQLSentinel

(*)OWASP Xelenium

(*)XssSniper

(*)Rabbit VS

(*)Spacemonkey

(*)Kayra

(*)2gwvs

(*)Webarmy

(*)springenwerk

(*)Mopset 2

(*)XSSFuzz 1.1

(*)Witchxtoolv

(*)PHP-Injector

(*)XSS Assistant

(*)Fiddler XSSInspector/XSRFInspector Plugins

(*)GNUCitizen JAVASCRIPT XSS SCANNER

(*)Vulnerability Scanner 1.0 (by cmiN, RST)

The following is a partial list of SAAS scanners were not included in the benchmark, mainly due to time restrictions, but might be included in future benchmarks:

SAAS Online Scanning Services

Appscan On Demand (IBM), Click To Secure, Sentinel (WhiteHat), Veracode (Veracode), Quatrashield, Veracode Dynamic Analysis, edgescan, VUPEN Web Application Security Scanner (VUPEN Security), WebInspect (online service - HP), WebScanService (Elanize KG), Gamascan (GAMASEC – currently offline), Cloud Penetrator (Secpoint), Zero Day Scan, DomXSS Scanner, Golem Technologies, etc.

Web Application Testing Tools which are using Dynamic Runtime Analysis (IAST):

(*)Seeker (Quotium)

(*)Appscan Glassbox (IBM)

(*)Contrast (Contrast Security)

(*)PuzlBox (currently named PHP Vulnerability Hunter)

(*)Inspathx

The benchmark focused on web application scanners that are able to detect at least Reflected XSS or SQL Injection vulnerabilities, can be locally installed, and are also able to scan multiple URLs in the same execution.

As a result, the test did not include the following types of tools:

Scanners without RXSS / SQLi detection features:

(*)Dominator (Firefox Plugin)

(*)fimap

(*)lfimap

(*)phpBB-RFI Scanner

(*)DotDotPawn

(*)LFI (Library-level Fault Injector)

(*)lfi-scanner

(*)LFI-Scanner

(*)lfi-rfi2

(*)LFI/RFI Checker (astalavista)

(*)CSRF Tester

(*)Etc.

Passive Scanners (response analysis without verification):

(*)Watcher (Fiddler Plugin by Casaba Security)

(*)Skavanger (OWASP)

(*)Pantera (OWASP)

(*)Ratproxy (Google)

(*)CAT The Manual Application Proxy (Context)

(*)Etc.

Scanners of specific products or services (CMS scanners, Web Services, etc):

(*)WSDigger

(*)Sprajax

(*)ScanAjax

(*)Joomscan

(*)wpscan

(*)Joomlascan

(*)Joomsq

(*)WPSqli

(*)Etc.

Uncontrollable Scanners

Scanners that can’t be controlled or restricted to scan a single site, since they either receive the list of URLs to scan from Google Dork, or continue and scan external sites that are linked to the tested site. This list currently includes the following tools (and might include more):

(*)Darkjumper 5.8 (scans additional external hosts that are linked to the given tested host)

(*)Bako's SQL Injection Scanner 2.2 (only tests sites from a google dork)

(*)Serverchk (only tests sites from a google dork)

(*)XSS Scanner by Xylitol (only tests sites from a google dork)

(*)Hexjector by hkhexon – also falls into other categories

(*)d0rk3r by b4ltazar

Deprecated Scanners

Incomplete tools that were not maintained for a very long time; currently includes the following tools (and might include more):

(*)Wpoison (development stopped in 2003, the new official version was never released, although the 2002 development version can be obtained by manually composing the sourceforge URL which does not appear in the web site- http://sourceforge.net/projects/wpoison/files/ )

De facto Fuzzers

Tools that scan applications in a similar way to a scanner, but where the scanner attempts to conclude whether or not the application or is vulnerable (according to some sort of “intelligent” set of rules), the fuzzer simply collects abnormal responses to various inputs and behaviors, leaving the task of concluding to the human user.

(*)Lilith 0.4c/0.6a (both versions 0.4c and 0.6a were tested, and although the tool seems to be a scanner at first glimpse, it doesn’t perform any intelligent analysis on the results).

(*)Spike proxy 1.48 (although the tool has XSS and SQLi scan features, it acts like a fuzzer more then it acts like a scanner – it sends payloads of partial XSS and SQLi, and does not verify that the context of the returned output is sufficient for execution or that the error presented by the server is related to a database syntax injection, leaving the verification task for the user).

Fuzzers

Scanning tools that lack the independent ability to conclude whether a given response represents a vulnerable location, by using some sort of verification method (this category includes tools such as JBroFuzz, Firefuzzer, Proxmon, st4lk3r, etc). Fuzzers that had at least one type of exposure that was verified were included in the benchmark (Powerfuzzer).

CGI Scanners

Vulnerability scanners that focus on detecting hardening flaws and version specific hazards in web infrastructures (Nikto, Wikto, WHCC, st4lk3r, N-Stealth, etc)

Single URL Vulnerability Scanners

Scanners that can only scan one URL at a time, or can only scan information from a google dork (uncontrollable):

(*)Havij (by itsecteam.com)

(*)Hexjector (by hkhexon)

(*)Simple XSS Fuzzer [SiXFu] (by www.EvilFingers.com)

(*)Mysqloit (by muhaimindz)

(*)PHP Fuzzer (by RoMeO from DarkMindZ)

(*)SQLi-Scanner (by Valentin Hoebel)

(*)Etc.

Vulnerability Detection Toolkits

Tools that aid in discovering vulnerabilities, but do not detect the vulnerability themselves; for example:

(*)Exploit-Me Suite (XSS-Me, SQL Inject-Me, Access-Me)

(*)Fiddler X5s plugin

(*)XSSRays (chrome Addon)

Exploitation Tools

Tools that can exploit vulnerabilities without any independent ability to automatically detect vulnerabilities on a large scale. Examples:

(*)MultiInjector

(*)XSS-Proxy-Scanner

(*)Pangolin

(*)FGInjector

(*)Absinth

(*)Safe3 SQL Injector (an exploitation tool with scanning features (pentest mode) that are not available in the free version)

(*)Etc.

Exceptional Cases

(*)SecurityQA Toolbar (iSec) – various lists and rumors include this tool in the collection of free/open-source vulnerability scanners, but I wasn’t able to obtain it from the vendor’s web site, or from any other legitimate source, so I’m not really sure it fits the “free to use” category.

The 2013 scanner benchmark is coming soon!

2013-12-10T01:58:00.000-08:00

To all those who are interested in the latest an greatest, I'm currently working on the 2013/2014 web application scanner benchmark, and already I'm seeing some VERY interesting results.

The benchmark will be published soon, and I'm posting many of the results during the assessment process using the comparison twitter account @sectoolmarket, which also publishes news about other information security product comparisons performed around the globe.

This time, I received plenty of help from multiple entities -

Many entities (including the ZAP project and IronWASP project) contributed test cases to wavsep (not included in this benchmark scope, but might be in the next),

Several researchers around the globe offered their help in the assessment process (encouraging me to work on something that will someday make it easier),

And last but not least, I received plenty of help from the wonderful guys at Denim group, which did their best to adjust ThreadFix so I can use it to make the task of comparing and counting results easier (just started checking it - looks great so far)

Wavsep was already enhanced to v1.5 (with hundreds of additional test cases that will be published after the upcoming benchmark),

The vast majority of commercial vendors already provided me with a valid license and installation, and at least half of the planned open source projects were either tested or currently being tested.

I'm planning to release the information gathered in two or three bulks -

(*) The typical benchmark and analysis (including at least two new vulnerability detection comparison aspects which will remain obscure at the moment - for the sake of the competition).
(*) An analysis of the DAST market status, based on the results and additional information gathered during the test.

I'm also planning to upload the results into a dynamic publication framework (partially implemented), although the first bulk of information will probably be published in the blog and static sectoolmarket website.

In short, stay tuned, results will be published soon .

Security Benchmarks & Comparisons – Plans for 2013

2013-05-19T12:06:00.006-07:00

It's kind of hard to admit that your current strategy leads to a dead end… Hard, but liberating.

I initially started this blog because I was searching for a way to sort through an insane amount of tools I collected over the years - so we can all weed out the irrelevant and stick with what works.

Obviously, things got a little complicated, and after doing double shifts and spending half my nights over the past 4 years on comparisons, I realize now that I only covered 60-70 tools.

Sure, I had a good reason to do so - learning curve, comprehensiveness, accuracy, credibility, evolution… but the numbers don't lie.

As much as I like the idea of a one man army, the current rate is NOT what I expected, and to achieve something greater, I'll need to get some resources and some help (yeah yeah, mental too).

Nope, that DOES NOT mean that I'm about to stop any of my planned activities, researches or benchmarks. Giving up is for wusses.

It does mean, however, that I'm going to make some changes that will enable me to cover more, even if I have to make some decisions I was dreading and trying to postpone.

So what I'm planning for 2013 is to branch out and cover additional types of tools & products, in addition to vulnerability scanners.

That means updating WAVSEP with some hybrid issues, becoming less of a control freak, let go the leash I was so inclined on keeping, and probably even creating additional comparison platforms.

Yep… b a c k t o w o r k.

SAAS Scanners (Qualys) vs Commercial Scanners

2012-07-30T16:04:00.000-07:00

The results of Qualys - a SAAS scanning service provider were added to http://www.sectoolmarket.com.
Qualys got great scores in WIVET, SQLi & RXSS, but don't detect RFI/Path Traversal cases, and I'm currently not sure if it's my testing methodology, bugs or the absence of appropriate plugins.
Link to the 2012 benchmark: http://bit.ly/LloTfL (Qualys results are only included in sectoolmarket)

The Diviner - Clairvoyance in the Digital Frontier

2012-07-30T15:10:00.001-07:00

The Diviner

Digital Clairvoyance

Server-Side Source Code and Memory Divination

How to gain insight into the server-side source code and memory structure of any application, using black box techniques and without relying on any security exposures.

By Shay Chen and Eran Tamari

POC is implemented as a ZAP proxy extension, developed by Hacktics ASC.

Download:

The Diviner Extension Homepage

Demonstration Videos:

Source Code Divination

Persistent SQL Injection

Persistent XSS Detection

Introduction

There's a LOT of quality infosec publications lately, in blog posts, articles, videos and whitepapers. Even though I try my best, I admit it's hard for me to keep up.

Although this post is one of these publications, I already admit that the title sounds a bit confusing and maybe even scary, and I am aware of that since that's a response I got from many individuals.

So what's so special in this post that should make you want to invest 5 minutes of your precious time to read it?

I could tell you stories about research and development work that's been going on for more than a year, or mention the fact that it contains an entirely new concept in hacking, but I think I'll take the direct approach with this one:

Using a new technology that relies on black box techniques, the server-side source code of any application can be stolen, the server side memory can be mapped, and so can the data flow of server side values.

The technique is already implemented in a new tool, does not rely on any security exposures, and works regardless of any existing security enhancements.

No introductions, obscure concepts or murky waters. Just facts - Get Code, Get Memory Map, No Security, Any Application.

Let's assume for a moment that the proclamations are true - so how can this information be used in penetration tests?

Although the posts in this blog were recently focused at automated scanning, it's never too late to correct the faults. Any veteran knows that the focus of any tester should always be the manual testing process, and this new information, when properly presented to a tester, can dramatically enhance the process of a manual penetration test:

Optimization of the manual testing process - allow the tester to make better decisions, faster and test entry points that are more likely to be vulnerable first.
Gaining Intel - enable the tester to understand how a certain page / entry point behaves under various conditions, by viewing a representation of the server-side source code, memory and cross-entry-point processes.
Locate complex vulnerabilities - locate leads for vulnerabilities that require access to multiple entry points, while overriding session and database values, with various perquisites and in extreme scenarios. Vulnerabilities that cannot be detected by automated tools, and are hard to locate even in manual assessments.
Think about it… viewing the server-side source code of any component… criteria or not, it's simply awesome.

In addition, if the information can be delivered in a standard format to a black box web application scanner, it can enhance the coverage of the tool to include potential events and behaviors that only occur under extreme or rare conditions.

And what enables us to gather this information using nothing but black box techniques?

Well, I can only define it as... umm... breadcrumbs. Many tiny, seemingly useless pieces of information.

So if having the ability to gain Insight into the server side, reducing the time necessary to perform many types of tests and being able to locate vulnerabilities that nobody else can detect without sheer luck is of any interest to you, hang on a bit.

And just to make sure you're not losing track, here's one way to present it:

Activating Diviner's Clairvoyance feature - viewing a representation of the server side code

Viewing the Dynamic Server Memory & Processes Map Generated by Diviner

The Problem – The Limitations of Manual Pentesting

The process of manual penetration testing is a process of trial and error, which is composed of event-triggering attempts, behavior analysis and deduction;

Through a process of trial and error, the tester learns how a certain application entry point responds to specific input, access patterns and extreme conditions, locates behaviors that might be caused by potential vulnerabilities, and verifies (or rules out) the existence of these vulnerabilities through exploits, comparisons, etc.

Since there are dozens of potential generic application-level attacks (read the lists in OWASP, WASC and CWE if this number sounds exaggerated), excluding the use of scanners and fuzzers and with the exception of very small applications, this process can only be manually performed on part of the tested application entry points, and relies heavily on experience, intuition, methodology and sometimes luck.

The point I am trying to make is this - currently, there is an inefficient use of time in the process of manual penetration testing.

Don't jump to conclusions or take it personally... let me explain my intention:

Even though efficient information gathering enables the tester to narrow the list of tests that should be performed on each application, entry point, page or parameter - it still includes a lot of tests to perform, often more than the tester can do in the time allocated to the test.

Furthermore, since the most of the global information gathering processes rely on information disclosure, passive information gathering and fingerprinting, the tester needs to manually gather information on specific targets prior to testing them, or perform the test "blindly", while relying on other incentives.

Take SQL injection for example, one of the most common tests that penetration testers attempt to perform. In order to truly be certain that a certain location is (or isn't) vulnerable, the tester needs to receive different kinds of feedback; Sometimes a visible or hidden error make the task simple (blablabla.SQLException), Sometimes the tester needs to dig deeper and detect content differentiation, or compare responses to inputs that contain arithmetic or mathematical operations (id=4-2 vs id=5-3). When the tested entry point does not provide any feedback, he might be required to use payloads that are designed to delay the execution of SQL statements, and if an exposure with a similar obscure behavior affects an offline process or an indirectly affected backend server, he/she might even need to inject payloads that execute an exploit that alters content (risky) or sends a notification to external entities (mail, ping, etc).

Assuming the assessment method is a black box assessment, since there are various types of databases and syntax injection contexts, the tester will need to use a lot of payloads to truly verify the issue - in each field, and in each location.

Scanners attempt to tackle this issue by performing various tests on a wide range of targets, but conclude themselves whether or not the location is vulnerable, and currently, are far from performing these tests in a sufficient amount of extreme or complex scenarios.

Fuzzers on the other hand can store the different responses and behaviors of multiple entry points, but don't provide out-of-the-box support for complex processes or complex analysis methods, are usually not application-aware, and present the information in a way that is hard to digest.

The problem, however, could be handled using another method:

Divination attacks, a crossbreed between automated testing and human deduction, provide an alternate (or complementary) route:

Consider the methods required to detect the following complex vulnerability:

"SQL injection vulnerability, in which the *attack payload* is injected into a server variable in the *registration phase*, stored in the *database*, but only affects the application in the *event of writing an exception into a database log* (the vulnerable code segment), which only occurs in a module that generates the *monthly report* for a user, which requires *authentication*, while the log triggering exception requires the user to *directly access* the last phase of a multiphase report generation process while skipping the rest of the phases in the flow (forceful browsing)."

In other words, a vulnerability that affects the application indirectly, and only when certain extreme scenarios occur.

Although talented (or lucky) lucky testers might be able to detect it in a limited scope, it's unlikely that it will be detected by a black box automated vulnerability scanner, passive security scanner, or any other black-box tool… that is unless a certain process will make it possible…

Divination Attacks

When using the general term "Divination", this article refers to the following interpretation:

"Divination is the attempt to gain insight into a question or situation by way of an occultic standardized process or ritual. Used in various forms for thousands of years, diviners ascertain their interpretations of how a querent should proceed by reading signs, events, or omens." - Wikipedia's Definition for Divination.

For those of you that read this section first, and for those that got confused from the introduction, please, let me clarify: I am not proposing to hire the practitioners of witchcraft to participate in penetration tests.

I am however, proposing the following solution to the time management problem:

Inspect the direct and indirect effect of each parameter, on each page, with every possible sequence and under every possible condition, before deciding which attack to perform, and where.

Since obtaining this information manually is not probable, the process needs to be, at least in some aspects, automated.

And how can we obtain this information using an automated process?

Execute Scenarios -> Isolate Behaviors -> Perform Verifications -> Interpret -> GUI

Assume that interception proxy contains the following requests in its request history:

In order to analyze the effect of a given input parameter on other entry points (and on the origin entry point), we need to send a value to the target parameter, and then access another entry point - in order to see the effect (for example, send a value in the username input parameter to request 4, and then access request 6 to see if there was any special effect).

The process must be repeated for the next "exit point", while sending another value (identical or otherwise) to the target parameter, prior to accessing the "exit point".

The result of this analysis might change due to various factors, such as:

Authentication - Authenticate before accessing the entry point, before accessing the "exit point" (a.k.a target), or not at all.
Multiple Sessions - When an entry point responds by replacing the session identifier, the scenario could continue using the old session identifier (assuming it was not invalidated), or using the new one.
History Requirements – Certain entry points might require the execution of previous entry points using a shared session identifier. For example, testing a parameter sent to the fourth phase of a multiphase process might require access to previous entry points using the same session identifier, with, or without authentication.
Input Type - The target "exit point" and "entry point" might respond differently to other types of input (e.g. input with random values, valid values, invalid syntax characters, etc).
Required Tokens – Certain behaviors might only occur when a required token is sent to the entry point (or not sent to the entry point) – for example, the existence of a timestamp or anti-CSRF token might affect each entry point in different ways.
Invalid Access – accessing pages without meeting their "requirements" might still generate a "beneficial" behavior – for example, accessing a page without a valid anti-CSRF token might trigger a response that reuses a server variable that can be affected, and thus, expose the entry point to attacks.

So in order to truly analyze the effect of the parameter on the various entry points of the application, we need to try everything (or at the very least – try a lot of scenarios), and we need to do it to as many input parameters as possible, to as many entry/exit points as possible, and in various scenarios.

Furthermore, the behavior itself might vary according to the scenario, input and in-page logic: it can be input reflection, exception, a certain valid response, time delay, content differentiation or anything else; the behaviors that we are interested in are behaviors that can be traced back to a certain process, memory allocation, potential issue or a specific line of code.

The information gathered in such a process will be composed of a lot of behaviors, which vary per page, per input, and per scenario.

These "behaviors" can then be presented to the tester in a simple, visual form, which will enable him to decide which behaviors he should inspect manually.

Don't get me wrong - I am not suggesting that we limit the inspection only to the information presented by such a process - I'm merely stating that it is wise to focus on this information first, and verify the various leads it provides before using the hardcore manual approach. After using this approach for some time, I can clearly state the following:

The information provided by the process, when used by a tester, can transform even a very complex vulnerability into a low hanging fruit.

And that's not all. The collection of behaviors can also be "converted" into other useful forms, such as the ones presented in the following sections.

Source Code Divination

Source code divination is a new concept and approach (can also be referred to as source code fingerprinting).

Think about it - we use fingerprinting techniques to identify web servers, content management systems, operating systems, web application firewalls, and more.

Why not use the same approach to identify specific lines of code? Why not use it to detect all the lines of code, or at the very least, a large portion of the server code?

Nearly all of us classify source code disclosure, or attacks that can obtain the server source code as severe exposures (at least to some extent), and claim in the reports that we provide to customers that attackers can harness this information to enhance their attacks, learn about the system's structure and identify potential flaws in it.

If a large portion of the application's source code could be obtained using accurate "fingerprinting", wouldn't that lead to the same result?

In order to explain how this information can be obtained, let's use an example:

Connection pool exhaustion (or consumption) is one of the many forms of application denial of service attacks. It occurs when an attacker intentionally accesses an entry point (page/web service, etc) that requires a database* connection pool, using multiple threads – more threads the maximum amount of connections in the pool. The attack will delay the responses from entry points that rely on the pool, but won't affect entry points that don't use it (assuming the amount of threads don't affect other resources).

Although this behavior is an exposure in its own right, it also leads to the following conclusion:

It is highly likely that somewhere in the entry point's code, a connection is obtained from a connection pool, and since in many cases, a connection pool is a mechanism used to interact with databases, it's highly likely that the source code is similar to the following (jsp sample):

try {

Connection conn = DriverManager.getConnection(…);

…

} catch (…) {…}

Of course – this connection pool might serve a different type of resource, but using additional verifications we might be able to increase the level of certainty – for example, identifying erroneous databases responses in the same entry point, or even detecting certain exposures in other application entry points.

The same approach can be used to convert other behaviors to the lines of code that might have caused them, and since the previous process gathered a lot of behaviors – these can be converted into a fair amount of code - pseudo code that can be presented using any specific syntax, and enable the tester to understand how a certain page behaves – prior to testing that page.

For example, input sent from one page (the "source" page), but reflected in another (the "target" page) is probably shared through a session variable, file or database field. The origin can be isolated by accessing the target page using a different session identifier, but using the same identical process used to access it before (login, history, etc) - with the exception of the source page;

If the reflected input is not present in the target page, the probability for the existence of the following lines of code in the source page and target page increases:

Source Page:

String input1 = request.getParameter("input1");

session.setAttribute("sessionValue1", input1 );

Target Page:

out.println(session.getAttribute("sessionValue1"));

If however, the reflected input would have been present at the verification scenario, than the source code matching the pattern will probably include database access, file access or static server variables – and specific aspects of these behaviors can be isolated in turn (insert statements are more likely to exist in pages that rapidly increase in size, update statements in pages with relatively static size and persistent changes, etc).

At the end of the processes, after performing additional verifications and tests, the options with the highest probability will be selected and presented to the user.

And how will this code be sorted? Which lines will appear first?

Although the sorting problem has many solutions, one of the main solutions is probably "delay-of-service" attacks (yes, I said delay, not deny).

Presented in the research "Temporal Session Race Conditions", these attacks were originally meant to delay the execution of specific lines of code, in order to extend the lifespan of temporary session variables – but these attacks can also be used to sort some of the code – by inspecting if exceptions or conditional behaviors occur instead of the delay, before the delay, after the delay or not at all.

For example, performing a connection pool exhaustion attack on a page while simultaneously sending an error generating value to the same vulnerable page will provide a potentially important piece of information – which code is executed first: the code that attempts to obtain a connection from the pool, or the code that is prone to the exception.

Note - Although this method isn't exactly "safe", it will probably enhance the results more than other methods for sorting divined lines of code.

Like fingerprinting, this information might not be 100% accurate (although it can be VERY accurate, if the processes is performed properly and thoroughly), but can still be very beneficial for the purpose of the test – just like other forms of fingerprinting.

I won't expand the subject of source code divination in this post (I do have plans to discuss it further in separate posts), but it's already implemented in the diviner extension that will be discussed in the following sections.

Memory Structure Divination and Cross Entry-Point Effects

In the previous process, we have discussed how an identified behavior (such as an exception or input reflection) can be classified as persistent or temporary – by reproducing the scenario that caused it using a different session identifier, identical process, and without accessing the "entry point" (source page). This process, alongside additional verifications allowed us to conclude whether a behavior is persistent, temporary or something else.

Although not all the behaviors rely on specific variables that are stored in the server side, some do, and from these behaviors we can conclude how and where does the server stores some of the content.

By crossing the information obtained from interesting scenarios that were discovered in the process, we can even locate multiple entry points that affect the same database tables, fields, session variables and static variables, and thus, construct a general structure of database tables and session attributes.

It's key to understand that the process does not verify the existence of any exposures or attempts to exploit any vulnerability; instead, it's simply uses a method of deduction to attempt to present what's going on behind the scenes, in order for this information to enhance the abilities of a tester, or a scanner.

The Diviner Extension

During the last year, I collaborated with a number of individuals (especially with @Secure_ET, various colleagues and the OWASP ZAP project) so that these ideas will not remain a theory… and after numerous late night brainstorming sessions, various incarnations and a long development period – we have an initial version that works (beta phase).

The diviner platform – an active information gathering platform that implements many of the previously described concepts, is implemented as a ZAP proxy extension, and can be downloaded from the following address:

http://code.google.com/p/diviner/

It can already illustrate server side behaviors and processes, contains features such as the task list/advisor which provide invaluable leads to potential exposures, present a partial map of the server side memory, and present a partial representation of the server side code.

The extension is deployed using a windows installer (or in binary format for other operating systems), and requires java 1.7.x and ZAP 1.4.0.1 in order to run properly.

Furthermore, since it attempts to identify behaviors that result from valid & invalid scenarios, and can't guess what is valid on its own, it must be used after a short manual crawling process that covers the important application sections with valid values.

It was tested mostly on small scale applications (100+- parameters, +-50) – including real-life applications, and although it will probably work on larger applications (it's not stuck in the database analysis process – be patient) – due to various optimizations (and sacrifices) we didn't yet make – it's recommended not to exceed that size.

We can currently identify 20+- different lines of code, but have plans to implement tests that identify other lines of code, some with high probability, and some with absolute certainty.

We didn't yet implement features that sort the lines of code (and thus, currently rely on default positioning), but plan on implementing them in the future (with restrictions that will prevent their use for actual denial/delay of service attacks).

We have many additional experimental features that aren't mature enough, but are already working on refining them for the future versions.

We don't perform any form of automated vulnerability scanning, but plan on exporting the interesting leads to a format that can be used by external scanners to detect exposures in these abnormal scenarios.

Bottom line - It's not perfect yet, but it's already very useful, and can already help testers locate exposures that can't be located using other means, and make better decisions - quicker.

Acknowledgements

The diviner project was funded by Hacktics ASC.

The following individuals assisted me in various ways, and deserve acknowledgment for their contribution:

Eran Tamari (The lead developer) - for the countless hours of development, the sheer determination, and most of all, for being a true believer.

Simon Bennetts (psiinon) and Axel Neumann - The projects leaders of the OWASP Zed Attack Proxy (ZAP) project - for providing support, useful advice and adjustments that made the creation of Diviner possible.

Liran Sheinbox (Developer) - Diviner's Payload Manager (alpha).

Alex Mor, Oren Ofer and Michal Goldstein (Developers) - for their contribution to the development of Diviner's content differentiation analysis features (alpha).

Alex Ganelis, Tsachi Itschak and Lior Suliman (Developers) - Diviner Installer, ZAP Integration and various modifications.

Zafrir Grosman - material design.

The Flying Saucer Draught Emporium Bar at Houston, TX - for whatever substance that triggered the inspiration.

The 2012 Web Application Scanner Benchmark

2012-07-13T09:24:00.000-07:00

Top 10:

The Web Application Vulnerability Scanners Benchmark, 2012

Commercial & Open Source Scanners

An Accuracy, Coverage, Versatility, Adaptability, Feature and Price Comparison of 60 Commercial & Open Source Black Box Web Application Vulnerability Scanners

By Shay Chen

Information Security Consultant, Researcher and Instructor

http://sectooladdict.blogspot.com/, http://www.sectoolmarket.com/

sectooladdict-$at$-gmail-$dot$-com

July 2012

Assessment Environments: WAVSEP 1.2, ZAP-WAVE (WAVSEP integration), WIVET v3-rev148

Table of Contents

1. Introduction

2. List of Tested Web Application Scanners

3. Benchmark Overview & Assessment Criteria

4. A Glimpse at the Results of the Benchmark

5. Test I - Scanner Versatility - Input Vector Support

6. Test II – Attack Vector Support – Counting Audit Features

7. Introduction to the Various Accuracy Assessments

8. Test III – The Detection Accuracy of Reflected XSS

9. Test IV – The Detection Accuracy of SQL Injection

10. Test V – The Detection Accuracy of Path Traversal/LFI

11. Test VI – The Detection Accuracy of RFI (XSS via RFI)

12. Test VII - WIVET - Coverage via Automated Crawling

13. Test VIII – Scanner Adaptability - Crawling & Scan Barriers

14. Test IX – Authentication and Usability Feature Comparison

15. Test X – The Crown Jewel - Results & Features vs. Pricing

16. Additional Comparisons, Built-in Products and Licenses

17. What Changed?

18. Initial Conclusions – Open Source vs. Commercial

19. Verifying The Benchmark Results

20. So What Now?

21. Recommended Reading List: Scanner Benchmarks

22. Thank-You Note

23. FAQ - Why Didn't You Test NTO, Cenzic and N-Stalker?

24. Appendix A – List of Tools Not Included In the Test

1. Introduction

Detailed Result Presentation at

http://www.sectoolmarket.com/

Tools, Features, Results, Statistics and Price Comparison
(Delete Cache)

A Step by Step Guide for Choosing the Right Web Application Vulnerability Scanner for *You*

infosec-island

A Perfectionist Guide for Optimal Use of Web Application Vulnerability Scanners

[Placeholder]

Getting the information was the easy part. All I had to do was to invest a couple of years in gathering the list of tools, and a couple of more in documenting their various features. It's really a daily routine - you read a couple of posts in news groups in the morning, and couple blogs at the evening. Once you get used to it, it's fun, and even quite addictive.

Then came the "best" fantasy, and with it, the inclination to test the proclaimed features of all the web application vulnerability scanners against each other, only to find out that things are not that simple, and finding the "best", if there is such a tool, was not an easy task.

Inevitably, I tried searching for alternative assessment models, methods of measurements that will handle the imperfections of the previous assessments.

I tried to change the perspective, add tests (and hundreds of those - 940+, to be exact), examine different aspects, and even make parts of the test process obscure, and now, I'm finally ready for another shot.

In spite of everything I had invested in past researches, due to the focus I had on features and accuracy, and the policy I used when interacting with the various vendors, it was difficult, especially for me, to gain insights from the mass amounts of data that will enable me to choose, and more importantly, properly use the various tools in real life scenarios.

Is the most accurate scanner necessarily the best choice for a point and shoot scenario? and what good will it do if it can't scan an application due to a specific scan barrier it can't handle, or because if does not support the input delivery method?

I needed to gather other pieces of the puzzle, and even more importantly, I needed a method, or more accurately, a methodology.

I'm sorry to disappoint you, dear reader, so early in the article, but I still don't have a perfect answer or one recommendation... But I sure am much closer than I ever was, and although I might not have the answer, I have many answers, and a very comprehensive, logical and clear methodology for employing the use of all the information I'm about to present.

In the previous benchmarks , I focused on assessing 3 major aspects of web application scanners, which revolved mostly around features & accuracy, and even though the information was very interesting, it wasn't necessarily useful, at least not in all scenarios.

So decided to take it to the edge, but since I already reached the number of 60 scanners, it was hard to make an impression with a couple of extra tools, so instead, I focused my efforts on aspects.

This time, I compared 10 different aspects of the tools (or 14, if you consider non competitive charts), and chose the collection with the aim of providing practical tools for making a decision, and getting a glimpse of the bigger picture.

Let me assure you - this time, the information is presented in a manner that is very helpful, is easy to navigate, and is supported by presentation platforms, articles and step by step methodologies.

Furthermore, I wrapped it all in a summary that includes the major results and features in relation to the price, for those of us that prefer the overview, and avoid the drill down. Information and Insights that I believe, will help testers invest their time in better-suited tools, and consumers in properly investing their money, in the long term or the short term (but not necessarily both*).

As mentioned earlier, this research covers various aspects for the latest versions of 11 commercial web application scanners, and the latest versions of most of the 49 free & open source web application scanners. It also covers some scanners that were not covered in previous benchmarks, and includes, among others, the following components and tests:

• A Price Comparison - in Relation to the Rest of the Benchmark Results

• Scanner Versatility - A Measure for the Scanner's Support of Protocols & Input Delivery Vectors

• Attack Vector Support - The Amount & Type of Active Scan Plugins (Vulnerability Detection)

• Reflected Cross Site Scripting Detection Accuracy

• SQL Injection Detection Accuracy

• Path Traversal / Local File Inclusion Detection Accuracy

• Remote File Inclusion Detection Accuracy (XSS/Phishing via RFI)

• WIVET Score Comparison - Automated Crawling / Input Vector Extraction

• Scanner Adaptability - Complementary Coverage Features and Scan Barrier Support

• Authentication Features Comparison

• Complementary Scan Features and Embedded Products

• General Scanning Features and Overall Impression

• License Comparison and General Information

And just before we delve into the details, one last tip: don't focus solely on the charts - if you want to really understand what they reflect, dig in.

Lists and charts first, detailed description later.

2. List of Tested Web Application Scanners

The following commercial scanners were included in the benchmark:

IBM AppScan v8.5.0.1, Build 42-SR1434 (IBM)
WebInspect v9.20.277.0, SecureBase 4.08.00 (HP)
Netsparker v2.1.0, Build 45 (Mavituna Security)
Acunetix WVS v8.0, Build 20120613 (Acunetix)
Syhunt Dynamic (SandcatPro) v4.5.0.0/1 (Syhunt)
Burp Suite v1.4.10 (Portswigger)
ParosPro v1.9.12 (Milescan) - WIVET / Other
JSky v3.5.1-905 (NoSec) - WIVET / Other
WebCruiser v2.5.1 EE (Janus Security)
Nessus v5.0.1 - 20120701 (Tenable Network Security) - Web Scanning Features
Ammonite v1.2 (RyscCorp)

The following new free & open source scanners were included in the benchmark:

IronWASP v0.9.1.0

The updated versions of the following free & open source scanners were re-tested in the benchmark:

Zed Attack Proxy (ZAP) v1.4.0.1, sqlmap v1.0-Jul-5-2012 (Github), W3AF 1.2-rev509 (SVN), Acunetix Free Edition v8.0-20120509, Safe3WVS v10.1 FE (Safe3 Network Center) WebSecurify v0.9 (free edition - the new commercial version was not tested), Syhunt Mini (Sandcat Mini) v4.4.3.0, arachni v0.4.0.3, Skipfish 2.07b, N-Stalker 2012 Free Edition v7.1.1.121 (N-Stalker), Watobo v0.9.8-rev724 (a few new WATOBO 0.9.9 pre versions were released a few days before the publication of the benchmark, but I didn't managed to test them in time)

Different aspects of the following free & open source scanners were tested in the benchmark:

VEGA 1.0 beta (Subgraph), Netsparker Community Edition v1.7.2.13, Andiparos v1.0.6, ProxyStrike v2.2, Wapiti v2.2.1, Paros Proxy v3.2.13, Grendel Scan v1.0

The results were compared to those of unmaintained scanners tested in previous benchmarks:

PowerFuzzer v1.0, Oedipus v1.8.1 (v1.8.3 is around somewhere), Scrawler v1.0, WebCruiser v2.4.2 FE (corrections), Sandcat Free Edition v4.0.0.1, JSKY Free Edition v1.0.0, N-Stalker 2009 Free Edition v7.0.0.223, UWSS (Uber Web Security Scanner) v0.0.2, Grabber v0.1, WebScarab v20100820, Mini MySqlat0r v0.5, WSTool v0.14001, crawlfish v0.92, Gamja v1.6, iScan v0.1, LoverBoy v1.0, DSSS (Damn Simple SQLi Scanner) v0.1h, openAcunetix v0.1, ScreamingCSS v1.02, Secubat v0.5, SQID (SQL Injection Digger) v0.3, SQLiX v1.0, VulnDetector v0.0.2, Web Injection Scanner (WIS) v0.4, Xcobra v0.2, XSSploit v0.5, XSSS v0.40, Priamos v1.0, XSSer v1.5-1 (version 1.6 was released but I didn't manage to test it), aidSQL 02062011 (a newer revision exists in the SVN but was not officially released)

For a full list of commercial & open source tools that were not tested in this benchmark, refer to the appendix.

3. Benchmark Overview & Assessment Criteria

· The ability to detect Reflected XSS and/or SQL Injection and/or Path Traversal/Local File Inclusion/Remote File Inclusion vulnerabilities.

· The ability to scan multiple URLs at once (using either a crawler/spider feature, URL/Log file parsing feature or a built-in proxy).

· The ability to control and limit the scan to internal or external host (domain/IP).

The testing procedure of all the tools included the following phases:

Feature Documentation

Accuracy Assessment

The scanners were all tested against the latest version of WAVSEP (v1.2, integrating ZAP-WAVE), a benchmarking platform designed to assess the detection accuracy of web application scanners, which was released with the publication of this benchmark. The purpose of WAVSEP’s test cases is to provide a scale for understanding which detection barriers each scanning tool can bypass, and which common vulnerability variations can be detected by each tool.

· The various scanners were tested against the following test cases (GET and POST attack vectors):

o 816 test cases that were vulnerable to Path Traversal attacks.

o 108 test cases that were vulnerable to Remote File Inclusion (XSS via RFI) attacks.

o 66 test cases that were vulnerable to Reflected Cross Site Scripting attacks.

o 80 test cases that contained Error Disclosing SQL Injection exposures.

o 46 test cases that contained Blind SQL Injection exposures.

o 10 test cases that were vulnerable to Time Based SQL Injection attacks.

o 7 different categories of false positive RXSS vulnerabilities.

o 10 different categories of false positive SQLi vulnerabilities.

o 8 different categories of false positive Path Travesal / LFI vulnerabilities.

o 6 different categories of false positive Remote File Inclusion vulnerabilities.

· The benchmark included 8 experimental RXSS test cases and 2 experimental SQL Injection test cases, and although the scan results of these test cases were documented in the various scans, their results were not included in the final score, at least for now.

· In order to ensure the result consistency, the directory of each exposure sub category was individually scanned multiple times using various configurations, usually using a single thread and using a scan policy that only included the relevant plugins.

In order to ensure that the detection features of each scanner were truly effective, most of the scanners were tested against an additional benchmarking application that was prone to the same vulnerable test cases as the WAVSEP platform, but had a different design, slightly different behavior and different entry point format, in order to verify that no signatures were used, and that any improvement was due to the enhancement of the scanner's attack tree.

Attack Surface Coverage Assessment

This section of the benchmark also included the WIVET test (Web Input Vector Extractor Teaser), in which scanners were executed against a dedicated application that can assess their crawling mechanism in the aspect of input vector extraction. The specific details of this assessment are provided in the relevant section.

Public tests vs. Obscure tests

· Publically announced tests: the active scan feature comparison, and the detection accuracy assessment of the SQL Injection and Reflected Cross Site Scripting, composed out of tests cases which were published as a part of WAVSEP v1.1.1)

· Tests that were obscure to all vendors until the moment of the publication: the various new groups of feature comparisons, the WIVET assessment, and the detection accuracy assessment of the Path Traversal / LFI and Remote File Inclusion (XSS via RFI), implemented as 940+ test cases in WAVSEP 1.2 (a new version that was only published alongside this benchmark).

The results of the main test categories are presented within three graphs (commercial graph, free & open source graph, unified graph), and the detailed information of each test is presented in a dedicated section in benchmark presentation platform at http://www.sectoolmarket.com.

Now that were finally done with the formality, let's get to the interesting part... the results.

4. A Glimpse to the Results of the Benchmark

This presentation of results in this benchmark, alongside the dedicated website (http://www.sectoolmarket.com/) and a series of supporting articles and methodologies ([placeholder]), are all designed to help the reader to make a decision - to choose the proper product/s or tool/s for the task at hand, within the borders of the time or budget.

For those of us that can't wait, and want to get a glimpse to the summary of the unified results, there is a dedicated page available at the following links:

Price & Feature Comparison of Commercial Scanners
http://sectoolmarket.com/price-and-feature-comparison-of-web-application-scanners-commercial-list.html
Price & Feature Comparison of a Unified List of Commercial, Free and Open Source Products

http://sectoolmarket.com/price-and-feature-comparison-of-web-application-scanners-unified-list.html

Some of the sections might not be clear to some of the readers at this phase, which is why I advise you to read the rest of the article, prior to analyzing this summary.

5. Test I - Scanner Versatility - Input Vector Support

The first assessment criterion was the number of input vectors each tool can scan (and not just parse).

Modern web applications use a variety of sub-protocols and methods for delivering complex inputs from the browser to the server. These methods include standard input delivery methods such as HTTP querystring parameters and HTTP body parameters, modern delivery methods such as JSON and XML, and even binary delivery methods for technology specific objects such as AMF, Java serialized objects and WCF.

Since the vast majority of active scan plugins rely on input that is meant to be injected into client originating parameters, supporting the parameter (or rather, the input) delivery method of the tested application is a necessity.

Although the charts in this section don't necessarily represent the most important score, it is the most important perquisite for the scanner to comply with when scanning a specific technology.

Reasoning: An automated tool can't detect a vulnerability in a given parameter, if it can't scan the protocol or mimic the application's method of delivering the input. The more vectors of input delivery that the scanner supports, the more versatile it is in scanning different technologies and applications (assuming it can handle the relevant scan barriers, supports necessary features such as authentication, or alternatively, contains features that can be used to work around the specific limitations).

The detailed comparison of the scanners support for various input delivery methods is documented in detail in the following section of sectoolmarket (recommended - too many scanners in the chart):

http://www.sectoolmarket.com/input-vector-support-unified-list.html

The following chart shows how versatile each scanner is in scanning different input delivery vectors (and although not entirely accurate - different technologies):

The Number of Input Vectors Supported – Commercial Tools

The Number of Input Vectors Supported – Free & Open Source Tools

The Number of Input Vectors Supported – Unified List

6. Test II – Attack Vector Support – Counting Audit Features

The second assessment criterion was the number of audit features each tool supports.

Reasoning: An automated tool can't detect an exposure that it can't recognize (at least not directly, and not without manual analysis), and therefore, the number of audit features will affect the amount of exposures that the tool will be able to detect (assuming the audit features are implemented properly, that vulnerable entry points will be detected, that the tool will be able to handle the relevant scan barriers and scanning perquisites, and that the tool will manage to scan the vulnerable input vectors).

The definition of the assessment criterion rules out product specific exposures and infrastructure related vulnerabilities, while unique and extremely rare features were documented and presented in a different section of this research, and were not taken into account when calculating the results. Exposures that were specific to Flash/Applet/Silverlight and Web Services Assessment (with the exception of XXE) were treated in the same manner.

The detailed comparison of the scanners support for various audit features is documented in detail in the following section of sectoolmarket:

http://sectoolmarket.com/audit-features-comparison-unified-list.html

The Number of Audit Features in Web Application Scanners – Commercial Tools

The Number of Audit Features in Web Application Scanners – Free & Open Source Tools

The Number of Audit Features in Web Application Scanners – Unified List

So once again, now that were done with the quantity, let's get to the quality…

7. Introduction to the Various Accuracy Assessments

The following sections presents the results of the detection accuracy assessments performed for Reflected XSS, SQL Injection, Path Traversal and Remote File Inclusion (RXSS via RFI) - four of the most commonly supported features in web application scanners. Although the detection accuracy of a specific exposure might not reflect the overall condition of the scanner on its own, it is a crucial indicator for how good a scanner is at detecting specific vulnerability instances.

The various assessments were performed against the various test cases of WAVSEP v1.2, which emulate different common test case scenarios for generic technologies.

Reasoning: a scanner that is not accurate enough will miss many exposures, and might classify non-vulnerable entry points as vulnerable. These tests aim to assess how good is each tool at detecting the vulnerabilities it claims to support, in a supported input vector, which is located in a known entry point, without any restrictions that can prevent the tool from operating properly.

8. Test III – The Detection Accuracy of Reflected XSS

The third assessment criterion was the detection accuracy of Reflected Cross Site Scripting, a common exposure which is the 2nd most commonly implemented feature in web application scanners, and the one in which I noticed the greatest improvement in the various tested web application scanners.

The comparison of the scanners' reflected cross site scripting detection accuracy is documented in detail in the following section of sectoolmarket:

http://sectoolmarket.com/reflected-cross-site-scripting-detection-accuracy-unified-list.html

Result Chart Glossary

The Reflected XSS Detection Accuracy of Web Application Scanners – Commercial Tools

The Reflected XSS Detection Accuracy of Web Application Scanners – Open Source & Free Tools

The Reflected XSS Detection Accuracy of Web Application Scanners – Unified List

9. Test IV – The Detection Accuracy of SQL Injection

The fourth assessment criterion was the detection accuracy of SQL Injection, one of the most famous exposures and the most commonly implemented attack vector in web application scanners.

The evaluation was performed on an application that uses MySQL 5.5.x as its data repository, and thus, will reflect the detection accuracy of the tool when scanning an application that uses similar data repositories.

The comparison of the scanners' SQL injection detection accuracy is documented in detail in the following section of sectoolmarket:

http://sectoolmarket.com/sql-injection-detection-accuracy-unified-list.html

Result Chart Glossary

The SQL Injection Detection Accuracy of Web Application Scanners – Commercial Tools

The SQL Injection Detection Accuracy of Web Application Scanners – Open Source & Free Tools

The SQL Injection Detection Accuracy of Web Application Scanners – Unified List

Although there are many changes in the results since the last benchmark, both of these exposures (SQLi, RXSS) were previously assessed, so, I believe it's time to introduce something new... something none of the tested vendors could have prepared for in advance...

10. Test V – The Detection Accuracy of Path Traversal/LFI

The fifth assessment criterion was the detection accuracy of Path Traversal (a.k.a Directory Traversal), a newly implemented feature in WAVSEP v1.2, and the third most commonly implemented attack vector in web application scanners.

The reason it was tagged along with Local File Inclusion (LFI) is simple - many scanners don't make the differentiation between inclusion and traversal, and furthermore, a few online vulnerability documentation sources don't. In addition, the results obtained from the tests performed on the vast majority of tools lead to the same conclusion - many plugins listed under the name LFI detected the path traversal plugins.

While implementing the path traversal test cases and consuming nearly every relevant piece of documentation I could find on the subject, I decided to take the current path, in spite of some acute differences some of the documentation sources suggested (but did implemented an infrastructure in WAVSEP for "true" inclusion exposures).

The evaluation was performed on a WAVSEP v1.2 instance that was hosted on windows XP, and although there are specific test cases meant to emulate servers that are running with a low privileged OS user accounts (using the servlet context file access method), many of the test cases emulate web servers that are running with administrative user accounts.

Although I didn't perform the path traversal scans on Linux for all the tools, I did perform the initial experiments on Linux, and even a couple of verifications on Linux for some of the scanners, and as weird as it sounds, I can clearly state that the results were significantly worse, and although I won't get the opportunity to discuss the subject in this benchmark, I might handle it in the next.

In order to assess the detection accuracy of different path traversal instances, I designed a total of 816 OS-adapting path traversal test cases (meaning - the test cases adapt themselves to the OS they are executed in, and to the server they are executed in, in the aspects of file access delimiters and file access paths). I know it might seem a lot, and I guess I did got carried away with the perfectionism, but you will be surprised too see that these tests really represent common vulnerability instances, and not necessarily super extreme scenarios, and that results of the tests did prove the necessity.

The tests were deigned to emulate various combination of the following conditions and restrictions:

If you will take a closer look at the detailed scan-specific results at www.sectoolmarket.com, you'll notice that some scanners were completely unaffected by the response content type and HTTP code variation, while other scanners were dramatically affected by the variety (gee, it's nice to know that I didn't write them all for nothing... :) ).

In reality, there were supposed to more test cases, primarily because I intended to test injection entry points in which the input only affected the filename without the extension, or was injected directly into the directory name. However, due to the sheer amount of tests and the deadline I had for this benchmark, I decided to delete (literally) the test cases that handled these anomalies, and focus on test cases in which the entire filename/path was affected. That being said, I might publish these test cases in future versions of wavsep (they amount to a couple of hundreds).

The comparison of the scanners' path traversal detection accuracy is documented in detail in the following section of sectoolmarket:

http://sectoolmarket.com/path-traversal-local-file-inclusion-detection-accuracy-unified-list.html

Result Chart Glossary

The Path Traversal / LFI Detection Accuracy of Web Application Scanners – Commercial Tools

The Path Traversal / LFI Detection Accuracy of Web Application Scanners – Open Source & Free Tools

The Path Traversal / LFI Detection Accuracy of Web Application Scanners – Unified List

And what of LFI's evil counterpart, Remote File Inclusion?

(yeah yeah, I know, it was path traversal...)

11. Test VI – The Detection Accuracy of RFI (XSS via RFI)

The sixth assessment criterion was the detection accuracy of Remote File Inclusion (or more accurately, vectors of RFI that can result in XSS or Phishing - and currently, not necessarily in server code execution), a newly implemented feature in WAVSEP v1.2, and the one of most commonly implemented attack vector in web application scanners.

I didn't originally plan to assess the detection accuracy of RFI in this benchmark, however, since I implemented a new structure to wavsep that enables me to write a lot of test cases faster, I couldn't resist the urge to try it... and thus, found a new way to decrease the amount of sleep I get each night.

The interesting thing I found was that although RFI is supposed to work a bit differently than LFI/Path traversal, many LFI/Path traversal Plugins effectively detected RFI exposures, and in some instances, the tests for both of these vulnerabilities were actually implemented in the same plugin (usually named "file inclusions"); thus, while scanning for Traversal/LFI/RFI, I usually activated all the relevant plugins in the scanner, and low and behold - got results from the LFI/Path Traversal plugins that even the RFI dedicated plugins did not detect.

In order to assess the detection accuracy of different remote file inclusion exposures (again, RXSS/Phishing via RFI vectors), I designed a total of 108 remote file inclusion test cases.

The tests were deigned to emulate various combination of the following conditions and restrictions:

Just like the case of path traversal, In reality, there were supposed to be more XSS via RFI test cases, primarily because I intended to test injection entry points in which the input only affected the filename without the extension, or was injected directly into the directory name. However, due to the sheer amount of tests and the deadline I had for this benchmark, I decided to delete (literally) the test cases that handled these anomalies, and focus on test cases in which the entire filename/path was affected. That being said, I might publish these test cases in future versions of wavsep (they amount to dozens).

[Note: Although the tested versions of Appscan and Nessus contain RFI detection plugins, they did not support the detection of XSS via RFI.]

The comparison of the scanners' remote file inclusion detection accuracy is documented in detail in the following section of sectoolmarket:

http://sectoolmarket.com/remote-file-inclusion-detection-accuracy-unified-list.html

Result Chart Glossary

The RFI (XSS via RFI) Detection Accuracy of Web Application Scanners – Commercial Tools

The RFI (XSS via RFI) Detection Accuracy of Web Application Scanners – Open Source & Free Tools

The RFI (XSS via RFI) Detection Accuracy of Web Application Scanners – Unified List

And after covering all those accuracy aspects, it's time to cover a totally different subject - Coverge.

12. Test VII - WIVET - Coverage via Automated Crawling

The seventh assessment criterion was the scanner's WIVET score, which is related to coverage.

The concept of coverage can mean a lot of things, but in general, what I'm referring to is the ability of the scanner to increase the attack surface of the tested application - to locate additional resources and input delivery methods to attack.

Although a scanner can increase the attack surface in a number of ways, from detecting hidden files to exposing device-specific interfaces, this section of the benchmark focuses on automated crawling and an efficient input vector extraction.

In order to evaluate these aspects in scanners, I used a wonderful OWASP turkey project called WIVET (Web Input Vector Extractor Teaser); The WIVET project is a benchmarking project that was written by an application security specialist by the name of Bedirhan Urgun, and released under the GPL2 license.

The project is implemented as a web application which aims to "statistically analyze web link extractors", by measuring the amount of input vectors extracted by each scanner while crawling the WIVET website, in order to assess how well each scanner can increase the coverage of the attack surface.

Plainly speaking, the project simply measures how well a scanner is able to crawl the application, and how well can it locate input vectors, by presenting a collection of challenges that contain links, parameters and input delivery methods that the crawling process should locate and extract.

Although WIVET used to have an online instance, with my luck, by the time I decided to use it the online version was already gone... so I checked-out the latest subversion revision from the project's google code website (v3-revision148), installed FastCGI on an IIS server (Windows XP), copied the application files to a directory called wivet under the C:\Inetpub\wwwroot\ directory, and started the IIS default website.

In order for WIVET to work, the scanner must crawl the application while consistently using the same session identifier in its crawling requests, while avoiding the 100.php logout page (which initializes the session, and thus the results). The results can then be viewed by accessing the application index page, while using the session identifier used during the scan.

A very nice idea that makes the assessment process easy and effective, however, for me, things weren't that easy. Although some scanners did work properly with the platform, many scanners did not receive any score, even though I configured them exactly according to the recommendations (valid session identifier and logout URL exclusion), so after a careful examination, I discovered the source of my problem: some of the scanners don't send the predefined session identifier in their crawling requests (even though it's explicitly defined in the product), and others simply ignore URL exclusions (in certain conditions).

Since even without these bugs, not all the scanners supported URL exclusions (100.php logout page) and predefined cookies, I had to come up with a solution that will enable me to test all of them... so I changed the WIVET platform a little bit by deleting the link to the logout page (100.php) from the main menu page (menu.php), forwarded the communication of the vast majority of scanners through a fiddler instance, in which I defined a valid WIVET session identifier (using the filter features), and in extreme scenarios in which an upstream proxy was not supported by the scanner, defined the WIVET website as a proxy in an IE browser, loaded fiddler (so it will forward the communication to the system defined proxy - WIVET), defined burp as a transparent proxy that forwards the communication to fiddler (upstream proxy), and scanned burp instead of the WIVET application (the scanner will scan burp which will forward the communication to fiddler which will forward the communication to the system defined proxy - the WIVET website).

These solutions seemed to be working for most vendors, that is until I discovered two more bugs that caused these solutions not to work for another small group of products...

The first bug was related to the emulation of modern browser behavior when interpreting the relative context of links in a frameset (browsers use the link's target frame as the path basis, but some scanners used the path basis of the links origin page), and the other bug was related to another browser emulation issue - some scanners that did not manage to submit forms without an action property (while a browser usually submits such a form to the same URL that form originated from).

I managed to solve the first bug by editing the menu page and manually adding additional links with an alternate context (added "pages/" to all URLs) to the same WIVET pages , while the second bug was reported to some vendors (and was handled by them).

Finally, some of the scanners had bugs that I did not manage to isolate in the given timeframe, and thus, I didn't manage to get any WIVET score for them (a list of these products will presented at the end of this section).

However, the vast majority of the scanners did got a score, which can be viewed in the following charts and links.

The comparison of the scanners' WIVET score is documented in detail in the following section of sectoolmarket:

http://sectoolmarket.com/wivet-score-unified-list.html

The WIVET Score of Web Application Scanners – Commercial Tools

The WIVET Score of Web Application Scanners – Free and Open Source Tools

The WIVET Score of Web Application Scanners – Unified List

It is important to clarify that due to these scanner bugs (and the current WIVET structure) - low scores and non-existing scores might differ once minor bugs are fixed, but the scores presented in this chart are currently all I can offer.

The following scanners didn't manage to get a WIVET score at all (even after all the adjustments and enhancements I tried), and although this does not mean that their score is necessarily low, or that there isn't any possible way to execute them in-front of WIVET, simply that there isn't a simple method of doing it (at least not one that I discovered):

Syhunt Mini (Sandcat Mini), Webcruiser, IronWASP, Safe3WVS free edition, N-Stalker 2012 free edition, Vega, Skipfish.

In addition, I didn't try scanning WIVET with various unmaintained scanners, scanners that didn't have a spider feature (WATOBO in the assessed version, Ammonite, etc), or with the following assessed tools: Nessus, sqlmap.

It's crucial to note that scanners with burp-log parsing features (such sqlmap and IronWASP) can effectively be assigned with the WIVET score of burp, that scanners with internal proxy features (such as ZAP, Burpsuite, Vega, etc) can be used with the crawling mechanisms of other scanners (such as Acunetix FE), and that as a result of both of these conclusions, any scanner that supports any of those features can be assigned the WIVET score of any scanner in the possession of the tester (by using the crawling mechanism of a scanner through a proxy such as burp, in order to generate scan logs).

13. Test VIII – Scanner Adaptability - Crawling & Scan Barriers

By using the seemingly irrelevant term "adaptability" in relation to scanners, I'm actually referring to the scanner's ability to adapt and scan the application, despite different technologies, abnormal crawling requirements and varying scan barriers, such as Anti-CSRF tokens, CAPTCHA mechanisms, platform specific tokens (such as required viewstate values) or account lock mechanisms.

Although not necessarily a measurable quality, the ability of the scanner to handle different technologies and scan barriers is an important perquisite, and in a sense, almost as important as being able to scan the input delivery method.

Reasoning: An automated tool can't detect a vulnerability in a point and shoot scenario if it is can't locate & scan the vulnerable location due to the lack of support in a certain a browser add-on, the lack of support for extracting data from certain non-standard vectors, or the lack of support in overcoming a specific barrier, such as a required token or challenge. The more barriers the scanner is able to handle, the more useful it is when scanning complex applications that employ the use of various technologies and scan barriers (assuming it can handle the relevant input vectors, supports the necessary features such as authentication, or has a feature that can be used to work around the specific limitations).

The following charts shows how many types of barriers does each scanner claim to be able to handle (these features were not verified, and the information currently relies on documentation or vendor supplied information):

The Adaptability Score of Web Application Scanners – Commercial Tools

The Adaptability Score of Web Application Scanners – Free and Open Source Tools

The Adaptability Score of Web Application Scanners – Unified List

The detailed comparison of the scanners support for various barriers is documented in detail in the following of sectoolmarket:

http://sectoolmarket.com/coverage-features-comparison-unified-list.html

14. Test IX – Authentication and Usability Feature Comparison

Although supporting the authentication required by the application seems like a crucial quality, in reality, certain scanner chaining features can make-up for the lack of support in certain authentication methods, by employing the use of a 3rd party proxy to authenticate on the scanner's behalf.

For example, if we wanted to use a scanner that does not support NTLM authentication (but does support an upstream proxy), we could have defined the relevant credentials in burpsuite FE, and define it as an upstream proxy for the tested scanner.

However, chaining the scanner to an external tool that supports the authentication still has some disadvantages, such as potential stability issues, thread limitation and inconvenience.

The following comparison table shows which authentication methods and features are supported by the various assessed scanners:

http://sectoolmarket.com/authentication-features-comparison-unified-list.html

15. Test X – The Crown Jewel - Results & Features vs. Pricing

Finally, after reading through all the sections and charts, and analyzing the different aspects in which each scanner was measured, it's time to expose the price (at least for those of you that did manage to resist the temptation to access this link at the beginning).

The important thing to notice, specifically in relation to commercial scanner pricing, is that each product might be a bundle of several semi-independent products that cover different aspects of the assessment process, which are not necessarily related to the web application security. These products currently include web service scanners, flash application scanners and CGI scanners (SAST and IAST features were not included on purpose).

In short, the scanner price might reflect (or not) a set of products that might have been priced separately as an independent product.

Another issue to pay attention to is the type of license acquired. In general, I did not cover non commercial prices in this comparison, and in addition, did not include any vendor specific bundles, sales, discounts and sales pitches. I presented the base prices listed in the vendor website or provided to me by the vendor, according to a total of 6 predefined categories, which are in fact, combinations of the following concepts:

Website/Year - a license to install the software on a single station and use it for a single year against a single IP address (the exception to this rule is Netsparker, in which the per website price reflects 3 Websites).

Seat/Year - a license to install the software on a single station and use it for a single year.

The various prices can be viewed in the dedicated comparison in sectoolmarket, available in the following address:

http://www.sectoolmarket.com/price-and-feature-comparison-of-web-application-scanners-unified-list.html

It is important to remember that this prices might change, vary or be affected by numerous variables, from special discounts and sales to a strategic conscious decision of a vendors to invest in you as a customer or a beta testing site.

16. Additional Comparisons, Built-in Products and Licenses

While in the past I used to present additional information in external PDF files, with the new presentation platform I am now able to present the information in a media that is much easier to use and analyze. Although anyone can access the root URL of sectoolmarket and search the various sections on his own, I decided to provide a short summary of additional lists and features that were not covered in a dedicated section of this benchmark, but were still documented and published in sectoolmarket.

List of Tools

The list of tools tested in this benchmark, and in the previous benchmarks, can be accessed through the following link:

· List of Tested Scanners and Their Licenses, Notes, Vendor, Source Repository and Latest Update

Additional Features

Complementary scan features that were not evaluated or included in the benchmark:

· Complementary Scan Features

· General Scanner Features

In order to clarify what each column in the report table means, use the following glossary table:

Title	Possible Values
Configuration & Usage Scale	Very Simple - GUI + Wizard Simple - GUI with simple options, Command line with scan configuration file or simple options Complex - GUI with numerous options, Command line with multiple options Very Complex - Manual scanning feature dependencies, multiple configuration requirements
Stability Scale	Very Stable - Rarely crashes, Never gets stuck Stable - Rarely crashes, Gets stuck only in extreme scenarios Unstable - Crashes every once in a while, Freezes on a consistent basis Fragile – Freezes or Crashes on a consistent basis, Fails performing the operation in many cases
Performance Scale	Very Fast - Fast implementation with limited amount of scanning tasks Fast - Fast implementation with plenty of scanning tasks Slow - Slow implementation with limited amount of scanning tasks Very Slow - Slow implementation with plenty of scanning tasks

Scan Logs

· http://sectoolmarket.com/

17. What Changed?

Since the latest benchmark, many open source & commercial tools added new features and improved their detection accuracy.

The following list presents a summary of changes in the detection accuracy of commercial tools that were tested in the previous benchmark (+new):

· IBM AppScan - no significant changes, new results for Path Traversal and WIVET.

· WebInspect - a dramatic improvement in the detection accuracy of SQLi and XSS (fantastic result!), new results for Path Traversal, RFI (fantastic result!), and WIVET (fantastic result!)

· Netsparker - no significant changes, new results for Path Traversal and WIVET.

· Acunetix WVS - a dramatic improvement in the detection accuracy of SQLi (fantastic result!) and XSS (fantastic result!), and new results for Path Traversal, RFI and WIVET.

· Syhunt Dynamic - a dramatic improvement in the detection accuracy of XSS (fantastic result!) and SQLi, and new results for Path Traversal, RFI and WIVET.

· Burp Suite - a dramatic improvement in the detection accuracy of XSS and SQLi (fantastic result!), and new results for Path Traversal and WIVET.

· ParosPro - New results for Path Traversal and WIVET.

· JSky - New results for RFI, Path Traversal and WIVET.

· WebCruiser - No significant changes.

· Nessus - a dramatic improvement in the detection accuracy of Reflected XSS, potential bug in the LFI/RFI detection features.

· Ammonite - New results for RXSS, SQLi, RFI and Path Traversal (fantastic result!)

The following list presents a summary of changes in the detection accuracy of free and open source tools that were tested in the previous benchmark (+new):

· Zed Attack Proxy (ZAP) – a dramatic improvement in the detection accuracy of Reflected XSS exposures (fantastic result!), in addition to new results for Path Traversal and WIVET.

· IronWASP - New results for SQLi, XSS, Path Traversal and RFI (fantastic result!).

· arachni – an improvement in the detection accuracy of Reflected XSS exposures (mainly due to the elimination of false positives), but a decrease in the accuracy of SQL injection exposures (due to additional false positives being discovered). There's also new results for RFI, Path Traversal (incomplete due to a bug), and WIVET.

· sqlmap – a dramatic improvement in the detection accuracy of SQL Injection exposures (fantastic result!).

· Acunetix Free Edition – a dramatic improvement in the detection accuracy of Reflected XSS exposures, in addition to a new WIVET result.

· Syhunt Mini (Sandcat Mini) - a dramatic improvement in the detection accuracy of both XSS (fantastic result!) and SQLi. New results for RFI.

· Watobo – Identical results, in addition to new results for Path Traversal and WIVET. The author did not test the latest Watobo version, which was released a few days before the publication of this benchmark.

· N-Stalker 2012 FE – no significant changes, although it seems that the decreased accuracy is actually an unhandled bug in the release (unverified theory).

· Skipfish – insignificant changes that probably result from the testing methodology and/or testing environment. New results for Path Traversal, RFI and WIVET.

· WebSecurify – a major improvement in the detection accuracy of RXSS exposures, and new results for Path Traversal and WIVET.

· W3AF – a slight increase in the SQL Injection detection accuracy. New results for Path Traversal (fantastic result!), RFI and WIVET.

· Netsparker Community Edition – New results for WIVET.

· Andiparos & Paros – New results for WIVET.

· Wapiti – New results for Path Traversal, RFI and WIVET.

· ProxyStrike – New results for WIVET (Fantastic results for an open source product! again!)

· Vega - New results for Path Traversal, RFI and WIVET.

· Grendel Scan – New results for WIVET.

18. Initial Conclusions – Open Source vs. Commercial

The following section presents my own personal opinions on the results, and is not based purely on accurate statistics, like the rest of the benchmark.

After testing various versions of over 51 open source scanners on multiple occasions, and after comparing the results and experiences to the ones I had after testing 15 commercial ones (including tools tested in the previous benchmarks and tools I did not reported), I have reached the following conclusions:

· As far as accuracy & features, the distance between open source tools and commercial tools is insignificant, and open source already rival, and in some rare cases, even exceed the capabilities of commercial scanners (and vice versa).

· Although most open source scanners have not yet adjusted to support applications that use new technologies (AJAX, JSON, etc), recent advancement in the crawler of ZAP proxy (not tested in the benchmark, and might be reused by other projects), and the input vectors supported by a new project named IronWASP are a great beginning to the process. On the other hand, most of the commercial vendors already adjusted themselves to some of the new technologies, and can be used to scan them in a variety of models.

· The automated crawling capability of most commercial scanners is significantly better than those of open source projects, making these tools better for point and shot scenarios... the difference however, is not significant for some open source projects, which can "import" or employ the crawling capabilities of the a free version of a commercial product (requires some experience with certain tools - probably more suited for a consultant then a QA engineer).

· Some open source tools, even the most accurate ones, are relatively difficult to install & use, and still require fine-tuning in various fields, particularly stability. Other open source projects however, improved over the last year, and enhanced their user experience in many ways.

19. Verifying The Benchmark Results

The results of the benchmark can be verified by replicating the scan methods described in the scan log of each scanner, and by testing the scanner against WAVSEP v1.2 and WIVET v3-revision148.

The same methodology can be used to assess vulnerability scanners that were not included in the benchmark.

The latest version of WAVSEP can be downloaded from the web site of project WAVSEP (binary/source code distributions, installation instructions and the test case description are provided in the web site download section):

http://code.google.com/p/wavsep/

The latest version of WIVET can be downloaded from the project web site, or preferably, checked-out from the project subversion repository:

svn checkout http://wivet.googlecode.com/svn/trunk/ wivet-read-only

20. So What Now?

So now that we have all those statistics, it's time to analyze them properly, and see which conclusions we can get to. I already started writing a couple of articles that will make the information easy to use, and defined a methodology that will explain exactly how to use it. Analyzing the results however, will take me some time, since most of my time in the next few months will be invested in another project I'm working on (will be released soon), one I've been working on for the past year.

Since I didn't manage to test all the tools I wanted, I might update the results of the benchmark soon with additional tools (so you can think of it as a dynamic benchmark), and I will surely update the results in sectoolmarket (made some promises).

If you want to get notifications on new scan results, follow my blog or twitter account, and i'll do my best to tweet notification when I find the time to perform some major updates.

Since I have already been in the situation in the past, then I know what's coming… so I apologize in advance for any delays in my responses in the next few weeks, especially during august.

21. Recommended Reading List: Scanner Benchmarks

The following resources include additional information on previous benchmarks, comparisons and assessments in the field of web application vulnerability scanners:

· "SQL Injection through HTTP Headers", by Yasser Aboukir (an analysis and enhancement of the 2011 60 scanners benchmark, with a different approach for interpreting the results, March 2012)

· "The Scanning Legion: Web Application Scanners Accuracy Assessment & Feature Comparison", one of the predecessors of the current benchmark, by Shay Chen (a comparison of 60 commercial & open source scanners, August 2011)

· "Building a Benchmark for SQL Injection Scanners", by Andrew Petukhov (a commercial & opensource scanner SQL injection benchmark with a generator that produces 27680 (!!!) test cases, August 2011)

· "Webapp Scanner Review: Acunetix versus Netsparker", by Mark Baldwin (commercial scanner comparison, April 2011)

· "Effectiveness of Automated Application Penetration Testing Tools", by Alexandre Miguel Ferreira and Harald Kleppe (commercial & freeware scanner comparison, February 2011)

· "Web Application Scanners Accuracy Assessment", one of the predecessors of the current benchmark, by Shay Chen (a comparison of 43 free & open source scanners, December 2010)

· "State of the Art: Automated Black-Box Web Application Vulnerability Testing" (Original Paper), by Jason Bau, Elie Bursztein, Divij Gupta, John Mitchell (May 2010) – original paper

· "Analyzing the Accuracy and Time Costs of Web Application Security Scanners", by Larry Suto (commercial scanners comparison, February 2010)

· "Why Johnny Can’t Pentest: An Analysis of Black-box Web Vulnerability Scanners", by Adam Doup´e, Marco Cova, Giovanni Vigna (commercial & open source scanner comparison, 2010)

· "Web Vulnerability Scanner Evaluation", by AnantaSec (commercial scanner comparison, January 2009)

· "Analyzing the Effectiveness and Coverage of Web Application Security Scanners", by Larry Suto (commercial scanners comparison, October 2007)

· "Rolling Review: Web App Scanners Still Have Trouble with Ajax", by Jordan Wiens (commercial scanners comparison, October 2007)

· "Web Application Vulnerability Scanners – a Benchmark" , by Andreas Wiegenstein, Frederik Weidemann, Dr. Markus Schumacher, Sebastian Schinzel (Anonymous scanners comparison, October 2006)

22. Thank-You Note

During the research described in this article, I have received help from plenty of individuals and resources, and I’d like to take the opportunity to thank them all.

I might be reusing the texts, due to the late night hour and the constant lack of sleep I have been through in the last couple of months, but I mean every word that is written here.

For the kind souls that helped me obtain evaluation licenses for commercial products, for the CEO's, Marketing Executives, QA engineers, Support and Development teams of commercial vendors, which saved me tons of time, supported me throughout the process, helped me overcome obstacles and proved to me that the process of interacting with a commercial vendor can be a pleasant one, and for the various individuals that helped me contact these vendors.

I can't thank you enough, and wish you all the best.

For the information sources that helped me gather the list of scanners over the years, and gain knowledge, ideas, and insights, including (but not limited to) information security sources such as Security Sh3ll (http://security-sh3ll.blogspot.com/), PenTestIT (http://www.pentestit.com/), The Hacker News (http://thehackernews.com/), Toolswatch (http://www.vulnerabilitydatabase.com/toolswatch/), Darknet (http://www.darknet.org.uk/), Packet Storm (http://packetstormsecurity.org/), Google (of course), Twitter (my latest addiction) and many others great sources that I have used over the years to gather the list of tools.

I hope that the conclusions, ideas, information and payloads presented in this research (and the benchmarks and tools that will follow) will contribute to all the vendors, projects and most importantly, testers that choose to rely on them.

23. FAQ - Why Didn't You Test NTO, Cenzic and N-Stalker?

Prior to the benchmark, I made an important decision. I decided to go through official channels, and either contact vendors and work with them, or use public evaluation versions of relatively simple products. I had a huge amount of tasks, and needed the support to cut the learning curve of understanding how optimize the tools. I was determined to meet my deadline, didn't have any time to spare, and was willing to make certain sacrifices to meet my goals.

As for why specific vendors were not included, this is the short answer:

NTO: I only managed to get in touch with NTO about two weeks before the benchmark publication. I didn't have luck contacting the guys I worked with in the previous benchmarks, but was eventually contacted by Kim Dinerman. She was nice and polite, and apologized for the time the process took. After explaining to her which timeframe they have for enhancing the product (an action performed by other commercial vendors as well, in order to prepare for the publically known tests of the benchmark), they decided that the timeframe and circumstances don't provide an even opportunity and decided not to participate.

I admit that by the time they contacted me, I was so loaded with tasks, that it was somewhat relieved, even though I was curious and wanted to assess their product. That being said, I decided prior to the benchmark that I will respect the decisions of vendors, even if will cause me to not get to a round scanner number.

N-Stalker: I finally received a valid N-Stalker license one day before the publication of the benchmark - a couple of days after the final deadline I had for accepting any tool. I decided to give it a shot, just in case it will be a simple process, however, with my luck, I immediately discovered a bug that prevented me from properly assessing the product and it's features, and unlike the rest of tests which were performed with a sufficient timeframe... this time, I had no time to find a workaround. I decided not to publish the partial results I had (I did not want to create the wrong impression or hurt anyone's business), and notified the vendor on the bug and on my decision.

The vendor, from his part, thanked me for the bug report, and promised to look up the issue. Sorry guys... I wanted to test them too... next benchmark.

Cenzic: the story of Cenzic is much simpler than the rest. I simply didn't manage to get in touch, and even though I did have access to a license, I decided prior to the benchmark not to take that approach. As I mentioned earlier, I decided to respect the vendor decisions, and not to assess their product without their support.

24. Appendix A – List of Tools Not Included In the Test

The following commercial web application vulnerability scanners were not included in the benchmark, due to deadlines and time restrictions from my part, and in the case of specific vendors, for other reasons.

Commercial Scanners not included in this benchmark

· N-Stalker Commercial Edition (N-Stalker)

· Hailstorm (Cenzic)

· NTOSpider (NTO)

· McAfee Vulnerability Manager (McAfee / Foundstone)

· NeXpose Enterprise Edition Web Application Scanning Features (Rapid7)

· Retina Web Application Scanner (eEye Digital Security)

· SAINT Scanner Web Application Scanning Features (SAINT co.)

· WebApp360 (NCircle)

· Core Impact Pro Web Application Scanning Features (Core Impact)

· Parasoft Web Application Scanning Features (a.k.a WebKing, by Parasoft)

· MatriXay Web Application Scanner (DBAppSecurity)

· Falcove (BuyServers ltd, currently Unmaintained)

· Safe3WVS 13.1 Commercial Edition (Safe3 Network Center)

The following open source web application vulnerability scanners were not included in the benchmark, mainly due to time restrictions, but might be included in future benchmarks:

Open Source Scanners not included in this benchmark

· Vanguard

· WebVulScan

· SQLSentinel

· XssSniper

· Rabbit VS

· Spacemonkey

· Kayra

· 2gwvs

· Webarmy

· springenwerk

· Mopset 2

· XSSFuzz 1.1

· Witchxtoolv

· PHP-Injector

· XSS Assistant

· Fiddler XSSInspector/XSRFInspector Plugins

· GNUCitizen JAVASCRIPT XSS SCANNER - since WebSecurify, a more advanced tool from the same vendor is already tested in the benchmark.

· Vulnerability Scanner 1.0 (by cmiN, RST) - since the source code contained traces for remotely downloaded RFI lists from locations that do not exist anymore.

The benchmark focused on web application scanners that are able to detect either Reflected XSS or SQL Injection vulnerabilities, can be locally installed, and are also able to scan multiple URLs in the same execution.

As a result, the test did not include the following types of tools:

· Online Scanning Services – Online applications that remotely scan applications, including (but not limited to) Appscan On Demand (IBM), Click To Secure, QualysGuard Web Application Scanning (Qualys), Sentinel (WhiteHat), Veracode (Veracode), VUPEN Web Application Security Scanner (VUPEN Security), WebInspect (online service - HP), WebScanService (Elanize KG), Gamascan (GAMASEC – currently offline), Cloud Penetrator (Secpoint), Zero Day Scan, DomXSS Scanner, etc.

· Scanners without RXSS / SQLi detection features:

o Dominator (Firefox Plugin)

o fimap

o lfimap

o phpBB-RFI Scanner

o DotDotPawn

o LFI (Library-level Fault Injector)

o lfi-scanner

o LFI-Scanner

o lfi-rfi2

o LFI/RFI Checker (astalavista)

o CSRF Tester

o etc

· Passive Scanners (response analysis without verification):

o Watcher (Fiddler Plugin by Casaba Security)

o Skavanger (OWASP)

o Pantera (OWASP)

o Ratproxy (Google)

o CAT The Manual Application Proxy (Context)

o etc

· Scanners of specific products or services (CMS scanners, Web Services Scanners, etc):

o WSDigger

o Sprajax

o ScanAjax

o Joomscan

o wpscan

o Joomlascan

o Joomsq

o WPSqli

o etc

· Web Application Scanning Tools which are using Dynamic Runtime Analysis:

o PuzlBox (the free version was removed from the web site, and is now sold as a commercial product named PHP Vulnerability Hunter)

o Inspathx

o etc

· Uncontrollable Scanners - scanners that can’t be controlled or restricted to scan a single site, since they either receive the list of URLs to scan from Google Dork, or continue and scan external sites that are linked to the tested site. This list currently includes the following tools (and might include more):

o Darkjumper 5.8 (scans additional external hosts that are linked to the given tested host)

o Bako's SQL Injection Scanner 2.2 (only tests sites from a google dork)

o Serverchk (only tests sites from a google dork)

o XSS Scanner by Xylitol (only tests sites from a google dork)

o Hexjector by hkhexon – also falls into other categories

o d0rk3r by b4ltazar

o etc

· Deprecated Scanners - incomplete tools that were not maintained for a very long time. This list currently includes the following tools (and might include more):

o Wpoison (development stopped in 2003, the new official version was never released, although the 2002 development version can be obtained by manually composing the sourceforge URL which does not appear in the web site- http://sourceforge.net/projects/wpoison/files/ )

o etc

· De facto Fuzzers – tools that scan applications in a similar way to a scanner, but where the scanner attempts to conclude whether or not the application or is vulnerable (according to some sort of “intelligent” set of rules), the fuzzer simply collects abnormal responses to various inputs and behaviors, leaving the task of concluding to the human user.

o Lilith 0.4c/0.6a (both versions 0.4c and 0.6a were tested, and although the tool seems to be a scanner at first glimpse, it doesn’t perform any intelligent analysis on the results).

o Spike proxy 1.48 (although the tool has XSS and SQLi scan features, it acts like a fuzzer more then it acts like a scanner – it sends payloads of partial XSS and SQLi, and does not verify that the context of the returned output is sufficient for execution or that the error presented by the server is related to a database syntax injection, leaving the verification task for the user).

· Fuzzers – scanning tools that lack the independent ability to conclude whether a given response represents a vulnerable location, by using some sort of verification method (this category includes tools such as JBroFuzz, Firefuzzer, Proxmon, st4lk3r, etc). Fuzzers that had at least one type of exposure that was verified were included in the benchmark (Powerfuzzer).

· CGI Scanners: vulnerability scanners that focus on detecting hardening flaws and version specific hazards in web infrastructures (Nikto, Wikto, WHCC, st4lk3r, N-Stealth, etc)

· Single URL Vulnerability Scanners - scanners that can only scan one URL at a time, or can only scan information from a google dork (uncontrollable).

o Havij (by itsecteam.com)

o Hexjector (by hkhexon)

o Simple XSS Fuzzer [SiXFu] (by www.EvilFingers.com)

o Mysqloit (by muhaimindz)

o PHP Fuzzer (by RoMeO from DarkMindZ)

o SQLi-Scanner (by Valentin Hoebel)

o Etc.

· Vulnerability Detection Assisting Tools – tools that aid in discovering a vulnerability, but do not detect the vulnerability themselves; for example:

o Exploit-Me Suite (XSS-Me, SQL Inject-Me, Access-Me)

o Fiddler X5s plugin

o XSSRays (chrome Addon)

· Exploiters - tools that can exploit vulnerabilities but have no independent ability to automatically detect vulnerabilities on a large scale. Examples:

o MultiInjector

o XSS-Proxy-Scanner

o Pangolin

o FGInjector

o Absinth

o Safe3 SQL Injector (an exploitation tool with scanning features (pentest mode) that are not available in the free version).

o etc

· Exceptional Cases

o SecurityQA Toolbar (iSec) – various lists and rumors include this tool in the collection of free/open-source vulnerability scanners, but I wasn’t able to obtain it from the vendor’s web site, or from any other legitimate source, so I’m not really sure it fits the “free to use” category.

Started Testing Scanners for the 2012 Benchmark!

2012-05-28T15:28:00.001-07:00

Well, it took me some time to finish the new version of the assessment platform, but now I'm all set and ready to go.

I've just started testing Web Application Vulnerability Scanners for the 2012 scanner comparison. The benchmark will cover freeware, open-source and commercial web application scanners, including newly published tools, many tools that were not assessed in the last benchmark, and even tools that usually fall under different categories.

The test will enable any previously assessed product a chance to improve its previous score (all the previous wavsep test-cases will be included in each product assessment), and also allow tools to excel in new categories.

Want to improve your previous score, get your product in the list, or even conquer the top?

Now's a good time to notify me about your product, and provide me with a license.

SecToolMarket - A dynamic benchmark presentation website

2012-02-19T17:36:00.000-08:00

Although I can't really claim that what I'm about to present is perfect (I'm learning to control that one, hopefully), and the design is not yet memorable (U-N-D-E-R-S-T-A-T-E-M-E-N-T), it's certainly going to be useful for a lot of folks - pen-testers (first and foremost), vendors, analysts, researchers, security personal, and a bunch of people that stumbled upon this blog and are about to face a lot of scary words.

In short, the benchmark presentation framework is up and ready, and published as a web site called SecToolMarket (http://www.sectoolmarket.com).

I originally planned hosting it in Google sites (which is why there's no JS/AJAX/etc), but after a couple of hours of desperately trying to upload bulks of files to Google sites, I gave up and used the conventional method.

Although it doesn't yet contain a lot of new information (mostly additional information & analysis of the products tested in 2011), it's much easier to navigate through the data, and the analysis of the 2011 benchmark can provide additional insights, even to those that read the 2011 benchmark post.

A part from adding statistics, making things simpler, adding glossaries for everything and collecting vendor and product specific stats under dedicated pages, this framework can also be updated more frequently (and hopefully on a consistent basis), contains information that wasn't published, and allows you to track my progress as I'm performing my comparisons.

The two new categories (input-vector-support and coverage) are still incomplete (and will probably be updated soon, especially for commercial scanners - which will hopefully notify me if there's any missing information), but they already provide some insights, that might be relevant for some us.

Some screen captures of the content:

I'll probably post additional information on this website, and my future plans, but in the meantime, I'm going to crash, and hope you have fun with it.

Rules of the Game – Scanner Benchmarks

2011-10-23T15:46:00.000-07:00

The last couple of months have been very interesting (thanks for all the great feedback and constructive criticism), and I have some good news and several announcements.

First, the good news:

I had several discussions with Simon Bennetts (psiinon), one of the chapter leaders in OWASP and the leader / co-leader of several OWASP projects (ZAP, WAVE and OWASP-DEF).

One of the sub projects Simon is leading is ZAP-WAVE, the only additional web-app scanner evaluation framework which is actively maintained (the last publically available update of the third framework – "moth", was in mid 2009), and he suggested we merge our efforts so that everyone will benefit.

To make a long story short, Simon contributed the source code of the current test cases of ZAP-WAVE, allowing me adjust them into WAVSEP format and publish them under GPL 3.0 (currently available under ASF 2.0, lawyer comments aside). He even suggested that in the future, test cases that will be implemented by the ZAP team will be in WAVSEP format (structure and documentation).

That's obviously great news for me (and for anyone else using the project or the benchmark results – credits to Simon Bennetts and Axel Neumann), since the ZAP-WAVE project already contains test cases in several exposures that are not covered by WAVSEP, and any additional contribution will only enhance my current efforts (I'm currently working on dozens of additional test cases for new exposure categories).

I have already started to adjust these test cases (changes and integration notes will appear in the changelog), and I hope I'll manage to release them soon.

Now for several announcements that are related to the upcoming benchmark and the future versions of WAVSEP:

After the last benchmark was published, I got a lot of feedback, requests, interesting ideas and various suggestions. I read it all, and some of the requests and suggestions will be implemented in the upcoming benchmark and the future versions of WAVSEP.

The different feedbacks lead me to some important realizations:

· The rules of the test must be made public and clear to all vendors, in order to make sure that the process will be fair. In order to achieve this goal, certain changes must be implemented in the testing process.

· In order to enable vendors to show improvement quickly and in order to prevent any previous "negative" results from being perceived as a long term "punishment", the result presentation method must be updated more frequently, even between benchmarks.

As a result, I have constructed the following set of rules which will govern the testing processes in any future benchmark I will perform, and also require some changes in the publication cycles of WAVSEP:

· In order to enable vendors to show improvement, all the future benchmarks will be based on the WAVSEP test cases used in the previous benchmark, in addition to any other tests (interpretation: the upcoming benchmark will also include tests against all the SQLi and RXSS test cases of WAVSEP 1.0.3).

· Since a benchmark is much more interesting if the content in it is new, each major benchmark will include different test aspects and / or detection results for test cases in additional exposure categories.

· In order for the contest to be even more interesting (and in order to prevent one vendor from preparing for everything while another was not even notified, was not aware of the WAVSEP platform, has insufficient time to improve the tool prior to the benchmark, etc), the test cases of some of the new exposure categories will only be published after the first major benchmark that included tests against them – something that will add some spice to the results, make sure the process will be fair, but will still enable vendors to improve their previous score.

· The upcoming benchmark will include tests in some new categories: I'm currently aiming for at least 3 additional categories, in addition to the previous (and I hope that I'll manage to finish all the developments and tests before my next deadline… at least for most tools).

· Vendors that wish to update their score will be given an opportunity to do so, even between major benchmarks, by using a presentation method that will support dynamically updated content. The terms for these tests will be published separately, as soon as the presentation framework will be available (soon). Re-tests of additional versions of the same product will be performed under these terms.

· Since my final goal is to test all the vendors (almost 100), test additional types of scanning services / products, and eventually, test as many features of these tools as I possibly can, my time is a valuable asset, and contacting commercial vendors that don't offer a publically available evaluation version is very difficult. Although I will try my best to go through official channels and perform all the tests myself (or through members of the WAVSEP project), my experience shows that in some cases, the official channels might be time consuming, and sadly, sometimes more then I can afford. Therefore, I encourage vendors to contact me directly, starting of November 15, so I could test them properly, on equal terms.

Summary: future benchmarks will include test cases used in previous benchmarks (to enable vendors to show improvement), new test cases which will only be published after the benchmark (so that the tests will be fair and the content more interesting), and finally - the results will be more dynamic, to disable one more participation barrier.

As I said in my previous posts - I'm planning to continue to perform these comparisons for a long time, and intend to make sure that the community and vendors will both be able benefit from this initiative, if they only choose to.

Cheers

Session Puzzling and Session Race Conditions

2011-09-18T08:54:00.000-07:00

Is It Really That Complicated?

Session Puzzling – An Indirect Application Attack Vector – Now Simplified

A couple of months ago, I published a paper on an under-emphasized application level attack vector nicknamed "Session Puzzling" – an attack pattern that can abuse improper usage of session variables (a.k.a "Session Puzzles") in order to impersonate users, elevate privileges, bypass security restrictions and even execute "traditional" attack vectors against applications, while bypassing any existing security mechanisms by attacking the application using a trusted input source.

Even though the paper was published alongside a training kit that was meant to demonstrate the various attack vectors (a vulnerable application called "puzzlemall"), the vast majority of responses I got have made me realize that most of the 2000 security professionals that were exposed to this attack did not manage to understand it.

Some of the responses associated the paper to unrelated attacks, some didn't understand the impact or the mechanics, and some responses even claimed that the attacks is too complicated to perform (!?!).

Although I know that the attack is not simple, and that several session puzzling vectors require 10+ requests, I refuse to believe it's that complicated.

Over the last couple of years, I have seen many commercial applications that were vulnerable to this attack (Oracle E-Business Suite Included), so I'm giving it one more shot before I'll let the attack fall into the "too complicated to explain" category, and keep it all to myself.

The original whitepaper/presentation can be downloaded from the following addresses (contains background, additional attack vectors and mitigations):
Whitepaper
Presentation

The project homepage:
http://puzzlemall.googlecode.com/

The following short movies demonstrate a few simple session puzzling sequences:

Authentication Bypass via Session Puzzling (Abusing common session variables):

http://www.youtube.com/watch?v=-DackF8HsIE

User Impersonation via Session Puzzling (Abusing common session variables):

http://www.youtube.com/watch?v=ikIyInm0wAg

Session Puzzling via Redirection Prevention (Abusing Premature Session Population):

http://www.youtube.com/watch?v=iTcOooHbgog

Bypassing Restrictions in Multiphase Processes via Session Puzzling (Abusing Common Session Flags)

http://www.youtube.com/watch?v=HeP54b52IeQ

The following POC movie demonstrates the attack against Oracle E-Business Suite (exception scenario - not relying on input):

http://www.hacktics.com/content/advisories/AdvORA20091214.html

The training kit can be downloaded from the following address:

http://puzzlemall.googlecode.com/files/puzzlemall.war (derby version)

Temporal Session Race Conditions and Layer Targeted ADoS

Although the original attack relied on the existence of persistent session values, an extended attack was presented last week (15^th of September), in a local OWASP chapter meeting.

The extended method (nicknamed "Temporal Session Race Conditions") enables detecting & exploiting session puzzles even if the session variables have a lifespan of milliseconds (session-level race conditions), by increasing the latency of certain lines of code through the use of layer targeted denial of service attacks.

The original OWASP presentation:
Presentation

The following movies demonstrate a few simple TSRC attacks:

Exploiting Temporal Session Race Conditions via Connection Pool Consumption:

http://www.youtube.com/watch?v=woWECWwrsSk

Exploiting Temporal Session Race Conditions via RegEx DoS:

http://www.youtube.com/watch?v=3k_eJ1bcCro

An extended version of "puzzlemall" which includes TSRC vulnerabilities (premium login page, requires MySQL):

http://puzzlemall.googlecode.com/files/puzzlemall-v.1.1.2-mysql.zip

A simple tool that can assist in the detection of TSRC connection pool consumption scenarios:

http://puzzlemall.googlecode.com/files/SessionKeepAlive.exe

Acknowledgements

The following individuals contributed to the Session Puzzling / TSRC research in various ways, and helped me turn a bunch of ideas into a consistent methodology:

Oren Ofer, Oren Hafif, Alex Ganelis, Liran Sheinbox and Zafrir Grossman.

Additional Resources
An attack similar to session puzzling is mentioned under the name "session poisoning", but the session puzzling/TSRC sequences differ from this attack mainly by the lack of direct input dependency (see the multiphase restriction bypass scenario and the e-business suite exploit for the exception scenario), and expand the attack tool-set in the aspect of methodology, predefined sequences, scope of modules, complementary methods and usage of denial of service for extending the lifespan of temporary session variables.

Commercial Web Application Scanner Benchmark

2011-08-01T20:48:00.000-07:00

The Scanning Legion:

Web Application Scanners Accuracy Assessment & Feature Comparison

Commercial & Open Source Scanners

A Comparison of 60 Commercial & Open Source Black Box Web Application Vulnerability Scanners

By Shay Chen

Security Consultant, Researcher & Instructor

http://sectooladdict.blogspot.com/

sectooladdict-$at$-gmail-$dot$-com

August 2011

Assessment Environments: WAVSEP 1.0 / WAVSEP 1.0.3 (http://code.google.com/p/wavsep/)

Disclaimer

The results of this research are only valid for estimating the detection accuracy of SQLi & RXSS exposures, and for counting and comparing the various features of the tested tools.

The author did not evaluate every possible feature of each product, only the categories tested within the research, and thus, does not claim to be able to estimate the ROI from each individual product.

Furthermore, several vendors invested resources in improving their tools according to the recommendations of the WAVSEP platform which was publically available since December 2010. Some of them did so without any relation to the benchmark (and before they were aware of it), and some in preparation for it. Since the special structure of the WAVSEP testing platform actually requires the vendor to cover more vulnerable test scenarios, that action actually improves the detection ratio of the tool in any application (for the exposures covered by WAVSEP).

It is however, important to mention that a few vendors were not notified on this benchmark, and were not aware of the existence of the WAVSEP platform, and thus, could not have enhanced their tools in preparation for this benchmark (HP Webinspect, Tenable Nessus, and Janus security Webcruiser), while other vendors that were tested in the initial research phases released updated versions that were not tested (Portswigger Burpsuite and Cenzic Hailstorm)

That being said, the benchmark does represent the accuracy level of each tool in the date it was tested (the results of the vast majority of the tools are valid for the date this research was released), but future benchmark will use a different research model in order to ensure that the competition will be fair for all vendors.

Table of Contents

1. Prologue

2. List of Tested Web Application Scanners

3. Benchmark Overview & Assessment Criteria

4. Test I – The More The Merrier – Counting Audit Features

5. Test II – To the Victor Go the Spoils – SQL Injection

6. Test III – I Fight (For) the Users – Reflected XSS

7. Test IV – Knowledge is Power - Feature Comparison

8. What Changed?

9. Initial Conclusions – Open Source vs. Commercial

10. Morale Issues in Commercial Product Benchmarks

11. Verifying The Benchmark Results

12. Notifications and Clarifications

13. List of Tested Scanners

14. Source, License and Technical Details of Tested Scanners

15. Comparison of Active Vulnerability Detection Features

16. Comparison of Complementary Scanning Features

17. Comparison of Usability and Coverage Features

18. Comparison of Connection and Authentication Features

19. Comparison of Advanced Features

20. Detailed Results: Reflected XSS Detection Accuracy

21. Detailed Results: SQL Injection Detection Accuracy

22. Drilldown – Error Based SQL Injection Detection

23. Drilldown – Blind & Time Based SQL Injection Detection

24. Technical Benchmark Conclusions – Vendors & Users

25. So What Now?

26. Recommended Reading List: Scanner Benchmarks

27. Thank-You Note

28. Frequently Asked Questions

29. Appendix A – Assessing Web Application Scanners

30. Appendix B – A List of Tools Not Included In the Test

31. Appendix C – WAVSEP Scan Logs

32. Appendix D – Scanners with Abnormal Behavior

1. Prologue

I've always been curious about it… from the first moment I executed a commercial scanner, almost seven years ago, to the day I started performing this research. Although manual penetration testing has always been the main focus of the test, most of us use automated tools to easily detect "low hanging fruit" exposures, increase the coverage when testing large scale applications in limited timeframes and even to double check locations that were manually tested. The questions always pops up, in every penetration test in which these tools are used…

"Is it any good?", "Is it better than…" and "Can I rely on it to…" are questions that every pen-tester asks himself whenever he hits the scan button.

Well, curiosity is a strange beast… it can drive you to wander and search, consume all your time in a search for obscure solutions.

So recently, because of curiosity, I decided that I want to find out for myself, and invest whatever resources necessary to solve this mystery once and for all.

Although I can hardly state that all my questions were answered, I can definitely sate your curiosity for the moment, by sharing insights, interesting facts, useful information and even some surprises, all derived from my latest research which is focused on the subject of commercial & open source web application scanners.

This research covers the latest versions of 12 commercial web application scanners and 48 free & open source web application scanners, while comparing the following aspects of these tools:

· Number & Type of Vulnerability Detection Features

· SQL Injection Detection Accuracy

· Reflected Cross Site Scripting Detection Accuracy

· General & Special Scanning Features

Although my previous research included similar information, I regretted one thing after it was published; I did not present the information in a format that was useful to the common reader. In fact, as I found out later, many readers skipped the actual content, and focused on sections of the article that were actually a side effect of the main research.

As a result, the following article will focus on presenting the information in a simple comprehendible graphical format, while still providing the detailed research information to those interested… and there's a lot of new information to be shared – knowledge that can aid pen-testers in choosing the right tools, managers in budget related decisions, and visionaries, in properly reading the map;

But before you read the statistics and insights presented in this report, and reach a conclusion as to which tool is the "best", it is crucial that you read ‎Appendix A - Section 29, which explains the complexity of assessing the overall quality of web application scanners… As you're about to find out, this question cannot be answered so easily… at least not yet.

…

So without any further delay, let's focus on the information you seek, and discuss the insights and conclusions later.

2. List of Tested Web Application Scanners

The following commercial scanners were included in the benchmark:

· IBM Rational AppScan v8.0.03 - iFix Version (IBM)

· WebInspect v9.10.78.0, SecureBase 4.05.99 (HP)

· Hailstorm Professional v6.5-5267(Cenzic)

· Acunetix WVS v7.0-20110608 (Acunetix)

· NTOSpider v 5.4.098 (NT Objectives)

· Netsparker v2.0.0.0 (Mavituna Security)

· Burp Suite v1.3.09 (Portswigger)

· Sandcat v4.2.4.0 (Syhunt)

· ParosPro v1.9.12 (Milescan)

· JSky v3.5.1-905 (NoSec)

· WebCruiser v2.5.0 EE (Janus Security)

· Nessus v4.41-15078 (Tenable Network Security) – Only the Web Application Scanning Features

The following new free & open source scanners were included in the benchmark:

VEGA 1.0 beta (Subgraph), Safe3WVS v9.2 FE (Safe3 Network Center), N-Stalker 2012 Free Edition v7.1.1.106 (N-Stalker), DSSS (Damn Simple SQLi Scanner) v0.1h, SandcatCS v4.2.3.0

The updated versions of the following free & open source scanners were re-tested in the benchmark:

Zed Attack Proxy (ZAP) v1.3.0, sqlmap v0.9-rev4209 (SVN), W3AF 1.1-rev4350 (SVN), Watobo v0.9.7-rev544, Acunetix Free Edition v7.0-20110711, Netsparker Community Edition v1.7.2.13, WebSecurify v0.8, WebCruiser v2.4.2 FE (corrections), arachni v0.2.4 / v0.3, XSSer v1.5-1, Skipfish 2.02b, aidSQL 02062011

The results were compared to those of unmaintained scanners tested in the original benchmark:

Andiparos v1.0.6, ProxyStrike v2.2, Wapiti v2.2.1, Paros Proxy v3.2.13, PowerFuzzer v1.0, Grendel Scan v1.0, Oedipus v1.8.1, Scrawler v1.0, Sandcat Free Edition v4.0.0.1, JSKY Free Edition v1.0.0, N-Stalker 2009 Free Edition v7.0.0.223, UWSS (Uber Web Security Scanner) v0.0.2, Grabber v0.1, WebScarab v20100820, Mini MySqlat0r v0.5, WSTool v0.14001, crawlfish v0.92, Gamja v1.6, iScan v0.1, LoverBoy v1.0, openAcunetix v0.1, ScreamingCSS v1.02, Secubat v0.5, SQID (SQL Injection Digger) v0.3, SQLiX v1.0, VulnDetector v0.0.2, Web Injection Scanner (WIS) v0.4, Xcobra v0.2, XSSploit v0.5, XSSS v0.40, Priamos v1.0

For the full list of commercial & open source tools that were not tested in this benchmark, refer to ‎Appendix B - Section 30.

3. Benchmark Overview & Assessment Criteria

· The ability to detect Reflected XSS and/or SQL Injection vulnerabilities.

· The ability to scan multiple URLs at once (using either a crawler/spider feature, URL/Log file parsing feature or a built-in proxy).

· The ability to control and limit the scan to internal or external host (domain/IP).

The testing procedure of all the tools included the following phases:

· The scanners were all tested against the latest version of WAVSEP (v1.0.3), a benchmarking platform designed to assess the detection accuracy of web application scanners. The purpose of WAVSEP’s test cases is to provide a scale for understanding which detection barriers each scanning tool can bypass, and which vulnerability variations can be detected by each tool. The various scanners were tested against the following test cases (GET and POST attack vectors):

o 66 test cases that were vulnerable to Reflected Cross Site Scripting attacks.

o 80 test cases that contained Error Disclosing SQL Injection exposures.

o 46 test cases that contained Blind SQL Injection exposures.

o 10 test cases that were vulnerable to Time Based SQL Injection attacks.

o 7 different categories of false positive RXSS vulnerabilities.

o 10 different categories of false positive SQLi vulnerabilities.

· In order to ensure the result consistency, the directory of each exposure sub category was individually scanned multiple times using various configurations.

· The features of each scanner were documented and compared, according to documentation, configuration, plugins and information received from the vendor.

· In order to ensure that the detection features of each scanner were truly effective, most of the scanners were tested against an additional benchmarking application that was prone to the same vulnerable test cases as the WAVSEP platform, but had a different design, slightly different behavior and different entry point format (currently nicknamed "bullshit").

So, now that you've learned about the testing process, it's time for the results…

4. Test I – The More The Merrier – Counting Audit Features

The first assessment criterion was the number of audit features each tool supports.

Reasoning: An automated tool can't detect an exposure that it can't recognize (at least not directly, and not without manual analysis), and therefore, the number of audit features will affect the amount of exposures that the tool will be able to detect (assuming the audit features are implemented properly, that vulnerable entry points will be detected and that the tool will manage to scan the vulnerable input vectors).

The Number of Audit Features in Web Application Scanners – Commercial Tools

The Number of Audit Features in Web Application Scanners - Free & Open Source Tools

The Number of Audit Features in Web Application Scanners – Unified List

So, now that were done with the quantity, let's get to the quality…

5. Test II – To the Victor Go the Spoils – SQL Injection

The second assessment criterion was the detection accuracy of SQL Injection, one of the most famous exposures and the most commonly implemented attack vector in web application scanners.

Reasoning: a scanner that is not accurate enough will miss many exposures, and classify non-vulnerable entry points as vulnerable. This test aims to assess how good is each tool at detecting SQL Injection exposures in a supported input vector, which is located in a known entry point, without any restrictions that can prevent the tool from operating properly.

The evaluation was performed on an application that uses MySQL 5.5.x as its data repository, and thus, will reflect the detection accuracy of the tool when scanning similar data repositories.

Result Chart Glossary

Note that the BLUE bar represents the vulnerable test case detection accuracy, while the RED bar represents false positive categories detected by the tool (which may result in more instances then what the bar actually presents, when compared to the detection accuracy bar).

The SQL Injection Detection Accuracy of Web Application Scanners – Commercial Tools

The SQL Injection Detection Accuracy of Web Application Scanners – Open Source & Free Tools

The SQL Injection Detection Accuracy of Web Application Scanners – Unified List

It's obvious that testing one feature is not enough; ideally, the detection accuracy of all audit features should be assessed, but in the meantime, we will settle for one more…

6. Test III – I Fight (For) the Users – Reflected XSS

The third assessment criterion was the detection accuracy of Reflected Cross Site Scripting, a common exposure which is the 2nd most commonly implemented feature in web application scanners.

Result Chart Glossary

The Reflected XSS Detection Accuracy of Web Application Scanners – Commercial Tools

The Reflected XSS Detection Accuracy of Web Application Scanners – Open Source & Free Tools

The Reflected XSS Detection Accuracy of Web Application Scanners – Unified List

7. Test IV – Knowledge is Power - Feature Comparison

The list of tools tested in this benchmark is organized within the following reports:

· List of Tested Scanners

· Source, License and Technical Details of Tested Scanners

Additional information was gathered during the benchmark, including information related to the different features of the various scanners. These details are organized in the following reports, and might prove useful when searching for tools for specific tasks or tests:

· Comparison of Active Vulnerability Detection Features (Audit Features) – 1 of 2

· Comparison of Active Vulnerability Detection Features (Audit Features) – 2 of 2

· Comparison of Complementary Scanning Features - Passive Analysis, CGI Scanning, Etc

· Comparison of Usability, Coverage and Scan Initiation Features

· Comparison of Authentication, Scan Control and Connection Support Features

· Comparison of Advanced and Uncommon Features

For detailed information on the accuracy assessment results, refer to the following reports:

· Benchmark Results – Reflected XSS Detection Accuracy

· Benchmark Results – SQL Injection Detection Accuracy – Unified

· Benchmark Drilldown – Blind & Time Based SQL Injection Detection Accuracy

· Benchmark Drilldown – Error Dependant SQL Injection Detection Accuracy

· The Scan Logs (describing the executing process and configuration of each scanner)

Additional information on the scan logs, the list of untested tools and the abnormal behaviors of scanners can be found in the article appendix sections (at the end of the article):

‎Appendix B - Section 30 – an appendix that contains a list of tools that were not included in the benchmark

‎Appendix D - Section 32 – an appendix that describes scanners with abnormal behavior

8. What Changed?

Since the latest benchmark, many open source & commercial tools added new features and improved their detection accuracy.

The following list presents a summary of changes in the detection accuracy of free & open source tools that were tested in the previous benchmark:

· arachni – a dramatic improvement in the detection accuracy of Reflected XSS exposures, and a dramatic improvement in the detection accuracy of SQL Injection exposures (verified on mysql).

· sqlmap – a dramatic improvement in the detection accuracy of SQL Injection exposures (verified on mysql).

· Acunetix Free Edition – a major improvement in the detection accuracy of RXSS exposures.

· Watobo – a major improvement in the detection accuracy of SQL Injection exposures (verified on mysql).

· N-Stalker 2009 FE vs. 2012 FE – although this tool is a very similar to N-Stalker 2009 FE, the surprising discovery I had was that the detection accuracy of N-Stalker 2012 is very different – it detects only a quarter of what N-Stalker 2009 used to detect. Assuming this result is not related to a bug in the product or in my testing procedure, it means that the newer free version is significantly less effective than the previous free version, at least at detecting reflected XSS. A legitimate business decision, true, but surprising nevertheless.

· aidSQL – a major improvement in the detection accuracy of SQL Injection exposures (verified on mysql).

· XSSer – a major improvement in the detection accuracy of Reflected XSS exposures, even though the results were not consistent.

· Skipfish – a slight improvement in the detection accuracy of RXSS exposures (it is currently unknown if the RXSS detection improvement is related to changes in code or to the enhanced testing method), and a slight decrease in the detection accuracy of SQLi exposures (might be related to the different testing environment and the different method used to count the results).

· WebSecurify – a slight improvement in the detection accuracy of RXSS exposures (it is currently unknown if the RXSS detection improvement is related to changes in code or to the enhanced testing method).

· Zed Attack Proxy (ZAP) – Identical results. Any minor difference was probably caused due to the testing environment, configuration or minor issues.

· W3AF – slight improvement in the detection accuracy of RXSS exposures and slight decrease in the detection accuracy of SQL Injection exposures.

· Netsparker Community Edition – Identical results. Any minor difference was probably caused due to the testing environment, configuration or minor issues.

· WebCruiser Free Edition – a minor decrease in accuracy, due to fixing documentation mistakes from the previous benchmark.

9. Initial Conclusions – Open Source vs. Commercial

The following section presents my own personal opinions on the results of the benchmark, and since opinions are beliefs, which are affected by emotions and circumstances, you are entitled to your own.

After testing over 48 open source scanners multiple times, and after comparing the results and experiences to the ones I had after testing 12 commercial ones (and those are just the ones that I reported), I have reached the following conclusions:

· As far as accuracy & features, the distance between open source tools and commercial tools is not as big as it used to be – tools such as sqlmap, arachni, wapiti, w3af and others are slowly closing the gap. That being said, there still is a significant difference in stability & false positives, in which most open source tools tend to have more false positives and be relatively unstable when compared to most commercial tools.

· Some open source tools, even the most accurate ones, are relatively difficult to install & use, and still require fine-tuning in various fields. In my opinion, a non-technical QA engineer will have difficulties using these tools, and as a general rule, I'll recommend using them if your background is relatively technical (consultant, developer, etc). For all the rest, especially non-technical enterprise employees that prefer a decent usage experience - stick with commercial produces, with their free versions, or with the simple variations of open source tools.

· If you are using a commercial product, it's best to merge the use of tools with a wide variety of features with tools with high detection accuracy. It's possible to use tools that have relatively good scores in both of these aspects, or use a tool with a wide variety of features with another tool that has enhanced accuracy. Yes, this statement can be interpreted to using combinations of commercial and open source tools, and even to using two different commercial tools, so that one tool will complete the other. Budget? Take a look at the cost diversity of the tools, before you make any harsh decisions; I promise you'll be surprised.

10. Morale Issues in Commercial Product Benchmarks

While testing the various commercial tools, I have dealt with certain moral issues that I want to share. Many vendors that were aware of this research enhanced their tools in preparation for it, an action I respect, and consider a positive step. Since the testing platform that included most of the tests was available online, preparing for the benchmark was a relatively easy task for any vendor that invested the resources.

So, is the benchmark fair for vendors that couldn’t improve their tools due to various circumstances?

The testing process of a commercial tool is usually much more complicated and restrictive then testing a free or open source tool; it is necessary to contact the vendor to obtain an evaluation license, and the latest version of the tool (a process that can take several weeks), the evaluation licenses are usually restricted to a short evaluation timeframe (usually two weeks), and thus, updating and testing the tools in a future date can become a hassle (since some of the process will have to be performed all over again)… but why am I telling you all this?

Simply, because I believe that the relevance of the test I performed for vendors that provided me an extended evaluation period and access to new builds was better; for example, a few days before the latest benchmark, immediately after testing the latest versions of two major vendors, I decided to rescan the platform using the latest versions of all the commercial tools I have, to ensure that the benchmark will be published with the most updated results.

I verified that JSky, WebCruiser, and ParosPro didn't release a new version, tested the latest versions of AppScan, WebInspect, Acunetix, Netsparker, Sandcat and Nessus.

It made sense that builds that were tested a short while ago (like NTO spider), were also something that I can rely on to represent the currently state of the tool (I hopeJ).

I did however, have a problem with Cenzic and Burp, two of the first tools that I tested in this research, since my evaluation licenses were no longer valid, and I couldn't update the tools to their latest version and scan again, and since I had 2-3 days until the end of my planned schedule, with a million tasks pending, I simply couldn't afford going through the evaluation request phase again, with all of my good intentions, and the willingness to sacrifice my spare time to ensure these tools will be properly represented.

Even though the results of some updated products (WebInspect and Nessus being the best examples) didn't change at all, even after I updated them to the latest version, who could say that the result would be the same for other vendors?

So, were the terms unfair to burp and cenzic?

Finally, several vendors sent me multiple versions and builds – they all wanted to succeed, a legitimate desire of any human being, even more so for a firm. Apart from the time each test took (a price I was willing to pay at the time), the new builds were sent even in the last day of the benchmark, and afterwards.

But if the new version is better, and more accurate, by limiting the amount of tests I perform for a given vendor, isn't that against what I'm trying to achieve in all my benchmarks, which is to release the benchmark with the most updated results, for all the tools?

(For example, Syhunt, a vendor that did very well in the last benchmark, sent me its final build (2.4.2.5) a day after the deadline, and included a time based SQL injection detection feature in that build, but since I couldn't afford the time anymore, I couldn't test the build, so, am I really reflecting the tool's current state in the most accurate manner? But if I would have tested this build, shouldn't I provide the rest of the vendors the same opportunity?)

One of the questions I believe I can answer – the accuracy question.

A benchmark is, in a very real sense, a competition, and since I take the scientific approach, I believe that the results are absolute, at least for the subject that is being tested. Since I'm not claiming that one tool is "better" than the other in every category, only at the tested criterion, I believe that priorities do not matter – as long as the test really reflects the current situation, the result is reliable.

I leave the interpretation of the results to the reader, at least until I'll cover enough aspects of the tools.

As for the rest of the open issues, I don't have good answers for all of those questions, and although I did my very best in this benchmark, and even exceeded what I thought I'm capable of, I will probably have to think of some solutions that will make the next benchmark terms equal, even for scanners that were tested in the beginning of the benchmark, and less time consuming then it has been.

11. Verifying The Benchmark Results

The results of the benchmark can be verified by replicating the scan methods described in the scan log of each scanner, and by testing the scanner against WAVSEP v1.0.3.

http://code.google.com/p/wavsep/

12. Notifications and Clarifications

How to use the results of the benchmark

The results of the benchmark clearly show how accurate each tool is in detecting the tested vulnerabilities (SQL Injection (MySQL ) & Reflected Cross Site Scripting), as long as it is able to locate and scan the vulnerable entry points. The results might even help to estimate how accurate each tool is in detecting related vulnerabilities (for example SQL Injection vulnerabilities which are based on other databases), and determine which exposure instances cannot be detected by certain tools;

However, currently, the results DO NOT evaluate the overall quality of the tool, since they don't include detailed information on the subjects such as crawling quality, technology support, scoping, profiling, stability in extreme cases, tolerance, detection accuracy of other exposures and so on... at least NOT YET.

I highly recommend reading the detailed results, and the appendix that deals with web application scanner evaluation, before getting to any conclusions.

Additional Notifications

During the benchmark, I have reported bugs that had a major affect on the detection accuracy to several commercial and open source vendors:

· A performance improvement feature in NTOSpider caused it not to scan many POST XSS test cases, and thus, the detection accuracy of RXSS POST test cases was significantly smaller then the RXSS GET detection accuracy. The vendor was notified on this issue, and provided me with a special build that overrides this feature (at least until they will have a feature in the GUI to disable this mechanism).

· A similar performance improvement feature in Netsparker caused the same issue, however, the feature could have been disabled in Netsparker, and thus, with the support of the relevant personal at Netsparker, I was able to work around the problem.

· A few bugs in arachni prevented the blind sql injection diff plugins from working properly. I notified the author, Tasos, on the issue, and he quickly fixed the issue and released the new version.

· Acunetix RXSS detection result was updated to match the results of the latest free version (one version above the tested commercial version) - Since the tested commercial version of Acunetix was older than the tested free version (20110608 vs 20110711), and since the results of the upgraded free version were actually better than the older commercial version I had tested, I changed the results of the commercial tool to match the ones of the new free version (from 22 to 24 in both the GET & POST RXSS detection scores).

· Changes in results from the previous benchmark might be attributed to enhanced scanning features, and/or to enhanced stability in the test environment & method (connection pool, limited & divided scope).

13. List of Tested Scanners

The following report contains the list of scanners tested in this benchmark, and provides information on the tested version, the tool's vendor/author and the current status of product:

http://sectooladdict-benchmarks.googlecode.com/files/List%20of%20Tested%20Web%20Application%20Vulnerability%20Scanners%20-%20WAVSEP%20Benchmark%202011.pdf

14. Source, License and Technical Details of Tested Scanners

The following report compares the licenses, development technology and sources (home page) of the various scanners:

http://sectooladdict-benchmarks.googlecode.com/files/Details%20of%20Tested%20Web%20Application%20Vulnerability%20Scanners%20-%20WAVSEP%20Benchmark%202011.pdf

15. Comparison of Active Vulnerability Detection Features

The following reports compare the active vulnerability detection features (audit features) of the various tested scanners:

First Report:

http://sectooladdict-benchmarks.googlecode.com/files/Application%20Active%20Scan%20Features%20Comparison%201of2%20-%20WAVSEP%20Benchmark%202011.pdf

Second Report:

http://sectooladdict-benchmarks.googlecode.com/files/Application%20Active%20Scan%20Features%20Comparison%202of2%20-%20WAVSEP%20Benchmark%202011.pdf

Aside from the Count column (which represents the total amount of audit features supported by the tool, not including complementary features such as web server scanning and passive analysis), each column in the report represents an audit feature. The description of each column is presented in the following glossary table:

Title	Description
SQL	Error Dependant SQL Injection
BSQL	Blind & Intentional Time Delay SQL Injection
RXSS	Reflected Cross Site Scripting
PXSS	Persistent / Stored Cross Site Scripting
DXSS	DOM XSS
Redirect	External Redirect / Phishing via Redirection
Bck	Backup File Detection
Auth	Authentication Bypass
CRLF	CRLF Injection / Response Splitting
LDAP	LDAP Injection
XPath	X-Path Injection
MX	MX / SMTP / IMAP Injection
Session Test	Session Identifier Complexity Analysis
SSI	Server Side Include
RFI-LFI	Directory Traversal / Remote File Include / Local File Include (Will be separated into different categories in future benchmarks)
Cmd	Command Injection / OS Command Injection
Buffer	Buffer Overflow
CSRF	Cross Site Request Forgery
A-Dos	Application Denial of Service / RegEx DoS
Privilege Escalation	Privilege Escalation Between Different Roles and User Accounts (Resources / Features)
Format String	Format String Injection
File Upload	File Upload / Insecure File Upload
Code Injection	Code Injection (ASP/JSP/PHP/Perl/etc)
XML Injection	XML / SOAP Injection
Source Code Disclosure	Source Code Disclosure Detection
Integer Overflow	Integer Overflow
Padding Oracle	Padding Oracle Detection / Exploitation
Session Fixation	Session Fixation

16. Comparison of Complementary Scanning Features

The following report compares complementary vulnerability detection features in the tested scanners:

http://sectooladdict-benchmarks.googlecode.com/files/Complementary%20Scan%20Features%20Comparison%20-%20WAVSEP%20Benchmark%202011.pdf

In order to clarify what each column in the report table means, use the following glossary table:

Title	Description
Web Server Hardening	Features that are able to detect Insecure HTTP method support (PUT, Trace, WebDAV), directory listing, robots and cross-domain files information disclosure, version specific vulnerabilities, etc.
CGI Scanning	Default files, common vulnerable applications, etc.
Passive Analysis	Security tests that don’t require any actual attacks, and are instead based on information gathering and analysis of responses, including certificate & cipher tests, content & metadata analysis, mime type analysis, autocomplete detection, insecure transmission of credentials, google hacking, etc.
File / Dir Enumeration	Directory and file enumeration features
Notes and Other Features	Uncommon or Unique features

17. Comparison of Usability and Coverage Features

The following report compares the usability, coverage and scan initiation features of the tested scanners:
http://sectooladdict-benchmarks.googlecode.com/files/List%20of%20Scanner%20Features%20(1%20of%203)%20-%20WAVSEP%20Benchmark%202011%20-%20Final3.pdf

In order to clarify what each column in the report table means, use the following glossary table:

Title	Possible Values
Configuration & Usage Scale	Very Simple - GUI + Wizard Simple - GUI with simple options, Command line with scan configuration file or simple options Complex - GUI with numerous options, Command line with multiple options Very Complex - Manual scanning feature dependencies, multiple configuration requirements
Stability Scale	Very Stable - Rarely crashes, Never gets stuck Stable - Rarely crashes, Gets stuck only in extreme scenarios Unstable - Crashes every once in a while, Freezes on a consistent basis Fragile – Freezes or Crashes on a consistent basis, Fails performing the operation in many cases
Performance Scale	Very Fast - Fast implementation with limited amount of scanning tasks Fast - Fast implementation with plenty of scanning tasks Slow - Slow implementation with limited amount of scanning tasks Very Slow - Slow implementation with plenty of scanning tasks

18. Comparison of Connection and Authentication Features

The following report compares the connection, authentication and scan control features of the tested scanners:
http://sectooladdict-benchmarks.googlecode.com/files/List%20of%20Scanner%20Features%20(2%20of%203)%20-%20WAVSEP%20Benchmark%202011%20-%20Final.pdf

19. Comparison of Advanced Features

The following report contains a comparison of advanced and uncommon scanner features:

http://sectooladdict-benchmarks.googlecode.com/files/List%20of%20Scanner%20Features%20%283%20of%203%29%20-%20WAVSEP%20Benchmark%202011.pdf

20. Detailed Results: Reflected XSS Detection Accuracy

The results of the Reflected Cross Site Scripting (RXSS) accuracy assessment are presented in the following report (the graphical results representation is provided in the beginning of the article):

http://sectooladdict-benchmarks.googlecode.com/files/Web%20Application%20Scanner%20RXSS%20Detection%20Accuracy%20-%20WAVSEP%20ScoreChart%202011.pdf

The results that were taken into account only include vulnerable pages linked from the index-xss.jsp index page (the RXSS-GET and/or RXSS-POST directories, in addition to the RXSS-FalsePositive directory). XSS Vulnerable entry points in the SQL injection vulnerable pages were not taken into account, since they don’t necessarily represent a unique scenario (or at least, not until the “layered vulnerabilities” scenario will be implemented).

21. Detailed Results: SQL Injection Detection Accuracy

The overall results of the SQL Injection accuracy assessment are presented in the following report (the graphical results representation is provided in the beginning of the article):

http://sectooladdict-benchmarks.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20-%20WAVSEP%20ScoreChart%202011.pdf

22. Drilldown – Error Based SQL Injection Detection

The results of the Error-Based SQL Injection benchmark are presented in the following report:

http://sectooladdict-benchmarks.googlecode.com/files/Web%20Application%20Scanner%20Error-Based%20SQLi%20Detection%20Accuracy%20-%20WAVSEP%20ScoreChart%202011.pdf

23. Drilldown – Blind & Time Based SQL Injection Detection

The results of the Blind & Time based SQL Injection benchmarks are presented in the following report:

http://sectooladdict-benchmarks.googlecode.com/files/Web%20Application%20Scanner%20Blind%20SQLi%20Detection%20Accuracy%20-%20WAVSEP%20ScoreChart%202011.pdf

24. Technical Benchmark Conclusions – Vendors & Users

While testing the various tools in this benchmark, I dealt with numerous difficulties, witnessed many inconsistent results and noticed that some tools had difficulties optimizing their scanning features on the tested platform. I had however, dealt with the other end of the spectrum, and used tools the easily overcome most of the difficulties related to detecting the tested vulnerabilities.

I'd like to share my conclusions, with the authors and vendors that are interested in improving their tools, and aren't offended by someone that's giving advice.

As far as detecting SQL injection exposures, I have noticed that tools that implemented the following features, detected more exposures, had less false positives, and provided consistent results:

· Time based SQL Injection detection vectors are very effective. They are, however, very tricky to use, since they might be affected by other attacks that are simultaneously executed, or affect the detection of other tests in the same manner. As a result, I recommended to all the authors & vendors to implement the following behavior in their product: execute time based attacks at the end of the scanning process, after all the rest of the tests are done, while using a reduced number of concurrent connections. Executing other tests in parallel might have a negative effect on the detection accuracy.

· Since the upper/lower timeout values used to determine whether or not a time based exploit was successful may change due to various circumstances, I recommend calculating and re-calculating this value during the scan, and revalidating each time based result independently, after verifying that the timeout values are "normal".

· Implement various payloads of time based attacks – the sleep method is not enough to cover all the databases, and not even all the versions of mysql.

25. So What Now?

So now that we have all those statistics, it's time to analyze them properly, and see which conclusions we can get to. Since this process will take time, I have to set some priorities;

In the near future, I will try to achieve the following goals:

· Find a better way to present the vast amount of information on web application scanners features & accuracy. I have been struggling with this issue for almost 2 years, but I think that I finally found a solution that will make the information more useful for the common reader… stay tuned for updates.

· Provide recommendations for the best current method of executing free & open source web application scanners; the most useful combinations, and the tiny tweaks required to achieve the best results.

· Release the new test case categories of WAVSEP that I have been working on. Yep, help needed.

In addition to the short term goals, the following long term goals will still have a high priority:

· Improve the testing framework (WAVSEP); add additional test cases and additional security vulnerabilities.

· Perform additional benchmarks on the framework, and on a consistent basis. I previously aimed for one major benchmark per year, but that formula might completely change, if I'll manage to work a few issues around a new initiative I have in this field.

· Integration with external frameworks for assessing crawling capabilities, technology support, etc.

· Publish the results of tests against sample vulnerable web applications, so that some sort of feedback on other types of exposures will be available (until other types of vulnerabilities will be implemented in the framework), as well as features such as authentication support, crawling, etc.

· Gradually develop a framework for testing additional related features, such as authentication support, malformed HTML tolerance, abnormal response support, etc.

I hope that this content will help the various vendors improve their tools, help pen-testers choose the right tool for each task, and in addition, help create some method of testing the numerous tools out there.

Since I have already been in the situation in the past, then I know what's coming… so I apologize in advance for any delays in my responses in the next few weeks.

The following resources include additional information on previous benchmarks, comparisons and assessments in the field of web application vulnerability scanners:

· "Webapp Scanner Review: Acunetix versus Netsparker", by Mark Baldwin (commercial scanner comparison, April 2011)

· "Effectiveness of Automated Application Penetration Testing Tools", by Alexandre Miguel Ferreira and Harald Kleppe (commercial & freeware scanner comparison, February 2011)

· "Web Application Scanners Accuracy Assessment", the predecessor of the current benchmark, by Shay Chen (a comparison of 43 free & open source scanners, December 2010)

· "State of the Art: Automated Black-Box Web Application Vulnerability Testing" (Original Paper), by Jason Bau, Elie Bursztein, Divij Gupta, John Mitchell (May 2010) – original paper

· "Analyzing the Accuracy and Time Costs of Web Application Security Scanners", by Larry Suto (commercial scanners comparison, February 2010)

· "Why Johnny Can’t Pentest: An Analysis of Black-box Web Vulnerability Scanners", by Adam Doup´e, Marco Cova, Giovanni Vigna (commercial & open source scanner comparison, 2010)

· "Web Vulnerability Scanner Evaluation", by AnantaSec (commercial scanner comparison, January 2009)

· "Analyzing the Effectiveness and Coverage of Web Application Security Scanners", by Larry Suto (commercial scanners comparison, October 2007)

· "Rolling Review: Web App Scanners Still Have Trouble with Ajax", by Jordan Wiens (commercial scanners comparison, October 2007)

· "Web Application Vulnerability Scanners – a Benchmark" , by Andreas Wiegenstein, Frederik Weidemann, Dr. Markus Schumacher, Sebastian Schinzel (Anonymous scanners comparison, October 2006)

27. Thank-You Note

During the research described in this article, I have received help from quite a few individuals and resources, and I’d like to take the opportunity to thank them all.

For all the open source tool authors that assisted me in testing the various tools in unreasonable late night hours, for the kind souls that helped me obtain evaluation licenses for commercial products, for the QA, Support and Development teams of commercial vendors, which saved me tons of time and helped me overcome obstacles, and for the various individuals that helped me contact these vendors.

I would also like to continue my tradition, and thank all the information sources that helped me gather the list of scanners over the years, including (but not limited to) information security sources such as PenTestIT (http://www.pentestit.com/), Security Sh3ll (http://security-sh3ll.blogspot.com/), NETpeas Toolswatch Service (http://www.vulnerabilitydatabase.com/toolswatch/), Darknet (http://www.darknet.org.uk/), Packet Storm (http://packetstormsecurity.org/), Help Net Security (http://www.net-security.org/), Astalavista (http://www.astalavista.com/), Google (of course) and many others.

I hope that the conclusions, ideas, information and payloads presented in this research (and the benchmarks and tools that will follow) will be for the benefit of all vendors, open source community projects and commercial vendors alike.

28. Frequently Asked Questions

Q: 60 web application scanners is an awful lot, how many scanners exist?

A: Assuming you are using the same definition for a scanner that I do, then I'm currently aware of 95 web application scanners that can claim to support the detection of generic application level exposures, in a safe an controllable manner, and in multiple URLs (48 free & open source scanners that were tested, 12 commercial scanners that were tested, 25 open source scanners that I didn't test yet, and 10 commercial scanners that slipped my grip). And yes, I'm planning on testing them all.

Q: Why RXSS and SQLi again? Will the benchmarks ever include additional exposures?

A: Yes, they will. In fact, I'm already working on test case categories of two different exposures, and will use them both for my next research. Besides, the last benchmark focused on free & open source products, and I couldn't help myself, I had to test them against each other.

Q: I can't wait for the next research, what can I do to speed things up?

A: I'm currently looking for methods to speed up the processes related to these researches, so if you're willing to help, contact me.

Q: What’s with the titles that contain cheesy movie quotes?

A: That's just it - I happen to like cheese. Let's see you coming up with better titles at 4AM.

29. Appendix A – Assessing Web Application Scanners

Although this benchmark contains tons of information, and is very useful as a decision assisting tool, the content within it cannot be used to calculate the accurate ROI (return of investment) of each web application scanner. Furthermore, it can't predict on its own exactly how good will the results of each scanner be in every situation (but it can predict what won't be detected), since there are additional factors that need to be taken into account.

The results in this benchmark could serve as an accurate evaluation formula only if the scanner will be used to scan a technology that it supports, pages that it can detect (manual crawling features can be used to overcome many obstacles in this case), and locations without technological barriers that it cannot handle (for example, web application firewalls or anti-CSRF tokens).

In order for us to truly assess the full capability of web application vulnerability scanners, the following features must be tested:

· The entry point coverage of the web application scanner must be as high as possible; meaning, the tool must be able to locate and properly activate (or be manually "taught") all the application entry points (e.g. static & dynamic pages, in-page events, services, filters, etc). Vulnerabilities in an entry point that wasn't located will not be detected. The WIVET project can provide additional information on coverage and support.

· The attack vector coverage of the web application scanner – does it support input vectors such as GET / POST / Cookie parameters? HTTP headers? Parameter Names? Ajax Parameters? Serialized Objects? Each input vector that is not supported means exposures that won't be detected, regardless of the tool's accuracy level (assuming the unsupported attack/input vector is vulnerable).

· The scanner must be able to handle the technological barriers implemented in the application, ranging from authentication mechanism to automated access prevention mechanisms such as CAPTCHAs and anti-CSRF tokens.

· The scanner must be able to handle any application specific problems it encounters, including malformed HTML (tolerance), stability issues and other limitations. If the best scanner in the world will consistently cause the application to crash in a couple of seconds, then it's not useful for assessing the security of that application (in matters that don't relate to DoS attacks).

· The number of features (active & passive) implemented in the web application vulnerability scanner.

· The accuracy level of each and every plugin supported by the web application vulnerability scanner.

That being said, it's crucial to remember that even in the most ideal scenario, with the absence of human intelligence, scanners can't detect all the instances of exposures that are truly logical – meaning, are related to specific business logic, and thus, are not perceived as an issue by an entity that can't understand the business logic.

But the sheer complexity of the issue does not mean that we shouldn't start somewhere, and that's exactly what I'm trying to do in my benchmarks – create a scientific, accurate foundation for obtaining that goal, with enough investment, over time.

Note that my explanations describe only a portion of the actual tests that should be performed, and I'm sharing them only to emphasize the true complexity of the core issue; I haven't touched stability, bugs, and a lot of other subjects, which may affect the overall result you get.

Additional information on evaluation standards for web application vulnerability scanners can be found in the WASC Web Application Security Scanner Evaluation Criteria web site.

30. Appendix B – A List of Tools Not Included In the Test

The following commercial web application vulnerability scanners were not included in the benchmark, since I didn't manage to get an evaluation version until the article publication deadline, or in the case of one scanner (mcafee), had problems with the evaluation version that I didn't manage to work out until the benchmark's deadline:

Commercial Scanners not included in this benchmark

· N-Stalker Commercial Edition (N-Stalker)

· McAfee Vulnerability Manager (McAfee / Foundstone)

· NeXpose Enterprise Edition Web Application Scanning Features (Rapid7)

· Retina Web Application Scanner (eEye Digital Security)

· WebApp360 (NCircle)

· Core Impact Pro Web Application Scanning Features (Core Impact)

· Parasoft Web Application Scanning Features (a.k.a WebKing, by Parasoft)

· MatriXay Web Application Scanner (DBAppSecurity)

· Falcove (BuyServers ltd, currently Unmaintained)

· Safe3WVS 9.2 Commercial Edition (Safe3 Network Center)

The following open source web application vulnerability scanners were not included in the benchmark, mainly due to time restrictions, but will be included in future benchmarks:

Open Source Scanners not included in this benchmark

· Rabbit VS

· Spacemonkey

· Kayra

· 2gwvs

· Webarmy

· springenwerk

· Mopset 2

· XSSFuzz 1.1

· Witchxtoolv

· PHP-Injector

· XSS Assistant

· Fiddler XSSInspector/XSRFInspector Plugins

· GNUCitizen JAVASCRIPT XSS SCANNER - since WebSecurify, a more advanced tool from the same vendor is already tested in the benchmark.

· Vulnerability Scanner 1.0 (by cmiN, RST) - since the source code contained traces for remotely downloaded RFI lists from locations that do not exist anymore.

As a result, the test did not include the following types of tools:

· Scanners without RXSS / SQLi detection features:

o Dominator (Firefox Plugin)

o fimap

o lfimap

o phpBB-RFI Scanner

o DotDotPawn

o LFI (Library-level Fault Injector)

o lfi-scanner

o LFI-Scanner

o lfi-rfi2

o LFI/RFI Checker (astalavista)

o CSRF Tester

o etc

· Passive Scanners (response analysis without verification):

o Watcher (Fiddler Plugin by Casaba Security)

o Skavanger (OWASP)

o Pantera (OWASP)

o Ratproxy (Google)

o CAT The Manual Application Proxy (Context)

o etc

· Scanners of specific products or services (CMS scanners, Web Services Scanners, etc):

o WSDigger

o Sprajax

o ScanAjax

o Joomscan

o wpscan

o Joomlascan

o Joomsq

o WPSqli

o etc

· Web Application Scanning Tools which are using Dynamic Runtime Analysis:

o PuzlBox (the free version was removed from the web site, and is now sold as a commercial product named PHP Vulnerability Hunter)

o Inspathx

o etc

o Darkjumper 5.8 (scans additional external hosts that are linked to the given tested host)

o Bako's SQL Injection Scanner 2.2 (only tests sites from a google dork)

o Serverchk (only tests sites from a google dork)

o XSS Scanner by Xylitol (only tests sites from a google dork)

o Hexjector by hkhexon – also falls into other categories

o d0rk3r by b4ltazar

o etc

· Deprecated Scanners - incomplete tools that were not maintained for a very long time. This list currently includes the following tools (and might include more):

o etc

o Lilith 0.4c/0.6a (both versions 0.4c and 0.6a were tested, and although the tool seems to be a scanner at first glimpse, it doesn’t perform any intelligent analysis on the results).

· CGI Scanners: vulnerability scanners that focus on detecting hardening flaws and version specific hazards in web infrastructures (Nikto, Wikto, WHCC, st4lk3r, N-Stealth, etc)

· Single URL Vulnerability Scanners - scanners that can only scan one URL at a time, or can only scan information from a google dork (uncontrollable).

o Havij (by itsecteam.com)

o Hexjector (by hkhexon)

o Simple XSS Fuzzer [SiXFu] (by www.EvilFingers.com)

o Mysqloit (by muhaimindz)

o PHP Fuzzer (by RoMeO from DarkMindZ)

o SQLi-Scanner (by Valentin Hoebel)

o Etc.

· Vulnerability Detection Assisting Tools – tools that aid in discovering a vulnerability, but do not detect the vulnerability themselves; for example:

o Exploit-Me Suite (XSS-Me, SQL Inject-Me, Access-Me)

o Fiddler X5s plugin

o XSSRays (chrome Addon)

· Exploiters - tools that can exploit vulnerabilities but have no independent ability to automatically detect vulnerabilities on a large scale. Examples:

o MultiInjector

o XSS-Proxy-Scanner

o Pangolin

o FGInjector

o Absinth

o Safe3 SQL Injector (an exploitation tool with scanning features (pentest mode) that are not available in the free version).

o etc

· Exceptional Cases

31. Appendix C – WAVSEP Scan Logs

The execution logs, installation steps and configuration used while scanning with the various tools are all described in the following report:

http://sectooladdict-benchmarks.googlecode.com/files/Scan%20Log%20-%20WAVSEP%20Benchmark%202011.pdf

32. Appendix D – Scanners with Abnormal Behavior

The following appendix was published in my previous benchmark, but I decided to include in the current benchmark, mainly because I didn't manage to invest the time to get to the bottom of these mysteries, and didn't see any information on someone else that did.

During the current & previous assessment, parts of the source code of open source scanners and the HTTP communication of some of the scanners was analyzed; some tools behaved in an abnormal manner that should be reported:

· Priamos IP Address Lookup – The tool Priamos attempts to access “whatismyip.com” (or some similar site) whenever a scan is initiated (verified by channeling the communication through Burp proxy). This behavior might derive from a trojan horse that infected the content on the project web site, so I’m not jumping to any conclusions just yet.

· VulnerabilityScanner Remote RFI List Retrieval (listed in the scanners that were not tested, appendix A, developed by a group called RST, http://pastebin.com/f3c267935) – In the source code of the tool VulnerabilityScanner (a python script), I found traces for remote access to external web sites for obtaining RFI lists (might be used to refer the user to external URLs listed in the list). I could not verify the purpose of this feature since I didn’t manage to activate the tool (yet); in theory, this could be a legitimate list update feature, but since all the lists the tool uses are hardcoded, I didn’t understand the purpose of the feature. Again, I’m not jumping to any conclusions; this feature might be related to the tool’s initial design, which was not fully implemented due to various considerations.

Although I did not verify that any of these features is malicious in nature, these features and behaviors might be abused to compromise the security of the tester’s workstation (or to incriminate him in malicious actions), and thus, require additional investigation to disqualify this possibility.

Myth Breaker - The Best Open Source Web Application Vulnerability Scanner

2011-01-25T14:00:00.000-08:00

(The original benchmark post - comparison of 43 web application vulnerability scanners:

http://sectooladdict.blogspot.com/2010/12/web-application-scanner-benchmark.html)

It’s been a couple of weeks since the initial benchmark was published, and I used that time to contact most of the vendors and to come to some conclusions, as to which tool combinations are ideal for each task;

I believe that those of you that use these tools on a daily basis will find my conclusions interesting.

Please note that the conclusions refer to the condition of the tools in the day the benchmark was released (see the full explanation at the end of the post).

Glossary

AND – combining the tools is required to obtain the best results.

OR – using either one of the tools will provide nearly identical results.

AND/OR – it is currently unknown if combining them will provide additional benefits.

SAFE scan – a scan method in which the tester can select which URLs to scan, in order to prevent the scanner from accessing links that could delete data, lock user accounts or cause any other unintentional hazard (generally requires the scanner to have a proxy/manual crawling/URL file parsing/pre-configured URL restriction module); Recommended while scanning the internal section of an application that resides in a production environment.

UNSAFE scan – a scan method that scans all the URLs, without any restrictions or limitations; Recommended while scanning the public section of an application, and for scanning the internal section of an application that resides in the testing/development environment.

The Ideal Combination of Tools (Relevant to the release date of the initial benchmark – 26/12/2010):

(Constructed according to the cases detected by each tool, and according to tool capabilities and application scope restrictions)

Scan Type & Target

Reflected XSS

SQL Injection (MySQL)

Initial Public Scan

Initial Scan on the Application’s Public (unauthenticated) Section

(Purpose: gather as many “Low Hanging Fruit” exposures as possible with a minimal amount of false positives)

Netsparker AND Acunetix AND N-Stalker AND SkipFish

(Nearly False Positive Free Combination)

ProxyStrike AND WebCruiser

(Nearly False Positive Free Combination)

Internal Scan - Unsafe

The Application’s Internal (authenticated) Section

Netsparker AND Acunetix AND SkipFish

(Nearly False Positive Free Combination)

Wapiti

(Verification with other tools is recommended to reduce False Positives – ProxyStrike AND WebCruiser, In addition to one of the following: W3AF/Andipaors/ZAP/

Netsparker/Sandcat/

Oedipus)

Internal Scan - Safe

The Application’s Internal (authenticated) Section

(Method: scan internal application pages without activating any delete, logout or other dangerous operations).

ZAP AND W3AF

(Safe combination with relatively efficient accuracy)

W3AF AND Andiparos/Paros AND Oedipus AND ProxyStrike

Additional Public Scan

Detect additional potential exposures that require manual verification, and aren’t covered by previous tools

(Public Section)

ProxyStrike OR Sandcat (Grabber detects 1-2 additional POST cases - optional)

Wapiti

2nd Internal – Unsafe

Detect additional potential exposures that require manual verification, and aren’t covered by previous tools

ProxyStrike OR Sandcat

Wapiti

(No substantial change, so there’s no need to run another scan)

2nd Internal – Safe

Detect additional potential exposures that require manual verification, and aren’t covered by previous tools

(Method: scan internal application pages for additional exposure instances without activating any delete, logout or other dangerous operations)

ProxyStrike

W3AF AND Andiparos/Paros AND Oedipus AND ProxyStrike

(No substantial change, so there’s no need to run another scan)

Complementary Scan for Additional Exposures

Complementary Scan

Scan the applications with scanners that have a wider range of features, to cover additional security flaws

W3AF AND/OR Arachni AND/OR Skipfish AND/OR Sandcat

Notable Open Source & Freeware Tools – SQL Injection Detection

The highest SQLi detection ratio of open source & freeware tools belongs to Wapiti, currently the undisputed winner in this category.

A bit behind Wapiti were AndiParos, Zapproxy and Paros Proxy (all forks of the original Paros project), followed closely by Netsparker and W3AF (two tools that were prone to less false positives test cases, compared to all of the tools described so far - 30% compared to 40% or 50%).

* it is important to mention that Netsparker CE 1.5 does not contain Netsparker’s Blind-SQL injection module (disabled in this version), only the regular SQL-Injection module and the Boolean SQL-Injection module.

However, we cannot ignore the fact that the following tools had pretty decent accuracy with 0 false positives(!): WebCruiser (55.88%) and ProxyStrike (52.21%), making them ideal tools for an initial scan (Mini MySqlat0r and Scrawler had 0 false positives as well, but with lower accuracy).

Notable Open Source & Freeware Tools – XSS Detection

The Highest XSS detection ratio belongs to Sandcat, which detected nearly 100% of the overall test-cases (although like ProxyStrike & Grabber, it was misled by a few extra false positive test cases).

The highest XSS detection ratio of open source tools (and 2nd best in total) belongs to ProxyStrike (Grabber detected more POST test cases, but had a higher false positive ratio, and did not detect GET cases).

The best overall XSS detection ratio (while considering the low amount of false positives) belongs to Netsparker CE (63.64% and 3rd in the efficiency order, right after ProxyStrike), followed closely by N-Stalker and by Acunetix FE (and since Skipfish and these tools “complete” missing test cases in each other, they are ideal for initial scans, since they all have 0 false positives!).

The best overall XSS detection ratio (while considering the low amount of false positives) of open source tools belongs to WebSecurify.

The best HTTP GET XSS detection ratio (while considering the low amount of false positives) of open source tools belongs to XSSer.

The following open source tools had XSS detection modules that were free of false positives (while still having a relatively efficient detection ratio) – Grendel-Scan (GET) and Skipfish (Secubat had 0 false positives as well, but its detection ratio was a bit lower).

Notes

When using ProxyStrike for the initial scan, It’s probably best to use an external spider instead of the built in spider (e.g. use ProxyStrike as an outgoing/upstream proxy for Burp Suite FE or Paros/ZAP/Andiparos and then use the spider feature of the external tool through ProxyStrike).

As mentioned before, the conclusions reflect the condition of the various tools in the date the initial benchmark was published. Since the benchmark, many vendors had released new versions (some even in response to the benchmark), so the list of conclusions will change as soon as the next benchmark is released; I know for a fact that some vendors invested so much effort in improving their detection modules that some of the new versions get to nearly 100% detection ratio (but since I don’t have updated statistics, well have to wait).

Conclusions

So… it seems that I didn't find “the best web application vulnerability scanner” after all… but I did find combinations of open source & freeware tools that get pretty good results.

As I mentioned in previous posts, my work is only beginning.

Various open source vendors already released new versions that should be tested, tools that were improperly executed (or had a bug) should be retested as soon as their issues are mitigated, additional research led me to discover a couple of additional open source web application scanner projects, and at least one new open source web application scanner was released in the last couple of weeks (and I haven’t even mentioned commercial scanners).

Time to get back to work…

Follow Up & Clarifications

2011-01-11T14:51:00.000-08:00

I’ve been pretty busy trying to contact the various vendors and deliver materials that they can use for QA & development, and I must mention that so far every vendor / developer that I have contacted responded kindly, and many of them responded with excitement and already started enhancing their tool (which is GREAT news for all of us).

I managed to find the time to contact about 18 vendors, and hopefully I’ll manage to contact more in the following weeks (25 left to go). This process requires me to analyze the benefits of the tools of each vendor, and as a result, is more time consuming then I originally thought; however, thanks to this process, I believe that soon it will lead me to some additional interesting conclusions and insights, which I’ll publish separately.

In the process of contacting the vendors, I realized that I have neglected some of my duties and forgot to publish some important clarifications:

Although the test cases implemented as “False Positives” are by no means vulnerable to SQL Injection or Cross Site Scripting, some of the test cases still fall into a category of information that should be presented in the report under the context of another type of exposure:

o Pages that disclose sensitive information / exceptions (some SQL Injection False Positive test cases that are meant to simulate SQL errors that do not derive from user originating input, such as connection failures, etc).

o Pages that fall under the category of insecure coding practices (some of the False RXSS & SQLi pages).

Some tools are still in early beta, and some didn’t even publish an official alpha version (aidSQL, iScan, and some of the other tools that had zero accuracy); the accuracy of these tools was not really audited, due to limitations or bugs that will surely be mitigated in the future versions. The benchmark will be updated as soon as the tool vendors release a new stable version.
The execution of certain tools which were reported as having zero accuracy failed due to bugs or configuration flaws, and not accuracy related issues; These tools include SQLMap, aidSQL, VulnDetector, and a couple of more; I’m currently working with the various vendors to figure out how to execute them properly (or how to work around the specific bugs), so the test will actually reflect their accuracy level.

As a result, I believe that the next benchmark is going to be performed sooner then I planned;

It will probably include the same results alongside the corrected scans of the tools that had execution issues (particularly SQL tools), and maybe additional enhancements (under discussion).

I wish you all a Happy New Year :)

Web Application Scanner Benchmark (v1.0)

2010-12-26T03:26:00.000-08:00

Well, it’s finally done. What I originally thought will only take me a couple of days, and found myself doing for the past 9 months is finally ready for release, and it’s titled:

Web Application Scanners Accuracy Assessment

Freeware & Open Source Scanners

Comparison & Assessment of 43 Free & Open Source Black Box Web Application Vulnerability Scanners

By Shay Chen

Information Security Consultant, Researcher and Instructor

http://sectooladdict.blogspot.com/

sectooladdict -$at$- gmail -$dot$- com

December 2010

Assessment Environment: WAVSEP 1.0 (http://code.google.com/p/wavsep/)

Introduction

I’ve been collecting them for years, trying to get my hands on anything that was released within the genre. It started as a necessity, transformed into a hobby, and eventually turned into a relatively huge collection… But that’s when the problems started.

While back in 2005 I could barely find freeware web application scanners, by 2008 I had SO MANY of them that I couldn’t decide which ones to use. By 2010 the collection became so big that I came to the realization that I HAVE to choose.

I started searching for benchmarks in the field, but at the time, only located benchmarks the focused on comparing commercial web application scanners (with the exception of one benchmark that also covered 3 open source web application scanners), leaving the freeware & open source scanners in an uncharted territory;

http://www.virtualforge.de/index.php/en/library/white-papers/web-application-vulnerability-scanners-a-benchmark_en.html (Anonymous scanners)
http://anantasec.blogspot.com/2009/01/web-vulnerability-scanners-comparison.html (commercial scanners)
http://www.cs.ucsb.edu/~adoupe/static/black-box-scanners-dimva2010.pdf (mostly commercial, but including W3AF, paros and grendel-scan)
http://ha.ckers.org/files/Accuracy_and_Time_Costs_of_Web_App_Scanners.pdf (commercial scanners)

By 2010 I had over 50 tools, so I eventually decided to test them myself using the same model used in previous benchmarks (a big BIG mistake).

I initially tested the various tools against a vulnerable ASP.net web application and came to conclusions as to which tool is the “best”… and if it weren’t for my curiosity, that probably would have been the end of it and my conclusions might have mislead many more.

I decided to test the tools against another vulnerable web application, just to make sure the results were consistent, and arbitrarily selected “Insecure Web App” (a vulnerable JEE web application) as the second target… and to my surprise, the results of the tests against it were VERY different.

Some of the Tools that were efficient in the test against the vulnerable ASP.net application (which will stay anonymous for the time being) didn’t function very well and missed many exposures, while some of the tools that I previously classified as “useless” detected exposures that NONE of the other tools found.

After performing an in-depth analysis for the different vulnerabilities in the tested applications, I came to the conclusion that although the applications included a similar classification of exposures (SQL Injection, RXSS, Information disclosure, etc), the properties and restrictions in the exposure instances were VERY different in each application.

That’s when it dawned on me that the different methods that tools use to discover security exposures might be efficient for detecting certain common instances of a vulnerability while simultaneously being inefficient for detecting other instances of the same vulnerability, and that tools with “lesser” algorithms or different approaches (which might appear to be less effective at first) might be able to fill the gap.

So the question remains… Which tool is the best? Is there one that surpasses the others? Can there be only one?

I decided to find out…

It started as a bunch of test cases, and ended as a project containing hundreds of scenarios (currently focusing on Reflected XSS and SQL Injection) that will hopefully help in unveiling the mystery.

(A PDF version of this benchmark will be available shortly in the WAVSEP project home page at http://code.google.com/p/wavsep/)

Thank-You Note

Before I’ll describe project WAVSEP and the results of the first scanner benchmark performed using it, I’d like to thank all the tool developers and vendors that shared freeware & open source tools with the community over the years; if it weren’t for the long hours they’ve invested and the generosity they had to share their creations, then my job (and that of others in my profession) would have been much harder.

I’d like to express my sincere gratitude for Shimi Volkovich (http://il.linkedin.com/pub/shimi-volkovich/20/173/263), for taking the time to design the logo I’ll soon be using.

I would also like to thank all the sources that helped me gather the list of scanners over the years, including (but not limited to) information security sources such as PenTestIT (http://www.pentestit.com/), Security Sh3ll (http://security-sh3ll.blogspot.com/), Security Database (http://www.security-database.com/), Darknet (http://www.darknet.org.uk/), Packet Storm (http://packetstormsecurity.org/), Help Net Security (http://www.net-security.org/), Astalavista (http://www.astalavista.com/), Google (of course) and many others that I have neglected to mention due to my failing memory.

I hope that the conclusions, ideas, information and payloads presented in this research (and the benchmarks and tools that will follow) will benefit all vendors, and specifically help the open source community to locate code sections that all tool vendors could assimilate to improve their products; to that end I’ll try and contact each vendor in the next few weeks, in order to notify them on source codes that could be assimilated in their product to make it even better (on the basis of development technology and the license of each code section).

Phase I – The “Traditional” Benchmark

Testing the scanners against vulnerable training & real life applications.

As I mentioned earlier, In the initial phase of the benchmark, I have tested the various scanners in front of different vulnerable “training” applications (OWASP InsecureWebApp, a vulnerable .Net Application and a simple vulnerable application I have written myself), and tested many of them against real life applications (ASP.Net applications, Java applications based on Spring, Web application written in PHP, etc).

I decided not to publish the results just yet, and for a damn good reason which I did not predict in the first place; nevertheless, the initial process was very helpful because it helped me to learn about the different aspects of the tools: features, vulnerability list, coverage, installation processes, configuration methods, usage, adaptability, stability, performance and a bunch of other aspects.

I have found VERY interesting results that prove that certain old scanners might provide great benefits in many cases that many modern projects will not handle properly.

The process also enabled me to verify the support of the various tools in their proclaimed features (which I have literally done for the vast majority of the tools, using proxies, sniffers and other experiments), and even get a general measure of their accuracy and capabilities.

However, after seeing the results diversity in different applications and technologies, and after dealing with the countless challenges that came along the way, I have discovered several limitations and even a fundamental flaw in testing the accuracy, coverage, stability and performance of scanners in this manner (I have managed to test around 50 free and open source scanners by this point, as insane and unbelievable as this number might sound);

We may be able to estimate the general capabilities of a scanner from the amount of REAL exposures that it located, the amount of exposures that it missed (false negatives) and from the amount of FALSE exposures (false positives) it identified as security exposures, BUT on the other hand, the output of such a process will very much depend on the type of exposures that exist in the tested application, how much each scanner is adapted to the tested application technology and which private cases of exposures and barriers exist in the tested application.

A scanner that will be very useful for scanning PHP web sites might completely fail the task of scanning a ASP.Net web application, and a tool perfectly suited for that task might crash when faced with certain application behaviors, or be useless in detecting a private case of a specific vulnerability that is not supported by the tool.

I guess what I’m trying to say is this:

There are many forms and variations to each security exposure, and in order to prove my point, I’ll use the example of reflected cross site scripting;

Locations vulnerable to reflected cross site scripting might appear in many forms; they may require the attacker to send a whole HTML tag as a part of the crafted link, require the injection of an HTML event (in case the input-affected-output is printed in the context of a tag and the usage of tag-composing-characters is restricted), they may appear in locations vulnerable to SQL injection (and thus restrict the use of certain characters, or even require the usage of initial payloads that “disable” the SQL injection vulnerability first), require browser specific payloads or even direct injection of javascript/vbscript (in case the context is within a script tag, certain HTML events or even in the context of certain properties), and these cases are only a fragment of the whole list!

So, how can the tester know which of these cases is handled by each scanner from the figures and numbers presented in a general benchmark?

I believe he can’t. No matter how solid the difference appears, he really can’t.

Such information may allow him to root out useless tools (tools that miss even the most obvious exposures), and even identify what appears to be a significant difference in the accuracy of locating certain exposure instances, but the latter case might have been very different if the tested applications would have been prone to certain exposure instances that are the specialty of a different scanner, or would have included a technological barrier that requires a specific feature or behavior to bypass.

Thus, I have come to believe that the only way I could truly provide useful information to testers on the accuracy and coverage of freely available web application scanners is by writing detailed test cases for different exposures, starting with some core common exposures such as SQL Injection, cross site scripting and maybe a couple of others.

And thus, I have ended up investing countless nights in the development of a new test-case based evaluation application, designed specifically to test the support of each tool for detecting MANY different cases of certain common exposures.

The results of the original benchmark (against the vulnerable training web applications) will be published separately in a different article (since by now, many of them have been updated, and the results require modifications).

Phase II - Project WAVSEP

After documenting and testing the features of every free & open source web application scanner and scan script that I could get my hands on, I discovered that the most common features were Reflected Cross Site Scripting (RXSS) and SQL Injection (SQLi). I decided to focus my initial efforts on these two vulnerabilities, and develop a platform that could truly evaluate how good each scanner is in detecting them, which tool combinations provide the best results and which tool can bypass the largest amount of detection barriers.

Project WAVSEP (Web Application Vulnerability Scanner Evaluation Project) was implemented as a set of vulnerable JSP pages; each page implementing a unique test case.

A test case is defined as a unique combination of the following elements:

A certain instance of a given vulnerability.
Attack vectors with certain input origins (either GET or POST values, and in the future, also URL/path, cookie, various headers, file upload content and other origins).

Currently, only GET and POST attack vectors are covered, since most scanners support only GET and POST vectors (future versions of WAVSEP will include support for additional databases, additional response types, additional detection barriers, additional attack vector origins and additional vulnerabilities).

Project WAVSEP currently consists of the following test cases:

64 Reflected XSS test cases (32 GET cases, 32 POST cases -> 66 total vulnerabilities)
130 SQL Injection test cases, most of them implemented for MySQL & MSSQL (65 GET cases, 65 POST Vases -> 136 total vulnerabilities)

o The list of test cases includes vulnerable pages that respond with 500 HTTP errors, 200 HTTP Responses with erroneous text, 200 HTTP Responses with differentiation or completely identical 200 HTTP responses.

o 80 out of 136 cases are simple SQL injection test cases (500 & 200 erroneous HTTP responses), and 56 are Blind SQL Injection test cases (valid and identical 200 HTTP responses).

7 different categories of false positive Reflected XSS vulnerabilities (GET OR POST).
10 different categories of false positive SQL Injection vulnerabilities (GET OR POST).

Each exposure category in WAVSEP contains an index page with descriptions of different barriers in test cases, structures of a sample detection payloads and examples of such payloads.

A general description of each test case is also available in the following excel spreadsheet: http://code.google.com/p/wavsep/downloads/detail?name=VulnerabilityTestCases.xlsx&can=2&q=

Those that wish to verify the results of the benchmark can download the latest source code of project WAVSEP (including the list of test cases and their description) from the project’s web site:

http://code.google.com/p/wavsep/

Benchmark Overview

As mentioned before, the benchmark focused on testing free & open source tools that are able to detect (and not necessarily exploit) security vulnerabilities on a wide range of URLs, and thus, each tool tested needed to support the following features:

Either open source or free to use, so that open source projects and vendors generous enough to contribute to the community will benefit from the benchmark first.
The ability to detect Reflected XSS and/or SQL Injection vulnerabilities.
The ability to scan multiple URLs at once (using either a crawler/spider feature, URL/Log file parsing feature or a built-in proxy).
The ability to control and limit the scan to internal or external host (domain/IP).

As a direct implication, the test did NOT include the tools listed in Appendix A – A List of Tools Not Included In The Test.

The purpose of WAVSEP’s test cases is to provide a scale for understanding which detection barriers each scanning tool can bypass, and which vulnerability variations can be detected by each tool.

The Reflected Cross Site Scripting vulnerable pages are pretty standard & straightforward, and should provide reliable basis for assessing the detection capabilities of different scanners.

However, it is important to remember that the SQL Injection vulnerable pages used a MySQL database as a data repository, and thus, the SQL Injection detection results only reflect detection results of SQL Injection vulnerabilities in this type of database; the results that might vary when the back end data repository will be different (a theory that will be verified in the next benchmark).

Description of Comparison Tables

The list of tools tested in this benchmark is organized within the following reports:

List of Tested Scanners (http://wavsep.googlecode.com/files/List%20of%20Tested%20Web%20Application%20Vulnerability%20Scanners%20v1.0.pdf)

Source, License and Technical Details of Tested Scanners (http://wavsep.googlecode.com/files/Details%20of%20Tested%20Web%20Application%20Vulnerability%20Scanners%20v1.0.pdf)

For those of you that wish to get straight to the point, the results of the accuracy assessment are organized within the following reports:

Benchmark Results – Reflected XSS Detection Accuracy (http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20RXSS%20Detection%20Accuracy%20v1.0.pdf)

Benchmark Results – SQL Injection Detection Accuracy – Total (http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20v1.0.pdf)

Benchmark Drilldown – Blind SQL Injection Detection (http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20-%20Blind%20v1.0.pdf)

Benchmark Drilldown – Erroneous SQL Injection Detection

(http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20-%20ErrorBased%20v1.0.pdf)

Additional information was gathered during the benchmark, including information related to the different features of various scanners. These details are organized in the following reports, and might prove useful when searching for tools for specific tasks or tests:

Comparison of Active Vulnerability Detection Features (http://wavsep.googlecode.com/files/Active%20Vulnerability%20Detection%20Features%20Comparison%20v1.0.pdf)

Comparison of Complementary Scanning Features - Passive Analysis, CGI Scanning, Brute Force, etc (http://wavsep.googlecode.com/files/Complementary%20Scan%20Features%20Comparison%20v1.0.pdf)

Comparison of Usability, Coverage and Scan Initiation Features (http://wavsep.googlecode.com/files/List%20of%20Scanner%20Features%20%281%20of%203%29%20v1.0.pdf)

Comparison of Authentication, Scan Control and Connection Support Features (http://wavsep.googlecode.com/files/List%20of%20Scanner%20Features%20%282%20of%203%29%20v1.0.pdf)

Comparison of Advanced and Uncommon Features (http://wavsep.googlecode.com/files/List%20of%20Scanner%20Features%20%283%20of%203%29%20v1.0.pdf)

Information regarding the scan logs, list of untested tools and abnormal behaviors of scanners can be found in the article appendix sections:

The following appendix report contains a list of scanners that were not included in the test:

Appendix A – A List of Tools not included in the Test (The end of the article)

The scan logs (describing the executing process and configuration of each scanner) can be viewed in the following appendix report: Appendix B – WAVSEP Scanning Logs (http://wavsep.googlecode.com/files/WavsepScanLogs%20v1.0.pdf)

During the benchmark, certain tools with abnormal behavior were identified; the list of these tools is presented in the following appendix report:

Appendix C – Scanners with Abnormal Behavior (The end of the article)

List of Tested Scanners

The following report (PDF) contains the list of scanners tested in this benchmark, in addition to their version, their author and their status: http://wavsep.googlecode.com/files/List%20of%20Tested%20Web%20Application%20Vulnerability%20Scanners%20v1.0.pdf

For those of you that want a quick glimpse, the following scanners were tested in the benchmark:

Acunetix Web Vulnerability Scanner (Free Edition), aidSQL, Andiparos, arachni, crawlfish, Gamja, Grabber, Grendel Scan, iScan, JSKY Free Edition, LoverBoy, Mini MySqlat0r, Netsparker Community Edition, N-Stalker Free Edition, Oedipus, openAcunetix, Paros Proxy, PowerFuzzer, Priamos, ProxyStrike, Sandcat Free Edition, Scrawler, ScreamingCSS, ScreamingCobra, Secubat, SkipFish, SQID (SQL Injection Digger), SQLiX, sqlmap, UWSS(Uber Web Security Scanner), VulnDetector, W3AF, Wapiti, Watobo, Web Injection Scanner (WIS), WebCruiser Free Edition, WebScarab, WebSecurify, WSTool, Xcobra, XSSer, XSSploit, XSSS, ZAP.

Source, License and Technical Details of Tested Scanners

The following report (PDF) contains a comparison of licenses, development technology and sources (home page) of different scanners: http://wavsep.googlecode.com/files/Details%20of%20Tested%20Web%20Application%20Vulnerability%20Scanners%20v1.0.pdf

Comparison of Active Vulnerability Detection Features

The following report (PDF) contains a comparison of active vulnerability detection features in the various scanners: http://wavsep.googlecode.com/files/Active%20Vulnerability%20Detection%20Features%20Comparison%20v1.0.pdf

Aside from the Count column (which represents the total amount of active vulnerability detection features supported by the tool, not including complementary features such as web server scanning and passive analysis), each column in the report represents an active vulnerability detection feature, which translates to the exposure presented in the following list:

SQL – SQL Injection

BSQL – Blind SQL Injection

RXSS – Reflected Cross Site Scripting

PXSS – Persistent / Stored Cross Site Scripting

DXSS – DOM XSS

Redirect – External Redirect / Phishing via Redirection

Bck – Backup File Detection

Auth – Authentication Bypass

CRLF – CRLF Injection / Response Splitting

LDAP – LDAP Injection

XPath – X-Path Injection

MX – MX Injection

Session Test – Session Identifier Complexity Analysis

SSI – Server Side Include

RFI-LFI – Directory Traversal / Remote File Include / Local File Include (Will be separated into different categories in future benchmarks)

Cmd – Command Injection / OS Command Injection

Buffer – Buffer Overflow / Integer Overflow (Will be separated into different categories in future benchmarks)

CSRF – Cross Site Request Forgery

A-Dos – Application Denial of Service / RegEx DoS

Comparison of Complementary Scanning Features

The following report (PDF) contains a comparison of complementary vulnerability detection features in the various scanners: http://wavsep.googlecode.com/files/Complementary%20Scan%20Features%20Comparison%20v1.0.pdf

In order to clarify what each column in the report table means, use the following interpretation:

Web Server Hardening – plugins that scan for HTTP method support (Trace, WebDAV), directory listing, Robots and cross-domain information disclosure, version specific vulnerabilities, etc.

CGI Scanning - Default files, common vulnerable applications, etc.

Passive Analysis – security tests that don’t require any actual attacks, and are based instead on information gathering and analysis of responses, including certificate & cipher tests, gathering of comments, mime type analysis, autocomplete detection, insecure transmission of credentials, google hacking, etc.

File Enumeration – directory and file enumeration features.

Comparison of Usability and Coverage Features

The following report (PDF) contains a comparison of usability, coverage and scan initiation features of different scanners:

http://wavsep.googlecode.com/files/List%20of%20Scanner%20Features%20%281%20of%203%29%20v1.0.pdf

Configuration & Usage Scale

Very Simple - GUI + Wizard

Simple - GUI with simple options, Command line with scan configuration file or simple options

Complex - GUI with numerous options, Command line with multiple options

Very Complex - Manual scanning feature dependencies, multiple configuration requirements

Stability Scale

Very Stable - Rarely crashes, Never gets stuck

Stable - Rarely crashes, Gets stuck only in extreme scenarios

Unstable - Crashes every once in a while, Freezes on a consistent basis

Fragile – Freezes or Crashes on a consistent basis, Fails performing the operation in many cases

(Unlike the accuracy values presented in the benchmark for W3AF, which are up date, the stability values for W3AF represent the condition of 1.0-RC3, and not 1.0-RC4; the values will be updated in the next benchmark, after the new version will be thoroughly tested)

Performance Scale

Very Fast - Fast implementation with limited amount of scanning tasks

Fast - Fast implementation with plenty of scanning tasks

Slow - Slow implementation with limited amount of scanning tasks

Very Slow - Slow implementation with plenty of scanning tasks

Comparison of Connection and Authentication Features

The following report (PDF) contains a comparison of connection, authentication and scan control features of different scanners:

http://wavsep.googlecode.com/files/List%20of%20Scanner%20Features%20%282%20of%203%29%20v1.0.pdf

Comparison of Advanced Features

The following report (PDF) contains a comparison of advanced and uncommon scanner features:

http://wavsep.googlecode.com/files/List%20of%20Scanner%20Features%20%283%20of%203%29%20v1.0.pdf

Benchmark Results – Reflected XSS Detection Accuracy

The results of the Reflected Cross Site Scripting (RXSS) benchmark are presented in the following report (PDF format):

http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20RXSS%20Detection%20Accuracy%20v1.0.pdf

The results only include vulnerable pages linked from the index-xss.jsp index page (RXSS-GET or RXSS-POST directories, in addition to the RXSS-FalsePositive directory). XSS Vulnerable locations in the SQL injection vulnerable pages were not taken into account, since they don’t necessarily represent a unique scenario (or at least not until the “layered vulnerabilities” scenario will be implemented).

Benchmark Results – SQL Injection Detection Accuracy

The overall results of the SQL Injection benchmark are presented in the following report (PDF format):

http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20v1.0.pdf

Benchmark Drilldown – Erroneous SQL Injection Detection

The results of the Error-Based SQL Injection benchmark are presented in the following report (PDF format):

http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20-%20ErrorBased%20v1.0.pdf

Benchmark Drilldown – Blind SQL Injection Detection

The results of the Blind SQL Injection benchmark are presented in the following report (PDF format):

http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20-%20Blind%20v1.0.pdf

Initial Analysis & Conclusions

After performing an initial analysis on the data, I have come to a simple conclusion as to which combination of tools will be the most effective in detecting Reflected XSS vulnerabilities in the public (unauthenticated) section of a tested web site, while providing the least amount of false positives:

Netsparker CE (42 cases), alongside Acunetix Free Edition (38 cases, including case 27 which is missed by Netsparker), alongside Skipfish (detects case 12 which is missed by both tools). I’d also recommend executing N-Stalker on small applications since it able to detect certain cases that none of the other tested tools can (but the XSS scanning feature is limited to 100 URLs).

Using Sandcat or Proxy Strike alongside Burp Spider/Paros Spider/External Spider can help detect additional potentially vulnerable locations (cases 10, 11, 13-15 and 17-21) that could be manually verified by a human tester.

So combining four tools will give the best possible result of RXSS detection in the unauthenticated section of an application, using today’s free & open source tools… WOW, it took some time to get to that conclusion. However, scanning the public section of the application is one thing, and scanning the internal section (authenticated section) of the application is another; effectively scanning the authenticated section requires various features such as authentication support, URL scanning restrictions, manual crawling (in case damage might be caused from crawling certain URLs), etc; so the conclusions for the public section are not necessarily fit for the internal section.

During the next few days, I’ll try and analyze the results and come to additional conclusions (internal RXSS scanning, external & internal SQLi scanning, etc). Simply check my blog in a few days to see which conclusions were already published.

An updated benchmark document will be released in the WAVSEP project homepage after each addition, conclusion or change.

A comment about accuracy and inconsistent results

During the benchmark, I have executed each tool more than once, and on rare occasions, dozens of times. I have discovered that some of the tools have inconsistent results in certain fields (particularly SQL injection). The following tools produced inconsistent results in the SQLi detection field: Skipfish (my guess is the inconsistencies are related to crawling problems and connection timeouts), Oedipus, and probably a couple of others that I can’t remember.

It is important to note that the 100% Reflected XSS detection ratio that Sandcat and ProxyStrike produce comes with a huge amount of false positives, a fact that signifies that the detection algorithm works more like a passive scanner (such as watcher by casaba), and less like an active intelligent scanner that verifies that the injection returned is sufficient to exploit the exposure in the given scope. This conclusion does not necessarily pinpoint anything about other features of these scanners (for example, the SQL injection detection module of proxystrike is pretty decent), or presume that the XSS scanning features of these tools are “useless”; on the contrary, these tools can be used as means to obtain more leads for human verification, and can be very useful in the right context.

Furthermore, the 100% SQL Injection detection ratio of Wapiti needs to be further investigated since andiparos produced the same ratio when the titles of the various pages contained the word SQL (which is part of the reason that in the latest version of WAVSEP, this word does not appear anywhere).

Additional conclusions will follow.

So What Now?

So now that we have plenty of statistics to analyze, and a new framework for testing scanners, it’s time to discuss the next phases.

Although the calendar tells me that it took me 9 months to conduct this research, in reality, it took me a couple of years to collect all the tools, learn how to install and use them, gather everything that was freely available for more than 5 minutes and test them all together.

However, since my research led me to develop a whole framework for benchmarking (aside from the WAVSEP project which was already published), I believe (or at least hope) that thanks to the platform, future benchmarks will be much easier to conduct, and in fact, I’m planning on updating the content of the web site (http://sectooladdict.blogspot.com/) with additional related content on a regular basis.

In addition to different classes of benchmarks, the following goals will be in the highest priority:

Improve the testing framework (WAVSEP); add additional test cases and additional security vulnerabilities.
Perform additional benchmarks on the framework, and on a consistent basis. I'm currently aiming for one major benchmark per year, although I might start with twice per year, and a couple of initial releases that might come even sooner.
Publish the results of tests against sample vulnerable web applications, so that some sort of feedback on other types of exposures will be available (until other types of vulnerabilities will be implemented in the framework), as well as features such as authentication support, crawling, etc.
Gradually develop a framework for testing additional related features, such as authentication support, malformed HTML tolerance, abnormal response support, etc.
Integration with external frameworks for assessing crawling capabilities, technology support, etc.

The different vendors will receive an email message from an email address designated for communicating with them. I urge them to try and contact me through that address, and not using alternative means, so I’ll be able to set my priorities properly. I apologize in advance for any delays in my responses in the next few weeks.

Appendix A – A List of Tools Not Included In the Test

The benchmark focused on web application scanners that are free to use (freeware and/or open source), are able to detect either Reflected XSS or SQL Injection vulnerabilities, and are also able to scan multiple URLs in the same execution.

As a direct implication, the test did NOT include the following types of tools:

· Commercial scanners - The commercial versions of AppScan, WebInspect, Cenzic, NTOSpider, Acunetix, Netsparker, N-Stalker, WebCruiser, Sandcat and many other commercial tools that I failed to mention. Any tool in the benchmark that holds the same commercial name is actually a limited free version of the same product, and does not refer (or even necessarily reflect on) the full product.
· Online Scanning Services – Online applications that remotely scan applications, including (but not limited to) Zero Day Scan, Appscan On Demand, Click To Secure, QualysGuard Web Application Scanning (Qualys), Sentinel (WhiteHat), Veracode (Veracode), VUPEN Web Application Security Scanner (VUPEN Security), WebInspect (online service - HP), WebScanService (Elanize KG), Gamascan (GAMASEC – currently offline), etc.
· Scanners without RXSS / SQLi detection features, including (but not limited to):

o LFIMap

o phpBB-RFI Scanner

o DotDotPawn

o CSRF Tester

o etc

· Passive Scanners (response analysis without verification), including (but not limited to):

o Watcher (Fiddler Plugin by Casaba Security)

o Skavanger (OWASP)

o Pantera (OWASP)

o Rat proxy (Google)

o Etc

· Scanners for specific products or services (CMS scanners, Web Services Scanners, etc), including (but not limited to):

o WSDigger

o Sprajax

o ScanAjax

o Joomscan

o Joomlascan

o Joomsq

o WPSqli

o etc

· White box & Code Review Application Scan Tools, including (but not limited to):

o PuzlBox

o Inspathx

o etc

· Uncontrollable Scanners - scanners that can’t be controlled or restricted to scan a single site, since they either receive the list of URLs to scan from Google Dork, or continue and scan external sites that are linked to the tested site. This list currently includes the following tools (and might include more):

o Darkjumper 5.8 (scans additional external hosts that are linked to the given tested host)

o Bako's SQL Injection Scanner 2.2 (only tests sites from a google dork)

o Serverchk (only tests sites from a google dork)

o XSS Scanner by Xylitol (only tests sites from a google dork)

o Hexjector (by hkhexon) – also falls into other categories

o etc

· Deprecated Scanners - incomplete tools that were not maintained for a very long time. This list currently includes the following tools (and might include more):

o etc

· De facto Fuzzers – tools that scan applications in a similar way to a scanner, but where the scanner attempts to conclude whether or not the application or is vulnerable (according to some sort of “intelligent” set of rules), the fuzzer simply collects abnormal responses to various inputs and behaviors, leaving the task of concluding to the human user.

o Lilith 0.4c/0.6a (both versions 0.4c and 0.6a were tested, and although the tool seems to be a scanner at first glimpse, it doesn’t perform any intelligent analysis on the results).

· Fuzzers – scanning tools that lack the independent ability to conclude whether a given response represents a vulnerable location, by using some sort of verification method (this category includes tools such as JBroFuzz, Firefuzzer, Proxmon, st4lk3r, etc). Fuzzers that had at least one type of exposure that was verified were included in the benchmark (Powerfuzzer).
· CGI Scanners: vulnerability scanners that focus on detecting hardening flaws and version specific hazards in web infrastructures (Nikto, Wikto, WHCC, st4lk3r, N-Stealth, etc)
· Single URL Vulnerability Scanners - scanners that can only scan one URL at a time, or can only scan information from a google dork (uncontrollable).

o Havij (by itsecteam.com)

o Hexjector (by hkhexon)

o Simple XSS Fuzzer [SiXFu] (by www.EvilFingers.com)

o Mysqloit (by muhaimindz)

o PHP Fuzzer (by RoMeO from DarkMindZ)

o SQLi-Scanner (by Valentin Hoebel)

o Etc.

· The following scanners:

o sandcatCS 4.0.3.0 - Since sandcat 4.0 free edition, a more advanced tool from the same vendor is already tested in the benchmark.

o GNUCitizen JAVASCRIPT XSS SCANNER - since WebSecurify, a more advanced tool from the same vendor is already tested in the benchmark.

o Vulnerability Scanner 1.0 (by cmiN, RST) - since the source code contained traces for remotely downloaded RFI lists from locations that do not exist anymore. I might attempt to test it anyway in the next benchmark.

o XSSRays 0.5.5 - I might attempt to test it in the next benchmark.

o XSSFuzz 1.1 - I might attempt to test it in the next benchmark.

o XSS Assistant - I might attempt to test it in the next benchmark.

· Vulnerability Detection Helpers – tools that aid in discovering a vulnerability, but do not detect the vulnerability themselves; for example:

o Exploit-Me Suite (XSS-Me, SQL Inject-Me, Access-Me)

o Fiddler X5s plugin

· Exploiters - tools that can exploit vulnerabilities but have no independent ability to automatically detect vulnerabilities on a large scale. Examples:

o MultiInjector

o XSS-Proxy-Scanner

o Pangolin

o FGInjector

o Absinth

o Safe3 SQL Injector (an exploitation tool with scanning features (pentest mode) that are not available in the free version).

o etc

· Exceptional Cases

Appendix B – WAVSEP Scanning Logs

The execution logs, installation steps and configuration used while scanning with the various tools are all described in the following report (PDF format):

http://wavsep.googlecode.com/files/WavsepScanLogs%20v1.0.pdf

Appendix C – Scanners with Abnormal Behavior

During the assessment, parts of the source code of open source scanners and the HTTP communication of some of the scanners was analyzed; some tools behaved in an abnormal manner that should be reported:

· Priamos IP Address Lookup – The tool Priamos attempts to access “whatismyip.com” (or some similar site) whenever a scan is initiated (verified by channeling the communication through Burp proxy). This behavior might derive from a trojan horse that infected the content on the project web site, so I’m not jumping to any conclusions just yet.
· VulnerabilityScanner Remote RFI List Retrieval (listed in the scanners that were not tested, appendix A, developed by a group called RST, http://pastebin.com/f3c267935) – In the source code of the tool VulnerabilityScanner (a python script), I found traces for remote access to external web sites for obtaining RFI lists (might be used to refer the user to external URLs listed in the list). I could not verify the purpose of this feature since I didn’t manage to activate the tool (yet); in theory, this could be a legitimate list update feature, but since all the lists the tool uses are hardcoded, I didn’t understand the purpose of the feature. Again, I’m not jumping to any conclusions; this feature might be related to the tool’s initial design, which was not fully implemented due to various considerations. I’ll try and drill deeper in the next benchmark (and hopefully, manage to test the tool’s accuracy as well).

Where to begin…

2010-04-16T05:18:00.000-07:00

There’s a ton of security tools out there.

From the point of view of security consultants (pen-testers to be exact), most of these tools are there to make their job easier, aid them in improving test results and enable them to reduce the time required to perform their tests.

But that’s not how it works in reality...

Lately there so much tools that it’s hard to know what to use;

Some of these tools are obsolete, some contain numerous bugs that prevents their execution from being effective, and some simply don’t justify the time required to execute them. On the other hand, some relatively anonymous tools generate spectacular results and can provide great benefits, but for some unknown reason (that has nothing to do with their quality), do not receive the credit and recognition they deserve.

After several years in the profession, and as an official security tool addict, I have decided to invest some time in sharing my experiences from using these tools, and from time to time, publish detailed benchmarking articles that compare between the various tools features, usability, accuracy, advantages and disadvantages.

In hopes that the community will benefit from this initiative, and in hopes that it will inspire the various tool vendors to compete and improve their tools,

Let the contest begin.

#	Supported Technologies
DAST	Any application with WEB/REST/WebService back-end. Some exotic back-end listeners may be supported as well (web-sockets, DWR, AMF, etc). The support also depends on compatibility with input delivery vectors, as well as compatible crawling OR session recording features.
SAST	Java, ASP.Net, C#.Net, VB.Net, PHP, Noje.js, Html/JS, SQL, Ruby, Pyhon, C, C++, JSP, ASP3, VB6, VBScript, Groovy, Scala, Perl, Apex, VisualForce, Android/iOS/WinMobile, Objective C, Swift, PhoneGap, Flex ActionScript, COBOL, ABAP, Coldfusion CFML
IAST	Java, .Net, PHP (few vendors), Node.js (few vendors), Ruby (few vendors), Python (experimental)

Coverage	DAST	SAST	Passive-IAST	Active-IAST
Out-Of-The-Box Wide Coverage Min Effort	? In Unauthenticated/ Form/Basic/NTLM	✔ Most Scenarios	? In Tested/Used Instances	X Depending on Implementation
End-To-End Coverage Scan/Correlate Issues in All Client/FE/BE Layers	✔ In Client Triggered Sequences	X Depending on Implementation	X Depending on Implementation	? Depending on Implementation
3^rd Party Code Closed Source Libraries/Entry-Points	? For “Visible” Methods	X No DE-compilation	✔ Depending on Implementation	✔ Depending on Implementation
Dead/Blocked Code Non-Web Executable	X Depending on Implementation	✔ Most Scenarios	X Depending on Implementation	X Depending on Implementation