Security Tools Benchmarking

SecToolMarket - A dynamic benchmark presentation website

2012-02-19T17:36:00.000-08:00

I noticed a pattern in my behavior... and I haven't really decided if it's good or bad.

I have a tendency to take up a task (which always seems super-simple at first), and then somewhere in the middle, find myself investing twice the time I originally planned, due to an outburst of obsessive perfectionism.

Then again, the exaggerated optimism is probably the main reason why I manage to follow through with those tasks, and end up thinking they were fun... and this one actually was.

Although I can't really claim that what I'm about to present is perfect (I'm learning to control that one, hopefully), and the design is not yet memorable (U-N-D-E-R-S-T-A-T-E-M-E-N-T), but it's certainly going to be useful for a lot of folks - pen-testers (first and foremost), vendors, analysts, researchers, security personal, and a bunch of people that stumbled upon this blog and are about to face a lot of scary words.

In short, the benchmark presentation framework is up and ready, and published as a web site called SecToolMarket (http://www.sectoolmarket.com).

I originally planned hosting it in Google sites (which is why there's no JS/AJAX/etc), but after a couple of hours of desperately trying to upload bulks of files to Google sites, I gave up and used the conventional method.

Although it doesn't yet contain a lot of new information (mostly additional information & analysis of the products tested in 2011), it's much easier to navigate through the data, and the analysis of the 2011 benchmark can provide additional insights, even to those that read the 2011 benchmark post.

A part from adding statistics, making things simpler, adding glossaries for everything and collecting vendor and product specific stats under dedicated pages, this framework can also be updated more frequently (and hopefully on a consistent basis), contains information that wasn't published, and allows you to track my progress as I'm performing my comparisons.

The two new categories (input-vector-support and coverage) are still incomplete (and will probably be updated soon, especially for commercial scanners - which will hopefully notify me if there's any missing information), but they already provide some insights, that might be relevant for some us.

And for all the kind souls... please help me spread the news... tweets, blog posts and telepathy are all welcome :)

Some screen captures of the content:

I'll probably post additional information on this website, and my future plans, but in the meantime, I'm going to crash, and hope you have fun with it.

Rules of the Game – Scanner Benchmarks

2011-10-23T15:46:00.000-07:00

The last couple of months have been very interesting (thanks for all the great feedback and constructive criticism), and I have some good news and several announcements.

First, the good news:

I had several discussions with Simon Bennetts (psiinon), one of the chapter leaders in OWASP and the leader / co-leader of several OWASP projects (ZAP, WAVE and OWASP-DEF).

One of the sub projects Simon is leading is ZAP-WAVE, the only additional web-app scanner evaluation framework which is actively maintained (the last publically available update of the third framework – "moth", was in mid 2009), and he suggested we merge our efforts so that everyone will benefit.

To make a long story short, Simon contributed the source code of the current test cases of ZAP-WAVE, allowing me adjust them into WAVSEP format and publish them under GPL 3.0 (currently available under ASF 2.0, lawyer comments aside). He even suggested that in the future, test cases that will be implemented by the ZAP team will be in WAVSEP format (structure and documentation).

That's obviously great news for me (and for anyone else using the project or the benchmark results – credits to Simon Bennetts and Axel Neumann), since the ZAP-WAVE project already contains test cases in several exposures that are not covered by WAVSEP, and any additional contribution will only enhance my current efforts (I'm currently working on dozens of additional test cases for new exposure categories).

I have already started to adjust these test cases (changes and integration notes will appear in the changelog), and I hope I'll manage to release them soon.

Now for several announcements that are related to the upcoming benchmark and the future versions of WAVSEP:

After the last benchmark was published, I got a lot of feedback, requests, interesting ideas and various suggestions. I read it all, and some of the requests and suggestions will be implemented in the upcoming benchmark and the future versions of WAVSEP.

The different feedbacks lead me to some important realizations:

· The rules of the test must be made public and clear to all vendors, in order to make sure that the process will be fair. In order to achieve this goal, certain changes must be implemented in the testing process.

· In order to enable vendors to show improvement quickly and in order to prevent any previous "negative" results from being perceived as a long term "punishment", the result presentation method must be updated more frequently, even between benchmarks.

As a result, I have constructed the following set of rules which will govern the testing processes in any future benchmark I will perform, and also require some changes in the publication cycles of WAVSEP:

· In order to enable vendors to show improvement, all the future benchmarks will be based on the WAVSEP test cases used in the previous benchmark, in addition to any other tests (interpretation: the upcoming benchmark will also include tests against all the SQLi and RXSS test cases of WAVSEP 1.0.3).

· Since a benchmark is much more interesting if the content in it is new, each major benchmark will include different test aspects and / or detection results for test cases in additional exposure categories.

· In order for the contest to be even more interesting (and in order to prevent one vendor from preparing for everything while another was not even notified, was not aware of the WAVSEP platform, has insufficient time to improve the tool prior to the benchmark, etc), the test cases of some of the new exposure categories will only be published after the first major benchmark that included tests against them – something that will add some spice to the results, make sure the process will be fair, but will still enable vendors to improve their previous score.

· The upcoming benchmark will include tests in some new categories: I'm currently aiming for at least 3 additional categories, in addition to the previous (and I hope that I'll manage to finish all the developments and tests before my next deadline… at least for most tools).

· Vendors that wish to update their score will be given an opportunity to do so, even between major benchmarks, by using a presentation method that will support dynamically updated content. The terms for these tests will be published separately, as soon as the presentation framework will be available (soon). Re-tests of additional versions of the same product will be performed under these terms.

· Since my final goal is to test all the vendors (almost 100), test additional types of scanning services / products, and eventually, test as many features of these tools as I possibly can, my time is a valuable asset, and contacting commercial vendors that don't offer a publically available evaluation version is very difficult. Although I will try my best to go through official channels and perform all the tests myself (or through members of the WAVSEP project), my experience shows that in some cases, the official channels might be time consuming, and sadly, sometimes more then I can afford. Therefore, I encourage vendors to contact me directly, starting of November 15, so I could test them properly, on equal terms.

Summary: future benchmarks will include test cases used in previous benchmarks (to enable vendors to show improvement), new test cases which will only be published after the benchmark (so that the tests will be fair and the content more interesting), and finally - the results will be more dynamic, to disable one more participation barrier.

As I said in my previous posts - I'm planning to continue to perform these comparisons for a long time, and intend to make sure that the community and vendors will both be able benefit from this initiative, if they only choose to.

Cheers

Session Puzzling and Session Race Conditions

2011-09-18T08:54:00.000-07:00

Is It Really That Complicated?

Session Puzzling – An Indirect Application Attack Vector – Now Simplified

A couple of months ago, I published a paper on an under-emphasized application level attack vector nicknamed "Session Puzzling" – an attack pattern that can abuse improper usage of session variables (a.k.a "Session Puzzles") in order to impersonate users, elevate privileges, bypass security restrictions and even execute "traditional" attack vectors against applications, while bypassing any existing security mechanisms by attacking the application using a trusted input source.

Even though the paper was published alongside a training kit that was meant to demonstrate the various attack vectors (a vulnerable application called "puzzlemall"), the vast majority of responses I got have made me realize that most of the 2000 security professionals that were exposed to this attack did not manage to understand it.

Some of the responses associated the paper to unrelated attacks, some didn't understand the impact or the mechanics, and some responses even claimed that the attacks is too complicated to perform (!?!).

Although I know that the attack is not simple, and that several session puzzling vectors require 10+ requests, I refuse to believe it's that complicated.

Over the last couple of years, I have seen many commercial applications that were vulnerable to this attack (Oracle E-Business Suite Included), so I'm giving it one more shot before I'll let the attack fall into the "too complicated to explain" category, and keep it all to myself.

The original whitepaper/presentation can be downloaded from the following addresses (contains background, additional attack vectors and mitigations):
Whitepaper
Presentation

The project homepage:
http://puzzlemall.googlecode.com/

The following short movies demonstrate a few simple session puzzling sequences:

Authentication Bypass via Session Puzzling (Abusing common session variables):

http://www.youtube.com/watch?v=-DackF8HsIE

User Impersonation via Session Puzzling (Abusing common session variables):

http://www.youtube.com/watch?v=ikIyInm0wAg

Session Puzzling via Redirection Prevention (Abusing Premature Session Population):

http://www.youtube.com/watch?v=iTcOooHbgog

Bypassing Restrictions in Multiphase Processes via Session Puzzling (Abusing Common Session Flags)

http://www.youtube.com/watch?v=HeP54b52IeQ

The following POC movie demonstrates the attack against Oracle E-Business Suite (exception scenario - not relying on input):

http://www.hacktics.com/content/advisories/AdvORA20091214.html

The training kit can be downloaded from the following address:

http://puzzlemall.googlecode.com/files/puzzlemall.war (derby version)

Temporal Session Race Conditions and Layer Targeted ADoS

Although the original attack relied on the existence of persistent session values, an extended attack was presented last week (15^th of September), in a local OWASP chapter meeting.

The extended method (nicknamed "Temporal Session Race Conditions") enables detecting & exploiting session puzzles even if the session variables have a lifespan of milliseconds (session-level race conditions), by increasing the latency of certain lines of code through the use of layer targeted denial of service attacks.

The original OWASP presentation:
Presentation

The following movies demonstrate a few simple TSRC attacks:

Exploiting Temporal Session Race Conditions via Connection Pool Consumption:

http://www.youtube.com/watch?v=woWECWwrsSk

Exploiting Temporal Session Race Conditions via RegEx DoS:

http://www.youtube.com/watch?v=3k_eJ1bcCro

An extended version of "puzzlemall" which includes TSRC vulnerabilities (premium login page, requires MySQL):

http://puzzlemall.googlecode.com/files/puzzlemall-v.1.1.2-mysql.zip

A simple tool that can assist in the detection of TSRC connection pool consumption scenarios:

http://puzzlemall.googlecode.com/files/SessionKeepAlive.exe

Acknowledgements

The following individuals contributed to the Session Puzzling / TSRC research in various ways, and helped me turn a bunch of ideas into a consistent methodology:

Oren Ofer, Oren Hafif, Alex Ganelis, Liran Sheinbox and Zafrir Grossman.

Additional Resources
An attack similar to session puzzling is mentioned under the name "session poisoning", but the session puzzling/TSRC sequences differ from this attack mainly by the lack of direct input dependency (see the multiphase restriction bypass scenario and the e-business suite exploit for the exception scenario), and expand the attack tool-set in the aspect of methodology, predefined sequences, scope of modules, complementary methods and usage of denial of service for extending the lifespan of temporary session variables.

Commercial Web Application Scanner Benchmark

2011-08-01T20:48:00.000-07:00

The Scanning Legion:

Web Application Scanners Accuracy Assessment & Feature Comparison

Commercial & Open Source Scanners

A Comparison of 60 Commercial & Open Source Black Box Web Application Vulnerability Scanners

By Shay Chen

Security Consultant, Researcher & Instructor

http://sectooladdict.blogspot.com/

sectooladdict-$at$-gmail-$dot$-com

August 2011

Assessment Environments: WAVSEP 1.0 / WAVSEP 1.0.3 (http://code.google.com/p/wavsep/)

Disclaimer

The results of this research are only valid for estimating the detection accuracy of SQLi & RXSS exposures, and for counting and comparing the various features of the tested tools.

The author did not evaluate every possible feature of each product, only the categories tested within the research, and thus, does not claim to be able to estimate the ROI from each individual product.

Furthermore, several vendors invested resources in improving their tools according to the recommendations of the WAVSEP platform which was publically available since December 2010. Some of them did so without any relation to the benchmark (and before they were aware of it), and some in preparation for it. Since the special structure of the WAVSEP testing platform actually requires the vendor to cover more vulnerable test scenarios, that action actually improves the detection ratio of the tool in any application (for the exposures covered by WAVSEP).

It is however, important to mention that a few vendors were not notified on this benchmark, and were not aware of the existence of the WAVSEP platform, and thus, could not have enhanced their tools in preparation for this benchmark (HP Webinspect, Tenable Nessus, and Janus security Webcruiser), while other vendors that were tested in the initial research phases released updated versions that were not tested (Portswigger Burpsuite and Cenzic Hailstorm)

That being said, the benchmark does represent the accuracy level of each tool in the date it was tested (the results of the vast majority of the tools are valid for the date this research was released), but future benchmark will use a different research model in order to ensure that the competition will be fair for all vendors.

Table of Contents

1. Prologue

2. List of Tested Web Application Scanners

3. Benchmark Overview & Assessment Criteria

4. Test I – The More The Merrier – Counting Audit Features

5. Test II – To the Victor Go the Spoils – SQL Injection

6. Test III – I Fight (For) the Users – Reflected XSS

7. Test IV – Knowledge is Power - Feature Comparison

8. What Changed?

9. Initial Conclusions – Open Source vs. Commercial

10. Morale Issues in Commercial Product Benchmarks

11. Verifying The Benchmark Results

12. Notifications and Clarifications

13. List of Tested Scanners

14. Source, License and Technical Details of Tested Scanners

15. Comparison of Active Vulnerability Detection Features

16. Comparison of Complementary Scanning Features

17. Comparison of Usability and Coverage Features

18. Comparison of Connection and Authentication Features

19. Comparison of Advanced Features

20. Detailed Results: Reflected XSS Detection Accuracy

21. Detailed Results: SQL Injection Detection Accuracy

22. Drilldown – Error Based SQL Injection Detection

23. Drilldown – Blind & Time Based SQL Injection Detection

24. Technical Benchmark Conclusions – Vendors & Users

25. So What Now?

26. Recommended Reading List: Scanner Benchmarks

27. Thank-You Note

28. Frequently Asked Questions

29. Appendix A – Assessing Web Application Scanners

30. Appendix B – A List of Tools Not Included In the Test

31. Appendix C – WAVSEP Scan Logs

32. Appendix D – Scanners with Abnormal Behavior

1. Prologue

I've always been curious about it… from the first moment I executed a commercial scanner, almost seven years ago, to the day I started performing this research. Although manual penetration testing has always been the main focus of the test, most of us use automated tools to easily detect "low hanging fruit" exposures, increase the coverage when testing large scale applications in limited timeframes and even to double check locations that were manually tested. The questions always pops up, in every penetration test in which these tools are used…

"Is it any good?", "Is it better than…" and "Can I rely on it to…" are questions that every pen-tester asks himself whenever he hits the scan button.

Well, curiosity is a strange beast… it can drive you to wander and search, consume all your time in a search for obscure solutions.

So recently, because of curiosity, I decided that I want to find out for myself, and invest whatever resources necessary to solve this mystery once and for all.

Although I can hardly state that all my questions were answered, I can definitely sate your curiosity for the moment, by sharing insights, interesting facts, useful information and even some surprises, all derived from my latest research which is focused on the subject of commercial & open source web application scanners.

This research covers the latest versions of 12 commercial web application scanners and 48 free & open source web application scanners, while comparing the following aspects of these tools:

· Number & Type of Vulnerability Detection Features

· SQL Injection Detection Accuracy

· Reflected Cross Site Scripting Detection Accuracy

· General & Special Scanning Features

Although my previous research included similar information, I regretted one thing after it was published; I did not present the information in a format that was useful to the common reader. In fact, as I found out later, many readers skipped the actual content, and focused on sections of the article that were actually a side effect of the main research.

As a result, the following article will focus on presenting the information in a simple comprehendible graphical format, while still providing the detailed research information to those interested… and there's a lot of new information to be shared – knowledge that can aid pen-testers in choosing the right tools, managers in budget related decisions, and visionaries, in properly reading the map;

But before you read the statistics and insights presented in this report, and reach a conclusion as to which tool is the "best", it is crucial that you read ‎Appendix A - Section 29, which explains the complexity of assessing the overall quality of web application scanners… As you're about to find out, this question cannot be answered so easily… at least not yet.

…

So without any further delay, let's focus on the information you seek, and discuss the insights and conclusions later.

2. List of Tested Web Application Scanners

The following commercial scanners were included in the benchmark:

· IBM Rational AppScan v8.0.03 - iFix Version (IBM)

· WebInspect v9.10.78.0, SecureBase 4.05.99 (HP)

· Hailstorm Professional v6.5-5267(Cenzic)

· Acunetix WVS v7.0-20110608 (Acunetix)

· NTOSpider v 5.4.098 (NT Objectives)

· Netsparker v2.0.0.0 (Mavituna Security)

· Burp Suite v1.3.09 (Portswigger)

· Sandcat v4.2.4.0 (Syhunt)

· ParosPro v1.9.12 (Milescan)

· JSky v3.5.1-905 (NoSec)

· WebCruiser v2.5.0 EE (Janus Security)

· Nessus v4.41-15078 (Tenable Network Security) – Only the Web Application Scanning Features

The following new free & open source scanners were included in the benchmark:

VEGA 1.0 beta (Subgraph), Safe3WVS v9.2 FE (Safe3 Network Center), N-Stalker 2012 Free Edition v7.1.1.106 (N-Stalker), DSSS (Damn Simple SQLi Scanner) v0.1h, SandcatCS v4.2.3.0

The updated versions of the following free & open source scanners were re-tested in the benchmark:

Zed Attack Proxy (ZAP) v1.3.0, sqlmap v0.9-rev4209 (SVN), W3AF 1.1-rev4350 (SVN), Watobo v0.9.7-rev544, Acunetix Free Edition v7.0-20110711, Netsparker Community Edition v1.7.2.13, WebSecurify v0.8, WebCruiser v2.4.2 FE (corrections), arachni v0.2.4 / v0.3, XSSer v1.5-1, Skipfish 2.02b, aidSQL 02062011

The results were compared to those of unmaintained scanners tested in the original benchmark:

Andiparos v1.0.6, ProxyStrike v2.2, Wapiti v2.2.1, Paros Proxy v3.2.13, PowerFuzzer v1.0, Grendel Scan v1.0, Oedipus v1.8.1, Scrawler v1.0, Sandcat Free Edition v4.0.0.1, JSKY Free Edition v1.0.0, N-Stalker 2009 Free Edition v7.0.0.223, UWSS (Uber Web Security Scanner) v0.0.2, Grabber v0.1, WebScarab v20100820, Mini MySqlat0r v0.5, WSTool v0.14001, crawlfish v0.92, Gamja v1.6, iScan v0.1, LoverBoy v1.0, openAcunetix v0.1, ScreamingCSS v1.02, Secubat v0.5, SQID (SQL Injection Digger) v0.3, SQLiX v1.0, VulnDetector v0.0.2, Web Injection Scanner (WIS) v0.4, Xcobra v0.2, XSSploit v0.5, XSSS v0.40, Priamos v1.0

For the full list of commercial & open source tools that were not tested in this benchmark, refer to ‎Appendix B - Section 30.

3. Benchmark Overview & Assessment Criteria

The benchmark focused on testing commercial & open source tools that are able to detect (and not necessarily exploit) security vulnerabilities on a wide range of URLs, and thus, each tool tested was required to support the following features:

· The ability to detect Reflected XSS and/or SQL Injection vulnerabilities.

· The ability to scan multiple URLs at once (using either a crawler/spider feature, URL/Log file parsing feature or a built-in proxy).

· The ability to control and limit the scan to internal or external host (domain/IP).

The testing procedure of all the tools included the following phases:

· The scanners were all tested against the latest version of WAVSEP (v1.0.3), a benchmarking platform designed to assess the detection accuracy of web application scanners. The purpose of WAVSEP’s test cases is to provide a scale for understanding which detection barriers each scanning tool can bypass, and which vulnerability variations can be detected by each tool. The various scanners were tested against the following test cases (GET and POST attack vectors):

o 66 test cases that were vulnerable to Reflected Cross Site Scripting attacks.

o 80 test cases that contained Error Disclosing SQL Injection exposures.

o 46 test cases that contained Blind SQL Injection exposures.

o 10 test cases that were vulnerable to Time Based SQL Injection attacks.

o 7 different categories of false positive RXSS vulnerabilities.

o 10 different categories of false positive SQLi vulnerabilities.

· In order to ensure the result consistency, the directory of each exposure sub category was individually scanned multiple times using various configurations.

· The features of each scanner were documented and compared, according to documentation, configuration, plugins and information received from the vendor.

· In order to ensure that the detection features of each scanner were truly effective, most of the scanners were tested against an additional benchmarking application that was prone to the same vulnerable test cases as the WAVSEP platform, but had a different design, slightly different behavior and different entry point format (currently nicknamed "bullshit").

The results of the main test categories are presented within three graphs (commercial graph, free & open source graph, unified graph), and the detailed information of each test is presented in a dedicated report.

So, now that you've learned about the testing process, it's time for the results…

4. Test I – The More The Merrier – Counting Audit Features

The first assessment criterion was the number of audit features each tool supports.

Reasoning: An automated tool can't detect an exposure that it can't recognize (at least not directly, and not without manual analysis), and therefore, the number of audit features will affect the amount of exposures that the tool will be able to detect (assuming the audit features are implemented properly, that vulnerable entry points will be detected and that the tool will manage to scan the vulnerable input vectors).

For the purpose of the benchmark, an audit feature was defined as a common generic application-level scanning feature, supporting the detection of exposures which could be used to attack the tested web application, gain access to sensitive assets or attack legitimate clients.

The definition of the assessment criterion rules out product specific exposures and infrastructure related vulnerabilities, while unique and extremely rare features were documented and presented in a different section of this research, and were not taken into account when calculating the results. Exposures that were specific to Flash/Applet/Silverlight and Web Services Assessment were treated in the same manner.

The Number of Audit Features in Web Application Scanners – Commercial Tools

The Number of Audit Features in Web Application Scanners - Free & Open Source Tools

The Number of Audit Features in Web Application Scanners – Unified List

So, now that were done with the quantity, let's get to the quality…

5. Test II – To the Victor Go the Spoils – SQL Injection

The second assessment criterion was the detection accuracy of SQL Injection, one of the most famous exposures and the most commonly implemented attack vector in web application scanners.

Reasoning: a scanner that is not accurate enough will miss many exposures, and classify non-vulnerable entry points as vulnerable. This test aims to assess how good is each tool at detecting SQL Injection exposures in a supported input vector, which is located in a known entry point, without any restrictions that can prevent the tool from operating properly.

The evaluation was performed on an application that uses MySQL 5.5.x as its data repository, and thus, will reflect the detection accuracy of the tool when scanning similar data repositories.

Result Chart Glossary

Note that the BLUE bar represents the vulnerable test case detection accuracy, while the RED bar represents false positive categories detected by the tool (which may result in more instances then what the bar actually presents, when compared to the detection accuracy bar).

The SQL Injection Detection Accuracy of Web Application Scanners – Commercial Tools

The SQL Injection Detection Accuracy of Web Application Scanners – Open Source & Free Tools

The SQL Injection Detection Accuracy of Web Application Scanners – Unified List

It's obvious that testing one feature is not enough; ideally, the detection accuracy of all audit features should be assessed, but in the meantime, we will settle for one more…

6. Test III – I Fight (For) the Users – Reflected XSS

The third assessment criterion was the detection accuracy of Reflected Cross Site Scripting, a common exposure which is the 2nd most commonly implemented feature in web application scanners.

Result Chart Glossary

The Reflected XSS Detection Accuracy of Web Application Scanners – Commercial Tools

The Reflected XSS Detection Accuracy of Web Application Scanners – Open Source & Free Tools

The Reflected XSS Detection Accuracy of Web Application Scanners – Unified List

7. Test IV – Knowledge is Power - Feature Comparison

The list of tools tested in this benchmark is organized within the following reports:

· List of Tested Scanners

· Source, License and Technical Details of Tested Scanners

Additional information was gathered during the benchmark, including information related to the different features of the various scanners. These details are organized in the following reports, and might prove useful when searching for tools for specific tasks or tests:

· Comparison of Active Vulnerability Detection Features (Audit Features) – 1 of 2

· Comparison of Active Vulnerability Detection Features (Audit Features) – 2 of 2

· Comparison of Complementary Scanning Features - Passive Analysis, CGI Scanning, Etc

· Comparison of Usability, Coverage and Scan Initiation Features

· Comparison of Authentication, Scan Control and Connection Support Features

· Comparison of Advanced and Uncommon Features

For detailed information on the accuracy assessment results, refer to the following reports:

· Benchmark Results – Reflected XSS Detection Accuracy

· Benchmark Results – SQL Injection Detection Accuracy – Unified

· Benchmark Drilldown – Blind & Time Based SQL Injection Detection Accuracy

· Benchmark Drilldown – Error Dependant SQL Injection Detection Accuracy

· The Scan Logs (describing the executing process and configuration of each scanner)

Additional information on the scan logs, the list of untested tools and the abnormal behaviors of scanners can be found in the article appendix sections (at the end of the article):

‎Appendix B - Section 30 – an appendix that contains a list of tools that were not included in the benchmark

‎Appendix D - Section 32 – an appendix that describes scanners with abnormal behavior

8. What Changed?

Since the latest benchmark, many open source & commercial tools added new features and improved their detection accuracy.

The following list presents a summary of changes in the detection accuracy of free & open source tools that were tested in the previous benchmark:

· arachni – a dramatic improvement in the detection accuracy of Reflected XSS exposures, and a dramatic improvement in the detection accuracy of SQL Injection exposures (verified on mysql).

· sqlmap – a dramatic improvement in the detection accuracy of SQL Injection exposures (verified on mysql).

· Acunetix Free Edition – a major improvement in the detection accuracy of RXSS exposures.

· Watobo – a major improvement in the detection accuracy of SQL Injection exposures (verified on mysql).

· N-Stalker 2009 FE vs. 2012 FE – although this tool is a very similar to N-Stalker 2009 FE, the surprising discovery I had was that the detection accuracy of N-Stalker 2012 is very different – it detects only a quarter of what N-Stalker 2009 used to detect. Assuming this result is not related to a bug in the product or in my testing procedure, it means that the newer free version is significantly less effective than the previous free version, at least at detecting reflected XSS. A legitimate business decision, true, but surprising nevertheless.

· aidSQL – a major improvement in the detection accuracy of SQL Injection exposures (verified on mysql).

· XSSer – a major improvement in the detection accuracy of Reflected XSS exposures, even though the results were not consistent.

· Skipfish – a slight improvement in the detection accuracy of RXSS exposures (it is currently unknown if the RXSS detection improvement is related to changes in code or to the enhanced testing method), and a slight decrease in the detection accuracy of SQLi exposures (might be related to the different testing environment and the different method used to count the results).

· WebSecurify – a slight improvement in the detection accuracy of RXSS exposures (it is currently unknown if the RXSS detection improvement is related to changes in code or to the enhanced testing method).

· Zed Attack Proxy (ZAP) – Identical results. Any minor difference was probably caused due to the testing environment, configuration or minor issues.

· W3AF – slight improvement in the detection accuracy of RXSS exposures and slight decrease in the detection accuracy of SQL Injection exposures.

· Netsparker Community Edition – Identical results. Any minor difference was probably caused due to the testing environment, configuration or minor issues.

· WebCruiser Free Edition – a minor decrease in accuracy, due to fixing documentation mistakes from the previous benchmark.

9. Initial Conclusions – Open Source vs. Commercial

The following section presents my own personal opinions on the results of the benchmark, and since opinions are beliefs, which are affected by emotions and circumstances, you are entitled to your own.

After testing over 48 open source scanners multiple times, and after comparing the results and experiences to the ones I had after testing 12 commercial ones (and those are just the ones that I reported), I have reached the following conclusions:

· As far as accuracy & features, the distance between open source tools and commercial tools is not as big as it used to be – tools such as sqlmap, arachni, wapiti, w3af and others are slowly closing the gap. That being said, there still is a significant difference in stability & false positives, in which most open source tools tend to have more false positives and be relatively unstable when compared to most commercial tools.

· Some open source tools, even the most accurate ones, are relatively difficult to install & use, and still require fine-tuning in various fields. In my opinion, a non-technical QA engineer will have difficulties using these tools, and as a general rule, I'll recommend using them if your background is relatively technical (consultant, developer, etc). For all the rest, especially non-technical enterprise employees that prefer a decent usage experience - stick with commercial produces, with their free versions, or with the simple variations of open source tools.

· If you are using a commercial product, it's best to merge the use of tools with a wide variety of features with tools with high detection accuracy. It's possible to use tools that have relatively good scores in both of these aspects, or use a tool with a wide variety of features with another tool that has enhanced accuracy. Yes, this statement can be interpreted to using combinations of commercial and open source tools, and even to using two different commercial tools, so that one tool will complete the other. Budget? Take a look at the cost diversity of the tools, before you make any harsh decisions; I promise you'll be surprised.

10. Morale Issues in Commercial Product Benchmarks

While testing the various commercial tools, I have dealt with certain moral issues that I want to share. Many vendors that were aware of this research enhanced their tools in preparation for it, an action I respect, and consider a positive step. Since the testing platform that included most of the tests was available online, preparing for the benchmark was a relatively easy task for any vendor that invested the resources.

So, is the benchmark fair for vendors that couldn’t improve their tools due to various circumstances?

The testing process of a commercial tool is usually much more complicated and restrictive then testing a free or open source tool; it is necessary to contact the vendor to obtain an evaluation license, and the latest version of the tool (a process that can take several weeks), the evaluation licenses are usually restricted to a short evaluation timeframe (usually two weeks), and thus, updating and testing the tools in a future date can become a hassle (since some of the process will have to be performed all over again)… but why am I telling you all this?

Simply, because I believe that the relevance of the test I performed for vendors that provided me an extended evaluation period and access to new builds was better; for example, a few days before the latest benchmark, immediately after testing the latest versions of two major vendors, I decided to rescan the platform using the latest versions of all the commercial tools I have, to ensure that the benchmark will be published with the most updated results.

I verified that JSky, WebCruiser, and ParosPro didn't release a new version, tested the latest versions of AppScan, WebInspect, Acunetix, Netsparker, Sandcat and Nessus.

It made sense that builds that were tested a short while ago (like NTO spider), were also something that I can rely on to represent the currently state of the tool (I hopeJ).

I did however, have a problem with Cenzic and Burp, two of the first tools that I tested in this research, since my evaluation licenses were no longer valid, and I couldn't update the tools to their latest version and scan again, and since I had 2-3 days until the end of my planned schedule, with a million tasks pending, I simply couldn't afford going through the evaluation request phase again, with all of my good intentions, and the willingness to sacrifice my spare time to ensure these tools will be properly represented.

Even though the results of some updated products (WebInspect and Nessus being the best examples) didn't change at all, even after I updated them to the latest version, who could say that the result would be the same for other vendors?

So, were the terms unfair to burp and cenzic?

Finally, several vendors sent me multiple versions and builds – they all wanted to succeed, a legitimate desire of any human being, even more so for a firm. Apart from the time each test took (a price I was willing to pay at the time), the new builds were sent even in the last day of the benchmark, and afterwards.

But if the new version is better, and more accurate, by limiting the amount of tests I perform for a given vendor, isn't that against what I'm trying to achieve in all my benchmarks, which is to release the benchmark with the most updated results, for all the tools?

(For example, Syhunt, a vendor that did very well in the last benchmark, sent me its final build (2.4.2.5) a day after the deadline, and included a time based SQL injection detection feature in that build, but since I couldn't afford the time anymore, I couldn't test the build, so, am I really reflecting the tool's current state in the most accurate manner? But if I would have tested this build, shouldn't I provide the rest of the vendors the same opportunity?)

One of the questions I believe I can answer – the accuracy question.

A benchmark is, in a very real sense, a competition, and since I take the scientific approach, I believe that the results are absolute, at least for the subject that is being tested. Since I'm not claiming that one tool is "better" than the other in every category, only at the tested criterion, I believe that priorities do not matter – as long as the test really reflects the current situation, the result is reliable.

I leave the interpretation of the results to the reader, at least until I'll cover enough aspects of the tools.

As for the rest of the open issues, I don't have good answers for all of those questions, and although I did my very best in this benchmark, and even exceeded what I thought I'm capable of, I will probably have to think of some solutions that will make the next benchmark terms equal, even for scanners that were tested in the beginning of the benchmark, and less time consuming then it has been.

11. Verifying The Benchmark Results

The results of the benchmark can be verified by replicating the scan methods described in the scan log of each scanner, and by testing the scanner against WAVSEP v1.0.3.

The latest version of WAVSEP can be downloaded from the web site of project WAVSEP (binary/source code distributions, installation instructions and the test case description are provided in the web site download section):

http://code.google.com/p/wavsep/

12. Notifications and Clarifications

How to use the results of the benchmark

The results of the benchmark clearly show how accurate each tool is in detecting the tested vulnerabilities (SQL Injection (MySQL ) & Reflected Cross Site Scripting), as long as it is able to locate and scan the vulnerable entry points. The results might even help to estimate how accurate each tool is in detecting related vulnerabilities (for example SQL Injection vulnerabilities which are based on other databases), and determine which exposure instances cannot be detected by certain tools;

However, currently, the results DO NOT evaluate the overall quality of the tool, since they don't include detailed information on the subjects such as crawling quality, technology support, scoping, profiling, stability in extreme cases, tolerance, detection accuracy of other exposures and so on... at least NOT YET.

I highly recommend reading the detailed results, and the appendix that deals with web application scanner evaluation, before getting to any conclusions.

Additional Notifications

During the benchmark, I have reported bugs that had a major affect on the detection accuracy to several commercial and open source vendors:

· A performance improvement feature in NTOSpider caused it not to scan many POST XSS test cases, and thus, the detection accuracy of RXSS POST test cases was significantly smaller then the RXSS GET detection accuracy. The vendor was notified on this issue, and provided me with a special build that overrides this feature (at least until they will have a feature in the GUI to disable this mechanism).

· A similar performance improvement feature in Netsparker caused the same issue, however, the feature could have been disabled in Netsparker, and thus, with the support of the relevant personal at Netsparker, I was able to work around the problem.

· A few bugs in arachni prevented the blind sql injection diff plugins from working properly. I notified the author, Tasos, on the issue, and he quickly fixed the issue and released the new version.

· Acunetix RXSS detection result was updated to match the results of the latest free version (one version above the tested commercial version) - Since the tested commercial version of Acunetix was older than the tested free version (20110608 vs 20110711), and since the results of the upgraded free version were actually better than the older commercial version I had tested, I changed the results of the commercial tool to match the ones of the new free version (from 22 to 24 in both the GET & POST RXSS detection scores).

· Changes in results from the previous benchmark might be attributed to enhanced scanning features, and/or to enhanced stability in the test environment & method (connection pool, limited & divided scope).

13. List of Tested Scanners

The following report contains the list of scanners tested in this benchmark, and provides information on the tested version, the tool's vendor/author and the current status of product:

http://sectooladdict-benchmarks.googlecode.com/files/List%20of%20Tested%20Web%20Application%20Vulnerability%20Scanners%20-%20WAVSEP%20Benchmark%202011.pdf

14. Source, License and Technical Details of Tested Scanners

The following report compares the licenses, development technology and sources (home page) of the various scanners:

http://sectooladdict-benchmarks.googlecode.com/files/Details%20of%20Tested%20Web%20Application%20Vulnerability%20Scanners%20-%20WAVSEP%20Benchmark%202011.pdf

15. Comparison of Active Vulnerability Detection Features

The following reports compare the active vulnerability detection features (audit features) of the various tested scanners:

First Report:

http://sectooladdict-benchmarks.googlecode.com/files/Application%20Active%20Scan%20Features%20Comparison%201of2%20-%20WAVSEP%20Benchmark%202011.pdf

Second Report:

http://sectooladdict-benchmarks.googlecode.com/files/Application%20Active%20Scan%20Features%20Comparison%202of2%20-%20WAVSEP%20Benchmark%202011.pdf

Aside from the Count column (which represents the total amount of audit features supported by the tool, not including complementary features such as web server scanning and passive analysis), each column in the report represents an audit feature. The description of each column is presented in the following glossary table:

Title	Description
SQL	Error Dependant SQL Injection
BSQL	Blind & Intentional Time Delay SQL Injection
RXSS	Reflected Cross Site Scripting
PXSS	Persistent / Stored Cross Site Scripting
DXSS	DOM XSS
Redirect	External Redirect / Phishing via Redirection
Bck	Backup File Detection
Auth	Authentication Bypass
CRLF	CRLF Injection / Response Splitting
LDAP	LDAP Injection
XPath	X-Path Injection
MX	MX / SMTP / IMAP Injection
Session Test	Session Identifier Complexity Analysis
SSI	Server Side Include
RFI-LFI	Directory Traversal / Remote File Include / Local File Include (Will be separated into different categories in future benchmarks)
Cmd	Command Injection / OS Command Injection
Buffer	Buffer Overflow
CSRF	Cross Site Request Forgery
A-Dos	Application Denial of Service / RegEx DoS
Privilege Escalation	Privilege Escalation Between Different Roles and User Accounts (Resources / Features)
Format String	Format String Injection
File Upload	File Upload / Insecure File Upload
Code Injection	Code Injection (ASP/JSP/PHP/Perl/etc)
XML Injection	XML / SOAP Injection
Source Code Disclosure	Source Code Disclosure Detection
Integer Overflow	Integer Overflow
Padding Oracle	Padding Oracle Detection / Exploitation
Session Fixation	Session Fixation

16. Comparison of Complementary Scanning Features

The following report compares complementary vulnerability detection features in the tested scanners:

http://sectooladdict-benchmarks.googlecode.com/files/Complementary%20Scan%20Features%20Comparison%20-%20WAVSEP%20Benchmark%202011.pdf

In order to clarify what each column in the report table means, use the following glossary table:

Title	Description
Web Server Hardening	Features that are able to detect Insecure HTTP method support (PUT, Trace, WebDAV), directory listing, robots and cross-domain files information disclosure, version specific vulnerabilities, etc.
CGI Scanning	Default files, common vulnerable applications, etc.
Passive Analysis	Security tests that don’t require any actual attacks, and are instead based on information gathering and analysis of responses, including certificate & cipher tests, content & metadata analysis, mime type analysis, autocomplete detection, insecure transmission of credentials, google hacking, etc.
File / Dir Enumeration	Directory and file enumeration features
Notes and Other Features	Uncommon or Unique features

17. Comparison of Usability and Coverage Features

The following report compares the usability, coverage and scan initiation features of the tested scanners:
http://sectooladdict-benchmarks.googlecode.com/files/List%20of%20Scanner%20Features%20(1%20of%203)%20-%20WAVSEP%20Benchmark%202011%20-%20Final3.pdf

In order to clarify what each column in the report table means, use the following glossary table:

Title	Possible Values
Configuration & Usage Scale	Very Simple - GUI + Wizard Simple - GUI with simple options, Command line with scan configuration file or simple options Complex - GUI with numerous options, Command line with multiple options Very Complex - Manual scanning feature dependencies, multiple configuration requirements
Stability Scale	Very Stable - Rarely crashes, Never gets stuck Stable - Rarely crashes, Gets stuck only in extreme scenarios Unstable - Crashes every once in a while, Freezes on a consistent basis Fragile – Freezes or Crashes on a consistent basis, Fails performing the operation in many cases
Performance Scale	Very Fast - Fast implementation with limited amount of scanning tasks Fast - Fast implementation with plenty of scanning tasks Slow - Slow implementation with limited amount of scanning tasks Very Slow - Slow implementation with plenty of scanning tasks

18. Comparison of Connection and Authentication Features

The following report compares the connection, authentication and scan control features of the tested scanners:
http://sectooladdict-benchmarks.googlecode.com/files/List%20of%20Scanner%20Features%20(2%20of%203)%20-%20WAVSEP%20Benchmark%202011%20-%20Final.pdf

19. Comparison of Advanced Features

The following report contains a comparison of advanced and uncommon scanner features:

http://sectooladdict-benchmarks.googlecode.com/files/List%20of%20Scanner%20Features%20%283%20of%203%29%20-%20WAVSEP%20Benchmark%202011.pdf

20. Detailed Results: Reflected XSS Detection Accuracy

The results of the Reflected Cross Site Scripting (RXSS) accuracy assessment are presented in the following report (the graphical results representation is provided in the beginning of the article):

http://sectooladdict-benchmarks.googlecode.com/files/Web%20Application%20Scanner%20RXSS%20Detection%20Accuracy%20-%20WAVSEP%20ScoreChart%202011.pdf

The results that were taken into account only include vulnerable pages linked from the index-xss.jsp index page (the RXSS-GET and/or RXSS-POST directories, in addition to the RXSS-FalsePositive directory). XSS Vulnerable entry points in the SQL injection vulnerable pages were not taken into account, since they don’t necessarily represent a unique scenario (or at least, not until the “layered vulnerabilities” scenario will be implemented).

21. Detailed Results: SQL Injection Detection Accuracy

The overall results of the SQL Injection accuracy assessment are presented in the following report (the graphical results representation is provided in the beginning of the article):

http://sectooladdict-benchmarks.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20-%20WAVSEP%20ScoreChart%202011.pdf

22. Drilldown – Error Based SQL Injection Detection

The results of the Error-Based SQL Injection benchmark are presented in the following report:

http://sectooladdict-benchmarks.googlecode.com/files/Web%20Application%20Scanner%20Error-Based%20SQLi%20Detection%20Accuracy%20-%20WAVSEP%20ScoreChart%202011.pdf

23. Drilldown – Blind & Time Based SQL Injection Detection

The results of the Blind & Time based SQL Injection benchmarks are presented in the following report:

http://sectooladdict-benchmarks.googlecode.com/files/Web%20Application%20Scanner%20Blind%20SQLi%20Detection%20Accuracy%20-%20WAVSEP%20ScoreChart%202011.pdf

24. Technical Benchmark Conclusions – Vendors & Users

While testing the various tools in this benchmark, I dealt with numerous difficulties, witnessed many inconsistent results and noticed that some tools had difficulties optimizing their scanning features on the tested platform. I had however, dealt with the other end of the spectrum, and used tools the easily overcome most of the difficulties related to detecting the tested vulnerabilities.

I'd like to share my conclusions, with the authors and vendors that are interested in improving their tools, and aren't offended by someone that's giving advice.

As far as detecting SQL injection exposures, I have noticed that tools that implemented the following features, detected more exposures, had less false positives, and provided consistent results:

· Time based SQL Injection detection vectors are very effective. They are, however, very tricky to use, since they might be affected by other attacks that are simultaneously executed, or affect the detection of other tests in the same manner. As a result, I recommended to all the authors & vendors to implement the following behavior in their product: execute time based attacks at the end of the scanning process, after all the rest of the tests are done, while using a reduced number of concurrent connections. Executing other tests in parallel might have a negative effect on the detection accuracy.

· Since the upper/lower timeout values used to determine whether or not a time based exploit was successful may change due to various circumstances, I recommend calculating and re-calculating this value during the scan, and revalidating each time based result independently, after verifying that the timeout values are "normal".

· Implement various payloads of time based attacks – the sleep method is not enough to cover all the databases, and not even all the versions of mysql.

25. So What Now?

So now that we have all those statistics, it's time to analyze them properly, and see which conclusions we can get to. Since this process will take time, I have to set some priorities;

In the near future, I will try to achieve the following goals:

· Find a better way to present the vast amount of information on web application scanners features & accuracy. I have been struggling with this issue for almost 2 years, but I think that I finally found a solution that will make the information more useful for the common reader… stay tuned for updates.

· Provide recommendations for the best current method of executing free & open source web application scanners; the most useful combinations, and the tiny tweaks required to achieve the best results.

· Release the new test case categories of WAVSEP that I have been working on. Yep, help needed.

In addition to the short term goals, the following long term goals will still have a high priority:

· Improve the testing framework (WAVSEP); add additional test cases and additional security vulnerabilities.

· Perform additional benchmarks on the framework, and on a consistent basis. I previously aimed for one major benchmark per year, but that formula might completely change, if I'll manage to work a few issues around a new initiative I have in this field.

· Integration with external frameworks for assessing crawling capabilities, technology support, etc.

· Publish the results of tests against sample vulnerable web applications, so that some sort of feedback on other types of exposures will be available (until other types of vulnerabilities will be implemented in the framework), as well as features such as authentication support, crawling, etc.

· Gradually develop a framework for testing additional related features, such as authentication support, malformed HTML tolerance, abnormal response support, etc.

I hope that this content will help the various vendors improve their tools, help pen-testers choose the right tool for each task, and in addition, help create some method of testing the numerous tools out there.

Since I have already been in the situation in the past, then I know what's coming… so I apologize in advance for any delays in my responses in the next few weeks.

The following resources include additional information on previous benchmarks, comparisons and assessments in the field of web application vulnerability scanners:

· "Webapp Scanner Review: Acunetix versus Netsparker", by Mark Baldwin (commercial scanner comparison, April 2011)

· "Effectiveness of Automated Application Penetration Testing Tools", by Alexandre Miguel Ferreira and Harald Kleppe (commercial & freeware scanner comparison, February 2011)

· "Web Application Scanners Accuracy Assessment", the predecessor of the current benchmark, by Shay Chen (a comparison of 43 free & open source scanners, December 2010)

· "State of the Art: Automated Black-Box Web Application Vulnerability Testing" (Original Paper), by Jason Bau, Elie Bursztein, Divij Gupta, John Mitchell (May 2010) – original paper

· "Analyzing the Accuracy and Time Costs of Web Application Security Scanners", by Larry Suto (commercial scanners comparison, February 2010)

· "Why Johnny Can’t Pentest: An Analysis of Black-box Web Vulnerability Scanners", by Adam Doup´e, Marco Cova, Giovanni Vigna (commercial & open source scanner comparison, 2010)

· "Web Vulnerability Scanner Evaluation", by AnantaSec (commercial scanner comparison, January 2009)

· "Analyzing the Effectiveness and Coverage of Web Application Security Scanners", by Larry Suto (commercial scanners comparison, October 2007)

· "Rolling Review: Web App Scanners Still Have Trouble with Ajax", by Jordan Wiens (commercial scanners comparison, October 2007)

· "Web Application Vulnerability Scanners – a Benchmark" , by Andreas Wiegenstein, Frederik Weidemann, Dr. Markus Schumacher, Sebastian Schinzel (Anonymous scanners comparison, October 2006)

27. Thank-You Note

During the research described in this article, I have received help from quite a few individuals and resources, and I’d like to take the opportunity to thank them all.

For all the open source tool authors that assisted me in testing the various tools in unreasonable late night hours, for the kind souls that helped me obtain evaluation licenses for commercial products, for the QA, Support and Development teams of commercial vendors, which saved me tons of time and helped me overcome obstacles, and for the various individuals that helped me contact these vendors.

I would also like to continue my tradition, and thank all the information sources that helped me gather the list of scanners over the years, including (but not limited to) information security sources such as PenTestIT (http://www.pentestit.com/), Security Sh3ll (http://security-sh3ll.blogspot.com/), NETpeas Toolswatch Service (http://www.vulnerabilitydatabase.com/toolswatch/), Darknet (http://www.darknet.org.uk/), Packet Storm (http://packetstormsecurity.org/), Help Net Security (http://www.net-security.org/), Astalavista (http://www.astalavista.com/), Google (of course) and many others.

I hope that the conclusions, ideas, information and payloads presented in this research (and the benchmarks and tools that will follow) will be for the benefit of all vendors, open source community projects and commercial vendors alike.

28. Frequently Asked Questions

Q: 60 web application scanners is an awful lot, how many scanners exist?

A: Assuming you are using the same definition for a scanner that I do, then I'm currently aware of 95 web application scanners that can claim to support the detection of generic application level exposures, in a safe an controllable manner, and in multiple URLs (48 free & open source scanners that were tested, 12 commercial scanners that were tested, 25 open source scanners that I didn't test yet, and 10 commercial scanners that slipped my grip). And yes, I'm planning on testing them all.

Q: Why RXSS and SQLi again? Will the benchmarks ever include additional exposures?

A: Yes, they will. In fact, I'm already working on test case categories of two different exposures, and will use them both for my next research. Besides, the last benchmark focused on free & open source products, and I couldn't help myself, I had to test them against each other.

Q: I can't wait for the next research, what can I do to speed things up?

A: I'm currently looking for methods to speed up the processes related to these researches, so if you're willing to help, contact me.

Q: What’s with the titles that contain cheesy movie quotes?

A: That's just it - I happen to like cheese. Let's see you coming up with better titles at 4AM.

29. Appendix A – Assessing Web Application Scanners

Although this benchmark contains tons of information, and is very useful as a decision assisting tool, the content within it cannot be used to calculate the accurate ROI (return of investment) of each web application scanner. Furthermore, it can't predict on its own exactly how good will the results of each scanner be in every situation (but it can predict what won't be detected), since there are additional factors that need to be taken into account.

The results in this benchmark could serve as an accurate evaluation formula only if the scanner will be used to scan a technology that it supports, pages that it can detect (manual crawling features can be used to overcome many obstacles in this case), and locations without technological barriers that it cannot handle (for example, web application firewalls or anti-CSRF tokens).

In order for us to truly assess the full capability of web application vulnerability scanners, the following features must be tested:

· The entry point coverage of the web application scanner must be as high as possible; meaning, the tool must be able to locate and properly activate (or be manually "taught") all the application entry points (e.g. static & dynamic pages, in-page events, services, filters, etc). Vulnerabilities in an entry point that wasn't located will not be detected. The WIVET project can provide additional information on coverage and support.

· The attack vector coverage of the web application scanner – does it support input vectors such as GET / POST / Cookie parameters? HTTP headers? Parameter Names? Ajax Parameters? Serialized Objects? Each input vector that is not supported means exposures that won't be detected, regardless of the tool's accuracy level (assuming the unsupported attack/input vector is vulnerable).

· The scanner must be able to handle the technological barriers implemented in the application, ranging from authentication mechanism to automated access prevention mechanisms such as CAPTCHAs and anti-CSRF tokens.

· The scanner must be able to handle any application specific problems it encounters, including malformed HTML (tolerance), stability issues and other limitations. If the best scanner in the world will consistently cause the application to crash in a couple of seconds, then it's not useful for assessing the security of that application (in matters that don't relate to DoS attacks).

· The number of features (active & passive) implemented in the web application vulnerability scanner.

· The accuracy level of each and every plugin supported by the web application vulnerability scanner.

That being said, it's crucial to remember that even in the most ideal scenario, with the absence of human intelligence, scanners can't detect all the instances of exposures that are truly logical – meaning, are related to specific business logic, and thus, are not perceived as an issue by an entity that can't understand the business logic.

But the sheer complexity of the issue does not mean that we shouldn't start somewhere, and that's exactly what I'm trying to do in my benchmarks – create a scientific, accurate foundation for obtaining that goal, with enough investment, over time.

Note that my explanations describe only a portion of the actual tests that should be performed, and I'm sharing them only to emphasize the true complexity of the core issue; I haven't touched stability, bugs, and a lot of other subjects, which may affect the overall result you get.

Additional information on evaluation standards for web application vulnerability scanners can be found in the WASC Web Application Security Scanner Evaluation Criteria web site.

30. Appendix B – A List of Tools Not Included In the Test

The following commercial web application vulnerability scanners were not included in the benchmark, since I didn't manage to get an evaluation version until the article publication deadline, or in the case of one scanner (mcafee), had problems with the evaluation version that I didn't manage to work out until the benchmark's deadline:

Commercial Scanners not included in this benchmark

· N-Stalker Commercial Edition (N-Stalker)

· McAfee Vulnerability Manager (McAfee / Foundstone)

· NeXpose Enterprise Edition Web Application Scanning Features (Rapid7)

· Retina Web Application Scanner (eEye Digital Security)

· WebApp360 (NCircle)

· Core Impact Pro Web Application Scanning Features (Core Impact)

· Parasoft Web Application Scanning Features (a.k.a WebKing, by Parasoft)

· MatriXay Web Application Scanner (DBAppSecurity)

· Falcove (BuyServers ltd, currently Unmaintained)

· Safe3WVS 9.2 Commercial Edition (Safe3 Network Center)

The following open source web application vulnerability scanners were not included in the benchmark, mainly due to time restrictions, but will be included in future benchmarks:

Open Source Scanners not included in this benchmark

· Rabbit VS

· Spacemonkey

· Kayra

· 2gwvs

· Webarmy

· springenwerk

· Mopset 2

· XSSFuzz 1.1

· Witchxtoolv

· PHP-Injector

· XSS Assistant

· Fiddler XSSInspector/XSRFInspector Plugins

· GNUCitizen JAVASCRIPT XSS SCANNER - since WebSecurify, a more advanced tool from the same vendor is already tested in the benchmark.

· Vulnerability Scanner 1.0 (by cmiN, RST) - since the source code contained traces for remotely downloaded RFI lists from locations that do not exist anymore.

The benchmark focused on web application scanners that are able to detect either Reflected XSS or SQL Injection vulnerabilities, can be locally installed, and are also able to scan multiple URLs in the same execution.

As a result, the test did not include the following types of tools:

· Online Scanning Services – Online applications that remotely scan applications, including (but not limited to) Appscan On Demand (IBM), Click To Secure, QualysGuard Web Application Scanning (Qualys), Sentinel (WhiteHat), Veracode (Veracode), VUPEN Web Application Security Scanner (VUPEN Security), WebInspect (online service - HP), WebScanService (Elanize KG), Gamascan (GAMASEC – currently offline), Cloud Penetrator (Secpoint), Zero Day Scan, DomXSS Scanner, etc.

· Scanners without RXSS / SQLi detection features:

o Dominator (Firefox Plugin)

o fimap

o lfimap

o phpBB-RFI Scanner

o DotDotPawn

o LFI (Library-level Fault Injector)

o lfi-scanner

o LFI-Scanner

o lfi-rfi2

o LFI/RFI Checker (astalavista)

o CSRF Tester

o etc

· Passive Scanners (response analysis without verification):

o Watcher (Fiddler Plugin by Casaba Security)

o Skavanger (OWASP)

o Pantera (OWASP)

o Ratproxy (Google)

o CAT The Manual Application Proxy (Context)

o etc

· Scanners of specific products or services (CMS scanners, Web Services Scanners, etc):

o WSDigger

o Sprajax

o ScanAjax

o Joomscan

o wpscan

o Joomlascan

o Joomsq

o WPSqli

o etc

· Web Application Scanning Tools which are using Dynamic Runtime Analysis:

o PuzlBox (the free version was removed from the web site, and is now sold as a commercial product named PHP Vulnerability Hunter)

o Inspathx

o etc

· Uncontrollable Scanners - scanners that can’t be controlled or restricted to scan a single site, since they either receive the list of URLs to scan from Google Dork, or continue and scan external sites that are linked to the tested site. This list currently includes the following tools (and might include more):

o Darkjumper 5.8 (scans additional external hosts that are linked to the given tested host)

o Bako's SQL Injection Scanner 2.2 (only tests sites from a google dork)

o Serverchk (only tests sites from a google dork)

o XSS Scanner by Xylitol (only tests sites from a google dork)

o Hexjector by hkhexon – also falls into other categories

o d0rk3r by b4ltazar

o etc

· Deprecated Scanners - incomplete tools that were not maintained for a very long time. This list currently includes the following tools (and might include more):

o Wpoison (development stopped in 2003, the new official version was never released, although the 2002 development version can be obtained by manually composing the sourceforge URL which does not appear in the web site- http://sourceforge.net/projects/wpoison/files/ )

o etc

· De facto Fuzzers – tools that scan applications in a similar way to a scanner, but where the scanner attempts to conclude whether or not the application or is vulnerable (according to some sort of “intelligent” set of rules), the fuzzer simply collects abnormal responses to various inputs and behaviors, leaving the task of concluding to the human user.

o Lilith 0.4c/0.6a (both versions 0.4c and 0.6a were tested, and although the tool seems to be a scanner at first glimpse, it doesn’t perform any intelligent analysis on the results).

o Spike proxy 1.48 (although the tool has XSS and SQLi scan features, it acts like a fuzzer more then it acts like a scanner – it sends payloads of partial XSS and SQLi, and does not verify that the context of the returned output is sufficient for execution or that the error presented by the server is related to a database syntax injection, leaving the verification task for the user).

· Fuzzers – scanning tools that lack the independent ability to conclude whether a given response represents a vulnerable location, by using some sort of verification method (this category includes tools such as JBroFuzz, Firefuzzer, Proxmon, st4lk3r, etc). Fuzzers that had at least one type of exposure that was verified were included in the benchmark (Powerfuzzer).

· CGI Scanners: vulnerability scanners that focus on detecting hardening flaws and version specific hazards in web infrastructures (Nikto, Wikto, WHCC, st4lk3r, N-Stealth, etc)

· Single URL Vulnerability Scanners - scanners that can only scan one URL at a time, or can only scan information from a google dork (uncontrollable).

o Havij (by itsecteam.com)

o Hexjector (by hkhexon)

o Simple XSS Fuzzer [SiXFu] (by www.EvilFingers.com)

o Mysqloit (by muhaimindz)

o PHP Fuzzer (by RoMeO from DarkMindZ)

o SQLi-Scanner (by Valentin Hoebel)

o Etc.

· Vulnerability Detection Assisting Tools – tools that aid in discovering a vulnerability, but do not detect the vulnerability themselves; for example:

o Exploit-Me Suite (XSS-Me, SQL Inject-Me, Access-Me)

o Fiddler X5s plugin

o XSSRays (chrome Addon)

· Exploiters - tools that can exploit vulnerabilities but have no independent ability to automatically detect vulnerabilities on a large scale. Examples:

o MultiInjector

o XSS-Proxy-Scanner

o Pangolin

o FGInjector

o Absinth

o Safe3 SQL Injector (an exploitation tool with scanning features (pentest mode) that are not available in the free version).

o etc

· Exceptional Cases

o SecurityQA Toolbar (iSec) – various lists and rumors include this tool in the collection of free/open-source vulnerability scanners, but I wasn’t able to obtain it from the vendor’s web site, or from any other legitimate source, so I’m not really sure it fits the “free to use” category.

31. Appendix C – WAVSEP Scan Logs

The execution logs, installation steps and configuration used while scanning with the various tools are all described in the following report:

http://sectooladdict-benchmarks.googlecode.com/files/Scan%20Log%20-%20WAVSEP%20Benchmark%202011.pdf

32. Appendix D – Scanners with Abnormal Behavior

The following appendix was published in my previous benchmark, but I decided to include in the current benchmark, mainly because I didn't manage to invest the time to get to the bottom of these mysteries, and didn't see any information on someone else that did.

During the current & previous assessment, parts of the source code of open source scanners and the HTTP communication of some of the scanners was analyzed; some tools behaved in an abnormal manner that should be reported:

· Priamos IP Address Lookup – The tool Priamos attempts to access “whatismyip.com” (or some similar site) whenever a scan is initiated (verified by channeling the communication through Burp proxy). This behavior might derive from a trojan horse that infected the content on the project web site, so I’m not jumping to any conclusions just yet.

· VulnerabilityScanner Remote RFI List Retrieval (listed in the scanners that were not tested, appendix A, developed by a group called RST, http://pastebin.com/f3c267935) – In the source code of the tool VulnerabilityScanner (a python script), I found traces for remote access to external web sites for obtaining RFI lists (might be used to refer the user to external URLs listed in the list). I could not verify the purpose of this feature since I didn’t manage to activate the tool (yet); in theory, this could be a legitimate list update feature, but since all the lists the tool uses are hardcoded, I didn’t understand the purpose of the feature. Again, I’m not jumping to any conclusions; this feature might be related to the tool’s initial design, which was not fully implemented due to various considerations.

Although I did not verify that any of these features is malicious in nature, these features and behaviors might be abused to compromise the security of the tester’s workstation (or to incriminate him in malicious actions), and thus, require additional investigation to disqualify this possibility.

Myth Breaker - The Best Open Source Web Application Vulnerability Scanner

2011-01-25T14:00:00.000-08:00

(The original benchmark post - comparison of 43 web application vulnerability scanners:

http://sectooladdict.blogspot.com/2010/12/web-application-scanner-benchmark.html)

It’s been a couple of weeks since the initial benchmark was published, and I used that time to contact most of the vendors and to come to some conclusions, as to which tool combinations are ideal for each task;

I believe that those of you that use these tools on a daily basis will find my conclusions interesting.

Please note that the conclusions refer to the condition of the tools in the day the benchmark was released (see the full explanation at the end of the post).

Glossary

AND – combining the tools is required to obtain the best results.

OR – using either one of the tools will provide nearly identical results.

AND/OR – it is currently unknown if combining them will provide additional benefits.

SAFE scan – a scan method in which the tester can select which URLs to scan, in order to prevent the scanner from accessing links that could delete data, lock user accounts or cause any other unintentional hazard (generally requires the scanner to have a proxy/manual crawling/URL file parsing/pre-configured URL restriction module); Recommended while scanning the internal section of an application that resides in a production environment.

UNSAFE scan – a scan method that scans all the URLs, without any restrictions or limitations; Recommended while scanning the public section of an application, and for scanning the internal section of an application that resides in the testing/development environment.

The Ideal Combination of Tools (Relevant to the release date of the initial benchmark – 26/12/2010):

(Constructed according to the cases detected by each tool, and according to tool capabilities and application scope restrictions)

Scan Type & Target

Reflected XSS

SQL Injection (MySQL)

Initial Public Scan

Initial Scan on the Application’s Public (unauthenticated) Section

(Purpose: gather as many “Low Hanging Fruit” exposures as possible with a minimal amount of false positives)

Netsparker AND Acunetix AND N-Stalker AND SkipFish

(Nearly False Positive Free Combination)

ProxyStrike AND WebCruiser

(Nearly False Positive Free Combination)

Internal Scan - Unsafe

The Application’s Internal (authenticated) Section

Netsparker AND Acunetix AND SkipFish

(Nearly False Positive Free Combination)

Wapiti

(Verification with other tools is recommended to reduce False Positives – ProxyStrike AND WebCruiser, In addition to one of the following: W3AF/Andipaors/ZAP/

Netsparker/Sandcat/

Oedipus)

Internal Scan - Safe

The Application’s Internal (authenticated) Section

(Method: scan internal application pages without activating any delete, logout or other dangerous operations).

ZAP AND W3AF

(Safe combination with relatively efficient accuracy)

W3AF AND Andiparos/Paros AND Oedipus AND ProxyStrike

Additional Public Scan

Detect additional potential exposures that require manual verification, and aren’t covered by previous tools

(Public Section)

ProxyStrike OR Sandcat (Grabber detects 1-2 additional POST cases - optional)

Wapiti

2nd Internal – Unsafe

Detect additional potential exposures that require manual verification, and aren’t covered by previous tools

ProxyStrike OR Sandcat

Wapiti

(No substantial change, so there’s no need to run another scan)

2nd Internal – Safe

Detect additional potential exposures that require manual verification, and aren’t covered by previous tools

(Method: scan internal application pages for additional exposure instances without activating any delete, logout or other dangerous operations)

ProxyStrike

W3AF AND Andiparos/Paros AND Oedipus AND ProxyStrike

(No substantial change, so there’s no need to run another scan)

Complementary Scan for Additional Exposures

Complementary Scan

Scan the applications with scanners that have a wider range of features, to cover additional security flaws

W3AF AND/OR Arachni AND/OR Skipfish AND/OR Sandcat

Notable Open Source & Freeware Tools – SQL Injection Detection

The highest SQLi detection ratio of open source & freeware tools belongs to Wapiti, currently the undisputed winner in this category.

A bit behind Wapiti were AndiParos, Zapproxy and Paros Proxy (all forks of the original Paros project), followed closely by Netsparker and W3AF (two tools that were prone to less false positives test cases, compared to all of the tools described so far - 30% compared to 40% or 50%).

* it is important to mention that Netsparker CE 1.5 does not contain Netsparker’s Blind-SQL injection module (disabled in this version), only the regular SQL-Injection module and the Boolean SQL-Injection module.

However, we cannot ignore the fact that the following tools had pretty decent accuracy with 0 false positives(!): WebCruiser (55.88%) and ProxyStrike (52.21%), making them ideal tools for an initial scan (Mini MySqlat0r and Scrawler had 0 false positives as well, but with lower accuracy).

Notable Open Source & Freeware Tools – XSS Detection

The Highest XSS detection ratio belongs to Sandcat, which detected nearly 100% of the overall test-cases (although like ProxyStrike & Grabber, it was misled by a few extra false positive test cases).

The highest XSS detection ratio of open source tools (and 2nd best in total) belongs to ProxyStrike (Grabber detected more POST test cases, but had a higher false positive ratio, and did not detect GET cases).

The best overall XSS detection ratio (while considering the low amount of false positives) belongs to Netsparker CE (63.64% and 3rd in the efficiency order, right after ProxyStrike), followed closely by N-Stalker and by Acunetix FE (and since Skipfish and these tools “complete” missing test cases in each other, they are ideal for initial scans, since they all have 0 false positives!).

The best overall XSS detection ratio (while considering the low amount of false positives) of open source tools belongs to WebSecurify.

The best HTTP GET XSS detection ratio (while considering the low amount of false positives) of open source tools belongs to XSSer.

The following open source tools had XSS detection modules that were free of false positives (while still having a relatively efficient detection ratio) – Grendel-Scan (GET) and Skipfish (Secubat had 0 false positives as well, but its detection ratio was a bit lower).

Notes

When using ProxyStrike for the initial scan, It’s probably best to use an external spider instead of the built in spider (e.g. use ProxyStrike as an outgoing/upstream proxy for Burp Suite FE or Paros/ZAP/Andiparos and then use the spider feature of the external tool through ProxyStrike).

As mentioned before, the conclusions reflect the condition of the various tools in the date the initial benchmark was published. Since the benchmark, many vendors had released new versions (some even in response to the benchmark), so the list of conclusions will change as soon as the next benchmark is released; I know for a fact that some vendors invested so much effort in improving their detection modules that some of the new versions get to nearly 100% detection ratio (but since I don’t have updated statistics, well have to wait).

Conclusions

So… it seems that I didn't find “the best web application vulnerability scanner” after all… but I did find combinations of open source & freeware tools that get pretty good results.

As I mentioned in previous posts, my work is only beginning.

Various open source vendors already released new versions that should be tested, tools that were improperly executed (or had a bug) should be retested as soon as their issues are mitigated, additional research led me to discover a couple of additional open source web application scanner projects, and at least one new open source web application scanner was released in the last couple of weeks (and I haven’t even mentioned commercial scanners).

Time to get back to work…

Follow Up & Clarifications

2011-01-11T14:51:00.000-08:00

I’ve been pretty busy trying to contact the various vendors and deliver materials that they can use for QA & development, and I must mention that so far every vendor / developer that I have contacted responded kindly, and many of them responded with excitement and already started enhancing their tool (which is GREAT news for all of us).

I managed to find the time to contact about 18 vendors, and hopefully I’ll manage to contact more in the following weeks (25 left to go). This process requires me to analyze the benefits of the tools of each vendor, and as a result, is more time consuming then I originally thought; however, thanks to this process, I believe that soon it will lead me to some additional interesting conclusions and insights, which I’ll publish separately.

In the process of contacting the vendors, I realized that I have neglected some of my duties and forgot to publish some important clarifications:

Although the test cases implemented as “False Positives” are by no means vulnerable to SQL Injection or Cross Site Scripting, some of the test cases still fall into a category of information that should be presented in the report under the context of another type of exposure:

o Pages that disclose sensitive information / exceptions (some SQL Injection False Positive test cases that are meant to simulate SQL errors that do not derive from user originating input, such as connection failures, etc).

o Pages that fall under the category of insecure coding practices (some of the False RXSS & SQLi pages).

Some tools are still in early beta, and some didn’t even publish an official alpha version (aidSQL, iScan, and some of the other tools that had zero accuracy); the accuracy of these tools was not really audited, due to limitations or bugs that will surely be mitigated in the future versions. The benchmark will be updated as soon as the tool vendors release a new stable version.
The execution of certain tools which were reported as having zero accuracy failed due to bugs or configuration flaws, and not accuracy related issues; These tools include SQLMap, aidSQL, VulnDetector, and a couple of more; I’m currently working with the various vendors to figure out how to execute them properly (or how to work around the specific bugs), so the test will actually reflect their accuracy level.

As a result, I believe that the next benchmark is going to be performed sooner then I planned;

It will probably include the same results alongside the corrected scans of the tools that had execution issues (particularly SQL tools), and maybe additional enhancements (under discussion).

I wish you all a Happy New Year :)

Web Application Scanner Benchmark (v1.0)

2010-12-26T03:26:00.000-08:00

Well, it’s finally done. What I originally thought will only take me a couple of days, and found myself doing for the past 9 months is finally ready for release, and it’s titled:

Web Application Scanners Accuracy Assessment

Freeware & Open Source Scanners

Comparison & Assessment of 43 Free & Open Source Black Box Web Application Vulnerability Scanners

By Shay Chen

Information Security Consultant, Researcher and Instructor

http://sectooladdict.blogspot.com/

sectooladdict -$at$- gmail -$dot$- com

December 2010

Assessment Environment: WAVSEP 1.0 (http://code.google.com/p/wavsep/)

Introduction

I’ve been collecting them for years, trying to get my hands on anything that was released within the genre. It started as a necessity, transformed into a hobby, and eventually turned into a relatively huge collection… But that’s when the problems started.

While back in 2005 I could barely find freeware web application scanners, by 2008 I had SO MANY of them that I couldn’t decide which ones to use. By 2010 the collection became so big that I came to the realization that I HAVE to choose.

I started searching for benchmarks in the field, but at the time, only located benchmarks the focused on comparing commercial web application scanners (with the exception of one benchmark that also covered 3 open source web application scanners), leaving the freeware & open source scanners in an uncharted territory;

http://www.virtualforge.de/index.php/en/library/white-papers/web-application-vulnerability-scanners-a-benchmark_en.html (Anonymous scanners)
http://anantasec.blogspot.com/2009/01/web-vulnerability-scanners-comparison.html (commercial scanners)
http://www.cs.ucsb.edu/~adoupe/static/black-box-scanners-dimva2010.pdf (mostly commercial, but including W3AF, paros and grendel-scan)
http://ha.ckers.org/files/Accuracy_and_Time_Costs_of_Web_App_Scanners.pdf (commercial scanners)

By 2010 I had over 50 tools, so I eventually decided to test them myself using the same model used in previous benchmarks (a big BIG mistake).

I initially tested the various tools against a vulnerable ASP.net web application and came to conclusions as to which tool is the “best”… and if it weren’t for my curiosity, that probably would have been the end of it and my conclusions might have mislead many more.

I decided to test the tools against another vulnerable web application, just to make sure the results were consistent, and arbitrarily selected “Insecure Web App” (a vulnerable JEE web application) as the second target… and to my surprise, the results of the tests against it were VERY different.

Some of the Tools that were efficient in the test against the vulnerable ASP.net application (which will stay anonymous for the time being) didn’t function very well and missed many exposures, while some of the tools that I previously classified as “useless” detected exposures that NONE of the other tools found.

After performing an in-depth analysis for the different vulnerabilities in the tested applications, I came to the conclusion that although the applications included a similar classification of exposures (SQL Injection, RXSS, Information disclosure, etc), the properties and restrictions in the exposure instances were VERY different in each application.

That’s when it dawned on me that the different methods that tools use to discover security exposures might be efficient for detecting certain common instances of a vulnerability while simultaneously being inefficient for detecting other instances of the same vulnerability, and that tools with “lesser” algorithms or different approaches (which might appear to be less effective at first) might be able to fill the gap.

So the question remains… Which tool is the best? Is there one that surpasses the others? Can there be only one?

I decided to find out…

It started as a bunch of test cases, and ended as a project containing hundreds of scenarios (currently focusing on Reflected XSS and SQL Injection) that will hopefully help in unveiling the mystery.

(A PDF version of this benchmark will be available shortly in the WAVSEP project home page at http://code.google.com/p/wavsep/)

Thank-You Note

Before I’ll describe project WAVSEP and the results of the first scanner benchmark performed using it, I’d like to thank all the tool developers and vendors that shared freeware & open source tools with the community over the years; if it weren’t for the long hours they’ve invested and the generosity they had to share their creations, then my job (and that of others in my profession) would have been much harder.

I’d like to express my sincere gratitude for Shimi Volkovich (http://il.linkedin.com/pub/shimi-volkovich/20/173/263), for taking the time to design the logo I’ll soon be using.

I would also like to thank all the sources that helped me gather the list of scanners over the years, including (but not limited to) information security sources such as PenTestIT (http://www.pentestit.com/), Security Sh3ll (http://security-sh3ll.blogspot.com/), Security Database (http://www.security-database.com/), Darknet (http://www.darknet.org.uk/), Packet Storm (http://packetstormsecurity.org/), Help Net Security (http://www.net-security.org/), Astalavista (http://www.astalavista.com/), Google (of course) and many others that I have neglected to mention due to my failing memory.

I hope that the conclusions, ideas, information and payloads presented in this research (and the benchmarks and tools that will follow) will benefit all vendors, and specifically help the open source community to locate code sections that all tool vendors could assimilate to improve their products; to that end I’ll try and contact each vendor in the next few weeks, in order to notify them on source codes that could be assimilated in their product to make it even better (on the basis of development technology and the license of each code section).

Phase I – The “Traditional” Benchmark

Testing the scanners against vulnerable training & real life applications.

As I mentioned earlier, In the initial phase of the benchmark, I have tested the various scanners in front of different vulnerable “training” applications (OWASP InsecureWebApp, a vulnerable .Net Application and a simple vulnerable application I have written myself), and tested many of them against real life applications (ASP.Net applications, Java applications based on Spring, Web application written in PHP, etc).

I decided not to publish the results just yet, and for a damn good reason which I did not predict in the first place; nevertheless, the initial process was very helpful because it helped me to learn about the different aspects of the tools: features, vulnerability list, coverage, installation processes, configuration methods, usage, adaptability, stability, performance and a bunch of other aspects.

I have found VERY interesting results that prove that certain old scanners might provide great benefits in many cases that many modern projects will not handle properly.

The process also enabled me to verify the support of the various tools in their proclaimed features (which I have literally done for the vast majority of the tools, using proxies, sniffers and other experiments), and even get a general measure of their accuracy and capabilities.

However, after seeing the results diversity in different applications and technologies, and after dealing with the countless challenges that came along the way, I have discovered several limitations and even a fundamental flaw in testing the accuracy, coverage, stability and performance of scanners in this manner (I have managed to test around 50 free and open source scanners by this point, as insane and unbelievable as this number might sound);

We may be able to estimate the general capabilities of a scanner from the amount of REAL exposures that it located, the amount of exposures that it missed (false negatives) and from the amount of FALSE exposures (false positives) it identified as security exposures, BUT on the other hand, the output of such a process will very much depend on the type of exposures that exist in the tested application, how much each scanner is adapted to the tested application technology and which private cases of exposures and barriers exist in the tested application.

A scanner that will be very useful for scanning PHP web sites might completely fail the task of scanning a ASP.Net web application, and a tool perfectly suited for that task might crash when faced with certain application behaviors, or be useless in detecting a private case of a specific vulnerability that is not supported by the tool.

I guess what I’m trying to say is this:

There are many forms and variations to each security exposure, and in order to prove my point, I’ll use the example of reflected cross site scripting;

Locations vulnerable to reflected cross site scripting might appear in many forms; they may require the attacker to send a whole HTML tag as a part of the crafted link, require the injection of an HTML event (in case the input-affected-output is printed in the context of a tag and the usage of tag-composing-characters is restricted), they may appear in locations vulnerable to SQL injection (and thus restrict the use of certain characters, or even require the usage of initial payloads that “disable” the SQL injection vulnerability first), require browser specific payloads or even direct injection of javascript/vbscript (in case the context is within a script tag, certain HTML events or even in the context of certain properties), and these cases are only a fragment of the whole list!

So, how can the tester know which of these cases is handled by each scanner from the figures and numbers presented in a general benchmark?

I believe he can’t. No matter how solid the difference appears, he really can’t.

Such information may allow him to root out useless tools (tools that miss even the most obvious exposures), and even identify what appears to be a significant difference in the accuracy of locating certain exposure instances, but the latter case might have been very different if the tested applications would have been prone to certain exposure instances that are the specialty of a different scanner, or would have included a technological barrier that requires a specific feature or behavior to bypass.

Thus, I have come to believe that the only way I could truly provide useful information to testers on the accuracy and coverage of freely available web application scanners is by writing detailed test cases for different exposures, starting with some core common exposures such as SQL Injection, cross site scripting and maybe a couple of others.

And thus, I have ended up investing countless nights in the development of a new test-case based evaluation application, designed specifically to test the support of each tool for detecting MANY different cases of certain common exposures.

The results of the original benchmark (against the vulnerable training web applications) will be published separately in a different article (since by now, many of them have been updated, and the results require modifications).

Phase II - Project WAVSEP

After documenting and testing the features of every free & open source web application scanner and scan script that I could get my hands on, I discovered that the most common features were Reflected Cross Site Scripting (RXSS) and SQL Injection (SQLi). I decided to focus my initial efforts on these two vulnerabilities, and develop a platform that could truly evaluate how good each scanner is in detecting them, which tool combinations provide the best results and which tool can bypass the largest amount of detection barriers.

Project WAVSEP (Web Application Vulnerability Scanner Evaluation Project) was implemented as a set of vulnerable JSP pages; each page implementing a unique test case.

A test case is defined as a unique combination of the following elements:

A certain instance of a given vulnerability.
Attack vectors with certain input origins (either GET or POST values, and in the future, also URL/path, cookie, various headers, file upload content and other origins).

Currently, only GET and POST attack vectors are covered, since most scanners support only GET and POST vectors (future versions of WAVSEP will include support for additional databases, additional response types, additional detection barriers, additional attack vector origins and additional vulnerabilities).

Project WAVSEP currently consists of the following test cases:

64 Reflected XSS test cases (32 GET cases, 32 POST cases -> 66 total vulnerabilities)
130 SQL Injection test cases, most of them implemented for MySQL & MSSQL (65 GET cases, 65 POST Vases -> 136 total vulnerabilities)

o The list of test cases includes vulnerable pages that respond with 500 HTTP errors, 200 HTTP Responses with erroneous text, 200 HTTP Responses with differentiation or completely identical 200 HTTP responses.

o 80 out of 136 cases are simple SQL injection test cases (500 & 200 erroneous HTTP responses), and 56 are Blind SQL Injection test cases (valid and identical 200 HTTP responses).

7 different categories of false positive Reflected XSS vulnerabilities (GET OR POST).
10 different categories of false positive SQL Injection vulnerabilities (GET OR POST).

Each exposure category in WAVSEP contains an index page with descriptions of different barriers in test cases, structures of a sample detection payloads and examples of such payloads.

A general description of each test case is also available in the following excel spreadsheet: http://code.google.com/p/wavsep/downloads/detail?name=VulnerabilityTestCases.xlsx&can=2&q=

Those that wish to verify the results of the benchmark can download the latest source code of project WAVSEP (including the list of test cases and their description) from the project’s web site:

http://code.google.com/p/wavsep/

Benchmark Overview

As mentioned before, the benchmark focused on testing free & open source tools that are able to detect (and not necessarily exploit) security vulnerabilities on a wide range of URLs, and thus, each tool tested needed to support the following features:

Either open source or free to use, so that open source projects and vendors generous enough to contribute to the community will benefit from the benchmark first.
The ability to detect Reflected XSS and/or SQL Injection vulnerabilities.
The ability to scan multiple URLs at once (using either a crawler/spider feature, URL/Log file parsing feature or a built-in proxy).
The ability to control and limit the scan to internal or external host (domain/IP).

As a direct implication, the test did NOT include the tools listed in Appendix A – A List of Tools Not Included In The Test.

The purpose of WAVSEP’s test cases is to provide a scale for understanding which detection barriers each scanning tool can bypass, and which vulnerability variations can be detected by each tool.

The Reflected Cross Site Scripting vulnerable pages are pretty standard & straightforward, and should provide reliable basis for assessing the detection capabilities of different scanners.

However, it is important to remember that the SQL Injection vulnerable pages used a MySQL database as a data repository, and thus, the SQL Injection detection results only reflect detection results of SQL Injection vulnerabilities in this type of database; the results that might vary when the back end data repository will be different (a theory that will be verified in the next benchmark).

Description of Comparison Tables

The list of tools tested in this benchmark is organized within the following reports:

List of Tested Scanners (http://wavsep.googlecode.com/files/List%20of%20Tested%20Web%20Application%20Vulnerability%20Scanners%20v1.0.pdf)

Source, License and Technical Details of Tested Scanners (http://wavsep.googlecode.com/files/Details%20of%20Tested%20Web%20Application%20Vulnerability%20Scanners%20v1.0.pdf)

For those of you that wish to get straight to the point, the results of the accuracy assessment are organized within the following reports:

Benchmark Results – Reflected XSS Detection Accuracy (http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20RXSS%20Detection%20Accuracy%20v1.0.pdf)

Benchmark Results – SQL Injection Detection Accuracy – Total (http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20v1.0.pdf)

Benchmark Drilldown – Blind SQL Injection Detection (http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20-%20Blind%20v1.0.pdf)

Benchmark Drilldown – Erroneous SQL Injection Detection

(http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20-%20ErrorBased%20v1.0.pdf)

Additional information was gathered during the benchmark, including information related to the different features of various scanners. These details are organized in the following reports, and might prove useful when searching for tools for specific tasks or tests:

Comparison of Active Vulnerability Detection Features (http://wavsep.googlecode.com/files/Active%20Vulnerability%20Detection%20Features%20Comparison%20v1.0.pdf)

Comparison of Complementary Scanning Features - Passive Analysis, CGI Scanning, Brute Force, etc (http://wavsep.googlecode.com/files/Complementary%20Scan%20Features%20Comparison%20v1.0.pdf)

Comparison of Usability, Coverage and Scan Initiation Features (http://wavsep.googlecode.com/files/List%20of%20Scanner%20Features%20%281%20of%203%29%20v1.0.pdf)

Comparison of Authentication, Scan Control and Connection Support Features (http://wavsep.googlecode.com/files/List%20of%20Scanner%20Features%20%282%20of%203%29%20v1.0.pdf)

Comparison of Advanced and Uncommon Features (http://wavsep.googlecode.com/files/List%20of%20Scanner%20Features%20%283%20of%203%29%20v1.0.pdf)

Information regarding the scan logs, list of untested tools and abnormal behaviors of scanners can be found in the article appendix sections:

The following appendix report contains a list of scanners that were not included in the test:

Appendix A – A List of Tools not included in the Test (The end of the article)

The scan logs (describing the executing process and configuration of each scanner) can be viewed in the following appendix report: Appendix B – WAVSEP Scanning Logs (http://wavsep.googlecode.com/files/WavsepScanLogs%20v1.0.pdf)

During the benchmark, certain tools with abnormal behavior were identified; the list of these tools is presented in the following appendix report:

Appendix C – Scanners with Abnormal Behavior (The end of the article)

List of Tested Scanners

The following report (PDF) contains the list of scanners tested in this benchmark, in addition to their version, their author and their status: http://wavsep.googlecode.com/files/List%20of%20Tested%20Web%20Application%20Vulnerability%20Scanners%20v1.0.pdf

For those of you that want a quick glimpse, the following scanners were tested in the benchmark:

Acunetix Web Vulnerability Scanner (Free Edition), aidSQL, Andiparos, arachni, crawlfish, Gamja, Grabber, Grendel Scan, iScan, JSKY Free Edition, LoverBoy, Mini MySqlat0r, Netsparker Community Edition, N-Stalker Free Edition, Oedipus, openAcunetix, Paros Proxy, PowerFuzzer, Priamos, ProxyStrike, Sandcat Free Edition, Scrawler, ScreamingCSS, ScreamingCobra, Secubat, SkipFish, SQID (SQL Injection Digger), SQLiX, sqlmap, UWSS(Uber Web Security Scanner), VulnDetector, W3AF, Wapiti, Watobo, Web Injection Scanner (WIS), WebCruiser Free Edition, WebScarab, WebSecurify, WSTool, Xcobra, XSSer, XSSploit, XSSS, ZAP.

Source, License and Technical Details of Tested Scanners

The following report (PDF) contains a comparison of licenses, development technology and sources (home page) of different scanners: http://wavsep.googlecode.com/files/Details%20of%20Tested%20Web%20Application%20Vulnerability%20Scanners%20v1.0.pdf

Comparison of Active Vulnerability Detection Features

The following report (PDF) contains a comparison of active vulnerability detection features in the various scanners: http://wavsep.googlecode.com/files/Active%20Vulnerability%20Detection%20Features%20Comparison%20v1.0.pdf

Aside from the Count column (which represents the total amount of active vulnerability detection features supported by the tool, not including complementary features such as web server scanning and passive analysis), each column in the report represents an active vulnerability detection feature, which translates to the exposure presented in the following list:

SQL – SQL Injection

BSQL – Blind SQL Injection

RXSS – Reflected Cross Site Scripting

PXSS – Persistent / Stored Cross Site Scripting

DXSS – DOM XSS

Redirect – External Redirect / Phishing via Redirection

Bck – Backup File Detection

Auth – Authentication Bypass

CRLF – CRLF Injection / Response Splitting

LDAP – LDAP Injection

XPath – X-Path Injection

MX – MX Injection

Session Test – Session Identifier Complexity Analysis

SSI – Server Side Include

RFI-LFI – Directory Traversal / Remote File Include / Local File Include (Will be separated into different categories in future benchmarks)

Cmd – Command Injection / OS Command Injection

Buffer – Buffer Overflow / Integer Overflow (Will be separated into different categories in future benchmarks)

CSRF – Cross Site Request Forgery

A-Dos – Application Denial of Service / RegEx DoS

Comparison of Complementary Scanning Features

The following report (PDF) contains a comparison of complementary vulnerability detection features in the various scanners: http://wavsep.googlecode.com/files/Complementary%20Scan%20Features%20Comparison%20v1.0.pdf

In order to clarify what each column in the report table means, use the following interpretation:

Web Server Hardening – plugins that scan for HTTP method support (Trace, WebDAV), directory listing, Robots and cross-domain information disclosure, version specific vulnerabilities, etc.

CGI Scanning - Default files, common vulnerable applications, etc.

Passive Analysis – security tests that don’t require any actual attacks, and are based instead on information gathering and analysis of responses, including certificate & cipher tests, gathering of comments, mime type analysis, autocomplete detection, insecure transmission of credentials, google hacking, etc.

File Enumeration – directory and file enumeration features.

Comparison of Usability and Coverage Features

The following report (PDF) contains a comparison of usability, coverage and scan initiation features of different scanners:

http://wavsep.googlecode.com/files/List%20of%20Scanner%20Features%20%281%20of%203%29%20v1.0.pdf

Configuration & Usage Scale

Very Simple - GUI + Wizard

Simple - GUI with simple options, Command line with scan configuration file or simple options

Complex - GUI with numerous options, Command line with multiple options

Very Complex - Manual scanning feature dependencies, multiple configuration requirements

Stability Scale

Very Stable - Rarely crashes, Never gets stuck

Stable - Rarely crashes, Gets stuck only in extreme scenarios

Unstable - Crashes every once in a while, Freezes on a consistent basis

Fragile – Freezes or Crashes on a consistent basis, Fails performing the operation in many cases

(Unlike the accuracy values presented in the benchmark for W3AF, which are up date, the stability values for W3AF represent the condition of 1.0-RC3, and not 1.0-RC4; the values will be updated in the next benchmark, after the new version will be thoroughly tested)

Performance Scale

Very Fast - Fast implementation with limited amount of scanning tasks

Fast - Fast implementation with plenty of scanning tasks

Slow - Slow implementation with limited amount of scanning tasks

Very Slow - Slow implementation with plenty of scanning tasks

Comparison of Connection and Authentication Features

The following report (PDF) contains a comparison of connection, authentication and scan control features of different scanners:

http://wavsep.googlecode.com/files/List%20of%20Scanner%20Features%20%282%20of%203%29%20v1.0.pdf

Comparison of Advanced Features

The following report (PDF) contains a comparison of advanced and uncommon scanner features:

http://wavsep.googlecode.com/files/List%20of%20Scanner%20Features%20%283%20of%203%29%20v1.0.pdf

Benchmark Results – Reflected XSS Detection Accuracy

The results of the Reflected Cross Site Scripting (RXSS) benchmark are presented in the following report (PDF format):

http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20RXSS%20Detection%20Accuracy%20v1.0.pdf

The results only include vulnerable pages linked from the index-xss.jsp index page (RXSS-GET or RXSS-POST directories, in addition to the RXSS-FalsePositive directory). XSS Vulnerable locations in the SQL injection vulnerable pages were not taken into account, since they don’t necessarily represent a unique scenario (or at least not until the “layered vulnerabilities” scenario will be implemented).

Benchmark Results – SQL Injection Detection Accuracy

The overall results of the SQL Injection benchmark are presented in the following report (PDF format):

http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20v1.0.pdf

Benchmark Drilldown – Erroneous SQL Injection Detection

The results of the Error-Based SQL Injection benchmark are presented in the following report (PDF format):

http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20-%20ErrorBased%20v1.0.pdf

Benchmark Drilldown – Blind SQL Injection Detection

The results of the Blind SQL Injection benchmark are presented in the following report (PDF format):

http://wavsep.googlecode.com/files/Web%20Application%20Scanner%20SQLi%20Detection%20Accuracy%20-%20Blind%20v1.0.pdf

Initial Analysis & Conclusions

After performing an initial analysis on the data, I have come to a simple conclusion as to which combination of tools will be the most effective in detecting Reflected XSS vulnerabilities in the public (unauthenticated) section of a tested web site, while providing the least amount of false positives:

Netsparker CE (42 cases), alongside Acunetix Free Edition (38 cases, including case 27 which is missed by Netsparker), alongside Skipfish (detects case 12 which is missed by both tools). I’d also recommend executing N-Stalker on small applications since it able to detect certain cases that none of the other tested tools can (but the XSS scanning feature is limited to 100 URLs).

Using Sandcat or Proxy Strike alongside Burp Spider/Paros Spider/External Spider can help detect additional potentially vulnerable locations (cases 10, 11, 13-15 and 17-21) that could be manually verified by a human tester.

So combining four tools will give the best possible result of RXSS detection in the unauthenticated section of an application, using today’s free & open source tools… WOW, it took some time to get to that conclusion. However, scanning the public section of the application is one thing, and scanning the internal section (authenticated section) of the application is another; effectively scanning the authenticated section requires various features such as authentication support, URL scanning restrictions, manual crawling (in case damage might be caused from crawling certain URLs), etc; so the conclusions for the public section are not necessarily fit for the internal section.

During the next few days, I’ll try and analyze the results and come to additional conclusions (internal RXSS scanning, external & internal SQLi scanning, etc). Simply check my blog in a few days to see which conclusions were already published.

An updated benchmark document will be released in the WAVSEP project homepage after each addition, conclusion or change.

A comment about accuracy and inconsistent results

During the benchmark, I have executed each tool more than once, and on rare occasions, dozens of times. I have discovered that some of the tools have inconsistent results in certain fields (particularly SQL injection). The following tools produced inconsistent results in the SQLi detection field: Skipfish (my guess is the inconsistencies are related to crawling problems and connection timeouts), Oedipus, and probably a couple of others that I can’t remember.

It is important to note that the 100% Reflected XSS detection ratio that Sandcat and ProxyStrike produce comes with a huge amount of false positives, a fact that signifies that the detection algorithm works more like a passive scanner (such as watcher by casaba), and less like an active intelligent scanner that verifies that the injection returned is sufficient to exploit the exposure in the given scope. This conclusion does not necessarily pinpoint anything about other features of these scanners (for example, the SQL injection detection module of proxystrike is pretty decent), or presume that the XSS scanning features of these tools are “useless”; on the contrary, these tools can be used as means to obtain more leads for human verification, and can be very useful in the right context.

Furthermore, the 100% SQL Injection detection ratio of Wapiti needs to be further investigated since andiparos produced the same ratio when the titles of the various pages contained the word SQL (which is part of the reason that in the latest version of WAVSEP, this word does not appear anywhere).

Additional conclusions will follow.

So What Now?

So now that we have plenty of statistics to analyze, and a new framework for testing scanners, it’s time to discuss the next phases.

Although the calendar tells me that it took me 9 months to conduct this research, in reality, it took me a couple of years to collect all the tools, learn how to install and use them, gather everything that was freely available for more than 5 minutes and test them all together.

However, since my research led me to develop a whole framework for benchmarking (aside from the WAVSEP project which was already published), I believe (or at least hope) that thanks to the platform, future benchmarks will be much easier to conduct, and in fact, I’m planning on updating the content of the web site (http://sectooladdict.blogspot.com/) with additional related content on a regular basis.

In addition to different classes of benchmarks, the following goals will be in the highest priority:

Improve the testing framework (WAVSEP); add additional test cases and additional security vulnerabilities.
Perform additional benchmarks on the framework, and on a consistent basis. I'm currently aiming for one major benchmark per year, although I might start with twice per year, and a couple of initial releases that might come even sooner.
Publish the results of tests against sample vulnerable web applications, so that some sort of feedback on other types of exposures will be available (until other types of vulnerabilities will be implemented in the framework), as well as features such as authentication support, crawling, etc.
Gradually develop a framework for testing additional related features, such as authentication support, malformed HTML tolerance, abnormal response support, etc.
Integration with external frameworks for assessing crawling capabilities, technology support, etc.

The different vendors will receive an email message from an email address designated for communicating with them. I urge them to try and contact me through that address, and not using alternative means, so I’ll be able to set my priorities properly. I apologize in advance for any delays in my responses in the next few weeks.

Appendix A – A List of Tools Not Included In the Test

The benchmark focused on web application scanners that are free to use (freeware and/or open source), are able to detect either Reflected XSS or SQL Injection vulnerabilities, and are also able to scan multiple URLs in the same execution.

As a direct implication, the test did NOT include the following types of tools:

· Commercial scanners - The commercial versions of AppScan, WebInspect, Cenzic, NTOSpider, Acunetix, Netsparker, N-Stalker, WebCruiser, Sandcat and many other commercial tools that I failed to mention. Any tool in the benchmark that holds the same commercial name is actually a limited free version of the same product, and does not refer (or even necessarily reflect on) the full product.
· Online Scanning Services – Online applications that remotely scan applications, including (but not limited to) Zero Day Scan, Appscan On Demand, Click To Secure, QualysGuard Web Application Scanning (Qualys), Sentinel (WhiteHat), Veracode (Veracode), VUPEN Web Application Security Scanner (VUPEN Security), WebInspect (online service - HP), WebScanService (Elanize KG), Gamascan (GAMASEC – currently offline), etc.
· Scanners without RXSS / SQLi detection features, including (but not limited to):

o LFIMap

o phpBB-RFI Scanner

o DotDotPawn

o CSRF Tester

o etc

· Passive Scanners (response analysis without verification), including (but not limited to):

o Watcher (Fiddler Plugin by Casaba Security)

o Skavanger (OWASP)

o Pantera (OWASP)

o Rat proxy (Google)

o Etc

· Scanners for specific products or services (CMS scanners, Web Services Scanners, etc), including (but not limited to):

o WSDigger

o Sprajax

o ScanAjax

o Joomscan

o Joomlascan

o Joomsq

o WPSqli

o etc

· White box & Code Review Application Scan Tools, including (but not limited to):

o PuzlBox

o Inspathx

o etc

· Uncontrollable Scanners - scanners that can’t be controlled or restricted to scan a single site, since they either receive the list of URLs to scan from Google Dork, or continue and scan external sites that are linked to the tested site. This list currently includes the following tools (and might include more):

o Darkjumper 5.8 (scans additional external hosts that are linked to the given tested host)

o Bako's SQL Injection Scanner 2.2 (only tests sites from a google dork)

o Serverchk (only tests sites from a google dork)

o XSS Scanner by Xylitol (only tests sites from a google dork)

o Hexjector (by hkhexon) – also falls into other categories

o etc

· Deprecated Scanners - incomplete tools that were not maintained for a very long time. This list currently includes the following tools (and might include more):

o etc

· De facto Fuzzers – tools that scan applications in a similar way to a scanner, but where the scanner attempts to conclude whether or not the application or is vulnerable (according to some sort of “intelligent” set of rules), the fuzzer simply collects abnormal responses to various inputs and behaviors, leaving the task of concluding to the human user.

o Lilith 0.4c/0.6a (both versions 0.4c and 0.6a were tested, and although the tool seems to be a scanner at first glimpse, it doesn’t perform any intelligent analysis on the results).

· Fuzzers – scanning tools that lack the independent ability to conclude whether a given response represents a vulnerable location, by using some sort of verification method (this category includes tools such as JBroFuzz, Firefuzzer, Proxmon, st4lk3r, etc). Fuzzers that had at least one type of exposure that was verified were included in the benchmark (Powerfuzzer).
· CGI Scanners: vulnerability scanners that focus on detecting hardening flaws and version specific hazards in web infrastructures (Nikto, Wikto, WHCC, st4lk3r, N-Stealth, etc)
· Single URL Vulnerability Scanners - scanners that can only scan one URL at a time, or can only scan information from a google dork (uncontrollable).

o Havij (by itsecteam.com)

o Hexjector (by hkhexon)

o Simple XSS Fuzzer [SiXFu] (by www.EvilFingers.com)

o Mysqloit (by muhaimindz)

o PHP Fuzzer (by RoMeO from DarkMindZ)

o SQLi-Scanner (by Valentin Hoebel)

o Etc.

· The following scanners:

o sandcatCS 4.0.3.0 - Since sandcat 4.0 free edition, a more advanced tool from the same vendor is already tested in the benchmark.

o GNUCitizen JAVASCRIPT XSS SCANNER - since WebSecurify, a more advanced tool from the same vendor is already tested in the benchmark.

o Vulnerability Scanner 1.0 (by cmiN, RST) - since the source code contained traces for remotely downloaded RFI lists from locations that do not exist anymore. I might attempt to test it anyway in the next benchmark.

o XSSRays 0.5.5 - I might attempt to test it in the next benchmark.

o XSSFuzz 1.1 - I might attempt to test it in the next benchmark.

o XSS Assistant - I might attempt to test it in the next benchmark.

· Vulnerability Detection Helpers – tools that aid in discovering a vulnerability, but do not detect the vulnerability themselves; for example:

o Exploit-Me Suite (XSS-Me, SQL Inject-Me, Access-Me)

o Fiddler X5s plugin

· Exploiters - tools that can exploit vulnerabilities but have no independent ability to automatically detect vulnerabilities on a large scale. Examples:

o MultiInjector

o XSS-Proxy-Scanner

o Pangolin

o FGInjector

o Absinth

o Safe3 SQL Injector (an exploitation tool with scanning features (pentest mode) that are not available in the free version).

o etc

· Exceptional Cases

Appendix B – WAVSEP Scanning Logs

The execution logs, installation steps and configuration used while scanning with the various tools are all described in the following report (PDF format):

http://wavsep.googlecode.com/files/WavsepScanLogs%20v1.0.pdf

Appendix C – Scanners with Abnormal Behavior

During the assessment, parts of the source code of open source scanners and the HTTP communication of some of the scanners was analyzed; some tools behaved in an abnormal manner that should be reported:

· Priamos IP Address Lookup – The tool Priamos attempts to access “whatismyip.com” (or some similar site) whenever a scan is initiated (verified by channeling the communication through Burp proxy). This behavior might derive from a trojan horse that infected the content on the project web site, so I’m not jumping to any conclusions just yet.
· VulnerabilityScanner Remote RFI List Retrieval (listed in the scanners that were not tested, appendix A, developed by a group called RST, http://pastebin.com/f3c267935) – In the source code of the tool VulnerabilityScanner (a python script), I found traces for remote access to external web sites for obtaining RFI lists (might be used to refer the user to external URLs listed in the list). I could not verify the purpose of this feature since I didn’t manage to activate the tool (yet); in theory, this could be a legitimate list update feature, but since all the lists the tool uses are hardcoded, I didn’t understand the purpose of the feature. Again, I’m not jumping to any conclusions; this feature might be related to the tool’s initial design, which was not fully implemented due to various considerations. I’ll try and drill deeper in the next benchmark (and hopefully, manage to test the tool’s accuracy as well).

Where to begin…

2010-04-16T05:18:00.000-07:00

There’s a ton of security tools out there.

From the point of view of security consultants (pen-testers to be exact), most of these tools are there to make their job easier, aid them in improving test results and enable them to reduce the time required to perform their tests.

But that’s not how it works in reality...

Lately there so much tools that it’s hard to know what to use;

Some of these tools are obsolete, some contain numerous bugs that prevents their execution from being effective, and some simply don’t justify the time required to execute them. On the other hand, some relatively anonymous tools generate spectacular results and can provide great benefits, but for some unknown reason (that has nothing to do with their quality), do not receive the credit and recognition they deserve.

After several years in the profession, and as an official security tool addict, I have decided to invest some time in sharing my experiences from using these tools, and from time to time, publish detailed benchmarking articles that compare between the various tools features, usability, accuracy, advantages and disadvantages.

In hopes that the community will benefit from this initiative, and in hopes that it will inspire the various tool vendors to compete and improve their tools,

Let the contest begin.