Comments on: How NOT to Conduct a Penetration Test: Recent Rises in a Disturbing Trend

By: Roger's Information Security Blog » Blog Archive » Web Vulnerability Analysis the Wrong Way

Sun, 06 Jun 2010 23:16:06 +0000

[...] Kevin Lam’s blog at Impacta to the list of blogs I regularly follow. In a recent entry, he blogs that he’s been seeing companies use the same company that designed their website to perform [...]

By: Middle Zone Musings » What I Learned From 2008 - Kevin Lam

Middle Zone Musings » What I Learned From 2008 - Kevin Lam — Wed, 21 Jan 2009 12:01:35 +0000

[...] How NOT to Conduct a Penetration Test: Recent Rises in a Disturbing Trend [...]

By: Ahsan Mir

Ahsan Mir — Sun, 04 Jan 2009 04:15:42 +0000

Hi Kevin,

I totally agree with your assessment – however I want to share an observation:

A penetration test is essentially an audit – usually part of a bigger system audit. One of the first guideline of any audit is that auditor must not be involved in any activity related to the project which can compromise the auditor’s independence and objectivity.By agreeing to perform an audit of the system the company had developed, they have essentially shown any lack of audit understanding, capabilities and understanding and thus the results. The company who commissioned the audit is also responsible for this audit failure or more precisely the management of the company who by the looks of it just wanted it done, perhaps wanted a PCI stamp – which didn’t fly with PCI, as you were asked to do a review.
Ahsan

By: Kevin Lam (IMPACTA)

Kevin Lam (IMPACTA) — Sat, 03 Jan 2009 00:23:35 +0000

Hi Mike I think the tools were chosen based on their understanding of what constituted “secure” and PCI. Anyhow, you’ve got me thinking. Maybe the standards and guidance should be simple enough that even if the wrong people were doing the work they could conceivably get to the right results.

For instance, replacing your windshield wipers is easiest enough for anyone to do. That’s because the instructions and objectives are clear, and there aren’t that many moving parts. You don’t have to be a mechanic to install windshield wipers. What if you could do the same with security, but only call in security experts when you really really have to (have you tried changing the windshield wipers on a BMW!?). I don’t know if you meant to go down this route, but now you have me thinking …

Thanks again Mike for sharing and taking the time to post a comment!

–Kevin

By: Mike

Mike — Fri, 02 Jan 2009 15:07:39 +0000

Agreed. However, I’d be interested to see what portion of the PCI standard they were testing to. Meaning, 11.x “Penn Test” is just a network scan and wouln’t really turn up application flaws due to custom code (i.e. “you are using software version X with XSS vulnerabilities” versus exercising the application to determine if a flaw truly exists, which it appears you did).

PCI 6.6 should be looking at the application layer and they should be using web application scanners plus human testing to find flaws (as you did).

But, as stated earlier, a fool and their tool …

By: Kevin Lam (IMPACTA)

Kevin Lam (IMPACTA) — Thu, 01 Jan 2009 19:30:08 +0000

Hey Donna, thanks also for sharing and taking the time to post a comment. You got me thinking more about those companies that have good intentions (i.e to perform actual independent reviews) but often times they don’t have the resources so they do what they can, which unfortunately sometimes turns into reviews like these. Now you’ve got me really thinking about how these folks can be help (especially with the current economic conditions).

–Kevin

By: Donna

Donna — Thu, 01 Jan 2009 13:37:50 +0000

The truth is, companies do this sort of thing all the time. One of the most common examples I see with hi-tech companies is having the developers conducting the quality tests on what they developed! Or, just as insidious, is having the testing group reporting to the development manager (that conflict of interest would never cause the testing results to be changed, would it?) And yet, even when it is pointed out to them, often they still don’t see, or at least will not admit to, the inherent conflict of interest.

By: Web Vulnerability Analysis the Wrong Way - Industrial Paper Shredder Review -

Web Vulnerability Analysis the Wrong Way - Industrial Paper Shredder Review - — Wed, 31 Dec 2008 06:48:16 +0000

By: Kevin Lam (IMPACTA)

Kevin Lam (IMPACTA) — Wed, 24 Dec 2008 04:03:29 +0000

Hey Richard, thanks for taking the time to post as well. The tools aren’t really the problem, they are all good tools, just when the people using the tools aren’t properly trained then we start seeing stuff like this. Great work over there at eEye!

–Kevin

By: Richard McCrohan

Richard McCrohan — Tue, 23 Dec 2008 21:44:36 +0000

Kevin,

I’m a little biased here myself as I work for eEye Digital Security. I think it’s important for your readers to understand that there are 2 versions of Retina. Retina Network Security Scanner, which is probably what your example compnay tried to use in their testing, and Retina Web Security Scanner which is relatively new. I’ve yet to find a scanner that returns more thourough results with less false positives than our Retina Web Security Scanner.

However, I do agree with your point. I believe companies should employ an outside party to scan their web apps for SQL Injection and XSS. I know I always made a 100 when I graded my own paper.

:-)