I wanted to blog about a disturbing trend that I’ve been seeing recently. I might be slightly biased here, actually I know I am, but hear me out on this one for just a moment and I think you’ll agree with what I have to say.
I had the chance recently to review the results of a penetration test report of a company as a ‘second opinion.’ Turns out the company hired the same company that developed their Web-site, a web-development company, to also perform the security testing for PCI evaluation readiness — yikes! The web-development company used a formidable set of tools to do the analysis including prominent names like Nessus, Eeye’s Retina, Nikto, Nmap, GFI, etc., and yet I spent about 20 minutes on that site using tools developed here at Impacta and manual techniques, and dug up a handful of SQL injection, XSS and blatant configuration issues that were not included in the report. Worst of all, the penetration test was billed out as a 2 month engagement for a single online system. So not only do they not have a good understanding of their PCI compliance (at least the Web portion), they probably wasted about $32-40k for work they will probably have to do again.
I’ll cut straight to the chase: do not hire Web-development companies to assess the security of your online presence, especially when they created that online presence! That’s like:
- Having students grade their own finals
- Having baseball players test themselves for steroid use
- Having a security assessment company design your Web site for you :P
None of these make sense. You can say I am biased (I am. Impacta does penetration tests, security code reviews and other security services), but the whole goal of penetration tests and other assessment activities is to have ”an independent assessment of the risks present in a system.” Keyword: independent.
Of course Web-development companies are going to say that they are experts in security. Why? Because that’s what their competition is also saying. Just keep thinking ‘independent review, independent review’ and you’ll be fine.
–Kevin
20 Comments
I agree with Kevin’s take. You can never replace human experience regardless of the tool set being used. Anyone can run a series of tool, but without proper experience the deliverable is worthless.
I always suggest that the client ask for reference’s feedback on deliverables. This is a good place to start your due diligence.
Lance
This is a simple control failure; maker-checker rule overlooked. And not just in pen testing, I have seen it happening in almost all fields of security, especially when it comes to processes. It surprising how companies that spend so much on security forget the basic rules of security at times.
Bhavuk
Couple of points that I would like to make.
1. It is a well known fact that “A Fool with a Tool is still a Fool.”
2. Though tools may be used to autmate certain repeated tasks, but solely relying on tools for best results is sheer foolishness, no matter whatever combination of best tools being used.
3. Wasting thousands of Dollars would be a secondary concern, the primary concern is that the application still has many critical vulnerabilities, which when exploited might cause unimaginable damage to the organization.
4. Prior to signing a contract with a 3rd party, one should be very clear on the kind of assessment being done, the methodology involved etc to avoid any complications in future.
I see the same problem in organizations where the Information Security staff reports up through the IT department. The prison guards are reporting to the prisoners.
@Patrick – I would urge a little caution in generalising like this. If the IS staff are not responsible for admin/ops then depending on the requirements, it may be reasonable for them to perform certain types of tests. Also, having built an internal pen-test capability within a large enterprise, I can state from experience that not all pen-test companies can compete when it comes to business understanding/knowledge; i.e. they can’t really model an inside attacker. For perimeter/Internet facing testing, I agree that an independent 3rd party is preferred for both reasons of credibility with auditors but also that a well chosen pen-test company should have more in-depth web-specific assessment skills than an in-house team (who in turn may have better skillsets on software typically deployed *inside* an enterprise).
Horses for courses.
Cheers
Craig
I’m not at all sure that this is a new trend. There have always been vendors peddling security assessment services that are primarily snake-oil, and there have always been immature clients who can’t differentiate between a quality assessment and snake-oil.
But it isn’t just at the immature end of the scale that this kind of scenario happens; all too often it is an affliction of larger outsourcing arrangements too. Global megacorp inc. will outsource their IT delivery in general to a supplier, who will be providing both implementation & management services, and also be responsible for security assurance in the same environments.
In the land of the half-arsed, the two-buttocked man is king…
Martin…
“In the land of the half-arsed, the two-buttocked man is king…” (From Martin O’Neal’s response)
Ha ha! Seriously, Martin makes a great point. This isn’t something new and stems in part due to clients not being properly educated prior to engaging companies for security assessment work. Thanks for sharing your thoughts!
–Kevin
I agree with you Kevin, those doing the work should not be assessing their own work. However, I believe it is a much larger issue.
For quite some time there has been little, if any, independence in the security industry. Most standards have yet to address this issue and it leads to bad security practices and less than thorough testing.
Similarly, assessors should not be remediators and vice versa. But let’s face it, assessments are a commodity these days and often loss leaders for remediation companies. Not to mention the fact that most clients just want to pass their assessment and would rather do it open-book style, if and when at all possible.
I applaud the folks that have begun to develop penetration assessment standards and who work so diligently to address these issues. But for things to really change, it will require security leadership and standards bodies to formulate better processes as well as checks and balances.
A difficult task but much needed in this industry, thanks for raising the topic.
Shannon, great points! Assessments have indeed (unfortunately) become a commodity these days. Penetration/assessments standards … now that’s a tough subject. One difficulty with these is the threat landscape keeps changing and sometimes quite often. Thanks again for sharing your thoughts!
–Kevin
The question is ultimately due diligence, whether that diligence is taken with a vendor or internally.
Putting on my ‘business hat’ I can understand why it makes sense for the initial review to be conducted by the ‘implementation company’. However, with that said, I think it is important to note that the client actually went outside the company and got a ‘post-implementation review’ of their investment.
The fact that the original poster is critical of their client is almost shameful. I’m going to go out on a limb here and say ‘kudos to the client’ for asking for a second opinion.
I deal with this on a daily basis, whereby a vendor provides engineering support to engineers trying to build ‘to spec’. Based on that, another internal team reviews what the vendor engineering claims were provided. In the middle of that process, for example I’m now leading a team of external pen testers to provide a disinterested third party review of both the vendors and the engineering teams architecture and opine on their findings that may or may not impact design.
Be careful what you wish for (Impacta), whining about due diligence may be a false positive in the face of (in)effective advertising. (read: why would I hire you for a second opinion, when you complain about my previous business decisions? Why can’t you be happy you won a review contract in these tough times?)
Just my .02,
Cheers,
Thomas
Thanks Thomas for your comments and candor, I always encourage and appreciate feedback like this. A couple of points to help clarify:
1. I have a good enough relationship with this company’s management that I can ask them for permission first to blog about the lessons learned here (of course without using their name) since it can help other people who may not know they are in the same predicament. They were all for it, so I went ahead and shared with the community.
2. This company is not a ‘client’, in fact I did the cursory review them pro bono because I felt (a) they could use the help and (b) it doesn’t cost me much to punch in their IP address into my tool and run some quick analysis for them to help them out. I don’t believe I ever referred to this company ever in my post as a ‘client’ or indicated that some contract was won by Impacta, so I am not sure how this conclusion was made. So yes, if they were an actual client of mine and they weren’t very receptive about sharing lessons learned with the greater community then of course I would respect their decision and not write about it.
Perhaps, for point #1 I could have placed a note saying “posted with permission … etc.”, but given the relationship I have with that company and their willingness to share lessons learned I didn’t feel it was necessary. For point #2 about misinterpreting this blog post as negatively critizing a client not sure what I can do about this. Either way, as always thanks for sharing your thoughts (your comments gave me a great idea for another future post!).
–Kevin
Kevin,
First off, Excellent Article!
I wanted to point out that I as a web developer agree with you 100%. I am an independent developer, and – although I do use a series of tools on my own developments, and keep XSS / SQL Injection in mind to the best of MY ability, I can honestly say that I have never claimed to be a security expert.
In fact I always recommend an outside (other than me) independent security expert, yes expert always test my code, and part of purchasing such work from me I will fix holes found (if any).
I believe that I am a far better programmer (PHP) than I am a security expert. If a flaw is shown to me (how they got in) I have to date been able to take care of it, with that being said, the old saying “Jack of all trades, master of none” comes to mind.
Not to discount that a web-developer company can indeed employ security experts in house that are just as capable as a security company, I still believe you should leave each aspect of the development cycle to the expert for the most effective product.
My entire point really is – I am working from the side of a developer, and I agree and recommend to my own clients – a third party security EXPERT always check over the application(s) before considering it secure.
Michael,
You are my developer HERO! You make a fabulous point that web-developer companies can still employ security experts in house (business hat on here: they earned the business so keep the work on their paper), but just make sure they bring the right tools and people to the job. The world needs more developers like you! Thanks for taking the time to post!
–Kevin
Kevin,
I’m a little biased here myself as I work for eEye Digital Security. I think it’s important for your readers to understand that there are 2 versions of Retina. Retina Network Security Scanner, which is probably what your example compnay tried to use in their testing, and Retina Web Security Scanner which is relatively new. I’ve yet to find a scanner that returns more thourough results with less false positives than our Retina Web Security Scanner.
However, I do agree with your point. I believe companies should employ an outside party to scan their web apps for SQL Injection and XSS. I know I always made a 100 when I graded my own paper.
:-)
Hey Richard, thanks for taking the time to post as well. The tools aren’t really the problem, they are all good tools, just when the people using the tools aren’t properly trained then we start seeing stuff like this. Great work over there at eEye!
–Kevin
The truth is, companies do this sort of thing all the time. One of the most common examples I see with hi-tech companies is having the developers conducting the quality tests on what they developed! Or, just as insidious, is having the testing group reporting to the development manager (that conflict of interest would never cause the testing results to be changed, would it?) And yet, even when it is pointed out to them, often they still don’t see, or at least will not admit to, the inherent conflict of interest.
Hey Donna, thanks also for sharing and taking the time to post a comment. You got me thinking more about those companies that have good intentions (i.e to perform actual independent reviews) but often times they don’t have the resources so they do what they can, which unfortunately sometimes turns into reviews like these. Now you’ve got me really thinking about how these folks can be help (especially with the current economic conditions).
–Kevin
Agreed. However, I’d be interested to see what portion of the PCI standard they were testing to. Meaning, 11.x “Penn Test” is just a network scan and wouln’t really turn up application flaws due to custom code (i.e. “you are using software version X with XSS vulnerabilities” versus exercising the application to determine if a flaw truly exists, which it appears you did).
PCI 6.6 should be looking at the application layer and they should be using web application scanners plus human testing to find flaws (as you did).
But, as stated earlier, a fool and their tool …
Hi Mike I think the tools were chosen based on their understanding of what constituted “secure” and PCI. Anyhow, you’ve got me thinking. Maybe the standards and guidance should be simple enough that even if the wrong people were doing the work they could conceivably get to the right results.
For instance, replacing your windshield wipers is easiest enough for anyone to do. That’s because the instructions and objectives are clear, and there aren’t that many moving parts. You don’t have to be a mechanic to install windshield wipers. What if you could do the same with security, but only call in security experts when you really really have to (have you tried changing the windshield wipers on a BMW!?). I don’t know if you meant to go down this route, but now you have me thinking …
Thanks again Mike for sharing and taking the time to post a comment!
–Kevin
Hi Kevin,
I totally agree with your assessment – however I want to share an observation:
A penetration test is essentially an audit – usually part of a bigger system audit. One of the first guideline of any audit is that auditor must not be involved in any activity related to the project which can compromise the auditor’s independence and objectivity.By agreeing to perform an audit of the system the company had developed, they have essentially shown any lack of audit understanding, capabilities and understanding and thus the results. The company who commissioned the audit is also responsible for this audit failure or more precisely the management of the company who by the looks of it just wanted it done, perhaps wanted a PCI stamp – which didn’t fly with PCI, as you were asked to do a review.
Ahsan
3 Trackbacks/Pingbacks
[...] Kevin Lam’s blog at Impacta to the list of blogs I regularly follow. In a recent entry, he blogs that he’s been seeing companies use the same company that designed their website to perform [...]
[...] How NOT to Conduct a Penetration Test: Recent Rises in a Disturbing Trend [...]
[...] Kevin Lam’s blog at Impacta to the list of blogs I regularly follow. In a recent entry, he blogs that he’s been seeing companies use the same company that designed their website to perform [...]