In the 11+ years Depth has been in business we’ve had the opportunity to see some less than stellar work as far as assessment services go. Our clients often send us assessment reports they’ve received from other security firms. Sometimes they want us to check remediation status on a single item. Other times they aren’t sure whether a given vulnerability is even a vulnerability or why a vulnerability is rated “Critical” when it seems relatively minor. Often, they just want a general opinion on the quality of prior work or want to know if they overpaid.
I’d like to tell you that the quality of what we’ve seen follows a normal distribution curve with just as many examples of poor-quality work as there are of high quality. Admittedly it’s a small and biased sample size but alarmingly, what we’ve observed has been largely terrible in its quality. InfoSec is slowly getting more mature as an industry, but it still seems to behave like a bubble, with high demand, low supply, and a low barrier to entry. To be fair, I know there are some terrific companies out there as well, that we simply don’t run across very often.
I thought I’d try and help arm consumers of these services with some red flags to look out for. Specifically, I’ll be talking about runtime/black-box network and application security assessments. Some of these red flags are more significant than others. Also, any one of these alone may not be significant but if you’re considering contracting with a third party who show indications of several of these items, you might want to think twice. Let me entertain you with some examples and you tell me if any of this sounds familiar to you:
Planning/Scoping
Explanation of Assessment Offerings: Your assessment provider should clearly explain its different service offerings and help you pick the one that’s right for you. We know some companies who have something like 6 different levels of run-time application security assessments. That level of granularity takes some time to explain. If your assessor can’t articulate their offerings to you then it’s unlikely their consultants will apply the appropriate level of detail to the work.
Vulnerability vs Penetration Assessments: Make sure you are comparing apples to apples. It’s very common for companies to sell “Pen Tests” when the results they deliver are nothing of the sort.
Authenticated vs Unauthenticated: If you’re testing an application, it’s important to know whether you’re getting thorough authenticated testing or simple unauthenticated testing of only publicly exposed content. There are cases to be made for either, but this should be discussed up front. If authenticated testing is to be performed, then your assessor should ask you the number of roles and how they are different. If inter-role or inter-tenant authorization are to be checked, they should ask for test accounts representing multiple roles and tenants. If they don’t mention it, they probably haven’t accounted the time for it, and thus aren’t testing it.
Live Addresses VS Address Ranges: For network vuln/pen work, it’s crucial to know how many active IP addresses rather than just a range. For instance, a /23 network could have 500+ live IPs or it could be sparsely populated and contain just a few. If the assessor didn’t ask or look for themselves before sending you a proposal, it’s unlikely they are committed to doing a good job.
Application Size/Complexity: There are many companies who scope application security assessments at a fixed cost per application. Imagine the difference in effort it requires to assess, say, maps.google.com, compared to a small company’s website which contains only brochure-ware, and no dynamic content. Your assessor should ask about the application’s purpose as well as page/screen, parameter, and role counts to determine the proper amount of time it takes to do a good job covering the app.
Application Type (Web App / Mobile App / Web Service / Fat Client): If a consultancy offers you a proposal before ascertaining what type of application they’re assessing, that’s a bad sign. Different types of apps require different methodologies and have different requirements. Kicking off a web services assessment without asking for method documentation, test values, test harnesses, WSDLs, or whatever else necessary to invoke all methods properly is a sign you’re probably about to experience subpar work.
Example Report: You’re going to pay 4+ figures for an assessment and all you will get as a deliverable is a report. If your assessor cannot provide an example of what this report looks like, go another direction. If you do get a sample report, remember that they pre-selected a profusely vulnerable target app and were not constrained by normal project timelines. It should be pretty much immaculate or at least free of obvious errors. If it looks like garbage, yours will be worse.
Remediation Checking: You might want to check and see if your project price includes remediation checking. Some assessment companies provide this and others do not. We include one remediation check with all our external-facing assessments, albeit with some reasonable limitations.
Execution
Kickoff Call: I think it goes without saying that a kickoff call should be held before any work is done. The assessor should question you about the target, walk you through their process at a high level, and cover rules of engagement. You should exchange your points of contact and receive your their direct/mobile numbers. You should be able to directly communicate with your tester without going through a project manager as a middle man. They should provide their source IP addresses if they haven’t already.
Severe Finding Notification: Any reputable assessment firm should provide immediate notification of severe findings, so you don’t have to wait until report delivery to find out.
Requesting Credentials for Internal Assessments: We see this all the time. We get a new customer and they’re like, “Well last year assessment company X asked for Active Directory credentials before conducting the internal pen test.” There are many cases to be made for authenticated testing, but there are studio gangsters out there who just do it as a rule. In a standard pen test, nobody should be given credentials unless the test scenario explicitly calls for it.
NAC/802.1x/ICE/Port Security Exceptions: It’s fine to request a port security exemption for thorough testing, but any test worth its salt should at least attempt to bypass port security. More often than not this can be done by simply printing a test page from a printer and spoofing it’s MAC address.
Authenticated Testing While Logged Out: During application security assessments, it’s often important to test functionality that is hidden behind a login form. During automated scanning phases, it is easy for a tester to think they are testing authenticated functionality, but in fact their session got logged out 3 hours ago. That’s like testing a cinder block wall for weaknesses with your forehead and kneecaps; you shouldn’t pay someone to do that. Make sure you ask for an explanation as to how your tester will ensure their session is valid during authenticated, automated testing.
Subcontracted Assessors: You probably don’t want to hire someone for an assessment only to have your work subcontracted out to an unknown entity foreign or domestic. Unscrupulous companies often use subbing as a means of avoiding the hassle of staffing talented assessors. Maintaining a large bench of qualified employees in this industry is expensive and difficult. Couple that with the fact that InfoSec has a business cycle that often involves nobody doing any work in Q1 and everyone wanting to spend their budgets in Q4. Seriously, watch out for this and ask about it directly. Some security VARs offer all kinds of assessment work but have zero staff to do it in reality.
Certified but not Certified: We have a lot of OSCPs, OSWPs, some OSCEs, and even an OSEE as well. We see assessors all the time, advertising their consultants as having “Industry Certifications” but if you scratch the surface, one in 25 of their people have their CEH, and they have 3 CISSPs. Some of our best consultants have no offensive certs or just have CCIEs, MCSEs, and other infrastructure certs from back in the day because they’re straight OGs.
Results/Report
False Positive City: False positives, findings that aren’t findings, waste your time and indicate a lack of attention to detail on the assessor’s part. There are situations where they can occur, such as when the only way to check a vulnerability relies upon a version banner and exploitation is not possible or desired. However, reports should generally be free of any false positives.
Lack of Understanding: I’ll always remember this time when a client reached out with a report where an assessor had called out certificates that did not match their hostname. That’s a legit problem in that cert names should match the hostnames that users request to prevent SSL/TLS warnings. Problem was, they didn’t check the hostnames; they checked the IP addresses. When you visit an HTTPS site via its IP address rather than its name, you’re usually going to get an error! We’ve seen ridiculous stuff like this rated as high-severity and it just illustrates a complete lack of understanding on the assessor’s part.
Missing Vuln Instances: Great, they found a vulnerability in your application. The problem is that the issue exists across multiple pages and parameters and they only documented the first one they found. That’s a great indicator that you did not get good coverage. Did they just not test the rest of the app or were they just too lazy to document all the instances?
Too Few Findings in Report: Yep, your entire /24 had no vulns sir. That’ll be $10,000 please. It’s called gross negligence and it happens. The only time this should reasonably happen is if you want to verify X network is completely unreachable from Y.
Obvious Scanner Output: We’ve seen some reports where there has been no attempt to change the fact that the consultant clicked “Start” in Nessus and then just exported the results. Unless you’re conducting an evasive red team test or adversarial simulation, any vuln/pen work is going to include some automated scanning phase. However, if there’s no evidence of any manual work, or more than one tool used, how much effort do you think they really put into your assessment?
Pen Tests Sans Pen: An entire report without exploitation is called a vulnerability assessment. We’re still getting terms confused in 2018. If a pen test finding is rated critical or even high, it should usually be exploited unless you requested it not to be. Otherwise what are you paying for? This is especially true on internal pen tests. Just to give you an example, our hit rate for gaining Domain Admin privileges on internal pen tests is over 95%. That’s with no creds, no info, just a network jack. It’s just not very difficult behind the firewall with Active Directory to abuse in all but the most mature InfoSec organizations.
Shallow Pen: As a really simple example we see all the time, say your assessor found a SQL injection bug in your application. A lot of assessors will send a waitfor delay “(0:0:10)”;–, call it exploitation, and then stop. That’s a proof of concept pulled right out of their scanner, not exploitation. What else is possible? Did they check for remote code execution? Did they attempt to enumerate the database or find sensitive data? Is it so bad that you need to involve IR? If it’s a pen test, stopping at a PoC really doesn’t show you too much. That said, a PoC like this is enough to let you know you have a big problem and should probably resolve it very quickly.