Security Testing Programs
Establishing a security testing program is essential to maintaining implemented security measures. It's primary function is to ensure controls are working as expected. It's secondary function is to identify new areas of concern. The third function is to serve as a source of data for metrics, particularly the key indicators or risk and performance.
Without adequate security testing, programs are running at a reduced safety level. Just as industrial testing protects the consumer from faulty tools and materials, so the security testing program protects us from errors and overlooked opportunities.
But what is "adequate" testing?
Adequate testing breaks down into two concepts: testing testing needs identified by risk, and testing aligned to current industry practices. This means that the risk analysis for the business should be highlighting these items, and it means the technical systems people should be identifying best practices in their areas.
What should we be testing?
Testing needs to address all the components identified by the risk assessment process as cost effective. This should include, if your organization has them, External security, Internal security, Physical security, and SDLC security.
External Security Testing
This is the most commonly referred to type of security testing. It is the perspective of the business as viewed from the outside. It addresses the question: can criminals easily steal our valuables or otherwise wreak havoc and disrupt the business. It utilizes vulnerability testing tools and techniques to examine the exposed surfaces. It should not be confused with red team or penetration testing, which is discussed below. Be wary of vendors that conflate the two terms, and be sure of any regulatory or legal requirement that states "pen testing" is required. Failure to use the correct testing methodology may result in distinct legal exposure, overspending with no risk justification, or under-spending that increases risk. The relationship between the terminology is something like this:
Red Teaming > penetration testing > external security testing > vulnerability scans
This relates the types and applies to time, money, disruption potential, and intensity.
At a minimum, vulnerability testing needs to examine the external facing security controls using a quality scan run by a trained expert without limits across all surfaces, detectable or otherwise.
That last "or otherwise" is important, because organizations will frequently discover they have exposed external attack surfaces that they were unaware of. When an organization limits the testing to what it thinks is the real surface, they risk missing the parts that they are unaware of, but which attackers can find quite easily, because they are operating under no such assumption.
It is equally important to be using quality tools and quality personnel. Too often we see vulnerability scans being performed by personnel who have neither the training nor the background to adequately identify genuine risks from false positives. This is a two sided risk, because on the one hand you want the vendor to confirm that identified vulnerabilities are real, but on the other hand, if they lack the talent, they may not find all the vulnerabilities they should and the report may be misleading.
It is essential that different vendors be used in a cyclical fashion using different tooling and experts to ensure that a bias is not masking a vulnerability.
The frequency of this testing should be determined, as always, by risk, but at a minimum:
Annually, and
When any change to the external facing infrastructure or systems is made. That means every time a firewall is patched, every time an access control list is changed, and literally every time anything in that control surface is changed. This "change control testing" does not have to be the entire suite of external testing, but it must be adequate to encompass the changes and any related or interconnected dependencies. Failing to identify dependencies can lead to undetected problems. Have your assumptions reviewed by your auditors as part of the annual process.
Optionally, as indicated by risk or obvious need, the business should also consider the more intense forms of external testing. Red team testing, the most intense form of penetration testing, is a much more serious operation than vulnerability testing, although there are overlaps in the tools and techniques. Red teaming involves trained personnel who perform a series of activities designed to obtain access to key information or perform identified hostile actions.
For example, a red team may be tasked with obtaining access to:
Business secrets.
Executive communications.
Engineering laptops.
Critical systems, especially industrial components such as MRI, lasers, compactors, or any item capable of inflicting damage on people.
Key systems that impact business operations (such as those identified in the DR/BCP lists).
Third party systems and supporting infrastructure.
Each goal would be supported by some identified risk concern and the business would typically only pursue one or two success cases "goals" per engagement, as the costs for red teaming are high and the complexity involved typically means if one goal is met, the rest are also susceptible. Until testing can pass the simplest of these, it's no use pursuing the additional goals, because they all are exposed. Risk determines what goals should be set and in what order.
Red teaming can also optionally involve two other areas of testing:
Physical security, to obtain access to the digital materials.
Social engineering, or human testing, to determine how susceptible personnel are to manipulation, to obtain access to the digital materials.
In between the two ends of the spectrum of external testing are variations that add time, cost and manpower to the testing effort to obtain more specific results. For example, an external security test that goes beyond simple vulnerability scanning by incorporating manual interaction with perceived weaknesses and coordinated exploration to limit business interference. This is a better option, in that it validates any findings, but may be more disruptive.
As this disruption potential increases with the type of testing, and Red Teaming can be very disruptive in certain situations. Careful planning is required if it is determined either Red Team or penetration testing is required.
Physical Security Testing
Physical security testing can be performed in support of a red team engagement for cyber, or as part of a dedicated physical security requirement, such as in a retail or warehouse operation. If it can be stolen, disrupted, or destroyed through a kinetic engagement (any physical contact) it is a risk. The risk management program should identify what is of concern, and the testing should address those concerns.
For our purposes, the easiest example of how physical security interfaces with cybersecurity is through the simple act of stealing a laptop with company data. That one act, that happens thousands of times a day, even in high security environments, can provide attackers with critical information resources.
Another simple view is how an attacker can gain access to corporate property to obtain access to corporate infrastructure. If they can get into the building, they can plug a computer into the network.
There are, of course, layers to this security onion, and merely obtaining physical access might not be enough. There may be network access controls. Or encryption. To get past additional layers of protection, the physical security testing may need to go deeper, if justified by the business risk profile.
If there is sufficient reason, attackers may use physical interaction with personnel to obtain access. Our company has posed as prospective customers at data centers and requested tours to obtain physical access to servers deep inside the layers of protection.
Testing of physical security supports not only the security needs, but also the disaster recovery aspects that it touches on, such as reviews of fencing, fire escapes, and gate protocols. A very good assessment will also look at the business operations angles and try to identify any areas where operational improvement opportunities exist, such as consolidated purchasing, and unified standards and practices.
Social Engineering
Social engineering (SE) is the easiest way to obtain access to a well protected organization, and much preferred over any physical interaction where possible, because it can be done remotely, minimizing risk to the attacker. It is so easy that it is the preferred method of obtaining access in almost every attack. It most frequently takes the form of an email sent to an employee that contains malware or links to a malicious website (phishing).
Less often, but still quite frequently, employees may be asked to provide information under false pretense. This typically is a website that resembles a valid website and asks for authentication. Once the user provides the credentials, they are redirected to the correct site and assume they entered their credentials incorrectly if they are not automatically authenticated.
Even less frequently, but still happening today, we see direct interaction, where an attacker contacts the employee by phone to attempt and coerce their assistance through a facilitating dialogue under false pretense. This method is more expensive and higher risk.
Finally, we could see combinations of these forms of attack, such as an attempt by a person dressed like a PC tech trying to take a user's computer for "service", or a boringly dressed smoker waiting by the back door for a smoking employee to come and let them in.
Social engineering attacks and tests can be quite disruptive and even emotionally difficult for the impacted employees, so be quite certain your risk profile demands it, and approach the effort with due caution. The consequences for poor judgement also need to be well thought out ahead of time, and sanctions suited to the business risk and judgement failure. For example, firing an employee for letting a very convincing PC tech take their computer away probably is extreme, but firing an employee who clicks on a plainly ridiculous phishing email for the 3rd time may be entirely appropriate.
These techniques cost almost nothing to use and test, are easy to use, and carry almost no risk which is why the attackers use them so often.
For our purposes, they are also easy to test. Generating phishing email campaigns is a trivial business expense and part of any good employee awareness training. Additional testing should be driven by identified risks, such as real-person phone calls and dropped USB sticks with malware. A serious campaign might even mail a new iPhone to an executive who has just "won" an award for CIO of the year. Malware included, of course.
SDLC Testing
Systems (Software) Development Lifecycle (SDLC) Testing can be the most complicated aspect of security testing, if the environment is complex.
This type of testing tries to identify security flaws in systems and software prior to release where it would be picked up by other types of testing (and attackers). Addressing security issues before pushing the system into production is more cost effective than post-release, because any exploited flaw will easily consume more resources than the SDLC testing process, and changes after the fact cost more than changes during development (good references here).
The topic is simplest to understand in an environment with no software development, but with systems development (which is any business with a computer). Systems lifecycle activities involve a recurring process of identifying needs, mapping these to solutions, testing the solutions, and deploying to production, before returning to needs assessment to ensure the system continues to meet business needs.
Neglecting even one of these risks introducing systems that do not meet business needs and that create security risk.
Before you buy a firewall you should be asking do I need a firewall? Purchasing processes and policies are typically where we find requirements determination and needs assessments to prevent the business from spending money unnecessarily or buying insecure systems.
If we do need a firewall, before the purchase order is signed off on, some form of acceptance testing should take place to ensure the system does what the vendor claims it does and meets identified performance criteria, some of which should be security requirements.
Software development is far more complicated, and an entire discipline unto itself, but it is possible to reduce the complexity to a manageable process for the SDLC lifecycle, aligned to the development methodology of the particular application. For a lot more detail, you can check out Oliver Moradov's blog, but it all starts with the same steps: requirements analysis and solutions identification.
Once the software enters the business development process, testing should be done at every stage, both for application function acceptance and security, to ensure security is designed in from the initial stages. This then progresses through whichever development methodology cycle is being used, but at each stage in the cycle, security testing is incorporated, preferably using automation, but with manual examinations wherever risk definitions sufficient need.
The final stages of software testing before go-live, should include vulnerability testing and possibly red teaming, typically aligned with OWASP and other options.
Only by utilizing a comprehensive SDLC program can an organization be confident their exposure and risks are minimized and that the first deployment isn't met with problems that require expensive redesign.
Testing Metrics
Metrics, which we discuss extensively with clients, are useful for improving business management and enhancing financial performance. While metrics can become quite complex, it is possible to utilize simplified sets and still obtain worthwhile positive feedback for business management and competitive advantage. Especially in testing.
Testing metrics are key to good results, and they follow the same general construction and business integration as most others security metrics. There are foundational metrics, derived from the raw performance data, used to drive line efficiencies. There are compound metrics used to drive business operational performance. And there are derived metrics used to identify trending and map to business performance, commonly referred to as KPI/KRI.
Foundational metrics in security testing are simply the quantification of the results to identify problems that need to be addressed in the testing operations. These may be simple, singular items, such as a test failure, or they may be compound and incorporate counts over time or source problem areas. But these metrics are only of use at the line level and used to manage line processes. Supervisory management may use them to identify trends in line operations or source points of disruption, but they do not percolate up as they are items to be addressed well below the business.
Compound metrics are items that combine other security metrics or integrate time, or some other variable, to produce new information. These are most commonly seen in trending metrics, to identify performance changes. For example, we would trend attack types and sources over time to identify changes that may indicate a particular source of attacks has taken a clear interest in us. Another good use case is identifying security component failures over time, to tell us when a particular component in our layered defense is either failing or has a new vulnerability requiring attention. These are of interest to operational and management personnel, but hey are not Key indicators, because they are addressable within the management structure.
We (CISOs) keep repeating that security needs to align with the business but then we talk about vulnerabilities, attack surfaces, and threat actors. If we can’t express risks in money or time, our seat at the table is probably of better use to someone else. - Wim Remes, Managing Director Damovo Security Services EMEA
Derived metrics are where the magic happens for the business. These are security metrics that are mapped to business operational and production values and provide insight at a senior management level, such as quarterly or monthly production numbers. In the security world, these are the derivatives of the compound metrics that impact business operations. For example, a security failure is a line item, and an increase in failures is a management concern, but a failure to address either as an ongoing condition, especially with an identifiable rate of change that indicates the problem is going to continue to get worse is a significant business concern. It may indicate a foundational problem exists, including competency issues. The sort of thing the board of directors would like to see fixed before the next meeting. It will be a reportable issue, because it affects business.
Derived metrics can also be positive, although these are usually only noted as part of the purchasing or operational improvement process, and indicators of effectiveness are generalized with the success story. Once the positive change has happened, it becomes the new performance standard and we are only concerned with negative changes or drastic changes moving forward. Retaining these calculations for the SDLC reviews is recommended, as well as avoiding the need to recreate the wheel when the system is replaced. Don't throw away good metrics.
Scoping
Scoping is used to both enumerate and identify which areas of testing are to be examined in any given engagement or operation. Not everything needs to be tested in each cycle, and here again is where the risk analysis indicates how frequent testing should be in each definable area. Red teaming and physical security testing are two areas that are most frequently broken down into distinct scopes both to reduce financial impact and to reduce operational impact.
For example, we might test one business location each year, in a retail organization with multiple outlets, but we would use the individual results to refine security operations at each store.
Another example is electing to test one cloud vendor rather than all of them at once (for a multi-tenant operation). Or a statistically significant set of third party vendors. We reduce the scope to reduce the costs, time, and interference, because testing can be disruptive.
Provided the rotation of in-scope assets is aligned to risk requirements, it is entirely reasonable to test subsets, but we need to account for inter-system dependencies in the results. For example, if credentials harvested from an Azure specific test are also valid in the AWS cloud, the results should explicitly mention that the problem includes the additional assets.
Summary
If an organization follows these concepts and works towards positive results, they can be confident their program is functioning properly. Successful testing happens when the information security program is aligned with business needs. The testing metrics produced provide excellent feedback, and the derived metrics provide decision support, with the result being a clear view of the state of security in the business and an understanding of which security processes are most important to the business.
And of course, call us if we can help with any of this. We help businesses do better through security.