Received 24 February 2016; accepted 28 March 2016; published 31 March 2016
1. Security Investment in an Uncertain World
We hear about security breaches in the news almost daily, each bigger and more costly than the last. Does this reflect flawed technology, policy, or simply ineptitude? What if, instead, allowing some attacks to succeed is entirely rational for businesses? Rather than over-invest proactively, firms could wait to observe which attacks work and use this knowledge to better allocate security spending. In this article, we describe a model that weighs the merits of such an approach.
One key insight from the economics of information security literature is that attackers bent on undermining a system’s security operate strategically  . Moreover, information systems are often structured so that a system’s overall security depends on its weakest link  . A careless programmer in a software firm can intro- duce a critical vulnerability. The Internet’s global, distributed architecture leads to security being dominated by the weakest link―attackers compromise machines hosted at Internet service providers (ISPs) with lax enforce- ment policies or located in uncooperative countries. Attackers have repeatedly exhibited a knack for identifying the easiest way to bypass a system’s security, even when the system’s designer remains unaware of the parti- cular weakness.
However, systems do not exist in a vacuum; rather, defenders respond to attacks by plugging known holes. And yet, as soon as one weakest link is fixed, another weak point is often identified and exploited. Therefore, a strong dynamic component is at play: attackers find the weakest link; defenders fix the problem; attackers find new holes which are then plugged, and so on. We see that this pattern emerges repeatedly. For instance, attackers construct networks of compromised machines (so-called botnets) to pester legitimate users by emitting spam, distributing malware and hosting phishing websites. Attackers concentrate their efforts at the most irresponsible ISPs, moving on to others only after the ISP cleans up its act or is shut down. Likewise, technical countermeasures to payment card fraud have evolved over time, causing fraudsters to adopt new strategies as old weaknesses are fixed. For example, when UK banks migrated to PIN verification of transactions rather than signatures, in-person retail fraud declined while overseas ATM fraud and card-not-present fraud skyrocketed.
In this article, we devise a model that reflects this dynamic interaction between attackers and defenders. Our model captures the iterative aspect of attack and defense; we exclusively study the case where security depends on the weakest link; consequently, we model the iterated weakest link.
Plugging known holes come at a cost for the defender. And as often experienced in real systems, this cost is not always proportional to the number of controls in place. For example, organization science suggests that complexity adds super-linear administrative costs, for instance to pay a manager who directs several employees where each is an expert for a particular control. Conflicting defenses also emerge from incompatible systems, e.g., running two virus scanners on the same machine to increase coverage slows down the machine and causes errors. Lastly, human behavior fundamentally limits the composability of defenses: a password policy that re- quires both special characters (defense 1) and frequent password changes (defense 2) encourages people to stick their password on their monitor. Our model can account for interdependent defensive countermeasures.
Practical security engineers will note that it is already difficult to obtain detailed information on cost struc- tures, but what is even more unrealistic is attempts to determine the attackers’ cost and plug them into decision tables. In the real world, defenders almost always operate with incomplete information, and oftentimes a rough gauge on the relative magnitude of known threats is all the information available to a decision maker.
So another key characteristic of our model is to reflect uncertainty about which components are weakest, and therefore, which components are most susceptible to attack. All chief information security officers (CISOs) know that the security of their computer systems is imperfect at best; the better ones have catalogued the many ways an attacker might penetrate their systems, mitigating risks where possible and accepting the rest. What they remain uncertain about is which, if any, of the vulnerable components an attacker will choose to exploit. We capture this uncertainty into our model and use it to analyze how defense strategies change as knowledge about the attacker varies.
We reach several interesting conclusions upon examining the model. By comparing the static case (a single round of attack and defense) to the dynamic one (multiple rounds), we find that different defender strategies may prevail. When the defender only gets one chance to protect a system, increasing uncertainty about which link is weakest causes the defender to protect more assets, but only up to a point. When uncertainty is too high, the defender does not know which asset to protect and so chooses to protect none. If instead we allow for repeated defensive investments, an uncertain defender will initially protect fewer assets and wait for the attacker to “identify” the weakest links to be fixed in later rounds. Hence, it can be quite rational to under-invest in security until threats are realized. Unlike in other theories, this type of under-investment is not driven by the interrelation with other market participants and resulting incentive systems. Of course, security countermeasures may require significant capital investment from the outset. When we consider unrecoverable costs in our model, we find that for moderate levels of uncertainty, high unrecoverable update costs can raise the proactive protec- tion investment.
We then translate these findings about optimal defensive strategies into accepted security indicators such as annual loss expectancy (ALE) and return on security investment (ROSI). Return on investment drops as uncer- tainty about known attacks rises, even as the defender grows increasingly reactive when countering realized threats.
We do not deal with the problem of distributing costs if the defender is in fact a coalition of multiple parties (such as in the case of protecting global infrastructure) and the resulting incentive systems, which we regard as a separate problem to be studied independently. We refer the reader to the literature on externalities, such as  -  .
The remainder of this article is organized as follows: Section 2 specifies the model, which is solved later on in Section 3. We apply the model to real problems in two case studies on current topics of information security, namely online crime (Section 5.1) and payment card security (Section 5.2). Section 6 puts our contribution into perspective with prior art, and Section 7 concludes the article.
Readers who want to glance over the article without getting into the formal details may find the model summary in Section 2.6 along with the interpretation of example results in Section 4 most useful. Also the case studies (Section 5) and the general conclusion (Section 7) do not require deep understanding of the model. A non-technical summary of the model’s key ideas has also appeared in a magazine format  .
Imagine a simplistic world, in which a defender protects an asset of value a against a dispersed set of possibly heterogeneous attackers. There exist n possible threats, which can be regarded as distinct attack vectors against a single system.1 Each threat can be warded off by investing in its corresponding defense (or control). In other words, we assume a one-to-one mapping between threats and defenses, and defenses are always effective.
2.1. Defender’s Options
The model is “run” in an iterated player-versus-nature game with discrete time. In each round, the defender makes a security investment decision to define his configuration of defenses and extracts a net return from his asset before the attacker penetrates the system and, if successful, loots a fraction z of the asset. Gross returns are consumed or distributed so that the asset value does not accumulate over time. The defender seeks to maximize his expected return per round, where expectations are taken over the realizations of random variables and averages calculated over time.
Let elements of the binary column vector indicate whether a defense against the i-th threat is
implemented () or not, and let be the number of defenses in place.
The cost of defense in round t can be calculated from an upper triangular cost matrix to reflect possible interdependent defenses,
Diagonal elements hold the cost to implement a defense against the i-th threat, and off-diagonal elements indicate the extra costs if the i-th defense is implemented together with the j-th defense. If all off- diagonal elements are zero, then the defenses are independent:2
A nice property of the cost matrix is that for positive off-diagonal elements, decreasing marginal utility of defenses has become endogenous to our model. This compares favorably to, say, the Gordon-Loeb framework, in which this property appears as an assumption (A3 of  , p. 443).
2.2. Defender’s Knowledge
While the defender may possess some intuition about the relative difficulty of carrying out the n threats, such knowledge may very well be blurred. To model this uncertainty, we order the threats by increasing expected cost of attack, where the expectation is taken from the point of view of the defender. This constitutes our notion of an attack profile. This rank order is also convenient because it allows us to ignore the threats deemed too unlikely when deciding what to defend. This way, one can think of the n threats as those anti- cipated by the defender.
We define a simple functional form for the expected attack costs of the i-th threat as follows,
The unknown true cost, however, is modeled as a Gaussian random variable with mean and standard deviation (truncated to values),
The realizations are drawn only once and kept constant over time. We can vary the level of uncertainty by adjusting parameter. Modeling uncertainty in this way is crucial to the model, since it captures the diffi- culty defenders face in anticipating which of the undefended threats turns out to be the weakest link exploited by the attacker. Figure 1 illustrates the role of uncertainty when ordering threats. The left graph plots a perfect matching between expected and realized costs of attack under certainty; under uncertainty, by contrast, the right graph shows that threat 4, not threat 3 as expected, is the weakest link if defenses 1 and 2 are in place.
The defender’s knowledge about the attack profile will naturally increase over time once attacks occur, and so her uncertainty decreases as the observed attacks reveal which threat is in fact the weakest link with respect to a given configuration.
2.3. Unrecoverable Costs
So far we have assumed that the defender can upgrade his configuration at any time. This may be necessary in order to adjust to new information on the threat level or to changing risk appetite. For example, a start-up company is exposed to so many risks that is might tolerate a moderate level of information security risk. As the venture grows and develops a brand name, its risk aversion will increase to reflect the higher damage caused by potential reputation losses. Conversely, market competition (alongside herd behavior and short-sighted incen- tives to management) may drive firms to decrease their risk aversion for the sake of higher expected profits.
As is argued in  , security disinvestment is often possible to a certain degree, since personnel can be fired or equipment sold (though sometimes at high transaction costs). However, for many businesses, updating may be very expensive and the costs are spent irrevocably. For instance, the vast majority of the cost to incorporate new security features into bank notes or payment cards are borne when the changes are first made. Producing and distributing tailored tokens or devices to a large and dispersed community is expensive, and unlikely to be repeated often. As unrecoverable costs considerably affect the strategy of security investment in an iterated setting, we include them in our model as follows:
Figure 1. Attack profile. The defender forms expectations about the rank order of attack costs, but does not know the true costs until attacks are actually observed. (a) [certainty:]; (b) [uncertainty:].
Hence, parameter controls the amount of unrecoverable costs, on an ex post basis sometimes referred to as sunk costs.
2.4. Attacker’s Options and Knowledge
Our attacker model is very simple: the attacker identifies and exploits the weakest link, i.e., the threat least costly to the attacker. We do not require the attacker to make a trade-off between cost and potential gains by assuming the same utility is received for exploiting all threats. If the attacker succeeds, regardless of how, he profits, which is added to the defender’s cost. Unlike the defender, the attacker is certain of the cost for each realization of. The attacker does not operate indiscriminately; rather, he only attacks when it is pro- fitable to do so; that is, if the term is not negative.
These attacker assumptions are quite strong and probably unrealistic in some cases. For targeted attackers in particular, searching for the weakest link could be very expensive. So a more appropriate conception is the presence of many opportunistic attackers  who operate independently. Like in swarm intelligence, the chance that one of them finds the hole is quite high.
Moreover, the attacker’s gain from successful attacks may only represent a small fraction of the defender’s losses. But this is not crucial here since we do not consider attackers’ profits; we simply need an asset-dependent quantity to relate to the cost. A level shift of could easily account for the negative balance (destruction of wealth) after successful attacks. This assumption is also common in the literature (e.g.,  ), and justified there by attackers who mainly care about inflicting harm on the defender. In summary, we consider it prudent to err in the direction of a more powerful attacker than an unrealistically weak one.
2.5. Simplifying Assumptions
To make the analysis more tractable, we restrict the space of possible cost matrices by fixing all diagonal elements to a unit cost of value one and all off-diagonal elements, to the same constant. Hence, we can now derive from Equation (1) a simple expression for the cost per round to maintain k defenses,
Observe that if and if. Of course, if better knowledge on the cost of defenses is available in practical applications of the model, this assumption can be relaxed and populated with real data.
2.6. Model Summary and Example Parameters
Our model can be applied to many scenarios. For now, we demonstrate its parameters in action by considering an imaginary online music store to demonstrate its parameters in action (we describe other scenarios in Section 5). The music store’s asset is the library of audio files and the associated property rights. Let the total value of the asset be one million dollars. To avoid large numbers, we count in thousands of dollars, so asset value a = 1 000. Every year, indexed by t, the net return from online sales amounts to of the asset value.
Surrounded by malicious competitors, professional cybercriminals and “curious customers”, the store is ex- posed to various threats (say,). In case of a successful attack, the store loses of the asset value, or $25,000, due to forgone sales and incurred damage. It is known (from industry sources) that, on average over all similar businesses, the cost of a successful attack is at least 15,000 dollars (). This could be the price for an attack kit against standard software on the underground market for cybercrime supplies  . Attack costs also rise with novelty and difficulty of implementation, on average by 1000 dollars for each level of sophistication. Hence, the gradient of the attack cost is. However, these are only average values and it is unknown how much the actual costs are to attack the particular business. This uncertainty can be expressed by the deviation of the attack costs around the average. To put this parameter in perspective, means that all attacks can be predicted perfectly, means 96% of the attacks can be predicted correctly,3 and for, predicting attacks is only 65% successful. This is still somewhat better than random guessing.
The store owner is not entirely helpless: he may spend on security measures to defend against each of the threats. Each defense costs 1000 dollars per round (this is a constant that defines the unit, so no symbol is required). On top of that, defense interdependence with parameter accounts for additional cost for every pair of installed defenses. As a consequence, every investment in security must be carefully considered, since countermeasures added later cost more due to increased complexity and interoperability challenges. Finally, parameter controls (optional) unrecoverable costs. It describes the cost to update the defense con- figuration as a fraction of the asset value a. In the example of a music store, unrecoverable costs are negligible (i.e.,) since security upgrades do usually not render existing investment useless. However, if the music store served a proprietary audio format targeted to specific playback devices, an exchange of all customers’ physical playback devices to upgrade the defense configuration could be modeled as unrecoverable cost ().
Table 1 summarizes the parameters of our model and their values used in the subsequent examples. The following section will show how defenders can optimize their security spending in this model. We are primarily interested in exploring how the optimal strategy changes based on the level of uncertainty; we also elaborate on the role of unrecoverable costs.
We distinguish between static analysis, where the defender chooses the configuration in the first round () and keeps it constant further on, and dynamic analysis, where the configuration can be updated in every round. In case of certainty (), a static solution exists that is at least as good as any dynamic strategy. Unrecover- able costs are relevant in the dynamic setting only.
Table 1. Overview of model parameters.
3.1. Static Solution
We consider a defender with defense vector where the lowest cost threats are defended, leaving the remaining threats unprotected: and. Let be the return function for this defender,
where q is an indicator variable that takes value one if the attacker is successful and zero otherwise. In the case of certainty, q is fixed for all rounds, so that two outcomes are possible:
The cost term disappears in Equation (8) because if we accept attacks anyway, then there is no need to spend money to make it more difficult for the attacker (i.e., k = 0). Otherwise, the minimal number of defenses to defeat all attacks can be calculated by rearranging Equation (2) and comparing it to the attacker’s gain,
Typically, the best strategy for the defender is to follow whatever is higher, Equation (8) or (9), with. When Equation (9) is lower than Equation (8), the defender finds it profitable to operate without investing in any security measures. We call this situation indefensible, which does not necessarily imply that the business is unsustainable. Rather like in self-insurance, the defender extracts positive gross return after deducting the losses due to successful attacks. Only when both equations are negative, the cost of attacks and the cost of defending them are both so high that the defender’s business becomes unviable.
In the case of uncertainty (), q becomes a random variable and a risk neutral defender maximizes, where expectations are taken over the realizations of q. For a static solution, it is still rational to use configurations in which lower-order threats are defended first. So we can calculate as a function of k as the probability that the cost of at least one of the unprotected threats falls below the threshold,
where is the cumulative distribution function of the Gaussian normal distribution. The product term is the probability that none of the undefended threats materializes, so one minus this term is the probability of at least one successful attack. Inserting into Equation (7), we obtain
Figure 2 depicts expected returns of the static solution as a function of k for parameters as specified in Table 1. Each curve corresponds to different levels of uncertainty; the utility-maximizing selection of for each uncertainty level is circled. In the case of certainty (), the optimal security investment amounts to defenses (Equation (10)). Increasing uncertainty affects the optimal strategy in a non-linear manner: while some noise in the predicted attack costs lets the defender take more proactive counter-measures (), the cost of this over-investment start to outweigh the protected revenues when uncertainty is too high (). Eventually, it becomes rational for more uninformed defenders to refrain from any defense and instead accept moderate losses in each round. In other words, knowing your enemy is a prerequisite for taking tailored countermeasures, and tailored defense is the only option if comprehensive defense (i.e.,) is prohibitively expensive.
3.2. Dynamic (Iterated) Solution
When (uncertainty), the static solution is often sub-optimal because observed attacks reveal additional
Figure 2. Static solutions for the parameters asset value a = 1000, return r = 5%, loss given attack, minimum expected cost of attack, gradient of attack cost, defense interdependence, and. The optimal number of defenses is circled for each curve.
information about the true cost of attack. This information could be used to reconfigure the defenses adaptively. In the dynamic case, uncertainty is repeatedly eliminated for targets revealed to be the weakest link. Con- sequently, this reduction in uncertainty for later rounds can lead to a better outcome for the defender than in the static case. In general, the inter-temporal outcome is the sum of all returns over a period of time,
and can be maximized by adjusting adaptively. At each point in time t, the defender learns about the materialized threat (if any) of period. In Equation (13), , and, are functions of the sequence of configurations and the random realization of. Due to this stochastic component and its subsequent path dependencies through adaptation, the analysis must be based on expectations over all possible realizations. We will first discuss the simpler case without fully recoverable costs (i.e.,) before moving on to the general case.
3.2.1. Special Case: All Costs Recoverable
The defender has to choose a proactive defense against the most probable threats, so that = 1 and. Unless this configuration is already secure, the defender would iteratively add another defense in each round as the attacker targets new weak links. The order of these reactive defenses is determined by the order of, and thus by the realization of the random vector. This way, one weakest link is fixed after another. It follows from the path dependencies that once a secure configuration is found, the defender would not touch anymore: in this model, it is always better to start with a lower proactive defense than tinkering with existing defenses after a couple of rounds. This is so because the uncertainty about the realization of is not reduced, while the direct and indirect () cost of the -th defense has to be born for all intermediate rounds.
Let (with) be the number of rounds with successful attacks, then is the final secure configuration that is eventually reached if. If all costs are recoverable, then Equation (13) can be rewritten as
after inserting Equation (5) we obtain:
Equation (15) holds because is constant when is constant. A closed-form expression for Equation (16) can be obtained by replacing the remaining sums by first and second order arithmetic progression formulas. This expression gives the output y as a function of the proactive defense and the number of rounds under attack. The latter parameter in fact depends on the random realization of. The expectation of the term in Equation (16) can be calculated by the sum of all possible values for, weighted by the respective probability of occurrence.
To calculate, we interpret the condition for a successful attack as Bernoulli random variable with probability of attack
Hence, is the parameter vector of non-homogeneous independent Bernoulli variables. For reasonable para- meter choices, the sum of these random vectors can be approximated by the Gaussian distribution with location
and variance. This completes the analysis. Next, we compute the optimal
choices for the proactive defense (that maximize the expected inter-temporal outcome) numerically by searching for the supremum of in the integer range.
Figure 3 shows expected returns per round of the dynamic solution for the same parameters as in Figure 2, now as a function of the proactive defense. Similar to Figure 2, the optimal proactive defense for each value of is circled. Obviously, in the case of certainty (), the optimal proactive defense equals the result of the static analysis. If there is no uncertainty, a defender does not gain any information from observed attacks and can define the optimal configuration entirely proactively. However, as soon as uncertainty arises, the optimal strategies in the static and dynamic case diverge. While some uncertainty (, but small) makes a static defender more cautious (to reduce the probability of being vulnerable) and leads him to spend more, a dynamic defender knows that he will be able to adjust later and can thus take on more risk in the first place. The higher the uncertainty, the more valuable the information gleaned from observed attacks becomes and the lower the optimal share of proactive defense.
Moreover, observe that an indefensible state is already reached for in Figure 2 for the static case, whereas much higher uncertainty is needed to let a defender refrain from all proactive investment in the dynamic case (in Figure 3). So we gain the insight that the sheer possibility of adjusting defenses reactively can stimulate proactive security investment when the uncertainty is high.
3.2.2. General Case
To consider unrecoverable costs in the analysis, Equation (13) is rearranged to the form of Equation (14) as
Figure 3. Dynamic solutions for the same parameters as in Figure 2 (, , , , , , and). The opti- mal number of proactive defenses is circled for each curve.
The second sum does not include st because dt remains constant for t > tatt. Rearranging and inserting Equation (4) yields
We proceed as with Equation (16) in the special case with all costs recoverable. It is possible to keep the same strategy of iterated investment because the elements of the random vector are independent and observing one attack does not inform the defender on what the next weakest link will be. Also with unrecoverable costs at every update, it would be irrational to invest in bundles of reactive defenses if one had not done so proactively. However, if unrecoverable costs are not negligible, the effort spent on proactive measures changes substantially. For a finite horizons, however, it is rational for the defender to reflect on the strategy after having observed an attack in: for very high, he might prefer not to defend at all and accept losses in the remaining rounds. This is like a reduction to Equation (8) of the static case. This lends to another interpretation of introduc- ing unrecoverable costs in our model: they cause an increase in z for rounds (if the iterated patching starts) combined with the option to fall back to the static case (without the increase in z).
Figure 4 illustrates the structure of the general decision problem and Table 2 displays the corresponding expressions for calculating the costs at the different types of terminal nodes T1, T4. With unrecoverable costs, ignore becomes a valid choice as it may be cheaper than disinvest.
In Figure 5, we fix two settings for the level uncertainty, low () and high (), and plot expected
Figure 4. Decision tree of iterated weakest link model with unrecoverable costs. S is the start node, T1, T4 refer to terminal nodes, P denotes player (i.e., defender) and N denotes nature (i.e., attackers).
Table 2. Cost at terminal nodes as function of k1 and t (if applicable).
Figure 5. Dynamic solutions with unrecoverable costs for low (left) and high (right) uncertainty. All other parameters as in Figure 2 (, , , , , , and). controls the amount of unrecoverable update costs (as fraction of a), is a measure of uncertainty. The optimal number of proactive defenses is circled for each curve. (a) Low uncertainty: σ = 2; (b) High uncertainty: σ = 4.
returns per period as a function of the proactive defense for various settings of. Observe in Figure 5(a) that higher unrecoverable update costs triggers greater precaution and forces the defender to invest more in proactive security. This is consistent with common sense. However, if the uncertainty is high, then there exists a threshold for the unrecoverable costs above which a defender “surrenders” and prefers to cope with the attacks, as visible in Figure 5(a) for values of of the asset. The similarity to the static case can also be seen by comparing the similar shape of the curves. Note that these results should be interpreted with caution as they depend on a finite horizon, which might not apply to all business cases.
4. Security Investment Based on the Iterated Weakest Link Model
As we have analyzed the optimal strategies of the defender in the previous sections, we can now use the model to calculate some key indicators that relate to practical experience. Table 3 compares typical security indicators derived from the model for different degrees of uncertainty (in columns) and different defense strategies (row sections―static/dynamic/dynamic with unrecoverable costs). The list of indicators includes:
・ Optimal defense is a memo item that describes the optimal defense strategy. All other indicators are computed on the assumption that a rational defender follows this strategy.
・ Attack intensity measures the probability of a successful attack per round, averaged over all rounds.
・ Average gross return is the objective function of the optimization problem, scaled to a percentage of the asset value a. The average is computed over all periods and expectations are taken over all realizations of the stochastic component, weighted by their respective probability of occurrence. Higher values are better.
・ Average security spending is the average effort spent on defenses, comprising direct, indirect and un- recoverable costs (if applicable). Averages and expectations are computed as for gross returns. Lower values are not necessarily better; what matters is the efficiency of the security spending, i.e., how security spending compares against prevented losses. This depends on how targeted the defense configuration is to the reali- zation of the attack profile.
・ Annual loss expectation (ALE) measures expected foregone profits per period  .4 ALE can be computed for two scenarios: is the (hypothetical) ALE if no defenses were deployed, and is the ALE of the respective defense strategy. Expectations are taken over realizations of the random vector.
・ Return on information security investment (ROSI) is a summary measure to quantify the effectiveness of security investment. We follow the approach in  and normalize the indicators by the security spending, i.e.,
Table 3. Key security investment indicators derived from the model.
Parameters: asset value, return, loss given attack, min. expected cost of attack, gradient of attack cost, defense interdependence, sunk costs, ,. All figures rounded to one decimal digit.
Higher values denote more efficient security investment. The indicator is not defined for cases where no defenses are optimal.
Unsurprisingly, the three modes of adaptivity concur in the case of certainty (leftmost column in Table 3). However, uncertainty creates some remarkable results, which we now discuss.
For low levels of uncertainty (), replacing a static with a dynamic defense strategy primarily reduces security spending, which leads to more observed attacks. However, gross returns increase as well. In fact, se- curity spending is better targeted, over-investment is reduced, and the overall efficiency of security investment (as measured by the ROSI indicator) improves. From this result, we can draw an alternative interpretation to the omnipresent reports of security breaches in the media: rather than rashly framing them as engineering failures, one can also view them as unavoidable side-effects of smart defense strategies that balance the amount of proactive and reactive security investment.
Regarding attack intensity, a converse effect is observable when uncertainty is higher (). Here, the attack intensity drops substantially from (almost) 100% in the static case to a very low rate in the dynamic case. At the same time, average returns increase. The reason for this is that high uncertainty lets some defenders refrain from deploying any defenses (and thereby absorbing losses). Only a staged approach gives these in- vestors an incentive to defend against the most aggressive threats. Note that the model identifies the rational response to the private costs faced by defenders. We ignore the public costs created by insecurity. While not addressed by this article, negative externalities of poor security practices are widely considered to be a signifi- cant public problem  -  . Hence, while it may be narrowly better for some defenders to skimp on security when they are unsure whether they will be targeted, a public policy response may be necessary to compensate for the negative externalities of insecurity caused by such under-investment.
Unrecoverable costs muddy this picture somewhat. They raise incentives towards more proactive security investment. This leads to higher proactive defenses and thereby lower attack intensity. However, unrecoverable costs also eat up margins, so gross returns tend to decline. Even more importantly, the overall efficiency of the security investment drops, and the differential grows as uncertainty rises. So, some uncertainty combined with expensive infrastructure upgrades can stimulate over-investment―but only to a certain point. Eventually, high uncertainty can demotivate any security investment, similar to what happens in the static case. In the real world, we would expect to see relatively more weight on proactive security when updates case high unrecoverable costs (e.g., payment cards) and the uncertainty is low or moderate, compared to businesses where almost all costs are recoverable (e.g., websites). This prediction could be tested against cross-sectional empirical data in future work. In any case, the above-mentioned excuse of breaches as smart defense strategy does not apply to defenders with high unrecoverable costs. The related policy dimension here is to reduce the uncertainty when it is so high that it discourages security investment. This can be done, for instance, by encouraging information sharing so that the cost of reducing the uncertainty can be borne between several defenders   .
Note that all these conclusions remain fairly robust for other parameters than the example set in Table 1. For brevity, we refrain from discussing the influence of all other parameters. Instead, we demonstrate the validity of our model by applying it to two case studies where real data is available.
5. Empirical Evidence for Iterated Weakest Links
In the following, we will point the reader to past “iterated arms races” in the realm of information security. We argue that such dynamics, against the backdrop of our model, may in fact be rational and should not simply be framed as “failures”. We have selected two cases, one where almost all costs are recoverable (Section 5.1) and another one where unrecoverable costs should not be ignored (Section 5.2).
5.1. Case 1: Phishing and Online Crime
Our first argument focuses on the security of computers comprising the Internet follow an iterated weakest link model. Fundamental to the Internet’s design is the notion that any connected computer can communicate with every other computer. Attackers have relentlessly exploited this structure, constructing networks of compromised machines (so-called botnets) to pester legitimate users by emitting spam, distributing malware and hosting phishing websites. Internet service providers (ISPs) are usually well-placed to detect infection, because evidence of a user’s infection flows over an ISP’s communication systems. Moreover, large companies that act as Internet service providers have technical staff who can detect and clean up infected machines, while domestic users and small businesses are mostly helpless to even recognize when they are compromised; let alone to take appropriate remedial action. However, for every responsible ISP that keeps a clean network, there are countless others who find it more cost-effective to ignore the problem and let infections flourish.
There is substantial evidence that attackers concentrate their efforts at the most irresponsible ISPs, moving on to others once the ISP cleans up its act or is shut down. For instance, until 2007, a lot of the world’s malware was hosted by the Russian Business Network (RBN), which refused to comply with requests from international law enforcement  . After Russian Business Network suddenly went offline in late 2007, malware distribution shifted to other ISPs. In November 2008, a journalist from the Washington Post persuaded upstream bandwidth providers to shut off their connection to San Francisco-based McColo, which led to a temporary fall of almost 70% in the volume of spam worldwide  . Apparently, many botnet herders had been using McColo to host their control machines. Spam volume has since recovered as the spammers found new safe havens. In 2008 EstDomains, which served as the primary domain name registrar for malicious websites hosted by Russian Business Network  , became the first domain registrar to have its accreditation terminated by the Internet Corporation for Assigned Names and Numbers  . Unfortunately, other irresponsible registrars remain.
Attackers can often move on to exploit the next weakest link in the Internet far faster than defenders can knock them out. Moore and Clayton  showed how one leading group of online fraudsters, the rock-phish gang, operated. It registered many malicious web domain names to carry out attacks. Periodically, the gang picked a firm for registering Internet domain names that it had never used before and registered hundreds of web domains, using false names and stolen credit cards. These domains did not resemble normal bank names, so a registrar may not identify foul play, and the first domains to be used for false purposes were not removed quickly.
Figure 6 presents scatter plots of phishing site lifetime based on the date reported from data of  . Both .hk (Hong Kong) domains (left) and .cn (China) domains (right) lasted much longer in their first month of use than in later months. The gang targeted Hong Kong domains first in March 2007, followed by Chinese domains in May 2007 after the Hong Kong authorities wised up.
Such iteration over previously untargeted infrastructure operators is prevalent among many forms of cybercrime. Other researchers have found that websites hosting most malware move from one registrar to the next  . Liu et al. studied patterns in the registration and take-down of spam-advertised domains  , finding that miscreants register many domains at previously untargeted registrars, repeatedly moving on to new ones following crack-downs. McCoy et al. examined the use of payment processors by unlicensed online pharmacies  . They showed that the operators of unlicensed pharmacies initially relied on a small number of payment processors. Once banks were made aware of the criminal behavior, many severed ties with the pharmacies, who quickly made arrangements with new processors.
5.2. Case 2: Payment Card Security
Financial transactions have long been subject to fraud, and banks and payment processors have recognized that fraud has to be managed rather than eliminated. In 2000, the UK banks decided to adopt the EMV card payment system designed by Europay, Mastercard and Visa, popularly known as “Chip and PIN” EMV. The EMV roll-out provides a compelling example of taking security decisions to tackle the weakest link, only to find that the attackers react by exploiting new holes previously ignored. A May 2007 internal report prepared by APACS (since rebranded as the UK Cards Association) compared the expected benefits from EMV adoption to fraud incidence over time. While the report was intended to be kept confidential, it was accidentally leaked and is now hosted on the Cryptome website  . We use the data gleaned from this report, as well as more recent public statistics   , to demonstrate how the banks’ fight against card fraud has essentially been a sequence of patches to plug weak links.
Figure 7 plots the annual percentage of transaction turnover found to be fraudulent from 1972 to 2014.
Figure 6. Take-down latency for phishing attacks targeting different registrars in spring 2007; lines are five-day moving averages broken down by top-level domain (Source: own aggregation based on data of  ).
Figure 7. Long-term fraud rates as a fraction of overall turnover (Source:   ).
Notably, there are significant swings in the rate. According to the report  , “the industry has responded to crises in card fraud by making significant investment in new preventative measures,” including “terminalisation in the early 80s and, later, a major increase in levels of authorisation during the mid 90s.” Such behavior is consistent with our model of iterating through successive weakest links.
Around 2000, a fear arose that another uptick in the fraud rate was imminent. The dashed line with triangle markers in Figure 8 plots the forecast losses due to fraud based on maintaining the status quo as of 2000. To stem this expected rise in fraud, UK banks decided to adopt the EMV standard (Chip and PIN) over the course of the next few years. The estimated cost of implementing EMV across the UK was just over £1 billion, shared between the banks and retailers. We can interpret this investment to roll out EMV as unrecoverable costs. The dashed line with diamond markers in Figure 8 plots the forecast losses following EMV adoption. The uptick to 2003 is due to the phased roll-out of the technology.
In fact, the actual losses look rather different (solid line in Figure 8). The anticipated explosion of fraud in the early 2000s did not materialize; strikingly, the full adoption of EMV (completed in 2005) did not significantly alter the fraud totals. Instead, a slow but steady increase was observed, both before and after EMV adoption.
Digging a little deeper, we can see that the switch to EMV did significantly alter attacker strategy, even if it did not significantly shift the aggregate fraud rates. Instead, as some types of fraud diminished, other techniques quickly filled the gap. For instance, EMV has caused a dramatic drop in face-to-face retail fraud, since forging a signature is easier than guessing the right PIN. In 2004, face-to-face fraud totaled £219 million; in 2006, after EMV became mandatory, it fell to £72 million (see Figure 9). However, card-not-present (CNP) fraud (where only the card number is used, as in online transactions) has sharply risen since the introduction of EMV, from £150 million in 2004 to £328 million in 2008, before falling modestly and reaching a new high of £332 million in 2014. This is not surprising, since PINs are not verified for CNP transactions.
Another example of attackers moving to weaker links can be found by looking at where ATM fraud takes place. With the adoption of EMV, ATMs in the UK stopped verifying PINs using magnetic stripes, requiring the PIN be verified using the more secure chip. However, many ATMs outside the UK continued to allow the less secure verification via magnetic stripe. As UK ATMs switched over during 2006, fraudsters immediately adapted to cashing out via less secure international ATMs. Figure 10 shows how this transformation happened. In December 2005, when most UK ATMs still allowed magnetic stripe PIN verification, £6 million out of the total £6.5 million stolen from ATMs took place within the UK. By the end of the year, after the switch to more secure verification, the total monthly fraud increased to over £9 million, with nearly 75% of the losses occurring overseas. The shift to overseas fraud that is easier to carry out has continued since the EMV rollout. In Figure 11, we see that the share of the total fraud occurring overseas rose from £93M (18% of the total) in 2004 to a peak of £230M (37% of the total) in 2008. In fact, the share of overseas-originated fraud on UK cards in 2008 exceeded the total UK-originated fraud in 2004, just prior to the EMV deployment. This is compelling evidence of how quickly attackers can shift to find the next-weakest link once the biggest hole has been plugged.
The EMV standard actually provides several options for implementation that vary in the cost and security achieved. While the UK EMV implementation was certainly not inexpensive, its designers did opt for many of the lower-cost implementations offered by the standard. For instance, PIN verification is normally done offline, be- tween the card’s chip and the PIN entry device (PED). This enables a relay attack whereby a small, legitimate transaction authorized by a compromised terminal can be used to approve a simultaneous, much larger fraudulent
Figure 8. Ten-year fraud projections made in 2000 comparing whether EMV is adopted (marked dashed lines), plus actual fraud rates (solid line) (Source:  ).
Figure 9. Fraud losses for UK face-to-face (F2F) retail transactions protected by EMV and for card-not-present (CNP) transactions not protected by EMV (Source:  ).
Figure 10. Shifting attacker strategy following EMV adoption. The bar graph shows how ATM fraud has moved overseas to ATMs which have not made the switch to chip verification, but still rely on less secure magnetic stripes (Source:  , trend lines fitted with ordinary least squares).
transaction elsewhere  . Furthermore, the communications within the PED between the card’s chip and the PED’s processor are not encrypted. Combined with some shoddy tamper-proofed devices, this enables PINs to be read using a paper clip tapped inside the PED  . The same team later discovered an attack on the protocol that could enable a criminal to use a genuine card without knowing the PIN   . While entirely practical, there is no evidence that these attacks are being used by attackers at present. Instead, they seem to be focusing on the more obvious weak links of card-not-present transactions and magnetic stripe verification at foreign
Figure 11. Total fraud losses split according to whether the fraud took place in the UK or not (Source:  ).
ATMs. If (and when) these holes are plugged5, one might expect these attacks to be deployed, which would trigger defensive countermeasures by the banks.
5.3. Attack Profiles for the Case Studies
Figure 12 plots the expected and realized costs of different attacks for the case studies just presented. Figure 12(a) plots a sequence of top-level domains that potentially targeted by the rock-phish gang. First, many Chinese domains were registered by the gang, followed by domains from Hong Kong (as shown in Figure 6). The next top-level domain targeted―Austria―came as a surprise. Austria has an extensive Internet security community, and its law enforcement has long cooperated with cyber investigations from other countries. None- theless, the rock-phish gang registered many .at domains, and they were not removed for a very long time. It turns out that the Austrian domain registrar nic.at balked at removing the illicit domains without more extensive evidence documenting their illegality. The domains were only removed following a public row between the e-mail-blacklist operator Spamhaus and nic.at  . Finally, after .at domains began getting removed promptly, the Turkish top-level domain .tk was targeted.
Meanwhile, many top-level domains have not been targeted by the gang. This includes domains such as Paraguay (.py), which we might have expected to be an easy target for attackers6, as well as domains such as .se, .edu, and .gov that are harder to register.
The many threats to payment card security, along with their expected and actual costs, are shown in Figure 12(b). Face-to-face retail fraud might reasonably be seen as the weakest link in the payment card environment; the reduction in face-to-face retail fraud following the adoption of EMV supports this view. Similarly, the banks correctly anticipated that losses due to credit cards lost or stolen inside the UK (L&S in the figure) would drop once PINs were required for use. By contrast, fraud rates due to cards stolen outside the UK were lower than the banks expected. Card fraud at UK ATMs is a bit harder than retail fraud since a PIN is required. One area where the banks’ expectations were not met is with ATM fraud on UK cards outside the UK. It turns out that fraudsters can easily clone stolen UK cards and use them in foreign ATMs; hence the true cost of attack is lower than expected.
The banks’ losses due to card-not-present fraud were much higher than forecast; unsurprisingly, many banks have now decided to deploy readers that verify PINs in order to carry out online banking. Meanwhile, uncer- tainty remains regarding the attacks on PIN entry devices uncovered in  and  , though it is apparent that the costs are higher than for card-not-present fraud and foreign ATM fraud.
6. Related Work
The relevance of developing quantitative models of information security has been widely recognized in account- ing   , information security   , and dependable computing  -  . Of this and subsequent
Figure 12. Hypothetical attack profiles (derived ex post from predictions and actual attacks). (a) Rock-phish domain costs; (b) EMV attack costs.
literature, we briefly describe several works with similarities to our model.7
Gordon et al.’s “wait-and-see approach” comes closest to our mode l   . The authors explain deferred security spending in the framework of real option theory and conclude that “it is economically rational to ini- tially invest a portion of the information security budget and defer remaining investments until security breaches actually occur”. Similar to our argument, Gordon et al.  name uncertainty about the attack surface as main reason why one should “expect organizations to use security breaches as a critical determinant of their actual […] expenditures on information security.” In this sense, our model can be regarded as a substantial extension of the two-period case study they discuss. Gordon et al.’s work differs from our approach in that they treat security spending as an allocation problem of a fixed security budget rather than seeing it as capital investment to be put into relation to the assets at risk. Also, the weakest link attacker model is unique to our work. We and Gordon et al. back our claims with empirical evidence. While they draw on cross-sectional survey data collected from senior information officers, we cite time series of actual attacks in Section 5 to emphasize the iterated dimen- sion.
Herath and Herath  have transferred Gordon et al.’s  real option approach (ROA) to a case study of e-mail spam filtering. They compare net present values (NPV), which imply now-or-never investments, to ROA. Their results concur with our comparison between the static and dynamic solution: for their selected parameters, NPV suggests no security spending whereas ROA indicates positive returns for deferred partial security investment. Franqueira et al.  discuss a conceptual framework and tool support for implementing ROA in corporate security decision making.
Besides real options, the optimal timing of security investments has also been analyzed with other tools.  propose a graphical model for scenarios where defenders can learn from observed attacker behavior. The model shows the superiority of reactive defenses with concepts from online learning algorithms. Ogut et al.  use stochastic dynamic programming for the special case of reacting to signals from intrusion detection systems, which are notorious for false alarms. They compare a myopic strategy, which prescribes a fixed dynamic reac- tion, to the optimal adaptive response. In this setting, the difficulty of finding the optimal adaptive response does not outweigh the security benefit of a fixed dynamic response. One step closer to practice, Zhu et al.  simu- late reactive defense strategies based on reinforced learning algorithms for scenarios inspired by the discovery of the Heartbleed vulnerability in 2014. Kwon and Johnson  present the only empirical study we are aware of that specifically investigates proactive versus reactive security investment. Notably, their result contrasts the tenor of most analytical models! Proactive security investment was found to be more cost effective that reactive investment, and this effect is further amplified if indirect costs from breach disclosure are taken into account. It remains for future work to verify if these findings are due to the specific characteristics of the US healthcare sector, the subject of this study, or if they apply more broadly. In any case, it would be interesting to identify the driving factor behind this effect and tie it to the IWL model.
This article is an extended and updated version of our workshop version  . As a follow-up,  have studied a variant of this model which allows the defender to commission penetration tests that might reveal the next weakest link without risking the consequences of a successful attack.
7. Summary and Possible Extensions
We have presented an economic model that explains why and under which conditions under-investment in security can be rational, even against known threats for which defenses exist. Under-investment might reasonably occur when a) reactive investment is possible, b) uncertainty exists about the attacker’s relative capability to exploit different threats, c) successful attacks are not catastrophic, and d) the costs to upgrade the defense configuration are largely recoverable.
Unlike in other work explaining security under-investment with market failures, the ingredients to our model solely draw on the relation between defender and attacker, and do not involve actions of other market participants. Our results do not contradict or invalidate the well-known explanations of market failure. It rather complements the picture and highlights that market failure is a sufficient, but not always necessary, cause for security under-investment.
We believe that an iterated weakest link model accurately captures the challenges facing many information security threats today. We discussed two timely examples in the article. First, miscreants committing high- volume online crime exploit the Internet’s global, distributed architecture by compromising insecure machines and defrauding uninformed registrars, wherever they are located. Second, the security of payment cards has evolved over time by reacting to fraudster’s actions. Empirical data on losses show how the introduction of EMV in the UK has simply diverted attacker strategy to other methods, notably card-not-present and foreign ATM fraud.
As with any stylized model, ideas for extensions are abundant and implementation should be guided by the principle of parsimony, especially in the absence of reliable data to validate model assumptions. Nevertheless, a handful of extensions appear reasonable to make the model more applicable. One is to let the cost of attacks decrease over time to reflect learning and the creation of automated tools. Another direction could be to allow more ways for reducing uncertainty. This can be done either by including information gathering as an option for security investment (e.g., penetration testing or information sharing), or by relaxing the independence assumption between the stochastic realizations of the attack profile. The latter would allow the defender to infer knowledge about the probability of other threats from observed attacks, a phenomenon exploited by some technical early warning systems. If updates cause substantial unrecoverable costs, this opens new strategies of bundled investment in new defenses to better amortize the cost. Lastly, it may be of―at least academic―interest to consider other attack strategies. For example, attackers might not choose the weakest link to confuse the defender and trigger misallocations of security spending. Anecdotal evidence for such behavior can be found in the actions of some spammers, who send waves of messages with no other apparent purpose than to wear out self- learning spam filters.
One key future challenge is to find a case study where enough information is available to calibrate the model to empirical data. More generally, it may be worthwhile to explore further the question of when to deal with a problem. When is it better to move first and take proactive measures, and when is it better to defer and respond to other’s actions?
We thank Chad Heitzenrater, the anonymous reviewers and participants of the 2009 Workshop on the Economics of Information Security (WEIS), and the anonymous reviewers of this special issue on Cybersecurity Investment for valuable comments and suggestions on earlier versions of this paper. All errors and omissions are the authors’.
1An alternative interpretation is that threats represent distinct targets in a distributed system that together form asset a.
2We do not consider higher-order interdependence such as extra costs if three or more defenses are involved.
3All other parameters are as specified in Table 1.
4We stick to the term “annual” to be consistent with the literature and assume that time periods in the model correspond to financial years.
5Unfortunately, card readers used to secure online banking (and potentially all card-not-present e-commerce) have also been demonstrated to be insecure CAPFC  .
6We do not intend to single out Paraguay; many smaller countries have not been targeted by the rock-phish gang.
7For the sake of brevity, we do not review in detail the extensive literature explaining security underinvestment by interactions with other market players (e.g., by undersupply in a lemon market Anderson 2001  , moral hazard in risk sharing contracts Kunreuther 2003, market failure in residual risk transfer BS2010-WEIS  , defense effectiveness conditional to other players’ actions Varian 2004, or general externalities Bauer 2009). A trade-off between proactive and reactive measures is also studied in the literature on optimal configuration of response in intrusion detection and prevention systems Yue 2007  . We deem this related, but not very relevant, due to the narrow scope of adjusting a technical decision threshold.