Gartner and the Pope

By Nitesh Dhanjani
February 24, 2009 | Comments: 3

The question of how much phishing costs business in an important one. I am personally interested in it because I have recently been speaking of my research in phishing at various conferences and also because my clients often reach out to me for advise on the topic.

In December 2007, Gartner issued a press release stating that phishing cost businesses approximately $3.2 billion (based upon survey results of about 4,500 adults).

I concede that I have often leveraged the Gartner claim of $3.2 billion to make my case on why phishing is a major problem - it immediately gets the audience to understand and accept the fact that the phishing ecosystem is an incredible menace that must be dealt with.

In January 2009, Cormac Herley and Dinei Florencio of Microsoft Research published A Profitless Endeavor: Phishing as Tragedy of the Commons. In this paper, Herley and Florencio systematically and methodologically state the case for why they think the cost of phishing is not $3.2 billion annually as claimed by Gartner, but more around $61 million (section 4.4).

Even though I have leveraged the Gartner research in my presentations, I nodded in agreement as I read the hypothesis and the reasoning offered by Microsoft Research. I remembered going through the vast amount of underground message boards where phishers and scam-artists convene, noting how much of a constant struggle it was for the phishers to monetize (including cases where phishers attempted to scam other phishers), and wondering how it is that such a struggling system could correlate to a $3.2 billion loss.

The factor of difference between the Gartner and Microsoft Research numbers is x50. That's right: fifty.

It is important to understand that the claims by Gartner and Microsoft Research are scientific claims - they should be based upon reason and evidence. Even though it may not be possible to arrive at the hypothetical end-solution of an absolutely exact number, the goal of the exercise it to tend towards a reasonably accurate estimate.

My appreciation of the Microsoft Research publication stems from the scientific discourse utilized to make the arguments - reason and evidence, systematically presented. What did Gartner have to say in response? Here is a quote from a Dark Reading article:

Avivah Litan, vice president and distinguished analyst of information security and risk at Gartner, says the researchers' paper is more of an academic exercise than reality. "They are assuming their economic theories apply here -- there is no hard evidence that they do," Litan says.

I am extremely puzzled by this stance. Why wouldn't economic theories apply? After all, we are debating dollar figures here, aren't we? Herley and Florencio walk through well-reasoned arguments for their calculations. If Litan isn't in agreement, I feel she should be more specific on her stance - what exactly doesn't she agree with and why?

Here is another quote from a recent zdnet article:

"It's very misleading for the authors only to look at the phishing industry without looking at the malware business," said Litan. "In fact, it renders their entire economic argument meaningless."

Even though Herley and Florencio do not specifically mention malware in their paper (except for once, in a different context, page 8), their hypothesis on the dismal economics of phishing still stands (I am assuming that Litan was referring to malware served by phishing sites).

The original Gartner press release makes extraordinary claims: $3.2 billion is not an estimate that should be taken lightly by anyone. Extraordinary claims require extraordinary evidence (quoting Carl Sagan). As I read through the Gartner press release, I felt that the claims were unsupported because, besides the fact that a survey was conducted, it does not reveal the methodology used to arrive at the specific claims (and then there's Herley and Florencio's bias argument, section 4.2.1).

The Gartner press release, seems in essence, to be based on authority: here are the facts and they are true because we are a reputable brand and because we say so. There are only a few people that can get away with forming arguments based on authority - one of them is pictured below.

pope.png

In all seriousness, I am sincerely excited about the ongoing conversation on the real cost of phishing because it is something that advances our knowledge. The point is not for Gartner or Microsoft Research to have the final say - the goal is to have a conversation so we all arrive closer to what is true. Here is a quote from a Richard Dawkins' speech that illustrates my sentiment:

A formative influence on my undergraduate self was the response of a respected elder statesmen of the Oxford Zoology Department when an American visitor had just publicly disproved his favorite theory. The old man strode to the front of the lecture hall, shook the American warmly by the hand and declared in ringing, emotional tones: "My dear fellow, I wish to thank you. I have been wrong these fifteen years." And we clapped our hands red. Can you imagine a Government Minister being cheered in the House of Commons for a similar admission? "Resign, Resign" is a much more likely response!

I am excited that we are indulging in such important conversations in the information security, but I sincerely hope that we keep ourselves in check, and that we continue to press for critical thinking and encourage scientific discourse.

Getting back onto the original topic, it could very well be that the Microsoft Research paper includes errors, yet for now, I have not come across any well-reasoned counter-arguments that influence me otherwise. I would welcome any additional comments from Gartner - but be aware, they would have qualify to my "is this argument based on reason and evidence?" filter before being accepted.


You might also be interested in:

3 Comments

I agree that it is vital to have a serious conversation about the true costs of Phishing to both consumers and the financial industry. However, I feel strongly that Herley and Florencio should not be held up as a model for how to approach that conversation. Rather, their key conclusions suffer from three distinct sets of problems, two factual, one methodological.


1. Direct Evidence to the Contrary: First and most importantly, the paper lacks the simplest test for these hypotheses, i.e. asking the banks losing the money how much an attack pays the “Phisher”. They develop complex theories about how Phishing doesn't pay, but never asked the people with the hard dollar figures that concretely demonstrate this is a profoundly wrong conclusion. We know teh conclusion is wrong because we DID ask them, and they know exactly how big a bill they receive every time they have to pay back their customers for the lost dollars.


2. They Undercut Their Own Findings: The authors appear to estimate the profit from a typical victim is likely to be roughly $539. Even if this were true, and each attack captured only a single victim, this would weaken their own argument about total losses from Phishing given just the number of documented phishing attacks a single company like ours captures each year, which number in the hundreds of thousands, not thousands as the authors suggest.


3. Incorrect Construct: There are a number of serious flaws in applying the “Tragedy of the Commons” construct to the Phishing industry. The industry’s dynamics actually bear very little resemblance to finite-resource systems like fisheries or public grazing lands. Dramatic structural differences make a fishery a very poor analogy on which to model the Phishing industry.


For an in-depth look at each one of these points please go to http://www.cyveillance.com/web/forms/request.asp?getFile=114 to download the detailed paper.


For the sake of both banks and consumers everywhere, one would wish very much that Herley and Florencio’s conclusions were true. Unfortunately, Cyveillance believes that, when examined in light of the actual dynamics in today’s Phishing industry and when real dollars actually stolen from the banks are tallied, it remains just that – a wish.


In reality, Phishing does pay, it pays handsomely (if not unimaginably) well on a per-hour-of-effort basis, and the very low likelihood of prosecution provides a risk-reward ratio that ensures it will be with us far into the foreseeable future.

@Eric

Thank you for your comments!

I don't feel I'm holding up anyone as a model to approach the conversation - my point is to discredit authoritative modes of reasoning.

I read your paper, and here are my thoughts on the first section:

1.1 Why is asking banks the 'simplest test'?
1.2 How do you know that banks are able to measure the amount accurately?
1.3 Can you anonymize and share the data you collected from the banks?
1.4 What exact methodology did you have the banks use to come up with the Total Losses amount? I feel this should be open to scrutiny before it is accepted by anyone.
1.5 How do you know that the sample set you collected is representative?
1.6 How representative is the case where the funds were physically withdrawn within 6 minutes? Is it fair to mention this case when you are talking about an overall sample that includes 1200x365 attacks per year?
1.7 The data represents 20 cases for an 'attack that occurred less than 30 days prior'. How many total number of attacks occurred? Can you multiply this amount by x12 and confirm with the bank if they felt they actually lost that amount total?
1.8 What is the size of the bank that was surveyed?

And this is for the 1st section. I have many more questions and some disagreements about the remaining sections - perhaps I'll get to them later and post them to solicit your response.

I do appreciate your work - I feel it gets the conversation going. But, honestly, I ended up with more questions by the end of it. This isn't a bad thing necessarily - it did get my neurons flowing, but I came away more puzzled than before I went in.

Hi Nitesh -

As you say, the best part about this is, it does get the conversation going. :-) And I appreciate that. I hope you do as well, since I’m sure you’ll continue to disagree with lots of what I say, which is terrific, good fun, and good for the debate. I'm happy to answer your questions at length if you'd like to call me or email me but in short let me summarize as best I can.

The first thing to remember is I’m not publishing my own academic paper here, I’m not trying to convince the world my quess is right and theirs is wrong. My point is that Herley and Florencio could have ANSWERED rather than theorized about the questions they tackled. Whatever the answer is, why not just go get it? Wouldn’t that have been a better use of what are clearly two pretty smart guys and a lot of effort? This isn’t some cosmic unknowable we’re talking about here. The answer is explicitly resident in the computer systems of every bank that gets Phished. Why not just ask them?

Why is asking the banks the simplest way to find out if Phishing pays? Ummm…because it directly and factually answers the question of “does it pay to be a Phisher?” And how do we know banks are able to measure the amount accurately? And how do we know this truly represents “total cost”?

First, a clarification – I’m not talking “total cost”. Total cost is cash reimbursements, investigative and recovery costs, security staff overhead, incident tracking and forensics, brand equity loss and intangibles, blah blah blah. No, I was talking about right-now, clear-as-day, cash-on-the-nail reimbursement. So how do we know that information is accurate? Ummm, accounting systems are pretty unambiguous on this point. If Mr. Jones’ account is drained of $9231.23, the bank has to put $9231.23 back in the account.

Each bank knows at least that one piece of the “cost” - the reimbursement bill (forget all the other “total” costs) to the penny. Total COST is how much it hurt the bank, which wasn’t really the point. Total direct LOSSES (I’m talking about the money drained out of the accounts) is how much the Phisher MADE, which was the crux of the central claims in the paper.

Subjecting the methodology to scrutiny? Sure. We service several hundred clients. We detected several hundred thousand unique Phishing attacks last year. We know a lot of Phishing and Fraud people at banks. So our methodology went like this - We picked up the phone and said something like:

“Hi Bob, Eric here. Listen, Bob, each time you guys get Phished, do you guys track the losses in your fraud department or accounting systems? Ok, and what kind of losses do you incur? Uh huh…yeah, round figures is fine. Or maybe a few specific examples… Uh huh… Great. Listen, Bob, if I don’t use your name, can I quote those numbers? Great. Thanks.”

So are my examples, or the selection of the clients who happened to take my call that day, perfectly statistically representative of all banks being Phished? Nope.

Does that matter? Not one little bit.

Don’t forget, I’m not trying to give you the correct answer. I’m only making two points here. First, why use theory, modeling and guesswork to answer a question for which the answer is unambiguously available? Why didn’t Herley and Florencio do what I did, albeit with a bit more rigor, and actually get a realistic estimate based on facts? Call 500 banks, maybe 50 will talk to you, and write down the hard numbers they give you. Theory, schmeory.

Second, if they had done that, then whatever the exact dollar amount turned out to be, $1000 per attack or $100,000 per attack or whatever, doesn’t matter to me. What does matter is that whatever they are, I'll bet those actual numbers show that the notion “being a Phisher doesn’t pay, and you might as well go get a regular job” is not only unsupported by the data, it’s downright laughable.

Call your bank and ask the head of the fraud department whether they agree Phishing them is a “profitless” endeavor and that Phishers aren’t making any money. They’ll laugh. Probably long, loud and bitterly.

Hope that helps continue the debate!

Regards,

Eric


News Topics

Recommended for You

Got a Question?