Why most companies shouldn't run intrusion prevention

By John Viega
December 4, 2008 | Comments: 9

The IT security industry is filled with plenty of technologies that work, but not very well. Technologies that sell, even if they're not particularly cost effective. One of the most pervasive security technologies that doesn't work very well is the intrusion detection/prevention system. Vendors might have you believe every company needs this kind of technology, but I'd say it is only a good investment for the largest 5% of companies.

The idea behind network-based intrusion detection and intrusion prevention systems (NIDS and NIPS respectively) sounds pretty appealing. Stick a box on your network that will look at all traffic. The box will do some analysis and tell you when you're being attacked (in the case of a NIDS) or even drop attacker traffic automatically (in the case of a NIPS).

It sounds like a good thing to have all that insight into what's happening on your network, because it's insight that you didn't have before. But, turn on your typical intrusion detection system for the first time, and you will get spammed. Intrusion detection systems regularly give off over 10,000 alerts a day.

Clearly, not all of those alerts map to real intrusions. But, it's clear that, to get value out of an intrusion detection system, you need to be able to at least separate out some of the good alerts from the many irrelevant ones.

Why are intrusion detection devices so spammy? People love to talk about false positives, and certainly there are plenty. However, it's not nearly the whole problem, the way some people think.

What tends to happen is that bad guys are continually crawling the Internet, trying to find issues they can leverage. Yes, the entire Internet is continually under attack. For instance, anybody who runs an SSH server with password authentication on will notice a sea of login attempts (my personal server doesn't allow password authentication, which keeps most such attempts away, and yet, yesterday I still got almost 600 attempts).

SSH is a legitimate attack vector, and it's certainly not a false positive for an intrusion detection device to report it. But, most of these attack attempts are going to fail. Some of them can easily succeed, though, if you've got people with very poor (or blank) passwords. So it doesn't necessarily make sense to ignore everything.

Just because NIDS and NIPS technologies have lots of noise that is not due to false positives, doesn't mean false positives aren't a problem. They certainly are, common reports are that many devices can be good for thousands a day. But the point is, even if you can get rid of all the false positives, you don't get rid of the high management costs. Lowering the number of alerts takes a lot of work. You have to understand each class of problem, which takes time.

The whole "tuning" process is this very expensive up-front cost. And, even after tuning, there can be a significant ongoing cost to review data on those alerts that you might want to review. For instance, some people might want to try to correlate failed SSH logins to other network traffic that might indicate a successful intrusion (whether manually, or with a security event management product). The upfront costs alone are large enough that it doesn't make much sense for most small and medium businesses to do this kind of thing.

Let's say you're managing a network of 40 users on a corporate DSL line. And let's say you get your NIDS/NIPS and somehow manage to eat the upfront costs, and tune the system to the point where it is down to just 30 messages a day that you see and need to action. Let's say that each message takes only 5 minutes to investigate. If your team is spending 2.5 hours a day on the problem, that's going to cost potentially 30K a year in opportunity cost (time where your IT staff could be more productive). Does your team really spend an average of 2.5 hours a week cleaning up infections? And even if you do, is the NIDS/NIPS system actually going to stop you from incurring the clean-up costs, or just bring them to the surface faster??

In short, the economics don't look that great for small and even medium businesses. But, they do typically make sense in the enterprise. There's more tolerance for the up-front costs, and even the ongoing cost, since having even a half dozen people spending their day mining IPS data could make sense when monitoring a network of 40,000 users than it does to spend for one person to monitor 40 users.

The only way that there's any hope of NIDS/NIPS being cost effective for small businesses is if they can somehow benefit from scale. This is the entire idea around Managed Security Services (MSS), as offered by such companies as Symantec (through their Riptech acquisition) BT Counterpane and Verisign (through their Guardant acquisition). Let those guys monitor and analyze data for 40,000+ users, so they can charge a lot less to a small company than it would take that company to do it themselves.

But, even a managed service costs enough money that it probably isn't worth the cost if you don't have the need for dedicated server IT. Running intrusion detection for a bunch of desktops doesn't make too much sense, because you can stick your entire in-house network behind a NATing router, and then all of the sudden, (more or less), all of the traffic bound for your network will be due to connections initiated from your network. While you can still do intrusion monitoring, with the external attack surface so low, I'd say it's generally more cost effective to go spend your security budget on other things.

Certainly, as a small business owner, I'm very wary of seeing a real cost benefit before I spend anything on intrusion monitoring products or services--I will need to be spending enough in dealing with intrusions that demonstrably could have been prevented to make it worth while.

If you do have the need for dedicated server IT, you can always manage those costs by pushing them off to the cloud. Why pay for running and administering your own web servers (it's much cheaper to secure a cluster of machines dedicated to offering only a single service, by the way), when you can just host your content, and then let someone else deal with the security problem?

If the answer is that you have lots of backend application stuff, then instead of hosting content in the cloud, why not host your applications in the cloud. Let Amazon or Google handle the security. Of course, you have to trust them to do a good job here, but the big guys can generally show their methods and their success at protecting the infrastructure they manage.

For small and medium businesses, this will tend to be an excellent solution, because cloud-based computing has all the scale advantages that makes it cheap for the small guy. There does come a point where you have enough scale that it's worth doing yourself, but that doesn't apply to too many people.

In summary, NIDS/NIPS is good for the big guys, but tends to be way too noisy to be cost effective for everybody else. Managed services can help make it cost effective for medium sized shops, but effective NIPS, even if managed, requires not just the infrastructure, but an ongoing investment of money and time--two things which small guys usually lack.

And, they have better alternatives, like the cloud, or simply going without, incurring cost on an as-needed basis.


You might also be interested in:

9 Comments

John,

Great post. Most people don't realize what you pointed out so eloquently, until after they have deployed these products or services. By then, it's too late.

Good to see you blogging at O'Reilly.

Ben

I run snort on a box behind a NAT router for a small company. I look at the alert log once a week max. If something's wrong it's very obvious; I don't even bother trying to understand the non-obvious messages. I've picked up viruses and an account hijack that way. IDS has its place like everything else. Saying "IDS is bad!" is as stupid as saying "IDS is good!"

First, I never said IDS is bad. I only said that it is not cost effective for the average company. This is the primary driver for the MSS industry being as large as it is.

Second, while I've met a lot of people who can't make the economics work, there are always people with other experiences. I've seen one or two people who never look at the output, ever, but think it's a useful device (especially intrusion prevention devices, they just assume they're dropping lots of traffic). And, there are other people like you who ignore most of the output, but still feel positive about the devices.

From your description, it doesn't sound like you've saved yourself any real money. For instance, "picking up viruses"... no NIDS/NIPS is going to do a good job catching things a desktop AV product won't. Plus, if it triggers on every virus entering your network, and I were on your network, you would end up catching plenty of viruses that get sent to me as attachments. When you see such an alert, should you have to deal with them, or not?

In most cases, probably not-- I never open anything like that anyway. But even for people who might, they often won't. And, even if your NIDS vendor did outdo your AV vendor, this will probably be stopped at the desktop.

When viruses can be kept from the desktop in the first place, and when you have people on your network who may not be running AV, it may definitely be cost-effective to have potential viruses automatically dropped before it reaches the end user's PC. However, this is usually the domain of mail gateway products, not the NIDS.

But, I'm sure a few people do manage to get value they could measure, particularly out of a NIPS. That's certainly not the experience most people have.

You are broaching the gap between the top IPS products on the market and low priced or even free tools that claim some IDS/IPS functionality. The former group has a higher initial cost, the later creates notable running costs arising from their noisiness and need for tailoring. Let me very briefly point out several key differences between these groups to check why which one works.

The top IPS products come with predefined policies for user groups, protocol analysis and
recommended for blocking rules that generate automatic responses. This enables/disables classes of network events and reduces product tailoring. - e.g. to avoid raising an alert if someone accesses an SSH service. Only if this access is followed by a malicious action to the SSH server, the IT administrator should be notified and the action prevented. If the malicious action is seen without the prior SSH service access and no attack is possible then no alert should be raised. A number of ways exist in the top IPS products to prevent a DoS or high number of alerts. The basis is to perform protocol analysis. This ensures any time it is known which part of the communication is being executed for each session inspected by the IPS.

Free or commercial tools basing on Snort for example execute minimal protocol analysis and thus require a lot of tailoring to remove false positives. These tools look for tokens at certain offsets in the stream. It is challenging to write rules that do not trigger upon valid traffic on another protocol but still detect the attacks. So some rules are restricted to ports or IP ranges. This in turn means each time the network changes and services are moved, rule writers have to do a round of verification and adaptation. Some top products automatically detect the protocol running over an unknown port and apply the correct detection.

To get most out of the investment as a small business owner, I agree that one feasible way is to use MSS. Still the customer has to ensure to buy the services from top IPS products. Another possibility would be that top engines are made available at lower prices with restrictions e.g. regarding the number of connections and throughput.

Cheers,
Manuel.

Manuel,

First, sorry for the delayed response. I didn't notice this comment until putting up my next post.

I pretty much agree with you, if you pay a lot more up front, you get a product that costs less to maintain in the long term. However, from talking to lots of large enterprises about their high-end IDS deployments, it's clear there are still ongoing costs, both in terms of paying for signature updates, and reviewing the output (even if it is a lot less work per packet).

And, it's clear that the TCO for high-sticker products doesn't work out for small businesses, or even most medium ones. Most people can't put out the cash up front, and if you amortize it all out, I doubt it's too much cheaper, anyway. Certainly, I don't expect it will be cheaper than an MSS. But, I'd be interested if there were ever data to say otherwise...

John

Obviously I think the conclusion is incorrect and that the position needs to consider many more variables. I posted my thought on the matter here - http://www.snort.org/users/jbrvenik/Site/Blog/Entries/2008/12/31_The_latest_non_sequitur.html

Let's assume your number of ~$32,000 a year in labor to maintain your own IPS installation is correct. Note that's direct costs of labor. If we consider the opportunity cost of what we're not doing with that employee's time, that's another 32K. Or, if we hire someone new, the price is going to go up. So now we're close to about 60K.


For some companies, that might seem reasonable. For a lot of companies, it won't. Sure, you'd like to think that protection is valuable to people, but one of my points is that most people would spend a whole lot less than this in response if they have decent security practices (AV, NAT) and no IPS. Especially when you get down into companies with less than 10 people (the vast majority of all businesses are less than 5 people), the economics don't work out for doing it yourself.


Plus, you've totally ignored my point that, for small to medium businesses, managed security services are far cheaper than doing it yourself. There are companies that provide IPS monitoring and a host of other services for as little as $1,500 per year.


Why would I pay $30K+ for something that is worth no more than $1,500 (and, for many small business owners, is clearly not even worth that)? Your plea is more emotional than rational, which could easily be considered FUD.

I don't believe that I missed your point at all, the position I read can be summed up as "go without" which I believe is based on a narrow view and is extremely bad advice for many businesses for the reasons I presented. In short I believe that I used your data ( including 2.5 hrs a year, 30K etc ) to analyze the case presented and provided my opinion.

Changing the numbers in the middle of the analysis is intellectually dishonest. Obviously the economics are significantly different for a 5 person business than a 40 person company and the analysis I would present for that is significantly different too.

I asserted that the cost of a commercial system designed to reduce the burdens you cited has significant advantages and reduces maintenance cost in this case to ~$6000/yr, for a 5 person company it would be much less.

If I had to summarize it in a paragraph:

I like the way you are thinking about the problem but I believe the view is extremely narrow. The only way for the problems that would necessitate real security (monitored or in-house) to be solved is to have real security through technology or business. You cannot offload the business risk to a $1500/yr service and expect that service to indemnify your business in the case of a breach, the fallout is still on your business. You still have to handle it accordingly. If you moved the entirety of that information handling to the cloud then you could potentially offload that business risk but it is still likely to be catastrophic to your bottom line. Every case requires appropriate evaluation.

FWIW, the example of SSH brute forcing/password guessing examiple you gave is nonsensical, it is application payload that a network IDS/IPS can't even see. Change SSH to FTP and you've got yourself a winner.

In my opinion, the places that can get the MOST value from IDS (in particular, Snort + emerging threats rules) are the smaller to medium size companies that tend to be ignorant of common security best practices. I can setup a fairly autonomous Snort+emerging threats+base environment in just a few hours. Tune it for a week (i.e. disable irrelevant or false-positive generating sigs) and then the ongoing management of signatures is minimal, maybe an hour a week.

All IDS require tuning, even the big boys. A well-tuned IDS protecting a network of 40 users should not on average generate more than a single alert that requires investigation per day. I mean that.

News Topics

Recommended for You

Got a Question?