How NSS Labs Tested IE 8's Security
|
So the president of NSS Labs, Rick Moy, got back to me overnight about the two papers the company issued showing that Microsoft's Internet Explorer 8 Web browser is effective at blocking both phishing sites and socially engineered malware. The study was sponsored by Microsoft, which led to a number of reader e-mails declaring it biased from the start. I wanted to give Moy a chance to respond, and sent him a few questions about the study's methodology; he got back to me last night. "We can understand that some are shocked and angry about the report results," Moy wrote, adding that the reports documented the test procedures. "The data certainly challenges some long-held beliefs for some. But these are the results of an evenly administered test." The browsers in the two studies, he said, were tested at their default settings, and any adjustments to elements such as JavaScript had no effect on the ultimate test results. Furthermore, Moy said, the study started as a private test for Microsoft's engineering team, which was seeking to make internal improvements. "They decided to release it based on the positive results. Many of the test reports we write do not get released by vendors, but they do get used to improve products. So what does 'sponsored' mean in this case?" In a few of the online discussions I'd been monitoring, people cited the study's small test sample as a possible reason to suspect the final results. Moy drilled down into the methodology a little, describing what made the NSS Labs study different from similar browser security research that perhaps involved larger sample sizes: "Instead of taking old malware samples that have been shared with vendors and test labs for the past few months, we actually take new fresh malware from the live Internet every couple hours, injecting new URLs every 4 hours," Moy wrote. "This is a true test of cyber-attacks against users right now, not three months ago. We start with zero malware and build up over the course of the two-week test. We also took care to [triple-] validate that each URL we suspected was actually malware using two sandboxes and human review. Thus from 12,000 URLs we evaluated, 600 ultimately passed this high level of analysis." According to Moy, a baseline of 400 URLs proved statistically significant within NSS Labs' desired confidence interval. "This type of test makes much more sense for such a fast-moving set of threats." As I noted in my blog posting Aug. 17, the NSS Labs report is and will be just one information source about IE 8's security. I suspect that if other, non-Microsoft-sponsored studies show similar results, then it might ease certain commenters' claims of bias for the NSS Labs studies. NSS Labs will conduct a study under the same methodology with 22 different desktop anti-virus products this month. Somehow I doubt that, no matter what the results of that study, it will cause the same amount of chatter as its browser testing. |
Comments (10)
Its a brave new world it there for Microsoft and eWeek. The question is, can they adapt, or are they just dinosaurs, as some people say, just waiting to die?
Quoting from an "bought and paid for" report. quote: "The study was sponsored by Microsoft, which led to a number of reader emails declaring it biased from the start." Yes, Nick it was more than biased from the start. I hope eWeek is not putting pressure on you to do this type of "fluff" for Microsoft, like they most likely were doing with Joe. The fact that you would then go on to use this bought and paid for "source" as a basis for your article, is repugnant, to say the least.
Nick, I think where eWeek is failing, is in understanding where the market is going now. Hint, its moving away from MS. Sure, it has a long way to go, but if you look at the factors to why its moving, you will see that the trend to alternatives will not stop. As such, eWeek needs to derisively, and seek out other advertising revenue besides the Vole. Because the day is coming, that MS will not have the advertising bucks to spend soon.
Posted by chips b malroy | August 18, 2009 3:49 PM
If the test was faulty, why arn't the other browsers refuting the results. They did not refute the results in March, and they are not doing it this time. Except for a small blog by an Opera person, who only threw mud, but not data.
Also why people should care about malware & phishing.... It is a huge problem
IE’s blog about SmartScreen sheds some interesting light on the amount of phishing and malware on the Internet. http://blogs.msdn.com/ie/archive/2009/08/13/real-world-protection-with-ie8-s-smartscreen-filter.aspx
• IE8 is delivering a malware block for approximately 1 out of 40 users every week
• Approximately 1 of every 200 downloads is blocked as malicious
• In the four months since IE8’s launch, IE8 has delivered 70 million malware blocks
• IE7 & 8 have delivered 125 million phishing blocks
Posted by Anon | August 18, 2009 6:41 PM
Typical, the anti-MS crew can't stand it so they start flinging poo, about the studied being bought and paid for. Tell me something, how do you think a study is done, because somebody has to pay, and whoever pays is going to be biased. They seem to think they're the only unbiased ones in the world, a shining white light in a computer dark age, when in reality they are lifeless basement dwellers taking out their life's frustrations on other computer users (the "M$ sheep"). In reality, they are the most biased, I don't see MS claiming linux kernel panics every 5 minutes like I see many anti-MS types doing. Then they scream fud and oh yes, astroturfer. It's a test of their ability to reason, when they run out of arguments they just label everyone else a shill.
Anyways, MS is doing well in security, hopefully everyone else catches up. Security does everyone good, we should not be mad at that.
Posted by JamesG | August 19, 2009 3:13 AM
Chips, I think you're probably being unfair to accuse both NSS Labs and Nicholas of producing propaganda. Studies of anything take time, effort and therefore money. Someone's got to pay for these tests.
What I think is more likely (but since I'm not involved in the IE team in anyway, I have no way to know for sure) is that Microsoft paid for these tests wanting an honest result. If the tests had shown any serious security holes, the company would have kept its mouth shut and the IE team would have taken the results and tried to fix the holes. Since the test results were positive, someone thought, "Hey, why don't we tell people we've got it right?"
The fact that Nicholas made a point of asking for more details on the test suggests to me that he's trying to do honest reporting and get his facts straight before he posts an opinion.
Posted by Jess Meats | August 19, 2009 3:39 AM
What is there to "refute"? NSS has not shared the procedure, and they have not shared the data about which urls were tested at what times. If you don't open the procedure and the data to peer review, then you don't have a scientific study.
If a drug company
-commissioned a study on a new product,
-found positive results,
-and only after looking at the results decided to publish the study,
they would be in serious trouble with the FDA. That's exactly what Moy has described above.
All of this was brought to light 3 months ago when the first round was published, yet the press slupped up the sensational headlines and story once again.
Posted by Anonymous | August 19, 2009 10:58 AM
It seems hard for some people to admit MS may be doing something good, yet they are accusing others of bias. Look in your own backyard first. Personally my favourite browser is Chrome, but I have no problem admitting the IE8 is a great browser.
Posted by Craig | August 19, 2009 6:50 PM
What is there to "refute"? NSS has not shared the procedure...
--
Are you reading the same report I am?
The proceedure.. the navigated to to melicious URLs and noted if a user would have received a block to navigate to the page or download a melicious file
URLs were from a lot of sources, NSS and others. No more than 10% were from any one domian.
If the study was flawed, why is FF tied with IE for phishing protection?
Why are you not asking Google why different browsers that use thier solution score differently over the same URLs over the same time?
Why are you not asking Opera why their one source of malware (Haught Secure according to thier website) is no longer sending telemetry to them and they still say they have malware protection?
Why would you call BS the NSS/IE test, but blindly think Chrome is protecting you?
Posted by Anon | August 20, 2009 7:01 PM
Who wants to bet that NSS is going to claim the technical test details are under NDA with MSFT. You'll never see these.
Posted by Anon... | August 24, 2009 11:51 AM
Anon,
Has any journalist written about asking NSS about the technical details of the test and were denied? Has FF, GC, OP, or SF blogged about asking about the details of the test and were denied?
I have not seen anything written about this, so I think it is wrong for you to assume it.
Posted by Anon2 | August 24, 2009 6:51 PM
As someone who actually worked on the team that commissioned this trial I knew the kind of suspicions it would provoke from the bashers. In over a decade working at Microsoft in many product team and operations capacities I can say every one of our engagements had the purpose of truly testing the robustness and potential flaws of the applications. This was no different and indeed was an exercise originally intended for internal consumption only. Frankly, we were pleasantly surprised the conclusions were even more positive than we hoped, as we told NSS to not pull any punches. We didn't skew their results.
So why not expose something that could generate positive publicity regarding the security of our products? There will always be naysayers but let's be honest, someone has to pay for these things to be done. Do you not think Apple and Mozilla do the same thing?
Posted by Former MSer | September 12, 2009 2:01 PM