CFET Paper (2)

This was my second CFET paper.

A summary of how the Anti-Malware Testing Standards Organization has developed in the past few years and the way in which the AV and testing industries have responded to those developments. Presented at the 4th Cybercrime Forensics Education & Training (CFET 2010) Conference in September 2010.

Local copy: Antivirus-Testing-and-AMTSO

Here’s the abstract.

Since it was formally founded in May 2008, the Anti-Malware Testing Standards Organization has been through a number of changes and generated some serious documentation and significant press coverage. AMTSO was actually founded as the result of many years of concern, not to say rage on occasion, on the part of anti-malware vendors and mainstream product testers, at the low level of competence and accuracy demonstrated by so many of the individuals and organizations offering comparative testing and/or product certification.

The organization announced its intention of improving levels of objectivity, quality and relevance of anti-malware testing methodologies. Clearly that wasn’t going to happen overnight, but how far along the road to better testing practice have we travelled?

This paper looks at testing as it was, as it is, and as AMTSO would like it to be. Is testing really so difficult? Is it appropriate for the vendors who make the products under test to be so involved in the process of defining good practice? In the process, core issues will be considered such as:

  • Comparative testing versus certification
  • Detection testing versus performance testing, and why it’s rarely a good idea to mix the two
  • Detection testing in a time of glut: when a virus lab may process tens or hundreds of thousands of unique binaries on a daily basis, prioritization is not a trivial issue. How big is the margin for error?
  • Comparing apples to oranges: can you penalize an orange for not tasting like an apple?
  • Default configuration and level playing fields
  • Correct classification and selection of samples.
  • Validation: is that sample really malicious, and how does a tester check?
  • Static analysis and static testing: is there still a place for signatures and WildList testing?
  • Is a good static test better than a bad dynamic test?
  • The AMTSO fundamental principles of testing: do they help or hinder? Is standardization of testing even a good idea?

David Harley
Small Blue-Green World
ESET Senior Research Fellow

Advertisements

About David Harley

Computer Security Author/Editor; Independent Antimalware Researcher; CEO at Small Blue-Green World; Senior Research Fellow at ESET.
This entry was posted in AMTSO, CFET, conference papers, David Harley and tagged , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s