This was my second CFET paper.
A summary of how the Anti-Malware Testing Standards Organization has developed in the past few years and the way in which the AV and testing industries have responded to those developments. Presented at the 4th Cybercrime Forensics Education & Training (CFET 2010) Conference in September 2010.
Local copy: Antivirus-Testing-and-AMTSO
Here’s the abstract.
Since it was formally founded in May 2008, the Anti-Malware Testing Standards Organization has been through a number of changes and generated some serious documentation and significant press coverage. AMTSO was actually founded as the result of many years of concern, not to say rage on occasion, on the part of anti-malware vendors and mainstream product testers, at the low level of competence and accuracy demonstrated by so many of the individuals and organizations offering comparative testing and/or product certification.
The organization announced its intention of improving levels of objectivity, quality and relevance of anti-malware testing methodologies. Clearly that wasn’t going to happen overnight, but how far along the road to better testing practice have we travelled?
This paper looks at testing as it was, as it is, and as AMTSO would like it to be. Is testing really so difficult? Is it appropriate for the vendors who make the products under test to be so involved in the process of defining good practice? In the process, core issues will be considered such as:
- Comparative testing versus certification
- Detection testing versus performance testing, and why it’s rarely a good idea to mix the two
- Detection testing in a time of glut: when a virus lab may process tens or hundreds of thousands of unique binaries on a daily basis, prioritization is not a trivial issue. How big is the margin for error?
- Comparing apples to oranges: can you penalize an orange for not tasting like an apple?
- Default configuration and level playing fields
- Correct classification and selection of samples.
- Validation: is that sample really malicious, and how does a tester check?
- Static analysis and static testing: is there still a place for signatures and WildList testing?
- Is a good static test better than a bad dynamic test?
- The AMTSO fundamental principles of testing: do they help or hinder? Is standardization of testing even a good idea?
Small Blue-Green World
ESET Senior Research Fellow