A Security Comparison: Microsoft Office vs. Oracle Openoffice (CERT)

Background

Fuzz testing is a dynamic software testing technique that can be used to find bugs that result in the crashing of an application. Every bug that results in a crash has the potential of being a vulnerability. Depending on the specific circumstances of a crash, these bugs may also result in vulnerabilities that allow an attacker to execute arbitrary code. Fuzz testing can be used as one measure of the number of vulnerabilities that an application may contain.

The Test

I used a Python-based mutational fuzzing framework for Microsoft Windows. The fuzzer takes a starting, or "seed," file, mutates it, and opens it using the target application while monitoring that application for a crash. Microsoft's !exploitable Crash Analyzer debugger extension analyzes any crashes and assigns them each a hash identifier. The fuzzing framework uses the hash that was generated to determine if a crash is unique. Note that !exploitable may assign different hashes to the same underlying software defect. I configured the fuzzing framework to use a set of 19 different DOC files. The fuzzer used a random byte mutation strategy, which set the value of a range of bytes to a random value. The range used for this test was to mutate 0.001% to 1% of the seed file for each iteration. Each seed file was mutated in 10,000 different ways, resulting in a 190,000-iteration fuzzing campaign for each target application. I used the same set of 190,000 mutated files to test each Office suite.

The TargetsI tested currently supported Microsoft Office products:

• Microsoft Office XP Professional (released March 5, 2001)
• Microsoft Office 2003 Professional (released November 17, 2003)
• Microsoft Office 2007 Professional (released January 30, 2007)
• Microsoft Office 2010 Professional (File validation disabled)
• Microsoft Office 2010 Professional (released June 15, 2010)

All Microsoft Office products were fully patched with updates as of November 2010.

I tested Microsoft Office 2010 with file validation both enabled and disabled. The default configuration for Office 2010 has a feature called "Gatekeeper," which performs some preliminary validation of input files before opening them. If a file fails the validation, the user receives a warning but can continue to open the file. The standard fuzz testing run for Office 2010 indicates files that would crash the application without any additional user interaction. The "Office 2010 with File Validation disabled" fuzz testing run indicates files that would crash the application if the user proceeds to open the file despite the warning.

I also tested the following versions Oracle OpenOffice:

• OpenOffice 3.2.1 (released June 4, 2010)
• OpenOffice 3.3.0 RC7 (released November 26, 2010)

Because the fuzzing campaign focused on DOC file parsing, I used Microsoft Word in the Microsoft Office suite and OpenOffice Writer in the Oracle OpenOffice suite.

The Results

This graph shows the results from all of the products. As indicated by the first five columns, there is a clear decrease in the number of unique crashes with the Microsoft Office products. Office XP has the most unique crashes of any of the Office suites. However, there is not a one-to-one mapping of unique crash hashes to software bugs. More specifically, the !exploitable extension produces a Major hash and a Minor hash. Unique Major hashes are more likely to be unique bugs, while unique Minor hashes may be variations of the same bug.

More...

Fuente: CERT/CC Blog