OpenOffice 4.x wants you to use the setup.exe to launch the installation. You can supply the property SETUP_USED=1 on the command line to bypass this, or edit the MSI to enter that property directly.
My testing showed that putting this property in a transform and applying that to the MSI did not have the same effect. I think the installer is doing something devious and stupid, creating copies of itself with some tables restored to 'default', or MST properties not applied. Not OK, whatever the trick they are using is.
The purpose of the setup.exe, as near as I can tell, is to allow dialogs for file associations to be presented to the installing user. The easiest way I found to have the settings I wanted deployed was to edit the registry table using an MST. I sorted the Key column alphabetically using InstEdit, then deleted rows pertaining to the extensions I wanted left alone. These MST settings did seem to apply fine.
Setup Type: Windows Installer (MSI)
Deployment Method Used: Windows Installer with Custom Transform (MST)