FileMaker Pro 6 comes as a native MSI, but it will not install silently unless the [Silent Install] section of the "FileMaker Pro 6.pdf" file (note: not an Acrobat PDF - it's just a text file). The "FileMaker Pro 6.pdf" file acts as the 'unattended settings' file.
The only bit you /really/ need to edit in the .pdf file is the "Installation Code=" entry.
Apparently nobody at FileMaker has heard of MST's.
Once you have extracted the MSI from the original EXE file, you can create a transform using InstallShield Admin Studio. You need to use InstallShield to create the transform because the MSI was created with InstallShield. Once the transform is created, you can then edit the transform with Wise.
If you try creating the transform with Wise, it looks like it was created successfully until you try and install the MST with the MSI and it hangs during the installation. Once the transform is created with InstallShield Admin Studio and edited with Wise, the MST installs fine with the MSI.