BIG File Format help plz

Sy · 01-18-2005, 05:46 PM

I have a requirement to create BIG files dynamically. Using Hex Workshop I have managed to decode much of the file contents. I was originally under the misunderstanding that BIG files were compressed, however they aren't.

This is what I know, File Format:

4 Bytes, ID String, BIGF (Generals), BIG4 (BFME)
4 Bytes, ULONG, Size of BIG file, little endian.
4 Bytes, ULONG, Number of files in the BIG file, big endian.
4 Bytes, ULONG, Offset to beginning of first file in archive -1, big endian.
4 Bytes, ULONG, File 1: Offset to data in big file, big endian
4 Bytes, ULONG, File 1: Size of file
4 Bytes, String, File 1: File name

The name of the file name is padded to a 4 byte boundary then the next file entry begins starting again with the File 2: Offset and so on until all the files have been specified.

The offset to the data area is simple enough and if you jump to the location in the file you will find an unmodified verison of the file data at that location, followed by the next file and then the next with no padding.

What I don't understand atm is that at the end of the file list there is some data before the data for the first file begins, I do not know what this is. Can anyone help me out?

Sy · 01-19-2005, 07:38 AM

Figured it out...the same 9 bytes appear in all BIG files after the directory listing.

004C 3232 3500 0000 00

Followed by the data of the first file. I presume this is some kind of marker. With this in mind I can now create BIG files.

MOD Developers should consider not using BIG files at all as they don't compress very well being a mixture of text and binary data. Instead Mod Launchers should be left to the task of dynamically creating BIG files for the installed mods before launching. This way, mods can be compressed and kept reasonably small for downloading. As a simple example my INI's for the mod I am working on now as a BIG file is 5MB, this is no where near complete. The same INI's can be zipped to well under .5MB

mastermind2003 · 01-19-2005, 01:26 PM

DefconINI.big - 6.39Mb
DefconINI.zip - 560Kb
I think I'll go with creating my bigs and compressing them.

Lord Arkaine · 01-19-2005, 03:58 PM

The 12mb default INI.big can be zipped into 1.25 mb. Heck, with WinRar or Ace you can go even further, if you're anal about bandwidth.

Sy · 01-19-2005, 04:02 PM

The only reason the INI.BIG compresses well is because its a largely text data and text compresses very well. Art will not compress so well, mainly because TGA and DDS already compress the image and compression ontop of compression normally results in larger files.

Lord Arkaine · 01-20-2005, 12:45 AM

Okay, well FinalBig is what, 300 kb? Including it as part of an installation could allow you to include the necessary files in a zip, extract them to a temp folder, run FinalBig, create a big file, and then stick it in the user's folder, all with one easy to use install process, which even the dumbest of computer users can't mess up. No need to complicate mods with Mod Launcher requirements. People already have a hard time installing my mods, and all I told them to do was stick one file in a folder.

mastermind2003 · 01-20-2005, 01:27 AM

Defcon zipped to about 28% of its original size when I built the installer. Graphics seem to compress pretty well.

Sy · 01-20-2005, 04:59 AM

The main reason I am suggesting that things be done differently is that the new version of the mod swapper I am working on will allow developers to use Worldbuilder for Generals, Zero-Hour or BFME without problem or launch the game.

It handles the differences between the engines and the file locations for launching. With BFME launching a mod with worldbuilder is not the same as starting the game with -mod. Mod Swapper handles this, so you the developer can get on with the business of writing your mod and transparent switch between game and worldbuilder with issues.

ModSwapper is changing more into a developers aid than a mod launching package. As final mods can be launched using -mod and a simple icon shortcut.

Phoib · 01-20-2005, 07:59 AM

Sy, I tested the size of Blitzkrieg2 art with files loose or in BIG files. Loose, the Art files are 179.089.408 bytes. Compress that in a rar and you get 58.519.552 bytes.

The big file is 177.635.328 bytes, compress that to rar and you get 50.290.688 bytes. Or 85% of the loose Art files.

I'll stick to putting everything in BIG files, thank you.

Sy · 01-20-2005, 11:35 AM

I would suggest that the reason you are getting such good compression ratios is because of the content of ur BIG files. Do you have lots of TGA's?

My guess would be yes. If these were already in DDS format which is compressed then you wouldn't see much of a change in size.

As I said before compressing something that is already compressed results in a bigger file normally, because of the added overhead in the compression algorithm itself.

W3D's will compress quite well as these aren't compressed.

In general BIG files simply contain a contiguous block of whatever you put in them and it is very important to note that it is what you put in them that
governs how compressable they are.

Try compressing your textures seperatly from the W3D's and see where the high compression ratio is.

As I said in my earlier post my reasons for creating BIG files dynamically is for several reason, ease of development with WB and keeping my original game intact without having to stick my dev files into the game folder.

You do it whichever way suits you.