Christmas present: reverse-engineering Asset.dat

Koen · 12-26-2004, 04:50 AM

Want to add new art to BFME?

The easiest way to do it right now is to download TibEd 2 beta 3 ( http://www.tibed.net ), and look for instructions on page 6/7 of this thread. I'll quickly repeat the info:
- start TibEd 2 with a BFME project.
- go to the "Project Explorer" and add all your models (*.w3d) into the "Art\W3D" folder. Also, the textures go into "Art\Textures"
- you can also put all your INI files in there, if you want.
- Close the Project explorer
- Press the "Export mod" button... this will take some time
- if you have BFME Mod Launcher installed, you can now use your mod through it. BFME Mod Launcher can be downloaded from http://www.tibed.net as well.

Extra details

TibEd basically makes an asset.dat file out of your models (W3D files). This file is put into the BIG file in which your entire mod is packaged. You can find the BIG file in C:\Documents and settings\<your account name>\Application Data\My BFME files.
BFME Mod Launcher automatically extracts the asset.dat of your mod to the game folder while backing up the original file. When the game exits, the original asset.dat is restored.

Original initial post

I noticed there is a problem with asset.dat and how everyone seems to need it, so I've been working on reverse-engineering the format. My problem right now is that I don't know what all these strings mean exactly and what their relationship is. So, I will post part of the asset.dat file as I have found it and hopefully we'll be able to find the missing things together.

General: there are 2 lists of 'things' in asset.dat. I have termed the first one entries and the second ones 'stringentries'.

Entries (8460 total, I have made a selection):

Code:

Entry #0: 201_wp_01.tga 1280822537 94 201 452
  Sub #0: TEX 0 0
Entry #1: 204_wp_01.tga 1281291272 94 201 452
  Sub #0: TEX 0 0
Entry #2: 205_wp_01.tga 1281291272 94 201 452
  Sub #0: TEX 0 0
Entry #3: 208_ad_01_r1.tga 1281603762 94 201 452
  Sub #0: TEX 0 0
Entry #4: 211_ad_01.tga 1281760007 94 201 452
  Sub #0: TEX 0 0
Entry #5: 212_ad_01_r1.tga 1281916252 94 201 452
  Sub #0: TEX 0 0
Entry #6: 214_wp_01.tga 1282228742 94 201 452
  Sub #0: TEX 0 0
Entry #7: 215_ad_01_r1.tga 1282384987 94 201 452
  Sub #0: TEX 0 0
Entry #8: 218_ad_01_r1.tga 1282697477 94 201 452
  Sub #0: TEX 0 0
Entry #9: 222_ad_01_r1.tga 1283009967 94 201 452
  Sub #0: TEX 0 0
Entry #10: 223_ad_01_r1.tga 1283322457 94 201 452
  Sub #0: TEX 0 0
Entry #11: 226_ad_01.tga 1283478702 94 201 452
  Sub #0: TEX 0 0
Entry #12: 227_ad_01_r1.tga 1283634947 94 201 452
  Sub #0: TEX 0 0
Entry #13: 230_wp_01.tga 1283947437 94 201 452
  Sub #0: TEX 0 0
Entry #14: 231_ad_01_r1.tga 1284103682 94 201 452
  Sub #0: TEX 0 0
Entry #15: 232_ad_01_r1.tga 1284416172 94 201 452
  Sub #0: TEX 0 0
Entry #16: 234_wp_01.tga 1284572417 94 201 452
  Sub #0: TEX 0 0
Entry #17: 235_ad_01.tga 1284884907 94 201 452
  Sub #0: TEX 0 0
Entry #18: 238_wp_01.tga 1285353642 94 201 452
  Sub #0: TEX 0 0

Code:

Entry #56: abridge.tga 1297853242 94 201 452
  Sub #0: TEX 0 0
Entry #57: aicamp_skn.w3d 2423910957 94 201 452
  Sub #0: MESH 0 79007
  Sub #1: HLOD 79007 124
Entry #58: apt_CampaignReview_1.tga 878022927 94 201 452
  Sub #0: TEX 0 0
Entry #59: apt_CampaignReview_41.tga 878022927 94 201 452
  Sub #0: TEX 0 0
Entry #60: apt_CampaignReview_51.tga 878022927 94 201 452
  Sub #0: TEX 0 0

Code:

Entry #399: archrnget.tga 1300821897 94 201 452
  Sub #0: TEX 0 0
Entry #400: armyants.tga 1194785427 155 202 452
hmm
  Sub #0: TEX 0 0
Entry #401: armyantsa.tga 1301134387 94 201 452
  Sub #0: TEX 0 0
Entry #402: armyantsb.tga 1301915612 94 201 452
  Sub #0: TEX 0 0
Entry #403: armyantsglow.w3d 2424067202 94 201 452
  Sub #0: HIER 0 284
  Sub #1: MESH 284 7914
  Sub #2: HLOD 8198 124
Entry #404: armyantsloc.w3d 2424223447 94 201 452
  Sub #0: HIER 0 284
  Sub #1: MESH 284 853
  Sub #2: HLOD 1137 124
Entry #405: armyantsm_gm01.w3d 2424379692 94 201 452
  Sub #0: HIER 0 716
  Sub #1: ANIM 716 8724
  Sub #2: MESH 9440 1781
  Sub #3: MESH 11221 1781
  Sub #4: MESH 13002 1421
  Sub #5: MESH 14423 1781
  Sub #6: MESH 16204 1781
  Sub #7: HLOD 17985 300
Entry #406: armyantss_gm01.w3d 2424692182 94 201 452
  Sub #0: HIER 0 500
  Sub #1: MESH 500 1421
  Sub #2: MESH 1921 1781
  Sub #3: MESH 3702 1781
  Sub #4: HLOD 5483 212
Entry #407: banner_glow01.tga 1302384347 94 201 452
  Sub #0: TEX 0 0
Entry #408: banner_glow01.w3d 2425004672 94 201 452
  Sub #0: HIER 0 1256
  Sub #1: ANIM 1256 2044
  Sub #2: MESH 3300 96562
  Sub #3: HLOD 99862 124
Entry #409: banner_glow02.tga 1302696837 94 201 452
  Sub #0: TEX 0 0
Entry #410: banner_glow02.w3d 2425160917 94 201 452
  Sub #0: HIER 0 1256
  Sub #1: ANIM 1256 3132
  Sub #2: MESH 4388 10290
  Sub #3: HLOD 14678 124
Entry #411: barad-durtoplightingmap.tga 1303009327 94 201 452
  Sub #0: TEX 0 0
Entry #412: bibl.tga 1303790552 94 201 452
  Sub #0: TEX 0 0
Entry #413: bone_01.tga 1196504144 155 202 452
hmm
  Sub #0: TEX 0 0
Entry #414: bowl.w3d 2689110231 95 201 452
  Sub #0: MESH 0 30346
Entry #415: buildingradialbuttons_001.tga 1304884267 94 201 452
  Sub #0: TEX 0 0
Entry #416: buildingradialbuttons_002.tga 1305040512 94 201 452
  Sub #0: TEX 0 0

Code:

Entry #498: buildingradialbuttons_084.tga 1343945517 94 201 452
  Sub #0: TEX 0 0
Entry #499: buildingradialbuttons_085.tga 1344726742 94 201 452
  Sub #0: TEX 0 0
Entry #500: camp_flag.w3d 2425473407 94 201 452
  Sub #0: HIER 0 284
  Sub #1: MESH 284 2011
  Sub #2: HLOD 2295 124
Entry #501: camp_flag_tex.tga 1345351722 94 201 452
  Sub #0: TEX 0 0
Entry #502: castle_flag_netex.tga 1345820457 94 201 452
  Sub #0: TEX 0 0
Entry #503: castle_flag_nwtex.tga 1346914172 94 201 452
  Sub #0: TEX 0 0

And StringEntries (more than 14000 in total, format fully known. It is two strings (on the line after string entry) and then a variable number of strings after that):

Code:

StringEntry #:0 aicamp_skn.w3d AICAMP_SKN.STUFF 1
    ncampsite.tga
StringEntry #:1 aicamp_skn.w3d AICAMP_SKN 2
    aicamp_skn.stuff
    h*ncampsite_skl
StringEntry #:2 armyantsglow.w3d ARMYANTSGLOW.CYLINDER01 1
    exlightstreaks3.tga
StringEntry #:3 armyantsglow.w3d ARMYANTSGLOW 2
    armyantsglow.cylinder01
    h*armyantsglow
StringEntry #:4 armyantsloc.w3d ARMYANTSLOC.PLANE01 2
    exgradient2.tga
    sfe_bluernga.tga
StringEntry #:5 armyantsloc.w3d ARMYANTSLOC 2
    armyantsloc.plane01
    h*armyantsloc
StringEntry #:6 armyantsm_gm01.w3d ARMYANTSM_GM01.A8 3
    armyants.tga
    armyantsa.tga
    armyantsb.tga
StringEntry #:7 armyantsm_gm01.w3d ARMYANTSM_GM01.A4 3
    armyants.tga
    armyantsa.tga
    armyantsb.tga
StringEntry #:8 armyantsm_gm01.w3d ARMYANTSM_GM01.A2 3
    armyants.tga
    armyantsa.tga
    armyantsb.tga
StringEntry #:9 armyantsm_gm01.w3d ARMYANTSM_GM01.A1 3
    armyants.tga
    armyantsa.tga
    armyantsb.tga
StringEntry #:10 armyantsm_gm01.w3d ARMYANTSM_GM01.A7 3
    armyants.tga
    armyantsa.tga
    armyantsb.tga
StringEntry #:11 armyantsm_gm01.w3d ARMYANTSM_GM01 6
    armyantsm_gm01.a1
    armyantsm_gm01.a2
    armyantsm_gm01.a4
    armyantsm_gm01.a7
    armyantsm_gm01.a8
    h*armyantsm_gm01
StringEntry #:12 armyantss_gm01.w3d ARMYANTSS_GM01.A2 3
    armyants.tga
    armyantsa.tga
    armyantsb.tga
StringEntry #:13 armyantss_gm01.w3d ARMYANTSS_GM01.A1 3
    armyants.tga
    armyantsa.tga
    armyantsb.tga
StringEntry #:14 armyantss_gm01.w3d ARMYANTSS_GM01.A7 3
    armyants.tga
    armyantsa.tga
    armyantsb.tga
StringEntry #:15 armyantss_gm01.w3d ARMYANTSS_GM01 4
    armyantss_gm01.a1
    armyantss_gm01.a2
    armyantss_gm01.a7
    h*armyantss_gm01
StringEntry #:16 banner_glow01.w3d BANNER_GLOW01.LINE04 1
    banner_glow01.tga
StringEntry #:17 banner_glow01.w3d BANNER_GLOW01 2
    banner_glow01.line04
    h*banner_glow01
StringEntry #:18 banner_glow02.w3d BANNER_GLOW02.PLANE04 1
    banner_glow02.tga
StringEntry #:19 banner_glow02.w3d BANNER_GLOW02 2
    banner_glow02.plane04
    h*banner_glow02
StringEntry #:20 camp_flag.w3d CAMP_FLAG.CYLINDER01 1
    camp_flag_tex.tga
StringEntry #:21 camp_flag.w3d CAMP_FLAG 2
    camp_flag.cylinder01
    h*camp_flag
StringEntry #:22 castle_flagne.w3d CASTLE_FLAGNE 1
    castle_flag_netex.tga
StringEntry #:23 castle_flagnw.w3d CASTLE_FLAGNW 1
    castle_flag_nwtex.tga
StringEntry #:24 castle_flagse.w3d CASTLE_FLAGSE 1
    castle_flag_setex.tga
StringEntry #:25 castle_flagsw.w3d CASTLE_FLAGSW 1
    castle_flag_swtex.tga

Can anyone with W3D knowledge link entries and stringentries together? What does it all mean? As I have absolutely no knowledge about that. And suggestions for what the numbers in the entries (and subentries) can mean are welcome.

Relax everyone, asset.dat is not going to keep us from modding BFME

.

Alphathon · 12-26-2004, 06:44 AM

I think you should post on the Generals modding forum, coz not that many peeps have BFME yet, and some never will (I assume you know gens uses w3d also). I would also try and cantact coolfile, or someone else who has produced a w3d imprter (I do appologyse if you have done either of these already).

Koen · 12-26-2004, 07:01 AM

I posted in the Generals forum now, I just hope I won't get flamed for starting a topic related to BFME

Rene · 12-26-2004, 07:29 AM

Code:

Entry #405: armyantsm_gm01.w3d 2424379692 94 201 452 
  Sub #0: HIER 0 716 
  Sub #1: ANIM 716 8724 
  Sub #2: MESH 9440 1781 
  Sub #3: MESH 11221 1781 
  Sub #4: MESH 13002 1421 
  Sub #5: MESH 14423 1781 
  Sub #6: MESH 16204 1781 
  Sub #7: HLOD 17985 300

This is the exact format of the w3d file armyantsm_gm01.w3d. I don't understand the first entry yet but I can tell you the subs.

Sub #0: HIER 0 716.

The first node at the first level in the w3d structure. It contains hierarchy data (the objects you select export tranform in renx). It starts after 0 bytes in the w3d file and has a total size of 716 bytes.

Sub #1: ANIM 716 8724

The second node at the first level and this one contains animation keys that are set on nodes in the hierarchy (that's why that one always comes before animation). It starts after 716 bytes and has a total size of 8724 bytes.

I can continue for the other parts listed but I think you get the drill. It is very easy to read through the nodes at the first level and get their type and size so we should be able to replicate this.

Just to complete it, MESH is a geomerty object and HLOD contains the hierarchal level of detail. It is only used by Renegade (and maybe earth and beyond?) but it also contains links between geometry and hierarchal objects.

Rene · 12-26-2004, 07:54 AM

Code:

StringEntry #:0 aicamp_skn.w3d AICAMP_SKN.STUFF 1 
    ncampsite.tga 
StringEntry #:1 aicamp_skn.w3d AICAMP_SKN 2 
    aicamp_skn.stuff 
    h*ncampsite_skl 



StringEntry #:15 armyantss_gm01.w3d ARMYANTSS_GM01 4 
    armyantss_gm01.a1 
    armyantss_gm01.a2 
    armyantss_gm01.a7 
    h*armyantss_gm01

I checked this one with the data contained in the w3d files. It is currently mostly a guess because I didn't compare it to other entries but...

aicamp_skn.w3d = filename
AICAMP_SKN = container name
STUFF = mesh name
1 = number of textures on this mesh
ncampsite.tga = texture on this mesh

All of this data is found in the mesh header except for the texture name, it is in the texture node (duh....).

second one, has something to do with the hlod but it is a bit more complicated I think....

aicamp_skn = filename
AICAMP_SKN = container name in hlod
2 = unknown, I don't see a relation here yet
aicamp_skn.stuff = mesh in hlod
h*ncampsite_skl = skeleton name in hlod

Now this looks easy but lets take the third one. The first entries are the same but the sub entries are confusing. The file contains 5 meshes but only 3 are listed. All meshes show up in every node of the file. I look into this one when I've got some more time.

Koen · 12-26-2004, 09:25 AM

Quote:

Originally Posted by Rene";p=&quot

I can continue for the other parts listed but I think you get the drill. It is very easy to read through the nodes at the first level and get their type and size so we should be able to replicate this.

That is good, that means the sub format is always (offset),(size) and I will believe you when you see this can be constructed from a W3D file.

Everything 'after' the normal Entry (those numbers) can also be displayed in hex as its 'easier' to move the byte boundaries.

Here are some with hex entries:

Code:

Entry #0: 201_wp_01.tga 01C4C95E4C57CD09
  Sub #0: TEX 0 0
Entry #1: 204_wp_01.tga 01C4C95E4C5EF408
  Sub #0: TEX 0 0
Entry #2: 205_wp_01.tga 01C4C95E4C5EF408
  Sub #0: TEX 0 0
Entry #3: 208_ad_01_r1.tga 01C4C95E4C63B8B2
  Sub #0: TEX 0 0
Entry #4: 211_ad_01.tga 01C4C95E4C661B07
  Sub #0: TEX 0 0
Entry #5: 212_ad_01_r1.tga 01C4C95E4C687D5C
  Sub #0: TEX 0 0
Entry #6: 214_wp_01.tga 01C4C95E4C6D4206
  Sub #0: TEX 0 0
Entry #7: 215_ad_01_r1.tga 01C4C95E4C6FA45B
  Sub #0: TEX 0 0
Entry #8: 218_ad_01_r1.tga 01C4C95E4C746905
  Sub #0: TEX 0 0
Entry #9: 222_ad_01_r1.tga 01C4C95E4C792DAF
  Sub #0: TEX 0 0
Entry #10: 223_ad_01_r1.tga 01C4C95E4C7DF259
  Sub #0: TEX 0 0
Entry #11: 226_ad_01.tga 01C4C95E4C8054AE
  Sub #0: TEX 0 0
Entry #12: 227_ad_01_r1.tga 01C4C95E4C82B703
  Sub #0: TEX 0 0
Entry #13: 230_wp_01.tga 01C4C95E4C877BAD
  Sub #0: TEX 0 0
...
Entry #399: archrnget.tga 01C4C95E4D88F789
  Sub #0: TEX 0 0
Entry #400: armyants.tga 01C4CA9B4736FA93
  Sub #0: TEX 0 0
Entry #401: armyantsa.tga 01C4C95E4D8DBC33
  Sub #0: TEX 0 0
Entry #402: armyantsb.tga 01C4C95E4D99A7DC
  Sub #0: TEX 0 0
Entry #403: armyantsglow.w3d 01C4C95E907C5482
  Sub #0: HIER 0 284
  Sub #1: MESH 284 7914
  Sub #2: HLOD 8198 124
Entry #404: armyantsloc.w3d 01C4C95E907EB6D7
  Sub #0: HIER 0 284
  Sub #1: MESH 284 853
  Sub #2: HLOD 1137 124
Entry #405: armyantsm_gm01.w3d 01C4C95E9081192C
  Sub #0: HIER 0 716
  Sub #1: ANIM 716 8724
  Sub #2: MESH 9440 1781
  Sub #3: MESH 11221 1781
  Sub #4: MESH 13002 1421
  Sub #5: MESH 14423 1781
  Sub #6: MESH 16204 1781
  Sub #7: HLOD 17985 300
Entry #406: armyantss_gm01.w3d 01C4C95E9085DDD6
  Sub #0: HIER 0 500
  Sub #1: MESH 500 1421
  Sub #2: MESH 1921 1781
  Sub #3: MESH 3702 1781
  Sub #4: HLOD 5483 212
Entry #407: banner_glow01.tga 01C4C95E4DA0CEDB
  Sub #0: TEX 0 0
Entry #408: banner_glow01.w3d 01C4C95E908AA280
  Sub #0: HIER 0 1256
  Sub #1: ANIM 1256 2044
  Sub #2: MESH 3300 96562
  Sub #3: HLOD 99862 124
Entry #409: banner_glow02.tga 01C4C95E4DA59385
  Sub #0: TEX 0 0

I get the feeling it is at least related to which BIG file it is stored in.

Rene · 12-26-2004, 09:30 AM

I can make a piece of code for you that reconstructs the subs. Any language you prefer?

Koen · 12-26-2004, 09:44 AM

I am going to liberally quote you Rene. I think StringEntry #0 and #1 should be 'read' in reverse order:

Code:

StringEntry #:1 aicamp_skn.w3d AICAMP_SKN 2 
    aicamp_skn.stuff 
    h*ncampsite_skl

aicamp_skn.w3d = filename
AICAMP_SKN = container name in hlod <- is this always the filename without .w3d?
2 = it is the number of strings that follow for Asset.dat, you can ignore it as its not W3d-related and can be automatically constructed

Then, in the substrings we have: aicamp_skn.stuff which might just be a referal to #0. And h*ncampsite_skl = skeleton name in hlod.
So, my theory about these substrings is:
- it can be a referal to another string entry (container name.SOMETHING). This appears to match the number of MESH,BOX subentries in the other one...
- it can be h*bleh which (only???) occurs if the Entry with the same filename as this StringEntry has a HLOD. This could be the link between Entries and StringEntries, given both have the same filename.
- it can be a texture filename, which has an Entry for itself. If there are textures involved, then there are never other substrings.

Code:

StringEntry #:0 aicamp_skn.w3d AICAMP_SKN.STUFF 1 
    ncampsite.tga

All in all, lots of theories now. Is there a description of the W3D format somewhere?

Koen · 12-26-2004, 09:48 AM

Quote:

Originally Posted by Rene";p=&quot

I can make a piece of code for you that reconstructs the subs. Any language you prefer?

Delphi, C/C++, VB6, Python in that order

Though I can read just about any language, as long as I know the data type byte length (int=4bytes and stuff like that).

Rene · 12-26-2004, 11:16 AM

There is no real description of the w3d format but I have a c++ header file that was ripped of an early beta from Renegade.

I never used delphi and I just hate vb, it's just not my thing, so I'll pick c++

I heven't completely thought about the theory you posted but I can comment on one thing, every model has a hlod part except for skeletons and pure animations and the container name is without the file extension.