.GDT Image Format

[hollowaytape]

The .GDT is used for graphics, both static and animated. For a translation, there are 7 images that will need to be altered - main title screen and 6 chapter title screens. There may be more - with a codec, it would be trivial to check for more.

By playing around with the file MAP100.GDT pretty much one byte at a time, replacing TITLE1.GDT in disk B1 with it, and loading the game over and over, I've figured out a few different ways the pixels get encoded. I was mostly working on this a few months ago and lost steam with it. Here are my findings so far:

Code: Select all

*Headers:
**BIGBOMB.GDT 88-E4-00-00-00-00-80-02-C8-00-11-... (big, animated)
**TITLE1.GDT: 88-E4-00-00-00-00-80-02-C8-00-11-... (640x400)
**TITLE2.GDT: 88-E4-00-00-00-00-80-02-C8-00-11-... (640x400)
**TITLE3.GDT: 88-E4-00-00-00-00-80-02-C8-00-11-... (640x400)
**TITLE4.GDT: 88-E4-00-00-00-00-80-02-C8-00-11-... (640x400)
**TITLE5.GDT: 88-E4-00-00-00-00-80-02-C8-00-11-... (640x400)
**GAMEOVER.GDT88-E4-18-00-08-00-A0-01-90-00-11-... (416x288)
**SEN04.GDT:  88-E4-18-00-09-00-A0-01-8F-00-11-... (416x288)
**SEN09.GDT:  88-E4-18-00-08-00-A0-01-90-00-11-... (416x288)
**SEN10.GDT:  88-E4-18-00-08-00-A0-01-90-00-11-... (416x288)
**MAP100.GDT: 88-E4-C8-01-08-00-A0-00-40-00-11-... (160x128)
**MAP200.GDT: 88-E4-C8-01-08-00-A0-00-40-00-11-... (160x128)
**MAP300.GDT: 88-E4-C8-01-08-00-A0-00-40-00-11-... (160x128)
**MAP400.GDT: 88-E4-C8-01-08-00-A0-00-40-00-11-... (160x128)
**MAP500.GDT: 88-E4-C8-01-08-00-A0-00-40-00-11-... (160x128)

*Header Bytes:
**01-00: Begin header
**03-02: (?)
**05-04: (?)
**07-06: Image width (640 for titles, 416 for sad ends, 160 for maps)
**09-08: Image height, scanlines subtracted (200 for titles, 143-144 for sad ends, 64 for map)
**11-10: End of header (11=(decimal) length of header)

*Encoding Modes
The image is divided into skinny blocks, 8px wide and image-height tall. You have some number of color planes, each introduced with a byte specifying the encoding mode:
**04, 0C?: Run-length encoding. Two bytes:
***First byte: Pattern of the line in binary (which pixels are filled in).
***Second byte: how many lines to repeat the pattern in the first byte.
**81, 80?: Positional encoding. Two bytes:
***First byte: Position of the next line.
***Second byte: Pattern of the line in binary.
**06: Pattern encoding. One byte:
***First half-byte: Number of lines to repeat the pattern.
***Second half-byte: Pattern. See "Patterns" below.

*Patterns
**x0: 0000 0000
**x1: 0010 0010
**x2: 0101 0101
**x3: 0111 0111
**x4: 1111 1111
**x5: 1101 1101
**x6: 1010 1010
**x7: ? (breaks everything)
**x8: 1111 1111
**x9: checkerboard 0010 0010 / 1000 1000
**xa: checkerboard 0101 0101 / 1010 1010
**xb: checkerboard 0111 0111 / 1101 1101
**xc: 1111 1111
**xd: rev. checkerboard 1101 1101/ 0111 0111
**xe: rev. checkerboard 1010 1010 / 0101 0101
**xf: ? (breaks everything)

*Control Codes
**10: Ends a block (?)
**FF 84: Repeat the next line 4 times.
**00 00 00: Blank. The block is completely black. Skip it and start the next one.

What I'm wondering about at this point: What other sorts of info gets specified in image headers? How do PC-98 games generally treat drawing the different color planes?

Does anyone with more experience in image formats have any insights, or see any resemblances to file formats familiar to them? I will continue to buckle down and mess around with the individual bytes, but I feel like i'm missing some pieces in what I need to know to do it.

[SkyeWelse]

PC-98 Image formats are a bit new to us and there are many out there that were used. The MAG and PIC formats are typically the most common, and there are plenty of editors and converters for those. MAGs are usually used to for CG Background images.

Out of curiosity I did do a search earlier today to see how common the format of GDT was and if there were any existing editors or conversion tools. It doesn't appear to be too common from what I can tell. I only found one type of converter and I think it only manages to convert it to some other format I've never heard of, if it even does that. Haven't figured it out because it needs to be run in PC-98 DOS with various output options that I can't make out.

However, I did come across this post over at Tokugawa which may or may not be helpful. I'm a graphics editor, but I no very little about what makes up graphic formats and I'm still learning. Flame has taught me a little about how to look at where color tables start within certain image formats, but that's about the extent of my knowledge on all of that.

The thread is this one. Look at Ashura's post.

The PC-98 VRAM has 4 planes. The segment addresses are:

Blue: A800h
Red: B000h
Green: B800h
Extended: E000h

Every pixel is defined by 1 bit in each plane, so an 8 color image has 3 bits/pixel (no extended plane) and a 16 color image has 4 bits/pixel.
Each line requires 80 bytes (640 pixels, 16 colors), so an uncompressed 400 line image is 80 * 400 * 4 = 128000 bytes.
This structure made it more difficult to switch to 256 colors than in the IBM world.
How 256 color graphics are managed I haven't studied myself yet so I can't tell you.

Almost all formats use their own custom compression. The complexity of the compression algorithm varies. You can usually tell from the image sizes.
The usual size for a decently compressed 640x400 image is between 30 and 60 KB.
To determine which files are images it usually helps to look at the first few bytes in some of the bigger candidates for values like 80 (50h, since many formats store bytes-per-line rather than image width) or 400 (190h).
Some formats store values in big endian format btw. These formats probably originated on the X68000 or the programmer came from a X68000 background.

Since this is an early PC-98 game, I'm wondering if perhaps there may be instead some PC-88 tools that worked with GDT files. I did do some searches with this, but didn't find anything new really.

-Thomas

[josejl]

Tracing this game's code is truly a challenge,having being programmed in C instead of ASM,the compiler really generates different ASM than a human programmer.

It took me a while,but I found where TITLE01.GDT and other GDT files are processed.I have not had time to see what it does,but it's on GEAGRDRV.EXE around offset 0x1250.
When running the game it's on 1B62:1053.

By the way,I have customized NP2 Debug edition a little.I added a window where you can see the contents of the 4 VRAM banks graphically (as they are 1 bit per pixel,you only can see black and white).Now that I've got it working I'm afraid it won't be too useful for this,but now that I know how to add new debugging modes perhaps you might have some suggestions for extra debugging features that could be interesting.

[M_bot]

I started looking at the format today and managed to insert my name without modifying the existing part of the image. I don't have the time atm to go in depth but Hollowaytape's description only seems to be half right (unless I misread). I'll post more about what I know later this week. In terms of codecs its similar to others but not enough to take existing code and modify it, but I'm only basing that on my none PC98 knowledge. It's possible MAG and PIC are similar to it. In terms of approach to this it seems that looking at the ASM code or bruteforcing byte-by-byte will yield the same speed.

This was originally TITLE0.GDT, which now has my name in it. I did it by hand (hex-editor) in case you were thinking I had made a program already.

46MBOT.PNG (51.43 KiB) Viewed 23910 times

[hollowaytape]

Wow, nice work! I look forward to hearing more about it.

[M_bot]

I managed to decode around 95% of GAMEOVER.GDT, currently don't understand the 0x02 encoding mode and there appear to be some other quirks. Here is very incomplete analysis of the file. And below is an image of my program attempting to decode the image. The program atm only properly works for GAMEOVER.GDT because I have yet to work on other encoding modes like 0x01 and 0x03. I did the smallest file that had text in so that I could more quickly get a full image. I will upload program later. Anyway need to sleep.

[M_bot]

So I managed to completely decode GAMEOVER.GDT, however there still parts of the format that I don't understand.


My program (open and type "convert file.gdt") will decode any of the GDT files and tell you if something goes wrong. It outputs each encoding block in hex so you can find what went wrong, as well as outputs each successive write in its own file (the test directory that is created) so that you can see how far it got. Here's a cool animation of the file decoding.


MAP100.GDT also works "fine" however there's a certain bit in each encoding mode that must change how the file is read. For example the encoding mode 0x04 can be modified by setting bit 3 becoming 0x0C, its still the same mode but it also does something extra.

Never mind! Figured that out, basically if you copy a plane you add new data using exclusive or. Bit 3 means copy the first (or blue) plane.

There are still problems with other files, will upload new program later or maybe when I figure out more.

The majority of the files get decoded a couple non zero blocks in and then break. So still missing a lot but I think at this point I need to start thinking about how to get edits back into the game.

Here's an explanation of the very beginning of the format. There a little bit too much to write it all down. At this point its easier to read my code.

Code: Select all

Header that defines width, height, and stuff. It ends in 0x11.
8 pixel blocks, 3 color planes (BRG Order), single byte defined.

Plane define first nibble
	Value	 Meaning
	0x1		Copy the previous plane
	0x8		Copy the nth previous block's plane. N is the second nibble. Copy the same plane we are in.
			   Read two bytes
					1st byte: Position of next line
					2nd byte: Pattern of the line in binary

Plane define second nibble
	Value	 Meaning
	0x2		Too little info to really know, seems like patterns again
	0x4		Direct write unless nibbles equal each other
	0x6		Pattern write, also allows direct write
	0x8		Anything that has this bit set is still one of the modes above but is going to do something extra

[hollowaytape]

Looks like you're making great progress! I was wondering what I was missing when planes were being copied, that's great.

When I used the CompileTools CLI on some of the title GDTs I have, it fails at the first block with "Could not find a part of the path '(mydir)\test\b018p0.png". Do I need some more files to run the converter? Is it failing to create some files? (I tried running powershell as administrator too, no luck.)

As for reinserting the images after edits, worst-case scenario: we never find out how some of the encoding modes work. A BMP/PNG->GDT converter could still use the simplest encoding mode (direct write?), maybe some of the plane repeating codes if we're feeling sophisticated. I don't anticipate that the bigger file size will be an issue - replacing TITLE1.GDT with various bigger images didn't mess up the game file pointers or anything, the game worked fine.

[M_bot]

Oops, that problem is my fault. You have to manually create the test directory in that version. I just fixed that, along with the other fix I figured out above. You can get the updated version here.

If file size won't be a problem than I can go start writing an encoder using only one of the modes. And then try to slowly implement the other modes. I think we'll need at least some of the fancy stuff because otherwise the files will be quite large.

Edit : I just ran the updated version on all the files it manages to decode a quarter of them half way through. AV500 is quite funny, since its literally AV500 written, must've been a placeholder. Also bigbomb isn't an animation file, its different colored circles that I'm guessing are masked by the game in succession to create the explosion.

[hollowaytape]

Okay, I made a test directory and am running the new version. For each of the files I tried (again, just TITLEX.GDTs) it prints out a bunch of blocks interspersed with "Houston, we have a problem," and then fails with "Index was outside the bounds of the array." Is this the same way it's failing on your end? (Also, nothing gets written to the .BMP files it generates.)

That makes sense with BIGBOMB.GDT! Clever Enix programmers...

I wonder how big the image files can get without using all encoding modes... From editdisk, it looks like disk A has 1MB used, 135KB free; disk B1 has 295KB used, 975KB free; ...disk B4 has 786KB used, 391KB free. Right now, I've only found 2-3 images per disk that need edits. The largest image I found is ~14KB. If I'm not wrong, it looks like the images could get pretty big without running into space issues.

You can work on an encoder if you're up to it! I am excited to try some reinsertion.

[M_bot]

Are you sure you're running the latest version? It should be v1.1.1, though there's no way of checking that from the program. Mine fails but still writes to the BMP file. Even if it doesn't write the to BMP file, in the test directory the last file should be how far it got in decoding. I know for a fact that MAP100.GDT and GAMEOVER.GDT should have no problems if you want to make sure those work. TITLEX.GDT ones don't decode completely. Apparently the fixes I made today fixed half of the TITLEX.GDT files, still quite a lot of issues though.

The colored blocks are supposed to help with debug though it gets quite messy. The Houston thing means it went out of bounds and the program caught it. 4NR, 2NR, NR mean that a opcode/pattern of some sort wasn't recognized. They're mainly for me or anyone who understands the code. The code is quite experimental so it doesn't really have helpful debug yet.

I'm slowly adding fixes to my code for the 0x4 and 0x6 modes by going through the files. 0x2 however completely eludes me since I haven't tested it enough in NP Debug. There's also some other things like 0x43 that I haven't looked at either. I don't know if I should make a new version for every day's fixes. I might do a weekly update.

If you still have trouble getting it to decode files you can get on IRC (click the link above next to FAQ or you can connect to #heroes on Rizon) around the same time you posted (9PM CST). I'm usually on then and can help you. A lot of the time the problem is that I introduced a bug when fixing something else I didn't catch it. It's possible that I already fixed on my end but I haven't made a new version yet. I need to figure out how to make Travis CI make a build of my latest commit.

[M_bot]

And v1.1.2 is up. Made some addition to 0x04 and 0x06. I think I'll finally start looking at 0x02 even though it looks like a pain. On the other side though, I made a very stupid encoder. It uses absolutely none of the flags so very large file sizes but the game doesn't seem to care. Of course later I'll make it at least use some RLE but just wanted to prove insertion. Also it does absolutely no color approximation. Your file is treated as binary colors. If your color is a blue that has a non-zero value in all RGB then it will appear white, but if the blue is only B then it will appear blue.

I inserted my avatar into the game (after making it BW in Photoshop). And also reconverted MAP100 (albeit with incorrect colors) and that worked fine as well.

You can convert BMP (and PNG if you change the extension to BMP) to GDT by typing "convert file.bmp gdt"

[hollowaytape]

Heh, looks like I just made a mistake in copying files last time, everything works fine now. This new release has been especially fun to play with. Exciting to see what's in a lot of the smaller GDT images I hadn't looked at.

[hollowaytape]

M-bot, what is your status on this? I have made enough progress on the text reinserter for now, so I can turn my attention to this. I can work on the BMP -> GDT conversion if you'd like my help or my time. Have you made any updates that are not up on the github?

[M_bot]

I haven't done anything other than adding some comments to GDT which I just pushed. I was thinking of starting work on it again myself (hence why I made comments to GDT since I had to remind myself of how the code works). You are more than welcome to work on it though you'll have to wait as I slowly comment my code if you don't understand part of it.

Speaking of the code I think it looks pretty bad and could use some cleaning up. For example some of the switch statements can probably be reduced to an array that holds that data that need to written.

The BMP to GDT is just a straight copy at the moment so if you are able to work in some of the flags that are available that would improve you ability to reinsert. I was thinking of working some more on the decoder so it can properly decode the title screens.

[hollowaytape]

That sounds good to me! I just forked it, and I'll start familiarizing myself with what you've figured out of the format. I'll figure out what flags are most useful to integrate into inserting the title images for now.

[hollowaytape]

I made some fairly basic progress in my fork of the encoder a few weeks ago, not as much as I'd hoped. I'm able to use the RLE flag (0x04), and this image (a simplified chapter tile from SNES EVO) ends up being 3kb:



I also worked on the "repeat previous plane" flag (0x10), so I compare each plane's RLE representation to the one right before, and if they're the same, the encoder will just write 0x10 and continue. But when I did this, the text would be blue instead - like instead of repeating planes, it had just skipped them... Yup, I'm just doing something wrong in comparing planes.

The problem I'm running into is color. I re-encode GAMEOVER.GDT from a bmp fine, but the blues are cyans instead of the normal blue in the original. Worse, when I include cyan or blue in the chapter title image above, the game displays lots of garbage instead.

So I'll be looking more into how color works in the decoder, as well as how the plane repeater works.

It looks like GDT files will always stop rendering after the first 16kb, so that's probably the maximum file size we should aim for.

EDIT: Cool, I just, got all the colors working:



Looks like the colors for the planes are a little different than pure RGB - it's more of a burnt orange, darker green, and lighter blue, so that changes the logic of checking the bmp colors slightly. Once I looked at your GetColor() function that became clear to me.

I'll work out a few more quirks and then make a pull request. I'll ask kuoushi for a translation of TITLE1.GDT so I can get some art from Skye and try reinserting for real.

[SkyeWelse]

Excellent! Nice job! Yep, just let me know what type of format you would like to have the graphics file in and the bit size and I can work on some images. I had a HDD failure with my OS boot drive so a lot of my programs such as photoshop are going to all need to be reinstalled once the new harddrive arrives. It's taken me out of commission lately among other things.

-Thomas

[hollowaytape]

Cool! Sorry about your hard drive, that's no fun. Whenever you're ready, here are some details:

The encoder takes BMP files. 16-color (3-bit) bitmaps work well, but the 16 in-game colors are slightly different than those in bitmaps, so you're welcome to use higher-color bitmaps to work closer to the in-game colors. The image dimensions are 640x400.

Here's the image in question:



^ scanlined, actual in-game colors



^ interpolated, then saved as a 16-color bmp

And here's a translation from kuoushi:

Chapter One
Ordovician Period
Set Sail for Evolution

Thanks again!

[M_bot]

I'll be merging your request later today. I'm most likely going to squash your commits into one if that's fine with you. Alternatively you could squash them yourself into a couple more concise ones.

I got some progress done on decoding myself but the mess of code I originally wrote is getting to me. It's becoming harder to add in the code I want. I'm probably going to rewrite the decode function to be more object oriented. Somewhat like the LZ77 file that my friend wrote.

Once I'm done with that I'll help work with encoding if needed. Does it look like we'll need better encoding to insert the files you need? I'm going to continue work on decoding everything mainly for fun since its unlikely will need every single type of encoding method.