Posts Tagged Delphi

Xbox save resigning – a technical overview

This article covers the saves signing process used by the original Xbox.  Only the standard signing process is discussed, the “noroam” signatures are not covered.  All code is based on my own work or derivative work of others.  The language used in examples is Delphi and assumes a familiarity with programming concepts such as the use of records or structures for reading in data.

The reason I’ve never previously discussed the Xbox save signing procedure in public is concern that by doing so would negatively impact Xbox Live.  Now that the original Xbox and it’s games are unable to access the Xbox Live service the following information is relatively harmless.

A brief history

The original Xbox used digital signatures to validate not only executables it was loading but save data, with this it was impossible to change or tamper with a save without updating the file’s digital signature using the correct key.  This key was generated using data contained within the default.xbe and a key used by the Xbox Operating System.  The save data was then run through a SHA1 HMAC routine and the 20 byte result appended to the save data to confirm integrity. This result, or digest, was usually found at the beginning or the end of a file although it can potentially be located anywhere in the file if the developer was feeling adventurous.

Obtaining the XBE key

Each Xbox game has a default.xbe file, this is the executable that is loaded whenever you start a game. It contains various information including the title ID, age rating, game region and the all important signature key.  Don’t get confused, this is not the key used to sign the saves but is an essential piece in the process.
The first step is to obtain the key directly from the XBE. This is stored in a certificate area of the file, the address of which is located in the XBE header. Before we can retrieve the key we need to define the structure of the XBE Header and the Certificate data we will be retrieving:

?View Code DELPHI
// XBE sections
unit xbestruct;
 
interface
 
type
ByteArray = array[0..15] of byte;
 
type
TxbeHeader = record
	m_magic : cardinal;                    // magic number [should be "XBEH"]
	m_digsig : array[0..255] of char;      // digital signature
	m_base: cardinal;                      // base address
	m_sizeof_headers : cardinal;           // size of headers
	m_sizeof_image : cardinal;             // size of image
	m_sizeof_image_header : cardinal;      // size of image header
	m_timedate : cardinal;                 // timedate stamp
	m_certificate_addr : cardinal;         // certificate address
	m_sections : cardinal;                 // number of sections
	m_section_headers_addr : cardinal;     // section headers address
	m_init_flags : cardinal;
	m_entry : cardinal;                         // entry point address
	m_tls_addr : cardinal;                       // thread local storage directory address
	m_pe_stack_commit  : cardinal;                // size of stack commit
	m_pe_heap_reserve : cardinal;               // size of heap reserve
	m_pe_heap_commit  : cardinal;                 // size of heap commit
	m_pe_base_addr : cardinal;                   // original base address
	m_pe_sizeof_image : cardinal;               // size of original image
	m_pe_checksum : cardinal;                // original checksum
	m_pe_timedate : cardinal;                   // original timedate stamp
	m_debug_pathname_addr  : cardinal;            // debug pathname address
	m_debug_filename_addr : cardinal;           // debug filename address
	m_debug_unicode_filename_addr : cardinal;   // debug unicode filename address
	m_kernel_image_thunk_addr : cardinal;        // kernel image thunk address
	m_nonkernel_import_dir_addr : cardinal;      // non kernel import directory address
	m_library_versions : cardinal;               // number of library versions
	m_library_versions_addr : cardinal;          // library versions address
	m_kernel_library_version_addr : cardinal;    // kernel library version address
	m_xapi_library_version_addr : cardinal;      // xapi library version address
	m_logo_bitmap_addr : cardinal;               // logo bitmap address
	m_logo_bitmap_size : cardinal;               // logo bitmap size
end;
 
pTXbeHeader = ^TXbeHeader;
 
type
TxbeCertificate  = record
	m_size : cardinal;                          // size of certificate
	m_timedate : cardinal;                     // timedate stamp
	m_titleid : cardinal;                       // title id
	m_title_name : array[0..63] of widechar;                // title name (unicode)
	m_alt_title_id : byteArray;            // alternate title ids
	m_allowed_media  : cardinal;                 // allowed media types
	m_game_region  : cardinal;               // game region
	m_game_ratings  : cardinal;                  // game ratings
	m_disk_number : cardinal;                   // disk number
	m_version : cardinal;                       // version
	m_lan_key : byteArray;                 // lan key
	m_sig_key : byteArray;                  // signature key
	m_title_alt_sig_key : array[0..15, 0..15] of byte;     // alternate signature keys
end;
 
pTxbeCertificate = ^ TxbeCertificate;
 
implementation
 
end.

The TxbeHeader record contains a lot of data but the two most important sections are m_certificate_addr, and m_base.  These two values give us the address of the certificate.

The following code retrieves the key from an XBE file:

?View Code DELPHI
var
    MS : TMemoryStream;
    xbeHeader : pTxbeHeader;
    xbeCert : pTxbeCertificate;
begin
 
    MS := TmemoryStream.Create;
    MS.LoadFromFile(FileName);
    new(xbeHeader);
    new(xbeCert);
    //Read in header and certificate
    MS.Read(xbeHeader^, sizeof(xbeHeader^));
    MS.Position := xbeHeader^.m_certificate_addr - xbeHeader^.m_base;
    MS.Read(xbeCert^, sizeof(xbeCert^));

We can now access the sig_key directly

?View Code DELPHI
xbeCert^.m_sig_key

Generating the signing key

The key we retrieved from the XBE is not used to directly sign save data, instead it is used in a SHA1 HMAC with the Xbox key to produce the actual key we need. Thankfully we don’t need to determine the Xbox key every time we need to sign or verify a save as it is a constant.

The following is a textual representation of this key, to use it you must first convert it to a 16 byte array:

5C0733AE0401F7E8BA7993FDCD2F1FE0

Once you have this key in a byte array simply run both the XBE key and the Xbox key through a SHA1 HMAC (referring to the documentation for your SHA1 HMAC function as to whether you need to pass data or memory addresses to the function). The output should be a 160bit digest truncated to 16 bytes (the last 4 bytes are not required and should not be used or present).

?View Code DELPHI
var
    digest : T160BitDigest;
begin
    digest := CalcHMAC_SHA1(addr(xboxKey), 16, xbeCert^.m_sig_key, 16);

Determining the data to process

As noted before the digital signature is usually found before or after the actual save data.  It is entirely possible to store this signature at any location in the file and treat this location as all 0’s during the HMAC process.  Since we cannot rely on all developers using the same location for the signature we must determine the location ourselves.

The fastest and easiest way to do this is to start a new game and make a save at the earliest opportunity.  If the game saves any options you change this is ideal, otherwise start playing and save as soon as you can.  Copy this save to you PC and label is SAVE A.

Load the game again and create another save with as small a difference as you can but ensuring something is different.  As before, if the game saves option changes you should change only a single item and re-save. Copy this save to your PC and label it SAVE B.

Open both saves in a hex editor and visually compare the two, with little differences between them you should easily spot the 20 byte digital signature either at the start or the end of the file.  You can use an automatic file comparison if your editor supports it but in my experience I can find the sig faster by eye.  The signature in both files should be wildly different with the actual save data very consistent, aside from the minor differences you saved earlier.

Once you have found the sig location you should exclude this from the HMAC routines.  If the sig is at the end of the file you should HMAC all data up to the last 20 bytes, if it is at the beginning then all data after the first 20 bytes should be processed.  If the signature is located at another location in the file then you will need to experiment as to what data to HMAC.  A common trick when generating checksum’s is to treat the area containing the result of the checksum as all 0’s during the processing stage and write the result back to this location once complete.  Please note, it is rare to see this, most saves store their signature at the beginning or end of the file.

Generating the signature

Now you have the correct key and the data to process you can generate or check signatures for that particular games save files.  Similar to how the key was generated, the actual signature generation is a SHA1 HMAC of the save data (excluding the existing signature) with the key.

In the following example the save signature is located at the end of the file and the data is copied to a different memory stream for processing. The previously generated key is stored in a 16 byte array named sigKey

?View Code DELPHI
mem := TMemoryStream.Create;
mem2 := TMemoryStream.Create;
mem.LoadFromFile(FileName);
mem.Position := 0;
mem2.CopyFrom(mem, mem.Size - 20);
digest := CalcHMAC_SHA1(addr(sigKey), 16, mem2.memory^, mem2.size);

Before making any changes to a save you should confirm that the signature you generate matches that already present on an unaltered save.

It’s worth bearing in mind that a lot of saves contain a checksum as well as a digital signature.  The most common being a CRC32 or a simple addition of bytes, you must recalculate this before the digital signature should you make any changes to the file.

Conclusion

The process of creating a digital signature for Xbox saves is fairly simple and can be summarized as

  • SHA1 HMAC “XBE key” using “Xbox key”
  • Truncate resulting 160 bit digest to 16 bytes to create the “signature key”
  • SHA1 HMAC Save data using “signature key”
  • Compare resulting 160 bit digest to existing signature or write back to save file.

If you use the information above to create anything or produce a unique save, drop me a line in the comments, sometimes the most interesting time in a games life is long after it was released..

Tags: , , ,

Halo 3 Asset-O-Matic Beta – Soon

Halo 3 logoThe Asset-O-Matic is coming along nicely now. The database code has been completed and I’m now planning the best way to display the stored information, this is where a public beta will help no end with people feeding back as to what they want to see and how. Speed wise things are looking a lot better, I have just scanned my current game history (1150 games) and it took 19 minutes 50 seconds. A vast improvement from the original 50 minutes for 1000(ish) games.
I’ve got an occasional memory leak to track down, it’s currently escaping me as everything I create is destroyed correctly and the leak can occur on searches that previously had no problems. It’s a little random and hard to repeat on a regular basis so the first beta might come with a warning to restart the application for every new game history you scan as it’d be a shame to hold off on release for such a minor and occasional problem.

Update
I’ve fixed the memory leak, when scanning a gamertag with only a couple of games played the downloader threads would occasionally finish before the scanning thread had processed the games. Another bug left over from when I upped the amount of downloader threads from 1 to 5.

Tags: , , , ,

Halo 3 Asset-O-Matic now 50% faster

Halo 3 logoWorking down in Worcester last weekend left me little time to work on the Halo 3 Asset-O-Matic but I was able to steal an hour or so away from the current Double XP weekend playlist to do some further work. I’ve beefed up the amount of downloader threads from 1 to 5 which has halved the time needed to analyse my game history, 1167 games at the time of writing, down to 23 minutes from 50. It’s still more than I had hoped but the bottleneck is the speed at which Bungie.net is responding. The thread that strips the information from the pages never falls behind and is working on each page as it arrives. I could throw more downloader threads in there but I’m not sure what affect this will have on Bungie.net I guess the first release will have to be a slow but sure affair.

The database code hasn’t been started yet, the program is currently erroring on completion. Nothing show stopping but something that needs addressing before I move onto the DB code. With the recent announcement that Bungie are going to give the helm of Bungie Favourites to the community I can see an increased need to locate which games have assets in your history.
I’ll be working on stopping the completion error today, I’m not sure why it’s only just started happening so that’s an interesting challenge. I’ll also need to find a profile that hasn’t played many Halo 3 matches for testing, another challenge in itself 😀

Update
I’ve fixed the error, I hadn’t accounted for going from 1 thread downloading to 5 which meant another thread was exiting early. I also picked up on a subtle bug where 1 of your games was never processed. Hardly a killer bug but I’m glad I spotted it before the public beta (as soon as the DB code is written) 🙂

Tags: , , , ,

AR Max save format details coming soon

Ar max packagingThanks to some sterling work by Ross Ridge, who has already completed work on two projects I was looking at starting, the details of the .max PS2 save format have now been uncovered.

I’m documenting the file structure and compression used and also preparing a DLL that will allow people to add .max support to their programs. As a taster here’s the .max header, part 1 of a 3 part system:

type
TMaxheader = record
   magic : array[0..11] of char; //Ps2PowerSave
   checksum : integer; //CRC32 of entire file with checksum area treated as all 0’s
   dirname : array[0..31] of char; //parent directory name
   iconSysName : array[0..31] of char; //icon.sys text
   compressedSize : integer; //size of compressed data
   fileNum: integer; //number of files
   length: integer; //size of uncompressed data
end;

What really suprised me is that the .max format has no provision for recording file attributes! Unlike the .psu format which maintains file system attributes, cruicial for accurate restoration of files back to a PS2 memory card, the .max format appears to restore files with a generic set of attributes. Luckily, or even by design, these attributes match those required by GTA saves but this runs the risk that a file is restored without the correct attributes and a different game will treat the save as corrupt.

On the same note, I’m concerned that the .max format does not support PSOne and Pocketstation files as these require a specific attribute to be set. More investigation is required for clarification.

Tags: , ,

GTA: SA Censor Remover Online AKA Conversion Hell

I often get requests for versions of my tools that will work on non-Windows operating systems, such as Linux and MacOSX.
Because I use delphi, which is primarily a Windows programming tool, this often isn’t practical or even possible.

A few months ago I set about thinking of ways I could provide the same tools and abilities to those who don’t run Windows or have restrictions set on what they can download. I eventually decided to recreate one of my more popular tools, GTA: SA Censor Remover, in an online edition.

This wasn’t as easy a task as I had first assumed, my PHP skills were a little rusty from years of neglect in favour of Windows application development and PHP is an inherently poor language for working with binary files. I could have deviated from my original intention and written the online version in Perl, but I’m not that much of a masochist and I’m much more familiar with PHP.
After a few hours I was able to knock up a basic version that worked fine with PS2 and PC saves, even with PHP’s poor file handling routines, but came against a huge stumbling block with the Xbox saves.

Xbox saves are protected by a digital signature, change even one byte and the signature is invalidated and the save will not load. Of course I know how to fix this signature but this is where PHP let me down immensly.

PHP is a loosely type language, this means it’s great for beginners but sometimes you need to be able to control the language and it’s variables precisely, which is exactly what I needed to do. For reference, Delphi is a strongly typed language and is much better for it in my opinion. I can also access raw memory directly with Delphi which makes creating digital signatures a breeze.

I spent hours trying different ways of representing the binary data I needed to fix the digital signature in the save, I trawled the PHP manual for hints and comments. I even asked an experienced PHP programmer who writes scripts for a living daily and he was unable to help me or offer anything I hadn’t tried already.
Eventually I gave up, I’d chosen the wrong tool for the job and had paid bitterly for it.

Last night I was intrigued to see that the makers of Delphi, Codegear, are planning a “Delphi for PHP” and this rekindled my interest in PHP. Looking for a project to start I remembered about the work I had done on GTA: SA Censor Remover Online and how I had left it to rot on my dev machine.
I decided that I couldn’t let this go to waste and was a perfect challenge to get my PHP skills back on track, after all if I couldn’t find the answer from the manual and experienced users of PHP than I’d have to find it myself.
Eventually after a few tests, all producing the wrong digital signature, I commented out all the existing code and returned to the basic task: Create a binary representation of a series of bytes and pipe this along with the raw file data to the required function.

It was a matter of minutes before I had a working function. I spent a few hours tweaking the code and making sure it worked properly, again having to fight PHP’s poor file handling tooth and nail, until the final product was ready for release.

It’s been a hellish struggle and at times I wondered if the task was even possible given the poor performance of PHP in this area and lack of information out there, even from PHP experts. I won’t be retro-fitting any of my other tools at this time but if the need arises, or another prgram is created that would bendfit from an online version, I’ve got a solid groundwork to enable access to my software to everyone, regardless of what operating system they use.

Tags: , , , , , ,

Using Critical Sections with Delphi

Delphi makes threading a very easy process, I’ve used it on numerous projects however there’s always room to learn new tricks.

When multiple threads want to access the same shared data you have to lock it to avoid complications, in my projects I use a TCriticalSection.
The Delphi docs and even my Mastering Delphi book were very vague on the use of this and so I’m hoping that this post will help others avoid the pitfall of misuse I did:

I assumed that I could use a Critical Section like so:

//vars, I use them as part of a class
var
cs : TCriticalSection;
anInt: integer;
aList: TList;

//create the CriticalSection when my class is created
TMyClass.Create
begin
cs := TCriticalSection.Create;
end;

//thread wants to access the int
cs.Enter;
doSomethingtoInt(anInt);
cs.leave;

//thread want sto do something to the TList
cs.Enter;
aList.add(anItem);
cs.Leave;

This is very wrong for the following reasons (and probably more):
1) I should have use a TThreadSafeList with auto locking built in for thread safe use
2) Using the same CriticalSection on two different sections of data is completely wrong. Each data that needs locking and is used in different routines or accessed at different times should have it’s own CriticalSection like so:

//Global vars
var
iCS : TCriticalSection;
lCS : TCriticalSection
anInt: integer;
aList: TList;

//create the CriticalSection when my class was created
TMyClass.Create
begin
iCS := TCriticalSection.Create;
lCS := TCriticalSection.Create;
end;

//thread wants to access the int
iCS.Enter;
doSomethingtoInt(anInt);
iCS.leave;

//thread wants to do something to the TList
aCS.Enter;
aList.add(anItem);
aCS.Leave;

Again, I must stress that if you need to use a TList with threads a much saner and safer option is to use a TThreadSafeList which locks the contained TList for you. If you need to access the internal TList of a TThreadSafeList the following code works great:

with threadedList.LockList do
try
for x := 0 to Count – 1 do begin
doSomething(items[x]);
end;
finally
threadedList.UnlockList;
end;

However simple actions such as adding an item are like so:

threadedList.Add(anItem);

A simple unit containing a class that spawns several threads and uses TCriticalSections to protect the data would look like so:

unit myClassUnit;

interface
uses
SyncObjs;

type
myClass = class
private
anInt: Integer;
aWord: word;
intCS: TCriticalSection;
wordCS: TCriticalSection;
public
Constructor Create;
Destructor Destroy;
Procedure IncInt;
Procedure IncWord;
procedure decInt;
Procedure spawnThread;
end;

implementation

{ myClass }

constructor myClass.Create;
begin
anInt := 0;
aWord := 0;
intCS := TCriticalSection.Create;
wordCS := TCriticalSection.Create;
end;

destructor myClass.Destroy;
begin
intCS.Free;
wordCS.Free;
end;

procedure myClass.decInt;
begin
intCS.Enter;
dec(anInt);
intCS.Leave;
end;

procedure myClass.IncInt;
begin
intCs.Enter;
inc(anInt);
intCS.Leave;
end;

procedure myClass.IncWord;
begin
wordCS.Enter;
inc(aWord);
wordCS.Leave;
end;

procedure myClass.spawnThread;
var
aThread : TMyThread;
begin
aThread := TMyThread.Create(False);
end;

end.

Usage would be as simple as

var
theClass : TMyClass;
x : integer;
begin
theClass := TMyClass.Create;
for x := 0 to 3 do begin
theClass.spawnThread;
end;

with the threads calling the inc or dec procedures like so:

theClass.incInt;
theClass.decInt;
theClass.incWord;

with no risk of data corruption being caused by multiple or conflicting calls to the procedures by different threads.

Tags: , ,

Turbo Delphi, pcsx2 memory card format and a secret project…

I thought it was about time I started blogging again, my main site www.ps2savetools.com is too focused on one subject and not the place for my thoughts.

Turbo Delphi

I’ve been using Turbo Delphi since a day before it’s released (I figured out Borlands download system and grabbed the installer and a licence the day before the official release) and it’s great stuff.
The only down-side is not being able to use the Design Time part of components but the components I use the most, drag and drop controls and zip utilities don’t need any visual tweaking so I can easily create them in code. So far it’s taken less than 5 minutes per app to recode and recompile them in Turbo Delphi and I expect my productivity to increase again when I finally get time to code again.
I’d suggest that everyone gives Turbo Delphi a go, it’s a nice, fast and powerful language, a decent GUI and it’s free, what more do you want? The ability to create commercial apps? Yup that’s there too so you have the chance to make some money of your hard work, assuming someone wants to pay for it of course 😉

PCSX Memory card format

I’ve not had much chance to look at this but here’s what I’ve seen so far:

  • The file is a raw image of a PS2 memory card.
  • I believe it’s a similar system to FAT12/16, it’s got a similar FAT usage from my quick scan.
  • Every 512 bytes is a 16 byte footer, I assume it’s some sort of checksum.

If you’re writing apps to support this file format do some research on FAT16 (particulary how file and directory entries are stored) and don’t forget to account for the 16 byte footer, this should not be included in any files you extract from the memorycard image.

The secret project….

I can’t reveal much at this stage apart from it’s a PC utility for the Xbox360, it was started many many months ago and I’m going to start up development of it again. FNG might be happy to hear this 😉

Tags: , ,