Archive for category Programming

FFXIV Time released

It’s taken longer than I expected but FFXIV Time is finally available for the Pebble!

FFXIV Time is a watch app that keeps track of Eorzea time which is used in Square Enix’s successful MMORPG Final Fantasy XIV: A Realm Reborn. With Heavensward close to release there’s no better time to prepare by catching up on your mining, fishing, botany or sight-seeing log.  FFXIV Time helps you keep an eye on Eorzea time, wherever you are!

Picture of FFXIV Time in the Pebble App Store

Big thanks to Angie for hand drawing the chocobo sporting a red Pebble running a copy of the app which is used in both the apps icon and the banner image.

I’d also like to thank the following kind strangers from /r/ffxiv who helped me test the app and get it working across multiple time zones: ChaoticAgenda, OutcastKnight, tecrogue and GoldenCrater.

Whilst I mull over whether to build an enhanced Pebble Time version and investigate options for alerts check out my account of why writing FFXIV Time took 1 week instead of 1 day.

Tags: , , ,

FFXIV Time: The 1 day Pebble app that took 1 week.

You would think that writing a Pebble application to track the time in Eorzea, a region of the world found in Final Fantasy XIV, would be simple. After all displaying the time is pretty much the definition of what a watch should do.

Eorzea time isn’t the same as ‘Earth’ time, it’s considerably faster but follows a set of rules which means we can recreate it outside of the game if we know how. I thought I’d be done within a few hours, after all the maths behind it is already well documented and the Pebble SDK makes it easy to write watch apps, but I wasn’t expecting to hit so many issues along the way:

1) The Pebble uses 32 bit integers.  
This means that the Pebble is susceptible to the year 2038 problem, also known as the Unix Millennium Bug.  This normally wouldn’t be an issue, after all the flash storage and the battery in the Pebble will be long dead by the time we reach 2038 but it means the standard time_t C structure can’t hold FFXIV date/time values.
Fortunately the localtime() function we need to format date/time strings can accept the int64_t data type so we can use that for holding Eorzea time values.

2) The Pebble maths precision returns incorrect values with large numbers.
Instead of the correct Eorzea time being calculated the watch generates a number that’s 44 minutes behind the actual in-game time.  Luckily this is fairly consistent so we can manually correct in code.

3) The Pebble itself has no concept of time zones.
Eorzea time is based on the number of seconds since 01/01/1970, we can get that information from the Pebble but the number returned is based on the local time.  Without knowing what time zone the user is in we can’t calculate Eorzea time at all.  Luckily the Pebble SDK now includes the capability to run some helper code on the phone itself which means we can determine the users time zone and send that back to the watch.  We can’t do this too often though as any communication with the phone is additional battery drain and we don’t want to tie the app to working only when the phone is around.

4) The time on the watch seems to drift over extended periods of time.
This happens more noticibly on my 1st generation Pebble and less so on my Pebble Steel.  Normally this wouldn’t matter too much, if your watch ticks over to 5pm 10 seconds faster/slower than your friends the core functionality is still good enough.
Unfortunately drift of even a few seconds can have a noticeable impact on calculating Eorzea time.  We can use a similar solution to the one implemented to fix problem 3 to figure out if there is a difference between the time on the phone and that of the watch and compensate accordingly.

5) Communication with the phone isn’t reliable.  
Problems 3 & 4 were solved by retrieving information from the phone.  If we can’t communicate with the phone on first launch to get time zone information we can’t calculate Eorzea time and the app is effectively useless.  During testing I found that every so often phone communication failed at app launch and with the next phone communication scheduled for an hour later (to save battery) the only way for the user to fix it was to restart the app until the communication worked.  I like things that ‘just work’ so the the (heavy handed) fix I’ve implemented is to check if we have cached time zone information and if not, blast the phone with up to 30 communication calls to ensure we get something back. The feedback has been that after this routine was implemented the time self heals within seconds of the app launching without users having to do anything at all.


It’s been quite a journey creating FFXIV Time (soon to be available on the Pebble Marketplace), several of the problems took longer to identify and resolve than I had initially thought the entire application would take me to write.  In contrast, the proof of concept I wrote for Apple’s iWatch took only a few hours.  I’m still fond of the Pebble, particularly as the iWatch doesn’t run native apps at the moment, but the difference in the time required to write the core functionality of effectively the same application is staggering.

Next challenge: Update FFXIV Time to support the colour features of the Pebble Time.  Though I think I might wait for the SDK to become stable before embarking on that journey…

Tags: , , ,

Python script to generate Xbox save signing keys

A while ago I discussed how save signing for the original Xbox works and the methodology to generate the correct key to resign a save.  Save Hacking is becoming something of a legacy these days and I’m keen to do what I can to enable others to continue to work on what I guess is called ‘retro’ these days.

With that in mind I’ve uploaded a quick and dirty Python script to generate Xbox save signing keys to GitHub.

Using xbox-save-sig is as simple as passing in the path to a default.xbe file:

python -i /path/to/default.xbe

The output will be a text representation of the digital signature for save signing:


The script can output the key in a selection of formats and being Python can run on many different systems.

The script is licensed under the MIT License, which is pretty permissive, so feel free to fork, improve and put it to good use!

Tags: , ,

Xbox save resigning – a technical overview

This article covers the saves signing process used by the original Xbox.  Only the standard signing process is discussed, the “noroam” signatures are not covered.  All code is based on my own work or derivative work of others.  The language used in examples is Delphi and assumes a familiarity with programming concepts such as the use of records or structures for reading in data.

The reason I’ve never previously discussed the Xbox save signing procedure in public is concern that by doing so would negatively impact Xbox Live.  Now that the original Xbox and it’s games are unable to access the Xbox Live service the following information is relatively harmless.

A brief history

The original Xbox used digital signatures to validate not only executables it was loading but save data, with this it was impossible to change or tamper with a save without updating the file’s digital signature using the correct key.  This key was generated using data contained within the default.xbe and a key used by the Xbox Operating System.  The save data was then run through a SHA1 HMAC routine and the 20 byte result appended to the save data to confirm integrity. This result, or digest, was usually found at the beginning or the end of a file although it can potentially be located anywhere in the file if the developer was feeling adventurous.

Obtaining the XBE key

Each Xbox game has a default.xbe file, this is the executable that is loaded whenever you start a game. It contains various information including the title ID, age rating, game region and the all important signature key.  Don’t get confused, this is not the key used to sign the saves but is an essential piece in the process.
The first step is to obtain the key directly from the XBE. This is stored in a certificate area of the file, the address of which is located in the XBE header. Before we can retrieve the key we need to define the structure of the XBE Header and the Certificate data we will be retrieving:

?View Code DELPHI
// XBE sections
unit xbestruct;
ByteArray = array[0..15] of byte;
TxbeHeader = record
	m_magic : cardinal;                    // magic number [should be "XBEH"]
	m_digsig : array[0..255] of char;      // digital signature
	m_base: cardinal;                      // base address
	m_sizeof_headers : cardinal;           // size of headers
	m_sizeof_image : cardinal;             // size of image
	m_sizeof_image_header : cardinal;      // size of image header
	m_timedate : cardinal;                 // timedate stamp
	m_certificate_addr : cardinal;         // certificate address
	m_sections : cardinal;                 // number of sections
	m_section_headers_addr : cardinal;     // section headers address
	m_init_flags : cardinal;
	m_entry : cardinal;                         // entry point address
	m_tls_addr : cardinal;                       // thread local storage directory address
	m_pe_stack_commit  : cardinal;                // size of stack commit
	m_pe_heap_reserve : cardinal;               // size of heap reserve
	m_pe_heap_commit  : cardinal;                 // size of heap commit
	m_pe_base_addr : cardinal;                   // original base address
	m_pe_sizeof_image : cardinal;               // size of original image
	m_pe_checksum : cardinal;                // original checksum
	m_pe_timedate : cardinal;                   // original timedate stamp
	m_debug_pathname_addr  : cardinal;            // debug pathname address
	m_debug_filename_addr : cardinal;           // debug filename address
	m_debug_unicode_filename_addr : cardinal;   // debug unicode filename address
	m_kernel_image_thunk_addr : cardinal;        // kernel image thunk address
	m_nonkernel_import_dir_addr : cardinal;      // non kernel import directory address
	m_library_versions : cardinal;               // number of library versions
	m_library_versions_addr : cardinal;          // library versions address
	m_kernel_library_version_addr : cardinal;    // kernel library version address
	m_xapi_library_version_addr : cardinal;      // xapi library version address
	m_logo_bitmap_addr : cardinal;               // logo bitmap address
	m_logo_bitmap_size : cardinal;               // logo bitmap size
pTXbeHeader = ^TXbeHeader;
TxbeCertificate  = record
	m_size : cardinal;                          // size of certificate
	m_timedate : cardinal;                     // timedate stamp
	m_titleid : cardinal;                       // title id
	m_title_name : array[0..63] of widechar;                // title name (unicode)
	m_alt_title_id : byteArray;            // alternate title ids
	m_allowed_media  : cardinal;                 // allowed media types
	m_game_region  : cardinal;               // game region
	m_game_ratings  : cardinal;                  // game ratings
	m_disk_number : cardinal;                   // disk number
	m_version : cardinal;                       // version
	m_lan_key : byteArray;                 // lan key
	m_sig_key : byteArray;                  // signature key
	m_title_alt_sig_key : array[0..15, 0..15] of byte;     // alternate signature keys
pTxbeCertificate = ^ TxbeCertificate;

The TxbeHeader record contains a lot of data but the two most important sections are m_certificate_addr, and m_base.  These two values give us the address of the certificate.

The following code retrieves the key from an XBE file:

?View Code DELPHI
    MS : TMemoryStream;
    xbeHeader : pTxbeHeader;
    xbeCert : pTxbeCertificate;
    MS := TmemoryStream.Create;
    //Read in header and certificate
    MS.Read(xbeHeader^, sizeof(xbeHeader^));
    MS.Position := xbeHeader^.m_certificate_addr - xbeHeader^.m_base;
    MS.Read(xbeCert^, sizeof(xbeCert^));

We can now access the sig_key directly

?View Code DELPHI

Generating the signing key

The key we retrieved from the XBE is not used to directly sign save data, instead it is used in a SHA1 HMAC with the Xbox key to produce the actual key we need. Thankfully we don’t need to determine the Xbox key every time we need to sign or verify a save as it is a constant.

The following is a textual representation of this key, to use it you must first convert it to a 16 byte array:


Once you have this key in a byte array simply run both the XBE key and the Xbox key through a SHA1 HMAC (referring to the documentation for your SHA1 HMAC function as to whether you need to pass data or memory addresses to the function). The output should be a 160bit digest truncated to 16 bytes (the last 4 bytes are not required and should not be used or present).

?View Code DELPHI
    digest : T160BitDigest;
    digest := CalcHMAC_SHA1(addr(xboxKey), 16, xbeCert^.m_sig_key, 16);

Determining the data to process

As noted before the digital signature is usually found before or after the actual save data.  It is entirely possible to store this signature at any location in the file and treat this location as all 0’s during the HMAC process.  Since we cannot rely on all developers using the same location for the signature we must determine the location ourselves.

The fastest and easiest way to do this is to start a new game and make a save at the earliest opportunity.  If the game saves any options you change this is ideal, otherwise start playing and save as soon as you can.  Copy this save to you PC and label is SAVE A.

Load the game again and create another save with as small a difference as you can but ensuring something is different.  As before, if the game saves option changes you should change only a single item and re-save. Copy this save to your PC and label it SAVE B.

Open both saves in a hex editor and visually compare the two, with little differences between them you should easily spot the 20 byte digital signature either at the start or the end of the file.  You can use an automatic file comparison if your editor supports it but in my experience I can find the sig faster by eye.  The signature in both files should be wildly different with the actual save data very consistent, aside from the minor differences you saved earlier.

Once you have found the sig location you should exclude this from the HMAC routines.  If the sig is at the end of the file you should HMAC all data up to the last 20 bytes, if it is at the beginning then all data after the first 20 bytes should be processed.  If the signature is located at another location in the file then you will need to experiment as to what data to HMAC.  A common trick when generating checksum’s is to treat the area containing the result of the checksum as all 0’s during the processing stage and write the result back to this location once complete.  Please note, it is rare to see this, most saves store their signature at the beginning or end of the file.

Generating the signature

Now you have the correct key and the data to process you can generate or check signatures for that particular games save files.  Similar to how the key was generated, the actual signature generation is a SHA1 HMAC of the save data (excluding the existing signature) with the key.

In the following example the save signature is located at the end of the file and the data is copied to a different memory stream for processing. The previously generated key is stored in a 16 byte array named sigKey

?View Code DELPHI
mem := TMemoryStream.Create;
mem2 := TMemoryStream.Create;
mem.Position := 0;
mem2.CopyFrom(mem, mem.Size - 20);
digest := CalcHMAC_SHA1(addr(sigKey), 16, mem2.memory^, mem2.size);

Before making any changes to a save you should confirm that the signature you generate matches that already present on an unaltered save.

It’s worth bearing in mind that a lot of saves contain a checksum as well as a digital signature.  The most common being a CRC32 or a simple addition of bytes, you must recalculate this before the digital signature should you make any changes to the file.


The process of creating a digital signature for Xbox saves is fairly simple and can be summarized as

  • SHA1 HMAC “XBE key” using “Xbox key”
  • Truncate resulting 160 bit digest to 16 bytes to create the “signature key”
  • SHA1 HMAC Save data using “signature key”
  • Compare resulting 160 bit digest to existing signature or write back to save file.

If you use the information above to create anything or produce a unique save, drop me a line in the comments, sometimes the most interesting time in a games life is long after it was released..

Tags: , , ,

Fixing WP e-Commerce

I’ve been implementing the WordPress e-Commerce plug-in on another site, it’s not a bad freebie but if you’re selling anything more than a handful of items then you’ll need to let your users search your products.  To add this functionality, along with a few other features, you need to purchase the $50 Gold Cart add-in.

I duly purchased and installed the update only to find that the “search” only looked at product titles, meaning that all the description text, additional description text and meta tags were being ignored.   Looking around it appears I wasn’t the only person who needed this functionality and I was able to get hold of some code to change in gold_shopping_cart.php. I still didn’t get the results I was expecting so after checking out the Database table structure I went from

$category_list = $wpdb->get_col("SELECT `id` FROM `".WPSC_TABLE_PRODUCT_CATEGORIES."` WHERE `name` LIKE '".$search_string_title."'");


$category_list = $wpdb->get_col("SELECT id FROM `".WPSC_TABLE_PRODUCT_LIST."` WHERE name LIKE '".$search_string_title."' OR description LIKE '".$search_string_title."' OR additional_description LIKE '".$search_string_title."'");

Notice the change in the table being searched. The original code I found still looked in the WPSC_TABLE_PRODUCT_CATEGORIES table but this does not contain the additional_description field, an oversight on the original modders part.

Yet I still wasn’t getting the expected results. A quick count of the $category_list array showed I was getting results but they were getting filtered out further on in the function by this little check:

if($category_list != null) {
				$category_assoc_list = $wpdb->get_col("SELECT DISTINCT `product_id` FROM `".WPSC_TABLE_ITEM_CATEGORY_ASSOC."` WHERE `category_id` IN ('".implode("', '", $category_list)."')");
				$category_sql = "OR `".WPSC_TABLE_PRODUCT_LIST."`.`id` IN ('".implode("', '", $category_assoc_list)."')";

Remember that I’m now searching products directly and the above code is looking to match the products unique ID against a category ID, clearly this will fail on 99% of searches. A quick change to the search criteria is all that was needed to return valid results:

if($category_list != null) {
				$category_assoc_list = $wpdb->get_col("SELECT DISTINCT `product_id` FROM `".WPSC_TABLE_ITEM_CATEGORY_ASSOC."` WHERE `product_id` IN ('".implode("', '", $category_list)."')");
				$category_sql = "OR `".WPSC_TABLE_PRODUCT_LIST."`.`id` IN ('".implode("', '", $category_assoc_list)."')";

With that completed I looked at adding a separate search interface and came across WP e-Commerce Search Widget, a handy bit of code but unfortunately it doesn’t appear to be compatible with the latest version of WP e-Commerce so I had to quickly update

<form method='GET' name='product_search' id="productsearchform" action="<?php bloginfo('home');  ?>/">


<form method='GET' name='product_search' id="productsearchform" action="<?php bloginfo('home');  ?>/products-page/">

WP e-Commerce is a lot of work, especially considering you have to pay $50 just to get basic functionality and then hack in the desired features. Lets hope the checkout refresh bug doesn’t hit again….

Tags: , , ,

Halo 3 Asset-O-Matic R.I.P?

As expected, the redesign of; has stopped the Halo 3 Asset-O-Matic from working.

I’m not sure if I will be updating the app to support the new layout, I’ll have to see how inspired I am with the release of ODST…;)

Tags: ,

Halo 3 Asset-O-matic Beta 4

Download Halo 3 Asset-O-Matic Beta 4

What happened to Beta 3 you may ask. Well Bungie made a few changes in between me completing Beta 3 and actually finding time to upload it. By that time it no longer worked.

So, what you have in your metaphysical hands is Beta 4, it’s just like Beta 3 but it works!

The changes are:

*You can now search Matchmaking or Custom games

*You can now go directly to the files instead of the match details

*It is significantly faster (think 5 minutes to search over 2500+ matches..)

*Templates are working (sorta, no word wrapping enabled and no way to clear the output box). This allows you to enter some text in the top section, hit a button and have your text repeated with the following tokens replaced by data from the currently loaded database: %SCREENS%, %CLIPS%, %GAMEID% (used for linking directly to the game), %GAMETYPE%, %GAMEDATETIME% and %MAP%

Alas, Campaign searching did not make it in. I wrote half the code and then realised it needs a whole new DB format to support the relevent information found for a Campaign record. I will add support once my busy work life allows.

Databases from previous versions are not compatible with this new release. A fresh search is always preferred in this house 😀

Give it a try yourself, tell your friends! Download the Halo 3 Asset-O-Matic Beta 4 today!

(Feedback is always appreciated folks! 🙂 )

Tags: , , ,

Another update, another revision required

The latest changes to broke Halo 3 Asset-O-Matic again. I’m working on a new build featuring:

  • Search Matchmaking or Custom Games
  • Direct linking to the files page
  • Faster searching (5 times more grabbing threads)

A problem with increasing the amount of network activity is that the thread checking for the next page of results sometimes times out. It’s fine if a grabber thread times out because there’s 24 more left to continue to continue the work but there’s only one thread looking for the next page. I’ll have to see if I can add some fallback code to reinitilise the thread if it falls over.

Tags: ,

Beta 3?

Halo 3 Asset-O-Matic Beta 3

Halo 3 Asset-O-Matic Beta 3

Tags: , ,

Halo 3 Asset-O-Matic Beta 2

Halo 3 Asset-O-Matic Beta 2 is finished, tested and confirmed working with the redesign of 🙂

This is a hot fix build, all previously created databases are fully compatible with this version. I’m considering implementing support for Custom Games and Campaign game history but this will require the existing code being refactored, something I haven’t yet had time to do…

Tags: , ,