Tool for syncing game saves to/from Dropbox

Introduction

It’s no secret that apart from programming I enjoy a good relaxing game to take my mind off things. This is the sole reason I still have Windows OS installed in a partition. Recently with Steam embracing Linux more and more my Archlinux machine also allows me to play some nice games.

steam1.png

This post will address syncing saves for various games that don’t support Steam Cloud, but focus solely on Pillars Of Eternity as an example.

Syncing saves on the cloud

The easy way

Steam offers its games the ability to synchronize saves on the cloud across multiple machines and platforms. The developers of the game have to take advantage of the feature though. One such very nicely working example is Terraria.

terraria_cloud_1.png

All you need to do is make sure that the cloud icon under a character or a world is activated and then you are set.

Before I played around with Steam in Linux I was playing this game in Windows. When I installed Steam in ArchLinux and got Terraria, my character and the world I created was there ready for me to play. Really awesome.

The not so easy way

Unfortunately some games don’t utilize the Steam cloud sync in a cross platform way and can’t offer the same service that Terraria does. One such example is Pillars of Eternity as can also be seen by posts like this. True enough I encountered the same problem when trying to sync my saves from Windows to Linux. Steam cloud was just not working.

To make matters worse I also got 2 Linux machines which I would also like to keep in sync. Thankfully the PoE save games from Windows are indeed compatible with Linux as I have tested this myself.

A kind of brute-force way to sync them would be to use Dropbox. The save games location differs depending on your Operating system so check it here. Simply copy the whole directory to Dropbox and then copy from Dropbox to the target machine when you want to sync. Unfortunately this can be a hassle and hard to maintain.

The proposed tool

I really like Pillars Of Eternity and would appreciate to be able to sync saves between different machines without a hassle. This is why I spent a bit of time to write a tool to do that.

The tool is a python script and is in Github. It has been written with a more Generic purpose in mind for syncing saves to Dropbox for any game that does not support it. At the moment only Pillars of Eternity is tested and even for that DO NOT USE without backing up your saves directory first. You have been warned.

Usage

Calling the script is pretty straighforward:

./steamsaves_sync.py

In my Archlinux machine I use dunst as a notification daemon to show notifications in my windows manager (i3-wm).

dunst_notify1.png

This script assumes that you also have dunst installed. If you don’t or if you don’t want to get desktop notifications for each change the script does then also add the --no-notify option to the script.

Once called the script will attempt to check all the save files from the local location and see if a corresponding save file is in Dropbox. If yes then it will check the save timestamp to see which one should be kept and keep it. Finally all Dropbox saves will also be checked to see if any saves are missing from the local directory.

Script in Depth

As I mentioned above this was written with generic saves in mind but for the moment the only game for which the sync process is a bit trustworthy is Pillars of Eternity. The reason for that will be made clear soon.

Embedded in the script for now (later could, and probably should be a different file) is a list of GameEntry objects which should be filled in by the user and explain which games to sync, how, and from which directories.

GameEntry Class

The GameEntry class is:

def defaultSaveNameCB(f):
    return basename(f)


def defaultGetSaveTime(f):
    return getmtime(f)


class GameEntry():
    def __init__(
            self,
            name,
            steamPath,
            dropboxPath,
            saveSuffix=None,
            saveNameCB=None,
            saveTimeCB=None
    ):
        """
        A Game Entry. It consists of:
            name:               A name for the game
            steamPath:          Path where steam stores the saves for this game
            dropboxPath:        Dropbox path where you need the saves backedup.
                                Will be created if it does not exist.
            saveSuffix:         Optional suffix for the save game files. If it
                                is not None then only files with the given
                                suffix in the directory will be considered.
            saveNameCB:         Optional callback to determine the name
                                of the save file to compare by checking the
                                filename or contents. If the function returns
                                \"__IGNORE__\" then that save file is ignored.
                                if the function returns the empty string
                                there has been an error
            saveTimeCB:         Optional callback to determine the in-game
                                time the save was performed. If not given
                                then the last file modification timestamp is
                                taken which can't really be very trustworthy.
        """
        self.name = name
        self.dropboxPath = dropboxPath
        self.steamPath = steamPath
        self.saveSuffix = saveSuffix
        self.saveNameCB = saveNameCB
        if not saveNameCB:
            self.saveNameCB = defaultSaveNameCB
        self.getSaveTime = saveTimeCB
        if not saveTimeCB:
            self.getSaveTime = defaultGetSaveTime

A GameEntry is basically a desciption of a game and how its saves should be handled. It is composed of:

  • name: A name to identify the game with
  • dropboxPath: A path to the location to which the saves should be synced to and synced from. I recommend using Dropbox path for this purpose.
  • steamPath: The path where the Game places its save files which we want to sync.
  • saveSuffix: An optional suffix for the game files. If given, then any files in the directory not ending with this suffix will be ignored.
  • saveNameCB: A function to identify a name for the save game from the file itself. If none is given then the filename is used as the save name identifier.
  • saveTimeCB: A function to identify a save time from a save file. This is a really important function since it’s what determines which of 2 save files is newer and should be kept. The default function checks the file’s last modified date.

Pillars Of Eternity GameEntry

The tool can have many entries but comes with a premade entry for Pillars Of Eternity (PoE). This is how it looks like for my machine.

gamesList = [
    GameEntry(
        "PillarsOfEternity",
        "/home/lefteris/.local/share/PillarsOfEternity/SavedGames",
        "/home/lefteris/Dropbox/saves/PillarsOfEternity",
        "savegame",
        POESaveName,
        POESaveTime
    )
]

A nice TODO, would be to add different locations for different OSes as this now only works for Linux.

If you check inside a save file for PoE you will actually notice that it’s nothing more than a zip archive.

poe_save_1.png

There is one particular file of importance inside that archive which we are interested in and that is saveinfo.xml. Example of the contents of such a file are below:

<Complex name="Root" type="SaveGameInfo, Assembly-CSharp">
  <Properties>
    <Simple name="PlayerName" type="System.String, mscorlib" value="Celina" />
    <Simple name="MapName" type="System.String, mscorlib" value="AR_0301_Ondras_Gift_Exterior" />
    <Simple name="SceneTitle" type="System.String, mscorlib" value="Ondra's Gift" />
    <Simple name="SceneTitleId" type="System.Int32, mscorlib" value="5" />
    <Simple name="Chapter" type="System.Int32, mscorlib" value="2" />
    <Simple name="RealtimePlayDurationSeconds" type="System.Int32, mscorlib" value="84865" />
    <Simple name="PlaytimeSeconds" type="System.Int32, mscorlib" value="2947900" />
    <Simple name="TrialOfIron" type="System.Boolean, mscorlib" value="False" />
    <Simple name="RealTimestamp" type="System.DateTime, mscorlib" value="12/13/2015 02:18:17" />
    <Simple name="SessionID" type="System.Guid, mscorlib" value="81b92ceb-0faf-4ca2-bfbf-396026a95dd9" />
    <Simple name="FileName" type="System.String, mscorlib" value="" />
    <Simple name="SaveVersion" type="System.Int32, mscorlib" value="2" />
    <Simple name="GameComplete" type="System.Boolean, mscorlib" value="False" />
    <Simple name="UserSaveName" type="System.String, mscorlib" value="lef2" />
    <Simple name="Difficulty" type="GameDifficulty, Assembly-CSharp" value="Normal" />
    <Simple name="ActivePackages" type="ProductConfiguration+Package, Assembly-CSharp" value="BaseGame" />
  </Properties>
</Complex>

With that in mind we can now take a look at the 2 important functions of the PoE game entry definition.

def POESaveName(f):
    """
    Pillars of Eternity saves are actually zip archives. We are interested in
    the saveinfo.xml file inside the archive since that contains the save game
    name we use in-game.

    Ignore autosaves.
    """
    name = basename(f)
    res = name.rpartition(" ")
    if res[0] == "" and res[1] == "":
        # malformed name
        return ""
    if res[2].startswith("autosave_"):
        # ignore autosaves
        return "__IGNORE__"

    # If we get here it's a user save. Get the user save name
    archive = zipfile.ZipFile(f, 'r')
    xmldata = archive.read('saveinfo.xml')
    root = ET.fromstring(xmldata)
    ret_name = ""
    for p in root[0].findall('Simple'):
        if p.get('name') == 'UserSaveName':
            ret_name = p.get('value')
            break
    return ret_name


def POESaveTime(f):
    """
    Pillars of Eternity saves are actually zip archives. We are interested in
    the actual save was performed by the player.
    the saveinfo.xml file inside the archive since that contains the time

    Returns 0 if there is an error or the unix timestamp if it's determined
    """
    archive = zipfile.ZipFile(f, 'r')
    xmldata = archive.read('saveinfo.xml')
    root = ET.fromstring(xmldata)
    ret_ts = 0
    for p in root[0].findall('Simple'):
        if p.get('name') == 'RealTimestamp':
            sdate = p.get('value')
            ret_ts = time.mktime(datetime.datetime.strptime(
                sdate, "%m/%d/%Y %H:%M:%S").timetuple()
            )
            break
    return ret_ts

Those 2 functions get the zip contents, read the xml and return the in-game name of the save and the actual timestamp at which the user made the save. Specifically <Simple name="UserSaveName" type="System.String, mscorlib" value="lef2" /> and <Simple name="RealTimestamp" type="System.DateTime, mscorlib" value="12/13/2015 02:18:17" /> from the example save file we posted above.

Armed with the save name and save time the Tool can make a very safe and informed decision on which save file to sync, which ones to keep and which ones to overwrite.

Running the tool periodically

If you need some extra automation you can make the script run periodically. This very much depends on your Operating System. As I have ArchLinux I will talk about using systemd to accomplish this.

Make sure you have read the corresponding systemd documentation and tutorials. Nice links can be found here and here. Assuming we have a systemd hourly target then having a service that will run the sync each hour is as simple as having this service file:

[Unit]
Description=Runs the Tool that synchronizes game saves with Dropbox every hour
Wants=timer-hourly.timer

[Service]
User=lefteris
Nice=19
IOSchedulingClass=2
IOSchedulingPriority=7
ExecStart=/home/lefteris/.local/bin/steamsaves_sync.py

[Install]
WantedBy=timer-hourly.target

Place this in /etc/systemd/system/saves_sync.service and make sure that the syncing tool is under ~/.local/bin/. The timer-hourly.target should also be placed in the /etc/systemd/system/ directory.

Simply enable by systemctl enable saves_sync.service and enjoy your saves being synced every hour.

Conclusion

At the moment this script can be used for any game as long as you fill out the GameEntry properly. I would not recommend using it with the default functions though for 2 reasons.

  • The name of the save file may be the same for all saves or just have an increasing timestamp identifier in the save file name like Pillars of Eternity does. In this case the tool would always consider each save different.
  • Using the last modification date to determine which save file is dangerous. If you copy/overwite any of the save files around manually or if Dropbox does, then their modification date will change and then the tool will consider them as newer even if they represent an older save. As a result your new saves may be overwritten.

For the above I recommend using the script only with Games for which you have written proper callbacks just like I showed above for PoE.

And finally let me close with the warning that you are using this tool AT YOUR OWN RISK and that it may very well overwrite your saves. Always make a backup of the saves before you start using it.

Hope this tool turns out to be useful for some of you guys. As usual if you have any questions, fire away in the comments section.

Using Emacs yasnippet against repetitive boilerplate code

Yasnippet is an emacs package for snippet expansion into code according to predefined rules. In this post I show how to use it for doing away with common boilerplate code in C.

This blogpost will target the C language but all of its contents can easily be transferred to any other verbose enough language where repetition some times may arise.

I love working with C code, for the simplicity and clarity it gives if written well when compared to the higher-level alternative languages like C++. The other side of the coin for this is that C requires you to write a lot of boilerplate code. You can either go ahead, bite the bullet and write it every time or try to abstract it away in preprocessor macros.

The macros way can go really far if you combine it with preprocessor magic like counting the number of arguments passed to a variadic macro. I have used the macros approach personally in a C project I am working on. It was okay at first but then it became so ugly to look at and messed up with code jumping in any IDE (including Emacs) that I decided to abandon the approach.

Introducing yasnippet

So what is the alternative? I have been experimenting a bit with yasnippet lately and I find it to be a very useful tool for exactly the problem I described above.

Yasnippet works in a quite simple but powerfull manner. It has the definition of a code snippet which is a templated piece of code in a specific language. You can define a keyword after which you can press a particular key, the TAB key by default, and then your keyword will expand into the code snippet following the template rules. It comes with some predefined snippets for various language modes but also allows you to define your own.

You can have yasnippet enabled globally. But if you are like me and only need it in some specific modes you can have something like this in your emacs configuration:

;;; init-yasnippet --- my yasnippet configuration
;;; Commentary:
;;; Code snippets expansion
;;; https://capitaomorte.github.io/yasnippet/
;;;
;;; Code:
(require 'yasnippet)
(yas-reload-all)

(add-hook 'c-mode-common-hook #'yas-minor-mode)
(provide 'init-yasnippet)
;;; init-yasnippet.el ends here

The templated language for yasnippet is really simple. You can describe how you would like the code to look like and insert variables in the form of $n where n is a number that refers to the number of tab completions you want to do when filling in the snippet. At the expansion of the snippet you will be prompted for each of these variables. A variable can also have a default value which is specified like this: ${1:object}.

Using yasnippet for boilerplate code

Yasnippet has some snippets already predefined for C code which you can find under c-mode in the yasnippet, snippets directory but I personally find them lacking. As an example here is a snippet to expand pr into printf taken from the predefined snippets

# -*- mode: snippet -*-
# name: printf
# key: pr
# --
printf("${1:format string}"${2: ,a0,a1});

A simple example for C structs

But since we can create our own snippets let’s get something useful! For those of you who have programmed long in C you must already be quite familiar with the 4 much needed functions to:

  • Allocate a new object of a structure.
  • Initialize a new object of a structure.
  • Destroy an object of a structure.
  • Deinitialize an object of a structure.

Most of the times the code for the above is very repetitive and is the perfect candidate for a snippet. Below you can see a yasnippet that accomplishes exactly that.

# name : C struct Creation and destruction functions
# key : crtdstr
# --

bool ${1:prefix}_${2:name}_init(struct $1_$2 *${3:obj})
{
    return true;
}

struct $1_$2 *$1_$2_create()
{
    struct $1_$2 *ret;
    RF_MALLOC(ret, sizeof(*ret), return NULL);
    if (!$1_$2_init(ret)) {
        free(ret);
        ret = NULL;
    }
    return ret;
}

void $1_$2_deinit(struct $1_$2 *$3)
{

}

void $1_$2_destroy(struct $1_$2 *$3)
{
    $1_$2_deinit($3);
    free($3);
}

In order to use it, just add it under one of the directoris defined in yas-snippet-dirs under a subdirectory called c-mode.

The above snippet has 3 template variables. One for the prefix of the object, one for its name and one for the name of the object variable. These will be filled in with 3 consecutive TAB completions.

The prefix is usually used as a poor man’s way to namespace things in C, so it’s usually the name of the module or library.

An example of the above snippet in action can be seen below

yasnippet1.gif

The demo starts with the keyword crtdstr we defined in the snippet. Then the steps are:

  1. <Tab> -> Expand the snippet and prompt for first variable
  2. Input value for first variable, press <Tab> to continue
  3. Input value for second variable, press <Tab> to continue
  4. Input value for third variable, press <Tab> to finish the snippet

And here is the created C code. RF_MALLOC() is a macro I am using for malloc(), plus error reporting.

bool rf_string_init(struct rf_string *str)
{
    return true;
}

struct rf_string *rf_string_create()
{
    struct rf_string *ret;
    RF_MALLOC(ret, sizeof(*ret), return NULL);
    if (!rf_string_init(ret)) {
        free(ret);
        ret = NULL;
    }
    return ret;
}

void rf_string_deinit(struct rf_string *str)
{

}

void rf_string_destroy(struct rf_string *str)
{
    rf_string_deinit(str);
    free(str);
}

A more complete example for C code

We can also see a more complete version of the above example where we will do the same thing as above but form a full C file and infer the name of the structure from the filename.

# -*- mode: snippet -*-
# name : C full definition of a file with include guards and functions
# key : cfullfile
# --
#ifndef ${1:$(upcase yas-text)}_${2:`(upcase (file-name-nondirectory (file-name-sans-extension (buffer-file-name))))`_H}
#define ${1:$(upcase yas-text)}_$2

struct ${1:prefix}_${3:`(downcase (file-name-nondirectory (file-name-sans-extension (buffer-file-name))))`} {
$0
};

bool $1_$3_init(struct $1_$3 *${4:obj})
{
    return true;
}

struct $1_$3 *$1_$3_create()
{
    struct $1_$3 *ret;
    RF_MALLOC(ret, sizeof(*ret), return NULL);
    if (!$1_$3_init(ret)) {
        free(ret);
        ret = NULL;
    }
    return ret;
}

void $1_$3_deinit(struct $1_$3 *$4)
{

}

void $1_$3_destroy(struct $1_$3 *$4)
{
    $1_$3_deinit($4);
    free($4);
}
#endif /* ${1:$(upcase yas-text)}_$2 */

Add this to a new file under the c-mode snippets. Below you can see an example of the snippet expansion in action ran on a file named tree.c.

yasnippet2.gif

From the filename we infer all the needed information and we simply provide the rf prefix as required. After that we just <Tab> over the other variables confirming their values. We could also change them if we wanted. The resulting C code can be seen below

#ifndef RF_TREE_H
#define RF_TREE_H

struct rf_tree {
    // here fill in the struct
};

bool rf_tree_init(struct rf_tree *t)
{
    return true;
}

struct rf_tree *rf_tree_create()
{
    struct rf_tree *ret;
    RF_MALLOC(ret, sizeof(*ret), return NULL);
    if (!rf_tree_init(ret)) {
        free(ret);
        ret = NULL;
    }
    return ret;
}

void rf_tree_deinit(struct rf_tree *t)
{

}

void rf_tree_destroy(struct rf_tree *t)
{
    rf_tree_deinit(t);
    free(t);
}
#endif /* RF_TREE_H */

How is that achieved? If you take a careful look at the snippet you can see that we have used embedded elisp code in it. For example in order to get the value that should go to the #ifndef guards we used the following.

${1:$(upcase yas-text)}_${2:`(upcase (file-name-nondirectory (file-name-sans-extension (buffer-file-name))))`_H}

Remember that variables are numbered for a reason. This may be in the beginning of the snippet but assumes that $1 is already set and uppercases it. Following that it gets the filename and uppercases that. This, as usual in C code, forms the name of the #ifndef guard.

A bit further down we see the following:

struct ${1:prefix}_${3:`(downcase (file-name-nondirectory (file-name-sans-extension (buffer-file-name))))`} {
$0
};

Here is where $1 is first prompted for insertion and where we acquire the prefix. That in combination with similar code to the #ifndef guards form the name of the structure inferred from the file name.

A final thing to note is the special variable $0 in the body of the structure. This is where the cursor of the user will end up after the snippet expansion is finished.

Conclusion

This was just a simple example of using yasnippet to automate and simplify repetitive boilerplate code. If you put enough time into it you can come up with quite more powerful and empowering snippets. To do that important references would be: