Keystone

Last time out, we talked about how to ‘compile’ a python script into a form that could be stuffed inside a MEL file — and, therefore, could be launched directly from your desktop without the need for a BAT file, launcher app, or long cumbersome command line. I ended up by hinting that there was more to come on the same idea.

The strategy we sketched out for the first version worked by converting a single file to a base64-encoded binary representation that could be safely stored in a MEL string variable without complicated escaping of special characters and so on. That’s a great trick — as long as what you’re trying to accomplish can be done in a single file.

However, a single script is usually just a tiny part of a real-world production pipeline. Real Maya environments are usually much more sprawling — at my work ours is fairly modest and it’s still several hundred files and tens of thousands of lines of code. There are dozens of modules and libraries involved. Jamming all of that into a single string and executing it would be, well… let’s just call it “ambitious.”

Sprawl

How to maintain and version a big sprawling environment like that is a frequent subject for discussion going back many years. All of the different strategies, however, amount to roughtly the same basic ideas:

1. Make sure that everything you depend on is available to all your users in the same way — there’s no worse kind of bug-hunting than the dreaded “It worked on my machine…!” variety.
2. Getting the right bits to your users also means making sure they don’t have the wrong ones. An out of date library or a leftover .pyc file can be even worse than a missing file, since incorrect code can create subtler and more difficult to catch problems.
3. You’ll probably have to support multiple configurations, so make it easy to pick a toolkit at startup time instead of just pointing at one file location.

What I’ve been doing for many years is to maintain my entire environment as a single zip file. Since Python allows you to put zip files on your Python path and will find modules inside zips automatically this doesn’t require any special code — yet it does ensure that there is one and only one copy of the toolkit running in any given scenario. We use a launcher app that puts the zip on the Python path and tells Maya to run it at startup, so you can always run a different toolkit by simply bypassing the launcher, or by using a different launch script that points at a different zip file.

This zip-based system has worked like a charm for the better part of a decade, and I’ve never regretted adopting it. The only thing that has always frustrated me about it is the fact that you still need to find a way to tell Maya about that zip file. userSetup.py works fine for this, but many users want that userSetup for their own customization purposes and you don’t want people to have to hand edit that file to switch between projects. Custom launcher scripts (BAT files or other ways of formatting a command line argument to Maya) support project switching well — but they are separate software projects int their own right with their own distribution and versioning hassles. Plus, they add an extra support burden when you’re supporting people at remote studios who don’t necessarily participate in your entire studio ecosystem.

A distribution method that was simple and robust and untethered to external depenendencies has been a kind of obsession of mine for a long time now. Thanks to the Python-inside-MEL idea from our last outing, I think it’s finally within reach.

Keystone…

I’ve got a working first version of the concept bundled up into a short, single-file script that you can get from this GitHub — a project I’ve dubbed Keystone because, ya know, lame “pipelne” puns.

Keystone is a simple command line script that you point at a folder which contains your Python toolkit. It will compile all of the Python files to .pyc for speed and then zip them up into a single archive. Then, it will generate a startup script to find an launch the zipped-up toolkit. Finally it will compile that startup script and the binary data from the zip file into a double-clickable MEL file — a single-source distribution of the whole shebang which doesn’t need any extra infrastructure.

The strategy is actually very simple. Just as the last post showed how to stuff Python code into a binary blob inside the MEL file, Keystone does the same thing for a zipped-up Python tookit. The final result is a MEL file which makes sure the user has a zipped up copy of the Python environment, puts that zip on the user’s path, and launches it inside of Maya when the user double-clicks on the MEL. This combines all the ease-of-use advantages of having a MEL launcher instead of a separately maintained BAT file or a complex command line with the ability to distribute a big, complex toolset in a single bundle. The self-contained nature of the MEL-zip hybride makes it trivially easy to maintain multiple versions side-by-side. The launcher is the environment and vice-versa.

..the pipeline

Here’s a basic flowchart of what happens in a Keystone pipeline ( environmental destruction, petro-dollars, and giant protest puppets omitted for clarity):

The only ‘complexity’ — it barely deserves the name — is the way in which the zip file data is jammed into the MEL file. This amounts to simply generating a zip file, reading it in as binary bytes, and appended those bytes to the end of the MEL file after a comment mark so that Maya doesn’t try to read them. The offset where the comment begins is encoded into the script so that it’s eay to grab the relevant bits and put them back on the user’s hard drive as ordinary zip files.

The Keystone compiler automatically generates the code to add the zip file onto the Python path (this script is encoded using the base64 method we outlined in the previous post). When the user runs the MEL file it hands execution over to this startup code. The startup script checks to make see if there’s already a zip that matches the data stuffed into the MEL file. If there’s no script (or if there is a zip which is older than the MEL file) the startup code will extract the hidden binary data from the MEL and save it (currently it’s getting stashed in the Maya user directory, though it would be easy to change that).

The MEL file’s startup code adds the zip to the Python path, launches Maya with the new zip as the first item on the Python path, and then executes any startup code you’ve included in the zip file — you can put any initialization you want into the zip as a __main__.py and it will get fired off in the same way that an ordinary Maya session executes userSetup.py. The nice thing is that the code, however complex it wants to be, is part of the overall Python codebase and not part of the small bit of startup code. Bitter experience has taught me never to let the first bit of a startup script to be more complex than “put this stuff on the path and go” — it’s much easier to maintain and version and tweak production code than the first few lines of the bootstrapper which are always running in a very bare-bones context.

The upshot of all this is, hopefully, a complete Maya Python environment in a single file — as many modules and scripts as you need, but without the need for tools in other languages to get it plugged into a given Maya session. It’s MEL and Python, so it’s cross-platform (no need to translate .BATS into shell scripts for OSX or Linux). You can distribute this file through Perforce or a download link or simply by emailing it to one contractor; no matter how that file lands on their desk it should give them a complete environment. If you need to give them an update you simply replace that one file and the rest follows naturally.

Go with the flow

I’ve been fiddling around with variation on this idea for a long time. The primary reason I find it compelling that productions nowadays are distributed around the globe — I know, for example, that I met far less than half of the artists and animators who contributed to State of Decay 2. A toolkit that’s easy to drop in as a single file, and which doesn’t depend on a complex web of other tools like launcher apps or perforce automation is very appealing. It’s also nice to have a way to let outsource artists leave the party with as little friction as possible: it’s far better to simply drag one file to the trash than to remove an entire ecosystem.

Of course, all tech involves tradeoffs and I’m still trying to learn what the tradeoffs in this approach will be. The most obvious issue is that there’s the data in the zip portion of the package ends up getting duplicated. My work toolkit zips up about 25 mb, so the overhead is far from trememndous, but it would be nicer if there were a good way around that. A more annoying issue is the need to extract any binary tools (such as the P4Python module) from the zip file — Python currently won’t read compiled extensions out of a zip which is irritating though easy to get around by extracting binaries to a hidden folder.

So — I’d be curious to hear what other people think and whether this answers a need. As always, pull requests and issue on the Github. Until then, have fun checking it out. Help us make MEL great again!