In one of my projects I need to create and use several fairly large, non-changing data structures — hash tables of string values, array’s of objects, nested object definitions several layers deep, etc. For example, from my battlefield 2 stats website I have several very large arrays that describe every award you can get, a sample:

    '9' => array(
        'short'     => 'MgySgt', 
        'long'      => 'Master Gunnery Sergeant', 
        'unlock'    => true, 
        'notes'     => '', 
        'thisrank'  => '9',
        'nextrank'  => '12',
        'requires'  => array(
            'awards'  => '', 
            'rank'    => '7', 
            'score'   => '50000', 
            'round'   => ''
            ) 
        ),

So the question arises, how do you store this data, load it, and use it?

First Step: Research

Of course, I found a page on StackOverflow about php storage. Perusing the article, you’ll see most people break down the caching theory into three options:

1. json_encode the data into a file, load it, json_decode
2. serialize the data into a file, load it, unserialize
3. var_export the data into a file, require() it

One responder (who should have more votes, if it was up to me) went pretty far and did a serious amount of benchmarking for the php array storage issue.

I won’t go into a lot of details on each method as the linked articles cover them each pretty well, but I want to open two other options and redo the benchmarks because something smelled fishy when JSON has been coming out the winner…

The Purported Winner, JSON

JSON certainly has some advantages: the cached file sizes are smaller (especially when you saving a numericly indexed array), it is usually faster to write, and if you have a highly JS oriented site architecture being able to make HTTP requests for your JSON data files can be quite useful.

What irked me is that, for my purposes, none of those things really truly matter. File size isn’t a real concern because my largest cached file weighs in at about 50KB. I won’t be distributing the data to JS applications, and writes only happens once.

So, what does matter? Read speed. And with that in mind, what I found out is that JSON can’t hold a candle to the other two methods.

Cons of var_export

I’d first like to rebut the cons list that var_export was given on Procurious.nl:

1. Needs PHP wrapper code.
2. Can not encode Objects of classes missing the __set_state method.
3. When using an opcode cache your cache file will be stored in the opcode cache. If you do not need a persistant cache this is useless, most opcode caches support storing values in the shared memory. If you don’t mind storing the cache in memory, use the shared memory without writing the cache to disk first.
4. Another disadvantage is that your stored file has to be valid PHP. If it contains a parse error (which could happen when your script crashes while writing the cache) your application will not work anymore.

1. Wrapper code exists for your other caching methods as well, it’s just placed inside the loading function instead of in the cached file itself.
2. If you encoding objects created with classes, JSON is going to fail worse here. The more likely scenario, however, is that you are just using “dumb” stdClass objects, in which case it is super easy to overcome the stdClass::__set_state() blunder. str_replace('stdClass::__set_state(', '(object)', var_export($data,true)) done and fixed.
3. The point of pushing the cache data into a file is for longevity. Your APC cache will not survive a server restart, a cached file will. Additionally, I have too much data to reliably cache in memory¹; using the opcache provides the built in “most used gets kept around” optimization.
4. This point could be made about ANY of these methods. A broken serialized file or cut-in-half JSON object is just as likely to cause errors.

Don’t Discount The Opcode Cache

I find it odd that the opcode cache was so readily dismissed; it’s a surefire way to speed up ANY of these operations. To that point, let’s introduce two other caching methods that an that can make unserialize and json_decode even faster. How? Just wrap the data up inside a PHP file!

json_data_cache.php:
<?php return json_decode('{"your": "json", "goes": "here"}');

The same treatement can be applied to unserialize. By introducing this boiler plate routine into your serialized/json_encode’ed data you can take advantage of the opcode cache. When you’ll see later is that even if you decide to go with JSON, doing this will lead to a significant speed up.

Test Script

Here you will find the test script for benchmarking php caching with json, serialize, and var_export. A quick overview of how this is intended to be used:

1. Install to a webserver (apache, lighttpd, etc) that has PHP with APC installed.
2. Call script from browser as test.php?m=gen
   
   This step will create all the cached data files. These files should be the same as the data files I (or any one else) would use to test as long as the $setsize value is the same.
3. Call the sript again as test.php?m=test
   
   This will run the load & parse tests on each cache file created.

The data generated by this script is fairly comprehesive. It will test arrays of strings, ints, and objects. The arrays are indexed by numeric and string keys. Additionally, it will test two object collections. In all cases, the script creates a Large, Medium, and Small set of data based on the $setsize value (100% size, 50% and 10%).

When you are running the test, I highly recommend you run with the same $setsize and $itterations value with APC off, APC enabled (apc.stat=1), and finally APC enabled(apc.stat=0) restarting your webserver between each execution. apc.stat instructs APC to check the file for modifications to see if it should be recompiled before using the opcache variant. Turning this off eeks a bit of performance out of APC at the cost of needing to manually clear the APC cache if you update files. The testing, however, shows that you may or may not think this is valuable — the performance difference is very, very minimal.

Also, for the APC enabled test runs, perform then twice to ensure that your APC cache is primed.

Results, Man!

I ran this test on a Fedora 15 virtual box on top of a Win7 install on a Corei7 machine with the settings $setsize=2500 and $itterations=200. The webserver was Apache/2.2.21 with PHP 5.3.8 with APC 3.1.9.

Here’s a graph comparing the best runs of each decoding situation:

And an ODS (Open Office Calc) file of the raw results and a few other graphs.

What you are looking at here is the decoding of each type of data structure by method, as compared to the fastest method, which is var_export with APC.

Take aways:

1. var_export with APC caching (apc.stat on or off) is about 270% faster than anything else
2. If you want to keep your data in JSON format, keep it in the json.php data structure — the opcode cache makes this method an average of 150% faster than loading JSON from a file and even if you don’t have an opcache you’ll not lose any performance anyway
3. The jump in performance between var_export with and without opcode cacheing is as dramatic as I had expected
4. Important You must know your deployment environment — var_export without opcode caching is by far and away the very slowest method

If you have any suggestions, feedback, or wish to contest the results, feel free to leave a comment or submit a patch.

¹ If you have a small enough data cache, you should be reading up on Redis, and this whole discussion becomes pointless very quickly.

Had a nice long post about this, got eaten by expired session. Engage tl;dr

• Love Battlelog
• Battlelog better not stay stuck behind the login gate
• Forced to load game and spawn before you can change options? Yeah, dumb. People scream for options any time — no, better would be to move options into Battlelog.
• Gameplay and graphics are killer and perform quite well even on my starting-to-get-2-years-old hardware.
• The closely interwoven Blaze, ESN, and Origin system dependencies scream “Multiple single-points-of-failure!!” which does not bode well for initial stability nor game longevity.
• For the love of all that is holy, in-game text and squad systems better be about 1000% better by release.
• Overall: Probably taking some time off at release.

Who doesn’t like a little weekend project, right?

A combination of things has recently pushed me towards moving away from Wordpress — primarily the ever present threat of a high profile suite of software requiring constant upgrades to keep safe. Because of how rarely a I post (sometimes less often than Wordpress releases updates) I run a fairly high risk of having security holes in active code on my webserver. Between that and the fact that I simply don’t require such up-to-the-refresh dynamic flexibility (typically I only make layout changes once every 2 years or so) it just doesn’t make sense to use something so dynamic, thus I am ready to stop playing the upgrade game¹.

Being that I keep exposed to new technologies, the Jekyll project has come across my path a few times. Between that and wanting to play around with the Phar format, I decided to have a go at the same concept written in PHP.

First, I looked around for some alternatives, and I did find one: Phrozn. However, while Phrozn is geared towards static site generation, it lacks some functionality unique to blogs: namely “tags” and chronological pagination.

So I figured, if the world could use one more PHP MVC framework², then why not another static site generation script?

Thus, Alkemy was born.

Alkemy, like Jekyll, will take a list of posts, parse them up (Textile, Markdown, HTML, or just plaintext), smash them through some template files, and smoosh out a website. The website is fully generated and completely static. You serve nothing but set-in-stone HTML files and other non-code-interpreted content. This has 2 really big advantages:

1. Damn if it isn’t fast — Static content requires no programming overhead; served direct from disk (or memory) to network pipe.
2. Secure as all getout — there’s no php to hack, no python to work around, no perl to xss inject, no java stack traces, nothing interpreted but GET requests for static files.

Of course, one must eat his own dog food, so this blog is now managed by Alkemy. I have a local copy, versioning using Mercurial, where I get things written and tested. After I’m happy with my latest post, I commit locally, push to the server, and have the server run an update. My webserver has the repo’s “site” folder setup as the docroot and “TADA!” the site is updated.

You can find Alkemy at http://alkemy.info/ along with documentation and examples.

¹ I have often wished Wordpress offered a LTS version of their software for this sole reason.

² Not really.

---

UPDATE: Turns out static blogging like this kind of sucks. The overhead in producing the content, editing it in the right area, re-running the generation, and getting it synced up is too much of a burden. Compared to something like “Login, type, post” or even easier, bookmarklets (ala tumblr/posterous), it’s just no comparison. As such, this blog never even got off the ground with that code (not externally anyway) but instead is running some custom tumblresque software. Alkemy will stay released, in any case.