my name is mwd

Just thought I should put this on the Internet somewhere: if you’re using python-cmemcache, then you might want to move to python-memcached (or other alternative) if you plan on moving from just one memcached server to multiple ones.

We had an issue last week at Camvine when we moved from running our main django app on one server to multiple servers. We use memcached not just for caching, but also for storing transient state that we don’t mind if it’s lost occasionally. To access memcached from django we’d been using python-cmemcache, as it’s a bit faster than the pure python python-memcached.

When we moved to a pool of servers we did the standard thing of sharing the memcached across all machines as one large memcache. This is fine, except we found that randomly we’d find inconsistent state between the machines. It turns out, after 24 hours of chasing, that python-cmemcache was occasionally losing responses from remote memcached instances. Looking at the logs on the memcached instance you could see it replying when asked, but from the apache logs on the machine asking, you could see the reply from cmemcached was “no data”. Switching python-cmemcache for python-memcached cleared the issue up immediately.

We’ve ran python-cmemcache for ages, without issue. It was only as we moved to more servers we hit this issue. It seems to be something specific to it running in apache, as I couldn’t recreate it from the command line. The python-cmemcache page itself says it’s time to move on to other libraries, as it’s no longer supported, but I imagine there are many out there who have been using it for ages and haven’t thought to look. Hopefully this post can save someone a day of their life :)