We’re really sorry for the ongoing problems with our services.

The short story is that our datacentre had a disk problem, and the servers had to be restored from backup – but it looks like the backup was corrupted too.

We’re digging in to older backups, to see if we can fix the holes in our data.

As you can imagine, we have a massive amount of data and the corruption was widespread and almost random, so it’s a bit hard to spot all the files that are missing. To help us out, please report any problems to jim@snowcrashdigital.com so I can chase them down. Please include as much information about what’s happening with your system as possible – including specifics of album names and avatar names involved. Apologies in advance if I don’t reply to your mail quickly…I’m expecting a bucketload and my priority will be chasing down those pesky files to get things running properly again.

Once again, we’re sorry about this, we’re at the mercy of a ridiculously unlucky set of coincidences, but we’re working hard to get it fixed. Just in case things go wrong with our blog during this time, please follow us on twitter – @hermesannounce.

Thanks for your patience.

[update 9:55AM PDT April 3, 2010] After pulling a seriously late night (so late it was early), we seem to have restored most of our files from older backups, mixed in with newer ones. It was painstaking work, and it’s very possible we missed some, so please check your stuff and send reports of problems to jim@snowcrashdigital.com or by IM (not notecard please) to Hippyjim Starbrook in-world.

About jim

As our technical director Jim brings decades of experience and knowledge in security, stability and scalability to the team, earning his geeky spurs as an application developer for financial institutions. Jim knows his way around lots of server and browser based technologies and can make a website do pretty much anything you’d like (except persuade a teenager to tidy his room).