Archiving Twitter on Your Own Server

If you’re a regular ProfHacker reader, you already know that we are obsessed with backup. You probably even know that we begin every post about backup by telling you that we are obsessed with backup.

It’s true: we are obsessed with backup. And it’s true: we keep telling you that we’re obsessed. You might think we’re being a nuisance by relentlessly revisiting the topic of backing up your data, but we like to think of ourselves as a contagion, spreading the Got-To-Backup-Now Virus (G2BNV). In the end, we hope every one of our readers is infected.

Most recently, Brian took a comprehensive look at backing up your social network, primarily Twitter and Facebook. ProfHacker will have a future post looking at Facebook’s new built-in export/backup feature, but today, I want to look more closely at backing up your Twitter account. More specifically, I want to highlight several alternatives to the backup services Brian mentions. The solutions here differ significantly from other backup services because they archive your Twitter activity on your own server.

Why Archive Twitter?

But why would you want to archive your Twitter stream? Maybe you don’t. It probably depends on how often you tweet and what you usually tweet about. I certainly didn’t think it was important for my first several hundred tweets or so. But then I began using Twitter to try out ideas that could possibly work their way into my teaching, my research, or even into ProfHacker. It turns out that Twitter is often where the first draft of my serious thinking occurs. And I didn’t want to those fragments of my thinking to disappear—which is effectively what happens once you have more than 3,200 tweets.

Twitter’s public search record only goes back 3,200 tweets for each account. There is no service that can recover tweets beyond your more recent 3,200 tweets. All of your tweets are permanently stored, bound as they are for the Library of Congress, but unless you know the exact ID number of a tweet earlier than 3,200 ago, you cannot readily access it.

So, if you’re approaching 3,000 tweets (and it doesn’t take nearly as long as you might think), you’ll want to begin archiving your Twitter activity, so the earliest ones (which are often quite quaint) don’t disappear.

Why Archive on Your Own Server?

So if you want to archive, why not stick with third-party services like BackupMyTweets or TwapperKeeper? These services are free and reliable, and there’s no reason why you shouldn’t be satisfied with them. But what I like about backing up material onto my own server is that it’s mine and I have complete control over it. After nearly 10,000 tweets, I’ve invested some serious intellectual labor into these bullet-bursts of writing, and those ideas are mine, and I want to keep them safe, on my own domain.

Of course, you’ll need your own server for any of the following three Twitter archiving solutions. Luckily, ProfHacker can get you started with your own domain.

Solution 1: TwapperKeeper on Your Own Server

TwapperKeeper is one of the better known Twitter archiving services, allowing you to archive your own Twitter stream, hashtags, and even random keywords. So when TwapperKeeper announced an open version of TwapperKeeper that you could run on your own server—called Your TwapperKeeper—it was exciting news. However, I was disappointed to find that I could not get Your TwapperKeeper running on my server. I’m reasonably adept at installing software on a LAMP (Linux-Apache-MySql-PHP) server, but after a great deal of frustration I finally gave up on TwapperKeeper. If you’ve had success running your own version on TwapperKeeper on a server, let us know in the comments!

Solution 2: Archiving Twitter through WordPress

If you’ve already got WordPress running on a server, you can turn a WordPress blog into a platform for archiving Twitter. Douglas Bowman (who just happens to be Twitter’s Creative Director) has written up easy-to-follow directions that make use of the Twitter Tools WordPress plugin. In no time at all I got this solution working on my own server, where my most recent 4,000 or so tweets are easily accessible and even better, searchable. The tweets appear in a blog all of their own, so they don’t clutter up my regular blog, something readers only interested in my longer writing might appreciate.

Solution 3: Archiving Twitter through Tweet Nest

I like the WordPress solution, but it has one major flaw: the archive does not link back to the original tweet on Twitter. The tweets are duplicated on my server, but there’s no easy way to get back to the source.

Enter Tweet Nest, a stand-alone open source PHP application coded primarily by Andy Graulund. Tweet Nest requires a bit more backend tinkering on your server and in MySQL, but Graulund’s instructions are clear, and again, in no time at all, I had a workable archive on my server.

Unlike the above WordPress solution, Tweet Nest does indeed link back to the original tweet on Twitter, as seen in this post of mine, archived by Tweet Nest:

Clicking the time-date stamp leads to the original post on Twitter. Not only that, but by clicking the “in reply” link, you end up at the tweet that began the conversation.  Tweet Nest also provides a few data visualizations, such as a monthly calendar, showing the total number of daily tweets (and public replies).

What about You?

What about you? Are you likely to archive your Twitter account? Are you likely to try doing so on your own server? What other solutions have you found? (Hint: I have one more myself, which I’ll be sharing on ProfHacker very soon!)

[I'm Just Looking for Something image courtesy of Flickr user tpholland / Creative Commons Licensed]