tweetvaultHQ blog

Another much-requested feature from recent feedback I’ve received has been to address the delay between new users signing up and the archive process running to grab their data. Historically, new users had to wait until the next archive run (at worst case, up to an hour) before their tweets and favourites would appear in the main view.

I’m pleased to report that this has now been addressed – a new user signing up causes a background job to be started to grab their data, with live progress being reported to the user in the main view.

The same thing happens for first-time subscribers. I figured new subscribers shouldn’t have to wait to see the rest of their data in the app (trial mode limits a user’s view to 100 tweets and 25 favourites – subscribing lifts these limits) so they will also get to enjoy a live progress report as their additional data is retrieved after subscribing.

Hopefully this means more new users will stick around to explore tweetvaultHQ properly. Yay!

Hey y’all. Some recent feedback I got after posting tweetvaultHQ to Hacker News included a suggestion that I enhance the tweet view such that any tweets of yours that are replies to tweets from another user can now thread in those other tweets to see more of the conversation.

A tweet that replies to another user already has an “in reply to …” link attached to it, which used to take you to that tweet on Twitter itself. Now, however, clicking these links will load the other user’s tweet neatly below yours in the timeline.

If that other user’s tweet is itself a reply, it will include an “in reply to …” link too, and you can click that to load the tweet it was replying to, and so on and so forth until the original non-reply tweet is reached.

For bonus points, try holding Shift as you click an “in reply to …” link, and tweetvaultHQ will load the whole conversation. Clever eh?

More new functionality coming soon, with any luck .

Short notice I know, but if you’re coming to Devnest tonight at the Guardian offices in London you can hear me give tweetvaultHQ a proper introduction (building on the micro-demo I did last time before the app was launched), see a full demo and share a few thoughts on the experience of launching my first public web app. Hope to see a few of you there! I’ll be the one expelling hot air at the front.

Update: it went well! Thanks to everyone who turned up. You can find my slides here.

I started using DestroyTwitter‘s default prefix for retweets a while ago, which is a greater-than symbol (>) rather than “RT“. For some reason or other I tried searching in tweetvaultHQ for the > character yesterday and found that it wasn’t returning any results. I had a look in the database and realised that Twitter encodes this and the < character to their proper HTML entities, > and <. Searching for either of these strings returns results as expected.

So I added a line of code to encode entities in search queries, so searching for > and < would match their encoded counterparts in the database. So far, so good. I was about to tweet about this fix when I realised I should try searching for “&” as well as this should (in proper usage) also be encoded – to &. Annoyingly, this wasn’t returning any results in the app. I checked the database and found to my surprise that Twitter doesn’t encode all HTML entities, only < and >. This meant that my search for “&” (encoded as & before being sent to the database) wasn’t matching anything.

A quick search showed that this is a known issue, and there hasn’t yet been a non-hacky fix for it. The reason Twitter does this half-assed encoding is apparently to do with browsers behaving oddly when encountering & entities in JSON responses. If you retrieve API results as XML, all entities are encoded correctly. But in the JSON format, Twitter only encodes < and >. All other entities are left alone.

This is a tricky issue to solve as there’s not much I can do at the query level given the inconsistent state of the tweets already in the database. To fix this I would have to run a conversion routine against every record in the database, and then apply that same conversion to all new tweets as they come in. I haven’t ruled that out, but I’ll need to give it some consideration. That would involve decoding entities in the incoming tweet (to produce < and >), and then re-encoding them before storing to produce the proper entities for all special characters including “&“. Those characters in search queries would then match the encoded ones in the database.

So for the time being any searches for tweets or favourites containing “&” won’t return correct results. I’ll weigh up the implications of performing the database conversion I described, and if it looks OK I’ll put a fix out soon. Hopefully this post is useful to anyone else having the same problem!

So we’re 48 hours in to the tweetvaultHQ beta and nothing has caught fire yet. Big thanks to everyone who’s signed up so far, and I hope you’re finding the app useful and/or interesting!

There have been a couple of boogs which are now fixed, but I thought I’d share my findings during the squashing process in case anyone’s particularly curious.

The first bug was a pretty straightforward case of failing to (re)initialise a variable during a loop – the symptom was that the archiver was saving only a partial set of tweets/favourites for some users, and it turned out that I wasn’t properly resetting the ‘trial’ flag while looping over the users, so some users’ trial limitations (unsubscribed users can only see 100 tweets and 25 favourites) were being carried over to other, subscribed users. That’s been sorted now, red faces all round.

The second bug took a bit more digging to diagnose – some users were finding that once activated, they couldn’t log in, getting the message “invalid session” despite repeated attempts. It took me ages to figure out where the problem lay as there didn’t seem to be any pattern to the users having problems, but eventually I traced it to one particular query – the one that’s supposed to create a profile record for a user when they first log in. This query was failing to execute but the session cookie was still being set, which meant that when trying to log in the system would see the cookie but be unable to tie it to a known user, hence the session error. The problem field was the utc_offset, which is part of the user’s timezone setting. I’d looked at my own Twitter profile object when setting up the database schema and assumed that the utc_offset defaulted to 0, when it fact it defaults to null. Assuming it would always be an integer, I didn’t quote the value when passing it to MySQL (I did escape it, before you ask) – and this caused a bit of a wobbly when some users started logging in with empty values. It didn’t help that the failing of this query wasn’t noted in the front-end, or logged. That’ll learn me.

Anyway, all sorted out now. If you had any trouble logging in since you signed up, have another crack now and all should be well. If you’re still having trouble, get in touch.

And it’s done! tweetvaultHQ is live at last. You can (and should) go and register right now.

When you’re signed up, head to the subscription page and enter the code THQBETA for a free month’s subscription. That should give you a chance to try out the full-featured app for a bit.

I’m reasonably confident it works end-to-end, but it’s entirely possible there could be a few bugfoots lurking behind the scenes. If you do come across something, the best place for it is the bug tracker right here on this blog. You can @-me or the official @tweetvaultHQ account if you want to talk about it, but filing a bug is the easiest way to get something fixed.

There are plenty of features and enhancements in the pipeline – we’re only just getting started over here, so watch this space for future announcements.

In the meantime, it would be the most awesome thing ever if you could tell your friends about tweetvaultHQ! They can all use the code above if they like. Neat, eh?

Enjoy!

I’m currently dashing to finish the last few features on tweetvaultHQ and fix a few issues, and I’m almost there. One noteworthy feature that supercedes info in the previous post is that I’ve now implemented promocodes, so if you’re participating in the beta you can get a month’s free subscription as soon as you sign up.

Once the site’s live and you’re registered and activated, just head over to the Subscription page, scroll down to the promocodes section and enter the code THQBETA to get a month’s subscription. Bam. Just like that.

I’m posting this now in case you miss it in the release announcement (which will be soon, honest!). I’ve got two content pages left to write, one feature page (account administration) and one blocker bug to fix (OK, it’s not really a blocker, but Twitter won’t like me if I don’t fix it). This weekend is looking busy already, but I’ll see what I can do, OK?

Keep watching this space.

Welcome to the tweetvaultHQ dev blog, which I realised I should probably start tonight, ha. I’m @BigglesZX, and tweetvaultHQ is a hosted Twitter archiving service that’s about to launch into public beta. I’ll be posting beta-related news and stuff here.

I was lucky enough to be able to give tweetvaultHQ its public debut tonight in a short presentation at devnest London, and the response was amazing and encouraging. For me it was the culmination of several months of snatching a few hours here and there on evenings and weekends to poke away at a product that I wasn’t convinced would ever see the light of day, but now it seems it’s almost ready! And from the response I got, it seems a few people think it might be useful.

Big thanks to everyone who came to devnest tonight (and to @nuxnix for organising it), and extra thanks to those of you who came to speak to me after the presentation or @-replied me on Twitter about the app. Here’s how the beta is going to work.

Right now the app is fully functional, I just need to add a few bits of polish before I’m ready to hit the big button. I’ll do that in the next few days and then it’ll be all systems go. You’ll be able to sign up on the site using your Twitter account and access the site in trial mode, where only a small number of tweets/favourites will be pulled in. If you @-reply me to let me know you’ve signed up, I’ll add a subscription to your account to let you test the full range of tweetvaultHQ features for freesies.

If you discover anything that you think isn’t working right (or could work better) while you’re testing the app, you can let me know via Twitter or possibly via a more organised bug tracking system, the details of which I’ll confirm in a future post.

You can also @-reply the official @tweetvaultHQ account, where I’ll be posting shorter updates in line with this blog.

I’ll be responding to feedback with bug fixes and updates, and at some point will declare the beta period over. Anyone can sign up during the beta, and if they so choose they can contact me for a free subscription for testing. When the beta closes, anyone who helped out will get a further subscription bump by way of a thank-you.

Thanks again for the interest so far and watch this space for further announcements. I’m really excited about launching this product and glad to have you all along for the ride :].