Stuff that occurs to me

All of my 'how to' posts are tagged here. The most popular posts are about blocking and private accounts on Twitter, also the science communication jobs list. None of the science or medical information I might post to this blog should be taken as medical advice (I'm not medically trained).

Think of this blog as a sort of nursery for my half-baked ideas hence 'stuff that occurs to me'.

Contact: @JoBrodie Email: jo DOT brodie AT gmail DOT com

Science in London: The 2018/19 scientific society talks in London blog post

Saturday, 5 March 2011

Trapping tweets for archiving eg at conferences, or curiosity

I've divided this post into WHAT and HOW. If you're new to archiving tweets or can't think why anyone would bother please read the WHAT first. Hopefully the HOW is self-explanatory ;)

One of the most popular search terms that bring people to my blog relates to "finding your old tweets", a topic I've touched on previously in these posts:
With the news that a tweet archiving tool called Twapperkeeper is stopping its download and export function from the 20th March I thought I'd look again at some of the ways of storing tweets in the longer term.

Some examples of why you might want to do this include
  • you're following what everyone's saying at a conference, by following a hashtag (everyone puts a codeword into their tweets with a # in front of them, the # lets archiving tools pick them out from all the other tweets and present them to you on one page) and you want to make sure you don't miss any tweets, or read them later
  • you want to capture @ mentions - eg during a campaign you might want to see how often your company is mentioned
  • you want to capture other mentions, eg retweets
  • something else I've not thought of
As far as I know, Twitter doesn't treat any of these concepts any differently - pretty much everything on Twitter can be captured by an RSS feed (scroll to the bottom of any page, past all the 'trending topics' and you'll likely see an orange RSS icon - Twitter doesn't care if the RSS feed is made up of tweets from someone's profile or the results from a search.

1 Tools
a) **NEW** SearchHash has been around since October 2010 but I discovered it on 19 May 2011 - it's extremely good and lets you capture and export significant numbers of hashtagged tweets. It's been created by @lesteph and Dave Briggs. Steph's post introducing it is here and the tool is here - try it out with a hashtag.

b) **NEW** Storify has been around for quite a while but I only heard of it a month ago. Journalist / blogger types were using it to create stories in a very nice way, drawing together several strands from various bits of social media, including Twitter. When I realised that I understood its potential for storing archived tweets - it works very well indeed and is a good interface with a very attractive output, but it's hard work as you have to save each tweet manually. I've previously investigated it here.

c) What The Hashtag has now turned into What the Trend API which is technically beyond me but if you know what you're doing with coding / JSON / stuff like that, give it a go.

What The Hashtag or WTHashtag is a free tool (registration required) that lets you create an archive based on a hashtagged keyword.

During Paul Chamber's trial for sending a threatening message via Twitter people used the hashtag #twitterjoketrial to indicate their lack of warmth for the case coming to court. At the time someone set up WTHashtag to collect all of these together to see how many were generated.

If you
look at the page several months after it was used for this purpose you'll see some other tweets that don't appear to be particularly relevant to this case (a couple of tweets are using it to refer to Julian Assange).

To see older tweets you need to scroll below the bar chart window and click on the 'View transcript' tab and choose the relevant date (US style). Here are the #twitterjoketrial tweets from
15 Sep to 25 Sep (note, all days might not appear if the hashtag was invented on, say, the 19th! You'll note that evidently it's missing a tweet or two as the first one listed, at time of writing, is a retweet - where's the original tweet?).
d) Twapperkeeper lets you maintain two free archives, but any more than that and you'll either need to pay ($5-$15 per month depending on whether or not you want a few or up to 25 archives). You can archive a hashtagged event but you can't use the exporting facility to do much with them. I expect you can copy and paste the tweets into some other programme (eg Word) and do something with them that way.

Currently lets you create an archive of hashtags, your own tweets, tweets mentioning you etc. however this facility's usefulness is going to come to an unfortunate end on 20 March 2011 as no-one will be able to export /download them any more. This is because Twapperkeeper's been asked by Twitter to remove this function so that they fall in line with Twitter's plans for syndication of content. This could well mean that other archiving tools come up against problems. Twapperkeeper have made an open source version of their tool which I've not investigated but the implcation seems to be that this will be a viable alternative.

Incidentally Twapperkeeper has been working closely with JISC and UKOLN - I know that academics working in the area of HCI and Search work on huge Twitter datasets but I would also imagine that academics who communicate using Twitter might want to maintain access to this information too.

One thing to bear in mind is retrospectiveness of the tweets you want to capture. Twapperkeeper has, as far as I can tell, always been able to crawl backwards and access earlier tweets but I'm not sure that WTHashtag can do so - therefore this sort of archive is best set up in advance and monitored, with the content periodically 'tapped off' as a PDF etc. for later reading.

Email alerts for your @mention - see below.

2 Tasks (what you want to save)
Finding @mentions retrospectively
I can't think of an easy way to access old @ mentions in an ordered fashion. You can certainly find them via Google, going back to early 2010 but the interface is a bit clunky.

I used this system very succesfully to capture a series of tweets sent in response to a question that Ben Goldacre asked about nerdy days out (it would have been easier to collect if a hashtag had been applied by everyone to their tweet but I didn't think to ask at the time).

This system also lets you set up an email alert for your @mention - I've not tried this and don't know if it will present you with a neat continuous list of your mentions. (Scroll to the end of the page and you'll see the option availalbe).

Example: some tweets sent to or by me in early 2011
By the way here's what the search-results string looks like - someone nerdier than me might be able to URL hack it to get some better results...,mbl_hs:1293840000,mbl_he:1325375999

Setting up an @mention archive prospectively
Ah the benefit of hindsight eh. Go to Twitter ( type in your @mention of interest, eg @jobrodie, press enter and you'll see a list of tweets with your name. It's probably remarkably similar to what appears in your own @mention page but the difference is that this one has an orange RSS icon on it - click it to view, or right click/copy shortcut to save the URL.
What you do with your RSS feed is up to you - you can use any RSS reader (eg Google reader) to read it. Because feeds are regularly updated you'll probably have to keep on top of saving them somewhere (ie, don't just rely on visiting that RSS page - send it somewhere for safekeeping). I use FriendFeed for my tweets.

You might notice that, annoyingly, none of the tweets say who sent them! If you ignore the purple tweets (just duplicates of the blue ones, a quirk of RSS I think) and hover over the blue link you'll be able to see, in your status bar, who sent that tweet.

This search may well bring up some of your tweets that have been retweeted (which you wouldn't ordinarily see in your @mention window unless the retweet was sent manually).

But for belts and braces you might as well see what the search brings up without your @ in front, so here's a search just for jobrodie

{Long post - taking a temporary breather at midnight! Bound to remember some more options after a good night's sleep}

1 comment:

Comment policy: I enthusiastically welcome corrections and I entertain polite disagreement ;) Because of the nature of this blog it attracts a LOT - 5 a day at the moment - of spam comments (I write about spam practices,misleading marketing and unevidenced quackery) and so I'm more likely to post a pasted version of your comment, removing any hyperlinks.

Comments written in ALL CAPS LOCK will be deleted and I won't publish any pro-homeopathy comments, that ship has sailed I'm afraid (it's nonsense).