A list of tools for finding or capturing tweets

Shortened link for this post is http://is.gd/ennsBC
Last updated: 14 October 2011 - "h/t" means hat tip, basically where I heard about something.

UPDATE 7 July 2011: Google Realtime is no longer with us. Initially it seemed to be temporarily offline while under-bonnet tinkering happened and it was assumed it would be hooked up with Google+ however it now seems that Google is no longer accessing Twitter's stream as the deal ended on 2 July 2011. This isn't great.

Part A is the alphabetic list, with a bit more info about each individual tool.

Part B looks at a range of tools according to the function(s) they can perform - B1 for finding tweets, B2 for trapping tweets and B3 for handling conference or event hashtags.

I might have sucked all the fun out of this post by itemising everything but it was getting a bit chaotic ;)

Currently I am using Google Realtime to find any tweet posted within the last year and SearchHash and / or Twilert to trap hashtagged event tweets. I’m also using Storify for small events because the interface is so good and it lets you contextualise things. But Chirpstory is very good as a quick trapping tool too, for smaller numbers of tweets.

Part A: alphabetic listing of tools

• BackTweets - acquired by Twitter, no longer operating under its own steam but apparently it's been absorbed into Twitter's Web Analytics tool that was announced on 13 September 2011. I think you need a website to use this though.
• Bing site search – assume same as Google, not tried.
• Bing social – Bing’s Twitter search
• Blackbird Pie - lets you embed an example tweet in a blogpost - my example here.
• Chirpstory.com - suggested by @annindk. It's like Storify and Keepstream. This one lets you reverse the order of your chosen panel of tweets (start with the earliest) and also lets you move a whole block of tweets - both can be done with a single click. You can also remove RTs and duplicates. Storify is investigating this, and does have the slightly easier interface but I do like Chirpstory - here's one I made earlier http://chirpstory.com/li/1600
• Curated.by – like Storify and Keepstream
• Favotter - seems to be permanently down.
• Favstar – lets you see favourite tweets (yours, or someone else’s) – since not every single one of someone’s tweets will be favourite this is a highly selective filter of available tweets. Here are MY tweets that OTHERS have favourited.
• Freezepage - literally traps a web page as it is now and keeps it that way in case the page changes or is deleted. My favourite example of frozen tweets marks Gillian McKeith's rudeness to someone pointing out she has no real PhD qualification.
• FriendFeed - used to work beautifully, seems to have captured my entire tweetstream (began archiving well after started using Twitter so assume it can comfortably search retrospectively because I used it to find my first tweet). Search is no longer working very well.
• Google Advanced Search
• Google Realtime – formerly awesome, now no longer with us.
• Google site-restricted search – use site:twitter.com to search within Twitter
• Google Spreadsheets - "A hack for searching for tweets by date, with help from Google Spreadsheets" by @steveWINton and highlighted to me by @CliveAndrews
• Guardian Twitterfall – app for events where you want to pre-moderate the tweets that appear on the wall.
• hashtags.org - shows you a list of recently posted tweets containing that hashtag. Doesn't appear to do that much with them, but does show trends in a little graph which might be interesting - h/t this slideshare presentation.
• Hashtracking - h/t rjleaman. It's currently in beta (note URL might change as it currently includes 'beta' in it!) and I've applied to try it out, hopefully they'll let me. Apparently it collates hashtags and has some analytics (don't know what) so it might actually be a sort of replacement for WTHashtag / What the Hashtag
• Hashtweeps - type in a hashtag and the service will tell you people who've been tweeting with it - h/t this slideshare presentation.
• Healthcare Hashtags - if you have a health-related hashtag you can add it and capture tweets here. Not tried it for non-healthcare purposes.
• identi.ca – an alternative to Twitter, on signing up you agree to your tweets being open – wondered if enough people were using this to make it worthwhile using in a conference setting.
• Keepstream.com - bit like Storify and curated.by - I got an email about this recently, no longer fully functional.
• Library of Congress tweet archive – does anyone know if this is searchable?
• Monitter - lets you watch a stream of tweets in real time. See also TweetChat and Twitterfall.
• Research.ly - you can have a play around for free but after 8 searches I think you have to log in with Twitter. Not really sure what it's telling me to be honest. Opportunity to filter tweets by positive and negative (requires login).
• Rowfeeder - 500 hashtagged tweets a month free, but more than that is chargeable on a sliding scale. h/t this Quora post
• RSS - New Twitter seems to have omitted any RSS feeds on its pages (apparently RSS is 'dead' but personally I have no truck with this nonsense). I believe the following URL formats should work, you'll have to add in your own term obviously - ie you are manipulating the URL. The phrase "search.atom" means it's an RSS feed, or at least that's how it seems to work anyway - it may stop suddenly, Twitter can be a bit capricious:
  
  For an individual user: https://twitter.com/statuses/user_timeline/jobrodie.rss
  
  For a person search try http://search.twitter.com/search.atom?q=jobrodie - this will find all instances of people posting TO me as well as all instances of ME posting. Note that every tweet is duplicated, goodness knows why.
  
  For a person search restricted just to @mentions use http://search.twitter.com/search.atom?q=@jobrodie
  
  For your own favourites use
  https://twitter.com/favorites/jobrodie.rss or http://twitter.com/favorites/jobrodie.rss
  
  For a keyword search, just use %20 between words to make the URL continuous, eg
  http://search.twitter.com/search.atom?q=fluffy%20bunnies - all instances of fluffy bunnies :)
  
  For a hashtag search, use %23 which browsers read as # symbol
  http://search.twitter.com/search.atom?q=%23scicomm - searching for #scicomm
  
  If you see the %20 and %23 replacing [space] and # respectively and are thinking "what else can I find to play with" you might like http://en.wikipedia.org/wiki/Percent-encoding
  See also http://www.quora.com/How-can-I-find-an-RSS-feed-for-a-single-Twitter-user and http://www.quora.com/Twitter-1/Is-there-an-RSS-feed-for-a-Twitter-hashtag
  
• SearchHash - wonderful. Does almost exactly what WTHASHTAG used to do - lets you archive and export a pretty complete record of a hashtag. You are also given the tweet ID and the author, to find the original link, use the formula http://twitter.com/AUTHOR/status/TWITTERid eg https://twitter.com/JoBrodie/status/75299030430777344 You can export tweets as a .csv but the data table isn’t the prettiest of things to share with people so Twilert might suit better.
• Searchtastic - shut down as they didn't have time to maintain it
• SearchTweetsByDate
• SnapBird
• Social Searching
• Storify - been using this a bit recently, rather good, and lots of fun. See also http://brodiesnotes.blogspot.com/2011/05/some-quick-thoughts-on-storify-as-mini.html and see also notes in the Freezepage bit above. I wrote a tweet, trapped it in Storify, then deleted it and demo'd that Storify can keep tweets after deletion.
• tagdef - find a hashtag and see what it means, or define it yourself if you know - h/t this slideshare presentation. See also tagalus.
• tagalus or tagal.us - this appears to be similar to tagdef but at time of viewing (September 2011) the page is down with a 503 service temporarily unavailable note. From their Twitter account it appears the last tweet was sent in November - h/t this slideshare presentation.
• The Archivist – I found this quite good but you have to keep an eye on the tweets as they disappear ‘off the bottom’ – very easy to miss tweets if you don’t actively monitor it. Desktop version also available.
• ThinkUp – downloadable app - EDIT 24 June 2011 @martinjmckenna suggested this one to me today although I forgot I'd already listed it. Possibly the reasons I didn't get very far are that you have to download it and run it on your server (don't have one). It seems to be very suited to people who want to do a bit of processing on their tweets.
• Topsy – search the social web: this is very good and can find old tweets (I found some from 2009) but it doesn't let you scroll back and forward like Google Realtime used to let you.
• Twapperkeeper - will let you access two archives free, it costs for more than two archives. See also YourTwapperkeeper, the open source version. You can also query (but not create) Twapperkeeper archives using Eduserv's Summarizr (h/t @briankelly)
• Twapperlyzer - apparently a "a hashtag community analyser" but it involves GitHub which means it's probably technically beyond me - h/t this slideshare presentation.
• Twazzup - looks like a news story thing, akin to paper.li, not investigated in depth yet as it's currently reporting a 500 error, h/t @davoloid.
• TweetBoard
• TweetChat - really good for watching a live stream. You can type in your hashatag of interest and it will present you with all the tweets in real time. If you miss a few you can scroll back and it will pause for you. Used very successfully in the first Twitter Journal Club aka #twitjc. You can also sign in via Twitter authorisation and tweet directly from the programme (it will automatically add the relevant hashtag to all your tweets). It also lets you block spammers from view (hat tip @fidouglas and @silv24). See also Monitter and Twitterfall.
• Tweetdoc - lets you create a PDF document bringing together not more than 500 hashtagged tweets and keep them as a permanent record h/t @martinrue
• TweetGrid - automatically updates a hashtag, so similar to things like Monitter or Twitterfall, try the search facility and you can add a Twitter search widget to your page - h/t this slideshare presentation.
• Tweetnotes - stalled early in development and is no more, sadly, as it showed promise.
• Tweetreach - calculates how far your tweet travelled (I assume it counts up the maximum number of eyeballs that could have seen it based on followers)
• TweetReports - ££
• TweetScan - temporarily closed to new registrations so I can't tell you anything about it
• Twitterfountain - for use in foyer display screens where you want to present tweets prettily. No use for archiving though. Very pretty though.
• Twitter Search - Twitter's own (much improved) search facility, note that there's also an Advanced Search but it's still fairly limited (note, it's very good it's just that I compare everything to the no-longer-with-us awesomeness of Google Realtime).
• Twitter widgets - "Widgets let you display Twitter updates on your website or social network page" - if you want to show, on your webpage, what you're sharing on Twitter then use one of these widgets. There's an example on the linked page. See also B3.3 below.
• Twilert - I like this a lot. It's good for capturing event hashtags by emailing them to you and the appearance is nice too (not sure how you can display tweets sent to you by email though). It sends in batches of 100 and you can have them sent every 15 mins if it's a very busy event. I wrote about it here. It's been around since 2009 and I only heard it about it at the end of June beginning of July 2011!
• TwimeMachine – will let you access the last 3,200 tweets in a person's timeline
• Twitter Advanced Search - I'm personally not a fan as it doesn't offer me any advantage over Google Realtime, but it is an exceptionally easy user interface (OK it is easier than GRealtime, but you can't scroll back and forwards in it which you can on Google).
• Twitterfall - lets you watch a stream of tweets in real time. See also TweetChat and Monitter.
• Twitter itself: the ‘More…’ scrollback button at the bottom lets you access 160 pages of 20 tweets per page (3,200 tweets) https://twitter.com/JoBrodie?max_id=60838663235452928
• twittertoirc - "Twitter-to-IRC bridge: A hacky Ruby script to route Twitter status updates into IRC." by @tommorrisI said in an earlier edition of this post: Another tool I have ‘mentally’ investigated is IRC (internet relay chat) as a backchannel for conferences. I’m sure all nerdy conferences have an IRC channel but for regular conferences possibly not – these let everyone sign in, all posts are labelled as who wrote them and it’s possible to create a transcript. Seems perfect – what I’d like to do, but am not clever enough to work out how, is to export a live Twitter RSS of a hashtag into an IRC channel and capture the tweets that way (maintaining tweets’ authors identities). – then discovered Tom had written this programme.
• Twitter tools - Wordpress plugin, see How to auto publish your Twitter tweets as Wordpress posts and the Twitter tools plugin FAQ.
• Twoogel
• Wayback Machine - entirely random, very limited.http://classic-web.archive.org/web/*/http:/twitter.com/JoBrodie
• WordPress plugin - trap de novo, curate from this point forwards - see Twitter tools
• wthashtag - What the Hashtag, no longer functioning in the way it used to alas and is now What the Trend API. If you have programming skills (JSON...?) you may get something useful from it but it's beyond me.
• Yahoo pipes – haven’t the faintest idea how to get one to work, but clever people might play around with this and send me some instructions as it seems to be potentially useful. Some info at http://thecodetrain.co.uk/2009/05/following-twitter-conversations-using-yahoo-pipes/
• Yahoo site search – not investigated but assume it will be similar to Google’s
• Yahoo twitter search – I think this is similar to Google’s Realtime search
• YourTwapperKeeper – open source version of TwapperKeeper

Part B: tools ordered by functionB1. Finding tweetseither from May 2010 onwards or 3,200 tweets ago and no earlierNote that all of your tweets still exist somewhere on Twitter's servers but they stop being amenable to its searches once they are superseded by another 3,200 tweets.

To prove that old tweets don't really disappear here's Jack Dorsey's tweet from 22 March 2006 https://twitter.com/jack/statuses/81 - he co-founded Twitter I think. This might be the earliest tweet that remains on Twitter.

B1.1 Scrolling back on Twitter
If you've posted fewer than 3,200 tweets in total (including RTs) then you will be able to scroll continually back (160 pages maximum at 20 tweets per page) to your first one. On New Twitter the page will continually expand with another 20 tweets as you scroll to the end of the page, but on old Twitter you'll need to click on the More button.

B1.2 Fast scroll (jump) by tweaking a web address: 'URL hacking' - old Twitter onlyYou should also be able to find your own Twitter URL that will let you jump back pages by changing numbers in the URL. It seems that this might not work if you're using new Twitter and it's pretty fiddly anyway. Scroll to the bottom of the page until you see a More button - this might not appear on new Twitter - hover over it and make a note of the URL that appears in the status window (I open up a small notepad window to the side and type it in). You'd think you could right click, save URL - but no - for some reason left clicking and right clicking both activate the More button so you will have to make a note manually (you could also take a screen shot of the page using Print Screen, copy it into Paint or Word and take down the details at leisure).

Once you have the URL (eg something like this) https://twitter.com/jobrodie?max_id=86927876561190912&page=2&twttr=true - annoyingly the numbers bit changes so I don't think you can just make a note of this and use the same one. Paste it into a new tab or window and change the bit saying page=2 to page=20 and so on. The max is page=160.

- arrows point to the URL you need.

B1.3 Find your first tweet
Posted more than 3,200 tweets? Then it's probably not straightforward.
If it's working (and you have fewer than 3,200) you can cheat and use http://myfirsttweet.com/, or the 'jump' function mentioned above. If you've been a bit chattier and have several thousand then you'll have some difficulties finding your first.

Google Realtime is a wonderful thing from Google that lets you find tweets going back as far as May 2010 - I use it quite regularly, more details below in the list of tools.

It was my good fortune shortly after signing up to Twitter to import all of my tweets into my FriendFeed account meaning that I could search, very easily, for any of my tweets. FriendFeed's search has become much less reliable of late - but it did (though no longer does) let me find my very first tweet https://twitter.com/JoBrodie/statuses/841825864. Oddly, searching now for this no longer works - instead of picking up the original tweet, it picked up only the later search I ran for the tweet in 2009 ;)

B1.4 Finding tweets from a week or two agoI think Google Realtime is probably the best for this, see more details and alternative in the list below. Possibly much more difficult now unless you have tweets fed into FriendFeed. You could also give Topsy a go.

B1.5 Find tweets on a topic or hashtag
Twitter search is good for finding recent ones, just type in your keyword(s) or hashtag. For older ones, use Google Realtime.

B2 Trapping tweets – see also B3.2Freezepage, Chirpstory and Storify will let you capture a tweet. It will remain captured even after deletion, see Freezepage and Storify details above.

B3 Conferences and events: for events organisers and delegates
B3. 1 Follow an event hashtag in real-time, at conference or at home
Twitter itself is pretty good for this - just search for the hashtag and sit back and watch the tweets auto-refresh. Also particularly good is Tweetchat - neither requires you to have an account to watch the tweets. If you have a Twitter account you can sign in to Tweetchat and send tweets that have the hashtag of interest already written in the tweet for you. More info below, and see also Monitter and Twitterfall.

B3.2 Display a Twitterwall in the foyer or the conference roomPrettiest so far seems to be Twitterfountain, ideal for the anteroom display for people to coo over. For a backdrop that people can follow inside the event room I prefer Twitterfall (actually I prefer no Twitterwall but…). See also Guardian Twitterfall.

B3.3 Display event tweets on your website
If you want to bring in the results of a search for an event hashtag you can do that with Twitter Widgets. The 'Search' widget (the options are Profile, Search, Favorites, List) will pull in a hashtag or other keyword but if you worry that cheeky types will introduce rude words into the tag (or just want to show a selection) then you can instead use the 'Favorites' widget and as you favourite each relevant tagged tweet it will show up. This requires more work from you though... See also TweetGrid (listed above) for an option to let people search for anything on Twitter via a widget you can add to your site.

B3.4 Saving an event hashtag for later
Best in show is SearchHash or Twilert but there are a few other tools available too. Twitter's made it a great deal harder for people to download / export tweet archives. The sooner you set these up the better (ie before the event begins) because people often use the hashtag a day or even a week / month or two before an event begins. Chirpstory is great for bagging a run of tweets soon after an event and lets you capture them a page at a time whereas the similar Storify lets you capture them individually but also lets you intersperse commentary. The first two are better for big conferences, the latter are better for small things. Previously everyone used wthashtag (What the hashtag) but it no longer works, and Twapperkeeper only lets you create two free archives.

B3.5 Format tweets for conference output documents, eg blogsStorify is lovely for this – you can embed the final Storify story into most blogs, but to place an individual tweet into a blogpost directly… I need to find out what the tool is that I’ve seen used!

Found one - an example is Blackbird Pie which lets you select a tweet and 'bake' it into your blogpost. I've linked an example of me trying it out in the alphabetic list above. Although I've not tried it I understand the Twitter widget in WordPress will also do this.

Further information on tools mentioned above
The end of Twapperkeeper? (and what to do about it)March 8, 2011, 11:00 amBy Mark Samplehttp://chronicle.com/blogs/profhacker/the-end-of-twapperkeeper-and-what-to-do-about-it/31582

Top 10 Websites To Search Old TweetsBy YOUNG
February 16, 2011
http://freenuts.com/top-10-websites-to-search-old-tweets/

All the easiest ways to search old tweetsAugust 11, 2010
by DAVE LARSON
http://blog.tweetsmarter.com/twitter-search/10-ways-and-20-features-for-searching-old-tweets/ - covers some of the options listed above, and others I’ve not heard of.

Other posts by me, on related topics

• Some quick thoughts on Storify as a mini archive for Tweets (4 May 2011)
• Screencap video that I made about Google Realtime (30 April 2011)
• Google Realtime: Finding threaded conversations on Twitter as well as year old tweets (14 April 2011) - features Google Realtime, Storify, Wayback Machine and my failed attempts to find the searchable archive at the Library of Congress
• Trapping tweets for archiving eg at conferences, or curiosity (5 March 2011)