How to back up or download your blog that is hosted on free sites eg Blogger

Short version: get hold of a copy of the XML file for your blog. This contains all of the information (posts, site hierarchy, comments) in one convenient web file that can be used to relaunch your blog in case of disaster.

---

This post is for people like me, who have their blog hosted on sites like Blogger, Wordpress.com or Posterous. I've no idea about setting up, or backing up, a self-hosted blog (where you interact with a server). If someone who knows more about that fancies contributing a few lines for me to add here... that would be nice.

Things you can do now while your blog is still up and running
Remember, backing up your blog isn't a one-time thing but something you might want to do every few days or weeks (depending on how frequently you blog). Probably there's some way to automate this process but if there is, I don't know it.

1. Backing up your blog

Blogger (Google)
To download blog, click on the 'Design' icon at the top of your blog (being logged in) and then click on the Settings | Other menu, then click "Export Blog" and choose the Download option. You will then be saving an .xml file which you can keep safe until you need to import it into another blog host.
From http://support.google.com/blogger/bin/answer.py?hl=en&answer=97416

Posterous
http://posterous.com/#backup - then click on the Request Backup button for your blog(s).

WordPress
Go to your blog(s)' site admin area (stick /wp-admin/ after the end of your blog's address).
In the menu on the left click on Tools | Export and choose the free option which gives you an xml file
 
Tumblr
I am less familiar with this one so have taken advice from Google but it seems that the following option may help:
• HTTrack (this will back up any website to be honest, I use it for other purposes)
• See also all Tumblr posts tagged with Tumblr backup

2. Mirroring your blog while it's still up and running

I have a copy of this very blog at Blogger.com over at Wordpress.com though it's private as it's confusing to have two copies of the same information, I also have the xml files of blogs I'd be annoyed to lose.

You can mirror your blog in two ways I think.
(1) Authorise the two blog hosts to talk to each other and let the software do the importing for you
(2) Import a blog using your xml file (obviously this ought to work whether or not your original blog is up and running or suspended).

Importing into Wordpress from another site
• Create account and new blog to house your content
• Stick /wp-admin/import.php on the end of your blog's address to go to the admin / import site (or use Tools | Import) - you can choose to import from several different blogging platforms including Blogger, Posterous or Tumblr.
• You'll then be asked to authorise Wordpress to interact with your original blog and you can then import the posts.

Importing into Blogger from another site (not sure if this will work)
Not sure if this will work using the xml setting as it says 'import from a Blogger xml' rather than just any old xml - I can't see how one xml differs from any other but 'more research needed'. Apparently there is a Wordpress to Blogger conversion tool if your 'another site' is Wordpress.

• Create an account and set up a new blog at http://www.blogger.com/home.
• If doing so takes you to a different page, go back to the link above (home)
• To the right of your new blog is an orange pencil symbol (create new post) a couple of other icons and a 'view blog' icon. Click on the small down arrow between them to bring up the options, choose Settings | Other and then Import blog from the options along the top
• Upload your xml file and hope for the best.

See also Blogger's own advice on importing and exporting blogs (also mentioned above).

Recovering your posts when it's a bit late for the above
1. Stick your blog's URL into Google and capture what you can from Google's cache. You can also remove the http:// bit and replace it with cache: to do this.

The result will be pages of search results and so will involve quite a bit of labour in capturing them all, you can use 'File / Save page as...' to save them to your hard drive.

Edit: Alan Heness has suggested the following, accessed by using a Web Cache extension for the Chrome browser. Some of these I confess I've never heard of :)

Google's cache

Yahoo's cache

Bing's cache

CoralCDN

WebCite

GigaBlast

Wayback Machine

2. Once on a cached page make note of any table of contents or archives for each month. Google will return your blog's posts in no particular order making it difficult to know if you've got everything. It's much easier to know what you're searching for and you can also get hold of a cached copy of your month by month archives - the purpose of this is to bring up a page with the title and links of your posts for that month. The links themselves won't work if your blog is down but, again, stick cache: in front of them to see if you can grab a cached copy from Google.

3. Use the Internet Archive's Wayback Machine to find older posts if Google cache doesn't have them. Note also that Google Cache might not have very recently published posts.

Background
This post arose after a few posts and entire blogs I enjoy reading were apparently taken down including the Retraction Watch website (see also ArsTechnica on the story), the 21st Floor and Josephine Jones' blog (the latter turned out to be a glitch, I've done it myself when importing my blog to Wordpress - I triggered their spam warning and had to ask them to restore it, which they did within a couple of hours although it looks like someone is keen to see it taken down [copy]).

In the case of Retraction Watch (RW) a seemingly mistaken DMCA (Digital Millennium Copyright Act) 'takedown' request was made to the organisation hosting RW's blog (Wordpress) stating that RW had posted material that belonged to someone else. Nonsense as this may be it seems that the effect of DMCA takedowns are more like 'shoot first, ask questions later'. Similarly blogs can be terminated for violation of terms of service.

Sometimes blogs get it wrong and post stuff that isn't theirs to post, so fair enough, other times it's just a blunt tool to remove perfectly valid but perhaps inconvenient information.

Chilling Effects is a site where people can upload their DMCA notices - the site is collecting and commenting on them (not all are unreasonable). They also have an online DMCA counter claim if you believe your material has been removed in error.

Be careful about putting in a claim or counter-claim - you may be committing a crime if you're wrong.

If you spot any mistakes or omissions in methods listed above please let me know, thanks.

Using free tools to capture a handful of tweets or a larger bunch

These are my most-used tools:

• Chirpstory.com (log in with Twitter)
• SearchHash.com <-- from @lesteph (no login required) 
• Storify.com (log in with Twitter, Facebook or create a Storify account)
• GrabChat.com <-- new in, this is made by @dean_jenkins (no login required)
• Epilogger (log in with Twitter or Facebook) - this actually automates stuff.

Edit: 21 September, now with added Epilogger :)

Questions to ask yourself

• Has the series of tweets happened already, is it still unfolding or is the event yet to occur?
• Is it just a few tweets you want, eg a short conversation, or a whole bunch from a hashtag or even a user?

There are paid-for tools that will let you press a button and when you come back later all the hashtagged tweets you want are nicely collected and probably analysed in some interesting way. I'm afraid I know nothing about them and have always used the free tools - the cost is my (your) time in doing the manual processing. I also don't know anything about how to analyse your tweets (how many were sent, who sent the most etc).

If you have a server and some coding skills then you can use Twitter's API to collect lots of tweets too but I have neither and can't help you there either.

The event's about to start, currently happening, only just finished
You'll need at least one or two hashtagged event tweets to be able to search for them on any of the five tools listed above.

Chirpstory and Storify can be used once the event has ended or is winding down. Both will search further back in time than Twitter's own search so if you've left it a few days you might be OK (this is also partly dependent on the number of tweets - I don't know what the cap is but the most I've ever captured is definitely under 3,200 tweets (typically about 20 pages of 50 tweets is capturable on Chirpstory. Storify lets you capture them in bunches of 20 so I generally prefer Chirp for bigger events).

I often set up a Chirpstory then collect the max number of tweets available, then return to it later adding more. You'll end up with duplicates but there's a button that lets you remove those. You can also remove RTs if you want to (sometimes it's useful to keep these in to get a sense of how often an idea's been shared I suppose) and re-order them chronologically (earliest tweet first).

Both embed into Blogger very well but I understand that the free version of WordPress (ie wordpress.com rather than downloading the software to self-host the blog on your own site) doesn't permit this sort of embedding - something to do with restrictions on the use of javascript I think, not sure).

SearchHash will run in the background for you while the event's running and you can refresh it periodically before 'tapping off' a selection of tweets to store in a more permanent form. Older tweets will eventually fall off from the SearchHash search so don't just leave it running. It's not truly automated, but it's a great help to grab a bunch to copy and paste etc.

I've not tested GrabChat to destruction yet but it looks like you can get a really nice end product to embed in your blog - have a look at the examples given on the front page at the link above. I would probably use this after the event had finished (so that there were enough tweets to be bothering with).

Edit 21 September

Epilogger is (to me) the new kid on the block and thus far it's pretty amazing. I set one up for @ayisasophia a week or so ago to track all #talesfromtheriver hashtagged tweets and at tiime of writing this it's trapped 827 of them. It will also search backwards (I don't know how far back) so if I set it up on "Friday 5th Monthtember 2012" it will find the tagged tweets from Wednesday 3rd Monthtember too.

So far it doesn't embed into blogs but I believe that's something they're looking into.

Event happened ages ago
While you probably won't be able to search for all the tweets by hashtag after a week or two has passed you can, if you can be bothered, recover them with a bit of time and effort. Unless a tweet has been deleted it remains permanently on Twitter's servers but stops being amenable to Twitter's search (or third party search tools) a week or so after posting. Tools like Topsy.com can find much older tweets (I've found ones from 2009 there) but you can't follow a thread, and what turns up can be a bit random). I'm in a fortunate position that I signed up to FriendFeed.com in 2008 and sent my tweets there so I can quite often find what I'm after, also all my tweets with a link in them are captured by Delicious.com - however this is of no help if you've not already set this up. Sorry :)

Things that make a tweet more discoverable are if you or someone else has favourited it (you can simply scroll through your list of favourites and, if you've favourited fewer than 3,200 tweets, you'll find it listed there).

A note on 3,200 tweets This seems to be the maximum number of tweets in any one thread that Twitter will let you access at one go. You can scroll back on your own tweets, or someone else's, only as far as your most recent 3,200 tweets. Same goes for hashtags, favourites and any other search. That doesn't mean you can't collect more than that - if thousands of tweets are being emitted on a topic you need to collect a bunch of them on day one, more on day two etc.

There's no guarantee that you'll be able to access all the older ones back to the 3,200th-tweet-ago though - Twitter doesn't let you scroll back all the way on hashtag searches, and neither do most third party apps. Should be OK on an individual's account.

If you want to grab a few tweets, eg in a conversation, and don't have the luxury of a hashtag to fish them all out with at one go, then it's a little bit more work but still pretty straightforward. Every tweet ever sent has its own URL and web page (not kidding) and if you know what the URL - see below - is then you can use Chirpstory or Storify to capture that tweet. For a smaller number of tweets I tend to use Storify - it has a very nice interface.

First catch your URL (the link that points to the tweet on its own webpage)
This assumes you've been able to find your tweet(s) either by search, by scrolling back through yours or someone else's timeline, mentions or favourites.

If the tweet has a grey 'Expand'or 'Details'  link below it then you can get the tweet's URL from that - plonk your mouse cursor over it and right-click, copy link address to copy and paste it somewhere else (eg in the Storify link search).

If the tweet is part of a response to someone else then you'll see a blue 'View conversation' link at the bottom and you can get the URL from that. If there is a chain of tweets in the conversation then clicking on the first one (you don't literally have to click on 'view conversation', just click anywhere within the tweet to expand it).

Using Storify to add a single tweet using its URL
Once you've got your URL(s) log in to Storify and use the link tool to search for the tweet (paste in the link, press enter, the tweet will appear) and get it in a format that you can embed into a Storify story.

The link tool is highlighted in yellow below.

If you're collecting loads of tweets by hashtag then it's the Twitter icon (the bird on the second left in the picture below) that you need.

Further reading
A list of tools for finding or capturing tweets - much more detailed post covering more tools and their uses.


Further posts in the Twitter tips series...

Quora questions I've answered about Twitter and hashtags

I've joined Quora and have been answering questions. Others have answered them too and I've learned of new tools from these people.

When I stopped working as a scientist my first job in the world of science communication was ScienceLine/ ScienceNet where I got to find out answers to stuff and tell people about science. Awesome. This also features heavily in one of my current jobs too...

Quora lets me answer questions and read others' answers on train journeys and things and I can see why someone (probably an employee of Quora, but I still agree with them) has described it as the crack cocaine of the internet. I'm sure other things have also been described like that.

Anyway, as you may have picked up, I'm pretty interested in tools used to capture tweets. Increasingly this is because Twitter appears to be actively trying to stop me from doing this (it changed its terms of conditions earlier this year and lots of useful tools suddenly stopped working). Instead of just trying to capture stuff I'm also trying to get around this annoyance and generally find ways of thwarting Twitter.

Here are the questions relating to Twitter and hashtags to which I've contributed, with a very brief description of my stance on the topic. I've contributed to other questions too but these are just the Twittery ones.

• Would writing a Twitter Bot get me banned? No / depends
• Is there a reason to register a hashtag with a place like tagdef? Possibly not
• How can I search for tweets that are older than what's archived on Twitter? Up to a point yes, but fiddly
• Is there a way to archive tweets from a hashtag after a certain period of time? Up to a point yes, but fiddly
• What is the etiquette on tagging things on Twitter? Freestyle
• Did @mentions exist before Twitter? If so, how? Yes
• Are there any tools/ services that provide a way to curate, select and publicly display / publish tweets with a specific hashtag? Yes
• Why do people put a dot in front of tweets? Wider sharing
• On Twitter, why do people put a # sign in front of a word? Tagging
• What are the best tools for Twitter chat transcripts and why? Loads

A list of tools for finding or capturing tweets

Last updated: 13 October 2012
Shortened link for this post is http://is.gd/ennsBC

3 November 2012: Interim edits (cos I'm not doing this in one go!) - I'm at twapperlyzer in the list below. Anything below that hasn't been checked for a while, everything above has been.
 
Post history: This post was born on 30 May 2011 at a time when there were many more free tools available with which to wrangle tweets. Since then Twitter's changed its terms of service and Google Realtime no longer exists making some of the information now wildly out of date, in October 2012. The original post will be placed HERE as a PDF for completists and archivists who want to know how things used to be. But the text below has been significantly changed.

I've found Quora to be the best resource for finding out about new tools for a couple of reasons: (1) is that Quora answers persist and don't fall off the edge of the page compared to Twitter and (2) people organise the information into questions and answers making it much easier to find! Also people tend to post this info to Quora more than on Twitter (or I'm not following those people). 

What have we got here then?

Part A is the alphabetic list, with a bit more info about each individual tool.

Part B looks at a range of tools according to the function(s) they can perform - B1 for finding tweets, B2 for trapping tweets and B3 for handling conference or event hashtags.

I might have sucked all the fun out of this post by itemising everything but it was getting a bit chaotic ;)

Currently I use SearchHash, Epilogger, Chirpstory, Storify and / or Twilert to trap hashtagged event tweets. I’m also using Storify for small events because the interface is so good and it lets you contextualise things. But Chirpstory is very good as a quick trapping tool too, for larger numbers of tweets - last time I looked you can grab 20 tweets at a time with Storify and 50 at a time with Chirpstory.

Part A: alphabetic listing of tools 

• BackTweets – "search for links on Twitter"
  I searched for brodiesnotes.blogspot.com and found quite a few tweets including a couple from 2009. The search results don't actually give you either the link to an individual blog post (only the main blog homepage address) nor do they give you the link to the original tweet, though that is discoverable.
• Bing site search – assume same as Google, not tried.
• Bing social – Bing’s Twitter search
• Blackbird Pie – lets you embed an example tweet in a blogpost - my example here. Defunct as Twitter's now makes it possible to embed tweets in blogs very well. To get the embed code click on any tweet to expand it, then click on Details, then grab the code.
• Chirpstory.com – suggested by @annindk. Similar to Storify.
  This letss you grab a bunch of tweets (up to 50 at one go) and also lets you reorder them with one click so that readers see the earliest tweet first. Very useful tool, one of my favourites. You can also easily remove RTs and any duplicates you might have copied over twice. Here's one I made earlier http://chirpstory.com/li/1600
• Curated.by – no longer functions, kept in list for completeness.
• Embedding a tweet somewhere - see the information above in Blackbird Pie, it's easy to do now.
• Digestly - lets you receive, by email, tweets from any user with a public account
• Epilogger – brilliant. This automates the saving of hashtagged tweets and can pick up tweets earlier than the time at which you set up your 'log'. Currently can't embed into blogs, but does provide a basic analysis of who's tweeted the most on a hashtag.
• Exquisite Tweets – lets you thread a bunch of tweets together, similar to Chirpstory and Storify, h/t @tomscott
• Favotter – no longer functions, kept in list for completeness.
• Favstar – lets you see favourite tweets (yours, or someone else’s) – since not every single one of someone’s tweets will be favourite this is a highly selective filter of available tweets. Here are MY tweets that OTHERS have retweeted.
• Freezepage – this used to trap an exact copy of whatever web page you were looking at in case the page is changed or deleted in future, however I've noticed it doesn't work as well for Twitter and have stopped using it. Here's one I made on 13 October. My favourite example of frozen tweets marks Gillian McKeith's rudeness to someone pointing out she has no real PhD qualification.
• FriendFeed – n 2008 I imported my Twitter feed (http://twitter.com/JoBrodie) into this tool and it's still capturing them. I thought it would stop what with Twitter mucking about with what it lets escape from its 'ecosystem' but so far all is well. Here's my first tweet, which I originally found in FFeed but time has passed and it doesn't appear by searching anymore.
• Google Advanced Search – well, always worth a go I suppose.
• Google Realtime – formerly awesome, now no longer with us.
• Google site-restricted search – use site:twitter.com to search within Twitter
• Google Spreadsheets – "A hack for searching for tweets by date, with help from Google Spreadsheets" by @steveWINton and highlighted to me by @CliveAndrews - I have no idea if this will work still, it looks pretty technical. If the Twitter export doesn't work, try RSS feeds until Twitter stops supporting them too in March 2013.
• GrabChat.com – made by @dean_jenkins for people to 'archive your conversation'. It lets you export a whole bunch of hashtagged event tweets, really good.
• Guardian Twitterfall – app for events where you want to pre-moderate the tweets that appear on the wall – it's for people who understand what JSON and AJAX mean, sadly that doesn't include me.
• hashtags.org – apparently it's meant to show you tweets with a particular hashtag along with a little graph but I think you have to spend money for it to do so, so I've not bothered with it – via this slideshare presentation.
• Hashtracking - via rjleaman. I follow them on Twitter (@hashtracking) and often pick up some really interesting stuff, although I've not really tried out the tool very much. I tend to go for the cheapy freebies but I hear good things about this one, probably best for corporate use if you're a cheapskate like me.
• Hashtweeps - type in a hashtag and the service will tell you people who've been tweeting with it - via this slideshare presentation.
• Healthcare Hashtags - if you have a health-related hashtag you can add it and capture tweets here. Not tried it for non-healthcare purposes.
• identi.ca – an alternative to Twitter, on signing up you agree to your tweets being open – wondered if enough people were using this to make it worthwhile using in a conference setting.
• IFTTT – If This Then That.
  An excellent tool for all sorts of social media stuff. Twitter triggers have now been disabled at Twitter's request so you'll need to use RSS as the trigger instead. That will stop working after March 2013  less so for Twitter.
  >> Here are the suggested Twitter RSS feeds you can use.
  >> More info on the changes to IFTT here or here 
• Keepstream.com – it's read only now.
• Library of Congress – they keep every tweet but it's not available to the public, only to a small number of researchers. No idea if this will change in the future.
• Monitter - lets you watch a stream of tweets in real time. See also TweetChat and Twitterfall.
• PeopleBrowsr- lets you scroll back through several hours' worth of a person's tweets or tagged tweets. You get 10 free goes. Not really sure how it's unique.
• Research.ly – appears to have turned into PeopleBrowsr
• Roundteam - automatic retweet service [added 28 Feb, not investigated]
• Rowfeeder – 500 hashtagged tweets a month free, but more than that is chargeable on a sliding scale, via this Quora post
• RSS – Twitter will stop supporting RSS after 5 March 2013 which is a bit of a shame. If you're using IFTTT and your Twitter triggers aren't working, just edit your recipes so that the trigger is an RSS version of the same Twitter feed.
  
  For an individual user:
  https://api.twitter.com/1/statuses/user_timeline.rss?screen_name=jobrodie
  
  For a person search:
  http://search.twitter.com/search.atom?q=jobrodie - this will find all instances of people posting TO me as well as all instances of ME posting. Note that every tweet is duplicated, goodness knows why.
  
  For a person search restricted just to @mentions:http://search.twitter.com/search.atom?q=@jobrodie
  
  For your own favourites usehttp://api.twitter.com/1/favorites.rss?screen_name=jobrodie
  
  For a keyword search:http://search.twitter.com/search.atom?q=fluffy%20bunnies (%20 acts as 'space')
  
  For a hashtag search:http://search.twitter.com/search.atom?q=%23scicomm  (%23 acts as '#')
  
  If you see the %20 and %23 replacing [space] and # respectively and are thinking "what else can I find to play with" you might like http://en.wikipedia.org/wiki/Percent-encoding
  
  
• SearchHash - wonderful. Does almost exactly what WTHASHTAG used to do - lets you archive and export a pretty complete record of a hashtag.
  
  You are also given the tweet ID and the 'from user' author in the table. To find the link for the original tweet use the formula http://twitter.com/AUTHOR/status/TWITTERid eg https://twitter.com/JoBrodie/status/75299030430777344 You can export tweets as a .csv but the data table isn’t the prettiest of things to share with people so Twilert might suit better.
• Searchtastic – shut down as they didn't have time to maintain it
• SearchTweetsByDate – appears to be dead
• SnapBird – lets you search a little further back than Twitter's search (though Storify or Chirpstory do this and I use them more)
• Social Searching – no idea, log in with Facebook - can anyone tell me what it does?
• Storify – I use this a lot but not quite a much as Chirpstory. Once you've trapped a tweet in a Storify story it will remain there even after the original tweet has been deleted. Use this power wisely...
• tagdef – find a hashtag and see what it means, or define it yourself if you know - via this slideshare presentation. It's not possible to 'reserve' a hashtag and say 'it means this' and others can't use it, but sometimes it's helpful to know what it's supposed to mean. See also tagalus.
• tagalus or tagal.us – was similar to tagdef, now permanently down. From their Twitter account it appears the last tweet was sent in November - via this slideshare presentation.
• The Archivist – I found this quite good but you have to keep an eye on the tweets as they disappear ‘off the bottom’ – very easy to miss tweets if you don’t actively monitor it. You can download in .csv (spreadsheet) or zipped file formats. Desktop version also available.
• ThinkUp – downloadable free open-source app which analyses your tweets, Fb and Google_ activity. If you have your own server you can use it on that, if not they've partnered with a cloud company PHP Fog and you can host it there instead. h/t @martinjmckenna
• Topsy – search the social web: this is very good and can find old tweets (I found some from 2009) but it doesn't let you scroll back and forward like Google Realtime used to let you.
• Twapperkeeper - closed on 6 January 2012 - before that it let you access two archives free, and you had to pay for more than two archives. See also YourTwapperkeeper, the open source version - its current status is unknown to me. You can also query (but not create) Twapperkeeper archives using Eduserv's Summarizr (h/t @briankelly)
• Twapperlyzer - seems to be linked to YourTwapperkeeper (see Twapperkeeper) h/t this slideshare presentation.
  Point at which I stopped editing... info below may be out of date
• Twazzup - looks like a news story thing, akin to paper.li, not investigated in depth yet as it's currently reporting a 500 error, h/t @davoloid.
• TweetBoard - unavailable
• TweetChat - really good for watching a live stream. You can type in your hashatag of interest and it will present you with all the tweets in real time. If you miss a few you can scroll back and it will pause for you. Used very successfully in the first Twitter Journal Club aka #twitjc. You can also sign in via Twitter authorisation and tweet directly from the programme (it will automatically add the relevant hashtag to all your tweets). It also lets you block spammers from view (hat tip @fidouglas and @silv24). See also Monitter and Twitterfall.
• Tweetdoc - lets you create a PDF document bringing together not more than 500 hashtagged tweets and keep them as a permanent record h/t @martinrue
• TweetGrid - automatically updates a hashtag, so similar to things like Monitter or Twitterfall, try the search facility and you can add a Twitter search widget to your page - h/t this slideshare presentation.
• Tweetnotes - stalled early in development and is no more, sadly, as it showed promise.
• Tweetreach - calculates how far your tweet travelled (I assume it counts up the maximum number of eyeballs that could have seen it based on followers)
• TweetReports - ££
• TweetScan - temporarily closed to new registrations so I can't tell you anything about it
• Tweet Tunnel - lets you go back 3,200 tweets (as Twitter itself does) but you don't have to scroll through all of them but can 'page' through (random access versus linear access I suppose!)
• Twitterfountain - Currently inactive though website promises a revamp. Was a wonderful way to present tweets at conferences on a display screen. Until it's back, use Twitter Fontana etc.
• Twitter Fontana - pretty cool, for use in foyer display screens where you want to present tweets prettily. No use for archiving though.
• Twitter Search - Twitter's own (much improved) search facility, note that there's also an Advanced Search but it's still fairly limited (note, it's very good it's just that I compare everything to the no-longer-with-us awesomeness of Google Realtime).
• Twitter widgets - "Widgets let you display Twitter updates on your website or social network page" - if you want to show, on your webpage, what you're sharing on Twitter then use one of these widgets. There's an example on the linked page. See also B3.3 below.
• Twilert - I like this a lot. It's good for capturing event hashtags by emailing them to you and the appearance is nice too (not sure how you can display tweets sent to you by email though). It sends in batches of 100 and you can have them sent every 15 mins if it's a very busy event. I wrote about it here. It's been around since 2009 and I only heard it about it at the end of June beginning of July 2011!
• TwimeMachine – will let you access the last 3,200 tweets in a person's timeline
• Twitter Advanced Search - I'm personally not a fan as it doesn't offer me any advantage over Google Realtime, but it is an exceptionally easy user interface (OK it is easier than GRealtime, but you can't scroll back and forwards in it which you can on Google).
• Twitterfall - lets you watch a stream of tweets in real time. See also TweetChat and Monitter.
• Twitter itself: the ‘More…’ scrollback button at the bottom lets you access 160 pages of 20 tweets per page (3,200 tweets) https://twitter.com/JoBrodie?max_id=60838663235452928
• twittertoirc - "Twitter-to-IRC bridge: A hacky Ruby script to route Twitter status updates into IRC." by @tommorrisI said in an earlier edition of this post: Another tool I have ‘mentally’ investigated is IRC (internet relay chat) as a backchannel for conferences. I’m sure all nerdy conferences have an IRC channel but for regular conferences possibly not – these let everyone sign in, all posts are labelled as who wrote them and it’s possible to create a transcript. Seems perfect – what I’d like to do, but am not clever enough to work out how, is to export a live Twitter RSS of a hashtag into an IRC channel and capture the tweets that way (maintaining tweets’ authors identities). – then discovered Tom had written this programme.
• Twitter tools - Wordpress plugin, see How to auto publish your Twitter tweets as Wordpress posts and the Twitter tools plugin FAQ.
• Twoogel
• twtr2rsc beta - ostensibly sends you tweets by email (as Twilert does) or to Evernote (as IFTTT and a Twitter RSS feed trigger does, until March 2013) but I couldn't get anywhere with it.
• UgoTwitt - your Twitter replies sent by email. May not work now Twitter's tweaked things. 
• Visible Tweets - lovely, lets you display hashtagged event tweets in a nice way on a large screen in the foyer at your event or in the event room h/t @clangyandjammy. If you want people in the room to see tweets as they're coming in though I'd recommend something like Twitterfall or Monitter.
• Wayback Machine - entirely random, very limited.http://classic-web.archive.org/web/*/http:/twitter.com/JoBrodie
• WordPress plugin - trap de novo, curate from this point forwards - see Twitter tools
• wthashtag - What the Hashtag, no longer functioning in the way it used to alas and is now What the Trend API. If you have programming skills (JSON...?) you may get something useful from it but it's beyond me.
• Yahoo pipes – haven’t the faintest idea how to get one to work, but clever people might play around with this and send me some instructions as it seems to be potentially useful. Some info at http://thecodetrain.co.uk/2009/05/following-twitter-conversations-using-yahoo-pipes/
• Yahoo site search – not investigated but assume it will be similar to Google’s
• Yahoo twitter search – I think this is similar to Google’s Realtime search
• YourTwapperKeeper – open source version of TwapperKeeper

Part B: tools ordered by functionB1. Finding tweets either from May 2010 onwards or 3,200 tweets ago and no earlier
Note that all of your tweets still exist somewhere on Twitter's servers but they stop being amenable to its searches once they are superseded by a whole bunch of other tweets. 

To prove that old tweets don't really disappear here's Jack Dorsey's tweet from 22 March 2006 https://twitter.com/jack/statuses/81 - he co-founded Twitter I think. This might be the earliest tweet that remains on Twitter.

B1.1 Scrolling back on Twitter
If you've posted fewer than 3,200 tweets in total (including RTs) then you will be able to scroll continually back (160 pages maximum at 20 tweets per page) to your first one. On New Twitter the page will continually expand with another 20 tweets as you scroll to the end of the page, but on old Twitter you'll need to click on the More button.

B1.2 Fast scroll (jump) by tweaking a web address: 'URL hacking' - old Twitter onlyYou should also be able to find your own Twitter URL that will let you jump back pages by changing numbers in the URL. It seems that this might not work if you're using new Twitter and it's pretty fiddly anyway. Scroll to the bottom of the page until you see a More button - this might not appear on new Twitter - hover over it and make a note of the URL that appears in the status window (I open up a small notepad window to the side and type it in). You'd think you could right click, save URL - but no - for some reason left clicking and right clicking both activate the More button so you will have to make a note manually (you could also take a screen shot of the page using Print Screen, copy it into Paint or Word and take down the details at leisure).

Once you have the URL (eg something like this) https://twitter.com/jobrodie?max_id=86927876561190912&page=2&twttr=true - annoyingly the numbers bit changes so I don't think you can just make a note of this and use the same one. Paste it into a new tab or window and change the bit saying page=2 to page=20 and so on. The max is page=160.

- arrows point to the URL you need.

B1.3 Find your first tweet
Posted more than 3,200 tweets? Then it's probably not straightforward.
If it's working (and you have fewer than 3,200) you can cheat and use http://myfirsttweet.com/, or the 'jump' function mentioned above. If you've been a bit chattier and have several thousand then you'll have some difficulties finding your first.

Google Realtime is a wonderful thing from Google that lets you find tweets going back as far as May 2010 - I use it quite regularly, more details below in the list of tools.

It was my good fortune shortly after signing up to Twitter to import all of my tweets into my FriendFeed account meaning that I could search, very easily, for any of my tweets. FriendFeed's search has become much less reliable of late - but it did (though no longer does) let me find my very first tweet https://twitter.com/JoBrodie/statuses/841825864. Oddly, searching now for this no longer works - instead of picking up the original tweet, it picked up only the later search I ran for the tweet in 2009 ;)

B1.4 Finding tweets from a week or two agoI think Google Realtime is probably the best for this, see more details and alternative in the list below. Possibly much more difficult now unless you have tweets fed into FriendFeed. You could also give Topsy a go.

B1.5 Find tweets on a topic or hashtag
Twitter search is good for finding recent ones, just type in your keyword(s) or hashtag. For older ones, use Google Realtime.

B2 Trapping tweets – see also B3.2Freezepage, Chirpstory and Storify will let you capture a tweet. It will remain captured even after deletion, see Freezepage and Storify details above.

B3 Conferences and events: for events organisers and delegates
B3. 1 Follow an event hashtag in real-time, at conference or at home
Twitter itself is pretty good for this - just search for the hashtag and sit back and watch the tweets auto-refresh. Also particularly good is Tweetchat - neither requires you to have an account to watch the tweets. If you have a Twitter account you can sign in to Tweetchat and send tweets that have the hashtag of interest already written in the tweet for you. More info below, and see also Monitter and Twitterfall.

B3.2 Display a Twitterwall in the foyer or the conference roomPrettiest so far seems to be Twitterfountain, ideal for the anteroom display for people to coo over. For a backdrop that people can follow inside the event room I prefer Twitterfall (actually I prefer no Twitterwall but…). See also Guardian Twitterfall.

B3.3 Display event tweets on your website
If you want to bring in the results of a search for an event hashtag you can do that with Twitter Widgets. The 'Search' widget (the options are Profile, Search, Favorites, List) will pull in a hashtag or other keyword but if you worry that cheeky types will introduce rude words into the tag (or just want to show a selection) then you can instead use the 'Favorites' widget and as you favourite each relevant tagged tweet it will show up. This requires more work from you though... See also TweetGrid (listed above) for an option to let people search for anything on Twitter via a widget you can add to your site.

B3.4 Saving an event hashtag for later
Best in show is SearchHash or Twilert but there are a few other tools available too. Twitter's made it a great deal harder for people to download / export tweet archives. The sooner you set these up the better (ie before the event begins) because people often use the hashtag a day or even a week / month or two before an event begins. Chirpstory is great for bagging a run of tweets soon after an event and lets you capture them a page at a time whereas the similar Storify lets you capture them individually but also lets you intersperse commentary. The first two are better for big conferences, the latter are better for small things. Previously everyone used wthashtag (What the hashtag) but it no longer works, and Twapperkeeper only lets you create two free archives.

B3.5 Format tweets for conference output documents, eg blogs Storify is lovely for this – you can embed the final Storify story into most blogs, but to place an individual tweet into a blogpost directly… I need to find out what the tool is that I’ve seen used!

Found one - an example is Blackbird Pie which lets you select a tweet and 'bake' it into your blogpost. I've linked an example of me trying it out in the alphabetic list above. Although I've not tried it I understand the Twitter widget in WordPress will also do this.

Further information on tools mentioned above
The end of Twapperkeeper? (and what to do about it) March 8, 2011, 11:00 am By Mark Samplehttp://chronicle.com/blogs/profhacker/the-end-of-twapperkeeper-and-what-to-do-about-it/31582

Top 10 Websites To Search Old Tweets By YOUNG
February 16, 2011
http://freenuts.com/top-10-websites-to-search-old-tweets/

All the easiest ways to search old tweets August 11, 2010
by DAVE LARSON
http://blog.tweetsmarter.com/twitter-search/10-ways-and-20-features-for-searching-old-tweets/ - covers some of the options listed above, and others I’ve not heard of.

7 ‘Secret’ Ways To Use Twitter Search June 25, 2009
by Thomas @Baekdal
http://www.twitip.com/7-%E2%80%98secret%E2%80%99-ways-to-use-twitter-search/

Other posts by me, on related topics

• Some quick thoughts on Storify as a mini archive for Tweets (4 May 2011)
• Screencap video that I made about Google Realtime (30 April 2011)
• Google Realtime: Finding threaded conversations on Twitter as well as year old tweets (14 April 2011) - features Google Realtime, Storify, Wayback Machine and my failed attempts to find the searchable archive at the Library of Congress
• Trapping tweets for archiving eg at conferences, or curiosity (5 March 2011)

Some quick thoughts on Storify as a mini archive for Tweets

The shortened link for this post is http://is.gd/qYFuwp

EDIT: 30 May 2011 - Have discovered a useful thing that is a huge help in moving tweets (or other items from left to right): double-click on an item in the left and it will snap to the top of the right hand side panel, or use SHIFT double-click to make it snap to the bottom :)

----------------------------------------------------------------------------

Storify (http://storify.com/) is a lovely bit of web-based kit that sort of lets you trap tweets and it appeared on my radar during my mourning period for What the Hashtag and Twapperkeeper. Those two tools would do the work for you, sniffing the tweestream and picking out the hashtagged tweets you'd asked it to save, then presenting them to you (all nicely in order) for export to a text file. Lovely. But these are more or less no longer with us.

Twapperkeeper doesn't let you export an archive of hashtags and will only let you save two archives. What the Hashtag has been taken over by What the Trend and is no longer doing that sort of thing. In fact, according to Twitter's terms of service it seems that it's not possible to do this:

Additionally, and maybe most importantly, Twitter's recent changes to its Terms of Service (TOS) now prevent us from providing transcripts of conversations around a hashtag.

Source: http://wthashtag.com/whatthetrend.com

Storify however, absolutely definitely lets you create an archive of hashtagged tweets, it even lets you embed it in your blog. It's awesome, but the major disadvantage is that you have to manually add each tweet that you want, and you're restricted to a number of tweets (I've not worked out how many yet). It would seem to be at odds with Twitter's TOS (as I understand it from what WTHasthag says above) but perhaps Twitter doesn't seriously envisage anyone using this as an archiving tool because of the amount of effort involved.

I've begun testing Storify to see what I can do with it and will continue to report here :)

1. What happens when a Storified tweet is deleted on Twitter?

Seemingly not much. The tweet I deleted on Monday is still forming a part of the Storify story I created on the Sunday.

2. How many tweets can you access?

Don't know - I have just played around with the #streettalks tweets and it let me go back (you have to keep pressing 'load more' until it stops giving you that option) a day or so - however this was a small meeting in a pub and not a large science conference where tweets tend to ticker tape out at quite a rate.

There is a limit on both the number of tweets you can access and how many days back you can go, and I think these are independent. For example if you were searching for the hashtag #madeuptag but no-one had written a tweet mentioning it for, say, a year or several months, it may well not show up in your tweets, so there is a time thing to be aware of.

While investigating Storify for use by @marilyneb just now we found that her hashtag of interest, #pas2011 could find four days' worth of tweets because it hadn't been used very much. So for capturing low volume hashtaggeed tweets Storify seems to work beautifully.

3. If you can go back to, say, 500 tweets, does stripping out the RTs (eg let's assume 200 of the tweets are RTs) allow you to access an earlier batch of tweets?

Don't know.

4. Transferring tweets from the left hand side panel to the right hand side panel is not that easy on a small laptop with a pointing device rather than a mouse. It's just a bit fiddly. For serious use I think a desktop is a lot easier. I was only half joking when I tweeted the other day that one day we might see people paid to sit in conferences and do precisely this (sweeping tweets as they come in, into a Storify to save them), unless Twitter sorts out its TOS so that third party apps can archive stuff again.

Because it's a bit harder (for me) to move them, what I've done is bung them into a draft Storify - here's what I said:

These are a series of tweets that are not in the order in which they were posted. I just grabbed them from the feed and bunged them here. It will be easier to reorder them if I let them sit for a few hours or at least more than a day.

This is because a newly posted tweet will say it was posted 'x minutes ago' then 'x hours ago' then 'about x hours ago' and finally 23.06 May 4th - and that's when I'll swoop in and reorder them.

Or, what I can do is create a fresh Storify and select from THIS Storify the elements I want to use.

Source: http://storify.com/jobrodie/draft-of

5. If you're archiving tweets, it seems that the avatar of the one on the top of your 'stack' of tweets is used for the Storify's picture. This could be a bit annoying but I haven't investigated it yet - an obvious way to get around this is to add in a tweet of your own and use your avatar... but not ideal.

6. When you search on Storify for a set of tweets, eg a hashtag, the most recent tweet is at the top in the panel on the left. If you want your story to start from the beginning then the trick is (if starting from the top) to drop each tweet on the top of the next one, to reverse the order.

On my computer it's easier to wait 24 hours so that all the tweets are properly timestamped (see point 4) because Storify can be a bit fiddly to use on a very small laptop.

7. I think I might have just spotted a way to get a bit more granularity in Storify. I'm currently creating a record of #streettalks (Harry Rutter's talk on making cities more active-friendly) and am working from a rough draft which has ALL the tweets bunged in from a Twitter search.

While in edit mode, clicking on the S button in the panel on the left (which lets you search within Storify stories) for #streettalks brings up my story and I can drag the entire thing in to the panel on the right. But... I've just discovered that if I click on the Storify result while it's still in the left panel it will unfurl into the entire post, enabling me to pick out the individual elements (ie the tweets). If you hover over the link it does actually say "click to view the elements of this story".

In the pic below I've clicked on the S, highlighted by an orange circle, then typed #streettalks into the search bar and pressed 'get stories'; what shows up is a single result which is my Storify based on the #streettalks hashtag. If I hover over the link it invites me to click to view the elements, which I then did... and in the second panel you can see the entire Storify story unfurled and with all the individual tweets available for adding to the panel on the right to create a new story.

I expect there's more to follow...

Trapping tweets for archiving eg at conferences, or curiosity

I've divided this post into WHAT and HOW. If you're new to archiving tweets or can't think why anyone would bother please read the WHAT first. Hopefully the HOW is self-explanatory ;)

WHAT
One of the most popular search terms that bring people to my blog relates to "finding your old tweets", a topic I've touched on previously in these posts:

• How to find old tweets (5 June 2010)
• TweetNotes - tool for archiving hashtagged tweets at events and conferences etc (24 April 2010) - sadly, Tweetnotes, which looked really interesting, didn't come to anything.
• Following conference hashtag tweets in real time and saving them for later (3 January 2010)
• Cunning use of FriendFeed - find old tweets, yours or others (30 December 2009)
• Watching conference tweets in real time - how best? (21 June 2009)

With the news that a tweet archiving tool called Twapperkeeper is stopping its download and export function from the 20th March I thought I'd look again at some of the ways of storing tweets in the longer term.

Some examples of why you might want to do this include

• you're following what everyone's saying at a conference, by following a hashtag (everyone puts a codeword into their tweets with a # in front of them, the # lets archiving tools pick them out from all the other tweets and present them to you on one page) and you want to make sure you don't miss any tweets, or read them later
• you want to capture @ mentions - eg during a campaign you might want to see how often your company is mentioned
• you want to capture other mentions, eg retweets
• something else I've not thought of

As far as I know, Twitter doesn't treat any of these concepts any differently - pretty much everything on Twitter can be captured by an RSS feed (scroll to the bottom of any page, past all the 'trending topics' and you'll likely see an orange RSS icon - Twitter doesn't care if the RSS feed is made up of tweets from someone's profile or the results from a search.

HOW
1 Toolsa) **NEW** SearchHash has been around since October 2010 but I discovered it on 19 May 2011 - it's extremely good and lets you capture and export significant numbers of hashtagged tweets. It's been created by @lesteph and Dave Briggs. Steph's post introducing it is here and the tool is here - try it out with a hashtag.

b) **NEW** Storify has been around for quite a while but I only heard of it a month ago. Journalist / blogger types were using it to create stories in a very nice way, drawing together several strands from various bits of social media, including Twitter. When I realised that I understood its potential for storing archived tweets - it works very well indeed and is a good interface with a very attractive output, but it's hard work as you have to save each tweet manually. I've previously investigated it here.

c) What The Hashtag has now turned into What the Trend API which is technically beyond me but if you know what you're doing with coding / JSON / stuff like that, give it a go.

What The Hashtag or WTHashtag is a free tool (registration required) that lets you create an archive based on a hashtagged keyword.

During Paul Chamber's trial for sending a threatening message via Twitter people used the hashtag #twitterjoketrial to indicate their lack of warmth for the case coming to court. At the time someone set up WTHashtag to collect all of these together to see how many were generated.

If you look at the page several months after it was used for this purpose you'll see some other tweets that don't appear to be particularly relevant to this case (a couple of tweets are using it to refer to Julian Assange).

To see older tweets you need to scroll below the bar chart window and click on the 'View transcript' tab and choose the relevant date (US style). Here are the #twitterjoketrial tweets from 15 Sep to 25 Sep (note, all days might not appear if the hashtag was invented on, say, the 19th! You'll note that evidently it's missing a tweet or two as the first one listed, at time of writing, is a retweet - where's the original tweet?).

d) Twapperkeeper lets you maintain two free archives, but any more than that and you'll either need to pay ($5-$15 per month depending on whether or not you want a few or up to 25 archives). You can archive a hashtagged event but you can't use the exporting facility to do much with them. I expect you can copy and paste the tweets into some other programme (eg Word) and do something with them that way.

Twapperkeeper

http://twapperkeeper.com/
Currently lets you create an archive of hashtags, your own tweets, tweets mentioning you etc. however this facility's usefulness is going to come to an unfortunate end on 20 March 2011 as no-one will be able to export /download them any more. This is because Twapperkeeper's been asked by Twitter to remove this function so that they fall in line with Twitter's plans for syndication of content. This could well mean that other archiving tools come up against problems. Twapperkeeper have made an open source version of their tool which I've not investigated but the implcation seems to be that this will be a viable alternative.

Incidentally Twapperkeeper has been working closely with JISC and UKOLN - I know that academics working in the area of HCI and Search work on huge Twitter datasets but I would also imagine that academics who communicate using Twitter might want to maintain access to this information too.

One thing to bear in mind is retrospectiveness of the tweets you want to capture. Twapperkeeper has, as far as I can tell, always been able to crawl backwards and access earlier tweets but I'm not sure that WTHashtag can do so - therefore this sort of archive is best set up in advance and monitored, with the content periodically 'tapped off' as a PDF etc. for later reading.

Email alerts for your @mention - see below.

2 Tasks (what you want to save)
@mentions
Finding @mentions retrospectively
I can't think of an easy way to access old @ mentions in an ordered fashion. You can certainly find them via Google, going back to early 2010 but the interface is a bit clunky.

I used this system very succesfully to capture a series of tweets sent in response to a question that Ben Goldacre asked about nerdy days out (it would have been easier to collect if a hashtag had been applied by everyone to their tweet but I didn't think to ask at the time).

This system also lets you set up an email alert for your @mention - I've not tried this and don't know if it will present you with a neat continuous list of your mentions. (Scroll to the end of the page and you'll see the option availalbe).

Example: some tweets sent to or by me in early 2011
By the way here's what the search-results string looks like - someone nerdier than me might be able to URL hack it to get some better results...
http://www.google.co.uk/search?q=%40jobrodie&hl=en&biw=1006&bih=325&prmd=ivns&source=lnms&ei=98tyTablE46Sswav0JyEDg&sa=X&oi=mode_link&ct=mode&cd=9&ved=0CBUQ_AUoCA&prmdo=1&tbm=mbl:1&tbs=mbl:1,mbl_hs:1293840000,mbl_he:1325375999

Setting up an @mention archive prospectively
Ah the benefit of hindsight eh. Go to Twitter (http://twitter.com) type in your @mention of interest, eg @jobrodie, press enter and you'll see a list of tweets with your name. It's probably remarkably similar to what appears in your own @mention page but the difference is that this one has an orange RSS icon on it - click it to view, or right click/copy shortcut to save the URL.

• Search results for @jobrodie
• RSS feed for @jobrodie

What you do with your RSS feed is up to you - you can use any RSS reader (eg Google reader) to read it. Because feeds are regularly updated you'll probably have to keep on top of saving them somewhere (ie, don't just rely on visiting that RSS page - send it somewhere for safekeeping). I use FriendFeed for my tweets.

You might notice that, annoyingly, none of the tweets say who sent them! If you ignore the purple tweets (just duplicates of the blue ones, a quirk of RSS I think) and hover over the blue link you'll be able to see, in your status bar, who sent that tweet.

This search may well bring up some of your tweets that have been retweeted (which you wouldn't ordinarily see in your @mention window unless the retweet was sent manually).

But for belts and braces you might as well see what the search brings up without your @ in front, so here's a search just for jobrodie

{Long post - taking a temporary breather at midnight! Bound to remember some more options after a good night's sleep}