Technorati Blog

We’re asked all the time how platforms like Twitter and Facebook are impacting the Blogosphere. This is what we’re seeing: Twitter is not replacing blogs, but it has evolved as a major awareness vehicle for bloggers and people who read blogs (same goes for Facebook status updates).

So the idea behind Twittorati was to allow people to view the topical geist that the most influential bloggers are tweeting and blogging about.

We say this is where the Blogosphere meets the Twittersphere, but what does that mean? Twittorati shows what top bloggers are tweeting about, and how these trends compare to Blogosphere trends. You’ll be able to filter tweets by topic, see the most tweeted blog posts, and compare leading Blogosphere and twitter trends.

We’re featuring the Technorati Top 100 Bloggers at launch, but we’re going to expand very soon to include the many more authors in the active Blogosphere. You can find the Twitter content most relevant to them, compare the day’s top blog and hash tags to see the hottest topics in both spheres, as well as see the most popular links that are being tweeted and which blogs are linking to them. You can also really dig into the information source: writer pages display each tweeter’s blogs and Twitter information and Technorati Authority.

Please reach out and let us know what you think
… And special thanks to Sawhorse Media, publisher of Muckrack.com and VentureMaven.com, who helped us produce the site.

It’s been a tumultuous month. For us – and unfortunately for you.

We’re sorry for the frustration and the less than great service. While not an excuse, here is the reason:

We changed co-location facilities. We, well the four members of our ops team, moved 1,000 servers and ALL of our real-time search infrastructure. Things are now returning to normal, although slower than we would have liked.

Looking forward:
We haven’t lost any data – so when things are back to normal your link counts and authority will be exactly where they should be.

We’ll follow up in more detail so you have information on exactly which services have been affected and how, and when they’ll be completely functional again.

Our colo move from one end of San Francisco to the other was completed this past Monday. We have successfully moved nearly 900 machines while reducing our overall rack footprint by more than 50%. This major reduction will help us manage our systems better, keep things running better, save a ton of money, and, as an added benefit, consume a lot less power, thus being a wee bit more friendly to the planet.

We had a hard deadline and only 5 weeks to complete the move. We moved quickly and as cautiously as possible. We did have outages and I apologize for any inconvenience that may have caused you, our Technorati users. We still have cleanup, networking to optimize, systems to stabilize, and the routine work of running a real-time blog search engine and ad platform, but things are calming down. This move makes great economic sense for us going forward and the pain of the last few weeks, though unpleasant, was worth enduring since we are now much better positioned for success going forward.

I want to thank the Technorati Operations team for their die hard dedication, positive attitude, and professionalism. I'd also like to thank all of you for sticking with us. We look forward to serving you, the bloggers, readers, and advertisers, for a long, long time.

We have moved over 2/3 of the servers and have all systems enabled at this time. Performance has crept back down to much more acceptable levels, but not where we want it to be nor where it was before we started the move. We will continue to optimize the configuration of the servers and network.

You can see some of our progress, and some of what this ordeal has been like, here:

GrapPERF for Technorati

Separately, we had a few hosts fail over the weekend, which is unfortunate but to be expected when they get moved. Those have been rebooted or replaced and, as of this writing, search and tags are working, though a bit slow.

What's not working so well

A lot of you on Twitter have said that "Technorati has forgotten my blog". We haven't, but our blog display data servers are running very hot and not always returning data. Unfortunately, this particular error is not handled as we would like in the front-end and, instead of saying there was an error, it concludes that the blog doesn't exist. Our mistake, not yours. We'll be fixing this, but in the meantime, try a reload and sorry for the inconvenience.

Blog claiming continues to be problematic. This is more the result of our new crawler's diligence in not letting potential spam or non-blogs into the index than the result of the move, though the move has aggravated things. We are trying to balance content review policies for new sites that people are trying to claim with our desire to keep spam out. Some of what we are looking for in a blog can be found in the Technorati Blog Quality Guidelines.

Priorities

1. Stability and performance of the site.
2. Making sure blog claiming works.
3. Continued ping processing and indexing

There are engineers working on all of these and I'll keep you posted on our progress.

The Technorati Attention Index: These are the top sites with highest number of blogs linking to them in the past 30 days. This time around, in addition to rank, we've added Attention numbers. Attention is the number of blogs (not the number of links) that have linked to the site in the past 30 days. Here are the mainstream media gainers and losers in the blogosphere:

New to the top 50

The Dallas Morning News

San Jose Mercury News

Star Tribune

Out of the top 50

US News & World Report

Rolling Stone

Christian Science Monitor

International Herald Tribune (now part of nytimes.com)

5 biggest gains in Rank

PBS

The Houston Chronicle

Google News

NY Post

Slate

5 biggest losses in Rank

The Economist

Chicago Tribune

The White House

Financial Times

Newsweek

5 biggest gains in attention

YouTube

The Wall Street Journal

CNN

LA Times

Wired

5 biggest losses in attention

Reuters

Telegraph.co.uk

The Boston Globe

Financial Times

The Economist

Overall Rankings and Attention
 
1.   YouTube 60,644
2.   The New York Times 17,374
3.   guardian.co.uk 8,039
4.   The Wall Street Journal ­7,513
5.   The Washington Post 6,891
6.   CNN 6,330
7.   Telegraph.co.uk ­5,380
8.   Yahoo! News 5,070
9.   MSNBC 5,036
10. The Los Angeles Times 4,536
11. Reuters ­4,314
12. FOX News 4,001
13. The Boston Globe 3,838
14. USA Today 3,619
15. Daily Mail 3,530
16. Time 3,524
17. BBC News 3,399
18. NPR 3,189
19. NY Daily News 2,588
20. Forbes 2,534
21. San Francisco Chronicle 2,420
22. Slate 2,187
23. CBS News 2,156
24. Google News 2,093
25. Wired 2,062
26. Financial Times 2,056
27. PBS 2,053
28. NY Post 2,025
29. San Francisco Examiner 1,968
30. BusinessWeek 1,949
31. The White House 1,929
32. Salon 1,928
33. Chicago Tribune 1,924
34. Newsweek 1,880
35. CNNMoney 1,712
36. CBC 1,696
37. Yahoo! Finance 1,642
38. The Economist 1,565
39. New York Magazine 1,550
40. philly.com 1,288 
41. The Houston Chronicle 1,120
42. Science Daily 1,093
43. MarketWatch 1,076
44. People 1,066
45. Miami Herald 1,049
46. The Seattle Times 1,049
47. Yahoo! Sports 1,047
48. The Dallas Morning News 939
49. San Jose Mercury News 879
50. Star Tribune 877

Yesterday we completed 3 move phases, relocating about 60 machines. In particular we have moved the vast majority of our front end web farm and the back end search and crawling infrastructure. The team has been working non-stop to execute our adjusted plan.

If you have been trying to use the site in the last 24 hours, you have noticed we've had our difficulties. Despite rigorous configuration test scenarios and checks prior to the move, some of our internal network components failed to do what they were supposed to. This meant that, although everything we moved was up and running, requests coming from the internet were not getting everything they needed to complete a meaningful page of HTML.

You may have noticed that your claimed blogs didn't show up any more or that we seem to have forgotten that your favorite blogs exist. Believe, they are all safe and nothing has been lost.

The ops team has rerouted various pieces to start bringing the site online again and full results are now beginning to show up. Unfortunately, you may still see monster pages or slow responses.

We appreciate your patience and I'd like to say thank you to the team for their hard work and ingenuity. They are a small group dedicated to keeping Technorati running smoothly. A move of this magnitude is always a challenge and they are doing everything they can to get it done quickly and return all services to normalcy.

We've been running for about a week with our web server farm split between the two colo facilities. This approach has definitely made it difficult to guarantee fast or solid page loads.

This afternoon we will be moving a major portion of the web server farm so that it resides in the same facility as the search backend. The servers we have already moved are performing quite well and we expect that this will return most services to normal.

We appreciate your patience as we are working as fast as we can on getting Technorati working 100% again.

Support Blog

Our move between colos continues at full pace. We are working hard to keep services available but there will be outages, both scheduled and unscheduled. We apologize for the inconvenience and understand downtime makes it difficult to use the site.

You can keep up with more detailed updates on the Support Blog.

As I said in Colo move is in full swing now, we will be having regularly scheduled maintenance windows while we move from our old colo to our new colo.

We are experiencing some slowness over our network between the two colos which has caused a backlog in ping processing.

Rather than have this drag on for several days, we have opted to accelerate the move of the major parts of our crawling infrastructure. To make this go quickly, we will probably have to turn crawling and blog claiming off for several hours tomorrow (Friday May 1).

No data will be lost as we will simply queue the requests for later processing.

You can keep track of our updates in the Site status report.

We've been preparing for our colo move for the last several months. In fact, we're moving two colos at the same time. We expect to be completed prior to the end of May.

Since we'll be moving more than 600 servers that make up the 30+ sub-systems that drive Technorati, Blogcritics and AdEngage, there is a ton to coordinate. Our systems have been designed from the ground up with high availability and redundancy in mind. That being said, there are still a lot of moving parts and things don't always go quite as planned. We expect there will be planned outages of some systems and, unfortunately, there may be some unplanned outages as well. We'll keep you posted if things go bump in the night, but, rest assured, we'll keep your blog data safe. Our backup servers have already been moved successfully.

For the latest on the move or system status, please consult the Site status report in the right sidebar of our support site.