As many MySQL users noticed, MySQL.com and related sites disappeared from the Internet on Wednesday 22 July 2009 for about 10 hours. I’d like to give an update on what happened and what we’ve done to fix it, as such outages are nonacceptable. For this blog entry, I’ve talked to Adam Donnison, senior MySQL.com web developer/admin. Last Wednesday certainly also highlighted the amazing power of Twitter, where we could communicate with our community in real time using our @MySQL and @MySQL_Community identities as well as our personal ones.

So what happened? For the last several years, the MySQL.com servers were being hosted in Uppsala, Sweden where MySQL AB was headquartered before becoming part of Sun. During all this time, despite being the sole hosting location for MySQL, MySQL’s Web Team recorded a near “five-nine” uptime rate over 4 years. As we wanted to get rid of the single point of failure represented by MySQL.com being located in a single facility, we planned on moving all the servers to a new facility in Stockholm, Sweden with high available redundancy in Sun’s datacentres in the United States. Ironically, this was planned for last Saturday, 25 July 2009. As you can probably foresee, our plans changed.

Last week, the building in which the data centre resides in Uppsala, Sweden suffered a major power outage. It lasted long enough to overcome our extensive UPS and then damaged the main building power grid in a way that we could not be confident power would return to stability for days to come (and later, this was proven true). Global internal discussions were held, emails were sent and our Internal IT immediately started the move to the Stockholm Data Centre. Thanks, Ove Ewerlid, Jonathan Petersson, Thorild Selén and Danny Swälas of the MySQL IT team for excellent, focused 24×7 work at short notice!

Meanwhile in Adam’s own words: “As the servers were being moved over to Stockholm, the web team and I decided to finish activating the redundant data centre in the United States. This too was planned for the weekend so I had some work to do. We moved the most recent data sets and files and started our Master/Slave databases there and brought back a read-only version of the MySQL.com sites as quickly as we could.”

Adam continues: “The entire process of provisioning new servers was made easier by moving to the GlassFish Web Stack on our new Sun servers, which provided the key platform elements of the MySQL.com architecture: MySQL, Apache, PHP and Memcached. Instead of adding individual components and making sure they all played nice together, it was a single install and we were able to roll the site across from our existing LAMP stack with no changes to any software.”

Now, to be honest, we did learn a few lessons from our sites going down. Even though we knew about the disaster scenario that could happen in Uppsala and were prepared to move, we were caught unaware. We fully recognize that our external communication could have been better, and that our response could have been a bit faster. Our new disaster plans for the Web and Community sites will have to include such scenarios. Many in the community have given us suggestions on how we could improve our communication at these times and we appreciate and listen to all such feedback.

However, by the end of this last weekend, here’s what we have: Our servers are 100% back up-and-running, now redundant across two continents with data centres which are staffed and equipped with some of the best Sun Servers available. MySQL.com has survived over 10 years and tells the remarkable story of a database that grew to become the most popular open source database in the world. I’m very happy and proud that our web and IT teams have managed to move our web site and all its services so that going forward, we can have the highly available, highly redundant and high performance web site that you have come to expect from us.

Another year and another successful OSCON has been concluded. While I didn’t attend this year’s conference, let me hereby offer some reflections — basing it on reading blogs and talking to attendees both in person and over Twitter (I’m glad to see both the @MySQL and @MySQL_Community Twitter accounts have a large and quickly growing list of followers).

Let me start by highlighting the 2009 Google O’Reilly Open Source Awards. First on the list is Brian Aker, who is recognised as the Best Open Source Database Hacker. He joined MySQL many years ago having not just worked on Apache but also a major developer behind Slashdot. His award he gets for his contributions to MySQL in the past and Drizzle currently. Congratulations to Brian, and I’m sorry I won’t be attending Burning Man with you this year!

I also want to highlight some of the other winners. Evan Prodromou won the award for Best Social Networking Hacker and Clay Johnson who won the Best Community Builder award. Evan Prodromou wrote and runs the open-source microblogging tool Laconica which powers Identi.ca. The Laconica platform runs on MySQL as the database. The same can be said for Sunlight Labs of which Clay Johnson is the Director. Sunlight Labs produces technology to make government in the United States more transparent. Their platform also uses MySQL as a database.

Let me also grab the opportunity to congratulate Bruce Momjian, who was named Database Jedi Master for his work on PostgreSQL!

From what I sensed, highlighted topics of this years OSCON were web applications, cloud computing in addition to what could be labeled “regular applications“. In all of them, data and the web as a data driven operating system (to use Tim O’Reilly’s words from the keynote) is a self evident component, a fact of life. And MySQL continues to be one of the prime movers in this space.

In the MySQL Community team, our charter is to serve the MySQL community — new and old MySQL users alike. One of the ways we do this is by facilitating information exchange between community members, where the new can learn from the old.

And there’s been lots of that information exchange going on, such as over mailing lists (in the early days the dominant vehicle), forums, and Planet MySQL.

One problem with this information exchange has been its ephemeral nature. The same questions pop up for many new users, and should they for some reason not be amongst the issues solved in the MySQL documentation, chances are you’ll have to know quite precisely what you’re looking for when coming up with your Google search phrases. Of course, browsing Planet MySQL is a great way of keeping up to date about how others are using MySQL — but what happened on the Planet six months ago, or a year ago? With currently 16940 (sixteen thousand ninehundred and forty) entries, you won’t by chance pop into the most interesting ones.

Enter the MySQL Librarian. The MySQL Librarian knows what the library contains. Ask the MySQL Librarian for what’s popular, and you’ll be answered.

How does the librarian know? Well, the librarian asks the library visitors (that’s people like you!) for what’s interesting, and whether it’s good.

For more on MySQL Librarian,

• just take a look at it on http://dev.mysql.com/librarian/
• read Giuseppe Maxia’s intro from two weeks ago http://blogs.sun.com/datacharmer/entry/the_mysql_librarian_is_here ,
• read the Dev Zone introductory article http://dev.mysql.com/tech-resources/articles/introducing-librarian.html

or

• read the MySQL Librarian FAQ at http://dev.mysql.com/librarian/faq.html
• read Roberto Galoppini’s review at http://robertogaloppini.net/2009/07/19/the-mysql-librarian-initiative/

Currently, there are 217 entries in it. It contains what our user community think it is the-best-of-the-best from what the community has produced, and it’s tagged and voted for.

Thanks to Dups for coding this, to Giuseppe for leading the efforts and publishing it, and to everyone in the community who has already entered items into the library (at the same time not forgetting those who wrote the items in the first place).

Hence: Go take a look at the MySQL Librarian! Add new items to it (presentations, pictures, videos, articles, and books)! Tag items for easy searching! Vote, so others know what you liked!

As I’ve blogged many times, MySQLers frequently share off-work interests and running is one of them. I’ve also blogged about social media, which usually use MySQL under the hood. Now I’ve combined the two (running and social media) with the insight that running is a religion: I’m propagating Runnism, the Religion of Running.

It started as a thought experiment that I’m now pursuing in what in MySQL AB lingo used to be called my “copious free time” (of which there never was much). I’ve started Twitter accounts for Runnism, one for each language in which I tweet in. This is in sharp contrast to my @kajarno twitter account, where I happily mix Swedish, German and English — with the end result that certain of my Twitter followers have asked me whether I’ve already got the Twitter error message “maximum number of non-English tweets exceeded”.

I’ve so far published three Runnism blog entries in English:

• Runnism, the Religion of Running — why?
• The Seven Legs of Runnism
• Why a world religion? And in 20 languages?

In German, I’ve published the equivalent three blog entries

• Runnismus, die Religion des Laufens — wieso?
• Die Sieben Beine des Runnismus
• Weltreligion? Wieso unbedingt Weltreligion? Und 20 Sprachen?

plus a fourth one

• Ein großes Dankeschön an die anwachsende Runnisten-Gemeinschaft!

which is my thanks note to the German Twitterati. After a filmed eleven minute presentation last Wednesday to 50 German alpha twitterers, I was the happy recipient of plenty of tweets and even a long blog entry explaining “Runnismus, die neue Weltreligion” (”Runnism, the new world religion”).

Interested? Follow me on Twitter as @Runnism in English, @Runnismus in German and/or @Runnismen in Swedish!

The MySQL Runners Club meets whenever running MySQLers meet. Sun Microsystems GmbH here in Germany also has something similar — and a good week ago, a group of Sunnies teamed up for the 31. Münchner Stadtlauf half marathon. I decided to make myself an embedded reporter, combining Twitter, Flip Mino, YouTube and a bit of surplus electronics and Web 2.0 media.

What’s the result? 37 Tweets, an almost 10 minute YouTube video and, surprisingly, a new personal record (1:44:06).

You’ll see a bit more in the video. Sure, both voice and text are in German — but you’ll get most of it just looking and sensing the physical presence. I decided to increase the Web 2.0 ambition level from last year, when I just blogged the event.

So check out the results on YouTube and Twitter:

1. The film: http://www.youtube.com/watch?v=EkR8ap7AoCw
2. The Tweets: http://search.twitter.com/search?q=&from=kajarno&since=2009-06-28&until=2009-06-28&rpp=50

The commentators on YouTube and Twitter seem to like the film. “The orange mass of people resemble the disorder and rush when bringing the cattle down from the mountain pastures”. I hope the film conveys some of the feeling of running a half marathon!

More comments in my German blog entry: Der rasende Reporter mit dem Mittendrin-Video vom Münchner Stadtlauf 28.6.2009.

And — proudly presenting — my certificate with the new PR 1:44:06:

Yesterday, I was invited to hold a presentation at the Berlin Open ‘09 Open Source event. This was related to earlier activities, such as the Open Source Yearbook published by professor Berndt Lutterbeck of the Technical University of Berlin. Professor Lutterbeck was also the main organiser behind Berlin Open.

After some, ehh, logistic challenges (a scenic detour over Alexandersplatz), I got to the Marshall House at Messe Berlin and met with the hosts. A well organised event, with lots of twitterings under http://twitter.com/berlinopen, and plenty of good presentations.

My presentation was on the architectures of social networking using MySQL and myself as examples. I started by making some of my favourite points:

1. Easy wins (by walk-over) are still available for those who mirror their goals and activities into Web 2.0
2. This goes for companies, Open Source projects, developers, researchers, investors, anyone
3. The risk of a disadvantageous Internet presence isn’t mitigated by absence (as the description is then left to others, with their own interests and disinterests)
4. Too much theory at the beginning of such a new phenomenon (as Web 2.0) isn’t beneficial, as it scares us away from Just Doing It — but a too unsystematic approach isn’t good either (so go maintain a decent profile on  Facebook, LinkedIn, Twitter, Slideshare & Co)

I have uploaded the 60-slide presentation (which is partly in German) onto my Slideshare account, which you can access through http://kaj.arno.fi/slideshare. This specific presentation is available at http://www.slideshare.net/kajarno/berlin-open-09-architekturen-sozialer-netzwerke-fallbeispiel-mysql and through clicking on the below picture.

A particular blog entry usually feels relevant and topical when fresh, at least to the author. So let’s say a blog entry even carries some non-zero long-term value. How do you find it after a while? And more importantly, how will your readers find your blog entry?

Descriptive subjects go a long way. But your readers may be searching for “development model” when your header says “release plan”. And even if you anticipate the search words used by your readers, you can only pick one wording for your header.

Full-text search also helps. There’s now a brand new Search field in the top left corner of Planet MySQL. Chances are you’ll find what you look for, no matter if search for “Chinese”, “DRBD”, “development” or “PHP”. You may even search for several words, such as “Chinese, UTF”.

Easy searchability calls for yet a bit more, namely tagging. Tags are a great way for the author (or for the reader!) to underline key aspects of a blog post (or Flickr picture or Slideshare presentation or any item you post on the Web, of course).

And next to searching, tagging is the second area where PlanetMySQL has lately been improved, by Dups, as announced in his blog entry last week.

Four key points:

1. Tags are displayed under the blog header. We import the tags originally written by the blogger.
2. When you click on the tags, you’ll find blog posts with that tag across individual blogs. Example: http://planet.mysql.com/?tag_search=169 has all PHP tagged posts, on any blog aggregated to Planet MySQL.
3. You can also search for (multiple) tags in the search field. Example: “Tag: PHP, development”.
4. The community may edit tags (when logged in to MySQL.com). The wisdom of individual community members can thus be shared by others.

For more, go read Dups’s blog!

P.S. UPDATE: I got a question on what the numbers in the picture are. They’re “numeric tags” in the Firefox browser, as appearing with the “Mouseless Browsing” add-on. They enable me to “click” on links from the keyboard without moving my hand to the mouse.

Sometimes I miss the coding I did last century. Today I was reminded of some of the fun, when I had set my mind to doing some statistics on my blogging.

In a blog entry on http://blogs.arno.fi/isit/2009/05/14/home-made-blog-statistics-from-wordpress-mu/ I describe what I did.

The blog entry may be of interest for those who use WordPress and are set back by the huge amounts of tables it generates. I happen to host 18 blogs and with each blog requiring 8 tables, that’s a total of 144 tables. Add the 9 top-level blogs and I’ve got 153 tables to navigate.

The blog entry I wrote 

• identifies the key fields
• shows how to do stats on individual blogs
• creates a statistics table into which I aggregate relevant entries from individual blog tables
• uses SELECT to generate SELECT statements
• and ends up with some statistics on the 253 blog entries in my WordPress.

I also studied the wp_n_comments tables, and came to the conclusion that I’ll need to use some global DELETEs to clean up spam comments that have found their way to my site.

I found out that, strangely enough, I haven’t approved a single comment that includes the character string “viagra”

select comment_approved,count(*)
 from wp_4_comments
 where comment_content like "%viagra%"
 group by comment_approved;

The same applies to “cialis“.

So I issued

delete from wp_4_comments 
 where comment_content like "%cialis%";

delete from wp_4_comments
 where comment_content like "%viagra%";

but I still have quite a bit of cleaning up to do, since despite deleting first 759 cialis entries and then 411 viagra entries, I still have 3683 unapproved comments to clean up (and I suspect there are less than 10 real comments that have slipped my attention in my inbox, when I’ve got notification of them).

At any rate, I got my statistics and had more efficient and fun (albeit incomplete) spam cleaning than ever before!

On my non-MySQL blog, I just wrote a blog post called “Photo Manager: How do you keep track of your pictures?“. I’m looking for Open Source Software that helps me keep track of my 100,000+ digital pictures. I wrote specs on dreamt-up software called “Robfat” (for rename, order, backup, find, archive, tag) as I want to remove excess fat from my HDs (and CD/DVD cabinets).

And then I thought this may have a MySQL dimension:

What if we had an “EXIF Storage Engine”? What if we could update EXIF tags directly from the MySQL command line, with UPDATE statements, and SELECT picture file names based on tags and other file characteristics?

But beyond this technical aspect, I think many, many MySQL users are avid photographers and may have input about the specs themselves, or even know of existing software that satisfied the needs.

So please go to the blog post and comment!

Travelling to Hongkong and Taipei has made such an impression on me, that I couldn’t help but add two new blogs to my homepage kaj.arno.fi:

• http://blogs.arno.fi/guanxi/ is (or pretends to be) in Simplified Chinese, written on the Chinese mainland
• http://blogs.arno.fi/yilingyi/ is (or pretends to be) in Traditional Chinese, written here in Taiwan

Guanxi means “relations”, as in “Community Relations”. It’s also a very common word describing how to get things done in China. It even has its own English language Wikipedia entry.

Yi-ling-yi means one-oh-one, as in Taipei 101. This number sequence also means “special” in Chinese. Taipei 101 is the world’s highest completed skyscraper. Needless to say, that merits another Wikipedia entry, not to mention a blog entry in Traditional Chinese.

Tomorrow, I’ll make another attempt at giving a short MySQL speech in Mandarin. Well, technically speaking, it’s already today, as it’s 3:30am in Taipei. My internal clock has gone awry.