As many MySQL users noticed, MySQL.com and related sites disappeared from the Internet on Wednesday 22 July 2009 for about 10 hours. I’d like to give an update on what happened and what we’ve done to fix it, as such outages are nonacceptable. For this blog entry, I’ve talked to Adam Donnison, senior MySQL.com web developer/admin. Last Wednesday certainly also highlighted the amazing power of Twitter, where we could communicate with our community in real time using our @MySQL and @MySQL_Community identities as well as our personal ones.

So what happened? For the last several years, the MySQL.com servers were being hosted in Uppsala, Sweden where MySQL AB was headquartered before becoming part of Sun. During all this time, despite being the sole hosting location for MySQL, MySQL’s Web Team recorded a near “five-nine” uptime rate over 4 years. As we wanted to get rid of the single point of failure represented by MySQL.com being located in a single facility, we planned on moving all the servers to a new facility in Stockholm, Sweden with high available redundancy in Sun’s datacentres in the United States. Ironically, this was planned for last Saturday, 25 July 2009. As you can probably foresee, our plans changed.

Last week, the building in which the data centre resides in Uppsala, Sweden suffered a major power outage. It lasted long enough to overcome our extensive UPS and then damaged the main building power grid in a way that we could not be confident power would return to stability for days to come (and later, this was proven true). Global internal discussions were held, emails were sent and our Internal IT immediately started the move to the Stockholm Data Centre. Thanks, Ove Ewerlid, Jonathan Petersson, Thorild Selén and Danny Swälas of the MySQL IT team for excellent, focused 24×7 work at short notice!

Meanwhile in Adam’s own words: “As the servers were being moved over to Stockholm, the web team and I decided to finish activating the redundant data centre in the United States. This too was planned for the weekend so I had some work to do. We moved the most recent data sets and files and started our Master/Slave databases there and brought back a read-only version of the MySQL.com sites as quickly as we could.”

Adam continues: “The entire process of provisioning new servers was made easier by moving to the GlassFish Web Stack on our new Sun servers, which provided the key platform elements of the MySQL.com architecture: MySQL, Apache, PHP and Memcached. Instead of adding individual components and making sure they all played nice together, it was a single install and we were able to roll the site across from our existing LAMP stack with no changes to any software.”

Now, to be honest, we did learn a few lessons from our sites going down. Even though we knew about the disaster scenario that could happen in Uppsala and were prepared to move, we were caught unaware. We fully recognize that our external communication could have been better, and that our response could have been a bit faster. Our new disaster plans for the Web and Community sites will have to include such scenarios. Many in the community have given us suggestions on how we could improve our communication at these times and we appreciate and listen to all such feedback.

However, by the end of this last weekend, here’s what we have: Our servers are 100% back up-and-running, now redundant across two continents with data centres which are staffed and equipped with some of the best Sun Servers available. MySQL.com has survived over 10 years and tells the remarkable story of a database that grew to become the most popular open source database in the world. I’m very happy and proud that our web and IT teams have managed to move our web site and all its services so that going forward, we can have the highly available, highly redundant and high performance web site that you have come to expect from us.

Another year and another successful OSCON has been concluded. While I didn’t attend this year’s conference, let me hereby offer some reflections — basing it on reading blogs and talking to attendees both in person and over Twitter (I’m glad to see both the @MySQL and @MySQL_Community Twitter accounts have a large and quickly growing list of followers).

Let me start by highlighting the 2009 Google O’Reilly Open Source Awards. First on the list is Brian Aker, who is recognised as the Best Open Source Database Hacker. He joined MySQL many years ago having not just worked on Apache but also a major developer behind Slashdot. His award he gets for his contributions to MySQL in the past and Drizzle currently. Congratulations to Brian, and I’m sorry I won’t be attending Burning Man with you this year!

I also want to highlight some of the other winners. Evan Prodromou won the award for Best Social Networking Hacker and Clay Johnson who won the Best Community Builder award. Evan Prodromou wrote and runs the open-source microblogging tool Laconica which powers Identi.ca. The Laconica platform runs on MySQL as the database. The same can be said for Sunlight Labs of which Clay Johnson is the Director. Sunlight Labs produces technology to make government in the United States more transparent. Their platform also uses MySQL as a database.

Let me also grab the opportunity to congratulate Bruce Momjian, who was named Database Jedi Master for his work on PostgreSQL!

From what I sensed, highlighted topics of this years OSCON were web applications, cloud computing in addition to what could be labeled “regular applications“. In all of them, data and the web as a data driven operating system (to use Tim O’Reilly’s words from the keynote) is a self evident component, a fact of life. And MySQL continues to be one of the prime movers in this space.

Yesterday, I was invited to hold a presentation at the Berlin Open ‘09 Open Source event. This was related to earlier activities, such as the Open Source Yearbook published by professor Berndt Lutterbeck of the Technical University of Berlin. Professor Lutterbeck was also the main organiser behind Berlin Open.

After some, ehh, logistic challenges (a scenic detour over Alexandersplatz), I got to the Marshall House at Messe Berlin and met with the hosts. A well organised event, with lots of twitterings under http://twitter.com/berlinopen, and plenty of good presentations.

My presentation was on the architectures of social networking using MySQL and myself as examples. I started by making some of my favourite points:

1. Easy wins (by walk-over) are still available for those who mirror their goals and activities into Web 2.0
2. This goes for companies, Open Source projects, developers, researchers, investors, anyone
3. The risk of a disadvantageous Internet presence isn’t mitigated by absence (as the description is then left to others, with their own interests and disinterests)
4. Too much theory at the beginning of such a new phenomenon (as Web 2.0) isn’t beneficial, as it scares us away from Just Doing It — but a too unsystematic approach isn’t good either (so go maintain a decent profile on  Facebook, LinkedIn, Twitter, Slideshare & Co)

I have uploaded the 60-slide presentation (which is partly in German) onto my Slideshare account, which you can access through http://kaj.arno.fi/slideshare. This specific presentation is available at http://www.slideshare.net/kajarno/berlin-open-09-architekturen-sozialer-netzwerke-fallbeispiel-mysql and through clicking on the below picture.

A particular blog entry usually feels relevant and topical when fresh, at least to the author. So let’s say a blog entry even carries some non-zero long-term value. How do you find it after a while? And more importantly, how will your readers find your blog entry?

Descriptive subjects go a long way. But your readers may be searching for “development model” when your header says “release plan”. And even if you anticipate the search words used by your readers, you can only pick one wording for your header.

Full-text search also helps. There’s now a brand new Search field in the top left corner of Planet MySQL. Chances are you’ll find what you look for, no matter if search for “Chinese”, “DRBD”, “development” or “PHP”. You may even search for several words, such as “Chinese, UTF”.

Easy searchability calls for yet a bit more, namely tagging. Tags are a great way for the author (or for the reader!) to underline key aspects of a blog post (or Flickr picture or Slideshare presentation or any item you post on the Web, of course).

And next to searching, tagging is the second area where PlanetMySQL has lately been improved, by Dups, as announced in his blog entry last week.

Four key points:

1. Tags are displayed under the blog header. We import the tags originally written by the blogger.
2. When you click on the tags, you’ll find blog posts with that tag across individual blogs. Example: http://planet.mysql.com/?tag_search=169 has all PHP tagged posts, on any blog aggregated to Planet MySQL.
3. You can also search for (multiple) tags in the search field. Example: “Tag: PHP, development”.
4. The community may edit tags (when logged in to MySQL.com). The wisdom of individual community members can thus be shared by others.

For more, go read Dups’s blog!

P.S. UPDATE: I got a question on what the numbers in the picture are. They’re “numeric tags” in the Firefox browser, as appearing with the “Mouseless Browsing” add-on. They enable me to “click” on links from the keyboard without moving my hand to the mouse.

Sometimes I miss the coding I did last century. Today I was reminded of some of the fun, when I had set my mind to doing some statistics on my blogging.

In a blog entry on http://blogs.arno.fi/isit/2009/05/14/home-made-blog-statistics-from-wordpress-mu/ I describe what I did.

The blog entry may be of interest for those who use WordPress and are set back by the huge amounts of tables it generates. I happen to host 18 blogs and with each blog requiring 8 tables, that’s a total of 144 tables. Add the 9 top-level blogs and I’ve got 153 tables to navigate.

The blog entry I wrote 

• identifies the key fields
• shows how to do stats on individual blogs
• creates a statistics table into which I aggregate relevant entries from individual blog tables
• uses SELECT to generate SELECT statements
• and ends up with some statistics on the 253 blog entries in my WordPress.

I also studied the wp_n_comments tables, and came to the conclusion that I’ll need to use some global DELETEs to clean up spam comments that have found their way to my site.

I found out that, strangely enough, I haven’t approved a single comment that includes the character string “viagra”

select comment_approved,count(*)
 from wp_4_comments
 where comment_content like "%viagra%"
 group by comment_approved;

The same applies to “cialis“.

So I issued

delete from wp_4_comments 
 where comment_content like "%cialis%";

delete from wp_4_comments
 where comment_content like "%viagra%";

but I still have quite a bit of cleaning up to do, since despite deleting first 759 cialis entries and then 411 viagra entries, I still have 3683 unapproved comments to clean up (and I suspect there are less than 10 real comments that have slipped my attention in my inbox, when I’ve got notification of them).

At any rate, I got my statistics and had more efficient and fun (albeit incomplete) spam cleaning than ever before!

On my non-MySQL blog, I just wrote a blog post called “Photo Manager: How do you keep track of your pictures?“. I’m looking for Open Source Software that helps me keep track of my 100,000+ digital pictures. I wrote specs on dreamt-up software called “Robfat” (for rename, order, backup, find, archive, tag) as I want to remove excess fat from my HDs (and CD/DVD cabinets).

And then I thought this may have a MySQL dimension:

What if we had an “EXIF Storage Engine”? What if we could update EXIF tags directly from the MySQL command line, with UPDATE statements, and SELECT picture file names based on tags and other file characteristics?

But beyond this technical aspect, I think many, many MySQL users are avid photographers and may have input about the specs themselves, or even know of existing software that satisfied the needs.

So please go to the blog post and comment!

Just before the MySQL User Conference, Dups implemented a small little feature for Planet MySQL: “voting”. We wanted to see what a voting system might mean to you, our PlanetMySQL readers.

The question is now how to improve the voting mechanism to make it more useful. The goal is for everyone to see what you and your peers think are the best PlanetMySQL entries over a given week.

Here are some of the options:

1. Open up voting to everyone regardless of whether they are logged in or not. Currently you can only vote if you are logged in with a MySQL.com account. Let’s face it, a login gives a barrier to entry even as much as it gives us the security of knowing we won’t get spammed.

One option is to allow all to vote, within the constraints of spam control of some kind. Another option is to have voting for non-logged in users to count, but count less than those who login. This gives voters the incentive to login and magnify their voice, but allows anyone to have a vote if they choose not to login. An example would be to have a logged in vote worth 1 and a non-logged in vote worth 0.1

The problem, of course, is that we would be setting an artificial “quality” judgement on your vote.

2. We add voting links into the RSS Feed itself, so you can tell the world what you think of blog posts from within your RSS reader. We could also automate a posting of the Top voted entries on PlanetMySQL at the end of each week in case you decide not to come to the web site at all (though we hope you do come by once in a while).

3. We leave it as is!

Now it’s your turn to tell us what you think! Remember the intent of all this is for good posts to not just disappear as the feeds scroll by with ever more content.

We’ve also been asked whether there is any conspiracy to throw posts up and down by us manipulating votes. Let me answer categorically: No. We do not manipulate the votes in any way. What you vote is what you see.

A new start always provides an opportunity to reassess your way of doing things:

• “Am I doing the right things with my life?”
• “Which habits could I change — in order to be a better neighbour?” 

Since Karen Tegan Padir took over the MySQL product from Mårten Mickos, we at Sun have taken a thorough look in the mirror.  The result of this introspection allows us to publish our conclusions and new community commitments this week in conjunction with the MySQL Conference & Expo in Santa Clara. 

For those who don’t yet know Karen, let me state a few facts: 

• Karen is VP of Sun’s newly-founded MySQL & Software Infrastructure group
• She describes herself as a geek and a straight-shooter, and I can attest to both 
• She was deeply involved in Sun’s decision to acquire MySQL, as well as the subsequent integration work throughout 2008 

In my previous blog, ”MySQL Culture and Business Philosophy Goes Mainstream at Sun“, I stressed that the key point to remember is that Sun is completely committed to building a big open source-based business, and very much supportive of the various communities that Sun is engaged in. That may sound a bit abstract, so I have worked with my new boss, Karen, to spell out for the MySQL community what it actually means in practice: 

First: We shall now start releasing MySQL Community Server binaries as frequently as we release the MySQL Enterprise Server. We want everyone – community and customers — to get the best bits first from us. This is why future Monthly Rapid Updates of MySQL 5.1 will remain available for the community. Moreover, future MRUs of MySQL 5.0 will also become available for the community. 

Second: We shall focus even more on our traditional core product values of stability, Performance and Ease of Use. Like you, we hate bugs, we hate slow, we hate waiting for fixes, and we hate awkward usability — even more than we love new functionality. The MySQL 5.4 performance release, which is considerably faster than MySQL 5.0 or 5.1 in most use cases, is a case in point.

Third: We shall allocate additional resources to the health and well-being of the MySQL Community. We focus both on users and developers. We shall now devote a greater portion of our internal MySQL Engineering Team resources to reviewing and eventually merging architecturally-compatible features written by external contributors, even if those features weren’t on our own roadmap.

Finally, fourth: We shall do more to improve our internal software engineering practices. We will brush our teeth every morning and evening, exercise several times a week, and eat (mostly) healthy food. Some of us may at times still take a vodka shot, but if so, then only to be social and to provide an excuse for singing badly, but not while writing code. 

In the spirit of open-ness, please give us ideas on how we can implement these changes. Let the MySQL community team know what you think. For those at the show this week, stop Karen in the halls and introduce yourself. Above all, enjoy!

Jetlagged from transatlantic travel, I woke up in the middle of the Californian night thinking about what has changed since I arrived at the MySQL Conference in Santa Clara on Sunday evening. I was pondering all the questions MySQL users and Sun colleagues were asking at the event, and what the user base was thinking out loud on Twitter yesterday.

What has changed is obviously that Sun Microsystems and Oracle announced they have entered into a definitive agreement under which Oracle will acquire Sun.

What further changes we will see as a result of that is a different story. Evidently, I don’t sit in with a crystal ball predicting what will happen next. Nor do I have insight into Oracle’s plans for MySQL, once the deal is closed. Nor am I even in a position to comment upon the acquisition, so I won’t do it.

However, what I do know and what I can say is what has not changed with MySQL:

1. There still is a huge base of MySQL users out there. They have economic interests that are independent of whoever owns MySQL. The users in the MySQL community come in all flavors, ranging from casual users to those who intimately know the inner workings of MySQL and have contributed to the code base.
2. There still is a huge talent pool of MySQL experts in Sun Microsystems, in Support, in Consulting, in Training, in Engineering, in other parts of Sun. They have a strong loyalty towards the MySQL users they have served over many years.
3. MySQL is still licensed under the GPL. The GPL license used to form a safety net for the users not certain about whether MySQL AB would follow the spirit of Open Source. It continued to be so with Sun Microsystems. And the Open Source license continues to provide a safety net for its user base, regardless of the owner of MySQL.
4. MySQL has founders, one in particular, who still haven’t fallen off the face of the planet. Moreover, their passion for MySQL and its users continues.
5. Sun Microsystems still is a separate legal entity, practising what’s known as “business as usual“. This is familiar to MySQLers from the time between Sun’s acquisition of MySQL was announced mid-January 2008 to the closing at the end of February 2008. During the period between announcement and closing, we continue to behave as separate entities, even competing with each other.
6. Part of Business as Usual is a number of product announcements at the MySQL Conference this week. I’m looking forward to these!

While I cannot and will not personally speculate about what happens next, nor about Oracle’s intentions with MySQL, I think our users are looking to what the names most inimately associated with MySQL are saying — even if they no longer work for Sun Microsystems:

• First, Mårten Mickos, MySQL AB’s former CEO and long-time SVP at Sun, has several positive comments in his Forbes interview “Why Oracle Won’t Kill MySQL”.
• Second, Michael “Monty” Widenius, MySQL AB’s co-founder, also finds many positive things to say in his blog statement “To be (free) or not to be (free)“.

My humble suggestions: Keep using MySQL! Follow the announcements from the MySQL Conference this week! Keep helping each other within the MySQL community!

Go MySQL!

The last open keynote slot in the MySQL Conference, Thursday 10:00am, is now filled with the keynoter we had in mind all the time: Sun co-founder Andy Bechtolsheim.

Andy’s bio is one of the longest and most impressive of MySQL Conference keynoters ever:

• Sun co-founder, employee number one
• invented the “Stanford University Network workstation” that eventually became the Sun-1 Workstation
• was instrumental in launching other successful Sun products, including the SparcStation 1
• now works with Sun’s Systems Group to help drive next generation X64 and storage servers product architecture as well as HPC opportunities
• left Sun in 1994 and rejoined 2005 through Sun acquiring his company Kealia
• was one of the first investors in Google

What will Andy talk about? “The Solid State Storage Revolution” is the title. For more details, come to the conference!

But before that, I’d encourage you to take a closer look at his bio

• Sun Executive Bio / Andy Bechtolsheim: http://www.sun.com/aboutsun/media/ceo/bio.jsp?name=Andy%20Bechtolsheim
• Wikipedia in English: http://en.wikipedia.org/wiki/Andy_Bechtolsheim
• Wikipedia in German: http://de.wikipedia.org/wiki/Andreas_von_Bechtolsheim