I suspect that few of the readers of my blog have missed it, but MySQL 5.1 is GA. And I suspect it was no surprise to my readers, nor those of Giuseppe, the Community Team Lead. Nor those who read Sheeri’s blog, with the picture of Dups unmistakably setting GA date expectations at the OpenSQL Camp a number of weeks ago.

In addition to our communication, there has been some, ehmm, complementary communication on MySQL 5.1 GA, its history, its quality, and its timing. Some of that may have been a tad one-sided, painting the world in a bit darker colours than absolutely needed.

As an outside observer, it may be hard to read properly between the lines, and interpret what’s really going on behind the scenes. Now, Giuseppe has made all that quite a bit easier with his fresh posting.

With the delivery of MySQL 5.1 GA, our release policy sees a couple of changes in relation to that of MySQL 5.0.

First, initially we will deliver both MySQL Enterprise Server and MySQL Community Server in parallel. The bug fixes to the initial MySQL 5.1 GA releases will thus be delivered also in binary form for both MySQL Community Server and MySQL Enterprise Server.

Second, after a period of time, we will revert to the different release schedule for MySQL Enterprise Server and MySQL Community Server that we have been using. MySQL Enterprise Server will have more frequent binary releases. However, at this point, we do not have an exact date or a release number, when the first Enterprise only release will happen.

Third, we have done away with odd and even numbering. Whenever MySQL Community Server is released, it will carry the same release number as the corresponding MySQL Enterprise Server.

Fourth, MySQL Enterprise Server comes in more flavours than the MySQL Community Server. This will allow us to offer different optional pricing levels for our commercial subscriptions — where a customer will only have to pay extra for MySQL Enterprise support if he/she plans to use the higher-level features. Hence, MySQL Enterprise Server “Advanced” includes support for partitioning. However, do note that the MySQL Community Server provides the full partitioning functionality.

This blog entry is about a specific storage engine in MySQL. The Federated storage engine enables data to be accessed from a remote MySQL database on a local server without using replication or cluster technology. When using a Federated table, queries on the local server are automatically executed on the remote (federated) tables. No data is stored on the local tables.

When we released MySQL 5.1.24, the Federated engine was not compiled in, pending decisions on our future steps. The reason for the removal was that we realised (albeit quite late in the game) that Federated has some bugs that expose the server to unnecessary risks. Fixing these bugs is a time consuming process, because the root cause lies in the design of the Federated engine.

The removal was a safety precaution, which made the server more secure. However, it also deprived some users of an engine that they had been using for some time (Federated was introduced in MySQL 5.0.3).

Thus, we were left with the dilemma of more security versus more features. After much internal discussion, we reached a compromise. In 5.1.25, we reintroduced Federated as it was, but in the meantime we prepared a change for 5.1.26 which was just released. Federated is now compiled in, but disabled by default. This means that normal users won’t be exposed to the possible side effects of using Federated tables. Users who require the Federated engine will be able to use it, by adding an option (–federated) to the mysqld command line or to the configuration file. Existing users of the Federated engine must be warned that using Federated can be risky, and it is not recommended for production.

We have a list of outstanding bugs affecting the Federated engine in our Bugs database.

Notice that the 5.0.x server is not affected by this decision. However, to allow security conscious users to disable Federated, we plan to introduce a similar configuration option in 5.0.66 and later releases.

We realise that the situation with Federated is undesirable. Therefore, we plan to replace Federated with a better designed, more robust engine, and we will welcome feedback about this task from the community and from our customers.

And as followers of MySQL Forge know (thanks Brian Aker for reminding me), there is already an initiative from the community, called FederatedX:

FederatedX Pluggable Storage Engine for MySQL is based off of the Federated Storage Engine, and is an attempt to moved the Federated Storage Engine forward to fix bugs, add new features and develop new concepts that are easier to achieve as a pluggable storage engine.

Thanks, Patrick Galbraith and Antony Curtis!

As for our general Federated plans: Please address your suggestions to community-contributions(at)mysql.com

Links:

• MySQL 5.0 manual entry on the “Federated” storage engine: http://dev.mysql.com/doc/refman/5.0/en/federated-storage-engine.html
• MySQL 5.1 manual entry on the “Federated” storage engine: http://dev.mysql.com/doc/refman/5.1/en/federated-storage-engine.html
• MySQL Forum dedicated to the Federated to the “Federated” storage engine: http://forums.mysql.com/list.php?105
• Community-based FederatedX storage engine on Forge: http://forge.mysql.com/projects/project.php?id=265
• FederatedX shortlog: http://hg.patg.net/federatedx-storage-engine/
• Bugs related to the “Federated” storage engine: http://bugs.mysql.com/saved/bugs_federated

Last week in Prague, I met with Mark Shuttleworth of Canonical at the Ubuntu Developer Summit for version 8.10.

myself, Mark Shuttleworth, Arseniy Kuznetsov (Sun, NetBeans), Matt Zimmermann (Canonical CTO)

As you would expect, we discussed Canonical’s and MySQL’s offerings to the market. And we pondered on possibilities for joint customers to have the best of both worlds: Wouldn’t it be great if MySQL users on Ubuntu could get the “best bits” from MySQL through the software delivery mechanisms of Ubuntu?

Mark drew the above picture on a piece of paper (which I used Omni Graffle to capture in .png form), and we decided to explore how the “curved orange” part of the customer experience could be improved.

The upcoming 5.1.25 release will see a change with regards to MySQL Cluster:

MySQL 5.1.25 binaries will be released without Cluster support.

This is due to a change in the development cycle of MySQL Cluster. The work on Cluster specific features is largely independent from the rest of the MySQL Server, which has resulted in our decision to release MySQL Cluster separately from the rest of the MySQL Server, effective with MySQL 5.1.25.

This change just goes for the binaries, though. Let me reassure you that:

• Sun is still releasing MySQL Cluster under the GPL, as before.
• MySQL 5.1 source code will still include Cluster, and you can download and compile it with Cluster, if you wish.

The only change is that we’re now releasing two separate packages.

MySQL Cluster is being developed and maintained at a different pace, compared to the regular server. In addition, we have learned that our current users of MySQL Cluster are not as concerned about the regular server’s features. Typical Cluster customer installations rely mostly on updated cluster features. Many users of Cluster use the direct API rather than the SQL interface.

We hope this division will bring benefits for all parties involved:

• Cluster releases will be happening more quickly, as Cluster releases are fairly independent of MySQL Server releases. With new releases of MySQL Cluster no longer tied to the release of the regular server, cluster users will get their updates as soon as the Cluster software is updated, regardless of whether the regular server is being patched or not.
• The new Cluster package includes the Carrier Grade Edition, which contains a juicy list of features (http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-changes-5-1-ndb-6-2.html).
• Users of the main MySQL Server will now be able to download a smaller package with a quicker test suite. This should really help those with low bandwidth connections.

The change is evident when you look at the developer zone (http://dev.mysql.com) where MySQL Cluster uses a different release numbering scheme from MySQL Server, and is labeled 6.2.15. This is not the regular server version. 6.2.15 is the Cluster version, and this package is based on MySQL 5.1.

Bug fixes to the Cluster will be ported to the main server when appropriate. Bug fixes to the main server will be merged to the Cluster sources on a regular basis.

Finally, where shall you turn for the downloads?

• If you need MySQL binaries with Cluster enabled, there is a specific downloads page http://dev.mysql.com/downloads/#mysql-cluster
• MySQL Server (slimmer now, without the NDB binaries), is available as always at http://dev.mysql.com/downloads/

With 130 entries in the “MySQL” category and no MySQL-related subcategories, my blog had become impossible to search and navigate easily.

And thus I created a number of new categories for the MySQL entries within my blog. They’re listed in the left navigation bar, below the months, as well as below:

• MySQL Server, MySQL Cluster, Falcon
• Connectors: PHP, Ruby on Rails
• Tools: GUI, MySQL Workbench, MySQL Proxy
• Events: MySQL Users Conferences
• Licensing: GPL
• Architecture of Participation, Summer of Code, Virtual company
• Other: Release Policy, Documentation, Use cases

I hope this will make my blog more (re)usable.

(The picture is from this summer, when navigating the way up the Großvenediger, a 3662 m high mountain in the Hohe Tauern region of Austria.)

MySQL has introduced Previews as a new way of delivering features for early testing by our community.

Previews are for testing of features, not of versions. The two previews we offer today are the Falcon Feature Preview based on 6.0, and the GIS Feature Preview based on 5.1. However, the version number as such is not intended by us to set any expectations on when the feature will be in a production version. For instance, Falcon is going into the 6.0 code base, but the GIS geographical data won’t be in before 6.1 (although the preview is based on 5.1).

Previews are by definition a bit “rough around the edges”. They have no inherent level of maturity — they aren’t pre-alpha, or alpha, or beta, or RC, or GA. They are a new way to do “release early & release often” for those who want to get things early, without MySQL tying itself into a promise on the specific delivery release.

Falcon 6.0.4 is available immediately in the preview release and soon will be part of a formal beta. If you want it immediately, go get the preview version!

Our purpose of having previews is to make new features available for those who want early access. That way, we hope for more community testing. For our users, Previews are meant for testing only, absolutely not something to use in production. It’s a new way for us to release early & often for experimentation, learning, and testing.

Later on, I hope we will do other feature previews for storage engines or specific features.

Links:

• Falcon Feature Preview
• GIS Feature Preview
• Download MySQL 6.0
• MySQL 6.0 in a nutshell
• New features in MySQL 6.0.3
• Documentation: The Falcon storage engine
• Forge: The Falcon storage engine
• Forge: MySQL Tutorials (look for the Falcon header)

First of all: Thank you for your positive feedback on the MySQL 5.1 Errata Sheet!

While I never doubted that publishing the 5.1 Errata Sheet was the right thing to do, I had expected a more mixed feedback. It turned out the feedback was very grateful. So thank you for your encouragement

• Kevin Burton for commenting in Jay’s blog
  

  “This was a big win guys. Good work.We were going to deploy 5.1.x in out or production slave configurations just to test the reliability but this give us a lot more confidence in this release.”

• Baron Schwartz for commenting in my blog
  

  “Kaj, this is VERY helpful. I just discovered the character set issue with mysqldump myself last week.This is a big step forward in quality, from the user’s point of view. Now it’s much easier to see the state of things.”

• Peter Zaitsev for explaining some background in the blog entry “Making bugs public - good job MySQL”

The Errata Sheet had somewhat of a difficult birth process, as our pompously titled “Release Philosophy” page optimistically refers to the ideal situation of “no known critical bugs”. If we follow that policy, there is no need for an errata sheet, is there? The Errata Sheet should by definition be empty.

But it isn’t.

And it wouldn’t have been, for the last four years, had we just had sufficient awareness of critical bugs, and a policy of publishing them.

The only logical conclusion from our Reality Check was that wishful thinking must stop, and we need to do the right thing under the circumstances: Get the best possible quality releases out with the given resources, and set the expectations of our customers and users according to what we can and will deliver.

And hence we appointed a cross-team working group, the “Release Criteria Committee“, which is defining both external and internal processes and policies when it comes to

• releasing new releases in general
• moving from one maturity stage to another
• dealing with bugs of all kinds
• aligning MySQL release policy documentation with reality

So let’s first look at the circumstances:

• Resources are finite: There is only so much time in a day, and there are only so many developers around to fix bugs.
• Timely releases have value to our users: Earlier access to bug fixes. Earlier access to new features. Earlier testing of both, i.e. faster convergence. Reduced guesswork about when the next release will be.
• Zero critical bugs is impossible: Sure, zero bugs is desirable. And at times, we internally glorify MySQL’s past, with “production level alpha releases”. Those were the days, but those days also had less complex and less interwoven features, and fewer developers to manage.

As such, it is MySQL’s responsibility to create a release policy that is aligned with the needs of the customers and the user base. This will involve a number of difficult judgement calls. However, we do think that we’ve got a lot to learn from our customers and users, when it comes to defining the best possible release criteria. So I am inviting your input.

To structure your thinking and the work, let me briefly explain some basics:

1. We have four maturity levels: Alpha, Beta, RC and GA.
1a. Alpha is for testing and development only. Features are added, and
syntax may change. (*)
1b. Beta means feature-freeze. We fix bugs. (*)
1c. RC or Release Candidate needs solid criteria. RC has some industry definitions, and is the final stage prior to GA.
1d. GA or General Availability means the version is for production use. The name GA is somewhat silly for Open Source products, as even alpha releases are available for the general public.
(*) These are my own simplifications, and our work group may refine them.

2. Bugs are described by their Severity, Defect Classification, Impact, Risk To Fix, and Effort. They may have even more dimensions.
2a. Severity describes the damage from the original user (bug reporter) point of view. The most severe bugs are about crashes, security problems and corruption, without any known workaround. The least severe refer to unpleasant but documented behaviour, i.e. feature requests. From the severity of the individual original reporter, we derive the Defect Classification, which attempts at being a neutral classification across our user base.
2b. Impact refers to how many users may be affected. They range from
the entire user base, all the way to corner cases that affect only users of a particular feature on one platform.
2c. Risk To Fix and Effort refer to the difficulty of fixing the bug. A bug fix may destabilise the code (i.e. introduce more problems than it fixes), requiring postponing the fix to a later version. And the time for fixing the bug may vary (influencing the complexity of reviews).

Some of my personal thinking on the release policy is as follows:

(i) Maturity changes need quality criteria.

• Progressing from Alpha to Beta is not useful if most testers find the same elementary critical bugs; basic usability should exist
• Progressing from Beta to RC needs for the impact of critical bugs to be very low
• Progressing from RC to GA needs the impact of critical bugs to be even lower
• Hence, I advocate quality based (not time based) release criteria for maturity changes

(ii) New releases within the same maturity level need different quality criteria (from above).

• Regression bugs must be avoided (what worked in release n must still work in release n+1) (**)
• Beyond that, we mainly need for the quality of release n+1 to be higher than that of release n. In practice, if you look at today’s (10 Nov 2007) version of the Errata Sheet, you will see a lot of bugs annotated with “Already fixed in 5.1.23″, while others have “Target fix: 5.1.23″. “Let’s not wait until we can cure all illnesses, before handing out the medicine we already have!” is my logic.
• Hence, I advocate mostly time based (seldom quality based) release criteria for new releases within the same maturity level
• (**) By “regression bugs” I mean unintentionally changed behaviour, i.e. pure bugs. There may be situations where behaviour has to be consciously changed to enable the fixing of a bug.

(iii) Simplicity rules.

• The definitions of the dimensions that describe bugs must be as simple as possible.
• The rules triggering maturity changes must be as simple as possible.
• It should be easy at any point in time for us to evaluate whether we should change a maturity level (as opposed to being a negotiation between self-proclaimed advocates of quality and timeliness, respectively)

(iv) Timeliness has a value, and RC criteria may differ from GA criteria.

• Some claim that by definition, an RC must fulfill all GA criteria except the exposure to wide user base testing
• Others (like me) think that it’s in the best interest of the user base to converge to a GA quicker, through marginally relaxed RC criteria, whereby we can fix the remaining less-critical bugs in parallel with the RC getting exposed to massive parallel testing by the user base

When we (the members of our Release Criteria Committee) think we’re done, we’ll upgrade our Release Policy (and, while we’re at it, no longer pompously describe it as a Philosophy). We’re working at a fairly hectic pace, and expect to be done within weeks rather than months.

In the meantime, your input is welcome!

As indicated last week, we have decided to start producing a condensed report of outstanding issues in our current development release, MySQL 5.1.

This list is now available as part of our reference manual at dev.mysql.com/doc/refman/5.1/en/open-bugs.html.

The list is updated daily based on the then-current status in our bug database.

The purpose of the list is to make it possible for our users to more easily make an informed judgement about whether our development release is ready for them to test and use. While the corresponding bugs have already been openly and publicly available for searching in our bug database, I hope this list is easier to navigate, as it is compiled into one web page that is sorted into categories such as

• C API
• Client
• Connector/J
• Server: Backup
• Server: Charsets
• Server: Cluster
• Server: Events
• Server: I_S (Information Schema)
• Server: InnoDB
• Server: Partition
• Server: RBR (Row-based replication)
• Server: Replication

Hence, if you’re into using Row-based replication, read through the corresponding section. Our goal is to make it easy for you to know whether any of the outstanding bugs are relevant for you.

The list also contains target fix releases (with 5.1.23 currently being the next one), and links directly into the bugs system for detailed inspection of the bug description.

A good two weeks ago, MySQL 5.1.22 was released. MySQL 5.1 is a new release of MySQL Server, with new features like

• partitioning (likely the top feature in 5.1)
• events (”crontab” triggers in the database)
• row-based replication
• table logs
• some XML functions

and also with major bug fixes, such as the AUTO_INCREMENT table-lock contention in InnoDB (fixed now in 5.1.22), as well as early indications of performance improvements of up to 20 % - 40 % on dual cores in some scenarios.

Jay Pipes has written an overview that compresses all of the pointers to 5.1 into one article, MySQL 5.1 Article Recap. I recommend you read it. I also recommend the manual section What’s New in MySQL 5.1.

With 5.1.22, MySQL also changed the maturity state to “RC”, Release Candidate. Looking at our own Support Policies, Release Candidate (aka Gamma) release is defined as follows:

Release Candidate binaries, also known as Gamma releases, are believed stable, having passed all of MySQL’s internal testing, and with all known fatal runtime bugs fixed. However this release has not been in widespread use long enough to know for sure that all bugs have been identified.

However, we recognise that this particular RC does not fit the definition exactly. We still have some fatal runtime bugs left. We are producing an errata list of these, and expect to have the list ready and published on by 23 Oct 2007. We know that we should have published this list together with the RC itself, and we are now working as fast as we can to fix this.

We know we have thus released our RC too early according to our own standards, and we are doing our best to fix it. We apologise for any inconvenience/miscommunication and are working on improving our internal guidelines to ensure it doesn’t happen again.

That said, 5.1.22 is a great release, one that we’re proud of, and very likely worthy of your attention!