Friday afternoon, I met with Tetsuro Ikeda-san and Teruyoshi Hazama-san of MySQL’s long-time key partner in Japan, Sumisho. Ikeda-san and Hazama-san taught me about their work on full text search in Japanese.

Senna is an engine for fast full text search in Japanese. The Senna project derives its name from Formula I driver Ayrton Senna. “But he’s dead”, I protested. “Sure, but he is a legend and will always be associated with speed.” I cannot protest there — and the numbers I saw for Senna’s full-text search defend the choice of name.

Tritonn is the combination of Senna into MySQL. The Tritonn name refers to two things: Triton Square in Tokyo, where Sumisho has its offices, and to the fact that MySQL through our dolphin logotype is associated with the sea. Tritonn is spelt with two n’s in order to simplify web search, so as not to be confused with the Greek god Triton (the messenger of the deep, son of Poseidon, god of the sea) and the many other things named after him.

I recommended Hazama-san to meet at the MySQL Users Conference with key guys in MySQL Engineering: Sergei Golubchik who wrote MySQL’s Full-Text Search feature, Peter Gulutzan who knows all there is to be known about standards and character sets, and Alexander Barkov who implements all there is to be implemented about character sets.

Good luck to Senna and Tritonn!

References:

• Tritonn home page: http://qwik.jp/tritonn/about_en.html

With 130 entries in the “MySQL” category and no MySQL-related subcategories, my blog had become impossible to search and navigate easily.

And thus I created a number of new categories for the MySQL entries within my blog. They’re listed in the left navigation bar, below the months, as well as below:

• MySQL Server, MySQL Cluster, Falcon
• Connectors: PHP, Ruby on Rails
• Tools: GUI, MySQL Workbench, MySQL Proxy
• Events: MySQL Users Conferences
• Licensing: GPL
• Architecture of Participation, Summer of Code, Virtual company
• Other: Release Policy, Documentation, Use cases

I hope this will make my blog more (re)usable.

(The picture is from this summer, when navigating the way up the Großvenediger, a 3662 m high mountain in the Hohe Tauern region of Austria.)

Stefan Hinz, MySQL’s Docs Team Lead, just showed me the new restructured documentation overview page http://dev.mysql.com/doc/. The intention of the restructuring is to make it easier for you to find the information you need.

We’ve amended the MySQL Reference Manual section with a subsection labeled “Excerpts from the Reference Manual“, examples of which are a standalone Connectors book (covering all MySQL connectors and APIs) and guides for each individual MySQL Connector.

The key new document there is the “MySQL Server Version Reference” that should make life easier for everyone who needs cross-version information. It’s available for download but you can also browse it online at http://dev.mysql.com/doc/mysqld-version-reference/en/.

Here’s an overview of the MySQL Server Version Reference document:

1. The mysqld Options/Variables Reference contains all MySQL server options and variables for all MySQL versions. You can easily find out whether or not a particular option is available in a specific MySQL version, when it was introduced or deprecated, and more. For variables, that chapter gives a quick overview of core properties, such as if it’s a status or a server system variable, if I can be changed dynamically, or if the scope is global or local.
2. The Reserved Words list contains all words reserved in a particular MySQL (major) version, with annotations about minor versions. The list is created by running a script against all MySQL versions starting from 4.1.0. (Yes, our Docs Team has installed all MySQL versions on their documentation machine!)
3. The Functions and Operators chapter shows which MySQL functions are available in which version, and when they were introduced or deprecated.
4. The Build (configure) Options chapter should be useful for anyone building MySQL from source.
5. Eventually, the Key changes in MySQL releases chapter lists security fixes and incompatible changes for the GA versions of MySQL 4.1 and 5.0 and for the beta and RC versions of MySQL 5.1.

The Server Version Reference is not a static document but is being recreated on a regular basis.

Kudos go to Martin C. Brown who created all this. We hope it will make your work with MySQL more productive and enjoyable!

References:

• Improved MySQL Documentation overview page http://dev.mysql.com/doc/
• New MySQL Server Version Reference document: http://dev.mysql.com/doc/mysqld-version-reference/en/

A while ago, I had a discussion with Stewart Smith, Vinay Joosery, Monty Taylor and a number of other MySQLers who know much more about MySQL Cluster than I do. The result is a model for using MySQL Proxy to offload MySQL Cluster from doing Table Scans, without touching the application.

The discussion started from me asking Stewart about the largest road block for expanding the number of use cases for MySQL Cluster. “Oh, that would probably be doing JOINs and other SELECTs requiring the scanning of large parts of the database”, he replied. “There, other storage engines are faster, such as MyISAM and InnoDB.”

In a very simple view, the application talks SQL with MySQL Cluster, and gets responses.

Stewart’s insight can be refined into the first simplistic diagram by adding the recognition that “SQL” can consist of

1. UPDATE, INSERT, DELETE statements (very light, usually invidual rows affected) — unidirectional blue arrow in the diagram below
2. Simple SELECT statements (also very light, defined as SELECTs that use indices and return invidual rows) — bidirectional black arrow
3. Complex SELECT statements (could be as easy as “SELECT *”, but defined as those not easily using indices and usually returning multiple rows) — dashed arrows having two arrowheads to show that plenty of data is being returned

This second figure doesn’t depict any change in application architecture from the first figure; it just shows a more granular view.

Now, enter the insight that plain MySQL Server (with MyISAM or InnoDB) can deliver the complex SELECTs faster.

In the new architecture represented by the above picture, we scale the application by

1. introducing Replication (replicating MySQL Cluster to plain MySQL Server)
2. changing the application to direct the complex SELECTs to MySQL Server instead of MySQL Cluster

This complicates life. Not only do we need to set up replication. We also need to touch the application all over the place, to direct queries to the appropriate server.

Now, enter MySQL Proxy.

Using LUA scripts, MySQL Proxy can relieve us of the second complication, i.e. having to change the application to point to different MySQL Servers depending on the type of the SQL. Let MySQL Proxy parse the traffic and direct it to the appropriate server! The application is left untouched, and the topmost part of the picture again has a simple bidirectional arrow saying “SQL”. The distinction of what type of SQL we’re talking about is left to MySQL Proxy.

MySQL Proxy can also be assigned to load balancing the queries to a number of MySQL Replication Slaves.

This picture clearly is the most complex architecture depicted here, but also represents the highest level of scaling.

All of the above can be done using current versions of MySQL Cluster and MySQL Server, and the MySQL Proxy.

Sources:

• MySQL Proxy: http://forge.mysql.com/wiki/MySQL_Proxy
• MySQL Cluster: http://www.mysql.com/products/database/cluster/
• MySQL Replication: http://dev.mysql.com/doc/refman/5.0/en/replication.html

As indicated last week, we have decided to start producing a condensed report of outstanding issues in our current development release, MySQL 5.1.

This list is now available as part of our reference manual at dev.mysql.com/doc/refman/5.1/en/open-bugs.html.

The list is updated daily based on the then-current status in our bug database.

The purpose of the list is to make it possible for our users to more easily make an informed judgement about whether our development release is ready for them to test and use. While the corresponding bugs have already been openly and publicly available for searching in our bug database, I hope this list is easier to navigate, as it is compiled into one web page that is sorted into categories such as

• C API
• Client
• Connector/J
• Server: Backup
• Server: Charsets
• Server: Cluster
• Server: Events
• Server: I_S (Information Schema)
• Server: InnoDB
• Server: Partition
• Server: RBR (Row-based replication)
• Server: Replication

Hence, if you’re into using Row-based replication, read through the corresponding section. Our goal is to make it easy for you to know whether any of the outstanding bugs are relevant for you.

The list also contains target fix releases (with 5.1.23 currently being the next one), and links directly into the bugs system for detailed inspection of the bug description.

A good two weeks ago, MySQL 5.1.22 was released. MySQL 5.1 is a new release of MySQL Server, with new features like

• partitioning (likely the top feature in 5.1)
• events (”crontab” triggers in the database)
• row-based replication
• table logs
• some XML functions

and also with major bug fixes, such as the AUTO_INCREMENT table-lock contention in InnoDB (fixed now in 5.1.22), as well as early indications of performance improvements of up to 20 % - 40 % on dual cores in some scenarios.

Jay Pipes has written an overview that compresses all of the pointers to 5.1 into one article, MySQL 5.1 Article Recap. I recommend you read it. I also recommend the manual section What’s New in MySQL 5.1.

With 5.1.22, MySQL also changed the maturity state to “RC”, Release Candidate. Looking at our own Support Policies, Release Candidate (aka Gamma) release is defined as follows:

Release Candidate binaries, also known as Gamma releases, are believed stable, having passed all of MySQL’s internal testing, and with all known fatal runtime bugs fixed. However this release has not been in widespread use long enough to know for sure that all bugs have been identified.

However, we recognise that this particular RC does not fit the definition exactly. We still have some fatal runtime bugs left. We are producing an errata list of these, and expect to have the list ready and published on by 23 Oct 2007. We know that we should have published this list together with the RC itself, and we are now working as fast as we can to fix this.

We know we have thus released our RC too early according to our own standards, and we are doing our best to fix it. We apologise for any inconvenience/miscommunication and are working on improving our internal guidelines to ensure it doesn’t happen again.

That said, 5.1.22 is a great release, one that we’re proud of, and very likely worthy of your attention!

With the my blog entry yesterday on Refining MySQL Community Server, did anything really change? If so, was it for the better or for worse for the community?

My answer is “not much”, and “for the better”, but feel free to disagree.

Judging from some reactions in the community, a lot changed, and for the worse.

• Mike Kruckenberg feels that MySQL takes another step away from Open Source, through no longer publishing the sources for the Enterprise incarnation of the MySQL Server.
• Jeremy Cole claims that the MySQL Community split is officially a failure, and that MySQL fundamentally misunderstands their community.

Strong words.

So let me expand on the “not much” part:

The negative reactions that I experienced on the #mysql-dev channel in the Freenode IRC chat stem (like Mike Kruckenberg’s blog) from the removal of the source code of MySQL Enterprise Server from ftp.mysql.com. I argue that while this may feel or appear like a step away from Open Source, the real effect on the core MySQL community member is minuscule. And this is by design. We don’t intend for the change to adversely affect core MySQL community members’ usage. What we do intend is related to positioning: MySQL Community Server is for our users, MySQL Enterprise Server is for our paying customers. We want people to associate MySQL Enterprise Server with a commercial relationship to MySQL as a company.

Hence, we thought about what we could do to encourage the use of MySQL Community Server, in particular by the Linux distributions. Part of that is discouraging the use of MySQL Enterprise Server, and that’s why we set a signal by removing the source code of the Enterprise incarnation of the MySQL Server code.

But I want to underline that we also took a number of positive measures — carrots, if you will — which brings me to the “for the better” part.

First and foremost: We stopped adding new features to a GA release. No more new patches to 5.0. Adding new features to 5.0 is the “that” that I was above all referring to in IRC when saying (as now quoted by Jeremy)

<kaj> JeremyC: Our past 10 months since Oct has been a struggle to make that work, and we’ve failed

Having a stable release which gets new features is like squaring the circle. It’s not doable in Euclidean geometry. This is also where I feel Jeremy’s pain. There are plenty of reasons to want to have new functionality into a stable release, without the burden of the destabilisation introduced by other new functionality.

As I understand Jeremy’s non-Euclidean solution (and there are other geometries than that of Euclid), he wants to pick and choose individual patches, and build an abundance of binaries with a free combination of those patches. While that may be technically feasible, it gives MySQL Community Server a different role from the one intended by MySQL AB. MySQL AB gave two mandates to MySQL Community Server: being experimental, and being the main one used by the MySQL user base. The mandate of MySQL Enterprise Server was the opposite: being stable, and being the one used by customers entering a commercial relationship with MySQL. That equation is the other “that” which hasn’t worked in the past ten months at MySQL AB.

Now, yesterday’s announcement recognised that these requirements are conflicting. That’s where we failed. The community doesn’t want to merely experiment, it wants to use a stable version. And the changes announced yesterday are about making Community Server stable. On top of that, we increase the predictability and planned frequency of Source releases (which we do to encourage Distributions to use MySQL Community Server). Jeremy correctly points out that the frequency doesn’t change much compared to our actual track record on release frequency during the last 6-7-8 months. Personally, I wouldn’t necessarily count it to our disadvantage that we’ve released more frequently than planned (which was twice a year). The more-frequent-than-planned releases is another indicator of the failure of the double mandate (”experimental” plus “main version”), that yesterday’s announcements are supposed to improve.

Another solution, which I suppose would have pleased Jeremy, is to fix the situation by removing the mandate of MySQL Enterprise Server to be associated with a commercial relationship with MySQL. Then we would have one stable and one experimental variety of MySQL Server, regardless of the dimension of paying vs. non-paying. That would dissociate the brand of “MySQL Enterprise Server” with paying-only. And different branding of servers, in turn, is one of the basic current assumptions of Community versus Enterprise. I do understand that this distinction is not one that the Community particularly cares about. However, indirectly, a successful commercial company behind MySQL fuels the virtuous circle from which the community benefits in the form of new GPL features developed by MySQL AB.

Speaking of GPL: I want to make it perfectly clear that we have no intentions of moving MySQL Enterprise Server to another license. Yes, we are evaluating GPLv3 instead of GPLv2, but our plan is for both Community Server and Enterprise Server to remain GPL.

So in summary, I do say “not much” has changed. We have recognised some impossible requirements on the Community Server, and made the difference between Enterprise Server and Community Server consist mainly of the packaging (release schedule and availability). Same features, same bug fixes, different schedule, different branding.

I hope to now focus on making the best out of both Enterprise and Community server, feeding the virtuous circle of open source software.

Back in October 2006, we introduced MySQL Community Server. Since then, we’ve learnt a thing or two, spent many man hours discussing how to improve our processes, and are now refining the concept. We feel that we’ve come up with some good middle-ground that fulfils not only our company interests but fosters community use and growth as well.

The changes are in the areas of release policy and stability of MySQL Community Server and in the availability of MySQL Enterprise Server.

The changes start from the question: “How can we better target MySQL Community Server to the community and MySQL Enterprise Server to the paying customers?“. Many of them originate from our ongoing discussions with the Linux Distributions, some of whom have been distributing MySQL Enterprise Server to their user base, since MySQL Community Server hasn’t conformed to their needs of feature stability and release schedule.

Our intention is for MySQL Community Server to be very good, and for MySQL Enterprise Server to provide further value on top of that. The five changes, in short, are:

1. New features and community contributions will go into the next development tree. The new features will not be applied to a current GA release, ensuring stability for the Community Server. At the time of writing, the development tree is MySQL 5.2.
2. There will be at least two yearly “mature GA” (currently MySQL 5.0) binary builds. They aren’t scheduled, but usually triggered by grave security vulnerabilities.
3. When a version of MySQL initially goes GA (as 5.1 soon will), the company will release binary builds of the new GA product every month for a period of several months until it reaches a point of suitable stability/maturity to be considered a “mature GA” release — as described above.
4. There will be four yearly “mature GA” (currently MySQL 5.0) source releases, predictably scheduled, to be released once every quarter. These will be ideal for use by distributions shipping MySQL.
5. The current Enterprise source tarballs will be removed from ftp.mysql.com. These will move to enterprise.mysql.com, and will be available for our paying subscribers only.

Let me expand upon the first and the last items.

Community contributions: As I know this is a topic important to our contributors and potential contributors, let me provide an example. I confess: We’ve been bad at incorporating new features. We haven’t entered much more significant than SHOW PROFILE, and it took many months and several builds to mature. And we cannot continue with this practice of applying patches to a stable release, because new features will destabilise the release. That’s what we’ve been hearing from both distributions and contributors. We’re listening, and we’re learning. By no longer going for the impossible (adding features to a feature frozen release), we hope to change this dynamic for the better.

Enterprise source code availability: I expect three questions, “Why?“, “Does this conform with the GPL?” and “What keeps a paying customer from building and posting Enterprise binaries?“. The rationale is to underline the positioning goal of “Community Server for community users, Enterprise Server for paying users”. And the GPL requires us (like anybody else) to hand out the code to those whom we give the binaries, which in the case of MySQL Enterprise Server is just the customers. So it does conform to the GPL, something that we’ve verified with the FSF to eliminate any doubt. And as for the third expected question, the answer is “Nothing”. Still, we feel that most business users will see the value of a MySQL Enterprise subscription that offers regularly-reliable software updates directly from the ’source’, along with premium 24×7 technical support and proactive monitoring/advisory tools.

To finish off, let me repeat a number of basic facts which stay unchanged:

1. both the Enterprise Server and the Community Server remain licensed under the GPLv2
2. MySQL Enterprise Server has Monthly Rapid Updates (MRUs) and Quarterly Service Packs (QSPs), i.e. binaries delivered to our customers; building these binaries is a service for paying customers only
3. bugs are first fixed in MySQL Enterprise Server builds, and while updates are delivered quicker and more frequently to paying customers, bug fixes are also added to the next source and binary builds of MySQL Community Server as stated above
4. when scheduled, we build Community binaries for all platforms
5. community binaries and sources are still available for download at dev.mysql.com/downloads
6. all source trees are still available via BitKeeper
7. the Enterprise Server is close to a full subset of the Community Server

With these changes that are intended to simplify life for the Linux Distributions that distribute MySQL, we hope to better serve the MySQL user base, while continuing to provide additional value to our paying customers in the form of more frequent, scheduled binary bug fix releases.

If you have any questions with regards to this, please do not hesitate to reply via email to community@mysql.com.

Today at OSCON, MySQL co-founder Michael “Monty” Widenius and I presented the “State of the Dolphin” lightning talk.

My slides for this preso weren’t too graphic, which makes them all the easier to reuse in this blog:

Use our new software!

• Use MySQL 5.1, it‘s soon going RC
• Use Falcon, it‘s soon going Beta (new transactional storage engine, faster than InnoDB on large servers)
• Use MySQL Workbench (ER Tool), Now Beta
• Use MySQL Proxy, just released
• PHPers: Use mysqlnd (Native Driver)

Go test MySQL 5.1!

• We‘re happy with the quality
  • More stable than 5.0 was four months after GA
  • RC happening very soon, GA within a few versions after that
  • A better MySQL 5.0 (thousands of small fixes)
• We‘re happy with the new functionality
  • Table / Index Partitioning
  • Row-based replication – Transfers data instead of commands
  • Full-text indexing parser plugins – Flexible full text search
  • Disk-based Data Support for MySQL Cluster
  • Replication Support for MySQL Cluster
  • XPath Support - helps any customer wanting to better navigate and search XML documents stored in MySQL
  • Internal Task Scheduler (Events)

Other goodies coming soon (5.2, 6.0, …)

• Global Backup API
• Falcon and Maria (MyISAM++) storage engine
• Further new storage engines
• Hash & Merge joins (faster subselects)
• Federated tables over ODBC
• Foreign key support for all engines

Participate in our Development!

• Report bugs! Test them! Submit patches!
• Hang out on Freenode IRC #mysql-dev
• Subscribe to commits@lists.mysql.com to see our code reviews
• Attend MySQL University, the foremost education for MySQL developers (of C/C++ code, not apps)
• MySQL Forge: List your MySQL apps! Upload your code snippets! Fix the missing documents!
• Go visit MySQL Forge Worklog
  • Voting for best features starting soon

Hiring!

We are hiring outstanding C/C++ developers with systems or database engineering experience.
Visit www.mysql.com/jobs or email resume to jpugh@mysql.com

A good three months ago, I had the pleasure of being the referee for seven DB egos. Now, the match has appeared as a Dev Zone article:

One of the most popular keynotes of the MySQL Conference & Expo 2007 was called “The Clash of the DB Egos”. It was a fight amongst seven database luminaries, all playing an important role either within MySQL AB or as providers of Storage Engines that work closely with MySQL. This article attempts at giving a picture of what the fight was about, through reciting the egos and the questions posed to them by the referee.