Back in October 2006, we introduced MySQL Community Server. Since then, we’ve learnt a thing or two, spent many man hours discussing how to improve our processes, and are now refining the concept. We feel that we’ve come up with some good middle-ground that fulfils not only our company interests but fosters community use and growth as well.

The changes are in the areas of release policy and stability of MySQL Community Server and in the availability of MySQL Enterprise Server.

The changes start from the question: “How can we better target MySQL Community Server to the community and MySQL Enterprise Server to the paying customers?“. Many of them originate from our ongoing discussions with the Linux Distributions, some of whom have been distributing MySQL Enterprise Server to their user base, since MySQL Community Server hasn’t conformed to their needs of feature stability and release schedule.

Our intention is for MySQL Community Server to be very good, and for MySQL Enterprise Server to provide further value on top of that. The five changes, in short, are:

1. New features and community contributions will go into the next development tree. The new features will not be applied to a current GA release, ensuring stability for the Community Server. At the time of writing, the development tree is MySQL 5.2.
2. There will be at least two yearly “mature GA” (currently MySQL 5.0) binary builds. They aren’t scheduled, but usually triggered by grave security vulnerabilities.
3. When a version of MySQL initially goes GA (as 5.1 soon will), the company will release binary builds of the new GA product every month for a period of several months until it reaches a point of suitable stability/maturity to be considered a “mature GA” release — as described above.
4. There will be four yearly “mature GA” (currently MySQL 5.0) source releases, predictably scheduled, to be released once every quarter. These will be ideal for use by distributions shipping MySQL.
5. The current Enterprise source tarballs will be removed from ftp.mysql.com. These will move to enterprise.mysql.com, and will be available for our paying subscribers only.

Let me expand upon the first and the last items.

Community contributions: As I know this is a topic important to our contributors and potential contributors, let me provide an example. I confess: We’ve been bad at incorporating new features. We haven’t entered much more significant than SHOW PROFILE, and it took many months and several builds to mature. And we cannot continue with this practice of applying patches to a stable release, because new features will destabilise the release. That’s what we’ve been hearing from both distributions and contributors. We’re listening, and we’re learning. By no longer going for the impossible (adding features to a feature frozen release), we hope to change this dynamic for the better.

Enterprise source code availability: I expect three questions, “Why?“, “Does this conform with the GPL?” and “What keeps a paying customer from building and posting Enterprise binaries?“. The rationale is to underline the positioning goal of “Community Server for community users, Enterprise Server for paying users”. And the GPL requires us (like anybody else) to hand out the code to those whom we give the binaries, which in the case of MySQL Enterprise Server is just the customers. So it does conform to the GPL, something that we’ve verified with the FSF to eliminate any doubt. And as for the third expected question, the answer is “Nothing”. Still, we feel that most business users will see the value of a MySQL Enterprise subscription that offers regularly-reliable software updates directly from the ’source’, along with premium 24×7 technical support and proactive monitoring/advisory tools.

To finish off, let me repeat a number of basic facts which stay unchanged:

1. both the Enterprise Server and the Community Server remain licensed under the GPLv2
2. MySQL Enterprise Server has Monthly Rapid Updates (MRUs) and Quarterly Service Packs (QSPs), i.e. binaries delivered to our customers; building these binaries is a service for paying customers only
3. bugs are first fixed in MySQL Enterprise Server builds, and while updates are delivered quicker and more frequently to paying customers, bug fixes are also added to the next source and binary builds of MySQL Community Server as stated above
4. when scheduled, we build Community binaries for all platforms
5. community binaries and sources are still available for download at dev.mysql.com/downloads
6. all source trees are still available via BitKeeper
7. the Enterprise Server is close to a full subset of the Community Server

With these changes that are intended to simplify life for the Linux Distributions that distribute MySQL, we hope to better serve the MySQL user base, while continuing to provide additional value to our paying customers in the form of more frequent, scheduled binary bug fix releases.

If you have any questions with regards to this, please do not hesitate to reply via email to community@mysql.com.

Today at OSCON, MySQL co-founder Michael “Monty” Widenius and I presented the “State of the Dolphin” lightning talk.

My slides for this preso weren’t too graphic, which makes them all the easier to reuse in this blog:

Use our new software!

• Use MySQL 5.1, it‘s soon going RC
• Use Falcon, it‘s soon going Beta (new transactional storage engine, faster than InnoDB on large servers)
• Use MySQL Workbench (ER Tool), Now Beta
• Use MySQL Proxy, just released
• PHPers: Use mysqlnd (Native Driver)

Go test MySQL 5.1!

• We‘re happy with the quality
  • More stable than 5.0 was four months after GA
  • RC happening very soon, GA within a few versions after that
  • A better MySQL 5.0 (thousands of small fixes)
• We‘re happy with the new functionality
  • Table / Index Partitioning
  • Row-based replication – Transfers data instead of commands
  • Full-text indexing parser plugins – Flexible full text search
  • Disk-based Data Support for MySQL Cluster
  • Replication Support for MySQL Cluster
  • XPath Support - helps any customer wanting to better navigate and search XML documents stored in MySQL
  • Internal Task Scheduler (Events)

Other goodies coming soon (5.2, 6.0, …)

• Global Backup API
• Falcon and Maria (MyISAM++) storage engine
• Further new storage engines
• Hash & Merge joins (faster subselects)
• Federated tables over ODBC
• Foreign key support for all engines

Participate in our Development!

• Report bugs! Test them! Submit patches!
• Hang out on Freenode IRC #mysql-dev
• Subscribe to commits@lists.mysql.com to see our code reviews
• Attend MySQL University, the foremost education for MySQL developers (of C/C++ code, not apps)
• MySQL Forge: List your MySQL apps! Upload your code snippets! Fix the missing documents!
• Go visit MySQL Forge Worklog
  • Voting for best features starting soon

Hiring!

We are hiring outstanding C/C++ developers with systems or database engineering experience.
Visit www.mysql.com/jobs or email resume to jpugh@mysql.com

A good three months ago, I had the pleasure of being the referee for seven DB egos. Now, the match has appeared as a Dev Zone article:

One of the most popular keynotes of the MySQL Conference & Expo 2007 was called “The Clash of the DB Egos”. It was a fight amongst seven database luminaries, all playing an important role either within MySQL AB or as providers of Storage Engines that work closely with MySQL. This article attempts at giving a picture of what the fight was about, through reciting the egos and the questions posed to them by the referee.

A good three years ago at the Linux Summit 26-27.2.2004 in the Finlandia Hall in Helsinki, I gave a presentation on “Why MySQL?” together with Osma Ahvenlampi, CTO of Sulake Labs, the creators of Habbo Hotel, one of the world’s largest teen online communities present in 29 countries.

Now, browsing the web pages of COSS, the Finnish Centre for Open Source Solutions, I ran across their slide repository and specifically Osma’s slides on the topic of “Infrastructure for Innovation, which is all about scale-out and describes their site which is built with MySQL as the database.

Sulake Corporation was founded May 2000 in Helsinki, and is an interactive entertainment company based on online communities and casual multiplayer games. Their tagline is “everyone can play“, which they expand in their credo “Habbo provides the most fun and creative hangout packed with friends and excitement“.

Now for the slightly more technical overview:

• Global infrastructure: 4 data centers, 29 countries
• Distributed game servers
  • Over 300 live CPU cores
  • 20 MySQL servers
  • Capacity for 130,000 concurrent users
• Self-developed game server technology
• Director/Shockwave based in-browser game client

The technology they build upon is the Linux-Java-MySQL stack, mostly with Linux servers, some Solaris. The server software Java 2 Standard Edition, with transaction processing using J2EE / JBoss. Look at Osma’s slides for more buzzwords: Hibernate, Spring, Jakarta Commons, EHCache, ICU4J, DOM4J, JMock, JGroups, JS Rhino, Junit, Sitemesh, Webwork…

Habbo runs on MySQL 5.0 with terabytes of data. Sulake claims they upgraded since they found 5.0 to perform better with complex queries, and their setup now includes the 5.0 features of federated tables and SQL views. Most interesting is probably their unique scale-out solution, with multi-master, backup+analytics replicas, averaging 10000 events / second. The nature of the online community application is such that they get an order of magnitude more write-intensive database load than traditional Master + Slaves applications. Distributed replication is used for online backups, where Ibbackup and filesystem snapshots minimise availability impact. Data volumes aren’t exactly modest, with over 10 gigs of transaction logs per day.

Go Habbo! Go Sulake! And go Osma!

Lenz made a good summary of the DotOrg Pavilion exhibitors at the Expo part of the MySQL Conference & Expo ten days from now. To recap, the DotOrg Pavilion is a part of the exhibition area reserved for Open Source projects. Our attempt has been to collect the DotOrg pavilion to be a mecca for OSS enthusiasts, writers, speakers, and advocates. We are fortunate to have attracted some of the world’s most recognised free software organisations, and new up and coming projects.

A related type of reason to register for the conference are the BOF sessions. Birds of a Feather sessions are the informal evening sessions where people sharing an interest meet face to face. You can still organise your own BoF. There are sixteen BoFs already:

BoFs Monday evening 23 April 2007

• MySQL Data Warehousing and BI, Lance Walter, V.P. Marketing, Pentaho Corporation

BoFs Tuesday evening 24 April 2007

• Backup and Recovery of MySQL, Paddy Sreenivasan, VP of Engineering and Co-founder, Zmanda, Inc., Zmanda, Inc.
• Using Perl’s DBD::mysql, Giuseppe Maxia, QA Developer, MySQL AB
• SNMP/AgentX Management of MySQL Servers, Mark Atwood
• MySQL Replication, Lars Thalmann, Replication and Clustering Technology, MySQL, Jeremy Cole, MySQL Geek, Proven Scaling LLC, Mats Kindahl, MySQL AB
• Testing Tools and Techniques, Giuseppe Maxia, QA Developer, MySQL AB
• Scalable BLOB Streaming Infrastructure, Paul McCullagh, CTO, SNAP Innovation GmbH
• Performance Tuning and Optimization, Peter Zaitsev, Co-Founder, Lead Consultant, Percona Ltd, Tobias Asplund, Instructor and Performance Tuning Consultant, MySQL AB, Jay Pipes, Community Relations Manager, North America, MySQL AB

BoFs Wednesday evening 25 April 2007

• MySQL Cluster (NDB), Stewart Smith, Cluster Developer, MySQL AB
• Quality Contribution to MySQL, Giuseppe Maxia, QA Developer, MySQL AB
• Making It Real in the Enterprise: Migrating from Access to Enterprise Web 2.0
• All Things .NET, Reggie Burnett, Software Developer, MySQL AB
• MySQL and Java, Mark D. Matthews, MySQL AB
• Online Backup, Lars Thalmann, Replication and Clustering Technology, MySQL, Charles Bell, Senior Developer, MySQL AB
• Pluggable Storage Engine API, Brian Miezejewski, Principal Consultant, MySQL Inc., Brian Aker, Director of Architecture, MySQL AB, Trudy Pelzer, Project Engineering Manager, MySQL AB
• MySQL Monitoring and Advisory Service Unleashed—Come Make Your Mark!, Andy Bang, Director of MySQL Network, MySQL, Rob Young, Senior Product Manager, MySQL Network

Please check the “Birds of a Feather Sessions” message board located in the Conference Registration Area for up-to-the-minute changes and information.

I’m just back from SNAP Innovation, the developers of the Primebase XT (PBXT) storage engine, here in Hamburg / Altona. What a pleasure to visit a company that, despite its closed-source roots, has understood and embraced what Open Source is all about and how it can open up business possibilities!

Left: Paul McCullagh

PBXT is a pluggable storage engine that fits in with MySQL 5.1. It is fully transactional, and supports MVCC. That makes it a good fit for many purposes, but the area where it is an especially good fit is for the industry that SNAP has been serving for the past decades: the publishing industry. No matter whether print or web, they have fairly similar needs, where BLOBs are a common denominator.

While MyISAM has supported BLOBs since the 1990s, and while the inventor of the concept of BLOB is Jim Starkey developing the Falcon engine, I can see PBXT satisfying needs beyond the mere moving the storage of pictures, films, MP3 files and other such data off the file system, into the database. Although PBXT isn’t there yet, Paul McCullagh (the PBXT author and SNAP CTO), Ulrich Zimmer (their CEO) and Volker Oboda (their director of Mktg & Sales) presented an interesting vision of PBXT as the engine for scalable BLOB streaming. While most access to the PBXT based data will be through MySQL, SNAP seeks to develop a dedicated API (over a separate protocol) accessing the streaming aspects of the BLOBs more efficiently than can be made through a SQL constrained interface.

I’m happy to note that Paul McCullagh will present at the MySQL Conference & Expo 23-26 April 2007, with a session PrimeBase XT: Design and Implementation of a Transactional Storage Engine (Wednesday 10:45-11:45 in Ballroom F), and on the panel in Lightning Rounds: State of the Partner Engines (Thursday 10:45-11:45, Ballroom H). I’m sure Paul will take advantage of the opportunity to integrate with not just MySQLers but also top PHP & Ruby developers, since I think there are some joint interests between PBXT and several development environments.

We talked about various aspects of better integrating PBXT into MySQL and our community activities. Let’s see if Paul manages to put up a task for MySQL’s Summer of Code page, for benchmarking of the PBXT engine.

Read more about PBXT in Robin Schumacher’s article “A look at the PBXT Storage Engine“. If you want to follow the ongoing development of PBXT, consider reading Paul’s blog (which is also aggregated on Planet MySQL) and take PBXT for a test drive yourself by downloading it from http://www.primebase.com/xt/.

It seems to me that SNAP intends to build up a sizable community of users of the PBXT storage engine under MySQL, and that they’re in it for the long run, aiming for their scalable BLOB streaming backend.

Finally!

We have a release of MySQL Community Server that contains community-provided contributions and it is ready for your download in binary form.

Thanks to Jeremy Cole for providing us with the SHOW PROFILE patch. Quoting our Reference Manual at http://dev.mysql.com/doc/refman/5.0/en/show-profiles.html:

SHOW PROFILES

SHOW PROFILE [type [, type] … ]
[FOR QUERY n]
[LIMIT n [OFFSET n]]

type:
ALL | BLOCK IO | CONTEXT SWITCHES | CPU | IPC | MEMORY | PAGE FAULTS | SOURCE | SWAPS

The SHOW PROFILES and SHOW PROFILE statements display profiling information that indicates resource usage for statements executed during the course of the current session.

Profiling is controlled by the profiling session variable, which has a default value of 0 (OFF). Profiling is enabled by setting profiling to 1 or ON:

mysql> SET profiling = 1;
SHOW PROFILES displays a list of the most recent statements sent to the master. The size of the list is controlled by the profiling_history_size session variable, which has a default value of 15. The maximum value is 100. Setting the value to 0 has the practical effect of disabling profiling.

All statements are profiled except SHOW PROFILES and SHOW PROFILE, so you will find neither of those statements in the profile list. Malformed statements are profiled. For example, SHOW PROFILING is an illegal statement, and a syntax error occurs if you try to execute it, but it will show up in the profiling list.

SHOW PROFILE displays detailed information about a single statement. Without the FOR QUERY n clause, the output pertains to the most recently executed statement. If FOR QUERY n is included, SHOW PROFILE displays information for statement n. The values of n correspond to the Query_ID values displayed by SHOW PROFILES.

The LIMIT n clause may be given to limit the output to n rows. If LIMIT is given, OFFSET n may be added to begin the output n rows into the full set of rows.

Some sample output:

mysql> SHOW PROFILES;
+----------+----------+--------------------------+
| Query_ID | Duration | Query                    |
+----------+----------+--------------------------+
|        0 | 0.000088 | SET PROFILING = 1        |
|        1 | 0.000136 | DROP TABLE IF EXISTS t1  |
|        2 | 0.011947 | CREATE TABLE t1 (id INT) |
+----------+----------+--------------------------+
3 rows in set (0.00 sec)

mysql> SHOW PROFILE;
+----------------------+----------+
| Status               | Duration |
+----------------------+----------+
| checking permissions | 0.000040 |
| creating table       | 0.000056 |
| After create         | 0.011363 |
| query end            | 0.000375 |
| freeing items        | 0.000089 |
| logging slow query   | 0.000019 |
| cleaning up          | 0.000005 |
+----------------------+----------+
7 rows in set (0.00 sec)

This information is also visible in the corresponding INFORMATION_SCHEMA.PROFILING table.

Unfortunately the SHOW PROFILE implementation as of 5.0.37 still has a small gotcha, that we will resolve for the next release: the MySQL command line client hangs if you issue the SHOW PROFILE command without having enabled the @@profiling session variable before. Hence, when you evaluate this new feature, make sure to enable it by issuing “SET @@profiling = 1;“ before.

In summary, the SHOW PROFILE patch provides a way for the MySQL Community Server users to get a more detailed understanding of where the response time is formed. Jeremy also contributed a patch to display the Uptime_since_flush_status status variable, which indicates the number of seconds since the most recent FLUSH STATUS statement. Thanks, Jeremy!

Looking beyond the community enhancements, MySQL Community Server 5.0.37 is our second full (source and binary) release of the MySQL Community Server since we made the split between the Community and Enterprise Version in October 2006. It includes all bug fixes applied to up to and including the MySQL 5.0.36 Enterprise Server. The prior MySQL Community Server 5.0.33 release early January 2007 was a source-only release.

The 5.0.37 release also resolves a crashing bug that could be exploited as a potential local Denial of Service attack. Thank you, Stefan Streichsbier from SEC Consult, who also informed us about this bug via our security@mysql.com mail address - he will send out a separate advisory about his findings.

We welcome and appreciate your feedback, bug reports, bug fixes, patches etc. as described on MySQL Forge on http://forge.mysql.com/wiki/Contributing.

The MySQL Community Team has been strengthened, getting an Engineering arm able to process code patches.

Giuseppe Maxia	Chad Miller

In an effort to align MySQL company resources to support an Architecture of Participation, we have reallocated Chad Miller and Giuseppe Maxia from the Engineering team to form the Community Engineering team. Chad is allocated towards enabling us to quickly review code patches (bug fixes, new features) and Giuseppe towards our Quality Contribution Program.

Looking at this from a Community perspective, this forms the following step in the process of opening up our Development Model and supporting the Architeture of Participation. Looking at it from an internal company perspective, this forms the following step in the specialisation of resources — a development which started more than five years ago when we dedicated some of our engineers mainly to Support, while others remained in Development.

No matter from which perspective you look at it, the establishment of the Community Engineering Team is intended this to increase our level of responsiveness to community contributions. And that is an area where we still have room for improvement. Our bugs system contains some skeletons in the form of pending community-provided patches, such as the performance enhancements of Gunnar von Boehn. The Engineering department has been busy fixing bugs according to certain priorities (which aren’t wrong per se), and the Community department has lacked the technical expertise to be able to safely apply patches. Now, we have an organisational unit which has as its main priority to work for and with community contributors, and which has the technical skill set required. This unit, Community Engineering, only depends on the rest of Engineering for reviews.

I’ve already mentioned it in my Summer of Code blog entry, but it bears repeating: We’ve seen some early wins based on this organisational change. One of our top Quality Contributors, Martin Friebe, submitted a bug report including a test case and a bug fix code patch. Through the allocation of Giuseppe and Chad to Community Engineering, the patch was verified and applied to the tree within two hours of the bug report. I hope we are able to continue down this road!

Welcome, Giuseppe! Welcome, Chad! And thanks to our VP Engineering, Jeffrey Pugh, for letting me have them!

I’m listening to the first MySQL University session. It’s the highest form of education available at MySQL AB, hence the name. Not about how to use MySQL, but how to write code patches for MySQL.

Right now, Sergei Golubchik is sitting in Cologne, Germany, speaking behind Skype Conferencing and two pillows (sound quality!) about memory management and mysys libraries and algorithms. Monty and others are assisting Sergei over our internal IRC, where there are 31 developers listening in.

Our presenter, Sergei Golubchik, in a non-virtual setting

Here’s an IRC transcript:

monty|uni ie, only destructior is called, memory is not freed until free_root()
rafal|school monty|uni: if I initialize String from other String, or char table[] or “string constant”, is memory allocated/deallocated correctly?
iggy|uni serg, While talking about alloc/dealloc for Strings, could you comment on the usefullness of SAFEMALLOC and how to take advantage of it?
monty|uni rafal|school: should be
rafal|school monty|uni: ok (thanks for all answers)
monty|uni Note that String can be both a pointer (like char*) and a C++ like string that holds the string itself
monty|uni If you use it as a pointer, you don’t get it freed. If you use it as a string (or convert it from pointer to string with String.copy(), you need to free it
monty|uni Using delete works on both cases
monty|uni so to be safe, you should just call delete on all String objects.
rafal|school monty|uni: how String can be used as a pointer?
mats|phone monty|uni, So that means that you have to allocate memory for a C string if you are passing it into the String constructor?
monty|uni Default is pointer
monty|uni mats|phone: or you use string.copy()

It’s a challenge. I’m not talking about understanding the routines, like

• init_alloc_root() - Initalize a MEMROOT object
• alloc_root() - Allocate an object
• multi_alloc_root() - Allocate many objects at once
• free_root() - Free everything

nor about understanding Serg’s accent. Everyone at MySQL has an accent. Our internal language is Bad English, and I feel competent to conform.

The challenge is the low bandwidth. I’m also sitting in Germany, and sadly, it sounds as if Serg were speaking through two pillows.

So why am I writing about this? Because we are planning to open up MySQL University. We are thinking of moving the IRC chat to freenode, and the Wiki pages to the MySQL University Library on MySQL Forge Wiki. The sound already is public, on Skype Conferencing.

With the open MySQL University, those who have an interest in learning about MySQL’s internal code structure can listen in, or follow the MP3 recording later on, or just browse the MySQL University Library. Stay tuned — we hope to be able to open up these weekly sessions later on in March.

I’m happy to now announce the full Quality Contribution program tentatively described in December.

Some basics: We understand that the main reason users report bugs is to get them fixed. That said, we still think we have plenty of opportunities of improving our ways when it comes to making it easier for you to help us.

The Quality Contribution Program goal is to improve the quality of MySQL products, with the active co-operation of the MySQL user community. The program facilitates this by

1. visibly acknowledging the participants by attributing individual quality enhancements to them;
2. rewarding the participants with benefits in proportion to their contribution (Awards are subscriptions to MySQL Enterprise);
3. streamlining the process of contributing to MySQL Quality for the benefit of both current and future contributors

Quality Contributions fall into three categories: (i) bug reports, (ii) test cases, and (iii) code patches. We evaluate each contribution with a publicly verifiable set of rules, and the contributor accumulates the QA points for all contributions submitted during the last 12 months.

The award is MySQL Enterprise subscriptions:

And let me thank the 60 Quality Contributors that have already enrolled! Specifically, I want to list by name the topmost 20 contributors, to which MySQL and its user community is greatly indebted:

Top contributors

For us to be able to accept test cases and bug patches from a legal perspective, we have to ask our contributors to agree to our Contributor License Agreement, which exists in a click-through form. Should you have any questions on this or other matters, please contact Giuseppe Maxia and/or myself at firstname@mysql.com.

So please enroll in the program and help us identify, test and fix our bugs!