The Problem

I've had MySQL on my Windows 7 laptop for a bit (as part of wampserver), mostly for local offline WordPress development.

However, even though MySQL is relatively stable, I've been observing a vast quantity of intermittent MySQL errors, as reported by WordPress in the PHP error log (C:\wamp\logs\php_error.log). Here are some examples:

[05-Jan-2010 09:47:51] WordPress database error Error on delete of
'C:\Windows\TEMP\#sql17e0_1a2_6.MYD' (Errcode: 13) for query SELECT t.*, tt.*
FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON tt.term_id = t.term_id
INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id =
tt.term_taxonomy_id WHERE tt.taxonomy IN ('category') AND tr.object_id IN (3)
ORDER BY t.name ASC made by require, require_once, include, get_footer,
locate_template, load_template, require_once, dynamic_sidebar,
call_user_func_array, widget_rrm_recent_posts, RecentPosts->execute,
ppl_expand_template, otf_categorylinks, get_the_category, wp_get_object_terms
 
[05-Jan-2010 09:50:42] WordPress database error Error on delete of
'C:\Windows\TEMP\#sql17e0_1b0_0.MYD' (Errcode: 13) for query  SELECT
SQL_CALC_FOUND_ROWS  wp_posts.* FROM wp_posts  INNER JOIN wp_term_relationships
ON (wp_posts.ID = wp_term_relationships.object_id) INNER JOIN wp_term_taxonomy ON
(wp_term_relationships.term_taxonomy_id = wp_term_taxonomy.term_taxonomy_id)
WHERE 1=1  AND wp_term_taxonomy.taxonomy = 'category' AND
wp_term_taxonomy.term_id IN ('3') AND wp_posts.post_type = 'post' AND
(wp_posts.post_status = 'publish' OR wp_posts.post_status = 'private') GROUP BY
wp_posts.ID ORDER BY wp_posts.post_date DESC LIMIT 0, 10 made by require, wp,
WP->main, WP->query_posts, WP_Query->query, WP_Query->get_posts

The important part here is "Errcode: 13", which is a file access error. The MySQL daemon process (mysqld.exe) randomly cannot access temporary tables it itself creates, which causes these errors and failed queries.

Digging Around

After looking around and finding nothing obvious, I tracked down a few forum posts mentioning the same issue.

Here is the gist – the problem is caused by an anti-virus program that is clearly not working properly.

The one mentioned on the forums is McAfee (big surprise, right? McAfee is a piece of junk – probably the worst anti-virus I've ever tried). In my case, however, it was the recently installed freeware security program from Microsoft called Microsoft Security Essentials, highly praised but problematic in this case nonetheless.

After thinking about it, I am confident that these security programs don't actually purposely prohibit access to the files MySQL creates. Instead, they lock these files for the duration of the check, so that the system doesn't get infected before they are approved. If you notice, the problems are related to temporary tables, created by MySQL on the fly. MySQL probably has a very short access timeout for these files, for performance reasons, and because it doesn't get this access fast enough, it considers it a failure (file deletions are failing, as you can see in the log).

The Solution

Now onto fixing the problem. The solution is to have Microsoft Security Essentials, in my case, or whatever your security program may be ignore these files.

You can block the whole Temp directory from being checked. It is not a good idea, as viruses can trickle down to that location and bypass your anti-virus:

Instead, you should just ignore by file extension: *.MYI and *.MYD. MySQL uses files with these extensions for its MyISAM table types. Using this approach is obviously safer as it doesn't single out a directory and instead targets specific files:

After I applied either of these exclusions, all MySQL Error 13 problems went away immediately.

Conclusion

The intermittent Error 13 problem on Windows is caused by 2 otherwise legitimate processes which, when mixed together, end up breaking MySQL.

Is this entirely the fault of the antivirus programs? Inadvertently, perhaps so.

Could MySQL be a bit smarter in this scenario? Perhaps so as well.

What do you, MySQL pros, think?

• How To Diagnose And Fix Incorrect Post Comment Counts In WordPress
• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• Swapping Column Values in MySQL
• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations
• MySQL Conference Liveblogging: Benchmarking Tools (Wednesday 4:25PM)

Similarities

• Both Solr and Sphinx satisfy all of your requirements. They're fast and designed to index and search large bodies of data efficiently.
• Both have a long list of high-traffic sites using them (Solr, Sphinx)
• Both offer commercial support. (Solr, Sphinx)
• Both offer client API bindings for several platforms/languages (Sphinx, Solr)
• Both can be distributed to increase speed and capacity (Sphinx, Solr)

Here are some differences

• Solr, being an Apache project, is obviously is Apache2-licensed. Sphinx is GPLv2. This means that if you ever need to embed or extend (not just "use") Sphinx in a commercial application, you'll have to buy a commercial license.
• Solr is easily embeddable in Java applications.
• Solr is built on top of Lucene, which is a proven technology over 7 years old with a huge user base (this is only a small part). Whenever Lucene gets a new feature or speedup, Solr gets it too. Many of the devs committing to Solr are also Lucene committers.
• Sphinx integrates more tightly with RDBMSs, especially MySQL.
• Solr can be integrated with Hadoop to build distributed applications
• Solr can be integrated with Nutch to quickly build a fully-fledged web search engine with crawler.
• Solr can index proprietary formats like Microsoft Word, PDF, etc. Sphinx can't.
• Solr comes with a spell-checker out of the box.
• Solr comes with facet support out of the box. Faceting in Sphinx takes more work.
• Sphinx doesn't allow partial index updates for field data.
• In Sphinx, all document ids must be unique unsigned non-zero integer numbers. Solr doesn't even require a unique key for many operations, and unique keys can be either integers or strings.
• Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.

Related questions

• http://stackoverflow.com/questions/1284083/choosing-a-stand-alone-full-text-search-server-sphinx-or-solr
• http://stackoverflow.com/questions/1132284/full-text-searching-with-rails
• http://stackoverflow.com/questions/737275/pros-cons-of-full-text-search-engine-lucene-sphinx-postgresql-full-text-searc

Conclusion

In my experience, Solr is very-very fast on the query side. It is also very powerful. The indexing side is very CPU and memory intensive and is an unfortunate side effect of having such a feature-rich, fast application. Nevertheless, I highly recommend Solr.

For disclaimer purposes, I have not had much experience with Sphinx and, again, all credit for this comparison goes to mausch.

By the way, here's a really good resource for Solr 1.4 that just came out: Solr 1.4 Enterprise Search. I have this book and it's quite helpful in explaining such topics as multicore setup, search methods, replication, etc.

• [Solr] How To Fix java.io.IOException: directory FOO exists and is a directory, but cannot be listed: list() returned null
• Delicious.com [Quietly] Rolls Out Domain And Url Searching/Filtering. Finally!
• Hidden Features Of Perl, PHP, Javascript, C, C++, C#, Java, Ruby, Python, And Others [Collection Of Incredibly Useful Lists]
• My MySQL Conference Schedule
• [Updated x3] Goodbye Outlook, I Don't Need You Anymore – Gmail Now Lets You Paste Images Directly From Clipboard

StackOverflow is an amazing site for coding questions. It was created by Joel Spolsky of joelonsoftware.com, Jeff Atwood of codinghorror.com, and some other incredibly smart guys who truly care about user experience. I have been a total fan of SO since it went mainstream and it's now a borderline addiction (you can see my StackOverflow badge on the right sidebar).

The Story

So, one day someone at StackOverflow started a "Hidden features of" post about a famous language (I don't feel like finding out which one was first exactly), and it turned out to be so popular that other posts in the same series started popping up.

Such questions were quickly turned into community wikis, for the purposes of harvesting and organizing information coming from the best developers on the planet and voted by users of the site. There are literally hundreds of answers, sorted by votes.

Hidden Features Of

Programming Languages

Hidden features of ActionScript3 / Flex

Hidden features of ASP.NET

Hidden features of x86 assembly

Hidden features of C

Hidden features of C++

Hidden features of C#

Hidden features of ColdFusion

Hidden features of D

Hidden features of Delphi

Hidden features of Erlang

Hidden features of F#

Hidden features of Flash development, Flash language (AS2/3), and Flash IDE

Hidden features of Java

Hidden features of JavaScript

Hidden features of Haskell

Hidden features of Lisp

Hidden features of Lua

Hidden features of Objective-C

Hidden features of Perl

Hidden features of PHP

Hidden features of Python

Hidden features of R

Hidden features of Ruby

Hidden features of Ruby on Rails

Hidden features of Scala

Hidden features of Silverlight

Hidden features and Dark Corners of STL?

Hidden features of TCL/TK

Hidden features of VB.Net

Hidden features of VBA

Databases

Hidden features of MySQL

Hidden features of Oracle

Hidden features of PL/SQL

Hidden features of PostgreSQL

Hidden features of SQL Server

Mobile

Hidden features of Android development

Other

Hidden features of Bash – also see my bash cheatsheet.

Hidden features of CSS

Hidden features of Django

Hidden features of Eclipse

Hidden features of Emacs

Hidden features of Grails

Hidden features of Greasemonkey

Hidden features of HTML

Hidden features of HTTP

Hidden features of jQuery

Hidden features of mod_rewrite

Hidden features of Qt

Hidden features of RegEx

Hidden features of Spring framework

Hidden features of VHDL

Hidden features of Visual Studio (2005-2008)

Hidden features of Windows.Forms

Hidden features of WPF and XAML

Hidden features of Xpath+Xslt

I will try to maintain this list, adding new languages that join the series as I find them. Now go learn something new!

• StackOverflow.com, SuperUser.com, ServerFault.com Fan? Here's A Greasemonkey Script To Keep Track Of All Your Accounts
• Comparison Between Solr And Sphinx Search Servers (Solr Vs Sphinx – Fight!)
• Twitter.com Autocomplete, Auto URL Expansion, Auto URL Shortener, RT Button, Nested Replies, Inline Media Embed, Search Tabs, And More
• WordPress Developers – How Do You Make A Living [Poll + Discussion]?
• Delicious.com [Quietly] Rolls Out Domain And Url Searching/Filtering. Finally!

A question I recently saw on Stack Overflow titled Faster way to delete matching [database] rows? prompted me to organize my thoughts and observations on the subject and quickly jot them down here.

Here is the brief description of the task: say, you have 2 MySQL tables a and b. The tables contain the same type of data, for example log entries. Now you want to delete all or a subset of the entries in table a that exist in table b.

Solutions Suggested By Others

DELETE FROM a WHERE EXISTS (SELECT b.id FROM b WHERE b.id = a.id);

DELETE a FROM a INNER JOIN b on a.id=b.id;

DELETE FROM a WHERE id IN (SELECT id FROM b)

The Problem With Suggested Solutions

Solutions above are all fine if the tables are quite small and the SELECT/JOIN is fast. However, in large scale situations with replication, these queries could potentially lock up the tables from writes and severely aggravate slave lag because, as I mentioned in the MySQL Slave Lag (Delay) Explained And 7 Ways To Battle It post, replication is single-threaded.

Thus, if a single UPDATE/DELETE query takes a considerable amount of time, when it propagates to the slaves, they will be stuck executing it and doing nothing else, lagging behind more and more.

My Thoughts And Solution

I have personally dealt with having to delete many rows from one table that exist in another and in my experience it's best to do the following, especially if you expect lots of rows to be deleted. This technique most importantly will improve replication slave lag.

So, here it is: do a SELECT first, as a separate query, remembering the IDs returned in your script/application, then continue on deleting in batches (say, 50,000 rows at a time). This will achieve the following:

• each one of the delete statements will not lock the table for too long, thus not letting replication lag get out of control. It is especially important if you rely on your replication to provide you relatively up-to-date data. The benefit of using batches is that if you find that each DELETE query still takes too long, you can adjust it to be smaller without touching any DB structures.
• another benefit of using a separate SELECT is that the SELECT itself might take a long time to run, especially if it can't for whatever reason use the best DB indexes. If the SELECT is inner to a DELETE, when the whole statement migrates to the slaves, it will have to do the SELECT all over again, potentially lagging the slaves because of how long that SELECT will take. If you use a separate SELECT query, this problem goes away, as all you're passing to the DELETE query is a list of IDs.

Do you have another opinion or see a fault with my logic? Feel free to share in the comments.

P.S. One thing to be careful about is, of course, potential edits to the table between the times the SELECT finishes and DELETEs start. I will let you handle such details by using transactions and/or logic pertinent to your application.

• MySQL Slave Lag (Delay) Explained And 7 Ways To Battle It
• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• MySQL Conference Liveblogging: MySQL Hidden Treasures (Thursday 11:55PM)
• MySQL Indexing Considerations Of Implementing A Priority Field In Your Application
• MySQL Conference Liveblogging: Portable Scale-out Benchmarks For MySQL (Wednesday 10:50AM)

If you, like me, are building or thinking of implementing a MySQL-powered application that has any need for prioritizing selecting certain data over other data, this article is for you.

Example

As a real world example, consider a queue-like video processing system. Your application receives new videos and processes them. The volume of incoming videos can at times be higher than the processing rate because the process is CPU bound, so occasionally a pretty long queue may form. You will try to process them as fast as you can but…

…suddenly, you need to process a video somewhere in the middle of the queue or an important video enters and needs immediate attention. What do you do?

An obvious solution is implementing a simple priority system where each item has a numeric priority field. Now you can sort first by priority from highest to lowest and then by id within the highest priority. Important and urgent items get a their priority changed to something higher and get processed first. There is only one problem.

Problem

The problem is pretty serious – let’s take a look at the SELECT statement. Before selecting, I’ve added 19 random rows to have some data to work on.

SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;

What kind of index would you put on this table to speed up this query? You do want to add a proper index, don’t you? DO YOU? Ok, good.

Here’s what happens without any indexes:

mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows | Extra          |
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
|  1 | SIMPLE      | queue | ALL  | NULL          | NULL | NULL    | NULL |   19 | Using filesort |
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
1 row in set (0.00 sec)

Using filesort, ugh, of course, due to sorting without an index.

Let’s see, how about a combined index on (priority, id)?

mysql> ALTER TABLE `queue` ADD INDEX `priority_id`(`priority`, `id`);
Query OK, 19 rows affected (0.05 sec)
Records: 19  Duplicates: 0  Warnings: 0
 
mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
| id | select_type | table | type  | possible_keys | key         | key_len | ref  | rows | Extra                       |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
|  1 | SIMPLE      | queue | index | NULL          | priority_id | 5       | NULL |   19 | Using index; Using filesort |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
1 row in set (0.00 sec)

Better because an index is being used but not very good because filesort is still present. “Of course!”, you slap yourself on the forehead. The first ORDER BY uses a DESCENDING order, and our key is in ASCENDING order.

So, let’s add the proper key with the right ordering instead.

mysql> ALTER TABLE `queue` DROP INDEX `priority_id`;
Query OK, 19 rows affected (0.05 sec)
Records: 19  Duplicates: 0  Warnings: 0
 
mysql> ALTER TABLE `queue` ADD INDEX `priority_id`(`priority` DESC, `id`);
Query OK, 19 rows affected (0.06 sec)
Records: 19  Duplicates: 0  Warnings: 0

mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
| id | select_type | table | type  | possible_keys | key         | key_len | ref  | rows | Extra                       |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
|  1 | SIMPLE      | queue | index | NULL          | priority_id | 5       | NULL |   19 | Using index; Using filesort |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
1 row in set (0.00 sec)

What the deuce? This is the same result as with the previous index. Time to dig up the documentation.

Here is what the MySQL manual has to say under the ORDER BY optimization section:

MySQL cannot use indexes to resolve the ORDER BY, although it still uses indexes to find the rows that match the WHERE clause … if you mix ASC and DESC:

SELECT * FROM t1 ORDER BY key_part1 DESC, key_part2 ASC;

Moreover, to confuse the user even more, the index creation command accepts the DESC instruction, without actually honoring it, as specified in the CREATE INDEX section:

An index_col_name specification can end with ASC or DESC. These keywords are allowed for future extensions for specifying ascending or descending index value storage. Currently, they are parsed but ignored; index values are always stored in ascending order.

So, after so many years MySQL still doesn’t support such basic functionality – you are either stuck with a query that uses filesort or have to look for a workaround.

Solution

Since it’s not possible to mix order directions, the solution is then to change the meaning of the priority column to match your needs. Thus, in the new approach priority 1 is higher than priority 10, and the application logic needs to accommodate to that. If you caught this while the application is still young, the code may be easy to change, but otherwise it could be a major pain in the butt.

Conclusion

The moral here is: plan your queries ahead and don’t mix and match DESC and ASC ordering as MySQL will not be able to use an index to resolve it. Do it even sooner if you’re putting lots and lots of data into your tables.

• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• Swapping Column Values in MySQL
• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations
• MySQL Conference Liveblogging: MySQL Performance Under A Microscope: The Tobias And Jay Show (Wednesday 2:00PM)
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)

Here are my findings:

1. The

   UPDATE swap_test SET x=y, y=x;

   approach doesn't work, as it'll just set both values to y.

   PostgreSQL seems to handle this query differently, as it apparently uses the old values throughout the whole query. [Reference]

2. Here's a method that uses a temporary variable. Thanks to Antony from the comments for the "IS NOT NULL" tweak. Without it, the query works unpredictably. See the table schema at the end of the post. This method doesn't swap the values if one of them is NULL. Use method #3 that doesn't have this limitation.

   UPDATE swap_test SET x=y, y=@temp WHERE (@temp:=x) IS NOT NULL;

   The parentheses around @temp:=x are critical. Omitting them will cause data corruption.

   1 2 3

   mysql> UPDATE swap_test SET x=y, y=@temp WHERE (@temp:=x) IS NOT NULL; Query OK, 3 rows affected Rows matched: 3 Changed: 3 Warnings: 0

3. This method was offered by Dipin in the comments. I think it’s the most elegant and clean solution. It works with both NULL and non-NULL values.

   UPDATE swap_test SET x=(@temp:=x), x = y, y = @temp;

4. Another approach I came up with that seems to work:

   UPDATE swaptest s1, swaptest s2 SET s1.x=s1.y, s1.y=s2.x WHERE s1.id=s2.id;

   1 2 3

   mysql> update swap_test s1, swap_test s2 set s1.x=s1.y, s1.y=s2.x where s1.id=s2.id; Query OK, 3 rows affected Rows matched: 3 Changed: 3 Warnings: 0

Essentially, the 1st table is the one getting updated and the 2nd one is used to pull the old data from.

Test schema used:

1
2
3
4
5
6
7
8
9
10

CREATE TABLE `swap_test` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `x` varchar(255) DEFAULT NULL,
  `y` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB;
 
INSERT INTO `swap_test` VALUES ('1', 'a', '10');
INSERT INTO `swap_test` VALUES ('2', NULL, '20');
INSERT INTO `swap_test` VALUES ('3', 'c', NULL);

Do you have a better approach? If so, please share in the comments.

Some references:

• http://stackoverflow.com/questions/37649/swapping-column-values-in-mysql/ – some discussion on various methods, which eventually prompted me to start this post
• http://www.marcworrell.com/article-3026-en.html – discussion on the 2nd approach, which doesn’t work

• MySQL Indexing Considerations Of Implementing A Priority Field In Your Application
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)
• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• How To Diagnose And Fix Incorrect Post Comment Counts In WordPress
• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations

Everyone and their mother are throwing out their predictions for 2009 nowadays, it’s a new fad. It’s like you’re not cool anymore if you don’t have twitter, a Mac, and a set of random predictions for the next 12 joyous months.

So I decided to throw in a few ideas of my own to be part of the cool crowd again (how much cooler can I be already, you might think, and I wouldn’t blame you).

Disclaimer (read it, tough guy)

What this post is:

• about the future of technology and the Internet, 2009 and beyond.
• my ideas on what is going to happen or should happen. If they happen to match someone else’s ideas – it doesn’t mean I ripped them off, it just means we share the same opinions and they’re more likely to come true.
• awesome.

What this post is not:

• predictions I pulled out of my ass, like “the market will bounce in August 2009” because some random douche said so.
• a collection of stolen ideas. I have reserved a separate post for that purpose.
• a raptor on hoverboard.

Things That Need To Happen

1. Socially Editable Maps

With the advent of the GPS technology in the last couple of years and GPS prices falling (my originally $800 Garmin Nuvi 660 now costs about $200), the biggest frustration I have now is the accuracy of information. In the world of Google Maps and Wikipedia, why is it that I have to wait a whole year for map updates that are obsolete by the time they come out? The Bay Bridge repairs change the roads on an almost monthly basis, for example.

I want to know about road changes as soon as they occur.

The next big thing will be a company, either existing, like Google, or a startup, that will introduce the social aspect into the mapping technology. It will do for maps and GPS what Wikipedia did for text, using the same approach. The details are of course to be worked out.

After this concept becomes successful, GPS companies, will need to support such updates over the air, with a push of a button. This ties in closely with prediction #2 and #4.

2. Open Source GPS Goes Mainstream

Google, the father of Android, needs to get behind this one as well. In fact, an open source GPS device would really be a subset of Android’s functionality, in a dedicated device, so it shouldn’t be that hard. In the far future, it will be built into automobile dashboards but I don’t foresee that happening in the near future.

Combined with prediction #1, this will be a killer device. I really believe that free, open source software is the wave of the future, the natural direction of where software development is heading. Look at Linux in the past few years. Look at Android now. Release one and I’m be #1 in line for one. Those who know me IRL know that I can’t go anywhere without a GPS anymore. I once forgot a GPS at home and ended up in Chili.

3. Photo and Video Cameras Will Go Wireless, Anywhere You Go

Photo and video cameras will be able to post pictures and videos to your home computer, sites like Flickr and YouTube, or email themselves directly to your friends… as soon as you record them, over cellular and Wifi networks.

Cell phones with cameras? Think cameras with cell phones. Some Wifi enabled cameras already came out, like this Canon SD430 (DP review here), and this freshly announced at CES 2009 Sony G3, and now it’s time to go cellular.

The next prediction is a generalization of this concept.

4. Wireless Connectivity Everywhere, In Every Device, Anywhere You Go

Cell phone carriers will sign an increasing amount of deals with companies that want to build devices which connect to the Internet or private networks anywhere you go: GPSes, cameras, digital picture frames, cars, fridges.

The future is completely wireless. People are sick and tired of cords, clutter, and Rick Astley – I know that for a fact and digg says so.

5. Android Will Become Hugely Successful

Alright, I’m dead tired of all the posts about how Google’s Android is going to flop in 2009. I sincerely feel that people with that point of view have never experienced Google and open source, played with Android itself, or realized what exactly an open source phone OS means to developers, manufacturers, and consumers.

Android will become a platform of choice in the next couple of years as everyone starts to realize its limitless possibilities, the OS matures, thousands of new apps get written for it, and new Android phones flood the market. The only Android phone released so far, HTC’s G1, has already received praise from consumers, as well as my friends (even the females ones are excited. And yes, I have female friends).

Motorola announced a few months ago that its new phones are going to run Android and Windows Mobile exclusively. If a company with an R&D department as big as Moto’s stands behind something, you bet your ass they’ve done they’re homework. And what’s not to like? They can now:

• take something that’s supported by the biggest Internet company in the world and the community, for free.
• stop worrying about writing and maintaining their own OS – BAM, tons of money saved.
• concentrate R&D on the hardware.
• if need be, freely develop the features they want that Android doesn’t support yet and contribute them back. Everyone wins, except maybe Symbian developers that don’t have a job anymore.

Android is going to be a revolution. Apple and Android fighting it out will be the best thing that happened to us since the invention of sushi.

6. Open Source HDTVs

The open source revolution will continue and start penetrating the HDTV market.

Personally, I think set top boxes are a waste of effort, time, and money but give me an HDTV that can run Youtube, Vimeo, LastFM, Pandora, and any other site through some sort of a plugin or browser, with a build in Media Center that connects to my computers and goes Wifi, that uses open source, upgradeable software (most likely Linux based), and I will buy it in a heartbeat.

Yes, what I’m saying is if my TV and HTPC did something dirty, I would totally dig their offspring, and so would millions of other people who don’t want/need/care/understand HTPC. TV in general holds a special place in my heart, and make it an even better experience, people.

7. Twitter’s Popularity Will Explode

Now for some social media. Twitter will continue dominating the microblogging arena, similar to YouTube dominating the video space. The site will grow extremely fast, much faster than now.

Competitors, like plurk, will probably secure certain small niches but nobody will be close to touch twitter.

Twitter’s significance in the business world for will be revolutionary. It will become second nature for every company to have one or many twitter accounts as means of connecting to consumers on a personal level. Think an opportunity for mini press releases, many of them, daily, not boring ones, the ones people are actually going to read.

8. Social Media Jobs Will Be In Increasing Demand

SEO, social marketing, and viral campaigns are very cost effective ways of brand promotion, and every company will want to jump on board.

As more and more of them realize this, they will start needing more people with marketing skills of John Chow, Chris Brogan, Nicky Cakes (this dude is hilarious), Jeremy Schoemaker, and the like.

9. YouTube Will Continue To Dominate

This one is easy:

• Biggest user base.
• HD. My favorite greasemonkey plugin YousableTubeFix exposes hi/lo FLVs, MP4, and HD MP4 options. Better quality equals better user experience.
• Chromeless player. YouTube is the only company I know of that has a Javascript controlled chromeless player, which can be embedded in any other flash player. Combined with already existing millions of embeds all over the Internet, YouTube’s popularity isn’t going anywhere any time soon.
• Google backed. Anything is possible when you Google owns you. Competition releases a good feature? YouTube has the resources to one-up it in no time. Need for more servers? YouTube will just buy a few thousand more with the $1.65Bln Google gave it. Traffic explosion? No problem – YouTube has been mooching off the Google CDN for months now.

At the end of 2007, I predicted that in a year we will experience unprecedented HD quality online video. This prediction came true when Hulu and fueled by its success CBS, ABC, NBC, and pretty much every other TV network released their free online TV sites. YouTube launched HD a few months later.

So, my YouTube prediction for 2009 is it will sign deals with major TV and movie networks to finally start showing legal TV episodes and movies. It will become the biggest legal TV and movie hub on the Internet.

10. PostgreSQL Will Gain Popularity

Sun’s buyout of MySQL in 2008 surely sent some shockwaves around. However, I predict that the following factors will contribute to PostgreSQL gaining momentum:

• certain features of MySQL were moved to Enterprise only. Open source enthusiasts don’t appreciate an open source project going partially closed source, so they will be looking for alternative software, like PostgreSQL.
• having spent years with MySQL, I am incredibly frustrated with certain quirks that should have been worked out a long time ago. As software architects look for stable, mature, cost effective, and easy to maintain databases, they will find PostgreSQL increasingly attractive.

Don’t take my word for it. I highly suggest taking a look at this whitepaper comparing MySQL and PostgreSQL. here are some highlights:

• Online operations and reorganization. This is my biggest beef with MySQL. Almost any ALTER table command will prevent writes to the table while it’s being altered. This operation requires double the table size because an ALTER simply makes a copy of the table, a rename, and then drop of the old one. This takes FOREVERRRR. PostgreSQL, on the other hand, supports a lot more online operations that will not take down the table. MySQL promised to support online ALTER TABLE by 2009. Will they keep their word? I highly doubt it.
• PostgreSQL supports function based and partial indexes.
• PostgreSQL supports nested triggers.
• PostgreSQL supports user defined datatypes.
• PostgreSQL has an IP address datatype (woot!).
• MySQL’s default engine doesn’t support online backup and recovery (*cough*, MyISAM, *cough*). Don’t even get me started on MyISAM, which doesn’t support referential integrity, transactions, of any other ACID properties. Yes, I know, there's InnoDB. It’s a lot better. But it’s still not good enough.

If you have spent time interfacing with both MySQL and PostgreSQL, I’d like to hear from you. Everyone I talked to so far who had used both, preferred PostgreSQL.

For all the MySQL fanboys, I was and still am one of you, I use MySQL every day. I’m only trying to open your eyes so you can see the rest of the world.

In fact, I think the only reason MySQL is so much more popular than Postgresql nowadays is luck and marketing by MySQL AB (hey, it sure paid off, I’m not saying anything).

So, here’s to PostgreSQL having a bright future.

Bonus

As a bonus, here is a collection of links to other interesting predictions for 2009:

• http://www.pr-squared.com/2008/12/social_media_predictions_2009.html – a great list of social media predictions.
• http://battellemedia.com/archives/004772.php – 14 online predictions as well as analysis of past years’ predictions.
• http://news.cnet.com/5-predictions-for-2009/ – 5 CNET Digital Home predictions.
• http://www.engadget.com/2009/01/01/predictions-for-2009/ – Engadget’s predictions.
• http://www.freedom-to-tinker.com/blog/felten/predictions-2009 – 38 predictions freedom and technology related predictions.
• http://www.readwriteweb.com/archives/2009_predictions_across_the_we.php – ReadWriteWeb’s predictions across the web.

• HTC Hero Coming To Sprint October 11th! $179.99, Powerful, And Sexy. Here Is Why You Need To Own It
• Sun Definitely Developing A Phone This Year
• The Real Reasons To Use Twitter (Get Over Your Prejudice Already)
• WordPress Developers – How Do You Make A Living [Poll + Discussion]?
• Meet Firefox For Mobile [Video + Feature Highlights + More Info]

A couple of reasons for sharing the primer:

• it is short and concise
• it has examples
• and most importantly, it finally pushed me to install Hadoop on a 4-machine cluster and start playing around with it

So, take a look at the primer PDF, download Hadoop, and quickstart it. Here’s a more detailed set up page.

The big guys are using it, why aren’t you?

• MySQL Conference Liveblogging: The Future Of MySQL (Tuesday 11:55AM)
• [Web Dev] Browser Breakdown Stats+Charts From Plaxo.com For December 2009 And Thoughts
• Comparison Between Solr And Sphinx Search Servers (Solr Vs Sphinx – Fight!)
• MySQL Slave Lag (Delay) Explained And 7 Ways To Battle It
• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations

• Between 1 and 100 UPDATE queries are constantly running on the master in parallel. If the slave IO is only fast enough to handle 50 of them without lagging, as soon as 51 start running, the slaves starts to lag.
• A more common problem is when one query takes an hour to run (let's say, it's an UPDATE with a big WHERE clause that doesn't use an index). In this case, the query runs on the master for an hour, which isn't a big problem because it doesn't block other queries. However, when the query moves over to the slaves, all of them start to lag because it plugs up the single replication thread.

There is nothing you can do right now to fix the way MySQL handles replication. If the replication threads could run in parallel, I'm guessing horrible things would happen to the data integrity due to race conditions, canceled queries, slave restarts, differences in query execution times due to server load and configuration, etc. Replication is already an asynchronous, prone to getting out of sync business (hint: use maatkit tools by Baron Schwartz and specifically mk-table-checksum and mk-table-sync to sync up your slaves).

In order to see if a slave is lagging, execute the 'show slave status' command and look for the Seconds_Behind_Master value. The way this value is calculated can be slightly ambiguous and unclear, so I'll explain. It is simply a difference between the 2 timestamps – the time of the last received (and queued up in the relay log) query that already executed on the master and the time of the currently executing query on the slave. Thus this value is not real time (it is possible to catch up to the master much faster); it's an approximation, or special metrics if you will, that helps point out problems.

So what can you do if you start hitting replication lag? This is the ultimate question, and the answer depends on your application. Here are the things I came up with after dealing with MySQL for a few years (there are undoubtedly other techniques, but these all come from my own experience):

1. Normalize your data, if it is not already. Non-normalized tables lead to repetition and is generally considered bad practice. More data – more IO in most cases. There can be cases, however, where you can normalize too much. Having JOINs is much slower than not having them, and it can hurt your queries if you JOIN a lot. Finally, the extreme case is mentioned at highscalability.com: How I Learned to Stop Worrying and Love Using a Lot of Disk Space to Scale. "You–pause for dramatic effect–duplicate data instead of normalize it. *shudder*". Flickr is provided as an example.
2. Shard (meaning, slice) your data, horizontally and vertically. For example, you can horizontally partition by some sort of key, hash, username, or other properties. You can also partition vertically by moving out some table columns into other databases. As an example, if you had a database of videos, storing view counts, number of favorites, etc. is OK but if these fields receive a lot of frequent updates, you are bound to have slave lag. Instead, separate these into a dedicated stats table(s). You don't have to shard all of your data – even sharding the most active bits helps immensely (for example, you can choose to shard your stats tables and leave the main one alone).
3. Upgrade machines running MySQL (first slaves, then master, for the reasons given above). 99% of the time, disk IO is the bottleneck, CPU being the other 1%. Move to RAIDed setups (RAID10 or RAID0) with 6-10 15K RPM SCSI or SSD drives. Add a lot of RAM. Make sure you're running a 64 bit OS if you have more than 3GB of RAM, so that the mysql process may utilize more of it. My search for the best MySQL server under $10K may be of some help here.
4. Separate your applications onto different MySQL instances. If you are running separate applications A, B, and C that don't depend on each other, consider giving them their own machines, otherwise a single long-running UPDATE or INSERT query in application A will delay all writes by application B and C. This is actually quite common – even though the server may not appear to be loaded, the annoying slave delay will still show its cowardly tail. I want to highlight this again: the replication thread is shared between all databases on the server.
5. Another solution to (4) is multiple MySQL instances, granted that the MySQL machine isn't generally overloaded already. In that case, installing more than 1 mysql daemons would separate replication threads and allow running multiple applications, like A, B, and C on one machine, without affecting each other. MySQL sandbox achieves just that – it is my preferred solution.
6. Split up longer running queries into shorter ones. This should be pretty straightforward – a single query on 10 million rows may run a few hours. Splitting it into batches of 50,000, for example, will give other queries a chance to run in between. Of course, you should take care of data integrity and generally double check what you are doing.
7. Don't overload the same slave by sending all queries to it, as it will just make the matter worse. You can round-robin the queries using either round-robin DNS (eww), round-robin within the application logic (better), smarter application logic, like checking slave load and status from time to time, or my personal favorite – using MySQL proxy and having it pick the least lagging slave for you. An official solution utilizing mysql proxy, called MySQL load balancer, is apparently in the works (I was promised beta access but haven't got it so far).

As a bonus, I wanted to throw in this idea of helping minimize a certain corner case cause of slave delay and feed it to the hungry MySQL minds. I'm not sure if it is mentioned anywhere else, as I have not Googled it. If it's a widely known fact, then I will consider this post as just adding my vote to the usefulness of the technique.

Well, there you have it. Comments are open, so feel free to share your own replication strategies and thoughts about mine.

• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations
• MySQL Conference Liveblogging: Portable Scale-out Benchmarks For MySQL (Wednesday 10:50AM)
• MySQL Conference Liveblogging: Performance Guide For MySQL Cluster (Tuesday 10:50AM)
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)
• Swapping Column Values in MySQL

I know that a multi-disk RAID array with a bunch of fast disks (15k RPM?) is probably the most important method of improving performance, followed by the amount of RAM, so I'm trying to find the best combination/balance of the two. However, server prices on the Internet range so much that I don't even know where to begin to tell a good deal from a bad one. I don't think SSDs can play a role here, because we need at least 200GB of usable space per machine. For comparison, we currently use the following spec: Dual quadcore Intel, 16GB RAM, 200GB RAID1 + 1TB RAID1 using SATA drives (eww?) in a 2U rack (a bit too chunky, isn't it?) made by Zantaz. It performs quite nicely but I think it chokes on the amount of writes way too early.

So, what does everyone think the best configuration is under $10,000? Bonus points if you can provide a link to the vendor site or at least a model number!

• AMD Plans Major CPU Price Drops Day After Conroe
• MySQL Conference: Presentation At The Kickfire Booth
• Beer Planet is Fast Again
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)
• MySQL Slave Lag (Delay) Explained And 7 Ways To Battle It

There is a different solution, however, that is not dependent on file sizes. Enter rdiff – rsync's backbone. You can read about it here: http://en.wikipedia.org/wiki/Rsync (search for rdiff).

The upsides of rdiff are:

• with the same 4.5GB files, rdiff only ate about 66MB of RAM and scaled very well. It never crashed to date.
• it is also MUCH faster than diff.
• rdiff itself combines both diff and patch capabilities, so you can create deltas and apply them using the same program

The downsides of rdiff are:

• it's not part of standard Linux/UNIX distribution – you have to install the librsync package.
• delta files rdiff produces have a slightly different format than diff's.
• delta files are slightly larger (but not significantly enough to care).
• a slightly different approach is used when generating a delta with rdiff, which is both good and bad – 2 steps are required. The first one produces a special signature file. In the second step, a delta is created using another rdiff call (all shown below). While the 2-step process may seem annoying, it has the benefits of providing faster deltas than when using diff. In fact, you can pipe the first step into the second one without any trouble if you want, which is what I ended up doing).

Usage:

1
2
3
4
5
6
7
8
9

$ rdiff signature ORIGINAL.txt SIGNATURE.sig
 
$ l -h SIGNATURE.sig
-rw-r--r-- 1 user users 25M 2008-04-23 22:32 SIGNATURE.sig
 
$ rdiff delta SIGNATURE.sig MODIFIED.txt DELTA.rdiff
 
$ l -h DELTA.rdiff
-rw-r--r-- 1 user users 82M 2008-04-23 22:36 DELTA.rdiff

And here's what you would do to reassemble MODIFIED.txt:

1
2
3
4
5
6

$ rdiff patch ORIGINAL.txt DELTA.rdiff MODIFIED_REASSEMBLED.txt
 
$ l *.txt
-rw-r--r-- 1 user users 4,471,493,588 2008-04-23 20:24 MODIFIED.txt
-rw-r--r-- 1 user users 4,471,493,588 2008-04-23 22:44 MODIFIED_REASSEMBLED.txt
-rw-r--r-- 1 user users 4,403,302,981 2008-04-23 20:20 ORIGINAL.txt

Just as expected – everything matches.

Now, all of this could have been done in one go like this:

1

rdiff signature ORIGINAL.txt | rdiff delta -- - MODIFIED.txt DELTA.rdiff

As far as my usage of such a useful diff program, I was doing CSV dumps of certain fields from a MySQL database, like so:

1

SELECT * FROM table WHERE some_condition='1' ORDER BY id DESC INTO OUTFILE '/home/dump/dump.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"';

and then applying rdiff to get the [quite small] daily deltas.

That's all folks!

• Watch – A Useful Linux Command You May Have Never Heard Of
• How To Download And Install Windows Media Player 11 Bypassing WGA
• Navicat For MySQL Bugs Filed
• Converting from CVS to SVN: Developer's Notes And Why SVN Is Better
• How To SVN Update All Your WordPress Plugins In One Go

Last year, Sun already made an announcement of JavaFX, a Java-based mobile platform but didn't provide any concrete timelines, so I was excited to hear the more on the subject. With Apple iPhone's advent last year and Google entering the same space later on this year with Android, Sun's addition to the game definitely won't hurt consumers. After all, competition usually leads to better products.

So when is the phone coming, Jonathan? My guess is you're going to try as hard as possible to compete with Google's second half of 2008 timeline but are you going to manage to beat it? And will it blow us away? I guess time will show.

• Meet Firefox For Mobile [Video + Feature Highlights + More Info]
• The Most Accurate Apple iPhone Review To Date From Maddox
• HTC Hero Coming To Sprint October 11th! $179.99, Powerful, And Sexy. Here Is Why You Need To Own It
• Impressions From The StackOverflow's DevDays Conference In San Francisco
• Sun buys MySQL for $1bln!

• the database
• CPU
• memory
• IO
• disk latency
• network latency
• slow queries

• 300 platforms (300 remote agents collecting data)
• 2,100 servers
• 21,000 services (10 services per server), sounds feasible
• 468,000 metrics (20 metrics per service)
• 28,800,000 metric data rows per day
• larger deployments have a lot more of these (sounds crazy)

• measurement_id
• timestamp
• value
• primary key (timestamp, measurement_id)

• agent collects data and sends reports to server with multiple data points
• server batch inserts metric data points
• if network connection fails, agent continues to collect but server "backfills" unavailable
• when agent reconnects, spooled data overwrite backfilled data points (why not use REPLACE for all inserts?)

• INSERT INTO TABLE (a,b,c) VALUES (0,0,0), (1,1,1),…
• using MySQL batch insert statements vs prepared statements with multiple queries in other databases seems to improve overall performance by 30%
• batch inserts are limited by 'max_allowed_packet'

• set unique_checks=0, insert, set unique_checks=1 (definitely need to make sure data is valid first)
• set foreign_key_checks=0, insert, set foreign_key_checks=1 (same concerns as above)
• Hyperic doesn't use the 2 above

• when regular INSERT fails, retry batch with INSERT ON DUPLICATE KEY syntax
• it's much slower but it allows

• queue metric data from separate agent reports
• minimize number of inserts, connections, CPU load
• maximize workload efficiency
• optimal configuration for 700 agents
• 3 workers
• 2000 batch size seems to work best
• queue size of 4,000,000
• this seems to peak at 2.2mil metric data inserts per minute

• inspired by rrdtool
• lower resolution tables track min, avg, and max
• data compression runs hourly
• size limit 2 days
• every hour, data is rolled up into another table that holds hourly aggregated values with size limit 14 days, then that one gets rolled up into a monthly table, etc
• this is is a good approach if you don't care about each data point

I'm overwhelmed by the amount of "you know"s from the speaker. Parasite words, ahh! Sorry Charles

• measurement data split into 18 tables, representing 9 days (2 per day)
• they didn't want to do more than 2 SELECTs to get data per day, hence such sharding
• oddly, Charles didn't actually use the word 'shard' once
• tables truncated, rather than deleting rows => huge performance boost
• truncation vs deletion
• deletion causes contention on rows
• truncation doesn't produce fragmentation
• truncation just drops and recreates the table – single DDL operation

• every InnoDB table has a special index called the clustered index (based on primary key) where the physical data for the rows is stored
• advantages
• selects faster – row data is on the same page where the index search leads
• inserts in (timestamp) order – avoid page splits and fragmentation
• shows comparison between non-clustered index and clustered index (see slides)

• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations
• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• MySQL Conference Liveblogging: Performance Guide For MySQL Cluster (Tuesday 10:50AM)
• Swapping Column Values in MySQL
• MySQL Indexing Considerations Of Implementing A Priority Field In Your Application

• an enum is both a string and a number
• sorted by number
• displayed as string
• can be sorted by string if it's cast as string

• compacts storage
• faster to search
• if (var)char is turned into enum, some space can be saved, shows example

• order by rand(1) – obviously
• the integer parameter is actually a seed
• really slow, also obviously, especially for larger tables because it has to order first, then apply rand() to the list
• another solution is to add an extra column, put random values into it, and add index, then
• UPDATE tbl SET chaos=RAND();
• random extraction
• SELECT id, col FROM tbl JOIN
    (SELECT CEIL(RAND() *
      (SELECT MAX(id) FROM tbl)) AS r)
  AS r2 ON id=r;
• for multiple rows, Damien offers a solution involving an integer table (see his own slides). This solution is supposed to be really fast.

• generate a table with only 10 integers
• do cross joins to get 100, 10,000, etc integers
• 10mil rows takes only 13 seconds (on his laptop), then either store it in another table or just use the result set
• yet again, code is shows, so see slides

• shows a handy query

• offers a long query that uses ELT(), CONCAT(), and RAND() (see slides)
• a SELECT that builds words uses GROUP_CONCAT()
• looks pretty useful

• SELECT version, REPEAT('*', percentage * 2) AS bars FROM php_versions;
• shows a nifty result set with stars as fillers in percentage bars
• shows another result set, similar to the one above, with spaces as fillers and stars at the end
• shows another example that looks like some sort of a spaceship, from mysql forge

• CONCAT() and CONCAT_WS() now for groups
• ORDER BY
• SEPARATOR
• limited to 1kb by default, change group_concat_max_len
• shows example (see slides)

• uses GROUP_CONCAT, looks horribly long and inefficient, there has to be a better way
• somebody suggests using ORDER BY with LIMIT

• uses MySQL vars
• not as many people as I thought seem to know about it, judging by the low amount of raised hands
• SET @var:=3
• SELECT @var;
• variables are not shared between connections, destroyed at the end of the connection
• uses LEAST()
• see slides

• SET @i:=0
• LOAD DATA INFILE '/tmp/stats.txt'
  INTO TABLE statPHPload
    (@date, @php, @country, @continent)
  SET
    id = 0,
    period = date(STR_TO_DATE(@date, '%d-%b-%y')),
    rank = (@i := @i + 1),
    php = CAST(REPLACE(@php, ',', '.') AS DECIMAL),
    country = @country;

• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• MySQL Conference Liveblogging: Applied Partitioning And Scaling your (OLTP) Database System (Wednesday 11:55AM)
• MySQL Conference Liveblogging: MySQL Performance Under A Microscope: The Tobias And Jay Show (Wednesday 2:00PM)
• MySQL Indexing Considerations Of Implementing A Priority Field In Your Application
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)

• MySQL Conference Liveblogging: Benchmarking Tools (Wednesday 4:25PM)
• Monitor All Your Domains From One Location
• Navicat For MySQL Bugs Filed
• MySQL Conference 2008
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)

• sysbench – Linux Test Bench
• MySQL Conference Liveblogging: Monitoring Tools (Wednesday 5:15PM)
• MySQL Conference Liveblogging: MySQL Performance Under A Microscope: The Tobias And Jay Show (Wednesday 2:00PM)
• Navicat For MySQL Bugs Filed
• MySQL Conference Liveblogging: Portable Scale-out Benchmarks For MySQL (Wednesday 10:50AM)

These questions included things like "how many MySQL servers do you have", "how many DBAs", etc. It was a lot of fun, hopefully someone (Sheeri) will edit and post the video soon.

Keith has a nice summary of everything that went on together with the numbers here.

Update: Venu has even better notes here.

• Must-Know People In The MySQL Field
• IT'S HERE. STARCRAFT 2 IS ANNOUNCED (VIDEOS)!!!!!!!!!!!!!!!
• MySQL Conference: Presentation At The Kickfire Booth
• How To Find Out The Number Of Videos On Youtube
• Swapping Column Values in MySQL

• invented by Jim Starkey
• Basic Large OBject
• Binary Large OBject
• photos, films, mp4 files, pdfs, etc

• mysql client send buffer -> receive buffer on the server (max_allowed_packet)
• streaming a BLOB
• continuous data stream
• stream BLOB data directly in and out of the database
• store BLOBs of any size (>4GB) in the database
• create a scalable back-end that can handle any throughput and storage requirements. Wouldn't need to know in advance how big the database will get
• provide an open system that can be used by all engines
• provide extensions for BLOB streaming to existing MySQL clients

• referential integrity (no invalid references), can take a lot of time to sort out missing data
• all data in one place – structured and unstructured. Good for testing
• small BLOBs are handled better by databases
• backups are consistent
• BLOBs can be replicated

• a BLOB column makes a table slow
• big rows in memory
• sequential scans are not possible
• database becomes too big
• cannot be copied easily
• backups become slow
• space not freed on delete
• database doesn't scale well

• a separate BLOB repository, outside of database rows
• references are stored in the table
• allows for incremental backups
• automatic defrag and compaction
• BLOBs not written to binlogs
• repository can be scaled-out

• BLOB storage engine (MyBS, heh heh, Baron especially likes this acronym)
• PBXT – streaming enabled engine or other storage engines
• HTTP-based API extension

• does not provide conventional table storage (CREATE TABLE syntax only for system tables)
• built-in HTTP server (port 8080 by default)
• BLOB repository
• provides a server-side API

• BLOBs that are not referenced are deleted from the repository
• initially all BLOBs don't have references to them yet
• timeout is used (mysql_temp_blob_timeout)

• creates a table of type PBXT
• uploads an image into the BLOB engine directly (using curl)
• gets back a reference id for the newly inserted BLOB
• accesses the image right in the browser using a GET request with the reference id from above
• Paul does a SELECT on the BLOB engine and gets back the earlier inserted BLOB with such things as BLOB size, content type (MIME), and EXIF data
• now Paul shows what happens if you insert a BLOB and don't access it. It then gets automatically purged from the BLOB engine after the BLOB timeout (mentioned above)
• I like what I'm seeing so far. Very interesting work, Paul. Definitely worth checking out the actual slides

• JDBC already works get/setBinaryStream(), get/setBlob()
• Google Summer of Code 2008 to add connector to PHP
• still to be done
• mysqlclient library – simple extension to do HTTP GET and PUT
• other languages (Perl, Ruby, etc)

• mysqldump will dump repository names and references

• distributed repository

• Paul has a convenient benchmark of a BLOB engine compared to xfs
• on the chart, creating and reading BLOBs stays practically the same no matter how many objects are present (in the engine or on the file system), while the file system starts high but quickly degrades almost to 0
• Why does the file system degrade to 0? Did you put all the files in the same directory? This looks very odd
• the next slide shows that when adding threads, the BLOB engine yet again performs the same and scales well while the file system starts degrading after 4-8 threads
• why is the performance better than, let's say, lighttpd pulling from a file system?

• MySQL Conference Liveblogging: The Future Of MySQL (Tuesday 11:55AM)
• MySQL Falcon Storage Engine Enters Beta Stage.
• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• Setting Up A MySQL Cluster
• MySQL Conference Liveblogging: Monitoring Tools (Wednesday 5:15PM)

• SQL_CALC_FOUND_ROWS and FOUND_ROWS()
• COUNT(*)
• MEMORY table
• if query cache is on, then it makes no difference
• if it's off
• Memory MyISAM is fastest
• FOUND_ROWS() is slightly slower than count(*)
• more in the slides that I'll add later

• SELECT … WHERE a UNION SELECT … WHERE b
  vs
  SELECT … WHERE a AND b
• index_merge wins

• composite index is faster
• of course, multiple indexes are more flexible than composite index

• …

• query cache disabled
• 100k rows
• now and 20 years back
• retrieval range between 50 and 365 days
• all data fits into memory
• UNIX_TIMESTAMP is worst
• DATETIME in the middle
• strtotime() fastest

• NOW()
• MONTH(NOW())
• MONTH('YYYY-MM-DD')
• DATE_FORMAT()
• SUBSTRING()
• all about the same if php benchmark script is used
• BENCHMARK() is very unreliable, the results vary a lot

• 3 different methods on mysql forge
• 2 UDFs and 1 using native functions
• query cache off
• 1K rows
• all rows have "/"
• count the number of "/"
• Marcelo's solution "COUNT_CHAR_OCCURRENCE": 4th place
• Garrett Hill's solution "countstring" (it's technically slightly wrong, because it doesn't account for multibyte characters): 2nd place
• Roland Bouman's solution using mysql functions: 1st place
• Jay's way: function in php: 3rd place

• BTREE is faster
• HASH uses less memory

• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• MySQL Conference Liveblogging: The Future Of MySQL (Tuesday 11:55AM)
• MySQL Conference Liveblogging: Benchmarking Tools (Wednesday 4:25PM)
• MySQL Conference Liveblogging: Performance Guide For MySQL Cluster (Tuesday 10:50AM)
• MySQL Indexing Considerations Of Implementing A Priority Field In Your Application

• old school – union in the archive tables
• auto partitioning and partition pruning
• great for data warehousing
• query performance improved
• maintenance is clearly improved

• often id driven access vs date driven access
• 1 big clients could be 80% of the whole database, so there's a difficulty selecting partitioning schemes

• reducing seek and scan set sizes
• improving inserts/updates durations
• making maintenance easier

• table sizes and predicted growth patterns – partition big tables and also partition in advance, if you predict quick growth
• access patterns – select what you want to partition by in a smart way, date, id, etc
• keys and indexes – there are a few restrictions, foreign keys are currently not supported
• availability requirements
• manageability considerations – choosing to partition by hash if there is a TON of data
• reuse / scope considerations – think ahead, think of the usage

• range partitioning
• data usually accessed by date
• limited number of primary partitions needed
• ordered intelligent keys
• support sub partitions
• list partitioning
• grouping data in partitions out of order (1,5,7 in partition x)
• limited number of primary partitions needed
• intelligent keys
• supports sub partitions
• hash partitioning
• low maintenance
• works with limited or large number of partitions
• non-intelligent keys (can work in some cases with intelligent keys)
• key partitioning
• non-integer based partitioned keys (MySQL converts to int for you)
• low maintenance

• hash(mod%num_partitions)
• in this example, Phil has stores, employees, and inventory. He decided to partition by store.
• http://dev.mysql.com/doc/refman/5.1/en/partitioning-management-hash-key.html
• 50 stores
• ALTER TABLE my_store PARTITION BY HASH(id) PARTITIONS 50;
• ALTER TABLE my_employee PARTITION BY HASH(store_id) PARTITIONS 50;
• ALTER TABLE my_inventory PARTITION BY HASH(store_id) PARTITIONS 50;
• ALTER obviously takes a long time and blocks (grr)
• adding partitions
• ALTER TABLE my_store ADD PARTITION PARTITIONS 2;
• ALTER TABLE my_employee ADD PARTITION PARTITIONS 2;
• ALTER TABLE my_inventory ADD PARTITION PARTITIONS 2;
• ALTER takes some time again, though less (how come if the partitions are empty?)
• SELECT table_name, partition_name, table_rows FROM information_schema.partitions … shows info on partitions
• remove 4 partitions
• ALTER TABLE my_store COALESCE PARTITION 4;
• ALTER TABLE my_employee COALESCE PARTITION 4;
• ALTER TABLE my_inventory COALESCE PARTITION 4;

• MySQL Conference Liveblogging: Performance Guide For MySQL Cluster (Tuesday 10:50AM)
• MySQL Indexing Considerations Of Implementing A Priority Field In Your Application
• MySQL Conference Liveblogging: The Future Of MySQL (Tuesday 11:55AM)
• MySQL Conference Liveblogging: MySQL Hidden Treasures (Thursday 11:55PM)
• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)