Introduction

If your WordPress comment counts got messed up, whether because of a plugin (I'm talking about you, DISQUS) or you messed with your database manually and did something wrong (yup, that's what I just did), fear not – I have a solution for you.

But first, a little background.

Comment Counts In WordPress

Here's how comment counts work in WP:

• Posts live in a table called wp_posts and each has an ID.
• Comments reside in a table called wp_comments, each referring to an ID in wp_posts.
• However, to make queries faster, the comment count is also cached in the wp_posts table, rather than getting calculated on every page load.
  If this count ever gets out of sync with the actual number of comments for some reason, WordPress, while still displaying all comments properly, will simply show the wrong count.

How To Find Out Which Posts Are Out Of Sync

Fire up a MySQL shell or your favorite MySQL software (mine is Navicat) and run this query.

It assumes your database is called wordpress and the prefix is wp_, so adjust those accordingly.

1
2
3
4
5
6
7
8

SELECT wpp.id, wpp.post_title, wpp.comment_count, wpc.cnt
FROM wordpress.wp_posts wpp
LEFT JOIN
(SELECT comment_post_id AS c_post_id, count(*) AS cnt FROM wordpress.wp_comments
 WHERE comment_approved = 1 GROUP BY comment_post_id) wpc
ON wpp.id=wpc.c_post_id
WHERE wpp.post_type IN ('post', 'page')
      AND (wpp.comment_count!=wpc.cnt OR (wpp.comment_count != 0 AND wpc.cnt IS NULL));

The result of this query is a list of posts whose comment_counts differ from the actual number of comments associated with each of them.

The left count is the cached number, while the right one is the right one.

How To Fix The Counts Automatically

The following query will recalculate and fix the comment counts for all posts that are out of sync (ones we just queried for above):

1
2
3
4
5
6
7
8

UPDATE wordpress.wp_posts wpp
LEFT JOIN
(SELECT comment_post_id AS c_post_id, count(*) AS cnt FROM wordpress.wp_comments
 WHERE comment_approved = 1 GROUP BY comment_post_id) wpc
ON wpp.id=wpc.c_post_id
SET wpp.comment_count=wpc.cnt
WHERE wpp.post_type IN ('post', 'page')
      AND (wpp.comment_count!=wpc.cnt OR (wpp.comment_count != 0 AND wpc.cnt IS NULL));

I tested this approach on a few test cases but if you experience any problems, please do alert me in the comments and desribe your problem.

Happy WP hacking!

• How To Fix Intermittent MySQL Errcode 13 Errors On Windows
• MySQL Conference Liveblogging: MySQL Performance Under A Microscope: The Tobias And Jay Show (Wednesday 2:00PM)
• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations
• Navicat For MySQL Bugs Filed

The Problem

I've had MySQL on my Windows 7 laptop for a bit (as part of wampserver), mostly for local offline WordPress development.

However, even though MySQL is relatively stable, I've been observing a vast quantity of intermittent MySQL errors, as reported by WordPress in the PHP error log (C:\wamp\logs\php_error.log). Here are some examples:

[05-Jan-2010 09:47:51] WordPress database error Error on delete of
'C:\Windows\TEMP\#sql17e0_1a2_6.MYD' (Errcode: 13) for query SELECT t.*, tt.*
FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON tt.term_id = t.term_id
INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id =
tt.term_taxonomy_id WHERE tt.taxonomy IN ('category') AND tr.object_id IN (3)
ORDER BY t.name ASC made by require, require_once, include, get_footer,
locate_template, load_template, require_once, dynamic_sidebar,
call_user_func_array, widget_rrm_recent_posts, RecentPosts->execute,
ppl_expand_template, otf_categorylinks, get_the_category, wp_get_object_terms
 
[05-Jan-2010 09:50:42] WordPress database error Error on delete of
'C:\Windows\TEMP\#sql17e0_1b0_0.MYD' (Errcode: 13) for query  SELECT
SQL_CALC_FOUND_ROWS  wp_posts.* FROM wp_posts  INNER JOIN wp_term_relationships
ON (wp_posts.ID = wp_term_relationships.object_id) INNER JOIN wp_term_taxonomy ON
(wp_term_relationships.term_taxonomy_id = wp_term_taxonomy.term_taxonomy_id)
WHERE 1=1  AND wp_term_taxonomy.taxonomy = 'category' AND
wp_term_taxonomy.term_id IN ('3') AND wp_posts.post_type = 'post' AND
(wp_posts.post_status = 'publish' OR wp_posts.post_status = 'private') GROUP BY
wp_posts.ID ORDER BY wp_posts.post_date DESC LIMIT 0, 10 made by require, wp,
WP->main, WP->query_posts, WP_Query->query, WP_Query->get_posts

The important part here is "Errcode: 13", which is a file access error. The MySQL daemon process (mysqld.exe) randomly cannot access temporary tables it itself creates, which causes these errors and failed queries.

Digging Around

After looking around and finding nothing obvious, I tracked down a few forum posts mentioning the same issue.

Here is the gist – the problem is caused by an anti-virus program that is clearly not working properly.

The one mentioned on the forums is McAfee (big surprise, right? McAfee is a piece of junk – probably the worst anti-virus I've ever tried). In my case, however, it was the recently installed freeware security program from Microsoft called Microsoft Security Essentials, highly praised but problematic in this case nonetheless.

After thinking about it, I am confident that these security programs don't actually purposely prohibit access to the files MySQL creates. Instead, they lock these files for the duration of the check, so that the system doesn't get infected before they are approved. If you notice, the problems are related to temporary tables, created by MySQL on the fly. MySQL probably has a very short access timeout for these files, for performance reasons, and because it doesn't get this access fast enough, it considers it a failure (file deletions are failing, as you can see in the log).

The Solution

Now onto fixing the problem. The solution is to have Microsoft Security Essentials, in my case, or whatever your security program may be ignore these files.

You can block the whole Temp directory from being checked. It is not a good idea, as viruses can trickle down to that location and bypass your anti-virus:

Instead, you should just ignore by file extension: *.MYI and *.MYD. MySQL uses files with these extensions for its MyISAM table types. Using this approach is obviously safer as it doesn't single out a directory and instead targets specific files:

After I applied either of these exclusions, all MySQL Error 13 problems went away immediately.

Conclusion

The intermittent Error 13 problem on Windows is caused by 2 otherwise legitimate processes which, when mixed together, end up breaking MySQL.

Is this entirely the fault of the antivirus programs? Inadvertently, perhaps so.

Could MySQL be a bit smarter in this scenario? Perhaps so as well.

What do you, MySQL pros, think?

• How To Diagnose And Fix Incorrect Post Comment Counts In WordPress
• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• Swapping Column Values in MySQL
• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations
• MySQL Conference Liveblogging: Benchmarking Tools (Wednesday 4:25PM)

Similarities

• Both Solr and Sphinx satisfy all of your requirements. They're fast and designed to index and search large bodies of data efficiently.
• Both have a long list of high-traffic sites using them (Solr, Sphinx)
• Both offer commercial support. (Solr, Sphinx)
• Both offer client API bindings for several platforms/languages (Sphinx, Solr)
• Both can be distributed to increase speed and capacity (Sphinx, Solr)

Here are some differences

• Solr, being an Apache project, is obviously is Apache2-licensed. Sphinx is GPLv2. This means that if you ever need to embed or extend (not just "use") Sphinx in a commercial application, you'll have to buy a commercial license.
• Solr is easily embeddable in Java applications.
• Solr is built on top of Lucene, which is a proven technology over 7 years old with a huge user base (this is only a small part). Whenever Lucene gets a new feature or speedup, Solr gets it too. Many of the devs committing to Solr are also Lucene committers.
• Sphinx integrates more tightly with RDBMSs, especially MySQL.
• Solr can be integrated with Hadoop to build distributed applications
• Solr can be integrated with Nutch to quickly build a fully-fledged web search engine with crawler.
• Solr can index proprietary formats like Microsoft Word, PDF, etc. Sphinx can't.
• Solr comes with a spell-checker out of the box.
• Solr comes with facet support out of the box. Faceting in Sphinx takes more work.
• Sphinx doesn't allow partial index updates for field data.
• In Sphinx, all document ids must be unique unsigned non-zero integer numbers. Solr doesn't even require a unique key for many operations, and unique keys can be either integers or strings.
• Solr supports field collapsing to avoid duplicating similar results. Sphinx doesn't seem to provide any feature like this.

Related questions

• http://stackoverflow.com/questions/1284083/choosing-a-stand-alone-full-text-search-server-sphinx-or-solr
• http://stackoverflow.com/questions/1132284/full-text-searching-with-rails
• http://stackoverflow.com/questions/737275/pros-cons-of-full-text-search-engine-lucene-sphinx-postgresql-full-text-searc

Conclusion

In my experience, Solr is very-very fast on the query side. It is also very powerful. The indexing side is very CPU and memory intensive and is an unfortunate side effect of having such a feature-rich, fast application. Nevertheless, I highly recommend Solr.

For disclaimer purposes, I have not had much experience with Sphinx and, again, all credit for this comparison goes to mausch.

By the way, here's a really good resource for Solr 1.4 that just came out: Solr 1.4 Enterprise Search. I have this book and it's quite helpful in explaining such topics as multicore setup, search methods, replication, etc.

• [Solr] How To Fix java.io.IOException: directory FOO exists and is a directory, but cannot be listed: list() returned null
• Delicious.com [Quietly] Rolls Out Domain And Url Searching/Filtering. Finally!
• Hidden Features Of Perl, PHP, Javascript, C, C++, C#, Java, Ruby, Python, And Others [Collection Of Incredibly Useful Lists]
• My MySQL Conference Schedule
• [Updated x3] Goodbye Outlook, I Don't Need You Anymore – Gmail Now Lets You Paste Images Directly From Clipboard

StackOverflow is an amazing site for coding questions. It was created by Joel Spolsky of joelonsoftware.com, Jeff Atwood of codinghorror.com, and some other incredibly smart guys who truly care about user experience. I have been a total fan of SO since it went mainstream and it's now a borderline addiction (you can see my StackOverflow badge on the right sidebar).

The Story

So, one day someone at StackOverflow started a "Hidden features of" post about a famous language (I don't feel like finding out which one was first exactly), and it turned out to be so popular that other posts in the same series started popping up.

Such questions were quickly turned into community wikis, for the purposes of harvesting and organizing information coming from the best developers on the planet and voted by users of the site. There are literally hundreds of answers, sorted by votes.

Hidden Features Of

Programming Languages

Hidden features of ActionScript3 / Flex

Hidden features of ASP.NET

Hidden features of x86 assembly

Hidden features of C

Hidden features of C++

Hidden features of C#

Hidden features of ColdFusion

Hidden features of D

Hidden features of Delphi

Hidden features of Erlang

Hidden features of F#

Hidden features of Flash development, Flash language (AS2/3), and Flash IDE

Hidden features of Java

Hidden features of JavaScript

Hidden features of Haskell

Hidden features of Lisp

Hidden features of Lua

Hidden features of Objective-C

Hidden features of Perl

Hidden features of PHP

Hidden features of Python

Hidden features of R

Hidden features of Ruby

Hidden features of Ruby on Rails

Hidden features of Scala

Hidden features of Silverlight

Hidden features and Dark Corners of STL?

Hidden features of TCL/TK

Hidden features of VB.Net

Hidden features of VBA

Databases

Hidden features of MySQL

Hidden features of Oracle

Hidden features of PL/SQL

Hidden features of PostgreSQL

Hidden features of SQL Server

Mobile

Hidden features of Android development

Other

Hidden features of Bash – also see my bash cheatsheet.

Hidden features of CSS

Hidden features of Django

Hidden features of Eclipse

Hidden features of Emacs

Hidden features of Grails

Hidden features of Greasemonkey

Hidden features of HTML

Hidden features of HTTP

Hidden features of jQuery

Hidden features of mod_rewrite

Hidden features of Qt

Hidden features of RegEx

Hidden features of Spring framework

Hidden features of VHDL

Hidden features of Visual Studio (2005-2008)

Hidden features of Windows.Forms

Hidden features of WPF and XAML

Hidden features of Xpath+Xslt

I will try to maintain this list, adding new languages that join the series as I find them. Now go learn something new!

• StackOverflow.com, SuperUser.com, ServerFault.com Fan? Here's A Greasemonkey Script To Keep Track Of All Your Accounts
• Comparison Between Solr And Sphinx Search Servers (Solr Vs Sphinx – Fight!)
• Twitter.com Autocomplete, Auto URL Expansion, Auto URL Shortener, RT Button, Nested Replies, Inline Media Embed, Search Tabs, And More
• WordPress Developers – How Do You Make A Living [Poll + Discussion]?
• Delicious.com [Quietly] Rolls Out Domain And Url Searching/Filtering. Finally!

A question I recently saw on Stack Overflow titled Faster way to delete matching [database] rows? prompted me to organize my thoughts and observations on the subject and quickly jot them down here.

Here is the brief description of the task: say, you have 2 MySQL tables a and b. The tables contain the same type of data, for example log entries. Now you want to delete all or a subset of the entries in table a that exist in table b.

Solutions Suggested By Others

DELETE FROM a WHERE EXISTS (SELECT b.id FROM b WHERE b.id = a.id);

DELETE a FROM a INNER JOIN b on a.id=b.id;

DELETE FROM a WHERE id IN (SELECT id FROM b)

The Problem With Suggested Solutions

Solutions above are all fine if the tables are quite small and the SELECT/JOIN is fast. However, in large scale situations with replication, these queries could potentially lock up the tables from writes and severely aggravate slave lag because, as I mentioned in the MySQL Slave Lag (Delay) Explained And 7 Ways To Battle It post, replication is single-threaded.

Thus, if a single UPDATE/DELETE query takes a considerable amount of time, when it propagates to the slaves, they will be stuck executing it and doing nothing else, lagging behind more and more.

My Thoughts And Solution

I have personally dealt with having to delete many rows from one table that exist in another and in my experience it's best to do the following, especially if you expect lots of rows to be deleted. This technique most importantly will improve replication slave lag.

So, here it is: do a SELECT first, as a separate query, remembering the IDs returned in your script/application, then continue on deleting in batches (say, 50,000 rows at a time). This will achieve the following:

• each one of the delete statements will not lock the table for too long, thus not letting replication lag get out of control. It is especially important if you rely on your replication to provide you relatively up-to-date data. The benefit of using batches is that if you find that each DELETE query still takes too long, you can adjust it to be smaller without touching any DB structures.
• another benefit of using a separate SELECT is that the SELECT itself might take a long time to run, especially if it can't for whatever reason use the best DB indexes. If the SELECT is inner to a DELETE, when the whole statement migrates to the slaves, it will have to do the SELECT all over again, potentially lagging the slaves because of how long that SELECT will take. If you use a separate SELECT query, this problem goes away, as all you're passing to the DELETE query is a list of IDs.

Do you have another opinion or see a fault with my logic? Feel free to share in the comments.

P.S. One thing to be careful about is, of course, potential edits to the table between the times the SELECT finishes and DELETEs start. I will let you handle such details by using transactions and/or logic pertinent to your application.

• MySQL Slave Lag (Delay) Explained And 7 Ways To Battle It
• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• MySQL Conference Liveblogging: MySQL Hidden Treasures (Thursday 11:55PM)
• MySQL Indexing Considerations Of Implementing A Priority Field In Your Application
• MySQL Conference Liveblogging: Portable Scale-out Benchmarks For MySQL (Wednesday 10:50AM)

If you, like me, are building or thinking of implementing a MySQL-powered application that has any need for prioritizing selecting certain data over other data, this article is for you.

Example

As a real world example, consider a queue-like video processing system. Your application receives new videos and processes them. The volume of incoming videos can at times be higher than the processing rate because the process is CPU bound, so occasionally a pretty long queue may form. You will try to process them as fast as you can but…

…suddenly, you need to process a video somewhere in the middle of the queue or an important video enters and needs immediate attention. What do you do?

An obvious solution is implementing a simple priority system where each item has a numeric priority field. Now you can sort first by priority from highest to lowest and then by id within the highest priority. Important and urgent items get a their priority changed to something higher and get processed first. There is only one problem.

Problem

The problem is pretty serious – let’s take a look at the SELECT statement. Before selecting, I’ve added 19 random rows to have some data to work on.

SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;

What kind of index would you put on this table to speed up this query? You do want to add a proper index, don’t you? DO YOU? Ok, good.

Here’s what happens without any indexes:

mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows | Extra          |
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
|  1 | SIMPLE      | queue | ALL  | NULL          | NULL | NULL    | NULL |   19 | Using filesort |
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
1 row in set (0.00 sec)

Using filesort, ugh, of course, due to sorting without an index.

Let’s see, how about a combined index on (priority, id)?

mysql> ALTER TABLE `queue` ADD INDEX `priority_id`(`priority`, `id`);
Query OK, 19 rows affected (0.05 sec)
Records: 19  Duplicates: 0  Warnings: 0
 
mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
| id | select_type | table | type  | possible_keys | key         | key_len | ref  | rows | Extra                       |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
|  1 | SIMPLE      | queue | index | NULL          | priority_id | 5       | NULL |   19 | Using index; Using filesort |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
1 row in set (0.00 sec)

Better because an index is being used but not very good because filesort is still present. “Of course!”, you slap yourself on the forehead. The first ORDER BY uses a DESCENDING order, and our key is in ASCENDING order.

So, let’s add the proper key with the right ordering instead.

mysql> ALTER TABLE `queue` DROP INDEX `priority_id`;
Query OK, 19 rows affected (0.05 sec)
Records: 19  Duplicates: 0  Warnings: 0
 
mysql> ALTER TABLE `queue` ADD INDEX `priority_id`(`priority` DESC, `id`);
Query OK, 19 rows affected (0.06 sec)
Records: 19  Duplicates: 0  Warnings: 0

mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
| id | select_type | table | type  | possible_keys | key         | key_len | ref  | rows | Extra                       |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
|  1 | SIMPLE      | queue | index | NULL          | priority_id | 5       | NULL |   19 | Using index; Using filesort |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
1 row in set (0.00 sec)

What the deuce? This is the same result as with the previous index. Time to dig up the documentation.

Here is what the MySQL manual has to say under the ORDER BY optimization section:

MySQL cannot use indexes to resolve the ORDER BY, although it still uses indexes to find the rows that match the WHERE clause … if you mix ASC and DESC:

SELECT * FROM t1 ORDER BY key_part1 DESC, key_part2 ASC;

Moreover, to confuse the user even more, the index creation command accepts the DESC instruction, without actually honoring it, as specified in the CREATE INDEX section:

An index_col_name specification can end with ASC or DESC. These keywords are allowed for future extensions for specifying ascending or descending index value storage. Currently, they are parsed but ignored; index values are always stored in ascending order.

So, after so many years MySQL still doesn’t support such basic functionality – you are either stuck with a query that uses filesort or have to look for a workaround.

Solution

Since it’s not possible to mix order directions, the solution is then to change the meaning of the priority column to match your needs. Thus, in the new approach priority 1 is higher than priority 10, and the application logic needs to accommodate to that. If you caught this while the application is still young, the code may be easy to change, but otherwise it could be a major pain in the butt.

Conclusion

The moral here is: plan your queries ahead and don’t mix and match DESC and ASC ordering as MySQL will not be able to use an index to resolve it. Do it even sooner if you’re putting lots and lots of data into your tables.

• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• Swapping Column Values in MySQL
• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations
• MySQL Conference Liveblogging: MySQL Performance Under A Microscope: The Tobias And Jay Show (Wednesday 2:00PM)
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)

Here are my findings:

1. The

   UPDATE swap_test SET x=y, y=x;

   approach doesn't work, as it'll just set both values to y.

   PostgreSQL seems to handle this query differently, as it apparently uses the old values throughout the whole query. [Reference]

2. Here's a method that uses a temporary variable. Thanks to Antony from the comments for the "IS NOT NULL" tweak. Without it, the query works unpredictably. See the table schema at the end of the post. This method doesn't swap the values if one of them is NULL. Use method #3 that doesn't have this limitation.

   UPDATE swap_test SET x=y, y=@temp WHERE (@temp:=x) IS NOT NULL;

   The parentheses around @temp:=x are critical. Omitting them will cause data corruption.

   1 2 3

   mysql> UPDATE swap_test SET x=y, y=@temp WHERE (@temp:=x) IS NOT NULL; Query OK, 3 rows affected Rows matched: 3 Changed: 3 Warnings: 0

3. This method was offered by Dipin in the comments. I think it’s the most elegant and clean solution. It works with both NULL and non-NULL values.

   UPDATE swap_test SET x=(@temp:=x), x = y, y = @temp;

4. Another approach I came up with that seems to work:

   UPDATE swaptest s1, swaptest s2 SET s1.x=s1.y, s1.y=s2.x WHERE s1.id=s2.id;

   1 2 3

   mysql> update swap_test s1, swap_test s2 set s1.x=s1.y, s1.y=s2.x where s1.id=s2.id; Query OK, 3 rows affected Rows matched: 3 Changed: 3 Warnings: 0

Essentially, the 1st table is the one getting updated and the 2nd one is used to pull the old data from.

Test schema used:

1
2
3
4
5
6
7
8
9
10

CREATE TABLE `swap_test` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `x` varchar(255) DEFAULT NULL,
  `y` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB;
 
INSERT INTO `swap_test` VALUES ('1', 'a', '10');
INSERT INTO `swap_test` VALUES ('2', NULL, '20');
INSERT INTO `swap_test` VALUES ('3', 'c', NULL);

Do you have a better approach? If so, please share in the comments.

Some references:

• http://stackoverflow.com/questions/37649/swapping-column-values-in-mysql/ – some discussion on various methods, which eventually prompted me to start this post
• http://www.marcworrell.com/article-3026-en.html – discussion on the 2nd approach, which doesn’t work

• MySQL Indexing Considerations Of Implementing A Priority Field In Your Application
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)
• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• How To Diagnose And Fix Incorrect Post Comment Counts In WordPress
• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations

Clickjacking is a malicious technique of tricking web users into revealing confidential information or taking control of their computer while clicking on seemingly innocuous web pages. A vulnerability across a variety of browsers and platforms, a clickjacking takes the form of embedded code or script that can execute without the user's knowledge, such as clicking on a button that appears to perform another function (credit: Wikipedia).

Clickjacking is hard to combat. From a technical standpoint, the attack is executed using a combination of CSS and iFrames, which are both harmless web technologies, and relies mostly on tricking users by means of social engineering. Additionally, the only server side technique against clickjacking known to me is “frame breaking”, which would cause a legitemate site to break out of any iFrames it may be embedded in. This is not always the desired behavior and is generally frowned upon.

Generic Example

In laymen’s terms, clickjacking means that it is quite possible for websites to trick you into, for example, clicking a button to show a cute kitty while in reality prompting a deletion of all your hotmail email. A malicious site uses an iFrame (which essentially allows embedding sites within other sites) with hotmail loaded inside and hidden using CSS (which is a web language for styling HTML elements). A button named “Show Me The Next Awwww Kitty” is then placed by the malicious site and positioned below the iFrame layer (manipulated by CSS, yet again). However, because the iFrame is hidden, it looks like the “Aww” button is all you’re clicking. Wrong!

Latest Example: Twitter

This morning a new, though harmless, epidemic hit twitter. Hundreds and thousands of messages saying “Don’t Click: http://tinyurl.com/amgzs6” started showing up. Clicking the link shows a simple page with 1 button:

Clicking (which I of course did) uses clickjacking to repost the message to your own twitter account. Take a look yourself: http://search.twitter.com/search?q=don%27t+click.

All of these are a result of an experiment by some French guys to mess around with twitter and show the effects of clickjacking. Thank you for that, French guys. Creating awareness via the most social platform on the web is the best thing they could do for us.

Fight Clickjacking

James Padolsey recently wrote an excellent blog post about clickjacking and mentioned Twitter specifically. Because clickjacking relies mostly on social hacking (i.e. tricking people into clicking malicious links and buttons), Twitter is nothing but a perfect platform. James gives some nice background info and code examples. He correctly highlights, as I did earlier, that clickjacking is not a software bug – it’s a malicious technique exploiting harmless technologies.

So how does one fight clickjacking?

At this point the most reliable way is to use Firefox and the NoScript extension. NoScript provides a simple, yet amazingly effective feature, called ClearClick. From their site:

“…it's enabled by default, protecting NoScript users from Clickjacking everywhere: it even remains active if you switch NoScript in the less safe Allow scripts globally mode. How does it work? Clickjacking hides or displaces or partially covers something you wouldn't want to click, if you could see it in its original context. ClearClick does the opposite: whenever you click a plugin object or a framed page, it takes a screenshot of it alone and opaque (i.e. an image of it with no transparencies and no overlaying objects), then compares it with a screenshot of the parent page as you can see it. If the two images differ, a clickjacking attack is probably happening and NoScript raises a "ClearClick warning", showing you the contextualized and "clear" object you were about to click, so you can evaluate by yourself if that was really something you wanted to do.”

Did ClearClick work in the earlier twitter attack? Sure did! After I clicked the “Don’t click” button Noscript promptly popped up a warning showing the hidden iFrame (since the original malicious page has been removed, I found another similar page from the same author for screenshot purposes).

So, even if you don’t want to enable NoScript globally, install it anyway, just for ClearClick.

That about covers what I had to say about clickjacking. Stay safe, folks!

• Modern-Day Frame Busting With X-FRAME-OPTIONS And "This content cannot be displayed in a frame" Warnings
• Enable A Twitter Retweet (RT) Button That Lets You Add Comments Before Retweeting
• How *Not* To Implement A Web Application That Handles External Authentication, Using BeTwittered.com As An Example
• StackOverflow.com, SuperUser.com, ServerFault.com Fan? Here's A Greasemonkey Script To Keep Track Of All Your Accounts
• Hidden Features Of Perl, PHP, Javascript, C, C++, C#, Java, Ruby, Python, And Others [Collection Of Incredibly Useful Lists]

Everyone and their mother are throwing out their predictions for 2009 nowadays, it’s a new fad. It’s like you’re not cool anymore if you don’t have twitter, a Mac, and a set of random predictions for the next 12 joyous months.

So I decided to throw in a few ideas of my own to be part of the cool crowd again (how much cooler can I be already, you might think, and I wouldn’t blame you).

Disclaimer (read it, tough guy)

What this post is:

• about the future of technology and the Internet, 2009 and beyond.
• my ideas on what is going to happen or should happen. If they happen to match someone else’s ideas – it doesn’t mean I ripped them off, it just means we share the same opinions and they’re more likely to come true.
• awesome.

What this post is not:

• predictions I pulled out of my ass, like “the market will bounce in August 2009” because some random douche said so.
• a collection of stolen ideas. I have reserved a separate post for that purpose.
• a raptor on hoverboard.

Things That Need To Happen

1. Socially Editable Maps

With the advent of the GPS technology in the last couple of years and GPS prices falling (my originally $800 Garmin Nuvi 660 now costs about $200), the biggest frustration I have now is the accuracy of information. In the world of Google Maps and Wikipedia, why is it that I have to wait a whole year for map updates that are obsolete by the time they come out? The Bay Bridge repairs change the roads on an almost monthly basis, for example.

I want to know about road changes as soon as they occur.

The next big thing will be a company, either existing, like Google, or a startup, that will introduce the social aspect into the mapping technology. It will do for maps and GPS what Wikipedia did for text, using the same approach. The details are of course to be worked out.

After this concept becomes successful, GPS companies, will need to support such updates over the air, with a push of a button. This ties in closely with prediction #2 and #4.

2. Open Source GPS Goes Mainstream

Google, the father of Android, needs to get behind this one as well. In fact, an open source GPS device would really be a subset of Android’s functionality, in a dedicated device, so it shouldn’t be that hard. In the far future, it will be built into automobile dashboards but I don’t foresee that happening in the near future.

Combined with prediction #1, this will be a killer device. I really believe that free, open source software is the wave of the future, the natural direction of where software development is heading. Look at Linux in the past few years. Look at Android now. Release one and I’m be #1 in line for one. Those who know me IRL know that I can’t go anywhere without a GPS anymore. I once forgot a GPS at home and ended up in Chili.

3. Photo and Video Cameras Will Go Wireless, Anywhere You Go

Photo and video cameras will be able to post pictures and videos to your home computer, sites like Flickr and YouTube, or email themselves directly to your friends… as soon as you record them, over cellular and Wifi networks.

Cell phones with cameras? Think cameras with cell phones. Some Wifi enabled cameras already came out, like this Canon SD430 (DP review here), and this freshly announced at CES 2009 Sony G3, and now it’s time to go cellular.

The next prediction is a generalization of this concept.

4. Wireless Connectivity Everywhere, In Every Device, Anywhere You Go

Cell phone carriers will sign an increasing amount of deals with companies that want to build devices which connect to the Internet or private networks anywhere you go: GPSes, cameras, digital picture frames, cars, fridges.

The future is completely wireless. People are sick and tired of cords, clutter, and Rick Astley – I know that for a fact and digg says so.

5. Android Will Become Hugely Successful

Alright, I’m dead tired of all the posts about how Google’s Android is going to flop in 2009. I sincerely feel that people with that point of view have never experienced Google and open source, played with Android itself, or realized what exactly an open source phone OS means to developers, manufacturers, and consumers.

Android will become a platform of choice in the next couple of years as everyone starts to realize its limitless possibilities, the OS matures, thousands of new apps get written for it, and new Android phones flood the market. The only Android phone released so far, HTC’s G1, has already received praise from consumers, as well as my friends (even the females ones are excited. And yes, I have female friends).

Motorola announced a few months ago that its new phones are going to run Android and Windows Mobile exclusively. If a company with an R&D department as big as Moto’s stands behind something, you bet your ass they’ve done they’re homework. And what’s not to like? They can now:

• take something that’s supported by the biggest Internet company in the world and the community, for free.
• stop worrying about writing and maintaining their own OS – BAM, tons of money saved.
• concentrate R&D on the hardware.
• if need be, freely develop the features they want that Android doesn’t support yet and contribute them back. Everyone wins, except maybe Symbian developers that don’t have a job anymore.

Android is going to be a revolution. Apple and Android fighting it out will be the best thing that happened to us since the invention of sushi.

6. Open Source HDTVs

The open source revolution will continue and start penetrating the HDTV market.

Personally, I think set top boxes are a waste of effort, time, and money but give me an HDTV that can run Youtube, Vimeo, LastFM, Pandora, and any other site through some sort of a plugin or browser, with a build in Media Center that connects to my computers and goes Wifi, that uses open source, upgradeable software (most likely Linux based), and I will buy it in a heartbeat.

Yes, what I’m saying is if my TV and HTPC did something dirty, I would totally dig their offspring, and so would millions of other people who don’t want/need/care/understand HTPC. TV in general holds a special place in my heart, and make it an even better experience, people.

7. Twitter’s Popularity Will Explode

Now for some social media. Twitter will continue dominating the microblogging arena, similar to YouTube dominating the video space. The site will grow extremely fast, much faster than now.

Competitors, like plurk, will probably secure certain small niches but nobody will be close to touch twitter.

Twitter’s significance in the business world for will be revolutionary. It will become second nature for every company to have one or many twitter accounts as means of connecting to consumers on a personal level. Think an opportunity for mini press releases, many of them, daily, not boring ones, the ones people are actually going to read.

8. Social Media Jobs Will Be In Increasing Demand

SEO, social marketing, and viral campaigns are very cost effective ways of brand promotion, and every company will want to jump on board.

As more and more of them realize this, they will start needing more people with marketing skills of John Chow, Chris Brogan, Nicky Cakes (this dude is hilarious), Jeremy Schoemaker, and the like.

9. YouTube Will Continue To Dominate

This one is easy:

• Biggest user base.
• HD. My favorite greasemonkey plugin YousableTubeFix exposes hi/lo FLVs, MP4, and HD MP4 options. Better quality equals better user experience.
• Chromeless player. YouTube is the only company I know of that has a Javascript controlled chromeless player, which can be embedded in any other flash player. Combined with already existing millions of embeds all over the Internet, YouTube’s popularity isn’t going anywhere any time soon.
• Google backed. Anything is possible when you Google owns you. Competition releases a good feature? YouTube has the resources to one-up it in no time. Need for more servers? YouTube will just buy a few thousand more with the $1.65Bln Google gave it. Traffic explosion? No problem – YouTube has been mooching off the Google CDN for months now.

At the end of 2007, I predicted that in a year we will experience unprecedented HD quality online video. This prediction came true when Hulu and fueled by its success CBS, ABC, NBC, and pretty much every other TV network released their free online TV sites. YouTube launched HD a few months later.

So, my YouTube prediction for 2009 is it will sign deals with major TV and movie networks to finally start showing legal TV episodes and movies. It will become the biggest legal TV and movie hub on the Internet.

10. PostgreSQL Will Gain Popularity

Sun’s buyout of MySQL in 2008 surely sent some shockwaves around. However, I predict that the following factors will contribute to PostgreSQL gaining momentum:

• certain features of MySQL were moved to Enterprise only. Open source enthusiasts don’t appreciate an open source project going partially closed source, so they will be looking for alternative software, like PostgreSQL.
• having spent years with MySQL, I am incredibly frustrated with certain quirks that should have been worked out a long time ago. As software architects look for stable, mature, cost effective, and easy to maintain databases, they will find PostgreSQL increasingly attractive.

Don’t take my word for it. I highly suggest taking a look at this whitepaper comparing MySQL and PostgreSQL. here are some highlights:

• Online operations and reorganization. This is my biggest beef with MySQL. Almost any ALTER table command will prevent writes to the table while it’s being altered. This operation requires double the table size because an ALTER simply makes a copy of the table, a rename, and then drop of the old one. This takes FOREVERRRR. PostgreSQL, on the other hand, supports a lot more online operations that will not take down the table. MySQL promised to support online ALTER TABLE by 2009. Will they keep their word? I highly doubt it.
• PostgreSQL supports function based and partial indexes.
• PostgreSQL supports nested triggers.
• PostgreSQL supports user defined datatypes.
• PostgreSQL has an IP address datatype (woot!).
• MySQL’s default engine doesn’t support online backup and recovery (*cough*, MyISAM, *cough*). Don’t even get me started on MyISAM, which doesn’t support referential integrity, transactions, of any other ACID properties. Yes, I know, there's InnoDB. It’s a lot better. But it’s still not good enough.

If you have spent time interfacing with both MySQL and PostgreSQL, I’d like to hear from you. Everyone I talked to so far who had used both, preferred PostgreSQL.

For all the MySQL fanboys, I was and still am one of you, I use MySQL every day. I’m only trying to open your eyes so you can see the rest of the world.

In fact, I think the only reason MySQL is so much more popular than Postgresql nowadays is luck and marketing by MySQL AB (hey, it sure paid off, I’m not saying anything).

So, here’s to PostgreSQL having a bright future.

Bonus

As a bonus, here is a collection of links to other interesting predictions for 2009:

• http://www.pr-squared.com/2008/12/social_media_predictions_2009.html – a great list of social media predictions.
• http://battellemedia.com/archives/004772.php – 14 online predictions as well as analysis of past years’ predictions.
• http://news.cnet.com/5-predictions-for-2009/ – 5 CNET Digital Home predictions.
• http://www.engadget.com/2009/01/01/predictions-for-2009/ – Engadget’s predictions.
• http://www.freedom-to-tinker.com/blog/felten/predictions-2009 – 38 predictions freedom and technology related predictions.
• http://www.readwriteweb.com/archives/2009_predictions_across_the_we.php – ReadWriteWeb’s predictions across the web.

• HTC Hero Coming To Sprint October 11th! $179.99, Powerful, And Sexy. Here Is Why You Need To Own It
• Sun Definitely Developing A Phone This Year
• The Real Reasons To Use Twitter (Get Over Your Prejudice Already)
• WordPress Developers – How Do You Make A Living [Poll + Discussion]?
• Meet Firefox For Mobile [Video + Feature Highlights + More Info]