One would give up and accept this futility but I had 2 reasons to keep trying to figure out why:

• my co-worker's battery would consistently hold twice as much charge as mine – by the time I was at 50%, he was at 75%
• a wide range of responses on Internet forums and blog posts suggested some people experienced excellent battery life, while others, like me, did not have as much luck

I thought I tried everything:

• I kept WiFi and Bluetooth off when I didn't need them (I even got an app that turned WiFi on and off on schedule – Timerriffic; unfortunately until Hero gets Android 2.0 we won't see automatic Bluetooth scheduling)
  
• I tried to use the Spare Parts app which provided insight into battery usage history. I did not, however, find out anything significant
  
• I tried to avoid using GPS altogether – at one point Spare Parts told me GPS usage by Google Maps, Camera, and Bump were 5-6 hours each, even though I turned the GPS off. I think it's a bug in Spare Parts itself or what Android was recording but it made me paranoid.
• I switched to using Handcent SMS due to the bug in default Message app (which was later fixed in the firmware upgrade).
• I got a task manager (Advanced Task Manager) to see what apps are running and make sure especially heinous is eating up my resources. It would also automatically kill long running apps and certain apps I know I don't use and which I can't uninstall, such as IM, Gmail (I use Mail), and Finance.
• On some days, I stopped using the phone altogether, just to see how long it would last. The result, however, was the same.

Nothing worked. Consistently, my phone would run down to 80% by the time I got to work and by about 3pm, it would be at 50%.

The Solution

Finally, a few days ago, when comparing my phone with my co-worker's, I noticed he didn't set up automatic email checking, opting in for the manual method instead. I, on the other hand, set up my email the day I got my Hero and never revisited the settings since. "How much power can downloading emails suck out?" I thought. Boy, was I wrong…

Turns out a LOT. I had 3 email accounts and they were checked at the following intervals:

• my gmail was being checked every 5 minutes – I wanted near real-time notification but could not even fathom the consequences at the time
• my beerpla.net account was hit every 15 minutes
• my Plaxo work Exchange email fired every 30 minutes

In order to change these intervals, load up each email account's inbox, click Menu, find Settings, go to Send & Receive and then Set Download Frequency.

After I set each email account to check every hour instead, my battery life improved dramatically. To be fair, I watched the battery use for the last 3 days without changing my usage habits and recorded the results.

Here they are:

• by the time I left for work in the morning, before the change I would oftentimes be between 85 and 90% battery. After the change, I was at 95-97%.
• by lunch time, my phone would usually run down to 50-60%. After the change, the battery had 75-80% juice.
• by 9-10pm, after I came back from work, I would usually need to put the phone on a charger because it was <15%. After the change, I was at least at 50%!

The Side Effect

Lately, I started to realize that having email notifications pop up this often was actually more distracting than useful. I was losing concentration and constant context switching made me less productive.

The fix for my battery problem was, incidentally, also the fix for my concentration problem.

Conclusion

Be very much aware of your email polling intervals or they will suck all the life out of your phone.

I would love to hear about your experiences with tweaking email settings and the effects those tweaks had on your battery life, as well as other dramatic Android battery saving tips. Feel free to share in the comments.

• How To Fix Palm Pre's "Error Sending Mail" Problem
• 2 Gmail Tricks I Bet You Didn't Know About
• Sprint And HTC Release The First HTC Hero Firmware Update 1.56: Fixes Major Bugs, Adds Small Tweaks
• The Not So Good, The Bad, And The Ugly: My List Of 20+ Problems With HTC Hero
• [Updated x3] Goodbye Outlook, I Don't Need You Anymore – Gmail Now Lets You Paste Images Directly From Clipboard

SimplifyMedia is an extremely useful program, which allows super easy song and playlist sharing via streaming between multiple computers. It also works as a Winamp plugin, so the shared songs simply show up inside Winamp's own media library – each computer sharing music appears under the special "Shared" tab. And the beauty is – there is no need to muck around with network settings, open ports, or even touch your router.

Here's a screenshot of how it's supposed to work:

The Problem

Since I recently installed Windows 7 64bit on my laptop, I wanted to set up SimplifyMedia on it and listen to some tracks in my favorite music player Winamp. However, after I downloaded and installed it and restarted Winamp, the Shared tab did not show up. The SimplifyMedia installer did tell me that a 64 bit version of Windows isn't explicitly supported but mentioned it should still work fine.

The Solution

After reinstalling, analyzing, and comparing with my other computers, I found that the SimplifyMedia installer didn't copy the needed Winamp plugin files, namely ml_simplify.dll and ml_simplifyexp.dll. After I copied them over into the Winamp/Plugins directory and restarted Winamp, the Shared tab showed up like a charm.

• Supercharge Your GNU Screen With A Power "Taskbar" And Never Feel Lost Again
• How To Download And Install Windows Media Player 11 Bypassing WGA
• How To Update Eclipse From Galileo (3.5) To Helios (3.6) In-Place Without Reinstalling
• Jocko Sims’ (Panic’s) Rap On Starz Show Crash – Head Up (MP3)
• How To View A Specific SVN Revision In Your Browser

My fiancée Emilie has had her Palm Pre for about 2 weeks now, and one specific problem that started showing up last week has been incredibly annoying, causing us all kinds of trouble.

After Emilie got the phone, she added 3 email accounts – Gmail, Hotmail, and Yahoo. One day, after sending an email through the Pre, a popup showed up that said "Error sending" followed by the subject of that specific email. Swiping the error away didn't help – it just came back 5 minutes later. In fact, it has been coming back every 5 minutes of every day since, lighting up the screen each time. Thus, not only it was annoying, but it was running down the battery like crazy – a full charge lasted between 1/2 and 3/4 of the day with very mild usage.

I deleted the problematic message from the Outbox a long time ago. The error didn't disappear and kept popping up. Additionally, whichever Outbox clicking on the annoying popup led to was having trouble refreshing, even though it wouldn't display what the error actually was – just a yellow exclamation sign would show up. Refreshing the folder didn't help and would take an unusually long time.

Well, finally, I got sick of waiting for Palm to update its OS with possible fixes and set out to find one myself.

Turns out a ton of people are having the same issue, as evident in all Palm Pre forums. For example, Palm.com forum or Precentral.net forum. I really didn't like the common suggestion to delete and re-add the problematic account because

• that just most likely temporarily pushes the problem back as I'm sure it would come back again to haunt me
• this could potentially erase or disconnect all contacts associated with that account
• my engineering mind would be left puzzled
• I wasn't sure which account was having the problem (later it turned out the culprit was Yahoo)

Surprisingly, I found the clue that led me to the solution on Yahoo Answers – yeah, that place where nothing intelligent is allowed to exist (with certain exceptions).

The Solution

To make a long story short, Pre messed up Yahoo's outgoing mail settings. My particular problem was the SMTP (that's the outgoing mail server) port was set to 995, which is in fact an incoming POP3 mail server port. I don't know how it happened or if it even worked in the first place but the fact is: the right outgoing port for secure (SSL) SMTP connections is 465. Also, for some reason the SMTP username was blank but the password was filled in. WTF, Pre?

When I changed SMTP port to 465 and entered the username, as specified in that Yahoo Answers post (without the @yahoo.com part), the error message went away and the Outbox refreshed just fine. BTW, my Pre has the incoming mail server set to POP3, while the Yahoo Answers post was set to IMAP – again, why the differences? I'd frankly prefer IMAP anyway.

To get to your mail account settings, open up your email application, go to the Email menu in Pre's top left corner, tap Preferences & Accounts, then tap the offending account, scroll down, and tap Change Login Settings. When you are done, just swipe back (half swipe from right to left in the gesture area). Here's a handy screenshot I took of the relevant area:

-&gt;

If you are experiencing the same symptoms, check ALL your mail settings, not just the ones I mentioned. Who knows what else your Pre may have messed up?

Update: if you are having trouble sending your mail only while on WiFi, it is possible that your home broadband router is blocking the needed ports. Try to edit the router configuration and open the ports used by your Pre's mail configuration – hopefully the problem will be solved.

Conclusion

Palm, are you hearing us? We are not happy with the bugs and the unhelpful error messages (or lack thereof altogether)! 3 measly updates in the last 2 months is quite pathetic. I hope you are working on a much bigger update behind the scenes.

• How I Doubled My Android Phone's (HTC Hero) Battery Life or Just How Much Email Polling Affects Your Battery
• Top 10 Reasons Why Digsby ROCKS
• 2 Gmail Tricks I Bet You Didn't Know About
• How *Not* To Implement A Web Application That Handles External Authentication, Using BeTwittered.com As An Example
• How To Fix Redrawing Problems In Your VNC

If you, like me, are building or thinking of implementing a MySQL-powered application that has any need for prioritizing selecting certain data over other data, this article is for you.

Example

As a real world example, consider a queue-like video processing system. Your application receives new videos and processes them. The volume of incoming videos can at times be higher than the processing rate because the process is CPU bound, so occasionally a pretty long queue may form. You will try to process them as fast as you can but…

…suddenly, you need to process a video somewhere in the middle of the queue or an important video enters and needs immediate attention. What do you do?

An obvious solution is implementing a simple priority system where each item has a numeric priority field. Now you can sort first by priority from highest to lowest and then by id within the highest priority. Important and urgent items get a their priority changed to something higher and get processed first. There is only one problem.

Problem

The problem is pretty serious – let’s take a look at the SELECT statement. Before selecting, I’ve added 19 random rows to have some data to work on.

SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;

What kind of index would you put on this table to speed up this query? You do want to add a proper index, don’t you? DO YOU? Ok, good.

Here’s what happens without any indexes:

mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows | Extra          |
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
|  1 | SIMPLE      | queue | ALL  | NULL          | NULL | NULL    | NULL |   19 | Using filesort |
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
1 row in set (0.00 sec)

Using filesort, ugh, of course, due to sorting without an index.

Let’s see, how about a combined index on (priority, id)?

mysql> ALTER TABLE `queue` ADD INDEX `priority_id`(`priority`, `id`);
Query OK, 19 rows affected (0.05 sec)
Records: 19  Duplicates: 0  Warnings: 0
 
mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
| id | select_type | table | type  | possible_keys | key         | key_len | ref  | rows | Extra                       |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
|  1 | SIMPLE      | queue | index | NULL          | priority_id | 5       | NULL |   19 | Using index; Using filesort |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
1 row in set (0.00 sec)

Better because an index is being used but not very good because filesort is still present. “Of course!”, you slap yourself on the forehead. The first ORDER BY uses a DESCENDING order, and our key is in ASCENDING order.

So, let’s add the proper key with the right ordering instead.

mysql> ALTER TABLE `queue` DROP INDEX `priority_id`;
Query OK, 19 rows affected (0.05 sec)
Records: 19  Duplicates: 0  Warnings: 0
 
mysql> ALTER TABLE `queue` ADD INDEX `priority_id`(`priority` DESC, `id`);
Query OK, 19 rows affected (0.06 sec)
Records: 19  Duplicates: 0  Warnings: 0

mysql> EXPLAIN SELECT * FROM queue ORDER BY priority DESC, id LIMIT 1;
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
| id | select_type | table | type  | possible_keys | key         | key_len | ref  | rows | Extra                       |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
|  1 | SIMPLE      | queue | index | NULL          | priority_id | 5       | NULL |   19 | Using index; Using filesort |
+----+-------------+-------+-------+---------------+-------------+---------+------+------+-----------------------------+
1 row in set (0.00 sec)

What the deuce? This is the same result as with the previous index. Time to dig up the documentation.

Here is what the MySQL manual has to say under the ORDER BY optimization section:

MySQL cannot use indexes to resolve the ORDER BY, although it still uses indexes to find the rows that match the WHERE clause … if you mix ASC and DESC:

SELECT * FROM t1 ORDER BY key_part1 DESC, key_part2 ASC;

Moreover, to confuse the user even more, the index creation command accepts the DESC instruction, without actually honoring it, as specified in the CREATE INDEX section:

An index_col_name specification can end with ASC or DESC. These keywords are allowed for future extensions for specifying ascending or descending index value storage. Currently, they are parsed but ignored; index values are always stored in ascending order.

So, after so many years MySQL still doesn’t support such basic functionality – you are either stuck with a query that uses filesort or have to look for a workaround.

Solution

Since it’s not possible to mix order directions, the solution is then to change the meaning of the priority column to match your needs. Thus, in the new approach priority 1 is higher than priority 10, and the application logic needs to accommodate to that. If you caught this while the application is still young, the code may be easy to change, but otherwise it could be a major pain in the butt.

Conclusion

The moral here is: plan your queries ahead and don’t mix and match DESC and ASC ordering as MySQL will not be able to use an index to resolve it. Do it even sooner if you’re putting lots and lots of data into your tables.

• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• Swapping Column Values in MySQL
• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations
• MySQL Conference Liveblogging: MySQL Performance Under A Microscope: The Tobias And Jay Show (Wednesday 2:00PM)
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)

• Between 1 and 100 UPDATE queries are constantly running on the master in parallel. If the slave IO is only fast enough to handle 50 of them without lagging, as soon as 51 start running, the slaves starts to lag.
• A more common problem is when one query takes an hour to run (let's say, it's an UPDATE with a big WHERE clause that doesn't use an index). In this case, the query runs on the master for an hour, which isn't a big problem because it doesn't block other queries. However, when the query moves over to the slaves, all of them start to lag because it plugs up the single replication thread.

There is nothing you can do right now to fix the way MySQL handles replication. If the replication threads could run in parallel, I'm guessing horrible things would happen to the data integrity due to race conditions, canceled queries, slave restarts, differences in query execution times due to server load and configuration, etc. Replication is already an asynchronous, prone to getting out of sync business (hint: use maatkit tools by Baron Schwartz and specifically mk-table-checksum and mk-table-sync to sync up your slaves).

In order to see if a slave is lagging, execute the 'show slave status' command and look for the Seconds_Behind_Master value. The way this value is calculated can be slightly ambiguous and unclear, so I'll explain. It is simply a difference between the 2 timestamps – the time of the last received (and queued up in the relay log) query that already executed on the master and the time of the currently executing query on the slave. Thus this value is not real time (it is possible to catch up to the master much faster); it's an approximation, or special metrics if you will, that helps point out problems.

So what can you do if you start hitting replication lag? This is the ultimate question, and the answer depends on your application. Here are the things I came up with after dealing with MySQL for a few years (there are undoubtedly other techniques, but these all come from my own experience):

1. Normalize your data, if it is not already. Non-normalized tables lead to repetition and is generally considered bad practice. More data – more IO in most cases. There can be cases, however, where you can normalize too much. Having JOINs is much slower than not having them, and it can hurt your queries if you JOIN a lot. Finally, the extreme case is mentioned at highscalability.com: How I Learned to Stop Worrying and Love Using a Lot of Disk Space to Scale. "You–pause for dramatic effect–duplicate data instead of normalize it. *shudder*". Flickr is provided as an example.
2. Shard (meaning, slice) your data, horizontally and vertically. For example, you can horizontally partition by some sort of key, hash, username, or other properties. You can also partition vertically by moving out some table columns into other databases. As an example, if you had a database of videos, storing view counts, number of favorites, etc. is OK but if these fields receive a lot of frequent updates, you are bound to have slave lag. Instead, separate these into a dedicated stats table(s). You don't have to shard all of your data – even sharding the most active bits helps immensely (for example, you can choose to shard your stats tables and leave the main one alone).
3. Upgrade machines running MySQL (first slaves, then master, for the reasons given above). 99% of the time, disk IO is the bottleneck, CPU being the other 1%. Move to RAIDed setups (RAID10 or RAID0) with 6-10 15K RPM SCSI or SSD drives. Add a lot of RAM. Make sure you're running a 64 bit OS if you have more than 3GB of RAM, so that the mysql process may utilize more of it. My search for the best MySQL server under $10K may be of some help here.
4. Separate your applications onto different MySQL instances. If you are running separate applications A, B, and C that don't depend on each other, consider giving them their own machines, otherwise a single long-running UPDATE or INSERT query in application A will delay all writes by application B and C. This is actually quite common – even though the server may not appear to be loaded, the annoying slave delay will still show its cowardly tail. I want to highlight this again: the replication thread is shared between all databases on the server.
5. Another solution to (4) is multiple MySQL instances, granted that the MySQL machine isn't generally overloaded already. In that case, installing more than 1 mysql daemons would separate replication threads and allow running multiple applications, like A, B, and C on one machine, without affecting each other. MySQL sandbox achieves just that – it is my preferred solution.
6. Split up longer running queries into shorter ones. This should be pretty straightforward – a single query on 10 million rows may run a few hours. Splitting it into batches of 50,000, for example, will give other queries a chance to run in between. Of course, you should take care of data integrity and generally double check what you are doing.
7. Don't overload the same slave by sending all queries to it, as it will just make the matter worse. You can round-robin the queries using either round-robin DNS (eww), round-robin within the application logic (better), smarter application logic, like checking slave load and status from time to time, or my personal favorite – using MySQL proxy and having it pick the least lagging slave for you. An official solution utilizing mysql proxy, called MySQL load balancer, is apparently in the works (I was promised beta access but haven't got it so far).

As a bonus, I wanted to throw in this idea of helping minimize a certain corner case cause of slave delay and feed it to the hungry MySQL minds. I'm not sure if it is mentioned anywhere else, as I have not Googled it. If it's a widely known fact, then I will consider this post as just adding my vote to the usefulness of the technique.

Well, there you have it. Comments are open, so feel free to share your own replication strategies and thoughts about mine.

• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations
• MySQL Conference Liveblogging: Portable Scale-out Benchmarks For MySQL (Wednesday 10:50AM)
• MySQL Conference Liveblogging: Performance Guide For MySQL Cluster (Tuesday 10:50AM)
• MySQL Conference Liveblogging: Optimizing MySQL For High Volume Data Logging Applications (Thursday 2:50PM)
• Swapping Column Values in MySQL