I am throwing up a quick post about a relatively cryptic error that Solr started throwing the other day here at Plaxo. After happily running for a few days, I suddenly started getting pages about failed Solr indexing.

Upon closer examination, I saw the following repeatedly in the log file:

catalina.2009-09-18.log:SEVERE: java.io.IOException: directory 'DATADIR/index'
exists and is a directory, but cannot be listed: list() returned null

I tried to see if sending an OPTIMIZE command would help but the server returned the same response.

Digging Deeper

The reason was these errors was quite simple – Solr was running into the system level limit on allowed number of open files (ulimit). This limit can be seen by running

ulimit -n
1024

or simply

ulimit -a | grep 'open files'
open files                      (-n) 1024

This means that if a process tries to open that many files at the same time, the kernel will prohibit opening any more, which in my case caused the Java IOException.

In my case, I haven't been using the Solr OPTIMIZE command for a while, so after a lot of COMMITs, the Solr data got pretty fragmented, thus hitting the open files limit.

The Solution

There are 2 things to be done here:

1. OPTIMIZE Solr more often. When an OPTIMIZE occurs, multiple index files are merged into 1, thus reducing the number of files that need to be opened. However, before you can OPTIMIZE, you have to raise the allowed number of open files (see the next bullet).

2. Set a higher open files limit for the user that runs Solr (in my case, the solr user) – for example to 4096 instead of 1024. One way to do it is by adding a file /etc/security/limits.d/solr.conf with the following contents:

   solr hard nofile 4096
   solr soft nofile 4096

   and then logging out and back in. The file should be automatically loaded, which you can verify by running the ulimit commands from the section above.

Happy Solring!

By the way, here's a really good resource for Solr 1.4 that just came out: Solr 1.4 Enterprise Search. I have this book and it's quite helpful in explaining such topics as multicore setup, search methods, replication, etc.

• How To Show Hidden/Invisible Files In Total Commander, Both Locally And On An FTP Server
• Comparison Between Solr And Sphinx Search Servers (Solr Vs Sphinx – Fight!)
• Watch – A Useful Linux Command You May Have Never Heard Of
• [Perl] Finding Files, The Fun And Elegant Way
• How To Sort Folders The Same Way As Files In Total Commander

• the database
• CPU
• memory
• IO
• disk latency
• network latency
• slow queries

• 300 platforms (300 remote agents collecting data)
• 2,100 servers
• 21,000 services (10 services per server), sounds feasible
• 468,000 metrics (20 metrics per service)
• 28,800,000 metric data rows per day
• larger deployments have a lot more of these (sounds crazy)

• measurement_id
• timestamp
• value
• primary key (timestamp, measurement_id)

• agent collects data and sends reports to server with multiple data points
• server batch inserts metric data points
• if network connection fails, agent continues to collect but server "backfills" unavailable
• when agent reconnects, spooled data overwrite backfilled data points (why not use REPLACE for all inserts?)

• INSERT INTO TABLE (a,b,c) VALUES (0,0,0), (1,1,1),…
• using MySQL batch insert statements vs prepared statements with multiple queries in other databases seems to improve overall performance by 30%
• batch inserts are limited by 'max_allowed_packet'

• set unique_checks=0, insert, set unique_checks=1 (definitely need to make sure data is valid first)
• set foreign_key_checks=0, insert, set foreign_key_checks=1 (same concerns as above)
• Hyperic doesn't use the 2 above

• when regular INSERT fails, retry batch with INSERT ON DUPLICATE KEY syntax
• it's much slower but it allows

• queue metric data from separate agent reports
• minimize number of inserts, connections, CPU load
• maximize workload efficiency
• optimal configuration for 700 agents
• 3 workers
• 2000 batch size seems to work best
• queue size of 4,000,000
• this seems to peak at 2.2mil metric data inserts per minute

• inspired by rrdtool
• lower resolution tables track min, avg, and max
• data compression runs hourly
• size limit 2 days
• every hour, data is rolled up into another table that holds hourly aggregated values with size limit 14 days, then that one gets rolled up into a monthly table, etc
• this is is a good approach if you don't care about each data point

I'm overwhelmed by the amount of "you know"s from the speaker. Parasite words, ahh! Sorry Charles

• measurement data split into 18 tables, representing 9 days (2 per day)
• they didn't want to do more than 2 SELECTs to get data per day, hence such sharding
• oddly, Charles didn't actually use the word 'shard' once
• tables truncated, rather than deleting rows => huge performance boost
• truncation vs deletion
• deletion causes contention on rows
• truncation doesn't produce fragmentation
• truncation just drops and recreates the table – single DDL operation

• every InnoDB table has a special index called the clustered index (based on primary key) where the physical data for the rows is stored
• advantages
• selects faster – row data is on the same page where the index search leads
• inserts in (timestamp) order – avoid page splits and fragmentation
• shows comparison between non-clustered index and clustered index (see slides)

• [MySQL] Deleting/Updating Rows Common To 2 Tables – Speed And Slave Lag Considerations
• MySQL Conference Liveblogging: EXPLAIN Demystified (Tuesday 2:00PM)
• MySQL Conference Liveblogging: Performance Guide For MySQL Cluster (Tuesday 10:50AM)
• Swapping Column Values in MySQL
• MySQL Indexing Considerations Of Implementing A Priority Field In Your Application