I am throwing up a quick post about a relatively cryptic error that Solr started throwing the other day here at Plaxo. After happily running for a few days, I suddenly started getting pages about failed Solr indexing.

Upon closer examination, I saw the following repeatedly in the log file:

catalina.2009-09-18.log:SEVERE: java.io.IOException: directory 'DATADIR/index'
exists and is a directory, but cannot be listed: list() returned null

I tried to see if sending an OPTIMIZE command would help but the server returned the same response.

Digging Deeper

The reason was these errors was quite simple – Solr was running into the system level limit on allowed number of open files (ulimit). This limit can be seen by running

ulimit -n
1024

or simply

ulimit -a | grep 'open files'
open files                      (-n) 1024

This means that if a process tries to open that many files at the same time, the kernel will prohibit opening any more, which in my case caused the Java IOException.

In my case, I haven't been using the Solr OPTIMIZE command for a while, so after a lot of COMMITs, the Solr data got pretty fragmented, thus hitting the open files limit.

The Solution

There are 2 things to be done here:

1. OPTIMIZE Solr more often. When an OPTIMIZE occurs, multiple index files are merged into 1, thus reducing the number of files that need to be opened. However, before you can OPTIMIZE, you have to raise the allowed number of open files (see the next bullet).

2. Set a higher open files limit for the user that runs Solr (in my case, the solr user) – for example to 4096 instead of 1024. One way to do it is by adding a file /etc/security/limits.d/solr.conf with the following contents:

   solr hard nofile 4096
   solr soft nofile 4096

   and then logging out and back in. The file should be automatically loaded, which you can verify by running the ulimit commands from the section above.

Happy Solring!

By the way, here's a really good resource for Solr 1.4 that just came out: Solr 1.4 Enterprise Search. I have this book and it's quite helpful in explaining such topics as multicore setup, search methods, replication, etc.

• How To Show Hidden/Invisible Files In Total Commander, Both Locally And On An FTP Server
• Comparison Between Solr And Sphinx Search Servers (Solr Vs Sphinx – Fight!)
• Watch – A Useful Linux Command You May Have Never Heard Of
• [Perl] Finding Files, The Fun And Elegant Way
• How To Sort Folders The Same Way As Files In Total Commander