| Share |
Updated: September 28th, 2009
The Problem
I am throwing up a quick post about a relatively cryptic error that Solr started throwing the other day here at Plaxo. After happily running for a few days, I suddenly started getting pages about failed Solr indexing.
Upon closer examination, I saw the following repeatedly in the log file:
catalina.2009-09-18.log:SEVERE: java.io.IOException: directory 'DATADIR/index' exists and is a directory, but cannot be listed: list() returned null
I tried to see if sending an OPTIMIZE command would help but the server returned the same response.
Digging Deeper
The reason was these errors was quite simple – Solr was running into the system level limit on allowed number of open files (ulimit). This limit can be seen by running
ulimit -n 1024
or simply
ulimit -a | grep 'open files' open files (-n) 1024
This means that if a process tries to open that many files at the same time, the kernel will prohibit opening any more, which in my case caused the Java IOException.
In my case, I haven't been using the Solr OPTIMIZE command for a while, so after a lot of COMMITs, the Solr data got pretty fragmented, thus hitting the open files limit.
The Solution
There are 2 things to be done here:
- OPTIMIZE Solr more often. When an OPTIMIZE occurs, multiple index files are merged into 1, thus reducing the number of files that need to be opened. However, before you can OPTIMIZE, you have to raise the allowed number of open files (see the next bullet).
- Set a higher open files limit for the user that runs Solr (in my case, the solr user) – for example to 4096 instead of 1024. One way to do it is by adding a file /etc/security/limits.d/solr.conf with the following contents:
solr hard nofile 4096 solr soft nofile 4096
and then logging out and back in. The file should be automatically loaded, which you can verify by running the ulimit commands from the section above.
Happy Solring!
By the way, here's a really good resource for Solr 1.4 that just came out: Solr 1.4 Enterprise Search. I have this book and it's quite helpful in explaining such topics as multicore setup, search methods, replication, etc.
Artem Russakovskii is a San Francisco programmer, blogger, and future millionaire (that last part is in the works). Follow Artem on Twitter (@ArtemR) or subscribe to the RSS feed.
In the meantime, if you found this article useful, feel free to buy me a cup of coffee below.
beer planet is a blog about technology, programming, computers, and geek life. It is run by Artem Russakovskii - a local San Francisco geek who currently works at 
Artem loves commenting on other people's blogs and reading comments on his own, so go ahead and leave one.
[...] [Solr] How To Fix java.io.IOException: directory FOO exists and is a directory, but cannot be listed… (beerpla.net) [...]