[Solr] How To Fix java.io.IOException: directory FOO exists and is a directory, but cannot be listed: list() returned null
Updated: June 11th, 2015
I am throwing up a quick post about a relatively cryptic error that Solr started throwing the other day here at Plaxo. After happily running for a few days, I suddenly started getting pages about failed Solr indexing.
Upon closer examination, I saw the following repeatedly in the log file:
catalina.2009-09-18.log:SEVERE: java.io.IOException: directory 'DATADIR/index' exists and is a directory, but cannot be listed: list() returned null
I tried to see if sending an OPTIMIZE command would help but the server returned the same response.
The reason was these errors was quite simple – Solr was running into the system level limit on allowed number of open files (ulimit). This limit can be seen by running
ulimit -n 1024
ulimit -a | grep 'open files' open files (-n) 1024
This means that if a process tries to open that many files at the same time, the kernel will prohibit opening any more, which in my case caused the Java IOException.
In my case, I haven't been using the Solr OPTIMIZE command for a while, so after a lot of COMMITs, the Solr data got pretty fragmented, thus hitting the open files limit.
There are 2 things to be done here:
- OPTIMIZE Solr more often. When an OPTIMIZE occurs, multiple index files are merged into 1, thus reducing the number of files that need to be opened. However, before you can OPTIMIZE, you have to raise the allowed number of open files (see the next bullet).
- Set a higher open files limit for the user that runs Solr (in my case, the solr user) – for example to 4096 instead of 1024. One way to do it is by adding a file /etc/security/limits.d/solr.conf with the following contents:
solr hard nofile 4096 solr soft nofile 4096
and then logging out and back in. The file should be automatically loaded, which you can verify by running the ulimit commands from the section above.
By the way, here's a really good resource for Solr 1.4 that just came out: Solr 1.4 Enterprise Search. I have this book and it's quite helpful in explaining such topics as multicore setup, search methods, replication, etc.
In the meantime, if you found this article useful, feel free to buy me a cup of coffee below.