Opensearch failed to obtain node locks error

My cluster would not start up after killing the containers using the force, which means elasticsearch / opensearch did not get a chance to cleanly shutdown and remove the lock files. The error looks like this:

[][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-node] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/opensearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
...
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/opensearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing
... 6 more

The fix is to remove the node.lock file in the PV data location nodes/0/

On the instance:

rm /usr/share/opensearch/data/nodes/0/node.lock

For me I just nuked it from outside:

rm  /mnt/iscsi/ssd/k8s/opensearch/nodes/0/node.lock

Comments

comments powered by Disqus