Linux Enterprise Cluster: Build a Highly Available Cluster with Commodity Hardware and Free Software

The NFS was originally conceived and built to manage distributed access to a single data storage device, with separate programs added to provide a cooperative lock arbitration method. Because the filesystem and its locking method were implemented separately, the original NFS developers felt that they were providing a generic means of network lock arbitration that could be used in conjunction with any network file system. However, the network lock manager (NLM), as it became known, has only been widely used as a lock arbitration method by NFSv3 servers and NFS version 3 clients. The NFSv4 protocol does not use a separate daemon or protocol for locking, but many of the concepts are the same, so we'll introduce NFS locking by discussing the NFSv3 NLM.

The NLM consists of two daemons called statd and lockd. Both daemons are supposed to be running all the time on the NFS server and on all NFS clients to ensure that everyone agrees upon what is locked and which programs or processes own the locks. The statd daemon running on both the client and the server keeps a list of the hosts that are currently holding or granting locks (NFS clients, if the machine is acting as an NFS server, and NFS servers, if the machine is an NFS client^[5]). The lockd daemon on the NFS client is in charge of making a lock request over the network by talking to the lockd daemon running on the NFS server.

Let's examine these two daemons more closely.

statd

• In a cluster, statd, also called rpc.statd, runs on each cluster node in case the node crashes while it holds a lock on the NFS server. If this happens, the rpc.statd program on the cluster node will notify the NFS server when the cluster node is rebooted that it is now operational again. statd on the NFS client knows that it should do this, because it writes the name of each NFS server to a local disk drive the first time a process running on the cluster node tries to lock a file on the NFS server. When the NFS server receives this notification after an NFS client reboots, it assumes all of the processes that were holding locks from this cluster node (or NFS client) are no longer running, and it releases them.

lockd

• The Linux kernel knows when a Posix fcntl lock request is for a file or portion of a file stored on an NFS server, because the file is stored on an NFS-mounted filesystem. If the Linux machine is an NFS client, the kernel knows it needs to pass the lock request to the locally running lockd daemon.^[6] The lockd daemon on the NFS client then contacts the lockd daemon running on the NFS server and requests the lock. The lockd daemon on the NFS server then asks its own kernel for the lock on behalf of the NFS client.

  If the same program or process asks for the same lock^[7] more than once, it will be granted the lock each time; the kernel doesn't consider this to be a violation of the lock. Before lockd became a part of the kernel, the ability for a single process to be granted the same lock more than once meant that the lockd daemon had to track lock requests that it had made on behalf of the NFS clients and deny any conflicts before asking the kernel for the lock. Were it not to do so, lockd could unknowingly ask for, and be granted, the same lock twice on behalf of two different NFS clients.

^[5]Actually, it always keeps a list of both if the machine is both an NFS client and an NFS server.

^[6]The lockd daemon actually runs inside the kernel as a kernel thread on Linux and is called klockd, but for our purposes, we'll talk about it as if it were outside the kernel.

^[7]At least this is true for Posix fcntl locks in NFS version 3.