File locking issue on NFS home directories (workbench w/ load balancing)

[rundel]

We are trying to configure a cluster of machines to use load balancing and we've gotten stuck on file locking step of the configuration.

User's home directories are mounted from a FreeNAS box via nfs4 but when running the locktester script on two separate machine both claim to able to acquire an advisory and link-based lock of the same testing file (see output below).

This occurs with both the current release and preview release and with trying /etc/rstudio/file-locks configured with either lock-type=linkbased and lock-type=advisory.

Any guidance or alternative approaches to configuration or testing would be greatly appreciated.

Machine 1:

cr173@statistics3 [~]$ /usr/lib/rstudio-server/bin/locktester
RStudio Workbench Lock Test
---------------------------
Host: statistics3.-----
Lock type: Link-based
User: cr173
File: /home/c/cr173/.local/share/rstudio/locktester_file

* Acquired advisory lock
* Acquired link-based lock

If this is the first node on which you are running the
lock tester, run it on another node now. If you see this
message on both nodes, your file system does not support file
locking.
^C

Machine 2:

cr173@linearalgebra3 [~]$ /usr/lib/rstudio-server/bin/locktester
RStudio Workbench Lock Test
---------------------------
Host: linearalgebra3.stat.duke.edu
Lock type: Link-based
User: cr173
File: /home/c/cr173/.local/share/rstudio/locktester_file

* Acquired advisory lock
* Acquired link-based lock

If this is the first node on which you are running the
lock tester, run it on another node now. If you see this
message on both nodes, your file system does not support file
locking.
^C

NFS Details

The following are the mount details for the NFS home share

statfs:/mnt/vol1/home on /home type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=--.--.--.--,local_lock=none,addr=--.--.--.--)

[cole]

I have been struggling to reason about this, and I must admit that I do not understand much about how our file locking works. Have you tested symlinks on your filesystem to make sure that they do in fact work properly? I'm not sure how FreeNAS / ZFS works or what a good test would be... I would create some links and make sure they work in an expected way.

[rundel]

The whole NFS locking process is also a bit of a mystery for me as well, I haven't been able to track down a decent example or test case that I could use outside of the locktester tool.

The files in .local/share/rstudio are definitely being created and are accessible from both machines at the same time. The pid that is written to the file is also clearly being overwritten when the 2nd locktester instance is run and matches the pid on the 2nd machine.

Symlinks work as expected within the NFS mount

[rundel]

I've put in a support ticket for this issue and referenced this post - hopefully it can get escalated to someone on the ide team who can take a look.