Change Gluster Volume Connection Timeout for GlusterFS Native Client

• Updated July 1, 2018


After setting up a two node replica Gluster cluster to use as a web content backend, I began testing what would happen when I brought down one of the Gluster nodes. The web nodes access the Gluster cluster using the GlusterFS native client, and I expected them to quickly see that one of the Gluster nodes was down and begin serving content from the healthy Gluster node, but that was not happening. My test simply involved rebooting one of the Gluster nodes. That node would come back online within about 30 seconds, but within those 30 seconds my website was down.

I discovered the default connection timeout for a Gluster client to flag a Gluster node as down is 42 seconds. You would think this would be easy enough to change, but it turned out to be a bit trickier.

First, I stumbled upon the network.ping-timeout Gluster volume setting. Looking at the documentation, this value is set to 42 seconds. On one of the Gluster nodes, I changed it to 5 seconds with the following command (gv0 is the Gluster volume):

gluster volume set gv0 network.ping-timeout 5

However, this did not do anything.

Second, on both of the Gluster nodes, I tried applying a similar configuration change to the main Gluster configuration file at /etc/glusterfs/glusterd.vol by adding option network.ping-timeout 5:

volume management
    type mgmt/glusterd
    option working-directory /var/lib/glusterd
    option transport-type socket,rdma
    option transport.socket.keepalive-time 10
    option transport.socket.keepalive-interval 2
    option transport.socket.read-fail-log off
    option network.ping-timeout 5
#   option base-port 49152
end-volume

After restarting the glusterd service on both Gluster nodes, nothing had changed.

Finally, on both of the Gluster nodes, I found that I needed to add a specific volume configuration block for the Gluster volume in /etc/glusterfs/glusterd.vol and set option ping-timeout 5 inside it:

volume management
    type mgmt/glusterd
    option working-directory /var/lib/glusterd
    option transport-type socket,rdma
    option transport.socket.keepalive-time 10
    option transport.socket.keepalive-interval 2
    option transport.socket.read-fail-log off
#   option base-port 49152
end-volume

volume gv0
    type protocol/client
    option ping-timeout 5
end-volume

After restarting the glusterd service on both Gluster nodes, I was able to reboot one of the Gluster nodes (with a healthy node still online), and, within about 5 seconds, continue serving content from my website.

References

If you found this post useful and would like to help support this site - and get something for yourself - sign up for any of the services listed below through the provided links.

The following links are part of affiliate programs, and I will receive a referral payment from any of the services you sign-up for.

  • Get Faster Shipping: About to order some stuff on Amazon but want to get more value out of the money you would normally pay for shipping? Sign-up for a free, 30-day trial of Amazon Prime to get free two-day shipping, access to thousands of movies and TV shows, and more.
  • Start Investing Now: Looking to start investing some of your saved income? Get started by opening an account with Webull, where upon sign-up and opening a brokerage account with an initial deposit of $100 or more you will receive free stock(s). The promotions are always changing, so be sure to click the link to see the latest.

Thanks for reading and take care.