Prerequisites and Assumptions
- Debian Lenny stable (not backports) packages are used in this document
- In order to prevent split-brain syndrome, there must be two network paths between the servers. One of these paths will be the normal uplink to the network. The second path should be on a direct link between the servers using a non-routable 172.16 subnet. For convenience, each of the servers participating in the cluster should have eth1 configured with the same two last octets as eth0. For example, a server with an IP address of 10.2.3.4 on eth0 should have 172.16.3.4 bound to eth1.
- It is also a good idea to use LVM to manage the volumes that we are going to be setting up. This allows for future growth. The examples here assume the use of LVM.
- Unless otherwise noted, everything outlined here should be executed/configured on both nodes.
Required Packages
Install the following packages, each of which will be configured later.
aptitude install nfs-kernel-server drbd8-modules-2.6-amd64 drbd8-utils heartbeat-2
NFS Server
-
Create the NFS server share directory.
mkdir -p /mnt/nfs
-
Export the NFS share to the servers that should have access to it in /etc/exports
/mnt/nfs 10.0.0.0/8(rw,sync,no_subtree_check,no_root_squash)
-
Update the STATDOPTS variable in /etc/default/nfs-common. "-p 4000" forces NFS to operate over port 4000-4002, allowing for easy firewall filtering if needed. "-n nfs" forces the NFS server to announce itself as "nfs" instead of the real hostname. This allows for seamless failover between the two NFS servers.
STATDOPTS="-p 4000 -n nfs"
-
Update the RPCMOUNTDOPTS variable in /etc/default/nfs-kernel-server. "-p 4002" is the counterpart of "-p 4000" set in STATDOPTS above
RPCMOUNTDOPTS="-p 4002 -g"
-
Disable the NFS server init script. This will all be handled by Heartbeat below.
/etc/init.d/nfs-kernel-server stop update-rc.d -f nfs-kernel-server remove
DRBD
-
Create new LVM volume for use by DRBD. This example configures a 2GB volume on the vg0 volume group
lvcreate -L 2G -n drbd0 vg0
-
Allow heartbeat to call drbdsetup and drbdmeta with root privileges
chgrp haclient /sbin/drbdsetup chmod o-x /sbin/drbdsetup chmod u+s /sbin/drbdsetup chgrp haclient /sbin/drbdmeta chmod o-x /sbin/drbdmeta chmod u+s /sbin/drbdmeta
-
Create the DRBD configuration in /etc/drbd.conf. Note that the outdate-peer entry should reference the opposite node of where the config file lives.
global { usage-count yes; } common { protocol C; syncer { rate 100M; } disk { fencing resource-only; } } resource nfs { handlers { outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5 -p node1.example.com -r nfs"; } on node1.example.com { device /dev/drbd0; disk /dev/vg0/drbd0; address 172.16.0.10:7788; flexible-meta-disk internal; } on node2.example.com { device /dev/drbd0; disk /dev/vg0/drbd0; address 172.16.0.20:7788; meta-disk internal; } } -
Load the DRBD kernel module
modprobe drbd
-
Initialize the DRBD volume
drbdadm create-md nfs drbdadm up nfs
-
Synchronize the DRBD volume, and create a filesystem by running this on ONE of the servers
drbdadm -- --overwrite-data-of-peer primary nfs mkfs.xfs /dev/drbd0
Heartbeat
-
Create the initial heartbeat configuration in /etc/ha.d/ha.cf
use_logd on keepalive 3 warntime 5 deadtime 7 initdead 10 udpport 694 node node1.example.com node node2.example.com ucast eth1 172.16.0.10 ucast eth1 172.16.0.20 ucast eth0 10.0.0.10 ucast eth0 10.0.0.20 crm respawn autojoin any respawn hacluster /usr/lib/heartbeat/dopd apiauth dopd gid=haclient uid=hacluster
-
Create an authentication key. I like to generate this key using the following
# head /dev/urandom | md5sum 135536ec47dce951d0778fda8102beeb -
Then, add this key to /etc/ha.d/authkeys. This key should be the same on both servers
auth 1 1 md5 135536ec47dce951d0778fda8102beeb
-
Start heartbeat
chmod 600 /etc/ha.d/authkeys touch /etc/ha.d/haresources /etc/init.d/heartbeat start
-
Wait for the cluster to come online before proceeding. You can watch the progress with
crm_mon -i 1 -nr
-
Set the default resource "stickiness" to infinity, so that a resource will not try to fail back to its original location once it becomes available again. This prevents resources from "flapping" between the two cluster servers. This only needs to be executed on one of the cluster nodes
crm_attribute --attr-name default-resource-stickiness --attr-value INFINITY
-
Configured heartbeat to continue trying a failed command. Without this, a cluster will become unresponsive on the first failure or timeout. Sometimes services just need a little time to get going.
crm_attribute --attr-name start-failure-is-fatal --attr-value FALSE
-
Configure heartbeat to manage the NFS share mountpoint, and the floating IP address. Heartbeat automatically propagates the configuration to each cluster member, so this only needs to be done on one node.
cat << EOF | cibadmin -U -p <resources> <group ordered="true" collocated="true" id="nfs"> <primitive class="heartbeat" type="drbddisk" provider="heartbeat" id="nfs-disk"> <instance_attributes> <attributes> <nvpair name="1" value="nfs"/> </attributes> </instance_attributes> </primitive> <primitive class="ocf" type="Filesystem" provider="heartbeat" id="nfs-fs"> <instance_attributes> <attributes> <nvpair name="device" value="/dev/drbd0"/> <nvpair name="directory" value="/mnt/nfs"/> <nvpair name="type" value="xfs"/> </attributes> </instance_attributes> </primitive> <primitive class="ocf" provider="heartbeat" type="IPaddr2" id="nfs-ip"> <instance_attributes> <attributes> <nvpair name="ip" value="10.0.0.30"/> <nvpair name="cidr_netmask" value="24"/> <nvpair name="nic" value="eth0"/> </attributes> </instance_attributes> </primitive> <primitive class="lsb" type="nfs-kernel-server" provider="heartbeat" id="nfs-kernel-server"> <operations> <op name="monitor" interval="3s" timeout="30s"/> </operations> <instance_attributes> <attributes/> </instance_attributes> </primitive> </group> </resources> EOF
NFS Client
- Configure your NFS client to mount to the floating IP address managed by heartbeat, 10.0.0.30 in our example.