Short answer: Qumulo Continuous Replication keeps a read-only copy of your directories on a second cluster. To fail over, you stop writes on the primary, let the last replication job finish, make the target directories writable, publish them with the same shares and exports, and move client access to the secondary. Failback reverses these steps and decides what happens to data written on the secondary in the meantime.
What Continuous Replication does and doesn’t do
Continuous Replication takes a snapshot of a source directory and transfers it to a target directory on another cluster. After the first run, it only transfers changed regions of modified files. The target directory stays read-only while the relationship is active.
Replication copies data. It does not copy the configuration clients need to reach that data: SMB shares, NFS exports, quotas, and the floating IP addresses clients connect to. It also doesn’t switch anything over by itself. A failover is a sequence of administrative steps on two clusters, and the order matters.
Before you need it: three preparations
- Separate client access by protocol in DNS. Use one FQDN for NFS exports, one for SMB shares connected directly, and one for SMB shares published through DFS. Each FQDN resolves to its own floating IPs. This lets you move NFS access by IP without touching DFS clients, which fail over differently (see our article on DFS failover).
- Make replication targets recognisable. Replicate into a directory named after the source cluster, for example
/Mediaon the source to/qumulo-prod-01/Mediaon the target. During an incident, nobody has to guess which directory is a copy of what. - Keep shares, exports and quotas on the secondary up to date. If a share exists only on the primary, clients have nothing to connect to after failover. This is the most common reason a failover “works” on the storage side and still fails for users.
Planned failover (both clusters online)
Use this for maintenance windows and failover tests.
- Check replication health. On the source cluster, confirm that every relationship you want to fail over is enabled and that the last recovery point is recent.
- Stop client writes on the primary. Set the source directories read-only for clients, for example by switching share permissions and export rules to read-only. Anything written after this point would not reach the secondary.
- Run a final replication job and wait for it to finish. Only then is the target identical to the source.
- Make the target directories writable on the secondary. In the Web UI or with
qq replication_make_target_writable --id <relationship-id>. This reverts the target to its latest recovery point and makes it writable. - Take the source shares on the primary out of service. Rename or remove them, so clients that still resolve the old address cannot write to the primary by mistake.
- Publish the data on the secondary. Shares and exports must point to the replicated directories, with the same names, permissions and quotas as on the primary.
- Move client access.
– NFS and directly connected SMB clients: remove the floating IPs from the primary’s network configuration (Web UI orqq network_modify_network) and add them to the corresponding network on the secondary. Create that network on the secondary first if it doesn’t exist.
– SMB clients usually have to reconnect, because their Kerberos ticket was issued for the primary.
– NFS clients reach the secondary under the same address. Whether they continue on their own or need a remount depends on NFS version and mount options (see below).
– DFS clients: remove the DFS-related floating IPs from the primary without moving them; DFS then redirects clients to the secondary target. - Verify from a client. Open a share, write a test file, check permissions and quotas.
NFS mount options decide whether clients must remount
NFSv4.1 with soft mounts is the only combination in which clients continue after a failover without a remount. With every other combination, plan the remount as a step in your runbook.
| NFS version | Mount | Remount after planned failover (primary online) | Remount after failover in an outage (primary down) |
|---|---|---|---|
| 3 | hard | Yes | Yes |
| 3 | soft | Yes | Possibly |
| 4.1 | hard | Yes | Possibly |
| 4.1 | soft | No | No |
The trade-off: with a hard mount, a client waits indefinitely until the server answers again. With a soft mount, it gives up after its retry timeout and returns an I/O error to the application. Before you switch write-intensive workloads to soft mounts, check how those applications handle such errors.
Example for NFSv4.1 with a soft mount on Linux:
mount -t nfs -o vers=4.1,nconnect=16,soft,tcp,rdirplus,rsize=1048576,wsize=1048576 nfs.example.com:/Sales /mnt/sales
On macOS, add intr and readahead=128:
mount -t nfs -o vers=4.1,nconnect=16,intr,soft,tcp,rdirplus,readahead=128,rsize=1048576,wsize=1048576 nfs.example.com:/Sales ~/Sales_NFS
nconnect=16 opens several TCP connections per mount and raises throughput; it has no effect on failover behaviour. Use the FQDN reserved for NFS exports, so the mount follows the floating IPs to the secondary.
Unplanned failover (primary is down)
When the primary is unreachable, steps 2, 3 and 5 are not possible. You make the targets writable, publish the data and add the floating IPs to the secondary. Two risks remain:
- Data loss window. The secondary only has what was replicated up to the last recovery point. Writes after that are on the failed primary.
- The primary comes back. It brings its floating IPs and writable source directories with it. You then have duplicate IP addresses in the network and two writable copies of the same data. Remove the floating IPs and set the source directories read-only on the primary immediately after it boots, before clients reconnect.
Failback
Failback reverses the failover once the primary is healthy again. Data written on the secondary in the meantime is replicated back first, then client access returns:
- On the secondary, reverse the relationship with
qq replication_reverse_target_relationshipand reconnect it from the primary, so the changes flow back. - When the primary has caught up, stop writes on the secondary and run a final replication job.
- Make the primary directories writable and reverse the relationship again to restore the original direction.
- Move the floating IPs back, restore the original share names on the primary and set the secondary read-only.
Failback typically takes longer than failover, because changed data has to travel back before you can switch.
Where failovers go wrong
- A share, export or quota exists on the primary only.
- Floating IPs are moved before the target is writable, so clients connect to a read-only copy.
- The primary returns after an outage and nobody removes its IPs.
- Clients are switched back before the data written on the secondary has reached the primary.
- The runbook was written once and never tested, so it no longer matches the clusters.
All of these have the same cause: many manual steps on two systems, under time pressure, by whoever is on call.
How yuzuy automates this
The yuzuy software runs as a virtual machine (VM) in your data center and orchestrates these steps through the Qumulo API. You configure a failover relationship between two clusters once: which floating IPs move to the secondary (IP Failover) and which are only removed from the primary (DFS Failover).
A failover is then one click or one API call. yuzuy finishes running replication jobs where possible, saves the replication state, sets the source read-only, renames the source shares, moves or removes the floating IPs, creates missing networks on the secondary and makes the targets writable. Every step is guarded by a technical check, and the history shows each step with its result.
After an unplanned failover, yuzuy checks every minute whether the primary is back and removes its floating IPs, so duplicate addresses exist for one minute at most. For failback, yuzuy replicates the changed data back first and asks you to confirm the final switch; it can notify you by email when the data is back.
Because shares, exports and quotas are kept in sync between the clusters (see our article on synchronization), the secondary is ready when you need it. That makes regular failover tests realistic: one click to fail over, one to fail back.
Failover / Failback · Request a demo
FAQ
Does Qumulo fail over automatically? Qumulo provides the building blocks: replication, writable targets, relationship reversal and floating IPs. Switching client access between two clusters is a sequence of administrative steps that you run yourself or automate.
What happens to data written on the secondary during a failover? It stays on the secondary until failback, when it is replicated back to the primary before clients switch.
Do I lose data in an unplanned failover? Possibly the changes since the last recovery point. How much depends on the change rate and on how far replication was behind when the primary failed.
Do NFS clients need to remount after a failover? With NFSv4.1 and soft mounts, no. With NFSv3, or with hard mounts, usually yes, at least after a planned failover.
