Guide: The Silent Corruption Killer – A High-Safety ZFS Backup Architecture

As the “Cheerio Lab” storage needs expanded, I faced a classic homelab dilemma: My high-performance “Hot NAS” is a low-power Mini PC running non-ECC RAM. While fast and efficient, it carries the risk of “Silent Data Corruption”—where a flipped bit in RAM gets written to disk with a valid checksum, effectively “poisoning” the file forever.

The solution wasn’t to throw money at enterprise ECC hardware immediately. Instead, I engineered a ZFS Pull-Backup Architecture that leverages Sanoid/Syncoid and aggressive snapshot retention to act as a “Time Machine” against corruption.

1. The Core Architecture: Split-Brain Deep Storage

The strategy treats the Main NAS as a “Production” environment and the Cold NAS (Deep NAS) as a dual-purpose “Vault.” Because the Deep NAS has significantly more storage than the Hot NAS, it serves two distinct roles:

  • Role A: The Safety Net (Replica)
    • Source: Hot NAS (tank)
    • Destination: Deep NAS (deeptank/nas-replica)
    • Function: A direct mirror of the Hot NAS. If the Hot NAS corrupts a file today, the Deep NAS holds a healthy snapshot from last month.
  • Role B: The Deep Archive (Expansion)
    • Source: None (Exists only on Deep NAS)
    • Destination: Deep NAS (deeptank/archive)
    • Function: Storage for massive files that don’t fit on the Hot NAS (old ISOs, raw footage). These live exclusively on the ECC-protected server.

2. The Secret Sauce: “Raw” Send & Pull Methodology

To minimize the risk of the Hot NAS corrupting data during the backup process, I use a “Pull” configuration. The Deep NAS wakes up, reaches out to the Hot NAS, and grabs the data.

Why “Raw” Send (-w) is Critical: Standard ZFS replication might decompress and re-process data in RAM. By using the raw flag, we tell the Hot NAS to read the encrypted/compressed blocks directly from the disk and stream them over the network without processing them. This bypasses the non-ECC RAM vulnerability almost entirely during transfer.

3. Implementation Steps

A. Configure the Hot NAS (Source)

The Hot NAS needs to take local snapshots so users can recover accidentally deleted files quickly. We use Sanoid for this. File: /etc/sanoid/sanoid.conf

[tank/media]
    use_template = media
    recursive = yes

[template_media]
    # We must be explicit because defaults are hidden
    autosnap = yes
    autoprune = yes
    
    # Keep local snapshots for immediate "oops" recovery
    daily = 30
    monthly = 12
    yearly = 2

B. Configure the Deep NAS (The Vault)

This is where the “Split” architecture matters. We need two different policies: one that receives snapshots (Replica) and one that creates them (Archive). File: /etc/sanoid/sanoid.conf (on Deep NAS)

# 1. THE REPLICA (Incoming from Main NAS)
[deeptank/nas-replica]
    use_template = passive_replica
    recursive = yes

# 2. THE ARCHIVE (Your "Excess" Data)
[deeptank/archive]
    use_template = active_archive
    recursive = yes

# --- TEMPLATES ---

[template_passive_replica]
    # Syncoid brings snapshots from the Hot NAS, so we DON'T create them here.
    autosnap = no
    autoprune = yes
    daily = 60
    monthly = 12
    yearly = 10 

[template_active_archive]
    # This data only exists here, so we MUST create snapshots locally.
    autosnap = yes
    autoprune = yes
    daily = 30
    monthly = 12
    yearly = 10

C. User Permissions (The “Rootless” Approach)

Running backups as root is bad practice. I used ZFS Permission Delegation to allow a dedicated user (cheerio) to perform backups without full sudo access. On the Deep NAS:

# Allow cheerio to manage the backup dataset without sudo
sudo zfs allow -u cheerio compression,create,destroy,mount,mountpoint,receive,rollback,send,snapshot,hold,bookmark deeptank
  • Critical: Do not forget snapshot, hold, and bookmark.
    • For Replica: Syncoid needs these to track replication state.
    • For Archive: Sanoid needs snapshot to create the daily history for your archival files.

D. The “One-Shot” Systemd Service

Instead of leaving the Deep NAS on 24/7 (wasting electricity), I built a systemd service that:

    1. Boots up (via Wake-on-LAN or BIOS timer).
    1. Runs the Backup (Syncoid) for the Replica.
    1. Runs the Cleanup/Creation (Sanoid) for the Archive.
    1. Shuts down.

File: /etc/systemd/system/deep-backup-shutdown.service

[Unit]
Description=Pull Backup from Main NAS and Shutdown
After=network-online.target
Wants=network-online.target

[Service]
User=cheerio
Group=cheerio
Type=oneshot

# The Magic Command
# 1. Syncoid pulls data for the Replica (using "Raw" send)
# 2. Sanoid runs cleanup on Replica AND creates new snapshots for Archive
# 3. Echo statements provide logs
ExecStart=/bin/bash -c 'echo "--- START $(date) ---"; \
 /usr/sbin/syncoid -r --no-privilege-elevation --sendoptions="w" cheerio@nas:tank deeptank/nas-replica --recvoptions="-x sharenfs -x sharesmb -x mountpoint"; \
 echo "--- CLEANUP & ARCHIVE SNAPSHOTS $(date) ---"; \
 /usr/sbin/sanoid --cron; \
 echo "--- FINISHED $(date) ---"'

# The Kill Switch
# Requires 'cheerio' to have NOPASSWD for poweroff in /etc/sudoers
ExecStopPost=/usr/bin/sudo /usr/sbin/poweroff

StandardOutput=append:/var/log/deep-backup-shutdown.log
StandardError=append:/var/log/deep-backup-shutdown.log

4. Why Future Me Is Doing This

  • The “Silent Corruption” Mitigation: By keeping 2+ years of monthly snapshots on the Deep NAS, I have a massive window to discover corrupted files. Even if the Hot NAS writes garbage, the Deep NAS holds the history.
  • Automation Anxiety: The ExecStopPost ensures the server always turns off, even if the backup crashes, saving money on the electric bill.
  • Least Privilege: If the cheerio account is compromised, the attacker can only mess with ZFS datasets, not the entire root OS.

Troubleshooting Tips

  • Permission Denied? Check zfs allow. If snapshot is missing, Sanoid will fail to protect your deeptank/archive data.
  • Server won’t shutdown? Check /etc/sudoers. The user cheerio must have NOPASSWD: /usr/sbin/poweroff.
  • Disk filling up? Check if sanoid.timer is running on the Hot NAS, and ensure sanoid --cron is explicitly called in the Deep NAS script.