Last updated: January 28th, 2023 - Referencing OpenZFS v2.1.7
ZFS is a magical filesystem created by Sun Microsystems, with an initial release of 2006. While ZFS may be almost 20 years old by this post (over 22 if you count when they started development), it was ahead of it's time then, and even now, with newer filesystems like BTRFS, APFS, and ReFS, it is still ahead of its time. In just about every aspect, ZFS has more features, better data integrity, and the most amount of data and files possible on a single filesystem.
If you are storing data on servers, especially across drives in an array, you should be considering ZFS, unless it is not possible on the hardware or OS you are running.
Below I will group the Getting Started information in expandable info boxes.
While this guide is detailed enough for almost all cases, if you need even more information on doing more advanced administration, please refer to the OpenZFS Documentation, The FreeBSD ZFS Documentation, or even the Oracle ZFS Documentation
ZFS System Requirements
- Direct access to drives - ZFS can only work correctly if it can directly see and write to drives. This means that if you use a RAID card or a storage controller, you must have the drives passthrough or in HBA mode and have write caching off.
- Linux, BSD, Solaris, or supported UNIX OS - Solaris has the original ZFS by Oracle (formerly Sun). OpenZFS is available on Linux and BSDs like FreeBSD and NetBSD. OpenZFS is the opensource continuation of ZFS, developed by the community. The Linux and BSD versions are in parity. macOS has an OpenZFS port that works, but it's state may not be as consistent as the Linux and BSD OpenZFS.
- Adequate CPU and Memory - ZFS has features like memory caching and file compression. Compression on modern CPUs is quite fast, and the memory cache for ZFS can utilize a lot of memory. The cache is dynamic so it will free itself if the OS needs it.
Operating Systems for ZFS
For this guide, I'll assume you're using an OS that has OpenZFS included already without needing to set it up.
- Proxmox (my preferred Linux server host OS and Linux ZFS option)
- Ubuntu 20.04 Focal
- Ubuntu 22.04 Jammy (if you use Ubuntu, this is preferred)
- TrueNAS Scale
- TrueNAS Core (Preferred BSD option)
- Oracle Solaris (Proprietary)
- illumos (Fork of OpenSolaris)
- OpenIndiana (Based on illumos, preferred Solaris option)
Other OSs Supported but will need work to setup ZFS
Check OpenZFS Wiki on how to do that
- Red Hat 8.0+ / AlmaLinux / Rocky Linux / CentOS Stream / Fedora
- Arch Linux
- Void Linux
- macOS / Mac OS X - Please check on OpenZFS on OS X for more info
- Pool - A storage array consisting of VDEVs. Pools have names associated with them at creation.
- VDEV - VDEVs are a name for Virtual Devices. Virtual Devices are made up of Physical Devices or Physical Drives that collectively make up different configurations of devices for RAID, Mirrors, Stripes, RAID-Z, Special Devices, Cache, or SLOGs.
- Physical Device/Drive - A storage device being used for a pool
- RAID, Stripe, and Mirror - While ZFS has its own spin on RAID 5/6/7, ZFS can also do traditional RAID like RAID 0/1/10, or in ZFS terms Stripe (0), Mirror (1), and Mirror + Stripe (10).
- RAID-Z - A ZFS specific RAID type based on RAID 5 and RAID 6. There are 3 levels/types of RAID-Z: RAID-Z1 (RAID 5), RAID-Z2 (Raid 6), RAID-Z3 (Raid 7, in theory). Each type/level adds more parity allowing for more redundancy and resilience to drive failure. Once a Pool is created as a RAID-Z type it can not be changed until you remake the Pool.
- Dataset - A virtual filesystem that resides within a Pool. While a Pool can act as a filesystem on its own, you can utilize Datasets to segment out the Pool into specific filesystems that can have their own properties or Feature Flags set entirely.
- Feature Flags - Each Pool and Dataset can have features flags enabled on them to. This includes but is not limited to: mount points, compression, encryption, quotas, deduplication, atime, checksum algorithm, etc.
- Scrub - This a filesystem check for ZFS Pools. It can be performed online while the filesystem is mounted and active. It will read through the entire pool's data from each device and check if there are data errors found. It will then attempt to repair the data on drives that have an error on that read data from data/drives with known good data. Scrubs can take a while to perform on large pools, especially hard drive based pools. Scrubs can reduce IO of the pool while active, but scrubbing can also be tuned to pause while IO is being performed. Scrubs should be performed regularly, ideally monthly. Some OSs like Proxmox automatically have pools perform scrubs on a monthly basis.
- ARC - Standing for Adaptive Replacement Cache, ARC is a RAM/Memory based read cache for ZFS to speed up reads greatly for frequently accessed files. ARC will by default utilize as much free memory as it can. It is dynamic and will release memory when requested by the OS so overall the OS and it's apps will not realistically be affected by ARC's utilization. ARC is not used for write operations or cache.
- L2ARC - This means Level 2 ARC. L2ARC resides on a dedicated drive or partition on a drive, usually an SSD, but it should be faster than the associated Pool array it is added to. L2ARC is not used for write operations or cache.
- ZIL / Log - ZIL stands for ZFS Intent Log. It is a log of the data ZFS writes-ahead before committing data to a pool. In the case of a system failure, ZFS will recover and read this log to determine where it left off.
- SLOG - A SLOG means Separate Log, meaning a specified device or pair of devices, usually something fast like NVMe drives, are dedicated or have dedicated partitions for ZFS to utilize for the ZIL. The best type of NVMe drive would be one with very low latency and no drive cache, like an Intel Optane drive.
- Snapshots - ZFS has the ability to take Snapshots of a Pool or Dataset. Snapshots are created instantly and only take up space based on the changed file differences between the Snapshot and the live Pool/Dataset filesystem. Snapshots can be rolled back in their entirety or you can pull specific files from the hidden directory in that Dataset in ".zfs/snapshot/snapshot_name/"
- ZFS Send / ZFS Recv - ZFS has a built in ability to Send and Receive ZFS Pools or Datasets in their entirety to another Pool or Dataset, locally or remote. This also has the ability to just send the changes since the last Send/Recv. This works by first Snapshotting a Pool or Dataset and then sending that snapshot to the target Pool or Dataset. Because this works with snapshots, you are not working with live active data that is changing.
- CoW / Copy-on-Write - When ZFS writes data to a Pool/Devices, it will use unused or forward space on the drives to write the new data. When it is finished writing the data, it verified that this data was correctly written before it decides to have the old data removed and queued for deletion/overwriting. This ensures that data integrity is kept for new writes and that the old data is safe before overwriting it with data.
- Data Integrity / Checksum - When ZFS writes data to the pool, it performs a checksum hash on the data. When this is read later or a scrub is performed, this data and checked against the checksum to ensure data integrity is kept. This is critical as the data could be corrupted at any time, from events like a flipped bit, a bad cable, a bad drive, or the drive simply reporting the data wrong, even if that drive isn't showing signs that it is bad.
- ashift - How much data writes to devices should be adjusted to handle sector writes. ZFS will by default try to detect the sector size of the drives and conform to the lowest common denominator. But if you know that the drives in the Pool will accept 4K sectors, you can specify ashift=12 to have ZFS treat the devices as 4K sector size for writes for optimal performance.
Utilize a newer OpenZFS on older Ubuntu LTS versions
You can utilize a PPA repository to have the latest OpenZFS version on an older Ubuntu install (16.04 - 20.04). Ubuntu 22.04 has a new enough version of OpenZFS to not need this.
The Launchpad PPA Url: https://launchpad.net/~jonathonf/+archive/ubuntu/zfs
How to install this, step by step:
sudo add-apt-repository ppa:jonathonf/zfs
sudo apt update
sudo apt install zfs-dkms
sudo apt autoremove
sudo apt update
sudo apt dist-upgrade -V
sudo zfs version
You should now be on a 2.x version of OpenZFS
Creating ZFS Pools
Commands and Examples for creating ZFS Pool Arrays
The examples below will assume you are doing this on Linux but ZFS commands themselves are universal. ZFS requires root privileges so either be root or use sudo/doas/etc
Creating a ZFS Pool
When using drives for ZFS, you need to utilize their actual drive names or WWNs. We also need to check for the sector sizes these drives support.
While you can create Pools with drive partitions, it is best to use the whole drive when possible.
To start, we'll take an inventory of the drives available.
lsblk -o NAME,SIZE,MODEL,FSTYPE,LOG-SEC,PHY-SEC,SERIAL,MOUNTPOINT
And we get an output similar to this, but I am including only the drives I want to use in this example.
NAME SIZE MODEL FSTYPE LOG-SEC PHY-SEC sdc 14.6T ST16000NM000J-2TW103 512 4096 sdf 14.6T ST16000NM000J-2TW103 512 4096 sdg 14.6T ST16000NM000J-2TW103 512 4096 sdh 14.6T ST16000NM000J-2TW103 512 4096 sdi 14.6T ST16000NM000J-2TW103 512 4096
From this information, we know we want to get the drive ID names of
sdi. As because these are 4K native drives, we can include
ashift=12 in our pool creation command to force 4K sector mode. If it just says 512 or otherwise, leave out the ashift in the creation command. But ZFS is smart and it will check the reported Physical Block Size of the drive and choose what's best, so this is almost never needed.
So we will check the
/dev/disk/by-id/ directory in Linux to see what the drive IDs are. We can use grep to filter the output easier.
ls -lh /dev/disk/by-id | grep -e sdc -e sdf -e sdg -e sdh -e sdi lrwxrwxrwx 1 root root 9 Jan 9 00:38 ata-ST16000NM000J-2TW103_ZR52CQQK -> ../../sdi lrwxrwxrwx 1 root root 9 Jan 9 00:38 ata-ST16000NM000J-2TW103_ZR57LBNF -> ../../sdh lrwxrwxrwx 1 root root 9 Jan 9 00:38 ata-ST16000NM000J-2TW103_ZR5CZTZ6 -> ../../sdg lrwxrwxrwx 1 root root 9 Jan 9 00:38 ata-ST16000NM000J-2TW103_ZR5D4D23 -> ../../sdf lrwxrwxrwx 1 root root 9 Jan 9 00:38 ata-ST16000NM000J-2TW103_ZR60QV5P -> ../../sdc lrwxrwxrwx 1 root root 9 Jan 9 00:38 wwn-0x5000c5007443c744 -> ../../sdf lrwxrwxrwx 1 root root 9 Jan 9 00:38 wwn-0x5000c500e3c9cfab -> ../../sdh lrwxrwxrwx 1 root root 9 Jan 9 00:38 wwn-0x5000c500e53da9b9 -> ../../sdc lrwxrwxrwx 1 root root 9 Jan 9 00:38 wwn-0x5000c500e5c8b8a5 -> ../../sdg lrwxrwxrwx 1 root root 9 Jan 9 00:38 wwn-0x5000c500e5e2bfee -> ../../sdi
We want ideally the ones that look like
ata-DRIVEMODEL_SERIAL in this case, but if you are using NVMe drives, they will start with
wwn- entries will also be acceptable but not all drives use those and this creates extra steps in identifying bad drives later. So for now, copy and paste those names somewhere.
For the following, replace
POOLNAME with your desired name of the Pool storage. It is case sensitive and can not be the following:
drive#-name-serial with the actual Drive ID names you found earlier.
DO NOT USE THE TYPICAL NAMES OF DRIVES IN THE FORMAT OF
/dev/nvme0n1or one day you will find after a reboot or a change of cables, your drives will have different names and the ZFS pool will go offline. This can be fixed if you do this, later in this guide.
Single Drive Pool
- Min/Max Drives: 1/1
- Fault Tolerance: None
- Drive Space Overhead: None
- Read / Write Speed: Native Drive Speed
zpool create POOLNAME drive1-name-serial
Striped (RAID 0) Pool
- Min Drives: 1
- Fault Tolerance: None
- Drive Space Overhead: None
- Read / Write Speed: Native speed multiplied by # of Drives
zpool create POOLNAME drive1-name-serial drive2-name-serial
Mirrored (RAID 1) Pool
- Min Drives: 2
- Fault Tolerance: (# of Drives) - 1
- Drive Space Overhead: ((# of Drives) - 1) / (# of Drives)
- Read / Write Speed: 2x Drives / Native Drive Speed
zpool create POOLNAME mirror drive1-name-serial drive2-name-serial
Example of a real command
zpool create VENTURE mirror nvme-T-FORCE_TM8FP8002T_112104060080114 nvme-T-FORCE_TM8FP8002T_112104060080143
Striped + Mirrored (RAID 10) Pool
- Min Drives: 4
- Fault Tolerance: (# of Drives in each mirror) - 1
- Drive Space Overhead: ((# of Drives) x Parity Stripes over # of Mirrors
- Read / Write Speed: Typically 2x Drives / 2x Drives - Read speed increases with # of mirror groups
zpool create POOLNAME mirror drive1-name-serial drive2-name-serial mirror drive3-name-serial drive4-name-serial
Example of a real command
zpool create ENTERPRISE mirror ata-ST16000NM000J-2TW103_ZR52CQQK ata-ST16000NM000J-2TW103_ZR57LBNF mirror ata-ST16000NM000J-2TW103_ZR5CZTZ6 ata-ST16000NM000J-2TW103_ZR5D4592 mirror ata-ST16000NM000J-2TW103_ZR5D4D23 ata-ST16000NM000J-2TW103_ZR60QV5P
You may have noticed how the syntax here shows that a RAID 10 is made up of 2 mirror commands, with a striped layout of the mirrors. This also matters how you add drives to these Pools later.
Here's an example of what a RAID 10 ZFS Pool looks like after created.
zpool status pool: CHALLENGER state: ONLINE scan: scrub repaired 0B in 00:10:24 with 0 errors on Sun Jan 8 00:34:25 2023 config: NAME STATE READ WRITE CKSUM CHALLENGER ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nvme-TEAM_TM8FP4004T_112211290580504 ONLINE 0 0 0 nvme-TEAM_TM8FP4004T_112211290580505 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 nvme-TEAM_TM8FP4004T_112211290580958 ONLINE 0 0 0 nvme-TEAM_TM8FP4004T_112211290581167 ONLINE 0 0 0 errors: No known data errors
RAID-Z Type Pools
These are more specialized for ZFS. These will provide the most usable storage space, the best redundancy, the best data integrity, but also the slowest write speeds for ZFS Pool arrays. To calculate storage and speed, use this ZFS RAID-Z Calculator.
Which RAID-Z level should I choose?
You will have to decide based on your needs and your drives. The safest best is to choose RAID-Z2 by default. If you value your data, then RAID-Z3. RAID-Z1 can be useful if you're on a budget and your ability to quickly change out a failed drive. SSDs can work well on RAID-Z1 due to how they most likely won't die from resilvering a new drive replacement and can do it very quickly, as well as maximizing the space utilization.
If you're running high IO intensive applications like databases on ZFS, you may want to consider ZFS RAID 10 and not a RAID-Z, since it will have the best write speeds.
RAID-Z Pool (RAID-Z1)
- Min Drives: 3
- Fault Tolerance: 1 Drive
- Drive Space Overhead: 1 Drive
- Read / Write Speed: # of Drives - 1 / Single Drive Speed
zpool create POOLNAME raidz drive1-name-serial drive2-name-serial drive3-name-serial drive4-name-serial ...
Example of a real command
zpool create INTREPID raidz1 ata-TEAM_T2532TB_112208190540104 ata-TEAM_T2532TB_112208190540533 ata-TEAM_T2532TB_112208290490136 ata-TEAM_T2532TB_112208190540057
- Min Drives: 4
- Fault Tolerance: 2 Drives
- Drive Space Overhead: 2 Drives
- Read / Write Speed: # of Drives - 2 / Single Drive Speed
zpool create POOLNAME raidz2 drive1-name-serial drive2-name-serial drive3-name-serial drive4-name-serial ...
- Min Drives: 5
- Fault Tolerance: 3 Drives
- Drive Space Overhead: 3 Drives
- Read / Write Speed: # of Drives - 3 / Single Drive Speed
zpool create POOLNAME raidz3 drive1-name-serial drive2-name-serial drive3-name-serial drive4-name-serial drive5-name-serial ...
Example of a RAID-Z type Pool in Status output
pool: INTREPID state: ONLINE scan: scrub repaired 0B in 00:52:52 with 0 errors on Sun Jan 8 01:16:54 2023 config: NAME STATE READ WRITE CKSUM INTREPID ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ata-TEAM_T2532TB_112208190540104 ONLINE 0 0 0 ata-TEAM_T2532TB_112208190540533 ONLINE 0 0 0 ata-TEAM_T2532TB_112208290490136 ONLINE 0 0 0 ata-TEAM_T2532TB_112208190540057 ONLINE 0 0 0 errors: No known data errors
After Creating Your Pool
Now that your Pool is created, you need to set your default global properties for it and then a mountpoint. Check below for how to do that in the property flags section.
I personally like to set these properties on my Pools at the Pool level before I do anything with the Pool.
# Set ZStandard Compression globally for space and sped up reads zfs set compression=zstd-7 POOLNAME # Disable atime (Access Time) for performance and reduce SSD wear zfs set atime=off POOLNAME # Set the mountpoint zfs set mountpoint=/storage/POOLNAME POOLNAME # Create Datasets for major directories/categories zfs create POOLNAME/DATASET_NAME
Virtual Filesystems as directories in your Pool
If you looked at the ZFS Terminology area, you'll see that Datasets are virtual filesystems in ZFS. They show up as directories in the Pool. There's a lot of benefits to utilizing Datasets for your data.
Datasets have their own properties
Datasets inherit properties set from their parent Pool or parent Dataset. But you can set properties specifically on Datasets so that they have addition or changed features. Such as but not limited to:
- Different level of compression - Maybe you want more or less compression for that Dataset for the type of data it is holding
- Quotas - You can set limits of how much data that Dataset can have. This is useful for many situations including setting a user's home directory as a Dataset to limit their disk space utilization.
- ATIME - Maybe the data in that Dataset benefits from Access Times
- Encryption - You can specifically encrypt the data in a Dataset
- SAMBA Mode - You can set a property that makes the Dataset SAMBA friendly for access from Windows machines that are case insensitive
- Checksum type - You can change what checksum algorithm is used for that data
- Mountpoint - You can set a mountpoint for that Dataset to be somewhere else in the Filesystem
Datasets show disk space utilization info instantly
zfs list command, you can see live information instantly about how much space is being utilized by the Datasets. No more running and waiting for
du -h on multiple directories to see what's using up the disk space.
Datasets can be Snapshot and Recovered
You can snapshot a Dataset instantly and rollback to that snapshot anytime. Snapshots only use space for file differences or deltas between the snapshot and the live Dataset filesystem. This is great for backing up the existing data before you make major changes to the data. Or you can have a script that snapshots the Dataset regularly incase one day you accidently delete or change a file. You can then mount that snapshot to recover the data, roll back the snapshot (overwriting the new data), or even access the snapshot files from the hidden folder in the Dataset
Datasets can be Sent to other ZFS Pools locally and remotely
In conjunction with the Snapshot feature, you can send the Snapshot of a Dataset to another pool and retain all the properties and permissions, and it only sends the difference in files, sort of like a ZFS specific RSYNC.
Create a ZFS Dataset
Reminder: Datasets are case sensitive, name them how you'd name a directory
# Dataset at the root of the Pool zfs create POOLNAME/DATASET_NAME # Dataset in another Dataset zfs create POOLNAME/DATASET_NAME/ANOTHER_DATASET
Rename a ZFS Dataset
zfs rename POOLNAME/CURRENT_DATASET_NAME POOLNAME/NEW_DATASET_NAME
Destroy a ZFS Dataset
WARNING: THIS WILL DELETE ALL THE DATA IN THE DATASET INSTANTLY.
If there are children Datasets under this Dataset, you will need to specify
# Destroy a Dataset (no children Datasets) zfs destroy POOLNAME/DATASET_NAME # Destroy Dataset and all children Datasets underneath zfs destroy -r POOLNAME/DATASET_NAME
Destroy a ZFS Dataset Snapshot
Same principle as the Dataset, but you specify the Snapshot only
# Destroy a Dataset Snapshot zfs destroy POOLNAME/DATASET_NAME@SNAPSHOT_NAME # Recursively destroy all Snapshots of Dataset and Children Dataset Snapshots with the same name zfs destroy -r POOLNAME/DATASET_NAME@SNAPSHOT_NAME
ZFS Property Flags
Commands and Examples for Pool and Dataset Properties
See all properties for a POOL or DATASET
zfs get all POOLNAME zfs get all POOLNAME/DATASET_NAME
See specific properties for a POOL or DATASET
# Single Property zfs get propertyA POOLNAME zfs get propertyA POOLNAME/DATASET_NAME # Multiple Properties zfs get propertyA,propertyB,propertyC POOLNAME zfs get propertyA,propertyB,propertyC POOLNAME/DATASET_NAME
Example of some properties you'd check and the output
# Single Property in a Pool zfs get compression INTREPID NAME PROPERTY VALUE SOURCE INTREPID compression zstd-7 local # Single Property in a Dataset zfs get compression INTREPID/mirrors/apt NAME PROPERTY VALUE SOURCE INTREPID/mirrors/apt compression zstd-12 inherited from INTREPID/mirrors # Multiple Properties in a Pool zfs get compression,compressratio,mountpoint,encryption,atime INTREPID NAME PROPERTY VALUE SOURCE INTREPID compression zstd-7 local INTREPID compressratio 1.14x - INTREPID mountpoint /storage/INTREPID local INTREPID encryption off default INTREPID atime off local
Pool and Dataset mountpoint
When you set this on a Pool or a Dataset, ZFS will automatically mount the Pool or Dataset to this filesystem path. The path is case sensitive and doesn't have to match your
Mountpoint for a POOL
zfs set mountpoint=/path/to/dir POOLNAME
Mountpoint for a Dataset
zfs set mountpoint=/path/to/dir POOLNAME/DATASET_NAME
Example of my mountpoint
zfs set mountpoint=/storage/ENTERPRISE ENTERPRISE
Compression on a Pool or Dataset
ZFS allows you to define compression globally on a pool and specifically on Datasets. Compression is inherited from the parent filesystem unless it has its own compression set. You can change the compression on a pool or dataset anytime, but the data is not compressed to the new format and level until new data is written.
zfs set compression=OPTION POOLNAME zfs set compression=OPTION POOLNAME/DATASET_NAME
Here are the options available
on | off | lzjb | gzip | gzip-[1-9] | zle | lz4 | zstd | zstd-[1-19] | zstd-fast | zstd-fast-[1-10,20,30,40,50,60,70,80,90,100,500,1000]
on defaults to
zstd alone means
zstandard level 3 when set. But you can specify different
zstd levels with
zstd-# from compressional level 1 to 19.
zstd-fast and its options utilize a faster
zstd compression method, but the compression ratio will be less.
My recommendation is to utilize
zstd-7 as a global option on the Pool, and use
zstd-12 for Datasets that can benefit from higher compression and the data write speeds don't matter as much. You can also set
zstd-19 for maximum space savings for backup or archival data.
# Set global zstandard level 7 on POOL globally zfs set compression=zstd-7 POOLNAME # Set higher zstandard compression on specific Dataset zfs set compression=zstd-12 POOLNAME/DATASET_NAME
See what compression is being used on a Pool or Dataset
zfs get compression POOLNAME zfs get compression POOLNAME/DATASET_NAME
See how much compression is helping - Compression Ratio
zfs get compressratio POOLNAME zfs get compressratio POOLNAME/DATASET_NAME
Example of the output
The higher the value, the higher % of the data is compressed.
1.82x basically means
82% of the space is saved by compression.
zfs get compressratio INTREPID NAME PROPERTY VALUE SOURCE INTREPID compressratio 1.13x - zfs get compressratio INTREPID/containers/mikesulsenti.com NAME PROPERTY VALUE SOURCE INTREPID/containers/mikesulsenti.com compressratio 1.82x -
atime (Access Timestamps)
atime on a Pool or Dataset
zfs set atime=off POOLNAME zfs set atime=off POOLNAME/DATASET_NAME
atime on a Pool or Dataset
zfs get atime POOLNAME zfs get atime POOLNAME/DATASET_NAME
ZFS Disk Quotas
Limit Disk Space Usage by Dataset, Users, and Groups
Dataset and User Disk Space Quotas
ZFS allows you to set disk space quotas on datasets which limits how much space a Dataset can use total or limit how much a Dataset can use in Data and ZFS overheads like snapshots and metadata, or even specific Users or Groups that can use that Dataset filesystem.
These properties accept inputs in
Byte units like:
T - Terabytes,
G - Gigabytes,
M - Megabytes,
K - Kilobytes
You can check on a Dataset's
refquota settings like this:
zfs get refquota,quota POOLNAME/DATASET_NAME
And the available disk space for the Datasets with quotas should show in
Quota - Limit Total Disk Usage in a Dataset
The ZFS property flag
quota sets the hard limit of how much disk space a Dataset can use, including actual data, snapshots, and any other extras. Snapshots will use disk space if there are changes between snapshot(s) and the live data. This caps both.
# Syntax of quota zfs set quota=[SIZE] POOLNAME/DATASET_NAME # Real Example setting this is 1,200 Gigabytes max Total Quota zfs set quota=1200G INTREPID/users/mike
Or remove it with
zfs set quota=none INTREPID/users/mike
RefQuota - Limit Data Disk Usage in a Dataset
refquota sets a limit on the live data disk usage of the dataset but does not have a limit on the snapshots and other ZFS data that can add to the total disk usage by the Dataset. You can set both
quota to provide two levels of disk quotas, a hard limit of the Data and a hard limit of the total ZFS data.
# Syntax of refquota zfs set refquota=[SIZE] POOLNAME/DATASET_NAME # Real Example setting this is 1,000 Gigabytes max Data Quota zfs set refquota=1000G INTREPID/users/mike
Now I can also set the
quota on this Dataset to a slightly higher number to give some space to potential snapshot utilization even if the data is maxed.
Or remove it with
zfs set refquota=none INTREPID/users/mike
User and Group Quotas - Limit Disk Usage by Users and Groups
Similar to how you can set a
quota for a Dataset, you can set quotas for specific users or specific groups.
View Disk Usage by User or Group
# View disk usage by Users zfs userspace POOLNAME/DATASET_NAME # View disk usage by Groups zfs groupspace POOLNAME/DATASET_NAME
zfs userspace INTREPID/users/mike TYPE NAME USED QUOTA OBJUSED OBJQUOTA POSIX User mike 1.00T none 1.64M none POSIX User root 3K none 6 none zfs groupspace INTREPID/users/mike TYPE NAME USED QUOTA OBJUSED OBJQUOTA POSIX Group mike 1.00T none 1.64M none POSIX Group root 3K none 6 none
Set User Quota
# Syntax for setting a user quota zfs set userquota@USERNAME=[SIZE] POOLNAME/DATASET_NAME # Real example for setting user 'mike' to 200 Gigabytes max zfs set userquota@mike=200G INTREPID/shares/Documents
Or remove it with
zfs set userquota@mike=none INTREPID/shares/Documents
Set Group Quota
# Syntax for setting a group quota zfs set groupquota@GROUPNAME=[SIZE] POOLNAME/DATASET_NAME # Real example for setting group 'accounting' to 500 Gigabytes max zfs set groupquota@accounting=500G INTREPID/shares/Documents
Or remove it with
zfs groupquota@accounting=none INTREPID/shares/Documents
Commands to check and manage a ZFS Pool
Check Pool Status
zpool status is going to be the way to status of the Pool health and the drives
# Check zpool status of all Pools zpool status # Checkk all Pools with extra Verbose information zpool status -v # Check a specific Pool (with Verbose info) zpool status -v POOLNAME
state shows whether the Pool or Device is online or healthy
CKSUM indicate if there's ERRORs with the data. You don't want to see anything in these columns but
0. But if you see error counts, it could indicate a bad cable or a bad drive.
pool: ENTERPRISE state: ONLINE scan: scrub repaired 0B in 09:00:00 with 0 errors on Sun Jan 8 09:24:01 2023 config: NAME STATE READ WRITE CKSUM ENTERPRISE ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 wwn-0x5000cca24cc243a9 ONLINE 0 0 0 wwn-0x5000cca24cc3cf60 ONLINE 0 0 0 wwn-0x5000cca24cc3c39a ONLINE 0 0 0 wwn-0x5000cca24cc3c304 ONLINE 0 0 0 errors: No known data errors pool: INTREPID state: ONLINE scan: scrub repaired 0B in 00:52:52 with 0 errors on Sun Jan 8 01:16:54 2023 config: NAME STATE READ WRITE CKSUM INTREPID ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ata-TEAM_T2532TB_112208190540104 ONLINE 0 0 0 ata-TEAM_T2532TB_112208190540533 ONLINE 0 0 0 ata-TEAM_T2532TB_112208290490136 ONLINE 0 0 0 ata-TEAM_T2532TB_112208190540057 ONLINE 0 0 0 errors: No known data errors
Example of an output after a resilvering (re-syncing data on a replaced or offline drive)
pool: INTREPID state: ONLINE scan: resilvered 70.0G in 00:17:40 with 0 errors on Tue Jan 17 12:06:45 2023 config: NAME STATE READ WRITE CKSUM INTREPID ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 ata-TEAM_T2532TB_TPBF2209160130700561 ONLINE 0 0 0 ata-TEAM_T2532TB_TPBF2209160130700881 ONLINE 0 0 0 ata-TEAM_T2532TB_TPBF2209160130700462 ONLINE 0 0 0 ata-TEAM_T2532TB_TPBF2209160130700420 ONLINE 0 0 0 errors: No known data errors
Pool Filesystem Statistics
zpool list outputs information about the Pool filesystem like Total Size, Allocated Usage, Free Space, Fragmented Free Space, Capacity, Deduplication Ratio, and Health status.
You can use
zpool list -v to show more detailed information of the Pool broken down by Device and Pool Type.
You can also customize the command to show the properties you want to list out. Use
zpool list -h to show what options you can list out.
Example usage and outputs
zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT ENTERPRISE 14.5T 7.34T 7.20T - - 3% 50% 1.00x ONLINE - INTREPID 7.44T 2.93T 4.51T - - 2% 39% 1.00x ONLINE - zpool list -v NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT ENTERPRISE 14.5T 7.34T 7.20T - - 3% 50% 1.00x ONLINE - raidz1-0 14.5T 7.34T 7.20T - - 3% 50.5% - ONLINE wwn-0x5000cca24cc243a9 3.64T - - - - - - - ONLINE wwn-0x5000cca24cc3cf60 3.64T - - - - - - - ONLINE wwn-0x5000cca24cc3c39a 3.64T - - - - - - - ONLINE wwn-0x5000cca24cc3c304 3.64T - - - - - - - ONLINE INTREPID 7.44T 2.93T 4.51T - - 2% 39% 1.00x ONLINE - raidz1-0 7.44T 2.93T 4.51T - - 2% 39.4% - ONLINE ata-TEAM_T2532TB_112208190540104 1.86T - - - - - - - ONLINE ata-TEAM_T2532TB_112208190540533 1.86T - - - - - - - ONLINE ata-TEAM_T2532TB_112208290490136 1.86T - - - - - - - ONLINE ata-TEAM_T2532TB_112208190540057 1.86T - - - - - - - ONLINE
Pool IO Stats
zpool iostat provides IO statistics. Passing the verbose option
zpool iostat -v provides a greater breakdown of the IO information. You can also target a specific pool with
zpool iostat -v POOLNAME
zpool iostat capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- ENTERPRISE 7.34T 7.20T 33 60 5.60M 3.50M INTREPID 2.93T 4.51T 39 132 2.09M 1.70M ---------- ----- ----- ----- ----- ----- ----- zpool iostat -v capacity operations bandwidth pool alloc free read write read write ------------------------------------ ----- ----- ----- ----- ----- ----- ENTERPRISE 7.34T 7.20T 33 60 5.60M 3.50M raidz1-0 7.34T 7.20T 33 60 5.60M 3.50M wwn-0x5000cca24cc243a9 - - 8 15 1.41M 902K wwn-0x5000cca24cc3cf60 - - 8 14 1.39M 891K wwn-0x5000cca24cc3c39a - - 8 15 1.41M 902K wwn-0x5000cca24cc3c304 - - 8 14 1.39M 892K ------------------------------------ ----- ----- ----- ----- ----- ----- INTREPID 2.93T 4.51T 39 132 2.09M 1.70M raidz1-0 2.93T 4.51T 39 132 2.09M 1.70M ata-TEAM_T2532TB_112208190540104 - - 10 34 535K 438K ata-TEAM_T2532TB_112208190540533 - - 9 32 535K 435K ata-TEAM_T2532TB_112208290490136 - - 10 34 536K 438K ata-TEAM_T2532TB_112208190540057 - - 9 32 535K 434K ------------------------------------ ----- ----- ----- ----- ----- -----
ZFS Pool Scrub
zpool scrub initiates a full Pool data check, while the Pool is still online and usable. It will scan through all the data in the Pool and verify that the data matches the checksum. If it finds an issue, it will attempt to repair that data on the Device with data it knows is good.
Scrubs should be done on a monthly basis. How long a scrub takes to perform depends on if it's being done on spinning Hard Drives or SSDs, how much data is on the Pool, how many drives are in the Pool to speed up read times, and how much IO is using the Pool during the duration. Large SSD arrays can do their scrubs in hours while large (old) HDD arrays I have seen take half a month to almost a month to finish.
Start a scrub manually on a POOL (or resume a paused scrub)
zpool scrub POOLNAME
Pause a scrub in progress (can be resumed later)
zpool scrub -p POOLNAME
Stop/cancel a scrub in progress
zpool scrub -s POOLNAME
Check on the Status of a Scrub (zpool status)
zpool status -v POOLNAME ... scan: scrub in progress since Sun Jul 25 16:07:49 2021 403M scanned at 100M/s, 68.4M issued at 10.0M/s, 405M total 0B repaired, 16.91% done, 00:00:04 to go ...
Enable Periodic Scrubs
Proxmox automatically does this, but for other Linux systems, you can enable a
systemd timer for this like so:
# Scrub POOLNAME weekly systemctl enable [email protected] --now # Scrub POOLNAME Monthly (recommended) systemctl enable [email protected] --now
Or simply just setup a cronjob that calls the scrub command.
# Scrub all pools on the second Sunday of every month. 24 0 8-14 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi # Or Simply on the 15th of every month for a specific Pool 0 2 15 * * zpool scrub POOLNAME >/dev/null 2>&1
ZFS Pool TRIM
While Linux/Systemd has a FSTRIM.timer that is usually enabled by default on most distros, this may not actually TRIM ZFS Pools.
zpool trim POOLNAME
Enable Periodic TRIMs
Proxmox automatically does this, but for other Linux systems, you can enable a
systemd timer for this like so:
# TRIM POOLNAME weekly systemctl enable [email protected] --now # TRIM POOLNAME monthly (recommended) systemctl enable [email protected] --now
Or setup a cronjob for it
# TRIM all Pools the first Sunday of every month. 24 0 1-7 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi # Or Simply on the 5th of every month for a specific Pool 0 2 5 * * zpool trim POOLNAME >/dev/null 2>&1
ZFS ARC Stats and Summary
arc_summary can provide useful and advanced information on how ZFS's ARC Cache is being used.
arcstat time read miss miss% dmis dm% pmis pm% mmis mm% size c avail 16:11:59 0 0 0 0 0 0 0 0 0 6.6G 8.5G 1.9G
arc_summary ------------------------------------------------------------------------ ZFS Subsystem Report Fri Jan 20 16:12:06 2023 Linux 5.15.74-1-pve 2.1.6-pve1 Machine: virgo (x86_64) 2.1.6-pve1 ARC status: HEALTHY Memory throttle count: 0 ARC size (current): 43.2 % 6.6 GiB Target size (adaptive): 55.1 % 8.5 GiB Min size (hard limit): 6.2 % 983.5 MiB Max size (high water): 16:1 15.4 GiB Most Frequently Used (MFU) cache size: 21.2 % 1.0 GiB Most Recently Used (MRU) cache size: 78.8 % 3.8 GiB Metadata cache size (hard limit): 75.0 % 11.5 GiB Metadata cache size (current): 24.4 % 2.8 GiB Dnode cache size (hard limit): 10.0 % 1.2 GiB Dnode cache size (current): 90.4 % 1.0 GiB ARC hash breakdown: Elements max: 776.7k Elements current: 25.6 % 198.9k Collisions: 24.9M Chain max: 5 Chains: 4.5k ARC misc: Deleted: 185.8M Mutex misses: 50.7k Eviction skips: 325.7k Eviction skips due to L2 writes: 0 L2 cached evictions: 0 Bytes L2 eligible evictions: 10.1 TiB L2 eligible MFU evictions: 33.2 % 3.3 TiB L2 eligible MRU evictions: 66.8 % 6.7 TiB L2 ineligible evictions: 10.6 TiB ARC total accesses (hits + misses): 3.6G Cache hit ratio: 96.7 % 3.5G Cache miss ratio: 3.3 % 119.7M Actual hit ratio (MFU + MRU hits): 96.5 % 3.5G Data demand efficiency: 99.4 % 1.3G Data prefetch efficiency: 1.2 % 86.8M Cache hits by cache type: Most frequently used (MFU): 86.2 % 3.0G Most recently used (MRU): 13.6 % 475.0M Most frequently used (MFU) ghost: 0.1 % 4.7M Most recently used (MRU) ghost: < 0.1 % 1.3M Anonymously used: < 0.1 % 772.9k Cache hits by data type: Demand data: 36.6 % 1.3G Prefetch data: < 0.1 % 1000.0k Demand metadata: 63.1 % 2.2G Prefetch metadata: 0.2 % 7.7M Cache misses by data type: Demand data: 6.3 % 7.5M Prefetch data: 71.6 % 85.8M Demand metadata: 17.3 % 20.7M Prefetch metadata: 4.8 % 5.8M DMU prefetch efficiency: 366.1M Hit ratio: 21.6 % 79.1M Miss ratio: 78.4 % 287.0M L2ARC not detected, skipping section ### The output continues so I just cut it off here.
ZFS Pool Upgrade
Every so often, a ZFS Utils update introduces new features. ZFS Pools on BSD, Linux, and macOS do not use a version number, but feature flags. When a ZFS Pool Upgrade is available,
zpool status -v will show that upgrades are available. Note: If you do the upgrade, the Pool won't be able to be used in an older ZFS system. This usually is not an issue to be concerned about.
You can upgrade the pool with:
# Check all Pools for upgrades zpool upgrade # Enable all supported features on all Pools zpool upgrade -a # Enable for specific Pool zpool upgrade -a POOLNAME
ZFS Pool Drive Management
Manage Drives and Devices in Pools
Temporarily Offline a Drive
A Temporarily offline Drive comes back online automatically after a reboot or you put it back Online
zpool offline -t POOLNAME DRIVE_NAME
Example of real world use
zpool offline -t INTREPID ata-TEAM_T2532TB_112208190540533
Offline a Drive permanently (and by force)
-f flag forces the drive to be offline in a faulted state. This is a step more than a normal Offline as it persistence between Pool imports. Do not use the
-f flag unless you need the drive to stay offline between imports.
zpool offline -f POOLNAME DRIVE_NAME
Online a Drive
Brings the physical Drive online
zpool online POOLNAME DRIVE_NAME
Bring the Drive online AND Expand
-e the Drive to use available space on Drive. If the Device is part of a mirror or RAID-Z, all Devices must be expanded before te new space is usable on the Pool.
zpool online -e POOLNAME DRIVE_NAME
Replace a Drive
You can replace a drive in a ZFS Pool while the old drive is still in the Pool. The new drive will be added the Pool and resilvered. After this is completed, the OLD Drive will be detached from the Pool. This can be before or after the OLD Drive is removed from the system.
zpool replace POOLNAME OLD_DRIVE_NAME NEW_DRIVE_NAME
If errors occured on a ZFS Pool, and a solution was applied, you can Clear the errors from the Pool to bring the Pool back from a degraded state.
# Clear All Errors on a Pool zpool clear POOLNAME # Specific Drive zpool clear POOL DRIVE_NAME
Export ZFS Pool
You can export a ZFS Pool to either move it (and the drives) to another system or to utilize proper Drive names, if you accidently made the Pool with the
/dev/sda type drive naming convention, that is NOT recommended to have.
zpool export POOLNAME
All Datasets and Pools will be unmounted and the Pool will probably disappear from
Import ZFS Pool
You an import a ZFS Pool that was exported or came from another system.
List importable Pools. Not all importable Pools may show up this method.
zpool import pool: POOLNAME id: 15451357997522795478 state: ONLINE action: The pool can be imported using its name or numeric identifier.
Then import via the name or ID
zpool import POOLNAME
Import by scanning Drives
If you're unsure about what Drives to import by or there's a lot to import, you can use this to scan through the Disks by ID name and import them. For some HDDs, it may default to using
wwn- names, which isn't as good as the actual Drive name.
-a Means that it searches for and imports found Pools
-N Means it Imports the Pool without automounting filesystems
-d Means it searches a directory or a device
zpool import -d /dev/disk/by-id -aN
So you can specify the
-d option multiple times to import the Drive name you specifically want:
zpool import -d ata-ST16000NM000J-2TW103_ZR52CQQK -d ata-ST16000NM000J-2TW103_ZR57LBNF -d ata-ST16000NM000J-2TW103_ZR5CZTZ6 -d ata-ST16000NM000J-2TW103_ZR5D4592 -d ata-ST16000NM000J-2TW103_ZR5D4D23 -d ata-ST16000NM000J-2TW103_ZR60QV5P -aN
Import a Pool that was DESTROYED
You can re-import a Pool that was Destroyed with
zpool destroy POOLNAME with the
# Import Destroyed Pool by autoscanning Disk IDs zpool import -D -d /dev/disk/by-id -aN # Import Destroyed Pool by specific Drive names zpool import -D -d ata-ST16000NM000J-2TW103_ZR52CQQK -d ata-ST16000NM000J-2TW103_ZR57LBNF -d ata-ST16000NM000J-2TW103_ZR5CZTZ6 -d ata-ST16000NM000J-2TW103_ZR5D4592 -d ata-ST16000NM000J-2TW103_ZR5D4D23 -d ata-ST16000NM000J-2TW103_ZR60QV5P -aN
Adding to your Pool
Mirrors, Cache Drives, LOG Drives
Adding to your ZFS Pool usually means a couple of things. Do not mistake this with "expanding" your Pool.
mirror to your Pool. If you are running a ZPOOL Mirror already or a ZFS RAID 10, you can add additional mirrors to your Pool
zpool add POOLNAME mirror DRIVE_NAME_1 DRIVE_NAME_2 ...
Cache Drive - L2ARC
You can add a Cache or L2ARC to your ZFS Pool to speed up Read speeds and latency. Your Cache device should have a faster speed than your actual Pool, so you'll want to utilize SSDs for a HDD Pool, or NVMe SSD for a SATA SSD Pool. You can have multiple Cache drives for the Pool, which allows redundancy. More space though for L2ARC is not usually going to be beneficial, this is after your ARC Cache in memory is used.
zpool add POOLNAME cache DRIVE_NAME [DRIVE_NAME_2 if you use more than 1 drive]
You can remove this Drive from the Pool
zpool remove POOLNAME CACHE_DRIVE_NAME
Log Drive - ZIL - ZFS Intent Log Drive
You can add a ZIL or Intent Log drive to your ZFS Pool. These should ideally be very fast and Cache-less NVMe Drives. Intel Optane drives make the best Log Drives. They only need to be around 16-20GB in size. ZFS will only store about 5 seconds of writes to the drives before committing them to the Pool. If you have SSDs as your Pool storage already, this may not benefit you much if at all.
Additionally you should ideally have TWO Log drives in case one of them fails. A failed Log drive alone is not an issue but a failed Log drive and a power failure of the system could cause data corruption for the data that was about to be written.
# Single Log drive zpool add POOLNAME log DRIVE_NAME # Redundent Log drives zpool add POOLNAME log mirror DRIVE_NAME_1 DRIVE_NAME_2
You can remove the ZIL Log Drive from the Pool
zpool remove POOLNAME LOG_DRIVE_NAME
ZFS Pool Hot Spares
These Drives can be added to your Pool to automatically replace a failed Drive.
zpool add POOLNAME spare DRIVE_NAME
The Spares will be listed in their own category from the active Drives
Remove a spare at anytime (as long as a Drive is not currently failed)
zpool remove POOLNAME SPARE_DRIVE_NAME
Instant and space efficient life savers
ZFS has the ability to Snapshot entire Pools and Datasets. These snapshots take no space except for the change or delta in files from the live filesystem. They can be sent to other Datasets, ZFS Pools, mounted, or rollbacked to.
Create a ZFS Snapshot
ZFS Snapshots are named with the
@ symbol first and then a name for that Snapshot. This is case sensitive and it can be whatever you want. Do not use spaces and I would avoid certain special characters. But this could be, but not limited to:
# Snapshot a Pool zfs snapshot POOLNAME@SNAPSHOT_NAME zfs snapshot POOLNAME/DATASET_NAME@SNAPSHOT_NAME
Recursively Create Snapshots for Dataset and Children Datasets
Useful if you want all the Datasets of the target Dataset to have their own Snapshots at the same time, and for scripting.
-r is the Recursive flag to apply
zfs snapshot -r POOLNAME/DATASET_NAME@SNAPSHOT_NAME
Rollback a Dataset to its last Snapshot
You can easily
rollback all changes made to a Dataset's filesystem/files/data to the last Snapshot made of that Dataset.
Note: Rolling back to the last Snapshot will delete all changes made including: new files, modified files, etc
# Syntax of a Snapshot Rollback zfs rollback POOLNAME/DATASET_NAME@SNAPSHOT_NAME # Real Example zfs rollback INTREPID/containers/mikesulsenti.com@2023-01-08
You can only rollback to the latest Snapshot. If you want to rollback to an even earlier Snapshot that still exists, you must either destroy the snapshots between that snapshot and the live Dataset. Or run
rollback with the
-r option to automatically delete all snapshots inbetween.
zfs rollback -r POOLNAME/DATASET_NAME@SNAPSHOT_NAME
Rename A ZFS Snapshot
Similar to renaming a Dataset, you can rename a Snapshot. useful for scripting or moving things around.
# Change a specific @SNAPSHOT_NAME to a New Name zfs rename POOLNAME/DATASET_NAME@SNAPSHOT_NAME @NEW_SNAPSHOT_NAME # Recursively change all @SNAPSHOT_NAME Snapshots to a New Name zfs rename -r POOLNAME/DATASET_NAME@SNAPSHOT_NAME @NEW_SNAPSHOT_NAME
Destroy or Delete a ZFS Snapshot
Destroys a ZFS Snapshot and removes it and the data from the filesystem. Does not touch the live filesystem/Dataset
# Destroy a specific Dataset's Snapshot zfs destroy POOLNAME/DATASET_NAME@SNAPSHOT_NAME # Recursively Destroy @SNAPSHOT_NAME Snapshots zfs destroy -r POOLNAME/DATASET_NAME@SNAPSHOT_NAME
Clone a Snapshot to it's own Dataset
You can take a Dataset's Snapshot and make it into it's own Dataset and live filesystem
zfs clone POOLNAME/DATASET_NAME@SNAPSHOT_NAME POOLNAME/NEW_DATASET_NAME
You then need to Promote the clone Dataset to no longer depend on the origin Snapshot
zfs promote POOLNAME/NEW_DATASET_NAME
ZFS Send and Receive
Send ZFS filesystem locally and remote
ZFS Send and Receive
-R is Replicate and it replicates the filesystem properties and children filesystems, including all snapshots.
-i is for snapshot and sends incremental changes
-v is verbose output and shows how much data has been sent and more each second
First you need to snapshot the Dataset or Pool. Check the section above on how to do that
zfs snapshot POOLNAME/DATASET_NAME@SNAPSHOT_NAME
ZFS Send and Receive Locally
# Replicate a Snapshot zfs send -R -v POOLNAME/DATASET_NAME@SNAPSHOT_NAME | zfs receive OTHER_POOLNAME/OTHER_DATASET_NAME # Replicate a Snapshot with Incremental Changes zfs send -R -i -v POOLNAME/DATASET_NAME@SNAPSHOT_NAME | zfs receive OTHER_POOLNAME/OTHER_DATASET_NAME
ZFS Send and Receive Remotely
Simply just pipe into the
ssh command with the
# Replicate a Snapshot zfs send -R POOLNAME/DATASET_NAME@SNAPSHOT_NAME | ssh user@hostname zfs receive OTHER_POOLNAME/OTHER_DATASET_NAME # Replicate a Snapshot with Incremental Changes zfs send -R -i POOLNAME/DATASET_NAME@SNAPSHOT_NAME | ssh user@hostname zfs receive OTHER_POOLNAME/OTHER_DATASET_NAME
Monitor Progress of a ZFS Send
zfs send has a
-v verbose option, but this may produce too much information. You can utilize
pv to have a more simplified info, but you may not get a full picture like
You can install the utility
pv which is Pipe View. If you put
pv between the pipes, you can watch the data stream being sent. You can install it on Debian/Ubuntu/Proxmox with
apt install pv and on Red Hat/CentOS/Fedora/AlmaLinux/Rocky Linux with
yum install pv
# Monitor locally zfs send -R -i POOLNAME/DATASET_NAME@SNAPSHOT_NAME | pv | zfs receive OTHERPOOLNAME/OTHER_DATASET_NAME # Monitor Remotely over SSH zfs send -R -i POOLNAME/DATASET_NAME@SNAPSHOT_NAME | pv | ssh user@hostname zfs receive OTHER_POOLNAME/OTHER_DATASET_NAME
Encrypt ZFS Datasets and Decrypt them
Create an Encrypted Dataset
The important options here are:
oncurrently and recommended),
prompt(prompts in shell when you tell ZFS to load Dataset),
passphrase(8-512 bytes long and recommended),
raw(hex and raw must be 32 bytes long)
Create an Encrypted Dataset with
prompt with command, using
zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase POOLNAME/ENCRYPTED_DATASET_NAME
Load the key to mount Dataset after a reboot
-r Recursively loads the key for any child Dataset encrypted with the key. If a Child Dataset has its own key, you will need to target
load-key for that Dataset
zfs load-key -r POOLNAME/ENCRYPTED_DATASET_NAME
load-key using a key file
zfs load-key -r -L file:///path/to/key_file.txt POOLNAME/ENCRYPTED_DATASET_NAME
You can also specify
-a to load that key for any encrypted volume on any active Pool in the system
Change User Encryption Key
You can not change the Master Key to an encrypted Dataset. You'll have to move the data to another new Encrypted Dataset to fulled change out the Master Key. But this changes the user key wrapper.
zfs change-key [-l] [-o keylocation=location] [-o keyformat=format] [-o pbkdf2iters=value] POOLNAME/ENCRYPTED_DATASET_NAME
Sending an Encrypted Dataset
When you send a Dataset with
zfs send, if you want that data to be encrypted on the other receiving end, you must specify
-w to keep the data encrypted as is. This also means you can not compress the data stream as the encrypted data is incompressible.
zfs send -w -R -i POOLNAME/ENCRYPTED_DATASET_NAME@SNAPSHOT_NAME | zfs receive OTHER_POOLNAME/OTHER_ENCRYPTED_DATASET_NAME
I may update this guide and cheat sheet with more information later or with any new major changes OpenZFS introduces.
If you see an issue in this post, feel free to contact me about it!