Context
I'm using multiple ZFS datasets in one pool backed by HDDs to store backups and am forwarding those backups using RSYNC into some other pool on a different HDD, which can be replaced regularly. All of the datasets used the property xattr=on, while only the backup datasets additionally used encryption.
At some point a few weeks ago I decided to enable encryption with the internal datasets as well. So I created a new root dataset and sent+received all other datasets into that encrypted one, really applying encryption to the data that way.
Backup became much slower
The downside of my change is that while RSYNC took ~3 hours in the past when only the backup HDD was encrypted, it now takes ~7,5 hours with both encrypted. I've debugged things and the problem is with xattr, because when RSYNC ignores ACLs and(!) extended attributes, it runs less than 3 hours again. If either of both is enabled, RSYNC takes 7,5 hours again. My setting on seems to map to dir, which uses additional file system objects to store things and for some reason the combination of that and encryption seems to have slowed things down a lot.
Changing xattr=sa
So, I've changed all my datasets to use xattr=sa by really only changing that property. I didn't rewrite data into new datasets or something, because in theory I don't care about the old data stored with dir too much. I regularly create new backups and extended attributes would just be rewritten with the new mode.
Problem: I don't see any performance improvement. My expectation was that after changing the property, accesses would be much faster, because the additional overhead of dir doesn't effect me anymore. I expected that without rewriting extended attributes in the backups, those would simply be empty, because they aren't stored as system attribute yet.
I wonder if one ever used dir, that ZFS might still know that regardless of the current setting sa and still do additional lookups. The docs mention that sa is only used for ~64K of data and afterwards things are stored like with dir in additional file system objects. So there's such possibility in general, but I would expect that it is really only used when those 64K of data are exhausted, which can't be the case for my setup, because I didn't ran any backups after changing the property.
Up to 64K of data may be stored per-file in the space reserved for system attributes. If there is not enough space available for an extended attribute then it will be automatically written as a directory-based xattr.
https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html#xattr
Question
Does changing ZFS property XATTR take effect immediately or need a dataset be rewritten?
Notes
This is not a question about backup performance in general, I use RSYNC for reasons currently. I know about send+receive and that that might perform better. It's on the list in case things are just that slow and can't really be improved with encryption.
