I have quite old mail server (CentOS 6) of 4+ Tb mail in maildirs (so you may imagine numbe of files). It is VM (VMware 7) on SSD storage.
I wonder if I should care for FS fragmentation at all? I can afford to rsync whole storage to new VM disk (effectively make fragmentation to almost zero level), or run online defragmentation tool (since this is XFS), but should I care at all? SSD backed storage seems to forgive anything?
When storing data on an HDD with a spinning disk, defragmentation reduces the mechanical seek time penalty significantly. Highly-fragmented filesystems on those devices can become extremely slow. Solid state devices do not have any seek penalty. However, modern high-speed SSDs are often IOPS limited, meaning that the rate at which commands can be sent over the bus is the bottleneck, not the ability of the device to access the actual data and service those requests.
Defragmenting a filesystem increases the number contiguous logical sectors in a given file. This can improve performance, even for an SSD, because reading a small portion of a file can be done with a single I/O request. If that same portion was fragmented, it might take multiple requests. This could easily saturate the bus when using an extremely-fast SSD with a heavily-fragmented filesystem.
However, there are better ways to improve performance.
If you want to improve performance on an SSD, you have to help the SSD know which blocks can be erased by using TRIM. A TRIM command allows the filesystem to tell the SSD which sectors are empty and can be handed over to the garbage collector, even if they still have (unallocated) data on them. This greatly improves the device's ability to manage storage. On Linux, you can execute the command fstrim /var/mail (or wherever your partition is) to give the SSD a list of unallocated sectors that can be discarded. See fstrim(8) for more details on command usage.
TRIM greatly improves the ability of the SSD to counter write amplification and allows for better wear leveling. This will not speed up reads from the device, but it will speed up writes. If your drive has very little free space, there will be less benefit (after all, you wouldn't have many free sectors).
Because the FTL (flash translation layer, a part of the flash controller) of an SSD is responsible for mapping logical sectors to physical sectors, there's no serious need to defragment a filesystem in most situations. The time it takes to defragment and the wear is generally not worth it.
SSDs will automatically attempt to optimize the storage of files. This optimization is subject to the limitations of NAND flash. To understand the limitations, we have to describe some terminology:
Cells are the smallest physical unit which stores data in a flash device. In SLC devices, each cell stores a single bit. MLC devices, each cell stores two (by holding four different voltage levels). TLC devices store three (by holding 8), and QLC, four (by holding 16). Multiple cells, plus some extra to hold error-correcting codes that help recover from failures of individual cells, make up a page.
Pages are a collection of cells that can be read from or written to at once. A page is how NAND flash devices implement physical sectors, and most pages are between 512 bytes and 4 KiB in size. Pages can only be written to if they have been erased (all bits initialized to 1). If a page has data on it, it cannot be overwritten without first being erased. This is due to the fact that a single bit can only be changed from a 1 to a 0. The other way around can only be done by erasing an entire page. However, it's not possible to erase individual pages. To erase a page, you must erase the entire block.
Blocks are a collection of pages and are measured in tens or hundreds of kilobytes. 128 individual 4 KiB pages may make up a single 512 KiB block. The only action that can be taken on an individual block is erasure. This will erase all pages that it contains. Any data in any of those pages that needs to be preserved must be moved to another block that has some erased pages already. This is done by the flash controller transparently. If you have very few free blocks, the controller may need to move some data around in order to write anything, which slows down writes continuously.
This poses a problem: In order to most effectively optimize writes, the SSD needs to know which sectors the filesystem no longer needs. However, block storage devices don't even have a concept of filesystems. They can't distinguish between a sector that is no longer allocated to any file but still contains leftover data and a sector that is still part of a file and must be preserved. Because making SSDs aware of filesystems would be a Sisyphean task, it's better to simply tell the SSD which sectors the filesystem no longer needs. Modern SSDs support this via TRIM.
You should not attempt to defragment a file / folder on an SSD. Defragment will not do anything useful for solid state storage and should not be used. "Fragmentation" does not really apply to the organization of cells in an SSD.
If you need to Shrink (reduce the space occupied by the machine) you can do this in VMware using the shrink command.
I use shrink a lot but you would have to try for the older machine. It should work.
Open a terminal session, from there run:
sudo vmware-toolbox-cmd disk shrink /
Allow the shrink to finish. There is a terminal component that will reach 100% and then a VMware GUI Window that will shrink the machine.
Shrink will recover space and remove unused space in the machine but little if any impact in performance
SSDs don't cost extra time to seek between distant parts of the disk, so the potential gains are very small. Like at best just a tiny reduction in CPU usage for the server, maybe fractions of a percent as a wild guess.
But it's a valid question; if reads for multiple small files can be coalesced at the block level into one larger read, that's fewer commands sent to the SSD, so you'll get more of its available bandwidth. At least that's a reasonable thing to hope for or wonder about. Same for reading one larger file.
But it still probably doesn't justify actually doing anything, especially because XFS can only defrag individual files, not AFAIK group multiple small files into contiguous storage (in order of filename or otherwise).
Your files are mostly small and each individual file was probably written all at once, not appended later. You say you have maildir, one file per email message. Not mbox like traditional /var/mail, where it's one file per mailbox with messages appended as they arrive. In that case there might be fragmentation, although deleting a message involves rewriting at least the tail of the file from that point.
So that being the case, you probably don't have much fragmentation of the kind XFS can do anything about; most files probably only have one extent. You can check with filefrag /var/mail/**/* | grep -v '1 extent found'.
For more detail you can use filefrag -e to see the individual extents of a file, or since you're on XFS you can use xfs_bmap to get the same info. See a unix.SE answer for example output.
As @Roger Lipscombe comments, the physical flash cells that store a file's data might not be contiguous even if the logical block addresses are; the flash remapping layer inside the SSD might fragment at that level. But that's fine; the SSD firmware should be able to read the necessary parts of flash with minimal overhead; unlike the kernel having to send commands in some protocol (SATA, SAS, or NVMe), the firmware is directly connected to the raw flash and it's probably not slower to read 2 separate regions than one longer region.
Defragmenting a solid state drives does nothing to increase disk performance as the seek/head movement times do not exist as in drives with moving parts. It will also increase wear on the chips which have a limited number of write cycles resulting in reduced lifetime.
If you need to to shrink a partition, you can do that with the drive unmounted.
