I've a VirtualBox VM which configured a very large hard disk size (bigger than host). By my mistake, a program on the VM generated lots of log files and the VDI file size keeps growing until there is no space on the host.
Now I've deleted the log files but the VDI file size are not getting smaller after using VBoxManage.exe modifyhd "C:\Virts\mybox-i386.vdi" compact
Is there a way to really compact the VDI file size? Thanks!
You have to do the following steps:
Run defrag in the guest (Windows only)
Nullify free space:
With a Linux Guest run this:
dd if=/dev/zero of=/var/tmp/bigemptyfile bs=4096k ; rm /var/tmp/bigemptyfile
Or:
telinit 1
mount -o remount,ro /dev/sda1
zerofree -v /dev/sda1
With a Windows Guest, download SDelete from Sysinternals and run this:
sdelete.exe c: -z
(replace C: with the drive letter of the VDI)
Shutdown the guest VM
Now run VBoxManage's modifymedium command with the --compact option:
With a Linux Host run this:
vboxmanage modifymedium --compact /path/to/thedisk.vdi
With a Windows Host run this:
VBoxManage.exe modifymedium --compact c:\path\to\thedisk.vdi
With a Mac Host run this:
VBoxManage modifymedium --compact /path/to/thedisk.vdi
VBoxManage is located here: /Applications/VirtualBox.app/Contents/MacOS/VBoxManage
This reduces the vdi size.
This isn't a direct answer per se, as I'm addressing the problem, not the question. Instead of periodically compacting the image, this solution uses discard to automatically remove unused blocks in the host's VM disk image.
This solution requires a guest filesystem that supports continuous TRIM. The Arch Linux wiki has a list of filesystems supporting TRIM operations.
FDE and cryptoroot are specifically not covered, as there are security concerns and none of the other solutions to this question would allow compacting either. The Arch Linux wiki has information about TRIM and dm-crypt devices.
In theory, this will work for all Linux guests on VBox hosts using VDI storage.
With VBox exited and no VMs running, add discard support to your disks by setting both discard and nonrotational for each disk in the config file for the VM. At this time discard is not in the GUI, but nonrotational is exposed as the "Solid-state Drive" checkbox. (ref: vbox forums, discard support)
<AttachedDevice discard="true" nonrotational="true" type="HardDisk" [..other options..] >
Boot the VM up, and verify that TRIM support is enabled:
sudo hdparm -I /dev/sda | grep TRIM
If LVM is in use, change the discard setting in /etc/lvm/lvm.conf. (ref: debian wiki, lvm.conf example)
devices {
...
issue_discards = 1
}
In fstab, add the discard option to the filesystems you wish to auto-discard (ref: debian wiki, fstab example)
UUID=8db6787f-1e82-42d8-b39f-8b7491a0523c / ext4 discard,errors=remount-ro 0 1
UUID=70bfca92-8454-4777-9d87-a7face32b7e7 /build ext4 discard,errors=remount-ro,noatime 0 1
Remount the filesystems to have them pick up their new options.
sudo mount -o remount /
sudo mount -o remount /build
Manually trim free blocks now with fstrim. fstrim uses the mounted filesystem, not the block device backing it. Instead of setting continuous discard in fstab, this could be done on a weekly cron. (The weekly cron is recommended for physical SSDs which may have questionable support for TRIM, but this is not relevant here since underlying SSDs are handled by the host OS. see: ssd trim warning).
fstrim /
fstrim /build
At this point, the size of the filesystems inside the VM and the size of the VM images should be pretty close in value.
Tested with:
I'm on a Windows 7 host with Windows guests, Here is a batch file I wrote to Compact all of the VDIs in a folder tree
echo off
mode con:cols=140 lines=200
cls
:: see https://forums.virtualbox.org/viewtopic.php?p=29272#p29272
:: How can I reduce the size of a dynamic VDI on disk?
:: but that page says to use sdelete -s which is suboptimal.
:: use -z as per http://technet.microsoft.com/en-us/sysinternals/bb897443.aspx
:: First run the sdelete -z c: inside the VMs that zero-out all the free space
:: THEN run this batch file
Title Compacting Free space on Virtual Machine VMs
:: http://ss64.com/nt/for_r.html
:: http://stackoverflow.com/questions/8836368/windows-batch-file-how-to-loop-through-files-in-a-directory/8836401#8836401
Setlocal EnableDelayedExpansion
:: http://ss64.com/nt/delayedexpansion.html ...
:: Notice that within the for loop we use !variable! instead of %variable%.
For /R %CD% %%G IN (*.vdi) DO (
set ohai=%%G
set lastfive=!ohai:~-5!
:: Skip snapshots which are named {guid}.vdi
if NOT !lastfive!==}.vdi (
echo .
echo Compacting %%G
"C:\Program Files\Oracle\VirtualBox\VboxManage.exe" modifyhd "%%G" --compact )
)
pause
exit
I left the links in the comments so you can (sort of) tell how it works.
edit
For MacOS Guest do this:
Nullify free space in guest system:
diskutil secureErase freespace 0 "/Volumes/Macintosh HD"
(replace /Volumes/Macintosh HD with your drive name)
Shutdown the guest VM
Run this command to reduce VDI disk image size
VBoxManage modifyhd /path/to/thedisk.vdi --compact
OR
VBoxManage modifymedium /path/to/thedisk.vdi --compact
IMPORTANT NOTE FOR LEGACY (~1997-2007) OPERATING SYSTEMS
In general, the techniques in the answers previously given are valid; HOWEVER, there is a very important special case.
For a period of some years-- perhaps 1997-2007 or so-- 32-bit operating systems were still the norm, but hard disks larger than 2GB were already in use. As a result, when attempting to consume all free space by writing a file of zeroes (which should always be done as root, to include root's privileged free space, which no one else can touch), you may see:
File too large
instead of what you expect:
No space left on device.
If this occurs, you have most likely hit a 2GB file size limitation. This was common at the time because many file operations returned results in signed 32-bit integers, so that negative values could report error codes. This effectively meant that offset results were limited to 2^31 bytes without special measures.
The workaround is straightforward: keep creating separate, differently-named zeroing files until the disk actually runs out of space.
If you are an instructor wishing to demonstrate this situation for a class, a 4GB disk image with an old copy of Red Hat Linux 7.0 is sufficient.
I use this for my VDI image mounted to virtual Debian in Windows VirtualBox. It isn't a general solution, but it should at least give you a gist of what I do.
Commands in Debian:
root@debian:~# lsblk # show partitions
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdb 8:16 0 128G 0 disk └─sdb1 8:17 0 128G 0 part /mnt/web # THIS IS THE PARTITION OF INTEREST! sda 8:0 0 64G 0 disk ├─sda1 8:1 0 61,4G 0 part / ├─sda2 8:2 0 1K 0 part └─sda5 8:5 0 2,7G 0 part [SWAP] sr0 11:0 1 56,3M 0 rom
root@debian:~# service mysql stop # terminate all operations with partition
root@debian:~# service apache2 stop # terminate all operations with partition
root@debian:~# umount /mnt/web # unplug partition
root@debian:~# apt-get install zerofree # install tool for filling in zeros to empty space
root@debian:~# zerofree -v /dev/sdb1 # fill with zeros
root@debian:~# poweroff # shut down machine
Commands in Windows:
C:\Program Files\Oracle\VirtualBox>VBoxManage.exe modifyhd --compact "D:\VirtualBox VMs\web.vdi" # convert zeros to empty space
Hope it helps :)
I don't want to enable TRIM support in OS, because every data deletion will force data compacting in VDI file, making guest system unusable when VDI file is on classic rotational disc. For me better is to perform compacting by hand e.g. once per month.
During normal compacting, VDI file content is copied to new file. This requires some (sometimes big) free space on host disc.
I've solution similar to pointed by Andrew Domaszek. It works very well even with NTFS (Windows10).
To do this:
Change VDI based discs to be visible as SSD with TRIM support:
VBoxManage storageattach "gpared live" --storagectl "SATA" --port 0 --discard on --nonrotational on
VBoxManage storageattach "gpared live" --storagectl "SATA" --port 1 --discard on --nonrotational on
start machine
mount /dev/sda2 /mntdd if=/dev/zero of=/mnt/bigfilerm /mnt/bigfilefstrim -v /mntA very neat trick to supplement the accepted answer is that you can get away without doing any compacting at all after zeroing guest space, by using a compressed file system on the host (e.g. selecting to compress the folder of virtual drives on NTFS properties on a Windows host). This in fact has the benefit to save a lot more space because operating systems tend to hold a lot of repetitive text or binary files (e.g. a 30GB guest drive that had 15GB of space zeroed can turn to 4GB on the host drive).
Caveats include that drive access on the real hardware may increase and there is a slight increase in CPU usage.
Just to add an alternative to the accepted answer: vboxmanage clonehd not only clones but also compacts virtual disks, so if you also need to clone it you can use a similar process and do it in one go (in my case I was moving the VM from an external disk to an internal disk with less space so I needed clone+compact):
vboxmanage clonehd /mnt/externaldisk/VMs/win10.vdi /home/myuser/VMs/win10.vdi
Just use the Live CD from this Open Source Project, on Linux Hosts / Guests ( this Live CD has the zerofree utility built in; also read the instructions on the site ) :
https://sourceforge.net/projects/live-cd-with-zerofree-utility/
This will also help in reducing the size of an exported virtual machine appliance as well ( i.e. .ova file ).
As others mentioned, after zerofree is run, use the compact option, to reduce the vdi file size.
P.S. A 32-Bit Version ( i686 architecture ) of this Live CD is available at: https://sourceforge.net/projects/live-cd-with-zerofree-32-bit/
For very OLD Architectures, Please try this Live CD : https://sourceforge.net/projects/live-cd-with-zerofree-x86/
I compared the working method proposed by @magicandre1981 above versus CloneVDI tool (again suggested here, by CAD Bloke) in 2 scenarios:
For a VM Windows (guest machine) that had been defragged multiple times using a an external program that also defrags the white space (makes sure no bubble of white space remains between the files). Then I had zeroed all the bits of white space.
For the same VM Windows but without any defragging and also without nullifying (wiping) the white space. That extra defrag program was installed in both VM guest Windows (so both virtual hard disks were identical), but was not used in the second scenario.
Results:
The method proposed by Andre (5 steps) worked very efficiently in the first scenario but not in the second scenario (as expected). The only downside was it was time consuming to follow the steps.
The CloneVDI tool worked perfectly in both scenarios, with the same results in terms of hard disk size in both scenarios, regardless of the initital state of VM virtual hard disk (not defragged or nullified). And it was extremely fast and easy.
