I'd like to use dd to create some disk image backups to S3 buckets, however the extra space that's included from empty space is not tolerable. I have thought I could pass it through xz to compress out the zeros, however when this happens, it seems that it's not really losing the empty space padding.
Any ideas on how to properly to pass a dd stream on through a compression algorithm so as to create minimum sized images directly on S3 mounts? These images will range from tens of MB to tens of GB of data with very little local working space for the upper bounds and so it's necessary to remove the 0s as early as possible (ie, perhaps reducing the buffer or chunk size for the compression tool).
