9

I have about 12 GB of image tiles made up of about 2 million files. I'd like to zip these up to make transferring them to a server simpler. I just plan on storing the files in the zip files for transferring, no compression. Helm is present on the web server and can handle unzipping files.

I'd like to point a program at all these files in one go and get it to zip them up into files of approx 1 GB each, but each zip file needs to be independent of the others.

I have 7-zip installed with supports splitting across volumes, but these volumes are dependent upon one another to be unzipped.

Anyone have any suggestions? Thanks in advance!

Journeyman Geek
  • 133,878
Gavin
  • 325

7 Answers7

4

The freeware on Windows called "Spinzip" should do the work for your purpose ! ;) http://skwire.dcmembers.com/wb/pages/software/spinzip.php

It is based on IZARCC (automatically included in Spinzip). You have to check but the full original path may be kept in the zipped files !

See ya

Erb
  • 415
3

I'm not aware of a program that can do that, since if you are making one zip in multi-volumes they will all be related. Your best bet may be to make 12 folders and put a GB in each one, then zip the folders individually.

JNK
  • 8,426
3

In the end I created a quick python script to split the files in to sub directories for me before zipping each individually.

In case it's useful to anyone else, here's my script:

import os
import csv
import shutil

def SplitFilesIntoGroups(dirsrc, dirdest, bytesperdir):
    dirno = 1
    isdircreated = False
    bytesprocessed = 0

    for file in os.listdir(dirsrc):
        filebytes = os.path.getsize(dirsrc+'\\'+file)

        #start new dir?
        if bytesprocessed+filebytes > bytesperdir:
            dirno += 1
            bytesprocessed = 0
            isdircreated = False

        #create dir?
        if isdircreated == False:
            os.makedirs(dirdest+'\\'+str(dirno))
            isdircreated = True

        #copy file
        shutil.copy2(dirsrc+'\\'+file, dirdest+'\\'+str(dirno)+'\\'+file)
        bytesprocessed += filebytes

def Main():
    dirsrc='C:\\Files'
    dirdest='C:\\Grouped Files'

    #1,024,000,000 = approx 1gb
    #512,000,000 = approx 500mb
    SplitFilesIntoGroups(dirsrc, dirdest, 512000000) 

if __name__ == "__main__":
    Main()
Gavin
  • 325
1

Replying after 10 years and 8 months later

Just wrote my own tool for exact necessity.

https://github.com/hamza02x/zip-split-independent

$ zip-split-independent -h

-d string the directory which need to be zipped -o string output zip directory (not zip file) (default "zip-splits") -s float split size (in MB) (default 1)

1

OK here is a way out of it, but not all that good. You can try if you really need.

Assumptions: You need to divide in 12GB of data into 3 4GB DVDs.

Solution

  1. Take 3 empty pen drives of size 4GB.
  2. Insert pen drive #1.
  3. Now start copying your whole of the folder to pen drive #1 using a tool like TeraCopy which pauses on some error and doesn't terminate.
  4. When pen drive #1 is full, it will pause and by the time you can change to pen drive #2.
  5. Now resume after pen drive #2 is loaded.
  6. Repeat from step 4 when pen drive #2 is full. Just increase the pen drive #.

Now you have your data divided. Write them to DVDs or whatever you want to write on.

If you don't have 3 pen drives, you can write the first DVD there itself and then delete whole of the data on pen drive before you resume the copy process.

IsmailS
  • 719
0

We now use a free program called DirectorySlicer. It makes "copies" (uses "hardlinks" if destination is the same drive, so it doesn't use up more drive space) the files into folders of a specified size. This helps us create folders of files that will fit a 700MB CD.

NOTED DOWNSIDE: the files aren't necessarily in the same order; meaning the sequenced filenames (like photo images) might be spread across "chunks" to fit better

You can then create ZIP files of each folder.


Take a look at http://bkprograms.weebly.com/folder-axe.html it is something we've used in the past to split a batch of files into smaller chunks to fit onto 4.7GB DVDs.

slhck
  • 235,242
J. Chin
  • 293
  • 2
  • 7
0

SpinZip is the rigth tool for no compression. I wanted to use compression, so the result was unsatisfactory. Zipsplit does not work for files above 2GB, so I ended up to write my own quick and dirty perl script, which does its work. Its adds files to the archive as long as the file + archive is lower than the max. specified size:

# Use strict Variable declaration
use strict;
use warnings;
use File::Find;

# use constant MAXSIZE    => 4700372992; # DVD File size
use constant MAXSIZE    => 1566790997; # File size for DVD to keep below 2GB limit
# use constant MAXSIZE    => 100000000; # Test

use constant ROOTDIR    => 'x:/dir_to_be_zipped'; # to be zipped directory

my $zipfilename    = "backup"; # Zip file name
my $zipfileext    = "zip"; # extension

my $counter = 0;
my $zipsize = undef;
my $flushed = 1;

my $arr = [];

find({wanted =>\&wanted, no_chdir => 1}, ROOTDIR);

flush(@{$arr});

# Callback function of FIND
sub wanted {
    my $filesize = (-s $File::Find::name);

    LABEL: {
        if ($flushed) {
            $zipsize = (-s "$zipfilename$counter.$zipfileext");
            $zipsize = 0 unless defined $zipsize;

            printf("Filesize Zip-File %s: %d\n", 
                "$zipfilename$counter.$zipfileext", $zipsize);

            $flushed = 0;

            if (($zipsize + $filesize) >= MAXSIZE) {
                $counter++;
                $flushed = 1;
                printf("Use next Zip File %d, Filesize old File: %d\n",
                    $counter, ($zipsize + $filesize));
                goto LABEL;
            }
        }
    }

    if ( $zipsize + $filesize  < MAXSIZE ) {
        printf("Adding %s (%d) to Buffer %d (%d)\n",
            $File::Find::name, $filesize, $counter, $zipsize);

        push @{$arr}, $File::Find::name;
        $zipsize += $filesize;
    }
    else {
        printf("Flushing File Buffer\n");
        flush(@{$arr});

        $flushed = 1;
        $arr = [];
        goto LABEL;

    }
}

# Flush File array to zip file
sub flush {

    # open handle to write to STDIN of zip call
    open(my $fh, "|zip -9 $zipfilename$counter.$zipfileext -@")
        or die "cannot open < $zipfilename$counter.$zipfileext: $!";

    printf("Adding %d files\n", scalar(@_));
    print $fh map {$_, "\n"} @_;
    close $fh;
}
hwol
  • 1