Split big files using PHP

Question

I want to split huge files (to be specific, tar.gz files) in multiple part from php code. Main reason to do this is, php's 2gb limit on 32bit system.

SO I want to split big files in multiple part and process each part seperately.

Is this possible? If yes, how?

Answer 1

My comment was voted up twice, so maybe my guess was onto something :P

If on a unix environment, try this...

exec('split -d -b 2048m file.tar.gz pieces');

split

Your pieces should be pieces1, pieces2, etc.

You could get the number of resulting pieces easily by using stat() in PHP to get the file size and then do the simple math (int) ($stat['size'] / 2048*1024*1024) (I think).

Answer 2

A simple method (if using Linux based server) is to use the exec command and to run the split command:

exec('split Large.tar.gz -b 4096k SmallParts'); // 4MB parts
/*    |     |            |      | |
      |     |            |______| |
      App   |                 |   |_____________
            The source file   |                 |
                              The split size    Out Filename
*/

See here for more details: http://www.computerhope.com/unix/usplit.htm

Or you can use: http://www.computerhope.com/unix/ucsplit.htm

exec('csplit -k -s -f part_ -n 3 LargeFile.tar.gz');

---

PHP runs within a single thread and the only way to increase this thread count is to create child process using the fork commands.

This is not resource friendly. What I would suggest is to look into a language that can do this fast and effectively. I would suggest using node.js.

Just install node on the server and then create a small script, called node_split for instance, that can do the job on its own for you.

But I do strongly advise that you do not use PHP for this job but use exec to allow the host operating system to do this.

Answer 3

HJSPLIT

http://www.hjsplit.org/php/

Answer 4

function split_file($source, $targetpath='/split/', $lines=1000){

    $i=0;
    $j=1;
    $date = date("m-d-y");
    $buffer='';

    $handle = fopen ($_SERVER['DOCUMENT_ROOT'].$source, "r");

    while (!feof ($handle)) {
        $buffer .= fgets($handle, 4096);
        $i++;
        if ($i >= $lines) {
            $fname = $_SERVER['DOCUMENT_ROOT'].$targetpath."part_".$date.$j.".txt";

                 $fhandle = fopen($fname, "w") or die($php_errormsg);

            if (!$fhandle) {
                echo "Cannot open file ($fname)";
                //exit;
            }


            if (!fwrite($fhandle, $buffer)) {
                echo "Cannot write to file ($fname)";
                //exit;
            }
            fclose($fhandle);
            $j++;
            $buffer='';
            $i=0;
            $line+=10; // add 10 to $lines after each iteration. Modify this line as required
        }
    }
    fclose ($handle);
}

Answer 5

$handle = fopen('source/file/path','r'); 
        $f = 1; //new file number
        while(!feof($handle))
        {
            $newfile = fopen('newfile/path/'.$f.'.txt','w'); //create new file to write to with file number
            for($i = 1; $i <= 5000; $i++) //for 5000 lines
            {
                $import = fgets($handle);
                //print_r($import);
                fwrite($newfile,$import);
                if(feof($handle))
                {break;} //If file ends, break loop
            }
            fclose($newfile);
            $f++; //Increment newfile number
        }
        fclose($handle);

Answer 6

PHP itself might not be able to... If you can figure out how to do this from your computers' command line, You should be able to then execute these commands using exec();

Answer 7

Splits are named as filename.part0 filename.part1 ...

<?php
function fsplit($file,$buffer=1024){
    //open file to read
    $file_handle = fopen($file,'r');
    //get file size
    $file_size = filesize($file);
    //no of parts to split
    $parts = $file_size / $buffer;

    //store all the file names
    $file_parts = array();

    //path to write the final files
    $store_path = "splits/";

    //name of input file
    $file_name = basename($file);

    for($i=0;$i<$parts;$i++){
        //read buffer sized amount from file
        $file_part = fread($file_handle, $buffer);
        //the filename of the part
        $file_part_path = $store_path.$file_name.".part$i";
        //open the new file [create it] to write
        $file_new = fopen($file_part_path,'w+');
        //write the part of file
        fwrite($file_new, $file_part);
        //add the name of the file to part list [optional]
        array_push($file_parts, $file_part_path);
        //close the part file handle
        fclose($file_new);
    }    
    //close the main file handle

    fclose($file_handle);
    return $file_parts;
}
?>

Answer 8

• If you want to split files which are already on server, you can do it (simply use the file functions fread, fopen, fwrite, fseek to read/write part of the file).
• If you want to split files which are uploaded from the client, I am afraid you cannot.

Answer 9

This would probably be possible in php, but php was built for web development and trying to this whole operation in one request will result in the request timing out.

You could however use another language like java or c# and build a background process that you can notify from php to perform the operation. Or even run from php, depending on your Security settings on the host.