unix prepend line to file without buffer/copy

Question

I want this, the sed solution looks good, but when I tried it seems to create a huge temp file.

I have big sql file that is like 28 gigs, I dont have much space on file system and just want to add one line to the front of the file. How can I do this without using up more file space?

Can you put the one line in a separate file, and then use `cat single-line-file 28-gigabyte-file | program-that-reads-the-data`? This avoids creating the file explicitly on disk. — Jonathan Leffler, Aug 28 '12 at 14:53

score 3 · Accepted Answer · answered Aug 28 '12 at 14:40

Unfortunately, on every OS and file system I've ever seen, prepending can't generally be done in place like appending can. One might argue that file systems could do it efficiently, if the amount of data was some multiple of the underlying file system block size, but since that would not generally be the case, I don't know of any that have actually implemented such functionality. So, probably, the only way to do it is through a temporary file or copy. You could, however, use compression to somewhat alleviate the space crunch, but that would require some prework, and may not ultimately be suitable. Something along these lines:

1) gzip original_file.sql    # or bzip2 or whatever
2) create new_file.sql with data to be prepended
3) (cat new_file.sql; zcat original_file.sql.gz) | gzip > updated_file.sql.gz
4) zcat updated_file.sql.gz | less  # inspect the top of the file to make sure it looks right
5) rm original_file.sql.gz new_file.sql
6) gunzip updated_file.sql.gz # if necessary to have uncompressed text available - if not, just leave it compressed

Even though not the answer I was looking for, seems to be true and the consensus. — Joelio, Sep 04 '12 at 12:34

score 0 · Answer 2 · answered Aug 28 '12 at 14:36

Compile with gcc:

#include <stdio.h>
#include <string.h>
#include <malloc.h>

// Prepend size must be less than this value
#define bufSize 1024000

int main( int argc, char **argv )
{
    FILE *fil;
    unsigned char *smallBuf, *mainBuf;
    size_t sReadSize, mReadSize;
    long readPos = 0, writePos = 0;
    int appendSize;

    if( argc != 3 )
    {
        printf( "Usage: %s, <prepend_line> <file>\n", argv[0] );
        return 1;
    }

    sReadSize = appendSize = strlen( argv[1] ) + 1;

    smallBuf = (unsigned char *) malloc( appendSize );
    mainBuf = (unsigned char *) malloc( bufSize );
    if( !smallBuf || !mainBuf )
    {
        printf( "No memory\n" );
        return 1;
    }

    memcpy( smallBuf, argv[1], appendSize );
    smallBuf[ appendSize - 1 ] = '\n';

    fil = fopen( argv[2], "rb+" );
    if( !fil )
    {
        printf( "Cannot open file\n" );
        return 1;
    }

    while( 1 )
    {
        fseek( fil, readPos, 0 );
        readPos += mReadSize = fread( mainBuf, 1, bufSize, fil );

        fseek( fil, writePos, 0 );
        writePos += fwrite( smallBuf, 1, sReadSize, fil );

        if( mReadSize < bufSize )
        {
            if( mReadSize > 0 )
                fwrite( mainBuf, 1, mReadSize, fil );
            break;
        }

        fseek( fil, readPos, 0 );
        readPos += sReadSize = fread( smallBuf, 1, appendSize, fil );

        fseek( fil, writePos, 0 );
        writePos += fwrite( mainBuf, 1, mReadSize, fil );

        if( sReadSize < appendSize )
        {
            if( sReadSize > 0 )
                fwrite( smallBuf, 1, sReadSize, fil );
            break;
        }
    }

    fclose( fil );
    return 0;
}

I have a 2 buffers, first I read 1 block, write prepend, read second, write first,... Please, be more attentive. — Pavel Ognev, Aug 28 '12 at 15:59
Yes, I see that on a closer inspection. I'll edit the original comment, but I would still observe that this is a dangerous operation, especially as the original question is about a multi-GB file. Modifying in place is prone to catastrophic failure in case of power outages, accidental early termination of the program, or any number of other factors, which would leave the only copy of the file partially modified (although probably in at least a somewhat predictable way, ignoring caching/writeback timing issues). — twalberg, Aug 28 '12 at 16:12
Joelio asked for a simple suggestion which doesn't need additional memory. Of cause, It may be fault-resistant, by adding a temporary file with buffers content and current progress information. — Pavel Ognev, Aug 28 '12 at 19:47

score 0 · Answer 3 · answered Aug 29 '12 at 07:02

0

You can use perl for this:

perl -i -n -e 'print "xxx\n$_" if $.==1;print if $.!=1' your_file

answered Aug 29 '12 at 07:02

Vijay

65,327
90
227
319

4

`-i` uses a temp file behind the scenes. – choroba Aug 29 '12 at 07:11

unix prepend line to file without buffer/copy

3 Answers3