Automatically inserting specified file into another (using sed, AWK, or Perl)

Question

Let's say I have a file, Foo.in:

Contents of Foo

and Bar.in

INSERT Foo.in
Contents of Bar

I would like to write a sed script that replaces INSERT xyz with actual contents of the xyz file. In this particular example I'd like to produce Bar.out file containing:

Contents of Foo
Contents of Bar

I thought about using sed's r command as demonstrated in this Stack Overflow question, but the problem is that the name of the file to be inserted is specified in the file itself. I thought about scanning the file for INSERT commands and then running sed separately for every INSERT found, but that is a horrible solution with O(n^2) complexity. I'd prefer to do this using sed or AWK, but if all else fails a Perl script would be acceptable.

I would argue that to be O(n). More importantly though, unless you have thousands of files and need to this every other day, why care for the asymptotic complexity. If the problem really was bound by resources, neither script language would be the right answer. edit: Wait, is this about doing that for only one file? Then complexity is completely meaningless... — DeVadder, Feb 19 '14 at 12:09
I have many files to process in that way and each may contain multiple lines. If I scan each file N times (where N is the number of of INSERT pragmas) then complexity is O(n^2), not O(n). I agree this is not a big issue here, but nevertheless it is an indication of a bad design. — Jan Stolarek, Feb 19 '14 at 12:20
As you mention, sed will not replace a dynamic content where content to insert is a value used as reference in the sed script. File name are "hardcoded". Now you can dynamicaly created a sed script but in this case, awk is a lot better — NeronLeVelu, Feb 19 '14 at 12:32
If n is the number of pragmas you are right, sorry. Nevertheless, in the world of scripts i would keep one sentence from the camel book in the back of my head: 'A Perl script is "correct" if it gets the job done before your boss fires you.' If you start thinking about runtime efficiency and resource management, Perl might be the wrong language and that is most likely just as true for sed and awk. — DeVadder, Feb 19 '14 at 12:37

score 4 · Accepted Answer · edited Apr 18 '15 at 17:23

4

$ cat Foo.in 
Contents of Foo

$ cat Bar.in 
INSERT Foo.in
Contents of Bar.in

$ awk '/INSERT/{while((getline line < $2) > 0 ){print line}close($2);next}1' Bar.in 
Contents of Foo
Contents of Bar.in

edited Apr 18 '15 at 17:23

Peter Mortensen

30,738
21
105
131

answered Feb 19 '14 at 12:07

Akshay Hegde

16,536
2
22
36

Is it possible to make this solution work recursively? – Jan Stolarek Feb 19 '14 at 12:48
@Jan Stolarek I just posted in another post please have a look – Akshay Hegde Feb 19 '14 at 13:13

score 1 · Answer 2 · answered Feb 19 '14 at 12:08

This is fairly easy to do. Here is a small Perl script:

#!/usr/bin/env perl
use strict;
use warnings;
use autodie;

while(<>) {
  # If this line is an INSERT command, capture the filename and insert it
  if (my ($filename) = /^INSERT\s+(.+)$/) {
    open my $fh, "<", $filename;
    local $/;
    print <$fh>;
  }
  # Otherwise just print out the line as it is.
  else {
    print;
  }
}

Usage: $ perl the-script.pl some_file.txt > output.txt

There are some issues with this solution: INSERTs in an inserted file won't be processed. The file path in an INSERT is interpreted relative to the working directory of the invocation, not to the file that issued the INSERT.

score 1 · Answer 3 · answered Feb 19 '14 at 12:12

1

Amon using your regex,

perl -MFile::Slurp -pe '$_= read_file($1) if /^INSERT\s+(.+)$/' file

answered Feb 19 '14 at 12:12

mpapec

50,217
8
67
127

score 1 · Answer 4 · edited Jun 20 '20 at 09:12

Recursion

$ cat Foo.in
Contents of Foo
INSERT test

$ cat test
1
2
3
4
5

$ cat Bar.in
INSERT Foo.in
Contents of Bar.in

AWK code:

awk '
function fetch(inp){
                     while( (getline p < inp) > 0)
                     print p
                     close(inp)
                   }
           /INSERT/{
                    while((getline line < $2) > 0)
                    {
                     if(line ~ /INSERT/){
                                         split(line,A)
                                         fetch(A[2])
                                         next
                                        }
                                    else{
                                         print
                                        }
                    }
                     close($2)
                     next
                   }1
   ' Bar.in

Result:

INSERT Foo.in
1
2
3
4
5
Contents of Bar.in

---Edit---

$ cat test.awk
function fetch(inp){
                 while( (getline p < inp) > 0)
                 print p
                 close(inp)
               }
       /INSERT/{
                while((getline line < $2) > 0)
                {
                 if(line ~ /INSERT/){
                                     split(line,A)
                                     fetch(A[2])
                                     next
                                    }
                                else{
                                     print
                                    }
                }
                 close($2)
                 next
               }1

Usage:

$ awk -f test.awk Bar.in

Hmm.. it looks like it doesn't work for me. I also tried putting this into a file and running `awk` with `-f` option but with no luck. — Jan Stolarek, Feb 19 '14 at 13:30
check the content of your test files in your edit it looks like you removed `.in`, BTW what's the error message ? — Akshay Hegde, Feb 19 '14 at 14:25
No error message - simply nothing gets substituted. When I replace the recursive version with the original, non-recursive one all works fine. — Jan Stolarek, Feb 19 '14 at 15:08
I really could not find problem I checked with `gawk`, `mawk` `original-awk` and `nawk`, one reason could be if file mentioned in column2 doesn't exist in present working directory, then it returns you nothing, but return contents of `Bar.in` have you missed anything while copying script ? — Akshay Hegde, Feb 19 '14 at 15:45

score 0 · Answer 5 · edited Apr 18 '15 at 17:17

0

You can also do it without loading Perl modules, in even fewer characters:

perl -pe 'open $F,"<$1" and $_=<$F> if /^INSERT\s+(.+)$/' [file]

edited Apr 18 '15 at 17:17

Peter Mortensen

30,738
21
105
131

answered Feb 19 '14 at 12:36

Jens Erat

37,523
16
80
96

Automatically inserting specified file into another (using sed, AWK, or Perl)

5 Answers5

Recursion

Linked