how to convert bash script to C++ using boost::iostreams

Question

I'm trying to convert the following bash code into C++ using boost::iostreams:

#!/usr/bin/bash
(
    gzip -cd file1.ext.gz
    cat file2.ext
) | grep '^regex' # or sed 's/search/replace/'

I can open a file and decompress it:

std::ifstream s("file.ext.gz", std::ios_base::in | std::ios_base::binary);
boost::iostreams::filtering_istreambuf in;
in.push(boost::iostreams::gzip_decompressor());
in.push(s);

Then open an uncompressed file:

std::ifstream s2("file.ext", std::ios_base::in | std::ios_base::binary);

Now I'm a bit stuck, so here are my questions:

1) What's the boost::iostreams solution to concat the two streams?

2) How to output the result through a regex filter to emulate grep/sed?

As a result I'd like to have a an istream that i can copy to cout:

boost::iostream::copy(result, std::cout);

UPDATE complete solution using Hamigaki's concatenate:

/*
 * convert the following bash script into C++
 *
 * #!/bin/bash
 * (
 *     gzip -cd file1.ext.gz
 *     cat file2.ext
 * ) | grep '^filter' | 'sed s/search/replace/g'
 *
 */

#include <iostream>
#include <boost/bind.hpp>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/device/file.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <boost/iostreams/filter/regex.hpp>
#include <boost/iostreams/filter/grep.hpp>
#include <boost/iostreams/copy.hpp>

// http://hamigaki.sourceforge.jp/hamigaki/iostreams/concatenate.hpp
#include "concatenate.hpp"

namespace io = boost::iostreams;

int main(int argc, char const* argv[])
{
    io::file_source file1("file1.ext.gz");
    io::file_source file2("file2.ext");
    io::gzip_decompressor gzip;
    io::regex_filter sed(boost::regex("search"), "replace");
    io::grep_filter grep(boost::regex("^filter"));

    io::filtering_istreambuf in1(gzip | file1);
    io::filtering_istreambuf in2(file2);

    io::filtering_istreambuf combined(sed | grep | 
            hamigaki::iostreams::concatenate(
                boost::ref(in1),
                boost::ref(in2)
            )
        );

    io::copy(combined, std::cout);

    return 0;
}

As far as the shell side of things, do you know about `zgrep` (unless, of course, you need the file to be uncompressed)? Won't your script send the file to `grep` twice? — Dennis Williamson, Feb 22 '11 at 00:11
@Dennis: I think you misread the bash script. I have two files, one of them is compressed, and I want to concat them together and search(/replace) for a pattern in the result. — chris, Feb 22 '11 at 06:48
Sorry, I saw `file.ext.gz` and `file.ext` and took that to mean the second was the result of the first (I overlooked the `-c`). It would have been clearer if you had said `file1.ext.gz` and `file2.ext`. — Dennis Williamson, Feb 22 '11 at 14:44
@Dennis: I see. I edited the question accordingly. Sorry for the confusion. — chris, Feb 22 '11 at 15:32

Pablo · Accepted Answer · 2011-02-22T17:11:28.423

1) I don't know if there's anything built into boost, but this class seems to be exactly what you want: http://hamigaki.sourceforge.jp/hamigaki/iostreams/concatenate.hpp

The catch here is that it expects CopyConstructible devices to concatenate and Chains seem to not be CopyConstructible. However, we can easily work around that using boost::ref. This code does (almost) what I understood you're asking:

int main(int argc, char const* argv[])
{
  boost::iostreams::filtering_istreambuf in;
  boost::regex regex("search");
  boost::iostreams::regex_filter rf(regex, "replace");
  in.push(rf);

  boost::iostreams::file_source file1(argv[1]);
  in.push(file1);

  boost::iostreams::file_source file2(argv[2]);
  boost::iostreams::copy(hamigaki::iostreams::concatenate(boost::ref(in), file2), std::cout);

  return 0;
}

I just used the regex filter instead of gzip, for testing.

2) boost::iostreams has a regex filter: http://www.boost.org/doc/libs/1_45_0/libs/iostreams/doc/classes/regex_filter.html

EDIT: You seem to have this working, now.

I found both of these, but couldn't get any of them to work the way I want. If you can show how to solve my problem with these I'll accept it. — chris, Feb 22 '11 at 06:46

chris · Answer 2 · 2011-02-23T22:51:50.943

1) Not available in boost

Hamigakis's concatenate sounds interesting, but I couldn't figure out how to use it to combine two boost::iostreams::chains. The code mentions it's meant for "concatenation of devices", so it might not be usable for chains. Please correct me if I'm wrong.

EDIT: updated my question with the complete solution.

2a) grep behavior (filter):

#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/filter/grep.hpp>

boost::iostreams::filtering_istreambuf in;
boost::regex regex("^search")
boost::iostreams::grep_filter grep(regex);
in.push(grep);

2b) sed behavior (search/replace):

#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/filter/regex.hpp>

struct formatter {
    std::string operator()(const boost::match_results<const char*>& match)
    {
        return str(boost::format("%s | %s") % match[2] % match[1]);
    }
};
boost::iostreams::filtering_istreambuf in;
boost::regex regex("^([a-z]+) ([0-9]+)");
boost::iostreams::regex_filter sed(regex, formatter());
in.push(sed);

@bta: No it is not. The answer is referring to my own question. — chris, Feb 22 '11 at 18:02

how to convert bash script to C++ using boost::iostreams

2 Answers2