How to read compressed files c++

Question

My Question is closed so I have to update this.

1- My purpose is to send my compressed file google cloud storage URL.

2- To do that I have generated a postman request. I have stored my file to my google cloud storage by using the postman tool and the tool has generated the following code.

CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
  curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
  curl_easy_setopt(curl, CURLOPT_URL, "my URL");
  curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
  curl_easy_setopt(curl, CURLOPT_DEFAULT_PROTOCOL, "https");
  struct curl_slist *headers = NULL;
  headers = curl_slist_append(headers, "Content-Type: 
      application/octet-stream");
  curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
  curl_easy_setopt(curl,CURLOPT_POSTFIELDS,"<file contents here>");
  res = curl_easy_perform(curl);
}
curl_easy_cleanup(curl);

3- Then I have copied the code above into my c++ project to send my compressed file to the URL.

4- To create CURLOPT_POSTFIELDS content I did implement the following code;

std::ifstream ifs;
ifs.open ("./compressed.gz", std::ios::binary | 
    std::ios::ate);

PRINT("Size ->", ifs.tellg());

curl_easy_setopt(curl, CURLOPT_POSTFIELDS, &ifs);

And when I compile and run my code, the request returns me the 200 success response. But when I checked the google storage dashboard, it just contains 6 bytes of data. Actually, the size of my ifstream data is 1090.

So my problem is that why my request uploads all bytes of the compressed file to cloud storage? Whats wrong in my code ?

I believe you need to decompress using `zlib`. Related: [https://stackoverflow.com/questions/20762094/how-are-zlib-gzip-and-zip-related-what-do-they-have-in-common-and-how-are-they](https://stackoverflow.com/questions/20762094/how-are-zlib-gzip-and-zip-related-what-do-they-have-in-common-and-how-are-they) — drescherjm, Mar 23 '21 at 21:28
Don't use `strlen()` on binary data, use `file_data.size()`. — Galik, Mar 23 '21 at 21:30
`strlen` will read until it sees the first `null` character. Are you sure there aren't any `null` characters in the file? You should use `file_data.size()` instead as Galik suggests — alter_igel, Mar 23 '21 at 21:31
You probably want to use the output of `gzip -d < ./compressed.gz` or use `zlib` to decompress the file. Also, a `.gz` can have binary data in it, so even if you get the valid uncompressed data using `strlen` may not be a valid thing to do. — Craig Estey, Mar 23 '21 at 21:32
Why do you do `strlen` on a `std::string` object? You already have `string.length()` which gives you the correct length even if the string contains `0-bytes`. And this is not C. Also it's not clear if you want to send the compressed file to libcurl or the uncompressed one. — Devolus, Mar 23 '21 at 21:33
Use `std::vector` for binary data, not `std::string`. Also use `std::ios::binary` open mode on the file stream. — paddy, Mar 23 '21 at 21:38
What @paddy said. After opening the stream, `std::vector file_data(std::istreambuf_iterator(ifs), std::istreambuf_iterator{});` would put the whole content in `file_data`. — Ted Lyngmo, Mar 23 '21 at 21:40

eerorika · Answer 1 · 2021-03-23T21:56:42.897

How to read compressed files c++

Compressed files are generally binary formats, they are not null terminated text. You cannot use strlen to get their length because that requires a null terminated text as input.

You can use any UnformattedInputFunction to read binary data. Don't forget to open any stream in binary mode.

I need to put the compressed file in a char pointer to put it to the server by using libcurl.

You don't need to read the file in order to do that. You can let libcurl take care of reading the file by passing a FILE* to CURLOPT_READDATA.

This way it won't be necessary to store the entire file in memory. Reading the entire file could be a problem if the file is very large.

score 1 · Answer 2 · answered Mar 23 '21 at 21:46

You can do something like that when you want to read binary data:

std::ifstream file("./compressed.gz", std::ios::binary);
if (!file.is_open())
    ERROR();

std::vector<unsigned char> buffer(std::istreambuf_iterator<char>(file), {});

Then you will have it in the vector buffer. Access the size via buffer.size() and to get raw data use buffer.data() Note that since buffer is an std::vector if it goes outside of scope it will be destructed so the data will be deleted.

How to read compressed files c++

2 Answers2