0

I have an API pull request that returns a .gz file, which R recognizes as a "raw" file. I have not been able to find any R package that can decompress it in-memory from the saved file. I've tried fread(), rawToChar() and unzip().

The structure of the file I need to unzip is below. Specifically req$content.

str

ekoam
  • 8,744
  • 1
  • 9
  • 22
Pceam
  • 46
  • 5
  • 1
    Try this `memDecompress(req$content, "gzip")` – ekoam Nov 24 '20 at 04:06
  • Are you using `httr`? It should decompress for you when you access the `content()` of the response. It would be easier to help you if you provide a [reproducible example](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) that we can use for testing. – MrFlick Nov 24 '20 at 04:26
  • @ekoam Have you successfully used `memDecompress` on standard gzip data? I've never seen it actually work (see https://stackoverflow.com/questions/55600738/how-can-i-unzip-a-base64-encoded-string-in-r). – MrFlick Nov 24 '20 at 04:29
  • @MrFlick I tried the string in that post with `memDecompress` and got the expected [output](https://imgur.com/a/X2T6DCH). I am using R v4.0.3. – ekoam Nov 24 '20 at 05:20
  • @ekoam Sure enough, it was listed as a [new feature in R 4.0.0](https://cran.r-project.org/doc/manuals/r-devel/NEWS.html). That's good news. – MrFlick Nov 24 '20 at 05:24
  • The issue I was running into was with 00 in the string. When removed, I got a lexical error. I also tried memDecompress but I did not run on my latest version of R, for some reason. – Pceam Nov 24 '20 at 21:26

1 Answers1

0

Thanks for your responses. I ended up saving the variables as .gz files in a folder and decompressed them using foreach and fread() from data.table. This solution worked better because data set is really large >80gb so it would not have made sense to do it in-memory. Apologies for not being able to give a fully reproducible example as the data is purchased and not possible to share on public forum.

Pceam
  • 46
  • 5