S3 file to local using luigi raises UnicodeDecodeError

Question

I am copying a pdf file to local, using the following piece of code:

with self.input_target().open('r') as r:
    with self.output_target().open('w') as w:
       for line in r: 
           w.write(line)

Which is based in this question (kind of)

But when I execute that code I get the following:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 11:
      invalid continuation byte

I tried this other approach, without good results:

   with self.input_target().open('r') as r, self.output_target().open('w') as w:                                                                                                                        
       w.write(r.read())

What is the correct way of doing it?

could you prepare a minimal example to reproduce it and the pdf? — Alexey Grigorev, Nov 08 '15 at 20:52

score 1 · Answer 1 · answered Nov 01 '19 at 00:10

1

Another option would be to use the format kwarg luigi.format.Nop to Target to indicate that it is a binary file that is being read.

e.g.

def output(self):
    return LocalTarget(target, format=luigi.format.Nop)

answered Nov 01 '19 at 00:10

KCP

23
3

score 0 · Answer 2 · edited May 23 '17 at 11:44

0

It seems that you're dealing with a binary file as if it was text - but it's not. You probably need to do something like this:

with self.input().open('r') as i, self.output().open('w') as o:
    o.write(i.read())

(not tested!)

Also, I think you may find this answer useful: Python writing binary files, bytes

edited May 23 '17 at 11:44

Community

1
1

answered Nov 07 '15 at 08:07

Alexey Grigorev

2,415
28
47

Thank you for the answer, but I didn 't work. Same error: `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 11: invalid continuation byte` `:(` I will update the question – nanounanue Nov 07 '15 at 15:08
try `open('rb')` and `open('wb')` – Alexey Grigorev Nov 07 '15 at 17:40
Do you know the encoding of the pdf file? If it's Latin-1, then you should read it as such: `self.input_target().open('r', encoding="ISO-8859-1")`. – matagus Nov 08 '15 at 19:16
`luigi` doesn` t support the `encoding` argument – nanounanue Nov 12 '15 at 01:17

score 0 · Answer 3 · answered Nov 12 '15 at 01:20

I ended using luigi.s3.S3Clientand the method get which copies a binary file from/to the s3.

Snippet of code:

import luigi

class ATask(luigi.Task):
    s3_path = luigi.Parameter()
    local_path = luigi.Parameter()
    ...

    def run(self):
        client = luigi.s3.S3Client()
        client.get(s3_path, local_path)  ## This gets the file
        ...

I think that the underlying reason is that luigi uses boto for getting/putting files from/to the s3. (As you can see in the source code)

S3 file to local using luigi raises UnicodeDecodeError

3 Answers3

Linked