Python 3- check if buffered out bytes form a valid char

Question

I am porting some code from python 2.7 to 3.4.2, I am struck at the bytes vs string complication.

I read this 3rd point in the wolf's answer

Exactly n bytes may cause a break between logical multi-byte characters (such as \r\n in binary mode and, I think, a multi-byte character in Unicode) or some underlying data structure not known to you;

So, when I buffer read a file (say - 1 byte each time) & the very first characters happens to be a 6-byte unicode how do I figure out how many more bytes to be read? Because if I do not read till the complete char, it will be skipped from processing; as next time read(x) will read x bytes relative to it's last position (i.e. halfway between it char's byte equivalent)

I tried the following approach:

import sys, os

def getBlocks(inputFile, chunk_size=1024):
    while True:
        try:
            data=inputFile.read(chunk_size)
            if data:
                yield data
            else:
                break
        except IOError as strerror:
            print(strerror)
            break

def isValid(someletter):
    try:
        someletter.decode('utf-8', 'strict')
        return True
    except UnicodeDecodeError:
        return False

def main(src):
    aLetter = bytearray()
    with open(src, 'rb') as f:
        for aBlock in getBlocks(f, 1):
            aLetter.extend(aBlock)
            if isValid(aLetter):
                # print("char is now a valid one") # just for acknowledgement
                # do more
            else:
                aLetter.extend( getBlocks(f, 1) )

Questions:

1. Am I doomed if I try fileHandle.seek(-ve_value_here, 1)
2. Python must be having something in-built to deal with this, what is it?
3. how can I really test if the program meets its purpose of ensuring complete characters are read (right now I have only simple english files)
4. how can I determine best chunk_size to make program faster. I mean reading 1024 bytes where first 1023 bytes were 1-byte-representable-char & last was a 6-byter leaves me with the only option of reading 1 byte each time

Note: I can't prefer buffered reading as I do not know range of input file sizes in advance

Answer 1

The answer to #2 will solve most of your issues. Use an IncrementalDecoder via codecs.getincrementaldecoder. The decoder maintains state and only outputs fully decoded sequences:

#!python3
import codecs
import sys
byte_string = '\u5000\u5001\u5002'.encode('utf8')

# Get the UTF-8 incremental decoder.
decoder_factory = codecs.getincrementaldecoder('utf8')
decoder_instance = decoder_factory()

# Simple example, read two bytes at a time from the byte string.
result = ''
for i in range(0,len(byte_string),2):
    chunk = byte_string[i:i+2]
    result += decoder_instance.decode(chunk)
    print('chunk={} state={} result={}'.format(chunk,decoder_instance.getstate(),ascii(result)))
result += decoder_instance.decode(b'',final=True)
print(ascii(result))

Output:

chunk=b'\xe5\x80' state=(b'\xe5\x80', 0) result=''
chunk=b'\x80\xe5' state=(b'\xe5', 0) result='\u5000'
chunk=b'\x80\x81' state=(b'', 0) result='\u5000\u5001'
chunk=b'\xe5\x80' state=(b'\xe5\x80', 0) result='\u5000\u5001'
chunk=b'\x82' state=(b'', 0) result='\u5000\u5001\u5002'
'\u5000\u5001\u5002'

Note after the first two bytes are processed the internal decoder state just buffers them and appends no characters to the result. The next two complete a character and leave one in the internal state. The last call with no additional data and final=True just flushes the buffer. It will raise an exception if there is an incomplete character pending.

Now you can read your file in whatever chunk size you want, pass them all through the decoder and be sure that you only have complete code points.

Note that with Python 3, you can just open the file and declare the encoding. The chunk you read will actually be processed Unicode code points using an IncrementalDecoder internally:

input.csv (saved in UTF-8 without BOM)

我是美国人。
Normal text.

code

with open('input.txt',encoding='utf8') as f:
    while True:
        data = f.read(2)   # reads 2 Unicode codepoints, not bytes.
        if not data: break
        print(ascii(data))

Result:

'\u6211\u662f'
'\u7f8e\u56fd'
'\u4eba\u3002'
'\nN'
'or'
'ma'
'l '
'te'
'xt'
'.'