Calling toString() on potentially incomplete buffers

[matthiasg]

It seems byline is just calling toString() on buffers without correctly respecting unicode encoding rules. Since the buffers going into transform can be on any arbitrary position inside the original byte stream it is possible for it to be in the middle of a character.

This would potentially influence all characters encoded with more than 1 bytes.

if (encoding == 'buffer') {
  chunk = chunk.toString(); // utf8
  encoding = 'utf8';
}

Is this handled somewhere else ?

[LeonFedotov]

i just got an error like that with some Chinese characters where I didn't get the whole json that is present as expected, @matthiasg do you know how can I correct this issue?

[matthiasg]

@LeonFedotov you could use the stringdecoder as built into node directly. it can be continuously fed with new bytes and will emit proper unicode parsed strings only. each string can then be fed into byline (or just do it yourself).

[pludov]

As a workaround, for file streams, you can set the source stream encoding:
const stream = byline(fs.createReadStream(filename, {encoding: 'utf-8'}))