Skip to content

fix(writer): escape the escape byte when double_quote is disabled - #432

Open
jaideeppyne wants to merge 1 commit into
BurntSushi:masterfrom
jaideeppyne:fix/escape-char-not-escaped-in-writer
Open

fix(writer): escape the escape byte when double_quote is disabled#432
jaideeppyne wants to merge 1 commit into
BurntSushi:masterfrom
jaideeppyne:fix/escape-char-not-escaped-in-writer

Conversation

@jaideeppyne

Copy link
Copy Markdown

When double_quote is disabled, the writer escapes quote characters with the escape character (default \), but it did not escape occurrences of the escape character itself in field data. Because the reader treats the escape character inside a quoted field as escaping the following byte, any escape character in the data was silently consumed on read-back.

Repro (current master):

let mut wtr = WriterBuilder::new().double_quote(false).from_writer(vec![]);
wtr.write_record([r"C:\path\to", "END"]).unwrap();
let data = wtr.into_inner().unwrap();
let mut rdr = ReaderBuilder::new().has_headers(false)
    .double_quote(false).escape(Some(b'\\')).from_reader(&*data);
// reads back ["C:pathto", "END"] instead of ["C:\path\to", "END"]

A lone \ field is worse: it swallows the closing quote and the following delimiter, mangling the whole record.

The writer already force-quotes fields containing the escape byte (requires_quotes[escape] = true), so the fix is to also double the escape byte inside csv_core::write::quote when double_quote is false, making the escaping self-consistent and the round-trip lossless. The default (double_quote = true) path is unchanged. Adds regression tests at the csv-core and csv levels.

When `double_quote` is disabled, the writer escapes quote characters with
the escape character (default `\`). However, it did not escape occurrences
of the escape character itself in field data. Because the reader treats the
escape character inside a quoted field as escaping the following byte, any
escape character present in the data was silently consumed on read-back,
corrupting fields (e.g. `C:\path\to` round-tripped to `C:pathto`, and a
field consisting of a lone `\` swallowed the closing quote and following
delimiter, mangling the whole record).

The writer already force-quotes fields containing the escape character
(`requires_quotes[escape] = true`), so the fix is to also double the escape
byte inside `csv_core::write::quote` when `double_quote` is false, making
the escaping self-consistent and the write/read round-trip lossless.

Adds regression tests at both the `csv-core` and `csv` levels.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant