Support for IBM037 and IBM1140?

[jbowtie]

By my count there are four EBCDIC code pages likely to be encountered on the internet, though the comprehensive list of all registered IANA character sets found at https://www.iana.org/assignments/character-sets/character-sets.xhtml lists about 20 EBCDIC codepages.

Most of them are cross-referenced to the code page definitions in https://datatracker.ietf.org/doc/html/rfc1345 (which aggregates many of the original sources into a single reference document).

I'm happy to do this work for the two most common pages (IBM037 and IBM1140) as I am implementing an XML parser, however I don't know if it's something you would want in scope, behind a cargo feature, or segregated into an EBCDIC-specific crate.

[bonega]

Hi, I am open to including them.
Looking at how to implement them I discovered a bug with how Yore handles codepages that deviate from ascii.
Got to fix that first.
I expect that EBCDIC decoding performance will be quite a bit slower because we can't just memcpy from <=0xFF to utf8.
I will give you a ping when I have implemented it

[bonega]

I published ebcdic branch
Can you check it out?

[jbowtie]

It works as expected with my test cases so far - though to be fair I'm not exercising the encode functionality.

Do you have a test suite to build out? I could contribute some files, or maybe you want to encode with iconv / decode with yore?

[bonega]

I have fixed some bugs in a private branch, will push that shortly.
Will probably use iconv to make some tests.
The rest of yore is tested against oem_cp and encoding_rs, but they have no ebcdic encodings

[bonega]

I force pushed the ebcdic branch: https://github.com/bonega/yore/tree/ebcdic
Use "ebcdic" feature flag.
Please add PR for test data.
At the moment I am only running it through the invariant fuzzer.