Обсуждение: Errors in our encoding conversion tables
There's a discussion over at http://www.postgresql.org/message-id/flat/2sa.Dhu5.1hk1yrpTNFy.1MLOlb@seznam.cz of an apparent error in our WIN1250 -> LATIN2 conversion. I looked into this and found that indeed, the code will happily translate certain characters for which there seems to be no justification. I made up a quick script that would recompute the conversion tables in latin2_and_win1250.c from the Unicode mapping files in src/backend/utils/mb/Unicode, and what it computes is shown in the attached diff. (Zeroes in the tables indicate codes with no translation, for which an error should be thrown.) Having done that, I thought it would be a good idea to see if we had any other conversion tables that weren't directly based on the Unicode data. The only ones I could find were in cyrillic_and_mic.c, and those seem to be absolutely filled with errors, to the point where I wonder if they were made from the claimed encodings or some other ones. The attached patch recomputes those from the Unicode data, too. None of this data seems to have been touched since Tatsuo-san's original commit 969e0246, so it looks like we simply didn't vet that submission closely enough. I have not attempted to reverify the files in utils/mb/Unicode against the original Unicode Consortium data, but maybe we ought to do that before taking any further steps here. Anyway, what are we going to do about this? I'm concerned that simply shoving in corrections may cause problems for users. Almost certainly, we should not back-patch this kind of change. regards, tom lane diff --git a/src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c b/src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c index 5d1c59b..97e890d 100644 *** a/src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c --- b/src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c *************** iso2mic(const unsigned char *l, unsigned *** 433,439 **** 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0xb3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xe1, 0xe2, 0xf7, 0xe7, 0xe4, 0xe5, 0xf6, 0xfa, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, 0xf0, --- 433,439 ---- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x9a, 0xb3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xe1, 0xe2, 0xf7, 0xe7, 0xe4, 0xe5, 0xf6, 0xfa, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, 0xf0, *************** mic2iso(const unsigned char *mic, unsign *** 458,464 **** 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xf1, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xa1, 0x00, 0x00, 0x00, 0x00, --- 458,464 ---- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0xa0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xf1, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xa1, 0x00, 0x00, 0x00, 0x00, *************** win12512mic(const unsigned char *l, unsi *** 485,494 **** 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xbd, 0x00, 0x00, ! 0xb3, 0x00, 0xb4, 0x00, 0x00, 0x00, 0x00, 0xb7, ! 0x00, 0x00, 0xb6, 0xa6, 0xad, 0x00, 0x00, 0x00, ! 0xa3, 0x00, 0xa4, 0x00, 0x00, 0x00, 0x00, 0xa7, 0xe1, 0xe2, 0xf7, 0xe7, 0xe4, 0xe5, 0xf6, 0xfa, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, 0xf0, 0xf2, 0xf3, 0xf4, 0xf5, 0xe6, 0xe8, 0xe3, 0xfe, --- 485,494 ---- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x9a, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0xb3, 0xbf, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x9c, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x9e, ! 0xa3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xe1, 0xe2, 0xf7, 0xe7, 0xe4, 0xe5, 0xf6, 0xfa, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, 0xf0, 0xf2, 0xf3, 0xf4, 0xf5, 0xe6, 0xe8, 0xe3, 0xfe, *************** mic2win1251(const unsigned char *mic, un *** 510,520 **** 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0xb8, 0xba, 0x00, 0xb3, 0xbf, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xb4, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0xa8, 0xaa, 0x00, 0xb2, 0xaf, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xa5, 0x00, 0x00, 0xfe, 0xe0, 0xe1, 0xf6, 0xe4, 0xe5, 0xf4, 0xe3, 0xf5, 0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, 0xff, 0xf0, 0xf1, 0xf2, 0xf3, 0xe6, 0xe2, --- 510,520 ---- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0xa0, 0x00, 0xb0, 0x00, 0xb7, 0x00, + 0x00, 0x00, 0x00, 0xb8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0xa8, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xa9, 0xfe, 0xe0, 0xe1, 0xf6, 0xe4, 0xe5, 0xf4, 0xe3, 0xf5, 0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, 0xff, 0xf0, 0xf1, 0xf2, 0xf3, 0xe6, 0xe2, *************** win8662mic(const unsigned char *l, unsig *** 539,554 **** 0xfb, 0xfd, 0xff, 0xf9, 0xf8, 0xfc, 0xe0, 0xf1, 0xc1, 0xc2, 0xd7, 0xc7, 0xc4, 0xc5, 0xd6, 0xda, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, 0xd0, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xbd, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xd2, 0xd3, 0xd4, 0xd5, 0xc6, 0xc8, 0xc3, 0xde, 0xdb, 0xdd, 0xdf, 0xd9, 0xd8, 0xdc, 0xc0, 0xd1, ! 0xb3, 0xa3, 0xb4, 0xa4, 0xb7, 0xa7, 0x00, 0x00, ! 0xb6, 0xa6, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }; latin2mic_with_table(l, p, len, LC_KOI8_R, PG_WIN866, win8662koi); --- 539,554 ---- 0xfb, 0xfd, 0xff, 0xf9, 0xf8, 0xfc, 0xe0, 0xf1, 0xc1, 0xc2, 0xd7, 0xc7, 0xc4, 0xc5, 0xd6, 0xda, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, 0xd0, ! 0x90, 0x91, 0x92, 0x81, 0x87, 0xb2, 0xb4, 0xa7, ! 0xa6, 0xb5, 0xa1, 0xa8, 0xae, 0xad, 0xac, 0x83, ! 0x84, 0x89, 0x88, 0x86, 0x80, 0x8a, 0xaf, 0xb0, ! 0xab, 0xa5, 0xbb, 0xb8, 0xb1, 0xa0, 0xbe, 0xb9, ! 0xba, 0xb6, 0xb7, 0xaa, 0xa9, 0xa2, 0xa4, 0xbd, ! 0xbc, 0x85, 0x82, 0x8d, 0x8c, 0x8e, 0x8f, 0x8b, 0xd2, 0xd3, 0xd4, 0xd5, 0xc6, 0xc8, 0xc3, 0xde, 0xdb, 0xdd, 0xdf, 0xd9, 0xd8, 0xdc, 0xc0, 0xd1, ! 0xb3, 0xa3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x9c, 0x95, 0x9e, 0x96, 0x00, 0x00, 0x94, 0x9a }; latin2mic_with_table(l, p, len, LC_KOI8_R, PG_WIN866, win8662koi); *************** static void *** 559,572 **** mic2win866(const unsigned char *mic, unsigned char *p, int len) { static const unsigned char koi2win866[] = { ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0xf1, 0xf3, 0x00, 0xf9, 0xf5, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xad, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0xf0, 0xf2, 0x00, 0xf8, 0xf4, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xbd, 0x00, 0x00, 0xee, 0xa0, 0xa1, 0xe6, 0xa4, 0xa5, 0xe4, 0xa3, 0xe5, 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, 0xef, 0xe0, 0xe1, 0xe2, 0xe3, 0xa6, 0xa2, --- 559,572 ---- mic2win866(const unsigned char *mic, unsigned char *p, int len) { static const unsigned char koi2win866[] = { ! 0xc4, 0xb3, 0xda, 0xbf, 0xc0, 0xd9, 0xc3, 0xb4, ! 0xc2, 0xc1, 0xc5, 0xdf, 0xdc, 0xdb, 0xdd, 0xde, ! 0xb0, 0xb1, 0xb2, 0x00, 0xfe, 0xf9, 0xfb, 0x00, ! 0x00, 0x00, 0xff, 0x00, 0xf8, 0x00, 0xfa, 0x00, ! 0xcd, 0xba, 0xd5, 0xf1, 0xd6, 0xc9, 0xb8, 0xb7, ! 0xbb, 0xd4, 0xd3, 0xc8, 0xbe, 0xbd, 0xbc, 0xc6, ! 0xc7, 0xcc, 0xb5, 0xf0, 0xb6, 0xb9, 0xd1, 0xd2, ! 0xcb, 0xcf, 0xd0, 0xca, 0xd8, 0xd7, 0xce, 0x00, 0xee, 0xa0, 0xa1, 0xe6, 0xa4, 0xa5, 0xe4, 0xa3, 0xe5, 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, 0xef, 0xe0, 0xe1, 0xe2, 0xe3, 0xa6, 0xa2, diff --git a/src/backend/utils/mb/conversion_procs/latin2_and_win1250/latin2_and_win1250.c b/src/backend/utils/mb/conversion_procs/latin2_and_win1250/latin2_and_win1250.c index 8f831ba..5e35c75 100644 *** a/src/backend/utils/mb/conversion_procs/latin2_and_win1250/latin2_and_win1250.c --- b/src/backend/utils/mb/conversion_procs/latin2_and_win1250/latin2_and_win1250.c *************** static void *** 154,163 **** win12502mic(const unsigned char *l, unsigned char *p, int len) { static const unsigned char win1250_2_iso88592[] = { ! 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, ! 0x88, 0x89, 0xA9, 0x8B, 0xA6, 0xAB, 0xAE, 0xAC, ! 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, ! 0x98, 0x99, 0xB9, 0x9B, 0xB6, 0xBB, 0xBE, 0xBC, 0xA0, 0xB7, 0xA2, 0xA3, 0xA4, 0xA1, 0x00, 0xA7, 0xA8, 0x00, 0xAA, 0x00, 0x00, 0xAD, 0x00, 0xAF, 0xB0, 0x00, 0xB2, 0xB3, 0xB4, 0x00, 0x00, 0x00, --- 154,163 ---- win12502mic(const unsigned char *l, unsigned char *p, int len) { static const unsigned char win1250_2_iso88592[] = { ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0xA9, 0x00, 0xA6, 0xAB, 0xAE, 0xAC, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0xB9, 0x00, 0xB6, 0xBB, 0xBE, 0xBC, 0xA0, 0xB7, 0xA2, 0xA3, 0xA4, 0xA1, 0x00, 0xA7, 0xA8, 0x00, 0xAA, 0x00, 0x00, 0xAD, 0x00, 0xAF, 0xB0, 0x00, 0xB2, 0xB3, 0xB4, 0x00, 0x00, 0x00, *************** static void *** 180,189 **** mic2win1250(const unsigned char *mic, unsigned char *p, int len) { static const unsigned char iso88592_2_win1250[] = { ! 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, ! 0x88, 0x89, 0x00, 0x8B, 0x00, 0x00, 0x00, 0x00, ! 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, ! 0x98, 0x99, 0x00, 0x9B, 0x00, 0x00, 0x00, 0x00, 0xA0, 0xA5, 0xA2, 0xA3, 0xA4, 0xBC, 0x8C, 0xA7, 0xA8, 0x8A, 0xAA, 0x8D, 0x8F, 0xAD, 0x8E, 0xAF, 0xB0, 0xB9, 0xB2, 0xB3, 0xB4, 0xBE, 0x9C, 0xA1, --- 180,189 ---- mic2win1250(const unsigned char *mic, unsigned char *p, int len) { static const unsigned char iso88592_2_win1250[] = { ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xA0, 0xA5, 0xA2, 0xA3, 0xA4, 0xBC, 0x8C, 0xA7, 0xA8, 0x8A, 0xAA, 0x8D, 0x8F, 0xAD, 0x8E, 0xAF, 0xB0, 0xB9, 0xB2, 0xB3, 0xB4, 0xBE, 0x9C, 0xA1,
> There's a discussion over at > http://www.postgresql.org/message-id/flat/2sa.Dhu5.1hk1yrpTNFy.1MLOlb@seznam.cz > of an apparent error in our WIN1250 -> LATIN2 conversion. I looked into this > and found that indeed, the code will happily translate certain characters > for which there seems to be no justification. I made up a quick script > that would recompute the conversion tables in latin2_and_win1250.c from > the Unicode mapping files in src/backend/utils/mb/Unicode, and what it > computes is shown in the attached diff. (Zeroes in the tables indicate > codes with no translation, for which an error should be thrown.) > > Having done that, I thought it would be a good idea to see if we had any > other conversion tables that weren't directly based on the Unicode data. > The only ones I could find were in cyrillic_and_mic.c, and those seem to > be absolutely filled with errors, to the point where I wonder if they were > made from the claimed encodings or some other ones. The attached patch > recomputes those from the Unicode data, too. > > None of this data seems to have been touched since Tatsuo-san's original > commit 969e0246, so it looks like we simply didn't vet that submission > closely enough. > > I have not attempted to reverify the files in utils/mb/Unicode against the > original Unicode Consortium data, but maybe we ought to do that before > taking any further steps here. > > Anyway, what are we going to do about this? I'm concerned that simply > shoving in corrections may cause problems for users. Almost certainly, > we should not back-patch this kind of change. I have started to looking into it. I wonder how do you create the part of your patch: *** 154,163 **** win12502mic(const unsigned char *l, unsigned char *p, int len) { static const unsigned char win1250_2_iso88592[]= { ! 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, ! 0x88, 0x89, 0xA9, 0x8B, 0xA6, 0xAB, 0xAE, 0xAC, ! 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, ! 0x98, 0x99, 0xB9, 0x9B, 0xB6, 0xBB, 0xBE, 0xBC, 0xA0, 0xB7, 0xA2, 0xA3, 0xA4, 0xA1, 0x00, 0xA7, 0xA8, 0x00, 0xAA, 0x00, 0x00, 0xAD, 0x00, 0xAF, 0xB0, 0x00, 0xB2, 0xB3, 0xB4, 0x00, 0x00, 0x00, --- 154,163 ---- win12502mic(const unsigned char *l, unsigned char *p, int len) { static const unsigned char win1250_2_iso88592[]= { ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0xA9, 0x00, 0xA6, 0xAB, 0xAE, 0xAC, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0xB9, 0x00, 0xB6, 0xBB, 0xBE, 0xBC, 0xA0, 0xB7, 0xA2, 0xA3, 0xA4, 0xA1, 0x00, 0xA7, 0xA8, 0x00, 0xAA, 0x00, 0x00, 0xAD, 0x00, 0xAF, 0xB0, 0x00, 0xB2, 0xB3, 0xB4, 0x00, 0x00, 0x00, In the above you seem to disable the conversion from 0x96 of win1250 to ISO-8859-2 by using the Unicode mapping files in src/backend/utils/mb/Unicode. But the corresponding mapping file (iso8859_2_to_utf8.amp) does include following entry: {0x0096, 0xc296}, How do you know 0x96 should be removed from the conversion? Best regards, -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese:http://www.sraoss.co.jp
Tatsuo Ishii <ishii@postgresql.org> writes:
> I have started to looking into it. I wonder how do you create the part
> of your patch:
The code I used is below.
> In the above you seem to disable the conversion from 0x96 of win1250
> to ISO-8859-2 by using the Unicode mapping files in
> src/backend/utils/mb/Unicode. But the corresponding mapping file
> (iso8859_2_to_utf8.amp) does include following entry:
> {0x0096, 0xc296},
> How do you know 0x96 should be removed from the conversion?
Right, but there is no mapping in the win1250-utf8 files that matches
U+C296. The complaint over in the other thread is precisely that we
have no business translating 0x96 in WIN1250 to this character. What
WIN1250 0x96 could translate to is U+E28093 (at least, according to
win1250_to_utf8.map) but that Unicode character has no equivalent in
LATIN2.
AFAICS, whoever made these tables just arbitrarily decided that 0x96
in WIN1250 could be mapped to 0x96 in LATIN2, and likewise for a number
of other codes; but those are false equivalences, as you find out if
you try to perform the same conversion via other encoding conversion
paths, ie convert to UTF8 and then to the other encoding.
regards, tom lane
#include "c.h"
#include "mb/pg_wchar.h"
#include "src/backend/utils/mb/Unicode/iso8859_2_to_utf8.map"
#include "src/backend/utils/mb/Unicode/iso8859_5_to_utf8.map"
#include "src/backend/utils/mb/Unicode/win1250_to_utf8.map"
#include "src/backend/utils/mb/Unicode/win1251_to_utf8.map"
#include "src/backend/utils/mb/Unicode/win866_to_utf8.map"
#include "src/backend/utils/mb/Unicode/koi8r_to_utf8.map"
#include "src/backend/utils/mb/Unicode/koi8u_to_utf8.map"
typedef struct
{
const pg_local_to_utf *map1; /* to UTF8 map name */
int size1; /* size of map1 */
const pg_local_to_utf *map2; /* to UTF8 map name */
int size2; /* size of map2 */
const char *tabname;
int upper;
} pg_conv_map;
static const pg_conv_map maps[] = {
{
LUmapWIN1250, lengthof(LUmapWIN1250),
LUmapISO8859_2, lengthof(LUmapISO8859_2),
"win1250_2_iso88592", 1
},
{
LUmapISO8859_2, lengthof(LUmapISO8859_2),
LUmapWIN1250, lengthof(LUmapWIN1250),
"iso88592_2_win1250", 1
},
{
LUmapISO8859_5, lengthof(LUmapISO8859_5),
LUmapKOI8R, lengthof(LUmapKOI8R),
"iso2koi", 0
},
{
LUmapKOI8R, lengthof(LUmapKOI8R),
LUmapISO8859_5, lengthof(LUmapISO8859_5),
"koi2iso", 0
},
{
LUmapWIN1251, lengthof(LUmapWIN1251),
LUmapKOI8R, lengthof(LUmapKOI8R),
"win2koi", 0
},
{
LUmapKOI8R, lengthof(LUmapKOI8R),
LUmapWIN1251, lengthof(LUmapWIN1251),
"koi2win", 0
},
{
LUmapWIN866, lengthof(LUmapWIN866),
LUmapKOI8R, lengthof(LUmapKOI8R),
"win8662koi", 0
},
{
LUmapKOI8R, lengthof(LUmapKOI8R),
LUmapWIN866, lengthof(LUmapWIN866),
"koi2win866", 0
},
};
static void
domap(const pg_conv_map *info)
{
uint32 c;
printf(" static const unsigned char %s[] = {\n", info->tabname);
for (c = 0x80; c <= 0xff; c++)
{
uint32 u = 0;
uint32 c2 = 0;
int i;
for (i = 0; i < info->size1; i++)
{
if (info->map1[i].code == c)
{
u = info->map1[i].utf;
break;
}
}
if (u != 0)
{
for (i = 0; i < info->size2; i++)
{
if (info->map2[i].utf == u)
{
c2 = info->map2[i].code;
break;
}
}
}
#if 0
if (c2)
printf("0x%02x maps to 0x%02x via U+%04X\n", c, c2, u);
else
printf("0x%02x has no equivalent\n", c);
#endif
if (c % 8 == 0)
printf("\t\t");
if (info->upper)
printf("0x%02X", c2);
else
printf("0x%02x", c2);
if (c == 0xff)
printf("\n");
else if (c % 8 == 7)
printf(",\n");
else
printf(", ");
}
printf("\t};\n\n");
}
int
main()
{
int i;
for (i = 0; i < lengthof(maps); i++)
domap(maps + i);
return 0;
}
Tom Lane wrote: > There's a discussion over at > http://www.postgresql.org/message-id/flat/2sa.Dhu5.1hk1yrpTNFy.1MLOlb@seznam.cz > of an apparent error in our WIN1250 -> LATIN2 conversion. I looked into this > and found that indeed, the code will happily translate certain characters > for which there seems to be no justification. I made up a quick script > that would recompute the conversion tables in latin2_and_win1250.c from > the Unicode mapping files in src/backend/utils/mb/Unicode, and what it > computes is shown in the attached diff. (Zeroes in the tables indicate > codes with no translation, for which an error should be thrown.) > > Having done that, I thought it would be a good idea to see if we had any > other conversion tables that weren't directly based on the Unicode data. > The only ones I could find were in cyrillic_and_mic.c, and those seem to > be absolutely filled with errors, to the point where I wonder if they were > made from the claimed encodings or some other ones. The attached patch > recomputes those from the Unicode data, too. > > None of this data seems to have been touched since Tatsuo-san's original > commit 969e0246, so it looks like we simply didn't vet that submission > closely enough. > > I have not attempted to reverify the files in utils/mb/Unicode against the > original Unicode Consortium data, but maybe we ought to do that before > taking any further steps here. > > Anyway, what are we going to do about this? I'm concerned that simply > shoving in corrections may cause problems for users. Almost certainly, > we should not back-patch this kind of change. Thanks for picking this up. I agree with your proposed fix, the only thing that makes me feel uncomfortable is that you get error messages like: ERROR: character with byte sequence 0x96 in encoding "WIN1250" has no equivalent inencoding "MULE_INTERNAL" which is a bit misleading. But the main thing is that no corrupt data can be entered. I can understand the reluctance to back-patch; nobody likes his application to suddenly fail after a minor database upgrade. However, the people who would fail if this were back-patched are people who will certainly run into trouble if they a) upgrade to a release where this is fixed or b) try to convert their database to, say, UTF8. The least thing we should do is stick a fat warning into the release notes of the first version where this is fixed, along with some guidelines what to do (though I am afraid that there is not much more helpful to say than "If your database encoding is X and data have been entered with client_encoding Y, fix your data in the old system"). But I think that this fix should be applied to 9.6. PostgreSQL has a strong reputation for being strict about correct encoding (not saying that everybody appreciates that), and I think we shouldn't mar that reputation. Yours, Laurenz Albe
Albe Laurenz <laurenz.albe@wien.gv.at> writes:
> I agree with your proposed fix, the only thing that makes me feel uncomfortable
> is that you get error messages like:
> ERROR: character with byte sequence 0x96 in encoding "WIN1250" has no equivalent in encoding "MULE_INTERNAL"
Hm, yeah. It's pretty silly that this code uses a double conversion when
it has a table that would work fine for a direct conversion, anyway. But
I think improving that is a separate question, especially since the above
behavior occurred already for a few code points.
regards, tom lane
I wrote:
> I have not attempted to reverify the files in utils/mb/Unicode against the
> original Unicode Consortium data, but maybe we ought to do that before
> taking any further steps here.
I downloaded the mapping files from unicode.org and attempted to verify
that the Unicode/*.map files could be reproduced from the stated sources.
Most of them are okay, but I failed to verify these:
euc_cn_to_utf8.map utf8_to_euc_cn.map
Could not find the reference file GB2312.TXT; it is not at unicode.org
gb18030_to_utf8.map utf8_to_gb18030.map
Could not find the reference file gb-18030-2000.xml, whose origin is
unstated anyway.
euc_jp_to_utf8.map utf8_to_euc_jp.map
euc_kr_to_utf8.map utf8_to_euc_kr.map
johab_to_utf8.map utf8_to_johab.map
uhc_to_utf8.map utf8_to_uhc.map
These four all have minor to significant differences from what I got by
running the generation scripts. See attached diffs.
utf8_to_sjis.map
It's very disturbing that this fails to verify when its allegedly inverse
file does verify; either the script is broken or somebody did sloppy
manual editing.
Anyway, this seems to mean that it's okay to go ahead with fixing the
encoding conversion discrepancies I complained of yesterday; the data
those proposed diffs are based on is solid. But we've evidently got
a number of other issues with these Far Eastern encodings.
regards, tom lane
*** euc_jp_to_utf8.map.orig Fri May 15 17:56:06 2015
--- euc_jp_to_utf8.map Fri Nov 27 16:25:37 2015
***************
*** 1,6 **** /* src/backend/utils/mb/Unicode/euc_jp_to_utf8.map */
! static const pg_local_to_utf LUmapEUC_JP[] = { {0x8ea1, 0xefbda1}, {0x8ea2, 0xefbda2}, {0x8ea3, 0xefbda3},
--- 1,6 ---- /* src/backend/utils/mb/Unicode/euc_jp_to_utf8.map */
! static const pg_local_to_utf LUmapEUC_JP[ 13007 ] = { {0x8ea1, 0xefbda1}, {0x8ea2, 0xefbda2}, {0x8ea3,
0xefbda3},
***************
*** 95,103 **** {0xa1bd, 0xe28095}, {0xa1be, 0xe28090}, {0xa1bf, 0xefbc8f},
! {0xa1c0, 0xefbcbc},
! {0xa1c1, 0xefbd9e},
! {0xa1c2, 0xe288a5}, {0xa1c3, 0xefbd9c}, {0xa1c4, 0xe280a6}, {0xa1c5, 0xe280a5},
--- 95,102 ---- {0xa1bd, 0xe28095}, {0xa1be, 0xe28090}, {0xa1bf, 0xefbc8f},
! {0xa1c1, 0xe3809c},
! {0xa1c2, 0xe28096}, {0xa1c3, 0xefbd9c}, {0xa1c4, 0xe280a6}, {0xa1c5, 0xe280a5},
***************
*** 124,130 **** {0xa1da, 0xe38090}, {0xa1db, 0xe38091}, {0xa1dc, 0xefbc8b},
! {0xa1dd, 0xefbc8d}, {0xa1de, 0xc2b1}, {0xa1df, 0xc397}, {0xa1e0, 0xc3b7},
--- 123,129 ---- {0xa1da, 0xe38090}, {0xa1db, 0xe38091}, {0xa1dc, 0xefbc8b},
! {0xa1dd, 0xe28892}, {0xa1de, 0xc2b1}, {0xa1df, 0xc397}, {0xa1e0, 0xc3b7},
***************
*** 144,151 **** {0xa1ee, 0xe28483}, {0xa1ef, 0xefbfa5}, {0xa1f0, 0xefbc84},
! {0xa1f1, 0xefbfa0},
! {0xa1f2, 0xefbfa1}, {0xa1f3, 0xefbc85}, {0xa1f4, 0xefbc83}, {0xa1f5, 0xefbc86},
--- 143,150 ---- {0xa1ee, 0xe28483}, {0xa1ef, 0xefbfa5}, {0xa1f0, 0xefbc84},
! {0xa1f1, 0xc2a2},
! {0xa1f2, 0xc2a3}, {0xa1f3, 0xefbc85}, {0xa1f4, 0xefbc83}, {0xa1f5, 0xefbc86},
***************
*** 182,188 **** {0xa2c1, 0xe288a9}, {0xa2ca, 0xe288a7}, {0xa2cb, 0xe288a8},
! {0xa2cc, 0xefbfa2}, {0xa2cd, 0xe28792}, {0xa2ce, 0xe28794}, {0xa2cf, 0xe28880},
--- 181,187 ---- {0xa2c1, 0xe288a9}, {0xa2ca, 0xe288a7}, {0xa2cb, 0xe288a8},
! {0xa2cc, 0xc2ac}, {0xa2cd, 0xe28792}, {0xa2ce, 0xe28794}, {0xa2cf, 0xe28880},
***************
*** 588,676 **** {0xa8be, 0xe294a5}, {0xa8bf, 0xe294b8}, {0xa8c0, 0xe29582},
- {0xada1, 0xe291a0},
- {0xada2, 0xe291a1},
- {0xada3, 0xe291a2},
- {0xada4, 0xe291a3},
- {0xada5, 0xe291a4},
- {0xada6, 0xe291a5},
- {0xada7, 0xe291a6},
- {0xada8, 0xe291a7},
- {0xada9, 0xe291a8},
- {0xadaa, 0xe291a9},
- {0xadab, 0xe291aa},
- {0xadac, 0xe291ab},
- {0xadad, 0xe291ac},
- {0xadae, 0xe291ad},
- {0xadaf, 0xe291ae},
- {0xadb0, 0xe291af},
- {0xadb1, 0xe291b0},
- {0xadb2, 0xe291b1},
- {0xadb3, 0xe291b2},
- {0xadb4, 0xe291b3},
- {0xadb5, 0xe285a0},
- {0xadb6, 0xe285a1},
- {0xadb7, 0xe285a2},
- {0xadb8, 0xe285a3},
- {0xadb9, 0xe285a4},
- {0xadba, 0xe285a5},
- {0xadbb, 0xe285a6},
- {0xadbc, 0xe285a7},
- {0xadbd, 0xe285a8},
- {0xadbe, 0xe285a9},
- {0xadc0, 0xe38d89},
- {0xadc1, 0xe38c94},
- {0xadc2, 0xe38ca2},
- {0xadc3, 0xe38d8d},
- {0xadc4, 0xe38c98},
- {0xadc5, 0xe38ca7},
- {0xadc6, 0xe38c83},
- {0xadc7, 0xe38cb6},
- {0xadc8, 0xe38d91},
- {0xadc9, 0xe38d97},
- {0xadca, 0xe38c8d},
- {0xadcb, 0xe38ca6},
- {0xadcc, 0xe38ca3},
- {0xadcd, 0xe38cab},
- {0xadce, 0xe38d8a},
- {0xadcf, 0xe38cbb},
- {0xadd0, 0xe38e9c},
- {0xadd1, 0xe38e9d},
- {0xadd2, 0xe38e9e},
- {0xadd3, 0xe38e8e},
- {0xadd4, 0xe38e8f},
- {0xadd5, 0xe38f84},
- {0xadd6, 0xe38ea1},
- {0xaddf, 0xe38dbb},
- {0xade0, 0xe3809d},
- {0xade1, 0xe3809f},
- {0xade2, 0xe28496},
- {0xade3, 0xe38f8d},
- {0xade4, 0xe284a1},
- {0xade5, 0xe38aa4},
- {0xade6, 0xe38aa5},
- {0xade7, 0xe38aa6},
- {0xade8, 0xe38aa7},
- {0xade9, 0xe38aa8},
- {0xadea, 0xe388b1},
- {0xadeb, 0xe388b2},
- {0xadec, 0xe388b9},
- {0xaded, 0xe38dbe},
- {0xadee, 0xe38dbd},
- {0xadef, 0xe38dbc},
- {0xadf0, 0xe28992},
- {0xadf1, 0xe289a1},
- {0xadf2, 0xe288ab},
- {0xadf3, 0xe288ae},
- {0xadf4, 0xe28891},
- {0xadf5, 0xe2889a},
- {0xadf6, 0xe28aa5},
- {0xadf7, 0xe288a0},
- {0xadf8, 0xe2889f},
- {0xadf9, 0xe28abf},
- {0xadfa, 0xe288b5},
- {0xadfb, 0xe288a9},
- {0xadfc, 0xe288aa}, {0xb0a1, 0xe4ba9c}, {0xb0a2, 0xe59496}, {0xb0a3, 0xe5a883},
--- 587,592 ----
***************
*** 7037,7043 **** {0x8fa2b8, 0xce84}, {0x8fa2b9, 0xce85}, {0x8fa2c2, 0xc2a1},
! {0x8fa2c3, 0xefbfa4}, {0x8fa2c4, 0xc2bf}, {0x8fa2eb, 0xc2ba}, {0x8fa2ec, 0xc2aa},
--- 6953,6959 ---- {0x8fa2b8, 0xce84}, {0x8fa2b9, 0xce85}, {0x8fa2c2, 0xc2a1},
! {0x8fa2c3, 0xc2a6}, {0x8fa2c4, 0xc2bf}, {0x8fa2eb, 0xc2ba}, {0x8fa2ec, 0xc2aa},
***************
*** 13091,13201 **** {0x8fede0, 0xe9bea1}, {0x8fede1, 0xe9bea2}, {0x8fede2, 0xe9bea3},
! {0x8fede3, 0xe9bea5},
! {0x8ff3f3, 0xe285b0},
! {0x8ff3f4, 0xe285b1},
! {0x8ff3f5, 0xe285b2},
! {0x8ff3f6, 0xe285b3},
! {0x8ff3f7, 0xe285b4},
! {0x8ff3f8, 0xe285b5},
! {0x8ff3f9, 0xe285b6},
! {0x8ff3fa, 0xe285b7},
! {0x8ff3fb, 0xe285b8},
! {0x8ff3fc, 0xe285b9},
! {0x8ff3fd, 0xe285a0},
! {0x8ff3fe, 0xe285a1},
! {0x8ff4a1, 0xe285a2},
! {0x8ff4a2, 0xe285a3},
! {0x8ff4a3, 0xe285a4},
! {0x8ff4a4, 0xe285a5},
! {0x8ff4a5, 0xe285a6},
! {0x8ff4a6, 0xe285a7},
! {0x8ff4a7, 0xe285a8},
! {0x8ff4a8, 0xe285a9},
! {0x8ff4a9, 0xefbc87},
! {0x8ff4aa, 0xefbc82},
! {0x8ff4ab, 0xe388b1},
! {0x8ff4ac, 0xe28496},
! {0x8ff4ad, 0xe284a1},
! {0x8ff4ae, 0xe782bb},
! {0x8ff4af, 0xe4bbbc},
! {0x8ff4b0, 0xe583b4},
! {0x8ff4b1, 0xe587ac},
! {0x8ff4b2, 0xe58c87},
! {0x8ff4b3, 0xe58ca4},
! {0x8ff4b4, 0xefa88e},
! {0x8ff4b5, 0xe5928a},
! {0x8ff4b6, 0xe59d99},
! {0x8ff4b7, 0xefa88f},
! {0x8ff4b8, 0xefa890},
! {0x8ff4b9, 0xe5a29e},
! {0x8ff4ba, 0xe5afac},
! {0x8ff4bb, 0xe5b3b5},
! {0x8ff4bc, 0xe5b593},
! {0x8ff4bd, 0xefa891},
! {0x8ff4be, 0xe5beb7},
! {0x8ff4bf, 0xe68285},
! {0x8ff4c0, 0xe684a0},
! {0x8ff4c1, 0xe6958e},
! {0x8ff4c2, 0xe698bb},
! {0x8ff4c3, 0xe699a5},
! {0x8ff4c4, 0xefa892},
! {0x8ff4c5, 0xefa4a9},
! {0x8ff4c6, 0xe6a081},
! {0x8ff4c7, 0xefa893},
! {0x8ff4c8, 0xefa894},
! {0x8ff4c9, 0xe6a9ab},
! {0x8ff4ca, 0xe6aba2},
! {0x8ff4cb, 0xe6b7b8},
! {0x8ff4cc, 0xe6b7b2},
! {0x8ff4cd, 0xe780a8},
! {0x8ff4ce, 0xefa895},
! {0x8ff4cf, 0xefa896},
! {0x8ff4d0, 0xe79481},
! {0x8ff4d1, 0xe79a82},
! {0x8ff4d2, 0xe79a9e},
! {0x8ff4d3, 0xefa897},
! {0x8ff4d4, 0xe7a4b0},
! {0x8ff4d5, 0xefa898},
! {0x8ff4d6, 0xefa899},
! {0x8ff4d7, 0xefa89a},
! {0x8ff4d8, 0xefa89b},
! {0x8ff4d9, 0xe7aba7},
! {0x8ff4da, 0xefa89c},
! {0x8ff4db, 0xefa89d},
! {0x8ff4dc, 0xe7b6a0},
! {0x8ff4dd, 0xe7b796},
! {0x8ff4de, 0xefa89e},
! {0x8ff4df, 0xe88da2},
! {0x8ff4e0, 0xefa89f},
! {0x8ff4e1, 0xe896b0},
! {0x8ff4e2, 0xefa8a0},
! {0x8ff4e3, 0xefa8a1},
! {0x8ff4e4, 0xe8a087},
! {0x8ff4e5, 0xefa8a2},
! {0x8ff4e6, 0xe8adbf},
! {0x8ff4e7, 0xe8b3b4},
! {0x8ff4e8, 0xe8b5b6},
! {0x8ff4e9, 0xefa8a3},
! {0x8ff4ea, 0xefa8a4},
! {0x8ff4eb, 0xefa8a5},
! {0x8ff4ec, 0xe9839e},
! {0x8ff4ed, 0xefa8a6},
! {0x8ff4ee, 0xe98495},
! {0x8ff4ef, 0xefa8a7},
! {0x8ff4f0, 0xefa8a8},
! {0x8ff4f1, 0xe99692},
! {0x8ff4f2, 0xefa79c},
! {0x8ff4f3, 0xefa8a9},
! {0x8ff4f4, 0xe99cbb},
! {0x8ff4f5, 0xe99d8d},
! {0x8ff4f6, 0xe99d91},
! {0x8ff4f7, 0xefa8aa},
! {0x8ff4f8, 0xefa8ab},
! {0x8ff4f9, 0xefa8ac},
! {0x8ff4fa, 0xe9a69e},
! {0x8ff4fb, 0xe9ab99},
! {0x8ff4fc, 0xe9adb2},
! {0x8ff4fd, 0xefa8ad},
! {0x8ff4fe, 0xe9bb91}, };
--- 13007,13011 ---- {0x8fede0, 0xe9bea1}, {0x8fede1, 0xe9bea2}, {0x8fede2, 0xe9bea3},
! {0x8fede3, 0xe9bea5} };
*** euc_kr_to_utf8.map.orig Fri May 15 17:56:06 2015
--- euc_kr_to_utf8.map Fri Nov 27 16:25:41 2015
***************
*** 1,4 ****
! static const pg_local_to_utf LUmapEUC_KR[ 8227 ] = { {0xa1a1, 0xe38080}, {0xa1a2, 0xe38081}, {0xa1a3,
0xe38082},
--- 1,6 ----
! /* src/backend/utils/mb/Unicode/euc_kr_to_utf8.map */
!
! static const pg_local_to_utf LUmapEUC_KR[ 8224 ] = { {0xa1a1, 0xe38080}, {0xa1a2, 0xe38081}, {0xa1a3,
0xe38082},
***************
*** 162,170 **** {0xa2e3, 0xe38f82}, {0xa2e4, 0xe38f98}, {0xa2e5, 0xe284a1},
- {0xa2e6, 0xe282ac},
- {0xa2e7, 0xc2ae},
- {0xa2e8, 0xe389be}, {0xa3a1, 0xefbc81}, {0xa3a2, 0xefbc82}, {0xa3a3, 0xefbc83},
--- 164,169 ----
*** johab_to_utf8.map.orig Fri May 15 17:56:06 2015
--- johab_to_utf8.map Fri Nov 27 16:26:10 2015
***************
*** 1,4 ****
! static const pg_local_to_utf LUmapJOHAB[ 17049 ] = { {0x8444, 0xe384b3}, {0x8446, 0xe384b5}, {0x8447,
0xe384b6},
--- 1,6 ----
! /* src/backend/utils/mb/Unicode/johab_to_utf8.map */
!
! static const pg_local_to_utf LUmapJOHAB[ 17046 ] = { {0x8444, 0xe384b3}, {0x8446, 0xe384b5}, {0x8447,
0xe384b6},
***************
*** 11385,11393 **** {0xd9e3, 0xe38f82}, {0xd9e4, 0xe38f98}, {0xd9e5, 0xe284a1},
- {0xd9e6, 0xe282ac},
- {0xd9e7, 0xc2ae},
- {0xd9e8, 0xe389be}, {0xda31, 0xefbc81}, {0xda32, 0xefbc82}, {0xda33, 0xefbc83},
--- 11387,11392 ----
*** uhc_to_utf8.map.orig Fri May 15 17:56:06 2015
--- uhc_to_utf8.map Fri Nov 27 16:26:11 2015
***************
*** 1,4 ****
! static const pg_local_to_utf LUmapUHC[ 17237 ] = { {0x8141, 0xeab082}, {0x8142, 0xeab083}, {0x8143, 0xeab085},
--- 1,6 ----
! /* src/backend/utils/mb/Unicode/uhc_to_utf8.map */
!
! static const pg_local_to_utf LUmapUHC[ 17048 ] = { {0x8141, 0xeab082}, {0x8142, 0xeab083}, {0x8143, 0xeab085},
***************
*** 6028,6034 **** {0xa2e5, 0xe284a1}, {0xa2e6, 0xe282ac}, {0xa2e7, 0xc2ae},
- {0xa2e8, 0xe389be}, {0xa341, 0xeca5b1}, {0xa342, 0xeca5b2}, {0xa343, 0xeca5b3},
--- 6030,6035 ----
***************
*** 12160,12259 **** {0xc8fc, 0xed9e99}, {0xc8fd, 0xed9e9b}, {0xc8fe, 0xed9e9d},
- {0xc9a1, 0xee8080},
- {0xc9a2, 0xee8081},
- {0xc9a3, 0xee8082},
- {0xc9a4, 0xee8083},
- {0xc9a5, 0xee8084},
- {0xc9a6, 0xee8085},
- {0xc9a7, 0xee8086},
- {0xc9a8, 0xee8087},
- {0xc9a9, 0xee8088},
- {0xc9aa, 0xee8089},
- {0xc9ab, 0xee808a},
- {0xc9ac, 0xee808b},
- {0xc9ad, 0xee808c},
- {0xc9ae, 0xee808d},
- {0xc9af, 0xee808e},
- {0xc9b0, 0xee808f},
- {0xc9b1, 0xee8090},
- {0xc9b2, 0xee8091},
- {0xc9b3, 0xee8092},
- {0xc9b4, 0xee8093},
- {0xc9b5, 0xee8094},
- {0xc9b6, 0xee8095},
- {0xc9b7, 0xee8096},
- {0xc9b8, 0xee8097},
- {0xc9b9, 0xee8098},
- {0xc9ba, 0xee8099},
- {0xc9bb, 0xee809a},
- {0xc9bc, 0xee809b},
- {0xc9bd, 0xee809c},
- {0xc9be, 0xee809d},
- {0xc9bf, 0xee809e},
- {0xc9c0, 0xee809f},
- {0xc9c1, 0xee80a0},
- {0xc9c2, 0xee80a1},
- {0xc9c3, 0xee80a2},
- {0xc9c4, 0xee80a3},
- {0xc9c5, 0xee80a4},
- {0xc9c6, 0xee80a5},
- {0xc9c7, 0xee80a6},
- {0xc9c8, 0xee80a7},
- {0xc9c9, 0xee80a8},
- {0xc9ca, 0xee80a9},
- {0xc9cb, 0xee80aa},
- {0xc9cc, 0xee80ab},
- {0xc9cd, 0xee80ac},
- {0xc9ce, 0xee80ad},
- {0xc9cf, 0xee80ae},
- {0xc9d0, 0xee80af},
- {0xc9d1, 0xee80b0},
- {0xc9d2, 0xee80b1},
- {0xc9d3, 0xee80b2},
- {0xc9d4, 0xee80b3},
- {0xc9d5, 0xee80b4},
- {0xc9d6, 0xee80b5},
- {0xc9d7, 0xee80b6},
- {0xc9d8, 0xee80b7},
- {0xc9d9, 0xee80b8},
- {0xc9da, 0xee80b9},
- {0xc9db, 0xee80ba},
- {0xc9dc, 0xee80bb},
- {0xc9dd, 0xee80bc},
- {0xc9de, 0xee80bd},
- {0xc9df, 0xee80be},
- {0xc9e0, 0xee80bf},
- {0xc9e1, 0xee8180},
- {0xc9e2, 0xee8181},
- {0xc9e3, 0xee8182},
- {0xc9e4, 0xee8183},
- {0xc9e5, 0xee8184},
- {0xc9e6, 0xee8185},
- {0xc9e7, 0xee8186},
- {0xc9e8, 0xee8187},
- {0xc9e9, 0xee8188},
- {0xc9ea, 0xee8189},
- {0xc9eb, 0xee818a},
- {0xc9ec, 0xee818b},
- {0xc9ed, 0xee818c},
- {0xc9ee, 0xee818d},
- {0xc9ef, 0xee818e},
- {0xc9f0, 0xee818f},
- {0xc9f1, 0xee8190},
- {0xc9f2, 0xee8191},
- {0xc9f3, 0xee8192},
- {0xc9f4, 0xee8193},
- {0xc9f5, 0xee8194},
- {0xc9f6, 0xee8195},
- {0xc9f7, 0xee8196},
- {0xc9f8, 0xee8197},
- {0xc9f9, 0xee8198},
- {0xc9fa, 0xee8199},
- {0xc9fb, 0xee819a},
- {0xc9fc, 0xee819b},
- {0xc9fd, 0xee819c},
- {0xc9fe, 0xee819d}, {0xcaa1, 0xe4bcbd}, {0xcaa2, 0xe4bdb3}, {0xcaa3, 0xe58187},
--- 12161,12166 ----
***************
*** 17141,17239 **** {0xfdfb, 0xe7a6a7}, {0xfdfc, 0xe7a880}, {0xfdfd, 0xe7beb2},
! {0xfdfe, 0xe8a9b0},
! {0xfea1, 0xee819e},
! {0xfea2, 0xee819f},
! {0xfea3, 0xee81a0},
! {0xfea4, 0xee81a1},
! {0xfea5, 0xee81a2},
! {0xfea6, 0xee81a3},
! {0xfea7, 0xee81a4},
! {0xfea8, 0xee81a5},
! {0xfea9, 0xee81a6},
! {0xfeaa, 0xee81a7},
! {0xfeab, 0xee81a8},
! {0xfeac, 0xee81a9},
! {0xfead, 0xee81aa},
! {0xfeae, 0xee81ab},
! {0xfeaf, 0xee81ac},
! {0xfeb0, 0xee81ad},
! {0xfeb1, 0xee81ae},
! {0xfeb2, 0xee81af},
! {0xfeb3, 0xee81b0},
! {0xfeb4, 0xee81b1},
! {0xfeb5, 0xee81b2},
! {0xfeb6, 0xee81b3},
! {0xfeb7, 0xee81b4},
! {0xfeb8, 0xee81b5},
! {0xfeb9, 0xee81b6},
! {0xfeba, 0xee81b7},
! {0xfebb, 0xee81b8},
! {0xfebc, 0xee81b9},
! {0xfebd, 0xee81ba},
! {0xfebe, 0xee81bb},
! {0xfebf, 0xee81bc},
! {0xfec0, 0xee81bd},
! {0xfec1, 0xee81be},
! {0xfec2, 0xee81bf},
! {0xfec3, 0xee8280},
! {0xfec4, 0xee8281},
! {0xfec5, 0xee8282},
! {0xfec6, 0xee8283},
! {0xfec7, 0xee8284},
! {0xfec8, 0xee8285},
! {0xfec9, 0xee8286},
! {0xfeca, 0xee8287},
! {0xfecb, 0xee8288},
! {0xfecc, 0xee8289},
! {0xfecd, 0xee828a},
! {0xfece, 0xee828b},
! {0xfecf, 0xee828c},
! {0xfed0, 0xee828d},
! {0xfed1, 0xee828e},
! {0xfed2, 0xee828f},
! {0xfed3, 0xee8290},
! {0xfed4, 0xee8291},
! {0xfed5, 0xee8292},
! {0xfed6, 0xee8293},
! {0xfed7, 0xee8294},
! {0xfed8, 0xee8295},
! {0xfed9, 0xee8296},
! {0xfeda, 0xee8297},
! {0xfedb, 0xee8298},
! {0xfedc, 0xee8299},
! {0xfedd, 0xee829a},
! {0xfede, 0xee829b},
! {0xfedf, 0xee829c},
! {0xfee0, 0xee829d},
! {0xfee1, 0xee829e},
! {0xfee2, 0xee829f},
! {0xfee3, 0xee82a0},
! {0xfee4, 0xee82a1},
! {0xfee5, 0xee82a2},
! {0xfee6, 0xee82a3},
! {0xfee7, 0xee82a4},
! {0xfee8, 0xee82a5},
! {0xfee9, 0xee82a6},
! {0xfeea, 0xee82a7},
! {0xfeeb, 0xee82a8},
! {0xfeec, 0xee82a9},
! {0xfeed, 0xee82aa},
! {0xfeee, 0xee82ab},
! {0xfeef, 0xee82ac},
! {0xfef0, 0xee82ad},
! {0xfef1, 0xee82ae},
! {0xfef2, 0xee82af},
! {0xfef3, 0xee82b0},
! {0xfef4, 0xee82b1},
! {0xfef5, 0xee82b2},
! {0xfef6, 0xee82b3},
! {0xfef7, 0xee82b4},
! {0xfef8, 0xee82b5},
! {0xfef9, 0xee82b6},
! {0xfefa, 0xee82b7},
! {0xfefb, 0xee82b8},
! {0xfefc, 0xee82b9},
! {0xfefd, 0xee82ba},
! {0xfefe, 0xee82bb} };
--- 17048,17052 ---- {0xfdfb, 0xe7a6a7}, {0xfdfc, 0xe7a880}, {0xfdfd, 0xe7beb2},
! {0xfdfe, 0xe8a9b0} };
*** utf8_to_euc_jp.map.orig Fri May 15 17:56:06 2015
--- utf8_to_euc_jp.map Fri Nov 27 16:25:37 2015
***************
*** 1,11 ****
! static const pg_utf_to_local ULmapEUC_JP[ 13175 ] = { {0xc2a1, 0x8fa2c2}, {0xc2a4, 0x8fa2f0}, {0xc2a6,
0x8fa2c3}, {0xc2a7, 0xa1f8}, {0xc2a8, 0xa1af}, {0xc2a9, 0x8fa2ed}, {0xc2aa, 0x8fa2ec}, {0xc2ae, 0x8fa2ee},
{0xc2af,0x8fa2b4}, {0xc2b0, 0xa1eb},
--- 1,16 ----
! /* src/backend/utils/mb/Unicode/utf8_to_euc_jp.map */
!
! static const pg_utf_to_local ULmapEUC_JP[ 13007 ] = { {0xc2a1, 0x8fa2c2},
+ {0xc2a2, 0xa1f1},
+ {0xc2a3, 0xa1f2}, {0xc2a4, 0x8fa2f0}, {0xc2a6, 0x8fa2c3}, {0xc2a7, 0xa1f8}, {0xc2a8, 0xa1af}, {0xc2a9,
0x8fa2ed}, {0xc2aa, 0x8fa2ec},
+ {0xc2ac, 0xa2cc}, {0xc2ae, 0x8fa2ee}, {0xc2af, 0x8fa2b4}, {0xc2b0, 0xa1eb},
***************
*** 386,391 ****
--- 391,397 ---- {0xd19f, 0x8fa7fe}, {0xe28090, 0xa1be}, {0xe28095, 0xa1bd},
+ {0xe28096, 0xa1c2}, {0xe28098, 0xa1c6}, {0xe28099, 0xa1c7}, {0xe2809c, 0xa1c8},
***************
*** 399,428 **** {0xe280b3, 0xa1ed}, {0xe280bb, 0xa2a8}, {0xe28483, 0xa1ee},
! {0xe28496, 0xade2},
! {0xe284a1, 0xade4}, {0xe284a2, 0x8fa2ef}, {0xe284ab, 0xa2f2},
- {0xe285a0, 0xadb5},
- {0xe285a1, 0xadb6},
- {0xe285a2, 0xadb7},
- {0xe285a3, 0xadb8},
- {0xe285a4, 0xadb9},
- {0xe285a5, 0xadba},
- {0xe285a6, 0xadbb},
- {0xe285a7, 0xadbc},
- {0xe285a8, 0xadbd},
- {0xe285a9, 0xadbe},
- {0xe285b0, 0x8ff3f3},
- {0xe285b1, 0x8ff3f4},
- {0xe285b2, 0x8ff3f5},
- {0xe285b3, 0x8ff3f6},
- {0xe285b4, 0x8ff3f7},
- {0xe285b5, 0x8ff3f8},
- {0xe285b6, 0x8ff3f9},
- {0xe285b7, 0x8ff3fa},
- {0xe285b8, 0x8ff3fb},
- {0xe285b9, 0x8ff3fc}, {0xe28690, 0xa2ab}, {0xe28691, 0xa2ac}, {0xe28692, 0xa2aa},
--- 405,413 ---- {0xe280b3, 0xa1ed}, {0xe280bb, 0xa2a8}, {0xe28483, 0xa1ee},
! {0xe28496, 0x8fa2f1}, {0xe284a2, 0x8fa2ef}, {0xe284ab, 0xa2f2}, {0xe28690, 0xa2ab}, {0xe28691, 0xa2ac},
{0xe28692,0xa2aa},
***************
*** 435,454 **** {0xe28887, 0xa2e0}, {0xe28888, 0xa2ba}, {0xe2888b, 0xa2bb},
! {0xe28891, 0xadf4}, {0xe2889a, 0xa2e5}, {0xe2889d, 0xa2e7}, {0xe2889e, 0xa1e7},
- {0xe2889f, 0xadf8}, {0xe288a0, 0xa2dc},
- {0xe288a5, 0xa1c2}, {0xe288a7, 0xa2ca}, {0xe288a8, 0xa2cb}, {0xe288a9, 0xa2c1}, {0xe288aa, 0xa2c0},
{0xe288ab,0xa2e9}, {0xe288ac, 0xa2ea},
- {0xe288ae, 0xadf3}, {0xe288b4, 0xa1e8}, {0xe288b5, 0xa2e8}, {0xe288bd, 0xa2e6},
--- 420,436 ---- {0xe28887, 0xa2e0}, {0xe28888, 0xa2ba}, {0xe2888b, 0xa2bb},
! {0xe28892, 0xa1dd}, {0xe2889a, 0xa2e5}, {0xe2889d, 0xa2e7}, {0xe2889e, 0xa1e7}, {0xe288a0, 0xa2dc},
{0xe288a7,0xa2ca}, {0xe288a8, 0xa2cb}, {0xe288a9, 0xa2c1}, {0xe288aa, 0xa2c0}, {0xe288ab, 0xa2e9}, {0xe288ac,
0xa2ea}, {0xe288b4, 0xa1e8}, {0xe288b5, 0xa2e8}, {0xe288bd, 0xa2e6},
***************
*** 464,491 **** {0xe28a86, 0xa2bc}, {0xe28a87, 0xa2bd}, {0xe28aa5, 0xa2dd},
- {0xe28abf, 0xadf9}, {0xe28c92, 0xa2de},
- {0xe291a0, 0xada1},
- {0xe291a1, 0xada2},
- {0xe291a2, 0xada3},
- {0xe291a3, 0xada4},
- {0xe291a4, 0xada5},
- {0xe291a5, 0xada6},
- {0xe291a6, 0xada7},
- {0xe291a7, 0xada8},
- {0xe291a8, 0xada9},
- {0xe291a9, 0xadaa},
- {0xe291aa, 0xadab},
- {0xe291ab, 0xadac},
- {0xe291ac, 0xadad},
- {0xe291ad, 0xadae},
- {0xe291ae, 0xadaf},
- {0xe291af, 0xadb0},
- {0xe291b0, 0xadb1},
- {0xe291b1, 0xadb2},
- {0xe291b2, 0xadb3},
- {0xe291b3, 0xadb4}, {0xe29480, 0xa8a1}, {0xe29481, 0xa8ac}, {0xe29482, 0xa8a2},
--- 446,452 ----
***************
*** 558,565 **** {0xe38093, 0xa2ae}, {0xe38094, 0xa1cc}, {0xe38095, 0xa1cd},
! {0xe3809d, 0xade0},
! {0xe3809f, 0xade1}, {0xe38181, 0xa4a1}, {0xe38182, 0xa4a2}, {0xe38183, 0xa4a3},
--- 519,525 ---- {0xe38093, 0xa2ae}, {0xe38094, 0xa1cc}, {0xe38095, 0xa1cd},
! {0xe3809c, 0xa1c1}, {0xe38181, 0xa4a1}, {0xe38182, 0xa4a2}, {0xe38183, 0xa4a3},
***************
*** 737,778 **** {0xe383bc, 0xa1bc}, {0xe383bd, 0xa1b3}, {0xe383be, 0xa1b4},
- {0xe388b1, 0xadea},
- {0xe388b2, 0xadeb},
- {0xe388b9, 0xadec},
- {0xe38aa4, 0xade5},
- {0xe38aa5, 0xade6},
- {0xe38aa6, 0xade7},
- {0xe38aa7, 0xade8},
- {0xe38aa8, 0xade9},
- {0xe38c83, 0xadc6},
- {0xe38c8d, 0xadca},
- {0xe38c94, 0xadc1},
- {0xe38c98, 0xadc4},
- {0xe38ca2, 0xadc2},
- {0xe38ca3, 0xadcc},
- {0xe38ca6, 0xadcb},
- {0xe38ca7, 0xadc5},
- {0xe38cab, 0xadcd},
- {0xe38cb6, 0xadc7},
- {0xe38cbb, 0xadcf},
- {0xe38d89, 0xadc0},
- {0xe38d8a, 0xadce},
- {0xe38d8d, 0xadc3},
- {0xe38d91, 0xadc8},
- {0xe38d97, 0xadc9},
- {0xe38dbb, 0xaddf},
- {0xe38dbc, 0xadef},
- {0xe38dbd, 0xadee},
- {0xe38dbe, 0xaded},
- {0xe38e8e, 0xadd3},
- {0xe38e8f, 0xadd4},
- {0xe38e9c, 0xadd0},
- {0xe38e9d, 0xadd1},
- {0xe38e9e, 0xadd2},
- {0xe38ea1, 0xadd6},
- {0xe38f84, 0xadd5},
- {0xe38f8d, 0xade3}, {0xe4b880, 0xb0ec}, {0xe4b881, 0xc3fa}, {0xe4b882, 0x8fb0a1},
--- 697,702 ----
***************
*** 934,940 **** {0xe4bbb6, 0xb7ef}, {0xe4bbb7, 0xd0c3}, {0xe4bbbb, 0xc7a4},
- {0xe4bbbc, 0x8ff4af}, {0xe4bbbd, 0x8fb0cf}, {0xe4bbbe, 0x8fb0d0}, {0xe4bbbf, 0x8fb0d1},
--- 858,863 ----
***************
*** 1281,1287 **** {0xe583af, 0x8fb2c4}, {0xe583b1, 0x8fb2c5}, {0xe583b2, 0x8fb2d1},
- {0xe583b4, 0x8ff4b0}, {0xe583b5, 0xd1ac}, {0xe583b6, 0x8fb2c6}, {0xe583b9, 0xd1ab},
--- 1204,1209 ----
***************
*** 1457,1463 **** {0xe587a7, 0xc2fc}, {0xe587a9, 0xd1de}, {0xe587aa, 0xc6e4},
- {0xe587ac, 0x8ff4b1}, {0xe587ad, 0xd1df}, {0xe587ae, 0x8fb3ab}, {0xe587b0, 0xd1e0},
--- 1379,1384 ----
***************
*** 1658,1664 **** {0xe58c83, 0x8fb3fc}, {0xe58c85, 0xcaf1}, {0xe58c86, 0xd2b2},
- {0xe58c87, 0x8ff4b2}, {0xe58c88, 0xd2b3}, {0xe58c8a, 0x8fb3fd}, {0xe58c8b, 0x8fb3fe},
--- 1579,1584 ----
***************
*** 1682,1688 **** {0xe58ca0, 0xbea2}, {0xe58ca1, 0xb6a9}, {0xe58ca3, 0xd2ba},
- {0xe58ca4, 0x8ff4b3}, {0xe58ca5, 0x8fb4a9}, {0xe58ca7, 0x8fb4aa}, {0xe58ca8, 0x8fb4ab},
--- 1602,1607 ----
***************
*** 1922,1928 **** {0xe59286, 0xd2f6}, {0xe59288, 0x8fb5a4}, {0xe59289, 0x8fb5a5},
- {0xe5928a, 0x8ff4b5}, {0xe5928b, 0xbaf0}, {0xe5928c, 0xcfc2}, {0xe5928d, 0x8fb5a6},
--- 1841,1846 ----
***************
*** 2334,2340 **** {0xe59d90, 0xbac1}, {0xe59d91, 0xb9a3}, {0xe59d92, 0x8fb7c0},
- {0xe59d99, 0x8ff4b6}, {0xe59da1, 0xd4b3}, {0xe59da2, 0x8fb7c1}, {0xe59da4, 0xbaa5},
--- 2252,2257 ----
***************
*** 2516,2522 **** {0xe5a297, 0xc1fd}, {0xe5a29c, 0xc4c6}, {0xe5a29d, 0x8fb8c7},
- {0xe5a29e, 0x8ff4b9}, {0xe5a29f, 0xd4d2}, {0xe5a2a0, 0x8fb8c8}, {0xe5a2a1, 0x8fb8c9},
--- 2433,2438 ----
***************
*** 3001,3007 **** {0xe5afa8, 0xdccd}, {0xe5afa9, 0xbfb3}, {0xe5afab, 0xd5ed},
- {0xe5afac, 0x8ff4ba}, {0xe5afae, 0xcec0}, {0xe5afaf, 0x8fbae5}, {0xe5afb0, 0xd5ee},
--- 2917,2922 ----
***************
*** 3154,3160 **** {0xe5b3b1, 0x8fbbbf}, {0xe5b3b2, 0x8fbbc0}, {0xe5b3b4, 0x8fbbc1},
- {0xe5b3b5, 0x8ff4bb}, {0xe5b3b6, 0xc5e7}, {0xe5b3ba, 0xd6b8}, {0xe5b3bb, 0xbdd4},
--- 3069,3074 ----
***************
*** 3199,3205 **** {0xe5b590, 0xcdf2}, {0xe5b591, 0x8fbbd5}, {0xe5b592, 0xd6c9},
- {0xe5b593, 0x8ff4bc}, {0xe5b595, 0x8fbbd4}, {0xe5b599, 0x8fbbd6}, {0xe5b59c, 0xd6bf},
--- 3113,3118 ----
***************
*** 3592,3598 **** {0xe5beb1, 0x8fbdb6}, {0xe5beb3, 0xc6c1}, {0xe5beb4, 0xc4a7},
- {0xe5beb7, 0x8ff4be}, {0xe5beb8, 0x8fbdb7}, {0xe5beb9, 0xc5b0}, {0xe5bebc, 0xd7d5},
--- 3505,3510 ----
***************
*** 3724,3730 **** {0xe68282, 0x8fbded}, {0xe68283, 0xd7fd}, {0xe68284, 0xd8a1},
- {0xe68285, 0x8ff4bf}, {0xe68286, 0x8fbdee}, {0xe68288, 0x8fbdef}, {0xe68289, 0xbcbd},
--- 3636,3641 ----
***************
*** 3840,3846 **** {0xe6849c, 0x8fbec4}, {0xe6849e, 0x8fbec5}, {0xe6849f, 0xb4b6},
- {0xe684a0, 0x8ff4c0}, {0xe684a1, 0xd8bb}, {0xe684a2, 0x8fbec6}, {0xe684a7, 0xd8c3},
--- 3751,3756 ----
***************
*** 4494,4500 **** {0xe69588, 0xdac3}, {0xe69589, 0x8fc1cc}, {0xe6958d, 0xdac6},
- {0xe6958e, 0x8ff4c1}, {0xe6958f, 0xc9d2}, {0xe69590, 0x8fc1cd}, {0xe69591, 0xb5df},
--- 4404,4409 ----
***************
*** 4653,4659 **** {0xe698b6, 0xdae4}, {0xe698b7, 0x8fc2b0}, {0xe698b9, 0x8fc2af},
- {0xe698bb, 0x8ff4c2}, {0xe698bc, 0xc3eb}, {0xe698bf, 0xdba6}, {0xe69980, 0x8fc2b1},
--- 4562,4567 ----
***************
*** 4683,4689 **** {0xe699a1, 0x8fc2be}, {0xe699a2, 0xdaf1}, {0xe699a4, 0xdaed},
- {0xe699a5, 0x8ff4c3}, {0xe699a6, 0xb3a2}, {0xe699a7, 0xdaee}, {0xe699a8, 0xdaef},
--- 4591,4596 ----
***************
*** 4956,4962 **** {0xe69fbb, 0xbaba}, {0xe69fbe, 0xcbef}, {0xe69fbf, 0xb3c1},
- {0xe6a081, 0x8ff4c6}, {0xe6a082, 0xc4ce}, {0xe6a083, 0xc6ca}, {0xe6a084, 0xb1c9},
--- 4863,4868 ----
***************
*** 5344,5350 **** {0xe6a9a6, 0xdcf5}, {0xe6a9a7, 0x8fc5b2}, {0xe6a9aa, 0x8fc5b3},
- {0xe6a9ab, 0x8ff4c9}, {0xe6a9b1, 0x8fc5b4}, {0xe6a9b2, 0xdcef}, {0xe6a9b3, 0x8fc5b5},
--- 5250,5255 ----
***************
*** 5406,5412 **** {0xe6ab9d, 0x8fc5d4}, {0xe6ab9e, 0xdda9}, {0xe6ab9f, 0xddab},
- {0xe6aba2, 0x8ff4ca}, {0xe6aba4, 0x8fc5d5}, {0xe6aba7, 0x8fc5d6}, {0xe6aba8, 0xc8a7},
--- 5311,5316 ----
***************
*** 5854,5866 **** {0xe6b7af, 0x8fc7e2}, {0xe6b7b0, 0x8fc7e3}, {0xe6b7b1, 0xbfbc},
- {0xe6b7b2, 0x8ff4cc}, {0xe6b7b3, 0xbddf}, {0xe6b7b4, 0x8fc7e4}, {0xe6b7b5, 0xcaa5}, {0xe6b7b6, 0x8fc7e5},
{0xe6b7b7,0xbaae},
- {0xe6b7b8, 0x8ff4cb}, {0xe6b7b9, 0xdebb}, {0xe6b7ba, 0xdec9}, {0xe6b7bb, 0xc5ba},
--- 5758,5768 ----
***************
*** 6170,6176 **** {0xe780a3, 0x8fc9b1}, {0xe780a6, 0xc3f5}, {0xe780a7, 0xc2ed},
- {0xe780a8, 0x8ff4cd}, {0xe780ac, 0xc0a5}, {0xe780af, 0x8fc9b2}, {0xe780b0, 0xdfd0},
--- 6072,6077 ----
***************
*** 6236,6242 **** {0xe782b8, 0xdfda}, {0xe782b9, 0xc5c0}, {0xe782ba, 0xb0d9},
- {0xe782bb, 0x8ff4ae}, {0xe78388, 0xcef5}, {0xe7838a, 0x8fc9d8}, {0xe7838b, 0xdfde},
--- 6137,6142 ----
***************
*** 6810,6816 **** {0xe793bb, 0x8fccfb}, {0xe793bc, 0x8fccfc}, {0xe793bf, 0x8fccfd},
- {0xe79481, 0x8ff4d0}, {0xe79483, 0xe1ac}, {0xe79484, 0xe1ab}, {0xe79485, 0xe1ad},
--- 6710,6715 ----
***************
*** 7075,7081 **** {0xe799be, 0xc9b4}, {0xe79a80, 0xe2a5}, {0xe79a81, 0x8fceb5},
- {0xe79a82, 0x8ff4d1}, {0xe79a83, 0xe2a6}, {0xe79a84, 0xc5aa}, {0xe79a85, 0x8fceb6},
--- 6974,6979 ----
***************
*** 7095,7101 **** {0xe79a9b, 0x8fceba}, {0xe79a9c, 0x8fcebb}, {0xe79a9d, 0x8fcebc},
- {0xe79a9e, 0x8ff4d2}, {0xe79a9f, 0x8fcebd}, {0xe79aa0, 0x8fcebe}, {0xe79aa2, 0x8fcebf},
--- 6993,6998 ----
***************
*** 7458,7464 **** {0xe7a4ab, 0xe3aa}, {0xe7a4ac, 0xe3a9}, {0xe7a4ad, 0x8fd0c7},
- {0xe7a4b0, 0x8ff4d4}, {0xe7a4b1, 0x8fd0c8}, {0xe7a4b4, 0x8fd0c9}, {0xe7a4b5, 0x8fd0ca},
--- 7355,7360 ----
***************
*** 7736,7742 **** {0xe7aba3, 0xbdd7}, {0xe7aba5, 0xc6b8}, {0xe7aba6, 0xe3f0},
- {0xe7aba7, 0x8ff4d9}, {0xe7aba8, 0x8fd1ea}, {0xe7aba9, 0x8fd1eb}, {0xe7abaa, 0xc3a8},
--- 7632,7637 ----
***************
*** 8191,8197 **** {0xe7b69d, 0x8fd3fe}, {0xe7b69e, 0x8fd4a1}, {0xe7b69f, 0xe5c6},
- {0xe7b6a0, 0x8ff4dc}, {0xe7b6a2, 0xe5c2}, {0xe7b6a3, 0xe5bc}, {0xe7b6a6, 0x8fd4a2},
--- 8086,8091 ----
***************
*** 8232,8238 **** {0xe7b791, 0xced0}, {0xe7b792, 0xbdef}, {0xe7b795, 0xe5ee},
- {0xe7b796, 0x8ff4dd}, {0xe7b797, 0x8fd4b1}, {0xe7b798, 0xe5c8}, {0xe7b799, 0x8fd4b2},
--- 8126,8131 ----
***************
*** 9034,9040 **** {0xe88d96, 0x8fd7fb}, {0xe88d97, 0x8fd7fc}, {0xe88d98, 0xc1f1},
- {0xe88da2, 0x8ff4df}, {0xe88db0, 0x8fd7fd}, {0xe88db3, 0xe8ba}, {0xe88db5, 0xe8bb},
--- 8927,8932 ----
***************
*** 9372,9378 **** {0xe896ad, 0x8fd9f4}, {0xe896ae, 0xccf9}, {0xe896af, 0xbdf2},
- {0xe896b0, 0x8ff4e1}, {0xe896b4, 0x8fd9e7}, {0xe896b6, 0x8fd9e8}, {0xe896b7, 0x8fd9e9},
--- 9264,9269 ----
***************
*** 9711,9717 **** {0xe8a083, 0x8fdbe9}, {0xe8a085, 0xeaa4}, {0xe8a086, 0x8fdbea},
- {0xe8a087, 0x8ff4e4}, {0xe8a089, 0x8fdbeb}, {0xe8a08a, 0x8fdbec}, {0xe8a08b, 0x8fdbed},
--- 9602,9607 ----
***************
*** 10276,10282 **** {0xe8adbc, 0x8fded5}, {0xe8adbd, 0xeca5}, {0xe8adbe, 0x8fded6},
- {0xe8adbf, 0x8ff4e6}, {0xe8ae80, 0xeca6}, {0xe8ae81, 0x8fded7}, {0xe8ae83, 0xbbbe},
--- 10166,10171 ----
***************
*** 10432,10438 **** {0xe8b3af, 0x8fdfb8}, {0xe8b3b0, 0x8fdfb9}, {0xe8b3b2, 0x8fdfba},
- {0xe8b3b4, 0x8ff4e7}, {0xe8b3b5, 0x8fdfbb}, {0xe8b3b7, 0x8fdfbc}, {0xe8b3b8, 0x8fdfbd},
--- 10321,10326 ----
***************
*** 10473,10479 **** {0xe8b5b1, 0xece1}, {0xe8b5b3, 0xece2}, {0xe8b5b4, 0xc9eb},
- {0xe8b5b6, 0x8ff4e8}, {0xe8b5b7, 0xb5af}, {0xe8b5bf, 0x8fdfca}, {0xe8b681, 0xece3},
--- 10361,10366 ----
***************
*** 10997,11003 **** {0xe9839b, 0xeebe}, {0xe9839c, 0x8fe2a9}, {0xe9839d, 0x8fe2aa},
- {0xe9839e, 0x8ff4ec}, {0xe9839f, 0x8fe2ab}, {0xe983a1, 0xb7b4}, {0xe983a2, 0xeebb},
--- 10884,10889 ----
***************
*** 11025,11031 **** {0xe98490, 0x8fe2bb}, {0xe98492, 0xeec0}, {0xe98494, 0x8fe2bc},
- {0xe98495, 0x8ff4ee}, {0xe98496, 0x8fe2bd}, {0xe98497, 0x8fe2be}, {0xe98498, 0x8fe2bf},
--- 10911,10916 ----
***************
*** 11599,11605 **** {0xe9968e, 0x8fe6a6}, {0xe9968f, 0xb1bc}, {0xe99691, 0xb4d7},
- {0xe99692, 0x8ff4f1}, {0xe99693, 0xb4d6}, {0xe99694, 0xefdc}, {0xe99696, 0xefdd},
--- 11484,11489 ----
***************
*** 11848,11854 **** {0xe99cb3, 0x8fe7ac}, {0xe99cb8, 0xdbb1}, {0xe99cb9, 0xf0c8},
- {0xe99cbb, 0x8ff4f4}, {0xe99cbd, 0xf0c9}, {0xe99cbe, 0xf0ca}, {0xe99d81, 0x8fe7ad},
--- 11732,11737 ----
***************
*** 11859,11868 **** {0xe99d88, 0xf0cd}, {0xe99d89, 0xf0cf}, {0xe99d8a, 0x8fe7af},
- {0xe99d8d, 0x8ff4f5}, {0xe99d8e, 0x8fe7b0}, {0xe99d8f, 0x8fe7b1},
- {0xe99d91, 0x8ff4f6}, {0xe99d92, 0xc0c4}, {0xe99d95, 0x8fe7b2}, {0xe99d96, 0xccf7},
--- 11742,11749 ----
***************
*** 12178,12184 **** {0xe9a699, 0xb9e1}, {0xe9a69b, 0x8fe8f7}, {0xe9a69d, 0x8fe8f8},
- {0xe9a69e, 0x8ff4fa}, {0xe9a69f, 0x8fe8f9}, {0xe9a6a5, 0xf1c6}, {0xe9a6a6, 0x8fe8fa},
--- 12059,12064 ----
***************
*** 12329,12335 **** {0xe9ab96, 0x8fe9ea}, {0xe9ab97, 0x8fe9eb}, {0xe9ab98, 0xb9e2},
- {0xe9ab99, 0x8ff4fb}, {0xe9ab9b, 0x8fe9ec}, {0xe9ab9c, 0x8fe9ed}, {0xe9ab9e, 0xf1f4},
--- 12209,12214 ----
***************
*** 12436,12442 **** {0xe9adad, 0x8feaca}, {0xe9adae, 0x8feacb}, {0xe9adaf, 0xcfa5},
- {0xe9adb2, 0x8ff4fc}, {0xe9adb3, 0x8feacc}, {0xe9adb4, 0xf2b7}, {0xe9adb5, 0x8feacd},
--- 12315,12320 ----
***************
*** 12846,12852 **** {0xe9bb8e, 0xf3d5}, {0xe9bb8f, 0xf3d6}, {0xe9bb90, 0xf3d7},
- {0xe9bb91, 0x8ff4fe}, {0xe9bb92, 0xb9f5}, {0xe9bb94, 0xf3d8}, {0xe9bb95, 0x8fecf4},
--- 12724,12729 ----
***************
*** 12977,13029 **** {0xe9bea2, 0x8fede1}, {0xe9bea3, 0x8fede2}, {0xe9bea5, 0x8fede3},
- {0xefa4a9, 0x8ff4c5},
- {0xefa79c, 0x8ff4f2},
- {0xefa88e, 0x8ff4b4},
- {0xefa88f, 0x8ff4b7},
- {0xefa890, 0x8ff4b8},
- {0xefa891, 0x8ff4bd},
- {0xefa892, 0x8ff4c4},
- {0xefa893, 0x8ff4c7},
- {0xefa894, 0x8ff4c8},
- {0xefa895, 0x8ff4ce},
- {0xefa896, 0x8ff4cf},
- {0xefa897, 0x8ff4d3},
- {0xefa898, 0x8ff4d5},
- {0xefa899, 0x8ff4d6},
- {0xefa89a, 0x8ff4d7},
- {0xefa89b, 0x8ff4d8},
- {0xefa89c, 0x8ff4da},
- {0xefa89d, 0x8ff4db},
- {0xefa89e, 0x8ff4de},
- {0xefa89f, 0x8ff4e0},
- {0xefa8a0, 0x8ff4e2},
- {0xefa8a1, 0x8ff4e3},
- {0xefa8a2, 0x8ff4e5},
- {0xefa8a3, 0x8ff4e9},
- {0xefa8a4, 0x8ff4ea},
- {0xefa8a5, 0x8ff4eb},
- {0xefa8a6, 0x8ff4ed},
- {0xefa8a7, 0x8ff4ef},
- {0xefa8a8, 0x8ff4f0},
- {0xefa8a9, 0x8ff4f3},
- {0xefa8aa, 0x8ff4f7},
- {0xefa8ab, 0x8ff4f8},
- {0xefa8ac, 0x8ff4f9},
- {0xefa8ad, 0x8ff4fd}, {0xefbc81, 0xa1aa},
- {0xefbc82, 0x8ff4aa}, {0xefbc83, 0xa1f4}, {0xefbc84, 0xa1f0}, {0xefbc85, 0xa1f3}, {0xefbc86, 0xa1f5},
- {0xefbc87, 0x8ff4a9}, {0xefbc88, 0xa1ca}, {0xefbc89, 0xa1cb}, {0xefbc8a, 0xa1f6}, {0xefbc8b, 0xa1dc},
{0xefbc8c,0xa1a4},
- {0xefbc8d, 0xa1dd}, {0xefbc8e, 0xa1a5}, {0xefbc8f, 0xa1bf}, {0xefbc90, 0xa3b0},
--- 12854,12869 ----
***************
*** 13070,13076 **** {0xefbcb9, 0xa3d9}, {0xefbcba, 0xa3da}, {0xefbcbb, 0xa1ce},
- {0xefbcbc, 0xa1c0}, {0xefbcbd, 0xa1cf}, {0xefbcbe, 0xa1b0}, {0xefbcbf, 0xa1b2},
--- 12910,12915 ----
***************
*** 13104,13110 **** {0xefbd9b, 0xa1d0}, {0xefbd9c, 0xa1c3}, {0xefbd9d, 0xa1d1},
- {0xefbd9e, 0xa1c1}, {0xefbda1, 0x8ea1}, {0xefbda2, 0x8ea2}, {0xefbda3, 0x8ea3},
--- 12943,12948 ----
***************
*** 13168,13177 **** {0xefbe9d, 0x8edd}, {0xefbe9e, 0x8ede}, {0xefbe9f, 0x8edf},
- {0xefbfa0, 0xa1f1},
- {0xefbfa1, 0xa1f2},
- {0xefbfa2, 0xa2cc}, {0xefbfa3, 0xa1b1},
- {0xefbfa4, 0x8fa2c3}, {0xefbfa5, 0xa1ef} };
--- 13006,13011 ----
*** utf8_to_euc_kr.map.orig Fri May 15 17:56:06 2015
--- utf8_to_euc_kr.map Fri Nov 27 16:25:41 2015
***************
*** 1,11 ****
! static const pg_utf_to_local ULmapEUC_KR[ 8227 ] = { {0xc2a1, 0xa2ae}, {0xc2a4, 0xa2b4}, {0xc2a7, 0xa1d7},
{0xc2a8,0xa1a7}, {0xc2aa, 0xa8a3}, {0xc2ad, 0xa1a9},
- {0xc2ae, 0xa2e7}, {0xc2b0, 0xa1c6}, {0xc2b1, 0xa1be}, {0xc2b2, 0xa9f7},
--- 1,12 ----
! /* src/backend/utils/mb/Unicode/utf8_to_euc_kr.map */
!
! static const pg_utf_to_local ULmapEUC_KR[ 8224 ] = { {0xc2a1, 0xa2ae}, {0xc2a4, 0xa2b4}, {0xc2a7, 0xa1d7},
{0xc2a8,0xa1a7}, {0xc2aa, 0xa8a3}, {0xc2ad, 0xa1a9}, {0xc2b0, 0xa1c6}, {0xc2b1, 0xa1be}, {0xc2b2, 0xa9f7},
***************
*** 189,195 **** {0xe28282, 0xa9fc}, {0xe28283, 0xa9fd}, {0xe28284, 0xa9fe},
- {0xe282ac, 0xa2e6}, {0xe28483, 0xa1c9}, {0xe28489, 0xa2b5}, {0xe28493, 0xa7a4},
--- 190,195 ----
***************
*** 806,812 **** {0xe389b9, 0xa8ca}, {0xe389ba, 0xa8cb}, {0xe389bb, 0xa8cc},
- {0xe389be, 0xa2e8}, {0xe389bf, 0xa2de}, {0xe38e80, 0xa7c9}, {0xe38e81, 0xa7ca},
--- 806,811 ----
*** utf8_to_johab.map.orig Fri May 15 17:56:06 2015
--- utf8_to_johab.map Fri Nov 27 16:26:10 2015
***************
*** 1,11 ****
! static const pg_utf_to_local ULmapJOHAB[ 17049 ] = { {0xc2a1, 0xd9ae}, {0xc2a4, 0xd9b4}, {0xc2a7, 0xd967},
{0xc2a8,0xd937}, {0xc2aa, 0xdca3}, {0xc2ad, 0xd939},
- {0xc2ae, 0xd9e7}, {0xc2b0, 0xd956}, {0xc2b1, 0xd94e}, {0xc2b2, 0xdd99},
--- 1,12 ----
! /* src/backend/utils/mb/Unicode/utf8_to_johab.map */
!
! static const pg_utf_to_local ULmapJOHAB[ 17046 ] = { {0xc2a1, 0xd9ae}, {0xc2a4, 0xd9b4}, {0xc2a7, 0xd967},
{0xc2a8,0xd937}, {0xc2aa, 0xdca3}, {0xc2ad, 0xd939}, {0xc2b0, 0xd956}, {0xc2b1, 0xd94e}, {0xc2b2, 0xdd99},
***************
*** 189,195 **** {0xe28282, 0xdd9e}, {0xe28283, 0xdd9f}, {0xe28284, 0xdda0},
- {0xe282ac, 0xd9e6}, {0xe28483, 0xd959}, {0xe28489, 0xd9b5}, {0xe28493, 0xdc34},
--- 190,195 ----
***************
*** 806,812 **** {0xe389b9, 0xdcca}, {0xe389ba, 0xdccb}, {0xe389bb, 0xdccc},
- {0xe389be, 0xd9e8}, {0xe389bf, 0xd9de}, {0xe38e80, 0xdc59}, {0xe38e81, 0xdc5a},
--- 806,811 ----
*** utf8_to_sjis.map.orig Fri May 15 17:56:06 2015
--- utf8_to_sjis.map Fri Nov 27 16:25:56 2015
***************
*** 1,11 ****
! static const pg_utf_to_local ULmapSJIS[ 7398 ] = {
! {0xc19c, 0x815f},
! {0xc2a2, 0x8191},
! {0xc2a3, 0x8192},
! {0xc2a5, 0x5c}, {0xc2a7, 0x8198}, {0xc2a8, 0x814e},
- {0xc2ac, 0x81ca}, {0xc2b0, 0x818b}, {0xc2b1, 0x817d}, {0xc2b4, 0x814c},
--- 1,8 ----
! /* src/backend/utils/mb/Unicode/utf8_to_sjis.map */
!
! static const pg_utf_to_local ULmapSJIS[ 7389 ] = { {0xc2a7, 0x8198}, {0xc2a8, 0x814e}, {0xc2b0, 0x818b},
{0xc2b1,0x817d}, {0xc2b4, 0x814c},
***************
*** 128,134 **** {0xd191, 0x8476}, {0xe28090, 0x815d}, {0xe28095, 0x815c},
- {0xe28096, 0x8161}, {0xe28098, 0x8165}, {0xe28099, 0x8166}, {0xe2809c, 0x8167},
--- 125,130 ----
***************
*** 141,147 **** {0xe280b2, 0x818c}, {0xe280b3, 0x818d}, {0xe280bb, 0x81a6},
- {0xe280be, 0x7e}, {0xe28483, 0x818e}, {0xe28496, 0xfa59}, {0xe284a1, 0xfa5a},
--- 137,142 ----
***************
*** 179,185 **** {0xe28888, 0x81b8}, {0xe2888b, 0x81b9}, {0xe28891, 0x8794},
- {0xe28892, 0x817c}, {0xe2889a, 0x81e3}, {0xe2889d, 0x81e5}, {0xe2889e, 0x8187},
--- 174,179 ----
***************
*** 302,308 **** {0xe38093, 0x81ac}, {0xe38094, 0x816b}, {0xe38095, 0x816c},
- {0xe3809c, 0x8160}, {0xe3809d, 0x8780}, {0xe3809f, 0x8781}, {0xe38181, 0x829f},
--- 296,301 ----
*** utf8_to_uhc.map.orig Fri May 15 17:56:06 2015
--- utf8_to_uhc.map Fri Nov 27 16:26:11 2015
***************
*** 1,4 ****
! static const pg_utf_to_local ULmapUHC[ 17237 ] = { {0xc2a1, 0xa2ae}, {0xc2a4, 0xa2b4}, {0xc2a7, 0xa1d7},
--- 1,6 ----
! /* src/backend/utils/mb/Unicode/utf8_to_uhc.map */
!
! static const pg_utf_to_local ULmapUHC[ 17048 ] = { {0xc2a1, 0xa2ae}, {0xc2a4, 0xa2b4}, {0xc2a7, 0xa1d7},
***************
*** 806,812 **** {0xe389b9, 0xa8ca}, {0xe389ba, 0xa8cb}, {0xe389bb, 0xa8cc},
- {0xe389be, 0xa2e8}, {0xe389bf, 0xa2de}, {0xe38e80, 0xa7c9}, {0xe38e81, 0xa7ca},
--- 808,813 ----
***************
*** 16680,16873 **** {0xed9ea1, 0xc650}, {0xed9ea2, 0xc651}, {0xed9ea3, 0xc652},
- {0xee8080, 0xc9a1},
- {0xee8081, 0xc9a2},
- {0xee8082, 0xc9a3},
- {0xee8083, 0xc9a4},
- {0xee8084, 0xc9a5},
- {0xee8085, 0xc9a6},
- {0xee8086, 0xc9a7},
- {0xee8087, 0xc9a8},
- {0xee8088, 0xc9a9},
- {0xee8089, 0xc9aa},
- {0xee808a, 0xc9ab},
- {0xee808b, 0xc9ac},
- {0xee808c, 0xc9ad},
- {0xee808d, 0xc9ae},
- {0xee808e, 0xc9af},
- {0xee808f, 0xc9b0},
- {0xee8090, 0xc9b1},
- {0xee8091, 0xc9b2},
- {0xee8092, 0xc9b3},
- {0xee8093, 0xc9b4},
- {0xee8094, 0xc9b5},
- {0xee8095, 0xc9b6},
- {0xee8096, 0xc9b7},
- {0xee8097, 0xc9b8},
- {0xee8098, 0xc9b9},
- {0xee8099, 0xc9ba},
- {0xee809a, 0xc9bb},
- {0xee809b, 0xc9bc},
- {0xee809c, 0xc9bd},
- {0xee809d, 0xc9be},
- {0xee809e, 0xc9bf},
- {0xee809f, 0xc9c0},
- {0xee80a0, 0xc9c1},
- {0xee80a1, 0xc9c2},
- {0xee80a2, 0xc9c3},
- {0xee80a3, 0xc9c4},
- {0xee80a4, 0xc9c5},
- {0xee80a5, 0xc9c6},
- {0xee80a6, 0xc9c7},
- {0xee80a7, 0xc9c8},
- {0xee80a8, 0xc9c9},
- {0xee80a9, 0xc9ca},
- {0xee80aa, 0xc9cb},
- {0xee80ab, 0xc9cc},
- {0xee80ac, 0xc9cd},
- {0xee80ad, 0xc9ce},
- {0xee80ae, 0xc9cf},
- {0xee80af, 0xc9d0},
- {0xee80b0, 0xc9d1},
- {0xee80b1, 0xc9d2},
- {0xee80b2, 0xc9d3},
- {0xee80b3, 0xc9d4},
- {0xee80b4, 0xc9d5},
- {0xee80b5, 0xc9d6},
- {0xee80b6, 0xc9d7},
- {0xee80b7, 0xc9d8},
- {0xee80b8, 0xc9d9},
- {0xee80b9, 0xc9da},
- {0xee80ba, 0xc9db},
- {0xee80bb, 0xc9dc},
- {0xee80bc, 0xc9dd},
- {0xee80bd, 0xc9de},
- {0xee80be, 0xc9df},
- {0xee80bf, 0xc9e0},
- {0xee8180, 0xc9e1},
- {0xee8181, 0xc9e2},
- {0xee8182, 0xc9e3},
- {0xee8183, 0xc9e4},
- {0xee8184, 0xc9e5},
- {0xee8185, 0xc9e6},
- {0xee8186, 0xc9e7},
- {0xee8187, 0xc9e8},
- {0xee8188, 0xc9e9},
- {0xee8189, 0xc9ea},
- {0xee818a, 0xc9eb},
- {0xee818b, 0xc9ec},
- {0xee818c, 0xc9ed},
- {0xee818d, 0xc9ee},
- {0xee818e, 0xc9ef},
- {0xee818f, 0xc9f0},
- {0xee8190, 0xc9f1},
- {0xee8191, 0xc9f2},
- {0xee8192, 0xc9f3},
- {0xee8193, 0xc9f4},
- {0xee8194, 0xc9f5},
- {0xee8195, 0xc9f6},
- {0xee8196, 0xc9f7},
- {0xee8197, 0xc9f8},
- {0xee8198, 0xc9f9},
- {0xee8199, 0xc9fa},
- {0xee819a, 0xc9fb},
- {0xee819b, 0xc9fc},
- {0xee819c, 0xc9fd},
- {0xee819d, 0xc9fe},
- {0xee819e, 0xfea1},
- {0xee819f, 0xfea2},
- {0xee81a0, 0xfea3},
- {0xee81a1, 0xfea4},
- {0xee81a2, 0xfea5},
- {0xee81a3, 0xfea6},
- {0xee81a4, 0xfea7},
- {0xee81a5, 0xfea8},
- {0xee81a6, 0xfea9},
- {0xee81a7, 0xfeaa},
- {0xee81a8, 0xfeab},
- {0xee81a9, 0xfeac},
- {0xee81aa, 0xfead},
- {0xee81ab, 0xfeae},
- {0xee81ac, 0xfeaf},
- {0xee81ad, 0xfeb0},
- {0xee81ae, 0xfeb1},
- {0xee81af, 0xfeb2},
- {0xee81b0, 0xfeb3},
- {0xee81b1, 0xfeb4},
- {0xee81b2, 0xfeb5},
- {0xee81b3, 0xfeb6},
- {0xee81b4, 0xfeb7},
- {0xee81b5, 0xfeb8},
- {0xee81b6, 0xfeb9},
- {0xee81b7, 0xfeba},
- {0xee81b8, 0xfebb},
- {0xee81b9, 0xfebc},
- {0xee81ba, 0xfebd},
- {0xee81bb, 0xfebe},
- {0xee81bc, 0xfebf},
- {0xee81bd, 0xfec0},
- {0xee81be, 0xfec1},
- {0xee81bf, 0xfec2},
- {0xee8280, 0xfec3},
- {0xee8281, 0xfec4},
- {0xee8282, 0xfec5},
- {0xee8283, 0xfec6},
- {0xee8284, 0xfec7},
- {0xee8285, 0xfec8},
- {0xee8286, 0xfec9},
- {0xee8287, 0xfeca},
- {0xee8288, 0xfecb},
- {0xee8289, 0xfecc},
- {0xee828a, 0xfecd},
- {0xee828b, 0xfece},
- {0xee828c, 0xfecf},
- {0xee828d, 0xfed0},
- {0xee828e, 0xfed1},
- {0xee828f, 0xfed2},
- {0xee8290, 0xfed3},
- {0xee8291, 0xfed4},
- {0xee8292, 0xfed5},
- {0xee8293, 0xfed6},
- {0xee8294, 0xfed7},
- {0xee8295, 0xfed8},
- {0xee8296, 0xfed9},
- {0xee8297, 0xfeda},
- {0xee8298, 0xfedb},
- {0xee8299, 0xfedc},
- {0xee829a, 0xfedd},
- {0xee829b, 0xfede},
- {0xee829c, 0xfedf},
- {0xee829d, 0xfee0},
- {0xee829e, 0xfee1},
- {0xee829f, 0xfee2},
- {0xee82a0, 0xfee3},
- {0xee82a1, 0xfee4},
- {0xee82a2, 0xfee5},
- {0xee82a3, 0xfee6},
- {0xee82a4, 0xfee7},
- {0xee82a5, 0xfee8},
- {0xee82a6, 0xfee9},
- {0xee82a7, 0xfeea},
- {0xee82a8, 0xfeeb},
- {0xee82a9, 0xfeec},
- {0xee82aa, 0xfeed},
- {0xee82ab, 0xfeee},
- {0xee82ac, 0xfeef},
- {0xee82ad, 0xfef0},
- {0xee82ae, 0xfef1},
- {0xee82af, 0xfef2},
- {0xee82b0, 0xfef3},
- {0xee82b1, 0xfef4},
- {0xee82b2, 0xfef5},
- {0xee82b3, 0xfef6},
- {0xee82b4, 0xfef7},
- {0xee82b5, 0xfef8},
- {0xee82b6, 0xfef9},
- {0xee82b7, 0xfefa},
- {0xee82b8, 0xfefb},
- {0xee82b9, 0xfefc},
- {0xee82ba, 0xfefd},
- {0xee82bb, 0xfefe}, {0xefa480, 0xcbd0}, {0xefa481, 0xcbd6}, {0xefa482, 0xcbe7},
--- 16681,16686 ----
I wrote:
> gb18030_to_utf8.map utf8_to_gb18030.map
> Could not find the reference file gb-18030-2000.xml, whose origin is
> unstated anyway.
Ah, scratch that complaint; digging in our git history turned up the
origin of that file, so I double-checked it and then updated the script
with a comment. We're good on that conversion ... but the other ones
still have problems.
regards, tom lane
> I wrote: >> I have not attempted to reverify the files in utils/mb/Unicode against the >> original Unicode Consortium data, but maybe we ought to do that before >> taking any further steps here. > > I downloaded the mapping files from unicode.org and attempted to verify > that the Unicode/*.map files could be reproduced from the stated sources. > Most of them are okay, but I failed to verify these: > > euc_cn_to_utf8.map utf8_to_euc_cn.map > > Could not find the reference file GB2312.TXT; it is not at unicode.org > > gb18030_to_utf8.map utf8_to_gb18030.map > > Could not find the reference file gb-18030-2000.xml, whose origin is > unstated anyway. > > euc_jp_to_utf8.map utf8_to_euc_jp.map > euc_kr_to_utf8.map utf8_to_euc_kr.map > johab_to_utf8.map utf8_to_johab.map > uhc_to_utf8.map utf8_to_uhc.map > > These four all have minor to significant differences from what I got by > running the generation scripts. See attached diffs. > > utf8_to_sjis.map > > It's very disturbing that this fails to verify when its allegedly inverse > file does verify; > either the script is broken or somebody did sloppy > manual editing. Manual editing. I explain why the manual editing is necessary. One of the most famous problems with Unicode is "wave dash" (U+301C). According the Unicode consortium's Unicode/SJIS map, it corresponds to 0x8160 of Shift_JIS. Unfortunately this was a mistake in Unicode (the glyph of Shift_JIS and Unicode is slightly different - looks like to be rotated in 90 degrees of wave dash in vertical scripting. Probably they did not understand the Japanese vertical writing at that time). So later on the Unicode consortium decided to add another "wave dash" as U+FF5E which has a correct glyph of "wave dash". However since Unicode already decided that U+301C corresponds to 0x8160 of Shift_JIS, there's no Shift_JIS code corresponding to U+FF5E. Unlike Unicode's definition, Microsoft defines that 0x8160 (wave dash) corresponds to U+FF5E. This is widely used in Japan. So I decided to hire this for "wave dash". i.e. 0x8160 -> U+FF5E (sjis_to_utf8.map) U+301C -> 0x8160 (utf_to_sjis.map) U+FF5E -> 0x8160 (utf_to_sjis.map) Another problem is vendor extension. There are several standards for SJIS and EUC_JP in Japan. There is a standard "Shift_JIS" defined by Japanese Government (probably the Unicode consortium's map can be based on this, but I need to verify). However several major vendors include IBM, NEC added their own additional characters to Shift_JIS and they are widely used in Japan. Unfortunately they are not compatible. So as a compromise I and other developers decided to "merge" NEC and IBM extension part and added to Shift_JIS. Same thing can be said to EUC_JP. In short, there are number of reasons we cannot simply import the consortium's mapping regarding SJIS (and EUC_JP). > Anyway, this seems to mean that it's okay to go ahead with fixing the > encoding conversion discrepancies I complained of yesterday; the data > those proposed diffs are based on is solid. But we've evidently got > a number of other issues with these Far Eastern encodings. > > regards, tom lane
I wrote: > There's a discussion over at > http://www.postgresql.org/message-id/flat/2sa.Dhu5.1hk1yrpTNFy.1MLOlb@seznam.cz > of an apparent error in our WIN1250 -> LATIN2 conversion. Attached is an updated patch (against today's HEAD) showing proposed changes to bring cyrillic_and_mic.c and latin2_and_win1250.c into sync with the Unicode Consortium's conversion data. In addition, I've attached the C program I used to generate the proposed new conversion tables from the Unicode/*.map files, a simple SQL script to print out the conversion behavior for the affected conversions, and a diff of the script's output between 9.5 and the proposed patch. While the changes in the WIN1250 <-> LATIN2 conversions just amount to removal of some translations that seem to have no basis in reality, the changes in the Cyrillic mappings are quite a bit more extensive. It would be good if we could get those checked by some native Russian speakers. regards, tom lane diff --git a/src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c b/src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c index 1847287..539c3d3 100644 *** a/src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c --- b/src/backend/utils/mb/conversion_procs/cyrillic_and_mic/cyrillic_and_mic.c *************** static const unsigned char iso2koi[] = { *** 65,71 **** 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0xB3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xE1, 0xE2, 0xF7, 0xE7, 0xE4, 0xE5, 0xF6, 0xFA, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, 0xF0, --- 65,71 ---- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x9A, 0xB3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xE1, 0xE2, 0xF7, 0xE7, 0xE4, 0xE5, 0xF6, 0xFA, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, 0xF0, *************** static const unsigned char koi2iso[] = { *** 84,90 **** 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF1, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xA1, 0x00, 0x00, 0x00, 0x00, --- 84,90 ---- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0xA0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF1, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xA1, 0x00, 0x00, 0x00, 0x00, *************** static const unsigned char win12512koi[] *** 105,114 **** 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xBD, 0x00, 0x00, ! 0xB3, 0x00, 0xB4, 0x00, 0x00, 0x00, 0x00, 0xB7, ! 0x00, 0x00, 0xB6, 0xA6, 0xAD, 0x00, 0x00, 0x00, ! 0xA3, 0x00, 0xA4, 0x00, 0x00, 0x00, 0x00, 0xA7, 0xE1, 0xE2, 0xF7, 0xE7, 0xE4, 0xE5, 0xF6, 0xFA, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, 0xF0, 0xF2, 0xF3, 0xF4, 0xF5, 0xE6, 0xE8, 0xE3, 0xFE, --- 105,114 ---- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x9A, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0xB3, 0xBF, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x9C, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x9E, ! 0xA3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xE1, 0xE2, 0xF7, 0xE7, 0xE4, 0xE5, 0xF6, 0xFA, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, 0xF0, 0xF2, 0xF3, 0xF4, 0xF5, 0xE6, 0xE8, 0xE3, 0xFE, *************** static const unsigned char koi2win1251[] *** 124,134 **** 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0xB8, 0xBA, 0x00, 0xB3, 0xBF, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xB4, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0xA8, 0xAA, 0x00, 0xB2, 0xAF, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xA5, 0x00, 0x00, 0xFE, 0xE0, 0xE1, 0xF6, 0xE4, 0xE5, 0xF4, 0xE3, 0xF5, 0xE8, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, 0xFF, 0xF0, 0xF1, 0xF2, 0xF3, 0xE6, 0xE2, --- 124,134 ---- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0xA0, 0x00, 0xB0, 0x00, 0xB7, 0x00, + 0x00, 0x00, 0x00, 0xB8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0xA8, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xA9, 0xFE, 0xE0, 0xE1, 0xF6, 0xE4, 0xE5, 0xF4, 0xE3, 0xF5, 0xE8, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, 0xFF, 0xF0, 0xF1, 0xF2, 0xF3, 0xE6, 0xE2, *************** static const unsigned char win8662koi[] *** 147,174 **** 0xFB, 0xFD, 0xFF, 0xF9, 0xF8, 0xFC, 0xE0, 0xF1, 0xC1, 0xC2, 0xD7, 0xC7, 0xC4, 0xC5, 0xD6, 0xDA, 0xC9, 0xCA, 0xCB, 0xCC, 0xCD, 0xCE, 0xCF, 0xD0, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xBD, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xD2, 0xD3, 0xD4, 0xD5, 0xC6, 0xC8, 0xC3, 0xDE, 0xDB, 0xDD, 0xDF, 0xD9, 0xD8, 0xDC, 0xC0, 0xD1, ! 0xB3, 0xA3, 0xB4, 0xA4, 0xB7, 0xA7, 0x00, 0x00, ! 0xB6, 0xA6, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }; /* KOI8-R to WIN866 */ static const unsigned char koi2win866[] = { ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0xF1, 0xF3, 0x00, 0xF9, 0xF5, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xAD, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0xF0, 0xF2, 0x00, 0xF8, 0xF4, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xBD, 0x00, 0x00, 0xEE, 0xA0, 0xA1, 0xE6, 0xA4, 0xA5, 0xE4, 0xA3, 0xE5, 0xA8, 0xA9, 0xAA, 0xAB, 0xAC, 0xAD, 0xAE, 0xAF, 0xEF, 0xE0, 0xE1, 0xE2, 0xE3, 0xA6, 0xA2, --- 147,174 ---- 0xFB, 0xFD, 0xFF, 0xF9, 0xF8, 0xFC, 0xE0, 0xF1, 0xC1, 0xC2, 0xD7, 0xC7, 0xC4, 0xC5, 0xD6, 0xDA, 0xC9, 0xCA, 0xCB, 0xCC, 0xCD, 0xCE, 0xCF, 0xD0, ! 0x90, 0x91, 0x92, 0x81, 0x87, 0xB2, 0xB4, 0xA7, ! 0xA6, 0xB5, 0xA1, 0xA8, 0xAE, 0xAD, 0xAC, 0x83, ! 0x84, 0x89, 0x88, 0x86, 0x80, 0x8A, 0xAF, 0xB0, ! 0xAB, 0xA5, 0xBB, 0xB8, 0xB1, 0xA0, 0xBE, 0xB9, ! 0xBA, 0xB6, 0xB7, 0xAA, 0xA9, 0xA2, 0xA4, 0xBD, ! 0xBC, 0x85, 0x82, 0x8D, 0x8C, 0x8E, 0x8F, 0x8B, 0xD2, 0xD3, 0xD4, 0xD5, 0xC6, 0xC8, 0xC3, 0xDE, 0xDB, 0xDD, 0xDF, 0xD9, 0xD8, 0xDC, 0xC0, 0xD1, ! 0xB3, 0xA3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x9C, 0x95, 0x9E, 0x96, 0x00, 0x00, 0x94, 0x9A }; /* KOI8-R to WIN866 */ static const unsigned char koi2win866[] = { ! 0xC4, 0xB3, 0xDA, 0xBF, 0xC0, 0xD9, 0xC3, 0xB4, ! 0xC2, 0xC1, 0xC5, 0xDF, 0xDC, 0xDB, 0xDD, 0xDE, ! 0xB0, 0xB1, 0xB2, 0x00, 0xFE, 0xF9, 0xFB, 0x00, ! 0x00, 0x00, 0xFF, 0x00, 0xF8, 0x00, 0xFA, 0x00, ! 0xCD, 0xBA, 0xD5, 0xF1, 0xD6, 0xC9, 0xB8, 0xB7, ! 0xBB, 0xD4, 0xD3, 0xC8, 0xBE, 0xBD, 0xBC, 0xC6, ! 0xC7, 0xCC, 0xB5, 0xF0, 0xB6, 0xB9, 0xD1, 0xD2, ! 0xCB, 0xCF, 0xD0, 0xCA, 0xD8, 0xD7, 0xCE, 0x00, 0xEE, 0xA0, 0xA1, 0xE6, 0xA4, 0xA5, 0xE4, 0xA3, 0xE5, 0xA8, 0xA9, 0xAA, 0xAB, 0xAC, 0xAD, 0xAE, 0xAF, 0xEF, 0xE0, 0xE1, 0xE2, 0xE3, 0xA6, 0xA2, *************** static const unsigned char win8662win125 *** 188,202 **** 0xE0, 0xE1, 0xE2, 0xE3, 0xE4, 0xE5, 0xE6, 0xE7, 0xE8, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xA5, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF0, 0xF1, 0xF2, 0xF3, 0xF4, 0xF5, 0xF6, 0xF7, 0xF8, 0xF9, 0xFA, 0xFB, 0xFC, 0xFD, 0xFE, 0xFF, ! 0xA8, 0xB8, 0xAA, 0xBA, 0xAF, 0xBF, 0x00, 0x00, ! 0xB2, 0xB3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }; /* WIN1251 to WIN866 */ --- 188,202 ---- 0xE0, 0xE1, 0xE2, 0xE3, 0xE4, 0xE5, 0xE6, 0xE7, 0xE8, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF0, 0xF1, 0xF2, 0xF3, 0xF4, 0xF5, 0xF6, 0xF7, 0xF8, 0xF9, 0xFA, 0xFB, 0xFC, 0xFD, 0xFE, 0xFF, ! 0xA8, 0xB8, 0xAA, 0xBA, 0xAF, 0xBF, 0xA1, 0xA2, ! 0xB0, 0x00, 0xB7, 0x00, 0xB9, 0xA4, 0x00, 0xA0 }; /* WIN1251 to WIN866 */ *************** static const unsigned char win12512win86 *** 205,214 **** 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0xBD, 0x00, 0x00, 0xF0, 0x00, 0xF2, 0x00, 0x00, 0x00, 0x00, 0xF4, ! 0x00, 0x00, 0xF8, 0xF9, 0xAD, 0x00, 0x00, 0x00, ! 0xF1, 0x00, 0xF3, 0x00, 0x00, 0x00, 0x00, 0xF5, 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8A, 0x8B, 0x8C, 0x8D, 0x8E, 0x8F, 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, --- 205,214 ---- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0xFF, 0xF6, 0xF7, 0x00, 0xFD, 0x00, 0x00, 0x00, 0xF0, 0x00, 0xF2, 0x00, 0x00, 0x00, 0x00, 0xF4, ! 0xF8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFA, ! 0xF1, 0xFC, 0xF3, 0x00, 0x00, 0x00, 0x00, 0xF5, 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8A, 0x8B, 0x8C, 0x8D, 0x8E, 0x8F, 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, *************** static const unsigned char iso2win1251[] *** 225,232 **** 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0xA8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xC0, 0xC1, 0xC2, 0xC3, 0xC4, 0xC5, 0xC6, 0xC7, 0xC8, 0xC9, 0xCA, 0xCB, 0xCC, 0xCD, 0xCE, 0xCF, 0xD0, 0xD1, 0xD2, 0xD3, 0xD4, 0xD5, 0xD6, 0xD7, --- 225,232 ---- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0xA0, 0xA8, 0x80, 0x81, 0xAA, 0xBD, 0xB2, 0xAF, ! 0xA3, 0x8A, 0x8C, 0x8E, 0x8D, 0xAD, 0xA1, 0x8F, 0xC0, 0xC1, 0xC2, 0xC3, 0xC4, 0xC5, 0xC6, 0xC7, 0xC8, 0xC9, 0xCA, 0xCB, 0xCC, 0xCD, 0xCE, 0xCF, 0xD0, 0xD1, 0xD2, 0xD3, 0xD4, 0xD5, 0xD6, 0xD7, *************** static const unsigned char iso2win1251[] *** 235,254 **** 0xE8, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, 0xF0, 0xF1, 0xF2, 0xF3, 0xF4, 0xF5, 0xF6, 0xF7, 0xF8, 0xF9, 0xFA, 0xFB, 0xFC, 0xFD, 0xFE, 0xFF, ! 0x00, 0xB8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }; /* WIN1251 to ISO-8859-5 */ static const unsigned char win12512iso[] = { ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0xA1, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0xF1, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xB0, 0xB1, 0xB2, 0xB3, 0xB4, 0xB5, 0xB6, 0xB7, 0xB8, 0xB9, 0xBA, 0xBB, 0xBC, 0xBD, 0xBE, 0xBF, 0xC0, 0xC1, 0xC2, 0xC3, 0xC4, 0xC5, 0xC6, 0xC7, --- 235,254 ---- 0xE8, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, 0xF0, 0xF1, 0xF2, 0xF3, 0xF4, 0xF5, 0xF6, 0xF7, 0xF8, 0xF9, 0xFA, 0xFB, 0xFC, 0xFD, 0xFE, 0xFF, ! 0xB9, 0xB8, 0x90, 0x83, 0xBA, 0xBE, 0xB3, 0xBF, ! 0xBC, 0x9A, 0x9C, 0x9E, 0x9D, 0xA7, 0xA2, 0x9F }; /* WIN1251 to ISO-8859-5 */ static const unsigned char win12512iso[] = { ! 0xA2, 0xA3, 0x00, 0xF3, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0xA9, 0x00, 0xAA, 0xAC, 0xAB, 0xAF, ! 0xF2, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0xF9, 0x00, 0xFA, 0xFC, 0xFB, 0xFF, ! 0xA0, 0xAE, 0xFE, 0xA8, 0x00, 0x00, 0x00, 0xFD, ! 0xA1, 0x00, 0xA4, 0x00, 0x00, 0xAD, 0x00, 0xA7, ! 0x00, 0x00, 0xA6, 0xF6, 0x00, 0x00, 0x00, 0x00, ! 0xF1, 0xF0, 0xF4, 0x00, 0xF8, 0xA5, 0xF5, 0xF7, 0xB0, 0xB1, 0xB2, 0xB3, 0xB4, 0xB5, 0xB6, 0xB7, 0xB8, 0xB9, 0xBA, 0xBB, 0xBC, 0xBD, 0xBE, 0xBF, 0xC0, 0xC1, 0xC2, 0xC3, 0xC4, 0xC5, 0xC6, 0xC7, *************** static const unsigned char iso2win866[] *** 265,272 **** 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0xF0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8A, 0x8B, 0x8C, 0x8D, 0x8E, 0x8F, 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, --- 265,272 ---- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0xFF, 0xF0, 0x00, 0x00, 0xF2, 0x00, 0x00, 0xF4, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF6, 0x00, 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8A, 0x8B, 0x8C, 0x8D, 0x8E, 0x8F, 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, *************** static const unsigned char iso2win866[] *** 275,282 **** 0xA8, 0xA9, 0xAA, 0xAB, 0xAC, 0xAD, 0xAE, 0xAF, 0xE0, 0xE1, 0xE2, 0xE3, 0xE4, 0xE5, 0xE6, 0xE7, 0xE8, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, ! 0x00, 0xF1, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }; /* WIN866 to ISO-8859-5 */ --- 275,282 ---- 0xA8, 0xA9, 0xAA, 0xAB, 0xAC, 0xAD, 0xAE, 0xAF, 0xE0, 0xE1, 0xE2, 0xE3, 0xE4, 0xE5, 0xE6, 0xE7, 0xE8, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, ! 0xFC, 0xF1, 0x00, 0x00, 0xF3, 0x00, 0x00, 0xF5, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF7, 0x00 }; /* WIN866 to ISO-8859-5 */ *************** static const unsigned char win8662iso[] *** 295,302 **** 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xE0, 0xE1, 0xE2, 0xE3, 0xE4, 0xE5, 0xE6, 0xE7, 0xE8, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, ! 0xA1, 0xF1, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 }; --- 295,302 ---- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xE0, 0xE1, 0xE2, 0xE3, 0xE4, 0xE5, 0xE6, 0xE7, 0xE8, 0xE9, 0xEA, 0xEB, 0xEC, 0xED, 0xEE, 0xEF, ! 0xA1, 0xF1, 0xA4, 0xF4, 0xA7, 0xF7, 0xAE, 0xFE, ! 0x00, 0x00, 0x00, 0x00, 0xF0, 0x00, 0x00, 0xA0 }; diff --git a/src/backend/utils/mb/conversion_procs/latin2_and_win1250/latin2_and_win1250.c b/src/backend/utils/mb/conversion_procs/latin2_and_win1250/latin2_and_win1250.c index 1260b62..2cc223d 100644 *** a/src/backend/utils/mb/conversion_procs/latin2_and_win1250/latin2_and_win1250.c --- b/src/backend/utils/mb/conversion_procs/latin2_and_win1250/latin2_and_win1250.c *************** PG_FUNCTION_INFO_V1(win1250_to_latin2); *** 37,46 **** /* WIN1250 to ISO-8859-2 */ static const unsigned char win1250_2_iso88592[] = { ! 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, ! 0x88, 0x89, 0xA9, 0x8B, 0xA6, 0xAB, 0xAE, 0xAC, ! 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, ! 0x98, 0x99, 0xB9, 0x9B, 0xB6, 0xBB, 0xBE, 0xBC, 0xA0, 0xB7, 0xA2, 0xA3, 0xA4, 0xA1, 0x00, 0xA7, 0xA8, 0x00, 0xAA, 0x00, 0x00, 0xAD, 0x00, 0xAF, 0xB0, 0x00, 0xB2, 0xB3, 0xB4, 0x00, 0x00, 0x00, --- 37,46 ---- /* WIN1250 to ISO-8859-2 */ static const unsigned char win1250_2_iso88592[] = { ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0xA9, 0x00, 0xA6, 0xAB, 0xAE, 0xAC, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0xB9, 0x00, 0xB6, 0xBB, 0xBE, 0xBC, 0xA0, 0xB7, 0xA2, 0xA3, 0xA4, 0xA1, 0x00, 0xA7, 0xA8, 0x00, 0xAA, 0x00, 0x00, 0xAD, 0x00, 0xAF, 0xB0, 0x00, 0xB2, 0xB3, 0xB4, 0x00, 0x00, 0x00, *************** static const unsigned char win1250_2_iso *** 57,66 **** /* ISO-8859-2 to WIN1250 */ static const unsigned char iso88592_2_win1250[] = { ! 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, ! 0x88, 0x89, 0x00, 0x8B, 0x00, 0x00, 0x00, 0x00, ! 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, ! 0x98, 0x99, 0x00, 0x9B, 0x00, 0x00, 0x00, 0x00, 0xA0, 0xA5, 0xA2, 0xA3, 0xA4, 0xBC, 0x8C, 0xA7, 0xA8, 0x8A, 0xAA, 0x8D, 0x8F, 0xAD, 0x8E, 0xAF, 0xB0, 0xB9, 0xB2, 0xB3, 0xB4, 0xBE, 0x9C, 0xA1, --- 57,66 ---- /* ISO-8859-2 to WIN1250 */ static const unsigned char iso88592_2_win1250[] = { ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, ! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xA0, 0xA5, 0xA2, 0xA3, 0xA4, 0xBC, 0x8C, 0xA7, 0xA8, 0x8A, 0xAA, 0x8D, 0x8F, 0xAD, 0x8E, 0xAF, 0xB0, 0xB9, 0xB2, 0xB3, 0xB4, 0xBE, 0x9C, 0xA1, #include "c.h" #include "mb/pg_wchar.h" #include "iso8859_2_to_utf8.map" #include "iso8859_5_to_utf8.map" #include "win1250_to_utf8.map" #include "win1251_to_utf8.map" #include "win866_to_utf8.map" #include "koi8r_to_utf8.map" typedef struct { const char *enname1; const pg_local_to_utf *map1; /* to UTF8 map name */ int size1; /* size of map1 */ const char *enname2; const pg_local_to_utf *map2; /* to UTF8 map name */ int size2; /* size of map2 */ const char *tabname; } pg_conv_map; static const pg_conv_map maps[] = { { "WIN1250", LUmapWIN1250, lengthof(LUmapWIN1250), "ISO-8859-2", LUmapISO8859_2, lengthof(LUmapISO8859_2), "win1250_2_iso88592" }, { "ISO-8859-2", LUmapISO8859_2, lengthof(LUmapISO8859_2), "WIN1250", LUmapWIN1250, lengthof(LUmapWIN1250), "iso88592_2_win1250" }, { "ISO-8859-5", LUmapISO8859_5, lengthof(LUmapISO8859_5), "KOI8-R", LUmapKOI8R, lengthof(LUmapKOI8R), "iso2koi" }, { "KOI8-R", LUmapKOI8R, lengthof(LUmapKOI8R), "ISO-8859-5", LUmapISO8859_5, lengthof(LUmapISO8859_5), "koi2iso" }, { "WIN1251", LUmapWIN1251, lengthof(LUmapWIN1251), "KOI8-R", LUmapKOI8R, lengthof(LUmapKOI8R), "win12512koi" }, { "KOI8-R", LUmapKOI8R, lengthof(LUmapKOI8R), "WIN1251", LUmapWIN1251, lengthof(LUmapWIN1251), "koi2win1251" }, { "WIN866", LUmapWIN866, lengthof(LUmapWIN866), "KOI8-R", LUmapKOI8R, lengthof(LUmapKOI8R), "win8662koi" }, { "KOI8-R", LUmapKOI8R, lengthof(LUmapKOI8R), "WIN866", LUmapWIN866, lengthof(LUmapWIN866), "koi2win866" }, { "WIN866", LUmapWIN866, lengthof(LUmapWIN866), "WIN1251", LUmapWIN1251, lengthof(LUmapWIN1251), "win8662win1251" }, { "WIN1251", LUmapWIN1251, lengthof(LUmapWIN1251), "WIN866", LUmapWIN866, lengthof(LUmapWIN866), "win12512win866" }, { "ISO-8859-5", LUmapISO8859_5, lengthof(LUmapISO8859_5), "WIN1251", LUmapWIN1251, lengthof(LUmapWIN1251), "iso2win1251" }, { "WIN1251", LUmapWIN1251, lengthof(LUmapWIN1251), "ISO-8859-5", LUmapISO8859_5, lengthof(LUmapISO8859_5), "win12512iso" }, { "ISO-8859-5", LUmapISO8859_5, lengthof(LUmapISO8859_5), "WIN866", LUmapWIN866, lengthof(LUmapWIN866), "iso2win866" }, { "WIN866", LUmapWIN866, lengthof(LUmapWIN866), "ISO-8859-5", LUmapISO8859_5, lengthof(LUmapISO8859_5), "win8662iso" }, }; static void domap(const pg_conv_map *info) { uint32 c; printf("/* %s to %s */\n", info->enname1, info->enname2); printf("static const unsigned char %s[] = {\n", info->tabname); for (c = 0x80; c <= 0xff; c++) { uint32 u = 0; uint32 c2 = 0; int i; for (i = 0; i < info->size1; i++) { if (info->map1[i].code == c) { u = info->map1[i].utf; break; } } if (u != 0) { for (i = 0; i < info->size2; i++) { if (info->map2[i].utf == u) { c2 = info->map2[i].code; break; } } } #if 0 if (c2) printf("0x%02x maps to 0x%02x via U+%04X\n", c, c2, u); else printf("0x%02x has no equivalent\n", c); #endif if (c % 8 == 0) printf("\t"); printf("0x%02X", c2); if (c == 0xff) printf("\n"); else if (c % 8 == 7) printf(",\n"); else printf(", "); } printf("};\n\n"); } int main(int argc, char **argv) { int i; for (i = 0; i < lengthof(maps); i++) domap(maps + i); return 0; } set bytea_output = hex; \pset tuples_only on \pset format unaligned create or replace function check_conv( src_encoding name, dest_encoding name, mule_id int ) returns setof text strict language plpgsql as $$ declare printerrs bool := false; -- "true" prints error message text too, which is not too helpful when -- comparing 9.5 against HEAD, because of bogus "MULE_INTERNAL" references c int; cbytea bytea; trbytea bytea; trback bytea; errtext text; begin for c in 0..255 loop if c >= 128 and src_encoding = 'MULE_INTERNAL' then cbytea := set_byte(' ', 0, mule_id); cbytea := set_byte(cbytea, 1, c); else cbytea := set_byte(' ', 0, c); end if; begin trbytea = convert(cbytea, src_encoding, dest_encoding); errtext := ''; exception when sqlstate '22P05' or sqlstate '22021' then GET STACKED DIAGNOSTICS errtext = MESSAGE_TEXT; trbytea := '\x00'; end; if trbytea <> '\x00' then return next 'convert '::text || cbytea || ' from ' || src_encoding || ' to ' || dest_encoding || ' produced ' || trbytea; else if printerrs then return next 'convert '::text || cbytea || ' from ' || src_encoding || ' to ' || dest_encoding || ' failed: ' || errtext; else return next 'convert '::text || cbytea || ' from ' || src_encoding || ' to ' || dest_encoding || ' failed'; end if; end if; if trbytea <> '\x00' then trback = convert(trbytea, dest_encoding, src_encoding); if trback <> cbytea then return next 'convert '::text || cbytea || ' from ' || src_encoding || ' to ' || dest_encoding || ' reverse conversionproduced ' || trback; end if; end if; end loop; end $$; select check_conv('KOI8-R', 'MULE_INTERNAL', 139); select check_conv('MULE_INTERNAL', 'KOI8-R', 139); select check_conv('ISO-8859-5', 'MULE_INTERNAL', 139); select check_conv('MULE_INTERNAL', 'ISO-8859-5', 139); select check_conv('WIN1251', 'MULE_INTERNAL', 139); select check_conv('MULE_INTERNAL', 'WIN1251', 139); select check_conv('WIN866', 'MULE_INTERNAL', 139); select check_conv('MULE_INTERNAL', 'WIN866', 139); select check_conv('ISO-8859-5', 'KOI8-R', 139); select check_conv('KOI8-R', 'ISO-8859-5', 139); select check_conv('WIN1251', 'KOI8-R', 139); select check_conv('KOI8-R', 'WIN1251', 139); select check_conv('WIN866', 'KOI8-R', 139); select check_conv('KOI8-R', 'WIN866', 139); select check_conv('WIN866', 'WIN1251', 139); select check_conv('WIN1251', 'WIN866', 139); select check_conv('ISO-8859-5', 'WIN1251', 139); select check_conv('WIN1251', 'ISO-8859-5', 139); select check_conv('ISO-8859-5', 'WIN866', 139); select check_conv('WIN866', 'ISO-8859-5', 139); select check_conv('WIN1250', 'ISO-8859-2', 130); select check_conv('ISO-8859-2', 'WIN1250', 130); --- check95.out 2015-11-28 15:06:34.384735691 -0500 +++ check.out 2015-11-28 15:06:37.603812190 -0500 @@ -674,7 +674,7 @@ convert \x9d from ISO-8859-5 to MULE_INTERNAL failed convert \x9e from ISO-8859-5 to MULE_INTERNAL failed convert \x9f from ISO-8859-5 to MULE_INTERNAL failed -convert \xa0 from ISO-8859-5 to MULE_INTERNAL failed +convert \xa0 from ISO-8859-5 to MULE_INTERNAL produced \x8b9a convert \xa1 from ISO-8859-5 to MULE_INTERNAL produced \x8bb3 convert \xa2 from ISO-8859-5 to MULE_INTERNAL failed convert \xa3 from ISO-8859-5 to MULE_INTERNAL failed @@ -924,7 +924,7 @@ convert \x8b97 from MULE_INTERNAL to ISO-8859-5 failed convert \x8b98 from MULE_INTERNAL to ISO-8859-5 failed convert \x8b99 from MULE_INTERNAL to ISO-8859-5 failed -convert \x8b9a from MULE_INTERNAL to ISO-8859-5 failed +convert \x8b9a from MULE_INTERNAL to ISO-8859-5 produced \xa0 convert \x8b9b from MULE_INTERNAL to ISO-8859-5 failed convert \x8b9c from MULE_INTERNAL to ISO-8859-5 failed convert \x8b9d from MULE_INTERNAL to ISO-8859-5 failed @@ -1186,38 +1186,38 @@ convert \x9d from WIN1251 to MULE_INTERNAL failed convert \x9e from WIN1251 to MULE_INTERNAL failed convert \x9f from WIN1251 to MULE_INTERNAL failed -convert \xa0 from WIN1251 to MULE_INTERNAL failed +convert \xa0 from WIN1251 to MULE_INTERNAL produced \x8b9a convert \xa1 from WIN1251 to MULE_INTERNAL failed convert \xa2 from WIN1251 to MULE_INTERNAL failed convert \xa3 from WIN1251 to MULE_INTERNAL failed convert \xa4 from WIN1251 to MULE_INTERNAL failed -convert \xa5 from WIN1251 to MULE_INTERNAL produced \x8bbd +convert \xa5 from WIN1251 to MULE_INTERNAL failed convert \xa6 from WIN1251 to MULE_INTERNAL failed convert \xa7 from WIN1251 to MULE_INTERNAL failed convert \xa8 from WIN1251 to MULE_INTERNAL produced \x8bb3 -convert \xa9 from WIN1251 to MULE_INTERNAL failed -convert \xaa from WIN1251 to MULE_INTERNAL produced \x8bb4 +convert \xa9 from WIN1251 to MULE_INTERNAL produced \x8bbf +convert \xaa from WIN1251 to MULE_INTERNAL failed convert \xab from WIN1251 to MULE_INTERNAL failed convert \xac from WIN1251 to MULE_INTERNAL failed convert \xad from WIN1251 to MULE_INTERNAL failed convert \xae from WIN1251 to MULE_INTERNAL failed -convert \xaf from WIN1251 to MULE_INTERNAL produced \x8bb7 -convert \xb0 from WIN1251 to MULE_INTERNAL failed +convert \xaf from WIN1251 to MULE_INTERNAL failed +convert \xb0 from WIN1251 to MULE_INTERNAL produced \x8b9c convert \xb1 from WIN1251 to MULE_INTERNAL failed -convert \xb2 from WIN1251 to MULE_INTERNAL produced \x8bb6 -convert \xb3 from WIN1251 to MULE_INTERNAL produced \x8ba6 -convert \xb4 from WIN1251 to MULE_INTERNAL produced \x8bad +convert \xb2 from WIN1251 to MULE_INTERNAL failed +convert \xb3 from WIN1251 to MULE_INTERNAL failed +convert \xb4 from WIN1251 to MULE_INTERNAL failed convert \xb5 from WIN1251 to MULE_INTERNAL failed convert \xb6 from WIN1251 to MULE_INTERNAL failed -convert \xb7 from WIN1251 to MULE_INTERNAL failed +convert \xb7 from WIN1251 to MULE_INTERNAL produced \x8b9e convert \xb8 from WIN1251 to MULE_INTERNAL produced \x8ba3 convert \xb9 from WIN1251 to MULE_INTERNAL failed -convert \xba from WIN1251 to MULE_INTERNAL produced \x8ba4 +convert \xba from WIN1251 to MULE_INTERNAL failed convert \xbb from WIN1251 to MULE_INTERNAL failed convert \xbc from WIN1251 to MULE_INTERNAL failed convert \xbd from WIN1251 to MULE_INTERNAL failed convert \xbe from WIN1251 to MULE_INTERNAL failed -convert \xbf from WIN1251 to MULE_INTERNAL produced \x8ba7 +convert \xbf from WIN1251 to MULE_INTERNAL failed convert \xc0 from WIN1251 to MULE_INTERNAL produced \x8be1 convert \xc1 from WIN1251 to MULE_INTERNAL produced \x8be2 convert \xc2 from WIN1251 to MULE_INTERNAL produced \x8bf7 @@ -1436,44 +1436,44 @@ convert \x8b97 from MULE_INTERNAL to WIN1251 failed convert \x8b98 from MULE_INTERNAL to WIN1251 failed convert \x8b99 from MULE_INTERNAL to WIN1251 failed -convert \x8b9a from MULE_INTERNAL to WIN1251 failed +convert \x8b9a from MULE_INTERNAL to WIN1251 produced \xa0 convert \x8b9b from MULE_INTERNAL to WIN1251 failed -convert \x8b9c from MULE_INTERNAL to WIN1251 failed +convert \x8b9c from MULE_INTERNAL to WIN1251 produced \xb0 convert \x8b9d from MULE_INTERNAL to WIN1251 failed -convert \x8b9e from MULE_INTERNAL to WIN1251 failed +convert \x8b9e from MULE_INTERNAL to WIN1251 produced \xb7 convert \x8b9f from MULE_INTERNAL to WIN1251 failed convert \x8ba0 from MULE_INTERNAL to WIN1251 failed convert \x8ba1 from MULE_INTERNAL to WIN1251 failed convert \x8ba2 from MULE_INTERNAL to WIN1251 failed convert \x8ba3 from MULE_INTERNAL to WIN1251 produced \xb8 -convert \x8ba4 from MULE_INTERNAL to WIN1251 produced \xba +convert \x8ba4 from MULE_INTERNAL to WIN1251 failed convert \x8ba5 from MULE_INTERNAL to WIN1251 failed -convert \x8ba6 from MULE_INTERNAL to WIN1251 produced \xb3 -convert \x8ba7 from MULE_INTERNAL to WIN1251 produced \xbf +convert \x8ba6 from MULE_INTERNAL to WIN1251 failed +convert \x8ba7 from MULE_INTERNAL to WIN1251 failed convert \x8ba8 from MULE_INTERNAL to WIN1251 failed convert \x8ba9 from MULE_INTERNAL to WIN1251 failed convert \x8baa from MULE_INTERNAL to WIN1251 failed convert \x8bab from MULE_INTERNAL to WIN1251 failed convert \x8bac from MULE_INTERNAL to WIN1251 failed -convert \x8bad from MULE_INTERNAL to WIN1251 produced \xb4 +convert \x8bad from MULE_INTERNAL to WIN1251 failed convert \x8bae from MULE_INTERNAL to WIN1251 failed convert \x8baf from MULE_INTERNAL to WIN1251 failed convert \x8bb0 from MULE_INTERNAL to WIN1251 failed convert \x8bb1 from MULE_INTERNAL to WIN1251 failed convert \x8bb2 from MULE_INTERNAL to WIN1251 failed convert \x8bb3 from MULE_INTERNAL to WIN1251 produced \xa8 -convert \x8bb4 from MULE_INTERNAL to WIN1251 produced \xaa +convert \x8bb4 from MULE_INTERNAL to WIN1251 failed convert \x8bb5 from MULE_INTERNAL to WIN1251 failed -convert \x8bb6 from MULE_INTERNAL to WIN1251 produced \xb2 -convert \x8bb7 from MULE_INTERNAL to WIN1251 produced \xaf +convert \x8bb6 from MULE_INTERNAL to WIN1251 failed +convert \x8bb7 from MULE_INTERNAL to WIN1251 failed convert \x8bb8 from MULE_INTERNAL to WIN1251 failed convert \x8bb9 from MULE_INTERNAL to WIN1251 failed convert \x8bba from MULE_INTERNAL to WIN1251 failed convert \x8bbb from MULE_INTERNAL to WIN1251 failed convert \x8bbc from MULE_INTERNAL to WIN1251 failed -convert \x8bbd from MULE_INTERNAL to WIN1251 produced \xa5 +convert \x8bbd from MULE_INTERNAL to WIN1251 failed convert \x8bbe from MULE_INTERNAL to WIN1251 failed -convert \x8bbf from MULE_INTERNAL to WIN1251 failed +convert \x8bbf from MULE_INTERNAL to WIN1251 produced \xa9 convert \x8bc0 from MULE_INTERNAL to WIN1251 produced \xfe convert \x8bc1 from MULE_INTERNAL to WIN1251 produced \xe0 convert \x8bc2 from MULE_INTERNAL to WIN1251 produced \xe1 @@ -1714,54 +1714,54 @@ convert \xad from WIN866 to MULE_INTERNAL produced \x8bce convert \xae from WIN866 to MULE_INTERNAL produced \x8bcf convert \xaf from WIN866 to MULE_INTERNAL produced \x8bd0 -convert \xb0 from WIN866 to MULE_INTERNAL failed -convert \xb1 from WIN866 to MULE_INTERNAL failed -convert \xb2 from WIN866 to MULE_INTERNAL failed -convert \xb3 from WIN866 to MULE_INTERNAL failed -convert \xb4 from WIN866 to MULE_INTERNAL failed -convert \xb5 from WIN866 to MULE_INTERNAL failed -convert \xb6 from WIN866 to MULE_INTERNAL failed -convert \xb7 from WIN866 to MULE_INTERNAL failed -convert \xb8 from WIN866 to MULE_INTERNAL failed -convert \xb9 from WIN866 to MULE_INTERNAL failed -convert \xba from WIN866 to MULE_INTERNAL failed -convert \xbb from WIN866 to MULE_INTERNAL failed -convert \xbc from WIN866 to MULE_INTERNAL failed -convert \xbd from WIN866 to MULE_INTERNAL produced \x8bbd -convert \xbe from WIN866 to MULE_INTERNAL failed -convert \xbf from WIN866 to MULE_INTERNAL failed -convert \xc0 from WIN866 to MULE_INTERNAL failed -convert \xc1 from WIN866 to MULE_INTERNAL failed -convert \xc2 from WIN866 to MULE_INTERNAL failed -convert \xc3 from WIN866 to MULE_INTERNAL failed -convert \xc4 from WIN866 to MULE_INTERNAL failed -convert \xc5 from WIN866 to MULE_INTERNAL failed -convert \xc6 from WIN866 to MULE_INTERNAL failed -convert \xc7 from WIN866 to MULE_INTERNAL failed -convert \xc8 from WIN866 to MULE_INTERNAL failed -convert \xc9 from WIN866 to MULE_INTERNAL failed -convert \xca from WIN866 to MULE_INTERNAL failed -convert \xcb from WIN866 to MULE_INTERNAL failed -convert \xcc from WIN866 to MULE_INTERNAL failed -convert \xcd from WIN866 to MULE_INTERNAL failed -convert \xce from WIN866 to MULE_INTERNAL failed -convert \xcf from WIN866 to MULE_INTERNAL failed -convert \xd0 from WIN866 to MULE_INTERNAL failed -convert \xd1 from WIN866 to MULE_INTERNAL failed -convert \xd2 from WIN866 to MULE_INTERNAL failed -convert \xd3 from WIN866 to MULE_INTERNAL failed -convert \xd4 from WIN866 to MULE_INTERNAL failed -convert \xd5 from WIN866 to MULE_INTERNAL failed -convert \xd6 from WIN866 to MULE_INTERNAL failed -convert \xd7 from WIN866 to MULE_INTERNAL failed -convert \xd8 from WIN866 to MULE_INTERNAL failed -convert \xd9 from WIN866 to MULE_INTERNAL failed -convert \xda from WIN866 to MULE_INTERNAL failed -convert \xdb from WIN866 to MULE_INTERNAL failed -convert \xdc from WIN866 to MULE_INTERNAL failed -convert \xdd from WIN866 to MULE_INTERNAL failed -convert \xde from WIN866 to MULE_INTERNAL failed -convert \xdf from WIN866 to MULE_INTERNAL failed +convert \xb0 from WIN866 to MULE_INTERNAL produced \x8b90 +convert \xb1 from WIN866 to MULE_INTERNAL produced \x8b91 +convert \xb2 from WIN866 to MULE_INTERNAL produced \x8b92 +convert \xb3 from WIN866 to MULE_INTERNAL produced \x8b81 +convert \xb4 from WIN866 to MULE_INTERNAL produced \x8b87 +convert \xb5 from WIN866 to MULE_INTERNAL produced \x8bb2 +convert \xb6 from WIN866 to MULE_INTERNAL produced \x8bb4 +convert \xb7 from WIN866 to MULE_INTERNAL produced \x8ba7 +convert \xb8 from WIN866 to MULE_INTERNAL produced \x8ba6 +convert \xb9 from WIN866 to MULE_INTERNAL produced \x8bb5 +convert \xba from WIN866 to MULE_INTERNAL produced \x8ba1 +convert \xbb from WIN866 to MULE_INTERNAL produced \x8ba8 +convert \xbc from WIN866 to MULE_INTERNAL produced \x8bae +convert \xbd from WIN866 to MULE_INTERNAL produced \x8bad +convert \xbe from WIN866 to MULE_INTERNAL produced \x8bac +convert \xbf from WIN866 to MULE_INTERNAL produced \x8b83 +convert \xc0 from WIN866 to MULE_INTERNAL produced \x8b84 +convert \xc1 from WIN866 to MULE_INTERNAL produced \x8b89 +convert \xc2 from WIN866 to MULE_INTERNAL produced \x8b88 +convert \xc3 from WIN866 to MULE_INTERNAL produced \x8b86 +convert \xc4 from WIN866 to MULE_INTERNAL produced \x8b80 +convert \xc5 from WIN866 to MULE_INTERNAL produced \x8b8a +convert \xc6 from WIN866 to MULE_INTERNAL produced \x8baf +convert \xc7 from WIN866 to MULE_INTERNAL produced \x8bb0 +convert \xc8 from WIN866 to MULE_INTERNAL produced \x8bab +convert \xc9 from WIN866 to MULE_INTERNAL produced \x8ba5 +convert \xca from WIN866 to MULE_INTERNAL produced \x8bbb +convert \xcb from WIN866 to MULE_INTERNAL produced \x8bb8 +convert \xcc from WIN866 to MULE_INTERNAL produced \x8bb1 +convert \xcd from WIN866 to MULE_INTERNAL produced \x8ba0 +convert \xce from WIN866 to MULE_INTERNAL produced \x8bbe +convert \xcf from WIN866 to MULE_INTERNAL produced \x8bb9 +convert \xd0 from WIN866 to MULE_INTERNAL produced \x8bba +convert \xd1 from WIN866 to MULE_INTERNAL produced \x8bb6 +convert \xd2 from WIN866 to MULE_INTERNAL produced \x8bb7 +convert \xd3 from WIN866 to MULE_INTERNAL produced \x8baa +convert \xd4 from WIN866 to MULE_INTERNAL produced \x8ba9 +convert \xd5 from WIN866 to MULE_INTERNAL produced \x8ba2 +convert \xd6 from WIN866 to MULE_INTERNAL produced \x8ba4 +convert \xd7 from WIN866 to MULE_INTERNAL produced \x8bbd +convert \xd8 from WIN866 to MULE_INTERNAL produced \x8bbc +convert \xd9 from WIN866 to MULE_INTERNAL produced \x8b85 +convert \xda from WIN866 to MULE_INTERNAL produced \x8b82 +convert \xdb from WIN866 to MULE_INTERNAL produced \x8b8d +convert \xdc from WIN866 to MULE_INTERNAL produced \x8b8c +convert \xdd from WIN866 to MULE_INTERNAL produced \x8b8e +convert \xde from WIN866 to MULE_INTERNAL produced \x8b8f +convert \xdf from WIN866 to MULE_INTERNAL produced \x8b8b convert \xe0 from WIN866 to MULE_INTERNAL produced \x8bd2 convert \xe1 from WIN866 to MULE_INTERNAL produced \x8bd3 convert \xe2 from WIN866 to MULE_INTERNAL produced \x8bd4 @@ -1780,20 +1780,20 @@ convert \xef from WIN866 to MULE_INTERNAL produced \x8bd1 convert \xf0 from WIN866 to MULE_INTERNAL produced \x8bb3 convert \xf1 from WIN866 to MULE_INTERNAL produced \x8ba3 -convert \xf2 from WIN866 to MULE_INTERNAL produced \x8bb4 -convert \xf3 from WIN866 to MULE_INTERNAL produced \x8ba4 -convert \xf4 from WIN866 to MULE_INTERNAL produced \x8bb7 -convert \xf5 from WIN866 to MULE_INTERNAL produced \x8ba7 +convert \xf2 from WIN866 to MULE_INTERNAL failed +convert \xf3 from WIN866 to MULE_INTERNAL failed +convert \xf4 from WIN866 to MULE_INTERNAL failed +convert \xf5 from WIN866 to MULE_INTERNAL failed convert \xf6 from WIN866 to MULE_INTERNAL failed convert \xf7 from WIN866 to MULE_INTERNAL failed -convert \xf8 from WIN866 to MULE_INTERNAL produced \x8bb6 -convert \xf9 from WIN866 to MULE_INTERNAL produced \x8ba6 -convert \xfa from WIN866 to MULE_INTERNAL failed -convert \xfb from WIN866 to MULE_INTERNAL failed +convert \xf8 from WIN866 to MULE_INTERNAL produced \x8b9c +convert \xf9 from WIN866 to MULE_INTERNAL produced \x8b95 +convert \xfa from WIN866 to MULE_INTERNAL produced \x8b9e +convert \xfb from WIN866 to MULE_INTERNAL produced \x8b96 convert \xfc from WIN866 to MULE_INTERNAL failed convert \xfd from WIN866 to MULE_INTERNAL failed -convert \xfe from WIN866 to MULE_INTERNAL failed -convert \xff from WIN866 to MULE_INTERNAL failed +convert \xfe from WIN866 to MULE_INTERNAL produced \x8b94 +convert \xff from WIN866 to MULE_INTERNAL produced \x8b9a convert \x00 from MULE_INTERNAL to WIN866 failed convert \x01 from MULE_INTERNAL to WIN866 produced \x01 convert \x02 from MULE_INTERNAL to WIN866 produced \x02 @@ -1922,70 +1922,69 @@ convert \x7d from MULE_INTERNAL to WIN866 produced \x7d convert \x7e from MULE_INTERNAL to WIN866 produced \x7e convert \x7f from MULE_INTERNAL to WIN866 produced \x7f -convert \x8b80 from MULE_INTERNAL to WIN866 failed -convert \x8b81 from MULE_INTERNAL to WIN866 failed -convert \x8b82 from MULE_INTERNAL to WIN866 failed -convert \x8b83 from MULE_INTERNAL to WIN866 failed -convert \x8b84 from MULE_INTERNAL to WIN866 failed -convert \x8b85 from MULE_INTERNAL to WIN866 failed -convert \x8b86 from MULE_INTERNAL to WIN866 failed -convert \x8b87 from MULE_INTERNAL to WIN866 failed -convert \x8b88 from MULE_INTERNAL to WIN866 failed -convert \x8b89 from MULE_INTERNAL to WIN866 failed -convert \x8b8a from MULE_INTERNAL to WIN866 failed -convert \x8b8b from MULE_INTERNAL to WIN866 failed -convert \x8b8c from MULE_INTERNAL to WIN866 failed -convert \x8b8d from MULE_INTERNAL to WIN866 failed -convert \x8b8e from MULE_INTERNAL to WIN866 failed -convert \x8b8f from MULE_INTERNAL to WIN866 failed -convert \x8b90 from MULE_INTERNAL to WIN866 failed -convert \x8b91 from MULE_INTERNAL to WIN866 failed -convert \x8b92 from MULE_INTERNAL to WIN866 failed +convert \x8b80 from MULE_INTERNAL to WIN866 produced \xc4 +convert \x8b81 from MULE_INTERNAL to WIN866 produced \xb3 +convert \x8b82 from MULE_INTERNAL to WIN866 produced \xda +convert \x8b83 from MULE_INTERNAL to WIN866 produced \xbf +convert \x8b84 from MULE_INTERNAL to WIN866 produced \xc0 +convert \x8b85 from MULE_INTERNAL to WIN866 produced \xd9 +convert \x8b86 from MULE_INTERNAL to WIN866 produced \xc3 +convert \x8b87 from MULE_INTERNAL to WIN866 produced \xb4 +convert \x8b88 from MULE_INTERNAL to WIN866 produced \xc2 +convert \x8b89 from MULE_INTERNAL to WIN866 produced \xc1 +convert \x8b8a from MULE_INTERNAL to WIN866 produced \xc5 +convert \x8b8b from MULE_INTERNAL to WIN866 produced \xdf +convert \x8b8c from MULE_INTERNAL to WIN866 produced \xdc +convert \x8b8d from MULE_INTERNAL to WIN866 produced \xdb +convert \x8b8e from MULE_INTERNAL to WIN866 produced \xdd +convert \x8b8f from MULE_INTERNAL to WIN866 produced \xde +convert \x8b90 from MULE_INTERNAL to WIN866 produced \xb0 +convert \x8b91 from MULE_INTERNAL to WIN866 produced \xb1 +convert \x8b92 from MULE_INTERNAL to WIN866 produced \xb2 convert \x8b93 from MULE_INTERNAL to WIN866 failed -convert \x8b94 from MULE_INTERNAL to WIN866 failed -convert \x8b95 from MULE_INTERNAL to WIN866 failed -convert \x8b96 from MULE_INTERNAL to WIN866 failed +convert \x8b94 from MULE_INTERNAL to WIN866 produced \xfe +convert \x8b95 from MULE_INTERNAL to WIN866 produced \xf9 +convert \x8b96 from MULE_INTERNAL to WIN866 produced \xfb convert \x8b97 from MULE_INTERNAL to WIN866 failed convert \x8b98 from MULE_INTERNAL to WIN866 failed convert \x8b99 from MULE_INTERNAL to WIN866 failed -convert \x8b9a from MULE_INTERNAL to WIN866 failed +convert \x8b9a from MULE_INTERNAL to WIN866 produced \xff convert \x8b9b from MULE_INTERNAL to WIN866 failed -convert \x8b9c from MULE_INTERNAL to WIN866 failed +convert \x8b9c from MULE_INTERNAL to WIN866 produced \xf8 convert \x8b9d from MULE_INTERNAL to WIN866 failed -convert \x8b9e from MULE_INTERNAL to WIN866 failed +convert \x8b9e from MULE_INTERNAL to WIN866 produced \xfa convert \x8b9f from MULE_INTERNAL to WIN866 failed -convert \x8ba0 from MULE_INTERNAL to WIN866 failed -convert \x8ba1 from MULE_INTERNAL to WIN866 failed -convert \x8ba2 from MULE_INTERNAL to WIN866 failed +convert \x8ba0 from MULE_INTERNAL to WIN866 produced \xcd +convert \x8ba1 from MULE_INTERNAL to WIN866 produced \xba +convert \x8ba2 from MULE_INTERNAL to WIN866 produced \xd5 convert \x8ba3 from MULE_INTERNAL to WIN866 produced \xf1 -convert \x8ba4 from MULE_INTERNAL to WIN866 produced \xf3 -convert \x8ba5 from MULE_INTERNAL to WIN866 failed -convert \x8ba6 from MULE_INTERNAL to WIN866 produced \xf9 -convert \x8ba7 from MULE_INTERNAL to WIN866 produced \xf5 -convert \x8ba8 from MULE_INTERNAL to WIN866 failed -convert \x8ba9 from MULE_INTERNAL to WIN866 failed -convert \x8baa from MULE_INTERNAL to WIN866 failed -convert \x8bab from MULE_INTERNAL to WIN866 failed -convert \x8bac from MULE_INTERNAL to WIN866 failed -convert \x8bad from MULE_INTERNAL to WIN866 produced \xad -convert \x8bad from MULE_INTERNAL to WIN866 reverse conversion produced \x8bce -convert \x8bae from MULE_INTERNAL to WIN866 failed -convert \x8baf from MULE_INTERNAL to WIN866 failed -convert \x8bb0 from MULE_INTERNAL to WIN866 failed -convert \x8bb1 from MULE_INTERNAL to WIN866 failed -convert \x8bb2 from MULE_INTERNAL to WIN866 failed +convert \x8ba4 from MULE_INTERNAL to WIN866 produced \xd6 +convert \x8ba5 from MULE_INTERNAL to WIN866 produced \xc9 +convert \x8ba6 from MULE_INTERNAL to WIN866 produced \xb8 +convert \x8ba7 from MULE_INTERNAL to WIN866 produced \xb7 +convert \x8ba8 from MULE_INTERNAL to WIN866 produced \xbb +convert \x8ba9 from MULE_INTERNAL to WIN866 produced \xd4 +convert \x8baa from MULE_INTERNAL to WIN866 produced \xd3 +convert \x8bab from MULE_INTERNAL to WIN866 produced \xc8 +convert \x8bac from MULE_INTERNAL to WIN866 produced \xbe +convert \x8bad from MULE_INTERNAL to WIN866 produced \xbd +convert \x8bae from MULE_INTERNAL to WIN866 produced \xbc +convert \x8baf from MULE_INTERNAL to WIN866 produced \xc6 +convert \x8bb0 from MULE_INTERNAL to WIN866 produced \xc7 +convert \x8bb1 from MULE_INTERNAL to WIN866 produced \xcc +convert \x8bb2 from MULE_INTERNAL to WIN866 produced \xb5 convert \x8bb3 from MULE_INTERNAL to WIN866 produced \xf0 -convert \x8bb4 from MULE_INTERNAL to WIN866 produced \xf2 -convert \x8bb5 from MULE_INTERNAL to WIN866 failed -convert \x8bb6 from MULE_INTERNAL to WIN866 produced \xf8 -convert \x8bb7 from MULE_INTERNAL to WIN866 produced \xf4 -convert \x8bb8 from MULE_INTERNAL to WIN866 failed -convert \x8bb9 from MULE_INTERNAL to WIN866 failed -convert \x8bba from MULE_INTERNAL to WIN866 failed -convert \x8bbb from MULE_INTERNAL to WIN866 failed -convert \x8bbc from MULE_INTERNAL to WIN866 failed -convert \x8bbd from MULE_INTERNAL to WIN866 produced \xbd -convert \x8bbe from MULE_INTERNAL to WIN866 failed +convert \x8bb4 from MULE_INTERNAL to WIN866 produced \xb6 +convert \x8bb5 from MULE_INTERNAL to WIN866 produced \xb9 +convert \x8bb6 from MULE_INTERNAL to WIN866 produced \xd1 +convert \x8bb7 from MULE_INTERNAL to WIN866 produced \xd2 +convert \x8bb8 from MULE_INTERNAL to WIN866 produced \xcb +convert \x8bb9 from MULE_INTERNAL to WIN866 produced \xcf +convert \x8bba from MULE_INTERNAL to WIN866 produced \xd0 +convert \x8bbb from MULE_INTERNAL to WIN866 produced \xca +convert \x8bbc from MULE_INTERNAL to WIN866 produced \xd8 +convert \x8bbd from MULE_INTERNAL to WIN866 produced \xd7 +convert \x8bbe from MULE_INTERNAL to WIN866 produced \xce convert \x8bbf from MULE_INTERNAL to WIN866 failed convert \x8bc0 from MULE_INTERNAL to WIN866 produced \xee convert \x8bc1 from MULE_INTERNAL to WIN866 produced \xa0 @@ -2211,7 +2210,7 @@ convert \x9d from ISO-8859-5 to KOI8-R failed convert \x9e from ISO-8859-5 to KOI8-R failed convert \x9f from ISO-8859-5 to KOI8-R failed -convert \xa0 from ISO-8859-5 to KOI8-R failed +convert \xa0 from ISO-8859-5 to KOI8-R produced \x9a convert \xa1 from ISO-8859-5 to KOI8-R produced \xb3 convert \xa2 from ISO-8859-5 to KOI8-R failed convert \xa3 from ISO-8859-5 to KOI8-R failed @@ -2461,7 +2460,7 @@ convert \x97 from KOI8-R to ISO-8859-5 failed convert \x98 from KOI8-R to ISO-8859-5 failed convert \x99 from KOI8-R to ISO-8859-5 failed -convert \x9a from KOI8-R to ISO-8859-5 failed +convert \x9a from KOI8-R to ISO-8859-5 produced \xa0 convert \x9b from KOI8-R to ISO-8859-5 failed convert \x9c from KOI8-R to ISO-8859-5 failed convert \x9d from KOI8-R to ISO-8859-5 failed @@ -2723,38 +2722,38 @@ convert \x9d from WIN1251 to KOI8-R failed convert \x9e from WIN1251 to KOI8-R failed convert \x9f from WIN1251 to KOI8-R failed -convert \xa0 from WIN1251 to KOI8-R failed +convert \xa0 from WIN1251 to KOI8-R produced \x9a convert \xa1 from WIN1251 to KOI8-R failed convert \xa2 from WIN1251 to KOI8-R failed convert \xa3 from WIN1251 to KOI8-R failed convert \xa4 from WIN1251 to KOI8-R failed -convert \xa5 from WIN1251 to KOI8-R produced \xbd +convert \xa5 from WIN1251 to KOI8-R failed convert \xa6 from WIN1251 to KOI8-R failed convert \xa7 from WIN1251 to KOI8-R failed convert \xa8 from WIN1251 to KOI8-R produced \xb3 -convert \xa9 from WIN1251 to KOI8-R failed -convert \xaa from WIN1251 to KOI8-R produced \xb4 +convert \xa9 from WIN1251 to KOI8-R produced \xbf +convert \xaa from WIN1251 to KOI8-R failed convert \xab from WIN1251 to KOI8-R failed convert \xac from WIN1251 to KOI8-R failed convert \xad from WIN1251 to KOI8-R failed convert \xae from WIN1251 to KOI8-R failed -convert \xaf from WIN1251 to KOI8-R produced \xb7 -convert \xb0 from WIN1251 to KOI8-R failed +convert \xaf from WIN1251 to KOI8-R failed +convert \xb0 from WIN1251 to KOI8-R produced \x9c convert \xb1 from WIN1251 to KOI8-R failed -convert \xb2 from WIN1251 to KOI8-R produced \xb6 -convert \xb3 from WIN1251 to KOI8-R produced \xa6 -convert \xb4 from WIN1251 to KOI8-R produced \xad +convert \xb2 from WIN1251 to KOI8-R failed +convert \xb3 from WIN1251 to KOI8-R failed +convert \xb4 from WIN1251 to KOI8-R failed convert \xb5 from WIN1251 to KOI8-R failed convert \xb6 from WIN1251 to KOI8-R failed -convert \xb7 from WIN1251 to KOI8-R failed +convert \xb7 from WIN1251 to KOI8-R produced \x9e convert \xb8 from WIN1251 to KOI8-R produced \xa3 convert \xb9 from WIN1251 to KOI8-R failed -convert \xba from WIN1251 to KOI8-R produced \xa4 +convert \xba from WIN1251 to KOI8-R failed convert \xbb from WIN1251 to KOI8-R failed convert \xbc from WIN1251 to KOI8-R failed convert \xbd from WIN1251 to KOI8-R failed convert \xbe from WIN1251 to KOI8-R failed -convert \xbf from WIN1251 to KOI8-R produced \xa7 +convert \xbf from WIN1251 to KOI8-R failed convert \xc0 from WIN1251 to KOI8-R produced \xe1 convert \xc1 from WIN1251 to KOI8-R produced \xe2 convert \xc2 from WIN1251 to KOI8-R produced \xf7 @@ -2973,44 +2972,44 @@ convert \x97 from KOI8-R to WIN1251 failed convert \x98 from KOI8-R to WIN1251 failed convert \x99 from KOI8-R to WIN1251 failed -convert \x9a from KOI8-R to WIN1251 failed +convert \x9a from KOI8-R to WIN1251 produced \xa0 convert \x9b from KOI8-R to WIN1251 failed -convert \x9c from KOI8-R to WIN1251 failed +convert \x9c from KOI8-R to WIN1251 produced \xb0 convert \x9d from KOI8-R to WIN1251 failed -convert \x9e from KOI8-R to WIN1251 failed +convert \x9e from KOI8-R to WIN1251 produced \xb7 convert \x9f from KOI8-R to WIN1251 failed convert \xa0 from KOI8-R to WIN1251 failed convert \xa1 from KOI8-R to WIN1251 failed convert \xa2 from KOI8-R to WIN1251 failed convert \xa3 from KOI8-R to WIN1251 produced \xb8 -convert \xa4 from KOI8-R to WIN1251 produced \xba +convert \xa4 from KOI8-R to WIN1251 failed convert \xa5 from KOI8-R to WIN1251 failed -convert \xa6 from KOI8-R to WIN1251 produced \xb3 -convert \xa7 from KOI8-R to WIN1251 produced \xbf +convert \xa6 from KOI8-R to WIN1251 failed +convert \xa7 from KOI8-R to WIN1251 failed convert \xa8 from KOI8-R to WIN1251 failed convert \xa9 from KOI8-R to WIN1251 failed convert \xaa from KOI8-R to WIN1251 failed convert \xab from KOI8-R to WIN1251 failed convert \xac from KOI8-R to WIN1251 failed -convert \xad from KOI8-R to WIN1251 produced \xb4 +convert \xad from KOI8-R to WIN1251 failed convert \xae from KOI8-R to WIN1251 failed convert \xaf from KOI8-R to WIN1251 failed convert \xb0 from KOI8-R to WIN1251 failed convert \xb1 from KOI8-R to WIN1251 failed convert \xb2 from KOI8-R to WIN1251 failed convert \xb3 from KOI8-R to WIN1251 produced \xa8 -convert \xb4 from KOI8-R to WIN1251 produced \xaa +convert \xb4 from KOI8-R to WIN1251 failed convert \xb5 from KOI8-R to WIN1251 failed -convert \xb6 from KOI8-R to WIN1251 produced \xb2 -convert \xb7 from KOI8-R to WIN1251 produced \xaf +convert \xb6 from KOI8-R to WIN1251 failed +convert \xb7 from KOI8-R to WIN1251 failed convert \xb8 from KOI8-R to WIN1251 failed convert \xb9 from KOI8-R to WIN1251 failed convert \xba from KOI8-R to WIN1251 failed convert \xbb from KOI8-R to WIN1251 failed convert \xbc from KOI8-R to WIN1251 failed -convert \xbd from KOI8-R to WIN1251 produced \xa5 +convert \xbd from KOI8-R to WIN1251 failed convert \xbe from KOI8-R to WIN1251 failed -convert \xbf from KOI8-R to WIN1251 failed +convert \xbf from KOI8-R to WIN1251 produced \xa9 convert \xc0 from KOI8-R to WIN1251 produced \xfe convert \xc1 from KOI8-R to WIN1251 produced \xe0 convert \xc2 from KOI8-R to WIN1251 produced \xe1 @@ -3251,54 +3250,54 @@ convert \xad from WIN866 to KOI8-R produced \xce convert \xae from WIN866 to KOI8-R produced \xcf convert \xaf from WIN866 to KOI8-R produced \xd0 -convert \xb0 from WIN866 to KOI8-R failed -convert \xb1 from WIN866 to KOI8-R failed -convert \xb2 from WIN866 to KOI8-R failed -convert \xb3 from WIN866 to KOI8-R failed -convert \xb4 from WIN866 to KOI8-R failed -convert \xb5 from WIN866 to KOI8-R failed -convert \xb6 from WIN866 to KOI8-R failed -convert \xb7 from WIN866 to KOI8-R failed -convert \xb8 from WIN866 to KOI8-R failed -convert \xb9 from WIN866 to KOI8-R failed -convert \xba from WIN866 to KOI8-R failed -convert \xbb from WIN866 to KOI8-R failed -convert \xbc from WIN866 to KOI8-R failed -convert \xbd from WIN866 to KOI8-R produced \xbd -convert \xbe from WIN866 to KOI8-R failed -convert \xbf from WIN866 to KOI8-R failed -convert \xc0 from WIN866 to KOI8-R failed -convert \xc1 from WIN866 to KOI8-R failed -convert \xc2 from WIN866 to KOI8-R failed -convert \xc3 from WIN866 to KOI8-R failed -convert \xc4 from WIN866 to KOI8-R failed -convert \xc5 from WIN866 to KOI8-R failed -convert \xc6 from WIN866 to KOI8-R failed -convert \xc7 from WIN866 to KOI8-R failed -convert \xc8 from WIN866 to KOI8-R failed -convert \xc9 from WIN866 to KOI8-R failed -convert \xca from WIN866 to KOI8-R failed -convert \xcb from WIN866 to KOI8-R failed -convert \xcc from WIN866 to KOI8-R failed -convert \xcd from WIN866 to KOI8-R failed -convert \xce from WIN866 to KOI8-R failed -convert \xcf from WIN866 to KOI8-R failed -convert \xd0 from WIN866 to KOI8-R failed -convert \xd1 from WIN866 to KOI8-R failed -convert \xd2 from WIN866 to KOI8-R failed -convert \xd3 from WIN866 to KOI8-R failed -convert \xd4 from WIN866 to KOI8-R failed -convert \xd5 from WIN866 to KOI8-R failed -convert \xd6 from WIN866 to KOI8-R failed -convert \xd7 from WIN866 to KOI8-R failed -convert \xd8 from WIN866 to KOI8-R failed -convert \xd9 from WIN866 to KOI8-R failed -convert \xda from WIN866 to KOI8-R failed -convert \xdb from WIN866 to KOI8-R failed -convert \xdc from WIN866 to KOI8-R failed -convert \xdd from WIN866 to KOI8-R failed -convert \xde from WIN866 to KOI8-R failed -convert \xdf from WIN866 to KOI8-R failed +convert \xb0 from WIN866 to KOI8-R produced \x90 +convert \xb1 from WIN866 to KOI8-R produced \x91 +convert \xb2 from WIN866 to KOI8-R produced \x92 +convert \xb3 from WIN866 to KOI8-R produced \x81 +convert \xb4 from WIN866 to KOI8-R produced \x87 +convert \xb5 from WIN866 to KOI8-R produced \xb2 +convert \xb6 from WIN866 to KOI8-R produced \xb4 +convert \xb7 from WIN866 to KOI8-R produced \xa7 +convert \xb8 from WIN866 to KOI8-R produced \xa6 +convert \xb9 from WIN866 to KOI8-R produced \xb5 +convert \xba from WIN866 to KOI8-R produced \xa1 +convert \xbb from WIN866 to KOI8-R produced \xa8 +convert \xbc from WIN866 to KOI8-R produced \xae +convert \xbd from WIN866 to KOI8-R produced \xad +convert \xbe from WIN866 to KOI8-R produced \xac +convert \xbf from WIN866 to KOI8-R produced \x83 +convert \xc0 from WIN866 to KOI8-R produced \x84 +convert \xc1 from WIN866 to KOI8-R produced \x89 +convert \xc2 from WIN866 to KOI8-R produced \x88 +convert \xc3 from WIN866 to KOI8-R produced \x86 +convert \xc4 from WIN866 to KOI8-R produced \x80 +convert \xc5 from WIN866 to KOI8-R produced \x8a +convert \xc6 from WIN866 to KOI8-R produced \xaf +convert \xc7 from WIN866 to KOI8-R produced \xb0 +convert \xc8 from WIN866 to KOI8-R produced \xab +convert \xc9 from WIN866 to KOI8-R produced \xa5 +convert \xca from WIN866 to KOI8-R produced \xbb +convert \xcb from WIN866 to KOI8-R produced \xb8 +convert \xcc from WIN866 to KOI8-R produced \xb1 +convert \xcd from WIN866 to KOI8-R produced \xa0 +convert \xce from WIN866 to KOI8-R produced \xbe +convert \xcf from WIN866 to KOI8-R produced \xb9 +convert \xd0 from WIN866 to KOI8-R produced \xba +convert \xd1 from WIN866 to KOI8-R produced \xb6 +convert \xd2 from WIN866 to KOI8-R produced \xb7 +convert \xd3 from WIN866 to KOI8-R produced \xaa +convert \xd4 from WIN866 to KOI8-R produced \xa9 +convert \xd5 from WIN866 to KOI8-R produced \xa2 +convert \xd6 from WIN866 to KOI8-R produced \xa4 +convert \xd7 from WIN866 to KOI8-R produced \xbd +convert \xd8 from WIN866 to KOI8-R produced \xbc +convert \xd9 from WIN866 to KOI8-R produced \x85 +convert \xda from WIN866 to KOI8-R produced \x82 +convert \xdb from WIN866 to KOI8-R produced \x8d +convert \xdc from WIN866 to KOI8-R produced \x8c +convert \xdd from WIN866 to KOI8-R produced \x8e +convert \xde from WIN866 to KOI8-R produced \x8f +convert \xdf from WIN866 to KOI8-R produced \x8b convert \xe0 from WIN866 to KOI8-R produced \xd2 convert \xe1 from WIN866 to KOI8-R produced \xd3 convert \xe2 from WIN866 to KOI8-R produced \xd4 @@ -3317,20 +3316,20 @@ convert \xef from WIN866 to KOI8-R produced \xd1 convert \xf0 from WIN866 to KOI8-R produced \xb3 convert \xf1 from WIN866 to KOI8-R produced \xa3 -convert \xf2 from WIN866 to KOI8-R produced \xb4 -convert \xf3 from WIN866 to KOI8-R produced \xa4 -convert \xf4 from WIN866 to KOI8-R produced \xb7 -convert \xf5 from WIN866 to KOI8-R produced \xa7 +convert \xf2 from WIN866 to KOI8-R failed +convert \xf3 from WIN866 to KOI8-R failed +convert \xf4 from WIN866 to KOI8-R failed +convert \xf5 from WIN866 to KOI8-R failed convert \xf6 from WIN866 to KOI8-R failed convert \xf7 from WIN866 to KOI8-R failed -convert \xf8 from WIN866 to KOI8-R produced \xb6 -convert \xf9 from WIN866 to KOI8-R produced \xa6 -convert \xfa from WIN866 to KOI8-R failed -convert \xfb from WIN866 to KOI8-R failed +convert \xf8 from WIN866 to KOI8-R produced \x9c +convert \xf9 from WIN866 to KOI8-R produced \x95 +convert \xfa from WIN866 to KOI8-R produced \x9e +convert \xfb from WIN866 to KOI8-R produced \x96 convert \xfc from WIN866 to KOI8-R failed convert \xfd from WIN866 to KOI8-R failed -convert \xfe from WIN866 to KOI8-R failed -convert \xff from WIN866 to KOI8-R failed +convert \xfe from WIN866 to KOI8-R produced \x94 +convert \xff from WIN866 to KOI8-R produced \x9a convert \x00 from KOI8-R to WIN866 failed convert \x01 from KOI8-R to WIN866 produced \x01 convert \x02 from KOI8-R to WIN866 produced \x02 @@ -3459,70 +3458,69 @@ convert \x7d from KOI8-R to WIN866 produced \x7d convert \x7e from KOI8-R to WIN866 produced \x7e convert \x7f from KOI8-R to WIN866 produced \x7f -convert \x80 from KOI8-R to WIN866 failed -convert \x81 from KOI8-R to WIN866 failed -convert \x82 from KOI8-R to WIN866 failed -convert \x83 from KOI8-R to WIN866 failed -convert \x84 from KOI8-R to WIN866 failed -convert \x85 from KOI8-R to WIN866 failed -convert \x86 from KOI8-R to WIN866 failed -convert \x87 from KOI8-R to WIN866 failed -convert \x88 from KOI8-R to WIN866 failed -convert \x89 from KOI8-R to WIN866 failed -convert \x8a from KOI8-R to WIN866 failed -convert \x8b from KOI8-R to WIN866 failed -convert \x8c from KOI8-R to WIN866 failed -convert \x8d from KOI8-R to WIN866 failed -convert \x8e from KOI8-R to WIN866 failed -convert \x8f from KOI8-R to WIN866 failed -convert \x90 from KOI8-R to WIN866 failed -convert \x91 from KOI8-R to WIN866 failed -convert \x92 from KOI8-R to WIN866 failed +convert \x80 from KOI8-R to WIN866 produced \xc4 +convert \x81 from KOI8-R to WIN866 produced \xb3 +convert \x82 from KOI8-R to WIN866 produced \xda +convert \x83 from KOI8-R to WIN866 produced \xbf +convert \x84 from KOI8-R to WIN866 produced \xc0 +convert \x85 from KOI8-R to WIN866 produced \xd9 +convert \x86 from KOI8-R to WIN866 produced \xc3 +convert \x87 from KOI8-R to WIN866 produced \xb4 +convert \x88 from KOI8-R to WIN866 produced \xc2 +convert \x89 from KOI8-R to WIN866 produced \xc1 +convert \x8a from KOI8-R to WIN866 produced \xc5 +convert \x8b from KOI8-R to WIN866 produced \xdf +convert \x8c from KOI8-R to WIN866 produced \xdc +convert \x8d from KOI8-R to WIN866 produced \xdb +convert \x8e from KOI8-R to WIN866 produced \xdd +convert \x8f from KOI8-R to WIN866 produced \xde +convert \x90 from KOI8-R to WIN866 produced \xb0 +convert \x91 from KOI8-R to WIN866 produced \xb1 +convert \x92 from KOI8-R to WIN866 produced \xb2 convert \x93 from KOI8-R to WIN866 failed -convert \x94 from KOI8-R to WIN866 failed -convert \x95 from KOI8-R to WIN866 failed -convert \x96 from KOI8-R to WIN866 failed +convert \x94 from KOI8-R to WIN866 produced \xfe +convert \x95 from KOI8-R to WIN866 produced \xf9 +convert \x96 from KOI8-R to WIN866 produced \xfb convert \x97 from KOI8-R to WIN866 failed convert \x98 from KOI8-R to WIN866 failed convert \x99 from KOI8-R to WIN866 failed -convert \x9a from KOI8-R to WIN866 failed +convert \x9a from KOI8-R to WIN866 produced \xff convert \x9b from KOI8-R to WIN866 failed -convert \x9c from KOI8-R to WIN866 failed +convert \x9c from KOI8-R to WIN866 produced \xf8 convert \x9d from KOI8-R to WIN866 failed -convert \x9e from KOI8-R to WIN866 failed +convert \x9e from KOI8-R to WIN866 produced \xfa convert \x9f from KOI8-R to WIN866 failed -convert \xa0 from KOI8-R to WIN866 failed -convert \xa1 from KOI8-R to WIN866 failed -convert \xa2 from KOI8-R to WIN866 failed +convert \xa0 from KOI8-R to WIN866 produced \xcd +convert \xa1 from KOI8-R to WIN866 produced \xba +convert \xa2 from KOI8-R to WIN866 produced \xd5 convert \xa3 from KOI8-R to WIN866 produced \xf1 -convert \xa4 from KOI8-R to WIN866 produced \xf3 -convert \xa5 from KOI8-R to WIN866 failed -convert \xa6 from KOI8-R to WIN866 produced \xf9 -convert \xa7 from KOI8-R to WIN866 produced \xf5 -convert \xa8 from KOI8-R to WIN866 failed -convert \xa9 from KOI8-R to WIN866 failed -convert \xaa from KOI8-R to WIN866 failed -convert \xab from KOI8-R to WIN866 failed -convert \xac from KOI8-R to WIN866 failed -convert \xad from KOI8-R to WIN866 produced \xad -convert \xad from KOI8-R to WIN866 reverse conversion produced \xce -convert \xae from KOI8-R to WIN866 failed -convert \xaf from KOI8-R to WIN866 failed -convert \xb0 from KOI8-R to WIN866 failed -convert \xb1 from KOI8-R to WIN866 failed -convert \xb2 from KOI8-R to WIN866 failed +convert \xa4 from KOI8-R to WIN866 produced \xd6 +convert \xa5 from KOI8-R to WIN866 produced \xc9 +convert \xa6 from KOI8-R to WIN866 produced \xb8 +convert \xa7 from KOI8-R to WIN866 produced \xb7 +convert \xa8 from KOI8-R to WIN866 produced \xbb +convert \xa9 from KOI8-R to WIN866 produced \xd4 +convert \xaa from KOI8-R to WIN866 produced \xd3 +convert \xab from KOI8-R to WIN866 produced \xc8 +convert \xac from KOI8-R to WIN866 produced \xbe +convert \xad from KOI8-R to WIN866 produced \xbd +convert \xae from KOI8-R to WIN866 produced \xbc +convert \xaf from KOI8-R to WIN866 produced \xc6 +convert \xb0 from KOI8-R to WIN866 produced \xc7 +convert \xb1 from KOI8-R to WIN866 produced \xcc +convert \xb2 from KOI8-R to WIN866 produced \xb5 convert \xb3 from KOI8-R to WIN866 produced \xf0 -convert \xb4 from KOI8-R to WIN866 produced \xf2 -convert \xb5 from KOI8-R to WIN866 failed -convert \xb6 from KOI8-R to WIN866 produced \xf8 -convert \xb7 from KOI8-R to WIN866 produced \xf4 -convert \xb8 from KOI8-R to WIN866 failed -convert \xb9 from KOI8-R to WIN866 failed -convert \xba from KOI8-R to WIN866 failed -convert \xbb from KOI8-R to WIN866 failed -convert \xbc from KOI8-R to WIN866 failed -convert \xbd from KOI8-R to WIN866 produced \xbd -convert \xbe from KOI8-R to WIN866 failed +convert \xb4 from KOI8-R to WIN866 produced \xb6 +convert \xb5 from KOI8-R to WIN866 produced \xb9 +convert \xb6 from KOI8-R to WIN866 produced \xd1 +convert \xb7 from KOI8-R to WIN866 produced \xd2 +convert \xb8 from KOI8-R to WIN866 produced \xcb +convert \xb9 from KOI8-R to WIN866 produced \xcf +convert \xba from KOI8-R to WIN866 produced \xd0 +convert \xbb from KOI8-R to WIN866 produced \xca +convert \xbc from KOI8-R to WIN866 produced \xd8 +convert \xbd from KOI8-R to WIN866 produced \xd7 +convert \xbe from KOI8-R to WIN866 produced \xce convert \xbf from KOI8-R to WIN866 failed convert \xc0 from KOI8-R to WIN866 produced \xee convert \xc1 from KOI8-R to WIN866 produced \xa0 @@ -3777,7 +3775,7 @@ convert \xba from WIN866 to WIN1251 failed convert \xbb from WIN866 to WIN1251 failed convert \xbc from WIN866 to WIN1251 failed -convert \xbd from WIN866 to WIN1251 produced \xa5 +convert \xbd from WIN866 to WIN1251 failed convert \xbe from WIN866 to WIN1251 failed convert \xbf from WIN866 to WIN1251 failed convert \xc0 from WIN866 to WIN1251 failed @@ -3834,16 +3832,16 @@ convert \xf3 from WIN866 to WIN1251 produced \xba convert \xf4 from WIN866 to WIN1251 produced \xaf convert \xf5 from WIN866 to WIN1251 produced \xbf -convert \xf6 from WIN866 to WIN1251 failed -convert \xf7 from WIN866 to WIN1251 failed -convert \xf8 from WIN866 to WIN1251 produced \xb2 -convert \xf9 from WIN866 to WIN1251 produced \xb3 -convert \xfa from WIN866 to WIN1251 failed +convert \xf6 from WIN866 to WIN1251 produced \xa1 +convert \xf7 from WIN866 to WIN1251 produced \xa2 +convert \xf8 from WIN866 to WIN1251 produced \xb0 +convert \xf9 from WIN866 to WIN1251 failed +convert \xfa from WIN866 to WIN1251 produced \xb7 convert \xfb from WIN866 to WIN1251 failed -convert \xfc from WIN866 to WIN1251 failed -convert \xfd from WIN866 to WIN1251 failed +convert \xfc from WIN866 to WIN1251 produced \xb9 +convert \xfd from WIN866 to WIN1251 produced \xa4 convert \xfe from WIN866 to WIN1251 failed -convert \xff from WIN866 to WIN1251 failed +convert \xff from WIN866 to WIN1251 produced \xa0 convert \x00 from WIN1251 to WIN866 failed convert \x01 from WIN1251 to WIN866 produced \x01 convert \x02 from WIN1251 to WIN866 produced \x02 @@ -4004,12 +4002,12 @@ convert \x9d from WIN1251 to WIN866 failed convert \x9e from WIN1251 to WIN866 failed convert \x9f from WIN1251 to WIN866 failed -convert \xa0 from WIN1251 to WIN866 failed -convert \xa1 from WIN1251 to WIN866 failed -convert \xa2 from WIN1251 to WIN866 failed +convert \xa0 from WIN1251 to WIN866 produced \xff +convert \xa1 from WIN1251 to WIN866 produced \xf6 +convert \xa2 from WIN1251 to WIN866 produced \xf7 convert \xa3 from WIN1251 to WIN866 failed -convert \xa4 from WIN1251 to WIN866 failed -convert \xa5 from WIN1251 to WIN866 produced \xbd +convert \xa4 from WIN1251 to WIN866 produced \xfd +convert \xa5 from WIN1251 to WIN866 failed convert \xa6 from WIN1251 to WIN866 failed convert \xa7 from WIN1251 to WIN866 failed convert \xa8 from WIN1251 to WIN866 produced \xf0 @@ -4020,17 +4018,16 @@ convert \xad from WIN1251 to WIN866 failed convert \xae from WIN1251 to WIN866 failed convert \xaf from WIN1251 to WIN866 produced \xf4 -convert \xb0 from WIN1251 to WIN866 failed +convert \xb0 from WIN1251 to WIN866 produced \xf8 convert \xb1 from WIN1251 to WIN866 failed -convert \xb2 from WIN1251 to WIN866 produced \xf8 -convert \xb3 from WIN1251 to WIN866 produced \xf9 -convert \xb4 from WIN1251 to WIN866 produced \xad -convert \xb4 from WIN1251 to WIN866 reverse conversion produced \xed +convert \xb2 from WIN1251 to WIN866 failed +convert \xb3 from WIN1251 to WIN866 failed +convert \xb4 from WIN1251 to WIN866 failed convert \xb5 from WIN1251 to WIN866 failed convert \xb6 from WIN1251 to WIN866 failed -convert \xb7 from WIN1251 to WIN866 failed +convert \xb7 from WIN1251 to WIN866 produced \xfa convert \xb8 from WIN1251 to WIN866 produced \xf1 -convert \xb9 from WIN1251 to WIN866 failed +convert \xb9 from WIN1251 to WIN866 produced \xfc convert \xba from WIN1251 to WIN866 produced \xf3 convert \xbb from WIN1251 to WIN866 failed convert \xbc from WIN1251 to WIN866 failed @@ -4261,22 +4258,22 @@ convert \x9d from ISO-8859-5 to WIN1251 failed convert \x9e from ISO-8859-5 to WIN1251 failed convert \x9f from ISO-8859-5 to WIN1251 failed -convert \xa0 from ISO-8859-5 to WIN1251 failed +convert \xa0 from ISO-8859-5 to WIN1251 produced \xa0 convert \xa1 from ISO-8859-5 to WIN1251 produced \xa8 -convert \xa2 from ISO-8859-5 to WIN1251 failed -convert \xa3 from ISO-8859-5 to WIN1251 failed -convert \xa4 from ISO-8859-5 to WIN1251 failed -convert \xa5 from ISO-8859-5 to WIN1251 failed -convert \xa6 from ISO-8859-5 to WIN1251 failed -convert \xa7 from ISO-8859-5 to WIN1251 failed -convert \xa8 from ISO-8859-5 to WIN1251 failed -convert \xa9 from ISO-8859-5 to WIN1251 failed -convert \xaa from ISO-8859-5 to WIN1251 failed -convert \xab from ISO-8859-5 to WIN1251 failed -convert \xac from ISO-8859-5 to WIN1251 failed -convert \xad from ISO-8859-5 to WIN1251 failed -convert \xae from ISO-8859-5 to WIN1251 failed -convert \xaf from ISO-8859-5 to WIN1251 failed +convert \xa2 from ISO-8859-5 to WIN1251 produced \x80 +convert \xa3 from ISO-8859-5 to WIN1251 produced \x81 +convert \xa4 from ISO-8859-5 to WIN1251 produced \xaa +convert \xa5 from ISO-8859-5 to WIN1251 produced \xbd +convert \xa6 from ISO-8859-5 to WIN1251 produced \xb2 +convert \xa7 from ISO-8859-5 to WIN1251 produced \xaf +convert \xa8 from ISO-8859-5 to WIN1251 produced \xa3 +convert \xa9 from ISO-8859-5 to WIN1251 produced \x8a +convert \xaa from ISO-8859-5 to WIN1251 produced \x8c +convert \xab from ISO-8859-5 to WIN1251 produced \x8e +convert \xac from ISO-8859-5 to WIN1251 produced \x8d +convert \xad from ISO-8859-5 to WIN1251 produced \xad +convert \xae from ISO-8859-5 to WIN1251 produced \xa1 +convert \xaf from ISO-8859-5 to WIN1251 produced \x8f convert \xb0 from ISO-8859-5 to WIN1251 produced \xc0 convert \xb1 from ISO-8859-5 to WIN1251 produced \xc1 convert \xb2 from ISO-8859-5 to WIN1251 produced \xc2 @@ -4341,22 +4338,22 @@ convert \xed from ISO-8859-5 to WIN1251 produced \xfd convert \xee from ISO-8859-5 to WIN1251 produced \xfe convert \xef from ISO-8859-5 to WIN1251 produced \xff -convert \xf0 from ISO-8859-5 to WIN1251 failed +convert \xf0 from ISO-8859-5 to WIN1251 produced \xb9 convert \xf1 from ISO-8859-5 to WIN1251 produced \xb8 -convert \xf2 from ISO-8859-5 to WIN1251 failed -convert \xf3 from ISO-8859-5 to WIN1251 failed -convert \xf4 from ISO-8859-5 to WIN1251 failed -convert \xf5 from ISO-8859-5 to WIN1251 failed -convert \xf6 from ISO-8859-5 to WIN1251 failed -convert \xf7 from ISO-8859-5 to WIN1251 failed -convert \xf8 from ISO-8859-5 to WIN1251 failed -convert \xf9 from ISO-8859-5 to WIN1251 failed -convert \xfa from ISO-8859-5 to WIN1251 failed -convert \xfb from ISO-8859-5 to WIN1251 failed -convert \xfc from ISO-8859-5 to WIN1251 failed -convert \xfd from ISO-8859-5 to WIN1251 failed -convert \xfe from ISO-8859-5 to WIN1251 failed -convert \xff from ISO-8859-5 to WIN1251 failed +convert \xf2 from ISO-8859-5 to WIN1251 produced \x90 +convert \xf3 from ISO-8859-5 to WIN1251 produced \x83 +convert \xf4 from ISO-8859-5 to WIN1251 produced \xba +convert \xf5 from ISO-8859-5 to WIN1251 produced \xbe +convert \xf6 from ISO-8859-5 to WIN1251 produced \xb3 +convert \xf7 from ISO-8859-5 to WIN1251 produced \xbf +convert \xf8 from ISO-8859-5 to WIN1251 produced \xbc +convert \xf9 from ISO-8859-5 to WIN1251 produced \x9a +convert \xfa from ISO-8859-5 to WIN1251 produced \x9c +convert \xfb from ISO-8859-5 to WIN1251 produced \x9e +convert \xfc from ISO-8859-5 to WIN1251 produced \x9d +convert \xfd from ISO-8859-5 to WIN1251 produced \xa7 +convert \xfe from ISO-8859-5 to WIN1251 produced \xa2 +convert \xff from ISO-8859-5 to WIN1251 produced \x9f convert \x00 from WIN1251 to ISO-8859-5 failed convert \x01 from WIN1251 to ISO-8859-5 produced \x01 convert \x02 from WIN1251 to ISO-8859-5 produced \x02 @@ -4485,23 +4482,23 @@ convert \x7d from WIN1251 to ISO-8859-5 produced \x7d convert \x7e from WIN1251 to ISO-8859-5 produced \x7e convert \x7f from WIN1251 to ISO-8859-5 produced \x7f -convert \x80 from WIN1251 to ISO-8859-5 failed -convert \x81 from WIN1251 to ISO-8859-5 failed +convert \x80 from WIN1251 to ISO-8859-5 produced \xa2 +convert \x81 from WIN1251 to ISO-8859-5 produced \xa3 convert \x82 from WIN1251 to ISO-8859-5 failed -convert \x83 from WIN1251 to ISO-8859-5 failed +convert \x83 from WIN1251 to ISO-8859-5 produced \xf3 convert \x84 from WIN1251 to ISO-8859-5 failed convert \x85 from WIN1251 to ISO-8859-5 failed convert \x86 from WIN1251 to ISO-8859-5 failed convert \x87 from WIN1251 to ISO-8859-5 failed convert \x88 from WIN1251 to ISO-8859-5 failed convert \x89 from WIN1251 to ISO-8859-5 failed -convert \x8a from WIN1251 to ISO-8859-5 failed +convert \x8a from WIN1251 to ISO-8859-5 produced \xa9 convert \x8b from WIN1251 to ISO-8859-5 failed -convert \x8c from WIN1251 to ISO-8859-5 failed -convert \x8d from WIN1251 to ISO-8859-5 failed -convert \x8e from WIN1251 to ISO-8859-5 failed -convert \x8f from WIN1251 to ISO-8859-5 failed -convert \x90 from WIN1251 to ISO-8859-5 failed +convert \x8c from WIN1251 to ISO-8859-5 produced \xaa +convert \x8d from WIN1251 to ISO-8859-5 produced \xac +convert \x8e from WIN1251 to ISO-8859-5 produced \xab +convert \x8f from WIN1251 to ISO-8859-5 produced \xaf +convert \x90 from WIN1251 to ISO-8859-5 produced \xf2 convert \x91 from WIN1251 to ISO-8859-5 failed convert \x92 from WIN1251 to ISO-8859-5 failed convert \x93 from WIN1251 to ISO-8859-5 failed @@ -4511,44 +4508,44 @@ convert \x97 from WIN1251 to ISO-8859-5 failed convert \x98 from WIN1251 to ISO-8859-5 failed convert \x99 from WIN1251 to ISO-8859-5 failed -convert \x9a from WIN1251 to ISO-8859-5 failed +convert \x9a from WIN1251 to ISO-8859-5 produced \xf9 convert \x9b from WIN1251 to ISO-8859-5 failed -convert \x9c from WIN1251 to ISO-8859-5 failed -convert \x9d from WIN1251 to ISO-8859-5 failed -convert \x9e from WIN1251 to ISO-8859-5 failed -convert \x9f from WIN1251 to ISO-8859-5 failed -convert \xa0 from WIN1251 to ISO-8859-5 failed -convert \xa1 from WIN1251 to ISO-8859-5 failed -convert \xa2 from WIN1251 to ISO-8859-5 failed -convert \xa3 from WIN1251 to ISO-8859-5 failed +convert \x9c from WIN1251 to ISO-8859-5 produced \xfa +convert \x9d from WIN1251 to ISO-8859-5 produced \xfc +convert \x9e from WIN1251 to ISO-8859-5 produced \xfb +convert \x9f from WIN1251 to ISO-8859-5 produced \xff +convert \xa0 from WIN1251 to ISO-8859-5 produced \xa0 +convert \xa1 from WIN1251 to ISO-8859-5 produced \xae +convert \xa2 from WIN1251 to ISO-8859-5 produced \xfe +convert \xa3 from WIN1251 to ISO-8859-5 produced \xa8 convert \xa4 from WIN1251 to ISO-8859-5 failed convert \xa5 from WIN1251 to ISO-8859-5 failed convert \xa6 from WIN1251 to ISO-8859-5 failed -convert \xa7 from WIN1251 to ISO-8859-5 failed +convert \xa7 from WIN1251 to ISO-8859-5 produced \xfd convert \xa8 from WIN1251 to ISO-8859-5 produced \xa1 convert \xa9 from WIN1251 to ISO-8859-5 failed -convert \xaa from WIN1251 to ISO-8859-5 failed +convert \xaa from WIN1251 to ISO-8859-5 produced \xa4 convert \xab from WIN1251 to ISO-8859-5 failed convert \xac from WIN1251 to ISO-8859-5 failed -convert \xad from WIN1251 to ISO-8859-5 failed +convert \xad from WIN1251 to ISO-8859-5 produced \xad convert \xae from WIN1251 to ISO-8859-5 failed -convert \xaf from WIN1251 to ISO-8859-5 failed +convert \xaf from WIN1251 to ISO-8859-5 produced \xa7 convert \xb0 from WIN1251 to ISO-8859-5 failed convert \xb1 from WIN1251 to ISO-8859-5 failed -convert \xb2 from WIN1251 to ISO-8859-5 failed -convert \xb3 from WIN1251 to ISO-8859-5 failed +convert \xb2 from WIN1251 to ISO-8859-5 produced \xa6 +convert \xb3 from WIN1251 to ISO-8859-5 produced \xf6 convert \xb4 from WIN1251 to ISO-8859-5 failed convert \xb5 from WIN1251 to ISO-8859-5 failed convert \xb6 from WIN1251 to ISO-8859-5 failed convert \xb7 from WIN1251 to ISO-8859-5 failed convert \xb8 from WIN1251 to ISO-8859-5 produced \xf1 -convert \xb9 from WIN1251 to ISO-8859-5 failed -convert \xba from WIN1251 to ISO-8859-5 failed +convert \xb9 from WIN1251 to ISO-8859-5 produced \xf0 +convert \xba from WIN1251 to ISO-8859-5 produced \xf4 convert \xbb from WIN1251 to ISO-8859-5 failed -convert \xbc from WIN1251 to ISO-8859-5 failed -convert \xbd from WIN1251 to ISO-8859-5 failed -convert \xbe from WIN1251 to ISO-8859-5 failed -convert \xbf from WIN1251 to ISO-8859-5 failed +convert \xbc from WIN1251 to ISO-8859-5 produced \xf8 +convert \xbd from WIN1251 to ISO-8859-5 produced \xa5 +convert \xbe from WIN1251 to ISO-8859-5 produced \xf5 +convert \xbf from WIN1251 to ISO-8859-5 produced \xf7 convert \xc0 from WIN1251 to ISO-8859-5 produced \xb0 convert \xc1 from WIN1251 to ISO-8859-5 produced \xb1 convert \xc2 from WIN1251 to ISO-8859-5 produced \xb2 @@ -4773,21 +4770,21 @@ convert \x9d from ISO-8859-5 to WIN866 failed convert \x9e from ISO-8859-5 to WIN866 failed convert \x9f from ISO-8859-5 to WIN866 failed -convert \xa0 from ISO-8859-5 to WIN866 failed +convert \xa0 from ISO-8859-5 to WIN866 produced \xff convert \xa1 from ISO-8859-5 to WIN866 produced \xf0 convert \xa2 from ISO-8859-5 to WIN866 failed convert \xa3 from ISO-8859-5 to WIN866 failed -convert \xa4 from ISO-8859-5 to WIN866 failed +convert \xa4 from ISO-8859-5 to WIN866 produced \xf2 convert \xa5 from ISO-8859-5 to WIN866 failed convert \xa6 from ISO-8859-5 to WIN866 failed -convert \xa7 from ISO-8859-5 to WIN866 failed +convert \xa7 from ISO-8859-5 to WIN866 produced \xf4 convert \xa8 from ISO-8859-5 to WIN866 failed convert \xa9 from ISO-8859-5 to WIN866 failed convert \xaa from ISO-8859-5 to WIN866 failed convert \xab from ISO-8859-5 to WIN866 failed convert \xac from ISO-8859-5 to WIN866 failed convert \xad from ISO-8859-5 to WIN866 failed -convert \xae from ISO-8859-5 to WIN866 failed +convert \xae from ISO-8859-5 to WIN866 produced \xf6 convert \xaf from ISO-8859-5 to WIN866 failed convert \xb0 from ISO-8859-5 to WIN866 produced \x80 convert \xb1 from ISO-8859-5 to WIN866 produced \x81 @@ -4853,21 +4850,21 @@ convert \xed from ISO-8859-5 to WIN866 produced \xed convert \xee from ISO-8859-5 to WIN866 produced \xee convert \xef from ISO-8859-5 to WIN866 produced \xef -convert \xf0 from ISO-8859-5 to WIN866 failed +convert \xf0 from ISO-8859-5 to WIN866 produced \xfc convert \xf1 from ISO-8859-5 to WIN866 produced \xf1 convert \xf2 from ISO-8859-5 to WIN866 failed convert \xf3 from ISO-8859-5 to WIN866 failed -convert \xf4 from ISO-8859-5 to WIN866 failed +convert \xf4 from ISO-8859-5 to WIN866 produced \xf3 convert \xf5 from ISO-8859-5 to WIN866 failed convert \xf6 from ISO-8859-5 to WIN866 failed -convert \xf7 from ISO-8859-5 to WIN866 failed +convert \xf7 from ISO-8859-5 to WIN866 produced \xf5 convert \xf8 from ISO-8859-5 to WIN866 failed convert \xf9 from ISO-8859-5 to WIN866 failed convert \xfa from ISO-8859-5 to WIN866 failed convert \xfb from ISO-8859-5 to WIN866 failed convert \xfc from ISO-8859-5 to WIN866 failed convert \xfd from ISO-8859-5 to WIN866 failed -convert \xfe from ISO-8859-5 to WIN866 failed +convert \xfe from ISO-8859-5 to WIN866 produced \xf7 convert \xff from ISO-8859-5 to WIN866 failed convert \x00 from WIN866 to ISO-8859-5 failed convert \x01 from WIN866 to ISO-8859-5 produced \x01 @@ -5111,20 +5108,20 @@ convert \xef from WIN866 to ISO-8859-5 produced \xef convert \xf0 from WIN866 to ISO-8859-5 produced \xa1 convert \xf1 from WIN866 to ISO-8859-5 produced \xf1 -convert \xf2 from WIN866 to ISO-8859-5 failed -convert \xf3 from WIN866 to ISO-8859-5 failed -convert \xf4 from WIN866 to ISO-8859-5 failed -convert \xf5 from WIN866 to ISO-8859-5 failed -convert \xf6 from WIN866 to ISO-8859-5 failed -convert \xf7 from WIN866 to ISO-8859-5 failed +convert \xf2 from WIN866 to ISO-8859-5 produced \xa4 +convert \xf3 from WIN866 to ISO-8859-5 produced \xf4 +convert \xf4 from WIN866 to ISO-8859-5 produced \xa7 +convert \xf5 from WIN866 to ISO-8859-5 produced \xf7 +convert \xf6 from WIN866 to ISO-8859-5 produced \xae +convert \xf7 from WIN866 to ISO-8859-5 produced \xfe convert \xf8 from WIN866 to ISO-8859-5 failed convert \xf9 from WIN866 to ISO-8859-5 failed convert \xfa from WIN866 to ISO-8859-5 failed convert \xfb from WIN866 to ISO-8859-5 failed -convert \xfc from WIN866 to ISO-8859-5 failed +convert \xfc from WIN866 to ISO-8859-5 produced \xf0 convert \xfd from WIN866 to ISO-8859-5 failed convert \xfe from WIN866 to ISO-8859-5 failed -convert \xff from WIN866 to ISO-8859-5 failed +convert \xff from WIN866 to ISO-8859-5 produced \xa0 convert \x00 from WIN1250 to ISO-8859-2 failed convert \x01 from WIN1250 to ISO-8859-2 produced \x01 convert \x02 from WIN1250 to ISO-8859-2 produced \x02 @@ -5253,34 +5250,34 @@ convert \x7d from WIN1250 to ISO-8859-2 produced \x7d convert \x7e from WIN1250 to ISO-8859-2 produced \x7e convert \x7f from WIN1250 to ISO-8859-2 produced \x7f -convert \x80 from WIN1250 to ISO-8859-2 produced \x80 -convert \x81 from WIN1250 to ISO-8859-2 produced \x81 -convert \x82 from WIN1250 to ISO-8859-2 produced \x82 -convert \x83 from WIN1250 to ISO-8859-2 produced \x83 -convert \x84 from WIN1250 to ISO-8859-2 produced \x84 -convert \x85 from WIN1250 to ISO-8859-2 produced \x85 -convert \x86 from WIN1250 to ISO-8859-2 produced \x86 -convert \x87 from WIN1250 to ISO-8859-2 produced \x87 -convert \x88 from WIN1250 to ISO-8859-2 produced \x88 -convert \x89 from WIN1250 to ISO-8859-2 produced \x89 +convert \x80 from WIN1250 to ISO-8859-2 failed +convert \x81 from WIN1250 to ISO-8859-2 failed +convert \x82 from WIN1250 to ISO-8859-2 failed +convert \x83 from WIN1250 to ISO-8859-2 failed +convert \x84 from WIN1250 to ISO-8859-2 failed +convert \x85 from WIN1250 to ISO-8859-2 failed +convert \x86 from WIN1250 to ISO-8859-2 failed +convert \x87 from WIN1250 to ISO-8859-2 failed +convert \x88 from WIN1250 to ISO-8859-2 failed +convert \x89 from WIN1250 to ISO-8859-2 failed convert \x8a from WIN1250 to ISO-8859-2 produced \xa9 -convert \x8b from WIN1250 to ISO-8859-2 produced \x8b +convert \x8b from WIN1250 to ISO-8859-2 failed convert \x8c from WIN1250 to ISO-8859-2 produced \xa6 convert \x8d from WIN1250 to ISO-8859-2 produced \xab convert \x8e from WIN1250 to ISO-8859-2 produced \xae convert \x8f from WIN1250 to ISO-8859-2 produced \xac -convert \x90 from WIN1250 to ISO-8859-2 produced \x90 -convert \x91 from WIN1250 to ISO-8859-2 produced \x91 -convert \x92 from WIN1250 to ISO-8859-2 produced \x92 -convert \x93 from WIN1250 to ISO-8859-2 produced \x93 -convert \x94 from WIN1250 to ISO-8859-2 produced \x94 -convert \x95 from WIN1250 to ISO-8859-2 produced \x95 -convert \x96 from WIN1250 to ISO-8859-2 produced \x96 -convert \x97 from WIN1250 to ISO-8859-2 produced \x97 -convert \x98 from WIN1250 to ISO-8859-2 produced \x98 -convert \x99 from WIN1250 to ISO-8859-2 produced \x99 +convert \x90 from WIN1250 to ISO-8859-2 failed +convert \x91 from WIN1250 to ISO-8859-2 failed +convert \x92 from WIN1250 to ISO-8859-2 failed +convert \x93 from WIN1250 to ISO-8859-2 failed +convert \x94 from WIN1250 to ISO-8859-2 failed +convert \x95 from WIN1250 to ISO-8859-2 failed +convert \x96 from WIN1250 to ISO-8859-2 failed +convert \x97 from WIN1250 to ISO-8859-2 failed +convert \x98 from WIN1250 to ISO-8859-2 failed +convert \x99 from WIN1250 to ISO-8859-2 failed convert \x9a from WIN1250 to ISO-8859-2 produced \xb9 -convert \x9b from WIN1250 to ISO-8859-2 produced \x9b +convert \x9b from WIN1250 to ISO-8859-2 failed convert \x9c from WIN1250 to ISO-8859-2 produced \xb6 convert \x9d from WIN1250 to ISO-8859-2 produced \xbb convert \x9e from WIN1250 to ISO-8859-2 produced \xbe @@ -5509,34 +5506,34 @@ convert \x7d from ISO-8859-2 to WIN1250 produced \x7d convert \x7e from ISO-8859-2 to WIN1250 produced \x7e convert \x7f from ISO-8859-2 to WIN1250 produced \x7f -convert \x80 from ISO-8859-2 to WIN1250 produced \x80 -convert \x81 from ISO-8859-2 to WIN1250 produced \x81 -convert \x82 from ISO-8859-2 to WIN1250 produced \x82 -convert \x83 from ISO-8859-2 to WIN1250 produced \x83 -convert \x84 from ISO-8859-2 to WIN1250 produced \x84 -convert \x85 from ISO-8859-2 to WIN1250 produced \x85 -convert \x86 from ISO-8859-2 to WIN1250 produced \x86 -convert \x87 from ISO-8859-2 to WIN1250 produced \x87 -convert \x88 from ISO-8859-2 to WIN1250 produced \x88 -convert \x89 from ISO-8859-2 to WIN1250 produced \x89 +convert \x80 from ISO-8859-2 to WIN1250 failed +convert \x81 from ISO-8859-2 to WIN1250 failed +convert \x82 from ISO-8859-2 to WIN1250 failed +convert \x83 from ISO-8859-2 to WIN1250 failed +convert \x84 from ISO-8859-2 to WIN1250 failed +convert \x85 from ISO-8859-2 to WIN1250 failed +convert \x86 from ISO-8859-2 to WIN1250 failed +convert \x87 from ISO-8859-2 to WIN1250 failed +convert \x88 from ISO-8859-2 to WIN1250 failed +convert \x89 from ISO-8859-2 to WIN1250 failed convert \x8a from ISO-8859-2 to WIN1250 failed -convert \x8b from ISO-8859-2 to WIN1250 produced \x8b +convert \x8b from ISO-8859-2 to WIN1250 failed convert \x8c from ISO-8859-2 to WIN1250 failed convert \x8d from ISO-8859-2 to WIN1250 failed convert \x8e from ISO-8859-2 to WIN1250 failed convert \x8f from ISO-8859-2 to WIN1250 failed -convert \x90 from ISO-8859-2 to WIN1250 produced \x90 -convert \x91 from ISO-8859-2 to WIN1250 produced \x91 -convert \x92 from ISO-8859-2 to WIN1250 produced \x92 -convert \x93 from ISO-8859-2 to WIN1250 produced \x93 -convert \x94 from ISO-8859-2 to WIN1250 produced \x94 -convert \x95 from ISO-8859-2 to WIN1250 produced \x95 -convert \x96 from ISO-8859-2 to WIN1250 produced \x96 -convert \x97 from ISO-8859-2 to WIN1250 produced \x97 -convert \x98 from ISO-8859-2 to WIN1250 produced \x98 -convert \x99 from ISO-8859-2 to WIN1250 produced \x99 +convert \x90 from ISO-8859-2 to WIN1250 failed +convert \x91 from ISO-8859-2 to WIN1250 failed +convert \x92 from ISO-8859-2 to WIN1250 failed +convert \x93 from ISO-8859-2 to WIN1250 failed +convert \x94 from ISO-8859-2 to WIN1250 failed +convert \x95 from ISO-8859-2 to WIN1250 failed +convert \x96 from ISO-8859-2 to WIN1250 failed +convert \x97 from ISO-8859-2 to WIN1250 failed +convert \x98 from ISO-8859-2 to WIN1250 failed +convert \x99 from ISO-8859-2 to WIN1250 failed convert \x9a from ISO-8859-2 to WIN1250 failed -convert \x9b from ISO-8859-2 to WIN1250 produced \x9b +convert \x9b from ISO-8859-2 to WIN1250 failed convert \x9c from ISO-8859-2 to WIN1250 failed convert \x9d from ISO-8859-2 to WIN1250 failed convert \x9e from ISO-8859-2 to WIN1250 failed
On Fri, Nov 27, 2015 at 8:54 PM, Tatsuo Ishii <ishii@postgresql.org> wrote: > I explain why the manual editing is necessary. > > One of the most famous problems with Unicode is "wave dash" > (U+301C). According the Unicode consortium's Unicode/SJIS map, it > corresponds to 0x8160 of Shift_JIS. Unfortunately this was a mistake > in Unicode (the glyph of Shift_JIS and Unicode is slightly different - > looks like to be rotated in 90 degrees of wave dash in vertical > scripting. Probably they did not understand the Japanese vertical > writing at that time). So later on the Unicode consortium decided to > add another "wave dash" as U+FF5E which has a correct glyph of "wave > dash". However since Unicode already decided that U+301C corresponds > to 0x8160 of Shift_JIS, there's no Shift_JIS code corresponding to > U+FF5E. Unlike Unicode's definition, Microsoft defines that 0x8160 > (wave dash) corresponds to U+FF5E. This is widely used in Japan. So I > decided to hire this for "wave dash". i.e. > > 0x8160 -> U+FF5E (sjis_to_utf8.map) > > U+301C -> 0x8160 (utf_to_sjis.map) > U+FF5E -> 0x8160 (utf_to_sjis.map) > > Another problem is vendor extension. > > There are several standards for SJIS and EUC_JP in Japan. There is a > standard "Shift_JIS" defined by Japanese Government (probably the > Unicode consortium's map can be based on this, but I need to > verify). However several major vendors include IBM, NEC added their > own additional characters to Shift_JIS and they are widely used in > Japan. Unfortunately they are not compatible. So as a compromise I and > other developers decided to "merge" NEC and IBM extension part and > added to Shift_JIS. Same thing can be said to EUC_JP. > > In short, there are number of reasons we cannot simply import the > consortium's mapping regarding SJIS (and EUC_JP). I haven't seen a response to this point, but it seems important. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> On Fri, Nov 27, 2015 at 8:54 PM, Tatsuo Ishii <ishii@postgresql.org> wrote:
>> In short, there are number of reasons we cannot simply import the
>> consortium's mapping regarding SJIS (and EUC_JP).
> I haven't seen a response to this point, but it seems important.
I'll defer to Tatsuo-san concerning whether the Far Eastern conversions
should act the way they do. However, I still think the Cyrillic and
Latin-2 conversions are broken. There is no reason to question the
Unicode consortium's mappings in those cases AFAIK, and even if somebody
wants to, our current tables fail to round-trip some characters, which
is surely wrong. (See the "inconsistent reverse conversion" complaints
in the test output in <32464.1448742262@sss.pgh.pa.us>.)
Regardless of that, it's dismaying that we have files in our tree that
claim to produce our mapping tables from authoritative sources, when in
fact those tables were not produced in that way. This is a documentation
failure even if you consider the actual conversion behavior valid.
regards, tom lane