Hackers,
I just had a discussion on IRC about unicode normalization in
PostgreSQL. Apparently there is not support for it, currently. Andrew
Gierth points out that it's part of the SQL spec to support it, though:
> RhodiumToad:e.g. NORMALIZE(foo,NFC,len)
> justatheory:Oh, just a function then, really.
> RhodiumToad:where the normal form can be any of NFC, NFD, NFKC, NFKD
> RhodiumToad:except that the normal form is an identifier, not a string
> RhodiumToad:also the normal form and length are optional
> RhodiumToad:so NORMALIZE(foo) is equivalent to NORMALIZE(foo,NFC)
I looked around and found the Public Software Group's utf8proc
project, which even includes some PostgreSQL support (not, alas, for
normalization). It has an MIT-licensed C library that offers these
functions:
> uint8_t utf8proc_NFD(uint8_t str)
>
> Returns a pointer to newly allocated memory of a NFD normalized
> version of the null-terminated stringstr.
>
> uint8_t utf8proc_NFC(uint8_t str)
>
> Returns a pointer to newly allocated memory of a NFC normalized
> version of the null-terminated stringstr.
>
> uint8_t utf8proc_NFKD(uint8_t str)
>
> Returns a pointer to newly allocated memory of a NFKD normalized
> version of the null-terminated stringstr.
>
> uint8_t utf8proc_NFKC(uint8_t str)
>
> Returns a pointer to newly allocated memory of a NFKC normalized
> version of the null-terminated stringstr.
Anyone got any interest in porting these functions to PostgreSQL? I
guess the parser would need to be updated to support the use of
identifiers in the NORMALIZE() function, but otherwise it should be a
fairly straight-forward port for an experienced C coder, no?
Best,
David