Re: PostgreSQL fails to convert decomposed utf-8 to other encodings

On 08/06/2014 09:14 AM, Tom Lane wrote:
> We don't actually support "decomposed" utf8; if there is any bug here,
> it's that the input you show isn't rejected.  But I think there was
> some intentional choice to not check \u escapes fully.

Combining characters (i.e. decomposed utf-8 form, for chars where there
is a combined equivalent) are part of utf-8. They're not an optional add-on.

So if Pg doesn't support them, it doesn't fully support utf-8. Which is
fine as far as it goes, but must be documented as a limitation at
minimum. (I'll deal with that).

It also means that you get fun anomalies like:

regress=> SELECT 'á' = 'á';
 ?column?
----------
 f
(1 row)

which is IMO insane.

Not only that, but we can't reject decomposed forms, because they will
already exist in live installs. That'd break dump and reload of such
installs and cause exciting problems with pg_upgrade.

The "we'll just reject part of utf-8" opportunity has flown. It needs to
be documented as a bug in existing versions, and I guess given that I'm
the one complaining I get to see if I can find a sane fix for 9.5...

--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services