Обсуждение: Release notes for February minor releases
I've pushed the first draft for $SUBJECT at https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=ab22eea83169c8d0eb15050ce61cbe3d7dae4de6 Please send comments/corrections by Sunday. regards, tom lane
On Fri, Feb 04, 2022 at 02:58:59PM -0500, Tom Lane wrote: > I've pushed the first draft for $SUBJECT at > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=ab22eea83169c8d0eb15050ce61cbe3d7dae4de6 > > Please send comments/corrections by Sunday. + <para> + Build extended statistics for partitioned tables (Justin Pryzby) + </para> + <para> + A previous bug fix disabled building of extended statistics for + old-style inheritance trees, but it also prevented building them for + partitioned tables, which was an unnecessary restriction. + If you have created statistics objects for partitioned tables, you + may wish to explicitly <command>ANALYZE</command> those tables after + installing this update, rather than waiting for auto-analyze to do it. Since autoanalyze still doesn't process partitioned tables, the last part should be removed. Probably it should say "..you *should* explicitly ANALYZE thse tables..". + <para> + Ignore extended statistics for inheritance trees (Justin Pryzby) + </para> + <para> + A previous bug fix disabled building of extended statistics for + old-style inheritance trees, but any existing statistics data was + not removed, and that data would become more and more out-of-date + over time. Adjust the planner to ignore such data. Extended + statistics for the individual child tables are still built and used, + however. + </para> The issue here isn't that old stats were never updated. For inheritance, they *were* updated with non-inherited stats (for SELECT FROM ONLY). But then "SELECT FROM tbl*" used the stats anyway... + <para> + Fix failure of SP-GiST indexes when indexed column's data type is + binary-compatible with the declared input type of the operator class + (Tom Lane) + </para> maybe: when *the* -- Justin
On Fri, Feb 04, 2022 at 02:58:59PM -0500, Tom Lane wrote: > I've pushed the first draft for $SUBJECT at > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=ab22eea83169c8d0eb15050ce61cbe3d7dae4de6 > > Please send comments/corrections by Sunday. > + <para> > + Fix corruption of HOT chains when a RECENTLY_DEAD tuple changes > + state to fully DEAD during page pruning (Andres Freund) > + </para> > + > + <para> > + This happens when the last transaction that could <quote>see</quote> > + the tuple ends while the page is being pruned. It was then possible > + to remove a tuple that is pointed to by a redirect item elsewhere on > + the page. While that causes no immediate problem, when the item slot > + is re-used by some new tuple, that tuple would be thought to be part > + of the pre-existing HOT chain, creating a form of index corruption. Well, ouchy. > + If this seems to have affected a table, <command>REINDEX</command> > + should repair the damage. I don't think this is very helpful to the reader, are their indexes corrupt or not? If we can't tell them a specific command to run to check, can we at least mention that running amcheck would detect that (if it actually does)? Otherwise, I guess the only way to be sure is to just reindex every index? Or is this at least specific to b-trees? > + <para> > + Enforce standard locking protocol for TOAST table updates, to prevent > + problems with <command>REINDEX CONCURRENTLY</command> (Michael Paquier) > + </para> > + > + <para> > + If applied to a TOAST table or TOAST table's index, <command>REINDEX > + CONCURRENTLY</command> tended to produce a corrupted index. This > + happened because sessions updating TOAST entries released > + their <literal>ROW EXCLUSIVE</literal> locks immediately, rather > + than holding them until transaction commit as all other updates do. > + The fix is to make TOAST updates hold the table lock according to the > + normal rule. Any existing corrupted indexes can be repaired by > + reindexing again. > + </para> > + </listitem> Same, but at least here the admin can cut it down to only those indexes which were added concurrently. Michael -- Michael Banck Teamleiter PostgreSQL-Team Projektleiter Tel.: +49 2166 9901-171 Email: michael.banck@credativ.de credativ GmbH, HRB Mönchengladbach 12080 USt-ID-Nummer: DE204566209 Trompeterallee 108, 41189 Mönchengladbach Geschäftsführung: Dr. Michael Meskes, Geoff Richardson, Peter Lilley Unser Umgang mit personenbezogenen Daten unterliegt folgenden Bestimmungen: https://www.credativ.de/datenschutz
Justin Pryzby <pryzby@telsasoft.com> writes:
> On Fri, Feb 04, 2022 at 02:58:59PM -0500, Tom Lane wrote:
>> Please send comments/corrections by Sunday.
[ assorted comments ]
Thanks for the corrections.
> + A previous bug fix disabled building of extended statistics for
> + old-style inheritance trees, but any existing statistics data was
> + not removed, and that data would become more and more out-of-date
> + over time. Adjust the planner to ignore such data. Extended
> + statistics for the individual child tables are still built and used,
> + however.
> The issue here isn't that old stats were never updated. For inheritance, they
> *were* updated with non-inherited stats (for SELECT FROM ONLY). But then
> "SELECT FROM tbl*" used the stats anyway...
I'm confused about this bit. Are we still building bogus stats for
inheritance parents, or has that stopped?
regards, tom lane
Michael Banck <michael.banck@credativ.de> writes:
> On Fri, Feb 04, 2022 at 02:58:59PM -0500, Tom Lane wrote:
>> + If this seems to have affected a table, <command>REINDEX</command>
>> + should repair the damage.
> I don't think this is very helpful to the reader, are their indexes
> corrupt or not? If we can't tell them a specific command to run to
> check, can we at least mention that running amcheck would detect that
> (if it actually does)? Otherwise, I guess the only way to be sure is to
> just reindex every index? Or is this at least specific to b-trees?
Yeah, I wasn't too happy with that advice either, but it seems like the
best we can do [1]. I don't think we should advise blindly reindexing
your whole installation, because it's a very low-probability bug.
regards, tom lane
[1] https://www.postgresql.org/message-id/20220204041935.gf4uwbdxddq6rffh%40alap3.anarazel.de
On Fri, Feb 04, 2022 at 04:29:19PM -0500, Tom Lane wrote: > > + A previous bug fix disabled building of extended statistics for > > + old-style inheritance trees, but any existing statistics data was > > + not removed, and that data would become more and more out-of-date > > + over time. Adjust the planner to ignore such data. Extended > > + statistics for the individual child tables are still built and used, > > + however. > > > The issue here isn't that old stats were never updated. For inheritance, they > > *were* updated with non-inherited stats (for SELECT FROM ONLY). But then > > "SELECT FROM tbl*" used the stats anyway... > > I'm confused about this bit. Are we still building bogus stats for > inheritance parents, or has that stopped? To make a long story long: - before 859b3003de, an ERROR occurred when a stats object was created on an inheritance parent. - To avoid the error, 859b3003de changed to no longer build "whole tree" stats on the table heirarchy. Non-inheried stats were still collected. - However, the stats were *also* applied to inherited queries (FROM tbl*). 36c4bc6 stops applying stats that shouldn't be applied (and doesn't change their collection during ANALYZE). 20b9fa3 then changes to collect inherited stats on partitioned tables, since they have no non-inherited stats, and since extended stats on partitioned tables were intended to work since v10, and did work until 859b3003de stopped collecting them. In back branches, pg_statistic has inherited stats for partitioned tables, and non-inherited stats otherwise. -- Justin
Justin Pryzby <pryzby@telsasoft.com> writes:
> On Fri, Feb 04, 2022 at 04:29:19PM -0500, Tom Lane wrote:
>> I'm confused about this bit. Are we still building bogus stats for
>> inheritance parents, or has that stopped?
> To make a long story long:
> - before 859b3003de, an ERROR occurred when a stats object was created on an
> inheritance parent.
> - To avoid the error, 859b3003de changed to no longer build "whole tree" stats
> on the table heirarchy. Non-inheried stats were still collected.
> - However, the stats were *also* applied to inherited queries (FROM tbl*).
> 36c4bc6 stops applying stats that shouldn't be applied (and doesn't change
> their collection during ANALYZE).
Got it. So we collected (and still do collect) non-inherited stats
for inheritance parents, but prior to 36c4bc6 those were mistakenly
applied in estimating both inheritance and non-inheritance queries.
Now we only do the latter.
(Since 269b532ae, this is all better in HEAD, but that's not
relevant for the back-branch release notes.)
regards, tom lane
On Fri, Feb 04, 2022 at 04:35:07PM -0500, Tom Lane wrote: > Michael Banck <michael.banck@credativ.de> writes: > > On Fri, Feb 04, 2022 at 02:58:59PM -0500, Tom Lane wrote: > >> + If this seems to have affected a table, <command>REINDEX</command> > >> + should repair the damage. > > > I don't think this is very helpful to the reader, are their indexes > > corrupt or not? If we can't tell them a specific command to run to > > check, can we at least mention that running amcheck would detect that > > (if it actually does)? Otherwise, I guess the only way to be sure is to > > just reindex every index? Or is this at least specific to b-trees? > > Yeah, I wasn't too happy with that advice either, but it seems like the > best we can do [1]. I don't think we should advise blindly reindexing > your whole installation, because it's a very low-probability bug. Right ok. I wonder whether it makes sense to at hint at the low probability then; I guess if you know Postgres well you can deduct from the "when the last transaction that could see the tuple ends while the page is being pruned" that it is a low-probability corner-case, but I fear lots of users will be unable to gauge the chances they got hit by this bug and just blindly assume they are affected (and/or ask around). I just woke up, so I don't have any good wording suggetsions yet. Michael -- Michael Banck Teamleiter PostgreSQL-Team Projektleiter Tel.: +49 2166 9901-171 Email: michael.banck@credativ.de credativ GmbH, HRB Mönchengladbach 12080 USt-ID-Nummer: DE204566209 Trompeterallee 108, 41189 Mönchengladbach Geschäftsführung: Dr. Michael Meskes, Geoff Richardson, Peter Lilley Unser Umgang mit personenbezogenen Daten unterliegt folgenden Bestimmungen: https://www.credativ.de/datenschutz
Hi, On 2022-02-04 22:27:54 +0100, Michael Banck wrote: > On Fri, Feb 04, 2022 at 02:58:59PM -0500, Tom Lane wrote: > > I've pushed the first draft for $SUBJECT at > > > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=ab22eea83169c8d0eb15050ce61cbe3d7dae4de6 > > > > Please send comments/corrections by Sunday. > > > + <para> > > + Fix corruption of HOT chains when a RECENTLY_DEAD tuple changes > > + state to fully DEAD during page pruning (Andres Freund) > > + </para> > > + > > + <para> > > + This happens when the last transaction that could <quote>see</quote> > > + the tuple ends while the page is being pruned. It was then possible > > + to remove a tuple that is pointed to by a redirect item elsewhere on > > + the page. While that causes no immediate problem, when the item slot > > + is re-used by some new tuple, that tuple would be thought to be part > > + of the pre-existing HOT chain, creating a form of index corruption. > > Well, ouchy. I don't think the above description is quite accurate / makes it sound much easier to hit than it is. The time window in which the stars need to align badly is not per-page window, but per-vacuum. And the window is very narrow. Even if that prerequisite was fulfilled, one additionally needs to encounter a pretty rare combination of tids of very specific xid "ages". > > + If this seems to have affected a table, <command>REINDEX</command> > > + should repair the damage. > > I don't think this is very helpful to the reader, are their indexes > corrupt or not? If we can't tell them a specific command to run to > check, can we at least mention that running amcheck would detect that > (if it actually does)? It does not reliably. Unfortunately heap amcheck does not verify HOT chains to any meaningful degree. Nor does btree amcheck check whether index tuples point to matching heap tuples :( > Otherwise, I guess the only way to be sure is to > just reindex every index? Or is this at least specific to b-trees? It's an issue on the heap side, so unfortunately it is not btree specific. Greetings, Andres Freund
Hi, On 2022-02-04 14:58:59 -0500, Tom Lane wrote: > I've pushed the first draft for $SUBJECT at > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=ab22eea83169c8d0eb15050ce61cbe3d7dae4de6 +Author: Andres Freund <andres@anarazel.de> +Branch: master [18b87b201] 2022-01-13 18:13:41 -0800 +Branch: REL_14_STABLE [dad1539ae] 2022-01-14 10:56:12 -0800 +--> + <para> + Fix corruption of HOT chains when a RECENTLY_DEAD tuple changes + state to fully DEAD during page pruning (Andres Freund) + </para> Even if that happens, it's still pretty unlikely to cause corruption - so maybe s/corruption/chance of corruption/? + <para> + This happens when the last transaction that could <quote>see</quote> + the tuple ends while the page is being pruned. The transaction doesn't need to have ended while the page is vacuumed - the horizon needs to have been "refined/updated" while the page is pruned so that a tuple version that was first considered RECENTLY_DEAD is now considered DEAD. Which can only happen if RecentXmin changed after vacuum_set_xid_limits(), which only can happen if catalog snapshot invalidations and other invalidations are processed in vac_open_indexes() and RecentXmin changed since vacuum_set_xid_limits(). Then a page involving tuples in a specific "arrangement" need to be encountered. That's obviously to complicated for the release notes. Trying to make it more understandable I came up with the following, which still does not seem great: This can only happen if transactions, some having performed DDL, commit within a narrow window at the start of VACUUM. If VACUUM then prunes a page containing several tuple version that started to be removable within the aforementioned time window, the bug may cause corruption on that page (but no further pages). A tuple that is pointed to by a redirect item elsewhere on the page can get removed. [...] Greetings, Andres Freund
Andres Freund <andres@anarazel.de> writes:
> That's obviously to complicated for the release notes. Trying to make it more
> understandable I came up with the following, which still does not seem great:
> ...
How do you like this wording?
<para>
Fix corruption of HOT chains when a RECENTLY_DEAD tuple changes
state to fully DEAD during page pruning (Andres Freund)
</para>
<para>
It was possible for <command>VACUUM</command> to remove a
recently-dead tuple while leaving behind a redirect item that
pointed to it. When the tuple's item slot is later re-used by
some new tuple, that tuple would be seen as part of the
pre-existing HOT chain, creating a form of index corruption.
If this has happened, reindexing the table should repair the
damage. However, this is an extremely low-probability scenario,
so we do not recommend reindexing just on the chance that it might
have happened.
</para>
I'm also going to swap the order of this item and the TOAST locking
item, since that one is seeming like it's much more relevant to
most people.
regards, tom lane
Hi, On 2022-02-06 13:09:41 -0500, Tom Lane wrote: > Andres Freund <andres@anarazel.de> writes: > > That's obviously to complicated for the release notes. Trying to make it more > > understandable I came up with the following, which still does not seem great: > > ... > > How do you like this wording? > I'm also going to swap the order of this item and the TOAST locking > item, since that one is seeming like it's much more relevant to > most people. +1 Thanks!
On 2/6/22 1:44 PM, Andres Freund wrote: > Hi, > > On 2022-02-06 13:09:41 -0500, Tom Lane wrote: >> Andres Freund <andres@anarazel.de> writes: >>> That's obviously to complicated for the release notes. Trying to make it more >>> understandable I came up with the following, which still does not seem great: >>> ... >> >> How do you like this wording? > >> I'm also going to swap the order of this item and the TOAST locking >> item, since that one is seeming like it's much more relevant to >> most people. > > +1 I'm working on the release announcement and have been following this thread. Are there steps we can provide to help a user detect that this occurred, even though it's a low-probability? Thanks, Jonathan
Вложения
Hi, On 2022-02-06 14:22:25 -0500, Jonathan S. Katz wrote: > On 2/6/22 1:44 PM, Andres Freund wrote: > I'm working on the release announcement and have been following this thread. > > Are there steps we can provide to help a user detect that this occurred, > even though it's a low-probability? Not realiably currently: https://postgr.es/m/20220205221742.qylnkze5ykc4mabv%40alap3.anarazel.de https://www.postgresql.org/message-id/20220204041935.gf4uwbdxddq6rffh@alap3.anarazel.de Greetings, Andres Freund
On 2/6/22 3:42 PM, Andres Freund wrote: > Hi, > > On 2022-02-06 14:22:25 -0500, Jonathan S. Katz wrote: >> On 2/6/22 1:44 PM, Andres Freund wrote: >> I'm working on the release announcement and have been following this thread. >> >> Are there steps we can provide to help a user detect that this occurred, >> even though it's a low-probability? > > Not realiably currently: > https://postgr.es/m/20220205221742.qylnkze5ykc4mabv%40alap3.anarazel.de > https://www.postgresql.org/message-id/20220204041935.gf4uwbdxddq6rffh@alap3.anarazel.de Thanks. For the release announcement then, it seems like the phrasing would to indicate the probability of hitting this issue is low, but if you are concerned, you should reindex. I should have the first release announcement draft later tonight, which I'll put on its usual separate thread. Thanks, Jonathan