Обсуждение: PostgreSql: Canceled on conflict out to old pivot

Hi,

> On 28. Nov 2023, at 06:41, Wirch, Eduard <eduard.w@smart-host.com> wrote:
>
> 
>
> :
>
> ERROR: could not serialize access due to read/write dependencies among transactions
>   Detail: Reason code: Canceled on identification as a pivot, with conflict out to old committed transaction
61866959.
>
> There is a variation of the error:
>
> PSQLException: ERROR: could not serialize access due to read/write dependencies among transactions
>   Detail: Reason code: Canceled on conflict out to old pivot 61940806.

Could you show explain analyze output for those queries which fail with such errors?


>
>

On 28/11/2023 07:41, Wirch, Eduard wrote:
> ERROR: could not serialize access due to read/write dependencies among 
> transactions
>    Detail: Reason code: Canceled on identification as a pivot, with 
> conflict out to old committed transaction 61866959.
> 
> There is a variation of the error:
> 
> PSQLException: ERROR: could not serialize access due to read/write 
> dependencies among transactions
>    Detail: Reason code: Canceled on conflict out to old pivot 61940806.

Both of these errors are coming from CheckForSerializableConflictOut(), 
and are indeed variations of the same kind of conflict.

> We're logging the id, begin and end of every transaction. Transaction 
> 61940806 was committed without errors. The transaction responsible for 
> the above error was started 40min later (and failed immediately). With 
> 61866959 it is even more extreme: the first conflict error occurred 2.5h 
> after 61866959 was committed.

Weird indeed. There is only one caller of 
CheckForSerializableConflictOut(), and it does this:

>     /*
>      * Find top level xid.  Bail out if xid is too early to be a conflict, or
>      * if it's our own xid.
>      */
>     if (TransactionIdEquals(xid, GetTopTransactionIdIfAny()))
>         return;
>     xid = SubTransGetTopmostTransaction(xid);
>     if (TransactionIdPrecedes(xid, TransactionXmin))
>         return;
> 
>     CheckForSerializableConflictOut(relation, xid, snapshot);

That check with TransactionXmin is very clear: if 'xid' precedes the 
xmin of the current transaction, IOW if there were no transactions with 
'xid' or older running when the current transcaction started, 
CheckForSerializableConflictOut() is not called.

> The DB table access pattern is too complex to lay out here. There are 
> like 20 tables that are read/written to. Transactions are usually short 
> living. The longest transaction that could occur is 1 min long. My 
> understanding of serializable isolation is that only overlapping 
> transactions can conflict. I can be pretty sure that in the above cases 
> there is no single transaction, which overlaps with 61940806 and with 
> the failing transaction 40 min later.

I hate to drill on this, but are you very sure about that? I don't see 
how this could happen if there are no long-running transactions. Maybe a 
forgotten two-phase commit transaction? A transaction in a different 
database? A developer who did "begin;" in psql and went for lunch?

> Such long running transactions 
> would cause different types of errors in our system ("out of shared 
> memory", "You might need to increase max_pred_locks_per_transaction").

I don't see why that would necessarily be the case, unless it's 
something very specific to your application.

> Why does PostgreSql detect a conflict with a transaction which was 
> committed more than 1h before? Can there be a long dependency chain 
> between many short running transactions? Does the high load prevent 
> Postgres from doing some clean up?

The dependencies don't chain like that, but there is a system of 
"summarizing" old transactions to limit the shared memory usage. When a 
transaction has dependencies on other transactions, we track those 
dependencies in shared memory. But if we run short on the space reserved 
for that, we summarize the dependencies, losing granularity. We lose 
information of which relations/pages/tuples the xid accessed and which 
transactions exactly it had a dependency on. That is safe, but can cause 
false positives.

The amount of shared memory reserved for tracking the dependencies is 
determined by max_pred_locks_per_transaction, so you could try 
increasing that to reduce those false positives, even if you never get 
the "out of shared memory" error.

-- 
Heikki Linnakangas
Neon (https://neon.tech)

On 30/11/2023 18:24, Wirch, Eduard wrote:
>  > > The longest transaction that could occur is 1 min long.
>> I hate to drill on this, but are you very sure about that? A transaction 
> in a different database?
> 
> Don't be sorry for that, drilling down is important. ;) It took me so 
> long to reply because I had to prepare the information carefully. You're 
> right, on that day I observed the behavior, there were indeed long 
> running transactions in different DBs!

A-ha! :-D

> My understanding of serializable isolation is that only transactions
> which can somehow affect each other can conflict. It should be clear
> for PostgreSql, that transactions belonging to different databases
> cannot affect each other. Why do they cause serializable conflicts?

When the system runs low on the memory reserved to track the potential 
conflicts, it "summarizes" old transactions and writes them to disk. The 
summarization loses a lot of information: all we actually store on disk 
is the commit sequence number of the earliest "out-conflicting" 
transaction. We don't store the database oid that the transaction ran in.

The whole SSI mechanism is conservative so that you can get false 
serialization errors even when there is no actual problem. This is just 
an extreme case of that.

Perhaps we should keep more information in the on-disk summary format. 
Preserving the database OID would make a lot of sense, it would be only 
4 bytes extra per transaction. But we don't preserve it today.

> If you want something visual, I prepared a SO question with similar 
> content like this mail, but added an image of the tx flow: 
> https://stackoverflow.com/questions/77544821/postgresql-canceled-on-conflict-out-to-old-pivot
<https://stackoverflow.com/questions/77544821/postgresql-canceled-on-conflict-out-to-old-pivot>

Nice graphs!

-- 
Heikki Linnakangas
Neon (https://neon.tech)

Heikki Linnakangas <hlinnaka@iki.fi> writes:
> On 30/11/2023 18:24, Wirch, Eduard wrote:
>> My understanding of serializable isolation is that only transactions
>> which can somehow affect each other can conflict. It should be clear
>> for PostgreSql, that transactions belonging to different databases
>> cannot affect each other. Why do they cause serializable conflicts?

On what grounds do you assert that?  Operations on shared catalogs
are visible across databases.  Admittedly they can't be written by
ordinary DML, and I'm not sure that we make any promises about DDL
writes honoring serializability.  But I'm unwilling to add
"optimizations" that assume that that will never happen.

            regards, tom lane

Hi,

On 2023-11-30 18:51:35 -0500, Tom Lane wrote:
> On what grounds do you assert that?  Operations on shared catalogs
> are visible across databases.  Admittedly they can't be written by
> ordinary DML, and I'm not sure that we make any promises about DDL
> writes honoring serializability.  But I'm unwilling to add
> "optimizations" that assume that that will never happen.

I'd say the issue is more that it's quite expensive to collect the
information. I tried in the past to make the xmin computation in
GetSnapshotData() be database specific, but it quickly shows in profiles, and
GetSnapshotData() unfortunately is really performance / scalability critical.

If that weren't the case, we could check a shared horizon for shared tables,
and a non-shared horizon otherwise.

In some cases we can compute a "narrower" horizon when it's worth the cost,
but quite often we lack the necessary data, because various backends have
stored the "global" xmin in the procarray.

Greetings,

Andres Freund