Обсуждение: "ERROR: could not open relation with OID 16391" error was encountered when reindexing

Hi,

> This issue has been reported in the <pgsql-bugs> list at the link below, but has not received a reply.
> https://www.postgresql.org/message-id/18286-f6273332500c2a62%40postgresql.org
> Hopefully to get some response from kernel hackers, thanks!
>
> Hi,
> When reindex the partitioned table's index and the drop index are executed concurrently, we may encounter the error
"couldnot open relation with OID”. 
>
> The reconstruction of the partitioned table's index is completed in multiple transactions and can be simply
summarizedinto the following steps: 
> 1. Obtain the oids of all partition indexes in the ReindexPartitions function, and then commit the transaction to
releaseall locks. 
> 2. Reindex each index in turn
>    2.1 Start a new transaction
>    2.2 Check whether the index still exists
>    2.3 Call the reindex_index function to complete the index rebuilding work
>    2.4 Submit transaction
>
> There is no lock between steps 2.2 and 2.3 to protect the heap table and index from being deleted, so whether the
heaptable still exists is determined in the reindex_index function, but the index is not checked. 
>
> One fix I can think of is: after successfully opening the heap table in reindex_index, check again whether the index
stillexists, Something like this: 
> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
> index 143fae01eb..21777ec98c 100644
> --- a/src/backend/catalog/index.c
> +++ b/src/backend/catalog/index.c
> @@ -3594,6 +3594,17 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence,
>         if (!heapRelation)
>                 return;
>
> +       /*
> +        * Before opening the index, check if the index relation still exists.
> +        * If index relation is gone, leave.
> +        */
> +       if (params->options & REINDEXOPT_MISSING_OK != 0 &&
> +               !SearchSysCacheExists1(RELOID, ObjectIdGetDatum(indexId)))
> +       {
> +               table_close(heapRelation, NoLock);
> +               return;
> +       }
> +
>         /*
>          * Switch to the table owner's userid, so that any index functions are run
>          * as that user.  Also lock down security-restricted operations and
>
> The above analysis is based on the latest master branch.
>
> I'm not sure if my idea is reasonable, I hope you can give me some suggestions. Thanks.

Any chance you could provide minimal steps to reproduce the issue on
an empty PG instance, ideally as a script? That's going to be helpful
to reproduce / investigate the issue and also make sure that it's
fixed.

--
Best regards,
Aleksander Alekseev

I have provided a python script in the attachment to minimize the reproduction of the issue.

"=?ISO-8859-1?B?ZmVpY2hhbmdob25n?=" <feichanghong@qq.com> writes:
> 2. session1 reindex and gdb break at index.c:3585

This is extremely nonspecific, as line numbers in our code change
constantly.  Please quote a chunk of code surrounding that
and indicate which line you are trying to stop at.

            regards, tom lane

On Wed, Jan 17, 2024 at 12:54:26AM +0800, feichanghong wrote:
>> This is extremely nonspecific, as line numbers in our code change
>> constantly.  Please quote a chunk of code surrounding that
>> and indicate which line you are trying to stop at.
>
> Thanks for the suggestion, I've refined the steps below to reproduce:

Yeah, thanks for the steps.  I am not surprised that there are still a
few holes in this area.  CONCURRENTLY can behave differently depending
on the step where the old index is getting opened.

For this specific job, I have always wanted a try_index_open() that
would attempt to open the index with a relkind check, perhaps we could
introduce one and reuse it here?
--
Michael

On Wed, Jan 17, 2024 at 04:03:46PM +0800, feichanghong wrote:
> It has been verified that the patch in the attachment can solve the
> above problems. I sincerely look forward to your suggestions!

Thanks for the patch.  I have completely forgotten to update this
thread.  Except for a few comments and names that I've tweaked, this
was OK, so applied and backpatched after splitting things into two:
- One commit for try_index_open().
- Second commit for the fix in reindex_index().

I've looked at the concurrent paths as well, and even if these involve
more relations opened we maintain a session lock on the parent
relations that we manipulate, so I could not see a pattern where the
index would be dropped and where we'd try to open it.  Now, there are
cases where it is possible to deadlock for the concurrent paths, but
that's not new: schema or database level reindexes can also hit that.

This is one of these areas where tests are hard to write now because
we want to stop operations at specific points but we cannot.
--
Michael