Re: MaxOffsetNumber for Table AMs

On Tue, May 4, 2021 at 5:40 PM Andres Freund <andres@anarazel.de> wrote:
> What does the deduplication actually require from tids? Isn't it just
> that you need to be able to compare tids?

It's hard to know for sure what is essential to the design, and what
can be discarded. Though I can say for sure that it depends on there
being cheap to compare TIDs.

> Hm. It doesn't seems look like that'd be all that hard to adjust / that
> it'd be meaningfully easier to support only one other type of tid width.

It depends on equi-sized TIDs within a posting list, too -- see
"Posting list splits" from the nbtree README for some idea of what I
mean. The reason that nbtree deduplication could be enabled by default
(the reason why it has very little downside) is because there are
virtually no special cases, and because the WAL overhead for posting
list splits is so low (it's only a tiny bit higher than a simple
btinsert()).

Anyway, deduplication could probably still work in about the same way
if there were some sensible limits on how generalized TIDs could work.

> > Obviously there are some ways in which that could be revised if there
> > was a really good reason to do so -- like an actual concrete reason
> > with some clear basis in reality.
>
> The example of indirect indexes has been brought up repeatedly - you
> just didn't respond to it?

I did respond to it -- in detail. I don't see how it will ever be
possible to make indirect indexes work -- at least within anything
like the current framework. And even if I was wrong there, it still
wouldn't mean that the project was particularly worth pursuing.

Whether or not you find my arguments about indirect indexes convincing
is ultimately beside the point. The fact is that I just don't believe
that that's ever going to happen (it is a fact that that's what I
believe). This will discourage me from becoming involved in anything
that touches on how the table AM thinks about TIDs, particularly in
index AMs. As far as I'm concerned this TID stuff is up in the air
(except maybe for something like zheap which is sufficiently close to
heapam that it doesn't matter).

> So far nobody has expressed any expectation of you doing specific work
> in this thread as far as I can see? I certainly didn't intend to. I
> think it's perfectly normal to discuss tradeoffs and disagree about
> them?

That's why I was careful to say "You have no obligation to make me
happy". Of course it's true that I could easily just not get involved
-- that was never my concern.

Here is my concern: I have an obligation to make it clear that I think
that you really ought to straighten out this business with
generalizing TIDs before too long. Not because I say so, but because
it's holding up progress in general. If you aren't getting cooperation
from people who work on indexing (could be somebody else), then
consider the possibility that this business with TIDs and bitmap scans
has a lot to do with it. Most people are not as outspoken as I am.

I'm not at all surprised that this happened, simply because of the
history -- it makes sense. I'm glad that the table AM interface
exists, and it was always going to be something that required
refinement over time. I want the table AM to be successful. If I
didn't care then I would say nothing at all.
-- 
Peter Geoghegan