Hi hackers!
There were several discussions where the limitations of the existing TOAST pointers
were mentioned [1], [2] and [3] and from time to time this topic appears in other places.
We proposed a fresh approach to the TOAST mechanics in [2], but unfortunately the
patch was met quite unfriendly, and after several iterations was rejected, although we
still have hopes for it and have several very promising features based on it.
Anyway, the old TOAST pointer is also the cause of problems like [4], and this part of
the PostgreSQL screams to be revised and improved.
The TOAST begins with the pointer to the externalized value - the TOAST Pointer, which
is very limited in means of storing data, and all TOAST improvements require revision
of this Pointer structure. So we decided to ask the community for thoughts and ideas on
how to rework this pointer.
The TOAST Pointer (varatt_external structure) stores 4 fields:
[varlena header][<4b - original data size><4b - size in TOAST table><4b - TOAST table OID><4b - ID of chunk>]
In [2] we proposed the new Custom TOAST pointer structure where main feature is
extensibility:
[varlena header][<2b - total size of the TOAST pointer><4b size of original data><4b - OID of algorithm used for TOASTing><variable length field used for storing any custom data>]
where Custom TOAST Pointer is distinguished from Regular one by va_flag field which
is a part of varlena header, so new pointer format does not interfere with the old (regular) one.
The first field is necessary because the Custom TOAST pointer has variable length due to the
tail used for inline storage, and original value size is used by the Executor. The third field could
be a subject for discussion.
Thoughts? Objections?