Re: TOAST (was: BLOB)

Tom Lane wrote:

> wieck@debis.com (Jan Wieck) writes:
> >     To share some info with the other developers:
> >     I  just  successfully  stored the entire PG sources (~10M) in
> >     one table.  Stored are all 936  .[chyl]  files.
>
> Cool!  Sounds like you are making great progress.
   At least I think so :-)


> >     Anyway, TOAST is on it's way. And as soon as we go  for  7.1,
> >     I'll  need  alot  of  help  to  make  all  our existing types
> >     toastable, add  administrative  utility  commands  and  teach
> >     pg_dump to deal with all that.
>
> Question: pg_dump shouldn't need to know about toasted tuples as
> such, should it?  I'm guessing that what you are thinking of is
> adding administration capabilities to control whether particular
> columns are toast-able, and then pg_dump would need to know about
> saving and loading those control fields.  But toasting a particular
> field value ought to be invisible to applications...
   Answer:  Right!  The  only  change from a clients PoV is that   attribute values can be bigger now.
   So it's only to teach pg_dump to correctly  restore  all  the   possible  tuning.   For  now  there  is  an
attstorage(char)   attribute telling in the TupleDesc, what can be done  with  a   particular value in a HeapTuple.
Themeanings are:
 
   'p'   don't  touch, either DBA doesn't want or the type isn't         capable of dealing with it.   'e'   never try
tocompress, if needed store external as  is.   'x'   try  to compress and if needed store compression result
external.  'm'   try to keep in main  tuple  as  long  as  possible,  if         needed handle like 'x'.
 
   If  a  tuple doesn't fit into a block, the toaster performs 4   loops over the attributes, each one  stops
immediately when   the tuple fits.
 
   1.  Inline compress storage 'x' attributes.
   2.  Move   storage   'x'  and  'e'  attributes  to  secondary       relation.
   3.  Inline compress storage 'm' attributes.
   4.  Move storage 'm' attributes to secondary relation.
   My plan now is to add a default storage value to pg_type.  At   CREATE  TABLE  time, a secondary relation plus it's
indexare   automatically  created  if  attributes  of  the   table   are   toastable.  This is questionable, because
it'llcost 2 inodes   and 16K per table, but  we  can  change  it  later  and  have   special  admin  commands to deal
withit. The toaster doesn't   jump in if a relation doesn't have a secondary  one.  But  it   will  expand values from
otherrelations in that case (INSERT   ... SELECT).
 
   With the pg_type entry telling what to do by default, we  can   work on all our types one by one, and enable them at
thetime   all their type specific functions can handle it.
 
   Finally, I'd like to give  the  secondary  relations  another   relkind,  so they are ignored by vacuum, and then
vacuumthem   explicitly while the LOCK on the master table is still  held.
 


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #