TOAST (was: BLOB)

> Hello
>
> I would like to know
> - how to create under psql a BLOB field in sql language
> - how store a value (and not a file) in sql language
> - the maximum size of the BLOB field
   For  now  (and  7.0),  there  is no way to have rows >8K in a   regular table.
   But I'm actually making alot  of  progress  in  breaking  it.   Seems we'll get rid of this restriction in 7.1.
   To share some info with the other developers:
   I  just  successfully  stored the entire PG sources (~10M) in   one table.  Stored are all 936  .[chyl]  files.
The tables   schema is:
 
       sources (         s_path    text,         s_data    clob       )
   clob  is  a  user  defined,  very  simple varsize datatype, I   created for testing. The entire source  tree
inserts in  23   seconds.  If  I  cut  off all sources at 7K and insert into a   text field it needs 8 seconds.  Not
bad,because  the  amount   of data shrinks <4M due to the cutoff.
 
   Reading the data back shows similar results, 0.8 secs vs. 0.3   secs.
   And the toaster already uses a  heap+index  to  store  values   external.
   This  all done with full toasting - i.e. try to compress, and   only  if  it  still  doesn't  fit  move  already
compressed  attribute  into secondary relation. As a side effect, the 10M   of sources only need 3.4M main and 2.6M
secondaryheap.
 
   I  experienced  some  trouble  with   occational   "Deadlock"   detections,  when  using  multiple processes
hammeringon the   toaster. But I was able to reproduce the same  error  with  a   plain  7.0  and  "text"  attributes
(onlyfar less frequent).   Seems there's  something  wrong  in  our  deadlock  detection   algorithm.
 
   Another  thing  is,  that  ISTM  that index's never shrink on   vacuum. One requirement of TOAST  is,  that  the
secondaries  tables  index-OID is remembered in the external reference, so   the ref can be  passed  around  in  the
entire backend  and   whenever needed, it's real value can be fetched quickly by an   index scan. Thus, it's not
possibleto drop/recreate an index   on that right now. Seems we need a "REBUILD INDEX" utility or   the like.
 
   Anyway, TOAST is on it's way. And as soon as we go  for  7.1,   I'll  need  alot  of  help  to  make  all  our
existingtypes   toastable, add  administrative  utility  commands  and  teach   pg_dump to deal with all that.
 


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #