Allocate hash join files in a separate memory context
Should a hash join exceed memory limit, the hashtable is split up into
multiple batches. The number of batches is doubled each time a given
batch is determined not to fit in memory. Each batch file is allocated
with a block-sized buffer for buffering tuples and parallel hash join
has additional sharedtuplestore accessor buffers.
In some pathological cases requiring a lot of batches, often with skewed
data, bad stats, or very large datasets, users can run out-of-memory
solely from the memory overhead of all the batch files' buffers.
Batch files were allocated in the ExecutorState memory context, making
it very hard to identify when this batch explosion was the source of an
OOM. This commit allocates the batch files in a dedicated memory
context, making it easier to identify the cause of an OOM and work to
avoid it.
Based on initial draft by Tomas Vondra, with significant reworks and
improvements by Jehan-Guillaume de Rorthais.
Author: Jehan-Guillaume de Rorthais <jgdr@dalibo.com>
Author: Tomas Vondra <tomas.vondra@enterprisedb.com>
Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
Discussion: https://postgr.es/m/20190421114618.z3mpgmimc3rmubi4@development
Discussion: https://postgr.es/m/20230504193006.1b5b9622%40karst#273020ff4061fc7a2fbb1ba96b281f17
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/8c4040edf456d9241816176eacb79e4d9a0034fc
Modified Files
--------------
src/backend/executor/nodeHash.c | 46 ++++++++++++++++++++++---------
src/backend/executor/nodeHashjoin.c | 31 +++++++++++++++++----
src/backend/utils/sort/sharedtuplestore.c | 8 ++++++
src/include/executor/hashjoin.h | 30 ++++++++++++++------
src/include/executor/nodeHashjoin.h | 2 +-
5 files changed, 88 insertions(+), 29 deletions(-)