While testing something on spgist I found that at certain point while inserting in spgist it is going for doPickSplit, but even after split is is not able to find a place to insert a tuple and it keeping going in that loop infinitely it seems and finally error out with OOM because in this loop we are continuously allocating memory for the tuple in temp context but since we are never coming out of the loop it is erroring out with OOM.
My first idea is that this is the case when index tuple doesn't fit into one index page. As INCLUDED columns are added as is the tuple can not be made shorter by prefix-stripping. Seems we should check every index tuple length to fit the page before its insertion. Will see the code little bit later.