Out of memory condition

Hi,

I've facing an out of memory condition after running SLONY several hours to get a 1TB database with about 23,000 tables replicated. The error occurs after about 50% of the tables were replicated.

Most of the 48GB memory is being used for file system cache but for some reason the initial copy of one table performed by SLONY abended due to an out of memory condition. The table that was being transferred at the moment of the abend has two text columns.

After the OOM condition is raised, "select *" of that specific table also returns out of memory condition.

I guess postgresql is trying to perform an atomic allocation (those which cannot wait for reclaim) to get a continues memory area and is failing due to memory fragmentation.

My idea to prevent this issue is to reserve 500MB of free storage for atomic allocations using vm.min_free_kbytes = 500000000 in the /etc/sysctl.conf.

Is this a good approach to solve it?

Another question: is it safe to flush file system cache using these steps:

1) Shutdown postgresql
2) sync
3) echo 1 > /proc/sys/vm/drop_caches;
4) Startup postgresql

Some data about the issue:

SLONY error:
2014-12-01 12:14:56 BRST ERROR  remoteWorkerThread_1: copy to stdout on provider - PGRES_FATAL_ERROR ERROR:  out of memory
DETAIL:  Failed on request of size 123410655.

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.3 (Santiago)

# uname -m
x86_64

# free
             total       used       free     shared    buffers     cached
Mem:      49422076   49038348     383728          0     268488   47520476
-/+ buffers/cache:    1249384   48172692
Swap:     16777208          0   16777208

# cat /proc/meminfo | grep Commit
CommitLimit:    41488244 kB
Committed_AS:     689312 kB

# /sbin/sysctl vm.min_free_kbytes
vm.min_free_kbytes = 135168

After SLONY gets the out of memory condition, select * of the table also does not work:
FiscalWeb=# select * from "8147_spunico"."sincdc";
ERROR:  out of memory
DETAIL:  Failed on request of size 268435456.

Backup of the table using pg_dump also gives out of memory condition.

Buddyinfo indicates memory fragmentation after getting out of memory condition:
# cat /proc/buddyinfo
Node 0, zone      DMA      3      2      2      3      2      1      1      0      1      0      3
Node 0, zone    DMA32  94091  69426  30367   7531    996    126      8      0      0      1      0
Node 0, zone   Normal   6840     23      0      0      0      0      0      0      0      0      1
Node 1, zone   Normal    730    338    159     93     44     26     11      9      3      1      3
Node 2, zone   Normal     68    535    309    144     60     18     13     12     32     29      7
Node 3, zone   Normal 319246 341233 173115  52602   5989    646    232     63      8      3      1

postgres=# select version();
                                                  version                                                 
-----------------------------------------------------------------------------------------------------------
 PostgreSQL 8.3.21 on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.6 20120305 (Red Hat 4.4.6-4)
(1 row)


Thank you!

Carlos Reimer