Обсуждение: Out of memory condition
Hi,
I've facing an out of memory condition after running SLONY several hours to get a 1TB database with about 23,000 tables replicated. The error occurs after about 50% of the tables were replicated.
Most of the 48GB memory is being used for file system cache but for some reason the initial copy of one table performed by SLONY abended due to an out of memory condition. The table that was being transferred at the moment of the abend has two text columns.
After the OOM condition is raised, "select *" of that specific table also returns out of memory condition.
I guess postgresql is trying to perform an atomic allocation (those which cannot wait for reclaim) to get a continues memory area and is failing due to memory fragmentation.
My idea to prevent this issue is to reserve 500MB of free storage for atomic allocations using vm.min_free_kbytes = 500000000 in the /etc/sysctl.conf.
Is this a good approach to solve it?
Another question: is it safe to flush file system cache using these steps:
1) Shutdown postgresql
2) sync
3) echo 1 > /proc/sys/vm/drop_caches;
4) Startup postgresql
Some data about the issue:
SLONY error:
2014-12-01 12:14:56 BRST ERROR remoteWorkerThread_1: copy to stdout on provider - PGRES_FATAL_ERROR ERROR: out of memory
DETAIL: Failed on request of size 123410655.
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.3 (Santiago)
# uname -m
x86_64
# free
total used free shared buffers cached
Mem: 49422076 49038348 383728 0 268488 47520476
-/+ buffers/cache: 1249384 48172692
Swap: 16777208 0 16777208
# cat /proc/meminfo | grep Commit
CommitLimit: 41488244 kB
Committed_AS: 689312 kB
# /sbin/sysctl vm.min_free_kbytes
vm.min_free_kbytes = 135168
After SLONY gets the out of memory condition, select * of the table also does not work:
FiscalWeb=# select * from "8147_spunico"."sincdc";
ERROR: out of memory
DETAIL: Failed on request of size 268435456.
Backup of the table using pg_dump also gives out of memory condition.
Buddyinfo indicates memory fragmentation after getting out of memory condition:
# cat /proc/buddyinfo
Node 0, zone DMA 3 2 2 3 2 1 1 0 1 0 3
Node 0, zone DMA32 94091 69426 30367 7531 996 126 8 0 0 1 0
Node 0, zone Normal 6840 23 0 0 0 0 0 0 0 0 1
Node 1, zone Normal 730 338 159 93 44 26 11 9 3 1 3
Node 2, zone Normal 68 535 309 144 60 18 13 12 32 29 7
Node 3, zone Normal 319246 341233 173115 52602 5989 646 232 63 8 3 1
postgres=# select version();
version
-----------------------------------------------------------------------------------------------------------
PostgreSQL 8.3.21 on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.6 20120305 (Red Hat 4.4.6-4)
(1 row)
Thank you!
Carlos Reimer
I've facing an out of memory condition after running SLONY several hours to get a 1TB database with about 23,000 tables replicated. The error occurs after about 50% of the tables were replicated.
Most of the 48GB memory is being used for file system cache but for some reason the initial copy of one table performed by SLONY abended due to an out of memory condition. The table that was being transferred at the moment of the abend has two text columns.
After the OOM condition is raised, "select *" of that specific table also returns out of memory condition.
I guess postgresql is trying to perform an atomic allocation (those which cannot wait for reclaim) to get a continues memory area and is failing due to memory fragmentation.
My idea to prevent this issue is to reserve 500MB of free storage for atomic allocations using vm.min_free_kbytes = 500000000 in the /etc/sysctl.conf.
Is this a good approach to solve it?
Another question: is it safe to flush file system cache using these steps:
1) Shutdown postgresql
2) sync
3) echo 1 > /proc/sys/vm/drop_caches;
4) Startup postgresql
Some data about the issue:
SLONY error:
2014-12-01 12:14:56 BRST ERROR remoteWorkerThread_1: copy to stdout on provider - PGRES_FATAL_ERROR ERROR: out of memory
DETAIL: Failed on request of size 123410655.
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.3 (Santiago)
# uname -m
x86_64
# free
total used free shared buffers cached
Mem: 49422076 49038348 383728 0 268488 47520476
-/+ buffers/cache: 1249384 48172692
Swap: 16777208 0 16777208
# cat /proc/meminfo | grep Commit
CommitLimit: 41488244 kB
Committed_AS: 689312 kB
# /sbin/sysctl vm.min_free_kbytes
vm.min_free_kbytes = 135168
After SLONY gets the out of memory condition, select * of the table also does not work:
FiscalWeb=# select * from "8147_spunico"."sincdc";
ERROR: out of memory
DETAIL: Failed on request of size 268435456.
Backup of the table using pg_dump also gives out of memory condition.
Buddyinfo indicates memory fragmentation after getting out of memory condition:
# cat /proc/buddyinfo
Node 0, zone DMA 3 2 2 3 2 1 1 0 1 0 3
Node 0, zone DMA32 94091 69426 30367 7531 996 126 8 0 0 1 0
Node 0, zone Normal 6840 23 0 0 0 0 0 0 0 0 1
Node 1, zone Normal 730 338 159 93 44 26 11 9 3 1 3
Node 2, zone Normal 68 535 309 144 60 18 13 12 32 29 7
Node 3, zone Normal 319246 341233 173115 52602 5989 646 232 63 8 3 1
postgres=# select version();
version
-----------------------------------------------------------------------------------------------------------
PostgreSQL 8.3.21 on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.6 20120305 (Red Hat 4.4.6-4)
(1 row)
Thank you!
Carlos Reimer
Carlos Henrique Reimer <carlos.reimer@opendb.com.br> writes:
> I've facing an out of memory condition after running SLONY several hours to
> get a 1TB database with about 23,000 tables replicated. The error occurs
> after about 50% of the tables were replicated.
I'd try bringing this up with the Slony crew.
> I guess postgresql is trying to perform an atomic allocation (those which
> cannot wait for reclaim) to get a continues memory area and is failing due
> to memory fragmentation.
This theory has nothing to do with reality. More likely it's just a
garden variety memory leak. If it was an out-of-memory error reported
by Postgres, there should have been a memory statistics dump written in
the postmaster log --- can you find that and post it?
Another possible theory is that you're just looking at lots of memory
needed to hold relcache entries for all 23000 tables :-(. If so there
may not be any easy way around it, except perhaps replicating subsets
of the tables. Unless you can boost the memory available to the backend
--- since this is a 64 bit build, the only reason I can see for
out-of-memory failures would be a restrictive ulimit setting.
> After SLONY gets the out of memory condition, select * of the table also
> does not work:
> FiscalWeb=# select * from "8147_spunico"."sincdc";
> ERROR: out of memory
> DETAIL: Failed on request of size 268435456.
That's odd ... looks more like data corruption than anything else.
Does this happen even in a fresh session? What do you have to do
to get rid of the failure?
> PostgreSQL 8.3.21 on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC)
> 4.4.6 20120305 (Red Hat 4.4.6-4)
You realize of course that this version is years out of support, and that
even if this problem traces to a bug in Postgres, 8.3 is not going to get
fixed.
regards, tom lane
Hi,
Yes, I agree, 8.3 is out of support for a long time and this is the reason we are trying to migrate to 9.3 using SLONY to minimize downtime.
I eliminated the possibility of data corruption as the limit/offset technique indicated different rows each time it was executed. Actually, the failure is still happening and as it is running in a virtual machine, memory size configuration for this virtual machine was increased from 48GB to 64GB and we have scheduled a server shutdown/restart for the next coming weekend in order to try to get rid of the failure.
The replication activity was aborted: SLONY triggers removed, SLONY processes terminated and SLONY schemas removed.
Ulimit output was appended at the end of this note.
Memory statistics dump from postmaster log resulted from a select * from "8147_spunico"."sincdc"; command:
Ulimit output was appended at the end of this note.
Memory statistics dump from postmaster log resulted from a select * from "8147_spunico"."sincdc"; command:
Thank you!
TopMemoryContext: 80800 total in 9 blocks; 4088 free (10 chunks); 76712 used
Operator class cache: 8192 total in 1 blocks; 1680 free (0 chunks); 6512 used
TopTransactionContext: 8192 total in 1 blocks; 7648 free (1 chunks); 544 used
MessageContext: 57344 total in 3 blocks; 40760 free (6 chunks); 16584 used
smgr relation table: 24576 total in 2 blocks; 13904 free (4 chunks); 10672 used
TransactionAbortContext: 32768 total in 1 blocks; 32736 free (0 chunks); 32 used
Portal hash: 8192 total in 1 blocks; 1680 free (0 chunks); 6512 used
PortalMemory: 8192 total in 1 blocks; 7888 free (0 chunks); 304 used
PortalHeapMemory: 1024 total in 1 blocks; 720 free (0 chunks); 304 used
ExecutorState: 381096528 total in 6 blocks; 49856 free (30 chunks); 381046672 used
ExprContext: 0 total in 0 blocks; 0 free (0 chunks); 0 used
Relcache by OID: 24576 total in 2 blocks; 14912 free (3 chunks); 9664 used
CacheMemoryContext: 817392 total in 20 blocks; 230456 free (3 chunks); 586936 used
pg_toast_729119_index: 2048 total in 1 blocks; 608 free (0 chunks); 1440 used
idx_sincdc_situacao: 2048 total in 1 blocks; 752 free (0 chunks); 1296 used
idx_sincdc_esqtab: 2048 total in 1 blocks; 656 free (0 chunks); 1392 used
idx_sincdc_datahoraexp: 2048 total in 1 blocks; 752 free (0 chunks); 1296 used
pk_sincdc: 2048 total in 1 blocks; 752 free (0 chunks); 1296 used
pg_index_indrelid_index: 2048 total in 1 blocks; 704 free (0 chunks); 1344 used
pg_attrdef_adrelid_adnum_index: 2048 total in 1 blocks; 608 free (0 chunks); 1440 used
pg_ts_dict_oid_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
pg_aggregate_fnoid_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
pg_language_name_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
pg_statistic_relid_att_index: 3072 total in 2 blocks; 1600 free (2 chunks); 1472 used
pg_ts_dict_dictname_index: 3072 total in 2 blocks; 1648 free (2 chunks); 1424 used
pg_namespace_nspname_index: 3072 total in 2 blocks; 1696 free (2 chunks); 1376 used
pg_opfamily_oid_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
pg_opclass_oid_index: 3072 total in 2 blocks; 1696 free (2 chunks); 1376 used
pg_ts_parser_prsname_index: 3072 total in 2 blocks; 1648 free (2 chunks); 1424 used
pg_amop_fam_strat_index: 3072 total in 2 blocks; 1384 free (2 chunks); 1688 used
pg_opclass_am_name_nsp_index: 3072 total in 2 blocks; 1624 free (3 chunks); 1448 used
pg_trigger_tgrelid_tgname_index: 3072 total in 2 blocks; 1600 free (2 chunks); 1472 used
pg_cast_source_target_index: 3072 total in 2 blocks; 1648 free (2 chunks); 1424 used
pg_auth_members_role_member_index: 3072 total in 2 blocks; 1648 free (2 chunks); 1424 used
pg_attribute_relid_attnum_index: 3072 total in 2 blocks; 1600 free (2 chunks); 1472 used
pg_ts_config_cfgname_index: 3072 total in 2 blocks; 1648 free (2 chunks); 1424 used
pg_authid_oid_index: 3072 total in 2 blocks; 1696 free (2 chunks); 1376 used
pg_ts_config_oid_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
pg_conversion_default_index: 3072 total in 2 blocks; 1432 free (3 chunks); 1640 used
pg_language_oid_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
pg_enum_oid_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
pg_proc_proname_args_nsp_index: 3072 total in 2 blocks; 1624 free (3 chunks); 1448 used
pg_ts_parser_oid_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
pg_database_oid_index: 3072 total in 2 blocks; 1696 free (2 chunks); 1376 used
pg_conversion_name_nsp_index: 3072 total in 2 blocks; 1648 free (2 chunks); 1424 used
pg_class_relname_nsp_index: 3072 total in 2 blocks; 1600 free (2 chunks); 1472 used
pg_attribute_relid_attnam_index: 3072 total in 2 blocks; 1648 free (2 chunks); 1424 used
pg_class_oid_index: 3072 total in 2 blocks; 1696 free (2 chunks); 1376 used
pg_amproc_fam_proc_index: 3072 total in 2 blocks; 1384 free (2 chunks); 1688 used
pg_operator_oprname_l_r_n_index: 3072 total in 2 blocks; 1432 free (3 chunks); 1640 used
pg_index_indexrelid_index: 3072 total in 2 blocks; 1696 free (2 chunks); 1376 used
pg_type_oid_index: 3072 total in 2 blocks; 1696 free (2 chunks); 1376 used
pg_rewrite_rel_rulename_index: 3072 total in 2 blocks; 1648 free (2 chunks); 1424 used
pg_authid_rolname_index: 3072 total in 2 blocks; 1696 free (2 chunks); 1376 used
pg_auth_members_member_role_index: 3072 total in 2 blocks; 1648 free (2 chunks); 1424 used
pg_enum_typid_label_index: 3072 total in 2 blocks; 1648 free (2 chunks); 1424 used
pg_constraint_oid_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
pg_conversion_oid_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
pg_ts_template_tmplname_index: 3072 total in 2 blocks; 1648 free (2 chunks); 1424 used
pg_ts_config_map_index: 3072 total in 2 blocks; 1624 free (3 chunks); 1448 used
pg_namespace_oid_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
pg_type_typname_nsp_index: 3072 total in 2 blocks; 1648 free (2 chunks); 1424 used
pg_operator_oid_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
pg_amop_opr_fam_index: 3072 total in 2 blocks; 1648 free (2 chunks); 1424 used
pg_proc_oid_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
pg_opfamily_am_name_nsp_index: 3072 total in 2 blocks; 1624 free (3 chunks); 1448 used
pg_ts_template_oid_index: 3072 total in 2 blocks; 1744 free (3 chunks); 1328 used
MdSmgr: 8192 total in 1 blocks; 7648 free (0 chunks); 544 used
LOCALLOCK hash: 24576 total in 2 blocks; 15984 free (5 chunks); 8592 used
Timezones: 53584 total in 2 blocks; 3744 free (0 chunks); 49840 used
ErrorContext: 8192 total in 1 blocks; 8160 free (3 chunks); 32 used
Ulimit output:
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 385725
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 385725
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
On Thu, Dec 11, 2014 at 1:30 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Carlos Henrique Reimer <carlos.reimer@opendb.com.br> writes:
> I've facing an out of memory condition after running SLONY several hours to
> get a 1TB database with about 23,000 tables replicated. The error occurs
> after about 50% of the tables were replicated.
I'd try bringing this up with the Slony crew.
> I guess postgresql is trying to perform an atomic allocation (those which
> cannot wait for reclaim) to get a continues memory area and is failing due
> to memory fragmentation.
This theory has nothing to do with reality. More likely it's just a
garden variety memory leak. If it was an out-of-memory error reported
by Postgres, there should have been a memory statistics dump written in
the postmaster log --- can you find that and post it?
Another possible theory is that you're just looking at lots of memory
needed to hold relcache entries for all 23000 tables :-(. If so there
may not be any easy way around it, except perhaps replicating subsets
of the tables. Unless you can boost the memory available to the backend
--- since this is a 64 bit build, the only reason I can see for
out-of-memory failures would be a restrictive ulimit setting.
> After SLONY gets the out of memory condition, select * of the table also
> does not work:
> FiscalWeb=# select * from "8147_spunico"."sincdc";
> ERROR: out of memory
> DETAIL: Failed on request of size 268435456.
That's odd ... looks more like data corruption than anything else.
Does this happen even in a fresh session? What do you have to do
to get rid of the failure?
> PostgreSQL 8.3.21 on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC)
> 4.4.6 20120305 (Red Hat 4.4.6-4)
You realize of course that this version is years out of support, and that
even if this problem traces to a bug in Postgres, 8.3 is not going to get
fixed.
regards, tom lane
--
Just wondering what slony version you're using?
Slony version is 2.2.3
On Thu, Dec 11, 2014 at 3:29 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
Just wondering what slony version you're using?
--
On Thu, Dec 11, 2014 at 10:30 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
needed to hold relcache entries for all 23000 tables :-(. If so there
may not be any easy way around it, except perhaps replicating subsets
of the tables. Unless you can boost the memory available to the backend
I'd suggest this. Break up your replication into something like 50 sets of 500 tables each, then add one at a time to replication, merging it into the main set. Something like this:
create & replicate set 1.
create & replicate set 2.
merge 2 into 1.
create & replicate set 3.
merge 3 into 1.
repeat until done. this can be scripted.
Given you got about 50% done before it failed, maybe even 4 sets of 6000 tables each may work out.
That was exactly what the process was doing and the out of memory error happened while one of the merges to set 1 was being executed.
On Thu, Dec 11, 2014 at 4:42 PM, Vick Khera <vivek@khera.org> wrote:
On Thu, Dec 11, 2014 at 10:30 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:needed to hold relcache entries for all 23000 tables :-(. If so there
may not be any easy way around it, except perhaps replicating subsets
of the tables. Unless you can boost the memory available to the backend
I'd suggest this. Break up your replication into something like 50 sets of 500 tables each, then add one at a time to replication, merging it into the main set. Something like this:create & replicate set 1.create & replicate set 2.merge 2 into 1.create & replicate set 3.merge 3 into 1.repeat until done. this can be scripted.Given you got about 50% done before it failed, maybe even 4 sets of 6000 tables each may work out.
--
On Thu, Dec 11, 2014 at 12:05 PM, Carlos Henrique Reimer <carlos.reimer@opendb.com.br> wrote: > That was exactly what the process was doing and the out of memory error > happened while one of the merges to set 1 was being executed. You sure you don't have a ulimit getting in the way?
Yes, all lines of /etc/security/limits.conf are commented out and session ulimit -a indicates the defaults are being used:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 385725
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 385725
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
On Thu, Dec 11, 2014 at 5:19 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
On Thu, Dec 11, 2014 at 12:05 PM, Carlos Henrique Reimer
<carlos.reimer@opendb.com.br> wrote:
> That was exactly what the process was doing and the out of memory error
> happened while one of the merges to set 1 was being executed.
You sure you don't have a ulimit getting in the way?
--
Carlos Henrique Reimer <carlos.reimer@opendb.com.br> writes:
> Yes, all lines of /etc/security/limits.conf are commented out and session
> ulimit -a indicates the defaults are being used:
I would not trust "ulimit -a" executed in an interactive shell to be
representative of the environment in which daemons are launched ...
have you tried putting "ulimit -a >sometempfile" into the postmaster
start script?
regards, tom lane
Extracted ulimits values from postmaster pid and they look as expected:
[root@00002-NfseNet ~]# ps -ef | grep /postgres
postgres 2992 1 1 Nov30 ? 03:17:46 /usr/local/pgsql/bin/postgres -D /database/dbcluster
root 26694 1319 0 18:19 pts/0 00:00:00 grep /postgres
[root@00002-NfseNet ~]# cat /proc/2992/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1024 385725 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space 1024000000 unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 385725 385725 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
[root@00002-NfseNet-SGDB ~]#
[root@00002-NfseNet ~]# ps -ef | grep /postgres
postgres 2992 1 1 Nov30 ? 03:17:46 /usr/local/pgsql/bin/postgres -D /database/dbcluster
root 26694 1319 0 18:19 pts/0 00:00:00 grep /postgres
[root@00002-NfseNet ~]# cat /proc/2992/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1024 385725 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space 1024000000 unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 385725 385725 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
[root@00002-NfseNet-SGDB ~]#
On Thu, Dec 11, 2014 at 6:01 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Carlos Henrique Reimer <carlos.reimer@opendb.com.br> writes:
> Yes, all lines of /etc/security/limits.conf are commented out and session
> ulimit -a indicates the defaults are being used:
I would not trust "ulimit -a" executed in an interactive shell to be
representative of the environment in which daemons are launched ...
have you tried putting "ulimit -a >sometempfile" into the postmaster
start script?
regards, tom lane
--
So if you watch processes running with sort by memory turned on in top or htop can you see your machine running out of memory etc? You have enough swap if needed? 48G is pretty small for a modern pgsql server with as much data and tables as you have, so I'd assume you have plenty of swap just in case.
Carlos Henrique Reimer <carlos.reimer@opendb.com.br> writes:
> Extracted ulimits values from postmaster pid and they look as expected:
> [root@00002-NfseNet ~]# cat /proc/2992/limits
> Limit Soft Limit Hard Limit
> Units
> Max address space 1024000000 unlimited
> bytes
So you've got a limit of 1GB on process address space ... that's
probably why it's burping on allocations of a couple hundred meg,
especially if you have a reasonably large shared_buffers setting.
You might as well be running a 32-bit build (in fact, a 32-bit
build could still do a lot better than that).
regards, tom lane