Обсуждение: How many Mandatory Process are there in Postgres

> Hi All,
>
> When we start the postgres server, the writer process, wal process,
> postmaster, autovacuum ( if autovacuum is on), stats collector will come
> into picture as mandotory process. My question is, is there any processes
> apart from these process, what are the mandotory process come along with
> the
> postgres server and how many sleeping processes are there.

Well, there is a bunch of processes started at the beginning, and then
there is one backend process for each connection (see the postgresql.conf
how many connections are allowed in your case).

> Few important question about Postgres Architecture
> ======================================
> 1. When does temp_buffer comes into existence in a database session ( like
> when shared_buffer completely filled or any wal_buffer filled) ?

AFAIK Those are completely different buffers I.

Shared buffers are used as a "shared cache" for all the backends /
sessions. Wal_buffers are used when writing data to the write-ahead-log
(transaction log) and are shared by all backends just like the shared
buffers.

Temp_buffers on the other hand are used to access temporary tables - so
they serve s a completely different purpose and are session-specific. Each
session may consume the given amount of memory.

> 2. What is process array in shared memory?

No, idea (not a PostgreSQL hacker) but I'd guess it has something to do
with the backends (list of backends).

> 3. maintenance_work_mem is used for vacuuming(does this memory allocated
> if
> autovacuum is off)?

Yes, it's used for vacuuming (and many other things related to
maintenance). AFAIK the amount of memory is 'upper limit' and does not
mean the autovacuum will consume that.

Anyway disabling the autovacuum is a bad idea, just as using not enough
memory.

> 4. As oracle,  Postgres doesnt have any undo_tablespace, for rollback
> transaction. But it is handled in BEGIN /END block. So question is where
> the
> transaction data is stored.

Oracle uses a completely different implementation of MVCC architecture. It
overwrites the data and then uses rollback segments to provide 'previous
versions' to running transactions etc.

PostgreSQL does not overwrite the data - it just creates a copy of the row
and then decides which version should each session see (depending on the
transaction IDs etc.). So it does not need to do rollbacks the way Oracle
does, but it has to remove stale copies of the rows (such that no running
transaction can see) - that's why there is VACUUM.

Regards
Tomas

Well, there is a bunch of processes started at the beginning, and then
there is one backend process for each connection (see the postgresql.conf
how many connections are allowed in your case).

Temp_buffers on the other hand are used to access temporary tables - so
they serve s a completely different purpose and are session-specific. Each
session may consume the given amount of memory.

Yes, it's used for vacuuming (and many other things related to
maintenance). AFAIK the amount of memory is 'upper limit' and does not
mean the autovacuum will consume that.

Anyway disabling the autovacuum is a bad idea, just as using not enough
memory.

Oracle uses a completely different implementation of MVCC architecture. It
overwrites the data and then uses rollback segments to provide 'previous
versions' to running transactions etc.

PostgreSQL does not overwrite the data - it just creates a copy of the row
and then decides which version should each session see (depending on the
transaction IDs etc.). So it does not need to do rollbacks the way Oracle
does, but it has to remove stale copies of the rows (such that no running
transaction can see) - that's why there is VACUUM.

> Hi  Tomas,
>
> Thank you for the reply.
>
>
>> Well, there is a bunch of processes started at the beginning, and then
>> there is one backend process for each connection (see the
>> postgresql.conf
>> how many connections are allowed in your case).
>>
> I do agree with you, that there would be bunch of process. Can you
> describe
> any?

The names of the processes are quite descriptive I guess - just start the
database and use 'ps ax' command. And then use the names to search the
docs, there's a plenty of details available there.

>> Temp_buffers on the other hand are used to access temporary tables - so
>> they serve s a completely different purpose and are session-specific.
>> Each
>> session may consume the given amount of memory.
>
>
> So, what happen when the shared_buffer is filled completely, where it does
> its operation.

See this:
http://wiki.postgresql.org/wiki/User:Gsmith#How_is_the_shared_buffer_cache_organized

Shared buffers is a LRU cache, i.e. when it's full and a new buffer is
requested, some of the 'oldest' buffers are removed.

>
>
>> Yes, it's used for vacuuming (and many other things related to
>> maintenance). AFAIK the amount of memory is 'upper limit' and does not
>> mean the autovacuum will consume that.
>>
>> Anyway disabling the autovacuum is a bad idea, just as using not enough
>> memory.
>
>
> My question is does maintenance_work_mem is occupied or allocated, even
> the
> autovacuum is off.

No. As I wrote the maintenance_work_mem is the maximum amount of memory
the vacuum may consume, so if it is not running the memory is not
allocated. BTW you could find this out simply by starting the server with
enabled / disabled autovacuum and comparing the amount of memory consumed.

>> Oracle uses a completely different implementation of MVCC architecture.
>> It
>> overwrites the data and then uses rollback segments to provide 'previous
>> versions' to running transactions etc.
>>
>> PostgreSQL does not overwrite the data - it just creates a copy of the
>> row
>> and then decides which version should each session see (depending on the
>> transaction IDs etc.). So it does not need to do rollbacks the way
>> Oracle
>> does, but it has to remove stale copies of the rows (such that no
>> running
>> transaction can see) - that's why there is VACUUM.
>
>
> Here, if you have issued a command pg_start_backup() at that time the
> cluster is freezed, and if any transaction takes place before the
> pg_stop_backup() issued at that time where the transaction data will be
> kept
> if the undo's are not there.

What do you mean by 'freezed'? The cluster operates normally, the
pg_start_backup() just creates a backup label (and performs a checkpoint),
but that's not a problem. OK, there could be a performance decrease
because of full page writes, but the data will be processed as if there is
no backup running. PostgreSQL does not need the checkpoints to perform
backup.

Tomas

>> Oracle uses a completely different implementation of MVCC architecture.

>> It

>> overwrites the data and then uses rollback segments to provide 'previous

>> versions' to running transactions etc.

>>

>> PostgreSQL does not overwrite the data - it just creates a copy of the

>> row

>> and then decides which version should each session see (depending on the

>> transaction IDs etc.). So it does not need to do rollbacks the way

>> Oracle

>> does, but it has to remove stale copies of the rows (such that no

>> running

>> transaction can see) - that's why there is VACUUM.

>

>

> Here, if you have issued a command pg_start_backup() at that time the

> cluster is freezed, and if any transaction takes place before the

> pg_stop_backup() issued at that time where the transaction data will be

> kept

> if the undo's are not there.

 

 

What do you mean by 'freezed'? The cluster operates normally, the
pg_start_backup() just creates a backup label (and performs a checkpoint),
but that's not a problem. OK, there could be a performance decrease
because of full page writes, but the data will be processed as if there is
no backup running. PostgreSQL does not need the checkpoints to perform
backup.

The 'UNDO' data is just kept in the main data files.  Then, based on what your xid is, and the xmin / xmax on each row, it's either visible or not.  That's what vacuum does, clean up the rows that are for 'undo' and can never been seen anymore.  The REDO itself is kept in the transaction logs.

Like Thomas says, the pg_start_backup() just creates a label that tells the restoring database what pg_xlog record to start with when you 'restore' your data.  The 'UNDO' type of data (just the old rows actually) is just kept in the main table until vacuum nukes 'em.

See this:
http://wiki.postgresql.org/wiki/User:Gsmith#How_is_the_shared_buffer_cache_organized

Shared buffers is a LRU cache, i.e. when it's full and a new buffer is
requested, some of the 'oldest' buffers are removed.

On Thu, Mar 25, 2010 at 9:03 PM, Tadipathri Raghu <traghu.dba@gmail.com> wrote:
> Hi Scott, Thomas,
>
> Thank you for the update.
>
>>
>> >> Oracle uses a completely different implementation of MVCC architecture.
>> >> It
>> >> overwrites the data and then uses rollback segments to provide
>> >> 'previous
>> >> versions' to running transactions etc.
>> >>
>> >> PostgreSQL does not overwrite the data - it just creates a copy of the
>> >> row
>> >> and then decides which version should each session see (depending on
>> >> the
>> >> transaction IDs etc.). So it does not need to do rollbacks the way
>> >> Oracle
>> >> does, but it has to remove stale copies of the rows (such that no
>> >> running
>> >> transaction can see) - that's why there is VACUUM.
>> >
>> >
>> > Here, if you have issued a command pg_start_backup() at that time the
>> > cluster is freezed, and if any transaction takes place before the
>> > pg_stop_backup() issued at that time where the transaction data will be
>> > kept
>> > if the undo's are not there.
>>
>>
>> What do you mean by 'freezed'? The cluster operates normally, the
>> pg_start_backup() just creates a backup label (and performs a checkpoint),
>> but that's not a problem. OK, there could be a performance decrease
>> because of full page writes, but the data will be processed as if there is
>> no backup running. PostgreSQL does not need the checkpoints to perform
>> backup.
>
>> The 'UNDO' data is just kept in the main data files.  Then, based on what
>> your xid is, and the xmin / xmax on each row, it's either visible or not.
>>  That's what vacuum does, clean up the rows that are for 'undo' and can
>> never been seen anymore.  The REDO itself is kept in the transaction logs.
>
>
>>
>> Like Thomas says, the pg_start_backup() just creates a label that tells
>> the restoring database what pg_xlog record to start with when you 'restore'
>> your data.  The 'UNDO' type of data (just the old rows actually) is just
>> kept in the main table until vacuum nukes 'em.
>
> --Scott
> Q1. Does vacuum process come into existence even you turn off the
> autovacuum.? What is the main work for vacuum process, to the clean the
> buffers or work only when autovacuum in on and to clean up the rows that are
> 'undo'

1a: Yes, if you are approaching transaction id wraparound (txid wrap)
then autovacuum will crank up a vacuum to fix that situation whether
you like it or not.  The alternative is db shutdown.
1b: The main job of vacuum is to free dead tuples, which postgresql
accumulates as it updates or deletes tuples. When you have several old
versions of a tuple to collect, vacuum recycles the space and makes it
available to other updates to now use instead of allocating on the end
of the relation.

> Q2. Do you mean to say there is a vacuum process which will clean up the
> buffers that are for 'undo'?

No buffers for undo.  It's all stored on disc, ready for instant
access.  Try it on a test db.  begin; load a table with 1M rows;
rollback; sub second response.  A million changes just disappeared
instantly.  And now you've got 1M dead tuples in that table.  Vacuum's
job is to make them available for re-use.  But they're all in one big
consecutive file so the performance is actually pretty good as the
table repopulates, especially if it'll get back up to 1M records again
in the near future.  As long as it got vacuumed.

> Q3. Fine, if the vacuum is off then your data files will hold the undo data
> also in it?

Yep.  Until a forced vacuum due to txid wrap occurs.

On Sun, Mar 28, 2010 at 1:14 AM, Tadipathri Raghu <traghu.dba@gmail.com> wrote:
> Hi All,
>
> Thank you for the prompt reply on this.
>
> Please find the output of the top command and the process availabe. Could
> explain what logger process is here for

Logging?  I'm just guessing there.   My machines don't have it and I'm
guessing it's the process that starts up if you're using syslog as
opposed to local logging.

Le 28/03/2010 19:30, Scott Marlowe a écrit :
> On Sun, Mar 28, 2010 at 1:14 AM, Tadipathri Raghu <traghu.dba@gmail.com> wrote:
>> Hi All,
>>
>> Thank you for the prompt reply on this.
>>
>> Please find the output of the top command and the process availabe. Could
>> explain what logger process is here for
>
> Logging?  I'm just guessing there.   My machines don't have it and I'm
> guessing it's the process that starts up if you're using syslog as
> opposed to local logging.
>

That's the other way around. The logger process is launched when you
enable the logging_collector (or redirect_stderr in 8.2 and older releases).


--
Guillaume.
 http://www.postgresqlfr.org
 http://dalibo.com