Обсуждение: pg_standby and build farm
I've written up a Warm Standby script as a .c program rather than scripts, to allow it to be portable and potentially shipped as part of PostgreSQL core. pg_standby is designed to be a wait-for restore_command, required to turn a normal archive recovery into a Warm Standby. Within the restore_command of the recovery.conf you could configure pg_standby in the following way: restore_command = 'pg_standby archiveDir %f %p' It has a number of other useful options and works with Win32 also. So far I've written this as a contrib module. Would people like to see this submitted to -patches, or should I make it available via PgFoundry HAPITR? Should this be reworked as an src/bin program? Or is the contrib module the best form for this in 8.3? I've also written a test program that uses this. I'm using this on a server at EDB to run a continuous Warm Standby test. Would anybody consider this test script worth including somewhere in core? and/or: Might it become part of the build farm? -- Simon Riggs EnterpriseDB http://www.enterprisedb.com
I'd love to see your program. Could you please submit to -pacthes? -- Tatsuo Ishii SRA OSS, Inc. Japan > I've written up a Warm Standby script as a .c program rather than > scripts, to allow it to be portable and potentially shipped as part of > PostgreSQL core. > > pg_standby is designed to be a wait-for restore_command, required to > turn a normal archive recovery into a Warm Standby. Within the > restore_command of the recovery.conf you could configure pg_standby in > the following way: > > restore_command = 'pg_standby archiveDir %f %p' > > It has a number of other useful options and works with Win32 also. > > So far I've written this as a contrib module. Would people like to see > this submitted to -patches, or should I make it available via PgFoundry > HAPITR? > > Should this be reworked as an src/bin program? Or is the contrib module > the best form for this in 8.3? > > I've also written a test program that uses this. I'm using this on a > server at EDB to run a continuous Warm Standby test. > > Would anybody consider this test script worth including somewhere in > core? and/or: Might it become part of the build farm? > > -- > Simon Riggs > EnterpriseDB http://www.enterprisedb.com > > > > ---------------------------(end of broadcast)--------------------------- > TIP 9: In versions below 8.0, the planner will ignore your desire to > choose an index scan if your joining column's datatypes do not > match >
Simon Riggs wrote: > I've written up a Warm Standby script as a .c program rather than > scripts, to allow it to be portable and potentially shipped as part of > PostgreSQL core. > Great! It'll make the DBA's work much easier, could help the adoption of PITR (nowadays people thinks it's to complicate to code a script) and help Win32 users. > So far I've written this as a contrib module. Would people like to see > this submitted to -patches, or should I make it available via PgFoundry > HAPITR? > > Should this be reworked as an src/bin program? Or is the contrib module > the best form for this in 8.3? > Please submit it to -patches. IMHO it should stay in src/bin because it'll be part of a solution that is tightly integrated to core (PITR). -- Euler Taveira de Oliveira http://www.timbira.com/
Euler Taveira de Oliveira <euler@timbira.com> writes:
> Simon Riggs wrote:
>> Should this be reworked as an src/bin program? Or is the contrib module
>> the best form for this in 8.3?
>>
> Please submit it to -patches. IMHO it should stay in src/bin because
> it'll be part of a solution that is tightly integrated to core (PITR).
I think src/bin would be a seriously bad place for it, because that
carries the presumption that "there can be only one", whereas in reality
this will be more like an example of one way to do it. contrib seems OK.
(And no, it should NOT be perceived as "tightly integrated to core".
The entire point of the design is to keep that behavior separate.)
regards, tom lane
Simon Riggs wrote: > > I've also written a test program that uses this. I'm using this on a > server at EDB to run a continuous Warm Standby test. > > Would anybody consider this test script worth including somewhere in > core? and/or: Might it become part of the build farm? > Buildfarm automatically runs all regression tests in core, including in contrib. Running some other test where the code is in core could be trivial, as long as there was a boolean success criterion. But of course it has to run on a single machine. Send me details and I can take a look. cheers andrew
On Wed, 2006-12-13 at 19:43 -0500, Tom Lane wrote: > Euler Taveira de Oliveira <euler@timbira.com> writes: > > Simon Riggs wrote: > >> Should this be reworked as an src/bin program? Or is the contrib module > >> the best form for this in 8.3? > >> > > Please submit it to -patches. IMHO it should stay in src/bin because > > it'll be part of a solution that is tightly integrated to core (PITR). Posted to pgsql-patches > I think src/bin would be a seriously bad place for it, because that > carries the presumption that "there can be only one", whereas in reality > this will be more like an example of one way to do it. contrib seems OK. Agreed. I will be updating it each time we have feedback on integration with various BAR systems to provide more options as standard. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com
On Thu, 2006-12-14 at 07:35 +0900, Tatsuo Ishii wrote: > Could you please submit to -pacthes? Patch submitted to -patches as requested last Thursday. Any feedback would be gratefully received. Thanks, -- Simon Riggs EnterpriseDB http://www.enterprisedb.com
Hi all,<br /> I'm new to the forums, so bear with me on my questions. I've set up an auto-archive and auto-recover pairof databases using pg_standby, which I'm prototyping various products for high availability. I've noticed that when Iattempt to fail from the primary archiver to the secondary recovery db using the pg_standby trigger file, the secondarydetects the trigger file, flags that it couldn't read the current WAL file pg_standby was waiting on, then attemptsto read in the previous WAL file. I use the -m option in pg_standby, so the previous WAL file no longer exists, whichcauses the secondary postgres to "panic" on not being able to open the previous WAL and terminates. Is there a way toprevent the looking for the previous, or preserving the previous WAL file so that when the trigger file is detected, thesecondary will come all the way up, completely its recovery mode? <br /><br /> Thanks,<br /> Doug Knight
On Tue, 2006-12-26 at 10:34 -0500, Doug Knight wrote: > I'm new to the forums, so bear with me on my questions. I've set up an > auto-archive and auto-recover pair of databases using pg_standby, > which I'm prototyping various products for high availability. Thanks for the feedback. pg_standby was written as a by-product of some WAL recovery testing, so its in use, but only by me so far. > I've noticed that when I attempt to fail from the primary archiver to > the secondary recovery db using the pg_standby trigger file, the > secondary detects the trigger file, flags that it couldn't read the > current WAL file pg_standby was waiting on, then attempts to read in > the previous WAL file. I use the -m option in pg_standby, so the > previous WAL file no longer exists, which causes the secondary > postgres to "panic" on not being able to open the previous WAL and > terminates. Is there a way to prevent the looking for the previous, or > preserving the previous WAL file so that when the trigger file is > detected, the secondary will come all the way up, completely its > recovery mode? Well, on a separate point I realised today that the -m option is faster but just doesn't work perfectly with restartable recovery, so I had already decided before reading this post that the move option needs to be removed. Doubly so, now. Apologies for my slip. Use the default copy mode and it should work fine. The reason for the -m option was performance. Recovery is I/O-bound, with 50% of the CPU it does use coming from IsRecordValid() - which is where the CRC checking takes place. (I can add an option to recover without CRC checks, if anyone wants it, once I've rejigged the option parsing for recovery.conf.) Should be able to use hard links, i.e. ln -f -s /archivepath/%f %p instead. I'll test that tomorrow then issue a new version. I'll also add a signal handler for SIGINT and SIGQUIT. Comments? -- Simon Riggs EnterpriseDB http://www.enterprisedb.com
On Wed, 2006-12-27 at 20:09 +0000, Simon Riggs wrote: > The reason for the -m option was performance. Recovery is I/O-bound, > with 50% of the CPU it does use coming from IsRecordValid() - which is > where the CRC checking takes place. (I can add an option to recover > without CRC checks, if anyone wants it, once I've rejigged the option > parsing for recovery.conf.) Make that 70% of the CPU, for long running recoveries, but the CPU only gets as high as 20% on my tests, so still I/O bound. > Should be able to use hard links, i.e. ln -f -s /archivepath/%f %p > instead. I'll test that tomorrow then issue a new version. The ln works, and helps, but not that much. I'll remove the -m option and replace it with an -l option. Must be careful to use the -f option. The majority of the I/O comes from writing dirty buffers, so enabling the bgwriter during recovery would probably be more helpful. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com
Thanks for the response, Simon. I'm continuing to use your script, so if there's anything I can help you with (testing,ideas, etc), just let me know.<br /><br /> Doug<br /><br /> On Thu, 2006-12-28 at 13:40 +0000, Simon Riggs wrote:<blockquote type="CITE"><pre> <font color="#000000">On Wed, 2006-12-27 at 20:09 +0000, Simon Riggs wrote:</font> <font color="#000000">> The reason for the -m option was performance. Recovery is I/O-bound,</font> <font color="#000000">> with 50% of the CPU it does use coming from IsRecordValid() - which is</font> <font color="#000000">> where the CRC checking takes place. (I can add an option to recover</font> <font color="#000000">> without CRC checks, if anyone wants it, once I've rejigged the option</font> <font color="#000000">> parsing for recovery.conf.)</font> <font color="#000000">Make that 70% of the CPU, for long running recoveries, but the CPU only</font> <font color="#000000">gets as high as 20% on my tests, so still I/O bound.</font> <font color="#000000">> Should be able to use hard links, i.e. ln -f -s /archivepath/%f %p</font> <font color="#000000">> instead. I'll test that tomorrow then issue a new version.</font> <font color="#000000">The ln works, and helps, but not that much. I'll remove the -m option</font> <font color="#000000">and replace it with an -l option. Must be careful to use the -f option.</font> <font color="#000000">The majority of the I/O comes from writing dirty buffers, so enabling</font> <font color="#000000">the bgwriter during recovery would probably be more helpful.</font> </pre></blockquote>
On Thu, 2006-12-28 at 08:45 -0500, Doug Knight wrote: > Thanks for the response, Simon. I'm continuing to use your script, so > if there's anything I can help you with (testing, ideas, etc), just > let me know. My earlier comments about mv were not correct; when fully retesting things, I noted another error instead and have corrected that. pg_standby now supports cp, mv and ln - all 3 now tested on Linux. Posting new version to patches; no signal handling (yet). Another other comments gratefully received. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com