Обсуждение: Re: open items for 9.4
On Sun, Sep 28, 2014 at 11:36 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Josh Berkus <josh@agliodbs.com> writes: >> So, can we get Beta3 out now? > > If nobody else steps up and says they want to do some performance > testing, I'll push the latest lengths+offsets patch tomorrow. > > Are any of the other open items listed at > https://wiki.postgresql.org/wiki/PostgreSQL_9.4_Open_Items > things that we must-fix-before-beta3? The items I see are: - Remove xloginsert_slots/xloginsert_locks GUC - Not yet!! The text seems to indicate that there's some disagreement on this point. I don't have a strong opinion on whether or not to keep the GUC, but if we're going to remove it it should probably happen before beta3. It's going to be impossible to remove once we've released with it, I suspect. - TAP tests still have some pretty severe problems Some of these issues have been fixed; but maybe not all. - psql output change in 9.4 I'm still happy with what's committed, but if somebody else wants to tweak it, it should get done soon. - autovacuum scheduling starvation and frenzy This doesn't seem to be a new problem, so I don't think we should consider it a stop-ship issue. - jsonb data format doesn't work well with toast compression Sounds like we are near to resolving this. - pg_dump fails with --if-exists and blobs This looks like a 9.4 regression. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 2014-09-29 11:50:19 -0400, Robert Haas wrote: > The items I see are: > > - Remove xloginsert_slots/xloginsert_locks GUC - Not yet!! > > The text seems to indicate that there's some disagreement on this > point. I don't have a strong opinion on whether or not to keep the > GUC, but if we're going to remove it it should probably happen before > beta3. It's going to be impossible to remove once we've released with > it, I suspect. I vote for keeping it. > - TAP tests still have some pretty severe problems > > Some of these issues have been fixed; but maybe not all. If there are, they don't seem blockers to me personally. > - autovacuum scheduling starvation and frenzy > > This doesn't seem to be a new problem, so I don't think we should > consider it a stop-ship issue. +1. > - pg_dump fails with --if-exists and blobs > > This looks like a 9.4 regression. Alvaro, IIRC you were looking at this one? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Mon, Sep 29, 2014 at 5:53 PM, Andres Freund <andres@2ndquadrant.com> wrote: > On 2014-09-29 11:50:19 -0400, Robert Haas wrote: >> The items I see are: >> >> - Remove xloginsert_slots/xloginsert_locks GUC - Not yet!! >> >> The text seems to indicate that there's some disagreement on this >> point. I don't have a strong opinion on whether or not to keep the >> GUC, but if we're going to remove it it should probably happen before >> beta3. It's going to be impossible to remove once we've released with >> it, I suspect. > > I vote for keeping it. No real preference. But ISTM that if we're uncertain, it's probably a good idea to keep them. >> - TAP tests still have some pretty severe problems >> >> Some of these issues have been fixed; but maybe not all. > > If there are, they don't seem blockers to me personally. +1. Good to have, but not release blockers. >> - autovacuum scheduling starvation and frenzy >> >> This doesn't seem to be a new problem, so I don't think we should >> consider it a stop-ship issue. > > +1. +1. >> - pg_dump fails with --if-exists and blobs >> >> This looks like a 9.4 regression. > > Alvaro, IIRC you were looking at this one? This does seem to be the "major" release stopper, other than the json stuff. -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
Andres Freund wrote: > On 2014-09-29 11:50:19 -0400, Robert Haas wrote: > > - pg_dump fails with --if-exists and blobs > > > > This looks like a 9.4 regression. > > Alvaro, IIRC you were looking at this one? I am. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
<p dir="ltr"><br /> Dne 29.9.2014 18:00 "Magnus Hagander" <<a href="mailto:magnus@hagander.net">magnus@hagander.net</a>>napsal(a):<br /> ><br /> > On Mon, Sep 29, 2014 at 5:53PM, Andres Freund <<a href="mailto:andres@2ndquadrant.com">andres@2ndquadrant.com</a>> wrote:<br /> > > On2014-09-29 11:50:19 -0400, Robert Haas wrote:<br /> > >> The items I see are:<br /> > >><br /> > >>- Remove xloginsert_slots/xloginsert_locks GUC - Not yet!!<br /> > >><br /> > >> The text seemsto indicate that there's some disagreement on this<br /> > >> point. I don't have a strong opinion on whetheror not to keep the<br /> > >> GUC, but if we're going to remove it it should probably happen before<br />> >> beta3. It's going to be impossible to remove once we've released with<br /> > >> it, I suspect.<br/> > ><br /> > > I vote for keeping it.<br /> ><br /> > No real preference. But ISTM that ifwe're uncertain, it's probably a<br /> > good idea to keep them.<br /> ><br /> ><br /> > >> - TAP testsstill have some pretty severe problems<br /> > >><br /> > >> Some of these issues have been fixed;but maybe not all.<br /> > ><br /> > > If there are, they don't seem blockers to me personally.<br /> ><br/> > +1. Good to have, but not release blockers.<br /> ><br /> ><br /> > >> - autovacuum schedulingstarvation and frenzy<br /> > >><br /> > >> This doesn't seem to be a new problem, so I don'tthink we should<br /> > >> consider it a stop-ship issue.<br /> > ><br /> > > +1.<br /> ><br/> > +1.<br /> ><br /> ><br /> > >> - pg_dump fails with --if-exists and blobs<br /> > >><br/> > >> This looks like a 9.4 regression.<br /> > ><br /> > > Alvaro, IIRC you were lookingat this one?<br /> ><br /> > This does seem to be the "major" release stopper, other than the json stuff.<br/> ><p dir="ltr">I sent two variants of fix month ago. It should be somewhere in mailing list archive<p dir="ltr">><br/> > --<br /> > Magnus Hagander<br /> > Me: <a href="http://www.hagander.net/">http://www.hagander.net/</a><br/> > Work: <a href="http://www.redpill-linpro.com/">http://www.redpill-linpro.com/</a><br/> ><br /> ><br /> > --<br /> > Sentvia pgsql-hackers mailing list (<a href="mailto:pgsql-hackers@postgresql.org">pgsql-hackers@postgresql.org</a>)<br />> To make changes to your subscription:<br /> > <a href="http://www.postgresql.org/mailpref/pgsql-hackers">http://www.postgresql.org/mailpref/pgsql-hackers</a><br/>
On 09/29/2014 08:53 AM, Andres Freund wrote: >> - Remove xloginsert_slots/xloginsert_locks GUC - Not yet!! >> > >> > The text seems to indicate that there's some disagreement on this >> > point. I don't have a strong opinion on whether or not to keep the >> > GUC, but if we're going to remove it it should probably happen before >> > beta3. It's going to be impossible to remove once we've released with >> > it, I suspect. > I vote for keeping it. Time for me to put on my "we have too many darned GUCs" curmudgeon hat. 1. What does it do? 2. Are there reasons why users would want to change it from the default? 3. Can we explain those reasons in the form of documentation? IMHO, if the answers to 2 or 3 are "no", then we shouldn't have a GUC. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On 2014-09-29 11:28:07 -0700, Josh Berkus wrote: > On 09/29/2014 08:53 AM, Andres Freund wrote: > >> - Remove xloginsert_slots/xloginsert_locks GUC - Not yet!! > >> > > >> > The text seems to indicate that there's some disagreement on this > >> > point. I don't have a strong opinion on whether or not to keep the > >> > GUC, but if we're going to remove it it should probably happen before > >> > beta3. It's going to be impossible to remove once we've released with > >> > it, I suspect. > > I vote for keeping it. > > Time for me to put on my "we have too many darned GUCs" curmudgeon hat. > > 1. What does it do? Change scalability in write heavy loads. > 2. Are there reasons why users would want to change it from the > default? Yes, to improve scalability. > 3. Can we explain those reasons in the form of documentation? Yes. Try and benchmark it. It'll be hardware and workload dependant. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Robert Haas <robertmhaas@gmail.com> writes:
> The items I see are:
> - Remove xloginsert_slots/xloginsert_locks GUC - Not yet!!
> The text seems to indicate that there's some disagreement on this
> point. I don't have a strong opinion on whether or not to keep the
> GUC, but if we're going to remove it it should probably happen before
> beta3. It's going to be impossible to remove once we've released with
> it, I suspect.
The lack of any documentation for the GUC (neither in config.sgml or
postgresql.conf.sample) suggests very very strongly that it was not
meant to be shipped. If we don't remove it I will certainly insist
that it be documented adequately.
Personally I think a hardwired #define should be plenty. What's the
argument that users will need to tune this at runtime?
regards, tom lane
On Mon, Sep 29, 2014 at 2:44 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> The items I see are: > >> - Remove xloginsert_slots/xloginsert_locks GUC - Not yet!! > >> The text seems to indicate that there's some disagreement on this >> point. I don't have a strong opinion on whether or not to keep the >> GUC, but if we're going to remove it it should probably happen before >> beta3. It's going to be impossible to remove once we've released with >> it, I suspect. > > The lack of any documentation for the GUC (neither in config.sgml or > postgresql.conf.sample) suggests very very strongly that it was not > meant to be shipped. If we don't remove it I will certainly insist > that it be documented adequately. > > Personally I think a hardwired #define should be plenty. What's the > argument that users will need to tune this at runtime? +1. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 2014-09-29 14:44:42 -0400, Tom Lane wrote: > Robert Haas <robertmhaas@gmail.com> writes: > > The items I see are: > > > - Remove xloginsert_slots/xloginsert_locks GUC - Not yet!! > > > The text seems to indicate that there's some disagreement on this > > point. I don't have a strong opinion on whether or not to keep the > > GUC, but if we're going to remove it it should probably happen before > > beta3. It's going to be impossible to remove once we've released with > > it, I suspect. > > The lack of any documentation for the GUC (neither in config.sgml or > postgresql.conf.sample) suggests very very strongly that it was not > meant to be shipped. That's definitely the case. > If we don't remove it I will certainly insist that it be documented > adequately. Definitely. > Personally I think a hardwired #define should be plenty. What's the > argument that users will need to tune this at runtime? That right now it can make quite noticeable differences in scalability. And we have not much data to justify 8 as the default. And that's surely not going to change if we require recompiles. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Andres Freund <andres@2ndquadrant.com> writes:
> On 2014-09-29 14:44:42 -0400, Tom Lane wrote:
>> Personally I think a hardwired #define should be plenty. What's the
>> argument that users will need to tune this at runtime?
> That right now it can make quite noticeable differences in
> scalability. And we have not much data to justify 8 as the default. And
> that's surely not going to change if we require recompiles.
I wonder why it's a fixed constant at all, and not something like
"wal_buffers / 8".
regards, tom lane
On 2014-09-29 16:16:38 -0400, Tom Lane wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > On 2014-09-29 14:44:42 -0400, Tom Lane wrote: > >> Personally I think a hardwired #define should be plenty. What's the > >> argument that users will need to tune this at runtime? > > > That right now it can make quite noticeable differences in > > scalability. And we have not much data to justify 8 as the default. And > > that's surely not going to change if we require recompiles. > > I wonder why it's a fixed constant at all, and not something like > "wal_buffers / 8". Because that'd be horrible performancewise on a system with many wal_buffers. There's several operations where all locks are checked in sequence (to see whether there's any stragglers that need to finish inserting) and even some where they're acquired concurrently (e.g. for xlog switch, checkpoint and such). Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Andres Freund <andres@2ndquadrant.com> writes:
> On 2014-09-29 16:16:38 -0400, Tom Lane wrote:
>> I wonder why it's a fixed constant at all, and not something like
>> "wal_buffers / 8".
> Because that'd be horrible performancewise on a system with many
> wal_buffers. There's several operations where all locks are checked in
> sequence (to see whether there's any stragglers that need to finish
> inserting) and even some where they're acquired concurrently (e.g. for
> xlog switch, checkpoint and such).
Hm. Well, if there are countervailing considerations as to how large is a
good value, that makes it even less likely that it's sensible to expose
it as a user tunable. A relevant analogy is that we don't expose a way
to adjust the number of lock table partitions at runtime.
regards, tom lane
On 2014-09-29 16:35:12 -0400, Tom Lane wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > On 2014-09-29 16:16:38 -0400, Tom Lane wrote: > >> I wonder why it's a fixed constant at all, and not something like > >> "wal_buffers / 8". > > > Because that'd be horrible performancewise on a system with many > > wal_buffers. There's several operations where all locks are checked in > > sequence (to see whether there's any stragglers that need to finish > > inserting) and even some where they're acquired concurrently (e.g. for > > xlog switch, checkpoint and such). > > Hm. Well, if there are countervailing considerations as to how large is a > good value, that makes it even less likely that it's sensible to expose > it as a user tunable. Aren't there such considerations for most of the performance critical gucs? > A relevant analogy is that we don't expose a way > to adjust the number of lock table partitions at runtime. Which has worked out badly for e.g. the number of buffer partitions... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
<div dir="ltr"><br /><div class="gmail_extra"><br /><div class="gmail_quote">On Tue, Sep 30, 2014 at 3:44 AM, Tom Lane <spandir="ltr"><<a href="mailto:tgl@sss.pgh.pa.us" target="_blank">tgl@sss.pgh.pa.us</a>></span> wrote:<br /><blockquoteclass="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">RobertHaas <<a href="mailto:robertmhaas@gmail.com">robertmhaas@gmail.com</a>> writes:<br /> > The itemsI see are:<br /><br /> > - Remove xloginsert_slots/xloginsert_locks GUC - Not yet!!<br /><br /> > The text seemsto indicate that there's some disagreement on this<br /> > point. I don't have a strong opinion on whether or notto keep the<br /> > GUC, but if we're going to remove it it should probably happen before<br /> > beta3. It's goingto be impossible to remove once we've released with<br /> > it, I suspect.<br /><br /></span>The lack of any documentationfor the GUC (neither in config.sgml or<br /> postgresql.conf.sample) suggests very very strongly that it wasnot<br /> meant to be shipped. If we don't remove it I will certainly insist<br /> that it be documented adequately.<br/><br /> Personally I think a hardwired #define should be plenty. What's the<br /> argument that users willneed to tune this at runtime?<br /></blockquote></div>I tend to go in this direction too. It is unclear how it is reallyable to improve scalability, or at least some documentation should be here to help users to set it. An additional thoughtas well: set it with a configure switch at compilation instead of a GUC.<br />-- <br />Michael<br /></div></div>
On 09/29/2014 11:41 PM, Andres Freund wrote: > On 2014-09-29 16:35:12 -0400, Tom Lane wrote: >> Andres Freund <andres@2ndquadrant.com> writes: >>> On 2014-09-29 16:16:38 -0400, Tom Lane wrote: >>>> I wonder why it's a fixed constant at all, and not something like >>>> "wal_buffers / 8". >> >>> Because that'd be horrible performancewise on a system with many >>> wal_buffers. There's several operations where all locks are checked in >>> sequence (to see whether there's any stragglers that need to finish >>> inserting) and even some where they're acquired concurrently (e.g. for >>> xlog switch, checkpoint and such). >> >> Hm. Well, if there are countervailing considerations as to how large is a >> good value, that makes it even less likely that it's sensible to expose >> it as a user tunable. > > Aren't there such considerations for most of the performance critical > gucs? > >> A relevant analogy is that we don't expose a way >> to adjust the number of lock table partitions at runtime. > > Which has worked out badly for e.g. the number of buffer partitions... Number of buffer partitions is the analogy I've also had in mind. It has worked out pretty well, IMHO. Sure, with a system with a lot of CPUs you sometimes want to increase the hardwired default, but most configurations are not too sensitive to it. No-one has stepped up to do any testing on the effects if the GUC during the beta period, while the GUC has been there. Somehow I don't think anyone's going to do any rigorous tuning of it before commissioning a system into production, either. This might come up again in 9.5 cycle, once we get the improvements in LWLock and buffer cache scalability. That can make scalability of the WAL insertion more visible again. There seems to be no decisive consensus here. I'm going to put my foot on the ground and go remove it, as I'm leaning towards that option, and we need to get the release out. But if someone objects loudly enough to actually write the documentation and commit it that way, I'm happy with that too. - Heikki
On 09/30/2014 04:56 AM, Heikki Linnakangas wrote: > There seems to be no decisive consensus here. I'm going to put my foot > on the ground and go remove it, as I'm leaning towards that option, and > we need to get the release out. But if someone objects loudly enough to > actually write the documentation and commit it that way, I'm happy with > that too. I'm also in favor of removing it. We really don't need more GUCs which nobody has any clear idea how to change or why. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On 9/29/14, 2:30 PM, Andres Freund wrote: > > Can we explain those reasons in the form of documentation? > Yes. Try and benchmark it. It'll be hardware and workload dependant. I missed this whole thing, and eventually I have to circle back to it. I could do it this week. Could you (or someone else familiar with the useful benchmarks) give me more detail on how to replicate one case where things should improve? I can do that, and I have a lab full of hardware if it's easier to spot on one type of server. That exercise will probably lead to a useful opinion on the feature in its final form, any associated GUC, tunables, and necessary level of associated documentation in a day or two. This is a pretty low bar, right? Examples and documentation sufficient for just me to catch up is not asking for very much. If it isn't possible to do that in a very short period of time, it would not bode well for broader consumption. -- Greg Smith greg.smith@crunchydatasolutions.com Chief PostgreSQL Evangelist - http://crunchydatasolutions.com/
On 09/30/2014 09:10 PM, Gregory Smith wrote: > On 9/29/14, 2:30 PM, Andres Freund wrote: >>> Can we explain those reasons in the form of documentation? >> Yes. Try and benchmark it. It'll be hardware and workload dependant. > > I missed this whole thing, and eventually I have to circle back to it. > I could do it this week. Ah, that would be great! > Could you (or someone else familiar with the > useful benchmarks) give me more detail on how to replicate one case > where things should improve? I can do that, and I have a lab full of > hardware if it's easier to spot on one type of server. That exercise > will probably lead to a useful opinion on the feature in its final form, > any associated GUC, tunables, and necessary level of associated > documentation in a day or two. To see the most improvement from the patch, try a workload that's otherwise bottlenecked on XLogInsert. For example, with pgbench: psql postgres -c "create table foo (id int4)" pgbench postgres -n -f fooinsert.sql -c 4 -T10 and in the test script: insert into foo select g from generate_series(1, 10000) g; - Heikki