Обсуждение: Question about non-blocking mode in libpq

Поиск
Список
Период
Сортировка

Question about non-blocking mode in libpq

От
Yugo NAGATA
Дата:
Hello,

During reading the documentation of libpq [1] , I found the following
description:

 In the nonblocking state, calls to PQsendQuery, PQputline, PQputnbytes,
 PQputCopyData, and PQendcopy will not block but instead return an error
 if they need to be called again.

[1] https://www.postgresql.org/docs/devel/libpq-async.html

However, looking into the code, PQsendQuery seems not to return an error
in non-bloking mode even if unable to send all data. In such cases,
pqSendSome will return 1 but it doesn't cause an error. Moreover,
we would not need to call PQsendQuery again. Indead, we need to call
PQflush until it returns 0, as documented with regard to PQflush.

Do we need to fix the description of PQsetnonblocking?

Regards,
Yugo Nagata

-- 
Yugo NAGATA <nagata@sraoss.co.jp>



Re: Question about non-blocking mode in libpq

От
Yugo NAGATA
Дата:
On Tue, 13 Jul 2021 11:59:49 +0900
Yugo NAGATA <nagata@sraoss.co.jp> wrote:

> Hello,
> 
> During reading the documentation of libpq [1] , I found the following
> description:
> 
>  In the nonblocking state, calls to PQsendQuery, PQputline, PQputnbytes,
>  PQputCopyData, and PQendcopy will not block but instead return an error
>  if they need to be called again.
> 
> [1] https://www.postgresql.org/docs/devel/libpq-async.html
> 
> However, looking into the code, PQsendQuery seems not to return an error
> in non-bloking mode even if unable to send all data. In such cases,
> pqSendSome will return 1 but it doesn't cause an error. Moreover,
> we would not need to call PQsendQuery again. Indead, we need to call
> PQflush until it returns 0, as documented with regard to PQflush.
> 
> Do we need to fix the description of PQsetnonblocking?

I have further questions. Reading the following statement:

 "In the nonblocking state, calls to PQsendQuery, PQputline, PQputnbytes,
 PQputCopyData, and PQendcopy will not block" 

this seems to me that this is a list of functions that could block in blocking
mode, but I wander PQflush also could block because it calls pqSendSome, right?

Also, in the last paragraph of the section, I can find the following:

 "After sending any command or data on a nonblocking connection, call PQflush. ..."

However, ISTM we don't need to call PQflush in non-bloking mode and we can
call PQgetResult immediately because PQgetResult internally calls pqFlush
until it returns 0 (or -1).

        /*
         * If data remains unsent, send it.  Else we might be waiting for the
         * result of a command the backend hasn't even got yet.
         */
        while ((flushResult = pqFlush(conn)) > 0) 
        {
            if (pqWait(false, true, conn))
            {
                flushResult = -1;
                break;
            }
        }

Therefore, I wander the last paragraph of this section is
now unnecessary. right?

-- 
Yugo NAGATA <nagata@sraoss.co.jp>



Re: Question about non-blocking mode in libpq

От
Alvaro Herrera
Дата:
On 2021-Jul-19, Yugo NAGATA wrote:

> On Tue, 13 Jul 2021 11:59:49 +0900
> Yugo NAGATA <nagata@sraoss.co.jp> wrote:

> > However, looking into the code, PQsendQuery seems not to return an error
> > in non-bloking mode even if unable to send all data. In such cases,
> > pqSendSome will return 1 but it doesn't cause an error. Moreover,
> > we would not need to call PQsendQuery again. Indead, we need to call
> > PQflush until it returns 0, as documented with regard to PQflush.
> > 
> > Do we need to fix the description of PQsetnonblocking?

Yeah, I think you're right -- these functions don't error out, the
commands are just stored locally in the output buffer.

>  "In the nonblocking state, calls to PQsendQuery, PQputline, PQputnbytes,
>  PQputCopyData, and PQendcopy will not block" 
> 
> this seems to me that this is a list of functions that could block in blocking
> mode, but I wander PQflush also could block because it calls pqSendSome, right?

I don't see that.  If pqSendSome can't write anything, it'll just return 1.

> Also, in the last paragraph of the section, I can find the following:
> 
>  "After sending any command or data on a nonblocking connection, call PQflush. ..."
> 
> However, ISTM we don't need to call PQflush in non-bloking mode and we can
> call PQgetResult immediately because PQgetResult internally calls pqFlush
> until it returns 0 (or -1).

Well, maybe you don't *need* to PQflush(); but if you don't call it,
then the commands will sit in the output buffer indefinitely, which
means the server won't execute them.  So even if it works to just call
PQgetResult and have it block, surely you would like to only call
PQgetResult when the query has already been executed and the result
already been received and processed; that is, so that you can call
PQgetResult and obtain the result immediately, and avoid (say) blocking
a GUI interface while PQgetResult flushes the commands out, the server
executes the query and sends the results back.

> Therefore, I wander the last paragraph of this section is
> now unnecessary. right?

Doesn't seem so to me.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/



Re: Question about non-blocking mode in libpq

От
Yugo NAGATA
Дата:
Hello Alvaro,

On Tue, 20 Jul 2021 12:05:11 -0400
Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:

> On 2021-Jul-19, Yugo NAGATA wrote:
> 
> > On Tue, 13 Jul 2021 11:59:49 +0900
> > Yugo NAGATA <nagata@sraoss.co.jp> wrote:
> 
> > > However, looking into the code, PQsendQuery seems not to return an error
> > > in non-bloking mode even if unable to send all data. In such cases,
> > > pqSendSome will return 1 but it doesn't cause an error. Moreover,
> > > we would not need to call PQsendQuery again. Indead, we need to call
> > > PQflush until it returns 0, as documented with regard to PQflush.
> > > 
> > > Do we need to fix the description of PQsetnonblocking?
> 
> Yeah, I think you're right -- these functions don't error out, the
> commands are just stored locally in the output buffer.

Thank you for your explanation!
I attached a patch fix the description.

> >  "In the nonblocking state, calls to PQsendQuery, PQputline, PQputnbytes,
> >  PQputCopyData, and PQendcopy will not block" 
> > 
> > this seems to me that this is a list of functions that could block in blocking
> > mode, but I wander PQflush also could block because it calls pqSendSome, right?
> 
> I don't see that.  If pqSendSome can't write anything, it'll just return 1.

Well, is this the case of non-blocking mode, nor? If I understood correctly,
pqSendSome could block in blocking mode, so PQflush could block, too.  I thought
we should add PQflush to the list in the description to enphasis that this would
not block  in non-blocking mode. However, now I don't think so because PQflush
seems useful only in non-blocking mode.

> > Also, in the last paragraph of the section, I can find the following:
> > 
> >  "After sending any command or data on a nonblocking connection, call PQflush. ..."
> > 
> > However, ISTM we don't need to call PQflush in non-bloking mode and we can
> > call PQgetResult immediately because PQgetResult internally calls pqFlush
> > until it returns 0 (or -1).
> 
> Well, maybe you don't *need* to PQflush(); but if you don't call it,
> then the commands will sit in the output buffer indefinitely, which
> means the server won't execute them.  So even if it works to just call
> PQgetResult and have it block, surely you would like to only call
> PQgetResult when the query has already been executed and the result
> already been received and processed; that is, so that you can call
> PQgetResult and obtain the result immediately, and avoid (say) blocking
> a GUI interface while PQgetResult flushes the commands out, the server
> executes the query and sends the results back.

I understood that, although PQgetResult() also flushes the buffer, we still
should call PQflush() beforehand because we would not like get blocked after
calling PQgetResult(). Thanks.

Regards,
Yugo Nagata

-- 
Yugo NAGATA <nagata@sraoss.co.jp>

Вложения

Re: Question about non-blocking mode in libpq

От
Bruce Momjian
Дата:
On Wed, Jul 21, 2021 at 10:15:09AM +0900, Yugo NAGATA wrote:
> I understood that, although PQgetResult() also flushes the buffer, we still
> should call PQflush() beforehand because we would not like get blocked after
> calling PQgetResult(). Thanks.

I modified your patch, attached, that I would like to apply to all
supported versions.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Only you can decide what is important to you.

Вложения

Re: Question about non-blocking mode in libpq

От
Tom Lane
Дата:
Bruce Momjian <bruce@momjian.us> writes:
> I modified your patch, attached, that I would like to apply to all
> supported versions.

This seems to have lost the information about what to do if these
functions fail.  I think probably the only possible failure cause
in nonblock mode is "unable to enlarge the buffer because OOM",
but that's certainly not the same thing as "cannot fail".

            regards, tom lane



Re: Question about non-blocking mode in libpq

От
Bruce Momjian
Дата:
On Tue, Oct 31, 2023 at 01:58:34PM -0400, Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> > I modified your patch, attached, that I would like to apply to all
> > supported versions.
> 
> This seems to have lost the information about what to do if these
> functions fail.  I think probably the only possible failure cause
> in nonblock mode is "unable to enlarge the buffer because OOM",
> but that's certainly not the same thing as "cannot fail".

Okay, I added "_successful_ calls", attached.  I am not sure what else
to add.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Only you can decide what is important to you.

Вложения

Re: Question about non-blocking mode in libpq

От
Tom Lane
Дата:
Bruce Momjian <bruce@momjian.us> writes:
> Okay, I added "_successful_ calls", attached.  I am not sure what else
> to add.

What I'm objecting to is removal of the bit about "if they need to be
called again".  That provides a hint that retry is the appropriate
response to a failure.  Admittedly, it's not 100% clear, but your
version makes it 0% clear.

            regards, tom lane



Re: Question about non-blocking mode in libpq

От
Bruce Momjian
Дата:
On Tue, Oct 31, 2023 at 09:11:06PM -0400, Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> > Okay, I added "_successful_ calls", attached.  I am not sure what else
> > to add.
> 
> What I'm objecting to is removal of the bit about "if they need to be
> called again".  That provides a hint that retry is the appropriate
> response to a failure.  Admittedly, it's not 100% clear, but your
> version makes it 0% clear.

I thought the original docs said you had to re-call on failure (it would
not block but it would fail if it could not be sent), while we are now
saying that it will be queued in the input buffer.

Is retry really something we need to mention now?  If out of memory is
our only failure case now ("unable to enlarge the buffer because OOM"),
is retry really a realistic option?

Am I missing something?

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Only you can decide what is important to you.



Re: Question about non-blocking mode in libpq

От
Tom Lane
Дата:
Bruce Momjian <bruce@momjian.us> writes:
> On Tue, Oct 31, 2023 at 09:11:06PM -0400, Tom Lane wrote:
>> What I'm objecting to is removal of the bit about "if they need to be
>> called again".  That provides a hint that retry is the appropriate
>> response to a failure.  Admittedly, it's not 100% clear, but your
>> version makes it 0% clear.

> I thought the original docs said you had to re-call on failure (it would
> not block but it would fail if it could not be sent), while we are now
> saying that it will be queued in the input buffer.

For these functions in nonblock mode, failure means "we didn't queue it".

> Is retry really something we need to mention now?  If out of memory is
> our only failure case now ("unable to enlarge the buffer because OOM"),
> is retry really a realistic option?

Well, ideally the application would do something to alleviate the
OOM problem before retrying.  I don't know if we want to go so far
as to discuss that.  I do object to giving the impression that
failure is impossible, which I think your proposed wording does.

An orthogonal issue with your latest wording is that it's unclear
whether *unsuccessful* calls to these functions will block.

            regards, tom lane



Re: Question about non-blocking mode in libpq

От
Bruce Momjian
Дата:
On Tue, Oct 31, 2023 at 10:16:07PM -0400, Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> > On Tue, Oct 31, 2023 at 09:11:06PM -0400, Tom Lane wrote:
> >> What I'm objecting to is removal of the bit about "if they need to be
> >> called again".  That provides a hint that retry is the appropriate
> >> response to a failure.  Admittedly, it's not 100% clear, but your
> >> version makes it 0% clear.
> 
> > I thought the original docs said you had to re-call on failure (it would
> > not block but it would fail if it could not be sent), while we are now
> > saying that it will be queued in the input buffer.
> 
> For these functions in nonblock mode, failure means "we didn't queue it".
> 
> > Is retry really something we need to mention now?  If out of memory is
> > our only failure case now ("unable to enlarge the buffer because OOM"),
> > is retry really a realistic option?
> 
> Well, ideally the application would do something to alleviate the
> OOM problem before retrying.  I don't know if we want to go so far
> as to discuss that.  I do object to giving the impression that
> failure is impossible, which I think your proposed wording does.
> 
> An orthogonal issue with your latest wording is that it's unclear
> whether *unsuccessful* calls to these functions will block.

Okay, I see your point now.  Here is an updated patch that addresses
both issues.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Only you can decide what is important to you.

Вложения

Re: Question about non-blocking mode in libpq

От
Bruce Momjian
Дата:
On Wed, Nov  1, 2023 at 08:47:33AM -0400, Bruce Momjian wrote:
> On Tue, Oct 31, 2023 at 10:16:07PM -0400, Tom Lane wrote:
> > Bruce Momjian <bruce@momjian.us> writes:
> > > On Tue, Oct 31, 2023 at 09:11:06PM -0400, Tom Lane wrote:
> > >> What I'm objecting to is removal of the bit about "if they need to be
> > >> called again".  That provides a hint that retry is the appropriate
> > >> response to a failure.  Admittedly, it's not 100% clear, but your
> > >> version makes it 0% clear.
> > 
> > > I thought the original docs said you had to re-call on failure (it would
> > > not block but it would fail if it could not be sent), while we are now
> > > saying that it will be queued in the input buffer.
> > 
> > For these functions in nonblock mode, failure means "we didn't queue it".
> > 
> > > Is retry really something we need to mention now?  If out of memory is
> > > our only failure case now ("unable to enlarge the buffer because OOM"),
> > > is retry really a realistic option?
> > 
> > Well, ideally the application would do something to alleviate the
> > OOM problem before retrying.  I don't know if we want to go so far
> > as to discuss that.  I do object to giving the impression that
> > failure is impossible, which I think your proposed wording does.
> > 
> > An orthogonal issue with your latest wording is that it's unclear
> > whether *unsuccessful* calls to these functions will block.
> 
> Okay, I see your point now.  Here is an updated patch that addresses
> both issues.

Patch applied to master.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Only you can decide what is important to you.



Re: Question about non-blocking mode in libpq

От
Bruce Momjian
Дата:
On Mon, Nov 13, 2023 at 01:01:32PM -0500, Bruce Momjian wrote:
> On Wed, Nov  1, 2023 at 08:47:33AM -0400, Bruce Momjian wrote:
> > On Tue, Oct 31, 2023 at 10:16:07PM -0400, Tom Lane wrote:
> > > Bruce Momjian <bruce@momjian.us> writes:
> > > > On Tue, Oct 31, 2023 at 09:11:06PM -0400, Tom Lane wrote:
> > > >> What I'm objecting to is removal of the bit about "if they need to be
> > > >> called again".  That provides a hint that retry is the appropriate
> > > >> response to a failure.  Admittedly, it's not 100% clear, but your
> > > >> version makes it 0% clear.
> > > 
> > > > I thought the original docs said you had to re-call on failure (it would
> > > > not block but it would fail if it could not be sent), while we are now
> > > > saying that it will be queued in the input buffer.
> > > 
> > > For these functions in nonblock mode, failure means "we didn't queue it".
> > > 
> > > > Is retry really something we need to mention now?  If out of memory is
> > > > our only failure case now ("unable to enlarge the buffer because OOM"),
> > > > is retry really a realistic option?
> > > 
> > > Well, ideally the application would do something to alleviate the
> > > OOM problem before retrying.  I don't know if we want to go so far
> > > as to discuss that.  I do object to giving the impression that
> > > failure is impossible, which I think your proposed wording does.
> > > 
> > > An orthogonal issue with your latest wording is that it's unclear
> > > whether *unsuccessful* calls to these functions will block.
> > 
> > Okay, I see your point now.  Here is an updated patch that addresses
> > both issues.
> 
> Patch applied to master.

My apologies, I forgot this needed to backpatched, so done now.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Only you can decide what is important to you.