Обсуждение: Listen/Notify feedback
I am investigating various pub/sub tools such as ActiveMQ, Rabbit, Redis, etc.I came across Postgresql Listen/Notify and was easily able to write code to listen to messages. For the people who have been using this for a while: what are its downsides, things to consider when writing good code that use pub/sub, how do you deal with large messages, can I have subscribers listen to replica nodes?
Thanks
--
--
--- Get your facts first, then you can distort them as you please.--
One aspect is if there is no one listening when a notify happens, the message is lost (e.g. no durability). If this is important to you, it can be addressed by writing the messages to a table as well when you NOTIFY, and the listener deletes messages after they are processed. On connection the listener can query the table to catch up on any missed messages, or messages that were mid-process during a crash. This is trickier with more than one listener. This isn't a whole lot more efficient than just using the table alone, but it saves you from having to poll so better response times.
On Sat, Jul 11, 2020 at 8:58 AM Rita <rmorgan466@gmail.com> wrote:
I am investigating various pub/sub tools such as ActiveMQ, Rabbit, Redis, etc.I came across Postgresql Listen/Notify and was easily able to write code to listen to messages. For the people who have been using this for a while: what are its downsides, things to consider when writing good code that use pub/sub, how do you deal with large messages, can I have subscribers listen to replica nodes?Thanks
----- Get your facts first, then you can distort them as you please.--
Thats good to know. Are there some standard patterns or best practices I should follow when using messaging and with listen/notify?
On Sat, Jul 11, 2020 at 1:44 PM Brian Dunavant <dunavant@gmail.com> wrote:
One aspect is if there is no one listening when a notify happens, the message is lost (e.g. no durability). If this is important to you, it can be addressed by writing the messages to a table as well when you NOTIFY, and the listener deletes messages after they are processed. On connection the listener can query the table to catch up on any missed messages, or messages that were mid-process during a crash. This is trickier with more than one listener. This isn't a whole lot more efficient than just using the table alone, but it saves you from having to poll so better response times.On Sat, Jul 11, 2020 at 8:58 AM Rita <rmorgan466@gmail.com> wrote:I am investigating various pub/sub tools such as ActiveMQ, Rabbit, Redis, etc.I came across Postgresql Listen/Notify and was easily able to write code to listen to messages. For the people who have been using this for a while: what are its downsides, things to consider when writing good code that use pub/sub, how do you deal with large messages, can I have subscribers listen to replica nodes?Thanks
----- Get your facts first, then you can distort them as you please.--
--- Get your facts first, then you can distort them as you please.--
On Sun, 12 Jul 2020 at 21:39, Rita <rmorgan466@gmail.com> wrote:
Thats good to know. Are there some standard patterns or best practices I should follow when using messaging and with listen/notify?On Sat, Jul 11, 2020 at 1:44 PM Brian Dunavant <dunavant@gmail.com> wrote:One aspect is if there is no one listening when a notify happens, the message is lost (e.g. no durability). If this is important to you, it can be addressed by writing the messages to a table as well when you NOTIFY, and the listener deletes messages after they are processed. On connection the listener can query the table to catch up on any missed messages, or messages that were mid-process during a crash. This is trickier with more than one listener. This isn't a whole lot more efficient than just using the table alone, but it saves you from having to poll so better response times.On Sat, Jul 11, 2020 at 8:58 AM Rita <rmorgan466@gmail.com> wrote:I am investigating various pub/sub tools such as ActiveMQ, Rabbit, Redis, etc.I came across Postgresql Listen/Notify and was easily able to write code to listen to messages. For the people who have been using this for a while: what are its downsides, things to consider when writing good code that use pub/sub, how do you deal with large messages, can I have subscribers listen to replica nodes?Thanks
----- Get your facts first, then you can distort them as you please.--
A couple of years ago I started looking into listen/notify in PG10 and found that the throughput decreased quite a bit as I added more and more listeners. Given the number of apps I needed to have listening and the number of messages that I expected to be consuming, I ended up writing a single listener app which then republished the messages via MQTT. Not sure if the performance has improved in subsequent versions (or whether this will affect you at all) but it's something to keep in mind.
Andrew Smith <laconical@gmail.com> writes:
> A couple of years ago I started looking into listen/notify in PG10 and
> found that the throughput decreased quite a bit as I added more and more
> listeners. Given the number of apps I needed to have listening and the
> number of messages that I expected to be consuming, I ended up writing a
> single listener app which then republished the messages via MQTT. Not sure
> if the performance has improved in subsequent versions (or whether this
> will affect you at all) but it's something to keep in mind.
FWIW, we made some efficiency improvements in v13 for the case of NOTIFY
with a lot of listeners [1]. Can't say of course whether that would
have fixed your problem, and v13 is still in beta anyway.
regards, tom lane
[1] https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=51004c717
Good to know about potential performance problems. I don't plan to have more than 5 hosts. Also, good to know about MQTT.
On Sun, Jul 12, 2020 at 8:52 AM Andrew Smith <laconical@gmail.com> wrote:
On Sun, 12 Jul 2020 at 21:39, Rita <rmorgan466@gmail.com> wrote:Thats good to know. Are there some standard patterns or best practices I should follow when using messaging and with listen/notify?On Sat, Jul 11, 2020 at 1:44 PM Brian Dunavant <dunavant@gmail.com> wrote:One aspect is if there is no one listening when a notify happens, the message is lost (e.g. no durability). If this is important to you, it can be addressed by writing the messages to a table as well when you NOTIFY, and the listener deletes messages after they are processed. On connection the listener can query the table to catch up on any missed messages, or messages that were mid-process during a crash. This is trickier with more than one listener. This isn't a whole lot more efficient than just using the table alone, but it saves you from having to poll so better response times.On Sat, Jul 11, 2020 at 8:58 AM Rita <rmorgan466@gmail.com> wrote:I am investigating various pub/sub tools such as ActiveMQ, Rabbit, Redis, etc.I came across Postgresql Listen/Notify and was easily able to write code to listen to messages. For the people who have been using this for a while: what are its downsides, things to consider when writing good code that use pub/sub, how do you deal with large messages, can I have subscribers listen to replica nodes?Thanks
----- Get your facts first, then you can distort them as you please.--A couple of years ago I started looking into listen/notify in PG10 and found that the throughput decreased quite a bit as I added more and more listeners. Given the number of apps I needed to have listening and the number of messages that I expected to be consuming, I ended up writing a single listener app which then republished the messages via MQTT. Not sure if the performance has improved in subsequent versions (or whether this will affect you at all) but it's something to keep in mind.
--- Get your facts first, then you can distort them as you please.--
On Sat, Jul 11, 2020 at 10:44 AM Brian Dunavant <dunavant@gmail.com> wrote:
One aspect is if there is no one listening when a notify happens, the message is lost (e.g. no durability). If this is important to you, it can be addressed by writing the messages to a table as well when you NOTIFY, and the listener deletes messages after they are processed. On connection the listener can query the table to catch up on any missed messages, or messages that were mid-process during a crash. This is trickier with more than one listener. This isn't a whole lot more efficient than just using the table alone, but it saves you from having to poll so better response times.
Good advice from Brian here that mirrors my own experience, I'd like to point out that if you do go the multiple listener route working a persistent table, it's important to avoid races with SELECT FOR UPDATE SKIP LOCKED.
I know this is old news to most of the people on this list, but I've run into enough folks who don't know about this little gem that I figured I'd mention it, it's saved my bacon more than once.
-Michel
On Sat, Jul 11, 2020 at 8:58 AM Rita <rmorgan466@gmail.com> wrote:I am investigating various pub/sub tools such as ActiveMQ, Rabbit, Redis, etc.I came across Postgresql Listen/Notify and was easily able to write code to listen to messages. For the people who have been using this for a while: what are its downsides, things to consider when writing good code that use pub/sub, how do you deal with large messages, can I have subscribers listen to replica nodes?Thanks
----- Get your facts first, then you can distort them as you please.--