Обсуждение: [GENERAL] BDR node removal and rejoin

Hi, I am having difficulty removing a node from a BDR group (with nodes node1 through node5) then rejoin the group.

Prior to removing a node, the BDR is running fine, query on bdr.bdr_nodes table shows all nodes having the status ‘r’.

Here is what I have done for removing node5 and rejoining:

• On node1, do bdr.bdr_part_by_node_names

At this point the status of node5 in bdr.bdr_nodes becomes ‘k’

• On node5, do bdr.remove_bdr_from_local_node
• On node5, drop and recreate the database, then rejoin using bdr.bdr_group_join (using the same node name and external dsn)

At this point the status of node5 on node1 though node4 still remains ‘k’, and the status of node5 on node5 (there is only one record) is ‘i’, and they stuck at these status codes.

[note: I tried using a different node name on rejoining, same result]

What have I done wrong, what is the correct way of doing removal and rejoining?

Thanks

An update… after manually removing the record for ‘node4’ from bdr.bdr_nodes, corresponding record in bdr.bdr_connections, and associated replication slot (with pg_drop_replication_slot), rejoining was successful.

I was under the impression that there is no need to perform manual cleanup before a removed node (with database dropped and recreated) rejoining a BDR group.

Hi, I am having difficulty removing a node from a BDR group (with nodes node1 through node5) then rejoin the group.

Prior to removing a node, the BDR is running fine, query on bdr.bdr_nodes table shows all nodes having the status ‘r’.

Here is what I have done for removing node5 and rejoining:

• On node1, do bdr.bdr_part_by_node_names

At this point the status of node5 in bdr.bdr_nodes becomes ‘k’

• On node5, do bdr.remove_bdr_from_local_node
• On node5, drop and recreate the database, then rejoin using bdr.bdr_group_join (using the same node name and external dsn)

At this point the status of node5 on node1 though node4 still remains ‘k’, and the status of node5 on node5 (there is only one record) is ‘i’, and they stuck at these status codes.

[note: I tried using a different node name on rejoining, same result]

What have I done wrong, what is the correct way of doing removal and rejoining?

Thanks

Thanks for the clarification.

Looks like I am running into a different issue: while trying to pin down precisely the steps (and the order in which to perform them) needed to remove/rejoin a node, the removal/rejoining exercise was repeated a number of times, and stuck again:

1. The status of the re-joining node (node4) on other nodes is “I”
2. The status of the re-joining node on the node4 itself started at “I”, changed to “o”, then stuck there
3. From the log file for node4, the following entries are constantly being generated:

2017-07-12 10:37:46 PDT [24943:bdr (6334686800251932108,1,43865,):receive:::1(33883)]DEBUG:  00000: received replication command: IDENTIFY_SYSTEM

2017-07-12 10:37:46 PDT [24943:bdr (6334686800251932108,1,43865,):receive:::1(33883)]LOCATION:  exec_replication_command, walsender.c:1309

2017-07-12 10:37:46 PDT [24943:bdr (6334686800251932108,1,43865,):receive:::1(33883)]DEBUG:  08003: unexpected EOF on client connection

2017-07-12 10:37:46 PDT [24943:bdr (6334686800251932108,1,43865,):receive:::1(33883)]LOCATION:  SocketBackend, postgres.c:355

2017-07-12 10:37:46 PDT [24944:bdr (6408408103171110238,1,24713,):receive:::1(33884)]DEBUG:  00000: received replication command: IDENTIFY_SYSTEM

2017-07-12 10:37:46 PDT [24944:bdr (6408408103171110238,1,24713,):receive:::1(33884)]LOCATION:  exec_replication_command, walsender.c:1309

2017-07-12 10:37:46 PDT [24944:bdr (6408408103171110238,1,24713,):receive:::1(33884)]DEBUG:  08003: unexpected EOF on client connection

2017-07-12 10:37:46 PDT [24944:bdr (6408408103171110238,1,24713,):receive:::1(33884)]LOCATION:  SocketBackend, postgres.c:355

2017-07-12 10:37:46 PDT [24946:bdr (6334686760735153516,1,43845,):receive:::1(33885)]DEBUG:  00000: received replication command: IDENTIFY_SYSTEM

2017-07-12 10:37:46 PDT [24946:bdr (6334686760735153516,1,43845,):receive:::1(33885)]LOCATION:  exec_replication_command, walsender.c:1309

2017-07-12 10:37:46 PDT [24946:bdr (6334686760735153516,1,43845,):receive:::1(33885)]DEBUG:  08003: unexpected EOF on client connection

2017-07-12 10:37:46 PDT [24946:bdr (6334686760735153516,1,43845,):receive:::1(33885)]LOCATION:  SocketBackend, postgres.c:355

2017-07-12 10:37:49 PDT [24949:bdr (6394432535408825526,1,37325,):receive:::1(33892)]DEBUG:  00000: received replication command: IDENTIFY_SYSTEM

2017-07-12 10:37:49 PDT [24949:bdr (6394432535408825526,1,37325,):receive:::1(33892)]LOCATION:  exec_replication_command, walsender.c:1309

2017-07-12 10:37:49 PDT [24949:bdr (6394432535408825526,1,37325,):receive:::1(33892)]DEBUG:  08003: unexpected EOF on client connection

2017-07-12 10:37:49 PDT [24949:bdr (6394432535408825526,1,37325,):receive:::1(33892)]LOCATION:  SocketBackend, postgres.c:355

What do these entries say? and what can be done to correct the situation (there have been no change with respect to either postgres  or network configuration in the remove/rejoin exercise)?

Thanks

From: Craig Ringer [mailto:craig@2ndquadrant.com]
Sent: Wednesday, July 12, 2017 1:59 AM
To: Zhu, Joshua <jzhu@thalesesec.net>
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] BDR node removal and rejoin

Found these log entries from one of the other node:

t=2017-07-13 08:35:34 PDT p=27292 a=DEBUG:  00000: found valid replication identifier 15

t=2017-07-13 08:35:34 PDT p=27292 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:604

t=2017-07-13 08:35:34 PDT p=27292 a=ERROR:  53400: no free replication state could be found for 15, increase max_replication_slots

Increased max_replication_slots, things are looking good now, thanks.

This does bring up a couple of questions:

1. Given the fact there is no real increase in the number of nodes in this repeated removal/rejoining exercise, yet it caused replication slots being used up, wouldn’t removal of a node also automatically free up the replication slot allocated for the node? Or is there a way to manually free up no longer needed slots? (the don’t seem to show up in pg_replication_slots view, I made sure to use pg_drop_replication_slot when they do show up there)
2. If there is such a thing, what is the rule of thumb for best value of max_replication_slots (are they somehow related to the value max_wal_senders as well), with respect to, say, the max number of nodes intended to support?

Thanks

From: Craig Ringer [mailto:craig@2ndquadrant.com]
Sent: Wednesday, July 12, 2017 11:59 PM
To: Zhu, Joshua <jzhu@thalesesec.net>
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] BDR node removal and rejoin