If you're doing minor upgrades, you don't need to bring down the whole cluster, you can take a single replica out and upgrade pg on it, then put it back in the cluster. Again, if you have enough kept wal segments, or you use replication slots, then it will just catch right up.
What do you mean by "take it out of the cluster" and "put it back in the cluster"? Shut it down and rename the recovery.conf? Or more complexity than that? Why would that be needed at all?