Re: logical changeset generation v6.2

On Fri, Oct 11, 2013 at 12:57 PM, Andres Freund <andres@2ndquadrant.com> wrote:
>> I don't see any need for SQL syntax.  I was just thinking that the
>> _PG_init function could fill in a structure and then call
>> RegisterLogicalReplicationOutputPlugin(&mystruct).
>
> Hm. We can do that, but what'd be the advantage of that? The current
> model will correctly handle things like a'shared_preload_libraries'ed
> output plugin, because its _PG_init() will not register it. With the
> handling done in _PG_init() there would be two.
> Being able to use the same .so for output plugin handling and some other
> replication solution specific stuff is imo useful.

Well, I just think relying on specific symbol names in the .so file is
kind of unfortunate.  It means that, for example, you can't have
multiple output plugins provided by a single .so.  And in general I
think it's something that we've tried to minimize.

>> I don't see why you're so pessimistic about that.  I know you haven't
>> worked it out yet, but what makes this harder than sitting down and
>> designing something?
>
> Because every replication solution has different requirements for the
> format and they will want filter the output stream with regard to their
> own configuration.
> E.g. bucardo will want to include the transaction timestamp for conflict
> resolution and such.

But there's only so much information available here.  Why not just
have a format that logs it all?

> Sure, that's no problem. Do I understand correctly that you'd like
> wal_decoding: Add information about a tables primary key to struct RelationData
> wal_decoding: Add wal_level = logical and log data required for logical decoding
>
> earlier?

Yes.

>> > I'd really like to do so. I am travelling atm, but I will be back
>> > tomorrow evening and will push an updated patch this weekend. The issue
>> > I know of in the latest patches at
>> > http://www.postgresql.org/message-id/20131007133232.GA15202@awork2.anarazel.de
>> > is renaming from http://www.postgresql.org/message-id/20131008194758.GB3718183@alap2.anarazel.de
>>
>> I'm a bit nervous about the way the combo CID logging.  I would have
>> thought that you would emit one record per combo CID, but what you're
>> apparently doing is emitting one record per heap tuple that uses a
>> combo CID.
>
> I thought and implemented that in the beginning. Unfortunately it's not
> enough :(. That's probably the issue that took me longest to understand
> in this patchseries...
>
> Combocids can only fix the case where a transaction actually has create
> a combocid:
>
> 1) TX1: INSERT id = 1 at 0/1: (xmin = 1, xmax=Invalid, cmin = 55, cmax = Invalid)
> 2) TX2: DELETE id = 1 at 0/1: (xmin = 1, xmax=2, cmin = Invalid, cmax = 1)
>
> So, if we're decoding data that needs to lookup those rows in TX1 or TX2
> we both times need access to cmin and cmax, but neither transaction will
> have created a multixact. That can only be an issue in transaction with
> catalog modifications.

Oh, yuck.  So that means you have to write an extra WAL record for
EVERY heap insert, update, or delete to a catalog table?  OUCH.

> Couldn't measure anything either, which is not surprising that I
> couldn't measure the overhead in the first place.
>
> I've done some parallel INSERT/DELETE pgbenching around the
> wal_level=logical and I couldn't measure any overhead with it
> disabled. With wal_level = logical, UPDATEs and DELETEs do get a bit
> slower, but that's to be expected.
>
> It'd probably not hurt to redo those benchmarks to make sure...

Yes, I think it would be good to characterize it more precisely than
"a bit", so people know what to expect.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company