Обсуждение: Is a PS - MSSQL Logical Replication Possible??

Поиск
Список
Период
Сортировка

Is a PS - MSSQL Logical Replication Possible??

От
Erik Serrano
Дата:

Dear Masters,

Is it possible to make a logical replica between a postgresql 12 and a sql server 2019?
Technically I want to make a multimaster logical replica to a subscriber and this in turn replicates to a MSSQL 2019 AlwaysOn server...
Is it possible to make this type of structure or impossible?
Please tell me...

Regards and thank you very much

Atte.




image.png


Erik R. Serrano Saavedra
Ingeniero de Sistemas Informáticos
      Data Base Administrator
        eserranos@gmail.com
                 998596691
Вложения

Re: Is a PS - MSSQL Logical Replication Possible??

От
Dan Smith
Дата:
I think what you are looking for is a Change Data Capture (CDC) solution.  This is going to require a system in the middle that can speak to both database systems.  Note these implementations can be brittle (replication can fail) and take some tinkering to get working well.

I recommend you look in to tools such as Debezium (https://debezium.io), AWS Data Migration Service (DMS), or maybe some data lake implementation and scripts could be used (apache hudi, data bricks delta lake, or other).


Best regards,

Dan Smith

On Sun, Jul 23, 2023, 22:30 Erik Serrano <eserranos@gmail.com> wrote:

Dear Masters,

Is it possible to make a logical replica between a postgresql 12 and a sql server 2019?
Technically I want to make a multimaster logical replica to a subscriber and this in turn replicates to a MSSQL 2019 AlwaysOn server...
Is it possible to make this type of structure or impossible?
Please tell me...

Regards and thank you very much

Atte.




image.png


Erik R. Serrano Saavedra
Ingeniero de Sistemas Informáticos
      Data Base Administrator
        eserranos@gmail.com
                 998596691
Вложения

Re: Is a PS - MSSQL Logical Replication Possible??

От
Erik Serrano
Дата:
Yes, I believe that it is not possible to carry out this architecture without an additional layer or application (which I do not know) exists. While it only occurs to me to make an unattended replica (Cron or Daemon linux in origin that generates a flat format dump (.sql) on a specific share and that windows can pick it up and in turn carry out the load through an intermediate job), I don't know... I think it could work... although it is not what I would have liked to do.

In any case, your collaboration is appreciated... and who knows, an issue will open up to be carried out

greetings friends

Sincerely


Erik R. Serrano Saavedra
Ingeniero de Sistemas Informáticos
      Data Base Administrator
        eserranos@gmail.com
                 998596691


El lun, 24 jul 2023 a las 8:42, Dan Smith (<j.daniel.smith1@gmail.com>) escribió:
I think what you are looking for is a Change Data Capture (CDC) solution.  This is going to require a system in the middle that can speak to both database systems.  Note these implementations can be brittle (replication can fail) and take some tinkering to get working well.

I recommend you look in to tools such as Debezium (https://debezium.io), AWS Data Migration Service (DMS), or maybe some data lake implementation and scripts could be used (apache hudi, data bricks delta lake, or other).


Best regards,

Dan Smith

On Sun, Jul 23, 2023, 22:30 Erik Serrano <eserranos@gmail.com> wrote:

Dear Masters,

Is it possible to make a logical replica between a postgresql 12 and a sql server 2019?
Technically I want to make a multimaster logical replica to a subscriber and this in turn replicates to a MSSQL 2019 AlwaysOn server...
Is it possible to make this type of structure or impossible?
Please tell me...

Regards and thank you very much

Atte.




image.png


Erik R. Serrano Saavedra
Ingeniero de Sistemas Informáticos
      Data Base Administrator
        eserranos@gmail.com
                 998596691
Вложения

Re: Is a PS - MSSQL Logical Replication Possible??

От
Ozgur Kulu
Дата:
Hi there,

Replicating between postgresql instances on different platforms (windows and linux) but database should be postgres. In other words, you should use cdc tool like debezium or attunity.

Best Regards,
Ozgur

25 Temmuz 2023 Salı tarihinde Erik Serrano <eserranos@gmail.com> yazdı:
Yes, I believe that it is not possible to carry out this architecture without an additional layer or application (which I do not know) exists. While it only occurs to me to make an unattended replica (Cron or Daemon linux in origin that generates a flat format dump (.sql) on a specific share and that windows can pick it up and in turn carry out the load through an intermediate job), I don't know... I think it could work... although it is not what I would have liked to do.

In any case, your collaboration is appreciated... and who knows, an issue will open up to be carried out

greetings friends

Sincerely


Erik R. Serrano Saavedra
Ingeniero de Sistemas Informáticos
      Data Base Administrator
        eserranos@gmail.com
                 998596691


El lun, 24 jul 2023 a las 8:42, Dan Smith (<j.daniel.smith1@gmail.com>) escribió:
I think what you are looking for is a Change Data Capture (CDC) solution.  This is going to require a system in the middle that can speak to both database systems.  Note these implementations can be brittle (replication can fail) and take some tinkering to get working well.

I recommend you look in to tools such as Debezium (https://debezium.io), AWS Data Migration Service (DMS), or maybe some data lake implementation and scripts could be used (apache hudi, data bricks delta lake, or other).


Best regards,

Dan Smith

On Sun, Jul 23, 2023, 22:30 Erik Serrano <eserranos@gmail.com> wrote:

Dear Masters,

Is it possible to make a logical replica between a postgresql 12 and a sql server 2019?
Technically I want to make a multimaster logical replica to a subscriber and this in turn replicates to a MSSQL 2019 AlwaysOn server...
Is it possible to make this type of structure or impossible?
Please tell me...

Regards and thank you very much

Atte.




image.png


Erik R. Serrano Saavedra
Ingeniero de Sistemas Informáticos
      Data Base Administrator
        eserranos@gmail.com
                 998596691
Вложения

Re: Is a PS - MSSQL Logical Replication Possible??

От
Erik Serrano
Дата:
What you indicate is interesting, I am going to find out about the debezium and attunity programs.
Thank you very much for answering

Greetings


Erik R. Serrano Saavedra
Ingeniero de Sistemas Informáticos
      Data Base Administrator
        eserranos@gmail.com
                 998596691


El mar, 25 jul 2023 a las 3:32, Ozgur Kulu (<kuluozgur@gmail.com>) escribió:
Hi there,

Replicating between postgresql instances on different platforms (windows and linux) but database should be postgres. In other words, you should use cdc tool like debezium or attunity.

Best Regards,
Ozgur

25 Temmuz 2023 Salı tarihinde Erik Serrano <eserranos@gmail.com> yazdı:
Yes, I believe that it is not possible to carry out this architecture without an additional layer or application (which I do not know) exists. While it only occurs to me to make an unattended replica (Cron or Daemon linux in origin that generates a flat format dump (.sql) on a specific share and that windows can pick it up and in turn carry out the load through an intermediate job), I don't know... I think it could work... although it is not what I would have liked to do.

In any case, your collaboration is appreciated... and who knows, an issue will open up to be carried out

greetings friends

Sincerely


Erik R. Serrano Saavedra
Ingeniero de Sistemas Informáticos
      Data Base Administrator
        eserranos@gmail.com
                 998596691


El lun, 24 jul 2023 a las 8:42, Dan Smith (<j.daniel.smith1@gmail.com>) escribió:
I think what you are looking for is a Change Data Capture (CDC) solution.  This is going to require a system in the middle that can speak to both database systems.  Note these implementations can be brittle (replication can fail) and take some tinkering to get working well.

I recommend you look in to tools such as Debezium (https://debezium.io), AWS Data Migration Service (DMS), or maybe some data lake implementation and scripts could be used (apache hudi, data bricks delta lake, or other).


Best regards,

Dan Smith

On Sun, Jul 23, 2023, 22:30 Erik Serrano <eserranos@gmail.com> wrote:

Dear Masters,

Is it possible to make a logical replica between a postgresql 12 and a sql server 2019?
Technically I want to make a multimaster logical replica to a subscriber and this in turn replicates to a MSSQL 2019 AlwaysOn server...
Is it possible to make this type of structure or impossible?
Please tell me...

Regards and thank you very much

Atte.




image.png


Erik R. Serrano Saavedra
Ingeniero de Sistemas Informáticos
      Data Base Administrator
        eserranos@gmail.com
                 998596691
Вложения

Re: Is a PS - MSSQL Logical Replication Possible??

От
Scott Ribe
Дата:
> On Jul 25, 2023, at 11:29 AM, Erik Serrano <eserranos@gmail.com> wrote:
>
> What you indicate is interesting, I am going to find out about the debezium and attunity programs.
> Thank you very much for answering

I don't know how those two are implemented--but for background on what is possible, you might want to read up on
PostgreSQLlogical replication slots. Essentially, one can write a plug-in which has access to a serialized stream of
allupdates being applied. There are examples which exist. So one could write such a thing that pushes to MSSQL, or one
coulduse such a thing which publishes change stream as JSON, and develop something to push this to MSSQL. Not
suggestingthat *you* have to this--just that it's likely tools are based on this, and it would be good for your
understanding.

Is a PS - MSSQL Logical Replication Possible??

От
"Wetmore, Matthew (CTR)"
Дата:
Correct, and with dblink on logical slots.  The gotcha is making the logical slots persistent in your network config.

I have done db link from one table to another on my primary and then use that table as the source for processing data
orwhatever. 

But you will have to do custom work.

The most annoying part of this is the MSSQL ::Postgres inconsistencies such as date, numeric and float , bit::bool
types.

-----Original Message-----
From: Scott Ribe <scott_ribe@elevated-dev.com>
Sent: Tuesday, July 25, 2023 10:42 AM
To: Erik Serrano <eserranos@gmail.com>
Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Subject: [EXTERNAL] Re: Is a PS - MSSQL Logical Replication Possible??

> On Jul 25, 2023, at 11:29 AM, Erik Serrano <eserranos@gmail.com> wrote:
>
> What you indicate is interesting, I am going to find out about the debezium and attunity programs.
> Thank you very much for answering

I don't know how those two are implemented--but for background on what is possible, you might want to read up on
PostgreSQLlogical replication slots. Essentially, one can write a plug-in which has access to a serialized stream of
allupdates being applied. There are examples which exist. So one could write such a thing that pushes to MSSQL, or one
coulduse such a thing which publishes change stream as JSON, and develop something to push this to MSSQL. Not
suggestingthat *you* have to this--just that it's likely tools are based on this, and it would be good for your
understanding.

Re: Is a PS - MSSQL Logical Replication Possible??

От
Erik Serrano
Дата:
Dear all,

Yes, I see that there is a small possibility of realizing this architecture, however I also realize that the consistency is put at risk and that the HH are high.  I am going to deliver a report giving the pros and cons of this architecture.  

Thank you all very much for your contributions.

Erik Serrano






El mar., 25 de julio de 2023 2:54 p. m., Wetmore, Matthew (CTR) <Matthew.Wetmore@express-scripts.com> escribió:
Correct, and with dblink on logical slots.  The gotcha is making the logical slots persistent in your network config.

I have done db link from one table to another on my primary and then use that table as the source for processing data or whatever.

But you will have to do custom work.

The most annoying part of this is the MSSQL ::Postgres inconsistencies such as date, numeric and float , bit::bool types.

-----Original Message-----
From: Scott Ribe <scott_ribe@elevated-dev.com>
Sent: Tuesday, July 25, 2023 10:42 AM
To: Erik Serrano <eserranos@gmail.com>
Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Subject: [EXTERNAL] Re: Is a PS - MSSQL Logical Replication Possible??

> On Jul 25, 2023, at 11:29 AM, Erik Serrano <eserranos@gmail.com> wrote:
>
> What you indicate is interesting, I am going to find out about the debezium and attunity programs.
> Thank you very much for answering

I don't know how those two are implemented--but for background on what is possible, you might want to read up on PostgreSQL logical replication slots. Essentially, one can write a plug-in which has access to a serialized stream of all updates being applied. There are examples which exist. So one could write such a thing that pushes to MSSQL, or one could use such a thing which publishes change stream as JSON, and develop something to push this to MSSQL. Not suggesting that *you* have to this--just that it's likely tools are based on this, and it would be good for your understanding.