Re: How to schedule long running SQL job

Поиск
Список
Период
Сортировка
От Doug Reynolds
Тема Re: How to schedule long running SQL job
Дата
Msg-id 0100018970b9776d-9a599f5a-f267-4de9-b014-6de7f0ec010c-000000@email.amazonses.com
обсуждение исходный текст
Ответ на How to schedule long running SQL job  (M Sarwar <sarwarmd02@outlook.com>)
Ответы RE: How to schedule long running SQL job
Список pgsql-admin
Run a postgres container on Fargate, dumping CSV from psql to a file, upload to S3.  Kick job from CloudWatch cron trigger.

You could do a lambda, but it would time out after 15 minutes.

Sent from my iPhone

On Jul 19, 2023, at 7:41 PM, M Sarwar <sarwarmd02@outlook.com> wrote:


Hi,
Db: Postgres
Cloud: aws / RDS
Tool using so far: PgAdmin

SQL got aborted after running for 1 hr 40 minutes from pgadmin.
I need CSV file output.
What is the recommended approach for this issue.

SQL:

SELECT P1.FILE_ID,

                F.FILENAME,

                P1.STAGE,

                P1.SERIAL_NUMBER,

                P1.TEST_IMAGE_SET_VALUE,

                SUBSTRING(P1.SERIAL_NUMBER, '[A-Z][0-9]+') AS MCM_ID_SERIAL_NUMBER,

                P1.RUN_ID,

                SPLIT_PART(P1.RUN_ID, ':', 1) TEST_LOT

FROM BRONX.TEST_PART_DETAILS_ALL_MCM P1,

                BRONX.FILES_METADATA F

WHERE F.FILE_ID = P1.FILE_ID

                                AND EXISTS         (              SELECT SUBSTRING(P2.SERIAL_NUMBER, '[A-Z][0-9]+') AS MCM_ID2 ,

                                                                                                                COUNT(DISTINCT SPLIT_PART(P2.RUN_ID, ':', 1)) AS NUM_DISTINCT_TEST_LOTS

                                                                                FROM BRONX.TEST_PART_DETAILS_ALL_MCM P2

                                                                               WHERE SUBSTRING(P2.SERIAL_NUMBER, '[A-Z][0-9]+') = SUBSTRING(P1.SERIAL_NUMBER, '[A-Z][0-9]+')

                                                                                GROUP BY MCM_ID2

                                                                                HAVING COUNT(DISTINCT SPLIT_PART(P2.RUN_ID, ':', 1))  > 1

                                                                )

order BY               MCM_ID_SERIAL_NUMBER,

                                                TEST_LOT ;


I used to use crontab or OEM to schedule these types jobs in the past. Now it is on RDS/AWS.


Thanks,
Sarwar

В списке pgsql-admin по дате отправления:

Предыдущее
От: M Sarwar
Дата:
Сообщение: Re: How to schedule long running SQL job
Следующее
От: M Sarwar
Дата:
Сообщение: RE: How to schedule long running SQL job