Rick Casey <caseyrick@gmail.com> writes:
> So, I am wondering if there is any to optimize this process? I have been using Postgres for several years, but have
neverhad to partition or optimize it for files
> of this size until now.
> Any comments or suggestions would be most welcomed from this excellent forum.
The pgloader tool will import your data as batches of N lines, you get
to say how many lines you want to consider in each transaction. Plus,
you can have more than one python thread importing your big file, either
sharing one writer and having the other threads doing the parsing and
COPY, or having N independent threads doing the reading/parsing/COPY.
http://pgloader.projects.postgresql.org/
Hope this helps,
--
dim