Hi,
I'm using Postgre's COPY feature to populate a large table (~500M
rows, 100 GB) with frequent updates from CSV text files. Currently
I'm using psycopg's copy_expert():
cur.copy_expert("COPY Table (c1, c2, ...) FROM STDIN WITH CSV",
open(filename))
This avoids any issue with client vs. server file permissions, but I'm
wondering if it carries a significant performance penalty by reading
the text file at the Python level rather than the OS level. Instead I
could do:
cur.execute("COPY Table (c1, c2, ...) FROM $$%s$$ WITH CSV" % fn)
Of course, this opens up a whole host of permissions and security
issues. Is there any reason to believe that the Python file interface
significantly slows down COPY FROM?
Thanks,
Dan