Re: COPY enhancements

On Mon, 5 Oct 2009, Josh Berkus wrote:

>> I think that this was the original idea but we should probably rollback
>> the error logging if the command has been rolled back. It might be more
>> consistent to use the same hi_options as the copy command. Any idea what
>> would be best?
>
> Well, if we're logging to a file, you wouldn't be *able* to roll them
> back.  Also, presumbly, if you abort a COPY because of errors, you
> probably want to keep the errors around for later analysis.  No?

Absolutely, that's the whole point of logging to a file in the first 
place.  What needs to happen here is that when one is aborted, you need to 
make sure that fact is logged, and with enough information (the pid?) to 
tie it to the COPY that failed.  Then someone can crawl the logs to figure 
out what happened and what data did and didn't get loaded.  Ideally you'd 
want to have that as database table information instead, to allow 
automated recovery and re-commit in the cases where the error wasn't 
inherent in the data but instead some other type of database failure.

I know this patch is attracting more reviewers lately, is anyone tracking 
the general architecture of the code yet?  Emmanuel's work is tough to 
review just because there's so many things mixed together, and there's 
other inputs I think should be considered at the same time while we're all 
testing in there (such as the COPY patch Andrew Dunstan put together).

What I'd like to see is for everything to get broken more into component 
chunks that can get commited and provide something useful one at a time, 
because I doubt taskmaster Robert is going to let this one linger around 
with scope creep for too long before being pushed out to the next 
CommitFest.  It would be nice to have a clear game plan that involves the 
obvious 3 smaller subpatches that can be commited one at a time as they're 
ready to focus the work on, so that something might be polished enough for 
this CF.  And, yes, I'm suggesting work only because I now have some time 
to help with that if the idea seems reasonable to persue.  Be glad to set 
up a public git repo or something to serve as an integration point for 
dependency merge management and testing that resists bit-rot while 
splitting things up functionally.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD