Обсуждение: [QUESTIONS] lo_write cannot > 640Kb? memory leaks?
Hi all, Can anybody pinpoint where is wrong on lo_write(and possibly lo_read)? I tried to lo_import(and lo_write) few MegaBytes of data with large_object but it was painfully SLOW! So I simplified the problem, just test "testlo.c" under test/examples directory with few MegaBytes of file X.dat and printout debug messages like this with lo_import ()-> importFile() in main() IT CANNOT LO_WRITE > 640KB !!! WHY? testlo.c------------------- ... in importFile() { ... // with BUFSIZE -> 1024 or 8*1024 or 64*1024 i.e. 1K, 8K, 64K, ... /* * read in from the Unix file and write to the inversion file */ while ((nbytes = read(fd, buf, BUFSIZE)) > 0) { printf("COUNT [%d] %d Kb\n", ++count, count*BUFSIZE/1024); tmp = lo_write(conn, lobj_fd, buf, nbytes); ... if (tmp < nbytes) { fprintf(stderr, "error while reading \"%s\"", filename); } printf("READ %d , WRITE %d\n", nbytes, tmp); _________________________ ./testlo template1 X.dat out .... importing file "in" ... COUNT [1] 0 Kb READ 65536 , WRITE 65536 COUNT [2] 64 Kb ... READ 65536 , WRITE 65536 COUNT [9] 512 Kb error while reading "in"READ 65536 , WRITE -1 COUNT [10] 576 Kb error while reading "in"READ 65536 , WRITE -1 COUNT [11] 640 Kb ________________________________________________ and "strace ./testlo template1 in out" shows ... write(3, "F \0", 3) = 3 write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024 write(3, "\331\5\200\t\317\t\331]\334\331E"..., 64512) = 64512 write(3, "\213\r\30Lv\10I\211\312\1\322\1\312"..., 20) = 20 read(4, "EERROR: no empty local buffer.\n"..., 1024) = 33 write(2, "error while reading \"in\"", 24error while reading "in") = 24 read(5, "\0\0\0\0\1\310\272\300\276\345\t"..., 65536) = 65536 write(1, "COUNT [10] 576 Kb\n", 20COUNT [10] 576 Kb ) = 20 write(3, "F \0", 3) = 3 write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024 write(3, "\'j\4j\4h\4\273j\10\241\30Lv\10H"..., 64512) = 64512 write(3, "\331\311\331]\370\331E\370\336\311"..., 20) = 20 read(4, "EERROR: cannot read block 62 of"..., 1024) = 45 write(2, "error while reading \"in\"", 24error while reading "in") = 24 read(5, "u5hp\366v\10\350pk\374\377\203\304"..., 65536) = 65536 write(1, "COUNT [11] 640 Kb\n", 20COUNT [11] 640 Kb ... ___________________________________________________________ So I lo_open/lo_lseek(SEEK_END)/lo_write/lo_close() INSIDE of the loop, it solved the problem, which means MEM LEAKS or buffer size problem? And just lobj_fd = lo_open(conn, lobjId, INV_WRITE| INV_READ); ... /* * read in from the Unix file and write to the inversion file */ while ((nbytes = read(fd, buf, BUFSIZE)) > 0) { printf("COUNT [%d] %d Kb\n", ++count, count*BUFSIZE/1024); o_lseek(conn, lobj_fd, 0, SEEK_END); tmp = lo_write(conn, lobj_fd, buf, nbytes); ... Also solved the problem... But I want reliable(lo_read/lo_write) large object with BIG data. Please Help me out. C.S.Park
Re: [HACKERS] [QUESTIONS] lo_write cannot > 640Kb? memory leaks?
От
dg@illustra.com (David Gould)
Дата:
> Hi all, > > Can anybody pinpoint where is wrong on lo_write(and possibly > lo_read)? I tried to lo_import(and lo_write) few MegaBytes of data > with large_object but it was painfully SLOW! So I simplified the > problem, just test "testlo.c" under test/examples directory > > with few MegaBytes of file X.dat > > and printout debug messages like this with lo_import ()-> > importFile() in main() > > IT CANNOT LO_WRITE > 640KB !!! WHY? > > testlo.c------------------- > ... > in importFile() > { > ... > // with BUFSIZE -> 1024 or 8*1024 or 64*1024 i.e. 1K, 8K, 64K, ... > /* > * read in from the Unix file and write to the inversion file > */ > while ((nbytes = read(fd, buf, BUFSIZE)) > 0) > { > printf("COUNT [%d] %d Kb\n", ++count, count*BUFSIZE/1024); > tmp = lo_write(conn, lobj_fd, buf, nbytes); > ... > if (tmp < nbytes) > { > fprintf(stderr, "error while reading \"%s\"", filename); > } > printf("READ %d , WRITE %d\n", nbytes, tmp); > _________________________ > > ./testlo template1 X.dat out > > .... > importing file "in" ... > COUNT [1] 0 Kb > READ 65536 , WRITE 65536 > COUNT [2] 64 Kb > ... > READ 65536 , WRITE 65536 > COUNT [9] 512 Kb > error while reading "in"READ 65536 , WRITE -1 > COUNT [10] 576 Kb > error while reading "in"READ 65536 , WRITE -1 > COUNT [11] 640 Kb > ________________________________________________ > > and "strace ./testlo template1 in out" shows > > ... > write(3, "F \0", 3) = 3 > write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024 > write(3, "\331\5\200\t\317\t\331]\334\331E"..., 64512) = 64512 > write(3, "\213\r\30Lv\10I\211\312\1\322\1\312"..., 20) = 20 > read(4, "EERROR: no empty local buffer.\n"..., 1024) = 33 > write(2, "error while reading \"in\"", 24error while reading "in") = 24 > read(5, "\0\0\0\0\1\310\272\300\276\345\t"..., 65536) = 65536 > write(1, "COUNT [10] 576 Kb\n", 20COUNT [10] 576 Kb > ) = 20 > write(3, "F \0", 3) = 3 > write(3, "\0\0\3\273\0\0\0\2\0\0\0\4\0\0\0"..., 1024) = 1024 > write(3, "\'j\4j\4h\4\273j\10\241\30Lv\10H"..., 64512) = 64512 > write(3, "\331\311\331]\370\331E\370\336\311"..., 20) = 20 > read(4, "EERROR: cannot read block 62 of"..., 1024) = 45 > write(2, "error while reading \"in\"", 24error while reading "in") = 24 > read(5, "u5hp\366v\10\350pk\374\377\203\304"..., 65536) = 65536 > write(1, "COUNT [11] 640 Kb\n", 20COUNT [11] 640 Kb > ... > ___________________________________________________________ > > So I lo_open/lo_lseek(SEEK_END)/lo_write/lo_close() INSIDE of the loop, > it solved the problem, which means MEM LEAKS or buffer size problem? > And just > lobj_fd = lo_open(conn, lobjId, INV_WRITE| INV_READ); > ... > /* > * read in from the Unix file and write to the inversion file > */ > while ((nbytes = read(fd, buf, BUFSIZE)) > 0) > { > printf("COUNT [%d] %d Kb\n", ++count, count*BUFSIZE/1024); > o_lseek(conn, lobj_fd, 0, SEEK_END); > tmp = lo_write(conn, lobj_fd, buf, nbytes); > ... > > Also solved the problem... But I want reliable(lo_read/lo_write) large > object with BIG data. > > Please Help me out. > > C.S.Park Have you tried increasing the number of postgres buffer cache buffers? I am speculating that the lo_write() is using these buffers but perhaps not flushing them. As the default configuration has way too few buffers anyway this might be the problem. Could you try increasing the buffers to say 1024 or so and rerun your test and post the results? Thanks, -dg David Gould dg@illustra.com 510.628.3783 or 510.305.9468 Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612 "Of course, someone who knows more about this will correct me if I'm wrong, and someone who knows less will correct me if I'm right." --David Palmer (palmer@tybalt.caltech.edu)
Hi, > Have you tried increasing the number of postgres buffer cache buffers? I am > speculating that the lo_write() is using these buffers but perhaps not > flushing them. As the default configuration has way too few buffers anyway > this might be the problem. Could you try increasing the buffers to say 1024 or > so and rerun your test and post the results? > > -dg Thanks for your reply, but as I posted I tested many different size of buffers in lotest.c, e.g. 1Kb, 4Kb, 8Kb, 9Kb, 10Kb, 16Kb, 32Kb, 64Kb, 640Kb, 1Mb, 10Mb, ... or your "cache" or "buffer" size have some diffferent meaning? but lo_write cannot exceed 640kb always, only open/write(less than 640kb)/close looping OR open LOOP lo_write(buffer<640Kb)) , lo_lseek to SEEK_END, write, ... seems to solve this problem(as in my last email), and I found that 16Kb buffer seems to be the optimal buffer size for the SPEED(not 8Kb) for network & socket connections why?... and I think that above lo_lseek write loop is not a real solution, because lo_read seems does not have such defects(but 16kb buffer seems to be the optimal size also), there must be some problems in lo_write or buffer managing... does anybody have some guess or fixes? c.s.park
Re: [HACKERS] [QUESTIONS] lo_write cannot > 640Kb? memory leaks?
От
dg@illustra.com (David Gould)
Дата:
> Hi, > > > Have you tried increasing the number of postgres buffer cache buffers? I am > > speculating that the lo_write() is using these buffers but perhaps not > > flushing them. As the default configuration has way too few buffers anyway > > this might be the problem. Could you try increasing the buffers to say 1024 or > > so and rerun your test and post the results? > > > > -dg > > Thanks for your reply, but as I posted I tested many different size of > buffers in lotest.c, e.g. 1Kb, 4Kb, 8Kb, 9Kb, 10Kb, 16Kb, 32Kb, 64Kb, 640Kb, > 1Mb, 10Mb, ... or your "cache" or "buffer" size have some diffferent meaning? Yes, my "cache" or "buffer" size has a different meaning. I am referring to the size of the postgres buffer pool which can be changed by a command line option when you start the postmaster. I forget what the exact option is but it is something like "-o -B <number_of_buffers>". It is documented I think. -dg David Gould dg@illustra.com 510.628.3783 or 510.305.9468 Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612 "Of course, someone who knows more about this will correct me if I'm wrong, and someone who knows less will correct me if I'm right." --David Palmer (palmer@tybalt.caltech.edu)