| | <<<Back 1 day (to 2017/05/31) | 20170601 |
| dch26 | hi. i am joining pdf files together in ghostscript 8.7 on redhat. it works well but i seem to have hit a limit when joining 6000+ files. the error i am getting seems to be "io" related, but i wondered if anyone had any further ideas? Last OS error: 28 , Unrecoverable error, exit code 1 , ERROR -12 closing pdfwrite device. See gs/src/ierrors.h for code explanation | 14:04.45 |
| kens | You can't 'join' PDF files whith Ghostscript and pdfwrite | 14:05.53 |
| | You can create a new PDF file whose content shopuld be visibly the same. | 14:06.07 |
| | The first thing you shoudl do is upgrade, 8.7 is seriously old, the current version is 9.21 | 14:06.38 |
| dch26 | yes, i understand the process. | 14:07.06 |
| kens | 8.70 is 7 years old now | 14:07.09 |
| dch26 | i will try that - but 8.7 seemed to only be last year | 14:07.19 |
| | oh, is it??? wow | 14:07.27 |
| kens | That's almost certainly the last update produced bu whoever packages it | 14:07.35 |
| dch26 | yes, i will do that and come back | 14:07.40 |
| | sorry/thx | 14:07.44 |
| kens | I imagine you are on a stable Red Hat release | 14:07.46 |
| | SO they only upsdate *very* ionfrequently | 14:07.56 |
| dch26 | yup, RHEL 6.? | 14:08.04 |
| | many thanks kens | 14:08.18 |
| kens | Sorry I'm not too up to speed with the version numbers :-( | 14:08.24 |
| | -12 is an ioerror, it may simply mean you've hit the maximum file size or run out of disk space. | 14:08.58 |
| | Depends on the size of the PDF files. | 14:09.05 |
| dch26 | memory fine (unlimited) - disk space fine (50+GB free) - i was wondering about number of processes.....but i will upgrade | 14:10.03 |
| kens | In ortder to produce a new PDF file pdfwrite maintains a couple of temporary files, and all the content is stored in there. Sometimes more than once. In order to do auto compression of images, for instance, we store the data compressed with different compression filters. Then on oputput we choose the smallest one. | 14:10.05 |
| | pdfwrite is a single process, and a single thread of execution. | 14:10.23 |
| | Its conceivable that files aren't being closed properly untilthe job completes, in which case you may hit a maximum number of open files | 14:10.50 |
| chrisl | RHEL 6 dates from 2011...... | 14:11.00 |
| dch26 | it happily did 5000 files yesterday, todays batch was 6000 - it may be number of files - or maybe even some "bad" content | 14:11.31 |
| kens | A broken input file could conceivably cause a problem | 14:11.52 |
| | THe PDF itnerpreter, by default, tries to ignore errors and carry on. | 14:12.06 |
| dch26 | i know that RHEL is old - but you know customers......can't get them to upgrade unless something broken | 14:12.11 |
| kens | THe problem with that is that the underlying code exepcts to halt on errors, so if you cartry on, the state can be messed up | 14:12.31 |
| | If you are suspicious tha'ts a problem, try running with -dPDFSTOPONERROR | 14:12.52 |
| | Which will exit and stop if it encounters problems | 14:13.01 |
| dch26 | if it fails on the new version, then i will run it with debug on and see where that gets me | 14:13.06 |
| chrisl | Um, I don't thing PDFSTOPONERROR existed in 8.70 | 14:13.20 |
| kens | Umm, don't try PDFDEBUG! | 14:13.21 |
| chrisl | s/thing/think | 14:13.34 |
| kens | chrsl, you could well be right :-) | 14:13.34 |
| dch26 | lol, too verbose or doesnt work? | 14:13.37 |
| kens | PDFDEBUG is *WAY* too verbose with 6000 input files | 14:13.51 |
| | You'll never find anything int the output | 14:14.04 |
| dch26 | thanks for the warning | 14:14.05 |
| kens | It will also slow the processing down significantly | 14:14.15 |
| chrisl | Running a recent release (9.12) with -dPDFSTOPONERROR wil be a better indicator of a problem with the input | 14:14.59 |
| kens | Indeed | 14:15.07 |
| chrisl | I mean 9.21 - jeeze | 14:15.12 |
| kens | :-) | 14:15.16 |
| dch26 | i'm on it - downloading now | 14:15.24 |
| chrisl | Mind's all over the place just now..... | 14:15.35 |
| kens | dch26 : You are quite correct that its an ioerror, but that could mean a problem with the input *or* the output | 14:16.02 |
| dch26 | so the stop on error *might* show me which - it would help if it told me the filename or the number of the file it was processing | 14:17.03 |
| kens | The only other thing is, was there any other context in the back channel output ? Usually you would get some more information, such as the operator that was performed, the opeand stack dump and so on | 14:17.16 |
| dch26 | a little....hang on...... | 14:17.36 |
| kens | dch26 : In all honesty we've never really anticipated people running thousands of files..... | 14:17.42 |
| dch26 | ok to paste 20 lines of dump/log here? | 14:18.10 |
| kens | Th original intended purpose of pdfwrite was as a mimic of Adobe Acorbat Distiller, to convert PostScript to PDF | 14:18.20 |
| | Might be better to pastebin | 14:18.26 |
| chrisl | Oh my, RHEL 6 was based Fedora Core 12, which came out in 2009.... | 14:18.30 |
| kens | freenode might throrttle that much | 14:18.36 |
| | chrisl we know that a lot of people are using old 'stable' versions. Look at the NHS, still using WIndows XP :-( | 14:19.25 |
| dch26 | the purpose of me "joining" (i know!) 5000 files is because each file is approx 1mb, which includes images and fonts. they are pretty common over my pdf files, so ghostscript is re-building for me and "compressing" - so my 1.3gb of pdf files are turning into 40mb :) | 14:20.00 |
| kens | I'm surprised it saves that much | 14:20.46 |
| dch26 | oh yeah :) | 14:20.52 |
| chrisl | kens: I know - but they have to accept the trade offs..... | 14:20.52 |
| kens | If only they would :-( | 14:21.35 |
| dch26 | https://pastebin.com/yA6TRVaR | 14:23.28 |
| kens | THat would suggest a problem writing a file | 14:23.45 |
| | At a guess that was the last page of the last file, and pdfwrite had a problem writing to the final output file | 14:24.20 |
| | Hmm last OS error 28 | 14:24.44 |
| dch26 | ok, it is not free space or permissons. hmm. unless it writes multiple temporary files per input pdf - might be blowing a linux folder limit | 14:24.47 |
| kens | Anyone know what error 28 is on Linux ? | 14:24.56 |
| | It should only write a relatively small nuymber of files | 14:25.11 |
| | 3 or 4 IIRC | 14:25.21 |
| | error 28 is "No space left on device" apparently | 14:25.45 |
| dch26 | linux error 28 googles to be "no space left on device" - but i know different | 14:25.49 |
| kens | temp partition ? | 14:26.00 |
| | THa'ts where the temp files are going | 14:26.09 |
| dch26 | some searches also mention memory as a "device" | 14:26.12 |
| kens | Hmm, memory is less likely | 14:26.21 |
| dch26 | what, gs uses /tmp ?? | 14:26.26 |
| kens | Not impossible, we do have to hold a lot of context, but most stuff goes to disk | 14:26.36 |
| dch26 | my /tmp is not big | 14:26.40 |
| kens | THat'll be the problem then | 14:26.47 |
| dch26 | lol | 14:26.50 |
| kens | pdfwrite writes a lot of temporary files | 14:26.59 |
| | I believe those go to /tmp chrisl am I right ? | 14:27.09 |
| dch26 | i thought it would make temp files in the current working folder? | 14:27.18 |
| kens | Nope | 14:27.22 |
| | I believe we use the OS tmeporary files, so that if we crash they get cleaned up | 14:27.43 |
| | I am not a Linux expert though | 14:27.51 |
| | I *think* it may be possible to tell GS not to do that, give me a minute | 14:28.38 |
| dch26 | many thanks for your help guys, i have to drop off this now and swap machines. i WILL pop back on tomorrow to let you know how it panned out. :) | 14:29.09 |
| | ok, i'll hang around for 2 mins | 14:29.20 |
| kens | Looku up TMPDIR in the Ghostscript docs | 14:29.27 |
| | Yeah according to Use.htm you can set TMPDIR which changes where temp files go | 14:29.56 |
| | No idea if that will work, but its worth a try | 14:30.03 |
| | and yeah tha same doc says Unix temp files go to /tmp | 14:30.20 |
| dch26 | lovely job - on it now. will report back tomorrow :) | 14:30.20 |
| kens | Most of us are travelling tomorrow, but feel free to post and it will get logged | 14:30.36 |
| | Staff meeting in the US | 14:30.41 |
| dch26 | ok, have a good trip....you UK based? | 14:30.58 |
| kens | I am, we're spread around a bit, people in UK, Sweden, US... | 14:31.15 |
| | Hope it works for you anyway | 14:31.24 |
| dch26 | same as us, me in sunny Essex | 14:31.30 |
| chrisl | By default we use whatever mkstemp gives us | 14:31.35 |
| kens | Ah, its a nice day here today :-) | 14:31.40 |
| dch26 | hopefully with your input, my day just improved | 14:31.56 |
| kens | Well let us know if not, might be able to do something in future | 14:32.11 |
| dch26 | thanks | 14:33.02 |
| kens | coffees | 14:33.22 |
| | Forward 1 day (to 2017/06/02)>>> | |
Log of #ghostscript at irc.freenode.net.