| | <<<Back 1 day (to 2015/04/02) | 20150403 |
| jrgifford | (Might be better off in imagemagick or #rails, but figured i'd take a stab here): Anyone aware of a way to speed up PDF > JPG processing? Our thumbnails take _forever_ to generate for small PDFs... even hitting up ghostscript directly is slow. | 03:39.08 |
| | and by small i mean 500K, and by forever i mean 5 hours on a quad-core box. | 03:39.21 |
| | (and that's when I gave up) | 03:39.30 |
| chrisl | jrgifford: it's really hard to comment without seeing an example PDF. But the first question is: what version of gs are you using? | 06:54.17 |
| kens | 500Kb is not small for a PDF file. | 06:54.52 |
| | But the length of time sounds more like a bug | 06:55.04 |
| chrisl | kens: on Linux, did you install gsview in the default directory? | 09:16.00 |
| kens | Yeah I believe so | 09:16.07 |
| chrisl | so ~/gsview | 09:16.16 |
| kens | Yes | 09:16.20 |
| | Give me a minute, I'm just doing it again | 09:16.30 |
| | FWIW I had deleted the previous installation too | 09:16.40 |
| | The result is exactly the same as before | 09:18.03 |
| chrisl | okay, at the command line, can you try 'LD_PRELOAD=~/gsview/libs/libc.so.6 ~/gsview/gsview' | 09:18.09 |
| kens | ERROR: ld.so: object '~gsview/libs/libc.so.6' from LD_PRELOAD cannot be preloaded: ignored. | 09:19.21 |
| | There's no libc.so.6 in gsview/libs | 09:20.10 |
| chrisl | Really? There is in mine - hang on..... | 09:20.28 |
| kens | I'm using the 32-bit installer.... | 09:20.36 |
| | The installer I'm using is GSView-6.0-Installer-32 dated Apr 3 10:12 and is 59457505 bytes in length | 09:22.19 |
| chrisl | kens: you are correct..... strange. | 09:22.40 |
| kens | At least I'm not going completely mad :-) | 09:22.56 |
| chrisl | Going to try the 64 bit one again..... | 09:23.23 |
| kens | OK that'll be interesting. | 09:23.33 |
| chrisl | Yep, libc is in the 64 bit one. I guess it's just a config oversight between the two installers | 09:24.55 |
| kens | Yeah, I wondered if that might be the case. | 09:25.06 |
| | I'll send a followup email | 09:25.18 |
| Robin_Watts | jrgifford: Have you tried MuPDF? | 09:34.02 |
| Robin_Watts | wonders if tor is taking today off... | 10:00.37 |
| | it being good friday. | 10:00.44 |
| | paulgardiner: Do you have any thoughts on bug 694388 ? | 11:21.08 |
| paulgardiner | Sounds sensible. Not surprised it's over calling. | 11:22.43 |
| Robin_Watts | paulgardiner: If I do a commit with the change suggested in his second comment and test it, will you review it? | 11:23.27 |
| paulgardiner | Sure. Great | 11:23.37 |
| jrgifford | kens: chrisl well, for our application its small - normally it's got complicated artwork in there, and ends up weighing in around 10-20MB. | 11:28.38 |
| | As for what versions, one sec. | 11:28.53 |
| kens | OK | 11:28.57 |
| Robin_Watts | jrgifford: Can you make an example file available ? | 11:29.19 |
| jrgifford | Robin_Watts: first I've heard of it. Looks interesting, will have to dive into it. | 11:29.21 |
| Robin_Watts | For doing thumbnails mupdf is probably faster/easier than gs - assuming you don't want postscript input of course. | 11:29.49 |
| | (MuPDF is developed by us too, hence this is the right irc channel to ask questions about it) | 11:30.20 |
| jrgifford | Robin_Watts: should be able to. It's a customer submitted file, so I need to find/edit a file that doesn't say who they are. | 11:30.43 |
| | I'm using 9.10 | 11:32.51 |
| | On Ubuntu 14.04 (64) | 11:33.11 |
| kens | You should probably cosider upgrading then, the current version is 9.16 | 11:33.11 |
| | It may be a fixed bug, if you can let us look at a file we could tell more easily (or if its not fixed, but is still a bug....) | 11:33.42 |
| chrisl | jrgifford: OKay, can you try adding -dNOINTERPOLATE to your command line | 11:33.53 |
| kens | Or it could be something Ubuntu have done, we've recently ahd an experience of that with font handling | 11:34.01 |
| chrisl | (and removing -dDOINTERPOLATE if it's there) | 11:34.38 |
| kens | Hallelujah, I may have an answer to this stupid broken PDF file, its only taken a day and a half :-( And I still have the second problem to look at, which the customer didn't even notice. I'm tempted to send it back and wait until they do. | 11:35.42 |
| kens | lunches, if jrgifford can post an example file, I'll look at it when I get back. If so, a command line which exhibits the problem would be good. | 11:40.15 |
| jrgifford | chrisl: ok, so tried dNOINTERPOLATE - no change. | 11:55.33 |
| | I'n not in the office yet, so I'll dive into it more in an hour or so. | 11:55.49 |
| chrisl | jrgifford: well, the other thing would be -dNOTRANSPARENCY | 11:56.20 |
| | jrgifford: Just checking: you are testing this by calling gs directly, and not through imagemagick? | 12:13.00 |
| jrgifford | Direct yes | 12:14.56 |
| chrisl | OKay, just checking - I *think* imagemagick calls gs with -dDOINTERPOLATE | 12:16.30 |
| | jrgifford: can you post your command line? | 12:16.42 |
| Robin_Watts | One of the tasks that dropped on myself and ray at the staff meeting was writing a quick guide for when to use MuPDF and when to use GS. | 12:54.36 |
| | This is my first draft: http://twiki.ghostscript.com/do/view/Ghostscript/GhostscriptOrMuPDF | 12:54.44 |
| | opinions gratefully received | 12:55.15 |
| | It should couple up with: http://twiki.ghostscript.com/do/view/MuPDF/EmbeddingMuPDF | 12:56.18 |
| jrgifford | chrisl: ok, so what i found is this: On my mac (not the linux box), using 9.15 (thanks homebrew), it processes. | 13:07.57 |
| | So before I ask another question, I'm going to spin up a new VM, install the newest ghostscript | 13:08.21 |
| kens | THat would seem like the most sensible approach | 13:08.32 |
| jrgifford | and then try it. | 13:08.34 |
| kens | Can I suggest you get and build the sources, rather than trying to use a package manager ? | 13:08.56 |
| | THe latest version often isn't packaged anyway, and the packagers sometimes do 'stuff' to Ghostscript | 13:09.26 |
| jrgifford | sure! | 13:09.29 |
| | it's a VM, i don't care. :ð | 13:09.44 |
| kens | :-D | 13:09.49 |
| | It does sort of sound like its a fixed bug though | 13:09.58 |
| chrisl | jrgifford: you can grab a cut-down, prebuilt exe here: http://www.ghostscript.com/download/gsdnld.html | 13:10.26 |
| | It *may* work - vagaries of Linux distros permitting | 13:10.49 |
| jrgifford | chrisl: well, i guess i need to setup new ghostscript binaries. | 13:18.54 |
| | that did it. | 13:18.56 |
| chrisl | jrgifford: The binary from above? | 13:19.24 |
| jrgifford | chrisl: yes | 13:19.30 |
| | if I can get permission to release a file, I will open a bug report with details. otherwise, i'll spend a few hours trying to build a PDF that causes this issue, and then submit a bug report. | 13:19.51 |
| kens | Umm, there's probably no point, if the current code doesn't exhibit the problem | 13:20.14 |
| jrgifford | ok. | 13:20.18 |
| | that works too. ;-) | 13:20.22 |
| kens | We would just close the report as 'worksforme' | 13:20.35 |
| jrgifford | coolio. | 13:20.43 |
| | :D thanks everyone. | 13:20.52 |
| chrisl | You can probably use the binary from our site, as long as it covers all the features you need | 13:20.56 |
| kens | We don't try to fix old versions, exept for commercial customers | 13:20.56 |
| jrgifford | chrisl: yeah, that's the next step. testing all the features of the application for that binary. | 13:24.58 |
| chrisl | jrgifford: well, if you know what devices you require, that's easy to check | 13:25.47 |
| Robin_Watts | paulgardiner: many commits on robin/master | 14:03.43 |
| | Probably you can review the first 3. | 14:03.58 |
| | The rest I suspect tor will want to weigh in on. | 14:04.08 |
| paulgardiner | Okay. Will look now | 14:05.12 |
| mvrhel_laptop | kens and Robin_Watts thanks for all the feedback. I will spend today trying to fix some of these things | 14:08.10 |
| kens | NP | 14:08.20 |
| Robin_Watts | mvrhel_laptop: No worries. It looks really good. | 14:08.24 |
| kens | SOMe of the weirdness may be hard to reproduce | 14:08.30 |
| mvrhel_laptop | well, I can see some scrolling issues | 14:08.44 |
| | and the zoom resize when when scrolling I can see | 14:08.54 |
| kens | O that's better than I thought it might be :-) | 14:09.09 |
| mvrhel_laptop | and all the issues Robin_Watts had I could see except that DLL issue he is getting | 14:09.25 |
| | I am suspicious about that one | 14:09.34 |
| kens | Yeah that's fixed for me now | 14:09.35 |
| Robin_Watts | mvrhel_laptop: Is there an additional test you'd like me to do ? | 14:09.50 |
| mvrhel_laptop | What size is the installer that you used? | 14:09.53 |
| Robin_Watts | 25630738 | 14:10.10 |
| mvrhel_laptop | ok that is the right size | 14:10.46 |
| Robin_Watts | Let me uninstall/reinstall. | 14:10.56 |
| mvrhel_laptop | well wait | 14:11.00 |
| | look in Program Files | 14:11.06 |
| | Artifex | 14:11.09 |
| | gsview | 14:11.12 |
| | bin | 14:11.14 |
| Robin_Watts | too slow. sorry. | 14:11.23 |
| mvrhel_laptop | what size is msvcr120 | 14:11.53 |
| | what size is msvcr120.cll | 14:11.59 |
| | dll | 14:12.01 |
| Robin_Watts | ooh. installer crash. | 14:12.03 |
| mvrhel_laptop | really? | 14:12.31 |
| Robin_Watts | It complains that it can't open C:\Program Files\Artifex Software\gsview6.0\bin\mupdfnet64.dll for writing. | 14:13.01 |
| mvrhel_laptop | hmm what is in the folder? | 14:13.18 |
| Robin_Watts | and indeed, the installation hasn't removed a load of files. | 14:13.23 |
| mvrhel_laptop | did you uninstall? | 14:13.34 |
| Robin_Watts | I did. | 14:13.38 |
| | That's why I said "too slow" | 14:13.43 |
| mvrhel_laptop | ok that is weird. I will try that here too | 14:13.59 |
| kens | I hsould try the uninstaller too, I did on Linux.... | 14:14.06 |
| mvrhel_laptop | is msvcr120.dll there? | 14:14.06 |
| Robin_Watts | http://pastebin.com/b43u3vLd | 14:14.13 |
| paulgardiner | Robin_Watts: all three lgtm | 14:14.22 |
| Robin_Watts | Let me rm -rf that, then reinstall. | 14:14.28 |
| | paulgardiner: Thanks. | 14:14.32 |
| mvrhel_laptop | oh! | 14:14.47 |
| | you don't have msvcr120 | 14:14.53 |
| | brb | 14:14.57 |
| | have to run son to school | 14:15.04 |
| Robin_Watts | mvrhel_laptop: That's what's left after I uninstalled. | 14:15.19 |
| | ooh. I can't rm -rf them. Permission denied. | 14:15.42 |
| mvrhel_laptop | is the application running? | 14:16.57 |
| | that may be why the installer did not delete them | 14:17.13 |
| Robin_Watts | Ok, apparently it was running, but I didn't have a window up... | 14:17.58 |
| | I pskilled it. | 14:18.05 |
| mvrhel_laptop | so wait, there was no window hidden either, it was just running? | 14:18.25 |
| Robin_Watts | There was certainly nothing on the task bar. | 14:18.49 |
| mvrhel_laptop | ok that is not good. i need to figure out how you did that | 14:19.02 |
| Robin_Watts | go take son to school. | 14:19.03 |
| mvrhel_laptop | ok bbiab | 14:19.10 |
| kens | Hmm, wehn I uninstall gsview it leaves teh Artifex Software/gsview6.0/bin folder tree and leaves gsprint64.dll, gsview.exe, msvcr120.dll abd mupdfnet64.dll in that folder. | 14:31.00 |
| mvrhel_laptop | it certainly should not do that | 14:31.24 |
| | unless you are having the same issue as Robin_Watts | 14:31.39 |
| kens | I htink I must be | 14:31.45 |
| Robin_Watts | kens: is gsview running? | 14:31.54 |
| kens | Just got amn error 'cannot open file for writing' | 14:31.58 |
| Robin_Watts | use psexplorer to see. | 14:31.59 |
| kens | 2 secs | 14:32.05 |
| Robin_Watts | yeah, I think you'll find there is a gsview process somewhere. | 14:32.15 |
| kens | yes gsview.exe is running | 14:32.23 |
| Robin_Watts | procexp, sory. | 14:32.25 |
| kens | No window, but its htere | 14:32.28 |
| | Robin_Watts : task manager suffices :-) | 14:32.36 |
| mvrhel_laptop | ok. I will need to see if I can duplicate that. I dont see that on windows 8 | 14:32.57 |
| kens | I'll try again | 14:33.04 |
| kens | thought michael was off | 14:33.27 |
| mvrhel_laptop | i am back | 14:33.31 |
| kens | That was fast | 14:33.36 |
| | Hmm file-exit isn't working, still have a window | 14:33.57 |
| | close button works, but..... | 14:34.06 |
| mvrhel_laptop | huh? | 14:34.08 |
| | exit does not work | 14:34.14 |
| kens | Seems not | 14:34.21 |
| | But the red x works | 14:34.26 |
| | And tehre is now no process running | 14:34.37 |
| | NB I had no file open, just launched and tried to exit | 14:35.08 |
| mvrhel_laptop | oh I get the same issue on win 7 | 14:35.14 |
| kens | Same again | 14:35.15 |
| mvrhel_laptop | well no there it went away | 14:35.21 |
| kens | Hmm, still htere for me | 14:35.28 |
| | How long did you have to wait ? | 14:35.37 |
| mvrhel_laptop | exit is slooow on win 7 | 14:35.56 |
| | weird | 14:35.57 |
| | actually this time it did not work | 14:36.05 |
| kens | Ah, not just me then | 14:36.13 |
| mvrhel_laptop | ok, I should be able to debug this | 14:36.13 |
| kens | The close button at top right does work | 14:36.25 |
| mvrhel_laptop | add it to my growing list..... | 14:36.32 |
| kens | :-( | 14:36.39 |
| mvrhel_laptop | its ok, this is the way testing and bug tails go | 14:36.57 |
| Robin_Watts | Never test. | 14:37.15 |
| kens | If I have a file open, file->exit works | 14:37.17 |
| mvrhel_laptop | yes, that is what I am seeing kens | 14:37.27 |
| kens | and gsvioew.exe is gone | 14:37.28 |
| Robin_Watts | Do, or do not. There is no test. | 14:37.45 |
| mvrhel_laptop | :) | 14:37.51 |
| | I do want to know how Robin_Watts had a process running but no window open | 14:38.03 |
| kens | I did have that too | 14:38.10 |
| mvrhel_laptop | need to find out how have that happen | 14:38.16 |
| kens | I don't know how | 14:38.16 |
| Robin_Watts | mvrhel_laptop: If I can reproduce it, I will let you know. | 14:38.27 |
| mvrhel_laptop | Robin_Watts: can you reinstall and see if you have msvcr120? | 14:38.48 |
| Robin_Watts | I have reinstalled. | 14:39.39 |
| mvrhel_laptop | I wonder if some background threads are not getting shut down. | 14:40.00 |
| Robin_Watts | http://pastebin.com/Wss1zWcu | 14:40.24 |
| | No msvcr120 that I can see. | 14:40.31 |
| kens | "Debugging? Klingons do not debug. Our software does not coddle the weak. | 14:40.39 |
| mvrhel_laptop | :) | 14:40.47 |
| | oh so you must have msvcr120.dll in Windows/System32 | 14:41.03 |
| Robin_Watts | I do. | 14:41.29 |
| | 970912 Oct 5 2013 /c/Windows/System32/msvcr120.dll | 14:41.41 |
| mvrhel_laptop | oh interesting | 14:42.08 |
| | so that is different then mine. I wonder if there are different versions of this library | 14:42.21 |
| | other than 32bit vs 64bit | 14:42.28 |
| kens | There almost always are | 14:42.31 |
| mvrhel_laptop | ok, then I guess me being clever to check is not so clever | 14:42.43 |
| | and I should just install it. | 14:42.51 |
| | that is what MS suggests | 14:43.00 |
| Robin_Watts | I'd say always install it locally. | 14:43.00 |
| mvrhel_laptop | yes | 14:43.05 |
| | ok, I will fix that | 14:43.10 |
| | thanks Robin_Watts | 14:43.13 |
| Robin_Watts | np. | 14:43.18 |
| mvrhel_laptop | ok, now I need to go help get daughter out door, this may take a bit longer.... | 14:43.51 |
| Robin_Watts | rayjj, henrys: You remember at the meeting that Ray and I got tasked with 'write a document to say when to use gs and when to use mupdf' ? | 15:19.08 |
| | First draft: http://twiki.ghostscript.com/do/view/Ghostscript/GhostscriptOrMuPDF | 15:19.32 |
| | and I found this while looking at the Twiki, which looks like it might be related: http://twiki.ghostscript.com/do/view/MuPDF/EmbeddingMuPDF | 15:20.04 |
| henrys | Robin_Watts: can you send a link to tech and solicit feedback? | 15:23.46 |
| Robin_Watts | ok. | 15:24.45 |
| mvrhel_laptop | fixed the exit problem and the installer dll issue. found the issue with the thumbs progress rendering. that fix requires a bit more work | 16:14.19 |
| | there is a competition for ui updates as the progress bar vs setting the pages to the thumb renderings | 16:47.06 |
| | and updating the pages is hogging everything | 16:47.28 |
| Robin_Watts | So when the thumbnail rendering finishes, you set the page to have that as its bitmap? | 16:48.23 |
| | let me try that again. | 16:48.31 |
| | You have a process in the background that runs through each page rendering it to a thumbnail. | 16:48.50 |
| | Every time a thumbnail rendering finishes, you 'set' the appropriate page in the main window to have that as its representation? | 16:49.26 |
| | Does windows have the concept of an 'idle' message? | 16:50.10 |
| | I bet you're not working at the GetMessage level, are you? | 16:52.08 |
| mvrhel_laptop | oh I think I see why I am having this issue | 16:52.37 |
| | it has to do with the type of bindings | 16:52.45 |
| | the update binding for the pages is somewhat special | 16:53.12 |
| | I suspect I need to move the progress bar update to the same type | 16:53.27 |
| | essentially the "source" for the xaml page content is set, and if a change occurs it will update | 16:54.08 |
| | the progress bar is a bit different | 16:54.24 |
| | I simply set its value as I the pages are set to the thumbnails | 16:54.46 |
| | I am going to do a bit of refactoring here | 16:55.03 |
| Robin_Watts | mvrhel_laptop: I was going to suggest that after you set each new page source to the new thumbnail, you should wait until the message loop is empty before doing the next one. | 16:56.38 |
| mvrhel_laptop | Robin_Watts: unfortunately, that stuff is all hidden from me in this managed environment | 16:57.02 |
| Robin_Watts | MFC has an 'OnIdle' call for that. | 16:57.14 |
| | it wouldn't surprise me to find there was a .net equivalent. | 16:57.31 |
| | but you have a plan, so I'll shut up and let you try it out. | 16:57.47 |
| | (Yeah, .net has the Application.Idle event) | 16:58.44 |
| mvrhel_laptop | I can see what that is about | 17:00.23 |
| Robin_Watts | Cool. My burbling is all ignorant speculation so just ignore it :) | 17:01.15 |
| mvrhel_laptop | but I suspect that I should make my own progress class with INotifyPropertyChanged | 17:01.38 |
| | and bind the value to the progress bar xaml | 17:02.11 |
| | will try that first to see what happens | 17:02.54 |
| | ok. that did absolutely nothing for me | 17:42.31 |
| | let me look at Application.Idle | 17:42.47 |
| Robin_Watts | mvrhel_laptop: A cheesy test that you can do quickly... | 17:44.09 |
| | In the thrumbnail generation thread, put 'Sleep(500);' after setting the thumbnail. | 17:44.58 |
| | See if that solves the issue. | 17:45.01 |
| | If it doesn't, then Application.idle won't help. | 17:45.18 |
| mvrhel_laptop | I think I see how I need to do this now | 17:47.07 |
| Robin_Watts | Did you try the Sleep thing? | 17:48.00 |
| mvrhel_laptop | instead of a background thread that does progress calls back to the UI thread to do updates, I am going to do a standard thread that does dispatches to the ui thread | 17:48.21 |
| Robin_Watts | I'd hate for my fevered imaginings to lead you down a blind alley. | 17:48.25 |
| mvrhel_laptop | Robin_Watts: no I have not tried the sleep yet. so the way the background threads work is the following | 17:49.01 |
| Robin_Watts | mvrhel_laptop: You plan to have you background thread say "here is a new thumbnail" | 17:49.12 |
| mvrhel_laptop | here is what we have now: | 17:49.25 |
| Robin_Watts | and your ui thread to pick that up, set the thumbnail, and set the progress at the same time ? | 17:49.28 |
| mvrhel_laptop | that is what we have now | 17:49.36 |
| | but it is running in a background thread, which does updates to the ui through is progress call backs | 17:51.11 |
| | after reading a bit, I am thinking that I should do a standard thread (not a background thread) and do dispatches to the ui thread to do the updates. it is a subtle difference but the dispatching is supposed to result in very responsive ui. I actually did this for the printing thread due to some funny ownership issues | 17:52.32 |
| | I think I am trying to pack too much into the progress call back with the standard background thread object | 17:53.07 |
| Robin_Watts | I can't claim to have a strong feeling for any of this, but... ok. | 17:56.05 |
| | dispatches to the ui thread are what? Sending messages to the ui thread? | 17:56.41 |
| | I can't really see how that differs much from what you're doing now. | 17:56.53 |
| | but you know more than me. | 17:57.29 |
| fredross-perry | gang: see email regarding 32-bit linux. Thanks. | 17:58.37 |
| mvrhel_laptop | aha! | 18:00.13 |
| | I found a very simple solution to this | 18:00.19 |
| | I don't set the binding until the thumbs are donw | 18:00.32 |
| | done | 18:00.33 |
| | then everything is quite fast | 18:00.41 |
| | and the progress bar works | 18:00.47 |
| Robin_Watts | but then no thumbnails appear until they all do ? | 18:00.53 |
| mvrhel_laptop | essentially nothing is shown until it finishes them | 18:03.01 |
| Robin_Watts | That's a shame. | 18:03.11 |
| mvrhel_laptop | yes I am going to work on my dispatch approach | 18:03.39 |
| Robin_Watts | cool. | 18:03.46 |
| mvrhel_laptop | bbiaw | 18:09.47 |
| Robin_Watts | tor8: For the logs... Loads of commits on robin/master | 19:04.34 |
| | Forward 1 day (to 2015/04/04)>>> | |
IRC Logs