Log of #ghostscript at irc.freenode.net.

Search:
 <<<Back 1 day (to 2016/09/26)20160927 
ray_pc I don't have a good feeling about this. Doing Windows update and it has taken 20 minutes so far :-(00:45.27 
  what't that chance that everything will still work ?00:45.58 
  so far it has rebooted about 12 times01:09.16 
  mow it asks me to login and say "Getting things ready..."01:13.55 
  finally up. Email tool works01:17.41 
  hmm... even though this was supposed to be ther "anniversary update", I don't see the bash stuff (trying various hints frp, google_01:30.46 
  ahh, afterI turned on developer options,I rebooted (no prompt, but goolge said I might have to) and ow I see the "Windows Subsytem for Linux (Beta)" option01:36.14 
  which (no surpise here) require YET ANOTHER REBOOT. I think that brings me up to about 1601:36.49 
  time to get dinner for the family ...01:37.57 
sebras Robin_Watts: tor8: four gif-related fuzzing issues fixed on sebras/master05:27.27 
  ... and a jpeg-crash fix.06:22.23 
  kens: morning!06:54.33 
kens morning sebras06:54.39 
Robin_Watts morning sebras08:38.53 
  all lgtm.08:41.52 
  chrisl: Do you remember how the openjpeg stuff works?09:19.49 
  Do we generate the gs openjpeg stuff from mupdf? Or vice versa? or are they separate?09:20.10 
kens2 I thought they were separate09:24.29 
Robin_Watts fab.09:24.36 
kens2 Don't quote me though, its not my area09:24.50 
chrisl Robin_Watts: No idea.....09:33.53 
Robin_Watts I suspect they are completely separate then.09:34.08 
chrisl I'll have a quick look on casper , there might be a hint in there09:34.32 
  Robin_Watts: the mupdf thirdparty/openjpeg is cloned from the GhostPDL repo, apparently09:35.26 
Robin_Watts arse. OK. Ta.09:35.51 
chrisl Robin_Watts: to what are you trying to update?09:37.01 
  2.1.1?09:37.21 
Robin_Watts chrisl: The latest version (git) has loads of fixes, apparently.09:37.36 
  and indeed, it's enough to solve a whole raft of crashes with the files npw in our test repo.09:38.04 
chrisl Robin_Watts: So you plan to use some random SHA from their repo?09:38.18 
Robin_Watts Yes.09:38.32 
chrisl Robin_Watts: I'm not keen on doing that unless there's a really compelling reason09:40.53 
Robin_Watts The 78 test files that no longer SEGV seems a compelling enough reason to me.09:41.27 
  We are running a fork anyway.09:41.45 
chrisl No, we're not09:41.52 
  Oh, shit, yes we are :-(09:42.11 
Robin_Watts If we're not running a fork, then I wasted several hours merging it all yesterday :)09:42.34 
sebras do we keep a big pile of patches on top of upstream?09:43.16 
  maybe it's feasible to get them merged..?09:43.32 
chrisl Yeh. good luck with that......09:43.48 
Robin_Watts sebras: Sadly it's not that simple.09:43.54 
  What I could do (I think) is generate a diff from their code to ours, so that our diffs sit on top of theirs as a single commit.09:44.49 
chrisl Robin_Watts: "Several hours"? The extra patches are pretty trivial......09:44.57 
Robin_Watts But I'm not sure that is that friendly.09:45.00 
chrisl http://git.ghostscript.com/?p=user/chrisl/ghostpdl.git;a=commitdiff;h=f01b75c42145cfbff1b827ed9e988abddbca435c09:45.02 
  Robin_Watts: if it can wait until after the staff meeting, I'll update the Ghostscript one, which will also update the mupdf one09:46.10 
Robin_Watts The MuPDF one is updated already. Then it dawned on me that we generated one from the other :(09:47.04 
chrisl The top commit in thirdparty/openjpeg.git is still my "update to OpenJPEG 2.1.0" commit09:47.53 
Robin_Watts I thought our patches were more extensive.09:48.56 
chrisl All the others have been adopted upstream, as of 2.1.009:49.20 
Robin_Watts hence I generated the diffs from their v2.1 to current, and reapplied those enmasse to our treee.09:49.23 
  chrisl: Let me diff back between the merged version of our code and their code - that should give just the leftover patches.09:50.33 
  The 'interesting' thing is that they have changed their allocation macros to be functions.09:51.05 
chrisl Robin_Watts: They may, of course, have made further changes in the same areas as our patches.....09:51.23 
Robin_Watts Which makes it explicit that they are calling down to malloc/free rather than any functions we might prefer them to use.09:51.50 
chrisl Yeh, that's where we hit a roadblock with them......09:52.18 
Robin_Watts Now, we CAN implement our own versions of those calls, but we don't have a gs_memory_t.09:52.19 
chrisl They refused to change the API/ABI09:52.35 
Robin_Watts So we could do it using TLS.09:52.46 
  and wrapping the calls from gs.09:52.54 
  In mupdf I have a lock and a static to make it safe.09:53.11 
chrisl Which is great, until we port to a platform without TLS.....09:53.22 
Robin_Watts chrisl: Indeed. Hence my lock and a static approach with mupdf.09:53.49 
  it means only 1 thread can be accessing openjpeg at a time, but that's not a disaster.09:54.14 
chrisl It wouldn't matter for Ghostscript09:54.27 
Robin_Watts So I can do the same there.09:54.35 
  Are you saying that gs doesn't even need a lock?09:57.33 
chrisl At the moment no, because gs is not thread safe09:57.58 
Robin_Watts That's a different statement to what I was hoping for.09:58.26 
  I was hoping for "We only ever call opj from the interpreting thread, and we only ever have one of those by design"09:58.57 
chrisl Well, except for the "by design" part......09:59.25 
Robin_Watts OK, I get the distinction.09:59.57 
chrisl If you're heading down that route, I would rather you did the lock and static approach - I'd rather not make gs *less* thread safe10:00.38 
Robin_Watts chrisl: That is my thought too.10:01.14 
  Of course, this will mean that we CANNOT accept a system openjpeg lib.10:01.44 
chrisl Why?10:01.57 
Robin_Watts cos opj_malloc will end up being defined in gs in our build.10:02.08 
  And opj_malloc (which calls direct to malloc) will be defined in the system openjpeg build.10:02.30 
chrisl Sure, so when we use a shared one, we just leave out that definition10:02.31 
Robin_Watts chrisl: OK, so ifdeffery is required around that.10:02.51 
chrisl There's already some for another reason, IIRC10:03.09 
  Oh, that's gone - there used to be. We now just vary on the OPJ version10:04.25 
  Robin_Watts: if you add a lock, it'll have to go in gs_lib_ctx10:14.24 
Robin_Watts Why?10:14.38 
chrisl Where else would you put it?10:14.49 
Robin_Watts In a static.10:14.55 
chrisl And have the same hassle we have with lcms?10:15.10 
Robin_Watts If we have 1 static, might as well have 2.10:15.21 
  what hassle with lcms is that? (there are so many)10:15.35 
chrisl Robin_Watts: the nonsense with the initialisation of the lock10:16.00 
Robin_Watts chrisl: Yes, init is a pain.10:16.11 
  OK, I will look at adding it to the libctx.10:16.19 
chrisl Robin_Watts: d'you want me to add the pre-processor flag for sharing openjpeg?10:18.35 
Robin_Watts That would be appreciated, thanks.10:19.07 
chrisl Robin_Watts: that's on golden/master now10:23.09 
Robin_Watts and -DUSE_OPENJPEG_JP2 doesn't make it into the libctx stuff.10:23.16 
chrisl Does that matter?10:23.33 
Robin_Watts so either we need to extend that, or I extend BOTH openjpeg implementations to take an 'init'/'fin' call.10:23.41 
  and the luratech one does nothing, and the openjpeg one allocates the monitor.10:24.01 
  which would you prefer ?10:24.08 
chrisl I'd prefer both to have the init/fin calls10:24.32 
Robin_Watts then that is what I shall do. thanks again.10:24.47 
chrisl I tend to prefer consistency......10:25.06 
sebras chrisl: that statement is not correct. sometimes you use five periods at the end of your sentences, sometimes six.....(.?)10:31.25 
chrisl sebras: *I* am consistent: what you are seeing there is the variation in the time for the interrupt handler for my keyboard to run.... it doesn't seem important enough to warrant a realtime or even low latency kernel ;-)10:59.57 
sebras Robin_Watts: tor8: mind taking another look at sebras/master?11:06.20 
  Robin already looked at the fixes for 697152, 697163, 697162, 697161, 697160 another set of eyes wouldn't hurt.11:07.29 
tor8 sebras: bug 697162, we generally have the count before the array when passing arguments. or at least I think we do :)11:18.51 
  "image dimensions might overflow" -- "image too large" would probably be clearer for users11:19.53 
  "cannot decode tile" when checking the decoded size. maybe "decoded tile is the wrong size"11:23.32 
sebras tor8: ok, those are simple fixes.11:25.57 
  I haven't forgotten about jpx, that patch is still pending.11:26.26 
  I'm just cleaning out my queue of half baked fuzzing patches.11:26.37 
  there is still one of png (haven't looked at it) and one for cbz (I think this really a problem in mutool)11:27.15 
tor8 sebras: yes. the actual code changes LGTM.11:28.01 
sebras Robin_Watts: how long does a cluster run take now?11:52.13 
Robin_Watts sebras: mupdf is 5 mins or so.11:53.44 
  mujstest is slower.11:53.49 
  167 seconds - dunno if that includes building etc. I guess it must.11:54.28 
sebras Robin_Watts: can I see how long my currently running test has been executing?11:55.02 
kens The start time is at the top11:55.13 
sebras kens: oh!11:55.30 
  so... 20min..?11:55.35 
  unless my timezones are off again...11:55.44 
kens yes 20 mins11:55.50 
  currently at 907% jus t finishing11:56.00 
sebras mmm, I just noticed the percentage.11:56.25 
kens Looks like there was a problem11:56.39 
  Its dropped back to 80%11:56.45 
  and there's 'went down, rerunning jobs' messages11:57.05 
  It should all self-correct11:57.20 
sebras I'm not sure I understand quite how it works.11:57.40 
kens I'm not sure anyone really understands it totally11:58.02 
Robin_Watts sebras: I suspect that your current run has introduced some errors (or something else) that is causing stuff to time out.11:58.07 
kens Its up to 100% now though11:58.16 
Robin_Watts consequently those jobs get 'rescheduled'. That's the only way I can think that the percentage can drop.11:58.26 
kens So when these 3 jobs are done, it'll be finished11:58.28 
Robin_Watts Certainly the last normal pdf run went through in 167 seconds.11:58.46 
  Note that those 2 jobs are jp2s. :(11:59.12 
kens So were a numbre of the others11:59.23 
Robin_Watts If the jp2 stuff is still causing slowdowns we may need to disable those tests again.12:00.05 
sebras Robin_Watts: the problem was I didn't rebase on top of your openjpeg import.12:05.12 
Robin_Watts sebras: Ah!12:05.23 
sebras tor8: what both you and I missed is that fz_decode_tiff_strips() creates it own rp, separate from tiff->rp :-(12:34.08 
  btw... fz_load_tiff(fz_context *ctx, unsigned char *buf, size_t len) the buffer passed here. am I allowed to modify its contents?!12:35.38 
tor8 sebras: no.12:40.06 
  it really ought to be const12:40.17 
sebras tor8: shouldn't fz_open_memory() take const unsigned char *?13:05.58 
tor8 sebras: yes. we haven't constified everything yet. it's a work in progress :)13:14.50 
sebras tor8: you will face problems in the chaining of filters, one writing to the next.13:15.38 
  if you make fz_stream->rp const.13:15.44 
tor8 sebras: yes. the data of fz_buffer can't be const since we use buffers both for reading and writing13:19.41 
  it gets convoluted fast, and needs a lot of casts, so we haven't bothered putting const in for filters and buffers13:22.15 
Robin_Watts chrisl: So, you are correct that the only diffs from the vanilla openjpeg source are some sumatrapdf ones (plus whitespace changes).14:53.09 
  How would you prefer the history to look in git?14:53.09 
  At the moment it's what was there before + 1 commit that inserts all the changes between the relavent openjpeg points.14:53.41 
  I could redo it as: "Copy in the new openjpeg source and then reapply our patches" which might be clearer.14:54.07 
  (and easier to update next time)14:54.28 
chrisl Robin_Watts: I use a branch, commit the complete openjpeg release, the remove the bits we don't need (they ship a load of stuff we don't use), apply our patch(es), then squash all that to a single commit on master14:55.08 
  Robin_Watts: like this: http://git.ghostscript.com/?p=user/chrisl/ghostpdl.git;a=shortlog;h=refs/heads/openjpeg-update14:55.41 
  So we keep the history on the branch (for future reference), but avoid any non-working commits on master14:56.35 
Robin_Watts And the master commit includes a pointer to that branch name (so I'd need to use openjpeg-update-20160927 or something)14:57.42 
chrisl You can do that, sure - I didn't before because the last OpenJPEG import was before we ratified that process14:58.30 
Robin_Watts When I looked at the mupdf generated repo, it didn't have any such pointer, which is why I worked the long way around.14:58.34 
  Sure, but it's evidence that we should be doing that in future.14:58.55 
chrisl Oh yes, I agree14:59.24 
Robin_Watts Thanks.14:59.27 
sebras during a windy typhoon the entire building might shake if you are on the tenth floor. this is not normal in .se :)15:25.52 
  just so you know if I lose my internet connection.15:26.08 
Robin_Watts sebras: ooh, typhoon. Fun. Time to order several pizzas and take bets as to which ones will arrive.15:27.15 
sebras Robin_Watts: or fix fuzzing bugs in mupdf!15:27.43 
  Robin_Watts: this is the third one since I arrived but this one almost hit the capital whereas the others made landfall down south.15:28.19 
Robin_Watts chrisl: http://git.ghostscript.com/?p=user/robin/ghostpdl.git;a=commitdiff;h=9a78bfb8698061e22e3d460b8c15825992e44def15:54.56 
chrisl Robin_Watts: I notice the addition of thread.h and tls_keys.h in the OPJ sources - is that going to bite us at some point?15:59.34 
  Robin_Watts: And I assume it clusters okay16:01.29 
Robin_Watts chrisl: It is clustering as we speak.16:02.36 
  I hope the thread stuff won't bite us. I certainly haven't done anything to enable it.16:02.58 
chrisl Yeh, it looks like they don't assume it's available - I just have trust issues where things like that are concerned16:03.41 
Robin_Watts yeah. If MUTEX_whatever is not defined, it does nothing.16:04.34 
  There are some diffs. All progressions. (See my bmpcmp if interested)16:08.36 
chrisl Cool, thanks for doing that16:10.47 
Robin_Watts chrisl: No worries. Best to do things properly, if only so that they will confuse me less next time.16:11.37 
  Now, let's see if I can find the mechanism that regenerates openjpeg on casper...16:12.11 
chrisl It should get run by a cron job - the actual script is in tor8's bin directory16:15.27 
Robin_Watts @daily /home/tor/bin/mirror-thirdparty.sh >/dev/null16:16.22 
chrisl That script then runs the individual ones for each mirror16:16.59 
Robin_Watts chrisl: Just running the whole lot now.16:17.11 
  Done.16:17.15 
  That gets it as far as /home/tor/src/thirdparty/openjpeg16:19.33 
  Ah, and then to /home/git/thirdparty/openjpeg16:20.15 
  Which is somehow not up to date.16:21.26 
  Ah, the mirror script has been broken since gs/openjpeg became openjpeg16:22.15 
  tor8: You here?16:22.45 
  tor8: ping16:42.37 
sebras Robin_Watts: ugh.. I finally sorted out the issues with tiff over at sebras/master16:52.06 
Robin_Watts sebras: Ah, cool. Gimme a mo, and I'll look.16:52.44 
  ok, sebras: 1 commit on robin/master then. Just correcting the thirdparty/openjpeg pointer so it's on the commit autogenerated from the gs ones.16:57.07 
  Now let me look at your commits.16:57.20 
  sebras: Not sure about the 3rd from the top.17:01.01 
  No, it's OK, I get it.17:01.33 
  OK, what's the final commit about? I mean, I get that it's nasty to reverse an input buffer, and then reverse it back again afterwards, but did you actually have a bug showing up because of that?17:04.30 
  I am not hugely keen on the fact we'll malloc/free the buffer for every chunk, but I guess that's OK.17:08.15 
  All lgtm then.17:08.18 
sebras Robin_Watts: no, no bug. but tor8 mentioned that the buffer really ought to be const.17:08.38 
  Robin_Watts: and it feels better not to modify the input buffer.17:08.50 
Robin_Watts It does. I might be tempted to have tiff->reversebuf and tiff->reversebuf_size, and thus avoid the remalloc.17:09.22 
sebras Robin_Watts: tiles are specified to only have one size.17:10.17 
  Robin_Watts: and strips have an upper bound.17:10.25 
Robin_Watts so we could malloc it once and be sure we'd never need to release it?17:11.03 
sebras Robin_Watts: yes.17:12.07 
  Robin_Watts: I haven't thought about the error handling though.17:12.51 
Robin_Watts sebras: We alloc it just in time. We always free it when we free tiff17:13.23 
sebras Robin_Watts: but moving the reversing up one notch into decode_tiff_tiles()/decode_tiff_strips() where they have knowledge about the size of the buffer, then they could allocate _once_, populate it, send it to the chunk-function and then free the buffer once all tiles/strips are done.17:14.14 
Robin_Watts sebras: Which is milliseconds before tiff is freed anyway?17:14.42 
  so we'd still only need to malloc once just doing it where it is, wouldn't we?17:14.59 
sebras Robin_Watts: true. now, for sane TIFFs I doubt that this will have noticable effect. 17:15.25 
  Robin_Watts: I'm tempted to push this as is and then get to this allocation thing later this/next week. is that feasible?17:16.48 
Robin_Watts of course.17:16.56 
sebras Robin_Watts: thanks.17:18.03 
Robin_Watts sebras: http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=c04e6cbada524d05a49546bc86ab7125ecdf97c417:28.07 
sebras Robin_Watts: yes.17:29.15 
Robin_Watts You're happy with that? Thanks.17:29.33 
sebras Robin_Watts: no.17:29.39 
  Robin_Watts: not yet.17:29.41 
  Robin_Watts: yes, I should be looking at it.17:29.50 
Robin_Watts ah, thanks.17:30.07 
sebras Robin_Watts: it would have been nice if the openjpeg commit message mentioned _what_ version of openjpeg was being imported on the subject line. ;)17:32.11 
  Robin_Watts: but I guess that's too late now.17:32.19 
Robin_Watts sebras: There is no specific version (it's not a release version).17:33.01 
  The commit message DOES list the SHA.17:33.02 
sebras Robin_Watts: right. 17:34.08 
  Robin_Watts: looks reasonable to me.17:34.19 
Robin_Watts Thanks.17:34.24 
sebras Robin_Watts: but it is almost 2am again...17:34.27 
sebras sleeps.17:34.31 
Robin_Watts Night. THanks!17:34.35 
  I shall be heading to Orlando when I wake up, so good luck with the typhoon.17:34.59 
sebras Robin_Watts: you're welcome, thanks for helping me with my commits. :)17:35.01 
  Robin_Watts: see to mostly have passed now.17:35.16 
Robin_Watts http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=54fb65fae4d5ce47e32e54d0da50cbc3aa8c5d6417:35.33 
  Time for a 1 liner?17:35.40 
  (just 2 chars!)17:35.45 
sebras Robin_Watts: LGTM.17:37.14 
  Robin_Watts: I didn't know about this function.17:37.19 
Robin_Watts It just performs a sum on pixel values, so it can trip valgrind if there are any undefined pixels.17:37.58 
sebras Robin_Watts: would have been convenient to know about _before_ I implemented support for all those image formats and did a hackish sum-thingy over all samples just to trigger valgrind. :)17:38.10 
Robin_Watts It's not used in normal builds obviously :)17:38.13 
sebras Robin_Watts: no, no, I get it!17:38.21 
Robin_Watts sebras: Ha! This IS my hackish thing :)17:38.24 
sebras Robin_Watts: I never checked mine in.17:38.34 
  s/mine/my/17:38.46 
Robin_Watts No, mine was correct.17:38.58 
sebras Robin_Watts: what if there were multiple?17:39.22 
sebras really does sleep.17:40.09 
Robin_Watts This is my dog. These dogs are mine. But: That dog is yours, which is mine?17:40.09 
ray_laptop I got the bash up now on Windows 10 (the Beta) and am able to build and run gs :-) I ran the benchmark that I had been running on peeved on my laptop and compared the gcc built version running in the bash shell and the time is the same as running the VS 2015 built version 18:12.56 
Robin_Watts ray_laptop: That's about what we'd expect, right?18:16.38 
  Speed will be determined by native performance mostly, and the performance of syscalls - which are all implemented on the same windows kernel.18:17.24 
ray_laptop Robin_Watts: yeah -- but I was curious if it would run as fast (if the connection to the system services and file system was OK)18:17.30 
Robin_Watts ray_laptop: They are missing the ability for linux to make calls out to windows programs.18:18.32 
  At some point I will investigate using VisualGDB to debug code running on the linux subshell.18:19.11 
ray_laptop I'm trying to get the X11 package installed18:19.40 
Robin_Watts I was about to ask. Do you have mingw?18:19.53 
ray_laptop Robin_Watts: I have an X server that I can use to peeved/peeves 18:20.20 
  so when I putty in, it tunnels the X11 18:20.50 
  so I was going to see if I can get it hooked on the same machine18:21.17 
  but the main thing I want it for is to do pg builds (VS performance tools are a pain and I don't trust them)18:22.34 
Robin_Watts gprof build?18:25.25 
 Forward 1 day (to 2016/09/28)>>> 
ghostscript.com
Search: