| <<<Back 1 day (to 2016/09/26) | 20160927 |
ray_pc | I don't have a good feeling about this. Doing Windows update and it has taken 20 minutes so far :-( | 00:45.27 |
| what't that chance that everything will still work ? | 00:45.58 |
| so far it has rebooted about 12 times | 01:09.16 |
| mow it asks me to login and say "Getting things ready..." | 01:13.55 |
| finally up. Email tool works | 01:17.41 |
| hmm... even though this was supposed to be ther "anniversary update", I don't see the bash stuff (trying various hints frp, google_ | 01:30.46 |
| ahh, afterI turned on developer options,I rebooted (no prompt, but goolge said I might have to) and ow I see the "Windows Subsytem for Linux (Beta)" option | 01:36.14 |
| which (no surpise here) require YET ANOTHER REBOOT. I think that brings me up to about 16 | 01:36.49 |
| time to get dinner for the family ... | 01:37.57 |
sebras | Robin_Watts: tor8: four gif-related fuzzing issues fixed on sebras/master | 05:27.27 |
| ... and a jpeg-crash fix. | 06:22.23 |
| kens: morning! | 06:54.33 |
kens | morning sebras | 06:54.39 |
Robin_Watts | morning sebras | 08:38.53 |
| all lgtm. | 08:41.52 |
| chrisl: Do you remember how the openjpeg stuff works? | 09:19.49 |
| Do we generate the gs openjpeg stuff from mupdf? Or vice versa? or are they separate? | 09:20.10 |
kens2 | I thought they were separate | 09:24.29 |
Robin_Watts | fab. | 09:24.36 |
kens2 | Don't quote me though, its not my area | 09:24.50 |
chrisl | Robin_Watts: No idea..... | 09:33.53 |
Robin_Watts | I suspect they are completely separate then. | 09:34.08 |
chrisl | I'll have a quick look on casper , there might be a hint in there | 09:34.32 |
| Robin_Watts: the mupdf thirdparty/openjpeg is cloned from the GhostPDL repo, apparently | 09:35.26 |
Robin_Watts | arse. OK. Ta. | 09:35.51 |
chrisl | Robin_Watts: to what are you trying to update? | 09:37.01 |
| 2.1.1? | 09:37.21 |
Robin_Watts | chrisl: The latest version (git) has loads of fixes, apparently. | 09:37.36 |
| and indeed, it's enough to solve a whole raft of crashes with the files npw in our test repo. | 09:38.04 |
chrisl | Robin_Watts: So you plan to use some random SHA from their repo? | 09:38.18 |
Robin_Watts | Yes. | 09:38.32 |
chrisl | Robin_Watts: I'm not keen on doing that unless there's a really compelling reason | 09:40.53 |
Robin_Watts | The 78 test files that no longer SEGV seems a compelling enough reason to me. | 09:41.27 |
| We are running a fork anyway. | 09:41.45 |
chrisl | No, we're not | 09:41.52 |
| Oh, shit, yes we are :-( | 09:42.11 |
Robin_Watts | If we're not running a fork, then I wasted several hours merging it all yesterday :) | 09:42.34 |
sebras | do we keep a big pile of patches on top of upstream? | 09:43.16 |
| maybe it's feasible to get them merged..? | 09:43.32 |
chrisl | Yeh. good luck with that...... | 09:43.48 |
Robin_Watts | sebras: Sadly it's not that simple. | 09:43.54 |
| What I could do (I think) is generate a diff from their code to ours, so that our diffs sit on top of theirs as a single commit. | 09:44.49 |
chrisl | Robin_Watts: "Several hours"? The extra patches are pretty trivial...... | 09:44.57 |
Robin_Watts | But I'm not sure that is that friendly. | 09:45.00 |
chrisl | http://git.ghostscript.com/?p=user/chrisl/ghostpdl.git;a=commitdiff;h=f01b75c42145cfbff1b827ed9e988abddbca435c | 09:45.02 |
| Robin_Watts: if it can wait until after the staff meeting, I'll update the Ghostscript one, which will also update the mupdf one | 09:46.10 |
Robin_Watts | The MuPDF one is updated already. Then it dawned on me that we generated one from the other :( | 09:47.04 |
chrisl | The top commit in thirdparty/openjpeg.git is still my "update to OpenJPEG 2.1.0" commit | 09:47.53 |
Robin_Watts | I thought our patches were more extensive. | 09:48.56 |
chrisl | All the others have been adopted upstream, as of 2.1.0 | 09:49.20 |
Robin_Watts | hence I generated the diffs from their v2.1 to current, and reapplied those enmasse to our treee. | 09:49.23 |
| chrisl: Let me diff back between the merged version of our code and their code - that should give just the leftover patches. | 09:50.33 |
| The 'interesting' thing is that they have changed their allocation macros to be functions. | 09:51.05 |
chrisl | Robin_Watts: They may, of course, have made further changes in the same areas as our patches..... | 09:51.23 |
Robin_Watts | Which makes it explicit that they are calling down to malloc/free rather than any functions we might prefer them to use. | 09:51.50 |
chrisl | Yeh, that's where we hit a roadblock with them...... | 09:52.18 |
Robin_Watts | Now, we CAN implement our own versions of those calls, but we don't have a gs_memory_t. | 09:52.19 |
chrisl | They refused to change the API/ABI | 09:52.35 |
Robin_Watts | So we could do it using TLS. | 09:52.46 |
| and wrapping the calls from gs. | 09:52.54 |
| In mupdf I have a lock and a static to make it safe. | 09:53.11 |
chrisl | Which is great, until we port to a platform without TLS..... | 09:53.22 |
Robin_Watts | chrisl: Indeed. Hence my lock and a static approach with mupdf. | 09:53.49 |
| it means only 1 thread can be accessing openjpeg at a time, but that's not a disaster. | 09:54.14 |
chrisl | It wouldn't matter for Ghostscript | 09:54.27 |
Robin_Watts | So I can do the same there. | 09:54.35 |
| Are you saying that gs doesn't even need a lock? | 09:57.33 |
chrisl | At the moment no, because gs is not thread safe | 09:57.58 |
Robin_Watts | That's a different statement to what I was hoping for. | 09:58.26 |
| I was hoping for "We only ever call opj from the interpreting thread, and we only ever have one of those by design" | 09:58.57 |
chrisl | Well, except for the "by design" part...... | 09:59.25 |
Robin_Watts | OK, I get the distinction. | 09:59.57 |
chrisl | If you're heading down that route, I would rather you did the lock and static approach - I'd rather not make gs *less* thread safe | 10:00.38 |
Robin_Watts | chrisl: That is my thought too. | 10:01.14 |
| Of course, this will mean that we CANNOT accept a system openjpeg lib. | 10:01.44 |
chrisl | Why? | 10:01.57 |
Robin_Watts | cos opj_malloc will end up being defined in gs in our build. | 10:02.08 |
| And opj_malloc (which calls direct to malloc) will be defined in the system openjpeg build. | 10:02.30 |
chrisl | Sure, so when we use a shared one, we just leave out that definition | 10:02.31 |
Robin_Watts | chrisl: OK, so ifdeffery is required around that. | 10:02.51 |
chrisl | There's already some for another reason, IIRC | 10:03.09 |
| Oh, that's gone - there used to be. We now just vary on the OPJ version | 10:04.25 |
| Robin_Watts: if you add a lock, it'll have to go in gs_lib_ctx | 10:14.24 |
Robin_Watts | Why? | 10:14.38 |
chrisl | Where else would you put it? | 10:14.49 |
Robin_Watts | In a static. | 10:14.55 |
chrisl | And have the same hassle we have with lcms? | 10:15.10 |
Robin_Watts | If we have 1 static, might as well have 2. | 10:15.21 |
| what hassle with lcms is that? (there are so many) | 10:15.35 |
chrisl | Robin_Watts: the nonsense with the initialisation of the lock | 10:16.00 |
Robin_Watts | chrisl: Yes, init is a pain. | 10:16.11 |
| OK, I will look at adding it to the libctx. | 10:16.19 |
chrisl | Robin_Watts: d'you want me to add the pre-processor flag for sharing openjpeg? | 10:18.35 |
Robin_Watts | That would be appreciated, thanks. | 10:19.07 |
chrisl | Robin_Watts: that's on golden/master now | 10:23.09 |
Robin_Watts | and -DUSE_OPENJPEG_JP2 doesn't make it into the libctx stuff. | 10:23.16 |
chrisl | Does that matter? | 10:23.33 |
Robin_Watts | so either we need to extend that, or I extend BOTH openjpeg implementations to take an 'init'/'fin' call. | 10:23.41 |
| and the luratech one does nothing, and the openjpeg one allocates the monitor. | 10:24.01 |
| which would you prefer ? | 10:24.08 |
chrisl | I'd prefer both to have the init/fin calls | 10:24.32 |
Robin_Watts | then that is what I shall do. thanks again. | 10:24.47 |
chrisl | I tend to prefer consistency...... | 10:25.06 |
sebras | chrisl: that statement is not correct. sometimes you use five periods at the end of your sentences, sometimes six.....(.?) | 10:31.25 |
chrisl | sebras: *I* am consistent: what you are seeing there is the variation in the time for the interrupt handler for my keyboard to run.... it doesn't seem important enough to warrant a realtime or even low latency kernel ;-) | 10:59.57 |
sebras | Robin_Watts: tor8: mind taking another look at sebras/master? | 11:06.20 |
| Robin already looked at the fixes for 697152, 697163, 697162, 697161, 697160 another set of eyes wouldn't hurt. | 11:07.29 |
tor8 | sebras: bug 697162, we generally have the count before the array when passing arguments. or at least I think we do :) | 11:18.51 |
| "image dimensions might overflow" -- "image too large" would probably be clearer for users | 11:19.53 |
| "cannot decode tile" when checking the decoded size. maybe "decoded tile is the wrong size" | 11:23.32 |
sebras | tor8: ok, those are simple fixes. | 11:25.57 |
| I haven't forgotten about jpx, that patch is still pending. | 11:26.26 |
| I'm just cleaning out my queue of half baked fuzzing patches. | 11:26.37 |
| there is still one of png (haven't looked at it) and one for cbz (I think this really a problem in mutool) | 11:27.15 |
tor8 | sebras: yes. the actual code changes LGTM. | 11:28.01 |
sebras | Robin_Watts: how long does a cluster run take now? | 11:52.13 |
Robin_Watts | sebras: mupdf is 5 mins or so. | 11:53.44 |
| mujstest is slower. | 11:53.49 |
| 167 seconds - dunno if that includes building etc. I guess it must. | 11:54.28 |
sebras | Robin_Watts: can I see how long my currently running test has been executing? | 11:55.02 |
kens | The start time is at the top | 11:55.13 |
sebras | kens: oh! | 11:55.30 |
| so... 20min..? | 11:55.35 |
| unless my timezones are off again... | 11:55.44 |
kens | yes 20 mins | 11:55.50 |
| currently at 907% jus t finishing | 11:56.00 |
sebras | mmm, I just noticed the percentage. | 11:56.25 |
kens | Looks like there was a problem | 11:56.39 |
| Its dropped back to 80% | 11:56.45 |
| and there's 'went down, rerunning jobs' messages | 11:57.05 |
| It should all self-correct | 11:57.20 |
sebras | I'm not sure I understand quite how it works. | 11:57.40 |
kens | I'm not sure anyone really understands it totally | 11:58.02 |
Robin_Watts | sebras: I suspect that your current run has introduced some errors (or something else) that is causing stuff to time out. | 11:58.07 |
kens | Its up to 100% now though | 11:58.16 |
Robin_Watts | consequently those jobs get 'rescheduled'. That's the only way I can think that the percentage can drop. | 11:58.26 |
kens | So when these 3 jobs are done, it'll be finished | 11:58.28 |
Robin_Watts | Certainly the last normal pdf run went through in 167 seconds. | 11:58.46 |
| Note that those 2 jobs are jp2s. :( | 11:59.12 |
kens | So were a numbre of the others | 11:59.23 |
Robin_Watts | If the jp2 stuff is still causing slowdowns we may need to disable those tests again. | 12:00.05 |
sebras | Robin_Watts: the problem was I didn't rebase on top of your openjpeg import. | 12:05.12 |
Robin_Watts | sebras: Ah! | 12:05.23 |
sebras | tor8: what both you and I missed is that fz_decode_tiff_strips() creates it own rp, separate from tiff->rp :-( | 12:34.08 |
| btw... fz_load_tiff(fz_context *ctx, unsigned char *buf, size_t len) the buffer passed here. am I allowed to modify its contents?! | 12:35.38 |
tor8 | sebras: no. | 12:40.06 |
| it really ought to be const | 12:40.17 |
sebras | tor8: shouldn't fz_open_memory() take const unsigned char *? | 13:05.58 |
tor8 | sebras: yes. we haven't constified everything yet. it's a work in progress :) | 13:14.50 |
sebras | tor8: you will face problems in the chaining of filters, one writing to the next. | 13:15.38 |
| if you make fz_stream->rp const. | 13:15.44 |
tor8 | sebras: yes. the data of fz_buffer can't be const since we use buffers both for reading and writing | 13:19.41 |
| it gets convoluted fast, and needs a lot of casts, so we haven't bothered putting const in for filters and buffers | 13:22.15 |
Robin_Watts | chrisl: So, you are correct that the only diffs from the vanilla openjpeg source are some sumatrapdf ones (plus whitespace changes). | 14:53.09 |
| How would you prefer the history to look in git? | 14:53.09 |
| At the moment it's what was there before + 1 commit that inserts all the changes between the relavent openjpeg points. | 14:53.41 |
| I could redo it as: "Copy in the new openjpeg source and then reapply our patches" which might be clearer. | 14:54.07 |
| (and easier to update next time) | 14:54.28 |
chrisl | Robin_Watts: I use a branch, commit the complete openjpeg release, the remove the bits we don't need (they ship a load of stuff we don't use), apply our patch(es), then squash all that to a single commit on master | 14:55.08 |
| Robin_Watts: like this: http://git.ghostscript.com/?p=user/chrisl/ghostpdl.git;a=shortlog;h=refs/heads/openjpeg-update | 14:55.41 |
| So we keep the history on the branch (for future reference), but avoid any non-working commits on master | 14:56.35 |
Robin_Watts | And the master commit includes a pointer to that branch name (so I'd need to use openjpeg-update-20160927 or something) | 14:57.42 |
chrisl | You can do that, sure - I didn't before because the last OpenJPEG import was before we ratified that process | 14:58.30 |
Robin_Watts | When I looked at the mupdf generated repo, it didn't have any such pointer, which is why I worked the long way around. | 14:58.34 |
| Sure, but it's evidence that we should be doing that in future. | 14:58.55 |
chrisl | Oh yes, I agree | 14:59.24 |
Robin_Watts | Thanks. | 14:59.27 |
sebras | during a windy typhoon the entire building might shake if you are on the tenth floor. this is not normal in .se :) | 15:25.52 |
| just so you know if I lose my internet connection. | 15:26.08 |
Robin_Watts | sebras: ooh, typhoon. Fun. Time to order several pizzas and take bets as to which ones will arrive. | 15:27.15 |
sebras | Robin_Watts: or fix fuzzing bugs in mupdf! | 15:27.43 |
| Robin_Watts: this is the third one since I arrived but this one almost hit the capital whereas the others made landfall down south. | 15:28.19 |
Robin_Watts | chrisl: http://git.ghostscript.com/?p=user/robin/ghostpdl.git;a=commitdiff;h=9a78bfb8698061e22e3d460b8c15825992e44def | 15:54.56 |
chrisl | Robin_Watts: I notice the addition of thread.h and tls_keys.h in the OPJ sources - is that going to bite us at some point? | 15:59.34 |
| Robin_Watts: And I assume it clusters okay | 16:01.29 |
Robin_Watts | chrisl: It is clustering as we speak. | 16:02.36 |
| I hope the thread stuff won't bite us. I certainly haven't done anything to enable it. | 16:02.58 |
chrisl | Yeh, it looks like they don't assume it's available - I just have trust issues where things like that are concerned | 16:03.41 |
Robin_Watts | yeah. If MUTEX_whatever is not defined, it does nothing. | 16:04.34 |
| There are some diffs. All progressions. (See my bmpcmp if interested) | 16:08.36 |
chrisl | Cool, thanks for doing that | 16:10.47 |
Robin_Watts | chrisl: No worries. Best to do things properly, if only so that they will confuse me less next time. | 16:11.37 |
| Now, let's see if I can find the mechanism that regenerates openjpeg on casper... | 16:12.11 |
chrisl | It should get run by a cron job - the actual script is in tor8's bin directory | 16:15.27 |
Robin_Watts | @daily /home/tor/bin/mirror-thirdparty.sh >/dev/null | 16:16.22 |
chrisl | That script then runs the individual ones for each mirror | 16:16.59 |
Robin_Watts | chrisl: Just running the whole lot now. | 16:17.11 |
| Done. | 16:17.15 |
| That gets it as far as /home/tor/src/thirdparty/openjpeg | 16:19.33 |
| Ah, and then to /home/git/thirdparty/openjpeg | 16:20.15 |
| Which is somehow not up to date. | 16:21.26 |
| Ah, the mirror script has been broken since gs/openjpeg became openjpeg | 16:22.15 |
| tor8: You here? | 16:22.45 |
| tor8: ping | 16:42.37 |
sebras | Robin_Watts: ugh.. I finally sorted out the issues with tiff over at sebras/master | 16:52.06 |
Robin_Watts | sebras: Ah, cool. Gimme a mo, and I'll look. | 16:52.44 |
| ok, sebras: 1 commit on robin/master then. Just correcting the thirdparty/openjpeg pointer so it's on the commit autogenerated from the gs ones. | 16:57.07 |
| Now let me look at your commits. | 16:57.20 |
| sebras: Not sure about the 3rd from the top. | 17:01.01 |
| No, it's OK, I get it. | 17:01.33 |
| OK, what's the final commit about? I mean, I get that it's nasty to reverse an input buffer, and then reverse it back again afterwards, but did you actually have a bug showing up because of that? | 17:04.30 |
| I am not hugely keen on the fact we'll malloc/free the buffer for every chunk, but I guess that's OK. | 17:08.15 |
| All lgtm then. | 17:08.18 |
sebras | Robin_Watts: no, no bug. but tor8 mentioned that the buffer really ought to be const. | 17:08.38 |
| Robin_Watts: and it feels better not to modify the input buffer. | 17:08.50 |
Robin_Watts | It does. I might be tempted to have tiff->reversebuf and tiff->reversebuf_size, and thus avoid the remalloc. | 17:09.22 |
sebras | Robin_Watts: tiles are specified to only have one size. | 17:10.17 |
| Robin_Watts: and strips have an upper bound. | 17:10.25 |
Robin_Watts | so we could malloc it once and be sure we'd never need to release it? | 17:11.03 |
sebras | Robin_Watts: yes. | 17:12.07 |
| Robin_Watts: I haven't thought about the error handling though. | 17:12.51 |
Robin_Watts | sebras: We alloc it just in time. We always free it when we free tiff | 17:13.23 |
sebras | Robin_Watts: but moving the reversing up one notch into decode_tiff_tiles()/decode_tiff_strips() where they have knowledge about the size of the buffer, then they could allocate _once_, populate it, send it to the chunk-function and then free the buffer once all tiles/strips are done. | 17:14.14 |
Robin_Watts | sebras: Which is milliseconds before tiff is freed anyway? | 17:14.42 |
| so we'd still only need to malloc once just doing it where it is, wouldn't we? | 17:14.59 |
sebras | Robin_Watts: true. now, for sane TIFFs I doubt that this will have noticable effect. | 17:15.25 |
| Robin_Watts: I'm tempted to push this as is and then get to this allocation thing later this/next week. is that feasible? | 17:16.48 |
Robin_Watts | of course. | 17:16.56 |
sebras | Robin_Watts: thanks. | 17:18.03 |
Robin_Watts | sebras: http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=c04e6cbada524d05a49546bc86ab7125ecdf97c4 | 17:28.07 |
sebras | Robin_Watts: yes. | 17:29.15 |
Robin_Watts | You're happy with that? Thanks. | 17:29.33 |
sebras | Robin_Watts: no. | 17:29.39 |
| Robin_Watts: not yet. | 17:29.41 |
| Robin_Watts: yes, I should be looking at it. | 17:29.50 |
Robin_Watts | ah, thanks. | 17:30.07 |
sebras | Robin_Watts: it would have been nice if the openjpeg commit message mentioned _what_ version of openjpeg was being imported on the subject line. ;) | 17:32.11 |
| Robin_Watts: but I guess that's too late now. | 17:32.19 |
Robin_Watts | sebras: There is no specific version (it's not a release version). | 17:33.01 |
| The commit message DOES list the SHA. | 17:33.02 |
sebras | Robin_Watts: right. | 17:34.08 |
| Robin_Watts: looks reasonable to me. | 17:34.19 |
Robin_Watts | Thanks. | 17:34.24 |
sebras | Robin_Watts: but it is almost 2am again... | 17:34.27 |
sebras | sleeps. | 17:34.31 |
Robin_Watts | Night. THanks! | 17:34.35 |
| I shall be heading to Orlando when I wake up, so good luck with the typhoon. | 17:34.59 |
sebras | Robin_Watts: you're welcome, thanks for helping me with my commits. :) | 17:35.01 |
| Robin_Watts: see to mostly have passed now. | 17:35.16 |
Robin_Watts | http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=54fb65fae4d5ce47e32e54d0da50cbc3aa8c5d64 | 17:35.33 |
| Time for a 1 liner? | 17:35.40 |
| (just 2 chars!) | 17:35.45 |
sebras | Robin_Watts: LGTM. | 17:37.14 |
| Robin_Watts: I didn't know about this function. | 17:37.19 |
Robin_Watts | It just performs a sum on pixel values, so it can trip valgrind if there are any undefined pixels. | 17:37.58 |
sebras | Robin_Watts: would have been convenient to know about _before_ I implemented support for all those image formats and did a hackish sum-thingy over all samples just to trigger valgrind. :) | 17:38.10 |
Robin_Watts | It's not used in normal builds obviously :) | 17:38.13 |
sebras | Robin_Watts: no, no, I get it! | 17:38.21 |
Robin_Watts | sebras: Ha! This IS my hackish thing :) | 17:38.24 |
sebras | Robin_Watts: I never checked mine in. | 17:38.34 |
| s/mine/my/ | 17:38.46 |
Robin_Watts | No, mine was correct. | 17:38.58 |
sebras | Robin_Watts: what if there were multiple? | 17:39.22 |
sebras | really does sleep. | 17:40.09 |
Robin_Watts | This is my dog. These dogs are mine. But: That dog is yours, which is mine? | 17:40.09 |
ray_laptop | I got the bash up now on Windows 10 (the Beta) and am able to build and run gs :-) I ran the benchmark that I had been running on peeved on my laptop and compared the gcc built version running in the bash shell and the time is the same as running the VS 2015 built version | 18:12.56 |
Robin_Watts | ray_laptop: That's about what we'd expect, right? | 18:16.38 |
| Speed will be determined by native performance mostly, and the performance of syscalls - which are all implemented on the same windows kernel. | 18:17.24 |
ray_laptop | Robin_Watts: yeah -- but I was curious if it would run as fast (if the connection to the system services and file system was OK) | 18:17.30 |
Robin_Watts | ray_laptop: They are missing the ability for linux to make calls out to windows programs. | 18:18.32 |
| At some point I will investigate using VisualGDB to debug code running on the linux subshell. | 18:19.11 |
ray_laptop | I'm trying to get the X11 package installed | 18:19.40 |
Robin_Watts | I was about to ask. Do you have mingw? | 18:19.53 |
ray_laptop | Robin_Watts: I have an X server that I can use to peeved/peeves | 18:20.20 |
| so when I putty in, it tunnels the X11 | 18:20.50 |
| so I was going to see if I can get it hooked on the same machine | 18:21.17 |
| but the main thing I want it for is to do pg builds (VS performance tools are a pain and I don't trust them) | 18:22.34 |
Robin_Watts | gprof build? | 18:25.25 |
| Forward 1 day (to 2016/09/28)>>> | |