| <<<Back 1 day (to 2016/03/20) | 20160321 |
tor8 | Robin_Watts: argh. I've figured out why annotations disappeared. | 11:25.51 |
| transformation madness... | 11:25.57 |
| I removed the beginPage call, and moved the initial 'cm' that sets up the page space to match mupdf's top-down coord system | 11:26.39 |
| but for annots, the initial 'cm' was never emitted, so this flips the drawing outside the bbox :( | 11:27.16 |
kens | tor8 welcoem to my world :-) | 11:27.39 |
Robin_Watts | tor8: Ah. | 13:31.22 |
tor8 | Robin_Watts: double argh! | 14:44.59 |
| java dynamically loads libfreetype from the system | 14:45.11 |
| and that collides with our built-in thirdparty freetype which is also loaded as a shared library | 14:45.27 |
Robin_Watts | tor8: urgh. | 15:03.22 |
| Can we insert a shim header to define all the FT_ stuff as MUFT_ ? | 15:04.02 |
tor8 | I'm thinking we could just use the system freetype for desktop java builds. | 15:06.41 |
| the android build builds the native so using its ndk-build right, so the platform/java/Makefile is only used for desktop? | 15:10.16 |
| ... and possibly to build the .jar? | 15:10.27 |
| or is it used for more? | 15:10.37 |
| Robin_Watts: the freetype include system is a nightmare ... and it uses its own module system, etc. so I think trying to mess with redefining as MUFT_ is going to be more trouble than its worth | 15:13.00 |
| it's, even | 15:13.08 |
| Robin_Watts: a bunch of commits on tor/master for review (ignore the URW font one) | 15:14.05 |
Robin_Watts | The android ndk-build is currently used for all the C stuff (libs, core, and jni layer) | 15:15.57 |
| but fredross-perry is fiddling with this stuff. | 15:16.10 |
| The "Fix unintialized value errors" one. Those are all spurious compiler warnings :( | 15:48.15 |
tor8 | Robin_Watts: actually, no. | 15:48.30 |
| in the case of an exception stack overflow, they are real uninitialized errors | 15:48.41 |
| because then the fz_push_try part will fail, we'll drop into catch and then the jni_rethrow (which just sets the exception flag), and then return the unitilialized value to java | 15:49.22 |
Robin_Watts | Ah, yes, good catch. | 15:49.54 |
| Though, hopefully java doesn't do anything with those values as there is an exception flagged. | 15:50.21 |
tor8 | and oddly enough the compiler didn't spot it when we had the old formulation of the try macro | 15:50.33 |
| I suspect it turns off a bunch of warnings when it sees a setjmp rather than get confused :( | 15:50.47 |
| so having the first if-test separate from the if(setjmp) has shown a number of possible bugs due to more warnings appearing | 15:51.16 |
Robin_Watts | tor8: I'm wondering if the new formulation of setjmp is what is causing the slowdown on the pi. | 15:51.26 |
| I had the idea that we could do: | 15:51.37 |
| { jmpbuf_t *FZ_BUF; if ((FZ_BUF = fz_try_push()) != NULL) { if (setjmp(FZ_BUF) { ... | 15:52.30 |
| Which we might hope would compile down to something closer to what we had before. | 15:52.59 |
tor8 | that's troublesome because of the type of jmp_buf is defined as an array type | 15:53.15 |
| the only way to do it sanely is to use void *FZ_BUF. | 15:53.28 |
| it's an array type of an implementation-defined and hidden magic type :( | 15:54.05 |
Robin_Watts | Whatever type was returned from fz_try_push before, I mean. | 15:54.11 |
| Your commits lgtm then. | 15:54.21 |
tor8 | I believe that was void* :) | 15:54.23 |
| Robin_Watts: thanks. | 15:54.27 |
| I'm confused to how that would generate any better code than what we do now... it's still two branches | 15:55.05 |
Robin_Watts | We called setjmp with a void? | 15:55.09 |
tor8 | void pointer | 15:55.15 |
Robin_Watts | We can avoid the extra ctx->error->top dereference. | 15:55.35 |
tor8 | ah, no, we returned a fz_errro_stack_slot that had a jmpbuf in it | 15:55.45 |
Robin_Watts | Ah, right. That then. | 15:55.58 |
| but let's wait to see where the time has gone. | 15:56.08 |
tor8 | I'd be annoyed if it turns out to be this | 15:56.25 |
Robin_Watts | The other possibility is the glyph plotting changes. | 15:56.34 |
| but I can't really see how I can have made it slower (famous last words) | 15:56.54 |
tor8 | yeah, that was a mechanical change (the #include voodoo shouldn't change implementations) that only added cmyk functions right? | 15:57.35 |
Robin_Watts | tor8: Yes. | 15:59.32 |
| and added optimised rgb/greyscale functions for glyph plotting. | 15:59.50 |
tor8 | does the pi have any fancy branch prediction? | 16:00.12 |
Robin_Watts | I believe ray did timings at 4b6c0ab659b3feb5f8e3683e393d61ccfdc09c3c before. | 16:00.15 |
| tor8: Nope. | 16:00.21 |
| Hmm. building noto.o kills the pi. | 16:03.36 |
tor8 | Robin_Watts: $(OUT)/fitz/noto.o : $(FONT_GEN) | 16:07.55 |
| + $(CC) $(CFLAGS) -O0 -o $@ -c $< | 16:07.55 |
| it might help if we turn off optimizations completely when building noto.o | 16:08.21 |
rayjj | tor8: When building on the Pi, I had to use NOTO_SMALL | 16:11.15 |
| but building with -O0 would probably speed it up as well | 16:11.41 |
| Robin_Watts: I did apt-get upgrade, but my gcc is still at 4.6.3 :-( | 16:12.43 |
tor8 | + $(CC) $(CFLAGS) -O0 -o $@ -c source/fitz/noto.c | 16:12.46 |
Robin_Watts | rayjj: You may need to dist-upgrade | 16:13.04 |
rayjj | tor8: yeah, I saw that on the logs | 16:13.05 |
| Robin_Watts: OK, trying that... | 16:14.05 |
tor8 | rayjj: you may also want to try using clang rather than gcc | 16:14.17 |
rayjj | the upgrade loaded 77Mb, the dist-upgrade after that calculated needing 243Mb | 16:14.53 |
Robin_Watts | I'm seeing the speed difference on windows even more than on the pi. | 16:15.00 |
rayjj | tor8: I'll try that after re-running with gcc upraded | 16:15.26 |
Robin_Watts | Waas 16.5 seconds. Now is 23 seconds. | 16:15.28 |
rayjj | Robin_Watts: slowing down ? | 16:15.39 |
Robin_Watts | Yes. | 16:15.47 |
rayjj | :-( | 16:15.51 |
Robin_Watts | rayjj: bisecting will be MUCH faster on the PC :) | 16:16.06 |
rayjj | no kidding | 16:16.16 |
| Robin_Watts: and you are correct about the version I tested with previously being 4b6c0ab659b3feb5f8e3683e393d61ccfdc09c3c | 16:18.34 |
Robin_Watts | Ok, so it was a0960a7 | 16:41.24 |
| Presumably cos I close and reopen the output file now. Let me see if I can work around that. | 16:42.31 |
rayjj | Robin_Watts: close and re-open how often ? | 16:46.42 |
| (more than once per page?) | 16:46.53 |
Robin_Watts | once a page. Seems an odd result. Let me double check it. | 16:46.57 |
rayjj | I *really* need to get a bigger sd card on the Pi. The Pi 2 I got has a 8Gb, but the older Pi B only has a 4Gb. With multiple trees of GS and mupdf, it ran out of space trying to dist-upgrade :-( | 16:51.54 |
| I think I'll get 32Gb chips for both. | 16:52.16 |
Robin_Watts | rayjj: Yeah, I got a 32gb card off amazon on saturday for exactly that reason :) | 16:52.51 |
| ok. Previous best time was 16.6 seconds. | 17:19.40 |
| I just had a 15.8 second one. | 17:19.54 |
| with the latest code + a fix. | 17:20.10 |
rayjj | hmm... even after dist-upgrade my gcc is _still_ 4.6.3 | 17:22.35 |
| Robin_Watts: which file, etc. and which machine ? | 17:23.14 |
Robin_Watts | rayjj: Windows. | 17:23.38 |
| My point is that I'm back down to the proper timings rather than being up in the 23s. | 17:23.57 |
rayjj | oooh, that is quite a difference | 17:25.04 |
| and *MUCH* worse than I saw on the Pi | 17:25.23 |
| Robin_Watts: BTW, the CMYK 8-bit (contone) numbers for mupdf look reasonable now, some faster than gs, some slower. As with the other modes, gs is still much faster on the PLRM | 17:33.37 |
| Robin_Watts: and the 'mono' (1-bit) is faster than it was with 4b6c0a (PLRM is up by 7%, although it is still only 30% of gs throughput on that file) | 17:36.43 |
Robin_Watts | tor8: (For the logs) 1 commit on robin/master to get ray his speed back. | 18:04.24 |
| Hmm. On windows, doing the first 100 pages of pdf_reference17.pdf at 600dpi, I still get 1/3 of the time wasted in the file writing code even when going to NULL. | 18:07.54 |
lotsoftuna | Hi, I'm wondering if it's possible to enable constant page scrolling in mupdf | 18:14.52 |
| as in, to have one page flow to the next, instead of having the previous page disappear | 18:15.09 |
Robin_Watts | On Android, presumably. | 18:15.09 |
lotsoftuna | How about desktop? | 18:15.15 |
Robin_Watts | ok, let me back up a bit. | 18:15.36 |
| MuPDF has a core C library that does PDF opening/rendering/manipulation etc. | 18:15.52 |
lotsoftuna | mhm | 18:16.06 |
Robin_Watts | That bit will happily run multiple pages at a time. | 18:16.10 |
lotsoftuna | so.. | 18:16.56 |
Robin_Watts | The simple example viewers that we've done don't make use of that flexibility though. | 18:18.46 |
lotsoftuna | Ah | 18:18.52 |
rayjj | Robin_Watts: oops. I did the dist-upgrade and it did eat up almost as much space as it estimated, so now I don't have enough left to build mupdf :-( | 18:19.06 |
lotsoftuna | Would it be possible to... or is this the right process, to recompile it with an option like that? | 18:19.09 |
rayjj | at least until I get the 32Gb chip | 18:19.27 |
Robin_Watts | (Sorry, am on hold on phone...) | 18:19.58 |
| ok, back. | 18:21.09 |
| lotsoftuna: You could try using gsview. | 18:21.35 |
| That's a viewer based on mupdf that does continuous page scrolling. | 18:21.51 |
mvrhel_laptop | ok. I seem to have the page grafting stuff working now | 18:21.54 |
Robin_Watts | nice one. | 18:22.01 |
mvrhel_laptop | error leaks fixed. So I like how memento does the little stack trace print out | 18:22.15 |
Robin_Watts | mvrhel_laptop: Yeah, pleased with that :) | 18:22.44 |
mvrhel_laptop | so I added a mutool graft that lets you specify a source document with a page number and a destination document with a page number. It will then copy the page object and all the dependencies from one to the other. pdf_insert_page is used to do the actual insertion of the created object. The only trick is that the PDF_NAME_Parent entry in the page dictionary has to be deleted from the... | 18:26.09 |
| ...source page dictionary prior to the copy. This avoids the copying of the entire page tree from the source | 18:26.10 |
| I wonder if I we should look into issues with internal links etc that are going to occur | 18:26.45 |
| or contents too | 18:26.52 |
lotsoftuna | ah geez | 18:27.35 |
Robin_Watts | mvrhel_laptop: I suspect it would be safer if rather than copying the page dictionary, we make an empty dictionary, and then copy all the known entries from the page dictionary into it. | 18:27.38 |
lotsoftuna | I gotta compile gsview from aur :/ | 18:27.43 |
Robin_Watts | lotsoftuna: For what OS? | 18:27.52 |
mvrhel_laptop | Robin_Watts: actually a new dictionary is created | 18:28.02 |
lotsoftuna | Arch >.> | 18:28.03 |
Robin_Watts | ArchLinux, right. | 18:28.18 |
lotsoftuna | oh geez, Ghostscript is quite large | 18:28.31 |
Robin_Watts | But you like compiling things, right? otherwise you wouldn't be a linux user. | 18:28.35 |
lotsoftuna | Yeah | 18:28.47 |
mvrhel_laptop | Robin_Watts: as we encounter new dictionaries or arrays we create new ones that are populated | 18:28.55 |
lotsoftuna | I think I'll stick with MuPDF for now, but I'll note that gsview does continuous scrolling | 18:28.59 |
| thanks :) | 18:29.00 |
Robin_Watts | np. | 18:29.12 |
| mvrhel_laptop: I mean, instead of taking the page dictionary P from file 1 and putting it into file 2, we should make a new dict Q. | 18:29.49 |
mvrhel_laptop | That is what I do | 18:30.07 |
| I create a new Dict Q | 18:30.20 |
Robin_Watts | and then put Q.blah = P.blah for all blah in X, where X is the set of names that we know we need to copy over (and that are safe to copy over) | 18:30.45 |
mvrhel_laptop | yes. that is what I do | 18:31.17 |
| you can look at the code in my branch | 18:31.24 |
| pdf-graft.c | 18:31.32 |
| it is a recursive calling function that creates new arrays and dicts as it goes into the object | 18:32.15 |
Robin_Watts | So, if in the original document, I have a page tree entry P, that contains a /RobinRules entry, would that get copied across? | 18:33.19 |
mvrhel_laptop | yes | 18:34.35 |
Robin_Watts | Right, that's what I'm saying shouldn't happen. | 18:34.46 |
| We should only copy *known* entries across at the top level. | 18:35.01 |
mvrhel_laptop | Robin_Watts: I am not sure I am quite following you. Originally I was creating new objects for everything. And tor told me not to create new objects for names, null, ints, reals, and strings | 18:36.40 |
| so I am not only creating new objects for arrays, and dicts | 18:36.58 |
| s/not/now | 18:37.03 |
Robin_Watts | We'd copy, Type, LastModified, Resources, MediaBox, CropBox, BleedBox, TrimBox, ArtBox, BoxColorInfo, Contents, Rotate, Group, Thumb, B, Dur, Trans, Annots, AA, Metadata, PieceInfo, ID, PZ, SeparationInfo, Tabs, TemplateInstantiated, PresSteps, UserUnit, VP. | 18:38.06 |
mvrhel_laptop | everything is copied | 18:38.21 |
Robin_Watts | mvrhel_laptop: It sounds like you've written a generic 'deep copy' routine. | 18:38.28 |
| And that's great, and a useful thing to have. | 18:38.37 |
mvrhel_laptop | I suppose I could strip out the unknown name entries | 18:38.53 |
| if that is what you want | 18:38.56 |
Robin_Watts | BUT for copying a page tree P to put into a new document, I reckon you should not deep copy P. | 18:39.16 |
| You should do Q = { deep_copy(P.Type), deep_copy(P.LastModified), deep_copy(P.Resources), .... } | 18:39.43 |
malc_ | Robin_Watts: your inner haskellite is trying to break free | 18:40.24 |
Robin_Watts | Which would be the same net effect as deep copying P and then deleting the unwanted entries, yes. | 18:40.56 |
| I also wonder whether deep_copy should have special knowledge about Keys with 'Parent' in the names. | 18:41.37 |
| :) | 18:41.40 |
mvrhel_laptop | Robin_Watts: I am not saying that we delete unwanted entries. Just avoid doing the copy of them. | 18:41.41 |
| Robin_Watts: no | 18:41.45 |
| I had that earlier | 18:41.50 |
| and tor told me to take at it | 18:41.57 |
| it was up to the caller to do any "special" stuff | 18:42.08 |
Robin_Watts | ha. OK. | 18:42.10 |
mvrhel_laptop | s/at it/ it out/ | 18:42.22 |
Robin_Watts | So, I think we're at the same place now. | 18:42.31 |
mvrhel_laptop | I will let him look this over and we call all chat about it tomorrow | 18:42.58 |
| you can have a look to if you have a chance | 18:43.07 |
Robin_Watts | deep_copy P but restricted to just the known set of children. | 18:43.08 |
| I will try to. | 18:43.11 |
mvrhel_laptop | Perhaps the restriction of not copying unknown children should be an option | 18:43.56 |
| there may be applications where we want to do it | 18:44.12 |
Robin_Watts | mvrhel_laptop: We could recast deep_copy(P) to be deep_copy(P,X) | 18:45.15 |
| which would mean deep_copy P. If X is non NULL, restrict the copy to the names in X. | 18:46.05 |
mvrhel_laptop | yes | 18:46.23 |
lotsoftuna | ... | 18:46.59 |
| So now my pages fade in/out... | 18:47.05 |
Robin_Watts | malc_: It's not haskell as such. I was tortured in a mathematics department for 3 years, so sometimes I have painful flashbacks. | 18:47.07 |
lotsoftuna | I don't know what button I pressed... | 18:47.08 |
| also they seem to be changing on their own | 18:47.20 |
| like a slideshow | 18:47.27 |
Robin_Watts | Sounds like you hit presentation mode. | 18:47.28 |
lotsoftuna | waa | 18:47.31 |
| oh, it's just p | 18:47.44 |
| lmao | 18:47.44 |
| why isn't that documented :S | 18:47.53 |
Robin_Watts | It is. | 18:48.01 |
malc_ | Robin_Watts: I only lasted 2... | 18:51.30 |
Robin_Watts | tor8: (For the logs) I think the noto.o line in the Makefile is wrong, cos it never actually passes noto.c to the C compiler. | 20:18.47 |
tor8 | mvrhel_laptop: Robin_Watts: I want to keep the pdf_graft_object simple and predictable without magic stuff happening, hence me wishing that the user would deep copy only the stuff he wants. | 20:23.54 |
Robin_Watts | tor8: Right. | 20:24.10 |
tor8 | in page_graft, do what robin suggests and create a new dictionary and deep copy the known entries to that dictionary, then insert the new dictionary into the page tree | 20:24.26 |
| the current page_graft code *destroys* the source document by deleting the parent entry | 20:24.51 |
| new_page_obj = pdf_new_dict(); pdf_dict_put(new_page_obj, "Resources", pdf_graft_object(pdf_dict_get(pageref, "Resources"))); | 20:25.58 |
| etc. | 20:26.03 |
| safer to copy what we know, than to filter out what we can only hope is complete | 20:26.44 |
| new versions of PDF may add other references that we don't know about | 20:27.00 |
| Robin_Watts: yes, the noto.o line I pasted was wrong. I did paste an updated line a bit later, that you may have missed. | 20:28.35 |
Robin_Watts | I did miss it. | 20:28.42 |
| Another fix robin/master | 20:33.47 |
tor8 | I don't like adding another branch to each fz_write call :( | 20:46.53 |
| that feels like it should just make every other non-benchmark use slower | 20:47.52 |
Robin_Watts | tor8: The alternative is to force all fz_write callers to check for NULL | 20:50.28 |
tor8 | the null device seemed like a decent compromise | 20:50.50 |
| if you don't want output, don't call the function to create it... | 20:51.08 |
Robin_Watts | tor8: OK... I could have a: fz_is_output_null() function that returns true if it's a NULL device then? | 20:51.24 |
tor8 | this looks like benchmark wankery, with no real life applications | 20:51.40 |
Robin_Watts | excellent use of the term wankery, BTW. | 20:51.56 |
tor8 | the other two commits LGTM (l2factor and avoiding reopening) | 20:52.27 |
Robin_Watts | tor8: OK, I'll take another run at the NULL one. | 20:52.52 |
tor8 | can you benchmark actual (inefficiently written) output that writes stuff in smallish chunks? | 20:53.42 |
Robin_Watts | tor8: pgm is an example. | 20:54.13 |
tor8 | it may be that the extra if test doesn't hurt us measurably, in which case I can live with it. but I want numbers for real files not just dumping to /dev/null which is really just cheating :) | 20:54.16 |
| the benchmark will be faster, but is it fair? does what we compare with also bail out early of writing data. | 20:55.08 |
Robin_Watts | gs? Yes. | 20:55.22 |
| Essentially the point is that we generate data into a buffer, and then write it out. | 20:55.41 |
| printer manufacturers will use the data from the buffer. | 20:55.50 |
| Hence they aren't interested in the time taken to write the buffer to disc. | 20:56.01 |
| We are making the same decision/optimisation here that gs does. | 20:56.23 |
| OK, so on my windows PC: platform/win32/Release/mudraw.exe -o nul: -r600 -F pam -c gray pdf_reference17.pdf 1-300 | 20:59.44 |
tor8 | do we care about formatting the output as an image file? or just the rendering time? | 21:00.03 |
Robin_Watts | Just the rendering time. | 21:00.15 |
tor8 | -F benchmark ? | 21:00.38 |
| like the default OUT_NONE but trigger a render | 21:00.57 |
Robin_Watts | With the nul: special case removed so we actually call the output functions... 10 pages takes 10.250 seconds. | 21:02.16 |
malc_ | Robin_Watts: starting with page 1131 is fairer i think... lots of pretty pictures ahead | 21:02.31 |
Robin_Watts | malc_: On the contrary. I am looking for pages where rendering time is minimal, so as to get the most emphasis on the output times. | 21:03.30 |
| tor8: With the if (out) removed from fz_write... | 21:04.00 |
| 10.352s :) | 21:04.12 |
malc_ | Robin_Watts: the range you cited 0-300 contains quite a lot of vectors graphics (44, 91, etc) | 21:04.33 |
| so uhm | 21:04.35 |
Robin_Watts | so the if (out) in fz_write is actually saving us time if you believe that. | 21:04.40 |
tor8 | repeat the job 100 times :) | 21:04.42 |
malc_ | s;vectors;vector | 21:04.44 |
Robin_Watts | malc_: I cut it back to just 50 pages. | 21:04.53 |
| tor8: I've repeated the timings 5 times each. | 21:05.01 |
| As far as I can tell the if (out) doesn't cost us anything. | 21:05.19 |
tor8 | Robin_Watts: you're probably right. | 21:05.31 |
| Robin_Watts: still, -F benchmark might be worth doing anyway | 21:05.41 |
Robin_Watts | and that's the only real objection to the proposed solution, I think. | 21:05.44 |
| tor8: Except we need to say 1bpp or 8bpp as well as colorspace. | 21:06.07 |
tor8 | Robin_Watts: we should make sure all fz_output functions can handle null as well | 21:06.13 |
Robin_Watts | and currently that's done using -F | 21:06.14 |
| tor8: I did :) | 21:06.18 |
| fz_write is the only one that wasn't testing anyway. | 21:06.31 |
tor8 | Robin_Watts: ah. well in that case :) | 21:06.56 |
| Robin_Watts: we have -c for colorspace, mono for 1-bit grayscale | 21:07.05 |
| might want cmykmono for 1-bit cmyk | 21:07.14 |
Robin_Watts | we have no way to say... yeah, that. | 21:07.20 |
| if you're happy, I'll push this stuff, and then ray can retime at his leisure. | 21:07.48 |
tor8 | Robin_Watts: go for it. | 21:08.00 |
Robin_Watts | I want to do some profiles to see why gs beats us for text only output. | 21:08.07 |
| tor8: Thanks. | 21:08.40 |
tor8 | Robin_Watts: the runlength coded font cache, the branching may be slower than brute force bit pushing? | 21:09.16 |
Robin_Watts | tor8: yeah. It may be that we can write an optimised glyph cacher/plotter for 0 bits of AA. | 21:09.58 |
Robin_Watts | is called for food. | 21:10.04 |
rayjj | Robin_Watts: tor: We used to have a -b # option to let us control the output bit depth. My objection to overloading the -F is that currently (AFAIK) the only way to get 1-bit mono is | 23:16.24 |
| -F pbm | 23:16.34 |
| and we've had interest from customers in 2-bit and 4-bit output (with or without AA), so IMHO, keeping bit depth as a separate parameter makes life easier. | 23:18.47 |
| Robin_Watts: tor: I have no problem with using something different to -o /dev/null to just render buffers and skip any writing. In fact, if you implement something reasonable (that doesn't conflict with some gs parameter), I'd be glad to switch gs over | 23:20.22 |
| but good luck on side-stepping the myriad gs command line parameters ;-) | 23:20.48 |
| the thing is that MANY important gs customers take data directly from our buffers | 23:21.33 |
| (or the clist, for one top-ten customer) | 23:22.12 |
| obviously, mupdf isn't going to provide clist output, but since that just goes to their "GRS" graphics library calls now, maybe someday ??? | 23:23.17 |
| BTW, can we *PLEASE* alphabetize the options in the usage ??? If you don't object, I'll be glad to do it | 23:24.58 |
| Forward 1 day (to 2016/03/22)>>> | |