| | <<<Back 1 day (to 2020/10/21) | Fwd 1 day (to 2020/10/23)>>> | 20201022 |
| sh4rm4^bnc | added libmupdf support to my ebook viewer https://github.com/rofl0r/SDLBook/ | 00:27.10 |
| | unlike mupdf-x11 it supports smooth scrolling over page boundaries | 00:27.27 |
| malc_ | sh4rm4^bnc: https://tpaste.us/QW4b is needed on my system to build sdlbook thusly: "LIBRARY_PATH=$HOME/x/rcs/git/mupdf/build/native CPATH=$HOME/x/rcs/git/mupdf/include make" | 03:06.22 |
| sh4rm4^bnc | malc_, is that with a static build ? | 03:14.38 |
| | of mupdf | 03:14.50 |
| malc_ | sh4rm4^bnc: yes | 03:16.58 |
| sh4rm4^bnc | hmph, it seems mupdf ships without pkgconfig file | 03:17.50 |
| | i've never seen CPATH and LIBRARY_PATH env vars, what are they for ? | 03:19.20 |
| | also its odd that you need -lm | 03:19.37 |
| malc_ | sh4rm4^bnc: back in the day motivation for starting my own program to read pdfs was also continuous scroll, it even had a optional djvu support years ago, but i've abandoned it | 03:19.50 |
| | sh4rm4^bnc: they are like adding -I and -L respectively to gcc/clang | 03:20.34 |
| | sh4rm4^bnc: why is it odd? | 03:20.49 |
| sh4rm4^bnc | because i removed usage of ceil(), the only math func i used | 03:21.08 |
| | is it required by libmudf ? | 03:21.29 |
| | does make process the CPATH and LIBRARY_PATH vars ? | 03:21.50 |
| malc_ | sh4rm4^bnc: gcc/clang do | 03:24.12 |
| | /usr/bin/ld: /home/malc/x/rcs/git/mupdf/build/native/libmupdf-third.a(one.o): undefined reference to symbol 'acos@@GLIBC_2.2.5' | 03:25.48 |
| | | 03:25.48 |
| sh4rm4^bnc | interesting. btw there's also config.mak where you could stuff your C/LDFLAGS into | 03:26.25 |
| | ok, will re-add -lm then in any case | 03:26.40 |
| | i use musl libc which has libm inside libc so i dont catch such errors | 03:27.19 |
| | what's your impression so far of the prog? | 03:30.44 |
| malc_ | sh4rm4^bnc: well, i'm biased. | 03:32.16 |
| sebras | sh4rm4^bnc: https://bugs.ghostscript.com/show_bug.cgi?id=703024 | 10:26.51 |
| malc_ | feels almost like Elon Masks latest kid reported the issue | 10:36.35 |
| sebras | malc_: ? | 10:41.07 |
| Robin_Watts | sebras: We should run that patch past cgdae, cos he uses openbsd, which is awkward for the sake of it. | 10:44.52 |
| | likewise, it'd be good to test it on the mac. | 10:45.08 |
| sebras | Robin_Watts: that's why I didn't simply apply sh4rm4^bnc's patch. | 10:45.12 |
| Robin_Watts | yeah, no criticism at all! | 10:45.28 |
| sebras | Robin_Watts: I know. sh4rm4^bnc seems impatient though. :) | 10:46.07 |
| Robin_Watts | sebras: Any thoughts on: https://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=eaa20f4e9f9a7dcca1d3d1cc7f27736b8e092416 | 10:46.13 |
| sebras | s/seems/seemed/ | 10:46.15 |
| | Robin_Watts: I'm looking at replacing some of the digital signatures fixes with pauls suggestion of removing cleared signatures from the unsaved signatures list. | 10:47.09 |
| | Robin_Watts: I have found differences to the current approach but I don't know why yet. | 10:47.24 |
| | Robin_Watts: I'd prefer to work on that until settled, if that's ok? | 10:47.41 |
| Robin_Watts | I'm trying to parse that... | 10:47.53 |
| sebras | Robin_Watts: TLDR: not now. | 10:48.08 |
| Robin_Watts | I mean, the bit about what paul is suggesting. | 10:48.19 |
| sebras | Robin_Watts: ok. | 10:48.24 |
| | Robin_Watts: remember the commit that removed the signer from the signature info stored in doc->unsaved_sigs? | 10:48.52 |
| | Robin_Watts: paul proposed removing the entire entry from doc->unsaved_sigs, not just clearing its signer. | 10:49.11 |
| Robin_Watts | right. | 10:49.17 |
| sebras | Robin_Watts: in an attempt to undo everything that pdf_sign_signature() did. | 10:49.28 |
| | Robin_Watts: when I tested this I didn't end up with any exceptions | 10:49.40 |
| | Robin_Watts: but I _do_ end up with xref sections whose /Prev skip over a prior xref sections. | 10:50.01 |
| Robin_Watts | so if I sign a sig, don't save it, and then clear the sig the document basically doesn't change? | 10:50.03 |
| sebras | I don't know why it skips. | 10:50.06 |
| Robin_Watts | Getting a new xref section, I would expect. | 10:50.20 |
| sebras | Robin_Watts: open a form pdf, sign a signature, clear the signature, sign the signature with another cert, save the file. | 10:50.27 |
| Robin_Watts | but skipping over... that seems odd, yes. | 10:50.28 |
| | OK. I'll shut up and let you be. | 10:50.46 |
| | Yell if I can help. | 10:50.51 |
| sebras | Robin_Watts: yes, my initial thought is that pdf_clear_signature() doesn't yet fully undo what pdf_sign_signature() did because one of the things it did was to request a new xref section and update the annotations appearance stream. I think both of those changes are still in the file when saved. | 10:51.29 |
| | they are saved with the current approach as well, but the /Prev pointer in the Root object doesn't skip over the xref section where the signature was cleared. | 10:52.10 |
| | at least this is what I _think_ is happening. | 10:52.18 |
| | Robin_Watts: do you mind looking at the freetype things jogux raised? https://bugs.ghostscript.com/show_bug.cgi?id=703022 | 10:52.50 |
| | Robin_Watts: that issue seems to be high prio for them as well. | 10:53.09 |
| Robin_Watts | ok. | 10:56.06 |
| jogux | Chris said it wasn’t exploitable in gs. I have no idea if that applies to mupdf. | 10:56.33 |
| | Chris said “Since we only allow outline glyphs, it's not actually going to affect us” | 10:57.01 |
| malc_ | sebras: it sucks having to explain (attempted) humor... https://en.wikipedia.org/wiki/Elon_musk#Relationships | 10:57.09 |
| | cf sh4rm4^bnc | 10:57.29 |
| Robin_Watts | jogux: It probably won't hurt for me to pull freetype up to the latest version anyway. | 10:57.37 |
| sebras | malc_: ah. :) | 10:57.41 |
| Robin_Watts | jogux: gs and mupdf use freetype differently. | 10:57.57 |
| | mupdf uses a bit more of it's capabilities than gs does, AIUI. | 10:58.08 |
| jogux | Robin_Watts: agree. knowing if it is exploitable or not may still help. (Lisa may need to send out a notification…) | 10:58.14 |
| Robin_Watts | but quite possibly not this much. | 10:58.26 |
| | jogux: Well, that'd be a question for ator, I suspect. | 10:58.51 |
| jogux | https://savannah.nongnu.org/bugs/?59308 has a link to the font that causes the problem. Not sure what the easiest way to try and get mupdf to process that font is. | 11:00.32 |
| sebras | Robin_Watts: jogux: probably to use mutool create (https://mupdf.com/docs/manual-mutool-create.html) to build a PDF that references that font file. | 11:04.50 |
| jogux | I’m surprised no one else seems to have created such a pdf yet (or at least they’ve not made it public if they have…). | 11:15.29 |
| Robin_Watts | I have created a PDF. It doesn't crash. | 11:20.26 |
| jogux | I’m not sure if it’s meant to crash or just show an address sanitiser issue | 11:20.45 |
| Robin_Watts | will test with sanitize and valgrind. | 11:21.29 |
| jogux | you can probably breakpoint also breakpoint load_sbit_image, if that’s not called I suspect we can’t end up in the exploitable code. Or maybe Load_SBit_Png as that’s the problem code I think. | 11:22.05 |
| Robin_Watts | sanitize doesn't complain. | 11:24.31 |
| sebras | Robin_Watts: tested valgrinding the debug build too? | 11:25.29 |
| | Robin_Watts: sometimes valgrind catches things ASAN doesn't | 11:25.38 |
| malc_ | jogux: why would you need pdf? fitz can do html and external fonts within them | 11:26.31 |
| | so it's rather simple really | 11:26.40 |
| jogux | malc_: well it wasn’t my suggestion :-) But you make a good point, even if it’s not expolitable via pdf it may be exploitable via html. | 11:27.44 |
| Robin_Watts | valgrind also works fine. | 11:28.47 |
| malc_ | jogux: and i've just tried, mupdf errs on this font | 11:29.10 |
| Robin_Watts | Load_SBit_Png calls pnglib, right? | 11:29.18 |
| jogux | yeah, I believe so. | 11:29.27 |
| Robin_Watts | We don't use pnglib in mupdf, ergo, we can't be affected. | 11:29.31 |
| malc_ | i take that back... | 11:29.52 |
| sam_ | (so what did you make crash? ;)) | 11:31.06 |
| malc_ | sam_: no | 11:31.31 |
| jogux | Robin_Watts: This seems like a good point :-) | 11:31.35 |
| malc_ | sam_: still i have used the wrong methodology | 11:31.48 |
| sam_ | ahh | 11:31.52 |
| jogux | I guess we must have this code path disabled in freetype somehow. | 11:33.37 |
| | presumably FT_CONFIG_OPTION_USE_PNG just isn’t defined. | 11:35.19 |
| | yeah, that seems right. scripts/freetype/slimftoptions.h:#undef FT_CONFIG_OPTION_USE_PNG. ace. | 11:37.08 |
| | So there’s no panic to upgrade freetype. Thanks Robin_Watts. | 11:43.54 |
| sam_ | thanks all :) | 12:01.53 |
| user38 | Greetings everyone... | 12:09.35 |
| kens | Afternoon user38 | 12:10.36 |
| Robin_Watts | user38: Hi | 12:10.40 |
| | I updated the patch last night, a link to it is on the bug. | 12:10.56 |
| user38 | Robin_Watts, I applied the patch this morning, after I took out a/ and b/, it applies cleanly, but now the 64-bit version of mupdf-x11 works while the 32-bit version crashes. I've been working on getting build=debug working since. It's compiling the 32-bit version now, hopefully it will finish. | 12:13.15 |
| Robin_Watts | Ta. | 12:18.23 |
| user38 | Robin_Watts, the crash is still happening at 0x0047acfc in cmsSetAdaptationState (ContextID=ContextID@entry=0x2759f38, d=-1) at thirdparty/lcms2/src/cmsxform.c:64 | 13:20.02 |
| | 64 prev = ptr ->AdaptationState; | 13:20.17 |
| | That's 32-bit mupdf-x11. 32-bit mupdf-gl crashes in the same place. | 13:21.22 |
| | I'll go build=debug 64-bit now. | 13:22.48 |
| Robin_Watts | user38: Even with CMS_WOSSNAME set to 8 ? | 13:25.27 |
| user38 | Let me check. | 13:25.42 |
| Robin_Watts | CMS_PTR_ALIGNMENT | 13:26.49 |
| user38 | Robin_Watts, that's gone: | 13:29.16 |
| | > fgrep CMS_PTR_ALIGNMENT ./thirdparty/lcms2/src/lcms2_internal.h#ifndef CMS_PTR_ALIGNMENT# define CMS_PTR_ALIGNMENT sizeof(void *)#define _cmsALIGNMEM(x) (((x)+(CMS_PTR_ALIGNMENT - 1)) & ~(CMS_PTR_ALIGNMENT - 1)) | 13:29.29 |
| | ARGH! | 13:29.35 |
| | I removed my patch, thinking yours would put it back in... | 13:30.06 |
| | Robin_Watts, okay, let me put it back in... | 13:31.09 |
| | Then it's clear why it's crashing. | 13:31.18 |
| Robin_Watts | user38: Right. lcms is a separate beast. It's a submodule. | 13:32.36 |
| | I have a fix for that working its way through, but it's not there yet. | 13:32.52 |
| | user38: Thanks! | 13:33.02 |
| user38 | I'm so sorry. | 13:33.16 |
| Robin_Watts | no worries at all. | 13:33.29 |
| sh4rm4^bnc | sebras, thanks for adding the bug report. the description isnt quite right, mupdf does build successfully but mupdf-x11 fails at runtime because the full path to the .so is hardcoded in DT_NEEDED table | 13:50.15 |
| | malc_, does your joke refer to the continuous renaming of musk's kid ? | 13:50.44 |
| Robin_Watts | and the fact that he is named with lots of strange characters. | 13:51.35 |
| malc_ | sh4rm4^bnc: you and sebras are hell bent on making me doubt my satirical prowess don't you?? | 13:51.48 |
| | finally a voice or reason | 13:52.10 |
| sh4rm4^bnc | :) | 13:52.13 |
| malc_ | and from an englishman no less... my dry rosso humor and stiff lower lip are probably close enough... | 13:53.01 |
| sh4rm4^bnc | in case the patch isnt understood: $^ expands to all dependencies of the target, we pass it through subst to replace foo/bar/libmupdf.ext with -Lfoo/bar -lmupdf | 13:53.22 |
| | and the path is saved in $(OUT) | 13:54.06 |
| | $(subst...) is a GNU make feature, but i believe i had seen other GNU-specific things, though i might be wrong | 13:55.23 |
| malc_ | sh4rm4^bnc: the Make* files are choke full of GNUisms the very first non comment line in Makerules has two gnuisms in it | 13:58.06 |
| user38 | Robin_Watts, after putting the lcms2 patch back in, except for the 64-bit mupdf-gl, all other mupdf-* 32- and 64-bit binaries work, and that without build=debug and with -O3. | 14:24.23 |
| | I find it curious that the 32-bit mupdf-gl works while the 64-bit doesn't. | 14:25.10 |
| | Thank you kindly for your help. | 14:25.34 |
| | For me this is a good stopping point for the day, I'll be checking the bug for updates (if any) tomorrow morning and checking in here afterwards. I wish you all a nice and pleasant evening / the rest of the day. | 14:27.28 |
| sh4rm4^bnc | i'm using pix = fz_new_pixmap_from_page(ctx, page, ctm, fz_device_rgb(ctx), 0) to render a pdf page, but it seems quite slow (takes up to 300 ms on my machine). is there a faster way ? | 14:28.15 |
| Robin_Watts | sh4rm4^bnc: Depends what you mean. | 14:29.06 |
| | Clearly the time is dependent on page contents. | 14:29.20 |
| sh4rm4^bnc | right. in this case it's scanned books from the last century. | 14:29.44 |
| Robin_Watts | If you're repeatedly rendering the same page (perhaps at different zooms) then using a displaylist will help. | 14:30.01 |
| | Ah, if you use scans of more modern books, they'll probably be faster. | 14:30.25 |
| sh4rm4^bnc | example https://0x0.st/iDlf.png | 14:31.32 |
| | i've been examining whether i could request a 8 or 16 bit image format but that doesn't seem to be available | 14:34.10 |
| Robin_Watts | sh4rm4^bnc: request from what? | 14:34.25 |
| sh4rm4^bnc | from fz_new_pixmap_from_page() via the colorspace | 14:34.55 |
| malc_ | sh4rm4^bnc: you can have 16bit BW image | 14:35.20 |
| Robin_Watts | You can get an 8bpp greyscale image. | 14:35.59 |
| | You can't get an 8bpp color image. | 14:36.06 |
| malc_ | sh4rm4^bnc: i would also recommend using llpp as the benchmark, it is (to the best of my knowledge) most efficient pdf reader out there | 14:36.09 |
| sh4rm4^bnc | i see only fz_device_gray/rgb/bgr/lab/cmyk | 14:36.41 |
| malc_ | Robin_Watts: huh? the b&w pixmap mupdyf produces is 16bit... y+alpha | 14:36.58 |
| sh4rm4^bnc | how would i request those formats? is fz_device_gray 8bit ? | 14:38.21 |
| Robin_Watts | sh4rm4^bnc: It is. | 14:47.49 |
| | I suspect most of the time is in the decoding of the image, and possibly the scaling of it. | 14:48.20 |
| | the decoding of the image will be the same time regardless of what colorspace you ask for. | 14:48.42 |
| | malc_: mupdf can produce stuff with or without alpha. | 14:49.07 |
| sh4rm4^bnc | i see. so requesting an inferior format might only save some milliseconds | 14:49.24 |
| | probably i should use a thread that preloads bordering pages | 14:49.52 |
| | so there's no delay in the UI | 14:50.14 |
| Robin_Watts | sh4rm4^bnc: That would work. | 14:50.20 |
| | but as I say, if you're planning to allow stuff to be zoomed, then use a display list. | 14:50.56 |
| | what format are the images within the PDF? | 14:51.18 |
| sh4rm4^bnc | i have no idea | 14:51.27 |
| | what's the advantage re: zooming with a display list ? | 14:52.17 |
| Robin_Watts | Display list holds everything nicely in memory to rerender the page. | 14:53.46 |
| | It removes the PDF format parsing overhead. | 14:53.58 |
| | On top of that, subsequent renders will be faster anyway, assuming you have a reasonably sized store, cos the images will be cached decompressed. | 14:54.37 |
| malc_ | Robin_Watts: fz_device_colorspace is Y+A | 15:08.47 |
| Robin_Watts | Nope. | 15:09.03 |
| | Trust me, I wrote large sections of the rendering code. :) | 15:09.33 |
| | It is perfectly possible to get mupdf to return rendered pixmaps with no alpha plane. | 15:09.54 |
| | It maybe that the particular helper function you are using returns alpha by default, but it doesn't have to. | 15:10.25 |
| malc_ | Robin_Watts: i do trust you, however when i specify fz_device_gray i get Y+A which i feed to LUMALPHA textures | 15:10.25 |
| | Robin_Watts: arguing from my side is stupid, you are either absolutely right and i miss some api detail or deranged and beyond salvation | 15:11.48 |
| Robin_Watts | fz_pixmap *fz_new_pixmap_from_page(fz_context *ctx, fz_page *page, fz_matrix ctm, fz_colorspace *cs, int alpha); | 15:12.47 |
| | IF you pass alpha = 1, you get alpha data. If you pass alpha = 0, you should get just raw 8 bit data. | 15:13.07 |
| sh4rm4^bnc | oh interesting | 15:15.36 |
| | is the pixel data packed ? if so i could request rgb+alpha and just memcpy it onto my surface | 15:16.00 |
| | but if stride is > x*bpp then that won't work | 15:16.27 |
| | s/x/w/ | 15:16.39 |
| | though the pixel-copy part only takes 8ms anyway | 15:17.35 |
| malc_ | Robin_Watts: i don't use fz_new_pixmap_from_page | 15:19.13 |
| Robin_Watts | sh4rm4^bnc: If you get mupdf to make the pixmaps, then the stride will always be n*w. | 15:22.30 |
| | (where n = 2 for grey+alpha, or 1 for just grey) | 15:22.47 |
| | If you want a different stride, then you have to create the pixmap structure yourself. | 15:23.02 |
| | but mupdf will honour it. | 15:23.16 |
| sh4rm4^bnc | nice, saves one conversion pass then | 15:23.38 |
| Robin_Watts | MuPDF data is always packed as <process colors><spot colors><alpha> where some of those may be optional. | 15:23.56 |
| sh4rm4^bnc | hmmpf, caching display lists doesnt seem to make a big difference | 21:29.19 |
| | rendering from the display list with zoom factor applied still takes 280ms | 21:30.20 |
| | isn't there any function to apply a fz_matrix for scaling to an existing fz_pixmap? | 22:19.25 |
| | i had hoped to be able to use fz_get_pixmap_from_image to do so, but it doesnt seem to work | 22:20.01 |
| | that would allow me to save the rendered page at original resolution, then scale as needed when drawing | 22:34.41 |
| Moult | peeks in the channel | 22:42.30 |
| Robin_Watts | sh4rm4^bnc: There is a pixmap scaling function, yes. | 23:22.04 |
| | fz_scale_pixmap, but it's not in the public API, just the internal one, currently. | 23:22.25 |
| sh4rm4^bnc | i just discovered it reading the source | 23:23.06 |
| | scaled = fz_scale_pixmap(ctx, image, m.e, m.f, m.a, m.d, clip) | 23:23.14 |
| | hmm, doesn't seem to like clip being NULL... the scaled image returned is 2x2 | 23:31.26 |
| Robin_Watts | I'm off to bed. I'm going to be playing with that tomorrow myself, I think. | 23:32.31 |
| | FWIW, my test code with that works fine with clip being NULL. | 23:33.27 |
| sh4rm4^bnc | ok thanks | 23:35.14 |
| | it works when i replace the values from the transform (with a zoomfactor of ~ 1.5 in m.a/m.d) with the w/h of the output rect | 23:45.41 |
| | s/transform/matrix/ | 23:45.56 |
| | hmm, applying that scale func produces a far less crispy image | 23:51.43 |
| | https://0x0.st/iD5p.png | 23:53.56 |
| | oh hmm, wonder where that 120 dpi value in mupdf's title comes from | 23:55.35 |
| | <<<Back 1 day (to 2020/10/21) | Forward 1 day (to 2020/10/23)>>> | |
Log of #mupdf at irc.freenode.net.
