| | <<<Back 1 day (to 2020/03/24) | Fwd 1 day (to 2020/03/26)>>> | 20200325 |
| Robin_Watts_ | Hey scottschafer | 16:03.05 |
| scottschafer | Hi Robin - I was replying to your email but we can try chatting here | 16:25.57 |
| | so I tried opening a PDF with an annotation using the pdf.js viewer app here: http://mozilla.github.io/pdf.js/web/viewer.html | 16:27.01 |
| | The pdf has a text annotation. I used https://tcpdf.org/examples/example_036/ | 16:27.21 |
| Robin_Watts_ | Hi scottschafer, go for it. | 16:27.43 |
| scottschafer | As I suspected, the pdf.js getAnnotations() API returns an array of annotations, and the viewer application renders it | 16:27.48 |
| Robin_Watts_ | ok, where an annotation is what? | 16:28.29 |
| scottschafer | So, if I fully implement getAnnotations() in mupdf.js (again, it returns an empty array now), then the viewer application will render the annotations in the "annotation layer" in the DOM | 16:28.33 |
| | > where an annotation is what? | 16:28.41 |
| Robin_Watts_ | It's a javascript object, presumably? | 16:29.13 |
| | with what methods/fields? | 16:29.24 |
| scottschafer | An annotation is some pdf.js object with fields….the documentation doesn't help much, but I can reverse engineer it and give it a typescript definition | 16:29.29 |
| | anyway, if I implement getAnnotations(), then the viewer app will render the annotations. But also, mupdf will render the annotations through drawPageAsPNG() | 16:30.09 |
| Robin_Watts_ | OK, so change drawPageAsPNG, so that instead of calling fz_run_page(), it calls fz_run_page_contents() | 16:30.51 |
| | How can the caller edit annotations? By tweaking the objects found in the getAnnotations array? | 16:31.54 |
| scottschafer | OK. I guess that's one of my questions. IMO, things will go more easily for us if we have the viewer application render the annotations rather than mupdf | 16:32.08 |
| Robin_Watts_ | scottschafer: Not sure I agree. | 16:32.22 |
| scottschafer | I'm sure there are a lot of considerations here | 16:32.41 |
| Robin_Watts_ | I think having the viewer responsible for calling the render function is fine. | 16:33.01 |
| scottschafer | ok, at least as a first stab at this | 16:33.17 |
| Robin_Watts_ | but the actual plotting of the pixels that make up the annotations can be done by mupdf, right? | 16:33.30 |
| ator | scottschafer: some annotations need to render with a specific blend mode on top of the page (like highlights using 'multiply' blend) | 16:33.37 |
| scottschafer | ok, this is really the question | 16:33.47 |
| Robin_Watts_ | ator: Yeah, but I think we're happy to drop that, aren't we? | 16:33.55 |
| | If we're not happy to drop that, then we can't work to the pdf.js interface. | 16:34.17 |
| ator | Robin_Watts_: we really need the blend mode for highlights. everything else can just use plain alpha compositing. | 16:34.28 |
| | the client can do the blending though, at the web canvas (or however you display them) | 16:34.47 |
| scottschafer | so…if we were to render the annotations in mupdf, then we'd have to: 1) provide a modified version of the viewer application that did not use the annotation layer and 2) modify the page in mupdf and re-render it whenever annotations change | 16:35.07 |
| ator | we can render annotations to separate images, but compositing them with the page needs care for the blend modes | 16:35.13 |
| Robin_Watts_ | ator: Right. The way this stuff works, AIUI is that we have a <div> in the page for the contents and another for the annotations. | 16:36.18 |
| | We render the page to a bitmap, and pass that bitmap back as a data URL. | 16:36.38 |
| scottschafer | one of my challenges is to make a library that uses mupdf but is compatible with the pdf.js API. But if I implement getAnnotations(), then the pdf.js viewer application will try to render them in a div | 16:36.46 |
| Robin_Watts_ | That is then inserted into the div. | 16:36.50 |
| | scottschafer: Is that a fair description? | 16:36.58 |
| scottschafer | yes robin | 16:37.03 |
| ator | scottschafer: right, so the pdf.js viewer app draws the annotations using Javascript and HTML? | 16:37.13 |
| scottschafer | yes | 16:37.18 |
| Robin_Watts_ | I was thinking that we'd do annotations the same way, and get another bitmap (this time a transparent one) and put that in the annotation div. | 16:37.39 |
| ator | change (or add a variant) drawPageAsPNG to call fz_new_pixmap_from_page_contents instead then, I think. | 16:37.44 |
| Robin_Watts_ | but ator seems to be saying that's not possible. | 16:37.49 |
| | or not good enough. | 16:37.54 |
| ator | Robin_Watts_: if the rendered annotations are PNG files with transparent areas, that might work to overlay them | 16:38.19 |
| | but we might have some difficulty telling it to do the multiply blend mode for highlights | 16:38.34 |
| Robin_Watts_ | ator: Right, but we can't get funky blend modes that way. | 16:38.38 |
| | snap. | 16:38.40 |
| ator | depends, might be possible with CSS | 16:38.47 |
| Robin_Watts_ | ator: We'd have a single annotation bitmap, AIUI. | 16:39.07 |
| | so even if we could apply blendmodes to the whole bitmap, that'd probably be wrong. | 16:39.25 |
| ator | ah, then that would not work. we'd need one bitmap per annotation (or two, one for normal and one for multiply) | 16:39.28 |
| Robin_Watts_ | ator: And even then we could have parts of an annotation using a blend mode, and parts not. | 16:39.49 |
| ator | Robin_Watts_: not worth worrying about, IMO. the only ones that use blend modes in practice are highlight annots | 16:40.13 |
| | and they always use multiply | 16:40.16 |
| Robin_Watts_ | (or maybe we couldn't, depends how complex they are). | 16:40.20 |
| | rendering highlights without multiply looks crap? or doesn't show up at all? | 16:40.46 |
| ator | any other types would be non-standard annotation types with funky appearance streams, not sure we'd care enough about those cases | 16:40.56 |
| scottschafer | if we went the route of mupdf rendering annotations, why not just have it render the page with annotations? | 16:40.59 |
| | and not bother with the layer? | 16:41.09 |
| ator | Robin_Watts_: they show up as opaque rectangles blocking out the text they're supposed to highlight | 16:41.14 |
| | like a redaction, basically | 16:41.22 |
| | scottschafer: that's what mupdf does now isn't it? the drawPageAsPNG certainly renders annotations. | 16:42.03 |
| Robin_Watts_ | scottschafer: That depends if callers will get confused expecting the annotations to not be in the page bitmap. | 16:42.03 |
| ator | "background-blend-mode: multiply" is a thing in CSS | 16:42.31 |
| scottschafer | it is, it doesn't work in IE though | 16:42.41 |
| | I had to fake it in another project with pixel manipulation in canvas | 16:42.56 |
| ator | scottschafer: ah, not even in edge? | 16:42.59 |
| scottschafer | nope | 16:43.04 |
| Robin_Watts_ | scottschafer: For instance, I could imagine that callers might not expect to have to redraw the page bitmap once they edit an annotation. | 16:43.11 |
| ator | then scratch that for now, or use canvas for everything? | 16:43.18 |
| | or WebGL (shudder) | 16:43.25 |
| Robin_Watts_ | ator: That won't fit under the pdf.js interface that we're trying to work to. | 16:43.37 |
| scottschafer | yeah | 16:43.55 |
| | so again….if I implement getAnnotations(), then the pdf.js viewer will attempt to render them in the annotations layer | 16:44.30 |
| Robin_Watts_ | scottschafer: HOW will attempt to render them? | 16:44.44 |
| | It will call the 'render' method for each annotation? | 16:44.54 |
| ator | is there a primer on pdf.js somewhere? | 16:45.13 |
| scottschafer | it's just built into the viewer application http://localhost:8888/web/viewer.html. There's a file called "annotation_layer.js" | 16:45.33 |
| Robin_Watts_ | So, the annotation rendering code is not actually under the pdf.js interface? | 16:46.03 |
| | i.e. every caller of pdf.js is expected to implement its own annotation rendering? (which presumably they all do by using the same annotation_layer.js) | 16:46.35 |
| scottschafer | pdf.js is the low-level library providing APIs, but the viewer application provides the UI and builds a lot of functionality on top of pdf.js. It's what polar-bookshelf uses and I think many other pdf.js users | 16:46.38 |
| | > every caller of pdf.js is expected to implement its own annotation rendering | 16:46.56 |
| | yes | 16:46.57 |
| Robin_Watts_ | OK. So there are a couple of ways we could go here that I can see. | 16:47.11 |
| scottschafer | unless they use the viewer application directly in their code | 16:47.15 |
| Robin_Watts_ | 1) We can return a list of annotations with duff types, so that the annotation_layer doesn't render them. | 16:47.50 |
| | 2) We can imagine that annotation_layer.js is part of a combined "pdf.js + annotations" API and work to make mupdf.js implement all of that. | 16:48.30 |
| | How are annotations created? | 16:48.56 |
| | Is there an annotation_layer.js method like "createAnnotation"? | 16:49.12 |
| scottschafer | currently pdf.js doesn't support modifying pdfs, so we'd have to add that | 16:49.24 |
| Robin_Watts_ | but annotations can be created/edited, right? They just don't go into the PDF. They are faked as non-PDF things. | 16:50.02 |
| scottschafer | no, that's something that polar-bookshelf hacked together. It tracks and renders its "annotations" separately (in a firebase DB) | 16:50.45 |
| Robin_Watts_ | Ah, ok. | 16:50.53 |
| scottschafer | So #1 wouldn't be strictly pdf.js API compatible. You could imagine a use case where you wanted to call getAnnotations() without rendering | 16:51.35 |
| Robin_Watts_ | indeed. #1 doesn't appeal to me. | 16:51.58 |
| scottschafer | maybe not a big deal | 16:51.59 |
| Robin_Watts_ | cos at the very least I'd like to be able to say "what type of annotation is this?" in the caller. | 16:52.23 |
| scottschafer | yeah | 16:52.29 |
| Robin_Watts_ | I think, my favoured way to work would be for us to replace annotation_layer.js too. | 16:52.53 |
| | If we do that, we have more freedom. | 16:53.11 |
| | Suppose we do that, and we have mupdf rendering the annotations as we do now. | 16:54.05 |
| | The 'render' function for each individual annotation can do nothing. | 16:54.23 |
| scottschafer | We could give the annotation layer 0 opacity. So the user could still click on elements, etc, but mupdf would render. Not sure about performance though. | 16:54.24 |
| Robin_Watts_ | scottshafer: OOOOH, that's clever. | 16:54.48 |
| | I like that. | 16:54.54 |
| scottschafer | Dragging annotations around would be very choppy I'd imagine. Is that something we'd want to do? | 16:55.06 |
| Robin_Watts_ | As long as you can still click on an annotation even when it's got an alpha of 0. | 16:55.39 |
| | generally you don't drag the annotation live, you drag a bbox for it. | 16:55.54 |
| scottschafer | ok, sounds like this approach is worth pursuing | 16:56.16 |
| Robin_Watts_ | For editing text, it means that the whole doc gets redrawn on every change. | 16:56.40 |
| scottschafer | yeah, I suppose we could try having a creation mode in which the viewer application renders the annotations while you were acting on them and then commits to the mupdf page | 16:57.24 |
| Robin_Watts_ | So *maybe* we want to consider having 2 bitmaps for such cases. | 16:57.24 |
| scottschafer | or yeah, do that | 16:57.40 |
| Robin_Watts_ | scottschafer: Well, you want fonts to match. | 16:57.41 |
| scottschafer | true | 16:57.53 |
| Robin_Watts_ | So either you abandon the thought of doing inline editing for text, and pop up a simple modal window, OR you have to get mupdf to render. | 16:58.11 |
| | I believe we do the former in mupdf-gl, but I could be wrong. | 16:58.32 |
| | Using a modal window means you don't have to worry about matching fonts etc. | 16:59.04 |
| ator | we do the former in mupdf-gl --getting inline editing to work well is a lot of work | 16:59.16 |
| scottschafer | I'm happy to do the modal window | 16:59.33 |
| Robin_Watts_ | ator: yeah. so much work that I don't think we should be ashamed about punting on it for now. | 16:59.41 |
| ator | in HTML you could just drop an input field on the right place | 16:59.46 |
| Robin_Watts_ | scottschafer: Fab. sounds like we're agreed on that then. | 16:59.53 |
| scottschafer | sounds like I have some things to try. let's check in again after I've made some progress and go from there | 17:00.16 |
| Robin_Watts_ | scottschafer: Fab. | 17:00.23 |
| scottschafer | thanks to both you! ttyl | 17:00.34 |
| Robin_Watts_ | Do you want to register your nickname? | 17:00.37 |
| | If you do I can get you into the private channel. | 17:00.53 |
| scottschafer | sure…not sure how though | 17:01.08 |
| Robin_Watts_ | just a tick... | 17:01.15 |
| | try: /msg nickserv register ? | 17:01.44 |
| scottschafer | ok, thanks - think that did it | 17:02.59 |
| Robin_Watts_ | ok, now I need to do some magic. just a mo. | 17:03.13 |
| | ok, try doing: /msg chanserv invite #artifex | 17:04.58 |
| scottschafer | chanserv | 17:06.01 |
| | (notice) You are not authorized to perform this operation. | 17:06.01 |
| Robin_Watts_ | ok, just a mo... | 17:06.20 |
| | Apparently you've not completed registration verification. | 17:06.46 |
| | that might be why. | 17:06.50 |
| | You should get a mail from nickserv with some magic runes in it. | 17:07.15 |
| scottschafer | hm, nothing yet | 17:07.43 |
| | I'm going to have to go, let's figure this out later | 17:08.13 |
| Robin_Watts_ | Sure. np. | 17:08.17 |
| | ttyl. | 17:08.19 |
| scottschafer | ttfn | 17:08.23 |
| Robin_Watts_ | we're always here if you need to talk. | 17:08.25 |
| | too slow. | 17:08.29 |
| | <<<Back 1 day (to 2020/03/24) | Forward 1 day (to 2020/03/26)>>> | |
Log of #mupdf at irc.freenode.net.