| | <<<Back 1 day (to 2017/05/28) | 20170529 |
| Robin_Watts | tor8: Right, this smaskindata thing... | 11:34.46 |
| | We already have code that reads the alpha plane within a JPX file as an alpha plane for the image. | 11:35.35 |
| tor8 | Robin_Watts: I was under the impression we handled SMaskInData already by virtue of supporting alpha channels in fz_image | 11:35.44 |
| Robin_Watts | SMaskInData says there is an extra alpha channel. | 11:35.51 |
| tor8 | does that mean there are *two* alpha channels? | 11:36.02 |
| Robin_Watts | So a cmyk + alpha + smaskindata image will have 6 channels. | 11:36.06 |
| | tor8: Yes. | 11:36.10 |
| | I can do you a bmpcmp run to show you the clear progressions if you want. | 11:36.46 |
| tor8 | "If SMaskInData is nonzero, there must be only one opacity channel in the JPEG2000 data and it must apply to all color channels." | 11:36.52 |
| | says the 1.7 spec | 11:37.02 |
| Robin_Watts | tor8: spec lies :( | 11:37.13 |
| | fts_17_1700.pdf is a case in point. | 11:38.31 |
| | It contains a 1200x1600 CMYK image with 6 channels of data. | 11:39.01 |
| tor8 | openjpeg warning: Bad COLR header box (bad size: 35) | 11:39.07 |
| | warning: assert: overwrite hash slot | 11:39.11 |
| | ick | 11:39.13 |
| Robin_Watts | and 1701, and 1704, and 1705, and 184. | 11:39.17 |
| | and 1701, and 1704, and 1705, and 1804. | 11:39.19 |
| tor8 | Robin_Watts: it looks like in 1804 the alpha channel is applied twice | 11:43.39 |
| | with your commit | 11:43.53 |
| | does gs do the right thing here? | 11:44.08 |
| Robin_Watts | tor8: I believe with my commit acrobat, gs and mupdf all agree. | 11:44.25 |
| | Let me double check | 11:44.29 |
| tor8 | I suspect we should only need to look at SMaskInData and nuke the alpha channel if it is 0, and do some other voodoo if it is 2 | 11:46.25 |
| | IIRC we handle it as always being 1 | 11:46.32 |
| | but this is assuming the spec is correct when it states there is 1 alpha channel | 11:46.46 |
| Robin_Watts | My memory of the one in 1700.pdf is that it has 6 channels, and the 2 alphas are both different. | 11:48.24 |
| | Feel free to poke at it yourself. 2 sets of eyes etc. | 11:48.44 |
| avih | tor8: do you plan to support 0 in strings? i don't think it's critical, but if you don't, i'd probably add this info to my docs | 11:50.47 |
| | i believe api wise you're half way there. there's pushlstring but not tolstring. implementation wise, no clue :) | 11:52.08 |
| tor8 | avih: no such plans in the immediate future, implementation wise it's going to be a fair amount of work. | 12:00.36 |
| avih | k, thx | 12:00.46 |
| tor8 | the API is designed so that it will work with a minimum amount of change | 12:00.57 |
| avih | was it really designed with that in mind? | 12:01.26 |
| tor8 | the public api? yes. | 12:01.51 |
| avih | (other than pushlstring, which is an api function, but not "design" in general) | 12:02.09 |
| tor8 | but I don't expose a js_tolstring because it's not implemented | 12:02.13 |
| avih | may i ask where it would have an effect on the api design other than adding a tolstring? | 12:03.00 |
| tor8 | what is an api function if not design? | 12:03.06 |
| sebras | Robin_Watts: just a question: the reason that the luratech hbased jpx_read_image() always returns mask == NULL is that I'm supposed to add that bit of code, right? ;) | 12:03.10 |
| Robin_Watts | sebras: oh, yeah, I forgot we'll need to do luratech too :( | 12:03.50 |
| avih | tor8: an api function is local. design is a more holistic thing which encompass how different functions complete eachother, imo | 12:03.59 |
| sebras | Robin_Watts: ideally at least. | 12:04.09 |
| Robin_Watts | sebras: No, we certainly need to do it. | 12:04.22 |
| | either you can do it, or I can/ | 12:04.28 |
| sebras | Robin_Watts: I'm still tryign to piece together what this thing does. | 12:04.42 |
| | Robin_Watts: I'm thinking that you are alreay up to speed somewhat on it, so the momentum would help you. | 12:05.00 |
| Robin_Watts | If SMaskInData is set, then we should look for an extra alpha channel and return that in 'mask'. | 12:05.19 |
| | If no such extra channel exists (as is the case in at least one of our test files), we should not return it. | 12:05.42 |
| | sebras: I'm happy to give it a go. | 12:06.14 |
| | but having another set of eyes on it wouldn't be a bad idea. | 12:06.28 |
| sebras | Robin_Watts: of course. I'm looking at the first patch already. | 12:07.33 |
| avih | so WRT to tolstring, i'd say it definitely fits in the design, but i don't see specific design considerations needed to make it so | 12:08.38 |
| | (which are unique to tolstring) | 12:09.48 |
| tor8 | Robin_Watts: with your patch 1700 looks better, but not the same as gs | 12:10.23 |
| | it still looks like we're squaring the alpha channel | 12:10.31 |
| Robin_Watts | thinks I should maybe let tor8 play with this for a while. | 12:11.06 |
| | I tried all sorts of different combinations, and this was the best I could come up with. | 12:11.25 |
| | If you can find better, great. | 12:11.31 |
| avih | thinks Robin_Watts should first make sure he has a backup :p | 12:11.37 |
| tor8 | I suspect it might be down to us not coping with CMYK and alpha correctly | 12:11.44 |
| Robin_Watts | avih: git FTW. | 12:11.47 |
| tor8 | when loading JPX images | 12:11.50 |
| avih | :) | 12:11.55 |
| kens | Regarding FTS_176_1700, Ghostscript gives a whole slew of warnings on that file | 12:14.06 |
| tor8 | Robin_Watts: but holy crap what do they expect us to do with that file? the PDF image dictionary has no colorspace info at all, and the JPX has a broken color-header | 12:14.22 |
| kens | GS appears to match Acrobat on FTS_17_1700 | 12:14.58 |
| | But the back channel says we specifically ignore a bunch of stuff | 12:15.15 |
| sebras | microns reports: nodeFail: too many timeouts | 12:15.55 |
| | I'm not sure what to make of my cluster run. :-/ | 12:16.01 |
| kens | I'd suggest that means it failed | 12:16.09 |
| | Probably triggered endless loops | 12:16.19 |
| Robin_Watts | sebras: You broke stuff :) | 12:16.20 |
| sebras | tor8: does it _really_ have a broken color header or is it just openjpeg reporting this? | 12:16.35 |
| tor8 | well ... yeah | 12:18.13 |
| | gimp refuses to open the file | 12:18.30 |
| kens | "Acrobat can open it " :-D | 12:19.23 |
| sebras | tor8: what does the luratech CLI decoder make of it? | 12:19.35 |
| kens | WHich image and file are you looking at ? | 12:19.46 |
| tor8 | Robin_Watts: so, if we assume the file is RGB instead of CMYK when n > 4 we match gs's behaviour | 12:22.53 |
| | we're interpreting the alpha channel as K and then the garbage extra channels as alpha | 12:23.17 |
| | for 1700 | 12:23.26 |
| kens | GS says (multiple times) that its ignoring the second COLR box | 12:23.44 |
| Robin_Watts | ok... if you cluster test, that may bite us for other files. | 12:23.47 |
| kens | We do also flag the bad COLR header | 12:24.21 |
| | We say 'COLR BOX meth value is not a regular value (3), so we will ignore the entire colour specification box (twice) | 12:25.38 |
| | The 'A conforming JP2 reader shall ignore all Colour Specificatio boxes afer the first, so we ignore this one. (twice, different images) | 12:26.13 |
| tor8 | with luratech we match gs behaviour with no modifications | 12:27.18 |
| nilsont | Hy | 12:42.20 |
| sebras | mubot: hi | 12:43.14 |
| mubot | Welcome to #mupdf, the channel for MuPDF. If you have a question, please ask it, don't ask to ask it. Do be prepared to wait for a reply as devs will check the logs and reply when they come on line. | 12:43.14 |
| nilsont | I can't create a type TEXT annotation?? can you help me?? it's for android app | 12:44.00 |
| | also, I don't know how to get the content text inserted? how can I do this? | 12:44.54 |
| Robin_Watts | nilsont: Are you a commercial or supported customer? | 12:45.57 |
| nilsont | yes sr | 12:47.38 |
| kens | Canyou tell us your customer numebr then please ? | 12:48.18 |
| | Or identify the organisationyou work for ? Feel free to contact one of us privately if you don't want to share that information | 12:48.49 |
| | Mysefl, Robin, chrisl, tor8 and sebras all work for Artifex | 12:49.41 |
| nilsont | I don't have a costumer munber, but its neccessary?? | 12:50.07 |
| kens | Well we need you to show us that you are a commercial or otherwise supported customer | 12:50.31 |
| | How else should we know you are not a free GPL user ? | 12:50.46 |
| nilsont | OK, How can I get that number? | 12:51.30 |
| Robin_Watts | nilsont: Either you're a free user (which is fine), or you're a supported/commercial user (in which case you work for a company that has a contract with Artifex). | 12:51.42 |
| | All we want to know is what company that is. | 12:51.56 |
| kens | Well, if you don;t know it, we cna look it up for you. But you will need to tell one of us which company you represent | 12:51.58 |
| Robin_Watts | Now, some companies don't like to advertise what software they use internally, so they prefer to keep their use of MuPDF secret. | 12:52.36 |
| nilsont | well I'm a free user | 12:52.45 |
| kens | OK that's fine | 12:53.01 |
| Robin_Watts | accordingly, we give such customers a company number that means nothing to anyone else except us. | 12:53.03 |
| | nilsont: OK, fair enough. | 12:53.10 |
| | So, there are 2 things to note. | 12:53.18 |
| | 1) You must therefore be using the GNU AGPL. As such any android app you produce will have to be released under the GNU AGPL too. | 12:53.56 |
| | i.e. lots of things you need to agree to, arguably the most important of which is source code must be given away to any user of your app that asks for it. | 12:54.44 |
| | Are you prepared to abide by the terms of the GNU AGPL? | 12:55.02 |
| nilsont | can I distribuite my product without problems? | 12:57.48 |
| Robin_Watts | nilsont: Are you prepared to give away the source code to your entire product ? | 12:58.20 |
| sebras | nilsont: you can distribute your product given that you give the full source code for your app to any user who asks for it. | 12:58.39 |
| | nilsont: assuming that you follow all parts of GNU AGPLv3. | 12:59.13 |
| nilsont | All my code or just everything related to mupdf? | 13:00.56 |
| Robin_Watts | All your code. | 13:01.58 |
| sebras | nilsont: if you don't want to do that you can choose to become a paying customer to Artifex, which includes support. | 13:06.22 |
| | nilsont: I don't know the details about prices, but I believe that if you are simply an indie developer doing an app on your own the cost might be too high. | 13:06.58 |
| | nilsont: if you are working for a company or on behalf of a customer, then the a commercial license for MuPDF along with support is likely suitable. | 13:09.27 |
| avih | tor8: interesting piece of info: the same mujs revision and general code (mpv), comparing windows native execution vs linux in virtualbox, both compiled with gcc 6.3, and the js code barely makes calls to c functions, at the linux vm it runs 50+% faster. any idea where it could come from? | 13:12.20 |
| | (the linux vm is hosted on the windows system i used for the comparison. i.e. the same system, except linux is also inside a vm) | 13:13.33 |
| nilsont | great! well I'll evaluate the proposal | 13:15.14 |
| tor8 | avih: not a clue. if you're using a lot of try/catch the performance of setjmp can be an important factor. | 13:17.10 |
| Robin_Watts | nilsont: Cool. mail sales@artifex.com and explain what you want to Scott, and he'll ask you some questions to produce a tailored license proposal. | 13:17.20 |
| avih | tor8: i don't know how much "a lot is" but the whole thing executes inside a single try clause without ever exiting it before completion | 13:18.15 |
| | (and it doesn't throw anything) | 13:18.53 |
| tor8 | "a lot" is if you're using it in an inner loop :) | 13:18.59 |
| | are both compiled for the same bit width? | 13:19.39 |
| avih | i'm not. my whole script is run inside a single try catch to catch uncaught user errors, but the code which i benchmarked is inside it and never leaves the try clause | 13:19.44 |
| tor8 | I generally expect 32-bit code to be faster for everything | 13:19.58 |
| avih | hmm.. good point. no. linux 64, windows 32 | 13:20.02 |
| | oh.. interesting. | 13:20.23 |
| tor8 | fatter pointers make for more cache misses | 13:20.34 |
| avih | i could build the windows for 64 or the linux for 32, but it's too much PITA with my current build setup :) | 13:20.50 |
| | but anyway, linux is 64 _and_ 50% faster, inside a vm, on the same machine | 13:21.18 |
| | one diff though, this linux uses musl libc. while the windows build uses glibc | 13:21.47 |
| tor8 | inside a vm should make little to no performance difference | 13:21.58 |
| | well, there you have it. glibc is a horrible bloated mess :) | 13:22.12 |
| avih | :) | 13:22.31 |
| tor8 | is this cygwin or mingw? | 13:22.41 |
| avih | mingw | 13:22.45 |
| tor8 | I thought mingw uses msvcrt where it can | 13:22.59 |
| avih | and everything is statically linked for windows, but .so (mujs included) on linux | 13:23.11 |
| | it does use msvcrt where it can, but at least the *printf functions are glibc/mingw ones for format compliance | 13:24.04 |
| | but anyway, the js code barely makes c calls. whatever it does, it's inside mujs for the vast majority of it | 13:24.50 |
| | (and it doesn't load files or do "system" calls. generally a loop with some arrays manipulation. i'm testing how quickly it can iterate setInterval and setTimeout(fn, 0). on windows i get ~50/ms intervals and ~40/ms timeouts. on linux i get ~90/ms intervals and ~60/ms timeouts) | 13:28.39 |
| tor8 | time to break out the profilers | 13:29.10 |
| avih | maybe after it gets merged :) | 13:29.24 |
| | (i'm now generally happy with all the code and finalizing the docs. my code is c: https://0x0.st/6rI.txt and js: https://0x0.st/6rl.txt ) | 13:30.14 |
| | could be some thread safety measures though. it probably uses them at least once per iteration (waiting for an event between timer callbacks) | 13:31.29 |
| | (setTimeout is slower since i have to also create the timer object, while with intervals it just re-inserts the existing timer back to the queue) | 13:32.26 |
| bmwiedemann2 | Hi. Can I contribute https://www.zq1.de/~bernhard/temp/mupdf-sort-input-files.patch without signing a CLA, by declaring the changes to be CC0 (aka public domain) ? | 14:36.19 |
| | which effectively allows anyone to relicense it for his purposes | 14:37.52 |
| Robin_Watts | bmwiedemann2: I think that's a trivial enough change, that we don't need a CLA for it. | 14:38.19 |
| bmwiedemann2 | good :-) | 14:38.30 |
| | been contributing dozens of such patches this week already | 14:38.45 |
| | Robin_Watts: should I file a bug or what is the process? | 14:39.29 |
| Robin_Watts | A bug is the best way, yes. | 14:40.59 |
| tor8 | Robin_Watts: bmwiedemann2: you missed one line MUTOOL_SRC += $(sort $(wildcard source/tools/pdf*.c)) | 14:41.17 |
| Robin_Watts | bmwiedemann2: The makefiles are tors, so I'll leave you with him :) | 14:41.55 |
| bmwiedemann2 | tor8: ah, good catch. I have been working on the 1.10a version which did not have this line. Will add it. | 14:42.47 |
| tor8 | bmwiedemann2: I've got a local fix here. | 14:43.48 |
| bmwiedemann2 | tor8: https://bugs.ghostscript.com/show_bug.cgi?id=697958 | 14:45.05 |
| | has my updated patch attached | 14:45.18 |
| tor8 | bmwiedemann2: thanks. | 14:46.19 |
| | Robin_Watts: on tor/master is a fix for the openjpeg stuff. we were treating the 3-color 2-alpha jp2 image in fts 1700 as a CMYK+alpha image | 16:18.53 |
| | I've improved the colorspace detection so we now accurately detect it as RGB and now it matches gs (and hopefully acrobat) | 16:19.18 |
| Robin_Watts | tor8: And that works without looking at SMaskInData ? | 16:19.34 |
| tor8 | Robin_Watts: it does. | 16:20.23 |
| Robin_Watts | That's not what your bmpcmp is for right? | 16:20.27 |
| tor8 | I think my bmpcmp has gotten confused | 16:20.41 |
| Robin_Watts | Does it fix 1700, 1701, 1704, 1705, 1804 ? | 16:20.57 |
| | or at least "does it render those all correctly now" ? | 16:21.10 |
| tor8 | I did not see a problem with 1804 | 16:21.25 |
| | but the 1700-series now matches gs | 16:21.35 |
| Robin_Watts | tor8: Ok, seems plausible then. Go for it. | 16:22.26 |
| tor8 | so if I want to do a fresh bmpcmp of my local worktree vs current origin/master, I should just need to "clusterpush" and "clusterpush bmpcmp"? | 16:22.36 |
| Robin_Watts | Yes. | 16:22.43 |
| tor8 | do I need to wait for one to finish before queueing up the second? | 16:22.57 |
| Robin_Watts | no. | 16:23.16 |
| | You might need to wait 60 seconds. | 16:23.23 |
| | cos the cluster only looks for incoming jobs every so often. | 16:23.44 |
| kens | I always wait untilthe clusterpush has started building, just to be sure | 16:23.52 |
| Robin_Watts | kens: until it shows on the dashboard, is enough. | 16:24.07 |
| tor8 | okay, thanks | 16:24.17 |
| kens | Fair enough, might save me a little time in future | 16:24.24 |
| tor8 | so, my next bmpcmp should have the right diffs then I hope | 16:24.41 |
| | Robin_Watts: updated bmpcmp is live | 16:34.44 |
| | does that look good to you? | 16:34.47 |
| Robin_Watts | It does. | 16:36.51 |
| | Thanks. | 16:36.53 |
| sebras | tor8 (for the logs): ok, so openjpeg reorders the color components to always be Gray, RGB, YCbCr, CMYK. | 18:13.28 |
| | tor8: any opacity channels are NOT reordered. | 18:14.07 |
| | tor8: also opacity channels come in one of two varieties: opacity or premultiplied opacity. | 18:14.38 |
| | tor8: which one it is can be detected by looking at the ->comps[i].alpha value. if it is 1 it is opacity, if it is 2 it is premultiplied if it is 0 then it is a normal color and if has some other value type type of the component is unknown. | 18:15.59 |
| | tor8: so technically you could have a 5 component image where the r,g,b components come first, then an opacity for b and then opacity for r/g (combined) | 18:29.11 |
| | the spec thankfully restricts the number of opacities associated with a component to a single one, so you can't have both premultiplied and normal opacity for a single component. | 18:30.33 |
| | also these definitions are at the jpx level, if you look inside the code stream then it may have a number of components that is different (hopefully it correlates with the jpx-level one) | 18:31.55 |
| | this format is... flexible. yeah, that's the word. | 18:32.09 |
| | oh, and of course you can also have component independent opacity. | 18:33.23 |
| kens | sebras I think the descrioption you were searching for is 'insanely comlicazted' | 18:47.46 |
| sebras | kens: oh, and I found that for jpx there is an additional variant that is different yet again. | 18:48.25 |
| kens | Other than baseline you mean ? | 18:48.41 |
| sebras | kens: all covered by the jpx baseline that pdf supposed requires. | 18:48.45 |
| | dly. | 18:48.50 |
| kens | I seem to recall that Adobe used a draft version of the spec for JPX and ended up with an incompatibility | 18:49.21 |
| sebras | kens: as I read it jpx encapsulates jp2 which encapsulates j2k codestreams | 18:49.23 |
| kens | Umm, there are restrictions in PDF, and this is all an awfully long time ago | 18:49.48 |
| sebras | kens: pdf says it it is jpx baseline except it allows for ciejab but not enumerated CMYK. | 18:52.18 |
| kens | Yes I remember that one | 18:52.34 |
| sebras | other than that I can't see that there are any really. | 18:54.04 |
| kens | I'd have to go back and reead the spec | 18:54.22 |
| sebras | it even mentions animation but doesn't state whether it should be observed. | 18:54.28 |
| | kens: no need. | 18:54.34 |
| kens | Well GS certainly isn't going to do animation :) | 18:54.44 |
| sebras | oh.. and it has chromakey transparency too (according to the pdf spec, haven't found that in the jpx spec yet) | 18:55.55 |
| | and it supports t.4, t.6, jbig, jpeg, jpeg-ls, jpeg2000 and jbig2 compression internally. | 18:57.33 |
| | just... wow. | 18:57.37 |
| kens | Yes, kitchen sink specification | 18:57.51 |
| | Everything everyone on the committee thought might be useful one day, possibly, perhaps, just incase, you never know.... | 18:58.12 |
| | Which results in a spec which is all but impossible to understand and well-nigh impossible to fully implement | 18:58.53 |
| sebras | kens: oh... and any box inside the nested levels of boxes may of course be gzipped. | 18:59.46 |
| | tor8 so in conclusion I'm not sure the way you implemented this is foolproof. | 19:04.10 |
| | tor8 and the only image with which I noticed there might be something strange is with issue414.jp2 from openjpeg. | 19:05.07 |
| | I believe that this is actually an RGB image, but we assume it is CMYK with alpha. | 19:05.21 |
| | by studying the .alpha value as mentioned above we might be able to get it right. | 19:05.42 |
| sebras | sleeps | 19:05.52 |
| | Forward 1 day (to 2017/05/30)>>> | |
Log of #mupdf at irc.freenode.net.