| | <<<Back 1 day (to 2013/11/05) | 2013/11/06 |
| Robin_Watts | unleashes the changes. | 00:20.44 |
| | Gigs-: You can call pdf_is_{array,dictionary,int} etc if you want. | 00:21.32 |
| | Or if you're pretty sure of the type, just convert to that type (pdf_to_name for example) and if it's not the right type, you'll get a sane default back. | 00:22.25 |
| ray_laptop | Robin_Watts: that J12 file is unfortunate :-( It's painting text with 4 component CMYK, so most of what we are getting is 4 planes. That has to make the text blurry when printed. | 00:41.19 |
| | Robin_Watts: what changes ? | 00:41.31 |
| | most of the pages on J12 process 4 planes for most bands :-( | 00:43.07 |
| | The text is being drawn with CMYK = 0.75 0.679688 0.669922 0.902344 YUCK!!! | 00:49.44 |
| Robin_Watts | oh, hi ray_laptop | 00:54.16 |
| | I've pushed the 3 commits you looked at on robin/master into golden. | 00:54.31 |
| | And if they don't have SSE3 or 4 available, I don't think we can do much better. | 00:55.11 |
| | Let me refresh 801.tar.gz with the latest version. | 00:55.25 |
| | Are J12 and J6 part of the JEITA files by any chance? | 00:56.29 |
| ray_laptop | Robin_Watts: yes. Did we ever get "official" copies via JT ? | 00:57.20 |
| | I thought henrys had requested them | 00:57.48 |
| | I do have other J12 versions that I will look at (from cust 532) | 00:58.11 |
| Robin_Watts | yeah, I've just pulled up the version from tests_private/pdf/JEITA/ | 00:58.41 |
| | So, it's probably worth doing timings for all the JEITA files we have, in the hopes that most of them will be improvements :) | 00:59.21 |
| | It must be the planar rendering that's hurting us on J12, cos all the device processing must be faster. | 01:00.08 |
| ray_laptop | Robin_Watts: having to do 'copy_mono' on all of the planes probably hurts, yes | 01:00.47 |
| | there's a lot of text in there | 01:01.00 |
| Robin_Watts | The top two functions in the profile are lcms2, accounting for 20% of the CPU time. | 01:09.34 |
| | In fact over half the time is spent in lcms. | 01:10.27 |
| | So, do you think that your color usage stuff won't help here ? | 01:11.14 |
| ray_laptop | Our J12 ps file has the same screwy color .750179 .679652 .670191 .901488 cmyk | 01:11.19 |
| Robin_Watts | So 60% of the time is spend in gx_image1_plane_data, of which almost all of that is in lcms. | 01:13.24 |
| | I wonder if perhaps we aren't caching a link properly or something. | 01:14.15 |
| | We should not be spending 50% of the CPU in gsicc_get_link, surely? | 01:14.53 |
| ray_laptop | Robin_Watts: not if the link cache is working correcly. It can spend a lot of time regenerating a link if it thinks it has to. | 01:17.38 |
| | Maybe mvrhel_laptop's patch to recognize ICC colorspaces is really needed | 01:17.56 |
| | he did that for cust 532, but I don't think it made it into the HEAD (yet) | 01:18.27 |
| Robin_Watts | Ah, well, that'd be a massive win for us, I think. | 01:18.47 |
| ray_laptop | I have to run get a car from the garage before they close. bbiab. | 01:18.52 |
| | I'll check that (simple) patch tonight | 01:19.06 |
| | and maybe mvrhel_laptop can go ahead and commit it | 01:19.29 |
| Robin_Watts | And the award for top function name of the day, goes to Mvrhel for "gsicc_mash_hash" :) | 01:22.50 |
| | Hmm. Because we don't have CMM_THREAD_SAFE set, each rendering thread gets its own icc cache - so we load the same profiles 4 times maybe? | 01:29.15 |
| | it looks like I'm seeing 4 loads per page, which doesn't seem overly excessive. | 01:29.33 |
| | We seem to take locks around lcms calls. | 01:31.12 |
| | which makes me wonder if we really can't share an icc cache any more? | 01:31.37 |
| | bedtime for me. | 01:31.42 |
| mvrhel_laptop | Robin_Watts: the patch for not re-loading icc spaces is in the head | 02:37.05 |
| | Robin_Watts: also, threads cannot share an ICC cache amongst them unless we put a lock around the lcms use of the links | 02:38.26 |
| | unfortunately lcms is not thread safe with respect to using the same link | 02:38.54 |
| | amongst two different threads | 02:39.04 |
| orlok | I have a PDF that ghostscript, specifically pdf2ps craps out on. | 04:36.32 |
| | Its a v1.6 PDF made with Acrobat Distiller/QuarkXPress for mac - | 04:37.23 |
| ray_laptop | quit | 05:00.48 |
| orlok | you first! | 05:01.26 |
| sebras | orlok: have you reported your pdf2ps problems as a bug and attached the pdf? | 09:03.47 |
| | (I know nothing about gs, but I know the devs prefer bug reports) | 09:04.08 |
| Amr_ | Hello, I am using MuPDF in my Android application, my question is Can I open more than one pdf file to be read as one file during viewing? I have for example 3 pdf files and I want to open the first one and while scrolling and after finishing the first one, the second one open in the same view | 09:17.31 |
| chrisl | Amr_: you're a bit early for (most of) the mupdf devs, another hour or so..... | 09:25.38 |
| Robin_Watts | Amr_: Not in the standard code, but it probably wouldn't be hard to do. | 10:32.53 |
| Amr_ | Ok Robin, is this possible from the java source code or I have to modify the C++ files? | 10:50.34 |
| Robin_Watts | You will need to alter stuff at least down to the JNI level. | 10:58.19 |
| | i.e. it cannot be achieved purely in java, currently. | 10:58.35 |
| | but we have no C++ files in MuPDF. Just C. | 10:58.55 |
| | Does the ICC link cache persist from page to page? | 11:00.48 |
| | When NumRenderingThreads is in use it's freshly created for each thread for each page. | 11:01.25 |
| | I wonder if we can make it so that the caches persist across pages. | 11:01.46 |
| chrisl | Aren't the rendering threads created and destroyed for each page? | 11:02.49 |
| Robin_Watts | chrisl: They are. | 11:03.02 |
| | but we create/destroy the same number of threads on each page. | 11:03.23 |
| chrisl | Yeh, I just thought the ICC link caches had to be tied to the threads for thread safety..... | 11:04.14 |
| Robin_Watts | They do, I'm not suggesting otherwise. | 11:04.29 |
| | Rather than 'create thread, create icc link for thread, run rendering operations, destroy icc link cache, destroy thread' | 11:05.01 |
| | I'm suggesting that we do... | 11:05.06 |
| | 'create thread, do we have an icc link cache we can reuse, if so reuse it, if not create one, run rendering operations, store icc link cache for reuse, destroy thread' | 11:05.55 |
| | So every icc link cache we make is still used strictly in one thread at a time. | 11:06.13 |
| | but for a 100 page file, with 4 threads, we'd get the link 4 times, rather than 400. | 11:06.40 |
| chrisl | Hmm, I wouldn't have thought that would work with the memory management - I thought each thread got it's own top level allocator. Long time since I was in that part of the code, though | 11:07.37 |
| Robin_Watts | chrisl: Ah, yes, every thread gets its own top level allocator. | 11:08.07 |
| | Maybe store those too ? | 11:08.18 |
| chrisl | Or have the "main" thread create the link caches, and ensure the color management code *always* uses the memory pointer in the cache object? | 11:08.57 |
| Robin_Watts | chrisl: That sounds smarter. | 11:09.22 |
| chrisl | It might need an additional, thread safe allocator for the main thread to use for the link caches | 11:09.48 |
| sebblonline | Hi, question for mupdf/artifex people. Im trying to get licencing information on mupdf and wrote a mail to sales@artifex last week but didnt get answer yet. Someone here who can "push" that internally at artifex? :) | 12:28.26 |
| Robin_Watts | sebblonline: Remail to sales@artifex.com and copy it to me (robin.watts@artifex.com) | 12:28.52 |
| | and I'll make sure it gets through. | 12:28.58 |
| sebblonline | thanks alot! | 12:31.21 |
| Robin_Watts | paulgardiner, tor7: So, the big potential mupdf customer has been back in touch (395). | 12:53.30 |
| | They have sent a list of questions. | 12:53.41 |
| tor7 | Robin_Watts: aha! | 12:54.26 |
| paulgardiner | Did I miss something on tech? | 12:54.31 |
| Robin_Watts | no, it's not gone to tech. Let me forward the mail. | 12:54.55 |
| | The timings are the interesting thing. | 12:56.35 |
| | Does anyone have cold startup times to hand? | 12:57.01 |
| tor7 | in reading order *based on tagging*, and the other accessibility feature is the only thing I don't feel warm and fuzzy about | 12:58.05 |
| Robin_Watts | tor7: I would answer "yes" to that, and then, in the supporting email I was planning to write say something like: We currently completely ignore any tagging for text extraction, mostly because it is never used in files in the wild. | 12:59.09 |
| tor7 | Robin_Watts: you'd need to specify version and hardware for cold startup times | 12:59.15 |
| Robin_Watts | We therefore extract reading order by heuristics in all cases. | 12:59.55 |
| tor7 | but I'm assuming we're talking the android java app here | 13:00.07 |
| Robin_Watts | tor7: I would assume the same. | 13:00.22 |
| tor7 | Robin_Watts: yeah, fair enough. | 13:00.34 |
| | want me to set the svg aside for a bit and look at the jni and other foreign language bindings now? | 13:00.43 |
| Robin_Watts | I'm tempted to say "On my DesireHD, small files open within 100ms or so. Huge, complex files (such as the 1300 page pdf reference manual) take around 2 seconds to open. | 13:01.20 |
| tor7 | Robin_Watts: how about rigging mudraw to do a cold open of the file and timing that on your raspberry pi? | 13:01.59 |
| | then we can split the library and the java timings | 13:02.15 |
| Robin_Watts | tor7: I'd prefer to use a beagleboard. | 13:02.20 |
| tor7 | (or any comparable device) | 13:02.31 |
| Robin_Watts | But frankly, I suspect that those timings are more than good enough for them. | 13:02.45 |
| | I bet they just don't want to click on a file and have to wait 30 seconds while it is loaded. | 13:03.12 |
| tor7 | Robin_Watts: it would be cool to say that the library is awesome fast, all the slowness is in the java framework wrappers, so out of our hands | 13:03.14 |
| Robin_Watts | tor7: I was planning to explain the division between the lib and the app in the mail. | 13:03.49 |
| paulgardiner | So these are questions about the Android app rather than about the C library? | 13:04.01 |
| Robin_Watts | paulgardiner: I suspect it's a bit of both. | 13:04.26 |
| tor7 | on an i5 we can open and render tiger.pdf at 72 dpi in 70ms | 13:04.28 |
| | and page 1 of pdfref17 in 55ms | 13:04.47 |
| Robin_Watts | I'll draft a response after lunch and send it to both of you for comments. | 13:05.19 |
| sebras | tor7: I wish I could tiger, but my wacom tries to tell me I can't. :/ | 13:05.40 |
| paulgardiner | So do we lose if a competitor just lies? | 13:05.55 |
| Robin_Watts | Anyone have a clue what they mean by 'Paging' ? | 13:05.56 |
| | paulgardiner: I could make the point in the email that relying purely on numbers supplied by vendors would be bad, and invite them to try for themselves. | 13:06.51 |
| tor7 | nope. I would guess they mean drawing pages as we do, not continuous scroll? | 13:06.58 |
| Robin_Watts | tor7: That's what I would think too. | 13:07.17 |
| | I can mention our 'smart advance' stuff for easy reading. | 13:07.29 |
| | We don't do a white-on-black view in the android app yet. | 13:07.44 |
| tor7 | Robin_Watts: I definitely think you should mention that. and that we have -to-html for reflow and accessibility | 13:07.54 |
| Robin_Watts | i.e. a color inverted bitmap. | 13:08.04 |
| paulgardiner | Robin_Watts: yeah, I'm a little worried as to who might be evaluating this. If it were people that were looking at the code itself as part of the judgement, I'd imagine they'd want to do their own timings | 13:08.09 |
| tor7 | Robin_Watts: we do on the mupdf windows/x11 viewer (button 'i') | 13:08.24 |
| Robin_Watts | I can say that we do it on other platforms. | 13:08.27 |
| | tor7: yeah. | 13:08.31 |
| | Any notable extensions we should list ? | 13:08.46 |
| tor7 | Robin_Watts: something interesting there would be to invert colors based on content though | 13:08.49 |
| | so line art is inverted, but images are not | 13:09.05 |
| paulgardiner | tor7: would that invert any actual black and white within images though? | 13:09.38 |
| tor7 | paulgardiner: I would expect not. we'd have to take care with type3 bitmap fonts though. | 13:10.02 |
| | shouldn't be too hard to hook up a custom inverted-rgb colorspace with custom functions so the pixmap conversions all go the regular rgb case, and invert for text and line art color conversions | 13:10.50 |
| Robin_Watts | I think with all such things, we just play the cards we have as best we can. | 13:10.56 |
| | If people are looking for an excuse to go another way, there is nothing we can do to stop them. | 13:11.13 |
| tor7 | Robin_Watts: shouldn't be more than an afternoon of hacking to do, but I haven't seen a compelling reason yet | 13:11.27 |
| paulgardiner | There are things there we don't have but could do very quickly: white on black and png out | 13:11.33 |
| tor7 | with my astigmatism, I really can't stand white text on a black background, it tires my eyes way too much | 13:11.46 |
| | paulgardiner: both which are supported in the library, and (I believe) by robin's new jni bindings | 13:12.29 |
| paulgardiner | I'm not sure what they mean by rotation. | 13:12.31 |
| tor7 | paulgardiner: yes, we autorotate to follow screen rotation? ;) | 13:12.50 |
| | or maybe they want manual rotation, if the page layout is borked | 13:12.58 |
| paulgardiner | Gah! They don't ask about form filling or annotations, after previously saying that was important. | 13:13.39 |
| Robin_Watts | Extensions: "Transparency, Forms" | 13:14.40 |
| paulgardiner | ah yes | 13:15.59 |
| tor7 | javascript? | 13:16.29 |
| Robin_Watts | ah, yes, thanks. | 13:16.48 |
| | sebblonline: Scott tells me he has now replied? | 13:17.54 |
| sebblonline | yes, got response, thanks. | 13:18.35 |
| Robin_Watts | sebblonline: Huge list of questions? :) | 13:18.54 |
| sebblonline | oh yes :) | 13:19.46 |
| paulgardiner | Robin_Watts: on timings, we could draw attention to the fact that we have a huge number of reviews on the Play store that praise MuPDF for its speed, perhaps. | 13:20.27 |
| kens | Robin_Watts : ping | 13:45.02 |
| Robin_Watts | pong | 13:45.13 |
| kens | Your commits last night have made 2 ps2write files produce diffs, any idea why ? Also a number of psdcmyk files now seg afult | 13:45.44 |
| Robin_Watts | sebblonline: Just answer them as best you can. Some of them won't be relevant. The more information you give Scott, the better he can understand what you need and hence tailor the quote. | 13:46.11 |
| | kens: No idea about the ps2write stuff. | 13:46.22 |
| kens | Did you look at the diffs ? | 13:46.32 |
| Robin_Watts | I will look into the SEGVs after I get this stuff out of the way. | 13:46.36 |
| | I did. | 13:46.38 |
| kens | OK thenI guess its not a problem | 13:46.45 |
| Robin_Watts | The calculation of raster has changed slightly. | 13:47.01 |
| | hence I suspect the banding may have changed a bit. | 13:47.10 |
| | and we get slight differences over band boundaries. | 13:47.20 |
| kens | pdfwrite/ps2write don;'t do banding | 13:47.22 |
| Robin_Watts | no, but the resulting files are rendered then diffed, right? | 13:47.38 |
| kens | Hmm, yes true | 13:47.49 |
| | I'd be slightly owrried that the same PostScript rendered 2 different ways, but I guess if you looked at it, its OK | 13:48.21 |
| Robin_Watts | All the diffs I could see (and I went through several times) were plausibly down to banding differences. | 13:48.33 |
| kens | Oh, and the files were different rendered at 72 dpoi as well, so I wouldn't have thought banding could affect that | 13:48.54 |
| | But if you're happy with it, I won't worry | 13:49.32 |
| kens | fetches fresh coffee | 13:50.20 |
| punknroll | Hi what can I do if gs gives me segemntation fault with pdfwriter and compatibiltiy 1.4? | 13:55.16 |
| kens | report a bug | 13:55.26 |
| | Or,of courfse, debug it yourself, it is opensource | 13:56.07 |
| punknroll | the debug output is Program received signal SIGSEGV, Segmentation fault. | 14:00.38 |
| | cos_stream_add (pcs=0x8e84118, size=512) at ./devices/vector/gdevpdfo.c:1565 | 14:00.38 |
| | 1565./devices/vector/gdevpdfo.c: File not found ...any hints what could cause that? | 14:00.38 |
| kens | Yes, there is a bug. | 14:00.51 |
| Robin_Watts | punknroll: What version of ghostscript ? | 14:02.08 |
| kens | I can't tell you anything without seeing the source file, and running it myself. THe way to get me the file is to open a bug report and attach theoffending file to it. You should also make sure you give the command line, and tell us the version of Ghostcript being used (also where you got it fomr if on Linux) | 14:02.12 |
| punknroll | GPL Ghostscript 9.10 (2013-08-30) | 14:02.30 |
| | i guess the problem is transparent png...the pdf has multiple pages and too big for the bug report. I have to try to reproduce the bug again with a smaller file | 14:04.49 |
| kens | You can't have a .PNG as an input, either in PostScript or PDF | 14:05.20 |
| | And if you are using pdfwrite, you can't be producing PNG | 14:05.35 |
| | So 'transparent png' isn't a possible problem | 14:06.02 |
| punknroll | no i have generated a multipage pdf file which has transparent png files in it | 14:06.04 |
| kens | No you haven't | 14:06.11 |
| | You may have started with transparent PNG fikles, but the PDF file will *NOT* contain PNG | 14:06.25 |
| | PNG is not a possible image format in PDF | 14:06.35 |
| punknroll | i use a php library for pdf creation...to i guess the png is converted there | 14:07.12 |
| kens | Possibly | 14:07.19 |
| | But the poitn remains that it can't be transpoarent PNG because you can't have PNG as an input. | 14:07.39 |
| | There really isn't any point in randomly guessing. If you can make a sensibly sized file go wrong, then make a bug report | 14:08.03 |
| punknroll | ok, i will try that the segfault goes away if i change the compatibiltiy level to 1.3 but then the quality is too bad | 14:09.08 |
| kens | is tempted to point out again that the design goal of pdfwrite is not a general purpose PDF mangling tool | 14:09.52 |
| punknroll | i didn't find a better tool to reduce the filesize of a pdf yet | 14:12.50 |
| kens | Possibly so, but that's not what it was intended to do | 14:13.16 |
| | OPh and reducing to 1.3 will flatten the transparency because PDF 1.3 doesn't have a transparent imaging model | 14:14.19 |
| punknroll | i think this is maybe the problem...but only with some specific png files which get convert by tcpdf | 14:21.13 |
| | no segmentation fault with GPL Ghostscript 9.05... can you point me to some debugging tips? | 14:27.15 |
| kens | Not really | 14:27.36 |
| | You could use git bisect to see where teh problem arose | 14:29.25 |
| | Robin_Watts : ping | 14:34.18 |
| punknroll | when i add -dNOTRANSPARENCY the segmentation fault is gone, but the images look ugly | 14:36.57 |
| kens | Becvause | 14:37.05 |
| | The transparency is ignored, obviously | 14:37.19 |
| | There really isn't any point in making random posts, you need to come up with a bug report | 14:37.35 |
| | And your file must be truly massive it is too big for our file attachments,so I wouldn't thank you for it anyway | 14:37.57 |
| punknroll | ok, sorry for that...i will find a file | 14:45.03 |
| Robin_Watts | kens:pong | 14:47.17 |
| kens | Could you have a quick look at bug #694756 please ? | 14:47.37 |
| | You might be able to answer it quickly | 14:47.45 |
| chrisl | Hmmm, "your code works, my code doesn't - you have a bug"! | 14:49.43 |
| kens | Well it is when using the gs_api stuff | 14:49.56 |
| Robin_Watts | kens: chrisl seems to have answered it well. | 14:50.05 |
| chrisl | It seems crazy that he would find it works with the normal executable, and not take 5 minutes to look at what it's doing..... | 14:51.05 |
| punknroll | so can i upload a 50mb file to the bugtracker? I will try:-) | 15:01.36 |
| chrisl | punknroll: compression often helps..... | 15:02.52 |
| kens | I will likely ignore it I have to tell you, as it wil be too painful to debug. Eventually I mayget round to it, smaller files will get looked at faster | 15:03.46 |
| punknroll | the segfault is on page 2 | 15:04.50 |
| kens | then amke a 2 page file | 15:04.57 |
| punknroll | so maybe i can cut away some other pages | 15:05.11 |
| kens | everything after page 2 would semm the most obvious | 15:09.26 |
| sebras | tor7: a few low hanging patches over at sebras/master | 15:30.38 |
| Gigs- | Robin_Watts: ok thanks. I've got the pdf and can enumerate the objects and detect colorspace definition arrays so I'm getting there. I don't know if you wrote all this code, but I have to say this is some of the cleanest C I've ever worked with. It was written by someone that knows when to ignore the "rules" and when not to, which is very nice. | 15:30.39 |
| Robin_Watts | Gigs-: The code was originally written by tor7. Myself, paulgardiner and sebras have all contributed to it, but the overall style was set by Tor. | 15:31.36 |
| punknroll | kens: i finally entered the bug report: http://bugs.ghostscript.com/show_bug.cgi?id=694758 | 15:31.43 |
| Gigs- | Ah, well I appreciate his style :) | 15:31.47 |
| kens | OK punknroll | 15:31.51 |
| Robin_Watts | Gigs-: We've felt free to rejig the code at various points in an attempt to keep it clean, so we are better off than some projects that 'freeze early' and then bend over backwards to avoid breaking anything. | 15:32.41 |
| Gigs- | yeah, and just little things too, like not overly committing to some paradigm.. for example you could have thrown exceptions all over the place for the invalid type conversions | 15:33.28 |
| | but I like this way a lot better | 15:33.38 |
| Robin_Watts | Ah, well, exceptions were added after most of the rest of the code :) | 15:33.49 |
| tor7 | yeah, I blame Robin_Watts for the exceptions! | 15:34.03 |
| Gigs- | probably for the best, they should be used sparingly IMO :) | 15:34.11 |
| Robin_Watts | but yes, exception should be used for exceptions, not for usual things. | 15:34.20 |
| Robin_Watts | is happy to be blamed for them :) | 15:34.39 |
| kens | punknroll : I really, really wouldn't use PDFSETTINGS at all. | 15:35.06 |
| tor7 | the exceptions are nicer, but it's easy to trip up on some gotchas :( | 15:35.09 |
| kens | Set the individual controls you want, don't use /ebook | 15:35.19 |
| | I'm not surprised the qulaity sucks that way | 15:35.32 |
| Gigs- | in the end every exception is basically a non-local goto, so it can get messy | 15:36.47 |
| punknroll | no quality is quite fine...basically i want to reduze the filesize and compress the images...but I don't know if this will solve my problem | 15:37.19 |
| tor7 | Gigs-: the usual caveats with setjmp (the scoping of when a local variable is valid) appy, but also care not to return or break out of the try/catch scope | 15:38.03 |
| Gigs- | yeah try/catch constructs are a little more structured if you use them right | 15:39.05 |
| kens | by using PDFSETTINGS you are getting a lot of controls set that you probably don't want. I would suggest that you instead set the compression controls individually | 15:39.07 |
| punknroll | ok, i will leave the fonts embedded and basically just compress the images | 15:40.26 |
| Gigs- | I've got a degenerate case type file to submit a bug on. I'd consider it a medium-low priority because even adobe's engine hates the file, but it does slog through it whereas gs gives a rangecheck. | 15:40.57 |
| | Probably implementation limit differences | 15:41.13 |
| kens | punknroll : by using /ebook you are (IIRC) converting everything to sRGB, which is not a good idea | 15:41.29 |
| Gigs- | on the other hand, mupdf flies through it :) | 15:41.50 |
| | but without overprinting | 15:42.01 |
| punknroll | thanx kens: i try to work on those options maybe I find the option that create the segsigv | 15:42.50 |
| henrys | paulgardiner: LO is sounding a lot less appealing to me. | 15:42.58 |
| Robin_Watts | henrys: Hell, I loved the idea of 120Megs of lib. :) | 15:43.24 |
| kens | punknroll : most likely its caused by the colour conversion. Converting the images to sRGBwill require rendering (flattening) the transparency | 15:43.59 |
| paulgardiner | henrys: yeah me too. It was looking like a really quite clean solution, but is definitely less so now. | 15:44.20 |
| | henrys: if it were modular and we really could get just the bits we need then I think I'd still be favourable towards it | 15:45.09 |
| henrys | paulgardiner: I think we should have a schedule estimate for doing it ourselves, of course a very rough estimate, but a bit more than finger in the wind. | 15:45.20 |
| Robin_Watts | paulgardiner, henryrs, tor7: I've sent my proposed response to customer 395 to tech. Any comments gratefully received. | 15:46.01 |
| henrys | paulgardiner: we could also reopen the idea of requiring a jvm and using poi | 15:46.30 |
| Gigs- | need attachment set private please http://bugs.ghostscript.com/show_bug.cgi?id=694759 | 15:46.35 |
| | also correct the component because I doubt I got it right | 15:46.47 |
| paulgardiner | Michael came up with 3 man months, but I've afraid I didn't carefully enough write down the list of tasks he listed. Also I was unsure whether he meant an LO dev or someone new to LO doing it. | 15:46.47 |
| Gigs- | mythical or real | 15:47.00 |
| punknroll | kens: this sounds promising...i don't want no color conversion there | 15:47.20 |
| henrys | Robin_Watts: yes I'll read it. | 15:47.40 |
| Robin_Watts | Gigs-: Done. | 15:47.42 |
| tor7 | Robin_Watts: C with JNI bindings and some Java wrapper classes? | 15:47.45 |
| Gigs- | Robin_Watts: thanks | 15:47.46 |
| henrys | paulgardiner: docx just viewing? | 15:48.02 |
| Robin_Watts | tor7: In the spreadsheet? | 15:48.09 |
| tor7 | yes | 15:48.14 |
| | Robin_Watts: I also think we've improved the large document startup time with the lazy page-tree loading. or did you retry opening a big document with the latest code? | 15:48.39 |
| paulgardiner | henrys: I assumed all office document types. | 15:48.41 |
| | but yes just viewing. | 15:48.53 |
| Robin_Watts | tor7: I did not. | 15:49.09 |
| tor7 | Robin_Watts: maybe they mean the adobe pdf extension level thing for features beyond the ISO spec? | 15:49.28 |
| henrys | paulgardiner: are there open standards now for all the office file types? | 15:49.32 |
| Gigs- | office supports an open xml format but it's not the defaults | 15:50.22 |
| | I'm sorry, it's not the default until office 2007 | 15:51.48 |
| paulgardiner | henrys: I don't think so. So I guess LO support for other than docx may not be perfect. I just imagined with all the years it has been about it probably does a fairly good job with the older formats. | 15:51.51 |
| chrisl | "Office Open XML" is an ISO standard | 15:51.57 |
| paulgardiner | Pure assumption though | 15:51.59 |
| Robin_Watts | To throw something else into the mix... | 15:52.38 |
| | Picsel phoenixed into "Smart Office Technologies" | 15:53.01 |
| | and they too have gone bust. | 15:53.12 |
| | The administrator is shutting them down, but before they go, he has been selling 'perpetual licenses' to the code. | 15:53.47 |
| | so there is potential for us picking up a license for their office formats code. | 15:54.20 |
| paulgardiner | Robin_Watts: do you think our previous worries about ownership no longer apply. | 15:54.51 |
| Robin_Watts | clearly, whether we would be interested would depend on a) the price, and b) any restrictions that would be placed upon us. | 15:54.58 |
| | paulgardiner: Well, the administrator has apparently employed people to ensure that the ownership of the code is clear. | 15:55.30 |
| paulgardiner | Sounds promising then. | 15:55.48 |
| Robin_Watts | I fired an email off to them at the weekend, just inquiring about the possibility, and that was one of the questions I asked. | 15:56.06 |
| | I've had no reply, so I've got a phone call into them now. | 15:56.23 |
| tor7 | anything would be better than a 120M C++ library | 15:56.35 |
| Robin_Watts | I haven't committed us to anything (obviously!), but I thought it was worth finding out what the options were. | 15:56.58 |
| tor7 | Robin_Watts: I take it an open source license buyout is out of the question? | 15:57.21 |
| paulgardiner | tor7: pure C and very small (compared with LO) | 15:57.24 |
| Robin_Watts | tor7: One of the questions I asked was "could we open source it" | 15:57.42 |
| | They have supposedly raised 500K or so by selling perpetual licenses (figure on 2 or 3 licensees). | 15:58.19 |
| tor7 | Robin_Watts: I hope if the answer to that is yes, we'd be willing to pay more for it | 15:58.24 |
| Robin_Watts | but that would be for all the technology; rendering engine, apps, DAs, the whole shebang. | 15:58.55 |
| | We're just after (a subset of) the DAs. | 15:59.09 |
| tor7 | Robin_Watts: or do what blender did in the late 90's and crowd fund the buyout... | 15:59.12 |
| Robin_Watts | I suspect that given they have licensed it already they would be really averse to allowing someone to open source the whole thing. | 15:59.39 |
| punknroll | kens: -dDownsampleColorImages=true causes the segsigv | 15:59.52 |
| Robin_Watts | but it's quite possible that they might be open to opensourcing the DAs. | 16:00.00 |
| tor7 | Robin_Watts: re the proposed response, all looks good with the (minor) detail that we only use droidsansfallback for CJK fonts, and have a few options (droidsans regular, and the base14 fonts) for non-CJK | 16:00.05 |
| kens | OK can you add that to the report please | 16:00.06 |
| tor7 | which might not be worth mentioning, given the target audience | 16:00.20 |
| Robin_Watts | tor7: Want to mail me back a revised paragraph? | 16:00.29 |
| tor7 | or might, just to say, we have it covered | 16:00.32 |
| henrys | Robin_Watts: perhaps trivial but opening your xls in google docs the data is wrapped strangely almost unreadable - his original is okay. What did you use to edit the spreadsheet? | 16:02.01 |
| Robin_Watts | Libre Office :) | 16:02.18 |
| punknroll | kens: i did | 16:02.41 |
| henrys | Robin_Watts: of course | 16:02.52 |
| | Robin_Watts: if I could stop laughing I'd check it in MS office. | 16:04.05 |
| paulgardiner | That's not funny. All that time I spent discovering reasons not to use LO, and Robin has managed it as an aside to another task. :-) | 16:07.22 |
| henrys | Robin_Watts: I do think it should be edited in office before sending. LO has made a mess opt it. | 16:07.24 |
| | s/opt/of | 16:07.29 |
| Robin_Watts | henrys: I don't (won't) have office. | 16:07.52 |
| henrys | I'll do it if I can make out what LO has done to it. | 16:08.35 |
| Robin_Watts | but I'll happily send you the final version and have you send me it back. thanks. | 16:08.41 |
| henrys | Is this a political stance you're aware office is an expected expense? | 16:09.30 |
| | Robin_Watts: yes send me your final and I'll reformat and send it back to you, we should also make sure it looks good in Google Docs, I'll do that also. | 16:10.50 |
| chrisl | Or we could send them a PDF - that's what PDF is for, after all! | 16:11.21 |
| Robin_Watts | henrys: I object to paying that much for something so bloated where I'll only ever use a fraction of it. And I have a problem with their proprietry formats etc. | 16:11.34 |
| | So, yes, it's a vaguely political stance. | 16:11.45 |
| punknroll | kens: thanx for help!!!! | 16:12.45 |
| henrys | chrisl: he said fill out the questionnaire … | 16:12.59 |
| sebras | tor7: hey! for the first time I triggered an automatic git gc! | 16:13.29 |
| | tor7: did you see my call for review? | 16:13.49 |
| tor7 | Robin_Watts: I'm rewriting the paragraph on font substitution. | 16:14.13 |
| | sebras: yes. will look soon. | 16:14.39 |
| Robin_Watts | tor7: for Engine wrapper I now have: C with JNI interface to Java app. Generic JNI interface under development. | 16:15.24 |
| henrys | fairly impressed with the new cdt + eclipse release. It used to suck but the new one has really come along looking more like VS. | 16:15.51 |
| Robin_Watts | tor7: I've just retested with the latest code on android. Still 2 seconds to load. | 16:15.59 |
| chrisl | henrys: I realise we're aiming for the sale, so trying to fit in with them, but I can't help but feel that if you use application specific file formats, all bets are off - especially dealing with a company so very *not* tied to M$..... | 16:16.47 |
| sebras | tor7: ok, you can postpone it until tomorrow if you're busy. just wanted an ack. :) | 16:16.59 |
| tor7 | Robin_Watts: rats. I had hoped it would improve :( | 16:17.39 |
| Robin_Watts | paulgardiner points out that I should mention the 'copy to clipboard' stuff in text extraction. | 16:19.53 |
| henrys | I'd like to say something about signatures too, even though he didn't ask. | 16:20.28 |
| Robin_Watts | Ah! | 16:20.39 |
| | I should put that in the extensions section. | 16:20.47 |
| | along with google print? | 16:21.02 |
| henrys | sounds good to me. | 16:24.15 |
| Robin_Watts | tor7: I think we just omit your last paragraph. | 16:27.16 |
| | That's covered by the last sentence of para 2 I feel. | 16:27.32 |
| tor7 | Robin_Watts: fair enough | 16:27.40 |
| Robin_Watts | Ok, revised version sent to tech. | 16:36.43 |
| | ray_laptop: Morning. | 16:50.41 |
| ray_laptop | morning, Robin_Watts | 16:51.12 |
| | mvrhel_laptop: good morning. Are you here ? | 16:51.31 |
| mvrhel_laptop | good morning. I am | 16:51.41 |
| ray_laptop | mvrhel_laptop: I was thinking about the screwy text color of J12 (the file I have from cust 532 is the same way). How difficult would it be to have the ICC profile for text map colors very close to black to just the K plane | 16:53.06 |
| | Not only would it let us skip most of the planes, but it would make the text look better when printed | 16:53.52 |
| Robin_Watts | ray_laptop, mvrhel_laptop: I had a random idea this morning that might help with performance. | 16:54.09 |
| | Currently we create a new icc cache for every rendering thread. | 16:54.45 |
| | so if I have 4 threads and 100 pages, we create 400 icc caches. | 16:55.01 |
| ray_laptop | On either laser or inkjet, doing text with lots of C+M+Y+K is going to look a lot worse, plus cust 801 prints CMY at 300 dpi :-( | 16:55.09 |
| Robin_Watts | Wouldn't it be nice if there was some way to carry the icc caches forward between pages? | 16:55.34 |
| ray_laptop | Robin_Watts: I saw that discussion (with chrisl ?) in the logs.. | 16:55.37 |
| Robin_Watts | ray_laptop: Yeah. | 16:55.48 |
| mvrhel_laptop | ray_laptop: We could certainly do a destination text profile that mapped near neutrals to K only | 16:55.54 |
| ray_laptop | mvrhel_laptop: is that something you can whip out ? | 16:56.18 |
| | or is a pain | 16:56.24 |
| kens | As the actress said to the bishop..... | 16:56.34 |
| chrisl | LOL | 16:57.03 |
| Robin_Watts | I was wondering about having a set of icc caches held in the lib context. | 16:57.04 |
| mvrhel_laptop | ray_laptop: To do it in a clean way is a pain. To throw together a hack that maps all the text colors to the black plane is easy | 16:57.10 |
| ray_laptop | kens: I don't know that one | 16:57.12 |
| paulgardiner | henrys: certainly reliance on a JVM has become more attractive in light of the alternative horrors of LO. The main issue disinclining me towards it was the difficulties of a JVM if someone wished to use MuPDF/office in an embedded environment, but perhaps that's not a likely use. What do you see as the main target for MuPDF with office support? | 16:57.22 |
| kens | ray_laptop : its a common 'double entendre' in the UK | 16:57.28 |
| mvrhel_laptop | Robin_Watts: carrying the caches to the next page is a great idea | 16:57.55 |
| Robin_Watts | kens: or rather it's an indicator that the previous thing said should be read as a double entendre. | 16:58.01 |
| kens | Yeah, that's what I mean | 16:58.13 |
| Robin_Watts | mvrhel_laptop: The problem is that currently the icc_cache's all use the threads memory pointers. | 16:58.24 |
| ray_laptop | Robin_Watts: I'm not sure how to do that in the rendering, however | 16:58.25 |
| paulgardiner | Robin_Watts: another little commit on paul/master | 16:58.49 |
| Robin_Watts | and the threads memory pointers are torn down at the end of each thread. | 16:58.56 |
| mvrhel_laptop | yes. this sounds like a nice task for you and ray_laptop | 16:59.14 |
| henrys | paulgardiner: the current customer wants their customer to direct print to their printer line. | 16:59.25 |
| ray_laptop | Robin_Watts: not at the end of each thread, but after the page is done | 16:59.28 |
| Robin_Watts | So either we'd need to carry the threads memory pointers across too, or we'd need to move the icc stuff to use different memory pointers. | 16:59.59 |
| | ray_laptop: I was thinking that in the lib_ctx we'd have an array of 'n' icc_cache pointers. | 17:00.17 |
| | We'd start with them all being NULL. | 17:00.26 |
| henrys | paulgardiner: the problem is ios | 17:00.41 |
| Robin_Watts | Then when we create the threads, we say: is there an icc_cache pointer for thread [0] stored in the lib_ctx? If so use that, if not, create one. | 17:01.21 |
| ray_laptop | Robin_Watts: Each thread has its own icc_cache unless we have CMM_THREAD_SAFE (which lcms* isn't) | 17:01.26 |
| henrys | paulgardiner: I just wish apple would cave on the jvm stuff. | 17:01.39 |
| Robin_Watts | And when we come to tear down, we move the icc_cache pointers from the thread to the lib_ctx. | 17:01.40 |
| | ray_laptop: Indeed. | 17:01.43 |
| chrisl | Robin_Watts: how would each thread identify the index it should use? | 17:02.26 |
| ray_laptop | Robin_Watts: so the array would be of a fixed size ? or set to num_rendering_threads_requested | 17:02.28 |
| Robin_Watts | ray_laptop: That can't easily be changed, AIUI. but having thread[0] for page 0 use the same cache as thread[0] for page 1 does would be safe, right? | 17:02.40 |
| | ray_laptop: It would be set to num_rendering_threads_requested. | 17:02.55 |
| ray_laptop | Robin_Watts: but that is a run time parameter that can change between jobs (although it usually doesn't) | 17:03.31 |
| Robin_Watts | chrisl: The icc_cache creation is done in the main thread in a: for(i =0; i < num_rendering_threads_requested; i++) { ...} | 17:03.31 |
| ray_laptop | but I guess we can free and re-allocate the array if it NRT changes | 17:04.00 |
| Robin_Watts | ray_laptop: Right, so the lib_ctx holds a variable length array. That's not hard, right? If the length changes, we throw away the contents. | 17:04.03 |
| henrys | Robin_Watts: did you intend a '?' after the timing's or is that some LO addition? | 17:04.09 |
| Robin_Watts | I intended a ? | 17:04.17 |
| chrisl | Robin_Watts: yes, then each rendering thread picks up it's icc_cache from there, but how do you guarantee two rendering threads won't try to use the same cache pointer? | 17:04.41 |
| Robin_Watts | I wanted to point out that asking for timings without asking for a reference machine was odd. | 17:04.44 |
| ray_laptop | chrisl: each thread has a 'slot' in the render_threads array | 17:05.16 |
| henrys | Robin_Watts: I think the pointer to the attached document is enough but I'll leave them in if you like. | 17:05.20 |
| Robin_Watts | chrisl: Rendering threads are always numbered 0...n, right? So thread i will always use the cache stored in position i in the render_threads array. | 17:05.24 |
| paulgardiner | henrys: I didn't realise we already had a customer for this. | 17:05.50 |
| Robin_Watts | henrys: My point is that I didn't want people to look at the spreadsheet and take the figures as gospel. | 17:05.58 |
| henrys | Robin_Watts: right | 17:06.08 |
| ray_laptop | chrisl: and each thread has a gp_thread_id | 17:06.37 |
| chrisl | ray_laptop: okay, doesn't that just increment with each thread creation? | 17:07.14 |
| ray_laptop | is going to wait until the cust 395 discussion wraps up since Robin_Watts is involved in both. | 17:07.35 |
| Robin_Watts | chrisl: I'm failing to see the issue here. The threads on page 0 are 0,1,2,3. On page 1 they are 0,1,2,3 etc ? | 17:07.42 |
| chrisl | Robin_Watts: most of the gs_*_id variables just increment with each use, they don't decrement when the objects are destroyed | 17:08.31 |
| henrys | paulgardiner: nothing definite but we did speak to a large printer company and that is one of the features they pointed that would make mupdf more attractive to them. | 17:09.08 |
| ray_laptop | chrisl: well, it isn't at all hard for a thread to get an id that corresponds to its slot in thre render_threads array | 17:09.23 |
| chrisl | ray_laptop: no, indeed, it's a little extra complication | 17:09.48 |
| | Robin_Watts: my main reason for asking was if you were thinking of passing an index to the thread, you'd be as well to pass the cache pointer directly...... but if there's an existing index you can reuse, then no need | 17:10.42 |
| henrys | Robin_Watts: the only problem I see in LO is it imposes a very small column width on the document. Once I manually stretch out those columns it looks right. I'll send out a new spreadsheet shortly. | 17:11.26 |
| chrisl | ray_laptop: it is, however, something we'll need to be careful of if we start using threads elsewhere...... | 17:11.35 |
| ray_laptop | Robin_Watts: but I thought you were seeing that each band was reloading the links (I saw mention of 400 times) | 17:12.09 |
| | chrisl: the information passed to the rendering thread is *NOT* general to threads | 17:12.50 |
| Robin_Watts | ray_laptop: I said that the link was loaded into every cache. And we have 4 caches per page, for 100 pages, hence 400. | 17:13.22 |
| punknroll | lens: without dDownsampleColorImages=true my ebook version and screen version have the same filesize. | 17:13.24 |
| chrisl | ray_laptop: gs_thread_ids wouldn't be used for other threads? | 17:13.38 |
| Robin_Watts | I don't see how gs_thread_ids come into this. | 17:13.51 |
| paulgardiner | henrys: so would the JVM requirement be likely not a barrier for the large printer company? I guess JVMs are more commonly used these days. | 17:13.53 |
| Robin_Watts | The threads aren't the ones creating/reusing the icc_caches. | 17:14.09 |
| ray_laptop | Robin_Watts: they don't | 17:14.13 |
| Robin_Watts | The code setting up the threads on the main thread is the one that sets up the icc_cache for the threads it's about to create. | 17:14.40 |
| ray_laptop | Robin_Watts: right. The thread just uses it | 17:15.11 |
| henrys | paulgardiner: maybe not - you are thinking install a jvm on an iphone? | 17:15.16 |
| Robin_Watts | Right. | 17:15.18 |
| chrisl | Robin_Watts: then I'm not clear how each rendering thread knows which icc_cache to use from the array in the gs_lib_ctx | 17:15.20 |
| ray_laptop | chrisl: each thread has an 'index' in the render_threads array. The cache would correspond to the index | 17:16.01 |
| Robin_Watts | henrys: Let's be clear here. Relying on a JVM is still a bad solution. It may not be as bad a solution as relying on LO, but it's still bad. | 17:16.27 |
| chrisl | ray_laptop: right, so each thread would search for "self" in the render_threads array, and use the corresponding entry in the icc_cache array? | 17:16.52 |
| Robin_Watts | chrisl: We set up some structures for the thread, and then we say "create a thread, and pass it this structure" | 17:16.53 |
| ray_laptop | but it really isn't critical that page 1 and page 2 assign the same cache to a certain thread. If a link isn't present, it just computes one | 17:17.01 |
| Robin_Watts | and part of that structure is a pointer to the icc_cache. | 17:17.07 |
| paulgardiner | henrys: no I'm just floundering really given we are not currently seeing a good way forward | 17:17.12 |
| chrisl | Robin_Watts: I thought, from what you said above, that each rendering thread would grab it's icc_cache from the gs_lib_ctx, rather than each thread have the appropriate icc_cache passed to it | 17:18.13 |
| ray_laptop | Robin_Watts: specifically, the icc_cache_cl is pointed to by the clone of the clist reader device unique to each thread | 17:18.27 |
| Robin_Watts | No. The setup code would grab an icc_cache from the ctx rather than create one. And then that would be passed to the new thread. | 17:18.53 |
| ray_laptop | The modification would (partly) be in setup_device_and_mem_for_thread | 17:19.09 |
| chrisl | Robin_Watts: Right, I follow now - I think that's the right solution | 17:19.25 |
| | Robin_Watts: I just misinterpreted your description earlier, sorry. | 17:19.58 |
| Robin_Watts | So... the problem is that the icc_caches use the threads own wrapped chunk allocator. | 17:19.59 |
| ray_laptop | so we'd probably factor that out of that fn and have the caller pass the icc_cache_cl pointer to use | 17:20.00 |
| Robin_Watts | and that gets torn down at the end of the page. | 17:20.20 |
| | So either we ALSO need to store the allocators to be reused on a page by page basis, or we need to move the icc_cache to use a different allocator that doesn't get torn down. | 17:20.55 |
| | And as chrisl pointed out this morning, that new allocator would need to be thread safe. | 17:21.11 |
| chrisl | Doesn't the icc_cache object contain a memory pointer? | 17:21.34 |
| Robin_Watts | I prefer the first idea, personally, but I could believe that there might be problems with that. | 17:21.42 |
| ray_laptop | I think we'd need to make sure that the cache maintenance logic always allocs and frees using the memory pointer that is part of the gsicc_link_cache_t | 17:21.50 |
| Robin_Watts | ray_laptop: OK. And is that memory pointer the threads own one ? | 17:22.19 |
| paulgardiner | I suppose there's still the possibility to carve out the parts of LO we need ourselves avoiding all the extra dross and overhead, but it would be a difficult job. | 17:22.24 |
| ray_laptop | if the icc_cache_cl is passed into setup_... then it could be allocated by the caller using any allocator, right ? | 17:22.56 |
| Robin_Watts | ray_laptop: It could. But it would need to be a thread safe one, right? | 17:23.23 |
| ray_laptop | the gsicc_cache_new gets called with a memory pointer that is _supposed_ to be what it uses for everything | 17:23.39 |
| Robin_Watts | ray_laptop: Indeed, but I *thought* that that was currently the one that was wrapped for the thread. | 17:24.08 |
| | and as such it's safe cos only one person will be calling it at a time. | 17:24.23 |
| ray_laptop | Robin_Watts: yes, it would need to be thread safe, but it can be common amongst threads | 17:24.24 |
| Robin_Watts | Right. That would seem ideal to me then. | 17:24.46 |
| ray_laptop | Robin_Watts: icc caches are currently started with the thread's memory because that's what's in setup... | 17:25.22 |
| Robin_Watts | ray_laptop: Right. | 17:25.33 |
| | So, how do we make a thread safe allocator? | 17:25.51 |
| ray_laptop | The locking performed by a shared thread safe allocator wouldn't kill us because links don't get built that often | 17:25.59 |
| | Robin_Watts: every allocator has a mem->thread_safe_memory | 17:26.33 |
| Robin_Watts | ray_laptop: OK. | 17:26.42 |
| mvrhel_laptop | yea. I finally heard back from one of my contacts at MS | 17:26.50 |
| ray_laptop | for the main thread, it's usually the gsmalloc (heap) allocator | 17:27.00 |
| mvrhel_laptop | so I need to rework the printing in the windows8 app as to this correctly requires DirectX | 17:27.14 |
| Robin_Watts | ray_laptop: How much effort is it to make and release chunk allocators? | 17:27.46 |
| ray_laptop | the only reason each thread has its own chunk allocator (which goes to the outer, shared thread_safe_memory for chunks) is to avoid the frequent locks | 17:27.55 |
| Robin_Watts | Is it worth considering storing the chunk allocators across pages too? | 17:28.04 |
| ray_laptop | Robin_Watts: making chunk allocators is trivial and releasing is as well | 17:28.32 |
| Robin_Watts | Or does that open us up to the possibility of things leaking? | 17:28.36 |
| ray_laptop | that doesn't hurt our performance | 17:28.46 |
| Robin_Watts | ok. | 17:29.01 |
| ray_laptop | Robin_Watts: and chunks always free up all chunks on exit so no leaks are possible | 17:29.14 |
| Robin_Watts | ray_laptop: Right, but if we kept the thread allocators across pages, we *wouldn't* be exiting each page. | 17:29.43 |
| ray_laptop | in debug mode, we report the contents of any non free chunks on exit to help spot leaks in the rendering code | 17:29.44 |
| Robin_Watts | ray_laptop: Yeah, and we currently complain about icc blocks being leaked. | 17:30.02 |
| ray_laptop | Robin_Watts: I don't think that it is a good idea, or that it would help to share allocators across pages. Doesn't save any measurable time | 17:30.27 |
| Robin_Watts | ray_laptop: Fair enough. | 17:30.39 |
| | So, shall I have a go at making the icc_cache's persistent, or would you like to ? | 17:31.03 |
| ray_laptop | Robin_Watts: I did play (> 1 year ago) with having the threads remain active and just wait for a 'signal' to start the rendering of a band (after the main thread filled in the info). | 17:32.08 |
| | it didn't seem to save much | 17:32.28 |
| | right now, it fires up a thread for every band and then the thread exits | 17:33.00 |
| Robin_Watts | ray_laptop: Eh? | 17:33.13 |
| | I thought it fired up the threads at the start of the page and they all lived until the bottom of the page. | 17:33.44 |
| | i.e. one thread did multiple bands ? | 17:33.52 |
| kens2 | ANyone know who owns the tiffsep device ? | 17:34.16 |
| ray_laptop | rather than startup starting all threads that just sit and idle waiting for a semaphore that lets them run | 17:34.27 |
| henrys | Robin_Watts: sent you the xls , it looks good in Google docs too. | 17:34.43 |
| ray_laptop | Robin_Watts: see clist_start_render_thread | 17:34.48 |
| Robin_Watts | ray_laptop: So for a page with 100 bands, we start 100 threads? | 17:34.57 |
| ray_laptop | Robin_Watts: with no more than 4 at a time, yes | 17:35.21 |
| | 4 == NR | 17:35.27 |
| | NRT | 17:35.30 |
| Robin_Watts | oh, but we only do 4 setup_mem_and_buffer things right ? | 17:35.31 |
| ray_laptop | Robin_Watts: yes | 17:35.39 |
| Robin_Watts | right, that makes sense. | 17:35.44 |
| ray_laptop | so it's not a big change to work the other way. | 17:36.09 |
| | but the time to start a thread is small compared to the rest of the stuff that gets done to render a band | 17:37.06 |
| kens2 | O goodnight all | 17:37.13 |
| ray_laptop | so it was not measurable | 17:37.19 |
| Robin_Watts | night kens. | 17:37.34 |
| | yeah, I understand. | 17:37.45 |
| | ray_laptop: So do you want to try this icc_cache thing or should I ? | 17:38.00 |
| henrys | Robin_Watts: have to run and errand, I guess mvrhel_laptop or ray_laptop can do a quick xl fix if you need it. | 17:38.07 |
| Robin_Watts | henrys: Unless ray_laptop or mvrhel_laptop have any changes to suggest to the email I was going to send it. | 17:38.34 |
| ray_laptop | what's a quick xl fix ? | 17:38.39 |
| mvrhel_laptop | i was going to ask the same thing | 17:38.48 |
| Robin_Watts | ray_laptop, mvrhel_laptop: There is a proposed response to 395's questions on tech. | 17:39.12 |
| | That has an xl file attached. | 17:39.18 |
| mvrhel_laptop | ah ok | 17:39.18 |
| | let me read over real quick | 17:39.25 |
| Robin_Watts | but it seems that Libre Office sets the columns in the spreadsheet to be narrow, and that makes it hard to read. | 17:39.40 |
| ray_laptop | Robin_Watts right, but I didn't think I would have anything to contribute to that. | 17:39.40 |
| Robin_Watts | so henrys has been fixing it in excel. | 17:39.53 |
| ray_laptop | Robin_Watts: Oh, OK. I have excel if you want me to fix it | 17:40.11 |
| henrys | bbiab | 17:40.21 |
| mvrhel_laptop | :) that is what you were complaining about with respect to timing | 17:41.22 |
| Robin_Watts | mvrhel_laptop: Yeah. | 17:41.44 |
| chrisl | ray_laptop: I strongly suspect that Windows "caches" threads in some way these days. I remember, for a while, "thread pools" were quite a fad when creating and destroying threads was fairly expensive, but fell out of favour, presumably because the costs became much lower. | 17:42.44 |
| ray_laptop | mvrhel_laptop: on the profile for text, it couldn't map all colors to black. That would break J6. It would need to just map the colors that are close to neutral, or with very low L (where color doesn't matter). How hard is that ? | 17:43.57 |
| mvrhel_laptop | ray_laptop: to create an ICC profile that is reasonably smooth as we go from where you want black to where you don't takes a bit of time. I don't have an ICC profile editor available any more (at least one that I can use to do this quickly) | 17:45.24 |
| | Robin_Watts: your reply looks good to me | 17:45.59 |
| Robin_Watts | mvrhel_laptop: Thanks. | 17:46.09 |
| | So, unless anyone else has anything to add I will send it... | 17:46.20 |
| | sent! | 17:48.14 |
| | ray_laptop: So, did you do timings for 801, or should I ? | 17:48.48 |
| ray_laptop | Robin_Watts: I am still straightening out my mess with my repo. It sure would be nicer to get git patches from you | 17:49.54 |
| Robin_Watts | ray_laptop: OK, let me commit what I've got into git. | 17:50.15 |
| ray_laptop | because some of the files you zap are ones that have changes that aren't in the master | 17:50.29 |
| | Robin_Watts: you mean on a local branch ? | 17:50.45 |
| Robin_Watts | ray_laptop: yeah. | 17:50.51 |
| ray_laptop | if you want. | 17:51.03 |
| mvrhel_laptop | ray_laptop; I used to have some tools to extract the tables out of a profile, which I would then edit as an image and then store back | 17:51.16 |
| | I have not done this in a long time. If we decide we wanted to do this sort of thing, I would probably need to rewrite the tools | 17:51.43 |
| ray_laptop | but I am now at a point where I cleaned up a merge problem, then did a git add base/gdevprn.c and git status doesn't show anything, but git rebase --continue gives me: No changes - did you forget to use 'git add'? | 17:52.24 |
| Robin_Watts | ray_laptop: Maybe the changes resolved away to nothing ? | 17:53.21 |
| ray_laptop | damn. my changes to the color_usage aren't on this branch :-( | 17:54.16 |
| Robin_Watts | ray_laptop: So, would you like a git patch ? | 17:54.54 |
| ray_laptop | I'm going to dump this branch and start from the original 0822 code, load your changes, then merge with my master | 17:55.00 |
| Robin_Watts | eh? | 17:55.13 |
| | My changes are on top of 9.10, NOT 0822. | 17:55.29 |
| ray_laptop | sorry, forget about the 0822 | 17:55.30 |
| Robin_Watts | right. | 17:55.33 |
| | Would a git patch be easier for you? | 17:55.42 |
| ray_laptop | easier for me to combine changes when we are both working on the same modules, yes, for the future. Not needed right now. | 17:57.02 |
| | I'll just pull in your tarball on top of the branch | 17:57.17 |
| Robin_Watts | So, I shall do timings for 801. | 17:59.33 |
| | And then tomorrow I will look at the possibilities in sharing icc_caches, unless you tell me that you'd rather do that. | 18:00.12 |
| ray_laptop | Robin_Watts: OK, I can re-do them after the improved (fixed) color_usage lets me skip some planes. | 18:00.42 |
| Robin_Watts | ray_laptop: Right. I'll tell that we are working on more improvements. | 18:01.03 |
| | ray_laptop: Unless you're close to having that finished... | 18:01.22 |
| ray_laptop | Robin_Watts: good idea. | 18:01.27 |
| | maybe I'll just edit J12 to fix the colors for the near black text | 18:02.57 |
| | OK, testing my edited file (this way I don't need a fancy profile from mvrhel_laptop) | 18:14.28 |
| mvrhel_laptop | ray_laptop: that is going to be a *much* faster solution | 18:14.53 |
| | sorry | 18:15.00 |
| ray_laptop | mvrhel_laptop: np. but if the timings are a lot better, we can suggest that we might be able to do such a profile, right? | 18:15.41 |
| | since it'll help the text quality IMMENSELY | 18:16.03 |
| mvrhel_laptop | We can suggest that | 18:16.16 |
| ray_laptop | Of course, in the real solution, they should *NOT* be doing host based color correction when creating the file | 18:18.48 |
| | OK. my fixed file works great | 18:19.05 |
| | I guess we can send that to let them print it and look at the difference on paper :-) That should convince them that it's a good idea | 18:19.56 |
| Robin_Watts | The JEITA files we have are too small to give a meaningful comparison, I think. | 18:23.35 |
| | except for J11 which causes a SEGV :( | 18:26.24 |
| ray_laptop | oops | 18:26.58 |
| | Robin_Watts: that's probably why they mashed up a file with 100 copies of J6 :-) | 18:27.29 |
| Robin_Watts | But I feel we've been silent for long enough. I'll send them some timings to be getting on with, and say we are working on a couple more ideas. | 18:28.11 |
| ray_laptop | Robin_Watts: yes, agreed. | 18:30.02 |
| | Robin_Watts: are you going to send them the changes ? | 18:30.58 |
| Robin_Watts | I will ask them how they would like them. | 18:31.14 |
| ray_laptop | Robin_Watts: OK. | 18:31.27 |
| | Robin_Watts: BTW, did you do timings with -dBGPrint=true ? This would (in theory) overlap parsing | 18:32.32 |
| | and I know they want that | 18:32.51 |
| Robin_Watts | I did not, and it would not. | 18:33.09 |
| | because I haven't used bg_outpyt_page :) | 18:33.22 |
| ray_laptop | Robin_Watts: would not work ? | 18:33.22 |
| Robin_Watts | I'm still calling gdev_prn_output_page. | 18:33.38 |
| ray_laptop | Robin_Watts: OK, so I'll include that in my timings. | 18:33.41 |
| | and if it doesn't work, I can debug it :-) | 18:33.52 |
| Robin_Watts | great. | 18:34.00 |
| Gigs- | I wonder if Kens has figured out I just put all my bugs under pdf interpreter because I have no idea :) | 18:35.42 |
| | ray what component goes to you, it seems most of them wind up in your lap anyway | 18:36.36 |
| mvrhel_laptop | bbiab | 18:36.54 |
| | off to get some new eyeglasses | 18:37.01 |
| ray_laptop | Gigs-: I noticed kens asked who owned tiffsep. Were those your bugs ? | 19:14.31 |
| | another screwy thing in J12. There is an Indexed colorspace with a hival == 0 ??? | 19:32.50 |
| | Robin_Watts: So with a release build, but without my 'skip plane if not marked', I am seeing 17.2 seconds for J12. Is that reasonable ? | 19:52.55 |
| Robin_Watts | ray_laptop: At what res? | 19:54.18 |
| ray_laptop | 600 dpi | 19:54.25 |
| Robin_Watts | On my machine with the new device at 600dpi, and a bandheight of 128, it takes 21.043 | 19:54.47 |
| | so, as you probably have a machine with a faster clockspeed than my 1.6GHz, I'd say, yes, that was a reasonable timing. | 19:55.22 |
| ray_laptop | I didn't force BandHeight. Let me try that | 19:55.25 |
| Robin_Watts | I force bandheight to be 128 to match their old device. | 19:55.43 |
| | Without that, their old device will crash. | 19:55.51 |
| ray_laptop | That gives me 17.25 seconds (best case in 3 runs) | 19:57.28 |
| | with NumRenderingThreads=2 or 3 it is 15.4 sec. More than 3 gets slower (4 is 16.6, 5 is 17.7) | 20:00.32 |
| | that's on my i7 2640 laptop at 2.8 GHz | 20:01.13 |
| Robin_Watts | ray_laptop: Possibly, I only tested NumRenderingThreads=4 | 20:08.09 |
| | Urm... if crdev->data_size = 400000, would it be bad for crdev->page_info.tile_cache_size = 600000 ? | 20:09.08 |
| ray_laptop | OK, so now I'll proceed to skip planes. First I have to change the pms function(s) to pass the color_usage_bits somehow. | 20:15.45 |
| | Robin_Watts: what do you think -- pass in the rij_process_buffer_t (and add it to that) and let the pms functions get the data pointer from buffer->params.data[0] | 20:17.22 |
| | Robin_Watts: or add an argument to the pms fn's ? | 20:17.40 |
| Robin_Watts | add an arg. | 20:21.27 |
| ray_laptop | Robin_Watts: OK. | 20:21.46 |
| Robin_Watts | I wonder if this SEGV predates my changes. | 20:25.11 |
| | I'm seeing the buffer device closed in clist_playback_band before I get a change to getbits from it. | 20:25.43 |
| ray_laptop | Robin_Watts: do you think that the GB_SELECT_PLANES will work if I zap (after saving) the unused data[] pointers to NULL ? Thinking that since I'm going to skip the processing anyway, why even write them | 20:27.28 |
| Robin_Watts | ray_laptop: Eh? | 20:28.03 |
| | ray_laptop: Typically a process_fn asks for GB_RETURN_POINTER | 20:28.30 |
| ray_laptop | in rij_process, for the call to get_bits_rectangle | 20:28.35 |
| Robin_Watts | we ask for a pointer to the buffer devices data. | 20:28.48 |
| | if we get NULL back, then we will crash. | 20:29.04 |
| ray_laptop | who "we" ? | 20:29.29 |
| Robin_Watts | the device. | 20:29.50 |
| ray_laptop | in the planar memory device ? | 20:29.54 |
| Robin_Watts | No, in 801s device. | 20:30.05 |
| | 801s device has a process_fn. | 20:30.17 |
| | That gets called. | 20:30.19 |
| | It does a get_bits_rectangle with GB_RETURN_POINTER | 20:30.32 |
| | it therefore expects to be given a set of data pointers where it can read the data from. | 20:30.48 |
| ray_laptop | Robin_Watts: this would be in the process_fn, and it would save the pointers, call the get_bits_rectangle, then restore them before proceeding | 20:30.53 |
| Robin_Watts | ray_laptop: Oh, I see. | 20:31.23 |
| | why save them then? | 20:31.27 |
| ray_laptop | the buffer device hangs around, and I need at least data[0] to call the pms fn | 20:32.06 |
| Robin_Watts | We call get_bits_rectangle with GB_RETURN_POINTER, so the data pointers within the options structure are undefined. | 20:32.13 |
| | when we come back from get_bits_rectangle, the values have been filled in. | 20:32.32 |
| Gigs- | ray_laptop: yeah there's a new one today | 20:33.13 |
| Robin_Watts | If you then want to blank those values to NULL if the color_usage says they are unused, then feel free. | 20:33.15 |
| Gigs- | http://bugs.ghostscript.com/show_bug.cgi?id=694759 | 20:33.18 |
| | well it's not really that new, but I just filed it today | 20:33.29 |
| Robin_Watts | then the pms functions can spot the pointers being NULL, right? | 20:33.36 |
| ray_laptop | I was looking at the comments above GB_SELECT_PLANES -- it says to set the plane data pointers to NULL | 20:33.38 |
| Gigs- | we've been seeing it a while but it's one of those screwy files | 20:33.40 |
| Robin_Watts | ray_laptop: Sorry, where? | 20:33.47 |
| | We don't use GB_SELECT_PLANES, do we ? | 20:34.07 |
| ray_laptop | in gxbitfmt.h | 20:34.15 |
| Robin_Watts | Right, we don't use GB_SELECT_PLANES, so whether they are null or not on entry is irrelavent. | 20:35.00 |
| | And we are only asking for pointers to the data. No copying is done. | 20:35.38 |
| | hence there is no work to be saved by using GB_SELECT_PLANES. | 20:35.58 |
| ray_laptop | oh, it looks like GB_SELECT_PLANES can only return a single plane if I'm reading gdevmpla.c correctly | 20:36.04 |
| Robin_Watts | ray_laptop: quite possibly, but why would we want to use it anyway? | 20:36.27 |
| | GB_SELECT_PLANES has advantages if you are copying the data out into your own buffers (GB_RETURN_COPY), but we are deliberately not doing that. | 20:37.07 |
| ray_laptop | Robin_Watts: won't it be faster to only paint the planes that we are going to use ? | 20:37.51 |
| Robin_Watts | ray_laptop: The clist rendering has painted all the planes anyway. | 20:38.27 |
| | AIUI the call to getbits can never influence what planes are drawn, only what planes are returned. | 20:38.54 |
| ray_laptop | Robin_Watts: clist_get_bits_rectangle gets the options and sets up it's buffer device. In theory, for a planar mem device it could communicate which planes to skip during the painting. I think | 20:42.04 |
| Robin_Watts | ray_laptop: It possibly *could*, but I don't believe it *does*. | 20:43.27 |
| ray_laptop | one method might be to setup line ptrs that have NULL for the planes we skip. Then the actual painting routines that iterate over the planes can skip if the line pointer for the plane is NULL | 20:43.50 |
| | Robin_Watts: No, it's apparent that it doesn't do it now. | 20:44.08 |
| Robin_Watts | right, except we don't "paint all the cyan", "paint all the magenta" etc. | 20:44.27 |
| | we "paint the first object", "paint the second object". | 20:44.52 |
| | and within each object we "paint cyan", "paint magenta" etc... | 20:45.18 |
| | so we'd have to update all the planar drawing routines to check for line_ptrs being null. | 20:45.44 |
| ray_laptop | Robin_Watts: in mem_planar_fill_rectangle or mem_planar_fill_rectangle_hl_color, it iterates over the planes | 20:46.00 |
| Robin_Watts | and we'd need to update the line_ptrs setup routines to correctly set them all null. | 20:46.05 |
| ray_laptop | Robin_Watts: that's right as far as what's involved. | 20:46.35 |
| Robin_Watts | It's possible, but it's not a trivial job. | 20:46.39 |
| | and aren't we already smart enough to skip the writes in mem_planar_fill_Rectangle if the given colorant value is 0 ? | 20:47.04 |
| ray_laptop | I'll look into it after doing the skip process for the planes | 20:47.12 |
| Robin_Watts | hence we might not actually be saving anything. | 20:47.16 |
| | skipping on colorant value is going to be less of a blunt instrument than skipping on color_usage_bits anyway. | 20:48.08 |
| ray_laptop | Robin_Watts: it doesn't skip if 0 since it may have to paint 0's (not knowing that the buffer has never been anything *but* 0's) | 20:48.11 |
| Robin_Watts | Oh, and now I come to think of it, we REALLY don't want to skip the painting! | 20:48.28 |
| ray_laptop | Robin_Watts: that's only for overprint mode | 20:48.30 |
| | Robin_Watts: why don't we want to skip ? | 20:48.53 |
| Robin_Watts | a lot of the time with planar stuff people might 'get the C plane', then 'get the M plane', then 'get the Y plane' etc. | 20:49.05 |
| | We really don't want to have to rerender the whole lot each time. | 20:49.17 |
| ray_laptop | re-render the whole lot ??? | 20:49.30 |
| Robin_Watts | better for it to render entirely when asked for the C, then the M and the Y etc gets can be supplied instantly. | 20:49.47 |
| | when I ask for the C plane, it should render CMYK, and return the C. | 20:50.12 |
| | then, when I ask for the M plane it can trivially return it. | 20:50.24 |
| | That will be faster than asking for C, and just rendering C. Then asking for M and just rendering M. Then asking for Y and just rendering Y. | 20:50.50 |
| ray_laptop | well, if that's the device's mode, yes. | 20:50.51 |
| | so, the calling model you have is for devices that ask for a plane at a time, but expect it to render all the planes | 20:51.38 |
| Robin_Watts | That's the common case I believe. | 20:51.54 |
| ray_laptop | and so, there's no way to tell it "I'm never going to need C, M, and Y since I know the page only has K that I need) | 20:52.27 |
| Robin_Watts | This does get us into trouble when we have devices where we want to pull all the C out, then all the M out, then all the Y out then all the K out, if we're banding. | 20:52.32 |
| | ray_laptop: Indeed. | 20:52.40 |
| ray_laptop | oh, well... | 20:56.56 |
| | It just irks me to be painting 0's on top of 0's for all the planes we are going to ignore anyway :-/ | 20:57.29 |
| | oooh. How about this. for each line, keep an 'is_dirty' bit (or a 'marked' rectangle or something). If we are painting where it isn't marked, we can skip painting components that are 0's ??? | 20:59.27 |
| | This is valid only for subtractive devices, of course. | 20:59.53 |
| | even just a 'marked' bit per plane | 21:00.36 |
| Robin_Watts | ray_laptop: We could do that, yes. | 21:01.03 |
| ray_laptop | we'd go ahead and let fillpage ignore the marked bits (actually clear the marked bits) | 21:01.15 |
| | then the other painting ops would inspect it based on the plane's colorant | 21:01.43 |
| | and set the marked bit when the colorant isn't "white" | 21:02.08 |
| | I like that a lot better. | 21:02.32 |
| | OK. Time for lunch... | 21:03.57 |
| Robin_Watts | I now have a stupidly simple file that causes the device to crash. | 21:11.55 |
| | I fear it's down to compositor handling. | 21:12.18 |
| | and hence not actually my fault. | 21:12.38 |
| | I will try to reproduce it with fpng tomorrow. | 21:12.53 |
| sebras | paulgardiner: do we normally re-order the annots on a page when editing them? | 21:44.59 |
| | paulgardiner: I'm thinking about commit 2880476. if there are many annotations on a page it will take time to iterate through the list. | 21:45.49 |
| ray_laptop | so, with BandHeight=128, 600 dpi, out of 1100 bands, the bands we actually run the process on (CMYKB) are: [928 939 942 999 352]. Big win on B | 23:02.23 |
| | with the j12-fixed-text.ps that I made, the counts are: [ 560 571 574 999 352]. Quite a reduction !!! | 23:02.25 |
| | timings coming up... | 23:02.43 |
| mvrhel_laptop | yes! I was finally able to get printing in the windows 8 app via directX which fixes things for me. need to do a bit more work and clean up but the hard part (which was decoding windows documentation) isover | 23:08.39 |
| | Robin_Watts: so probably in the next couple days I will have a new commit for you to review which reworks the previous print approach. then I can finally get on to finish up the windows phone | 23:15.19 |
| | Forward 1 day (to 2013/11/07)>>> | |
IRC Logs