| | <<<Back 1 day (to 2012/05/14) | 2012/05/15 |
| tkamppeter | chrisl, hi | 09:02.04 |
| chrisl | tkamppeter: hi | 09:05.00 |
| tkamppeter | chrisl, it is about your last comment on Ubuntu bug #998087. Is it true that Ghostscript's color managament is only for raster output and there is no color management for PS/PDF/PCL-XL output? | 09:06.44 |
| chrisl | tkamppeter: it would be more accurate to say that the vector output doesn't take *full* advantage of the ICC workflow, but I didn't want to get into that in the bug, since it's not relevant to the bug | 09:07.59 |
| tkamppeter | chrisl, and if it is only available for raster, is it available for all raster drivers, like cljet, PNG, TIFF, ... or only for CUPS Raster? | 09:08.06 |
| kens | PostScript and PDF output will get ICC colour management in a future release. At themoment it is 'partially' implemented, but not thoroughly. | 09:08.54 |
| tkamppeter | chrisl, so do we have a color management advantage for ps2write output against Poppler or can I return to Poppler for Ubuntu 12.04 LTS? I only did the switchover under the assumption that I get color management. | 09:10.05 |
| chrisl | tkamppeter: the ICC color workflow is in place for all devices. The vector devices could (and will) do more with it. | 09:10.07 |
| kens | Broadly speaking the inpu tis colour managed, the output isn't, though even that is an over-simplification. | 09:11.07 |
| | I'm expecting to start revamping the pdfwrite colour management 'soon', but we need the ability to create V2 ICC profiles for PDF/A output. | 09:11.47 |
| chrisl | tkamppeter: Yes, I think there is an advantage. The other problem is, if you want the "full" functionality when we get it implemented, we're just going to hit these same problems at that point. | 09:12.00 |
| | tkamppeter: and if you remember, that HP problem has been reported before, but the reporter stopped answering our e-mails | 09:15.24 |
| tkamppeter | chrisl, kens, for me it is now important to know about how to react to user complaints. Users complain that rendering for PS printers got too slow in 12.04 and it also showed up that Konica Minolta and also other HP printers do not like GS's PostScript. | 09:32.42 |
| kens | kamppeter, there are several points there. | 09:33.12 |
| | The 'slow rendering' was, iirc, due to converting PDF files with transparency and not setting a resolution, which led to 30-0 dpi files being compared to 720 dpi files. As far as I am aware, chenging the resolution to match poppler results in files at least as fast. | 09:34.08 |
| | Also it seems that Poppler simply ignored resolution requests in the past, which masks hte problem. | 09:34.40 |
| | THe solution to that one is to use a more appropriate resolution. | 09:34.56 |
| | As regards the pritners; these are bugs in the implementations in those printers. | 09:35.13 |
| tkamppeter | chrisl, kens, if I return to Poppler only for the already released Ubuntu 12.04 and keep using Ghostscript for the 12.10 development cycle I could satisfy production users of 12.04 but users who test 12.10 during development still can give me input about the compatibility of GS with printers and perhaps also about color output quality. | 09:35.43 |
| kens | Now obviously we cna work around those (we have been doing so) but there is no way for us to predict bugs in other people's implementations. All we can do is try to resolve the problems later. | 09:36.05 |
| chrisl | tkamppeter: how viable would it be to make poppler an option - keep GS as the default, meaning we can get a list of printers with problems (and hopefully users with those printers willing to help), and we can try to address those over time. In the interim, you can point them to the poppler option as a workaround. | 09:36.14 |
| tkamppeter | chrisl, I already thought about this, changing the pdftops filter to allow switching between GS and Poppler by run-time configuration, ideally even per-queue and also issuing this change as update for 12.04. | 09:39.40 |
| | Current pdftops sets the actual printing resolution for both GS and Poppler (current Poppler supports this). I can also introduce a per-queue-runtime-configurable resolution limit, so that the GS/Poppler rendering resolution does never go higher than that. | 09:41.51 |
| chrisl | tkamppeter: but the point kens was making was that the comparisons the reporters were making were against a poppler version that hard coded to 300dpi | 09:42.43 |
| | tkamppeter: I guess for 12.04 you could keep gs for the raster based drivers (where the largest benefit is currently), and revert to poppler for Postscript. And as you say, use ps2write for the 12.10 release. | 09:43.15 |
| tkamppeter | chrisl, so if someone has slowness problems under 12.04 he siomply setrs the limit to 300dpi and then GS should be as fast for him than former Poppler. If a printer does not print at all, he switches it to Poppler. | 09:45.08 |
| kens | sorry, went to get a coffee | 09:46.14 |
| chrisl | tkamppeter: yes, that's what I was thinking. The only way we're get past these issues is if we have people actually using the GS PS output. | 09:46.33 |
| tkamppeter | chrisl, for 12.04 defaulting back to Poppler for PS printers is a good idea, independent whether I introduce the runtime-configurable pdftops or not. For raster printers GS was always used for built-in GS drivers and for CUPS Raster drivers GS was used at least since 11.10 and worked perfectly there and also now in 12.04. | 09:48.32 |
| chrisl | tkamppeter: as an aside, I was aware of the variable quality of various Postscript implementations, but some of the problems are truly *shocking* - I'm fairly sure some of the things we've worked around would cause the printer to fail the Quality Logic test suite. | 09:50.50 |
| tkamppeter | chrisl, so I think I will implement the runtime-configurable pdftops ASAP and also include it in the 12.04 update. In 12.04 it will default to Poppler with 360-dpi limit and in 12.10 it will default to GS with 1440-dpi limit (default limit can be lowered here during the development cycle depending on user complaints). | 09:51.22 |
| kens | I wouldn't start at 1440, that's very high. The default of 720 should be fine for most practical purposes. | 09:52.05 |
| tkamppeter | chrisl, what is the "Quality Logic test suite"? Is this a PS interpreter tester at Artifex? | 09:52.41 |
| kens | tkamppeter : its a PostScript test suite available form the testing company 'Quality Logic' (formerly Genoa) | 09:53.07 |
| | Its an industry standard test. | 09:53.15 |
| | http://www.qualitylogic.com/Contents/Independent-Software-Vendors/Technology/PostScript.aspx | 09:53.51 |
| tkamppeter | kens, this is an upper limit. if actual printing resolution is 600 dpi, on a system with thjs limit the job will get rendered at 600 dpi, but a 2400-dpi job will get rendered at 1200 dpi. | 09:53.57 |
| kens | OK fair enough till, I didn't realise that was a maximum :-) | 09:54.19 |
| tkamppeter | chrisl, kens, but what are actual human-eye-reading resolutions, I have heard that it is around 300 dpi, so 600 dpi and 1200 dpi are probably already difficult to distinguish and more serving for extra color depth with a larger dither matrix. Am I right? This would allow me to set the default ceiling at 720 dpi. | 09:56.57 |
| kens | tkamppeter the eye can resolve around 1000 dpi at comfortable reading distances | 09:57.22 |
| | text rendered at this resolution or above is 'smooth' | 09:57.36 |
| | However, colour reproduction involves halftones with 4 inks | 09:57.50 |
| | So the size of teh halftone cell becomes important. | 09:58.00 |
| | Halftones in colour reproduction range up to 150 lines per inch | 09:58.14 |
| | with 256 gray levels | 09:58.24 |
| | This is not a simple topic.... | 09:58.37 |
| chrisl | The resolution we're talking about here won't affect the halftone cells, though | 09:58.52 |
| kens | You cna trade-off higher frequencies (smoother output) against colours | 09:59.00 |
| tkamppeter | kens, so then it should be better to stay witrh the 1440-dpi ceiling. | 09:59.08 |
| kens | As a maximum ? yes, that's fine | 09:59.31 |
| | rule of thumb is images need to be nredered at ~1/4 of the touptu resolution for 'decent' colour reproduction. | 10:00.01 |
| chrisl | kens: this isn't printer resolution, though, so shouldn't have any effect on color or halftones | 10:01.36 |
| kens | chrisl the resolution of the images in rendered transparency will have an effect on the printed output | 10:06.43 |
| kens | is trying to apply for Olympics tickets, and tehrefore somewhat distracted | 10:07.00 |
| chrisl | kens: not the color or the halftone, though. The color will be contone, and the halftone is done by the printer | 10:07.34 |
| kens | chrisl yes, but if I render a transparent image at 72 dpi and send it to a 1200 dpi engine, teh conteon image will be full of jagged edges | 10:08.09 |
| | The 'flattened' (rendered) data needs to be contone at a reasonable resolution | 10:08.34 |
| | rednering at 1200 dpi and sending to a 600 dpi printer is a waste of time, rendering at 150 dpi for a 1200 dpit printer will give poor output | 10:09.09 |
| chrisl | True, but if you render a transparency image at 600dpi send it to a 2400dpi, it will look just fine. | 10:09.19 |
| kens | Yes, hence my 'rule of thumb' above | 10:09.28 |
| | The point is there is no 'correct' resolution to render at. | 10:09.42 |
| tkamppeter | kens, chrisl, can I tell GS to render text with 1200 dpi and images with 300 dpi? | 10:10.20 |
| kens | I like your original idea of a 'slilder' or 3 check boxes for 'quick but poor' 'slower but OK' and 'slow but nice' output | 10:10.20 |
| | tkamppeter no | 10:10.27 |
| chrisl | Yeh, so how many printers are we likely to be generating data for that run at 5760 dpi? | 10:10.45 |
| kens | Apart from anything else this only affects transparency, so the distinction isn't feasible | 10:10.50 |
| | chrisl, yes, which is why I said 1440 as a maximum was fine | 10:11.05 |
| chrisl | I'd reckon 720 would be fine as a maximum - but as you say, this is partially informed guesswork...... | 10:11.53 |
| kens | TBH 600 is probably fine as a maximum, but as long as its not what gets used in general, its not a problem. | 10:12.21 |
| | Olympics web site waiting time going up and down like a yoyo | 10:13.20 |
| chrisl | I'd figured 720 would be good for the 1440 and 2880 dpi printers | 10:13.37 |
| kens | SOrry, wasn't paying attention. Got first application declined, trying again. | 10:16.40 |
| | 720 is ifne, 1440 is fine. As long as these are a maximum its really not a problem. | 10:17.02 |
| | Robin_Watts : How do I configure Memento for leak detection only ? | 10:17.54 |
| chrisl | kens: it looks like you need to minimally change the memento source file: see line 17 in base/memento.c | 10:21.06 |
| kens | chrisl thanks I'll look in a moment, trying my 3rd request.... | 10:21.28 |
| | How cna it take 15 minutes to query a database ? | 10:22.37 |
| | Should have bought a bigger mainframe..... | 10:23.04 |
| chrisl | That Babbage didn't design very fast computers....... | 10:23.32 |
| kens | Did you see that Minecraft URL ? | 10:23.48 |
| | I htink they are using one of those | 10:23.59 |
| | I notice the waiting time is going down, I wonder if a lot of people are giving up..... | 10:24.54 |
| chrisl | I think the minecraft thing is too advanced, something more clockwork oriented, I think | 10:25.49 |
| kens | 5th search, estimated waiting time 1 minute, first search was estimated at 15 minutes | 10:27.21 |
| | Of course it could also be that there are no actual tickets left.... | 10:27.38 |
| | Well, time to give up, I can't find any ticket combination which doesn't get me a 'no tickets found' screen | 10:33.27 |
| paulgardiner | Robin_Watts, tor8: Ping | 10:45.04 |
| oy | mvrhel: ping | 10:45.23 |
| tor8 | paulgardiner: hey | 10:47.27 |
| paulgardiner | tor8: Hi. I'm just looking at handling errors arising in the C++ v8 layer. I think it may be dangerous to call fz_throw from C++, so I was thinking I should explicitly return an error value from every pdf_jsimp method then the C layer above can turn those into fz_throw calls. Does char * sound right for an error type? With NULL as no error? | 10:50.17 |
| tor8 | char* as an error message? I do that in the xml parser in xps, so it's not unprecedented :) | 10:51.34 |
| | return the error message as a char* if there's an error, or NULL if no error, that is | 10:52.02 |
| paulgardiner | tor8: Ok good. We don't seem to have a separate error type, and fz_throw looks to produce a message, so it makes sense to me. | 10:52.48 |
| tor8 | paulgardiner: yeah. all errors now go through the fz_throw exception mechanism, but at a lower level they can arrive from various other places (error codes for freetype and other third party libraries). the xml parser returns the error message as a string. | 10:53.56 |
| | so it makes perfect sense, IMO. robin's opinion may differ though :) | 10:54.29 |
| | paulgardiner: I guess I (and Robin) should go over the combined non-interactive mupdf changes commit you mention in your status report and see if we can merge that into master | 10:55.19 |
| paulgardiner | :-) I'll work on that assumption for now, and see whether Robin chips in with something later | 10:55.37 |
| | tor8: Ah right. That would be good. You feel it makes sense to merge it in then? | 10:56.27 |
| tor8 | is longjmp from C++ to C code inherently unsafe, due to automatic destructors of stack variables etc? | 10:57.29 |
| paulgardiner | tor8: I don't know, but automatic destructors were exactly the worry I had. | 10:58.03 |
| tor8 | well, I think it makes sense anyway not to mix runtimes at that level just for keeping things separate | 10:58.41 |
| paulgardiner | The functions from which the fz_throw calls would be made are all declared as extern "C" but I don't think that helps. I could ensure that all local variables were in inner braces, but that might not guarantee destruction because of optimisations. | 10:59.30 |
| tor8 | yeah. I'd be wary as well, C++ scares me :) | 10:59.52 |
| | I'd rather mess with prolog! | 11:00.04 |
| paulgardiner | local variable declarations in inner braces, and the longjmp calls outside them that is. | 11:00.08 |
| | Prolog. I love prolog. Been many years though. | 11:00.31 |
| tor8 | yeah. my introduction to prolog was made more difficult by learning it in combination with constraint language programming... | 11:01.04 |
| | one of the few programming classes where I actually learned something really new :) | 11:01.37 |
| paulgardiner | Hmmm. constraint language programming sounds familiar but I don't think I ever "did any" so to speak. | 11:01.48 |
| tor8 | it's brain twisting! | 11:02.06 |
| paulgardiner | Lasy evaluating languages are fun too, where you can deal with infinite lists. | 11:02.16 |
| tor8 | but they changed the class in later years so now they use java instead of prolog :( | 11:02.34 |
| | I never got much experience with Haskell. | 11:02.53 |
| chrisl | Prolog is a cool way to bring down a computer....... | 11:02.57 |
| paulgardiner | Shame. As you say Prolog is really someting different. | 11:03.06 |
| Robin_Watts | kens: look at the top of memento.c | 11:11.08 |
| | you'll see #undef MEMENTO_LEAKONLY | 11:11.20 |
| | I'll leave the rest to your imagination :) | 11:11.30 |
| kens | Robin_Watts : chrisl already pointed me at it, but thanks | 11:13.54 |
| Robin_Watts | paulgardiner: I think you're right to catch exceptions and turn them into C. | 11:13.59 |
| | kens: yeah, just read that in the logs. | 11:14.07 |
| kens | Not showing any leaks, but I'm not sure of that's really true | 11:14.07 |
| Robin_Watts | paulgardiner: I think you're right to catch exceptions and turn them into errors, sorry. | 11:14.19 |
| | Then throw them. | 11:14.42 |
| | I dislike the fact that our error class is char * only personally. | 11:14.58 |
| | I would prefer a structure that was an int, followed by the char *. | 11:15.18 |
| paulgardiner | Robin_Watts: Good. Thanks. Yeah, char * will be problem if ever we want to test for a particular error, but until then... | 11:15.47 |
| Robin_Watts | so we can compare error types, and still have meaningful messages. | 11:15.55 |
| | Also I'd prefer if we had a scheme in place for internationalisation too, but... | 11:16.09 |
| paulgardiner | Robin_Watts: Oh yes, that too. | 11:17.07 |
| | Hmmm, I'm finding myself doing this. | 11:20.34 |
| | Try again | 11:20.52 |
| | --- /* blah blah blah | 11:21.07 |
| | --- * blah */ | 11:21.15 |
| Robin_Watts | TAB SPACE * ? | 11:21.34 |
| | Yeah, me too. | 11:21.36 |
| paulgardiner | Sometimes with tab sometimes tight to the left | 11:22.05 |
| | Alignment wont be good if ' ' and '/' have different widths. | 11:22.30 |
| Robin_Watts | Oh, I always use TAB SPACE *. Not sure what the 'right' thing to do there is. | 11:22.40 |
| paulgardiner | Well if you are getting away with it... :-) | 11:23.13 |
| | It's not something that will be upset by tab size at least | 11:23.46 |
| tor8 | Robin_Watts: paulgardiner: I wish we could just use // comments... | 11:26.52 |
| paulgardiner | Could we? | 11:27.13 |
| tor8 | I'm not in favor of the star banner multi-line comments you get with 'TAB SPACE *' | 11:27.39 |
| | paulgardiner: we don't at the moment, but maybe we should reconsider. which C compilers these days don't support // comments? | 11:28.06 |
| kens | very old ones :-) | 11:28.47 |
| tor8 | I find the star banner comments messy to edit, especially when joining or wrapping lines | 11:29.03 |
| | but the same is true of // comments I guess :) | 11:29.16 |
| | paulgardiner: which commit/branch has the stuff you want merged into master? | 11:33.59 |
| | also, the android stuff, should we take that on as well? | 11:34.06 |
| paulgardiner | 994bd7d | 11:35.21 |
| | Also the one just before it on the forms branch | 11:36.02 |
| kens | 123456 ? | 11:36.03 |
| | ah, commit hash | 11:36.17 |
| paulgardiner | tor8: Not sure there's any android stuff not on master. Can you see something? | 11:37.16 |
| tor8 | Android app: explicitly release resources when page moves out of cache area | 11:37.29 |
| | miha's patch to produce 2-up display | 11:37.35 |
| | Android app: build safe AsyncTask behaviour into a derived class | 11:37.44 |
| | those three are what I see at a glance | 11:37.50 |
| paulgardiner | OH yes. So there are. explicit release and safe asynctask are probably worth taking | 11:39.10 |
| | miha's patch is there just so I don't lose it. I think it will drastically increase memory usage in some cases. | 11:40.41 |
| Robin_Watts | I think we should carefully consider mihas patch before taking it on. | 11:41.15 |
| tor8 | paulgardiner: right. might want to drop that patch on a separate branch so I don't get confused then :) | 11:41.45 |
| Robin_Watts | tor8: Did you have any thoughts on the linearisation patch on my repo on casper ? | 11:42.00 |
| tor8 | Robin_Watts: haven't had a chance to look at it yet, will do that today | 11:42.19 |
| Robin_Watts | sebras: You said "hint stream?" last night. Linearized pdf contains an object with a hint stream in it. The hint stream contains 'hints' about where in the file different pages etc can be found. | 11:42.43 |
| | It's horrible. | 11:42.54 |
| paulgardiner | tor8: What the utility functions one? Or all the ones we've mentioned? | 11:43.44 |
| tor8 | Robin_Watts: ", with non-local resources." what does that mean? in the commit message | 11:43.46 |
| | paulgardiner: miha's | 11:44.00 |
| Robin_Watts | tor8: Certain things are 'inheritable' in the page tree. | 11:44.25 |
| | MediaBox, CropBox etc can be specified once at the top of the page tree, and inherited by all the kids. | 11:44.43 |
| | In linearized pdf, they need to be on the page tree leaf node. | 11:45.01 |
| paulgardiner | tor8: It is on a separate branch, but my master in my repo has yet to branch off it. | 11:45.08 |
| Robin_Watts | i.e. resources/boxes must be 'local' to the page node. | 11:45.21 |
| tor8 | Robin_Watts: okay. I still don't understand the sentence though. | 11:46.03 |
| Robin_Watts | oh. It should be 'with local resources' | 11:46.44 |
| | too many negations. | 11:46.48 |
| tor8 | :) | 11:46.58 |
| Robin_Watts | updated patch. | 11:47.21 |
| paulgardiner | tor8: I could rebase android-2up somewhere out of the way of confusion. :-) | 11:47.28 |
| tor8 | paulgardiner: ah, so it is! my bad :) | 11:47.47 |
| | I just read the paul/master on the line under it! | 11:48.01 |
| | Robin_Watts: pdf_new_ref ... we already have a fz_new_indirect. maybe pdf_alloc_object instead? | 11:53.03 |
| Robin_Watts | new_ref is different to new_indirect. Hold on and I'll try and remember why. | 11:53.49 |
| tor8 | Robin_Watts: it allocates space in the xref :) | 11:53.57 |
| | I just checked. | 11:53.59 |
| | which I think a longer name like pdf_alloc_object would help let you avoid looking it up | 11:54.19 |
| Robin_Watts | pdf_new_xref_entry ? | 11:54.21 |
| | new_xref_ref ? | 11:54.41 |
| tor8 | Robin_Watts: but it's not *just* a 'new', it also extends the xref and reserves an object number | 11:55.17 |
| Robin_Watts | Right. hence pdf_new_xref_ref | 11:55.32 |
| | It's a new reference in the xref. | 11:55.41 |
| paulgardiner | tor8: I've just rejigged things a bit. I believe it's now everything on my master branch that we want to merge | 11:55.45 |
| Robin_Watts | pdf_alloc_object doesn't say anything about the xref to me; it implies allocating a new object in memory. | 11:56.31 |
| tor8 | Robin_Watts: yeah... I'm still thinking. | 11:56.55 |
| Robin_Watts | pdf_xref_insert_obj ? | 11:57.02 |
| tor8 | I was thinking allocate more as allocate some space in the xref than 'malloc' but you're right | 11:57.25 |
| | pdf_xref_append_new_object but that's getting unwieldy | 11:57.43 |
| Robin_Watts | I follow your logic, but alloc so strongly means "in memory" to me that I'd forget every time. | 11:57.58 |
| tor8 | pdf_extend_xref? | 11:57.59 |
| Robin_Watts | pdf_xref_append ? | 11:58.29 |
| | That implies sticking something on the end of the xref. | 11:58.54 |
| | pdf_extend_xref implies extending the xref (at least allowing more space in), but not necessarily putting anything into that space (to me) | 11:59.26 |
| tor8 | append is probably the best one. but... reference counting, you make a new indirect object as well so it should have one of the magic words in it | 11:59.40 |
| Robin_Watts | pdf_xref_append_ref ? | 12:00.01 |
| tor8 | pdf_new_indirect_by_appending_to_xref if you wanna be like Apple :) | 12:00.16 |
| Robin_Watts | No, with apple, it'd have :'s in it. | 12:00.36 |
| tor8 | Robin_Watts: or MuPDFNewIndirectReferenceByAppendingToPdfCrossReferenceTable() if pre-Cocoa ;) | 12:01.46 |
| | Robin_Watts: hm, pdf_new_ref takes an existing pdf_obj as well to insert. | 12:03.29 |
| | let me mull this a bit more while I read the rest of the patch | 12:03.37 |
| Robin_Watts | OK. It'll pale compared to the rest of it :) | 12:04.12 |
| tor8 | bubble sort -- would it be difficult to change to a qsort call? | 12:04.36 |
| Robin_Watts | Yes. | 12:04.45 |
| | Or at least, not trivial. | 12:05.00 |
| | We'd need to rejig the data structures. | 12:05.12 |
| tor8 | right. then morph it to a shell sort and I'll be happy. | 12:05.18 |
| Robin_Watts | (which may not be a bad thing, they do hurt my head) | 12:05.25 |
| tor8 | well, that can be done in a later patch | 12:05.41 |
| | Robin_Watts: I don't see anything hugely offensive in the patch :) a few scrunchedupnames that should probably be underscored, but nothing I will argue passionately about | 12:07.14 |
| | not considering that pdfclean.c was probably the original offender :) | 12:07.44 |
| Robin_Watts | paulgardiner: Looking at your utility review now... | 12:20.54 |
| paulgardiner | ta | 12:21.12 |
| Robin_Watts | I think you need fz_var(item); fz_var(arr) in pdf_new_rect | 12:21.21 |
| | Or, better, fz_var(item); and move the arr = out of the fz_try | 12:21.56 |
| | similar in pdf_new_matrix | 12:22.12 |
| paulgardiner | Oh yes. Oops! | 12:23.00 |
| Robin_Watts | You rely on vsnprintf too in fz_buffer_printf, and that's C99 only. | 12:23.40 |
| paulgardiner | Ah. What should I do about that? | 12:24.34 |
| Robin_Watts | I don't know. | 12:24.53 |
| | Still reading the patch :) | 12:25.09 |
| paulgardiner | Damn! Relies heavily on being able to test for overlow | 12:25.27 |
| Robin_Watts | The way we use fz_buffer_printf, it can never be called with more than 256 bytes. | 12:25.45 |
| | hence I'd document that it assumes that, and thus be able to avoid the need for vsnprintf. tor8 any thoughts? | 12:26.25 |
| paulgardiner | If it's C99, how come there's two possibilities for the way it reports overflow? You'd think they'd have standardised that. | 12:26.41 |
| Robin_Watts | http://linux.die.net/man/3/snprintf | 12:26.55 |
| sebras | Robin_Watts: I wonder why pdfref doesn't have any examples of the hint stream. oh well. | 12:28.06 |
| Robin_Watts | sebras: I think the hint stream is a complete pain in the ass, and I can't really believe anyone uses it. | 12:28.32 |
| | Anyone know of a "PDF linearisation checker" ? | 12:28.41 |
| | i.e. something that checks that a PDF file is validly linearised? | 12:28.52 |
| | I'd love to just not put a hint stream in, but I fear it wouldn't conform to the standard. | 12:29.22 |
| sebras | Robin_Watts: never heard of such a tool. | 12:29.27 |
| Robin_Watts | I was wondering if the PDF preflight stuff does it - but I've never used it. kens has though I think? | 12:29.51 |
| | tor8, paulgardiner: It looks like pdf_new_stream_indirection is another contender :) | 12:31.16 |
| tor8 | Robin_Watts: you could try and see if acrobat opens it as expected on the web? | 12:31.56 |
| | ie fast first page display before the rest in downloaded | 12:32.07 |
| Robin_Watts | tor8: Yes. I really need to set up a throttled webserver somewhere. | 12:32.17 |
| | (actually, I have have one here somewhere) | 12:32.26 |
| paulgardiner | Robin_Watts: contender for longest function name? | 12:32.27 |
| Robin_Watts | paulgardiner: Contender for doing almost the same thing as the function that tor8 were just arguing over. | 12:32.48 |
| kens | Robin_Watts : I missed the question ? | 12:32.54 |
| | ah linearisation. No, I don't believe it checks it | 12:33.17 |
| Robin_Watts | Can the adobe pdf preflight tester (or any other test you know of) test for linearisation... ah, ok thanks. | 12:33.27 |
| kens | It can check general validity though | 12:33.33 |
| paulgardiner | Robin_Watts: I did wonder if there was an overlap. I tried to allert you, but not in a very clear way. | 12:33.42 |
| kens | which is probably good enough ? | 12:33.42 |
| Robin_Watts | kens: I load the file into acrobat and then close it - if it doesn't offer to save a fixed version, I count it as "generally valid" :) | 12:34.02 |
| kens | :-) Not the same as the preflight cehcker | 12:34.17 |
| Robin_Watts | paulgardiner: Your function sets stmofs to 1 to tell it that it's a stream rather than a dictionary. | 12:34.41 |
| paulgardiner | Robin_Watts: Yes. not ideal | 12:35.02 |
| Robin_Watts | how does that work when it actually wants to read the stream contents ? | 12:35.03 |
| | I use -1 in my code. Clearly an infinitely superior choice :) | 12:35.18 |
| paulgardiner | Damn! I wish I'd thought of -1. :-) | 12:35.53 |
| Robin_Watts | I had to test the stmofs > 0 to be stmofs != 0 in the "stream or dict" test too. | 12:36.24 |
| | but presumably you must do something cunning when things come to read the stream data? | 12:37.22 |
| paulgardiner | That's quite an assumption. I may have just cocked it up, but not enough for the current use case. | 12:38.17 |
| | Do these words make sense: "I ensured that the stream was in the cache so it wouldn't be read again"? | 12:39.31 |
| Robin_Watts | Quite possibly. | 12:39.42 |
| paulgardiner | I've forgotten how it worked now. Do we ever takes stuff out of the stream cache? | 12:40.07 |
| Robin_Watts | I think I prefer the idea of -1 meaning "stream data not in the original file" | 12:40.19 |
| | Do we have a stream cache? | 12:40.44 |
| paulgardiner | Ah. "stream cache" not existing would go a long way towards making that sentence of mine nonsensical | 12:41.53 |
| | I've forgotten how this works, but I think the call to pdf_store_item in pdf_new_xobject is important. | 12:47.06 |
| | I think attempt to read the stream go to the cache first, or the place where pdf_store_item stores | 12:48.18 |
| | No. I'll shut up until I've remembered how this works. | 12:51.52 |
| | Ah yes. First thing pdf_load_xobject does is call pdf_find_item. If that works it goes no further. I rely on that. That may well not be good enough for what you are intending to do. | 12:55.15 |
| tor8 | paulgardiner: if you dig far back in the git history, we had a stream cache for objects in the xref | 12:57.07 |
| | basically we had a fz_buffer *stmbuf attached to the xref entry as well as the stmofs | 12:57.20 |
| | I think we only used it when creating new streams or updating the contents of a stream, not when reading | 12:57.46 |
| | so any subsequent stream reading operations would pick up the stmbuf rather than going to disk | 12:58.04 |
| | we could add that back if it will help the annotation synthesis workings | 12:58.29 |
| | (and maybe the linearisation stuff too, for the hint stream) | 12:58.41 |
| Robin_Watts | tor8: That sounds like a good fit for what I need. | 13:01.01 |
| | paulgardiner: Right, so xobjects go into the store ? | 13:01.39 |
| paulgardiner | yep | 13:02.07 |
| | So it's an xobject cache, or something like that. | 13:02.21 |
| Robin_Watts | God, that's embarassing. | 13:02.27 |
| | I should really remember this stuff given I reworked the store. | 13:02.40 |
| tor8 | Robin_Watts: basically revert parts of d219624c5cfb98556c71cf71d6eed6727846badc | 13:03.09 |
| paulgardiner | He he. No more than I should remember how those few functions work, espectially as it wasn't long ago. :-) | 13:03.25 |
| Robin_Watts | So, the only problem might be if we run low on memory and stuff gets evicted from the store. | 13:03.33 |
| tor8 | -fz_error *pdf_allocobject(pdf_xref *, int *oidp, int *genp); | 13:03.36 |
| | -fz_error *pdf_deleteobject(pdf_xref *, int oid, int gen); | 13:03.36 |
| | -fz_error *pdf_updateobject(pdf_xref *, int oid, int gen, fz_obj *obj); | 13:03.37 |
| | -fz_error *pdf_updatestream(pdf_xref *, int oid, int gen, fz_buffer *stm); | 13:03.37 |
| | pdf_allocobject does what you pdf_new_ref does, but without returning the fz_new_indirect object | 13:04.03 |
| Robin_Watts | (which you can probably avoid by keeping a reference to the xobject) | 13:04.10 |
| | I'm being called for lunch. bbs. | 13:04.45 |
| tor8 | Robin_Watts: paulgardiner: I can take a stab at resurrecting those functions unless you both are halfway there already with various patches. | 13:05.08 |
| paulgardiner | tor8: sounds sensible to me, although I'm not sure I've followed all that's been said. | 13:06.32 |
| tor8 | it means you should update the appearance streams by messing with the xref streams rather than xobjects directly. I think we may need to change the xobject implementation to read the stream directly rather than keep a fz_buffer of the contents though. | 13:07.43 |
| | shouldn't be a problem with the recent changes to pdf_open_stream that robin did that will let us have multiple streams open for reading at the same time | 13:08.03 |
| paulgardiner | Where will the newly created streams be held if not in an fz_buffer? | 13:11.53 |
| | Oh I guess there are buffers associated with the xref. | 13:13.46 |
| | So xobject always refers to its stream via the xref. Yeah, makes sense. | 13:16.27 |
| tor8 | paulgardiner: yeah, it'd store the pdf_obj indirect object instead of the fz_buffer | 13:24.54 |
| | that'll reduce the memory footprint of xobjects in the cache too | 13:25.04 |
| Robin_Watts | tor8: I'm planning to stare at some ghostscript code today, so feel free to resurrect away. | 13:45.20 |
| tor8 | Robin_Watts: pdf_update_object, keep or zap the 'gen' argument? | 13:53.34 |
| Robin_Watts | It's never used. So lose it. | 13:54.19 |
| | Well, it's used in the error message, but who cares ? | 13:54.44 |
| tor8 | indeed. | 13:54.47 |
| henrys`` | sent you mupdf folks some mail about the new customer. | 13:55.11 |
| henrys | the customer is confidential. | 13:56.15 |
| tor8 | Robin_Watts: I also think fz_write and fz_write_options are a bit premature. I'd be happier with them restricted to the pdf namespace for now. | 13:57.19 |
| Robin_Watts | henrys: So, do you want one of us to write an email reply to that ? | 13:58.07 |
| henrys | no I think I pretty well covered it in the meeting. If there is followup I'll hand it over to one of you. | 13:59.01 |
| Robin_Watts | ok. | 13:59.08 |
| henrys | I am curious about this app - rasterizing to printer resolutions on a phone and sending that to a printer? | 14:00.50 |
| tor8 | henrys: did you see bug 693042 from norbert? | 14:06.40 |
| henrys | tor8:I just read it. | 14:06.55 |
| Robin_Watts | Am I going blind, or has marcosw failed to say what files it fails with on bug 693039 ? | 14:10.37 |
| | oh, he shows one, indirectly. | 14:11.27 |
| henrys | tor8:so the right answer is slow I don't think that's a regression but it should be investigated. | 14:11.43 |
| | so 9.04 was correct and faster. | 14:14.15 |
| | tor8:you can assign it to marcosw for bisection. | 14:15.23 |
| tor8 | henrys: done. | 14:17.35 |
| kens | Robin_Watts : marcos does give *one* falinig file | 14:18.10 |
| Robin_Watts | yeah. I thought it was a random (or autogenerated) filename on first glance ;) | 14:18.35 |
| kens | :-) | 14:18.46 |
| henrys | tor8:but you did enable the pdf 1.4 compositor for some patterns in that time frame right? | 14:19.32 |
| tor8 | henrys: oh drats, that's what flying will do for your memory. I made a patch to that effect in Londor, but ran into odd regressions that I forgot to follow up. | 14:21.49 |
| | henrys: michael did another patch which enables the compositor for some pattern (but the detection for when to do it doesn't catch all cases), that might be it | 14:23.04 |
| | that detection is what I tried to figure out in London and then forgot | 14:23.20 |
| Robin_Watts | TTOTD: When using a laptop, hide the mouse for the desktop to save much frustrated clicking. | 14:23.50 |
| chrisl | Robin_Watts: that's the main reason I stopped using a mouse with the laptop...... | 14:24.35 |
| Robin_Watts | I am NOT using a mouse on the laptop :) | 14:24.52 |
| chrisl | Ah, well move the laptop further away, then | 14:25.51 |
| Robin_Watts | The available deskspace in my office is limited :) | 14:26.13 |
| | I'd go and work in the garden, but then it'd hail on me... | 14:26.38 |
| chrisl | :-) | 14:26.46 |
| | henrys: I made the assumption that the non-configure ghostpdl Unix build was no longer supported, so I moved the tiff configure call into the ghostpdl configure script. | 14:27.55 |
| henrys | chrisl:that makes sense. | 14:28.36 |
| chrisl | Good - building without running configure hadn't worked for some time, anyway | 14:29.07 |
| Robin_Watts | forms meeting in 15 mins, right? | 14:45.06 |
| Robin_Watts | fetches tea. | 14:45.23 |
| henrys | yes | 14:45.55 |
| | paulgardiner:seems like a lot of time messing with the v8 api, I wonder if another js would be simpler. | 14:59.58 |
| Robin_Watts | api, or build system ? | 15:00.31 |
| henrys | oh I guess it is mostly the build system, but also memory management might be simpler with another js | 15:01.12 |
| paulgardiner | henrys: I don't get the impression it's particualary complicated as engines go. I'd imagine they all have the same problems to cope with, and it would take some time to understand any one of them. | 15:01.56 |
| Robin_Watts | I thought that *all* the javascript things we'd looked at used C++, so presumably they will all have the same build problems. | 15:01.57 |
| | Where were we having problems with memory management? | 15:02.19 |
| paulgardiner | Wasn't so much problems. It was just a bit intricate. | 15:02.40 |
| henrys | Robin_Watts:second to last paragraph in the report | 15:03.00 |
| paulgardiner | I believe it's all working ok now. | 15:03.08 |
| | I did a few tests and the number of outstanding objects repeatedly came back to 0. | 15:03.37 |
| Robin_Watts | oh, right, yes, the context stuff. | 15:04.30 |
| | That's a javascript context, or a mupdf one ? | 15:04.59 |
| | All MuPDF contexts share the same allocator, so freeing something in one that was allocated in another isn't a problem. | 15:05.53 |
| paulgardiner | Sort of both. I think we have to keep the engine alive over multiple pdf file loads. Each one will need a separate context. | 15:05.59 |
| Robin_Watts | Again, I'm not seeing why that should be a problem. | 15:06.45 |
| paulgardiner | I thought it was wise, when finishing with one pdf file that we make sure that the engine didn't still have ownership of any pdf side things. | 15:07.24 |
| Robin_Watts | Right. Hard to argue against that. | 15:07.46 |
| paulgardiner | You want it to garbage collect the native representative of any js objects, but when you'v e finished with a context, you cannot be sure it has let go of them all. | 15:08.36 |
| | So you have to sort of unlink them. | 15:08.48 |
| | It works quite neatly. It just took me a while to get to understand it. | 15:09.09 |
| | The build problem on the other hand is mysterious. I wonder if it's just some C++ think that's well known, but I haven't come across before. | 15:10.13 |
| | It's not holding me up because I can force the link, but I'd like to get it sorted out eventually. | 15:10.50 |
| | Could be to do with calling C++ from C, possibly. | 15:11.02 |
| henrys | paulgardiner:have you tried platforms other than windows? | 15:11.28 |
| Robin_Watts | The wierdness in linking is the single largest thing that worries me about this work. | 15:11.39 |
| paulgardiner | henrys: No I haven't yet. That's something we should probably look at soon, but I was planning to do any necessary cleaning up first. | 15:12.49 |
| | I knew it had memory leaks, although those are sorted out now. | 15:13.32 |
| henrys | I was thinking the linking problem might be unique to windows. | 15:13.41 |
| Robin_Watts | The fact that it works with their hello world, and not with ours is very odd. | 15:14.04 |
| | I'd be tempted to start with the simplest possible version of an application we can do that causes the problem, and try to migrate it towards their hello world example. | 15:14.45 |
| | And vice versa. | 15:14.47 |
| paulgardiner | henrys: yes it might well be. I probably gave the issue too much weight in the report. I haven't been spending much time on it. I just tried out a suggestion of Robin's and then posted a quewstion on a forum | 15:14.50 |
| | Robin_Watts: Yes, I'd thought that, but I was hoping to get a reply to my forum posting. | 15:15.45 |
| Robin_Watts | How long have you been waiting? | 15:16.08 |
| henrys | tor8:anything for the forms meeting? | 15:16.34 |
| paulgardiner | I did find someone else complaining about the problem, but there were no replies other than people saying that you shduldn't have implementations in h files. | 15:16.50 |
| Robin_Watts | So this is a C++ forum rather than a V8 specific forum ? | 15:17.14 |
| paulgardiner | The thing about trying another platform is that it'll involve some app-side work, and hence wont be a quick test. | 15:17.40 |
| tor8 | yeah, but I would still like to see us (eventually, anyway) try spidermonkey if that one is more portable | 15:18.15 |
| paulgardiner | Robin_Watts: Can't remember where that one was. The question was about v8 | 15:18.20 |
| Robin_Watts | It's hard to argue against us at least trying more than 1 engine. It's not clear to me that this is the best time to schedule that work or not - I guess Paul probably is best placed to decide that. | 15:19.31 |
| paulgardiner | tor8: I had a quick look at SpiderMonkey. Its API certainly looks no less complicated. :-) | 15:19.36 |
| Robin_Watts | I think it's worth giving this linking thing a long hard look (even if it takes a whole week to sort it or to find out that it can't be sorted). The last thing we want to do is to get almost to the end and find there is a fundamental problem that we haven't addressed (though I guess that's unlikely). | 15:20.35 |
| | We could always add some #ifdefs to remove the implementations from the .h file for when it's included externally, right? | 15:21.01 |
| | But it would be a shame to not be using a vanilla V8. | 15:21.18 |
| paulgardiner | Robin_Watts: that may be a possibility, but seeing as the HelloWorld example links that should be unnecessary. | 15:22.02 |
| | I cannot imagine there's any really serious problem here. calc.pdf works fine and that's not a completley trivial test. | 15:22.39 |
| Robin_Watts | Yes. Hence my suggestion that we try migrating a simple app towards hello world, and hello world towards that simple app. | 15:22.40 |
| | presumably we'll hit a point where the problem occurs/goes away and hopefully we'll be able to deduce why. | 15:23.17 |
| | I am aware that could be a slow, painstaking, time consuming process though. | 15:23.43 |
| paulgardiner | Robin_Watts: No, I had considered doing that. I was just hoping that a forum question might make it unnecessary. | 15:24.21 |
| | It might be worth my asking in a VS forum. | 15:24.39 |
| | Or a C++ forum. | 15:24.48 |
| Robin_Watts | Sorry, wasn't criticising there - a forum question is a good approach (but in my experience you either get a useful answer quickly or not at all). | 15:25.19 |
| paulgardiner | Anyway, it's not as though v8 seems to be playing up in general. Mostly it's been far easier than I'd anticipated. | 15:25.47 |
| Robin_Watts | And such slow tasks are not an immediately attractive thing when you're trying to make progress. | 15:26.12 |
| paulgardiner | Quite. | 15:26.31 |
| | No. Wasn't reading it as criticism. | 15:29.11 |
| | I'd be happy to look at another engine, but it might be just a detour on the way to getting something releasable. | 15:30.09 |
| Robin_Watts | We shouldn't release without having looked at another engine. We could *demo* without having looked at another engine though. | 15:31.00 |
| | (or at least we shouldn't release without "This is subject to change" notices plastered on the code) | 15:31.31 |
| | <tumbleweeds> | 15:33.21 |
| paulgardiner | :-) | 15:33.30 |
| | I'd hope that other engines will require very little change outside of the engine wrapper. Admittedly I had to change the API for v8, but it was only one small change. | 15:35.08 |
| Robin_Watts | paulgardiner: Yes, it's a reasonable hope, but until we've at least looked at another engine, it would be hard to be 100% sure. | 15:36.08 |
| henrys | paulgardiner:I didn't find much else in the report meeting stuff - is there anything else you can thing of for the meeting. Do you need anything? | 15:36.20 |
| paulgardiner | Robin_Watts: No true. | 15:36.34 |
| tor8 | Robin_Watts: after having done pdf_write(_document), should we swap over to use the new file for the pdf_document, or do you think that's overkill? | 15:37.37 |
| paulgardiner | henrys: No thanks. I could probably do with some help with prioritising what order to do things in, but I really need to generate a list of what the things are before that's possible. :-) | 15:37.39 |
| tor8 | paulgardiner: Robin_Watts: I have a first (incomplete) stab at resurrecting the mutation api on tor/master. just so you can see what I am aiming for there. | 15:39.00 |
| paulgardiner | If looking at another engine is prioriy, I'm happy to do that. | 15:39.05 |
| henrys | yes a list would be useful - I assume you are okay tool-wise? Something like valgrind would be very useful for something like this I would think - windows equivalents cost $$ | 15:39.26 |
| paulgardiner | henrys: Oh right. I'll keep that in mind. I have MacOS and Linux here too. | 15:40.32 |
| Robin_Watts_ | That only has an effect on writing, so not sure what we'd gain by using it elsewhere. | 15:41.10 |
| paulgardiner | Robin_Watts, tor8 , henrys: Oh there's the question of committing parts of this. I guess from the discussions earlier, I should wait for Tor's resurrection of the mutation api, and use that in place of some of my utility functions. | 15:43.19 |
| tor8 | paulgardiner: I don't think it changes a lot of what you need to do. the pdf_update_xobject call will just call the resurrected api behind the scenes. | 15:44.19 |
| | and it won't conflict until that's in place | 15:44.41 |
| | or well, it won't conflict anyway | 15:44.55 |
| | paulgardiner: pdf_new_stream_indirection is the only one that really needs to change from my reading of your patch | 15:45.59 |
| paulgardiner | tor8: yes, it doesn't look like a huge change. Should this make saving of form data work? | 15:46.12 |
| tor8 | paulgardiner: it should, yes. | 15:46.28 |
| paulgardiner | tor8: magic | 15:46.35 |
| tor8 | if you save the form data in the synthesized annotation streams that you hook up | 15:46.45 |
| | once we update the underlying xref table entry with the fz_buffer containing those, they should get written out with pdf_write_document | 15:47.13 |
| | so pdf_xobject_set_contents will need to call pdf_update_stream with the new appearance stream and then it should save fine | 15:47.51 |
| | I need to make sure the stm-buf entry in the xref table is saved out properly in pdf_write_document and deal with the encryption issues before I merge the commit I posted. | 15:48.40 |
| paulgardiner | Right. I almost understand that! :-) | 15:50.28 |
| henrys | so I need to warm up for the next fun filled meeting, I'll be here if you need me. | 15:51.22 |
| | Robin_Watts:I think I now have everything working on the gl/2 side with your new fill code. The dash pattern is the only thing open, I believe. | 15:55.31 |
| Robin_Watts | henrys: Cool. I'm looking into the memento crashes on macosx. Then I plan to look at the dashes. | 15:56.17 |
| henrys | sounds good. | 15:56.34 |
| | I guess the new meeting time messes up kens plans but meetings in August can't work - too many vacations. | 15:57.24 |
| ray_laptop | what's with the "Some glyphs of the font ___ requires a patented True Type interpreter" ? I thought we use FT with the patent stuff enabled. | 15:58.41 |
| henrys | chrisl's project is to integrate ft with the languages, until then we have the old ft 1 stuff. | 15:59.27 |
| Robin_Watts | henrys: I guess they could work, but would need more warning (people book summer holidays more than 3 months ahead) | 15:59.40 |
| henrys | Oh I guess it is time for the next meeting. | 16:01.21 |
| kens | ray_laptop : not with PCL or XPS yet | 16:02.39 |
| henrys | chrisl:since ray_laptop brought it up how is the ft integration. I have time to help if you think it would be useful. | 16:02.45 |
| chrisl | henrys: thanks, but I'm not quite there yet. I've just started looking at changing how FAPI gets built in, so it's not a psi option anymore...... | 16:03.45 |
| mvrhel | sorry I am late | 16:04.34 |
| chrisl | henrys: TBH, I don't think the pcl integration will be too hard, I'll probably just need you to clear up a few questions when I get to that stage | 16:04.48 |
| henrys | chrisl:getting everything into the graphics library without adding any new functionality would be good... small steps | 16:04.51 |
| chrisl | henrys: yeh, I've just reached the point where I can't remove any more Postscript specific stuff from the FAPI "core" with the build working as it does now | 16:05.30 |
| henrys | the other project I'm kind of worried about is the mupdf parser integration, alexcher any news on that ? | 16:07.17 |
| alexcher | henrys: I have a question. | 16:07.43 |
| | henrys: At what level to we want it to take place. | 16:08.10 |
| henrys | chrisl:will we get email if tor8's mooscript branch is modified? | 16:08.14 |
| chrisl | henrys: I don't know, it was tor8 who setup the notifications | 16:08.48 |
| alexcher | henrys: Should it be a mupdf with gs renderer or just a pdf parser in C. | 16:09.06 |
| henrys | alexcher:the level tor8 established ;-) | 16:09.08 |
| alexcher | henrys: OK, this is mupdf with gs renderer. | 16:09.37 |
| ray_laptop | alexcher: good. I also thought what we wanted was the mupdf C parser with the gs rendering | 16:10.18 |
| henrys | alexcher:I sort of expect this to be a project that no one person is going to do solo - lots of questions and discussion. | 16:10.21 |
| alexcher | So what's wrong with mupdf renderer? | 16:11.27 |
| henrys | tor8:where was mooscript left - did you run a pdf file though it? | 16:11.28 |
| tor8 | henrys: yes, I could run a pdf file through it and get some vector graphics and text out | 16:12.01 |
| henrys | alexcher:nothing | 16:12.04 |
| alexcher | henrys: not yet. There's no makefile. | 16:12.09 |
| henrys | tor8:just high level output? | 16:12.35 |
| Robin_Watts | alexcher: Nothing is wrong with mupdf renderer - it's just we can't get access to all the devices within gs with it. | 16:12.43 |
| tor8 | the fonts all came through as bitmaps to the pdfwrite device | 16:12.49 |
| | but I never had a working build system for it. I just dumped object files in a directory and manually linked them together. | 16:13.04 |
| Robin_Watts | No funky halftoning, no pdfwrite, no pswrite, no pcl writers etc. | 16:13.09 |
| henrys | alexcher:the goal is to have a C parser in ghostscript. | 16:13.11 |
| | alexcher:I thought we all agreed to that. | 16:13.49 |
| tor8 | the way I integrated it was: a main function like gxps that creates a pdf_document to drive the parsing, and a fitz device that calls the ghostscript graphics library | 16:14.19 |
| ray_laptop | tor8: that seems to be sensible -- as Robin_Watts said, what we are striving for is the range of devices on the back end of GS rendering. | 16:15.36 |
| | and we might pick up some performance improvement. | 16:16.05 |
| mvrhel | is marcosw here? | 16:17.29 |
| marcosw | I'm here. | 16:17.36 |
| alexcher | So what code will maintain PDF graphic state? | 16:17.49 |
| mvrhel | Did my email make sense to you. What I wanted to test? | 16:17.50 |
| | marcosw | 16:17.56 |
| henrys | the fitz device confuses me. I was expecting calls into the graphics library directly gs_moveto, gs_lineto etc. | 16:18.07 |
| tor8 | henrys: the fitz device interface is at a higher level than that | 16:19.16 |
| mvrhel | marcosw: at one point you were going to show me what I can do to run timing tests on the clusters myself I thought. Then I don't need to bother you | 16:19.35 |
| marcosw | mvrhel: yes, it makes sense. I presume we just need to test 300 dpi, is it necessary to test both banded and page mode? | 16:19.37 |
| mvrhel | marcosw: just 300dpi is fine | 16:19.52 |
| ray_laptop | henrys: we want the higher level to better support devices that take advantage of 'fill_path', 'stroke_path', etc, | 16:20.04 |
| henrys | mvrhel:best if I download it and step through some simple stuff. | 16:20.09 |
| mvrhel | all the time diff will be during the drawing part | 16:20.11 |
| | henrys: you mean tor8 | 16:20.44 |
| tor8 | henrys: http://git.ghostscript.com/?p=ghostpdl.git;a=blob;f=moo/dev_ghost.c;h=6324f9e2e3915ef386e7bf247d18d2601e88daa1;hb=947ada5438090f2bfe7e29da82e93f62bf7bdff3#l240 for example | 16:20.48 |
| Robin_Watts | henrys: MuPDF is designed so that whatever happens, we call the fitz device. | 16:21.01 |
| henrys | mvrhel: I do mean tor8 sorry. | 16:21.06 |
| marcosw | mvrhel: I did send out an email a while ago with (sort of) complete instructions. | 16:21.09 |
| Robin_Watts | It gets called for things like "fill path", "draw this text" etc. | 16:21.25 |
| mvrhel | ok maybe that is what I am remembering marcosw | 16:21.30 |
| | let me find the email | 16:21.36 |
| henrys | tor8:can you enable notifications on the branch? | 16:21.45 |
| Robin_Watts | So mooscript (presumably) implements a fitz device that converts fitz device calls into gs device calls. | 16:22.05 |
| tor8 | Robin_Watts: yes. | 16:22.16 |
| Robin_Watts | That will mean converting paths from fitz -> gs format etc. | 16:22.19 |
| | tor8: sounds eminently sane to me. | 16:22.41 |
| marcosw | there are probably major steps missing, so I think I should setup this test following the instructions and see how it goes… | 16:22.48 |
| tor8 | henrys: there's only one commit on that branch, and it uses a somewhat bitrotted version of mupdf. if you want me to update it to the latest ghostscript and mupdf versions I can do that. | 16:22.59 |
| mvrhel | marcosw: ok sounds good | 16:23.01 |
| | thank you | 16:23.06 |
| marcosw | I'll should have the test started later today, shouldn't take too long to run. Did you say page mode, banded. orboth? | 16:23.54 |
| tor8 | the mooscript code as it stands tigers and renders plain text okay. no images or gradients or patterns. | 16:24.24 |
| henrys | so you have it so you create say a path in fitz then you convert is to a gs path then it is rendered? | 16:24.47 |
| tor8 | henrys: yes, the pdf parser in mupdf creates a fz_path object; calls the fz_device interface with say fill_path(). | 16:25.30 |
| ray_laptop | marcosw: IMHO, just default BufferSpace MaxBitmap should be fine (it will probably be mostly banded, but that's how many people run it) | 16:25.33 |
| tor8 | the fitz to ghostscript bridge device then looks at the fz_path and calls gs_moveto etc | 16:25.55 |
| | and juggles all the graphics state that needs to be set. | 16:26.10 |
| | the fitz device interface doesn't have a 'gstate' object or anything like that. almost all state is passed explicitly. | 16:26.32 |
| | only clipping and blend groups and patterns have 'state' in the fitz device | 16:26.46 |
| henrys | well then you've just done memory handling and processing for one path twice, are we sure this is going to perform reasonably? | 16:27.21 |
| mvrhel | marcosw: you can run them in page mode. the extra timing will be during the reading clist phase anyway | 16:27.37 |
| marcosw | will do. | 16:27.45 |
| chrisl | tor8: (probably silly question) why do you need to do the conversion of the path when the is passing the move, lineto, curveto in the graphics library anyway? | 16:28.36 |
| Robin_Watts | henrys: The alternative would be to 'bend' mupdf so that instead of an fz_path, it used a gs_path internally. | 16:29.06 |
| tor8 | chrisl: I think you're missing a word or two in that sentence... | 16:29.07 |
| henrys | kens:do you have anything? - almost half past. | 16:29.24 |
| tor8 | chrisl: the fz_path is there primarily as an object to stick in the display list. | 16:29.56 |
| | and maintaining cross call state (like separate moveto lineto calls) in the device gets hairy fast | 16:30.13 |
| Robin_Watts | That might be possible, but it'd make a fork in the process. There is no way I am taking ghostscripts bonkers path representation into mupdf :( | 16:30.31 |
| henrys | Robin_Watts:paths that come close to memory limits aren't unusual, it seems like the implementation would double the memory requirements for paths unless I don't understand something. | 16:30.55 |
| Robin_Watts | henrys: mupdf paths are smaller than gs ones in general. | 16:31.20 |
| chrisl | tor8: so the calls to gs_moveto/gs_lineto etc shouldn't be there? | 16:31.35 |
| tor8 | mupdf paths are pretty much the most minimal representation needed. | 16:31.37 |
| Robin_Watts | but yes, you will end up having twin representations of each in there. | 16:31.47 |
| ray_laptop | rather than calling moveto lineto curveto, converting paths from one to another would be more efficient, wouldn't it ? | 16:31.55 |
| tor8 | chrisl: no, you could go directly from fz_path to gs_path, but I went with the path of least resistance when bashing out the mooscript prototype | 16:32.04 |
| Robin_Watts | ray_laptop: Yes, but that would require a new function to be added to the gs graphics lib. | 16:32.24 |
| tor8 | chrisl: or maybe I'm just not understanding what your question is | 16:32.33 |
| Robin_Watts | If it's an issue, we should just put the mupdf path format into gs and be done with it :) | 16:32.54 |
| chrisl | tor8: what I mean is that currently it might be building the GS path twice | 16:33.04 |
| henrys | maybe alexcher can give us some relative memory and timing performance numbers. | 16:33.13 |
| Robin_Watts | chrisl: perhaps I confused you with what I said earlier. | 16:34.02 |
| alexcher | henrys: OK, I need to do the measurements first. | 16:34.16 |
| chrisl | Robin_Watts: I'm looking at the mooscript code | 16:34.23 |
| Robin_Watts | MuPDF builds itself an fz_path by calling fz_moveto, fz_lineto etc. Then it calls the fitz device with that as a param. | 16:34.57 |
| tor8 | henrys, chrisl: http://git.ghostscript.com/?p=mupdf.git;a=blob;f=fitz/fitz-internal.h#l724 there is the fz_path structure definition. a 'moveto' takes 12 bytes (4 for the command, and 8 for the coords). that could be squeezed further with a separate command array (so only 1 byte per command) | 16:35.00 |
| henrys | alexcher:are you comfortable moving forward with this nebulous project? Just pull off small pieces at a time like the this path issue and move forward, I think everyone is going to have to pitch in on this one. | 16:35.10 |
| ray_laptop | compared to other stuff we store, paths are not likely to stress memory usage -- paths are usually ephemeral and rarely very big. | 16:35.20 |
| Robin_Watts | The fitz device then runs through the fz_path calling gs_moveto/gs_lineto etc and ends up with a gs_path that's equivalent. | 16:35.33 |
| chrisl | Robin_Watts: in the mooscript code the move_to function calls the gs_moveto, and the line_to calls the gs_lineto etc | 16:35.52 |
| tor8 | chrisl: those are for the font building stuff | 16:36.05 |
| henrys | Robin_Watts:I wonder how painful it would be to change the gs representation? | 16:36.25 |
| Robin_Watts | henrys: Me too :) | 16:36.35 |
| tor8 | the real path ones are further down (fz_ghost_fill_path, and fz_ghost_path at the top) | 16:36.46 |
| Robin_Watts | My fear (though I haven't looked into it) is that there are ways in postscript to walk a path. | 16:37.09 |
| henrys | Robin_Watts:ref counting and gc would be the hell of it, I imagine. | 16:37.13 |
| chrisl | Robin_Watts: there is a way to walk a path in PS | 16:37.26 |
| Robin_Watts | chrisl: Right. That could potentially be a pain. | 16:37.51 |
| chrisl | tor8: that's cool, I just wondered if you were ending up creating a path as you go, and then overwriting it at drawing time. | 16:38.07 |
| ray_laptop | I think changing the gs path representation is REALLY hairy, plus it would break the device API that needs gs paths | 16:38.15 |
| Robin_Watts | s/changing the// s/path representation// | 16:38.58 |
| ray_laptop | Robin_Watts: we aren't concerned with PS for the mupdf parser project, right ? | 16:39.00 |
| henrys | it will be interesting to see alexcher's numbers for some real world files we may be worrying about nothing. | 16:39.03 |
| tor8 | chrisl: ah, no. I just walk the fz_path calling gs_moveto/lineto/curveto followed by a gs_fill. | 16:39.19 |
| chrisl | Robin_Watts: it's pretty horrific - you can walk clipped paths :-( | 16:39.33 |
| tor8 | the move_to, line_to functions are doing the same but as freetype callbacks to draw glyphs | 16:39.45 |
| alexcher | henrys: can I have a list of files? | 16:39.46 |
| Robin_Watts | ray_laptop: The idea of putting mupdf paths into gs would be to get gs the benefit of a much streamlined representation. We wouldn't want to end up with 2 types of paths in gs (gs_path and fz_path) | 16:40.01 |
| chrisl | tor8: Okay, i follow you. I would be concerned about performance with large paths, though..... | 16:40.40 |
| ray_laptop | Robin_Watts: but the device API _requires_ a gx_path * | 16:40.50 |
| tor8 | chrisl: yeah, but mupdf already sucks for these huge paths. | 16:40.58 |
| Robin_Watts | Right. We'd be changing that. | 16:41.00 |
| | tor8: It does? | 16:41.11 |
| ray_laptop | Robin_Watts: we CAN'T change the device API | 16:41.30 |
| tor8 | chrisl: Robin_Watts: a bit, yeah. the ones where they put a whole page of line art graphics into one big path. | 16:41.31 |
| ray_laptop | the gxdevcli.h is a 'published' API that all devices can count on. | 16:41.59 |
| Robin_Watts | ray_laptop: The device api relies on an abstract type 'gs_path'. I don't think devices have access to what is within that type. | 16:42.16 |
| tor8 | but I don't know if anyone else deals with them any better. the scan converter has a lot of data to create and sort for those. | 16:42.27 |
| Robin_Watts | They just have a set of functions they can call on it (to iterate down it, for instance) | 16:42.29 |
| chrisl | tor8, Robin_Watts: you often find blue prints and schematics are drawn using immense paths, so it's not something we can ignore :-( | 16:42.49 |
| henrys | I would write a loop in postscript that generated a lot of path data, send it to pdfwrite and use that, also tiger is a pretty good path example - I assume not much is working so you have to use simple files. | 16:43.09 |
| ray_laptop | Robin_Watts: the fill_path (etc.) get gx_path * and gx_clip_path * types | 16:43.10 |
| Robin_Watts | ray_laptop: Right. | 16:43.23 |
| tor8 | chrisl: but I think that performance is more down to the mupdf scan converter being all brute force | 16:43.28 |
| | chrisl: not due to the fz_path representation | 16:43.39 |
| henrys | alexcher:last message was to you. | 16:43.41 |
| Robin_Watts | ray_laptop: Right. but again device users don't have access to the internals of those types. | 16:44.07 |
| alexcher | henrys: I'll try. | 16:44.31 |
| Robin_Watts | I'm not saying it's something to try lightly, but it may be possible. | 16:44.54 |
| chrisl | tor8: no, I rather meant the double path creation (once for fitz and once for gslib) might be an overhead we could do without | 16:44.57 |
| Robin_Watts | If this *is* an issue,we can think about it then. | 16:45.19 |
| ray_laptop | Robin_Watts: But device DEVELOPERS do have access to those types. Recall, that we have lots of devices that are developed by others -- not just our small set | 16:45.26 |
| henrys | alexcher:enough data so get near an error in gs, I belive postscript does a limitcheck at some device dependent value right? | 16:45.39 |
| tor8 | chrisl: yeah, but splitting that up means hell for device implementers to puzzle the paths back together, and blows up the fitz display list memory use for paths immensely. | 16:45.43 |
| Robin_Watts | ray_laptop: I really hope they don't. | 16:45.51 |
| henrys | alexcher:for paths that are too long. | 16:46.07 |
| ray_laptop | Robin_Watts: hope they don't what ?? have access to the gxdevcli.h and all of the object types specified ?? How else can they develop devices | 16:46.30 |
| alexcher | henrys: there's no 64K limit on the path size. | 16:46.41 |
| Robin_Watts | Device developers get passed "a path". And they get access to functions to operate with those paths. | 16:46.48 |
| | They don't (or shouldn't) get the information required to fiddle around inside the path themselves. | 16:47.08 |
| chrisl | tor8: sure, but there are ways we can deal with it that shouldn't hurt either case...... if it proves necessary. | 16:47.08 |
| Robin_Watts | If they do, then that's a real failure of modularity. | 16:47.23 |
| ray_laptop | Robin_Watts: there is no requirement that developers use particular functions to access paths | 16:47.33 |
| kens | I have to go folks, godnight all | 16:47.56 |
| tor8 | nn kens | 16:48.03 |
| Robin_Watts | For instance, I believe they have a function they can call on a path, which will call them back once per segment. | 16:48.03 |
| henrys | bye kens | 16:48.07 |
| Robin_Watts | ray_laptop: Well, if they have their fingers inside the engine, they can expect them to get caught in the fan belt when it moves. | 16:48.31 |
| tor8 | chrisl: yeah. I'd prefer to wait until it's necessary rather than complicate things prematurely :) | 16:48.49 |
| ray_laptop | Robin_Watts: the device API as a published spec is invariant. We can't change existing functions. All we can do is deprecate functions and add new ones that are different. | 16:49.30 |
| henrys | marcosw:will you be able to bisect that problem from norbert? | 16:49.49 |
| chrisl | ray_laptop: obviously we want to maintain a stable API as much as we can, but life moves on, and so must APIs, at some point...... | 16:50.00 |
| Robin_Watts | ray_laptop: Right. The *API* is constant. The internal representation of paths is not (unless it's document somewhere that I haven't seen) | 16:50.48 |
| ray_laptop | chrisl: yes, the API has 'moved', but only as I described -- the internals stop using an existing function and start using a new function | 16:50.57 |
| Robin_Watts | People will still generate their paths in exactly the same way by ca;;ing the same path API. | 16:51.48 |
| chrisl | ray_laptop: so we can *never* change the public API? | 16:52.03 |
| Robin_Watts | If internally it's now powered by steam rather than clockwork, they should never know. | 16:52.07 |
| | but we'll worry about this if it ever becomes a real issue. | 16:53.13 |
| | henrys: marcosw found some memento crashes on macos 64bit. | 16:54.17 |
| marcosw | henrys: yes, I just saw the bug was assigned to me. | 16:54.25 |
| Robin_Watts | and I've tracked it down to gdev_prn_tear_down | 16:54.45 |
| ray_laptop | chrisl: we can't change an existing function (name _or_ parameters) -- otherwise some device that works now might break and we don't "own" all the devices | 16:54.48 |
| henrys | Robin_Watts:sounds like a ray_laptop bug. | 16:55.07 |
| ray_laptop | which bug ? | 16:55.16 |
| marcosw | Robin_Watts: speaking of memento, I'm working on the leak detector but am having trouble testing it. I created some intentional leaks (by calling gs_malloc) and they didn't get reported. Am I doing something wrong? | 16:55.18 |
| Robin_Watts | marcosw: Depends if you're within an allocator that gc's maybe? | 16:55.50 |
| | ray_laptop: http://bugs.ghostscript.com/show_bug.cgi?id=693039 | 16:56.27 |
| ray_laptop | Robin_Watts: thanks | 16:56.32 |
| henrys | ray_laptop:robin_watts said there is a problem with gdev_prn_tear_down | 16:56.36 |
| ray_laptop | does the LEAK DETECTOR work before the 'alloc_restore_all' ??? | 16:57.14 |
| Robin_Watts | In the second part of that function, /* point at the device bitmap, no need to close mem dev */ | 16:57.17 |
| marcosw | I put | 16:57.24 |
| | dummy=gs_malloc(em, 1024, 1, "memento test"); | 16:57.24 |
| chrisl | ray_laptop: as I said, we want to maintain a stable API, but a revamp and cleanup every 5-10 years doesn't seem unreasonable to me - I can't think of another library that maintains that level of compatibility long term. | 16:57.34 |
| marcosw | in gs_lib_init0() | 16:57.43 |
| Robin_Watts | pmemdev->base is coming back as a pointer that's not in any allocated block. | 16:57.58 |
| ray_laptop | chrisl: Note that I don't have a problem with creating new device API functions that use mupdf type paths and just issue a statement in the News that the old calls are no longer used. | 16:59.24 |
| Robin_Watts | wait... hold on a sec, maybe I'm wrong. | 17:00.10 |
| ray_laptop | I don't think there are many devices that implement fill_path etc. There ARE more devices that implement begin_typed_image that take the gx_clip_path * | 17:01.01 |
| henrys | marcosw:also did you notify norbert when ray_laptop fixed the memory problem? I didn't see anything? | 17:01.19 |
| Robin_Watts | ray_laptop: Right. And they call API functions to enumerate/use the gx_clip_path. | 17:01.30 |
| henrys | or maybe it was emeric | 17:01.31 |
| chrisl | ray_laptop: I find it hard to believe that the API today is still compatible with the API as it was in 1990, for example. | 17:01.35 |
| ray_laptop | Robin_Watts: not the begin_typed_image functions | 17:01.48 |
| Robin_Watts | They do NOT reach inside the gx_clip_path and fiddle with bits of it itself. | 17:01.49 |
| ray_laptop | Robin_Watts: sure they do. | 17:01.59 |
| Robin_Watts | really? example please? | 17:02.16 |
| | (no sarcasm, just surprise) | 17:02.30 |
| ray_laptop | cust 532's device is a recent example | 17:02.32 |
| | they don't implement fill_path stroke_path but the _do_ handle begin_typed image and fill_trapezoid | 17:03.07 |
| marcosw | henrys: I usually go through the resolved bugs once or twice a week, but haven't done so since the staff meeting. I'll add doing that to my rapidly growing "do this today or else" list. | 17:03.39 |
| ray_laptop | and HP labs did as well. | 17:03.51 |
| Robin_Watts | Well, as I've said, if they have their fingers on moving bits of the engine, then they deserve to have them caught in the fan belt when we start it up. | 17:04.47 |
| ray_laptop | hmm.. and a MAJOR gotcha. the 'gdevijs.c' implements stroke_path | 17:05.06 |
| chrisl | Robin_Watts: the gx_(clip_)path structure is documented as part of the public API | 17:05.28 |
| ray_laptop | and IJS has several CLOSED SOURCE devices | 17:05.28 |
| Robin_Watts | (Of course, if there isn't an API for accessing the internals of a clip_path, then that's a bad thing. | 17:05.36 |
| henrys | the gx_path type is opaque. | 17:05.48 |
| Robin_Watts | but gx_clip_path isn't ? | 17:06.01 |
| chrisl | henrys: gx_path is listed in Drivers.htm so I would classify that as "public" | 17:07.05 |
| ray_laptop | henrys: how can they be opaque ? How can someone implement an IJS device ? | 17:07.10 |
| henrys | the graphics state is opaque I don't understand what you are saying. opaque means you don't have direct access to the fields in the structure. | 17:08.25 |
| Robin_Watts | chrisl: Right. But the definition of the gx_path structure is in gzpath.h | 17:09.32 |
| henrys | IJS is looking at the internals of gx_path_s? | 17:09.33 |
| Robin_Watts | people can call all the functions in gxpath.h and expect them to still work in new versions. | 17:09.57 |
| ray_laptop | henrys: the graphics state may be opaque, but the gs_imager_state and gx_path and gx_clip_path are not | 17:10.03 |
| Robin_Watts | They cannot access into the structure and expect it to still work. | 17:10.14 |
| ray_laptop | henrys: I don't know -- I just see that see it has the calls | 17:10.36 |
| Robin_Watts | Likewise gx_clip_path_s is defined in gzcpath.h. That isn't part of the public API. Therefore they shouldn't touch the internals. | 17:11.07 |
| ray_laptop | ijs is not an issue -- the external devices are 'dumb' | 17:11.24 |
| chrisl | Robin_Watts: yes, that's how it *should* be, I'm less that sure that's how it is. Personally, I don't see the big deal in changing the API every decade or so...... | 17:11.39 |
| henrys | it sounds to me like there is no need to change the api or did I miss something? We want to consider changing the internal workings of paths. | 17:12.30 |
| Robin_Watts | henrys: Indeed. | 17:12.58 |
| | but let's not worry about this until it actually is proven to be an issue. | 17:13.39 |
| ray_laptop | there are a LOT of places to change that handle paths | 17:13.54 |
| henrys | agreed | 17:13.55 |
| Robin_Watts | oh, god, I'm thick. Found the memento crash. | 17:18.36 |
| ray_laptop | Robin_Watts: so it's not my bug after all ? | 17:22.42 |
| Robin_Watts | ray_laptop: Sorry, no. | 17:22.58 |
| ray_laptop | I'm not sorry :-) | 17:23.07 |
| Robin_Watts | marcosw: So your memory checking problem. You were saying ? | 17:26.17 |
| marcosw | Robin_Watts: that I was trying to find an easy way of generating a leak to see if my code correctly reports it. | 17:26.58 |
| Robin_Watts | So, dumb questions... you're building the memento version? And running the memento version? of gs? or pcl ? | 17:27.32 |
| ray_laptop | marcosw: just run PCL with the gs_imager_state_release commented out in gstate_free_contents ;-) | 17:29.17 |
| | marcosw: that was the cause of norbert's bug that I just fixed | 17:30.14 |
| marcosw | Robin_Watts: I'm building the memento version with the MEMENTO_LEAKONLY option defined. I've been using gs, but based on ray_laptop's comment I'll try the pcl build. | 17:30.37 |
| Robin_Watts | marcosw: I don't see any leaks at all from running tiger.eps through gs (unmodified) | 17:32.19 |
| | so either we aren't leaking, or something is freeing everything. Memento_fin is being called, and there are indeed no allocated blocks at that point. | 17:32.42 |
| marcosw | correct, so I added a call to gs_malloc in gs_lib_init0() but still didn't get any leaks reported. | 17:33.07 |
| Robin_Watts | marcosw: Right. Which makes me suspect gc. | 17:33.24 |
| ray_laptop | Robin_Watts: GS frees all blocks when it exits | 17:33.24 |
| Robin_Watts | ray_laptop: Frees all blocks how? | 17:33.37 |
| ray_laptop | gs_heap_free_all | 17:33.45 |
| Robin_Watts | Is that a magic gc thing? | 17:33.54 |
| | or does gs keep a linked list thing? | 17:34.09 |
| | a linked list through all blocks allocated? | 17:34.19 |
| ray_laptop | Robin_Watts: no, the gsmalloc.c (heap allocator that calls malloc) keeps a list of all blocks and frees them at the end | 17:34.29 |
| Robin_Watts | Right. So marcosw - there's the reason. | 17:34.42 |
| | ray_laptop: If we remove that call (and maybe trigger a gc at that point), we'd get a list of all leaked non-gc able blocks, right? | 17:35.34 |
| | marcosw: That's what we want, right? | 17:35.44 |
| marcosw | ray_laptop's suggestion is of removing his recent memory leak fix for pcl should be good enough for me to test with, so you shouldn't need to spend more time thinking about this. | 17:35.57 |
| Robin_Watts | main/memobj/pcl6 -sDEVICE=ppmraw -o test.ppm tools/owl.pcl reports many leaked blocks for me | 17:36.59 |
| | marcosw: BUT... I can't help feeling that the behaviour of ghostscript is going to be masking any leaks. | 17:37.41 |
| | We might leak memory during the run, causing our usage to get larger and larger and larger, and we'll never detect it because it all gets freed at the end. | 17:38.10 |
| | Making it so that MEMENTO runs don't do that global freeing and instead trigger a gc seems like a sensible move to me. | 17:38.41 |
| | ray_laptop, henrys: Any opinions ? | 17:38.48 |
| marcosw | Robin_Watts: I agree. We need a build that doesn't force a free at the end of the run (or a call to memento_leak_detection() before that call). | 17:39.08 |
| henrys` | my original thought was to just use pcl to catch memory leaks but Robin_Watts does have a point. | 17:41.19 |
| ray_laptop | for PS, the only thing that works (as far as I know) is to run files multiple times in -dJOBSERVER mode with a 'ctrl-d' between each file. Then if the allocation grows, we know we leak. | 17:42.15 |
| | the 'ctrl-d' in JOBSERVER mode does a 'restore' and GC and is intended to isolate jobs from each other, freeing up fonts, etc. that a job allocates | 17:43.10 |
| Robin_Watts | gsapi_delete_instance calls gs_malloc_release | 17:43.51 |
| ray_laptop | otherwise there is some stuff that is INTENTIONALLY kept around | 17:43.52 |
| Robin_Watts | I'm pondering just removing that in MEMENTO builds. | 17:44.06 |
| | ray_laptop: Right. The idea of the automated checker on the cluster is to ensure that the list of stuff that leaks doesn't grow. | 17:44.51 |
| | so having a few blocks that are always there after a job doesn't matter. | 17:45.07 |
| | we want to be able to spot if we go from 64 block leaks to 10000064 :) | 17:45.29 |
| ray_laptop | Robin_Watts: for PS, the list _should_ be consistent as long as the file is run with -dJOBSERVER and we execute a ^D (or false 0 startjob pop) after the file | 17:46.08 |
| Robin_Watts | Is there a magic incantation for "run a garbage collect now" ? | 17:46.32 |
| ray_laptop | the ^D does the 'restore' to the initial 'save' object and runs a GC. | 17:47.12 |
| | to just run a GC do '2 .vmreclaim' | 17:47.54 |
| Robin_Watts | From C :) | 17:48.54 |
| ray_laptop | Robin_Watts: but a GC won't free up objects that were allocated during the job (fonts, other stuff in dicts) | 17:49.38 |
| Robin_Watts | Right. | 17:49.54 |
| | I think it's fair to assume that anything that *can* be GC'd away can't count as a leak. | 17:50.13 |
| | and we're interested in getting a list of the blocks that have leaked. | 17:50.33 |
| | so running a gc then listing all the extant blocks should do that for us, right? | 17:50.46 |
| ray_laptop | Robin_Watts: The GC is only performed by the PS interpreter loop when the GC has been signalled | 17:51.34 |
| Robin_Watts | So there is no cheap and cheerful gs_gc(mem) call? shame. | 17:52.02 |
| ray_laptop | Robin_Watts: without doing the 'restore' to the outermost save level, you cannot assume that stuff laying around is 'leaked' since the restore would have freed it. | 17:52.36 |
| Robin_Watts | ray_laptop: I'm doing this when gs exits. | 17:52.55 |
| | I'd have hoped that we'd have restored all the way at that point! :) | 17:53.20 |
| ray_laptop | Robin_Watts: gs doesn't (usually) do the restore to the outermost level before exiting | 17:53.41 |
| Robin_Watts | can we make it do so? | 17:54.01 |
| | and if we do make it do so, will it have gc'd ? | 17:54.11 |
| ray_laptop | although if you check for leaks after the 'alloc_restore_all' then stuff will have been freed. | 17:55.39 |
| Robin_Watts | I'm checking on a function called from atexit. | 17:56.16 |
| | so will alloc_restore_all have been called by then ? | 17:56.26 |
| ray_laptop | this is called by gs_main_finit and happens before the gs_lib_finit | 17:56.46 |
| Robin_Watts | so, yes. Fab. | 17:56.58 |
| ray_laptop | so checking for leaks before the call to gp_exit in gs_lib_finit _should_ work | 17:57.25 |
| Robin_Watts | ray_laptop: No, I'm checking for leaks on atexit. | 17:57.48 |
| | So I'll nobble gs_heap_free_all to not free all the data blocks in the MEMENTO case. | 17:58.17 |
| ray_laptop | Robin_Watts: that's too late -- then we will have done the 'delete_instance' call that does the gs_heap_free_all | 17:58.20 |
| Robin_Watts | I get 42 leaked blocks. | 17:59.08 |
| ray_laptop | Robin_Watts: if you disable the heap_free_all, right ? | 17:59.32 |
| Robin_Watts | I've nobbled gs_heap_free_all to not free all the data blocks in the MEMENTO case. | 17:59.54 |
| | Still frees the allocator itself, and the monitor. | 18:00.16 |
| ray_laptop | OK, sounds reasonable | 18:00.16 |
| | Robin_Watts: so if you run the same file multiple times, it stays the same, right ? | 18:01.12 |
| Robin_Watts | Multiple runs give identical results. | 18:01.46 |
| | Let me try multiple copies of the file on one invocation. | 18:01.56 |
| ray_laptop | Robin_Watts: multiple runs, or a run with the multiple files in the same run ? | 18:02.22 |
| Robin_Watts | multiple runs. Trying multiple files in the same run now. | 18:02.47 |
| | 56 blocks leaked. | 18:03.00 |
| ray_laptop | Robin_Watts: and what we hope is that we will see that same count for any file, any number of times. | 18:03.16 |
| Robin_Watts | The number of command line args has an effect :) | 18:03.36 |
| ray_laptop | that's not good -- how many times did you run the same file ? | 18:03.36 |
| Robin_Watts | The extra 14 leaked blocks are all arg_copy and argproc | 18:04.57 |
| ray_laptop | for leak checking, I usually run the JOBSERVER mode with ^D between the same file 10 times | 18:04.59 |
| Robin_Watts | So if we clean up the command line handlings cleanup then we'll be stable. | 18:05.25 |
| ray_laptop | it doesn't tell me what leaked -- just if there were leaks | 18:05.27 |
| Robin_Watts | but at any rate it's already fine for what we want. | 18:05.37 |
| ray_laptop | Robin_Watts: OK. | 18:05.42 |
| Robin_Watts | Hmm. We arg_copy, then we call runarg( with that copy. Can we then safely free it? | 18:07.52 |
| | yes, I think we can. | 18:08.53 |
| ray_laptop | Robin_Watts: I think that's OK | 18:09.15 |
| Robin_Watts | Why are we arg_copying at all? | 18:09.27 |
| | runarg copies it again. | 18:09.34 |
| ray_laptop | leaving coffee shop. I'll check logs later.... | 18:16.59 |
| | Robin_Watts: since we generally only run one file at a time during testing, the arg_copy can be ignored (as part of the 42 leaked blocks), right ? | 18:18.01 |
| Robin_Watts | ray_laptop: Right. but *why* do we bother arg_copying? | 18:18.30 |
| ray_laptop | Robin_Watts: I don't know -- Peter may have had a reason (or a case of belt and braces) | 18:19.02 |
| | some of the arg_copy calls are needed to get the values into one of our allocators, but run_arg may not need it | 18:19.51 |
| | since it does copying itself. | 18:20.08 |
| | be back later... | 18:21.51 |
| henrys` | I thought the args could be sent in from the api and may not reside in the operating systems environment for the process. | 18:21.56 |
| Robin_Watts | henrys`: Right. I can understand that we might need to copy args into the heap, but runarg promptly makes a second copy of them. | 18:22.47 |
| henrys` | oh I don't know about that. | 18:25.43 |
| Robin_Watts | I see various icc profiles being loaded and never freed. | 18:42.26 |
| | and 1 monitor per each one of those. | 18:42.34 |
| | Is that expected? | 18:42.39 |
| | 4 in total (grey, rgb, cmyk, and ... ?) | 18:43.07 |
| mvrhel | lab | 18:45.14 |
| Robin_Watts | ok, that makes sense then. | 18:45.33 |
| mvrhel | the profiles live through the life of the icc manager which survives through the life of the imager state | 18:49.21 |
| Robin_Watts | As far as I can tell, they are never freed. | 18:49.36 |
| mvrhel | ray was working on an issue with this I thought | 18:49.37 |
| | henrys` wasnt ray working on this? | 18:49.55 |
| Robin_Watts | (and if that's intentional, fine - I just thought I'd mention it in case they were supposed to get cleaned up and weren't being). | 18:50.18 |
| henrys` | why do they need to be freed? | 18:50.34 |
| mvrhel | there was an issue with the profiles the ht cache and other member variables | 18:50.37 |
| | I think in the case of the profiles, there was an issue with semaphores | 18:50.52 |
| | in windows | 18:50.55 |
| | iirc | 18:51.05 |
| Robin_Watts | henrys: Maybe they don't need to be freed. They aren't freed for me :) | 18:51.25 |
| | If they were freed, they wouldn't show up in the list of leaked blocks, which is nicer, but it's a small thing. | 18:51.50 |
| mvrhel | they really should survive through the live of the document rendering | 18:51.50 |
| | if we are tidy about our shut down then they would be freed | 18:52.11 |
| Robin_Watts | Right, they are absolutely doing that. My leaked blocks is on atexit. | 18:52.14 |
| henrys` | a leak is continuous growth ... IMHO | 18:52.20 |
| mvrhel | henrys yes this is a good definition | 18:52.47 |
| Robin_Watts | right, but Memento can't tell the difference, so it just reports all the blocks sill allocated at the end. | 18:53.07 |
| henrys` | right but marcosw will report the differences and we won't have to look at it each time. | 18:53.50 |
| Robin_Watts | Yes. | 18:53.55 |
| mvrhel | oh that will work nicely | 18:54.15 |
| Robin_Watts | (neither marcosw or I will report 'leaks' in the sense you want - he'll report the CHANGE in leakage each time) | 18:54.24 |
| | So I'll report too much, and him too little. | 18:54.46 |
| mvrhel | why is his too little? | 18:55.22 |
| Robin_Watts | Suppose I have a file that creates a 4 byte block of memory whenever it meets a shade of blue that it likes. | 18:55.51 |
| | We'll run it once, and it'll leak (by henrys definition). | 18:56.08 |
| | When marcosw tests it, every time he'll see the same number of leaked blocks - no change, so no reports. | 18:56.30 |
| | marcosw will catch NEW leaks though, which is the main thing. | 18:57.11 |
| henrys` | I did ask marcosw about running multiple jobs perhaps repeating jobs we never decided about that though. | 18:57.19 |
| mvrhel | yes | 18:57.21 |
| | that is what I would think we would want | 18:57.30 |
| Robin_Watts | henrys: Oh, do gs blah blah fred.ps then gs blah blah fred.ps fred.ps and see if the list of blocks are the same ? | 18:58.11 |
| marcosw | henrys: i figured I'd get it working for the single job case and then we'd discuss how to expand to multiple jobs (either repeated or not). | 18:58.17 |
| Robin_Watts | That'd be cunning (and only be foiled by the blocks that the gs argument handling leaks) | 18:58.43 |
| henrys` | once we get to a zero leak state in the base code that becomes unnecessary assuming that any new allocation is investigated. | 19:00.31 |
| Robin_Watts | OK. I've unleashed some changes on the cluster that should help. If I've broken anything, yell at me. | 19:02.55 |
| mvrhel | henrys: are you here? | 19:42.05 |
| | or henrys` | 19:44.07 |
| henrys` | yes | 19:46.52 |
| mvrhel | so I was looking at the bug 692926 | 19:47.39 |
| | and going back to look at some profiling in 692674 | 19:47.58 |
| | for the case where they did not see improvement which was pdf to ps | 19:48.08 |
| | I see that they are using pswrite | 19:48.15 |
| | not ps2write | 19:48.19 |
| | the pswrite device is going off and rendering the image in their file to rectangles | 19:48.48 |
| | which is using an icc conversion in the process | 19:49.09 |
| | ps2write does none of this business | 19:49.19 |
| | did we not deprecate pswrite henrys`? or am I mistaken | 19:49.45 |
| henrys` | we did | 19:49.50 |
| | but it seems to live on. | 19:50.01 |
| mvrhel | ok. So I should tell them to rerun but with ps2write | 19:50.08 |
| henrys` | I would push back and tell them to move on with ps2write and report a new batch of problems as is their way. | 19:50.32 |
| mvrhel | ok. thanks | 19:50.52 |
| henrys` | I'll talk to chrisl_away about actually removing the device don't know if we can do it by august. | 19:51.44 |
| mvrhel | bbiaw | 20:03.47 |
| henrys` | Robin_Watts:oh about the filling stuff I forgot high level stuff doesn't work yet, do you want to disable the mode for those devices? | 20:14.49 |
| Robin_Watts | High level stuff doesn't work? | 20:15.11 |
| | I thought I'd updated pdfwrite... | 20:15.56 |
| | maybe just enough so it wouldn't crash. | 20:16.03 |
| | Do you have a case where they go more wrong than you'd imagine? | 20:16.25 |
| henrys` | yes the first entry of my current bmpcmp | 20:17.12 |
| Robin_Watts | Right. I'm treating gapto as lineto in pdfwrite then I guess. | 20:18.39 |
| | That's right for fills, but wrong for strokes. | 20:18.58 |
| henrys | I am all for not supporting the mode in high level devices for now, but whatever you think. | 20:21.42 |
| oy | mvrhel: ping | 20:22.29 |
| sebras | tor8: hm... would it be possible to implement a long-click thing in the android version of mupdf to support higlighting and copying text without hampering the existing behaviour? | 21:16.47 |
| | i.e. one measures the time from press to release and if sufficently short it is a normal click and if veeeeery loooong it's a long-click that may let you do box highlighting and copying text..? | 21:17.48 |
| mvrhel | oy: I am here now | 22:25.47 |
| | Forward 1 day (to 2012/05/16)>>> | |
IRC Logs