| <<<Back 1 day (to 2015/11/02) | 20151103 |
mvrhel_laptop | henrys: are you there? | 00:03.18 |
| done for the night | 03:52.37 |
iamaregee1 | Hi , I am trying to convert multipage pdf to png (I would prefer base64 encoded images) | 06:47.34 |
| I am taking input from stdin and redirecting the output to stdout | 06:47.51 |
| but I can't figure out how to get individual images from the Output | 06:48.37 |
| https://gist.github.com/aregee/01e161bd932b77e22109 | 06:48.49 |
| Here what I am trying to do is setup a complete streaming process, where user uploads PDF on frontend and gets Images of all the pages in response | 06:49.41 |
chrisl | iamaregee1: if spotting the PNG header in the stream doesn't work, probably your only workable solution is to write to individual temp PNG files, serve the PNG data from there | 07:55.20 |
iamaregee1 | yes, I guess I am going to resort to that approach only | 07:56.54 |
chrisl | You're other option, if you are storing the uploaded PDF would be to only convert one page at a time - that would then allow you do to pages on demand | 07:58.54 |
| Bearing in mind that in sending a PDF to gs on stdin, gs has to buffer the entire PDF because it can interpret it anyway | 08:00.31 |
| s/because/before | 08:00.41 |
norbertj | chrisl: did you try to compile ghostpcl using the windows sln? I think on plwmain.c #50 an 'e' which shouldn't be there, and 'gserrors.h' is not included. | 08:15.16 |
kens | I built ghostpcl from scratch 32 and 64 bit yesterday using the SHA current at the time and it worked for me on VS 2008 and VS 2015 | 08:15.55 |
| Let me look at plwmain | 08:16.18 |
chrisl | norbertj: with the e_* errors removed commit? | 08:16.19 |
iamaregee1 | @chrisl yes, something like this https://gist.github.com/aregee/2fa24a75907de8dbe2e2 | 08:16.23 |
kens | AH, I haven't upgraded to that SHA yet | 08:16.35 |
norbertj | kens: chrisl: should be plwmainc.c | 08:16.39 |
chrisl | Hmm, sorry, I get I messed that up :-( | 08:17.10 |
kens | i sjust updating master | 08:17.25 |
| Oh yes, that's obviously wrong | 08:18.00 |
chrisl | I'll fix it in a second.... | 08:18.15 |
kens | One reason we need a Windows cluster machine I guess | 08:18.23 |
chrisl | More to the point, a good reason for me to sort out my mouse so it works properly :-( | 08:19.06 |
| Just testing now - if this builds okay, I'll push the commit | 08:23.20 |
| norbertj: thanks for spotting that | 08:24.09 |
kens | Typical VS2015 has overwritten the status for VS 2008, so when I open 2008 all my previously open WIndows haev gone :-( | 08:26.57 |
chrisl | kens: well, you should know better! | 08:29.04 |
| Okay, the Windows builds should be better now.... | 08:29.17 |
kens | I'll go update nd rebuild | 08:29.30 |
| chrisl builds fine for me with that commit | 08:34.14 |
chrisl | Thanks | 08:35.07 |
Robin_Watts | chrisl: You here? | 10:30.02 |
chrisl | yeh | 10:31.35 |
Robin_Watts | I started to look at that bug last night. | 10:33.18 |
| Couple of interesting things: | 10:33.27 |
| It's a shame that a 're' call can end up with a path area that's not a rectangle. | 10:33.41 |
kens | Can it ? | 10:34.17 |
chrisl | Well, that's dependent on the ctm | 10:34.19 |
kens | AH yes, of coruse | 10:34.27 |
Robin_Watts | chrisl: This is with an axis aligned ctm. | 10:34.34 |
chrisl | In this case, I think it's a rounding error | 10:34.42 |
Robin_Watts | It's to do with the use of rlineto rather than lineto, I think. | 10:34.53 |
| and float rounding, as you say. | 10:35.07 |
chrisl | Well, we could resolve that, but again, it would just hide the read issue | 10:35.10 |
| s/read/real | 10:35.17 |
Robin_Watts | Indeed. | 10:35.19 |
| But I think it might be worth doing. | 10:35.28 |
| (after we fix the main problem) | 10:35.35 |
kens | It sounds likely, if its easy, might give alittle performance | 10:35.47 |
| I presume this is 're' from PDF | 10:35.56 |
Robin_Watts | Then, talking about the flattened_iterator__init_line stuff you say that you can't see how it ever adds more than a 1/4 of the length to the active Y list. | 10:36.25 |
chrisl | In these circumstances, yes | 10:36.49 |
kens | Ah, we use rlinteto because its easier to code in PS | 10:36.56 |
Robin_Watts | This code is new to me too, but I think what happens is that it puts a quarter of the line into the list, and sets the repeats to 4. | 10:37.10 |
| The iterator then repeats that quarter 4 times. | 10:37.26 |
| hence we get the full line. | 10:37.31 |
chrisl | Yes, but when we build the active y list, we only process one of the four sections] | 10:37.55 |
Robin_Watts | ok, so you've walked through further than me then. | 10:38.16 |
| I will keep looking. | 10:38.23 |
chrisl | I'll fire up the debugger, and give you a more specific location - but I'm wary about pointing you at a red herring.... | 10:39.01 |
| Robin_Watts: I'm looking in scan_contour in gxfill.c | 10:43.57 |
Robin_Watts | ok... | 10:48.06 |
chrisl | From scan_contour() we to add_y_line(), and add_y_line_aux(), then init_al(). In init_al(), for the !curve case, we call step_al(), which only ever (as far as I can tell) uses the first set of values returned from gx_flattened_iterator__next() | 10:48.17 |
Robin_Watts | right. | 10:48.55 |
| gx_flattened_iterator__next loops 4 times internally, but I haven't understood all this yet. | 10:49.21 |
| I'll walk it through. | 10:49.42 |
chrisl | gx_flattened_iterator__next() doesn't have a loop | 10:49.58 |
| It expects to be called from a loop | 10:50.28 |
| Anyway, that's what I was suspicious about | 10:51.06 |
| It seemed to me that scan_contour() loops calling gx_flattened_iterator__next() until it finds a section of the current path segment that intersects with the current clipping area box, but then the downstream code ignores the possibility that the path may have been segmented to avoid difference overflows | 10:54.00 |
Robin_Watts | Right. | 11:00.51 |
| I was looking at the loop in scan_contour itself. | 11:01.41 |
| I'll add some debug so I can try to understand what's going on. | 11:01.54 |
| Thanks. | 11:01.59 |
chrisl | Yeh, I find the sheer number of times we setup and use a flattened iterator a little depressing | 11:02.51 |
Robin_Watts | OK, I added some debug into scan_contour, getting it to print the line segments it's returned by gx_flattened_iterator__next | 11:11.02 |
| scan_contour: 7ffc17ff,803bbeb8->7ffc17ff,c02bd50a | 11:11.22 |
| scan_contour: 7ffc17ff,c02bd50a->7ffc17ff,1beb5c | 11:11.23 |
| scan_contour: 7ffc17ff,1beb5c->7ffc17ff,400c01ae | 11:11.25 |
| scan_contour: 7ffc17ff,400c01ae->7ffc17ff,7ffc17ff | 11:11.26 |
chrisl | I *think* those should be sane | 11:11.41 |
Robin_Watts | That looks like 4 sub sections of a single subdivided line to me. | 11:11.54 |
chrisl | Yeh, so I think that is working as intended | 11:12.30 |
Robin_Watts | Which is a shame, cos it means the bug is 'somewhere' in this code (waves hand vaguely) rather than 'there' in this code :( | 11:13.13 |
chrisl | So, one of the subsections actually ends up getting added to the y line list | 11:14.11 |
| By add_y_line() | 11:14.31 |
| But from add_y_line()/init_al()/step_al() only uses the first subdivision from the path segment | 11:17.20 |
Robin_Watts | falls in. | 11:24.01 |
| I see what you mean. | 11:24.05 |
| We loop around the gx_flattened_iterator__next(p.fi) calls, and get 4 subsections of the line out, but only the last one counts as having been 'sufficiently flattened' for the loop to exit and for it to then be processed by the rest of the code. | 11:25.13 |
chrisl | Oh, I thought it was the dir != 2 that gave the exit condition, but my memory may be faulty | 11:26.11 |
Robin_Watts | The first loop in scan_contour only exits when the 'code' returned by gx_flattened_iterator__next returns 0. | 11:28.55 |
chrisl | Isn't the top loop only checking for only_horizontal? | 11:30.00 |
Robin_Watts | __next returns 'true' whenever there is more iteration to do on a given curve segment. | 11:30.13 |
| ok, yes, but I think it's the same issue further down too. | 11:30.49 |
chrisl | For me, it's carrying on when p.more_flattened = 1 and p.dir = 1 | 11:32.00 |
Robin_Watts | I have p.moreflattened = 0 && p.dir == 2 | 11:32.25 |
chrisl | In the second loop through the path segments? | 11:33.12 |
Robin_Watts | __next has now been called 8 times. 4 in the loop before the for (pseg loop, 4 in the loop inside the for(pseg loop. | 11:34.24 |
chrisl | Ah, yes, but we won't call add_y_line() on that pass | 11:35.43 |
Robin_Watts | I'm confused by this stuff. | 11:36.39 |
chrisl | Well, that makes me feel a little better...... | 11:36.52 |
Robin_Watts | The gx_flattened_iterator stuff is a cunning way of walking down a path segment using differencing. | 11:37.17 |
| It avoids having to explicitly expand the path segments into a list of lines, and also avoids having to use recursion to do it. | 11:37.44 |
chrisl | Yep, and then the subdivision stuff is intended to prevent the differencing causing an overflow | 11:38.02 |
Robin_Watts | Urm... ignore the current subdivision thing for now. | 11:38.31 |
| That's a 'cunning' reuse I reckon. | 11:38.41 |
chrisl | So, I was going to suggest hacking the code so it doesn't do the subdivision, and walk through that | 11:39.14 |
Robin_Watts | I reckon in normal use (if I'm understanding this correctly), we look at the curve, and 'guess' a number of subdivisions - that number being the number of subdivisions required to get the required flatness. | 11:39.29 |
| Then the curve is iterated through, and it'll take that number of 'next' calls to enumerate out the subdivided lines. | 11:40.10 |
chrisl | And then we overload that to handle extreme paths | 11:40.23 |
Robin_Watts | Hence the working with subdivided curves is absolutely vital. | 11:40.35 |
| Right, but ignore the extreme path case for now. | 11:40.45 |
| Think about this routine being called on just a 'normal' curve. | 11:41.06 |
chrisl | Well, that's what I mean - nobble it, so it doesn't treat this as a extreme path | 11:41.06 |
Robin_Watts | On just a normal curve, every returned section needs to be handled. | 11:41.36 |
| hence I can't see how it's correct for the code to loop around waiting for p.more_flattened to be 0. | 11:42.25 |
chrisl | It's looping waiting for *either* that or p.dir != 2 | 11:43.06 |
| p.dir == 2 (I think) indicates that segment lies entirely outside the current clip | 11:43.36 |
Robin_Watts | Ah! | 11:43.59 |
| Stupid magic numbers. | 11:44.11 |
| Yes. | 11:44.14 |
chrisl | Yeh, I think we should fix that magic number | 11:44.33 |
Robin_Watts | ok, let me keep walking. | 11:44.51 |
| Ok, so when we're walking the vertical section from (8032639a,7ffc17ff) -> (8032639a, 8xxxxxxxx), it does indeed exit that loop with more_flattned=1 and dir = -1 | 11:56.37 |
| The subsegment in question being: 80326e9a,400c01ad->80326e9a,1beb5b | 11:57.14 |
| As far as I can tell then, we don't call add_y_line or anything on that subsegment. We end up in the if (p.dir = DOWN || HORIZONTAL) section at the bottom. | 12:01.17 |
chrisl | Yes, we'll go through add_y_line a bit later | 12:02.22 |
Robin_Watts | That code seems to swap p and q. | 12:05.08 |
| What the hell is this doing? | 12:05.13 |
chrisl | Honestly, I would happily tell you if I'd fathomed that out | 12:05.47 |
Robin_Watts | chrisl; 2 WIP commits on robin/master for your opinion if you're still interested in this problem. | 14:11.24 |
| Slight rejigs of the code to make it nicer (IMHO). | 14:11.41 |
| Specifically it avoids splitting the contour setup between the caller of scan_contour and the implementation. | 14:12.17 |
kens | meeting time ? | 14:31.54 |
| http://www.reuters.com/article/2015/11/03/us-britain-usa-tourism-idUSKCN0SS14320151103 | 14:32.43 |
| Finally | 14:32.47 |
henrys | kens: in an hour | 14:35.29 |
kens | Ah, my mistake | 14:35.39 |
henrys | kens: great news about travel I hope you guys can get registered soon. | 14:36.26 |
kens | Notice, no actual start date :-( | 14:36.39 |
| But yeah it would be good | 14:36.49 |
henrys | jogux: do you know if pdfkit is using their own pdf or have they possibly taken the android bsd licensed code derived from foxit? | 15:18.45 |
jogux | henrys: it sounded like they'd developed their own engine in house | 15:19.06 |
henrys | s/pdfkit/pspdfkit | 15:20.31 |
jogux | "However, as we expanded to Android, we came to the realization that we had to develop our own renderer. As many of you know, Android has only shipped with a native renderer since 4.4, and their renderer is far too limited for the robust features we offer. So we sat down and worked hard to develop our own renderer." | 15:20.51 |
| 'PDF is our core business and we have over 20 engineers working full-time on it.' | 15:21.37 |
| I think the 20 is about their whole product, not just the renderer. but still, that's a lot of people. | 15:21.58 |
tor8 | henrys: I'm getting frustrated with the lack of man power on mupdf. | 15:23.15 |
henrys | jogux: when we looked at them last we found several web site claims suffering inflation ... I thought you might have inside knowledge perhaps not from the web site. | 15:23.36 |
tor8 | pspdfkit are going to eat mupdf's business for lunch... they're making an actual usable library with these interactive features | 15:23.42 |
| where we've got half finished prototypes that barely work for our android app, and are in no way suitable for use as a general purpose library | 15:24.12 |
| and nobody with time to work on finishing and polishing them | 15:24.30 |
henrys | tor8: so we'll start the meeting early ;-) | 15:24.58 |
jogux | henrys: Everything I know it from the website or other articles from Peter Steinburger, or from playing with the sample apps. | 15:25.23 |
tor8 | and the only attention mupdf gets is with gsview ... the core features needed to support the interactive features not so much. | 15:25.25 |
Robin_Watts | It is hard not to agree with tor8 on this. | 15:26.06 |
| mupdf is basically back to being just tor8 working on it. | 15:26.17 |
tor8 | large tracts of mupdf core is now massively complicated to deal with interactive editing etc | 15:26.27 |
henrys | first off welcome fredross-perry! This meeting is every tuesday at 7:30 and please come to the skype meeting at 8:00. | 15:26.46 |
fredross-perry | As always! | 15:26.56 |
jogux | tor8: Yeah, the iOS app is in a similar state. I think I've been trying to say this for a while, I quite agree. | 15:27.10 |
mvrhel_laptop | I do feel like mupdf core could use some manpower too. | 15:27.50 |
tor8 | the way I see it, we've got two options here: allocate more man power and work on finishing the interactive features we've already started on. | 15:27.55 |
| or give up interactive features, and go back to focusing on rendering only. | 15:28.34 |
mvrhel_laptop | I am at a point in gsview where we need to do some work on mupdf | 15:28.53 |
henrys | tor8: we can't back out of interactive features. | 15:29.01 |
tor8 | then we need more people working on mupdf. and we don't have time for XFA for quite some time. I've been reading the spec last week. it's dreadful. | 15:29.25 |
Robin_Watts | MuPDF rendering is 'good enough'. I reckon. | 15:29.29 |
| We don't get transparency correct, but we're not too far off. | 15:29.42 |
tor8 | Robin_Watts: with 'rendering' I'm also thinking about all the format conversion stuff we've started | 15:30.03 |
kens2 | Or transfer functions | 15:30.06 |
mvrhel_laptop | or color... | 15:30.10 |
marcosw | morning | 15:30.12 |
Robin_Watts | And overall we are as close to correct with other files as any other renderer. | 15:30.15 |
tor8 | with pdfwrite, xps and svg output. the text extraction pdf2html for reflow, etc. | 15:30.19 |
Robin_Watts | kens, mvrhel: absolutely. | 15:30.25 |
| OK, format conversions are a separate thing I reckon, but yes, that area needs work. | 15:30.42 |
tor8 | there are about a half dozen to a dozen unfinished areas where people have started trying to use it only to be told "no, that's unfinished. come back in a decade." | 15:31.19 |
Robin_Watts | tor8: Can't argue with that. | 15:32.03 |
henrys | I was hoping to keep Robin_Watts on SOT for a bit to accelerate Paul's project. What if mvrhel went full time mupdf? | 15:32.37 |
mvrhel_laptop | I would be happy to do that | 15:33.08 |
henrys | and fredross-perry split time SOT/NUI ? | 15:33.14 |
| and fredross-perry split time MuPDF/NUI ? | 15:33.28 |
| sorry | 15:33.29 |
fredross-perry | I could | 15:33.32 |
mvrhel_laptop | gsview is sort of slowing down a bit due to the limitation of mupdf anyway | 15:33.52 |
fredross-perry | mvrhel_laptop: âsplain? | 15:34.09 |
mvrhel_laptop | I had spent the last week hacking at my huge list of gs bugs | 15:34.19 |
kens2 | ?me has been pleased to see a number of those closed | 15:34.38 |
mvrhel_laptop | fredross-perry: It would be nice to get in the text editing, image addition, etc that Acrobat pro does | 15:35.04 |
henrys | that would also have the advantage of getting mvrhel_laptop more into mupdf generally, which I think is a good thing... I realize moving Robin_Watts to MuPDF would probably get things going quicker but long term this seems like a better play to me. | 15:35.22 |
mvrhel_laptop | kens2: yes some of those had been sitting around a bit too long | 15:35.30 |
| I little embarrassing | 15:35.38 |
| s/I/A | 15:35.44 |
henrys | tor8: can we have a list of things that you think should be worked on and we'll start from that? | 15:36.38 |
| tor8: perhaps making a list like Paul's list on the Wiki for NUI? | 15:37.04 |
tor8 | henrys: I've been writing a document with all unfinished in-need-of-work tasks | 15:37.28 |
| I could put that on the wiki I guess | 15:37.41 |
| some of these tasks are fairly isolated and could be handed to mvrhel to help him get up to speed with mupdf internals | 15:38.11 |
jogux | tor8: if we want to go head-to-head with pspdfkit there's a whole bunch of extra things we'd need too? | 15:38.21 |
henrys | chrisl: I've fallen behind on language switch, distracted by other nonsense, and the damn reality of the display device. I know kens2 bolted that in hurriedly but it really is a wrench in the works. | 15:38.25 |
tor8 | jogux: quite possibly. would be good to have a list of targets. | 15:38.35 |
| I'm assuming they have nice UI components clients can just drop into the apps? | 15:38.59 |
rayjj | I had thought that transfer functions were a customer issue for mupdf, but the bug tracker doesn't show it as such | 15:39.01 |
jogux | tor8: just to pick one, pspdfkit have sample apps (plural) for each platform | 15:39.12 |
chrisl | henrys: really? Hmm, I wouldn't have thought it too bad. Maybe we need to refactor how it's integrated with the PCL/XPS code | 15:39.27 |
tor8 | jogux: I've been wanting a java-based android viewer component for a while, where the JNI stuff is limited to mirroring the core api | 15:39.49 |
henrys | chrisl: I'm doing that yes, it's not a huge deal. | 15:40.01 |
kens2 | I thought it was broadly the same as the way it works in the Ghostscript code. But I don't recall now how it works | 15:40.01 |
jogux | tor8: and yeah, it's all nicely packaged components, in iOS they have static and dynamic frameworks, the user just drops the one they want into their xcode project. | 15:40.04 |
Robin_Watts | Having a JNI interface to the core is something I've wanted for ages. Even started on it, but never had time to finish it. | 15:40.16 |
tor8 | it would let clients use the core api from java, so all the people hacking the android core class could get what they want easier | 15:40.20 |
| Robin_Watts: I think that should get high priority for the 1.9 release in spring, if at all possible. | 15:40.42 |
jogux | tor8: and http://stackoverflow.com/questions/33409913/how-to-integrate-mupdf-in-android-studio-step-by-step really should be answereable in a single sentence :-) | 15:40.56 |
Robin_Watts | Absolutely. And people should be able to drive the device interface as a 'MuCanvas' so they can create pages. | 15:41.00 |
chrisl | henrys: we may want to change how it works entirely - IIRC, it rather goes around the side of the actual API which is icky | 15:41.04 |
tor8 | annotation creation should be exposed to a nice friendly set of calls the clients can use in the core library | 15:41.11 |
| Robin_Watts: yes. but that needs work on the pdf output device. | 15:41.29 |
jogux | tor8: plus the mobile UIs need a good tidy up | 15:41.49 |
tor8 | Robin_Watts: but still definitely something that should be able to do from java | 15:41.49 |
Robin_Watts | tor8: Indeed. Lots of these tasks are interelated. | 15:41.52 |
tor8 | having looked into XFA, I think these really do need a higher priority than tackling that beast. | 15:42.18 |
Robin_Watts | jogux: Yes, the point of the JNI interface is that it'd be exposed in a sane way to java. | 15:42.23 |
| I see XFA as something that'll eat time and not get us any new customers. | 15:42.49 |
jogux | Robin_Watts: Indeed - that's only a small part of the issue though; how it's packaged is really important. | 15:42.51 |
tor8 | with a new sane JNI interface, writing a sample android demo app should be trivial without having to do any native development on the app side | 15:43.03 |
Robin_Watts | jogux: Absolutely. Having an up to date android sample lib is really desirable. | 15:43.20 |
henrys | rayjj: I have a sales call this week with miles and scott to customer #535 and #534 - do you understand the specifics of either of those workflows beyond they are using pdfwrite? | 15:44.01 |
jogux | Robin_Watts: Indeed. | 15:44.16 |
henrys | tor8: make the list get on the twiki and we'll prioritize it. | 15:44.52 |
| tor8: what is the release state? | 15:45.02 |
rayjj | henrys: I thought they used ps2write to convert PDF's to PS for their printers | 15:45.07 |
kens2 | thnks that makes more sense | 15:45.22 |
jogux | is tor8 writing all this down? :) | 15:45.25 |
rayjj | but I haven't had any contact with them since Nori left | 15:45.36 |
henrys | rayjj: oh right yes, and I just noticed they are a mupdf customer. | 15:45.55 |
tor8 | henrys: release candidate is up, but haven't heard much noise | 15:46.01 |
jogux | huh. not sure what happened then. | 15:46.18 |
tor8 | ran into trouble building the 64-bit windows (msvc project file stuff I'm woefully inexperienced with) | 15:46.23 |
rayjj | henrys: well, mupdf doesn't fit the printer workflow at all, so I don't know what they do with that | 15:46.38 |
| but it's a big collection of companies, so it could be a totally different division | 15:47.13 |
| henrys: and I thought one division was doing a host 'document management' product. That _might_ make sense for mupdf to create thumbnails or something | 15:48.47 |
henrys | jogux: by the way I really want to have "contact sales" work just like pspdfkit.com - that little questionaire can be hooked up to a database. Ted is supposed to be looking at that but he seems to be swamped with other stuff right now. | 15:49.15 |
jogux | henrys: yeah... we should have something similar for SO too (I think Ted is working on the SO one at the moment) | 15:50.02 |
rayjj | thinks it's sort of backwards to expect engineers to keep track of what the customers are doing with the product. That's usually a marketing/product management function | 15:50.03 |
henrys | marcosw: anything for the meeting? | 15:50.30 |
jogux | I really think it would be helpful if we had some kind of customer database that useful info about a customer could be dumped into | 15:50.38 |
| I'd suggest a wiki page but I can't imagine marketing ever using such much, and something better structured would probably be beneficial anyhow. | 15:51.32 |
rayjj | particularly since the 'inquiry' they fill out that says what they want to do with the product very rarely makes it past Scott or Miles | 15:51.32 |
mvrhel_laptop | tor8: let me know when you want to talk about what I should get started on | 15:52.27 |
marcosw | henrys: not really, I'm going through the windows regression results and entering bugs (though I think I might be done, at least for the 64 bit build). | 15:52.31 |
henrys | rayjj: actually the lack of transparency is exactly what I don't like, I'm even on the sales mailing list I still can't figure out what's going on because Scott won't CC: me . | 15:52.32 |
Robin_Watts | I don't believe the inquiry every actually helps. | 15:52.32 |
| It's utterly terrifying for most people. | 15:52.57 |
marcosw | chrisl: I did want to ask chris about http://bugs.ghostscript.com/show_bug.cgi?id=693011 | 15:52.58 |
chrisl | marcosw: working on it.... been distracted by other things | 15:53.22 |
marcosw | chrisl: no problem; I was just thinking I should either add a -dLastPage=1 to the command or add a %d to the output filename and then collect the pages. Can't decide which is better. | 15:54.26 |
| currently we test the last page, so that's clearly not ideal. | 15:54.45 |
henrys | I didn't have anything else for the meeting, more on skype if anyone wants to join beyond the SOT regulars, that'd be great. see you in 5 minutes | 15:54.58 |
chrisl | marcosw: I was thinking about adding an exception that piped output wouldn't throw the error | 15:55.03 |
tor8 | http://twiki.ghostscript.com/do/view/MuPDF/Tasks | 15:55.08 |
chrisl | marcosw: essentially on the assumption that if someone wants to stream psd output, they figure their own way to identify where new files begin | 15:56.04 |
rayjj | Robin_Watts: The few inquiries I've seen do have more information than we usually get. At least they describe the product that will use our stuff | 15:56.12 |
tor8 | mvrhel_laptop: where do you want to start? :) | 15:56.26 |
marcosw | chrisl: you mean piped to stdout? sending to a named pipe (i.e.-sOutputFile="| gzip >test.psd.gz") doesn't work, you get the last page. | 15:56.45 |
mvrhel_laptop | well.... | 15:56.51 |
chrisl | marcosw: that makes not sense at all :-( | 15:57.25 |
marcosw | yeah, I know. | 15:57.31 |
chrisl | We can't suck back the data we've already written | 15:57.54 |
tor8 | mvrhel_laptop: I think one of the core features we need to solve sooner or later will be related to the pdf editing interfaces | 15:57.54 |
mvrhel_laptop | tor8: yes. that one interests me. | 15:58.33 |
tor8 | more specifically, we should be able to create blank pdf documents and start filling them with pages and annotations from scratch | 15:58.35 |
mvrhel_laptop | right | 15:58.42 |
tor8 | and copy and delete and move pages around, etc | 15:58.44 |
mvrhel_laptop | yes. | 15:58.56 |
tor8 | so page tree editing, making sure annotations and bookmarks don't get confused, etc | 15:58.57 |
mvrhel_laptop | the ability to take a scanned form (image) and drop in form filling annotations would be nice | 15:59.14 |
tor8 | now I see three ways to create a page: manually with printf-like stuff, manually using pdf_op_xxx calls to call operators to fill out the page (the pdf_processor interface the interpreter calls), and using a pdf write like device that robin started | 15:59.52 |
| but all those three ways at the bottom terminate in having a fz_buffer with data and a list of resources used that need to be wrapped into a page and stuffed into the page tree and xref | 16:00.28 |
Robin_Watts | tor8: yeah, the pdf writing device works enough for appearance stream synthesis, I think. | 16:00.29 |
marcosw | chrisl: I'm 99% sure it works that way. I was presuming that ghostscript was closing and reopening the output device and that was causing the first N pages to be lost. | 16:00.31 |
tor8 | and those bits at the bottom are what need some good polishing in order to make them usable for customers | 16:01.02 |
chrisl | marcosw: that would be true of writing to a file, but not to a stream, once we've written to the stream, we cannot overwrite what's already been sent | 16:01.17 |
tor8 | we've got all the core bits, but they're a bit finicky and fragile to put together | 16:01.18 |
| having an api to create a document, create a blank page, fill it with drawing and resources, and saving it back out would be a good start for what we'll need | 16:02.03 |
mvrhel_laptop | tor8: that sounds reasonable | 16:02.19 |
marcosw | chrisl: I don't disagree with you, but it's doesn't match wjat O | 16:02.40 |
| what I'm seeing. | 16:02.48 |
tor8 | this is something that we should then wrap up in the JNI interface, and be able to use to create annotations and possibly edit page graphics with etc | 16:02.49 |
mvrhel_laptop | right | 16:03.02 |
marcosw | I'll double check and update the bug. | 16:03.08 |
tor8 | mvrhel: I suspect if you start by writing a new command line tool to do some of this, you'll see what needs work and what works currently | 16:03.44 |
chrisl | marcosw: I'll grab the test file and have a check | 16:03.58 |
tor8 | maybe something that will take an input file with a content stream and wrap that up in a new pdf | 16:04.18 |
| mvrhel_laptop: the functions we have for use are in include/mupdf/pdf/xref.h, pdf_create_object, pdf_update_object, pdf_update_stream but those are very low level | 16:05.56 |
marcosw | chrisl: confirmed, the last page is the one that is saved. | 16:06.43 |
mvrhel_laptop | tor8: ok thanks. I will look this over and probably have a pile of questions. | 16:06.43 |
chrisl | marcosw: I have to disagree with you there...... | 16:07.04 |
marcosw | chrisl: hold on... | 16:07.19 |
tor8 | mvrhel: source/pdf/pdf-write.c has a lot of stuff in it that might need extracting into parts | 16:07.23 |
| like the page tree rebalancing etc | 16:07.27 |
chrisl | marcosw: If I write the entire file using -sOutputFile="| gzip >test.psd.gz", I get a 1.8Mb gzip file. If I do -dFirstPage=8 I get a 182k gzip file | 16:08.08 |
tor8 | pdf_insert_page in source/pdf/pdf-page.c exists, but I'm not sure how robust it is | 16:08.11 |
| I think I saw somebody talking about using it to copy a page from one pdf_document to another .. .but that's *never* going to work so maybe we need to fix the interface so people don't try it | 16:08.42 |
| or fix it so that it will work, by copying all the resources along | 16:09.13 |
mvrhel_laptop | right | 16:09.18 |
tor8 | I think the pdf-page.c stuff is what needs a proper polishing so that it will work as a general purpose library, not just for our current slim needs | 16:09.48 |
| pdf_create_page is currently only used by mudraw in the pdfwrite mode | 16:10.44 |
mvrhel_laptop | I was just looking at the function | 16:11.08 |
| that | 16:11.10 |
marcosw | chrisl: that's not what I'm showing. | 16:11.29 |
tor8 | anyway, you should have enough code to read for a while now :) I've got to go out for a few hours, will pop back later if you have any questions | 16:11.37 |
chrisl | marcosw: are you using Brandypot.pdf? | 16:11.55 |
mvrhel_laptop | tor8: I will have questions. Just not sure I will be ready to ask them in a few hours... | 16:12.04 |
marcosw | No, BFAZCFPB.PDF | 16:12.23 |
| ./gs/bin/gs -I./gs/lib -sOutputFile='| gzip >./temp.psd.gz' -sDEVICE=psdcmyk -r72 -dNOPAUSE -dBATCH ./tests_private/comparefiles/BFAZCFPB.PDF | 16:12.29 |
mvrhel_laptop | working on the command line application to do what you said makes a lot of sense to get started | 16:12.30 |
marcosw | vs | 16:12.39 |
| ./gs/bin/gs -I./gs/lib -sOutputFile=temp.%d.psd -sDEVICE=psdcmyk -r72 -dNOPAUSE -dBATCH ./tests_private/comparefiles/BFAZCFPB.PDF | 16:12.39 |
| and then 'gzip temp.*.psd' | 16:12.48 |
| 128 -rw-rw-r-- 1 marcos marcos 127058 Nov 3 08:10 temp.1.psd.gz | 16:13.03 |
| 136 -rw-rw-r-- 1 marcos marcos 136733 Nov 3 08:10 temp.2.psd.gz | 16:13.03 |
| 160 -rw-rw-r-- 1 marcos marcos 162751 Nov 3 08:10 temp.3.psd.gz | 16:13.04 |
| 400 -rw-rw-r-- 1 marcos marcos 407569 Nov 3 08:10 temp.4.psd.gz | 16:13.04 |
| 400 -rw-rw-r-- 1 marcos marcos 407558 Nov 3 08:12 temp.psd.gz | 16:13.04 |
mvrhel_laptop | This will help me get more familiar with this part of the code and hopefully make it clear what needs to be done | 16:13.06 |
| rayjj: did you get a chance to look at the open embedded stuff? | 16:13.19 |
marcosw | note that temp.psd.gz is the same size as temp.4.psd.gz (sort of, I think the temp4.psd.gz is larger because gzip stores the original file name and maybe the creation date/time). | 16:14.05 |
chrisl | Hmm, yes, that's very strange...... | 16:14.55 |
marcosw | chrisl: yeah. | 16:15.21 |
chrisl | I also get the same size of output if I use -dFIXEDMEDIA which should prevent the device being closed and reopened..... | 16:16.16 |
marcosw | chrisl: I don't think we should write multi-page psd files at all. Produce an error if %d isn't in the output filename when asked to write the second page. While you are at it fix psdrgb to act that way too, writing out files that don't meet the spec can't be right. | 16:19.09 |
chrisl | marcosw: psdcmyk and psdrgb share most of their code, so changing one will change the other, the odd man out is psdcmykog - but not a big problem | 16:20.09 |
rayjj | mvrhel: sorry, I was off looking at the openembedded stuff :-/ | 16:20.29 |
marcosw | chrisl: so that leads back to my original question, should the cluster use -dLastPage=1 with psdcmyk[og] files so that it only the first page is tested or do some messy stuff with -sOutputFile=temp.%d.psd? I'm voting for the former, since I think most of the interesting color stuff happens in specially constructed single page test files. | 16:25.57 |
chrisl | marcosw: Hmm, you're going to hate me, but I think the latter - many of the test suite files that have taxing color tests are multipage files :-( | 16:27.12 |
henrys | marcosw: oh I forgot to bring up the windows segfaults a couple of those looked bad, maybe P1's? | 16:27.56 |
marcosw | henrys: I didn't make them P1 since the segfaults are only in the 64 bit build and I don' t think anyone uses that on windows. | 16:29.21 |
rayjj | chrisl: marcosw: since you are concerned with cluster testing, and it does -sOutputFile='|md5sum' can't we just allow piping for psd... (not trigger the problem) and assume that the program we are piping to is smart enough to handle it ? | 16:29.23 |
chrisl | rayjj: it seems that doesn't work | 16:29.48 |
rayjj | marcosw: we have several customers that use the 64-bit gs on windos | 16:29.55 |
| marcosw: mainly because their app is 64 bit and they want to call the 64-bit lib | 16:30.35 |
| chrisl: I know that we can't write a valid psd that way, but what doesn't work -- does the PSD device need a seekable output file or something ? | 16:31.39 |
marcosw | rayjj: I always see 'gswin32c.exe' in bug reports but perhaps customers are just using the 32 bit build to duplicate the problem on the command line. | 16:31.57 |
chrisl | rayjj: You end up with only the last page - somehow...... | 16:32.07 |
henrys | marcosw, rayjj, chrisl: I do really think P1 for these no? | 16:32.38 |
marcosw | chrisl: looking at the nightly regression code it's not too bad, not sure about the cluster... | 16:32.47 |
kens2 | I'm intrigued to know how that's possible | 16:32.48 |
marcosw | henrys: i agree, based rayjj's comment that we have customers using 64 bit. | 16:33.06 |
kens2 | I thnk the image one should be P1, it could be very general | 16:33.12 |
chrisl | I haven't a clue - I would have said impossible, but..... | 16:33.13 |
marcosw | chrisl: I'm pleased to have found one impossible thing before breakfast :-) | 16:33.44 |
| (of course I actually found this several weeks ago and probably late at night, but nevermind). | 16:34.06 |
kens2 | I thnk we only have 1 seg fault ? | 16:34.24 |
chrisl | marcosw: I was about to say that this must be Thursday, I've always had problems with Thursdays..... but that's clearly wrong | 16:34.46 |
marcosw | It's the Apple Watch a digital watch? | 16:35.18 |
kens2 | aha the other culprit is GhostXPS | 16:35.21 |
marcosw | yeah, gxpswin64.exe is very broken. | 16:35.51 |
henrys | don't p1 anything for ghostxps | 16:35.53 |
kens2 | THat's the setjmp/longjmp in the PNG code. Probably needs Tor's attention | 16:35.56 |
| THen there's only 1 GhostScript one | 16:36.10 |
chrisl | gxpswin64.exe has *always* been very broken | 16:36.10 |
henrys | kens2: so that should be reproducibly in mupdf? | 16:36.24 |
kens2 | henrys I rather doubt it | 16:36.34 |
chrisl | But I don't recall it crashing before | 16:36.35 |
kens2 | As chrisl pointed out before, GhostsXPS and MuPDF's XPS interpreter are no longer very similar | 16:37.04 |
marcosw | at some point in the past building a 64 bit gxps.exe resulted in a 32 bit executable, so there's that. | 16:37.25 |
chrisl | marcosw: No, I fixed that a *long* time ago. More recently, it just produced no output, ever | 16:37.50 |
marcosw | chrisl: so there was never a time that gxpswin64.exe produced correct output? (and was actually 64 bit). | 16:38.24 |
chrisl | marcosw: not that I can recall | 16:38.38 |
rayjj | marcosw: one customer we hear from that uses gswin64 is cust 384 | 16:38.43 |
marcosw | that's sort of what I found with bisect | 16:38.57 |
chrisl | marcosw: which is why I never distributed a 64 windows binary of it | 16:39.09 |
| 64 bit windows binary | 16:39.23 |
marcosw | chrisl: maybe I should stop testing it :-) | 16:39.24 |
kens2 | feels it should work | 16:39.43 |
chrisl | The setjmp/longjmp is probably alignment related | 16:40.03 |
kens2 | And should be made to work. Wasn't norbert looking at it at on point | 16:40.04 |
rayjj | marcosw: also cust 210 | 16:40.10 |
henrys | no way should we have an ghostscript bug out there, xps we can live with for a while | 16:40.22 |
kens2 | chrisl I'm uncertain, Its involving hideous macros so I gave up | 16:40.30 |
chrisl | kens2: I'm going by the fact we had a problem like that previously with the png output devices | 16:40.57 |
marcosw | rayjj: since valgrind suggests the http://bugs.ghostscript.com/show_bug.cgi?id=696320 crash is related to the clist I'll assign it to you. | 16:41.02 |
kens2 | But I would not be surprised to find its an alignment problem | 16:41.07 |
| marcosw that's a preliominary look by me, I'm not saying its a clist problem | 16:41.36 |
marcosw | kens2: valgrind produced clist related issues as well, so that's two half-hearted votes. | 16:42.15 |
kens2 | :-) | 16:42.23 |
marcosw | besides rayjj actually uses windows as his developement environment (and can hopefully build and debug 64 bit code). | 16:43.01 |
| btw, what's all the talk about moving to VS2015? Chris just walked me through setting up VS2003 (or something like that) to do regression testing. | 16:44.07 |
kens2 | I'm actually using 2008 | 16:44.18 |
| But I can't debug 64-bit buolds with it, so I installed 2015 to see if it would work. WIth a considerable amount of prodding it does. | 16:44.48 |
| Needs me to disable my firewall :-( | 16:44.58 |
chrisl | marcosw: we generally do try to make it possible to *build* with more recent versions | 16:44.58 |
kens2 | VS 2015 ooks dreadful. Can't see which line of code its executing, and it's search clutters up the whole page with coloured boxen | 16:45.35 |
mvrhel_laptop | kens2: I had no problems like that | 16:45.43 |
| VS 2015 interface is not much different than 2013 | 16:46.02 |
| or 2012 for that matter | 16:46.06 |
kens2 | I never used any of those | 16:46.13 |
marcosw | chrisl: yeah, well, when I tried to get VS2012 (?) working you suggested VS2005 (which is what I'm actually using). | 16:46.14 |
kens2 | I detest the 'flat' look, don't like the fact I can't change colours etc, etc | 16:46.33 |
marcosw | of course I'm also running on Windows 7, so presumably VS2015 isn't an option in any cae. | 16:46.41 |
| ^cae^case | 16:46.51 |
chrisl | marcosw: Yes, because I felt that testing should (at least to start with) should be on the version we use for the binaries we ship | 16:47.02 |
mvrhel_laptop | the only issue I see with 2015 is that some of the extensions are not yet available | 16:47.05 |
marcosw | chrisl: sounded reasonable at the time and still make sense. just wondering why this sudden interest in vs2015. | 16:47.37 |
kens2 | I just don't like the look, and you can't change it | 16:47.39 |
| marcosw : I'm runnig Windows 7 and VS 2015 | 16:47.57 |
chrisl | marcosw: current version fever | 16:48.07 |
kens2 | will only be dragged into the revolting flat look kicking and screaming all the way | 16:48.46 |
chrisl | I'm a little surprised VS isn't skinnable | 16:49.30 |
kens2 | If only...... | 16:49.38 |
rayjj | sorry. My laptop power supply came unplugged and I had to reboot | 16:50.30 |
| (it wouldn't come up from hibernate that it does with critical low power) | 16:50.59 |
marcosw | just to change the subject, I've set the fans on my computer to have a minimum rpm of 0, so they don't spin at all when the computer is idle. the cpu is water cooled and the pump never shuts off, so it's not 100% silent but it's really noticeable when a cluster run starts (like it did a few minutes ago). | 16:51.13 |
mvrhel_laptop | rayjj: so were you able to figure out what the OE stuff is about? | 16:51.51 |
rayjj | marcosw: on the cluster subject, has peeved been OK now ? | 16:52.04 |
marcosw | rayjj: yup. | 16:52.15 |
| henrys: otoh, we haven't seen your new machine on the cluster. | 16:52.26 |
rayjj | mvrhel: it looks like ghostscript is part of the openembedded-core | 16:52.27 |
| (probably because it's there for cups and printing) | 16:52.50 |
| but it looks like a lot of other packages are 'meta-***' things | 16:53.23 |
mvrhel_laptop | what version is it installing. I could not tell. MQ just sent me an email that âdNumRenderingThreads=4 was not having any effect | 16:53.24 |
| I will ask MQ what version he has.. | 16:54.02 |
rayjj | mvrhel: I saw 9.15->9.16 as the most recent commit by Robert Yang last May: http://cgit.openembedded.org/openembedded-core/commit/?id=c5adb5903bd93f1c71acd52062d8c8620de2f944 | 16:55.08 |
| mvrhel: so presumably, that's what MQ pulled | 16:55.30 |
marcosw | chrisl: I will re-jigger the nightly regression code to use temp.%d.psd so multi-page psdcmyk files will work correctly. I'll presumably have to do the same thing on the cluster... | 16:56.24 |
| bbiaw, latte and breakfast just showed up. | 16:57.17 |
rayjj | mvrhel: w.r.t NumRenderingThreads, if the Outputpage start and end time are small, then the rendering won't be helped | 16:57.39 |
mvrhel_laptop | rayjj: ok that makes sense. He is running a 12 page doc that is finishing in less than 2 secs | 16:58.08 |
rayjj | mvrhel: -dBGPrint=true might help (overlaps parsing and rendering) | 16:58.36 |
| mvrhel: *not* to be used with NumRenderingThreads | 16:58.54 |
mvrhel_laptop | rayjj: ok | 16:59.03 |
rayjj | mvrhel: but on simple pages without disk I/O (memory based everything), we are probably memory bandwidth (not CPU) limited | 17:00.31 |
mvrhel_laptop | ok that makes sense | 17:00.47 |
chrisl | marcosw: okay. Do you want me to hold off committing until you have that all setup? I must admit, I'll like to understand the issue with the piped output | 17:01.03 |
kens2 | discovers how to get rid of the stupid SDI tabbed interface and back to an MDI interface. THere may be more work to do hre. Also; discovered how to get rid of the shouting in menus | 17:01.04 |
rayjj | mvrhel: which is why I wanted to get a board and profile to see the hot spots | 17:02.04 |
marcosw | chrisl: if you could wait that would be good. if I do my change first, your change won't cause any md5sum differences. | 17:02.13 |
kens2 | Bizarrely every time I move focus away from my floating windows, they disappear, what's that all aout then ? | 17:02.15 |
mvrhel_laptop | rayjj: I will work on getting one | 17:02.20 |
chrisl | rayjj, mvrhel_laptop: I'm really not convinced that using a pre-packaged build is the best way to go for performance testing | 17:02.25 |
| marcosw: not a problem | 17:02.31 |
marcosw | chrisl: I should be done later today (i.e. in the morning your time). | 17:02.36 |
mvrhel_laptop | chrisl: I agree | 17:02.38 |
rayjj | mvrhel: OK, thanks. I'll let you eat now :-) | 17:02.42 |
mvrhel_laptop | I already warned him | 17:02.43 |
chrisl | But we are due to get a board to play with? | 17:03.00 |
mvrhel_laptop | I think we are supposed to get a few | 17:03.10 |
rayjj | chrisl: I agree, sort of. It's using our makefiles (and configure) | 17:03.28 |
Robin_Watts | mvrhel: Or could he let us ssh in to his board? That way we can build for him. | 17:03.29 |
chrisl | Well, I guess we just wait until we get the hardware, and really see where we stand..... | 17:03.47 |
rayjj | Robin_Watts: he cross compiles it, AFAIK | 17:03.58 |
marcosw | chrisl: you will figure out why the last page is saved when using pipes, right? | 17:04.00 |
chrisl | marcosw: I am going to look at that, but I can't promise! | 17:04.21 |
marcosw | chrisl: good enough. | 17:04.34 |
chrisl | I was suspicious about the openembedded stuff as it *seems* to be building (at least) without freetype and without jbig2dec..... | 17:05.18 |
rayjj | Robin_Watts: all the openembedded stuff looks like it is made for cross compiling, and part of the "recipe" for gs plugs in an arch.h for the platform | 17:05.32 |
| chrisl: without jbig2dec is probably OK for most files, but I was also puzzled about the freetype. Maybe it's relying on an so lib ? | 17:06.35 |
chrisl | rayjj: no, IIRC, they've --disable-freetype | 17:07.05 |
rayjj | chrisl: can we do text without freetype anymore ? | 17:07.13 |
chrisl | rayjj: yes, the old code still works - sort of | 17:07.42 |
mvrhel_laptop | oh that is not good | 17:08.16 |
rayjj | chrisl: OK. Because mvrhel mentioned (on the phone) that the pages look like they are coming out OK | 17:08.27 |
mvrhel_laptop | well from the command line output | 17:08.38 |
| I have not seen any pages... | 17:08.43 |
henrys | marcosw: are you seeing any error on your end for my machine? I thought I did everything. | 17:08.54 |
mvrhel_laptop | this is out to the null device | 17:08.55 |
rayjj | chrisl: iirc, freetype was a little bit faster than our "AFS" | 17:08.57 |
chrisl | rayjj: for some cases | 17:09.20 |
| The output probably will look okay for well formed, well behaved fonts | 17:09.32 |
marcosw | henrys: I'm not seeing your machine at all. it's never sending the 'I'm alive' signal to the cluster. | 17:09.46 |
rayjj | mvrhel: Oh, you were just going from the lack of error messages ? So, it could be generating blank pages :-/ | 17:09.47 |
mvrhel_laptop | it could. how am I suppose to know with the null device | 17:10.16 |
marcosw | henrys: is the clusterpull.sh script running? | 17:10.24 |
kens2 | GOodnight all | 17:10.35 |
Robin_Watts | henrys: you here ? | 17:10.40 |
mvrhel_laptop | rayjj: I will have him go out to a different device to make sure all is ok | 17:11.00 |
rayjj | mvrhel: he can run it to something else (local SD card file? ) and look at it (maybe use png16m) | 17:11.16 |
mvrhel_laptop | yes | 17:11.20 |
henrys | marcosw: yes it is | 17:11.21 |
| Robin_Watts: yes I am | 17:11.25 |
Robin_Watts | henrys: There is a customer reported bug, that chrisl traced to a clipping problem. | 17:11.46 |
rayjj | mvrhel: I usually check the output independently of doing the timings | 17:11.54 |
Robin_Watts | The problem appears to be in scan_contour | 17:11.58 |
chrisl | Robin_Watts: those scan_contour changes seem to improve the clarity quite a lot - and comments.... crumbs! | 17:12.03 |
Robin_Watts | which is a function of igors, that he claims is a "revised and much improved" version of something you wrote. | 17:12.33 |
marcosw | henrys: could you try running './heartbeat.pl <machineName>'? | 17:12.45 |
henrys | Robin_Watts: which bug is this? | 17:13.13 |
Robin_Watts | Now, putting aside the fact that "revised and much improved" from igor generally means "obfuscated and broken", do you remember anything about this code? | 17:13.16 |
| http://bugs.ghostscript.com/show_bug.cgi?id=696305 | 17:13.28 |
henrys | I was just looking at it a couple days ago and now I've lost it | 17:13.30 |
marcosw | henrys: never mind, I think I found the (or at least a) problem. | 17:15.04 |
henrys | Robin_Watts: No I don't but these guys need to be told we don't expect good progress until Q1 2016. I know that's tough but that is the right call priority wise. | 17:16.13 |
marcosw | oops, better wait until Robin_Watts' cluster run finishes before I do anything bad to the cluster. | 17:16.45 |
henrys | marcosw: did I screw something up? At the end of this exercise I would like to have an engineer be able to add a machine without help beyond the readme. | 17:17.42 |
marcosw | henrys: i don't think it's anything you did. | 17:18.20 |
henrys | chrisl: I meant to ask about shelly, I didn't hear back on my last review. I hope I'm not being difficult | 17:19.53 |
mvrhel_laptop | ok. before I dig into mupdf I want to wrap up a couple gs bugs | 17:20.33 |
chrisl | henrys: he's probably got day job stuff going on | 17:20.35 |
mvrhel_laptop | including 696321 | 17:21.11 |
henrys | rayjj: it's going to be really weird to walk into my old office with Miles and Scott. 3 colleagues still remain from the old QMS group. | 17:22.21 |
marcosw | henrys: okay, I fixed one problem. The cluster monitor script on casper hadn't been restarted so wasn't accepting connections from your machine (this step is documented in the readme, I just didn't follow the instructions). | 17:23.21 |
| henrys: so now I can sent hearbeats that impersonate your machine and they show up, but your machine doesn't appear to be sending any. Could you try 'heartbeat.pl henrysi7' from the cluster directory? | 17:24.14 |
henrys | marcosw: I guess I did screw something up I didn't copy that over. | 17:25.21 |
marcosw | you don't need to, clusterpull.sh does it. | 17:25.33 |
| the line is "scp -i ~/.ssh/cluster_key -q regression@cluster.ghostscript.com:/home/regression/cluster/heartbeat.pl ." | 17:25.47 |
| or are you running an old clusterpull.sh? | 17:26.11 |
henrys | heartbeat.pl had no output. | 17:27.32 |
marcosw | that's good, no error message :-) | 17:27.50 |
| and I do see the updated timestamp on casper. | 17:28.03 |
henrys | marcosw: I just copied clusterpull.sh as per the readme | 17:28.29 |
marcosw | so that begs the question why clusterpull.sh didn't copy that file over. | 17:28.31 |
rayjj | henrys: are you going to try and pitch them GhostPDL for their printers :-) | 17:28.49 |
| marcosw: if the host hasn't been accessed from a new machine, it won't be in the hosts list, and may error out | 17:30.05 |
henrys | rayjj: no the plan is to tread lightly, mupdf as an add on for GDI printers, Smart office. Selling the language group a rip seems like a non starter to me. | 17:30.08 |
rayjj | henrys: yeah, an uphill battle at best | 17:30.39 |
henrys | marcosw: in the log here I have a test run from robin? | 17:30.41 |
Robin_Watts | henrys: I'm running a test run now. | 17:31.04 |
marcosw | rayjj: true, but the readme instructs the user to run an `ssh touch` and answer 'yes' to the question. | 17:31.04 |
| henrys: henrysi7 does seem to be running on the cluster now. | 17:31.19 |
| though it failed in some way, let me check the log | 17:31.56 |
rayjj | marcosw: sounds reasonable. I hadn't read the readme | 17:32.20 |
Robin_Watts | marcosw: First test run probably failed to update everything before everything else finished. (it was a bmpcmp run, so very quick) | 17:32.29 |
marcosw | Robin_Watts: that's possible but doesn't quite match the symptoms. | 17:33.33 |
henrys | marcosw: I'll be leaving in a bit for cross fit class, but I'll check the logs and email when I get back. | 17:38.22 |
marcosw | henrys: that would be great, I'll be around most of the day. | 17:38.56 |
mvrhel_laptop | Robin_Watts: I am wondering if I can throw bug 694916 to you. A problem is that a call dest_height = fixed2int_var_rounded(any_abs(penum->y_extent.y)); in image_render_color_thresh is overflowing its sign by the macro fixed2int_var_rounded. Unsigned int seems to work ok. I will put the details in the bug. I am sure you don't have time to get to it, but this fixed point macro... | 17:44.04 |
| ...stuff is more up your alley. | 17:44.06 |
Robin_Watts | mvrhel_laptop: Sure. | 17:44.48 |
mvrhel_laptop | rayjj: so looks like MQ has 9.07 | 17:48.45 |
| in his open embedded build | 17:48.59 |
| so I suspect -dBGPrint is not available | 17:49.23 |
| when was -dNumRenderingThreads added? | 17:49.43 |
Robin_Watts | -dNumRenderingThreads has been there for longer than -dBGPrint. | 17:50.24 |
chrisl | It predates 9.07 by quite a long time | 17:50.34 |
mvrhel_laptop | ok I thought so | 17:50.40 |
| but the background thread overlap of the interpreter and the rendering is not there | 17:51.05 |
chrisl | But we have made a lot of performance improvements since 9.07 | 17:51.10 |
mvrhel_laptop | yes | 17:51.13 |
| I am going to push him to get us some boards | 17:51.20 |
chrisl | For a case like this, not using the ICC color workflow is probably a big one | 17:52.30 |
mvrhel_laptop | right. I dont think the fast color was available then either | 17:52.52 |
chrisl | Actually, UseFastColor should be available in 9.07. | 17:56.38 |
mvrhel_laptop | oh ok | 18:03.32 |
| oh great. they have run out of boards... | 18:04.10 |
marcosw | feels a bit like a character in "The Martian", I'm remotely debugging henrys new cluster node by modifying the part of the code that appears to be working to send back debugging information on the parts that aren't. | 18:14.55 |
mvrhel_laptop | brb | 18:17.12 |
marcosw | Robin_Watts: you probably received an email but your cluster job failed because you are writing out too much debugging information (i.e. scan_contour (early): 40cd,40cd->40cd,28800). To avoid filling up the disk the cluster abort when the log gets too big. | 18:35.47 |
Robin_Watts | marcosw: Yeah, that was a slip, sorry. I aborted and reran. | 18:36.06 |
marcosw | I was worried that my fiddling with the cluster code broke something. I tend to do that :-) | 18:36.39 |
Robin_Watts | too. | 18:38.13 |
| /* q stands at the first segment, which isn't last. */ | 18:43.39 |
| Top comment. | 18:43.43 |
chrisl | It's hard to argue with, though..... | 18:44.40 |
Robin_Watts | in order to argue with it, you'd have to be able to understand what it was claiming. | 18:45.40 |
chrisl | Maybe it makes sense in Russian | 18:46.07 |
Robin_Watts | ÐопÑÐ¾Ñ ÑÑÐ¾Ð¸Ñ Ð½Ð° пеÑвом ÑегменÑе, коÑоÑÑй не пÑодлиÑÑÑ. | 18:46.56 |
| That makes no less sense, it's true. | 18:47.08 |
| I have a fix for that bug. | 18:48.26 |
| Let me tidy the commit up. | 18:48.58 |
chrisl | There's another file that would be worth trying....... | 18:49.14 |
| tests/ps/bigpaths.ps | 18:49.57 |
mvrhel_laptop | tor8: so I was going to start a pdf_create branch for this work. I guess I should make a pdfcreate_main addition to the mutool and then work from there with starting out with a blank pdf, and then add pages, content and annotations | 18:59.04 |
| does that sound reasonable | 18:59.37 |
| I assume that as I start down this road it will be clear what needs to be fixed/cleaned up in the core | 19:00.05 |
| and simplified | 19:00.17 |
chrisl | I'm heading off.... g'nite all! | 19:00.57 |
mvrhel_laptop | good night | 19:01.03 |
Robin_Watts | mvrhel_laptop: Personally, I'd not work in the way tor suggests. | 19:01.54 |
| If our intention is to make the pdf writing device work, then we can start much more simply. | 19:02.23 |
| mutool draw -o out.pdf in.xps | 19:02.37 |
| That's enough to demonstrate the limitations in the pdf output device. | 19:02.57 |
| and to immediately see improvements as you extend it. | 19:03.14 |
mvrhel_laptop | Robin_Watts: I see so basically writing pdfwrite to handle all the fitz graphics commands | 19:03.52 |
Robin_Watts | The pdfwrite device is there already. | 19:04.15 |
| It's just incomplete. | 19:04.19 |
| It's fill in the blanks coding. | 19:04.25 |
mvrhel_laptop | that does sound like it would be a better option | 19:05.01 |
Robin_Watts | There is other work that needs to be done that wouldn't be immediately shown by this approach (like adding pages from one PDF to another) etc. | 19:05.41 |
mvrhel_laptop | Robin_Watts: ok that is what I was just going to ask | 19:05.57 |
Robin_Watts | but the pdfwrite device is the biggest incomplete piece, I reckon. | 19:06.06 |
| I'm not sure I see the independent value of a tool that takes a content stream and wraps it into a PDF. | 19:06.35 |
mvrhel_laptop | Robin_Watts: looking at what the competitor has though, being able to drop in various annotations and content is going to be be needed | 19:06.44 |
Robin_Watts | especially as that will fail to cope with resources like images/fonts etc. | 19:06.46 |
| mvrhel_laptop: Imagine opening a pdfwrite device, then running an existing pdf page to it. | 19:07.33 |
| Then running extra content to it. | 19:07.41 |
mvrhel_laptop | Robin_Watts: yes. | 19:07.51 |
Robin_Watts | That's a mechanism for getting extra stuff onto a page. | 19:07.58 |
mvrhel_laptop | I can see that | 19:08.14 |
| I would prefer to leverage what you already have | 19:08.28 |
| let me look over the pdfwrite stuff | 19:08.35 |
Robin_Watts | Now, I think one of the issues with the pdfwrite stuff was that there isn't quite the information within the fz_font that we'd like. | 19:09.12 |
| We need to be able to output the font data to embed fonts etc, for example, and that is not readily available. | 19:09.37 |
| So it will probably require some rejigging of the fz_font/fz_fontdesc stuff. | 19:09.52 |
| but I'm sure tor8 will have ideas on that. | 19:10.21 |
| I think he's got designs on some of the fz_text stuff as it is. | 19:10.35 |
captainralf | hi all, is this just a dev channel or support as well? | 19:11.52 |
Robin_Watts | captainralf: Depends on our mood :) | 19:12.35 |
| captainralf: Is this for gs? | 19:12.46 |
captainralf | lol, fair enough and yep for gs | 19:13.12 |
Robin_Watts | What's your problem? | 19:13.22 |
captainralf | well, my friend has a bash script that calls gs to merge a few docs. but he can't send a string with several document paths in it. Tried it from commandline as well, but gs can't find the files, even though the paths look fine in the error output. Shouldn't be a bash issue... | 19:19.10 |
| he doesn't use irc, that's why i'm acting as proxy here ;) | 19:19.27 |
Robin_Watts | OK, so for us to look at it, he'd have to do a bit of legwork. | 19:20.56 |
| We're not going to debug his bash script. | 19:21.05 |
| But if he can give us a command that we can run that goes wrong, we can try to help out. | 19:21.27 |
| So basically if he can figure out the exact command that his bash script is doing, and try that on the command line, if it goes wrong, we'll help. | 19:21.59 |
| I'm guessing he's doing something like: | 19:22.08 |
| gs -sDEVICE=pdfwrite -o out.pdf in1.pdf in2.pdf | 19:22.23 |
| It's also worth noting that that does not 'merge' the PDF files. It makes a completely new PDF file that is unrelated to the input ones other than the contents should hopefully look almost identical :) | 19:23.13 |
captainralf | yeah of course, no problem. he did try on commanline, but it gave him the same error. he's going to send the command+output in a sec | 19:23.45 |
| oh that's interesting. never had any issues using gs to merge. is there an actual merge program out there? I assumed most GUIs just use gs in the background in a similar fashion, except those creating image based docs only | 19:25.03 |
| this is the commandline he uses in the script: gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -dPDFSETTINGS=/prepress -sOutputFile="$mergedFile" "$sourceFiles" | 19:26.08 |
Robin_Watts | captainralf: OK. That's got bash bletcherisms in it. | 19:26.40 |
| For a start, I'd guess that "$sourceFiles" is not what you want. | 19:26.57 |
| "in1.pdf in2.pdf in3.pdf" is not the same as "in1.pdf" "in2.pdf" "in3.pdf" | 19:27.14 |
tor8 | mvrhel_laptop: yeah, that sounds reasonable. | 19:27.18 |
henrys | marcosw: well I put it under my user name, if you think you are going to need access I'd rather change the user. | 19:27.49 |
Robin_Watts | guesses tor8 is reading the logs... 5...4...3...2...1... | 19:28.00 |
marcosw | henrys: I found and fixed the problem with your new machine not working as a clusternode. The code had a 'heartbeat.pl` in it, which worked on the other machines because '.' is in the path. I fixed it so now it does './heartbeat.pl'. | 19:28.00 |
tor8 | Robin_Watts: mvrhel_laptop: yeah, reading logs. | 19:28.06 |
| Robin_Watts: mvrhel_laptop: my thinking was that my original idea would be a warming up for filling in the blanks in the pdfwrite device | 19:28.23 |
| and at the same time create a top level api for how the pdfwrite device can be used | 19:28.40 |
marcosw | as I mentioned earlier I did it by modifying the parts that worked to send log files and debugging output to casper. Probably wouldn't have taken less time if I had been sitting in front of your machine. | 19:29.08 |
Robin_Watts | We have code to create new pages, I think. | 19:29.32 |
tor8 | the way mudraw -o foo.pdf uses the pdfwrite device is probably too limited for the general use we envision | 19:29.45 |
mvrhel_laptop | tor8: so you think adding a create option in mutool is reasonable and have that leverage pdfwrite | 19:29.46 |
tor8 | I'm thinking robin and my "projects" are separate things | 19:30.13 |
Robin_Watts | I think merging pages from PDF to PDF shouldn't need pdfwrite at all really. If we use pdfwrite we're as bad as gs. | 19:30.44 |
tor8 | my original thought would be that the pdf page and annotation creation stuff would be independent of pdfwrite (it just takes a ready made content stream and creates all the necessary page tree objects etc) | 19:30.56 |
captainralf | Robin_Watts: yep, I agree. he told me that if he uses individual variables for each file it works, but as the list can get quite long, so he was hoping to avoid it(as this is basically the point of the script). Also, it didn't work if the $sourceFiles had "" around each individual file. | 19:31.11 |
tor8 | it needs to sort juggling resource dictionaries, etc. | 19:31.16 |
marcosw | There is still an issue with your machine and cups but I've seen this before so should be easy to sort (as I recall some package is missing, presumably the readme is incomplete in the list of packages that need to be installed). | 19:31.29 |
tor8 | so copying pages between pdf files would just mean pulling the resources over as well, not re-interpreting with the pdfwrite device | 19:31.38 |
Robin_Watts | Anything that takes a "ready made content streams" is essentially useless though. | 19:31.39 |
henrys | marcosw: I may have been speed reading through that part too ;-) | 19:32.09 |
Robin_Watts | tor8: I have some code to copy resources between trees. | 19:32.14 |
tor8 | input a ready made content stream + resource dictionary, insert into a page tree. | 19:32.17 |
marcosw | no, that's like the second step. | 19:32.28 |
Robin_Watts | That's in the pdf operator filtering stuff I think. | 19:32.30 |
tor8 | then we plug in the pdfwrite device to create the content stream and resource dictionary | 19:32.34 |
| separate lego bricks; and that frees the pdfwrite device to not have to worry about whether it's creating a page or annotation stream | 19:33.15 |
henrys | marcosw: I think I was lacking libcupsimage2-dev which is not installed | 19:33.55 |
| marcosw: I think I was lacking libcupsimage2-dev which is now installed | 19:34.04 |
marcosw | henrys: right, ghostscript is reporting "Unknown device: cups". This happens because ./configure doesn't find the cups libraries. BTW, what distribution are you running? | 19:34.08 |
| okay, I'll try again. | 19:34.16 |
tor8 | Robin_Watts: it would be nice to have those bits of code accessible as independent functions we can use without being tied to a specific work flow | 19:34.27 |
Robin_Watts | tor8: It would. | 19:34.36 |
henrys | marcosw: 15.04 | 19:34.39 |
Robin_Watts | I mention it not to say "it's there and finished", but just "there may be something you can refactor" | 19:34.58 |
tor8 | Robin_Watts: fair point. | 19:35.07 |
marcosw | henrys: me too (at least on the cluster nodes in my home office), the ones in my garage and miles' office run 14.04 LTS | 19:35.20 |
| which means I get to deal with updating them in 6 months :-( | 19:35.40 |
tor8 | "mutool create" would not be a very useful tool on its own, but as a developer tool it wouldn't be completely useless | 19:35.44 |
| mutool create which creates font and image resources from command line arguments would be handy for concocting test files | 19:36.41 |
Robin_Watts | Yes. | 19:36.58 |
mvrhel_laptop | tor8: so, are we thinking that I should start with doing the mutool create and see about adding in content. Then when all the content has been "added" one would use pdfwrite to save it. | 19:37.00 |
tor8 | mutool create -o out.pdf -f font.ttf -i image.jpg contents.txt | 19:37.11 |
Robin_Watts | mutool create -o out.pdf -f F0 font.ttf -i I0 image.jpg contents.txt | 19:37.41 |
| where contents.txt uses F0 and I0. | 19:37.51 |
tor8 | the pdfwrite we are talking about is something different, it's a fz_device that takes graphics commands and creates a content stream only. | 19:37.51 |
mvrhel_laptop | ok | 19:38.08 |
tor8 | we already have code to write out pdf objects and streams | 19:38.09 |
| it's creating said objects and streams to make a brand new pdf that's missing parts | 19:38.22 |
Robin_Watts | mvrhel_laptop: There are 2 aspects to this work. The first is the actual pdf device (that I was talking about). The second is the fiddling around inside a PDF file to add new pages etc. | 19:38.51 |
mvrhel_laptop | so in the above what exactly is in contents.txt | 19:38.52 |
tor8 | mudraw with pdfwrite creates some of it, mixed into the pdf content stream creation. that's what I want you to separate out. | 19:39.05 |
Robin_Watts | I think tor was thinking you'd tackle the second part first. | 19:39.12 |
| contents.txt would be something like: 1 0 0 1 100 100 cm /F0 10 Tf "Ass Biscuits" Tj 0 0 1000 1000 re f | 19:40.05 |
mvrhel_laptop | ok | 19:40.11 |
tor8 | Yes. Fiddling with the PDF files to add pages etc. It's something I believe we get a few requests for help in how to do and it's also a lot easier to start with. | 19:40.19 |
marcosw | henrys: your machine is pretty quick, only my i7, yards, hubbles, and points are faster at compiling ghostscript. If you installed ccache I think you might beat them as well. | 19:41.00 |
mvrhel_laptop | so in the above example of , mutool create -o out.pdf -f F0 font.ttf -i I0 image.jpg contents.txt would we ever want to start with an existing pdf (or xps) content and add an additional page of content? | 19:41.57 |
Robin_Watts | mvrhel_laptop: That's a different use case, not covered by mutool create as I understand it. | 19:42.28 |
tor8 | that would come later, in another tool, I imagine | 19:42.38 |
Robin_Watts | mutool merge -o out.pdf in1.pdf in2.pdf | 19:42.49 |
tor8 | mutool create should be a very small program, once the library interface is up to snuff | 19:42.54 |
mvrhel_laptop | ok that sound manageble | 19:43.02 |
| sounds | 19:43.07 |
Robin_Watts | mutool merge should not need pdf write. | 19:43.10 |
mvrhel_laptop | right | 19:43.13 |
Robin_Watts | Nor would mutool create for that matter. | 19:43.26 |
marcosw | we should treat pdf files like psd files and just cat them together :-) | 19:43.30 |
tor8 | eventually, once pdf write is in better shape, "mudraw -o out.pdf one.pdf two.pdf three.xps" should be able to convert formats and merge files like the current gs pdfwrite workflows people always abuse | 19:44.11 |
mvrhel_laptop | I would imagine though that mutool create would be doing some of the same things that are going on in pdfwrite now | 19:45.01 |
| for the final output | 19:45.08 |
| or am I mistaken | 19:45.14 |
tor8 | you would be creating some of the same objects and streams, but the actual saving to disk part is done | 19:45.40 |
| and both pdfwrite and mutool create should be calling the same functions to do the work of setting up pages etc | 19:46.03 |
Robin_Watts | mutool draw -o out.pdf in.whatever calls functions to create a blank PDF, and add a blank page to it, then populates the page. | 19:46.04 |
| You'll be doing the same, just populating the page in a different way. | 19:46.18 |
marcosw | henrys: installing the cups library fixed the problem. | 19:46.55 |
mvrhel_laptop | so there may be some refactoring for pdfwrite needed | 19:47.27 |
tor8 | mvrhel_laptop: not may, will :) | 19:47.36 |
mvrhel_laptop | some of the statics that are in will need to be opened up | 19:47.49 |
Robin_Watts | Yes. A lot of this work is refactor and extend. | 19:47.50 |
mvrhel_laptop | ok | 19:47.50 |
tor8 | also, don't feel wedded to the current functions at all. nobody but us should be using this part of the code. | 19:48.05 |
Robin_Watts | simon91 might be :) | 19:48.15 |
tor8 | like robin just said, the job is mostly refactoring and extension | 19:48.31 |
Robin_Watts | But the MuPDF way is to rip up and relay the tracks if it gives a nicer layout in the end. | 19:48.49 |
tor8 | Robin_Watts: true. but he should be happy that we are giving him proper tools to work with. in the end. :) | 19:49.05 |
mvrhel_laptop | ok this sounds reasonable. I suspect I will make a mess of things to being with. I think I am familiar enough with the mupdf style to keep things clean and clear but let me know if things are not making sense or if I do a no no. | 19:51.36 |
| s/being/begin/ | 19:51.46 |
Robin_Watts | Whatever you do, tor8 will find a way of simplifying it. I've learned to live with that :) | 19:52.23 |
mvrhel_laptop | yes | 19:52.28 |
| A big concern is making sure I don't rewrite something that is already there | 19:52.56 |
Robin_Watts | feel free to run stuff past us and we can have a quick grep :) | 19:53.21 |
mvrhel_laptop | ok sounds good. thanks | 19:53.30 |
tor8 | Always a concern with a sprawling code base. Don't hesitate to ask here if something exists. | 19:53.54 |
henrys | marcosw: wow the clusters were running and I didn't hear a thing. These fans are pretty nice... | 19:53.57 |
Robin_Watts | It's probably worth stepping through mutool draw -o out.pdf tiger.pdf in the debugger. | 19:54.10 |
tor8 | and if it exists, but in a slightly wrong shape, don't worry about fixing it so it works for new code either. | 19:54.22 |
| rather than duplicating it | 19:54.26 |
Robin_Watts | henrys: yeah, silent fans are awesome. My machine is way quieter than my last one. | 19:54.36 |
henrys | Robin_Watts: did you open it up and have a look at them? pretty cool | 19:55.13 |
| no pun intended | 19:55.33 |
Robin_Watts | henrys: Yeah, I've had my machine open to add a 1TB SSD. | 19:55.47 |
mvrhel_laptop | tor8: ok thanks | 19:56.27 |
marcosw | what model fans are they? | 19:56.34 |
mvrhel_laptop | off to lunch | 19:56.44 |
Robin_Watts | To go along with the 512GB M2 PCI-e SSD that the OS is on, and the 4TB slow hard drive for bulk data. | 19:56.48 |
| marcosw: http://www.scan.co.uk/products/fractal-design-arc-midi-r2-black-mid-tower-quiet-performance-case-with-side-window-usb-30-w-o-psu | 19:58.31 |
| http://www.scan.co.uk/products/noctua-nh-u14s-ultra-quiet-slim-cpu-cooler-with-140mm-nf-a15-fan-for-all-intel-and-amd-cpus | 19:59.38 |
henrys | marcosw: for the cpu a gelid tranquillo rev3 | 20:01.08 |
marcosw | I think the cpu cooler is a newer version of what I'm running in yards. I switched to water cooling on my main machine to better handle overclocking. | 20:01.33 |
tor8 | I used to swear by noctua fans, but the latest batch I got were all much noisier than their old models :( | 20:01.46 |
Robin_Watts | chrisl: tests/ps/bigpaths.ps is in the cluster tests, right? I guess we must be working with that :) | 20:14.13 |
| OK, 2 commits on robin/master for review. | 20:14.31 |
| oh, whoops. let me update the commit message on one of them. | 20:14.47 |
| OK, fixed. | 20:15.43 |
| Forward 1 day (to 2015/11/04)>>> | |