IRC Logs

Log of #ghostscript at irc.freenode.net.

Search:
 <<<Back 1 day (to 2015/11/02)20151103 
mvrhel_laptop henrys: are you there?00:03.18 
  done for the night03:52.37 
iamaregee1 Hi , I am trying to convert multipage pdf to png (I would prefer base64 encoded images)06:47.34 
  I am taking input from stdin and redirecting the output to stdout06:47.51 
  but I can't figure out how to get individual images from the Output06:48.37 
  https://gist.github.com/aregee/01e161bd932b77e2210906:48.49 
  Here what I am trying to do is setup a complete streaming process, where user uploads PDF on frontend and gets Images of all the pages in response06:49.41 
chrisl iamaregee1: if spotting the PNG header in the stream doesn't work, probably your only workable solution is to write to individual temp PNG files, serve the PNG data from there07:55.20 
iamaregee1 yes, I guess I am going to resort to that approach only07:56.54 
chrisl You're other option, if you are storing the uploaded PDF would be to only convert one page at a time - that would then allow you do to pages on demand07:58.54 
  Bearing in mind that in sending a PDF to gs on stdin, gs has to buffer the entire PDF because it can interpret it anyway08:00.31 
  s/because/before08:00.41 
norbertj chrisl: did you try to compile ghostpcl using the windows sln? I think on plwmain.c #50 an 'e' which shouldn't be there, and 'gserrors.h' is not included.08:15.16 
kens I built ghostpcl from scratch 32 and 64 bit yesterday using the SHA current at the time and it worked for me on VS 2008 and VS 201508:15.55 
  Let me look at plwmain08:16.18 
chrisl norbertj: with the e_* errors removed commit?08:16.19 
iamaregee1 @chrisl yes, something like this https://gist.github.com/aregee/2fa24a75907de8dbe2e208:16.23 
kens AH, I haven't upgraded to that SHA yet08:16.35 
norbertj kens: chrisl: should be plwmainc.c08:16.39 
chrisl Hmm, sorry, I get I messed that up :-(08:17.10 
kens i sjust updating master08:17.25 
  Oh yes, that's obviously wrong08:18.00 
chrisl I'll fix it in a second....08:18.15 
kens One reason we need a Windows cluster machine I guess08:18.23 
chrisl More to the point, a good reason for me to sort out my mouse so it works properly :-(08:19.06 
  Just testing now - if this builds okay, I'll push the commit08:23.20 
  norbertj: thanks for spotting that08:24.09 
kens Typical VS2015 has overwritten the status for VS 2008, so when I open 2008 all my previously open WIndows haev gone :-(08:26.57 
chrisl kens: well, you should know better!08:29.04 
  Okay, the Windows builds should be better now....08:29.17 
kens I'll go update nd rebuild08:29.30 
  chrisl builds fine for me with that commit08:34.14 
chrisl Thanks08:35.07 
Robin_Watts chrisl: You here?10:30.02 
chrisl yeh10:31.35 
Robin_Watts I started to look at that bug last night.10:33.18 
  Couple of interesting things:10:33.27 
  It's a shame that a 're' call can end up with a path area that's not a rectangle.10:33.41 
kens Can it ?10:34.17 
chrisl Well, that's dependent on the ctm10:34.19 
kens AH yes, of coruse10:34.27 
Robin_Watts chrisl: This is with an axis aligned ctm.10:34.34 
chrisl In this case, I think it's a rounding error10:34.42 
Robin_Watts It's to do with the use of rlineto rather than lineto, I think.10:34.53 
  and float rounding, as you say.10:35.07 
chrisl Well, we could resolve that, but again, it would just hide the read issue10:35.10 
  s/read/real10:35.17 
Robin_Watts Indeed.10:35.19 
  But I think it might be worth doing.10:35.28 
  (after we fix the main problem)10:35.35 
kens It sounds likely, if its easy, might give alittle performance10:35.47 
  I presume this is 're' from PDF10:35.56 
Robin_Watts Then, talking about the flattened_iterator__init_line stuff you say that you can't see how it ever adds more than a 1/4 of the length to the active Y list.10:36.25 
chrisl In these circumstances, yes10:36.49 
kens Ah, we use rlinteto because its easier to code in PS10:36.56 
Robin_Watts This code is new to me too, but I think what happens is that it puts a quarter of the line into the list, and sets the repeats to 4.10:37.10 
  The iterator then repeats that quarter 4 times.10:37.26 
  hence we get the full line.10:37.31 
chrisl Yes, but when we build the active y list, we only process one of the four sections]10:37.55 
Robin_Watts ok, so you've walked through further than me then.10:38.16 
  I will keep looking.10:38.23 
chrisl I'll fire up the debugger, and give you a more specific location - but I'm wary about pointing you at a red herring....10:39.01 
  Robin_Watts: I'm looking in scan_contour in gxfill.c10:43.57 
Robin_Watts ok...10:48.06 
chrisl From scan_contour() we to add_y_line(), and add_y_line_aux(), then init_al(). In init_al(), for the !curve case, we call step_al(), which only ever (as far as I can tell) uses the first set of values returned from gx_flattened_iterator__next()10:48.17 
Robin_Watts right.10:48.55 
  gx_flattened_iterator__next loops 4 times internally, but I haven't understood all this yet.10:49.21 
  I'll walk it through.10:49.42 
chrisl gx_flattened_iterator__next() doesn't have a loop10:49.58 
  It expects to be called from a loop10:50.28 
  Anyway, that's what I was suspicious about10:51.06 
  It seemed to me that scan_contour() loops calling gx_flattened_iterator__next() until it finds a section of the current path segment that intersects with the current clipping area box, but then the downstream code ignores the possibility that the path may have been segmented to avoid difference overflows10:54.00 
Robin_Watts Right.11:00.51 
  I was looking at the loop in scan_contour itself.11:01.41 
  I'll add some debug so I can try to understand what's going on.11:01.54 
  Thanks.11:01.59 
chrisl Yeh, I find the sheer number of times we setup and use a flattened iterator a little depressing11:02.51 
Robin_Watts OK, I added some debug into scan_contour, getting it to print the line segments it's returned by gx_flattened_iterator__next11:11.02 
  scan_contour: 7ffc17ff,803bbeb8->7ffc17ff,c02bd50a11:11.22 
  scan_contour: 7ffc17ff,c02bd50a->7ffc17ff,1beb5c11:11.23 
  scan_contour: 7ffc17ff,1beb5c->7ffc17ff,400c01ae11:11.25 
  scan_contour: 7ffc17ff,400c01ae->7ffc17ff,7ffc17ff11:11.26 
chrisl I *think* those should be sane11:11.41 
Robin_Watts That looks like 4 sub sections of a single subdivided line to me.11:11.54 
chrisl Yeh, so I think that is working as intended11:12.30 
Robin_Watts Which is a shame, cos it means the bug is 'somewhere' in this code (waves hand vaguely) rather than 'there' in this code :(11:13.13 
chrisl So, one of the subsections actually ends up getting added to the y line list11:14.11 
  By add_y_line()11:14.31 
  But from add_y_line()/init_al()/step_al() only uses the first subdivision from the path segment11:17.20 
Robin_Watts falls in.11:24.01 
  I see what you mean.11:24.05 
  We loop around the gx_flattened_iterator__next(p.fi) calls, and get 4 subsections of the line out, but only the last one counts as having been 'sufficiently flattened' for the loop to exit and for it to then be processed by the rest of the code.11:25.13 
chrisl Oh, I thought it was the dir != 2 that gave the exit condition, but my memory may be faulty11:26.11 
Robin_Watts The first loop in scan_contour only exits when the 'code' returned by gx_flattened_iterator__next returns 0.11:28.55 
chrisl Isn't the top loop only checking for only_horizontal?11:30.00 
Robin_Watts __next returns 'true' whenever there is more iteration to do on a given curve segment.11:30.13 
  ok, yes, but I think it's the same issue further down too.11:30.49 
chrisl For me, it's carrying on when p.more_flattened = 1 and p.dir = 111:32.00 
Robin_Watts I have p.moreflattened = 0 && p.dir == 211:32.25 
chrisl In the second loop through the path segments?11:33.12 
Robin_Watts __next has now been called 8 times. 4 in the loop before the for (pseg loop, 4 in the loop inside the for(pseg loop.11:34.24 
chrisl Ah, yes, but we won't call add_y_line() on that pass11:35.43 
Robin_Watts I'm confused by this stuff.11:36.39 
chrisl Well, that makes me feel a little better......11:36.52 
Robin_Watts The gx_flattened_iterator stuff is a cunning way of walking down a path segment using differencing.11:37.17 
  It avoids having to explicitly expand the path segments into a list of lines, and also avoids having to use recursion to do it.11:37.44 
chrisl Yep, and then the subdivision stuff is intended to prevent the differencing causing an overflow11:38.02 
Robin_Watts Urm... ignore the current subdivision thing for now.11:38.31 
  That's a 'cunning' reuse I reckon.11:38.41 
chrisl So, I was going to suggest hacking the code so it doesn't do the subdivision, and walk through that11:39.14 
Robin_Watts I reckon in normal use (if I'm understanding this correctly), we look at the curve, and 'guess' a number of subdivisions - that number being the number of subdivisions required to get the required flatness.11:39.29 
  Then the curve is iterated through, and it'll take that number of 'next' calls to enumerate out the subdivided lines.11:40.10 
chrisl And then we overload that to handle extreme paths11:40.23 
Robin_Watts Hence the working with subdivided curves is absolutely vital.11:40.35 
  Right, but ignore the extreme path case for now.11:40.45 
  Think about this routine being called on just a 'normal' curve.11:41.06 
chrisl Well, that's what I mean - nobble it, so it doesn't treat this as a extreme path11:41.06 
Robin_Watts On just a normal curve, every returned section needs to be handled.11:41.36 
  hence I can't see how it's correct for the code to loop around waiting for p.more_flattened to be 0.11:42.25 
chrisl It's looping waiting for *either* that or p.dir != 211:43.06 
  p.dir == 2 (I think) indicates that segment lies entirely outside the current clip11:43.36 
Robin_Watts Ah!11:43.59 
  Stupid magic numbers.11:44.11 
  Yes.11:44.14 
chrisl Yeh, I think we should fix that magic number11:44.33 
Robin_Watts ok, let me keep walking.11:44.51 
  Ok, so when we're walking the vertical section from (8032639a,7ffc17ff) -> (8032639a, 8xxxxxxxx), it does indeed exit that loop with more_flattned=1 and dir = -111:56.37 
  The subsegment in question being: 80326e9a,400c01ad->80326e9a,1beb5b11:57.14 
  As far as I can tell then, we don't call add_y_line or anything on that subsegment. We end up in the if (p.dir = DOWN || HORIZONTAL) section at the bottom.12:01.17 
chrisl Yes, we'll go through add_y_line a bit later12:02.22 
Robin_Watts That code seems to swap p and q.12:05.08 
  What the hell is this doing?12:05.13 
chrisl Honestly, I would happily tell you if I'd fathomed that out12:05.47 
Robin_Watts chrisl; 2 WIP commits on robin/master for your opinion if you're still interested in this problem.14:11.24 
  Slight rejigs of the code to make it nicer (IMHO).14:11.41 
  Specifically it avoids splitting the contour setup between the caller of scan_contour and the implementation.14:12.17 
kens meeting time ?14:31.54 
  http://www.reuters.com/article/2015/11/03/us-britain-usa-tourism-idUSKCN0SS1432015110314:32.43 
  Finally14:32.47 
henrys kens: in an hour14:35.29 
kens Ah, my mistake14:35.39 
henrys kens: great news about travel I hope you guys can get registered soon.14:36.26 
kens Notice, no actual start date :-(14:36.39 
  But yeah it would be good14:36.49 
henrys jogux: do you know if pdfkit is using their own pdf or have they possibly taken the android bsd licensed code derived from foxit?15:18.45 
jogux henrys: it sounded like they'd developed their own engine in house15:19.06 
henrys s/pdfkit/pspdfkit15:20.31 
jogux "However, as we expanded to Android, we came to the realization that we had to develop our own renderer. As many of you know, Android has only shipped with a native renderer since 4.4, and their renderer is far too limited for the robust features we offer. So we sat down and worked hard to develop our own renderer."15:20.51 
  'PDF is our core business and we have over 20 engineers working full-time on it.'15:21.37 
  I think the 20 is about their whole product, not just the renderer. but still, that's a lot of people.15:21.58 
tor8 henrys: I'm getting frustrated with the lack of man power on mupdf.15:23.15 
henrys jogux: when we looked at them last we found several web site claims suffering inflation ... I thought you might have inside knowledge perhaps not from the web site.15:23.36 
tor8 pspdfkit are going to eat mupdf's business for lunch... they're making an actual usable library with these interactive features15:23.42 
  where we've got half finished prototypes that barely work for our android app, and are in no way suitable for use as a general purpose library15:24.12 
  and nobody with time to work on finishing and polishing them15:24.30 
henrys tor8: so we'll start the meeting early ;-)15:24.58 
jogux henrys: Everything I know it from the website or other articles from Peter Steinburger, or from playing with the sample apps.15:25.23 
tor8 and the only attention mupdf gets is with gsview ... the core features needed to support the interactive features not so much.15:25.25 
Robin_Watts It is hard not to agree with tor8 on this.15:26.06 
  mupdf is basically back to being just tor8 working on it.15:26.17 
tor8 large tracts of mupdf core is now massively complicated to deal with interactive editing etc15:26.27 
henrys first off welcome fredross-perry! This meeting is every tuesday at 7:30 and please come to the skype meeting at 8:00.15:26.46 
fredross-perry As always!15:26.56 
jogux tor8: Yeah, the iOS app is in a similar state. I think I've been trying to say this for a while, I quite agree.15:27.10 
mvrhel_laptop I do feel like mupdf core could use some manpower too. 15:27.50 
tor8 the way I see it, we've got two options here: allocate more man power and work on finishing the interactive features we've already started on.15:27.55 
  or give up interactive features, and go back to focusing on rendering only.15:28.34 
mvrhel_laptop I am at a point in gsview where we need to do some work on mupdf15:28.53 
henrys tor8: we can't back out of interactive features.15:29.01 
tor8 then we need more people working on mupdf. and we don't have time for XFA for quite some time. I've been reading the spec last week. it's dreadful.15:29.25 
Robin_Watts MuPDF rendering is 'good enough'. I reckon.15:29.29 
  We don't get transparency correct, but we're not too far off.15:29.42 
tor8 Robin_Watts: with 'rendering' I'm also thinking about all the format conversion stuff we've started15:30.03 
kens2 Or transfer functions15:30.06 
mvrhel_laptop or color...15:30.10 
marcosw morning15:30.12 
Robin_Watts And overall we are as close to correct with other files as any other renderer.15:30.15 
tor8 with pdfwrite, xps and svg output. the text extraction pdf2html for reflow, etc.15:30.19 
Robin_Watts kens, mvrhel: absolutely.15:30.25 
  OK, format conversions are a separate thing I reckon, but yes, that area needs work.15:30.42 
tor8 there are about a half dozen to a dozen unfinished areas where people have started trying to use it only to be told "no, that's unfinished. come back in a decade."15:31.19 
Robin_Watts tor8: Can't argue with that.15:32.03 
henrys I was hoping to keep Robin_Watts on SOT for a bit to accelerate Paul's project. What if mvrhel went full time mupdf?15:32.37 
mvrhel_laptop I would be happy to do that15:33.08 
henrys and fredross-perry split time SOT/NUI ?15:33.14 
  and fredross-perry split time MuPDF/NUI ?15:33.28 
  sorry15:33.29 
fredross-perry I could15:33.32 
mvrhel_laptop gsview is sort of slowing down a bit due to the limitation of mupdf anyway15:33.52 
fredross-perry mvrhel_laptop: ‘splain?15:34.09 
mvrhel_laptop I had spent the last week hacking at my huge list of gs bugs15:34.19 
kens2 ?me has been pleased to see a number of those closed15:34.38 
mvrhel_laptop fredross-perry: It would be nice to get in the text editing, image addition, etc that Acrobat pro does15:35.04 
henrys that would also have the advantage of getting mvrhel_laptop more into mupdf generally, which I think is a good thing... I realize moving Robin_Watts to MuPDF would probably get things going quicker but long term this seems like a better play to me.15:35.22 
mvrhel_laptop kens2: yes some of those had been sitting around a bit too long15:35.30 
  I little embarrassing 15:35.38 
  s/I/A15:35.44 
henrys tor8: can we have a list of things that you think should be worked on and we'll start from that?15:36.38 
  tor8: perhaps making a list like Paul's list on the Wiki for NUI?15:37.04 
tor8 henrys: I've been writing a document with all unfinished in-need-of-work tasks15:37.28 
  I could put that on the wiki I guess15:37.41 
  some of these tasks are fairly isolated and could be handed to mvrhel to help him get up to speed with mupdf internals15:38.11 
jogux tor8: if we want to go head-to-head with pspdfkit there's a whole bunch of extra things we'd need too?15:38.21 
henrys chrisl: I've fallen behind on language switch, distracted by other nonsense, and the damn reality of the display device. I know kens2 bolted that in hurriedly but it really is a wrench in the works.15:38.25 
tor8 jogux: quite possibly. would be good to have a list of targets.15:38.35 
  I'm assuming they have nice UI components clients can just drop into the apps?15:38.59 
rayjj I had thought that transfer functions were a customer issue for mupdf, but the bug tracker doesn't show it as such15:39.01 
jogux tor8: just to pick one, pspdfkit have sample apps (plural) for each platform15:39.12 
chrisl henrys: really? Hmm, I wouldn't have thought it too bad. Maybe we need to refactor how it's integrated with the PCL/XPS code15:39.27 
tor8 jogux: I've been wanting a java-based android viewer component for a while, where the JNI stuff is limited to mirroring the core api15:39.49 
henrys chrisl: I'm doing that yes, it's not a huge deal.15:40.01 
kens2 I thought it was broadly the same as the way it works in the Ghostscript code. But I don't recall now how it works15:40.01 
jogux tor8: and yeah, it's all nicely packaged components, in iOS they have static and dynamic frameworks, the user just drops the one they want into their xcode project.15:40.04 
Robin_Watts Having a JNI interface to the core is something I've wanted for ages. Even started on it, but never had time to finish it.15:40.16 
tor8 it would let clients use the core api from java, so all the people hacking the android core class could get what they want easier15:40.20 
  Robin_Watts: I think that should get high priority for the 1.9 release in spring, if at all possible.15:40.42 
jogux tor8: and http://stackoverflow.com/questions/33409913/how-to-integrate-mupdf-in-android-studio-step-by-step really should be answereable in a single sentence :-)15:40.56 
Robin_Watts Absolutely. And people should be able to drive the device interface as a 'MuCanvas' so they can create pages.15:41.00 
chrisl henrys: we may want to change how it works entirely - IIRC, it rather goes around the side of the actual API which is icky15:41.04 
tor8 annotation creation should be exposed to a nice friendly set of calls the clients can use in the core library15:41.11 
  Robin_Watts: yes. but that needs work on the pdf output device.15:41.29 
jogux tor8: plus the mobile UIs need a good tidy up15:41.49 
tor8 Robin_Watts: but still definitely something that should be able to do from java15:41.49 
Robin_Watts tor8: Indeed. Lots of these tasks are interelated.15:41.52 
tor8 having looked into XFA, I think these really do need a higher priority than tackling that beast.15:42.18 
Robin_Watts jogux: Yes, the point of the JNI interface is that it'd be exposed in a sane way to java.15:42.23 
  I see XFA as something that'll eat time and not get us any new customers.15:42.49 
jogux Robin_Watts: Indeed - that's only a small part of the issue though; how it's packaged is really important.15:42.51 
tor8 with a new sane JNI interface, writing a sample android demo app should be trivial without having to do any native development on the app side15:43.03 
Robin_Watts jogux: Absolutely. Having an up to date android sample lib is really desirable.15:43.20 
henrys rayjj: I have a sales call this week with miles and scott to customer #535 and #534 - do you understand the specifics of either of those workflows beyond they are using pdfwrite?15:44.01 
jogux Robin_Watts: Indeed.15:44.16 
henrys tor8: make the list get on the twiki and we'll prioritize it.15:44.52 
  tor8: what is the release state?15:45.02 
rayjj henrys: I thought they used ps2write to convert PDF's to PS for their printers15:45.07 
kens2 thnks that makes more sense15:45.22 
jogux is tor8 writing all this down? :)15:45.25 
rayjj but I haven't had any contact with them since Nori left15:45.36 
henrys rayjj: oh right yes, and I just noticed they are a mupdf customer.15:45.55 
tor8 henrys: release candidate is up, but haven't heard much noise15:46.01 
jogux huh. not sure what happened then.15:46.18 
tor8 ran into trouble building the 64-bit windows (msvc project file stuff I'm woefully inexperienced with)15:46.23 
rayjj henrys: well, mupdf doesn't fit the printer workflow at all, so I don't know what they do with that15:46.38 
  but it's a big collection of companies, so it could be a totally different division15:47.13 
  henrys: and I thought one division was doing a host 'document management' product. That _might_ make sense for mupdf to create thumbnails or something15:48.47 
henrys jogux: by the way I really want to have "contact sales" work just like pspdfkit.com - that little questionaire can be hooked up to a database. Ted is supposed to be looking at that but he seems to be swamped with other stuff right now.15:49.15 
jogux henrys: yeah... we should have something similar for SO too (I think Ted is working on the SO one at the moment)15:50.02 
rayjj thinks it's sort of backwards to expect engineers to keep track of what the customers are doing with the product. That's usually a marketing/product management function15:50.03 
henrys marcosw: anything for the meeting?15:50.30 
jogux I really think it would be helpful if we had some kind of customer database that useful info about a customer could be dumped into15:50.38 
  I'd suggest a wiki page but I can't imagine marketing ever using such much, and something better structured would probably be beneficial anyhow.15:51.32 
rayjj particularly since the 'inquiry' they fill out that says what they want to do with the product very rarely makes it past Scott or Miles15:51.32 
mvrhel_laptop tor8: let me know when you want to talk about what I should get started on15:52.27 
marcosw henrys: not really, I'm going through the windows regression results and entering bugs (though I think I might be done, at least for the 64 bit build).15:52.31 
henrys rayjj: actually the lack of transparency is exactly what I don't like, I'm even on the sales mailing list I still can't figure out what's going on because Scott won't CC: me .15:52.32 
Robin_Watts I don't believe the inquiry every actually helps.15:52.32 
  It's utterly terrifying for most people.15:52.57 
marcosw chrisl: I did want to ask chris about http://bugs.ghostscript.com/show_bug.cgi?id=69301115:52.58 
chrisl marcosw: working on it.... been distracted by other things15:53.22 
marcosw chrisl: no problem; I was just thinking I should either add a -dLastPage=1 to the command or add a %d to the output filename and then collect the pages. Can't decide which is better.15:54.26 
  currently we test the last page, so that's clearly not ideal. 15:54.45 
henrys I didn't have anything else for the meeting, more on skype if anyone wants to join beyond the SOT regulars, that'd be great. see you in 5 minutes15:54.58 
chrisl marcosw: I was thinking about adding an exception that piped output wouldn't throw the error15:55.03 
tor8 http://twiki.ghostscript.com/do/view/MuPDF/Tasks15:55.08 
chrisl marcosw: essentially on the assumption that if someone wants to stream psd output, they figure their own way to identify where new files begin15:56.04 
rayjj Robin_Watts: The few inquiries I've seen do have more information than we usually get. At least they describe the product that will use our stuff15:56.12 
tor8 mvrhel_laptop: where do you want to start? :)15:56.26 
marcosw chrisl: you mean piped to stdout? sending to a named pipe (i.e.-sOutputFile="| gzip >test.psd.gz") doesn't work, you get the last page.15:56.45 
mvrhel_laptop well....15:56.51 
chrisl marcosw: that makes not sense at all :-(15:57.25 
marcosw yeah, I know.15:57.31 
chrisl We can't suck back the data we've already written15:57.54 
tor8 mvrhel_laptop: I think one of the core features we need to solve sooner or later will be related to the pdf editing interfaces15:57.54 
mvrhel_laptop tor8: yes. that one interests me. 15:58.33 
tor8 more specifically, we should be able to create blank pdf documents and start filling them with pages and annotations from scratch15:58.35 
mvrhel_laptop right15:58.42 
tor8 and copy and delete and move pages around, etc15:58.44 
mvrhel_laptop yes. 15:58.56 
tor8 so page tree editing, making sure annotations and bookmarks don't get confused, etc15:58.57 
mvrhel_laptop the ability to take a scanned form (image) and drop in form filling annotations would be nice15:59.14 
tor8 now I see three ways to create a page: manually with printf-like stuff, manually using pdf_op_xxx calls to call operators to fill out the page (the pdf_processor interface the interpreter calls), and using a pdf write like device that robin started15:59.52 
  but all those three ways at the bottom terminate in having a fz_buffer with data and a list of resources used that need to be wrapped into a page and stuffed into the page tree and xref16:00.28 
Robin_Watts tor8: yeah, the pdf writing device works enough for appearance stream synthesis, I think.16:00.29 
marcosw chrisl: I'm 99% sure it works that way. I was presuming that ghostscript was closing and reopening the output device and that was causing the first N pages to be lost.16:00.31 
tor8 and those bits at the bottom are what need some good polishing in order to make them usable for customers16:01.02 
chrisl marcosw: that would be true of writing to a file, but not to a stream, once we've written to the stream, we cannot overwrite what's already been sent16:01.17 
tor8 we've got all the core bits, but they're a bit finicky and fragile to put together16:01.18 
  having an api to create a document, create a blank page, fill it with drawing and resources, and saving it back out would be a good start for what we'll need16:02.03 
mvrhel_laptop tor8: that sounds reasonable16:02.19 
marcosw chrisl: I don't disagree with you, but it's doesn't match wjat O16:02.40 
  what I'm seeing.16:02.48 
tor8 this is something that we should then wrap up in the JNI interface, and be able to use to create annotations and possibly edit page graphics with etc16:02.49 
mvrhel_laptop right16:03.02 
marcosw I'll double check and update the bug.16:03.08 
tor8 mvrhel: I suspect if you start by writing a new command line tool to do some of this, you'll see what needs work and what works currently16:03.44 
chrisl marcosw: I'll grab the test file and have a check16:03.58 
tor8 maybe something that will take an input file with a content stream and wrap that up in a new pdf16:04.18 
  mvrhel_laptop: the functions we have for use are in include/mupdf/pdf/xref.h, pdf_create_object, pdf_update_object, pdf_update_stream but those are very low level16:05.56 
marcosw chrisl: confirmed, the last page is the one that is saved.16:06.43 
mvrhel_laptop tor8: ok thanks. I will look this over and probably have a pile of questions.16:06.43 
chrisl marcosw: I have to disagree with you there......16:07.04 
marcosw chrisl: hold on...16:07.19 
tor8 mvrhel: source/pdf/pdf-write.c has a lot of stuff in it that might need extracting into parts16:07.23 
  like the page tree rebalancing etc16:07.27 
chrisl marcosw: If I write the entire file using -sOutputFile="| gzip >test.psd.gz", I get a 1.8Mb gzip file. If I do -dFirstPage=8 I get a 182k gzip file16:08.08 
tor8 pdf_insert_page in source/pdf/pdf-page.c exists, but I'm not sure how robust it is16:08.11 
  I think I saw somebody talking about using it to copy a page from one pdf_document to another .. .but that's *never* going to work so maybe we need to fix the interface so people don't try it16:08.42 
  or fix it so that it will work, by copying all the resources along16:09.13 
mvrhel_laptop right16:09.18 
tor8 I think the pdf-page.c stuff is what needs a proper polishing so that it will work as a general purpose library, not just for our current slim needs16:09.48 
  pdf_create_page is currently only used by mudraw in the pdfwrite mode16:10.44 
mvrhel_laptop I was just looking at the function16:11.08 
  that16:11.10 
marcosw chrisl: that's not what I'm showing.16:11.29 
tor8 anyway, you should have enough code to read for a while now :) I've got to go out for a few hours, will pop back later if you have any questions16:11.37 
chrisl marcosw: are you using Brandypot.pdf?16:11.55 
mvrhel_laptop tor8: I will have questions. Just not sure I will be ready to ask them in a few hours...16:12.04 
marcosw No, BFAZCFPB.PDF16:12.23 
  ./gs/bin/gs -I./gs/lib -sOutputFile='| gzip >./temp.psd.gz' -sDEVICE=psdcmyk -r72 -dNOPAUSE -dBATCH ./tests_private/comparefiles/BFAZCFPB.PDF 16:12.29 
mvrhel_laptop working on the command line application to do what you said makes a lot of sense to get started 16:12.30 
marcosw vs 16:12.39 
  ./gs/bin/gs -I./gs/lib -sOutputFile=temp.%d.psd -sDEVICE=psdcmyk -r72 -dNOPAUSE -dBATCH ./tests_private/comparefiles/BFAZCFPB.PDF 16:12.39 
  and then 'gzip temp.*.psd'16:12.48 
  128 -rw-rw-r-- 1 marcos marcos 127058 Nov 3 08:10 temp.1.psd.gz16:13.03 
  136 -rw-rw-r-- 1 marcos marcos 136733 Nov 3 08:10 temp.2.psd.gz16:13.03 
  160 -rw-rw-r-- 1 marcos marcos 162751 Nov 3 08:10 temp.3.psd.gz16:13.04 
  400 -rw-rw-r-- 1 marcos marcos 407569 Nov 3 08:10 temp.4.psd.gz16:13.04 
  400 -rw-rw-r-- 1 marcos marcos 407558 Nov 3 08:12 temp.psd.gz16:13.04 
mvrhel_laptop This will help me get more familiar with this part of the code and hopefully make it clear what needs to be done16:13.06 
  rayjj: did you get a chance to look at the open embedded stuff?16:13.19 
marcosw note that temp.psd.gz is the same size as temp.4.psd.gz (sort of, I think the temp4.psd.gz is larger because gzip stores the original file name and maybe the creation date/time).16:14.05 
chrisl Hmm, yes, that's very strange......16:14.55 
marcosw chrisl: yeah.16:15.21 
chrisl I also get the same size of output if I use -dFIXEDMEDIA which should prevent the device being closed and reopened.....16:16.16 
marcosw chrisl: I don't think we should write multi-page psd files at all. Produce an error if %d isn't in the output filename when asked to write the second page. While you are at it fix psdrgb to act that way too, writing out files that don't meet the spec can't be right.16:19.09 
chrisl marcosw: psdcmyk and psdrgb share most of their code, so changing one will change the other, the odd man out is psdcmykog - but not a big problem16:20.09 
rayjj mvrhel: sorry, I was off looking at the openembedded stuff :-/16:20.29 
marcosw chrisl: so that leads back to my original question, should the cluster use -dLastPage=1 with psdcmyk[og] files so that it only the first page is tested or do some messy stuff with -sOutputFile=temp.%d.psd? I'm voting for the former, since I think most of the interesting color stuff happens in specially constructed single page test files.16:25.57 
chrisl marcosw: Hmm, you're going to hate me, but I think the latter - many of the test suite files that have taxing color tests are multipage files :-(16:27.12 
henrys marcosw: oh I forgot to bring up the windows segfaults a couple of those looked bad, maybe P1's?16:27.56 
marcosw henrys: I didn't make them P1 since the segfaults are only in the 64 bit build and I don' t think anyone uses that on windows.16:29.21 
rayjj chrisl: marcosw: since you are concerned with cluster testing, and it does -sOutputFile='|md5sum' can't we just allow piping for psd... (not trigger the problem) and assume that the program we are piping to is smart enough to handle it ?16:29.23 
chrisl rayjj: it seems that doesn't work16:29.48 
rayjj marcosw: we have several customers that use the 64-bit gs on windos16:29.55 
  marcosw: mainly because their app is 64 bit and they want to call the 64-bit lib16:30.35 
  chrisl: I know that we can't write a valid psd that way, but what doesn't work -- does the PSD device need a seekable output file or something ?16:31.39 
marcosw rayjj: I always see 'gswin32c.exe' in bug reports but perhaps customers are just using the 32 bit build to duplicate the problem on the command line.16:31.57 
chrisl rayjj: You end up with only the last page - somehow......16:32.07 
henrys marcosw, rayjj, chrisl: I do really think P1 for these no?16:32.38 
marcosw chrisl: looking at the nightly regression code it's not too bad, not sure about the cluster...16:32.47 
kens2 I'm intrigued to know how that's possible16:32.48 
marcosw henrys: i agree, based rayjj's comment that we have customers using 64 bit.16:33.06 
kens2 I thnk the image one should be P1, it could be very general16:33.12 
chrisl I haven't a clue - I would have said impossible, but.....16:33.13 
marcosw chrisl: I'm pleased to have found one impossible thing before breakfast :-)16:33.44 
  (of course I actually found this several weeks ago and probably late at night, but nevermind).16:34.06 
kens2 I thnk we only have 1 seg fault ?16:34.24 
chrisl marcosw: I was about to say that this must be Thursday, I've always had problems with Thursdays..... but that's clearly wrong16:34.46 
marcosw It's the Apple Watch a digital watch?16:35.18 
kens2 aha the other culprit is GhostXPS16:35.21 
marcosw yeah, gxpswin64.exe is very broken.16:35.51 
henrys don't p1 anything for ghostxps16:35.53 
kens2 THat's the setjmp/longjmp in the PNG code. Probably needs Tor's attention16:35.56 
  THen there's only 1 GhostScript one16:36.10 
chrisl gxpswin64.exe has *always* been very broken16:36.10 
henrys kens2: so that should be reproducibly in mupdf?16:36.24 
kens2 henrys I rather doubt it16:36.34 
chrisl But I don't recall it crashing before16:36.35 
kens2 As chrisl pointed out before, GhostsXPS and MuPDF's XPS interpreter are no longer very similar16:37.04 
marcosw at some point in the past building a 64 bit gxps.exe resulted in a 32 bit executable, so there's that.16:37.25 
chrisl marcosw: No, I fixed that a *long* time ago. More recently, it just produced no output, ever16:37.50 
marcosw chrisl: so there was never a time that gxpswin64.exe produced correct output? (and was actually 64 bit).16:38.24 
chrisl marcosw: not that I can recall16:38.38 
rayjj marcosw: one customer we hear from that uses gswin64 is cust 38416:38.43 
marcosw that's sort of what I found with bisect16:38.57 
chrisl marcosw: which is why I never distributed a 64 windows binary of it16:39.09 
  64 bit windows binary16:39.23 
marcosw chrisl: maybe I should stop testing it :-)16:39.24 
kens2 feels it should work16:39.43 
chrisl The setjmp/longjmp is probably alignment related16:40.03 
kens2 And should be made to work. Wasn't norbert looking at it at on point16:40.04 
rayjj marcosw: also cust 21016:40.10 
henrys no way should we have an ghostscript bug out there, xps we can live with for a while16:40.22 
kens2 chrisl I'm uncertain, Its involving hideous macros so I gave up16:40.30 
chrisl kens2: I'm going by the fact we had a problem like that previously with the png output devices16:40.57 
marcosw rayjj: since valgrind suggests the http://bugs.ghostscript.com/show_bug.cgi?id=696320 crash is related to the clist I'll assign it to you. 16:41.02 
kens2 But I would not be surprised to find its an alignment problem16:41.07 
  marcosw that's a preliominary look by me, I'm not saying its a clist problem16:41.36 
marcosw kens2: valgrind produced clist related issues as well, so that's two half-hearted votes.16:42.15 
kens2 :-)16:42.23 
marcosw besides rayjj actually uses windows as his developement environment (and can hopefully build and debug 64 bit code).16:43.01 
  btw, what's all the talk about moving to VS2015? Chris just walked me through setting up VS2003 (or something like that) to do regression testing. 16:44.07 
kens2 I'm actually using 200816:44.18 
  But I can't debug 64-bit buolds with it, so I installed 2015 to see if it would work. WIth a considerable amount of prodding it does.16:44.48 
  Needs me to disable my firewall :-(16:44.58 
chrisl marcosw: we generally do try to make it possible to *build* with more recent versions16:44.58 
kens2 VS 2015 ooks dreadful. Can't see which line of code its executing, and it's search clutters up the whole page with coloured boxen16:45.35 
mvrhel_laptop kens2: I had no problems like that16:45.43 
  VS 2015 interface is not much different than 201316:46.02 
  or 2012 for that matter16:46.06 
kens2 I never used any of those16:46.13 
marcosw chrisl: yeah, well, when I tried to get VS2012 (?) working you suggested VS2005 (which is what I'm actually using).16:46.14 
kens2 I detest the 'flat' look, don't like the fact I can't change colours etc, etc16:46.33 
marcosw of course I'm also running on Windows 7, so presumably VS2015 isn't an option in any cae.16:46.41 
  ^cae^case16:46.51 
chrisl marcosw: Yes, because I felt that testing should (at least to start with) should be on the version we use for the binaries we ship16:47.02 
mvrhel_laptop the only issue I see with 2015 is that some of the extensions are not yet available16:47.05 
marcosw chrisl: sounded reasonable at the time and still make sense. just wondering why this sudden interest in vs2015.16:47.37 
kens2 I just don't like the look, and you can't change it16:47.39 
  marcosw : I'm runnig Windows 7 and VS 201516:47.57 
chrisl marcosw: current version fever16:48.07 
kens2 will only be dragged into the revolting flat look kicking and screaming all the way16:48.46 
chrisl I'm a little surprised VS isn't skinnable16:49.30 
kens2 If only......16:49.38 
rayjj sorry. My laptop power supply came unplugged and I had to reboot16:50.30 
  (it wouldn't come up from hibernate that it does with critical low power)16:50.59 
marcosw just to change the subject, I've set the fans on my computer to have a minimum rpm of 0, so they don't spin at all when the computer is idle. the cpu is water cooled and the pump never shuts off, so it's not 100% silent but it's really noticeable when a cluster run starts (like it did a few minutes ago).16:51.13 
mvrhel_laptop rayjj: so were you able to figure out what the OE stuff is about?16:51.51 
rayjj marcosw: on the cluster subject, has peeved been OK now ?16:52.04 
marcosw rayjj: yup.16:52.15 
  henrys: otoh, we haven't seen your new machine on the cluster.16:52.26 
rayjj mvrhel: it looks like ghostscript is part of the openembedded-core16:52.27 
  (probably because it's there for cups and printing)16:52.50 
  but it looks like a lot of other packages are 'meta-***' things 16:53.23 
mvrhel_laptop what version is it installing. I could not tell. MQ just sent me an email that –dNumRenderingThreads=4 was not having any effect 16:53.24 
  I will ask MQ what version he has..16:54.02 
rayjj mvrhel: I saw 9.15->9.16 as the most recent commit by Robert Yang last May: http://cgit.openembedded.org/openembedded-core/commit/?id=c5adb5903bd93f1c71acd52062d8c8620de2f94416:55.08 
  mvrhel: so presumably, that's what MQ pulled16:55.30 
marcosw chrisl: I will re-jigger the nightly regression code to use temp.%d.psd so multi-page psdcmyk files will work correctly. I'll presumably have to do the same thing on the cluster... 16:56.24 
  bbiaw, latte and breakfast just showed up.16:57.17 
rayjj mvrhel: w.r.t NumRenderingThreads, if the Outputpage start and end time are small, then the rendering won't be helped16:57.39 
mvrhel_laptop rayjj: ok that makes sense. He is running a 12 page doc that is finishing in less than 2 secs16:58.08 
rayjj mvrhel: -dBGPrint=true might help (overlaps parsing and rendering)16:58.36 
  mvrhel: *not* to be used with NumRenderingThreads16:58.54 
mvrhel_laptop rayjj: ok16:59.03 
rayjj mvrhel: but on simple pages without disk I/O (memory based everything), we are probably memory bandwidth (not CPU) limited17:00.31 
mvrhel_laptop ok that makes sense17:00.47 
chrisl marcosw: okay. Do you want me to hold off committing until you have that all setup? I must admit, I'll like to understand the issue with the piped output17:01.03 
kens2 discovers how to get rid of the stupid SDI tabbed interface and back to an MDI interface. THere may be more work to do hre. Also; discovered how to get rid of the shouting in menus17:01.04 
rayjj mvrhel: which is why I wanted to get a board and profile to see the hot spots17:02.04 
marcosw chrisl: if you could wait that would be good. if I do my change first, your change won't cause any md5sum differences.17:02.13 
kens2 Bizarrely every time I move focus away from my floating windows, they disappear, what's that all aout then ?17:02.15 
mvrhel_laptop rayjj: I will work on getting one17:02.20 
chrisl rayjj, mvrhel_laptop: I'm really not convinced that using a pre-packaged build is the best way to go for performance testing17:02.25 
  marcosw: not a problem17:02.31 
marcosw chrisl: I should be done later today (i.e. in the morning your time).17:02.36 
mvrhel_laptop chrisl: I agree17:02.38 
rayjj mvrhel: OK, thanks. I'll let you eat now :-)17:02.42 
mvrhel_laptop I already warned him17:02.43 
chrisl But we are due to get a board to play with?17:03.00 
mvrhel_laptop I think we are supposed to get a few17:03.10 
rayjj chrisl: I agree, sort of. It's using our makefiles (and configure)17:03.28 
Robin_Watts mvrhel: Or could he let us ssh in to his board? That way we can build for him.17:03.29 
chrisl Well, I guess we just wait until we get the hardware, and really see where we stand.....17:03.47 
rayjj Robin_Watts: he cross compiles it, AFAIK17:03.58 
marcosw chrisl: you will figure out why the last page is saved when using pipes, right?17:04.00 
chrisl marcosw: I am going to look at that, but I can't promise!17:04.21 
marcosw chrisl: good enough.17:04.34 
chrisl I was suspicious about the openembedded stuff as it *seems* to be building (at least) without freetype and without jbig2dec.....17:05.18 
rayjj Robin_Watts: all the openembedded stuff looks like it is made for cross compiling, and part of the "recipe" for gs plugs in an arch.h for the platform17:05.32 
  chrisl: without jbig2dec is probably OK for most files, but I was also puzzled about the freetype. Maybe it's relying on an so lib ?17:06.35 
chrisl rayjj: no, IIRC, they've --disable-freetype17:07.05 
rayjj chrisl: can we do text without freetype anymore ?17:07.13 
chrisl rayjj: yes, the old code still works - sort of17:07.42 
mvrhel_laptop oh that is not good17:08.16 
rayjj chrisl: OK. Because mvrhel mentioned (on the phone) that the pages look like they are coming out OK17:08.27 
mvrhel_laptop well from the command line output17:08.38 
  I have not seen any pages...17:08.43 
henrys marcosw: are you seeing any error on your end for my machine? I thought I did everything.17:08.54 
mvrhel_laptop this is out to the null device17:08.55 
rayjj chrisl: iirc, freetype was a little bit faster than our "AFS"17:08.57 
chrisl rayjj: for some cases17:09.20 
  The output probably will look okay for well formed, well behaved fonts17:09.32 
marcosw henrys: I'm not seeing your machine at all. it's never sending the 'I'm alive' signal to the cluster.17:09.46 
rayjj mvrhel: Oh, you were just going from the lack of error messages ? So, it could be generating blank pages :-/17:09.47 
mvrhel_laptop it could. how am I suppose to know with the null device17:10.16 
marcosw henrys: is the clusterpull.sh script running?17:10.24 
kens2 GOodnight all17:10.35 
Robin_Watts henrys: you here ?17:10.40 
mvrhel_laptop rayjj: I will have him go out to a different device to make sure all is ok17:11.00 
rayjj mvrhel: he can run it to something else (local SD card file? ) and look at it (maybe use png16m)17:11.16 
mvrhel_laptop yes17:11.20 
henrys marcosw: yes it is17:11.21 
  Robin_Watts: yes I am17:11.25 
Robin_Watts henrys: There is a customer reported bug, that chrisl traced to a clipping problem.17:11.46 
rayjj mvrhel: I usually check the output independently of doing the timings17:11.54 
Robin_Watts The problem appears to be in scan_contour17:11.58 
chrisl Robin_Watts: those scan_contour changes seem to improve the clarity quite a lot - and comments.... crumbs!17:12.03 
Robin_Watts which is a function of igors, that he claims is a "revised and much improved" version of something you wrote.17:12.33 
marcosw henrys: could you try running './heartbeat.pl <machineName>'?17:12.45 
henrys Robin_Watts: which bug is this?17:13.13 
Robin_Watts Now, putting aside the fact that "revised and much improved" from igor generally means "obfuscated and broken", do you remember anything about this code?17:13.16 
  http://bugs.ghostscript.com/show_bug.cgi?id=69630517:13.28 
henrys I was just looking at it a couple days ago and now I've lost it17:13.30 
marcosw henrys: never mind, I think I found the (or at least a) problem.17:15.04 
henrys Robin_Watts: No I don't but these guys need to be told we don't expect good progress until Q1 2016. I know that's tough but that is the right call priority wise.17:16.13 
marcosw oops, better wait until Robin_Watts' cluster run finishes before I do anything bad to the cluster.17:16.45 
henrys marcosw: did I screw something up? At the end of this exercise I would like to have an engineer be able to add a machine without help beyond the readme.17:17.42 
marcosw henrys: i don't think it's anything you did. 17:18.20 
henrys chrisl: I meant to ask about shelly, I didn't hear back on my last review. I hope I'm not being difficult17:19.53 
mvrhel_laptop ok. before I dig into mupdf I want to wrap up a couple gs bugs17:20.33 
chrisl henrys: he's probably got day job stuff going on17:20.35 
mvrhel_laptop including 69632117:21.11 
henrys rayjj: it's going to be really weird to walk into my old office with Miles and Scott. 3 colleagues still remain from the old QMS group.17:22.21 
marcosw henrys: okay, I fixed one problem. The cluster monitor script on casper hadn't been restarted so wasn't accepting connections from your machine (this step is documented in the readme, I just didn't follow the instructions).17:23.21 
  henrys: so now I can sent hearbeats that impersonate your machine and they show up, but your machine doesn't appear to be sending any. Could you try 'heartbeat.pl henrysi7' from the cluster directory?17:24.14 
henrys marcosw: I guess I did screw something up I didn't copy that over.17:25.21 
marcosw you don't need to, clusterpull.sh does it.17:25.33 
  the line is "scp -i ~/.ssh/cluster_key -q regression@cluster.ghostscript.com:/home/regression/cluster/heartbeat.pl ."17:25.47 
  or are you running an old clusterpull.sh?17:26.11 
henrys heartbeat.pl had no output.17:27.32 
marcosw that's good, no error message :-)17:27.50 
  and I do see the updated timestamp on casper.17:28.03 
henrys marcosw: I just copied clusterpull.sh as per the readme17:28.29 
marcosw so that begs the question why clusterpull.sh didn't copy that file over.17:28.31 
rayjj henrys: are you going to try and pitch them GhostPDL for their printers :-)17:28.49 
  marcosw: if the host hasn't been accessed from a new machine, it won't be in the hosts list, and may error out17:30.05 
henrys rayjj: no the plan is to tread lightly, mupdf as an add on for GDI printers, Smart office. Selling the language group a rip seems like a non starter to me.17:30.08 
rayjj henrys: yeah, an uphill battle at best17:30.39 
henrys marcosw: in the log here I have a test run from robin?17:30.41 
Robin_Watts henrys: I'm running a test run now.17:31.04 
marcosw rayjj: true, but the readme instructs the user to run an `ssh touch` and answer 'yes' to the question. 17:31.04 
  henrys: henrysi7 does seem to be running on the cluster now.17:31.19 
  though it failed in some way, let me check the log17:31.56 
rayjj marcosw: sounds reasonable. I hadn't read the readme17:32.20 
Robin_Watts marcosw: First test run probably failed to update everything before everything else finished. (it was a bmpcmp run, so very quick)17:32.29 
marcosw Robin_Watts: that's possible but doesn't quite match the symptoms. 17:33.33 
henrys marcosw: I'll be leaving in a bit for cross fit class, but I'll check the logs and email when I get back.17:38.22 
marcosw henrys: that would be great, I'll be around most of the day.17:38.56 
mvrhel_laptop Robin_Watts: I am wondering if I can throw bug 694916 to you. A problem is that a call dest_height = fixed2int_var_rounded(any_abs(penum->y_extent.y)); in image_render_color_thresh is overflowing its sign by the macro fixed2int_var_rounded. Unsigned int seems to work ok. I will put the details in the bug. I am sure you don't have time to get to it, but this fixed point macro...17:44.04 
  ...stuff is more up your alley. 17:44.06 
Robin_Watts mvrhel_laptop: Sure.17:44.48 
mvrhel_laptop rayjj: so looks like MQ has 9.0717:48.45 
  in his open embedded build17:48.59 
  so I suspect -dBGPrint is not available17:49.23 
  when was -dNumRenderingThreads added?17:49.43 
Robin_Watts -dNumRenderingThreads has been there for longer than -dBGPrint.17:50.24 
chrisl It predates 9.07 by quite a long time17:50.34 
mvrhel_laptop ok I thought so17:50.40 
  but the background thread overlap of the interpreter and the rendering is not there17:51.05 
chrisl But we have made a lot of performance improvements since 9.0717:51.10 
mvrhel_laptop yes17:51.13 
  I am going to push him to get us some boards17:51.20 
chrisl For a case like this, not using the ICC color workflow is probably a big one17:52.30 
mvrhel_laptop right. I dont think the fast color was available then either17:52.52 
chrisl Actually, UseFastColor should be available in 9.07.17:56.38 
mvrhel_laptop oh ok18:03.32 
  oh great. they have run out of boards...18:04.10 
marcosw feels a bit like a character in "The Martian", I'm remotely debugging henrys new cluster node by modifying the part of the code that appears to be working to send back debugging information on the parts that aren't.18:14.55 
mvrhel_laptop brb18:17.12 
marcosw Robin_Watts: you probably received an email but your cluster job failed because you are writing out too much debugging information (i.e. scan_contour (early): 40cd,40cd->40cd,28800). To avoid filling up the disk the cluster abort when the log gets too big.18:35.47 
Robin_Watts marcosw: Yeah, that was a slip, sorry. I aborted and reran.18:36.06 
marcosw I was worried that my fiddling with the cluster code broke something. I tend to do that :-)18:36.39 
Robin_Watts too.18:38.13 
  /* q stands at the first segment, which isn't last. */18:43.39 
  Top comment.18:43.43 
chrisl It's hard to argue with, though.....18:44.40 
Robin_Watts in order to argue with it, you'd have to be able to understand what it was claiming.18:45.40 
chrisl Maybe it makes sense in Russian18:46.07 
Robin_Watts Ð’опрос стоит на первом сегменте, который не продлится.18:46.56 
  That makes no less sense, it's true.18:47.08 
  I have a fix for that bug.18:48.26 
  Let me tidy the commit up.18:48.58 
chrisl There's another file that would be worth trying.......18:49.14 
  tests/ps/bigpaths.ps18:49.57 
mvrhel_laptop tor8: so I was going to start a pdf_create branch for this work. I guess I should make a pdfcreate_main addition to the mutool and then work from there with starting out with a blank pdf, and then add pages, content and annotations18:59.04 
  does that sound reasonable18:59.37 
  I assume that as I start down this road it will be clear what needs to be fixed/cleaned up in the core19:00.05 
  and simplified19:00.17 
chrisl I'm heading off.... g'nite all!19:00.57 
mvrhel_laptop good night19:01.03 
Robin_Watts mvrhel_laptop: Personally, I'd not work in the way tor suggests.19:01.54 
  If our intention is to make the pdf writing device work, then we can start much more simply.19:02.23 
  mutool draw -o out.pdf in.xps19:02.37 
  That's enough to demonstrate the limitations in the pdf output device.19:02.57 
  and to immediately see improvements as you extend it.19:03.14 
mvrhel_laptop Robin_Watts: I see so basically writing pdfwrite to handle all the fitz graphics commands19:03.52 
Robin_Watts The pdfwrite device is there already.19:04.15 
  It's just incomplete.19:04.19 
  It's fill in the blanks coding.19:04.25 
mvrhel_laptop that does sound like it would be a better option19:05.01 
Robin_Watts There is other work that needs to be done that wouldn't be immediately shown by this approach (like adding pages from one PDF to another) etc.19:05.41 
mvrhel_laptop Robin_Watts: ok that is what I was just going to ask19:05.57 
Robin_Watts but the pdfwrite device is the biggest incomplete piece, I reckon.19:06.06 
  I'm not sure I see the independent value of a tool that takes a content stream and wraps it into a PDF.19:06.35 
mvrhel_laptop Robin_Watts: looking at what the competitor has though, being able to drop in various annotations and content is going to be be needed19:06.44 
Robin_Watts especially as that will fail to cope with resources like images/fonts etc.19:06.46 
  mvrhel_laptop: Imagine opening a pdfwrite device, then running an existing pdf page to it.19:07.33 
  Then running extra content to it.19:07.41 
mvrhel_laptop Robin_Watts: yes.19:07.51 
Robin_Watts That's a mechanism for getting extra stuff onto a page.19:07.58 
mvrhel_laptop I can see that19:08.14 
  I would prefer to leverage what you already have19:08.28 
  let me look over the pdfwrite stuff19:08.35 
Robin_Watts Now, I think one of the issues with the pdfwrite stuff was that there isn't quite the information within the fz_font that we'd like.19:09.12 
  We need to be able to output the font data to embed fonts etc, for example, and that is not readily available.19:09.37 
  So it will probably require some rejigging of the fz_font/fz_fontdesc stuff.19:09.52 
  but I'm sure tor8 will have ideas on that.19:10.21 
  I think he's got designs on some of the fz_text stuff as it is.19:10.35 
captainralf hi all, is this just a dev channel or support as well?19:11.52 
Robin_Watts captainralf: Depends on our mood :)19:12.35 
  captainralf: Is this for gs?19:12.46 
captainralf lol, fair enough and yep for gs19:13.12 
Robin_Watts What's your problem?19:13.22 
captainralf well, my friend has a bash script that calls gs to merge a few docs. but he can't send a string with several document paths in it. Tried it from commandline as well, but gs can't find the files, even though the paths look fine in the error output. Shouldn't be a bash issue...19:19.10 
  he doesn't use irc, that's why i'm acting as proxy here ;)19:19.27 
Robin_Watts OK, so for us to look at it, he'd have to do a bit of legwork.19:20.56 
  We're not going to debug his bash script.19:21.05 
  But if he can give us a command that we can run that goes wrong, we can try to help out.19:21.27 
  So basically if he can figure out the exact command that his bash script is doing, and try that on the command line, if it goes wrong, we'll help.19:21.59 
  I'm guessing he's doing something like:19:22.08 
  gs -sDEVICE=pdfwrite -o out.pdf in1.pdf in2.pdf19:22.23 
  It's also worth noting that that does not 'merge' the PDF files. It makes a completely new PDF file that is unrelated to the input ones other than the contents should hopefully look almost identical :)19:23.13 
captainralf yeah of course, no problem. he did try on commanline, but it gave him the same error. he's going to send the command+output in a sec19:23.45 
  oh that's interesting. never had any issues using gs to merge. is there an actual merge program out there? I assumed most GUIs just use gs in the background in a similar fashion, except those creating image based docs only19:25.03 
  this is the commandline he uses in the script: gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -dPDFSETTINGS=/prepress -sOutputFile="$mergedFile" "$sourceFiles"19:26.08 
Robin_Watts captainralf: OK. That's got bash bletcherisms in it.19:26.40 
  For a start, I'd guess that "$sourceFiles" is not what you want.19:26.57 
  "in1.pdf in2.pdf in3.pdf" is not the same as "in1.pdf" "in2.pdf" "in3.pdf"19:27.14 
tor8 mvrhel_laptop: yeah, that sounds reasonable.19:27.18 
henrys marcosw: well I put it under my user name, if you think you are going to need access I'd rather change the user.19:27.49 
Robin_Watts guesses tor8 is reading the logs... 5...4...3...2...1...19:28.00 
marcosw henrys: I found and fixed the problem with your new machine not working as a clusternode. The code had a 'heartbeat.pl` in it, which worked on the other machines because '.' is in the path. I fixed it so now it does './heartbeat.pl'.19:28.00 
tor8 Robin_Watts: mvrhel_laptop: yeah, reading logs.19:28.06 
  Robin_Watts: mvrhel_laptop: my thinking was that my original idea would be a warming up for filling in the blanks in the pdfwrite device19:28.23 
  and at the same time create a top level api for how the pdfwrite device can be used19:28.40 
marcosw as I mentioned earlier I did it by modifying the parts that worked to send log files and debugging output to casper. Probably wouldn't have taken less time if I had been sitting in front of your machine.19:29.08 
Robin_Watts We have code to create new pages, I think.19:29.32 
tor8 the way mudraw -o foo.pdf uses the pdfwrite device is probably too limited for the general use we envision19:29.45 
mvrhel_laptop tor8: so you think adding a create option in mutool is reasonable and have that leverage pdfwrite19:29.46 
tor8 I'm thinking robin and my "projects" are separate things19:30.13 
Robin_Watts I think merging pages from PDF to PDF shouldn't need pdfwrite at all really. If we use pdfwrite we're as bad as gs.19:30.44 
tor8 my original thought would be that the pdf page and annotation creation stuff would be independent of pdfwrite (it just takes a ready made content stream and creates all the necessary page tree objects etc)19:30.56 
captainralf Robin_Watts: yep, I agree. he told me that if he uses individual variables for each file it works, but as the list can get quite long, so he was hoping to avoid it(as this is basically the point of the script). Also, it didn't work if the $sourceFiles had "" around each individual file.19:31.11 
tor8 it needs to sort juggling resource dictionaries, etc.19:31.16 
marcosw There is still an issue with your machine and cups but I've seen this before so should be easy to sort (as I recall some package is missing, presumably the readme is incomplete in the list of packages that need to be installed).19:31.29 
tor8 so copying pages between pdf files would just mean pulling the resources over as well, not re-interpreting with the pdfwrite device19:31.38 
Robin_Watts Anything that takes a "ready made content streams" is essentially useless though.19:31.39 
henrys marcosw: I may have been speed reading through that part too ;-)19:32.09 
Robin_Watts tor8: I have some code to copy resources between trees.19:32.14 
tor8 input a ready made content stream + resource dictionary, insert into a page tree.19:32.17 
marcosw no, that's like the second step.19:32.28 
Robin_Watts That's in the pdf operator filtering stuff I think.19:32.30 
tor8 then we plug in the pdfwrite device to create the content stream and resource dictionary19:32.34 
  separate lego bricks; and that frees the pdfwrite device to not have to worry about whether it's creating a page or annotation stream19:33.15 
henrys marcosw: I think I was lacking libcupsimage2-dev which is not installed19:33.55 
  marcosw: I think I was lacking libcupsimage2-dev which is now installed19:34.04 
marcosw henrys: right, ghostscript is reporting "Unknown device: cups". This happens because ./configure doesn't find the cups libraries. BTW, what distribution are you running? 19:34.08 
  okay, I'll try again.19:34.16 
tor8 Robin_Watts: it would be nice to have those bits of code accessible as independent functions we can use without being tied to a specific work flow19:34.27 
Robin_Watts tor8: It would.19:34.36 
henrys marcosw: 15.0419:34.39 
Robin_Watts I mention it not to say "it's there and finished", but just "there may be something you can refactor"19:34.58 
tor8 Robin_Watts: fair point.19:35.07 
marcosw henrys: me too (at least on the cluster nodes in my home office), the ones in my garage and miles' office run 14.04 LTS19:35.20 
  which means I get to deal with updating them in 6 months :-(19:35.40 
tor8 "mutool create" would not be a very useful tool on its own, but as a developer tool it wouldn't be completely useless19:35.44 
  mutool create which creates font and image resources from command line arguments would be handy for concocting test files19:36.41 
Robin_Watts Yes.19:36.58 
mvrhel_laptop tor8: so, are we thinking that I should start with doing the mutool create and see about adding in content. Then when all the content has been "added" one would use pdfwrite to save it.19:37.00 
tor8 mutool create -o out.pdf -f font.ttf -i image.jpg contents.txt19:37.11 
Robin_Watts mutool create -o out.pdf -f F0 font.ttf -i I0 image.jpg contents.txt19:37.41 
  where contents.txt uses F0 and I0.19:37.51 
tor8 the pdfwrite we are talking about is something different, it's a fz_device that takes graphics commands and creates a content stream only.19:37.51 
mvrhel_laptop ok19:38.08 
tor8 we already have code to write out pdf objects and streams19:38.09 
  it's creating said objects and streams to make a brand new pdf that's missing parts19:38.22 
Robin_Watts mvrhel_laptop: There are 2 aspects to this work. The first is the actual pdf device (that I was talking about). The second is the fiddling around inside a PDF file to add new pages etc.19:38.51 
mvrhel_laptop so in the above what exactly is in contents.txt19:38.52 
tor8 mudraw with pdfwrite creates some of it, mixed into the pdf content stream creation. that's what I want you to separate out.19:39.05 
Robin_Watts I think tor was thinking you'd tackle the second part first.19:39.12 
  contents.txt would be something like: 1 0 0 1 100 100 cm /F0 10 Tf "Ass Biscuits" Tj 0 0 1000 1000 re f19:40.05 
mvrhel_laptop ok19:40.11 
tor8 Yes. Fiddling with the PDF files to add pages etc. It's something I believe we get a few requests for help in how to do and it's also a lot easier to start with.19:40.19 
marcosw henrys: your machine is pretty quick, only my i7, yards, hubbles, and points are faster at compiling ghostscript. If you installed ccache I think you might beat them as well.19:41.00 
mvrhel_laptop so in the above example of , mutool create -o out.pdf -f F0 font.ttf -i I0 image.jpg contents.txt would we ever want to start with an existing pdf (or xps) content and add an additional page of content?19:41.57 
Robin_Watts mvrhel_laptop: That's a different use case, not covered by mutool create as I understand it.19:42.28 
tor8 that would come later, in another tool, I imagine19:42.38 
Robin_Watts mutool merge -o out.pdf in1.pdf in2.pdf19:42.49 
tor8 mutool create should be a very small program, once the library interface is up to snuff19:42.54 
mvrhel_laptop ok that sound manageble19:43.02 
  sounds19:43.07 
Robin_Watts mutool merge should not need pdf write.19:43.10 
mvrhel_laptop right19:43.13 
Robin_Watts Nor would mutool create for that matter.19:43.26 
marcosw we should treat pdf files like psd files and just cat them together :-)19:43.30 
tor8 eventually, once pdf write is in better shape, "mudraw -o out.pdf one.pdf two.pdf three.xps" should be able to convert formats and merge files like the current gs pdfwrite workflows people always abuse19:44.11 
mvrhel_laptop I would imagine though that mutool create would be doing some of the same things that are going on in pdfwrite now19:45.01 
  for the final output19:45.08 
  or am I mistaken19:45.14 
tor8 you would be creating some of the same objects and streams, but the actual saving to disk part is done19:45.40 
  and both pdfwrite and mutool create should be calling the same functions to do the work of setting up pages etc19:46.03 
Robin_Watts mutool draw -o out.pdf in.whatever calls functions to create a blank PDF, and add a blank page to it, then populates the page.19:46.04 
  You'll be doing the same, just populating the page in a different way.19:46.18 
marcosw henrys: installing the cups library fixed the problem. 19:46.55 
mvrhel_laptop so there may be some refactoring for pdfwrite needed19:47.27 
tor8 mvrhel_laptop: not may, will :)19:47.36 
mvrhel_laptop some of the statics that are in will need to be opened up19:47.49 
Robin_Watts Yes. A lot of this work is refactor and extend.19:47.50 
mvrhel_laptop ok19:47.50 
tor8 also, don't feel wedded to the current functions at all. nobody but us should be using this part of the code.19:48.05 
Robin_Watts simon91 might be :)19:48.15 
tor8 like robin just said, the job is mostly refactoring and extension19:48.31 
Robin_Watts But the MuPDF way is to rip up and relay the tracks if it gives a nicer layout in the end.19:48.49 
tor8 Robin_Watts: true. but he should be happy that we are giving him proper tools to work with. in the end. :)19:49.05 
mvrhel_laptop ok this sounds reasonable. I suspect I will make a mess of things to being with. I think I am familiar enough with the mupdf style to keep things clean and clear but let me know if things are not making sense or if I do a no no.19:51.36 
  s/being/begin/19:51.46 
Robin_Watts Whatever you do, tor8 will find a way of simplifying it. I've learned to live with that :)19:52.23 
mvrhel_laptop yes19:52.28 
  A big concern is making sure I don't rewrite something that is already there19:52.56 
Robin_Watts feel free to run stuff past us and we can have a quick grep :)19:53.21 
mvrhel_laptop ok sounds good. thanks19:53.30 
tor8 Always a concern with a sprawling code base. Don't hesitate to ask here if something exists.19:53.54 
henrys marcosw: wow the clusters were running and I didn't hear a thing. These fans are pretty nice...19:53.57 
Robin_Watts It's probably worth stepping through mutool draw -o out.pdf tiger.pdf in the debugger.19:54.10 
tor8 and if it exists, but in a slightly wrong shape, don't worry about fixing it so it works for new code either.19:54.22 
  rather than duplicating it19:54.26 
Robin_Watts henrys: yeah, silent fans are awesome. My machine is way quieter than my last one.19:54.36 
henrys Robin_Watts: did you open it up and have a look at them? pretty cool19:55.13 
  no pun intended19:55.33 
Robin_Watts henrys: Yeah, I've had my machine open to add a 1TB SSD.19:55.47 
mvrhel_laptop tor8: ok thanks19:56.27 
marcosw what model fans are they?19:56.34 
mvrhel_laptop off to lunch 19:56.44 
Robin_Watts To go along with the 512GB M2 PCI-e SSD that the OS is on, and the 4TB slow hard drive for bulk data.19:56.48 
  marcosw: http://www.scan.co.uk/products/fractal-design-arc-midi-r2-black-mid-tower-quiet-performance-case-with-side-window-usb-30-w-o-psu19:58.31 
  http://www.scan.co.uk/products/noctua-nh-u14s-ultra-quiet-slim-cpu-cooler-with-140mm-nf-a15-fan-for-all-intel-and-amd-cpus19:59.38 
henrys marcosw: for the cpu a gelid tranquillo rev320:01.08 
marcosw I think the cpu cooler is a newer version of what I'm running in yards. I switched to water cooling on my main machine to better handle overclocking. 20:01.33 
tor8 I used to swear by noctua fans, but the latest batch I got were all much noisier than their old models :(20:01.46 
Robin_Watts chrisl: tests/ps/bigpaths.ps is in the cluster tests, right? I guess we must be working with that :)20:14.13 
  OK, 2 commits on robin/master for review.20:14.31 
  oh, whoops. let me update the commit message on one of them.20:14.47 
  OK, fixed.20:15.43 
 Forward 1 day (to 2015/11/04)>>> 
ghostscript.com
Search: