IRC Logs

Log of #ghostscript at irc.freenode.net.

Search:
 <<<Back 1 day (to 2013/03/24)2013/03/25 
sebras I'll probably be more active here during the easter... overloaded with pointless assignments at work at the moment. :-P09:50.26 
Robin_Watts sebras: If you want, we can load you up with some pointless mupdf contract work to ensure your work/fun balance? :)11:47.17 
tor8 Robin_Watts: are we building for x86 or amd64 on the cluster tests?11:54.44 
Robin_Watts tor8: yes.11:55.05 
  but I don't know which :)11:55.28 
  Whatever a standard 'make' gives us on the cluster nodes, I think.11:55.57 
chrisl Robin_Watts: didn't marcosw say it was amd64 on Linux and x86 on Mac?11:57.17 
Robin_Watts chrisl: Could be, yes.11:57.25 
  x86 vs amd 64 isn't causing us problems. FP is :(11:57.45 
tor8 -ffloat-store may be a workaround for the clusters11:57.51 
  or -msse2 on the x8611:58.04 
Robin_Watts That's why marcosw disabled the macs doing mupdf cluster tests.11:58.06 
tor8 Robin_Watts: FP instructions used differs between x86 and amd64 by default11:58.29 
Robin_Watts tor8: bug 693467...11:58.40 
  ah.11:58.41 
tor8 with -msse2 they should both use sse2 floating point math on x8611:58.43 
  -msse2 is default on amd6411:58.51 
Robin_Watts I'm looking at the gentoo bugs link, and I can't find how to just download a copy of the whole bloody file. Am I being myopic?11:59.37 
tor8 Robin_Watts: the debian custom is to not ship debian-specific packaging files upstream11:59.52 
  we do to make it easier for them, but maybe we should revisit that12:00.18 
Robin_Watts tor8: but there is no problem in us grabbing them, right?12:00.20 
tor8 I don't mind us having a debian/ directory, but it should be a good one if we do.12:03.19 
  not sure about the copyright issues by us grabbing them back from debian's package12:03.37 
paulgardiner Robin_Watts, tor8: some commits on paul/master need reviewing when you have time.12:04.20 
Robin_Watts tor8: As far as I can tell, the changes to the file (at least the ones given on the gentoo bug) are so tiny and obvious to be fine, I think.12:05.09 
chrisl tor8: I can always zap the debian/ directory when I do the commercial release archive, if that eases the copyright concern?12:05.32 
Robin_Watts http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-text/mupdf/files/mupdf-1.1_p20121127-desktop-integration.patch?revision=1.1&view=markup12:05.33 
tor8 chrisl: that would work too, but if what robin says is true we likely don't need to12:06.23 
chrisl tor8: okay - as long as I know.... :-)12:06.42 
Robin_Watts chrisl: It seems to me, that it certainly wouldn't HURT for you to zap it.12:07.02 
chrisl Robin_Watts: but it's (currently) entirely ours, so there's no need12:07.30 
Robin_Watts tor8: I think paulgardiner was happy with the reflow stuff. (is that right?) have you had a chance to look?12:08.00 
paulgardiner I have looked and there was nothing not to like, but I can't claim to have been very thorough. I'd need to take some considerable time over it to understand the algorithms and data structures completely.12:09.20 
tor8 paulgardiner: I'd name "get_color" something more appropriate, the rest LGTM12:10.05 
Robin_Watts tor8: You've looked at all 5 ?12:10.34 
  ooh, I bet those will conflict with my reflow commits.12:11.22 
tor8 Robin_Watts: yes, but another set of eyes on the java won't hurt12:11.28 
Robin_Watts cos you've abstracted outthe popup message stuff too.12:11.44 
paulgardiner tor8: yes, got a suggestion for a better name?12:11.57 
tor8 paulgardiner: pdf_to_color to go with the to_rect, to_matrix, etc12:12.41 
  ?12:13.00 
paulgardiner Robin_Watts: ah yes. :-( I was expecting that to conflict when I rebased thinking your popup thing was already in. I forgot about it when the conflict never happened12:13.07 
tor8 Robin_Watts: I looked at the three oldest before my attention started to wander12:13.26 
Robin_Watts paulgardiner: no wirries. one of us would have had to do it, and it might as well be me.12:13.39 
  wirries? It must be minday.12:13.51 
paulgardiner I prefer your name for the function12:14.14 
Robin_Watts So an ink annotation is a list of points?12:15.45 
  and to draw it, we join 'em up?12:15.58 
paulgardiner It's a list of lists and we join all the subpoints12:16.38 
  join all the sublists12:16.49 
  For the C interface, since ragged arrays require multiple allocations, I used a single array of points and and array of counts (lengths of sublists12:17.53 
  I was tempted to use a pdf_obj array of arrays, but that would have required going outside the fz_ space (maybe a reason to again consider Tor's idea of removing fz_interactive and allowing direct access to the pdf_ api)12:19.19 
Robin_Watts no, I like your interface.12:20.13 
paulgardiner Ok good12:21.58 
  ... it was just that the use of pdf_obj would have avoided one stage of conversion. At the moment it's Java PointF => fz_ C single array and counts => mupdf internal C pdf_obj array of arrays12:23.27 
  JAva PointF[][] I mean12:23.46 
Robin_Watts Electrician here, so there is a risk I may fall off the net.12:33.41 
paulgardiner Robin_Watts: any problems in the commits?12:34.16 
Robin_Watts still looking.12:34.24 
  trying to spot a lack of error checking in the jni and failing.12:34.43 
paulgardiner I'm sure it's there but well hidden :-(12:35.15 
  :-) I mean12:35.19 
Robin_Watts I think we need fz_var(counts) and a free of the counts ?12:35.24 
paulgardiner Ah not that well hidden12:35.39 
  Oh. Nicely spotted. Leaks even in the non-error case at the moment12:36.44 
Robin_Watts Other than that, they all look good.12:36.53 
  tor8: any idea when you'll get a chance to look at the reflow stuff?12:37.06 
paulgardiner I'll fix the leak and change the name of get_color.12:37.26 
Robin_Watts I've been talking to a mupdf customer via skype for the past few days. They wanted the caching of tiling bitmaps, and I've implement that. That's all reviewed and in the repo.12:39.18 
  They spotted a problem with the store bookkeeping in that the store thought it was full when it wasn't.12:39.35 
paulgardiner tor8: so I guess I should move pdf_to_color into pdf_parse.c and make it global?12:39.37 
Robin_Watts paulgardiner: Needn't be global until we need it to be.12:39.54 
  This fixes the immediate problem: http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=04b6107617a0d9a0784e1d3d4723018d2a22838612:40.35 
paulgardiner Robin_Watts: it was just that tor8 was likening it to pdf_to_rect and pdf_to_matrix and they are global.12:40.49 
Robin_Watts But it still leaves a problem.12:40.53 
  paulgardiner: Right. Keeping the naming in line makes sense so we can make it global easily if we ever need to.12:41.17 
tor8 paulgardiner: no need to move it and make it global, we can do that if someone else uses it. a "global" name is still good.12:41.30 
paulgardiner ok will do12:41.58 
tor8 Robin_Watts: I'm looking through them now12:42.00 
  Robin_Watts: the text structures look fine12:42.09 
  is_unicode_ws can possibly (if a bit more slowly) be done with the unicode database I have sitting around12:43.04 
Robin_Watts The problem arises, because when we get to the tile_end, we put the tiles into the store. If they are ALREADY in the store, we spot that they exist already, and I had previously been forgetting to remove the count for the new copy from the store.12:43.36 
tor8 Robin_Watts: right, so that's what happened with the second of your robin/master patches12:44.20 
  the oldest one (the fp fix) I'd rather try solving with proper compiler flags first12:44.52 
Robin_Watts The leftover problem (after my patch above) is that if we try to put X into the store, we first free enough from the store to fit X in. Then try to put X in. If X is already in the store, we spot it at this stage. This means we've already thrown store entries out when we didn't need to.12:46.08 
  tor8: Yeah.12:46.13 
  (I mean, I agree about the FP stuff. I'll move that onto a local branch)12:46.41 
tor8 Robin_Watts: you have large chunks of duplicated code in the bullet point detection12:46.47 
paulgardiner tor8: that fp one: I'm still struggling to understand how an optimisation could cause it. I can only think that the CPU instruction itself is giving a different result depending on the order of the arguments12:47.05 
tor8 paulgardiner: could be x86 intermediate floating point registers being 80 bits wide12:47.29 
  and not truncating them to gain performance12:47.36 
  or the order of execution differing12:48.00 
paulgardiner I couldn't see a way with that expression to alter the order.12:48.21 
tor8 could alse be turned into a madd instruction if such things exist12:48.46 
paulgardiner truncation is a possibility, although I thought the C standard defined all that stuff.12:49.07 
Robin_Watts In order to do the store stuff properly, I'd either need to: 1) search the hash table first to see if it already exists before freeing things to make room for it (which is nasty, cos it takes extra time in cases that doesn't matter)12:50.04 
tor8 order seems unlikely, yes. but consider: t0 = b - a; return a + t0 * x;12:50.12 
  that a + t0 * x might be done as a multiply-add instruction on amd64 (just hypothesizing here)12:50.37 
  Robin_Watts: how often do you see yourself adding a new tile to the store that the extra lookup would hurt?12:51.27 
paulgardiner madd is an interesting possibility, although you'd hope it would give the same result.12:51.36 
tor8 paulgardiner: it doesn't on GPUs :)12:51.47 
  but GPUs generally don't give a squat about precision...12:52.08 
Robin_Watts or 2) put the thing into the store early (and spot any copies there) and keep a pointer to the thing in the store. Then try to free any space required, and if it can't work, remove us quickly from the hashtable using the pointer.12:52.23 
tor8 hence all the problems with CUDA and OpenCL not giving good results12:52.25 
paulgardiner Really :-) Anyway I'll shut up about it now: Robin's current problem looks far more important12:52.28 
Robin_Watts tor8: It would be on *every* insertion, not just fz_object ones.12:53.09 
tor8 Robin_Watts: just to rewind a bit about how the store works12:53.16 
  for other objects (fonts, for instance) we first do a lookup to see if it exists before we even start parsing12:53.35 
  so the problem here is that we parse the tile before we look it up and/or insert it in the store?12:54.06 
Robin_Watts tor8: Right. in begin_tile we 'fz_find_object' to see if is there.12:54.11 
  If it's there, we keep a hold of the pixmaps that are part of the stored object, but not the stored object itself.12:54.43 
  Then in 'end_tile' we make an object with the pixmaps in, and store that back into the store.12:55.14 
  (We do this regardless of whether we found it in the store or not, because I don't have that information).12:55.47 
  If we find it's there already, that's fine, we just live with the existing one.12:56.07 
  I *could* make it so that we only store the tile back if we didn't get it from the store already (at the cost of a bit more complexity in the draw device)12:56.42 
tor8 as an aside, what did we decide about ownership on find? it returns a reference counted object that you have to drop when you're done, as opposed to lookup?12:56.47 
Robin_Watts yes12:57.03 
  BUT that doesn't entirely solve the problem, as we can hit this same case in multi-threaded rendering of other objects.12:57.30 
  (say 2 threads need an image, both fail to find it in the store, both decode it, then both try to insert it in the store. Only one should end up leaving it in the store.)12:58.26 
  I think the correct solution is to implement 2), but that requires a slight change to the hash table code, so I thought I'd float it here for comments first.12:59.16 
tor8 is the only problem the store size updating?13:03.01 
  we already know we may have races to insert in the multi-threaded case and decided not to worry about that since we'd just drop one of them13:03.30 
Robin_Watts tor8: The size is all fine now.13:04.05 
tor8 ahem. maybe I should read the message that I didn't see first.13:04.28 
Robin_Watts The problem is that if the size of the object we are trying to insert twice is large, we can evict quite a lot from the cache when we really don't need to.13:04.37 
tor8 yes. I just read that. it scrolled past so I didn't see it then I was wondering what problem you were really talking about13:05.16 
  we could insert first, then evict?13:05.38 
  except we don't want to evict what we just inserted I guess13:05.49 
  how does the eviction pick what to toss?13:06.00 
Robin_Watts inserting then evicting would work fine, as we have references to the thing that's been inserted, so it won't be tossed.13:06.32 
tor8 we've already allocated the memory for the big thing that would evict others before inserting it, so we won't be in the case where we run out of memory by doing it the "wrong" order13:06.57 
Robin_Watts The problem is that if we insert early, then toss, we may find that we can't toss enough to bring the store size down. in which case we need to back out the insert.13:07.06 
  and currently backing out the insert means another hash table search.13:07.30 
tor8 Robin_Watts: or live with a slightly oversized store until the next insertion?13:07.32 
  which would trigger another round of eviction which should remove the easy pickings13:08.04 
Robin_Watts It could be more than 'slightly' oversized.13:08.25 
tor8 of course, it could be the case that it's too big to ever fit the store so shouldn't punish the user by evicting everything else and then not fit...13:08.55 
Robin_Watts and the next wave of eviction may not be able to free any more because we've already binned all the free ones.13:09.06 
tor8 what happens if the store is full of used resources and we want to add more?13:09.27 
Robin_Watts tor8: Right, we are currently at pains never to throw *anything* out of the store until we are sure that we have enough that can be thrown.13:09.36 
  tor8: The broad strokes are: when asked to insert an object X, check to see how much the current store + X will be overbudget. If we can safely evict at least that overbudgetness, then evict that much, then add X.13:10.43 
tor8 and if we can't don't cache the new item13:11.08 
Robin_Watts if something equivalent to X happened to be in the store already, then remove X.13:11.13 
  right.13:11.15 
  So the new broad strokes would be:13:11.28 
  When asked to insert X, insert X into the hash table, keeping a pointer P to where it is inserted. (If something equivalent to X happens to be in the store already, then we don't insert X, and just exit). Check to see how much the current store + X will be overbudget. If we can safely evict at least that much, then great. If not, then use P to back out X leaving the store unchanged.13:13.21 
tor8 that sounds reasonable13:15.31 
Robin_Watts so the change there is that I need to alter the hashtable code to give access to P.13:15.48 
  This is all made more interesting by the constant taking/dropping of locks etc of course.13:16.17 
tor8 Robin_Watts: or do another hash to remove in the rare case of being over budget13:16.20 
Robin_Watts tor8: Depends how rare that is.13:16.35 
  If we're running with a low store, it may be quite often.13:16.45 
tor8 the hash is designed for speed, but I think we'll hurt more by not being able to cache stuff than doing two hash lookups in the case of a low store13:17.08 
paulgardiner Robin_Watts, tor8: I've fixed the problems you pointed out and repushed.13:18.46 
Robin_Watts tor8: If we extend the hash table so that when we insert something we keep a 'secret' pointer to it so we can whip it out again quickly if we need to, we should get the best of all worlds I think.13:29.07 
  paulgardiner: Looks good to me.13:32.17 
paulgardiner Great. Ta13:32.29 
Robin_Watts I always get bad results when trying to pull --rebase with pauls stuff.13:32.57 
paulgardiner Is that initially or after I've updated because of feedback?13:33.54 
Robin_Watts paulgardiner: If I fetch, and then rebase it's fine.13:34.31 
  If I pull --rebase it gives me all sorts of messages I don't get otherwise.13:34.42 
  but never mind, it's working now.13:34.49 
paulgardiner Interesting though. What sort of messages?13:35.07 
Robin_Watts It seems to rewind further than you'd expect.13:35.38 
  but don't worry about it.13:35.50 
  I'm about to lose power for a mo.13:35.58 
paulgardiner Does pull --rebase rebase my stuff over your or the other way around?13:36.12 
Robin_Watts It rebases my stuff over yours.13:36.25 
paulgardiner If I haven't rebased to the end of origin/master, and you have, wouldn't that rebase some of the origin/master commits?13:38.57 
Robin_Watts paulgardiner: It would, but I checked you had before I pulled.14:10.16 
paulgardiner You must pull --rebase from origin at times, but presumably from no other user-repo other than mine.14:12.23 
Robin_Watts I pull --rebase from both origin and golden.14:12.44 
  and tor.14:13.39 
  and sebras.14:13.42 
  and only yours gives me grief. But it could still be something local to me, so don't worry.14:14.04 
paulgardiner I thought you did it with mine just because you were going to commit them14:14.06 
Robin_Watts I do the same with tor/sebras code when I review them.14:14.30 
  and I shift code between various boxes via my casper repo14:14.55 
  Anyway, lunchtime.14:15.49 
paulgardiner Is origin your repo and golden the main one?14:15.51 
Robin_Watts yes.14:41.09 
  oh, ass.14:52.40 
  We can't quickly remove from the hash table.14:52.55 
  oh, yes, we can, kinda.14:53.40 
tor8 Robin_Watts: fz_hash_remove?15:12.16 
Robin_Watts tor8: That requires linear probing. I have an idea to minimise that.15:20.49 
henrys Robin_Watts:I know you aren't familiar with XL but I pushed some stuff to my local tree I'd like you to look at, since it is mostly about using the path code. no hurry15:22.18 
Robin_Watts henrys: sure.15:22.30 
henrys now I have to figure out how to get rid of all the new lines indent added to local declarations in PCL - do they test that code? That's quite an obvious bug.15:25.49 
  of course I committed it before noticing ;-(15:30.25 
Robin_Watts henrys: git revert the commit.15:34.46 
  Then indent it all properly and do a new commit.15:35.02 
  The squash the two together?15:35.11 
henrys Robin_Watts:I was just going to fix it.15:35.26 
Robin_Watts henrys: radical.15:35.41 
henrys something about going back and erasing my mistakes upsets my kharma15:36.16 
Robin_Watts I assumed from your question that there was no trivial way to fix it.15:36.21 
  henrys: I wasn't suggesting that you change the published history.15:36.40 
  If you've pushed to the golden repo, it's too late now (unless we want to be naughty).15:37.08 
henrys yes I have pushed.15:37.20 
Robin_Watts What I was suggesting was a way to make a single commit that would correct the mistake.15:37.37 
  presumably if you run a (fixed) indent command, it will fix the indentation, but not remove all the added lines?15:38.04 
henrys The problem is the new lines are a side effect bug of a feature we do want, so if I fix that it will break something else, I was more thinking of global search and replace kung fu without indent.15:41.20 
Robin_Watts henrys: ah, right.15:42.04 
kens Morning mvrhel_laptop16:05.32 
mvrhel_laptop hi kens16:05.51 
  so it was the 3CLR type16:06.45 
  kens16:06.46 
kens seems to be mvrhel_laptop16:06.55 
  I did eventually find it in the PDF Reference16:07.05 
mvrhel_laptop kens: so is this a profile that we are generating?16:07.09 
kens mvrhel_laptop : yes it is the icc_equivalent to a CIEBasedDEF space16:07.24 
mvrhel_laptop kens: ok. I can fix that to avoid the 3CLR type16:07.36 
kens Really ? THat would be great16:07.43 
  THe spec says it supports RGB, Gray, CMYK or Lab16:08.19 
Robin_Watts tor8, paulgardiner: 2 commits on robin/master16:08.26 
mvrhel_laptop We will simply call it RGB even though it may not be RGB colors . But since we are using it to map from source values to CIE to destination it should not matter16:09.07 
kens Oh, THat should be interesting :-)16:09.26 
mvrhel_laptop yes :)16:09.43 
  but it should work16:09.46 
kens I oucld probably hack that as a test if I'd thought of it16:09.54 
mvrhel_laptop kens: it is probably only one line if you want to do it16:10.31 
  let me find it16:10.34 
kens Please16:10.41 
mvrhel_laptop line 2045 in gsicc_create.c16:12.47 
kens one second16:12.57 
mvrhel_laptop oh and line 1983 is using 4CLR16:13.19 
kens ok16:13.20 
mvrhel_laptop for CIEDEFG16:13.23 
  replace 3CLR with icSigRgbData16:13.37 
kens Hmmm line 2045 is:16:13.56 
  gsicc_create_init_luta2bpart(&icc_luta2bparts);16:13.56 
  SHold it be:16:14.08 
  header->colorSpace = icSig3colorData;16:14.08 
mvrhel_laptop weird that your lines numbers are different16:14.56 
kens OK quick test time :-)16:14.56 
mvrhel_laptop header->colorSpace = icSig3colorData;16:14.58 
kens Yes that's the line I changed16:15.05 
mvrhel_laptop ok.16:15.10 
  other line to change is header->colorSpace = icSig4colorData; to header->colorSpace = icSigCmykData;16:15.43 
kens Yes did that too by inference16:16.00 
mvrhel_laptop ok good deal16:16.07 
kens oops, pdfwrite, not ps2write....16:16.21 
  OK says scnrRGB lets see what Acrobat says16:16.47 
  Yay! Actobat is happy now. THanks Michael16:17.04 
mvrhel_laptop kens: great16:17.11 
kens I'll do a fuller test after I find out why my type 2 shadings don't work16:17.47 
  Is it OK if I commit that change by the way ?16:18.04 
  mvrhel_laptop : ^^16:19.17 
mvrhel_laptop kens: yes, I am fine with the change16:19.29 
kens Thanks, I'll do it now then, as that will fix the 'new' CMS stuff in pdfwrite, apart from my werid shading bug16:20.00 
mvrhel_laptop those profiles are (up until now) all used internally so until now they had not been saved16:20.01 
kens Aha, OK that's fine then, should not be a problem (I hope!)16:20.15 
mvrhel_laptop I can't see any issue16:20.24 
paulgardiner Robin_Watts: two commits look good to me16:30.46 
Robin_Watts Thanks.16:31.30 
kens Robin_Watts : you keeping an eye out for Cassie on your runs ?17:29.36 
  http://www.theregister.co.uk/2013/03/25/space_hedgehog/17:29.36 
Robin_Watts kens: No running recently, cos of the weather. but I will look for it :)17:30.14 
kens :-) Maybe you dog can find it17:30.29 
ray_laptop chrisl: I actually read the nmake documentation, and it says that recursive nmake invocations _do_ inherit the macros from the outer level. I tested, and guess what ? It works as documented.17:30.31 
kens Off for now, goodnight all17:30.47 
ray_laptop chrisl: henrys: so I thought that the mess in msvc_top.mak was because nmake didn't pass things in, but maybe we don't need that ?17:31.20 
chrisl ray_laptop: just what I was wondering.... we only need the ones that need redefined17:31.52 
ray_laptop maybe is was some really old MSVC_VERSION where it was needed.17:31.53 
  sure would be nice to clean that up !17:32.36 
henrys does not remember and hopes it will go away when chrisl revisits build restructuring17:33.56 
chrisl There's also the stuff in msvc.mak, too, which could probably be tidied up17:34.46 
ray_laptop chrisl: if you mean the DEVSTUDIO="$(DEVSTUDIO)" FT_BRIDGE=$(FT_BRIDGE) then I agree.17:58.30 
chrisl ray_laptop: yeh, all that stuff17:58.44 
ray_laptop maybe even the WINDEFS since the BUILD_SYSTEM macro should be inherited17:59.10 
  chrisl: I tested with both VS 2005 and VS 2008 and it is OK. Maybe it was an older version than that, but I don't think we need to support anything older than 200518:16.05 
chrisl ray_laptop: cool! I've actually got VS2003 in a VM here, so I can test with that - definitely not going to worry about VC6 or earlier!18:16.55 
ray_laptop chrisl: I'll go ahead an commit my change for gs.mak that removes the *_EXTRA= definitions that are problematic for nmake then, and no other changes for now18:18.34 
chrisl ray_laptop: okay, that sounds fine18:18.53 
ray_laptop chrisl: I have 2003 as well, but not loaded (and I prefer not to)18:18.54 
  bbiaw -- coffee...18:29.45 
Robin_Watts tor8, paulgardiner 1 tiny fix on robin master20:20.33 
paulgardiner Robin_Watts: LGTM20:22.32 
Robin_Watts Thanks.20:31.11 
mvrhel_laptop bbiaw20:50.45 
 Forward 1 day (to 2013/03/26)>>> 
ghostscript.com
Search: