IRC Logs

Log of #ghostscript at irc.freenode.net.

Search:
 <<<Back 1 day (to 2012/05/07)2012/05/08 
kens I like yesterday's xkcd :-)07:23.47 
chrisl Heh! I Hadn't seen it until now...... :-)07:33.14 
henrys oh damn I'm up, hoped if I stayed up until bedtime I'd sleep through.10:18.28 
kens Doesn't always work10:19.04 
Robin_Watts henrys: Ah, you've flown home? Explains why I can see the sun.10:19.45 
kens henrys (or Robin_Watts) either of you know anything about GS and memory management ?10:20.10 
Robin_Watts bits.10:20.26 
kens Any idea if tehre's some way to tell when a chunk of memory is freed ?10:20.45 
Robin_Watts In what way "tell' ?10:21.06 
kens OK bear with me a second and I'll explain10:21.15 
  In the 'pdfwrite server' mode we do one file per page with pdfwrite.10:21.35 
  On page 2 I get a seg fault. The reason is that its following a linked list of 'functions' (PS function resource objects) to see if a new one is a duplicate.10:22.07 
  The problem is that the memory teh poitner is poiting at is 'corrupted'.10:22.23 
Robin_Watts where "corrupted" is "has already been freed" ?10:22.48 
kens No, hang on a mo10:23.01 
  After a fair bit of debugging I find that the memory is originally allocated, and teh pointer stored when defining a function.10:23.06 
Robin_Watts ok, sorry, carry on.10:23.08 
kens Later we allocate storage for a 'base font' resoruce object10:23.22 
  We get back teh exact same pointer from alloc. This of course results in 'corruption' as we store different data in the same place.10:23.48 
  I can't see any place where teh original data is freed.10:23.57 
  Is there a way I can tell ?10:24.05 
  Alternatively, how can I figure out what GS's memory manager is doing giving me back already allocated memory ?10:24.28 
Robin_Watts Right. So the first block has either been freed, or the free list structure has been corrupted.10:24.36 
henrys you could use -ZA and search for it's debug client name and relevant addresses.10:24.40 
kens FWIW it appears to be giving me memory form the 'free list' each time10:24.40 
  henrys, tried that, gives me 104Mb of output10:24.51 
Robin_Watts The first of those cases actually splits into 2 cases.10:24.58 
kens Robin_Watts : Yes, exactly10:25.00 
  My feeling is the free list is broken10:25.12 
Robin_Watts The block may have been freed by an explicit free, or the block has been freed by the chunk it's in being freed.10:25.31 
kens Because I had a similar problem before I added the code to free functions.10:25.34 
  OK10:25.40 
  I don't believe its an explicit free10:25.52 
Robin_Watts Is the chunk memory manager in use ?>10:26.04 
kens though I could be wrong. I've put a conditional breakpoint int he free routine and it doesn't trigger10:26.10 
  Robin_Watts : How do I tell ?10:26.19 
Robin_Watts gs or pcl ?10:26.28 
kens GS10:26.32 
  THe memory is not in the garbage collector10:26.40 
Robin_Watts then the chunk manager is NOT used, AIUI.10:26.42 
  kens: Do a memento build.10:26.54 
kens Robin_Watts : I tried that too.10:27.02 
  Takes 40 minutyes to finish page 110:27.12 
  Gies me no errors10:27.19 
henrys 104 MB is not a problem just open it in a file search for the client debug string and then search for addresses.10:27.30 
kens henrys if I knew what I was doing I'd try that, but i have nio idea what the data is telling me.10:27.51 
Robin_Watts Hmm.10:28.16 
henrys I assume you've done -Z?10:28.21 
kens I also can't be sure the same problem exhibits. I can't get stderr redirection to work in the debugger, so I haev to use the command line to get the output. I can't tell if the seg fault I see on the command line is the same as the one in the debugger10:28.35 
Robin_Watts Typically the free list structure is a linked list (or tree etc) that goes through the bytes just ahead of each block.10:28.44 
kens henrys I still don't know what that would tell me, pluss see above.10:28.51 
Robin_Watts So if you over/underwrite a block you can overwrite the structure.10:29.00 
kens Robin_Watts : That's the sort of thin g I'm looking for10:29.05 
Robin_Watts Memento puts a series of guard bytes in there (16 before each block, 16 afterwards) so it can spot such trampling.10:29.28 
kens Umm, well it didn't give me any warnings or errors10:29.44 
Robin_Watts and if you're not seeing that, then that's a good indication that that is not happening.10:29.51 
chrisl Have you tried running with -Z@10:30.00 
Robin_Watts You say you put a conditional breakpoint on free?10:30.01 
kens I can try again to be sure, btu ti will take nearly an hour10:30.05 
  Robin_Watts : yes, the specific free associated with the memory allocater10:30.25 
Robin_Watts Which free function ?10:30.33 
kens one sec10:30.42 
  i_free_object10:31.12 
  in gsalloc.c10:31.25 
  The memory is allocated by i_alloc_strut10:31.55 
Robin_Watts So, that IS the chunk allocator, right?10:31.55 
kens has no idea10:32.04 
  gsalloc.c comment says 'Standard memory allocater'10:32.25 
Robin_Watts Now, memento has a couple of hacks in the chunk allocator.10:32.32 
  grep for MEMENTO10:32.39 
kens In gsalloc.c ?10:32.47 
Robin_Watts They are in gsmchunk10:33.06 
kens OK I see that in several files10:33.39 
Robin_Watts actually, it's the very first MEMENTO in gsmchunk.c10:33.42 
  That may be masking the problem you're seeing?10:33.59 
kens Yes, possibly10:34.20 
Robin_Watts So you could try disabling that.10:34.28 
henrys -Z? will validate memory frequently and catch the problem when it happens. Is this something we can reproduce if we have your file?10:34.31 
kens No hang on it still fails with MEMENTO IIRC10:34.33 
  henrys I'm using pdf_reference17.pdf10:34.53 
  I don't know if it will reproduce on another machine10:35.05 
Robin_Watts oh, it fails with memento, but memento doesn't report corruption?10:35.12 
kens I mailed ray last night but I guess he hasn't had a chance to look at it10:35.16 
  Robin_Watts : That's what memory says, but that was last week (my memory is terrible). I'm willing to give it another shot, btu it will take an hour to get results.10:35.44 
Robin_Watts kens: If it was me, I'd want to find out first off whether the chunk memory manager is in play or not.10:36.08 
  by putting breakpoints in gsmchunk.c10:36.16 
kens I can do that, one second.10:36.23 
Robin_Watts If it *is* in play, then stuff is probably being freed by the ripcord being pulled.10:36.41 
henrys the chunk manager would only be used for MT banding.10:36.46 
Robin_Watts Another possibility is that the block you were looking at was moved due to being realloced.10:37.04 
kens breakpoint in chunk_obj_alloc is not being called10:37.14 
Robin_Watts (realloc to size 0 == free, realloc to a larger size would move it in the heap, leaving the old memory free)10:37.33 
kens But -ZA says 'opening chunk' and so on10:37.35 
Robin_Watts I fear overloading of the word "chunk" :(10:37.54 
kens Robin_Watts : it wil not be reallocated by my code10:37.57 
Robin_Watts kens: I added a new thing to memento the other day.10:38.31 
kens Tell you what, I'll rebase, rebuid and try with a memento buidl10:39.01 
  I can leave it running over lunch10:39.11 
henrys I wonder why pdf_reference17.pdf is so slow I was running the plrm in server mode and it seemed to be moving right along, until it crashed ;-)10:39.19 
kens henrys I think its because page 1 references a *load* of stuff.10:39.46 
henrys kens:so all the code you are testing is in the tree?10:39.48 
Robin_Watts Memento_breakOnRealloc((void *)address) will stop when a given address is freed (or realloced)10:39.51 
kens Also opening such a big PDF is slow10:39.55 
  henrys, yes its all in trunk now10:40.02 
  It works fine for non-server mode10:40.09 
  For some reason I ghot confused and thought you had been testing with the PDF reference rather than the PLRM10:40.33 
henrys well I'll give it a go and see if I trip over something obvious.10:40.37 
kens Does it crash on page 2 for you ?10:40.47 
henrys probably won't ... 10:40.55 
kens It will be interersting if it does.10:41.03 
  But no, I expect not.10:41.08 
henrys probably won't trip over anything obvious I mean.10:41.14 
kens One reason I asked ray is because he's on Windows too10:41.18 
Robin_Watts kens: If memento is not catching any corruption, then you can compile with MEMENTO_LEAKONLY (edit memento.c line 19), and it will run a lot faster.10:41.28 
kens Robin_Watts : let me try it the normal way first.10:41.45 
henrys did you try NOGC?10:41.48 
kens I'll get back to you after lunch10:41.53 
Robin_Watts goes for a run, will confer again when I'm back, and when the memento thing has finished.10:42.00 
kens henrys no I didn't. I dont' really want the problem to go away....10:42.05 
  A crash on page 2 is not to bad, the PLRM crashes on page one hundreed and something in a suspiciously similar fashion (for me)10:42.40 
henrys I hope I have the same version of the manual anyway have a nice lunch and run.10:42.40 
chrisl kens: is your crash happening in pdf_close()?10:55.59 
kens no10:56.06 
  while processing page 210:56.12 
chrisl I tried with the PDF reference "core", and using -Z@ and I get a crash on page 1 in pdf_close10:56.50 
kens Hmm, well that suggests a similar problem10:57.23 
  where in pdf_close ?10:57.32 
kammerer Hi all, is there any difference in draw_scale and draw_simple_scale in perfomance and memory usage? Could i use any of them or maybe one of them work faster or used less memory?10:57.49 
chrisl In the master code, at line 169010:57.51 
kens chrisl that's a surprise, and probably a different problem, though it might be related10:58.23 
chrisl prev = pres->prev; - because pres = (pdf_resource_t *) 0xf1f1f1f1f1f1f1f110:58.29 
kens I think that menas its already freed teh object.10:58.42 
chrisl It went on to page 2 without -Z@10:59.19 
kens obviously it changes the memory layout...10:59.32 
chrisl Filling freed memory shouldn't change the memory layout11:00.52 
kens sorry, misread the paramaeter11:01.09 
chrisl "-Z?" gives me this: http://pastebin.com/MbrHaWrL11:04.07 
henrys and the -Z? tells you where the problem starts: GPL Ghostscript GIT PRERELEASE 9.06: ./base/gsalloc.c(811): Free Widths array: freeing 0x10378de28, not owned by memory 0x101841c58!11:04.08 
kens Intersting, I can look at that after memento run finishes11:05.16 
  not the same problem though11:05.22 
  different memory allocations11:05.27 
henrys well that happens on page 1 - you probably need to get that one first11:05.58 
chrisl But if the memory is being freed into the wrong allocator, the results are likely to be undefined......11:06.21 
henrys this file is indeed very slow.11:07.46 
  chrisl:me too, I just gave the first line.11:08.21 
chrisl henrys: I felt it was significant that all the errors seem to come from the same allocator.11:09.56 
henrys yes and all are font related - I wonder if the new server code is freeing the fonts before they are accessed in some circumstance.11:10.51 
  it reproduces with NOGC which is a relief.11:12.23 
kens It won't be before they are used. It might be the wrong allocater or somethign I can look at that, I desk checked it yesterday, but its tricky11:12.58 
  Have to let memnto churn for a bit11:13.46 
henrys chrisl:did you run with -ZA to see which allocator is being used for the addresses? I guess we shouldn't all work on this.11:14.48 
kens Leave it with me for now folks11:17.23 
  I may ask more dum,b questions11:17.31 
henrys the -ZA stuff has a cryptic code it prints out to tell you the allocator, origin (free list etc). We really should make that more readable it takes a long time to decrypt if you don't use it frequently.11:20.37 
chrisl henrys: I didn't do very much, I just felt that -Z@ was worth trying, and didn't get an answer when I suggested it above.......11:21.34 
kens apologises, but can only try one thing at a time11:24.04 
chrisl Not a problem - I didn't want to make a fuss, and then find it made no difference, so it was easy enough just to run it11:24.55 
  It looks like there's a few "pdf_resource_t: object 0x2e3ed28 already free!" errors in there.......11:26.06 
henrys I'm moving on to my own hell then, good luck. It is also reasonable assign it to ray if you are head bashing.11:26.09 
chrisl henrys: what's your current hell? The funky HP/GL path stuff?11:27.33 
henrys after the tech agenda followup yes.11:28.12 
  had a wonderful trip in the UK and the weather wasn't really a hindrance I think we did everything we set out to do.11:28.35 
chrisl That's good, I'm glad you both enjoyed it. As to the weather, well, you've experienced Britain "properly" now!11:29.32 
henrys I would be uncomfortable driving there though, the first 30 minutes would go okay then complacency would set in and I'd be in the right lane, and I do mean the "right" lane.11:30.10 
  chrisl:just the opposite of here - our claim to fame in Boulder is 300+ days of sunshine.11:31.13 
  per year11:31.23 
chrisl Well, we could claim 300+ days of sunshine..... per decade (maybe)11:31.58 
  As to driving on the left, if it weren't for that, how could the car manufacturers over charge us for *everything* if we didn't have right hand drive cars :-(11:33.15 
henrys It's kind of a sad commentary you have 2000 years of incredible western history and everyone is going on about Harry Potter's family's house and such.11:34.12 
kens History is dull :-)11:34.34 
henrys ;-)11:35.28 
chrisl henrys: did you get to see the little bit of US territory in the middle of southern England?11:38.23 
henrys no I don't know anything about it. Do they drive on the right there ;-) ?11:39.14 
chrisl maybe in model cars.....11:39.35 
  http://en.wikipedia.org/wiki/Runnymede#John_F._Kennedy_Memorial11:39.38 
henrys I did see a statue of George Washington in London which I found odd.11:45.30 
kens We have statues to everyone :-)11:52.00 
  chrisl the Memento buil,d crashed in pdf_close, look slike it is picking up the same problem.11:52.20 
  Now to find out what it is11:52.25 
henrys Is it just me that dislikes Scott's email logos? Seems like it would just annoy folks.12:05.38 
kens hadn't really noticed12:08.18 
  logos are common in email, but Eudora often screws them up :-)12:08.32 
chrisl I have my mail client set to "plain text" so I don't really see them - they are listed as attachments.12:08.41 
Robin_Watts I am relatively unoffended by scotts logos - the most annoying thing is the way they are completely unbalanced.12:52.36 
  kens: just dug myself out of my mupdf hole.14:40.11 
kens NP14:40.17 
Robin_Watts Do you want me to have a look at this memory thing?14:40.22 
kens I am digging my own hole14:40.22 
  No its OK I have plenty to look at14:40.30 
Robin_Watts ok.14:40.46 
kens thanks though14:40.57 
Robin_Watts np.14:41.03 
  tor8, henrys: We have a meeting with paulgardiner in 20 mins, right?14:41.21 
henrys Robin_Watts:I'd believe that.14:42.20 
Robin_Watts he sent around a progress report yesterday.14:42.52 
henrys has tor8 been around?14:45.55 
Robin_Watts I haven't spoken to him today (but then I've not tried)14:46.28 
tor8 I am silently here.14:46.42 
paulgardiner me too14:49.32 
henrys Is there a url to the API code?14:49.46 
Robin_Watts http://git.ghostscript.com/?p=user/paulg/mupdf.git;a=shortlog;h=refs/heads/forms ?14:49.58 
paulgardiner It's in mupdf-internal.h14:50.44 
  Right at the end14:50.59 
henrys thanks14:51.43 
Robin_Watts and in fitz.h - right at the end.14:51.51 
paulgardiner Oh yes. There too. I was thinking of just the Javascript stuff.14:52.59 
Robin_Watts This build thing, about it being a library being a problem...14:53.43 
  It's a problem because of the inline functions in V8, right ?14:54.00 
paulgardiner I think that's what causes it.14:54.20 
Robin_Watts and the only place those are called is from pdf_js_v8, right?14:54.35 
paulgardiner method bodies in the class declaration14:54.41 
  Robin_Watts: ah yes true14:54.57 
Robin_Watts So, if we build all of mupdf except pdf_js_v8 as a lib, we're fine.14:55.16 
  We just need to build pdf_js_v8 + v8 at the same time?14:55.30 
  or am I being stupidly hopeful?14:55.46 
paulgardiner I wouldn't say stupidly so :-)14:56.17 
  I think you might be right.14:56.31 
Robin_Watts It does seem like a design flaw if people can't build 'a V8 library'.14:56.46 
henrys I need to download a snapshot getting around in web git sucks.14:56.48 
Robin_Watts henrys: git fetch paul then git diff 96a87dc paul/forms14:57.23 
  (assuming you have remote 'paul' set up)14:57.52 
  (git remote add paul henrys@ghostscript.com:/home/paulg/repos/mupdf.git)14:58.28 
henrys I'll set that up later.14:58.46 
paulgardiner Robin_Watts: Yes: if we avoid including pdf_jsimp_v8 in the library, instead compiling it and linking it as part of the final link...14:59.46 
Robin_Watts paulgardiner: I think the key thing is toe make pdf_jsimp+v8 part of the V8 build rather than part of the mupdf lib build.15:00.24 
  I'd hope that "V8 + pdf_jsimp_v8" can be safely built into 1 library.15:00.52 
  Whether that works may depend on whether we have mutual calls between libmupdf and libV8+15:01.48 
  (i.e. I'm not sure that unix linkers will cope if libmupdf calls into libV8 and libV8 also calls into libmupdf.15:02.24 
  libs are required to be a DAG, if memory serves.15:02.34 
paulgardiner Not sure that V8 + pdf_jsimp_v8 can be combined into one library15:02.52 
  Well not basing it on the v8 lib files.15:03.38 
  I couldn't find a way under windows to part link libraries15:04.05 
Robin_Watts No, I was thinking that you could link all the V8 .o's together with pdf_jsimp_v8.o into libV8+15:04.38 
  Does the V8 make system even have an option to make a libV8 ?15:05.10 
paulgardiner Yeah, that's a possibility, although playing with the V8 build is a bit scary.15:05.13 
Robin_Watts It's "just" adding one more .c file.15:05.31 
  Where the quotes in that sentence are semantically crucial.15:05.57 
paulgardiner Yeah, I guess. I was thinking about the GYP stage, but then I guess we can take the output of GYP and add the extra file15:06.04 
  It might make sense to link V8 + pdf_jsimp_v8 + pdf_js15:07.05 
Robin_Watts pdf_js calls into mupdf though (presumably using mupdf-internal.h type interfaces)15:07.50 
  I was hoping that pdf_jsimp_v8 only made mupdf.h and fitz.h calls to pdf_js ?15:08.30 
paulgardiner ah yes.15:08.30 
tor8 v8 takes forever to compile...15:08.41 
ray_laptop Hi, all. I have a doctor's appt today at 9:15. I'll try and check in, but don't know how long I'll be online15:08.59 
paulgardiner tor8: I could bung you my lib files.15:09.11 
tor8 paulgardiner: I figured I'd try it out on a linux box, see if I get the same link issues there15:09.34 
ray_laptop I'm taking my youngest to school now. back in about 20 minutes or so15:09.37 
Robin_Watts paulgardiner: pdf_new_rect... you make a rect that contains [ originx, originy, width, height ]15:09.46 
paulgardiner tor8: Oh right. Good15:09.48 
Robin_Watts IIRC pdf rectangles tend to be [ left bottom right top ]15:10.10 
paulgardiner Robin_Watts: should that be x0, y0, x1, y1?15:10.24 
  Ok. I'll fix that.15:10.34 
henrys has the error handling been sorted out? For example suppose we have division by zero with our calculator. and a real js arithmetic exception occurs what is the flow of execution after that?15:10.45 
Robin_Watts I'd check it before believing me blindly :)15:10.48 
paulgardiner henrys: No. But I did have that in mind. Forgot to mention it in the report.15:11.19 
Robin_Watts paulgardiner: What happens in such cases? (what *should* happen I mean)15:11.47 
paulgardiner henrys: I think I'll have to make pdf_jsimp_v8 return error values and then turn them into fz exceptions in pdf_js15:12.38 
  Robin_Watts: Not sure what should happen, but at the moment it will crash.15:12.52 
Robin_Watts So, a division by zero in V8 actually generates a division by zero C++ exception?15:13.22 
paulgardiner Robin_Watts: Again don't know, but certainly V8 generates C++ exceptions.15:13.48 
Robin_Watts In JavaScript, division by zero does not cause an error. Instead, it evaluates to Infinity , which is a reserved word (note the capital “I”)15:14.19 
paulgardiner Robin_Watts: Actually I do know: V8 methods fail silently or return NULL and you can ask if there was an exception.15:14.28 
Robin_Watts hence, I'd have expected V8 to swallow the exception and do the infinity thing all by itself.15:14.49 
henrys BTW, It seems the progress is excellent as usual, thank you.15:14.56 
paulgardiner Robin_Watts: I mean in the case of a Javascript exception15:14.58 
Robin_Watts (Frantic googling here)15:15.29 
  so javascript has exceptions built in to the language.15:15.40 
  again, I don't see why we should need to do anything.15:15.49 
  Unless we expect a javascript exception to get 'promoted' up to an fz_exception15:16.16 
  and I would have thought that would be odd, personally.15:16.27 
paulgardiner henrys: Thanks. I'm enjoying it a lot.15:16.43 
Robin_Watts (maybe there will be specific cases where we may want to do that - out of memory maybe? - but that would be exception (ha!) rather than the rule, I'd have thought)15:17.07 
henrys Well I'm sure in chrome or acrobat division by zero doesn't just shut everything down.15:17.18 
Robin_Watts henrys: No, as I say, anything / 0 is Infinity in javascript.15:17.41 
paulgardiner Robin_Watts: Yes you are right. I was confusing myself with engine exceptions. V8 does produce C++ exceptions in some cases.15:17.47 
Robin_Watts there are no exceptions involved.15:17.49 
paulgardiner Unless my memory is letting my down, I had it produce a C++ exception when I passed it bad JavaScript to parse.15:18.40 
Robin_Watts OK, so we have js exceptions, fz exceptions and v8 exceptions? (just trying to find a consistent terminology that my mind can cope with)15:18.41 
henrys well whatever produces and exception has to be handled in some way I would think.15:18.42 
  s/and/an15:18.48 
Robin_Watts js exceptions are not our problem.15:19.01 
paulgardiner Yes15:19.07 
  And I think the C++ exceptions should be turned into fz ones.15:19.26 
Robin_Watts v8 exceptions (presumably, like the engine running out of memory) are our problem. We'll need to catch them in the pdf_jsimp layer and deal with them (which may involve turning them into fz ones).15:19.52 
paulgardiner I may need to add a method to enquire after js exceptions.15:19.53 
henrys I assume there is something in the library API - probably different for each library sigh15:20.03 
Robin_Watts henrys: Right, but it should be hidden in the pdf_jsimp_ code, so that's fine.15:20.26 
paulgardiner henrys: Hopefully we can make them all look the same through the interface though15:21.09 
henrys right15:21.16 
Robin_Watts paulgardiner: Ff_NoToggleToOff - what's the Ff ?15:23.34 
  Fitz Forms ? Form Field ?15:23.51 
paulgardiner Field Flag15:23.54 
henrys I know it is early but let's start thinking about how long until we have something to show for customers. At this stage the only thing I'd be comfortable saying is we are still researching which is not so great for a potential customer to hear.15:23.59 
  tor8:comments?15:25.30 
Robin_Watts bets Paul has evil 4 space tabs set in his editor.15:26.14 
tor8 henrys: no. I'm liking what's going on so far. except the non-indentation alignment of variables in pdf_jsimp_v8.cpp...15:27.03 
paulgardiner henrys: It's very difficult to know. The DOM is fairly huge, and I don't know how broadly it's used in files in the wild. There's also a lots of intricate detail yet to deal updating fields.15:27.16 
henrys I'm sure indentation can be fixed.15:27.45 
paulgardiner Robin_Watts: I though I changed to 4 space tab to match the project. Typically I expand to spaces.15:28.06 
tor8 I think we should get JS working to a degree enough that we know our approach is solid, then spend more time on getting the forms working on the various platforms and apps15:28.21 
Robin_Watts was pondering a macro to do: fz_catch(ctx) { fz_rethrow(ctx); } fz_catch_and_release(ctx) maybe ?15:28.39 
henrys paulgardiner:just something to keep in mind, don't stress over it. I would like to announce an expected date in our August newsletter and if it comes up earlier great.15:29.09 
tor8 paulgardiner: mupdf source is tab-width agnostic :)15:29.20 
paulgardiner tor8: :-)15:29.33 
Robin_Watts (there are some places in the diff where I'm seeing odd indentations due to spaces, but it's a really minor thing, and I do far worse, I'm sure)15:30.19 
paulgardiner I quite like getting alignment right. I'd be happy to change it. But what to?15:31.10 
  henrys: will certainly keep it in mind. It's difficult to know how well it's going. Certainly not badly, but I may have been picking off all the low-hanging fruit so far.15:32.10 
henrys so I assume Robin_Watts can give paulgardiner studio tab settings.15:35.55 
Robin_Watts I use 8 space tabs.15:36.41 
paulgardiner Yes. I also use vi at times, but that's eqally configurable.15:36.42 
henrys anything else for the meeting? if not let's call it so I can get ready for the next one.15:36.46 
Robin_Watts in fz_buffer_printf calls you use %02X15:37.38 
  Should that be %02x ?15:37.57 
  (not sure if case matters, X may be a gnuism?)15:38.14 
paulgardiner Possibly. I'll check.15:38.31 
  One thing on my mind is how we will judge when it is sufficiently complete.15:38.58 
  Should we build up a set of test files?15:39.13 
  Is there some way we can judge what form features are important and what not.15:39.33 
  ?15:39.36 
Robin_Watts paulgardiner: We should grep our test files to see which ones have form data in them.15:40.07 
  and look to see which features are used.15:40.21 
paulgardiner That sounds like a good plan.15:40.38 
Robin_Watts I suspect to demo stuff to customers we want to be able to open a PDF file with some forms, fill in some fields, and submit it.15:42.26 
  A US tax form is probably a good demo.15:42.34 
paulgardiner Confused on the tab thing: mupdf.h seems to assume 4-space tabs.15:42.35 
Robin_Watts paulgardiner: what line?15:42.46 
paulgardiner Loads of lines, but specifically 15515:43.07 
  ... unless it's just my local copy15:43.22 
Robin_Watts PDF_PERM_... ?15:43.24 
paulgardiner ... or it's supposed to display differently than I thought.15:43.40 
  Robin_Watts: yes15:43.51 
  When you say 8-space tab, do you also mean 8-space indentation?15:44.13 
Robin_Watts There are no occurences of SPACE SPACE in that file15:44.16 
  I do.15:44.21 
  tor8 uses proportional fonts in his editor (cos he's mad, bonkers, crazy, loco, ...)15:45.01 
  hence all indentation has to be done with tabs.15:45.16 
henrys which font do you use for programming tor8?15:46.12 
paulgardiner Ah right. So working with 4-space tab and 4-space indentation is fine for the lefthand edge, but will mess up other alighnments15:46.17 
Robin_Watts The problem with using proportional fonts, is that there can be no other alignments.15:46.47 
  splitting conditions over multiple lines is problematic.15:47.07 
tor8 henrys: http://ghostscript.com/~tor/repos/codec-font/15:47.55 
paulgardiner Ah so there's no need to chose between 8 and 4, but it's best not to try to alighn anything other than the lefthand edge.15:48.10 
Robin_Watts Indeed.15:48.16 
tor8 paulgardiner: exactly.15:48.18 
  (I also find fiddling with alignments a waste of time)15:48.40 
  (or a distraction, if I'm feeling favorable)15:49.10 
Robin_Watts My "evil 4 space tabs" comment was flippant, because I've lost hair dealing with code drops from other companies before that mix spaces and tabs and assume that tab = 4 spaces, and that's JustWrong(TM).15:49.13 
paulgardiner Yeah. All tabs or all spaces.15:49.46 
tor8 I think everyone agrees that it must be either all tabs, or all spaces.15:49.56 
Robin_Watts being really picky...15:50.29 
tor8 paulgardiner: it may amuse you to know that the ghostscript source code mandated mixed tabs and spaces for over a decade!15:50.30 
Robin_Watts ... do you remember picsels "no right hand creep" rule ?15:50.49 
paulgardiner Of course, with git, one could rewrite history and make it look like we were always sensible.15:51.23 
  Robin_Watts: No. What was that?15:51.32 
Robin_Watts paulgardiner: Which is exactly what tor8 did :)15:51.36 
paulgardiner Robin_Watts, tor8: :-)15:51.52 
Robin_Watts Look at fz_free_widget15:52.07 
  The contents are entirely inside if (widget) { ... }15:52.31 
  which pushes them further to the right.15:52.38 
  If instead we write: if (!widget) return;15:52.54 
  then the whole rest of the function is less indented.15:53.07 
  And it reads more naturally, IMHO. The flow of the code is clearer.15:53.31 
  (This is particularly evident in code that nests several such ifs)15:53.50 
  if (no_problem) { do next thing; } else { raise error }15:54.19 
  that's clearer as if (problem) raise error; do next thing;15:54.36 
paulgardiner Yeah. I have this slight dislike of returns and breaks within the body of constructs.15:55.13 
Robin_Watts In general, if you see code creeping rightwards, it's a sign that you could have simplified the code a bit.15:55.35 
  paulgardiner: Yeah, I understand that. It's a 'hint' rather than a rule, I guess.15:56.39 
Robin_Watts hopes paulgardiner has realised the fact that we (well me) is being picky over such trivial points indicates that there is nothing larger to be picky over :)15:57.24 
paulgardiner That thought had occured to me :-)15:57.58 
Robin_Watts And I like the indentation of variables in pdf_js - but tor8 will beat me for saying it.15:59.14 
paulgardiner But those thing are worth mentioning too... although I think it's swings and roundabouts. Using break and return from the middle of blocks is good if you are designing an algorithm in turns of it's step by step behaviour, but it's not good when you are thinking in terms of the predicates that hold at various places in the code.16:00.22 
henrys I agree all tabs or all spaces but all tabs is sort of like driving on the left - the world is not on your side.16:00.29 
kens tuesday meeting time16:00.33 
mvrhel good morning16:00.34 
  just in time16:00.45 
henrys I'm still on UK time so started around 4:00 am today.16:01.06 
mvrhel oh I see a space vs. tab argument16:01.56 
  we seem to have one of those every few years16:02.05 
henrys mvrhel did your open printing summit bugs get turned into bug reports?16:02.10 
mvrhel hmm we need to ask marcosw16:02.32 
  he was there and I thought added them16:02.42 
henrys they are listed in the agenda followup but I can't find the bugs.16:02.54 
  I'll just make a note of that and follow up with him.16:03.19 
mvrhel ok16:03.32 
kens FWIW the ps2write file size seems broadly similar to poppler16:03.44 
  Form Till's results on CUPS and Ubuntu16:03.52 
  If anything ps2write is smaller16:04.00 
mvrhel ok sounds like you have a bug...16:04.00 
  i mean a bug number16:04.06 
  maybe not a bug16:04.16 
kens No, those results are fomr irc chats16:04.18 
mvrhel oh ok16:04.23 
kens chrisl and I ahev worked with Till and a number of end users to resolve problems with printers16:04.40 
Robin_Watts mvrhel: I can't see any new bugs having been opened.16:05.08 
kens I think any bug would be an old one16:05.16 
  relatively16:05.20 
henrys so we sort of left the meeting and Robin_Watts did not have a current hot project and I think mvrhel talked about a planar separation device.16:05.20 
tor8 chrisl: http://ghostscript.com/~tor/gs-browse/ rough prototype of web-based source browser for ghostscript16:05.45 
Robin_Watts henrys: I was hoping to look at 1) MuPDF loading on the fly, 2) linearised files, 3) scan conversion.16:05.53 
  But I spoke to mvrhel about the tiffsep1 thing.16:06.14 
  Would it not make more sense to ditch tiffsep1 now, and add a -dBitsPerComponent to tiffsep ?16:06.33 
henrys Robin_Watts:oh okay then you are set with a full plate.16:06.34 
chrisl tor8: looks promising - what are you using for it?16:06.50 
mvrhel henrys: it is not much of a project other then that tiffsep1 is broken now since it seemed to be broken without compressed color encoding. moving the device to planar should be pretty easy. 16:06.53 
ray_laptop kens: did you get your answer about knowing when memory is freed ? (finalization proc)16:07.32 
Robin_Watts AIUI the significant differences between tiffsep and tiffsep1 are 1) 1bpc rather than 8, 2) composite vs no composite (-dComposite=0 ?)16:07.38 
mvrhel or as Robin_Watts had mentioned, merging the device with tiffsep but with a bit depth option may be an approach to consider16:07.44 
kens ray_laptop : no, but I have some other threads to pull at for now thanks16:07.49 
mvrhel since they both render to 8 bit contone anyway16:08.00 
kens I think it likely that the wrong allocater is being used for free.16:08.08 
henrys mvrhel:do we have open customer problems that would be fixed with planar or do you just want to get rid of the compressed color business?16:08.09 
chrisl Robin_Watts: Can tiff do composite 1 bpc?16:08.25 
Robin_Watts mvrhel: I was thinking that a -dBitsPerComponent would STOP the device rendering to contone.16:08.38 
mvrhel Robin_Watts: not sure how overprint could work then16:08.56 
  you really have to do the halftone afterwards16:09.05 
Robin_Watts So overprint only works with 8bpp planes?16:09.26 
mvrhel hmm maybe not 16:10.03 
  I might be thinking of something else16:10.12 
chrisl No overprint should be fine, since the channels are essentially independent16:10.26 
henrys ray_laptop:do you want me to look at the pcl memory bug. I'm surprised he hasn't pinged us yet he was fairly anxious about it when it was reported?16:10.37 
Robin_Watts I was thinking it would just be a simple(*) matter of setting each plane to be the appropriate number of bits. (* not necessarily simple).16:10.57 
mvrhel ray_laptop: was there a reason that the tiffsep1 device did the threshold array in the device after rendering to contone?16:11.19 
  it doesnt make any sense to me. since if we had done the halftone in the graphics lib, we would readily handle more than 8 colors in the color index16:12.23 
  unless we had transparency16:12.32 
  in which case we are contone16:12.37 
Robin_Watts contone, but effectively planar.16:12.45 
henrys perhaps performance?16:12.56 
ray_laptop mvrhel: at the time, the separation code needed 8 bits. (in order to form the CMYK composite)16:13.16 
Robin_Watts but tiffsep1 doesn't store a composite ?16:13.35 
mvrhel but tiffsep1 does not even output composite16:13.38 
  right16:13.40 
chrisl Robin_Watts: while I remember, there was a jbig2dec bug/patch you thought we could punt to Shelly for review, but we wanted to check with henrys........16:14.24 
Robin_Watts chrisl: oh, yes, I'll try and dig that up.16:14.38 
chrisl Cool - I promised Shelly I'd remind you16:15.52 
henrys you can just go ahead and mark any jbig2dec problem bountiable and point shelly at it.16:15.52 
kens I think these were patches from zeniko ?16:16.09 
henrys ah so fixes really for mupdf?16:16.36 
kens well maybe, but still jbig2dec16:16.52 
chrisl There we go: http://bugs.ghostscript.com/show_bug.cgi?id=69302516:17.32 
Robin_Watts bug 69302516:17.37 
  damn, too slow.16:17.40 
chrisl :-)16:18.06 
henrys about the halftone performance request from Drupa - they are halftoning everything as a postscript process - rendering to CMYK contone so hardware would likely pay off. Of course sse2 might do just as well.16:19.25 
  anyway I'm very hesitant to do any more project for them until we see 9.x integrated. We are likely to work on something that will never be used.16:20.04 
Robin_Watts henrys: Right, so an efficient C implementation would probably be much faster than the postscript implementation.16:20.18 
henrys s/postscript/post16:20.22 
chrisl Wouldn't it be faster to render to a halftone buffer?16:20.23 
  Rather than post process?16:20.38 
Robin_Watts henrys: Oh, a post process. Well, then my original objection stands I feel.16:20.57 
henrys It would have to be evaluated. With mvrhel new image ht code it might be much better.16:21.17 
mvrhel a good point for them to upgrade16:21.38 
Robin_Watts I can't believe that the post process halftone phase done efficiently in software is nearly as big a CPU hit as the rendering in the first place.16:21.55 
mvrhel depends upon the resolution and the content I suppose16:23.03 
Robin_Watts IF they are doing it inefficiently, then moving to a more efficient mechanism may be worthwhile, but it could be SSE2 rather than GPU based as you say.16:23.07 
  but yes, it should be evaluated. I'll shut up.16:23.24 
kens threshold halftoning is fast16:23.24 
henrys Robin_Watts:I recall ray_laptop and I did some measurement and halftoning was really quite significant ... but I think we were measuring raphs wts code which is slower than simple halftoning.16:23.46 
  do you remember ray_laptop?16:23.57 
ray_laptop mvrhel: the other reason for halftoning in tiffsep1 as a post process is that building of the LCM halftone tiles for 'real world' C, M, Y, K halftones at different angles is really slow16:24.01 
mvrhel oh good point16:24.12 
Robin_Watts why is that faster as a post process?16:24.46 
ray_laptop mvrhel: if each plane had it's own halftone (for planar mode) then we wouldn't need the LCM tile (which is chunky)16:25.05 
mvrhel ray_laptop: yes. so I think going to planar for tiffsep1 makes sense16:25.20 
tor8 chrisl: ctags and a python script16:25.34 
henrys okay let me get the jbig2dec bug bountiable and ping shelly.16:26.46 
chrisl tor8: ah, I tried something a while back, and it crashed and burned on the gs code - can't remember what it was though.16:26.51 
Robin_Watts tor8: Any objection to me pushing those 2 patches onto the main repo? (load contents on the fly during interpretation, and cope better with corrupted pages?)16:26.52 
ray_laptop Robin_Watts: the post process is slower for sparse pages (but those are fast anyway), but for artwork and photo images, building the halftone tile for the 4 components is CPU intensive, and the large LCM tile size means that the HT cache isn't very useful16:26.55 
Robin_Watts Right, the LCM being an issue, gottit.16:27.22 
ray_laptop I'm in the exam room now waiting for the dr., so when she shows up, I'll just close my laptop.16:27.46 
mvrhel turn and cough please16:27.59 
Robin_Watts ray_laptop: Can you not type whilst bent over? :)16:28.11 
mvrhel hehe16:28.18 
ray_laptop ha ha16:28.29 
Robin_Watts mvrhel: Do you know if the planar code gets that right?16:29.20 
tor8 Robin_Watts: go ahead16:29.35 
mvrhel Robin_Watts: if it get what right?16:29.37 
Robin_Watts Do halftone tiles get set up for each plane separately? So no LCM issues?16:29.48 
ray_laptop mvrhel: does the planar 1-bit code halftone with per-plane tiles ?16:29.48 
mvrhel the threshold arrays are independent and there is not LCM issue16:30.06 
ray_laptop mvrhel: great. That lets us change tiffsep to be more sane16:30.31 
Robin_Watts so moving the halftoning into the device shouldn't be a performance hit then, right?16:30.33 
henrys as someone suggested in the thread I think we really need to ask for some sample files and see wheat exactly they expect to speed up before making any recommendations.16:30.39 
ray_laptop tiffsep1, thatis16:30.43 
henrys alexcher:so are you good with the mupdf parser project - I didn't have bugs on the customer bug aging list so can you spend a little time with it.16:32.19 
kens have to go put dinner on, brb16:32.23 
mvrhel Robin_Watts: I need to see how solid colors are handled though in the code. I can't remember if we create a tile cache for solid fills. The fast threshold stuff is used only for images iirc.16:32.45 
henrys oh well 10:30 meeting over sorry to keep you kens16:32.56 
alexcher henrys: Yes, I'm looking into mupdf parser.16:33.14 
henrys okay great.16:33.32 
tor8 alexcher: http://git.ghostscript.com/?p=ghostpdl.git;a=commit;h=947ada5438090f2bfe7e29da82e93f62bf7bdff3 there is my old mupdf parser attempt. it's likely out of date, but it could be a start.16:34.25 
alexcher OK16:34.41 
mvrhel henrys: I am going to work on the delayed icc loading stuff this week16:34.42 
  I want to get that squeaky wheel off my list 16:35.05 
henrys okay sounds good.16:35.10 
alexcher There was a message about pdfopt problems. What's out opinion about replacing PostScript-based PDF utilities with mupdf-based ones?16:35.19 
kens I think that's a sound idea16:35.42 
Robin_Watts mupdf cannot produce linearised PDF currently.16:35.56 
  I don't know if that's a blocker...16:36.14 
ray_laptop it would be better if pdfwrite could write linearized files directly (instead of a post process by gs or mupdf)16:36.26 
kens ray_laptop : that's still a possibility16:36.43 
  I would have to read up on linearized PDF again to see what would have to be done16:36.58 
  But I think it should be perfectly possible16:37.09 
Robin_Watts hint tables. shudder.16:37.19 
ray_laptop writing a linearized PDF is similar to what we do for ps2write, but we'd have to code up the 'hints' that is now in PS16:37.26 
henrys we really should talk to tkamppeter about the utilities, he would have to change the package distribution - I assume other distros just copy ubuntu since we work with them the most.16:37.33 
alexcher pdfwrite re-writes PDF completely. The user may prefer to change the file as little as possible.16:37.52 
chrisl My concern would be for customers who use the utilities - they'd need licenses for both gs and mupdf, and incorporate the two tool sets16:38.59 
ray_laptop alexcher: right, it depends on whether or not they are trying to linearize an existing PDF, or are creating one using gs 16:39.10 
kens well I oculd look at linearized PDF as another enhancement, I think there may even be a bug for it16:39.26 
ray_laptop chrisl: oh boy! double the license fees ;-)16:39.36 
kens Where people are working with an existing PDF I have suggested MuPDF to them16:39.46 
chrisl ray_laptop: I'm not sure that would fly, too often - but I may be wrong.... if so, great!16:40.17 
ray_laptop ken: http://bugs.ghostscript.com/show_bug.cgi?id=68991616:40.27 
henrys kens:one of the things I wanted to discuss at the meeting is marking enhancement that we want to do P3 or something other than P4 which should be reserved for things that would be nice to do if we had more resources.16:40.45 
kens I don't mind henrys16:41.01 
ray_laptop kens: except mupdf doesn't emit linearized pdf's16:41.04 
kens yes that's the bug ray_laptop16:41.36 
  And I didn't mean linearizing with MuPDF, I meant 'other stuff'16:41.37 
henrys then we'll start with that enhancement, done!16:41.45 
kens henrys bear in mind that when Michael gets the ICC V2 profiles done I have another large project to look at.16:41.53 
Robin_Watts We probably should add the ability to linearise PDFs to mupdf.16:41.58 
ray_laptop that bug has been around since 2008.16:42.07 
henrys ray_laptop:I asked about the pcl memory bug earlier don't know if you saw it.16:42.34 
kens I know, there's always somethin g more important to do16:42.39 
mvrhel kens: I would like to help out some with the pdfwrite color project when we get to that point (and assuming you would like the help)16:42.45 
henrys ray_laptop:do you want me to work on it?16:42.46 
kens mvrhel I would welcom any help at that point ;-)16:43.09 
ray_laptop The other enhancement that I think should be elevated is http://bugs.ghostscript.com/show_bug.cgi?id=68991516:43.11 
  (write pdf 1.5)16:43.23 
kens That may be a more hideous task16:43.49 
ray_laptop kens: compressed objects and xref ?16:44.13 
kens because of the way pdfwrite works16:45.47 
  But I haven't looked at it16:45.47 
  yes16:45.47 
  currently all compressed objects are stream16:45.47 
  streams16:45.48 
  There is no provision for compressing anything which is not a stream16:45.49 
  Objects are written in *m,any* places scattered through the code16:45.50 
  But it may be possible to do something tricky with teh main stream and have it all 'just work'16:45.51 
ray_laptop henrys: if you can take the PCL memory leak, that would help. I still have the transparency performance improvements for cust 532 in process16:45.54 
henrys anyway let's have owners mark enhancement that we *really* think should be done to P3, then they are simple to filter out and review at meetings.16:46.30 
kens Hmm, OK I'll review mine16:46.43 
henrys ray_laptop:will do.16:46.47 
  bbiab16:47.22 
kens I'm off now, night all16:47.30 
Robin_Watts I just had a thought; by making 693025 bountiable, so Shelly can look at the patch, might that not make zeniko want to claim the bounty? After all, it's his patch...17:09.35 
henrys we can give them dual credit - if there is absolutely nothing that needs to be added to Zeniko's maybe shelley will give us a discount.18:41.08 
marcosw_ sorry I missed the meeting. I did open a bug for the issue that Til reported at the printing conference, but it's already been closed as invalid: http://bugs.ghostscript.com/show_bug.cgi?id=69301618:42.52 
henrys no problem marcosw_, how goes it?18:54.49 
mvrhel bbiaw20:17.11 
henrys I created the new bugzilla keyword "unsupported" for folks we are pursuing to get a support contract.20:26.00 
Robin_Watts tor8: (For the logs) I just pushed a commit to my repo on casper. It solves bug 693021 by avoiding overflows in the texture position calculations by keeping them in floats for longer.23:44.42 
  It changes 2000 or so images though, but all by tiny amounts - so before I commit it, I thought I'd check to see whether you can see a different way.23:45.14 
 Forward 1 day (to 2012/05/09)>>> 
ghostscript.com
Search: