| | <<<Back 1 day (to 2011/11/16) | 2011/11/17 |
| Robin_Watts | alexcher: If your cluster job misbehaves, please tell me. It *shouldn't* do, but... | 00:50.27 |
| alexcher | Robin_Watts: the cluster is fine for me. | 01:46.39 |
| mvrhel | good night | 07:05.02 |
| kens | marcosw_ I can reproduce the problem Gemma submitted | 08:29.33 |
| | Bug #692688 | 08:30.07 |
| marcosw_ | kens: that's good news. I tried to cut down the file to just the barcode but couldn't see the problem when I did that. | 08:30.10 |
| kens | I've reduced the file to the barcode and some other stuff, trying to reduce it still further | 08:30.33 |
| | The offending text is not part of the barcode. | 08:30.41 |
| | OK got it down to just the text. | 08:31.30 |
| | Still goes wrong | 08:31.36 |
| | Just need to hack the page size now | 08:31.42 |
| marcosw_ | yes, I know that, I meant just the text and the barcode. | 08:31.47 |
| kens | :-) You can remove the barcode too | 08:32.02 |
| marcosw_ | are you using acrobat to remove items from the file? If so I recommend the page size plug in from Windjack for changing the page size. | 08:32.26 |
| kens | Acrobat and then hand editing | 08:32.36 |
| | Here's a good one. | 08:32.43 |
| | The file renders incorrectly as expeted in GS, but opens bllank in Acrobat.... | 08:32.59 |
| marcosw_ | The resize page plugin is part of their AcroPack Demo: http://www.windjack.com/products/freestuff.html | 08:33.26 |
| kens | I just edit the MediaSize | 08:33.47 |
| | Ah, if I do that, tehn it comes out OK | 08:35.11 |
| | So it looks like its a problem with the page size. | 08:35.30 |
| | Which is odd | 08:35.36 |
| | OK file added to bug along with my minor findings | 08:46.05 |
| Robin_Watts | Anyone planning on using the cluster in the next hour ? | 11:20.05 |
| chrisl | Not me - hack away! | 11:20.26 |
| kens | nto me | 11:20.29 |
| Robin_Watts | If you do want to use it then disable watts from the dashboard before you run a job. | 11:20.41 |
| | I'm off for a run, and the rsync is taking ages. | 11:21.06 |
| kens | Noy going to be a problem, a long way from testing anything | 11:21.07 |
| Robin_Watts | Well, it got as far as trying to build the files that time. | 12:37.13 |
| | kens: FWIW: calling devenv from the command line is no good - cos it wants a UAC permission thing before it loads. | 12:57.41 |
| kens | Can't you exempt it ? | 13:07.54 |
| Robin_Watts | How ? | 13:09.17 |
| | (I'd like to do that anyway :) ) | 13:09.27 |
| kens | Not sure | 13:09.29 |
| | can always turn off UAC | 13:10.09 |
| Robin_Watts | marcosw_: Where are you at the moment ? | 16:37.40 |
| marcosw_ | Robin_Watts: I'm in Germany. Why? | 16:37.58 |
| Robin_Watts | Just wondered. | 16:38.07 |
| | I'm trying to get a windows cluster node set up. | 16:38.16 |
| | I have it so that it gets as far as trying to do the build. | 16:38.37 |
| | I'm fighting cygwin etc. | 16:38.43 |
| | I've had to make various changes in run.pl | 16:38.50 |
| | (I check to see if $^O is "cygwin" and then do windows specific stuff. | 16:39.27 |
| marcosw_ | Are you using gcc or the windows c compiler? | 16:40.55 |
| Robin_Watts | MSVC | 16:40.59 |
| | otherwise there would be no point :) | 16:41.05 |
| | This means all the binary names are different :( | 16:41.38 |
| marcosw_ | and you checked to see if the md5sums from the binary match the ones from gcc? Not much point otherwise... | 16:41.38 |
| Robin_Watts | marcosw_: I am fairly sure they won't match :) | 16:41.50 |
| | until we sort the 64bit arch stuff for windows being the same as the 64bit arch stuff for linux. | 16:42.26 |
| | But then they should. | 16:42.36 |
| | And even if we just have the cluster node do the build and not do the tests, it would be worthwhile. | 16:43.14 |
| | It's my personal windows machine, so the cluster won't be enabled full time. | 16:43.31 |
| | It's just a proof of concept for now. | 16:43.38 |
| marcosw_ | As long as you don't suggest moving all the cluster nodes from Linux to Windows :-) | 16:48.44 |
| henrys` | I see why 32 vs 64 bit bitmap rasters would lead to difference in memory but how does that result in visual differences? | 16:50.12 |
| | detectable by md5sum | 16:50.27 |
| Robin_Watts | different bitmap raster leads to different memory usage for a given size. That leads to different bandheight. | 16:51.03 |
| | Different bandheight leads to small differences in rendering. | 16:51.15 |
| henrys` | so full frame we should be okay and we could use the windows machine for that only. | 16:51.59 |
| Robin_Watts | henrys`: Theoretically, yeah. | 16:52.11 |
| | but the cluster doesn't know what machine is running what OS. | 16:52.21 |
| | (currently) | 16:52.27 |
| henrys` | right | 16:52.54 |
| | we don't want to fix the band height for some reason? | 16:53.23 |
| Robin_Watts | 'fix'? as in 'make it constant' ? | 16:53.39 |
| | Not all our test files are the same size, so a height that's good for one might not be good for another. | 16:54.12 |
| henrys` | meaning the bandheight would be calculated consistently on 32 and 64. | 16:54.39 |
| Robin_Watts | The bandheight is calculated by taking the max pattern size and dividing that by the size of the line pointer array and the size of the bitmap etc. | 16:55.28 |
| | the line pointer array changes size between 32 and 64bit. | 16:55.39 |
| | the size of the bitmap changes size with the bitmap_raster alignment | 16:55.59 |
| | hence to make a consistent bandheight we'd have to calculate it as if pointers were always 64bit and the raster alignment was always 8. | 16:57.32 |
| | which would result in smaller bandheights that we might expect on 32bit. | 16:57.57 |
| | (I'm sure ray will correct any mistakes in the above when he arrives and reads logs) | 16:59.02 |
| kens | doesn't understand this problem from Till at all :-( | 17:06.46 |
| | Heading off, goodnight all | 17:13.18 |
| henrys` | bye kens | 17:13.37 |
| Robin_Watts | night kens | 17:13.46 |
| mvrhel | bye kens | 17:13.50 |
| henrys` | I've been meaning to experiment with this (http://phash.org/) which might allow us to do approximate compares on the cluster. I'm betting it will be too slow. | 17:15.13 |
| | bbiab | 17:17.29 |
| henrys | Robin_Watts:looks like http://bugs.ghostscript.com/show_bug.cgi?id=689098 should be yours | 17:25.34 |
| Robin_Watts | it does, doesn't it :/ | 17:27.27 |
| henrys | ray_laptop:I assume you want to skip the meeting. | 17:56.59 |
| ray_laptop | well, that'd be fine with me. Marcos is traveling today anyway, isn't he ? | 17:57.42 |
| henrys | yes I think so. | 17:58.59 |
| | tkamppeter:all these package commands on ubuntu apt-cache apt-get dpkg apt-file, is there one program that does everything you'd expect? | 18:03.38 |
| Robin_Watts | marcosw_: In run.pl, there is a lot of "touch x; rm -f x; do something that creates x;" | 18:07.55 |
| | What's the purpose of the touch ? | 18:08.07 |
| | chrisl: feeling brave? :) | 18:15.29 |
| chrisl | Oops, are you still tinkering - it slipped my mind | 18:16.00 |
| Robin_Watts | I've taken my machine offline, so you should get proper results. | 18:16.36 |
| chrisl | OKay, thanks. TBH, I've been dredging my brain for new and interesting swear words to describe some Postscript producers, *and* Distiller for silently accepting the sh*te they emit - that obviously drove memories of cluster hacking from my mind! :-( | 18:18.22 |
| ray_laptop | mvrhel: I've posted the profile logs for 871 and 904 on bug 692674. I think we should make this enhancement. | 18:26.48 |
| | mvrhel: (I've done so). Can I assign it to you for the 'trivial CMS' work ? | 18:27.25 |
| mvrhel | ray_laptop: yes. I think I am going to tackle that one when I get this ripping out of wts finished | 18:27.54 |
| | I took a break from that, to get our png, tiff and jpeg devices to embed the device profile | 18:28.22 |
| ray_laptop | henrys: it took a little longer to get the timings on peeves because I wanted to make sure and do the timings a few times and had to make sure that a cluster run wan't in progress. | 18:28.24 |
| | mvrhel: awesome. That sounds like a real improvement | 18:28.47 |
| henrys | ray_laptop:are they still not using gs as a server? just one job at a time? | 18:30.41 |
| mvrhel | it is nice. now when we open up those files with photoshop, the colors are correct | 18:30.41 |
| ray_laptop | mvrhel: OK. I assigned it to you. Let me know if you find anything that it is doing now that maybe it shouldn't be (that may be a partial improvement). | 18:30.51 |
| mvrhel | ok | 18:31.08 |
| ray_laptop | henrys: yes, that's the point I made on the last comment #12. 904 as a server is faster that (except ps2pdf) running as a server compared to 871 run with single invocation. | 18:32.08 |
| mvrhel | based on that, I would almost close this bug and tell them to operate in that manner | 18:33.57 |
| Robin_Watts | tor8: I've updated my commits as per our discussion. | 18:34.26 |
| ray_laptop | the 'pdf_make_text_glyphs_table' however, isn't CMS related and is a LOT slower than it was in 871 | 18:35.55 |
| | and I'm not sure if some of the FT_ are freetype related | 18:36.27 |
| henrys | Bezier_Up is ft right? | 18:36.50 |
| chrisl | Yes, and any FT_ should be freetype. | 18:37.09 |
| ray_laptop | I'll post the log for 904 with -dDisableFAPI then | 18:37.57 |
| mvrhel | I see. So in the case of our current CMM when you do the individual invokes of gs, we regenerate the link for the initial white fill (mapping gray to RGB or CMYK what ever their device is). When you run it as a server we only do that once | 18:41.21 |
| | like wise for any other links | 18:43.04 |
| tkamppeter | henrys, I use usually "sudo apt-get install <package>" to install a package with all its dependencies. | 18:44.59 |
| ray_laptop | chrisl: please see comment #13 on bug 692674 -- it turns out that -dDisableFAPI is MUCH faster on ps2pdf conversion | 18:53.22 |
| | chrisl: but on the other conversions, it doesn't seem to make any difference | 18:54.08 |
| chrisl | ray_laptop: what about using ps2write? | 18:54.58 |
| | In *general* pdfwrite shouldn't be using FAPI (by that, I mean, it shouldn't be rendering glyphs), so it's hard to see where the time would be spent..... | 18:56.46 |
| ray_laptop | chrisl: they didn't time ps2write -- just pswrite | 18:57.13 |
| | chrisl: their scripts test ps2pdf, ps to ljet4, ps to pswrite and pdf to pswrite | 18:58.28 |
| | chrisl: I didn't go looking for work by looking at other conversions that they don't use. | 18:59.11 |
| chrisl | ray_laptop: does the job in question contain stuff that pdfwrite has to emit as raster? | 18:59.14 |
| ray_laptop | One of the issues is that they couldn't use gs in 'server' mode with pdfwrite because we don't have a way to force pdfwrite (or ps2write) to output the PDF and start a new one. | 19:00.12 |
| | chrisl: I haven't looked at the 'input.ps' so I'd be surprised if it had to emit anything as raster. I'll check... | 19:01.05 |
| chrisl | ray_laptop: the DisableFAPI profile seems to be empty | 19:01.36 |
| ray_laptop | chrisl: oops. let me check what happened | 19:03.31 |
| chrisl | ray_laptop: looking at prof_PS2PDF_904.log Freetype is definitely rendering glyphs in there, which it probably shouldn't be given that the text is still text when run through pdfwrite. | 19:04.47 |
| ray_laptop | chrisl: the PDF has Type1 font and just a bunch of text. | 19:06.19 |
| chrisl | ray_laptop: so whatever mechanism for short circuiting the "show" machinery that pdfwrite uses isn't working right | 19:06.37 |
| ray_laptop | chrisl: ahh. that could be. | 19:06.58 |
| chrisl | ray_laptop: I need to finish soon (brain's already melting!): could you send a short e-mail to Ken and me, just as a reminder we need to talk about this, please? | 19:08.01 |
| ray_laptop | chrisl: OK. I put the profile there. Sorry | 19:08.51 |
| | chrisl: OK. I'll summarize the ps2pdf FT issue to you and ken | 19:09.19 |
| | mvrhel: would you rather have profiles for a different case, like ljet4, that doesn't have the FT stuff intermingled with the CMS stuff ? | 19:10.58 |
| chrisl | ray_laptop: thanks. Looking at the DisableFAPI profile, I don't see any AFS functions featuring in there at all, so *something* with FAPI is causing pdfwrite to also render glyphs (then ignore the bitmap!). | 19:10.59 |
| ray_laptop | chrisl: and that would explain why DisableFAPI doesn't help with the other cases that render anyway (with FT or AFS). It just says that FT and AFS are about the same | 19:11.58 |
| | but if we are rendering with pdfwrite with FT and not with AFS, that would make a big difference | 19:12.45 |
| chrisl | ray_laptop: about the same is what I expect - the performance difference was generally too small to measure accurately. | 19:12.45 |
| | Yep, exactly | 19:12.55 |
| | ray_laptop: three functions near the top of the list (Bezier_Up, Split_Cubic and Insert_Y_Turn) are all functions from the FT renderer. | 19:14.45 |
| | Anyway, I'm heading off now - I'll talk to Ken about how to divide this one up tomorrow....... | 19:16.08 |
| | night all! | 19:16.10 |
| mvrhel | ray_laptop: no dont do any extra profiling for me | 19:19.05 |
| | it is pretty clear to me what the situation is color wise with these timings | 19:19.30 |
| ray_laptop | mvrhel: something funny with the profiles I think. _cmsComputePrelinearizationTablesFromXFORM shows up as a big hit on one profile, but on the other it is cmsSample3DGrid | 19:19.44 |
| | mvrhel: OK. probably the issue is to avoid the lcms altogether | 19:20.13 |
| | mvrhel: I'll leave it to you then, and go back to other stuff. | 19:20.39 |
| | mvrhel: are you available to look at gdevdevn.c for a few minutes. I'd like to discuss the 'bug' I found | 19:22.33 |
| mvrhel | sure | 19:22.46 |
| ray_laptop | in base/gdevdevn.c line 1302 we scan to find the most common color value | 19:24.00 |
| mvrhel | ok | 19:24.48 |
| ray_laptop | but it was sometimes finding that value == 0 was the most common, and encoding that as solid_not_100 value 0 | 19:24.54 |
| mvrhel | seems like that would be encoded differently | 19:25.29 |
| ray_laptop | so I changed line 1309 to: if (value > 0 && group_size[value] > largest_group_size) { | 19:25.35 |
| mvrhel | yes. that makes sense | 19:25.50 |
| ray_laptop | mvrhel: because 0 value components are handled via the 'colorants' bitmap | 19:26.01 |
| mvrhel | yes | 19:26.06 |
| | you are looking for the most "common" color with values not 100 or 0 | 19:26.22 |
| ray_laptop | so far, so good. But... I ended up with non-encodable colors this way and I didn't get them before the change :-/ | 19:26.55 |
| mvrhel | seems like you should have more room with this change | 19:27.33 |
| ray_laptop | mvrhel: we don't have to have a special check for value == 255 because it is already accunted for in solid_comp_count, and we don't switch to solid_not_100 unless (largest_group_size > (solid_comp_count + 1) | 19:28.40 |
| mvrhel | I fear I am not sufficiently familiar with the encoding that is done here. I probably need to spend a bit of time with it | 19:29.37 |
| ray_laptop | so I'm inclined to go ahead and make the change (and clusterpush, of course), but I just wanted to see if you could think of why it would be _worse_ | 19:29.49 |
| | mvrhel: one of wasting time on it is probably enough :-) | 19:30.11 |
| mvrhel | From my small understanding of how things work in the encoding, I would have expected the opposite result | 19:30.30 |
| | so, no I don't have any ideas why you would suddenly not be able to encode | 19:31.00 |
| ray_laptop | mvrhel: I think our time would be better spent doing a gdevmsep planar device that can handle all of the separation planes _without_ encoding | 19:31.04 |
| mvrhel | yes | 19:31.08 |
| | that is what I would like to do | 19:31.12 |
| ray_laptop | mvrhel: then the it could have a put_image (used by pdf14_cmykspot_put_image) to directly write the planar buffers | 19:31.59 |
| mvrhel | yes | 19:32.23 |
| ray_laptop | plus all of the painting of the separations for non-transparency files would be more like the case with transparency | 19:32.50 |
| | I'm thinking that initially we could have it like the pdf14 device that gets encoded colors much of the time, then migrate it to implement more hl_color painting procs | 19:34.33 |
| Robin_Watts | ray_laptop: If you're finished with mvrhel... I've been looking at getting a windows cluster node up. I was discussing it with henrys earlier, and I said that I'd expect differences between the windows cluster nodes and the linux nodes because of the different bandheights. | 19:37.49 |
| ray_laptop | Robin_Watts: yes, in particular 32-bit vs 64-bit line_pointers | 19:38.39 |
| Robin_Watts | 64bit windows uses a bitmap_raster of 8, 32bit windows uses a bitmap raster of 4. | 19:38.49 |
| | I'm using 64bit windows. | 19:38.53 |
| | so both have 64bit line pointers. | 19:38.59 |
| ray_laptop | Robin_Watts: well, then you _should_ have the same band heights | 19:39.12 |
| Robin_Watts | ray_laptop: AIUI no. | 19:39.29 |
| | 64bit windows has a sizeof(long) of 4, whereas on linux it's 8. | 19:39.46 |
| ray_laptop | oops. I missed your comment about the bitmap_raster alignment | 19:39.48 |
| Robin_Watts | right. | 19:39.53 |
| | So... I have 2 questions, which are really complements of each other. | 19:40.10 |
| ray_laptop | a long on linux 64-bit is 8 ?? | 19:40.16 |
| Robin_Watts | 1) If windows can get away with a raster alignment of 4, why can't linux? | 19:40.41 |
| ray_laptop | I thought that 'long' was only 64-bits on alpha | 19:40.58 |
| Robin_Watts | 2) If linux gets an advantage from having a raster alignment of 8, can't we get the same advantage for windows? | 19:41.26 |
| ray_laptop | Robin_Watts: on x86, it doesn't need _any_ alignment except for efficiency | 19:41.37 |
| Robin_Watts | right. | 19:41.47 |
| | I'm not convinced that we'll ever see any measurable advantage to aligning to 8 rather than 4. | 19:43.24 |
| ray_laptop | Robin_Watts: I just checked the gdevm64 and it always writes 32-bit chunks, max, so 32-bit alignment should be OK | 19:43.38 |
| | Robin_Watts: so would you change bitmap_raster to use ARCH_ALIGN_INT_MOD ??? | 19:46.14 |
| henrys | we really need to verify the performance assumption. | 19:47.22 |
| Robin_Watts | ray_laptop: If we only ever write using ints, then using ARCH_ALIGN_INT_MOD would make sense. | 19:48.13 |
| henrys | seems to me all archs should use ARCH_ALIGN_PTR_MOD for bitmap raster, is there some good reason we don't do that. | 19:48.27 |
| | ? | 19:48.36 |
| Robin_Watts | because we aren't reading/writing pointers ? :) | 19:48.48 |
| | ooh! My cluster node got through the building phase. | 19:49.39 |
| ray_laptop | hurray | 19:49.47 |
| Robin_Watts | It's getting jobs now, and I need to rewrite the jobs to use 'gswin64.exe' rather than 'gs' etc | 19:50.06 |
| ray_laptop | Robin_Watts: that or rename gswin64c.exe to gs.exe | 19:50.57 |
| | Robin_Watts: I recommend gswin64c.exe instead of gswin64.exe | 19:51.48 |
| Robin_Watts | ray_laptop: Yeah, I misttyped, sorry. | 19:52.01 |
| ray_laptop | np | 19:52.07 |
| Robin_Watts | Will invoking from cygwin automatically add the .exe ? | 19:52.31 |
| mvrhel | crap: when did all those segvs get introduced? | 19:53.14 |
| | was that me? | 19:53.15 |
| Robin_Watts | ray_laptop: That is a most excellent idea, and one I shall now use. Thanks. | 19:53.57 |
| mvrhel | ugh | 19:55.15 |
| | that is what I get for committing without a clusterpush test first. I thought my testing of the jpeg device was sufficient but apparenty I broke something in pdfwrite with my jpeg change | 19:56.55 |
| ray_laptop | mvrhel: oops. That means my clusterpush will fail, too, right ? | 19:58.49 |
| mvrhel | sorry! | 19:58.56 |
| ray_laptop | (I just pulled updates before doing the clusterpush) | 19:59.11 |
| ray_laptop | goes to abort. | 19:59.23 |
| mvrhel | hmm. let me undo that commit | 20:01.03 |
| ray_laptop | good! 'pending' jobs get aborted before they start! my thanks to marcos or Robin for that | 20:01.35 |
| | Robin_Watts: you may want to abort your run if you were at commit 23a2b8e | 20:02.20 |
| | off to lunch. bbiaw | 20:02.52 |
| mvrhel | ok | 20:06.43 |
| | sorry about that | 20:06.46 |
| | is it reverted now | 20:06.49 |
| | ray_laptop if you want to recommit | 20:06.56 |
| | you job | 20:07.00 |
| | your job | 20:07.04 |
| | oops | 20:07.15 |
| | ok. testing my ripping out of wts. off to lunch now. I will fool with my jpeg issue tonight. sorry about the bad commit everyone | 20:09.31 |
| | it seemed like such a simple change | 20:10.09 |
| kens | mvrhel2 pdfwrite uses DCT compression for all images by default. | 21:12.05 |
| | I imagine this is what is causing the problem when you add the ICC rpofile. | 21:12.18 |
| | Forward 1 day (to 2011/11/18)>>> | |
IRC Logs