| | <<<Back 1 day (to 2013/03/05) | 2013/03/06 |
| Robin_Watts | ray_laptop: gah. more bugs :( | 00:11.37 |
| ray_laptop | OK, I've analyzed the gevmpla (planar memory device) problem and have a fix for the problem I found (104-01.ps). Doing a run now... | 03:20.44 |
| | how we've kept going so long with some of these things not showing up (at least as indeterminate results), I don't understand | 03:27.01 |
| henrys | marcosw:I was thinking about a new mac but now I hear X11 support for high res is nowhere in sight. I heard they are pixel doubling. | 03:44.57 |
| marcosw | that's my understanding. | 03:46.10 |
| henrys | ugh | 03:47.56 |
| marcosw | otoh, having an i7 laptop is nice. | 03:48.51 |
| henrys | yes and being able to control multiple monitors was attractive to me. | 03:55.51 |
| | I already have i7 but it is not Ivy Bridge | 03:57.43 |
| marcosw | ssh amd64 | 04:01.08 |
| | run window :-) | 04:01.18 |
| | ^run^wrong | 04:01.24 |
| ray_laptop | Maybe, just maybe, I can get a reasonable base to work against to get my pdf14_clist_optimize branch to be "good enough". Given what I've tripped over, I probably could have gone ahead days ago and not made things any worse. :-/ | 04:40.56 |
| | hmm... a LOT of psdcmyk seg faults (9 seems like a lot) | 05:05.11 |
| | and none that are NOT psdcmyk. Probably a devn isssue | 05:06.17 |
| kens | Hmm Ray's fix for indeterminism sewems to have introdcued 8 seg faults with psdcmyk device. | 08:02.13 |
| chrisl | Well, his last clusterpush shows the seg faults, so he must have opted to commit with the problem, and presumably fix the seg faults next.... | 08:04.50 |
| kens | I hope so, he doesn't seem to have mentioned it on IRC | 08:05.13 |
| | Ah actually he has, right at the end of the log | 08:05.58 |
| Hijackal | Hello everyone. I'm trying to optizmize a pdf using gs64win with pdfwrite and dPDFSETTINGS=/screen. It's working pretty well, but on pages with raster graphics, the entire page (including, unfortunately, text flowing around the graphics) is rasterized. Is there a way to prevent this and restrict downsampling to the actual raster graphics? | 12:37.46 |
| Robin_Watts | Hijackal: You really need to speak to kens. He'll be back in an hour or two. | 12:39.23 |
| | The reason for the rasterisation is almost certainly that the file uses transparency. | 12:39.55 |
| | I suspect the suggestion will be to avoid the use of -dPDFSETTINGS | 12:40.17 |
| Hijackal | Is there a different way to achieve downsampling of raster images in the file? I assume they are resonsible for most of the file size. | 12:41.24 |
| Robin_Watts | Hijackal: Possibly. I'd suggest you read the documentation for all the pdfwrite switches, as I don't know them all. | 12:43.11 |
| henrys | Meeting day is also the daylight savings change here in the states. | 14:25.53 |
| kens | Extra confusion ! | 14:26.05 |
| nilli | I'm trying to modify pdf files with php, and the gs command line works fine in the terminal but fails when php tries to execute it. I've tried running it in the terminal as the same user as php (www-data) and it works fine in terminal. any idea how to troubleshoot this? command and output: http://pastebin.com/ZUWu9QJy | 14:32.15 |
| | I checked "which gs" from both places and got the same result, I've changed the command to specifically use /usr/bin/gs instead of just gs, but that didn't help | 14:33.19 |
| | there are no issues with write permission in the specified directory, which is evident since the user www-data can run the script just fine from terminal | 14:33.58 |
| chrisl | nilli: calling gs from php has been discussed at length over at stackoverflow - a quick search there would probably be profitable | 14:34.38 |
| nilli | I've searched for hours already, they all seem to agree that it's write permissions | 14:35.01 |
| chrisl | Have you established whether gs actually runs? | 14:35.38 |
| nilli | yes, as you can see in the pastebin it runs but stops at '**** Unable to open the initial device, quitting.' | 14:35.57 |
| | from php only, when run in terminal it processes the pages | 14:36.16 |
| chrisl | Well, that error does suggest that it cannot open the output file, which, in turn, suggests a permission problem | 14:36.56 |
| | Oh, and you are using a truly *ancient* version - we've just release 9.07 | 14:37.29 |
| nilli | I can't upgrade the server OS so it's stuck at Ubuntu 8.04.4 | 14:38.02 |
| chrisl | You could try using changing your code to call gs with strace to see why the file open (apparently) fails | 14:40.31 |
| | nilli: I don't know how PHP works, but in the call to gs, are | 14:42.43 |
| | the "-o" and the "/var/www/produktion/test/SHQKEQS4_CMYK.pdf" separate options - they should be | 14:43.04 |
| nilli | the php function is simply exec($command); where $command is any valid shell command | 14:43.12 |
| chrisl | Oh, hang on, your input and output are the same file? That may not be wise | 14:43.57 |
| nilli | no, the output has a _CMYK suffix | 14:44.10 |
| | input is a.pdf and output is a_CMYK.pdf | 14:44.33 |
| chrisl | Oops, yeh, so it does. | 14:44.35 |
| nilli | -o and the file path is separated by a space, that should be enough right? | 14:44.58 |
| chrisl | Yeh, should be fine | 14:45.07 |
| nilli | could -sDEVICE=pdfwrite be an issue, since the error message says it can't open the initial device? | 14:45.53 |
| chrisl | The option is correct | 14:46.42 |
| henrys | nilli: so if you generate a file in your home directory not /var/... it works? | 14:48.14 |
| nilli | not from php, no. if I login as the php user in my terminal, it works, but not if I run the php code from a browser. | 14:49.00 |
| henrys | or say /tmp where perms are not a problem | 14:49.04 |
| nilli | it really isn't a write permission in the directory, the directory has 777 | 14:49.26 |
| | the output file doesn't exist, so it's not blocking itself | 14:49.43 |
| henrys | chrisl's strace idea should work. | 14:52.34 |
| | well diagnose the issue | 14:53.00 |
| nilli | strace looks difficult :s | 14:53.56 |
| chrisl | Well, all the possibly useful (in this case) debug options in Ghostscript are only available in debug builds | 14:55.21 |
| nilli | somebody wrote 'I have since discovered the cause of this: a now defunct TEMP assignment in the .profile file for my user account. Removing that piece of code resolved the problem.' | 14:57.09 |
| henrys | nilli: perhaps error logged in /var/log - look at the time stamps of the files there after you run the job | 14:57.11 |
| nilli | but the user www-data doesn't have a home folder | 14:57.17 |
| chrisl | nilli: okay, but www-data will still have an environment, so it may have a value for TEMP | 14:58.10 |
| | nilli: but then, strace will tell you if a file open is failing, which file it is, and hopefully why | 15:00.57 |
| nilli | no relevant logs in /var/log | 15:02.41 |
| | someone at stackoverflow seems to have had the same issue, but the only clue to fix it is 'I have been fighting with this all morning, and I have no idea what I did, but the exact same code works now. ' -.- | 15:04.09 |
| chrisl | nilli: if you suspect it's the setting for TEMP, you could set TEMP yourself | 15:04.34 |
| nilli | to what? null? | 15:05.14 |
| chrisl | No, to a sensible value - like "/tmp" | 15:05.54 |
| nilli | ok, there was no TEMP environment and adding one as /tmp didn't change anything | 15:10.39 |
| henrys | chrisl:so clearly if I have a change to one of the vector devices (I do) - I stash - a stash pop is going to screw thing up? | 15:11.06 |
| Robin_Watts | henrys: possibly not. | 15:11.37 |
| chrisl | henrys: it *shouldn't* - git *ought* to realise the file has moved, and do the right thing - in theory | 15:11.54 |
| henrys | I'll just save a patch to be sure. | 15:12.09 |
| chrisl | Good plan ;-) | 15:12.21 |
| Robin_Watts | But a smarter move might be to commit the change properly. Then when you pull --rebase, it will do the right thing. | 15:12.23 |
| | or commit it on a branch and rebase the branch. | 15:12.42 |
| henrys | thanks guys | 15:13.31 |
| | I was all set to get the retina laptop and now I find out X11 doesn't support the resolution and won't in the near future. I use a lot of X apps. | 15:14.55 |
| | X may be dying - Ubuntu is replacing it. | 15:15.44 |
| nilli | thanks for trying, I might come back once I figure out the strace thing | 15:15.53 |
| henrys | nilli: normally the output is abbreviated there is an option to be verbose, you'll probably need that. | 15:16.35 |
| | see the man page | 15:16.48 |
| nilli | strace man? will do | 15:16.58 |
| henrys | right | 15:17.13 |
| chrisl | henrys: that's likely a big problem - X should die, but now there's two major players pulling in different directions to replace it. Ubuntu should work on Wayland, I reckon | 15:17.18 |
| Robin_Watts | That's the nice thing about standards... | 15:17.49 |
| | http://xkcd.com/927/ | 15:18.08 |
| | http://www.coffitivity.com/ | 15:18.46 |
| henrys | I guess if I had an ImageMagick substitute on the mac that supported the new resolution I'd be okay. | 15:24.11 |
| Robin_Watts | gah. The ghostscript stroking code is killing me. | 15:24.19 |
| | Hacks on hacks on hacks... | 15:24.24 |
| | henrys: What do you use ImageMagick for? | 15:24.38 |
| | I use convert, but then I view stuff in preview. | 15:24.59 |
| henrys | image comparison, discovering pixel coordinates that kind of stuff. | 15:25.02 |
| | no coordinates in preview | 15:25.22 |
| Robin_Watts | What formats ? | 15:25.32 |
| henrys | that doesn't matter much I can always convert as you do, some reasonable collection of raster formats. | 15:26.22 |
| Robin_Watts | henrys: I could update my javascript image viewer to give coords. | 15:27.05 |
| | That does pgm and pams. | 15:27.13 |
| | Image comparison can be done at the command line; compare in1 in1 out | 15:27.43 |
| henrys | that's actually interesting because chrome will be supporting the retina stuff soon. | 15:28.16 |
| Robin_Watts | I thought that macs didn't use the native resolutions anyway? | 15:28.38 |
| | (without a hack) | 15:28.50 |
| henrys | what I hear about retina is X needs to go into pixel doubling and things look really bad. That is what I hear, haven't verified myself | 15:30.42 |
| Robin_Watts | When Marcos showed me his retina mac back in SF 6+ months ago, you couldn't set unhacked macos to use the full resolution of the screen. | 15:31.40 |
| | I guess because it made all the window furniture stupidly small. | 15:31.57 |
| | So it ran in a mode where the extra resolution was used to make text etc look better (but only on apps that knew about it), but window borders etc were still the normal size. | 15:32.44 |
| | On apps that didn't know about the retina stuff it would pixel double. | 15:33.04 |
| | It sounds like they just haven't updated X to "know" about the retina stuff. | 15:33.34 |
| | And I bet they won't be unusably bad. | 15:33.44 |
| henrys | no the talk about X is it is beyond the architecture - no fix in sight. | 15:34.39 |
| Robin_Watts | henrys: right, so X apps will never look better than native apps that don't know about retina. | 15:35.42 |
| henrys | so I wonder if this is just text improvements: http://www.engadget.com/2012/07/31/google-chrome-21-stable-release-adds-retina-macbook-pro-support/ | 15:35.52 |
| Robin_Watts | looks like it to me. So X apps will look like the left hand edge of that picture. Hardly unusuable. | 15:36.48 |
| | unusable, even. | 15:36.53 |
| henrys | right but if 90% of the motivation to update is higher resolution ... | 15:37.55 |
| Robin_Watts | What's the resolution of your current mac? | 15:39.05 |
| | According to prof google, the retina ones have more than 4 times the number of pixels of the non-retina ones. | 15:40.12 |
| | So even when pixel doubling, you'd have a higher resolution. | 15:40.22 |
| henrys | maybe so I should just go look at one carefully and see if it is worth it. | 15:42.34 |
| Robin_Watts | henrys: I would spend some time looking at marcosw's next weekend, cos he can show you it with the hack to give full screen resolution. | 15:44.41 |
| henrys | yeah I will does he have a 13 or 15? | 15:45.06 |
| Robin_Watts | he doesn't fly economy. | 15:45.16 |
| tor8 | chrisl: wayland should have been "done" 5 years ago... if it's still not in a usable form I'd say it's a failure | 16:20.36 |
| | Robin_Watts: this password encoding stuff is hairy | 16:21.33 |
| | Robin_Watts: it helped a lot when I added actual unicode support to the win32 password dialog :) | 16:22.00 |
| Robin_Watts | tor8: hehe | 16:22.10 |
| tor8 | I'd been bashing my head against the wall trying to figure out why on earth I couldn't input unicode passwords properly... >.< | 16:22.35 |
| | but I'm having zero luck trying to open either of the A0 or kekeke files :( | 16:24.05 |
| | neither old nor with utf-8 / pdfdocenc patches can open them... | 16:24.39 |
| Robin_Watts | If you build the password in C, and pass in 0xa0 etc that way, will they open ? | 16:26.20 |
| ray_laptop | Looks like the segfaults are unlimited recursion. Sorry about that. Shouldn't take too long. | 16:48.28 |
| kens | chrisl Alex's last regression test had some grumles about extra dependencies in makefile, don't know if its benign | 16:48.35 |
| chrisl | kens: yes, they are not new | 16:49.24 |
| kens | chrisl no problem then :-) | 16:50.37 |
| chrisl | I think most of them are not "wrong" - for example, gx.h is included by another header included by gdevdevn.c | 16:51.41 |
| Robin_Watts | Does the makefile dependency checker not check for that? | 16:53.30 |
| chrisl | Dunno, I looked at them and decided not to address them in the same commit as the directory reord | 16:54.14 |
| | s/reord/reorganisation.... | 16:54.30 |
| Robin_Watts | IIRC we look in the makefile to find the list of .h dependencies given for each .c file. And we look in the .c file to see which ones are included. And any that are included, but not listed are flagged as errors. | 16:55.24 |
| chrisl | There's also the new directories to cope with | 16:56.02 |
| Robin_Watts | Why should that make a difference? | 16:56.26 |
| chrisl | I thought the dependency checker only worked on "base" | 16:56.51 |
| Robin_Watts | hmm, not sure. | 16:57.16 |
| | I think I've hit a nasty case here where my stroking code breaks down. | 16:57.28 |
| | Suppose we have a circular path that circles the origin at a radius of 10. | 16:57.56 |
| | and we stroke that with a line width of 1. | 16:58.05 |
| | The way the fast code works is that it forms a forward and a reverse path as we move along the path to be stroked. | 16:59.09 |
| | The forward path is the 'left' edge, the reverse path the right. | 16:59.27 |
| | When we get to the end we reverse the reverse path, and append that to the forward path. | 16:59.47 |
| | and then we can fill that. | 16:59.59 |
| | And that works just fine. | 17:03.10 |
| | but if the line width is too great (so that the expansion of the line causes the 'inside' edge to circle the origin in the opposite direction) we're screwed, cos the winding makes it look like an empty region | 17:04.33 |
| ray_laptop | OK, found the infinite recursion I had put in. | 17:09.19 |
| gem_ | hi | 17:27.12 |
| ghostbot | hey | 17:27.12 |
| gem_ | can you provide support for muPdf in android ? | 17:27.33 |
| | i am facing one prob | 17:27.40 |
| | ? | 17:27.41 |
| | there ? | 17:28.21 |
| Robin_Watts | gem_: hi. | 17:29.46 |
| gem_ | hi | 17:29.58 |
| Robin_Watts | the MuPDF devs are here, so shoot. | 17:30.10 |
| gem_ | i am facing one prob in muPdf in android | 17:30.11 |
| | ok | 17:30.18 |
| | i am getting Fatal signal 11 error if in finish activity before rendering page | 17:31.10 |
| Robin_Watts | gem_: So you're using our lib in your own app? | 17:31.32 |
| gem_ | yes | 17:32.01 |
| Robin_Watts | Fatal signal 11 is SEGV. | 17:32.11 |
| gem_ | yes | 17:32.17 |
| | thats correct | 17:32.23 |
| Robin_Watts | so, while you have a native call going on, you kill the activity? And you're surprised that it dies? :) | 17:32.45 |
| | OK. My standard spiel about MuPDF on Android... | 17:33.36 |
| gem_ | actually i make back press on the phone while the page is being reddered | 17:33.46 |
| Robin_Watts | MuPDF is a set of core C libraries for manipulating/loading/rendering PDF (and XPS and CBZ) files. | 17:34.32 |
| gem_ | is there any way to cancel the rendering process ? | 17:34.36 |
| Robin_Watts | Together with those libraries we have some tools (utilities, and example viewers for various systems). | 17:35.07 |
| gem_ | ohh great | 17:35.21 |
| | so can you solve my problem | 17:35.29 |
| | ? | 17:35.32 |
| Robin_Watts | The tools tend to be thin layers of OS specific stuff to expose functions within the libs. | 17:35.51 |
| | As such, we try to bury the complexities within the libs as much as possible. | 17:36.11 |
| | The Android viewer is an example of this; the C level is the published API. | 17:36.30 |
| ray_laptop | hmm... git managed to cope with moving files (into devices) AND merging in my changes. | 17:36.54 |
| ray_laptop | is relieved | 17:37.01 |
| Robin_Watts | the java classes you get in that app are not a published API - they are just enough to make our example viewer work. | 17:37.23 |
| | So, now to your question... | 17:37.36 |
| | The C level does indeed have a way to cancel an ongoing render. | 17:37.47 |
| gem_ | but i can make the complete app with your lib | 17:37.47 |
| Robin_Watts | When you call the render functions (either fz_document_run_page or fz_run_display_list) you have the option to pass in a cookie. | 17:38.22 |
| ray_laptop | but git didn't fix up my source views in Visual Studio :-( | 17:38.28 |
| Robin_Watts | The cookie is a pointer to a structure. | 17:38.31 |
| gem_ | i have this mupdf.c file | 17:38.47 |
| Robin_Watts | one element of this structure is an 'abort' field. | 17:38.59 |
| gem_ | is that possible to cancel rendering through this | 17:39.02 |
| | ? | 17:39.03 |
| Robin_Watts | If you set the abort field during a render, then the render will abort as soon as possible. | 17:39.33 |
| | mupdf.c is not part of the libs; it's part of the android app specifics. | 17:40.08 |
| | As such mupdf.c does not (IIRC) pass a cookie in, and does not provide a way to set the abort field in that cookie. | 17:40.43 |
| | You could of course change it to do so. | 17:40.57 |
| gem_ | So how can i pass that cookie in through android java code | 17:41.28 |
| | have you exposed that feature outside ? | 17:41.40 |
| Robin_Watts | gem_: As the code stands at the moment, you cannot do it from java. | 17:41.59 |
| | We've not exposed that feature, because we haven't needed it in our viewer. | 17:42.11 |
| gem_ | But i think thats a really basic requirement that every app should have | 17:42.54 |
| | !! | 17:42.56 |
| Robin_Watts | If you want it, you'll need to make the changes to the java/JNI. | 17:42.59 |
| | gem_: Which bit of "the published API is the C level library API" did you not understand ? | 17:43.20 |
| ray_laptop | chrisl: I thought you wanted contrib kept off to itself, but 'devices' has contrib.mak. shouldn't it just remain in base ? | 17:43.29 |
| Robin_Watts | gem_: It's not a big change to push through. | 17:44.22 |
| chrisl | ray_laptop: that contrib.mak builds devices in "devices", it used to live in "base" | 17:44.26 |
| Robin_Watts | It's something we'd do for a supported customer or a commercial licensee. | 17:44.42 |
| ray_laptop | chrisl: OK, so this is different to the contrib.mak that's in contrib (file name is a bit confusing, maybe that could be changed) | 17:45.20 |
| Robin_Watts | It's something we are unlikely to do for a free user, because our app does not need it, and would not use it. The interface you are asking us to embellish is not a public one. | 17:45.40 |
| | gem_: I'm sorry I can't offer you a simple instant fix. | 17:46.23 |
| chrisl | ray_laptop: it is different yes - I don't know the history of it. If we consider the devices in "devices/contrib.mak" to be "ours" now, we could move them to "devices/devs.mak" | 17:46.32 |
| ray_laptop | or change the 'contib' directory to 'contrib_unsupported' | 17:46.36 |
| Robin_Watts | gem_: You say you are using this in your android app... can you give us details of this app? | 17:47.00 |
| chrisl | ray_laptop: we could rename the "contrib" directory just to "unsupported" | 17:47.35 |
| Robin_Watts | chrisl: ugh. | 17:47.51 |
| chrisl | Robin_Watts: okay, how about "not_ours"? | 17:48.14 |
| Robin_Watts | I think 'contrib' us understood to mean 'not_ours' | 17:48.36 |
| ray_laptop | chrisl: actually, I have no problem with putting the contrib.mak devices into devs.mak | 17:49.10 |
| Robin_Watts | ray_laptop: How does that work for a commercial release when the contrib devices aren't included? | 17:49.36 |
| chrisl | Actually, some of these devices in "devices/contrib.mak" could probably be dropped: "The Hercules Graphics display"? | 17:49.57 |
| Robin_Watts | chrisl: but!!!! | 17:50.11 |
| chrisl | Robin_Watts: the "contributed" devices in "devices" aren't dropped for the commercial release | 17:50.22 |
| | I wonder how many still functional SPARCprinters are out there? | 17:52.10 |
| ray_laptop | chrisl: good question. Maybe in the Smithsonian ??? | 17:52.32 |
| Robin_Watts | gem_: You've gone quiet... | 17:52.51 |
| marcosw | chrisl: we could find out by removing support from ghostscript and see how many people notice. | 17:52.53 |
| chrisl | marcosw: good idea! | 17:53.23 |
| | Also, who is still using a "AT&T 3b1 Unixpc monochrome display"..... | 17:53.29 |
| marcosw | I think henrys should add an agenda to discuss removing obsolete devices | 17:54.05 |
| kens | I'm in favour | 17:54.14 |
| ray_laptop | chrisl: how about making a directory in devices "old" or "obsolete" | 17:54.26 |
| chrisl | ray_laptop: we discussed that ages ago - there was considerable resistance...... | 17:55.20 |
| ray_laptop | it would clean things up a lot and make the builds faster (since we default to including a lot of this crap) and the executable smaller | 17:55.26 |
| | chrisl: do you remember who was resistant ? Surely not I ;-) | 17:56.03 |
| Robin_Watts | ray_laptop: We shouldn't stop including devices in the distro just because we don't use them. We could stop building them by default though. BUT... if we did that we should have a 'christmas tree' build that was done to test that they don't at least stop compiling. | 17:56.43 |
| chrisl | ray_laptop: I can't remember - the consensus was, IIRC, if we want to remove stuff, just do it. | 17:56.57 |
| ray_laptop | we could leave them in the devices tree under obsolete for a bit, and take them out of the default build to see who squawks | 17:57.07 |
| chrisl | My suggestion would be that we remove candidate devices from the default build for a release, if no one complains, then we purge the source. | 17:57.40 |
| ray_laptop | chrisl: how about we collect them in a obsolete or other folder so when nobody squawks, we don't have to dig out which files to delete | 17:59.35 |
| Robin_Watts | contrib/obselete seems entirely reasonable. | 18:00.13 |
| kens | I think we should leave this to Sunday and discuss it then..... | 18:00.17 |
| ray_laptop | or just delete the files when we remove them from the build (and rely on being able to pull them back from git if we get a squawk) | 18:00.24 |
| kens | In the meantime, goodnight all | 18:00.26 |
| Robin_Watts | night kens | 18:00.31 |
| ray_laptop | nite, kken | 18:00.33 |
| chrisl | ray_laptop: I think it was preferred that a simple build option could reintroduce the device, so if anyone needs it, they're not stuck until the next release, or need to faff about with copying files around. | 18:01.32 |
| | ray_laptop: frankly, I don't much care, as long as we come up with a list of devices we want to ditch, and follow through with it | 18:02.36 |
| Robin_Watts | chrisl: right, so put stuff in contrib/obselete, and then have a --obselete option to put them back in? | 18:03.20 |
| chrisl | Robin_Watts: so two lots of build changes :-( | 18:03.42 |
| | ray_laptop: in the meantime, how about renaming "devices/contrib.mak" to "devices/devs2.mak" or something? | 18:09.17 |
| Robin_Watts | So, excuse me for being dim here... devices/contrib.mak currently contains the build rules for what set of devices? | 18:10.38 |
| chrisl | a subset of the devices contained in "devices" | 18:11.23 |
| Robin_Watts | Ah, then I agree, giving that a name that doesn't include contrib would be a good thing. | 18:11.46 |
| chrisl | These *seem* to be contributed devices that we "took on", as opposed to the ones in "contrib" which we didn't (I'm guessing!) | 18:12.25 |
| ray_laptop | OK, fixed the SEGV's that I had put in. Commit pushed. | 18:13.50 |
| | now, maybe, I can get a clean enough regression run of my pdf14_clist_optimize branch to get it in :-/ | 18:15.16 |
| | I did happen to see some code that concerned me in gx_dc_devn_masked_fill_rect. There's if (state.pcdev == dev || ptile->is_simple) and sometimes ptile comes in == 0, so we are relying on an invariant that ptile will ONLY be 0 if state.pcdev == dev | 18:18.58 |
| chrisl | Robin_Watts: alternatively, we move all the devices built by "devices/contrib.mak" into "contrib" and have them built by "contrib/contrib.mak" | 18:19.25 |
| ray_laptop | but if it is ever not the case, it will cause an immediate SEGV, so I guess I'll leave it alone | 18:19.52 |
| Robin_Watts | Devices in contrib are unsupported, right? Wheras the ones in devices (even though they may have been contributed) are supported (at least nominally), yes ? | 18:20.41 |
| chrisl | Robin_Watts: that's what I thought, but not according to the comments in devs.mak | 18:21.19 |
| ray_laptop | strange. The first regression run I did after the devices were moved gave me a LOT of 'new warnings' that look bogus: | 18:35.26 |
| | ./base/gdevcif.c:51: warning: 'start' may be used uninitialized in this function | 18:35.27 |
| | ./base/gdevpdf.c:1219: warning: statement with no effect | 18:35.29 |
| | but those files aren't even in 'base' anymore | 18:35.42 |
| | also a lot of 'extra dependency' makefile warnings | 18:36.25 |
| chrisl | ray_laptop: there may some spurious "new warnings" warnings because of how the script detects actual new ones | 18:36.40 |
| | The makefile warnings are also not new | 18:37.12 |
| ray_laptop | chrisl: OK. I figured they were spurious. Seems like a funky way to report 'new warnings', however if the warning is from an older build | 18:38.00 |
| chrisl | ray_laptop: it is odd that the path was wrong, but I don't think anything to worry about | 18:38.55 |
| Robin_Watts | malc_: Aha. Missed you yesterday. | 18:43.08 |
| | I've looked, and we are culling text correctly. | 18:43.28 |
| | So I looked at your example code, and became confused. | 18:43.39 |
| malc_ | Robin_Watts: all ears | 18:43.47 |
| Robin_Watts | Are you saying that you did: "ttest 256" and "ttest 4096" and got more or less the same results ? | 18:44.19 |
| malc_ | yep | 18:44.28 |
| | and this with both main.pdf (giant text page) and out.pdf made of a single giant jpeg for instance | 18:44.57 |
| Robin_Watts | As far as I can see from that code, ttest 256 will draw main.pdf at 256x256 size. | 18:45.06 |
| malc_ | aye | 18:45.13 |
| Robin_Watts | and ttest 4096 will draw the same pdf at 4096x4096 size. | 18:45.27 |
| malc_ | aye | 18:45.30 |
| Robin_Watts | That is not what you lead me to believe. | 18:45.37 |
| malc_ | what were you beliving in then? | 18:45.57 |
| Robin_Watts | I understood that you were telling me that ttest 256 would draw a 256x256 section of the 4096x4096 page. | 18:45.59 |
| | i.e. same zoom, just a smaller region. | 18:46.12 |
| malc_ | erm that's what it does | 18:46.15 |
| | it shows fragment of an image argv[1]xargv[1] sized | 18:46.36 |
| Robin_Watts | So the output from ttest 256x256 will be a subsection of the 4096x4096 image? | 18:47.10 |
| malc_ | yes | 18:47.15 |
| Robin_Watts | i.e. I could load the 4096 image into a bitmap editor and cut out exactly the 256x256 region that would be produced? | 18:47.39 |
| malc_ | yes | 18:48.02 |
| Robin_Watts | Unless I've misread the code, that's not the case. | 18:48.23 |
| malc_ | Robin_Watts: specifically for you i made the code to be be as portable as humanly possible, why don't you give it a whirl? | 18:48.45 |
| | it even has an #ifdef _WIN32 (so that i can avoid recalling the exact semantics of getsystemtimeasfiletime) | 18:49.15 |
| Robin_Watts | You read the page size as rect. | 18:49.31 |
| | Then you make a bbox from this. | 18:49.41 |
| | oh. I see. | 18:49.53 |
| malc_ | .x0 +w .y0 + h you see i suppose | 18:50.14 |
| Robin_Watts | yes, I was confused by you changing bbox, but not rect, sorry. | 18:50.41 |
| malc_ | and the code in draw/ does abosollutely no culling whatsoever leave alone correctly | 18:51.28 |
| Robin_Watts | malc_: The code that does the culling is in fz_run_display_list. | 18:52.16 |
| | Each node in the displaylist has a rectangle stored with it. | 18:52.32 |
| | and we map that rectangle through the ctm to see if that node is visible or not. | 18:54.14 |
| | If it's not visible, then we don't call the draw/ stuff at all. | 18:54.27 |
| malc_ | Robin_Watts: you are mistaken, tiles are not stored in this particular list produced by the code ttest.c, why - escapes me | 18:56.46 |
| Robin_Watts | tiles? | 18:57.12 |
| malc_ | sperate nodes | 18:57.23 |
| | whatever you want to call this | 18:57.27 |
| Robin_Watts | nodes most certainly must be stored. | 18:57.35 |
| malc_ | only one things gets pushed into draw text which contains EVERYTHING | 18:57.40 |
| Robin_Watts | The display list is a list of nodes. | 18:57.41 |
| malc_ | that's why hash dominated the profile | 18:57.48 |
| Robin_Watts | Ah! | 18:57.49 |
| | You have a single text object that contains all the text on the page. | 18:58.25 |
| malc_ | yep | 18:58.36 |
| Robin_Watts | hence the node has a rectangle that covers the entire page. | 18:58.37 |
| | Well, there is nothing we can do about that, short of splitting the text object into smaller ones. | 18:59.00 |
| | So, all the text on this PDFs page must be in the same font/size etc. | 18:59.19 |
| malc_ | you can at least test the glyph bboxes inside draw_text against supplied scissor (which as of now is simply ignored more or less) | 18:59.31 |
| | it is | 18:59.40 |
| | and as i mentioned, same thing happens with images, and large images are not at all uncommon in pdfs | 19:00.10 |
| Robin_Watts | The scissor rectangle doesn't make it into the draw functions. | 19:00.55 |
| | Ignore me. | 19:01.02 |
| malc_ | gladly, given that i stepped through this code two days ago :) | 19:01.26 |
| Robin_Watts | ok, so in fz_render_glyph we get the scissor. | 19:02.26 |
| | We lookup the glyph in the hash. | 19:02.42 |
| | sorry, let's go up one stage. | 19:04.18 |
| | In fz_draw_fill_text... | 19:04.36 |
| | For each glyph, we request the glyph from the cache. | 19:04.50 |
| | by calling fz_render_glyph. Once we have that we attempt to draw it, and that falls out pretty quickly if it's not in the scissor region, I think. | 19:05.45 |
| malc_ | profile is dominated by the "hash" not rendering | 19:06.44 |
| Robin_Watts | so I'm guessing that you'd like us to avoid doing the work in fz_render_glyph if the glyph is invisible ? | 19:06.46 |
| malc_ | i'd like it not to attempt to do the work that'll be thrown away anyways, and my guess is you want the same | 19:07.18 |
| Robin_Watts | In order to know whether the glyph is visible or not, we need to know how big the glyph is. | 19:07.56 |
| | In order to know how big the glyph is, we need to look it up in the cache. | 19:08.13 |
| malc_ | let's switch to images then, why is situation the same there too? | 19:08.56 |
| Robin_Watts | What your test doesn't do, which it perhaps should, is to rerender the page several times (just put a for loop around the call to display lists). | 19:09.06 |
| | That way the first one will be slow as the cache fills up, but then should be faster as we won't need to actually do the freetype work each time. | 19:09.27 |
| | OK, for images... | 19:09.30 |
| | You were testing a PDF file with a single big image in? | 19:09.42 |
| malc_ | aye | 19:10.27 |
| Robin_Watts | we can't cull that at the displaylist level because it's in every tile. | 19:10.42 |
| malc_ | ttest$ ./ttest 4096 | 19:11.15 |
| | 0 took 0.620970 sec | 19:11.15 |
| | 1 took 0.299794 sec | 19:11.15 |
| | 2 took 0.297864 sec | 19:11.15 |
| | 3 took 0.297874 sec | 19:11.15 |
| Robin_Watts | and when we come to display the image in the first tile, we decode the entire thing, and cache it. | 19:11.16 |
| malc_ | 4 took 0.299131 sec | 19:11.18 |
| | ttest$ . | 19:11.20 |
| | ttest$ ./ttest 64 | 19:11.26 |
| | 0 took 0.621625 sec | 19:11.27 |
| | 1 took 0.296477 sec | 19:11.27 |
| | 2 took 0.297068 sec | 19:11.27 |
| | 3 took 0.296430 sec | 19:11.27 |
| | 4 took 0.296206 sec | 19:11.29 |
| ray_laptop | Robin_Watts: For text positions that are totally outside the clip path (visible region), you skip the rendering ? (as long as the text advance is moving away from the visible region) | 19:11.38 |
| Robin_Watts | ray_laptop: Nope. | 19:12.05 |
| | That would be a possible improvement. | 19:12.18 |
| ray_laptop | Is that not possible? I know if the text advance is _toward_ the visible region, you have to get the advance info from the glyph to update the position | 19:12.58 |
| Robin_Watts | ray_laptop: It may be possible, yes. But we don't do it yet. | 19:13.33 |
| ray_laptop | Robin_Watts: just checking. | 19:13.43 |
| Robin_Watts | malc_: So for those timings you do see that runs after the first are much faster. | 19:14.30 |
| ray_laptop | gs doesn't do that either, but we are less concerned with rendering a subset quickly (as when zoomed in) | 19:14.52 |
| Robin_Watts | but you're also seeing that (in this case), the time to run the list is dominating the timings overall. | 19:15.19 |
| malc_ | Robin_Watts: and are the same regardless of the target size | 19:15.31 |
| Robin_Watts | malc_: Right. The actually rendering times are insignificant. | 19:15.55 |
| | We could slow down the rendering a bit, and then you'd see what you want? :) | 19:17.38 |
| | back in 10 mins, sorry. | 19:18.00 |
| malc_ | Robin_Watts: thing is, i am not on core i7, so i do care about speed | 19:18.08 |
| Robin_Watts | malc_: Can you do timings for the image one? | 19:19.11 |
| | malc_: I will ponder on ray's suggestion. | 19:19.20 |
| malc_ | actually no, the situation is different there | 19:21.03 |
| | ttest$ ./ttest 4096 | 19:21.13 |
| | 0 took 0.683050 sec | 19:21.14 |
| | 1 took 0.083070 sec | 19:21.14 |
| | 2 took 0.082947 sec | 19:21.14 |
| | 3 took 0.083551 sec | 19:21.14 |
| | 4 took 0.082358 sec | 19:21.16 |
| | ttest$ . | 19:21.18 |
| | ttest$ ./ttest 64 | 19:21.23 |
| | 0 took 0.605439 sec | 19:21.23 |
| | 1 took 0.002354 sec | 19:21.23 |
| | 2 took 0.002351 sec | 19:21.23 |
| | 3 took 0.002180 sec | 19:21.23 |
| | 4 took 0.002223 sec | 19:21.26 |
| ray_laptop | Robin_Watts: Can you get the max vertical extent of a font without rendering all the glyphs ? (for lines enough above or below the visible region). Previously I was thinking mostly horizontally | 19:22.21 |
| | but skipping entire lines above or below would be nice. It just means knowing when part of the glyph may drop down into the visible region (below baseline) or paint into the visible region when the baseline is below the region | 19:24.24 |
| malc_ | Robin_Watts: and besides rendering _the same_ tile is pointless, we are rendering successive ones anyways and these will not benefit from caching so every 64x64 tile will take the same time (the same as one 4096 one, which is bollocks) | 19:29.22 |
| Robin_Watts | malc_: Right. Populating the font cache will help with successive tiles too. | 19:31.23 |
| | Likewise images. | 19:31.42 |
| malc_ | reality disagrees with you | 19:31.50 |
| Robin_Watts | as politeness with you, it seems. | 19:32.05 |
| sebras | Robin_Watts: only if the rendering is a significant factor... | 19:32.13 |
| Robin_Watts | In the image case above, the majority of the first run time is clearly the image decode. | 19:32.44 |
| | and the subsequent renders completely almost immediately. | 19:32.55 |
| | s/completely/complete/ | 19:33.05 |
| malc_ | Robin_Watts: this test draws the same tile over and over again | 19:33.18 |
| | not neighboring ones | 19:33.24 |
| Robin_Watts | The same image is in all tiles, so we'd expect to see the same vastly reduced time for a test that rendered those too. | 19:33.46 |
| | So I declare the image case to be not a problem. | 19:34.08 |
| | actually... that's not quite true. | 19:34.24 |
| | For images there are 3 essential stages. | 19:34.32 |
| sebras | Robin_Watts: yes, but in the image case you can see that the "64" case is faster (0.002s) than the "4096" case (0.08s)... why is there not a similar reduction in time for text? | 19:34.36 |
| Robin_Watts | The first stage is the decode from source format to pixmap. | 19:35.08 |
| sebras | Robin_Watts: in that case the time is about 0.3s regardless of whether in the "64" or "4096" case. | 19:35.18 |
| Robin_Watts | That pixmap is then cached (and will be useful for all tiles at the same zoom level that include any part of the image) | 19:35.37 |
| | Stage 2 is any scaling of the pixmap that may occur. This *will* be specific to the tile. Again it's cached, but that won't help subsequent tiles. | 19:36.31 |
| | Stage 3 is the actual rendering of the scaled pixmap to the output pixmap. This will happen on every render. | 19:36.53 |
| | So in the image case, we are doing the best we can I believe. | 19:37.13 |
| | Anyone want to disagree with that? | 19:37.35 |
| | malc_: By all means update your test to draw subsequent tiles. | 19:38.11 |
| malc_ | Robin_Watts: doing it now | 19:39.21 |
| Robin_Watts | OK, I'll wait to see the results from that. | 19:39.49 |
| sebras | regrets having to leave briefly at this thrilling moment... | 19:44.42 |
| malc_ | Robin_Watts: well, the results show such an amazing variability so i can not draw any real conlusions from them | 19:52.26 |
| Robin_Watts | malc_: Can you share the results anyway, please? | 19:54.43 |
| malc_ | Robin_Watts: i can share the code, as i said the output i get depends on things i do not understand | 19:55.17 |
| Robin_Watts | You pasted times for the runs of the old tests on text and image files here. Can you not do the same for the new test? (In a pastebin if they are large) | 19:56.17 |
| malc_ | http://pastebin.ca/2328831 | 19:57.46 |
| Robin_Watts | Is that for a text file, or an image file? | 19:58.38 |
| malc_ | image | 19:58.42 |
| | http://boblycat.org/~malc/ttest.c | 19:59.10 |
| Robin_Watts | ok. It would seem to suggest that the first 128 tile doesn't include any part of the image. | 19:59.23 |
| malc_ | it's white yes | 19:59.59 |
| Robin_Watts | The 1.5 seconds time is clearly dominated by the time taken to decode the image. | 20:00.01 |
| | tiles 4/5/6 are then purely painting from the cached decoded version. | 20:00.57 |
| | the 128 tiles are 1/4 of the size of the 256 ones. | 20:01.09 |
| | and the 128 times are roughly 1/4 of the time of the 256 ones. | 20:01.23 |
| | so that seems entirely sane to me. | 20:01.42 |
| | Is there some aspect I am overlooking? | 20:02.44 |
| malc_ | it does, sorry for doubting you | 20:02.53 |
| Robin_Watts | no worries. | 20:02.58 |
| malc_ | yes, the times i get from real application are somewhat more variable | 20:03.09 |
| | due to cache eviction i suppose | 20:03.16 |
| Robin_Watts | possibly, yes. | 20:03.22 |
| malc_ | so image tiling question is off of the agenda, text stays | 20:03.55 |
| Robin_Watts | So, I will try to look into optimisations for the text culling. | 20:04.13 |
| | I'll have to look to see what information is available to me at that stage. | 20:04.43 |
| malc_ | thing is, there's this multicolumn mode that i use, and it really takes a beating due to repeated glyph hashing rendering and such like | 20:05.01 |
| Robin_Watts | ray's suggestion would help to evict half of the stuff. | 20:05.31 |
| | If I have an overall worst case glyph bbox, then I can evict even more. | 20:05.52 |
| malc_ | anyhow, time to try to get some sleep, thanks, good night | 20:09.51 |
| | Forward 1 day (to 2013/03/07)>>> | |
IRC Logs