| | <<<Back 1 day (to 2012/12/02) | 2012/12/03 |
| ray_laptop | Finally able to throw some code over the transom to cust 532. Several false starts (down rat holes) and a major git glitch, but now the changes to limit interpolation to less than full device resolution are decent (with minimal testing). Trying to regression test this isn't reasonable, so I have to depend on their QA. | 06:25.14 |
| | Currently it is a compile time limit since making it a parameter is a bit more work. The simplest way would be to make it a device parameter. but that's for later. | 06:28.06 |
| | If anyone thinks this is useful for other users/customers, let me know. It should be pretty easy to port (now, with the simplified changes) to HEAD. | 06:30.00 |
| kens | chrisl ping | 09:20.42 |
| chrisl | kens: pong | 09:20.49 |
| kens | can you sopare me some time to talk about TrueType fonts, specifically the one I'm having trouble with ? | 09:21.18 |
| chrisl | Eek! Yeh, okay | 09:21.55 |
| kens | Well I'm having troulbe understanding what ttfdump is making of the loca table | 09:22.11 |
| | If you look at teh font I sent, I see that the loca table is located at offset 0x3780 and has length 0xB4 | 09:22.47 |
| | If I look at offset 0x3780 in the file, I see that it begins 0000 0036 | 09:23.23 |
| | THe length of B4 is 180 bytes, and since these are shorts that means 90 Glyphs | 09:23.54 |
| | MNow ttfdump says that the first glyf offset is 0036, while it *should* be 0000 | 09:24.16 |
| | IIRC one of the tools you were using said that the loca table was too short | 09:24.39 |
| | SO I'm kind of puzzled. | 09:24.46 |
| | Is the offset to the start of the table gven from the start of the TT file ? | 09:25.00 |
| | THe spec says it is. | 09:25.17 |
| chrisl | More than one tool - both freetype *and* Ghostscript's Postscript TTF parser got a too short loca | 09:25.30 |
| kens | OK so can you tell me how they are determining that ? ttfdump didn't give me that error | 09:26.06 |
| | NB the glyf offsets that the loca table comes olujt with are ridiculous though | 09:27.35 |
| | ttfdump is giving me 32-bit values, not 16-bit shorts | 09:27.48 |
| | SO it *will* run off the end of the table | 09:28.00 |
| chrisl | Hmm, freetype's table parsing is "interesting" - IIRC, that's one of the places it fills in a structure using byte offsets from a "template", so it's not too easy to follow it | 09:28.10 |
| kens | Well it 'looks like' ttfdump is using a different format of loca table (and I think there was some suggestion of a loca table having the 'wrong' format in one of the tools) | 09:28.51 |
| | But the 1.0 spec only has the one format of loca table.... | 09:29.01 |
| | Ah, I think I see | 09:29.46 |
| chrisl | OpenType has two version | 09:30.00 |
| | s | 09:30.01 |
| kens | According to the APple TT spec, the size is specified in the HEAD table | 09:30.06 |
| | I bet we copy the HEAD table, but generate the loca, and the HEAD table specifies a 32bit offset | 09:30.29 |
| chrisl | Hmm, that *should* be easy to resolve.... in theory! | 09:31.06 |
| kens | I think so, the indexToLocFormat is 256 in teh HEAD table, I wonder what that means ;-) | 09:31.32 |
| chrisl | It's broken! | 09:31.51 |
| kens | THe spec says 0 for short offsets, 1 for long.... | 09:31.54 |
| chrisl | I bet interpreters treat it as 0 for shorts, !0 for longs | 09:32.13 |
| kens | I was thinking that very thing | 09:32.21 |
| | OK thanks chrisl, I think I see where to go now | 09:33.01 |
| | talking about it helps a lot | 09:33.07 |
| chrisl | NP. BTW, I found the problem with the CID substitution problem...... | 09:33.33 |
| kens | Oh even better, what was it ? | 09:33.41 |
| chrisl | This file uses a non-identity CIDtoGIDMap, which FAPI wasn't handling - all the other examples we have use either no CIDtoGIDMap or Identity.... | 09:34.37 |
| kens | Oh, that's weird, I don't think I've ever seen that before. | 09:34.58 |
| chrisl | No, I thought it strange, but... ho hum | 09:35.46 |
| kens | Hmm, odd, we do support long loca formats in TT writing, I bet we don't set it properly though, that should be easy to fix. | 09:35.52 |
| chrisl | kens: so, if I change the GS PS to read 16 bit loca entries, I get numloca = 89, but still no glyphs showing in the output :-( | 09:48.17 |
| kens | Hm.... | 09:48.50 |
| | maybe there are more problems, let me fix the one I see fiorst | 09:49.00 |
| | Hmm, the code is 'fixing' the HEAD table to read 16-bit loca, so something is clearly wrong. | 09:50.41 |
| chrisl | kens: I wonder if the Type42 interpreting code isn't handling this correctly? | 09:54.15 |
| kens | Its not obvious | 09:54.33 |
| | The code to write the TT font specificaslly sets teh locaFormat to 0 | 09:54.51 |
| | But in the output file it seems to be 256, which suggests some other deeper format problem | 09:55.10 |
| | I'm just reading my way through the head table now | 09:55.30 |
| chrisl | Oh, yes, that's not good :-( | 09:55.35 |
| kens | THe TT writing code sets a byte at a specific offset in the table, I wonder if the table has some other entry bigger, which would make the value incorrect. | 09:56.48 |
| chrisl | It does sound like something's got an extra byte - or padding, maybe? | 09:57.30 |
| kens | Not sure yet, I'm just disassembling ther data I see in memory and comparing it with the ttfdump output | 09:58.00 |
| | Well, reading the table in memory, ttfdump is quite correct, so I need to step back up a bit and see what's going on with the code. Simply setting a byte offset may always have been wrong.... | 10:02.57 |
| | Want to bet our code only zeros the bottom byte, and the original font had the top byte set ? | 10:03.35 |
| | And indeed that seems to be exactly the problem. | 10:04.44 |
| | Looks like whoever wrote (or patched) the header table didn't realise that the data is defined as big-endian and they wrote it little-endian | 10:05.23 |
| chrisl | It's surprising there haven't been more problems | 10:06.22 |
| kens | Well, this may not be the only problem.... | 10:06.38 |
| | I've just hacked the code here, I'll see what the output looks like | 10:06.49 |
| | drat, we write it multiple times :-( | 10:07.22 |
| | quick code patch and rebuild | 10:07.53 |
| | Ah, that seems to work, I have text now :-) | 10:13.11 |
| | lets try the original file | 10:13.22 |
| | good the original works too, time for a cluster test | 10:14.09 |
| Robin_Watts | kens: chrisl: Just had virgin on the line. | 11:51.41 |
| | They offered me a promotional upgrade to PE both ways for 200 quidish. | 11:52.01 |
| kens | nice | 11:52.07 |
| Robin_Watts | And it let me keep the tier points. | 11:52.26 |
| kens | :-) | 11:52.32 |
| Robin_Watts | So, assuming the extra trip to see the potential big MuPDF customer comes off before the end of feb, I'll get to stay in Silver too. | 11:53.22 |
| kens | congratulations :-( | 11:53.32 |
| Robin_Watts | I assume you two will be offered the same offer when they ring you. | 11:54.01 |
| chrisl | I'm already in PE..... | 11:54.15 |
| kens | Me too | 11:54.20 |
| Robin_Watts | Ah! | 11:54.40 |
| | Where? | 11:54.42 |
| chrisl | I think it's on a plane ;-) | 11:55.09 |
| | Huh, computer and/or website it causing me trouble - can't login to VA right now.... | 11:59.18 |
| Robin_Watts | tor8: ping | 12:01.25 |
| tor8 | Robin_Watts: pong. | 12:02.02 |
| Robin_Watts | http://bugs.ghostscript.com/show_bug.cgi?id=693459 | 12:02.16 |
| kens | FWIW I'm in 33C on the way out | 12:02.29 |
| Robin_Watts | I'm in 34D both ways. | 12:02.40 |
| kens | 32C coming back | 12:02.54 |
| tor8 | Robin_Watts: right. the non-aa clip mask thing you mentioned earlier? | 12:03.00 |
| Robin_Watts | Right. | 12:03.04 |
| | The 'fix' is to change to always not use AA for clipmasks. | 12:03.18 |
| | which (if memory serves) fits with what Acrobat does. | 12:03.30 |
| tor8 | will make vector art that uses clipping with gradients or images look terrible though | 12:04.14 |
| chrisl | And I'm 36C both ways (now that VA website has deigned to talk to me!) | 12:04.19 |
| tor8 | but maybe that's a decent trade off to deal with these stupid draw-an-image-as-many-images-with-clips | 12:04.40 |
| Robin_Watts | (ISTR doing some investigations into Acrobats behaviour on 'thin' clip regions a few years ago, and it's not entirely consistent) | 12:04.41 |
| | tor8: Did you look at my bmpcmp ? | 12:04.48 |
| | Let me rerun the bmpcmp restricted to ppmraw.200.0 | 12:05.01 |
| tor8 | Robin_Watts: no, sorry. didn't have time. | 12:05.04 |
| | just saw the message before I ran out | 12:05.11 |
| | egads it's cold! | 12:05.25 |
| | temperature dropped 15 degrees over the weekend... | 12:05.33 |
| Robin_Watts | tor8: yeah, here too. | 12:05.36 |
| tor8 | -10 last night | 12:05.43 |
| Robin_Watts | Ah, we're not quite that bad yet. | 12:05.53 |
| | The differences I saw in the bmpcmp were really not that bad. | 12:06.06 |
| | I can't think of any other way to solve this really. | 12:06.19 |
| tor8 | Robin_Watts: rerun and ping me when you want me to look | 12:06.21 |
| Robin_Watts | will do. | 12:06.26 |
| tor8 | Robin_Watts: not without doing real multisampling, no... | 12:07.25 |
| Robin_Watts | tor8: Bugger. It looks to me like that if anything throws within the fz_always block, we'll reenter the fz_always block. | 13:19.10 |
| | So our choices would appear to be: | 13:20.35 |
| | 1) Never throw within the fz_always block. | 13:20.45 |
| | 2) Arrange so that anything thrown in the fz_always block goes immediately to the fz_catch block. | 13:21.11 |
| | 3) Arrange so that anything thrown in the fz_always block goes immediately to the next level of exception handling up. | 13:21.55 |
| | 3 sounds wrong. | 13:22.04 |
| | 1 sounds like it's probably something we should aim for, but I'd feel safer with 2 in place. What do you think? | 13:22.34 |
| tor8 | 4) throw a big ASSERT("STOOOPID STOOOPID PROGRAMMER!") and abort | 13:35.17 |
| | or well, at least fz_warn and (3) for safety | 13:35.43 |
| | rather leak than get double frees from (2) | 13:35.48 |
| Robin_Watts | Urm... | 13:36.02 |
| | 2) would not give double frees. | 13:36.10 |
| | 2 might leak. | 13:36.21 |
| tor8 | right. stupid me :) still monday morning. | 13:36.41 |
| | *grabs another red bull* | 13:37.02 |
| Robin_Watts | I have an implementation for 2, I think. | 13:37.08 |
| | (into which we can easily insert an fz_warn if we want) | 13:37.38 |
| | tor8: Opinions on the following 2 please: | 14:17.24 |
| | http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=a5ec4db0d3eb0225453003e61e5f3bbbd612cecf | 14:17.26 |
| | http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=00cf2d6a2a786922ab6f88624b8866288686e59d | 14:17.35 |
| LuSP | Could anybody help with building mupdf on Windows, please? Here is my MVS Error List: http://pastebin.com/hWatCMAt ... and here is whole MVS output: http://pastebin.com/KcLz4JEg | 14:58.45 |
| Robin_Watts | LuSP: Are you building from git? | 14:59.44 |
| LuSP | Yes | 14:59.50 |
| Robin_Watts | git submodule update --init | 15:00.00 |
| | tor8: Actually, ignore the second of those. | 15:05.15 |
| LuSP | Sorry. I am stacked. I cloned git repository via GUI... | 15:05.31 |
| Robin_Watts | Morning mvrhel_laptop. At the show? | 15:07.01 |
| | LuSP: Right, but we now use git submodules too. | 15:08.18 |
| | All the thirdparty libs are supplied as git submodules. | 15:08.32 |
| | So either you need to invoke the command I just gave you, or you need to find a way to do that under whatever GUI you are choosing to work with. | 15:09.17 |
| kens | Oh no, unpaid technical support mode again, my father has broken his PC :-( | 15:18.04 |
| Robin_Watts | hehe. | 15:19.10 |
| | Is your dad like mine? Just competent enough to get himself into trouble? | 15:19.35 |
| henrys_mac | kens:pdftk would seem to the tool for 692541. Feel free to recommend it not sure if you were worried about suggesting a non artifex product. | 15:20.03 |
| Robin_Watts | Morning henrys_mac. | 15:20.32 |
| henrys_mac | hi Robin_Watts , I should get it out later today ;-) | 15:21.08 |
| Robin_Watts | I don't know what you're trying to say. :) | 15:21.49 |
| henrys_mac | you usually ask about the agenda, which I'm always a bit late on. | 15:22.18 |
| kens | Robin_Watts : yes, and osmehow he breaks stuff he shouldn't beable to, he';s broken a RAID mirror this time :-( | 15:22.40 |
| | henrys_mac : I did mention pdftk in the bug thread I think | 15:22.53 |
| | I'm intending to look at the 'missing text' just to make sure that an up to date version (user is using 8.71) doens't suffer from the problem. | 15:23.22 |
| henrys_mac | I don't think anyone over the age of 65 should use raid. | 15:23.38 |
| kens | henrys_mac : its a simple hardware mirror, because he manages to break HDDs regularly | 15:24.04 |
| | How he's managed to break the mirror I have no idea. | 15:24.17 |
| | But at least I can probably rescue one of the disks | 15:24.29 |
| henrys_mac | it's all that kicking of the case after the blue screen | 15:24.52 |
| kens | funnily enough he did get a BSOD | 15:25.05 |
| | and then tried a few things before calling me :-( | 15:25.20 |
| | It looks like when he uninstalled ACG he didn't get all of it, and a driver has surfaced and interacted badly with something | 15:26.00 |
| | AVG* | 15:26.05 |
| Robin_Watts | tor8: ping me when you get back into range. | 15:27.42 |
| henrys_mac | I fix my neighbors computer and spend most of my time unraveling what she did to fix it. | 15:27.48 |
| kens | Yes, that's pretty much how it works for me too :-( | 15:28.15 |
| | Having got him to the safe mode screen, he tells me 'the mouse doens't work'. | 15:28.30 |
| | This turns out to be because he unplugged it when removing 'unnecessary' peripherals | 15:28.49 |
| LuSP | It's me, again. I have made git submodule update --init command but it still gives errors. MVS Error List: http://pastebin.com/Qjp2uE3j MVS output: http://pastebin.com/29n5ackG | 15:52.05 |
| Robin_Watts | LuSP: Right. So that's because you're trying to build the -v8 targets. | 15:53.56 |
| | If you build mupdf-v8 or mudraw-v8 then you'll need to have an additional thirdparty dependency installed. | 15:54.26 |
| | If you don't care about javascript, just don't build those targets. | 15:54.37 |
| | If you do care, then you can download... http://ghostscript.com/~robin/v8-3.9.win32.zip and unpack that into thirdparty. | 15:55.38 |
| LuSP | It creates 11 MB (!) mupdf.exe which works slower then default mupdf v1.1 and still hasn't continuous scrolling feature (( | 16:05.01 |
| | So currently there isn't good pdf viewer for Windows ( | 16:07.12 |
| Robin_Watts | LuSP: Are you building release or debug configuration? | 16:09.12 |
| | The current MuPDF viewer does not have a continuous scrolling mode, nor will it ever have. We have a new viewer in progress that we hope to release in February. That may have continuous scrolling - you need to ask tor8. | 16:10.41 |
| tor8 | LuSP: the new viewer that's in progress has continuous scrolling, but still only Gtk+ so no windows | 16:11.22 |
| | LuSP: you can try sumatrapdf, it uses mupdf as the rendering engine | 16:11.37 |
| Robin_Watts | In release mode, mupdf.exe is 8Meg for me. 10.6 meg for mupdf-v8.exe | 16:11.48 |
| tor8 | Robin_Watts: I'm around. first patch (jpeg throwing) looks fine | 16:11.49 |
| Robin_Watts | tor8: Fab. | 16:11.55 |
| | Ignore the second patch. | 16:12.01 |
| | and instead look at: | 16:12.07 |
| | http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=d9b21f436f22b6a6a744db48c69a59f8d73fca31 | 16:12.24 |
| tor8 | right, that one makes sense | 16:14.25 |
| | ... I hope. | 16:15.02 |
| Robin_Watts | Did you see zenikos answer about the code we didn't understand? | 16:15.08 |
| tor8 | considering I made a pretty bad thinking error earlier today about fz_always :) | 16:15.19 |
| Robin_Watts | his fix for use after freeing ? | 16:15.20 |
| | In pdf_load_obj_stm, when reading an object. We throw the existing one away and keep the new one. He keeps the old one and throws the new one away. | 16:16.24 |
| tor8 | Robin_Watts: yeah, something complicated... | 16:16.44 |
| Robin_Watts | He explains why in comment 13 of 693290. | 16:16.51 |
| | basically, he has a file which starts to read in, and populates some objects. | 16:17.12 |
| | Then it gets an error, so we drop back to the repair code. | 16:17.23 |
| | and as we repair, we read an object in again. | 16:17.37 |
| | If we throw away the old one, and keep the new one, someone still has a reference to the old one. and hence boom. | 16:17.55 |
| tor8 | Robin_Watts: oh yeah. he has a "borrowed" reference in his example, from fz_dict_gets | 16:18.51 |
| | ugh. objc's autorelease stuff is really quite neat. | 16:19.08 |
| Robin_Watts | tor8: You'll be approving of ^{ } next. | 16:19.37 |
| tor8 | Robin_Watts: I've used ^{} quite a lot you know! ios viewer is full of it. | 16:20.01 |
| | theory is nice, but damn all the voodoo reference counting and stuffing of objc magic objects into the "closures" is just evily | 16:20.36 |
| | s/y$// | 16:20.42 |
| LuSP | Currently, IMHO, the fastest pdf viewer for Windows is "FoxitReader v2.3" (but it has bad search implementation and some other issues) and "Evince" for Linux (poppler engine). With Evince there isn't any lags completely. | 16:21.16 |
| tor8 | Robin_Watts: http://git.ghostscript.com/?p=user/tor/mupdf.git;a=blob;f=ios/main.m;h=0f80505d240ea4f4c4a35e2eb7d159fa7c93758c;hb=HEAD#l645 | 16:21.24 |
| Robin_Watts | yeah. shudder. | 16:22.03 |
| | Given people can't yet write C compilers that don't screw the pooch on a regular basis, why do apple want to make them more and more complex? | 16:22.29 |
| tor8 | LuSP: too bad foxit renders text like sh*t... | 16:22.29 |
| Robin_Watts | LuSP: It sounds to me like you're building MuPDF in debug mode. That will be both bigger and larger. | 16:23.12 |
| tor8 | Robin_Watts: I think you can look at iTunes and Xcode for more examples of apple's gone completely bananas on complexity... | 16:23.16 |
| | they don't fix bugs, they add features instead. s/feature/bug/. | 16:23.34 |
| | http://www.textfromxcode.com/ | 16:24.07 |
| Robin_Watts | LuSP: And the structure of MuPDF is that it's a set of core libraries with the viewers being thin layers on top; you can reimplement those thin layers to offer viewers with any features you want. | 16:24.12 |
| | So feel free! | 16:24.25 |
| LuSP | Probably my hardware is too old to render simple pdfs. )) | 16:26.40 |
| Robin_Watts | marcosw: Has anything changed with mupdf and the cluster recently? | 16:27.48 |
| | I keep sending jobs that ought to have rendering differences, and getting no differences reported. | 16:28.06 |
| mvrhel_laptop | Morning Robin_Watts. No I am still here at home | 16:29.18 |
| | I am heading down there tonight | 16:29.29 |
| Robin_Watts | righto. So you've got everything you need from us for the demo, right? | 16:30.03 |
| LuSP | Sorry. I don't know is it Debug mode or not. I just made "git clone git://git.ghostscript.com/mupdf.git" ) | 16:32.21 |
| Robin_Watts | You did the build in MSVC, right? | 16:32.37 |
| | in Visual Studio, I mean. | 16:33.17 |
| | At the top of that, you get to choose between 'Debug', 'Release', etc. | 16:33.33 |
| marcosw_ | Robin_Watts: we now send passwords to the command line for the two files in tests_private/pdf/sumatra that are password protected | 16:33.35 |
| Robin_Watts | marcosw_: Right, but other than that? | 16:33.49 |
| | I added some code on friday to render clip masks without antialiasing, and when I tested that I got (as expected) lots of diffs - all trivial pixel rounding changes. | 16:34.25 |
| marcosw_ | I also made a change that fixed http://bugs.ghostscript.com/show_bug.cgi?id=693469 but that shouldn't affect mupdf. | 16:34.42 |
| Robin_Watts | I never committed that, and when I retest it now, I get '0 differences found'. | 16:34.48 |
| | Ok. The test I have running now, for example, should differ from the trunk in more than 1000 files. | 16:35.38 |
| marcosw_ | I can trivially undo that change, let me know if you don't see the expected changes and I'll do that and you can re-run your test. | 16:36.59 |
| | Robin_Watts: no changes in the bitmaps. i'll undo the change and force another cluster run | 16:41.40 |
| Robin_Watts | marcosw_: Thanks | 16:41.52 |
| LuSP | Release mode is definitely better. It's great! But... continuous mode. I saw a patch at github for such feature. Probably I should try it... | 16:42.52 |
| mvrhel_laptop | Robin_Watts: I think I am all set demo wise | 16:43.02 |
| | everything seems to be working fine | 16:43.07 |
| marcosw_ | Robin_Watts: never mind, as I was modifying clustermaster.pl I realized that the code change is in a block that is if(!$userMupdfRegression), so definitely isn't the cause of the problem. | 16:43.11 |
| mvrhel_laptop | Robin_Watts: My wife did ask me to see if you could bring a box of decaf tea when you come . She has been enjoying the regular tea in the morning but can't drink it past noon | 16:44.09 |
| marcosw_ | Robin_Watts: can you give me one file that should have had pixel differences and I'll follow the md5sum through the rabbit hole that is the cluster code. | 16:44.16 |
| Robin_Watts | tests/Ghent_V3.0/010_CMYK_OP_x3.pdf.pgmraw.200.0 | 16:44.44 |
| | mvrhel_laptop: I'll try and pick some up tonight. Any particular variety of decaff? | 16:45.11 |
| marcosw_ | okay, I'm going to take the cluster down while I do this, so that the logs are replaced while I'm looking at them :-) | 16:45.15 |
| mvrhel_laptop | She liked the Twinnings Robin_Watts | 16:45.26 |
| Robin_Watts | ok. | 16:45.30 |
| mvrhel_laptop | thanks | 16:45.33 |
| marcosw_ | Robin_Watts: that file doesn't show a difference in the md5sum. It's 5838b133b2ce62500549d071474f3edc when run via clustepush of your code and in the mupdf-current.tab file (which hasn't been modified since Friday). | 16:52.34 |
| | Can you try the command line on your copy of mudraw: mudraw -r200 -d -o ./temp/tests__Ghent_V3.0__010_CMYK_OP_x3.pdf.pgmraw.200.0.%04d ./tests/Ghent_V3.0/010_CMYK_OP_x3.pdf | 16:53.10 |
| | Robin_Watts: oops, I just realized I have to run to uni. Will be back online in an hour. I'll reenable the cluster; send me email or wait until I'm back on IRC. | 16:54.28 |
| Robin_Watts | marcosw: (For the logs) Sorry, being stupid! | 16:59.52 |
| ray_laptop | (the MT rendering thread memory conflict) | 17:04.52 |
| | Robin_Watts: can I use you as a sounding board for what I'm seeing with bug 693361 ??? | 17:05.01 |
| Robin_Watts | ray_laptop: Sure. Give me a couple of minutes to clear my desk? | 17:05.21 |
| ray_laptop | OK. | 17:05.29 |
| | hmm... gdb "info thread" only shows 4 threads, {1, 348, 350, 352}, but it says "using 4 rendering threads" -- wonder why I don't see 5 threads (main + 4 rendering threads) | 17:17.46 |
| Robin_Watts | ray_laptop: OK, go for it. | 17:18.00 |
| ray_laptop | Robin_Watts: OK, thanks. The first issue is that I may not be able to trust gdb, but I'll ignore that for now. | 17:18.55 |
| Robin_Watts | ray_laptop: Can you get your thread starter code to print the thread id as it starts? | 17:19.28 |
| | printf rules OK :) | 17:19.50 |
| ray_laptop | Robin_Watts: my last comment in the bug is bogus -- when I create the chunk manager for each thread, I use the 'thread_safe_memory' as the base (to provide more chunks), which is also the base for the parser's chunk allocator | 17:20.14 |
| | and the underlying gsmalloc allocator (heap allocator) _is_ thread safe, | 17:20.50 |
| Robin_Watts | Right, so can you restate what you believe the problem to be now ? | 17:21.10 |
| ray_laptop | well, that's the problem. gdb reports the messages for me: | 17:23.09 |
| | [New Thread 0x7ffff5bfe910 (LWP 28016)] | 17:23.11 |
| | [Thread 0x7ffff5bfe910 (LWP 28016) exited] | 17:23.12 |
| | so I think there are only 3 rendering threads because when a thread exits and releases the semaphore for the main thread (1) that we haven't yet started the next thread. | 17:24.46 |
| Robin_Watts | ray_laptop: And that's causing a crash ? | 17:25.10 |
| ray_laptop | no. | 17:25.45 |
| Robin_Watts | A thread is starting up, and then shutting down, and then on the shutdown you have code that says "If I'm the last thread to be shutdown, then tell the main thread to continue"? | 17:25.59 |
| LuSP | So, currently I have "semi-continuous" mode in mupdf with the patch by mariusmuja and its good. Thanks everybody who involved in mupdf development. | 17:27.37 |
| ray_laptop | so what I see (this time) is: | 17:27.42 |
| | % Using 4 rendering threads | 17:27.43 |
| | [Thread 0x7ffff5bfe910 (LWP 28321) exited] | 17:27.45 |
| | [New Thread 0x7ffff5bfe910 (LWP 28330)] | 17:27.46 |
| | [Thread 0x7ffff5bfe910 (LWP 28330) exited] | 17:27.48 |
| LuSP | I hope that you or someone else will create well working pdf viewer based on mupdf. SumatraPDF is too slow (As compared with separate mupdf). | 17:27.49 |
| ray_laptop | [Thread 0x7ffff4c5b910 (LWP 28323) exited] | 17:27.49 |
| | [New Thread 0x7ffff4c5b910 (LWP 28331)] | 17:27.52 |
| | [Switching to Thread 0x7ffff4089910 (LWP 28327)] | 17:27.52 |
| | and I hit my breakpoint when a chunk allocator has in_use != 0 | 17:27.54 |
| | the info thread says: | 17:27.55 |
| | 352 Thread 0x7ffff4c5b910 (LWP 28331) 0x0000000000845cc6 in mem_true24_copy_mono (dev=0x24c5b38, | 17:27.57 |
| | base=0x2ce72b8 "\254\254", sourcex=22, sraster=608, id=0, x=150, y=162, w=4800, h=0, zero=18446744073709551615, | 17:27.58 |
| | one=0) at ../gs/base/gdevm24.c:423 | 17:28.00 |
| | 350 Thread 0x7ffff34b7910 (LWP 28329) __lll_lock_wait () | 17:28.01 |
| | at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136 | 17:28.03 |
| | * 348 Thread 0x7ffff4089910 (LWP 28327) chunk_obj_alloc (mem=0x130b320, size=72, type=0xa73800, cname=0x8f7ff0 "lcms") | 17:28.04 |
| | at ../gs/base/gsmchunk.c:439 | 17:28.06 |
| | 1 Thread 0x7ffff7fd7730 (LWP 27964) 0x0000000000833271 in check_device_separable (dev=0x2d08218) | 17:28.07 |
| | at ../gs/base/gdevdflt.c:457 | 17:28.09 |
| | sorry to all for the flood. Robin_Watts should we work in a private chat ? | 17:28.19 |
| kens | OK I'm off, goodnight everyone | 17:28.27 |
| Robin_Watts | ray_laptop: Could do. | 17:30.45 |
| ray_laptop | Robin_Watts: the thread in the lock_wait state is waiting for the monitor lock from inside gsicc_get_link_profile (so it is _not_ doing any allocation) | 17:30.53 |
| Robin_Watts | ray_laptop: OK. Can we rewind a bit please? | 17:31.10 |
| | Right, so can you restate what you believe the problem to be now ? | 17:31.22 |
| ray_laptop | I know henrys_mac likes discussions here so others can follow along, so we'll stay here until others need the channel | 17:31.37 |
| Robin_Watts | or do you no longer have an idea? | 17:31.41 |
| ray_laptop | Robin_Watts: I don't know why we have a problem, BUT I do see the breakpoints where the 'in_use' is != 0 for a chunk memory which says another thread is using it. | 17:32.34 |
| Robin_Watts | OK, so somehow we are getting 2 (or more) threads accessing the same chunked memory manager. | 17:33.15 |
| | What chunked memory managers are there in the system? 1 per rendering thread? or 1 per rendering thread + main thread? | 17:34.00 |
| ray_laptop | specifically in this case, the thread that detected it is 348 and the thread that had set the in_use was thread 1 | 17:34.46 |
| | Robin_Watts: the latter. 1 per rendering thread + the main (parser) chunk allocator | 17:35.10 |
| Robin_Watts | AIUI, the rendering threads will have chunk memory managers that are based upon the main threads chunk memory manager. | 17:36.18 |
| | So they will mostly run by themselves, but every now and then they will call down to the main threads one to get memory for a new chunk. | 17:36.49 |
| ray_laptop | ahh. the main thread is the one that is supposed to be using cmem 130b320, but thread 348 is trying to use it !!! | 17:36.54 |
| Robin_Watts | What's the call stack at that point? | 17:37.26 |
| ray_laptop | it's in lcms. So the problem is the ICC code using the wrong allocator ! | 17:38.02 |
| Robin_Watts | AIUI, we can permissibly see a rendering thread calling it's chunked memory manager, that running out of memory, and that calling down to the main threads chunk mananger. | 17:38.05 |
| | And the call stack is NOT showing that lcms is calling the threads chunk allocator, and that's calling through to the base one? | 17:38.59 |
| ray_laptop | thread 348 _should only be using chunk allocator at 0x1aa6d00 | 17:39.04 |
| Robin_Watts | lcms is actually calling the base one directly? | 17:39.19 |
| ray_laptop | Robin_Watts: right. OK now I understand it, and can figure out why it (gsicc) is not using the correct allocator | 17:39.37 |
| Robin_Watts | Do you understand what I'm asking? | 17:39.50 |
| ray_laptop | thanks for being a sounding board. | 17:39.51 |
| | Robin_Watts: I have to trace back through thread 348 to see where it is pulling the main threads allocator from (which structure) | 17:40.18 |
| Robin_Watts | thread 348 might call 0x1aa6d00 to get some memory, and that might call 130b320 quite permissibly. | 17:40.39 |
| ray_laptop | but now I know how to proceed. | 17:40.40 |
| Robin_Watts | but 348 should never call 130b320 directly. | 17:40.51 |
| | If it's a direct call, then yes, I agree, it's wrong. | 17:41.19 |
| ray_laptop | the base memory for 0x1aa6d00 is 0x130b010 (not 0x130b320) | 17:41.42 |
| Robin_Watts | Fair enough. | 17:42.06 |
| | So it really most likely is an icc profile being created in the main thread, and somehow making its way to the render thread. | 17:42.42 |
| | I thought mvrhel_laptop was at pains to ensure that didn't happen. | 17:42.54 |
| ray_laptop | Oh. the 'stable_memory' is not correct. It is 0x130b320 _not_ 0x1aa6d00 which it should be. I think the lcms is probably using the stable_memory or the non_gc_memory (instead of itself). Probably I set that up wrong. | 17:44.28 |
| | no, I think it is my mistake in setting up the chunk allocators | 17:44.56 |
| Robin_Watts | ray_laptop: Yeah, it's quite possible that lcms uses the stable memory, cos it doesn't want to be dealing with gc etc. | 17:45.12 |
| ray_laptop | Robin_Watts: I think it uses non_gc_memory, but both are set wrong. I'll go off and fix it now. Thanks, again. | 17:46.12 |
| Robin_Watts | no worries. Not sure I did much :) | 17:46.39 |
| ray_laptop | Like I said, I needed a sounding board :-) | 17:47.08 |
| | it always helps me to have to explain what I am seeing to another | 17:47.33 |
| | then the nonsense I am spouting becomes obvious | 17:48.12 |
| henrys_mac | ray_laptop:http://madkingsmusings.blogspot.com/2010/10/tell-it-to-bear.html | 17:48.13 |
| ray_laptop | henrys_mac: yeah, but the bear doesn't request explanations. if my very first programming class (as we called it back in 1968), the favorite TA I went to for help would mostly just sit there and let me explain, and when I finished explaining a part all he would say was "yes, and then?" | 17:51.07 |
| henrys_mac | Seems like valgrind --tool=helgrind ./pcl6 ⦠would be worth a try, to see if there are other issues. | 17:54.34 |
| mvrhel_laptop | ray_laptop: so there is not something going on that I need to worry about? | 17:55.03 |
| ray_laptop | mvrhel_laptop: nope. It was my mistake. A one line change seems to have fixed it | 18:04.11 |
| Robin_Watts | henrys_mac: The mail to support about mupdf... | 18:04.41 |
| ray_laptop | oops. spoke to soon. still a problem (just made it through many more pages) | 18:04.44 |
| Robin_Watts | It seems he wants the android app to view n 1-page pdfs as if it was 1 n-page pdf. | 18:05.12 |
| | This of course is not how the android app is structured. | 18:05.26 |
| | I think we can probably make it work, but it'll require some digging around in the code. | 18:05.48 |
| | I was going to reply to him, and copy it to paulgardiner on the grounds that paulgardiner was the most recent person in this area. | 18:06.23 |
| henrys_mac | I'd like to entertain the request, it's a new market for us and we should be proactive. | 18:06.41 |
| Robin_Watts | Right, and you're happy for me to drag paul into it too? | 18:07.03 |
| henrys_mac | I am | 18:07.07 |
| Robin_Watts | Fab. | 18:07.11 |
| henrys_mac | It does sound like a terrible mess though | 18:07.55 |
| Robin_Watts | eek. I may need to be in California by Wednesday morning... | 18:14.54 |
| henrys_mac | why's that? | 18:15.26 |
| | I thought you got in thursday? | 18:15.58 |
| Robin_Watts | Miles just called. Apparently we have been offered a meeting with the big boss at the big company that we're trying to sell MuPDF to. | 18:17.07 |
| henrys_mac | wow great news | 18:17.52 |
| ray_laptop | Yeah. That is great! | 18:18.31 |
| | Robin_Watts: at least it is in CA, right ? | 18:19.02 |
| marcosw_ | Robin_Watts: I'm back online. Did you do any analysis on the mupdf clusterpush issue? | 18:19.36 |
| Robin_Watts | marcosw_: Yes. I discovered that I am a fool. | 18:19.54 |
| marcosw_ | Robin_Watts: welcome to the club :-) | 18:20.04 |
| | I'm betting git ate your changes ? | 18:20.22 |
| Robin_Watts | marcosw_: No. I 'tidied' the code a bit. | 18:24.45 |
| ray_laptop | mvrhel_laptop: OK, I take it back. I may need to chat with you. The cmsReverseToneCurveEx has (InCurve->InterpParams->ContextID set to the main thread's memory NOT the ContextID I see further up the call stack. It is correct in DefaultICCintents, but that calls _cmsReadOutputLUT which calls BuildRGBOutputMatrixShaper which calls cmsReverseToneCurve which calls cmsReverseToneCurveEx | 18:24.59 |
| henrys_mac | another day in the land of milk shakes ;-) | 18:27.46 |
| Robin_Watts | 2 days. I'd have to fly tomorrow. | 18:28.01 |
| | :) | 18:28.08 |
| ray_laptop | mvrhel_laptop: Since lcms keeps it's ContextID tucked away (potentially with different memory allocators) the only way I can see is to have lcms ALWAYS use the thread_safe_memory. Hopefully that won't kill our performance since we don't allocate too frequently | 18:28.21 |
| | mvrhel_laptop: it seems to be using some elements allocated by the main thread in the other threads. We think of them as "read only", but each is carrying it's own ContextID | 18:29.56 |
| Robin_Watts | ok, flights changed. | 18:35.56 |
| ray_laptop | Robin_Watts: bon voyage! | 18:38.23 |
| henrys_mac | Robin_Watts:I bet that cost miles a few $'s | 18:43.37 |
| Robin_Watts | 100UKP to change flights. | 19:48.22 |
| henrys_mac | Robin_Watts:oh not too bad | 19:49.36 |
| Robin_Watts | tor8: Thoughts.... http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=a8946fcf2caf4d9dfb3ddaf329c795cb750feabe | 20:09.36 |
| henrys_mac | mvrhel_laptop:on your nexus 10 are you asked if you want to open with an alternate PDF interpreter? - it does that on my 7. | 20:24.17 |
| mvrhel_laptop | henrys_mac: yes. when I open from the file explorer it lists two options mupdf and the native viewer | 20:25.01 |
| henrys_mac | have you compared our performance with the native viewer, for example how fast can you scroll pages in say a manual or something? | 20:25.51 |
| | it was okay on the 7 but you're moving around more pixels. | 20:26.23 |
| mvrhel_laptop | henrys_mac: at first we were slower. Now we are about the same | 20:26.39 |
| | at least comparing to zoom time | 20:26.50 |
| | theirs is slower starting up though | 20:27.07 |
| | going to grab some lunch and then get ready to head to the airport | 20:28.45 |
| henrys_mac | okay good luck! | 20:28.56 |
| mvrhel_laptop | thanks | 20:30.21 |
| tor8 | Robin_Watts: idea seems good, will review code tomorrow | 21:58.27 |
| Robin_Watts | tor8: Can you carry on talking to Paul Hudson on the support email please? | 22:02.34 |
| | Obviously I'm going to be out of contact... | 22:02.41 |
| tor8 | Robin_Watts: I'm leaving on thursday morning too... or are you going elsewhere? | 22:08.27 |
| | oh, right, nvm. | 22:09.02 |
| | I remember his earlier questions; did we ever mention that he could write his own fz_document type to wrap multiple documents in sequence and present their pages as one document? | 22:10.49 |
| Robin_Watts | tor8: I'm flying in about 12 hours time. | 22:12.10 |
| | I don't remember ever hearing about his desire to do multiple files at once. | 22:12.46 |
| | and I get the feeling that he's not keen on C coding. | 22:12.55 |
| | Though maybe I'm doing him a disservice. | 22:13.10 |
| tor8 | well, I'll butt in and mention it. if he's a supported customer I think we could easily whip one up in an afternoon or two. | 22:13.39 |
| | exposing it to android will be trickier | 22:13.54 |
| Robin_Watts | tor8: he's specifically asking about links now. | 22:14.01 |
| | I might have a go at writing some jni stuff to expose the C API as java on the plane. | 22:14.39 |
| tor8 | we have links supported in the android app -- both highlighting and following. | 22:15.23 |
| Robin_Watts | right, I thought so, but I've not touched any of that. | 22:15.49 |
| tor8 | and I've hardly touched it other than changing the color and enabling the button | 22:16.14 |
| Robin_Watts | ok, so if you can mention it to paulgardiner tomorrow, that'd be gret. | 22:16.37 |
| | great. | 22:16.40 |
| tor8 | should I offer to make a fz_open_document_sequence fz_document api? | 22:16.42 |
| Robin_Watts | up to henrys I guess. | 22:17.03 |
| | Gah. Virgin won't let me check in online, so I have no doubt got a crap seat :( | 22:18.01 |
| henrys_mac | tor8:butt away especially if paul isn't going to be around until tomorrow. | 22:37.48 |
| ray_work | in case he wants to just merge the pdf's on the server, I mentioned ghostscript as a way to do it. | 22:38.28 |
| | but if each user has different content, that could be difficult. It would probably be OK if they just have a fairly small number of 'mash ups' to deploy for classes of users | 22:39.58 |
| | I don't recall -- has anyone done a http stream for working from remote content for mupdf ? (I had started one for gs a while back, but mupdf seems better for that) | 22:41.24 |
| | off to pick up the kids from school ... bbiaw | 22:42.24 |
| Robin_Watts | ray_work: He is providing all the content himself (it's a magazine publisher). It's not a 'user' based system. | 23:07.17 |
| | Forward 1 day (to 2012/12/04)>>> | |
IRC Logs