| | <<<Back 1 day (to 2012/08/08) | 2012/08/09 |
| Robin_Watts | kens: I just got a mail, supposedly from you to robin@ghostscript.com with a zipfile attached called price09-Aug-2012.zip | 09:21.35 |
| kens | I've had several of those, apparently from you, ray, etc | 09:21.53 |
| Robin_Watts | It looks like a fairly typical spam, but it's odd that it came from your artifex address to my gs address. | 09:21.57 |
| kens | Its spoofed, its not from my address | 09:22.08 |
| | Some spammer has obviously got hold of our addresses and is sending mail apparently form one domain address to another. Its a slightly more sophisticated attempt than usual | 09:23.02 |
| Robin_Watts | Yes. | 09:23.16 |
| kens | If you look in your gmail spam folder you may find others | 09:23.34 |
| Robin_Watts | Given that spam is so highly automated, it's just surprising that this one is quite good. | 09:23.50 |
| | and looking in the headers I can't see anything obviously wrong with it. | 09:24.08 |
| kens | Yes, its a new one to me, quite a sensible appraoch (from teh spammers POV) | 09:24.11 |
| Robin_Watts | (I have had the same email address since 1995,so I get a lot of spam) | 09:24.23 |
| kens | Robin_Watts : the forgeries in the headers can be undetectable | 09:24.27 |
| | OK I have a memory problem, anyone got any ideas how to track this down ? The memory I have allocated is being overwritten by gs_heap_alloc_bytes fomr the Flate filter. | 09:56.10 |
| | ANd no, Memento offers no clues either | 09:56.19 |
| | It 'seems'like hte memory must be marked as free for this to happen, but I don't see where this occurs, if it does | 09:56.53 |
| chrisl | Ray sent some notes to tech about tracking down memory issues a while back | 09:57.32 |
| kens | Hmm, actually, it looks like this isn't where my memory corruption occurs. | 09:57.50 |
| Robin_Watts | kens: So the memory in use is being blatted on by someone allocating some more memory ? | 09:57.57 |
| kens | Robin_Watts : I thought that was what was happening, but it looks like that's not quite right | 09:58.19 |
| | Its setting the 'prev' member of the memory I have allocated to 'bp' which is 0x00 | 09:58.42 |
| | Hmm, even that isn't tru, now I'm confused.... | 09:59.11 |
| | I don't really wnt to use the debugging flags if I can avoid it, this is an 8 page PostScript file... | 10:00.24 |
| chrisl | -Z@ shouldn't slow things down too much | 10:00.43 |
| kens | chrisl but how much data will it dump ? | 10:00.55 |
| chrisl | None, it overwrites freed memory | 10:01.17 |
| Robin_Watts | kens: If you send me a patch and the file, I'll see if I can reproduce it and bash on it for a bit. | 10:01.19 |
| kens | Robin_Watts : Its a *huge* diff, its all my linearisation stuff. | 10:01.42 |
| | I'll hit it with a hammer some more | 10:01.53 |
| Robin_Watts | So, git commit it, then git format-patch, and I'll git am. | 10:02.01 |
| | The size of the patch makes no odds to me. | 10:02.09 |
| kens | Hmm, I may take you up on it if I don't get anywhere in teh next few minutes | 10:02.21 |
| chrisl | Or push it to a private repos on casper | 10:02.22 |
| Robin_Watts | chrisl: Yeah, or that. | 10:02.32 |
| | ARguably you should be doing that anyway to avoid data loss. | 10:02.49 |
| kens | Robin_Watts : not worried about that right now | 10:03.01 |
| Robin_Watts | That sounds like helens grandmother, who, when asked if she had made a will, said "Closer to the time dear, closer to the time..." | 10:04.00 |
| kens | Robin_Watts : if it worked at all I might be more worried.... | 10:04.17 |
| chrisl | Robin_Watts: that reads like "it's all going to Battersea Dogs' Home, anyway"! ;-) | 10:05.04 |
| kens | OK the memory was stomped on by a gc_objects_set_reolc called from gs_gc_reclaim | 10:06.47 |
| | But the memory was allocated using alloc_bytes_immovable | 10:07.06 |
| chrisl | And it's definitely not being freed by some other means, first? | 10:07.44 |
| kens | chrisl as far as I can tell, no. | 10:07.53 |
| | The reclaim is stomping on the 'fstype' in the memory header | 10:08.06 |
| | Can someone remind me how to set up a private reporistory ? I can't seem to find it in the irc logs :( | 10:10.02 |
| chrisl | kens: just a sec, I'll do the casper end for you..... | 10:11.51 |
| kens | didn't realise it involved Linux twiddling .... | 10:12.10 |
| chrisl | Just one little bit | 10:12.19 |
| Robin_Watts | kens: It involves you running a single git command: git clone blah ghostpdl | 10:13.53 |
| kens | Well that sounds easy :-) | 10:14.12 |
| Robin_Watts | The same command (different blah) that you'd run on windows :) | 10:14.14 |
| | kens: On your windows box, do: git remote -v | 10:14.42 |
| kens | I was planning on using mingw | 10:14.44 |
| | On MingW that gets me 4 results | 10:15.15 |
| Robin_Watts | and that will list a bunch of lines. For me I get: | 10:15.18 |
| | casper robin@ghostscript.com:/home/robin/repos/ghostpdl.git/ (fetch) | 10:15.29 |
| | casper robin@ghostscript.com:/home/robin/repos/ghostpdl.git/ (push) | 10:15.31 |
| | cluster regression@ghostscript.com:/home/regression/cluster/gitbridge/ghostpdl (fetch) | 10:15.33 |
| | cluster regression@ghostscript.com:/home/regression/cluster/gitbridge/ghostpdl (push) | 10:15.35 |
| | origin robin@ghostscript.com:/home/git/ghostpdl.git (fetch) | 10:15.37 |
| | origin robin@ghostscript.com:/home/git/ghostpdl.git (push) | 10:15.39 |
| | brb. | 10:15.40 |
| kens | $ git remote -v | 10:15.50 |
| | cluster regression@ghostscript.com:/home/regression/cluster/gitbridge/ghostpdl (fetch) | 10:15.50 |
| | cluster regression@ghostscript.com:/home/regression/cluster/gitbridge/ghostpdl (push) | 10:15.51 |
| | origin ken@ghostscript.com:/home/git/ghostpdl.git (fetch) | 10:15.51 |
| | origin ken@ghostscript.com:/home/git/ghostpdl.git (push) | 10:15.51 |
| chrisl | kens: okay, your private repos is up on casper | 10:16.50 |
| kens | OK so all I need is the git clone magic runes | 10:17.18 |
| chrisl | You should just need to add the remote repos, but I can't remember incantation off the top of my head... | 10:18.17 |
| kens | is searching irc logs | 10:18.29 |
| Robin_Watts | back | 10:18.39 |
| chrisl | git remote add ken@ghostscript.com:/home/ken/repos/ghostpdl.git | 10:19.04 |
| Robin_Watts | kens: Try: git remote add casper kens@ghostscript.com:/home/kens/repos/ghostpdl.git | 10:19.08 |
| | Once again, trumped by chris' typing speed :) | 10:19.23 |
| chrisl | Oh, but I forgot the repos name, so..... | 10:19.36 |
| Robin_Watts | Then you can 'git push casper master' | 10:19.50 |
| | which will push the 'master' branch to the 'casper' repo | 10:20.08 |
| | You can also do: | 10:20.19 |
| | git push casper hash:master | 10:20.27 |
| kens | wait a moment, going too fast | 10:20.34 |
| chrisl | I went for the name "personal" since our central repos is also on casper..... | 10:20.39 |
| kens | $ git push casper master | 10:21.21 |
| | FATAL ERROR: Disconnected: No supported authentication methods available | 10:21.21 |
| | fatal: The remote end hung up unexpectedly | 10:21.21 |
| Robin_Watts | git remote -v and paste the casper lines ? | 10:21.56 |
| kens | Hmm, is that home/ken or home/kens ? | 10:21.59 |
| Robin_Watts | Are you ken or kens on casper? | 10:22.11 |
| kens | Robin_Watts : I have no idea | 10:22.20 |
| chrisl | it should be ken | 10:22.24 |
| kens | let me look | 10:22.25 |
| chrisl | Your username on casper is ken | 10:22.38 |
| kens | ken | 10:22.41 |
| Robin_Watts | OK, my bad, sorry. | 10:22.52 |
| kens | $ git remote -v | 10:23.15 |
| | casper kens@ghostscript.com:/home/kens/repos/ghostpdl.git (fetch) | 10:23.15 |
| | casper kens@ghostscript.com:/home/kens/repos/ghostpdl.git (push) | 10:23.15 |
| | I will ned to remove these two first I think | 10:23.26 |
| Robin_Watts | git remote rm casper | 10:23.28 |
| | git remote add casper ken@ghostscript.com:/home/ken/repos/ghostpdl.git | 10:23.55 |
| kens | yes done that | 10:24.03 |
| Robin_Watts | git push casper master | 10:24.12 |
| kens | $ git push casper master | 10:24.56 |
| | To ken@ghostscript.com:/home/ken/repos/ghostpdl.git | 10:24.56 |
| | ! [rejected] master -> master (non-fast-forward) | 10:24.56 |
| | error: failed to push some refs to 'ken@ghostscript.com:/home/ken/repos/ghostpdl | 10:24.56 |
| | .git' | 10:24.56 |
| | To prevent you from losing history, non-fast-forward updates were rejected | 10:24.57 |
| | Merge the remote changes (e.g. 'git pull') before pushing again. See the | 10:24.57 |
| | 'Note about fast-forwards' section of 'git push --help' for details. | 10:24.58 |
| | THis is presumably because I have unstaged changes | 10:25.05 |
| | Or something | 10:25.13 |
| Robin_Watts | OK. So your local repo is less up to date than the one chrisl has cloned for you. | 10:25.23 |
| | Do: git push -f casper master | 10:25.30 |
| kens | yeah thought that, updating now | 10:25.34 |
| Robin_Watts | OR you can update. | 10:25.45 |
| kens | may as well be up to date here too | 10:25.59 |
| Robin_Watts | "git push -f casper master" will FORCE your changes onto the remote repo. | 10:26.15 |
| | You will need to use that at various points. | 10:26.23 |
| kens | OK looks like casper is up to date with my stuff | 10:26.38 |
| Robin_Watts | (Like if you put a version of a commit up, then amend it and want to put the new version up) | 10:26.41 |
| kens | But I am unconvinced that it has my changes | 10:26.53 |
| | I cna check I guess | 10:27.03 |
| Robin_Watts | Have you committed your changes? | 10:27.05 |
| kens | No | 10:27.13 |
| Robin_Watts | Well, there you are :) | 10:27.22 |
| | You need to commit your stuff to git before git will have anything to push. | 10:27.53 |
| kens | Well I'm not finished with them yet. I dislike comitting unless I'm happy | 10:27.58 |
| | Oh well.... | 10:28.06 |
| Robin_Watts | git add -u then git commit (and call it "Work in progress" or something). Then git push. | 10:28.27 |
| | Then later you can git add again, and git commit --amend to update the existing commit (and message) | 10:28.47 |
| kens | I'll just carry on wiht git gui if you don't mind ? I can understand that | 10:28.54 |
| Robin_Watts | Sure. There must be a similar "amend" option in that. I was trying to describe the conceptual steps. | 10:29.22 |
| kens | hopes git won't whine about white space at this point, but bets it will | 10:29.24 |
| | Yes, it did, there will be a delay | 10:29.55 |
| chrisl | You can force it to ignore whitespace errors - can't remember how, right now..... | 10:30.33 |
| kens | Well I *beleive* that's done it | 10:31.23 |
| chrisl | kens: http://git.ghostscript.com/?p=user/ken/ghostpdl.git;a=summary | 10:31.59 |
| kens | Yeah, I'd like to look at the source though, I'm a suspicious old Hector | 10:32.35 |
| chrisl | Click the "tree" link next to the comment, you can navigate the sources | 10:33.06 |
| kens | Well, it looks OK | 10:33.49 |
| chrisl | Or the "commitdiff" link to see a color coded diff of the changes | 10:33.53 |
| kens | Robin_Watts : can you grab that ? Or the diff ? | 10:34.27 |
| Robin_Watts | I can grab that. | 10:34.59 |
| | I'll just add your repo to my git as a remote 'ken' | 10:35.14 |
| kens | OK | 10:35.19 |
| | Command line to show the problem is: | 10:35.38 |
| | -sDEVICE=pdfwrite -dLinearise -sOutputFile="/temp/out.pdf" j:/tests/09-11.ps | 10:35.38 |
| | Obviously you'll have to amned the paths. Do you ahev a copy of the 09-11.ps file ? | 10:36.00 |
| | Umm, actually I'd better make sure this still fails with the new code. | 10:36.26 |
| Robin_Watts | if it's one of the standard ones, yes. | 10:36.31 |
| kens | Robin_Watts : its a Quality Logic file | 10:36.42 |
| | OK it does still crash, but in an excitingly different way which invalidates all my breakpoints. Oh well. | 10:37.56 |
| Robin_Watts | Yeah, it SEGVs for me too under memento. | 10:38.20 |
| | so, I'll have a bash. I've got to run helen to the station soon. | 10:38.40 |
| kens | Yes, I found it always crashes, but it cna be difficutl to tell where/when | 10:38.42 |
| | OK thanks Robin_Watts I'll plod on | 10:38.53 |
| | BTW this wasn't originally using 'immovable' memory, it was using GC'ed memory and had a poitner and everythign set up, this was a 'simplification' to try and figure out the problem | 10:39.36 |
| chrisl | You've definitely removed all the ENUM_PTRS etc crap? | 10:40.23 |
| kens | chrisl just to be inconsistent, pdfwrite doesn't do it that way | 10:40.40 |
| | It has its own set of mactos.... | 10:40.47 |
| | macros | 10:40.50 |
| | And I've definitely removed what I added. | 10:41.03 |
| chrisl | What's the memory for? | 10:41.56 |
| kens | chrisl it tracks which page a resource is used on, indexed by resource number | 10:42.21 |
| chrisl | Okay, so it's not copying an existing object, or anything like.... | 10:42.46 |
| kens | SO the size changes a lot, meaning I allocate new memory, copy the old content and then free the old memory | 10:42.53 |
| | Its not an object as such, just a bunch of bytes | 10:43.05 |
| | On the 36th time it gets here, the newly allocated memory pointer is the same as the 'old' memory pointer | 10:43.37 |
| | Which causes bad things to happen | 10:43.52 |
| chrisl | Hmm, that's odd...... | 10:44.27 |
| kens | Actually it happnes before that, I'm just narrowing down where it happens | 10:44.50 |
| | the 33rd time in fact | 10:46.48 |
| Robin_Watts | --debug=gc-disable ? | 10:49.22 |
| kens | Haven't tried it, but I want to fix the problem :-) | 10:49.44 |
| Robin_Watts | OK, That solves the segv but tells me that the postguard to a block has been overwritten. | 10:50.27 |
| kens | Ah, well *that* I can well believe. | 10:50.42 |
| | Its exactly the sort of problem I was looking for | 10:50.56 |
| | Let me do a new memento build | 10:51.05 |
| Robin_Watts | kens: BTW, you'll get complaints that it should be "Linearize". | 10:52.36 |
| kens | Robin_Watts : Yes, but I'm writing the code and comments, therefore I am correct :-) | 10:52.57 |
| Robin_Watts | IME the person that writes the cheques is correct :) | 10:53.30 |
| kens | Its not important at the moment, and probably won't be the final name of the parameter | 10:53.48 |
| | I just needed 'something' to put in there | 10:53.59 |
| Robin_Watts | Sure. | 10:54.04 |
| | It's the linear_params.Offsets block that is claiming to be corrupted. | 10:54.18 |
| kens | Really ? good grief.... | 10:54.34 |
| | Not what I expected.... | 10:54.41 |
| | That's a different block of memory altogether, it records the positions of each object as they are stored in teh xref | 10:55.36 |
| | Simple test | 10:56.07 |
| Robin_Watts | and it's corrupted between it being allocated, and the next malloc event. | 10:56.08 |
| kens | Which suggests 'something else' wrote over it | 10:56.29 |
| Robin_Watts | (it's allocated on event 498303, and it's corrupt by 498304) | 10:56.29 |
| | It suggests that you're allocating it, then immediately over filling it. | 10:56.43 |
| kens | Which is kind of to be expected | 10:56.46 |
| | It gets used immediatley | 10:56.52 |
| | gdevpdf.c 1754 | 10:57.13 |
| | change : | 10:57.26 |
| | write_xref_section(pdev, tfile, start_section, end_section, resource_pos, linear_params.Offsets); | 10:57.26 |
| | To: | 10:57.37 |
| | write_xref_section(pdev, tfile, start_section, end_section, resource_pos, NULL | 10:57.37 |
| | and see if the segv goes away | 10:57.43 |
| Robin_Watts | just in the middle of something, sorry. | 10:58.00 |
| kens | NP I'm trying it ehre too | 10:58.08 |
| Robin_Watts | Yes, it's definitely in the write_xref_section where it gets trashed. | 10:59.13 |
| | Memento_checkAllMemory() runs fine before that, but flags an error afterwards. | 10:59.28 |
| | (You can run that from Ctrl-Alt-Q in VS) | 10:59.41 |
| kens | It does seem to resolve the seg fault, so all I have to do is figure out why the allocation is too small, thanks Robin | 11:00.31 |
| | Then I can unmangle the code again.... | 11:00.46 |
| | Ah, sadly it does not solve teh problem in a regular debug build.... | 11:02.42 |
| Robin_Watts | so, it's a 312 byte block. | 11:02.54 |
| kens | So it must be a different problem :-( | 11:02.55 |
| Robin_Watts | Why is 'start' an int64_t and end an int? and i a long ? | 11:03.28 |
| kens | I don't remember | 11:03.40 |
| | Oh start is an offset into the file | 11:03.48 |
| | end is an int for reasons that escape me | 11:04.02 |
| | Bear in mind this is code that is in flux | 11:04.39 |
| Robin_Watts | OK, the block is getting corrupted when i = 63 | 11:04.43 |
| kens | The real question is why i > the LastResource member | 11:04.59 |
| | But in any event, teh non-memento code does not get this far | 11:05.10 |
| | s/code/build | 11:05.17 |
| paulgardiner | Robin_Watts: tiny commit on paul/master if you have a moment later. | 11:05.21 |
| Robin_Watts | is the LastResource member increasing between allocation and usage? | 11:05.26 |
| kens | Robin_Watts : it may do so | 11:05.36 |
| | sorry, *not* | 11:05.44 |
| | It could be incorrectly calculated at the allocation stage though, it is independent of the code which writes xref | 11:06.04 |
| | Because this code is all a horrible mess | 11:06.11 |
| | LastResource is only 62 | 11:07.25 |
| Robin_Watts | yeah linear_params.LastResource is 62 at allocation time. | 11:07.26 |
| | so end being 64 would indeed cause it problems. | 11:07.46 |
| kens | Yes, but as I say, this isn't what's causing the original problem | 11:08.03 |
| | In a normal debug build we don't get this far | 11:08.17 |
| Robin_Watts | Right, but this is *a* bug, and if you fix it it's possible we'll run into the same thing that's killing the debug build a bit further on? | 11:09.17 |
| kens | No, because the debug build doesn't get this far | 11:09.28 |
| | It crashes before we get ehre | 11:09.36 |
| Robin_Watts | Right, but the same problem may recurr later? | 11:09.53 |
| kens | Though I am fixing ti anyway. I can't recall why I didn't use pdev->next_id in the first place, I think I needed the number at an earlier stage in the past | 11:10.15 |
| Robin_Watts | anyway, I'll leave you to it and check back in in a bit. | 11:10.28 |
| kens | Robin_Watts : no, later isn't important, the crash occurs earlier | 11:10.29 |
| | Oh, I do need it earlier. | 11:12.14 |
| | Ah, it's OK I rewrote that code, I do need the number, but not until later now. | 11:13.11 |
| | Robin_Watts : I pushed new code. With memento and --debug=gc-disable this does not crash and shows no problems. Wihtout --debug=gc-disable, it still crashes on both Memento and debug builds | 11:25.51 |
| | BTW memento *dos* show memory leaks, but they aren't pdfwrite ones.... | 11:26.25 |
| | OK I'm going to grab some lunch back shortly | 11:31.07 |
| Robin_Watts | back. | 12:33.26 |
| | kens: Hmm. You've got 3 commits on your branch, some with broken code, right? | 12:34.34 |
| | No problem as yet, but you'll need to flatten them before you push to the real repo. | 12:34.55 |
| kens | Robin_Watts : probably true | 12:35.06 |
| | I'll flatten them all anyway | 12:35.36 |
| Robin_Watts | If you are working from the command line you can use the --amend option on a commit to mean "merge it into the last one". Presumably gitk has a similar option there. | 12:35.46 |
| | OK. I can confirm that it still goes wrong using: | 12:53.23 |
| | gs/membin/gswin32c.exe -dNOGC -sDEVICE=pdfwrite -dLinearise -o out.pdf ../ghostpcl/tests_private/ps/ps3cet/09-11.PS | 12:53.32 |
| | (I'm using -dNOGC to try to reduce the amount of gc it does as much as possible) | 12:53.55 |
| kens | Fair enough | 12:54.00 |
| | I *do* believe there's a problem, and its mine, I just can't figure out what I've done.... | 12:54.17 |
| Robin_Watts | it looks to me like pdev->ResourceUsage has been freed. | 12:56.59 |
| kens | Robin_Watts : that's what I believe too, I just don't know how. | 13:00.13 |
| Robin_Watts | It's block 445930 for me. | 13:00.22 |
| kens | THe same memroy gets reused a lot | 13:00.23 |
| Robin_Watts | kens: Not under memento it shouldn't be. | 13:00.37 |
| | It gets freed later on under gc_free_empty_chunks | 13:00.50 |
| | I'm guessing you're not marking it under gc enumeration? | 13:01.17 |
| | Morning mvrhel_laptop | 13:03.02 |
| | It does look like it's allocated by a stable allocator though (chunk allocator) | 13:04.33 |
| | looks like the most stable allocator going. | 13:04.49 |
| kens | It could be me freeing it | 13:07.32 |
| | I'll keep looking | 13:07.43 |
| Robin_Watts | No, you're not freeing it. | 13:07.52 |
| kens | Oh, well that's something.... | 13:08.01 |
| Robin_Watts | Memento_breakOnRealloc(block number) stops when the block is freed or realloced. | 13:08.15 |
| kens | I'll need to switch back to memento, currenlty on a debug build | 13:08.37 |
| Robin_Watts | And it stops when the block is freed in gc_free_empty_chunks | 13:08.45 |
| | Not saying you should swap, just saying how I found it. | 13:09.04 |
| kens | Hmm I guess the question is why the chunk is empty | 13:09.07 |
| Robin_Watts | I think the chunk pointer in the reclaim may be bollocks. | 13:17.23 |
| | no, sorry. | 13:19.16 |
| | I need to grab some lunch, but I'll try tracing the chunk pointers in that block to see when the block goes 'empty'. | 13:20.39 |
| | afterwards. | 13:20.43 |
| kens | OK thanks Robin_Watts | 13:20.46 |
| Robin_Watts | OK, so initially cp->cbot points to the top of the chunk, meaning it's got something in. | 14:09.45 |
| | but by the time we hit the empty chunk test cp->cbot has been set down to cp->cbase, implying it's empty. | 14:10.06 |
| | I'll try and use a data breakpoint to spot where that change is happening. | 14:10.21 |
| kens | makes snse, so I guess the next thigng is where tha happens | 14:10.27 |
| | TBH you are a long way ahead of me | 14:10.53 |
| | Robin_Watts : are we alwatys using the same chunk ? | 14:12.02 |
| Robin_Watts | In memento builds, 1 object per chunk. | 14:12.27 |
| kens | ah, of course, that helps | 14:12.37 |
| Robin_Watts | gc_objects_compact... | 14:14.20 |
| | gc_objects_compact is somehow deciding that that chunk can be compacted to be empty. | 14:21.44 |
| kens | Umm well that seems 'odd' | 14:22.12 |
| | I don't have to mark the bytes specially or anythign do I ? | 14:23.08 |
| Robin_Watts | kens: Are you free to look at gc_objects_compact for a mo? | 14:23.10 |
| | kens: I don't believe so. | 14:23.16 |
| kens | I set a BP there, I'm waiting for VS to get tehre | 14:23.24 |
| | First break | 14:23.35 |
| | OK Robin_Watts | 14:23.50 |
| Robin_Watts | /* An object is free iff its back pointer points to */ | 14:23.54 |
| | /* the chunk_head structure. */ | 14:23.56 |
| | if (pre->o_back << obj_back_shift != (byte *) pre - (byte *) chead) { | 14:23.58 |
| | I'm trying to decide if that condition means "object is free" or "object is NOT free". | 14:24.17 |
| | I'm thinking that it means "is not free". | 14:24.25 |
| kens | NOT free I beleive | 14:24.34 |
| Robin_Watts | OK, so as far as I can see, that test is FAILING, so we're thinking an object is free, and hence compacting the chunk. | 14:24.54 |
| kens | WHich means hte back pointer is the chunk_head | 14:25.09 |
| Robin_Watts | So presumably pre->o_back must be changing. | 14:25.32 |
| kens | That's stramge though, what does the first object back pointer point to, if its not the head? | 14:25.35 |
| | o_back seems to be a macro | 14:26.00 |
| Robin_Watts | of course it is :( | 14:26.17 |
| kens | As is obj_back_shift | 14:26.25 |
| Robin_Watts | pre is the block just before the actual data. | 14:27.09 |
| kens | Hmm, a header ? | 14:27.23 |
| Robin_Watts | so if we are corrupting the start of a block, that might explain this. | 14:27.25 |
| | yes. | 14:27.27 |
| kens | Corrupting the start of a block would make sense if I got a resource ID < 1 | 14:27.53 |
| Robin_Watts | When you get a block of memory from the chunk allocator, there is an obj_header_t before the actual block. | 14:27.53 |
| kens | Yes, I think I was looking at that earlier | 14:28.13 |
| | o_back = d.o.f.b.back | 14:28.36 |
| Robin_Watts | Memento puts preguards in for just this reason, but of course the chunk allocator is running inside memento :( | 14:29.14 |
| kens | exc ept aht b doesn't have a member 'back'.... | 14:29.23 |
| Robin_Watts | back is probably a macro. | 14:30.04 |
| kens | I'm guesing so. | 14:30.11 |
| | Typical GS stuff :( | 14:30.19 |
| | * - In free objects, the back pointer is an offset from the object | 14:30.44 |
| | * header back to a chunk_head_t structure that contains the location | 14:30.44 |
| | * to which all the data in this chunk will get moved; the reloc field | 14:30.44 |
| | * contains the amount by which the following run of useful objects | 14:30.44 |
| | * will be relocated downwards. | 14:30.44 |
| | * - In useful objects, the back pointer is an offset from the object | 14:30.44 |
| | * back to the previous free object; the reloc field is not used (it | 14:30.45 |
| | * overlays the type field). | 14:30.45 |
| | Huh despite what VS says, it looks like back is a member of a structure in a union | 14:32.53 |
| ronulirific | hi everyone | 14:38.12 |
| | who do I contact if I want to denounce a violation of the ghostscript licence? more specifically, ghostscript being used in a commercial product without releasing the source or even acknowledging it. | 14:39.20 |
| kens | Either our CEO or VP of sales. | 14:39.36 |
| | Or support@artifex.com | 14:39.48 |
| Robin_Watts | ronulirific: miles.jones@artifex.com | 14:39.58 |
| | (That's our CEO) | 14:40.05 |
| | ronulirific: And thank you. | 14:40.34 |
| | Though I'm interested to know for myself which product it is if you can say... | 14:41.11 |
| ronulirific | yes, sure | 14:42.51 |
| Robin_Watts | kens: looks like gc_objects_clear_marks may be involved. | 14:43.48 |
| kens | Hmm, I suppose that is likely | 14:45.07 |
| Robin_Watts | I don't understand that. | 14:54.37 |
| kens | does not understand most of this, its all greek to me | 14:55.43 |
| chrisl | Why does my router want me to login to it just so I can logout - seems daft..... | 15:05.26 |
| henrys | marcosw:ping | 15:22.33 |
| Robin_Watts | kens: Interesting. | 15:23.05 |
| | In alloc_obj, at line 1241 I have: | 15:23.29 |
| | pre->o_alone = 1; | 15:23.35 |
| | ptr->o_alone = 1; even | 15:23.42 |
| kens | give me a few minutes | 15:23.59 |
| Robin_Watts | Oh, ignore me. | 15:24.03 |
| henrys | kens:about gs and pdftk - pipitas reports an order of magnitude faster performance in gs bursting pages. pdftk is written in java. | 15:24.32 |
| henrys | wonders how much we could reduce the carbon footprint by porting all java to C. | 15:25.14 |
| Robin_Watts | Just think of all the blu-ray players you'd be sending to landfill. | 15:25.36 |
| kens | henrys we are faster ? :-O | 15:25.43 |
| henrys | kens:yes we're faster. | 15:27.03 |
| kens | I'm ... surprised.... | 15:27.14 |
| Robin_Watts | kens: A random idea here... | 15:28.49 |
| | Is the problem that we're allocating stuff from a memory handler that has been destroyed by the time we try to use it? | 15:29.35 |
| kens | Robin_Watts : I certianly hope not | 15:29.51 |
| paulgardiner | henrys: Hi. There have been a few bugs reported in the MuPDF Android app, plus a change request from Tor. Should I look into those if they don't take too long? | 15:29.54 |
| kens | That is the overall pdfwrite memory handler and much is still alloctaed (I htink) | 15:30.06 |
| Robin_Watts | The reclaim I am seeing is within gs_main_finit | 15:30.40 |
| | which is inside gsapi_exit | 15:30.54 |
| kens | That's not the one causing me problems I don't think | 15:30.57 |
| | The memory should have been freed long before then | 15:31.10 |
| henrys | paulgardiner:yes as long as it doesn't change the forms schedule appreciably. | 15:31.12 |
| kens | Robin_Watts : the memory is freed at teh end of gdevpdf.c | 15:31.43 |
| | before we close the device | 15:31.50 |
| | But the seg faults I see are *much* earlier that that | 15:32.11 |
| paulgardiner | henrys: Okay ta. None of them are show stoppers, so I'll tackle only what can be done quickly. | 15:32.30 |
| Robin_Watts | kens: The close device happens after this? | 15:32.45 |
| kens | after what ? | 15:32.58 |
| Robin_Watts | In imain.c, gs_main_finit | 15:33.02 |
| kens | The file writing is part of the device close | 15:33.07 |
| | Robin_Watts : I believe the device should be clsoed by then, maybe I'm wrong | 15:33.23 |
| Robin_Watts | we're within interp_reclaim at line 830. | 15:33.27 |
| | The device may be being closed at line 853 ? | 15:33.41 |
| kens | Just a moment | 15:33.52 |
| | The device is closed in gs_main_finit | 15:34.09 |
| | Yes at 853 | 15:34.23 |
| Robin_Watts | Right, so this reclaim is happening before the device is closed. | 15:34.43 |
| kens | Seems to be yes. | 15:34.51 |
| | But at the time we get to closedevice the pdfwrite device still has lots of memory allocated | 15:35.15 |
| | Which does not seem to cause a problem | 15:35.22 |
| Robin_Watts | Interestingly, the memory pointer looks different in the gc to when we were allocating. | 15:36.07 |
| kens | Hmm, did Ray's chnages for teh server mode of pdfwrte change anything do you think ? | 15:37.52 |
| | I'm not sure this commetn : | 15:39.06 |
| | * Close the "main" device, because it may need to write out | 15:39.06 |
| | * data before destruction. pdfwrite needs so. | 15:39.06 |
| | */ | 15:39.06 |
| | matches up with the code. | 15:39.13 |
| Robin_Watts | It's not possible that pdfwrite is being called twice? Once early, and once late? And it's the late one that's crashing? | 15:39.44 |
| | straw.grasp(); | 15:39.58 |
| kens | TO be honest, I don't know. | 15:40.07 |
| | I wouldn't have thought so, I don;t think that would work | 15:40.17 |
| | Robin_Watts : it does seem to be that reclaim causing hte problem | 15:45.50 |
| Robin_Watts | With -dNOGC I think that may be the only reclaim in town. | 15:46.12 |
| kens | Ah, that may be why then :-) | 15:46.22 |
| kens | tries without -dNOGC | 15:47.03 |
| | debug and memenot builds don't seg fault until after that recalim | 15:48.11 |
| | It seems that is the only vm reclaim that takes place, regardless of whether -dNOGC is set or not | 15:49.01 |
| | Possibly it only causes a problem when a reclaim executes. | 15:49.32 |
| | Robin_Watts : would this be easier to track if the pointer was in GC'ed memory ? | 15:49.56 |
| Robin_Watts | no :) | 15:50.07 |
| kens | OK that's a good answer :-) | 15:50.16 |
| henrys | kens:writing the newsletter; hard to find anything substantial to say about A-2 after reading the related patches. I'll just say we can now produce compliant output. | 15:52.07 |
| kens | henrys, yes that seems to be about alll there is to say about it :-) | 15:52.28 |
| | Please remember to note that we only produce PDF/A2-b | 15:52.50 |
| paulgardiner | Robin_Watts: Can you easily give me an up-to-date 22043_tcs_drive_mastercard_de.mjs? | 15:53.37 |
| henrys | kens:okay | 15:53.58 |
| Robin_Watts | paulgardiner: Do you have an account on peeves? | 15:54.23 |
| paulgardiner | I think I do | 15:54.41 |
| Robin_Watts | /home/marcos/cluster/tests_private/pdf/forms/... | 15:54.57 |
| paulgardiner | Yep. In. | 15:55.00 |
| | ta | 15:55.03 |
| Robin_Watts | That's always as up to date as the last cluster run | 15:58.40 |
| kens | Robin_Watts : we go through ireclaim with the same memory pointer severl times before the one that causes a problem, when -dNOGC is not specified | 16:14.14 |
| Robin_Watts | I think we should have a rule that every employee needs to remove 1 macro from gs per day. | 16:16.23 |
| marcosw | henrys: hello | 16:17.20 |
| kens | Robin_Watts : I wonder if you are correct. We do allocate a new array after the reclaim,. and it 'seems' to be this which causes the problem. | 16:18.35 |
| marcosw | Robin_Watts: you don't know what it was like 10+ years ago; the macro situation is so much better... | 16:18.38 |
| Robin_Watts | If we adopted my rule then in 10 years it'd be almost sane. | 16:19.10 |
| kens | If you limit the crteria for sane to macros.... | 16:20.06 |
| henrys | marcosw:wondering what became of the pdf output size comparison, hoping to get something in this newsletter. | 16:20.37 |
| marcosw | is wondering that myself | 16:20.53 |
| henrys | okay next newsletter ;-) | 16:21.18 |
| marcosw | what likely happened is a started the job and then forgot to look at the results; that's usually what happens. I'll check. | 16:21.42 |
| kens | Robin_Watts : I made a quick hack to ensure that we basiclly only make one allocarion of 'ResourceUsage', and that still crashes, though it now crashes when trying to free the memory. | 16:23.40 |
| | Again it looks like the memory has already been freed. | 16:24.01 |
| | I have to run, Melanie's riding lesson. I'll check the logs later. | 16:24.45 |
| Robin_Watts | Morning mvrhel_laptop. | 16:40.21 |
| | Did you get anywhere with that halftoning stuff? | 16:40.31 |
| | kens: (For the logs) I have a fix. | 16:46.15 |
| | Everything else in the gx_device_pdf is enumerated for the gc, except for this. | 16:46.37 |
| | If you add it to gdevpdfx.h line 793: | 16:47.08 |
| | m(36,Identity_ToUnicode_CMaps[0]) m(37,Identity_ToUnicode_CMaps[1])\ | 16:47.28 |
| | m(38,ResourceUsage) | 16:47.30 |
| | #define gx_device_pdf_num_ptrs 39 | 16:47.32 |
| | (Notice the increase of gxdevice_pdf_num_ptrs) then everything runs fine. | 16:47.52 |
| | Basically the garbage collection was clearing the marks of every object, then scans through all the reachable objects to mark them. | 16:48.15 |
| | because we weren't enumerating it, it wasn't being marked, and was therefore being considered free. | 16:48.32 |
| | I have to confess to being confused as to why we need to mark things that are unmovable, but this does seem like the right fix. | 16:49.19 |
| | paulgardiner: Looking at your patches now. | 16:51.40 |
| henrys | mvrhel_laptop or anyone the new PDFX3 profile option refers to the prepress stuff 10.10.4? | 17:17.47 |
| mvrhel_laptop | hi Robin_Watts I am still working on the halftone stuff | 17:25.37 |
| | I have chopped up a simple example | 17:25.50 |
| | and am going to be going through the scaling stuff and there appears to be an issue with the levels | 17:26.04 |
| | I am wondering though if we have this issue even with the cases that currently run | 17:26.23 |
| | in the head | 17:26.27 |
| | henrys: where is the PDFX3 profile option described? | 17:27.20 |
| Robin_Watts | mvrhel_laptop: If there is anything I can do to help, please say. | 17:27.46 |
| mvrhel_laptop | ok. Robin_Watts, I may drag you in this | 17:28.21 |
| Robin_Watts | feel free, I'm at a loose end right now. | 17:34.21 |
| henrys | mvrhel_laptop:the new icc color parameter added in this release. | 17:37.25 |
| | UsePDFX3Profile | 17:37.52 |
| mvrhel_laptop | oh ours | 17:38.19 |
| | crap | 17:39.03 |
| | oh there it is | 17:39.46 |
| | in Use.htm | 17:39.57 |
| | henrys in the ICC section | 17:40.09 |
| henrys | yes I read that I'm trying to tie it in with the pdf spec so I can understand exactly what to say in the newsletter. | 17:41.09 |
| mvrhel_laptop | oh. the pdf spec says little about the output intent other than defining it | 17:41.46 |
| | the ghent group pushes its usuage | 17:42.23 |
| | henrys: I can get you something about it | 17:42.32 |
| | I have to head out now for a bit though | 17:42.39 |
| henrys | okay thanks | 17:43.24 |
| Robin_Watts | Damn. I have a clist question that I suspect I need ray for. | 17:56.22 |
| ray_work | Robin_Watts: got your sms | 18:01.33 |
| Robin_Watts | Morning ray | 18:01.40 |
| ray_laptop | morning (still, barely) | 18:01.51 |
| Robin_Watts | Thanks. I have a clist question... | 18:01.53 |
| | When I use the downscaler with the planar device I fetch 1 plane at a time from the rendered buffer. | 18:02.50 |
| | Hence it would be advantageous to have the bandheight be a multiple of the downscalefactor. | 18:03.18 |
| | So I thought I'd step through the device open stuff to see where the BandHeight is calculated, and to see if it's somewhere that I can influence it. | 18:04.13 |
| | but as far as I can tell with: -r300 -dMaxBitmap=10000 -sDEVICE=psdcmyk -o out.psd gs/examples/tiger.eps the space_params.band details are never filled in. | 18:04.56 |
| ray_laptop | Robin_Watts: if you just want the planar device to use the BandHeight, then clist_get_band_height(dev) (macro in gxclist.h) will give it | 18:08.36 |
| Robin_Watts | ray_laptop: I need to be able to read the bandheight, and write back a slightly smaller one. | 18:09.06 |
| ray_laptop | Robin_Watts: I think that is better than trying to influence it, since that may not respect the command line options | 18:09.37 |
| | Robin_Watts: why smaller ? | 18:09.50 |
| Robin_Watts | Well, if the BandHeight has been calculated to fit a given buffersize, I don't want to increase it and hence make it take more than the buffersize. | 18:10.30 |
| ray_laptop | but in general, if the BandHeight is determined automagically from the BufferSpace you can't change it | 18:10.45 |
| Robin_Watts | Right. That's bad. | 18:10.58 |
| ray_laptop | but why do you need it to be slightly smaller ??? | 18:11.18 |
| Robin_Watts | Suppose we have a BandHeight of 100 | 18:11.30 |
| | and I have a downscale factor of 3. | 18:11.38 |
| | And we have a CMYK planar device. | 18:11.48 |
| | When we come to do output line 33, the downscaler will read lines 99,100 and 101 for C. | 18:12.33 |
| | So 99 will come from the already rendered band. | 18:12.40 |
| | 100 and 101 will cause the second band to render. | 18:12.50 |
| | Then I'll ask for 99, 100 and 101 for M, which will cause the first band to render again, then the second. | 18:13.09 |
| ray_laptop | right, so to avoid re-rendering the band with line 99, you need to tuck the other planes away | 18:13.15 |
| Robin_Watts | That doesn't fit with the interface to the downscaler. | 18:13.31 |
| ray_laptop | Robin_Watts: but realize the BandHeight can be < the downscale factor | 18:13.49 |
| Robin_Watts | I'd far rather have some way to say "I'd like the BandHeight to be a multiple of 3". | 18:13.55 |
| | ray_laptop: Well, in that case we get lots of rerendering :) | 18:14.04 |
| | OK. I'll investigate having the downscaler tuck the components away. | 18:16.17 |
| | Thanks. | 18:16.29 |
| ray_laptop | Robin_Watts: so, the downscale factor needs to be part of the BandHeight calculation in gdev_prn_allocate | 18:16.31 |
| | Robin_Watts: we take the pdf14 transparency into account, so making an already ugly routine slightly uglier isn't too bad. | 18:17.09 |
| Robin_Watts | ray_laptop: Yes. I did vaguely consider having that call a devspecop to ask if there was a bandheight multiple. | 18:17.16 |
| | Find the multiple; if the bandheight < multiple then bandheight = multiple, otherwise bandheight -= (bandheight % multiple); | 18:18.01 |
| ray_laptop | but I would suggest adding a space_param to identify the integer sub-multiple requirement | 18:18.02 |
| Robin_Watts | Having the downscaler buffer may not be too bad. | 18:19.15 |
| ray_laptop | I tend toward params because that is easier to control from the command line, but maybe a specop is better because the command line may violate the device requirements | 18:19.24 |
| Robin_Watts | ray_laptop: Right. It's a device imposed thing. | 18:19.42 |
| | but having the downscaler buffer avoids the issue. | 18:20.17 |
| ray_laptop | Robin_Watts: I have no objections to the dev_specop approach | 18:20.26 |
| Robin_Watts | The only thing that worries me with the devspecop approach is that someone might demand a particular bandheight and we'd give them a different one. | 18:21.04 |
| | hence giving us device to device differences. | 18:21.16 |
| ray_laptop | Robin_Watts: I thought you were going to just give the multiple constraint. If the prn_allocate can't meet it, we wouldn't open | 18:22.07 |
| Robin_Watts | oh, I see. | 18:22.35 |
| | Well, buffering in the downscaler is nicer - keeps the complexity localised. Let me try for that first. | 18:22.56 |
| ray_laptop | Robin_Watts: similarly to what happens if a BandHeight is given, but the allocate can't get enough RAM | 18:23.13 |
| Robin_Watts | but if it all gets too hairy, I'll come back to this constraint idea. | 18:23.21 |
| ray_laptop | Robin_Watts: but you realize that the clist_band_height can ONLY be used in clist mode so you have to make sure and check that you aren't in page buffer mode | 18:24.30 |
| Robin_Watts | If I'm buffering in the downscaler clist or not makes no odds. | 18:25.10 |
| ray_laptop | Robin_Watts: but I thought you needed to know how much to buffer | 18:25.47 |
| Robin_Watts | I buffer the downscale factor number of lines. | 18:26.03 |
| ray_laptop | Robin_Watts: or are you going to buffer up by the downscaler facter | 18:26.08 |
| ray_laptop | didn't type question fast enough. | 18:26.24 |
| Robin_Watts | :) | 18:26.29 |
| ray_laptop | that makes sense then. And it isn't really that much memory (unless some bozo has LOTS of separations) | 18:26.51 |
| Robin_Watts | some bozo = any PS file. | 18:27.14 |
| ray_laptop | combined with whatever the max downscale factor is (4?) | 18:27.18 |
| Robin_Watts | ray_laptop: 4 is typical. | 18:27.55 |
| ray_laptop | Robin_Watts: at render time, when the downscale happens, we know how many separations were used | 18:27.59 |
| | so PS shouldn't be any differenct | 18:28.21 |
| Robin_Watts | So color_info.num_comps has the correct number when we enter print_page ? | 18:28.57 |
| | Oh. | 18:31.29 |
| | The downscaler appears to be buffering data already in this case. | 18:31.52 |
| Robin_Watts | pushes against an open door and falls flat on his face. | 18:33.08 |
| henrys | was there a mupdf 1.1 candidate? | 18:33.22 |
| ray_laptop | oh, well. | 18:33.24 |
| Robin_Watts | henrys: Yes. It's on googlecode | 18:33.35 |
| | http://code.google.com/p/mupdf/downloads/list | 18:33.52 |
| henrys | thanks | 18:34.15 |
| ray_laptop | I'll send an email, but I'm going to take the afternoon off (beach day -- it's hot here) | 18:34.41 |
| Robin_Watts | have fun. | 18:35.05 |
| henrys | ray_laptop:get that in before the kids go back | 18:35.41 |
| | I can't believe how fast this summer has gone. | 18:36.02 |
| | Robin_Watts:do you have any desire for a forerunner 305? I can't seem to get rid of it here. | 18:59.20 |
| Robin_Watts | henrys: Thanks for the offer, but Helen has a 305 and I have a 310. | 18:59.46 |
| | One for the dog would be excessive :) | 19:00.00 |
| henrys | oh for some reason I thought you were using a pedometer. | 19:00.19 |
| Robin_Watts | I started with a Nike+ thing, but upgraded soon after. | 19:01.09 |
| pipitas1 | henrys: kens: Please don't take my rather accidental result about pdftk/gs bursting PDF pages for granted, where gs came out on top. I'll do a proper benchmarking later. It could have well been that pdftk was handicapped by parallel system load or swapping or somethingâ¦. | 19:07.28 |
| kens | Robin_Watts : your code duplicates what I had before, which caused 1200 seg faults.... | 19:27.54 |
| | I had changed to make the memory immovable in the (obviously mistaken) belief that this made it non-gc too. | 19:28.21 |
| | I'll try putting it back that way tomorrow. | 19:28.28 |
| Robin_Watts | Sorry, you'd previously had it enumerated, and it caused seg faults? | 19:28.36 |
| | I suspect you may need both enumerated and non movable. | 19:28.46 |
| kens | Robin_Watts : yes, 1200 seg faults in the cluster | 19:28.47 |
| Robin_Watts | but try it and if not, we can look at why tomorrow :) | 19:29.03 |
| kens | I figured if it was enumerated it didn't need to be immvable... | 19:29.08 |
| | Robin_Watts : yes, that's what I plan :-) | 19:29.18 |
| | If I get very entusiastic I may try it shortly | 19:29.30 |
| | hmm cluster is idle, lets see what happens ;-) | 19:32.56 |
| | AH, that's not too good | 19:33.39 |
| | clusterpush doesn't work now :-( | 19:33.53 |
| | maybe gitpush.sh doesn't know which remote to use ? | 19:34.30 |
| | D'oh, no I forgto to run ssh | 19:34.46 |
| | Robin_Watts : that seems much better, I will undo all the other changes tomorrow and retest | 21:00.34 |
| | BTW I'm getting 2 warnings in libopenjepg now as well | 21:01.34 |
| | goodnight all | 21:01.45 |
| Robin_Watts | kens, chrisl, tor8: I just got an email telling me that my ESTA (which I renewed just before they started charging, deliberately) is about to run out. | 23:54.49 |
| | If you guys did the same, you'll need to get new ESTAs too. (I printed mine out and stapled it in my passport so I'd have a record of the expiry date). | 23:55.21 |
| | Forward 1 day (to 2012/08/10)>>> | |
IRC Logs