| | <<<Back 1 day (to 2018/07/18) | 20180719 |
| chrisl_laptop | NancyDurgin: So, kens noticed you were storing the graphics in the erasepage optimization device.... | 14:38.03 |
| | Ah, kens is on the other channel.... | 14:38.32 |
| kens | Only for checking availability | 14:38.47 |
| chrisl_laptop | I figured it was easier to avoid treading on the other discussion | 14:39.10 |
| kens | Yes goo idea | 14:39.17 |
| | Go right ahead | 14:39.23 |
| Nancy_iMac | I am saving the pointer to the graphics state but I really don't think it's correct. I don't know wht is correct. I need to be able to save the color information. | 14:39.47 |
| | I just saved that pointer because then at least I would have *something* to use later. | 14:40.04 |
| kens | defers to chrisl | 14:40.07 |
| chrisl_laptop | NancyDurgin: It's fine, but for Ghostscript, you need to account for garbage collection | 14:40.19 |
| Nancy_iMac | The problem is I need to save the color that was current when the fillpage (which is being deferred) was originally called. | 14:40.29 |
| | And sometimes the color is some weird complicated thing in the graphics state. | 14:40.39 |
| kens | Saving a pointer to the GS is (IMO) absolutely the easiest and best thing to do | 14:40.51 |
| Nancy_iMac | Plus the other problem is when I do call the deferred fillpage later, it wants a graphic state | 14:40.53 |
| chrisl_laptop | As things stand, with the graphics not being passed into all the device methods, you *have* to store it - no choice really | 14:41.14 |
| Nancy_iMac | So if saving the GS is okay, there is still the question of how to grab the color info out of it. I actually did find a function squirelled away somewhere (I am not at the code right now) that could save the hl color state out of the graphics state | 14:41.56 |
| | but there is no corresponding "restore" for this function. | 14:42.07 |
| kens | Don't bother getting the colour | 14:42.19 |
| | just use the gs for teh fillpage, since it takes it | 14:42.30 |
| Nancy_iMac | How do I fill with the correct color, then? | 14:42.30 |
| | The problem is the color could have changed. | 14:42.40 |
| kens | Yeah that's true | 14:42.50 |
| Nancy_iMac | If you look at results from my latest run (which is from yesterday) then it works quite a bit, but sometimes has the wrong color. | 14:42.59 |
| kens | But you need the gstate anyway | 14:43.23 |
| Nancy_iMac | I think it was all pdf that were failing, possibly because they commonly do fillpage without using white | 14:43.35 |
| kens | So probably best to cover that and gc right now | 14:43.41 |
| Nancy_iMac | yeah I need the gstate, but it doesn't have the right color | 14:43.47 |
| | okay, tell me about gc and gstate :) | 14:44.16 |
| kens | It will only have an incorrect colour if something changes the current colour between the fillpage and the first marking operation | 14:44.17 |
| | I'll let chrisl describe teh GC stuff | 14:44.27 |
| Nancy_iMac | kens: Right, but I need to handle that case (incorrect color) | 14:44.38 |
| kens | Lets get GC done first and come back to that | 14:44.49 |
| Nancy_iMac | ok | 14:44.52 |
| chrisl_laptop | The gc stuff is here: https://git.ghostscript.com/?p=user/chrisl/ghostpdl.git;a=commitdiff;h=471938fc63bc54efef281361b54abfad078a968d | 14:45.07 |
| | In this case, fairely simple.... | 14:45.21 |
| | I did some quick tests, and it *seems* to be working as we want | 14:45.50 |
| kens | The point here is that the garbage collector is allowed to move garbage collected objects around in memory. | 14:46.33 |
| | So if you keep a pointer to (most) GC'ed objects. you need to be able to deal with it moving during a mark and sweep | 14:46.59 |
| | So you have to declare to the garbage collector that you are maintaining a pointer to that object | 14:47.16 |
| | Makes sense ? | 14:47.27 |
| chrisl_laptop | The ENUM_PTRS function tells the garbage collector that you have a reference to an object, the RELOC_PTRS retrieves from the garbage collector where your object has moved to | 14:48.16 |
| NancyDurgin | yeah, more or less. I understand the concept, just not sure of that code :) | 14:49.12 |
| kens | As long as you understand the concept, that's the important thing | 14:49.27 |
| | Getting GC stuff correct in Ghostscript is, well, non-trivial | 14:49.53 |
| | But its important to understand that, for the PostScript interpreter, you need to | 14:50.10 |
| chrisl_laptop | NancyDurgin: I did write a summary of gs memory management for the twiki a while back: https://twiki.ghostscript.com/do/view/Ghostscript/GhostscriptMemoryManagement | 14:50.17 |
| NancyDurgin | the actually code you wrote is stll Magikal Makros at this time | 14:50.27 |
| kens | Sadly, it always is | 14:50.40 |
| | Generally I do cargo cult programming with GC descriptors | 14:50.59 |
| | I have on that worked before so I copy it and hit it with a large hammer until it works | 14:51.16 |
| NancyDurgin | lol that's an awesome name for it.... copy/paste/edit by pattern | 14:51.18 |
| chrisl_laptop | In concept the garbage collector is pretty straight forward: it is complicated by a) the large amount of data it deals with, and b) all the bonkers macros | 14:52.33 |
| | kens: didn't you write a subclass device to force all text to black? | 14:53.33 |
| kens | Hmm, I think I might have once | 14:53.44 |
| | Is that the example one ? | 14:53.50 |
| | But doing that is easier than tracking colour changes | 14:54.02 |
| chrisl_laptop | I can't remember.... but it must save the color, replace it, and put it back again | 14:54.35 |
| NancyDurgin | I was looking at the color thing, and I found that in most cases it is a device that supports "hl_color", and it stores things in the graphics state. And I found a function that "saves" the color (makes a copy of it). | 14:54.41 |
| kens | Yes the documented subclass example is the 'black text' device | 14:54.43 |
| NancyDurgin | gx_hld_save_color() | 14:55.13 |
| kens | Well copying the colour sounds good | 14:55.16 |
| NancyDurgin | but there is not corresponding "restore" function. I was thinking I could try to write an appropriate restore function, assuming that is even the right idea in general. | 14:55.44 |
| kens | Yeah I doubt if there's a restore. | 14:56.12 |
| | I can't see ay solution that doesn't involve copying the colour, and forcing it into the graphics state later | 14:56.39 |
| NancyDurgin | I need to "save" when the original fillpage happens, then right before I do the actual deferred fillpage, I save whatever is current at that time. Then I "restore" the one I had saved with the fillpage, do the fillpage, and then "restore" the one I had just saved. | 14:56.46 |
| kens | But you'll need 2 copies, one for the initial state, and one for the 'current' state when you .... Yes what you just said | 14:57.10 |
| NancyDurgin | if you think that's the right track, I can try it. The function that saves is not very simple looking. | 14:57.19 |
| chrisl_laptop | But you'll need to save more than just the color: halftone, transfer function, maybe ctm..... possibly other stuff | 14:57.59 |
| NancyDurgin | right, that's where it gets a bit crazy. It looked to me like there was tons of stuff in the graphics state that might be relevant. hl_color isn't all of it? | 14:58.28 |
| kens | Well *anything* might have changed in the graphics state | 14:58.50 |
| | And it all needs to be 'correct' for the page to be properly erased | 14:59.16 |
| NancyDurgin | I know. Once I thoguht about it, it seemed potentially very messy to me. | 14:59.22 |
| kens | changes to CTM will be problematic, as will clip, colour etc. | 14:59.27 |
| chrisl_laptop | So, maybe taking a copy of the graphics is a better solution | 14:59.38 |
| kens | I wonder if we might be better messing with the gstate stack | 14:59.39 |
| | Copying the gstate is possible | 14:59.54 |
| | In some ways perhaps makes more sense | 15:00.02 |
| | We have a copy_gstate somewhere don't we ? | 15:00.10 |
| | I think we use it for gsave | 15:00.16 |
| NancyDurgin | hmm yes there is something called gstate_copy | 15:01.03 |
| kens | That sounds about right | 15:01.10 |
| | I believe that when we do a gsave we make a new gstate, copy thte current contents into it, then point the current gstate 'next' at the newly created state | 15:01.35 |
| | I could be mistaken though | 15:01.42 |
| NancyDurgin | so then.... is copying the entire gstate on every fillpage going to have a similar amount of overhead to jsut doing the fillpage? | 15:01.44 |
| | especially if in banding mode? | 15:01.55 |
| chrisl_laptop | Good question..... | 15:02.00 |
| kens | Probably worse for a clist | 15:02.13 |
| | For a page device, not so clear | 15:02.23 |
| NancyDurgin | is there a flag somewhere to say it is in clist mode? | 15:02.30 |
| kens | Sadly no | 15:02.36 |
| chrisl_laptop | Why worse for clist? | 15:02.42 |
| kens | There may be when I get done though | 15:02.43 |
| NancyDurgin | argh... | 15:02.45 |
| kens | I mean the copy is worse than executing fillpage for a clist | 15:03.03 |
| NancyDurgin | clist already has some check to not waste its time doing multiple fillpages I think. | 15:03.08 |
| kens | Migh tnot be though, depends how clever the clist is | 15:03.16 |
| | Right | 15:03.22 |
| NancyDurgin | If I have a quick test I can just not even install the erasepage optimization if in clist mode. | 15:03.41 |
| | I don't know how to do that check | 15:03.44 |
| chrisl_laptop | Oh, I see... I was worried the device was being subclass post clist | 15:03.45 |
| kens | So as normal for Ghostscript we have both copy_gstate and gstate_copy :-( | 15:03.48 |
| NancyDurgin | kens: lol... any idea which one I should be looking at? | 15:04.08 |
| kens | Well copy_gstaet is PostScript-specific, gstate_copy is grapchis library, so its probably that one | 15:04.37 |
| | Oh and we have *2* PostScript copy_gstate procedures | 15:04.59 |
| NancyDurgin | I only see a zcopy_gstate... | 15:05.02 |
| kens | zcopy_gstate and z2copy_gstate..... | 15:05.08 |
| | see zdevice2 | 15:05.24 |
| NancyDurgin | I meant I don't see 'copy_gstate', just 'zcopy_gstate'. But it seems like gstate_copy is the place to look? | 15:05.45 |
| kens | The 'z' prefix is the way we identify PostScript operators in C | 15:06.06 |
| NancyDurgin | it takes reasonable looking parameters and stuff (gstate_copy) | 15:06.17 |
| kens | so zshowpage is the impleenmtnation of showpage etc | 15:06.18 |
| NancyDurgin | yeah I don't want to do the postscript operator in the middle of my c code, so I think I want the gstate_copy | 15:06.41 |
| kens | There's a gs_copygstate | 15:06.53 |
| NancyDurgin | cries | 15:07.01 |
| kens | WHich looks like its the right thing | 15:07.03 |
| | You get used to this with Ghostscript, there's never *one* way to do things | 15:07.34 |
| NancyDurgin | yeah that looks good. It calls the other one anyway, just fills in the two parameters I wasn't sure of | 15:07.35 |
| | okay so instead of saving the pointer, I will copy the entire gstate. In that case I don't need to worry about the gc thing? | 15:08.04 |
| kens | The string parameter is what shows up in the memory debugging | 15:08.14 |
| chrisl_laptop | NancyDurgin: If I were you, I make the copy in garbage collected memory | 15:09.04 |
| kens | OK so if you want to make a copy of the gstate, then I think you want to *no* maintain a GC pointer to it (obviously) | 15:09.07 |
| chrisl_laptop | <sigh> | 15:09.18 |
| kens | chrisl_laptop : Yes I was coming to that, you want to make sure you are using the same vm state as the original gstate I think | 15:09.31 |
| | SO yes it needs to be in GC'ed memory and you need to make sure its local or global VM | 15:09.56 |
| NancyDurgin | Okay I am not sure how to allocate the memory for it, then. | 15:10.14 |
| kens | Which (sorty, tyypo above0 means that yes, you do need a GC pointer | 15:10.16 |
| | NancyDurgin : the gstate has a pointer to its memory in it I believe | 15:10.39 |
| chrisl_laptop | Mainly because something of the things the gstate will point to will also be in gc memory | 15:10.42 |
| NancyDurgin | okay right, the savepage state will still just have a pointer in it, but it will be a pointer to the copy of the gstate I allocated. | 15:10.51 |
| chrisl_laptop | yeh, pgs->memory | 15:10.54 |
| | That's right, yes | 15:11.21 |
| NancyDurgin | s/savepage/erasepage | 15:12.04 |
| kens | NancyDurgin : yes, you alloc (the right type of) memory big enough to hold a copy of the gstate, copy the gstate into it, and then hold a pointer to that memory. That pointer iss what you enumerate in the GC descriptor and reloc_ptr | 15:12.07 |
| NancyDurgin | okay now for multiple deferred fillpages, is it cool that I just keep re-using my spare allocated gstate, or do I have to keep freeing and making a new one for some obscure GC reason? | 15:12.58 |
| kens | I can't see any problem with just overwriting it each time | 15:13.17 |
| NancyDurgin | i.e. I only want to keep the most recent fillpage state | 15:13.18 |
| chrisl_laptop | Um, reference counting..... | 15:14.10 |
| kens | Yeah I was just typing that | 15:14.16 |
| | Does gstate_copy take care of that ? | 15:14.40 |
| chrisl_laptop | It's so long since I looked at this..... :-( | 15:15.16 |
| kens | Same problem here | 15:15.26 |
| NancyDurgin | Well, I can play it safe and free/reallocate it each time | 15:15.28 |
| | then I can look into whether that is necessary | 15:15.35 |
| kens | You won't want to free it directly | 15:15.49 |
| | You'll need to use the gstate_free function or whatever it is called | 15:16.06 |
| NancyDurgin | this is seeming like a ton of overhead :( | 15:16.19 |
| kens | I can't disagree | 15:16.30 |
| | It does make me wonder how useful this is really going to be | 15:16.40 |
| NancyDurgin | I am getting skeptical that this will actually be an optimization | 15:16.42 |
| | Well, worst case we found all those issues with the subclass stuff? | 15:17.21 |
| chrisl_laptop | So, here's a suggestion: how about looking at what in the graphics state fillpage *actually* uses, and only saving that | 15:17.31 |
| kens | Yes, which is worth doing anyway | 15:17.35 |
| NancyDurgin | chrisl_laptop: there are lots of different implementations of fillpage? | 15:18.00 |
| chrisl_laptop | Of course there are.... :-( | 15:19.05 |
| kens | The fillpage device method takes the entire graphics state as a parameter | 15:19.09 |
| | So every device cna do its own thing | 15:19.20 |
| NancyDurgin | yeah that was kind of my conclusion | 15:19.27 |
| | At some "theoretical level", if the whole graphics state is a parameter, then I need to provide that same value to the deferred call. I think trying to do it piecemeal will result in weird bugs? | 15:20.30 |
| kens | chrisl I think that copying/freeing gstates ought to be reasonably quick, because gsave and grestore do this, and we use those a *lot* | 15:20.38 |
| NancyDurgin | I can go ahead and try it. That seems pretty clean/straightforward to try. | 15:21.00 |
| kens | NancyDurgin : I don't thin we can do it partially, I think the only viabl;e solution is to copy the gstate on each fillpage, freeing the old one if present | 15:21.19 |
| NancyDurgin | kens: yes | 15:21.27 |
| chrisl_laptop | I agree, yes. | 15:22.02 |
| NancyDurgin | as far as how useful it is, I think it wil be most useful on higher res devices? But at the same time, higher res devices tend to use clist? Do we have examples of hi-res devices that are not clist (giant frame buffers)? | 15:22.25 |
| chrisl_laptop | Anyway rendering device can be made to use a giant page buffer | 15:23.01 |
| kens | At a guess (you'll need to poke into this yourself) on each fillpage check to see if we have already allocated a gstate (NULLL pointer check seems easiest). If not then allocate one and copy the gstate to it using gstate_copy. If we have already got a copy, use gstate_free_contents to discard it and then use gstate_copy to copy the current one | 15:23.02 |
| chrisl_laptop | gstate_free_contents() will need to be made non-static | 15:23.40 |
| kens | Oh is it static ? Didn't realise | 15:23.53 |
| chrisl_laptop | You could use gs_gstate_finalize() | 15:24.07 |
| kens | We could define an accessor instead gs_gstate_free_contents | 15:24.14 |
| | Oh it has a finalize ? | 15:24.27 |
| chrisl_laptop | It does, and it *is* public | 15:24.40 |
| kens | OK well that's doable | 15:24.47 |
| chrisl_laptop | But, frankly, I wouldn't have a problem making gstate_free_contents() non-static | 15:25.12 |
| kens | Either works for me | 15:25.28 |
| NancyDurgin | err... doesn't gs_gstate_finalize() havea. potential bug in it? | 15:25.34 |
| kens | In what way ? | 15:25.48 |
| NancyDurgin | the cmem thing (which was to be used for debugging?) looks like it could sometimes happen to be zero, so it would return early | 15:25.59 |
| | I mean, it is an unitialized stack variable? What am I missing? | 15:26.17 |
| chrisl_laptop | In the case, the cmem parameter would be pgs->memory | 15:26.30 |
| NancyDurgin | nevermind I can't read... | 15:26.39 |
| kens | :-) | 15:26.43 |
| kens | is totally familiar with that situation | 15:26.51 |
| NancyDurgin | i see... that "cmem;" thing threw me. I guess that is to trick the compiler. | 15:27.04 |
| kens | To avoid a compiler warning or a Coverity warning or something I suspect | 15:27.40 |
| NancyDurgin | anyway, I think I can just call gs_gstate_finalize() | 15:28.01 |
| chrisl_laptop | The finalize methods are called from the memory manager (normally) via a function pointer, so they all take a gs_memory_t * and a void * - in this specific case, the cmem parameter isn't required | 15:28.09 |
| NancyDurgin | so in that case, I don't have reallocate it over and over again, I just have to "free" its contents before copying new things in | 15:28.17 |
| kens | Yeah that's the plan | 15:28.30 |
| chrisl_laptop | That *should* speed things up a bit | 15:28.41 |
| kens | Its still a shed load of stuff to copy, but at least its just the copy, and the functios are already in place to do it | 15:28.56 |
| NancyDurgin | okay I will try this | 15:29.48 |
| | thanks! | 15:29.51 |
| chrisl_laptop | Good luck! | 15:29.56 |
| kens | seconds that | 15:30.07 |
| chrisl_laptop | NancyDurgin: If you do start to get bogged down in (suspected) garbage collector stuff, feel free to toss it to me to look at..... | 15:30.56 |
| NancyDurgin | okay thanks | 15:31.11 |
| kens | Bear in mind that until I finish the clist stuff we can't test this with a clist anyway, it'll all have to be page buffer devices | 15:31.36 |
| | I'm not currently looking at the clist, I need to talk to michael now | 15:31.53 |
| NancyDurgin | a flag to indicate whether a device in clist mode seems like a good thing to me.... I am unclear what you are doing with the clist stuff at this time, though. | 15:33.17 |
| kens | That's because I haven't explained it :-) | 15:33.31 |
| | And I'm facing the exact problem of not knowing if a device is a clist user or not | 15:33.43 |
| | There's a really hacky test in the code already, but I think we need something more... reliable.... | 15:34.18 |
| chrisl_laptop | I think there are a few | 15:34.34 |
| kens | All the tests I've seen rely on castin gthe device to a gdevprn device and testing some of the members | 15:35.08 |
| chrisl_laptop | IIRC, we usually check one of the procs | 15:35.54 |
| kens | Ah, that mihgt do it | 15:37.00 |
| | I should look more closely when I get done with this Lab thing | 15:37.19 |
| | Forward 1 day (to 2018/07/20)>>> | |
Log of #ghostscript at irc.freenode.net.