[gs-devel] Page (clist) queuing -- Seperate Rendering thread(s)
Ray.Johnston at Artifex.com
Wed Nov 14 23:40:00 PST 2007
Igor V. Melichev wrote:
> I agree with Ray's functionality
Good. That is the main thing I am looking for.
> and disagree with Ray's architecture.
I have not yet described an 'architecture', but I'll respond below.
I think you may have been confused by looking at the OLD gxpageq/
gdevprna/gdevbmpa method that I think is a kloodge and I think
should be eliminated. The consultant that came up with this did
not (apparently) talk to Peter or I (before I came to Artifex)
and came up with this.
> Ray assumes that both clist writer and reader exist in same memory space.
This statement is not accurate.
I suppose that I did not make this clear in the email or during our
staff meeting. It is a key point that (when the clist is resident on
a file system) the writer and reader do NOT exist in the same
memory space, nor do they need to use the same gs 'allocator'
Note that if the clist is memfile based, then the reader and the
writer MUST share (at least part of) the same memory space. This
will be the case on embedded systems without a disk drive.
In fact, I proposed that the 'reader' rendering task may use the
simpler 'plalloc.c' allocator even if the write used the PS type
garbage collecting allocator.
The clist files do not (invariant) contain pointers, merely offsets
within the file. The 'gx_saved_page' and the page queue entry (TBD)
also must be 'serialized' so that they do not contain pointers.
There are two issues when serializing the gx_band_page_info part
of the gx_saved_page: (1) the (clist_io_procs *)io_procs entry must
be reduced to a selector that can be used to select memory or disk
clist procs (I suggest an 'int' so that this can be extensible to
other methods). (2) If the clist is memory based, the 'cfname' and
'bfname' entries need to be pointers (for the reader memory space)
so an address must be stored in this area -- there are several
simple methods for this.
> I don't think that it's an useful constraint.
Agreed. see above.
> IMO they may run at different sides of the Earth
> or on different planets.
Agreed. As a more 'real world' example, CalComp used this method
to render saved pages from a file system after a reboot, even
if the firmware (OS) was a different version afterwards. The
three files (cfile, bfile and saved_page_file) were able to be
sent over the internet to allow analysis or printing at our
development facility (to duplicate or work on problems).
> Besides others, dropping this constraint
> gives a simpler architecture, simpler coding,
> and more flexibility as I'll explain below.
> In my view the module gxpageq.c provides some useful
> function prototypes but it is incomplete.
> Particularly it is restricted with FIFO discipline.
> Here is an example why FIFO is not acceptable.
> Suppose an user prints a 100 page document
> with a conveyor-like hardware, which performs
> 3 actions at a time :
> 1. Take the next page from the input tray and warm it;
> 2. Print the current page.
> 3. Cut off the last with TrimBox page and stock it to the output tray.
> If a glitch happens, the user may want to continue
> the conveyor printing because it is fast,
> and to reprint the failed page later.
> The user will insert the re-printed page manually into the booklet.
> I did so multiple times.
> In this example the failed page must not
> be dequeued until it is reprinted, but
> other pages to be dequeued
> to save the printer's RAM.
OK. At the staff meeting, I agreed that there should be an API
hookable procedure to allow the reader (renderer) to determine
when a page is to be deleted. I think that the 'simple' case
of "delete after rendering" is a worthwhile example for users/
customers that want a simple queue, but agreed that making it
easy for more advanced users/customer to control deleting from
the queue is worthwhile.
> gxpageq.c doesn't allow that.
The gxpageq.c implementation is part of the OLD gdevprna/
gdevbmpa kloodge method that I proposed be discarded. This
was implemented by a consultant that did not understand
the saved_page support that Peter developed with us at
CalComp. I agree that it has MANY problems.
> Another hard problem with the old design
> is a wrong subclassing. We need a class for
> normal rendering and another class
> for delayed rendering, and we want
> to apply each of them to each raster device.
> Instead doing so gdevbmpa.c simply duplicates
> some devices, and hopes that we will continue
> the duplicating till the hell. Thanks, I won't.
> The right way is to derive with 2 base classes,
> or to attach the second base class dynamically.
I proposed dicarding the gxpageq/gdevbmpa/gdevprna kloodge.
My proposal was that ANY 'page device' (that can be handled by the
clist) would support queueing.
> I think these 2 problems are enough for requesting another design.
> So here is my one.
> 1. Any raster device may employ clist
> if the user requests a delayed rendering.
Apparently this was not clear to you from my initial proposal, but
this WAS a key point.
> 2. If a delayed rendering is requested,
> Each page is written as (blist,clist) into
> a separate couple of files (bfile, cfile).
> They will get unique file names.
Or these will be separate memfile base addresses when memory based clist
is used. This is exactly what I anticipated and attempted to describe.
This is what the 'gx_saved_page.gx_band_page_info_s' structure
conveys. Peter and I are WAY ahead (>10 years) ahead of you here.
> 3. If a delayed rendering is requested,
> 'showpage' does not run the clist interpreter.
> Instead that it just writes a page title to the output file.
> The title contains the file names mentioned above.
> Then it applies flush() to the output file.
Exactly. This is similar to what gdev_prn_save_page does. What is
missing writing this to a queue with using the NEW queue API.
> That's all changes to the old product.
> gxpageq.c and gp_wsync.c will die.
> 4. In addition to document interpreter we will ship
> another application, which reads the page headers from
> a file and runs the clist interpreter for each page.
> The user can connect it with the first application
> with a pipe. The new application frees page data
I will respond to this later. This concept is OK for an OS based
implementation with file (non-volatile) based clist, but is NOT
appropriate for embedded or multi-threaded applications with a
memory based clist.
> 5. If somebody will need to render pages
> in a random order chosen by a human intervention,
> he will need to insert a filter, which changes the
> order of pages in the pipe. Also it will take care of
> freeing page data when he finds it useful.
> Therefore pages are stored in a file system
> rather than in a single file.
I agree (and described) that being able to render pages in other
than FIFO order is important. In fact I described many more
The key point in the design is to make sure that the queue can
be maintained by the writer and reader and kept in synchrony.
Please wait for a detailed design document on the queue API and
invariants before jumping ahead to a proposal that does not
meet all requirements.
> Thanks to Ralph for noticing the "file system"
> in the last meeting.
> Actually the pager were never stored in a single file
> due to (bfile and cfile).
One, two or three (logical) files is NOT an important issue. The
writer manages the bfile and cfile as separate files (as I am
well aware). The primary issue is the queue API and providing
the information to the reader so that it can get the information
it needs to render the pages. This does NOT need to be a single
> 6. In this design, it is not important
> whether a build uses romfs or
> the underlying OS's file system.
The 'romfs' has NOTHING to do with this. The important thing is to
support both memory and file BAND_LIST_STORAGE methods and to
support separate memory spaces (processes) for file based clist
> 7. The project name to be changed to
> "delayed page rendering".
> The appearance of the word "async" in gdevprna.c
> is misleading and must be removed.
We already agreed to this, in fact this thread is and ALWAYS HAS
BEEN "Page (clist) queuing - Separate Rendering thread(s)".
> 8. An additional advantage of this design
> is no change to the language switch build
> and other top level makefiles.
> That's all. I believe that the pipe and flush()
> provides enough synchronization,
> which is pretty portable.
I don't agree -- the main problem is that these methods don't
provide for the writer interaction needed for memory full or
queue full blocking, nor do they account for the reader and
writer having different gs_memory allocators when the clist
files are memory based. In this case the writer MUST perform
the actual 'free' call to its allocator, therefore it must
be notified (messaged) by the queue controller when a page is
to be deleted.
> Moreover, pipe exactly matches the
> needs of the suggested functionality,
> i.e. it doesn't set any extra constraint.
> We will ship the technology,
> and customers will be able to
> create various solutions with it.
> BTW, gp_wsync.c assumes too much about
> the underlying OS, so it should't go to production
> due to poor portability.
TBD. If other basic gp* level functionality is needed, we can
handle this during implementation. As far as portability, the
gp_wsync API has been supported on all primary OS's (Windows,
Mac and unix) as well as some RT execs and VMS. If we need
some extensions or choose to deprecate some calls, then that's
fine with me. This (IMO) is a design detail, and not part of
what we need to decide at this time.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the gs-devel