| | <<<Back 1 day (to 2016/01/07) | 20160108 |
| Robin_Watts | Martchus: I'd guess that arch linux have removed the copy of lcms2 that we bundle. | 00:00.34 |
| | instead they are probably expecting you to have a system version installed. | 00:00.58 |
| | we don't support that. go whinge to arch linux, or use the sources that we supply. | 00:01.15 |
| tor8 | mvrhel (for the logs): if you want to look things up by, say, an md5-sum we have a few structures you can use: fz_tree (a self-balancing binary search tree, keyed on strings) and fz_hash_table (keyed on fixed length keys) | 00:41.47 |
| | mvrhel (for the logs): we also use tend to prefer linked lists to growing arrays of pointers -- they're simpler to write in c. | 00:42.35 |
| | and don't require as much boiler plate code | 00:42.46 |
| | for walking the page tree, I'd just add some iterator functions: pdf_obj *pdf_first_page_object(ctx, doc), pdf_next_page_object(ctx, doc, page_obj) | 00:43.51 |
| | if that's easier than doing a callback dance | 00:44.29 |
| | just doing a walk of the page tree is fairly trivial, the looking for resources does a lot of extra stuff (looking through xobjects etc recursively) | 00:45.26 |
| | s/looking for resources/looking for transparency/ | 00:45.38 |
| chrisl | Martchus: TBH, our configure script does (yet) support cross compiling very well. You'll have to get as close as you can with configure, and then edit the Makefile by hand | 07:23.23 |
| tor8 | Robin_Watts: cleaned up commits on tor/master with fixes for issues you mentioned yesterday | 12:05.12 |
| Robin_Watts | tor8: look good to me. | 12:48.25 |
| | tor8: So i'm looking back at my JNI code. | 12:48.36 |
| | I'm revising it so that it avoids passing java objects to the native code as much as possible. | 12:49.00 |
| | i.e. for each Matrix we pass 6 floats, for each java object that wraps an fz_ object, we pass the long etc. | 12:50.00 |
| | We still need to pass strings/arrays as java objects. | 12:50.21 |
| | and for devices implemented in java we have a choice to make. | 12:50.53 |
| tor8 | I was thinking of taking your code and adapting it to use the c mirroring ffi.java class style. should make it easier to extend things in the future, if we can just create a thin ffi.java wrapper for each function and then do the magic in java to expose java-like interfaces. | 12:51.00 |
| | right, but how about returning rects and matrices? | 12:51.16 |
| | and the fz_stroke_state struct | 12:51.34 |
| Robin_Watts | For returning rects and matrixes we still need to create java objects and return them. | 12:51.52 |
| tor8 | we could do the c style of passing arrays/rects as java objects that the c code mutates | 12:52.03 |
| Robin_Watts | The StrokeState java object just has an fz_stroke_state pointer. | 12:52.44 |
| tor8 | right, but unpack the Matrix fields when calling the native function so the unpacking happens in java rather than by calling (*env)->GetFloatField etc | 12:52.53 |
| | right, so you wrap that. need to take care with the refs there. | 12:53.16 |
| | since the -2 magic number for having them on the stack | 12:53.25 |
| Robin_Watts | tor8: For calling from java to C, I unpack the matrix fields in the java, yes. | 12:53.26 |
| tor8 | that's a lot of arguments :/ | 12:53.47 |
| Robin_Watts | tor8: Yes. | 12:53.53 |
| | but you can't have it both ways :) | 12:54.04 |
| | Essentially, I'm hoping that what I'm ending up with now, is equivalent to your ffi thing. | 12:54.28 |
| tor8 | I made a start at packing/unpacking stroke states like I did for rects and matrices | 12:54.39 |
| | but then I bumped into the dash array | 12:54.44 |
| Robin_Watts | (but without having all the native stuff forced through a single class, which I dislike) | 12:54.45 |
| tor8 | I rather like that single class approach... | 12:55.21 |
| Robin_Watts | I don't. It's like a hernia. | 12:55.45 |
| tor8 | it's foreign stuff we're calling; I don't like pretending we have classes in C when we don't. | 12:55.48 |
| | and the single class approach should allow code generation to write the bulk of it from a simple template given how regular the code ends up | 12:58.01 |
| Robin_Watts | The ffi gains us nothing, is an extra lump in the code that everything calls through, and loses us the ability to ever do anything with the 'this' pointers. | 12:58.02 |
| | tor8: I don't believe we'll ever get to the simple template point. | 12:58.28 |
| | We'll never find a tool that properly copes with wrapping exceptions etc. | 12:58.43 |
| tor8 | I think they would, with a 1-to-1 function call mapping | 12:59.02 |
| Robin_Watts | So if you accept we're never going to get there, we're making life hard for ourselves for no benefit. | 12:59.08 |
| tor8 | the exception handling and clean-up is trivial and repetitive for the ffi.c code I wrote | 12:59.40 |
| | you mention doing stuff with the 'this' pointers; that worries me | 13:00.18 |
| | for each argument, depending on type either unpack long into a pointer, unmarshal a jobject into a stack-struct, or convert the jobject into a heap struct | 13:01.54 |
| | then fz_try, call the function, fz_always clean up the heap structs, fz_catch and jni_rethrow | 13:02.12 |
| | return would do the reverse of the argument | 13:02.30 |
| Robin_Watts | tor8: Right, but none of that is 'standard' in any tool other than mupdf. | 13:03.08 |
| | hence we are not going to find a 'standard' tool to create this for us. | 13:03.27 |
| tor8 | we just need to parse and have snippets of code for the different possible types. it's been too long since I looked at SWIG, but I was hoping that could either do the job or I'd have to write my own tool | 13:03.38 |
| Robin_Watts | You'd have to be talking about writing our own template generator code, and that seems utterly pointless and a massive waste of time to me. | 13:03.56 |
| | We're going to write the interface once and be done with it, barring occasional minor tweaks. | 13:04.15 |
| tor8 | yes, copy & paste is pretty fast for now. but we're still talking about hundreds of functions in the end. | 13:04.23 |
| Robin_Watts | When we make tweaks to the API we have to change all our code to follow that, and that's not a huge burden. I fail to see why this should be any difference. | 13:04.51 |
| | At any rate, I'm finding this massively disheartening. | 13:05.07 |
| tor8 | copy & paste from a regular template type of function, which takes care of error handling and clean up; does your native bindings allow that? I think you're adding a lot of exceptions by trying to use the 'this' pointers and native jobjects. | 13:05.22 |
| | but then you did just mention you're going to start passing longs rather than jobjects | 13:05.39 |
| | so maybe I should just wait and see what you come up with | 13:05.51 |
| Robin_Watts | Either I'm writing the JNI interfaces, in which case I'll just get on and do it, or I'm not, in which case I'll go do something else. | 13:05.56 |
| | tor8: I'm *trying* the long passing to see how it turns out. | 13:06.15 |
| | It's not a panacea for all ills, and at the moment I'm thinking that the mix of having to do some stuff in C and some in java is just ugly. | 13:06.46 |
| tor8 | Sorry. I don't mean to step on your toes; but I'm also curious to see how the various approaches are going to look like. | 13:07.02 |
| Robin_Watts | Doing it all in the C gives much more regular looking code to my eyes. | 13:07.12 |
| | If we DO adopt the long passing, then that doesn't stop the C having to create java objects. | 13:08.28 |
| tor8 | well, I doubt performance is ever going to be critical here so passing a jobject and GetLongField-ing the 'ptr' should look fine | 13:08.30 |
| Robin_Watts | (for return values, certainly). | 13:08.55 |
| tor8 | but then you'll need to figure out which class each C function belongs to, something that I fear might be more difficult once you start getting into more obscure functions | 13:09.01 |
| | given your preference for that, and your intense dislike of the ffi C mirror class | 13:09.49 |
| Robin_Watts | tor8: Every java reflection of the C functions is going to have to belong to a class. | 13:10.04 |
| tor8 | then it would make more sense to wrap each struct in a jobject | 13:10.09 |
| | and create pure java copies of pass by value stuff like rects and matrices | 13:10.31 |
| Robin_Watts | so, the underlying native call can go in that same class. | 13:11.13 |
| | It has to go somewhere. | 13:11.17 |
| | Either a call will not be making use of a 'this' pointer (in which case it can be static, and it makes bugger all difference where it is, other than the naming of the internal function that's invisible to anyone but us), or it WILL be using the this pointer, in which case it's obvious which class it can live in. | 13:12.36 |
| tor8 | http://pastebin.com/raw/ThzpynpF something like that? | 13:13.50 |
| Robin_Watts | yeah, that's basically what I have now, except I wrap the PTR((*env)->GetLongField(env, self, Pixmap_ptr_fid)); up as fz_pixmap_from_Pixmap(env, self) | 13:15.13 |
| tor8 | Robin_Watts: actually, if you're going to put the native functions in the java classes, I think I'd prefer passing the jobject and extracting the 'ptr' from C. | 13:15.15 |
| Robin_Watts | (which I think is more typesafe and readable) | 13:15.25 |
| | tor8: Which is what I do now, right? | 13:16.02 |
| tor8 | the question is how to deal with the fz_context | 13:16.05 |
| | yeah, with a weird dance around the constructors and finalizers though | 13:16.16 |
| Robin_Watts | The context still needs to be handled in java. | 13:16.30 |
| | cos of the threading gubbins. | 13:16.49 |
| tor8 | couldn't the C code just call a java method to get it? | 13:16.59 |
| | do the call hidden from C rather than put it on all java classes | 13:17.14 |
| Robin_Watts | (well, I could do it in the C, but it'd require callbacks from C to java to get it, which is icky if we don't need it) | 13:17.14 |
| | every API call would transition from java -> c, c->java, return java->c, return c->java. | 13:18.10 |
| | i.e. doubling the amount of time lost in the jni layers. | 13:18.22 |
| tor8 | fair enough | 13:18.27 |
| | a shame we can't have macros in java to hide the rather long "Context.getContext()" call... | 13:18.56 |
| Robin_Watts | the lack of a preprocessor in java drives me potty. | 13:19.20 |
| | more for #ifdef stuff than for macros, but... | 13:19.33 |
| | The SOT stuff all uses unifdef to get us that back. | 13:19.59 |
| tor8 | Robin_Watts: doing it in C means you could dispense with the *Native double functions | 13:20.17 |
| | now you have getPage which calls getPageNative with the context | 13:20.32 |
| Robin_Watts | tor8: yeah. | 13:20.42 |
| tor8 | if you extract the context from C (and performance overhead be damned) you could just make getPage native and save a *lot* of typing | 13:21.05 |
| Robin_Watts | But I see that as a boilerplate regular thing with no surprises. | 13:21.06 |
| | yes. | 13:21.15 |
| tor8 | is there a way to associate user data with JNIEnv? | 13:21.33 |
| | the JNIEnv is per-thread | 13:21.40 |
| Robin_Watts | You mean, handle the context entirely within the C, lose the context class. Nice idea. | 13:24.08 |
| | That would have the effect of making the java classes very thin indeed. | 13:24.27 |
| tor8 | yes. | 13:24.40 |
| Robin_Watts | if possible, I like that a lot. | 13:24.47 |
| | let me investigate that. | 13:24.59 |
| tor8 | my thought is to either keep the C thin (the ffi single class approach) or the java thin (by having each java class basically be a grab bag of native functions with only the constructor in java | 13:26.02 |
| | my preference was for keeping C thin (because writing boilerplate C is more tedious than writing boilerplate java) | 13:26.34 |
| Robin_Watts | tor8: Can finalisers be native? | 13:27.21 |
| | The Context class gives us somewhere to set things like 'AA level' etc. | 13:28.03 |
| tor8 | I don't see why not, they're really just functions like any other | 13:28.27 |
| Robin_Watts | so we'd need a different class to affect that sort of thing. | 13:28.39 |
| tor8 | make a grab bag MuPDF with public static functions to set context style settings? | 13:29.20 |
| Robin_Watts | (i.e. we'd lose the need for Context.getContext, but we'd still need to have a Context (or System, or Core, or Config or some better name) kicking around. | 13:29.20 |
| | yeah. | 13:29.35 |
| tor8 | and set the store size, etc | 13:29.38 |
| Robin_Watts | I wonder if Config would be most appropriate. | 13:29.51 |
| | names can change easily. | 13:30.04 |
| tor8 | Robin_Watts: also, JNI_OnLoad could look up all the method and field id's | 13:31.15 |
| | and unref them on JNI_OnUnload | 13:31.23 |
| Robin_Watts | no unreffing required. | 13:31.39 |
| | lookups don't take references, they are just numbers. | 13:32.04 |
| tor8 | well, if you want to keep the jclass around you need to make a reference | 13:32.13 |
| | but yes, for the method/field ids no need to unref | 13:32.21 |
| Robin_Watts | yeah, I don't keep the jclasses around. | 13:32.29 |
| tor8 | well, you might need them for running constructors to return jobjects | 13:32.59 |
| Robin_Watts | I'm *sure* I've had to do the 'make a new JENV for this thread' thing somewhere, but can I find it in the code? | 13:33.08 |
| tor8 | I think the JVM creates and attaches a JNIEnv when you spawn threads in java | 13:33.37 |
| | if you create your own threads, you have to do the Attach/Detach thing | 13:33.52 |
| Robin_Watts | yeah. I've had to do that somewhere. Oh, probably in SOT. | 13:34.13 |
| tor8 | Robin_Watts: crazy idea... | 13:37.59 |
| Robin_Watts | yeah, was in SOT. Sanity restored. | 13:38.31 |
| | tor8: go on... | 13:38.38 |
| tor8 | make every native wrapper java class inherit from a class that has a context pointer through thread local storage | 13:38.40 |
| | probably just insanity, there's probably a catch with how threadlocal stuff works in java | 13:39.05 |
| | nah, ignore that. stupid idea. | 13:39.18 |
| Robin_Watts | Interesting idea, but I think driving thread local storage through the JNIEnv would be easier/nicer/faster (if possible :) ) | 13:40.15 |
| tor8 | are JNIEnv's stable pointers? we could have a global list where we associate each jnienv with a fz_context | 13:40.20 |
| | if we can't attach custom data to the JNIEnv | 13:40.30 |
| Robin_Watts | Ok, all threads in java are pthreads, so we can use pthread_setspecific | 13:41.48 |
| | The only question is where do we store the 'base' context ? | 13:42.03 |
| tor8 | is that true of desktop java on windows? | 13:42.24 |
| | shouldn't matter, we can #ifdef that come that day | 13:42.45 |
| Robin_Watts | Yeah, some #ifdeffery that should be restricted to the 'getContext' call in the C layer. | 13:46.43 |
| | I can store the base context in a static probably. destroy it in JNI_unload. | 13:47.02 |
| tor8 | sounds good. | 13:48.23 |
| Robin_Watts | ok, will give that a go. | 13:48.49 |
| | sorry for having a minor handbag earlier. | 13:48.55 |
| tor8 | morning mvrhel_laptop | 14:25.02 |
| mvrhel_laptop | morning tor8 | 15:35.22 |
| | just read you comments | 15:36.05 |
| | I will go with the link lists | 15:36.26 |
| Robin_Watts | Just to be awkward... | 15:36.51 |
| | linked lists can't be easily binary searched. | 15:37.13 |
| | maybe that's not an issue here. | 15:37.23 |
| mvrhel_laptop | True. But I don't think that is an issue. There is no real compare to make a decision | 15:37.49 |
| | unfortunately | 15:38.02 |
| Robin_Watts | just thinking that if we are going to be asking 'is this image in the resources already' a lot, then search time may be significant. | 15:38.04 |
| | search on the md5s. | 15:38.11 |
| mvrhel_laptop | I suppose. then we would have to keep them sorted | 15:38.24 |
| Robin_Watts | but then we'd have to make insertions shuffle things up. | 15:38.27 |
| tor8 | if this is for looking up existing resources, search on md5s using either fz_tree or fz_hash_table | 15:38.29 |
| Robin_Watts | ooh, I'd forgotten fz_hash_table | 15:38.56 |
| tor8 | fz_tree does good with both insertion and deletion... or well, it would if I had bothered to add a deletion method | 15:39.07 |
| | I have one lying around in another piece of code somewhere | 15:39.17 |
| Robin_Watts | splay trees FTW! | 15:39.32 |
| tor8 | fz_hash_table is bad for deletion, but I gather that won't be much of an issue | 15:39.34 |
| Robin_Watts | I think go with whatever is simplest now. | 15:40.03 |
| | Let's not prematurely optimise. | 15:40.15 |
| | (I'm forever guilty of that) | 15:40.26 |
| tor8 | the fz_tree stuff is taken directly from mujs | 15:40.27 |
| mvrhel_laptop | although, it could be that someone does want to delete an image | 15:40.45 |
| tor8 | where I use it for object properties, which need to be searched quickly | 15:40.49 |
| | and that one has a delete function we can trivially copy over | 15:41.00 |
| mvrhel_laptop | so is fz_hash_remove_fast / fz_hash_remove a delete from the hash table? | 15:43.09 |
| tor8 | fz_hash_remove is a delete, yes | 15:43.53 |
| mvrhel_laptop | ok. I will look this over. These are the kinds of things lurking around in the code that need to learn about | 15:44.32 |
| Robin_Watts | tor8: How does fz_hash_remove work? | 15:44.47 |
| mvrhel_laptop | correctly according to the comments | 15:45.54 |
| tor8 | fz_hash_table is a linear probe thing, so hash collisions get dumped at the next available slot instead of being chained into a list | 15:45.55 |
| | so it's awkward because it has to scan to the next hole in the hash table and rearrange entries if needed | 15:47.00 |
| Robin_Watts | yeah, I'm intrigued as to how you rearrange entries. | 15:47.25 |
| tor8 | "it's complicated" | 15:47.42 |
| | do_removal in source/fitz/hash.c | 15:47.49 |
| Robin_Watts | Whenever you make a hole, you look to see if the next one should be relocated into it. If so, you move it in there, and then check for that as the hole. | 15:48.07 |
| | yeah, ok, that should work. | 15:48.14 |
| | interesting. | 15:48.18 |
| tor8 | something like that, yes | 15:48.20 |
| | it's been a decade or so since I wrote that bit of code... | 15:48.34 |
| Robin_Watts | Urgh. For finalize, I need to call super.finalize. | 16:53.00 |
| tor8 | really? we don't inherit from any classes that have finalizers do we? | 16:53.27 |
| Robin_Watts | All objects potentially have finalizers. | 16:53.39 |
| | Good practice says you should always call super.finalize. | 16:53.54 |
| tor8 | I think we can rely on java.lang.Object not requiring one | 16:53.58 |
| Robin_Watts | OK. | 16:54.03 |
| tor8 | but yes, you're correct in theory | 16:54.18 |
| | and then you need the try { cleanup } finally { super.finalize() } dance | 16:54.38 |
| Robin_Watts | I bet there is not a nice way to call super.finalize from the JNI. | 16:56.04 |
| | There is a way, but it's hairy, so I'll ignore it. | 16:57.51 |
| HenryStiles | wow tor8 are you sleeping too? | 16:58.44 |
| tor8 | HenryStiles: eh? | 16:59.17 |
| HenryStiles | it looks like you've been cranking out commits at a pretty good pace | 16:59.49 |
| tor8 | HenryStiles: gotta make up for slacking off most of the time ;) | 17:00.20 |
| mvrhel_laptop | tor8: just looking for the fz_hash stuff. So the *val entry in fz_hash_entry_s is the opaque pointer that will point to what ever stuff I have for that entry type. Would I still have my linked list also in addition to the fz_hash_table ? Can I get away with just having the fz_hash_table(s) | 17:06.49 |
| | I am trying to see why I can't just have the table | 17:07.10 |
| tor8 | just the table is fine. | 17:08.31 |
| | the linked list comment is more in terms of general data structure use in mupdf | 17:08.44 |
| mvrhel_laptop | ok. thanks | 17:08.50 |
| tor8 | we use linked lists in preference of arrays for lists | 17:09.10 |
| mvrhel_laptop | good to know. That is what I usually prefer too | 17:09.25 |
| tor8 | given the absence of useful generic containers in c, a single linked list is the easiest alternative | 17:09.37 |
| mvrhel_laptop | Yes. Now, to rewrite everything I did yesterday.... | 17:09.57 |
| tor8 | so you'll see tons of them throughout the mupdf code | 17:10.03 |
| Robin_Watts | tor8: fz_outline should have an fz_document pointer inside it. | 18:36.16 |
| | In the same way that fz_page has an fz_document, and fz_annot has an fz_page etc. | 18:36.32 |
| | OK, so fz_page doesn't have an fz_document, but pdf_page etc does. | 18:37.37 |
| tor8 | why does fz_outline need a fz_document inside it? | 18:46.59 |
| | Robin_Watts: ^ | 18:47.03 |
| Robin_Watts | It doesn't. | 18:47.25 |
| | Ignore me, sorry. | 18:47.29 |
| | I was thinking that to traverse an fz_outline you used iterators like we did for annots etc, and those needed pages. | 18:47.57 |
| | so, sorry, ignore me. | 18:48.11 |
| tor8 | yeah, the fz_outline is created wholesale at load time | 18:48.28 |
| | I might need to revisit how links and linkdests are done for epub links | 18:48.46 |
| Robin_Watts | tor8: So, the one nasty thing about this stuff is that I can't find any docs on how to define constructors in JNI. | 18:51.11 |
| | So we're still going to have: public Path() { nativePath = newNative(); } etc in the java. | 18:51.38 |
| tor8 | it might be enough to mangle the "<init>" method name? | 18:51.53 |
| Robin_Watts | How do I define a C function with <init> in the name? :) | 18:52.12 |
| tor8 | public Path(long ptr) { this.ptr = ptr; } should do the trick no? | 18:52.13 |
| | Robin_Watts: one sec | 18:52.18 |
| Robin_Watts | tor8: No, I want people to be able to create a java Path object from java. | 18:52.47 |
| | so java programmers will call p = new Path(); | 18:52.58 |
| tor8 | Error: modifier native not allowed here | 18:53.08 |
| | Robin_Watts: oh, right. | 18:53.23 |
| | so I don't think you can :/ | 18:53.32 |
| Robin_Watts | Other than constructors, everything is in the JNI now, pretty much. | 18:53.47 |
| | I'll polish it a bit and put it up for you to poke at. | 18:54.02 |
| tor8 | public static native long createNative(); private Path(long ptr) { this.ptr = ptr; } public Path() { this(Path.createNative()); } | 18:54.45 |
| | so yeah, same as you just wrote above | 18:55.02 |
| Robin_Watts | Why 2 constructors there? | 18:56.05 |
| tor8 | one for private use that just sets the 'long' | 18:56.15 |
| | one for public use that hides the ugly guts | 18:56.25 |
| Robin_Watts | why not just public Path() { ptr = createNative(); } | 18:56.26 |
| tor8 | because you can call the Path(ptr) one from JNI to create a wrapper Path object | 18:56.49 |
| Robin_Watts | oh, right, yes, I need to do that too. | 18:57.11 |
| tor8 | but yeah, no real need for one to call the other | 18:57.13 |
| | but you still want both constructors | 18:57.24 |
| Robin_Watts | Ok, sounds like we're in agreement. Ta. | 18:58.06 |
| tor8 | I'm not sure what's best, createNative a class method or init method on the object | 18:59.12 |
| | you know about 'javah' right? | 18:59.20 |
| | takes a java class and spits out the header file with all the mangled function names | 18:59.38 |
| Robin_Watts | yeah. | 18:59.44 |
| | I was aware of it, but I wasn't using it. | 18:59.53 |
| tor8 | back in a bit, installing an extra disk... | 19:10.02 |
| HenryStiles | el nino is really improving my shoveling. | 19:21.23 |
| Robin_Watts | tor8: OK, robin/jni2 has the revised jni stuff in. It builds. Haven't tested it. See what you think of the style. | 21:10.45 |
| | HenryStiles: You here? | 23:59.00 |
| | Forward 1 day (to 2016/01/09)>>> | |
IRC Logs