| | <<<Back 1 day (to 2014/10/30) | 20141031 |
| rayjj | UnderSampled: Sorry I didn't see your response. | 00:04.05 |
| | UnderSampled: I don't know what you mean by "cl.exe crash" | 00:08.24 |
| Robin_Watts | chrisl: Is requiring a trailing slash a "feature" or a "bug" ? | 12:02.29 |
| kens | Sounds liek a bug to me. | 12:05.11 |
| | Though I do notice that -sICCProfilesDir also requires a trailing directory delimiter | 12:05.33 |
| chrisl | Robin_Watts: I have no idea - I'd say it was a bug, but...... | 12:21.56 |
| | It's *just* about possible that CPSI device paths are expected to have a trailing slash..... It probably wouldn't be hard to "fix" | 12:22.49 |
| | Whether it's intended or not, I do feel it very, very crap..... | 12:23.40 |
| Robin_Watts | I bet it's just that some of our code has an "if it doesn't have a trailing slash, add one", and some of it doesn't. | 12:24.19 |
| chrisl | Robin_Watts: It is clearly the code to handle the %device% syntax in the search path code that doesn't append a slash - I believe that is totally separate from the ICCProfilesDir handling | 12:34.09 |
| kens2 | Yes it is separate, I didn't mean to imply they were the same, merely that we have other stuff that works the same way. | 12:35.14 |
| avih | tor8: i'm having a crash where js_freestate tries to free unallocated chunks. it's being called only once (i added a print inside js_freestate). crash callstack shows about 10 freed strings and the crash happens at js_defaultalloc . any idea what to explore? | 13:52.26 |
| | fwiw, i had registry with some userdata while calling to freestate | 13:53.31 |
| | also tried deleteregistry for the single userdata object i had there just before freestate, still crashes | 13:59.31 |
| tor8 | avih: no idea. if you can create a reduced test program that exhibits the crash that would help immensely. | 14:00.10 |
| | but free errors in js_freestate are usually problems with the garbage collector not marking everything as it should | 14:00.49 |
| avih | tor8: i would have, but it's currently inside some complex code. so that would take non trivial effort. | 14:01.13 |
| | tor8: oh, indeed, i freestate if i encounter an error at the script | 14:01.30 |
| | (which if fatal, so shutting down the js) | 14:01.47 |
| | so, in case of errors, what workaround i could do? maybe pop all the objects at the stack before freestate? or some other sequence which should make freestate safer? | 14:02.31 |
| tor8 | the js_State context struct holds a linked list of *every* bit of GC memory ever allocated so it can free it | 14:02.40 |
| | if that gets corrupted, you've got big problems | 14:02.51 |
| avih | right, so what could corrupt it? | 14:03.02 |
| tor8 | memory overwrites, buffer overruns, that sort of thing | 14:03.16 |
| | the only serious problems it could cause is if the gc thinks an objects is unused and frees it, while there are still other places holding pointers to it | 14:03.49 |
| | but that would crash elsewhere, not in the js_freestate call | 14:03.57 |
| | so if js_freestate crashes, that means the linked list of objects contains garbage data | 14:04.28 |
| avih | the crash happens at js_defaultalloc | 14:04.32 |
| | at the free(ptr) line | 14:04.44 |
| | or would you cosider it also freestate because it's called from freestate? | 14:05.24 |
| tor8 | avih: yeah, so it's trying to free data which doesn't exist (or is double freed) which should never happen | 14:05.28 |
| | all js_freestate does is run through the linked list of objects and frees them | 14:06.11 |
| avih | we're in agreement on the "should" part. question is, what tools i have to debug it? maybe expose some js* functions to help me debug it? | 14:06.19 |
| tor8 | have you tried valgrind yet? | 14:06.28 |
| avih | no | 14:06.33 |
| tor8 | do that. | 14:06.36 |
| avih | i _should_ but i'm unfamiliar with it tbh | 14:06.50 |
| tor8 | it's the #1 debugging tool for spotting memory errors and using uninitialized data | 14:06.54 |
| | just do "valgrind ...command line..." | 14:07.01 |
| | it'll run your program as normal, but very slowly, and print out any sort of memory misbehaviour with a small stack trace at every location | 14:07.35 |
| avih | "Error: valgrind @3.10.0 is only compatible with Mac OS X 10.5, 10.6, 10.7, 10.8 and 10.9" | 14:07.50 |
| tor8 | "valgrind ./build/mujs test.js" f.ex. | 14:07.54 |
| avih | (10.10 here) | 14:08.01 |
| tor8 | avih: see, you shouldn't have upgraded ;) | 14:08.13 |
| avih | heh | 14:08.18 |
| tor8 | do you have access to a linux machine that exhibits the same crashes? | 14:08.29 |
| avih | no, all my code currently is on osx. i got a windows machine but the video player i'm embedding into is a b*tch to compile on windows, so osx it is for this project | 14:09.15 |
| tor8 | right. apple might have something similar to valgrind, but really, valgrind is so good that it's reason enough to run linux rather than anything else. | 14:09.48 |
| avih | tor8: any suggestions for good old prints or some state debugging apis at mujs? | 14:11.13 |
| tor8 | valgrind would tell you whether the crash you see from free() is due to a double free or garbage memory, and if so, where it was freed the first time, etc | 14:11.29 |
| | avih: run js_gc() manually at various locations and see if that crashes | 14:12.07 |
| | it also loops through the same data structures as js_freestate so should crash at the same spots | 14:12.38 |
| avih | k, let me try to reduce the case. luckily, at the c code, it's not too far after newstate, so i could go backwards try to find out which stuff causes it | 14:12.48 |
| | (at though after adding some 10 new c functions, running a js file, etc, but at the c-level, not far) | 14:13.38 |
| | thx. will let you know once i get something. | 14:14.00 |
| tor8 | avih: good luck! memory errors can be quite the chore to debug. | 14:14.30 |
| avih | aye aye | 14:14.38 |
| | thx :) | 14:14.43 |
| | tor8: hopefully this is the same crash as i got with splice... | 14:15.02 |
| tor8 | running js_gc is my best suggestion, it does run through all of the mujs data structures so any corruption should be visible there | 14:15.13 |
| avih | indeed, i took it with both hands :) | 14:15.23 |
| zeniko | tor8: does the script for updating the standalone jbig2dec repo run nightly or only on demand? | 14:31.26 |
| | tor8: also, do you remember if there's been a particular reason for decoding JPEG2000 streams all at once instead of streamed like JPEG? | 14:34.44 |
| | we've encountered some documents consisting of huge JP2 images which would benefit from decoding at lower resolution during rendering instead of at full resolution during interpretation | 14:35.33 |
| chrisl | zeniko: JPEG2000 has to be decoded all once - unless they have multiple resolution levels which is *extremely* rare | 14:36.08 |
| zeniko | chrisl: do you know whether openjpeg is able to downscale during decoding | 14:37.14 |
| chrisl | zeniko: I don't know for sure, but I do not think so. | 14:37.55 |
| | zeniko: on the Ghostscript end, we've vaguely discussed looking into whether openjpeg could be coaxed to produce the image in bands, simply discarding anything outside the band - it would be slow, but might be useful in memory usage extremis. But nothing's been done to follow up on it | 14:39.29 |
| zeniko | looking at the opj_dparameters struct, it seems that the API only allows to discard higher resolutions (which given your first comment wouldn't help) | 14:39.58 |
| | Adobe Reader handles such documents so much faster and efficiently | 14:41.21 |
| chrisl | It is possible you have one of the rare cases with multiple resolution levels..... | 14:42.30 |
| avih | tor8: so my sequence is: main: js_pcall(c-func1 with userdata object as argument): set registry to the userdata and add some c-functions, -> js_call (c-func2): loadstring (succeeds and gc runs fine), pushglobal, js_call(J, 0) -> error while running -> main's cpcall returns non-0, freestate -> crash | 14:43.03 |
| zeniko | chrisl: I guess looking into it might still be worth a shot then, thanks for the hint | 14:44.34 |
| kens2 | It hsould at leats be possible to determine if the image has multiple resolution levels | 14:45.08 |
| | I think one of the boxes tells you that | 14:45.22 |
| avih | tor8: where "[p]call(c-func)" is pushglobal, newcfunc(c-func), [push arguments if needed), js_[p]call(J, <num-args>) | 14:45.49 |
| | hmm.. i've been trying that with the debug branch with the line info. let me see if it also happens with master... | 14:47.02 |
| chrisl | I properly *hate* sourceforge...... | 14:47.41 |
| kens2 | has no experience thereof. | 14:47.53 |
| | But I did just manage to undo a Git screwup all on my own :-) | 14:48.07 |
| chrisl | Woohoo - kens2 reaches git guru status ;-) | 14:48.29 |
| avih | tor8: yup, identical crash with git-master | 14:48.33 |
| kens2 | thinks more like Git-not-quite-a-noob status | 14:48.52 |
| tor8 | zeniko: the jbig2dec repository conversion script runs nightly | 14:48.56 |
| | zeniko: I can trigger a run manually if you need it, though | 14:49.44 |
| chrisl | It actually does look like openjpeg might support multiple resolution levels...... | 14:49.51 |
| nsz | sourceforge.net is a spam site, the ui is conciously designed to maximize pain (human attention time spent in front of their ads) | 14:49.57 |
| kens2 | chrisl, that *might* be useful, if we can identify which one we need at interpretation time | 14:50.18 |
| | For GS that is of course. | 14:50.23 |
| avih | tor8: anything in my sequence sounds fishy? | 14:50.37 |
| chrisl | nsz: I'm trying to get ssh access to a project - followed the instructions, but it keeps timing out | 14:50.46 |
| | kens2: we'd also have to mess with the matrix in the PS world | 14:51.06 |
| kens2 | And the width/height in the image dictioanry | 14:51.19 |
| | But since we already tinker with the colourspace.... | 14:51.30 |
| zeniko | tor8: the most recent jbig2dec commits in ghostpdl are from 34 hours ago, shouldn't the script have copied these last night then? | 14:51.55 |
| tor8 | avih: not immediately. have you got the code online somewhere, I could take a quick look | 14:51.58 |
| | zeniko: it should. let me check what's gone wrong. | 14:52.14 |
| avih | tor8: not yet, due to the occasional crashes | 14:52.16 |
| chrisl | kens: I still feel it would be more effort than it was worth - unless it's a feature that gains more traction | 14:54.35 |
| avih | tor8: hmm.. here's some interesting info. if the string i try to run starts with some reference error, like |x = xxx;|, then the error is intercepted at the pcall and free state works fine, but if that line appears right after entering function scope, i.e. |(function(){x = xxx; ... })()| then it it still get trapped by pcall but then freestate crashes | 15:00.30 |
| | (where xxx is an unknown identified) | 15:01.48 |
| | tor8: you think you got enough info to look into it? | 15:02.45 |
| tor8 | where exactly are you calling freestate? | 15:02.55 |
| avih | at main, after pcall returned non-0 | 15:03.09 |
| | see the sequence above | 15:03.31 |
| tor8 | avih: have you tried to isolate said sequence into a simple test program? | 15:03.52 |
| avih | tor8: no, but now i can. will let you know | 15:04.06 |
| tor8 | there's not a lot I can do without being able to run the actual code | 15:04.12 |
| avih | ok | 15:04.30 |
| | i'll try to reproduce it with main.c | 15:04.39 |
| tor8 | zeniko: chrisl: okay, so for some reason git subtree now creates the commits with different checksums than before | 15:05.29 |
| | and as such refuses to clobber the existing 'jbig2dec' split out branch | 15:05.44 |
| | yeah. the old git subtree reformatted the commit messages to add line breaks whereas the current git subtree doesn't... | 15:08.19 |
| chrisl | Oh, that's... poor :-( | 15:08.52 |
| tor8 | that's effing useless! | 15:08.59 |
| chrisl | That too.... | 15:09.13 |
| tor8 | chrisl: I guess the behaviour changed when git-subtree got mainlined into the git distribution | 15:11.43 |
| | if I run my old git-subtree shell script that was used until git was upgraded on casper, I get the old results with the new code | 15:12.13 |
| | sitting on top of the old branch, just like it should | 15:12.24 |
| avih | tor8: got it. http://pastebin.mozilla.org/7027540 | 15:48.53 |
| tor8 | chrisl: zeniko: okay, I've retuned the script to use the old version of git-subtree and the jbig2dec repo should now be up to date | 15:51.33 |
| | avih: thanks. I'll look into it. | 15:51.45 |
| avih | tor8: can you reproduce? | 15:51.53 |
| | (at this sample i doesn't actually get to freestate at all, i.e. doesn't even print the error) | 15:52.21 |
| tor8 | avih: yeah, it looks like a try/catch cleanup error | 15:52.34 |
| avih | tor8: cool. thx. | 15:52.41 |
| tor8 | avih: there's a missing js_endtry in Fp_toString :( | 15:55.59 |
| avih | tor8: why the sadly? fixing this can potentially fix tons of issues... | 15:56.33 |
| | and it's a damn easy fix too... | 15:56.46 |
| tor8 | which means you jump back to a stack frame that has already been passed or reused... causing lots of havoc | 15:56.51 |
| avih | yeah | 15:57.01 |
| tor8 | avih: the sadly because it's an embarassing and stupid mistake to make :) | 15:57.07 |
| avih | well.. good thing you can fix it :) | 15:57.18 |
| | easily ;) | 15:57.22 |
| tor8 | avih: yeah. thanks for spotting this one! | 15:58.41 |
| | or well, narrowing down the error case, if not the actual spotting :) | 15:58.53 |
| | I've pushed a fix | 15:59.14 |
| avih | tor8: np. i really hope this is the cause for my other occasional/splice crashes as well... | 15:59.22 |
| tor8 | and a quick grep over the source doesn't show up any more unmatched try/endtry cases | 15:59.30 |
| | avih: I doubt it, unless you're converting function objects to strings | 15:59.57 |
| avih | tor8: any chance you could polish the debug patch and push it to master at some stage? | 16:00.00 |
| | i'm not converting functions to strings. iirc. probably not. | 16:00.20 |
| tor8 | avih: I'm not happy enough with how the debug patch works | 16:00.33 |
| avih | hmm.. it's _really_ useful... | 16:00.49 |
| tor8 | it will give the wrong error if you get an error in a C function, etc | 16:00.57 |
| avih | yeah | 16:01.06 |
| tor8 | wrong location* | 16:01.07 |
| | so I'm going to need to mull it over | 16:01.20 |
| avih | sure. | 16:01.25 |
| tor8 | hopefully come up with a more robust debugging stack trace mechanism | 16:01.38 |
| avih | but when it does work, which is most of the time for normal js files, it's really useful. | 16:01.52 |
| tor8 | but that means I need to devote some serious thinking time to MuJS and I'm a bit swamped in other projects at the moment | 16:02.15 |
| avih | and it also crashes if the first error is compile error and there's no prior line info | 16:02.25 |
| tor8 | avih: yeah, that crash should be a trivial fix | 16:02.35 |
| avih | which i workedaround by first running a string ";0;" | 16:02.43 |
| tor8 | just initialize the curfun/curline when creating the js_State | 16:02.49 |
| | avih: I've pushed a fixed version of the debug commit to tor/debug that shouldn't crash if you get an error before running javascript code | 16:04.56 |
| avih | tor8: cool. did you also rebase it with recent master changes? | 16:05.33 |
| tor8 | yes. | 16:05.39 |
| avih | nice. thx. | 16:05.44 |
| rayjj | chrisl: Thanks for doing the jbig2dec png fixes. One comment on the jbig2dec commit. I'm not sure what version of libpng got rid of CVT_PTR, but you may want to bracket the #define CVT_PTR inside #ifndef CVT_PTR -- definitely optional. | 16:12.13 |
| | chrisl: I am not sure what CVT_PTR was used for | 16:12.49 |
| chrisl | rayjj: I'll check it out | 16:13.02 |
| rayjj | probably some sort of pointer conversion :-) | 16:13.18 |
| avih | tor8: the update (at master) fixes the sample crash bug indeed doesn't fix the maybe-splice-related one. maybe some day i'll be able to isolate it, for now i'm using my own function instead of splice which seems to work. where work == "only" very rare crashes which i wasn't able to isolate yet, | 16:13.55 |
| | (when i use splice it happens after few seconds of running js stuff) | 16:14.40 |
| tor8 | avih: how are you calling splice? | 16:15.12 |
| avih | (many js related functions and LOC executed till the crash) | 16:15.12 |
| tor8 | try calling js_gc() directly after using splice, see if that triggers the crash | 16:15.42 |
| avih | tor8: timers.timers.splice(i, 0, timer); | 16:16.13 |
| | hmm... ok will do | 16:16.32 |
| | where: var timers = {nextId: 1, timers: []}; and timer is an object with few numbers and reference to a callback function | 16:17.22 |
| rayjj | marcosw: (for the logs). I had the IBM people cc you on the email about their linux event. There might be free food, but otherwise I told her that we pretty much know about linux. About the only thing that might be relevant is "Explore how to leverage the IBM Power Development Platform, allowing you nochargeremote access to IBM Hardware." (in case that's useful to us for support) | 16:19.15 |
| tor8 | avih: yeah, nothing suspicious there | 16:19.27 |
| | avih: the 'add = top - 3' line is counting how many extra arguments there are (this, index, howManyToDelete are the 3 to ignore) | 16:23.59 |
| avih | tor8: yeah, i followed it, and the top-3 thingy. didn't try to follow the actual shifts though | 16:24.29 |
| tor8 | avih: they are tricky, but there's nothing there that should be able to cause any form of state corruption--the functions used are all public safe functions without any gotchas... | 16:25.19 |
| | if there were bugs in has/set/delproperty we'd be in real trouble. | 16:26.01 |
| mvrhel_laptop | bbiaw. need to run to the school | 16:28.03 |
| avih | anyway, calling gc right after splice makes it crash way earlier now. good thing. the crash happens at the 4th splice call, which is the first which operates on an array with length 1 and insert at index 0. the 3 prior calls which didn't cause crash at the following gc are (for array.length and insertAt): 0,0, 1,1, 2,2, and then crash after 1,0 | 16:29.32 |
| | tor8: ^ | 16:29.37 |
| | hmm.. this gc thingy is a great debugging feature. now i can try to reduce the STR for this crash as well. | 16:31.42 |
| | (the array length changed at other places which is what made it at length 1 at the 4th call) | 16:33.01 |
| Shelly_ | chrisl: kens: caught the tail end of your chat about max resolutions in jpeg2000, you can change cp_reduce in opj_set_default_decoder_parameters to set the max resolution level it will decode | 17:38.11 |
| kens | Shelly hi. | 17:38.33 |
| chrisl | Shelly_: Yeh, stumbled across in the docs - maybe one day..... | 17:38.39 |
| Shelly_ | np, you know where to find me if you have any interesting work to do....catch you guys later. | 17:41.52 |
| chrisl | see yah | 17:42.01 |
| avih | tor8: splice crash: http://pastebin.mozilla.org/7033967 | 23:11.02 |
| | tor8: just noticed that if i remove the 'print' line then it becomes much more rare, even with gc. but i still saw it crash once with "TypeError: called object is not a function" on the line with the 2nd splice, and the crashes which happen at my own code have identical symptoms (crashes with segfault 11 or freed obj was not allocated or suddenly js objects get messed up, and i also saw bus error and iirc also segfault 10) | 23:31.48 |
| | i also modified the implementation of jsB_print { js_tostring(J, 1); js_push_undefined(J); } (i.e. no output) and it crashes the same. i then changed the js print to print(i) and still crashes. it stopped crashing when i changed to print(). something is extremely weird and inconsistent.. :/ | 23:47.13 |
| | also print(3) doesn't crash, but print(i) does crash. | 23:48.56 |
| | Forward 1 day (to 2014/11/01)>>> | |
IRC Logs