Log of #mupdf at irc.freenode.net.

Search:
 <<<Back 1 day (to 2018/03/21)20180322 
sebras tor8 (for the logs): do the three first on sebras/master patches look reasonable to you?04:34.11 
  I'04:34.14 
  m thinking of reintroducing the algo/steps comments to the encryption code and checking that each step is done.04:34.45 
  before introducing the last two on sebras/master, even though both you and robin LGTMed them I still feel you are both hesitant about them.04:35.14 
velix I'm still not able to build #mutools on Windows. My system is full of MinGW's trash... hundrets of versions.06:25.20 
  Does anyone know the guy, who's building the official binaries of mutool?06:25.38 
  I mean, somebody must have done it before.06:25.44 
  MY GOD06:33.09 
  It finally worked.06:33.11 
  I can't believe it.06:33.14 
  I've built Mutools in Cygwin native!06:33.23 
  muuuuuh!06:33.30 
velix does the cow dance06:33.37 
  it works (tm)07:05.17 
  kens: Did you read the good news?07:59.09 
kens Err, not yet07:59.18 
  whatever it is07:59.23 
velix I was able to build mutool in Ghostscript using its oen GCC ;)08:01.13 
  Finally, after 5 days.08:01.17 
kens Oh, OK08:01.27 
  Umm how do you build mutool 'in' Ghostscript ?08:01.54 
velix d'oh08:02.30 
  s/Ghostscript/cygwin08:02.37 
kens Ah08:02.42 
velix kens: Actually, porting mutool to PS and compiling it in Ghostscript might be a challenge.08:03.02 
  :D08:03.16 
kens Well Ghostscript' PDF interpreter is written in PostScript08:03.29 
velix Can it play doom? ;)08:04.18 
kens Can you put Doom into a PDF ?08:04.32 
velix kens: https://www.doomworld.com/idgames/utils/misc/dmpsmu3208:04.41 
  ;)08:04.42 
kens And anywya, no, because Ghostscript isn't interactive08:04.51 
velix hehe, I know. Just trolling.08:04.58 
  kens: I like the font-to-path function of Ghostscript a lot.08:06.16 
kens All fonts are paths, at the lowest level08:06.35 
velix kens: You know what I mean ;)08:06.46 
kens Actually that's not quite true08:06.46 
  There do exsit (still) bitmap fonts08:06.54 
velix Need to debug this. Played a lot with pure PDF in the last days. Very nice.08:07.34 
tor8 velix: the official binaries for windows are built using visual studio.11:11.48 
youcanlinux_ Any chance of Artifex providing SHA1 checksums for MuPDF, as it does for Ghostscript, GhostPCL, and GhostXPS? Please and thank you.13:59.04 
mvrhel_laptop Robin_Watts: so in updating the thirdparty stuff in mupdf, git gives me complaints about freeglut14:58.50 
Robin_Watts mvrhel_laptop: how so?14:59.07 
mvrhel_laptop It can't read from the remote repository14:59.13 
  that was the error reported :)14:59.53 
Robin_Watts What command are you giving it ?15:00.01 
mvrhel_laptop submodule update --init15:00.27 
Robin_Watts cd thirdparty/freeglut15:00.55 
  git remote -v15:00.58 
mvrhel_laptop hmm it says /home/mvrhel/repos/third-party-freeglut.git does not appear to be a git repository15:01.13 
  oh15:01.18 
Robin_Watts You probably initially cloned from your private repo.15:01.46 
mvrhel_laptop why would it get it from there and not golden15:01.55 
  yes15:02.00 
  how do I reset that15:02.15 
Robin_Watts Submodules are specified relative to the main repo.15:02.17 
  2 possible solutions.15:02.22 
  1) on casper, ln -s /home/git/thirdparty-freeglut.git /home/mvrhel/repos/thirdparty-freeglut.git15:03.08 
  or 2) on your local machine: cd thirdparty/freeglut && git remote rm origin && git remote add origin mvrhel@ghostscript.com:/home/git/thirdparty-freeglut.git15:04.08 
mvrhel_laptop I am having network issues. brb15:10.03 
  blah this is so screwed up.15:18.26 
  I have to head out for a bit. will continue with this when I get back15:26.19 
youcanlinux_ No reply on my checksums inquiry above ?15:57.25 
malc_ youcanlinux_: ask sebras, Robin_Watts or tor8 directly16:23.50 
  they tend to get caught up in things and miss stuff that happens here16:24.04 
Robin_Watts youcanlinux_: It seems a reasonable request. I'll try to remember to mention it to tor8 and chrisl.16:24.37 
chrisl It'll have to be tor8, I have nothing to do with the normal mupdf release16:26.16 
youcanlinux_ malc_ Thank you16:27.21 
  Robin_Watts, thank you16:27.41 
malc_ youcanlinux_: you are welcome16:27.50 
youcanlinux_ chris1, thank you.16:38.56 
sebras Robin_Watts: did you ever hear back from them? http://git.ghostscript.com/?p=thirdparty-libjpeg.git;a=commitdiff;h=8942057756dedd70030e8072b92303006d56a2bb;hp=cae42f6ea01ac85921e50dc8bb63ec34b6ab857317:49.37 
Robin_Watts sebras: I remember sending a problem to IJG, that might have been this one.17:51.53 
  They did the fix a different way in a way that doesn't fix it completely.17:52.17 
sebras yes, that's what I think have happened too.17:52.44 
  how did you know what DEQUANTIZE() places to patch?17:53.18 
Robin_Watts IIRC the issue was that the DC component can drift arbitrarily out of range.17:53.36 
  cos it's always send as an offset from the previous one.17:53.52 
sebras I think it is only the ones with the 0 in DEQUANTIZE(inptr[DCTSIZE*0], quantptr[DCTSIZE*0])17:53.58 
Robin_Watts No *sane* file should ever have it outside the 0..1023 range, but we had an example where it went further.17:54.24 
  They fudged it by extending a table lookup IIRC.17:54.35 
  sebras: It's the DC ones that need patching (i.e. entry (0,0))17:55.19 
sebras ok.17:55.35 
Robin_Watts I could (in theory) construct a file that goes wrong with their code.17:56.21 
sebras thanks for explaining, that helped me patch this into v9c. :)17:58.29 
Robin_Watts No worries, thanks for handling that.17:58.59 
sebras though after reading https://en.wikipedia.org/wiki/Libjpeg it seems to me libjpeg might not be the best jpeg library.17:59.01 
Robin_Watts sebras: Possibly not anymore, but it is ubiquitous.17:59.18 
sebras Robin_Watts: seems like people are arguing that it might not produce standards compliant files.18:00.00 
Robin_Watts Only if you use "smartscale", right? (whatever that is)18:01.39 
sebras I hope so, and possibly that requires some define or something.18:02.53 
  Robin_Watts: seems like libjpeg-turbo is where most development happens today.18:06.03 
Robin_Watts sebras: yes.18:07.34 
sebras Robin_Watts: thoes looks reasonable to you, yes? https://pastebin.com/raw/H4w9DERN18:11.42 
  I could have merged each tag, but I felt that this was unnecessary.18:12.06 
  I have also verified that PDF that you originally fixed is fixed with my merge.18:12.37 
  not sure how to test this in the cluster though.18:12.49 
Robin_Watts sebras: Yeah, looks great.18:17.39 
mvrhel_laptop So I have the origin for freeglut set to /home/git/thirdparty-freeglut.git18:31.26 
  Robin_Watts ^^18:31.34 
  and still it fails18:31.38 
Robin_Watts git remote -v18:31.53 
  I have:18:32.15 
  origin robin@ghostscript.com:/home/git/thirdparty-freeglut.git (fetch)18:32.25 
  origin robin@ghostscript.com:/home/git/thirdparty-freeglut.git (push)18:32.27 
mvrhel_laptop me too18:32.37 
  except no robin18:32.45 
  mvrhel instead :)18:32.52 
Robin_Watts but mvrhel@ghostscript.com:/... right ?18:33.00 
  Ok.18:33.14 
mvrhel_laptop yes. mvrhel@ghostscript.com:/home/git/thirdparty-freeglut.git18:33.24 
  for fetch and push18:33.31 
Robin_Watts What does cd thirdparty && git fetch do ?18:33.39 
  cd thirdparty/freeglut.git && git fetch18:33.54 
  I mean.18:33.56 
mvrhel_laptop wtf18:34.54 
  could not resolve ghostscript.com18:35.08 
  oh18:35.12 
  crap18:35.14 
  sorry18:35.26 
  misspelled18:35.31 
Robin_Watts no worries. We've all done that :)18:35.36 
mvrhel_laptop I had ghostscipt18:35.40 
Robin_Watts ghostsrcipt is one of my favorurites.18:36.03 
mvrhel_laptop :)18:36.08 
fredross-perry Robin, r u back? Take a look at yesterday's discussion with sebras and me, regarding fz_clear_pixmap_with_value.18:36.39 
Robin_Watts fredross-perry: Will do. Just a mo.18:36.49 
mvrhel_laptop ok . now it is happy18:37.31 
fredross-perry no rush18:37.35 
Robin_Watts mvrhel_laptop: So this is what I have in mind:18:37.57 
  https://pastebin.com/kkTjHqn318:37.58 
  oops: https://pastebin.com/WeQ0vQrL18:39.18 
  fredross-perry: Right. looking now.19:19.09 
fredross-perry ok19:22.09 
Robin_Watts fredross-perry: OK, so I've read through.19:23.21 
  and I'm looking at the code and it looks OK.19:23.31 
  Which bug?19:23.50 
sebras Robin_Watts: the only thing I discovered was if AndroidBitmap_lockPixels() fails we don't handle the exception.19:24.10 
Robin_Watts sebras: Right.19:24.21 
fredross-perry 699048. muso testing has produced a bug that I can't repro, where the memset that's inside fz_clear_pixmap_with_value crashes.19:24.41 
Robin_Watts fredross-perry: So, clear_cmyk_bitmap also contains a memset, and is called from that routine in a way that means it might be inlined...19:27.04 
  but I can't see that we'd be using a cmyk (or spots) bitmap.19:27.18 
fredross-perry I don't think we are (in that I tried stopping there in the debugger, and didn't).19:27.39 
sebras could it be tied to android version?19:27.53 
fredross-perry You mean the version of Android on the user;s device?19:28.18 
Robin_Watts Given that it crashes on a specific device, I'd be tempted to give credence to the idea that it's something going wrong with the bitmap lock call.19:28.18 
  If it was me, I'd add some logging around the locks, and in this routine to output w/h/stride/ptr.19:28.59 
  That way when it crashes I would have evidence to be confused with :)19:29.15 
fredross-perry I could simulate a failing lock and see what happens.19:29.22 
Robin_Watts fredross-perry: If the lock fails, the ptr is (in technical terms) bollocks, and you'll see exactly this crash.19:29.52 
fredross-perry Right.19:30.10 
Robin_Watts it's the kind of thing that might happen when a device is running close to the maximum of its memory (or its texture memory or something).19:31.04 
sebras Robin_Watts: how about abort() if the lock fails? that way it will be _VERY_ visible in any backtraces.19:31.14 
Robin_Watts sebras: That'd do.19:31.22 
fredross-perry He's also reporting a crash atnew_draw_device as well.19:31.25 
  Why might the lock fail? Insufficient memory?19:32.03 
sebras has anyone run addr2line to figure out what the +offsets points to after the symbol names?19:32.27 
Robin_Watts fredross-perry: That would be my guess.19:32.41 
  When we 'lock' a bitmap, it means it has to be mapped into user accessible memory, and cannot be moved.19:33.16 
fredross-perry addr2line - I have not.19:33.28 
Robin_Watts Normally, I would expect that bitmaps would be in memory anyway, so it's really just a message to the gc system to say "don't move this one until I say otherwise".19:34.09 
sebras Robin_Watts: also the memory must be continuous when locked, but that might not be the case when it is not locked..?19:34.36 
Robin_Watts but it's not outside the bounds of possibility that there is something screwy going on with bitmaps being in special memory that is shared with the GPU on the phone.19:34.40 
  sebras: True.19:34.55 
  (but I hope unlikley)19:35.07 
  or unlikely.19:35.17 
fredross-perry But, we lock the entire bitmap, ur just the part we're rendering to?19:35.49 
Robin_Watts We have to lock the whole thing.19:36.07 
  That's all Android gives us an API for.19:36.23 
fredross-perry right19:36.40 
  so I can give the tester a version that does abort() if the lock fails, and see what that turns up. And additionally some logging.19:37.20 
Robin_Watts yeah, sounds good.19:37.28 
sebras for this particular phone there are at least four different SoCs being used in the specs I found.19:37.29 
fredross-perry ok thanks.19:37.32 
sebras so even getting a similar device might not reproduce the issue.19:37.39 
Robin_Watts As to a crash in new_draw_device, that sounds very odd.19:37.57 
fredross-perry his device is Galaxy J7 with 16 GB internal memory and 1.5 GB RAM.19:38.30 
sebras Robin_Watts: indeed. running addr2line on that one might help.19:38.35 
Robin_Watts Hmm. new_draw_device should be static.19:39.26 
sebras fredross-perry: I know, but that doesn't help (still two different SoCs in the 2015 version)19:39.58 
Robin_Watts Yeah, same named phones in different regions can have radically different hardware cos of different radio specs :(19:40.55 
  If he's lucky there will be model number somewhere (for instance my LG G3 is the D855 variant)19:41.31 
sebras Robin_Watts: in this case Qualcomm or Samsungs own.19:41.32 
fredross-perry how does that knowledge help us?19:42.03 
Robin_Watts Well, in getting one locally, I guess.19:42.38 
sebras Robin_Watts: also he mentions that some files were OK, but those files were all single page document indicating that less memory may be used perhaps..?19:42.39 
Robin_Watts or in seeing whether this is a known issue with other phones based on the same GPU.19:43.04 
  yeah, I wonder if it could be something like "no more than 2 bitmaps can be locked at a time" or something, and we're hitting that in the threading or something.19:43.42 
sebras it's a possiblity.19:44.14 
fredross-perry he said that they are ALL single page.19:44.38 
sebras Robin_Watts: my suggestion would be to put the abort() in the code to reproduce it so we clearly see what is happening without having the device.19:44.44 
Robin_Watts "The crash is not reproducible with single page document."19:45.05 
sebras Robin_Watts: but if this _is_ the problem, then the lock callback must return an int to indicate error, and the only way the device callback can react is by returning (with the java exception already having been thrown). this will fz_throw() back to the java-device which will then react accordingly.19:45.52 
Robin_Watts X works, O doesn't, right?19:45.54 
fredross-perry oh - 'and all of them were single page documents' probably means just the three.19:45.58 
  I think O = crash, X = ok19:46.36 
  yes19:46.43 
Robin_Watts new_draw_device is called from other functions like fz_new_draw_device.19:47.29 
  It can fail if we run out of memory, but that's all.19:47.41 
  and if we run out of memory, we fz_throw.19:47.55 
fredross-perry 'put the abort() in the code to reproduce it' - just unconditionally abort?19:48.02 
Robin_Watts I wonder if we're not in an exception stack at that point...19:48.12 
sebras fredross-perry: perhaps ndk-stack might help? https://developer.android.com/ndk/guides/ndk-stack.html19:48.22 
Robin_Watts fredross-perry: No... check the return value from lock()19:48.49 
  if (AndroidBitmap_lockPixels()) abort();19:49.37 
sebras fredross-perry: I think ndk-stack may help with the new_draw_device() backtrace.19:49.43 
fredross-perry hmm.19:49.55 
  So, where it throws in androidDrawDevice_lock(), just abort() instead?19:51.10 
Robin_Watts Yes.19:51.17 
  There are at least 2 places.19:51.24 
  Ah!!!!19:52.03 
fredross-perry the other is in AndroidImage_newImageFromBitmap19:52.05 
  Ah!!??19:52.08 
Robin_Watts jni_throw doesn't actually throw.19:52.14 
  It just sets a flag in the environment.19:52.22 
fredross-perry right.19:52.40 
Robin_Watts so we leave androidDrawDevice_lock, and the caller doesn't know.19:52.45 
  so it continues onwards, and runs into the crash.19:52.57 
  OK, so this is all consistent at least.19:53.02 
sebras Robin_Watts: that is what I wrote yesterday...19:53.05 
Robin_Watts sebras: sorry, I failed to see that.19:53.15 
sebras Robin_Watts: maybe I didn't explain it clearly enough. :)19:53.17 
Robin_Watts But you were right :)19:53.22 
sebras Robin_Watts: sorry about that. :)19:53.23 
Robin_Watts So we could make androidDrawDevice_lock return an int maybe...19:53.41 
sebras Robin_Watts: at least now we are two who semi-independently came to the same conclusion.19:53.47 
  Robin_Watts: check my message from 8 min ago. :)19:54.12 
  Robin_Watts: but that only papers over the hard crash and replaces it with a java exception.19:54.48 
  Robin_Watts: that doesn't make it _work_, which is the real problem.19:55.03 
Robin_Watts sebras: It means the caller can at least degrade nicely.19:55.07 
  Like maybe the caller can defer the redraw and try it later.19:55.23 
sebras Robin_Watts: yup. better than nothing.19:55.26 
Robin_Watts It could flush the store and retry or something.19:55.50 
fredross-perry I can try that but again, since I can't reproduce this, it's speculative.19:56.09 
sebras Robin_Watts: also I coun't see a case where we might fail to unlock and then run out of locks..?19:56.17 
Robin_Watts We can add the abort, and the logging, and see if silver sees an abort.19:56.30 
fredross-perry 'flush the store' ?19:56.33 
Robin_Watts If he doesn't then we know we're barking up the wrong tree.19:56.43 
  fredross-perry: In MuPDF we have a cache we use, called the store.19:57.00 
  It holds decoded versions of things like images.19:57.13 
sebras Robin_Watts: agreed.19:57.22 
Robin_Watts If we try to malloc something, and that fails, we evict stuff from the store, and retry.19:57.42 
  But if the system tries to malloc stuff (like for bitmaps) and fails, we can't do the same thing there.19:58.18 
  Does that make sense ?19:58.27 
  (we can't do the same thing *automatically*)19:58.42 
fredross-perry understood.19:58.55 
  OK, I'll do some abort/logging and send him something. I'll also see about reproducing with some of the other files.20:00.10 
  thanks gents20:01.37 
Robin_Watts no worries.20:03.46 
sebras Robin_Watts: I have uploaded sebras/test-libjpeg-v9c, where I have repointed Makethird to use the patched v9c that I put in another directory.20:05.28 
  Robin_Watts: this causes ~1500 diffs in the cluster.20:05.49 
Robin_Watts git cluster bmpcmp -t 3220:06.14 
  (oh, urm, clusterpush.sh bmpcmp -t 32 probably for you)20:06.35 
  (I have a git alias to start cluster jobs)20:06.53 
sebras Robin_Watts: I do to, but what is -t 32?20:07.02 
Robin_Watts run the bmpcmp with a threshold value of 32.20:07.18 
  i.e. it ignores any diffs where the pixels differ by <= 32.20:07.46 
sebras Robin_Watts: also libjpeg has changed the definition of FALSE/TRUE/boolean so there might be something that needs doing for windows.20:08.59 
  Robin_Watts: if you look at the cluster I have two runs running right now, one bmpcmp -t and one 32... but I just ran "git cluster bmpcmp -t 32" like you said..?20:10.05 
Robin_Watts sebras: ass.20:10.18 
  32 is a magic number.20:10.40 
  abort those and try again with 31 :)20:10.51 
sebras Robin_Watts: how do I abort?20:11.05 
Robin_Watts clusterpush.sh abort20:11.10 
sebras Robin_Watts: seems better, but now I need to wait for shelley. :)20:13.27 
Robin_Watts yeah, sorry. I blame marcos for using '32' to mean "do a 32bit build".20:14.16 
sebras Robin_Watts: oh.20:14.27 
Robin_Watts so... newNativeAndroidDrawDevice20:16.39 
  That makes a pixmap with a dummy pointer.20:16.56 
  Then locks, then clears the pixmap, then unlocks, then makes a new draw device.20:17.44 
  If the lock fails, the pixmap ptr will be duff, and possibly pointing to something on the stack.20:18.24 
  So maybe the clear might succeed by luck, but then the stack might be corrupted, which might cause the fz_new_draw_device to crash.20:18.50 
  so this could all be the same problem.20:18.56 
sebras Robin_Watts: the dummy pointer is stack allocated.20:21.28 
  Robin_Watts: why can't we just have pix->samples be == NULL when it is not locked?20:21.43 
Robin_Watts The pointer is on the stack, yes.20:21.48 
  How would that help? :)20:21.54 
  (other than maybe being easier to spot in a debugger)20:22.33 
  (or backtrace)20:22.37 
sebras Robin_Watts: because the memset() wouldn't accidentaly scribble on the stack in that case.20:22.52 
Robin_Watts it'd still crash.20:23.00 
sebras of course, but it wouldn't be able to blame new_draw_device() in that case perhaps.20:23.20 
  Robin_Watts: that's my bet at least.20:23.34 
  Robin_Watts: and I can't see that a random value in pix->samples helps us any either..?20:23.54 
Robin_Watts It doesn't.20:24.04 
sebras Robin_Watts: actually this fiddling with the sample pointer indicates that this might be something we want proper APIs for in fz_pixmap imho.20:24.30 
  some kind of callback to lock/unlock the samples in fz_pixmap..?20:25.26 
Robin_Watts http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=686490e08588add7c199c10d5aa473e446a56cab20:25.51 
  sebras: That'd need careful thought.20:26.43 
sebras Robin_Watts: NativeDeviceLockFn needs a int return type20:27.12 
Robin_Watts http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=53d365f201f04f873013dc610fa7e15ef72a7b9e20:27.50 
  Thanks20:27.52 
  fredross-perry: I'm not in a position to trivially test that here, but it *should* do what you want.20:28.37 
  If you want to give it a whirl, let me know how you get on.20:28.57 
  I'm about to be called away for the night.20:29.03 
sebras Robin_Watts: me too...20:31.23 
Robin_Watts sebras: Thanks. have a good one.20:31.36 
sebras Robin_Watts: your patch looks reasonable to me.20:31.45 
  Robin_Watts: wait...20:33.18 
  Robin_Watts: we set ninfo->object in newNativeAndroidDrawDevice() and then we reference it in androidDrawDevice_unlock().20:34.29 
  Robin_Watts: how can we know that this pointer is not stale at that point..?20:34.43 
  Robin_Watts: oh, info->object is reset in lockNativeDevice()... I guess that means that setting it in newNativeAndroidDrawDevice() is actually unnecessary.20:35.39 
  seems like by bmpcmp is all single pixel diffs with -t 31, so I'm guessing this v9c is acceptable.20:44.05 
fredross-perry I found oneo little compile issue (return instead of return 0 in one place)21:02.53 
  it seems to be working fine. I've put my 'new AndroidDrawDevice' inside try as well.21:03.26 
  and some logging. I'll givie this over to silver21:03.37 
  Robin, sebras - SO allocates two large bitmaps for rendering, which are used throughout the life of the document. Why would I not lock them and leave them locked until the document is closed?22:53.22 
Robin_Watts My understanding is that it's very bad form to do that.23:52.11 
  I believe the idea is that you lock, update the bitmap, unlock.23:52.39 
  and the contents get flushed back to texture memory on the unlock.23:53.02 
  so until you unlock, you might not get the results showing on screen.23:53.21 
  (But I could be wrong here)23:53.41 
 Forward 1 day (to 2018/03/23)>>> 
ghostscript.com #ghostscript
Search: