| | <<<Back 1 day (to 2018/03/25) | 20180326 |
| malc_ | tor8, kens: for the logs, https://www.tug.org/TUGboat/tb34-2/tb107bigelow-wang.pdf (hinting at page 10 or there abouts) | 06:04.00 |
| | tor8: https://boblycat.org/~malc/scratch/shot-go-mono.png | 11:04.43 |
| Robin_Watts | tor8, sebras: I wrote this after the discussion with jogu and co about the desire for more platform support: https://twiki.ghostscript.com/do/view/MuPDF/ChangeBuildSystem | 12:11.53 |
| | If you can see anything I missed, please let me know. | 12:12.12 |
| sebras | reads Robin_Watts wiki page. | 12:36.46 |
| Robin_Watts | updated it to be clearer. | 12:42.19 |
| sebras | Robin_Watts: just to be clear, does "solution" anywhere imply "VS solution"? (them having reused that word is annoying) | 12:44.44 |
| Robin_Watts | sebras: No. | 12:44.57 |
| | I've tried to say "VS solution", when I mean "VS solution", but I may have slipped. | 12:45.21 |
| sebras | Robin_Watts: ok, what platforms does muso build for today? | 12:45.38 |
| | Robin_Watts: android only? | 12:45.43 |
| Robin_Watts | android and ios. | 12:45.52 |
| sebras | Robin_Watts: but someone is wanting for it to compile on windows and there's where the problem started? | 12:46.11 |
| Robin_Watts | and what prompted this is that they need it for windows (and maybe macos, IDK) | 12:46.13 |
| | I have no problem with us using a 'meta tool' like CMake. | 12:46.50 |
| tor8 | Robin_Watts: the gradle build file only delegates to a gnu makefile for ndk-build | 12:47.14 |
| Robin_Watts | as long as a) it's *us* that have to use it, not users, and b) that it's not a retrograde step. | 12:47.32 |
| | That's what I've tried to capture on this page. | 12:47.39 |
| sebras | tor8: note that this is being deprecated in android. so sooner or later we likely have to switch to cmake for mupdf-android-fitz | 12:47.58 |
| Robin_Watts | tor8: right. | 12:48.00 |
| | sebras: Oh, Really? | 12:48.18 |
| tor8 | Robin_Watts: the number 1 issue with our build system is the need for 'make generate' | 12:48.19 |
| sebras | Robin_Watts: really. | 12:48.22 |
| Robin_Watts | Well, I guess that's a reason to move towards CMake anyway. | 12:48.33 |
| tor8 | if we drop that step, all of the builds are trivial 'compile a list of files' with a few minimal sets of -D arguments for the various thirdparty libraries | 12:48.48 |
| Robin_Watts | tor8: That's on my list of requirements (last point) | 12:48.57 |
| | I can't believe that other projects don't have a similar thing. | 12:49.17 |
| tor8 | I'm not a huge fan of requiring CMake... | 12:49.24 |
| Robin_Watts | So surely CMake must cope. | 12:49.26 |
| | tor8: I hate the idea of requiring CMake for users. | 12:49.39 |
| tor8 | it's a big fat dependency that's a veritable pain in the ass to install and use on windows | 12:49.47 |
| Robin_Watts | but I'm sanguine about requiring it for developers. | 12:49.56 |
| | very few people (i.e. you, me, sebras, mvrhel) ever actually add new files to MuPDF. | 12:50.29 |
| tor8 | Robin_Watts: if we bite the sour apple and include the 'generate' folder in the checkin we can probably get rid of a huge headache for our more clueless users | 12:51.23 |
| Robin_Watts | and that's a relatively rare thing for us to do. | 12:51.28 |
| sebras | tor8: would it be possible to generate the windows build files using cmake on linux and checking those in? | 12:51.37 |
| tor8 | but I'm loathe to include 100Mb of autogenerated crap source files | 12:51.37 |
| sebras | tor8: it might not be the most beautiful solution, but is it possible? | 12:51.49 |
| Robin_Watts | tor8: That's the price we'd pay, yes. | 12:51.49 |
| | sebras: Yes, AIUI, absolutely. | 12:52.12 |
| tor8 | another option is to mandate "use our gnu makefile on ALL systems. here's the gnumake.exe for windows." | 12:52.23 |
| Robin_Watts | The point is that any one of us can work on whatever platform we like and not worry about breaking other platforms. | 12:52.41 |
| malc_ | tor8: 170M on my system | 12:52.42 |
| Robin_Watts | tor8: No. That'll make customers cry. | 12:53.08 |
| tor8 | malc_: yes. 100M give or take :) | 12:53.13 |
| malc_ | tor8: ... lamost twice as much :) | 12:53.24 |
| Robin_Watts | malc_: It's within an order of magnitude, and doesn't affect tor8's point. | 12:53.48 |
| tor8 | Robin_Watts: the makefiles have grown to the point of creaking... I wouldn't mind simplifying the build system | 12:54.19 |
| sebras | Robin_Watts: https://developer.android.com/studio/projects/add-native-code.html "Android Studio also supports ndk-build due to the large number of existing projects that use the build toolkit to compile their native code. [...] However, if you are creating a new native library, you should use CMake." | 12:54.35 |
| malc_ | Robin_Watts: fine.. my point was always to use objcopy... but tor's reply was always non portable | 12:54.35 |
| tor8 | but I would not like losing some of the nicer features, like being able to trivially switch between release/debug/asanitizer build | 12:54.51 |
| Robin_Watts | tor8: And even if we did do that, and we had a skeleton VS solution that called gnumake, we'd still presumably want a list of files within the solution for easy browsing. | 12:55.22 |
| | And that'd fall out of sync too (that's a probably with gs' solution, albeit a small one). | 12:55.40 |
| tor8 | Robin_Watts: if we remove 'make generate' from the build system by checking in the results, we can make trivial 'nmake' makefiles for visual studio | 12:55.43 |
| Robin_Watts | tor8: If we move, the idea would be to move so that we don't have to maintain multiple lists of files. | 12:56.09 |
| tor8 | at least then I won't break the build when we add/remove/rename files | 12:56.09 |
| | and we can use the same 'makefile' list of files for both the GNUMake and nmake files | 12:56.30 |
| Robin_Watts | Having a single list of files (in the CMake/whatever files), and then having a script that makes all the platform specific versions from that would achieve what we want. | 12:56.52 |
| | Add a file, tweak the list, run the script, check everything in. | 12:57.05 |
| sebras | as I see it is modifications to the Makefiles, usually adding/removing files, that cause compilation problems. unless we routinely build all platforms on our systems, it is automated by the cluster or cmake/script/whatever-tool always works flawlessly I forsee that I might still add a new file and it messes up on another platform. and with muso I wouldn't know until someone there reports an issue. | 12:57.37 |
| Robin_Watts | sebras: The standard failure is "add a file, modify the makefile, forget to modify the solution (or the ndk-build or something)" | 12:58.23 |
| sebras | Robin_Watts: yes. | 12:58.38 |
| | Robin_Watts: but I don't know that until you holler. | 12:58.46 |
| Robin_Watts | Under the new scheme when we add a file, modify the cmake file, and run the script to regenerate, ALL the platforms get updated. | 12:58.57 |
| sebras | Robin_Watts: if the cluster built win/ios/whatever I'd know faster and perhaps could fix it too. | 12:59.04 |
| Robin_Watts | hence it's *less* likely that we'd get breakages. | 12:59.17 |
| tor8 | I agree that the current approach of having to maintain 2 lists of files is awkward (since the gnumake uses wildcards, I don't really count it as a maintenance headache) | 12:59.26 |
| Robin_Watts | It would be nice if the cluster built win/ios etc, but that's a separate problem. | 12:59.32 |
| tor8 | actually, it's just 1 list of files. the android makefile also uses wildcards. | 12:59.49 |
| Robin_Watts | tor8: If the CMake file can use wildcards, then we'd maybe be able to avoid having any :) | 13:00.31 |
| sebras | Robin_Watts: btw, I think android's new build system is gradle+cmake instead of gradle+ndk-build | 13:00.38 |
| tor8 | Robin_Watts: if you want I could take a look at cmake and try it out for mupdf and see how much I hate it | 13:00.51 |
| Robin_Watts | tor8: If you wanted to, that'd be great. | 13:01.11 |
| tor8 | and in either case, I think we should consider simplifying the current build systems anyway | 13:01.41 |
| Robin_Watts | I know that I haven't got time immediately, and I wanted to write this to avoid anyone else going off and trying something that was unacceptable to us (me). | 13:02.00 |
| sebras | tor8: what kind of simplications do you have in mind? | 13:02.01 |
| tor8 | remove the need for 'make generate' for users (which would simplify cross compilation and the buggy 'generate.bat' that doesn't detect changes and always rebuilds on visual studio) | 13:02.14 |
| Robin_Watts | I'd be happy to take the hit and check in the generated files, personally. | 13:02.38 |
| | I'd be unhappy if windows builds DIDN'T generate automatically though. | 13:03.06 |
| | cos we'll get people (customers/users/me) making changes, rebuilding and then getting confused as to why stuff didn't work. | 13:03.40 |
| tor8 | sebras: make generate as an explicit step, less use of magic gnumake features, explicit list of source files, fewer weird targets, purge some untested/unused combinations of libraries/cross compiler targets/etc | 13:03.46 |
| | Robin_Watts: if you could get the 'generate.bat' script to actually detect and not completely break when resources/pdf/names.txt changes I'd be happy | 13:04.33 |
| | that has driven me bonkers on more than one occasion | 13:04.50 |
| Robin_Watts | tor8: I'll ponder it. | 13:07.21 |
| Robin_Watts | lunches. bbs | 13:07.34 |
| fredross-perry | robin, sebras - re: #699048. Silver produced on 'failed to lock' crash. Return code was -3, which is ANDROID_BITMAP_RESULT_ALLOCATION_FAILED. I asked him to get a few more of these, to see if that's all he's getting. | 15:54.22 |
| | s/on/one/ | 15:54.38 |
| Robin_Watts | fredross-perry: Ah, cool. | 15:54.40 |
| | That at least would seem to indicate that it's a genuine error rather than us having forgotten to unlock somewhere or something. | 15:55.14 |
| sebras | Robin_Watts: ok, so now with the new code we'll throw a java exception instead. then what? :) | 15:55.20 |
| fredross-perry | I catch it in Java and deal with it. | 15:59.30 |
| sebras | fredross-perry: what do you do? empty the store? | 16:00.41 |
| | fredross-perry: because the underlying problems is still unknown. | 16:00.55 |
| fredross-perry | I would just try it again. That would be the next step in testing I would think. Is this intermittent, or something permanent once it happens? | 16:01.46 |
| sebras | fredross-perry: one question is: if he sees this exception once, will it be returned again even if we free up memory? is there a specific type of memory we would need to free? | 16:01.48 |
| | fredross-perry: we think alike. :) | 16:02.04 |
| fredross-perry | I don't frankly know what should be freed. | 16:03.32 |
| | ... or how to determine that. | 16:03.43 |
| Robin_Watts | fredross-perry: So you only ever have 2 bitmaps, right? | 16:06.17 |
| fredross-perry | Yes. | 16:06.55 |
| tor8 | Robin_Watts: okay, so I've got a CMakeLists.txt that manages to build (with pre-generated files) on linux | 16:07.57 |
| | the makefiles it spits out are *NOT* suitable for distribution | 16:08.08 |
| | they have hardcoded system paths to compilers and all kinds of cruft in them | 16:08.19 |
| Robin_Watts | tor8: Is that soluble? | 16:08.26 |
| fredross-perry | robin: I should add that the document that's originally referred to in the bug does have a lot of images. | 16:09.34 |
| | ... and I assume that's the one he's continuing to use. | 16:09.52 |
| Robin_Watts | fredross-perry: So... if it's a general issue of "bitmap allocation may fail when memory gets low", we can think about ways to fix that. | 16:10.43 |
| fredross-perry | You mean within mupdf? | 16:11.11 |
| Robin_Watts | fredross-perry: Yeah. | 16:11.17 |
| | Just a tick and I'll try and find some code... | 16:11.29 |
| fredross-perry | All right. | 16:11.32 |
| Robin_Watts | ok, so we have various functions in MuPDF you could try to call. | 16:13.15 |
| | When an fz_malloc(size) fails, it performs a "scavenge" operation. | 16:13.40 |
| | Which means it runs through the store, freeing the oldest stuff, and retrying until it actually manages the allocation (or until there is nothing left to free). | 16:14.10 |
| | If you look in memory.c at do_scavenging_malloc, you can see that. | 16:15.02 |
| | So we could amend mupdf_native.c to do the same kind of thing around lockPixel calls. | 16:16.20 |
| fredross-perry | hmmm. | 16:20.48 |
| Robin_Watts | scribbling now. | 16:22.17 |
| jogux | I think I'd be in the 'treat the cmake output as generated files that don't go into git' camp. There's even a windows .msi for cmake, android uses it, etc. I don't think a single dependency completely kills the idea. | 16:22.29 |
| fredross-perry | robin - does it make sense to use the same lock (FZ_LOCK_ALLOC) for this? | 16:23.27 |
| Robin_Watts | FZ_LOCK_ALLOC must be taken whenever we operate on the store. | 16:23.50 |
| | so we have no choice but to take that lock. | 16:24.02 |
| fredross-perry | got it. | 16:24.06 |
| | I can see creating a 'do_scavenging_function' and pass in an int-returning function that the caller provides. | 16:24.54 |
| | ... so this can apply in different cases. | 16:25.12 |
| Robin_Watts | fredross-perry: https://pastebin.com/mgEThru1 | 16:25.22 |
| fredross-perry | Or that. | 16:27.03 |
| Robin_Watts | http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=6507657a9620a4abdda0a7828013a137535da8fe | 16:28.58 |
| | That's my first cut. | 16:29.06 |
| | oops. | 16:29.46 |
| fredross-perry | I guess I'd put that bit of business in a local function and use it in two places. | 16:29.49 |
| Robin_Watts | better version: http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=f79225ece5a5522c746977911b350a985d2fc009 | 16:30.16 |
| fredross-perry | are we confident that we'll -always- break out of that loop one way or another? | 16:30.22 |
| Robin_Watts | fredross-perry: fz_store_scavenge counts phase up as it goes. | 16:30.40 |
| | Yes, I think we're confident of that. | 16:31.20 |
| | (If not, then MuPDF would grind to a halt already in some cases) | 16:31.36 |
| | It'll take at most 16 calls to free everything freeable in the store. | 16:32.23 |
| fredross-perry | how does fz_store_scavenge use the 'size' param? | 16:32.30 |
| jogux | what if something other than mupdf tries to allocate a bitmap? | 16:32.36 |
| Robin_Watts | jogux: Then the failure is it's problem :) | 16:33.11 |
| | s/it's/its/ | 16:33.18 |
| jogux | I'd argue that's not entirely true if mupdf has stolen all the memory | 16:33.32 |
| Robin_Watts | fredross-perry: size is an indication of how much we should try to free. | 16:34.04 |
| | but as the phases go on, it has less and less effect. | 16:34.17 |
| | jogux: The alternative is to give mupdf a lower max store size on initialisation. | 16:34.41 |
| jogux | FWIW pre-nui smartoffice handled this situation by a slightly involved mechanism that set a limit on how much memory the native code could allocate, so that things in the host app would always have enough memory to do 'things'. | 16:34.52 |
| Robin_Watts | jogux: Well, the allocators that mupdf uses are passed in, so you are free to do that if you want. | 16:35.26 |
| jogux | I do feel that might be a saner way to go. | 16:36.00 |
| Robin_Watts | That requires there to me a mechanism to ask "what's the maximum safe amount of memory I can allocate" of course. | 16:36.48 |
| | which there never is. | 16:37.00 |
| jogux | yeah. That code is annoying in smartoffice. But, it did stand the test of time. | 16:37.27 |
| Robin_Watts | jogux: Of couse, smartoffice also did/does the scavenging thing too. | 16:37.47 |
| jogux | note that Fred has the added issue that he may have multiple documents loaded in his mupdf instance. | 16:37.50 |
| sebras | Robin_Watts: you could start at 0, try to render a page, if it fails, increment 1byte, then try again until you get a page or give up. | 16:38.11 |
| Robin_Watts | jogux: This is why we want all the contexts to share a store. | 16:38.20 |
| | sebras: start what at 0? | 16:38.28 |
| sebras | Robin_Watts: the maximum size of memory available to native implementation. | 16:38.44 |
| jogux | Robin_Watts: right | 16:39.02 |
| fredross-perry | you used 'things' twice. Is that like a double-negative? | 16:39.12 |
| Robin_Watts | fredross-perry: s/'things'/shizzle/. | 16:40.20 |
| fredross-perry | ;-) | 16:40.31 |
| sebras | Robin_Watts: why is implementing a different allocator better than setting a lower max size of the store at init time? | 16:40.39 |
| | Robin_Watts: unless we allocate a _LOT_ of things outside of the store, we have not reason to believe that this is the cause of the issue...? | 16:41.04 |
| Robin_Watts | sebras: The maximum store size is "the maximum amount of memory that we *try* to keep to in storable items". | 16:41.32 |
| | Which means "if we ever have more stuff in the store than the max, we bin the freeable stuff until we don't" | 16:42.04 |
| | Sadly things in the store can be "not freeable" (cos they might be in use). | 16:42.18 |
| | And there are large things in MuPDF that aren't storable (like display lists). | 16:42.38 |
| | So if you want a hard ceiling on what the app allocates, then you need to implement that yourself by fiddling the allocators. | 16:43.13 |
| | So even if you set the store max to 100Meg, you could still have 200Meg of storable objects. | 16:44.10 |
| | That'll drop back to 100Meg as soon as stuff becomes freeable, but peak usage can be higher. | 16:44.30 |
| sebras | Robin_Watts: right. | 16:44.45 |
| Robin_Watts | Now, on most sane OS, we should get allocation failures and degrade nicely. | 16:44.49 |
| | but certain OS (like iOS, I believe) will just kill you if you allocate too much. | 16:45.16 |
| | And they won't tell you what too much is. | 16:45.29 |
| sebras | Robin_Watts: that means that the custom allocators ceiling must be variable depending on whether mupdf fails to draw, then it does something magic, raises the ceiling and tries to have mupdf draw the page again. | 16:45.35 |
| | Robin_Watts: is that allocate or is that _use_? | 16:46.04 |
| | Robin_Watts: I'm thinking about overcommit in linux where malloc() succeeds, but the OOM killer takes your process out if you attempt to write to the memory. | 16:46.28 |
| Robin_Watts | sebras: No. Having a ceiling in the custom allocators means "we will fail to render (at least some elements of) a page if we hit that ceiling" | 16:46.33 |
| | sebras: I am not sure which one iOS does. | 16:46.52 |
| sebras | Robin_Watts: how do you know we will even be able to start creating the displaylist? | 16:47.16 |
| Robin_Watts | The custom allocators ceiling is a blunt instrument. | 16:47.20 |
| | sebras: Indeed. | 16:47.23 |
| sebras | Robin_Watts: we might not be able to.. | 16:47.23 |
| Robin_Watts | There is no nice solution here. | 16:47.40 |
| sebras | Robin_Watts: how do we determine if this is indeed the problem? | 16:47.54 |
| Robin_Watts | sebras: The fact that the logging from silver is saying that it's failing with -3 (bitmap allocation failed), means that I suspect this is the case. | 16:48.37 |
| jogux | iOS just kills an app if it tries to malloc too much memory. malloc rarely ever fails apart from if you allocate memory /very/ quickly or in very large chunks (megabytes) | 16:48.41 |
| sebras | Robin_Watts: could we somehow check how much memory our process uses and how many locked Pixelregions we have? | 16:48.43 |
| Robin_Watts | We only have 2 bitmaps, hence can only have 2 pixelregions locked. | 16:49.04 |
| | sebras: Potentially fred could add some logging before the lockPixels call to say what the memory status was. | 16:49.38 |
| jogux | I'm not convinced that's max 2 is always true? isn't it 4 in splitscreen mode, and potentially more if (as Fred has been playing with) we present as fragments and the app puts multiple ones on the screen. | 16:50.14 |
| Robin_Watts | jogux: Yes, but in the case where Silver has been testing it's 2. | 16:50.38 |
| fredross-perry | yes, what info would I log, then? | 16:50.44 |
| jogux | Robin_Watts: ah :) | 16:50.51 |
| Robin_Watts | My point was "it's not the case that we are producing a bitmap per thread and they aren't being tidied up fast enough" or something. | 16:51.10 |
| fredross-perry | Fragments might be an issue. Not settled on exactly how that would work. | 16:51.17 |
| Robin_Watts | I'm slightly surprised that the OS ever releases the underlying memory. | 16:51.54 |
| fredross-perry | In the multiple activities case (which Silver is not testing), there are still only two. | 16:52.26 |
| Robin_Watts | If I create a bitmap, fill it with pixels, and plot it, presumably that memory must go "somewhere" so that when I lockPixels again in future, I still have the old values in there. | 16:52.45 |
| | It's odd to me that lockPixels is failing other than the very first time. | 16:53.12 |
| | fredross-perry: Do we know if this is the very first time we're trying to lock? | 16:53.27 |
| fredross-perry | We don't, but I suspect it's not. | 16:53.39 |
| Robin_Watts | The alternative is that when we 'unlockPixels', the data is copied into GL memory or something, and our bitmap is removed. When we 'lockPixels' it must copy the data back. | 16:54.16 |
| fredross-perry | By the bug description he has to zoom out to show several pages. | 16:54.16 |
| Robin_Watts | Maybe the GL data is in a different format? | 16:54.32 |
| | Just to be clear, it is "2 bitmaps", not "2 bitmaps at a time", right ? | 16:55.45 |
| fredross-perry | When the doc is opened, we make two bitmaps. Then it's the same two for the life of the doc. | 17:02.07 |
| Robin_Watts | Perfect. | 17:02.18 |
| | Given that he has to pan out to be able to see multiple pages, he must get redraws during that time, so this can't be the first lock. | 17:05.35 |
| | If a lockPixels/unlockPixels is going to require an allocation (and hence presumably a copy or conversion), then this means it's not the cheap operation I'd hoped for. | 17:06.14 |
| jogux | errr. we don't free them when that instance goes into the background? | 17:06.22 |
| Robin_Watts | It'll have implications for the performance of a native Java device. | 17:06.38 |
| fredross-perry | jogux - yes we do. | 17:07.25 |
| | robin - that's why I asked the other day of we could lock them and leave them locked. | 17:07.51 |
| | I assume though that the performance implications are unrelated to this allocation failure? | 17:12.08 |
| sebras | fredross-perry: what is the bug number again? | 17:12.09 |
| fredross-perry | 699048 | 17:12.55 |
| Robin_Watts | fredross-perry: Yes, the performance implications don't matter here. | 17:18.01 |
| | locking and leaving them locked sounds potentially very bad to me. | 17:18.25 |
| fredross-perry | I knew you'd say that. Unless there's abig performance win, and we could do so safely. ;-) | 17:19.03 |
| Robin_Watts | I don't know that I've ever seen code that doesn't unlock before returning to the java side. | 17:20.19 |
| sebras | fredross-perry: how do I see in the log from comment 25 that we get ANDROID_BITMAP_RESULT_ALLOCATION_FAILED ? | 17:21.19 |
| fredross-perry | look for LOG_ABORT messages, you'll see return value = -3. I looked that value up in the online docs | 17:22.01 |
| sebras | fredross-perry: if I search in that file for "Abort message" I get one hit, and that line ends with -2... | 17:23.21 |
| fredross-perry | ang on... | 17:23.30 |
| sebras | fredross-perry: what am I doing wrong? | 17:23.31 |
| | fredross-perry: should I be looking at another log? | 17:23.48 |
| | fredross-perry: the reason I'm asking is that -2 == ANDROID_BITMAP_RESULT_JNI_EXCEPTION | 17:24.51 |
| Robin_Watts | E/libmupdf(26167): LOG_ABORT_1 called from androidDrawDevice_lock | 17:24.52 |
| | E/libmupdf(26167): LOG_ABORT_1 return value = -3 | 17:24.54 |
| fredross-perry | thanks RObin. | 17:25.03 |
| Robin_Watts | The first log in the bug was showing lots of -2's. | 17:25.06 |
| | The second one shows -3's. | 17:25.29 |
| sebras | Robin_Watts: ok, so wrong log. thanks. | 17:25.38 |
| Robin_Watts | np. | 17:25.44 |
| fredross-perry | This is just one result from the version that has abort(). I asked for a few more just to determine that this is what's up. | 17:25.46 |
| sebras | fredross-perry: oh, search for createSurf and you will see at some calls that SO has been expanded incorrectly at some places. | 17:28.16 |
| | doesn't matter perhaps, I don't know. | 17:28.28 |
| fredross-perry | 'expanded incorrectly' ?? | 17:34.44 |
| sebras | fredross-perry: it writes tmart instead of smart. | 17:35.05 |
| fredross-perry | Oh that. Hmm. | 17:35.22 |
| | In each of those error messages, the first letter is +1 from what it should be. Not just 'tmartoffice'. Odd. | 17:37.16 |
| sebras | fredross-perry: oh, true. | 17:37.53 |
| | Robin_Watts: after digging a bit in android it seems like the bitmsp can reside in the heap, but external (whatever that means), belong to ashmem or be hardware buffers. | 17:47.53 |
| Robin_Watts | Who is ashmem? | 17:49.24 |
| sebras | Robin_Watts: android shared memory implementation? something they stuck in the kernel I believe, but I might be wrong. | 17:49.58 |
| | https://developer.android.com/studio/profile/memory-profiler.html#how-counted seems to be able to indicate how much native memory has ben allocated and how much is GL surfaces etc. | 17:50.28 |
| Robin_Watts | sebras: Ah, so it might be a buffer shared between GL and java or something? | 17:50.37 |
| sebras | Robin_Watts: could this be a tool that would help us? | 17:50.40 |
| | Robin_Watts: it seems to me like the implementation would support a shared buffer, yes. | 17:51.03 |
| Robin_Watts | sebras: Yes, that looks interesting. | 17:51.14 |
| sebras | Robin_Watts: it can create heap dumps, but I'm not sure if that would help us (would it include native memory stats? I'm not sure) | 17:51.38 |
| Robin_Watts | sebras: I suspect that this comes down to the exact GPU in use on each phone, and the way it's driven. | 17:51.43 |
| | "Native" is listed in the screenshot. | 17:52.03 |
| sebras | Robin_Watts: perhaps, but in this case I'm more interested in getting more proof that we really are runnign out of memory as we believe. | 17:52.26 |
| Robin_Watts | sebras: There is a call to tell you some memory stats, I think. | 17:53.11 |
| sebras | Robin_Watts: true, perhaps it would be interesting to have the reporter run this (if possible)? | 17:53.20 |
| | Robin_Watts: oh, yes https://developer.android.com/reference/android/os/Debug.html#dumpHprofData(java.lang.String) | 17:53.47 |
| Robin_Watts | ActivityManager.MemoryInfo looks interesting too. | 17:54.38 |
| | https://developer.android.com/reference/android/app/ActivityManager.MemoryInfo.html | 17:54.49 |
| sebras | Robin_Watts: https://developer.android.com/reference/android/os/Debug.html#getNativeHeapFreeSize() https://developer.android.com/reference/android/os/Debug.html#getMemoryInfo(android.os.Debug.MemoryInfo) | 17:56.46 |
| | Robin_Watts: now, if we think this is the issue we should be able to reproduce this easily. | 17:57.07 |
| | Robin_Watts: render many pages from a PDF containting JPEG2000 data. | 17:57.33 |
| | Robin_Watts: those images usually cover the entire page and so ought to be BIG. | 17:57.55 |
| | Robin_Watts: ideally on an older phone (with less memory). | 17:58.15 |
| Robin_Watts | sebras: Yes, but I suspect that on most devices Bitmaps have the memory allocated when they start up, and never freed until they are destroyed. | 17:58.31 |
| sebras | Robin_Watts: notably there is nothing I find in the log that talks about memory. | 17:58.33 |
| Robin_Watts | lock and unlock just "pins" the memory so it can't be moved. | 17:58.43 |
| | So on such devices, we won't see this fail in the same way. | 17:59.14 |
| | We need a device of the same type that Silver is using to be able to reproduce this. | 17:59.34 |
| fredross-perry | Its a Galaxy J7 | 18:00.31 |
| sebras | Robin_Watts: this will lead us to buying all the crappy android devices (i.e. not the good ones). | 18:00.36 |
| Robin_Watts | sebras: Yes. I am not suggesting that we do that. | 18:00.51 |
| sebras | fredross-perry: it's a 2015 Galaxy J7 (the 2016 version has more meory) | 18:00.55 |
| Robin_Watts | I'm just saying that it's not going to be easy to reproduce here. | 18:01.07 |
| fredross-perry | 'My Galaxy J7 has 16 GB internal memory and 1.5 GB RAM' | 18:01.11 |
| | I've never reproduced it. | 18:01.18 |
| sebras | fredross-perry: I konw, he's describing the 2015 version of it' | 18:01.46 |
| | Robin_Watts: if that is the case, how do we pick a reasonable limit for the allocator? | 18:02.22 |
| | Robin_Watts: because I assume that the idea is that the app shouldn't crash. | 18:02.40 |
| paulgardiner | Just a couple of comment towards the CMake discussion: this tutorial has a section "Adding a Generated File and Generator" - https://cmake.org/cmake-tutorial/ | 18:03.24 |
| Robin_Watts | sebras: We look at ActivityManager.MemoryInfo and pick 80% of what it says? | 18:03.33 |
| | paulgardiner: Ta. Did you see tor8's comment earlier? | 18:03.51 |
| | The generated makefiles from CMake contain hardcoded paths etc :( | 18:04.02 |
| paulgardiner | Ah. Bad | 18:04.14 |
| Robin_Watts | Which kinda buggers it up for us distributing them. | 18:04.16 |
| fredross-perry | roin, sebras - perhaps what's next is to roll a version with scavenging, plus catching/dealing with lock exceptions, and see if Silver can break it. | 18:05.05 |
| | ... and some informative logging. | 18:05.21 |
| Robin_Watts | fed: Yes. | 18:05.56 |
| sebras | fredross-perry: the informative logging is probably obtained by some of the calls in the links that Robin_Watts and I posted above. | 18:06.27 |
| | fredross-perry: unclear exactly which ones, so perhaps print all of them at the abort()..? :) | 18:06.47 |
| fredross-perry | Oh. And this next version would NOT abort90, imho. | 18:07.14 |
| | abort() | 18:07.22 |
| Robin_Watts | fredross-perry: Actually, if it was me, my next version would still abort() (and log) if we fail to lock even after the scavenge. | 18:07.45 |
| | That way we can see if the scavenge is enough to solve the problem (for Silvers testing). | 18:08.12 |
| fredross-perry | What does abort() get us, please? | 18:08.13 |
| | all right then. | 18:08.24 |
| Robin_Watts | It gives us a hard stop and information if it ever fails rather than just a redraw glitch that he might miss. | 18:08.42 |
| sebras | and it is also VERY visible in the log. | 18:08.55 |
| fredross-perry | ok | 18:09.02 |
| | popping out for a bit. | 18:11.09 |
| Robin_Watts | If all else fails: https://www.ebay.com/itm/Samsung-Galaxy-J7-2015-SM-J700H-16GB-White-UNLOCKED-Smartphone/232708919838?epid=238493124&hash=item362e88521e:g:YVwAAOSw2CJatlXk | 18:12.02 |
| paulgardiner | For which platforms would it be painful to use cmake actually in place of make? I've been assuming on linux, installing cmake is just adding a package. | 18:15.50 |
| Robin_Watts | paulgardiner: No, no, no, no, no. | 18:21.50 |
| | Requirement 1: "All platforms should only require the "most native" mechanisms for working." | 18:22.31 |
| | Next time I have to port to a dev board, I don't want to have to port cmake first. | 18:22.58 |
| | and I really don't want to tell customers they have to. | 18:23.12 |
| | https://alternativeto.net/software/premake/ ? | 18:24.22 |
| | Doesn't support VS2005, but DOES support VS2008, and the differences between them are tiny, so we might be able to fix that. | 18:27.24 |
| | hmm. There is code in there to do vs2005. | 18:28.55 |
| | Ah, so maybe 2005 dropped out of the prerelease of 5. | 18:29.42 |
| paulgardiner | I was thinking maybe, on linux, cmake is becoming one of the most native mechanisms. Just a thought. | 18:30.56 |
| Robin_Watts | https://premake.github.io/ looks tempting. Need to look at it. | 18:39.11 |
| | In particular it's portable, and a single library-less executable. | 18:39.36 |
| | and it uses lua, so you'll like it :) | 18:39.47 |
| paulgardiner | I'm sold. :-) | 18:40.55 |
| tor8 | paulgardiner: cmake is *not* a build system. it's a build system generator. | 18:45.47 |
| | on unix it creates makefiles, on windows it creates visual studio projects, etc. | 18:45.59 |
| paulgardiner | I realise that but, on linux, do you have to perform the two stages explicitly? | 18:46.54 |
| Robin_Watts | tor8: premake is the same, but looks like it might be better suited to what we want. | 18:49.05 |
| paulgardiner | ... I suppose you would: you'd run cmake once and then repeatedly run make. | 18:54.39 |
| sebras | tor8: I tried your cmake file in android, and I seem to ahve to make all libraries SHARED. | 18:57.02 |
| tor8 | Robin_Watts: okay, I guess I can take a look at premake as well | 18:57.03 |
| | sebras: okay. I haven't gotten that far yet. | 18:57.20 |
| | Robin_Watts: but I am somewhat favorably impressed by what cmake seems capable of | 18:58.56 |
| | setting up the Makethird stuff was trivial, and now works better than our current makefiles | 18:59.13 |
| sebras | tor8: yeah, it only looks for .so and executables from cmake. | 18:59.20 |
| tor8 | Robin_Watts: premake seems a bit immature. the syntax for premake files changed completely between premake4 (which is what is available on my debian) and premake5 | 19:07.47 |
| sebras | tor8: I ought to be sleeping but I wanted to finish this: http://git.ghostscript.com/?p=user/sebras/mupdf-android-fitz.git;a=commitdiff;h=07cc41c05006ff9f716369df30fb76f1aab5c3b5;hp=360c83670fba2fb815df0a9aea23bf2a432f80a3 | 19:13.37 |
| | tor8: so I just crammed everything into a single shared library. | 19:13.56 |
| | tor8: but using this it builds. | 19:14.03 |
| tor8 | sebras: can you use a final shared library that links to static libraries? | 19:14.18 |
| | so that 'mupdf' is a shared library that links to the static libraries mupdf-core mupdf-third-zlib, etc | 19:14.37 |
| sebras | tor8: I'm not sure, it seemed to skip all libraries for which the type was not defined (which it was not if I set STATIC) | 19:14.57 |
| | oh and it would be the externalNativeBuild thing in the gradle plugin doing the skipping. | 19:17.16 |
| tor8 | Robin_Watts: and premake4 doesn't solve the problem of Android going to cmake in the future | 19:26.00 |
| | sebras: well, if the final target is shared it would still have to build the static libraries the shared thing has as dependencies, no? | 19:26.35 |
| sebras | seems reasonable. | 19:26.47 |
| | tor8: I just verified that using this I can still produce an .apk | 19:27.01 |
| | tor8: yes, it worked. | 19:30.54 |
| | tor8: nice. I'll separate out this into static libraries tomorrow. | 19:31.16 |
| | tor8: need to sleep. | 19:31.29 |
| malc_ | tor8: CC build/native/thirdparty/freetype/ftlcdfil.o | 19:35.30 |
| | make: *** No rule to make target 'build/native/thirdparty/harfbuzz/hb-ot-shape-complex-khmer.o', needed by 'build/native/libmupdfthird.a'. Stop. | 19:35.30 |
| | expected? | 19:35.33 |
| Robin_Watts | malc_: Do you have thirdparty/harfbuzz ? | 19:36.02 |
| malc_ | Robin_Watts: yeah | 19:36.42 |
| Robin_Watts | Are you using a new CMake thing? | 19:36.58 |
| malc_ | running submodule update | 19:37.01 |
| | Robin_Watts: nope | 19:37.04 |
| Robin_Watts | ok. | 19:37.13 |
| tor8 | malc_: are you using my master? | 19:42.38 |
| malc_ | tor8: aye | 19:44.26 |
| | tor8: after updating submodules it built | 19:44.33 |
| | and the built version even runs | 19:45.04 |
| tor8 | Robin_Watts: so, I got premake4 to build mutool | 19:45.43 |
| Robin_Watts | tor8: What does the output of the tool look like? presentable ? | 19:46.04 |
| tor8 | the makefiles it creates are much simpler than cmakes | 19:46.12 |
| Robin_Watts | That sounds good. | 19:46.40 |
| tor8 | it doesn't do transitive library dependencies, etc | 19:46.47 |
| Robin_Watts | This is where I found it: https://alternativeto.net/software/cmake/ | 19:47.09 |
| tor8 | Robin_Watts: on tor/cmake are two commits that add a cmake and premake4 build files | 19:47.42 |
| Robin_Watts | tor8: By which you mean? | 19:47.57 |
| tor8 | if you want to try it on windows | 19:48.05 |
| | or just look at how the source ends up | 19:48.21 |
| Robin_Watts | tor8: The premake files look nicely readable. | 19:49.52 |
| tor8 | Robin_Watts: it's regular lua source code | 19:50.03 |
| | it just predefines a bunch of functions like 'project' and 'kind' etc | 19:50.14 |
| Robin_Watts | right. | 19:50.19 |
| tor8 | and lua has a bit of nice syntactic sugar | 19:50.28 |
| | foo "bar" is equivalent to foo(bar) | 19:50.34 |
| Robin_Watts | We could presumably simplify them further by using wildcards. | 19:50.35 |
| tor8 | foo("bar") | 19:50.38 |
| | yeah. wildcards don't work as well for the thirdparty libraries (they tend to have junk files in there) | 19:50.52 |
| Robin_Watts | thirdparty/freetype/src/*/*.c ? | 19:51.15 |
| tor8 | freetype uses mega-files that include other c files (one 'top' c file per module that includes the other files) | 19:51.42 |
| | it's things like that we need to work around :( | 19:51.59 |
| Robin_Watts | I'll have a poke at that on windows tomorrow to see how well it works out. | 19:52.04 |
| tor8 | the premake-generated makefiles look easy enough to tweak for cross compiling | 19:53.23 |
| malc_ | lovely... xcode beta installation image has ~600M worth of fonts, mono-spaced subset = Apple Color Eoji, Menlo, Courier new, Last resort... shrug | 20:01.33 |
| fredross-perry | feh. fz_lock and fz_unlock are not in the public api (or so it seems). | 21:02.42 |
| Robin_Watts | fredross-perry: Hmm. | 23:36.28 |
| | I'll add a new scavenging function tomorrow. | 23:37.42 |
| fredross-perry | for now I copied a few things into mupdf_native.c to get it working for Silver. | 23:40.05 |
| Robin_Watts | Better version: http://git.ghostscript.com/?p=user/robin/mupdf.git;a=commitdiff;h=ae9f48228022917f2084639489be4c72e89221b7 | 23:45.07 |
| | Forward 1 day (to 2018/03/27)>>> | |
Log of #mupdf at irc.freenode.net.
