| | <<<Back 1 day (to 2016/03/08) | 20160309 |
| tkamppeter | chrisl, hi | 14:27.01 |
| chrisl | tkamppeter: Hello | 14:28.09 |
| tkamppeter | chrisl, it is about the openjpeg embedded in GS. | 14:28.47 |
| chrisl | Okay | 14:29.00 |
| tkamppeter | chrisl, in the Linux distributions one is always trying to use the system's libraries so that if a bug is discovered in a library one only needs to fix the library and not also GS. | 14:29.48 |
| | chrisl, so Debian arrived at the point not using any of GS's embedded libs any more, and I would like to sync this package into Ubuntu, to not maintain a separate GS package any more. | 14:31.01 |
| | chrisl, but as GS is in Ubuntu's fully supported "Main" part, all its dependencies, including openjpeg must be in "Main", too and for getting there it has to fulfill a certain security standard (Debian does not care here). | 14:32.27 |
| | See the results of the security audit by Ubuntu here: | 14:32.52 |
| | ttps://bugs.launchpad.net/ubuntu/+source/openjpeg/+bug/711061/comments/41 | 14:33.32 |
| | chrisl, and for this they do not accept openjpeg in Ubuntu "Main". | 14:33.58 |
| chrisl | tkamppeter: surely, this should be given to the OpenJPEG developers? | 14:35.00 |
| tkamppeter | chrisl, and I do not only want to have it in Main due to GS but also for Poppler which also is using it for reading embedded JPEG2000 images (Poppler is dropping their own solution). | 14:35.13 |
| chrisl | tkamppeter: also, I simply don't believe that openjpeg is worse than jasper for security! | 14:36.50 |
| | cd obj | 14:37.10 |
| | Oops | 14:37.14 |
| tkamppeter | chrisl, my question to you also is what are you taking from upstream openjpeg> Is it only a small part or everything? Did you modify it a lot? Clean up? Fixing of bugs? Or has the instance embedded in GS the same security flaws as the original? Could you do something with GS's build system that the embedded libopenjpeg can be built as shared lib and used by other programs, like it is done with libijs?> | 14:38.19 |
| | chrisl, and is JPEG2000 very complex? Or could a GSoC student write a completely new lib for it (only for reading it)? | 14:39.50 |
| kens2 | JPEG200 is very complex to implement, I did it once before | 14:40.03 |
| | THe specification is hard to read, and simple implementations are either slow, memory hungry, or both. Also for PDF we would not implement the full specification. | 14:40.37 |
| chrisl | tkamppeter: We pull in basically the whole library. We've fixed a few issues that could be security related, also issues with incorrect decoding, and incorrect/incomplete API. As far as I know, all our changes have been fed upstream, whether they have been accepted, I do not know | 14:41.03 |
| kens2 | It took me several months to implement the limited subset needed for a PDF implementation. | 14:41.04 |
| tkamppeter | chrisl, so you probably did not fix everything which is mentioned in my link as it is not all of openjpeg needed for PDF. As at Ubuntu we can only include a complete upstream package or do not do it, it would perhaps make sense to have a PDF-only fork. | 14:44.19 |
| | chrisl, this fork could be done by GS. | 14:44.43 |
| chrisl | tkamppeter: It could, but it won't | 14:44.54 |
| kens2 | Why would we want to invest time in this ? | 14:44.55 |
| tkamppeter | chrisl, as GS provides the shared library libijs it could also provide a libopenjpeg, which in contrary to the original contains only the PDF-relevant parts. | 14:45.43 |
| chrisl | tkamppeter: not going to happen, no | 14:46.03 |
| kens2 | We don't use openjpeg for our commercial customers, only open source. | 14:46.07 |
| | So we have no real incentive to work on openjpeg | 14:46.22 |
| tkamppeter | kens2, Ubuntu (and any other distro) needs JPEG2000 support as part of PDF interpreters. | 14:46.32 |
| kens2 | tkamppeter : but that's not *our* problem :-) | 14:46.46 |
| | I really do think that you need to talk to the OpenJPEG developers | 14:47.28 |
| Robin_Watts | Could we do the same extraction thing for libopenjpeg that we do for jbig2dec ? | 14:47.34 |
| kens2 | extraction thing ? | 14:47.47 |
| Robin_Watts | We extract jbig2dec automatically from the gs source to its own repo. | 14:48.16 |
| tkamppeter | kens2, there are GS and Poppler needing it and the only upstream maintained solution is libopenjpeg where upstream seems not to take care of security. | 14:48.20 |
| kens2 | Robin_Watts : how would that help ? THe openkpeg source is the problem | 14:48.33 |
| chrisl | Robin_Watts: the problem is the magnitude of the work to make it acceptable, and the long term implications of supporting it | 14:48.43 |
| kens2 | tkamppeter : I agree completely, the OpenJPEG team are not hugely responsive | 14:48.51 |
| | We definitely don't want to be supporting an open source JPEG2000 implementation. | 14:49.13 |
| Robin_Watts | kens2: Well, it would mean tkamppeter and co could use our extracted repo as their shared source. | 14:49.18 |
| kens2 | Robin_Watts : but the quality of the rce itself is the problem | 14:49.41 |
| | source* | 14:49.47 |
| tkamppeter | chrisl, kens2 Robin_Watts So someone more responsible should fork it. | 14:49.51 |
| kens2 | tkamppeter : but we don't care enough, it works well enough ofr our purposes | 14:50.10 |
| Robin_Watts | Ideally, yes, someone prepared to actually fix it. | 14:50.12 |
| chrisl | tkamppeter: we simply don't have the resources to do that | 14:50.18 |
| kens2 | And as I said, our commercial customers get a different library | 14:50.31 |
| tkamppeter | kens2, a closed-source one they pay a high price for? | 14:51.06 |
| kens2 | Yes | 14:51.11 |
| | Except that we have a deal for it. | 14:51.21 |
| | But its not open-source so no good to you | 14:51.33 |
| tkamppeter | chrisl, kens2 I am not a security expert to either fix it myself nor mentor a student on it. | 14:52.39 |
| kens2 | Nor is anybody here, really | 14:52.52 |
| tkamppeter | chrisl, kens2, is the JPEG2000 format something constant or do there appear new versions regularly? | 14:54.34 |
| kens2 | I've never seen a new version | 14:54.43 |
| | Just the original specification | 14:54.49 |
| | Of course, I haven't looked either, since that's the version that the PDF specificatoin uses | 14:55.07 |
| chrisl | I suspect too few people understand the original to try to author a new one! | 14:55.17 |
| tkamppeter | kens2, great, so this would make doing a fork easier. | 14:55.54 |
| kens2 | I suppose so, but I'm not privy tot he workings of the Joint Photographic Experts Group, its possible they are working on a new version | 14:56.28 |
| tkamppeter | kens2, the fork would be some PDF-only thingy. Perhaps I could get a GSoC student plus people of Ubuntu's security team as mentors. | 15:02.15 |
| kens2 | Well, we would be happy to see a 'better' JPEG200 implementation, but I thnk it would be a lot of work for a student | 15:02.56 |
| chrisl | tkamppeter: I suspect that even understanding just what PDF requires is going to be a very non-trivial project, never mind the tidying up the source, fixing the security issues etc..... | 15:03.27 |
| tkamppeter | kens2, the student should be one with a C programming security background and he should audit libopenjpeg and fix the security holes. | 15:04.08 |
| chrisl | And probably break the decoding in the process :-( | 15:04.30 |
| kens2 | OK. | 15:04.31 |
| tkamppeter | So perhaps better continue GS as it is with embedded libopenjpeg and report a bug to Poppler to make them return to their own JPEG2000 solution or go also with an embedded fork of libopenjpeg, for example ripping it from GS. | 15:06.18 |
| | Note that we need correctly working Poppler for mobile devices to avoid having both GS and Poppler. | 15:07.23 |
| | How does MuPDF do JPEG2000? | 15:08.25 |
| chrisl | OpenJPEG | 15:08.36 |
| Robin_Watts | Ah, it seems we ALREADY have libopenjpeg extracted automatically. | 15:09.07 |
| | http://git.ghostscript.com/?p=thirdparty/openjpeg.git;a=summary | 15:09.13 |
| | cos that's what MuPDF uses. | 15:09.19 |
| chrisl | Just typing that.... | 15:09.29 |
| Robin_Watts | tkamppeter: If your security guy would care to retry his audit on that, it might be interesting. | 15:10.07 |
| | We certainly haven't solved the indentation/layout etc stuff, as that would make it harder to take on any changes from upstream. | 15:10.40 |
| | But we HAVE solved some crashes (we had loads of fuzzing bugs reported to us, and Shelly fixed those crashes). | 15:11.10 |
| | And I think, broadly, we'd be prepared to take security fixes into our repo as long as a) they don't break examples in our test repos, and b) we aren't swamped with them. | 15:12.08 |
| | chrisl: Does that seem fair? | 15:12.18 |
| | +kens. | 15:12.28 |
| kens | We'd want to pass them upstream to the OpenJPEG people and try to get them adopted I'd thnk | 15:12.43 |
| Robin_Watts | kens: We would. | 15:13.01 |
| kens | But I'm reasonably sure we'd take fixes | 15:13.05 |
| chrisl | Robin_Watts: TBH, I'd really rather not be the upstream source - we have enough hassle with the URW fonts, the IJS code, etc...... but given the behaviour of the OPenJPEG devs, we may have to consider it | 15:13.36 |
| Robin_Watts | chrisl: Do you know how successful Shelly was at getting fixes passed back? | 15:15.15 |
| kens | As I recall he did (eventually) get a number taken on. | 15:15.36 |
| | The info is probably in Bugzilla somewhere | 15:15.53 |
| chrisl | Robin_Watts: Fits and starts: the devs were receptive, and took on most/all his patches, *when* they were talking.... they often went dark for months at a time | 15:16.19 |
| tkamppeter | Which version of libopenjpeg did you extract automatically? Is this the newest one (2.xxx)? | 15:16.46 |
| kens | From our git repo, we are using 2.1 | 15:17.04 |
| | 2.1.0 specifically | 15:17.22 |
| chrisl | IIRC, there were fixes on top of 2.1 | 15:17.23 |
| tkamppeter | kens, this seems to be the current one. | 15:17.30 |
| kens | Yeah but as chrisl says, we have stuff on top of that | 15:17.42 |
| | There's a list of stuff from Shelly here: | 15:18.29 |
| | http://bugs.ghostscript.com/show_bug.cgi?id=694691 | 15:18.29 |
| tkamppeter | kens, this shows that it is really difficult to communicate with OpenJPEG upstreams. | 15:22.52 |
| kens | Yes I'm afraid that has been our experience | 15:23.25 |
| tkamppeter | kens, what do you have on top of libopenjpeg 2.1? Only bug fixes or also feature additions? | 15:23.40 |
| | kens, is the API the same of as of the original? | 15:23.58 |
| kens | tkamppeter : I'm afradi I don't recall precisely. I thnk there's at least one bug fix we have that they didn't (yet) take. Beyond that I don't remember | 15:24.15 |
| Robin_Watts | tkamppeter: Our stuff is only ever bugfixes, with the exception of the allocator stuff, AIUI. | 15:24.17 |
| | The allocator stuff may well require some API changes. | 15:24.30 |
| | (basically, we want the ability to tell them not to call malloc/free etc directly, but to call allocators that we supply) | 15:24.54 |
| kens | I thougth the memory allocator was accepted. | 15:24.58 |
| chrisl | Accepted, not released | 15:25.10 |
| kens | Ah fair point | 15:25.15 |
| Robin_Watts | That bug is all about the allocator stuff. | 15:25.27 |
| kens | Robin_Watts : it covers some other stuff too | 15:25.40 |
| Robin_Watts | From the description on the bug they were balking at the API changes it required. | 15:25.43 |
| chrisl | We did change the API, but that was prior to 2.1 - currently, we use the "released" API | 15:25.53 |
| kens | Yes, but they proposed alternatives, and we agreed on one | 15:25.59 |
| Robin_Watts | kens: yes, sorry, I see that now. | 15:26.17 |
| tkamppeter | Does it mean that their GIT head snapshot would equal the version in GS/MuPDF? | 15:26.20 |
| kens | I do not thnk so, no | 15:26.32 |
| | There is at least one bug fix that we have which they have not (formally) accepted. | 15:26.50 |
| | Which doesn't mean it isn't there of course. | 15:26.58 |
| chrisl | Which one? | 15:27.07 |
| kens | Well the URL is https://code.google.com/archive/p/openjpeg/issues/235 but that doesn't work for me | 15:27.46 |
| tkamppeter | kens, I think Google has shut down code.google.com. | 15:28.23 |
| kens | The last comment from Shelly was that he'd made a pull request to them for that bug, so it might be done | 15:28.25 |
| tkamppeter | So they do not answer when they accept a request/fix from a contributor. | 15:29.13 |
| kens | As has been said, when they talk they are reasonable, but they go for months at a time without saying anything | 15:30.08 |
| chrisl | I wonder if they bothered to migrate the issues list from google code to github..... | 15:31.23 |
| tkamppeter | chrisl, so probably that way they have lost all the inquiries. | 15:32.24 |
| chrisl | tkamppeter: the google code tracker is still accessible (just read-only, I guess) | 15:32.58 |
| tkamppeter | One problem with free software is that people have ideas, start a project, and suddenly, the project makes it into the core of the OS and so responsability gets imposed to these people and they cannot or do not want to cope with it. | 15:34.19 |
| chrisl | tkamppeter: My suspicion is that the OPJ devs maintain as far and as often as their own requirements dictate, and either don't have the time, or the inclination to do more | 15:35.42 |
| mvrhel_laptop | tkamppeter: personally I think the mobile folks should drop Poppler and go with MuPDF. | 15:39.36 |
| chrisl | OMFG!! https://code.google.com/archive/p/jj2000/ | 15:40.01 |
| kens | Hmm, bet that's slow :-) | 15:40.42 |
| chrisl | Glacial would be my guess, and memory hungry..... | 15:40.58 |
| tkamppeter | chrisl, please keep Java out of the OS core, especially for mobile/ | 15:41.59 |
| chrisl | tkamppeter: I'd get hanged if I tried to add it to Ghostscript!!! | 15:42.28 |
| HenryStiles | actually it gets worse than that: https://github.com/mozilla/pdf.js/blob/master/src/core/jpx.js | 15:43.26 |
| tkamppeter | chrisl, imagine a smart phone which shoots 20 images per seconds with continuous AF and takes 20 days to print 20 images (and during these 20 days you cannot shoot more images). | 15:43.35 |
| chrisl | HenryStiles: I knew about the jpx.js - it doesn't work too well, either..... | 15:44.58 |
| tkamppeter | HenryStiles, is this the JPEG2000 decoder of Mozilla's PDF interpreter implementation in JavaScript? | 15:45.24 |
| HenryStiles | tkamppeter: right | 15:47.12 |
| chrisl | I'm not convinced that 2k lines of javascript is a complete JPEG200 decoder...... | 15:50.00 |
| tkamppeter | jpx.js are 2233 lines, can one let a student translate these to C and this would be our new JPEG2000 library? | 15:51.22 |
| | chrisl, not even good enough for PDF? | 15:52.16 |
| Robin_Watts | Can't we just use mujs? :) | 15:52.19 |
| kens | As chrisl said I doubt that is complete. It woudl also undoubtedly be slow and probably still full of security issues. And we would need the ability to have memory allocators defined byb ues etc etc. | 15:52.33 |
| chrisl | tkamppeter: the other JPX decoder that I have source for is only for PDF, is part of a library from which it can pull various utility functions, and amounts to over 6k lines of C..... | 15:53.35 |
| tkamppeter | chrisl, what is this "other JPX deccoder"? The one which comes with GS? | 15:55.02 |
| kens | No, GS uses OpenJPEG | 15:55.12 |
| chrisl | tkamppeter: I'm not at liberty to say...... | 15:55.28 |
| tkamppeter | chrisl, so it is the one which Artifex has bought for their commercial customers? | 15:56.07 |
| chrisl | tkamppeter: nope | 15:56.37 |
| tkamppeter | Perhaps one should look into a project of creating an API "adapter" to allow the MuPDF library to be used by programs made for using the Poppler library. | 15:59.41 |
| kens | struggles to see why we would want to do that | 16:00.01 |
| tkamppeter | This could perhaps be a good startup for getting MuPDF introduced into Linux distros and mobile devices. | 16:00.30 |
| kens | By making it easier for our customers to switch to Poppler ? ..... | 16:00.50 |
| tkamppeter | kens, no the other way around, I want that Poppler gets replaced by MuPDF. | 16:01.18 |
| chrisl | Customers are unlikely to switch to GPL only libraries | 16:01.38 |
| kens | Well I'm reasonably sure we don't have any resource to do that | 16:01.46 |
| chrisl | I thought somebody had something not unlike that, so mupdf can be used as the back end for one of the PDF viewer applications - not evince, maybe the other one.... | 16:02.59 |
| tkamppeter | So the problem is that of the 3 free PDF interpreters GS takes most resources, Poppler is somewhere in the middle and MuPDF takes least resources? | 16:05.29 |
| | But license-wise Poppler is a license which allows free use for closed-source app developers and GS and MuPDF libs are GPL (not LGPL)? | 16:06.40 |
| mvrhel_laptop | why would linux developers favor a license that allows closed-source app development vs one that is AGPL? | 16:08.02 |
| | linux developers don't make closed source apps I thought | 16:08.44 |
| tkamppeter | mvrhel_laptop, its more about Linux-using phone manufacturers who want perhaps to be open to as many app developers as possible in the hope to get an app arsenal as the two established phone OSes have. | 16:09.37 |
| mvrhel_laptop | well, one needs to look at the features and performance of the libraries then and make that decision. Developers are free to use MuPDF in the app. They will need to pay a license fee for MuPDF if they choose the closed source route but that is only fair if they want the better performance and the ability to do epub, xps, interactive forms, digital signatures, annotations etc | 16:12.04 |
| tkamppeter | My thoughts were about what the OS supplies by itself, being as less resource-consuming as possible and being feature-complete, supplying all needed/expected standard tasks. | 16:13.56 |
| mvrhel_laptop | I see. So it sounds like you will be staying with Poppler then due to the licensing issues | 16:15.59 |
| tkamppeter | mvrhel_laptop, I must take all things into account, especially requirements of the OS architects. | 16:18.30 |
| | I would like to see MuPDF being more accepted as it is less resource-consuming/. | 16:19.12 |
| | I am also promoting it by GSoC projects towards supporting it in cups-filters. | 16:19.52 |
| | Do you have PDF sample files with embedded JPG2000 images? | 16:20.58 |
| chrisl | There are ones in our tests archive, yes | 16:22.59 |
| tkamppeter | chrisl, can you post links here or send them to me by e-mail? Thanks. | 16:26.32 |
| | I have posted | 16:26.41 |
| | http://git.ghostscript.com/?p=thirdparty/openjpeg.git;a=summary | 16:26.44 |
| | on https://bugs.launchpad.net/ubuntu/+source/openjpeg/+bug/711061 and they tell last commit is 2 years old. Is this not your current repo of openjpeg fork? What is your current one? | 16:27.39 |
| chrisl | That actually does seem to be up to date - so we're using 2.1.0 as released | 16:28.26 |
| tkamppeter | Is 2.1.0 already 2 years old? | 16:28.59 |
| Robin_Watts | almost, evidently. | 16:29.31 |
| chrisl | My 2014 commit says; "Update OpenJPEG to 2.1.0" | 16:30.04 |
| | tkamppeter: I'd need to work out which PDF/JPX files we'd be allowed to share | 16:30.48 |
| vtorri_ | hello | 19:22.24 |
| ghostbot | Welcome to #ghostscript, the channel for Ghostscript and MuPDF. If you have a question, please ask it, don't ask to ask it. Do be prepared to wait for a reply as devs will check the logs and reply when they come on line. | 19:22.24 |
| vtorri_ | is ghostscript supposed to build with MSYS2 + gcc (mingw-w64) ? | 19:22.56 |
| | i think i found something wrong in configure.ac of ghostscript | 19:58.29 |
| | with MSYS2 : | 19:58.34 |
| | uname reports MSYS_NT-6.1 | 19:58.53 |
| | so all the : | 19:58.59 |
| | case `uname` in | 19:59.21 |
| | MINGW*) | 19:59.21 |
| | should actually be | 19:59.26 |
| | case `uname` in | 19:59.28 |
| | MINGW*|MSYS*) | 19:59.28 |
| | there is still several problems with MSYS2 : | 20:00.11 |
| | x86_64-w64-mingw32-gcc -DSHARE_LCMS=0 -DCMS_USE_BIG_ENDIAN=0 -Dsqrtf="(float)sqrt" -DCMS_PTR_ALIGNMENT=8 etc.... | 20:00.30 |
| | <command-line>:0:8: error: expected identifier or '(' before 'float' | 20:00.41 |
| | there are several other imilar defines in Makefile | 20:00.58 |
| | they are just not correctly interpreted by MSYS2 | 20:01.14 |
| | better putting all those defines in a .h and include it | 20:01.54 |
| Robin_Watts | vtorri_: As far as I know, we don't actively support building with MSYS. | 20:36.47 |
| | Having said that, we'd probably take on fixes if you have any. | 20:37.24 |
| | The person to speak to about this is chrisl, but he's probably gone for the night. | 20:37.39 |
| malc_ | https://www.boblycat.org/~malc/fira-source.png | 20:37.40 |
| Robin_Watts | We have a new release due soon, so if you have fixes, we could do with them asap in order to get them into that. | 20:38.04 |
| malc_ | who's the thief and who's the victim? | 20:38.16 |
| vtorri_ | Robin_Watts: i'll try | 20:40.30 |
| | Robin_Watts: i can''t clone the repo, btw | 20:55.24 |
| Robin_Watts | You can't clone ghostscript? | 20:55.40 |
| vtorri_ | yes | 20:55.47 |
| Robin_Watts | Using what version of git, on what OS, using what URL ? | 20:55.54 |
| vtorri_ | git clone https://git.ghostscript.com/ghostpdl.git | 20:56.34 |
| | git clone http://git.ghostscript.com/ghostpdl.git | 20:56.35 |
| Robin_Watts | I would have expected the latter to work. | 20:56.56 |
| vtorri_ | oups | 20:57.04 |
| | yes | 20:57.07 |
| | the latter | 20:57.10 |
| | it fails | 20:57.13 |
| Robin_Watts | trying here... | 20:57.44 |
| vtorri_ | msys2 | 20:57.49 |
| | cloning usually works with it | 20:58.00 |
| Robin_Watts | It's cloning using msys bash for me. | 21:18.00 |
| | What error are you seeing? | 21:18.07 |
| vtorri_ | nothing | 21:25.03 |
| | it fails because of timeout | 21:25.10 |
| Robin_Watts | ok, worked for me. | 21:25.19 |
| vtorri_ | not with msys2 | 21:26.10 |
| Robin_Watts | I'm using msys git. | 21:27.11 |
| | In the msys shell. | 21:27.15 |
| | I think we have an official github mirror too. | 21:35.35 |
| vtorri_ | Robin_Watts: what's the link ? | 21:37.33 |
| tor8 | vtorri_: if the git:// fails, you can also use http:// instead | 21:38.07 |
| | oh, you already tried that. | 21:38.15 |
| | no idea then... | 21:38.18 |
| vtorri_ | ok | 21:38.29 |
| | git:// means i have to have the rights to clone from it, right ? | 21:38.55 |
| Robin_Watts | No, git is open access. | 21:41.16 |
| vtorri_ | working with git:// | 21:42.15 |
| sebras | Robin_Watts: actually git clone here "hangs" at "Cloning into 'ghostpdl'" too. | 21:42.23 |
| | Robin_Watts: but using wireshark I can see that some data is being transferred. | 21:42.37 |
| vtorri_ | sebras: wait for 5 mn :) | 21:42.53 |
| Robin_Watts | Doesn't hang for me, just takes a while to run. | 21:43.04 |
| sebras | Robin_Watts: yeah, hard to say that it's quick though. | 21:43.29 |
| HenryStiles | mvrhel_laptop: did see the larry david debate - hilarious | 22:00.10 |
| mvrhel_laptop | HenryStiles: he has him spot on. | 22:00.32 |
| | HenryStiles: Did Miles say to go ahead and buy the london tickets or to wait a bit longer. I can't remember | 22:19.35 |
| HenryStiles | mvrhel_laptop: I thought it part of his agenda but I'll ping him for an official email | 22:29.09 |
| mvrhel_laptop | HenryStiles: ok thanks | 22:33.18 |
| | Forward 1 day (to 2016/03/10)>>> | |
IRC Logs
