| | <<<Back 1 day (to 2014/02/20) | 2014/02/21 |
| ray_laptop | henrys: if you want, I can edit it into the logs file on casper | 15:20.13 |
| | (the same way we take out customer names) | 15:20.48 |
| | hmm... something going on with infobot I think. It is running, but there still is no 2014/02/21 file so I don't think the logging is happening. I sure don't see it in the html | 15:23.58 |
| mvrhel_laptop | Robin_Watts: great | 15:25.12 |
| | I want to ask you a bunch of questions too | 15:25.20 |
| henrys | naively wonders why there isn't a cron job somewhere checking if ghostbot is running and starts it if it isn't... | 15:25.24 |
| | ray_laptop: okay I sent it to you. | 15:25.33 |
| mvrhel_laptop | but I have to get kids out door first | 15:25.34 |
| ray_laptop | henrys: I think marcos has one | 15:25.39 |
| | henrys: but the pd -Fughostbot shows it as running | 15:25.55 |
| | henrys: thanks for the logs. | 15:26.08 |
| | I'll dig into the infobot a bit to see what might be happening | 15:26.31 |
| henrys | I wouldn't want anyone to miss out on that fine reading ;-) | 15:26.41 |
| Robin_Watts | ok. smartoffice.apk is in my homedir on casper. | 15:29.32 |
| | That's SmartOfficePlus with the new fonts. If people with android devices could test it I would appreciate it. | 15:29.58 |
| henrys | Robin_Watts: I have the commercial version installed will that replace it? | 15:30.24 |
| Robin_Watts | It will install SmartOfficePlus alongside SmartOffice. | 15:30.44 |
| | Presumably you got Smart Office 2 ? | 15:30.50 |
| henrys | yes | 15:30.59 |
| Robin_Watts | then this will install alongside, and should have more features. | 15:31.17 |
| henrys | okay I'll check it. | 15:31.25 |
| | paulgardiner, Robin_Watts: does it make sense for marcosw to set up a regreession test using the pdf output as a target. | 15:37.14 |
| | ? | 15:37.22 |
| Robin_Watts | henrys: There are lots of different things we could test. | 15:37.33 |
| | From individual components of the code up to complete builds of apps. | 15:37.58 |
| | Picsel had an Automated Testing System (ATS) for this. | 15:38.26 |
| henrys | I'm just interesting in avoiding unforseen changes from you modifications | 15:38.27 |
| | s/you/your | 15:38.35 |
| Robin_Watts | and we got the code for that. | 15:38.44 |
| henrys | okay so I guess that should be set up | 15:38.56 |
| Robin_Watts | The problem is, there are literally thousands (if not more) of valid build/option combinations. | 15:39.19 |
| ray_laptop | Robin_Watts: so what was the ATS written in, MUMPS ? | 15:39.38 |
| Robin_Watts | so to realistically set it up we would ideally need the ATS database. | 15:39.57 |
| | ATS was written in perl, IIRC. | 15:40.07 |
| ray_laptop | Robin_Watts: marcosw will be so pleased -- having to work with somebody else's perl code ;-) | 15:40.34 |
| Robin_Watts | The guy largely responsible for it (both originally writing it and maintaining it) is Miah Gregory, who did our new Artifex site. | 15:40.53 |
| | He could probably set it all up for us remotely. | 15:41.15 |
| henrys | perfect so he sets it up, bills us, is it a burdensome system to keep after? | 15:44.14 |
| ray_laptop | henrys: that log file looks funny. most timestamps missing and messages are truncated | 15:45.08 |
| Robin_Watts | henrys: It's got a web interface, so theoretically, no worse than bugzilla. | 15:46.11 |
| henrys | ray_laptop: right you are I'll send you another | 15:46.16 |
| Robin_Watts | It's currently written to use CVS to get the source though. | 15:47.04 |
| | so there might be some work involved in making it use git. | 15:47.21 |
| | The review system used to hook into ATS, but I don't think ATS depends on anything else. | 15:48.21 |
| | Want me to drag miah in here for a quick consult ? | 15:48.33 |
| henrys | I do | 15:48.39 |
| | ray_laptop: resent | 15:50.22 |
| Robin_Watts | trying now. | 15:50.28 |
| henrys | ray_laptop: there is an "ERC>" on the last line which can be removed. | 15:51.14 |
| Robin_Watts | Hi mace. | 15:52.26 |
| henrys | Robin_Watts: so this sounds like the sort of system that tells you something broke the next day. Or is there a way to get results faster. | 15:52.43 |
| | ? | 15:52.44 |
| ray_laptop | henrys: thanks. I'll have a go | 15:52.50 |
| Robin_Watts | What would be involved in setting up ATS on a server of ours? | 15:52.54 |
| | henrys: You can upload patches to the system and have it run them. | 15:53.20 |
| mace | i might be setting up a new ats right at this second, in theory ;) | 15:53.24 |
| Robin_Watts | so you can run specific tests in advance of committing. | 15:54.05 |
| mace | Robin_Watts: setting it up isn't too difficult. logistically if you need windows hosted builds or tests then you need to have windows machines to run on | 15:54.08 |
| henrys | Robin_Watts: I was asking about the run time, if it is going to check all these configurations I assume that will take a while. | 15:54.09 |
| mace | henrys: it can run builds/tests in parallel, depending on how much hardware you want to throw at it | 15:54.31 |
| Robin_Watts | But yes, most of the large tests are once a day ones. | 15:54.36 |
| | The smoke tests for a commit used to take about 20-30 mins IIRC. | 15:55.04 |
| henrys | mace:so windows is going to be easiest, we currently do most testing stuff on linux. | 15:55.18 |
| | ? | 15:55.20 |
| Robin_Watts | henrys: ATS has a server component, and many clients that can connect to it. | 15:55.50 |
| | clients can be linux or windows or ... | 15:55.57 |
| | the server is a linux box, and presumably can be a client too. IF we want windows hosted builds, then in addition, we'd need a windows box. | 15:56.55 |
| henrys | I'm talking about how the server runs. Presumably we want marcosw to eventually handle it and he doesn't like windows much | 15:56.56 |
| mace | henrys: the server is linux based | 15:57.26 |
| henrys | mace: great | 15:57.34 |
| | maces:why are you set setting up an ATS server now? or is that private? | 15:58.35 |
| | mace ^^^ | 15:58.52 |
| Robin_Watts | ATS currently gets the source from cvs. How much work would be involved in updating it to work with git? | 16:00.15 |
| henrys | mace: but it you are interested we will give you the keys to a server and you can set it up and bill us. Are you interested in doing that? | 16:01.26 |
| mace | sorry was afk | 16:02.08 |
| | henrys: i "might" be setting one up for another entity ;) | 16:02.32 |
| | henrys: yeah i don't have a problem with that | 16:02.57 |
| | would strongly prefer debian as the distro | 16:03.11 |
| henrys | sounds mysterious but it looks like we can move on. | 16:03.33 |
| | maces:I'm thinking an aws host? | 16:03.47 |
| mace | Robin_Watts: offhand, i'm not sure, however if you are dropping cvs entirely then switching over to git shouldn't be too painful | 16:03.51 |
| | henrys: that should be fine; not sure what the costs will be for a server of suitable spec | 16:04.19 |
| henrys | mace: but we could be more flexible I suppose, marcosw is our testing guru, like to let him weigh in. | 16:04.21 |
| mace | henrys: debian is just a flavour of linux; unrelated to who does the hosting | 16:04.43 |
| henrys | oh I thought aws defaulted you to ubuntu, see how much I know? ;-) | 16:05.23 |
| | mace: I run linux at home | 16:05.37 |
| Robin_Watts | ubuntu is a debian derivative I think. | 16:05.47 |
| henrys | yes it is | 16:05.52 |
| | but I guess marcosw installed that on our aws server and could have installed anything. | 16:06.11 |
| | I didn't understand that. | 16:06.18 |
| mace | ubuntu should be fine; it's not my choice for servers, but it is basically debian ;) | 16:06.48 |
| henrys | mace: what do you require hardware wise? | 16:06.49 |
| mace | that depends entirely on what testing you want to do; robin are you in a better position to quantify that? | 16:07.15 |
| Robin_Watts | mace: I guess we want the standard smoke tests. | 16:07.41 |
| | plus overnight builds of the smart office variants. | 16:07.55 |
| mace | Robin_Watts: do you recall what they were? i believe you have used ats more recently than me | 16:08.09 |
| Robin_Watts | I had to get the build line for smart office from alex yesterday :) | 16:08.32 |
| mace | heh | 16:08.41 |
| Robin_Watts | I *hope* we should be getting the ATS database as part of the kit from the liquidators. | 16:08.53 |
| mace | Robin_Watts: i imagine so, although you probably only want a fraction of the data in it (build commands etc) as the results are *huge* | 16:09.30 |
| mace | is currently copying ~ 80gb of db | 16:09.43 |
| Robin_Watts | yeah. | 16:09.47 |
| mace | re machine spec, i guess an aws machine should be fine to start off with; it can always be upgraded or migrated to a more beefy machine once ats is set up and running | 16:10.34 |
| henrys | Robin_Watts, mace: all of this is regression only? Are there unit tests or something else? | 16:11.01 |
| Robin_Watts | I think basically, we don't care about old builds. We can add more stuff as it becomes an issue. | 16:11.09 |
| mace | certainly the client nodes (if you want any) would work well under aws; with some work ats could fire up new aws instances to handle load and shut them down afterwards | 16:11.18 |
| Robin_Watts | There are unit tests, yes. | 16:11.24 |
| | mace: We currently have about 20 real physical server machines of various sizes around the net. | 16:11.52 |
| | We have 1 central aws node that runs our cluster server. | 16:12.10 |
| | And all the clients connect into that. | 16:12.20 |
| mace | i presume you don't want ats clients on them though? | 16:12.40 |
| Robin_Watts | We use AWS for the main server as it's generally always on, whereas the other machines are in peoples offices etc, and so have less reliable uptimes. | 16:12.57 |
| | We could install ats clients on at least some of them. | 16:13.05 |
| | so initially I think we'd probably work on an AWS server, and non AWS clients, but that might change depending on loading etc. | 16:13.56 |
| | Marcos is the guy to talk to here really, he might well have a different opinion. | 16:14.35 |
| henrys | mace: marcosw will probably want to decide about that. Alot depends on how much data is moved around the local machines don't have very fast pull times. | 16:14.43 |
| Robin_Watts | mace: Do clients check out from cvs? Or do they rsync a whole tree ? | 16:15.21 |
| henrys | mace: the local boxes are typically cable modem connected with good down time but lousy up time | 16:15.36 |
| mace | nods | 16:15.57 |
| | Robin_Watts: they check out a clean tree each time, however we can probably figure out some kind of rsync or git local cache on the nodes if the tree doesn't change 20 times a second | 16:16.43 |
| Robin_Watts | mace: It doesn't :) | 16:17.10 |
| mace | we would also need to consider security; ats wasn't really designed to run in a hostile environment | 16:17.20 |
| Robin_Watts | That might fall out naturally if we move over to using git rather than cvs. | 16:17.32 |
| | mace: the existing cluster has the nodes connecting in via ssh. | 16:17.50 |
| mace | that may mean we look at adding ssl to the connections between client/server, or it may mean vpn connections between them | 16:17.52 |
| mace | nods | 16:17.59 |
| Robin_Watts | With our cluster the nodes basically 'poll' the server every 30 seconds. | 16:18.23 |
| | In ATS, are the nodes permanently connected? Or do the clients poll? Or does the server connect out to the nodes? | 16:18.55 |
| | Hi marcosw_. The logs may not be working at the moment. | 16:22.13 |
| | but we have been talking about setting up regression testing on the picsel code. | 16:22.33 |
| marcosw_ | Robin_Watts: yeah, I saw that. | 16:22.57 |
| Robin_Watts | Picsel had an Automated Testing System that ran all sorts of tests (regression, unit tests, etc), and ... ah ok. | 16:23.18 |
| mace | Robin_Watts: they are permanently connected, they connect in to the server when they start up | 16:23.40 |
| Robin_Watts | mace: OK, so we could add a sprinkling of ssh to those, and we'd be Ok? | 16:24.11 |
| henrys | mace: can you email Miles and estimate once you understand all the pieces we need? | 16:24.18 |
| | s/and/an | 16:24.57 |
| mace | Robin_Watts: they could probably run over ssh tunnels, aye; i'd probably be more inclined to look at adding ssl support or using ssl wrappers though as it's one less layer to break | 16:25.18 |
| | henrys: urm, it's pretty open ended so difficult to give much of an estimate | 16:25.49 |
| ray_laptop | Robin_Watts: The logs have been working AFAICT (since I goosed ghostbot) | 16:26.17 |
| Robin_Watts | ray_laptop: OK. | 16:26.28 |
| mace | maybe a day or so to get the server installed and set up, but after that it's down to tweaking builds etc. migration from cvs to git isn't included | 16:26.28 |
| henrys | mace: maybe we should bring you on hourly. | 16:27.50 |
| mace | i suspect that's probably simpler. same applies to the website project (which i need to email miles back about) | 16:28.40 |
| marcosw_ | mace: so I think this as all been discussed but in summary we maintain a "cluster" of ~25 machines all running ubuntu linux for Ghostscript testing. | 16:32.24 |
| | they communicate with an AWS machine (also running ubuntu) . | 16:32.50 |
| henrys | mace: that sounds good, but I'd like him to have a heads up this work is coming. So if you want to send me your hourly rate for this new job or you can contact him directly. But I'd like to avoid presenting a bill to him after many hours of work have accumulated. | 16:33.06 |
| mvrhel_laptop | Robin_Watts: grabbing the new version you made. so what exactly is the difference to smart office 2? | 16:33.18 |
| marcosw_ | the nodes are all behind firewalls and/or NAT, so they poll the AWS machine every 10 seconds to see if a new testing job is queued. | 16:33.45 |
| | they then run the jobs they are assigned and send the results back to the AWS machine, which emails them out and puts them onto the internets. | 16:34.13 |
| mace | how does ats relate to your existing testing system? | 16:34.18 |
| Robin_Watts | mvrhel_laptop: Smart Office 2 costs 5.99. Smart Office Plus costs 20.99 :) | 16:34.28 |
| | I think Smart Office Plus does PDF annotation, and hebrew. | 16:34.42 |
| | but that's about all. | 16:34.46 |
| mvrhel_laptop | Robin_Watts: I got ripped off I paid 9.99 for Smart Office 2 :) | 16:34.58 |
| Robin_Watts | The difference between *my* smart office plus build and the standard one is that I've pulled in the new fonts. | 16:35.19 |
| henrys | mvrhel_laptop: try to get a refund ;-) | 16:35.19 |
| Robin_Watts | I may have the amounts wrong :) | 16:35.29 |
| marcosw_ | I think the plan is to use the same cluster to run the jobs but set up a separate AWS machine to act as the server. | 16:35.36 |
| mvrhel_laptop | maybe your prices are in pounds | 16:35.47 |
| Robin_Watts | yes, sorry. | 16:36.02 |
| mace | marcosw_: will there be any problems with running jobs on both systems concurrently on nodes? | 16:36.11 |
| Robin_Watts | Hmm. Smart Office Plus is now refusing to start for me :( | 16:36.15 |
| marcosw_ | mace: there shouldn't be. the performance will obviously suffer if both systems are running a job at the same time, but presumably we could setup a semaphore so that one job completes before the next starts. | 16:37.18 |
| mvrhel_laptop | Robin_Watts: so something that is certain to come up in Japan is memory requirements and footprint for something that does what ghostdocs does (mupdf + office parser) and likewise for something that includes editing of office content | 16:37.22 |
| Robin_Watts | mvrhel_laptop: Gawd, now you're asking :( | 16:37.45 |
| marcosw_ | the same nodes that run jobs for commits and user initiated tests also run a large set up jobs overnight, the overnight jobs pause when a higher priority job is started. | 16:38.01 |
| mvrhel_laptop | sorry :( | 16:38.06 |
| marcosw_ | ray_laptop: did you end up having to restart ghostbot? | 16:40.20 |
| ray_laptop | marcosw: yep | 16:40.34 |
| Robin_Watts | mvrhel_laptop: Gah. That version of Smart Office is duff. | 16:41.12 |
| | Let me do another, sorry. | 16:41.30 |
| mvrhel_laptop | ok. np. I just got it and had not yet installed | 16:41.52 |
| ray_laptop | marcosw: it was the same as previously -- it was running, but not logged into the channel | 16:42.18 |
| marcosw_ | ray_laptop: I'm going to modify my script that checks if it's running to also check if the log file has been updated in the last 30 minutes and restart it not. This will result in unnecessary restarts during the weekend and evening hours (PST) but better than the alternative. | 16:42.31 |
| henrys | Robin_Watts: I got "unfortunately smart office plus has stopped" upon opening a word document | 16:43.14 |
| Robin_Watts | henrys: yeah. | 16:43.23 |
| ray_laptop | henrys: complain to the support team ;-) | 16:43.36 |
| Robin_Watts | Trying for a rebuild now. | 16:43.40 |
| marcosw_ | henrys: you can file a bug | 16:43.50 |
| Robin_Watts | I git stashed some changes and forgot to git stash pop before I built this version. | 16:44.07 |
| henrys | mace: are you an independent consultant full time? | 16:45.23 |
| mace | henrys: i run my own company full time, which includes consultancy | 16:48.39 |
| henrys | marcosw, ray_laptop : geez guys Robin_Watts asked me to downlad it and try it. | 16:49.22 |
| Robin_Watts | henrys: Uploading a new version now. It'll take 11 minutes more. Sorry! | 16:49.48 |
| marcosw_ | okay, ghostbot cron job updated, no one say anything for 30 minutes and we'll see if it works :-) | 16:53.22 |
| mvrhel_laptop | so is the new code in a git repository now? I had thought it was in cvs or svn | 17:00.19 |
| Robin_Watts | mvrhel_laptop, henrys: OK, new version in my home dir as casper. Sorry about that. | 17:01.32 |
| mvrhel_laptop | ok | 17:01.42 |
| Robin_Watts | mvrhel_laptop: I did a conversion from cvs to git of the "epage" module. | 17:01.51 |
| | and that's what I've been working on. | 17:01.59 |
| mvrhel_laptop | ok | 17:02.01 |
| henrys | dropbox is nice for installation | 17:02.47 |
| mvrhel_laptop | ok it runs... | 17:04.43 |
| | this is a nicer app than the smart office 2 one. at least I like its interface | 17:06.11 |
| henrys | yes looks good | 17:06.19 |
| | a looked at a pdf file too. Looks okay | 17:06.47 |
| Robin_Watts | The interface looks very much like the Smart Office 2 build I've been looking at. | 17:07.05 |
| mvrhel_laptop | hmm seems different than the one I had. | 17:07.28 |
| | so what is the difference between 2 and plus then? | 17:08.25 |
| | Robin_Watts: ^^ | 17:08.28 |
| Robin_Watts | hebrew, and PDF annotation. | 17:08.42 |
| | basically. | 17:08.48 |
| mvrhel_laptop | for me, the smart office 2 ui has some issues. I will have to show you at the staff meeting | 17:09.23 |
| Robin_Watts | ok. You got Smart office 2 from the store. I got it from paulgardiner. | 17:09.46 |
| | so I would have expected your version to be newer. | 17:09.52 |
| | odd. | 17:09.56 |
| | I can't figure out how to do PDF annotation though :( | 17:10.16 |
| mvrhel_laptop | the add button | 17:10.57 |
| Robin_Watts | ah, yes, I have it. | 17:11.12 |
| mvrhel_laptop | then there is a sub selection for annotation | 17:11.13 |
| Robin_Watts | but I can't type anything... | 17:11.23 |
| | ah, got it. | 17:11.43 |
| mvrhel_laptop | the ui is struggling though... | 17:11.59 |
| mace | you folks need me for any more questions? | 17:12.22 |
| Robin_Watts | mace: So you're going to drop Miles a mail with some vague time/cost estimates? | 17:12.54 |
| | n days for setting up basic ATS. | 17:13.23 |
| mace | Robin_Watts: already did; it's very vague :) | 17:13.53 |
| Robin_Watts | ok :) | 17:14.03 |
| mvrhel_laptop | Robin_Watts: so if you select the little pages icon in the top left, do you get something odd on your screen | 17:14.06 |
| | top right I mean | 17:14.27 |
| | it is doing a weird display | 17:14.38 |
| Robin_Watts | mvrhel_laptop: Try it with a PDF with many pages :) | 17:14.39 |
| henrys | wow lots of stuff in this app. | 17:14.41 |
| mvrhel_laptop | Robin_Watts: I am | 17:14.45 |
| Robin_Watts | You should get a funky 3d page carousel thing. | 17:15.01 |
| mvrhel_laptop | I have a 100+ page pdf | 17:15.02 |
| Robin_Watts | ok. | 17:15.09 |
| mvrhel_laptop | yes, but it is rotated weird | 17:15.09 |
| | I can't see the page really | 17:15.18 |
| Robin_Watts | hmm. I'm seeing what I expect here :( | 17:15.40 |
| mvrhel_laptop | only 1/2 the page is really drawn here | 17:15.55 |
| Robin_Watts | Is it just the one file? | 17:16.16 |
| | or does it look odd with all PDFs? | 17:16.24 |
| mvrhel_laptop | no. all the files do this | 17:16.25 |
| Robin_Watts | portrait or landscape? What device? | 17:16.36 |
| mvrhel_laptop | nexus 10 landscape | 17:16.46 |
| | oh | 17:16.55 |
| | portrait is ok | 17:17.02 |
| | landscape is wacked | 17:17.22 |
| Robin_Watts | maybe something can't cope with the size of the screen on the Nexus 10? | 17:17.32 |
| | let me try on the transformer. | 17:17.39 |
| mvrhel_laptop | this thing must not have much of a page cache | 17:18.03 |
| Robin_Watts | paulgardiner wrote the page carousel, so we can blame him :) | 17:18.53 |
| mvrhel_laptop | excellent | 17:19.02 |
| | and mine has locked up | 17:19.17 |
| Robin_Watts | The page carousel is build using Lomond. | 17:19.31 |
| | Lomond is the UI project. | 17:19.41 |
| mvrhel_laptop | there it has come back | 17:19.44 |
| Robin_Watts | It's not too bad to use, but it's a bitch to develop on. | 17:19.53 |
| henrys | mvrhel_laptop: page carousel works fine for me. | 17:20.21 |
| mvrhel_laptop | in landscape too henrys? | 17:20.37 |
| henrys | yes | 17:20.51 |
| mvrhel_laptop | with the nexus 10? | 17:21.01 |
| henrys | I have the small nexus | 17:21.10 |
| | maybe performance with larger screen? | 17:21.19 |
| mvrhel_laptop | weird. I will show you all at the meeting. henrys: I am wondering if a smaller device might be better for this app | 17:21.36 |
| Robin_Watts | Works fine on the transformer too. | 17:22.06 |
| mvrhel_laptop | it is struggling to keep up as I scroll through the pages | 17:22.11 |
| | shows me how good mupdf is | 17:22.55 |
| | ok. so Robin_Watts that brings me back to the bomb that I dropped earlier | 17:23.51 |
| henrys | mine is slower than mupdf but I don't fell it is awkwardly slow. | 17:24.07 |
| Robin_Watts | I refer the honorable gentleman to the answer given earlier. | 17:24.44 |
| mvrhel_laptop | I will def. bring the 10 to the staff meeting. I am going to have to be careful what docs I bring up | 17:24.52 |
| | oh there was an answer earlier.... | 17:25.01 |
| Robin_Watts | yeah. I said I didn't know. | 17:25.53 |
| mvrhel_laptop | :) | 17:26.05 |
| | I can see a whole table full of frowning Japanese when I say that at the meeting | 17:26.29 |
| | that is what woke me up in the middle of the night last night | 17:26.50 |
| Robin_Watts | OK, so, for every filetype except PDF, we load the entire file and lay it out at the start. | 17:27.21 |
| | There is no 'page by page loading'. | 17:27.36 |
| | So if you load a 1000 page document, it will all be loaded into memory. | 17:27.51 |
| | To a certain extent that's inescapable as stuff like word is laid out continuously. | 17:28.30 |
| | With PDF files it can be smarter and just load pages into memory as it needs them. | 17:28.47 |
| mvrhel_laptop | really? I am surprised | 17:28.47 |
| Robin_Watts | Such is my understanding of the situation, yes. | 17:29.01 |
| | Word files are really like HTML files as they are broken onto pages for printing. | 17:29.27 |
| mvrhel_laptop | granted, but at some point I would think there would be an endpage sort of thing | 17:30.15 |
| Robin_Watts | You can get page breaks, yes. | 17:30.25 |
| ray_laptop | mvrhel_laptop: most docs don't have forced page breaks | 17:30.43 |
| mvrhel_laptop | certainly for power point there is a concept of a page/slide | 17:30.53 |
| Robin_Watts | But if you've got a 100 page document that's all one chapter, you're still going to have to lay out 100 pages. | 17:30.56 |
| | Right, yes, powerpoint is different. | 17:31.01 |
| ray_laptop | mvrhel_laptop: rught | 17:31.03 |
| | s/u/i/ | 17:31.09 |
| Robin_Watts | But the picsel stuff doesn't do page by page loading for powerpoint. | 17:31.17 |
| | (or at least, I don't believe it does). | 17:31.34 |
| mvrhel_laptop | that seems like an obvious improvment we could look into | 17:31.37 |
| Robin_Watts | That's the kind of improvement that might be much easier to add during the process of ripping out a lib and refactoring. | 17:32.09 |
| mvrhel_laptop | right | 17:32.17 |
| ray_laptop | well that makes it easier to move from page to page. PPT docs, even though individual pages are often complex, aren't often very many pages | 17:32.28 |
| Robin_Watts | I wouldn't like to bet how hard it would be to do it in situ. But it's certainly something we could look at. | 17:32.33 |
| mvrhel_laptop | I still don't see why I can't do a single page though in word. At some point a decision is made that this is a new page | 17:32.55 |
| ray_laptop | unless the PPT is for a history teacher's class. Then they can be massive ;-) | 17:33.07 |
| Robin_Watts | Essentially, documents are loaded into an internal format called Edr. That is then 'laid out' into displaylists. | 17:33.31 |
| | Some things are shared between Edr and DisplayList, I think, but not all. | 17:34.00 |
| | text for example lives in both. | 17:34.08 |
| mvrhel_laptop | I see. so this "loading" is when the whole thing is brought in | 17:34.17 |
| Robin_Watts | images (like JPEGs etc) can possibly be shared. | 17:34.20 |
| | Yes. | 17:34.24 |
| ray_laptop | marcosw: stopping and restarting ghostbot doesn't truncate the logs does it ? | 17:34.50 |
| Robin_Watts | When you open a multipage document you'll see the first page appear as a thumbnail, with the spinning thing on top of it while it loads the rest. | 17:35.01 |
| mvrhel_laptop | wait. that smells like I could do individual pages then | 17:35.43 |
| ray_laptop | Robin_Watts: so rather than a thumbnail, the full size page could be shown ? | 17:35.51 |
| | while it works in the background on the next page | 17:36.06 |
| Robin_Watts | mvrhel_laptop, ray_laptop: yes, you can look at page 1, while it's loading page 2. | 17:36.22 |
| | You can't look at page 100 without pages 1-99 being loaded first though. | 17:36.35 |
| | ray_laptop: This code is all VERY thread happy. | 17:36.54 |
| ray_laptop | Robin_Watts: that makes sense in any reflow | 17:37.06 |
| Robin_Watts | It goes to huge lengths to allow interaction to continue while other stuff runs. | 17:37.25 |
| | mvrhel_laptop: Does that answer your thought about individual pages ? | 17:38.21 |
| | It is possible that the code is smart enough to avoid holding both Edr and DisplayLists for every page (if it can remember the start/end of each page within the Edr, it maybe could ditch display lists and then recreate them). | 17:39.03 |
| | I do not know if that is implemented or not - not my area of the code. | 17:39.15 |
| mvrhel_laptop | Robin_Watts: that seems like the behavior as I scroll on the nexus 10 | 17:39.33 |
| | it is having to redraw pages earlier | 17:39.47 |
| | and it seems like it is doing a lot of work | 17:39.57 |
| Robin_Watts | mvrhel_laptop: The standard way of working in the picsel stuff is to have a 'thumbnail' bitmap for each page. | 17:40.07 |
| | So when you jump to a page, it can very quickly scale that bitmap up onto the screen. | 17:40.19 |
| | Then it does an HQ redraw on top of it. | 17:40.25 |
| | which takes longer. | 17:40.31 |
| mvrhel_laptop | Robin_Watts: I understand. that is what I did in the windows 8 stuff and I have done for the new version of gsview | 17:40.43 |
| Robin_Watts | This means that you can pan and zoom and still get the impression of instant results. | 17:40.47 |
| | Right. It is possible that thumbnails get binned if they are too far out of range. | 17:41.11 |
| | so jumping large numbers of pages may require the thumbnail to be redrawn. | 17:41.31 |
| | whether that requires the layout to be done first will depend on whether we still have the displaylist or not. | 17:41.49 |
| | For most text heavy files, the in memory stuff would likely be not hugely more massive than the size of the saved file, I'd expect. | 17:44.11 |
| mvrhel_laptop | Robin_Watts: so given how powerpoints can be a disk is maybe going to be needed in any application that is going to be using this stuff. | 17:45.11 |
| Robin_Watts | Possibly, yes. | 17:45.57 |
| | Now, the GhostDocs demo basically uses a doc2pdf step. | 17:46.15 |
| mvrhel_laptop | Robin_Watts: right. but it still needs to load the whole doc yes? | 17:46.32 |
| Robin_Watts | This loads the file in, lays it out, converts it to a "pdfexport" structure in memory and writes it out. | 17:46.54 |
| | So that is actually currently worse than the app. | 17:47.08 |
| mvrhel_laptop | so it needs the size of the input, the display list and the pdf output | 17:47.28 |
| Robin_Watts | BUT that could (in the fullness of time, given enough work etc) be done better. | 17:47.31 |
| mvrhel_laptop | yes | 17:47.35 |
| | what are the memory suggestions for just vanilla mupdf. I just realized I don't know that | 17:47.58 |
| Robin_Watts | as we could load the doc in, and layout each page at a time, and convert each page at a time, and then discard the bits we'd loaded. | 17:47.59 |
| mvrhel_laptop | Robin_Watts: right | 17:48.11 |
| Robin_Watts | mvrhel_laptop: I can't remember those either. Scott and Miles presumably have those. | 17:48.23 |
| mvrhel_laptop | Robin_Watts: I don't think they do. We have the footprint numbers | 17:48.37 |
| Robin_Watts | As with all these things, it depends on the file in question. | 17:48.43 |
| mvrhel_laptop | sure | 17:48.48 |
| | I have some numbers for gs that ray_laptop had given me | 17:49.00 |
| | but nothing similar for mupdf | 17:49.13 |
| Robin_Watts | Now, for MuPDF I can do a memento build and do a mudraw, and that will give me a peak figure. | 17:49.29 |
| | pdf_reference17.pdf for example ? | 17:49.40 |
| mvrhel_laptop | that would be a good one | 17:49.47 |
| | if that is easy for you to do I would great appreciate. | 17:50.04 |
| Robin_Watts | oh, god this will take hours to run. Let me do a quick tweak. | 17:50.11 |
| mvrhel_laptop | doing the above will take hourse? | 17:50.38 |
| | hours? | 17:50.40 |
| Robin_Watts | Memento keeps old (freed) blocks around and marks them with known values. | 17:51.48 |
| mvrhel_laptop | ah | 17:51.52 |
| Robin_Watts | and every now and then it checks those for corruption. | 17:52.00 |
| | so it's very slow for large files. | 17:52.06 |
| | but I've defined MEMENTO_LEAKONLY and it won't be doing those tests. | 17:52.21 |
| | so it's running now. I'll tell you when I have an answer. | 17:52.32 |
| mvrhel_laptop | nice | 17:52.34 |
| | Thanks a bunch Robin_Watts | 17:52.39 |
| Robin_Watts | no worries. | 17:52.45 |
| mvrhel_laptop | oh Robin_Watts I was wondering if I could bother you for some English Breakfast tea... | 17:52.59 |
| | Do you need anything from Japan? | 17:53.05 |
| Robin_Watts | Sure. | 17:53.07 |
| | Urm... I can't think of anything :) | 17:53.15 |
| mvrhel_laptop | I need to bring you something | 17:53.26 |
| | you have been keeping my wife supplied with her morning caffeine | 17:53.46 |
| Robin_Watts | It's a pleasure. | 17:53.57 |
| | I'm a transatlantic drug dealer :) | 17:54.04 |
| mvrhel_laptop | exactly | 17:54.08 |
| Robin_Watts | I'm wondering about the best way to get figures out from smart office. | 17:54.30 |
| | Picsel use a tool called Fortify, which is what inspired memento. | 17:54.46 |
| | If I do a fortify build of doc2pdf, then I can run a conversion of some input files, and that will tell me peak memory usage etc. | 17:55.19 |
| | Do you want to suggest some files and I'll try and run them through that? | 17:55.41 |
| mvrhel_laptop | Robin_Watts: perhaps just to get some figure let me give you two files that I have been playing around with | 18:01.40 |
| Robin_Watts | Sure. | 18:01.53 |
| | mudraw still plodding. | 18:02.20 |
| mvrhel_laptop | Robin_Watts: so in mvrhel/TestOffice there are 2 files | 18:04.01 |
| Robin_Watts | I have the pptx already, cos I fixed that in GhostDocs. | 18:05.42 |
| | I'd imagine that with ppts etc, a lot of the size is images etc. | 18:06.54 |
| | The Picsel code has an "Image Component" that would handle such images, decompressing them in the background, keeping them in a cache etc. | 18:08.10 |
| mvrhel_laptop | ok | 18:08.24 |
| Robin_Watts | so you'd hope that we wouldn't decompress them until we need to. | 18:08.32 |
| | But the work I had to do last week to make the PDFExport stuff export the missing images makes me wonder whether that's true in all cases. | 18:09.02 |
| | So the memory usage for such files might be heavier than we'd hope. | 18:09.28 |
| | MuPDF is still running. It's holding steady at 293Meg, but Memento keeps a certain amount of freed blocks around. | 18:10.16 |
| | Possibly 256Meg. | 18:10.35 |
| mvrhel_laptop | Robin_Watts: ok. I have to head to my daughters school for some volunteer work. | 18:14.16 |
| | I am going to be out for about 2 hours | 18:14.26 |
| Robin_Watts | I'll try and have figures for when you get back. | 18:14.29 |
| mvrhel_laptop | thanks | 18:18.14 |
| Robin_Watts | Linux_April_2013.pptx took 22Meg to convert to PDF. | 18:20.16 |
| | developing-v4-print-drivers2.docx took 26Meg to convert to PDF | 18:21.22 |
| mvrhel_laptop | Robin_Watts: Ok and those files were about 1.5 and 3.2 Meg each | 18:24.16 |
| | I need to go... going to be later | 18:24.22 |
| ray_laptop | I'll need to stop ghostbot to add in the missing log info. Let me know when we can do without a few minutes of log | 18:24.38 |
| | No response, so I'll try it now | 18:26.29 |
| | hmm.. I must have edited the wrong file | 18:39.57 |
| henrys | the pcl/xl argument processing procedure is getting to be a bit much. | 18:48.18 |
| Robin_Watts | ray_laptop: You added it to the non html file, right? | 18:52.34 |
| | mvrhel_laptop: Yes, but there is presumably a base cost to running the exes. | 18:54.02 |
| | Let me try a figure for an empty docx file. | 18:54.12 |
| | empty.docx takes 1Meg. | 18:56.03 |
| ray_laptop | Robin_Watts: yes, I put it there in 20140221 (not 20140221.html) | 19:30.39 |
| | It is likely that there is something I messed up that caused the converter to skip those lines | 19:31.19 |
| henrys | Robin_Watts, ray_laptop: what does the C library do with argc & argv with respect to utf 8? Was the code added to gsargs to handle unicode compatible with the C library? | 19:34.50 |
| Robin_Watts | Urm... | 19:35.21 |
| | Under unix, the main is called with argc pointing to locale encoded args. | 19:35.59 |
| | where locale is generally UTF8. | 19:36.06 |
| | so we can just take those values direct. | 19:36.16 |
| henrys | right it's all coming back to me now, like a nightmare nevermind | 19:36.34 |
| Robin_Watts | under windows, we don't get called via main, we get called via WinMain, which gets wchar_t *'s rather than char *s. | 19:37.01 |
| | And we encode that into UTF8 and call main. | 19:37.12 |
| | So the basic upshot is that main is now always called with utf8. | 19:37.39 |
| henrys | I need to set an option in pcl that for debugging and I can't use gsargs because it is before the memory is set up so I'm just going to scan the plain argument list. | 19:38.46 |
| | and I'm wondering if I should fool with that, but it's a simple boolean option so I should be okay. | 19:39.59 |
| malc_ | Robin_Watts: where did you get this idea that argv is encoded with anything? | 19:40.03 |
| Robin_Watts | henrys: The nice thing about UTF8 is that you can pretty much treat it as ascii. | 19:40.46 |
| | malc_: reality? | 19:41.07 |
| | Under linux (modern linux at least), you can call programs with unicode arguments, because they get passed to the programs as utf8. | 19:42.19 |
| malc_ | Robin_Watts: reality is that it's just an array of chars nothing locale specific at all | 19:42.31 |
| Robin_Watts | It is just an array of chars. But how the shell encodes the contents of those chars depends on the locale. | 19:43.10 |
| malc_ | Robin_Watts: huh? | 19:43.31 |
| Robin_Watts | Suppose I have a filename on disc with a unicode char in it. | 19:44.07 |
| | say the file is called fred# where # is unicode char 512 | 19:44.45 |
| | how can I do: "gs fred#" ? | 19:45.02 |
| | I have to go. | 19:45.24 |
| malc_ | Robin_Watts: really i don't have to assume my CTYPE is koi8-r, and arguments not being magically transformed into anything by the shell | 19:45.27 |
| | not to mention the whole exec[..] posix_spawn.. family of functions, you can pass anything there | 19:46.00 |
| Robin_Watts | I'm being called for dinner. The fact is that we reliably get utf8 encoded args in, and that's good enough for me. | 19:46.16 |
| mvrhel_laptop | I am back Robin_Watts | 21:53.51 |
| | ok. so the number that you have above give me some idea. in the mupdf case that you had with mudraw running, what resolution were you rendering to? | 22:05.11 |
| Robin_Watts | default, which is... 96dpi I think. | 23:21.47 |
| | mupdf takes 24Meg for the first 100 pages of pdf_reference17.pdf | 23:22.04 |
| | 33Meg for the first 1000 pages. | 23:22.16 |
| mvrhel_laptop | Robin_Watts: thank you | 23:50.24 |
| | need to start packing now | 23:50.32 |
| Robin_Watts | likewise (I'm off tomorrow to my parents in law for the day). | 23:50.58 |
| | Let me know if you need any other figures or information while you're away. | 23:51.11 |
| | and I hope you have a good trip. | 23:51.26 |
| mvrhel_laptop | thanks | 23:51.45 |
| | Forward 1 day (to 2014/02/22)>>> | |
IRC Logs