The main items of discussion during this meeting were using the fnndsc.childrens server for the hackathon and making it more persistent, optimizing file ops that occur within swift, and the node parameter config UI.
Gideon Demo
- bugs in the UI around animation in legacy coode. has been working on fixing them.
- added 'share feed' button & capability
- working with mo on adding new node mockups
- a production bug working on
- will be working on deleting node feature
- will also be working on implementing mo's new mockups (adding new node)
Hackathon Server and Performance Discussion
Parul: When can we put new ChRIS UI on MOC? I'm talking about the time it takes for the job to kick off, long time it takes because of the binding time for the PVC, he's working on that. But if you can do some testing from your side, making a note of all the issues we face, then we can coordinate and talk things out.
Rudolph: That's fantastic, not sure if we have the most recent version on fnndsc.childrens.harvard.edu page...
Gideon: I don't have the same bug in devel but I have it in production. I fixed that yesterday. If I can get through that today, I'll build it for production and have it up on fnndsc. That was the goal yesterday, hoping to do today.
Rudolph: I think Jorge might have production end ideas. Parul, yeh, two things:
1 - we should have this up on the public exposed site. 2 - on the public exposed site - all the ones that end with _moc are registered to run at the moc.
So yes, as long as a chris is instantiated and exposed, you shoul dbe able to dispatch any of these *_moc plugins and ru non the moc.
Paul: Sounds good.
Parul: For hackathon, are we going to use public facing fnndsc.childrens site, or some other server?
Rudolph: We can do either. The public facing server is not on a fantastic machine, it's a VM with just 2 CPUs and 16 GB ram. My laptop is more powerful than that. So we can do either/or, whatever makes the most sense then.
Dan: If we're doing the hackathon, I think - correct me if i'm wrong - part of the goal is to utilize the MOC resources, play aroud iwth chris and learn more about it, write plugins etc. Build out interest. If we don't have something there runing they can go back to consistently, we won't get as much attachment as we'd like .So should we devote more resources to that vm, get more VMs at childrens? Or set up more resources elsewhere, but then how would that tie into childrens?
Rudolph: the public-facing one should be fine, just suggesting - we have more powerful machines just sitting around probably.
Dan: Can we fix that problem? Seems like if we're ready for hackathon, seems like we also need to have an instance we can point them to consistently
Rudolph: yes we can do that. In the meantime I can put in a request to have that VM beefed up a bit. You might be suggesting, if I'm reading between the lines, the idea of a more premanent ChRIS as opposed to currently how it goes up and down, we restarted it, wipe it, switch storage, etc. That's something we should think about - having a persisten almost production site.
Dan: Yeh, at a minimum, a permanent playground, we can always point people to, let them build stuff, make it work. For permanent production - not sure about use case... not sure about any org that would run something on it permanently. but some level of prod capability so people can develop againt it, and it'll be there fairly reliably.
Rudolph: esp if you have a hackathon, if they want to come back a week later and see what they put into it. From that POV it makes sense. Altho usually hackathon playground, are just up for the few days of the workshop. So it's 4 cpus and 16GB of ram but we can def make that more permanent. We can just decide, have the one on fnndsc become more of a permanent resource. I think it's fine for now to have it available as a playground for a stretch of time as well.
Jorge: It's fine... we can make a kind of a single machine production deployment. was working in the past, maybe give me some access to that machinem, i don't have access right now. Really important to show devs the workflow of writing the plugins and submitting to the chris store.
Rudolph: I Was hoping you could take the lead on those instructions / write up, using the ChRIS store UI.
Jorge: Maybe try to do this once with your own plugin through the chris store UI, also running on fnndsc, not difficult at all.
Rudolph: I haven't done that for a long time, will try the UI.
Parul: Check with whatever the capacity of the VMs right now, how many concurrent users it supports. We should be prepared at least for 10 (or just decide on a number) for concurrent users a single instance of time. I think we shoul dhave a dry run of how many users can be supported yb the VM we have right now. Just an idea, we can discuss on it, or think on another solution so we have an approximation of how many users we can support so the hackathon is less chaotic.
Rudolph: Yeh that's a good idea
Jorge: The resources the machine has should be enough for 1000s of users, could be the network though...
Dan: do you have a perf run that shows how well it scales?
Jorge: no but, as I see typical amazon services machines... when you use them you get 32gb or 64gb ram machine...
Parul: it's not just hte machine, the capacity of the app as well, how many users can cube handle, we've never touched such a scenario.
Jorge: the slow part could be files coming back from pfcon?
Dan: Maybe let's not guess and just if we can perf test it, it's probably worth it.
Jorge: We can use the hackathon to see ;-) At least the ChRIS store will work fine, the most important part they'll interact with.
Dan: gut feel or do you have a data point there on chris store?
Jorge: The ChRIS store should be fine bc nothing compute intensive there, nothing extraordinary.
Dan: Im sure youre right but I've seen apps that are pretty simple that have flaws that cause issues too. It's ok either way, if we want to take a risk so be it, but could be worthwhile to perf test.
Rudolph: I totally agree.
Jorge: We were also going to discuss some optimizations we have in mind today.
Parul: these are good points, I'll make a note on it and follow up again some time later during our meeting in a few weeks.
== Note: Exposing JSON feed status data in UI ==
Parul: Another Q - not important but Rudolph, you were talking about JSON?
Rudolph: you mean the feed status feedback? That's an important one, from the UI perspective, we have the information already, so it shouldn't be too much to capture that. To provide more meaningful feedback to user.
Parul: I created an issue under MVP
Jorge Status
Backend modifications to enable front end to work with files
- I enabled the front end by working on the back end, to enable front end to choose files when in the new node UI, could choose files from everywhere in chris from the user's feed, filesystems, or the users upload filesystems. the backend also needs to be able to validate that any request is accessing files in an authorized path in swift, for that user. I implemented that and seems to be working fine now.
Discussion: Inefficiency in swift copy operations
-
[Jorge] Rudolph and I were talking about optimizations for ChRIS. Especially for plugins that only want to move files around in swift. Let's say currently a plugin that only wants to move a file from somwehwere in swift to somewhere else in swift - goes thorugh a very inefficient pipeline Let's say data first needs to be copied out of swift? Then data is sent to pfioh, then pman runs the plugin, plugin copies the data to its output there, pfioh sends data back to pfcon, pfcon put data back in swift again. All of that just to move a file from somewhere in swift, to somewhere else in swift. It's a very inefficient pipeline. So I was thinking about how we could optimize these things:
-
Basically a plugin that wants to work with swift, can only download a file from swift, process, and then upload it back to swift. I think that can be done by pfioh in the usual way. That use case is pretty much done properly by pfioh.
-
The other case is copying files from one place in swift to another place in swift. Important thing is a plugin can only write files to its output dir. That's the key in the ChRIS system. A plugin can only write files to its output dir. It means a plugin can not just go around in swift and access things, copying whatever it wants. That can make the system inconsistent and other things. Having that in mind... the best solution - instead of adding new services and credential secrets - i think we can just add a property to the path data type. If that property is true, then please don't download the files from swift but just copy them to the output dir of that plugin. The service doesn't know where the output dir of that plugin is going to be in swift...
-
Rudolph: There's a whole bunch of stuff here, let's not get too lost in the weeds bc conceptually and technically can get very complicated. Short story is, when we were putting together how to do the file upload and choose places in chris storage to create a new feed - it's a very powerful thing to do. Upload local files. What you cannot do right now but for arugment's sake - if you can select more than one dir in the chris file select. It's an extremely powerful operation here - it's a dircopy that creates a new root feed with info taken from somewhere else in the swift storage space. This comes down to what we've talked about, harvester plugins or some mechanism by which you can pull data from diff sources in the tree. Imagine ChRIS feed tree here - we're kind of making a JOIN between arbitrary data points in the tree. What we're doing in this dialog box here is pulling together data from diff sources across the tree which is extremely powerful. GO into it deeper, there's definitely a case to be made for some kind of behavior in the system where something we call a plugin is able to harvest data from multiple places in the tree. That's a bit of the discussion Jorge is touching on. I do think something like that owuld be super powerful to do. We've essentially done this but using an fs metaphor. If we could do this with a ds type plugin, we could link any kind of previous directory to a next node down the chain. Detail can be hairy and complex, but a very powerful thing to do. We don't know a good way in which to do it, but thinking along those lines. Could mean a new class of plugin or behavior in the system that doesn't yet exist. In our current metaphor / context / syntax we can't do it.
Jorge: That's part of the thing - we currently are able to copy files from one of those folders in the path in the ChRIS tree... that can be done by dsplugin or fsplugin. Currently system is able to do that. Multipe choices in the UI is a bit more complicated, bc it means you are sending a list of files to the backend, and that is difficult to put the maximum number of connectors on that. But we can do that actually with whatever in the chris path, choose a file or dir and will be made available for it to process. Rudolph. I was also talking about the other part of how the system works for these plugins that copy a file from swift to another place in swift. Pipeline is very inefficient for that.
Rudolph: Yes it's super inefficient for that.
Jorge: We were talking about that, maybe adding new services, bc you were thinking along the lines, maybe plugins could access swift directly. That would be very difficult, would imply creating new services, passing around credentials.
Rudolph: We're thinking about this and a lot of technical ipmlications. SImple one line message, we're thinking about a new way of internacting with the system not currently captured by the current design. IF anyone has any ideas or thinking about this - some mechanism by which you can do some operations without going through the whole pipeline of things we do now, you can boost efficiency by orders of magnitutde. A simple copy right now involves 5-6 network operations, when just involves copying things within swift. Should be almost instant but takes minutes to do right now. Our pipeline was never geared to do those kind of internal ops. That's the short story.
Jorge: Proposing a solution for that - adding a flag to a parameter so the system knows not to go thorugh the typical pipeline, and instead handle internally in swift. Would make pipeline more efficient. This is easy to do with amazon services, easy to do if you want to have swift local from private to public / hybrid cloud. my question is, is it easy to do this with the moc? Can we have a swift cluster there, on premise to the cloud? Only on-premise chris system that doesnt interact with the cloud - it's obvious having two swift clusters doesn't make sense. For hybrid cloud, also makes sense to have a single swift cluster, simplify a lot of the pipelines.
Rudolph: This is an important discussion and we need to get to it. Technical and hairy but could make a huge difference to the system.
Jorge: Yep, just want to get a take on how difficult this would be in the moc.
Rudolph. Probably not necessarily easy but probably doable, not happening likely before the next 6-8 months.
Parul: We can come back then to discuss.
Mo Status
Talked to Gideon and Rudolph about add new node mockups - would like a copy/paste freeform text field for specifying paramters + arguments to them rather than just the form. Considered having a 'mode' where you're in freeform vs form mode. Or allowing users to override freeform with the form. Mo will noodle on this and figure out an approach.
Mockups are available here: (https://github.com/FNNDSC/cube-design/tree/master/CHRIS-UI/PNG/add-new-node-2)[https://github.com/FNNDSC/cube-design/tree/master/CHRIS-UI/PNG/add-new-node-2]