27 Feb Status - More Hackathon planning and fsplugin UX discussion

The two major discussions we had during this meeting beyond the usual status reports was Hackathon planning, how to integrate fsplugins in the ChRIS UI, and Jorge's implementation of an extended path parameter type to enable a list of multiple files to be passed into a plugin to operate on.

Mo status

• Not much status to report, did another iteration of plugin config UI with Gideon: (https://github.com/FNNDSC/cube-design/tree/master/CHRIS-UI/PNG/add-new-new-3)[https://github.com/FNNDSC/cube-design/tree/master/CHRIS-UI/PNG/add-new-new-3]

Bill Status

• Bill has code review pending - Parul is still working on tensor flow, hasn't gotten to code review yet
• Bill has 10 interns out of Westford, working with Steve Huels, Sarah Coghlan & Hugh Brock - working on putting together bootcamp for new interns, so if anybody has interns coming in this summer, get in touch with Sarah and let them know and they can be included in this orientation session. New space in Boston will not be available in time for interns in Boston office this summer, will be tight.
• Bill will be unavailable after this Friday for 2 weeks on PTO... Parul has his mobile if needed to get in touch.
• Bill and Parul will sync up on what's next.

Rudolph Status

• Travelling last week and this week so not much specific to report

Discussion points:

• 1 - Hackathon dates: Thinking about a time & date for hackathon. Don't want to push out too long. Need to decide.
• 2 - fsplugins: How do we address the fsplugins in a way that doesn't have them scattered all over the place... maybe revive a design conversation we had a long time ago about fsplugins.
• 3 - How do we incorporate javsacript apps in ChRIS: Making progress on a PACS query/retrieve javascript app. Within Children's, having the PACS query/retrieve working internally an MVP. Most ppl use ChRIS right now to pull data from PACS, so getting this working will allow us to use the latest version in ChRIS internally, which we cannot currently do without this functionality. Another thing that's important to talk about from a design perspective - based on what I'm hitting up against with PACSpull, how do we incorporate these external apps seamlessly into ChRIS, since PACSpull is an important one.
• 4 - PACSpull/cube changes: Changes we're considering internally wrt CUBE that relate to this PACSpull request stuff. I can speak to it briefly, it's on the CUBE side.

Dates for hackathon

• Can push back to april-may. Conferences happen in June so people leave the area. Before summer would be optimal.
• Need to show system working end to end before hackathon. Is it reliable and can be tested in hackathon env?
  • Front end: depends on progress on front end - gideon will have plugin config mockups ready by end of week, has been working on plugin registration functionality
  • User registration: do we need to have user registration ready for hackathon? gideon unsure. rudolph thinks not required. we can stick w existing user.
  • Stress test: is there a way to stress test plugins running on MOC? rudolph says we don't have that at the moment, or have an automated set of tests that run through a couple of plugins... we need more feedback in the UI about what is happening in the MOC since it takes so long - 3 minute delay before job is executed. When MOC decides to schedule it, spins up init container, etc. If you see spinner for > 20s.
    • 3 minute job execution delay: dmcphers says it shouldn't take 3 minutes, that should be considered a bug. parul says yes it's a bug related to persistent volume issues, we need to figure it out
    • pman/pfioh stress testing: parul is doing pman/pfioh stress testing through openshift. script a little broken and she is working on it - by end of week should have some pfioh push and pman pull MOC perf findings. testing them directly and not through pfcon. Tried to do that earlier but ran into issues, sometimes pfcon doesn't provide a response with the scripts so it's hard to parse response and move to the next step. Have removed pfcon from the picture for now.
    • CCC course intern is working on getting pfioh/pman working on k/openshift (x86) on power9... the work won't be applicable to benchmark / perf testing.
  • ChRIS store plugin upload [jorge] Plugin developers need to be able to upload plugins to the ChRIS store, needs to be ready for hackathon. We have ChRIS store instance running at fnndsc.childrens.harvard.edu. It's an important part of the workflow.
  • Make decision on CUBE server django vs apache [jorge] CUBE is runing in a development environment rihgt now, not sure if what you are thinking for stress test. Is running django server there. We have a single machine prod env that we can run if you want based on apache. But the current ChRIS is running on the normal django devel server so it won't be as robust as a prod apache server. Make decision if we are going to stick with django dev server or to a prod apache server. CUBE server won't be hammered too hard by 10-12 hackfest participants. But would be worthwhile to investigate how much effort involved in switching from django to apache server. Can talk about next week.

TO-DOs for Hackathon

• Probably do not need to switch from django dev to apache prod server for front ends.
• Parul complete pman/pfioh stress testing.
• Need to fix 3 minute MOC job execution delay bug.
• Gideon to complete plugin registration work.
• Need to ensure ChRIS store plugin upload functioning

fsplugin Discussion

• Current feed create workflow... how do we run an arbitrary fsplugin. Crosses different conceptual domains... runing a pacs query, is to create a new feed, or just creating in the library. We're hitting up against that. Have PACSquery thing coming online soonish. How do we get it into ChRIS? Larger question - bunch of other fsplugins that just do something and create data - how do we run them with the current UI.

• classes of fsplugins that fall outside of these categories... generate a bunch of data from some internal process, have flags and create data. Basically a dsplugin workflow without input...

• [mizmo] there's a couple of spectrums as to how we could handle plugins in the UI:

  1. On one side is we think about who is producing the plugins:
     • We produce the plugins <===> Third-party produced plugins
  2. On the other side is how deeply the plugin functionality should/could be integrated into the UI:
     1. It's a default piece of the core UI
     2. It's an optional part of the core UI
     3. It integrates in ways visible across the core UI, but is installed and managed under an apps sandbox (similar to how WordPress plugins work)
     4. It exclusively lives in a plugins sandbox and does not integrate outside of that box
     5. It's a completely separate UI with different base URL connected to the main UI backend via APIs etc

• [mizmo] for a pacspull fsplugin specifically I would suggest the functionality live as a piece of the core UI or an optional component of the core UI under the 'library' tab. It can also exist where it currently lives in the create new feed workflow, or the new feed workflow tool can reference it via the library.

• [rudolph] what about other fsplugins that emit data, e.g. that produce reference data?

• [mizmo] I would also suggest treating these as library components. We could have different classes or categories within the library, and these could be categorized as 'reference' datasets. I would want to emphasize the content the fsplugin provides to users instead of how it is provided to them (whether that's an fsplugin output, a shared network filesystem mount point, etc.)

Jorge Status

• Internal storage in CUBE/ChRIS Swift storage... hospital-wide repository where you can put data from various hospital DBs. One thing I've been working on is all the file-related APIs in CUBE, such as user-uploaded files, plugin-generated files within a feed, new PACS file API I've been working on. Have to represent path to the file within internal Swift storage. Will expose within the APIs, the path within Swift storage, so plugins can now use that path to access any file within any feed or within the user storage space. We have to make sure the user running the plugin is authorized to access those paths. Many things will be in swift from different sources - CUBE will register those files with the APIs so those files are available to users in ChRIS.
• Also worked on optimization we wanted to implement. Created new parameter data type: xpath / extracted path - list of strings, paths separated by commas. Now you can pass to a plugin a list of paths in the internal storage so that the plugin can grab those files and do whatever it wants with it. All of this is part of an effort to make any file in Swift available to cube users.
  • extracted path data type can also indicate just copy from swift, do not go out to the MOC to optimize movement of files
  • [rudolph] once this is enabled.... topological implication. If we have the ability for cube to access files arbitrarily from swift storage, plugins can now in theory be able to pull from nodes that are not their direct predecessor. Eliminates need for bucket brigade. Plugins can work with data of different ancestry down the tree.
• At some point we need to think about the idea of a single swift as the backbone between the remote compute environment (MOC) and the ChRIS servers inside the hospital.
• Added a new PACSfile django app to cube to manage files downloaded from the hospital pacs. So he has some services and apps that can download files from the hospital pacs and store them in swift. Those files can now be registered with cube so they are available to the users of ChRIS. They will also be searchable using PACS query parameters.
• Next thing I'm working on now - will be added an asynchronous broker to cube, to enable a sync process running in a diff container or machine to register files. Right now, when the files come back from pfcon and they need to be registered in cube, it makes a get request that takes quite some time, shouldn't take more than a fraction of a section. So we will offland that process to an external containers.