We went through everyone's status, and The main items of discussion during this meeting were the new asynchronous task queue Jorge developed and further work needed for the frontend handling of that, and we also walked through some mockups on how to integrate fsplugins in the UI."

• Want to show Mo a demo of how the new PACS lookup tool works, after meeting because not ready yet
• For usage of ChRIS inside institution, PACS lookup is the main way people interact with the system, so it's an important workflow to get done and get done right.
• New member of the team : Sandip. Will start out developing plugins. Worked with him end of last year - was a grad student in cloud computing course with Parul. Plugins will work on deployment of network for cementing brain images. Will try to productize and have as a usable workflow. He will start by looking at that.

Gideon Status

• Still working on adding plugin node workflow
• Once that's done, tomorrow will start registering the plugins, one that is created in the lab by someone else. See if the workflow works end to end, with a demo likely next Thursday on the entire workflow
• Entire workflow: from adding a new node, to registering someone else's node, then adding their dataset to swift storage and run their plugin through the new wizard that Mo mocked up

Jorge Status

New Asynchronous Task Queue in CUBE

• Implementing asynchronous task queue for CUBE. Have it almost totally implemented now. It's working and works fine. This queue is to offload some of the work out of the CUBE container into a separate ocntainer. For example - registering output files with CUBE can be done from a separate container. That should improve the UX because the requests will resolve more quickly.
• The main issue here is with integration tests... the workers for asynchronous queue don't pick up Django temporary testing DB, they pick up the normal DB. Django is testing with one DB, this is working with another. Getting those integration tests working now, otherwise whole thing is working nicely.
• [Rudolph] This adds some extra complexity but we believe the payoff is worth it. There are some actions in CUBE that are super important, registering all the files after the plugin is finished and they come back. Unfortunately these are synchronous, so it freezes the web front end for 10 seconds registering files into Swift. So we're offloading it to threaded / async workers so there's no lag visible to the user during file registration.
• [Jorge] Very common in web apps... server shouldn't spend more of a fractoin of a second attending each req. Any req that requires data processing or external REST API calls, etc. should be offloaded to asynchronous worker. Everything is about contacting pfconf, so pfcon has to contact remote services. Everything outside of CUBE will be offlanded to async queue.
• [Jorge] There is a GET request to check running status of task in remote compute..... that GET req has to contact remote services. If everything is finished... come back and register file.
• [Dan] Is the result of registering the files previously required and now will have to be polled in a second request?
  • [Jorge] User made a GET req, internal svcs contact remote. IF the job didn't finish in the remote, then the req came back immediately saying 'status started' - then front end kept polling (not sure how it's implemented now) until the GET request comes back with successful finish status.
  • [Jorge] We could implement as polling directly from the workers, polling the backend services instead of the front end doing it. Not sure.
  • [Rudolph] When you're running a plugin, all of that handled by pfcon. When it's finished, pfcon will have received all the data and packed into swift. But CUBE doesn't pull or know about it - runs async to CUBE. Only way CUBE knows is that in the front end theres a GET status to find out the current status - it asks pfcon. So no polling in the system, all external event driven. At some point it's finished and sitting in swift. If pfcon says it's done, then and only then, will CUBE in its thread of exec will register the files sitting in swift to itself.
  • [Dan] if there's no polling from client to swift side.... [1] hasn't finished yet [2] finished but not ready to return result bc registering it [3] finished and reg finished. So at a minimum client has to ask twice - when it's done but results not registered.... then again when results are done and registered. So you'll never get the result on the first try unless it finished an hour ago.
  • [Jorge] Once the plugin makes the GET request, entire thing is pushed off to the async service .Will come back with not finished yet - async svc will contact backend, will know that the status is finished, db updated, all files downloaded, then next time when the user makes another GET request, it says yes finished successfully - files will be there.
  • [Dan] The weird part I'm describing - it'll never be there on the first request, because the first request kicks off the registration process?
  • [Jorge] It never says it's done if the files are not there
  • [Dan] Let's say it finished an hour ago. IF I make request now, it'll say done but not registered.... even if finished 2 months ago... then I'll be confused... that delay is a weird UX.
  • [Rudolph] It's so client dependent bc didn't want CUBE To internally poll bc it would hit synchronous thread... it'll freeze for X amount of time bc it hits before file reg is done.
  • [ACTION] We should adjust the expreience so when the user comes back they don't always have to wait for registering on the fly.

Mizmo Status

Library mockups for fsplugin UI integration

• Gave a quick demo of rough library mockups to demonstrate how fsplugins that output reference data could be integrated into the ChRIS UI:
• Overall workflow similar to RAW image processing tools like Adobe Lightroom or Darktable - you gather the raw data first, then go to processing (feeds)

1. This is a very rough sketch of what the library would look like. From here you can do a new PACS lookup, upload local data, or search for reference data (which is essentially searching for fsplugins that output reference data). You can also view a list of data you accessed via any of these methods in reverse chronological order.
2. This is a very rough sketch of how the reference lookup could work. Essentially this is a searchable listing of the fsplugins available on the system that output reference data.
3. Once you've selected the data you want, you should be able to:
   • Create a new feed straight from the data or
   • Go to the feeds tab at a later time, click create a new feed, and access the data you retrieved into your library there

Next steps

• [ACTION] Continue with the library mockups and iterate on them
• [ACTION] take a look at Rudolph's PACS search demo and mock up how to integrate into the library

Sandip Status

(I missed this because I had to step off the call for a moment. :( )

Parul Status

• Finished PR, couple of comments on tensorflow work, finished it off.

• Close to finishing off health check app to check the services on the MOC. Some trouble with the MOC and working with intern to get result. Once resolved, will get idea of the perf of the services running on MOC.

• Having trouble running simple-dsapp on MOC. Getting python file not found in path error. Should not be the case, use #! in the file and should be running itself, so exec shouldn't be happening. Looking into that. First thinking of running a different plugin to see if it's this plugin only or an issue with MOC.

• Had a workshop on Tuesday and it went well - the MOC workshop - a lot of researchers from the medical college at BU and seemed pretty interested in the ChRIS framework. We have an interested party who might be willing to explore a project. We extended invitations to the hackathon. And at least now we have contacts we can reach out to - inviting grad students in the med college would give them good exposure to ChRIS.

• Before we close I want to recap what we achieved for hackathon and what we have to do:

  • next week demo for new plugin registration
  • Parul working on MOC perf. Rudolph, a script for performance would be good.

• [Rudolph] Looking at domains available to have a more established presence on the web instead of our little DMZ machine here. Were thinking about getting domains and getting an ISP for it. A standard open source project setup with info - docs, repos, etc.

• [Jorge] Two websites - .org for everything about ChRIS, and one for the ChRIS store as a unique entity, will be unique for all ChRIS instances.

Dan Status

• Have nothing to add