Update SIM processing workflow - #855
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #855 +/- ##
==========================================
+ Coverage 54.51% 54.65% +0.13%
==========================================
Files 104 104
Lines 11131 11160 +29
Branches 1474 1475 +1
==========================================
+ Hits 6068 6099 +31
+ Misses 4742 4741 -1
+ Partials 321 320 -1 🚀 New features to boost your workflow:
|
tieneupin
added
enhancement
| machine_config.calibrations.get("pysimrecon_config") | ||
| ) | ||
| if not pysimrecon_config: | ||
| logger.error(f"No PySIMRecon configuration found for {instrument_name}") |
There was a problem hiding this comment.
If there aren't parameters, do you need to reject the processing or can you just let the service use defaults?
There was a problem hiding this comment.
That makes sense. I could provide the latest wavelength values we currently have as the default values, and the ones read from the MachineConfig could be updated as and when the instrument's calibrations are changed.
| # Construct the output directory for the PySIMRecon outputs to be saved to | ||
| try: | ||
| visit_idx = sim_data.file.parent.parts.index(visit_name) | ||
| raw_dir = sim_data.file.parents[-(visit_idx + 2)] |
There was a problem hiding this comment.
The -(visit_idx + 2) is correct, but confused me a lot initially
There was a problem hiding this comment.
Yes, I can appreciate that. Path().parents arranges the parent paths in descending order (immediate parent to root), so finding the parent based on index will have to start from the end of the list.
| try: | ||
| visit_idx = sim_data.file.parent.parts.index(visit_name) | ||
| raw_dir = sim_data.file.parents[-(visit_idx + 2)] | ||
| output_dir = ( |
There was a problem hiding this comment.
There's probably as easier way to do this - I think it's just a substitution of processed for raw*?
There was a problem hiding this comment.
Yes, that would be the most direct way. I have probably overengineered it slightly, but this method would allow for the code to still work if we decide to use a directory name other than "raw" in the future.
3b94457 to
f193c3c
Compare
3583032 to
99a2622
Compare
…of using Path().parents
99a2622 to
e2c2640
Compare
2 participants
Having fleshed out the PySIMRecon service in
cryoem-services(see DiamondLightSource/cryoem-services#286 and DiamondLightSource/cryoem-services#291), this PR updates therequest_sim_processingendpoint so that it includes the correct parameters in the outbound message.request_sim_processingAPI endpoint torequest_sim_reconstructionSIMContextlogic so that only SIM files ending with"_FL"are processed; raw data files without"_FL"will be transferred but skipped