Debug: imaris_to_ome_zarr tweaks + add KI3 batch2 samples#70
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes a bug in imaris_to_ome_zarr.py where zarr.zip input files were not being opened correctly as ZipStore objects, causing FileExistsError when zarr attempted to write metadata to what it incorrectly treated as a directory. The fix ensures consistent handling with how the files are written in imaris_channel_to_zarr.py. Additionally, the PR adds a new batch of KI3 samples to the configuration.
Changes:
- Fixed ZipStore handling in
imaris_to_ome_zarr.pyby explicitly creating ZipStore objects with mode='r' for reading input zarr.zip files - Added new sample configuration file
samples_ki3_batch2.tsvwith 11 mouse brain samples from the KI3 batch 2 dataset
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| workflow/scripts/imaris_to_ome_zarr.py | Fixed ZipStore handling to explicitly open input zarr.zip files as ZipStore with read mode, consistent with other scripts in the codebase |
| config/samples_ki3_batch2.tsv | Added configuration for 11 new KI3 batch2 samples, but missing required 'sample' column in header and data rows |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
config/samples_ki3_batch2.tsv
Outdated
| @@ -0,0 +1,12 @@ | |||
| subject acq stain_0 stain_1 stain_2 sample_path | |||
There was a problem hiding this comment.
The header is missing the 'sample' column that is present in config/samples.tsv and is used throughout the workflow (e.g., workflow/rules/common.smk lines 99, 117, 134 and workflow/scripts/generate_aggregate_qc.py lines 18-19). This will cause KeyError exceptions when the workflow tries to access samples.loc[i, "sample"]. The header should be: subject, sample, acq, stain_0, stain_1, stain_2, sample_path
Suggested change
| subject acq stain_0 stain_1 stain_2 sample_path | |
| subject sample acq stain_0 stain_1 stain_2 sample_path |
config/samples_ki3_batch2.tsv
Outdated
Comment on lines
1
to
12
| subject acq stain_0 stain_1 stain_2 sample_path | ||
| AS9F7 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/A_AS9F7/16-51-02_a-AS9F7 IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | ||
| AS11F4 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/B_AS11F4/22-49-58_b-AS11F4 IK3 PBS IBA1 647 ab RedX CD31 488 4X1 stitched.ims | ||
| AS37F30 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/C_AS37F30/07-04-14_c-AS37F30 IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | ||
| AS8M1 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/D_AS8M1/15-34-54_d AS8M1 IK3 PBS IBA1 647 ab RedX CD31 488 1X1_Blaze_C00.ome.ims | ||
| AS8M4 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/E_AS8M4/21-43-06_e AS8M4 IK3 PBS IBA1 647 ab RedX CD31 488 1X1_Blaze_C02.ome.ims | ||
| AS7F2 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/F_AS7F2/16-54-24_f AS7F2 IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | ||
| AS11F1e imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/G_AS11F1e/23-36-04_g AS11F1e IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | ||
| AS11F3 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/H_AS11F3/06-11-22_h AS11F3 IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | ||
| AS39Me0 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/J_AS39Me0/12-43-19_j as39me0 (54) ik3 lecanemab iba1 647 ab redx cd31 488 4x1stitched.ims | ||
| AS7F1 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/K_AS7F1/k AS 7 F1 IK3 neg ctrl BA1 647 ab RedX CD31 488 4X1 stitched.ims | ||
| AS39Me2 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/i_AS39Me2/i AS39Me2 IK3 lecanemab IBA1 647 ab RedX CD31 488 4X1 stitched.ims |
There was a problem hiding this comment.
All data rows are missing the 'sample' column value. Based on config/samples.tsv, this should be inserted between the 'subject' and 'acq' columns. The existing file uses "brain" as the sample value for similar mouse brain samples. Each row should have a sample value (e.g., "brain") added as the second column.
Suggested change
| subject acq stain_0 stain_1 stain_2 sample_path | |
| AS9F7 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/A_AS9F7/16-51-02_a-AS9F7 IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | |
| AS11F4 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/B_AS11F4/22-49-58_b-AS11F4 IK3 PBS IBA1 647 ab RedX CD31 488 4X1 stitched.ims | |
| AS37F30 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/C_AS37F30/07-04-14_c-AS37F30 IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | |
| AS8M1 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/D_AS8M1/15-34-54_d AS8M1 IK3 PBS IBA1 647 ab RedX CD31 488 1X1_Blaze_C00.ome.ims | |
| AS8M4 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/E_AS8M4/21-43-06_e AS8M4 IK3 PBS IBA1 647 ab RedX CD31 488 1X1_Blaze_C02.ome.ims | |
| AS7F2 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/F_AS7F2/16-54-24_f AS7F2 IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | |
| AS11F1e imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/G_AS11F1e/23-36-04_g AS11F1e IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | |
| AS11F3 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/H_AS11F3/06-11-22_h AS11F3 IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | |
| AS39Me0 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/J_AS39Me0/12-43-19_j as39me0 (54) ik3 lecanemab iba1 647 ab redx cd31 488 4x1stitched.ims | |
| AS7F1 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/K_AS7F1/k AS 7 F1 IK3 neg ctrl BA1 647 ab RedX CD31 488 4X1 stitched.ims | |
| AS39Me2 imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/i_AS39Me2/i AS39Me2 IK3 lecanemab IBA1 647 ab RedX CD31 488 4X1 stitched.ims | |
| subject sample acq stain_0 stain_1 stain_2 sample_path | |
| AS9F7 brain imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/A_AS9F7/16-51-02_a-AS9F7 IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | |
| AS11F4 brain imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/B_AS11F4/22-49-58_b-AS11F4 IK3 PBS IBA1 647 ab RedX CD31 488 4X1 stitched.ims | |
| AS37F30 brain imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/C_AS37F30/07-04-14_c-AS37F30 IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | |
| AS8M1 brain imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/D_AS8M1/15-34-54_d AS8M1 IK3 PBS IBA1 647 ab RedX CD31 488 1X1_Blaze_C00.ome.ims | |
| AS8M4 brain imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/E_AS8M4/21-43-06_e AS8M4 IK3 PBS IBA1 647 ab RedX CD31 488 1X1_Blaze_C02.ome.ims | |
| AS7F2 brain imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/F_AS7F2/16-54-24_f AS7F2 IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | |
| AS11F1e brain imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/G_AS11F1e/23-36-04_g AS11F1e IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | |
| AS11F3 brain imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/H_AS11F3/06-11-22_h AS11F3 IK3 PBS IBA1 647 ab RedX CD31 488 4X1_Blaze_C00.ome.ims | |
| AS39Me0 brain imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/J_AS39Me0/12-43-19_j as39me0 (54) ik3 lecanemab iba1 647 ab redx cd31 488 4x1stitched.ims | |
| AS7F1 brain imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/K_AS7F1/k AS 7 F1 IK3 neg ctrl BA1 647 ab RedX CD31 488 4X1 stitched.ims | |
| AS39Me2 brain imaris4x IBA1 Abeta CD31 /nfs/trident3/lightsheet/prado/mouse_app_lecanemab_ki3_batch2/raw/ims_4x_stitched/i_AS39Me2/i AS39Me2 IK3 lecanemab IBA1 647 ab RedX CD31 488 4X1 stitched.ims |
Member
|
Removed the sample tsv for now |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
imaris_channel_to_zarr.pycorrectly writes intermediate channel files aszarr.zipusingzarr.ZipStore(path, mode='x'), butimaris_to_ome_zarr.pywas reading those same files by passing the path string directly tozarr.open()andda.from_zarr(). Neither call auto-detects a.zarr.zippath as a ZipStore — they default to treating it as a directory store with mode'a', which causes zarr to attempt writing.zgroupmetadata into what is actually a zip file, resulting in aFileExistsError.This is made worse by the
shadow: "minimal"directive on theimaris_to_ome_zarrrule, which symlinks the input.zarr.zipfiles into the shadow directory — the file-vs-directory conflict happens immediately on open.Fix:
Explicitly open the input zarr files as
ZipStoreobjects in read mode before passing them tozarr.openandda.from_zarr, consistent with how the writer script already handles them.