Chardata first working on feature branch#6975
Chardata first working on feature branch#6975pp-mo wants to merge 65 commits intoSciTools:FEATURE_chardatafrom
Conversation
…Mostly working? Get 'create_cf_data_variable' to call 'create_generic_cf_array_var': Mostly working?
Rename; addin parts of old investigation; add temporary notes.
…or overlength writes.
…t_cf_var_data' function.
pp-mo
changed the title
There was a problem hiding this comment.
Looking pretty good. 👍🏼
I've got a few comments, questions and suggestions.
I have not looked at the tests yet - just the main Iris code. I thought it was worth submitting the review at this point so you can see the comments. I'll take a look at the tests next.
Also - remind me - what are we doing in the case where data is stored as a netCDF string type - i.e. the variable length string type? At the moment that just loads in as an object array in numpy. Were we just leaving that as-is? We can't write that kind of datatype in Iris.
Edit: Discussed with @pp-mo and he reminded me that we never intended to handle the variable length string cases.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## FEATURE_chardata #6975 +/- ##
====================================================
+ Coverage 90.11% 90.18% +0.07%
====================================================
Files 91 92 +1
Lines 24912 25092 +180
Branches 4675 4689 +14
====================================================
+ Hits 22449 22629 +180
- Misses 1684 1688 +4
+ Partials 779 775 -4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ukmo-ccbunney
left a comment
ukmo-ccbunney
left a comment
There was a problem hiding this comment.
Tests looks sensible as far as I can tell, with the expectation that there will be more coverage added as part of #6898.
| # Except if it already is one, since they forbid "re-wrapping". | ||
| if not hasattr(self._dataset, "THREAD_SAFE_FLAG"): | ||
| self._dataset = _thread_safe_nc.DatasetWrapper.from_existing( | ||
| self._dataset = bytecoding_datasets.DatasetWrapper.from_existing( |
There was a problem hiding this comment.
Oops.
I think this should possibly (also) be an 'EncodedDataset'.
I need to think about this one, I guess it depends on what kind of 'dataset-like' is passed in here.
| self._dataset = bytecoding_datasets.DatasetWrapper.from_existing( | |
| self._dataset = bytecoding_datasets.EncodedDataset.from_existing( |
| # Create a data-writeable object that we can stream into, which | ||
| # encapsulates the file to be opened + variable to be written. | ||
| write_wrapper = _thread_safe_nc.NetCDFWriteProxy( | ||
| write_wrapper = bytecoding_datasets.EncodedNetCDFWriteProxy( |
There was a problem hiding this comment.
| write_wrapper = bytecoding_datasets.EncodedNetCDFWriteProxy( | |
| # Note: we do *not* support selectable string encoding for writes, | |
| # so this never needs to be a _thread_safe_nc.NetCDFWriteProxy. | |
| write_wrapper = bytecoding_datasets.EncodedNetCDFWriteProxy( |
Co-authored-by: Patrick Peglar <patrick.peglar@metoffice.gov.uk>
|
Nearly there, I think. Edit: The docs failures might be a transient error - seems to be related to a link failure in the InterSphinx links: https://pandas.pydata.org seems to be unresponsive at the time of wtriting. |
2 participants
Successor to #6898
Now targetting (new) FEATURE_chardata feature branch in main repo
TODO: please check that any remaining unresolved issues on #6898 are now resolved here