feat: add CMIP6-to-CMIP7 Data Request variable mappings #530

lewisjared · 2026-02-12T11:42:31Z

Description

Add structured CMIP6-to-CMIP7 variable mappings sourced from the CMIP7 Data Request (DReq).

In the data request there are unique mappings of cmip6_compound_name ({table_id}.{variable_id}) to cmip7_compound_name. This is now the source of truth for mapping to branded variables.

Key changes:

extract-data-request-mappings.py script: Downloads the DReq release export, extracts variable mappings filtered to mon/fx tables and REF provider variables, and writes the bundled JSON.
Bundled JSON data (cmip6_cmip7_variable_map.json): Pre-extracted subset of DReq mappings shipped with climate-ref-core.
CMIP7 conversion utilities: Functions for branding suffix lookup, realm/out_name resolution, attribute conversion, filename/path generation per MIP-DRS7 spec.

Checklist

Please confirm that this pull request has done the following:

Tests added
Documentation added (where applicable)
Changelog item added to changelog/

Copilot

Pull request overview

Adds bundled CMIP6→CMIP7 Data Request (DReq) variable mappings and updates Climate REF’s CMIP7 conversion + ESMValTool provider/diagnostics so CMIP7 datasets can be handled via OR-logic data requirements and correct CMIP7 DRS/filename conventions.

Changes:

Introduces a frozen DReqVariableMapping model and loads a bundled cmip6_cmip7_variable_map.json at import time for branding/realm/compound-name lookups.
Adds/updates conversion and filename/path generation to align with CMIP7 (MIP-DRS7) and updates CMIP7 conversion caching behavior.
Updates ESMValTool diagnostics/recipe/config handling to accept either CMIP6 or CMIP7 inputs, plus provider setup to install pinned ESMValTool/ESMValCore via pip URLs.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
scripts/extract-data-request-mappings.py	New script to download/filter DReq export and generate the bundled mapping JSON.
scripts/create-cmip7-datasets.py	Writes CMIP7-style filenames when converting sample datasets.
packages/climate-ref-esmvaltool/tests/unit/diagnostics/test_base.py	Updates expectations for the generated ESMValTool config (projects/search settings).
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/recipe.py	Adds CMIP7 facet mapping and pins ESMValTool/ESMValCore git URLs.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/base.py	Adds CMIP6/CMIP7 selector helper + rewrites ESMValTool config to include CMIP7 local templates.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/zec.py	Adds CMIP7 alternative requirements + selects CMIP source dynamically for recipe updates.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/tcre.py	Adds CMIP7 alternative requirements + selects CMIP source dynamically.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/tcr.py	Adds CMIP7 alternative requirements + selects CMIP source dynamically.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/sea_ice_sensitivity.py	Adds CMIP7 alternative requirements + selects CMIP source dynamically.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/sea_ice_area_basic.py	Adds CMIP7 alternative requirements + selects CMIP source dynamically.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/regional_historical_changes.py	Adds CMIP7 requirements + CMIP7 ESGF test cases using `CMIP7Request`.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/example.py	Adds CMIP7 alternative requirements + selects CMIP source dynamically.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/enso.py	Adds CMIP7 alternative requirements + selects CMIP source dynamically.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/ecs.py	Adds CMIP7 alternative requirements + selects CMIP source dynamically.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/cloud_scatterplots.py	Generalizes requirements to CMIP6+CMIP7; suptitle now uses project.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/cloud_radiative_effects.py	Adds CMIP7 alternative requirements + selects CMIP source dynamically.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/climate_drivers_for_fire.py	Adds CMIP7 alternative requirements + selects CMIP source dynamically.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/diagnostics/climate_at_global_warming_levels.py	Adds CMIP7 alternative requirements + CMIP7-specific grouping/matching facets in recipe update.
packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/init.py	Switches provider setup to install ESMValTool/ESMValCore via `pip_packages`.
packages/climate-ref-core/tests/unit/test_providers.py	Adds unit test coverage for `pip_packages` installation behavior.
packages/climate-ref-core/tests/unit/test_cmip6_to_cmip7.py	Reworks tests around DReq-backed branding/realm/compound-name lookups + serialization.
packages/climate-ref-core/tests/unit/esgf/test_cmip7.py	Updates CMIP7 conversion tests for new filename behavior.
packages/climate-ref-core/src/climate_ref_core/providers.py	Replaces single dev install URL with a list of `pip_packages` installed post-conda-create.
packages/climate-ref-core/src/climate_ref_core/esgf/cmip7.py	Generates CMIP7-style output filename for cached conversions.
packages/climate-ref-core/src/climate_ref_core/data/cmip6_cmip7_variable_map.json	Adds bundled subset of DReq mappings shipped with the package.
packages/climate-ref-core/src/climate_ref_core/cmip6_to_cmip7.py	Loads bundled DReq mappings; updates branding/realm/compound-name logic; adds CMIP7 filename/path helpers.
changelog/519.feature.md	Changelog entry for CMIP7 support via OR-logic requirements.
.vscode/settings.json	Editor setting update for Python REPL smart send.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/extract-data-request-mappings.py

packages/climate-ref-core/src/climate_ref_core/data/cmip6_cmip7_variable_map.json

packages/climate-ref-core/src/climate_ref_core/esgf/cmip7.py

scripts/create-cmip7-datasets.py

packages/climate-ref-core/src/climate_ref_core/providers.py

scripts/extract-data-request-mappings.py

codecov · 2026-02-12T11:53:48Z

Codecov Report

❌ Patch coverage is 95.71429% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...limate-ref-core/src/climate_ref_core/esgf/cmip7.py	81.81%	2 Missing ⚠️
...te-ref-core/src/climate_ref_core/cmip6_to_cmip7.py	98.30%	0 Missing and 1 partial ⚠️

Files with missing lines	Coverage Δ
...te-ref-core/src/climate_ref_core/cmip6_to_cmip7.py	`92.35% <98.30%> (+4.94%)`	⬆️
...limate-ref-core/src/climate_ref_core/esgf/cmip7.py	`95.00% <81.81%> (-1.74%)`	⬇️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…ppings Replace raw dict usage with a frozen attrs class for type-safe serialisation/deserialisation of Data Request variable mappings. The class is used both in the extract script (to_dict for JSON output) and at load time (from_dict when reading the bundled JSON).

…ping functions

- Replace __file__ with stable relative path in extract script metadata - Fix bundled JSON description containing absolute path - Validate branding suffix format before splitting in extract script - Fix docstring to match raise-on-duplicate behavior - Move cache check before xr.open_dataset in CMIP7 converter

The function was using cmip6_path.name for the output filename instead of generating a proper CMIP7 filename. Import and use create_cmip7_filename to produce correct CMIP7 DRS filenames.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

packages/climate-ref-core/src/climate_ref_core/esgf/cmip7.py

packages/climate-ref-core/tests/unit/esgf/test_cmip7.py

scripts/extract-data-request-mappings.py

packages/climate-ref-core/src/climate_ref_core/cmip6_to_cmip7.py

packages/climate-ref-core/tests/unit/esgf/test_cmip7.py

- Fix docstring grammar in extract script - Add 60s timeout to urllib.request.urlopen - Add assert_not_called checks for mock_open/mock_convert in cache test

…nstruction _convert_file_to_cmip7 now looks up the DReq entry using table_id and variable_id to inject branding_suffix and region into the facets before calling create_cmip7_filename. This prevents empty branding components in the generated filenames. Tests no longer mock create_cmip7_filename, instead providing table_id in the facets so real filename construction is exercised.

For variables where out_name differs from variable_id (e.g. tasmax -> tas), the filename and DRS path now correctly use the CMIP7 out_name. The variable_id attribute in the dataset stays as the CMIP6 identity. - create_cmip7_filename and create_cmip7_path prefer out_name over variable_id (with fallback) - convert_cmip6_to_cmip7_attrs sets out_name and branded_variable from the DReq entry - _convert_file_to_cmip7 injects out_name during DReq enrichment - Added end-to-end tests for tasmax filename generation

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

packages/climate-ref-core/src/climate_ref_core/esgf/cmip7.py

packages/climate-ref-core/src/climate_ref_core/cmip6_to_cmip7.py

packages/climate-ref-core/tests/unit/test_cmip6_to_cmip7.py

Copilot · 2026-02-12T12:54:58Z

packages/climate-ref-core/src/climate_ref_core/esgf/cmip7.py

        str(cmip7_facets.get("frequency", "mon")),
        str(cmip7_facets.get("variable_id", "tas")),
        str(cmip7_facets.get("grid_label", "gn")),


output_file is now generated from facets via create_cmip7_filename(cmip7_facets) but no time_range is passed. For multi-file datasets split by time (typical on ESGF), every slice for the same variable/experiment will map to the same CMIP7 filename and collide/overwrite in the cache directory. Consider extracting the time range from cmip6_path.name (or from the dataset’s time coordinate) and passing it through to create_cmip7_filename, or otherwise incorporating the original CMIP6 timerange suffix to keep filenames unique per file.

- Make _get_dreq_entry a public API (get_dreq_entry) for cross-module use - Use out_name in DRS path construction in _convert_file_to_cmip7 - Use DReq region instead of hardcoded 'glb' in convert_cmip6_to_cmip7_attrs - Fix pytest parametrize ids to use pytest.param with explicit id strings - Add tests for out_name != variable_id (tasmax/tasmin) and non-glb region (ImonAnt)

…ata Request API

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

packages/climate-ref-core/src/climate_ref_core/esgf/cmip7.py:71

The inline comment describing the CMIP7 DRS path still says the variable component is {variable_id}, but the code now uses out_name (and the core create_cmip7_path uses out_name too). Update the comment to avoid documenting the wrong facet and confusing future changes.

    # Build CMIP7 DRS path
    # CMIP7 DRS: {activity_id}/{institution_id}/{source_id}/{experiment_id}/
    #            {variant_label}/{frequency}/{variable_id}/{grid_label}/{version}
    # Ensure all facet values are strings (some may be integers from metadata)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-12T13:27:34Z

packages/climate-ref-core/src/climate_ref_core/esgf/cmip7.py

+    # Create output filename and check cache before opening the source file
+    output_file = drs_path / create_cmip7_filename(cmip7_facets)



create_cmip7_filename(cmip7_facets) will emit empty components when keys like source_id/experiment_id/variant_label are missing (it defaults to empty strings), producing filenames with double underscores. Since _convert_file_to_cmip7 already applies defaults when building drs_path, consider normalizing cmip7_facets with the same defaults before calling create_cmip7_filename so cache paths/filenames are consistent and spec-like.

packages/climate-ref-core/src/climate_ref_core/cmip6_to_cmip7.py

Copilot · 2026-02-12T13:27:36Z

scripts/extract-data-request-mappings.py

+    }
+
+    output.parent.mkdir(parents=True, exist_ok=True)
+    with open(output, "w") as f:


When writing the JSON output, consider opening the file with an explicit encoding="utf-8" to make the script’s output deterministic across platforms/locales.

Suggested change

with open(output, "w") as f:

with open(output, "w", encoding="utf-8") as f:

…ments * origin/dr-mappings: fix: remove redundant information from breaking changelog entry for Data Request API docs: add breaking changelog entry for DReq API changes fix: address third round of PR review comments fix: use out_name from DReq for CMIP7 filenames and paths fix: enrich cmip7_facets with DReq branding_suffix before filename construction chore: exclude setting chnage fix: address second round of PR review comments fix(core): use create_cmip7_filename in _convert_file_to_cmip7 fix: address PR review comments docs: add changelog entry for PR #530 refactor(core): update documentation and remove unused CMIP7 name mapping functions feat(core): add DReqVariableMapping attrs class for CMIP6-to-CMIP7 mappings # Conflicts: # packages/climate-ref-core/src/climate_ref_core/cmip6_to_cmip7.py # packages/climate-ref-core/src/climate_ref_core/esgf/cmip7.py

Copilot AI review requested due to automatic review settings February 12, 2026 11:42

Copilot started reviewing on behalf of lewisjared February 12, 2026 11:42 View session

lewisjared added a commit that referenced this pull request Feb 12, 2026

docs: add changelog entry for PR #530

52c7275

Copilot AI reviewed Feb 12, 2026

View reviewed changes

lewisjared added 4 commits February 12, 2026 22:56

refactor(core): update documentation and remove unused CMIP7 name map…

8b2ca0a

…ping functions

docs: add changelog entry for PR #530

2ee82bc

lewisjared force-pushed the dr-mappings branch from 52c7275 to e327bfa Compare February 12, 2026 12:07

fix(core): use create_cmip7_filename in _convert_file_to_cmip7

5a14fcf

The function was using cmip6_path.name for the output filename instead of generating a proper CMIP7 filename. Import and use create_cmip7_filename to produce correct CMIP7 DRS filenames.

Copilot AI review requested due to automatic review settings February 12, 2026 12:12

Copilot started reviewing on behalf of lewisjared February 12, 2026 12:13 View session

lewisjared requested a review from bouweandela February 12, 2026 12:16

Copilot AI reviewed Feb 12, 2026

View reviewed changes

lewisjared added 3 commits February 12, 2026 23:33

fix: address second round of PR review comments

5df9bc4

- Fix docstring grammar in extract script - Add 60s timeout to urllib.request.urlopen - Add assert_not_called checks for mock_open/mock_convert in cache test

chore: exclude setting chnage

6fc315a

Copilot AI review requested due to automatic review settings February 12, 2026 12:47

Copilot started reviewing on behalf of lewisjared February 12, 2026 12:49 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

lewisjared added 2 commits February 13, 2026 00:10

docs: add breaking changelog entry for DReq API changes

936a553

Copilot AI review requested due to automatic review settings February 12, 2026 13:19

fix: remove redundant information from breaking changelog entry for D…

8624ebf

…ata Request API

Copilot started reviewing on behalf of lewisjared February 12, 2026 13:20 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

		# Create output filename and check cache before opening the source file
		output_file = drs_path / create_cmip7_filename(cmip7_facets)

	with open(output, "w") as f:
	with open(output, "w", encoding="utf-8") as f:

feat: add CMIP6-to-CMIP7 Data Request variable mappings #530

Are you sure you want to change the base?

feat: add CMIP6-to-CMIP7 Data Request variable mappings #530

Uh oh!

Conversation

lewisjared commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

lewisjared commented Feb 12, 2026 •

edited

Loading

codecov bot commented Feb 12, 2026 •

edited

Loading