CAL spectra : daily energy calibration #427

nicHoch · 2025-07-22T15:54:50Z

this PR started a while ago as first version of general L2 products as QL calibration spectra and sci-spectra as L2

then it moved its goal to daily energy calibration files that have become an own "level": CAL

so some code parts for L2 files are commented out to be eneabled later again with our first L2 products

codecov-commenter · 2025-07-22T15:57:44Z

Codecov Report

❌ Patch coverage is 33.64929% with 140 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.14%. Comparing base (3ae9671) to head (2e3480b).
⚠️ Report is 12 commits behind head on master.

Files with missing lines	Patch %	Lines
stixcore/products/CAL/energy.py	27.27%	112 Missing ⚠️
stixcore/products/product.py	30.76%	9 Missing ⚠️
stixcore/products/level2/scienceL2.py	63.63%	8 Missing ⚠️
stixcore/products/level0/scienceL0.py	0.00%	6 Missing ⚠️
stixcore/processing/pipeline_cron.py	25.00%	3 Missing ⚠️
stixcore/processing/L1toL2.py	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #427      +/-   ##
==========================================
- Coverage   72.23%   68.14%   -4.10%     
==========================================
  Files          78       78              
  Lines        8085     8296     +211     
==========================================
- Hits         5840     5653     -187     
- Misses       2245     2643     +398

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

stixcore/products/level2/scienceL2.py

samaloney

The managers are all very similar I wonder if we shouldn't make a general manager and then have specific versions?

Also I think we should avoid adding more of the singleton pattern, I know I was the one who started with it but it really an anti-pattern in python in general.

stixcore/calibration/ecc_post_fit.py

stixcore/products/level1/scienceL1.py

stixcore/soop/manager.py

Copilot

Pull request overview

This PR introduces a new CAL processing “level” focused on generating daily energy calibration products (migrating prior “L2 energy calibration” intent into CAL), and adds supporting calibration/config management utilities.

Changes:

Add a new CAL EnergyCalibration product implementation and adjust FITS naming/writing/publishing to include CAL outputs.
Introduce ECC/ELUT manager utilities (plus tests) to support date-dependent calibration configuration and ELUT lookup.
Re-enable parts of L2 processing and add scaffolding for L2 science spectrogram handling and extra FITS extensions.

Reviewed changes

Copilot reviewed 24 out of 26 changed files in this pull request and generated 20 comments.

Show a summary per file

File	Description
`stixcore/soop/manager.py`	Temporarily disables SOOP API fallback when the cached “all” file is missing.
`stixcore/products/product.py`	Adds loading of optional additional FITS extensions when reading products; adds `L2Mixin.get_additional_extensions()`.
`stixcore/products/level2/scienceL2.py`	New L2 spectrogram product scaffold with optional corrected data extension.
`stixcore/products/level2/quicklookL2.py`	Removes the L2 energy calibration product (moved to CAL).
`stixcore/products/level1/scienceL1.py`	Minor class metadata edit (processing version).
`stixcore/products/level0/scienceL0.py`	Copies optional additional-extension tables when splitting into file chunks.
`stixcore/products/__init__.py`	Exposes the new CAL energy calibration product and L2 science product(s).
`stixcore/products/CAL/energy.py`	New CAL energy calibration product that runs ECC and derives energy edges.
`stixcore/processing/publish.py`	Adds `CAL` to default published levels.
`stixcore/processing/pipeline_cron.py`	Re-enables L2 processing step (currently with a result-reset issue).
`stixcore/processing/FLtoL3.py`	New (currently broken) flare-list to L3 processing scaffold.
`stixcore/io/product_processors/fits/processors.py`	Adjusts filename generation for `cal` type and allows writing CAL via L2 processor.
`stixcore/io/ProcessingHistoryStorage.py`	Comments out a future DB migration section.
`stixcore/ecc/manager.py`	New ECC configuration manager with temporary context creation.
`stixcore/ecc/tests/test_ecc_manager.py`	Adds ECC manager tests.
`stixcore/calibration/elut_manager.py`	New ELUT manager wrapper around `stixpy` readers.
`stixcore/calibration/tests/test_elut_manager.py`	Adds ELUT manager tests (currently assumes missing repo data).
`stixcore/calibration/ecc_post_fit.py`	Adds ECC post-fit logic used by CAL energy calibration.
`stixcore/data/test.py`	Adds `test_data.ecc` path.
`stixcore/data/stixcore.ini`	Adds `[ECC] ecc_path` config key.
`stixcore/__init__.py`	Attempts to guard `__version_conf__` import (currently incorrect).
`pyproject.toml`	Adds `lmfit==1.3.4` dependency.
`docs/pipelineconfiguration.rst`	Removes pipeline monitor documentation section.
`.github/workflows/end2end_run.yml`	Whitespace cleanup.

Comments suppressed due to low confidence (1)

stixcore/products/level1/scienceL1.py:154

PRODUCT_PROCESSING_VERSION is now defined twice in this class. The duplicate assignment is redundant and makes it easy for the two values to diverge later; keep only one definition.

    PRODUCT_PROCESSING_VERSION = 4

    NAME = "xray-spec"

    PRODUCT_PROCESSING_VERSION = 4

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

stixcore/processing/FLtoL3.py

stixcore/products/product.py

stixcore/products/CAL/energy.py

stixcore/processing/pipeline_cron.py

stixcore/calibration/ecc_post_fit.py

Add support for trigger scaling in compression mode S=0,k=0,M=7 (i4Ds#377) * Unscale data when compressed with mode S=0,k=0,M=7 * QL - LC, BKG, SPECTRA * BSD - CPD, SCPD, VIS and SPEC bump all product versions to 2 (i4Ds#378) * bump all product versions to 2 * roleout IDB version 2.26.27 * fix test with new version number in mock object * fix test * fix for latest IDB as fallback Fix duplicated hk time bin bug Ddpdupdates2 (i4Ds#380) * bump all product versions to 2 * prep for L2 ddpd * integrate IDB release 2.26.38 * update for common CONF submodule * fir text for latest IDB to 2.26.38 Config fix (i4Ds#381) * bump all product versions to 2 * fix module path Fix bugs found when processing new scaled trigger data. (i4Ds#382) * Fix bug processing new scaled trigger data. * Allow for QL file to unscale by packet * Fix bug that create extra (replica) triggers in Spectrogram Publish_L2 / RID LUT Manager (i4Ds#385) * bump all product versions to 2 * RID LUT data manager update and use in processing for trigger descaling * Update stixcore/io/RidLutManager.py * fixed API mock via static files on pup99 --------- Co-authored-by: Shane Maloney <maloneys@tcd.ie> Add warning if trigger scaling changes in request (i4Ds#388) End2end rid lut manager (i4Ds#390) * bump all product versions to 2 * add copy and parent option * fix missing init of singleton bump all product versions to 2 (i4Ds#389) Bump conf version (i4Ds#391) * bump all product versions to 2 * bump common version Flexible way to fill DATAMAX/MIN BUNIT EXPOSURE in FITS header keywords (i4Ds#393) * bump all product versions to 2 * add properties for dedicated fits header keywords with class inhe. / override * add test and minor fixes after review * fix XPOSURE/XPOMAX header error * fix tests for dmax... * relax end2end fitsdiff * less tolerance in end2end test Solo anc stix ephemeris (i4Ds#399) * bump all product versions to 2 * add more FITS header keywords * fix for ANC also get fits updated headers while publishing Upgrade_watchdog (i4Ds#401) * bump all product versions to 2 * bump version * fix version label * pin common CONF version for end2end test * ignore VERS_CFG in end2end testing * add test for TM folder observing * bump watchdog==6.0.0 * integrate review fixes Update docks (i4Ds#409) * bump all product versions to 2 * update documentation * add doc for end to end tetsing * fix format * add doc for manual (re)processing of fits products E2e fix sym add (i4Ds#412) * bump all product versions to 2 * dep np.bool8 * disable test Get rid reason/scaling lookup error handling (i4Ds#411) * bump all product versions to 2 * add global error detection in pipeline logger (not just main process) * add exception handling for get_reason daily processing pipeline (i4Ds#405) * bump all product versions to 2 * initial * daily processing task * add singleStep * move ANC-aspect from pipeline to daily_pipeline * removed requirement for the moment * fix i4Ds#283 and option to process only latest fits file versions * update ddpd scripts * fix circular imports * test git dep * skio test on win * skip test on win/mac * Update stixcore/io/FlareListManager.py Co-authored-by: Shane Maloney <maloneys@tcd.ie> * fix format error --------- Co-authored-by: Shane Maloney <maloneys@tcd.ie> Add Zenodo file (i4Ds#413) * Add Zenodo author information Update high resolution transmission to match IDL defaults (i4Ds#414) Pin major and minor versions of some packages (i4Ds#415) V1.5.1_fixes (i4Ds#420) * bump all product versions to 2 * fix i4Ds#417 * fix i4Ds#418 * daily report log files could have different names if the scripts run over multiple days * fix i4Ds#419 * end2end test: better report if no corresponding file found * rewrite pipeline to cronjob based approach i4Ds#422 * cleanup before merge Fix bug in spectrogram time bin calculations (i4Ds#423) * Only use the closing time offset (NIX00269) for the last science substrure Update .zenodo.json

Copilot

Pull request overview

Copilot reviewed 25 out of 28 changed files in this pull request and generated 23 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-12T17:18:37Z

stixcore/calibration/ecc_post_fit.py

+    PIXELs = LARGEs + SMALLs
+
+    # Prep Pandas DataFrame
+    df = pd.DataFrame()  # dict will be define later during concactenation with appropriated keys


Typo in comment: "concactenation" should be "concatenation".

Suggested change

df = pd.DataFrame() # dict will be define later during concactenation with appropriated keys

df = pd.DataFrame() # dict will be define later during concatenation with appropriated keys

stixcore/calibration/ecc_post_fit.py

Copilot · 2026-02-12T17:18:38Z

stixcore/calibration/elut_manager.py

+
+# Create singleton instance
+if "pytest" in sys.modules:
+    ELUTManager.instance = ELUTManager(test_data.elut if hasattr(test_data, "elut") else None)


The singleton instance initialization checks for test_data.elut but this attribute doesn't exist in the TestData class (see stixcore/data/test.py). This will always evaluate to None in pytest runs, which may cause issues if tests expect ELUT data to be available. Consider adding an ELUTTestData class to test.py or removing the hasattr check.

Suggested change

ELUTManager.instance = ELUTManager(test_data.elut if hasattr(test_data, "elut") else None)

ELUTManager.instance = ELUTManager()

Copilot · 2026-02-12T17:18:38Z

stixcore/products/CAL/energy.py

+    def is_datasource_for(cls, *, service_type, service_subtype, ssid, **kwargs):
+        return (
+            kwargs["level"] in [EnergyCalibration.LEVEL, "L2"]
+            and service_type == 21
+            and service_subtype == 6
+            and ssid == 41
+        )


The is_datasource_for method accepts both "CAL" and "L2" levels. While this might be intentional for migration purposes (as mentioned in the PR description), it could lead to ambiguity. Consider whether this is truly necessary or if it should only accept "CAL" level. If both are required, document why in a comment.

Copilot · 2026-02-12T17:18:38Z

stixcore/products/CAL/energy.py

+                bash_script = f"""#!/bin/bash
+                              cd {ecc_run_context_path}
+
+                              {ecc_install_path}/Bkg
+                              {ecc_install_path}/ECC --f_obs "{spec_filename}"
+                              {ecc_install_path}/STX_Calib spec_all.fits ECC_para.fits[1]
+                              """


Potential command injection vulnerability: spec_filename is used in the bash script after simple string replacement but is still derived from FITS header data. Although there's a regex validation (line 170), the filename is interpolated into the bash script at line 195 inside double quotes, which could allow command injection if the filename contains special characters like backticks or dollar signs that bypass the regex. Consider using subprocess.run with a list of arguments instead of shell=True, or use shlex.quote() to properly escape the filename before interpolation.

Copilot · 2026-02-12T17:18:42Z

stixcore/calibration/ecc_post_fit.py

+
+
+def poly(x, degree, slope, intercept):
+    """a line"""


The docstring for the poly function says "a line" but it actually represents a quadratic polynomial (degree * x^2 + slope * x + intercept). Update the docstring to "a quadratic polynomial" for accuracy.

Suggested change

"""a line"""

"""a quadratic polynomial"""

Copilot · 2026-02-12T17:18:42Z

stixcore/calibration/ecc_post_fit.py

+def Read_fits_STIX_One_Pixel(data, PIX, DETECTOR_ID, Nbin=2024, NRebin=1, NSigma=1):
+    Pix = PIX
+    Pix = Pix + (DETECTOR_ID - 1) * 12
+    obs = [0] * Nbin


This assignment to 'obs' is unnecessary as it is redefined before this value is used.

Suggested change

obs = [0] * Nbin

Copilot · 2026-02-12T17:18:42Z

stixcore/calibration/ecc_post_fit.py

+    # Prep Pandas DataFrame
+    df = pd.DataFrame()  # dict will be define later during concactenation with appropriated keys


This assignment to 'df' is unnecessary as it is redefined before this value is used.

Suggested change

# Prep Pandas DataFrame

df = pd.DataFrame() # dict will be define later during concactenation with appropriated keys

# Prep Pandas DataFrame: dict will be defined later during concatenation with appropriate keys

Copilot · 2026-02-12T17:18:43Z

stixcore/calibration/ecc_post_fit.py

+    nbin = Nbin
+    Nbin = [Nbin / NRebin]


Variable Nbin is not used.

Suggested change

nbin = Nbin

Nbin = [Nbin / NRebin]

# determine number of bins to use, taking into account any requested rebinning

nbin = int(Nbin / NRebin)

Copilot · 2026-02-12T17:18:43Z

stixcore/calibration/ecc_post_fit.py

+    accumulator = []
+
+    # Proceed to fit individually each pixel spectrum in the list of files
+    for DET in DETs:


This for-loop may attempt to iterate over a non-iterable instance of class int.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

nicHoch commented Jul 23, 2025

View reviewed changes

stixcore/products/level2/scienceL2.py Show resolved Hide resolved

samaloney mentioned this pull request Sep 30, 2025

Daily Energy Calibration Data #395

Open

nicHoch force-pushed the L2_spec branch from cde5018 to c3b8921 Compare November 25, 2025 14:09

nicHoch changed the title ~~L2_spec~~ CAL spectra : daily energy calibration Feb 10, 2026

samaloney approved these changes Feb 12, 2026

View reviewed changes

stixcore/calibration/ecc_post_fit.py Show resolved Hide resolved

stixcore/products/level1/scienceL1.py Outdated Show resolved Hide resolved

stixcore/soop/manager.py Outdated Show resolved Hide resolved

samaloney requested a review from Copilot February 12, 2026 10:48

Copilot started reviewing on behalf of samaloney February 12, 2026 10:48 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

nicHoch added 16 commits February 12, 2026 13:09

add ecc context manager

3a06055

add elut manager

3848748

L2 calibration spectra and L2 sci-spectra

067651d

update ecc conf

d4b0ea5

prep fits file for post ecc fit

641354d

integration of ecc post fit step with own conf

9ec1392

alternate ebin edges for direct usage with data

8f876ad

add alternative e bin edges definition based on/not on applied ELUT

c45320f

fix for multiple spectra per day and more logging

941b581

fix typos

b1a7447

create a own CAL product "level"

a23eb33

fix soop startup and add check for elut same at the end

958b970

add CAL to publishing step

8f0b116

cleanup

4355806

add missing dependencies

4e7dd47

nicHoch force-pushed the L2_spec branch from 9a56b85 to 4e7dd47 Compare February 12, 2026 12:09

nicHoch added 4 commits February 12, 2026 13:17

undo SOOP API deactivation

1fb8bdf

fixes after review

f924fa8

remove obsolete file

7e966a5

fixes after review

7c59ac0

nicHoch self-assigned this Feb 12, 2026

nicHoch added this to the v1.7.0 milestone Feb 12, 2026

samaloney requested a review from Copilot February 12, 2026 17:10

Copilot started reviewing on behalf of samaloney February 12, 2026 17:11 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

Update stixcore/calibration/ecc_post_fit.py

2e3480b

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

	df = pd.DataFrame() # dict will be define later during concactenation with appropriated keys
	df = pd.DataFrame() # dict will be define later during concatenation with appropriated keys

	ELUTManager.instance = ELUTManager(test_data.elut if hasattr(test_data, "elut") else None)
	ELUTManager.instance = ELUTManager()

		# Prep Pandas DataFrame
		df = pd.DataFrame() # dict will be define later during concactenation with appropriated keys

	# Prep Pandas DataFrame
	df = pd.DataFrame() # dict will be define later during concactenation with appropriated keys
	# Prep Pandas DataFrame: dict will be defined later during concatenation with appropriate keys

CAL spectra : daily energy calibration #427

Are you sure you want to change the base?

CAL spectra : daily energy calibration #427

Uh oh!

Conversation

nicHoch commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

samaloney left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nicHoch commented Jul 22, 2025 •

edited

Loading

codecov-commenter commented Jul 22, 2025 •

edited

Loading