Fix delete snapshot policy expunged volume

[Damans227]

Description

Fixes: #12441

deleteSnapshotPolicies() uses findById() which fails for expunged volumes. Changed to findByIdIncludingRemoved() to allow deleting orphaned policies.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

• Manually verified fix on test environment
• Added unit tests for deleteSnapshotPolicies()

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 17.60%. Comparing base (2941b51) to head (a32e7bc).
⚠️ Report is 67 commits behind head on 4.22.

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #12474      +/-   ##
============================================
+ Coverage     17.56%   17.60%   +0.03%     
- Complexity    15544    15608      +64     
============================================
  Files          5910     5910              
  Lines        529128   529819     +691     
  Branches      64636    64730      +94     
============================================
+ Hits          92931    93252     +321     
- Misses       425740   426070     +330     
- Partials      10457    10497      +40     

Flag	Coverage Δ
uitests	3.60% <ø> (+0.01%)	⬆️
unittests	18.67% <100.00%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sureshanaparti]

@Damans227 Check if it's possible to remove all snapshot policies of expunged / non-existing volumes during the upgrade (as one-time task as these are not applicable any more, and not relevant to show them in the UI). I think, it's better to remove all the snapshot policies associated with the volume during expunge volume itself, and refer them with findByIdIncluding Removed() wherever the snapshot policy is referred for the existing snapshots for the expunged volumes.

[Damans227]

@Damans227 Check if it's possible to remove all snapshot policies of expunged / non-existing volumes during the upgrade (as one-time task as these are not applicable any more, and not relevant to show them in the UI). I think, it's better to remove all the snapshot policies associated with the volume during expunge volume itself, and refer them with findByIdIncluding Removed() wherever the snapshot policy is referred for the existing snapshots for the expunged volumes.

@sureshanaparti So, the policies are already deleted during volume destroy - VolumeServiceImpl.destroyVolume() calls snapshotMgr.deletePoliciesForVolume().

The issue seems to me is when an orphaned policy exists due to an edge case (eg: failures, etc.). To reproduce the issue, I had to manually insert a record in DB to simulate an orphan policy. This fix allows handling those orphans created via UI/ API. Example:

An upgrade script to clean existing orphans is a good idea. Lemme know if I shall further investigate that?

[sureshanaparti]

@Damans227 destroy volume can be recovered, so better delete these policies while expunging the volume. also, check if you can keep the delete policies and remove volume in a transaction on destroy / expunge volume.

check if such orphans can be cleaned during ms start (maybe here -

cloudstack/framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java

Line 1210 in 036489b

private boolean cleanupVolume(final long volumeId) {

or other volume/snapshot service classes)

[Damans227]

@Damans227 destroy volume can be recovered, so better delete these policies while expunging the volume. also, check if you can keep the delete policies and remove volume in a transaction on destroy / expunge volume.

check if such orphans can be cleaned during ms start (maybe here -

cloudstack/framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java

Line 1210 in 036489b

private boolean cleanupVolume(final long volumeId) {

or other volume/snapshot service classes)

@sureshanaparti I see what you mean. I'll move policy deletion from destroy to expunge and add orphan cleanup on MS startup.

[Damans227]

@blueorangutan package

[blueorangutan]

@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16472