Versioning metadata and/or files?

[LukasKalbertodt]

Opencast currently has versioning for files, kind of. So the question came up whether we want it in the new data model or not. When we talked about this in the architecture meeting once, no one had amazing arguments/use cases for the feature, except one:

Data recovery after humans errors, bugs or hardware failure. Of course, everyone should have DB backups, but a full DB backup is large and thus they won't be created super regularly. And even if there is a suitable backup, it's usually infeasible to use it in a selective fashion, i.e. only restore one event. DB backups are rather for catastrophic failure situations where you want to roll back the whole DB. A versioning feature inside Opencast could potentially allow for fine-grained data recovery and a more complete "backup" (e.g. accidentally deleting a video that was uploaded the same day).

Of course, file and metadata are completely different beasts: keeping old file versions around seriously eats into your storage budget, while keeping metadata around eats very few server resources but is likely more complex to implement.

Files

Due to the large file size, admins usually don't want to keep them around for too long. So Opencast should support tools to automatically clean old versions based on rules (file age, file size, storage usage, ...). Depending on our file system discussion, we might just never delete files, but we have a garbage collector that prunes files that are not referenced by anything.

Metadata

(To be clear: in this discussion, metadata is all data about Opencast items that are not files, and thus includes ACLs and more.)

This is mostly a question about implementation and if it's worth the complexity. From a software architecture viewpoint, I think it's important that metadata versioning is implemented mostly as an independent layer on top of the core Opencast logic. I don't want this to seep through every piece of code dealing with metadata, but it should be ideally be handled at one central place. One idea would be to have DB triggers (in the DBMS itself or as an ORM layer in code) that, whenever some data is modified, writes the old data to a separate table. Or alternatively, write the change set to a separate table.

---

Having written this all down, it occured to me that this issue very likely does not need to be discussed as part of the core data model! We want this to be a thing on top anyway and we won't expose this versioning information in a new API anytime soon. It's an admin tool, at least for the foreseeable future. So maybe this is rather low priority.

Previous discussion:

• 2025-08-14

[mtneug]

I very much liked the idea of the hash-based file system. That could solve a lot of complexity, especially for S3 asset stores. Maybe to flesh that out a bit, here is my understanding:

"Data model instance" describes a single version of an event/mediapackage and is in principle equivalent to the current XML manifest. We could add versioning by having a history of "data model instances" stored in the DB. Each "data model instance" consists of elements that refer to files in an asset store (fs, S3, etc.). Crucially, versioning is not happening within the asset store, i.e. there are no archive/{org}/{id}/{version}/{element} paths anymore. Instead, the asset store is "flat", i.e. something like archive/{org}/{checksum}. To make this actually efficient on the file system, {checksum} can actually have multiple levels, e.g. given a file with checksum a34b085431775bd3c76db42d81840c0338fc5a07cc2931ecb3141e720d1f1913 map it to the path archive/{org}/a/34/b08/5431775bd3c76db42d81840c0338fc5a07cc2931ecb3141e720d1f1913.

What this will give us:

• No more hard links. No custom code for deduplication in S3.
• We could have configurable policies to keep the latest n versions of "data model instances". Independently, the asset store could remove files that are no longer referenced after x days.

What could be harder:

• If we want a unified store for archive and published files (now called distribution), this could require a "smart" web server that maps from published URL paths to paths in the hash-based fs. In some cases, we could also simply add HTTP redirects from Opencast to the web server with a resolved path.