Video language (metadata, tracks, ...)

[LukasKalbertodt]

• Per track language: each subtitle, audio or video track has a language: LangCode? field
  • Or LangCode[], i.e. potentially multiple languages if multiple languages are spoken?
• Top level language field?
  • What does that mean?
    • "Main spoken language" of the video? Probably pretty clear most of the time, but there can be exceptions (multiple audio tracks)
    • Language of metadata?
• Support user-facing metadata in multiple languages?
  • Was never fully supported in OC, complicates everything quite a bit, requires frontend support, useless if video is still in one fixed language.
  • We have multi-language universities (e.g. Swiss ones, tho Bern & ETH don't seem to be that interested in this feature), also potentially useful for PR (public relation) kind of videos

Re "per track language" and/or "top level field", we probably have to decide what special case we simply don't properly support. Most videos that most people upload will have one audio track and no subtitle tracks. When uploading, a user might thus expect to have one single language selector. If there are additional audio or subtitle tracks, it would be nice to: load language info from the container, but also let the user set the per-track language if they want. Problem is that a frontend application cannot inspect tracks during upload, this is just something that Opencast does later. But it can be very advantageous to already have language information on upload, e.g. to give an automatic subtitle engine that information.

Previous discussions:

• 2025-04-10
• 2025-04-24
• Discuss: Event Metadata #1 (comment)

[mtneug]

As with other issues, I'm always an opponent of restricting the data model, i.e. I'm for having LangCode for each stream in a track at the data layer. Simplifications should be made at the frontend level for the most common use cases. I don't want to have another data model discussion in 5 years, when the next feature stumbles upon limitations. Instead, the frontend can be changed more easily. So LangCode[] at the data layer and one selection in the frontend seems fine to me for how Opencast is currently used.

If there are additional audio or subtitle tracks, it would be nice to: load language info from the container, but also let the user set the per-track language if they want. Problem is that a frontend application cannot inspect tracks during upload, this is just something that Opencast does later.

You can very well decode the container in the frontend, e.g. using Mediabunny. However, in my experience the streams in a container mostly have an undefined language. This might be different for users that actually mux multiple audio tracks in one container. But I'm not sure if that's actually the case and worth the effort to implement.

[LukasKalbertodt]

How would you suggest an uploader would work?

• Not give the user the option to select a language? Then things like subtitle generation might not work or just suboptimally.
• Give the user a single language select and then put that value... where?
• Let user select language for each track/stream... and use media bunny to figure out the number and kinds of streams in the uploaded file?

[mtneug]

How would you suggest an uploader would work?

Make the excuse that this issue is only about the data model 🙈

But more seriously, I guess the most common case is a single language, so the UIs would only display a single option. Maybe with an exception for caption uploads that could already have a per-track selection for the language. I'm not so sure where a single language selection should be stored. I guess it can still make sense to store it at the event level and then leave the track languages undefined.

BTW selecting no language still works surprisingly well with the Whisper auto detection, at least for German/English.