ci : discuss optimization strategies

[ggerganov]

Overview

With the current trend of the Github Actions runners becoming increasing slower and slower, we are close to not having a useable CI pipeline. Opening this discussion to discuss what we can do about this.

Queue time for last 6 months

TODOs

• Optimize ubuntu-24-cmake-vulkan - currently very slow
• Run all riscv workflows only on the master branch and not in PRs
• Separate GH-hosted and self-hosted workflows in separate .yml files
• Update ccache-action: ci : discuss optimization strategies #20446 (reply in thread)

Upcoming self-hosted runners

One of the best way to minimize the queue times is to move the workflows to self-hosted runners that are dedicated to running ggml workflows. This requires provisioning and hosting hardware.

• 1 runner hosted by Intel (ci: Setup self-hosted CI for Intel Linux Vulkan backend #20154)
• 1 upcoming runner hosted by Intel for OpenVINO workflows (Add OpenVINO backend #15307)
• DGX Spark donated by NVIDIA (should be up in a few days)

[CISC]

I think we also need to look into minimizing (however possible without compromising test coverage) the amount of builds running.

[ggerganov]

Yes, the total runtime also keeps increasing. Though I am less concern about it. The queue time is what we have much less control of.

[ggerganov]

How do we feel about disabling all automatic Github-hosted workflows for Pull Requests and delegate it to maintainers to manually decide which workflows to run and when for a given PR? The master branch continues to run all workflows by default.

[CISC]

How do we feel about disabling all automatic Github-hosted workflows for Pull Requests and delegate it to maintainers to manually decide which workflows to run and when for a given PR? The master branch continues to run all workflows by default.

I've also toyed with this idea, this adds some overhead for us, but I think in general it would be beneficial.

[danbev]

I think this sounds reasonable and worth doing 👍

[ggerganov]

We'll need to separate the workflow definitions so that self-hosted workflows are in separate .yml files from the non-self-hosted.

[aldehir]

Maybe a minimal workflow for Ubuntu/macOS/Windows cpu-only builds to run tests? Then when the PR is refined enough, manually trigger the entire suite prior to merging.

[CISC]

I wonder if our main issue is that we have many long running ubuntu runnners (ubuntu-24-cmake-vulkan in particular) saturating availability of it and causing a long backlog...

[ggerganov]

Yes, you might be right. This workflow seems to take too long:

https://github.com/ggml-org/llama.cpp/actions/runs/22963567444/job/66660728706

[CISC]

Oh, BTW, we need to update our ccache-action:
https://github.com/ggml-org/llama.cpp/actions/runs/22963567444/job/66660728706#annotation:19:2

[netrunnereve]

Well I was the one who set up the ubuntu-24-cmake-vulkan runner, it's super slow since it basically emulates a GPU on CPU to run the Vulkan tests. I've optimized it already but ultimately it's not going to go any faster because of the slow machines and the growing number of tests. Maybe it's possible to get it running on ARM but I've never gotten that to work. While it has its uses (mainly in finding interesting bugs that don't pop up in a real GPU, and also so that it can be run in forks) considering we have the CI machines with real GPUs now we can get rid of it if you feel it's too slow.

We also have way too many jobs in general. Aside from removing jobs you can also try spreading out the load between ARM and x86 machines if possible, some stuff like those cross compiles or webgpu runs can probably be done on ARM. There's also the new ubuntu-slim machine which they hopefully have more of and which we can use for simple jobs that only need 1 core and 5 gigs of memory.

[CISC]

Well I was the one who set up the ubuntu-24-cmake-vulkan runner, it's super slow since it basically emulates a GPU on CPU to run the Vulkan tests. I've optimized it already but ultimately it's not going to go any faster because of the slow machines and the growing number of tests. Maybe it's possible to get it running on ARM but I've never gotten that to work. While it has its uses (mainly in finding interesting bugs that don't pop up in a real GPU, and also so that it can be run in forks) considering we have the CI machines with real GPUs now we can get rid of it if you feel it's too slow.

We can keep it, but I think this is one we will have to either only run on Vulkan changes and/or manually.

We also have way too many jobs in general. Aside from removing jobs you can also try spreading out the load between ARM and x86 machines if possible, some stuff like those cross compiles or webgpu runs can probably be done on ARM.

Indeed, spreading them across different arches may help.

There's also the new ubuntu-slim machine which they hopefully have more of and which we can use for simple jobs that only need 1 core and 5 gigs of memory.

Yep, already transitioned a few jobs to that. :)