Eval bug: #17795 introduces subtle correctness errors

[IMbackK]

On HIP 2cd20b7 has introduced a subtle correctness problem. For some workloads model (any model, tested with gpt-oss, devstral-2 small/large, GLM-4.5-air and Qwen3-30B-A3B) quality collapses, the most reliable way of reproducing this i have found is mistral vibe, sample output:

Please write a python script that calculates the elevation angle of the sun based on a location and time given on the command line

Here's a Python script that calculates the elevation angle of the sun based on location and time:
solar_elevation.py
import
I need to create a more detailed plan. The
The The The The The The The The The The The The The The The The The The The The The The The The The The The The The The The The The The The The The The The The .

This problem dose not show in llamacpp's webui or cli for some reason, nor dose lama-perplexity show anything unusual. Other clients of the api are sometimes affected, but only mistral-vibe is consistently affected.

Operating systems

Linux

GGML backends

HIP

Hardware

CDNA

[IMbackK]

I dont immediately see whats wrong with this commit
@aendk @ggerganov @JohannesGaessler any ideas?

Perhapse some could try and repoduce on cuda, mistral-vibe 2.4 was used with this llama-server command for the bisect:

llama-server -m Devstral-Small-2-24B-Instruct-2512-Q8_0.gguf --cache-reuse 128 -cram 65536 --host 127.0.0.1 -c 128000 --temp 0.2 -ub 1024 --temp 0.2 --repeat-penalty 1.1

two gfx908 gpus where in play.

[am17an]

Does reverting this commit fix this issue? We can just disable this feature for HIP (I'm not able to reproduce this on CUDA)

[ggerganov]

We can just disable this feature for HIP

The scheduler logic must be compatible with all backends, so we should trace the cause of the issue.

@IMbackK Does it work if you remove --cache-reuse 128?

[JohannesGaessler]

I think the smoking gun would be if anyone could find a llama-perplexity setup where the results are not bit-for-bit identical since nothing should have changed.

[IMbackK]

@am17an yes
@JohannesGaessler i will try

[IMbackK]

since 57819b8 is functional revert of 2cd20b7 this "solves" the problem

[IMbackK]

via fuzzing (only took 3 itterations) I have found a llama-perplexity input that reproduces the problem see attached:

llama-perplexity -m Devstral-Small-2-24B-Instruct-2512-Q8_0.gguf -f data.txt
two mi100 gpus.

Master at b5e1212:

[1]271.7086,[2]240.7233,[3]242.0661,[4]233.7855,[5]234.4785,[6]236.5923,[7]239.9286,[8]239.2050,[9]238.7995,[10]237.2691,[11]239.5767,[12]239.7625,[13]237.2254,

Master at b5e1212 with 57819b8 reverted:

[1]271.4103,[2]240.7857,[3]242.0849,[4]233.7903,[5]234.4714,[6]236.5981,[7]239.9492,[8]239.2379,[9]238.8027,[10]237.3037,[11]239.5958,[12]239.7967,[13]237.2436,

data.txt