Function args sometimes not decref:ed after return

[mandolaerik]

Bug report

Bug description:

I have an application written in C and Python, which implements threading using pthreads for performance-critical C code, and occasionally runs python code from these threads, protected by the GIL. I have somewhat complex reference counting logic which I unit test using weakref.Ref. These tests work flawlessly in 3.13 and earlier, but started to break in 3.14 (specifically 3.14.2, compiled with free threading disabled, linux/SLES12). My test looks something like:

from weakref import Ref
import gc
# my_c_module is implemented using the Python C API
from my_c_module import call_in_thread, f

class Data: pass

data = Data()
ref = Ref(data)
def g():
    f(data)

call_in_thread(g)

assert ref() is not None
del data
gc.collect()
assert ref() is None

By inspecting reference counts it seems that data sometimes gets an extra reference while returning from f: Py_REFCNT always returns 4 in the end of f, but right after returning into g, sys.getrefcount returns either 3 or 4. With 3.14 the test fails maybe 75% of the time, and with 3.10..3.13 it has never failed over thousands of runs. Unfortunately, I have not yet been able to reproduce it in a toy environment.

Playing around a bit, it seems to only happen when the variable is referenced from different pthreads, and only when a value is passed as an argument to a function implemented in C.

I also dug a bit with GDB, seems like the difference between the failing and succeeding path is this code path in _Py_Dealloc, which seems to defer deallocation of the args tuple, thus inhibiting arg decref:

    if (margin < 2 && gc_flag) {
        _PyTrash_thread_deposit_object(tstate, (PyObject *)op);
        return;
    }

It seems that these objects are leaked forever; that tuple object seems to remain even after I do things like call_in_thread(lambda: gc.collect()).

Any ideas? In particular, do you have any ideas on how to create a reproducer I can share? (the full application is in part proprietary, so cannot share that)

CPython versions tested on:

3.14

Operating systems tested on:

Linux

[sergey-miryanov]

Few comments:

1. Is call_in_thread blocking, or not? Can we be sure that after this call no other thread will use the data?
2. In the normal case you don't need to call gc.collect. del data is enough to "deallocate" the data
3. _PyTrash_thread_deposit_object means that you have "too deep" callstack. This means that the VM doesn't have enough space to free up the object, so it needs to postpone it. Is this your case?

[mandolaerik]

1. call_in_thread is blocking. In the real application it's somewhat more complex, something like:

initialize()
run_after_steps(1, g)
run_steps(2)

So the thread will be done with the data, and apart from that only the initial Python thread will use the data.

2. Understood. My original test did not collect(); I added the call now just in case.
3. The call stack is very shallow, by all metrics I can find. While playing around, traceback.print_stack never printed more than two frames, and bt in gdb never printed more than 20 frames.

[sergey-miryanov]

Thanks!
Could you please test with 3.14.0 and 3.14.1 versions?