[BUG] UDFs with external dependencies aren’t supported consistently across execution modes

[oresttkachukd]

Describe the bug
Databricks docs provide an example of UDFs with external dependencies here:
https://docs.databricks.com/aws/en/dev-tools/databricks-connect/python/udf#udfs-with-dependencies
In my testing, it works when running as “Run Python File”, but it doesn’t work when running via “Debug Current File with Databricks Connect”, “Run Current File with Databricks Connect”, or “Run Current File in Interactive Window.”

To Reproduce
Steps to reproduce the behavior:

1. Create a .py file with simplified version of the UDFs demo:

# Databricks notebook source
# COMMAND ----------
from databricks.connect import DatabricksSession, DatabricksEnv
from pyspark.sql.functions import udf, col, pandas_udf
from pyspark.sql.types import IntegerType, LongType, StringType

# COMMAND ----------
pypi_deps = ["pyjokes>=0.8,<1"]

env = DatabricksEnv().withDependencies(pypi_deps)
spark = DatabricksSession.builder.withEnvironment(env).getOrCreate()

# COMMAND ----------
# UDFs
@udf(returnType=StringType())
def get_joke():
    from pyjokes import get_joke

    return get_joke()


# COMMAND ----------
df = spark.range(1, 10)
df = df.withColumns(
    {
        "joke": get_joke(),
        # "doubled": double_and_json_parse(col("id")),
        # "mutliplied_with_roll": multiply_and_add_roll(col("id"), col("doubled")),
    }
)
df.show()

# COMMAND ----------

2. Click UI Run Cell button within the file or Debug Current File with Databricks Connect/Run Current File with Databricks Connect
3. See error: "No module named 'pyjokes'"

System information:

1. Paste the output ot the Help: About command (CMD-Shift-P).
   Version: 1.109.5 (user setup)
   Commit: 072586267e68ece9a47aa43f8c108e0dcbf44622
   Date: 2026-02-19T19:43:32.382Z
   Electron: 39.3.0
   ElectronBuildId: 13313481
   Chromium: 142.0.7444.265
   Node.js: 22.21.1
   V8: 14.2.231.22-electron.0
   OS: Windows_NT x64 10.0.26200
2. Databricks Extension Version
   package: databricks-connect~=16.4.13
   extension: 2.10.4