Databricks Python Environment Setup

A robust, automated makefile-based tool for creating and managing local Python environments that match specific Databricks Serverless Environment Versions.

🚀 Quick Start

# Install prerequisites
# Install uv: https://github.com/astral-sh/uv

# Create environment for Databricks version 4
make env ENV_VER=4

# Activate the environment
source .venv-db4/bin/activate

That's it! You now have a local Python environment matching Databricks Environment Version 4.

✨ Features

Automatic Python Version Detection: Dynamically fetches the correct Python version from Databricks documentation
Smart Package Management:
- Removes Ubuntu-specific/system packages that won't work on macOS
- Handles binary-only packages gracefully (installs if available, skips if not)
- Cleans Databricks-specific version suffixes from packages
Version Validation: Only allows creation of environments for valid Databricks versions
Modular Pipeline: Separate targets for each step (requirements, Python install, venv setup, dependencies)
Lock File Generation: Creates requirements-env-X.lock for reproducible environments
Clean Management: Easy cleanup for specific versions or all environments
Comprehensive Testing: Built-in test suite to validate functionality

📋 Prerequisites

uv - Fast Python package installer and environment manager

# Install uv (macOS/Linux)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Add to PATH
export PATH="$HOME/.local/bin:$PATH"

make - Usually pre-installed on macOS/Linux
curl - For downloading requirements files
Internet connection - For fetching Databricks documentation and packages

📖 Usage

Main Commands

# Show all available commands
make help

# List available Databricks environment versions
make list-versions

# Create complete environment (default: version 4)
make env ENV_VER=4

# Clean up specific version
make clean ENV_VER=4

# Clean up all environments
make clean-all

Incremental Commands

For more control over the process:

# 1. Download and process requirements
make requirements ENV_VER=4

# 2. Detect and install Python version
make python-version ENV_VER=4
make install-python ENV_VER=4

# 3. Create virtual environment
make setup-venv ENV_VER=4

# 4. Install dependencies
make install-deps ENV_VER=4

# 5. Generate lock file
make create-lockfile ENV_VER=4

Using the Environment

Once created, activate and use your environment:

# Activate
source .venv-db4/bin/activate

# Verify Python version
python --version

# Test imports
python -c "import pandas, numpy, pyspark; print('All imports successful!')"

# Deactivate
deactivate

🗂️ Generated Files

When you run make env ENV_VER=4, the following files are created:

.venv-db4/                      # Virtual environment directory
requirements-env-4.txt          # Processed requirements file
requirements-env-4.txt.binary   # Binary-only packages (internal use)
requirements-env-4.lock         # Lock file with installed packages

🔧 Configuration

Default Environment Version

Change the default version by modifying ENV_VER in the makefile:

ENV_VER ?= 4  # Change to your preferred default

Excluded Packages

The makefile automatically excludes packages that won't work on macOS. To modify the list, edit:

EXCLUDED_PACKAGES = unattended-upgrades|ssh-import-id|...

Binary-Only Packages

Packages that require binary wheels (will be skipped if unavailable):

BINARY_ONLY_PACKAGES = pyodbc

🎯 Available Databricks Versions

Currently supported versions (automatically fetched from Microsoft documentation):

Version 1
Version 2
Version 3
Version 4

Run make list-versions to see the latest available versions.

📦 Package Handling

Automatically Excluded Packages

The following packages are removed because they're Ubuntu/system-specific or lack ARM64 macOS wheels:

unattended-upgrades - Ubuntu system package
ssh-import-id - Ubuntu utility
dbus-python - Requires D-Bus system library
psycopg2 - PostgreSQL library requiring system dependencies
psutil - No compatible wheels for some versions
PyGObject, pycairo - GTK bindings
wadllib, lazr.uri, lazr.restfulclient - Launchpad utilities
google-api-core - Compatibility issues

Binary-Only Packages

Packages attempted with --only-binary (skipped if no wheel available):

pyodbc - ODBC database connector

Version Cleaning

Databricks-specific version suffixes are automatically removed:

pyspark==4.0.0+databricks.connect.17.0.1  →  pyspark==4.0.0

🧪 Testing

Quick Start

Run the full test suite:

make test

Or run the test script directly:

./test_makefile.sh

For quick validation (30 seconds):

./quick_test.sh

What Gets Tested

1. Prerequisites Checks

✓ uv is installed and accessible
✓ make is installed
✓ makefile exists

2. Information Targets

✓ make help displays correctly
✓ make list-versions returns available versions

3. Validation

✓ make validate-version accepts valid versions (e.g., 4)
✓ make validate-version rejects invalid versions (e.g., 999)
✓ make check-uv verifies uv installation

4. Requirements Processing

✓ make requirements downloads requirements file
✓ Requirements file is created and not empty
✓ Excluded packages are removed (psycopg2, psutil, dbus-python, etc.)
✓ Databricks version suffixes are cleaned from pyspark
✓ Binary-only packages are separated

5. Python Version Detection

✓ make python-version extracts correct Python version from docs

6. Full Environment Creation

✓ make env creates complete environment
✓ Virtual environment directory is created
✓ Python executable exists and is functional
✓ Lock file is created and contains packages
✓ Key packages are installed (pandas, numpy, pyspark)

7. Cleanup

✓ make clean removes files for specific version
✓ make clean-all removes all generated files

Test Output

The test suite provides colored output:

🟡 YELLOW: Test being run
🟢 GREEN: Test passed
🔴 RED: Test failed

Example output:

TEST: Testing 'make requirements' target (ENV_VER=4)
✓ PASS: requirements target executes successfully
✓ PASS: requirements-env-4.txt file created
✓ PASS: requirements-env-4.txt is not empty

========================================================================
TEST RESULTS SUMMARY
========================================================================
Total tests run:    25
Tests passed:       25
Tests failed:       0
========================================================================
All tests passed! ✓

Test Options

Quick Tests (30 seconds) ⚡

./quick_test.sh

Validates basic functionality without creating full environment.

Full Test Suite (5-10 minutes) 🔍

make test

Comprehensive tests including full environment creation and validation.

Manual Testing

Test individual targets:

make help
make list-versions
make validate-version ENV_VER=4
make requirements ENV_VER=4

Running Individual Tests

You can modify test_makefile.sh to run specific tests by commenting out sections you don't want to run.

Continuous Integration

GitHub Actions Workflows

This repository includes two automated workflows:

1. Test Makefile (`.github/workflows/test.yml`)

Runs on:

Push to main branch → Quick tests only (~30 sec)
Pull requests to main → Full test suite (~5-10 min)

What it does:

✅ Installs uv
✅ Runs quick tests (always)
✅ Runs full test suite (PRs only)
✅ Uploads logs on failure
✅ Reports results in GitHub summary

2. Validate Pull Request (`.github/workflows/validate-pr.yml`)

Runs on:

Pull request opened/updated

What it does:

✅ Quick validation (syntax, version detection, requirements)
✅ Posts comment on PR with results
✅ Fast feedback (~1 minute)

Manual CI/CD Integration

For other CI/CD systems:

# Example GitHub Actions
- name: Test Makefile
  run: |
    curl -LsSf https://astral.sh/uv/install.sh | sh
    export PATH="$HOME/.local/bin:$PATH"
    make test

# Example GitLab CI
test:
  script:
    - curl -LsSf https://astral.sh/uv/install.sh | sh
    - export PATH="$HOME/.local/bin:$PATH"
    - make test

Test Troubleshooting

Test Hangs

If the full environment creation test hangs, it will timeout after 10 minutes. Check /tmp/make_env_output.log for details.

PATH Issues

If tests fail with "uv is not installed", ensure uv is in your PATH:

export PATH="$HOME/.local/bin:$PATH"
make test

Cleanup Between Tests

The test script automatically runs make clean-all between major tests to ensure a clean state.

Adding New Tests

To add new tests to test_makefile.sh:

Use print_test "Test description" to start a new test
Run your test command
Use pass "message" for successful assertions
Use fail "message" for failed assertions

Example:

print_test "Testing custom target"
if make custom-target >/dev/null 2>&1; then
    pass "custom-target executed successfully"
else
    fail "custom-target failed"
fi

Test Performance

Full test suite typically takes:

Fast tests (validation, help, etc.): ~10 seconds
Full environment creation: 3-5 minutes
Total runtime: ~5-10 minutes

To skip the slow full environment test, comment out that section in test_makefile.sh.

🐛 Troubleshooting

"uv is not installed"

Install uv and add it to your PATH:

curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"

"Could not extract Python version from documentation"

This usually means:

Network connectivity issues
Databricks documentation format changed
Invalid environment version number

Try running make list-versions to see available versions.

"No such file or directory: .venv-dbX/bin/uv"

This error shouldn't occur with the current version. If you see it, ensure you're using the latest makefile.

Package Installation Failures

If specific packages fail to install:

Check if it's a binary-only package that lacks ARM64 wheels
Add it to EXCLUDED_PACKAGES or BINARY_ONLY_PACKAGES in the makefile
The environment will still be created with other packages

Virtual Environment Activation Issues

Make sure you're using the correct command for your shell:

# bash/zsh
source .venv-db4/bin/activate

# fish
source .venv-db4/bin/activate.fish

# csh/tcsh
source .venv-db4/bin/activate.csh

🏗️ Project Structure

.
├── makefile                 # Main automation script
├── README.md               # This file
├── test_makefile.sh        # Full test suite
├── quick_test.sh           # Quick validation tests
├── .venv-db{X}/            # Virtual environments (generated)
├── requirements-env-{X}.txt        # Processed requirements (generated)
├── requirements-env-{X}.txt.binary # Binary-only packages (generated)
└── requirements-env-{X}.lock       # Lock files (generated)

🔄 Workflow

┌─────────────────────────────────────────────────────────┐
│                     make env ENV_VER=4                  │
└────────────────────────┬────────────────────────────────┘
                         │
                         ▼
            ┌────────────────────────┐
            │   Check Prerequisites  │
            │   - Validate uv        │
            │   - Validate version   │
            └───────────┬────────────┘
                        │
                        ▼
            ┌────────────────────────┐
            │  Download Requirements │
            │  - Fetch from MS docs  │
            │  - Process packages    │
            │  - Exclude system pkgs │
            └───────────┬────────────┘
                        │
                        ▼
            ┌────────────────────────┐
            │   Detect Python Ver    │
            │  - Parse from docs     │
            │  - Install with uv     │
            └───────────┬────────────┘
                        │
                        ▼
            ┌────────────────────────┐
            │   Create Virtual Env   │
            │  - uv venv with ver    │
            └───────────┬────────────┘
                        │
                        ▼
            ┌────────────────────────┐
            │  Install Dependencies  │
            │  - Main packages       │
            │  - Binary-only (skip)  │
            └───────────┬────────────┘
                        │
                        ▼
            ┌────────────────────────┐
            │   Generate Lock File   │
            │  - uv pip freeze       │
            └───────────┬────────────┘
                        │
                        ▼
                   ✅ Done!

💡 Tips & Best Practices

Always specify the version: make env ENV_VER=4 is clearer than relying on defaults
Check available versions first: Run make list-versions before creating an environment
Test incrementally: Use individual targets (make requirements, make setup-venv) when debugging
Keep environments separate: Use different ENV_VER values for different projects
Use lock files: Commit requirements-env-X.lock to ensure reproducible environments
Clean regularly: Run make clean-all to remove old environments and free up disk space

🤝 Contributing

To contribute improvements:

Fork the repository
Create a feature branch: git checkout -b feature/my-improvement
Make your changes to the makefile
Run the test suite: make test (locally)
Ensure all tests pass
Commit your changes: git commit -m "Add my improvement"
Push to your fork: git push origin feature/my-improvement
Create a Pull Request
- The Validate PR workflow will run automatically
- Review the validation results posted as a comment
Once approved and merged, the Test Makefile workflow runs on main

Setting Up Your Repository

After cloning/creating this repository:

Update badge URLs in README.md:

Replace YOUR_USERNAME/YOUR_REPO with your actual GitHub username and repository name

Enable GitHub Actions:
- Go to repository Settings → Actions → General
- Ensure "Allow all actions and reusable workflows" is selected

First Push:

git add .
git commit -m "Initial commit"
git push origin main

The workflows will run automatically!

📝 License

This tool is provided as-is for use with Databricks environments.

🔗 Related Links

❓ FAQ

Q: Why use this instead of pip/conda?
A: This tool automatically matches Databricks environments exactly, handling version-specific quirks and platform differences.

Q: Can I use this on Linux/Windows?
A: It's designed for macOS but should work on Linux. Windows support via WSL2 is untested.

Q: What if a package I need was excluded?
A: You can manually install it in the venv, or modify EXCLUDED_PACKAGES in the makefile if you know how to handle dependencies.

Q: How do I update an existing environment?
A: Run make clean ENV_VER=X followed by make env ENV_VER=X to rebuild from scratch.

Q: Can I use this for multiple Databricks workspace versions?
A: Yes! Create separate environments: make env ENV_VER=1, make env ENV_VER=4, etc.

Made with ❤️ for Databricks developers

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.github/workflows		.github/workflows
.gitignore		.gitignore
CI-CD-SETUP.md		CI-CD-SETUP.md
CI-CD.md		CI-CD.md
README.md		README.md
TESTING.md		TESTING.md
makefile		makefile
quick_test.sh		quick_test.sh
test_makefile.sh		test_makefile.sh

Folders and files

Latest commit

History

Repository files navigation

Databricks Python Environment Setup

🚀 Quick Start

✨ Features

📋 Prerequisites

📖 Usage

Main Commands

Incremental Commands

Using the Environment

🗂️ Generated Files

🔧 Configuration

Default Environment Version

Excluded Packages

Binary-Only Packages

🎯 Available Databricks Versions

📦 Package Handling

Automatically Excluded Packages

Binary-Only Packages

Version Cleaning

🧪 Testing

Quick Start

What Gets Tested

1. Prerequisites Checks

2. Information Targets

3. Validation

4. Requirements Processing

5. Python Version Detection

6. Full Environment Creation

7. Cleanup

Test Output

Test Options

Quick Tests (30 seconds) ⚡

Full Test Suite (5-10 minutes) 🔍

Manual Testing

Running Individual Tests

Continuous Integration

GitHub Actions Workflows

1. Test Makefile (.github/workflows/test.yml)

2. Validate Pull Request (.github/workflows/validate-pr.yml)

Manual CI/CD Integration

Test Troubleshooting

Test Hangs

PATH Issues

Cleanup Between Tests

Adding New Tests

Test Performance

🐛 Troubleshooting

"uv is not installed"

"Could not extract Python version from documentation"

"No such file or directory: .venv-dbX/bin/uv"

Package Installation Failures

Virtual Environment Activation Issues

🏗️ Project Structure

🔄 Workflow

💡 Tips & Best Practices

🤝 Contributing

Setting Up Your Repository

📝 License

🔗 Related Links

❓ FAQ

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

1. Test Makefile (`.github/workflows/test.yml`)

2. Validate Pull Request (`.github/workflows/validate-pr.yml`)

Packages