USE 373 - Update to 2026-03 python project conventions

[ghukill]

Purpose and background context

Why these changes are being introduced

This project was due for a few key updates: pipenv to uv, python version, etc. These updates fall under a 2026-03 snapshot of targets and conventions for python projects.

How this addresses that need:

• Migrates from pipenv to uv
• Updates to python 3.13
• Updates to linting, pre-commit, and Makefile conventions
• Default to arm64 architecture and Makefile commands build to that
• Minor linting and formatting updates going from black to ruff format

How this updated was implemented

This update was performed via an experimental process of using an AI agent to follow a specification. The specification can be found here: 2026-03-05 - Python Projects DevOps Updates.

Additionally, the first pass of work by the agent can be seen in this PDF file.

A secondary pass was taken when realized that Makefile would need updates for dist-dev, publish-dev, etc. The confluence specification document was updated, and a secondary round of updates performed. Here is additional output from that:

 Good. The Confluence checklist now includes the "Container Build & Deploy (Terraform-Generated)" items. Here's the audit of the Terraform-generated targets:                                                                                                                                                               
                                                                                                                                                                                                                                                                                                                            
 Audit — Container Build & Deploy                                                                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                                                            
 ┌───┬───────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────┐                                                                                                                                                                                
 │ # │ Item                                                                                                  │ Status                      │                                                                                                                                                                                
 ├───┼───────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────┤                                                                                                                                                                                
 │ 1 │ Makefile header includes CPU_ARCH reading from .aws-architecture                                      │ ❌ Missing                  │                                                                                                                                                                                
 ├───┼───────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────┤                                                                                                                                                                                
 │ 2 │ Replace docker build with docker buildx pattern (check-arch, dist-dev, publish-dev, docker-clean)     │ ❌ Old docker build pattern │                                                                                                                                                                                
 ├───┼───────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────┤                                                                                                                                                                                
 │ 3 │ Replace docker login -u AWS -p $(...) with aws ecr get-login-password | docker login --password-stdin │ ❌ Old insecure pattern     │                                                                                                                                                                                
 ├───┼───────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────┤                                                                                                                                                                                
 │ 4 │ Add .arch_tag to .gitignore                                                                           │ ❌ Missing                  │                                                                                                                                                                                
 └───┴───────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────┘                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                                            
 Items to preserve (unique to this repo):                                                                                                                                                                                                                                                                                   
 - dist-stage / publish-stage targets (with stage deploy instructions comment)                                                                                                                                                                                                                                              
 - Local OpenSearch commands (local-opensearch-start, local-opensearch-stop, local-opensearch-teardown)                                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                                                                            
 I'll update the Terraform header, replace all docker build/publish targets with the new buildx pattern, and add .arch_tag to .gitignore. Proceeding now.

...
...

Here's what changed in this pass:                                                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                                                            
 Makefile — Terraform-generated header & build targets:                                                                                                                                                                                                                                                                     
 - Added CPU_ARCH ?= $(shell cat .aws-architecture 2>/dev/null || echo "linux/amd64") to header                                                                                                                                                                                                                             
 - Added check-arch target that validates architecture and writes .arch_tag                                                                                                                                                                                                                                                 
 - Replaced docker build with docker buildx build in dist-dev (architecture-aware, named builder, --load flag, richer tagging: latest-arm64, make-latest-arm64, make-{git-sha})                                                                                                                                             
 - Replaced insecure docker login -u AWS -p $(...) with aws ecr get-login-password | docker login --password-stdin in publish-dev                                                                                                                                                                                           
 - Applied the same buildx + check-arch + secure login pattern to dist-stage / publish-stage (preserved the stage deploy instructions comment)                                                                                                                                                                              
 - Added docker-clean target for cleanup of images, builders, and .arch_tag                                                                                                                                                                                                                                                 
 - Preserved all local OpenSearch targets untouched                                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                                            
 .gitignore — Added .arch_tag

How can a reviewer manually see the effects of these changes?

1- Remove pipenv environment and replace with uv environment

# maybe some slight variations depending on terminal environmenet + IDE, but roughly
# remove pipenv venv
pipenv --rm 

# recreate venv
make install

2- Set env vars:

WORKSPACE=dev
TIMDEX_OPENSEARCH_ENDPOINT=localhost
OPENSEARCH_INITIAL_ADMIN_PASSWORD=SuperSecret42!

3- Run Opensearch locally

docker compose up

4- Run TIM via uv:

uv run tim ping

5- Build docker container and ping local opensearch:

# build image
make dist-dev

# docker container ping
# NOTE: this work because --network makes "localhost" the same host as the
#   "opensearch" service from the docker compose project.
docker run \
--network container:timdex-index-manager-opensearch-1 \
-e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=SuperSecret42\!" \
timdex-index-manager-dev:latest-arm64 \
ping

Lastly, see this run in Dev1 for confirmation that TIM is working as before updates: https://us-east-1.console.aws.amazon.com/states/home?region=us-east-1#/v2/executions/details/arn:aws:states:us-east-1:222053980223:execution:timdex-ingest-dev:04c0956a-908b-4a35-a945-c757ffa9c717.

Includes new or updated dependencies?

YES

Changes expectations for external applications?

NO

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/USE-373

Code review

• Code review best practices are documented here and you are encouraged to have a constructive dialogue with your reviewers about their preferences and expectations.

[ehanson8]

Works as expected!