Skip to content

Improve running end-to-end tests locally #972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
NBardelot wants to merge 7 commits into
base: master
Choose a base branch
from
+105 −10
Open

Improve running end-to-end tests locally #972

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
4c28678
Improve running end-to-end locally
NBardelot Jan 26, 2026
f81403a
Overriding the git-sync repo URL in e2e tests
NBardelot Jan 27, 2026
d9181a7
End-to-end tests selection in the Makefile
NBardelot Jan 27, 2026
3f7b800
Revert "Clarify end-to-end tests script"
NBardelot Jan 27, 2026
8354e91
Clarify end-to-end script usage
NBardelot Jan 27, 2026
5b9e7f5
End-to-end tests selection in the Makefile
NBardelot Jan 27, 2026
69cbf3a
PR #972 review improvements
NBardelot Jan 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
/bin
/.go
/.push-*
/.buildx-initialized
/.container-*
/.dockerfile-*
/.buildx-initialized
/.go
/.idea
/.licenses
/.push-*
/bin
56 changes: 56 additions & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,59 @@ make all-container \
BUILD_IMAGE=$BUILD_IMAGE_IN_PRIVATE_REGISTRY \
BASEIMAGE=$BASEIMAGE_IN_PRIVATE_REGISTRY
```

## Running end-to-end tests using fully-qualified base images

By default the `_test_tools/*/Dockerfile` images used by the end-to-end tests are built in the `Makefile`'s `.container-test_tool.%` goals
using an unqualified `alpine` image.

In order to pull the `alpine` image from a private registry and/or with a fully-qualified name, and run the tests, you can
use for example:

```sh
docker login $YOUR_PRIVATE_REGISTRY
ALPINE_REGISTRY_PREFIX=$YOUR_PRIVATE_REGISTRY/$YOUR_ALPINE_NAMESPACE_PREFIX/ # Please note the final '/'
docker pull ${ALPINE_REGISTRY_PREFIX}alpine
make test ALPINE_REGISTRY_PREFIX=$ALPINE_REGISTRY_PREFIX
```

## Running end-to-end tests locally with docker configured behind a proxy

If you are using proxy configurations in your `~/.docker/config` file, you must add the `docker0` subnet (created by the
*bridge* network) as an exception in order for the containers executed by the tests to be able to call each other.

For example to get the subnets associated with the *bridge* network in a default Docker configuration you can run:

```bash
docker network ls # you can verify that a network called bridge is present
docker network inspect bridge --format '{{ range .IPAM.Config }}{{ .Subnet }}{{ end }}'
```

If for example your *bridge* subnet is `172.16.0.0/12`, then you'd want your `~/.docker/config.json` to look like this:

```json
{
"proxies": {
"default": {
"httpProxy": "...",
"httpsProxy": "...",
"noProxy": "...,172.16.0.0/12"
}
}
}
```

And you'd want to run the tests like this:

```bash
make test HTTP_PROXY="..." HTTPS_PROXY="..." NO_PROXY"...,172.16.0.0/12"
```

Or manually:

```bash
export HTTP_PROXY="..."
export HTTPS_PROXY="..."
export NO_PROXY"...,172.16.0.0/12"
VERBOSE=1 ./test_e2e.sh
```
28 changes: 26 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,21 @@ DBG ?=
# These are passed to docker when building and testing.
HTTP_PROXY ?=
HTTPS_PROXY ?=
NO_PROXY ?=

# Allow some buildx adaptation for local builds
BUILDX_BUILDER_NAME := git-sync
BUILDX_BUILDER_SKIP_CREATION ?=

# Allow alpine to be pulled from a private registry when building the end-to-end tests images
ALPINE_REGISTRY_PREFIX ?=

# By default all end-to-end tests are executed, but this allows for a manual selection.
# NOTE: Each item in this list will be used as a regex to run functions of the form 'e2e::${MATCHING_TEST_NAME}'
# in the test_e2e.sh script. For example using 'auth' will include all authentication end-to-end tests.
# If multiple items match a same test, it will be executed only once.
LIST_OF_E2E_TESTS ?=

###
### These variables should not need tweaking.
###
Expand Down Expand Up @@ -134,6 +144,7 @@ $(OUTBIN): .go/$(OUTBIN).stamp
-v $$(pwd)/.go/cache:/.cache \
--env HTTP_PROXY=$(HTTP_PROXY) \
--env HTTPS_PROXY=$(HTTPS_PROXY) \
--env NO_PROXY=$(NO_PROXY) \
$(BUILD_IMAGE) \
/bin/sh -c " \
ARCH=$(ARCH) \
Expand Down Expand Up @@ -190,6 +201,7 @@ container: .container-$(DOTFILE_IMAGE) container-name
--platform "$(OS)/$(ARCH)" \
--build-arg HTTP_PROXY=$(HTTP_PROXY) \
--build-arg HTTPS_PROXY=$(HTTPS_PROXY) \
--build-arg NO_PROXY=$(NO_PROXY) \
-t $(IMAGE):$(OS_ARCH_TAG) \
-f .dockerfile-$(OS)_$(ARCH) \
.
Expand Down Expand Up @@ -237,6 +249,8 @@ release:
version:
echo $(VERSION)

# Run the 'test' goal in a single shell
test: .SHELLFLAGS = -c
test: $(BUILD_DIRS)
docker run \
-i \
Expand All @@ -248,17 +262,27 @@ test: $(BUILD_DIRS)
-v $$(pwd)/.go/cache:/.cache \
--env HTTP_PROXY=$(HTTP_PROXY) \
--env HTTPS_PROXY=$(HTTPS_PROXY) \
--env NO_PROXY=$(NO_PROXY) \
$(BUILD_IMAGE) \
/bin/sh -c " \
./build/test.sh ./... \
"
VERBOSE=1 ./test_e2e.sh
@if [ -n "$(HTTP_PROXY)" ]; then \
Copy link
Member

@thockin thockin Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this do anything? Each of these blocks runs in its own shell, so export has no meaning:

thockin@thockin-glaptop4:/tmp/m$ cat Makefile 
foo:
	@export FOO=bar
	@echo $$FOO

bar:
	@export FOO=bar; \
	echo $$FOO
thockin@thockin-glaptop4:/tmp/m$ make foo

thockin@thockin-glaptop4:/tmp/m$ make bar
bar

Copy link
Contributor Author

@NBardelot NBardelot Jan 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad. Let me add a commit to fix this.

I think someone who uses a proxy will have all three variables already configured in a dotenv file or profile, expecting the NO_PROXY standard variable to be passed alongside the two others. It looks strange when this one is not. So the goal here was to be consistent (and not having to debug why something will be routed through the proxy when the user's config contains an exclusion, for example: the private/local images registries, or the github project's local mirror).

export HTTP_PROXY="$(HTTP_PROXY)"; \
fi
@if [ -n "$(HTTPS_PROXY)" ]; then \
export HTTPS_PROXY="$(HTTPS_PROXY)"; \
fi
@if [ -n "$(NO_PROXY)" ]; then \
export NO_PROXY="$(NO_PROXY)"; \
fi
VERBOSE=1 ./test_e2e.sh $(LIST_OF_E2E_TESTS)
Copy link
Member

@thockin thockin Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does make pass "" here if the list is empty? I think not, but I am not sure

Copy link
Contributor Author

@NBardelot NBardelot Jan 30, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without quotes no argument is passed.

Example:

printargs.sh

#!/bin/bash
for a in "$@"; do
  echo "Arg: >$a<"
done
> ./printargs.sh foo $not_declared bar $not_declared
Arg: >foo<
Arg: >bar<

vs.

> ./printargs.sh foo "$not_declared" bar "$not_declared"
Arg: >foo<
Arg: ><
Arg: >bar<
Arg: ><


TEST_TOOLS := $(shell find _test_tools/* -type d -printf "%f ")
test-tools: $(foreach tool, $(TEST_TOOLS), .container-test_tool.$(tool))

.container-test_tool.%: _test_tools/% _test_tools/%/*
docker build -t $(REGISTRY)/test/$$(basename $<) $<
docker build --build-arg ALPINE_REGISTRY_PREFIX="$(ALPINE_REGISTRY_PREFIX)" -t $(REGISTRY)/test/$$(basename $<) $<
docker images -q $(REGISTRY)/test/$$(basename $<) > $@

# Help set up multi-arch build tools. This assumes you have the tools
Expand Down
3 changes: 2 additions & 1 deletion _test_tools/httpd/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@

# Stolen from https://github.com/linuxkit/linuxkit/tree/master/pkg/sshd/

FROM alpine AS base
ARG ALPINE_REGISTRY_PREFIX=""
FROM ${ALPINE_REGISTRY_PREFIX}alpine AS base

RUN mkdir -p /out/etc/apk && cp -r /etc/apk/* /out/etc/apk/
RUN apk add --no-cache --initdb -p /out \
Expand Down
3 changes: 2 additions & 1 deletion _test_tools/ncsvr/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@

# Stolen from https://github.com/linuxkit/linuxkit/tree/master/pkg/sshd/

FROM alpine AS base
ARG ALPINE_REGISTRY_PREFIX=""
FROM ${ALPINE_REGISTRY_PREFIX}alpine AS base

RUN mkdir -p /out/etc/apk && cp -r /etc/apk/* /out/etc/apk/
RUN apk add --no-cache --initdb -p /out \
Expand Down
3 changes: 2 additions & 1 deletion _test_tools/sshd/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@

# Stolen from https://github.com/linuxkit/linuxkit/tree/master/pkg/sshd/

FROM alpine AS base
ARG ALPINE_REGISTRY_PREFIX=""
FROM ${ALPINE_REGISTRY_PREFIX}alpine AS base

RUN mkdir -p /out/etc/apk && cp -r /etc/apk/* /out/etc/apk/
RUN apk add --no-cache --initdb -p /out \
Expand Down
13 changes: 12 additions & 1 deletion test_e2e.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@ set -o errexit
set -o nounset
set -o pipefail

# If the Makefile and/or the user do set up the upper-case flavor of a proxy variable,
# then let us also set up the lower-case flavor that is used by tools like the git CLI.
[ -n "${HTTP_PROXY:-}" ] && export http_proxy="$HTTP_PROXY"
[ -n "${HTTPS_PROXY:-}" ] && export https_proxy="$HTTPS_PROXY"
[ -n "${NO_PROXY:-}" ] && export no_proxy="$NO_PROXY"

# shellcheck disable=SC2120
function caller() {
local stack_skip=${1:-0}
Expand Down Expand Up @@ -3443,9 +3449,11 @@ function e2e::touch_file_abs_path() {
# Test github HTTPS
##############################################
function e2e::github_https() {
local default_repo="https://github.com/kubernetes/git-sync"
local repo="${REMOTE_TEST_REPO_URL:-$default_repo}"
GIT_SYNC \
--one-time \
--repo="https://github.com/kubernetes/git-sync" \
--repo="$repo" \
--root="$ROOT" \
--link="link"
assert_file_exists "$ROOT/link/LICENSE"
Expand Down Expand Up @@ -3670,6 +3678,9 @@ if [[ "$#" == 0 ]]; then
fi

# Build it
# NOTE: If you want to run the end-to-end tests locally and you need specific Makefile arguments
Copy link
Member

@thockin thockin Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not pass the important args?

Copy link
Contributor Author

@NBardelot NBardelot Jan 30, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At first glance I was surpised that the script was running its prerequisites by calling the Makefile. And I ended up calling the given goals ahead of time myself, and finding it simpler. The alternative would be to map 1..1 every Makefile arg as an env variable in the script, which is possible but not very practical.

# you might want to run `make test` once first with those arguments, in order to pre-build this image.
# Otherwise this call to the Makefile will made without customization.
$build_container && make container REGISTRY=e2e VERSION="${E2E_TAG}" ALLOW_STALE_APT=1
make test-tools REGISTRY=e2e

Expand Down