Standardize all Docker images.

This change moves all Docker images to a standard location, and abstracts the
build process so that they can be maintained in an automated fashion. This also
allows the images to be architecture-independent.

All images will now be referred to by the test framework via the canonical
`gvisor.dev/images/<name>`, where `<name>` is a function of the path within the
source tree.

In a subsequent change, continuous integration will be added so that the images
will always be correct and available locally.

In the end, using `bazel` for Docker containers is simply not possible. Given
that we already have the need to use `make` with the base container (for
Docker), we extend this approach to get more flexibility.

This change also adds a self-documenting and powerful Makefile that is intended
to replace the collection of scripts in scripts. Canonical (self-documenting)
targets can be added here for targets that understand which images need to be
loaded and/or built.

PiperOrigin-RevId: 308322438
This commit is contained in:
Adin Scannell
2020-04-24 14:11:42 -07:00
committed by gVisor bot
parent f13f26d17d
commit c60613475c
47 changed files with 559 additions and 201 deletions
+4 -7
View File
@@ -1,22 +1,19 @@
language: minimal
sudo: required
language: shell
dist: xenial
cache:
directories:
- /home/travis/.cache/bazel/
os: linux
services:
- docker
matrix:
jobs:
include:
- os: linux
arch: amd64
env: RUNSC_PATH=./bazel-bin/runsc/linux_amd64_pure_stripped/runsc
- os: linux
arch: arm64
env: RUNSC_PATH=./bazel-bin/runsc/linux_arm64_pure_stripped/runsc
script:
- uname -a
- make DOCKER_RUN_OPTIONS="" BAZEL_OPTIONS="build runsc:runsc" bazel && $RUNSC_PATH --alsologtostderr --network none --debug --TESTONLY-unsafe-nonroot=true --rootless do ls
- uname -a && make smoke-test
branches:
except:
# Skip copybara branches.
+7 -24
View File
@@ -108,32 +108,15 @@ ignored.
### Build and test with Docker
`scripts/dev.sh` is a convenient script that builds and installs `runsc` as a
new Docker runtime for you. The scripts tries to extract the runtime name from
your local environment and will print it at the end. You can also customize it.
The script creates one regular runtime and another with debug flags enabled.
Here are a few examples:
Running `make dev` is a convenient way to build and install `runsc` as a Docker
runtime. The output of this command will show the runtimes installed.
You may use `make refresh` to refresh the binary after any changes. For example:
```bash
# Default case (inside branch my-branch)
$ scripts/dev.sh
...
Runtimes my-branch and my-branch-d (debug enabled) setup.
Use --runtime=my-branch with your Docker command.
docker run --rm --runtime=my-branch --rm hello-world
If you rebuild, use scripts/dev.sh --refresh.
Logs are in: /tmp/my-branch/logs
# --refresh just updates the runtime binary and doesn't restart docker.
$ git/my_branch> scripts/dev.sh --refresh
# Using a custom runtime name
$ git/my_branch> scripts/dev.sh my-runtime
...
Runtimes my-runtime and my-runtime-d (debug enabled) setup.
Use --runtime=my-runtime with your Docker command.
docker run --rm --runtime=my-runtime --rm hello-world
make dev
docker run --rm --runtime=my-branch --rm hello-world
make refresh
```
### The small print
-9
View File
@@ -1,9 +0,0 @@
FROM fedora:31
RUN dnf install -y dnf-plugins-core && dnf copr enable -y vbatts/bazel
RUN dnf install -y bazel2 git gcc make golang gcc-c++ glibc-devel python3 which python3-pip python3-devel libffi-devel openssl-devel pkg-config glibc-static libstdc++-static patch
RUN pip install pycparser
WORKDIR /gvisor
+161 -38
View File
@@ -1,50 +1,173 @@
UID := $(shell id -u ${USER})
GID := $(shell id -g ${USER})
GVISOR_BAZEL_CACHE := $(shell readlink -f ~/.cache/bazel/)
#!/usr/bin/make -f
# The --privileged is required to run tests.
DOCKER_RUN_OPTIONS ?= --privileged
# Copyright 2019 The gVisor Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
all: runsc
# Described below.
OPTIONS :=
TARGETS := //runsc
ARGS :=
docker-build:
docker build -t gvisor-bazel .
default: runsc
.PHONY: default
bazel-shutdown:
docker exec -i gvisor-bazel bazel shutdown && \
docker kill gvisor-bazel
## usage: make <target>
## or
## make <build|test|copy|run|sudo> OPTIONS="..." TARGETS="..." ARGS="..."
##
## Basic targets.
##
## This Makefile wraps basic build and test targets for ease-of-use. Bazel
## is run inside a canonical Docker container in order to simplify up-front
## requirements.
##
## There are common arguments that may be passed to targets. These are:
## OPTIONS - Build or test options.
## TARGETS - The bazel targets.
## ARGS - Arguments for run or sudo.
##
## Additionally, the copy target expects a DESTINATION to be provided.
##
## For example, to build runsc using this Makefile, you can run:
## make build OPTIONS="" TARGETS="//runsc"'
##
help: ## Shows all targets and help from the Makefile (this message).
@grep --no-filename -E '^([a-z.A-Z_-]+:.*?|)##' $(MAKEFILE_LIST) | \
awk 'BEGIN {FS = "(:.*?|)## ?"}; { \
if (length($$1) > 0) { \
printf " \033[36m%-20s\033[0m %s\n", $$1, $$2; \
} else { \
printf "%s\n", $$2; \
} \
}'
build: ## Builds the given $(TARGETS) with the given $(OPTIONS). E.g. make build TARGETS=runsc
test: ## Tests the given $(TARGETS) with the given $(OPTIONS). E.g. make test TARGETS=pkg/buffer:buffer_test
copy: ## Copies the given $(TARGETS) to the given $(DESTINATION). E.g. make copy TARGETS=runsc DESTINATION=/tmp
run: ## Runs the given $(TARGETS), built with $(OPTIONS), using $(ARGS). E.g. make run TARGETS=runsc ARGS=-version
sudo: ## Runs the given $(TARGETS) as per run, but using "sudo -E". E.g. make sudo TARGETS=test/root:root_test ARGS=-test.v
.PHONY: help build test copy run sudo
bazel-server-start: docker-build
mkdir -p "$(GVISOR_BAZEL_CACHE)" && \
docker run -d --rm --name gvisor-bazel \
--user 0:0 \
-v "$(GVISOR_BAZEL_CACHE):$(HOME)/.cache/bazel/" \
-v "$(CURDIR):$(CURDIR)" \
--workdir "$(CURDIR)" \
--tmpfs /tmp:rw,exec \
$(DOCKER_RUN_OPTIONS) \
gvisor-bazel \
sh -c "while :; do sleep 100; done" && \
docker exec --user 0:0 -i gvisor-bazel sh -c "groupadd --gid $(GID) --non-unique gvisor && useradd --uid $(UID) --non-unique --gid $(GID) -d $(HOME) gvisor"
# Load all bazel wrappers.
#
# This file should define the basic "build", "test", "run" and "sudo" rules, in
# addition to the $(BRANCH_NAME) variable.
ifneq (,$(wildcard tools/google.mk))
include tools/google.mk
else
include tools/bazel.mk
endif
bazel-server:
docker exec gvisor-bazel true || \
$(MAKE) bazel-server-start
##
## Docker image targets.
##
## Images used by the tests must also be built and available locally.
## The canonical test targets defined below will automatically load
## relevant images. These can be loaded or built manually via these
## targets.
##
## (*) Note that you may provide an ARCH parameter in order to build
## and load images from an alternate archiecture (using qemu). When
## bazel is run as a server, this has the effect of running an full
## cross-architecture chain, and can produce cross-compiled binaries.
##
define images
$(1)-%: ## Image tool: $(1) a given image (also may use 'all-images').
@$(MAKE) -C images $$@
endef
rebuild-...: ## Rebuild the given image. Also may use 'rebuild-all-images'.
$(eval $(call images,rebuild))
push-...: ## Push the given image. Also may use 'push-all-images'.
$(eval $(call images,pull))
pull-...: ## Pull the given image. Also may use 'pull-all-images'.
$(eval $(call images,push))
load-...: ## Load (pull or rebuild) the given image. Also may use 'load-all-images'.
$(eval $(call images,load))
list-images: ## List all available images.
@$(MAKE) -C images $$@
BAZEL_OPTIONS := build runsc
bazel: bazel-server
docker exec -u $(UID):$(GID) -i gvisor-bazel bazel $(BAZEL_OPTIONS)
##
## Canonical build and test targets.
##
## These targets are used by continuous integration and provide
## convenient entrypoints for testing changes. If you're adding a
## new subsystem or workflow, consider adding a new target here.
##
runsc: ## Builds the runsc binary.
@$(MAKE) build TARGETS="//runsc"
.PHONY: runsc
bazel-alias:
@echo "alias bazel='docker exec -u $(UID):$(GID) -i gvisor-bazel bazel'"
smoke-test: ## Runs a simple smoke test after build runsc.
@$(MAKE) run DOCKER_RUN_OPTIONS="" ARGS="--alsologtostderr --network none --debug --TESTONLY-unsafe-nonroot=true --rootless do true"
.PHONY: smoke-tests
runsc:
$(MAKE) BAZEL_OPTIONS="build runsc" bazel
unit-tests: ## Runs all unit tests in pkg runsc and tools.
@$(MAKE) test OPTIONS="pkg/... runsc/... tools/..."
.PHONY: unit-tests
tests:
$(MAKE) BAZEL_OPTIONS="test --test_tag_filters runsc_ptrace //test/syscalls/..." bazel
tests: ## Runs all local ptrace system call tests.
@$(MAKE) test OPTIONS="--test_tag_filter runsc_ptrace test/syscalls/..."
.PHONY: tests
unit-tests:
$(MAKE) BAZEL_OPTIONS="test //pkg/... //runsc/... //tools/..." bazel
##
## Development helpers and tooling.
##
## These targets faciliate local development by automatically
## installing and configuring a runtime. Several variables may
## be used here to tweak the installation:
## RUNTIME - The name of the installed runtime (default: branch).
## RUNTIME_DIR - Where the runtime will be installed (default: temporary directory with the $RUNTIME).
## RUNTIME_BIN - The runtime binary (default: $RUNTIME_DIR/runsc).
## RUNTIME_LOG_DIR - The logs directory (default: $RUNTIME_DIR/logs).
## RUNTIME_LOGS - The log pattern (default: $RUNTIME_LOG_DIR/runsc.log.%TEST%.%TIMESTAMP%.%COMMAND%).
##
ifeq (,$(BRANCH_NAME))
RUNTIME := runsc
RUNTIME_DIR := $(shell dirname $(shell mktemp -u))/runsc
else
RUNTIME := $(BRANCH_NAME)
RUNTIME_DIR := $(shell dirname $(shell mktemp -u))/$(BRANCH_NAME)
endif
RUNTIME_BIN := $(RUNTIME_DIR)/runsc
RUNTIME_LOG_DIR := $(RUNTIME_DIR)/logs
RUNTIME_LOGS := $(RUNTIME_LOG_DIR)/runsc.log.%TEST%.%TIMESTAMP%.%COMMAND%
.PHONY: docker-build bazel-shutdown bazel-server-start bazel-server bazel runsc tests
dev: ## Installs a set of local runtimes. Requires sudo.
@$(MAKE) refresh ARGS="--net-raw"
@$(MAKE) configure RUNTIME="$(RUNTIME)" ARGS="--net-raw"
@$(MAKE) configure RUNTIME="$(RUNTIME)-d" ARGS="--net-raw --debug --strace --log-packets"
@$(MAKE) configure RUNTIME="$(RUNTIME)-p" ARGS="--net-raw --profile"
@sudo systemctl restart docker
.PHONY: dev
refresh: ## Refreshes the runtime binary (for development only). Must have called 'dev' or 'test-install' first.
@mkdir -p "$(RUNTIME_DIR)"
@$(MAKE) copy TARGETS=runsc DESTINATION="$(RUNTIME_BIN)" && chmod 0755 "$(RUNTIME_BIN)"
.PHONY: install
test-install: ## Installs the runtime for testing. Requires sudo.
@$(MAKE) refresh ARGS="--net-raw --TESTONLY-test-name-env=RUNSC_TEST_NAME --debug --strace --log-packets $(ARGS)"
@$(MAKE) configure
@sudo systemctl restart docker
.PHONY: install-test
configure: ## Configures a single runtime. Requires sudo. Typically called from dev or test-install.
@sudo sudo "$(RUNTIME_BIN)" install --experimental=true --runtime="$(RUNTIME)" -- --debug-log "$(RUNTIME_LOGS)" $(ARGS)
@echo "Installed runtime \"$(RUNTIME)\" @ $(RUNTIME_BIN)"
@echo "Logs are in: $(RUNTIME_LOG_DIR)"
@sudo rm -rf "$(RUNTIME_LOG_DIR)" && mkdir -p "$(RUNTIME_LOG_DIR)"
.PHONY: configure
test-runtime: ## A convenient wrapper around test that provides the runtime argument. Target must still be provided.
@$(MAKE) test OPTIONS="$(OPTIONS) --test_arg=--runtime=$(RUNTIME)"
.PHONY: runtime-test
+11
View File
@@ -0,0 +1,11 @@
package(licenses = ["notice"])
# The images filegroup is definitely not a hermetic target, and requires Make
# to do anything meaningful with. However, this will be slurped up and used by
# the tools/installer/images.sh installer, which will ensure that all required
# images are available locally when running vm_tests.
filegroup(
name = "images",
srcs = glob(["**"]),
visibility = ["//tools/installers:__pkg__"],
)
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/make -f
# Copyright 2018 The gVisor Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ARCH is the architecture used for the build. This may be overriden at the
# command line in order to perform a cross-build (in a limited capacity).
ARCH := $(shell uname -m)
# Note that the image prefixes used here must match the image mangling in
# runsc/testutil.MangleImage. Names are mangled in this way to ensure that all
# tests are using locally-defined images (that are consistent and idempotent).
REMOTE_IMAGE_PREFIX ?= gcr.io/gvisor-presubmit
LOCAL_IMAGE_PREFIX ?= gvisor.dev/images
ALL_IMAGES := $(subst /,_,$(subst ./,,$(shell find . -name Dockerfile -exec dirname {} \;)))
ifneq ($(ARCH),$(shell uname -m))
DOCKER_PLATFORM_ARGS := --platform=$(ARCH)
else
DOCKER_PLATFORM_ARGS :=
endif
list-all-images:
@for image in $(ALL_IMAGES); do echo $${image}; done
.PHONY: list-build-images
%-all-images:
@$(MAKE) $(patsubst %,$*-%,$(ALL_IMAGES))
# tag is a function that returns the tag name, given an image.
#
# The tag constructed is used to memoize the image generated (see README.md).
# This scheme is used to enable aggressive caching in a central repository, but
# ensuring that images will always be sourced using the local files if there
# are changes.
path = $(subst _,/,$(1))
tag = $(shell find $(call path,$(1)) -type f -print | sort | xargs -n 1 sha256sum | sha256sum - | cut -c 1-16)
remote_image = $(REMOTE_IMAGE_PREFIX)/$(subst _,/,$(1))_$(ARCH):$(call tag,$(1))
local_image = $(LOCAL_IMAGE_PREFIX)/$(subst _,/,$(1))
# rebuild builds the image locally. Only the "remote" tag will be applied. Note
# we need to explicitly repull the base layer in order to ensure that the
# architecture is correct. Note that we use the term "rebuild" here to avoid
# conflicting with the bazel "build" terminology, which is used elsewhere.
rebuild-%: register-cross
FROM=$(shell grep FROM $(call path,$*)/Dockerfile | cut -d' ' -f2-) && \
docker pull $(DOCKER_PLATFORM_ARGS) $$FROM
T=$$(mktemp -d) && cp -a $(call path,$*)/* $$T && \
docker build $(DOCKER_PLATFORM_ARGS) -t $(call remote_image,$*) $$T && \
rm -rf $$T
# pull will check the "remote" image and pull if necessary. If the remote image
# must be pulled, then it will tag with the latest local target. Note that pull
# may fail if the remote image is not available.
pull-%:
docker pull $(DOCKER_PLATFORM_ARGS) $(call remote_image,$*)
# load will either pull the "remote" or build it locally. This is the preferred
# entrypoint, as it should never file. The local tag should always be set after
# this returns (either by the pull or the build).
load-%:
docker inspect $(call remote_image,$*) >/dev/null 2>&1 || $(MAKE) pull-$* || $(MAKE) rebuild-$*
docker tag $(call remote_image,$*) $(call local_image,$*)
# push pushes the remote image, after either pulling (to validate that the tag
# already exists) or building manually.
push-%: load-%
docker push $(call remote_image,$*)
# register-cross registers the necessary qemu binaries for cross-compilation.
# This may be used by any target that may execute containers that are not the
# native format.
register-cross:
ifneq ($(ARCH),$(shell uname -m))
ifeq (,$(wildcard /proc/sys/fs/binfmt_misc/qemu-*))
docker run --rm --privileged multiarch/qemu-user-static --reset --persistent yes
else
@true # Already registered.
endif
else
@true # No cross required.
endif
.PHONY: register-cross
+61
View File
@@ -0,0 +1,61 @@
# Container Images
This directory contains all images used by tests.
Note that all these images must be pushed to the testing project hosted on
[Google Container Registry][gcr]. This will happen automatically as part of
continuous integration. This will speed up loading as images will not need to be
built from scratch for each test run.
Image tooling is accessible via `make`, specifically via `tools/images.mk`.
## Why make?
Make is used because it can bootstrap the `default` image, which contains
`bazel` and all other parts of the toolchain.
## Listing images
To list all images, use `make list-all-images` from the top-level directory.
## Loading and referencing images
To build a specific image, use `make load-<image>` from the top-level directory.
This will ensure that an image `gvisor.dev/images/<image>:latest` is available.
Images should always be referred to via the `gvisor.dev/images` canonical path.
This tag exists only locally, but serves to decouple tests from the underlying
image infrastructure.
The continuous integration system can either take fine-grained dependencies on
single images via individual `load` targets, or pull all images via a single
`load-all-images` invocation.
## Adding new images
To add a new image, create a new directory under `images` containing a
Dockerfile and any other files that the image requires. You may choose to add to
an existing subdirectory if applicable, or create a new one.
All images will be tagged and memoized using a hash of the directory contents.
As a result, every image should be made completely reproducible if possible.
This means using fixed tags and fixed versions whenever feasible.
Notes that images should also be made architecture-independent if possible. The
build scripts will handling loading the appropriate architecture onto the
machine and tagging it with the single canonical tag.
Add a `load-<image>` dependency in the Makefile if the image is required for a
particular set of tests. This target will pull the tag from the image repository
if available.
## Building and pushing images
All images can be built manually by running `build-<image>` and pushed using
`push-<image>`. Note that you can also use `build-all-images` and
`push-all-images`. Note that pushing will require appropriate permissions in the
project.
The continuous integration system can either take fine-grained dependencies on
individual `push` targets, or ensure all images are up-to-date with a single
`push-all-images` invocation.
+1
View File
@@ -0,0 +1 @@
FROM alpine:3.11.5
+1
View File
@@ -0,0 +1 @@
FROM busybox:1.31.1
+1
View File
@@ -0,0 +1 @@
FROM httpd:2.4.43
+1
View File
@@ -0,0 +1 @@
FROM mysql:8.0.19
+1
View File
@@ -0,0 +1 @@
FROM nginx:1.17.9
+2
View File
@@ -0,0 +1,2 @@
FROM python:3
ENTRYPOINT ["python", "-m", "http.server", "8080"]
+1
View File
@@ -0,0 +1 @@
FROM k8s.gcr.io/busybox:latest
+1
View File
@@ -0,0 +1 @@
FROM ruby:2.7.1
+1
View File
@@ -0,0 +1 @@
FROM tomcat:8.0
+1
View File
@@ -0,0 +1 @@
FROM ubuntu:trusty
+11
View File
@@ -0,0 +1,11 @@
FROM fedora:31
RUN dnf install -y dnf-plugins-core && dnf copr enable -y vbatts/bazel
RUN dnf install -y git gcc make golang gcc-c++ glibc-devel python3 which python3-pip python3-devel libffi-devel openssl-devel pkg-config glibc-static libstdc++-static patch
RUN pip install pycparser
RUN dnf install -y bazel3
RUN curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-289.0.0-linux-x86_64.tar.gz | \
tar zxvf - google-cloud-sdk && \
google-cloud-sdk/install.sh && \
ln -s /google-cloud-sdk/bin/gcloud /usr/bin/gcloud
WORKDIR /workspace
ENTRYPOINT ["/usr/bin/bazel"]
+2
View File
@@ -0,0 +1,2 @@
FROM ubuntu
RUN apt update && apt install -y iptables
@@ -1,5 +1,4 @@
FROM ubuntu:bionic
RUN apt-get update && apt-get install -y net-tools git iptables iputils-ping \
netcat tcpdump jq tar bison flex make
RUN hash -r

Some files were not shown because too many files have changed in this diff Show More