netbird/gvisor
0
0
Fork 0
mirror of https://github.com/netbirdio/gvisor.git synced 2026-05-22 17:12:49 -07:00
Code Issues Packages Projects Releases Wiki Activity

Remove old benchmark tools.

Browse Source 
Remove the old benchmark-tools directory, including
imports in the WORKSPACE file and associated bazel rules.

The new Golang benchmark-tools can be found at //test/benchmarks
and it is functionally equivalent, excepting syscall_test
which can be found in //test/perf/linux.

PiperOrigin-RevId: 325529075
This commit is contained in:
Zach Koopmans
2020-08-07 16:18:51 -07:00
committed by gVisor bot
parent 94447aeab3
commit 80c80a1410
130 changed files with 8 additions and 7014 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Makefile
+1 -1
View File
@@ -122,7 +122,7 @@ smoke-tests: ## Runs a simple smoke test after build runsc.
.PHONY: smoke-tests
unit-tests: ## Local package unit tests in pkg/..., runsc/, tools/.., etc.
@$(call submake,test TARGETS="pkg/... runsc/... tools/... benchmarks/... benchmarks/runner:runner_test")
@$(call submake,test TARGETS="pkg/... runsc/... tools/...")
tests: ## Runs all unit tests and syscall tests.
tests: unit-tests
WORKSPACE
-20
View File
@@ -113,26 +113,6 @@ rules_proto_dependencies()
rules_proto_toolchains()
# Load python dependencies.
git_repository(
name = "rules_python",
commit = "abc4869e02fe9b3866942e89f07b7341f830e805",
remote = "https://github.com/bazelbuild/rules_python.git",
shallow_since = "1583341286 -0500",
)
load("@rules_python//python:pip.bzl", "pip_import")
pip_import(
name = "pydeps",
python_interpreter = "python3",
requirements = "//benchmarks:requirements.txt",
)
load("@pydeps//:requirements.bzl", "pip_install")
pip_install()
# Load bazel_toolchain to support Remote Build Execution.
# See releases at https://releases.bazel.build/bazel-toolchains.html
http_archive(
benchmarks/BUILD
-37
View File
@@ -1,37 +0,0 @@
load("//tools:defs.bzl", "bzl_library")
package(licenses = ["notice"])
config_setting(
name = "gcloud_rule",
values = {
"define": "gcloud=off",
},
)
py_binary(
name = "benchmarks",
testonly = 1,
srcs = ["run.py"],
data = select({
":gcloud_rule": [],
"//conditions:default": [
"//tools/vm:ubuntu1604",
"//tools/vm:zone",
],
}),
main = "run.py",
python_version = "PY3",
srcs_version = "PY3",
tags = [
"local",
"manual",
],
deps = ["//benchmarks/runner"],
)
bzl_library(
name = "defs_bzl",
srcs = ["defs.bzl"],
visibility = ["//visibility:private"],
)
benchmarks/README.md
-186
View File
@@ -1,186 +0,0 @@
# Benchmark tools
These scripts are tools for collecting performance data for Docker-based tests.
## Setup
The scripts assume the following:
* There are two sets of machines: one where the scripts will be run
(controller) and one or more machines on which docker containers will be run
(environment).
* The controller machine must have bazel installed along with this source
code. You should be able to run a command like `bazel run //benchmarks --
--list`
* Environment machines must have docker and the required runtimes installed.
More specifically, you should be able to run a command like: `docker run
--runtime=$RUNTIME your/image`.
* The controller has ssh private key which can be used to login to environment
machines and run docker commands without using `sudo`. This is not required
if running locally via the `run-local` command.
* The docker daemon on each of your environment machines is listening on
`unix:///var/run/docker.sock` (docker's default).
For configuring the environment manually, consult the
[dockerd documentation][dockerd].
## Running benchmarks
### Locally
The tool is built to, by default, use Google Cloud Platform to run benchmarks,
but it does support GCP workflows. To run locally, run the following from the
benchmarks directory:
```bash
bazel run --define gcloud=off //benchmarks -- run-local startup
...
method,metric,result
startup.empty,startup_time_ms,652.5772
startup.node,startup_time_ms,1654.4042000000002
startup.ruby,startup_time_ms,1429.835
```
The above command ran the startup benchmark locally, which consists of three
benchmarks (empty, node, and ruby). Benchmark tools ran it on the default
runtime, runc. Running on another installed runtime, like say runsc, is as
simple as:
```bash
bazel run --define gcloud=off //benchmarks -- run-local startup --runtime=runsc
```
There is help:
```bash
bazel run --define gcloud=off //benchmarks -- --help
bazel run --define gcloud=off //benchmarks -- run-local --help
```
To list available benchmarks, use the `list` commmand:
```bash
bazel --define gcloud=off run //benchmarks -- list
...
Benchmark: sysbench.cpu
Metrics: events_per_second
Run sysbench CPU test. Additional arguments can be provided for sysbench.
:param max_prime: The maximum prime number to search.
```
You can choose benchmarks by name or regex like:
```bash
bazel run --define gcloud=off //benchmarks -- run-local startup.node
...
metric,result
startup_time_ms,1671.7178000000001
```
or
```bash
bazel run --define gcloud=off //benchmarks -- run-local s
...
method,metric,result
startup.empty,startup_time_ms,1792.8292
startup.node,startup_time_ms,3113.5274
startup.ruby,startup_time_ms,3025.2424
sysbench.cpu,cpu_events_per_second,12661.47
sysbench.memory,memory_ops_per_second,7228268.44
sysbench.mutex,mutex_time,17.4835
sysbench.mutex,mutex_latency,3496.7
sysbench.mutex,mutex_deviation,0.04
syscall.syscall,syscall_time_ns,2065.0
```
You can run parameterized benchmarks, for example to run with different
runtimes:
```bash
bazel run --define gcloud=off //benchmarks -- run-local --runtime=runc --runtime=runsc sysbench.cpu
```
Or with different parameters:
```bash
bazel run --define gcloud=off //benchmarks -- run-local --max_prime=10 --max_prime=100 sysbench.cpu
```
### On Google Compute Engine (GCE)
Benchmarks may be run on GCE in an automated way. The default project configured
for `gcloud` will be used.
An additional parameter `installers` may be provided to ensure that the latest
runtime is installed from the workspace. See the files in `tools/installers` for
supported install targets.
```bash
bazel run //benchmarks -- run-gcp --installers=head --runtime=runsc sysbench.cpu
```
When running on GCE, the scripts generate a per run SSH key, which is added to
your project. The key is set to expire in GCE after 60 minutes and is stored in
a temporary directory on the local machine running the scripts.
## Writing benchmarks
To write new benchmarks, you should familiarize yourself with the structure of
the repository. There are three key components.
## Harness
The harness makes use of the [docker py SDK][docker-py]. It is advisable that
you familiarize yourself with that API when making changes, specifically:
* clients
* containers
* images
In general, benchmarks need only interact with the `Machine` objects provided to
the benchmark function, which are the machines defined in the environment. These
objects allow the benchmark to define the relationships between different
containers, and parse the output.
## Workloads
The harness requires workloads to run. These are all available in the
`workloads` directory.
In general, a workload consists of a Dockerfile to build it (while these are not
hermetic, in general they should be as fixed and isolated as possible), some
parsers for output if required, parser tests and sample data. Provided the test
is named after the workload package and contains a function named `sample`, this
variable will be used to automatically mock workload output when the `--mock`
flag is provided to the main tool.
## Writing benchmarks
Benchmarks define the tests themselves. All benchmarks have the following
function signature:
```python
def my_func(output) -> float:
return float(output)
@benchmark(metrics = my_func, machines = 1)
def my_benchmark(machine: machine.Machine, arg: str):
return "3.4432"
```
Each benchmark takes a variable amount of position arguments as
`harness.Machine` objects and some set of keyword arguments. It is recommended
that you accept arbitrary keyword arguments and pass them through when
constructing the container under test.
To write a new benchmark, open a module in the `suites` directory and use the
above signature. You should add a descriptive doc string to describe what your
benchmark is and any test centric arguments.
[dockerd]: https://docs.docker.com/engine/reference/commandline/dockerd/
[docker-py]: https://docker-py.readthedocs.io/en/stable/
benchmarks/defs.bzl
-14
View File
@@ -1,14 +0,0 @@
"""Provides attributes common to many workload tests."""
load("//tools:defs.bzl", "py_requirement")
test_deps = [
py_requirement("attrs", direct = False),
py_requirement("atomicwrites", direct = False),
py_requirement("more-itertools", direct = False),
py_requirement("pathlib2", direct = False),
py_requirement("pluggy", direct = False),
py_requirement("py", direct = False),
py_requirement("pytest"),
py_requirement("six", direct = False),
]
benchmarks/examples/localhost.yaml
-2
View File
@@ -1,2 +0,0 @@
client: localhost
server: localhost
benchmarks/harness/BUILD
-201
View File
@@ -1,201 +0,0 @@
load("//tools:defs.bzl", "pkg_tar", "py_library", "py_requirement")
package(
default_visibility = ["//benchmarks:__subpackages__"],
licenses = ["notice"],
)
pkg_tar(
name = "installers",
srcs = [
"//tools/installers:head",
"//tools/installers:master",
],
mode = "0755",
)
filegroup(
name = "files",
srcs = [
":installers",
],
)
py_library(
name = "harness",
srcs = ["__init__.py"],
data = [
":files",
],
)
py_library(
name = "benchmark_driver",
srcs = ["benchmark_driver.py"],
deps = [
"//benchmarks/harness/machine_mocks",
"//benchmarks/harness/machine_producers:machine_producer",
"//benchmarks/suites",
],
)
py_library(
name = "container",
srcs = ["container.py"],
deps = [
"//benchmarks/workloads",
py_requirement(
"asn1crypto",
direct = False,
),
py_requirement(
"chardet",
direct = False,
),
py_requirement(
"certifi",
direct = False,
),
py_requirement("docker"),
py_requirement(
"docker-pycreds",
direct = False,
),
py_requirement(
"idna",
direct = False,
),
py_requirement(
"ptyprocess",
direct = False,
),
py_requirement(
"requests",
direct = False,
),
py_requirement(
"urllib3",
direct = False,
),
py_requirement(
"websocket-client",
direct = False,
),
],
)
py_library(
name = "machine",
srcs = ["machine.py"],
deps = [
"//benchmarks/harness",
"//benchmarks/harness:container",
"//benchmarks/harness:ssh_connection",
"//benchmarks/harness:tunnel_dispatcher",
"//benchmarks/harness/machine_mocks",
py_requirement(
"asn1crypto",
direct = False,
),
py_requirement(
"chardet",
direct = False,
),
py_requirement(
"certifi",
direct = False,
),
py_requirement("docker"),
py_requirement(
"docker-pycreds",
direct = False,
),
py_requirement(
"idna",
direct = False,
),
py_requirement(
"ptyprocess",
direct = False,
),
py_requirement(
"requests",
direct = False,
),
py_requirement(
"six",
direct = False,
),
py_requirement(
"urllib3",
direct = False,
),
py_requirement(
"websocket-client",
direct = False,
),
],
)
py_library(
name = "ssh_connection",
srcs = ["ssh_connection.py"],
deps = [
"//benchmarks/harness",
py_requirement(
"bcrypt",
direct = False,
),
py_requirement("cffi"),
py_requirement("paramiko"),
py_requirement(
"cryptography",
direct = False,
),
],
)
py_library(
name = "tunnel_dispatcher",
srcs = ["tunnel_dispatcher.py"],
deps = [
py_requirement(
"asn1crypto",
direct = False,
),
py_requirement(
"chardet",
direct = False,
),
py_requirement(
"certifi",
direct = False,
),
py_requirement("docker"),
py_requirement(
"docker-pycreds",
direct = False,
),
py_requirement(
"idna",
direct = False,
),
py_requirement("pexpect"),
py_requirement(
"ptyprocess",
direct = False,
),
py_requirement(
"requests",
direct = False,
),
py_requirement(
"urllib3",
direct = False,
),
py_requirement(
"websocket-client",
direct = False,
),
],
)
benchmarks/harness/__init__.py
-62
View File
@@ -1,62 +0,0 @@
# python3
# Copyright 2019 The gVisor Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Core benchmark utilities."""
import getpass
import os
import subprocess
import tempfile
# LOCAL_WORKLOADS_PATH defines the path to use for local workloads. This is a
# format string that accepts a single string parameter.
LOCAL_WORKLOADS_PATH = os.path.dirname(__file__) + "/../workloads/{}/tar.tar"
# REMOTE_WORKLOADS_PATH defines the path to use for storing the workloads on the
# remote host. This is a format string that accepts a single string parameter.
REMOTE_WORKLOADS_PATH = "workloads/{}"
# INSTALLER_ROOT is the set of files that needs to be copied.
INSTALLER_ARCHIVE = os.readlink(os.path.join(
os.path.dirname(__file__), "installers.tar"))
# SSH_KEY_DIR holds SSH_PRIVATE_KEY for this run. bm-tools paramiko requires
# keys generated with the '-t rsa -m PEM' options from ssh-keygen. This is
# abstracted away from the user.
SSH_KEY_DIR = tempfile.TemporaryDirectory()
SSH_PRIVATE_KEY = "key"
# DEFAULT_USER is the default user running this script.
DEFAULT_USER = getpass.getuser()
# DEFAULT_USER_HOME is the home directory of the user running the script.
DEFAULT_USER_HOME = os.environ["HOME"] if "HOME" in os.environ else ""
# Default directory to remotely installer "installer" targets.
REMOTE_INSTALLERS_PATH = "installers"
def make_key():
"""Wraps a valid ssh key in a temporary directory."""
path = os.path.join(SSH_KEY_DIR.name, SSH_PRIVATE_KEY)
if not os.path.exists(path):
cmd = "ssh-keygen -t rsa -m PEM -b 4096 -f {key} -q -N".format(
key=path).split(" ")
cmd.append("")
subprocess.run(cmd, check=True)
return path
def delete_key():
"""Deletes temporary directory containing private key."""
SSH_KEY_DIR.cleanup()
benchmarks/harness/benchmark_driver.py
-85
View File
@@ -1,85 +0,0 @@
# python3
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Main driver for benchmarks."""
import copy
import statistics
import threading
import types
from benchmarks import suites
from benchmarks.harness.machine_producers import machine_producer
# pylint: disable=too-many-instance-attributes
class BenchmarkDriver:
"""Allocates machines and invokes a benchmark method."""
def __init__(self,
producer: machine_producer.MachineProducer,
method: types.FunctionType,
runs: int = 1,
**kwargs):
self._producer = producer
self._method = method
self._kwargs = copy.deepcopy(kwargs)
self._threads = []
self.lock = threading.RLock()
self._runs = runs
self._metric_results = {}
def start(self):
"""Starts a benchmark thread."""
for _ in range(self._runs):
thread = threading.Thread(target=self._run_method)
thread.start()
self._threads.append(thread)
def join(self):
"""Joins the thread."""
# pylint: disable=expression-not-assigned
[t.join() for t in self._threads]
def _run_method(self):
"""Runs all benchmarks."""
machines = self._producer.get_machines(
suites.benchmark_machines(self._method))
try:
result = self._method(*machines, **self._kwargs)
for name, res in result:
with self.lock:
if name in self._metric_results:
self._metric_results[name].append(res)
else:
self._metric_results[name] = [res]
finally:
# Always release.
self._producer.release_machines(machines)
def median(self):
"""Returns the median result, after join is finished."""
for key, value in self._metric_results.items():
yield key, [statistics.median(value)]
def all(self):
"""Returns all results."""
for key, value in self._metric_results.items():
yield key, value
def meanstd(self):
"""Returns all results."""
for key, value in self._metric_results.items():
mean = statistics.mean(value)
yield key, [mean, statistics.stdev(value, xbar=mean)]
benchmarks/harness/container.py
-181
View File
@@ -1,181 +0,0 @@
# python3
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Container definitions."""
import contextlib
import logging
import pydoc
import types
from typing import Tuple
import docker
import docker.errors
from benchmarks import workloads
class Container:
"""Abstract container.
Must be a context manager.
Usage:
with Container(client, image, ...):
...
"""
def run(self, **env) -> str:
"""Run the container synchronously."""
raise NotImplementedError
def detach(self, **env):
"""Run the container asynchronously."""
raise NotImplementedError
def address(self) -> Tuple[str, int]:
"""Return the bound address for the container."""
raise NotImplementedError
def get_names(self) -> types.GeneratorType:
"""Return names of all containers."""
raise NotImplementedError
# pylint: disable=too-many-instance-attributes
class DockerContainer(Container):
"""Class that handles creating a docker container."""
# pylint: disable=too-many-arguments
def __init__(self,
client: docker.DockerClient,
host: str,
image: str,
count: int = 1,
runtime: str = "runc",
port: int = 0,
**kwargs):
"""Trys to setup "count" containers.
Args:
client: A docker client from dockerpy.
host: The host address the image is running on.
image: The name of the image to run.
count: The number of containers to setup.
runtime: The container runtime to use.
port: The port to reserve.
**kwargs: Additional container options.
"""
assert count >= 1
assert port == 0 or count == 1
self._client = client
self._host = host
self._containers = []
self._count = count
self._image = image
self._runtime = runtime
self._port = port
self._kwargs = kwargs
if port != 0:
self._ports = {"%d/tcp" % port: None}
else:
self._ports = {}
@contextlib.contextmanager
def detach(self, **env):
env = ["%s=%s" % (key, value) for (key, value) in env.items()]
# Start all containers.
for _ in range(self._count):
try:
# Start the container in a detached mode.
container = self._client.containers.run(
self._image,
detach=True,
remove=True,
runtime=self._runtime,
ports=self._ports,
environment=env,
**self._kwargs)
logging.info("Started detached container %s -> %s", self._image,
container.attrs["Id"])
self._containers.append(container)
except Exception as exc:
self._clean_containers()
raise exc
try:
# Wait for all containers to be up.
for container in self._containers:
while not container.attrs["State"]["Running"]:
container = self._client.containers.get(container.attrs["Id"])
yield self
finally:
self._clean_containers()
def address(self) -> Tuple[str, int]:
assert self._count == 1
assert self._port != 0
container = self._client.containers.get(self._containers[0].attrs["Id"])
port = container.attrs["NetworkSettings"]["Ports"][
"%d/tcp" % self._port][0]["HostPort"]
return (self._host, port)
def get_names(self) -> types.GeneratorType:
for container in self._containers:
yield container.name
def run(self, **env) -> str:
env = ["%s=%s" % (key, value) for (key, value) in env.items()]
return self._client.containers.run(
self._image,
runtime=self._runtime,
ports=self._ports,
remove=True,
environment=env,
**self._kwargs).decode("utf-8")
def _clean_containers(self):
"""Kills all containers."""
for container in self._containers:
try:
container.kill()
except docker.errors.NotFound:
pass
class MockContainer(Container):
"""Mock of Container."""
def __init__(self, workload: str):
self._workload = workload
def __enter__(self):
return self
def run(self, **env):
# Lookup sample data if any exists for the workload module. We use a
# well-defined test locate and a well-defined sample function.
mod = pydoc.locate(workloads.__name__ + "." + self._workload)
if hasattr(mod, "sample"):
return mod.sample(**env)
return "" # No output.
def address(self) -> Tuple[str, int]:
return ("example.com", 80)
def get_names(self) -> types.GeneratorType:
yield "mock"
@contextlib.contextmanager
def detach(self, **env):
yield self
benchmarks/harness/machine.py
-265
View File
@@ -1,265 +0,0 @@
# python3
# Copyright 2019 The gVisor Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Machine abstraction passed to benchmarks to run docker containers.
Abstraction for interacting with test machines. Machines are produced
by Machine producers and represent a local or remote machine. Benchmark
methods in /benchmarks/suite are passed the required number of machines in order
to run the benchmark. Machines contain methods to run commands via bash,
possibly over ssh. Machines also hold a connection to the docker UNIX socket
to run contianers.
Typical usage example:
machine = Machine()
machine.run(cmd)
machine.pull(path)
container = machine.container()
"""
import logging
import os
import re
import subprocess
import time
from typing import List, Tuple
import docker
from benchmarks import harness
from benchmarks.harness import container
from benchmarks.harness import machine_mocks
from benchmarks.harness import ssh_connection
from benchmarks.harness import tunnel_dispatcher
log = logging.getLogger(__name__)
class Machine(object):
"""The machine object is the primary object for benchmarks.
Machine objects are passed to each metric function call and benchmarks use
machines to access real connections to those machines.
Attributes:
_name: Name as a string
"""
_name = ""
def run(self, cmd: str) -> Tuple[str, str]:
"""Convenience method for running a bash command on a machine object.
Some machines may point to the local machine, and thus, do not have ssh
connections. Run runs a command either local or over ssh and returns the
output stdout and stderr as strings.
Args:
cmd: The command to run as a string.
Returns:
The command output.
"""
raise NotImplementedError
def read(self, path: str) -> str:
"""Reads the contents of some file.
This will be mocked.
Args:
path: The path to the file to be read.
Returns:
The file contents.
"""
raise NotImplementedError
def pull(self, workload: str) -> str:
"""Send the given workload to the machine, build and tag it.
All images must be defined by the workloads directory.
Args:
workload: The workload name.
Returns:
The workload tag.
"""
raise NotImplementedError
def container(self, image: str, **kwargs) -> container.Container:
"""Returns a container object.
Args:
image: The pulled image tag.
**kwargs: Additional container options.
Returns:
:return: a container.Container object.
"""
raise NotImplementedError
def sleep(self, amount: float):
"""Sleeps the given amount of time."""
time.sleep(amount)
def __str__(self):
return self._name
class MockMachine(Machine):
"""A mocked machine."""
_name = "mock"
def run(self, cmd: str) -> Tuple[str, str]:
return "", ""
def read(self, path: str) -> str:
return machine_mocks.Readfile(path)
def pull(self, workload: str) -> str:
return workload # Workload is the tag.
def container(self, image: str, **kwargs) -> container.Container:
return container.MockContainer(image)
def sleep(self, amount: float):
pass
def get_address(machine: Machine) -> str:
"""Return a machine's default address."""
default_route, _ = machine.run("ip route get 8.8.8.8")
return re.search(" src ([0-9.]+) ", default_route).group(1)
class LocalMachine(Machine):
"""The local machine.
Attributes:
_name: Name as a string
_docker_client: a pythonic connection to to the local dockerd unix socket.
See: https://github.com/docker/docker-py
"""
def __init__(self, name):
self._name = name
self._docker_client = docker.from_env()
def run(self, cmd: str) -> Tuple[str, str]:
process = subprocess.Popen(
cmd.split(" "), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = process.communicate()
return stdout.decode("utf-8"), stderr.decode("utf-8")
def read(self, path: str) -> bytes:
# Read the exact path locally.
return open(path, "r").read()
def pull(self, workload: str) -> str:
# Run the docker build command locally.
logging.info("Building %s@%s locally...", workload, self._name)
with open(harness.LOCAL_WORKLOADS_PATH.format(workload),
"rb") as dockerfile:
self._docker_client.images.build(
fileobj=dockerfile, tag=workload, custom_context=True)
return workload # Workload is the tag.
def container(self, image: str, **kwargs) -> container.Container:
# Return a local docker container directly.
return container.DockerContainer(self._docker_client, get_address(self),
image, **kwargs)
def sleep(self, amount: float):
time.sleep(amount)
class RemoteMachine(Machine):
"""Remote machine accessible via an SSH connection.
Attributes:
_name: Name as a string
_ssh_connection: a paramiko backed ssh connection which can be used to run
commands on this machine
_tunnel: a python wrapper around a port forwarded ssh connection between a
local unix socket and the remote machine's dockerd unix socket.
_docker_client: a pythonic wrapper backed by the _tunnel. Allows sending
docker commands: see https://github.com/docker/docker-py
"""
def __init__(self, name, **kwargs):
self._name = name
self._ssh_connection = ssh_connection.SSHConnection(name, **kwargs)
self._tunnel = tunnel_dispatcher.Tunnel(name, **kwargs)
self._tunnel.connect()
self._docker_client = self._tunnel.get_docker_client()
self._has_installers = False
def run(self, cmd: str) -> Tuple[str, str]:
return self._ssh_connection.run(cmd)
def read(self, path: str) -> str:
# Just cat remotely.
stdout, stderr = self._ssh_connection.run("cat '{}'".format(path))
return stdout + stderr
def install(self,
installer: str,
results: List[bool] = None,
index: int = -1):
"""Method unique to RemoteMachine to handle installation of installers.
Handles installers, which install things that may change between runs (e.g.
runsc). Usually called from gcloud_producer, which expects this method to
to store results.
Args:
installer: the installer target to run.
results: Passed by the caller of where to store success.
index: Index for this method to store the result in the passed results
list.
"""
# This generates a tarball of the full installer root (which will generate
# be the full bazel root directory) and sends it over.
if not self._has_installers:
archive = self._ssh_connection.send_installers()
self.run("tar -xvf {archive} -C {dir}".format(
archive=archive, dir=harness.REMOTE_INSTALLERS_PATH))
self._has_installers = True
# Execute the remote installer.
self.run("sudo {dir}/{file}".format(
dir=harness.REMOTE_INSTALLERS_PATH, file=installer))
if results:
results[index] = True
def pull(self, workload: str) -> str:
# Push to the remote machine and build.
logging.info("Building %s@%s remotely...", workload, self._name)
remote_path = self._ssh_connection.send_workload(workload)
remote_dir = os.path.dirname(remote_path)
# Workloads are all tarballs.
self.run("tar -xvf {remote_path} -C {remote_dir}".format(
remote_path=remote_path, remote_dir=remote_dir))
self.run("docker build --tag={} {}".format(workload, remote_dir))
return workload # Workload is the tag.
def container(self, image: str, **kwargs) -> container.Container:
# Return a remote docker container.
return container.DockerContainer(self._docker_client, get_address(self),
image, **kwargs)
def sleep(self, amount: float):
time.sleep(amount)
benchmarks/harness/machine_mocks/BUILD
-9
View File
@@ -1,9 +0,0 @@
package(
default_visibility = ["//benchmarks:__subpackages__"],
licenses = ["notice"],
)
py_library(
name = "machine_mocks",
srcs = ["__init__.py"],
)
benchmarks/harness/machine_mocks/__init__.py
-81
View File
@@ -1,81 +0,0 @@
# python3
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Machine mock files."""
MEMINFO = """\
MemTotal: 7652344 kB
MemFree: 7174724 kB
MemAvailable: 7152008 kB
Buffers: 7544 kB
Cached: 178856 kB
SwapCached: 0 kB
Active: 270928 kB
Inactive: 68436 kB
Active(anon): 153124 kB
Inactive(anon): 880 kB
Active(file): 117804 kB
Inactive(file): 67556 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 900 kB
Writeback: 0 kB
AnonPages: 153000 kB
Mapped: 129120 kB
Shmem: 1044 kB
Slab: 60864 kB
SReclaimable: 22792 kB
SUnreclaim: 38072 kB
KernelStack: 2672 kB
PageTables: 5756 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3826172 kB
Committed_AS: 663836 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 94196 kB
DirectMap2M: 4624384 kB
DirectMap1G: 3145728 kB
"""
CONTENTS = {
"/proc/meminfo": MEMINFO,
}
def Readfile(path: str) -> str:
"""Reads a mock file.
Args:
path: The target path.
Returns:
Mocked file contents or None.
"""
return CONTENTS.get(path, None)
benchmarks/harness/machine_producers/BUILD
-84
View File
@@ -1,84 +0,0 @@
load("//tools:defs.bzl", "py_library", "py_requirement")
package(
default_visibility = ["//benchmarks:__subpackages__"],
licenses = ["notice"],
)
py_library(
name = "harness",
srcs = ["__init__.py"],
)
py_library(
name = "machine_producer",
srcs = ["machine_producer.py"],
)
py_library(
name = "mock_producer",
srcs = ["mock_producer.py"],
deps = [
"//benchmarks/harness:machine",
"//benchmarks/harness/machine_producers:gcloud_producer",
"//benchmarks/harness/machine_producers:machine_producer",
],
)
py_library(
name = "yaml_producer",
srcs = ["yaml_producer.py"],
deps = [
"//benchmarks/harness:machine",
"//benchmarks/harness/machine_producers:machine_producer",
py_requirement(
"PyYAML",
direct = False,
),
],
)
py_library(
name = "gcloud_mock_recorder",
srcs = ["gcloud_mock_recorder.py"],
)
py_library(
name = "gcloud_producer",
srcs = ["gcloud_producer.py"],
deps = [
"//benchmarks/harness:machine",
"//benchmarks/harness/machine_producers:gcloud_mock_recorder",
"//benchmarks/harness/machine_producers:machine_producer",
],
)
filegroup(
name = "test_data",
srcs = [
"testdata/get_five.json",
"testdata/get_one.json",
],
)
py_library(
name = "gcloud_producer_test_lib",
srcs = ["gcloud_producer_test.py"],
deps = [
"//benchmarks/harness/machine_producers:machine_producer",
"//benchmarks/harness/machine_producers:mock_producer",
],
)
py_test(
name = "gcloud_producer_test",
srcs = [":gcloud_producer_test_lib"],
data = [
":test_data",
],
python_version = "PY3",
tags = [
"local",
"manual",
],
)
benchmarks/harness/machine_producers/__init__.py
-13
View File
@@ -1,13 +0,0 @@
# python3
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
benchmarks/harness/machine_producers/gcloud_mock_recorder.py
-97
View File
@@ -1,97 +0,0 @@
# python3
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""A recorder and replay for testing the GCloudProducer.
MockPrinter and MockReader handle printing and reading mock data for the
purposes of testing. MockPrinter is passed to GCloudProducer objects. The user
can then run scenarios and record them for playback in tests later.
MockReader is passed to MockGcloudProducer objects and handles reading the
previously recorded mock data.
It is left to the user to check if data printed is properly redacted for their
own use. The intended usecase for this class is data coming from gcloud
commands, which will contain public IPs and other instance data.
The data format is json and printed/read from the ./test_data directory. The
data is the output of subprocess.CompletedProcess objects in json format.
Typical usage example:
recorder = MockPrinter()
producer = GCloudProducer(args, recorder)
machines = producer.get_machines(1)
with open("my_file.json") as fd:
recorder.write_out(fd)
reader = MockReader(filename)
producer = MockGcloudProducer(args, mock)
machines = producer.get_machines(1)
assert len(machines) == 1
"""
import io
import json
import subprocess
class MockPrinter(object):
"""Handles printing Mock data for MockGcloudProducer.
Attributes:
_records: list of json object records for printing
"""
def __init__(self):
self._records = []
def record(self, entry: subprocess.CompletedProcess):
"""Records data and strips out ip addresses."""
record = {
"args": entry.args,
"stdout": entry.stdout.decode("utf-8"),
"returncode": str(entry.returncode)
}
self._records.append(record)
def write_out(self, fd: io.FileIO):
"""Prints out the data into the given filepath."""
fd.write(json.dumps(self._records, indent=4))
class MockReader(object):
"""Handles reading Mock data for MockGcloudProducer.
Attributes:
_records: List[json] records read from the passed in file.
"""
def __init__(self, filepath: str):
with open(filepath, "rb") as file:
self._records = json.loads(file.read())
self._i = 0
def __iter__(self):
return self
def __next__(self, args) -> subprocess.CompletedProcess:
"""Returns the next record as a CompletedProcess."""
if self._i < len(self._records):
record = self._records[self._i]
stdout = record["stdout"].encode("ascii")
returncode = int(record["returncode"])
return subprocess.CompletedProcess(
args=args, returncode=returncode, stdout=stdout, stderr=b"")
raise StopIteration()
benchmarks/harness/machine_producers/gcloud_producer.py
-250
View File
@@ -1,250 +0,0 @@
# python3
# Copyright 2019 The gVisor Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""A machine producer which produces machine objects using `gcloud`.
Machine producers produce valid harness.Machine objects which are backed by
real machines. This producer produces those machines on the given user's GCP
account using the `gcloud` tool.
GCloudProducer creates instances on the given GCP account named like:
`machine-XXXXXXX-XXXX-XXXX-XXXXXXXXXXXX` in a randomized fashion such that name
collisions with user instances shouldn't happen.
Typical usage example:
producer = GCloudProducer(args)
machines = producer.get_machines(NUM_MACHINES)
# run stuff on machines with machines[i].run(CMD)
producer.release_machines(NUM_MACHINES)
"""
import datetime
import json
import subprocess
import threading
from typing import List, Dict, Any
import uuid
from benchmarks.harness import machine
from benchmarks.harness.machine_producers import gcloud_mock_recorder
from benchmarks.harness.machine_producers import machine_producer
class GCloudProducer(machine_producer.MachineProducer):
"""Implementation of MachineProducer backed by GCP.
Produces Machine objects backed by GCP instances.
Attributes:
image: image name as a string.
zone: string to a valid GCP zone.
machine_type: type of GCP to create (e.g. n1-standard-4).
installers: list of installers post-boot.
ssh_key_file: path to a valid ssh private key. See README on vaild ssh keys.
ssh_user: string of user name for ssh_key
ssh_password: string of password for ssh key
internal: if true, use internal IPs of instances. Used if bm-tools is
running on a GCP vm when a firewall is set for external IPs.
mock: a mock printer which will print mock data if required. Mock data is
recorded output from subprocess calls (returncode, stdout, args).
condition: mutex for this class around machine creation and deleteion.
"""
def __init__(self,
image: str,
zone: str,
machine_type: str,
installers: List[str],
ssh_key_file: str,
ssh_user: str,
ssh_password: str,
internal: bool,
mock: gcloud_mock_recorder.MockPrinter = None):
self.image = image
self.zone = zone
self.machine_type = machine_type
self.installers = installers
self.ssh_key_file = ssh_key_file
self.ssh_user = ssh_user
self.ssh_password = ssh_password
self.internal = internal
self.mock = mock
self.condition = threading.Condition()
def get_machines(self, num_machines: int) -> List[machine.Machine]:
"""Returns requested number of machines backed by GCP instances."""
if num_machines <= 0:
raise ValueError(
"Cannot ask for {num} machines!".format(num=num_machines))
with self.condition:
names = self._get_unique_names(num_machines)
instances = self._build_instances(names)
self._add_ssh_key_to_instances(names)
machines = self._machines_from_instances(instances)
# Install all bits in lock-step.
#
# This will perform paralell installations for however many machines we
# have, but it's easy to track errors because if installing (a, b, c), we
# won't install "c" until "b" is installed on all machines.
for installer in self.installers:
threads = [None] * len(machines)
results = [False] * len(machines)
for i in range(len(machines)):
threads[i] = threading.Thread(
target=machines[i].install, args=(installer, results, i))
threads[i].start()
for thread in threads:
thread.join()
for result in results:
if not result:
raise NotImplementedError(
"Installers failed on at least one machine!")
# Add this user to each machine's docker group.
for m in machines:
m.run("sudo setfacl -m user:$USER:rw /var/run/docker.sock")
return machines
def release_machines(self, machine_list: List[machine.Machine]):
"""Releases the requested number of machines, deleting the instances."""
if not machine_list:
return
cmd = "gcloud compute instances delete --quiet".split(" ")
names = [str(m) for m in machine_list]
cmd.extend(names)
cmd.append("--zone={zone}".format(zone=self.zone))
self._run_command(cmd, detach=True)
def _machines_from_instances(
self, instances: List[Dict[str, Any]]) -> List[machine.Machine]:
"""Creates Machine Objects from json data describing created instances."""
machines = []
for instance in instances:
name = instance["name"]
external = instance["networkInterfaces"][0]["accessConfigs"][0]["natIP"]
internal = instance["networkInterfaces"][0]["networkIP"]
kwargs = {
"hostname": internal if self.internal else external,
"key_path": self.ssh_key_file,
"username": self.ssh_user,
"key_password": self.ssh_password
}
machines.append(machine.RemoteMachine(name=name, **kwargs))
return machines
def _get_unique_names(self, num_names) -> List[str]:
"""Returns num_names unique names based on data from the GCP project."""
return ["machine-" + str(uuid.uuid4()) for _ in range(0, num_names)]
def _build_instances(self, names: List[str]) -> List[Dict[str, Any]]:
"""Creates instances using gcloud command.
Runs the command `gcloud compute instances create` and returns json data
on created instances on success. Creates len(names) instances, one for each
name.
Args:
names: list of names of instances to create.
Returns:
List of json data describing created machines.
"""
if not names:
raise ValueError(
"_build_instances cannot create instances without names.")
cmd = "gcloud compute instances create".split(" ")
cmd.extend(names)
cmd.append("--image=" + self.image)
cmd.append("--zone=" + self.zone)
cmd.append("--machine-type=" + self.machine_type)
res = self._run_command(cmd)
data = res.stdout
data = str(data, "utf-8") if isinstance(data, (bytes, bytearray)) else data
return json.loads(data)
def _add_ssh_key_to_instances(self, names: List[str]) -> None:
"""Adds ssh key to instances by calling gcloud ssh command.
Runs the command `gcloud compute ssh instance_name` on list of images by
name. Tries to ssh into given instance.
Args:
names: list of machine names to which to add the ssh-key
self.ssh_key_file.
Raises:
subprocess.CalledProcessError: when underlying subprocess call returns an
error other than 255 (Connection closed by remote host).
TimeoutError: when 3 unsuccessful tries to ssh into the host return 255.
"""
for name in names:
cmd = "gcloud compute ssh {user}@{name}".format(
user=self.ssh_user, name=name).split(" ")
if self.internal:
cmd.append("--internal-ip")
cmd.append("--ssh-key-file={key}".format(key=self.ssh_key_file))
cmd.append("--zone={zone}".format(zone=self.zone))
cmd.append("--command=uname")
timeout = datetime.timedelta(seconds=5 * 60)
start = datetime.datetime.now()
while datetime.datetime.now() <= timeout + start:
try:
self._run_command(cmd)
break
except subprocess.CalledProcessError:
if datetime.datetime.now() > timeout + start:
raise TimeoutError(
"Could not SSH into instance after 5 min: {name}".format(
name=name))
def _run_command(self,
cmd: List[str],
detach: bool = False) -> [None, subprocess.CompletedProcess]:
"""Runs command as a subprocess.
Runs command as subprocess and returns the result.
If this has a mock recorder, use the record method to record the subprocess
call.
Args:
cmd: command to be run as a list of strings.
detach: if True, run the child process and don't wait for it to return.
Returns:
Completed process object to be parsed by caller or None if detach=True.
Raises:
CalledProcessError: if subprocess.run returns an error.
"""
cmd = cmd + ["--format=json"]
if detach:
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if self.mock:
out, _ = p.communicate()
self.mock.record(
subprocess.CompletedProcess(
returncode=p.returncode, stdout=out, args=p.args))
return
res = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if self.mock:
self.mock.record(res)
if res.returncode != 0:
raise subprocess.CalledProcessError(
cmd=" ".join(res.args),
output=res.stdout,
stderr=res.stderr,
returncode=res.returncode)
return res
benchmarks/harness/machine_producers/gcloud_producer_test.py
-48
View File
@@ -1,48 +0,0 @@
# python3
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tests GCloudProducer using mock data.
GCloudProducer produces machines using 'get_machines' and 'release_machines'
methods. The tests check recorded data (jsonified subprocess.CompletedProcess
objects) of the producer producing one and five machines.
"""
import os
import types
from benchmarks.harness.machine_producers import machine_producer
from benchmarks.harness.machine_producers import mock_producer
TEST_DIR = os.path.dirname(__file__)
def run_get_release(producer: machine_producer.MachineProducer,
num_machines: int,
validator: types.FunctionType = None):
machines = producer.get_machines(num_machines)
assert len(machines) == num_machines
if validator:
validator(machines=machines, cmd="uname -a", workload=None)
producer.release_machines(machines)
def test_run_one():
mock = mock_producer.MockReader(TEST_DIR + "get_one.json")
producer = mock_producer.MockGCloudProducer(mock)
run_get_release(producer, 1)
def test_run_five():
mock = mock_producer.MockReader(TEST_DIR + "get_five.json")
producer = mock_producer.MockGCloudProducer(mock)
run_get_release(producer, 5)
benchmarks/harness/machine_producers/machine_producer.py
-51
View File
@@ -1,51 +0,0 @@
# python3
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Abstract types."""
import threading
from typing import List
from benchmarks.harness import machine
class MachineProducer:
"""Abstract Machine producer."""
def get_machines(self, num_machines: int) -> List[machine.Machine]:
"""Returns the requested number of machines."""
raise NotImplementedError
def release_machines(self, machine_list: List[machine.Machine]):
"""Releases the given set of machines."""
raise NotImplementedError
class LocalMachineProducer(MachineProducer):
"""Produces Local Machines."""
def __init__(self, limit: int):
self.limit_sem = threading.Semaphore(value=limit)
def get_machines(self, num_machines: int) -> List[machine.Machine]:
"""Returns the request number of MockMachines."""
self.limit_sem.acquire()
return [machine.LocalMachine("local") for _ in range(num_machines)]
def release_machines(self, machine_list: List[machine.MockMachine]):
"""No-op."""
if not machine_list:
raise ValueError("Cannot release an empty list!")
self.limit_sem.release()
machine_list.clear()
benchmarks/harness/machine_producers/mock_producer.py
-52
View File
@@ -1,52 +0,0 @@
# python3
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Producers of mocks."""
from typing import List, Any
from benchmarks.harness import machine
from benchmarks.harness.machine_producers import gcloud_mock_recorder
from benchmarks.harness.machine_producers import gcloud_producer
from benchmarks.harness.machine_producers import machine_producer
class MockMachineProducer(machine_producer.MachineProducer):
"""Produces MockMachine objects."""
def get_machines(self, num_machines: int) -> List[machine.MockMachine]:
"""Returns the request number of MockMachines."""
return [machine.MockMachine() for i in range(num_machines)]
def release_machines(self, machine_list: List[machine.MockMachine]):
"""No-op."""
return
class MockGCloudProducer(gcloud_producer.GCloudProducer):
"""Mocks GCloudProducer for testing purposes."""
def __init__(self, mock: gcloud_mock_recorder.MockReader, **kwargs):
gcloud_producer.GCloudProducer.__init__(
self, project="mock", ssh_private_key_path="mock", **kwargs)
self.mock = mock
def _validate_ssh_file(self):
pass
def _run_command(self, cmd):
return self.mock.pop(cmd)
def _machines_from_instances(
self, instances: List[Any]) -> List[machine.MockMachine]:
return [machine.MockMachine() for _ in instances]

Some files were not shown because too many files have changed in this diff Show More