BuildKite: Make post-command more robust against stuck containers.

Some BuildKite builds have been getting stuck on this step.
Interestingly, this was not on the first container that they try to stop,
implying that the problem is container-specific, rather than the Docker
daemon being unresponsive as a whole.

This change removes the dependency on `docker inspect` and adds a timeout
to the container kill step. Will it be enough? Not sure.

#codehealth

PiperOrigin-RevId: 501132910
This commit is contained in:
Etienne Perot
2023-01-10 17:17:25 -08:00
committed by gVisor bot
parent 44e0246e29
commit 9ba9d78ca9
+5 -4
View File
@@ -71,11 +71,12 @@ if test "${BUILDKITE_COMMAND_EXIT_STATUS}" -ne "0"; then
fi
# Kill any running containers (clear state), except for "bootstrap".
for container in $(docker ps -q); do
maybe_kill="$(docker inspect -f '{{if ne "/bootstrap" .Name}}true{{ end }}' "${container}")"
if test -n "${maybe_kill}"; then
docker container kill "${container}"
for id_and_name in $(docker ps --format='{{.ID}} {{.Names}}'); do
if [[ "$(echo "$id_and_name" | cut -d' ' -f2-)" == 'bootstrap' ]]; then
continue
fi
timeout --kill-after=10s --preserve-status 8s \
docker container kill "$(echo "$id_and_name" | cut -d' ' -f1)"
done
set -euo pipefail