Official Docker image?

[DarrienG]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Issue

System information. For client/server mode post info for both machines.

N/A

Your borg version (borg -V).

N/A

Operating system (distribution) and version.

N/A

Hardware / network configuration, and filesystems used.

N/A

There are a couple of unofficial Docker images for borg, but there's interest in an official Docker image. Especially after the runc vulnerability that was reported the other day, an officially supported image would inspire more trust in it.

I know I could always build one of my own, but the world is moving towards containerized services, and it would likely drive up usage.

I don't mind making a Docker related PR if desired :)

[ThomasWaldmann]

I personally don't use docker.

For which of the supported platforms would you like to add docker images?
What exactly would such an image offer?

[DarrienG]

Docker runs on most platforms, so any OS is fine.

Most "single binary" images tend to be based on alpine linux, but Debian, Ubuntu, Fedora, or the like is a fine base image, I don't think anyone would care much.

We're thinking about using Borg at big company XYZ and we have a huge Kubernetes cluster. We would like to use Borg for automating backups and all that jazz.

The image would be used as a sidecar container to run alongside another container in our cluster, and back up any contents that are important.

Containers are supposed to be single purpose, single process, so installing a cron and the borg binary alongside say our Jenkins container isn't much of an option.

As I said before, we can make our own, but an image with official support that is updated as borg is updated, and changed as borg has different dependencies would be great.

[pierreozoux]

I have 2 comments here :) I came to open the same issue ;)

Yes, an official image would be amazing, I already created this one and I'd be happy to PR upstream if it is something you want to have.
(It does a bit more than just borg, but well.. we can figure that out during the PR)

We also have 2 docker-compose that goes with that:
https://github.com/libresh/compose-borg-server
https://github.com/libresh/compose-borg-client

@DarrienG for kubernetes you are actually looking at https://appscode.com/products/stash/ ;) I know it is based on restic, but it is so cool ;) (And I'd love to have that with borg, restic doesn't have compression)

[ThomasWaldmann]

Considering the linux fat binaries are built on debian, debian 64bit would be a good base. In master branch we use stretch currently. Have a look at the Vagrantfile "stretch64" machine.

If this shall have a chance to become part of release process, it must be fully automated.

Also, there is a unfinished ticket of specifying a standard client/server setup, so guess the docker images could be based on that. #1553

[DarrienG]

At work we use a modified version of docker-library, which is likely overkill.

It looks like the docker foundation has their own GitHub integration which will probably do this in just a few clicks.

Set it up to only build when a new tag is added, and you should be good to go.

[ThomasWaldmann]

I'ld like to build binaries on machines / OS I control, not in the "cloud".

[DarrienG]

If you're ok with Travis (which isn't on your machine, but it does look like your build system) you can build through that on a tag release: https://docs.travis-ci.com/user/docker/ and https://docs.travis-ci.com/user/deployment/releases/

Otherwise you'll want to add a webhook that fires a POST or whatever to a server you own.

Building the image will look like this:

# With an authenticated docker client, see https://docs.docker.com/v17.12/docker-cloud/builds/push-images/
LATEST="borg_collective/borg-backup:latest"
DOCKER_IMAGE="borg_collective/borg-backup:$GIT_TAG_FROM_WEBHOOK"

git clone https://github.com/borgbackup/borg
cd borgbackup
git checkout $GIT_TAG

docker build -t "$DOCKER_IMAGE" .
docker tag "$DOCKER_IMAGE" "$LATEST"

docker push "$DOCKER_IMAGE"
docker push "$LATEST"

[DarrienG]

FWIW - I'm not entirely certain how the release process works for borgbackup. If you on the other hand make tags on your local machine and then push the tag up, you could make a hook on your git client itself which will in turn run the above commands quietly in the background.

https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks

[pierreozoux]

Where is the pipeline to push to pip?
The easiest is really to let https://hub.docker.com/ do the build and host the docker image.

Do you have some infrastructure to host a docker registry and run a pipeline?

I do plan to build one here if you need. (there is no runner nor registry yet, but I'll work on that on the coming weeks).

[ThomasWaldmann]

The stuff needs to be build in the vagrant VM, see Vagrantfile "stretch64" (as I already said above).

No travis, no github, no AWS, ...

I'ld accept a PR that is tested and modifies the Vagrantfile (master branch) so it builds 2(?) docker images.

But as I don't use / don't need docker, I won't implement this myself.

[pierreozoux]

Ok, I understand, I took a look at the vagrant, it will be easy.
For the 2 docker image, do you mean:

• a simple one with just Borg bin
• a complexe one with my client/server code to implement create specs for backup service? #1553

This would be cool indeed and cover most of what people need :)

I'll modify my PR later with that.

@DarrienG you'll have to help me maintainer that, if we share the work, it should be easier :)

@ThomasWaldmann I have more questions:

How do you push to pip? After reading the vagrant file, it was not clear to me.
In the Dockerfile, should I use the local bin, or the pip version (like in the PR) (in the second case, I have to make sure the build happens after the pip push).
Is it ok for you to host the docker image on hub.docker.com or would you prefer your own registry?
In the complexe Docker image, is it ok to ship Prometheus metrics like I currently do, or should I separate this logic? (I think separating would make more sense, but well, up to you :) )

Thanks for your time!

[ThomasWaldmann]

I thought you wanted to make one for the client and one for the server. But just do what makes sense. :)

The source dist (sdist, then uploaded to pypi after signing) currently comes from my development machine, the fat binaries are built and tested in the (self-installed, if possible) vagrant VMs you see in the Vagrantfile and fetched from there to dev machine via vagrant scp. I'ld fetch the docker image from there in the same way.

Inside the docker image, I guess working from source (via the sdist, python setup.py sdist, no need to wait for pypi or downloading it from there) is better than using the fat binary (which is a rather huge wrapper, but that is not needed as that is solved by docker, AFAIK).

I don't know any docker stuff, but an obvious requirement is that users can verify authenticity of any downloaded file via a gpg signature before running it as root and letting it access all their files. Another requirement is not "polluting" the vagrant machines with low-trust stuff (e.g. binary downloads from 3rd party sources, not debian).

Prometheus stuff sounds rather specific, so I guess without makes more sense for broad use.

[DarrienG]

@pierreozoux Happy to help get this going and keep it up to snuff. We're using a company image in some alpha services right now and would definitely like official support. Since you already have a Dockerfile and all that, feel free to add me to the review you make.

I personally don't think Prometheus metrics should be P1/built in.