BUG REPORT INFORMATION - Docker CE 1.13.1 issue while changing kernel semaphore changes(kernel.sem)

[jram-nwed]

---

BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:
You do NOT have to include this information if this is a FEATURE REQUEST
--> docker run --name webserver1 -d -p 9091:80 --sysctl kernel.sem="250 32000 100 2048" nginx

Description

Steps to reproduce the issue:

1. Make sure u are on docker version 1.13.1 on RHEL-7.5/RHEL-7.4
2. Change the host kernel settings : sysctl -w kernel.sem="250 32000 100 2048"
3. Check weather the changes are reflected on host cat /proc/sys/kernel/sem
4. Run docker run --name webserver1 -d -p 9091:80 --sysctl kernel.sem="250 32000 100 2048" nginx
   if successful:
5. docker exec -it webserver1 bash
6. cat /proc/sys/kernel/sem - to cross check weather the changes are reflected inside the container

** Output of error **
aa71efee7bf149794a11fb27eab1a25640c6cc3c09192f610d5b14cafe186b26
/usr/bin/docker-latest: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:364: container init caused "open /proc/sys/kernel/sem: permission denied"".

Describe the results you received:
Docker service was unable to pick the desired kernel changes on host.

Describe the results you expected:
the kernel semaphore changes are supposed to be reflected inside the container.

Additional information you deem important (e.g. issue happens only occasionally):
It perfectly works fine with docker 1.12.x and docker 18.03.

Output of docker version:

Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-latest-1.13.1-58.git87f2fab.el7.x86_64
 Go version:      go1.9.2
 Git commit:      85fc86f-unsupported
 Built:           Mon Mar 19 14:51:16 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-latest-1.13.1-58.git87f2fab.el7.x86_64
 Go version:      go1.9.2
 Git commit:      85fc86f-unsupported
 Built:           Mon Mar 19 14:51:16 2018
 OS/Arch:         linux/amd64
 Experimental:    false

Output of docker info:

Containers: 41
 Running: 4
 Paused: 0
 Stopped: 37
Images: 36
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: false
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc docker-runc-debug runc
Default Runtime: docker-runc-debug
Init Binary: docker-init
containerd version: 85fc86fe510461b1920d35e9819c792b1ccc054d (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: e9c345b3f906d5dc5e8100b05ce37073a811c74a (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  Profile: default
 selinux
 userns
Kernel Version: 3.10.0-862.3.2.el7.x86_64
Operating System: Red Hat Enterprise Linux
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 4
Total Memory: 15.51 GiB
Name:XXXXXXXXXXXXXX
ID: Q5L7:E6VG:MSG2:QKKU:FZEO:QTCV:G4IY:2GX5:5FHI:LIG2:B7OW:6BRK
Docker Root Dir: /app/docker-latest/100000.100000
Debug Mode (client): false
Debug Mode (server): false
Registry: https://registry.access.redhat.com/v1/
Experimental: false
Insecure Registries:
 XXXXXXXXXXXX
Live Restore Enabled: false
Registries: registry.access.redhat.com (secure), docker.io (secure)

Additional environment details (AWS, VirtualBox, physical, etc.):
AWS- EC2 instance
OS -RHEL-7.5

[rhatdan]

Did you get any AVC messages?

[jram-nwed]

No, SElinux isn't the factor.

error:
/usr/bin/docker-latest: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:364: container init caused "open /proc/sys/kernel/sem: permission denied"".

[rhatdan]

I take it SELinux is disabled? Could be capabitliies? Do you know if the kernel/sem is a namespaced sysctl?

[rhatdan]

Does it work if you set

--cap-add ALL
If yes then try
--cap-add SYS_ADMIN

[rhatdan]

BTW Have you tried podman...

Worked successfully with podman on Fedora 28.

# podman  run --name webserver1 -d -p 9091:80 --sysctl kernel.sem="250 32000 100 2048" nginx
Trying to pull quay.io/nginx:latest...Failed
Trying to pull docker.io/nginx:latest...Getting image source signatures
Copying blob sha256:f2aa67a397c49232112953088506d02074a1fe577f65dc2052f158a3e5da52e8
 21.45 MB / 21.45 MB [======================================================] 1s
Copying blob sha256:1cd0975d4f456a5ab3196ae351a2a9d72077ee24da60b42004eb0c3e8d153aad
 21.10 MB / 21.10 MB [======================================================] 1s
Copying blob sha256:72fd2d3be09a6d5bb4eee2e6c66e6929f9f9c71853dd250035a9bdb37fccce89
 205 B / 205 B [============================================================] 0s
Copying config sha256:cd5239a0906a6ccf0562354852fae04bc5b52d72a2aff9a871ddb6bd57553569
 5.88 KB / 5.88 KB [========================================================] 0s
Writing manifest to image destination
Storing signatures
2c68a75d7318421f5329bddf792a22d522f7481fd0754acd7e92da218c8a8340
# podman exec -it webserver1 bash
root@2c68a75d7318:/# cat /proc/sys/kernel/sem
250	32000	100	2048

[icqworld]

selinux is enabled.
username is enabled.
Other kernel settings can be changed.

We have only noticed this param kernel.sem not changed.
We tried adding sysctl.conf inside container and that was not able to update the kernel setting.
Seems like bug specific to this kernel setting and docker 1.13.1-rhel

[jram-nwed]

@rhatdan It works fine when I disable user namespaces(--userns=host).

[rhatdan]

Ah a clue. So usernamespace is blocking it.
I wonder if this is a bug in usernamespace or in runc?
@giuseppe could you try this out with usernamespace enabled.

Quick tests with podman shows that it is failing.

# ./bin/podman rm webserver4; podman run --uidmap "0:90000:1000" --name webserver4 -d -p 9091:80 --net=host nginx
69acc32b31089e615fbbe24d085e1a7a65008836630e7b3e9197988ee5a2bbd4
3abf029d5ad68fbfe71bca2f7565d8bd43e460281a3c4f90b420b24105c5f296
# ./bin/podman rm webserver4; podman run --uidmap "0:90000:1000" --name webserver4 -d -p 9091:80 --sysctl kernel.sem="250 32000 100 2048" --net=host nginx
3abf029d5ad68fbfe71bca2f7565d8bd43e460281a3c4f90b420b24105c5f296
container "4d436239d5061958ca3d14e8c86df5b6f67c58ec6c00d85d95131f609ad23d95" does not exist
`/usr/bin/runc start 4d436239d5061958ca3d14e8c86df5b6f67c58ec6c00d85d95131f609ad23d95` failed: exit status 1

[rhatdan]

Without usernamespace:

 ./bin/podman rm webserver4; podman run --name webserver4 -d -p 9091:80 --sysctl kernel.sem="250 32000 100 2048" --net=host nginx
4d436239d5061958ca3d14e8c86df5b6f67c58ec6c00d85d95131f609ad23d95
5aa16e1946ac626028d317c65a23c1c0b978bea5fd61a977b08c85e389a1d444

[rhatdan]

@ebeiderman Do you think this is a bug in UserNamespace or in Runc?

[giuseppe]

@rhatdan I think that is expected. Writing to /proc/sys/* is done from the container process itself during the setup and AFAIK only root in the init user namespace has write access to /proc/sys/* that is not userNS aware.

[rhatdan]

Can we change the writing to the sysctl to not be done by the the root in the init user namespace? IE Can this be fixed in the OCI RUntimes?

[giuseppe]

I am not sure, it might be possible that the parent process writes to the /proc/fs mounted by the container (as a hook would do) but it will be half solution as it won't work with rootless containers.

Tagging @ebiederm