[Bug]: Two NC instances on one server interfere with each other - disabling apcu helps

[aadlung]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• Nextcloud Server is running on 64bit capable CPU, PHP and OS.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

I updated from nextcloud 25 to 26, and I have 2 instances of nextcloud running on the same server with differnet domains, databases and data folders. The open_basedir and permission structure does not allow them to see/read each other.

Nevertheless, after upgrading I could see errors in nextcloud.log that one instance tries to access files of the oter, and after restarting php-fpm and accessing the second instance, it tried to access files of the other as well - only the first instance I opened kept working, the second showed a HTTP500 and the errors in log with file access.

I have "memcache.local": "\\OC\\Memcache\\Redis", enabled with a Redis configuration using a separate dbindex for each of the instances.

Only after disabling apcu it worked. Even if the memcache config should use Redis with its own database.

Steps to reproduce

1. Have 2 instances on the same system
2. Upgrade from NC 25 to NC 26
3. Enable apcu

Expected behavior

Two instances should not interfere with each other, even with apcu they should use an apcuPrefix or similar. I have no idea if this is related to the upgrade or anything else - but that's the only action I executed.

Installation method

Community Manual installation with Archive

Nextcloud Server version

26

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Nginx

Database engine version

MySQL

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 22.2.3 to 23.0.1)
--> Also tested with a FRESH install, same behavior after adding memcached+redis config with active apcu module

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "simpleSignUpLink.shown": false,
        "trusted_domains": [
            "files.example.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/files.example.com",
        "dbtype": "mysql",
        "version": "26.0.0.11",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "theme": "",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "dbindex": 2,
            "port": 6379
        },
        "updater.release.channel": "stable",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "loglevel": 2,
        "default_phone_region": "AT",
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - activity: 2.18.0
  - circles: 26.0.0
  - cloud_federation_api: 1.9.0
  - comments: 1.16.0
  - contactsinteraction: 1.7.0
  - dav: 1.25.0
  - federatedfilesharing: 1.16.0
  - federation: 1.16.0
  - files: 1.21.1
  - files_pdfviewer: 2.7.0
  - files_rightclick: 1.5.0
  - files_sharing: 1.18.0
  - files_trashbin: 1.16.0
  - files_versions: 1.19.1
  - firstrunwizard: 2.15.0
  - logreader: 2.11.0
  - lookup_server_connector: 1.14.0
  - nextcloud_announcements: 1.15.0
  - notifications: 2.14.0
  - oauth2: 1.14.0
  - password_policy: 1.16.0
  - photos: 2.2.0
  - privacy: 1.10.0
  - provisioning_api: 1.16.0
  - recommendations: 1.5.0
  - related_resources: 1.1.0-alpha1
  - serverinfo: 1.16.0
  - settings: 1.8.0
  - sharebymail: 1.16.0
  - support: 1.9.0
  - survey_client: 1.14.0
  - systemtags: 1.16.0
  - text: 3.7.2
  - theming: 2.1.1
  - twofactor_backupcodes: 1.15.0
  - updatenotification: 1.16.0
  - user_status: 1.6.0
  - viewer: 1.10.0
  - weather_status: 1.6.0
  - workflowengine: 2.8.0
Disabled:
  - admin_audit: 1.16.0
  - bruteforcesettings: 2.6.0
  - dashboard: 7.6.0 (installed 7.0.0)
  - encryption: 2.14.0
  - files_external: 1.18.0
  - suspicious_login: 4.4.0
  - twofactor_totp: 8.0.0-alpha.0
  - user_ldap: 1.16.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Tested with a FRESH install. It worked, until I added the memcached config with Redis

'memcache.local' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'localhost',
    'dbindex' => 2,
    'port' => 6379,
  ),

{"reqId":"dqf7hKO1FtScvE769qZz","level":3,"time":"2023-04-06T08:20:11+00:00","remoteAddr":"127.0.0.1","user":"webapp2","app":"PHP","method":"GET","url":"/apps/files/?dir=/&fileid=2","message":"include(): open_basedir restriction in effect. File(/var/www/webapp1/apps/firstrunwizard/lib//AppInfo/Application.php) is not within the allowed path(s): /var/www/webapp2/:/usr/lib/php/:/var/lib/php/sessions/:/usr/share/GeoIP/:/tmp/:/opt/nextcloud-data-webapp2) at /var/www/webapp2/lib/composer/composer/ClassLoader.php#578","userAgent":"Mozilla/5.0 (X11; Linux aarch64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36","version":"26.0.0.11","data":{"app":"PHP"}}

Additional info

No response

[toxpal]

This is unbelievable!

After reading your post, I thought "this is crazy, how is it even possible for one NC instance to access files from other instance?" and now I have the same problem. If the same server runs two or more NC26 installations, only one of them works. What's even more crazy is that each installation contains tons of errors about OTHER installation files (for example, if 1st instance is running at /home/user1 and other instance is running in /home/user2, 1st instance nextcloud.log file contains errors with /home/user2 path). And some of instances stop working randomly, resulting in 500 internal error.

I used NC for years and never had the same problem. WTF??

[aadlung]

I assume that just the cache is not using a corresponding prefix or similar - with Redis you can provide a database at least but for caching there should be ideally at least the instance ID or similar which is unique.

[toxpal]

I don't have Redis installed, I use apcu, and never had such problem. Only after upgrading to NC26 it started returning 500 internal server error, and it happens because each installation somehow tries to access files from other installation - https://help.nextcloud.com/t/nc-25-0-5-to-nc-26-internal-server-error/159632/6

[markushagge]

I can confirm the problem. Here also some Nextcloud installations after update to Nextcloud26 suddenly with error 500 no longer accessible, in the nextcloud log are then various error messages regarding include errors with paths to completely different installations.

Disabling APCu then brought relief.

Doesn't anyone at Nextcloud test new releases on a system running multiple Nextcloud instances?

[Bewo]

I can confirm the problem as well. I manage several servers, both with Nextcloud and with other online products. All instances always cleanly separated in their own PHP sockets, etc. I have never seen a problem of this kind. I haven't bothered to analyze the cache entries yet, but I can confirm that disabling APCu works.

I now use Redis for the local cache as well. I haven't done a benchmark yet, but imho there is no noticeable performance degradation. => So, that's a workaround / solution for me.

[szaimen]

Looks like krakjoe/apcu#238

[Bewo]

Looks like krakjoe/apcu#238

I can not confirm this, but i don't think so.
It worked fine for years before NC 26. I had separate apcu-caches for each instance using apcu.

[markushagge]

Here the same. PHP-FPM is used, but it has worked for several years with various versions of Nextcloud and APCu without problems

Same here after update from 25 to 26.
Two webhosted Nextclouds on the same server.
Using different PHP versions (8.1 and 8.2) solves the problem for me.

Support says: The problem is probably that APCu builds up a cache in the working memory which cannot be identified correctly. As a result, one Nextcloud tries to access data from the other.

[toxpal]

Support says: The problem is probably that APCu builds up a cache in the working memory which cannot be identified correctly

How it comes everything works fine on NC25 and problem happens only after upgrading NC, not apcu?

[eehmke]

I have the same problem. I run a test instance and a work instance on the same server. Worked fine for a long time. Since the upgrade to 26, even to 26.0.1, I get this errors of the test instance trying to access files of the work instance.

I even deleted the test instance, deleted the database, deleted the data directory, reinstalled the test instance using the web installer, and the same problem pops up again. In the test instance there is no cache activated, no apcu, no redis. Just the naked config.php from the installer.

The test instance has its own php-fpm instance too.

The work instance works fine so far.

[blizzz]

A prefix is used across all caches, and it's a hash of concatenated values of the instanceId, the NC version and the webroot of the instance. But it does not matter for APCu when you have not have it configured. APCu is then not used by Nextcloud itself.

While I did not observe strange behaviour (might be luck) i could find that composer was caching autoloaded classes without a path. We never configured composer to act like this however.

If you like to doublecheck on your instance, make this a script like this available via web (ideally not public; CLI won't work);

<?php
print("<pre>");
print_r(apcu_fetch('composer_autoloadOCA\Activity\AppInfo\Application'));
print("<br/>");
print_r(apcu_fetch('composer_autoloadOCA\GroupFolders\Listeners\LoadAdditionalScriptsListener'));
print("<p></p>");
print_r(apcu_cache_info());

Replace the keys in apcu_fetch commands with ones you see in the output below from apcu_cache_info().