[Bug]: NC 25.0.x - authentication - a need to recalculate credentials after secret is updated from empty to non-empty value

[piknew]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• Nextcloud Server is running on 64bit capable CPU, PHP and OS.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

According to reported issue #35347 my installation didn't have "secret" defined. Once issue has been reported it was arbitrarily marked as "resolved by #35368". Nobody has noticed that I have commented that if I add secret to my config (add manually) then authentication will not work with LDAP users:

In the log (see below) - there is an exception reported with HMAC does not match.
Remark: the same issue was for 24.0.7 - when I added secret manually - LDAP authentication was not working.

So, this issue is about it. Secret is already defined but I am not able to login by LDAP users. Definitely the problem is with the secret itself as when I remove it and additionally apply code fix (procedure in OC_Util to check existence of secret) - everything is working.

Of course for now I will apply my "code workaround" (see below) again and remove secret from config:

Steps to reproduce

1. Add secret to config (manually or automatically by upgrade procedure)
2. Try to login by LDAP user
3. Login will fail

Expected behavior

Login for LDAP users with "secret" defined shall working correctly - the same as without secret.

Installation method

Community Web installer on a VPS or web space

Operating system

Debian/Ubuntu

PHP engine version

PHP 7.4

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 22.2.3 to 23.0.1)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "25.0.2.0",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "installed": true,
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory",
        "log_type": "syslog",
        "logfile": "\/share\/spool\/nextcloud\/data\/nextcloud.log",
        "loglevel": 2,
        "cron_log": true,
        "syslog_tag": "nextcloud",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "maintenance": false,
        "theme": "",
        "updater.release.channel": "beta",
        "mysql.utf8mb4": true,
        "mail_sendmailmode": "smtp",
        "defaultapp": "files",
        "default_phone_region": "pl",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "htaccess.RewriteBase": "\/",
        "updater.secret": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - activity: 2.17.0
  - admin_audit: 1.15.0
  - circles: 25.0.0
  - cloud_federation_api: 1.8.0
  - comments: 1.15.0
  - contactsinteraction: 1.6.0
  - dashboard: 7.5.0
  - dav: 1.24.0
  - drawio: 1.0.5
  - federatedfilesharing: 1.15.0
  - federation: 1.15.0
  - files: 1.20.1
  - files_external: 1.17.0
  - files_pdfviewer: 2.6.0
  - files_rightclick: 1.4.0
  - files_sharing: 1.17.0
  - files_trashbin: 1.15.0
  - files_versions: 1.18.0
  - firstrunwizard: 2.14.0
  - groupfolders: 13.1.0
  - logreader: 2.10.0
  - lookup_server_connector: 1.13.0
  - maps: 0.2.1
  - metadata: 0.17.0
  - nextcloud_announcements: 1.14.0
  - notifications: 2.13.1
  - oauth2: 1.13.0
  - password_policy: 1.15.0
  - photos: 2.0.1
  - previewgenerator: 5.1.1
  - privacy: 1.9.0
  - provisioning_api: 1.15.0
  - recommendations: 1.4.0
  - related_resources: 1.0.3
  - serverinfo: 1.15.0
  - settings: 1.7.0
  - sharebymail: 1.15.0
  - support: 1.8.0
  - survey_client: 1.13.0
  - systemtags: 1.15.0
  - text: 3.6.0
  - theming: 2.0.1
  - twofactor_backupcodes: 1.14.0
  - updatenotification: 1.15.0
  - user_ldap: 1.15.0
  - user_status: 1.5.0
  - viewer: 1.9.0
  - weather_status: 1.5.0
  - workflowengine: 2.7.0
Disabled:
  - bruteforcesettings: 1.0.2
  - encryption: 2.9.0
  - suspicious_login
  - twofactor_totp

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Nov 26 11:56:56 PKSERVER nextcloud[236053]: {"reqId":"Y4Hw91ny8BdJHSjE_0X4oAAAAAs","level":3,"time":"2022-11-26T10:56:56+00:00","remoteAddr":"192.168.10.190","user":"73d281ca-8d75-1038-9188-13c3b0f6c6ce","app":"index","method":"POST","url":"/login","message":"{\"Exception\":\"Exception\",\"Message\":\"HMAC does not match.\",\"Code\":0,\"Trace\":[{\"file\":\"/share/data/www/nextcloud/lib/private/Security/CredentialsManager.php\",\"line\":104,\"function\":\"decrypt\",\"class\":\"OC\\\\Security\\\\Crypto\",\"type\":\"->\",\"args\":[\"*** sensitive parameters replaced ***\"]},{\"file\":\"/share/data/www/nextcloud/apps/files_external/lib/Listener/StorePasswordListener.php\",\"line\":53,\"function\":\"retrieve\",\"class\":\"OC\\\\Security\\\\CredentialsManager\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php\",\"line\":87,\"function\":\"handle\",\"class\":\"OCA\\\\Files_External\\\\Listener\\\\StorePasswordListener\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php\",\"line\":251,\"function\":\"__invoke\",\"class\":\"OC\\\\EventDispatcher\\\\ServiceEventListener\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php\",\"line\":73,\"function\":\"callListeners\",\"class\":\"Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php\",\"line\":88,\"function\":\"dispatch\",\"class\":\"Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php\",\"line\":100,\"function\":\"dispatch\",\"class\":\"OC\\\\EventDispatcher\\\\EventDispatcher\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Server.php\",\"line\":625,\"function\":\"dispatchTyped\",\"class\":\"OC\\\\EventDispatcher\\\\EventDispatcher\",\"type\":\"->\"},{\"function\":\"OC\\\\{closure}\",\"class\":\"OC\\\\Server\",\"type\":\"->\",\"args\":[\"*** sensitive parameters replaced ***\"]},{\"file\":\"/share/data/www/nextcloud/lib/private/Hooks/EmitterTrait.php\",\"line\":106,\"function\":\"call_user_func_array\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Hooks/PublicEmitter.php\",\"line\":40,\"function\":\"emit\",\"class\":\"OC\\\\Hooks\\\\BasicEmitter\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/User/Session.php\",\"line\":400,\"function\":\"emit\",\"class\":\"OC\\\\Hooks\\\\PublicEmitter\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Authentication/Login/CompleteLoginCommand.php\",\"line\":44,\"function\":\"completeLogin\",\"class\":\"OC\\\\User\\\\Session\",\"type\":\"->\",\"args\":[\"*** sensitive parameters replaced ***\"]},{\"file\":\"/share/data/www/nextcloud/lib/private/Authentication/Login/ALoginCommand.php\",\"line\":40,\"function\":\"process\",\"class\":\"OC\\\\Authentication\\\\Login\\\\CompleteLoginCommand\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Authentication/Login/LoggedInCheckCommand.php\",\"line\":60,\"function\":\"processNextOrFinishSuccessfully\",\"class\":\"OC\\\\Authentication\\\\Login\\\\ALoginCommand\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Authentication/Login/ALoginCommand.php\",\"line\":40,\"function\":\"process\",\"class\":\"OC\\\\Authentication\\\\Login\\\\LoggedInCheckCommand\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Authentication/Login/EmailLoginCommand.php\",\"line\":70,\"function\":\"processNextOrFinishSuccessfully\",\"class\":\"OC\\\\Authentication\\\\Login\\\\ALoginCommand\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Authentication/Login/ALoginCommand.php\",\"line\":40,\"function\":\"process\",\"class\":\"OC\\\\Authentication\\\\Login\\\\EmailLoginCommand\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Authentication/Login/UidLoginCommand.php\",\"line\":54,\"function\":\"processNextOrFinishSuccessfully\",\"class\":\"OC\\\\Authentication\\\\Login\\\\ALoginCommand\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Authentication/Login/ALoginCommand.php\",\"line\":40,\"function\":\"process\",\"class\":\"OC\\\\Authentication\\\\Login\\\\UidLoginCommand\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Authentication/Login/UserDisabledCheckCommand.php\",\"line\":58,\"function\":\"processNextOrFinishSuccessfully\",\"class\":\"OC\\\\Authentication\\\\Login\\\\ALoginCommand\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Authentication/Login/ALoginCommand.php\",\"line\":40,\"function\":\"process\",\"class\":\"OC\\\\Authentication\\\\Login\\\\UserDisabledCheckCommand\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Authentication/Login/PreLoginHookCommand.php\",\"line\":53,\"function\":\"processNextOrFinishSuccessfully\",\"class\":\"OC\\\\Authentication\\\\Login\\\\ALoginCommand\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Authentication/Login/Chain.php\",\"line\":108,\"function\":\"process\",\"class\":\"OC\\\\Authentication\\\\Login\\\\PreLoginHookCommand\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/core/Controller/LoginController.php\",\"line\":318,\"function\":\"process\",\"class\":\"OC\\\\Authentication\\\\Login\\\\Chain\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php\",\"line\":225,\"function\":\"tryLogin\",\"class\":\"OC\\\\Core\\\\Controller\\\\LoginController\",\"type\":\"->\",\"args\":[\"*** sensitive parameters replaced ***\"]},{\"file\":\"/share/data/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php\",\"line\":133,\"function\":\"executeController\",\"class\":\"OC\\\\AppFramework\\\\Http\\\\Dispatcher\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/AppFramework/App.php\",\"line\":172,\"function\":\"dispatch\",\"class\":\"OC\\\\AppFramework\\\\Http\\\\Dispatcher\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/lib/private/Route/Router.php\",\"line\":298,\"function\":\"main\",\"class\":\"OC\\\\AppFramework\\\\App\",\"type\":\"::\"},{\"file\":\"/share/data/www/nextcloud/lib/base.php\",\"line\":1047,\"function\":\"match\",\"class\":\"OC\\\\Route\\\\Router\",\"type\":\"->\"},{\"file\":\"/share/data/www/nextcloud/index.php\",\"line\":36,\"function\":\"handleRequest\",\"class\":\"OC\",\"type\":\"::\"}],\"File\":\"/share/data/www/nextcloud/lib/private/Security/Crypto.php\",\"Line\":156,\"CustomMessage\":\"--\"}","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:107.0) Gecko/20100101 Firefox/107.0","version":"25.0.2.0"}

Additional info

No response

[szaimen]

Cc @come-nc @blizzz @PVince81 is it possible that user_ldap uses an empty secret if none is configured?

[piknew]

Does it matter if I see exception to be thrown at files_external (retrieve function, see logs as I have pasted)? Maybe stored credentials (stored with "empty" secret) cannot be retrieved with new secret? In such a case - maybe some kind of fixup for data will be the solution (or such a solution shall be a part of job as implemented in #35368).

Please take a look at CredentialsManager.php and then at Crypto.php. It seems that definitely secret is used to calculate HMAC.

[piknew]

Edit: success - I have simply analyzed the code and it seems that credentials are refreshed, as for my settings:

So, I have deleted all records from oc_storages_credentials table. And... now login succeeds.

Current state (after login for 1st of LDAP user):

So, I strongly suggest to add "recalculation" of credentials for job added in #35368 (only in case secret was empty/not defined and upgrade is making non-empty value). This suggestion is really for all data that may have been previously stored with empty secret.

[blizzz]

CredentialsManager, cc @CarlSchwan

[piknew]

Shall the title of issue be changed? Currently it may not fully describe what the issue is (it was resulting that authentication was failed but the reason/root-cause was different).

[piknew]

Are you aware of any other places (besides listed here: "oc_storages_credentials") that such a recalculation may be required?

Proposal of new title "[Bug]: NC 25.0.x - authentication - a need to recalculate credentials after secret is updated from empty to non-empty value".