Official build uptake failure: "Unable to find an entry point named 'AppleCryptoNative_SslCreateContext'", "CryptoNative_Tls13Supported", during dotnet restore

[dagood]

dotnet/toolset#3884 has errors like this trying to update to the dotnet/runtime official build:

Unable to find an entry point named 'AppleCryptoNative_SslCreateContext' in shared library 'System.Security.Cryptography.Native.Apple'.

Details

macOS https://dev.azure.com/dnceng/public/_build/results?buildId=464720&view=ms.vss-test-web.build-test-results-tab&runId=14788342&paneView=debug&resultId=100006

/Users/runner/runners/2.163.1/work/1/s/artifacts/bin/redist/Release/dotnet/sdk/5.0.100-ci/NuGet.targets(124,5): error : Unable to load the service index for source https://api.nuget.org/v3/index.json. [/Users/runner/runners/2.163.1/work/1/s/artifacts/tmp/Release/dotnet-msbuild.Tests/WhenRestoreSourcesStartsWithUnixPathThenHttpsSourceIsParsedCorrectly/LibraryWithUnresolvablePackageReference/LibraryWithUnresolvablePackageReference.csproj]
/Users/runner/runners/2.163.1/work/1/s/artifacts/bin/redist/Release/dotnet/sdk/5.0.100-ci/NuGet.targets(124,5): error : The SSL connection could not be established, see inner exception. [/Users/runner/runners/2.163.1/work/1/s/artifacts/tmp/Release/dotnet-msbuild.Tests/WhenRestoreSourcesStartsWithUnixPathThenHttpsSourceIsParsedCorrectly/LibraryWithUnresolvablePackageReference/LibraryWithUnresolvablePackageReference.csproj]
/Users/runner/runners/2.163.1/work/1/s/artifacts/bin/redist/Release/dotnet/sdk/5.0.100-ci/NuGet.targets(124,5): error : Authentication failed, see inner exception. [/Users/runner/runners/2.163.1/work/1/s/artifacts/tmp/Release/dotnet-msbuild.Tests/WhenRestoreSourcesStartsWithUnixPathThenHttpsSourceIsParsedCorrectly/LibraryWithUnresolvablePackageReference/LibraryWithUnresolvablePackageReference.csproj]
/Users/runner/runners/2.163.1/work/1/s/artifacts/bin/redist/Release/dotnet/sdk/5.0.100-ci/NuGet.targets(124,5): error : Unable to find an entry point named 'AppleCryptoNative_SslCreateContext' in shared library 'System.Security.Cryptography.Native.Apple'. [/Users/runner/runners/2.163.1/work/1/s/artifacts/tmp/Release/dotnet-msbuild.Tests/WhenRestoreSourcesStartsWithUnixPathThenHttpsSourceIsParsedCorrectly/LibraryWithUnresolvablePackageReference/LibraryWithUnresolvablePackageReference.csproj]" does not contain visually same fragment string "NU1101".

Linux https://dev.azure.com/dnceng/public/_build/results?buildId=464720&view=ms.vss-test-web.build-test-results-tab&runId=14790786&paneView=debug&resultId=100004

 Restore completed in 423.9 ms for /opt/code/artifacts/tmp/Release/EndToEnd.Tests/ItCanRunToolsInACSProj/MSBuildTestApp/MSBuildTestApp.csproj.
/opt/code/artifacts/bin/redist/Release/dotnet/sdk/5.0.100-ci/NuGet.targets(124,5): error : Unable to load the service index for source https://api.nuget.org/v3/index.json. [/opt/code/artifacts/tmp/Release/EndToEnd.Tests/ItCanRunToolsInACSProj/MSBuildTestApp/MSBuildTestApp.csproj]
/opt/code/artifacts/bin/redist/Release/dotnet/sdk/5.0.100-ci/NuGet.targets(124,5): error : The SSL connection could not be established, see inner exception. [/opt/code/artifacts/tmp/Release/EndToEnd.Tests/ItCanRunToolsInACSProj/MSBuildTestApp/MSBuildTestApp.csproj]
/opt/code/artifacts/bin/redist/Release/dotnet/sdk/5.0.100-ci/NuGet.targets(124,5): error : Authentication failed, see inner exception. [/opt/code/artifacts/tmp/Release/EndToEnd.Tests/ItCanRunToolsInACSProj/MSBuildTestApp/MSBuildTestApp.csproj]
/opt/code/artifacts/bin/redist/Release/dotnet/sdk/5.0.100-ci/NuGet.targets(124,5): error : The type initializer for 'SslMethods' threw an exception. [/opt/code/artifacts/tmp/Release/EndToEnd.Tests/ItCanRunToolsInACSProj/MSBuildTestApp/MSBuildTestApp.csproj]
/opt/code/artifacts/bin/redist/Release/dotnet/sdk/5.0.100-ci/NuGet.targets(124,5): error : The type initializer for 'Ssl' threw an exception. [/opt/code/artifacts/tmp/Release/EndToEnd.Tests/ItCanRunToolsInACSProj/MSBuildTestApp/MSBuildTestApp.csproj]
/opt/code/artifacts/bin/redist/Release/dotnet/sdk/5.0.100-ci/NuGet.targets(124,5): error : Unable to find an entry point named 'CryptoNative_Tls13Supported' in shared library 'System.Security.Cryptography.Native.OpenSsl'. [/opt/code/artifacts/tmp/Release/EndToEnd.Tests/ItCanRunToolsInACSProj/MSBuildTestApp/MSBuildTestApp.csproj]

There are some similar looking crypto issues in aspnet/extensions too: dotnet/extensions#2798.

Unable to find an entry point named 'CryptoNative_EvpCipherCreate2' in shared library 'System.Security.Cryptography.Native.OpenSsl'.

[janvorli]

@dagood is there anything different in the official build compared to the past related to this break?

[dagood]

I don't know, I'm not familiar with the parts of the build that I imagine would affect this (CoreCLR, Libraries). @dotnet/runtime-infrastructure

Here are some outputs to potentially investigate:
https://dotnetcli.blob.core.windows.net/dotnet/Runtime/5.0.0-alpha.1.19622.1/dotnet-runtime-5.0.0-alpha.1.19622.1-osx-x64.tar.gz
https://dotnetcli.blob.core.windows.net/dotnet/Runtime/5.0.0-alpha.1.19622.1/dotnet-runtime-5.0.0-alpha.1.19622.1-linux-x64.tar.gz

[safern]

For reference here are also available the assets produced by the Libraries step for OSX:

https://dev.azure.com/dnceng/_apis/resources/Containers/3080015?itemPath=libraries_bin_OSX_x64_Release%2Flibraries_bin_OSX_x64_Release.tar.gz

cc: @bartonjs @wfurt might have ideas.

[janvorli]

Thanks @safern, looking at e.g. System.Security.Cryptography.Native.Apple.dylib in the https://dev.azure.com/dnceng/_apis/resources/Containers/3080015?itemPath=libraries_bin_OSX_x64_Release%2Flibraries_bin_OSX_x64_Release.tar.gz using objdump -t, I can see that it doesn't export any of the AppleCryptoNative_ symbols. I have built it locally using the libraries.sh script in the root of the runtime repo and the symbols were there just fine. So I wonder what's different in the lab build.

[safern]

Let me look at how the build was setup in Corefx and what build pools we were using vs now in the runtime repo to see if there is something obvious.

[wfurt]

The OpenSSL depends on version of OpenSSL and it may vary between systems. However the Native.Apple should work. Since we fail to load the library, would expect tests to fail as well. I can take look tomorrow.

[janvorli]

@wfurt as I've mentioned above, the problem is that we somehow build or pack an incorrect System.Security.Cryptography.Native.Apple.dylib that is missing all the export AppleCrypto_xxxxxxxxxxx symbols. It has nothing to do with the dependencies of this library.

[am11]

@janvorli, it seems like the said dylib from 3.1 and 3.0 has approximately the same list of symbols exported (exception being _kSecClassKey) as 5.0.0-alpha.1.19622.1 (i.e. no AppleCryptoNative_ prefixed symbol), when we inspect with objdump -t in macOS terminal. However, unprefixed _SSLCreateContext etc. exist in both versions.

[bartonjs]

The unprefixed version with more capitals is an imported function, not an exported function.

[am11]

Ah, alright. Perhaps then objdump -t does not show the exported symbols on macOS at all, since 3.1 and 3.0 dylibs also do not have AppleCryptoNative_*? (btw i got similar results with nm -a and otool -tV)