slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1412798	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127909
1412485	dotnet/maui	Controls Shell	Log	dotnet/maui#35330
1412469	dotnet/maui	Controls Shell	Log
1412464	dotnet/maui	Controls Shell	Log
2969639	dotnet-maui	RunOniOS_MauiReleaseTrimFull x64	Log
1412446	dotnet/maui	Controls Layout	Log	dotnet/maui#35292
1412490	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127926
1412495	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66531
2969640	dotnet-maui	Controls CollectionView	Log
1412438	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66616
1412396	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127920
1412300	dotnet/maui	Controls Shell	Log	dotnet/maui#35325
1411983	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35335
1412246	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35250
1412277	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#125436
1412228	dotnet/maui	Controls Layout	Log
1412263	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#125437
1412162	dotnet/maui	Controls Layout	Log	dotnet/maui#35345
2969365	dotnet-maui	RunOniOS_MauiReleaseTrimFull ARM64	Log
2969543	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	#60854
1411917	dotnet/maui	Controls Shell	Log
2969526	dotnet-runtime	maccatalyst-x64 Release AllSubsets_Mono	Log	#60855
1412176	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127913
2969505	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#60864
2969434	dotnet-performance	Performance imagesharp ubuntu 2204 x64 Viper 8.0	Log
1412038	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66513
1409583	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35311
2969255	dotnet-maui	Controls CollectionView	Log
1411939	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125491
1411727	dotnet/maui	Controls CollectionView	Log
1411831	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125729
2969254	dotnet-maui	RunOniOS_BlazorRelease x64	Log
1411802	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127904
1411800	dotnet/runtime	osx-arm64 Release CoreCLR_Libraries	Log
1411798	dotnet/aspnetcore	Tests: macOS	Log
1411736	dotnet/runtime	osx-x64 release Runtime_Release	Log
1411610	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127228
2969122	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1411505	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1411424	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#126569
1411455	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66611
1411362	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127450
1411105	dotnet/maui	Controls Shell	Log	dotnet/maui#35330
2968897	dotnet-maui	Controls (vlatest) Editor,Effects,Essentials,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log
1411041	dotnet/maui	Controls CollectionView	Log
2968876	dotnet-runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	#60854
2968881	dotnet-runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	#60855
1411024	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127883
2968875	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#60853
1409635	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35315
1410945	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127844
1410936	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#126552
2968840	dotnet-sdk	Darwin	Log	#60849
2968823	dotnet-sdk	Darwin	Log	#60843
1410900	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127877
1410868	dotnet/runtime	osx-x64 Debug PALTests	Log
1410745	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log	dotnet/runtime#125437
1410714	dotnet/aspnetcore	Tests: macOS	Log
1410712	dotnet/aspnetcore	Tests: macOS	Log
1410677	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#100729
1410665	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127328
2968605	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT iOSStripSymbols osx x64 perfiphone17 net11.0	Log
1410597	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127864
2968551	dotnet-runtime	Libraries Test Run release coreclr osx x64 Debug	Log	#60831
1410512	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127749
1410331	dotnet/runtime	Libraries Test Run release coreclr osx arm64 Debug	Log	dotnet/runtime#127454
1410451	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127760
1410126	dotnet/maui	Controls Shell	Log	dotnet/maui#35307
2968521	dotnet-performance	Performance powershell ubuntu 2204 x64 Viper main	Log
1410377	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127762
1410338	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127281
1410319	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127621
1410315	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log	dotnet/runtime#127787
1410273	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126324
1410270	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127269
1410036	dotnet/maui	Controls CollectionView	Log	dotnet/maui#32674
1410034	dotnet/maui	Controls Shell	Log	dotnet/maui#29591
2968449	dotnet-performance	Performance scenarios ubuntu 2204 arm64 Ampere 8.0	Log
1410032	dotnet/maui	Controls Layout	Log	dotnet/maui#35131
1410216	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#100729
1410195	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127755
1408694	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127789
1410125	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127762
1410122	dotnet/runtime	osx-arm64 Release Libraries_CheckedCoreCLR	Log
2968363	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	#60646
2968356	dotnet-performance	Performance scenarios ubuntu 2204 arm64 Ampere 9.0	Log
1410075	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127749
1410030	dotnet/maui	Controls Layout	Log	dotnet/maui#35197
1410041	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127679
2968189	dotnet-sdk	AoT: macOS (x64)	Log	#60805
2968201	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1409965	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#54173
1409955	dotnet/roslyn	Build_macOS_Debug	Log
2968190	dotnet-sdk	AoT: macOS (x64)	Log	#60811
2968162	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2968177	dotnet-sdk	Darwin	Log	#60809
2968175	dotnet-sdk	Darwin	Log	#60808
1409943	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1409827	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127819
1409804	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#54133
Displaying 100 of 1494 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
45	269	1494

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.