Seg fault in Linq.Expression tests

[ericstj]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=759698
Build error leg or test failing: System.Linq.Expressions.Tests.WorkItemExecution
Pull request: #105636

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": ["SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer", "System.Linq.Expressions.Interpreter.FuncCallInstruction"],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Log: https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-105636-merge-95523dd4fead49c19e/System.Linq.Expressions.Tests/1/console.673b37a5.log?helixlogtype=result
Dump: https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-105636-merge-95523dd4fead49c19e/System.Linq.Expressions.Tests/1/coredump.20.dmp?helixlogtype=result

Relevant portion of crash analysis, some symbols missing cc @hoyosjs

Thread Id: 0x22
      Child SP               IP Call Site
 0x7294dbb1f260 0x72d5778fdef9 0x72d5778fdef9
 0x7294dbb1f310 0x72d576e924bb System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.FuncCallInstruction`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]]..ctor(System.Reflection.MethodInfo)
 0x7294dbb1f3a0 0x72d5f4cc3be4 0x72d5f4cc3be4
 0x7294dbb1f3c0 0x72d5f4b01155 libcoreclr.so!?? at ??:0:0
 0x7294dbb1f400 0x72d5f4bab122 libcoreclr.so!?? at ??:0:0
 0x7294dbb1f7a0 0x72d5779304a1 System.Private.CoreLib.dll!System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(System.Object, System.Span`1<System.Object>, System.Reflection.BindingFlags)
 0x7294dbb1f820 0x72d57791a176 System.Private.CoreLib.dll!System.Reflection.MethodBaseInvoker.InvokeWithOneArg(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
 0x7294dbb1f8c0 0x72d5777fba77 System.Private.CoreLib.dll!System.Reflection.RuntimeConstructorInfo.Invoke(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
 0x7294dbb1f920 0x72d576af0676 System.Private.CoreLib.dll!System.RuntimeType.CreateInstanceImpl(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
 0x7294dbb1f9f0 0x72d576acaac7 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.CallInstruction.SlowCreate(System.Reflection.MethodInfo, System.Reflection.ParameterInfo[])
 0x7294dbb1faf0 0x72d576ac67c2 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.CallInstruction.FastCreate(System.Reflection.MethodInfo, System.Reflection.ParameterInfo[])
 0x7294dbb1fc20 0x72d576ac60a0 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.CallInstruction.Create(System.Reflection.MethodInfo, System.Reflection.ParameterInfo[])
 0x7294dbb1fda0 0x72d576ac5495 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.InstructionList.EmitCall(System.Reflection.MethodInfo, System.Reflection.ParameterInfo[])
 0x7294dbb1fde0 0x72d576abf14d System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.LightCompiler.CompileMethodCallExpression(System.Linq.Expressions.Expression, System.Reflection.MethodInfo, System.Linq.Expressions.IArgumentProvider)
 0x7294dbb1ff50 0x72d577210910 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.LightCompiler.CompileMethodCallExpression(System.Linq.Expressions.Expression)
 0x7294dbb1ff90 0x72d576abd924 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.LightCompiler.CompileNoLabelPush(System.Linq.Expressions.Expression)
 0x7294dbb201c0 0x72d576abc619 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.LightCompiler.Compile(System.Linq.Expressions.Expression)
 0x7294dbb201f0 0x72d576abc349 System.Linq.Expressions.dll!System.Linq.Expressions.Interpreter.LightCompiler.CompileTop(System.Linq.Expressions.LambdaExpression)
 0x7294dbb20320 0x72d576e86be9 System.Linq.Expressions.dll!System.Linq.Expressions.LambdaExpression.Compile(Boolean)
 0x7294dbb20370 0x72d578fef633 System.Linq.Expressions.Tests.dll!System.Linq.Expressions.Tests.CallTests.Call_NoParameters(System.Linq.Expressions.Expression, System.Reflection.MethodInfo, System.Object, Boolean)
 0x7294dbb20400 0x72d577aab5b2 System.Private.CoreLib.dll!DynamicClass.InvokeStub_CallTests.Call_NoParameters(System.Object, System.Span`1<System.Object>)

Could be reflection, or codegen. cc @steveharter @AndyAyersMS in case they see anything.

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=759698
Error message validated: [SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer System.Linq.Expressions.Interpreter.FuncCallInstruction]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/30/2024 4:08:57 PM UTC

Report

Build	Definition	Test	Pull Request
771458	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution
771358	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#106083
770443	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#106167
770388	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#106165
770213	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#106163
768825	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105903
768664	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#106078
768349	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#99596
768237	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution
767931	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#106053
767917	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105841
767251	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#106015
767206	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105941
766489	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution
766122	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105941
765937	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105866
765575	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105928
765321	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105868
764956	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105909
764852	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution
764572	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105846
764162	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105875
763929	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105841
763102	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105666
763231	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#101963
763107	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105826
762507	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105050
761806	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#104562
761539	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105692
761433	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105749
760893	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105300
760243	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105471
760176	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105689
760299	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105610
760097	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105680
759698	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105636
759000	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#105308

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	20	37

[dotnet-policy-service]

Tagging subscribers to this area: @cston
See info in area-owners.md if you want to be subscribed.

[cston]

See also #105704.

[AndyAyersMS]

Let me take a look; this seems to be happing frequently.

[AndyAyersMS]

Based on the windows x64 crash in https://dev.azure.com/dnceng-public/public/_build/results?buildId=764230&view=ms.vss-test-web.build-test-results-tab&runId=19436920&resultId=218614&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab

There is a delegate that blows up the process when invoked. The method invoking the delegate is minopts.

0:010> !DumpObj /d 00000136ce117488
Name:        System.Func`2[[System.Linq.Expressions.Tests.IncDecAssignTests+TestPropertyClass`1[[System.Int32, System.Private.CoreLib]], System.Linq.Expressions.Tests],[System.Int32, System.Private.CoreLib]]
MethodTable: 00007fff9bd6aaa0
EEClass:     00007fff9a59b7e8
Tracked Type: false
Size:        64(0x40) bytes
File:        C:\h\w\B8A309B6\p\shared\Microsoft.NETCore.App\9.0.0\System.Private.CoreLib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007fff998d4530  400021c        8        System.Object  0 instance 00000136ce117488 _target
00007fff998d4530  400021d       10        System.Object  0 instance 00000136ce116978 _methodBase
00007fff99a15170  400021e       18        System.IntPtr  1 instance 00007FFF9AD5F010 _methodPtr
00007fff99a15170  400021f       20        System.IntPtr  1 instance 00007FFF9BD4A0E8 _methodPtrAux
00007fff998d4530  40002c3       28        System.Object  0 instance 0000000000000000 _invocationList
00007fff99a15170  40002c4       30        System.IntPtr  1 instance 0000000000000000 _invocationCount

Here the methodPtr is an invalid address. The methodPtrAux field is an indirection cell for

0:010> !ip2md 00007fff`9ba11e90 
MethodDesc:   00007fff9bd6a100
Method Name:          System.Linq.Expressions.Tests.IncDecAssignTests+TestPropertyClass`1[[System.Int32, System.Private.CoreLib]].get_TestInstance()
Class:                00007fff9bd6a148
MethodTable:          00007fff9bd6a148
mdToken:              0000000006004518
Module:               00007fff9a2e25a0
IsJitted:             yes
Current CodeAddr:     00007fff9ba11e90
Version History:
  ILCodeVersion:      0000000000000000
  ReJIT ID:           0
  IL Addr:            00000177600f5b5b
     CodeAddr:           00007fff9ba11e90  (MinOptJitted)
     NativeCodeVersion:  0000000000000000

Locally (using CI assets) I was able to get 5 crashes in 200 runs, so I may be able to catch this live in the debugger.

Aside from the AV there were also crashes like

Fatal error. Internal CLR error. (0x80131506)
   at System.Delegate.<BindToMethodInfo>g____PInvoke|21_0(System.Runtime.CompilerServices.ObjectHandleOnStack, System.Runtime.CompilerServices.ObjectHandleOnStack, System.RuntimeMethodHandleInternal, System.Runtime.CompilerServices.QCallTypeHandle, System.DelegateBindingFlags)
   at System.Delegate.CreateDelegateInternal(System.RuntimeType, System.Reflection.RuntimeMethodInfo, System.Object, System.DelegateBindingFlags)
   at System.Reflection.RuntimeMethodInfo.CreateDelegateInternal(System.Type, System.Object, System.DelegateBindingFlags)
   at System.Linq.Expressions.Interpreter.FuncCallInstruction`2[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Int32, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor(System.Reflection.MethodInfo)
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, Void**, System.Signature, Boolean)
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(System.Object, System.Span`1<System.Object>, System.Reflection.BindingFlags)
   at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)

Based on the above my guess is that this is an issue in the runtime with stub management?

cc @mangod9

[mangod9]

Don't believe there have been any recent changes I am aware of which might affect this. We can take a look though. Also adding @janvorli @VSadov if it rings a bell?

[AndyAyersMS]

Can't quite figure out how to script this under the debugger, because some of the tests intentionally divide by zero.

[steveharter]

Adding @AaronRobinsonMSFT ? - some delegate changes were made in #105584 on Jul 27 which was a couple days before the first report on July 29.

[mangod9]

this looks similar to #106072 (comment), and a possible root cause PR has been reverted.

[AaronRobinsonMSFT]

I think this was fixed with #106072 (comment)

/cc @jkotas