2019-01-15 00:00:00.4718|17|Error|ServiceControl.Transports.SQLServer.QueueLengthProvider|Error querying sql queue sizes.
System.InvalidOperationException: There is already an open DataReader associated with this Command which must be closed first.
at System.Data.SqlClient.SqlInternalConnectionTds.ValidateConnectionForExecute(SqlCommand command)
at System.Data.SqlClient.SqlConnection.ValidateConnectionForExecute(String method, SqlCommand command)
at System.Data.SqlClient.SqlCommand.ValidateCommand(String method, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.BeginExecuteReaderInternal(CommandBehavior behavior, AsyncCallback callback, Object stateObject, Int32 timeout, Boolean inRetry, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.BeginExecuteReaderAsync(CommandBehavior behavior, AsyncCallback callback, Object stateObject)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1](Func`4 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, Object state, TaskCreationOptions creationOptions)
at System.Threading.Tasks.TaskFactory`1.FromAsync[TArg1](Func`4 beginMethod, Func`2 endMethod, TArg1 arg1, Object state)
at System.Data.SqlClient.SqlCommand.ExecuteReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceControl.Transports.SQLServer.QueueLengthProvider.<UpdateChunk>d__14.MoveNext() in C:\BuildAgent\work\75ab57c3309c2d4a\src\ServiceControl.Transports.SQLServer\QueueLengthProvider.cs:line 147
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceControl.Transports.SQLServer.QueueLengthProvider.<QueryTableSizes>d__13.MoveNext() in C:\BuildAgent\work\75ab57c3309c2d4a\src\ServiceControl.Transports.SQLServer\QueueLengthProvider.cs:line 136
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ServiceControl.Transports.SQLServer.QueueLengthProvider.<<Start>b__10_0>d.MoveNext() in C:\BuildAgent\work\75ab57c3309c2d4a\src\ServiceControl.Transports.SQLServer\QueueLengthProvider.cs:line 77
Had a customer reporting the following exception appearing in their logs
After doing some research, I advised them to add
MultipleActiveResultSets=trueto their connection string (for the monitoring instance) to get them unstuck. This appears to have done the trick but it's not really a permanent solution. I had a look at the code and it's not obvious where this error would be caused. Although all of the calls to the datareader are async, they all should still happen serially so there should never be any kind of crossover.