Wednesday, April 02, 2008

Stackoverflow exception analysis in crashed .NET 2.0 application

This is second crash analysis I had to perform to find out what is causing web process to crash. Prerequisite as in original crash analysis are Windows Debugging Tools installed (WinDbg), point symbols locations to Microsoft download servers and load SOS for .NET 2.0 to see clr related objects. Commands entered by me and in bold letters, debugger output is in gray, areas of interest are in red. Detailed information about prerequisites are in previous blog in 2007.

Microsoft (R) Windows Debugger Version 6.7.0005.1
Copyright (c) Microsoft Corporation. All rights reserved.

Extracted C:DOCUME~1GSUVAL~1LOCALS~1Temp10000137c0_mdmpmem.hdmp from
C:WINDOWSPCHealthERRORREPQSIGNOFF47D95C22.cab

Loading Dump File [C:DOCUME~1GSUVAL~1LOCALS~1Temp10000137c0_mdmpmem.hdmp]
User Mini Dump File: Only registers, stack and portions of memory are available

Symbol search path is: SRV*c:symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Server 2003 Version 3790 (Service Pack 1) MP (4 procs) Free x86 compatible
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Debug session time: Tue Apr 1 21:09:53.000 2008 (GMT-5)
System Uptime: not available
Process Uptime: 0 days 5:06:56.000
.....................................................................................................................
The call to LoadLibrary(exts) failed, Win32 error 0n2
"The system cannot find the file specified."
Please check your debugger configuration and/or network access.
The call to LoadLibrary(ntsdexts) failed, Win32 error 0n2
"The system cannot find the file specified."
Please check your debugger configuration and/or network access.
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(ab4.11d0): Unknown exception - code e053534f (first/second chance not available)
eax=00a80000 ebx=77e670da ecx=00000019 edx=0000001c esi=00000154 edi=00000000
eip=7c82ed54 esp=01af3de8 ebp=01af3e58 iopl=0 nv up ei ng nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00200297
ntdll!KiFastSystemCallRet:
7c82ed54 c3 ret

Load sos.dll from your .NET 2.0 directory with command below to enable .NET managed code access

0:014> .load C:WINDOWSMicrosoft.NETFrameworkv2.0.50727sos.dll
------------------------------------------------------------
sos.dll needs a full memory dump for complete functionality.
You can create one with .dump /ma <filename>
------------------------------------------------------------

See what managed exception you got (print exception), we can see that we got Exception of type StackOverflowException

0:014> !pe
Exception object: 142d00a4
Exception type: System.StackOverflowException
Message: <none>
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 800703e9

Now we need to see what we got on CLR stack by executing command below. You can see that there is infinite loop of cache being inserted and CacheitemRemovedCallback being called on that event. This documented issue (http://blogs.msdn.com/yangxind/archive/2005/12/20/506151.aspx) and solution is to use Cache.Add instead of Cache.Insert statement for .NET 2.0 applications.

0:014> !clrstack
OS Thread Id: 0x11d0 (14)
ESP EIP
01af4b24 7c82ed54 [FaultingExceptionFrame: 01af4b24]
01af5104 7c82ed54 [HelperMethodFrame: 01af5104]
01af5170 7a608d56 System.Net.Sockets.Socket.MultipleSend(System.Net.BufferOffsetSize[], System.Net.Sockets.SocketFlags)
01af51b4 7a5d43b7 System.Net.Sockets.NetworkStream.MultipleWrite(System.Net.BufferOffsetSize[])
01af51f4 7a5af41d System.Net.Connection.Write(System.Net.ScatterGatherBuffers)
01af51fc 7a5b1433 System.Net.ConnectStream.ResubmitWrite(System.Net.ConnectStream, Boolean)
01af525c 7a57f835 System.Net.HttpWebRequest.EndWriteHeaders_Part2()
01af528c 7a57e002 System.Net.HttpWebRequest.SetRequestContinue(System.Net.CoreResponseData)
01af52a8 7a5af3be System.Net.Connection.ReadComplete(Int32, System.Net.WebExceptionStatus)
01af52e8 7a5af008 System.Net.Connection.SyncRead(System.Net.HttpWebRequest, Boolean, Boolean)
01af533c 7a5aede2 System.Net.Connection.PollAndRead(System.Net.HttpWebRequest, Boolean)
01af5350 7a5b412b System.Net.ConnectStream.PollAndRead(Boolean)
01af535c 7a57f697 System.Net.HttpWebRequest.EndWriteHeaders(Boolean)
01af538c 7a57dee2 System.Net.HttpWebRequest.WriteHeadersCallback(System.Net.WebExceptionStatus, System.Net.ConnectStream, Boolean)
01af5398 7a5b40f6 System.Net.ConnectStream.WriteHeaders(Boolean)
01af53e4 7a57f4da System.Net.HttpWebRequest.EndSubmitRequest()
01af5410 7a57c78b System.Net.HttpWebRequest.CheckDeferredCallDone(System.Net.ConnectStream)
01af5420 7a57ca5e System.Net.HttpWebRequest.GetResponse()
01af5468 6990bf7d System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(System.Net.WebRequest)
01af5498 6990c915 System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(System.Net.WebRequest)
01af549c 69919eb1 System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(System.String, System.Object[])
01af54e0 03745370 eServiceWS.eServiceLiteWS.getListofServiceRequestAreaSubArea()
01af54ec 037450b4 Controller.GetLOVList(LOVTypes)
01af5530 01d3bb07 StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01af5584 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01af55c0 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01af55fc 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af56cc 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af56f0 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
01af574c 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
01af5784 01d3bb8d StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01af57d8 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01af5814 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01af5850 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af5920 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af5944 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
01af59a0 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
01af59d8 01d3bb8d StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01af5a2c 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01af5a68 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01af5aa4 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af5b74 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af5b98 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
01af5bf4 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
01af5c2c 01d3bb8d StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01af5c80 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01af5cbc 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01af5cf8 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af5dc8 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af5dec 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
01af5e48 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
01af5e80 01d3bb8d StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01af5ed4 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01af5f10 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01af5f4c 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af601c 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af6040 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
01af609c 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
01af60d4 01d3bb8d StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01af6128 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01af6164 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01af61a0 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean,
01b1f570 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
01b1f5cc 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
01b1f604 01d3bb8d StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01b1f658 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01b1f694 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01b1f6d0 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01b1f7a0 6601c398 System.Web.Caching.CacheInternal.DoRemove(System.Web.Caching.CacheKey, System.Web.Caching.CacheItemRemovedReason)
01b1f7ac 66023ad2 System.Web.Caching.ExpiresBucket.FlushExpiredItems(System.DateTime, Boolean)
01b1f820 66023c6f System.Web.Caching.CacheExpires.FlushExpiredItems(Boolean, Boolean)
01b1f868 66023d65 System.Web.Caching.CacheExpires.TimerCallback(System.Object)
01b1f86c 793d9c1a System.Threading._TimerCallback.TimerCallback_Context(System.Object)
01b1f870 793683dd System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
01b1f888 793d9c7f System.Threading._TimerCallback.PerformTimerCallback(System.Object)
01b1fa18 79e88f63 [GCFrame: 01b1fa18]
01b1fb5c 79e88f63 [ContextTransitionFrame: 01b1fb5c]

Post a Comment