ClrMD Part 9 – Deciphering Tasks and Thread Pool items

This post of the series shows how to easily list pending tasks and work items managed by the .NET thread pool using DynaMD proxies.

Part 1: Bootstrap ClrMD to load a dump.

Part 2: Find duplicated strings with ClrMD heap traversing.

Part 3: List timers by following static fields links.

Part 4: Identify timers callback and other properties.

Part 5: Use ClrMD to extend SOS in WinDBG.

Part 6: Manipulate memory structures like real objects.

Part 7: Manipulate nested structs using dynamic.

Part 8: Spelunking inside the .NET Thread Pool

Introduction

The previous post showed you how to list the pending tasks and work items from the.NET thread pool. It is now time to find out which method will be called. Last but not least, the running work items will be listed.

Understanding .NET “tasks”

The Task case requires more work to extract meaningful information. The m_taskScheduler field of the class keeps track of the custom scheduler, if any (we need this at Criteo). More importantly, the m_action field stores an Action<T> instance wrapping the callback of the Task as a delegate:

The _target field is the implicit this parameter of instance methods that is pointed to by the _methodPtr/_methodPtrAux field.

The code to extract the name of the method behind the action relies on the ClrRuntime.GetMethodByAddress helper from ClrMD:

In case of anonymous methods used for tasks and work items, the call to GetMethodByAddress will succeed. However, if you code old style by passing a named method (static or not), then the other _methodPtrAux field should be used:

The next step is to figure out the name of the method and the name of the defining type. Again, anonymous methods are treated differently:

The last trick is that tasks and work items use different types to store the method details so the code branches based on the target type name.

How to decode basic thread pool items

When you use ThreadPool.QueueUserWorkItem to ask the .NET thread pool to execute a callback asynchronously, the expected parameter is a WaitCallback that is stored as a QueueUserWorkItem by the thread pool:

Once the WaitCallback is extracted from the callback field of QueueUserWorkItem, the same call to BuildDelegateMethodName does the rest:

And what about running ThreadPool threads?

The previous code allows you to list the pending asynchronous actions that will be run by the thread pool. Getting the list of the running work items and tasks is as easy as iterating the Threads property from ClrRuntime and checking if their IsThreadpoolWorker property is true!

The following code lists the thread waiting on a lock first , then the dead ones and finally the real running ones:

For each worker ClrThread, you know if there was an exception (via the CurrentException property) but more important, you have access to the stack trace. The ClrThread.EnumerateStackTrace method iterate on each stack frame represented by a ClrStackFrame object. The SetThreadWaiters method in the lockingInspection.cs shows how to detect on what locking patterns threads are blocked such as Monitor.Enter, lock, Join, wait on native handles or reader/writer locks.

Last but not least, the following code allows you to know if a thread currently processes a task or a simple work item:

Next step…

This is the last episode of the ClrMD series but it is not the end! We have started a short series about the new version of WinDBG. The next long series will show how to take advantage of the ETW (on Windows) and LTTng (on Linux) events emitted by the CLR to monitor your live applications at close range.

Post written by: