A short guide to urgent CPU usage profiling of .NET applications on Linux using perf

Introduction

Based on a true story. Imagine an evening on Friday. Your thoughts are already at home. But suddenly, your Site Reliability Engineer (SRE) informs you in a Slack channel that your application is consuming 100% of the CPU on the company’s Linux virtual machine. What would you do?

Well, Let me help you. In this hands-on guide, we will explore the use of the perf tool on Linux for profiling .NET applications. perf is a powerful performance analysis tool that can provide detailed insights into the CPU behavior of your applications.

Pros of perf

1. It is free.
2. Easy installation: Use it on any Linux machine.
3. Powerful: I successfully identified a function that was being called 500 million times in recursion (obviously a bug) on my production machine with a .NET 7 application installed.

Prerequisites

perftool: Ensure that perf is installed on your system. You can install it using the package manager specific to your Linux distribution.

# For Ubuntu/Debian
sudo apt-get install linux-tools-common linux-tools-$(uname -r)

# For CentOS/RHEL
sudo yum install perf

A few words about neccessary environment variables for .NET that we will be using

Before starting up dotnet application that you want to profile you need to set up a several environment variables for it

DOTNET_EnableEventLog=0

This environment variable controls the event logging of the application. While you probably need this in your production environment, it can introduce additional CPU load during profiling, as perf will also write event logs.

DOTNET_PerfMapEnabled=0

This environment variable is crucial for profiling if you want to see the names of your functions. With this variable enabled, .NET will write maps of CPU addresses to .NET functions into the file /tmp/perf-<PID>.map.

DOTNET_EnableWriteXorExecute=0

To remove [unknown] /memfd:doublemapper in function names.

Notice, that for .NET version < 6 you must use COMPlus_ instead of DOTNET_ prefix!

Lets create a new application to profile

mkdir perf-test
cd perf-test
dotnet new console
nano Program.cs

Paste the following code

using System;

public class ProfileMe {
    static int variable;
    
    public static void Function1() {
        for(var i = 0; i < 1000000; i++) {
            variable += 10;
            variable *= 100;
        }
    }
    
    public static void Function2() {
        for(var i = 0; i < 1000000; i++) {
            variable += 10;
            variable *= 100;
        }
    }
    
    public static void Main(string[] args) {
        Console.WriteLine ("Starting application");
        for(var i = 0;; i++) {
            if(i % 3 == 0) {
                Function1();
            }
            else {
                Function2();   
            }
        }
    }
}

As you see Function1 takes around 1/3 of the CPU cycles of this program, while Function2 takes another 2/3.

Compile and start the program.

dotnet publish
cd bin/Release/net<your-version>/publish

DOTNET_EnableWriteXorExecute=0 DOTNET_EnableEventLog=0 DOTNET_PerfMapEnabled=1 ./perf-test &

And check our top processes.

top -d 1 # Refresh every 1 sec

As you can see on the screenshot perf-test indeed takes 100% of single core of my CPU.

See what is going on inside your application

sudo perf top -F 100 -d 10 -p 28636 # use PID from top output

What does -F -d and -p flags mean?

-F is a sample frequency. So in the code above 100 samples per second will be generated to reduce overhead.
-d is a refresh rate of the output. Since we are running ‘perf top’ it will be writing current state of the program every 10 seconds.
-p is a process PID that you want to profile.

You must see something like this in your terminal

Here “Shared Object” is your process or program and “Overhead” sums up to 100% (total CPU usage of all your objects), where Function1 takes 66.6% of execution time as I mentioned earlier, while Function1 takes roughly 33.3%.

Also you can run perfwith -g flag to get a call-stack to investigate.

sudo perf top -F 100 -d 10 -p 28636 -g -K

In this case “Children” column will show you all the CPU cycles spent in all descendant functions in call graph, while “Self” CPU cycles spent in the function itself.

Further reading:

perf Examples

Debug high CPU usage - .NET Core - .NET

GitHub - microsoft/perfview: PerfView is a CPU and memory performance-analysis tool

Also there is a handy Firefox Profile web viewer, where you can upload files generated by perf report
https://profiler.firefox.com/

dotnet-trace for .NET Core tracing in PerfView, SpeedScope, Chromium Event Trace Profiling, Flame…