BenchmarkDotNet(BDN): .NET library for benchmarking you C#/F#/VB code.
BenchmarkDotNET(partial), JMeter, Vegeta, Bombardier
dotTrace, dotMemory, PerfView, PerfMon, Windows Performance Recorder
PIlot
procedure.We can build a quick Hello world example using .NET SDK with BenchmarkDotNET templates.
Generate a new Benchmark project:
Generated HelloBDN.csproj
file:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<OutputType>Exe</OutputType>
</PropertyGroup>
<PropertyGroup>
<PlatformTarget>AnyCPU</PlatformTarget>
<DebugType>pdbonly</DebugType>
<DebugSymbols>true</DebugSymbols>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<Optimize>true</Optimize>
<Configuration>Release</Configuration>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="BenchmarkDotNet" Version="0.14.0" />
<PackageReference Include="BenchmarkDotNet.Diagnostics.Windows" Version="0.14.0"/>
</ItemGroup>
</Project>
Release
mode.BenchmarkDotNet.Diagnostics.Windows
package is for Windows OS.Generated Benchmarks.cs
file:
using System;
using BenchmarkDotNet;
using BenchmarkDotNet.Attributes;
namespace HelloBDN
{
public class Benchmarks
{
[Benchmark]
public void Scenario1()
{
// Implement your benchmark here
}
[Benchmark]
public void Scenario2()
{
// Implement your benchmark here
}
}
}
[Benchmark]
attribute marks a method as a benchmark case, it will got “Invocation” multiple times and BDN will collect its execution time, calculate.Generated Program.cs
file:
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Running;
namespace HelloBDN
{
public class Program
{
public static void Main(string[] args)
{
var config = DefaultConfig.Instance;
var summary = BenchmarkRunner.Run<Benchmarks>(config, args);
// Use this to select benchmarks from the console:
// var summaries = BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, config);
}
}
}
BenchmarkRunner.Run<Benchmarks>(config, args)
will run all benchmarks in Benchmarks
class.IConfig
class or using fluid API style to create config object and use it in BenchmarkRunner.Run<T>()
method.Inside project folder, run:
After running the benchmark, BDN will generate a report in the console, and also generate report file(s) in BenchmarkDotNet.Artifacts/results
folder.
So we achieved a quick benchmark demo with BenchmarkDotNET.
When we run the benchmark, BDN do the following steps:
A microbenchmark example with BDN to prove characteristic of four Fibonacci Sequence generate algorithm:
Source code: https://github.com/windperson/DemoFibonacciBDN
FibonaccCore
project is a class library project, contains four Fibonacci Sequence generate algorithm.FibSeqMicroBench
project is a console project that we write BDN benchmark test code here.FibonaccCore
project:public static class SequenceLib
{
/// <summary>
/// Calculate Fibonacci number using loop implementation
/// </summary>
/// <param name="n"></param>
/// <returns></returns>
public static BigInteger FibonacciUsingLoop(int n)
{
if (n <= 1)
{
return n;
}
var a = new BigInteger(0);
var b = new BigInteger(1);
var result = new BigInteger(0);
for (var i = 2; i <= n; i++)
{
result = a + b;
a = b;
b = result;
}
return result;
}
/// <summary>
/// Calculate Fibonacci number using recursion implementation
/// </summary>
/// <param name="n"></param>
/// <returns></returns>
public static BigInteger FibonacciUsingRecursion(int n)
{
if (n <= 1)
{
return n;
}
else
{
return FibonacciUsingRecursion(n - 1) + FibonacciUsingRecursion(n - 2);
}
}
/// <summary>
/// Calculate Fibonacci number using Golden Ration approximation math formula
/// https://www.wikihow.com/Calculate-the-Fibonacci-Sequence#Using-Binet.27s-Formula-and-the-Golden-Ratio
/// </summary>
/// <param name="n"></param>
/// <returns></returns>
public static BigInteger FibonacciUsingGoldenRatio(int n)
{
if (n <= 1)
{
return n;
}
// will be inaccurate after 70th Fibonacci number
// https://stackoverflow.com/questions/41938313/n-th-fibonacci-with-binets-formula-not-accurate-after-70#41938441
var phi = (1 + Math.Sqrt(5)) / 2;
var result = (Math.Pow(phi, n) - Math.Pow(1 - phi, n)) / Math.Sqrt(5);
return new BigInteger(Math.Round(result));
}
/// <summary>
/// Calculate Fibonacci number using Matrix Exponentiation
/// https://www.nayuki.io/page/fast-fibonacci-algorithms
/// </summary>
/// <param name="n"></param>
/// <returns></returns>
public static BigInteger FibonacciUsingMatrixExponentiation(int n)
{
if (n <= 1)
{
return n;
}
BigInteger[,] F = { { 1, 1 }, { 1, 0 } };
Power(F, n - 1);
return F[0, 0];
}
private static void Power(BigInteger[,] F, int n)
{
if (n <= 1)
{
return;
}
BigInteger[,] M = { { 1, 1 }, { 1, 0 } };
Power(F, n / 2);
Multiply(F, F);
if (n % 2 != 0)
{
Multiply(F, M);
}
}
private static void Multiply(BigInteger[,] F, BigInteger[,] M)
{
var x = F[0, 0] * M[0, 0] + F[0, 1] * M[1, 0];
var y = F[0, 0] * M[0, 1] + F[0, 1] * M[1, 1];
var z = F[1, 0] * M[0, 0] + F[1, 1] * M[1, 0];
var w = F[1, 0] * M[0, 1] + F[1, 1] * M[1, 1];
F[0, 0] = x;
F[0, 1] = y;
F[1, 0] = z;
F[1, 1] = w;
}
/// <summary>
/// Calculate Fibonacci number using Fast Doubling ( https://www.nayuki.io/page/fast-fibonacci-algorithms )
/// </summary>
/// <param name="n"></param>
/// <returns></returns>
public static BigInteger FibonacciUsingFastDoubling(int n)
{
var a = BigInteger.Zero;
var b = BigInteger.One;
for (var i = 31; i >= 0; i--)
{
var d = a * (b * 2 - a);
var e = a * a + b * b;
a = d;
b = e;
if ((((uint)n >> i) & 1) != 0)
{
var c = a + b;
a = b;
b = c;
}
}
return a;
}
}
FibonaccCore
project:[RankColumn(NumeralSystem.Roman)]
public class FibonacciSeqBenchmarks
{
[ParamsSource(nameof(NthValues))] public int Nth { get; set; }
public static IEnumerable<int> NthValues => ActualFibonacci.Keys;
private static int RecursionUpperLimit =>
int.TryParse(Environment.GetEnvironmentVariable(Const.RecursionUpperLimit), out var limit)
? limit
: Const.RecursionUpperLimitValue;
[Benchmark(Baseline = true), BenchmarkCategory("simple", "canonical")]
public BigInteger FibSeqUsingLoop()
{
var result = FibonacciCore.SequenceLib.FibonacciUsingLoop(Nth);
ValidateCorrectness(Nth, result);
return result;
}
[Benchmark, BenchmarkCategory("simple", "slow")]
public BigInteger FibSeqUsingRecursion()
{
if (Nth > RecursionUpperLimit)
{
throw new NotSupportedException($"Recursion will run too long for {Nth}th over {RecursionUpperLimit}th");
}
var result = FibonacciCore.SequenceLib.FibonacciUsingRecursion(Nth);
ValidateCorrectness(Nth, result);
return result;
}
[Benchmark, BenchmarkCategory("math", "approximate")]
public BigInteger FibSeqUsingGoldenRatio()
{
var result = FibonacciCore.SequenceLib.FibonacciUsingGoldenRatio(Nth);
ValidateCorrectness(Nth, result);
return result;
}
[Benchmark, BenchmarkCategory("math", "fast")]
public BigInteger FibSeqUsingMatrixExponentiation()
{
var result = FibonacciCore.SequenceLib.FibonacciUsingMatrixExponentiation(Nth);
ValidateCorrectness(Nth, result);
return result;
}
[Benchmark, BenchmarkCategory("math", "faster")]
public BigInteger FibSeqUsingFastDoubling()
{
var result = FibonacciCore.SequenceLib.FibonacciUsingFastDoubling(Nth);
ValidateCorrectness(Nth, result);
return result;
}
#region Check Fibonacci correctness
// see https://r-knott.surrey.ac.uk/Fibonacci/fibtable.html for precomputed Fibonacci series
private static IReadOnlyDictionary<int, BigInteger> ActualFibonacci => new Dictionary<int, BigInteger>()
{
{ 1, 1 },
/* too long so omit it ... */
{ 300, BigInteger.Parse("222232244629420445529739893461909967206666939096499764990979600") }
};
[MethodImpl(MethodImplOptions.NoInlining)]
private static void ValidateCorrectness(int Nth, BigInteger result)
{
if (ActualFibonacci[Nth] != result)
{
throw new ArithmeticException(
$"Fibonacci calculation failed, actual {Nth}th is '{ActualFibonacci[Nth]}', but calculated is '{result}'");
}
}
#endregion
}
FibSeqMicroBench
project:static void Main(string[] args)
{
var bdnConfig =
DefaultConfig.Instance.AddJob(
Job.ShortRun.WithStrategy(RunStrategy.Throughput)
.WithIterationCount(5)
.WithEnvironmentVariable(new EnvironmentVariable(Const.RecursionUpperLimit,
$"{Const.RecursionUpperLimitValue}"))
.WithPowerPlan(PowerPlan.UserPowerPlan)
.AsDefault());
BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, bdnConfig);
}
FibSeqMicroBench.csproj
file:<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net8.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<Optimize>true</Optimize>
<Configuration>release</Configuration>
<IsPackable>false</IsPackable>
<DebugType>pdbonly</DebugType>
<DebugSymbols>true</DebugSymbols>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\FibonacciCore\FibonacciCore.csproj" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="BenchmarkDotNet" Version="0.14.0" />
</ItemGroup>
</Project>
FibonacciCore.csproj
file:<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
<Optimize>true</Optimize>
<DebugType>pdbonly</DebugType>
<DebugSymbols>true</DebugSymbols>
</PropertyGroup>
</Project>
At first you may attempt to run the benchmark with dotnet run -c Release
command directly inside the FibSeqMicroBench folder, but you will get an error message like this:
This is due to BDN default wants to run benchmark using optimized build, and by design of .NET Core SDK, if you don’t specify the solution file but just run the command in the project folder, the dependency project will not be built in optimized mode.
So you need to execuate the dotnet run
command in the solution folder.
(See readme file in the github project root folder)
You can invoke some BDN command line arguments to control benchmark running behavior:
-f --filter
: filter the benchmark cases to run.--list tree
or --list flat
: list all benchmark cases. --help
: show help message. For a basic ASP.NET Core Web API / minimal API project, we can create the WebApplication
instance and associated HttpClient in the [GlobalSetup] method, and dispose of them in the [GlobalCleanup]
method.
Example project: https://github.com/windperson/DemoAspNetCoreBenchmark
The [GlobalSetup]
and [GlobalCleanup]
can accept async Task
method return value signature:
WebApiBenchmarks/MvcVsMinimalApiBenchmarks.cs
[GlobalSetup]
public async Task Setup()
{
_sutMvcWebApi = WebApplicationHelper.CreateSutMvcWebApi(["--urls", MvcWebApiUrl]);
1 await _sutMvcWebApi.StartAsync();
_mvcWebApiClient = new HttpClient()
{
BaseAddress = new Uri(MvcWebApiUrl)
};
_sutMinimalApi = WebApplicationHelper.CreateSutMinimalApi(["--urls", MinimalApiUrl]);
2 await _sutMinimalApi.StartAsync();
_minimalApiClient = new HttpClient()
{
BaseAddress = new Uri(MinimalApiUrl)
};
}
[GlobalCleanup]
public async Task Cleanup()
{
_mvcWebApiClient.Dispose();
await _sutMvcWebApi.StopAsync();
_minimalApiClient.Dispose();
await _sutMinimalApi.StopAsync();
}
[Benchmark]
public async Task<string> MvcWebApi()
{
var response = await _mvcWebApiClient.GetAsync("/weatherforecast");
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
RunAsync()
because it will block the thread until WebApplication shutdown, use StartAsync()
instead.
WebApiBenchmarks/WebApplicationHelper.cs
public static WebApplication CreateSutMvcWebApi(string[] args)
{
var builder = WebApplication.CreateBuilder(args);
// Add services to the container.
builder.Services.AddAuthorization();
builder.Services.AddControllers()
.AddApplicationPart(
1 typeof(SutMvcWebApi.Controllers.WeatherForecastController).Assembly);
var app = builder.Build();
// Configure the HTTP request pipeline.
app.UseHttpsRedirection();
app.UseAuthorization();
app.MapControllers();
return app;
}
For microservice architecture,
We can verify the performance of ASP.NET Core Web API, gRPC, and Microsoft Orleans with BenchmarkDotNet.
(A kind of Macro benchmark project)
https://github.com/windperson/DemoAspNetCoreMacroBenchmark/tree/hwdc_2024
When design test data & benchmark method signature, Be caution of C# Compiler limitation:
CSC : error CS8103: Combined length of user strings used by the program exceeds allowed limit.
Caution
The workaround is to reduce the usage of primitive string type parameter in benchmark method signature.
[GlobalSetup] / [GlobalCleanup] / [IterationSetup] / [IterationCleanup] attributes to manually setup/cleanup the DI container.
private ServiceProvider _serviceProvider = null!;
private Echo.EchoClient _client = null!;
[GlobalSetup]
public void PrepareClient()
{
var serviceCollection = new ServiceCollection();
serviceCollection.AddGrpcClient<Echo.EchoClient>(
options =>
{
options.Address = new Uri("https://localhost:7228");
options.ChannelOptionsActions.Add(channelOptions =>
{
// you need to raise the message size limit to send large messages
// see https://github.com/grpc/grpc-dotnet/issues/2277#issuecomment-1728559455
channelOptions.MaxSendMessageSize = int.MaxValue;
channelOptions.MaxReceiveMessageSize = int.MaxValue;
});
});
_serviceProvider = serviceCollection.BuildServiceProvider();
}
[IterationSetup]
public void InitGrpcClient()
{
_client = _serviceProvider.GetRequiredService<Echo.EchoClient>();
}
[Benchmark]
[ArgumentsSource(nameof(GetTestData))]
public async Task<string> gRPC_Invoke(RequestMsg request)
{
var reply = await _client.EchoAsync(new EchoRequest { Message = request.Message });
return reply.Message;
}
[IterationCleanup]
public void CleanupClient()
{
_client = null!;
}
[GlobalCleanup]
public void CleanupServiceProvider()
{
_serviceProvider.Dispose();
}
BDN can support .NET Core/.NET5+ WPF / WinForm applications.
(Besure to set the [STAThread] attribute on benchmark method)
https://github.com/windperson/DemoWpfAppBDN
WpfFlowDocBenchmark\RichTextFlowDocBenchmarks.cs
And also this example project can use ETW Profiler of BenchmarkDotNet to analyze the performance of WPF view:
restore.cmd
on Windows or restore.sh
on Linux/macOS at the root of git repository.build.cmd
on Windows or build.sh
on Linux/macOS (Note: if there’s some build fail about NPM/Webpack, just ignore it)dotnet run -c Release -- -f '*'
to run benchmarks.Benchmark on Xamarin Forms / .NET MAUI Android & iOS:
https://github.com/dotnet/BenchmarkDotNet/issues/2242
https://github.com/dotnet/maui/tree/main/src/Core/tests/Benchmarks.Droid
Unity3D game engine => Can’t due to its underlying compiler:
https://github.com/dotnet/BenchmarkDotNet/issues/759
Godot game engine nuget package for running BDN on its C# script:
https://github.com/ZerxZ/BenchmarkDotNet.Godot
Benchmark Dispatcher.Sort() v.s. Array.Sort() of Stride 3D Game Engine:
https://github.com/stride3d/stride/issues/1792#issue-1901631515
Benchmark native memcpy v.s. Buffer.MemoryCopy:
https://github.com/stride3d/stride/pull/536#issue-511071979
GitHub Action plugin to include Micro Benchmark on CI/CD:
https://github.com/benchmark-action/github-action-benchmark default using BDN on .NET C#:
https://github.com/benchmark-action/github-action-benchmark/blob/master/examples/benchmarkdotnet/README.md
Or just using command line in GitHub Action .yml configuration file:
https://codezen.medium.com/buckle-up-for-benchmarkdotnet-def73302abed
A stress test platform for .NET CLR & ASP.NET Core develop team to run performance test:
https://github.com/dotnet/crank
ASP.NET Core “Benchmark” repository using Crank to do micro benchmark performance test:
https://github.com/aspnet/Benchmarks/tree/main/scenarios#micro-benchmarks
“Client-Server” architect, which can accept BDN project to build & run for doing micro benchmark:
Write a .yml file to specify the .NET SDK, runtime version, and how to run the benchmark project, then submit project folder data to a remote machine for building & running benchmark test on “Crank-Agent” server:
Crank-Controller side can use --json
argument to export Crank-Agent benchmark running raw data as json format file in controller side , then use compare
command option to merge results:
https://github.com/windperson/DemoMultipleNugetVersionBenchmark
Its purpose is to compare Object to JSON string serialization performance of different libraries:
It will help you resolve the issue when you need to do some performance comparison between different versions of the same library by using WithNuget()
API of BDN:
or different libraries that consume the same nuget package(eq. JSON.NET) but with different versions, and you want to write benchmarks to compare:
Using Benchmark.NET ETWProfiler with PerfView:
https://adamsitnik.com/ETW-Profiler/
https://adamsitnik.com/Sample-Perf-Investigation/
Using Visual Studio profiler to see BDN test case benchmark diagnostic data:
https://learn.microsoft.com/en-us/visualstudio/profiling/profiling-with-benchmark-dotnet
Using JetBrain dotTrace to diagnose BDN test case:
https://blog.jetbrains.com/dotnet/2023/07/11/dottrace-comes-to-benchmarkdotnet/
Use R lang to draw performance compare graph:
https://makolyte.com/comparing-performance-with-benchmarkdotnet-graphs/
Icon | Purpose | Document / Tutorial Video |
---|---|---|
![]() |
HTML Report | |
![]() |
Stack Viewer | document |
![]() |
Raw Trace Event | video |
![]() |
GC Heap Analyzer | video |
Use the official WPF Gallery Preview sample app’s Icons Page to record performance data.
E13B77A8-14B6-11DE-8069-001B212B5009:2:*
in “Additional Providers”, make sure the “.NET Symbol Collection” is checked, then press Start Collection button:IconsPageViewModel
in the top Find input box to search for the ViewModel class entries in the list, PerfView will auto switch to a Call Tree view to see call tree view, and select correct view style in GroupPats: list:Use “Flame Graph” to get visualized overview of performance data.
“dotnet performance” use BDN to do micro benchmark:
https://github.com/dotnet/performance/blob/main/src/benchmarks/micro/README.md
gRPC ASP.NET core microbenchmark:
https://github.com/grpc/grpc-dotnet/tree/master/perf/Grpc.AspNetCore.Microbenchmarks
ASP.NET Core routing component micro benchmarks: https://github.com/dotnet/aspnetcore/tree/main/src/Http/Routing/perf/Microbenchmarks
Micro Benchmarks of PowerShell 7:
https://github.com/PowerShell/PowerShell/tree/master/test/perf/benchmarks
RX.NET(ReactiveX) benchmark:
https://github.com/dotnet/reactive/tree/main/Rx.NET/Source/benchmarks/Benchmarks.System.Reactive
The dotnet runtime benchmark tool Microsoft.Crank can combine use of BDN to do micro benchmark:
https://github.com/dotnet/crank/blob/main/docs/microbenchmarks.md
C# Open-Telemetry SDK core component benchmark:
https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/test/Benchmarks
.NET MAUI performance test:
https://github.com/dotnet/maui/tree/main/src/Core/tests/Benchmarks.Droid
Avalonia UI framework benchmark:
https://github.com/AvaloniaUI/Avalonia/tree/master/tests/Avalonia.Benchmarks
Benchmark on ASP.NET Core Orleans:
https://github.com/dotnet/orleans/tree/main/test/Benchmarks
EFCore benchmark projects:
https://github.com/dotnet/efcore/tree/main/benchmark
Benchmark on Dapper:
https://github.com/DapperLib/Dapper/tree/main/benchmarks/Dapper.Tests.Performance
Newtonsoft.Json benchmarks:
https://github.com/JamesNK/Newtonsoft.Json/tree/master/Src/Newtonsoft.Json.Tests/Benchmarks
Math library benchmark of Silk.NET:
https://github.com/dotnet/Silk.NET/tree/main/src/Maths/Silk.NET.Maths.Benchmarks
Benchmark ML.NET:
https://github.com/dotnet/machinelearning/tree/main/test/Microsoft.ML.PerformanceTests
ComputeSharp’s various Benchmarks project:
https://github.com/Sergio0694/ComputeSharp/tree/main/samples/ComputeSharp.Benchmark