Fuzzlyn is a fuzzer which utilizes Roslyn to generate random C# programs. It runs these programs on .NET core and ensures that they give the same results when compiled in debug and release mode.
We developed Fuzzlyn as a project for the 2018 Language-Based Security course at Aarhus University. Using Fuzzlyn, we have found and reported several bugs in RyuJIT used both by .NET Core and the full .NET framework. We have also found and reported a (harmless) bug in Roslyn.
It has found many thousands of programs producing deviating behavior. Some of the first examples we found can be seen in the examples folder in the v1.0 tag (most of these have since been fixed). To take a couple of them, Fuzzlyn automatically found and produced the following programs:
git clone https://github.com/jakobbotsch/Fuzzlyn.git
Using Fuzzlyn to find bugs
To run Fuzzlyn you must specify the number of programs to generate and check. To run a Fuzzlyn instance that generates a million programs and tries to find bugs in these, run:
dotnet fuzzlyn.dll --num-programs 1000000
Note that this only works with .NET core (i.e. a host like dotnet must be running the program). This is because Fuzzlyn runs instances of itself and it uses the dotnet that started Fuzzlyn to do this.
This command will not produce any output to stdout. However, when a program with deviating behavior is found, Fuzzlyn will append its seed and information about its execution to a file in the current directory called Execution_Mismatch.txt.
Regenerating full programs
When a seed has been obtained, the full program can be regenerated by doing the following:
dotnet fuzzlyn.dll --seed <seed here> --output-source
This will generate exactly the code that produced deviating behavior at runtime. However, these examples cannot be run directly because they include checksumming of variables, which requires the IRuntime interface to be passed to the
Mainmethod. Instead, you can disable checksumming by passing the –checksum- (note the minus) switch:
dotnet fuzzlyn.dll --seed <seed here> --output-source --checksum-
These examples are not very useful because they are very big. For that reason, Fuzzlyn includes an automatic reducer, which takes a seed and reduces the program specified by that seed to something smaller, while retaining the interesting behavior.
To reduce a program, use the –reduce switch:
dotnet fuzzlyn.dll –seed <seed here> –reduce
Depending on the size of the program, this will take a while (the biggest example in the examples directory takes roughly 10 minutes to reduce on an i7-4770k). However, most programs do not take longer than a couple of minutes. The output of this command will be a small C# program, that includes information about its seed, size and runtime behavior.
Copyright (c) 2018 Jakob Botsch Nielsen