Max Belugin

Puzzler with structs, generics and nullability

2023-06-10T00:00:00+00:00

Recently I found a non-obvious behavior of combining structs with nullability.

All code for this post is available at my GitHub

The puzzler

Let’s say we want to write a generic transform function for nullable:

  public static TResult? Transform<TSource, TResult>(
        TSource? source,
        Func<TSource, TResult> transformation)
        => source is null
            ? default
            : transformation(source);

It gets a nullable argument, a transformation (function that gets an argument and returns a result) and applies transformation if argument is not null. Otherwise, it returns null.

It works great until you try to apply this to some value type, say DateTime:

DateTime? source = null;
DateTime? result = NullableStructsTransformer.Transform(source, x => x.AddDays(1));

You are getting the following:

Severity Code Description Project File Line Suppression State Error CS1061 ‘DateTime?’ does not contain a definition for ‘AddDays’ and no accessible extension method ‘AddDays’ accepting a first argument of type ‘DateTime?’ could be found

Ok you are are adding an overload with struct constrain and it compiles:

public static TResult? Transform<TSource, TResult>(
        TSource? source,
        Func<TSource, TResult> transformation)
        where TSource : struct // <-- This one
        => source is null
        ? default
        : transformation(source.Value);

But… it does not work. For non-null values it return transformed values, but for nulls it returns default values. For example, for dates it returns <0001-01-01 00:00:00.000>.

The following test fails:

 [Fact]
public void Transform_IfNullStruct_ReturnsNull()
{
    DateTime? source = null;
    DateTime? result = NullableStructsTransformer.Transform(source, x => x.AddDays(1));
    result.Should().BeNull();
}

Did not expect result to have a value, but found <0001-01-01 00:00:00.000>.

The fix

The fixed version is below

    public static TResult? TransformFixed<TSource, TResult>(
        TSource? source,
        Func<TSource, TResult> transformation)
        where TSource : struct
        where TResult : struct // <-- Add constraint on result
        => source is null
            ? default(TResult?) // <-- Specify  return type
            : transformation(source.Value); 

The explanation

Lets insert our code into sharplab to compare:

Not working version:

    [return: System.Runtime.CompilerServices.Nullable(2)]
    public static TResult Transform<TSource, [System.Runtime.CompilerServices.Nullable(2)] TResult>(Nullable<TSource> source, [System.Runtime.CompilerServices.Nullable(new byte[] { 1, 0, 1 })] Func<TSource, TResult> transformation) where TSource : struct
    {
        return (!source.HasValue) ? default(TResult) : transformation(source.Value);
    }

Working version:

    public static Nullable<TResult> TransformFixed<TSource, TResult>(Nullable<TSource> source, [System.Runtime.CompilerServices.Nullable(new byte[] { 1, 0, 0 })] Func<TSource, TResult> transformation) where TSource : struct where TResult : struct
    {
        return (!source.HasValue) ? null : new Nullable<TResult>(transformation(source.Value));
    }

.NET have two nullable types implementations:

reference nullable type
value nullable types

The first kind is not a true .net runtime type, but just an attribute + compiler support.

The second kind is a struct wrapper around value.

When compiler does not know whether result will be value type or not it generates a generic method which returns a result, but marked with nullable attribute. Which works with reference types (since they are always nullable from runtime standpoint), but produces a strange hybrid in case of value types: struct marked with a nullable attribute instead of Nullable<T> for struct.

So actually, instead of default(TResult?) in case of struct it produces default(TResult) which is empty value.

Further steps

We have two versions: 1st one to transform structs to reference types; 2nd one to transform structs to structs.

If somebody accidentally uses 1st version instead of second, we can have compiled code with a problem which will be found in runtime.

Fortunately, we can prevent it easily: just add

where TResult: class

into type constraints of the first version.

All code for this post is available at my GitHub

Perfview in Dyn365FO. Part 5 - How to analyze X++ traces in perfview.

2021-05-22T00:00:00+00:00

You can use PerfView for X++ traces analysis in troubleshooting scenarios.

See the First part for the introduction and a list of posts.

Introduction

Dynamic 365 for Finance and Operations uses ETW technology for doing it’s own traces.

Compiler generates code that inserts an event before and after each method call. Also database engine reports all queries as ETW events.

Traces can be collected from UI or by using Trace Parser.

After that it can be opened in trace parser for analysis. Trace parser provides list of events, tree of calls and list of SQL queries.

That works good. But in some trouble shooting scenarios it can be usefule to search across events. In that case we can use Perfview.

How to use PerfView

We can open eny ETL in PerfView since it shares the same ETW technology with TraceParser.

Unfortunately, we cannot use stack viewer for X++ methods in this trace (you can use perfview to collect .NET trace ) though, but we can use this tool so see event sequence.

Opening trace

Do the following steps on a machine with Dyn365FO installed:

Launch PerfView and navigate to a folder
Double click “events”
Select all event types at the left:
- Click on any event type
- Presss Ctrl+A
Press F5 to refresh

Analysing trace

Setup columns to display: FormattedMessage methodName sqlStatement * (where * means “all other columns” )
Filter events by text. Unfortunately Perfview shows limited number of events (MaxRet input box). We can reduce number of viewed events by:
- excluding event types from the left pane
- filtering by text (Text Filter)
- limiting by time range (Start, End)
After finding a time range where happens something that we are interested in, we can remove Text Filter and set Start and End instead.
We can also use text filter to filter by activity id (since TraceParser catches all the session on the same AOS).

Perfview in Dyn365FO. Part 4 - Bottom up analysis

2019-05-19T00:00:00+00:00

How to find hot spots in your code.

See the First part for the introduction and a list of posts.

Overall plan

When I do a bottom-up analysis the questions I am intereted in are the following:

What resource is the bottleneck – this one was discussed in previous post
What parts of my code eat most time/CPU – so I can optimize them or reduce their usage
What parts of underlying platform consume most of resources - I can not change them directly, but I can reduce their usage or change this usage somehow or ask an owner to optimize that code.

So, most time I do the bottom up analysis then I do the top down analysis of a hotspot then I decide how to optimize it.

How to find a hot spot

Actually I am inetested in 2 levels:

My code (code I can change directly)
Code I call

PerfView can show very deep details of code execution, including windows kernel functions and ODBC calls, but for quickly finding bottleneck these details are too many.

Fortunately, the tool have 2 very powerfull features: grouping and folding:

Grouping allows to combine data from several calls into one (for example, you can summarize calls by class name and have total time for calls of specific class).
Folding allows distribute statistics of some kind of calls to usages. For example, if I see that on the top of statistics there is some common operation like adding strings, I can fold it to see the whole picture - calls of this operation will receive it’s time and operation will disappear from view.

Therefore:

If I want to see statictics of the bottom level of my code, I:
- group modules of my interest by functions
- all other code group to a single path “Other”
- fold the group “Other”
- See what parts of my code consume the most time
If I want to see statistics of underlying code I do the same steps but “entry group” the group “other” - than means that PerfView will group by calls that are entry point to the group (so internal details of all other calls will be hidden, but I will be able to see, what specific calls of other code consue most time). In that case folding is not required.

How to use folding and grouping in PerfView

Quick demo

I want to profile and found of bottleneck in the loading of moodel mapping designer

I collect a trace for the wall-clock analysis
I open the trace, select w3wp process and open the “Thread Time” view
I select “No grouping” in “group pats” and try to find thread in the “CallTree” tab: just enter “modelmapping” in the “Find” input box:
Now I see that find right thread (with ERModelMappingDesigner::Clicked) I can right click on and select “include item” and I will see only the thread I inetersted in, but I after that I edit the “IncPats” slightly - remove module name and arguments (will explain later) so I will have something like “Process% w3wp (7756);ERModelMappingDesigner::Clicked”
Then I switch to “By Name” tab:
I see than the process is CPU bound and can double click CPU_TIME to find a reason (pressing checkboxes in the tree): but most of levels beween CPU and my code are unnecessary - I want to see the big picture.
So I apply grouping rules I prepared (load it to Presets, and select from presets ER -> Other) and repeat selecting CPU and see that all unnecessary details are grouped into node “Other”
Now I can switch back to “By Name” and fold all the ow level items including “Other” (folding means that item’s resources will be distributed across callers)
Now we see a bottleneck and can optimize it (for example ungroup and perfrom a top-down analysis starting with the node)
Other way is to look just under my code and see what calls of underlying platform eats much time: just switch to “ER => Other” preset (then unfold all and fold cpu):

We see the most CPU time spend for moving between records - double clicking on the top row can show callers.

Explanation of grouping presets

I have 2 presets:

“ER -> Other” groups all modules I interested in by methods and all other item are grouped into single group named “Other”
“ER => Other” is the same thing, but performs entry grouping for other cals - that means that all methods from external code will be grouped only when my code calls it, all internal calls between external code will be inside those entry groups.

Grouping syntax

This is my example (download presets file)

[ER=>Other] Microsoft.Dynamics365.{*}!{*}->D365.$1.$2;dynamics.ax.applicationsuite!Dynamics.Ax.Application.{*}->AppSuite.$1;dynamics.ax.electronicreporting{*}!Dynamics.Ax.Application.{*}->ER$1.$2;!=>Other

Explanation:

“[ER=>Other]” - a name of the preset

After the name goes sequence of grouping instructions, each intruction groups some type of paths and after that result is processed by the next instruction.

Grouping instructions are in the following form:

Microsoft.Dynamics365.{*}!{*}->D365.$1.$2;

where:

“Microsoft.Dynamics365.” - this is just a string to match - {*} - is like a .* regex matches any sequence of any symbol
! - in the method path it delimits module (dll) name and a method name
-> is grouping (=> entry grouping)
$1, $2 - like in regular expressions substitutes matches in corresponding curly braces

So if this intruction will meet string Microsoft.Dynamics365.Application!MyClass.myMethod it will convert it to D365.Application.MyClass.MyMethod.

Take into account that result does not contain the ! symbol so the alsready grouped strings will not be processed by the subsequent instructions.

The last instruction !=>Other will take all previously onprocessed strings and put it to groups by entry points (or to the single group named “Other” in case of !->Other)

Perfview in Dyn365FO. Part 3 - Wall clock analysis

2018-11-17T00:00:00+00:00

How to analyze actual time thread spend working or waiting for other threads.

See the First part for the introduction and a list of posts.

First step in analysis is detection, what kind of resource is a bottleneck. In my practice when testing performance on 1 box envinronment, most case is CPU bound tasks:

On onebox we have a very low latency betwen AOS and SQL server so even if we have lots of queries, they are using CPU, not network or disk
Dyn365FO X++ code often use lots of small queries instead of one giant query with join - partly beacuse of it allows to use AOS caching, partly because many queries are in a separate methods (like “find”).
I work on Electronic Reporting module that itself is more like platform then like application code.
The reason for performance degradation can be locks on SQL server side, so it can be reproduced only with simultaneous work of multiple users

Please keep in mind that test and production is a different environement and use other tools (like TraceParser and SQL server tools) to check whether a bottleneck is onthe same place there or not.

Anyway lets strart.

How to analyze thread time

Collect a trace with thread time enabled
Filter by a thread you are interested in
See the most used resource
Recollect trace for CPU, if a task is CPU-bound
Perform top down or bottom up analysis

After you press “Stop collection” and Perfview processed the trace, you will see the following window:

Double click on thread time to select process and see details:

We see list of processes sorted by CPU time used and since I use perfromance test, I see the Visausl Studio test execution engine on top. If you use onebox you can probably see SQL server (if it do the most job). Also you can se AOS (w3wp) batch service.

After pressing OK we will see data for callstacks:

Filtering by thread

This data is not very useful: since we measuring thread time it tells us all the time all thread spend on working and waiting, but we need to analyze only one thread - that is doing actual job.

Also by default data is grouped by module. For selecting thread we

Set “GroupPats” to nothing
Find the thread we actually want to explore: enter in “find” a name of the method that surely exist in callstacks that you are interested in. For my case it is “ERFormatMappingRun.run” and press enter

Look at the pt.3 - perfview says that the whole method run is 0.6% of time. That’s because working and waiting time is summarized for all threads and this is percentage from all that time.

Lets tell PerfView to see only stacks that include “ERFormatMappingRun.run” or method of your choice: type “;ERFormatMappingRun.run()” in the IncPats window and swithch to “by name” tab:

We can see that most time (exclusive %) is spent by CPU - so actually calculating things, not waiting for SQL or network.

The task is CPU bound.

Now you shoould recollect trace for CPU to get better precision. Or, if you have a different result, you can start top down or bottom up analysis:

Go to CallTree tab and investigate perfromance of different parts from the top
Or doubleclick CPU and see the callers tree

The later will be difficult since perfview now does not know what are borders of your task and so it is started from OS internals.

But the tool is very flexible and the next time we will see how to group stacks by meaningful properties.

Perfview in Dyn365FO. Part 2 - Collecting a trace

2018-11-11T00:00:00+00:00

The first step in working with PerfView is collecting data.

See the First part for the introduction and a list of posts.

The overall plan is:

Prepare the scenario (use ~1-2 mins long period)
Select what to trace (trace CPU for CPU-bound tasks, trace thread time otherwise)
Record the trace

Preparing the scenario

First of all, we need to prepare data. Perfview collects samples of callstak, the more samples to collect, the more precise statistics you will get, but trace file will be bigger. Therefore I usually prefer scenarios with duration about 1-2 mins.

Prefview manual says that 10-20 secs would be enough.

Because the samples are taken every millisecond per processor, each sample represents 1 millisecond of CPU time. However exactly where the sample is taken is effectively ‘random’, and so it is really ‘unfair’ to ‘charge’ the full millisecond to the routine that happened to be running at the time the sample was taken. While this is true, it is also true that as more samples are taken this ‘unfairness’ decreases as the square root of the number of samples. If a method has just 1 or 2 samples it could be just random chance that it happened in that particular method, but methods with 10 samples are likely to have truly used between 7 and 13 samples (30% error). Routines with 100 samples are likely to be within 90 and 110 (10% error). For ‘typical’ analysis this means you want at least 1000 and preferably more like 5000 samples (There are diminishing returns after 10K). By collecting a few thousand samples you insure that even moderately ‘warm’ methods will have at least 10 samples, and ‘hot’ methods will have at least 100s, which keep the error acceptably small. Because PerfView does not allow you to vary the sampling frequency, this means that you need to run the scenario for at least several seconds (for CPU bound tasks), and 10-20 seconds for less CPU bound activities.

If I have shorter scenario I just insert loop into the code
if I have longer scenario, I either meaure part of it or reduce data to have 1-2 minutes

Also, if you want to compare time, better to make sure that there is no other tasks on the same machine: open resource monitor (Win+R, resmon, enter) and watch for resource consumption.

Also take into account, that possibly you need to measure warm run (so run scenario once before profiling to load caches, etc. But sometimes cold start is also interesting).

Collect the trace

Collection from the command line

You can instruct perfview to collect trace from the command line. For example below is a simple PowerShell script that I use for collecting thread time trace. It starts collection, builds a trace name from a timestamp, and stops collection when Electroinic Reporting finishes format generation (it detects the moment using windows event log monitoring).

c:\programs\PerfView collect "e:\traces\$(date -format "ddMMyyyy_hhmm").etl" `
     -CircularMB:20000 -ThreadTime `
     -NoNGenRundown `
     -StopOnEtwEvent:Microsoft-Dynamics-ElectronicReporting/FormatMappingRun/Stop

Collection from the gui

Press Collect\Collect
Set parameters:
1. Name of the trace - set the name that will remind you of optimization phase and type of data (for example “initial-cpu.etw”) it can help to find baseline for comparison of traces later
2. Set trace length (1G is enough) - if it is too short, tool wil use it as a circular buffer, so events at start will be lost
3. Set “thread time”, if you are not sure that CPU is bottleneck, do not set if you are 100% sure the task is CPU bound
4. Set “No V3.X NGEN Symbols”
5. Set “Zip” and “merge” if you want to take the trace from current machine to view on other machine (for example, if you have sources and symbols there)
Press “Start collection”. During collection pay some attention to the trace size
Press “Stop collection”

Thats all for today, the next time we will start to analyze a trace.

Using Perfview in Dynamics 365 for Finance and Operations

2018-11-04T00:00:00+00:00

PerfView is my favorite .NET profiling tool. It uses ETW to collect performance data.

This post starts series on working with it for Dynamics 365 for Finance and Operation optimization tasks.

Currently we have following posts:

Why do I like PerfView:

It it is very flexible: it can answers questions like “what are the most resource consuming functions from the code under my control”, “what are the most consuming function from modules that are level below”, etc.
It has rich (but not very intuitive UI) - flexible grouping, flamegraph, etc
It can collect data on the whole call stack (including AOS and ODBC calls).
It can accept data in an easy to understand XML format, that can be writen by tools under my control (for example, in the Electronic Reporting module for Dyn365FO 7.3 we can collect and output trace for configurations without exposing other user’s data)

Perfview showing flamegraph for ER configration:

Perfview vs traceparser

What are upsides and downsides of using PerfView vs Trace parser?

For me the main downside of PerfView in comparison with TraceParse is that it is unable to include SQL queries in the call stacks. While it can work with custom ETW providers, I still have not found a way to show SQL queries data there.

Second, it is not integrated in the Dyn365FO so, you need to access particular box to collect the data. You can collect trace from Dyn365FO UI and use it in tre TraceParser, but for collecting Perfview trace you need to install PerfView or collect .NET related ETW events some other way.

Third, the PerfView UI needs to be adapted to.

How to collect data with perfview

The main sequence of steps to collect data is following. I hope I will cover each step in details in the future:

Determine a resource that limits performance (you can use other tools like resmon for that). If I don’t know, where is a bottleneck, I collect a thread time first, and then use the wall clock analysis to see what resource is limiting scenario performance.
Collect data targeted to bottleneck. Mostly in my practice there are 2 cases:
- The scenario is CPU bound - “w3wp” or “batch” proceses use CPU up to 100% of one core and other resources wait for those processes to complete the task. In this case better to collect CPU usage because wall clock data can be inaccurate and affected by random factors. Also you will not have to deal with selecting the thread, which you want to analyze.
- The scenario is IO bound or waiting for other processes to complete (for example sql server executing query). In this case better to collect thread time and prform wall clock analysis.
  1. Perform top-down or bottom up analyze. You can use rich grouping and folding capabilities PerfView provides, flamegraph, call trees, etc.

Useful links

PerfView has embedded html documentation.
Channel 9 contains old, but very usefull series of videos on PerfView - PerfView tutorial
Excelent video by Sasha Goldshtein from DotNext SpB
Using ETW to profile and fix PowerPoint presentation

Hello, world!

2018-04-11T00:00:00+00:00

Trying a new blog engine

I like it because it uses markdown so I can easilly edit it offline and use version control.

info("@AxCoder:HelloWord");