Dynamics 365 FO: Priority based throttling for integrations

Priority-based throttling introduces service protection settings that prevent the over-utilization of resources to preserve the system’s responsiveness and ensure consistent availability and performance for environments running Finance and Operations apps.

What is throttling and why?

The problem with the OData and custom webservice is that when you make more calls than the endpoints can process, the system performance will be impacted because of the resources that run the OData/Custom webservice endpoints and the rest of the system are shared. So the number of requests sent by oData and Custom web service to the system will start to affect system reliability, the system will start to throttle requests for OData and custom service requests to make sure there is no degradation scenario for users using the Dynamics 365 Unified operations.
So which endpoints are getting throttled

  1. oData EndPoints
  2. Custom web service
  3. DMF Rest API endpoints
  4. Recurring integration dequeue and enqueue API

If you are using oData and custom service-based integrations with D365 Finance and Operations implementation, then D365FO implemented a feature to manage how the application throttles over-utilized requests to preserve the performance of the application. It is called “Priority-based throttling

What causes throttling?

1. High volume of requests from oData/Custom web service causing High resource utlization BUT this not enforcing API rate limit
2. Batch job such as DMF/BYOD exports which causes resource (CPU or memory) spikes in D365FO DB server
3. Resource intensive complex / Slow SQL queries causing resource (CPU or memory) spikes in either AOS or DB server
4. Execution of bad design/code which causes High resource utlization

What is priority-based throttling?

When there is a degradation performance(either due to high integration volume or due to high resource-intensive internal processing), to ensure that users using the D365FO are not impacted, priority-based throttling has been introduced. In other words, OData / custom service requests will be denied and will therefore not be processed, when end-user or system performance is impacted. The priority-based integration gives the D365FO administrator control over Which integration will be throttled first. It gives the D365FO administrators the opportunity to categorize integrations based on business criticality to High, Medium, and Low priority. The integrations with low priority will be throttled first when there is a performance issue, then the medium, and in the last the High Prio ones.

How to assign Priority to D365FO integrations ?

The following steps should be used to make use of the priority based integration feature

  1. Categorize the integrations based on business criticality.
    Priority-based throttling allows the administrator to categorize integrations into three priorities: High, Medium, and Low. To make use of this feature, identify the business criticality of the integrations and assign the priorities.
  2. Create three or more integration accounts and update the connection information
    If you are already using integration accounts per integrations, then you are good to go. If you are not using multiple accounts (either a service account or Azure AD App) for integrations, then you need to create at least three accounts for integrations. These integration accounts will be used to decide the priority of the integrations. Then update the integration connection info to D365FO to use the correct integration account based on the business criticality of the integration.
  3. Assign the priority for the integration in D365FO
    Log into D365FO with an administrator account. Then go to “system Administration“, then click expand all and click on “Throttling priority mapping” under setup.
Set Throttling priority mapping for Priority based throttling for integrations

Then click on “New” and add the integration account and select the priority accordingly. Then click Save.

Set Throttling priority mapping for integrations

Note:
Please ensure KB 4615823, which is part of the quality update for 10.0.17, has been installed in D365FO. This KB ensures to get 429 error code as a response if conditions are met for resource-based throttling. Also fixed CPU-based throttling bug, in which it considered only one core of the CPU instead of the average of the Multi-core.

Priority based throttling for integrations with quality update installed

How to monitor the throttled exceptions ?

The D365FO logs all the throttled exceptions and which can be retrieved using LCS.
1. Log into LCS: https://lcs.dynamics.com/
2. Select the correct Project
3. Select the Environment such as PRD, ACC
4. Then scroll down to “Monitoring” and click on “Environment Monitoring”
5.Then click on Activity, select “Requests throttled” query. Then select the range of date to see the thtottled requests.

Priority based throttling for integrations with enviornment monitoring in LCS
LCS Enviornment Monitoring
Priority based throttling for integrations with enviornment monitoring in LCS for throttled events
LCS: Monitoring throttled requests

If you see the throttling requests, then it is worth looking into the health of your D365FO server for CPU utilization, memory utilization, etc. In our case, one of the CPU was overloaded. Additional information such as slow sql queries, blocking queries can be found via LCS.

Priority based throttling for integrations with enviornment monitoring in LCS with CPU utlization

How to handle request throttling in integrations?

The request to following end point will be throttled when you invoke from Logic App (via HTTPS/Dynamics connector), PowerApp, Azure Function (or any code which invokes these APIs).

When a request is throttled, the system provides a value indicating the duration before any new requests from the user can be processed. When a request is throttled and a 429 error occurs, the response header will include a Retry-After interval, which can be used to retry the request after a specific number of seconds. The following example shows this operation.

    if (!response.IsSuccessStatusCode) 
            { 
                if ((int)response.StatusCode == 429) 
                { 
                    int seconds = 30; 
                    //Try to use the Retry-After header value if it is returned. 
                    if (response.Headers.Contains("Retry-After")) 
                    { 
                        seconds = int.Parse(response.Headers.GetValues("Retry-After").FirstOrDefault()); 
                    } 
                    Thread.Sleep(TimeSpan.FromSeconds(seconds)); 



                    // Retry sending the request.
                } 
            } 

Currently Dynamics 365 UO connectors doesn’t support Throttling.

FAQ

Microsoft FAQ page for latest information

Published by Poojith Jain

Poojith Jain is an Azure Architect with good experience with software design and development. He has a thorough knowledge of Azure Integration and he is passionate about solving complex and challenging problems in the field of Azure

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: