Real-time programming in C# and the eCLR

To develop real-time critical function blocks (FB), libraries and applications (apps) using C# and the .Net™ framework of the eCLR, certain fundamental aspects must be considered during development.

Heap memory allocation and Garbage Collection (GC)

Heap memory allocation and Garbage Collection behavior is non-deterministic and can cause unwanted delays in a real-time task. Therefore the use of new() in real-time critical contexts should be avoided as much as possible. Instead, objects should ideally be created on the heap during the FB initialization.

Using new() inside loop quickly generates garbage, which can lead to timing issues due to the GC after some runtime.

Example:

for(int i = 0; i < maxValue; i++)
{
    IPEndpoint ip = new IPEndpoint(0,0);
    socket.Connect(ip);
}

Here, the ip variable is repeatedly created on the heap using new(), and after each loop iteration it is marked as garbage by the GC.

A better approach is to create the ip variable on the heap before the loop, and then modify and reuse its values within the loop.

Example:

IPEndpoint ip = new IPEndpoint(0,0);
for(int i = 0; i < maxValue; i++)
{
    socket.Connect(ip);
}

The same can be said about new allocation for variables inside frequently used functions, such as thread bodies or the FB's __Process() method:
Try to minimize the use of new() as much as possible. But if you need to use new() during the running process, use it deliberately and as infrequent as possible; for example, not every time the __Process() function is called.

It might also be worth noting that – unlike with objects (classes) – using the new() operator on a struct doesn't allocate more memory on the heap. Instead, it resets the struct to its default value.

Local Roots

In order for the GC to be able to track local objects and strings, our compiler generates two functions inside the relevant function that contains these local datatypes, regardless of where in the function or in what context these local objects are used. These functions are System.GCInternal.LocRootEnter() and System.GCInternal.LocRootLeave(). Both of these calls will result in the GC having to use an internal mutex.

While this is generally not an issue, in stressed systems this can lead to additional avoidable stress, especially in a multi-core environment with functions that are rapidly called. Like in loops, thread bodies or the __Process() function of an FB.

Example:

[Execution]
public void __Process() 
{
    if(someCondition)
    {
        // some logic
        Eclr.Log.Debug("This is an example, handle: {0}", someHandle);
    }
}

In this example, even if the Eclr.Log call itself is used conditionally, the compiler would still generate a Local Root for the entire function, which it would then run through each cycle.

In order to optimize this, we could instead do the following:

private void SomeLogic()
{
    // some logic
    Eclr.Log.Debug("This is an example, handle: {0}", someHandle);
}
[Execution]
public void __Process()
{
    if(someCondition)
    {
        SomeLogic();
    }
}

In this example, the Local Root is only generated for the SomeLogic function and not for the __Process() method. Besides the usage of Eclr.Log, this also applies to strings, objects and try/catch.

Try to avoid creating strings – especially string formatting or other new() operations – each time the __Process() function is called.

Ideally, local strings, Eclr.Log calls, try/catch blocks and other local objects get moved to functions that are then only called conditionally (this might not always be possible).

Call-by-reference instead of call-by-value

To avoid unnecessary copying and heap allocations, functions should use call-by-reference instead of call-by-value.
A good concept explanation can be found at ScholarHat: Call by value and Call by reference in C#: An overview

Garbage Collection (GC)

A good explanation of how the GC works is available at ScholarHat: Introduction to Garbage Collection in C#.
(Note: The explanations refers to the Microsoft® .Net™ framework. The eCLR GC may behave slightly differently but the concepts are the same.)

Using references in loops

In loops, using references is especially important, as it's not always obvious that new objects are being created.

Example:

foreach(IPEndpoint ip in ipEndpointList)
{
    // Some logic
}

Here, each IPEndpoint is copied into a new ip variable from the list, which indirectly allocates heap memory.

A better alternative would be:

for(int i = 0; i < ipEndpointList.Count; i++)
{
    // Some logic
}

This uses a classic for loop to iterate through a list/array without copying data indirectly.

String operations

String operations usually generate garbage because temporary buffers are created in the background. In IEC 61131-3 context, functions like CONCAT or  TO_STRING are examples for this. Strings are handled directly in the context of the garbage collector and thus have less impact on real-time behavior than a typical new() in the managed context.

As mentioned before, append, concat or formatting operations are especially expensive. Use them selectively and only when absolutely needed.

Any changes to the string will result in a new string being created and the old string turning into garbage, even if this might not be apparent to the user.

File I/O and buffer operations

File I/O and buffer operations (such as used in the TLS_RECEIVE_2 FB or when reading/writing files) generate garbage when the connection or file stream is closed, because internally temporary buffers are used.

It is also wise to move these actions to a background thread, such as the System.Threading.ThreadPool.

Arrays and Lists

If buffers are used (for example, as byte arrays), try to reuse those buffers as much as possible without resizing or calling new[] to reset them. For elementary data types, or even for complex datatypes, try resetting the default values manually instead, as long as the array size remains the same.

While the convenience of List<> is great, at the end of the day it’s internally an array with a default of 4 elements.

The Capacity property increases by 2 each time the field's value is changed after the initial creation of the List.  Setting the Capacity property can be done manually by setting it, or simply by calling Add() on a List that has become too small to include the new element. Internally, a new array is created with the new size, and all elements have to be copied over. This is expensive, especially during runtime, so you better think ahead and set a fitting default Capacity property (ideally inside the constructor) ahead of time.

Frequent connection setup/teardown

Frequent connection setup/teardown calls (for example, via the TLS_SOCKET_2 FB) generate garbage. After each closed connection the unused send/receive buffer and base object gets turned into garbage.

Function blocks

Function blocks have both a constructor and an _Init() function. These are executed when the project is loaded and they are not subject to real-time monitoring. Therefore, new() can be called here without the garbage collector affecting real-time behavior. In many cases, this allows avoiding new() in the real-time context of a block.

As mentioned in other sections, the _Process() function is called every cycle. Be aware of what is inside it, and if needed, offload certain costly or memory-intensive operations to helper methods or background tasks such as ThreadPool.

Testing under stress conditions

FBs should be tested under stress conditions on the target controller, and CPU usage should be measured during testing. In PLCnext Engineer, the Logic Analyzer can help to identify large peaks of 1 ms or more. Additionally, ESM properties like LAST_EXEC_DURATION and MAX_EXEC_DURATION provide further information on how much load a function block generates.

Using Objects vs using Structs

It can be vital to choose the correct data type for the needed task. In C# we have access to both Objects and Structs, which both work differently, unlike in C++.

You can read more about it at these ressources:

• ByteHide: Struct vs Class in C#: Choosing the right data type
• Microsoft: Choosing between Class and Struct
• NDepend: Class vs Struct in C#: Making informed choices
• ZeroToMastery: Beginners guide to C# Struct vs Class (with code examples)

Note: These articles are all written for Microsoft®’s just-in-time (JIT) compilers in a non real-time context, so these use cases vary a bit.

In the context of the eCLR and real-time programming, allocating ahead of time or reusing already allocated memory can be very useful, which is where both data types can make a big difference if used right.

• Be aware of how your objects and data flow are being used, and choose the data type accordingly.

• Roughly said, if the data is simply part of or tied to an object, if it is small/short-lived and doesn’t need to be copied or moved, then make it a struct. As mentioned before, calling new() on a struct simply resets its fields and values.

• If data is moved or copied elsewhere, use an object, but try to use the new() operator deliberately and carefully. Ideally only create objects in the initialization phase. Try and reuse object instances as much as you can by simply manually resetting the fields, if it makes sense, instead of calling the new() operator on them.

• Make sure you're aware of when boxing occurs. Boxing is expensive and you should avoid it if you can.
  You can read more about boxing at Microsoft: Boxing and Unboxing (C# Programming Guide)

Threading and thread safety

We won't go into details here, as this is a huge topic in of itself and one can generally apply the lessons that are learned elsewhere here. We want to quickly touch upon a couple of points that might be relevant to the current state of our eCLR.

• The System.Threading.ThreadPool is the generally preferred class to utilize for background tasks, such as I/O or throwing away background workers, but there seem to still be performance-related issues associated to it. A fully stressed ThreadPool also currently increases CPU load. That said, this thread is DCG safe, unlike a custom-created ThreadPool – this will be fixed down the road.

• Don't create endless threads that don't do anything and are just sleeping until there is an occasional task that it should do based on a condition variable. Use the System.Threading.ThreadPool instead.

• The System.Threading.Interlocked class currently isn't implemented with actual atomic operations but simple "fast mutexes" – this will be corrected down the road. Currently this means that the obvious upsides of atomics can't really be achieved using Interlocked. It's something to keep in mind.

• While locks/System.Threading.Monitor are great, be aware that in certain situations, where you have more reads than writes, a System.Threading. ReaderWriterLockSlim might be the smarter/more performant choice. Also try and use the try/finally System.Threading.Monitor.TryEnter() pattern if you don't need immediate access, especially when used directly in the ESM context.

• Always test and benchmark your code, especially when using different types of locking mechanisms.

• Depending on what the thread is supposed to do, an input parameter for CPU affinity should be created for threads. By optionally specifying CPU affinity, the user would be able to distribute processing across different CPU cores on multi-core systems. You can pin .Net™ threads by using the eclrlib's Eclr.Environment.SetCPUAffinity method. We're also currently in the process of creating an example pattern on how to best achieve this.

Documenting FB functions

FB functions known to cause certain delays or CPU usage should be documented accordingly. FBs or libraries using other FBs or libraries that can cause high CPU usage or delays should be documented.

Providing examples

Whenever you develop a function block or a .Net™ library, it would be a good practice to also provide a basic usage/test case example project.

Down the road this helps QA, customers and colleagues that might need to help with debugging customer projects.