Pull to refresh

A small overview of SIMD in .NET/C#

Reading time 9 min
Views 17K

Here’s a quick look at algorithm vectorization capabilities in .NET Framework and .NET Core. This article is for those who know nothing about these techniques. I will also show that .NET doesn’t actually lag behind "real compiled" languages for native development.


I’m just starting to learn vectorization techniques. So, I will appreciate if community members find clear errors or suggest improvements to the described algorithms.


Some history


SIMD appeared in .NET Framework 4.6 in 2015. That's when Matrix3x2, Matrix4x4, Plane, Quaternion, Vector2, Vector3 and Vector4 types were added. They allowed vectorized computations. Next was the Vector<T> type that gave more opportunities to vectorize algorithms. However, many programmers were still dissatisfied as these types restricted coders’ idea streams and didn’t let use the full capacity of SIMD instructions in modern processors. Now in .NET Core 3.0 Preview, we have System.Runtime.Intrinsics namespace that gives much freedom in the choice of instructions. To get the most in speed you need to use RyuJit and resort either to x64 assembly or switch off Prefer 32-bit and choose AnyCPU assembly. I ran all the benchmarks on Intel Core i7-6700 3.40 GHz (Skylake) CPU computer.


Summing array elements


I decided to start with a classic task which usually comes first if there’s vectorization involved. It deals with finding the sum of array elements. Let’s write four implementations of this task to sum the elements of Array.


The most obvious implementation:


public int Naive() {
    int result = 0;
    foreach (int i in Array) {
        result += i;
    }
    return result;
}

LINQ-based implementation:


public long LINQ() => Array.Aggregate<int, long>(0, (current, i) => current + i);

The implementation based on vectors from System.Numerics:


public int Vectors() {
    int vectorSize = Vector<int>.Count;
    var accVector = Vector<int>.Zero;
    int i;
    var array = Array;
    for (i = 0; i <= array.Length - vectorSize; i += vectorSize) {
        var v = new Vector<int>(array, i);
        accVector = Vector.Add(accVector, v);
    }
    int result = Vector.Dot(accVector, Vector<int>.One);
    for (; i < array.Length; i++) {
        result += array[i];
    }
    return result;
}

The implementation based on code from System.Runtime.Intrinsics namespace:


public unsafe int Intrinsics() {
    int vectorSize = 256 / 8 / 4;
    var accVector = Vector256<int>.Zero;
    int i;
    var array = Array;
    fixed (int* ptr = array) {
        for (i = 0; i <= array.Length - vectorSize; i += vectorSize) {
            var v = Avx2.LoadVector256(ptr + i);
            accVector = Avx2.Add(accVector, v);
        }
    }
    int result = 0;
    var temp = stackalloc int[vectorSize];
    Avx2.Store(temp, accVector);
    for (int j = 0; j < vectorSize; j++) {
        result += temp[j];
    }   
    for (; i < array.Length; i++) {
        result += array[i];
    }
    return result;
}

I benchmarked those 4 methods on my computer and got the following results:


Method ItemsCount Mean Error StdDev Ratio
Naive 10 3.531 ns 0.0336 ns 0.0314 ns 1.00
LINQ 10 76.925 ns 0.4166 ns 0.3897 ns 21.79
Vectors 10 2.750 ns 0.0210 ns 0.0196 ns 0.78
Intrinsics 10 6.513 ns 0.0623 ns 0.0582 ns 1.84
Naive 100 47.982 ns 0.3975 ns 0.3524 ns 1.00
LINQ 100 590.414 ns 3.8808 ns 3.4402 ns 12.31
Vectors 100 10.122 ns 0.0747 ns 0.0699 ns 0.21
Intrinsics 100 14.277 ns 0.0566 ns 0.0529 ns 0.30
Naive 1000 569.910 ns 2.8297 ns 2.6469 ns 1.00
LINQ 1000 5,658.570 ns 31.7465 ns 29.6957 ns 9.93
Vectors 1000 79.598 ns 0.3498 ns 0.3272 ns 0.14
Intrinsics 1000 66.970 ns 0.3937 ns 0.3682 ns 0.12
Naive 10000 5,637.571 ns 37.5050 ns 29.2814 ns 1.00
LINQ 10000 56,498.987 ns 294.8776 ns 275.8287 ns 10.02
Vectors 10000 772.900 ns 2.6802 ns 2.5070 ns 0.14
Intrinsics 10000 579.152 ns 2.8371 ns 2.6538 ns 0.10
Naive 100000 56,352.865 ns 230.7916 ns 215.8826 ns 1.00
LINQ 100000 562,610.571 ns 3,775.7631 ns 3,152.9332 ns 9.98
Vectors 100000 8,389.647 ns 165.9590 ns 227.1666 ns 0.15
Intrinsics 100000 7,261.334 ns 89.6468 ns 69.9903 ns 0.13

It’s clear that solutions with Vectors and Intrinsics are much faster than the obvious and LINQ-based solutions. Now we need to figure out what goes on in these two methods.


Let’s consider Vectors method more closely:


Vectors
public int Vectors() {
    int vectorSize = Vector<int>.Count;
    var accVector = Vector<int>.Zero;
    int i;
    var array = Array;
    for (i = 0; i <= array.Length - vectorSize; i += vectorSize) {
        var v = new Vector<int>(array, i);
        accVector = Vector.Add(accVector, v);
    }
    int result = Vector.Dot(accVector, Vector<int>.One);
    for (; i < array.Length; i++) {
        result += array[i];
    }
    return result;
}

  • int vectorSize = Vector<int>.Count; — the amount of 4-byte numbers we can place in a vector. If hardware acceleration is used, this value shows how many 4-byte numbers we can put in one SIMD register. In fact, it shows how many elements of this type can be handled concurrently;
  • accVector is a vector that accumulates the result of the function;
  • var v = new Vector<int>(array, i); — the data from array is loaded into a new v vector, starting from i index. The vectorSize of data will be loaded exactly;
  • accVector = Vector.Add(accVector, v); — two vectors are summed.
    For example, there are 8 numbers in Array: {0, 1, 2, 3, 4, 5, 6, 7} and vectorSize == 4.
    Then during the first cycle iteration accVector = {0, 0, 0, 0}, v = {0, 1, 2, 3} and after addition accVector will hold: {0, 0, 0, 0} + {0, 1, 2, 3} = {0, 1, 2, 3}.
    During the second iteration v = {4, 5, 6, 7} and after addition accVector = {0, 1, 2, 3} + {4, 5, 6, 7} = {4, 6, 8, 10}.
  • Now we just need to get the sum of all vector elements. To do this we can use scalar multiplication by a vector filled with ones: int result = Vector.Dot(accVector, Vector<int>.One);
    Then we get: {4, 6, 8, 10} * {1, 1, 1, 1} = 4 * 1 + 6 * 1 + 8 * 1 + 10 * 1 = 28.
  • If necessary, those numbers that don’t fit the last vector will be summed at the end.

Let's look into Intrinsics code:


Intrinsics
public unsafe int Intrinsics() {
    int vectorSize = 256 / 8 / 4;
    var accVector = Vector256<int>.Zero;
    int i;
    var array = Array;
    fixed (int* ptr = array) {
        for (i = 0; i <= array.Length - vectorSize; i += vectorSize) {
            var v = Avx2.LoadVector256(ptr + i);
            accVector = Avx2.Add(accVector, v);
        }
    }
    int result = 0;
    var temp = stackalloc int[vectorSize];
    Avx2.Store(temp, accVector);
    for (int j = 0; j < vectorSize; j++) {
        result += temp[j];
    }   
    for (; i < array.Length; i++) {
        result += array[i];
    }
    return result;
}

We can see that it’s like Vectors with one exception:


  • vectorSize is specified by a constant. This is because this method explicitly uses Avx2 instructions that operate with 256-bit registers. A real application should include a check of whether a current processor supports Avx2. If not, another code should be called. It looks like this:
    if (Avx2.IsSupported) {
    DoThingsForAvx2();
    }
    else if (Avx.IsSupported) {
    DoThingsForAvx();
    }
    ...
    else if (Sse2.IsSupported) {
    DoThingsForSse2();
    }
    ...
  • var accVector = Vector256<int>.Zero; accVector is declared as 256-bit vector filled with zeros.
  • fixed (int* ptr = Array) — the pointer to the array is placed in ptr.
  • Next are the same operations as in Vectors: loading data into a vector and addition of two vectors.
  • The summing of vector elements uses the following method:
    • create an array on stack: var temp = stackalloc int[vectorSize];
    • load a vector into this array: Avx2.Store(temp, accVector);
    • sum array elements during the cycle.
  • Next, the elements which don’t fit the last vector are summed up.

Comparing two arrays


We need to compare two arrays of bytes. It is exactly this task that made me study SIMD in .NET. Again, let’s write several methods for benchmarking and compare two arrays: ArrayA and ArrayB.


The most obvious solution:


public bool Naive() {
    for (int i = 0; i < ArrayA.Length; i++) {
        if (ArrayA[i] != ArrayB[i]) return false;
    }
    return true;
}

LINQ-based solution:


public bool LINQ() => ArrayA.SequenceEqual(ArrayB);

The solution based on MemCmp function:


[DllImport("msvcrt.dll", CallingConvention = CallingConvention.Cdecl)]
static extern int memcmp(byte[] b1, byte[] b2, long count);

public bool MemCmp() => memcmp(ArrayA, ArrayB, ArrayA.Length) == 0;

The solution based on vectors from System.Numerics:


public bool Vectors() {
    int vectorSize = Vector<byte>.Count;
    int i = 0;
    for (; i <= ArrayA.Length - vectorSize; i += vectorSize) {
        var va = new Vector<byte>(ArrayA, i);
        var vb = new Vector<byte>(ArrayB, i);
        if (!Vector.EqualsAll(va, vb)) {
            return false;
        }
    }
    for (; i < ArrayA.Length; i++) {
        if (ArrayA[i] != ArrayB[i])
            return false;
    }
    return true;
}

Intrinsics-based solution:


public unsafe bool Intrinsics() {
    int vectorSize = 256 / 8;
    int i = 0;
    const int equalsMask = unchecked((int) (0b1111_1111_1111_1111_1111_1111_1111_1111));
    fixed (byte* ptrA = ArrayA)
    fixed (byte* ptrB = ArrayB) {
        for (; i <= ArrayA.Length - vectorSize; i += vectorSize) {
            var va = Avx2.LoadVector256(ptrA + i);
            var vb = Avx2.LoadVector256(ptrB + i);
            var areEqual = Avx2.CompareEqual(va, vb);
            if (Avx2.MoveMask(areEqual) != equalsMask) {
                return false;
            }
        }
        for (; i < ArrayA.Length; i++) {
            if (ArrayA[i] != ArrayB[i])
                return false;
        }
        return true;
    }
}

The results of running the benchmark on my computer:


Method ItemsCount Mean Error StdDev Ratio
Naive 10000 7,033.8 ns 50.636 ns 47.365 ns 1.00
LINQ 10000 64,841.4 ns 289.157 ns 270.478 ns 9.22
Vectors 10000 504.0 ns 2.406 ns 2.251 ns 0.07
MemCmp 10000 368.1 ns 2.637 ns 2.466 ns 0.05
Intrinsics 10000 283.6 ns 1.135 ns 1.061 ns 0.04
Naive 100000 85,214.4 ns 903.868 ns 845.478 ns 1.00
LINQ 100000 702,279.4 ns 2,846.609 ns 2,662.720 ns 8.24
Vectors 100000 5,179.2 ns 45.337 ns 42.409 ns 0.06
MemCmp 100000 4,510.5 ns 24.292 ns 22.723 ns 0.05
Intrinsics 100000 2,957.0 ns 11.452 ns 10.712 ns 0.03
Naive 1000000 844,006.1 ns 3,552.478 ns 3,322.990 ns 1.00
LINQ 1000000 6,483,079.3 ns 42,641.040 ns 39,886.455 ns 7.68
Vectors 1000000 54,180.1 ns 357.258 ns 334.180 ns 0.06
MemCmp 1000000 49,480.1 ns 515.675 ns 457.133 ns 0.06
Intrinsics 1000000 36,633.9 ns 680.525 ns 636.564 ns 0.04

I guess the code of these methods is clear, except for two lines in Intrinsics:


var areEqual = Avx2.CompareEqual(va, vb);
if (Avx2.MoveMask(areEqual) != equalsMask) {
    return false;
}

In the first line two vectors are compared for equality and the result is saved in areEqual vector in which all bits in the element at a particular position are set to 1 if the corresponding elements in va and vb are equal. So, it turns out that if byte vectors va and vb are equal, all the elements in areEquals should equal to 255 (11111111b). As Avx2.CompareEqual is a wrapper over _mm256_cmpeq_epi8, we can go to Intel website and see the pseudocode of this operation:
MoveMask method makes a 32-bit number from a vector. The top bits of each 32 onebyte elements in a vector are the values of bits in the MoveMask result. The pseudocode is available here.


Thus, if some bytes in va and vb don’t match, the corresponding bytes in areEqual will be 0. Therefore, the top bits of these bytes will be 0 too. This means the corresponding bits in Avx2.MoveMask response will also be 0 and areEqual will not equal to equalsMask.


Let’s look at one example assuming that vector length is 8 bytes (to write less):


  • Let va = {100, 10, 20, 30, 100, 40, 50, 100} and vb = {100, 20, 10, 30, 100, 40, 80, 90}.
  • Then areEquals will be {255, 0, 0, 255, 255, 255, 0, 0}.
  • The MoveMask method will return 10011100b that should be compared with 11111111b mask. As these masks are not equal, va and vb vectors are not equal too.

Counting the times an element occurs in a collection.


Sometimes you need to count the occurrences of a particular element, e.g. integers, in a collection. We can speed up this algorithm too. For comparison let’s write several methods to search Item element in Array.


The most obvious one:


public int Naive() {
    int result = 0;
    foreach (int i in Array) {
        if (i == Item) {
            result++;
        }
    }
    return result;
}

Using LINQ:


public int LINQ() => Array.Count(i => i == Item);

Using vectors from System.Numerics.Vectors:


public int Vectors() {
    var mask = new Vector<int>(Item);
    int vectorSize = Vector<int>.Count;
    var accResult = new Vector<int>();
    int i;
    var array = Array;
    for (i = 0; i <= array.Length - vectorSize; i += vectorSize) {
        var v = new Vector<int>(array, i);
        var areEqual = Vector.Equals(v, mask);
        accResult = Vector.Subtract(accResult, areEqual);
    }
    int result = 0;
    for (; i < array.Length; i++) {
        if (array[i] == Item) {
            result++;
        }
    }
    result += Vector.Dot(accResult, Vector<int>.One);
    return result;
}

Using Intrinsics:


public unsafe int Intrinsics() {
    int vectorSize = 256 / 8 / 4;
    var temp = stackalloc int[vectorSize];
    for (int j = 0; j < vectorSize; j++) {
        temp[j] = Item;
    }
    var mask = Avx2.LoadVector256(temp);
    var accVector = Vector256<int>.Zero;
    int i;
    var array = Array;
    fixed (int* ptr = array) {
        for (i = 0; i <= array.Length - vectorSize; i += vectorSize) {
            var v = Avx2.LoadVector256(ptr + i);
            var areEqual = Avx2.CompareEqual(v, mask);
            accVector = Avx2.Subtract(accVector, areEqual);
        }
    }
    int result = 0;
    Avx2.Store(temp, accVector);
    for(int j = 0; j < vectorSize; j++) {
        result += temp[j];
    }
    for(; i < array.Length; i++) {
        if (array[i] == Item) {
            result++;
        }
    }
    return result;
}

The results of running the benchmark on my computer:


Method ItemsCount Mean Error StdDev Ratio
Naive 10 8.844 ns 0.0772 ns 0.0603 ns 1.00
LINQ 10 87.456 ns 0.9496 ns 0.8883 ns 9.89
Vectors 10 3.140 ns 0.0406 ns 0.0380 ns 0.36
Intrinsics 10 13.813 ns 0.0825 ns 0.0772 ns 1.56
Naive 100 107.310 ns 0.6975 ns 0.6183 ns 1.00
LINQ 100 626.285 ns 5.7677 ns 5.3951 ns 5.83
Vectors 100 11.844 ns 0.2113 ns 0.1873 ns 0.11
Intrinsics 100 19.616 ns 0.1018 ns 0.0903 ns 0.18
Naive 1000 1,032.466 ns 6.3799 ns 5.6556 ns 1.00
LINQ 1000 6,266.605 ns 42.6585 ns 39.9028 ns 6.07
Vectors 1000 83.417 ns 0.5393 ns 0.4780 ns 0.08
Intrinsics 1000 88.358 ns 0.4921 ns 0.4603 ns 0.09
Naive 10000 9,942.503 ns 47.9732 ns 40.0598 ns 1.00
LINQ 10000 62,305.598 ns 643.8775 ns 502.6972 ns 6.27
Vectors 10000 914.967 ns 7.2959 ns 6.8246 ns 0.09
Intrinsics 10000 931.698 ns 6.3444 ns 5.9346 ns 0.09
Naive 100000 94,834.804 ns 793.8585 ns 703.7349 ns 1.00
LINQ 100000 626,620.968 ns 4,696.9221 ns 4,393.5038 ns 6.61
Vectors 100000 9,000.827 ns 179.5351 ns 192.1005 ns 0.09
Intrinsics 100000 8,690.771 ns 101.7078 ns 95.1376 ns 0.09
Naive 1000000 959,302.249 ns 4,268.2488 ns 3,783.6914 ns 1.00
LINQ 1000000 6,218,681.888 ns 31,321.9277 ns 29,298.5506 ns 6.48
Vectors 1000000 99,778.488 ns 1,975.6001 ns 4,252.6877 ns 0.10
Intrinsics 1000000 96,449.350 ns 1,171.8067 ns 978.5116 ns 0.10

Vectors and Intrinsics methods completely coincide in logic but differ in the implementation of particular operations. The idea is the following:


  • create mask vector in which a required number is stored in each element;
  • load the part of an array in v vector and compare this part with a mask. As a result, all bits will set in equal elements of areEqual. As areEqual is an array of integers, then if we set all the bits of one element, we will get -1 in this element ((int)(1111_1111_1111_1111_1111_1111_1111_1111b) == -1);
  • subtract areEqual vector from accVector. Then, accVector will hold the count of how many times the item element occurred in all v vectors for each position (minus by minus is a plus).

The whole code from the article is on GitHub.


Conclusion


I described only a small part of .NET capabilities for computation vectorization. To see the full updated list of all intrinsics available in .NET Core under x86, turn to the source code. It’s convenient that the summary of each intrinsic in C# files contains its name in C world. This helps either to understand the purpose of this intrinsic and the transfer of existing C++/C algorithms to .NET. System.Numerics.Vector documentation is available on msdn.


I think .NET has a great advantage over C++. As JIT compilation already occurs on a client machine, a compiler can optimize code for a particular client processor, giving maximum performance. At the same time, a programmer can stay within one language and the same technologies to write fast code.

Tags:
Hubs:
+6
Comments 0
Comments Leave a comment

Articles