Примерно полгода назад я познакомился с VLIW‑процессором Эльбрус-8СВ. На тот момент у меня уже был опыт написания кода на ассемблере для VLIW‑процессора TMS320C66. Поэтому я захотел написать нечто похожее для Эльбруса. А именно, реализовать алгоритм FFT на ассемблере. Но из‑за нехватки документации на инструкции процессора мне пришлось начать с реализации какого‑нибудь простого алгоритма на Си, чтобы изучать его ассемблерный вывод. По результатам той деятельности была написана предыдущая статья.
После написания той статьи я решил попробовать реализовать алгоритм FFT на Си для Эльбруса. Работа ещё не завершена, но определённые успехи уже есть (сравнение с EML присутствует). В этой статье я хочу поделиться полученными на данный момент результатами.
Содержание:
Пишем функцию Reverse
reverse_radix2
Пишем функцию Stage
stage_radix2
stage_radix2_2x
stage_radix2_readConjSwap
stage_radix2_readConjSwap_2x
stage_radix4
stage_radix4_2x
stage_radix4_readConjSwap
Собираем FFT
Постановка задачи
Дано: указатель на входной массив комплексных чисел, количество элементов в нём и указатель на выходной массив.
Требуется: вычислить FFT от входных данных, ответ записать в выходной массив.
Будем считать, что комплексные числа имеют типfloat complex(то есть действительная и мнимая части имеют типfloat).
Будем считать, что количество элементов массива является степенью числа 2 (или 4, если надо будет).
И, конечно же, пусть массивы будут выровнены в памяти на удобные нам границы.
Ключи компиляции
При компиляции использовались следующие ключи lcc:
-Wall -O3 -faligned -ffast-math -march=elbrus-v5
Почему я компилирую для elbrus-v5
После написания предыдущей статьи у меня пропал доступ к elbrus‑v5 (он уехал в ремонт). Сначала я придумывал алгоритмы «на бумажке», потом мне был предоставлен доступ к elbrus‑v6. Я планировал делать код для elbrus‑v5, поэтому добавил в скрипт компиляции ключ ‑march=elbrus‑v5. В дальнейшем доступ к v5 восстановился, но я уже привык работать на v6, так как он был круглосуточно доступен.
Перед написанием этой статьи я решил убрать ключ -march=elbrus-v5. После перекомпиляции некоторый код стал работать медленнее, чем при наличии ключа. Выяснилось, что этот код компилируется для v5 эффективнее, чем для v6 (плотнее упакованы инструкции). Получилось так, что код, скомпилированный для v5, работает на v6 быстрее, чем тот же код, скомпилированный для v6.
Такое поведение можно наблюдать в следующих функциях:
Поэтому я пока остановился на компиляции для v5.
Посмотреть на различия компиляции можно на сайте ce.mentality.rip:
Вставляем в левое поле одну из указанных выше функций
Добавляем перед вставленной функцией такой код:
#include <e2kintrin.h> #include <stdint.h> typedef struct { float real; float imag; } myComplex;В строке «Compiler options» указываем ключи компиляции:
-Wall -O3 -faligned -ffast-math -march=elbrus-v5
После этого считаем количество тактов в цикле (делаем поиск по «loop_mode»).
При указании -march=elbrus-v5 тактов меньше, чем при -march=elbrus-v6.
Как измерялось время
Замеры времени делались с помощью функции clock_gettime():
struct timespec t0, t1; clock_gettime(CLOCK_REALTIME, &t0); /*** здесь измеряемый код ***/ clock_gettime(CLOCK_REALTIME, &t1); int usec = (t1.tv_sec - t0.tv_sec)*1000000 + (t1.tv_nsec - t0.tv_nsec)/1000;
Также использовалось чтение счётчика тактов процессора:
uint64_t get_clock_count() { uint64_t dst; #pragma asm_inline asm ("rrd %%clkr, %0" : "=r" (dst)); return dst; } ... uint64_t ticks0 = get_clock_count(); /*** здесь измеряемый код ***/ uint64_t ticks1 = get_clock_count(); uint64_t ticks = ticks1 - ticks0;
Про опции pragma
В этот раз я использовал #pragma prefetch вместо #pragma loop_count(100).
Как я понял, оба варианта включают APB. Разница в том, что prefetchне отключает предварительный запрос данных перед циклом, а loop_count отключает. Этот запрос немного уменьшает время исполнения цикла. При большом числе итераций это ускорение несущественно, но увеличивает размер кода.
Разницы почти нет, но prefetch выглядит проще, чем loop_count(100).
Также использовалась опция #pragma ivdep, которая указывает компилятору, что разные итерации цикла независимы между собой по обращениям к памяти, и можно начать выполнение следующей итерации, не дожидаясь завершения текущей.
Опция полезна, когда в цикле одновременно присутствуют чтение и запись в память.
Про раскрутку циклов
Иногда для ускорения полезно сделать раскрутку циклов. В одном такте могут одновременно выполняться 6 инструкций, потому что в этом процессоре есть 6 исполнительных юнитов. Если, например, итерация цикла состоит из 9 инструкций, то она будет выполняться за 2 такта, а две такие итерации, соответственно, за 4 такта. После раскрутки в 2 раза одна итерация нового цикла будет состоять из 2*9=18 инструкций, и можно ожидать, что она будет выполняться за 3 такта. Таким образом, две итерации исходного цикла будут выполняться не за 4 такта, а за 3. Но следует помнить, что по разным причинам не всегда удаётся уместить в каждый такт 6 инструкций. Например, потому что некоторые инструкции способны выполняться не на всех исполнительных юнитах.
Способы раскрутить цикл:
Раскрутка цикла компилятором (у таких функций я добавляю к названию суффикс
"_unroll2"/"_unroll3"/"_unroll4")Для её использования надо написать
#pragma unroll(k)перед циклом, где k — множитель раскрутки. И компилятор раскрутит цикл ровно в k раз. Также будет добавлен код, проверяющий кратность числа итераций исходного цикла параметру k. Если число итераций не кратно k, остаток будет обработан отдельным кодом. Раскрученный код будет выполнять те же действия и в том же порядке, что и изначальный код. Если поставить#pragma unroll(1), то раскрутка не будет производиться. Это бывает полезно, потому что по‑умолчанию компилятор пытается сделатьunroll(2).Ручная раскрутка цикла программистом (у таких функций я добавляю к названию суффикс
"_x2"/"_x3"/"_x4")При ручной раскрутке программист сам пишет код так, чтобы в одной итерации цикла выполнялось k итераций алгоритма. Проверка кратности числа итераций алгоритма величине k лежит на программисте. Если число итераций может быть не кратно k, нужно обрабатывать эти случаи отдельно. В приведённом в этой статье коде такие случаи не обрабатываются для упрощения. В отличие от раскрутки компилятором, полностью повторяющей все действия в исходном порядке, раскрученный вручную код можно оптимизировать, заменяя действия на другие и меняя их порядок.
Интересным вариантом является ручная раскрутка цикла, тело которого состоит из другого цикла, обрабатывающего один и тот же массив. В этом случае можно объединить k итераций внутреннего цикла. Например, если исходная итерация внутреннего цикла состоит из чтения данных из памяти, обработки данных и сохранения результата в память, то после раскрутки внешнего цикла в 2 раза можно убрать сохранение результата в конце первой итерации и чтение его обратно перед второй. В итоге останется просто чтение, обработка, ещё раз обработка и сохранение. Пример такой оптимизации находится в функциях с пометкой
"2x", например,stage_radix2_2x(не"x2"для лучшей сортировки файлов по имени).
Что такое FFT
Пусть дан набор комплексных чисел x0, …, xN-1.
Дискретным преобразованием Фурье (DFT) называется перевод этого набора в другой набор комплексных чисел X0, …, XN-1 по следующей формуле:
Вычисление X0, …, XN-1 по этой формуле требует O(N2) операций.
Быстрым преобразованием Фурье (FFT) называется алгоритм, позволяющий вычислить DFT значительно быстрее (обычно за O(N logN) операций).
Наиболее известным FFT является алгоритм Кули‑Тьюки (Cooley‑Tukey). Этот алгоритм рекурсивно вычисляет DFT через DFT меньшего размера (метод «разделяй и властвуй» / «divide‑and‑conquer»).
Например, если поделить исходный массив на два подмассива (вариант «radix-2»), исходную формулу можно преобразовать к такому виду:
Et — DFT от N/2 чётных (Even) элементов исходного массива (с индексами вида 2s+0).
Ot — DFT от N/2 нечётных (Odd) элементов исходного массива (с индексами вида 2s+1).
Вариант «radix-4»
По аналогии вариант «radix-4» преобразует формулу к такому виду:
EEt — DFT от N/4 элементов исходного массива с индексами вида 4s+0.
EOt — DFT от N/4 элементов исходного массива с индексами вида 4s+1.
OEt — DFT от N/4 элементов исходного массива с индексами вида 4s+2.
OOt — DFT от N/4 элементов исходного массива с индексами вида 4s+3.
Напишем псевдокод рекурсивной функции, реализующей вариант «radix-2»:
FFT(IN) // IN - набор входных данных { N = IN.length if N == 1 return IN E = FFT(IN[0:N:2]) // FFT от чётных элементов O = FFT(IN[1:N:2]) // FFT от нечётных элементов for t = 0:N/2 { c = e^(-2*pi*i*t/N) OUT[t ] = E[t] + c*O[t] OUT[t + N/2] = E[t] - c*O[t] } return OUT }
Напишем чуть ближе к реальному коду:
FFT(*IN, N, s) // IN - указатель на первый элемент // N - количество элементов // s - расстояние между элементами { if N == 1 return IN[0] E[0:N/2] = FFT(IN, N/2, 2s) // FFT от чётных элементов O[0:N/2] = FFT(IN + s, N/2, 2s) // FFT от нечётных элементов for t = 0:N/2 { c = e^(-2*pi*i*t/N) OUT[t ] = E[t] + c*O[t] OUT[t + N/2] = E[t] - c*O[t] } return OUT[0:N] }
Разместим элементы Et в первой половине OUT, а элементы Ot во второй половине OUT:
FFT(*IN, N, s) // IN - указатель на первый элемент // N - количество элементов // s - расстояние между элементами { if N == 1 return IN[0] OUT[0:N/2] = FFT(IN, N/2, 2s) // FFT от чётных элементов OUT[N/2:N] = FFT(IN + s, N/2, 2s) // FFT от нечётных элементов for t = 0:N/2 { x = OUT[t ] y = OUT[t + N/2] c = e^(-2*pi*i*t/N) OUT[t ] = x + c*y OUT[t + N/2] = x - c*y } return OUT[0:N] }
Для удобства реализации перед рекурсивными вызовами можно передвинуть все чётные элементы в начало массива, нечётные — в конец массива. Тогда подмассивы будут занимать подряд идущие ячейки памяти, а не перемежаться между собой.
Если количество элементов N = 2n, то данная перестановка равнозначна перестановке вида IN[k] → OUT[rotateRight(k, n)], где rotateRight(k, n) — операция, «прокручивающая вправо» младшие n битов числа k, т.е. переставляющая младший бит числа k с позиции 0 на позицию n-1, сдвигая биты с позиций n-1, …, 1 на одну позицию в сторону позиции 0.
Псевдокод станет таким:
FFT(*IN, N) // IN - указатель на первый элемент // N - количество элементов { if N == 1 return IN[0] // перестановка "чётные - в начало, нечётные - в конец" IN = Even2Beginning_Odd2Ending(IN, N) OUT[0:N/2] = FFT(IN, N/2) // FFT от начальной половины массива OUT[N/2:N] = FFT(IN + N/2, N/2) // FFT от конечной половины массива for t = 0:N/2 { x = OUT[t ] y = OUT[t + N/2] c = e^(-2*pi*i*t/N) OUT[t ] = x + c*y OUT[t + N/2] = x - c*y } return OUT[0:N] }
Если теперь мысленно проследить за ходом рекурсии, можно увидеть, что к моменту достижения самой глубины рекурсивных вызовов все элементы исходного массива будут переставлены в порядке, который обычно называется «bit reversal». В этой статье я буду называть такую перестановку просто «Reverse».
Что такое Reverse
Перестановка элементов вида IN[k] → OUT[reverseNumber(k)], где reverseNumber(k) — операция, переставляющая биты числа k в обратном порядке.
Результат reverseNumber(k) зависит от количества битов в числе k.
Например:
- если битов 3, то reverseNumber(6) = 3 ( 110 → 011 )
- если битов 4, то reverseNumber(6) = 6 ( 0110 → 0110 )
- если битов 5, то reverseNumber(6) = 12 (00110 → 01100)
В случае «radix-4» мы придём к аналогичной перестановке, где потребуется вместо двоичных цифр (битов) переставлять четверичные цифры (пары битов).
Таким образом, движение по рекурсии вглубь можно заменить на Reverse.
Обратный путь рекурсии состоит из обработки подмассивов, размер которых увеличивается по мере возврата из рекурсии. Один шаг возврата из рекурсии, обрабатывающий все подмассивы сразу, будем называть «Stage».
Stage(*OUT, N, stage_num) { m = 2^stage_num // m - размер подмассива for k = 0:N/m // k - номер подмассива { for t = 0:m/2 { x = OUT[m*k + t ] y = OUT[m*k + t + m/2] c = e^(-2*pi*i*t/N) OUT[m*k + t ] = x + c*y OUT[m*k + t + m/2] = x - c*y } } }
Псевдокод алгоритма теперь выглядит так (без рекурсии):
FFT(*IN, N, *OUT) { OUT = Reverse(IN, N) for stage_num = 1, ..., log2(N) Stage(OUT, N, stage_num) }
Таким образом, FFT = Reverse + log2(N)*Stage.
Для решения задачи нужно написать две функции: Reverse и Stage.
Особенности реализации FFT на Эльбрусе
В процессоре Эльбрус есть механизм APB, который позволяет быстро читать данные, расположенные в памяти с равным шагом. Число потоков чтения в APB ограничено 32 штуками.
В последнем варианте алгоритма (без рекурсии) разные Stage обрабатывают подмассивы разного размера:
первый Stage обрабатывает подмассивы длиной 2 (это можно представить в виде 2-х потоков чтения с равным шагом: чётные и нечётные элементы)
второй Stage обрабатывает подмассивы длиной 4 (это можно представить в виде 4-х потоков чтения с равным шагом)
третий Stage обрабатывает подмассивы длиной 8 (это можно представить в виде 8-ми потоков чтения с равным шагом)
и так далее
При достаточно большом количестве Stage перестанет хватать потоков чтения APB. Для эффективного использования APB модифицируем алгоритм Stage таким образом, чтобы элементы всегда читались парами (как сейчас в первом Stage).
Получится такой вариант для Эльбруса:
Stage(*IN, N, *OUT, stage_num) { m = 2^stage_num for k = 0:m/2 { c = e^(-2*pi*i*k/m) for t = 0:N/m { x = IN[2*t ] y = IN[2*t + 1] OUT[t ] = x + c*y OUT[t + N/2] = x - c*y } } }
Если мысленно проследить за ходом операций, можно увидеть, что в этом варианте на каждом Stage выполняются те же арифметические операции с теми же парами чисел и теми же коэффициентами, что и в классическом варианте. Просто эти пары чисел обрабатываются в другом порядке (на всех Stage, кроме первого). И в конце каждого Stage добавляется перестановка «чётные — в начало, нечётные — в конец». Как было написано выше, эта перестановка равнозначна перестановке вида IN[k] → OUT[rotateRight(k, n)], поэтому после прохода по всем Stage числа возвращаются на исходные позиции (прокрутка делается log2(N) раз).
Как выглядит Stage для версии «radix-4»
Именно этот вариант будет реализован в данной статье.
Замечание про коэффициенты в функциях Stage
Для вычисления Stage нам нужны входные данные и коэффициенты. Входные данные располагаются в памяти изначально. Коэффициенты же можно либо читать из памяти одновременно с входными данными (вычислять заранее), либо вычислять на ходу.
Вариант чтения коэффициентов из памяти хорошо работает в случае малого числа коэффициентов, когда они все помещаются в кэше.
Вариант вычисления на ходу оправдан при большом числе коэффициентов, так как в этом случае тонким место является канал доступа к памяти и отказ от чтения коэффициентов из памяти позволяет использовать весь канал только для чтения входных данных. Общий размер коэффициентов (в зависимости от алгоритма) может в несколько раз превосходить размер входных данных. Поэтому отказ от чтения коэффициентов из памяти может увеличить скорость чтения входных данных в несколько раз. При малом числе коэффициентов вычисление на ходу будет замедлять работу, так как оно требует дополнительных инструкций для собственно вычисления.
В приведенном в этой статье коде реализован только вариант чтения коэффициентов из памяти.
Пишем функцию Reverse
reverse_radix2
1. reverse_radix2_etalon
Эталонный вариант для сравнения на корректность.
Здесь reverseNumber вычисляется с помощью цикла.
int reverseNumber_radix2(int number, int bit_count) { int answer = 0; for(int i = 0; i < bit_count; ++i) { answer <<= 1; answer |= number & 1; number >>= 1; } return answer; } void reverse_radix2_etalon(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; for(int64_t i = 0; i < count; ++i) { int index = reverseNumber_radix2(i, bit_count); data_out[index] = data_in[i]; } }
2. reverse_radix2
В процессоре есть инструкция bitrevd, которая производит операцию reverseNumber_radix2 над 64-битным числом. Заменим reverseNumber_radix2() на __builtin_e2k_bitrevd().
Схема перемещения данных в памяти
Код на Си
void reverse_radix2(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count; ++i) { int64_t index = __builtin_e2k_bitrevd(i) >> shift; data_out[index] = data_in[i]; } }
Основной цикл на ассемблере
.L1554: { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=5, abs=0, disp=0 } .L1385: { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END bitrevd,0,sm %b[9], %b[1] addd,2,sm %b[9], 0x1, %b[7] shrd,3,sm %b[5], %r0, %b[10] shld,4,sm %b[12], 0x3, %b[11] std,5 %r2, %b[13], %b[8] movad,1 area=0, ind=0, am=1, be=0, %b[0] }
Теоретическая скорость: 1 комплексное число за 1 такт (1/1) = 8 Байт/такт
Замеры скорости
3. reverse_radix2_x2_bad
Попробуем ускорить с помощью ручной раскрутки цикла в 2 раза.
Схема перемещения данных в памяти
Код на Си
void reverse_radix2_x2_bad(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; myComplex *data_out_0 = &data_out[0]; myComplex *data_out_1 = &data_out[count/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count; i += 2) { int64_t index = __builtin_e2k_bitrevd(i) >> shift; data_out_0[index] = data_in[i + 0]; data_out_1[index] = data_in[i + 1]; } }
Основной цикл на ассемблере
.L1860: { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } .L1655: { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END bitrevd,0,sm %b[20], %b[12] addd,1,sm %b[20], 0x2, %b[18] std,2 %r2, %b[17], %b[10] shrd,3,sm %b[16], %r4, %b[19] shld,4,sm %b[21], 0x3, %b[13] std,5 %r0, %b[17], %b[11] movad,0 area=0, ind=0, am=0, be=0, %b[0] movad,1 area=0, ind=8, am=1, be=0, %b[1] }
Теоретическая скорость: 2 комплексных числа за 1 такт (2/1) = 16 Байт/такт
Замеры скорости
Видим ускорение в начале графика.
Здесь происходит два чтения из одного места памяти и запись в два разных места памяти.
Вероятно, отсутствие ускорения по всей длине графика связано с тем, что запись в память всё равно делается по очереди (в один банк памяти?).
4. reverse_radix2_x2_good
Попробуем сделать наоборот: будем читать из двух разных мест, а писать рядом.
Схема перемещения данных в памяти
Код на Си
void reverse_radix2_x2_good(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; myComplex *data_in_0 = &data_in[0]; myComplex *data_in_1 = &data_in[count/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count/2; ++i) { int64_t index = __builtin_e2k_bitrevd(i) >> shift; data_out[index + 0] = data_in_0[i]; data_out[index + 1] = data_in_1[i]; } }
Основной цикл на ассемблере
.L2162: { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=5, abs=0, disp=0 } .L1993: { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END bitrevd,0,sm %b[20], %b[12] addd,1,sm %b[20], 0x1, %b[18] std,2 %b[17], %r2, %b[11] shrd,3,sm %b[16], %r4, %b[19] shld,4,sm %b[21], 0x3, %b[13] std,5 %r0, %b[17], %b[10] movad,1 area=0, ind=0, am=1, be=0, %b[1] movad,3 area=0, ind=0, am=1, be=0, %b[0] }
Теоретическая скорость: 2 комплексных числа за 1 такт (2/1) = 16 Байт/такт
Замеры скорости
Видим желаемое ускорение по всей длине графика.
Строго говоря, это не раскрутка цикла. Честной раскруткой был предыдущий вариант. Здесь же произошло изменение алгоритма (данные обрабатываются в другом порядке). Но я не придумал, как это назвать («stream2»?), поэтому все дальнейшие «раскрутки» будут называться x4/x8 и т.д.
5. reverse_radix2_x2_best
Прежде, чем переходить к более сильным раскруткам, посмотрим, что будет, если вместо двух 64-битных записей в память сделать одну 128-битную запись.
Схема перемещения данных в памяти
Код на Си
void reverse_radix2_x2_best(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; uint64_t *data_in_0 = (uint64_t*)&data_in[0]; uint64_t *data_in_1 = (uint64_t*)&data_in[count/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count/2; ++i) { int64_t offset = 8 * (__builtin_e2k_bitrevd(i) >> shift); *(__v2du*)((void*)data_out + offset) = (__v2du){data_in_0[i], data_in_1[i]}; } }
Основной цикл на ассемблере
.L2350: { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=5, abs=0, disp=0 } .L2262: { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END bitrevd,0,sm %b[22], %b[21] qppackdl,1,sm %b[10], %b[11], %b[13] addd,2,sm %b[22], 0x1, %b[20] shrd,3,sm %b[25], %r0, %b[24] shld,4,sm %b[26], 0x3, %b[27] stqp,5 %r2, %b[29], %b[19] movad,1 area=0, ind=0, am=1, be=0, %b[1] movad,3 area=0, ind=0, am=1, be=0, %b[0] }
Теоретическая скорость: 2 комплексных числа за 1 такт (2/1) = 16 Байт/такт
Замеры скорости
Видим небольшое ускорение.
В дальнейшем будем всегда писать в память 128-битными кусками.
6. reverse_radix2_x4
Сделаем аналогичную «псевдо раскрутку» теперь в 4 раза.
Схема перемещения данных в памяти
Код на Си
void reverse_radix2_x4(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; uint64_t *data_in_00 = (uint64_t*)&data_in[0 * count/4]; uint64_t *data_in_01 = (uint64_t*)&data_in[1 * count/4]; uint64_t *data_in_10 = (uint64_t*)&data_in[2 * count/4]; uint64_t *data_in_11 = (uint64_t*)&data_in[3 * count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count/4; ++i) { int64_t offset = 8 * (__builtin_e2k_bitrevd(i) >> shift); *(__v2du*)((void*)data_out + offset + 0*16) = (__v2du){data_in_00[i], data_in_10[i]}; *(__v2du*)((void*)data_out + offset + 1*16) = (__v2du){data_in_01[i], data_in_11[i]}; } }
Основной цикл на ассемблере
.L2619: { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=4, abs=0, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=4, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=4, abs=16, disp=0 } .L2503: { loop_mode qppackdl,1,sm %b[15], %b[18], %b[20] shrd,3,sm %b[21], %r4, %b[1] qppackdl,4,sm %b[9], %b[12], %b[22] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END addd,1,sm %b[4], 0x1, %b[2] stqp,2 %r2, %b[17], %b[20] bitrevd,3,sm %b[4], %b[19] shld,4,sm %b[1], 0x3, %b[15] stqp,5 %r0, %b[17], %b[22] movad,0 area=1, ind=0, am=1, be=0, %b[6] movad,1 area=0, ind=0, am=1, be=0, %b[12] movad,2 area=1, ind=0, am=1, be=0, %b[3] movad,3 area=0, ind=0, am=1, be=0, %b[9] }
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
Видим сильное ускорение.
Однако, код перестал вмещаться в один такт. Хочется это исправить.
7. reverse_radix2_x4_oneTickVersion
Перепишем код, чтобы вместиться в один такт (убираем инструкции shrd и shld).
Схема перемещения данных в памяти
Код на Си
void reverse_radix2_x4_oneTickVersion(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; int64_t delta = (1LL << shift) / 8; uint64_t *data_in_00 = (uint64_t*)&data_in[0 * count/4]; uint64_t *data_in_01 = (uint64_t*)&data_in[1 * count/4]; uint64_t *data_in_10 = (uint64_t*)&data_in[2 * count/4]; uint64_t *data_in_11 = (uint64_t*)&data_in[3 * count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0, shifted_i = 0; i < count/4; ++i, shifted_i += delta) { int64_t offset = __builtin_e2k_bitrevd(shifted_i); *(__v2du*)((void*)data_out + offset + 0*16) = (__v2du){data_in_00[i], data_in_10[i]}; *(__v2du*)((void*)data_out + offset + 1*16) = (__v2du){data_in_01[i], data_in_11[i]}; } }
Основной цикл на ассемблере
.L2891: { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=4, abs=0, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=4, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=4, abs=16, disp=0 } .L2777: { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END bitrevd,0,sm %b[34], %b[25] qppackdl,1,sm %b[22], %b[23], %b[31] stqp,2 %r2, %b[29], %b[33] qppackdl,3,sm %b[10], %b[11], %b[35] addd,4,sm %b[32], %r4, %b[30] stqp,5 %r0, %b[29], %b[37] movad,0 area=1, ind=0, am=1, be=0, %b[1] movad,1 area=0, ind=0, am=1, be=0, %b[13] movad,2 area=1, ind=0, am=1, be=0, %b[0] movad,3 area=0, ind=0, am=1, be=0, %b[12] }
Теоретическая скорость: 4 комплексных числа за 1 такт (4/1) = 32 Байт/такт
Замеры скорости
Видим ускорение в начале и замедление в конце графика.
Должно было быть либо лучше предыдущего варианта, либо так же. Не знаю, как это объяснить.
Примерно в этот момент у меня была мысль, что дальше оптимизировать не получится. Но, посмотрев на схемы алгоритмов, возник вопрос: «а если раскрутить по аналогии ещё в 2 раза?» Можно ли читать из 8-ми разных мест одновременно без потери скорости?
8. reverse_radix2_x8
Продолжаем «псевдо раскручивать» дальше.
Схема перемещения данных в памяти
Код на Си
void reverse_radix2_x8(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; uint64_t *data_in_000 = (uint64_t*)&data_in[0 * count/8]; uint64_t *data_in_001 = (uint64_t*)&data_in[1 * count/8]; uint64_t *data_in_010 = (uint64_t*)&data_in[2 * count/8]; uint64_t *data_in_011 = (uint64_t*)&data_in[3 * count/8]; uint64_t *data_in_100 = (uint64_t*)&data_in[4 * count/8]; uint64_t *data_in_101 = (uint64_t*)&data_in[5 * count/8]; uint64_t *data_in_110 = (uint64_t*)&data_in[6 * count/8]; uint64_t *data_in_111 = (uint64_t*)&data_in[7 * count/8]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count/8; ++i) { int64_t offset = 8 * (__builtin_e2k_bitrevd(i) >> shift); *(__v2du*)((void*)data_out + offset + 0*16) = (__v2du){data_in_000[i], data_in_100[i]}; *(__v2du*)((void*)data_out + offset + 1*16) = (__v2du){data_in_010[i], data_in_110[i]}; *(__v2du*)((void*)data_out + offset + 2*16) = (__v2du){data_in_001[i], data_in_101[i]}; *(__v2du*)((void*)data_out + offset + 3*16) = (__v2du){data_in_011[i], data_in_111[i]}; } }
Основной цикл на ассемблере
.L3314: { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=8, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=7, asz=3, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=6, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=5, asz=3, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=3, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=3, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=3, abs=24, disp=0 } .L3146: { loop_mode bitrevd,0,sm %b[32], %b[29] qppackdl,1,sm %b[20], %b[21], %b[33] stqp,2 %r2, %b[42], %b[38] shld,3,sm %b[34], 0x3, %b[40] qppackdl,4,sm %b[6], %b[7], %b[37] stqp,5 %r0, %b[42], %b[36] movad,0 area=3, ind=0, am=1, be=0, %b[1] movad,1 area=2, ind=0, am=1, be=0, %b[15] movad,2 area=3, ind=0, am=1, be=0, %b[0] movad,3 area=2, ind=0, am=1, be=0, %b[14] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppackdl,0,sm %b[26], %b[27], %b[36] addd,1,sm %b[30], 0x1, %b[28] stqp,2 %r4, %b[42], %b[35] qppackdl,3,sm %b[12], %b[13], %b[34] shrd,4,sm %b[31], %r6, %b[32] stqp,5 %r5, %b[42], %b[39] movad,0 area=1, ind=0, am=1, be=0, %b[7] movad,1 area=0, ind=0, am=1, be=0, %b[21] movad,2 area=1, ind=0, am=1, be=0, %b[6] movad,3 area=0, ind=0, am=1, be=0, %b[20] }
Теоретическая скорость: 8 комплексных чисел за 2 такта (8/2) = 32 Байт/такт
Замеры скорости
Видим замедление в начале и ускорение в конце графика.
Однако же, система справилась с чтением из 8-ми разных мест.
9. reverse_radix2_x16
А если в 16 раз?
Схема перемещения данных в памяти
Код на Си
void reverse_radix2_x16(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; uint64_t *data_in_0000 = (uint64_t*)&data_in[ 0 * count/16]; uint64_t *data_in_0001 = (uint64_t*)&data_in[ 1 * count/16]; uint64_t *data_in_0010 = (uint64_t*)&data_in[ 2 * count/16]; uint64_t *data_in_0011 = (uint64_t*)&data_in[ 3 * count/16]; uint64_t *data_in_0100 = (uint64_t*)&data_in[ 4 * count/16]; uint64_t *data_in_0101 = (uint64_t*)&data_in[ 5 * count/16]; uint64_t *data_in_0110 = (uint64_t*)&data_in[ 6 * count/16]; uint64_t *data_in_0111 = (uint64_t*)&data_in[ 7 * count/16]; uint64_t *data_in_1000 = (uint64_t*)&data_in[ 8 * count/16]; uint64_t *data_in_1001 = (uint64_t*)&data_in[ 9 * count/16]; uint64_t *data_in_1010 = (uint64_t*)&data_in[10 * count/16]; uint64_t *data_in_1011 = (uint64_t*)&data_in[11 * count/16]; uint64_t *data_in_1100 = (uint64_t*)&data_in[12 * count/16]; uint64_t *data_in_1101 = (uint64_t*)&data_in[13 * count/16]; uint64_t *data_in_1110 = (uint64_t*)&data_in[14 * count/16]; uint64_t *data_in_1111 = (uint64_t*)&data_in[15 * count/16]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count/16; ++i) { int64_t offset = 8 * (__builtin_e2k_bitrevd(i) >> shift); *(__v2du*)((void*)data_out + offset + 0*16) = (__v2du){data_in_0000[i], data_in_1000[i]}; *(__v2du*)((void*)data_out + offset + 1*16) = (__v2du){data_in_0100[i], data_in_1100[i]}; *(__v2du*)((void*)data_out + offset + 2*16) = (__v2du){data_in_0010[i], data_in_1010[i]}; *(__v2du*)((void*)data_out + offset + 3*16) = (__v2du){data_in_0110[i], data_in_1110[i]}; *(__v2du*)((void*)data_out + offset + 4*16) = (__v2du){data_in_0001[i], data_in_1001[i]}; *(__v2du*)((void*)data_out + offset + 5*16) = (__v2du){data_in_0101[i], data_in_1101[i]}; *(__v2du*)((void*)data_out + offset + 6*16) = (__v2du){data_in_0011[i], data_in_1011[i]}; *(__v2du*)((void*)data_out + offset + 7*16) = (__v2du){data_in_0111[i], data_in_1111[i]}; } }
Основной цикл на ассемблере
.L4040: { fapb ct=0, dcd=0, fmt=4, mrng=8, d=1, incr=0, ind=0, asz=2, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=15, asz=2, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=14, asz=2, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=13, asz=2, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=12, asz=2, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=11, asz=2, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=10, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=9, asz=2, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=8, asz=2, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=7, asz=2, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=6, asz=2, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=5, asz=2, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=2, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=2, abs=24, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=2, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=2, abs=28, disp=0 } .L3765: { loop_mode qppackdl,1,sm %b[21], %b[24], %b[40] stqp,2 %r2, %b[37], %b[41] qppackdl,4,sm %b[7], %b[10], %b[42] stqp,5 %r0, %b[37], %b[38] movad,0 area=7, ind=0, am=1, be=0, %b[4] movad,1 area=6, ind=0, am=1, be=0, %b[18] movad,2 area=7, ind=0, am=1, be=0, %b[1] movad,3 area=6, ind=0, am=1, be=0, %b[15] } { loop_mode qppackdl,1,sm %b[25], %b[28], %b[41] stqp,2 %r5, %b[37], %b[40] shrd,3,sm %b[31], %r12, %b[29] qppackdl,4,sm %b[11], %b[14], %b[38] stqp,5 %r6, %b[37], %b[42] movad,0 area=5, ind=0, am=1, be=0, %b[10] movad,1 area=4, ind=0, am=1, be=0, %b[24] movad,2 area=5, ind=0, am=1, be=0, %b[7] movad,3 area=4, ind=0, am=1, be=0, %b[21] } { loop_mode qppackdl,1,sm %b[17], %b[20], %b[32] stqp,2 %r7, %b[37], %b[41] shld,3,sm %b[29], 0x3, %b[35] qppackdl,4,sm %b[3], %b[6], %b[31] stqp,5 %r9, %b[37], %b[38] movad,0 area=1, ind=0, am=1, be=0, %b[14] movad,1 area=0, ind=0, am=1, be=0, %b[28] movad,2 area=1, ind=0, am=1, be=0, %b[11] movad,3 area=0, ind=0, am=1, be=0, %b[25] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppackdl,0,sm %b[27], %b[30], %b[39] addd,1,sm %b[2], 0x1, %b[0] stqp,2 %r10, %b[37], %b[34] qppackdl,3,sm %b[13], %b[16], %b[36] bitrevd,4,sm %b[2], %b[29] stqp,5 %r11, %b[37], %b[33] movad,0 area=3, ind=0, am=1, be=0, %b[6] movad,1 area=2, ind=0, am=1, be=0, %b[20] movad,2 area=3, ind=0, am=1, be=0, %b[3] movad,3 area=2, ind=0, am=1, be=0, %b[17] }
Теоретическая скорость: 16 комплексных чисел за 4 такта (16/4) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
10. reverse_radix2_x32
В 32 раза?
Код на Си
void reverse_radix2_x32(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; uint64_t *data_in_00000 = (uint64_t*)&data_in[ 0 * count/32]; uint64_t *data_in_00001 = (uint64_t*)&data_in[ 1 * count/32]; uint64_t *data_in_00010 = (uint64_t*)&data_in[ 2 * count/32]; uint64_t *data_in_00011 = (uint64_t*)&data_in[ 3 * count/32]; uint64_t *data_in_00100 = (uint64_t*)&data_in[ 4 * count/32]; uint64_t *data_in_00101 = (uint64_t*)&data_in[ 5 * count/32]; uint64_t *data_in_00110 = (uint64_t*)&data_in[ 6 * count/32]; uint64_t *data_in_00111 = (uint64_t*)&data_in[ 7 * count/32]; uint64_t *data_in_01000 = (uint64_t*)&data_in[ 8 * count/32]; uint64_t *data_in_01001 = (uint64_t*)&data_in[ 9 * count/32]; uint64_t *data_in_01010 = (uint64_t*)&data_in[10 * count/32]; uint64_t *data_in_01011 = (uint64_t*)&data_in[11 * count/32]; uint64_t *data_in_01100 = (uint64_t*)&data_in[12 * count/32]; uint64_t *data_in_01101 = (uint64_t*)&data_in[13 * count/32]; uint64_t *data_in_01110 = (uint64_t*)&data_in[14 * count/32]; uint64_t *data_in_01111 = (uint64_t*)&data_in[15 * count/32]; uint64_t *data_in_10000 = (uint64_t*)&data_in[16 * count/32]; uint64_t *data_in_10001 = (uint64_t*)&data_in[17 * count/32]; uint64_t *data_in_10010 = (uint64_t*)&data_in[18 * count/32]; uint64_t *data_in_10011 = (uint64_t*)&data_in[19 * count/32]; uint64_t *data_in_10100 = (uint64_t*)&data_in[20 * count/32]; uint64_t *data_in_10101 = (uint64_t*)&data_in[21 * count/32]; uint64_t *data_in_10110 = (uint64_t*)&data_in[22 * count/32]; uint64_t *data_in_10111 = (uint64_t*)&data_in[23 * count/32]; uint64_t *data_in_11000 = (uint64_t*)&data_in[24 * count/32]; uint64_t *data_in_11001 = (uint64_t*)&data_in[25 * count/32]; uint64_t *data_in_11010 = (uint64_t*)&data_in[26 * count/32]; uint64_t *data_in_11011 = (uint64_t*)&data_in[27 * count/32]; uint64_t *data_in_11100 = (uint64_t*)&data_in[28 * count/32]; uint64_t *data_in_11101 = (uint64_t*)&data_in[29 * count/32]; uint64_t *data_in_11110 = (uint64_t*)&data_in[30 * count/32]; uint64_t *data_in_11111 = (uint64_t*)&data_in[31 * count/32]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count/32; ++i) { int64_t offset = 8 * (__builtin_e2k_bitrevd(i) >> shift); *(__v2du*)((void*)data_out + offset + 0*16) = (__v2du){data_in_00000[i], data_in_10000[i]}; *(__v2du*)((void*)data_out + offset + 1*16) = (__v2du){data_in_01000[i], data_in_11000[i]}; *(__v2du*)((void*)data_out + offset + 2*16) = (__v2du){data_in_00100[i], data_in_10100[i]}; *(__v2du*)((void*)data_out + offset + 3*16) = (__v2du){data_in_01100[i], data_in_11100[i]}; *(__v2du*)((void*)data_out + offset + 4*16) = (__v2du){data_in_00010[i], data_in_10010[i]}; *(__v2du*)((void*)data_out + offset + 5*16) = (__v2du){data_in_01010[i], data_in_11010[i]}; *(__v2du*)((void*)data_out + offset + 6*16) = (__v2du){data_in_00110[i], data_in_10110[i]}; *(__v2du*)((void*)data_out + offset + 7*16) = (__v2du){data_in_01110[i], data_in_11110[i]}; *(__v2du*)((void*)data_out + offset + 8*16) = (__v2du){data_in_00001[i], data_in_10001[i]}; *(__v2du*)((void*)data_out + offset + 9*16) = (__v2du){data_in_01001[i], data_in_11001[i]}; *(__v2du*)((void*)data_out + offset + 10*16) = (__v2du){data_in_00101[i], data_in_10101[i]}; *(__v2du*)((void*)data_out + offset + 11*16) = (__v2du){data_in_01101[i], data_in_11101[i]}; *(__v2du*)((void*)data_out + offset + 12*16) = (__v2du){data_in_00011[i], data_in_10011[i]}; *(__v2du*)((void*)data_out + offset + 13*16) = (__v2du){data_in_01011[i], data_in_11011[i]}; *(__v2du*)((void*)data_out + offset + 14*16) = (__v2du){data_in_00111[i], data_in_10111[i]}; *(__v2du*)((void*)data_out + offset + 15*16) = (__v2du){data_in_01111[i], data_in_11111[i]}; } }
Основной цикл на ассемблере
.L5406: { fapb ct=0, dcd=0, fmt=4, mrng=8, d=17, incr=0, ind=0, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=16, incr=0, ind=0, asz=1, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=15, incr=0, ind=0, asz=1, abs=2, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=14, incr=0, ind=0, asz=1, abs=2, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=13, incr=0, ind=0, asz=1, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=12, incr=0, ind=0, asz=1, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=11, incr=0, ind=0, asz=1, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=10, incr=0, ind=0, asz=1, abs=6, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=9, incr=0, ind=0, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=8, incr=0, ind=0, asz=1, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=7, incr=0, ind=0, asz=1, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=6, incr=0, ind=0, asz=1, abs=10, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=5, incr=0, ind=0, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=4, incr=0, ind=0, asz=1, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=3, incr=0, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=2, incr=0, ind=0, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=1, incr=0, ind=0, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=15, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=14, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=13, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=12, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=11, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=10, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=9, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=8, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=7, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=6, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=5, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=1, abs=30, disp=0 } .L4875: { loop_mode qppackdl,1,sm %b[46], %b[47], %b[63] stqp,2 %r2, %b[61], %b[32] qppackdl,4,sm %b[38], %b[39], %b[62] stqp,5 %r4, %b[61], %b[51] movad,0 area=15, ind=0, am=1, be=0, %b[5] movad,1 area=14, ind=0, am=1, be=0, %b[13] movad,2 area=15, ind=0, am=1, be=0, %b[1] movad,3 area=14, ind=0, am=1, be=0, %b[9] } { loop_mode qppackdl,1,sm %b[20], %b[26], %b[65] stqp,2 %r5, %b[61], %b[63] qppackdl,4,sm %b[8], %b[14], %b[64] stqp,5 %r6, %b[61], %b[62] movad,0 area=13, ind=0, am=1, be=0, %b[25] movad,1 area=12, ind=0, am=1, be=0, %b[32] movad,2 area=13, ind=0, am=1, be=0, %b[17] movad,3 area=12, ind=0, am=1, be=0, %b[33] } { loop_mode qppackdl,1,sm %b[59], %b[60], %b[38] stqp,2 %r7, %b[61], %b[65] qppackdl,4,sm %b[55], %b[56], %b[39] stqp,5 %r9, %b[61], %b[64] movad,0 area=11, ind=0, am=1, be=0, %b[14] movad,1 area=10, ind=0, am=1, be=0, %b[26] movad,2 area=11, ind=0, am=1, be=0, %b[8] movad,3 area=10, ind=0, am=1, be=0, %b[20] } { loop_mode qppackdl,1,sm %b[57], %b[58], %b[47] stqp,2 %r10, %b[61], %b[40] qppackdl,4,sm %b[54], %b[53], %b[46] stqp,5 %r11, %b[61], %b[41] movad,0 area=9, ind=0, am=1, be=0, %b[51] movad,1 area=8, ind=0, am=1, be=0, %b[56] movad,2 area=9, ind=0, am=1, be=0, %b[52] movad,3 area=8, ind=0, am=1, be=0, %b[55] } { loop_mode addd,0,sm %b[4], 0x1, %b[2] qppackdl,1,sm %b[22], %b[28], %b[40] stqp,2 %r12, %b[61], %b[49] bitrevd,3,sm %b[4], %b[59] qppackdl,4,sm %b[10], %b[16], %b[41] stqp,5 %r13, %b[61], %b[48] movad,0 area=7, ind=0, am=1, be=0, %b[54] movad,1 area=6, ind=0, am=1, be=0, %b[58] movad,2 area=7, ind=0, am=1, be=0, %b[53] movad,3 area=6, ind=0, am=1, be=0, %b[57] } { loop_mode qppackdl,1,sm %b[35], %b[34], %b[48] stqp,2 %r14, %b[61], %b[42] shrd,3,sm %b[59], %r0, %b[49] qppackdl,4,sm %b[19], %b[27], %b[28] stqp,5 %r15, %b[61], %b[43] movad,0 area=5, ind=0, am=1, be=0, %b[10] movad,1 area=4, ind=0, am=1, be=0, %b[22] movad,2 area=5, ind=0, am=1, be=0, %b[4] movad,3 area=4, ind=0, am=1, be=0, %b[16] } { loop_mode qppackdl,1,sm %b[9], %b[13], %b[19] stqp,2 %r16, %b[61], %b[50] shld,3,sm %b[49], 0x3, %b[59] qppackdl,4,sm %b[1], %b[5], %b[27] stqp,5 %r17, %b[61], %b[30] movad,0 area=3, ind=0, am=1, be=0, %b[35] movad,1 area=2, ind=0, am=1, be=0, %b[43] movad,2 area=3, ind=0, am=1, be=0, %b[34] movad,3 area=2, ind=0, am=1, be=0, %b[42] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppackdl,0,sm %b[11], %b[15], %b[30] stqp,2 %r18, %b[61], %b[23] qppackdl,3,sm %b[3], %b[7], %b[49] stqp,5 %r19, %b[61], %b[31] movad,0 area=1, ind=0, am=1, be=0, %b[5] movad,1 area=0, ind=0, am=1, be=0, %b[13] movad,2 area=1, ind=0, am=1, be=0, %b[1] movad,3 area=0, ind=0, am=1, be=0, %b[9] }
Теоретическая скорость: 32 комплексных числа за 8 тактов (32/8) = 32 Байт/такт
Замеры скорости
Видим замедление в начале и ускорение в конце графика.
При попытке «псевдо раскрутить» в 64 раза получается резко менее эффективный код. APB может читать максимум из 32 потоков, поэтому для чтения из 64 потоков компилятор вставляет операции обычного чтения ldd. В итоге скорость резко проседает.
Можно ли ускорить ещё?
В голову приходит разве что попробовать читать не 64-битными кусками, а 128-битными.
11. reverse_radix2_x32x2
Попробуем увеличить скорость чтения версии reverse_radix2_x32.
По сути, в этом варианте сделана честная раскрутка в 2 раза.
Код на Си
void reverse_radix2_x32x2(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; __v2di *data_in_00000 = (__v2di*)&data_in[ 0 * count/32]; __v2di *data_in_00001 = (__v2di*)&data_in[ 1 * count/32]; __v2di *data_in_00010 = (__v2di*)&data_in[ 2 * count/32]; __v2di *data_in_00011 = (__v2di*)&data_in[ 3 * count/32]; __v2di *data_in_00100 = (__v2di*)&data_in[ 4 * count/32]; __v2di *data_in_00101 = (__v2di*)&data_in[ 5 * count/32]; __v2di *data_in_00110 = (__v2di*)&data_in[ 6 * count/32]; __v2di *data_in_00111 = (__v2di*)&data_in[ 7 * count/32]; __v2di *data_in_01000 = (__v2di*)&data_in[ 8 * count/32]; __v2di *data_in_01001 = (__v2di*)&data_in[ 9 * count/32]; __v2di *data_in_01010 = (__v2di*)&data_in[10 * count/32]; __v2di *data_in_01011 = (__v2di*)&data_in[11 * count/32]; __v2di *data_in_01100 = (__v2di*)&data_in[12 * count/32]; __v2di *data_in_01101 = (__v2di*)&data_in[13 * count/32]; __v2di *data_in_01110 = (__v2di*)&data_in[14 * count/32]; __v2di *data_in_01111 = (__v2di*)&data_in[15 * count/32]; __v2di *data_in_10000 = (__v2di*)&data_in[16 * count/32]; __v2di *data_in_10001 = (__v2di*)&data_in[17 * count/32]; __v2di *data_in_10010 = (__v2di*)&data_in[18 * count/32]; __v2di *data_in_10011 = (__v2di*)&data_in[19 * count/32]; __v2di *data_in_10100 = (__v2di*)&data_in[20 * count/32]; __v2di *data_in_10101 = (__v2di*)&data_in[21 * count/32]; __v2di *data_in_10110 = (__v2di*)&data_in[22 * count/32]; __v2di *data_in_10111 = (__v2di*)&data_in[23 * count/32]; __v2di *data_in_11000 = (__v2di*)&data_in[24 * count/32]; __v2di *data_in_11001 = (__v2di*)&data_in[25 * count/32]; __v2di *data_in_11010 = (__v2di*)&data_in[26 * count/32]; __v2di *data_in_11011 = (__v2di*)&data_in[27 * count/32]; __v2di *data_in_11100 = (__v2di*)&data_in[28 * count/32]; __v2di *data_in_11101 = (__v2di*)&data_in[29 * count/32]; __v2di *data_in_11110 = (__v2di*)&data_in[30 * count/32]; __v2di *data_in_11111 = (__v2di*)&data_in[31 * count/32]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count/32/2; ++i) { int64_t offset0 = 8 * (__builtin_e2k_bitrevd(2*i + 0) >> shift); __v2di mask0 = {0x0706050403020100, 0x0706050403020100}; *(__v2du*)((void*)data_out + offset0 + 0*16) = __builtin_e2k_qpshufb(data_in_10000[i], data_in_00000[i], mask0); *(__v2du*)((void*)data_out + offset0 + 1*16) = __builtin_e2k_qpshufb(data_in_11000[i], data_in_01000[i], mask0); *(__v2du*)((void*)data_out + offset0 + 2*16) = __builtin_e2k_qpshufb(data_in_10100[i], data_in_00100[i], mask0); *(__v2du*)((void*)data_out + offset0 + 3*16) = __builtin_e2k_qpshufb(data_in_11100[i], data_in_01100[i], mask0); *(__v2du*)((void*)data_out + offset0 + 4*16) = __builtin_e2k_qpshufb(data_in_10010[i], data_in_00010[i], mask0); *(__v2du*)((void*)data_out + offset0 + 5*16) = __builtin_e2k_qpshufb(data_in_11010[i], data_in_01010[i], mask0); *(__v2du*)((void*)data_out + offset0 + 6*16) = __builtin_e2k_qpshufb(data_in_10110[i], data_in_00110[i], mask0); *(__v2du*)((void*)data_out + offset0 + 7*16) = __builtin_e2k_qpshufb(data_in_11110[i], data_in_01110[i], mask0); *(__v2du*)((void*)data_out + offset0 + 8*16) = __builtin_e2k_qpshufb(data_in_10001[i], data_in_00001[i], mask0); *(__v2du*)((void*)data_out + offset0 + 9*16) = __builtin_e2k_qpshufb(data_in_11001[i], data_in_01001[i], mask0); *(__v2du*)((void*)data_out + offset0 + 10*16) = __builtin_e2k_qpshufb(data_in_10101[i], data_in_00101[i], mask0); *(__v2du*)((void*)data_out + offset0 + 11*16) = __builtin_e2k_qpshufb(data_in_11101[i], data_in_01101[i], mask0); *(__v2du*)((void*)data_out + offset0 + 12*16) = __builtin_e2k_qpshufb(data_in_10011[i], data_in_00011[i], mask0); *(__v2du*)((void*)data_out + offset0 + 13*16) = __builtin_e2k_qpshufb(data_in_11011[i], data_in_01011[i], mask0); *(__v2du*)((void*)data_out + offset0 + 14*16) = __builtin_e2k_qpshufb(data_in_10111[i], data_in_00111[i], mask0); *(__v2du*)((void*)data_out + offset0 + 15*16) = __builtin_e2k_qpshufb(data_in_11111[i], data_in_01111[i], mask0); int64_t offset1 = 8 * (__builtin_e2k_bitrevd(2*i + 1) >> shift); __v2di mask1 = {0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}; *(__v2du*)((void*)data_out + offset1 + 0*16) = __builtin_e2k_qpshufb(data_in_10000[i], data_in_00000[i], mask1); *(__v2du*)((void*)data_out + offset1 + 1*16) = __builtin_e2k_qpshufb(data_in_11000[i], data_in_01000[i], mask1); *(__v2du*)((void*)data_out + offset1 + 2*16) = __builtin_e2k_qpshufb(data_in_10100[i], data_in_00100[i], mask1); *(__v2du*)((void*)data_out + offset1 + 3*16) = __builtin_e2k_qpshufb(data_in_11100[i], data_in_01100[i], mask1); *(__v2du*)((void*)data_out + offset1 + 4*16) = __builtin_e2k_qpshufb(data_in_10010[i], data_in_00010[i], mask1); *(__v2du*)((void*)data_out + offset1 + 5*16) = __builtin_e2k_qpshufb(data_in_11010[i], data_in_01010[i], mask1); *(__v2du*)((void*)data_out + offset1 + 6*16) = __builtin_e2k_qpshufb(data_in_10110[i], data_in_00110[i], mask1); *(__v2du*)((void*)data_out + offset1 + 7*16) = __builtin_e2k_qpshufb(data_in_11110[i], data_in_01110[i], mask1); *(__v2du*)((void*)data_out + offset1 + 8*16) = __builtin_e2k_qpshufb(data_in_10001[i], data_in_00001[i], mask1); *(__v2du*)((void*)data_out + offset1 + 9*16) = __builtin_e2k_qpshufb(data_in_11001[i], data_in_01001[i], mask1); *(__v2du*)((void*)data_out + offset1 + 10*16) = __builtin_e2k_qpshufb(data_in_10101[i], data_in_00101[i], mask1); *(__v2du*)((void*)data_out + offset1 + 11*16) = __builtin_e2k_qpshufb(data_in_11101[i], data_in_01101[i], mask1); *(__v2du*)((void*)data_out + offset1 + 12*16) = __builtin_e2k_qpshufb(data_in_10011[i], data_in_00011[i], mask1); *(__v2du*)((void*)data_out + offset1 + 13*16) = __builtin_e2k_qpshufb(data_in_11011[i], data_in_01011[i], mask1); *(__v2du*)((void*)data_out + offset1 + 14*16) = __builtin_e2k_qpshufb(data_in_10111[i], data_in_00111[i], mask1); *(__v2du*)((void*)data_out + offset1 + 15*16) = __builtin_e2k_qpshufb(data_in_11111[i], data_in_01111[i], mask1); } }
Основной цикл на ассемблере
.L7238: { fapb ct=0, dcd=0, fmt=5, mrng=16, d=17, incr=0, ind=0, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=16, incr=0, ind=0, asz=1, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=15, incr=0, ind=0, asz=1, abs=2, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=14, incr=0, ind=0, asz=1, abs=2, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=13, incr=0, ind=0, asz=1, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=12, incr=0, ind=0, asz=1, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=11, incr=0, ind=0, asz=1, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=10, incr=0, ind=0, asz=1, abs=6, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=9, incr=0, ind=0, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=8, incr=0, ind=0, asz=1, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=7, incr=0, ind=0, asz=1, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=6, incr=0, ind=0, asz=1, abs=10, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=5, incr=0, ind=0, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=4, incr=0, ind=0, asz=1, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=3, incr=0, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=2, incr=0, ind=0, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=1, abs=30, disp=0 } .L5673: { loop_mode qpshufb,1,sm %b[42], %b[41], %r0, %b[40] stqp,2 %r2, %b[4], %b[48] qpshufb,3,sm %b[40], %b[39], %r0, %b[39] stqp,5 %r4, %b[4], %b[47] } { loop_mode qpshufb,1,sm %b[46], %b[45], %r1, %b[40] stqp,2 %r2, %g16, %b[40] qpshufb,4,sm %b[44], %b[43], %r1, %b[39] stqp,5 %r4, %g16, %b[39] } { loop_mode qpshufb,1,sm %b[46], %b[45], %r0, %b[40] stqp,2 %r6, %b[4], %b[40] qpshufb,4,sm %b[44], %b[43], %r0, %b[39] stqp,5 %r7, %b[4], %b[39] } { loop_mode qpshufb,1,sm %b[38], %b[37], %r0, %b[38] stqp,2 %r6, %g16, %b[40] qpshufb,4,sm %b[38], %b[37], %r1, %b[37] stqp,5 %r7, %g16, %b[39] } { loop_mode qpshufb,1,sm %b[36], %b[35], %r0, %b[36] stqp,2 %r9, %g16, %b[38] qpshufb,4,sm %b[36], %b[35], %r1, %b[35] stqp,5 %r9, %b[4], %b[37] movaqp,0 area=15, ind=0, am=1, be=0, %b[5] movaqp,1 area=14, ind=0, am=1, be=0, %b[9] movaqp,2 area=15, ind=0, am=1, be=0, %b[1] movaqp,3 area=14, ind=0, am=1, be=0, %b[6] } { loop_mode qpshufb,1,sm %b[34], %b[33], %r0, %b[34] stqp,2 %r10, %g16, %b[36] qpshufb,4,sm %b[34], %b[33], %r1, %b[33] stqp,5 %r10, %b[4], %b[35] movaqp,0 area=13, ind=0, am=1, be=0, %b[13] movaqp,1 area=12, ind=0, am=1, be=0, %b[17] movaqp,2 area=13, ind=0, am=1, be=0, %b[10] movaqp,3 area=12, ind=0, am=1, be=0, %b[14] } { loop_mode qpshufb,1,sm %b[32], %b[30], %r0, %b[32] stqp,2 %r11, %g16, %b[34] qpshufb,4,sm %b[32], %b[30], %r1, %b[30] stqp,5 %r11, %b[4], %b[33] movaqp,0 area=11, ind=0, am=1, be=0, %b[21] movaqp,1 area=10, ind=0, am=1, be=0, %b[25] movaqp,2 area=11, ind=0, am=1, be=0, %b[18] movaqp,3 area=10, ind=0, am=1, be=0, %b[22] } { loop_mode qpshufb,1,sm %g17, %g18, %r0, %b[32] stqp,2 %r12, %g16, %b[32] qpshufb,4,sm %g17, %g18, %r1, %b[30] stqp,5 %r12, %b[4], %b[30] movaqp,0 area=9, ind=0, am=1, be=0, %b[29] movaqp,1 area=8, ind=0, am=1, be=0, %g17 movaqp,2 area=9, ind=0, am=1, be=0, %b[26] movaqp,3 area=8, ind=0, am=1, be=0, %g18 } { loop_mode qpshufb,1,sm %b[31], %b[28], %r0, %b[34] stqp,2 %r13, %g16, %b[32] qpshufb,4,sm %b[31], %b[28], %r1, %b[33] stqp,5 %r13, %b[4], %b[30] movaqp,0 area=7, ind=0, am=1, be=0, %b[30] movaqp,1 area=6, ind=0, am=1, be=0, %b[32] movaqp,2 area=7, ind=0, am=1, be=0, %b[28] movaqp,3 area=6, ind=0, am=1, be=0, %b[31] } { loop_mode qpshufb,1,sm %b[27], %b[24], %r0, %b[38] stqp,2 %r14, %g16, %b[34] qpshufb,4,sm %b[27], %b[24], %r1, %b[37] stqp,5 %r14, %b[4], %b[33] movaqp,0 area=5, ind=0, am=1, be=0, %b[34] movaqp,1 area=4, ind=0, am=1, be=0, %b[36] movaqp,2 area=5, ind=0, am=1, be=0, %b[33] movaqp,3 area=4, ind=0, am=1, be=0, %b[35] } { loop_mode qpshufb,1,sm %b[23], %b[20], %r0, %b[42] stqp,2 %r15, %g16, %b[38] qpshufb,4,sm %b[23], %b[20], %r1, %b[41] stqp,5 %r15, %b[4], %b[37] movaqp,0 area=1, ind=0, am=1, be=0, %b[38] movaqp,1 area=0, ind=0, am=1, be=0, %b[40] movaqp,2 area=1, ind=0, am=1, be=0, %b[37] movaqp,3 area=0, ind=0, am=1, be=0, %b[39] } { loop_mode addd,0,sm 0x2, %b[2], %b[0] qpshufb,1,sm %b[19], %b[16], %r0, %b[47] stqp,2 %r16, %g16, %b[42] addd,3,sm %b[2], 0x1, %b[45] qpshufb,4,sm %b[19], %b[16], %r1, %b[46] stqp,5 %r16, %b[4], %b[41] movaqp,0 area=3, ind=0, am=1, be=0, %b[42] movaqp,1 area=2, ind=0, am=1, be=0, %b[44] movaqp,2 area=3, ind=0, am=1, be=0, %b[41] movaqp,3 area=2, ind=0, am=1, be=0, %b[43] } { loop_mode bitrevd,0,sm %b[2], %b[46] qpshufb,1,sm %b[15], %b[12], %r0, %b[49] stqp,2 %r17, %g16, %b[47] bitrevd,3,sm %b[45], %b[45] qpshufb,4,sm %b[15], %b[12], %r1, %b[47] stqp,5 %r17, %b[4], %b[46] } { loop_mode shrd,0,sm %b[46], %r5, %b[45] qpshufb,1,sm %b[11], %b[8], %r0, %b[48] stqp,2 %r18, %g16, %b[49] shrd,3,sm %b[45], %r5, %b[46] qpshufb,4,sm %b[11], %b[8], %r1, %b[47] stqp,5 %r18, %b[4], %b[47] } { loop_mode qpshufb,1,sm %b[7], %b[3], %r0, %b[47] stqp,2 %r19, %g16, %b[48] shld,3,sm %b[46], 0x3, %b[2] qpshufb,4,sm %b[7], %b[3], %r1, %b[46] stqp,5 %r19, %b[4], %b[47] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpshufb,0,sm %b[40], %b[39], %r1, %b[46] stqp,2 %r20, %g16, %b[47] qpshufb,3,sm %b[38], %b[37], %r1, %b[45] shld,4,sm %b[45], 0x3, %g16 stqp,5 %r20, %b[4], %b[46] }
Теоретическая скорость: 64 комплексных числа за 16 тактов (64/16) = 32 Байт/такт
Замеры скорости
Видим замедление в середине графика.
Итоги по reverse_radix2
Победителем можно считать вариант reverse_radix2_x32.
При реализации Radix-2 FFT будем использовать его.
reverse_radix4
1. reverse_radix4_etalon
Эталонный вариант для сравнения на корректность.
Здесь reverseNumber вычисляется с помощью цикла.
int reverseNumber_radix4(int number, int bit_count) { int answer = 0; for(int i = 0; i < bit_count/2; ++i) { answer <<= 2; answer |= number & 3; number >>= 2; } return answer; } void reverse_radix4_etalon(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; for(int64_t i = 0; i < count; ++i) { int index = reverseNumber_radix4(i, bit_count); data_out[index] = data_in[i]; } }
2. reverse_radix4
В процессоре нет готовых инструкций, производящих операцию reverseNumber_radix4.
Поэтому выполним инструкцию bitrevd и переставим соседние биты местами.
Схема перемещения данных в памяти
Код на Си
void reverse_radix4(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i) >> shift; int64_t index = ((rev<<1) & 0xAAAAAAAAAAAAAAAA) | ((rev>>1) & 0x5555555555555555); data_out[index] = data_in[i]; } }
Основной цикл на ассемблере
.L1601: { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=5, abs=0, disp=0 } .L1398: { loop_mode shrd,2,sm %b[16], %r0, %b[1] shld,4,sm %b[17], 0x3, %b[18] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END bitrevd,0,sm %b[2], %b[14] shr_andd,1,sm %b[1], 0x1, %r5, %b[5] addd,2,sm %b[2], 0x1, %b[0] ord,3,sm %b[13], %b[9], %b[15] shl_andd,4,sm %b[3], 0x1, %r4, %b[11] std,5 %r2, %b[18], %b[12] movad,1 area=0, ind=0, am=1, be=0, %b[4] }
Теоретическая скорость: 1 комплексное число за 2 такта (1/2) = 4 Байт/такт
Замеры скорости
Заметим, что код можно вместить в один такт, если немного перетасовать инструкции.
3. reverse_radix4_oneTickVersion
Перепишем код, чтобы вместиться в один такт (убираем инструкции shrd и shld).
Схема перемещения данных в памяти
Код на Си
void reverse_radix4_oneTickVersion(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = ((rev>>(shift-3-1)) & 0x5555555555555555) | ((rev>>(shift-3+1)) & 0xAAAAAAAAAAAAAAAA); *(myComplex*)((void*)data_out + offset) = data_in[i]; } }
Основной цикл на ассемблере
.L1873: { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=5, abs=0, disp=0 } .L1686: { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END bitrevd,0,sm %b[21], %b[14] shr_andd,1,sm %b[16], %r0, %r6, %b[1] addd,2,sm %b[21], 0x1, %b[19] ord,3,sm %b[17], %b[9], %b[20] shr_andd,4,sm %b[18], %r4, %r5, %b[11] std,5 %r2, %b[22], %b[12] movad,1 area=0, ind=0, am=1, be=0, %b[0] }
Теоретическая скорость: 1 комплексное число за 1 такт (1/1) = 8 Байт/такт
Замеры скорости
Видим ускорение в начале графика.
4. reverse_radix4_x4_bad
Попробуем ускорить с помощью ручной раскрутки цикла в 4 раза.
Схема перемещения данных в памяти
Код на Си
void reverse_radix4_x4_bad(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; myComplex *data_out_0 = &data_out[0 * count/4]; myComplex *data_out_1 = &data_out[1 * count/4]; myComplex *data_out_2 = &data_out[2 * count/4]; myComplex *data_out_3 = &data_out[3 * count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count; i += 4) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t index = ((rev>>(shift-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift+1)) & 0x5555555555555555); data_out_0[index] = data_in[i + 0]; data_out_1[index] = data_in[i + 1]; data_out_2[index] = data_in[i + 2]; data_out_3[index] = data_in[i + 3]; } }
Основной цикл на ассемблере
.L2338: { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=1, ind=1, asz=5, abs=0, disp=16 } .L2049: { loop_mode bitrevd,0,sm %b[29], %b[23] std,2 %r0, %b[26], %b[18] addd,3,sm %b[29], 0x4, %b[27] shld,4,sm %b[30], 0x3, %b[24] std,5 %r5, %b[26], %b[19] movad,0 area=0, ind=0, am=0, be=0, %b[1] movad,1 area=0, ind=8, am=1, be=0, %b[11] movad,2 area=0, ind=0, am=0, be=0, %b[0] movad,3 area=0, ind=8, am=1, be=0, %b[10] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END shr_andd,1,sm %b[23], %r7, %r9, %b[18] std,2 %r4, %b[26], %b[8] ord,3,sm %b[21], %b[22], %b[28] shr_andd,4,sm %b[25], %r6, %r8, %b[19] std,5 %r2, %b[26], %b[9] }
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
Видим ускорение в начале графика.
Как мы помним из reverse_radix2, запись в разные места памяти работает хуже, чем запись рядом.
5. reverse_radix4_x4_good
Попробуем сделать наоборот: будем читать из разных мест, а писать рядом.
Схема перемещения данных в памяти
Код на Си
void reverse_radix4_x4_good(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; myComplex *data_in_0 = &data_in[0 * count/4]; myComplex *data_in_1 = &data_in[1 * count/4]; myComplex *data_in_2 = &data_in[2 * count/4]; myComplex *data_in_3 = &data_in[3 * count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count/4; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t index = ((rev>>(shift-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift+1)) & 0x5555555555555555); data_out[index + 0] = data_in_0[i]; data_out[index + 1] = data_in_1[i]; data_out[index + 2] = data_in_2[i]; data_out[index + 3] = data_in_3[i]; } }
Основной цикл на ассемблере
.L2807: { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=4, abs=0, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=4, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=4, abs=16, disp=0 } .L2518: { loop_mode bitrevd,0,sm %b[29], %b[23] std,2 %r5, %b[26], %b[19] addd,3,sm %b[29], 0x1, %b[27] shld,4,sm %b[30], 0x3, %b[24] std,5 %r0, %b[26], %b[18] movad,0 area=1, ind=0, am=1, be=0, %b[1] movad,1 area=0, ind=0, am=1, be=0, %b[0] movad,2 area=1, ind=0, am=1, be=0, %b[11] movad,3 area=0, ind=0, am=1, be=0, %b[10] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END shr_andd,1,sm %b[23], %r7, %r9, %b[18] std,2 %r4, %b[26], %b[9] ord,3,sm %b[21], %b[22], %b[28] shr_andd,4,sm %b[25], %r6, %r8, %b[19] std,5 %b[26], %r2, %b[8] }
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
Видим желаемое ускорение по всей длине графика.
Строго говоря, это не раскрутка цикла. Честной раскруткой был предыдущий вариант. Здесь же произошло изменение алгоритма (данные обрабатываются в другом порядке). Но я не придумал, как это назвать («stream4»?), поэтому все дальнейшие «раскрутки» будут называться x4/x16 и т.д.
6. reverse_radix4_x4_best
Вместо четырёх 64-битных записей в память сделаем две 128-битные записи.
Схема перемещения данных в памяти
Код на Си
void reverse_radix4_x4_best(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; uint64_t *data_in_0 = (uint64_t*)&data_in[0 * count/4]; uint64_t *data_in_1 = (uint64_t*)&data_in[1 * count/4]; uint64_t *data_in_2 = (uint64_t*)&data_in[2 * count/4]; uint64_t *data_in_3 = (uint64_t*)&data_in[3 * count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count/4; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 8 * (((rev>>(shift-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift+1)) & 0x5555555555555555)); *(__v2du*)((void*)data_out + offset + 0*16) = (__v2du){data_in_0[i], data_in_1[i]}; *(__v2du*)((void*)data_out + offset + 1*16) = (__v2du){data_in_2[i], data_in_3[i]}; } }
Основной цикл на ассемблере
.L3099: { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=4, abs=0, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=4, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=4, abs=16, disp=0 } .L2975: { loop_mode qppackdl,0,sm %b[10], %b[16], %b[9] shr_andd,1,sm %b[23], %r5, %r7, %b[0] qppackdl,3,sm %b[21], %b[22], %b[5] shr_andd,4,sm %b[25], %r4, %r6, %b[13] ord,5,sm %b[15], %b[4], %b[26] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END bitrevd,0,sm %b[3], %b[21] stqp,2 %r0, %b[24], %b[11] addd,3,sm %b[3], 0x1, %b[1] shld,4,sm %b[26], 0x3, %b[22] stqp,5 %r2, %b[24], %b[7] movad,0 area=1, ind=0, am=1, be=0, %b[10] movad,1 area=0, ind=0, am=1, be=0, %b[16] movad,2 area=1, ind=0, am=1, be=0, %b[4] movad,3 area=0, ind=0, am=1, be=0, %b[15] }
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
Видим сильное ускорение.
В дальнейшем будем всегда писать в память 128-битными кусками.
7. reverse_radix4_x16
Продолжим «псевдо раскручивать» дальше.
Схема перемещения данных в памяти
Код на Си
void reverse_radix4_x16(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; uint64_t *data_in_00 = (uint64_t*)&data_in[ 0 * count/16]; uint64_t *data_in_01 = (uint64_t*)&data_in[ 1 * count/16]; uint64_t *data_in_02 = (uint64_t*)&data_in[ 2 * count/16]; uint64_t *data_in_03 = (uint64_t*)&data_in[ 3 * count/16]; uint64_t *data_in_10 = (uint64_t*)&data_in[ 4 * count/16]; uint64_t *data_in_11 = (uint64_t*)&data_in[ 5 * count/16]; uint64_t *data_in_12 = (uint64_t*)&data_in[ 6 * count/16]; uint64_t *data_in_13 = (uint64_t*)&data_in[ 7 * count/16]; uint64_t *data_in_20 = (uint64_t*)&data_in[ 8 * count/16]; uint64_t *data_in_21 = (uint64_t*)&data_in[ 9 * count/16]; uint64_t *data_in_22 = (uint64_t*)&data_in[10 * count/16]; uint64_t *data_in_23 = (uint64_t*)&data_in[11 * count/16]; uint64_t *data_in_30 = (uint64_t*)&data_in[12 * count/16]; uint64_t *data_in_31 = (uint64_t*)&data_in[13 * count/16]; uint64_t *data_in_32 = (uint64_t*)&data_in[14 * count/16]; uint64_t *data_in_33 = (uint64_t*)&data_in[15 * count/16]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count/16; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 8 * (((rev>>(shift-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift+1)) & 0x5555555555555555)); *(__v2du*)((void*)data_out + offset + 0*16) = (__v2du){data_in_00[i], data_in_10[i]}; *(__v2du*)((void*)data_out + offset + 1*16) = (__v2du){data_in_20[i], data_in_30[i]}; *(__v2du*)((void*)data_out + offset + 2*16) = (__v2du){data_in_01[i], data_in_11[i]}; *(__v2du*)((void*)data_out + offset + 3*16) = (__v2du){data_in_21[i], data_in_31[i]}; *(__v2du*)((void*)data_out + offset + 4*16) = (__v2du){data_in_02[i], data_in_12[i]}; *(__v2du*)((void*)data_out + offset + 5*16) = (__v2du){data_in_22[i], data_in_32[i]}; *(__v2du*)((void*)data_out + offset + 6*16) = (__v2du){data_in_03[i], data_in_13[i]}; *(__v2du*)((void*)data_out + offset + 7*16) = (__v2du){data_in_23[i], data_in_33[i]}; } }
Основной цикл на ассемблере
.L3848: { fapb ct=0, dcd=0, fmt=4, mrng=8, d=1, incr=0, ind=0, asz=2, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=15, asz=2, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=14, asz=2, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=13, asz=2, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=12, asz=2, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=11, asz=2, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=10, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=9, asz=2, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=8, asz=2, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=7, asz=2, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=6, asz=2, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=5, asz=2, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=2, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=2, abs=24, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=2, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=2, abs=28, disp=0 } .L3565: { loop_mode qppackdl,0,sm %b[51], %b[58], %b[1] stqp,2 %r0, %b[57], %b[42] qppackdl,3,sm %b[35], %b[50], %b[6] shr_andd,4,sm %b[60], %r12, %r15, %b[29] stqp,5 %r2, %b[57], %b[43] movad,0 area=7, ind=0, am=1, be=0, %b[18] movad,1 area=6, ind=0, am=1, be=0, %b[26] movad,2 area=7, ind=0, am=1, be=0, %b[13] movad,3 area=6, ind=0, am=1, be=0, %b[21] } { loop_mode ord,0,sm %b[61], %b[31], %b[59] qppackdl,1,sm %b[54], %b[55], %b[35] stqp,2 %r5, %b[57], %b[5] qppackdl,4,sm %b[46], %b[47], %b[34] stqp,5 %r6, %b[57], %b[10] movad,0 area=5, ind=0, am=1, be=0, %b[43] movad,1 area=4, ind=0, am=1, be=0, %b[51] movad,2 area=5, ind=0, am=1, be=0, %b[42] movad,3 area=4, ind=0, am=1, be=0, %b[50] } { loop_mode shld,0,sm %b[59], 0x3, %b[55] qppackdl,1,sm %b[23], %b[28], %b[5] stqp,2 %r7, %b[57], %b[39] addd,3,sm %b[4], 0x1, %b[2] qppackdl,4,sm %b[15], %b[20], %b[10] stqp,5 %r9, %b[57], %b[38] movad,0 area=3, ind=0, am=1, be=0, %b[46] movad,1 area=2, ind=0, am=1, be=0, %b[54] movad,2 area=3, ind=0, am=1, be=0, %b[31] movad,3 area=2, ind=0, am=1, be=0, %b[47] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppackdl,0,sm %b[19], %b[24], %b[38] shr_andd,1,sm %b[60], %r13, %r14, %b[59] stqp,2 %r10, %b[57], %b[11] qppackdl,3,sm %b[27], %b[32], %b[39] bitrevd,4,sm %b[4], %b[58] stqp,5 %r11, %b[57], %b[16] movad,0 area=1, ind=0, am=1, be=0, %b[20] movad,1 area=0, ind=0, am=1, be=0, %b[28] movad,2 area=1, ind=0, am=1, be=0, %b[15] movad,3 area=0, ind=0, am=1, be=0, %b[23] }
Теоретическая скорость: 16 комплексных чисел за 4 такта (16/4) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
При попытке «псевдо раскрутить» в 64 раза получается резко менее эффективный код. APB может читать максимум из 32 потоков, поэтому для чтения из 64 потоков компилятор вставляет операции обычного чтения ldd. В итоге скорость резко проседает.
Попробуем читать не 64-битными кусками, а 128-битными.
8. reverse_radix4_x16x2
Попробуем увеличить скорость чтения версии reverse_radix4_x16.
По сути, в этом варианте сделана честная раскрутка в 2 раза.
Код на Си
void reverse_radix4_x16x2(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; __v2di *data_in_00 = (__v2di*)&data_in[ 0 * count/16]; __v2di *data_in_01 = (__v2di*)&data_in[ 1 * count/16]; __v2di *data_in_02 = (__v2di*)&data_in[ 2 * count/16]; __v2di *data_in_03 = (__v2di*)&data_in[ 3 * count/16]; __v2di *data_in_10 = (__v2di*)&data_in[ 4 * count/16]; __v2di *data_in_11 = (__v2di*)&data_in[ 5 * count/16]; __v2di *data_in_12 = (__v2di*)&data_in[ 6 * count/16]; __v2di *data_in_13 = (__v2di*)&data_in[ 7 * count/16]; __v2di *data_in_20 = (__v2di*)&data_in[ 8 * count/16]; __v2di *data_in_21 = (__v2di*)&data_in[ 9 * count/16]; __v2di *data_in_22 = (__v2di*)&data_in[10 * count/16]; __v2di *data_in_23 = (__v2di*)&data_in[11 * count/16]; __v2di *data_in_30 = (__v2di*)&data_in[12 * count/16]; __v2di *data_in_31 = (__v2di*)&data_in[13 * count/16]; __v2di *data_in_32 = (__v2di*)&data_in[14 * count/16]; __v2di *data_in_33 = (__v2di*)&data_in[15 * count/16]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count/16/2; ++i) { uint64_t rev0 = __builtin_e2k_bitrevd(2*i+0); int64_t offset0 = 8 * (((rev0>>(shift-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev0>>(shift+1)) & 0x5555555555555555)); __v2di mask0 = {0x0706050403020100, 0x0706050403020100}; *(__v2du*)((void*)data_out + offset0 + 0*16) = __builtin_e2k_qpshufb(data_in_10[i], data_in_00[i], mask0); *(__v2du*)((void*)data_out + offset0 + 1*16) = __builtin_e2k_qpshufb(data_in_30[i], data_in_20[i], mask0); *(__v2du*)((void*)data_out + offset0 + 2*16) = __builtin_e2k_qpshufb(data_in_11[i], data_in_01[i], mask0); *(__v2du*)((void*)data_out + offset0 + 3*16) = __builtin_e2k_qpshufb(data_in_31[i], data_in_21[i], mask0); *(__v2du*)((void*)data_out + offset0 + 4*16) = __builtin_e2k_qpshufb(data_in_12[i], data_in_02[i], mask0); *(__v2du*)((void*)data_out + offset0 + 5*16) = __builtin_e2k_qpshufb(data_in_32[i], data_in_22[i], mask0); *(__v2du*)((void*)data_out + offset0 + 6*16) = __builtin_e2k_qpshufb(data_in_13[i], data_in_03[i], mask0); *(__v2du*)((void*)data_out + offset0 + 7*16) = __builtin_e2k_qpshufb(data_in_33[i], data_in_23[i], mask0); uint64_t rev1 = __builtin_e2k_bitrevd(2*i+1); int64_t offset1 = 8 * (((rev1>>(shift-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev1>>(shift+1)) & 0x5555555555555555)); __v2di mask1 = {0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}; *(__v2du*)((void*)data_out + offset1 + 0*16) = __builtin_e2k_qpshufb(data_in_10[i], data_in_00[i], mask1); *(__v2du*)((void*)data_out + offset1 + 1*16) = __builtin_e2k_qpshufb(data_in_30[i], data_in_20[i], mask1); *(__v2du*)((void*)data_out + offset1 + 2*16) = __builtin_e2k_qpshufb(data_in_11[i], data_in_01[i], mask1); *(__v2du*)((void*)data_out + offset1 + 3*16) = __builtin_e2k_qpshufb(data_in_31[i], data_in_21[i], mask1); *(__v2du*)((void*)data_out + offset1 + 4*16) = __builtin_e2k_qpshufb(data_in_12[i], data_in_02[i], mask1); *(__v2du*)((void*)data_out + offset1 + 5*16) = __builtin_e2k_qpshufb(data_in_32[i], data_in_22[i], mask1); *(__v2du*)((void*)data_out + offset1 + 6*16) = __builtin_e2k_qpshufb(data_in_13[i], data_in_03[i], mask1); *(__v2du*)((void*)data_out + offset1 + 7*16) = __builtin_e2k_qpshufb(data_in_33[i], data_in_23[i], mask1); } }
Основной цикл на ассемблере
.L4839: { fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=2, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=2, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=2, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=2, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=2, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=2, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=2, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=2, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=2, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=2, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=2, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=2, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=2, abs=24, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=2, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=2, abs=28, disp=0 } .L3987: { loop_mode qpshufb,0,sm %b[26], %b[21], %r12, %b[51] shr_andd,1,sm %b[23], %r14, %r17, %b[1] stqp,2 %r2, %b[4], %b[13] qpshufb,3,sm %b[45], %b[42], %r12, %b[50] shr_andd,4,sm %b[23], %r15, %r16, %b[5] stqp,5 %r0, %b[4], %b[12] } { loop_mode qpshufb,0,sm %b[35], %b[32], %r12, %b[54] shr_andd,1,sm %b[3], %r14, %r17, %b[23] stqp,2 %r5, %b[4], %b[51] qpshufb,3,sm %b[29], %b[11], %r12, %b[53] ord,4,sm %b[7], %b[25], %b[52] stqp,5 %r6, %b[4], %b[50] movaqp,0 area=7, ind=0, am=1, be=0, %b[12] movaqp,1 area=6, ind=0, am=1, be=0, %b[18] movaqp,2 area=7, ind=0, am=1, be=0, %b[6] movaqp,3 area=6, ind=0, am=1, be=0, %b[13] } { loop_mode qpshufb,1,sm %b[22], %b[17], %r12, %b[55] stqp,2 %r7, %b[4], %b[54] qpshufb,3,sm %b[22], %b[17], %r13, %b[51] shld,4,sm %b[52], 0x3, %b[50] stqp,5 %r9, %b[4], %b[53] movaqp,0 area=5, ind=0, am=1, be=0, %b[25] movaqp,1 area=4, ind=0, am=1, be=0, %b[31] movaqp,2 area=5, ind=0, am=1, be=0, %b[7] movaqp,3 area=4, ind=0, am=1, be=0, %b[28] } { loop_mode qpshufb,1,sm %b[45], %b[42], %r13, %b[53] stqp,2 %r10, %b[4], %b[55] qpshufb,4,sm %b[35], %b[32], %r13, %b[52] stqp,5 %r10, %b[50], %b[51] movaqp,0 area=3, ind=0, am=1, be=0, %b[41] movaqp,1 area=2, ind=0, am=1, be=0, %b[22] movaqp,2 area=3, ind=0, am=1, be=0, %b[38] movaqp,3 area=2, ind=0, am=1, be=0, %b[17] } { loop_mode addd,0,sm %b[2], 0x1, %b[48] qpshufb,1,sm %b[49], %b[46], %r13, %b[54] stqp,2 %r6, %b[50], %b[53] addd,3,sm 0x2, %b[2], %b[0] qpshufb,4,sm %b[26], %b[21], %r13, %b[51] stqp,5 %r7, %b[50], %b[52] movaqp,0 area=1, ind=0, am=1, be=0, %b[45] movaqp,1 area=0, ind=0, am=1, be=0, %b[35] movaqp,2 area=1, ind=0, am=1, be=0, %b[42] movaqp,3 area=0, ind=0, am=1, be=0, %b[32] } { loop_mode bitrevd,0,sm %b[2], %b[21] qpshufb,1,sm %b[39], %b[36], %r13, %b[52] stqp,2 %r0, %b[50], %b[54] ord,3,sm %b[5], %b[1], %b[26] qpshufb,4,sm %b[16], %b[10], %r13, %b[49] stqp,5 %r5, %b[50], %b[51] } { loop_mode bitrevd,0,sm %b[48], %b[1] qpshufb,1,sm %b[16], %b[10], %r12, %b[53] stqp,2 %r2, %b[50], %b[52] shld,3,sm %b[26], 0x3, %b[2] qpshufb,4,sm %b[29], %b[11], %r13, %b[51] stqp,5 %r11, %b[50], %b[49] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpshufb,0,sm %b[37], %b[34], %r12, %b[11] stqp,2 %r11, %b[4], %b[53] qpshufb,3,sm %b[47], %b[44], %r12, %b[10] shr_andd,4,sm %b[3], %r15, %r16, %b[5] stqp,5 %r9, %b[50], %b[51] }
Теоретическая скорость: 32 комплексных числа за 8 тактов (32/8) = 32 Байт/такт
Замеры скорости
Видим замедление в середине графика.
Ещё можно сделать раскрутку в 32 раза. Для этого напишем версию раскрутки в 64 раза и обработаем сначала одну половину строк в одном цикле, а потом вторую половину строк во втором цикле. В каждом цикле будут использованы 32 потока чтения APB.
9. reverse_radix4_x32
Сделаем «псевдо раскрутку» в 32 раза с помощью двух циклов.
Код на Си
void reverse_radix4_x32(int bit_count, myComplex *data_in, myComplex *data_out) { int count = 1 << bit_count; int shift = 64 - bit_count; uint64_t *data_in_000 = (uint64_t*)&data_in[ 0 * count/64]; uint64_t *data_in_001 = (uint64_t*)&data_in[ 1 * count/64]; uint64_t *data_in_002 = (uint64_t*)&data_in[ 2 * count/64]; uint64_t *data_in_003 = (uint64_t*)&data_in[ 3 * count/64]; uint64_t *data_in_010 = (uint64_t*)&data_in[ 4 * count/64]; uint64_t *data_in_011 = (uint64_t*)&data_in[ 5 * count/64]; uint64_t *data_in_012 = (uint64_t*)&data_in[ 6 * count/64]; uint64_t *data_in_013 = (uint64_t*)&data_in[ 7 * count/64]; uint64_t *data_in_020 = (uint64_t*)&data_in[ 8 * count/64]; uint64_t *data_in_021 = (uint64_t*)&data_in[ 9 * count/64]; uint64_t *data_in_022 = (uint64_t*)&data_in[10 * count/64]; uint64_t *data_in_023 = (uint64_t*)&data_in[11 * count/64]; uint64_t *data_in_030 = (uint64_t*)&data_in[12 * count/64]; uint64_t *data_in_031 = (uint64_t*)&data_in[13 * count/64]; uint64_t *data_in_032 = (uint64_t*)&data_in[14 * count/64]; uint64_t *data_in_033 = (uint64_t*)&data_in[15 * count/64]; uint64_t *data_in_100 = (uint64_t*)&data_in[16 * count/64]; uint64_t *data_in_101 = (uint64_t*)&data_in[17 * count/64]; uint64_t *data_in_102 = (uint64_t*)&data_in[18 * count/64]; uint64_t *data_in_103 = (uint64_t*)&data_in[19 * count/64]; uint64_t *data_in_110 = (uint64_t*)&data_in[20 * count/64]; uint64_t *data_in_111 = (uint64_t*)&data_in[21 * count/64]; uint64_t *data_in_112 = (uint64_t*)&data_in[22 * count/64]; uint64_t *data_in_113 = (uint64_t*)&data_in[23 * count/64]; uint64_t *data_in_120 = (uint64_t*)&data_in[24 * count/64]; uint64_t *data_in_121 = (uint64_t*)&data_in[25 * count/64]; uint64_t *data_in_122 = (uint64_t*)&data_in[26 * count/64]; uint64_t *data_in_123 = (uint64_t*)&data_in[27 * count/64]; uint64_t *data_in_130 = (uint64_t*)&data_in[28 * count/64]; uint64_t *data_in_131 = (uint64_t*)&data_in[29 * count/64]; uint64_t *data_in_132 = (uint64_t*)&data_in[30 * count/64]; uint64_t *data_in_133 = (uint64_t*)&data_in[31 * count/64]; uint64_t *data_in_200 = (uint64_t*)&data_in[32 * count/64]; uint64_t *data_in_201 = (uint64_t*)&data_in[33 * count/64]; uint64_t *data_in_202 = (uint64_t*)&data_in[34 * count/64]; uint64_t *data_in_203 = (uint64_t*)&data_in[35 * count/64]; uint64_t *data_in_210 = (uint64_t*)&data_in[36 * count/64]; uint64_t *data_in_211 = (uint64_t*)&data_in[37 * count/64]; uint64_t *data_in_212 = (uint64_t*)&data_in[38 * count/64]; uint64_t *data_in_213 = (uint64_t*)&data_in[39 * count/64]; uint64_t *data_in_220 = (uint64_t*)&data_in[40 * count/64]; uint64_t *data_in_221 = (uint64_t*)&data_in[41 * count/64]; uint64_t *data_in_222 = (uint64_t*)&data_in[42 * count/64]; uint64_t *data_in_223 = (uint64_t*)&data_in[43 * count/64]; uint64_t *data_in_230 = (uint64_t*)&data_in[44 * count/64]; uint64_t *data_in_231 = (uint64_t*)&data_in[45 * count/64]; uint64_t *data_in_232 = (uint64_t*)&data_in[46 * count/64]; uint64_t *data_in_233 = (uint64_t*)&data_in[47 * count/64]; uint64_t *data_in_300 = (uint64_t*)&data_in[48 * count/64]; uint64_t *data_in_301 = (uint64_t*)&data_in[49 * count/64]; uint64_t *data_in_302 = (uint64_t*)&data_in[50 * count/64]; uint64_t *data_in_303 = (uint64_t*)&data_in[51 * count/64]; uint64_t *data_in_310 = (uint64_t*)&data_in[52 * count/64]; uint64_t *data_in_311 = (uint64_t*)&data_in[53 * count/64]; uint64_t *data_in_312 = (uint64_t*)&data_in[54 * count/64]; uint64_t *data_in_313 = (uint64_t*)&data_in[55 * count/64]; uint64_t *data_in_320 = (uint64_t*)&data_in[56 * count/64]; uint64_t *data_in_321 = (uint64_t*)&data_in[57 * count/64]; uint64_t *data_in_322 = (uint64_t*)&data_in[58 * count/64]; uint64_t *data_in_323 = (uint64_t*)&data_in[59 * count/64]; uint64_t *data_in_330 = (uint64_t*)&data_in[60 * count/64]; uint64_t *data_in_331 = (uint64_t*)&data_in[61 * count/64]; uint64_t *data_in_332 = (uint64_t*)&data_in[62 * count/64]; uint64_t *data_in_333 = (uint64_t*)&data_in[63 * count/64]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count/32/2; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 8 * (((rev>>(shift-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift+1)) & 0x5555555555555555)); *(__v2du*)((void*)data_out + offset + 0*16) = (__v2du){data_in_000[i], data_in_100[i]}; *(__v2du*)((void*)data_out + offset + 1*16) = (__v2du){data_in_200[i], data_in_300[i]}; *(__v2du*)((void*)data_out + offset + 2*16) = (__v2du){data_in_010[i], data_in_110[i]}; *(__v2du*)((void*)data_out + offset + 3*16) = (__v2du){data_in_210[i], data_in_310[i]}; *(__v2du*)((void*)data_out + offset + 4*16) = (__v2du){data_in_020[i], data_in_120[i]}; *(__v2du*)((void*)data_out + offset + 5*16) = (__v2du){data_in_220[i], data_in_320[i]}; *(__v2du*)((void*)data_out + offset + 6*16) = (__v2du){data_in_030[i], data_in_130[i]}; *(__v2du*)((void*)data_out + offset + 7*16) = (__v2du){data_in_230[i], data_in_330[i]}; *(__v2du*)((void*)data_out + offset + 8*16) = (__v2du){data_in_001[i], data_in_101[i]}; *(__v2du*)((void*)data_out + offset + 9*16) = (__v2du){data_in_201[i], data_in_301[i]}; *(__v2du*)((void*)data_out + offset + 10*16) = (__v2du){data_in_011[i], data_in_111[i]}; *(__v2du*)((void*)data_out + offset + 11*16) = (__v2du){data_in_211[i], data_in_311[i]}; *(__v2du*)((void*)data_out + offset + 12*16) = (__v2du){data_in_021[i], data_in_121[i]}; *(__v2du*)((void*)data_out + offset + 13*16) = (__v2du){data_in_221[i], data_in_321[i]}; *(__v2du*)((void*)data_out + offset + 14*16) = (__v2du){data_in_031[i], data_in_131[i]}; *(__v2du*)((void*)data_out + offset + 15*16) = (__v2du){data_in_231[i], data_in_331[i]}; } #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count/32/2; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 8 * (((rev>>(shift-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift+1)) & 0x5555555555555555)); *(__v2du*)((void*)data_out + offset + 16*16) = (__v2du){data_in_002[i], data_in_102[i]}; *(__v2du*)((void*)data_out + offset + 17*16) = (__v2du){data_in_202[i], data_in_302[i]}; *(__v2du*)((void*)data_out + offset + 18*16) = (__v2du){data_in_012[i], data_in_112[i]}; *(__v2du*)((void*)data_out + offset + 19*16) = (__v2du){data_in_212[i], data_in_312[i]}; *(__v2du*)((void*)data_out + offset + 20*16) = (__v2du){data_in_022[i], data_in_122[i]}; *(__v2du*)((void*)data_out + offset + 21*16) = (__v2du){data_in_222[i], data_in_322[i]}; *(__v2du*)((void*)data_out + offset + 22*16) = (__v2du){data_in_032[i], data_in_132[i]}; *(__v2du*)((void*)data_out + offset + 23*16) = (__v2du){data_in_232[i], data_in_332[i]}; *(__v2du*)((void*)data_out + offset + 24*16) = (__v2du){data_in_003[i], data_in_103[i]}; *(__v2du*)((void*)data_out + offset + 25*16) = (__v2du){data_in_203[i], data_in_303[i]}; *(__v2du*)((void*)data_out + offset + 26*16) = (__v2du){data_in_013[i], data_in_113[i]}; *(__v2du*)((void*)data_out + offset + 27*16) = (__v2du){data_in_213[i], data_in_313[i]}; *(__v2du*)((void*)data_out + offset + 28*16) = (__v2du){data_in_023[i], data_in_123[i]}; *(__v2du*)((void*)data_out + offset + 29*16) = (__v2du){data_in_223[i], data_in_323[i]}; *(__v2du*)((void*)data_out + offset + 30*16) = (__v2du){data_in_033[i], data_in_133[i]}; *(__v2du*)((void*)data_out + offset + 31*16) = (__v2du){data_in_233[i], data_in_333[i]}; } }
Основной цикл на ассемблере
.L7926: { fapb ct=0, dcd=0, fmt=4, mrng=8, d=17, incr=0, ind=0, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=16, incr=0, ind=0, asz=1, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=15, incr=0, ind=0, asz=1, abs=2, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=14, incr=0, ind=0, asz=1, abs=2, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=13, incr=0, ind=0, asz=1, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=12, incr=0, ind=0, asz=1, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=11, incr=0, ind=0, asz=1, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=10, incr=0, ind=0, asz=1, abs=6, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=9, incr=0, ind=0, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=8, incr=0, ind=0, asz=1, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=7, incr=0, ind=0, asz=1, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=6, incr=0, ind=0, asz=1, abs=10, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=5, incr=0, ind=0, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=4, incr=0, ind=0, asz=1, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=3, incr=0, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=2, incr=0, ind=0, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=1, incr=0, ind=0, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=15, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=14, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=13, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=12, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=11, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=10, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=9, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=8, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=7, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=6, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=5, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=1, abs=30, disp=0 } .L6604: { loop_mode qppackdl,0,sm %b[55], %b[56], %b[1] shr_andd,1,sm %b[62], %r0, %r21, %b[67] stqp,2 %r4, %b[58], %b[14] qppackdl,3,sm %b[26], %b[50], %b[14] shr_andd,4,sm %b[62], %r1, %r20, %b[68] stqp,5 %r2, %b[58], %b[19] movad,0 area=15, ind=0, am=1, be=0, %b[62] movad,1 area=14, ind=0, am=1, be=0, %b[66] movad,2 area=15, ind=0, am=1, be=0, %b[60] movad,3 area=14, ind=0, am=1, be=0, %b[64] } { loop_mode qppackdl,1,sm %b[44], %b[49], %b[31] stqp,2 %r5, %b[58], %b[3] qppackdl,4,sm %b[25], %b[31], %b[26] stqp,5 %r6, %b[58], %b[16] movad,0 area=13, ind=0, am=1, be=0, %b[16] movad,1 area=12, ind=0, am=1, be=0, %b[25] movad,2 area=13, ind=0, am=1, be=0, %b[3] movad,3 area=12, ind=0, am=1, be=0, %b[19] } { loop_mode qppackdl,1,sm %b[63], %b[65], %b[33] stqp,2 %r7, %b[58], %b[33] qppackdl,4,sm %b[59], %b[61], %b[28] stqp,5 %r9, %b[58], %b[28] movad,0 area=11, ind=0, am=1, be=0, %b[49] movad,1 area=10, ind=0, am=1, be=0, %b[55] movad,2 area=11, ind=0, am=1, be=0, %b[44] movad,3 area=10, ind=0, am=1, be=0, %b[50] } { loop_mode qppackdl,1,sm %g18, %g19, %b[37] stqp,2 %r10, %b[58], %b[37] qppackdl,4,sm %g16, %g17, %b[32] stqp,5 %r11, %b[58], %b[32] movad,0 area=9, ind=0, am=1, be=0, %g17 movad,1 area=8, ind=0, am=1, be=0, %g19 movad,2 area=9, ind=0, am=1, be=0, %g16 movad,3 area=8, ind=0, am=1, be=0, %g18 } { loop_mode qppackdl,1,sm %b[52], %b[57], %b[41] stqp,2 %r12, %b[58], %b[41] qppackdl,4,sm %b[46], %b[51], %b[36] stqp,5 %r13, %b[58], %b[36] movad,0 area=7, ind=0, am=1, be=0, %b[59] movad,1 area=6, ind=0, am=1, be=0, %b[63] movad,2 area=7, ind=0, am=1, be=0, %b[57] movad,3 area=6, ind=0, am=1, be=0, %b[61] } { loop_mode addd,0,sm %b[4], 0x1, %b[2] ? %pcnt2 qppackdl,1,sm %b[21], %b[27], %b[5] stqp,2 %r14, %b[58], %b[45] ord,3,sm %b[68], %b[67], %b[65] qppackdl,4,sm %b[5], %b[18], %b[18] stqp,5 %r15, %b[58], %b[40] movad,0 area=5, ind=0, am=1, be=0, %b[27] movad,1 area=4, ind=0, am=1, be=0, %b[45] movad,2 area=5, ind=0, am=1, be=0, %b[21] movad,3 area=4, ind=0, am=1, be=0, %b[40] } { loop_mode bitrevd,0,sm %b[4], %b[60] qppackdl,1,sm %b[64], %b[66], %b[9] stqp,2 %r16, %b[58], %b[9] shld,3,sm %b[65], 0x3, %b[56] qppackdl,4,sm %b[60], %b[62], %b[4] stqp,5 %r17, %b[58], %b[22] movad,0 area=3, ind=0, am=1, be=0, %b[46] movad,1 area=2, ind=0, am=1, be=0, %b[52] movad,2 area=3, ind=0, am=1, be=0, %b[22] movad,3 area=2, ind=0, am=1, be=0, %b[51] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppackdl,0,sm %g22, %g23, %b[10] stqp,2 %r18, %b[58], %b[15] qppackdl,3,sm %g20, %g21, %b[15] stqp,5 %r19, %b[58], %b[10] movad,0 area=1, ind=0, am=1, be=0, %g23 movad,1 area=0, ind=0, am=1, be=0, %g21 movad,2 area=1, ind=0, am=1, be=0, %g22 movad,3 area=0, ind=0, am=1, be=0, %g20 } ... .L7272: { fapb ct=0, dcd=0, fmt=4, mrng=8, d=17, incr=0, ind=0, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=16, incr=0, ind=0, asz=1, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=15, incr=0, ind=0, asz=1, abs=2, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=14, incr=0, ind=0, asz=1, abs=2, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=13, incr=0, ind=0, asz=1, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=12, incr=0, ind=0, asz=1, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=11, incr=0, ind=0, asz=1, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=10, incr=0, ind=0, asz=1, abs=6, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=9, incr=0, ind=0, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=8, incr=0, ind=0, asz=1, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=7, incr=0, ind=0, asz=1, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=6, incr=0, ind=0, asz=1, abs=10, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=5, incr=0, ind=0, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=4, incr=0, ind=0, asz=1, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=3, incr=0, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=2, incr=0, ind=0, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=1, incr=0, ind=0, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=15, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=14, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=13, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=12, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=11, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=10, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=9, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=8, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=7, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=6, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=5, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=1, asz=1, abs=30, disp=0 } .L6518: { loop_mode qppackdl,0,sm %b[55], %b[56], %b[1] shr_andd,1,sm %b[62], %r0, %r21, %b[67] stqp,2 %r4, %b[58], %b[19] qppackdl,3,sm %b[26], %b[50], %b[14] shr_andd,4,sm %b[62], %r1, %r20, %b[68] stqp,5 %r2, %b[58], %b[14] movad,0 area=15, ind=0, am=1, be=0, %b[62] movad,1 area=14, ind=0, am=1, be=0, %b[66] movad,2 area=15, ind=0, am=1, be=0, %b[60] movad,3 area=14, ind=0, am=1, be=0, %b[64] } { loop_mode qppackdl,1,sm %b[44], %b[49], %b[26] stqp,2 %r5, %b[58], %b[3] qppackdl,4,sm %b[25], %b[31], %b[31] stqp,5 %r6, %b[58], %b[16] movad,0 area=13, ind=0, am=1, be=0, %b[16] movad,1 area=12, ind=0, am=1, be=0, %b[25] movad,2 area=13, ind=0, am=1, be=0, %b[3] movad,3 area=12, ind=0, am=1, be=0, %b[19] } { loop_mode qppackdl,1,sm %b[63], %b[65], %b[33] stqp,2 %r7, %b[58], %b[28] qppackdl,4,sm %b[59], %b[61], %b[28] stqp,5 %r9, %b[58], %b[33] movad,0 area=11, ind=0, am=1, be=0, %b[49] movad,1 area=10, ind=0, am=1, be=0, %b[55] movad,2 area=11, ind=0, am=1, be=0, %b[44] movad,3 area=10, ind=0, am=1, be=0, %b[50] } { loop_mode qppackdl,1,sm %g18, %g19, %b[37] stqp,2 %r10, %b[58], %b[37] qppackdl,4,sm %g16, %g17, %b[32] stqp,5 %r11, %b[58], %b[32] movad,0 area=9, ind=0, am=1, be=0, %g17 movad,1 area=8, ind=0, am=1, be=0, %g19 movad,2 area=9, ind=0, am=1, be=0, %g16 movad,3 area=8, ind=0, am=1, be=0, %g18 } { loop_mode qppackdl,1,sm %b[52], %b[57], %b[36] stqp,2 %r12, %b[58], %b[41] qppackdl,4,sm %b[46], %b[51], %b[41] stqp,5 %r13, %b[58], %b[36] movad,0 area=7, ind=0, am=1, be=0, %b[59] movad,1 area=6, ind=0, am=1, be=0, %b[63] movad,2 area=7, ind=0, am=1, be=0, %b[57] movad,3 area=6, ind=0, am=1, be=0, %b[61] } { loop_mode addd,0,sm %b[4], 0x1, %b[2] ? %pcnt2 qppackdl,1,sm %b[21], %b[27], %b[5] stqp,2 %r14, %b[58], %b[40] ord,3,sm %b[68], %b[67], %b[65] qppackdl,4,sm %b[5], %b[18], %b[18] stqp,5 %r15, %b[58], %b[45] movad,0 area=5, ind=0, am=1, be=0, %b[27] movad,1 area=4, ind=0, am=1, be=0, %b[45] movad,2 area=5, ind=0, am=1, be=0, %b[21] movad,3 area=4, ind=0, am=1, be=0, %b[40] } { loop_mode bitrevd,0,sm %b[4], %b[60] qppackdl,1,sm %b[64], %b[66], %b[4] stqp,2 %r16, %b[58], %b[9] shld,3,sm %b[65], 0x3, %b[56] qppackdl,4,sm %b[60], %b[62], %b[9] stqp,5 %r17, %b[58], %b[22] movad,0 area=3, ind=0, am=1, be=0, %b[46] movad,1 area=2, ind=0, am=1, be=0, %b[52] movad,2 area=3, ind=0, am=1, be=0, %b[22] movad,3 area=2, ind=0, am=1, be=0, %b[51] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppackdl,0,sm %g22, %g23, %b[15] stqp,2 %r18, %b[58], %b[10] qppackdl,3,sm %g20, %g21, %b[10] stqp,5 %r19, %b[58], %b[15] movad,0 area=1, ind=0, am=1, be=0, %g23 movad,1 area=0, ind=0, am=1, be=0, %g21 movad,2 area=1, ind=0, am=1, be=0, %g22 movad,3 area=0, ind=0, am=1, be=0, %g20 }
Теоретическая скорость: 32 комплексных числа за 8 тактов (32/8) = 32 Байт/такт
Замеры скорости
Видим замедление в начале и ускорение в конце графика.
Накладные расходы на организацию второго цикла не дают проявиться ускорению по всей длине графика.
Итоги по reverse_radix4
Победителем можно считать либо reverse_radix4_x16, либо reverse_radix4_x32.
Алгоритм FFT состоит из одного запуска Reverse и нескольких запусков Stage. Чем больше запусков Stage, тем меньший вклад вносит скорость Reverse в итоговую скорость FFT. Поэтому скорость Reverse важнее на меньших длинах входных данных, где меньше запусков Stage.
При реализации Radix-4 FFT будем использовать reverse_radix4_x16.
Потом можно заменить на reverse_radix4_x32 и посмотреть, как изменится скорость FFT.
Пишем функцию Stage
stage_radix2
Схема алгоритма Stage для версии «radix-2».
1. stage_radix2_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void stage_radix2_etalon(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef) { myComplex *x_in = &data_in[0]; myComplex *y_in = &data_in[1]; myComplex *c_in = coef; myComplex *out_add = &data_out[0]; myComplex *out_sub = &data_out[data_count/2]; #pragma ivdep #pragma unroll(1) // #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { myComplex x = x_in[2*i]; myComplex y = y_in[2*i]; myComplex c = c_in[i]; myComplex cy = complex_mul(c, y); out_add[i] = complex_add(x, cy); out_sub[i] = complex_sub(x, cy); } }
Основной цикл на ассемблере
.L444: { fapb ct=1, dcd=0, fmt=3, mrng=16, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=2, asz=5, abs=0, disp=0 } .L120: { loop_mode fmuls,0,sm %b[67], %b[6], %b[37] fsubs,1,sm %b[46], %b[24], %b[58] staaw,2 %b[62], %aad1[ %aasti3 + _f32s,_lts0 0x4 ] fmul_adds,3,sm %b[55], %b[13], %b[43], %b[14] fadds,4,sm %b[46], %b[24], %b[57] staaw,5 %b[61], %aad2[ %aasti4 + _f32s,_lts0 0x4 ] movaw,0 area=0, ind=8, am=0, be=0, %b[0] movaw,1 area=0, ind=12, am=0, be=0, %b[1] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END fmuls,0,sm %b[67], %b[7], %b[62] fsubs,1,sm %b[35], %b[56], %b[70] staaw,2 %b[74], %aad1[ %aasti3 ] incr,2 %aaincr3 fmul_rsubs,3,sm %b[55], %b[12], %b[68], %b[46] fadds,4,sm %b[35], %b[56], %b[69] staaw,5 %b[73], %aad2[ %aasti4 ] incr,5 %aaincr3 movaw,0 area=0, ind=0, am=0, be=0, %b[13] movaw,1 area=0, ind=4, am=1, be=0, %b[24] movaw,2 area=0, ind=4, am=1, be=0, %b[61] movaw,3 area=0, ind=0, am=0, be=0, %b[43] }
Теоретическая скорость: 2 комплексных числа за 2 такта (2/2) = 8 Байт/такт
Замеры скорости
2. stage_radix2_etalon_unroll2
Этот вариант появился случайно.
Здесь происходит раскрутка цикла в 2 раза с помощью опции unroll.
Можно видеть, что компилятор умеет использовать векторные инструкции.
Код на Си
void stage_radix2_etalon_unroll2(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef) { myComplex *x_in = &data_in[0]; myComplex *y_in = &data_in[1]; myComplex *c_in = coef; myComplex *out_add = &data_out[0]; myComplex *out_sub = &data_out[data_count/2]; #pragma ivdep #pragma unroll(2) // #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { myComplex x = x_in[2*i]; myComplex y = y_in[2*i]; myComplex c = c_in[i]; myComplex cy = complex_mul(c, y); out_add[i] = complex_add(x, cy); out_sub[i] = complex_sub(x, cy); } }
Основной цикл на ассемблере
.L1266: { fapb ct=0, dcd=0, fmt=3, mrng=12, d=0, incr=1, ind=1, asz=4, abs=0, disp=20 fapb dpl=0, dcd=0, fmt=3, mrng=20, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=16, disp=0 } .L463: { loop_mode pfmuls,0,sm %b[51], %b[32], %b[53] insfd,1,sm %b[28], %r8, %b[54], %b[1] pfmuls,2,sm %b[51], %b[13], %b[45] insfd,3,sm %b[23], %r8, %b[50], %b[0] pshufb,4,sm %b[9], %b[19], %r0, %b[56] pfadds,5,sm %b[33], %b[39], %b[10] movaw,1 area=0, ind=8, am=0, be=0, %b[38] movaw,3 area=0, ind=12, am=0, be=0, %b[44] } { loop_mode pfmul_rsubs,0,sm %b[5], %b[15], %b[55], %b[39] insfd,1,sm %b[20], %r8, %b[24], %b[23] pfmul_adds,2,sm %b[5], %b[34], %b[47], %b[33] insfd,3,sm %b[40], %r8, %b[46], %b[28] pshufb,4,sm %b[12], %b[16], %r0, %b[54] staad,5 %b[56], %aad1[ %aasti3 + _f32s,_lts0 0x8 ] movad,1 area=1, ind=0, am=0, be=0, %b[50] } { loop_mode pfsubs,0,sm %b[8], %b[43], %b[15] insfd,1,sm %b[9], %r8, %b[19], %b[57] pfsubs,2,sm %b[31], %b[37], %b[5] insfd,3,sm %b[12], %r8, %b[16], %b[55] staad,5 %b[54], %aad2[ %aasti4 + _f32s,_lts0 0x8 ] movad,0 area=1, ind=8, am=1, be=0, %b[24] movaw,1 area=0, ind=4, am=0, be=0, %b[34] movaw,2 area=0, ind=4, am=0, be=0, %b[20] movaw,3 area=0, ind=8, am=0, be=0, %b[40] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfadds,0,sm %b[8], %b[43], %b[12] insfd,1,sm %b[36], %r8, %b[42], %b[9] staad,2 %b[57], %aad1[ %aasti3 ] incr,2 %aaincr3 pshufb,4,sm %b[26], %b[52], %r0, %b[47] staad,5 %b[55], %aad2[ %aasti4 ] incr,5 %aaincr3 movaw,1 area=0, ind=0, am=1, be=0, %b[16] movaw,2 area=0, ind=0, am=1, be=0, %b[46] movaw,3 area=0, ind=16, am=0, be=0, %b[19] }
Теоретическая скорость: 4 комплексных числа за 4 такта (4/4) = 8 Байт/такт
Замеры скорости
Видим ускорение.
Теоретическая скорость не изменилась по сравнению с эталонным вариантом, но скорость выросла. В ассемблерном коде можно видеть, что компилятор вставил векторные инструкции.
3. stage_radix2_simd64
Прямую векторизацию сейчас пробовать не будем. Её посмотрим потом отдельно.
Сейчас попробуем использовать векторные инструкции SIMD64 для выполнения нескольких умножений одной инструкцией.
Умножение двух комплексных чисел c и y будем делать так:
читаем комплексные числа c и y из памяти в 64-битные регистры (в одну половину регистра попадает действительная часть, в другую половину — мнимая часть)
меняем знак у мнимой части c с помощью
xor(получаем conj_c) и перемножаем векторно conj_c и y — получаем полуфабрикат для действительной части cy (для завершения получения действительной части cy надо сложить половины регистра)меняем местами действительную и мнимую части c с помощью
shuf(получаем swap_c) и перемножаем векторно swap_c и y — получаем полуфабрикат для мнимой части cy (для завершения получения мнимой части cy надо сложить половины регистра)складываем половины регистров‑полуфабрикатов с помощью векторного горизонтального сложения
fhadd— получаем cy
Код на Си
void stage_radix2_simd64(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef) { uint64_t *x_in = (uint64_t*)&data_in[0]; uint64_t *y_in = (uint64_t*)&data_in[1]; uint64_t *c_in = (uint64_t*)coef; uint64_t *out_add = (uint64_t*)&data_out[0]; uint64_t *out_sub = (uint64_t*)&data_out[data_count/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { uint64_t x = x_in[2*i]; uint64_t y = y_in[2*i]; uint64_t c = c_in[i]; uint64_t conj_c = __builtin_e2k_pxord(c, 1LL<<63); uint64_t swap_c = __builtin_e2k_pshufb(0, c, 0x0302010007060504); uint64_t cy_real = __builtin_e2k_pfmuls(conj_c, y); uint64_t cy_imag = __builtin_e2k_pfmuls(swap_c, y); uint64_t cy = __builtin_e2k_pfhadds(cy_real, cy_imag); out_add[i] = __builtin_e2k_pfadds(x, cy); out_sub[i] = __builtin_e2k_pfsubs(x, cy); } }
Основной цикл на ассемблере
.L1588: { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=5, abs=0, disp=0 } .L1388: { loop_mode pfmuls,0,sm %b[35], %b[28], %b[18] pfmul_hadds,1,sm %b[33], %b[32], %b[22], %b[0] pshufb,4,sm 0x0, %b[7], %r5, %b[29] pfadds,5,sm %b[27], %b[10], %b[12] movad,3 area=0, ind=0, am=1, be=0, %b[1] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END xord,0,sm %b[5], %r0, %b[33] pfsubs,1,sm %b[27], %b[10], %b[32] staad,2 %b[36], %aad1[ %aasti3 ] incr,2 %aaincr0 staad,5 %b[16], %aad2[ %aasti4 ] incr,5 %aaincr0 movad,0 area=0, ind=8, am=1, be=0, %b[22] movad,1 area=0, ind=0, am=0, be=0, %b[7] }
После компиляции видим, что цикл состоит из 8 инструкций: xor, shuf, fmul, fmul_fhadd, fadd, fsub, std, std. Инструкция fhadd оказалась «сцеплена» с одной из инструкций fmul (оказывается, Эльбрус так умеет).
Теоретическая скорость: 2 комплексных числа за 2 такта (2/2) = 8 Байт/такт
Замеры скорости
Видим небольшое ускорение.
В одном такте помещается 6 инструкций, а у нас здесь 8 инструкций. Т.е. у нас занято 8/6 такта. В идеале, если раскрутить цикл в 3 раза, получится самая плотная упаковка (3 * 8/6 = 4 такта). Раскручивать будем с помощью опции unroll.
Но сначала посмотрим на раскрутку в 2 раза (2 * 8/6 = 3 такта).
4. stage_radix2_simd64_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью опции unroll.
Код на Си
void stage_radix2_simd64_unroll2(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef) { uint64_t *x_in = (uint64_t*)&data_in[0]; uint64_t *y_in = (uint64_t*)&data_in[1]; uint64_t *c_in = (uint64_t*)coef; uint64_t *out_add = (uint64_t*)&data_out[0]; uint64_t *out_sub = (uint64_t*)&data_out[data_count/2]; #pragma ivdep #pragma unroll(2) #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { uint64_t x = x_in[2*i]; uint64_t y = y_in[2*i]; uint64_t c = c_in[i]; uint64_t conj_c = __builtin_e2k_pxord(c, 1LL<<63); uint64_t swap_c = __builtin_e2k_pshufb(0, c, 0x0302010007060504); uint64_t cy_real = __builtin_e2k_pfmuls(conj_c, y); uint64_t cy_imag = __builtin_e2k_pfmuls(swap_c, y); uint64_t cy = __builtin_e2k_pfhadds(cy_real, cy_imag); out_add[i] = __builtin_e2k_pfadds(x, cy); out_sub[i] = __builtin_e2k_pfsubs(x, cy); } }
Основной цикл на ассемблере
.L2152: { fapb ct=1, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=5, abs=0, disp=0 } .L1710: { loop_mode pfmul_hadds,0,sm %b[51], %b[10], %b[36], %b[11] pfmuls,1,sm %b[57], %b[7], %b[44] pfsubs,2,sm %b[41], %b[25], %b[45] xord,4,sm %b[42], %r0, %b[52] pfadds,5,sm %b[41], %b[25], %b[48] movad,0 area=0, ind=24, am=0, be=0, %b[1] movad,1 area=0, ind=8, am=0, be=0, %b[0] } { loop_mode pshufb,0,sm 0x0, %b[32], %r9, %b[41] pfsubs,1,sm %b[26], %b[17], %b[51] staad,2 %b[47], %aad1[ %aasti3 + _f32s,_lts0 0x8 ] pfadds,3,sm %b[26], %b[17], %b[54] xord,4,sm %b[30], %r0, %b[55] staad,5 %b[50], %aad2[ %aasti4 + _f32s,_lts0 0x8 ] movad,0 area=0, ind=0, am=1, be=0, %b[10] movad,1 area=0, ind=16, am=0, be=0, %b[25] movad,3 area=0, ind=0, am=0, be=0, %b[36] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmul_hadds,0,sm %b[43], %b[9], %b[46], %b[17] pfmuls,1,sm %b[52], %b[6], %b[32] staad,2 %b[53], %aad1[ %aasti3 ] incr,2 %aaincr3 pshufb,4,sm 0x0, %b[42], %r9, %b[47] staad,5 %b[56], %aad2[ %aasti4 ] incr,5 %aaincr3 movad,3 area=0, ind=8, am=1, be=0, %b[26] }
Теоретическая скорость: 4 комплексных числа за 3 такта (4/3) = 10.67 Байт/такт
Замеры скорости
Видим ускорение.
Теперь посмотрим на раскрутку в 3 раза.
5. stage_radix2_simd64_unroll3
Здесь происходит раскрутка цикла в 3 раза с помощью опции unroll.
Код на Си
void stage_radix2_simd64_unroll3(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef) { uint64_t *x_in = (uint64_t*)&data_in[0]; uint64_t *y_in = (uint64_t*)&data_in[1]; uint64_t *c_in = (uint64_t*)coef; uint64_t *out_add = (uint64_t*)&data_out[0]; uint64_t *out_sub = (uint64_t*)&data_out[data_count/2]; #pragma ivdep #pragma unroll(3) #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { uint64_t x = x_in[2*i]; uint64_t y = y_in[2*i]; uint64_t c = c_in[i]; uint64_t conj_c = __builtin_e2k_pxord(c, 1LL<<63); uint64_t swap_c = __builtin_e2k_pshufb(0, c, 0x0302010007060504); uint64_t cy_real = __builtin_e2k_pfmuls(conj_c, y); uint64_t cy_imag = __builtin_e2k_pfmuls(swap_c, y); uint64_t cy = __builtin_e2k_pfhadds(cy_real, cy_imag); out_add[i] = __builtin_e2k_pfadds(x, cy); out_sub[i] = __builtin_e2k_pfsubs(x, cy); } }
Основной цикл на ассемблере
.L2815: { fapb ct=0, dcd=0, fmt=4, mrng=24, d=0, incr=2, ind=2, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=1, ind=1, asz=4, abs=16, disp=32 } .L2177: { loop_mode pfmuls,0,sm %b[61], %b[23], %b[41] pfmuls,1,sm %b[73], %b[12], %b[1] xord,2,sm %b[57], %r0, %b[59] pfmul_hadds,3,sm %b[78], %b[53], %b[28], %b[0] xord,4,sm %b[52], %r0, %b[66] pfadds,5,sm %b[48], %b[34], %b[58] } { loop_mode pfmul_hadds,0,sm %b[67], %b[25], %b[43], %b[28] pfsubs,1,sm %b[48], %b[34], %b[68] staad,2 %b[70], %aad1[ %aasti3 + _f32s,_lts0 0x10 ] pfsubs,3,sm %b[17], %b[4], %b[61] xord,4,sm %b[20], %r0, %b[71] staad,5 %b[60], %aad2[ %aasti4 + _f32s,_lts0 0x10 ] movad,1 area=0, ind=16, am=0, be=0, %b[53] } { loop_mode pfmul_hadds,0,sm %b[72], %b[14], %b[3], %b[60] pfsubs,1,sm %b[39], %b[64], %b[73] staad,2 %b[75], %aad1[ %aasti3 + _f32s,_lts0 0x8 ] pfadds,3,sm %b[17], %b[4], %b[67] pshufb,4,sm 0x0, %b[22], %r10, %b[70] staad,5 %b[63], %aad1[ %aasti3 ] incr,5 %aaincr3 movad,0 area=0, ind=0, am=0, be=0, %b[48] movad,1 area=1, ind=0, am=0, be=0, %b[34] movad,2 area=0, ind=16, am=0, be=0, %b[25] movad,3 area=0, ind=8, am=0, be=0, %b[43] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmuls,0,sm %b[66], %b[47], %b[22] pfadds,1,sm %b[39], %b[64], %b[72] staad,2 %b[74], %aad2[ %aasti4 + _f32s,_lts0 0x8 ] pshufb,3,sm 0x0, %b[57], %r10, %b[63] pshufb,4,sm 0x0, %b[56], %r10, %b[76] staad,5 %b[69], %aad2[ %aasti4 ] incr,5 %aaincr3 movad,0 area=1, ind=8, am=1, be=0, %b[17] movad,1 area=0, ind=8, am=1, be=0, %b[14] movad,2 area=0, ind=0, am=1, be=0, %b[3] movad,3 area=0, ind=24, am=0, be=0, %b[4] }
Теоретическая скорость: 6 комплексных чисел за 4 такта (6/4) = 12 Байт/такт
Замеры скорости
Видим ускорение в середине графика, но замедление в начале и в конце графика.
6. stage_radix2_simd128
Теперь попробуем использовать векторные инструкции SIMD128 по аналогии с SIMD64.
В отличие от SIMD64, здесь придётся перетасовать данные в начале и в конце цикла с помощью инструкции shuf. Это нужно для того, чтобы в одном 128-битном регистре оказались данные, относящиеся к двум числам x, а в другом — данные, относящиеся к двум числам y.
Код на Си
void stage_radix2_simd128(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *xy1_in = (__v2di*)&data_in[2]; __v2di *c_in = (__v2di*)coef; __v2di *out_add = (__v2di*)&data_out[0]; __v2di *out_sub = (__v2di*)&data_out[data_count/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { __v2di xy0 = xy0_in[2*i]; __v2di xy1 = xy1_in[2*i]; __v2di c = c_in[i]; __v2di x = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di conj_c = __builtin_e2k_qpxor(c, (__v2di){1LL<<63, 1LL<<63}); __v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy_real = __builtin_e2k_qpfmuls(conj_c, y); __v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y); __v2di cy_rrii = __builtin_e2k_qpfhadds(cy_real, cy_imag); __v2di cy = __builtin_e2k_qpshufb(cy_rrii, cy_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); out_add[i] = __builtin_e2k_qpfadds(x, cy); out_sub[i] = __builtin_e2k_qpfsubs(x, cy); } }
Основной цикл на ассемблере
.L3099: { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=5, abs=0, disp=0 } .L2840: { loop_mode qpshufb,0,sm %b[36], %b[45], %r6, %b[0] qpfmuls,1,sm %b[28], %b[5], %b[18] qpfsubs,2,sm %b[16], %b[47], %b[11] qpshufb,3,sm %b[34], %b[43], %r7, %b[1] qpxor,4,sm %b[23], %r0, %b[24] staaqp,5 %b[17], %aad1[ %aasti3 ] incr,5 %aaincr0 } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpshufb,0,sm %b[35], %b[35], %r5, %b[45] qpfmul_hadds,1,sm %b[42], %b[9], %b[22], %b[27] qpfadds,2,sm %b[16], %b[47], %b[44] qpshufb,4,sm %b[25], %b[25], %r9, %b[36] staaqp,5 %b[50], %aad2[ %aasti4 ] incr,5 %aaincr0 movaqp,0 area=0, ind=0, am=0, be=0, %b[37] movaqp,1 area=0, ind=16, am=1, be=0, %b[28] movaqp,3 area=0, ind=0, am=1, be=0, %b[17] }
После компиляции видим, что цикл состоит из 11 инструкций (такие же 8 инструкций, что были в варианте SIMD64, и ещё 3 дополнительные инструкции shuf).
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
Видим ускорение.
Прежде, чем переходить к раскрутке, заметим, что можно сделать цикл на одну инструкцию меньше. В дальнейшем это позволит выполнить более эффективную раскрутку.
7. stage_radix2_simd128_noConj
Воспользуется тем, что Эльбрус умеет сцеплять некоторые инструкции.
Откажемся от создания conj_c (убрали инструкцию xor) и будем использовать fhsub для получения действительной части cy из полуфабриката. Мнимую часть будем, как и раньше, получать с помощью fhadd. Обе эти инструкции будут сцеплены с двумя fmul, то есть будут «бесплатными». Итоговое соединение в единое комплексное число будет сделано в финальном shuf одновременно с уже имеющейся перетасовкой данных.
В версии SIMD64 такой приём сделать было нельзя, потому что там не было финального shuf.
(финальный shuf пришлось заменить на perm, инструкции аналогичны друг другу)
Код на Си
void stage_radix2_simd128_noConj(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *xy1_in = (__v2di*)&data_in[2]; __v2di *c_in = (__v2di*)coef; __v2di *out_add = (__v2di*)&data_out[0]; __v2di *out_sub = (__v2di*)&data_out[data_count/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { __v2di xy0 = xy0_in[2*i]; __v2di xy1 = xy1_in[2*i]; __v2di c = c_in[i]; __v2di x = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy_real = __builtin_e2k_qpfmuls( c, y); __v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y); __v2di cy_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy_real); __v2di cy_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy_imag); __v2di cy = __builtin_e2k_qppermb(cy_ii, cy_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); out_add[i] = __builtin_e2k_qpfadds(x, cy); out_sub[i] = __builtin_e2k_qpfsubs(x, cy); } }
Основной цикл на ассемблере
.L3385: { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=5, abs=0, disp=0 } .L3124: { loop_mode qpfmul_hsubs,0,sm %b[25], %b[28], %r9, %b[16] qpfmul_hadds,1,sm %b[27], %b[28], %r9, %b[1] qpfsubs,2,sm %b[14], %b[42], %b[37] qpshufb,3,sm %b[35], %b[36], %r6, %b[0] qppermb,4,sm %b[11], %b[26], %r7, %b[38] staaqp,5 %b[43], %aad1[ %aasti3 ] incr,5 %aaincr0 movaqp,0 area=0, ind=0, am=0, be=0, %b[30] movaqp,1 area=0, ind=16, am=1, be=0, %b[29] movaqp,3 area=0, ind=0, am=1, be=0, %b[19] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpshufb,0,sm %b[33], %b[34], %r0, %b[26] qpshufb,1,sm %b[23], %b[23], %r5, %b[25] qpfadds,2,sm %b[14], %b[42], %b[11] staaqp,5 %b[17], %aad2[ %aasti4 ] incr,5 %aaincr0 }
После компиляции видим, что цикл состоит из 10 инструкций (убрали инструкцию xor).
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
Скорость не изменилась по сравнению с предыдущим вариантом.
И теперь переходим к раскрутке.
Сейчас занято 10/6 такта. Раскрутка в 2 раза даст 2 * 10/6 = 4 такта, то есть ничего интересного (одна итерация цикла обработает в 2 раза больше данных за в 2 раза большее число тактов).
Поэтому сразу переходим к раскрутке в 3 раза (3 * 10/6 = 5 тактов).
8. stage_radix2_simd128_noConj_unroll3
Здесь происходит раскрутка цикла в 3 раза с помощью опции unroll.
Код на Си
void stage_radix2_simd128_noConj_unroll3(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *xy1_in = (__v2di*)&data_in[2]; __v2di *c_in = (__v2di*)coef; __v2di *out_add = (__v2di*)&data_out[0]; __v2di *out_sub = (__v2di*)&data_out[data_count/2]; #pragma ivdep #pragma unroll(3) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { __v2di xy0 = xy0_in[2*i]; __v2di xy1 = xy1_in[2*i]; __v2di c = c_in[i]; __v2di x = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy_real = __builtin_e2k_qpfmuls( c, y); __v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y); __v2di cy_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy_real); __v2di cy_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy_imag); __v2di cy = __builtin_e2k_qppermb(cy_ii, cy_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); out_add[i] = __builtin_e2k_qpfadds(x, cy); out_sub[i] = __builtin_e2k_qpfsubs(x, cy); } }
Основной цикл на ассемблере
.L3932: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=2, asz=4, abs=16, disp=32 } .L3410: { loop_mode qpfmul_hsubs,0,sm %b[19], %b[55], %r14, %b[0] qpshufb,1,sm %b[26], %b[23], %r12, %b[15] qpfmul_hsubs,2,sm %b[31], %b[63], %r14, %b[1] qpshufb,3,sm %b[52], %b[53], %r0, %b[18] qpfadds,4,sm %b[66], %b[69], %b[6] qpfadds,5,sm %b[64], %b[42], %b[68] } { loop_mode qpfmul_hsubs,0,sm %b[34], %b[15], %r14, %b[52] qpshufb,1,sm %b[34], %b[34], %r11, %b[61] staaqp,2 %b[35], %aad1[ %aasti3 + _f32s,_lts0 0x10 ] qpshufb,3,sm %b[44], %b[45], %r12, %b[53] qpfsubs,4,sm %b[62], %b[40], %b[57] qpfsubs,5,sm %b[18], %b[65], %b[58] movaqp,0 area=0, ind=0, am=0, be=0, %b[43] movaqp,1 area=0, ind=16, am=1, be=0, %b[42] } { loop_mode qpfmul_hadds,0,sm %b[61], %b[15], %r14, %b[34] qpshufb,1,sm %b[31], %b[31], %r11, %b[66] staaqp,2 %b[60], %aad1[ %aasti3 ] qpshufb,3,sm %b[30], %b[27], %r0, %b[64] qppermb,4,sm %b[38], %b[56], %r13, %b[67] qpfadds,5,sm %b[18], %b[65], %b[35] } { loop_mode qpfmul_hadds,0,sm %b[66], %b[63], %r14, %b[18] qpshufb,1,sm %b[19], %b[19], %r11, %b[56] staaqp,2 %b[8], %aad2[ %aasti4 + _f32s,_lts0 0x10 ] qppermb,3,sm %b[22], %b[5], %r13, %b[38] qpfsubs,4,sm %b[64], %b[67], %b[31] staaqp,5 %b[37], %aad2[ %aasti4 ] movaqp,1 area=2, ind=0, am=1, be=0, %b[27] movaqp,2 area=1, ind=16, am=1, be=0, %b[30] movaqp,3 area=1, ind=0, am=0, be=0, %b[15] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfmul_hadds,0,sm %b[56], %b[55], %r14, %b[37] qpshufb,1,sm %b[10], %b[7], %r12, %b[61] staaqp,2 %b[59], %aad1[ %aasti3 + _f32s,_lts0 0x20 ] incr,2 %aaincr3 qpshufb,3,sm %b[16], %b[13], %r0, %b[60] qppermb,4,sm %b[41], %b[4], %r13, %b[63] staaqp,5 %b[68], %aad2[ %aasti4 + _f32s,_lts0 0x20 ] incr,5 %aaincr3 movaqp,0 area=1, ind=0, am=0, be=0, %b[5] movaqp,1 area=1, ind=16, am=1, be=0, %b[8] movaqp,2 area=0, ind=0, am=0, be=0, %b[19] movaqp,3 area=0, ind=16, am=1, be=0, %b[22] }
Теоретическая скорость: 12 комплексных чисел за 5 тактов (12/5) = 19.2 Байт/такт
Замеры скорости
Видим ускорение в середине графика, но замедление в начале и в конце графика.
Итоги по stage_radix2
График FFT находится здесь.
stage_radix2_2x
Схема алгоритма Stage для версии «radix-2» 2x.
Один проход по stage_radix2_2x совершает ту же работу, что 2 прохода по stage_radix2. Поэтому скорость stage_radix2_2x будем умножать на 2 для удобства сравнения с stage_radix2 (этот факт подписан на оси графика и в выводе консоли).
1. stage_radix2_2x_etalon
Здесь происходит ручная раскрутка алгоритма stage_radix2_etalon в 2 раза.
Код на Си
void stage_radix2_2x_etalon(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef_a, myComplex *coef_b) { myComplex *x0_in = &data_in[0]; myComplex *y0_in = &data_in[1]; myComplex *x1_in = &data_in[2]; myComplex *y1_in = &data_in[3]; myComplex *c0a_in = &coef_a[0]; myComplex *c1a_in = &coef_a[1]; myComplex *c0b_in = &coef_b[0]; myComplex *c1b_in = &coef_b[data_count/4]; myComplex *out_add0 = &data_out[0*data_count/4]; myComplex *out_add1 = &data_out[1*data_count/4]; myComplex *out_sub0 = &data_out[2*data_count/4]; myComplex *out_sub1 = &data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) // #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { myComplex x0 = x0_in[4*i]; myComplex y0 = y0_in[4*i]; myComplex c0 = c0a_in[2*i]; myComplex x1 = x1_in[4*i]; myComplex y1 = y1_in[4*i]; myComplex c1 = c1a_in[2*i]; myComplex cy0 = complex_mul(c0, y0); myComplex cy1 = complex_mul(c1, y1); myComplex add0 = complex_add(x0, cy0); myComplex sub0 = complex_sub(x0, cy0); myComplex add1 = complex_add(x1, cy1); myComplex sub1 = complex_sub(x1, cy1); x0 = add0; y0 = add1; c0 = c0b_in[i]; x1 = sub0; y1 = sub1; c1 = c1b_in[i]; cy0 = complex_mul(c0, y0); cy1 = complex_mul(c1, y1); out_add0[i] = complex_add(x0, cy0); out_sub0[i] = complex_sub(x0, cy0); out_add1[i] = complex_add(x1, cy1); out_sub1[i] = complex_sub(x1, cy1); } }
Основной цикл на ассемблере
.L965: { fapb ct=0, dcd=0, fmt=3, mrng=16, d=0, incr=3, ind=2, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=4, asz=3, abs=8, disp=0 } { fapb ct=1, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=3, asz=4, abs=16, disp=0 } .L259: { loop_mode fmul_adds,0,sm %b[27], %b[84], %b[93], %b[45] fsub_adds,1,sm %b[15], %b[79], %b[88], %b[0] fsub_rsubs,2,sm %b[15], %b[79], %b[88], %b[1] fmuls,3,sm %b[56], %b[74], %b[90] fmuls,4,sm %b[45], %b[34], %b[89] fmuls,5,sm %b[68], %b[65], %b[88] } { loop_mode fmul_rsubs,0,sm %b[24], %b[67], %b[95], %b[56] fadd_adds,1,sm %b[15], %b[79], %b[47], %b[24] fadd_rsubs,2,sm %b[15], %b[79], %b[47], %b[47] fmul_rsubs,3,sm %b[73], %b[74], %b[94], %b[15] fmuls,5,sm %b[63], %b[51], %b[91] } { loop_mode fmul_rsubs,0,sm %b[27], %b[53], %b[96], %b[58] fsub_adds,1,sm %b[14], %b[83], %b[58], %b[27] fsub_rsubs,2,sm %b[14], %b[83], %b[58], %b[53] fmuls,4,sm %b[54], %b[50], %b[92] fmuls,5,sm %b[68], %b[85], %b[93] } { loop_mode fadd_adds,0,sm %b[14], %b[83], %b[60], %b[67] fadd_rsubs,1,sm %b[14], %b[83], %b[60], %b[68] staaw,2 %b[2], %aad2[ %aasti6 + _f32s,_lts0 0x4 ] fsubs,3,sm %b[48], %b[17], %b[63] fmuls,4,sm %b[63], %b[82], %b[94] staaw,5 %b[3], %aad1[ %aasti5 + _f32s,_lts0 0x4 ] movaw,0 area=2, ind=0, am=0, be=0, %b[14] movaw,1 area=2, ind=4, am=1, be=0, %b[60] movaw,2 area=0, ind=0, am=0, be=0, %b[2] movaw,3 area=0, ind=4, am=0, be=0, %b[3] } { loop_mode staaw,2 %b[26], %aad4[ %aasti8 + _f32s,_lts0 0x4 ] fmul_adds,3,sm %b[73], %b[52], %b[90], %b[73] fadds,4,sm %b[48], %b[17], %b[49] staaw,5 %b[49], %aad3[ %aasti7 + _f32s,_lts0 0x4 ] movaw,0 area=1, ind=0, am=0, be=0, %b[17] movaw,1 area=0, ind=12, am=0, be=0, %b[52] movaw,2 area=0, ind=8, am=0, be=0, %b[26] movaw,3 area=0, ind=28, am=0, be=0, %b[48] } { loop_mode fmul_rsubs,1,sm %b[42], %b[34], %b[87], %b[79] staaw,2 %b[29], %aad2[ %aasti6 ] incr,2 %aaincr4 fsubs,4,sm %b[80], %b[75], %b[83] staaw,5 %b[55], %aad1[ %aasti5 ] incr,5 %aaincr4 movaw,0 area=1, ind=4, am=1, be=0, %b[55] movaw,1 area=0, ind=0, am=0, be=0, %b[34] movaw,2 area=0, ind=12, am=0, be=0, %b[29] movaw,3 area=0, ind=20, am=0, be=0, %b[74] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END fmul_adds,0,sm %b[42], %b[37], %b[89], %b[75] fmul_adds,1,sm %b[22], %b[85], %b[88], %b[84] staaw,2 %b[69], %aad4[ %aasti8 ] incr,2 %aaincr4 fmuls,3,sm %b[43], %b[35], %b[85] fadds,4,sm %b[80], %b[75], %b[80] staaw,5 %b[70], %aad3[ %aasti7 ] incr,5 %aaincr4 movaw,0 area=0, ind=4, am=1, be=0, %b[37] movaw,1 area=0, ind=8, am=0, be=0, %b[69] movaw,2 area=0, ind=16, am=1, be=0, %b[42] movaw,3 area=0, ind=24, am=0, be=0, %b[70] }
Теоретическая скорость: 4 комплексных числа за 7 тактов (4/7) = 4.57 Байт/такт
Двойная теоретическая скорость: 9.14 Байт/такт
Замеры скорости
2. stage_radix2_2x_etalon_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью опции unroll.
Код на Си
void stage_radix2_2x_etalon_unroll2(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef_a, myComplex *coef_b) { myComplex *x0_in = &data_in[0]; myComplex *y0_in = &data_in[1]; myComplex *x1_in = &data_in[2]; myComplex *y1_in = &data_in[3]; myComplex *c0a_in = &coef_a[0]; myComplex *c1a_in = &coef_a[1]; myComplex *c0b_in = &coef_b[0]; myComplex *c1b_in = &coef_b[data_count/4]; myComplex *out_add0 = &data_out[0*data_count/4]; myComplex *out_add1 = &data_out[1*data_count/4]; myComplex *out_sub0 = &data_out[2*data_count/4]; myComplex *out_sub1 = &data_out[3*data_count/4]; #pragma ivdep #pragma unroll(2) // #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { myComplex x0 = x0_in[4*i]; myComplex y0 = y0_in[4*i]; myComplex c0 = c0a_in[2*i]; myComplex x1 = x1_in[4*i]; myComplex y1 = y1_in[4*i]; myComplex c1 = c1a_in[2*i]; myComplex cy0 = complex_mul(c0, y0); myComplex cy1 = complex_mul(c1, y1); myComplex add0 = complex_add(x0, cy0); myComplex sub0 = complex_sub(x0, cy0); myComplex add1 = complex_add(x1, cy1); myComplex sub1 = complex_sub(x1, cy1); x0 = add0; y0 = add1; c0 = c0b_in[i]; x1 = sub0; y1 = sub1; c1 = c1b_in[i]; cy0 = complex_mul(c0, y0); cy1 = complex_mul(c1, y1); out_add0[i] = complex_add(x0, cy0); out_sub0[i] = complex_sub(x0, cy0); out_add1[i] = complex_add(x1, cy1); out_sub1[i] = complex_sub(x1, cy1); } }
Основной цикл на ассемблере
.L2305: { fapb ct=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=16, d=0, incr=2, ind=2, asz=3, abs=0, disp=16 } { fapb ct=0, dcd=0, fmt=3, mrng=16, d=0, incr=2, ind=2, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=32 } { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=3, ind=3, asz=4, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=3, ind=4, asz=4, abs=16, disp=0 } .L1020: { loop_mode pfmul_rsubs,0,sm %b[71], %b[17], %b[75], %b[1] insfd,1,sm %b[10], %r6, %b[11], %b[87] pfmul_rsubs,2,sm %b[101], %b[47], %b[96], %b[0] insfd,3,sm %b[92], %r6, %b[98], %b[86] insfd,4,sm %b[86], %r6, %b[87], %b[10] pfsubs,5,sm %b[29], %b[40], %b[11] } { loop_mode pfmul_adds,0,sm %b[101], %b[13], %b[80], %b[17] insfd,1,sm %b[63], %r6, %b[38], %b[12] pfmul_rsubs,2,sm %b[87], %b[93], %b[85], %b[47] pfmul_adds,3,sm %b[90], %b[12], %b[97], %b[38] insfd,4,sm %b[76], %r6, %b[52], %b[13] } { loop_mode pfadd_adds,0,sm %b[18], %b[3], %b[49], %b[49] insfd,1,sm %b[82], %r6, %b[95], %b[29] pfadd_rsubs,2,sm %b[18], %b[3], %b[49], %b[40] pfadds,3,sm %b[29], %b[40], %b[52] pshufb,4,sm %b[43], %b[57], %r0, %b[80] pfmuls,5,sm %b[91], %b[93], %b[76] } { loop_mode pfsub_rsubs,0,sm %b[18], %b[3], %b[2], %b[55] insfd,1,sm %b[94], %r6, %b[81], %b[82] pfsub_rsubs,2,sm %b[37], %b[28], %b[19], %b[41] pshufb,3,sm %b[24], %b[25], %r0, %b[85] pshufb,4,sm %b[34], %b[48], %r0, %b[90] pfmuls,5,sm %b[41], %b[15], %b[81] } { loop_mode pfsub_adds,0,sm %b[18], %b[3], %b[2], %b[46] pfmuls,1,sm %b[82], %b[10], %b[92] pfsub_adds,2,sm %b[37], %b[28], %b[19], %b[32] pfadds,3,sm %b[32], %b[46], %b[91] pshufb,4,sm %b[64], %b[42], %r0, %b[93] movad,0 area=2, ind=0, am=0, be=0, %b[19] movad,1 area=2, ind=8, am=1, be=0, %b[18] movad,2 area=2, ind=0, am=0, be=0, %b[3] movad,3 area=2, ind=8, am=1, be=0, %b[2] } { loop_mode pfadd_rsubs,0,sm %b[37], %b[28], %b[84], %b[62] insfd,1,sm %b[89], %r6, %b[100], %b[56] staad,2 %b[80], %aad1[ %aasti5 + _f32s,_lts0 0x8 ] pshufb,3,sm %b[8], %b[9], %r0, %b[89] pshufb,4,sm %b[70], %b[51], %r0, %b[95] pfmuls,5,sm %b[85], %b[11], %b[94] movaw,0 area=1, ind=0, am=0, be=0, %b[63] movaw,1 area=1, ind=4, am=0, be=0, %b[66] movaw,2 area=1, ind=4, am=0, be=0, %b[80] movaw,3 area=1, ind=0, am=0, be=0, %b[59] } { loop_mode pfadd_adds,0,sm %b[37], %b[28], %b[84], %b[68] insfd,1,sm %b[78], %r6, %b[99], %b[28] staad,2 %b[90], %aad2[ %aasti6 + _f32s,_lts0 0x8 ] pfmuls,3,sm %b[85], %b[45], %b[78] insfd,4,sm %b[83], %r6, %b[68], %b[37] pfmuls,5,sm %b[89], %b[52], %b[83] movaw,0 area=0, ind=24, am=0, be=0, %b[96] movaw,1 area=0, ind=28, am=0, be=0, %b[85] movaw,2 area=1, ind=24, am=0, be=0, %b[90] movaw,3 area=1, ind=28, am=0, be=0, %b[84] } { loop_mode insfd,1,sm %b[79], %r6, %g16, %b[88] staad,2 %b[93], %aad3[ %aasti7 + _f32s,_lts0 0x8 ] insfd,4,sm %b[88], %r6, %b[65], %b[65] pfmuls,5,sm %b[39], %b[58], %b[71] movaw,0 area=1, ind=8, am=0, be=0, %g16 movaw,1 area=1, ind=12, am=1, be=0, %b[79] movaw,2 area=1, ind=8, am=0, be=0, %b[72] movaw,3 area=1, ind=20, am=0, be=0, %b[75] } { loop_mode pfmul_adds,0,sm %b[87], %b[54], %b[76], %b[82] insfd,1,sm %b[43], %r6, %b[57], %b[98] staad,2 %b[95], %aad4[ %aasti8 + _f32s,_lts0 0x8 ] pfmuls,3,sm %b[82], %b[86], %b[95] insfd,4,sm %b[34], %r6, %b[48], %b[97] pfsubs,5,sm %b[30], %b[44], %b[43] movaw,0 area=0, ind=4, am=0, be=0, %b[93] movaw,1 area=0, ind=0, am=0, be=0, %b[34] movaw,2 area=1, ind=16, am=0, be=0, %b[76] movaw,3 area=1, ind=12, am=1, be=0, %b[87] } { loop_mode pfmul_rsubs,0,sm %b[88], %b[86], %b[92], %b[42] insfd,1,sm %b[70], %r6, %b[51], %b[97] staad,2 %b[98], %aad1[ %aasti5 ] incr,2 %aaincr4 insfd,4,sm %b[64], %r6, %b[42], %b[98] staad,5 %b[97], %aad2[ %aasti6 ] incr,5 %aaincr4 movaw,0 area=0, ind=8, am=0, be=0, %b[48] movaw,1 area=0, ind=20, am=0, be=0, %b[51] movaw,2 area=0, ind=0, am=0, be=0, %b[86] movaw,3 area=0, ind=12, am=0, be=0, %b[92] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmul_adds,0,sm %b[69], %b[60], %b[81], %b[24] insfd,1,sm %b[24], %r6, %b[25], %b[99] staad,2 %b[97], %aad4[ %aasti8 ] incr,2 %aaincr4 insfd,4,sm %b[77], %r6, %b[53], %b[25] staad,5 %b[98], %aad3[ %aasti7 ] incr,5 %aaincr4 movaw,0 area=0, ind=16, am=0, be=0, %b[97] movaw,1 area=0, ind=12, am=1, be=0, %b[98] movaw,2 area=0, ind=4, am=1, be=0, %b[81] movaw,3 area=0, ind=8, am=0, be=0, %b[77] }
Так же, как это было в stage_radix2_etalon_unroll2, можно видеть, что компилятор вставил векторные инструкции.
Теоретическая скорость: 8 комплексных чисел за 11 тактов (8/11) = 5.82 Байт/такт
Двойная теоретическая скорость: 11.64 Байт/такт
Замеры скорости
3. stage_radix2_2x_simd64
Здесь происходит ручная раскрутка алгоритма stage_radix2_simd64 в 2 раза.
Код на Си
void stage_radix2_2x_simd64(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef_a, myComplex *coef_b) { uint64_t *x0_in = (uint64_t*)&data_in[0]; uint64_t *y0_in = (uint64_t*)&data_in[1]; uint64_t *x1_in = (uint64_t*)&data_in[2]; uint64_t *y1_in = (uint64_t*)&data_in[3]; uint64_t *c0a_in = (uint64_t*)&coef_a[0]; uint64_t *c1a_in = (uint64_t*)&coef_a[1]; uint64_t *c0b_in = (uint64_t*)&coef_b[0]; uint64_t *c1b_in = (uint64_t*)&coef_b[data_count/4]; uint64_t *out_add0 = (uint64_t*)&data_out[0*data_count/4]; uint64_t *out_add1 = (uint64_t*)&data_out[1*data_count/4]; uint64_t *out_sub0 = (uint64_t*)&data_out[2*data_count/4]; uint64_t *out_sub1 = (uint64_t*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { uint64_t x0 = x0_in[4*i]; uint64_t y0 = y0_in[4*i]; uint64_t c0 = c0a_in[2*i]; uint64_t x1 = x1_in[4*i]; uint64_t y1 = y1_in[4*i]; uint64_t c1 = c1a_in[2*i]; uint64_t conj_c0 = __builtin_e2k_pxord(c0, 1LL<<63); uint64_t conj_c1 = __builtin_e2k_pxord(c1, 1LL<<63); uint64_t swap_c0 = __builtin_e2k_pshufb(0, c0, 0x0302010007060504); uint64_t swap_c1 = __builtin_e2k_pshufb(0, c1, 0x0302010007060504); uint64_t cy0_real = __builtin_e2k_pfmuls(conj_c0, y0); uint64_t cy1_real = __builtin_e2k_pfmuls(conj_c1, y1); uint64_t cy0_imag = __builtin_e2k_pfmuls(swap_c0, y0); uint64_t cy1_imag = __builtin_e2k_pfmuls(swap_c1, y1); uint64_t cy0 = __builtin_e2k_pfhadds(cy0_real, cy0_imag); uint64_t cy1 = __builtin_e2k_pfhadds(cy1_real, cy1_imag); uint64_t add0 = __builtin_e2k_pfadds(x0, cy0); uint64_t sub0 = __builtin_e2k_pfsubs(x0, cy0); uint64_t add1 = __builtin_e2k_pfadds(x1, cy1); uint64_t sub1 = __builtin_e2k_pfsubs(x1, cy1); x0 = add0; y0 = add1; c0 = c0b_in[i]; x1 = sub0; y1 = sub1; c1 = c1b_in[i]; conj_c0 = __builtin_e2k_pxord(c0, 1LL<<63); conj_c1 = __builtin_e2k_pxord(c1, 1LL<<63); swap_c0 = __builtin_e2k_pshufb(0, c0, 0x0302010007060504); swap_c1 = __builtin_e2k_pshufb(0, c1, 0x0302010007060504); cy0_real = __builtin_e2k_pfmuls(conj_c0, y0); cy1_real = __builtin_e2k_pfmuls(conj_c1, y1); cy0_imag = __builtin_e2k_pfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_pfmuls(swap_c1, y1); cy0 = __builtin_e2k_pfhadds(cy0_real, cy0_imag); cy1 = __builtin_e2k_pfhadds(cy1_real, cy1_imag); out_add0[i] = __builtin_e2k_pfadds(x0, cy0); out_sub0[i] = __builtin_e2k_pfsubs(x0, cy0); out_add1[i] = __builtin_e2k_pfadds(x1, cy1); out_sub1[i] = __builtin_e2k_pfsubs(x1, cy1); } }
Основной цикл на ассемблере
.L2998: { fapb ct=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=3, abs=8, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=4, abs=16, disp=0 } .L2607: { loop_mode pfmul_hadds,0,sm %b[43], %b[20], %b[37], %b[24] pfmul_hadds,1,sm %b[41], %b[5], %b[39], %b[28] pfmuls,2,sm %b[75], %b[18], %b[35] xord,3,sm %b[44], %r0, %b[84] xord,4,sm %b[2], %r0, %b[81] pfsubs,5,sm %b[78], %b[49], %b[1] movad,1 area=0, ind=8, am=0, be=0, %b[0] } { loop_mode pfmul_hadds,1,sm %b[62], %b[9], %b[60], %b[20] pfmuls,2,sm %b[83], %b[3], %b[37] pshufb,3,sm 0x0, %b[71], %r6, %b[41] pshufb,4,sm 0x0, %b[58], %r6, %b[39] pfadds,5,sm %b[78], %b[49], %b[5] } { loop_mode pfmul_hadds,1,sm %b[73], %b[15], %b[53], %b[43] pfmuls,2,sm %b[84], %b[7], %b[58] pshufb,4,sm 0x0, %b[44], %r6, %b[60] pfmuls,5,sm %b[81], %b[11], %b[49] movad,3 area=0, ind=24, am=0, be=0, %b[9] } { loop_mode pfsub_adds,0,sm %b[33], %b[26], %b[30], %b[62] pfsub_rsubs,1,sm %b[33], %b[26], %b[30], %b[53] staad,2 %b[66], %aad2[ %aasti6 ] incr,2 %aaincr0 xord,3,sm %b[69], %r0, %b[73] pshufb,4,sm 0x0, %b[4], %r6, %b[71] staad,5 %b[57], %aad1[ %aasti5 ] incr,5 %aaincr0 movad,1 area=2, ind=0, am=1, be=0, %b[44] movad,3 area=0, ind=0, am=0, be=0, %b[15] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfadd_adds,0,sm %b[33], %b[26], %b[22], %b[78] pfadd_rsubs,1,sm %b[33], %b[26], %b[22], %b[75] staad,2 %b[82], %aad4[ %aasti8 ] incr,2 %aaincr0 xord,3,sm %b[56], %r0, %b[81] staad,5 %b[79], %aad3[ %aasti7 ] incr,5 %aaincr0 movad,0 area=1, ind=0, am=1, be=0, %b[30] movad,1 area=0, ind=0, am=1, be=0, %b[57] movad,2 area=0, ind=8, am=1, be=0, %b[4] movad,3 area=0, ind=16, am=0, be=0, %b[66] }
Теоретическая скорость: 4 комплексных числа за 5 тактов (4/5) = 6.4 Байт/такт
Двойная теоретическая скорость: 12.8 Байт/такт
Замеры скорости
4. stage_radix2_2x_simd128
Здесь происходит ручная раскрутка алгоритма stage_radix2_simd128 в 2 раза.
Код на Си
void stage_radix2_2x_simd128(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef_a, myComplex *coef_b) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *xy1_in = (__v2di*)&data_in[2]; __v2di *xy2_in = (__v2di*)&data_in[4]; __v2di *xy3_in = (__v2di*)&data_in[6]; __v2di *c0a_in = (__v2di*)&coef_a[0]; __v2di *c1a_in = (__v2di*)&coef_a[2]; __v2di *c0b_in = (__v2di*)&coef_b[0]; __v2di *c1b_in = (__v2di*)&coef_b[data_count/4]; __v2di *out_add0 = (__v2di*)&data_out[0*data_count/4]; __v2di *out_add1 = (__v2di*)&data_out[1*data_count/4]; __v2di *out_sub0 = (__v2di*)&data_out[2*data_count/4]; __v2di *out_sub1 = (__v2di*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/8; ++i) { __v2di xy0 = xy0_in[4*i]; __v2di xy1 = xy1_in[4*i]; __v2di c0 = c0a_in[2*i]; __v2di xy2 = xy2_in[4*i]; __v2di xy3 = xy3_in[4*i]; __v2di c1 = c1a_in[2*i]; __v2di x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di conj_c0 = __builtin_e2k_qpxor(c0, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_c1 = __builtin_e2k_qpxor(c1, (__v2di){1LL<<63, 1LL<<63}); __v2di swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy0_real = __builtin_e2k_qpfmuls(conj_c0, y0); __v2di cy1_real = __builtin_e2k_qpfmuls(conj_c1, y1); __v2di cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); __v2di cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); __v2di cy0_rrii = __builtin_e2k_qpfhadds(cy0_real, cy0_imag); __v2di cy1_rrii = __builtin_e2k_qpfhadds(cy1_real, cy1_imag); __v2di cy0 = __builtin_e2k_qpshufb(cy0_rrii, cy0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di cy1 = __builtin_e2k_qpshufb(cy1_rrii, cy1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di add0 = __builtin_e2k_qpfadds(x0, cy0); __v2di sub0 = __builtin_e2k_qpfsubs(x0, cy0); __v2di add1 = __builtin_e2k_qpfadds(x1, cy1); __v2di sub1 = __builtin_e2k_qpfsubs(x1, cy1); xy0 = add0; xy1 = add1; c0 = c0b_in[i]; xy2 = sub0; xy3 = sub1; c1 = c1b_in[i]; x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); conj_c0 = __builtin_e2k_qpxor(c0, (__v2di){1LL<<63, 1LL<<63}); conj_c1 = __builtin_e2k_qpxor(c1, (__v2di){1LL<<63, 1LL<<63}); swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); cy0_real = __builtin_e2k_qpfmuls(conj_c0, y0); cy1_real = __builtin_e2k_qpfmuls(conj_c1, y1); cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); cy0_rrii = __builtin_e2k_qpfhadds(cy0_real, cy0_imag); cy1_rrii = __builtin_e2k_qpfhadds(cy1_real, cy1_imag); cy0 = __builtin_e2k_qpshufb(cy0_rrii, cy0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); cy1 = __builtin_e2k_qpshufb(cy1_rrii, cy1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); out_add0[i] = __builtin_e2k_qpfadds(x0, cy0); out_sub0[i] = __builtin_e2k_qpfsubs(x0, cy0); out_add1[i] = __builtin_e2k_qpfadds(x1, cy1); out_sub1[i] = __builtin_e2k_qpfsubs(x1, cy1); } }
Основной цикл на ассемблере
.L3790: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=32 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=4, abs=16, disp=0 } .L3059: { loop_mode qpfmul_hadds,0,sm %b[56], %b[54], %b[70], %b[1] qpshufb,1,sm %b[9], %b[32], %r15, %b[52] qpfadds,2,sm %b[88], %b[87], %b[0] qpshufb,3,sm %b[33], %b[33], %r0, %b[94] qpshufb,4,sm %b[6], %b[30], %r16, %b[95] qpfadds,5,sm %b[61], %b[71], %b[92] } { loop_mode qpfmul_hadds,0,sm %b[74], %b[69], %b[7], %b[33] qpshufb,1,sm %b[29], %b[29], %r14, %b[56] qpfsubs,2,sm %b[55], %b[90], %b[30] qpxor,3,sm %b[60], %r13, %b[54] qpxor,4,sm %b[27], %r13, %b[57] qpfsubs,5,sm %b[95], %b[94], %b[96] movaqp,3 area=1, ind=0, am=0, be=0, %b[6] } { loop_mode qpfmul_hadds,0,sm %b[56], %b[73], %b[93], %b[29] qpshufb,1,sm %b[13], %b[36], %r16, %b[59] qpfsubs,2,sm %b[86], %b[85], %b[7] qpshufb,4,sm %b[62], %b[62], %r14, %b[61] qpfadds,5,sm %b[95], %b[94], %b[97] } { loop_mode qpshufb,1,sm %b[3], %b[3], %r0, %b[69] qpfmuls,2,sm %b[53], %b[52], %b[68] qpshufb,3,sm %b[14], %b[43], %r15, %b[62] qpshufb,4,sm %b[75], %b[76], %r15, %b[63] staaqp,5 %b[72], %aad1[ %aasti5 ] incr,5 %aaincr0 movaqp,0 area=2, ind=0, am=1, be=0, %b[36] movaqp,1 area=1, ind=0, am=1, be=0, %b[13] movaqp,3 area=1, ind=16, am=1, be=0, %b[56] } { loop_mode qpfmuls,0,sm %b[41], %b[65], %b[3] qpshufb,1,sm %b[0], %b[24], %r15, %b[71] qpfsubs,2,sm %b[59], %b[69], %b[70] qpshufb,3,sm %b[12], %b[12], %r14, %b[72] qpshufb,4,sm %b[83], %b[84], %r16, %b[53] staaqp,5 %b[92], %aad2[ %aasti6 ] incr,5 %aaincr0 } { loop_mode qpfmuls,0,sm %b[54], %b[64], %b[87] qpshufb,1,sm %b[35], %b[35], %r0, %b[88] qpfmuls,2,sm %b[57], %b[71], %b[91] qpshufb,3,sm %b[22], %b[51], %r16, %b[84] qpshufb,4,sm %b[39], %b[39], %r0, %b[83] staaqp,5 %b[96], %aad3[ %aasti7 ] incr,5 %aaincr0 movaqp,0 area=0, ind=0, am=0, be=0, %b[41] movaqp,1 area=0, ind=16, am=1, be=0, %b[12] movaqp,2 area=0, ind=0, am=0, be=0, %b[74] movaqp,3 area=0, ind=16, am=1, be=0, %b[73] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfmul_hadds,0,sm %b[61], %b[66], %b[89], %b[35] qpshufb,1,sm %b[50], %b[50], %r14, %b[54] qpfadds,2,sm %b[55], %b[90], %b[22] qpxor,3,sm %b[8], %r13, %b[39] qpxor,4,sm %b[48], %r13, %b[51] staaqp,5 %b[97], %aad4[ %aasti8 ] incr,5 %aaincr0 }
Теоретическая скорость: 8 комплексных чисел за 7 тактов (8/7) = 9.14 Байт/такт
Двойная теоретическая скорость: 18.29 Байт/такт
Замеры скорости
5. stage_radix2_2x_simd128_noConj
Здесь происходит ручная раскрутка алгоритма stage_radix2_simd128_noConj в 2 раза.
Код на Си
void stage_radix2_2x_simd128_noConj(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef_a, myComplex *coef_b) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *xy1_in = (__v2di*)&data_in[2]; __v2di *xy2_in = (__v2di*)&data_in[4]; __v2di *xy3_in = (__v2di*)&data_in[6]; __v2di *c0a_in = (__v2di*)&coef_a[0]; __v2di *c1a_in = (__v2di*)&coef_a[2]; __v2di *c0b_in = (__v2di*)&coef_b[0]; __v2di *c1b_in = (__v2di*)&coef_b[data_count/4]; __v2di *out_add0 = (__v2di*)&data_out[0*data_count/4]; __v2di *out_add1 = (__v2di*)&data_out[1*data_count/4]; __v2di *out_sub0 = (__v2di*)&data_out[2*data_count/4]; __v2di *out_sub1 = (__v2di*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/8; ++i) { __v2di xy0 = xy0_in[4*i]; __v2di xy1 = xy1_in[4*i]; __v2di c0 = c0a_in[2*i]; __v2di xy2 = xy2_in[4*i]; __v2di xy3 = xy3_in[4*i]; __v2di c1 = c1a_in[2*i]; __v2di x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy0_real = __builtin_e2k_qpfmuls( c0, y0); __v2di cy1_real = __builtin_e2k_qpfmuls( c1, y1); __v2di cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); __v2di cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); __v2di cy0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy0_real); __v2di cy1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy1_real); __v2di cy0_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy0_imag); __v2di cy1_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy1_imag); __v2di cy0 = __builtin_e2k_qppermb(cy0_ii, cy0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di cy1 = __builtin_e2k_qppermb(cy1_ii, cy1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di add0 = __builtin_e2k_qpfadds(x0, cy0); __v2di sub0 = __builtin_e2k_qpfsubs(x0, cy0); __v2di add1 = __builtin_e2k_qpfadds(x1, cy1); __v2di sub1 = __builtin_e2k_qpfsubs(x1, cy1); xy0 = add0; xy1 = add1; c0 = c0b_in[i]; xy2 = sub0; xy3 = sub1; c1 = c1b_in[i]; x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); cy0_real = __builtin_e2k_qpfmuls( c0, y0); cy1_real = __builtin_e2k_qpfmuls( c1, y1); cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); cy0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy0_real); cy1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy1_real); cy0_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy0_imag); cy1_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy1_imag); cy0 = __builtin_e2k_qppermb(cy0_ii, cy0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); cy1 = __builtin_e2k_qppermb(cy1_ii, cy1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); out_add0[i] = __builtin_e2k_qpfadds(x0, cy0); out_sub0[i] = __builtin_e2k_qpfsubs(x0, cy0); out_add1[i] = __builtin_e2k_qpfadds(x1, cy1); out_sub1[i] = __builtin_e2k_qpfsubs(x1, cy1); } }
Основной цикл на ассемблере
.L4577: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=32 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=4, abs=16, disp=0 } .L3851: { loop_mode qpfmul_hsubs,0,sm %b[53], %b[54], %r16, %b[5] qpshufb,1,sm %b[39], %b[39], %r15, %b[25] qpfmul_hadds,2,sm %b[92], %b[54], %r16, %b[1] qpshufb,3,sm %b[16], %b[8], %r14, %b[24] qpfsubs,4,sm %b[66], %b[22], %b[9] qpfsubs,5,sm %b[89], %b[69], %b[0] } { loop_mode qpfmul_hsubs,0,sm %b[39], %b[71], %r16, %b[58] qpshufb,1,sm %b[72], %b[75], %r0, %b[64] qpfmul_hadds,2,sm %b[25], %b[71], %r16, %b[54] qpshufb,3,sm %b[20], %b[20], %r15, %b[62] qpfsubs,4,sm %b[86], %b[59], %b[8] qpfadds,5,sm %b[24], %b[48], %b[61] movaqp,2 area=1, ind=0, am=0, be=0, %b[53] movaqp,3 area=1, ind=16, am=1, be=0, %b[16] } { loop_mode qpfmul_hsubs,0,sm %b[57], %b[64], %r16, %b[78] qpshufb,1,sm %b[57], %b[57], %r15, %b[88] staaqp,2 %b[87], %aad1[ %aasti5 ] incr,2 %aaincr0 qpshufb,3,sm %b[26], %b[34], %r0, %b[81] qpshufb,4,sm %b[32], %b[40], %r14, %b[84] qpfsubs,5,sm %b[24], %b[48], %b[85] movaqp,0 area=2, ind=0, am=1, be=0, %b[39] movaqp,1 area=1, ind=0, am=1, be=0, %b[25] movaqp,2 area=0, ind=0, am=0, be=0, %b[71] movaqp,3 area=0, ind=16, am=1, be=0, %b[68] } { loop_mode qpfmul_hadds,0,sm %b[88], %b[64], %r16, %b[48] qpshufb,1,sm %b[51], %b[51], %r15, %b[90] staaqp,2 %b[63], %aad2[ %aasti6 ] incr,2 %aaincr0 qpshufb,3,sm %b[76], %b[79], %r14, %b[87] qppermb,4,sm %b[15], %b[67], %r13, %b[57] qpfadds,5,sm %b[89], %b[69], %b[40] movaqp,0 area=0, ind=0, am=0, be=0, %b[32] movaqp,1 area=0, ind=16, am=1, be=0, %b[24] } { loop_mode qpfmul_hsubs,0,sm %b[20], %b[83], %r16, %b[63] qpshufb,1,sm %b[17], %b[42], %r0, %b[69] staaqp,2 %b[11], %aad3[ %aasti7 ] incr,2 %aaincr0 qppermb,3,sm %b[52], %b[82], %r13, %b[67] qpshufb,4,sm %b[21], %b[46], %r14, %b[64] qpfadds,5,sm %b[86], %b[59], %b[15] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfmul_hadds,0,sm %b[62], %b[83], %r16, %b[11] qpshufb,1,sm %b[10], %b[2], %r0, %b[52] staaqp,2 %b[23], %aad4[ %aasti8 ] incr,2 %aaincr0 qppermb,3,sm %b[3], %b[7], %r13, %b[46] qppermb,4,sm %b[56], %b[60], %r13, %b[20] qpfadds,5,sm %b[66], %b[22], %b[21] }
Теоретическая скорость: 8 комплексных чисел за 6 тактов (8/6) = 10.67 Байт/такт
Двойная теоретическая скорость: 21.33 Байт/такт
Замеры скорости
Итоги по stage_radix2_2x
Скорости выросли по сравнению с исходными версиями stage_radix2.
График FFT находится здесь.
stage_radix2_readConjSwap
Вернёмся к алгоритмам stage_radix2. Обратим внимание, что conj_c и swap_c получаются напрямую из c, который читается из памяти и больше нигде не используется.
Оптимизация: вместо вычисления conj_c и swap_c сразу читать их из памяти, чтение c больше не нужно. В результате уйдут две инструкции: xor и shuf.
Смотрим, что получится.
1. stage_radix2_readConjSwap_simd64
Развитие stage_radix2_simd64: замена вычисления conj и swap на чтение из памяти.
Код на Си
void stage_radix2_readConjSwap_simd64(int data_count, myComplex *data_in, myComplex *data_out, myComplex *conj_coef, myComplex *swap_coef) { uint64_t *x_in = (uint64_t*)&data_in[0]; uint64_t *y_in = (uint64_t*)&data_in[1]; uint64_t *conj_c_in = (uint64_t*)conj_coef; uint64_t *swap_c_in = (uint64_t*)swap_coef; uint64_t *out_add = (uint64_t*)&data_out[0]; uint64_t *out_sub = (uint64_t*)&data_out[data_count/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { uint64_t x = x_in[2*i]; uint64_t y = y_in[2*i]; uint64_t conj_c = conj_c_in[i]; uint64_t swap_c = swap_c_in[i]; uint64_t cy_real = __builtin_e2k_pfmuls(conj_c, y); uint64_t cy_imag = __builtin_e2k_pfmuls(swap_c, y); uint64_t cy = __builtin_e2k_pfhadds(cy_real, cy_imag); out_add[i] = __builtin_e2k_pfadds(x, cy); out_sub[i] = __builtin_e2k_pfsubs(x, cy); } }
Основной цикл на ассемблере
.L326: { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=4, abs=16, disp=0 } .L125: { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmuls,0,sm %b[67], %b[33], %b[45] pfsubs,1,sm %b[42], %b[62], %b[62] staad,2 %b[70], %aad1[ %aasti4 ] incr,2 %aaincr0 pfmul_hadds,3,sm %b[23], %b[45], %b[57], %b[42] pfadds,4,sm %b[42], %b[62], %b[67] staad,5 %b[75], %aad2[ %aasti5 ] incr,5 %aaincr0 movad,0 area=0, ind=0, am=1, be=0, %b[57] movad,1 area=1, ind=0, am=1, be=0, %b[1] movad,2 area=0, ind=8, am=1, be=0, %b[23] movad,3 area=0, ind=0, am=0, be=0, %b[0] }
Раньше было 8 инструкций в цикле, теперь стало 6.
6 инструкций идеально помещаются в 1 такт.
Теоретическая скорость: 2 комплексных числа за 1 такт (2/1) = 16 Байт/такт
Замеры скорости
2. stage_radix2_readConjSwap_simd128
Развитие stage_radix2_simd128: замена вычисления conj и swap на чтение из памяти.
Развитие stage_radix2_simd128_noConj приходит сюда же.
Код на Си
void stage_radix2_readConjSwap_simd128(int data_count, myComplex *data_in, myComplex *data_out, myComplex *conj_coef, myComplex *swap_coef) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *xy1_in = (__v2di*)&data_in[2]; __v2di *conj_c_in = (__v2di*)conj_coef; __v2di *swap_c_in = (__v2di*)swap_coef; __v2di *out_add = (__v2di*)&data_out[0]; __v2di *out_sub = (__v2di*)&data_out[data_count/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { __v2di xy0 = xy0_in[2*i]; __v2di xy1 = xy1_in[2*i]; __v2di conj_c = conj_c_in[i]; __v2di swap_c = swap_c_in[i]; __v2di x = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di cy_real = __builtin_e2k_qpfmuls(conj_c, y); __v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y); __v2di cy_rrii = __builtin_e2k_qpfhadds(cy_real, cy_imag); __v2di cy = __builtin_e2k_qpshufb(cy_rrii, cy_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); out_add[i] = __builtin_e2k_qpfadds(x, cy); out_sub[i] = __builtin_e2k_qpfsubs(x, cy); } }
Основной цикл на ассемблере
.L599: { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=4, abs=16, disp=0 } .L353: { loop_mode qpshufb,1,sm %b[31], %b[40], %r0, %b[16] qpshufb,3,sm %b[33], %b[42], %r7, %b[0] qpfsubs,4,sm %b[14], %b[44], %b[21] staaqp,5 %b[25], %aad1[ %aasti4 ] incr,5 %aaincr0 movaqp,0 area=0, ind=0, am=1, be=0, %b[13] movaqp,1 area=1, ind=0, am=1, be=0, %b[1] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfmuls,0,sm %b[19], %b[16], %b[33] qpfmul_hadds,2,sm %b[11], %b[20], %b[37], %b[22] qpshufb,3,sm %b[32], %b[32], %r6, %b[42] qpfadds,4,sm %b[14], %b[44], %b[39] staaqp,5 %b[43], %aad2[ %aasti5 ] incr,5 %aaincr0 movaqp,2 area=0, ind=0, am=0, be=0, %b[34] movaqp,3 area=0, ind=16, am=1, be=0, %b[25] }
Раньше было 11 инструкций в цикле, теперь стало 9.
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
Сейчас занято 9/6 такта. Раскрутка в 2 раза даст 2 * 9/6 = 3 такта.
3. stage_radix2_readConjSwap_simd128_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью опции unroll.
Код на Си
void stage_radix2_readConjSwap_simd128_unroll2(int data_count, myComplex *data_in, myComplex *data_out, myComplex *conj_coef, myComplex *swap_coef) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *xy1_in = (__v2di*)&data_in[2]; __v2di *conj_c_in = (__v2di*)conj_coef; __v2di *swap_c_in = (__v2di*)swap_coef; __v2di *out_add = (__v2di*)&data_out[0]; __v2di *out_sub = (__v2di*)&data_out[data_count/2]; #pragma ivdep #pragma unroll(2) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { __v2di xy0 = xy0_in[2*i]; __v2di xy1 = xy1_in[2*i]; __v2di conj_c = conj_c_in[i]; __v2di swap_c = swap_c_in[i]; __v2di x = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di cy_real = __builtin_e2k_qpfmuls(conj_c, y); __v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y); __v2di cy_rrii = __builtin_e2k_qpfhadds(cy_real, cy_imag); __v2di cy = __builtin_e2k_qpshufb(cy_rrii, cy_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); out_add[i] = __builtin_e2k_qpfadds(x, cy); out_sub[i] = __builtin_e2k_qpfsubs(x, cy); } }
Основной цикл на ассемблере
.L992: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=4, abs=0, disp=32 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=4, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=4, abs=16, disp=0 } .L626: { loop_mode qpfmul_hadds,0,sm %b[27], %b[14], %b[92], %b[1] qpfmuls,1,sm %b[81], %b[4], %b[9] qpfsubs,2,sm %b[89], %b[88], %b[34] qpshufb,3,sm %b[52], %b[51], %r0, %b[0] qpshufb,4,sm %b[20], %b[33], %r0, %b[8] qpfadds,5,sm %b[89], %b[88], %b[13] } { loop_mode qpfmuls,0,sm %b[73], %b[10], %b[88] qpfsubs,1,sm %b[84], %b[85], %b[89] staaqp,2 %b[36], %aad1[ %aasti4 ] qpshufb,3,sm %b[66], %b[65], %r13, %b[80] qpshufb,4,sm %b[74], %b[74], %r14, %b[81] staaqp,5 %b[15], %aad2[ %aasti5 ] movaqp,0 area=0, ind=0, am=0, be=0, %b[27] movaqp,1 area=0, ind=16, am=1, be=0, %b[14] movaqp,2 area=0, ind=0, am=0, be=0, %b[47] movaqp,3 area=0, ind=16, am=1, be=0, %b[48] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfmul_hadds,0,sm %b[46], %b[6], %b[11], %b[66] qpfadds,1,sm %b[82], %b[83], %b[74] staaqp,2 %b[91], %aad1[ %aasti4 + _f32s,_lts0 0x10 ] incr,2 %aaincr3 qpshufb,3,sm %b[32], %b[45], %r13, %b[85] qpshufb,4,sm %b[7], %b[7], %r14, %b[84] staaqp,5 %b[78], %aad2[ %aasti5 + _f32s,_lts0 0x10 ] incr,5 %aaincr3 movaqp,0 area=1, ind=0, am=0, be=0, %b[65] movaqp,1 area=1, ind=16, am=1, be=0, %b[73] movaqp,2 area=1, ind=0, am=0, be=0, %b[15] movaqp,3 area=1, ind=16, am=1, be=0, %b[36] }
Теоретическая скорость: 8 комплексных чисел за 3 такта (8/3) = 21.33 Байт/такт
Замеры скорости
Итоги по stage_radix2_readConjSwap
График FFT находится здесь.
stage_radix2_readConjSwap_2x
Один проход по stage_radix2_readConjSwap_2x совершает ту же работу, что 2 прохода по stage_radix2. Поэтому скорость stage_radix2_readConjSwap_2x будем умножать на 2 для удобства сравнения с stage_radix2 (этот факт подписан на оси графика и в выводе консоли).
1. stage_radix2_readConjSwap_2x_simd64
Здесь происходит ручная раскрутка алгоритма stage_radix2_readConjSwap_simd64 в 2 раза.
Код на Си
void stage_radix2_readConjSwap_2x_simd64(int data_count, myComplex *data_in, myComplex *data_out, myComplex *conj_coef_a, myComplex *conj_coef_b, myComplex *swap_coef_a, myComplex *swap_coef_b) { uint64_t *x0_in = (uint64_t*)&data_in[0]; uint64_t *y0_in = (uint64_t*)&data_in[1]; uint64_t *x1_in = (uint64_t*)&data_in[2]; uint64_t *y1_in = (uint64_t*)&data_in[3]; uint64_t *conj_c0a_in = (uint64_t*)&conj_coef_a[0]; uint64_t *conj_c1a_in = (uint64_t*)&conj_coef_a[1]; uint64_t *conj_c0b_in = (uint64_t*)&conj_coef_b[0]; uint64_t *conj_c1b_in = (uint64_t*)&conj_coef_b[data_count/4]; uint64_t *swap_c0a_in = (uint64_t*)&swap_coef_a[0]; uint64_t *swap_c1a_in = (uint64_t*)&swap_coef_a[1]; uint64_t *swap_c0b_in = (uint64_t*)&swap_coef_b[0]; uint64_t *swap_c1b_in = (uint64_t*)&swap_coef_b[data_count/4]; uint64_t *out_add0 = (uint64_t*)&data_out[0*data_count/4]; uint64_t *out_add1 = (uint64_t*)&data_out[1*data_count/4]; uint64_t *out_sub0 = (uint64_t*)&data_out[2*data_count/4]; uint64_t *out_sub1 = (uint64_t*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { uint64_t x0 = x0_in[4*i]; uint64_t y0 = y0_in[4*i]; uint64_t conj_c0 = conj_c0a_in[2*i]; uint64_t swap_c0 = swap_c0a_in[2*i]; uint64_t x1 = x1_in[4*i]; uint64_t y1 = y1_in[4*i]; uint64_t conj_c1 = conj_c1a_in[2*i]; uint64_t swap_c1 = swap_c1a_in[2*i]; uint64_t cy0_real = __builtin_e2k_pfmuls(conj_c0, y0); uint64_t cy1_real = __builtin_e2k_pfmuls(conj_c1, y1); uint64_t cy0_imag = __builtin_e2k_pfmuls(swap_c0, y0); uint64_t cy1_imag = __builtin_e2k_pfmuls(swap_c1, y1); uint64_t cy0 = __builtin_e2k_pfhadds(cy0_real, cy0_imag); uint64_t cy1 = __builtin_e2k_pfhadds(cy1_real, cy1_imag); uint64_t add0 = __builtin_e2k_pfadds(x0, cy0); uint64_t sub0 = __builtin_e2k_pfsubs(x0, cy0); uint64_t add1 = __builtin_e2k_pfadds(x1, cy1); uint64_t sub1 = __builtin_e2k_pfsubs(x1, cy1); x0 = add0; y0 = add1; conj_c0 = conj_c0b_in[i]; swap_c0 = swap_c0b_in[i]; x1 = sub0; y1 = sub1; conj_c1 = conj_c1b_in[i]; swap_c1 = swap_c1b_in[i]; cy0_real = __builtin_e2k_pfmuls(conj_c0, y0); cy1_real = __builtin_e2k_pfmuls(conj_c1, y1); cy0_imag = __builtin_e2k_pfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_pfmuls(swap_c1, y1); cy0 = __builtin_e2k_pfhadds(cy0_real, cy0_imag); cy1 = __builtin_e2k_pfhadds(cy1_real, cy1_imag); out_add0[i] = __builtin_e2k_pfadds(x0, cy0); out_sub0[i] = __builtin_e2k_pfsubs(x0, cy0); out_add1[i] = __builtin_e2k_pfadds(x1, cy1); out_sub1[i] = __builtin_e2k_pfsubs(x1, cy1); } }
Основной цикл на ассемблере
.L723: { fapb ct=0, dcd=0, fmt=4, mrng=16, d=0, incr=1, ind=3, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=1, asz=3, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=16, d=0, incr=1, ind=2, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=6, asz=3, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=7, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=5, asz=3, abs=24, disp=0 } .L322: { loop_mode pfmuls,1,sm %b[60], %b[45], %b[34] pfmul_hadds,2,sm %b[108], %b[47], %b[36], %b[0] pfmul_hadds,3,sm %b[41], %b[97], %b[111], %b[21] pfsub_adds,4,sm %b[77], %b[103], %b[25], %b[6] pfsub_rsubs,5,sm %b[77], %b[103], %b[25], %b[1] } { loop_mode pfmul_hadds,3,sm %b[76], %b[102], %b[107], %b[53] pfadd_adds,4,sm %b[77], %b[103], %b[57], %b[50] pfadd_rsubs,5,sm %b[77], %b[103], %b[57], %b[47] movad,0 area=0, ind=8, am=0, be=0, %b[56] movad,1 area=3, ind=0, am=1, be=0, %b[36] movad,2 area=0, ind=24, am=0, be=0, %b[41] movad,3 area=2, ind=0, am=1, be=0, %b[25] } { loop_mode staad,2 %b[10], %aad2[ %aasti9 ] incr,2 %aaincr0 pfsubs,3,sm %b[32], %b[4], %b[91] pfmuls,4,sm %b[22], %b[17], %b[92] staad,5 %b[5], %aad1[ %aasti8 ] incr,5 %aaincr0 movad,0 area=2, ind=0, am=1, be=0, %b[77] movad,1 area=1, ind=0, am=0, be=0, %b[76] movad,2 area=1, ind=0, am=1, be=0, %b[60] movad,3 area=0, ind=0, am=0, be=0, %b[57] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfadds,0,sm %b[32], %b[4], %b[96] pfmuls,1,sm %b[89], %b[98], %b[103] staad,2 %b[54], %aad4[ %aasti11 ] incr,2 %aaincr0 pfmuls,3,sm %b[48], %b[93], %b[107] pfmul_hadds,4,sm %b[90], %b[19], %b[94], %b[97] staad,5 %b[51], %aad3[ %aasti10 ] incr,5 %aaincr0 movad,0 area=1, ind=8, am=1, be=0, %b[102] movad,1 area=0, ind=0, am=1, be=0, %b[10] movad,2 area=0, ind=8, am=1, be=0, %b[5] movad,3 area=0, ind=16, am=0, be=0, %b[22] }
Теоретическая скорость: 4 комплексных числа за 4 такта (4/4) = 8 Байт/такт
Двойная теоретическая скорость: 16 Байт/такт
Замеры скорости
2. stage_radix2_readConjSwap_2x_simd64_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью опции unroll.
Код на Си
void stage_radix2_readConjSwap_2x_simd64_unroll2(int data_count, myComplex *data_in, myComplex *data_out, myComplex *conj_coef_a, myComplex *conj_coef_b, myComplex *swap_coef_a, myComplex *swap_coef_b) { uint64_t *x0_in = (uint64_t*)&data_in[0]; uint64_t *y0_in = (uint64_t*)&data_in[1]; uint64_t *x1_in = (uint64_t*)&data_in[2]; uint64_t *y1_in = (uint64_t*)&data_in[3]; uint64_t *conj_c0a_in = (uint64_t*)&conj_coef_a[0]; uint64_t *conj_c1a_in = (uint64_t*)&conj_coef_a[1]; uint64_t *conj_c0b_in = (uint64_t*)&conj_coef_b[0]; uint64_t *conj_c1b_in = (uint64_t*)&conj_coef_b[data_count/4]; uint64_t *swap_c0a_in = (uint64_t*)&swap_coef_a[0]; uint64_t *swap_c1a_in = (uint64_t*)&swap_coef_a[1]; uint64_t *swap_c0b_in = (uint64_t*)&swap_coef_b[0]; uint64_t *swap_c1b_in = (uint64_t*)&swap_coef_b[data_count/4]; uint64_t *out_add0 = (uint64_t*)&data_out[0*data_count/4]; uint64_t *out_add1 = (uint64_t*)&data_out[1*data_count/4]; uint64_t *out_sub0 = (uint64_t*)&data_out[2*data_count/4]; uint64_t *out_sub1 = (uint64_t*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(2) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { uint64_t x0 = x0_in[4*i]; uint64_t y0 = y0_in[4*i]; uint64_t conj_c0 = conj_c0a_in[2*i]; uint64_t swap_c0 = swap_c0a_in[2*i]; uint64_t x1 = x1_in[4*i]; uint64_t y1 = y1_in[4*i]; uint64_t conj_c1 = conj_c1a_in[2*i]; uint64_t swap_c1 = swap_c1a_in[2*i]; uint64_t cy0_real = __builtin_e2k_pfmuls(conj_c0, y0); uint64_t cy1_real = __builtin_e2k_pfmuls(conj_c1, y1); uint64_t cy0_imag = __builtin_e2k_pfmuls(swap_c0, y0); uint64_t cy1_imag = __builtin_e2k_pfmuls(swap_c1, y1); uint64_t cy0 = __builtin_e2k_pfhadds(cy0_real, cy0_imag); uint64_t cy1 = __builtin_e2k_pfhadds(cy1_real, cy1_imag); uint64_t add0 = __builtin_e2k_pfadds(x0, cy0); uint64_t sub0 = __builtin_e2k_pfsubs(x0, cy0); uint64_t add1 = __builtin_e2k_pfadds(x1, cy1); uint64_t sub1 = __builtin_e2k_pfsubs(x1, cy1); x0 = add0; y0 = add1; conj_c0 = conj_c0b_in[i]; swap_c0 = swap_c0b_in[i]; x1 = sub0; y1 = sub1; conj_c1 = conj_c1b_in[i]; swap_c1 = swap_c1b_in[i]; cy0_real = __builtin_e2k_pfmuls(conj_c0, y0); cy1_real = __builtin_e2k_pfmuls(conj_c1, y1); cy0_imag = __builtin_e2k_pfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_pfmuls(swap_c1, y1); cy0 = __builtin_e2k_pfhadds(cy0_real, cy0_imag); cy1 = __builtin_e2k_pfhadds(cy1_real, cy1_imag); out_add0[i] = __builtin_e2k_pfadds(x0, cy0); out_sub0[i] = __builtin_e2k_pfsubs(x0, cy0); out_add1[i] = __builtin_e2k_pfadds(x1, cy1); out_sub1[i] = __builtin_e2k_pfsubs(x1, cy1); } }
Основной цикл на ассемблере
.L1964: { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=3, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=3, ind=1, asz=3, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=3, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=2, asz=3, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=16, d=0, incr=1, ind=7, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=1, ind=6, asz=3, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=1, ind=5, asz=3, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=1, ind=4, asz=3, abs=24, disp=0 } .L1045: { loop_mode pfmul_hadds,0,sm %b[71], %b[68], %b[118], %b[31] pfsub_adds,1,sm %b[85], %b[94], %b[107], %b[1] pfsub_rsubs,2,sm %b[85], %b[94], %b[107], %b[0] pfmuls,3,sm %b[54], %b[50], %b[119] pfmuls,4,sm %b[31], %b[18], %b[117] pfmuls,5,sm %b[62], %b[103], %b[115] } { loop_mode pfmul_hadds,0,sm %b[43], %b[116], %g16, %b[85] pfadd_adds,1,sm %b[85], %b[94], %b[33], %b[71] pfadd_rsubs,2,sm %b[85], %b[94], %b[33], %b[68] pfmul_hadds,3,sm %b[108], %b[38], %g17, %b[62] pfmuls,5,sm %b[95], %b[66], %b[116] movad,0 area=3, ind=0, am=0, be=0, %b[43] movad,1 area=3, ind=8, am=1, be=0, %b[54] movad,2 area=3, ind=0, am=0, be=0, %b[33] movad,3 area=3, ind=8, am=1, be=0, %b[38] } { loop_mode pfmul_hadds,0,sm %b[61], %b[76], %g18, %b[107] pfsub_adds,1,sm %b[23], %b[101], %b[87], %b[94] pfsub_rsubs,2,sm %b[23], %b[101], %b[87], %b[95] pfmuls,4,sm %g19, %b[36], %g17 pfmuls,5,sm %b[51], %b[114], %g16 movad,0 area=2, ind=0, am=0, be=0, %b[76] movad,1 area=2, ind=8, am=1, be=0, %b[87] movad,2 area=2, ind=0, am=0, be=0, %b[51] movad,3 area=2, ind=8, am=1, be=0, %b[61] } { loop_mode pfadd_adds,0,sm %b[23], %b[101], %b[109], %b[108] pfadd_rsubs,1,sm %b[23], %b[101], %b[109], %b[109] staad,2 %b[3], %aad2[ %aasti9 + _f32s,_lts0 0x8 ] pfsubs,3,sm %b[30], %b[64], %b[101] pfmuls,4,sm %b[84], %b[74], %g18 staad,5 %b[2], %aad1[ %aasti8 + _f32s,_lts0 0x8 ] movad,0 area=1, ind=16, am=0, be=0, %b[23] movad,1 area=1, ind=0, am=0, be=0, %b[84] movad,2 area=1, ind=16, am=0, be=0, %b[2] movad,3 area=1, ind=0, am=0, be=0, %b[3] } { loop_mode staad,2 %b[73], %aad4[ %aasti11 + _f32s,_lts0 0x8 ] pfmul_hadds,3,sm %b[34], %b[50], %b[119], %b[70] pfadds,4,sm %b[30], %b[64], %b[64] staad,5 %b[70], %aad3[ %aasti10 + _f32s,_lts0 0x8 ] movad,0 area=1, ind=8, am=1, be=0, %b[50] movad,1 area=1, ind=24, am=0, be=0, %g19 movad,2 area=1, ind=8, am=0, be=0, %b[30] movad,3 area=0, ind=24, am=0, be=0, %b[34] } { loop_mode pfmul_hadds,1,sm %b[11], %b[104], %b[113], %b[97] staad,2 %b[96], %aad2[ %aasti9 ] incr,2 %aaincr4 pfsubs,4,sm %b[24], %b[72], %b[112] staad,5 %b[97], %aad1[ %aasti8 ] incr,5 %aaincr4 movad,0 area=0, ind=0, am=0, be=0, %b[11] movad,1 area=0, ind=8, am=0, be=0, %b[96] movad,2 area=1, ind=24, am=1, be=0, %b[104] movad,3 area=0, ind=0, am=0, be=0, %b[73] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmul_hadds,0,sm %b[10], %b[18], %b[117], %b[90] pfmul_hadds,1,sm %b[46], %b[103], %b[115], %b[103] staad,2 %b[110], %aad4[ %aasti11 ] incr,2 %aaincr4 pfmuls,3,sm %b[90], %b[102], %b[111] pfadds,4,sm %b[24], %b[72], %b[72] staad,5 %b[111], %aad3[ %aasti10 ] incr,5 %aaincr4 movad,0 area=0, ind=16, am=0, be=0, %b[18] movad,1 area=0, ind=24, am=1, be=0, %b[46] movad,2 area=0, ind=8, am=1, be=0, %b[10] movad,3 area=0, ind=16, am=0, be=0, %b[24] }
Теоретическая скорость: 8 комплексных чисел за 7 тактов (8/7) = 9.14 Байт/такт
Двойная теоретическая скорость: 18.29 Байт/такт
Замеры скорости
3. stage_radix2_readConjSwap_2x_simd64_unroll4
Здесь происходит раскрутка цикла в 4 раза с помощью опции unroll.
Код на Си
void stage_radix2_readConjSwap_2x_simd64_unroll4(int data_count, myComplex *data_in, myComplex *data_out, myComplex *conj_coef_a, myComplex *conj_coef_b, myComplex *swap_coef_a, myComplex *swap_coef_b) { uint64_t *x0_in = (uint64_t*)&data_in[0]; uint64_t *y0_in = (uint64_t*)&data_in[1]; uint64_t *x1_in = (uint64_t*)&data_in[2]; uint64_t *y1_in = (uint64_t*)&data_in[3]; uint64_t *conj_c0a_in = (uint64_t*)&conj_coef_a[0]; uint64_t *conj_c1a_in = (uint64_t*)&conj_coef_a[1]; uint64_t *conj_c0b_in = (uint64_t*)&conj_coef_b[0]; uint64_t *conj_c1b_in = (uint64_t*)&conj_coef_b[data_count/4]; uint64_t *swap_c0a_in = (uint64_t*)&swap_coef_a[0]; uint64_t *swap_c1a_in = (uint64_t*)&swap_coef_a[1]; uint64_t *swap_c0b_in = (uint64_t*)&swap_coef_b[0]; uint64_t *swap_c1b_in = (uint64_t*)&swap_coef_b[data_count/4]; uint64_t *out_add0 = (uint64_t*)&data_out[0*data_count/4]; uint64_t *out_add1 = (uint64_t*)&data_out[1*data_count/4]; uint64_t *out_sub0 = (uint64_t*)&data_out[2*data_count/4]; uint64_t *out_sub1 = (uint64_t*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(4) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { uint64_t x0 = x0_in[4*i]; uint64_t y0 = y0_in[4*i]; uint64_t conj_c0 = conj_c0a_in[2*i]; uint64_t swap_c0 = swap_c0a_in[2*i]; uint64_t x1 = x1_in[4*i]; uint64_t y1 = y1_in[4*i]; uint64_t conj_c1 = conj_c1a_in[2*i]; uint64_t swap_c1 = swap_c1a_in[2*i]; uint64_t cy0_real = __builtin_e2k_pfmuls(conj_c0, y0); uint64_t cy1_real = __builtin_e2k_pfmuls(conj_c1, y1); uint64_t cy0_imag = __builtin_e2k_pfmuls(swap_c0, y0); uint64_t cy1_imag = __builtin_e2k_pfmuls(swap_c1, y1); uint64_t cy0 = __builtin_e2k_pfhadds(cy0_real, cy0_imag); uint64_t cy1 = __builtin_e2k_pfhadds(cy1_real, cy1_imag); uint64_t add0 = __builtin_e2k_pfadds(x0, cy0); uint64_t sub0 = __builtin_e2k_pfsubs(x0, cy0); uint64_t add1 = __builtin_e2k_pfadds(x1, cy1); uint64_t sub1 = __builtin_e2k_pfsubs(x1, cy1); x0 = add0; y0 = add1; conj_c0 = conj_c0b_in[i]; swap_c0 = swap_c0b_in[i]; x1 = sub0; y1 = sub1; conj_c1 = conj_c1b_in[i]; swap_c1 = swap_c1b_in[i]; cy0_real = __builtin_e2k_pfmuls(conj_c0, y0); cy1_real = __builtin_e2k_pfmuls(conj_c1, y1); cy0_imag = __builtin_e2k_pfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_pfmuls(swap_c1, y1); cy0 = __builtin_e2k_pfhadds(cy0_real, cy0_imag); cy1 = __builtin_e2k_pfhadds(cy1_real, cy1_imag); out_add0[i] = __builtin_e2k_pfadds(x0, cy0); out_sub0[i] = __builtin_e2k_pfsubs(x0, cy0); out_add1[i] = __builtin_e2k_pfadds(x1, cy1); out_sub1[i] = __builtin_e2k_pfsubs(x1, cy1); } }
Основной цикл на ассемблере
.L3317: { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=3, ind=1, asz=2, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=3, ind=1, asz=2, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=3, ind=1, asz=2, abs=4, disp=64 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=3, ind=1, asz=2, abs=4, disp=96 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=3, asz=2, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=3, asz=2, abs=8, disp=32 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=2, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=2, asz=2, abs=12, disp=32 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=7, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=6, asz=3, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=5, asz=3, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=4, asz=3, abs=24, disp=0 } .L2286: { loop_mode pfmul_hadds,0,sm %b[13], %b[110], %g16, %b[110] pfadd_adds,1,sm %b[37], %b[72], %b[111], %b[115] pfmuls,2,sm %b[18], %b[116], %g17 pfadd_rsubs,3,sm %b[37], %b[72], %b[111], %b[111] pfmuls,4,sm %b[115], %g20, %g21 pfmuls,5,sm %g18, %b[107], %g19 } { loop_mode pfmul_hadds,0,sm %b[12], %b[103], %g24, %b[118] pfsub_adds,1,sm %b[30], %b[102], %b[118], %g23 pfmuls,2,sm %b[52], %b[114], %g25 pfsub_rsubs,3,sm %b[30], %b[102], %b[118], %g22 pfmuls,4,sm %g26, %b[99], %g27 pfmuls,5,sm %b[40], %b[80], %b[103] movad,0 area=5, ind=16, am=0, be=0, %b[12] movad,1 area=5, ind=0, am=0, be=0, %b[13] movad,2 area=5, ind=16, am=0, be=0, %b[0] movad,3 area=5, ind=0, am=0, be=0, %b[1] } { loop_mode pfmul_hadds,0,sm %b[7], %r0, %r1, %b[102] pfadd_adds,1,sm %b[30], %b[102], %g29, %g31 pfsubs,2,sm %b[92], %b[109], %r2 pfadd_rsubs,3,sm %b[30], %b[102], %g29, %g30 pfmuls,4,sm %b[21], %b[57], %r3 staad,5 %g28, %aad2[ %aasti9 + _f32s,_lts0 0x8 ] movad,0 area=5, ind=24, am=0, be=0, %b[21] movad,1 area=5, ind=8, am=1, be=0, %b[30] movad,2 area=5, ind=24, am=0, be=0, %b[7] movad,3 area=5, ind=8, am=1, be=0, %b[18] } { loop_mode pfmul_hadds,1,sm %b[46], %b[108], %r4, %b[109] pfadds,2,sm %b[92], %b[109], %r5 pfsubs,4,sm %b[81], %b[100], %b[108] staad,5 %b[119], %aad1[ %aasti8 + _f32s,_lts0 0x8 ] movad,0 area=4, ind=16, am=0, be=0, %b[46] movad,1 area=4, ind=0, am=0, be=0, %b[52] movad,2 area=4, ind=16, am=0, be=0, %b[37] movad,3 area=4, ind=0, am=0, be=0, %b[40] } { loop_mode pfmul_hadds,0,sm %b[49], %b[63], %r6, %b[100] pfmul_hadds,1,sm %b[6], %b[116], %g17, %b[116] pfmuls,2,sm %b[71], %b[88], %g17 pfadds,4,sm %b[81], %b[100], %b[101] staad,5 %b[101], %aad4[ %aasti11 + _f32s,_lts0 0x8 ] movad,0 area=4, ind=24, am=0, be=0, %b[63] movad,1 area=4, ind=8, am=1, be=0, %b[71] movad,2 area=4, ind=24, am=0, be=0, %b[6] movad,3 area=4, ind=8, am=1, be=0, %b[49] } { loop_mode pfmul_hadds,0,sm %b[43], %b[114], %g25, %g29 pfsub_adds,1,sm %b[68], %b[112], %b[117], %g28 pfsubs,2,sm %b[84], %r9, %b[114] pfsubs,4,sm %b[89], %b[106], %r0 staad,5 %r8, %aad3[ %aasti10 + _f32s,_lts0 0x8 ] movad,0 area=3, ind=16, am=0, be=0, %b[81] movad,1 area=3, ind=0, am=0, be=0, %b[93] movad,2 area=3, ind=0, am=0, be=0, %b[43] movad,3 area=3, ind=16, am=0, be=0, %b[72] } { loop_mode pfsub_rsubs,1,sm %b[68], %b[112], %b[117], %b[117] pfmuls,2,sm %b[34], %r2, %g25 pfadds,4,sm %b[89], %b[106], %b[106] staad,5 %r10, %aad2[ %aasti9 + _f32s,_lts0 0x18 ] movad,0 area=3, ind=8, am=1, be=0, %b[34] movad,1 area=3, ind=24, am=0, be=0, %b[96] movad,2 area=3, ind=8, am=1, be=0, %b[89] movad,3 area=3, ind=24, am=0, be=0, %b[92] } { loop_mode pfmul_hadds,0,sm %b[98], %b[99], %g27, %b[107] pfadd_adds,1,sm %b[68], %b[112], %r11, %b[99] pfmuls,2,sm %b[75], %r5, %r12 pfmul_hadds,3,sm %b[94], %b[107], %g19, %b[98] pfmuls,4,sm %b[25], %b[108], %g16 staad,5 %b[113], %aad1[ %aasti8 + _f32s,_lts0 0x18 ] movad,0 area=2, ind=8, am=0, be=0, %g19 movad,1 area=0, ind=24, am=0, be=0, %b[75] movad,2 area=2, ind=0, am=0, be=0, %b[25] movad,3 area=2, ind=8, am=0, be=0, %b[113] } { loop_mode pfmul_hadds,0,sm %b[91], %g20, %g21, %r9 pfadd_rsubs,1,sm %b[68], %b[112], %r11, %r8 staad,2 %r13, %aad4[ %aasti11 + _f32s,_lts0 0x18 ] pfadds,3,sm %b[84], %r9, %b[112] pfmuls,4,sm %b[67], %b[101], %g24 staad,5 %b[105], %aad3[ %aasti10 + _f32s,_lts0 0x18 ] movad,0 area=2, ind=0, am=0, be=0, %b[67] movad,1 area=1, ind=24, am=0, be=0, %g20 movad,2 area=2, ind=24, am=0, be=0, %g18 movad,3 area=1, ind=24, am=0, be=0, %b[105] } { loop_mode pfmul_hadds,0,sm %b[97], %b[88], %g17, %b[68] pfsub_adds,1,sm %b[62], %r16, %b[110], %r10 staad,2 %r15, %aad2[ %aasti9 ] pfmul_hadds,3,sm %b[36], %b[77], %b[104], %b[104] pfmuls,4,sm %b[17], %r0, %r1 staad,5 %r14, %aad1[ %aasti8 ] movad,0 area=2, ind=16, am=0, be=0, %b[17] movad,1 area=2, ind=24, am=1, be=0, %g26 movad,2 area=2, ind=16, am=1, be=0, %b[36] movad,3 area=0, ind=24, am=0, be=0, %b[97] } { loop_mode pfmul_hadds,0,sm %b[85], %b[57], %r3, %b[110] pfmul_hadds,1,sm %b[22], %r2, %g25, %b[115] staad,2 %b[115], %aad4[ %aasti11 ] pfsub_rsubs,3,sm %b[62], %r16, %b[110], %b[111] pfmuls,4,sm %b[56], %b[106], %r4 staad,5 %b[111], %aad3[ %aasti10 ] movad,0 area=1, ind=0, am=0, be=0, %b[22] movad,1 area=1, ind=8, am=0, be=0, %b[57] movad,2 area=1, ind=0, am=0, be=0, %b[56] movad,3 area=1, ind=16, am=0, be=0, %b[77] } { loop_mode pfmul_hadds,0,sm %b[76], %b[80], %b[103], %r16 pfadd_adds,1,sm %b[62], %r16, %b[118], %r13 staad,2 %g23, %aad2[ %aasti9 + _f32s,_lts0 0x10 ] incr,2 %aaincr4 pfadd_rsubs,3,sm %b[62], %r16, %b[118], %b[103] pfmuls,4,sm %b[29], %b[61], %r6 staad,5 %g22, %aad1[ %aasti8 + _f32s,_lts0 0x10 ] incr,5 %aaincr4 movad,0 area=1, ind=16, am=1, be=0, %b[80] movad,1 area=0, ind=0, am=0, be=0, %b[29] movad,2 area=1, ind=8, am=1, be=0, %b[76] movad,3 area=0, ind=0, am=0, be=0, %b[62] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmul_hadds,0,sm %b[53], %r5, %r12, %r11 pfsub_adds,1,sm %b[35], %b[70], %b[102], %r15 staad,2 %g31, %aad4[ %aasti11 + _f32s,_lts0 0x10 ] incr,2 %aaincr4 pfsub_rsubs,3,sm %b[35], %b[70], %b[102], %r14 pfmuls,4,sm %g19, %b[75], %b[102] staad,5 %g30, %aad3[ %aasti10 + _f32s,_lts0 0x10 ] incr,5 %aaincr4 movad,0 area=0, ind=16, am=0, be=0, %b[85] movad,1 area=0, ind=8, am=1, be=0, %b[84] movad,2 area=0, ind=8, am=1, be=0, %b[53] movad,3 area=0, ind=16, am=0, be=0, %b[88] }
Теоретическая скорость: 16 комплексных чисел за 13 тактов (16/13) = 9.85 Байт/такт
Двойная теоретическая скорость: 19.69 Байт/такт
Замеры скорости
4. stage_radix2_readConjSwap_2x_simd128
Здесь происходит ручная раскрутка алгоритма stage_radix2_readConjSwap_simd128 в 2 раза.
Код на Си
void stage_radix2_readConjSwap_2x_simd128(int data_count, myComplex *data_in, myComplex *data_out, myComplex *conj_coef_a, myComplex *conj_coef_b, myComplex *swap_coef_a, myComplex *swap_coef_b) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *xy1_in = (__v2di*)&data_in[2]; __v2di *xy2_in = (__v2di*)&data_in[4]; __v2di *xy3_in = (__v2di*)&data_in[6]; __v2di *conj_c0a_in = (__v2di*)&conj_coef_a[0]; __v2di *conj_c1a_in = (__v2di*)&conj_coef_a[2]; __v2di *conj_c0b_in = (__v2di*)&conj_coef_b[0]; __v2di *conj_c1b_in = (__v2di*)&conj_coef_b[data_count/4]; __v2di *swap_c0a_in = (__v2di*)&swap_coef_a[0]; __v2di *swap_c1a_in = (__v2di*)&swap_coef_a[2]; __v2di *swap_c0b_in = (__v2di*)&swap_coef_b[0]; __v2di *swap_c1b_in = (__v2di*)&swap_coef_b[data_count/4]; __v2di *out_add0 = (__v2di*)&data_out[0*data_count/4]; __v2di *out_add1 = (__v2di*)&data_out[1*data_count/4]; __v2di *out_sub0 = (__v2di*)&data_out[2*data_count/4]; __v2di *out_sub1 = (__v2di*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/8; ++i) { __v2di xy0 = xy0_in[4*i]; __v2di xy1 = xy1_in[4*i]; __v2di conj_c0 = conj_c0a_in[2*i]; __v2di swap_c0 = swap_c0a_in[2*i]; __v2di xy2 = xy2_in[4*i]; __v2di xy3 = xy3_in[4*i]; __v2di conj_c1 = conj_c1a_in[2*i]; __v2di swap_c1 = swap_c1a_in[2*i]; __v2di x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di cy0_real = __builtin_e2k_qpfmuls(conj_c0, y0); __v2di cy1_real = __builtin_e2k_qpfmuls(conj_c1, y1); __v2di cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); __v2di cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); __v2di cy0_rrii = __builtin_e2k_qpfhadds(cy0_real, cy0_imag); __v2di cy1_rrii = __builtin_e2k_qpfhadds(cy1_real, cy1_imag); __v2di cy0 = __builtin_e2k_qpshufb(cy0_rrii, cy0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di cy1 = __builtin_e2k_qpshufb(cy1_rrii, cy1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di add0 = __builtin_e2k_qpfadds(x0, cy0); __v2di sub0 = __builtin_e2k_qpfsubs(x0, cy0); __v2di add1 = __builtin_e2k_qpfadds(x1, cy1); __v2di sub1 = __builtin_e2k_qpfsubs(x1, cy1); xy0 = add0; xy1 = add1; conj_c0 = conj_c0b_in[i]; swap_c0 = swap_c0b_in[i]; xy2 = sub0; xy3 = sub1; conj_c1 = conj_c1b_in[i]; swap_c1 = swap_c1b_in[i]; x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); cy0_real = __builtin_e2k_qpfmuls(conj_c0, y0); cy1_real = __builtin_e2k_qpfmuls(conj_c1, y1); cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); cy0_rrii = __builtin_e2k_qpfhadds(cy0_real, cy0_imag); cy1_rrii = __builtin_e2k_qpfhadds(cy1_real, cy1_imag); cy0 = __builtin_e2k_qpshufb(cy0_rrii, cy0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); cy1 = __builtin_e2k_qpshufb(cy1_rrii, cy1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); out_add0[i] = __builtin_e2k_qpfadds(x0, cy0); out_sub0[i] = __builtin_e2k_qpfsubs(x0, cy0); out_add1[i] = __builtin_e2k_qpfadds(x1, cy1); out_sub1[i] = __builtin_e2k_qpfsubs(x1, cy1); } }
Основной цикл на ассемблере
.L4621: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=3, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=3, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=3, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=3, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=3, abs=24, disp=0 } .L3922: { loop_mode qpfsubs,2,sm %b[65], %b[78], %b[95] qpfadds,3,sm %b[28], %b[3], %b[0] qpshufb,4,sm %b[69], %b[69], %r20, %b[1] qpfadds,5,sm %b[65], %b[78], %b[96] } { loop_mode qpfmul_hadds,0,sm %b[74], %b[30], %b[76], %b[17] qpshufb,1,sm %b[50], %b[55], %r18, %b[28] qpfsubs,2,sm %b[77], %b[92], %b[97] qpshufb,4,sm %b[88], %b[91], %r18, %b[21] qpfadds,5,sm %b[77], %b[92], %b[98] movaqp,0 area=3, ind=0, am=1, be=0, %b[8] movaqp,1 area=2, ind=0, am=1, be=0, %b[3] } { loop_mode qpfmul_hadds,0,sm %b[68], %b[89], %b[71], %b[65] qpshufb,1,sm %b[90], %b[93], %r19, %b[59] qpfmuls,2,sm %b[86], %b[28], %b[74] qpshufb,4,sm %b[62], %b[62], %r20, %b[76] qpfmuls,5,sm %b[82], %b[87], %b[69] movaqp,0 area=0, ind=0, am=0, be=0, %b[51] movaqp,1 area=0, ind=16, am=1, be=0, %b[46] movaqp,2 area=3, ind=0, am=1, be=0, %b[33] movaqp,3 area=2, ind=0, am=1, be=0, %b[30] } { loop_mode qpshufb,0,sm %b[4], %b[29], %r18, %b[85] qpshufb,1,sm %b[2], %b[81], %r19, %b[71] qpfadds,2,sm %b[58], %b[54], %b[77] qpfsubs,3,sm %b[58], %b[54], %b[89] qpshufb,4,sm %b[27], %b[27], %r20, %b[90] qpfsubs,5,sm %b[26], %b[1], %b[86] movaqp,0 area=1, ind=16, am=1, be=0, %b[78] movaqp,1 area=1, ind=0, am=0, be=0, %b[82] movaqp,2 area=1, ind=16, am=1, be=0, %b[62] movaqp,3 area=1, ind=0, am=0, be=0, %b[68] } { loop_mode qpfmul_hadds,0,sm %b[47], %b[23], %b[94], %b[58] qpshufb,1,sm %b[52], %b[57], %r19, %b[54] staaqp,2 %b[95], %aad1[ %aasti8 ] incr,2 %aaincr0 qpfmuls,3,sm %b[20], %b[21], %b[92] qpshufb,4,sm %b[0], %b[79], %r18, %b[81] staaqp,5 %b[96], %aad2[ %aasti9 ] incr,5 %aaincr0 movaqp,2 area=0, ind=0, am=0, be=0, %b[27] movaqp,3 area=0, ind=16, am=1, be=0, %b[2] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfmul_hadds,0,sm %b[44], %b[83], %b[49], %b[23] qpshufb,1,sm %b[6], %b[31], %r19, %b[20] staaqp,2 %b[97], %aad3[ %aasti10 ] incr,2 %aaincr0 qpfmuls,3,sm %b[15], %b[81], %b[47] qpshufb,4,sm %b[19], %b[19], %r20, %b[52] staaqp,5 %b[98], %aad4[ %aasti11 ] incr,5 %aaincr0 }
Теоретическая скорость: 8 комплексных чисел за 6 тактов (8/6) = 10.67 Байт/такт
Двойная теоретическая скорость: 21.33 Байт/такт
Замеры скорости
5. stage_radix2_readConjSwap_2x_simd128_v2
Перетасовали код, чтобы уменьшить число инструкций.
Код на Си
void stage_radix2_readConjSwap_2x_simd128_v2(int data_count, myComplex *data_in, myComplex *data_out, myComplex *conj_coef_a, myComplex *conj_coef_b, myComplex *swap_coef_a, myComplex *swap_coef_b) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *xy1_in = (__v2di*)&data_in[2]; __v2di *xy2_in = (__v2di*)&data_in[4]; __v2di *xy3_in = (__v2di*)&data_in[6]; __v2di *conj_c0a_in = (__v2di*)&conj_coef_a[0]; __v2di *conj_c1a_in = (__v2di*)&conj_coef_a[2]; __v2di *conj_c0b_in = (__v2di*)&conj_coef_b[0]; __v2di *conj_c1b_in = (__v2di*)&conj_coef_b[data_count/4]; __v2di *swap_c0a_in = (__v2di*)&swap_coef_a[0]; __v2di *swap_c1a_in = (__v2di*)&swap_coef_a[2]; __v2di *swap_c0b_in = (__v2di*)&swap_coef_b[0]; __v2di *swap_c1b_in = (__v2di*)&swap_coef_b[data_count/4]; __v2di *out_add0 = (__v2di*)&data_out[0*data_count/4]; __v2di *out_add1 = (__v2di*)&data_out[1*data_count/4]; __v2di *out_sub0 = (__v2di*)&data_out[2*data_count/4]; __v2di *out_sub1 = (__v2di*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/8; ++i) { __v2di xy0 = xy0_in[4*i]; __v2di xy1 = xy1_in[4*i]; __v2di conj_c0 = conj_c0a_in[2*i]; __v2di swap_c0 = swap_c0a_in[2*i]; __v2di xy2 = xy2_in[4*i]; __v2di xy3 = xy3_in[4*i]; __v2di conj_c1 = conj_c1a_in[2*i]; __v2di swap_c1 = swap_c1a_in[2*i]; __v2di x0_rrii = __builtin_e2k_qppermb(xy1, xy0, (__v2di){0x1312111003020100, 0x1716151407060504}); __v2di x1_rrii = __builtin_e2k_qppermb(xy3, xy2, (__v2di){0x1312111003020100, 0x1716151407060504}); __v2di y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di cy0_real = __builtin_e2k_qpfmuls(conj_c0, y0); __v2di cy1_real = __builtin_e2k_qpfmuls(conj_c1, y1); __v2di cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); __v2di cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); __v2di cy0_rrii = __builtin_e2k_qpfhadds(cy0_real, cy0_imag); __v2di cy1_rrii = __builtin_e2k_qpfhadds(cy1_real, cy1_imag); __v2di add0_rrii = __builtin_e2k_qpfadds(x0_rrii, cy0_rrii); __v2di sub0_rrii = __builtin_e2k_qpfsubs(x0_rrii, cy0_rrii); __v2di add1_rrii = __builtin_e2k_qpfadds(x1_rrii, cy1_rrii); __v2di sub1_rrii = __builtin_e2k_qpfsubs(x1_rrii, cy1_rrii); __v2di xy0_rrii = add0_rrii; __v2di xy1_rrii = add1_rrii; conj_c0 = conj_c0b_in[i]; swap_c0 = swap_c0b_in[i]; __v2di xy2_rrii = sub0_rrii; __v2di xy3_rrii = sub1_rrii; conj_c1 = conj_c1b_in[i]; swap_c1 = swap_c1b_in[i]; __v2di x0 = __builtin_e2k_qppermb(xy1_rrii, xy0_rrii, (__v2di){0x0B0A090803020100, 0x1B1A191813121110}); __v2di x1 = __builtin_e2k_qppermb(xy3_rrii, xy2_rrii, (__v2di){0x0B0A090803020100, 0x1B1A191813121110}); y0 = __builtin_e2k_qppermb(xy1_rrii, xy0_rrii, (__v2di){0x0F0E0D0C07060504, 0x1F1E1D1C17161514}); y1 = __builtin_e2k_qppermb(xy3_rrii, xy2_rrii, (__v2di){0x0F0E0D0C07060504, 0x1F1E1D1C17161514}); cy0_real = __builtin_e2k_qpfmuls(conj_c0, y0); cy1_real = __builtin_e2k_qpfmuls(conj_c1, y1); cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); cy0_rrii = __builtin_e2k_qpfhadds(cy0_real, cy0_imag); cy1_rrii = __builtin_e2k_qpfhadds(cy1_real, cy1_imag); __v2di cy0 = __builtin_e2k_qpshufb(cy0_rrii, cy0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di cy1 = __builtin_e2k_qpshufb(cy1_rrii, cy1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); out_add0[i] = __builtin_e2k_qpfadds(x0, cy0); out_sub0[i] = __builtin_e2k_qpfsubs(x0, cy0); out_add1[i] = __builtin_e2k_qpfadds(x1, cy1); out_sub1[i] = __builtin_e2k_qpfsubs(x1, cy1); } }
Основной цикл на ассемблере
.L5345: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=3, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=3, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=3, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=3, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=3, abs=24, disp=0 } .L4696: { loop_mode qpfmul_hadds,0,sm %b[62], %b[79], %b[110], %b[62] qpshufb,1,sm %b[83], %b[83], %r11, %b[110] qpfsubs,2,sm %b[119], %b[118], %b[113] qpfmuls,3,sm %b[73], %b[103], %b[73] qppermb,4,sm %b[69], %b[12], %r9, %b[115] qpfadds,5,sm %b[119], %b[118], %b[114] movaqp,0 area=0, ind=0, am=0, be=0, %b[0] movaqp,1 area=1, ind=0, am=0, be=0, %b[69] movaqp,2 area=0, ind=0, am=0, be=0, %b[12] movaqp,3 area=0, ind=16, am=1, be=0, %b[1] } { loop_mode qpfmul_hadds,0,sm %b[44], %b[26], %b[108], %b[79] qppermb,1,sm %b[92], %b[76], %r0, %b[117] qpfmuls,2,sm %b[57], %b[77], %b[108] qpfmuls,3,sm %b[80], %b[109], %b[80] qpshufb,4,sm %b[66], %b[66], %r11, %b[116] qpfadds,5,sm %b[115], %b[25], %b[66] movaqp,0 area=0, ind=16, am=1, be=0, %b[57] movaqp,1 area=1, ind=16, am=1, be=0, %b[76] movaqp,2 area=3, ind=0, am=1, be=0, %b[26] movaqp,3 area=2, ind=0, am=1, be=0, %b[44] } { loop_mode qpfmul_hadds,0,sm %b[106], %b[111], %b[82], %b[92] qpshufb,1,sm %b[3], %b[14], %r12, %b[107] qpfmuls,2,sm %b[41], %b[24], %b[106] qpfadds,3,sm %g16, %b[98], %b[82] qpshufb,4,sm %b[59], %b[2], %r12, %b[101] qpfsubs,5,sm %b[115], %b[25], %b[83] movaqp,0 area=3, ind=0, am=1, be=0, %b[25] movaqp,1 area=2, ind=0, am=1, be=0, %b[41] movaqp,2 area=1, ind=0, am=0, be=0, %b[93] movaqp,3 area=1, ind=16, am=1, be=0, %b[100] } { loop_mode qpfsubs,0,sm %g18, %b[110], %g17 qppermb,1,sm %b[11], %b[22], %r9, %g16 staaqp,2 %g17, %aad1[ %aasti8 ] incr,2 %aaincr0 qpfsubs,3,sm %g16, %b[98], %b[11] qppermb,4,sm %b[13], %b[85], %r10, %b[22] staaqp,5 %b[112], %aad2[ %aasti9 ] incr,5 %aaincr0 } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfmul_hadds,0,sm %b[99], %b[105], %b[75], %b[19] qppermb,1,sm %b[84], %b[68], %r10, %b[75] staaqp,2 %b[113], %aad3[ %aasti10 ] incr,2 %aaincr0 qpfadds,3,sm %g18, %b[110], %b[110] qppermb,4,sm %b[19], %b[91], %r0, %g18 staaqp,5 %b[114], %aad4[ %aasti11 ] incr,5 %aaincr0 }
Теоретическая скорость: 8 комплексных чисел за 5 тактов (8/5) = 12.8 Байт/такт
Двойная теоретическая скорость: 25.6 Байт/такт
Замеры скорости
Итоги по stage_radix2_readConjSwap_2x
Скорости выросли по сравнению с исходными версиями stage_radix2_readConjSwap.
График FFT находится здесь.
stage_radix4
Схема алгоритма Stage для версии «radix-4».
Один проход по stage_radix4 совершает ту же работу, что 2 прохода по stage_radix2. Поэтому скорость stage_radix4 будем умножать на 2 для удобства сравнения с stage_radix2 (этот факт подписан на оси графика и в выводе консоли).
1. stage_radix4_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void stage_radix4_etalon(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE) { myComplex *x_in = &data_in[0]; myComplex *y_in = &data_in[1]; myComplex *z_in = &data_in[2]; myComplex *w_in = &data_in[3]; myComplex *c_in = coefC; myComplex *d_in = coefD; myComplex *e_in = coefE; myComplex *out_0 = &data_out[0*data_count/4]; myComplex *out_1 = &data_out[1*data_count/4]; myComplex *out_2 = &data_out[2*data_count/4]; myComplex *out_3 = &data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) // #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { myComplex x = x_in[4*i]; myComplex y = y_in[4*i]; myComplex z = z_in[4*i]; myComplex w = w_in[4*i]; myComplex c = c_in[i]; myComplex d = d_in[i]; myComplex e = e_in[i]; myComplex cy = complex_mul(c, y); myComplex dz = complex_mul(d, z); myComplex ew = complex_mul(e, w); myComplex add02 = complex_add( x, dz); myComplex sub02 = complex_sub( x, dz); myComplex add13 = complex_add(cy, ew); myComplex sub13 = complex_sub(cy, ew); myComplex sub13i = (myComplex){.real = -sub13.imag, .imag = sub13.real}; out_0[i] = complex_add(add02, add13); out_1[i] = complex_sub(sub02, sub13i); out_2[i] = complex_sub(add02, add13); out_3[i] = complex_add(sub02, sub13i); } }
Основной цикл на ассемблере
.L868: { fapb ct=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=4, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=3, asz=3, abs=8, disp=0 } { fapb ct=1, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=2, asz=4, abs=16, disp=0 } .L236: { loop_mode fmul_adds,0,sm %b[66], %b[72], %b[54], %b[1] fsub_rsubs,1,sm %b[12], %b[71], %b[55], %b[5] fsub_adds,2,sm %b[12], %b[71], %b[55], %b[45] fmuls,4,sm %b[59], %b[58], %b[52] fsubs,5,sm %b[79], %b[75], %b[53] movaw,3 area=0, ind=4, am=0, be=0, %b[0] } { loop_mode fmul_rsubs,0,sm %b[66], %b[60], %b[82], %b[71] fadd_adds,1,sm %b[24], %b[83], %b[90], %b[72] fadd_rsubs,2,sm %b[24], %b[83], %b[90], %b[76] fmuls,4,sm %b[59], %b[70], %b[80] fadds,5,sm %b[79], %b[75], %b[88] movaw,1 area=2, ind=4, am=0, be=0, %b[55] movaw,2 area=0, ind=0, am=0, be=0, %b[12] movaw,3 area=0, ind=24, am=0, be=0, %b[54] } { loop_mode fmul_rsubs,0,sm %b[42], %b[17], %b[84], %b[75] fmul_rsubs,1,sm %b[32], %b[67], %b[85], %b[79] staaw,2 %b[36], %aad3[ %aasti7 ] fmuls,3,sm %b[29], %b[46], %b[82] fmuls,4,sm %b[35], %b[23], %b[83] staaw,5 %b[39], %aad1[ %aasti5 ] movaw,0 area=2, ind=0, am=1, be=0, %b[60] movaw,1 area=1, ind=0, am=0, be=0, %b[24] movaw,2 area=0, ind=16, am=0, be=0, %b[59] movaw,3 area=0, ind=28, am=0, be=0, %b[66] } { loop_mode fmul_adds,0,sm %b[40], %b[46], %b[86], %b[39] fmul_adds,1,sm %b[32], %b[25], %b[87], %b[67] staaw,2 %b[11], %aad4[ %aasti8 + _f32s,_lts0 0x4 ] fmuls,3,sm %b[27], %b[13], %b[84] fmuls,4,sm %b[35], %b[65], %b[85] staaw,5 %b[51], %aad2[ %aasti6 + _f32s,_lts0 0x4 ] movaw,0 area=1, ind=4, am=1, be=0, %b[29] movaw,1 area=0, ind=0, am=0, be=0, %b[36] movaw,2 area=0, ind=20, am=0, be=0, %b[17] movaw,3 area=0, ind=12, am=0, be=0, %b[42] } { loop_mode fsub_rsubs,0,sm %b[22], %b[81], %b[48], %b[32] fsub_adds,1,sm %b[22], %b[81], %b[48], %b[35] staaw,2 %b[7], %aad3[ %aasti7 + _f32s,_lts0 0x4 ] incr,2 %aaincr3 fsubs,4,sm %b[3], %b[43], %b[46] staaw,5 %b[47], %aad1[ %aasti5 + _f32s,_lts0 0x4 ] incr,5 %aaincr3 movaw,1 area=0, ind=4, am=1, be=0, %b[25] movaw,3 area=0, ind=8, am=1, be=0, %b[11] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END fadd_adds,0,sm %b[10], %b[69], %b[50], %b[7] fadd_rsubs,1,sm %b[10], %b[69], %b[50], %b[47] staaw,2 %b[74], %aad4[ %aasti8 ] incr,2 %aaincr3 fadds,4,sm %b[43], %b[3], %b[48] staaw,5 %b[78], %aad2[ %aasti6 ] incr,5 %aaincr3 }
Теоретическая скорость: 4 комплексных числа за 6 тактов (4/6) = 5.33 Байт/такт
Двойная теоретическая скорость: 10.67 Байт/такт
Замеры скорости
2. stage_radix4_etalon_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью опции unroll.
Код на Си
void stage_radix4_etalon_unroll2(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE) { myComplex *x_in = &data_in[0]; myComplex *y_in = &data_in[1]; myComplex *z_in = &data_in[2]; myComplex *w_in = &data_in[3]; myComplex *c_in = coefC; myComplex *d_in = coefD; myComplex *e_in = coefE; myComplex *out_0 = &data_out[0*data_count/4]; myComplex *out_1 = &data_out[1*data_count/4]; myComplex *out_2 = &data_out[2*data_count/4]; myComplex *out_3 = &data_out[3*data_count/4]; #pragma ivdep #pragma unroll(2) // #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { myComplex x = x_in[4*i]; myComplex y = y_in[4*i]; myComplex z = z_in[4*i]; myComplex w = w_in[4*i]; myComplex c = c_in[i]; myComplex d = d_in[i]; myComplex e = e_in[i]; myComplex cy = complex_mul(c, y); myComplex dz = complex_mul(d, z); myComplex ew = complex_mul(e, w); myComplex add02 = complex_add( x, dz); myComplex sub02 = complex_sub( x, dz); myComplex add13 = complex_add(cy, ew); myComplex sub13 = complex_sub(cy, ew); myComplex sub13i = (myComplex){.real = -sub13.imag, .imag = sub13.real}; out_0[i] = complex_add(add02, add13); out_1[i] = complex_sub(sub02, sub13i); out_2[i] = complex_sub(add02, add13); out_3[i] = complex_add(sub02, sub13i); } }
Основной цикл на ассемблере
.L2050: { fapb ct=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=4, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=3, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=16, disp=0 } .L913: { loop_mode insfd,0,sm %b[67], %r7, %b[70], %b[0] pfmul_rsubs,1,sm %b[78], %b[5], %g16, %b[30] pfadd_adds,2,sm %b[50], %b[32], %b[77], %b[5] pshufb,3,sm %b[22], %b[47], %r0, %b[67] insfd,4,sm %b[30], %r7, %b[31], %b[1] pfmuls,5,sm %b[83], %b[58], %g16 } { loop_mode insfd,0,sm %b[24], %r7, %b[49], %b[78] pfsub_adds,1,sm %b[50], %b[32], %b[81], %b[31] pfsub_adds,2,sm %b[6], %b[25], %b[71], %b[24] pshufb,3,sm %b[61], %b[8], %r0, %b[83] pshufb,4,sm %b[26], %b[33], %r0, %b[87] pfmuls,5,sm %b[83], %b[3], %b[70] } { loop_mode insfd,0,sm %b[63], %r7, %b[10], %b[71] pfsub_rsubs,1,sm %b[50], %b[32], %b[81], %b[36] pfsub_rsubs,2,sm %b[6], %b[25], %b[71], %b[35] insfd,3,sm %b[76], %r7, %b[79], %b[10] pshufb,4,sm %b[37], %b[38], %r0, %b[76] } { loop_mode insfd,0,sm %b[85], %r7, %b[86], %b[43] pfadd_rsubs,1,sm %b[50], %b[32], %b[77], %b[39] pfadd_rsubs,2,sm %b[6], %b[25], %b[75], %b[32] insfd,3,sm %b[80], %r7, %b[84], %b[40] pshufb,4,sm %b[34], %b[41], %r0, %b[77] } { loop_mode pfmuls,0,sm %b[67], %b[43], %b[25] pfadd_adds,1,sm %b[6], %b[25], %b[75], %b[49] staad,2 %b[87], %aad1[ %aasti5 + _f32s,_lts0 0x8 ] pfsubs,3,sm %b[64], %b[21], %b[79] pshufb,4,sm %b[51], %b[7], %r0, %b[84] pfadds,5,sm %b[11], %b[20], %b[75] movad,0 area=2, ind=0, am=0, be=0, %b[6] movaw,1 area=0, ind=24, am=0, be=0, %b[81] movaw,3 area=0, ind=24, am=0, be=0, %b[80] } { loop_mode pfmuls,0,sm %b[83], %b[40], %b[50] pfmul_adds,1,sm %b[78], %b[45], %b[69], %b[17] staad,2 %b[76], %aad3[ %aasti7 + _f32s,_lts0 0x8 ] pfsubs,3,sm %b[11], %b[20], %b[69] insfd,4,sm %b[59], %r7, %b[17], %b[76] movad,0 area=1, ind=0, am=0, be=0, %b[45] movad,1 area=1, ind=8, am=1, be=0, %b[20] movad,3 area=1, ind=0, am=0, be=0, %b[11] } { loop_mode insfd,0,sm %b[72], %r7, %b[82], %b[54] pfmul_adds,1,sm %b[71], %b[42], %g17, %b[60] staad,2 %b[77], %aad2[ %aasti6 + _f32s,_lts0 0x8 ] pfmuls,3,sm %b[83], %b[14], %g17 insfd,4,sm %b[73], %r7, %b[74], %b[42] pfmuls,5,sm %b[67], %b[10], %b[67] movad,0 area=2, ind=8, am=1, be=0, %b[59] movaw,1 area=0, ind=4, am=0, be=0, %b[66] movad,2 area=1, ind=8, am=1, be=0, %b[53] movaw,3 area=0, ind=4, am=0, be=0, %b[63] } { loop_mode insfd,0,sm %b[26], %r7, %b[33], %b[83] pfmul_rsubs,1,sm %b[71], %b[16], %b[52], %b[16] staad,2 %b[84], %aad4[ %aasti8 + _f32s,_lts0 0x8 ] insfd,4,sm %b[37], %r7, %b[38], %b[82] pfadds,5,sm %b[21], %b[64], %b[73] movaw,0 area=0, ind=0, am=0, be=0, %b[72] movaw,1 area=0, ind=8, am=0, be=0, %b[77] movaw,2 area=0, ind=0, am=0, be=0, %b[71] movaw,3 area=0, ind=8, am=0, be=0, %b[74] } { loop_mode insfd,0,sm %b[51], %r7, %b[7], %b[85] pfmul_rsubs,1,sm %b[78], %b[12], %b[27], %b[7] staad,2 %b[83], %aad1[ %aasti5 ] incr,2 %aaincr3 insfd,4,sm %b[34], %r7, %b[41], %b[86] staad,5 %b[82], %aad3[ %aasti7 ] incr,5 %aaincr3 movaw,0 area=0, ind=28, am=0, be=0, %b[82] movaw,1 area=0, ind=12, am=0, be=0, %b[84] movaw,2 area=0, ind=28, am=0, be=0, %b[78] movaw,3 area=0, ind=12, am=0, be=0, %b[83] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmul_adds,1,sm %b[76], %b[58], %b[70], %b[21] staad,2 %b[85], %aad4[ %aasti8 ] incr,2 %aaincr3 insfd,3,sm %b[80], %r7, %b[81], %b[12] pshufb,4,sm %b[57], %b[15], %r0, %b[81] staad,5 %b[86], %aad2[ %aasti6 ] incr,5 %aaincr3 movaw,0 area=0, ind=16, am=0, be=0, %b[27] movaw,1 area=0, ind=20, am=1, be=0, %b[80] movaw,2 area=0, ind=16, am=0, be=0, %b[26] movaw,3 area=0, ind=20, am=1, be=0, %b[70] }
Теоретическая скорость: 8 комплексных чисел за 10 тактов (8/10) = 6.4 Байт/такт
Двойная теоретическая скорость: 12.8 Байт/такт
Замеры скорости
Видим ускорение.
3. stage_radix4_simd64
Вычисления делаем аналогично stage_radix2_simd64.
Код на Си
void stage_radix4_simd64(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE) { uint64_t *x_in = (uint64_t*)&data_in[0]; uint64_t *y_in = (uint64_t*)&data_in[1]; uint64_t *z_in = (uint64_t*)&data_in[2]; uint64_t *w_in = (uint64_t*)&data_in[3]; uint64_t *c_in = (uint64_t*)coefC; uint64_t *d_in = (uint64_t*)coefD; uint64_t *e_in = (uint64_t*)coefE; uint64_t *out_0 = (uint64_t*)&data_out[0*data_count/4]; uint64_t *out_1 = (uint64_t*)&data_out[1*data_count/4]; uint64_t *out_2 = (uint64_t*)&data_out[2*data_count/4]; uint64_t *out_3 = (uint64_t*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { uint64_t x = x_in[4*i]; uint64_t y = y_in[4*i]; uint64_t z = z_in[4*i]; uint64_t w = w_in[4*i]; uint64_t c = c_in[i]; uint64_t d = d_in[i]; uint64_t e = e_in[i]; uint64_t conj_c = __builtin_e2k_pxord(c, 1LL<<63); uint64_t conj_d = __builtin_e2k_pxord(d, 1LL<<63); uint64_t conj_e = __builtin_e2k_pxord(e, 1LL<<63); uint64_t swap_c = __builtin_e2k_pshufb(0, c, 0x0302010007060504); uint64_t swap_d = __builtin_e2k_pshufb(0, d, 0x0302010007060504); uint64_t swap_e = __builtin_e2k_pshufb(0, e, 0x0302010007060504); uint64_t cy_real = __builtin_e2k_pfmuls(conj_c, y); uint64_t dz_real = __builtin_e2k_pfmuls(conj_d, z); uint64_t ew_real = __builtin_e2k_pfmuls(conj_e, w); uint64_t cy_imag = __builtin_e2k_pfmuls(swap_c, y); uint64_t dz_imag = __builtin_e2k_pfmuls(swap_d, z); uint64_t ew_imag = __builtin_e2k_pfmuls(swap_e, w); uint64_t cy = __builtin_e2k_pfhadds(cy_real, cy_imag); uint64_t dz = __builtin_e2k_pfhadds(dz_real, dz_imag); uint64_t ew = __builtin_e2k_pfhadds(ew_real, ew_imag); uint64_t add02 = __builtin_e2k_pfadds( x, dz); uint64_t sub02 = __builtin_e2k_pfsubs( x, dz); uint64_t add13 = __builtin_e2k_pfadds(cy, ew); uint64_t sub13 = __builtin_e2k_pfsubs(cy, ew); //uint64_t conj_sub13 = __builtin_e2k_pxord(sub13, 1LL<<63); //uint64_t sub13i = __builtin_e2k_pshufb(0, conj_sub13, 0x0302010007060504); uint64_t swap_sub13 = __builtin_e2k_pshufb(0, sub13, 0x0302010007060504); uint64_t sub13i = __builtin_e2k_pxord(swap_sub13, 1LL<<31); out_0[i] = __builtin_e2k_pfadds(add02, add13); out_1[i] = __builtin_e2k_pfsubs(sub02, sub13i); out_2[i] = __builtin_e2k_pfsubs(add02, add13); out_3[i] = __builtin_e2k_pfadds(sub02, sub13i); } }
Основной цикл на ассемблере
.L2675: { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=3, abs=8, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=4, abs=16, disp=0 } .L2317: { loop_mode pfadds,0,sm %b[70], %b[67], %b[73] pfadd_adds,1,sm %b[40], %b[47], %b[75], %b[1] pfadd_rsubs,2,sm %b[40], %b[47], %b[75], %b[0] pfsubs,3,sm %b[66], %b[63], %b[39] xord,4,sm %b[51], %r0, %b[58] xord,5,sm %b[33], %r0, %b[79] } { loop_mode pfmuls,0,sm %b[60], %b[21], %b[70] pfsub_rsubs,1,sm %b[40], %b[47], %b[81], %b[67] pfsub_adds,2,sm %b[40], %b[47], %b[81], %b[33] pshufb,3,sm 0x0, %b[18], %r8, %b[75] pshufb,4,sm 0x0, %b[57], %r8, %b[84] xord,5,sm %b[18], %r0, %b[82] } { loop_mode pfmuls,0,sm %b[79], %b[13], %b[85] pfmul_hadds,1,sm %b[78], %b[15], %b[87], %b[57] staad,2 %b[5], %aad4[ %aasti8 ] incr,2 %aaincr0 pfmul_hadds,3,sm %b[84], %b[25], %b[74], %b[60] pshufb,4,sm 0x0, %b[41], %r8, %b[81] staad,5 %b[4], %aad2[ %aasti6 ] incr,5 %aaincr0 movad,1 area=0, ind=0, am=1, be=0, %b[47] movad,2 area=0, ind=0, am=0, be=0, %b[18] movad,3 area=0, ind=16, am=0, be=0, %b[40] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmuls,0,sm %b[82], %b[54], %b[78] pfmul_hadds,1,sm %b[77], %b[56], %b[80], %b[41] staad,2 %b[71], %aad3[ %aasti7 ] incr,2 %aaincr0 pshufb,3,sm 0x0, %b[31], %r8, %b[74] xord,4,sm %b[83], %r7, %b[79] staad,5 %b[37], %aad1[ %aasti5 ] incr,5 %aaincr0 movad,0 area=2, ind=0, am=1, be=0, %b[25] movad,1 area=1, ind=0, am=1, be=0, %b[4] movad,2 area=0, ind=24, am=0, be=0, %b[5] movad,3 area=0, ind=8, am=1, be=0, %b[15] }
Теоретическая скорость: 4 комплексных числа за 4 такта (4/4) = 8 Байт/такт
Двойная теоретическая скорость: 16 Байт/такт
Замеры скорости
Видим ускорение.
4. stage_radix4_simd128
Вычисления делаем аналогично stage_radix2_simd128.
Код на Си
void stage_radix4_simd128(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *zw0_in = (__v2di*)&data_in[2]; __v2di *xy1_in = (__v2di*)&data_in[4]; __v2di *zw1_in = (__v2di*)&data_in[6]; __v2di *c_in = (__v2di*)coefC; __v2di *d_in = (__v2di*)coefD; __v2di *e_in = (__v2di*)coefE; __v2di *out_0 = (__v2di*)&data_out[0*data_count/4]; __v2di *out_1 = (__v2di*)&data_out[1*data_count/4]; __v2di *out_2 = (__v2di*)&data_out[2*data_count/4]; __v2di *out_3 = (__v2di*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/8; ++i) { __v2di xy0 = xy0_in[4*i]; __v2di zw0 = zw0_in[4*i]; __v2di xy1 = xy1_in[4*i]; __v2di zw1 = zw1_in[4*i]; __v2di c = c_in[i]; __v2di d = d_in[i]; __v2di e = e_in[i]; __v2di x = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di conj_c = __builtin_e2k_qpxor(c, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_d = __builtin_e2k_qpxor(d, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_e = __builtin_e2k_qpxor(e, (__v2di){1LL<<63, 1LL<<63}); __v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d = __builtin_e2k_qpshufb(d, d, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e = __builtin_e2k_qpshufb(e, e, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy_real = __builtin_e2k_qpfmuls(conj_c, y); __v2di dz_real = __builtin_e2k_qpfmuls(conj_d, z); __v2di ew_real = __builtin_e2k_qpfmuls(conj_e, w); __v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y); __v2di dz_imag = __builtin_e2k_qpfmuls(swap_d, z); __v2di ew_imag = __builtin_e2k_qpfmuls(swap_e, w); __v2di cy_rrii = __builtin_e2k_qpfhadds(cy_real, cy_imag); __v2di dz_rrii = __builtin_e2k_qpfhadds(dz_real, dz_imag); __v2di ew_rrii = __builtin_e2k_qpfhadds(ew_real, ew_imag); __v2di cy = __builtin_e2k_qpshufb(cy_rrii, cy_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di dz = __builtin_e2k_qpshufb(dz_rrii, dz_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di ew = __builtin_e2k_qpshufb(ew_rrii, ew_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di add02 = __builtin_e2k_qpfadds( x, dz); __v2di sub02 = __builtin_e2k_qpfsubs( x, dz); __v2di add13 = __builtin_e2k_qpfadds(cy, ew); __v2di sub13 = __builtin_e2k_qpfsubs(cy, ew); //__v2di conj_sub13 = __builtin_e2k_qpxor(sub13, (__v2di){1LL<<63, 1LL<<63}); //__v2di sub13i = __builtin_e2k_qpshufb(conj_sub13, conj_sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13 = __builtin_e2k_qpshufb(sub13, sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di sub13i = __builtin_e2k_qpxor(swap_sub13, (__v2di){1LL<<31, 1LL<<31}); out_0[i] = __builtin_e2k_qpfadds(add02, add13); out_1[i] = __builtin_e2k_qpfsubs(sub02, sub13i); out_2[i] = __builtin_e2k_qpfsubs(add02, add13); out_3[i] = __builtin_e2k_qpfadds(sub02, sub13i); } }
Основной цикл на ассемблере
.L3309: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=4, abs=16, disp=0 } .L2728: { loop_mode qpfadd_rsubs,0,sm %b[13], %b[38], %b[21], %b[0] qpshufb,1,sm %b[44], %b[44], %r9, %b[8] qpshufb,3,sm %b[16], %b[16], %r10, %b[25] qpshufb,4,sm %b[17], %b[17], %r10, %b[48] qpfmuls,5,sm %b[60], %b[49], %b[1] } { loop_mode qpfsub_rsubs,0,sm %b[13], %b[38], %b[58], %b[44] qpshufb,1,sm %b[36], %b[36], %r9, %b[61] qpshufb,3,sm %b[29], %b[52], %r12, %b[16] qpfsub_adds,4,sm %b[13], %b[38], %b[58], %b[21] qpfadds,5,sm %b[48], %b[25], %b[17] } { loop_mode qpfmul_hadds,0,sm %b[10], %b[51], %b[3], %b[13] qpshufb,1,sm %b[56], %b[56], %r10, %b[36] qpxor,3,sm %b[34], %r0, %b[38] qpshufb,4,sm %b[59], %b[59], %r9, %b[52] qpfmuls,5,sm %b[57], %b[45], %b[29] } { loop_mode qpfmul_hadds,0,sm %b[55], %b[47], %b[31], %b[10] qpshufb,1,sm %b[37], %b[43], %r12, %b[3] staaqp,2 %b[26], %aad4[ %aasti8 ] incr,2 %aaincr0 qpxor,4,sm %b[52], %r7, %b[56] qpfmuls,5,sm %b[38], %b[20], %b[51] } { loop_mode qpfmul_hadds,0,sm %b[61], %b[22], %b[53], %b[52] qpxor,1,sm %b[4], %r0, %b[55] staaqp,2 %b[2], %aad2[ %aasti6 ] incr,2 %aaincr0 qpshufb,3,sm %b[35], %b[41], %r11, %b[47] qpshufb,4,sm %b[27], %b[50], %r11, %b[43] qpfsubs,5,sm %b[48], %b[25], %b[57] movaqp,0 area=1, ind=0, am=1, be=0, %b[38] movaqp,1 area=0, ind=0, am=0, be=0, %b[37] movaqp,2 area=1, ind=0, am=1, be=0, %b[26] movaqp,3 area=0, ind=0, am=0, be=0, %b[31] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfadd_adds,0,sm %b[11], %b[36], %b[19], %b[22] qpshufb,1,sm %b[6], %b[6], %r9, %b[53] staaqp,2 %b[46], %aad3[ %aasti7 ] incr,2 %aaincr0 qpxor,3,sm %b[42], %r0, %b[58] staaqp,5 %b[23], %aad1[ %aasti5 ] incr,5 %aaincr0 movaqp,0 area=2, ind=0, am=1, be=0, %b[2] movaqp,1 area=0, ind=16, am=1, be=0, %b[48] movaqp,3 area=0, ind=16, am=1, be=0, %b[25] }
Теоретическая скорость: 8 комплексных чисел за 6 тактов (8/6) = 10.67 Байт/такт
Двойная теоретическая скорость: 21.33 Байт/такт
Замеры скорости
Видим ускорение.
5. stage_radix4_simd128_noConj
Уменьшаем число инструкций аналогично stage_radix2_simd128_noConj.
Код на Си
void stage_radix4_simd128_noConj(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *zw0_in = (__v2di*)&data_in[2]; __v2di *xy1_in = (__v2di*)&data_in[4]; __v2di *zw1_in = (__v2di*)&data_in[6]; __v2di *c_in = (__v2di*)coefC; __v2di *d_in = (__v2di*)coefD; __v2di *e_in = (__v2di*)coefE; __v2di *out_0 = (__v2di*)&data_out[0*data_count/4]; __v2di *out_1 = (__v2di*)&data_out[1*data_count/4]; __v2di *out_2 = (__v2di*)&data_out[2*data_count/4]; __v2di *out_3 = (__v2di*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/8; ++i) { __v2di xy0 = xy0_in[4*i]; __v2di zw0 = zw0_in[4*i]; __v2di xy1 = xy1_in[4*i]; __v2di zw1 = zw1_in[4*i]; __v2di c = c_in[i]; __v2di d = d_in[i]; __v2di e = e_in[i]; __v2di x = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d = __builtin_e2k_qpshufb(d, d, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e = __builtin_e2k_qpshufb(e, e, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy_real = __builtin_e2k_qpfmuls( c, y); __v2di dz_real = __builtin_e2k_qpfmuls( d, z); __v2di ew_real = __builtin_e2k_qpfmuls( e, w); __v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y); __v2di dz_imag = __builtin_e2k_qpfmuls(swap_d, z); __v2di ew_imag = __builtin_e2k_qpfmuls(swap_e, w); __v2di cy_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy_real); __v2di dz_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz_real); __v2di ew_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew_real); __v2di cy_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy_imag); __v2di dz_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz_imag); __v2di ew_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew_imag); __v2di cy = __builtin_e2k_qppermb(cy_ii, cy_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz = __builtin_e2k_qppermb(dz_ii, dz_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew = __builtin_e2k_qppermb(ew_ii, ew_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di add02 = __builtin_e2k_qpfadds( x, dz); __v2di sub02 = __builtin_e2k_qpfsubs( x, dz); __v2di add13 = __builtin_e2k_qpfadds(cy, ew); __v2di sub13 = __builtin_e2k_qpfsubs(cy, ew); //__v2di conj_sub13 = __builtin_e2k_qpxor(sub13, (__v2di){1LL<<63, 1LL<<63}); //__v2di sub13i = __builtin_e2k_qpshufb(conj_sub13, conj_sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13 = __builtin_e2k_qpshufb(sub13, sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di sub13i = __builtin_e2k_qpxor(swap_sub13, (__v2di){1LL<<31, 1LL<<31}); out_0[i] = __builtin_e2k_qpfadds(add02, add13); out_1[i] = __builtin_e2k_qpfsubs(sub02, sub13i); out_2[i] = __builtin_e2k_qpfsubs(add02, add13); out_3[i] = __builtin_e2k_qpfadds(sub02, sub13i); } }
Основной цикл на ассемблере
.L3939: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=4, abs=16, disp=0 } .L3362: { loop_mode qpfmul_hsubs,0,sm %b[61], %b[63], %r12, %b[1] qpfmul_hadds,2,sm %b[64], %b[63], %r12, %b[0] qpshufb,3,sm %b[8], %b[9], %r9, %b[40] qpshufb,4,sm %b[19], %b[19], %r11, %b[31] qpfadds,5,sm %b[37], %b[30], %b[26] movaqp,0 area=1, ind=0, am=1, be=0, %b[17] movaqp,1 area=0, ind=0, am=0, be=0, %b[7] movaqp,3 area=0, ind=0, am=0, be=0, %b[6] } { loop_mode qpfmul_hsubs,0,sm %b[21], %b[42], %r12, %b[44] qpfmul_hadds,2,sm %b[33], %b[42], %r12, %b[41] qpshufb,3,sm %b[49], %b[52], %r9, %b[61] qpshufb,4,sm %b[59], %b[59], %r11, %b[62] qpfsubs,5,sm %b[37], %b[30], %b[58] movaqp,0 area=2, ind=0, am=1, be=0, %b[57] movaqp,1 area=0, ind=16, am=1, be=0, %b[50] movaqp,3 area=0, ind=16, am=1, be=0, %b[47] } { loop_mode qpfmul_hsubs,0,sm %b[16], %b[39], %r12, %b[21] qpshufb,1,sm %b[60], %b[60], %r11, %b[42] qpfadd_adds,2,sm %b[24], %b[56], %b[28], %b[30] qpshufb,3,sm %b[14], %b[14], %r11, %b[33] qpshufb,4,sm %b[51], %b[54], %r10, %b[37] staaqp,5 %b[34], %aad4[ %aasti8 ] incr,5 %aaincr0 } { loop_mode qpfmul_hadds,0,sm %b[35], %b[39], %r12, %b[34] qpxor,1,sm %b[42], %r7, %b[60] qpfadd_rsubs,2,sm %b[24], %b[56], %b[28], %b[51] qpshufb,3,sm %b[10], %b[11], %r10, %b[16] qppermb,4,sm %b[38], %b[25], %r0, %b[54] staaqp,5 %b[55], %aad2[ %aasti6 ] incr,5 %aaincr0 } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfsub_rsubs,0,sm %b[24], %b[56], %b[60], %b[25] qpfsub_adds,1,sm %b[24], %b[56], %b[60], %b[11] staaqp,2 %b[29], %aad3[ %aasti7 ] incr,2 %aaincr0 qppermb,3,sm %b[4], %b[5], %r0, %b[28] qppermb,4,sm %b[45], %b[48], %r0, %b[35] staaqp,5 %b[15], %aad1[ %aasti5 ] incr,5 %aaincr0 movaqp,3 area=1, ind=0, am=1, be=0, %b[10] }
Теоретическая скорость: 8 комплексных чисел за 5 тактов (8/5) = 12.8 Байт/такт
Двойная теоретическая скорость: 25.6 Байт/такт
Замеры скорости
Видим ускорение.
6. stage_radix4_simd128_noConj_unroll3
Здесь происходит раскрутка цикла в 3 раза с помощью опции unroll.
Код на Си
void stage_radix4_simd128_noConj_unroll3(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *zw0_in = (__v2di*)&data_in[2]; __v2di *xy1_in = (__v2di*)&data_in[4]; __v2di *zw1_in = (__v2di*)&data_in[6]; __v2di *c_in = (__v2di*)coefC; __v2di *d_in = (__v2di*)coefD; __v2di *e_in = (__v2di*)coefE; __v2di *out_0 = (__v2di*)&data_out[0*data_count/4]; __v2di *out_1 = (__v2di*)&data_out[1*data_count/4]; __v2di *out_2 = (__v2di*)&data_out[2*data_count/4]; __v2di *out_3 = (__v2di*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(3) #pragma prefetch for(int64_t i = 0; i < data_count/8; ++i) { __v2di xy0 = xy0_in[4*i]; __v2di zw0 = zw0_in[4*i]; __v2di xy1 = xy1_in[4*i]; __v2di zw1 = zw1_in[4*i]; __v2di c = c_in[i]; __v2di d = d_in[i]; __v2di e = e_in[i]; __v2di x = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d = __builtin_e2k_qpshufb(d, d, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e = __builtin_e2k_qpshufb(e, e, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy_real = __builtin_e2k_qpfmuls( c, y); __v2di dz_real = __builtin_e2k_qpfmuls( d, z); __v2di ew_real = __builtin_e2k_qpfmuls( e, w); __v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y); __v2di dz_imag = __builtin_e2k_qpfmuls(swap_d, z); __v2di ew_imag = __builtin_e2k_qpfmuls(swap_e, w); __v2di cy_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy_real); __v2di dz_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz_real); __v2di ew_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew_real); __v2di cy_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy_imag); __v2di dz_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz_imag); __v2di ew_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew_imag); __v2di cy = __builtin_e2k_qppermb(cy_ii, cy_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz = __builtin_e2k_qppermb(dz_ii, dz_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew = __builtin_e2k_qppermb(ew_ii, ew_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di add02 = __builtin_e2k_qpfadds( x, dz); __v2di sub02 = __builtin_e2k_qpfsubs( x, dz); __v2di add13 = __builtin_e2k_qpfadds(cy, ew); __v2di sub13 = __builtin_e2k_qpfsubs(cy, ew); //__v2di conj_sub13 = __builtin_e2k_qpxor(sub13, (__v2di){1LL<<63, 1LL<<63}); //__v2di sub13i = __builtin_e2k_qpshufb(conj_sub13, conj_sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13 = __builtin_e2k_qpshufb(sub13, sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di sub13i = __builtin_e2k_qpxor(swap_sub13, (__v2di){1LL<<31, 1LL<<31}); out_0[i] = __builtin_e2k_qpfadds(add02, add13); out_1[i] = __builtin_e2k_qpfsubs(sub02, sub13i); out_2[i] = __builtin_e2k_qpfsubs(add02, add13); out_3[i] = __builtin_e2k_qpfadds(sub02, sub13i); } }
Основной цикл на ассемблере
.L5038: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=128 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=160 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=2, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=4, asz=3, abs=16, disp=32 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=3, asz=3, abs=24, disp=32 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=2, asz=3, abs=24, disp=32 } .L3992: { loop_mode qpfsub_adds,0,sm %b[106], %b[103], %g16, %g17 qpfmul_hsubs,1,sm %b[10], %b[97], %r9, %b[0] qpfadds,2,sm %g18, %g19, %g20 qpshufb,3,sm %b[82], %b[81], %r0, %g22 qpshufb,4,sm %b[82], %b[81], %r5, %g21 qpfmul_hadds,5,sm %g23, %g24, %r9, %b[1] } { loop_mode qpfsub_rsubs,0,sm %g25, %g26, %g27, %b[117] qppermb,1,sm %g28, %b[117], %r1, %g29 qpfadds,2,sm %b[105], %b[108], %b[108] qpshufb,3,sm %b[54], %b[54], %r3, %g31 qpshufb,4,sm %b[13], %b[13], %r3, %g30 qpfmul_hsubs,5,sm %b[13], %g21, %r9, %b[105] } { loop_mode qpfsub_adds,0,sm %b[116], %b[109], %b[113], %r2 qpfmul_hadds,1,sm %r11, %b[43], %r9, %b[3] qpfsubs,2,sm %g29, %b[110], %r7 qppermb,3,sm %b[3], %b[64], %r1, %g19 qppermb,4,sm %b[118], %r10, %r1, %g18 qpfmul_hadds,5,sm %g30, %g21, %r9, %g21 } { loop_mode qpfsub_rsubs,0,sm %b[106], %b[103], %g16, %b[110] qpfmul_hsubs,1,sm %b[38], %b[114], %r9, %b[13] qpfadds,2,sm %g29, %b[110], %b[113] qppermb,3,sm %b[27], %b[16], %r1, %b[106] qppermb,4,sm %b[21], %b[17], %r1, %b[103] qpfsub_rsubs,5,sm %b[116], %b[109], %b[113], %b[109] } { loop_mode qpfmul_hsubs,0,sm %b[54], %g22, %r9, %r12 qpfmul_hsubs,1,sm %b[57], %b[41], %r9, %b[16] staaqp,2 %b[101], %aad2[ %aasti6 + _f32s,_lts0 0x10 ] qpshufb,3,sm %b[59], %b[42], %r5, %b[102] qppermb,4,sm %b[102], %r12, %r1, %b[101] qpfmul_hadds,5,sm %b[104], %b[114], %r9, %b[17] } { loop_mode qpfmul_hadds,0,sm %g31, %g22, %r9, %b[100] qpshufb,1,sm %b[74], %b[73], %r0, %b[104] staaqp,2 %b[115], %aad4[ %aasti8 + _f32s,_lts0 0x10 ] qpshufb,3,sm %b[100], %b[100], %r3, %b[115] qpshufb,4,sm %b[10], %b[10], %r3, %b[114] qpfmul_hsubs,5,sm %b[100], %b[102], %r9, %b[10] } { loop_mode qpfmul_hsubs,0,sm %b[51], %b[112], %r9, %b[115] qpfsubs,1,sm %g18, %g19, %b[27] staaqp,2 %r13, %aad2[ %aasti6 + _f32s,_lts0 0x20 ] qpshufb,3,sm %b[36], %b[36], %r3, %b[102] qpshufb,4,sm %b[88], %b[87], %r5, %b[116] qpfmul_hadds,5,sm %b[115], %b[102], %r9, %b[21] } { loop_mode qpfmul_hadds,0,sm %r14, %b[112], %r9, %g28 qpfsubs,1,sm %b[103], %b[106], %b[38] staaqp,2 %b[119], %aad2[ %aasti6 ] incr,2 %aaincr3 qpshufb,3,sm %b[31], %b[26], %r5, %b[112] qpshufb,4,sm %b[60], %b[60], %r3, %g22 qpfmul_hsubs,5,sm %b[60], %b[116], %r9, %r10 } { loop_mode qpfadd_rsubs,0,sm %b[104], %b[101], %b[113], %b[99] qppermb,1,sm %b[5], %b[20], %r1, %b[107] staaqp,2 %b[107], %aad4[ %aasti8 + _f32s,_lts0 0x20 ] qppermb,3,sm %b[99], %b[2], %r1, %g26 qpshufb,4,sm %b[39], %b[34], %r0, %g25 qpfmul_hadds,5,sm %g22, %b[116], %r9, %b[116] movaqp,1 area=5, ind=0, am=1, be=0, %b[2] } { loop_mode qpfmul_hadds,0,sm %b[114], %b[97], %r9, %b[97] qpshufb,1,sm %b[92], %b[91], %r0, %b[114] staaqp,2 %b[98], %aad4[ %aasti8 ] incr,2 %aaincr3 qpshufb,3,sm %b[96], %b[95], %r0, %b[39] qpshufb,4,sm %b[24], %b[24], %r3, %g23 qpfadd_adds,5,sm %b[104], %b[101], %b[113], %b[113] movaqp,0 area=4, ind=16, am=1, be=0, %b[5] movaqp,1 area=4, ind=0, am=0, be=0, %b[20] movaqp,2 area=5, ind=0, am=1, be=0, %b[98] movaqp,3 area=4, ind=0, am=1, be=0, %b[34] } { loop_mode qpfadd_adds,0,sm %b[114], %b[107], %g20, %b[96] qpshufb,1,sm %r7, %r7, %r3, %g17 staaqp,2 %g17, %aad1[ %aasti5 + _f32s,_lts0 0x10 ] qpshufb,3,sm %b[96], %b[95], %r5, %g24 qpshufb,4,sm %b[63], %b[46], %r0, %b[95] qpfadd_rsubs,5,sm %g25, %g26, %b[108], %r13 movaqp,0 area=3, ind=16, am=1, be=0, %b[43] movaqp,1 area=3, ind=0, am=0, be=0, %b[54] movaqp,2 area=3, ind=16, am=1, be=0, %b[46] movaqp,3 area=3, ind=0, am=0, be=0, %b[51] } { loop_mode qpfadd_rsubs,0,sm %b[114], %b[107], %g20, %b[117] qpshufb,1,sm %b[40], %b[40], %r3, %g22 staaqp,2 %b[117], %aad3[ %aasti7 + _f32s,_lts0 0x20 ] qpshufb,3,sm %b[57], %b[57], %r3, %r11 qpshufb,4,sm %b[29], %b[29], %r3, %g29 qpfmul_hsubs,5,sm %b[24], %g24, %r9, %b[60] movaqp,0 area=2, ind=0, am=0, be=0, %b[24] movaqp,1 area=2, ind=16, am=1, be=0, %b[40] movaqp,2 area=2, ind=0, am=0, be=0, %b[29] movaqp,3 area=2, ind=16, am=1, be=0, %b[57] } { loop_mode qpfadd_adds,0,sm %g25, %g26, %b[108], %b[105] qpxor,1,sm %g22, %r4, %g27 staaqp,2 %b[111], %aad1[ %aasti5 + _f32s,_lts0 0x20 ] qppermb,3,sm %g21, %b[105], %r1, %b[108] qpxor,4,sm %g29, %r4, %b[111] staaqp,5 %r2, %aad1[ %aasti5 ] incr,5 %aaincr3 movaqp,0 area=1, ind=16, am=1, be=0, %b[73] movaqp,1 area=1, ind=0, am=0, be=0, %b[63] movaqp,2 area=1, ind=16, am=1, be=0, %b[74] movaqp,3 area=1, ind=0, am=0, be=0, %b[64] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfsub_adds,0,sm %g25, %g26, %g27, %b[109] qpxor,1,sm %g17, %r4, %g16 staaqp,2 %b[110], %aad3[ %aasti7 + _f32s,_lts0 0x10 ] qpshufb,3,sm %b[49], %b[49], %r3, %r14 qpshufb,4,sm %b[70], %b[69], %r5, %b[110] staaqp,5 %b[109], %aad3[ %aasti7 ] incr,5 %aaincr3 movaqp,0 area=0, ind=0, am=0, be=0, %b[81] movaqp,1 area=0, ind=16, am=1, be=0, %b[91] movaqp,2 area=0, ind=0, am=0, be=0, %b[82] movaqp,3 area=0, ind=16, am=1, be=0, %b[92] }
Теоретическая скорость: 24 комплексных числа за 14 тактов (24/14) = 13.71 Байт/такт
Двойная теоретическая скорость: 27.43 Байт/такт
Замеры скорости
Видим ускорение в середине графика, но замедление в начале и в конце графика.
Итоги по stage_radix4
График FFT находится здесь.
stage_radix4_2x
Схема алгоритма Stage для версии «radix-4» 2x.
Один проход по stage_radix4_2x совершает ту же работу, что 2 прохода по stage_radix4. А один проход по stage_radix4 совершает ту же работу, что 2 прохода по stage_radix2. Поэтому скорость stage_radix4_2x будем умножать на 4 для удобства сравнения с stage_radix2 (этот факт подписан на оси графика и в выводе консоли).
1. stage_radix4_2x_etalon
Здесь происходит ручная раскрутка алгоритма stage_radix4_etalon в 2 раза.
Код на Си
void stage_radix4_2x_etalon(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coefC_a, myComplex *coefD_a, myComplex *coefE_a, myComplex *coefC_b, myComplex *coefD_b, myComplex *coefE_b) { myComplex *x0_in = &data_in[ 0]; myComplex *y0_in = &data_in[ 1]; myComplex *z0_in = &data_in[ 2]; myComplex *w0_in = &data_in[ 3]; myComplex *x1_in = &data_in[ 4]; myComplex *y1_in = &data_in[ 5]; myComplex *z1_in = &data_in[ 6]; myComplex *w1_in = &data_in[ 7]; myComplex *x2_in = &data_in[ 8]; myComplex *y2_in = &data_in[ 9]; myComplex *z2_in = &data_in[10]; myComplex *w2_in = &data_in[11]; myComplex *x3_in = &data_in[12]; myComplex *y3_in = &data_in[13]; myComplex *z3_in = &data_in[14]; myComplex *w3_in = &data_in[15]; myComplex *c0a_in = &coefC_a[0]; myComplex *c1a_in = &coefC_a[1]; myComplex *c2a_in = &coefC_a[2]; myComplex *c3a_in = &coefC_a[3]; myComplex *d0a_in = &coefD_a[0]; myComplex *d1a_in = &coefD_a[1]; myComplex *d2a_in = &coefD_a[2]; myComplex *d3a_in = &coefD_a[3]; myComplex *e0a_in = &coefE_a[0]; myComplex *e1a_in = &coefE_a[1]; myComplex *e2a_in = &coefE_a[2]; myComplex *e3a_in = &coefE_a[3]; myComplex *c0b_in = &coefC_b[0*data_count/16]; myComplex *c1b_in = &coefC_b[1*data_count/16]; myComplex *c2b_in = &coefC_b[2*data_count/16]; myComplex *c3b_in = &coefC_b[3*data_count/16]; myComplex *d0b_in = &coefD_b[0*data_count/16]; myComplex *d1b_in = &coefD_b[1*data_count/16]; myComplex *d2b_in = &coefD_b[2*data_count/16]; myComplex *d3b_in = &coefD_b[3*data_count/16]; myComplex *e0b_in = &coefE_b[0*data_count/16]; myComplex *e1b_in = &coefE_b[1*data_count/16]; myComplex *e2b_in = &coefE_b[2*data_count/16]; myComplex *e3b_in = &coefE_b[3*data_count/16]; myComplex *out_0 = &data_out[ 0*data_count/16]; myComplex *out_1 = &data_out[ 1*data_count/16]; myComplex *out_2 = &data_out[ 2*data_count/16]; myComplex *out_3 = &data_out[ 3*data_count/16]; myComplex *out_4 = &data_out[ 4*data_count/16]; myComplex *out_5 = &data_out[ 5*data_count/16]; myComplex *out_6 = &data_out[ 6*data_count/16]; myComplex *out_7 = &data_out[ 7*data_count/16]; myComplex *out_8 = &data_out[ 8*data_count/16]; myComplex *out_9 = &data_out[ 9*data_count/16]; myComplex *out_10 = &data_out[10*data_count/16]; myComplex *out_11 = &data_out[11*data_count/16]; myComplex *out_12 = &data_out[12*data_count/16]; myComplex *out_13 = &data_out[13*data_count/16]; myComplex *out_14 = &data_out[14*data_count/16]; myComplex *out_15 = &data_out[15*data_count/16]; #pragma ivdep #pragma unroll(1) // #pragma prefetch for(int64_t i = 0; i < data_count/16; ++i) { myComplex x0 = x0_in[16*i]; myComplex y0 = y0_in[16*i]; myComplex z0 = z0_in[16*i]; myComplex w0 = w0_in[16*i]; myComplex c0 = c0a_in[4*i]; myComplex d0 = d0a_in[4*i]; myComplex e0 = e0a_in[4*i]; myComplex x1 = x1_in[16*i]; myComplex y1 = y1_in[16*i]; myComplex z1 = z1_in[16*i]; myComplex w1 = w1_in[16*i]; myComplex c1 = c1a_in[4*i]; myComplex d1 = d1a_in[4*i]; myComplex e1 = e1a_in[4*i]; myComplex x2 = x2_in[16*i]; myComplex y2 = y2_in[16*i]; myComplex z2 = z2_in[16*i]; myComplex w2 = w2_in[16*i]; myComplex c2 = c2a_in[4*i]; myComplex d2 = d2a_in[4*i]; myComplex e2 = e2a_in[4*i]; myComplex x3 = x3_in[16*i]; myComplex y3 = y3_in[16*i]; myComplex z3 = z3_in[16*i]; myComplex w3 = w3_in[16*i]; myComplex c3 = c3a_in[4*i]; myComplex d3 = d3a_in[4*i]; myComplex e3 = e3a_in[4*i]; myComplex cy0 = complex_mul(c0, y0); myComplex cy1 = complex_mul(c1, y1); myComplex cy2 = complex_mul(c2, y2); myComplex cy3 = complex_mul(c3, y3); myComplex dz0 = complex_mul(d0, z0); myComplex dz1 = complex_mul(d1, z1); myComplex dz2 = complex_mul(d2, z2); myComplex dz3 = complex_mul(d3, z3); myComplex ew0 = complex_mul(e0, w0); myComplex ew1 = complex_mul(e1, w1); myComplex ew2 = complex_mul(e2, w2); myComplex ew3 = complex_mul(e3, w3); myComplex add02_0 = complex_add( x0, dz0); myComplex add02_1 = complex_add( x1, dz1); myComplex add02_2 = complex_add( x2, dz2); myComplex add02_3 = complex_add( x3, dz3); myComplex sub02_0 = complex_sub( x0, dz0); myComplex sub02_1 = complex_sub( x1, dz1); myComplex sub02_2 = complex_sub( x2, dz2); myComplex sub02_3 = complex_sub( x3, dz3); myComplex add13_0 = complex_add(cy0, ew0); myComplex add13_1 = complex_add(cy1, ew1); myComplex add13_2 = complex_add(cy2, ew2); myComplex add13_3 = complex_add(cy3, ew3); myComplex sub13_0 = complex_sub(cy0, ew0); myComplex sub13_1 = complex_sub(cy1, ew1); myComplex sub13_2 = complex_sub(cy2, ew2); myComplex sub13_3 = complex_sub(cy3, ew3); myComplex sub13i_0 = (myComplex){.real = -sub13_0.imag, .imag = sub13_0.real}; myComplex sub13i_1 = (myComplex){.real = -sub13_1.imag, .imag = sub13_1.real}; myComplex sub13i_2 = (myComplex){.real = -sub13_2.imag, .imag = sub13_2.real}; myComplex sub13i_3 = (myComplex){.real = -sub13_3.imag, .imag = sub13_3.real}; myComplex out0 = complex_add(add02_0, add13_0); myComplex out1 = complex_add(add02_1, add13_1); myComplex out2 = complex_add(add02_2, add13_2); myComplex out3 = complex_add(add02_3, add13_3); myComplex out4 = complex_sub(sub02_0, sub13i_0); myComplex out5 = complex_sub(sub02_1, sub13i_1); myComplex out6 = complex_sub(sub02_2, sub13i_2); myComplex out7 = complex_sub(sub02_3, sub13i_3); myComplex out8 = complex_sub(add02_0, add13_0); myComplex out9 = complex_sub(add02_1, add13_1); myComplex out10 = complex_sub(add02_2, add13_2); myComplex out11 = complex_sub(add02_3, add13_3); myComplex out12 = complex_add(sub02_0, sub13i_0); myComplex out13 = complex_add(sub02_1, sub13i_1); myComplex out14 = complex_add(sub02_2, sub13i_2); myComplex out15 = complex_add(sub02_3, sub13i_3); x0 = out0; y0 = out1; z0 = out2; w0 = out3; c0 = c0b_in[i]; d0 = d0b_in[i]; e0 = e0b_in[i]; x1 = out4; y1 = out5; z1 = out6; w1 = out7; c1 = c1b_in[i]; d1 = d1b_in[i]; e1 = e1b_in[i]; x2 = out8; y2 = out9; z2 = out10; w2 = out11; c2 = c2b_in[i]; d2 = d2b_in[i]; e2 = e2b_in[i]; x3 = out12; y3 = out13; z3 = out14; w3 = out15; c3 = c3b_in[i]; d3 = d3b_in[i]; e3 = e3b_in[i]; cy0 = complex_mul(c0, y0); cy1 = complex_mul(c1, y1); cy2 = complex_mul(c2, y2); cy3 = complex_mul(c3, y3); dz0 = complex_mul(d0, z0); dz1 = complex_mul(d1, z1); dz2 = complex_mul(d2, z2); dz3 = complex_mul(d3, z3); ew0 = complex_mul(e0, w0); ew1 = complex_mul(e1, w1); ew2 = complex_mul(e2, w2); ew3 = complex_mul(e3, w3); add02_0 = complex_add( x0, dz0); add02_1 = complex_add( x1, dz1); add02_2 = complex_add( x2, dz2); add02_3 = complex_add( x3, dz3); sub02_0 = complex_sub( x0, dz0); sub02_1 = complex_sub( x1, dz1); sub02_2 = complex_sub( x2, dz2); sub02_3 = complex_sub( x3, dz3); add13_0 = complex_add(cy0, ew0); add13_1 = complex_add(cy1, ew1); add13_2 = complex_add(cy2, ew2); add13_3 = complex_add(cy3, ew3); sub13_0 = complex_sub(cy0, ew0); sub13_1 = complex_sub(cy1, ew1); sub13_2 = complex_sub(cy2, ew2); sub13_3 = complex_sub(cy3, ew3); sub13i_0 = (myComplex){.real = -sub13_0.imag, .imag = sub13_0.real}; sub13i_1 = (myComplex){.real = -sub13_1.imag, .imag = sub13_1.real}; sub13i_2 = (myComplex){.real = -sub13_2.imag, .imag = sub13_2.real}; sub13i_3 = (myComplex){.real = -sub13_3.imag, .imag = sub13_3.real}; out_0[i] = complex_add(add02_0, add13_0); out_1[i] = complex_add(add02_1, add13_1); out_2[i] = complex_add(add02_2, add13_2); out_3[i] = complex_add(add02_3, add13_3); out_4[i] = complex_sub(sub02_0, sub13i_0); out_5[i] = complex_sub(sub02_1, sub13i_1); out_6[i] = complex_sub(sub02_2, sub13i_2); out_7[i] = complex_sub(sub02_3, sub13i_3); out_8[i] = complex_sub(add02_0, add13_0); out_9[i] = complex_sub(add02_1, add13_1); out_10[i] = complex_sub(add02_2, add13_2); out_11[i] = complex_sub(add02_3, add13_3); out_12[i] = complex_add(sub02_0, sub13i_0); out_13[i] = complex_add(sub02_1, sub13i_1); out_14[i] = complex_add(sub02_2, sub13i_2); out_15[i] = complex_add(sub02_3, sub13i_3); } }
Основной цикл на ассемблере
.L1379: { fapb ct=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=16, d=0, incr=3, ind=2, asz=1, abs=0, disp=16 } { fapb ct=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=1, abs=2, disp=64 fapb dpl=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=1, abs=2, disp=32 } { fapb ct=0, dcd=0, fmt=3, mrng=0, d=0, incr=3, ind=4, asz=1, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=1, abs=4, disp=96 } { fapb ct=0, dcd=0, fmt=3, mrng=16, d=0, incr=3, ind=2, asz=1, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=0, d=0, incr=3, ind=3, asz=1, abs=6, disp=0 } { fapb ct=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=15, asz=2, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=8, d=1, incr=2, ind=0, asz=2, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=13, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=14, asz=2, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=11, asz=2, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=12, asz=2, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=9, asz=2, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=10, asz=2, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=7, asz=2, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=8, asz=2, abs=24, disp=0 } { fapb ct=1, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=5, asz=2, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=6, asz=2, abs=28, disp=0 } .L285: { loop_mode disp %ctpr1, .L285 movaw,0 area=0, ind=24, am=0, be=0, %g17 movaw,1 area=0, ind=28, am=0, be=0, %g16 movaw,2 area=0, ind=8, am=0, be=0, %g19 movaw,3 area=0, ind=12, am=0, be=0, %g18 } { loop_mode movaw,0 area=0, ind=16, am=0, be=0, %g21 movaw,1 area=0, ind=20, am=0, be=0, %g20 movaw,2 area=0, ind=0, am=1, be=0, %g23 movaw,3 area=0, ind=4, am=0, be=0, %g22 } { loop_mode movaw,0 area=0, ind=8, am=0, be=0, %g25 movaw,1 area=0, ind=12, am=0, be=0, %g24 movaw,2 area=1, ind=24, am=0, be=0, %g27 movaw,3 area=1, ind=28, am=0, be=0, %g26 } { loop_mode movaw,0 area=0, ind=0, am=1, be=0, %g29 movaw,1 area=0, ind=4, am=0, be=0, %g28 movaw,2 area=1, ind=16, am=0, be=0, %g31 movaw,3 area=1, ind=20, am=0, be=0, %g30 } { loop_mode movaw,0 area=1, ind=24, am=0, be=0, %r3 movaw,1 area=1, ind=28, am=0, be=0, %r1 movaw,2 area=1, ind=8, am=0, be=0, %r5 movaw,3 area=1, ind=12, am=0, be=0, %r4 } { loop_mode movaw,0 area=1, ind=16, am=0, be=0, %r9 movaw,1 area=1, ind=20, am=0, be=0, %r7 movaw,2 area=1, ind=0, am=1, be=0, %r42 movaw,3 area=1, ind=4, am=0, be=0, %r41 } { loop_mode movaw,0 area=1, ind=8, am=0, be=0, %r44 movaw,1 area=1, ind=12, am=0, be=0, %r43 movaw,2 area=2, ind=24, am=0, be=0, %r46 movaw,3 area=2, ind=28, am=0, be=0, %r45 } { loop_mode movaw,0 area=1, ind=0, am=1, be=0, %r48 movaw,1 area=1, ind=4, am=0, be=0, %r47 movaw,2 area=2, ind=16, am=0, be=0, %r50 movaw,3 area=2, ind=20, am=0, be=0, %r49 } { loop_mode movaw,0 area=2, ind=24, am=0, be=0, %r52 movaw,1 area=2, ind=28, am=0, be=0, %r51 movaw,2 area=2, ind=8, am=0, be=0, %r54 movaw,3 area=2, ind=12, am=0, be=0, %r53 } { loop_mode fmuls,0 %g23, %r3, %r57 fmuls,1 %g22, %r1, %r58 fmuls,2 %g23, %r1, %g23 fmuls,3 %g22, %r3, %g22 movaw,0 area=2, ind=16, am=0, be=0, %r56 movaw,1 area=2, ind=20, am=0, be=0, %r55 movaw,2 area=2, ind=0, am=1, be=0, %r3 movaw,3 area=2, ind=4, am=0, be=0, %r1 } { loop_mode movaw,0 area=2, ind=8, am=0, be=0, %r60 movaw,1 area=2, ind=12, am=0, be=0, %r59 movaw,2 area=3, ind=24, am=0, be=0, %r62 movaw,3 area=3, ind=28, am=0, be=0, %r61 } { loop_mode fmuls,0 %g19, %r46, %r63 fmuls,1 %g18, %r45, %b[0] fmuls,2 %g19, %r45, %g19 fmuls,3 %g18, %r46, %g18 movaw,0 area=2, ind=0, am=1, be=0, %r46 movaw,1 area=2, ind=4, am=0, be=0, %r45 movaw,2 area=3, ind=16, am=0, be=0, %b[2] movaw,3 area=3, ind=20, am=0, be=0, %b[1] } { loop_mode movaw,0 area=3, ind=8, am=0, be=0, %b[4] movaw,1 area=3, ind=12, am=0, be=0, %b[3] movaw,2 area=3, ind=8, am=0, be=0, %b[6] movaw,3 area=3, ind=12, am=0, be=0, %b[5] } { loop_mode fmuls,0 %r52, %r54, %b[7] fmuls,1 %r51, %r53, %b[8] fmuls,2 %r52, %r53, %r52 fmuls,3 %r51, %r54, %r51 movaw,0 area=3, ind=0, am=1, be=0, %r54 movaw,1 area=3, ind=4, am=0, be=0, %r53 movaw,2 area=3, ind=0, am=1, be=0, %b[10] movaw,3 area=3, ind=4, am=0, be=0, %b[9] } { loop_mode fmuls,0 %r56, %r44, %b[11] fmuls,1 %r55, %r43, %b[12] fmuls,2 %r56, %r43, %r43 fmuls,3 %r55, %r44, %r44 movaw,0 area=4, ind=0, am=1, be=0, %r56 movaw,1 area=4, ind=4, am=0, be=0, %r55 movaw,2 area=4, ind=0, am=1, be=0, %b[14] movaw,3 area=4, ind=4, am=0, be=0, %b[13] } { loop_mode fmuls,0 %r60, %r5, %b[15] fmuls,1 %r59, %r4, %b[16] fmuls,2 %r60, %r4, %r4 fmuls,3 %r59, %r5, %r5 fmuls,4 %r61, %r49, %r59 fmuls,5 %r61, %r50, %r60 movaw,0 area=5, ind=0, am=1, be=0, %b[17] movaw,1 area=5, ind=4, am=0, be=0, %r61 movaw,2 area=5, ind=0, am=1, be=0, %b[19] movaw,3 area=5, ind=4, am=0, be=0, %b[18] } { loop_mode fmuls,0 %b[2], %r9, %b[20] fmuls,1 %b[1], %r7, %b[21] fmuls,2 %b[2], %r7, %r7 fmuls,3 %b[1], %r9, %r9 fmuls,4 %r62, %r50, %r50 fmuls,5 %r62, %r49, %r49 movaw,0 area=6, ind=0, am=1, be=0, %b[1] movaw,1 area=6, ind=4, am=0, be=0, %r62 movaw,2 area=6, ind=0, am=1, be=0, %b[22] movaw,3 area=6, ind=4, am=0, be=0, %b[2] } { loop_mode fmuls,0 %b[5], %g30, %b[23] fmuls,1 %b[5], %g31, %b[5] fmuls,2 %b[4], %g27, %b[24] fmuls,3 %b[3], %g26, %b[25] fmuls,4 %b[4], %g26, %g26 fmuls,5 %b[3], %g27, %g27 movaw,0 area=7, ind=0, am=1, be=0, %b[4] movaw,1 area=7, ind=4, am=0, be=0, %b[3] movaw,2 area=7, ind=0, am=1, be=0, %b[27] movaw,3 area=7, ind=4, am=0, be=0, %b[26] } { loop_mode fmuls,0 %b[6], %g31, %g31 fmuls,1 %b[6], %g30, %g30 fsubs,2 %r57, %r58, %r57 fadds,3 %g23, %g22, %g22 fsubs,4 %r63, %b[0], %g23 fadds,5 %g19, %g18, %g18 movaw,0 area=8, ind=0, am=1, be=0, %r58 movaw,1 area=8, ind=4, am=0, be=0, %g19 movaw,2 area=8, ind=0, am=1, be=0, %b[0] movaw,3 area=8, ind=4, am=0, be=0, %r63 } { loop_mode fsubs,0 %b[7], %b[8], %b[6] fadds,1 %r52, %r51, %r51 fsubs,2 %b[11], %b[12], %r52 fmuls,3 %r45, %g24, %b[7] fmuls,4 %r46, %g24, %g24 fmuls,5 %r45, %g25, %r45 movaw,0 area=9, ind=0, am=1, be=0, %b[11] movaw,1 area=9, ind=4, am=0, be=0, %b[8] movaw,2 area=9, ind=0, am=1, be=0, %b[28] movaw,3 area=9, ind=4, am=0, be=0, %b[12] } { loop_mode fadds,0 %r43, %r44, %r43 fsubs,1 %b[20], %b[21], %r44 fsubs,2 %b[15], %b[16], %b[15] fsubs,3 %r50, %r59, %r50 fadds,4 %r49, %r60, %r49 fmuls,5 %r46, %g25, %g25 } { loop_mode fadds,0 %r4, %r5, %r4 fmuls,1 %r53, %g16, %g27 fmuls,2 %r53, %g17, %r5 fadds,3 %g26, %g27, %g26 fmuls,4 %b[9], %g21, %r46 fmuls,5 %r54, %g17, %g17 } { loop_mode fadds,0 %r7, %r9, %r7 fsubs,1 %g31, %b[23], %g31 fadds,2 %g30, %b[5], %g30 fmuls,3 %r54, %g16, %g16 fmuls,4 %b[10], %g21, %g21 fmuls,5 %b[9], %g20, %r9 } { loop_mode fsubs,0 %b[24], %b[25], %r53 fmuls,1 %b[10], %g20, %g20 fadds,2 %r52, %r57, %r54 fadds,3 %g24, %r45, %g24 } { loop_mode fadds,0 %r48, %r44, %r45 fsubs,1 %r48, %r44, %r44 fsubs,2 %g22, %r43, %r48 fsubs,3 %r1, %r49, %r59 fadds,4 %r3, %r50, %r60 fadds,5 %r1, %r49, %r1 } { loop_mode fadds,0 %r43, %g22, %g22 fsubs,1 %g18, %r51, %r43 fadds,2 %b[6], %g23, %r49 fadds,3 %r51, %g18, %g18 fsubs,4 %b[6], %g23, %g23 fsubs,5 %r52, %r57, %r51 } { loop_mode fsubs,0 %r3, %r50, %r3 fadds,1 %r47, %r7, %r50 fsubs,2 %r47, %r7, %r7 fsubs,3 %g25, %b[7], %g25 fsubs,4 %g21, %r9, %g21 } { loop_mode fadds,0 %r4, %g26, %r9 fsubs,1 %g26, %r4, %g26 fadds,2 %r42, %g31, %r4 fsubs,3 %g17, %g27, %g17 fadds,4 %g16, %r5, %g16 } { loop_mode fadds,0 %r41, %g30, %g27 fsubs,1 %r42, %g31, %g31 fadds,2 %b[15], %r53, %r5 fsubs,3 %r41, %g30, %g30 } { loop_mode fsubs,0 %b[15], %r53, %r41 fadds,1 %g20, %r46, %g20 fsubs,2 %r44, %r48, %r42 fsubs,3 %r59, %g23, %r46 fadds,4 %r59, %g23, %g23 fadds,5 %r1, %g18, %r47 } { loop_mode fsubs,0 %r45, %r54, %r52 fadds,1 %r44, %r48, %r44 fadds,2 %r45, %r54, %r45 fsubs,3 %r1, %g18, %g18 fsubs,4 %g29, %g21, %r1 fadds,5 %g29, %g21, %g21 } { loop_mode fadds,0 %r50, %g22, %g29 fsubs,1 %r50, %g22, %g22 fadds,2 %r3, %r43, %r48 fadds,3 %r60, %r49, %r50 fsubs,4 %r60, %r49, %r49 fadds,5 %g24, %g16, %r53 } { loop_mode fsubs,0 %r3, %r43, %r3 fadds,1 %g27, %r9, %r43 fsubs,2 %r7, %r51, %r54 fadds,3 %r7, %r51, %r7 fsubs,4 %g25, %g17, %r51 fsubs,5 %g16, %g24, %g16 } { loop_mode fsubs,0 %g27, %r9, %g24 fadds,1 %g31, %g26, %g27 fadds,2 %r4, %r5, %r9 fadds,3 %g25, %g17, %g17 fmuls,4 %r62, %r46, %g25 fmuls,5 %b[1], %r46, %r46 } { loop_mode fadds,0 %g30, %r41, %r57 fsubs,1 %g31, %g26, %g26 fsubs,2 %g30, %r41, %g30 fsubs,3 %r4, %r5, %g31 fmuls,4 %b[8], %g23, %r4 fmuls,5 %b[11], %g23, %g23 } { loop_mode fadds,0 %g28, %g20, %r5 fsubs,1 %g28, %g20, %g20 fmuls,2 %b[22], %r42, %g28 fmuls,3 %b[2], %r42, %r41 fmuls,4 %b[18], %r47, %r42 fmuls,5 %b[19], %r47, %r47 } { loop_mode fmuls,0 %b[4], %r52, %r59 fmuls,1 %b[3], %r52, %r52 fmuls,2 %b[28], %r44, %r60 fmuls,3 %b[12], %r44, %r44 fmuls,4 %r56, %r45, %b[5] fmuls,5 %r55, %r45, %r45 } { loop_mode fmuls,0 %r63, %g18, %b[6] fmuls,1 %b[0], %g18, %g18 fmuls,2 %b[11], %r48, %b[7] fmuls,3 %b[8], %r48, %r48 fmuls,4 %r55, %g29, %r55 fmuls,5 %r56, %g29, %g29 } { loop_mode fmuls,0 %b[3], %g22, %r56 fmuls,1 %b[4], %g22, %g22 fmuls,2 %b[1], %r3, %b[1] fmuls,3 %r62, %r3, %r3 fmuls,4 %b[12], %r7, %r62 fmuls,5 %b[28], %r7, %r7 } { loop_mode fmuls,0 %b[19], %r50, %b[3] fmuls,1 %b[18], %r50, %r50 fmuls,2 %b[0], %r49, %b[0] fmuls,3 %r63, %r49, %r49 fmuls,4 %b[26], %g24, %r63 fmuls,5 %b[27], %g24, %g24 } { loop_mode fmuls,0 %g19, %r57, %b[4] fmuls,1 %r58, %r57, %r57 fmuls,2 %b[2], %r54, %b[2] fmuls,3 %b[22], %r54, %r54 fmuls,4 %b[13], %r43, %b[8] fmuls,5 %b[14], %r43, %r43 } { loop_mode fmuls,0 %r61, %g30, %b[9] fmuls,1 %b[17], %g30, %g30 fmuls,2 %r58, %g27, %r58 fmuls,3 %g19, %g27, %g19 fmuls,4 %b[14], %r9, %g27 fmuls,5 %b[13], %r9, %r9 } { loop_mode fmuls,0 %b[17], %g26, %b[10] fmuls,1 %r61, %g26, %g26 fmuls,2 %b[27], %g31, %r61 fmuls,3 %b[26], %g31, %g31 fsubs,4 %g20, %r51, %b[11] fadds,5 %g20, %r51, %g20 } { loop_mode fadds,0 %g21, %g17, %r51 fadds,1 %r5, %r53, %b[12] fsubs,2 %r1, %g16, %b[13] fsubs,3 %g21, %g17, %g17 fsubs,4 %r5, %r53, %g21 fadds,5 %r1, %g16, %g16 } { loop_mode fsubs,0 %b[5], %r55, %r1 fadds,1 %g29, %r45, %g29 fsubs,2 %r59, %r56, %r5 fadds,3 %g22, %r52, %g22 fsubs,4 %b[7], %r4, %r4 fadds,5 %g23, %r48, %g23 } { loop_mode fsubs,0 %b[3], %r42, %r42 fadds,1 %r47, %r50, %r45 fsubs,2 %b[1], %g25, %g25 fadds,3 %r46, %r3, %r3 fsubs,4 %b[0], %b[6], %r46 fadds,5 %g18, %r49, %g18 } { loop_mode fsubs,0 %r61, %r63, %r47 fsubs,1 %r58, %b[4], %r48 fsubs,2 %g28, %b[2], %g28 fadds,3 %r54, %r41, %r41 fsubs,4 %r60, %r62, %r49 fadds,5 %r7, %r44, %r7 } { loop_mode fsubs,0 %g27, %b[8], %g27 fadds,1 %r43, %r9, %r9 fadds,2 %g30, %g26, %g26 fadds,3 %g24, %g31, %g24 fadds,4 %r57, %g19, %g19 } { loop_mode fsubs,0 %b[10], %b[9], %g30 fsubs,1 %r51, %r1, %r43 fadds,2 %r51, %r1, %g22 fadds,3 %g21, %g22, %g31 fsubs,4 %g21, %g22, %g21 } { loop_mode fadds,0 %g17, %r5, %r1 fsubs,1 %g17, %r5, %g17 fadds,2 %b[12], %g29, %r5 } { loop_mode fsubs,0 %b[12], %g29, %g29 fadds,1 %b[13], %g28, %r44 fsubs,2 %b[13], %g28, %g28 fadds,3 %g20, %r7, %r50 fsubs,4 %g16, %r49, %r51 fsubs,5 %g20, %r7, %g20 } { loop_mode fsubs,0 %r48, %r4, %r7 fadds,1 %r47, %r46, %r49 fadds,2 %r48, %r4, %r4 fadds,3 %b[11], %r41, %r52 fadds,4 %g16, %r49, %g16 fsubs,5 %b[11], %r41, %r41 } { loop_mode fsubs,0 %r47, %r46, %r46 fadds,1 %r9, %r45, %r47 fsubs,2 %g27, %r42, %r53 fadds,3 %g19, %g23, %r48 fsubs,4 %g18, %g24, %r54 fsubs,5 %g23, %g19, %g19 } { loop_mode fsubs,0 %r45, %r9, %g23 fadds,1 %g27, %r42, %g27 fadds,2 %g26, %r3, %r9 fadds,3 %g24, %g18, %g18 fsubs,4 %r3, %g26, %g24 } { loop_mode fadds,0 %g30, %g25, %g26 fsubs,1 %g30, %g25, %g25 } { loop_mode fsubs,0 %r1, %r49, %g30 fadds,1 %r1, %r49, %r1 } { loop_mode fsubs,0 %g21, %r46, %r3 fadds,1 %g21, %r46, %g21 fsubs,2 %g20, %r7, %r42 fadds,3 %r51, %g19, %r45 fadds,4 %r50, %r48, %r46 fsubs,5 %r51, %g19, %g19 } { loop_mode fsubs,0 %r43, %g23, %r49 fadds,1 %r43, %g23, %g23 fadds,2 %g20, %r7, %g20 fsubs,3 %r50, %r48, %r7 fadds,4 %g16, %r4, %r43 fsubs,5 %g17, %r54, %r48 } { loop_mode fadds,0 %r5, %r47, %r50 fsubs,1 %r5, %r47, %r5 fadds,2 %g29, %r53, %r47 fsubs,3 %g29, %r53, %g29 fsubs,4 %g16, %r4, %g16 fadds,5 %g17, %r54, %g17 } { loop_mode fsubs,0 %g22, %g27, %r4 fadds,1 %r52, %r9, %r51 fadds,2 %g31, %g18, %r53 fsubs,3 %g28, %g24, %r54 fsubs,4 %r52, %r9, %r9 fsubs,5 %g31, %g18, %g18 } { loop_mode fadds,0 %g28, %g24, %g24 fadds,1 %g22, %g27, %g22 fsubs,2 %r44, %g26, %g26 fadds,3 %r44, %g26, %g27 fsubs,4 %r41, %g25, %g28 fadds,5 %r41, %g25, %g25 } { loop_mode stw,2 %r23, %r0, %g21 stw,5 %r32, %r0, %g30 } { loop_mode stw,2 %r6, %r0, %r3 stw,5 %r39, %r0, %r1 } { loop_mode stw,2 %r22, %r0, %g20 stw,5 %r21, %r0, %r42 } { loop_mode stw,2 %r27, %r0, %r45 stw,5 %r38, %r0, %r49 } { loop_mode stw,2 %r31, %r0, %g23 stw,5 %r34, %r0, %g19 } { loop_mode stw,2 %r17, %r0, %r7 stw,5 %r13, %r0, %r46 } { loop_mode stw,2 %r20, %r0, %r5 stw,5 %r2, %r0, %r50 } { loop_mode stw,2 %r16, %r0, %r47 stw,5 %r29, %r0, %r48 } { loop_mode stw,2 %r28, %r0, %g17 stw,5 %r12, %r0, %g29 } { loop_mode stw,2 %r26, %r0, %g16 stw,5 %r30, %r0, %r43 } { loop_mode stw,2 %r37, %r0, %r4 stw,5 %r14, %r0, %r53 } { loop_mode stw,2 %r18, %r0, %g18 stw,5 %r35, %r0, %r54 } { loop_mode stw,2 %r25, %r0, %g24 stw,5 %r15, %r0, %r51 } { loop_mode stw,2 %r19, %r0, %r9 stw,5 %r40, %r0, %g22 } { loop_mode stw,2 %r24, %r0, %g25 stw,5 %r11, %r0, %g28 } { loop_mode ct %ctpr1 ? %NOT_LOOP_END alc alcf=1, alct=1 stw,2 %r33, %r0, %g26 addd,3,sm 0x8, %r0, %r0 stw,5 %r36, %r0, %g27 }
Теоретическая скорость: 16 комплексных чисел за 77 тактов (16/77) = 1.66 Байт/такт
Четверная теоретическая скорость: 6.65 Байт/такт
Замеры скорости
2. stage_radix4_2x_simd64
Здесь происходит ручная раскрутка алгоритма stage_radix4_simd64 в 2 раза.
Код на Си
void stage_radix4_2x_simd64(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coefC_a, myComplex *coefD_a, myComplex *coefE_a, myComplex *coefC_b, myComplex *coefD_b, myComplex *coefE_b) { uint64_t *x0_in = (uint64_t*)&data_in[ 0]; uint64_t *y0_in = (uint64_t*)&data_in[ 1]; uint64_t *z0_in = (uint64_t*)&data_in[ 2]; uint64_t *w0_in = (uint64_t*)&data_in[ 3]; uint64_t *x1_in = (uint64_t*)&data_in[ 4]; uint64_t *y1_in = (uint64_t*)&data_in[ 5]; uint64_t *z1_in = (uint64_t*)&data_in[ 6]; uint64_t *w1_in = (uint64_t*)&data_in[ 7]; uint64_t *x2_in = (uint64_t*)&data_in[ 8]; uint64_t *y2_in = (uint64_t*)&data_in[ 9]; uint64_t *z2_in = (uint64_t*)&data_in[10]; uint64_t *w2_in = (uint64_t*)&data_in[11]; uint64_t *x3_in = (uint64_t*)&data_in[12]; uint64_t *y3_in = (uint64_t*)&data_in[13]; uint64_t *z3_in = (uint64_t*)&data_in[14]; uint64_t *w3_in = (uint64_t*)&data_in[15]; uint64_t *c0a_in = (uint64_t*)&coefC_a[0]; uint64_t *c1a_in = (uint64_t*)&coefC_a[1]; uint64_t *c2a_in = (uint64_t*)&coefC_a[2]; uint64_t *c3a_in = (uint64_t*)&coefC_a[3]; uint64_t *d0a_in = (uint64_t*)&coefD_a[0]; uint64_t *d1a_in = (uint64_t*)&coefD_a[1]; uint64_t *d2a_in = (uint64_t*)&coefD_a[2]; uint64_t *d3a_in = (uint64_t*)&coefD_a[3]; uint64_t *e0a_in = (uint64_t*)&coefE_a[0]; uint64_t *e1a_in = (uint64_t*)&coefE_a[1]; uint64_t *e2a_in = (uint64_t*)&coefE_a[2]; uint64_t *e3a_in = (uint64_t*)&coefE_a[3]; uint64_t *c0b_in = (uint64_t*)&coefC_b[0*data_count/16]; uint64_t *c1b_in = (uint64_t*)&coefC_b[1*data_count/16]; uint64_t *c2b_in = (uint64_t*)&coefC_b[2*data_count/16]; uint64_t *c3b_in = (uint64_t*)&coefC_b[3*data_count/16]; uint64_t *d0b_in = (uint64_t*)&coefD_b[0*data_count/16]; uint64_t *d1b_in = (uint64_t*)&coefD_b[1*data_count/16]; uint64_t *d2b_in = (uint64_t*)&coefD_b[2*data_count/16]; uint64_t *d3b_in = (uint64_t*)&coefD_b[3*data_count/16]; uint64_t *e0b_in = (uint64_t*)&coefE_b[0*data_count/16]; uint64_t *e1b_in = (uint64_t*)&coefE_b[1*data_count/16]; uint64_t *e2b_in = (uint64_t*)&coefE_b[2*data_count/16]; uint64_t *e3b_in = (uint64_t*)&coefE_b[3*data_count/16]; uint64_t *out_0 = (uint64_t*)&data_out[ 0*data_count/16]; uint64_t *out_1 = (uint64_t*)&data_out[ 1*data_count/16]; uint64_t *out_2 = (uint64_t*)&data_out[ 2*data_count/16]; uint64_t *out_3 = (uint64_t*)&data_out[ 3*data_count/16]; uint64_t *out_4 = (uint64_t*)&data_out[ 4*data_count/16]; uint64_t *out_5 = (uint64_t*)&data_out[ 5*data_count/16]; uint64_t *out_6 = (uint64_t*)&data_out[ 6*data_count/16]; uint64_t *out_7 = (uint64_t*)&data_out[ 7*data_count/16]; uint64_t *out_8 = (uint64_t*)&data_out[ 8*data_count/16]; uint64_t *out_9 = (uint64_t*)&data_out[ 9*data_count/16]; uint64_t *out_10 = (uint64_t*)&data_out[10*data_count/16]; uint64_t *out_11 = (uint64_t*)&data_out[11*data_count/16]; uint64_t *out_12 = (uint64_t*)&data_out[12*data_count/16]; uint64_t *out_13 = (uint64_t*)&data_out[13*data_count/16]; uint64_t *out_14 = (uint64_t*)&data_out[14*data_count/16]; uint64_t *out_15 = (uint64_t*)&data_out[15*data_count/16]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/16; ++i) { uint64_t x0 = x0_in[16*i]; uint64_t y0 = y0_in[16*i]; uint64_t z0 = z0_in[16*i]; uint64_t w0 = w0_in[16*i]; uint64_t c0 = c0a_in[4*i]; uint64_t d0 = d0a_in[4*i]; uint64_t e0 = e0a_in[4*i]; uint64_t x1 = x1_in[16*i]; uint64_t y1 = y1_in[16*i]; uint64_t z1 = z1_in[16*i]; uint64_t w1 = w1_in[16*i]; uint64_t c1 = c1a_in[4*i]; uint64_t d1 = d1a_in[4*i]; uint64_t e1 = e1a_in[4*i]; uint64_t x2 = x2_in[16*i]; uint64_t y2 = y2_in[16*i]; uint64_t z2 = z2_in[16*i]; uint64_t w2 = w2_in[16*i]; uint64_t c2 = c2a_in[4*i]; uint64_t d2 = d2a_in[4*i]; uint64_t e2 = e2a_in[4*i]; uint64_t x3 = x3_in[16*i]; uint64_t y3 = y3_in[16*i]; uint64_t z3 = z3_in[16*i]; uint64_t w3 = w3_in[16*i]; uint64_t c3 = c3a_in[4*i]; uint64_t d3 = d3a_in[4*i]; uint64_t e3 = e3a_in[4*i]; uint64_t conj_c0 = __builtin_e2k_pxord(c0, 1LL<<63); uint64_t conj_c1 = __builtin_e2k_pxord(c1, 1LL<<63); uint64_t conj_c2 = __builtin_e2k_pxord(c2, 1LL<<63); uint64_t conj_c3 = __builtin_e2k_pxord(c3, 1LL<<63); uint64_t conj_d0 = __builtin_e2k_pxord(d0, 1LL<<63); uint64_t conj_d1 = __builtin_e2k_pxord(d1, 1LL<<63); uint64_t conj_d2 = __builtin_e2k_pxord(d2, 1LL<<63); uint64_t conj_d3 = __builtin_e2k_pxord(d3, 1LL<<63); uint64_t conj_e0 = __builtin_e2k_pxord(e0, 1LL<<63); uint64_t conj_e1 = __builtin_e2k_pxord(e1, 1LL<<63); uint64_t conj_e2 = __builtin_e2k_pxord(e2, 1LL<<63); uint64_t conj_e3 = __builtin_e2k_pxord(e3, 1LL<<63); uint64_t swap_c0 = __builtin_e2k_pshufb(0, c0, 0x0302010007060504); uint64_t swap_c1 = __builtin_e2k_pshufb(0, c1, 0x0302010007060504); uint64_t swap_c2 = __builtin_e2k_pshufb(0, c2, 0x0302010007060504); uint64_t swap_c3 = __builtin_e2k_pshufb(0, c3, 0x0302010007060504); uint64_t swap_d0 = __builtin_e2k_pshufb(0, d0, 0x0302010007060504); uint64_t swap_d1 = __builtin_e2k_pshufb(0, d1, 0x0302010007060504); uint64_t swap_d2 = __builtin_e2k_pshufb(0, d2, 0x0302010007060504); uint64_t swap_d3 = __builtin_e2k_pshufb(0, d3, 0x0302010007060504); uint64_t swap_e0 = __builtin_e2k_pshufb(0, e0, 0x0302010007060504); uint64_t swap_e1 = __builtin_e2k_pshufb(0, e1, 0x0302010007060504); uint64_t swap_e2 = __builtin_e2k_pshufb(0, e2, 0x0302010007060504); uint64_t swap_e3 = __builtin_e2k_pshufb(0, e3, 0x0302010007060504); uint64_t cy0_real = __builtin_e2k_pfmuls(conj_c0, y0); uint64_t cy1_real = __builtin_e2k_pfmuls(conj_c1, y1); uint64_t cy2_real = __builtin_e2k_pfmuls(conj_c2, y2); uint64_t cy3_real = __builtin_e2k_pfmuls(conj_c3, y3); uint64_t dz0_real = __builtin_e2k_pfmuls(conj_d0, z0); uint64_t dz1_real = __builtin_e2k_pfmuls(conj_d1, z1); uint64_t dz2_real = __builtin_e2k_pfmuls(conj_d2, z2); uint64_t dz3_real = __builtin_e2k_pfmuls(conj_d3, z3); uint64_t ew0_real = __builtin_e2k_pfmuls(conj_e0, w0); uint64_t ew1_real = __builtin_e2k_pfmuls(conj_e1, w1); uint64_t ew2_real = __builtin_e2k_pfmuls(conj_e2, w2); uint64_t ew3_real = __builtin_e2k_pfmuls(conj_e3, w3); uint64_t cy0_imag = __builtin_e2k_pfmuls(swap_c0, y0); uint64_t cy1_imag = __builtin_e2k_pfmuls(swap_c1, y1); uint64_t cy2_imag = __builtin_e2k_pfmuls(swap_c2, y2); uint64_t cy3_imag = __builtin_e2k_pfmuls(swap_c3, y3); uint64_t dz0_imag = __builtin_e2k_pfmuls(swap_d0, z0); uint64_t dz1_imag = __builtin_e2k_pfmuls(swap_d1, z1); uint64_t dz2_imag = __builtin_e2k_pfmuls(swap_d2, z2); uint64_t dz3_imag = __builtin_e2k_pfmuls(swap_d3, z3); uint64_t ew0_imag = __builtin_e2k_pfmuls(swap_e0, w0); uint64_t ew1_imag = __builtin_e2k_pfmuls(swap_e1, w1); uint64_t ew2_imag = __builtin_e2k_pfmuls(swap_e2, w2); uint64_t ew3_imag = __builtin_e2k_pfmuls(swap_e3, w3); uint64_t cy0 = __builtin_e2k_pfhadds(cy0_real, cy0_imag); uint64_t cy1 = __builtin_e2k_pfhadds(cy1_real, cy1_imag); uint64_t cy2 = __builtin_e2k_pfhadds(cy2_real, cy2_imag); uint64_t cy3 = __builtin_e2k_pfhadds(cy3_real, cy3_imag); uint64_t dz0 = __builtin_e2k_pfhadds(dz0_real, dz0_imag); uint64_t dz1 = __builtin_e2k_pfhadds(dz1_real, dz1_imag); uint64_t dz2 = __builtin_e2k_pfhadds(dz2_real, dz2_imag); uint64_t dz3 = __builtin_e2k_pfhadds(dz3_real, dz3_imag); uint64_t ew0 = __builtin_e2k_pfhadds(ew0_real, ew0_imag); uint64_t ew1 = __builtin_e2k_pfhadds(ew1_real, ew1_imag); uint64_t ew2 = __builtin_e2k_pfhadds(ew2_real, ew2_imag); uint64_t ew3 = __builtin_e2k_pfhadds(ew3_real, ew3_imag); uint64_t add02_0 = __builtin_e2k_pfadds( x0, dz0); uint64_t add02_1 = __builtin_e2k_pfadds( x1, dz1); uint64_t add02_2 = __builtin_e2k_pfadds( x2, dz2); uint64_t add02_3 = __builtin_e2k_pfadds( x3, dz3); uint64_t sub02_0 = __builtin_e2k_pfsubs( x0, dz0); uint64_t sub02_1 = __builtin_e2k_pfsubs( x1, dz1); uint64_t sub02_2 = __builtin_e2k_pfsubs( x2, dz2); uint64_t sub02_3 = __builtin_e2k_pfsubs( x3, dz3); uint64_t add13_0 = __builtin_e2k_pfadds(cy0, ew0); uint64_t add13_1 = __builtin_e2k_pfadds(cy1, ew1); uint64_t add13_2 = __builtin_e2k_pfadds(cy2, ew2); uint64_t add13_3 = __builtin_e2k_pfadds(cy3, ew3); uint64_t sub13_0 = __builtin_e2k_pfsubs(cy0, ew0); uint64_t sub13_1 = __builtin_e2k_pfsubs(cy1, ew1); uint64_t sub13_2 = __builtin_e2k_pfsubs(cy2, ew2); uint64_t sub13_3 = __builtin_e2k_pfsubs(cy3, ew3); //uint64_t conj_sub13_0 = __builtin_e2k_pxord(sub13_0, 1LL<<63); //uint64_t conj_sub13_1 = __builtin_e2k_pxord(sub13_1, 1LL<<63); //uint64_t conj_sub13_2 = __builtin_e2k_pxord(sub13_2, 1LL<<63); //uint64_t conj_sub13_3 = __builtin_e2k_pxord(sub13_3, 1LL<<63); //uint64_t sub13i_0 = __builtin_e2k_pshufb(0, conj_sub13_0, 0x0302010007060504); //uint64_t sub13i_1 = __builtin_e2k_pshufb(0, conj_sub13_1, 0x0302010007060504); //uint64_t sub13i_2 = __builtin_e2k_pshufb(0, conj_sub13_2, 0x0302010007060504); //uint64_t sub13i_3 = __builtin_e2k_pshufb(0, conj_sub13_3, 0x0302010007060504); uint64_t swap_sub13_0 = __builtin_e2k_pshufb(0, sub13_0, 0x0302010007060504); uint64_t swap_sub13_1 = __builtin_e2k_pshufb(0, sub13_1, 0x0302010007060504); uint64_t swap_sub13_2 = __builtin_e2k_pshufb(0, sub13_2, 0x0302010007060504); uint64_t swap_sub13_3 = __builtin_e2k_pshufb(0, sub13_3, 0x0302010007060504); uint64_t sub13i_0 = __builtin_e2k_pxord(swap_sub13_0, 1LL<<31); uint64_t sub13i_1 = __builtin_e2k_pxord(swap_sub13_1, 1LL<<31); uint64_t sub13i_2 = __builtin_e2k_pxord(swap_sub13_2, 1LL<<31); uint64_t sub13i_3 = __builtin_e2k_pxord(swap_sub13_3, 1LL<<31); uint64_t out0 = __builtin_e2k_pfadds(add02_0, add13_0); uint64_t out1 = __builtin_e2k_pfadds(add02_1, add13_1); uint64_t out2 = __builtin_e2k_pfadds(add02_2, add13_2); uint64_t out3 = __builtin_e2k_pfadds(add02_3, add13_3); uint64_t out4 = __builtin_e2k_pfsubs(sub02_0, sub13i_0); uint64_t out5 = __builtin_e2k_pfsubs(sub02_1, sub13i_1); uint64_t out6 = __builtin_e2k_pfsubs(sub02_2, sub13i_2); uint64_t out7 = __builtin_e2k_pfsubs(sub02_3, sub13i_3); uint64_t out8 = __builtin_e2k_pfsubs(add02_0, add13_0); uint64_t out9 = __builtin_e2k_pfsubs(add02_1, add13_1); uint64_t out10 = __builtin_e2k_pfsubs(add02_2, add13_2); uint64_t out11 = __builtin_e2k_pfsubs(add02_3, add13_3); uint64_t out12 = __builtin_e2k_pfadds(sub02_0, sub13i_0); uint64_t out13 = __builtin_e2k_pfadds(sub02_1, sub13i_1); uint64_t out14 = __builtin_e2k_pfadds(sub02_2, sub13i_2); uint64_t out15 = __builtin_e2k_pfadds(sub02_3, sub13i_3); x0 = out0; y0 = out1; z0 = out2; w0 = out3; c0 = c0b_in[i]; d0 = d0b_in[i]; e0 = e0b_in[i]; x1 = out4; y1 = out5; z1 = out6; w1 = out7; c1 = c1b_in[i]; d1 = d1b_in[i]; e1 = e1b_in[i]; x2 = out8; y2 = out9; z2 = out10; w2 = out11; c2 = c2b_in[i]; d2 = d2b_in[i]; e2 = e2b_in[i]; x3 = out12; y3 = out13; z3 = out14; w3 = out15; c3 = c3b_in[i]; d3 = d3b_in[i]; e3 = e3b_in[i]; conj_c0 = __builtin_e2k_pxord(c0, 1LL<<63); conj_c1 = __builtin_e2k_pxord(c1, 1LL<<63); conj_c2 = __builtin_e2k_pxord(c2, 1LL<<63); conj_c3 = __builtin_e2k_pxord(c3, 1LL<<63); conj_d0 = __builtin_e2k_pxord(d0, 1LL<<63); conj_d1 = __builtin_e2k_pxord(d1, 1LL<<63); conj_d2 = __builtin_e2k_pxord(d2, 1LL<<63); conj_d3 = __builtin_e2k_pxord(d3, 1LL<<63); conj_e0 = __builtin_e2k_pxord(e0, 1LL<<63); conj_e1 = __builtin_e2k_pxord(e1, 1LL<<63); conj_e2 = __builtin_e2k_pxord(e2, 1LL<<63); conj_e3 = __builtin_e2k_pxord(e3, 1LL<<63); swap_c0 = __builtin_e2k_pshufb(0, c0, 0x0302010007060504); swap_c1 = __builtin_e2k_pshufb(0, c1, 0x0302010007060504); swap_c2 = __builtin_e2k_pshufb(0, c2, 0x0302010007060504); swap_c3 = __builtin_e2k_pshufb(0, c3, 0x0302010007060504); swap_d0 = __builtin_e2k_pshufb(0, d0, 0x0302010007060504); swap_d1 = __builtin_e2k_pshufb(0, d1, 0x0302010007060504); swap_d2 = __builtin_e2k_pshufb(0, d2, 0x0302010007060504); swap_d3 = __builtin_e2k_pshufb(0, d3, 0x0302010007060504); swap_e0 = __builtin_e2k_pshufb(0, e0, 0x0302010007060504); swap_e1 = __builtin_e2k_pshufb(0, e1, 0x0302010007060504); swap_e2 = __builtin_e2k_pshufb(0, e2, 0x0302010007060504); swap_e3 = __builtin_e2k_pshufb(0, e3, 0x0302010007060504); cy0_real = __builtin_e2k_pfmuls(conj_c0, y0); cy1_real = __builtin_e2k_pfmuls(conj_c1, y1); cy2_real = __builtin_e2k_pfmuls(conj_c2, y2); cy3_real = __builtin_e2k_pfmuls(conj_c3, y3); dz0_real = __builtin_e2k_pfmuls(conj_d0, z0); dz1_real = __builtin_e2k_pfmuls(conj_d1, z1); dz2_real = __builtin_e2k_pfmuls(conj_d2, z2); dz3_real = __builtin_e2k_pfmuls(conj_d3, z3); ew0_real = __builtin_e2k_pfmuls(conj_e0, w0); ew1_real = __builtin_e2k_pfmuls(conj_e1, w1); ew2_real = __builtin_e2k_pfmuls(conj_e2, w2); ew3_real = __builtin_e2k_pfmuls(conj_e3, w3); cy0_imag = __builtin_e2k_pfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_pfmuls(swap_c1, y1); cy2_imag = __builtin_e2k_pfmuls(swap_c2, y2); cy3_imag = __builtin_e2k_pfmuls(swap_c3, y3); dz0_imag = __builtin_e2k_pfmuls(swap_d0, z0); dz1_imag = __builtin_e2k_pfmuls(swap_d1, z1); dz2_imag = __builtin_e2k_pfmuls(swap_d2, z2); dz3_imag = __builtin_e2k_pfmuls(swap_d3, z3); ew0_imag = __builtin_e2k_pfmuls(swap_e0, w0); ew1_imag = __builtin_e2k_pfmuls(swap_e1, w1); ew2_imag = __builtin_e2k_pfmuls(swap_e2, w2); ew3_imag = __builtin_e2k_pfmuls(swap_e3, w3); cy0 = __builtin_e2k_pfhadds(cy0_real, cy0_imag); cy1 = __builtin_e2k_pfhadds(cy1_real, cy1_imag); cy2 = __builtin_e2k_pfhadds(cy2_real, cy2_imag); cy3 = __builtin_e2k_pfhadds(cy3_real, cy3_imag); dz0 = __builtin_e2k_pfhadds(dz0_real, dz0_imag); dz1 = __builtin_e2k_pfhadds(dz1_real, dz1_imag); dz2 = __builtin_e2k_pfhadds(dz2_real, dz2_imag); dz3 = __builtin_e2k_pfhadds(dz3_real, dz3_imag); ew0 = __builtin_e2k_pfhadds(ew0_real, ew0_imag); ew1 = __builtin_e2k_pfhadds(ew1_real, ew1_imag); ew2 = __builtin_e2k_pfhadds(ew2_real, ew2_imag); ew3 = __builtin_e2k_pfhadds(ew3_real, ew3_imag); add02_0 = __builtin_e2k_pfadds( x0, dz0); add02_1 = __builtin_e2k_pfadds( x1, dz1); add02_2 = __builtin_e2k_pfadds( x2, dz2); add02_3 = __builtin_e2k_pfadds( x3, dz3); sub02_0 = __builtin_e2k_pfsubs( x0, dz0); sub02_1 = __builtin_e2k_pfsubs( x1, dz1); sub02_2 = __builtin_e2k_pfsubs( x2, dz2); sub02_3 = __builtin_e2k_pfsubs( x3, dz3); add13_0 = __builtin_e2k_pfadds(cy0, ew0); add13_1 = __builtin_e2k_pfadds(cy1, ew1); add13_2 = __builtin_e2k_pfadds(cy2, ew2); add13_3 = __builtin_e2k_pfadds(cy3, ew3); sub13_0 = __builtin_e2k_pfsubs(cy0, ew0); sub13_1 = __builtin_e2k_pfsubs(cy1, ew1); sub13_2 = __builtin_e2k_pfsubs(cy2, ew2); sub13_3 = __builtin_e2k_pfsubs(cy3, ew3); //conj_sub13_0 = __builtin_e2k_pxord(sub13_0, 1LL<<63); //conj_sub13_1 = __builtin_e2k_pxord(sub13_1, 1LL<<63); //conj_sub13_2 = __builtin_e2k_pxord(sub13_2, 1LL<<63); //conj_sub13_3 = __builtin_e2k_pxord(sub13_3, 1LL<<63); //sub13i_0 = __builtin_e2k_pshufb(0, conj_sub13_0, 0x0302010007060504); //sub13i_1 = __builtin_e2k_pshufb(0, conj_sub13_1, 0x0302010007060504); //sub13i_2 = __builtin_e2k_pshufb(0, conj_sub13_2, 0x0302010007060504); //sub13i_3 = __builtin_e2k_pshufb(0, conj_sub13_3, 0x0302010007060504); swap_sub13_0 = __builtin_e2k_pshufb(0, sub13_0, 0x0302010007060504); swap_sub13_1 = __builtin_e2k_pshufb(0, sub13_1, 0x0302010007060504); swap_sub13_2 = __builtin_e2k_pshufb(0, sub13_2, 0x0302010007060504); swap_sub13_3 = __builtin_e2k_pshufb(0, sub13_3, 0x0302010007060504); sub13i_0 = __builtin_e2k_pxord(swap_sub13_0, 1LL<<31); sub13i_1 = __builtin_e2k_pxord(swap_sub13_1, 1LL<<31); sub13i_2 = __builtin_e2k_pxord(swap_sub13_2, 1LL<<31); sub13i_3 = __builtin_e2k_pxord(swap_sub13_3, 1LL<<31); out_0[i] = __builtin_e2k_pfadds(add02_0, add13_0); out_1[i] = __builtin_e2k_pfadds(add02_1, add13_1); out_2[i] = __builtin_e2k_pfadds(add02_2, add13_2); out_3[i] = __builtin_e2k_pfadds(add02_3, add13_3); out_4[i] = __builtin_e2k_pfsubs(sub02_0, sub13i_0); out_5[i] = __builtin_e2k_pfsubs(sub02_1, sub13i_1); out_6[i] = __builtin_e2k_pfsubs(sub02_2, sub13i_2); out_7[i] = __builtin_e2k_pfsubs(sub02_3, sub13i_3); out_8[i] = __builtin_e2k_pfsubs(add02_0, add13_0); out_9[i] = __builtin_e2k_pfsubs(add02_1, add13_1); out_10[i] = __builtin_e2k_pfsubs(add02_2, add13_2); out_11[i] = __builtin_e2k_pfsubs(add02_3, add13_3); out_12[i] = __builtin_e2k_pfadds(sub02_0, sub13i_0); out_13[i] = __builtin_e2k_pfadds(sub02_1, sub13i_1); out_14[i] = __builtin_e2k_pfadds(sub02_2, sub13i_2); out_15[i] = __builtin_e2k_pfadds(sub02_3, sub13i_3); } }
Основной цикл на ассемблере
.L3676: { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=1, abs=0, disp=16 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=1, abs=2, disp=64 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=1, abs=2, disp=32 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=4, asz=1, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=1, abs=4, disp=96 } { fapb ct=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=1, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=3, asz=1, abs=6, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=15, asz=2, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=1, incr=0, ind=0, asz=2, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=13, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=14, asz=2, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=11, asz=2, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=12, asz=2, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=9, asz=2, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=10, asz=2, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=7, asz=2, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=8, asz=2, abs=24, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=5, asz=2, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=6, asz=2, abs=28, disp=0 } .L2949: { loop_mode pfmul_hadds,0,sm %b[111], %b[62], %b[104], %b[62] pfmuls,1,sm %b[63], %b[57], %b[63] pfmuls,2,sm %b[97], %b[82], %b[97] pshufb,3,sm 0x0, %b[29], %r26, %b[99] } { loop_mode pfsubs,1,sm %b[68], %b[67], %b[103] pfmul_hadds,2,sm %b[100], %b[52], %b[103], %b[100] pshufb,3,sm 0x0, %b[32], %r26, %b[104] xord,4,sm %b[86], %r0, %b[105] xord,5,sm %b[105], %r6, %b[106] } { loop_mode pfmul_hadds,0,sm %b[80], %b[53], %b[98], %b[80] pfsubs,1,sm %b[81], %b[79], %b[93] pfmul_hadds,2,sm %b[93], %b[96], %b[89], %b[89] pshufb,3,sm 0x0, %b[49], %r26, %b[98] pfmuls,5,sm %b[105], %b[102], %b[96] } { loop_mode pfsub_adds,0,sm %b[13], %b[64], %b[106], %b[87] pfsubs,1,sm %b[91], %b[87], %b[91] pfadds,2,sm %b[91], %b[87], %b[105] xord,5,sm %b[21], %r0, %b[108] } { loop_mode pfmul_hadds,0,sm %b[99], %b[82], %b[97], %b[82] pfsubs,1,sm %b[94], %b[84], %b[84] pfadds,2,sm %b[94], %b[84], %b[86] pshufb,3,sm 0x0, %b[48], %r26, %b[94] pshufb,4,sm 0x0, %b[86], %r26, %b[97] xord,5,sm %b[74], %r0, %b[99] } { loop_mode pfsub_rsubs,0,sm %b[13], %b[64], %b[106], %b[64] pfsubs,1,sm %b[69], %b[15], %b[106] pfmuls,3,sm %b[99], %b[77], %b[99] pfmuls,4,sm %b[108], %b[10], %b[1] xord,5,sm %b[83], %r0, %b[108] } { loop_mode pfmul_hadds,0,sm %b[104], %b[60], %b[90], %b[63] pfadds,1,sm %b[81], %b[79], %b[79] pfmul_hadds,2,sm %b[98], %b[57], %b[63], %b[60] pshufb,3,sm 0x0, %b[23], %r26, %b[90] pfmuls,4,sm %b[108], %b[72], %b[81] pfmul_hadds,5,sm %b[97], %b[102], %b[96], %b[13] } { loop_mode pfadd_adds,0,sm %b[107], %b[71], %b[105], %b[75] pfsubs,1,sm %b[85], %b[75], %b[85] pfadds,2,sm %b[85], %b[75], %b[96] xord,5,sm %b[78], %r0, %b[97] } { loop_mode pfmul_hadds,0,sm %b[94], %b[61], %b[88], %b[93] xord,1,sm %b[47], %r0, %b[61] pfadd_rsubs,2,sm %b[101], %b[70], %b[86], %b[94] pshufb,3,sm 0x0, %b[93], %r26, %b[97] xord,4,sm %b[30], %r0, %b[88] pfmuls,5,sm %b[97], %b[92], %b[98] } { loop_mode pfadd_rsubs,0,sm %b[107], %b[71], %b[105], %b[83] pfadds,1,sm %b[69], %b[15], %b[91] pfadds,2,sm %b[62], %b[73], %b[102] pshufb,3,sm 0x0, %b[91], %r26, %b[104] pshufb,4,sm 0x0, %b[83], %r26, %b[105] xord,5,sm %b[109], %r0, %b[69] } { loop_mode pfmul_hadds,0,sm %b[90], %b[12], %b[3], %b[62] pshufb,1,sm 0x0, %b[74], %r26, %b[90] pfadd_adds,2,sm %b[101], %b[70], %b[86], %b[73] pshufb,3,sm 0x0, %b[84], %r26, %b[74] pfsubs,4,sm %b[62], %b[73], %b[84] xord,5,sm %b[76], %r0, %b[86] } { loop_mode pfadd_adds,0,sm %b[41], %b[80], %b[79], %b[68] pfadds,1,sm %b[68], %b[67], %b[78] pfadd_rsubs,2,sm %b[87], %b[89], %b[96], %b[72] pshufb,3,sm 0x0, %b[106], %r26, %b[81] pshufb,4,sm 0x0, %b[78], %r26, %b[105] pfmul_hadds,5,sm %b[105], %b[72], %b[81], %b[67] } { loop_mode pfadd_rsubs,0,sm %b[41], %b[80], %b[79], %b[90] pfmul_hadds,1,sm %b[90], %b[77], %b[99], %b[77] pfadd_adds,2,sm %b[87], %b[89], %b[96], %b[96] pshufb,3,sm 0x0, %b[103], %r26, %b[98] xord,4,sm %b[110], %r0, %b[92] pfmul_hadds,5,sm %b[105], %b[92], %b[98], %b[79] } { loop_mode pfadd_adds,0,sm %b[20], %b[100], %b[91], %b[85] pfmuls,1,sm %b[86], %b[66], %b[86] pfadd_adds,2,sm %b[64], %b[82], %b[102], %b[99] pshufb,3,sm 0x0, %b[85], %r26, %b[105] xord,4,sm %b[28], %r0, %b[103] xord,5,sm %b[104], %r6, %b[104] } { loop_mode pfadd_rsubs,0,sm %b[20], %b[100], %b[91], %b[74] xord,1,sm %b[95], %r0, %b[108] pfadd_rsubs,2,sm %b[64], %b[82], %b[102], %b[84] pshufb,3,sm 0x0, %b[84], %r26, %b[106] xord,4,sm %b[45], %r0, %b[91] xord,5,sm %b[74], %r6, %b[102] } { loop_mode pfadd_rsubs,0,sm %b[56], %b[60], %b[78], %b[97] pfmuls,1,sm %b[108], %b[65], %b[111] pfsub_adds,2,sm %b[107], %b[71], %b[104], %b[108] xord,3,sm %b[46], %r0, %b[81] xord,4,sm %b[97], %r6, %b[112] xord,5,sm %b[81], %r6, %b[113] } { loop_mode pfadd_adds,0,sm %b[56], %b[60], %b[78], %b[109] pshufb,1,sm 0x0, %b[110], %r26, %b[78] pfsub_rsubs,2,sm %b[101], %b[70], %b[102], %b[110] pshufb,3,sm 0x0, %b[109], %r26, %b[98] xord,4,sm %b[98], %r6, %b[114] addd,5,sm 0x8, %b[8], %b[6] ? %pcnt0 } { loop_mode pfsub_rsubs,0,sm %b[20], %b[100], %b[113], %b[71] pfsubs,1,sm %b[93], %b[63], %b[75] pfsub_rsubs,2,sm %b[107], %b[71], %b[104], %b[76] pshufb,3,sm 0x0, %b[76], %r26, %b[104] xord,4,sm %b[105], %r6, %b[105] std,5 %r25, %b[8], %b[75] } { loop_mode pfsub_adds,0,sm %b[20], %b[100], %b[113], %b[63] pfadds,1,sm %b[93], %b[63], %b[93] pfsub_adds,2,sm %b[101], %b[70], %b[102], %b[70] pshufb,3,sm 0x0, %b[95], %r26, %b[94] xord,4,sm %b[106], %r6, %b[95] std,5 %r23, %b[8], %b[94] } { loop_mode pfsub_adds,0,sm %b[56], %b[60], %b[114], %b[83] pfmuls,1,sm %b[91], %b[68], %b[100] pfsub_rsubs,2,sm %b[87], %b[89], %b[105], %b[101] pshufb,3,sm 0x0, %b[36], %r26, %b[91] xord,4,sm %b[44], %r0, %b[102] std,5 %r18, %b[8], %b[83] } { loop_mode pfsub_rsubs,0,sm %b[56], %b[60], %b[114], %b[60] pfmuls,1,sm %b[103], %b[90], %b[103] pfsub_adds,2,sm %b[64], %b[82], %b[95], %b[73] pshufb,3,sm 0x0, %b[45], %r26, %b[106] xord,4,sm %b[24], %r0, %b[107] std,5 %r2, %b[8], %b[73] } { loop_mode pfsub_adds,0,sm %b[87], %b[89], %b[105], %b[64] pfmuls,1,sm %b[102], %b[85], %b[82] pfsub_rsubs,2,sm %b[64], %b[82], %b[95], %b[72] pshufb,3,sm 0x0, %b[44], %r26, %b[87] xord,4,sm %b[33], %r0, %b[89] std,5 %r12, %b[8], %b[72] } { loop_mode pfmul_hadds,0,sm %b[94], %b[65], %b[111], %b[65] pfmuls,1,sm %b[107], %b[74], %b[95] pfsub_adds,2,sm %b[41], %b[80], %b[112], %b[94] pshufb,3,sm 0x0, %b[24], %r26, %b[96] xord,4,sm %b[40], %r0, %b[102] std,5 %r16, %b[8], %b[96] movad,0 area=9, ind=0, am=1, be=0, %b[15] movad,1 area=8, ind=0, am=1, be=0, %b[12] movad,2 area=9, ind=0, am=1, be=0, %b[3] movad,3 area=8, ind=0, am=1, be=0, %b[20] } { loop_mode pfmul_hadds,0,sm %b[104], %b[66], %b[86], %b[66] pfmuls,1,sm %b[89], %b[97], %b[86] pshufb,3,sm 0x0, %b[33], %r26, %b[89] xord,4,sm %b[36], %r0, %b[99] std,5 %r22, %b[8], %b[99] movad,0 area=7, ind=0, am=1, be=0, %b[24] movad,1 area=6, ind=0, am=1, be=0, %b[32] movad,2 area=7, ind=0, am=1, be=0, %b[29] movad,3 area=6, ind=0, am=1, be=0, %b[23] } { loop_mode pfsub_rsubs,0,sm %b[41], %b[80], %b[112], %b[80] pfmuls,1,sm %b[102], %b[109], %b[84] pshufb,3,sm 0x0, %b[40], %r26, %b[102] xord,4,sm %b[19], %r0, %b[104] std,5 %r19, %b[8], %b[84] movad,0 area=5, ind=0, am=1, be=0, %b[33] movad,1 area=4, ind=0, am=1, be=0, %b[41] movad,2 area=5, ind=0, am=1, be=0, %b[40] movad,3 area=4, ind=0, am=1, be=0, %b[36] } { loop_mode pfadd_adds,0,sm %b[11], %b[62], %b[93], %b[99] pfmuls,1,sm %b[99], %b[71], %b[111] pfadd_rsubs,2,sm %b[11], %b[62], %b[93], %b[105] pshufb,3,sm 0x0, %b[28], %r26, %b[112] xord,4,sm %b[16], %r0, %b[113] std,5 %r14, %b[8], %b[108] movad,0 area=3, ind=8, am=1, be=0, %b[93] movad,1 area=3, ind=0, am=0, be=0, %b[28] movad,2 area=3, ind=16, am=0, be=0, %b[108] movad,3 area=3, ind=24, am=0, be=0, %b[107] } { loop_mode pfmul_hadds,0,sm %b[87], %b[85], %b[82], %b[82] pfmuls,1,sm %b[104], %b[63], %b[87] pfmul_hadds,2,sm %b[96], %b[74], %b[95], %b[85] pshufb,3,sm 0x0, %b[19], %r26, %b[95] xord,4,sm %b[37], %r0, %b[96] std,5 %r24, %b[8], %b[110] movad,0 area=2, ind=0, am=0, be=0, %b[44] movad,1 area=2, ind=8, am=0, be=0, %b[74] movad,2 area=3, ind=8, am=1, be=0, %b[45] movad,3 area=3, ind=0, am=0, be=0, %b[19] } { loop_mode pfmul_hadds,0,sm %b[89], %b[97], %b[86], %b[89] pfmuls,1,sm %b[81], %b[59], %b[86] pfmuls,2,sm %b[113], %b[83], %b[97] pshufb,3,sm 0x0, %b[16], %r26, %b[104] std,5 %r15, %b[8], %b[76] movad,0 area=2, ind=16, am=0, be=0, %b[76] movad,1 area=2, ind=24, am=1, be=0, %b[81] movad,2 area=2, ind=0, am=0, be=0, %b[16] movad,3 area=2, ind=16, am=0, be=0, %b[48] } { loop_mode pfmul_hadds,0,sm %b[102], %b[109], %b[84], %b[92] pfmuls,1,sm %b[92], %b[51], %b[96] pfmuls,2,sm %b[96], %b[60], %b[102] pshufb,3,sm 0x0, %b[37], %r26, %b[109] xord,4,sm %b[7], %r0, %b[110] std,5 %r17, %b[8], %b[70] movad,0 area=1, ind=0, am=0, be=0, %b[37] movad,1 area=1, ind=16, am=0, be=0, %b[49] movad,2 area=2, ind=8, am=0, be=0, %b[70] movad,3 area=0, ind=8, am=0, be=0, %b[84] } { loop_mode pfmul_hadds,0,sm %b[106], %b[68], %b[100], %b[68] pfmuls,1,sm %b[69], %b[50], %b[101] pfmul_hadds,2,sm %b[112], %b[90], %b[103], %b[69] pshufb,3,sm 0x0, %b[75], %r26, %b[103] std,5 %r20, %b[8], %b[101] movad,0 area=1, ind=8, am=1, be=0, %b[90] movad,1 area=1, ind=24, am=0, be=0, %b[75] movad,2 area=2, ind=24, am=1, be=0, %b[100] movad,3 area=1, ind=0, am=0, be=0, %b[52] } { loop_mode pfmul_hadds,0,sm %b[91], %b[71], %b[111], %b[71] pfmuls,1,sm %b[110], %b[94], %b[87] pfmul_hadds,2,sm %b[95], %b[63], %b[87], %b[73] pshufb,3,sm 0x0, %b[7], %r26, %b[91] xord,4,sm %b[27], %r0, %b[95] std,5 %r11, %b[8], %b[73] movad,0 area=0, ind=0, am=0, be=0, %b[7] movad,1 area=0, ind=24, am=0, be=0, %b[56] movad,2 area=1, ind=16, am=0, be=0, %b[53] movad,3 area=1, ind=24, am=0, be=0, %b[63] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmul_hadds,0,sm %b[104], %b[83], %b[97], %b[83] pfmuls,1,sm %b[88], %b[58], %b[88] std,2 %r13, %b[8], %b[64] std,5 %r21, %b[8], %b[72] movad,0 area=0, ind=8, am=1, be=0, %b[57] movad,1 area=0, ind=16, am=0, be=0, %b[8] movad,2 area=1, ind=8, am=1, be=0, %b[64] movad,3 area=0, ind=0, am=1, be=0, %b[72] }
Теоретическая скорость: 16 комплексных чисел за 32 такта (16/32) = 4 Байт/такт
Четверная теоретическая скорость: 16 Байт/такт
Замеры скорости
3. stage_radix4_2x_simd128
Здесь происходит ручная раскрутка алгоритма stage_radix4_simd128 в 2 раза.
Код на Си
void stage_radix4_2x_simd128(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coefC_a, myComplex *coefD_a, myComplex *coefE_a, myComplex *coefC_b, myComplex *coefD_b, myComplex *coefE_b) { __v2di *xy0_in = (__v2di*)&data_in[ 0]; __v2di *zw0_in = (__v2di*)&data_in[ 2]; __v2di *xy1_in = (__v2di*)&data_in[ 4]; __v2di *zw1_in = (__v2di*)&data_in[ 6]; __v2di *xy2_in = (__v2di*)&data_in[ 8]; __v2di *zw2_in = (__v2di*)&data_in[10]; __v2di *xy3_in = (__v2di*)&data_in[12]; __v2di *zw3_in = (__v2di*)&data_in[14]; __v2di *xy4_in = (__v2di*)&data_in[16]; __v2di *zw4_in = (__v2di*)&data_in[18]; __v2di *xy5_in = (__v2di*)&data_in[20]; __v2di *zw5_in = (__v2di*)&data_in[22]; __v2di *xy6_in = (__v2di*)&data_in[24]; __v2di *zw6_in = (__v2di*)&data_in[26]; __v2di *xy7_in = (__v2di*)&data_in[28]; __v2di *zw7_in = (__v2di*)&data_in[30]; __v2di *c0a_in = (__v2di*)&coefC_a[0]; __v2di *c1a_in = (__v2di*)&coefC_a[2]; __v2di *c2a_in = (__v2di*)&coefC_a[4]; __v2di *c3a_in = (__v2di*)&coefC_a[6]; __v2di *d0a_in = (__v2di*)&coefD_a[0]; __v2di *d1a_in = (__v2di*)&coefD_a[2]; __v2di *d2a_in = (__v2di*)&coefD_a[4]; __v2di *d3a_in = (__v2di*)&coefD_a[6]; __v2di *e0a_in = (__v2di*)&coefE_a[0]; __v2di *e1a_in = (__v2di*)&coefE_a[2]; __v2di *e2a_in = (__v2di*)&coefE_a[4]; __v2di *e3a_in = (__v2di*)&coefE_a[6]; __v2di *c0b_in = (__v2di*)&coefC_b[0*data_count/16]; __v2di *c1b_in = (__v2di*)&coefC_b[1*data_count/16]; __v2di *c2b_in = (__v2di*)&coefC_b[2*data_count/16]; __v2di *c3b_in = (__v2di*)&coefC_b[3*data_count/16]; __v2di *d0b_in = (__v2di*)&coefD_b[0*data_count/16]; __v2di *d1b_in = (__v2di*)&coefD_b[1*data_count/16]; __v2di *d2b_in = (__v2di*)&coefD_b[2*data_count/16]; __v2di *d3b_in = (__v2di*)&coefD_b[3*data_count/16]; __v2di *e0b_in = (__v2di*)&coefE_b[0*data_count/16]; __v2di *e1b_in = (__v2di*)&coefE_b[1*data_count/16]; __v2di *e2b_in = (__v2di*)&coefE_b[2*data_count/16]; __v2di *e3b_in = (__v2di*)&coefE_b[3*data_count/16]; __v2di *out_0 = (__v2di*)&data_out[ 0*data_count/16]; __v2di *out_1 = (__v2di*)&data_out[ 1*data_count/16]; __v2di *out_2 = (__v2di*)&data_out[ 2*data_count/16]; __v2di *out_3 = (__v2di*)&data_out[ 3*data_count/16]; __v2di *out_4 = (__v2di*)&data_out[ 4*data_count/16]; __v2di *out_5 = (__v2di*)&data_out[ 5*data_count/16]; __v2di *out_6 = (__v2di*)&data_out[ 6*data_count/16]; __v2di *out_7 = (__v2di*)&data_out[ 7*data_count/16]; __v2di *out_8 = (__v2di*)&data_out[ 8*data_count/16]; __v2di *out_9 = (__v2di*)&data_out[ 9*data_count/16]; __v2di *out_10 = (__v2di*)&data_out[10*data_count/16]; __v2di *out_11 = (__v2di*)&data_out[11*data_count/16]; __v2di *out_12 = (__v2di*)&data_out[12*data_count/16]; __v2di *out_13 = (__v2di*)&data_out[13*data_count/16]; __v2di *out_14 = (__v2di*)&data_out[14*data_count/16]; __v2di *out_15 = (__v2di*)&data_out[15*data_count/16]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/32; ++i) { __v2di xy0 = xy0_in[16*i]; __v2di zw0 = zw0_in[16*i]; __v2di xy1 = xy1_in[16*i]; __v2di zw1 = zw1_in[16*i]; __v2di c0 = c0a_in[4*i]; __v2di d0 = d0a_in[4*i]; __v2di e0 = e0a_in[4*i]; __v2di xy2 = xy2_in[16*i]; __v2di zw2 = zw2_in[16*i]; __v2di xy3 = xy3_in[16*i]; __v2di zw3 = zw3_in[16*i]; __v2di c1 = c1a_in[4*i]; __v2di d1 = d1a_in[4*i]; __v2di e1 = e1a_in[4*i]; __v2di xy4 = xy4_in[16*i]; __v2di zw4 = zw4_in[16*i]; __v2di xy5 = xy5_in[16*i]; __v2di zw5 = zw5_in[16*i]; __v2di c2 = c2a_in[4*i]; __v2di d2 = d2a_in[4*i]; __v2di e2 = e2a_in[4*i]; __v2di xy6 = xy6_in[16*i]; __v2di zw6 = zw6_in[16*i]; __v2di xy7 = xy7_in[16*i]; __v2di zw7 = zw7_in[16*i]; __v2di c3 = c3a_in[4*i]; __v2di d3 = d3a_in[4*i]; __v2di e3 = e3a_in[4*i]; __v2di x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di conj_c0 = __builtin_e2k_qpxor(c0, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_c1 = __builtin_e2k_qpxor(c1, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_c2 = __builtin_e2k_qpxor(c2, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_c3 = __builtin_e2k_qpxor(c3, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_d0 = __builtin_e2k_qpxor(d0, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_d1 = __builtin_e2k_qpxor(d1, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_d2 = __builtin_e2k_qpxor(d2, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_d3 = __builtin_e2k_qpxor(d3, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_e0 = __builtin_e2k_qpxor(e0, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_e1 = __builtin_e2k_qpxor(e1, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_e2 = __builtin_e2k_qpxor(e2, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_e3 = __builtin_e2k_qpxor(e3, (__v2di){1LL<<63, 1LL<<63}); __v2di swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c2 = __builtin_e2k_qpshufb(c2, c2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c3 = __builtin_e2k_qpshufb(c3, c3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d0 = __builtin_e2k_qpshufb(d0, d0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d1 = __builtin_e2k_qpshufb(d1, d1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d2 = __builtin_e2k_qpshufb(d2, d2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d3 = __builtin_e2k_qpshufb(d3, d3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e0 = __builtin_e2k_qpshufb(e0, e0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e1 = __builtin_e2k_qpshufb(e1, e1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e2 = __builtin_e2k_qpshufb(e2, e2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e3 = __builtin_e2k_qpshufb(e3, e3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy0_real = __builtin_e2k_qpfmuls(conj_c0, y0); __v2di cy1_real = __builtin_e2k_qpfmuls(conj_c1, y1); __v2di cy2_real = __builtin_e2k_qpfmuls(conj_c2, y2); __v2di cy3_real = __builtin_e2k_qpfmuls(conj_c3, y3); __v2di dz0_real = __builtin_e2k_qpfmuls(conj_d0, z0); __v2di dz1_real = __builtin_e2k_qpfmuls(conj_d1, z1); __v2di dz2_real = __builtin_e2k_qpfmuls(conj_d2, z2); __v2di dz3_real = __builtin_e2k_qpfmuls(conj_d3, z3); __v2di ew0_real = __builtin_e2k_qpfmuls(conj_e0, w0); __v2di ew1_real = __builtin_e2k_qpfmuls(conj_e1, w1); __v2di ew2_real = __builtin_e2k_qpfmuls(conj_e2, w2); __v2di ew3_real = __builtin_e2k_qpfmuls(conj_e3, w3); __v2di cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); __v2di cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); __v2di cy2_imag = __builtin_e2k_qpfmuls(swap_c2, y2); __v2di cy3_imag = __builtin_e2k_qpfmuls(swap_c3, y3); __v2di dz0_imag = __builtin_e2k_qpfmuls(swap_d0, z0); __v2di dz1_imag = __builtin_e2k_qpfmuls(swap_d1, z1); __v2di dz2_imag = __builtin_e2k_qpfmuls(swap_d2, z2); __v2di dz3_imag = __builtin_e2k_qpfmuls(swap_d3, z3); __v2di ew0_imag = __builtin_e2k_qpfmuls(swap_e0, w0); __v2di ew1_imag = __builtin_e2k_qpfmuls(swap_e1, w1); __v2di ew2_imag = __builtin_e2k_qpfmuls(swap_e2, w2); __v2di ew3_imag = __builtin_e2k_qpfmuls(swap_e3, w3); __v2di cy0_rrii = __builtin_e2k_qpfhadds(cy0_real, cy0_imag); __v2di cy1_rrii = __builtin_e2k_qpfhadds(cy1_real, cy1_imag); __v2di cy2_rrii = __builtin_e2k_qpfhadds(cy2_real, cy2_imag); __v2di cy3_rrii = __builtin_e2k_qpfhadds(cy3_real, cy3_imag); __v2di dz0_rrii = __builtin_e2k_qpfhadds(dz0_real, dz0_imag); __v2di dz1_rrii = __builtin_e2k_qpfhadds(dz1_real, dz1_imag); __v2di dz2_rrii = __builtin_e2k_qpfhadds(dz2_real, dz2_imag); __v2di dz3_rrii = __builtin_e2k_qpfhadds(dz3_real, dz3_imag); __v2di ew0_rrii = __builtin_e2k_qpfhadds(ew0_real, ew0_imag); __v2di ew1_rrii = __builtin_e2k_qpfhadds(ew1_real, ew1_imag); __v2di ew2_rrii = __builtin_e2k_qpfhadds(ew2_real, ew2_imag); __v2di ew3_rrii = __builtin_e2k_qpfhadds(ew3_real, ew3_imag); __v2di cy0 = __builtin_e2k_qpshufb(cy0_rrii, cy0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di cy1 = __builtin_e2k_qpshufb(cy1_rrii, cy1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di cy2 = __builtin_e2k_qpshufb(cy2_rrii, cy2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di cy3 = __builtin_e2k_qpshufb(cy3_rrii, cy3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di dz0 = __builtin_e2k_qpshufb(dz0_rrii, dz0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di dz1 = __builtin_e2k_qpshufb(dz1_rrii, dz1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di dz2 = __builtin_e2k_qpshufb(dz2_rrii, dz2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di dz3 = __builtin_e2k_qpshufb(dz3_rrii, dz3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di ew0 = __builtin_e2k_qpshufb(ew0_rrii, ew0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di ew1 = __builtin_e2k_qpshufb(ew1_rrii, ew1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di ew2 = __builtin_e2k_qpshufb(ew2_rrii, ew2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di ew3 = __builtin_e2k_qpshufb(ew3_rrii, ew3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di add02_0 = __builtin_e2k_qpfadds( x0, dz0); __v2di add02_1 = __builtin_e2k_qpfadds( x1, dz1); __v2di add02_2 = __builtin_e2k_qpfadds( x2, dz2); __v2di add02_3 = __builtin_e2k_qpfadds( x3, dz3); __v2di sub02_0 = __builtin_e2k_qpfsubs( x0, dz0); __v2di sub02_1 = __builtin_e2k_qpfsubs( x1, dz1); __v2di sub02_2 = __builtin_e2k_qpfsubs( x2, dz2); __v2di sub02_3 = __builtin_e2k_qpfsubs( x3, dz3); __v2di add13_0 = __builtin_e2k_qpfadds(cy0, ew0); __v2di add13_1 = __builtin_e2k_qpfadds(cy1, ew1); __v2di add13_2 = __builtin_e2k_qpfadds(cy2, ew2); __v2di add13_3 = __builtin_e2k_qpfadds(cy3, ew3); __v2di sub13_0 = __builtin_e2k_qpfsubs(cy0, ew0); __v2di sub13_1 = __builtin_e2k_qpfsubs(cy1, ew1); __v2di sub13_2 = __builtin_e2k_qpfsubs(cy2, ew2); __v2di sub13_3 = __builtin_e2k_qpfsubs(cy3, ew3); __v2di swap_sub13_0 = __builtin_e2k_qpshufb(sub13_0, sub13_0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_1 = __builtin_e2k_qpshufb(sub13_1, sub13_1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_2 = __builtin_e2k_qpshufb(sub13_2, sub13_2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_3 = __builtin_e2k_qpshufb(sub13_3, sub13_3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di sub13i_0 = __builtin_e2k_qpxor(swap_sub13_0, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_1 = __builtin_e2k_qpxor(swap_sub13_1, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_2 = __builtin_e2k_qpxor(swap_sub13_2, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_3 = __builtin_e2k_qpxor(swap_sub13_3, (__v2di){1LL<<31, 1LL<<31}); __v2di out0 = __builtin_e2k_qpfadds(add02_0, add13_0); __v2di out1 = __builtin_e2k_qpfadds(add02_1, add13_1); __v2di out2 = __builtin_e2k_qpfadds(add02_2, add13_2); __v2di out3 = __builtin_e2k_qpfadds(add02_3, add13_3); __v2di out4 = __builtin_e2k_qpfsubs(sub02_0, sub13i_0); __v2di out5 = __builtin_e2k_qpfsubs(sub02_1, sub13i_1); __v2di out6 = __builtin_e2k_qpfsubs(sub02_2, sub13i_2); __v2di out7 = __builtin_e2k_qpfsubs(sub02_3, sub13i_3); __v2di out8 = __builtin_e2k_qpfsubs(add02_0, add13_0); __v2di out9 = __builtin_e2k_qpfsubs(add02_1, add13_1); __v2di out10 = __builtin_e2k_qpfsubs(add02_2, add13_2); __v2di out11 = __builtin_e2k_qpfsubs(add02_3, add13_3); __v2di out12 = __builtin_e2k_qpfadds(sub02_0, sub13i_0); __v2di out13 = __builtin_e2k_qpfadds(sub02_1, sub13i_1); __v2di out14 = __builtin_e2k_qpfadds(sub02_2, sub13i_2); __v2di out15 = __builtin_e2k_qpfadds(sub02_3, sub13i_3); xy0 = out0; zw0 = out1; xy1 = out2; zw1 = out3; c0 = c0b_in[i]; d0 = d0b_in[i]; e0 = e0b_in[i]; xy2 = out4; zw2 = out5; xy3 = out6; zw3 = out7; c1 = c1b_in[i]; d1 = d1b_in[i]; e1 = e1b_in[i]; xy4 = out8; zw4 = out9; xy5 = out10; zw5 = out11; c2 = c2b_in[i]; d2 = d2b_in[i]; e2 = e2b_in[i]; xy6 = out12; zw6 = out13; xy7 = out14; zw7 = out15; c3 = c3b_in[i]; d3 = d3b_in[i]; e3 = e3b_in[i]; x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); w0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0706050403020100, 0x0706050403020100}); w1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0706050403020100, 0x0706050403020100}); y2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0706050403020100, 0x0706050403020100}); w2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0706050403020100, 0x0706050403020100}); y3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0706050403020100, 0x0706050403020100}); w3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); conj_c0 = __builtin_e2k_qpxor(c0, (__v2di){1LL<<63, 1LL<<63}); conj_c1 = __builtin_e2k_qpxor(c1, (__v2di){1LL<<63, 1LL<<63}); conj_c2 = __builtin_e2k_qpxor(c2, (__v2di){1LL<<63, 1LL<<63}); conj_c3 = __builtin_e2k_qpxor(c3, (__v2di){1LL<<63, 1LL<<63}); conj_d0 = __builtin_e2k_qpxor(d0, (__v2di){1LL<<63, 1LL<<63}); conj_d1 = __builtin_e2k_qpxor(d1, (__v2di){1LL<<63, 1LL<<63}); conj_d2 = __builtin_e2k_qpxor(d2, (__v2di){1LL<<63, 1LL<<63}); conj_d3 = __builtin_e2k_qpxor(d3, (__v2di){1LL<<63, 1LL<<63}); conj_e0 = __builtin_e2k_qpxor(e0, (__v2di){1LL<<63, 1LL<<63}); conj_e1 = __builtin_e2k_qpxor(e1, (__v2di){1LL<<63, 1LL<<63}); conj_e2 = __builtin_e2k_qpxor(e2, (__v2di){1LL<<63, 1LL<<63}); conj_e3 = __builtin_e2k_qpxor(e3, (__v2di){1LL<<63, 1LL<<63}); swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c2 = __builtin_e2k_qpshufb(c2, c2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c3 = __builtin_e2k_qpshufb(c3, c3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d0 = __builtin_e2k_qpshufb(d0, d0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d1 = __builtin_e2k_qpshufb(d1, d1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d2 = __builtin_e2k_qpshufb(d2, d2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d3 = __builtin_e2k_qpshufb(d3, d3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e0 = __builtin_e2k_qpshufb(e0, e0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e1 = __builtin_e2k_qpshufb(e1, e1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e2 = __builtin_e2k_qpshufb(e2, e2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e3 = __builtin_e2k_qpshufb(e3, e3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); cy0_real = __builtin_e2k_qpfmuls(conj_c0, y0); cy1_real = __builtin_e2k_qpfmuls(conj_c1, y1); cy2_real = __builtin_e2k_qpfmuls(conj_c2, y2); cy3_real = __builtin_e2k_qpfmuls(conj_c3, y3); dz0_real = __builtin_e2k_qpfmuls(conj_d0, z0); dz1_real = __builtin_e2k_qpfmuls(conj_d1, z1); dz2_real = __builtin_e2k_qpfmuls(conj_d2, z2); dz3_real = __builtin_e2k_qpfmuls(conj_d3, z3); ew0_real = __builtin_e2k_qpfmuls(conj_e0, w0); ew1_real = __builtin_e2k_qpfmuls(conj_e1, w1); ew2_real = __builtin_e2k_qpfmuls(conj_e2, w2); ew3_real = __builtin_e2k_qpfmuls(conj_e3, w3); cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); cy2_imag = __builtin_e2k_qpfmuls(swap_c2, y2); cy3_imag = __builtin_e2k_qpfmuls(swap_c3, y3); dz0_imag = __builtin_e2k_qpfmuls(swap_d0, z0); dz1_imag = __builtin_e2k_qpfmuls(swap_d1, z1); dz2_imag = __builtin_e2k_qpfmuls(swap_d2, z2); dz3_imag = __builtin_e2k_qpfmuls(swap_d3, z3); ew0_imag = __builtin_e2k_qpfmuls(swap_e0, w0); ew1_imag = __builtin_e2k_qpfmuls(swap_e1, w1); ew2_imag = __builtin_e2k_qpfmuls(swap_e2, w2); ew3_imag = __builtin_e2k_qpfmuls(swap_e3, w3); cy0_rrii = __builtin_e2k_qpfhadds(cy0_real, cy0_imag); cy1_rrii = __builtin_e2k_qpfhadds(cy1_real, cy1_imag); cy2_rrii = __builtin_e2k_qpfhadds(cy2_real, cy2_imag); cy3_rrii = __builtin_e2k_qpfhadds(cy3_real, cy3_imag); dz0_rrii = __builtin_e2k_qpfhadds(dz0_real, dz0_imag); dz1_rrii = __builtin_e2k_qpfhadds(dz1_real, dz1_imag); dz2_rrii = __builtin_e2k_qpfhadds(dz2_real, dz2_imag); dz3_rrii = __builtin_e2k_qpfhadds(dz3_real, dz3_imag); ew0_rrii = __builtin_e2k_qpfhadds(ew0_real, ew0_imag); ew1_rrii = __builtin_e2k_qpfhadds(ew1_real, ew1_imag); ew2_rrii = __builtin_e2k_qpfhadds(ew2_real, ew2_imag); ew3_rrii = __builtin_e2k_qpfhadds(ew3_real, ew3_imag); cy0 = __builtin_e2k_qpshufb(cy0_rrii, cy0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); cy1 = __builtin_e2k_qpshufb(cy1_rrii, cy1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); cy2 = __builtin_e2k_qpshufb(cy2_rrii, cy2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); cy3 = __builtin_e2k_qpshufb(cy3_rrii, cy3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); dz0 = __builtin_e2k_qpshufb(dz0_rrii, dz0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); dz1 = __builtin_e2k_qpshufb(dz1_rrii, dz1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); dz2 = __builtin_e2k_qpshufb(dz2_rrii, dz2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); dz3 = __builtin_e2k_qpshufb(dz3_rrii, dz3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); ew0 = __builtin_e2k_qpshufb(ew0_rrii, ew0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); ew1 = __builtin_e2k_qpshufb(ew1_rrii, ew1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); ew2 = __builtin_e2k_qpshufb(ew2_rrii, ew2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); ew3 = __builtin_e2k_qpshufb(ew3_rrii, ew3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); add02_0 = __builtin_e2k_qpfadds( x0, dz0); add02_1 = __builtin_e2k_qpfadds( x1, dz1); add02_2 = __builtin_e2k_qpfadds( x2, dz2); add02_3 = __builtin_e2k_qpfadds( x3, dz3); sub02_0 = __builtin_e2k_qpfsubs( x0, dz0); sub02_1 = __builtin_e2k_qpfsubs( x1, dz1); sub02_2 = __builtin_e2k_qpfsubs( x2, dz2); sub02_3 = __builtin_e2k_qpfsubs( x3, dz3); add13_0 = __builtin_e2k_qpfadds(cy0, ew0); add13_1 = __builtin_e2k_qpfadds(cy1, ew1); add13_2 = __builtin_e2k_qpfadds(cy2, ew2); add13_3 = __builtin_e2k_qpfadds(cy3, ew3); sub13_0 = __builtin_e2k_qpfsubs(cy0, ew0); sub13_1 = __builtin_e2k_qpfsubs(cy1, ew1); sub13_2 = __builtin_e2k_qpfsubs(cy2, ew2); sub13_3 = __builtin_e2k_qpfsubs(cy3, ew3); swap_sub13_0 = __builtin_e2k_qpshufb(sub13_0, sub13_0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_1 = __builtin_e2k_qpshufb(sub13_1, sub13_1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_2 = __builtin_e2k_qpshufb(sub13_2, sub13_2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_3 = __builtin_e2k_qpshufb(sub13_3, sub13_3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); sub13i_0 = __builtin_e2k_qpxor(swap_sub13_0, (__v2di){1LL<<31, 1LL<<31}); sub13i_1 = __builtin_e2k_qpxor(swap_sub13_1, (__v2di){1LL<<31, 1LL<<31}); sub13i_2 = __builtin_e2k_qpxor(swap_sub13_2, (__v2di){1LL<<31, 1LL<<31}); sub13i_3 = __builtin_e2k_qpxor(swap_sub13_3, (__v2di){1LL<<31, 1LL<<31}); out_0[i] = __builtin_e2k_qpfadds(add02_0, add13_0); out_1[i] = __builtin_e2k_qpfadds(add02_1, add13_1); out_2[i] = __builtin_e2k_qpfadds(add02_2, add13_2); out_3[i] = __builtin_e2k_qpfadds(add02_3, add13_3); out_4[i] = __builtin_e2k_qpfsubs(sub02_0, sub13i_0); out_5[i] = __builtin_e2k_qpfsubs(sub02_1, sub13i_1); out_6[i] = __builtin_e2k_qpfsubs(sub02_2, sub13i_2); out_7[i] = __builtin_e2k_qpfsubs(sub02_3, sub13i_3); out_8[i] = __builtin_e2k_qpfsubs(add02_0, add13_0); out_9[i] = __builtin_e2k_qpfsubs(add02_1, add13_1); out_10[i] = __builtin_e2k_qpfsubs(add02_2, add13_2); out_11[i] = __builtin_e2k_qpfsubs(add02_3, add13_3); out_12[i] = __builtin_e2k_qpfadds(sub02_0, sub13i_0); out_13[i] = __builtin_e2k_qpfadds(sub02_1, sub13i_1); out_14[i] = __builtin_e2k_qpfadds(sub02_2, sub13i_2); out_15[i] = __builtin_e2k_qpfadds(sub02_3, sub13i_3); } }
Основной цикл на ассемблере
.L7295: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=2, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=2, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=4, disp=128 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=4, disp=160 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=6, disp=192 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=6, disp=224 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=1, abs=8, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=1, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=1, abs=10, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=1, abs=12, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=2, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=2, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=2, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=2, abs=24, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=2, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=2, abs=28, disp=0 } .L3969: { loop_mode disp %ctpr1, .L3969 movaqp,0 area=0, ind=0, am=1, be=0, %g17 movaqp,1 area=0, ind=16, am=0, be=0, %g16 movaqp,2 area=0, ind=0, am=1, be=0, %g19 movaqp,3 area=0, ind=16, am=0, be=0, %g18 } { loop_mode movaqp,0 area=1, ind=0, am=1, be=0, %g21 movaqp,1 area=1, ind=16, am=0, be=0, %g20 movaqp,2 area=1, ind=0, am=1, be=0, %g23 movaqp,3 area=1, ind=16, am=0, be=0, %g22 } { loop_mode movaqp,0 area=2, ind=0, am=1, be=0, %g25 movaqp,1 area=2, ind=16, am=0, be=0, %g24 movaqp,2 area=2, ind=0, am=1, be=0, %g27 movaqp,3 area=2, ind=16, am=0, be=0, %g26 } { loop_mode movaqp,0 area=3, ind=0, am=1, be=0, %g29 movaqp,1 area=3, ind=16, am=0, be=0, %g28 movaqp,2 area=3, ind=0, am=1, be=0, %g31 movaqp,3 area=3, ind=16, am=0, be=0, %g30 } { loop_mode movaqp,0 area=4, ind=0, am=1, be=0, %r26 movaqp,1 area=4, ind=16, am=0, be=0, %r9 movaqp,2 area=4, ind=0, am=1, be=0, %r28 movaqp,3 area=4, ind=16, am=0, be=0, %r27 } { loop_mode qpshufb,0 %g19, %g17, %r1, %r33 qpshufb,1 %g18, %g16, %r1, %r34 qpshufb,3 %g18, %g16, %r7, %g16 qpshufb,4 %g19, %g17, %r7, %g17 movaqp,0 area=5, ind=0, am=1, be=0, %r30 movaqp,1 area=5, ind=16, am=0, be=0, %r29 movaqp,2 area=5, ind=0, am=1, be=0, %r32 movaqp,3 area=5, ind=16, am=0, be=0, %r31 } { loop_mode qpshufb,0 %g23, %g21, %r1, %r37 qpshufb,1 %g22, %g20, %r1, %r38 qpshufb,3 %g22, %g20, %r7, %g20 qpshufb,4 %g23, %g21, %r7, %g21 movaqp,0 area=6, ind=0, am=1, be=0, %g19 movaqp,1 area=6, ind=16, am=0, be=0, %g18 movaqp,2 area=6, ind=0, am=1, be=0, %r36 movaqp,3 area=6, ind=16, am=0, be=0, %r35 } { loop_mode qpshufb,0 %g27, %g25, %r1, %g22 qpshufb,1 %g26, %g24, %r1, %g23 qpshufb,3 %g26, %g24, %r7, %g24 qpshufb,4 %g27, %g25, %r7, %g25 movaqp,0 area=8, ind=0, am=1, be=0, %r39 movaqp,1 area=7, ind=0, am=1, be=0, %g26 movaqp,2 area=8, ind=0, am=1, be=0, %r40 movaqp,3 area=7, ind=0, am=1, be=0, %g27 } { loop_mode qpshufb,0 %g31, %g29, %r1, %r41 qpshufb,1 %g30, %g28, %r1, %r42 qpshufb,3 %g30, %g28, %r7, %g28 qpshufb,4 %g31, %g29, %r7, %g29 movaqp,0 area=10, ind=0, am=1, be=0, %r43 movaqp,1 area=9, ind=0, am=1, be=0, %g30 movaqp,2 area=10, ind=0, am=1, be=0, %r44 movaqp,3 area=9, ind=0, am=1, be=0, %g31 } { loop_mode qpxor,0 %r26, %r5, %r45 qpxor,1 %r9, %r5, %r46 qpshufb,3 %r26, %r26, %r6, %r26 qpshufb,4 %r9, %r9, %r6, %r9 movaqp,0 area=12, ind=0, am=1, be=0, %r49 movaqp,1 area=11, ind=0, am=1, be=0, %r47 movaqp,2 area=12, ind=0, am=1, be=0, %r50 movaqp,3 area=11, ind=0, am=1, be=0, %r48 } { loop_mode qpxor,0 %r28, %r5, %r51 qpxor,1 %r27, %r5, %r52 qpfmuls,2 %r45, %r33, %r33 qpshufb,3 %r28, %r28, %r6, %r28 qpshufb,4 %r27, %r27, %r6, %r27 qpfmuls,5 %r26, %r33, %r26 } { loop_mode qpxor,0 %g19, %r5, %r45 qpxor,1 %g18, %r5, %r53 qpfmuls,2 %r46, %r37, %r37 qpshufb,3 %g19, %g19, %r6, %g19 qpshufb,4 %g18, %g18, %r6, %g18 qpfmuls,5 %r9, %r37, %r9 } { loop_mode qpxor,0 %r36, %r5, %r46 qpxor,1 %r35, %r5, %r54 qpfmuls,2 %r51, %g22, %g22 qpshufb,3 %r36, %r36, %r6, %r36 qpshufb,4 %r35, %r35, %r6, %r35 qpfmuls,5 %r28, %g22, %r28 } { loop_mode qpfmuls,0 %r52, %r41, %r41 qpfmuls,1 %r45, %r34, %r45 qpfmuls,2 %r53, %r38, %r51 qpfmuls,3 %r27, %r41, %r27 qpfmuls,4 %g19, %r34, %g19 qpfmuls,5 %g18, %r38, %g18 } { loop_mode qpfmuls,0 %r54, %r42, %r38 qpxor,1 %r29, %r5, %r36 qpfmuls,2 %r46, %g23, %r34 qpfmuls,3 %r35, %r42, %r35 qpshufb,4 %r29, %r29, %r6, %r29 qpfmuls,5 %r36, %g23, %g23 } { loop_mode qpfmuls,2 %r36, %g20, %r36 qpfmuls,5 %r29, %g20, %g20 } { loop_mode qpxor,1 %r30, %r5, %r29 qpfhadds,2 %r33, %r26, %r26 qpxor,4 %r31, %r5, %r42 } { loop_mode qpshufb,0 %r30, %r30, %r6, %r30 qpshufb,1 %r31, %r31, %r6, %r31 qpfmuls,2 %r29, %g16, %r29 qpxor,3 %r32, %r5, %r33 qpshufb,4 %r32, %r32, %r6, %r32 qpfmuls,5 %r42, %g28, %r42 } { loop_mode qpfmuls,0 %r30, %g16, %g16 qpfhadds,1 %r37, %r9, %r9 qpfmuls,2 %r31, %g28, %g28 qpfmuls,3 %r32, %g24, %g24 qpfhadds,4 %g22, %r28, %g22 qpfmuls,5 %r33, %g24, %r30 } { loop_mode qpfhadds,0 %r45, %g19, %g19 qpfhadds,1 %r51, %g18, %g18 qpfhadds,2 %r41, %r27, %r27 qpshufb,3 %g26, %g26, %r6, %r28 qpxor,4 %g26, %r5, %g26 } { loop_mode qpfhadds,0 %r38, %r35, %r31 qpfhadds,2 %r34, %g23, %g23 } { loop_mode qpfhadds,2 %r36, %g20, %g20 qpshufb,3 %r39, %r39, %r6, %r32 qpxor,4 %r39, %r5, %r33 } { loop_mode qpshufb,1 %r26, %r26, %r3, %r26 qpfhadds,2 %r29, %g16, %g16 qpshufb,3 %g22, %g22, %r3, %g22 qpshufb,4 %r40, %r40, %r6, %r29 qpfhadds,5 %r30, %g24, %g24 } { loop_mode qpshufb,0 %r9, %r9, %r3, %r9 qpshufb,1 %g19, %g19, %r3, %g19 qpfhadds,2 %r42, %g28, %g28 qpxor,3 %r40, %r5, %r30 qpshufb,4 %g31, %g31, %r6, %r34 } { loop_mode qpshufb,0 %r27, %r27, %r3, %r27 qpshufb,1 %g18, %g18, %r3, %g18 qpfsubs,2 %r26, %g19, %r35 qpxor,3 %g31, %r5, %g31 qpxor,4 %r43, %r5, %r36 } { loop_mode qpshufb,0 %r31, %r31, %r3, %r31 qpshufb,1 %g23, %g23, %r3, %g23 qpfsubs,2 %r9, %g18, %r37 qpshufb,3 %r43, %r43, %r6, %r38 qpxor,4 %r48, %r5, %r39 } { loop_mode qpfsubs,0 %g22, %g23, %r41 qpshufb,1 %g16, %g16, %r3, %g16 qpfsubs,2 %r27, %r31, %r40 qpshufb,3 %g24, %g24, %r3, %g24 qpxor,4 %r47, %r5, %r42 } { loop_mode qpshufb,0 %g28, %g28, %r3, %g28 qpshufb,1 %g20, %g20, %r3, %g20 qpfadds,2 %r26, %g19, %g19 qpfsubs,3 %g25, %g24, %g24 qpshufb,4 %r48, %r48, %r6, %g25 qpfadds,5 %g25, %g24, %r26 } { loop_mode qpfadds,0 %r9, %g18, %g18 qpfadds,1 %r27, %r31, %r9 qpfadds,2 %g22, %g23, %g22 qpshufb,3 %r47, %r47, %r6, %g23 qpxor,4 %r50, %r5, %r27 } { loop_mode qpfadds,0 %g17, %g16, %r31 qpfadds,1 %g29, %g28, %g17 qpfsubs,2 %g17, %g16, %g16 qpshufb,4 %r50, %r50, %r6, %r43 } { loop_mode qpfadds,0 %g21, %g20, %g29 qpfsubs,1 %g21, %g20, %g20 qpfsubs,2 %g29, %g28, %g28 qpshufb,3 %r35, %r35, %r6, %g21 qpxor,4 %g27, %r5, %r35 } { loop_mode qpshufb,3 %r37, %r37, %r6, %r37 qpxor,4 %g21, %r4, %g21 } { loop_mode qpshufb,3 %r40, %r40, %r6, %r40 qpshufb,4 %r41, %r41, %r6, %r41 } { loop_mode qpfsubs,0 %r31, %g19, %r45 qpfadds,1 %r31, %g19, %g19 qpfadds,2 %g17, %r9, %r31 qpxor,3 %r37, %r4, %r37 qpxor,4 %r41, %r4, %r41 } { loop_mode qpfsubs,0 %g17, %r9, %g17 qpfadds,1 %g29, %g18, %r9 qpfsubs,2 %g29, %g18, %g18 qpxor,3 %r40, %r4, %r40 qpfadds,4 %r26, %g22, %g29 qpfsubs,5 %r26, %g22, %g22 } { loop_mode qpfsubs,2 %g16, %g21, %r26 qpfsubs,3 %g24, %r41, %g21 qpfadds,4 %g24, %r41, %g24 qpfadds,5 %g16, %g21, %g16 } { loop_mode qpfsubs,3 %g20, %r37, %r46 qpfadds,4 %g28, %r40, %g28 qpfsubs,5 %g28, %r40, %r41 } { loop_mode qpshufb,0 %g27, %g27, %r6, %g27 qpshufb,1 %g30, %g30, %r6, %r37 qpfadds,2 %g20, %r37, %g20 } { loop_mode qpshufb,0 %r31, %r9, %r1, %r40 qpshufb,1 %g17, %g18, %r1, %r47 } { loop_mode qpfmuls,0 %r33, %r40, %r33 qpfmuls,1 %r42, %r47, %r40 qpfmuls,2 %r32, %r40, %r32 qpshufb,3 %g29, %g19, %r1, %r48 qpshufb,4 %g22, %r45, %r1, %r50 } { loop_mode qpxor,0 %g30, %r5, %g30 qpshufb,1 %r44, %r44, %r6, %r47 qpfmuls,2 %g23, %r47, %g23 qpshufb,3 %r41, %r46, %r1, %r42 qpshufb,4 %g24, %g16, %r1, %r51 qpfmuls,5 %r38, %r50, %r38 } { loop_mode qpshufb,3 %g21, %r26, %r1, %r52 qpfmuls,4 %g31, %r42, %g31 qpfmuls,5 %r34, %r42, %r34 } { loop_mode qpshufb,0 %g28, %g20, %r1, %r42 qpshufb,1 %g17, %g18, %r7, %g17 qpfmuls,3 %g26, %r48, %g26 qpfmuls,4 %r36, %r50, %r36 qpfmuls,5 %r28, %r48, %r28 } { loop_mode qpfmuls,0 %r43, %r42, %r39 qpxor,1 %r44, %r5, %r42 qpfmuls,2 %r27, %r42, %r27 qpfmuls,3 %r39, %r51, %g18 qpfmuls,4 %g25, %r51, %g25 qpfmuls,5 %r29, %r52, %r29 } { loop_mode qpxor,0 %r49, %r5, %r43 qpshufb,1 %r49, %r49, %r6, %r44 qpfmuls,2 %r42, %g17, %r42 qpfmuls,5 %r30, %r52, %r30 } { loop_mode qpfhadds,0 %r33, %r32, %r31 qpshufb,1 %r31, %r9, %r7, %r9 qpfmuls,2 %r47, %g17, %g17 qpfhadds,5 %g31, %r34, %g31 } { loop_mode qpshufb,0 %g28, %g20, %r7, %g20 qpshufb,1 %r41, %r46, %r7, %g28 qpfmuls,2 %r35, %r9, %r32 qpfhadds,3 %r36, %r38, %r28 qpfhadds,4 %r40, %g23, %g23 qpfhadds,5 %g26, %r28, %g26 } { loop_mode qpfmuls,0 %r43, %g20, %r33 qpfmuls,1 %r44, %g20, %g20 qpfmuls,2 %g27, %r9, %g27 qpshufb,3 %g29, %g19, %r7, %g19 qpshufb,4 %g22, %r45, %r7, %g22 qpfhadds,5 %g18, %g25, %g18 } { loop_mode qpfmuls,0 %g30, %g28, %g28 qpfhadds,1 %r27, %r39, %g29 qpfmuls,2 %r37, %g28, %g25 qpfhadds,5 %r30, %r29, %g30 } { loop_mode qpfhadds,2 %r42, %g17, %g17 qpshufb,3 %g31, %g31, %r3, %g31 qpshufb,4 %g24, %g16, %r7, %g16 } { loop_mode qpshufb,3 %g26, %g26, %r3, %g24 qpshufb,4 %r28, %r28, %r3, %g26 } { loop_mode qpfhadds,0 %r32, %g27, %g27 qpshufb,1 %r31, %r31, %r3, %r9 qpfhadds,2 %r33, %g20, %g20 qpshufb,3 %g23, %g23, %r3, %g23 qpshufb,4 %g18, %g18, %r3, %g18 } { loop_mode qpshufb,0 %g29, %g29, %r3, %g28 qpshufb,1 %g21, %r26, %r7, %g21 qpfhadds,2 %g28, %g25, %g25 qpshufb,3 %g30, %g30, %r3, %g29 qpfadds,4 %g26, %g23, %g23 qpfsubs,5 %g26, %g23, %g30 } { loop_mode qpshufb,1 %g17, %g17, %r3, %g17 qpfadds,3 %g29, %g31, %g29 qpfsubs,5 %g29, %g31, %g26 } { loop_mode qpfsubs,0 %g24, %r9, %g31 qpfadds,1 %g24, %r9, %g24 qpfadds,2 %g22, %g17, %r9 } { loop_mode qpshufb,0 %g20, %g20, %r3, %g20 qpshufb,1 %g27, %g27, %r3, %g27 qpfsubs,2 %g18, %g28, %r26 } { loop_mode qpfadds,0 %g18, %g28, %g18 qpfsubs,1 %g16, %g20, %g16 qpfadds,2 %g16, %g20, %g28 qpshufb,3 %g30, %g30, %r6, %g20 } { loop_mode qpshufb,0 %g25, %g25, %r3, %g25 qpfadds,1 %g19, %g27, %g19 qpfsubs,2 %g19, %g27, %g30 qpshufb,3 %g26, %g26, %r6, %g22 qpxor,4 %g20, %r4, %g20 qpfsubs,5 %g22, %g17, %g17 } { loop_mode qpfsubs,0 %g21, %g25, %g21 qpfadds,1 %r9, %g23, %g25 qpfadds,2 %g21, %g25, %g26 qpxor,3 %g22, %r4, %g22 } { loop_mode qpshufb,0 %g31, %g31, %r6, %g27 qpfsubs,1 %r9, %g23, %g23 } { loop_mode qpfadds,0 %g28, %g18, %g31 qpfsubs,1 %g28, %g18, %g18 } { loop_mode qpshufb,0 %r26, %r26, %r6, %g28 qpfadds,1 %g19, %g24, %r9 qpfsubs,2 %g19, %g24, %g19 qpfsubs,3 %g17, %g20, %g24 qpfadds,4 %g17, %g20, %g17 } { loop_mode qpfadds,0 %g26, %g29, %g20 qpfsubs,1 %g26, %g29, %g26 qpfadds,2 %g21, %g22, %g29 } { loop_mode qpxor,0 %g27, %r4, %g27 qpfsubs,1 %g21, %g22, %g21 stqp,2 %r18, %r0, %g23 } { loop_mode qpfsubs,0 %g30, %g27, %g22 qpfadds,1 %g30, %g27, %g23 stqp,2 %r12, %r0, %g18 stqp,5 %r25, %r0, %g25 } { loop_mode qpxor,0 %g28, %r4, %g18 stqp,2 %r16, %r0, %g31 stqp,5 %r14, %r0, %g17 } { loop_mode qpfsubs,0 %g16, %g18, %g17 qpfadds,1 %g16, %g18, %g16 stqp,2 %r23, %r0, %g19 stqp,5 %r15, %r0, %g24 } { loop_mode stqp,2 %r2, %r0, %r9 } { loop_mode stqp,2 %r24, %r0, %g22 stqp,5 %r22, %r0, %g20 } { loop_mode stqp,2 %r19, %r0, %g26 stqp,5 %r21, %r0, %g21 } { loop_mode stqp,2 %r17, %r0, %g23 stqp,5 %r11, %r0, %g29 } { loop_mode stqp,2 %r20, %r0, %g17 } { loop_mode ct %ctpr1 ? %NOT_LOOP_END alc alcf=1, alct=1 addd,0,sm 0x10, %r0, %r0 stqp,2 %r13, %r0, %g16 }
Теоретическая скорость: 32 комплексных числа за 73 такта (32/73) = 3.51 Байт/такт
Четверная теоретическая скорость: 14.03 Байт/такт
Замеры скорости
4. stage_radix4_2x_simd128_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью опции unroll.
Код на Си
void stage_radix4_2x_simd128_unroll2(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coefC_a, myComplex *coefD_a, myComplex *coefE_a, myComplex *coefC_b, myComplex *coefD_b, myComplex *coefE_b) { __v2di *xy0_in = (__v2di*)&data_in[ 0]; __v2di *zw0_in = (__v2di*)&data_in[ 2]; __v2di *xy1_in = (__v2di*)&data_in[ 4]; __v2di *zw1_in = (__v2di*)&data_in[ 6]; __v2di *xy2_in = (__v2di*)&data_in[ 8]; __v2di *zw2_in = (__v2di*)&data_in[10]; __v2di *xy3_in = (__v2di*)&data_in[12]; __v2di *zw3_in = (__v2di*)&data_in[14]; __v2di *xy4_in = (__v2di*)&data_in[16]; __v2di *zw4_in = (__v2di*)&data_in[18]; __v2di *xy5_in = (__v2di*)&data_in[20]; __v2di *zw5_in = (__v2di*)&data_in[22]; __v2di *xy6_in = (__v2di*)&data_in[24]; __v2di *zw6_in = (__v2di*)&data_in[26]; __v2di *xy7_in = (__v2di*)&data_in[28]; __v2di *zw7_in = (__v2di*)&data_in[30]; __v2di *c0a_in = (__v2di*)&coefC_a[0]; __v2di *c1a_in = (__v2di*)&coefC_a[2]; __v2di *c2a_in = (__v2di*)&coefC_a[4]; __v2di *c3a_in = (__v2di*)&coefC_a[6]; __v2di *d0a_in = (__v2di*)&coefD_a[0]; __v2di *d1a_in = (__v2di*)&coefD_a[2]; __v2di *d2a_in = (__v2di*)&coefD_a[4]; __v2di *d3a_in = (__v2di*)&coefD_a[6]; __v2di *e0a_in = (__v2di*)&coefE_a[0]; __v2di *e1a_in = (__v2di*)&coefE_a[2]; __v2di *e2a_in = (__v2di*)&coefE_a[4]; __v2di *e3a_in = (__v2di*)&coefE_a[6]; __v2di *c0b_in = (__v2di*)&coefC_b[0*data_count/16]; __v2di *c1b_in = (__v2di*)&coefC_b[1*data_count/16]; __v2di *c2b_in = (__v2di*)&coefC_b[2*data_count/16]; __v2di *c3b_in = (__v2di*)&coefC_b[3*data_count/16]; __v2di *d0b_in = (__v2di*)&coefD_b[0*data_count/16]; __v2di *d1b_in = (__v2di*)&coefD_b[1*data_count/16]; __v2di *d2b_in = (__v2di*)&coefD_b[2*data_count/16]; __v2di *d3b_in = (__v2di*)&coefD_b[3*data_count/16]; __v2di *e0b_in = (__v2di*)&coefE_b[0*data_count/16]; __v2di *e1b_in = (__v2di*)&coefE_b[1*data_count/16]; __v2di *e2b_in = (__v2di*)&coefE_b[2*data_count/16]; __v2di *e3b_in = (__v2di*)&coefE_b[3*data_count/16]; __v2di *out_0 = (__v2di*)&data_out[ 0*data_count/16]; __v2di *out_1 = (__v2di*)&data_out[ 1*data_count/16]; __v2di *out_2 = (__v2di*)&data_out[ 2*data_count/16]; __v2di *out_3 = (__v2di*)&data_out[ 3*data_count/16]; __v2di *out_4 = (__v2di*)&data_out[ 4*data_count/16]; __v2di *out_5 = (__v2di*)&data_out[ 5*data_count/16]; __v2di *out_6 = (__v2di*)&data_out[ 6*data_count/16]; __v2di *out_7 = (__v2di*)&data_out[ 7*data_count/16]; __v2di *out_8 = (__v2di*)&data_out[ 8*data_count/16]; __v2di *out_9 = (__v2di*)&data_out[ 9*data_count/16]; __v2di *out_10 = (__v2di*)&data_out[10*data_count/16]; __v2di *out_11 = (__v2di*)&data_out[11*data_count/16]; __v2di *out_12 = (__v2di*)&data_out[12*data_count/16]; __v2di *out_13 = (__v2di*)&data_out[13*data_count/16]; __v2di *out_14 = (__v2di*)&data_out[14*data_count/16]; __v2di *out_15 = (__v2di*)&data_out[15*data_count/16]; #pragma ivdep #pragma unroll(2) #pragma prefetch for(int64_t i = 0; i < data_count/32; ++i) { __v2di xy0 = xy0_in[16*i]; __v2di zw0 = zw0_in[16*i]; __v2di xy1 = xy1_in[16*i]; __v2di zw1 = zw1_in[16*i]; __v2di c0 = c0a_in[4*i]; __v2di d0 = d0a_in[4*i]; __v2di e0 = e0a_in[4*i]; __v2di xy2 = xy2_in[16*i]; __v2di zw2 = zw2_in[16*i]; __v2di xy3 = xy3_in[16*i]; __v2di zw3 = zw3_in[16*i]; __v2di c1 = c1a_in[4*i]; __v2di d1 = d1a_in[4*i]; __v2di e1 = e1a_in[4*i]; __v2di xy4 = xy4_in[16*i]; __v2di zw4 = zw4_in[16*i]; __v2di xy5 = xy5_in[16*i]; __v2di zw5 = zw5_in[16*i]; __v2di c2 = c2a_in[4*i]; __v2di d2 = d2a_in[4*i]; __v2di e2 = e2a_in[4*i]; __v2di xy6 = xy6_in[16*i]; __v2di zw6 = zw6_in[16*i]; __v2di xy7 = xy7_in[16*i]; __v2di zw7 = zw7_in[16*i]; __v2di c3 = c3a_in[4*i]; __v2di d3 = d3a_in[4*i]; __v2di e3 = e3a_in[4*i]; __v2di x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di conj_c0 = __builtin_e2k_qpxor(c0, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_c1 = __builtin_e2k_qpxor(c1, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_c2 = __builtin_e2k_qpxor(c2, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_c3 = __builtin_e2k_qpxor(c3, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_d0 = __builtin_e2k_qpxor(d0, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_d1 = __builtin_e2k_qpxor(d1, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_d2 = __builtin_e2k_qpxor(d2, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_d3 = __builtin_e2k_qpxor(d3, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_e0 = __builtin_e2k_qpxor(e0, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_e1 = __builtin_e2k_qpxor(e1, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_e2 = __builtin_e2k_qpxor(e2, (__v2di){1LL<<63, 1LL<<63}); __v2di conj_e3 = __builtin_e2k_qpxor(e3, (__v2di){1LL<<63, 1LL<<63}); __v2di swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c2 = __builtin_e2k_qpshufb(c2, c2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c3 = __builtin_e2k_qpshufb(c3, c3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d0 = __builtin_e2k_qpshufb(d0, d0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d1 = __builtin_e2k_qpshufb(d1, d1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d2 = __builtin_e2k_qpshufb(d2, d2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d3 = __builtin_e2k_qpshufb(d3, d3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e0 = __builtin_e2k_qpshufb(e0, e0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e1 = __builtin_e2k_qpshufb(e1, e1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e2 = __builtin_e2k_qpshufb(e2, e2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e3 = __builtin_e2k_qpshufb(e3, e3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy0_real = __builtin_e2k_qpfmuls(conj_c0, y0); __v2di cy1_real = __builtin_e2k_qpfmuls(conj_c1, y1); __v2di cy2_real = __builtin_e2k_qpfmuls(conj_c2, y2); __v2di cy3_real = __builtin_e2k_qpfmuls(conj_c3, y3); __v2di dz0_real = __builtin_e2k_qpfmuls(conj_d0, z0); __v2di dz1_real = __builtin_e2k_qpfmuls(conj_d1, z1); __v2di dz2_real = __builtin_e2k_qpfmuls(conj_d2, z2); __v2di dz3_real = __builtin_e2k_qpfmuls(conj_d3, z3); __v2di ew0_real = __builtin_e2k_qpfmuls(conj_e0, w0); __v2di ew1_real = __builtin_e2k_qpfmuls(conj_e1, w1); __v2di ew2_real = __builtin_e2k_qpfmuls(conj_e2, w2); __v2di ew3_real = __builtin_e2k_qpfmuls(conj_e3, w3); __v2di cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); __v2di cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); __v2di cy2_imag = __builtin_e2k_qpfmuls(swap_c2, y2); __v2di cy3_imag = __builtin_e2k_qpfmuls(swap_c3, y3); __v2di dz0_imag = __builtin_e2k_qpfmuls(swap_d0, z0); __v2di dz1_imag = __builtin_e2k_qpfmuls(swap_d1, z1); __v2di dz2_imag = __builtin_e2k_qpfmuls(swap_d2, z2); __v2di dz3_imag = __builtin_e2k_qpfmuls(swap_d3, z3); __v2di ew0_imag = __builtin_e2k_qpfmuls(swap_e0, w0); __v2di ew1_imag = __builtin_e2k_qpfmuls(swap_e1, w1); __v2di ew2_imag = __builtin_e2k_qpfmuls(swap_e2, w2); __v2di ew3_imag = __builtin_e2k_qpfmuls(swap_e3, w3); __v2di cy0_rrii = __builtin_e2k_qpfhadds(cy0_real, cy0_imag); __v2di cy1_rrii = __builtin_e2k_qpfhadds(cy1_real, cy1_imag); __v2di cy2_rrii = __builtin_e2k_qpfhadds(cy2_real, cy2_imag); __v2di cy3_rrii = __builtin_e2k_qpfhadds(cy3_real, cy3_imag); __v2di dz0_rrii = __builtin_e2k_qpfhadds(dz0_real, dz0_imag); __v2di dz1_rrii = __builtin_e2k_qpfhadds(dz1_real, dz1_imag); __v2di dz2_rrii = __builtin_e2k_qpfhadds(dz2_real, dz2_imag); __v2di dz3_rrii = __builtin_e2k_qpfhadds(dz3_real, dz3_imag); __v2di ew0_rrii = __builtin_e2k_qpfhadds(ew0_real, ew0_imag); __v2di ew1_rrii = __builtin_e2k_qpfhadds(ew1_real, ew1_imag); __v2di ew2_rrii = __builtin_e2k_qpfhadds(ew2_real, ew2_imag); __v2di ew3_rrii = __builtin_e2k_qpfhadds(ew3_real, ew3_imag); __v2di cy0 = __builtin_e2k_qpshufb(cy0_rrii, cy0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di cy1 = __builtin_e2k_qpshufb(cy1_rrii, cy1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di cy2 = __builtin_e2k_qpshufb(cy2_rrii, cy2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di cy3 = __builtin_e2k_qpshufb(cy3_rrii, cy3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di dz0 = __builtin_e2k_qpshufb(dz0_rrii, dz0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di dz1 = __builtin_e2k_qpshufb(dz1_rrii, dz1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di dz2 = __builtin_e2k_qpshufb(dz2_rrii, dz2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di dz3 = __builtin_e2k_qpshufb(dz3_rrii, dz3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di ew0 = __builtin_e2k_qpshufb(ew0_rrii, ew0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di ew1 = __builtin_e2k_qpshufb(ew1_rrii, ew1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di ew2 = __builtin_e2k_qpshufb(ew2_rrii, ew2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di ew3 = __builtin_e2k_qpshufb(ew3_rrii, ew3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di add02_0 = __builtin_e2k_qpfadds( x0, dz0); __v2di add02_1 = __builtin_e2k_qpfadds( x1, dz1); __v2di add02_2 = __builtin_e2k_qpfadds( x2, dz2); __v2di add02_3 = __builtin_e2k_qpfadds( x3, dz3); __v2di sub02_0 = __builtin_e2k_qpfsubs( x0, dz0); __v2di sub02_1 = __builtin_e2k_qpfsubs( x1, dz1); __v2di sub02_2 = __builtin_e2k_qpfsubs( x2, dz2); __v2di sub02_3 = __builtin_e2k_qpfsubs( x3, dz3); __v2di add13_0 = __builtin_e2k_qpfadds(cy0, ew0); __v2di add13_1 = __builtin_e2k_qpfadds(cy1, ew1); __v2di add13_2 = __builtin_e2k_qpfadds(cy2, ew2); __v2di add13_3 = __builtin_e2k_qpfadds(cy3, ew3); __v2di sub13_0 = __builtin_e2k_qpfsubs(cy0, ew0); __v2di sub13_1 = __builtin_e2k_qpfsubs(cy1, ew1); __v2di sub13_2 = __builtin_e2k_qpfsubs(cy2, ew2); __v2di sub13_3 = __builtin_e2k_qpfsubs(cy3, ew3); __v2di swap_sub13_0 = __builtin_e2k_qpshufb(sub13_0, sub13_0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_1 = __builtin_e2k_qpshufb(sub13_1, sub13_1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_2 = __builtin_e2k_qpshufb(sub13_2, sub13_2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_3 = __builtin_e2k_qpshufb(sub13_3, sub13_3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di sub13i_0 = __builtin_e2k_qpxor(swap_sub13_0, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_1 = __builtin_e2k_qpxor(swap_sub13_1, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_2 = __builtin_e2k_qpxor(swap_sub13_2, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_3 = __builtin_e2k_qpxor(swap_sub13_3, (__v2di){1LL<<31, 1LL<<31}); __v2di out0 = __builtin_e2k_qpfadds(add02_0, add13_0); __v2di out1 = __builtin_e2k_qpfadds(add02_1, add13_1); __v2di out2 = __builtin_e2k_qpfadds(add02_2, add13_2); __v2di out3 = __builtin_e2k_qpfadds(add02_3, add13_3); __v2di out4 = __builtin_e2k_qpfsubs(sub02_0, sub13i_0); __v2di out5 = __builtin_e2k_qpfsubs(sub02_1, sub13i_1); __v2di out6 = __builtin_e2k_qpfsubs(sub02_2, sub13i_2); __v2di out7 = __builtin_e2k_qpfsubs(sub02_3, sub13i_3); __v2di out8 = __builtin_e2k_qpfsubs(add02_0, add13_0); __v2di out9 = __builtin_e2k_qpfsubs(add02_1, add13_1); __v2di out10 = __builtin_e2k_qpfsubs(add02_2, add13_2); __v2di out11 = __builtin_e2k_qpfsubs(add02_3, add13_3); __v2di out12 = __builtin_e2k_qpfadds(sub02_0, sub13i_0); __v2di out13 = __builtin_e2k_qpfadds(sub02_1, sub13i_1); __v2di out14 = __builtin_e2k_qpfadds(sub02_2, sub13i_2); __v2di out15 = __builtin_e2k_qpfadds(sub02_3, sub13i_3); xy0 = out0; zw0 = out1; xy1 = out2; zw1 = out3; c0 = c0b_in[i]; d0 = d0b_in[i]; e0 = e0b_in[i]; xy2 = out4; zw2 = out5; xy3 = out6; zw3 = out7; c1 = c1b_in[i]; d1 = d1b_in[i]; e1 = e1b_in[i]; xy4 = out8; zw4 = out9; xy5 = out10; zw5 = out11; c2 = c2b_in[i]; d2 = d2b_in[i]; e2 = e2b_in[i]; xy6 = out12; zw6 = out13; xy7 = out14; zw7 = out15; c3 = c3b_in[i]; d3 = d3b_in[i]; e3 = e3b_in[i]; x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); w0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0706050403020100, 0x0706050403020100}); w1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0706050403020100, 0x0706050403020100}); y2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0706050403020100, 0x0706050403020100}); w2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0706050403020100, 0x0706050403020100}); y3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0706050403020100, 0x0706050403020100}); w3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); conj_c0 = __builtin_e2k_qpxor(c0, (__v2di){1LL<<63, 1LL<<63}); conj_c1 = __builtin_e2k_qpxor(c1, (__v2di){1LL<<63, 1LL<<63}); conj_c2 = __builtin_e2k_qpxor(c2, (__v2di){1LL<<63, 1LL<<63}); conj_c3 = __builtin_e2k_qpxor(c3, (__v2di){1LL<<63, 1LL<<63}); conj_d0 = __builtin_e2k_qpxor(d0, (__v2di){1LL<<63, 1LL<<63}); conj_d1 = __builtin_e2k_qpxor(d1, (__v2di){1LL<<63, 1LL<<63}); conj_d2 = __builtin_e2k_qpxor(d2, (__v2di){1LL<<63, 1LL<<63}); conj_d3 = __builtin_e2k_qpxor(d3, (__v2di){1LL<<63, 1LL<<63}); conj_e0 = __builtin_e2k_qpxor(e0, (__v2di){1LL<<63, 1LL<<63}); conj_e1 = __builtin_e2k_qpxor(e1, (__v2di){1LL<<63, 1LL<<63}); conj_e2 = __builtin_e2k_qpxor(e2, (__v2di){1LL<<63, 1LL<<63}); conj_e3 = __builtin_e2k_qpxor(e3, (__v2di){1LL<<63, 1LL<<63}); swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c2 = __builtin_e2k_qpshufb(c2, c2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c3 = __builtin_e2k_qpshufb(c3, c3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d0 = __builtin_e2k_qpshufb(d0, d0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d1 = __builtin_e2k_qpshufb(d1, d1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d2 = __builtin_e2k_qpshufb(d2, d2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d3 = __builtin_e2k_qpshufb(d3, d3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e0 = __builtin_e2k_qpshufb(e0, e0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e1 = __builtin_e2k_qpshufb(e1, e1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e2 = __builtin_e2k_qpshufb(e2, e2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e3 = __builtin_e2k_qpshufb(e3, e3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); cy0_real = __builtin_e2k_qpfmuls(conj_c0, y0); cy1_real = __builtin_e2k_qpfmuls(conj_c1, y1); cy2_real = __builtin_e2k_qpfmuls(conj_c2, y2); cy3_real = __builtin_e2k_qpfmuls(conj_c3, y3); dz0_real = __builtin_e2k_qpfmuls(conj_d0, z0); dz1_real = __builtin_e2k_qpfmuls(conj_d1, z1); dz2_real = __builtin_e2k_qpfmuls(conj_d2, z2); dz3_real = __builtin_e2k_qpfmuls(conj_d3, z3); ew0_real = __builtin_e2k_qpfmuls(conj_e0, w0); ew1_real = __builtin_e2k_qpfmuls(conj_e1, w1); ew2_real = __builtin_e2k_qpfmuls(conj_e2, w2); ew3_real = __builtin_e2k_qpfmuls(conj_e3, w3); cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); cy2_imag = __builtin_e2k_qpfmuls(swap_c2, y2); cy3_imag = __builtin_e2k_qpfmuls(swap_c3, y3); dz0_imag = __builtin_e2k_qpfmuls(swap_d0, z0); dz1_imag = __builtin_e2k_qpfmuls(swap_d1, z1); dz2_imag = __builtin_e2k_qpfmuls(swap_d2, z2); dz3_imag = __builtin_e2k_qpfmuls(swap_d3, z3); ew0_imag = __builtin_e2k_qpfmuls(swap_e0, w0); ew1_imag = __builtin_e2k_qpfmuls(swap_e1, w1); ew2_imag = __builtin_e2k_qpfmuls(swap_e2, w2); ew3_imag = __builtin_e2k_qpfmuls(swap_e3, w3); cy0_rrii = __builtin_e2k_qpfhadds(cy0_real, cy0_imag); cy1_rrii = __builtin_e2k_qpfhadds(cy1_real, cy1_imag); cy2_rrii = __builtin_e2k_qpfhadds(cy2_real, cy2_imag); cy3_rrii = __builtin_e2k_qpfhadds(cy3_real, cy3_imag); dz0_rrii = __builtin_e2k_qpfhadds(dz0_real, dz0_imag); dz1_rrii = __builtin_e2k_qpfhadds(dz1_real, dz1_imag); dz2_rrii = __builtin_e2k_qpfhadds(dz2_real, dz2_imag); dz3_rrii = __builtin_e2k_qpfhadds(dz3_real, dz3_imag); ew0_rrii = __builtin_e2k_qpfhadds(ew0_real, ew0_imag); ew1_rrii = __builtin_e2k_qpfhadds(ew1_real, ew1_imag); ew2_rrii = __builtin_e2k_qpfhadds(ew2_real, ew2_imag); ew3_rrii = __builtin_e2k_qpfhadds(ew3_real, ew3_imag); cy0 = __builtin_e2k_qpshufb(cy0_rrii, cy0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); cy1 = __builtin_e2k_qpshufb(cy1_rrii, cy1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); cy2 = __builtin_e2k_qpshufb(cy2_rrii, cy2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); cy3 = __builtin_e2k_qpshufb(cy3_rrii, cy3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); dz0 = __builtin_e2k_qpshufb(dz0_rrii, dz0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); dz1 = __builtin_e2k_qpshufb(dz1_rrii, dz1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); dz2 = __builtin_e2k_qpshufb(dz2_rrii, dz2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); dz3 = __builtin_e2k_qpshufb(dz3_rrii, dz3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); ew0 = __builtin_e2k_qpshufb(ew0_rrii, ew0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); ew1 = __builtin_e2k_qpshufb(ew1_rrii, ew1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); ew2 = __builtin_e2k_qpshufb(ew2_rrii, ew2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); ew3 = __builtin_e2k_qpshufb(ew3_rrii, ew3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); add02_0 = __builtin_e2k_qpfadds( x0, dz0); add02_1 = __builtin_e2k_qpfadds( x1, dz1); add02_2 = __builtin_e2k_qpfadds( x2, dz2); add02_3 = __builtin_e2k_qpfadds( x3, dz3); sub02_0 = __builtin_e2k_qpfsubs( x0, dz0); sub02_1 = __builtin_e2k_qpfsubs( x1, dz1); sub02_2 = __builtin_e2k_qpfsubs( x2, dz2); sub02_3 = __builtin_e2k_qpfsubs( x3, dz3); add13_0 = __builtin_e2k_qpfadds(cy0, ew0); add13_1 = __builtin_e2k_qpfadds(cy1, ew1); add13_2 = __builtin_e2k_qpfadds(cy2, ew2); add13_3 = __builtin_e2k_qpfadds(cy3, ew3); sub13_0 = __builtin_e2k_qpfsubs(cy0, ew0); sub13_1 = __builtin_e2k_qpfsubs(cy1, ew1); sub13_2 = __builtin_e2k_qpfsubs(cy2, ew2); sub13_3 = __builtin_e2k_qpfsubs(cy3, ew3); swap_sub13_0 = __builtin_e2k_qpshufb(sub13_0, sub13_0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_1 = __builtin_e2k_qpshufb(sub13_1, sub13_1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_2 = __builtin_e2k_qpshufb(sub13_2, sub13_2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_3 = __builtin_e2k_qpshufb(sub13_3, sub13_3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); sub13i_0 = __builtin_e2k_qpxor(swap_sub13_0, (__v2di){1LL<<31, 1LL<<31}); sub13i_1 = __builtin_e2k_qpxor(swap_sub13_1, (__v2di){1LL<<31, 1LL<<31}); sub13i_2 = __builtin_e2k_qpxor(swap_sub13_2, (__v2di){1LL<<31, 1LL<<31}); sub13i_3 = __builtin_e2k_qpxor(swap_sub13_3, (__v2di){1LL<<31, 1LL<<31}); out_0[i] = __builtin_e2k_qpfadds(add02_0, add13_0); out_1[i] = __builtin_e2k_qpfadds(add02_1, add13_1); out_2[i] = __builtin_e2k_qpfadds(add02_2, add13_2); out_3[i] = __builtin_e2k_qpfadds(add02_3, add13_3); out_4[i] = __builtin_e2k_qpfsubs(sub02_0, sub13i_0); out_5[i] = __builtin_e2k_qpfsubs(sub02_1, sub13i_1); out_6[i] = __builtin_e2k_qpfsubs(sub02_2, sub13i_2); out_7[i] = __builtin_e2k_qpfsubs(sub02_3, sub13i_3); out_8[i] = __builtin_e2k_qpfsubs(add02_0, add13_0); out_9[i] = __builtin_e2k_qpfsubs(add02_1, add13_1); out_10[i] = __builtin_e2k_qpfsubs(add02_2, add13_2); out_11[i] = __builtin_e2k_qpfsubs(add02_3, add13_3); out_12[i] = __builtin_e2k_qpfadds(sub02_0, sub13i_0); out_13[i] = __builtin_e2k_qpfadds(sub02_1, sub13i_1); out_14[i] = __builtin_e2k_qpfadds(sub02_2, sub13i_2); out_15[i] = __builtin_e2k_qpfadds(sub02_3, sub13i_3); } }
Основной цикл на ассемблере
.L11610: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=1, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=1, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=2, disp=128 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=2, disp=160 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=3, disp=192 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=3, disp=224 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=4, disp=256 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=4, disp=288 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=5, disp=320 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=5, disp=352 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=6, disp=384 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=6, disp=416 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=7, disp=448 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=7, disp=480 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=1, abs=8, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=1, abs=10, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=1, abs=10, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=1, abs=12, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=1, abs=14, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=1, abs=14, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=1, abs=16, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=1, abs=18, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=1, abs=18, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=1, incr=2, ind=0, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=15, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=14, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=13, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=12, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=11, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=10, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=9, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=8, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=7, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=6, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=5, asz=1, abs=30, disp=0 } .L7588: { loop_mode disp %ctpr1, .L7588 movaqp,0 area=0, ind=0, am=1, be=0, %g17 movaqp,1 area=0, ind=16, am=0, be=0, %g16 movaqp,2 area=0, ind=0, am=1, be=0, %g19 movaqp,3 area=0, ind=16, am=0, be=0, %g18 } { loop_mode movaqp,0 area=1, ind=0, am=1, be=0, %g21 movaqp,1 area=1, ind=16, am=0, be=0, %g20 movaqp,2 area=1, ind=0, am=1, be=0, %g23 movaqp,3 area=1, ind=16, am=0, be=0, %g22 } { loop_mode movaqp,0 area=2, ind=0, am=1, be=0, %g25 movaqp,1 area=2, ind=16, am=0, be=0, %g24 movaqp,2 area=2, ind=0, am=1, be=0, %g27 movaqp,3 area=2, ind=16, am=0, be=0, %g26 } { loop_mode movaqp,0 area=3, ind=0, am=1, be=0, %g29 movaqp,1 area=3, ind=16, am=0, be=0, %g28 movaqp,2 area=3, ind=0, am=1, be=0, %g31 movaqp,3 area=3, ind=16, am=0, be=0, %g30 } { loop_mode movaqp,0 area=4, ind=0, am=1, be=0, %b[20] movaqp,1 area=4, ind=16, am=0, be=0, %b[19] movaqp,2 area=4, ind=0, am=1, be=0, %b[22] movaqp,3 area=4, ind=16, am=0, be=0, %b[21] } { loop_mode qpshufb,0 %g19, %g17, %r24, %b[27] qpshufb,1 %g18, %g16, %r24, %b[28] qpshufb,3 %g18, %g16, %r7, %g16 qpshufb,4 %g19, %g17, %r7, %g17 movaqp,0 area=5, ind=0, am=1, be=0, %b[24] movaqp,1 area=5, ind=16, am=0, be=0, %b[23] movaqp,2 area=5, ind=0, am=1, be=0, %b[26] movaqp,3 area=5, ind=16, am=0, be=0, %b[25] } { loop_mode qpshufb,0 %g23, %g21, %r24, %b[31] qpshufb,1 %g22, %g20, %r24, %b[32] qpshufb,3 %g22, %g20, %r7, %g20 qpshufb,4 %g23, %g21, %r7, %g21 movaqp,0 area=6, ind=0, am=1, be=0, %g19 movaqp,1 area=6, ind=16, am=0, be=0, %g18 movaqp,2 area=6, ind=0, am=1, be=0, %b[30] movaqp,3 area=6, ind=16, am=0, be=0, %b[29] } { loop_mode qpshufb,0 %g27, %g25, %r24, %b[35] qpshufb,1 %g26, %g24, %r24, %b[36] qpshufb,3 %g26, %g24, %r7, %g24 qpshufb,4 %g27, %g25, %r7, %g25 movaqp,0 area=7, ind=0, am=1, be=0, %g23 movaqp,1 area=7, ind=16, am=0, be=0, %g22 movaqp,2 area=7, ind=0, am=1, be=0, %b[34] movaqp,3 area=7, ind=16, am=0, be=0, %b[33] } { loop_mode qpshufb,0 %g31, %g29, %r24, %b[39] qpshufb,1 %g30, %g28, %r24, %b[40] qpshufb,3 %g30, %g28, %r7, %g28 qpshufb,4 %g31, %g29, %r7, %g29 movaqp,0 area=8, ind=0, am=1, be=0, %g27 movaqp,1 area=8, ind=16, am=0, be=0, %g26 movaqp,2 area=8, ind=0, am=1, be=0, %b[38] movaqp,3 area=8, ind=16, am=0, be=0, %b[37] } { loop_mode qpshufb,0 %b[22], %b[20], %r24, %b[43] qpshufb,1 %b[21], %b[19], %r24, %b[44] qpshufb,3 %b[21], %b[19], %r7, %b[19] qpshufb,4 %b[22], %b[20], %r7, %b[20] movaqp,0 area=9, ind=0, am=1, be=0, %g31 movaqp,1 area=9, ind=16, am=0, be=0, %g30 movaqp,2 area=9, ind=0, am=1, be=0, %b[42] movaqp,3 area=9, ind=16, am=0, be=0, %b[41] } { loop_mode qpshufb,0 %b[26], %b[24], %r24, %b[47] qpshufb,1 %b[25], %b[23], %r24, %b[48] qpshufb,3 %b[25], %b[23], %r7, %b[23] qpshufb,4 %b[26], %b[24], %r7, %b[24] movaqp,0 area=10, ind=0, am=1, be=0, %b[22] movaqp,1 area=10, ind=16, am=0, be=0, %b[21] movaqp,2 area=10, ind=0, am=1, be=0, %b[46] movaqp,3 area=10, ind=16, am=0, be=0, %b[45] } { loop_mode qpshufb,0 %b[30], %g19, %r24, %b[51] qpshufb,1 %b[29], %g18, %r24, %b[52] qpshufb,3 %b[29], %g18, %r7, %g18 qpshufb,4 %b[30], %g19, %r7, %g19 movaqp,0 area=11, ind=0, am=1, be=0, %b[26] movaqp,1 area=11, ind=16, am=0, be=0, %b[25] movaqp,2 area=11, ind=0, am=1, be=0, %b[50] movaqp,3 area=11, ind=16, am=0, be=0, %b[49] } { loop_mode qpshufb,0 %b[34], %g23, %r24, %b[55] qpshufb,1 %b[33], %g22, %r24, %b[56] qpshufb,3 %b[33], %g22, %r7, %g22 qpshufb,4 %b[34], %g23, %r7, %g23 movaqp,0 area=12, ind=0, am=1, be=0, %b[30] movaqp,1 area=12, ind=16, am=0, be=0, %b[29] movaqp,2 area=12, ind=0, am=1, be=0, %b[54] movaqp,3 area=12, ind=16, am=0, be=0, %b[53] } { loop_mode qpxor,0 %g27, %r6, %b[59] qpxor,1 %g26, %r6, %b[60] qpxor,3 %b[38], %r6, %b[61] qpxor,4 %b[37], %r6, %b[62] movaqp,0 area=13, ind=0, am=1, be=0, %b[34] movaqp,1 area=13, ind=16, am=0, be=0, %b[33] movaqp,2 area=13, ind=0, am=1, be=0, %b[58] movaqp,3 area=13, ind=16, am=0, be=0, %b[57] } { loop_mode qpxor,0 %g31, %r6, %b[63] qpxor,1 %g30, %r6, %b[64] qpfmuls,2 %b[60], %b[31], %b[60] qpxor,3 %b[42], %r6, %b[65] qpxor,4 %b[41], %r6, %b[66] qpfmuls,5 %b[62], %b[39], %b[62] movaqp,0 area=14, ind=0, am=1, be=0, %b[68] movaqp,1 area=14, ind=16, am=0, be=0, %b[67] movaqp,2 area=14, ind=0, am=1, be=0, %b[70] movaqp,3 area=14, ind=16, am=0, be=0, %b[69] } { loop_mode qpfmuls,0 %b[63], %b[43], %b[63] qpfmuls,1 %b[64], %b[47], %b[64] qpfmuls,2 %b[59], %b[27], %b[59] qpfmuls,3 %b[65], %b[51], %b[65] qpxor,4 %b[22], %r6, %b[71] qpfmuls,5 %b[61], %b[35], %b[61] movaqp,0 area=15, ind=0, am=1, be=0, %b[73] movaqp,1 area=15, ind=16, am=0, be=0, %b[72] movaqp,2 area=15, ind=0, am=1, be=0, %b[75] movaqp,3 area=15, ind=16, am=0, be=0, %b[74] } { loop_mode qpxor,0 %b[21], %r6, %b[76] qpxor,1 %b[45], %r6, %b[77] qpxor,3 %b[46], %r6, %b[78] qpxor,4 %b[26], %r6, %b[79] qpfmuls,5 %b[66], %b[55], %b[66] movaqp,0 area=16, ind=0, am=1, be=0, %b[81] movaqp,1 area=16, ind=16, am=0, be=0, %b[80] movaqp,2 area=16, ind=0, am=1, be=0, %b[83] movaqp,3 area=16, ind=16, am=0, be=0, %b[82] } { loop_mode qpfmuls,0 %b[77], %g28, %b[77] qpfmuls,2 %b[76], %g20, %b[76] qpfmuls,3 %b[78], %g24, %b[78] qpxor,4 %b[30], %r6, %b[84] qpfmuls,5 %b[71], %g16, %b[71] movaqp,0 area=17, ind=0, am=1, be=0, %b[86] movaqp,1 area=17, ind=16, am=0, be=0, %b[85] movaqp,2 area=17, ind=0, am=1, be=0, %b[88] movaqp,3 area=17, ind=16, am=0, be=0, %b[87] } { loop_mode qpxor,0 %b[29], %r6, %b[89] qpxor,1 %b[53], %r6, %b[90] qpxor,3 %b[54], %r6, %b[91] qpxor,4 %b[34], %r6, %b[92] qpfmuls,5 %b[84], %b[28], %b[84] movaqp,0 area=18, ind=0, am=1, be=0, %b[94] movaqp,1 area=18, ind=16, am=0, be=0, %b[93] movaqp,2 area=18, ind=0, am=1, be=0, %b[96] movaqp,3 area=18, ind=16, am=0, be=0, %b[95] } { loop_mode qpfmuls,0 %b[90], %b[40], %b[90] qpxor,1 %b[33], %r6, %b[97] qpfmuls,2 %b[89], %b[32], %b[89] qpfmuls,3 %b[92], %b[44], %b[92] qpxor,4 %b[58], %r6, %b[98] qpfmuls,5 %b[91], %b[36], %b[91] movaqp,0 area=19, ind=0, am=1, be=0, %b[100] movaqp,1 area=19, ind=16, am=0, be=0, %b[99] movaqp,2 area=19, ind=0, am=1, be=0, %b[102] movaqp,3 area=19, ind=16, am=0, be=0, %b[101] } { loop_mode qpxor,0 %b[57], %r6, %b[103] qpxor,1 %b[25], %r6, %b[104] qpfmuls,2 %b[97], %b[48], %b[97] qpxor,3 %b[49], %r6, %b[105] qpxor,4 %b[50], %r6, %b[106] qpfmuls,5 %b[98], %b[52], %b[98] } { loop_mode qpfmuls,0 %b[79], %b[19], %b[79] qpfmuls,1 %b[104], %b[23], %b[104] qpfmuls,2 %b[103], %b[56], %b[103] qpfmuls,3 %b[106], %g18, %b[106] qpshufb,4 %g27, %g27, %r25, %g27 qpfmuls,5 %b[105], %g22, %b[105] } { loop_mode qpshufb,0 %g26, %g26, %r25, %g26 qpshufb,1 %b[37], %b[37], %r25, %b[37] qpshufb,3 %b[38], %b[38], %r25, %b[38] qpshufb,4 %g31, %g31, %r25, %g31 qpfmul_hadds,5 %g27, %b[27], %b[59], %g27 } { loop_mode qpfmul_hadds,0 %b[37], %b[39], %b[62], %b[27] qpfmul_hadds,2 %g26, %b[31], %b[60], %g26 qpfmul_hadds,3 %g31, %b[43], %b[63], %g31 qpshufb,4 %g30, %g30, %r25, %g30 qpfmul_hadds,5 %b[38], %b[35], %b[61], %b[31] } { loop_mode qpshufb,0 %b[41], %b[41], %r25, %b[35] qpshufb,1 %b[30], %b[30], %r25, %b[30] qpshufb,3 %b[29], %b[29], %r25, %b[29] qpshufb,4 %b[42], %b[42], %r25, %b[37] qpfmul_hadds,5 %g30, %b[47], %b[64], %g30 } { loop_mode qpshufb,0 %b[54], %b[54], %r25, %b[38] qpshufb,1 %b[53], %b[53], %r25, %b[39] qpfmul_hadds,2 %b[35], %b[55], %b[66], %b[35] qpshufb,3 %b[34], %b[34], %r25, %b[34] qpshufb,4 %b[33], %b[33], %r25, %b[33] qpfmul_hadds,5 %b[37], %b[51], %b[65], %b[37] } { loop_mode qpshufb,0 %b[58], %b[58], %r25, %b[41] qpshufb,1 %b[57], %b[57], %r25, %b[42] qpfmul_hadds,2 %b[30], %b[28], %b[84], %b[28] qpfmul_hadds,3 %b[29], %b[32], %b[89], %b[29] qpfmul_hadds,4 %b[34], %b[44], %b[92], %b[32] qpfmul_hadds,5 %b[33], %b[48], %b[97], %b[30] } { loop_mode qpfmul_hadds,0 %b[38], %b[36], %b[91], %b[34] qpfmul_hadds,1 %b[41], %b[52], %b[98], %b[36] qpfmul_hadds,2 %b[39], %b[40], %b[90], %b[33] qpshufb,3 %b[21], %b[21], %r25, %b[21] qpshufb,4 %b[22], %b[22], %r25, %b[22] } { loop_mode qpshufb,0 %b[46], %b[46], %r25, %b[39] qpshufb,1 %b[45], %b[45], %r25, %b[40] qpfmul_hadds,2 %b[42], %b[56], %b[103], %b[38] qpshufb,3 %b[26], %b[26], %r25, %b[26] qpshufb,4 %b[25], %b[25], %r25, %b[25] qpfmul_hadds,5 %b[21], %g20, %b[76], %g20 } { loop_mode qpfmul_hadds,0 %b[40], %g28, %b[77], %g28 qpshufb,1 %b[50], %b[50], %r25, %b[21] qpfmul_hadds,2 %b[39], %g24, %b[78], %g24 qpfmul_hadds,3 %b[26], %b[19], %b[79], %b[19] qpshufb,4 %b[49], %b[49], %r25, %b[41] qpfmul_hadds,5 %b[22], %g16, %b[71], %g16 } { loop_mode qpxor,0 %b[67], %r6, %b[21] qpxor,1 %b[68], %r6, %b[23] qpfmul_hadds,2 %b[21], %g18, %b[106], %g18 qpfmul_hadds,3 %b[41], %g22, %b[105], %g22 qpshufb,4 %g27, %g27, %r22, %g27 qpfmul_hadds,5 %b[25], %b[23], %b[104], %b[22] } { loop_mode qpshufb,0 %g26, %g26, %r22, %g26 qpshufb,1 %b[27], %b[27], %r22, %b[26] qpshufb,3 %b[31], %b[31], %r22, %b[25] qpshufb,4 %g31, %g31, %r22, %g31 } { loop_mode qpxor,0 %b[73], %r6, %b[27] qpxor,1 %b[72], %r6, %b[31] } { loop_mode qpshufb,3 %g30, %g30, %r22, %g30 qpshufb,4 %b[37], %b[37], %r22, %b[37] } { loop_mode qpshufb,0 %b[35], %b[35], %r22, %b[35] qpshufb,1 %b[28], %b[28], %r22, %b[28] qpshufb,3 %b[29], %b[29], %r22, %b[29] qpshufb,4 %b[32], %b[32], %r22, %b[32] } { loop_mode qpfsubs,0 %g27, %b[28], %b[39] qpshufb,1 %b[34], %b[34], %r22, %b[34] qpfadds,2 %g27, %b[28], %g27 qpfsubs,3 %g26, %b[29], %b[40] qpshufb,4 %b[30], %b[30], %r22, %b[30] qpfsubs,5 %g31, %b[32], %b[41] } { loop_mode qpshufb,0 %b[33], %b[33], %r22, %b[28] qpshufb,1 %b[36], %b[36], %r22, %b[33] qpfsubs,2 %b[25], %b[34], %b[36] qpfadds,3 %g26, %b[29], %g26 qpshufb,4 %g20, %g20, %r22, %g20 qpfsubs,5 %g30, %b[30], %b[42] } { loop_mode qpfsubs,0 %b[37], %b[33], %b[43] qpshufb,1 %b[38], %b[38], %r22, %b[29] qpfsubs,2 %b[26], %b[28], %b[38] qpfadds,3 %g31, %b[32], %g31 qpshufb,4 %g16, %g16, %r22, %g16 qpfadds,5 %g30, %b[30], %g30 } { loop_mode qpshufb,0 %g28, %g28, %r22, %g28 qpshufb,1 %g24, %g24, %r22, %g24 qpfsubs,2 %b[35], %b[29], %b[30] qpfadds,3 %g21, %g20, %b[32] qpshufb,4 %b[22], %b[22], %r22, %b[22] qpfsubs,5 %g21, %g20, %g20 } { loop_mode qpfadds,0 %b[26], %b[28], %b[19] qpshufb,1 %b[19], %b[19], %r22, %g21 qpfadds,2 %b[25], %b[34], %b[25] qpfadds,3 %b[24], %b[22], %b[26] qpshufb,4 %g22, %g22, %r22, %g22 qpfsubs,5 %b[24], %b[22], %b[22] } { loop_mode qpshufb,0 %g18, %g18, %r22, %g18 qpfadds,1 %b[37], %b[33], %b[24] qpfadds,2 %b[35], %b[29], %b[28] qpfadds,3 %g17, %g16, %b[29] qpfsubs,4 %g17, %g16, %g16 qpfadds,5 %g23, %g22, %g17 } { loop_mode qpfadds,0 %g29, %g28, %b[33] qpfsubs,1 %g29, %g28, %g28 qpfadds,2 %b[20], %g21, %g29 qpshufb,4 %b[39], %b[39], %r25, %g23 qpfsubs,5 %g23, %g22, %g22 } { loop_mode qpfsubs,0 %g25, %g24, %g24 qpfsubs,1 %g19, %g18, %g25 qpfsubs,2 %b[20], %g21, %g21 qpfsubs,3 %b[32], %g26, %b[34] qpfadds,4 %b[32], %g26, %g26 qpfadds,5 %g25, %g24, %b[20] } { loop_mode qpfadds,2 %g19, %g18, %g18 qpshufb,3 %b[40], %b[40], %r25, %g19 qpshufb,4 %b[36], %b[36], %r25, %b[32] qpfsubs,5 %b[26], %g30, %b[35] } { loop_mode qpfadds,3 %b[26], %g30, %g30 qpfadds,4 %b[29], %g27, %g27 qpfsubs,5 %b[29], %g27, %b[36] } { loop_mode qpshufb,0 %b[38], %b[38], %r25, %b[26] qpshufb,1 %b[42], %b[42], %r25, %b[29] qpfsubs,2 %g29, %g31, %b[38] qpshufb,3 %b[41], %b[41], %r25, %b[37] qpshufb,4 %b[30], %b[30], %r25, %b[30] } { loop_mode qpshufb,0 %b[43], %b[43], %r25, %b[39] qpxor,1 %g23, %r23, %g23 qpfsubs,2 %b[33], %b[19], %b[40] qpfadds,3 %b[20], %b[25], %b[41] qpfadds,4 %g17, %b[28], %b[25] qpfsubs,5 %b[20], %b[25], %b[20] } { loop_mode qpxor,0 %g19, %r23, %g19 qpxor,1 %b[32], %r23, %b[32] qpfadds,2 %g29, %g31, %g29 qpxor,3 %b[37], %r23, %b[37] qpxor,4 %b[30], %r23, %b[30] qpfadds,5 %b[33], %b[19], %g31 } { loop_mode qpxor,0 %b[29], %r23, %b[19] qpxor,1 %b[26], %r23, %b[26] qpfadds,2 %g20, %g19, %b[29] qpfsubs,3 %g17, %b[28], %g17 qpfsubs,4 %g21, %b[37], %b[28] qpfadds,5 %g21, %b[37], %g21 } { loop_mode qpxor,0 %b[39], %r23, %b[33] qpfsubs,1 %g20, %g19, %g19 qpfadds,2 %g18, %b[24], %g20 qpfsubs,3 %g18, %b[24], %g18 qpfsubs,4 %g22, %b[30], %g22 qpfadds,5 %g22, %b[30], %b[24] } { loop_mode qpfsubs,0 %g16, %g23, %b[30] qpfadds,1 %g16, %g23, %g16 qpfadds,2 %g24, %b[32], %g23 } { loop_mode qpfadds,0 %g28, %b[26], %b[37] qpfsubs,1 %g24, %b[32], %g24 qpfsubs,2 %g28, %b[26], %g28 } { loop_mode qpfsubs,0 %g25, %b[33], %b[22] qpfadds,1 %g25, %b[33], %g25 qpfsubs,2 %b[22], %b[19], %b[26] qpxor,3 %b[75], %r6, %b[32] qpxor,4 %b[83], %r6, %b[33] qpfadds,5 %b[22], %b[19], %b[19] } { loop_mode qpxor,3 %b[74], %r6, %b[39] qpxor,4 %b[82], %r6, %b[42] } { loop_mode qpxor,3 %b[86], %r6, %b[43] qpxor,4 %b[94], %r6, %b[44] } { loop_mode qpxor,0 %b[85], %r6, %b[45] qpxor,1 %b[93], %r6, %b[46] qpxor,3 %b[96], %r6, %b[47] qpxor,4 %b[102], %r6, %b[48] } { loop_mode qpxor,0 %b[95], %r6, %b[49] qpxor,1 %b[101], %r6, %b[50] qpshufb,3 %b[41], %g27, %r24, %b[51] qpshufb,4 %g31, %g26, %r24, %b[52] } { loop_mode qpshufb,0 %b[20], %b[36], %r24, %b[53] qpshufb,1 %b[40], %b[34], %r24, %b[54] qpshufb,3 %g20, %g29, %r24, %b[55] qpshufb,4 %b[25], %g30, %r24, %b[56] qpfmuls,5 %b[23], %b[51], %b[23] } { loop_mode qpshufb,0 %g18, %b[38], %r24, %b[57] qpshufb,1 %g17, %b[35], %r24, %b[58] qpfmuls,2 %b[43], %b[53], %b[43] qpshufb,3 %g24, %b[30], %r24, %b[59] qpshufb,4 %g28, %g19, %r24, %b[60] qpfmuls,5 %b[27], %b[52], %b[27] } { loop_mode qpshufb,0 %g23, %g16, %r24, %b[61] qpshufb,1 %b[37], %b[29], %r24, %b[62] qpfmuls,2 %b[44], %b[54], %b[44] qpshufb,3 %b[22], %b[28], %r24, %b[63] qpshufb,4 %g22, %b[26], %r24, %b[64] qpfmuls,5 %b[21], %b[55], %b[21] } { loop_mode qpshufb,0 %g25, %g21, %r24, %b[65] qpshufb,1 %b[24], %b[19], %r24, %b[66] qpfmuls,2 %b[45], %b[57], %b[45] qpfmuls,3 %b[31], %b[56], %b[31] qpfmuls,4 %b[32], %b[59], %b[32] qpfmuls,5 %b[33], %b[60], %b[33] } { loop_mode qpfmuls,0 %b[46], %b[58], %b[46] qpfmuls,1 %b[47], %b[61], %b[47] qpfmuls,2 %b[48], %b[62], %b[48] qpfmuls,3 %b[42], %b[64], %b[42] qpxor,4 %b[70], %r6, %b[71] qpfmuls,5 %b[39], %b[63], %b[39] } { loop_mode qpfmuls,0 %b[50], %b[66], %b[50] qpxor,1 %b[81], %r6, %b[76] qpfmuls,2 %b[49], %b[65], %b[49] } { loop_mode qpxor,4 %b[69], %r6, %b[77] } { loop_mode qpxor,1 %b[80], %r6, %b[78] qpxor,3 %b[88], %r6, %b[79] qpxor,4 %b[100], %r6, %b[84] } { loop_mode qpxor,0 %b[87], %r6, %b[89] qpxor,1 %b[99], %r6, %b[90] qpshufb,3 %g31, %g26, %r7, %g26 qpshufb,4 %b[40], %b[34], %r7, %g31 } { loop_mode qpshufb,0 %b[25], %g30, %r7, %g30 qpshufb,1 %g17, %b[35], %r7, %g17 qpshufb,3 %g28, %g19, %r7, %g19 qpshufb,4 %b[37], %b[29], %r7, %g28 qpfmuls,5 %b[79], %g31, %b[25] } { loop_mode qpshufb,0 %g22, %b[26], %r7, %g22 qpshufb,1 %b[24], %b[19], %r7, %b[19] qpfmuls,2 %b[77], %g30, %b[26] qpfmuls,3 %b[76], %g19, %b[29] qpfmuls,4 %b[84], %g28, %b[34] qpfmuls,5 %b[71], %g26, %b[24] } { loop_mode qpfmuls,0 %b[90], %b[19], %b[37] qpfmuls,1 %b[89], %g17, %b[40] qpfmuls,2 %b[78], %g22, %b[35] qpshufb,3 %b[68], %b[68], %r25, %b[68] qpshufb,4 %b[67], %b[67], %r25, %b[67] } { loop_mode qpshufb,0 %b[73], %b[73], %r25, %b[71] qpshufb,1 %b[72], %b[72], %r25, %b[72] qpfmul_hadds,3 %b[67], %b[55], %b[21], %b[21] qpfmul_hadds,5 %b[68], %b[51], %b[23], %b[23] } { loop_mode qpfmul_hadds,0 %b[72], %b[56], %b[31], %b[31] qpfmul_hadds,2 %b[71], %b[52], %b[27], %b[27] qpshufb,3 %b[75], %b[75], %r25, %b[51] qpshufb,4 %b[83], %b[83], %r25, %b[55] } { loop_mode qpshufb,0 %b[74], %b[74], %r25, %b[52] qpshufb,1 %b[82], %b[82], %r25, %b[56] qpshufb,3 %b[86], %b[86], %r25, %b[67] qpshufb,4 %b[94], %b[94], %r25, %b[68] qpfmul_hadds,5 %b[51], %b[59], %b[32], %b[32] } { loop_mode qpshufb,0 %b[85], %b[85], %r25, %b[51] qpshufb,1 %b[93], %b[93], %r25, %b[59] qpfmul_hadds,2 %b[52], %b[63], %b[39], %b[39] qpshufb,3 %b[96], %b[96], %r25, %b[71] qpshufb,4 %b[102], %b[102], %r25, %b[72] qpfmul_hadds,5 %b[55], %b[60], %b[33], %b[33] } { loop_mode qpshufb,0 %b[95], %b[95], %r25, %b[52] qpshufb,1 %b[101], %b[101], %r25, %b[55] qpfmul_hadds,2 %b[56], %b[64], %b[42], %b[42] qpfmul_hadds,3 %b[67], %b[53], %b[43], %b[43] qpfmul_hadds,4 %b[68], %b[54], %b[44], %b[44] qpfmul_hadds,5 %b[71], %b[61], %b[47], %b[47] } { loop_mode qpfmul_hadds,0 %b[59], %b[58], %b[46], %b[46] qpfmul_hadds,1 %b[52], %b[65], %b[49], %b[49] qpfmul_hadds,2 %b[51], %b[57], %b[45], %b[45] qpshufb,3 %b[70], %b[70], %r25, %b[51] qpshufb,4 %b[69], %b[69], %r25, %b[52] qpfmul_hadds,5 %b[72], %b[62], %b[48], %b[48] } { loop_mode qpshufb,0 %b[81], %b[81], %r25, %b[53] qpshufb,1 %b[88], %b[88], %r25, %b[54] qpfmul_hadds,2 %b[55], %b[66], %b[50], %b[50] qpfmul_hadds,3 %b[52], %g30, %b[26], %g30 qpshufb,4 %b[80], %b[80], %r25, %b[55] qpfmul_hadds,5 %b[51], %g26, %b[24], %g26 } { loop_mode qpfmul_hadds,0 %b[53], %g19, %b[29], %g19 qpshufb,1 %b[87], %b[87], %r25, %b[24] qpfmul_hadds,2 %b[54], %g31, %b[25], %g31 qpshufb,3 %b[100], %b[100], %r25, %b[26] qpshufb,4 %b[99], %b[99], %r25, %b[51] qpfmul_hadds,5 %b[55], %g22, %b[35], %g22 } { loop_mode qpshufb,0 %b[20], %b[36], %r7, %b[20] qpshufb,1 %b[41], %g27, %r7, %g27 qpfmul_hadds,2 %b[24], %g17, %b[40], %g17 qpfmul_hadds,3 %b[51], %b[19], %b[37], %b[19] qpshufb,4 %b[23], %b[23], %r22, %b[23] qpfmul_hadds,5 %b[26], %g28, %b[34], %g28 } { loop_mode qpshufb,0 %b[27], %b[27], %r22, %b[24] qpshufb,1 %b[31], %b[31], %r22, %b[25] qpshufb,3 %b[21], %b[21], %r22, %b[21] qpshufb,4 %g20, %g29, %r7, %g20 } { loop_mode qpshufb,0 %g18, %b[38], %r7, %g18 qpshufb,1 %g23, %g16, %r7, %g16 } { loop_mode qpshufb,3 %b[32], %b[32], %r22, %g23 qpshufb,4 %b[33], %b[33], %r22, %g29 } { loop_mode qpshufb,0 %b[39], %b[39], %r22, %b[26] qpshufb,1 %b[42], %b[42], %r22, %b[27] qpshufb,4 %b[43], %b[43], %r22, %b[29] } { loop_mode qpfsubs,0 %b[26], %b[27], %b[36] qpshufb,1 %b[45], %b[45], %r22, %b[31] qpfsubs,2 %b[23], %b[24], %b[34] qpshufb,3 %b[44], %b[44], %r22, %b[32] qpshufb,4 %b[47], %b[47], %r22, %b[33] qpfsubs,5 %b[21], %b[25], %b[35] } { loop_mode qpshufb,0 %b[46], %b[46], %r22, %b[37] qpshufb,1 %b[49], %b[49], %r22, %b[38] qpfadds,2 %b[23], %b[24], %b[23] qpfsubs,3 %b[29], %b[32], %b[41] qpshufb,4 %b[48], %b[48], %r22, %b[39] qpfsubs,5 %g23, %g29, %b[40] } { loop_mode qpfadds,0 %b[21], %b[25], %g25 qpshufb,1 %b[50], %b[50], %r22, %b[24] qpfsubs,2 %b[31], %b[37], %b[43] qpshufb,3 %g24, %b[30], %r7, %g24 qpshufb,4 %g25, %g21, %r7, %g21 qpfsubs,5 %b[33], %b[39], %b[42] } { loop_mode qpshufb,0 %b[22], %b[28], %r7, %b[22] qpshufb,1 %g26, %g26, %r22, %g26 qpfsubs,2 %b[38], %b[24], %b[21] qpfadds,3 %g23, %g29, %g23 qpshufb,4 %g30, %g30, %r22, %g30 qpfadds,5 %b[26], %b[27], %g29 } { loop_mode qpfadds,0 %b[38], %b[24], %b[24] qpshufb,1 %g17, %g17, %r22, %g17 qpfadds,2 %b[31], %b[37], %b[25] qpshufb,3 %g22, %g22, %r22, %g22 qpshufb,4 %g31, %g31, %r22, %g31 qpfadds,5 %b[33], %b[39], %b[26] } { loop_mode qpshufb,0 %g19, %g19, %r22, %g19 qpshufb,1 %g28, %g28, %r22, %g28 qpfadds,2 %b[29], %b[32], %b[27] qpfadds,3 %g20, %g30, %b[28] qpshufb,4 %b[19], %b[19], %r22, %b[19] qpfsubs,5 %g20, %g30, %g20 } { loop_mode qpfsubs,0 %g27, %g26, %g30 qpfadds,1 %g27, %g26, %g26 qpfadds,2 %g18, %g17, %b[20] qpfadds,3 %b[20], %g31, %g27 qpfsubs,4 %b[20], %g31, %g31 qpfsubs,5 %g21, %b[19], %b[29] } { loop_mode qpfsubs,0 %g18, %g17, %g17 qpfsubs,1 %g24, %g19, %g18 qpfadds,2 %g24, %g19, %g19 qpfadds,3 %b[22], %g22, %g24 qpfsubs,4 %b[22], %g22, %g22 qpfadds,5 %g21, %b[19], %g21 } { loop_mode qpfsubs,0 %g16, %g28, %g16 qpfadds,2 %g16, %g28, %b[19] } { loop_mode qpfsubs,3 %b[28], %g25, %g28 qpfadds,4 %b[28], %g25, %g25 } { loop_mode qpfadds,0 %g26, %b[23], %b[31] qpshufb,1 %b[35], %b[35], %r25, %b[22] qpfsubs,2 %g26, %b[23], %g26 qpshufb,3 %b[34], %b[34], %r25, %b[28] qpshufb,4 %b[40], %b[40], %r25, %b[30] } { loop_mode qpshufb,0 %b[36], %b[36], %r25, %b[23] qpshufb,1 %b[21], %b[21], %r25, %b[21] qpfsubs,2 %b[20], %b[25], %b[27] qpfadds,3 %g27, %b[27], %b[32] qpfsubs,4 %g27, %b[27], %g27 qpfadds,5 %g21, %b[24], %b[33] } { loop_mode qpfadds,0 %b[20], %b[25], %b[20] qpshufb,1 %b[43], %b[43], %r25, %b[34] qpfadds,2 %g19, %g23, %b[25] qpshufb,3 %b[42], %b[42], %r25, %b[35] qpshufb,4 %b[41], %b[41], %r25, %b[36] qpfsubs,5 %g21, %b[24], %g21 } { loop_mode qpxor,0 %b[22], %r23, %b[22] qpxor,1 %b[23], %r23, %b[23] qpfsubs,2 %b[19], %b[26], %g23 qpfsubs,3 %g19, %g23, %g19 qpfadds,4 %g24, %g29, %b[24] qpfsubs,5 %g24, %g29, %g24 } { loop_mode qpfadds,0 %b[19], %b[26], %b[19] qpxor,1 %b[28], %r23, %g29 qpfadds,2 %g20, %b[22], %b[26] qpxor,3 %b[30], %r23, %b[28] qpxor,4 %b[35], %r23, %b[30] stqp,5 %r33, %r0, %g28 } { loop_mode qpxor,0 %b[21], %r23, %g28 qpxor,1 %b[34], %r23, %b[21] qpfsubs,2 %g30, %g29, %b[34] qpfadds,3 %g18, %b[28], %b[35] qpfsubs,4 %g18, %b[28], %g18 qpfadds,5 %g16, %b[30], %b[28] } { loop_mode qpfadds,0 %g30, %g29, %g29 qpxor,1 %b[36], %r23, %b[36] qpfsubs,2 %g20, %b[22], %g20 qpfsubs,3 %g16, %b[30], %g16 stqp,5 %r18, %r0, %g26 } { loop_mode qpfsubs,0 %g22, %b[23], %g26 qpfadds,1 %g22, %b[23], %g22 qpfadds,2 %b[29], %g28, %g30 stqp,5 %r28, %r0, %g25 } { loop_mode qpfsubs,0 %g17, %b[21], %g25 qpfadds,1 %g17, %b[21], %g17 qpfsubs,2 %b[29], %g28, %g28 stqp,5 %r2, %r0, %b[31] } { loop_mode qpfsubs,0 %g31, %b[36], %b[21] qpfadds,1 %g31, %b[36], %g31 stqp,2 %r16, %r0, %g27 stqp,5 %r20, %r0, %b[32] } { loop_mode stqp,2 %r37, %r0, %b[27] stqp,5 %r26, %r0, %b[20] } { loop_mode stqp,2 %r13, %r0, %g19 stqp,5 %r21, %r0, %b[25] } { loop_mode stqp,2 %r36, %r0, %g21 stqp,5 %r3, %r0, %g23 } { loop_mode stqp,2 %r9, %r0, %b[33] stqp,5 %r27, %r0, %b[24] } { loop_mode stqp,2 %r32, %r0, %g24 stqp,5 %r17, %r0, %b[19] } { loop_mode stqp,2 %r35, %r0, %b[26] stqp,5 %r15, %r0, %g29 } { loop_mode stqp,2 %r31, %r0, %g20 stqp,5 %r19, %r0, %b[34] } { loop_mode stqp,2 %r1, %r0, %b[35] stqp,5 %r12, %r0, %g18 } { loop_mode stqp,2 %r40, %r0, %g22 stqp,5 %r30, %r0, %g26 } { loop_mode stqp,2 %r38, %r0, %g30 stqp,5 %r4, %r0, %b[28] } { loop_mode stqp,2 %r34, %r0, %g28 stqp,5 %r39, %r0, %g17 } { loop_mode stqp,2 %r14, %r0, %g16 stqp,5 %r5, %r0, %g31 } { loop_mode ct %ctpr1 ? %NOT_LOOP_END alc alcf=1, alct=1 stqp,2 %r29, %r0, %g25 addd,3,sm %r0, _f16s,_lts0lo 0x20, %r0 stqp,5 %r11, %r0, %b[21] }
Теоретическая скорость: 64 комплексных числа за 115 тактов (64/115) = 4.45 Байт/такт
Четверная теоретическая скорость: 17.81 Байт/такт
Замеры скорости
5. stage_radix4_2x_simd128_noConj
Здесь происходит ручная раскрутка алгоритма stage_radix4_simd128_noConj в 2 раза.
Код на Си
void stage_radix4_2x_simd128_noConj(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coefC_a, myComplex *coefD_a, myComplex *coefE_a, myComplex *coefC_b, myComplex *coefD_b, myComplex *coefE_b) { __v2di *xy0_in = (__v2di*)&data_in[ 0]; __v2di *zw0_in = (__v2di*)&data_in[ 2]; __v2di *xy1_in = (__v2di*)&data_in[ 4]; __v2di *zw1_in = (__v2di*)&data_in[ 6]; __v2di *xy2_in = (__v2di*)&data_in[ 8]; __v2di *zw2_in = (__v2di*)&data_in[10]; __v2di *xy3_in = (__v2di*)&data_in[12]; __v2di *zw3_in = (__v2di*)&data_in[14]; __v2di *xy4_in = (__v2di*)&data_in[16]; __v2di *zw4_in = (__v2di*)&data_in[18]; __v2di *xy5_in = (__v2di*)&data_in[20]; __v2di *zw5_in = (__v2di*)&data_in[22]; __v2di *xy6_in = (__v2di*)&data_in[24]; __v2di *zw6_in = (__v2di*)&data_in[26]; __v2di *xy7_in = (__v2di*)&data_in[28]; __v2di *zw7_in = (__v2di*)&data_in[30]; __v2di *c0a_in = (__v2di*)&coefC_a[0]; __v2di *c1a_in = (__v2di*)&coefC_a[2]; __v2di *c2a_in = (__v2di*)&coefC_a[4]; __v2di *c3a_in = (__v2di*)&coefC_a[6]; __v2di *d0a_in = (__v2di*)&coefD_a[0]; __v2di *d1a_in = (__v2di*)&coefD_a[2]; __v2di *d2a_in = (__v2di*)&coefD_a[4]; __v2di *d3a_in = (__v2di*)&coefD_a[6]; __v2di *e0a_in = (__v2di*)&coefE_a[0]; __v2di *e1a_in = (__v2di*)&coefE_a[2]; __v2di *e2a_in = (__v2di*)&coefE_a[4]; __v2di *e3a_in = (__v2di*)&coefE_a[6]; __v2di *c0b_in = (__v2di*)&coefC_b[0*data_count/16]; __v2di *c1b_in = (__v2di*)&coefC_b[1*data_count/16]; __v2di *c2b_in = (__v2di*)&coefC_b[2*data_count/16]; __v2di *c3b_in = (__v2di*)&coefC_b[3*data_count/16]; __v2di *d0b_in = (__v2di*)&coefD_b[0*data_count/16]; __v2di *d1b_in = (__v2di*)&coefD_b[1*data_count/16]; __v2di *d2b_in = (__v2di*)&coefD_b[2*data_count/16]; __v2di *d3b_in = (__v2di*)&coefD_b[3*data_count/16]; __v2di *e0b_in = (__v2di*)&coefE_b[0*data_count/16]; __v2di *e1b_in = (__v2di*)&coefE_b[1*data_count/16]; __v2di *e2b_in = (__v2di*)&coefE_b[2*data_count/16]; __v2di *e3b_in = (__v2di*)&coefE_b[3*data_count/16]; __v2di *out_0 = (__v2di*)&data_out[ 0*data_count/16]; __v2di *out_1 = (__v2di*)&data_out[ 1*data_count/16]; __v2di *out_2 = (__v2di*)&data_out[ 2*data_count/16]; __v2di *out_3 = (__v2di*)&data_out[ 3*data_count/16]; __v2di *out_4 = (__v2di*)&data_out[ 4*data_count/16]; __v2di *out_5 = (__v2di*)&data_out[ 5*data_count/16]; __v2di *out_6 = (__v2di*)&data_out[ 6*data_count/16]; __v2di *out_7 = (__v2di*)&data_out[ 7*data_count/16]; __v2di *out_8 = (__v2di*)&data_out[ 8*data_count/16]; __v2di *out_9 = (__v2di*)&data_out[ 9*data_count/16]; __v2di *out_10 = (__v2di*)&data_out[10*data_count/16]; __v2di *out_11 = (__v2di*)&data_out[11*data_count/16]; __v2di *out_12 = (__v2di*)&data_out[12*data_count/16]; __v2di *out_13 = (__v2di*)&data_out[13*data_count/16]; __v2di *out_14 = (__v2di*)&data_out[14*data_count/16]; __v2di *out_15 = (__v2di*)&data_out[15*data_count/16]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/32; ++i) { __v2di xy0 = xy0_in[16*i]; __v2di zw0 = zw0_in[16*i]; __v2di xy1 = xy1_in[16*i]; __v2di zw1 = zw1_in[16*i]; __v2di c0 = c0a_in[4*i]; __v2di d0 = d0a_in[4*i]; __v2di e0 = e0a_in[4*i]; __v2di xy2 = xy2_in[16*i]; __v2di zw2 = zw2_in[16*i]; __v2di xy3 = xy3_in[16*i]; __v2di zw3 = zw3_in[16*i]; __v2di c1 = c1a_in[4*i]; __v2di d1 = d1a_in[4*i]; __v2di e1 = e1a_in[4*i]; __v2di xy4 = xy4_in[16*i]; __v2di zw4 = zw4_in[16*i]; __v2di xy5 = xy5_in[16*i]; __v2di zw5 = zw5_in[16*i]; __v2di c2 = c2a_in[4*i]; __v2di d2 = d2a_in[4*i]; __v2di e2 = e2a_in[4*i]; __v2di xy6 = xy6_in[16*i]; __v2di zw6 = zw6_in[16*i]; __v2di xy7 = xy7_in[16*i]; __v2di zw7 = zw7_in[16*i]; __v2di c3 = c3a_in[4*i]; __v2di d3 = d3a_in[4*i]; __v2di e3 = e3a_in[4*i]; __v2di x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c2 = __builtin_e2k_qpshufb(c2, c2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c3 = __builtin_e2k_qpshufb(c3, c3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d0 = __builtin_e2k_qpshufb(d0, d0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d1 = __builtin_e2k_qpshufb(d1, d1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d2 = __builtin_e2k_qpshufb(d2, d2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d3 = __builtin_e2k_qpshufb(d3, d3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e0 = __builtin_e2k_qpshufb(e0, e0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e1 = __builtin_e2k_qpshufb(e1, e1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e2 = __builtin_e2k_qpshufb(e2, e2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e3 = __builtin_e2k_qpshufb(e3, e3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy0_real = __builtin_e2k_qpfmuls( c0, y0); __v2di cy1_real = __builtin_e2k_qpfmuls( c1, y1); __v2di cy2_real = __builtin_e2k_qpfmuls( c2, y2); __v2di cy3_real = __builtin_e2k_qpfmuls( c3, y3); __v2di dz0_real = __builtin_e2k_qpfmuls( d0, z0); __v2di dz1_real = __builtin_e2k_qpfmuls( d1, z1); __v2di dz2_real = __builtin_e2k_qpfmuls( d2, z2); __v2di dz3_real = __builtin_e2k_qpfmuls( d3, z3); __v2di ew0_real = __builtin_e2k_qpfmuls( e0, w0); __v2di ew1_real = __builtin_e2k_qpfmuls( e1, w1); __v2di ew2_real = __builtin_e2k_qpfmuls( e2, w2); __v2di ew3_real = __builtin_e2k_qpfmuls( e3, w3); __v2di cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); __v2di cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); __v2di cy2_imag = __builtin_e2k_qpfmuls(swap_c2, y2); __v2di cy3_imag = __builtin_e2k_qpfmuls(swap_c3, y3); __v2di dz0_imag = __builtin_e2k_qpfmuls(swap_d0, z0); __v2di dz1_imag = __builtin_e2k_qpfmuls(swap_d1, z1); __v2di dz2_imag = __builtin_e2k_qpfmuls(swap_d2, z2); __v2di dz3_imag = __builtin_e2k_qpfmuls(swap_d3, z3); __v2di ew0_imag = __builtin_e2k_qpfmuls(swap_e0, w0); __v2di ew1_imag = __builtin_e2k_qpfmuls(swap_e1, w1); __v2di ew2_imag = __builtin_e2k_qpfmuls(swap_e2, w2); __v2di ew3_imag = __builtin_e2k_qpfmuls(swap_e3, w3); __v2di cy0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy0_real); __v2di cy1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy1_real); __v2di cy2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy2_real); __v2di cy3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy3_real); __v2di dz0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz0_real); __v2di dz1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz1_real); __v2di dz2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz2_real); __v2di dz3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz3_real); __v2di ew0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew0_real); __v2di ew1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew1_real); __v2di ew2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew2_real); __v2di ew3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew3_real); __v2di cy0_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy0_imag); __v2di cy1_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy1_imag); __v2di cy2_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy2_imag); __v2di cy3_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy3_imag); __v2di dz0_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz0_imag); __v2di dz1_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz1_imag); __v2di dz2_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz2_imag); __v2di dz3_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz3_imag); __v2di ew0_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew0_imag); __v2di ew1_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew1_imag); __v2di ew2_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew2_imag); __v2di ew3_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew3_imag); __v2di cy0 = __builtin_e2k_qppermb(cy0_ii, cy0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di cy1 = __builtin_e2k_qppermb(cy1_ii, cy1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di cy2 = __builtin_e2k_qppermb(cy2_ii, cy2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di cy3 = __builtin_e2k_qppermb(cy3_ii, cy3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz0 = __builtin_e2k_qppermb(dz0_ii, dz0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz1 = __builtin_e2k_qppermb(dz1_ii, dz1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz2 = __builtin_e2k_qppermb(dz2_ii, dz2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz3 = __builtin_e2k_qppermb(dz3_ii, dz3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew0 = __builtin_e2k_qppermb(ew0_ii, ew0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew1 = __builtin_e2k_qppermb(ew1_ii, ew1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew2 = __builtin_e2k_qppermb(ew2_ii, ew2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew3 = __builtin_e2k_qppermb(ew3_ii, ew3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di add02_0 = __builtin_e2k_qpfadds( x0, dz0); __v2di add02_1 = __builtin_e2k_qpfadds( x1, dz1); __v2di add02_2 = __builtin_e2k_qpfadds( x2, dz2); __v2di add02_3 = __builtin_e2k_qpfadds( x3, dz3); __v2di sub02_0 = __builtin_e2k_qpfsubs( x0, dz0); __v2di sub02_1 = __builtin_e2k_qpfsubs( x1, dz1); __v2di sub02_2 = __builtin_e2k_qpfsubs( x2, dz2); __v2di sub02_3 = __builtin_e2k_qpfsubs( x3, dz3); __v2di add13_0 = __builtin_e2k_qpfadds(cy0, ew0); __v2di add13_1 = __builtin_e2k_qpfadds(cy1, ew1); __v2di add13_2 = __builtin_e2k_qpfadds(cy2, ew2); __v2di add13_3 = __builtin_e2k_qpfadds(cy3, ew3); __v2di sub13_0 = __builtin_e2k_qpfsubs(cy0, ew0); __v2di sub13_1 = __builtin_e2k_qpfsubs(cy1, ew1); __v2di sub13_2 = __builtin_e2k_qpfsubs(cy2, ew2); __v2di sub13_3 = __builtin_e2k_qpfsubs(cy3, ew3); __v2di swap_sub13_0 = __builtin_e2k_qpshufb(sub13_0, sub13_0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_1 = __builtin_e2k_qpshufb(sub13_1, sub13_1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_2 = __builtin_e2k_qpshufb(sub13_2, sub13_2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_3 = __builtin_e2k_qpshufb(sub13_3, sub13_3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di sub13i_0 = __builtin_e2k_qpxor(swap_sub13_0, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_1 = __builtin_e2k_qpxor(swap_sub13_1, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_2 = __builtin_e2k_qpxor(swap_sub13_2, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_3 = __builtin_e2k_qpxor(swap_sub13_3, (__v2di){1LL<<31, 1LL<<31}); __v2di out0 = __builtin_e2k_qpfadds(add02_0, add13_0); __v2di out1 = __builtin_e2k_qpfadds(add02_1, add13_1); __v2di out2 = __builtin_e2k_qpfadds(add02_2, add13_2); __v2di out3 = __builtin_e2k_qpfadds(add02_3, add13_3); __v2di out4 = __builtin_e2k_qpfsubs(sub02_0, sub13i_0); __v2di out5 = __builtin_e2k_qpfsubs(sub02_1, sub13i_1); __v2di out6 = __builtin_e2k_qpfsubs(sub02_2, sub13i_2); __v2di out7 = __builtin_e2k_qpfsubs(sub02_3, sub13i_3); __v2di out8 = __builtin_e2k_qpfsubs(add02_0, add13_0); __v2di out9 = __builtin_e2k_qpfsubs(add02_1, add13_1); __v2di out10 = __builtin_e2k_qpfsubs(add02_2, add13_2); __v2di out11 = __builtin_e2k_qpfsubs(add02_3, add13_3); __v2di out12 = __builtin_e2k_qpfadds(sub02_0, sub13i_0); __v2di out13 = __builtin_e2k_qpfadds(sub02_1, sub13i_1); __v2di out14 = __builtin_e2k_qpfadds(sub02_2, sub13i_2); __v2di out15 = __builtin_e2k_qpfadds(sub02_3, sub13i_3); xy0 = out0; zw0 = out1; xy1 = out2; zw1 = out3; c0 = c0b_in[i]; d0 = d0b_in[i]; e0 = e0b_in[i]; xy2 = out4; zw2 = out5; xy3 = out6; zw3 = out7; c1 = c1b_in[i]; d1 = d1b_in[i]; e1 = e1b_in[i]; xy4 = out8; zw4 = out9; xy5 = out10; zw5 = out11; c2 = c2b_in[i]; d2 = d2b_in[i]; e2 = e2b_in[i]; xy6 = out12; zw6 = out13; xy7 = out14; zw7 = out15; c3 = c3b_in[i]; d3 = d3b_in[i]; e3 = e3b_in[i]; x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); w0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0706050403020100, 0x0706050403020100}); w1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0706050403020100, 0x0706050403020100}); y2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0706050403020100, 0x0706050403020100}); w2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0706050403020100, 0x0706050403020100}); y3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0706050403020100, 0x0706050403020100}); w3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c2 = __builtin_e2k_qpshufb(c2, c2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c3 = __builtin_e2k_qpshufb(c3, c3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d0 = __builtin_e2k_qpshufb(d0, d0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d1 = __builtin_e2k_qpshufb(d1, d1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d2 = __builtin_e2k_qpshufb(d2, d2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d3 = __builtin_e2k_qpshufb(d3, d3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e0 = __builtin_e2k_qpshufb(e0, e0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e1 = __builtin_e2k_qpshufb(e1, e1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e2 = __builtin_e2k_qpshufb(e2, e2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e3 = __builtin_e2k_qpshufb(e3, e3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); cy0_real = __builtin_e2k_qpfmuls( c0, y0); cy1_real = __builtin_e2k_qpfmuls( c1, y1); cy2_real = __builtin_e2k_qpfmuls( c2, y2); cy3_real = __builtin_e2k_qpfmuls( c3, y3); dz0_real = __builtin_e2k_qpfmuls( d0, z0); dz1_real = __builtin_e2k_qpfmuls( d1, z1); dz2_real = __builtin_e2k_qpfmuls( d2, z2); dz3_real = __builtin_e2k_qpfmuls( d3, z3); ew0_real = __builtin_e2k_qpfmuls( e0, w0); ew1_real = __builtin_e2k_qpfmuls( e1, w1); ew2_real = __builtin_e2k_qpfmuls( e2, w2); ew3_real = __builtin_e2k_qpfmuls( e3, w3); cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); cy2_imag = __builtin_e2k_qpfmuls(swap_c2, y2); cy3_imag = __builtin_e2k_qpfmuls(swap_c3, y3); dz0_imag = __builtin_e2k_qpfmuls(swap_d0, z0); dz1_imag = __builtin_e2k_qpfmuls(swap_d1, z1); dz2_imag = __builtin_e2k_qpfmuls(swap_d2, z2); dz3_imag = __builtin_e2k_qpfmuls(swap_d3, z3); ew0_imag = __builtin_e2k_qpfmuls(swap_e0, w0); ew1_imag = __builtin_e2k_qpfmuls(swap_e1, w1); ew2_imag = __builtin_e2k_qpfmuls(swap_e2, w2); ew3_imag = __builtin_e2k_qpfmuls(swap_e3, w3); cy0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy0_real); cy1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy1_real); cy2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy2_real); cy3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy3_real); dz0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz0_real); dz1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz1_real); dz2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz2_real); dz3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz3_real); ew0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew0_real); ew1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew1_real); ew2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew2_real); ew3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew3_real); cy0_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy0_imag); cy1_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy1_imag); cy2_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy2_imag); cy3_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy3_imag); dz0_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz0_imag); dz1_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz1_imag); dz2_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz2_imag); dz3_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz3_imag); ew0_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew0_imag); ew1_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew1_imag); ew2_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew2_imag); ew3_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew3_imag); cy0 = __builtin_e2k_qppermb(cy0_ii, cy0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); cy1 = __builtin_e2k_qppermb(cy1_ii, cy1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); cy2 = __builtin_e2k_qppermb(cy2_ii, cy2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); cy3 = __builtin_e2k_qppermb(cy3_ii, cy3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); dz0 = __builtin_e2k_qppermb(dz0_ii, dz0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); dz1 = __builtin_e2k_qppermb(dz1_ii, dz1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); dz2 = __builtin_e2k_qppermb(dz2_ii, dz2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); dz3 = __builtin_e2k_qppermb(dz3_ii, dz3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); ew0 = __builtin_e2k_qppermb(ew0_ii, ew0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); ew1 = __builtin_e2k_qppermb(ew1_ii, ew1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); ew2 = __builtin_e2k_qppermb(ew2_ii, ew2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); ew3 = __builtin_e2k_qppermb(ew3_ii, ew3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); add02_0 = __builtin_e2k_qpfadds( x0, dz0); add02_1 = __builtin_e2k_qpfadds( x1, dz1); add02_2 = __builtin_e2k_qpfadds( x2, dz2); add02_3 = __builtin_e2k_qpfadds( x3, dz3); sub02_0 = __builtin_e2k_qpfsubs( x0, dz0); sub02_1 = __builtin_e2k_qpfsubs( x1, dz1); sub02_2 = __builtin_e2k_qpfsubs( x2, dz2); sub02_3 = __builtin_e2k_qpfsubs( x3, dz3); add13_0 = __builtin_e2k_qpfadds(cy0, ew0); add13_1 = __builtin_e2k_qpfadds(cy1, ew1); add13_2 = __builtin_e2k_qpfadds(cy2, ew2); add13_3 = __builtin_e2k_qpfadds(cy3, ew3); sub13_0 = __builtin_e2k_qpfsubs(cy0, ew0); sub13_1 = __builtin_e2k_qpfsubs(cy1, ew1); sub13_2 = __builtin_e2k_qpfsubs(cy2, ew2); sub13_3 = __builtin_e2k_qpfsubs(cy3, ew3); swap_sub13_0 = __builtin_e2k_qpshufb(sub13_0, sub13_0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_1 = __builtin_e2k_qpshufb(sub13_1, sub13_1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_2 = __builtin_e2k_qpshufb(sub13_2, sub13_2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_3 = __builtin_e2k_qpshufb(sub13_3, sub13_3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); sub13i_0 = __builtin_e2k_qpxor(swap_sub13_0, (__v2di){1LL<<31, 1LL<<31}); sub13i_1 = __builtin_e2k_qpxor(swap_sub13_1, (__v2di){1LL<<31, 1LL<<31}); sub13i_2 = __builtin_e2k_qpxor(swap_sub13_2, (__v2di){1LL<<31, 1LL<<31}); sub13i_3 = __builtin_e2k_qpxor(swap_sub13_3, (__v2di){1LL<<31, 1LL<<31}); out_0[i] = __builtin_e2k_qpfadds(add02_0, add13_0); out_1[i] = __builtin_e2k_qpfadds(add02_1, add13_1); out_2[i] = __builtin_e2k_qpfadds(add02_2, add13_2); out_3[i] = __builtin_e2k_qpfadds(add02_3, add13_3); out_4[i] = __builtin_e2k_qpfsubs(sub02_0, sub13i_0); out_5[i] = __builtin_e2k_qpfsubs(sub02_1, sub13i_1); out_6[i] = __builtin_e2k_qpfsubs(sub02_2, sub13i_2); out_7[i] = __builtin_e2k_qpfsubs(sub02_3, sub13i_3); out_8[i] = __builtin_e2k_qpfsubs(add02_0, add13_0); out_9[i] = __builtin_e2k_qpfsubs(add02_1, add13_1); out_10[i] = __builtin_e2k_qpfsubs(add02_2, add13_2); out_11[i] = __builtin_e2k_qpfsubs(add02_3, add13_3); out_12[i] = __builtin_e2k_qpfadds(sub02_0, sub13i_0); out_13[i] = __builtin_e2k_qpfadds(sub02_1, sub13i_1); out_14[i] = __builtin_e2k_qpfadds(sub02_2, sub13i_2); out_15[i] = __builtin_e2k_qpfadds(sub02_3, sub13i_3); } }
Основной цикл на ассемблере
.L15211: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=2, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=2, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=4, disp=128 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=4, disp=160 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=6, disp=192 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=6, disp=224 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=1, abs=8, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=1, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=1, abs=10, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=1, abs=12, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=2, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=2, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=2, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=2, abs=24, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=2, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=2, abs=28, disp=0 } .L11903: { loop_mode disp %ctpr1, .L11903 movaqp,0 area=0, ind=0, am=1, be=0, %g17 movaqp,1 area=0, ind=16, am=0, be=0, %g16 movaqp,2 area=0, ind=0, am=1, be=0, %g19 movaqp,3 area=0, ind=16, am=0, be=0, %g18 } { loop_mode movaqp,0 area=1, ind=0, am=1, be=0, %g21 movaqp,1 area=1, ind=16, am=0, be=0, %g20 movaqp,2 area=1, ind=0, am=1, be=0, %g23 movaqp,3 area=1, ind=16, am=0, be=0, %g22 } { loop_mode movaqp,0 area=2, ind=0, am=1, be=0, %g25 movaqp,1 area=2, ind=16, am=0, be=0, %g24 movaqp,2 area=2, ind=0, am=1, be=0, %g27 movaqp,3 area=2, ind=16, am=0, be=0, %g26 } { loop_mode movaqp,0 area=3, ind=0, am=1, be=0, %g29 movaqp,1 area=3, ind=16, am=0, be=0, %g28 movaqp,2 area=3, ind=0, am=1, be=0, %g31 movaqp,3 area=3, ind=16, am=0, be=0, %g30 } { loop_mode movaqp,0 area=4, ind=0, am=1, be=0, %r26 movaqp,1 area=4, ind=16, am=0, be=0, %r9 movaqp,2 area=4, ind=0, am=1, be=0, %r28 movaqp,3 area=4, ind=16, am=0, be=0, %r27 } { loop_mode qpshufb,0 %g19, %g17, %r1, %r33 qpshufb,1 %g18, %g16, %r1, %r34 qpshufb,3 %g18, %g16, %r6, %g16 qpshufb,4 %g19, %g17, %r6, %g17 movaqp,0 area=5, ind=0, am=1, be=0, %r30 movaqp,1 area=5, ind=16, am=0, be=0, %r29 movaqp,2 area=5, ind=0, am=1, be=0, %r32 movaqp,3 area=5, ind=16, am=0, be=0, %r31 } { loop_mode qpshufb,0 %g23, %g21, %r1, %r37 qpshufb,1 %g22, %g20, %r1, %r38 qpshufb,3 %g22, %g20, %r6, %g20 qpshufb,4 %g23, %g21, %r6, %g21 movaqp,0 area=6, ind=0, am=1, be=0, %g19 movaqp,1 area=6, ind=16, am=0, be=0, %g18 movaqp,2 area=6, ind=0, am=1, be=0, %r36 movaqp,3 area=6, ind=16, am=0, be=0, %r35 } { loop_mode qpshufb,0 %g27, %g25, %r1, %g22 qpshufb,1 %g26, %g24, %r1, %g23 qpshufb,3 %g26, %g24, %r6, %g24 qpshufb,4 %g27, %g25, %r6, %g25 movaqp,0 area=8, ind=0, am=1, be=0, %r39 movaqp,1 area=7, ind=0, am=1, be=0, %g26 movaqp,2 area=8, ind=0, am=1, be=0, %r40 movaqp,3 area=7, ind=0, am=1, be=0, %g27 } { loop_mode qpshufb,0 %g31, %g29, %r1, %r41 qpshufb,1 %g30, %g28, %r1, %r42 qpshufb,3 %g30, %g28, %r6, %g28 qpshufb,4 %g31, %g29, %r6, %g29 movaqp,0 area=10, ind=0, am=1, be=0, %r43 movaqp,1 area=9, ind=0, am=1, be=0, %g30 movaqp,2 area=10, ind=0, am=1, be=0, %r44 movaqp,3 area=9, ind=0, am=1, be=0, %g31 } { loop_mode qpshufb,0 %r26, %r26, %r5, %r45 qpshufb,1 %r9, %r9, %r5, %r46 qpfmul_hsubs,2 %r26, %r33, %r7, %r26 qpshufb,3 %r28, %r28, %r5, %r47 qpshufb,4 %r27, %r27, %r5, %r48 movaqp,0 area=12, ind=0, am=1, be=0, %r51 movaqp,1 area=11, ind=0, am=1, be=0, %r49 movaqp,2 area=12, ind=0, am=1, be=0, %r52 movaqp,3 area=11, ind=0, am=1, be=0, %r50 } { loop_mode qpfmul_hadds,0 %r46, %r37, %r7, %r37 qpfmul_hsubs,1 %r28, %g22, %r7, %r28 qpfmul_hsubs,2 %r9, %r37, %r7, %r9 qpshufb,3 %r30, %r30, %r5, %r46 qpshufb,4 %r29, %r29, %r5, %r53 qpfmul_hsubs,5 %r30, %g16, %r7, %r30 } { loop_mode qpshufb,0 %g19, %g19, %r5, %r54 qpshufb,1 %g18, %g18, %r5, %r55 qpfmul_hsubs,2 %r27, %r41, %r7, %r27 qpshufb,3 %r36, %r36, %r5, %r56 qpshufb,4 %r35, %r35, %r5, %r57 qpfmul_hsubs,5 %g19, %r34, %r7, %g19 } { loop_mode qpfmul_hsubs,0 %g18, %r38, %r7, %g18 qpfmul_hsubs,1 %r36, %g23, %r7, %r36 qpfmul_hsubs,2 %r35, %r42, %r7, %r35 qpfmul_hadds,3 %r47, %g22, %r7, %g22 qpfmul_hadds,4 %r48, %r41, %r7, %r41 qpfmul_hadds,5 %r56, %g23, %r7, %g23 } { loop_mode qpfmul_hadds,0 %r54, %r34, %r7, %r34 qpfmul_hadds,1 %r55, %r38, %r7, %r38 qpfmul_hadds,2 %r45, %r33, %r7, %r33 qpshufb,3 %r32, %r32, %r5, %r45 qpshufb,4 %r31, %r31, %r5, %r47 qpfmul_hadds,5 %r57, %r42, %r7, %r42 } { loop_mode qpfmul_hsubs,0 %r29, %g20, %r7, %r29 qpfmul_hsubs,1 %r32, %g24, %r7, %r32 qpfmul_hsubs,2 %r31, %g28, %r7, %r31 qpfmul_hadds,3 %r53, %g20, %r7, %g20 qpfmul_hadds,4 %r45, %g24, %r7, %g24 qpfmul_hadds,5 %r46, %g16, %r7, %g16 } { loop_mode qpshufb,0 %r40, %r40, %r5, %r47 qpshufb,1 %g31, %g31, %r5, %r48 qpshufb,3 %g26, %g26, %r5, %r45 qpshufb,4 %r39, %r39, %r5, %r46 qpfmul_hadds,5 %r47, %g28, %r7, %g28 } { loop_mode qpshufb,3 %r43, %r43, %r5, %r53 qpshufb,4 %r50, %r50, %r5, %r54 } { loop_mode nop 1 qpshufb,0 %r49, %r49, %r5, %r55 qpshufb,1 %r52, %r52, %r5, %r56 qpshufb,3 %g27, %g27, %r5, %r57 qpshufb,4 %g30, %g30, %r5, %r58 } { loop_mode nop 1 qpshufb,3 %r44, %r44, %r5, %r59 qpshufb,4 %r51, %r51, %r5, %r60 } { loop_mode qppermb,0 %r33, %r26, %r3, %r26 qppermb,1 %r34, %g19, %r3, %g19 qppermb,3 %r41, %r27, %r3, %r27 qppermb,4 %r37, %r9, %r3, %r9 } { loop_mode qppermb,0 %r38, %g18, %r3, %g18 qppermb,1 %g22, %r28, %r3, %g22 qpfsubs,2 %r26, %g19, %r33 qppermb,3 %g23, %r36, %r3, %g23 qppermb,4 %r42, %r35, %r3, %r28 } { loop_mode qpfadds,0 %r26, %g19, %g19 qppermb,3 %g16, %r30, %r3, %g16 qpfadds,4 %r27, %r28, %r27 qpfsubs,5 %r27, %r28, %r34 } { loop_mode qppermb,0 %g20, %r29, %r3, %g20 qppermb,1 %g24, %r32, %r3, %g24 qppermb,3 %g28, %r31, %r3, %g28 qpfadds,4 %g17, %g16, %r26 qpfsubs,5 %g17, %g16, %g16 } { loop_mode qpfsubs,0 %r9, %g18, %g17 qpfadds,1 %r9, %g18, %g18 qpfsubs,2 %g21, %g20, %r9 qpfadds,3 %g29, %g28, %r28 qpfsubs,4 %g29, %g28, %g28 } { loop_mode qpfadds,0 %g21, %g20, %g20 qpfsubs,1 %g25, %g24, %g21 qpfsubs,2 %g22, %g23, %g29 qpfadds,5 %g22, %g23, %g22 } { loop_mode qpfadds,2 %g25, %g24, %g23 } { loop_mode qpshufb,3 %r34, %r34, %r5, %g24 qpshufb,4 %r33, %r33, %r5, %g25 } { loop_mode qpshufb,0 %g17, %g17, %r5, %g17 qpxor,3 %g24, %r4, %g24 qpxor,4 %g25, %r4, %g25 qpfadds,5 %r26, %g19, %r29 } { loop_mode qpshufb,0 %g29, %g29, %r5, %g29 qpxor,1 %g17, %r4, %g17 qpfsubs,2 %r26, %g19, %g19 qpfadds,3 %r28, %r27, %r26 qpfsubs,4 %r28, %r27, %r27 qpfsubs,5 %g16, %g25, %r28 } { loop_mode qpxor,0 %g29, %r4, %g29 qpfadds,1 %g20, %g18, %r30 qpfsubs,2 %g20, %g18, %g18 qpfadds,3 %g16, %g25, %g16 qpfsubs,4 %g28, %g24, %g20 qpfadds,5 %g28, %g24, %g24 } { loop_mode qpfadds,0 %r9, %g17, %g25 qpfadds,1 %g23, %g22, %g28 qpfsubs,2 %g23, %g22, %g22 } { loop_mode nop 2 qpfsubs,0 %r9, %g17, %g17 qpfsubs,1 %g21, %g29, %g23 qpfadds,2 %g21, %g29, %g21 } { loop_mode qpshufb,0 %r26, %r30, %r1, %g29 qpshufb,1 %r27, %g18, %r1, %r9 } { loop_mode qpshufb,0 %g28, %r29, %r1, %r31 qpshufb,1 %g22, %g19, %r1, %r32 qpfmul_hadds,2 %r55, %r9, %r7, %r33 qpshufb,3 %r27, %g18, %r6, %g18 qpshufb,4 %r26, %r30, %r6, %r26 } { loop_mode qpshufb,0 %g23, %r28, %r1, %r27 qpshufb,1 %g20, %g17, %r1, %r30 qpfmul_hsubs,2 %r49, %r9, %r7, %r9 qpshufb,3 %g24, %g25, %r1, %r34 qpshufb,4 %g24, %g25, %r6, %g24 qpfmul_hadds,5 %r59, %g18, %r7, %g25 } { loop_mode qpshufb,0 %g21, %g16, %r1, %r35 qpfmul_hsubs,1 %r40, %r27, %r7, %r36 qpfmul_hadds,2 %r47, %r27, %r7, %r27 qpfmul_hadds,3 %r56, %r34, %r7, %r34 qpshufb,4 %g20, %g17, %r6, %g17 qpfmul_hsubs,5 %r52, %r34, %r7, %r37 } { loop_mode qpfmul_hsubs,0 %r39, %g29, %r7, %g20 qpfmul_hadds,1 %r46, %g29, %r7, %g29 qpfmul_hsubs,2 %r43, %r32, %r7, %r38 qpfmul_hsubs,3 %r44, %g18, %r7, %g18 qpfmul_hsubs,4 %g27, %r26, %r7, %g27 qpfmul_hadds,5 %r57, %r26, %r7, %r26 } { loop_mode qpfmul_hadds,0 %r45, %r31, %r7, %r39 qpfmul_hadds,1 %r53, %r32, %r7, %r32 qpfmul_hsubs,2 %g26, %r31, %r7, %g26 qpfmul_hadds,3 %r58, %g17, %r7, %r40 qpfmul_hadds,4 %r60, %g24, %r7, %g24 qpfmul_hsubs,5 %r51, %g24, %r7, %r31 } { loop_mode qpfmul_hadds,0 %r54, %r35, %r7, %r41 qpfmul_hadds,1 %r48, %r30, %r7, %r30 qpfmul_hsubs,2 %g31, %r30, %r7, %g31 qpshufb,3 %g22, %g19, %r6, %g19 qpshufb,4 %g28, %r29, %r6, %g22 qpfmul_hsubs,5 %g30, %g17, %r7, %g17 } { loop_mode nop 4 qpshufb,0 %g23, %r28, %r6, %g23 qpshufb,1 %g21, %g16, %r6, %g16 qpfmul_hsubs,2 %r50, %r35, %r7, %g28 } { loop_mode qppermb,3 %r33, %r9, %r3, %g21 qppermb,4 %r34, %r37, %r3, %g30 } { loop_mode qppermb,0 %r27, %r36, %r3, %r9 qppermb,1 %g29, %g20, %r3, %g20 qppermb,3 %r26, %g27, %r3, %g27 qppermb,4 %g25, %g18, %r3, %g18 } { loop_mode qppermb,0 %r39, %g26, %r3, %g25 qppermb,1 %r32, %r38, %r3, %g26 qppermb,3 %r40, %g17, %r3, %g17 qppermb,4 %g24, %r31, %r3, %g24 qpfsubs,5 %g19, %g18, %g29 } { loop_mode qppermb,0 %r30, %g31, %r3, %g31 qppermb,1 %r41, %g28, %r3, %g28 qpfsubs,2 %g25, %g20, %r26 qpfadds,3 %g22, %g27, %r27 qpfadds,4 %g19, %g18, %g18 qpfsubs,5 %g22, %g27, %g19 } { loop_mode qpfsubs,0 %g26, %g21, %g22 qpfsubs,1 %r9, %g31, %g27 qpfsubs,2 %g28, %g30, %r28 qpfadds,3 %g16, %g24, %r29 qpfsubs,4 %g23, %g17, %r30 qpfsubs,5 %g16, %g24, %g16 } { loop_mode qpfadds,0 %g25, %g20, %g20 qpfadds,1 %g26, %g21, %g21 qpfadds,2 %r9, %g31, %g23 qpfadds,3 %g23, %g17, %g17 } { loop_mode nop 1 qpfadds,0 %g28, %g30, %g24 } { loop_mode qpshufb,1 %r26, %r26, %r5, %g25 } { loop_mode qpshufb,0 %g22, %g22, %r5, %g22 qpshufb,1 %g27, %g27, %r5, %g26 qpfsubs,2 %r27, %g20, %g27 } { loop_mode qpshufb,0 %r28, %r28, %r5, %g28 qpxor,1 %g25, %r4, %g25 qpfadds,2 %r27, %g20, %g20 } { loop_mode qpxor,0 %g22, %r4, %g22 qpxor,1 %g26, %r4, %g26 qpfadds,2 %g18, %g21, %g30 qpfsubs,3 %g18, %g21, %g18 qpfadds,4 %g17, %g23, %g21 qpfsubs,5 %g17, %g23, %g17 } { loop_mode qpxor,0 %g28, %r4, %g23 qpfadds,1 %r29, %g24, %g28 qpfsubs,2 %r29, %g24, %g24 } { loop_mode qpfsubs,0 %g19, %g25, %g31 qpfadds,1 %g19, %g25, %g19 qpfadds,2 %g29, %g22, %g25 } { loop_mode qpfsubs,0 %g29, %g22, %g22 qpfsubs,1 %r30, %g26, %g29 qpfadds,2 %r30, %g26, %g26 } { loop_mode qpfsubs,0 %g16, %g23, %r9 qpfadds,1 %g16, %g23, %g16 stqp,2 %r25, %r0, %g30 stqp,5 %r23, %r0, %g27 } { loop_mode stqp,2 %r2, %r0, %g20 stqp,5 %r18, %r0, %g18 } { loop_mode stqp,2 %r16, %r0, %g28 stqp,5 %r19, %r0, %g17 } { loop_mode stqp,2 %r22, %r0, %g21 stqp,5 %r12, %r0, %g24 } { loop_mode stqp,2 %r24, %r0, %g31 stqp,5 %r17, %r0, %g19 } { loop_mode stqp,2 %r15, %r0, %g22 stqp,5 %r14, %r0, %g25 } { loop_mode stqp,2 %r21, %r0, %g29 stqp,5 %r11, %r0, %g26 } { loop_mode ct %ctpr1 ? %NOT_LOOP_END alc alcf=1, alct=1 addd,0,sm 0x10, %r0, %r0 stqp,2 %r20, %r0, %r9 stqp,5 %r13, %r0, %g16 }
Теоретическая скорость: 32 комплексных числа за 62 такта (32/62) = 4.13 Байт/такт
Четверная теоретическая скорость: 16.52 Байт/такт
Замеры скорости
6. stage_radix4_2x_simd128_noConj_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью опции unroll.
Код на Си
void stage_radix4_2x_simd128_noConj_unroll2(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coefC_a, myComplex *coefD_a, myComplex *coefE_a, myComplex *coefC_b, myComplex *coefD_b, myComplex *coefE_b) { __v2di *xy0_in = (__v2di*)&data_in[ 0]; __v2di *zw0_in = (__v2di*)&data_in[ 2]; __v2di *xy1_in = (__v2di*)&data_in[ 4]; __v2di *zw1_in = (__v2di*)&data_in[ 6]; __v2di *xy2_in = (__v2di*)&data_in[ 8]; __v2di *zw2_in = (__v2di*)&data_in[10]; __v2di *xy3_in = (__v2di*)&data_in[12]; __v2di *zw3_in = (__v2di*)&data_in[14]; __v2di *xy4_in = (__v2di*)&data_in[16]; __v2di *zw4_in = (__v2di*)&data_in[18]; __v2di *xy5_in = (__v2di*)&data_in[20]; __v2di *zw5_in = (__v2di*)&data_in[22]; __v2di *xy6_in = (__v2di*)&data_in[24]; __v2di *zw6_in = (__v2di*)&data_in[26]; __v2di *xy7_in = (__v2di*)&data_in[28]; __v2di *zw7_in = (__v2di*)&data_in[30]; __v2di *c0a_in = (__v2di*)&coefC_a[0]; __v2di *c1a_in = (__v2di*)&coefC_a[2]; __v2di *c2a_in = (__v2di*)&coefC_a[4]; __v2di *c3a_in = (__v2di*)&coefC_a[6]; __v2di *d0a_in = (__v2di*)&coefD_a[0]; __v2di *d1a_in = (__v2di*)&coefD_a[2]; __v2di *d2a_in = (__v2di*)&coefD_a[4]; __v2di *d3a_in = (__v2di*)&coefD_a[6]; __v2di *e0a_in = (__v2di*)&coefE_a[0]; __v2di *e1a_in = (__v2di*)&coefE_a[2]; __v2di *e2a_in = (__v2di*)&coefE_a[4]; __v2di *e3a_in = (__v2di*)&coefE_a[6]; __v2di *c0b_in = (__v2di*)&coefC_b[0*data_count/16]; __v2di *c1b_in = (__v2di*)&coefC_b[1*data_count/16]; __v2di *c2b_in = (__v2di*)&coefC_b[2*data_count/16]; __v2di *c3b_in = (__v2di*)&coefC_b[3*data_count/16]; __v2di *d0b_in = (__v2di*)&coefD_b[0*data_count/16]; __v2di *d1b_in = (__v2di*)&coefD_b[1*data_count/16]; __v2di *d2b_in = (__v2di*)&coefD_b[2*data_count/16]; __v2di *d3b_in = (__v2di*)&coefD_b[3*data_count/16]; __v2di *e0b_in = (__v2di*)&coefE_b[0*data_count/16]; __v2di *e1b_in = (__v2di*)&coefE_b[1*data_count/16]; __v2di *e2b_in = (__v2di*)&coefE_b[2*data_count/16]; __v2di *e3b_in = (__v2di*)&coefE_b[3*data_count/16]; __v2di *out_0 = (__v2di*)&data_out[ 0*data_count/16]; __v2di *out_1 = (__v2di*)&data_out[ 1*data_count/16]; __v2di *out_2 = (__v2di*)&data_out[ 2*data_count/16]; __v2di *out_3 = (__v2di*)&data_out[ 3*data_count/16]; __v2di *out_4 = (__v2di*)&data_out[ 4*data_count/16]; __v2di *out_5 = (__v2di*)&data_out[ 5*data_count/16]; __v2di *out_6 = (__v2di*)&data_out[ 6*data_count/16]; __v2di *out_7 = (__v2di*)&data_out[ 7*data_count/16]; __v2di *out_8 = (__v2di*)&data_out[ 8*data_count/16]; __v2di *out_9 = (__v2di*)&data_out[ 9*data_count/16]; __v2di *out_10 = (__v2di*)&data_out[10*data_count/16]; __v2di *out_11 = (__v2di*)&data_out[11*data_count/16]; __v2di *out_12 = (__v2di*)&data_out[12*data_count/16]; __v2di *out_13 = (__v2di*)&data_out[13*data_count/16]; __v2di *out_14 = (__v2di*)&data_out[14*data_count/16]; __v2di *out_15 = (__v2di*)&data_out[15*data_count/16]; #pragma ivdep #pragma unroll(2) #pragma prefetch for(int64_t i = 0; i < data_count/32; ++i) { __v2di xy0 = xy0_in[16*i]; __v2di zw0 = zw0_in[16*i]; __v2di xy1 = xy1_in[16*i]; __v2di zw1 = zw1_in[16*i]; __v2di c0 = c0a_in[4*i]; __v2di d0 = d0a_in[4*i]; __v2di e0 = e0a_in[4*i]; __v2di xy2 = xy2_in[16*i]; __v2di zw2 = zw2_in[16*i]; __v2di xy3 = xy3_in[16*i]; __v2di zw3 = zw3_in[16*i]; __v2di c1 = c1a_in[4*i]; __v2di d1 = d1a_in[4*i]; __v2di e1 = e1a_in[4*i]; __v2di xy4 = xy4_in[16*i]; __v2di zw4 = zw4_in[16*i]; __v2di xy5 = xy5_in[16*i]; __v2di zw5 = zw5_in[16*i]; __v2di c2 = c2a_in[4*i]; __v2di d2 = d2a_in[4*i]; __v2di e2 = e2a_in[4*i]; __v2di xy6 = xy6_in[16*i]; __v2di zw6 = zw6_in[16*i]; __v2di xy7 = xy7_in[16*i]; __v2di zw7 = zw7_in[16*i]; __v2di c3 = c3a_in[4*i]; __v2di d3 = d3a_in[4*i]; __v2di e3 = e3a_in[4*i]; __v2di x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c2 = __builtin_e2k_qpshufb(c2, c2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c3 = __builtin_e2k_qpshufb(c3, c3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d0 = __builtin_e2k_qpshufb(d0, d0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d1 = __builtin_e2k_qpshufb(d1, d1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d2 = __builtin_e2k_qpshufb(d2, d2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d3 = __builtin_e2k_qpshufb(d3, d3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e0 = __builtin_e2k_qpshufb(e0, e0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e1 = __builtin_e2k_qpshufb(e1, e1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e2 = __builtin_e2k_qpshufb(e2, e2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e3 = __builtin_e2k_qpshufb(e3, e3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy0_real = __builtin_e2k_qpfmuls( c0, y0); __v2di cy1_real = __builtin_e2k_qpfmuls( c1, y1); __v2di cy2_real = __builtin_e2k_qpfmuls( c2, y2); __v2di cy3_real = __builtin_e2k_qpfmuls( c3, y3); __v2di dz0_real = __builtin_e2k_qpfmuls( d0, z0); __v2di dz1_real = __builtin_e2k_qpfmuls( d1, z1); __v2di dz2_real = __builtin_e2k_qpfmuls( d2, z2); __v2di dz3_real = __builtin_e2k_qpfmuls( d3, z3); __v2di ew0_real = __builtin_e2k_qpfmuls( e0, w0); __v2di ew1_real = __builtin_e2k_qpfmuls( e1, w1); __v2di ew2_real = __builtin_e2k_qpfmuls( e2, w2); __v2di ew3_real = __builtin_e2k_qpfmuls( e3, w3); __v2di cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); __v2di cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); __v2di cy2_imag = __builtin_e2k_qpfmuls(swap_c2, y2); __v2di cy3_imag = __builtin_e2k_qpfmuls(swap_c3, y3); __v2di dz0_imag = __builtin_e2k_qpfmuls(swap_d0, z0); __v2di dz1_imag = __builtin_e2k_qpfmuls(swap_d1, z1); __v2di dz2_imag = __builtin_e2k_qpfmuls(swap_d2, z2); __v2di dz3_imag = __builtin_e2k_qpfmuls(swap_d3, z3); __v2di ew0_imag = __builtin_e2k_qpfmuls(swap_e0, w0); __v2di ew1_imag = __builtin_e2k_qpfmuls(swap_e1, w1); __v2di ew2_imag = __builtin_e2k_qpfmuls(swap_e2, w2); __v2di ew3_imag = __builtin_e2k_qpfmuls(swap_e3, w3); __v2di cy0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy0_real); __v2di cy1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy1_real); __v2di cy2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy2_real); __v2di cy3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy3_real); __v2di dz0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz0_real); __v2di dz1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz1_real); __v2di dz2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz2_real); __v2di dz3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz3_real); __v2di ew0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew0_real); __v2di ew1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew1_real); __v2di ew2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew2_real); __v2di ew3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew3_real); __v2di cy0_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy0_imag); __v2di cy1_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy1_imag); __v2di cy2_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy2_imag); __v2di cy3_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy3_imag); __v2di dz0_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz0_imag); __v2di dz1_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz1_imag); __v2di dz2_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz2_imag); __v2di dz3_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz3_imag); __v2di ew0_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew0_imag); __v2di ew1_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew1_imag); __v2di ew2_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew2_imag); __v2di ew3_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew3_imag); __v2di cy0 = __builtin_e2k_qppermb(cy0_ii, cy0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di cy1 = __builtin_e2k_qppermb(cy1_ii, cy1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di cy2 = __builtin_e2k_qppermb(cy2_ii, cy2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di cy3 = __builtin_e2k_qppermb(cy3_ii, cy3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz0 = __builtin_e2k_qppermb(dz0_ii, dz0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz1 = __builtin_e2k_qppermb(dz1_ii, dz1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz2 = __builtin_e2k_qppermb(dz2_ii, dz2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz3 = __builtin_e2k_qppermb(dz3_ii, dz3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew0 = __builtin_e2k_qppermb(ew0_ii, ew0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew1 = __builtin_e2k_qppermb(ew1_ii, ew1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew2 = __builtin_e2k_qppermb(ew2_ii, ew2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew3 = __builtin_e2k_qppermb(ew3_ii, ew3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di add02_0 = __builtin_e2k_qpfadds( x0, dz0); __v2di add02_1 = __builtin_e2k_qpfadds( x1, dz1); __v2di add02_2 = __builtin_e2k_qpfadds( x2, dz2); __v2di add02_3 = __builtin_e2k_qpfadds( x3, dz3); __v2di sub02_0 = __builtin_e2k_qpfsubs( x0, dz0); __v2di sub02_1 = __builtin_e2k_qpfsubs( x1, dz1); __v2di sub02_2 = __builtin_e2k_qpfsubs( x2, dz2); __v2di sub02_3 = __builtin_e2k_qpfsubs( x3, dz3); __v2di add13_0 = __builtin_e2k_qpfadds(cy0, ew0); __v2di add13_1 = __builtin_e2k_qpfadds(cy1, ew1); __v2di add13_2 = __builtin_e2k_qpfadds(cy2, ew2); __v2di add13_3 = __builtin_e2k_qpfadds(cy3, ew3); __v2di sub13_0 = __builtin_e2k_qpfsubs(cy0, ew0); __v2di sub13_1 = __builtin_e2k_qpfsubs(cy1, ew1); __v2di sub13_2 = __builtin_e2k_qpfsubs(cy2, ew2); __v2di sub13_3 = __builtin_e2k_qpfsubs(cy3, ew3); __v2di swap_sub13_0 = __builtin_e2k_qpshufb(sub13_0, sub13_0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_1 = __builtin_e2k_qpshufb(sub13_1, sub13_1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_2 = __builtin_e2k_qpshufb(sub13_2, sub13_2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_3 = __builtin_e2k_qpshufb(sub13_3, sub13_3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di sub13i_0 = __builtin_e2k_qpxor(swap_sub13_0, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_1 = __builtin_e2k_qpxor(swap_sub13_1, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_2 = __builtin_e2k_qpxor(swap_sub13_2, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_3 = __builtin_e2k_qpxor(swap_sub13_3, (__v2di){1LL<<31, 1LL<<31}); __v2di out0 = __builtin_e2k_qpfadds(add02_0, add13_0); __v2di out1 = __builtin_e2k_qpfadds(add02_1, add13_1); __v2di out2 = __builtin_e2k_qpfadds(add02_2, add13_2); __v2di out3 = __builtin_e2k_qpfadds(add02_3, add13_3); __v2di out4 = __builtin_e2k_qpfsubs(sub02_0, sub13i_0); __v2di out5 = __builtin_e2k_qpfsubs(sub02_1, sub13i_1); __v2di out6 = __builtin_e2k_qpfsubs(sub02_2, sub13i_2); __v2di out7 = __builtin_e2k_qpfsubs(sub02_3, sub13i_3); __v2di out8 = __builtin_e2k_qpfsubs(add02_0, add13_0); __v2di out9 = __builtin_e2k_qpfsubs(add02_1, add13_1); __v2di out10 = __builtin_e2k_qpfsubs(add02_2, add13_2); __v2di out11 = __builtin_e2k_qpfsubs(add02_3, add13_3); __v2di out12 = __builtin_e2k_qpfadds(sub02_0, sub13i_0); __v2di out13 = __builtin_e2k_qpfadds(sub02_1, sub13i_1); __v2di out14 = __builtin_e2k_qpfadds(sub02_2, sub13i_2); __v2di out15 = __builtin_e2k_qpfadds(sub02_3, sub13i_3); xy0 = out0; zw0 = out1; xy1 = out2; zw1 = out3; c0 = c0b_in[i]; d0 = d0b_in[i]; e0 = e0b_in[i]; xy2 = out4; zw2 = out5; xy3 = out6; zw3 = out7; c1 = c1b_in[i]; d1 = d1b_in[i]; e1 = e1b_in[i]; xy4 = out8; zw4 = out9; xy5 = out10; zw5 = out11; c2 = c2b_in[i]; d2 = d2b_in[i]; e2 = e2b_in[i]; xy6 = out12; zw6 = out13; xy7 = out14; zw7 = out15; c3 = c3b_in[i]; d3 = d3b_in[i]; e3 = e3b_in[i]; x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); w0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0706050403020100, 0x0706050403020100}); w1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0706050403020100, 0x0706050403020100}); y2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0706050403020100, 0x0706050403020100}); w2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0706050403020100, 0x0706050403020100}); y3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0706050403020100, 0x0706050403020100}); w3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c2 = __builtin_e2k_qpshufb(c2, c2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c3 = __builtin_e2k_qpshufb(c3, c3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d0 = __builtin_e2k_qpshufb(d0, d0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d1 = __builtin_e2k_qpshufb(d1, d1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d2 = __builtin_e2k_qpshufb(d2, d2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d3 = __builtin_e2k_qpshufb(d3, d3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e0 = __builtin_e2k_qpshufb(e0, e0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e1 = __builtin_e2k_qpshufb(e1, e1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e2 = __builtin_e2k_qpshufb(e2, e2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e3 = __builtin_e2k_qpshufb(e3, e3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); cy0_real = __builtin_e2k_qpfmuls( c0, y0); cy1_real = __builtin_e2k_qpfmuls( c1, y1); cy2_real = __builtin_e2k_qpfmuls( c2, y2); cy3_real = __builtin_e2k_qpfmuls( c3, y3); dz0_real = __builtin_e2k_qpfmuls( d0, z0); dz1_real = __builtin_e2k_qpfmuls( d1, z1); dz2_real = __builtin_e2k_qpfmuls( d2, z2); dz3_real = __builtin_e2k_qpfmuls( d3, z3); ew0_real = __builtin_e2k_qpfmuls( e0, w0); ew1_real = __builtin_e2k_qpfmuls( e1, w1); ew2_real = __builtin_e2k_qpfmuls( e2, w2); ew3_real = __builtin_e2k_qpfmuls( e3, w3); cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); cy2_imag = __builtin_e2k_qpfmuls(swap_c2, y2); cy3_imag = __builtin_e2k_qpfmuls(swap_c3, y3); dz0_imag = __builtin_e2k_qpfmuls(swap_d0, z0); dz1_imag = __builtin_e2k_qpfmuls(swap_d1, z1); dz2_imag = __builtin_e2k_qpfmuls(swap_d2, z2); dz3_imag = __builtin_e2k_qpfmuls(swap_d3, z3); ew0_imag = __builtin_e2k_qpfmuls(swap_e0, w0); ew1_imag = __builtin_e2k_qpfmuls(swap_e1, w1); ew2_imag = __builtin_e2k_qpfmuls(swap_e2, w2); ew3_imag = __builtin_e2k_qpfmuls(swap_e3, w3); cy0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy0_real); cy1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy1_real); cy2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy2_real); cy3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy3_real); dz0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz0_real); dz1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz1_real); dz2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz2_real); dz3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz3_real); ew0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew0_real); ew1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew1_real); ew2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew2_real); ew3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew3_real); cy0_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy0_imag); cy1_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy1_imag); cy2_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy2_imag); cy3_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy3_imag); dz0_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz0_imag); dz1_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz1_imag); dz2_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz2_imag); dz3_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz3_imag); ew0_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew0_imag); ew1_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew1_imag); ew2_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew2_imag); ew3_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew3_imag); cy0 = __builtin_e2k_qppermb(cy0_ii, cy0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); cy1 = __builtin_e2k_qppermb(cy1_ii, cy1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); cy2 = __builtin_e2k_qppermb(cy2_ii, cy2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); cy3 = __builtin_e2k_qppermb(cy3_ii, cy3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); dz0 = __builtin_e2k_qppermb(dz0_ii, dz0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); dz1 = __builtin_e2k_qppermb(dz1_ii, dz1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); dz2 = __builtin_e2k_qppermb(dz2_ii, dz2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); dz3 = __builtin_e2k_qppermb(dz3_ii, dz3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); ew0 = __builtin_e2k_qppermb(ew0_ii, ew0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); ew1 = __builtin_e2k_qppermb(ew1_ii, ew1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); ew2 = __builtin_e2k_qppermb(ew2_ii, ew2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); ew3 = __builtin_e2k_qppermb(ew3_ii, ew3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); add02_0 = __builtin_e2k_qpfadds( x0, dz0); add02_1 = __builtin_e2k_qpfadds( x1, dz1); add02_2 = __builtin_e2k_qpfadds( x2, dz2); add02_3 = __builtin_e2k_qpfadds( x3, dz3); sub02_0 = __builtin_e2k_qpfsubs( x0, dz0); sub02_1 = __builtin_e2k_qpfsubs( x1, dz1); sub02_2 = __builtin_e2k_qpfsubs( x2, dz2); sub02_3 = __builtin_e2k_qpfsubs( x3, dz3); add13_0 = __builtin_e2k_qpfadds(cy0, ew0); add13_1 = __builtin_e2k_qpfadds(cy1, ew1); add13_2 = __builtin_e2k_qpfadds(cy2, ew2); add13_3 = __builtin_e2k_qpfadds(cy3, ew3); sub13_0 = __builtin_e2k_qpfsubs(cy0, ew0); sub13_1 = __builtin_e2k_qpfsubs(cy1, ew1); sub13_2 = __builtin_e2k_qpfsubs(cy2, ew2); sub13_3 = __builtin_e2k_qpfsubs(cy3, ew3); swap_sub13_0 = __builtin_e2k_qpshufb(sub13_0, sub13_0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_1 = __builtin_e2k_qpshufb(sub13_1, sub13_1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_2 = __builtin_e2k_qpshufb(sub13_2, sub13_2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_3 = __builtin_e2k_qpshufb(sub13_3, sub13_3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); sub13i_0 = __builtin_e2k_qpxor(swap_sub13_0, (__v2di){1LL<<31, 1LL<<31}); sub13i_1 = __builtin_e2k_qpxor(swap_sub13_1, (__v2di){1LL<<31, 1LL<<31}); sub13i_2 = __builtin_e2k_qpxor(swap_sub13_2, (__v2di){1LL<<31, 1LL<<31}); sub13i_3 = __builtin_e2k_qpxor(swap_sub13_3, (__v2di){1LL<<31, 1LL<<31}); out_0[i] = __builtin_e2k_qpfadds(add02_0, add13_0); out_1[i] = __builtin_e2k_qpfadds(add02_1, add13_1); out_2[i] = __builtin_e2k_qpfadds(add02_2, add13_2); out_3[i] = __builtin_e2k_qpfadds(add02_3, add13_3); out_4[i] = __builtin_e2k_qpfsubs(sub02_0, sub13i_0); out_5[i] = __builtin_e2k_qpfsubs(sub02_1, sub13i_1); out_6[i] = __builtin_e2k_qpfsubs(sub02_2, sub13i_2); out_7[i] = __builtin_e2k_qpfsubs(sub02_3, sub13i_3); out_8[i] = __builtin_e2k_qpfsubs(add02_0, add13_0); out_9[i] = __builtin_e2k_qpfsubs(add02_1, add13_1); out_10[i] = __builtin_e2k_qpfsubs(add02_2, add13_2); out_11[i] = __builtin_e2k_qpfsubs(add02_3, add13_3); out_12[i] = __builtin_e2k_qpfadds(sub02_0, sub13i_0); out_13[i] = __builtin_e2k_qpfadds(sub02_1, sub13i_1); out_14[i] = __builtin_e2k_qpfadds(sub02_2, sub13i_2); out_15[i] = __builtin_e2k_qpfadds(sub02_3, sub13i_3); } }
Основной цикл на ассемблере
.L19508: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=1, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=1, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=2, disp=128 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=2, disp=160 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=3, disp=192 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=3, disp=224 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=4, disp=256 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=4, disp=288 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=5, disp=320 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=5, disp=352 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=6, disp=384 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=6, disp=416 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=7, disp=448 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=7, disp=480 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=1, abs=8, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=1, abs=10, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=1, abs=10, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=1, abs=12, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=1, abs=14, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=1, abs=14, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=1, abs=16, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=1, abs=18, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=1, abs=18, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=1, incr=2, ind=0, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=15, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=14, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=13, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=12, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=11, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=10, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=9, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=8, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=7, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=6, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=5, asz=1, abs=30, disp=0 } .L15504: { loop_mode disp %ctpr1, .L15504 movaqp,0 area=0, ind=0, am=1, be=0, %g17 movaqp,1 area=0, ind=16, am=0, be=0, %g16 movaqp,2 area=0, ind=0, am=1, be=0, %g19 movaqp,3 area=0, ind=16, am=0, be=0, %g18 } { loop_mode movaqp,0 area=1, ind=0, am=1, be=0, %g21 movaqp,1 area=1, ind=16, am=0, be=0, %g20 movaqp,2 area=1, ind=0, am=1, be=0, %g23 movaqp,3 area=1, ind=16, am=0, be=0, %g22 } { loop_mode movaqp,0 area=2, ind=0, am=1, be=0, %g25 movaqp,1 area=2, ind=16, am=0, be=0, %g24 movaqp,2 area=2, ind=0, am=1, be=0, %g27 movaqp,3 area=2, ind=16, am=0, be=0, %g26 } { loop_mode movaqp,0 area=3, ind=0, am=1, be=0, %g29 movaqp,1 area=3, ind=16, am=0, be=0, %g28 movaqp,2 area=3, ind=0, am=1, be=0, %g31 movaqp,3 area=3, ind=16, am=0, be=0, %g30 } { loop_mode movaqp,0 area=4, ind=0, am=1, be=0, %b[20] movaqp,1 area=4, ind=16, am=0, be=0, %b[19] movaqp,2 area=4, ind=0, am=1, be=0, %b[22] movaqp,3 area=4, ind=16, am=0, be=0, %b[21] } { loop_mode qpshufb,0 %g19, %g17, %r7, %b[27] qpshufb,1 %g18, %g16, %r7, %b[28] qpshufb,3 %g18, %g16, %r6, %g16 qpshufb,4 %g19, %g17, %r6, %g17 movaqp,0 area=5, ind=0, am=1, be=0, %b[24] movaqp,1 area=5, ind=16, am=0, be=0, %b[23] movaqp,2 area=5, ind=0, am=1, be=0, %b[26] movaqp,3 area=5, ind=16, am=0, be=0, %b[25] } { loop_mode qpshufb,0 %g23, %g21, %r7, %b[31] qpshufb,1 %g22, %g20, %r7, %b[32] qpshufb,3 %g22, %g20, %r6, %g20 qpshufb,4 %g23, %g21, %r6, %g21 movaqp,0 area=6, ind=0, am=1, be=0, %g19 movaqp,1 area=6, ind=16, am=0, be=0, %g18 movaqp,2 area=6, ind=0, am=1, be=0, %b[30] movaqp,3 area=6, ind=16, am=0, be=0, %b[29] } { loop_mode qpshufb,0 %g27, %g25, %r7, %b[35] qpshufb,1 %g26, %g24, %r7, %b[36] qpshufb,3 %g26, %g24, %r6, %g24 qpshufb,4 %g27, %g25, %r6, %g25 movaqp,0 area=7, ind=0, am=1, be=0, %g23 movaqp,1 area=7, ind=16, am=0, be=0, %g22 movaqp,2 area=7, ind=0, am=1, be=0, %b[34] movaqp,3 area=7, ind=16, am=0, be=0, %b[33] } { loop_mode qpshufb,0 %g31, %g29, %r7, %b[39] qpshufb,1 %g30, %g28, %r7, %b[40] qpshufb,3 %g30, %g28, %r6, %g28 qpshufb,4 %g31, %g29, %r6, %g29 movaqp,0 area=8, ind=0, am=1, be=0, %g27 movaqp,1 area=8, ind=16, am=0, be=0, %g26 movaqp,2 area=8, ind=0, am=1, be=0, %b[38] movaqp,3 area=8, ind=16, am=0, be=0, %b[37] } { loop_mode qpshufb,0 %b[22], %b[20], %r7, %b[43] qpshufb,1 %b[21], %b[19], %r7, %b[44] qpshufb,3 %b[21], %b[19], %r6, %b[19] qpshufb,4 %b[22], %b[20], %r6, %b[20] movaqp,0 area=9, ind=0, am=1, be=0, %g31 movaqp,1 area=9, ind=16, am=0, be=0, %g30 movaqp,2 area=9, ind=0, am=1, be=0, %b[42] movaqp,3 area=9, ind=16, am=0, be=0, %b[41] } { loop_mode qpshufb,0 %b[26], %b[24], %r7, %b[47] qpshufb,1 %b[25], %b[23], %r7, %b[48] qpshufb,3 %b[25], %b[23], %r6, %b[23] qpshufb,4 %b[26], %b[24], %r6, %b[24] movaqp,0 area=10, ind=0, am=1, be=0, %b[22] movaqp,1 area=10, ind=16, am=0, be=0, %b[21] movaqp,2 area=10, ind=0, am=1, be=0, %b[46] movaqp,3 area=10, ind=16, am=0, be=0, %b[45] } { loop_mode qpshufb,0 %b[30], %g19, %r7, %b[51] qpshufb,1 %b[29], %g18, %r7, %b[52] qpshufb,3 %b[29], %g18, %r6, %g18 qpshufb,4 %b[30], %g19, %r6, %g19 movaqp,0 area=11, ind=0, am=1, be=0, %b[26] movaqp,1 area=11, ind=16, am=0, be=0, %b[25] movaqp,2 area=11, ind=0, am=1, be=0, %b[50] movaqp,3 area=11, ind=16, am=0, be=0, %b[49] } { loop_mode qpshufb,0 %b[34], %g23, %r7, %b[55] qpshufb,1 %b[33], %g22, %r7, %b[56] qpshufb,3 %b[33], %g22, %r6, %g22 qpshufb,4 %b[34], %g23, %r6, %g23 movaqp,0 area=12, ind=0, am=1, be=0, %b[30] movaqp,1 area=12, ind=16, am=0, be=0, %b[29] movaqp,2 area=12, ind=0, am=1, be=0, %b[54] movaqp,3 area=12, ind=16, am=0, be=0, %b[53] } { loop_mode qpshufb,0 %g27, %g27, %r23, %b[59] qpshufb,1 %g26, %g26, %r23, %b[60] qpfmul_hsubs,2 %g27, %b[27], %r40, %g27 qpshufb,3 %b[38], %b[38], %r23, %b[61] qpshufb,4 %b[37], %b[37], %r23, %b[62] qpfmul_hsubs,5 %g26, %b[31], %r40, %g26 movaqp,0 area=13, ind=0, am=1, be=0, %b[34] movaqp,1 area=13, ind=16, am=0, be=0, %b[33] movaqp,2 area=13, ind=0, am=1, be=0, %b[58] movaqp,3 area=13, ind=16, am=0, be=0, %b[57] } { loop_mode qpshufb,0 %g31, %g31, %r23, %b[63] qpshufb,1 %g30, %g30, %r23, %b[64] qpfmul_hsubs,2 %b[38], %b[35], %r40, %b[38] qpshufb,3 %b[42], %b[42], %r23, %b[65] qpshufb,4 %b[41], %b[41], %r23, %b[66] qpfmul_hsubs,5 %b[37], %b[39], %r40, %b[37] movaqp,0 area=14, ind=0, am=1, be=0, %b[68] movaqp,1 area=14, ind=16, am=0, be=0, %b[67] movaqp,2 area=14, ind=0, am=1, be=0, %b[70] movaqp,3 area=14, ind=16, am=0, be=0, %b[69] } { loop_mode qpfmul_hsubs,0 %g31, %b[43], %r40, %g31 qpfmul_hsubs,1 %g30, %b[47], %r40, %g30 qpfmul_hsubs,2 %b[42], %b[51], %r40, %b[42] qpfmul_hadds,3 %b[61], %b[35], %r40, %b[35] qpfmul_hadds,4 %b[62], %b[39], %r40, %b[39] qpfmul_hadds,5 %b[65], %b[51], %r40, %b[51] movaqp,0 area=15, ind=0, am=1, be=0, %b[62] movaqp,1 area=15, ind=16, am=0, be=0, %b[61] movaqp,2 area=15, ind=0, am=1, be=0, %b[71] movaqp,3 area=15, ind=16, am=0, be=0, %b[65] } { loop_mode qpfmul_hadds,0 %b[59], %b[27], %r40, %b[27] qpfmul_hadds,1 %b[64], %b[47], %r40, %b[47] qpfmul_hadds,2 %b[60], %b[31], %r40, %b[31] qpfmul_hadds,3 %b[66], %b[55], %r40, %b[55] qpshufb,4 %b[21], %b[21], %r23, %b[59] qpfmul_hsubs,5 %b[41], %b[55], %r40, %b[41] movaqp,0 area=16, ind=0, am=1, be=0, %b[64] movaqp,1 area=16, ind=16, am=0, be=0, %b[60] movaqp,2 area=16, ind=0, am=1, be=0, %b[72] movaqp,3 area=16, ind=16, am=0, be=0, %b[66] } { loop_mode qpshufb,0 %b[30], %b[30], %r23, %b[73] qpshufb,1 %b[29], %b[29], %r23, %b[74] qpfmul_hsubs,2 %b[29], %b[32], %r40, %b[29] qpshufb,3 %b[54], %b[54], %r23, %b[75] qpshufb,4 %b[53], %b[53], %r23, %b[76] qpfmul_hsubs,5 %b[30], %b[28], %r40, %b[30] movaqp,0 area=17, ind=0, am=1, be=0, %b[78] movaqp,1 area=17, ind=16, am=0, be=0, %b[77] movaqp,2 area=17, ind=0, am=1, be=0, %b[80] movaqp,3 area=17, ind=16, am=0, be=0, %b[79] } { loop_mode qpshufb,0 %b[34], %b[34], %r23, %b[81] qpshufb,1 %b[33], %b[33], %r23, %b[82] qpfmul_hadds,2 %b[63], %b[43], %r40, %b[43] qpshufb,3 %b[58], %b[58], %r23, %b[83] qpshufb,4 %b[57], %b[57], %r23, %b[84] qpfmul_hsubs,5 %b[54], %b[36], %r40, %b[54] movaqp,0 area=18, ind=0, am=1, be=0, %b[85] movaqp,1 area=18, ind=16, am=0, be=0, %b[63] movaqp,2 area=18, ind=0, am=1, be=0, %b[87] movaqp,3 area=18, ind=16, am=0, be=0, %b[86] } { loop_mode qpfmul_hsubs,0 %b[53], %b[40], %r40, %b[53] qpfmul_hsubs,1 %b[34], %b[44], %r40, %b[34] qpfmul_hsubs,2 %b[33], %b[48], %r40, %b[33] qpfmul_hsubs,3 %b[58], %b[52], %r40, %b[58] qpfmul_hsubs,4 %b[57], %b[56], %r40, %b[57] qpfmul_hadds,5 %b[75], %b[36], %r40, %b[36] movaqp,0 area=19, ind=0, am=1, be=0, %b[88] movaqp,1 area=19, ind=16, am=0, be=0, %b[75] movaqp,2 area=19, ind=0, am=1, be=0, %b[90] movaqp,3 area=19, ind=16, am=0, be=0, %b[89] } { loop_mode qpfmul_hadds,0 %b[73], %b[28], %r40, %b[28] qpfmul_hadds,1 %b[74], %b[32], %r40, %b[32] qpfmul_hadds,2 %b[82], %b[48], %r40, %b[48] qpfmul_hadds,3 %b[84], %b[56], %r40, %b[56] qpfmul_hadds,4 %b[83], %b[52], %r40, %b[52] qpfmul_hadds,5 %b[76], %b[40], %r40, %b[40] } { loop_mode qpfmul_hsubs,0 %b[21], %g20, %r40, %b[21] qpfmul_hsubs,1 %b[22], %g16, %r40, %b[73] qpfmul_hadds,2 %b[81], %b[44], %r40, %b[44] qpfmul_hsubs,3 %b[46], %g24, %r40, %b[74] qpfmul_hsubs,4 %b[45], %g28, %r40, %b[76] qpfmul_hsubs,5 %b[25], %b[23], %r40, %b[81] } { loop_mode qpfmul_hsubs,0 %b[50], %g18, %r40, %b[82] qpfmul_hsubs,1 %b[49], %g22, %r40, %b[83] qpfmul_hsubs,2 %b[26], %b[19], %r40, %b[59] qpshufb,4 %b[22], %b[22], %r23, %b[22] qpfmul_hadds,5 %b[59], %g20, %r40, %g20 } { loop_mode qpshufb,0 %b[46], %b[46], %r23, %b[46] qpshufb,1 %b[45], %b[45], %r23, %b[45] qpshufb,3 %b[26], %b[26], %r23, %b[26] qpshufb,4 %b[25], %b[25], %r23, %b[25] qpfmul_hadds,5 %b[22], %g16, %r40, %g16 } { loop_mode qpshufb,0 %b[50], %b[50], %r23, %b[22] qpshufb,1 %b[49], %b[49], %r23, %b[49] qpfmul_hadds,2 %b[45], %g28, %r40, %g28 qpfmul_hadds,3 %b[25], %b[23], %r40, %b[23] qppermb,4 %b[39], %b[37], %r24, %b[25] qpfmul_hadds,5 %b[26], %b[19], %r40, %b[19] } { loop_mode nop 2 qpfmul_hadds,0 %b[22], %g18, %r40, %g18 qpfmul_hadds,1 %b[49], %g22, %r40, %g22 qpfmul_hadds,2 %b[46], %g24, %r40, %g24 } { loop_mode qppermb,3 %b[27], %g27, %r24, %g27 qppermb,4 %b[31], %g26, %r24, %g26 } { loop_mode qppermb,0 %b[35], %b[38], %r24, %b[22] qppermb,1 %b[51], %b[42], %r24, %b[26] qppermb,3 %b[47], %g30, %r24, %g30 qppermb,4 %b[55], %b[41], %r24, %b[27] } { loop_mode qppermb,0 %b[32], %b[29], %r24, %b[29] qppermb,1 %b[43], %g31, %r24, %g31 qppermb,4 %b[28], %b[30], %r24, %b[28] } { loop_mode qppermb,3 %b[40], %b[53], %r24, %b[30] qppermb,4 %b[48], %b[33], %r24, %b[31] qpfsubs,5 %g27, %b[28], %b[32] } { loop_mode qppermb,0 %b[56], %b[57], %r24, %b[33] qppermb,1 %b[36], %b[54], %r24, %b[35] qpfsubs,2 %g26, %b[29], %b[37] qppermb,3 %b[44], %b[34], %r24, %b[34] qppermb,4 %b[52], %b[58], %r24, %b[36] qpfsubs,5 %b[25], %b[30], %b[38] } { loop_mode qpfsubs,0 %b[27], %b[33], %b[42] qppermb,1 %g16, %b[73], %r24, %g16 qpfsubs,2 %b[22], %b[35], %b[39] qpfsubs,3 %b[26], %b[36], %b[41] qppermb,4 %g20, %b[21], %r24, %g20 qpfsubs,5 %g30, %b[31], %b[40] } { loop_mode qppermb,0 %g28, %b[76], %r24, %g28 qppermb,1 %g24, %b[74], %r24, %g24 qpfadds,2 %g27, %b[28], %g27 qppermb,3 %b[23], %b[81], %r24, %b[23] qppermb,4 %b[19], %b[59], %r24, %b[19] qpfsubs,5 %g31, %b[34], %b[21] } { loop_mode qpfadds,0 %g26, %b[29], %g26 qppermb,1 %g22, %b[83], %r24, %g22 qpfadds,2 %b[25], %b[30], %b[25] qpfadds,3 %g30, %b[31], %g30 qppermb,4 %g18, %b[82], %r24, %g18 qpfsubs,5 %g21, %g20, %b[28] } { loop_mode qpfadds,0 %b[27], %b[33], %b[27] qpfadds,1 %g17, %g16, %b[29] qpfsubs,2 %g17, %g16, %g16 qpfadds,3 %b[22], %b[35], %g17 qpfadds,4 %g31, %b[34], %g31 qpfadds,5 %b[26], %b[36], %b[22] } { loop_mode qpfadds,0 %g21, %g20, %g20 qpfadds,1 %g29, %g28, %g21 qpfsubs,2 %g29, %g28, %g28 qpfadds,3 %b[24], %b[23], %g29 qpfsubs,4 %b[24], %b[23], %b[23] qpfadds,5 %b[20], %b[19], %b[24] } { loop_mode qpfadds,0 %g25, %g24, %b[26] qpfsubs,1 %g25, %g24, %g24 qpfsubs,2 %b[20], %b[19], %g25 qpfsubs,3 %g19, %g18, %g18 qpfadds,5 %g19, %g18, %b[19] } { loop_mode qpfsubs,2 %g23, %g22, %g19 qpfadds,5 %g23, %g22, %g22 } { loop_mode qpfadds,0 %b[29], %g27, %g27 qpfsubs,2 %b[29], %g27, %b[20] qpshufb,4 %b[32], %b[32], %r23, %g23 } { loop_mode qpshufb,0 %b[37], %b[37], %r23, %b[29] qpshufb,1 %b[38], %b[38], %r23, %b[30] qpfsubs,2 %g20, %g26, %b[33] qpshufb,3 %b[40], %b[40], %r23, %b[31] qpshufb,4 %b[42], %b[42], %r23, %b[32] qpfadds,5 %g29, %g30, %b[34] } { loop_mode qpfadds,0 %g20, %g26, %g20 qpshufb,1 %b[39], %b[39], %r23, %b[35] qpfadds,2 %b[26], %g17, %g26 qpshufb,3 %b[21], %b[21], %r23, %b[21] qpshufb,4 %b[41], %b[41], %r23, %b[36] qpfsubs,5 %g29, %g30, %g29 } { loop_mode qpxor,0 %b[29], %r22, %g30 qpxor,1 %b[30], %r22, %b[29] qpfadds,2 %g21, %b[25], %b[31] qpxor,3 %g23, %r22, %g23 qpxor,4 %b[31], %r22, %b[30] qpfsubs,5 %g21, %b[25], %g21 } { loop_mode qpfsubs,0 %b[26], %g17, %g17 qpxor,1 %b[35], %r22, %b[32] qpfsubs,2 %b[24], %g31, %b[26] qpxor,3 %b[32], %r22, %b[25] qpxor,4 %b[21], %r22, %b[21] qpfadds,5 %b[24], %g31, %g31 } { loop_mode qpfsubs,0 %g22, %b[27], %b[35] qpfadds,1 %b[19], %b[22], %b[36] qpfadds,2 %g22, %b[27], %g22 qpxor,3 %b[36], %r22, %b[24] qpfsubs,4 %b[19], %b[22], %b[19] qpfsubs,5 %g16, %g23, %b[22] } { loop_mode qpfsubs,0 %b[28], %g30, %b[27] qpfadds,1 %b[28], %g30, %g30 qpfadds,2 %g28, %b[29], %g23 qpfadds,3 %g16, %g23, %g16 qpfsubs,4 %b[23], %b[30], %b[28] qpfadds,5 %b[23], %b[30], %b[23] } { loop_mode qpfsubs,0 %g28, %b[29], %g28 qpfsubs,1 %g24, %b[32], %b[25] qpfadds,2 %g24, %b[32], %g24 qpfadds,3 %g19, %b[25], %b[30] qpfsubs,4 %g19, %b[25], %g19 qpfsubs,5 %g25, %b[21], %b[29] } { loop_mode nop 1 qpfadds,2 %g25, %b[21], %g25 qpfsubs,3 %g18, %b[24], %g18 qpfadds,5 %g18, %b[24], %b[21] } { loop_mode qpshufb,0 %b[68], %b[68], %r23, %b[24] qpshufb,1 %b[67], %b[67], %r23, %b[32] qpshufb,4 %b[62], %b[62], %r23, %b[37] } { loop_mode qpshufb,0 %b[65], %b[65], %r23, %b[38] qpshufb,1 %b[61], %b[61], %r23, %b[39] qpshufb,3 %b[71], %b[71], %r23, %b[40] qpshufb,4 %b[72], %b[72], %r23, %b[41] } { loop_mode qpshufb,0 %b[77], %b[77], %r23, %b[42] qpshufb,1 %b[66], %b[66], %r23, %b[43] qpshufb,3 %b[78], %b[78], %r23, %b[44] qpshufb,4 %b[85], %b[85], %r23, %b[45] } { loop_mode qpshufb,0 %b[86], %b[86], %r23, %b[46] qpshufb,1 %b[63], %b[63], %r23, %b[47] qpshufb,3 %b[87], %b[87], %r23, %b[48] qpshufb,4 %b[90], %b[90], %r23, %b[49] } { loop_mode qpshufb,0 %b[89], %b[89], %r23, %b[50] qpshufb,1 %g21, %b[33], %r7, %b[51] qpshufb,3 %g26, %g27, %r7, %b[52] qpshufb,4 %b[31], %g20, %r7, %b[53] } { loop_mode qpshufb,0 %g17, %b[20], %r7, %b[54] qpshufb,1 %b[35], %g29, %r7, %b[55] qpfmul_hsubs,2 %b[85], %b[51], %r40, %b[58] qpshufb,3 %b[36], %g31, %r7, %b[56] qpshufb,4 %g22, %b[34], %r7, %b[57] qpfmul_hadds,5 %b[37], %b[53], %r40, %b[37] } { loop_mode qpshufb,0 %b[19], %b[26], %r7, %b[59] qpshufb,1 %g23, %g30, %r7, %b[73] qpfmul_hadds,2 %b[45], %b[51], %r40, %b[45] qpshufb,3 %g28, %b[27], %r7, %b[74] qpshufb,4 %g24, %g16, %r7, %b[76] qpfmul_hsubs,5 %b[62], %b[53], %r40, %b[51] } { loop_mode qpshufb,0 %g19, %b[28], %r7, %b[53] qpshufb,1 %b[30], %b[23], %r7, %b[62] qpfmul_hadds,2 %b[44], %b[54], %r40, %b[44] qpshufb,3 %b[25], %b[22], %r7, %b[81] qpshufb,4 %g18, %b[29], %r7, %b[82] qpfmul_hadds,5 %b[24], %b[52], %r40, %b[24] } { loop_mode qpshufb,0 %b[21], %g25, %r7, %b[83] qpfmul_hsubs,1 %b[68], %b[52], %r40, %b[52] qpfmul_hsubs,2 %b[63], %b[55], %r40, %b[63] qpfmul_hsubs,3 %b[67], %b[56], %r40, %b[67] qpfmul_hadds,4 %b[32], %b[56], %r40, %b[32] qpfmul_hadds,5 %b[39], %b[57], %r40, %b[39] } { loop_mode qpfmul_hsubs,0 %b[78], %b[54], %r40, %b[54] qpfmul_hadds,1 %b[47], %b[55], %r40, %b[47] qpfmul_hadds,2 %b[49], %b[73], %r40, %b[49] qpfmul_hsubs,3 %b[61], %b[57], %r40, %b[55] qpfmul_hsubs,4 %b[87], %b[76], %r40, %b[56] qpfmul_hsubs,5 %b[72], %b[74], %r40, %b[57] } { loop_mode qpfmul_hsubs,0 %b[77], %b[59], %r40, %b[61] qpfmul_hadds,1 %b[42], %b[59], %r40, %b[42] qpfmul_hsubs,2 %b[90], %b[73], %r40, %b[59] qpfmul_hadds,3 %b[48], %b[76], %r40, %b[48] qpfmul_hadds,4 %b[41], %b[74], %r40, %b[41] qpfmul_hsubs,5 %b[71], %b[81], %r40, %b[68] } { loop_mode qpfmul_hsubs,0 %b[66], %b[53], %r40, %b[66] qpfmul_hadds,1 %b[43], %b[53], %r40, %b[43] qpfmul_hsubs,2 %b[89], %b[62], %r40, %b[53] qpfmul_hadds,3 %b[50], %b[62], %r40, %b[50] qpfmul_hadds,4 %b[40], %b[81], %r40, %b[40] qpfmul_hsubs,5 %b[65], %b[82], %r40, %b[62] } { loop_mode qpfmul_hadds,0 %b[46], %b[83], %r40, %b[46] qpshufb,1 %b[70], %b[70], %r23, %b[71] qpfmul_hsubs,2 %b[86], %b[83], %r40, %b[65] qpshufb,3 %b[64], %b[64], %r23, %b[72] qpshufb,4 %b[69], %b[69], %r23, %b[73] qpfmul_hadds,5 %b[38], %b[82], %r40, %b[38] } { loop_mode qpshufb,0 %b[60], %b[60], %r23, %b[74] qpshufb,1 %b[80], %b[80], %r23, %b[76] qpshufb,3 %b[79], %b[79], %r23, %b[77] qpshufb,4 %b[75], %b[75], %r23, %b[78] } { loop_mode nop 3 qpshufb,0 %b[88], %b[88], %r23, %b[81] } { loop_mode qpshufb,1 %b[31], %g20, %r6, %g20 qpshufb,3 %g21, %b[33], %r6, %g21 qpshufb,4 %b[35], %g29, %r6, %g29 } { loop_mode qpshufb,0 %g22, %b[34], %r6, %g22 qpshufb,1 %g28, %b[27], %r6, %g28 qpfmul_hadds,2 %b[71], %g20, %r40, %b[23] qpshufb,3 %g23, %g30, %r6, %g23 qpshufb,4 %b[30], %b[23], %r6, %g30 qpfmul_hadds,5 %b[76], %g21, %r40, %b[27] } { loop_mode qpshufb,0 %g19, %b[28], %r6, %g19 qpfmul_hsubs,1 %b[70], %g20, %r40, %g20 qpfmul_hadds,2 %b[73], %g22, %r40, %b[30] qpfmul_hsubs,3 %b[80], %g21, %r40, %g21 qpfmul_hsubs,4 %b[79], %g29, %r40, %b[28] qpfmul_hadds,5 %b[77], %g29, %r40, %g29 } { loop_mode qpfmul_hsubs,0 %b[69], %g22, %r40, %g22 qpfmul_hadds,1 %b[72], %g28, %r40, %b[33] qpfmul_hsubs,2 %b[64], %g28, %r40, %g28 qpfmul_hsubs,3 %b[88], %g23, %r40, %b[31] qpfmul_hadds,4 %b[81], %g23, %r40, %g23 qpfmul_hsubs,5 %b[75], %g30, %r40, %b[34] } { loop_mode qpfmul_hadds,0 %b[74], %g19, %r40, %g19 qppermb,1 %b[45], %b[58], %r24, %b[45] qpfmul_hsubs,2 %b[60], %g19, %r40, %b[35] qppermb,3 %b[24], %b[52], %r24, %b[24] qppermb,4 %b[44], %b[54], %r24, %b[44] qpfmul_hadds,5 %b[78], %g30, %r40, %g30 } { loop_mode qppermb,0 %b[37], %b[51], %r24, %b[37] qppermb,1 %b[32], %b[67], %r24, %b[32] qppermb,3 %b[42], %b[61], %r24, %b[42] qppermb,4 %b[39], %b[55], %r24, %b[39] } { loop_mode qppermb,0 %b[47], %b[63], %r24, %b[47] qppermb,1 %b[49], %b[59], %r24, %b[49] qppermb,3 %b[40], %b[68], %r24, %b[40] qppermb,4 %b[48], %b[56], %r24, %b[48] } { loop_mode qppermb,0 %b[43], %b[66], %r24, %b[43] qppermb,1 %b[50], %b[53], %r24, %b[50] qppermb,3 %b[38], %b[62], %r24, %b[38] qppermb,4 %b[41], %b[57], %r24, %b[41] } { loop_mode qppermb,0 %b[46], %b[65], %r24, %b[46] qpfsubs,1 %b[24], %b[37], %b[51] qpfsubs,3 %b[44], %b[45], %b[52] qpfsubs,4 %b[40], %b[41], %b[53] } { loop_mode qpfsubs,0 %b[42], %b[47], %b[55] qpfsubs,1 %b[46], %b[50], %b[56] qpfsubs,2 %b[32], %b[39], %b[54] qpfadds,3 %b[24], %b[37], %b[24] qpfadds,4 %b[44], %b[45], %b[37] qpfadds,5 %b[32], %b[39], %b[32] } { loop_mode qpfadds,0 %b[48], %b[49], %b[44] qpfadds,1 %b[42], %b[47], %b[42] qpfsubs,2 %b[48], %b[49], %b[39] qpfadds,3 %b[40], %b[41], %b[40] } { loop_mode qpfadds,0 %b[38], %b[43], %b[38] qpfadds,1 %b[46], %b[50], %b[43] qpfsubs,2 %b[38], %b[43], %b[41] } { loop_mode qpshufb,4 %g26, %g27, %r6, %g26 } { loop_mode qpshufb,3 %g17, %b[20], %r6, %g17 qpshufb,4 %b[36], %g31, %r6, %g27 } { loop_mode qpshufb,0 %b[19], %b[26], %r6, %g31 qpshufb,1 %g24, %g16, %r6, %g16 qpshufb,3 %b[25], %b[22], %r6, %g24 qpshufb,4 %g18, %b[29], %r6, %g18 } { loop_mode qpshufb,0 %b[21], %g25, %r6, %g25 qppermb,1 %b[23], %g20, %r24, %g20 qppermb,3 %b[27], %g21, %r24, %g21 qppermb,4 %g29, %b[28], %r24, %g29 } { loop_mode qppermb,0 %b[30], %g22, %r24, %g22 qppermb,1 %b[33], %g28, %r24, %g28 qpfsubs,2 %g26, %g20, %b[19] qppermb,3 %g23, %b[31], %r24, %g23 qppermb,4 %g30, %b[34], %r24, %g30 qpfsubs,5 %g17, %g21, %b[20] } { loop_mode qppermb,0 %g19, %b[35], %r24, %g19 qpfsubs,1 %g27, %g22, %g21 qpfadds,2 %g26, %g20, %g20 qpshufb,3 %b[51], %b[51], %r23, %g26 qpshufb,4 %b[52], %b[52], %r23, %b[21] qpfadds,5 %g17, %g21, %g17 } { loop_mode qpfadds,0 %g27, %g22, %g22 qpfsubs,1 %g24, %g28, %g31 qpfadds,2 %g24, %g28, %g24 qpfadds,3 %g31, %g29, %b[22] qpfsubs,4 %g31, %g29, %g29 qpfadds,5 %g16, %g23, %g27 } { loop_mode nop 1 qpfsubs,0 %g18, %g19, %g18 qpfadds,2 %g18, %g19, %g28 qpfadds,3 %g25, %g30, %g23 qpfsubs,4 %g25, %g30, %g25 qpfsubs,5 %g16, %g23, %g16 } { loop_mode qpfadds,0 %g20, %b[24], %g30 qpshufb,1 %b[54], %b[54], %r23, %g19 qpfsubs,2 %g20, %b[24], %g20 qpfadds,3 %g17, %b[37], %b[23] qpfsubs,4 %g17, %b[37], %g17 } { loop_mode qpshufb,0 %b[39], %b[39], %r23, %b[24] qpshufb,1 %b[41], %b[41], %r23, %b[25] qpfsubs,2 %g22, %b[32], %b[28] qpshufb,3 %b[55], %b[55], %r23, %b[26] qpshufb,4 %b[53], %b[53], %r23, %b[27] qpfsubs,5 %b[22], %b[42], %b[29] } { loop_mode qpfadds,0 %g22, %b[32], %g22 qpshufb,1 %b[56], %b[56], %r23, %b[30] qpfsubs,2 %g24, %b[40], %b[33] qpfadds,3 %b[22], %b[42], %b[22] qpfsubs,4 %g27, %b[44], %b[31] qpfadds,5 %g23, %b[43], %b[32] } { loop_mode qpxor,0 %g26, %r22, %g26 qpxor,1 %g19, %r22, %g19 qpfadds,2 %g27, %b[44], %g27 qpxor,3 %b[21], %r22, %b[21] qpxor,4 %b[26], %r22, %b[26] qpfsubs,5 %g23, %b[43], %g23 } { loop_mode qpfadds,0 %g24, %b[40], %g24 qpxor,1 %b[25], %r22, %b[25] qpfsubs,2 %b[19], %g26, %b[34] qpfadds,3 %g28, %b[38], %b[35] qpfsubs,4 %g28, %b[38], %g28 qpfsubs,5 %b[20], %b[21], %b[36] } { loop_mode qpxor,0 %b[24], %r22, %b[24] qpxor,1 %b[27], %r22, %b[27] qpfadds,2 %b[19], %g26, %g26 qpfadds,3 %b[20], %b[21], %b[19] qpfadds,4 %g29, %b[26], %b[20] qpfsubs,5 %g29, %b[26], %g29 } { loop_mode qpfsubs,0 %g21, %g19, %b[26] qpxor,1 %b[30], %r22, %b[21] qpfadds,2 %g21, %g19, %g19 stqp,5 %r16, %r0, %g17 } { loop_mode qpfadds,0 %g16, %b[24], %g17 qpfsubs,1 %g18, %b[25], %g21 qpfadds,2 %g18, %b[25], %g18 stqp,5 %r20, %r0, %b[23] } { loop_mode qpfsubs,0 %g16, %b[24], %g16 qpfsubs,1 %g31, %b[27], %b[23] qpfadds,2 %g31, %b[27], %g31 stqp,5 %r18, %r0, %g20 } { loop_mode qpfadds,0 %g25, %b[21], %g20 qpfsubs,1 %g25, %b[21], %g25 stqp,2 %r2, %r0, %g30 stqp,5 %r36, %r0, %b[29] } { loop_mode stqp,2 %r25, %r0, %b[22] stqp,5 %r32, %r0, %b[28] } { loop_mode stqp,2 %r27, %r0, %g22 stqp,5 %r3, %r0, %b[31] } { loop_mode stqp,2 %r9, %r0, %b[32] stqp,5 %r35, %r0, %g23 } { loop_mode stqp,2 %r17, %r0, %g27 stqp,5 %r13, %r0, %b[33] } { loop_mode stqp,2 %r21, %r0, %g24 stqp,5 %r31, %r0, %g28 } { loop_mode stqp,2 %r15, %r0, %g26 stqp,5 %r26, %r0, %b[35] } { loop_mode stqp,2 %r19, %r0, %b[34] stqp,5 %r34, %r0, %g19 } { loop_mode stqp,2 %r30, %r0, %b[26] stqp,5 %r39, %r0, %g18 } { loop_mode stqp,2 %r5, %r0, %b[19] stqp,5 %r29, %r0, %g21 } { loop_mode stqp,2 %r11, %r0, %b[36] stqp,5 %r4, %r0, %g17 } { loop_mode stqp,2 %r38, %r0, %b[20] stqp,5 %r14, %r0, %g16 } { loop_mode stqp,2 %r28, %r0, %g29 stqp,5 %r1, %r0, %g31 } { loop_mode stqp,2 %r12, %r0, %b[23] stqp,5 %r37, %r0, %g20 } { loop_mode ct %ctpr1 ? %NOT_LOOP_END alc alcf=1, alct=1 stqp,2 %r33, %r0, %g25 addd,3,sm %r0, _f16s,_lts0lo 0x20, %r0 }
Теоретическая скорость: 64 комплексных числа за 106 тактов (64/106) = 4.83 Байт/такт
Четверная теоретическая скорость: 19.32 Байт/такт
Замеры скорости
7. stage_radix4_2x_simd128_noConj_unroll3
Здесь происходит раскрутка цикла в 3 раза с помощью опции unroll.
Код на Си
void stage_radix4_2x_simd128_noConj_unroll3(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coefC_a, myComplex *coefD_a, myComplex *coefE_a, myComplex *coefC_b, myComplex *coefD_b, myComplex *coefE_b) { __v2di *xy0_in = (__v2di*)&data_in[ 0]; __v2di *zw0_in = (__v2di*)&data_in[ 2]; __v2di *xy1_in = (__v2di*)&data_in[ 4]; __v2di *zw1_in = (__v2di*)&data_in[ 6]; __v2di *xy2_in = (__v2di*)&data_in[ 8]; __v2di *zw2_in = (__v2di*)&data_in[10]; __v2di *xy3_in = (__v2di*)&data_in[12]; __v2di *zw3_in = (__v2di*)&data_in[14]; __v2di *xy4_in = (__v2di*)&data_in[16]; __v2di *zw4_in = (__v2di*)&data_in[18]; __v2di *xy5_in = (__v2di*)&data_in[20]; __v2di *zw5_in = (__v2di*)&data_in[22]; __v2di *xy6_in = (__v2di*)&data_in[24]; __v2di *zw6_in = (__v2di*)&data_in[26]; __v2di *xy7_in = (__v2di*)&data_in[28]; __v2di *zw7_in = (__v2di*)&data_in[30]; __v2di *c0a_in = (__v2di*)&coefC_a[0]; __v2di *c1a_in = (__v2di*)&coefC_a[2]; __v2di *c2a_in = (__v2di*)&coefC_a[4]; __v2di *c3a_in = (__v2di*)&coefC_a[6]; __v2di *d0a_in = (__v2di*)&coefD_a[0]; __v2di *d1a_in = (__v2di*)&coefD_a[2]; __v2di *d2a_in = (__v2di*)&coefD_a[4]; __v2di *d3a_in = (__v2di*)&coefD_a[6]; __v2di *e0a_in = (__v2di*)&coefE_a[0]; __v2di *e1a_in = (__v2di*)&coefE_a[2]; __v2di *e2a_in = (__v2di*)&coefE_a[4]; __v2di *e3a_in = (__v2di*)&coefE_a[6]; __v2di *c0b_in = (__v2di*)&coefC_b[0*data_count/16]; __v2di *c1b_in = (__v2di*)&coefC_b[1*data_count/16]; __v2di *c2b_in = (__v2di*)&coefC_b[2*data_count/16]; __v2di *c3b_in = (__v2di*)&coefC_b[3*data_count/16]; __v2di *d0b_in = (__v2di*)&coefD_b[0*data_count/16]; __v2di *d1b_in = (__v2di*)&coefD_b[1*data_count/16]; __v2di *d2b_in = (__v2di*)&coefD_b[2*data_count/16]; __v2di *d3b_in = (__v2di*)&coefD_b[3*data_count/16]; __v2di *e0b_in = (__v2di*)&coefE_b[0*data_count/16]; __v2di *e1b_in = (__v2di*)&coefE_b[1*data_count/16]; __v2di *e2b_in = (__v2di*)&coefE_b[2*data_count/16]; __v2di *e3b_in = (__v2di*)&coefE_b[3*data_count/16]; __v2di *out_0 = (__v2di*)&data_out[ 0*data_count/16]; __v2di *out_1 = (__v2di*)&data_out[ 1*data_count/16]; __v2di *out_2 = (__v2di*)&data_out[ 2*data_count/16]; __v2di *out_3 = (__v2di*)&data_out[ 3*data_count/16]; __v2di *out_4 = (__v2di*)&data_out[ 4*data_count/16]; __v2di *out_5 = (__v2di*)&data_out[ 5*data_count/16]; __v2di *out_6 = (__v2di*)&data_out[ 6*data_count/16]; __v2di *out_7 = (__v2di*)&data_out[ 7*data_count/16]; __v2di *out_8 = (__v2di*)&data_out[ 8*data_count/16]; __v2di *out_9 = (__v2di*)&data_out[ 9*data_count/16]; __v2di *out_10 = (__v2di*)&data_out[10*data_count/16]; __v2di *out_11 = (__v2di*)&data_out[11*data_count/16]; __v2di *out_12 = (__v2di*)&data_out[12*data_count/16]; __v2di *out_13 = (__v2di*)&data_out[13*data_count/16]; __v2di *out_14 = (__v2di*)&data_out[14*data_count/16]; __v2di *out_15 = (__v2di*)&data_out[15*data_count/16]; #pragma ivdep #pragma unroll(3) #pragma prefetch for(int64_t i = 0; i < data_count/32; ++i) { __v2di xy0 = xy0_in[16*i]; __v2di zw0 = zw0_in[16*i]; __v2di xy1 = xy1_in[16*i]; __v2di zw1 = zw1_in[16*i]; __v2di c0 = c0a_in[4*i]; __v2di d0 = d0a_in[4*i]; __v2di e0 = e0a_in[4*i]; __v2di xy2 = xy2_in[16*i]; __v2di zw2 = zw2_in[16*i]; __v2di xy3 = xy3_in[16*i]; __v2di zw3 = zw3_in[16*i]; __v2di c1 = c1a_in[4*i]; __v2di d1 = d1a_in[4*i]; __v2di e1 = e1a_in[4*i]; __v2di xy4 = xy4_in[16*i]; __v2di zw4 = zw4_in[16*i]; __v2di xy5 = xy5_in[16*i]; __v2di zw5 = zw5_in[16*i]; __v2di c2 = c2a_in[4*i]; __v2di d2 = d2a_in[4*i]; __v2di e2 = e2a_in[4*i]; __v2di xy6 = xy6_in[16*i]; __v2di zw6 = zw6_in[16*i]; __v2di xy7 = xy7_in[16*i]; __v2di zw7 = zw7_in[16*i]; __v2di c3 = c3a_in[4*i]; __v2di d3 = d3a_in[4*i]; __v2di e3 = e3a_in[4*i]; __v2di x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c2 = __builtin_e2k_qpshufb(c2, c2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_c3 = __builtin_e2k_qpshufb(c3, c3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d0 = __builtin_e2k_qpshufb(d0, d0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d1 = __builtin_e2k_qpshufb(d1, d1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d2 = __builtin_e2k_qpshufb(d2, d2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d3 = __builtin_e2k_qpshufb(d3, d3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e0 = __builtin_e2k_qpshufb(e0, e0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e1 = __builtin_e2k_qpshufb(e1, e1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e2 = __builtin_e2k_qpshufb(e2, e2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e3 = __builtin_e2k_qpshufb(e3, e3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy0_real = __builtin_e2k_qpfmuls( c0, y0); __v2di cy1_real = __builtin_e2k_qpfmuls( c1, y1); __v2di cy2_real = __builtin_e2k_qpfmuls( c2, y2); __v2di cy3_real = __builtin_e2k_qpfmuls( c3, y3); __v2di dz0_real = __builtin_e2k_qpfmuls( d0, z0); __v2di dz1_real = __builtin_e2k_qpfmuls( d1, z1); __v2di dz2_real = __builtin_e2k_qpfmuls( d2, z2); __v2di dz3_real = __builtin_e2k_qpfmuls( d3, z3); __v2di ew0_real = __builtin_e2k_qpfmuls( e0, w0); __v2di ew1_real = __builtin_e2k_qpfmuls( e1, w1); __v2di ew2_real = __builtin_e2k_qpfmuls( e2, w2); __v2di ew3_real = __builtin_e2k_qpfmuls( e3, w3); __v2di cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); __v2di cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); __v2di cy2_imag = __builtin_e2k_qpfmuls(swap_c2, y2); __v2di cy3_imag = __builtin_e2k_qpfmuls(swap_c3, y3); __v2di dz0_imag = __builtin_e2k_qpfmuls(swap_d0, z0); __v2di dz1_imag = __builtin_e2k_qpfmuls(swap_d1, z1); __v2di dz2_imag = __builtin_e2k_qpfmuls(swap_d2, z2); __v2di dz3_imag = __builtin_e2k_qpfmuls(swap_d3, z3); __v2di ew0_imag = __builtin_e2k_qpfmuls(swap_e0, w0); __v2di ew1_imag = __builtin_e2k_qpfmuls(swap_e1, w1); __v2di ew2_imag = __builtin_e2k_qpfmuls(swap_e2, w2); __v2di ew3_imag = __builtin_e2k_qpfmuls(swap_e3, w3); __v2di cy0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy0_real); __v2di cy1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy1_real); __v2di cy2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy2_real); __v2di cy3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy3_real); __v2di dz0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz0_real); __v2di dz1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz1_real); __v2di dz2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz2_real); __v2di dz3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz3_real); __v2di ew0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew0_real); __v2di ew1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew1_real); __v2di ew2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew2_real); __v2di ew3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew3_real); __v2di cy0_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy0_imag); __v2di cy1_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy1_imag); __v2di cy2_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy2_imag); __v2di cy3_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy3_imag); __v2di dz0_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz0_imag); __v2di dz1_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz1_imag); __v2di dz2_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz2_imag); __v2di dz3_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz3_imag); __v2di ew0_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew0_imag); __v2di ew1_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew1_imag); __v2di ew2_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew2_imag); __v2di ew3_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew3_imag); __v2di cy0 = __builtin_e2k_qppermb(cy0_ii, cy0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di cy1 = __builtin_e2k_qppermb(cy1_ii, cy1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di cy2 = __builtin_e2k_qppermb(cy2_ii, cy2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di cy3 = __builtin_e2k_qppermb(cy3_ii, cy3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz0 = __builtin_e2k_qppermb(dz0_ii, dz0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz1 = __builtin_e2k_qppermb(dz1_ii, dz1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz2 = __builtin_e2k_qppermb(dz2_ii, dz2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz3 = __builtin_e2k_qppermb(dz3_ii, dz3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew0 = __builtin_e2k_qppermb(ew0_ii, ew0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew1 = __builtin_e2k_qppermb(ew1_ii, ew1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew2 = __builtin_e2k_qppermb(ew2_ii, ew2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew3 = __builtin_e2k_qppermb(ew3_ii, ew3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di add02_0 = __builtin_e2k_qpfadds( x0, dz0); __v2di add02_1 = __builtin_e2k_qpfadds( x1, dz1); __v2di add02_2 = __builtin_e2k_qpfadds( x2, dz2); __v2di add02_3 = __builtin_e2k_qpfadds( x3, dz3); __v2di sub02_0 = __builtin_e2k_qpfsubs( x0, dz0); __v2di sub02_1 = __builtin_e2k_qpfsubs( x1, dz1); __v2di sub02_2 = __builtin_e2k_qpfsubs( x2, dz2); __v2di sub02_3 = __builtin_e2k_qpfsubs( x3, dz3); __v2di add13_0 = __builtin_e2k_qpfadds(cy0, ew0); __v2di add13_1 = __builtin_e2k_qpfadds(cy1, ew1); __v2di add13_2 = __builtin_e2k_qpfadds(cy2, ew2); __v2di add13_3 = __builtin_e2k_qpfadds(cy3, ew3); __v2di sub13_0 = __builtin_e2k_qpfsubs(cy0, ew0); __v2di sub13_1 = __builtin_e2k_qpfsubs(cy1, ew1); __v2di sub13_2 = __builtin_e2k_qpfsubs(cy2, ew2); __v2di sub13_3 = __builtin_e2k_qpfsubs(cy3, ew3); __v2di swap_sub13_0 = __builtin_e2k_qpshufb(sub13_0, sub13_0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_1 = __builtin_e2k_qpshufb(sub13_1, sub13_1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_2 = __builtin_e2k_qpshufb(sub13_2, sub13_2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_3 = __builtin_e2k_qpshufb(sub13_3, sub13_3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di sub13i_0 = __builtin_e2k_qpxor(swap_sub13_0, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_1 = __builtin_e2k_qpxor(swap_sub13_1, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_2 = __builtin_e2k_qpxor(swap_sub13_2, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_3 = __builtin_e2k_qpxor(swap_sub13_3, (__v2di){1LL<<31, 1LL<<31}); __v2di out0 = __builtin_e2k_qpfadds(add02_0, add13_0); __v2di out1 = __builtin_e2k_qpfadds(add02_1, add13_1); __v2di out2 = __builtin_e2k_qpfadds(add02_2, add13_2); __v2di out3 = __builtin_e2k_qpfadds(add02_3, add13_3); __v2di out4 = __builtin_e2k_qpfsubs(sub02_0, sub13i_0); __v2di out5 = __builtin_e2k_qpfsubs(sub02_1, sub13i_1); __v2di out6 = __builtin_e2k_qpfsubs(sub02_2, sub13i_2); __v2di out7 = __builtin_e2k_qpfsubs(sub02_3, sub13i_3); __v2di out8 = __builtin_e2k_qpfsubs(add02_0, add13_0); __v2di out9 = __builtin_e2k_qpfsubs(add02_1, add13_1); __v2di out10 = __builtin_e2k_qpfsubs(add02_2, add13_2); __v2di out11 = __builtin_e2k_qpfsubs(add02_3, add13_3); __v2di out12 = __builtin_e2k_qpfadds(sub02_0, sub13i_0); __v2di out13 = __builtin_e2k_qpfadds(sub02_1, sub13i_1); __v2di out14 = __builtin_e2k_qpfadds(sub02_2, sub13i_2); __v2di out15 = __builtin_e2k_qpfadds(sub02_3, sub13i_3); xy0 = out0; zw0 = out1; xy1 = out2; zw1 = out3; c0 = c0b_in[i]; d0 = d0b_in[i]; e0 = e0b_in[i]; xy2 = out4; zw2 = out5; xy3 = out6; zw3 = out7; c1 = c1b_in[i]; d1 = d1b_in[i]; e1 = e1b_in[i]; xy4 = out8; zw4 = out9; xy5 = out10; zw5 = out11; c2 = c2b_in[i]; d2 = d2b_in[i]; e2 = e2b_in[i]; xy6 = out12; zw6 = out13; xy7 = out14; zw7 = out15; c3 = c3b_in[i]; d3 = d3b_in[i]; e3 = e3b_in[i]; x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); w0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0706050403020100, 0x0706050403020100}); w1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0706050403020100, 0x0706050403020100}); y2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0706050403020100, 0x0706050403020100}); w2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0706050403020100, 0x0706050403020100}); y3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0706050403020100, 0x0706050403020100}); w3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); swap_c0 = __builtin_e2k_qpshufb(c0, c0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c1 = __builtin_e2k_qpshufb(c1, c1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c2 = __builtin_e2k_qpshufb(c2, c2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_c3 = __builtin_e2k_qpshufb(c3, c3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d0 = __builtin_e2k_qpshufb(d0, d0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d1 = __builtin_e2k_qpshufb(d1, d1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d2 = __builtin_e2k_qpshufb(d2, d2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_d3 = __builtin_e2k_qpshufb(d3, d3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e0 = __builtin_e2k_qpshufb(e0, e0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e1 = __builtin_e2k_qpshufb(e1, e1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e2 = __builtin_e2k_qpshufb(e2, e2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_e3 = __builtin_e2k_qpshufb(e3, e3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); cy0_real = __builtin_e2k_qpfmuls( c0, y0); cy1_real = __builtin_e2k_qpfmuls( c1, y1); cy2_real = __builtin_e2k_qpfmuls( c2, y2); cy3_real = __builtin_e2k_qpfmuls( c3, y3); dz0_real = __builtin_e2k_qpfmuls( d0, z0); dz1_real = __builtin_e2k_qpfmuls( d1, z1); dz2_real = __builtin_e2k_qpfmuls( d2, z2); dz3_real = __builtin_e2k_qpfmuls( d3, z3); ew0_real = __builtin_e2k_qpfmuls( e0, w0); ew1_real = __builtin_e2k_qpfmuls( e1, w1); ew2_real = __builtin_e2k_qpfmuls( e2, w2); ew3_real = __builtin_e2k_qpfmuls( e3, w3); cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); cy2_imag = __builtin_e2k_qpfmuls(swap_c2, y2); cy3_imag = __builtin_e2k_qpfmuls(swap_c3, y3); dz0_imag = __builtin_e2k_qpfmuls(swap_d0, z0); dz1_imag = __builtin_e2k_qpfmuls(swap_d1, z1); dz2_imag = __builtin_e2k_qpfmuls(swap_d2, z2); dz3_imag = __builtin_e2k_qpfmuls(swap_d3, z3); ew0_imag = __builtin_e2k_qpfmuls(swap_e0, w0); ew1_imag = __builtin_e2k_qpfmuls(swap_e1, w1); ew2_imag = __builtin_e2k_qpfmuls(swap_e2, w2); ew3_imag = __builtin_e2k_qpfmuls(swap_e3, w3); cy0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy0_real); cy1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy1_real); cy2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy2_real); cy3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy3_real); dz0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz0_real); dz1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz1_real); dz2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz2_real); dz3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz3_real); ew0_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew0_real); ew1_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew1_real); ew2_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew2_real); ew3_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew3_real); cy0_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy0_imag); cy1_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy1_imag); cy2_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy2_imag); cy3_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy3_imag); dz0_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz0_imag); dz1_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz1_imag); dz2_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz2_imag); dz3_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz3_imag); ew0_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew0_imag); ew1_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew1_imag); ew2_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew2_imag); ew3_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew3_imag); cy0 = __builtin_e2k_qppermb(cy0_ii, cy0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); cy1 = __builtin_e2k_qppermb(cy1_ii, cy1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); cy2 = __builtin_e2k_qppermb(cy2_ii, cy2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); cy3 = __builtin_e2k_qppermb(cy3_ii, cy3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); dz0 = __builtin_e2k_qppermb(dz0_ii, dz0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); dz1 = __builtin_e2k_qppermb(dz1_ii, dz1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); dz2 = __builtin_e2k_qppermb(dz2_ii, dz2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); dz3 = __builtin_e2k_qppermb(dz3_ii, dz3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); ew0 = __builtin_e2k_qppermb(ew0_ii, ew0_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); ew1 = __builtin_e2k_qppermb(ew1_ii, ew1_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); ew2 = __builtin_e2k_qppermb(ew2_ii, ew2_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); ew3 = __builtin_e2k_qppermb(ew3_ii, ew3_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); add02_0 = __builtin_e2k_qpfadds( x0, dz0); add02_1 = __builtin_e2k_qpfadds( x1, dz1); add02_2 = __builtin_e2k_qpfadds( x2, dz2); add02_3 = __builtin_e2k_qpfadds( x3, dz3); sub02_0 = __builtin_e2k_qpfsubs( x0, dz0); sub02_1 = __builtin_e2k_qpfsubs( x1, dz1); sub02_2 = __builtin_e2k_qpfsubs( x2, dz2); sub02_3 = __builtin_e2k_qpfsubs( x3, dz3); add13_0 = __builtin_e2k_qpfadds(cy0, ew0); add13_1 = __builtin_e2k_qpfadds(cy1, ew1); add13_2 = __builtin_e2k_qpfadds(cy2, ew2); add13_3 = __builtin_e2k_qpfadds(cy3, ew3); sub13_0 = __builtin_e2k_qpfsubs(cy0, ew0); sub13_1 = __builtin_e2k_qpfsubs(cy1, ew1); sub13_2 = __builtin_e2k_qpfsubs(cy2, ew2); sub13_3 = __builtin_e2k_qpfsubs(cy3, ew3); swap_sub13_0 = __builtin_e2k_qpshufb(sub13_0, sub13_0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_1 = __builtin_e2k_qpshufb(sub13_1, sub13_1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_2 = __builtin_e2k_qpshufb(sub13_2, sub13_2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_3 = __builtin_e2k_qpshufb(sub13_3, sub13_3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); sub13i_0 = __builtin_e2k_qpxor(swap_sub13_0, (__v2di){1LL<<31, 1LL<<31}); sub13i_1 = __builtin_e2k_qpxor(swap_sub13_1, (__v2di){1LL<<31, 1LL<<31}); sub13i_2 = __builtin_e2k_qpxor(swap_sub13_2, (__v2di){1LL<<31, 1LL<<31}); sub13i_3 = __builtin_e2k_qpxor(swap_sub13_3, (__v2di){1LL<<31, 1LL<<31}); out_0[i] = __builtin_e2k_qpfadds(add02_0, add13_0); out_1[i] = __builtin_e2k_qpfadds(add02_1, add13_1); out_2[i] = __builtin_e2k_qpfadds(add02_2, add13_2); out_3[i] = __builtin_e2k_qpfadds(add02_3, add13_3); out_4[i] = __builtin_e2k_qpfsubs(sub02_0, sub13i_0); out_5[i] = __builtin_e2k_qpfsubs(sub02_1, sub13i_1); out_6[i] = __builtin_e2k_qpfsubs(sub02_2, sub13i_2); out_7[i] = __builtin_e2k_qpfsubs(sub02_3, sub13i_3); out_8[i] = __builtin_e2k_qpfsubs(add02_0, add13_0); out_9[i] = __builtin_e2k_qpfsubs(add02_1, add13_1); out_10[i] = __builtin_e2k_qpfsubs(add02_2, add13_2); out_11[i] = __builtin_e2k_qpfsubs(add02_3, add13_3); out_12[i] = __builtin_e2k_qpfadds(sub02_0, sub13i_0); out_13[i] = __builtin_e2k_qpfadds(sub02_1, sub13i_1); out_14[i] = __builtin_e2k_qpfadds(sub02_2, sub13i_2); out_15[i] = __builtin_e2k_qpfadds(sub02_3, sub13i_3); } }
Основной цикл на ассемблере
.L24812: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=1, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=1, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=2, disp=128 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=2, disp=160 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=3, disp=192 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=3, disp=224 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=4, disp=256 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=4, disp=288 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=5, disp=320 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=5, disp=352 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=6, disp=384 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=6, disp=416 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=7, disp=448 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=7, disp=480 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=8, disp=512 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=8, disp=544 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=9, disp=576 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=9, disp=608 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=10, disp=640 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=10, disp=672 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=11, disp=704 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=0, abs=11, disp=736 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=0, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=0, abs=12, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=0, abs=13, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=0, abs=13, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=0, abs=14, disp=128 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=4, asz=0, abs=14, disp=160 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=0, abs=15, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=0, abs=15, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=0, abs=16, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=0, abs=16, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=0, abs=17, disp=128 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=3, asz=0, abs=17, disp=160 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=0, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=0, abs=18, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=0, abs=19, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=0, abs=19, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=0, abs=20, disp=128 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=3, ind=2, asz=0, abs=20, disp=160 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=1, incr=2, ind=0, asz=0, abs=21, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=15, asz=0, abs=21, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=14, asz=0, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=13, asz=0, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=12, asz=0, abs=23, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=11, asz=0, abs=23, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=10, asz=0, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=9, asz=0, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=8, asz=0, abs=25, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=7, asz=0, abs=25, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=6, asz=0, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=5, asz=0, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=2, ind=0, asz=0, abs=27, disp=32 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=15, asz=0, abs=27, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=14, asz=0, abs=28, disp=32 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=13, asz=0, abs=28, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=12, asz=0, abs=29, disp=32 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=11, asz=0, abs=29, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=10, asz=0, abs=30, disp=32 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=9, asz=0, abs=30, disp=32 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=8, asz=0, abs=31, disp=32 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=7, asz=0, abs=31, disp=32 } .L19801: { loop_mode disp %ctpr1, .L19801 ldqp,0 %r58, %r0, %g20 addd,1,sm %r0, _f16s,_lts0lo 0x30, %g22 ldqp,2 %r55, %r0, %g21 movaqp,0 area=0, ind=0, am=1, be=0, %g17 movaqp,1 area=0, ind=16, am=0, be=0, %g16 movaqp,2 area=0, ind=0, am=1, be=0, %g19 movaqp,3 area=0, ind=16, am=0, be=0, %g18 } { loop_mode ldb,0,sm %r58, %g22, %empty, mas=0x20 ldb,2,sm %r55, %g22, %empty, mas=0x20 movaqp,0 area=1, ind=0, am=1, be=0, %g24 movaqp,1 area=1, ind=16, am=0, be=0, %g23 movaqp,2 area=1, ind=0, am=1, be=0, %g26 movaqp,3 area=1, ind=16, am=0, be=0, %g25 } { loop_mode movaqp,0 area=2, ind=0, am=1, be=0, %g27 movaqp,1 area=2, ind=16, am=0, be=0, %g22 movaqp,2 area=2, ind=0, am=1, be=0, %g29 movaqp,3 area=2, ind=16, am=0, be=0, %g28 } { loop_mode movaqp,0 area=3, ind=0, am=1, be=0, %g31 movaqp,1 area=3, ind=16, am=0, be=0, %g30 movaqp,2 area=3, ind=0, am=1, be=0, %b[38] movaqp,3 area=3, ind=16, am=0, be=0, %b[37] } { loop_mode movaqp,0 area=4, ind=0, am=1, be=0, %b[40] movaqp,1 area=4, ind=16, am=0, be=0, %b[39] movaqp,2 area=4, ind=0, am=1, be=0, %b[42] movaqp,3 area=4, ind=16, am=0, be=0, %b[41] } { loop_mode qpshufb,0 %g19, %g17, %r6, %b[47] qpshufb,1 %g18, %g16, %r6, %b[48] qpshufb,3 %g18, %g16, %r23, %g16 qpshufb,4 %g19, %g17, %r23, %g17 movaqp,0 area=5, ind=0, am=1, be=0, %b[44] movaqp,1 area=5, ind=16, am=0, be=0, %b[43] movaqp,2 area=5, ind=0, am=1, be=0, %b[46] movaqp,3 area=5, ind=16, am=0, be=0, %b[45] } { loop_mode qpshufb,0 %g26, %g24, %r6, %b[51] qpshufb,1 %g25, %g23, %r6, %b[52] qpshufb,3 %g25, %g23, %r23, %g23 qpshufb,4 %g26, %g24, %r23, %g24 movaqp,0 area=6, ind=0, am=1, be=0, %g19 movaqp,1 area=6, ind=16, am=0, be=0, %g18 movaqp,2 area=6, ind=0, am=1, be=0, %b[50] movaqp,3 area=6, ind=16, am=0, be=0, %b[49] } { loop_mode qpshufb,0 %g29, %g27, %r6, %b[55] qpshufb,1 %g28, %g22, %r6, %b[56] qpshufb,3 %g28, %g22, %r23, %g22 qpshufb,4 %g29, %g27, %r23, %g27 movaqp,0 area=7, ind=0, am=1, be=0, %g26 movaqp,1 area=7, ind=16, am=0, be=0, %g25 movaqp,2 area=7, ind=0, am=1, be=0, %b[54] movaqp,3 area=7, ind=16, am=0, be=0, %b[53] } { loop_mode qpshufb,0 %b[38], %g31, %r6, %b[59] qpshufb,1 %b[37], %g30, %r6, %b[60] qpshufb,3 %b[37], %g30, %r23, %g30 qpshufb,4 %b[38], %g31, %r23, %g31 movaqp,0 area=8, ind=0, am=1, be=0, %g29 movaqp,1 area=8, ind=16, am=0, be=0, %g28 movaqp,2 area=8, ind=0, am=1, be=0, %b[58] movaqp,3 area=8, ind=16, am=0, be=0, %b[57] } { loop_mode qpshufb,0 %b[42], %b[40], %r6, %b[63] qpshufb,1 %b[41], %b[39], %r6, %b[64] qpshufb,3 %b[41], %b[39], %r23, %b[39] qpshufb,4 %b[42], %b[40], %r23, %b[40] movaqp,0 area=9, ind=0, am=1, be=0, %b[38] movaqp,1 area=9, ind=16, am=0, be=0, %b[37] movaqp,2 area=9, ind=0, am=1, be=0, %b[62] movaqp,3 area=9, ind=16, am=0, be=0, %b[61] } { loop_mode qpshufb,0 %b[46], %b[44], %r6, %b[67] qpshufb,1 %b[45], %b[43], %r6, %b[68] qpshufb,3 %b[45], %b[43], %r23, %b[43] qpshufb,4 %b[46], %b[44], %r23, %b[44] movaqp,0 area=10, ind=0, am=1, be=0, %b[42] movaqp,1 area=10, ind=16, am=0, be=0, %b[41] movaqp,2 area=10, ind=0, am=1, be=0, %b[66] movaqp,3 area=10, ind=16, am=0, be=0, %b[65] } { loop_mode qpshufb,0 %b[50], %g19, %r6, %b[71] qpshufb,1 %b[49], %g18, %r6, %b[72] qpshufb,3 %b[49], %g18, %r23, %g18 qpshufb,4 %b[50], %g19, %r23, %g19 movaqp,0 area=11, ind=0, am=1, be=0, %b[46] movaqp,1 area=11, ind=16, am=0, be=0, %b[45] movaqp,2 area=11, ind=0, am=1, be=0, %b[70] movaqp,3 area=11, ind=16, am=0, be=0, %b[69] } { loop_mode qpshufb,0 %b[54], %g26, %r6, %b[75] qpshufb,1 %b[53], %g25, %r6, %b[76] qpshufb,3 %b[53], %g25, %r23, %g25 qpshufb,4 %b[54], %g26, %r23, %g26 movaqp,0 area=12, ind=0, am=1, be=0, %b[50] movaqp,1 area=12, ind=16, am=0, be=0, %b[49] movaqp,2 area=12, ind=0, am=1, be=0, %b[74] movaqp,3 area=12, ind=16, am=0, be=0, %b[73] } { loop_mode qpshufb,0 %b[58], %g29, %r6, %b[79] qpshufb,1 %b[57], %g28, %r6, %b[80] qpshufb,3 %b[57], %g28, %r23, %g28 qpshufb,4 %b[58], %g29, %r23, %g29 movaqp,0 area=13, ind=0, am=1, be=0, %b[54] movaqp,1 area=13, ind=16, am=0, be=0, %b[53] movaqp,2 area=13, ind=0, am=1, be=0, %b[78] movaqp,3 area=13, ind=16, am=0, be=0, %b[77] } { loop_mode qpshufb,0 %b[62], %b[38], %r6, %b[83] qpshufb,1 %b[61], %b[37], %r6, %b[84] qpshufb,3 %b[61], %b[37], %r23, %b[37] qpshufb,4 %b[62], %b[38], %r23, %b[38] movaqp,0 area=14, ind=0, am=1, be=0, %b[58] movaqp,1 area=14, ind=16, am=0, be=0, %b[57] movaqp,2 area=14, ind=0, am=1, be=0, %b[82] movaqp,3 area=14, ind=16, am=0, be=0, %b[81] } { loop_mode qpshufb,0 %b[66], %b[42], %r6, %b[87] qpshufb,1 %b[65], %b[41], %r6, %b[88] qpshufb,3 %b[65], %b[41], %r23, %b[41] qpshufb,4 %b[66], %b[42], %r23, %b[42] movaqp,0 area=15, ind=0, am=1, be=0, %b[62] movaqp,1 area=15, ind=16, am=0, be=0, %b[61] movaqp,2 area=15, ind=0, am=1, be=0, %b[86] movaqp,3 area=15, ind=16, am=0, be=0, %b[85] } { loop_mode qpshufb,0 %b[70], %b[46], %r6, %b[91] qpshufb,1 %b[69], %b[45], %r6, %b[92] qpshufb,3 %b[69], %b[45], %r23, %b[45] qpshufb,4 %b[70], %b[46], %r23, %b[46] movaqp,0 area=16, ind=0, am=1, be=0, %b[66] movaqp,1 area=16, ind=16, am=0, be=0, %b[65] movaqp,2 area=16, ind=0, am=1, be=0, %b[90] movaqp,3 area=16, ind=16, am=0, be=0, %b[89] } { loop_mode qpshufb,0 %b[50], %b[50], %r24, %b[95] qpshufb,1 %b[49], %b[49], %r24, %b[96] qpfmul_hsubs,2 %b[50], %b[47], %r25, %b[50] qpshufb,3 %b[74], %b[74], %r24, %b[97] qpshufb,4 %b[73], %b[73], %r24, %b[98] qpfmul_hsubs,5 %b[49], %b[51], %r25, %b[49] movaqp,0 area=17, ind=0, am=1, be=0, %b[70] movaqp,1 area=17, ind=16, am=0, be=0, %b[69] movaqp,2 area=17, ind=0, am=1, be=0, %b[94] movaqp,3 area=17, ind=16, am=0, be=0, %b[93] } { loop_mode qpshufb,0 %b[54], %b[54], %r24, %b[103] qpshufb,1 %b[53], %b[53], %r24, %b[104] qpfmul_hsubs,2 %b[74], %b[55], %r25, %b[74] qpshufb,3 %b[78], %b[78], %r24, %b[105] qpshufb,4 %b[77], %b[77], %r24, %b[106] qpfmul_hsubs,5 %b[73], %b[59], %r25, %b[73] movaqp,0 area=18, ind=0, am=1, be=0, %b[100] movaqp,1 area=18, ind=16, am=0, be=0, %b[99] movaqp,2 area=18, ind=0, am=1, be=0, %b[102] movaqp,3 area=18, ind=16, am=0, be=0, %b[101] } { loop_mode qpshufb,0 %b[58], %b[58], %r24, %b[111] qpshufb,1 %b[57], %b[57], %r24, %b[112] qpfmul_hsubs,2 %b[54], %b[63], %r25, %b[54] qpshufb,3 %b[82], %b[82], %r24, %b[113] qpshufb,4 %b[81], %b[81], %r24, %b[114] qpfmul_hsubs,5 %b[53], %b[67], %r25, %b[53] movaqp,0 area=19, ind=0, am=1, be=0, %b[108] movaqp,1 area=19, ind=16, am=0, be=0, %b[107] movaqp,2 area=19, ind=0, am=1, be=0, %b[110] movaqp,3 area=19, ind=16, am=0, be=0, %b[109] } { loop_mode qpfmul_hadds,0 %b[96], %b[51], %r25, %b[51] qpfmul_hsubs,1 %b[78], %b[71], %r25, %b[78] qpfmul_hsubs,2 %b[77], %b[75], %r25, %b[77] qpfmul_hadds,3 %b[97], %b[55], %r25, %b[55] qpfmul_hadds,4 %b[98], %b[59], %r25, %b[59] qpfmul_hsubs,5 %b[58], %b[79], %r25, %b[58] movaqp,0 area=20, ind=0, am=1, be=0, %b[116] movaqp,1 area=20, ind=16, am=0, be=0, %b[115] movaqp,2 area=20, ind=0, am=1, be=0, %b[118] movaqp,3 area=20, ind=16, am=0, be=0, %b[117] } { loop_mode qpfmul_hadds,0 %b[95], %b[47], %r25, %b[47] qpfmul_hsubs,1 %b[57], %b[83], %r25, %b[57] qpfmul_hsubs,2 %b[82], %b[87], %r25, %b[82] qpfmul_hadds,3 %b[105], %b[71], %r25, %b[71] qpfmul_hadds,4 %b[106], %b[75], %r25, %b[75] qpfmul_hsubs,5 %b[81], %b[91], %r25, %b[81] movaqp,0 area=21, ind=0, am=1, be=0, %b[96] movaqp,1 area=21, ind=16, am=0, be=0, %b[95] movaqp,2 area=21, ind=0, am=1, be=0, %b[98] movaqp,3 area=21, ind=16, am=0, be=0, %b[97] } { loop_mode qpfmul_hadds,0 %b[103], %b[63], %r25, %b[63] qpfmul_hadds,1 %b[112], %b[83], %r25, %b[83] qpfmul_hadds,2 %b[104], %b[67], %r25, %b[67] qpfmul_hadds,3 %b[114], %b[91], %r25, %b[91] qpshufb,4 %b[62], %b[62], %r24, %b[103] qpfmul_hadds,5 %b[113], %b[87], %r25, %b[87] movaqp,0 area=22, ind=0, am=1, be=0, %b[105] movaqp,1 area=22, ind=16, am=0, be=0, %b[104] movaqp,2 area=22, ind=0, am=1, be=0, %b[112] movaqp,3 area=22, ind=16, am=0, be=0, %b[106] } { loop_mode qpshufb,0 %b[100], %b[100], %r24, %b[113] qpshufb,1 %b[99], %b[99], %r24, %b[114] qpfmul_hsubs,2 %b[99], %b[52], %r25, %b[99] qpshufb,3 %b[102], %b[102], %r24, %b[119] qpshufb,4 %b[101], %b[101], %r24, %b[120] qpfmul_hadds,5 %b[111], %b[79], %r25, %b[79] movaqp,0 area=23, ind=0, am=1, be=0, %b[121] movaqp,1 area=23, ind=16, am=0, be=0, %b[111] movaqp,2 area=23, ind=0, am=1, be=0, %b[123] movaqp,3 area=23, ind=16, am=0, be=0, %b[122] } { loop_mode qpshufb,0 %b[108], %b[108], %r24, %b[124] qpshufb,1 %b[107], %b[107], %r24, %b[125] qpfmul_hsubs,2 %b[100], %b[48], %r25, %b[100] qpshufb,3 %b[110], %b[110], %r24, %b[126] qpshufb,4 %b[109], %b[109], %r24, %b[127] qpfmul_hsubs,5 %b[101], %b[60], %r25, %b[101] movaqp,0 area=24, ind=0, am=1, be=0, %b[35] movaqp,1 area=24, ind=16, am=0, be=0, %b[36] movaqp,2 area=24, ind=0, am=1, be=0, %b[33] movaqp,3 area=24, ind=16, am=0, be=0, %b[34] } { loop_mode qpshufb,0 %b[116], %b[116], %r24, %b[14] qpshufb,1 %b[115], %b[115], %r24, %b[13] qpfmul_hsubs,2 %b[102], %b[56], %r25, %b[102] qpshufb,3 %b[118], %b[118], %r24, %b[12] qpshufb,4 %b[117], %b[117], %r24, %b[11] qpfmul_hsubs,5 %b[107], %b[68], %r25, %b[107] movaqp,0 area=25, ind=0, am=1, be=0, %b[31] movaqp,1 area=25, ind=16, am=0, be=0, %b[32] movaqp,2 area=25, ind=0, am=1, be=0, %b[29] movaqp,3 area=25, ind=16, am=0, be=0, %b[30] } { loop_mode qpfmul_hsubs,0 %b[108], %b[64], %r25, %b[108] qpfmul_hsubs,1 %b[110], %b[72], %r25, %b[110] qpfmul_hsubs,2 %b[109], %b[76], %r25, %b[109] qpfmul_hadds,3 %b[119], %b[56], %r25, %b[56] qpfmul_hadds,4 %b[120], %b[60], %r25, %b[60] qpfmul_hsubs,5 %b[115], %b[84], %r25, %b[115] movaqp,0 area=26, ind=0, am=1, be=0, %b[27] movaqp,1 area=26, ind=16, am=0, be=0, %b[28] movaqp,2 area=26, ind=0, am=1, be=0, %b[25] movaqp,3 area=26, ind=16, am=0, be=0, %b[26] } { loop_mode qpfmul_hadds,0 %b[113], %b[48], %r25, %b[48] qpfmul_hadds,1 %b[114], %b[52], %r25, %b[52] qpfmul_hsubs,2 %b[116], %b[80], %r25, %b[113] qpfmul_hadds,3 %b[127], %b[76], %r25, %b[76] qpfmul_hsubs,4 %b[118], %b[88], %r25, %b[114] qpfmul_hsubs,5 %b[117], %b[92], %r25, %b[116] movaqp,0 area=28, ind=0, am=1, be=0, %b[22] movaqp,1 area=27, ind=0, am=1, be=0, %b[24] movaqp,2 area=28, ind=0, am=1, be=0, %b[21] movaqp,3 area=27, ind=0, am=1, be=0, %b[23] } { loop_mode qpfmul_hadds,0 %b[124], %b[64], %r25, %b[64] qpfmul_hadds,1 %b[125], %b[68], %r25, %b[68] qpfmul_hadds,2 %b[126], %b[72], %r25, %b[72] qpfmul_hadds,3 %b[11], %b[92], %r25, %b[11] qpshufb,4 %b[61], %b[61], %r24, %b[88] qpfmul_hadds,5 %b[12], %b[88], %r25, %b[12] movaqp,0 area=30, ind=0, am=1, be=0, %b[18] movaqp,1 area=29, ind=0, am=1, be=0, %b[20] movaqp,2 area=30, ind=0, am=1, be=0, %b[17] movaqp,3 area=29, ind=0, am=1, be=0, %b[19] } { loop_mode qpfmul_hadds,0 %b[13], %b[84], %r25, %b[13] qpshufb,1 %b[86], %b[86], %r24, %b[80] qpfmul_hadds,2 %b[14], %b[80], %r25, %b[14] qpshufb,3 %b[85], %b[85], %r24, %b[84] qpshufb,4 %b[66], %b[66], %r24, %b[92] qpfmul_hsubs,5 %b[61], %g23, %r25, %b[61] movaqp,1 area=31, ind=0, am=1, be=0, %b[16] movaqp,3 area=31, ind=0, am=1, be=0, %b[15] } { loop_mode qpshufb,0 %b[65], %b[65], %r24, %b[117] qpshufb,1 %b[90], %b[90], %r24, %b[118] qpfmul_hsubs,2 %b[62], %g16, %r25, %b[62] qpshufb,3 %b[89], %b[89], %r24, %b[119] qpshufb,4 %b[70], %b[70], %r24, %b[120] qpfmul_hsubs,5 %b[86], %g22, %r25, %b[86] } { loop_mode qpshufb,0 %b[69], %b[69], %r24, %b[124] qpshufb,1 %b[94], %b[94], %r24, %b[125] qpfmul_hsubs,2 %b[85], %g30, %r25, %b[85] qpshufb,3 %b[93], %b[93], %r24, %b[126] qpfmul_hsubs,4 %b[66], %b[39], %r25, %b[66] qpfmul_hsubs,5 %b[65], %b[43], %r25, %b[65] } { loop_mode qpfmul_hsubs,0 %b[90], %g18, %r25, %b[90] qpfmul_hsubs,1 %b[89], %g25, %r25, %b[89] qpfmul_hadds,2 %b[103], %g16, %r25, %g16 qpfmul_hadds,3 %b[88], %g23, %r25, %g23 qpfmul_hsubs,4 %b[70], %g28, %r25, %b[70] qpfmul_hsubs,5 %b[69], %b[37], %r25, %b[69] } { loop_mode qpfmul_hsubs,0 %b[94], %b[41], %r25, %b[88] qpfmul_hsubs,1 %b[93], %b[45], %r25, %b[93] qpfmul_hadds,2 %b[80], %g22, %r25, %g22 qpfmul_hadds,3 %b[84], %g30, %r25, %g30 qpfmul_hadds,4 %b[92], %b[39], %r25, %b[39] qpfmul_hadds,5 %b[119], %g25, %r25, %g25 } { loop_mode qpfmul_hadds,0 %b[117], %b[43], %r25, %b[43] qpfmul_hadds,1 %b[118], %g18, %r25, %g18 qpfmul_hadds,2 %b[124], %b[37], %r25, %b[37] qpfmul_hadds,3 %b[126], %b[45], %r25, %b[45] qppermb,4 %b[51], %b[49], %r7, %b[49] qpfmul_hadds,5 %b[120], %g28, %r25, %g28 } { loop_mode qppermb,1 %b[59], %b[73], %r7, %b[51] qpfmul_hadds,2 %b[125], %b[41], %r25, %b[41] qppermb,3 %b[47], %b[50], %r7, %b[47] qppermb,4 %b[55], %b[74], %r7, %b[50] } { loop_mode qppermb,0 %b[71], %b[78], %r7, %b[55] qppermb,1 %b[75], %b[77], %r7, %b[59] qppermb,3 %b[63], %b[54], %r7, %b[54] qppermb,4 %b[67], %b[53], %r7, %b[53] } { loop_mode nop 2 qppermb,0 %b[79], %b[58], %r7, %b[58] qppermb,1 %b[83], %b[57], %r7, %b[57] qppermb,3 %b[87], %b[82], %r7, %b[63] } { loop_mode qppermb,4 %b[91], %b[81], %r7, %b[67] } { loop_mode qppermb,0 %b[56], %b[102], %r7, %b[56] qppermb,1 %b[60], %b[101], %r7, %b[60] qppermb,3 %b[48], %b[100], %r7, %b[48] qppermb,4 %b[52], %b[99], %r7, %b[52] } { loop_mode qppermb,0 %b[68], %b[107], %r7, %b[68] qppermb,1 %b[76], %b[109], %r7, %b[71] qpfsubs,2 %b[51], %b[60], %b[73] qppermb,3 %b[64], %b[108], %r7, %b[64] qppermb,4 %b[72], %b[110], %r7, %b[72] qpfsubs,5 %b[49], %b[52], %b[74] } { loop_mode qppermb,0 %b[12], %b[114], %r7, %b[12] qppermb,1 %b[11], %b[116], %r7, %b[11] qpfsubs,2 %b[50], %b[56], %b[75] qppermb,3 %b[14], %b[113], %r7, %b[14] qppermb,4 %b[13], %b[115], %r7, %b[13] qpfsubs,5 %b[47], %b[48], %b[76] } { loop_mode qpfsubs,0 %b[59], %b[71], %b[77] qpfsubs,1 %b[53], %b[68], %b[78] qpfsubs,2 %b[63], %b[12], %b[79] qpfsubs,3 %b[55], %b[72], %b[81] qpfsubs,4 %b[58], %b[14], %b[82] qpfsubs,5 %b[54], %b[64], %b[80] } { loop_mode qppermb,0 %g16, %b[62], %r7, %g16 qppermb,1 %g23, %b[61], %r7, %g23 qpfsubs,2 %b[67], %b[11], %b[83] qppermb,3 %g22, %b[86], %r7, %g22 qppermb,4 %g30, %b[85], %r7, %g30 qpfsubs,5 %b[57], %b[13], %b[84] } { loop_mode qpfadds,0 %b[51], %b[60], %b[51] qpfadds,1 %b[47], %b[48], %b[47] qpfadds,2 %b[49], %b[52], %b[48] qpfadds,3 %b[50], %b[56], %b[49] qpfadds,4 %b[59], %b[71], %b[50] qpfadds,5 %b[54], %b[64], %b[52] } { loop_mode qppermb,0 %b[39], %b[66], %r7, %b[39] qppermb,1 %b[43], %b[65], %r7, %b[43] qpfadds,2 %b[53], %b[68], %b[53] qppermb,3 %g18, %b[90], %r7, %g18 qppermb,4 %g25, %b[89], %r7, %g25 qpfadds,5 %b[55], %b[72], %b[54] } { loop_mode qpfadds,0 %b[67], %b[11], %b[11] qpfadds,1 %g24, %g23, %b[55] qpfsubs,2 %g17, %g16, %b[56] qpfadds,3 %b[58], %b[14], %b[14] qpfadds,4 %b[57], %b[13], %b[13] qpfadds,5 %b[63], %b[12], %b[12] } { loop_mode qppermb,0 %b[37], %b[69], %r7, %b[37] qppermb,1 %g28, %b[70], %r7, %g28 qpfsubs,2 %g24, %g23, %g23 qppermb,3 %b[45], %b[93], %r7, %b[45] qppermb,4 %b[41], %b[88], %r7, %b[41] qpfadds,5 %g17, %g16, %g16 } { loop_mode qpfsubs,0 %g31, %g30, %g17 qpfadds,1 %g27, %g22, %g24 qpfadds,2 %g31, %g30, %g30 qpfsubs,3 %g27, %g22, %g22 qpfsubs,4 %g26, %g25, %g27 qpfadds,5 %g26, %g25, %g25 } { loop_mode qpfsubs,0 %b[44], %b[43], %g26 qpfadds,1 %b[40], %b[39], %g31 qpfadds,2 %b[44], %b[43], %b[43] qpfsubs,3 %b[40], %b[39], %b[39] qpfsubs,4 %g19, %g18, %b[40] qpfadds,5 %g19, %g18, %g18 } { loop_mode qpfadds,0 %b[38], %b[37], %g19 qpfsubs,1 %g29, %g28, %b[44] qpfsubs,2 %b[38], %b[37], %b[37] qpfsubs,3 %b[46], %b[45], %b[45] qpfadds,4 %b[42], %b[41], %b[46] qpfadds,5 %b[46], %b[45], %b[38] } { loop_mode qpfadds,0 %b[55], %b[48], %b[41] qpfsubs,1 %b[55], %b[48], %b[42] qpfadds,2 %g29, %g28, %g28 qpfadds,3 %g16, %b[47], %b[48] qpfsubs,4 %g16, %b[47], %g16 qpfsubs,5 %b[42], %b[41], %g29 } { loop_mode qpfadds,0 %g30, %b[51], %b[47] qpfadds,1 %g24, %b[49], %b[51] qpfsubs,2 %g30, %b[51], %g30 qpfsubs,3 %g25, %b[50], %g25 qpfadds,5 %g25, %b[50], %b[55] } { loop_mode qpfsubs,0 %g24, %b[49], %g24 qpfadds,1 %b[43], %b[53], %b[49] qpfsubs,2 %g31, %b[52], %b[50] qpfsubs,3 %g18, %b[54], %g18 qpfadds,5 %g18, %b[54], %b[57] } { loop_mode qpfsubs,0 %b[43], %b[53], %b[43] qpfadds,1 %g31, %b[52], %g31 qpfadds,2 %g19, %b[13], %b[52] qpfsubs,3 %b[38], %b[11], %b[53] qpshufb,4 %b[75], %b[75], %r24, %b[54] qpfadds,5 %b[46], %b[12], %b[58] } { loop_mode qpfadds,0 %g28, %b[14], %b[13] qpfsubs,1 %g28, %b[14], %g28 qpfsubs,2 %g19, %b[13], %g19 qpfsubs,3 %b[46], %b[12], %b[12] qpshufb,4 %b[73], %b[73], %r24, %b[59] qpfadds,5 %b[38], %b[11], %b[11] } { loop_mode qpshufb,4 %b[74], %b[74], %r24, %b[14] } { loop_mode qpshufb,4 %b[78], %b[78], %r24, %b[38] } { loop_mode qpshufb,0 %b[77], %b[77], %r24, %b[46] qpshufb,1 %b[83], %b[83], %r24, %b[60] qpshufb,3 %b[76], %b[76], %r24, %b[61] qpshufb,4 %b[80], %b[80], %r24, %b[62] } { loop_mode qpshufb,0 %b[81], %b[81], %r24, %b[63] qpshufb,1 %b[79], %b[79], %r24, %b[64] qpshufb,3 %b[82], %b[82], %r24, %b[65] qpshufb,4 %b[84], %b[84], %r24, %b[66] } { loop_mode qpxor,0 %b[54], %r22, %b[54] qpxor,1 %b[59], %r22, %b[59] qpxor,3 %b[14], %r22, %b[14] qpxor,4 %b[38], %r22, %b[38] } { loop_mode qpxor,0 %b[46], %r22, %b[46] qpxor,1 %b[60], %r22, %b[60] qpfadds,2 %g17, %b[59], %b[67] qpxor,3 %b[61], %r22, %b[61] qpxor,4 %b[62], %r22, %b[62] qpfadds,5 %g26, %b[38], %b[68] } { loop_mode qpxor,0 %b[63], %r22, %b[63] qpxor,1 %b[64], %r22, %b[64] qpfsubs,2 %g17, %b[59], %g17 qpxor,3 %b[65], %r22, %b[65] qpxor,4 %b[66], %r22, %b[66] qpfsubs,5 %g23, %b[14], %b[59] } { loop_mode qpfadds,0 %g27, %b[46], %b[69] qpfsubs,1 %g22, %b[54], %b[70] qpfadds,2 %g22, %b[54], %g22 qpfadds,3 %g23, %b[14], %g23 qpfsubs,4 %g26, %b[38], %g26 qpfsubs,5 %b[56], %b[61], %b[14] } { loop_mode qpfsubs,0 %g27, %b[46], %g27 qpfsubs,1 %b[45], %b[60], %b[38] qpfadds,2 %g29, %b[64], %b[46] qpfadds,3 %b[56], %b[61], %b[54] qpfsubs,4 %b[39], %b[62], %b[56] qpfadds,5 %b[39], %b[62], %b[39] } { loop_mode qpfadds,0 %b[45], %b[60], %b[45] qpfadds,1 %b[40], %b[63], %b[60] qpfsubs,2 %g29, %b[64], %g29 qpfadds,3 %b[37], %b[66], %b[37] qpfsubs,4 %b[44], %b[65], %b[62] qpfsubs,5 %b[37], %b[66], %b[61] } { loop_mode nop 1 qpfsubs,2 %b[40], %b[63], %b[40] qpfadds,5 %b[44], %b[65], %b[44] } { loop_mode qpshufb,0 %b[95], %b[95], %r24, %b[63] qpshufb,1 %g20, %g20, %r24, %b[64] qpshufb,3 %b[96], %b[96], %r24, %b[65] qpshufb,4 %b[104], %b[104], %r24, %b[66] } { loop_mode qpshufb,0 %b[112], %b[112], %r24, %b[71] qpshufb,1 %b[105], %b[105], %r24, %b[72] qpshufb,3 %b[106], %b[106], %r24, %b[73] qpshufb,4 %b[122], %b[122], %r24, %b[74] } { loop_mode qpshufb,0 %b[35], %b[35], %r24, %b[75] qpshufb,1 %b[123], %b[123], %r24, %b[76] qpshufb,3 %b[36], %b[36], %r24, %b[77] qpshufb,4 %b[32], %b[32], %r24, %b[78] } { loop_mode qpshufb,0 %b[29], %b[29], %r24, %b[79] qpshufb,1 %b[31], %b[31], %r24, %b[80] qpshufb,3 %b[30], %b[30], %r24, %b[81] qpshufb,4 %b[26], %b[26], %r24, %b[82] } { loop_mode qpshufb,0 %b[25], %b[25], %r24, %b[83] qpshufb,1 %b[24], %b[24], %r24, %b[84] qpshufb,3 %b[21], %b[21], %r24, %b[85] qpshufb,4 %b[22], %b[22], %r24, %b[86] } { loop_mode qpshufb,0 %b[18], %b[18], %r24, %b[87] qpshufb,1 %b[15], %b[15], %r24, %b[88] qpshufb,3 %b[19], %b[19], %r24, %b[89] qpshufb,4 %b[16], %b[16], %r24, %b[90] } { loop_mode qpshufb,0 %b[51], %b[48], %r6, %b[91] qpshufb,1 %b[47], %b[41], %r6, %b[92] qpshufb,3 %g24, %g16, %r6, %b[93] qpshufb,4 %g30, %b[42], %r6, %b[94] } { loop_mode qpshufb,0 %b[57], %g31, %r6, %b[99] qpshufb,1 %b[55], %b[49], %r6, %b[100] qpfmul_hadds,2 %b[65], %b[91], %r25, %b[65] qpshufb,3 %g18, %b[50], %r6, %b[101] qpshufb,4 %g25, %b[43], %r6, %b[102] qpfmul_hadds,5 %b[75], %b[93], %r25, %b[75] } { loop_mode qpshufb,0 %b[58], %b[13], %r6, %b[103] qpshufb,1 %b[11], %b[52], %r6, %b[107] qpfmul_hadds,2 %b[72], %b[92], %r25, %b[72] qpshufb,3 %b[12], %g28, %r6, %b[108] qpshufb,4 %b[53], %g19, %r6, %b[109] qpfmul_hadds,5 %b[80], %b[94], %r25, %b[80] } { loop_mode qpshufb,0 %g17, %b[59], %r6, %b[110] qpshufb,1 %b[67], %g23, %r6, %b[113] qpfmul_hsubs,2 %b[96], %b[91], %r25, %b[91] qpshufb,3 %g27, %g26, %r6, %b[114] qpshufb,4 %b[69], %b[68], %r6, %b[115] qpfmul_hsubs,5 %b[35], %b[93], %r25, %b[35] } { loop_mode qpshufb,0 %b[70], %b[14], %r6, %b[93] qpshufb,1 %g22, %b[54], %r6, %b[96] qpfmul_hsubs,2 %b[31], %b[94], %r25, %b[31] qpshufb,3 %b[60], %b[39], %r6, %b[116] qpshufb,4 %b[45], %b[37], %r6, %b[117] qpfmul_hsubs,5 %b[105], %b[92], %r25, %b[92] } { loop_mode qpshufb,0 %b[40], %b[56], %r6, %b[94] qpshufb,1 %g29, %b[62], %r6, %b[105] qpfmul_hsubs,2 %b[32], %b[102], %r25, %b[32] qpshufb,3 %b[38], %b[61], %r6, %b[118] qpshufb,4 %b[46], %b[44], %r6, %b[119] qpfmul_hadds,5 %b[78], %b[102], %r25, %b[78] } { loop_mode qpfmul_hsubs,0 %b[95], %b[99], %r25, %b[95] qpfmul_hsubs,1 %b[36], %b[101], %r25, %b[36] qpfmul_hsubs,2 %b[104], %b[100], %r25, %b[102] qpfmul_hadds,3 %b[63], %b[99], %r25, %b[63] qpfmul_hadds,4 %b[77], %b[101], %r25, %b[77] qpfmul_hadds,5 %b[66], %b[100], %r25, %b[66] } { loop_mode qpfmul_hsubs,0 %b[18], %b[108], %r25, %b[18] qpfmul_hsubs,1 %b[22], %b[107], %r25, %b[22] qpfmul_hsubs,2 %b[16], %b[109], %r25, %b[16] qpfmul_hadds,3 %b[87], %b[108], %r25, %b[87] qpfmul_hadds,4 %b[86], %b[107], %r25, %b[86] qpfmul_hadds,5 %b[90], %b[109], %r25, %b[90] } { loop_mode qpfmul_hsubs,0 %b[24], %b[103], %r25, %b[24] qpfmul_hadds,1 %b[84], %b[103], %r25, %b[84] qpfmul_hsubs,2 %b[25], %b[113], %r25, %b[25] qpfmul_hadds,3 %b[83], %b[113], %r25, %b[83] qpfmul_hsubs,4 %b[26], %b[115], %r25, %b[26] qpfmul_hadds,5 %b[82], %b[115], %r25, %b[82] } { loop_mode qpfmul_hsubs,0 %b[29], %b[96], %r25, %b[29] qpfmul_hsubs,1 %b[123], %b[110], %r25, %b[99] qpfmul_hadds,2 %b[79], %b[96], %r25, %b[79] qpfmul_hadds,3 %b[76], %b[110], %r25, %b[76] qpfmul_hsubs,4 %b[122], %b[114], %r25, %b[96] qpfmul_hadds,5 %b[74], %b[114], %r25, %b[74] } { loop_mode qpfmul_hsubs,0 %b[112], %b[93], %r25, %b[100] qpfmul_hadds,1 %b[71], %b[93], %r25, %b[71] qpfmul_hsubs,2 %b[30], %b[116], %r25, %b[30] qpfmul_hadds,3 %b[81], %b[116], %r25, %b[81] qpfmul_hsubs,4 %g20, %b[117], %r25, %g20 qpfmul_hadds,5 %b[64], %b[117], %r25, %b[64] } { loop_mode qpfmul_hsubs,0 %b[106], %b[94], %r25, %b[93] qpfmul_hadds,1 %b[73], %b[94], %r25, %b[73] qpfmul_hsubs,2 %b[15], %b[119], %r25, %b[15] qpfmul_hsubs,3 %b[19], %b[118], %r25, %b[19] qpfmul_hadds,4 %b[88], %b[119], %r25, %b[88] qpfmul_hadds,5 %b[89], %b[118], %r25, %b[89] } { loop_mode nop 5 qpfmul_hadds,0 %b[85], %b[105], %r25, %b[85] qpfmul_hsubs,2 %b[21], %b[105], %r25, %b[21] } { loop_mode qpshufb,1 %b[97], %b[97], %r24, %b[94] qpshufb,3 %g21, %g21, %r24, %b[101] qpshufb,4 %b[98], %b[98], %r24, %b[103] } { loop_mode qpshufb,0 %b[121], %b[121], %r24, %b[104] qpshufb,1 %b[111], %b[111], %r24, %b[105] qpshufb,3 %b[34], %b[34], %r24, %b[106] qpshufb,4 %b[33], %b[33], %r24, %b[107] } { loop_mode qpshufb,0 %b[27], %b[27], %r24, %b[108] qpshufb,1 %b[28], %b[28], %r24, %b[109] qpshufb,3 %b[23], %b[23], %r24, %b[110] qpshufb,4 %b[20], %b[20], %r24, %b[112] } { loop_mode qpshufb,0 %b[17], %b[17], %r24, %b[113] qpshufb,1 %b[47], %b[41], %r23, %b[41] qpshufb,3 %g30, %b[42], %r23, %g30 qpshufb,4 %b[55], %b[49], %r23, %b[42] } { loop_mode qpshufb,0 %g25, %b[43], %r23, %g25 qpshufb,1 %b[11], %b[52], %r23, %b[11] qpfmul_hsubs,2 %b[98], %b[41], %r25, %b[43] qpshufb,3 %b[53], %g19, %r23, %g19 qpshufb,4 %g17, %b[59], %r23, %g17 qpfmul_hsubs,5 %b[33], %g30, %r25, %b[33] } { loop_mode qpshufb,0 %b[67], %g23, %r23, %g23 qpshufb,1 %g27, %g26, %r23, %g26 qpfmul_hadds,2 %b[106], %g25, %r25, %b[47] qpshufb,3 %b[69], %b[68], %r23, %g27 qpshufb,4 %b[38], %b[61], %r23, %b[38] qpfmul_hsubs,5 %b[17], %g19, %r25, %b[17] } { loop_mode qpshufb,0 %b[45], %b[37], %r23, %b[37] qpfmul_hadds,1 %b[103], %b[41], %r25, %b[41] qpfmul_hsubs,2 %b[34], %g25, %r25, %g25 qpfmul_hadds,3 %b[107], %g30, %r25, %g30 qpfmul_hsubs,4 %b[97], %b[42], %r25, %b[34] qpfmul_hadds,5 %b[94], %b[42], %r25, %b[42] } { loop_mode qpfmul_hadds,0 %b[110], %b[11], %r25, %b[45] qpfmul_hsubs,1 %b[23], %b[11], %r25, %b[11] qpfmul_hadds,2 %b[108], %g23, %r25, %b[49] qpfmul_hsubs,3 %b[121], %g17, %r25, %b[23] qpfmul_hadds,4 %b[104], %g17, %r25, %g17 qpfmul_hadds,5 %b[113], %g19, %r25, %g19 } { loop_mode qpfmul_hsubs,0 %b[111], %g26, %r25, %b[52] qpfmul_hadds,1 %b[105], %g26, %r25, %g26 qpfmul_hsubs,2 %b[27], %g23, %r25, %g23 qpfmul_hsubs,3 %b[28], %g27, %r25, %b[28] qpfmul_hadds,4 %b[109], %g27, %r25, %g27 qpfmul_hsubs,5 %b[20], %b[38], %r25, %b[20] } { loop_mode qpfmul_hadds,0 %b[101], %b[37], %r25, %b[27] qppermb,1 %b[75], %b[35], %r7, %b[35] qpfmul_hsubs,2 %g21, %b[37], %r25, %g21 qppermb,3 %b[80], %b[31], %r7, %b[31] qppermb,4 %b[65], %b[91], %r7, %b[38] qpfmul_hadds,5 %b[112], %b[38], %r25, %b[37] } { loop_mode qppermb,0 %b[72], %b[92], %r7, %b[53] qppermb,1 %b[63], %b[95], %r7, %b[55] qppermb,3 %b[66], %b[102], %r7, %b[59] qppermb,4 %b[78], %b[32], %r7, %b[32] } { loop_mode qppermb,0 %b[90], %b[16], %r7, %b[16] qppermb,1 %b[77], %b[36], %r7, %b[36] qppermb,3 %b[87], %b[18], %r7, %b[18] qppermb,4 %b[84], %b[24], %r7, %b[24] } { loop_mode qppermb,0 %b[86], %b[22], %r7, %b[22] } { loop_mode qpfsubs,1 %b[38], %b[53], %b[61] qpfsubs,3 %b[35], %b[31], %b[63] qpfadds,4 %b[35], %b[31], %b[31] } { loop_mode qpfsubs,0 %b[36], %b[32], %b[65] qpfadds,1 %b[38], %b[53], %b[38] qpfsubs,2 %b[55], %b[59], %b[35] qpfadds,3 %b[55], %b[59], %b[53] } { loop_mode qpfsubs,0 %b[24], %b[22], %b[59] qpfadds,1 %b[36], %b[32], %b[32] qpfsubs,2 %b[18], %b[16], %b[55] qpfadds,3 %b[18], %b[16], %b[16] } { loop_mode qpfadds,0 %b[24], %b[22], %b[22] qppermb,4 %b[83], %b[25], %r7, %b[18] } { loop_mode qppermb,4 %b[79], %b[29], %r7, %b[24] } { loop_mode qppermb,1 %b[76], %b[99], %r7, %b[25] qppermb,3 %b[71], %b[100], %r7, %b[29] qppermb,4 %b[74], %b[96], %r7, %b[36] qpfsubs,5 %b[24], %b[18], %b[66] } { loop_mode qppermb,0 %b[82], %b[26], %r7, %b[26] qppermb,1 %b[64], %g20, %r7, %g20 qppermb,3 %b[81], %b[30], %r7, %b[30] qppermb,4 %b[85], %b[21], %r7, %b[21] qpfadds,5 %b[24], %b[18], %b[18] } { loop_mode qppermb,0 %b[88], %b[15], %r7, %b[15] qppermb,1 %b[89], %b[19], %r7, %b[19] qppermb,3 %b[73], %b[93], %r7, %b[24] qpshufb,4 %b[51], %b[48], %r23, %b[48] } { loop_mode qpshufb,0 %g24, %g16, %r23, %g16 qpshufb,1 %g18, %b[50], %r23, %g18 qpfsubs,2 %b[15], %g20, %b[64] qpshufb,3 %b[57], %g31, %r23, %g24 qpshufb,4 %b[58], %b[13], %r23, %g31 qpfsubs,5 %b[24], %b[36], %b[51] } { loop_mode qpshufb,0 %b[12], %g28, %r23, %g28 qpshufb,1 %g22, %b[54], %r23, %g22 qpfsubs,2 %b[29], %b[25], %b[13] qpshufb,3 %b[70], %b[14], %r23, %b[12] qpshufb,4 %b[40], %b[56], %r23, %b[14] qpfadds,5 %b[29], %b[25], %b[25] } { loop_mode qpfsubs,0 %b[21], %b[19], %b[40] qpshufb,1 %b[60], %b[39], %r23, %b[39] qpfsubs,2 %b[30], %b[26], %b[29] qpshufb,3 %b[46], %b[44], %r23, %b[44] qppermb,4 %b[41], %b[43], %r7, %b[41] qpfadds,5 %b[30], %b[26], %b[26] } { loop_mode qppermb,0 %b[47], %g25, %r7, %g25 qpshufb,1 %g29, %b[62], %r23, %g29 qpfadds,2 %b[15], %g20, %g20 qppermb,3 %g30, %b[33], %r7, %g30 qppermb,4 %b[42], %b[34], %r7, %b[30] qpfadds,5 %b[24], %b[36], %b[15] } { loop_mode qpfadds,0 %b[21], %b[19], %b[17] qppermb,1 %b[45], %b[11], %r7, %b[11] qpfadds,2 %g18, %g25, %b[21] qppermb,3 %g17, %b[23], %r7, %g17 qppermb,4 %g19, %b[17], %r7, %g19 qpfsubs,5 %b[48], %b[41], %b[19] } { loop_mode qppermb,0 %b[49], %g23, %r7, %g23 qppermb,1 %g26, %b[52], %r7, %g26 qpfsubs,2 %g18, %g25, %g18 qppermb,3 %g27, %b[28], %r7, %g27 qppermb,4 %b[37], %b[20], %r7, %b[20] qpfadds,5 %b[48], %b[41], %b[23] } { loop_mode qpfadds,0 %g31, %b[11], %b[27] qppermb,1 %b[27], %g21, %r7, %g21 qpfsubs,2 %g31, %b[11], %g31 qpfsubs,3 %g24, %b[30], %g25 qpfsubs,4 %g16, %g30, %b[24] qpfadds,5 %g24, %b[30], %g24 } { loop_mode qpfadds,0 %g16, %g30, %g16 qpfsubs,1 %g22, %g23, %b[12] qpfadds,2 %b[14], %g26, %b[28] qpfadds,3 %b[12], %g17, %g30 qpfsubs,4 %b[12], %g17, %g17 qpfsubs,5 %b[39], %g27, %b[11] } { loop_mode qpfadds,0 %g28, %g19, %b[30] qpfsubs,1 %g28, %g19, %g19 qpfsubs,2 %b[14], %g26, %g26 qpfadds,3 %b[39], %g27, %g27 qpfsubs,4 %g29, %b[20], %g29 qpfadds,5 %g29, %b[20], %g28 } { loop_mode qpfadds,0 %b[44], %g21, %g23 qpfsubs,1 %b[44], %g21, %g21 qpfadds,2 %g22, %g23, %g22 qpfadds,3 %b[23], %b[38], %b[14] qpfsubs,4 %b[23], %b[38], %b[20] } { loop_mode qpfadds,0 %b[21], %b[32], %b[23] qpfsubs,1 %b[21], %b[32], %b[21] qpfadds,2 %b[27], %b[22], %b[32] qpfsubs,3 %g24, %b[53], %b[33] qpfadds,4 %g24, %b[53], %g24 } { loop_mode qpfadds,0 %g16, %b[31], %b[34] qpfsubs,1 %g16, %b[31], %g16 qpfsubs,2 %b[27], %b[22], %b[22] qpfsubs,3 %g30, %b[25], %b[27] qpfadds,4 %g30, %b[25], %g30 } { loop_mode qpfadds,0 %b[30], %b[16], %b[25] qpfsubs,1 %b[30], %b[16], %b[16] qpfsubs,2 %b[28], %b[15], %b[31] qpfadds,3 %g27, %b[26], %b[30] qpfsubs,4 %g27, %b[26], %g27 qpfsubs,5 %g28, %b[17], %b[26] } { loop_mode qpfadds,0 %g22, %b[18], %b[36] qpfsubs,1 %g22, %b[18], %g22 qpfadds,2 %g23, %g20, %b[18] qpfadds,3 %b[28], %b[15], %b[15] qpfadds,4 %g28, %b[17], %g28 stqp,5 %r18, %r0, %b[20] } { loop_mode qpfsubs,0 %g23, %g20, %g20 stqp,2 %r36, %r0, %b[21] stqp,5 %r2, %r0, %b[14] } { loop_mode stqp,2 %r16, %r0, %g16 stqp,5 %r30, %r0, %b[33] } { loop_mode qpshufb,1 %b[61], %b[61], %r24, %g16 stqp,2 %r27, %r0, %b[23] qpshufb,3 %b[63], %b[63], %r24, %g23 qpshufb,4 %b[35], %b[35], %r24, %b[14] stqp,5 %r29, %r0, %g24 } { loop_mode qpshufb,0 %b[55], %b[55], %r24, %g24 qpshufb,1 %b[65], %b[65], %r24, %b[17] stqp,2 %r51, %r0, %b[22] qpshufb,3 %b[59], %b[59], %r24, %b[20] qpshufb,4 %b[66], %b[66], %r24, %b[21] stqp,5 %r20, %r0, %b[34] } { loop_mode qpshufb,0 %b[13], %b[13], %r24, %b[13] qpshufb,1 %b[51], %b[51], %r24, %b[22] stqp,2 %r38, %r0, %b[32] qpshufb,3 %b[29], %b[29], %r24, %b[23] qpshufb,4 %b[64], %b[64], %r24, %b[28] stqp,5 %r21, %r0, %g30 } { loop_mode qpshufb,0 %b[40], %b[40], %r24, %g30 qpxor,1 %g16, %r22, %g16 stqp,2 %r13, %r0, %b[27] qpxor,3 %g23, %r22, %g23 qpxor,4 %b[14], %r22, %b[14] stqp,5 %r35, %r0, %g27 } { loop_mode qpxor,0 %b[17], %r22, %g27 qpxor,1 %g24, %r22, %g24 qpfadds,2 %b[19], %g16, %b[21] qpxor,3 %b[20], %r22, %b[17] qpxor,4 %b[21], %r22, %b[20] qpfadds,5 %b[24], %g23, %b[27] } { loop_mode qpxor,0 %b[13], %r22, %b[13] qpxor,1 %b[22], %r22, %b[22] qpfsubs,2 %b[19], %g16, %g16 qpxor,3 %b[23], %r22, %b[23] qpxor,4 %b[28], %r22, %b[28] qpfsubs,5 %b[24], %g23, %g23 } { loop_mode qpxor,0 %g30, %r22, %g30 qpfsubs,1 %g19, %g24, %b[14] qpfadds,2 %g19, %g24, %g19 qpfsubs,3 %g25, %b[14], %b[19] qpfadds,4 %g25, %b[14], %g25 qpfadds,5 %g31, %b[17], %b[24] } { loop_mode qpfsubs,0 %g18, %g27, %g24 qpfadds,1 %g18, %g27, %g18 qpfadds,2 %g17, %b[13], %b[17] qpfsubs,3 %g31, %b[17], %g27 qpfsubs,4 %b[12], %b[20], %g31 qpfadds,5 %b[12], %b[20], %b[12] } { loop_mode qpfsubs,0 %g17, %b[13], %g17 qpfadds,1 %g26, %b[22], %b[20] qpfsubs,2 %g26, %b[22], %g26 qpfsubs,3 %b[11], %b[23], %b[13] qpfadds,4 %b[11], %b[23], %b[11] qpfsubs,5 %g21, %b[28], %b[23] } { loop_mode qpfadds,0 %g21, %b[28], %g21 qpfsubs,1 %g29, %g30, %b[22] qpfadds,2 %g29, %g30, %g29 stqp,5 %r26, %r0, %b[30] } { loop_mode stqp,2 %r37, %r0, %b[31] stqp,5 %r44, %r0, %b[25] } { loop_mode stqp,2 %r49, %r0, %b[16] stqp,5 %r3, %r0, %g22 } { loop_mode stqp,2 %r28, %r0, %b[15] stqp,5 %r17, %r0, %b[36] } { loop_mode stqp,2 %r54, %r0, %g20 stqp,5 %r43, %r0, %b[18] } { loop_mode stqp,2 %r50, %r0, %b[26] stqp,5 %r5, %r0, %b[27] } { loop_mode stqp,2 %r45, %r0, %g28 stqp,5 %r11, %r0, %g23 } { loop_mode stqp,2 %r15, %r0, %b[21] stqp,5 %r19, %r0, %g16 } { loop_mode stqp,2 %r34, %r0, %g25 stqp,5 %r9, %r0, %b[19] } { loop_mode stqp,2 %r57, %r0, %g19 stqp,5 %r47, %r0, %b[14] } { loop_mode stqp,2 %r53, %r0, %b[24] stqp,5 %r32, %r0, %g24 } { loop_mode stqp,2 %r40, %r0, %g18 stqp,5 %r42, %r0, %g27 } { loop_mode stqp,2 %r4, %r0, %b[12] stqp,5 %r14, %r0, %g31 } { loop_mode stqp,2 %r1, %r0, %b[17] stqp,5 %r12, %r0, %g17 } { loop_mode stqp,2 %r56, %r0, %g21 stqp,5 %r39, %r0, %b[11] } { loop_mode stqp,2 %r46, %r0, %b[23] stqp,5 %r31, %r0, %b[13] } { loop_mode stqp,2 %r41, %r0, %b[20] stqp,5 %r33, %r0, %g26 } { loop_mode addd,0,sm %r0, _f16s,_lts0lo 0x30, %r0 stqp,2 %r52, %r0, %g29 stqp,5 %r48, %r0, %b[22] } { loop_mode ct %ctpr1 ? %NOT_LOOP_END alc alcf=1, alct=1 } { loop_mode ldd,0 %r8, _f16s,_lts0lo 0xff30, %b[11], mas=0x4 ldd,2 %r8, _f16s,_lts0hi 0xff38, %b[12], mas=0x4 ldd,3 %r8, _f16s,_lts1lo 0xff40, %b[13], mas=0x4 ldd,5 %r8, _f16s,_lts1hi 0xff48, %b[14], mas=0x4 } { loop_mode ldd,0 %r8, _f16s,_lts0lo 0xff50, %b[15], mas=0x4 ldd,2 %r8, _f16s,_lts0hi 0xff58, %b[16], mas=0x4 ldd,3 %r8, _f16s,_lts1lo 0xff60, %b[17], mas=0x4 ldd,5 %r8, _f16s,_lts1hi 0xff68, %b[18], mas=0x4 } { loop_mode ldd,0 %r8, _f16s,_lts0lo 0xff70, %b[19], mas=0x4 ldd,2 %r8, _f16s,_lts0hi 0xff78, %b[20], mas=0x4 ldd,3 %r8, _f16s,_lts1lo 0xff80, %b[21], mas=0x4 ldd,5 %r8, _f16s,_lts1hi 0xff88, %b[22], mas=0x4 } { loop_mode ldd,0 %r8, _f16s,_lts0lo 0xff90, %b[23], mas=0x4 ldd,2 %r8, _f16s,_lts0hi 0xff98, %b[24], mas=0x4 ldd,3 %r8, _f16s,_lts1lo 0xffa0, %b[25], mas=0x4 ldd,5 %r8, _f16s,_lts1hi 0xffa8, %b[26], mas=0x4 } { loop_mode ldd,0 %r8, _f16s,_lts0lo 0xffb0, %b[27], mas=0x4 ldd,2 %r8, _f16s,_lts0hi 0xffb8, %b[28], mas=0x4 ldd,3 %r8, _f16s,_lts1lo 0xffc0, %b[29], mas=0x4 ldd,5 %r8, _f16s,_lts1hi 0xffc8, %b[30], mas=0x4 } { loop_mode ldd,0 %r8, _f16s,_lts0lo 0xffd0, %b[31], mas=0x4 ldd,2 %r8, _f16s,_lts0hi 0xffd8, %b[32], mas=0x4 ldd,3 %r8, _f16s,_lts1lo 0xffe0, %b[33], mas=0x4 ldd,5 %r8, _f16s,_lts1hi 0xffe8, %b[34], mas=0x4 } { loop_mode ldd,0 %r8, _f16s,_lts0lo 0xfff0, %b[35], mas=0x4 ldd,2 %r8, _f16s,_lts0hi 0xfff8, %b[36], mas=0x4 }
Теоретическая скорость: 96 комплексных чисел за 158 тактов (96/158) = 4.86 Байт/такт
Четверная теоретическая скорость: 19.44 Байт/такт
Замеры скорости
Итоги по stage_radix4_2x
Скорости упали по сравнению с исходными версиями stage_radix4.
График FFT находится здесь.
stage_radix4_readConjSwap
Один проход по stage_radix4_readConjSwap совершает ту же работу, что 2 прохода по stage_radix2. Поэтому скорость stage_radix4_readConjSwap будем умножать на 2 для удобства сравнения с stage_radix2 (этот факт подписан на оси графика и в выводе консоли).
1. stage_radix4_readConjSwap_simd64
Вычисления делаем аналогично stage_radix2_readConjSwap_simd64.
Код на Си
void stage_radix4_readConjSwap_simd64(int data_count, myComplex *data_in, myComplex *data_out, myComplex *conj_coefC, myComplex *conj_coefD, myComplex *conj_coefE, myComplex *swap_coefC, myComplex *swap_coefD, myComplex *swap_coefE) { uint64_t *x_in = (uint64_t*)&data_in[0]; uint64_t *y_in = (uint64_t*)&data_in[1]; uint64_t *z_in = (uint64_t*)&data_in[2]; uint64_t *w_in = (uint64_t*)&data_in[3]; uint64_t *conj_c_in = (uint64_t*)conj_coefC; uint64_t *conj_d_in = (uint64_t*)conj_coefD; uint64_t *conj_e_in = (uint64_t*)conj_coefE; uint64_t *swap_c_in = (uint64_t*)swap_coefC; uint64_t *swap_d_in = (uint64_t*)swap_coefD; uint64_t *swap_e_in = (uint64_t*)swap_coefE; uint64_t *out_0 = (uint64_t*)&data_out[0*data_count/4]; uint64_t *out_1 = (uint64_t*)&data_out[1*data_count/4]; uint64_t *out_2 = (uint64_t*)&data_out[2*data_count/4]; uint64_t *out_3 = (uint64_t*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { uint64_t x = x_in[4*i]; uint64_t y = y_in[4*i]; uint64_t z = z_in[4*i]; uint64_t w = w_in[4*i]; uint64_t conj_c = conj_c_in[i]; uint64_t conj_d = conj_d_in[i]; uint64_t conj_e = conj_e_in[i]; uint64_t swap_c = swap_c_in[i]; uint64_t swap_d = swap_d_in[i]; uint64_t swap_e = swap_e_in[i]; uint64_t cy_real = __builtin_e2k_pfmuls(conj_c, y); uint64_t dz_real = __builtin_e2k_pfmuls(conj_d, z); uint64_t ew_real = __builtin_e2k_pfmuls(conj_e, w); uint64_t cy_imag = __builtin_e2k_pfmuls(swap_c, y); uint64_t dz_imag = __builtin_e2k_pfmuls(swap_d, z); uint64_t ew_imag = __builtin_e2k_pfmuls(swap_e, w); uint64_t cy = __builtin_e2k_pfhadds(cy_real, cy_imag); uint64_t dz = __builtin_e2k_pfhadds(dz_real, dz_imag); uint64_t ew = __builtin_e2k_pfhadds(ew_real, ew_imag); uint64_t add02 = __builtin_e2k_pfadds( x, dz); uint64_t sub02 = __builtin_e2k_pfsubs( x, dz); uint64_t add13 = __builtin_e2k_pfadds(cy, ew); uint64_t sub13 = __builtin_e2k_pfsubs(cy, ew); //uint64_t conj_sub13 = __builtin_e2k_pxord(sub13, 1LL<<63); //uint64_t sub13i = __builtin_e2k_pshufb(0, conj_sub13, 0x0302010007060504); uint64_t swap_sub13 = __builtin_e2k_pshufb(0, sub13, 0x0302010007060504); uint64_t sub13i = __builtin_e2k_pxord(swap_sub13, 1LL<<31); out_0[i] = __builtin_e2k_pfadds(add02, add13); out_1[i] = __builtin_e2k_pfsubs(sub02, sub13i); out_2[i] = __builtin_e2k_pfsubs(add02, add13); out_3[i] = __builtin_e2k_pfadds(sub02, sub13i); } }
Основной цикл на ассемблере
.L640: { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=7, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=5, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=6, asz=3, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=3, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=4, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=2, asz=3, abs=24, disp=0 } .L275: { loop_mode pfmul_hadds,0,sm %b[43], %b[59], %b[80], %b[11] pfadd_adds,1,sm %b[34], %b[17], %b[87], %b[67] pfadd_rsubs,2,sm %b[34], %b[17], %b[87], %b[54] pfmuls,3,sm %b[12], %b[55], %b[76] pfsubs,4,sm %b[42], %b[25], %b[88] pfadds,5,sm %b[44], %b[27], %b[83] movad,1 area=0, ind=0, am=1, be=0, %b[70] movad,2 area=2, ind=0, am=1, be=0, %b[1] movad,3 area=1, ind=0, am=1, be=0, %b[0] } { loop_mode pfsub_rsubs,0,sm %b[34], %b[17], %b[99], %b[80] pfsub_adds,1,sm %b[34], %b[17], %b[99], %b[59] staad,2 %b[73], %aad4[ %aasti11 ] incr,2 %aaincr0 pfmuls,3,sm %b[100], %b[64], %b[87] pshufb,4,sm 0x0, %b[90], %r21, %b[93] staad,5 %b[60], %aad2[ %aasti9 ] incr,5 %aaincr0 movad,0 area=3, ind=0, am=1, be=0, %b[44] movad,1 area=2, ind=0, am=1, be=0, %b[27] movad,2 area=0, ind=0, am=0, be=0, %b[12] movad,3 area=0, ind=16, am=0, be=0, %b[43] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmul_hadds,0,sm %b[52], %b[68], %b[91], %b[17] pfmul_hadds,1,sm %b[9], %b[81], %b[94], %b[34] staad,2 %b[86], %aad3[ %aasti10 ] incr,2 %aaincr0 pfmuls,3,sm %b[74], %b[77], %b[90] xord,4,sm %b[95], %r9, %b[97] staad,5 %b[65], %aad1[ %aasti8 ] incr,5 %aaincr0 movad,1 area=1, ind=0, am=1, be=0, %b[96] movad,2 area=0, ind=8, am=1, be=0, %b[73] movad,3 area=0, ind=24, am=0, be=0, %b[60] }
Теоретическая скорость: 4 комплексных числа за 3 такта (4/3) = 10.67 Байт/такт
Двойная теоретическая скорость: 21.33 Байт/такт
Замеры скорости
2. stage_radix4_readConjSwap_simd128
Вычисления делаем аналогично stage_radix2_readConjSwap_simd128.
Код на Си
void stage_radix4_readConjSwap_simd128(int data_count, myComplex *data_in, myComplex *data_out, myComplex *conj_coefC, myComplex *conj_coefD, myComplex *conj_coefE, myComplex *swap_coefC, myComplex *swap_coefD, myComplex *swap_coefE) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *zw0_in = (__v2di*)&data_in[2]; __v2di *xy1_in = (__v2di*)&data_in[4]; __v2di *zw1_in = (__v2di*)&data_in[6]; __v2di *conj_c_in = (__v2di*)conj_coefC; __v2di *conj_d_in = (__v2di*)conj_coefD; __v2di *conj_e_in = (__v2di*)conj_coefE; __v2di *swap_c_in = (__v2di*)swap_coefC; __v2di *swap_d_in = (__v2di*)swap_coefD; __v2di *swap_e_in = (__v2di*)swap_coefE; __v2di *out_0 = (__v2di*)&data_out[0*data_count/4]; __v2di *out_1 = (__v2di*)&data_out[1*data_count/4]; __v2di *out_2 = (__v2di*)&data_out[2*data_count/4]; __v2di *out_3 = (__v2di*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/8; ++i) { __v2di xy0 = xy0_in[4*i]; __v2di zw0 = zw0_in[4*i]; __v2di xy1 = xy1_in[4*i]; __v2di zw1 = zw1_in[4*i]; __v2di conj_c = conj_c_in[i]; __v2di conj_d = conj_d_in[i]; __v2di conj_e = conj_e_in[i]; __v2di swap_c = swap_c_in[i]; __v2di swap_d = swap_d_in[i]; __v2di swap_e = swap_e_in[i]; __v2di x = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di cy_real = __builtin_e2k_qpfmuls(conj_c, y); __v2di dz_real = __builtin_e2k_qpfmuls(conj_d, z); __v2di ew_real = __builtin_e2k_qpfmuls(conj_e, w); __v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y); __v2di dz_imag = __builtin_e2k_qpfmuls(swap_d, z); __v2di ew_imag = __builtin_e2k_qpfmuls(swap_e, w); __v2di cy_rrii = __builtin_e2k_qpfhadds(cy_real, cy_imag); __v2di dz_rrii = __builtin_e2k_qpfhadds(dz_real, dz_imag); __v2di ew_rrii = __builtin_e2k_qpfhadds(ew_real, ew_imag); __v2di dz = __builtin_e2k_qpshufb(dz_rrii, dz_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di add02 = __builtin_e2k_qpfadds( x, dz); __v2di sub02 = __builtin_e2k_qpfsubs( x, dz); __v2di add13_rrii = __builtin_e2k_qpfadds(cy_rrii, ew_rrii); __v2di sub13_rrii = __builtin_e2k_qpfsubs(cy_rrii, ew_rrii); __v2di add13 = __builtin_e2k_qpshufb(add13_rrii, add13_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); //__v2di conj_sub13 = __builtin_e2k_qpxor(sub13_rrii, (__v2di){(1LL<<63) + (1LL<<31), 0}); //__v2di sub13i = __builtin_e2k_qpshufb(conj_sub13, conj_sub13, (__v2di){0x030201000B0A0908, 0x070605040F0E0D0C}); __v2di swap_sub13 = __builtin_e2k_qpshufb(sub13_rrii, sub13_rrii, (__v2di){0x030201000B0A0908, 0x070605040F0E0D0C}); __v2di sub13i = __builtin_e2k_qpxor(swap_sub13, (__v2di){1LL<<31, 1LL<<31}); out_0[i] = __builtin_e2k_qpfadds(add02, add13); out_1[i] = __builtin_e2k_qpfsubs(sub02, sub13i); out_2[i] = __builtin_e2k_qpfsubs(add02, add13); out_3[i] = __builtin_e2k_qpfadds(sub02, sub13i); } }
Основной цикл на ассемблере
.L1243: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=3, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=3, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=3, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=3, abs=24, disp=0 } .L699: { loop_mode qpfmuls,0,sm %b[59], %b[33], %b[48] qpfmul_hadds,1,sm %b[32], %b[41], %b[80], %b[6] qpfadd_adds,2,sm %b[24], %b[77], %b[62], %b[47] qpshufb,3,sm %b[54], %b[54], %g17, %b[60] qpshufb,4,sm %b[42], %b[42], %g16, %b[69] qpfadd_rsubs,5,sm %b[24], %b[77], %b[62], %b[7] movaqp,1 area=0, ind=0, am=0, be=0, %b[1] movaqp,3 area=0, ind=0, am=0, be=0, %b[0] } { loop_mode qpfmul_hadds,0,sm %b[19], %b[38], %b[73], %b[72] qpfmul_hadds,1,sm %b[29], %b[35], %b[50], %b[41] qpfsub_rsubs,2,sm %b[24], %b[77], %b[81], %b[54] qpshufb,3,sm %b[2], %b[3], %r22, %b[32] qpxor,4,sm %b[69], %g18, %b[79] qpfsub_adds,5,sm %b[24], %b[77], %b[81], %b[42] movaqp,1 area=0, ind=16, am=1, be=0, %b[62] movaqp,3 area=0, ind=16, am=1, be=0, %b[59] } { loop_mode qpfadds,0,sm %b[76], %b[10], %b[50] qpfsubs,1,sm %b[76], %b[10], %b[38] staaqp,2 %b[51], %aad4[ %aasti11 ] incr,2 %aaincr0 qpshufb,3,sm %b[61], %b[64], %r22, %b[35] qpshufb,4,sm %b[63], %b[66], %g19, %b[29] staaqp,5 %b[11], %aad2[ %aasti9 ] incr,5 %aaincr0 movaqp,1 area=3, ind=0, am=1, be=0, %b[19] movaqp,3 area=3, ind=0, am=1, be=0, %b[24] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfmuls,0,sm %b[70], %b[34], %b[69] qpfmuls,1,sm %b[67], %b[37], %b[76] staaqp,2 %b[58], %aad3[ %aasti10 ] incr,2 %aaincr0 qpshufb,3,sm %b[4], %b[5], %g19, %b[10] qpshufb,4,sm %b[45], %b[45], %g17, %b[73] staaqp,5 %b[46], %aad1[ %aasti8 ] incr,5 %aaincr0 movaqp,0 area=2, ind=0, am=1, be=0, %b[63] movaqp,1 area=1, ind=0, am=1, be=0, %b[66] movaqp,2 area=2, ind=0, am=1, be=0, %b[11] movaqp,3 area=1, ind=0, am=1, be=0, %b[51] }
Теоретическая скорость: 8 комплексных чисел за 4 такта (8/4) = 16 Байт/такт
Двойная теоретическая скорость: 32 Байт/такт
Замеры скорости
Итоги по stage_radix4_readConjSwap
График FFT находится здесь.
stage_radix4_readConjSwap_2x
Один проход по stage_radix4_readConjSwap_2x совершает ту же работу, что 2 прохода по stage_radix4_readConjSwap. А один проход по stage_radix4_readConjSwap совершает ту же работу, что 2 прохода по stage_radix2. Поэтому скорость stage_radix4_readConjSwap_2x будем умножать на 4 для удобства сравнения с stage_radix2 (этот факт подписан на оси графика и в выводе консоли).
1. stage_radix4_readConjSwap_2x_simd64
Здесь происходит ручная раскрутка алгоритма stage_radix4_readConjSwap_simd64 в 2 раза.
Код на Си
void stage_radix4_readConjSwap_2x_simd64(int data_count, myComplex *data_in, myComplex *data_out, myComplex *conj_coefC_a, myComplex *conj_coefD_a, myComplex *conj_coefE_a, myComplex *conj_coefC_b, myComplex *conj_coefD_b, myComplex *conj_coefE_b, myComplex *swap_coefC_a, myComplex *swap_coefD_a, myComplex *swap_coefE_a, myComplex *swap_coefC_b, myComplex *swap_coefD_b, myComplex *swap_coefE_b) { uint64_t *x0_in = (uint64_t*)&data_in[ 0]; uint64_t *y0_in = (uint64_t*)&data_in[ 1]; uint64_t *z0_in = (uint64_t*)&data_in[ 2]; uint64_t *w0_in = (uint64_t*)&data_in[ 3]; uint64_t *x1_in = (uint64_t*)&data_in[ 4]; uint64_t *y1_in = (uint64_t*)&data_in[ 5]; uint64_t *z1_in = (uint64_t*)&data_in[ 6]; uint64_t *w1_in = (uint64_t*)&data_in[ 7]; uint64_t *x2_in = (uint64_t*)&data_in[ 8]; uint64_t *y2_in = (uint64_t*)&data_in[ 9]; uint64_t *z2_in = (uint64_t*)&data_in[10]; uint64_t *w2_in = (uint64_t*)&data_in[11]; uint64_t *x3_in = (uint64_t*)&data_in[12]; uint64_t *y3_in = (uint64_t*)&data_in[13]; uint64_t *z3_in = (uint64_t*)&data_in[14]; uint64_t *w3_in = (uint64_t*)&data_in[15]; uint64_t *conj_c0a_in = (uint64_t*)&conj_coefC_a[0]; uint64_t *conj_c1a_in = (uint64_t*)&conj_coefC_a[1]; uint64_t *conj_c2a_in = (uint64_t*)&conj_coefC_a[2]; uint64_t *conj_c3a_in = (uint64_t*)&conj_coefC_a[3]; uint64_t *conj_d0a_in = (uint64_t*)&conj_coefD_a[0]; uint64_t *conj_d1a_in = (uint64_t*)&conj_coefD_a[1]; uint64_t *conj_d2a_in = (uint64_t*)&conj_coefD_a[2]; uint64_t *conj_d3a_in = (uint64_t*)&conj_coefD_a[3]; uint64_t *conj_e0a_in = (uint64_t*)&conj_coefE_a[0]; uint64_t *conj_e1a_in = (uint64_t*)&conj_coefE_a[1]; uint64_t *conj_e2a_in = (uint64_t*)&conj_coefE_a[2]; uint64_t *conj_e3a_in = (uint64_t*)&conj_coefE_a[3]; uint64_t *conj_c0b_in = (uint64_t*)&conj_coefC_b[0*data_count/16]; uint64_t *conj_c1b_in = (uint64_t*)&conj_coefC_b[1*data_count/16]; uint64_t *conj_c2b_in = (uint64_t*)&conj_coefC_b[2*data_count/16]; uint64_t *conj_c3b_in = (uint64_t*)&conj_coefC_b[3*data_count/16]; uint64_t *conj_d0b_in = (uint64_t*)&conj_coefD_b[0*data_count/16]; uint64_t *conj_d1b_in = (uint64_t*)&conj_coefD_b[1*data_count/16]; uint64_t *conj_d2b_in = (uint64_t*)&conj_coefD_b[2*data_count/16]; uint64_t *conj_d3b_in = (uint64_t*)&conj_coefD_b[3*data_count/16]; uint64_t *conj_e0b_in = (uint64_t*)&conj_coefE_b[0*data_count/16]; uint64_t *conj_e1b_in = (uint64_t*)&conj_coefE_b[1*data_count/16]; uint64_t *conj_e2b_in = (uint64_t*)&conj_coefE_b[2*data_count/16]; uint64_t *conj_e3b_in = (uint64_t*)&conj_coefE_b[3*data_count/16]; uint64_t *swap_c0a_in = (uint64_t*)&swap_coefC_a[0]; uint64_t *swap_c1a_in = (uint64_t*)&swap_coefC_a[1]; uint64_t *swap_c2a_in = (uint64_t*)&swap_coefC_a[2]; uint64_t *swap_c3a_in = (uint64_t*)&swap_coefC_a[3]; uint64_t *swap_d0a_in = (uint64_t*)&swap_coefD_a[0]; uint64_t *swap_d1a_in = (uint64_t*)&swap_coefD_a[1]; uint64_t *swap_d2a_in = (uint64_t*)&swap_coefD_a[2]; uint64_t *swap_d3a_in = (uint64_t*)&swap_coefD_a[3]; uint64_t *swap_e0a_in = (uint64_t*)&swap_coefE_a[0]; uint64_t *swap_e1a_in = (uint64_t*)&swap_coefE_a[1]; uint64_t *swap_e2a_in = (uint64_t*)&swap_coefE_a[2]; uint64_t *swap_e3a_in = (uint64_t*)&swap_coefE_a[3]; uint64_t *swap_c0b_in = (uint64_t*)&swap_coefC_b[0*data_count/16]; uint64_t *swap_c1b_in = (uint64_t*)&swap_coefC_b[1*data_count/16]; uint64_t *swap_c2b_in = (uint64_t*)&swap_coefC_b[2*data_count/16]; uint64_t *swap_c3b_in = (uint64_t*)&swap_coefC_b[3*data_count/16]; uint64_t *swap_d0b_in = (uint64_t*)&swap_coefD_b[0*data_count/16]; uint64_t *swap_d1b_in = (uint64_t*)&swap_coefD_b[1*data_count/16]; uint64_t *swap_d2b_in = (uint64_t*)&swap_coefD_b[2*data_count/16]; uint64_t *swap_d3b_in = (uint64_t*)&swap_coefD_b[3*data_count/16]; uint64_t *swap_e0b_in = (uint64_t*)&swap_coefE_b[0*data_count/16]; uint64_t *swap_e1b_in = (uint64_t*)&swap_coefE_b[1*data_count/16]; uint64_t *swap_e2b_in = (uint64_t*)&swap_coefE_b[2*data_count/16]; uint64_t *swap_e3b_in = (uint64_t*)&swap_coefE_b[3*data_count/16]; uint64_t *out_0 = (uint64_t*)&data_out[ 0*data_count/16]; uint64_t *out_1 = (uint64_t*)&data_out[ 1*data_count/16]; uint64_t *out_2 = (uint64_t*)&data_out[ 2*data_count/16]; uint64_t *out_3 = (uint64_t*)&data_out[ 3*data_count/16]; uint64_t *out_4 = (uint64_t*)&data_out[ 4*data_count/16]; uint64_t *out_5 = (uint64_t*)&data_out[ 5*data_count/16]; uint64_t *out_6 = (uint64_t*)&data_out[ 6*data_count/16]; uint64_t *out_7 = (uint64_t*)&data_out[ 7*data_count/16]; uint64_t *out_8 = (uint64_t*)&data_out[ 8*data_count/16]; uint64_t *out_9 = (uint64_t*)&data_out[ 9*data_count/16]; uint64_t *out_10 = (uint64_t*)&data_out[10*data_count/16]; uint64_t *out_11 = (uint64_t*)&data_out[11*data_count/16]; uint64_t *out_12 = (uint64_t*)&data_out[12*data_count/16]; uint64_t *out_13 = (uint64_t*)&data_out[13*data_count/16]; uint64_t *out_14 = (uint64_t*)&data_out[14*data_count/16]; uint64_t *out_15 = (uint64_t*)&data_out[15*data_count/16]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/16; ++i) { uint64_t x0 = x0_in[16*i]; uint64_t y0 = y0_in[16*i]; uint64_t z0 = z0_in[16*i]; uint64_t w0 = w0_in[16*i]; uint64_t conj_c0 = conj_c0a_in[4*i]; uint64_t conj_d0 = conj_d0a_in[4*i]; uint64_t conj_e0 = conj_e0a_in[4*i]; uint64_t swap_c0 = swap_c0a_in[4*i]; uint64_t swap_d0 = swap_d0a_in[4*i]; uint64_t swap_e0 = swap_e0a_in[4*i]; uint64_t x1 = x1_in[16*i]; uint64_t y1 = y1_in[16*i]; uint64_t z1 = z1_in[16*i]; uint64_t w1 = w1_in[16*i]; uint64_t conj_c1 = conj_c1a_in[4*i]; uint64_t conj_d1 = conj_d1a_in[4*i]; uint64_t conj_e1 = conj_e1a_in[4*i]; uint64_t swap_c1 = swap_c1a_in[4*i]; uint64_t swap_d1 = swap_d1a_in[4*i]; uint64_t swap_e1 = swap_e1a_in[4*i]; uint64_t x2 = x2_in[16*i]; uint64_t y2 = y2_in[16*i]; uint64_t z2 = z2_in[16*i]; uint64_t w2 = w2_in[16*i]; uint64_t conj_c2 = conj_c2a_in[4*i]; uint64_t conj_d2 = conj_d2a_in[4*i]; uint64_t conj_e2 = conj_e2a_in[4*i]; uint64_t swap_c2 = swap_c2a_in[4*i]; uint64_t swap_d2 = swap_d2a_in[4*i]; uint64_t swap_e2 = swap_e2a_in[4*i]; uint64_t x3 = x3_in[16*i]; uint64_t y3 = y3_in[16*i]; uint64_t z3 = z3_in[16*i]; uint64_t w3 = w3_in[16*i]; uint64_t conj_c3 = conj_c3a_in[4*i]; uint64_t conj_d3 = conj_d3a_in[4*i]; uint64_t conj_e3 = conj_e3a_in[4*i]; uint64_t swap_c3 = swap_c3a_in[4*i]; uint64_t swap_d3 = swap_d3a_in[4*i]; uint64_t swap_e3 = swap_e3a_in[4*i]; uint64_t cy0_real = __builtin_e2k_pfmuls(conj_c0, y0); uint64_t cy1_real = __builtin_e2k_pfmuls(conj_c1, y1); uint64_t cy2_real = __builtin_e2k_pfmuls(conj_c2, y2); uint64_t cy3_real = __builtin_e2k_pfmuls(conj_c3, y3); uint64_t dz0_real = __builtin_e2k_pfmuls(conj_d0, z0); uint64_t dz1_real = __builtin_e2k_pfmuls(conj_d1, z1); uint64_t dz2_real = __builtin_e2k_pfmuls(conj_d2, z2); uint64_t dz3_real = __builtin_e2k_pfmuls(conj_d3, z3); uint64_t ew0_real = __builtin_e2k_pfmuls(conj_e0, w0); uint64_t ew1_real = __builtin_e2k_pfmuls(conj_e1, w1); uint64_t ew2_real = __builtin_e2k_pfmuls(conj_e2, w2); uint64_t ew3_real = __builtin_e2k_pfmuls(conj_e3, w3); uint64_t cy0_imag = __builtin_e2k_pfmuls(swap_c0, y0); uint64_t cy1_imag = __builtin_e2k_pfmuls(swap_c1, y1); uint64_t cy2_imag = __builtin_e2k_pfmuls(swap_c2, y2); uint64_t cy3_imag = __builtin_e2k_pfmuls(swap_c3, y3); uint64_t dz0_imag = __builtin_e2k_pfmuls(swap_d0, z0); uint64_t dz1_imag = __builtin_e2k_pfmuls(swap_d1, z1); uint64_t dz2_imag = __builtin_e2k_pfmuls(swap_d2, z2); uint64_t dz3_imag = __builtin_e2k_pfmuls(swap_d3, z3); uint64_t ew0_imag = __builtin_e2k_pfmuls(swap_e0, w0); uint64_t ew1_imag = __builtin_e2k_pfmuls(swap_e1, w1); uint64_t ew2_imag = __builtin_e2k_pfmuls(swap_e2, w2); uint64_t ew3_imag = __builtin_e2k_pfmuls(swap_e3, w3); uint64_t cy0 = __builtin_e2k_pfhadds(cy0_real, cy0_imag); uint64_t cy1 = __builtin_e2k_pfhadds(cy1_real, cy1_imag); uint64_t cy2 = __builtin_e2k_pfhadds(cy2_real, cy2_imag); uint64_t cy3 = __builtin_e2k_pfhadds(cy3_real, cy3_imag); uint64_t dz0 = __builtin_e2k_pfhadds(dz0_real, dz0_imag); uint64_t dz1 = __builtin_e2k_pfhadds(dz1_real, dz1_imag); uint64_t dz2 = __builtin_e2k_pfhadds(dz2_real, dz2_imag); uint64_t dz3 = __builtin_e2k_pfhadds(dz3_real, dz3_imag); uint64_t ew0 = __builtin_e2k_pfhadds(ew0_real, ew0_imag); uint64_t ew1 = __builtin_e2k_pfhadds(ew1_real, ew1_imag); uint64_t ew2 = __builtin_e2k_pfhadds(ew2_real, ew2_imag); uint64_t ew3 = __builtin_e2k_pfhadds(ew3_real, ew3_imag); uint64_t add02_0 = __builtin_e2k_pfadds( x0, dz0); uint64_t add02_1 = __builtin_e2k_pfadds( x1, dz1); uint64_t add02_2 = __builtin_e2k_pfadds( x2, dz2); uint64_t add02_3 = __builtin_e2k_pfadds( x3, dz3); uint64_t sub02_0 = __builtin_e2k_pfsubs( x0, dz0); uint64_t sub02_1 = __builtin_e2k_pfsubs( x1, dz1); uint64_t sub02_2 = __builtin_e2k_pfsubs( x2, dz2); uint64_t sub02_3 = __builtin_e2k_pfsubs( x3, dz3); uint64_t add13_0 = __builtin_e2k_pfadds(cy0, ew0); uint64_t add13_1 = __builtin_e2k_pfadds(cy1, ew1); uint64_t add13_2 = __builtin_e2k_pfadds(cy2, ew2); uint64_t add13_3 = __builtin_e2k_pfadds(cy3, ew3); uint64_t sub13_0 = __builtin_e2k_pfsubs(cy0, ew0); uint64_t sub13_1 = __builtin_e2k_pfsubs(cy1, ew1); uint64_t sub13_2 = __builtin_e2k_pfsubs(cy2, ew2); uint64_t sub13_3 = __builtin_e2k_pfsubs(cy3, ew3); //uint64_t conj_sub13_0 = __builtin_e2k_pxord(sub13_0, 1LL<<63); //uint64_t conj_sub13_1 = __builtin_e2k_pxord(sub13_1, 1LL<<63); //uint64_t conj_sub13_2 = __builtin_e2k_pxord(sub13_2, 1LL<<63); //uint64_t conj_sub13_3 = __builtin_e2k_pxord(sub13_3, 1LL<<63); //uint64_t sub13i_0 = __builtin_e2k_pshufb(0, conj_sub13_0, 0x0302010007060504); //uint64_t sub13i_1 = __builtin_e2k_pshufb(0, conj_sub13_1, 0x0302010007060504); //uint64_t sub13i_2 = __builtin_e2k_pshufb(0, conj_sub13_2, 0x0302010007060504); //uint64_t sub13i_3 = __builtin_e2k_pshufb(0, conj_sub13_3, 0x0302010007060504); uint64_t swap_sub13_0 = __builtin_e2k_pshufb(0, sub13_0, 0x0302010007060504); uint64_t swap_sub13_1 = __builtin_e2k_pshufb(0, sub13_1, 0x0302010007060504); uint64_t swap_sub13_2 = __builtin_e2k_pshufb(0, sub13_2, 0x0302010007060504); uint64_t swap_sub13_3 = __builtin_e2k_pshufb(0, sub13_3, 0x0302010007060504); uint64_t sub13i_0 = __builtin_e2k_pxord(swap_sub13_0, 1LL<<31); uint64_t sub13i_1 = __builtin_e2k_pxord(swap_sub13_1, 1LL<<31); uint64_t sub13i_2 = __builtin_e2k_pxord(swap_sub13_2, 1LL<<31); uint64_t sub13i_3 = __builtin_e2k_pxord(swap_sub13_3, 1LL<<31); uint64_t out0 = __builtin_e2k_pfadds(add02_0, add13_0); uint64_t out1 = __builtin_e2k_pfadds(add02_1, add13_1); uint64_t out2 = __builtin_e2k_pfadds(add02_2, add13_2); uint64_t out3 = __builtin_e2k_pfadds(add02_3, add13_3); uint64_t out4 = __builtin_e2k_pfsubs(sub02_0, sub13i_0); uint64_t out5 = __builtin_e2k_pfsubs(sub02_1, sub13i_1); uint64_t out6 = __builtin_e2k_pfsubs(sub02_2, sub13i_2); uint64_t out7 = __builtin_e2k_pfsubs(sub02_3, sub13i_3); uint64_t out8 = __builtin_e2k_pfsubs(add02_0, add13_0); uint64_t out9 = __builtin_e2k_pfsubs(add02_1, add13_1); uint64_t out10 = __builtin_e2k_pfsubs(add02_2, add13_2); uint64_t out11 = __builtin_e2k_pfsubs(add02_3, add13_3); uint64_t out12 = __builtin_e2k_pfadds(sub02_0, sub13i_0); uint64_t out13 = __builtin_e2k_pfadds(sub02_1, sub13i_1); uint64_t out14 = __builtin_e2k_pfadds(sub02_2, sub13i_2); uint64_t out15 = __builtin_e2k_pfadds(sub02_3, sub13i_3); x0 = out0; y0 = out1; z0 = out2; w0 = out3; conj_c0 = conj_c0b_in[i]; conj_d0 = conj_d0b_in[i]; conj_e0 = conj_e0b_in[i]; swap_c0 = swap_c0b_in[i]; swap_d0 = swap_d0b_in[i]; swap_e0 = swap_e0b_in[i]; x1 = out4; y1 = out5; z1 = out6; w1 = out7; conj_c1 = conj_c1b_in[i]; conj_d1 = conj_d1b_in[i]; conj_e1 = conj_e1b_in[i]; swap_c1 = swap_c1b_in[i]; swap_d1 = swap_d1b_in[i]; swap_e1 = swap_e1b_in[i]; x2 = out8; y2 = out9; z2 = out10; w2 = out11; conj_c2 = conj_c2b_in[i]; conj_d2 = conj_d2b_in[i]; conj_e2 = conj_e2b_in[i]; swap_c2 = swap_c2b_in[i]; swap_d2 = swap_d2b_in[i]; swap_e2 = swap_e2b_in[i]; x3 = out12; y3 = out13; z3 = out14; w3 = out15; conj_c3 = conj_c3b_in[i]; conj_d3 = conj_d3b_in[i]; conj_e3 = conj_e3b_in[i]; swap_c3 = swap_c3b_in[i]; swap_d3 = swap_d3b_in[i]; swap_e3 = swap_e3b_in[i]; cy0_real = __builtin_e2k_pfmuls(conj_c0, y0); cy1_real = __builtin_e2k_pfmuls(conj_c1, y1); cy2_real = __builtin_e2k_pfmuls(conj_c2, y2); cy3_real = __builtin_e2k_pfmuls(conj_c3, y3); dz0_real = __builtin_e2k_pfmuls(conj_d0, z0); dz1_real = __builtin_e2k_pfmuls(conj_d1, z1); dz2_real = __builtin_e2k_pfmuls(conj_d2, z2); dz3_real = __builtin_e2k_pfmuls(conj_d3, z3); ew0_real = __builtin_e2k_pfmuls(conj_e0, w0); ew1_real = __builtin_e2k_pfmuls(conj_e1, w1); ew2_real = __builtin_e2k_pfmuls(conj_e2, w2); ew3_real = __builtin_e2k_pfmuls(conj_e3, w3); cy0_imag = __builtin_e2k_pfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_pfmuls(swap_c1, y1); cy2_imag = __builtin_e2k_pfmuls(swap_c2, y2); cy3_imag = __builtin_e2k_pfmuls(swap_c3, y3); dz0_imag = __builtin_e2k_pfmuls(swap_d0, z0); dz1_imag = __builtin_e2k_pfmuls(swap_d1, z1); dz2_imag = __builtin_e2k_pfmuls(swap_d2, z2); dz3_imag = __builtin_e2k_pfmuls(swap_d3, z3); ew0_imag = __builtin_e2k_pfmuls(swap_e0, w0); ew1_imag = __builtin_e2k_pfmuls(swap_e1, w1); ew2_imag = __builtin_e2k_pfmuls(swap_e2, w2); ew3_imag = __builtin_e2k_pfmuls(swap_e3, w3); cy0 = __builtin_e2k_pfhadds(cy0_real, cy0_imag); cy1 = __builtin_e2k_pfhadds(cy1_real, cy1_imag); cy2 = __builtin_e2k_pfhadds(cy2_real, cy2_imag); cy3 = __builtin_e2k_pfhadds(cy3_real, cy3_imag); dz0 = __builtin_e2k_pfhadds(dz0_real, dz0_imag); dz1 = __builtin_e2k_pfhadds(dz1_real, dz1_imag); dz2 = __builtin_e2k_pfhadds(dz2_real, dz2_imag); dz3 = __builtin_e2k_pfhadds(dz3_real, dz3_imag); ew0 = __builtin_e2k_pfhadds(ew0_real, ew0_imag); ew1 = __builtin_e2k_pfhadds(ew1_real, ew1_imag); ew2 = __builtin_e2k_pfhadds(ew2_real, ew2_imag); ew3 = __builtin_e2k_pfhadds(ew3_real, ew3_imag); add02_0 = __builtin_e2k_pfadds( x0, dz0); add02_1 = __builtin_e2k_pfadds( x1, dz1); add02_2 = __builtin_e2k_pfadds( x2, dz2); add02_3 = __builtin_e2k_pfadds( x3, dz3); sub02_0 = __builtin_e2k_pfsubs( x0, dz0); sub02_1 = __builtin_e2k_pfsubs( x1, dz1); sub02_2 = __builtin_e2k_pfsubs( x2, dz2); sub02_3 = __builtin_e2k_pfsubs( x3, dz3); add13_0 = __builtin_e2k_pfadds(cy0, ew0); add13_1 = __builtin_e2k_pfadds(cy1, ew1); add13_2 = __builtin_e2k_pfadds(cy2, ew2); add13_3 = __builtin_e2k_pfadds(cy3, ew3); sub13_0 = __builtin_e2k_pfsubs(cy0, ew0); sub13_1 = __builtin_e2k_pfsubs(cy1, ew1); sub13_2 = __builtin_e2k_pfsubs(cy2, ew2); sub13_3 = __builtin_e2k_pfsubs(cy3, ew3); //conj_sub13_0 = __builtin_e2k_pxord(sub13_0, 1LL<<63); //conj_sub13_1 = __builtin_e2k_pxord(sub13_1, 1LL<<63); //conj_sub13_2 = __builtin_e2k_pxord(sub13_2, 1LL<<63); //conj_sub13_3 = __builtin_e2k_pxord(sub13_3, 1LL<<63); //sub13i_0 = __builtin_e2k_pshufb(0, conj_sub13_0, 0x0302010007060504); //sub13i_1 = __builtin_e2k_pshufb(0, conj_sub13_1, 0x0302010007060504); //sub13i_2 = __builtin_e2k_pshufb(0, conj_sub13_2, 0x0302010007060504); //sub13i_3 = __builtin_e2k_pshufb(0, conj_sub13_3, 0x0302010007060504); swap_sub13_0 = __builtin_e2k_pshufb(0, sub13_0, 0x0302010007060504); swap_sub13_1 = __builtin_e2k_pshufb(0, sub13_1, 0x0302010007060504); swap_sub13_2 = __builtin_e2k_pshufb(0, sub13_2, 0x0302010007060504); swap_sub13_3 = __builtin_e2k_pshufb(0, sub13_3, 0x0302010007060504); sub13i_0 = __builtin_e2k_pxord(swap_sub13_0, 1LL<<31); sub13i_1 = __builtin_e2k_pxord(swap_sub13_1, 1LL<<31); sub13i_2 = __builtin_e2k_pxord(swap_sub13_2, 1LL<<31); sub13i_3 = __builtin_e2k_pxord(swap_sub13_3, 1LL<<31); out_0[i] = __builtin_e2k_pfadds(add02_0, add13_0); out_1[i] = __builtin_e2k_pfadds(add02_1, add13_1); out_2[i] = __builtin_e2k_pfadds(add02_2, add13_2); out_3[i] = __builtin_e2k_pfadds(add02_3, add13_3); out_4[i] = __builtin_e2k_pfsubs(sub02_0, sub13i_0); out_5[i] = __builtin_e2k_pfsubs(sub02_1, sub13i_1); out_6[i] = __builtin_e2k_pfsubs(sub02_2, sub13i_2); out_7[i] = __builtin_e2k_pfsubs(sub02_3, sub13i_3); out_8[i] = __builtin_e2k_pfsubs(add02_0, add13_0); out_9[i] = __builtin_e2k_pfsubs(add02_1, add13_1); out_10[i] = __builtin_e2k_pfsubs(add02_2, add13_2); out_11[i] = __builtin_e2k_pfsubs(add02_3, add13_3); out_12[i] = __builtin_e2k_pfadds(sub02_0, sub13i_0); out_13[i] = __builtin_e2k_pfadds(sub02_1, sub13i_1); out_14[i] = __builtin_e2k_pfadds(sub02_2, sub13i_2); out_15[i] = __builtin_e2k_pfadds(sub02_3, sub13i_3); } }
Основной цикл на ассемблере
.L2581: { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=1, asz=0, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=1, asz=0, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=1, asz=0, abs=1, disp=64 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=1, asz=0, abs=1, disp=96 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=7, asz=1, abs=2, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=6, asz=1, abs=2, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=5, asz=1, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=4, asz=1, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=3, asz=1, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=2, asz=1, abs=6, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=16, incr=0, ind=0, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=15, incr=0, ind=0, asz=1, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=14, incr=0, ind=0, asz=1, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=13, incr=0, ind=0, asz=1, abs=10, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=12, incr=0, ind=0, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=11, incr=0, ind=0, asz=1, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=10, incr=0, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=9, incr=0, ind=0, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=8, incr=0, ind=0, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=7, incr=0, ind=0, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=6, incr=0, ind=0, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=5, incr=0, ind=0, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=4, incr=0, ind=0, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=3, incr=0, ind=0, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=2, incr=0, ind=0, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=1, incr=0, ind=0, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=15, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=14, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=13, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=12, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=11, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=10, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=9, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=8, d=0, incr=0, ind=8, asz=1, abs=30, disp=0 } .L1747: { loop_mode pfadd_rsubs,1,sm %b[63], %b[62], %b[77], %b[86] pfmul_hadds,2,sm %b[9], %b[117], %b[86], %b[68] pfmul_hadds,3,sm %b[33], %b[68], %b[88], %b[88] } { loop_mode pfmuls,1,sm %b[113], %b[103], %b[73] xord,2,sm %b[107], %r3, %b[90] pfmul_hadds,3,sm %b[71], %b[73], %b[65], %b[71] pfmul_hadds,4,sm %b[40], %b[106], %b[101], %b[65] movad,0 area=16, ind=0, am=1, be=0, %b[1] movad,1 area=15, ind=0, am=1, be=0, %b[12] movad,2 area=16, ind=0, am=1, be=0, %b[8] movad,3 area=15, ind=0, am=1, be=0, %b[9] } { loop_mode pfsub_adds,2,sm %b[63], %b[62], %b[90], %b[92] pfadd_adds,3,sm %b[63], %b[62], %b[77], %b[95] pfmul_hadds,4,sm %b[5], %b[97], %b[105], %b[77] movad,0 area=14, ind=0, am=1, be=0, %b[16] movad,1 area=13, ind=0, am=1, be=0, %b[5] movad,2 area=14, ind=0, am=1, be=0, %b[13] movad,3 area=13, ind=0, am=1, be=0, %b[17] } { loop_mode pfsubs,1,sm %b[93], %b[82], %b[85] pfmul_hadds,2,sm %b[32], %b[64], %b[85], %b[64] movad,0 area=12, ind=0, am=1, be=0, %b[25] movad,1 area=11, ind=0, am=1, be=0, %b[24] movad,2 area=12, ind=0, am=1, be=0, %b[20] movad,3 area=11, ind=0, am=1, be=0, %b[21] } { loop_mode pfsubs,0,sm %b[91], %b[89], %b[96] pfadds,4,sm %b[74], %b[87], %b[93] pfadds,5,sm %b[93], %b[82], %b[82] movad,0 area=10, ind=0, am=1, be=0, %b[28] movad,1 area=9, ind=0, am=1, be=0, %b[33] movad,2 area=10, ind=0, am=1, be=0, %b[32] movad,3 area=9, ind=0, am=1, be=0, %b[29] } { loop_mode pfmul_hadds,0,sm %b[80], %b[103], %b[73], %b[62] pfadds,1,sm %b[91], %b[89], %b[72] pfmuls,2,sm %b[72], %b[108], %b[73] pfsub_rsubs,3,sm %b[63], %b[62], %b[90], %b[63] pfsubs,5,sm %b[74], %b[87], %b[74] movad,0 area=8, ind=0, am=1, be=0, %b[41] movad,1 area=7, ind=0, am=1, be=0, %b[36] movad,2 area=8, ind=0, am=1, be=0, %b[37] movad,3 area=7, ind=0, am=1, be=0, %b[40] } { loop_mode pfsubs,0,sm %b[75], %b[70], %b[80] pfsubs,5,sm %b[81], %b[78], %b[87] movad,0 area=6, ind=0, am=1, be=0, %b[49] movad,1 area=5, ind=0, am=1, be=0, %b[48] movad,2 area=6, ind=0, am=1, be=0, %b[44] movad,3 area=5, ind=0, am=1, be=0, %b[45] } { loop_mode pshufb,0,sm 0x0, %b[85], %r25, %b[84] pfmuls,1,sm %b[84], %b[102], %b[85] pfadds,4,sm %b[81], %b[78], %b[90] pfsubs,5,sm %b[83], %b[69], %b[89] movad,0 area=4, ind=0, am=0, be=0, %b[52] movad,1 area=4, ind=16, am=0, be=0, %b[78] movad,2 area=4, ind=0, am=0, be=0, %b[53] movad,3 area=4, ind=16, am=0, be=0, %b[81] } { loop_mode pshufb,1,sm 0x0, %b[96], %r25, %b[98] pfadds,3,sm %b[83], %b[69], %b[99] pfsubs,5,sm %b[88], %b[76], %b[97] movad,0 area=4, ind=24, am=0, be=0, %b[69] movad,1 area=4, ind=8, am=1, be=0, %b[83] movad,2 area=4, ind=24, am=0, be=0, %b[91] movad,3 area=4, ind=8, am=1, be=0, %b[96] } { loop_mode pfmul_hadds,0,sm %b[55], %b[108], %b[73], %b[75] pfadds,1,sm %b[75], %b[70], %b[100] pfadd_adds,2,sm %b[79], %b[66], %b[72], %b[74] pshufb,3,sm 0x0, %b[74], %r25, %b[101] pfadd_rsubs,4,sm %b[94], %b[71], %b[82], %b[76] pfadds,5,sm %b[88], %b[76], %b[104] movad,0 area=3, ind=0, am=0, be=0, %b[70] movad,1 area=3, ind=16, am=0, be=0, %b[88] movad,2 area=3, ind=0, am=0, be=0, %b[56] movad,3 area=3, ind=16, am=0, be=0, %b[73] } { loop_mode pfadd_rsubs,0,sm %b[79], %b[66], %b[72], %b[72] pshufb,1,sm 0x0, %b[80], %r25, %b[107] pfadd_adds,2,sm %b[86], %b[68], %b[93], %b[103] pshufb,4,sm 0x0, %b[87], %r25, %b[109] pfadd_rsubs,5,sm %b[86], %b[68], %b[93], %b[93] movad,0 area=3, ind=24, am=0, be=0, %b[105] movad,1 area=3, ind=8, am=1, be=0, %b[106] movad,2 area=3, ind=24, am=0, be=0, %b[80] movad,3 area=3, ind=8, am=1, be=0, %b[87] } { loop_mode pfmul_hadds,0,sm %b[58], %b[102], %b[85], %b[84] pfadd_adds,1,sm %b[94], %b[71], %b[82], %b[85] xord,2,sm %b[84], %r3, %b[112] pfadd_adds,3,sm %b[95], %b[65], %b[90], %b[102] pshufb,4,sm 0x0, %b[89], %r25, %b[111] pfadd_rsubs,5,sm %b[95], %b[65], %b[90], %b[90] movad,0 area=2, ind=0, am=0, be=0, %b[82] movad,1 area=2, ind=16, am=0, be=0, %b[108] movad,2 area=2, ind=24, am=0, be=0, %b[110] movad,3 area=0, ind=24, am=0, be=0, %b[89] } { loop_mode xord,0,sm %b[98], %r3, %b[116] pfsub_adds,1,sm %b[94], %b[71], %b[112], %b[94] pfsub_rsubs,2,sm %b[94], %b[71], %b[112], %b[97] pfadd_rsubs,3,sm %b[92], %b[77], %b[99], %b[98] pshufb,4,sm 0x0, %b[97], %r25, %b[115] pfadd_adds,5,sm %b[92], %b[77], %b[99], %b[99] movad,0 area=2, ind=24, am=0, be=0, %b[114] movad,1 area=2, ind=8, am=1, be=0, %b[113] movad,2 area=1, ind=16, am=0, be=0, %b[71] movad,3 area=0, ind=8, am=0, be=0, %b[112] } { loop_mode pfadd_adds,1,sm %b[57], %b[62], %b[100], %b[104] pfsub_adds,2,sm %b[79], %b[66], %b[116], %b[118] xord,3,sm %b[101], %r3, %g16 pfadd_rsubs,4,sm %b[63], %b[64], %b[104], %b[117] pfadd_adds,5,sm %b[63], %b[64], %b[104], %b[119] movad,0 area=1, ind=0, am=0, be=0, %b[55] movad,1 area=1, ind=16, am=0, be=0, %b[101] movad,2 area=1, ind=24, am=0, be=0, %g18 movad,3 area=1, ind=8, am=0, be=0, %g17 } { loop_mode xord,0,sm %b[107], %r3, %b[116] pfmuls,1,sm %b[67], %b[60], %g19 pfsub_rsubs,2,sm %b[79], %b[66], %b[116], %b[66] pfsub_rsubs,3,sm %b[86], %b[68], %g16, %b[107] xord,4,sm %b[109], %r3, %g16 pfsub_adds,5,sm %b[86], %b[68], %g16, %b[86] movad,0 area=1, ind=24, am=0, be=0, %b[68] movad,1 area=1, ind=8, am=1, be=0, %b[79] movad,2 area=2, ind=8, am=0, be=0, %b[109] movad,3 area=0, ind=16, am=0, be=0, %b[67] } { loop_mode pfsub_adds,0,sm %b[57], %b[62], %b[116], %b[95] pfsub_rsubs,3,sm %b[95], %b[65], %g16, %g16 pfsub_adds,4,sm %b[95], %b[65], %g16, %g21 xord,5,sm %b[111], %r3, %g20 movad,0 area=0, ind=0, am=0, be=0, %b[59] movad,1 area=0, ind=16, am=0, be=0, %b[58] movad,2 area=2, ind=0, am=1, be=0, %b[65] movad,3 area=2, ind=16, am=0, be=0, %b[111] } { loop_mode pfmuls,0,sm %b[106], %b[89], %g24 pfadd_rsubs,2,sm %b[57], %b[62], %b[100], %b[115] xord,3,sm %b[115], %r3, %g23 pfsub_rsubs,4,sm %b[92], %b[77], %g20, %g20 pfsub_adds,5,sm %b[92], %b[77], %g20, %g22 movad,0 area=0, ind=8, am=1, be=0, %b[100] movad,1 area=0, ind=24, am=0, be=0, %b[106] movad,2 area=1, ind=0, am=1, be=0, %b[92] movad,3 area=0, ind=0, am=1, be=0, %b[77] } { loop_mode pfmuls,0,sm %b[113], %b[112], %b[113] pfmuls,3,sm %b[110], %b[71], %b[63] pfsub_rsubs,4,sm %b[63], %b[64], %g23, %b[110] pfsub_adds,5,sm %b[63], %b[64], %g23, %b[64] } { loop_mode pfmuls,0,sm %b[105], %g18, %b[114] pfmuls,1,sm %b[114], %g17, %g19 pfmul_hadds,2,sm %b[54], %b[60], %g19, %b[60] pfmuls,4,sm %b[27], %b[76], %b[105] } { loop_mode pfsub_rsubs,0,sm %b[57], %b[62], %b[116], %b[62] pfsubs,1,sm %b[84], %b[75], %b[116] pfmuls,2,sm %b[51], %b[85], %g23 pfmuls,5,sm %b[109], %b[67], %b[109] } { loop_mode pfmuls,0,sm %b[88], %b[68], %b[88] pfmuls,1,sm %b[108], %b[79], %b[93] std,2 %r23, %b[6], %b[103] pfmuls,3,sm %b[26], %b[72], %b[103] addd,4,sm 0x8, %b[6], %b[4] ? %pcnt0 std,5 %r19, %b[6], %b[93] } { loop_mode pfmul_hadds,0,sm %b[96], %b[89], %g24, %b[87] pfmul_hadds,1,sm %b[87], %b[112], %b[113], %b[89] std,2 %r2, %b[6], %b[102] pfmuls,3,sm %b[50], %b[74], %b[90] std,5 %r20, %b[6], %b[90] } { loop_mode pfmul_hadds,0,sm %b[91], %g18, %b[114], %b[80] pfmul_hadds,1,sm %b[80], %g17, %g19, %b[91] std,2 %r11, %b[6], %b[98] pfmuls,3,sm %b[14], %b[94], %b[96] pfmuls,4,sm %b[35], %b[97], %b[98] std,5 %r22, %b[6], %b[99] } { loop_mode pfmul_hadds,0,sm %b[42], %b[85], %g23, %b[76] pfmuls,1,sm %b[47], %b[104], %b[99] std,2 %r16, %b[6], %b[117] pfmul_hadds,3,sm %b[19], %b[76], %b[105], %b[85] pfmuls,4,sm %b[18], %b[118], %b[102] std,5 %r24, %b[6], %b[119] } { loop_mode pfadds,0,sm %b[84], %b[75], %b[75] pfmuls,1,sm %b[23], %b[115], %b[84] std,2 %r14, %b[6], %b[107] pfmul_hadds,3,sm %b[22], %b[72], %b[103], %b[72] pfmuls,4,sm %b[43], %b[66], %b[86] std,5 %r13, %b[6], %b[86] } { loop_mode std,2 %r21, %b[6], %g16 pshufb,3,sm 0x0, %b[116], %r25, %b[105] pfmuls,4,sm %b[15], %b[95], %b[103] std,5 %r18, %b[6], %g21 } { loop_mode pfmul_hadds,0,sm %b[81], %b[68], %b[88], %b[68] pfmul_hadds,1,sm %b[73], %b[79], %b[93], %b[73] std,2 %r17, %b[6], %g20 pfmul_hadds,3,sm %b[46], %b[74], %b[90], %b[79] pfmul_hadds,4,sm %b[34], %b[97], %b[98], %b[74] std,5 %r12, %b[6], %g22 } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmul_hadds,0,sm %b[83], %b[67], %b[109], %b[64] pfmuls,1,sm %b[39], %b[62], %b[83] std,2 %r15, %b[6], %b[110] pfmul_hadds,3,sm %b[10], %b[94], %b[96], %b[67] pfmul_hadds,4,sm %b[11], %b[118], %b[102], %b[81] std,5 %r0, %b[6], %b[64] }
Теоретическая скорость: 16 комплексных чисел за 28 тактов (16/28) = 4.57 Байт/такт
Четверная теоретическая скорость: 18.29 Байт/такт
Замеры скорости
2. stage_radix4_readConjSwap_2x_simd128
Здесь происходит ручная раскрутка алгоритма stage_radix4_readConjSwap_simd128 в 2 раза.
Код на Си
void stage_radix4_readConjSwap_2x_simd128(int data_count, myComplex *data_in, myComplex *data_out, myComplex *conj_coefC_a, myComplex *conj_coefD_a, myComplex *conj_coefE_a, myComplex *conj_coefC_b, myComplex *conj_coefD_b, myComplex *conj_coefE_b, myComplex *swap_coefC_a, myComplex *swap_coefD_a, myComplex *swap_coefE_a, myComplex *swap_coefC_b, myComplex *swap_coefD_b, myComplex *swap_coefE_b) { __v2di *xy0_in = (__v2di*)&data_in[ 0]; __v2di *zw0_in = (__v2di*)&data_in[ 2]; __v2di *xy1_in = (__v2di*)&data_in[ 4]; __v2di *zw1_in = (__v2di*)&data_in[ 6]; __v2di *xy2_in = (__v2di*)&data_in[ 8]; __v2di *zw2_in = (__v2di*)&data_in[10]; __v2di *xy3_in = (__v2di*)&data_in[12]; __v2di *zw3_in = (__v2di*)&data_in[14]; __v2di *xy4_in = (__v2di*)&data_in[16]; __v2di *zw4_in = (__v2di*)&data_in[18]; __v2di *xy5_in = (__v2di*)&data_in[20]; __v2di *zw5_in = (__v2di*)&data_in[22]; __v2di *xy6_in = (__v2di*)&data_in[24]; __v2di *zw6_in = (__v2di*)&data_in[26]; __v2di *xy7_in = (__v2di*)&data_in[28]; __v2di *zw7_in = (__v2di*)&data_in[30]; __v2di *conj_c0a_in = (__v2di*)&conj_coefC_a[0]; __v2di *conj_c1a_in = (__v2di*)&conj_coefC_a[2]; __v2di *conj_c2a_in = (__v2di*)&conj_coefC_a[4]; __v2di *conj_c3a_in = (__v2di*)&conj_coefC_a[6]; __v2di *conj_d0a_in = (__v2di*)&conj_coefD_a[0]; __v2di *conj_d1a_in = (__v2di*)&conj_coefD_a[2]; __v2di *conj_d2a_in = (__v2di*)&conj_coefD_a[4]; __v2di *conj_d3a_in = (__v2di*)&conj_coefD_a[6]; __v2di *conj_e0a_in = (__v2di*)&conj_coefE_a[0]; __v2di *conj_e1a_in = (__v2di*)&conj_coefE_a[2]; __v2di *conj_e2a_in = (__v2di*)&conj_coefE_a[4]; __v2di *conj_e3a_in = (__v2di*)&conj_coefE_a[6]; __v2di *conj_c0b_in = (__v2di*)&conj_coefC_b[0*data_count/16]; __v2di *conj_c1b_in = (__v2di*)&conj_coefC_b[1*data_count/16]; __v2di *conj_c2b_in = (__v2di*)&conj_coefC_b[2*data_count/16]; __v2di *conj_c3b_in = (__v2di*)&conj_coefC_b[3*data_count/16]; __v2di *conj_d0b_in = (__v2di*)&conj_coefD_b[0*data_count/16]; __v2di *conj_d1b_in = (__v2di*)&conj_coefD_b[1*data_count/16]; __v2di *conj_d2b_in = (__v2di*)&conj_coefD_b[2*data_count/16]; __v2di *conj_d3b_in = (__v2di*)&conj_coefD_b[3*data_count/16]; __v2di *conj_e0b_in = (__v2di*)&conj_coefE_b[0*data_count/16]; __v2di *conj_e1b_in = (__v2di*)&conj_coefE_b[1*data_count/16]; __v2di *conj_e2b_in = (__v2di*)&conj_coefE_b[2*data_count/16]; __v2di *conj_e3b_in = (__v2di*)&conj_coefE_b[3*data_count/16]; __v2di *swap_c0a_in = (__v2di*)&swap_coefC_a[0]; __v2di *swap_c1a_in = (__v2di*)&swap_coefC_a[2]; __v2di *swap_c2a_in = (__v2di*)&swap_coefC_a[4]; __v2di *swap_c3a_in = (__v2di*)&swap_coefC_a[6]; __v2di *swap_d0a_in = (__v2di*)&swap_coefD_a[0]; __v2di *swap_d1a_in = (__v2di*)&swap_coefD_a[2]; __v2di *swap_d2a_in = (__v2di*)&swap_coefD_a[4]; __v2di *swap_d3a_in = (__v2di*)&swap_coefD_a[6]; __v2di *swap_e0a_in = (__v2di*)&swap_coefE_a[0]; __v2di *swap_e1a_in = (__v2di*)&swap_coefE_a[2]; __v2di *swap_e2a_in = (__v2di*)&swap_coefE_a[4]; __v2di *swap_e3a_in = (__v2di*)&swap_coefE_a[6]; __v2di *swap_c0b_in = (__v2di*)&swap_coefC_b[0*data_count/16]; __v2di *swap_c1b_in = (__v2di*)&swap_coefC_b[1*data_count/16]; __v2di *swap_c2b_in = (__v2di*)&swap_coefC_b[2*data_count/16]; __v2di *swap_c3b_in = (__v2di*)&swap_coefC_b[3*data_count/16]; __v2di *swap_d0b_in = (__v2di*)&swap_coefD_b[0*data_count/16]; __v2di *swap_d1b_in = (__v2di*)&swap_coefD_b[1*data_count/16]; __v2di *swap_d2b_in = (__v2di*)&swap_coefD_b[2*data_count/16]; __v2di *swap_d3b_in = (__v2di*)&swap_coefD_b[3*data_count/16]; __v2di *swap_e0b_in = (__v2di*)&swap_coefE_b[0*data_count/16]; __v2di *swap_e1b_in = (__v2di*)&swap_coefE_b[1*data_count/16]; __v2di *swap_e2b_in = (__v2di*)&swap_coefE_b[2*data_count/16]; __v2di *swap_e3b_in = (__v2di*)&swap_coefE_b[3*data_count/16]; __v2di *out_0 = (__v2di*)&data_out[ 0*data_count/16]; __v2di *out_1 = (__v2di*)&data_out[ 1*data_count/16]; __v2di *out_2 = (__v2di*)&data_out[ 2*data_count/16]; __v2di *out_3 = (__v2di*)&data_out[ 3*data_count/16]; __v2di *out_4 = (__v2di*)&data_out[ 4*data_count/16]; __v2di *out_5 = (__v2di*)&data_out[ 5*data_count/16]; __v2di *out_6 = (__v2di*)&data_out[ 6*data_count/16]; __v2di *out_7 = (__v2di*)&data_out[ 7*data_count/16]; __v2di *out_8 = (__v2di*)&data_out[ 8*data_count/16]; __v2di *out_9 = (__v2di*)&data_out[ 9*data_count/16]; __v2di *out_10 = (__v2di*)&data_out[10*data_count/16]; __v2di *out_11 = (__v2di*)&data_out[11*data_count/16]; __v2di *out_12 = (__v2di*)&data_out[12*data_count/16]; __v2di *out_13 = (__v2di*)&data_out[13*data_count/16]; __v2di *out_14 = (__v2di*)&data_out[14*data_count/16]; __v2di *out_15 = (__v2di*)&data_out[15*data_count/16]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/32; ++i) { __v2di xy0 = xy0_in[16*i]; __v2di zw0 = zw0_in[16*i]; __v2di xy1 = xy1_in[16*i]; __v2di zw1 = zw1_in[16*i]; __v2di conj_c0 = conj_c0a_in[4*i]; __v2di conj_d0 = conj_d0a_in[4*i]; __v2di conj_e0 = conj_e0a_in[4*i]; __v2di swap_c0 = swap_c0a_in[4*i]; __v2di swap_d0 = swap_d0a_in[4*i]; __v2di swap_e0 = swap_e0a_in[4*i]; __v2di xy2 = xy2_in[16*i]; __v2di zw2 = zw2_in[16*i]; __v2di xy3 = xy3_in[16*i]; __v2di zw3 = zw3_in[16*i]; __v2di conj_c1 = conj_c1a_in[4*i]; __v2di conj_d1 = conj_d1a_in[4*i]; __v2di conj_e1 = conj_e1a_in[4*i]; __v2di swap_c1 = swap_c1a_in[4*i]; __v2di swap_d1 = swap_d1a_in[4*i]; __v2di swap_e1 = swap_e1a_in[4*i]; __v2di xy4 = xy4_in[16*i]; __v2di zw4 = zw4_in[16*i]; __v2di xy5 = xy5_in[16*i]; __v2di zw5 = zw5_in[16*i]; __v2di conj_c2 = conj_c2a_in[4*i]; __v2di conj_d2 = conj_d2a_in[4*i]; __v2di conj_e2 = conj_e2a_in[4*i]; __v2di swap_c2 = swap_c2a_in[4*i]; __v2di swap_d2 = swap_d2a_in[4*i]; __v2di swap_e2 = swap_e2a_in[4*i]; __v2di xy6 = xy6_in[16*i]; __v2di zw6 = zw6_in[16*i]; __v2di xy7 = xy7_in[16*i]; __v2di zw7 = zw7_in[16*i]; __v2di conj_c3 = conj_c3a_in[4*i]; __v2di conj_d3 = conj_d3a_in[4*i]; __v2di conj_e3 = conj_e3a_in[4*i]; __v2di swap_c3 = swap_c3a_in[4*i]; __v2di swap_d3 = swap_d3a_in[4*i]; __v2di swap_e3 = swap_e3a_in[4*i]; __v2di x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di y3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di z3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di w3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di cy0_real = __builtin_e2k_qpfmuls(conj_c0, y0); __v2di cy1_real = __builtin_e2k_qpfmuls(conj_c1, y1); __v2di cy2_real = __builtin_e2k_qpfmuls(conj_c2, y2); __v2di cy3_real = __builtin_e2k_qpfmuls(conj_c3, y3); __v2di dz0_real = __builtin_e2k_qpfmuls(conj_d0, z0); __v2di dz1_real = __builtin_e2k_qpfmuls(conj_d1, z1); __v2di dz2_real = __builtin_e2k_qpfmuls(conj_d2, z2); __v2di dz3_real = __builtin_e2k_qpfmuls(conj_d3, z3); __v2di ew0_real = __builtin_e2k_qpfmuls(conj_e0, w0); __v2di ew1_real = __builtin_e2k_qpfmuls(conj_e1, w1); __v2di ew2_real = __builtin_e2k_qpfmuls(conj_e2, w2); __v2di ew3_real = __builtin_e2k_qpfmuls(conj_e3, w3); __v2di cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); __v2di cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); __v2di cy2_imag = __builtin_e2k_qpfmuls(swap_c2, y2); __v2di cy3_imag = __builtin_e2k_qpfmuls(swap_c3, y3); __v2di dz0_imag = __builtin_e2k_qpfmuls(swap_d0, z0); __v2di dz1_imag = __builtin_e2k_qpfmuls(swap_d1, z1); __v2di dz2_imag = __builtin_e2k_qpfmuls(swap_d2, z2); __v2di dz3_imag = __builtin_e2k_qpfmuls(swap_d3, z3); __v2di ew0_imag = __builtin_e2k_qpfmuls(swap_e0, w0); __v2di ew1_imag = __builtin_e2k_qpfmuls(swap_e1, w1); __v2di ew2_imag = __builtin_e2k_qpfmuls(swap_e2, w2); __v2di ew3_imag = __builtin_e2k_qpfmuls(swap_e3, w3); __v2di cy0_rrii = __builtin_e2k_qpfhadds(cy0_real, cy0_imag); __v2di cy1_rrii = __builtin_e2k_qpfhadds(cy1_real, cy1_imag); __v2di cy2_rrii = __builtin_e2k_qpfhadds(cy2_real, cy2_imag); __v2di cy3_rrii = __builtin_e2k_qpfhadds(cy3_real, cy3_imag); __v2di dz0_rrii = __builtin_e2k_qpfhadds(dz0_real, dz0_imag); __v2di dz1_rrii = __builtin_e2k_qpfhadds(dz1_real, dz1_imag); __v2di dz2_rrii = __builtin_e2k_qpfhadds(dz2_real, dz2_imag); __v2di dz3_rrii = __builtin_e2k_qpfhadds(dz3_real, dz3_imag); __v2di ew0_rrii = __builtin_e2k_qpfhadds(ew0_real, ew0_imag); __v2di ew1_rrii = __builtin_e2k_qpfhadds(ew1_real, ew1_imag); __v2di ew2_rrii = __builtin_e2k_qpfhadds(ew2_real, ew2_imag); __v2di ew3_rrii = __builtin_e2k_qpfhadds(ew3_real, ew3_imag); __v2di cy0 = __builtin_e2k_qpshufb(cy0_rrii, cy0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di cy1 = __builtin_e2k_qpshufb(cy1_rrii, cy1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di cy2 = __builtin_e2k_qpshufb(cy2_rrii, cy2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di cy3 = __builtin_e2k_qpshufb(cy3_rrii, cy3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di dz0 = __builtin_e2k_qpshufb(dz0_rrii, dz0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di dz1 = __builtin_e2k_qpshufb(dz1_rrii, dz1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di dz2 = __builtin_e2k_qpshufb(dz2_rrii, dz2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di dz3 = __builtin_e2k_qpshufb(dz3_rrii, dz3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di ew0 = __builtin_e2k_qpshufb(ew0_rrii, ew0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di ew1 = __builtin_e2k_qpshufb(ew1_rrii, ew1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di ew2 = __builtin_e2k_qpshufb(ew2_rrii, ew2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di ew3 = __builtin_e2k_qpshufb(ew3_rrii, ew3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); __v2di add02_0 = __builtin_e2k_qpfadds( x0, dz0); __v2di add02_1 = __builtin_e2k_qpfadds( x1, dz1); __v2di add02_2 = __builtin_e2k_qpfadds( x2, dz2); __v2di add02_3 = __builtin_e2k_qpfadds( x3, dz3); __v2di sub02_0 = __builtin_e2k_qpfsubs( x0, dz0); __v2di sub02_1 = __builtin_e2k_qpfsubs( x1, dz1); __v2di sub02_2 = __builtin_e2k_qpfsubs( x2, dz2); __v2di sub02_3 = __builtin_e2k_qpfsubs( x3, dz3); __v2di add13_0 = __builtin_e2k_qpfadds(cy0, ew0); __v2di add13_1 = __builtin_e2k_qpfadds(cy1, ew1); __v2di add13_2 = __builtin_e2k_qpfadds(cy2, ew2); __v2di add13_3 = __builtin_e2k_qpfadds(cy3, ew3); __v2di sub13_0 = __builtin_e2k_qpfsubs(cy0, ew0); __v2di sub13_1 = __builtin_e2k_qpfsubs(cy1, ew1); __v2di sub13_2 = __builtin_e2k_qpfsubs(cy2, ew2); __v2di sub13_3 = __builtin_e2k_qpfsubs(cy3, ew3); __v2di swap_sub13_0 = __builtin_e2k_qpshufb(sub13_0, sub13_0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_1 = __builtin_e2k_qpshufb(sub13_1, sub13_1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_2 = __builtin_e2k_qpshufb(sub13_2, sub13_2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13_3 = __builtin_e2k_qpshufb(sub13_3, sub13_3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di sub13i_0 = __builtin_e2k_qpxor(swap_sub13_0, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_1 = __builtin_e2k_qpxor(swap_sub13_1, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_2 = __builtin_e2k_qpxor(swap_sub13_2, (__v2di){1LL<<31, 1LL<<31}); __v2di sub13i_3 = __builtin_e2k_qpxor(swap_sub13_3, (__v2di){1LL<<31, 1LL<<31}); __v2di out0 = __builtin_e2k_qpfadds(add02_0, add13_0); __v2di out1 = __builtin_e2k_qpfadds(add02_1, add13_1); __v2di out2 = __builtin_e2k_qpfadds(add02_2, add13_2); __v2di out3 = __builtin_e2k_qpfadds(add02_3, add13_3); __v2di out4 = __builtin_e2k_qpfsubs(sub02_0, sub13i_0); __v2di out5 = __builtin_e2k_qpfsubs(sub02_1, sub13i_1); __v2di out6 = __builtin_e2k_qpfsubs(sub02_2, sub13i_2); __v2di out7 = __builtin_e2k_qpfsubs(sub02_3, sub13i_3); __v2di out8 = __builtin_e2k_qpfsubs(add02_0, add13_0); __v2di out9 = __builtin_e2k_qpfsubs(add02_1, add13_1); __v2di out10 = __builtin_e2k_qpfsubs(add02_2, add13_2); __v2di out11 = __builtin_e2k_qpfsubs(add02_3, add13_3); __v2di out12 = __builtin_e2k_qpfadds(sub02_0, sub13i_0); __v2di out13 = __builtin_e2k_qpfadds(sub02_1, sub13i_1); __v2di out14 = __builtin_e2k_qpfadds(sub02_2, sub13i_2); __v2di out15 = __builtin_e2k_qpfadds(sub02_3, sub13i_3); xy0 = out0; zw0 = out1; xy1 = out2; zw1 = out3; conj_c0 = conj_c0b_in[i]; conj_d0 = conj_d0b_in[i]; conj_e0 = conj_e0b_in[i]; swap_c0 = swap_c0b_in[i]; swap_d0 = swap_d0b_in[i]; swap_e0 = swap_e0b_in[i]; xy2 = out4; zw2 = out5; xy3 = out6; zw3 = out7; conj_c1 = conj_c1b_in[i]; conj_d1 = conj_d1b_in[i]; conj_e1 = conj_e1b_in[i]; swap_c1 = swap_c1b_in[i]; swap_d1 = swap_d1b_in[i]; swap_e1 = swap_e1b_in[i]; xy4 = out8; zw4 = out9; xy5 = out10; zw5 = out11; conj_c2 = conj_c2b_in[i]; conj_d2 = conj_d2b_in[i]; conj_e2 = conj_e2b_in[i]; swap_c2 = swap_c2b_in[i]; swap_d2 = swap_d2b_in[i]; swap_e2 = swap_e2b_in[i]; xy6 = out12; zw6 = out13; xy7 = out14; zw7 = out15; conj_c3 = conj_c3b_in[i]; conj_d3 = conj_d3b_in[i]; conj_e3 = conj_e3b_in[i]; swap_c3 = swap_c3b_in[i]; swap_d3 = swap_d3b_in[i]; swap_e3 = swap_e3b_in[i]; x0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0706050403020100, 0x0706050403020100}); y0 = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0706050403020100, 0x0706050403020100}); w0 = __builtin_e2k_qpshufb(zw1, zw0, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0706050403020100, 0x0706050403020100}); y1 = __builtin_e2k_qpshufb(xy3, xy2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0706050403020100, 0x0706050403020100}); w1 = __builtin_e2k_qpshufb(zw3, zw2, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0706050403020100, 0x0706050403020100}); y2 = __builtin_e2k_qpshufb(xy5, xy4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0706050403020100, 0x0706050403020100}); w2 = __builtin_e2k_qpshufb(zw5, zw4, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); x3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0706050403020100, 0x0706050403020100}); y3 = __builtin_e2k_qpshufb(xy7, xy6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); z3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0706050403020100, 0x0706050403020100}); w3 = __builtin_e2k_qpshufb(zw7, zw6, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); cy0_real = __builtin_e2k_qpfmuls(conj_c0, y0); cy1_real = __builtin_e2k_qpfmuls(conj_c1, y1); cy2_real = __builtin_e2k_qpfmuls(conj_c2, y2); cy3_real = __builtin_e2k_qpfmuls(conj_c3, y3); dz0_real = __builtin_e2k_qpfmuls(conj_d0, z0); dz1_real = __builtin_e2k_qpfmuls(conj_d1, z1); dz2_real = __builtin_e2k_qpfmuls(conj_d2, z2); dz3_real = __builtin_e2k_qpfmuls(conj_d3, z3); ew0_real = __builtin_e2k_qpfmuls(conj_e0, w0); ew1_real = __builtin_e2k_qpfmuls(conj_e1, w1); ew2_real = __builtin_e2k_qpfmuls(conj_e2, w2); ew3_real = __builtin_e2k_qpfmuls(conj_e3, w3); cy0_imag = __builtin_e2k_qpfmuls(swap_c0, y0); cy1_imag = __builtin_e2k_qpfmuls(swap_c1, y1); cy2_imag = __builtin_e2k_qpfmuls(swap_c2, y2); cy3_imag = __builtin_e2k_qpfmuls(swap_c3, y3); dz0_imag = __builtin_e2k_qpfmuls(swap_d0, z0); dz1_imag = __builtin_e2k_qpfmuls(swap_d1, z1); dz2_imag = __builtin_e2k_qpfmuls(swap_d2, z2); dz3_imag = __builtin_e2k_qpfmuls(swap_d3, z3); ew0_imag = __builtin_e2k_qpfmuls(swap_e0, w0); ew1_imag = __builtin_e2k_qpfmuls(swap_e1, w1); ew2_imag = __builtin_e2k_qpfmuls(swap_e2, w2); ew3_imag = __builtin_e2k_qpfmuls(swap_e3, w3); cy0_rrii = __builtin_e2k_qpfhadds(cy0_real, cy0_imag); cy1_rrii = __builtin_e2k_qpfhadds(cy1_real, cy1_imag); cy2_rrii = __builtin_e2k_qpfhadds(cy2_real, cy2_imag); cy3_rrii = __builtin_e2k_qpfhadds(cy3_real, cy3_imag); dz0_rrii = __builtin_e2k_qpfhadds(dz0_real, dz0_imag); dz1_rrii = __builtin_e2k_qpfhadds(dz1_real, dz1_imag); dz2_rrii = __builtin_e2k_qpfhadds(dz2_real, dz2_imag); dz3_rrii = __builtin_e2k_qpfhadds(dz3_real, dz3_imag); ew0_rrii = __builtin_e2k_qpfhadds(ew0_real, ew0_imag); ew1_rrii = __builtin_e2k_qpfhadds(ew1_real, ew1_imag); ew2_rrii = __builtin_e2k_qpfhadds(ew2_real, ew2_imag); ew3_rrii = __builtin_e2k_qpfhadds(ew3_real, ew3_imag); cy0 = __builtin_e2k_qpshufb(cy0_rrii, cy0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); cy1 = __builtin_e2k_qpshufb(cy1_rrii, cy1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); cy2 = __builtin_e2k_qpshufb(cy2_rrii, cy2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); cy3 = __builtin_e2k_qpshufb(cy3_rrii, cy3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); dz0 = __builtin_e2k_qpshufb(dz0_rrii, dz0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); dz1 = __builtin_e2k_qpshufb(dz1_rrii, dz1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); dz2 = __builtin_e2k_qpshufb(dz2_rrii, dz2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); dz3 = __builtin_e2k_qpshufb(dz3_rrii, dz3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); ew0 = __builtin_e2k_qpshufb(ew0_rrii, ew0_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); ew1 = __builtin_e2k_qpshufb(ew1_rrii, ew1_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); ew2 = __builtin_e2k_qpshufb(ew2_rrii, ew2_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); ew3 = __builtin_e2k_qpshufb(ew3_rrii, ew3_rrii, (__v2di){0x0B0A090803020100, 0x0F0E0D0C07060504}); add02_0 = __builtin_e2k_qpfadds( x0, dz0); add02_1 = __builtin_e2k_qpfadds( x1, dz1); add02_2 = __builtin_e2k_qpfadds( x2, dz2); add02_3 = __builtin_e2k_qpfadds( x3, dz3); sub02_0 = __builtin_e2k_qpfsubs( x0, dz0); sub02_1 = __builtin_e2k_qpfsubs( x1, dz1); sub02_2 = __builtin_e2k_qpfsubs( x2, dz2); sub02_3 = __builtin_e2k_qpfsubs( x3, dz3); add13_0 = __builtin_e2k_qpfadds(cy0, ew0); add13_1 = __builtin_e2k_qpfadds(cy1, ew1); add13_2 = __builtin_e2k_qpfadds(cy2, ew2); add13_3 = __builtin_e2k_qpfadds(cy3, ew3); sub13_0 = __builtin_e2k_qpfsubs(cy0, ew0); sub13_1 = __builtin_e2k_qpfsubs(cy1, ew1); sub13_2 = __builtin_e2k_qpfsubs(cy2, ew2); sub13_3 = __builtin_e2k_qpfsubs(cy3, ew3); swap_sub13_0 = __builtin_e2k_qpshufb(sub13_0, sub13_0, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_1 = __builtin_e2k_qpshufb(sub13_1, sub13_1, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_2 = __builtin_e2k_qpshufb(sub13_2, sub13_2, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); swap_sub13_3 = __builtin_e2k_qpshufb(sub13_3, sub13_3, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); sub13i_0 = __builtin_e2k_qpxor(swap_sub13_0, (__v2di){1LL<<31, 1LL<<31}); sub13i_1 = __builtin_e2k_qpxor(swap_sub13_1, (__v2di){1LL<<31, 1LL<<31}); sub13i_2 = __builtin_e2k_qpxor(swap_sub13_2, (__v2di){1LL<<31, 1LL<<31}); sub13i_3 = __builtin_e2k_qpxor(swap_sub13_3, (__v2di){1LL<<31, 1LL<<31}); out_0[i] = __builtin_e2k_qpfadds(add02_0, add13_0); out_1[i] = __builtin_e2k_qpfadds(add02_1, add13_1); out_2[i] = __builtin_e2k_qpfadds(add02_2, add13_2); out_3[i] = __builtin_e2k_qpfadds(add02_3, add13_3); out_4[i] = __builtin_e2k_qpfsubs(sub02_0, sub13i_0); out_5[i] = __builtin_e2k_qpfsubs(sub02_1, sub13i_1); out_6[i] = __builtin_e2k_qpfsubs(sub02_2, sub13i_2); out_7[i] = __builtin_e2k_qpfsubs(sub02_3, sub13i_3); out_8[i] = __builtin_e2k_qpfsubs(add02_0, add13_0); out_9[i] = __builtin_e2k_qpfsubs(add02_1, add13_1); out_10[i] = __builtin_e2k_qpfsubs(add02_2, add13_2); out_11[i] = __builtin_e2k_qpfsubs(add02_3, add13_3); out_12[i] = __builtin_e2k_qpfadds(sub02_0, sub13i_0); out_13[i] = __builtin_e2k_qpfadds(sub02_1, sub13i_1); out_14[i] = __builtin_e2k_qpfadds(sub02_2, sub13i_2); out_15[i] = __builtin_e2k_qpfadds(sub02_3, sub13i_3); } }
Основной цикл на ассемблере
.L6737: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=0, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=0, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=0, abs=1, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=0, abs=1, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=0, abs=2, disp=128 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=0, abs=2, disp=160 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=0, abs=3, disp=192 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=1, asz=0, abs=3, disp=224 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=0, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=0, abs=4, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=0, abs=5, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=0, abs=5, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=0, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=0, abs=6, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=0, abs=7, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=0, abs=7, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=0, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=0, abs=8, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=0, abs=9, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=0, abs=9, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=16, incr=0, ind=0, asz=0, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=15, incr=0, ind=0, asz=0, abs=10, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=14, incr=0, ind=0, asz=0, abs=11, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=13, incr=0, ind=0, asz=0, abs=11, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=12, incr=0, ind=0, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=11, incr=0, ind=0, asz=1, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=10, incr=0, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=9, incr=0, ind=0, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=8, incr=0, ind=0, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=7, incr=0, ind=0, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=6, incr=0, ind=0, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=5, incr=0, ind=0, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=4, incr=0, ind=0, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=3, incr=0, ind=0, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=2, incr=0, ind=0, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=1, abs=30, disp=0 } .L2988: { loop_mode qpfmul_hadds,0,sm %b[92], %b[83], %b[98], %b[83] qpshufb,1,sm %b[69], %b[69], %r4, %b[69] qpshufb,3,sm %b[63], %b[66], %r0, %b[92] qpfmuls,4,sm %b[48], %b[90], %b[98] qpfmuls,5,sm %b[21], %b[86], %b[99] } { loop_mode qpshufb,1,sm %b[62], %b[62], %r4, %b[74] qpshufb,3,sm %b[72], %b[74], %r3, %b[62] qpfmuls,5,sm %b[29], %b[92], %b[72] } { loop_mode qpshufb,1,sm %b[97], %b[97], %r4, %b[97] qpfmul_hadds,2,sm %b[32], %b[87], %b[102], %b[87] qpshufb,3,sm %b[115], %b[118], %r3, %b[100] qpfmuls,5,sm %b[45], %b[62], %b[103] } { loop_mode qpfmul_hadds,2,sm %b[44], %b[101], %b[93], %b[71] qpshufb,3,sm %b[71], %b[76], %r3, %b[76] qpfsubs,4,sm %b[79], %b[77], %b[93] qpfmuls,5,sm %b[28], %b[100], %b[101] } { loop_mode qpfmul_hadds,0,sm %b[49], %b[95], %b[84], %b[84] qpshufb,1,sm %b[96], %b[96], %r4, %b[73] qpfmul_hadds,2,sm %b[25], %b[91], %b[94], %b[89] qpshufb,3,sm %b[73], %b[89], %r3, %b[91] qpfsubs,4,sm %b[69], %b[70], %b[94] qpfmuls,5,sm %b[13], %b[76], %b[95] } { loop_mode qpshufb,1,sm %b[108], %b[109], %r3, %b[80] qpfmul_hadds,2,sm %b[41], %b[85], %b[88], %b[85] qpshufb,3,sm %b[80], %b[80], %r4, %b[102] qpfsubs,4,sm %b[74], %b[81], %b[96] qpfmuls,5,sm %b[24], %b[91], %b[88] } { loop_mode qpfmul_hadds,0,sm %b[36], %b[90], %b[98], %b[81] qpshufb,1,sm %b[82], %b[82], %r4, %b[64] qpfmul_hadds,2,sm %b[17], %b[86], %b[99], %b[82] qpshufb,3,sm %b[64], %b[67], %r3, %b[67] qpfadds,4,sm %b[74], %b[81], %b[74] qpfsubs,5,sm %b[102], %b[97], %b[86] movaqp,0 area=21, ind=0, am=1, be=0, %b[1] movaqp,1 area=20, ind=0, am=1, be=0, %b[17] movaqp,2 area=21, ind=0, am=1, be=0, %b[13] movaqp,3 area=20, ind=0, am=1, be=0, %b[8] } { loop_mode qpshufb,1,sm %b[75], %b[75], %r4, %b[72] qpfmul_hadds,2,sm %b[12], %b[92], %b[72], %b[77] qpshufb,3,sm %b[78], %b[105], %r3, %b[75] qpfadds,5,sm %b[79], %b[77], %b[78] movaqp,0 area=19, ind=0, am=1, be=0, %b[25] movaqp,1 area=18, ind=0, am=1, be=0, %b[12] movaqp,2 area=19, ind=0, am=1, be=0, %b[24] movaqp,3 area=18, ind=0, am=1, be=0, %b[21] } { loop_mode qpshufb,1,sm %b[83], %b[83], %r4, %b[62] qpfmul_hadds,2,sm %b[9], %b[62], %b[103], %b[69] qpshufb,3,sm %b[65], %b[68], %r3, %b[65] qpfadds,5,sm %b[69], %b[70], %b[68] movaqp,0 area=17, ind=0, am=1, be=0, %b[29] movaqp,1 area=16, ind=0, am=1, be=0, %b[33] movaqp,2 area=17, ind=0, am=1, be=0, %b[28] movaqp,3 area=16, ind=0, am=1, be=0, %b[9] } { loop_mode qpshufb,1,sm %b[93], %b[93], %r5, %b[83] qpfmul_hadds,2,sm %b[5], %b[100], %b[101], %b[70] qpshufb,3,sm %b[96], %b[96], %r5, %b[90] qpfadds,5,sm %b[102], %b[97], %b[79] movaqp,0 area=15, ind=0, am=1, be=0, %b[5] movaqp,1 area=14, ind=0, am=1, be=0, %b[36] movaqp,2 area=15, ind=0, am=1, be=0, %b[37] movaqp,3 area=14, ind=0, am=1, be=0, %b[32] } { loop_mode qpshufb,1,sm %b[94], %b[94], %r5, %b[92] qpfmul_hadds,2,sm %b[16], %b[76], %b[95], %b[76] qpshufb,3,sm %b[86], %b[86], %r5, %b[86] movaqp,0 area=13, ind=0, am=1, be=0, %b[44] movaqp,1 area=12, ind=0, am=1, be=0, %b[16] movaqp,2 area=13, ind=0, am=1, be=0, %b[41] movaqp,3 area=12, ind=0, am=1, be=0, %b[40] } { loop_mode qpxor,1,sm %b[92], %r1, %b[88] qpfmul_hadds,2,sm %b[20], %b[91], %b[88], %b[91] qpxor,3,sm %b[86], %r1, %b[86] movaqp,0 area=11, ind=0, am=1, be=0, %b[48] movaqp,1 area=10, ind=0, am=1, be=0, %b[49] movaqp,2 area=11, ind=0, am=1, be=0, %b[45] movaqp,3 area=10, ind=0, am=1, be=0, %b[20] } { loop_mode qpfadd_adds,0,sm %b[65], %b[62], %b[74], %b[87] qpshufb,1,sm %b[89], %b[89], %r4, %b[95] qpshufb,3,sm %b[87], %b[87], %r4, %b[96] movaqp,0 area=9, ind=0, am=0, be=0, %b[93] movaqp,1 area=9, ind=16, am=1, be=0, %b[94] movaqp,2 area=9, ind=0, am=0, be=0, %b[89] movaqp,3 area=9, ind=16, am=1, be=0, %b[92] } { loop_mode qpfadd_adds,0,sm %b[75], %b[72], %b[78], %b[71] qpxor,1,sm %b[90], %r1, %b[100] qpshufb,3,sm %b[71], %b[71], %r4, %b[101] movaqp,0 area=8, ind=16, am=1, be=0, %b[90] movaqp,1 area=8, ind=0, am=0, be=0, %b[98] movaqp,2 area=8, ind=16, am=1, be=0, %b[97] movaqp,3 area=8, ind=0, am=0, be=0, %b[99] } { loop_mode qpfadd_rsubs,0,sm %b[67], %b[64], %b[68], %b[52] qpshufb,1,sm %b[82], %b[82], %r4, %b[103] qpshufb,3,sm %b[84], %b[84], %r4, %b[106] movaqp,0 area=7, ind=16, am=1, be=0, %b[102] movaqp,1 area=7, ind=0, am=0, be=0, %b[104] movaqp,2 area=7, ind=0, am=0, be=0, %b[82] movaqp,3 area=7, ind=16, am=1, be=0, %b[84] } { loop_mode qpfadd_rsubs,0,sm %b[80], %b[73], %b[79], %b[53] qpxor,1,sm %b[83], %r1, %b[108] qpshufb,3,sm %b[85], %b[85], %r4, %b[110] qpfadds,4,sm %b[106], %b[101], %b[106] qpfsubs,5,sm %b[106], %b[101], %b[107] movaqp,0 area=6, ind=0, am=0, be=0, %b[101] movaqp,1 area=6, ind=16, am=1, be=0, %b[105] movaqp,2 area=6, ind=0, am=0, be=0, %b[83] movaqp,3 area=6, ind=16, am=1, be=0, %b[85] } { loop_mode qpfadd_rsubs,0,sm %b[65], %b[62], %b[74], %b[74] qpshufb,1,sm %b[69], %b[69], %r4, %b[109] qpfadd_rsubs,2,sm %b[75], %b[72], %b[78], %b[69] qpshufb,3,sm %b[81], %b[81], %r4, %b[113] qpfadds,4,sm %b[96], %b[95], %b[111] qpfsubs,5,sm %b[96], %b[95], %b[112] movaqp,0 area=5, ind=16, am=0, be=0, %b[78] movaqp,1 area=1, ind=16, am=0, be=0, %b[95] movaqp,2 area=5, ind=16, am=0, be=0, %b[96] movaqp,3 area=1, ind=16, am=0, be=0, %b[81] } { loop_mode qpfadd_adds,0,sm %b[67], %b[64], %b[68], %b[57] qpshufb,1,sm %b[60], %b[61], %r3, %b[79] qpfadd_adds,2,sm %b[80], %b[73], %b[79], %b[56] qpshufb,3,sm %b[77], %b[77], %r4, %b[113] qpfadds,4,sm %b[113], %b[110], %b[110] qpfsubs,5,sm %b[113], %b[110], %b[114] movaqp,0 area=5, ind=0, am=1, be=0, %b[60] movaqp,1 area=0, ind=16, am=0, be=0, %b[68] movaqp,2 area=5, ind=0, am=1, be=0, %b[77] movaqp,3 area=0, ind=16, am=0, be=0, %b[61] } { loop_mode qpfsub_adds,0,sm %b[65], %b[62], %b[100], %b[116] qpshufb,1,sm %b[76], %b[76], %r4, %b[119] qpfsub_adds,2,sm %b[75], %b[72], %b[108], %b[113] qpshufb,3,sm %b[55], %b[54], %r3, %b[118] qpfadds,4,sm %b[113], %b[103], %g17 qpfsubs,5,sm %b[113], %b[103], %g16 movaqp,0 area=3, ind=16, am=1, be=0, %b[117] movaqp,1 area=3, ind=0, am=0, be=0, %b[103] movaqp,2 area=3, ind=16, am=1, be=0, %b[115] movaqp,3 area=3, ind=0, am=0, be=0, %b[76] } { loop_mode qpfsub_rsubs,0,sm %b[65], %b[62], %b[100], %b[72] qpshufb,1,sm %b[91], %b[91], %r4, %b[108] qpfsub_rsubs,2,sm %b[75], %b[72], %b[108], %b[70] qpshufb,3,sm %b[70], %b[70], %r4, %b[75] movaqp,0 area=4, ind=16, am=0, be=0, %b[91] movaqp,1 area=0, ind=0, am=1, be=0, %b[65] movaqp,2 area=4, ind=16, am=0, be=0, %b[100] movaqp,3 area=0, ind=0, am=1, be=0, %b[62] } { loop_mode qpfsub_rsubs,0,sm %b[67], %b[64], %b[88], %b[59] qpshufb,1,sm %b[58], %b[59], %r3, %g18 qpfsub_rsubs,2,sm %b[80], %b[73], %b[86], %b[58] qpshufb,3,sm %b[63], %b[66], %r3, %g19 movaqp,0 area=4, ind=0, am=1, be=0, %g20 movaqp,1 area=1, ind=0, am=1, be=0, %b[66] movaqp,2 area=4, ind=0, am=1, be=0, %g21 movaqp,3 area=1, ind=0, am=1, be=0, %b[63] } { loop_mode qpfadd_rsubs,0,sm %g18, %b[108], %b[106], %b[112] qpshufb,1,sm %b[107], %b[107], %r5, %g22 qpfadd_adds,2,sm %g18, %b[108], %b[106], %g24 qpshufb,3,sm %b[112], %b[112], %r5, %g23 qpshufb,4,sm %b[81], %b[95], %r0, %g25 movaqp,0 area=2, ind=0, am=0, be=0, %b[107] movaqp,1 area=2, ind=16, am=1, be=0, %g26 movaqp,2 area=2, ind=0, am=0, be=0, %b[106] movaqp,3 area=2, ind=16, am=1, be=0, %g27 } { loop_mode qpfadd_rsubs,0,sm %b[118], %b[119], %b[111], %b[105] qpxor,1,sm %g22, %r1, %g22 qpfadd_adds,2,sm %b[118], %b[119], %b[111], %b[111] qpxor,3,sm %g23, %r1, %g23 qpshufb,4,sm %b[61], %b[68], %r0, %g29 qpfmuls,5,sm %b[105], %g25, %g28 } { loop_mode qpfsub_rsubs,0,sm %g18, %b[108], %g22, %b[108] qpshufb,1,sm %b[114], %b[114], %r5, %g30 qpfsub_adds,2,sm %g18, %b[108], %g22, %b[101] qpshufb,3,sm %b[115], %b[117], %r0, %b[114] qpshufb,4,sm %b[76], %b[103], %r0, %g22 qpfmuls,5,sm %b[101], %g29, %g18 } { loop_mode qpfadd_adds,0,sm %b[79], %b[109], %b[110], %b[100] qpshufb,1,sm %g16, %g16, %r5, %g16 qpfadd_adds,2,sm %g19, %b[75], %g17, %b[85] qpshufb,3,sm %b[62], %b[65], %r0, %g31 qpfmuls,4,sm %b[85], %b[114], %r7 qpfmuls,5,sm %b[100], %g22, %r6 } { loop_mode qpfadd_rsubs,0,sm %b[79], %b[109], %b[110], %g17 qpxor,1,sm %g16, %r1, %g16 qpfadd_rsubs,2,sm %g19, %b[75], %g17, %b[110] qpshufb,3,sm %b[63], %b[66], %r0, %r9 qpfmuls,5,sm %g20, %g31, %g20 } { loop_mode qpfsub_rsubs,0,sm %b[118], %b[119], %g23, %b[118] qpxor,1,sm %g30, %r1, %g30 qpfsub_adds,2,sm %b[118], %b[119], %g23, %b[91] qpshufb,3,sm %g27, %g26, %r0, %r26 qpfmuls,5,sm %b[91], %r9, %g23 } { loop_mode qpfsub_adds,0,sm %g19, %b[75], %g16, %b[83] qpfsub_rsubs,2,sm %b[79], %b[109], %g30, %r27 qpshufb,3,sm %b[106], %b[107], %r0, %r29 qpshufb,4,sm %g27, %g26, %r3, %g26 qpfmuls,5,sm %b[83], %r26, %r28 } { loop_mode qpfsub_rsubs,0,sm %g19, %b[75], %g16, %b[77] qpfmul_hadds,1,sm %b[94], %g25, %g28, %b[79] qpfsub_adds,2,sm %b[79], %b[109], %g30, %b[75] qpfmuls,4,sm %b[77], %g26, %b[94] qpfmuls,5,sm %g21, %r29, %b[109] } { loop_mode qpfsub_adds,0,sm %b[67], %b[64], %b[88], %b[64] qpfmul_hadds,1,sm %b[93], %g29, %g18, %b[68] qpfsub_adds,2,sm %b[80], %b[73], %b[86], %b[61] qpshufb,3,sm %b[61], %b[68], %r3, %b[80] qpshufb,4,sm %b[115], %b[117], %r3, %b[73] qpfmul_hadds,5,sm %b[104], %g31, %g20, %b[67] } { loop_mode qpfmul_hadds,0,sm %b[84], %g22, %r6, %b[86] qpfmul_hadds,2,sm %b[92], %b[114], %r7, %b[84] qpfmuls,3,sm %b[96], %b[73], %b[88] qpfmuls,4,sm %b[60], %b[80], %b[92] qpfmul_hadds,5,sm %b[102], %r9, %g23, %b[60] } { loop_mode addd,1,sm 0x10, %b[6], %b[4] ? %pcnt0 stqp,2 %r21, %b[6], %b[112] stqp,5 %r2, %b[6], %g24 } { loop_mode qpshufb,1,sm %b[81], %b[95], %r3, %b[81] stqp,2 %r20, %b[6], %b[105] stqp,5 %r24, %b[6], %b[111] } { loop_mode qpfmul_hadds,0,sm %b[89], %r26, %r28, %b[95] qpshufb,1,sm %b[69], %b[74], %r0, %b[89] stqp,2 %r22, %b[6], %b[108] qpshufb,3,sm %b[56], %b[57], %r0, %b[93] stqp,5 %r19, %b[6], %b[101] } { loop_mode qpfmul_hadds,0,sm %b[82], %r29, %b[109], %b[78] qpfmuls,1,sm %b[78], %b[81], %b[96] stqp,2 %r25, %b[6], %b[100] qpshufb,3,sm %b[53], %b[52], %r0, %b[85] qpfmuls,4,sm %b[51], %b[93], %b[82] stqp,5 %r23, %b[6], %b[85] } { loop_mode qpfmul_hadds,0,sm %b[99], %g26, %b[94], %b[94] stqp,2 %r17, %b[6], %g17 qpshufb,3,sm %b[71], %b[87], %r0, %b[99] qpfmuls,4,sm %b[35], %b[85], %b[100] stqp,5 %r12, %b[6], %b[110] } { loop_mode qpfmul_hadds,0,sm %b[98], %b[80], %b[92], %b[80] qpfmul_hadds,1,sm %b[97], %b[73], %b[88], %b[73] stqp,2 %r15, %b[6], %b[118] qpshufb,3,sm %b[58], %b[59], %r0, %b[88] qpfmuls,4,sm %b[50], %b[99], %b[91] stqp,5 %r14, %b[6], %b[91] } { loop_mode qpshufb,0,sm %b[79], %b[79], %r4, %b[79] qpshufb,1,sm %b[68], %b[68], %r4, %b[68] stqp,2 %r13, %b[6], %b[83] qpshufb,3,sm %b[70], %b[72], %r0, %b[83] qpfmuls,4,sm %b[31], %b[89], %b[92] stqp,5 %r16, %b[6], %r27 } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpshufb,0,sm %b[84], %b[84], %r4, %b[75] qpshufb,1,sm %b[86], %b[86], %r4, %b[77] stqp,2 %r18, %b[6], %b[77] qpshufb,3,sm %b[113], %b[116], %r0, %b[84] qpfmuls,4,sm %b[38], %b[83], %b[86] stqp,5 %r11, %b[6], %b[75] }
Теоретическая скорость: 32 комплексных числа за 39 тактов (32/39) = 6.56 Байт/такт
Четверная теоретическая скорость: 26.26 Байт/такт
Замеры скорости
Итоги по stage_radix4_readConjSwap_2x
Скорости упали по сравнению с исходными версиями stage_radix4_readConjSwap.
График FFT находится здесь.
Собираем FFT
fft_radix2
Собираем fft_radix2 из reverse_radix2_x32 и всех вариантов stage_radix2.
fft_radix2_2x
Собираем fft_radix2_2x из reverse_radix2_x32 и всех вариантов stage_radix2_2x.
fft_radix2_readConjSwap
Собираем fft_radix2_readConjSwap из reverse_radix2_x32 и всех вариантов stage_radix2_readConjSwap.
fft_radix2_readConjSwap_2x
Собираем fft_radix2_readConjSwap_2x из reverse_radix2_x32 и всех вариантов stage_radix2_readConjSwap_2x.
fft_radix4
Собираем fft_radix4 из reverse_radix4_x16 и всех вариантов stage_radix4.
fft_radix4_2x
Собираем fft_radix4_2x из reverse_radix4_x16 и всех вариантов stage_radix4_2x.
fft_radix4_readConjSwap
Собираем fft_radix4_readConjSwap из reverse_radix4_x16 и всех вариантов stage_radix4_readConjSwap.
fft_radix4_readConjSwap_2x
Собираем fft_radix4_readConjSwap_2x из reverse_radix4_x16 и всех вариантов stage_radix4_readConjSwap_2x.
Дальнейшие оптимизации имеющегося кода
Сейчас коэффициенты размещаются в нескольких массивах (conj/swap, coefC/coefD/coefE).
Можно разместить коэффициенты в одном массиве, перемежив их между собой.
Это позволит эффективнее производить чтение из памяти.
Хуже от этого стать не должно.
Дальнейшие направления создания кода
Что делать дальше:
Вариант
3xвradix2.
Сейчас2xвradix2ускоряет вычисления, поэтому можно сделать3x(и далее).
Сейчас2xвradix4замедляет вычисления, поэтому бесполезно делать3x.
Вряд ли это даст что‑то интересное, но для полноты картины можно сделать.Варианты Stage, в которых коэффициенты вычисляются на ходу, а не читаются из памяти.
Это должно ускорить вычисления в случаях, когда данные не помещаются в кэш.
Вradix4можно рассмотреть ещё такой случай: читать из памяти только coefC и вычислять coefD/coefE из coefC.Варианты Stage, на вход которым подаётся один постоянный коэффициент (или набор коэффициентов c/d/e для
radix4).
Можно заметить, что первый Stage использует один и тот же коэффициент для обработки всего входного массива. Второй Stage использует 2 разных коэффициента: один — для обработки первой половины массива, второй — для обработки второй половины. Третий Stage использует 4 разных коэффициента и так далее
Можно разбить обработку входного массива на части, обрабатываемые одинаковым коэффициентом. Получили коэффициент (из памяти или вычислением на ходу) — обработали часть массива, получили следующий коэффициент — обработали следующую часть массива.
Этот способ будет работать для начальных Stage, у которых такие части достаточно длинные. Для более поздних Stage надо использовать обычные методы получения коэффициентов (на каждом шаге читать из памяти или вычислять на ходу).
Поиск границы между начальными и поздними Stage придётся делать экспериментально.
Это отдельная задача.Объединить Reverse и первый Stage.
Сейчас есть объединение нескольких Stage в одном цикле (2x). Можно аналогично объединить Reverse и первый Stage (в общем случае — Reverse и несколько первых Stage).Входные данные типа
double complex.
Сейчас сделанfloat complex.
Надо сделать версию дляdouble complex.
Заключение
На примере stage_radix4_2x можно понять, что компилятор делает много работы по эффективной упаковке инструкций в такты. Вручную такая задача заняла бы много времени.
Я пишу код на Си так, чтобы его было удобно читать, а компилятор тасует операции так, чтобы, они эффективнее выполнялись. На ассемблере мне бы пришлось выбирать между понятным кодом и быстрым кодом.
На ассемблере приятно решать небольшие задачи, полностью осознавая, что где происходит. Однако, стоит немного увеличить сложность, и мозги начинают резко уставать.
Не уверен, смог бы я решить эту задачу на чистом ассемблере.
