Данная статья является продолжением предыдущей статьи.
Здесь рассматривается вариант прямой векторизации алгоритма FFT.
Перед чтением рекомендуется бегло ознакомиться с предыдущей статьёй.
Промежуточные результаты представлены на графиках в конце каждого раздела реализации.
Итоговые результаты показаны на графиках в разделе Собираем FFT
Исходный код
Исходный код лежит здесь.
Структура кода намеренно сделана как можно проще.
Приоритеты в порядке убывания:
— Скорость выполнения
— Единообразие
— Понятность
Компиляция:
./compile.sh */*.c
Проверка корректности:
./check_all.sh 12 1000 x
Измерение скорости:
taskset 1 ./speed_all_stage.sh 12 10000 x
taskset 1 ./speed_all_fft.sh 12 10000 x
taskset 1 нужен для корректного чтения счётчика тактов (привязка к первому ядру).
Опции компиляции
При компиляции использовались следующие опции lcc:
-Wall -O3 -faligned -ffast-math -march=elbrus-v5
Почему я компилирую для elbrus-v5 написано здесь.
Как измерялось время
Замеры времени делались с помощью функции clock_gettime():
struct timespec t0, t1; clock_gettime(CLOCK_REALTIME, &t0); /*** здесь измеряемый код ***/ clock_gettime(CLOCK_REALTIME, &t1); int usec = (t1.tv_sec - t0.tv_sec)*1000000 + (t1.tv_nsec - t0.tv_nsec)/1000;
Также использовалось чтение счётчика тактов процессора:
uint64_t get_clock_count() { uint64_t dst; #pragma asm_inline asm ("rrd %%clkr, %0" : "=r" (dst)); return dst; } ... uint64_t ticks0 = get_clock_count(); /*** здесь измеряемый код ***/ uint64_t ticks1 = get_clock_count(); uint64_t ticks = ticks1 - ticks0;
Как создавались графики
Графики рисовались в Google Sheets.
Данные для графиков собирались с помощью скрипта data_for_diagram.py, который запускает программу измерения скорости с разными параметрами и парсит её вывод.
Пример запуска скрипта:
taskset 1 ./data_for_diagram.py \
./fft_radix2/fft_radix2_speedtest 4 21 1 1000 -1 > out.txt
taskset 1 нужен для корректного чтения счётчика тактов (привязка к первому ядру).
Содержание:
Пишем векторный Reverse
reverse_radix2_vector2
reverse_radix2_vector4
reverse_radix4_vector2
Пишем векторный Stage
stage_radix2_vector2
stage_radix2_vector4
stage_radix4_vector2
Пишем LastStage
lastStage_radix2
Что такое прямая векторизация
Для ряда операций (сложение, умножение и так далее) в процессоре Эльбрус присутствуют векторные инструкции (SIMD). Векторные инструкции выполняют несколько одинаковых операций одновременно.
В предыдущей статье векторные инструкции использовались просто как набор из нескольких инструкций в одной. Например, вместо двух инструкций умножения float чисел, нужных по алгоритму, применялась одна векторная, выполняющая два умножения (SIMD64).
Проблема такого подхода в том, что нет простого способа ускорить алгоритм через замену инструкции SIMD64 на аналогичную SIMD128. Использование SIMD128 приводило к появлению инструкций перетасовки данных (shuf, perm), занимавших исполнительные юниты.
В этой статье векторные инструкции используются по‑другому.
Здесь с их помощью создаются «векторные» версии алгоритмов из исходных.
Векторные алгоритмы выполняют несколько одинаковых задач одновременно аналогично тому, как векторные инструкции выполняют несколько одинаковых операций одновременно.
Такой подход я и называю «прямой векторизацией».
Для выполнения прямой векторизации нужно заменить обычные инструкции на их векторные версии. Чтобы такое было возможно, в исходной программе должны использоваться инструкции, для которых существуют векторные версии. Обрабатываемые данные также требуется привести в векторную форму, то есть такой формат, который ожидается векторными инструкциями.
Прямая векторизация кода на Си
Пусть, например, алгоритм обрабатывает float числа.
Для выполнения прямой векторизации в 2 или 4 раза нужно заменить в Си‑коде исходного алгоритма тип float на __v2sf или __v4sf соответственно. И компилятор сгенерирует векторизованный код с использованием инструкций SIMD64 или SIMD128 соответственно.
При этом операции «+», «-», «*» будут скомпилированы в векторные версии инструкций.
Обрабатываемые данные также требуется предварительно привести в векторную форму:
там, где было одно
floatчисло, теперь должен быть «вектор»floatчисел (расположенные рядом 2 или 4floatчисла)там, где был массив
floatчисел, теперь должен быть массив «векторов»floatчисел (фактически исходный массив преобразуется в набор перемежённых массивов)
Прямая векторизация алгоритма Stage
Пусть изначально есть следующий код:
void stage_radix2_etalon(int data_count, myComplex *data_in, myComplex *data_out, myComplex *coef) { myComplex *x_in = &data_in[0]; myComplex *y_in = &data_in[1]; myComplex *c_in = coef; myComplex *out_add = &data_out[0]; myComplex *out_sub = &data_out[data_count/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { myComplex x = x_in[2*i]; myComplex y = y_in[2*i]; myComplex c = c_in[i]; myComplex cy = complex_mul(c, y); out_add[i] = complex_add(x, cy); out_sub[i] = complex_sub(x, cy); } }
Избавимся от типа myComplex, оставив просто тип float:
void stage_radix2_etalon(int data_count, float *data_in, float *data_out, float *coef) { float *x_real_in = &data_in[0]; float *x_imag_in = &data_in[1]; float *y_real_in = &data_in[2]; float *y_imag_in = &data_in[3]; float *c_real_in = &coef[0]; float *c_imag_in = &coef[1]; float *out_add_real = &data_out[0]; float *out_add_imag = &data_out[1]; float *out_sub_real = &data_out[data_count/2 + 0]; float *out_sub_imag = &data_out[data_count/2 + 1]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { float x_real = x_real_in[4*i]; float x_imag = x_imag_in[4*i]; float y_real = y_real_in[4*i]; float y_imag = y_imag_in[4*i]; float c_real = c_real_in[2*i]; float c_imag = c_imag_in[2*i]; float cy_real = c_real*y_real - c_imag*y_imag; float cy_imag = c_real*y_imag + c_imag*y_real; out_add_real[2*i] = x_real + cy_real; out_add_imag[2*i] = x_imag + cy_imag; out_sub_real[2*i] = x_real - cy_real; out_sub_imag[2*i] = x_imag - cy_imag; } }
Получившийся результат будем называть «базовым алгоритмом Stage» для прямой векторизации.
Если в этом коде заменить все float на __v2sf, получим прямую векторизацию в 2 раза.
Если в этом коде заменить все float на __v4sf, получим прямую векторизацию в 4 раза.
К сожалению, я узнал про этот приём уже после того, как написал весь код.
Поэтому сейчас в коде используются интринсики вместе с типами uint64_t и __v2di там, где в базовом алгоритме стоят операции «+», «-», «*».
В будущем я, скорее всего, исправлю код.
Прямая векторизация полного алгоритма FFT
Алгоритм FFT состоит из одного запуска Reverse и нескольких запусков Stage.
Stage был векторизован в предыдущем разделе.
Reverse векторизуется аналогичным образом.
Объединив векторные Reverse и Stage, получим векторный алгоритм FFT.
Этот алгоритм ожидает на входе и выходе набор перемежённых массивов float чисел.
В результате его работы будут вычислены FFT от каждого из перемежённых массивов.
Решение задачи FFT с помощью векторного FFT
Итак, мы умеем вычислять несколько FFT одновременно.
Как использовать это умение, чтобы вычислить FFT от одного набора комплексных чисел?
Для этого нужно вспомнить, как выглядел рекурсивный алгоритм Кули‑Тьюки:
Вычисляются FFT от каждого из перемежённых массивов комплексных чисел.
Вычисляется итоговый FFT с помощью этих перемежённых FFT.
Первый шаг сделаем с помощью векторного алгоритма FFT.
Векторный FFT ожидает на входе и выходе набор перемежённых массивов float чисел, а алгоритм Кули‑Тьюки ожидает набор перемежённых массивов комплексных чисел.
Поэтому добавим перед этим шагом преобразование исходных данных в векторную форму, а после этого шага — преобразование из векторной формы в исходную.
Схемы преобразования данных в векторную форму
Теперь на вход можно подавать массив комплексных чисел.
Алгоритм будет воспринимать его, как набор перемежённых массивов комплексных чисел.
В результате его работы будут вычислены FFT от каждого из перемежённых массивов.
Второй шаг похож на то, что делает адаптированный для Эльбруса алгоритм Stage из прошлой статьи. Если точнее, на этом шаге выполняются те же действия, что делает самый последний Stage в алгоритме FFT (тот Stage, в котором используется полный набор разных коэффициентов). Поэтому далее второй шаг будем называть «LastStage».
В случае выполнения прямой векторизации в 2 раза (когда получаются 2 перемежённых массива) в качестве LastStage можно использовать любой stage_radix2 из прошлой статьи.
В случае выполнения прямой векторизации в 4 раза (когда получаются 4 перемежённых массива) в качестве LastStage можно использовать любой stage_radix4 из прошлой статьи (или сначала stage_radix2_vector2 и затем любой stage_radix2 из прошлой статьи).
Итого, полный порядок действий выглядит так:
Исходные данные преобразуются в векторную форму.
Выполняется векторный Reverse.
Несколько раз выполняется векторный Stage.
Полученные данные преобразуются из векторной формы в исходную.
Выполняется LastStage.
Преобразования данных в векторную форму и обратно — достаточно простые процедуры. Поэтому в итоговой реализации они сразу включены внутрь Reverse и LastStage соответственно.
Пишем векторный Reverse
reverse_radix2_vector2
1. reverse_radix2_vector2_etalon
Эталонный вариант для сравнения на корректность.
Здесь reverseNumber_radix2 вычисляется с помощью цикла.
int reverseNumber_radix2(int number, int bit_count) { int answer = 0; for(int i = 0; i < bit_count; ++i) { answer <<= 1; answer |= number & 1; number >>= 1; } return answer; } void reverse_radix2_vector2_etalon(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; // int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; for(int64_t i = 0; i < count_out; ++i) { int index = reverseNumber_radix2(i, bit_count_out); myComplex a = data_in[2*i + 0]; myComplex b = data_in[2*i + 1]; data_out[index] = (myComplex2){.real = {a.real, b.real}, .imag = {a.imag, b.imag}}; } }
2. reverse_radix2_vector2
В процессоре есть инструкция bitrevd, которая производит операцию reverseNumber_radix2 над 64-битным числом. Заменим reverseNumber_radix2() на __builtin_e2k_bitrevd().
Код на Си
void reverse_radix2_vector2(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; // int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in = (__v2di*)data_in; __v2di *out = (__v2di*)data_out; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out; ++i) { int64_t index = __builtin_e2k_bitrevd(i) >> shift_out; __v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504}; out[index] = __builtin_e2k_qppermb(in[i], in[i], mask); } }
Основной цикл на ассемблере
.L4793: { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=5, abs=0, disp=0 } .L4668: { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END bitrevd,0,sm %b[11], %b[13] qppermb,1,sm %b[10], %b[10], %r4, %b[12] addd,2,sm %b[11], 0x1, %b[9] shrd,3,sm %b[17], %r0, %b[19] shld,4,sm %b[21], 0x4, %b[20] stqp,5 %r2, %b[22], %b[18] movaqp,1 area=0, ind=0, am=1, be=0, %b[0] }
Теоретическая скорость: 2 комплексных числа за 1 такт (2/1) = 16 Байт/такт
Замеры скорости
3. reverse_radix2_vector2_x2
Сделаем ручную раскрутку цикла в 2 раза.
Сразу напишем код так, чтобы вместиться в один такт (убираем инструкции shrd и shld).
Код на Си
void reverse_radix2_vector2_x2(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; // int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; int64_t delta = (1LL << shift_out) / 16; __v2di *in = (__v2di*) data_in; __v2di *out_0 = (__v2di*)&data_out[0]; __v2di *out_1 = (__v2di*)&data_out[count_out/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0, shifted_i = 0; i < count_out; i += 2, shifted_i += 2*delta) { int64_t offset = __builtin_e2k_bitrevd(shifted_i); __v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504}; *(__v2du*)((void*)out_0 + offset) = __builtin_e2k_qppermb(in[i + 0], in[i + 0], mask); *(__v2du*)((void*)out_1 + offset) = __builtin_e2k_qppermb(in[i + 1], in[i + 1], mask); } }
Основной цикл на ассемблере
.L5010: { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } .L4823: { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END bitrevd,0,sm %b[22], %b[13] qppermb,1,sm %b[10], %b[10], %r5, %b[19] stqp,2 %r4, %b[17], %b[21] qppermb,3,sm %b[11], %b[11], %r5, %b[23] addd,4,sm %b[20], %r0, %b[18] stqp,5 %r2, %b[17], %b[25] movaqp,0 area=0, ind=16, am=1, be=0, %b[0] movaqp,1 area=0, ind=0, am=0, be=0, %b[1] }
Теоретическая скорость: 4 комплексных числа за 1 такт (4/1) = 32 Байт/такт
Замеры скорости
Видим ускорение в начале графика.
Здесь происходит два чтения из одного места памяти и запись в два разных места памяти.
4. reverse_radix2_vector2_stream2
Теперь сделаем наоборот: будем читать из двух разных мест, а писать рядом.
Чтение в 2 потока из разных мест памяти, запись в одно место памяти.
Сразу напишем код так, чтобы вместиться в один такт (убираем инструкции shrd и shld).
Код на Си
void reverse_radix2_vector2_stream2(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; int64_t delta = (1LL << shift_out) / 16; __v2di *in_0 = (__v2di*)&data_in[0]; __v2di *in_1 = (__v2di*)&data_in[count_in/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0, shifted_i = 0; i < count_out/2; ++i, shifted_i += delta) { int64_t offset = __builtin_e2k_bitrevd(shifted_i); __v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_0[i], in_0[i], mask); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_1[i], in_1[i], mask); } }
Основной цикл на ассемблере
.L5212: { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=5, abs=0, disp=0 } .L5039: { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END bitrevd,0,sm %b[22], %b[13] qppermb,1,sm %b[11], %b[11], %r5, %b[23] stqp,2 %r2, %b[17], %b[25] qppermb,3,sm %b[10], %b[10], %r5, %b[19] addd,4,sm %b[20], %r4, %b[18] stqp,5 %r0, %b[17], %b[21] movaqp,1 area=0, ind=0, am=1, be=0, %b[1] movaqp,3 area=0, ind=0, am=1, be=0, %b[0] }
Теоретическая скорость: 4 комплексных числа за 1 такт (4/1) = 32 Байт/такт
Замеры скорости
Видим желаемое ускорение по всей длине графика.
5. reverse_radix2_vector2_stream4
Чтение в 4 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector2_stream4(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_00 = (__v2di*)&data_in[0 * count_in/4]; __v2di *in_01 = (__v2di*)&data_in[1 * count_in/4]; __v2di *in_10 = (__v2di*)&data_in[2 * count_in/4]; __v2di *in_11 = (__v2di*)&data_in[3 * count_in/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/4; ++i) { int64_t offset = 16 * (__builtin_e2k_bitrevd(i) >> shift_out); __v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_00[i], in_00[i], mask); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_10[i], in_10[i], mask); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_01[i], in_01[i], mask); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_11[i], in_11[i], mask); } }
Основной цикл на ассемблере
.L5516: { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=4, abs=0, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=4, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=4, abs=16, disp=0 } .L5255: { loop_mode bitrevd,0,sm %b[4], %b[1] qppermb,1,sm %b[27], %b[27], %r7, %b[9] stqp,2 %r2, %b[14], %b[28] shld,3,sm %b[24], 0x4, %b[12] qppermb,4,sm %b[22], %b[22], %r7, %b[5] stqp,5 %r4, %b[14], %b[26] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[19], %b[19], %r7, %b[26] addd,1,sm %b[2], 0x1, %b[0] stqp,2 %r0, %b[14], %b[11] qppermb,3,sm %b[10], %b[10], %r7, %b[24] shrd,4,sm %b[3], %r6, %b[22] stqp,5 %r5, %b[14], %b[7] movaqp,0 area=1, ind=0, am=1, be=0, %b[4] movaqp,1 area=0, ind=0, am=1, be=0, %b[13] movaqp,2 area=1, ind=0, am=1, be=0, %b[16] movaqp,3 area=0, ind=0, am=1, be=0, %b[21] }
Теоретическая скорость: 8 комплексных чисел за 2 такта (8/2) = 32 Байт/такт
Замеры скорости
Видим замедление в начале и ускорение в конце графика.
6. reverse_radix2_vector2_stream8
Чтение в 8 потоков из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector2_stream8(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_000 = (__v2di*)&data_in[0 * count_in/8]; __v2di *in_001 = (__v2di*)&data_in[1 * count_in/8]; __v2di *in_010 = (__v2di*)&data_in[2 * count_in/8]; __v2di *in_011 = (__v2di*)&data_in[3 * count_in/8]; __v2di *in_100 = (__v2di*)&data_in[4 * count_in/8]; __v2di *in_101 = (__v2di*)&data_in[5 * count_in/8]; __v2di *in_110 = (__v2di*)&data_in[6 * count_in/8]; __v2di *in_111 = (__v2di*)&data_in[7 * count_in/8]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/8; ++i) { int64_t offset = 16 * (__builtin_e2k_bitrevd(i) >> shift_out); __v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_000[i], in_000[i], mask); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_100[i], in_100[i], mask); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_010[i], in_010[i], mask); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_110[i], in_110[i], mask); *(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_001[i], in_001[i], mask); *(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_101[i], in_101[i], mask); *(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_011[i], in_011[i], mask); *(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_111[i], in_111[i], mask); } }
Основной цикл на ассемблере
.L6024: { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=3, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=3, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=3, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=3, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=3, abs=24, disp=0 } .L5591: { loop_mode qppermb,1,sm %b[21], %b[21], %r11, %b[26] stqp,2 %r2, %b[18], %b[24] qppermb,4,sm %b[13], %b[13], %r11, %b[25] stqp,5 %r4, %b[18], %b[23] } { loop_mode qppermb,1,sm %b[12], %b[12], %r11, %b[24] stqp,2 %r0, %b[18], %b[26] shrd,3,sm %b[22], %r12, %b[1] qppermb,4,sm %b[5], %b[5], %r11, %b[23] stqp,5 %r5, %b[18], %b[25] } { loop_mode qppermb,1,sm %b[9], %b[9], %r11, %b[27] stqp,2 %r6, %b[18], %b[24] shld,3,sm %b[1], 0x4, %b[16] qppermb,4,sm %b[8], %b[8], %r11, %b[25] stqp,5 %r9, %b[18], %b[23] movaqp,0 area=1, ind=0, am=1, be=0, %b[12] movaqp,1 area=0, ind=0, am=1, be=0, %b[13] movaqp,2 area=3, ind=0, am=1, be=0, %b[4] movaqp,3 area=2, ind=0, am=1, be=0, %b[5] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[15], %b[15], %r11, %b[22] addd,1,sm %b[2], 0x1, %b[0] stqp,2 %r7, %b[18], %b[27] qppermb,3,sm %b[14], %b[14], %r11, %b[21] bitrevd,4,sm %b[2], %b[20] stqp,5 %r10, %b[18], %b[25] movaqp,0 area=3, ind=0, am=1, be=0, %b[1] movaqp,1 area=2, ind=0, am=1, be=0, %b[8] movaqp,2 area=1, ind=0, am=1, be=0, %b[9] movaqp,3 area=0, ind=0, am=1, be=0, %b[17] }
Теоретическая скорость: 16 комплексных чисел за 4 такта (16/4) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
7. reverse_radix2_vector2_stream16
Чтение в 16 потоков из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector2_stream16(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_0000 = (__v2di*)&data_in[ 0 * count_in/16]; __v2di *in_0001 = (__v2di*)&data_in[ 1 * count_in/16]; __v2di *in_0010 = (__v2di*)&data_in[ 2 * count_in/16]; __v2di *in_0011 = (__v2di*)&data_in[ 3 * count_in/16]; __v2di *in_0100 = (__v2di*)&data_in[ 4 * count_in/16]; __v2di *in_0101 = (__v2di*)&data_in[ 5 * count_in/16]; __v2di *in_0110 = (__v2di*)&data_in[ 6 * count_in/16]; __v2di *in_0111 = (__v2di*)&data_in[ 7 * count_in/16]; __v2di *in_1000 = (__v2di*)&data_in[ 8 * count_in/16]; __v2di *in_1001 = (__v2di*)&data_in[ 9 * count_in/16]; __v2di *in_1010 = (__v2di*)&data_in[10 * count_in/16]; __v2di *in_1011 = (__v2di*)&data_in[11 * count_in/16]; __v2di *in_1100 = (__v2di*)&data_in[12 * count_in/16]; __v2di *in_1101 = (__v2di*)&data_in[13 * count_in/16]; __v2di *in_1110 = (__v2di*)&data_in[14 * count_in/16]; __v2di *in_1111 = (__v2di*)&data_in[15 * count_in/16]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/16; ++i) { int64_t offset = 16 * (__builtin_e2k_bitrevd(i) >> shift_out); __v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_0000[i], in_0000[i], mask); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_1000[i], in_1000[i], mask); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_0100[i], in_0100[i], mask); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_1100[i], in_1100[i], mask); *(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_0010[i], in_0010[i], mask); *(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_1010[i], in_1010[i], mask); *(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_0110[i], in_0110[i], mask); *(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_1110[i], in_1110[i], mask); *(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_0001[i], in_0001[i], mask); *(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_1001[i], in_1001[i], mask); *(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_0101[i], in_0101[i], mask); *(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_1101[i], in_1101[i], mask); *(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_0011[i], in_0011[i], mask); *(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_1011[i], in_1011[i], mask); *(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_0111[i], in_0111[i], mask); *(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_1111[i], in_1111[i], mask); } }
Основной цикл на ассемблере
.L6935: { fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=2, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=2, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=2, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=2, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=2, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=2, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=2, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=2, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=2, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=2, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=2, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=2, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=2, abs=24, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=2, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=2, abs=28, disp=0 } .L6163: { loop_mode qppermb,1,sm %b[24], %b[24], %r1, %b[28] stqp,2 %r2, %b[13], %b[9] qppermb,4,sm %b[22], %b[22], %r1, %b[27] stqp,5 %r5, %b[13], %b[4] } { loop_mode qppermb,1,sm %b[26], %b[26], %r1, %b[30] stqp,2 %r0, %b[13], %b[28] qppermb,4,sm %b[25], %b[25], %r1, %b[29] stqp,5 %r6, %b[13], %b[27] movaqp,0 area=7, ind=0, am=1, be=0, %b[10] movaqp,1 area=6, ind=0, am=1, be=0, %b[4] movaqp,2 area=7, ind=0, am=1, be=0, %b[7] movaqp,3 area=6, ind=0, am=1, be=0, %b[1] } { loop_mode qppermb,1,sm %b[23], %b[23], %r1, %b[27] stqp,2 %r7, %b[13], %b[30] qppermb,4,sm %b[16], %b[16], %r1, %b[26] stqp,5 %r10, %b[13], %b[29] movaqp,0 area=1, ind=0, am=1, be=0, %b[15] movaqp,1 area=0, ind=0, am=1, be=0, %b[17] movaqp,2 area=5, ind=0, am=1, be=0, %b[9] movaqp,3 area=4, ind=0, am=1, be=0, %b[12] } { loop_mode qppermb,1,sm %b[21], %b[21], %r1, %b[29] stqp,2 %r9, %b[13], %b[27] qppermb,4,sm %b[14], %b[14], %r1, %b[28] stqp,5 %r11, %b[13], %b[26] movaqp,0 area=5, ind=0, am=1, be=0, %b[16] movaqp,1 area=4, ind=0, am=1, be=0, %b[19] movaqp,2 area=1, ind=0, am=1, be=0, %b[20] movaqp,3 area=0, ind=0, am=1, be=0, %b[22] } { loop_mode addd,0,sm %b[2], 0x1, %b[0] qppermb,1,sm %b[18], %b[18], %r1, %b[27] stqp,2 %r12, %b[13], %b[29] bitrevd,3,sm %b[2], %b[25] qppermb,4,sm %b[11], %b[11], %r1, %b[26] stqp,5 %r13, %b[13], %b[28] movaqp,0 area=3, ind=0, am=1, be=0, %b[23] movaqp,1 area=2, ind=0, am=1, be=0, %b[24] movaqp,2 area=3, ind=0, am=1, be=0, %b[14] movaqp,3 area=2, ind=0, am=1, be=0, %b[21] } { loop_mode qppermb,1,sm %b[6], %b[6], %r1, %b[29] stqp,2 %r14, %b[13], %b[27] shrd,3,sm %b[25], %r20, %b[2] qppermb,4,sm %b[3], %b[3], %r1, %b[28] stqp,5 %r15, %b[13], %b[26] } { loop_mode qppermb,1,sm %b[10], %b[10], %r1, %b[6] stqp,2 %r16, %b[13], %b[29] shld,3,sm %b[2], 0x4, %b[11] qppermb,4,sm %b[7], %b[7], %r1, %b[3] stqp,5 %r17, %b[13], %b[28] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[17], %b[17], %r1, %b[7] stqp,2 %r18, %b[13], %b[8] qppermb,3,sm %b[15], %b[15], %r1, %b[2] stqp,5 %r19, %b[13], %b[5] }
Теоретическая скорость: 32 комплексных числа за 8 тактов (32/8) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
8. reverse_radix2_vector2_stream32
Чтение в 32 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector2_stream32(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_00000 = (__v2di*)&data_in[ 0 * count_in/32]; __v2di *in_00001 = (__v2di*)&data_in[ 1 * count_in/32]; __v2di *in_00010 = (__v2di*)&data_in[ 2 * count_in/32]; __v2di *in_00011 = (__v2di*)&data_in[ 3 * count_in/32]; __v2di *in_00100 = (__v2di*)&data_in[ 4 * count_in/32]; __v2di *in_00101 = (__v2di*)&data_in[ 5 * count_in/32]; __v2di *in_00110 = (__v2di*)&data_in[ 6 * count_in/32]; __v2di *in_00111 = (__v2di*)&data_in[ 7 * count_in/32]; __v2di *in_01000 = (__v2di*)&data_in[ 8 * count_in/32]; __v2di *in_01001 = (__v2di*)&data_in[ 9 * count_in/32]; __v2di *in_01010 = (__v2di*)&data_in[10 * count_in/32]; __v2di *in_01011 = (__v2di*)&data_in[11 * count_in/32]; __v2di *in_01100 = (__v2di*)&data_in[12 * count_in/32]; __v2di *in_01101 = (__v2di*)&data_in[13 * count_in/32]; __v2di *in_01110 = (__v2di*)&data_in[14 * count_in/32]; __v2di *in_01111 = (__v2di*)&data_in[15 * count_in/32]; __v2di *in_10000 = (__v2di*)&data_in[16 * count_in/32]; __v2di *in_10001 = (__v2di*)&data_in[17 * count_in/32]; __v2di *in_10010 = (__v2di*)&data_in[18 * count_in/32]; __v2di *in_10011 = (__v2di*)&data_in[19 * count_in/32]; __v2di *in_10100 = (__v2di*)&data_in[20 * count_in/32]; __v2di *in_10101 = (__v2di*)&data_in[21 * count_in/32]; __v2di *in_10110 = (__v2di*)&data_in[22 * count_in/32]; __v2di *in_10111 = (__v2di*)&data_in[23 * count_in/32]; __v2di *in_11000 = (__v2di*)&data_in[24 * count_in/32]; __v2di *in_11001 = (__v2di*)&data_in[25 * count_in/32]; __v2di *in_11010 = (__v2di*)&data_in[26 * count_in/32]; __v2di *in_11011 = (__v2di*)&data_in[27 * count_in/32]; __v2di *in_11100 = (__v2di*)&data_in[28 * count_in/32]; __v2di *in_11101 = (__v2di*)&data_in[29 * count_in/32]; __v2di *in_11110 = (__v2di*)&data_in[30 * count_in/32]; __v2di *in_11111 = (__v2di*)&data_in[31 * count_in/32]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/32; ++i) { int64_t offset = 16 * (__builtin_e2k_bitrevd(i) >> shift_out); __v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_00000[i], in_00000[i], mask); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_10000[i], in_10000[i], mask); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_01000[i], in_01000[i], mask); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_11000[i], in_11000[i], mask); *(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_00100[i], in_00100[i], mask); *(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_10100[i], in_10100[i], mask); *(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_01100[i], in_01100[i], mask); *(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_11100[i], in_11100[i], mask); *(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_00010[i], in_00010[i], mask); *(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_10010[i], in_10010[i], mask); *(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_01010[i], in_01010[i], mask); *(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_11010[i], in_11010[i], mask); *(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_00110[i], in_00110[i], mask); *(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_10110[i], in_10110[i], mask); *(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_01110[i], in_01110[i], mask); *(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_11110[i], in_11110[i], mask); *(__v2du*)((void*)data_out + offset + 16*16) = __builtin_e2k_qppermb(in_00001[i], in_00001[i], mask); *(__v2du*)((void*)data_out + offset + 17*16) = __builtin_e2k_qppermb(in_10001[i], in_10001[i], mask); *(__v2du*)((void*)data_out + offset + 18*16) = __builtin_e2k_qppermb(in_01001[i], in_01001[i], mask); *(__v2du*)((void*)data_out + offset + 19*16) = __builtin_e2k_qppermb(in_11001[i], in_11001[i], mask); *(__v2du*)((void*)data_out + offset + 20*16) = __builtin_e2k_qppermb(in_00101[i], in_00101[i], mask); *(__v2du*)((void*)data_out + offset + 21*16) = __builtin_e2k_qppermb(in_10101[i], in_10101[i], mask); *(__v2du*)((void*)data_out + offset + 22*16) = __builtin_e2k_qppermb(in_01101[i], in_01101[i], mask); *(__v2du*)((void*)data_out + offset + 23*16) = __builtin_e2k_qppermb(in_11101[i], in_11101[i], mask); *(__v2du*)((void*)data_out + offset + 24*16) = __builtin_e2k_qppermb(in_00011[i], in_00011[i], mask); *(__v2du*)((void*)data_out + offset + 25*16) = __builtin_e2k_qppermb(in_10011[i], in_10011[i], mask); *(__v2du*)((void*)data_out + offset + 26*16) = __builtin_e2k_qppermb(in_01011[i], in_01011[i], mask); *(__v2du*)((void*)data_out + offset + 27*16) = __builtin_e2k_qppermb(in_11011[i], in_11011[i], mask); *(__v2du*)((void*)data_out + offset + 28*16) = __builtin_e2k_qppermb(in_00111[i], in_00111[i], mask); *(__v2du*)((void*)data_out + offset + 29*16) = __builtin_e2k_qppermb(in_10111[i], in_10111[i], mask); *(__v2du*)((void*)data_out + offset + 30*16) = __builtin_e2k_qppermb(in_01111[i], in_01111[i], mask); *(__v2du*)((void*)data_out + offset + 31*16) = __builtin_e2k_qppermb(in_11111[i], in_11111[i], mask); } }
Основной цикл на ассемблере
.L8688: { fapb ct=0, dcd=0, fmt=5, mrng=16, d=17, incr=0, ind=0, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=16, incr=0, ind=0, asz=1, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=15, incr=0, ind=0, asz=1, abs=2, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=14, incr=0, ind=0, asz=1, abs=2, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=13, incr=0, ind=0, asz=1, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=12, incr=0, ind=0, asz=1, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=11, incr=0, ind=0, asz=1, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=10, incr=0, ind=0, asz=1, abs=6, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=9, incr=0, ind=0, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=8, incr=0, ind=0, asz=1, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=7, incr=0, ind=0, asz=1, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=6, incr=0, ind=0, asz=1, abs=10, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=5, incr=0, ind=0, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=4, incr=0, ind=0, asz=1, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=3, incr=0, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=2, incr=0, ind=0, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=1, abs=30, disp=0 } .L7202: { loop_mode qppermb,1,sm %b[37], %b[37], %r0, %b[21] stqp,2 %r2, %b[16], %b[20] qppermb,4,sm %b[35], %b[35], %r0, %b[18] stqp,5 %r6, %b[16], %b[18] } { loop_mode qppermb,1,sm %b[39], %b[39], %r0, %b[20] stqp,2 %r4, %b[16], %b[21] qppermb,4,sm %b[38], %b[38], %r0, %b[18] stqp,5 %r7, %b[16], %b[18] } { loop_mode qppermb,1,sm %b[23], %b[23], %r0, %b[19] stqp,2 %r9, %b[16], %b[20] qppermb,4,sm %b[19], %b[19], %r0, %b[18] stqp,5 %r11, %b[16], %b[18] } { loop_mode qppermb,1,sm %b[26], %b[26], %r0, %b[19] stqp,2 %r10, %b[16], %b[19] qppermb,4,sm %b[33], %b[33], %r0, %b[18] stqp,5 %r12, %b[16], %b[18] } { loop_mode qppermb,1,sm %b[25], %b[25], %r0, %b[19] stqp,2 %r13, %b[16], %b[19] qppermb,4,sm %b[32], %b[32], %r0, %b[18] stqp,5 %r14, %b[16], %b[18] } { loop_mode qppermb,1,sm %b[31], %b[31], %r0, %b[24] stqp,2 %r15, %b[16], %b[19] qppermb,4,sm %b[30], %b[30], %r0, %b[21] stqp,5 %r16, %b[16], %b[18] movaqp,0 area=15, ind=0, am=1, be=0, %b[19] movaqp,1 area=14, ind=0, am=1, be=0, %b[22] movaqp,2 area=15, ind=0, am=1, be=0, %b[18] movaqp,3 area=14, ind=0, am=1, be=0, %b[20] } { loop_mode qppermb,1,sm %b[29], %b[29], %r0, %b[26] stqp,2 %r17, %b[16], %b[24] qppermb,4,sm %b[17], %b[17], %r0, %b[25] stqp,5 %r18, %b[16], %b[21] movaqp,0 area=13, ind=0, am=1, be=0, %b[21] movaqp,1 area=12, ind=0, am=1, be=0, %b[24] movaqp,2 area=13, ind=0, am=1, be=0, %b[17] movaqp,3 area=12, ind=0, am=1, be=0, %b[23] } { loop_mode qppermb,1,sm %b[28], %b[28], %r0, %b[26] stqp,2 %r19, %b[16], %b[26] qppermb,4,sm %b[27], %b[27], %r0, %b[25] stqp,5 %r20, %b[16], %b[25] movaqp,0 area=11, ind=0, am=1, be=0, %b[1] movaqp,1 area=10, ind=0, am=1, be=0, %b[13] movaqp,2 area=11, ind=0, am=1, be=0, %b[0] movaqp,3 area=10, ind=0, am=1, be=0, %b[12] } { loop_mode qppermb,1,sm %g17, %g17, %r0, %b[28] stqp,2 %r21, %b[16], %b[26] qppermb,4,sm %g16, %g16, %r0, %b[27] stqp,5 %r22, %b[16], %b[25] movaqp,0 area=9, ind=0, am=1, be=0, %g17 movaqp,1 area=8, ind=0, am=1, be=0, %b[26] movaqp,2 area=9, ind=0, am=1, be=0, %g16 movaqp,3 area=8, ind=0, am=1, be=0, %b[25] } { loop_mode qppermb,1,sm %b[15], %b[15], %r0, %b[31] stqp,2 %r23, %b[16], %b[28] qppermb,4,sm %b[14], %b[14], %r0, %b[30] stqp,5 %r24, %b[16], %b[27] movaqp,0 area=7, ind=0, am=1, be=0, %b[27] movaqp,1 area=6, ind=0, am=1, be=0, %b[29] movaqp,2 area=7, ind=0, am=1, be=0, %b[15] movaqp,3 area=6, ind=0, am=1, be=0, %b[28] } { loop_mode qppermb,1,sm %b[3], %b[3], %r0, %b[36] stqp,2 %r25, %b[16], %b[31] qppermb,4,sm %b[2], %b[2], %r0, %b[33] stqp,5 %r26, %b[16], %b[30] movaqp,0 area=1, ind=0, am=1, be=0, %b[32] movaqp,1 area=0, ind=0, am=1, be=0, %b[34] movaqp,2 area=5, ind=0, am=1, be=0, %b[30] movaqp,3 area=4, ind=0, am=1, be=0, %b[31] } { loop_mode qppermb,1,sm %b[24], %b[24], %r0, %b[3] stqp,2 %r27, %b[16], %b[36] qppermb,4,sm %b[23], %b[23], %r0, %b[2] stqp,5 %r28, %b[16], %b[33] movaqp,0 area=5, ind=0, am=1, be=0, %b[23] movaqp,1 area=4, ind=0, am=1, be=0, %b[24] movaqp,2 area=1, ind=0, am=1, be=0, %b[33] movaqp,3 area=0, ind=0, am=1, be=0, %b[35] } { loop_mode addd,0,sm %r5, 0x1, %r5 qppermb,1,sm %b[21], %b[21], %r0, %b[5] stqp,2 %r29, %b[16], %b[5] bitrevd,3,sm %r5, %b[38] qppermb,4,sm %b[17], %b[17], %r0, %b[4] stqp,5 %r30, %b[16], %b[4] movaqp,0 area=3, ind=0, am=1, be=0, %b[36] movaqp,1 area=2, ind=0, am=1, be=0, %b[37] movaqp,2 area=3, ind=0, am=1, be=0, %b[17] movaqp,3 area=2, ind=0, am=1, be=0, %b[21] } { loop_mode qppermb,1,sm %b[22], %b[22], %r0, %b[7] stqp,2 %r31, %b[16], %b[7] shrd,3,sm %b[38], %r1, %b[20] qppermb,4,sm %b[20], %b[20], %r0, %b[6] stqp,5 %r32, %b[16], %b[6] } { loop_mode qppermb,1,sm %b[19], %b[19], %r0, %b[9] stqp,2 %r33, %b[16], %b[9] shld,3,sm %b[20], 0x4, %b[14] qppermb,4,sm %b[18], %b[18], %r0, %b[8] stqp,5 %r34, %b[16], %b[8] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[34], %b[34], %r0, %b[18] stqp,2 %r35, %b[16], %b[11] qppermb,3,sm %b[32], %b[32], %r0, %b[16] stqp,5 %r36, %b[16], %b[10] }
Теоретическая скорость: 64 комплексных числа за 16 тактов (64/16) = 32 Байт/такт
Замеры скорости
Видим замедление по всей длине графика.
При попытке чтения в 64 потока получается резко менее эффективный код.
APB не может читать в 64 потока, поэтому в дополнение к APB компилятор вставляет обычные операции чтения ldqp. В итоге скорость резко проседает.
Итоги по reverse_radix2_vector2
Победителем можно считать reverse_radix2_vector2_stream16.
При реализации Vector-2 Radix-2 FFT будем использовать его.
reverse_radix2_vector4
1. reverse_radix2_vector4_etalon
Эталонный вариант для сравнения на корректность.
Здесь reverseNumber_radix2 вычисляется с помощью цикла.
int reverseNumber_radix2(int number, int bit_count) { int answer = 0; for(int i = 0; i < bit_count; ++i) { answer <<= 1; answer |= number & 1; number >>= 1; } return answer; } void reverse_radix2_vector4_etalon(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; // int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; for(int64_t i = 0; i < count_out; ++i) { int index = reverseNumber_radix2(i, bit_count_out); myComplex a = data_in[4*i + 0]; myComplex b = data_in[4*i + 1]; myComplex c = data_in[4*i + 2]; myComplex d = data_in[4*i + 3]; data_out[index] = (myComplex4){.real = {a.real, b.real, c.real, d.real}, .imag = {a.imag, b.imag, c.imag, d.imag}}; } }
2. reverse_radix2_vector4
В процессоре есть инструкция bitrevd, которая производит операцию reverseNumber_radix2 над 64-битным числом. Заменим reverseNumber_radix2() на __builtin_e2k_bitrevd().
Код на Си
void reverse_radix2_vector4(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; // int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; int64_t delta = (1LL << shift_out) / 32; __v2di *in = (__v2di*)data_in; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0, shifted_i = 0; i < count_out; ++i, shifted_i += delta) { int64_t offset = __builtin_e2k_bitrevd(shifted_i); __v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110}; __v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in[2*i+1], in[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in[2*i+1], in[2*i+0], mask_imag); } }
Основной цикл на ассемблере
.L6535: { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } .L6346: { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END bitrevd,0,sm %b[22], %b[13] qppermb,1,sm %b[11], %b[10], %r5, %b[19] stqp,2 %r0, %b[17], %b[21] qppermb,3,sm %b[11], %b[10], %r6, %b[23] addd,4,sm %b[20], %r4, %b[18] stqp,5 %r2, %b[17], %b[25] movaqp,0 area=0, ind=0, am=1, be=0, %b[0] movaqp,1 area=0, ind=16, am=0, be=0, %b[1] }
Теоретическая скорость: 4 комплексных числа за 1 такт (4/1) = 32 Байт/такт
Замеры скорости
3. reverse_radix2_vector4_x2
Сделаем ручную раскрутку цикла в 2 раза.
Код на Си
void reverse_radix2_vector4_x2(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; // int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in = (__v2di*) data_in; __v2di *out_0 = (__v2di*)&data_out[0]; __v2di *out_1 = (__v2di*)&data_out[count_out/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out; i += 2) { int64_t offset = 32 * (__builtin_e2k_bitrevd(i) >> shift_out); __v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110}; __v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514}; *(__v2du*)((void*)out_0 + offset + 0*16) = __builtin_e2k_qppermb(in[2*(i+0)+1], in[2*(i+0)+0], mask_real); *(__v2du*)((void*)out_0 + offset + 1*16) = __builtin_e2k_qppermb(in[2*(i+0)+1], in[2*(i+0)+0], mask_imag); *(__v2du*)((void*)out_1 + offset + 0*16) = __builtin_e2k_qppermb(in[2*(i+1)+1], in[2*(i+1)+0], mask_real); *(__v2du*)((void*)out_1 + offset + 1*16) = __builtin_e2k_qppermb(in[2*(i+1)+1], in[2*(i+1)+0], mask_imag); } }
Основной цикл на ассемблере
.L6884: { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=32 } .L6561: { loop_mode bitrevd,0,sm %b[4], %b[1] qppermb,1,sm %b[25], %b[20], %r7, %b[28] stqp,2 %r0, %b[7], %b[26] shld,3,sm %b[22], 0x5, %b[5] qppermb,4,sm %b[17], %b[12], %r7, %b[27] stqp,5 %r4, %b[7], %b[24] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[23], %b[18], %r9, %b[24] addd,1,sm %b[2], 0x2, %b[0] stqp,2 %r2, %b[7], %b[28] qppermb,3,sm %b[15], %b[10], %r9, %b[22] shrd,4,sm %b[3], %r6, %b[20] stqp,5 %r5, %b[7], %b[27] movaqp,0 area=0, ind=0, am=1, be=0, %b[12] movaqp,1 area=0, ind=16, am=0, be=0, %b[17] movaqp,2 area=0, ind=0, am=1, be=0, %b[4] movaqp,3 area=0, ind=16, am=0, be=0, %b[9] }
Теоретическая скорость: 8 комплексных чисел за 2 такта (8/2) = 32 Байт/такт
Замеры скорости
Видим ускорение в начале графика.
Здесь происходит два чтения из одного места памяти и запись в два разных места памяти.
4. reverse_radix2_vector4_stream2
Теперь сделаем наоборот: будем читать из двух разных мест, а писать рядом.
Чтение в 2 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector4_stream2(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_0 = (__v2di*)&data_in[0]; __v2di *in_1 = (__v2di*)&data_in[count_in/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/2; ++i) { int64_t offset = 32 * (__builtin_e2k_bitrevd(i) >> shift_out); __v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110}; __v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_0[2*i+1], in_0[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_0[2*i+1], in_0[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_1[2*i+1], in_1[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_1[2*i+1], in_1[2*i+0], mask_imag); } }
Основной цикл на ассемблере
.L7199: { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } .L6909: { loop_mode bitrevd,0,sm %b[4], %b[1] qppermb,1,sm %b[25], %b[20], %r9, %b[28] stqp,2 %r0, %b[7], %b[26] shld,3,sm %b[22], 0x5, %b[5] qppermb,4,sm %b[17], %b[12], %r9, %b[27] stqp,5 %r5, %b[7], %b[24] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[23], %b[18], %r6, %b[24] addd,1,sm %b[2], 0x1, %b[0] stqp,2 %r2, %b[7], %b[28] qppermb,3,sm %b[15], %b[10], %r6, %b[22] shrd,4,sm %b[3], %r7, %b[20] stqp,5 %r4, %b[7], %b[27] movaqp,0 area=0, ind=0, am=1, be=0, %b[12] movaqp,1 area=0, ind=16, am=0, be=0, %b[17] movaqp,2 area=0, ind=0, am=1, be=0, %b[4] movaqp,3 area=0, ind=16, am=0, be=0, %b[9] }
Теоретическая скорость: 8 комплексных чисел за 2 такта (8/2) = 32 Байт/такт
Замеры скорости
Видим замедление в начале и ускорение в конце графика.
5. reverse_radix2_vector4_stream4
Чтение в 4 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector4_stream4(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_00 = (__v2di*)&data_in[0 * count_in/4]; __v2di *in_01 = (__v2di*)&data_in[1 * count_in/4]; __v2di *in_10 = (__v2di*)&data_in[2 * count_in/4]; __v2di *in_11 = (__v2di*)&data_in[3 * count_in/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/4; ++i) { int64_t offset = 32 * (__builtin_e2k_bitrevd(i) >> shift_out); __v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110}; __v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_00[2*i+1], in_00[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_00[2*i+1], in_00[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_10[2*i+1], in_10[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_10[2*i+1], in_10[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_01[2*i+1], in_01[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_01[2*i+1], in_01[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_11[2*i+1], in_11[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_11[2*i+1], in_11[2*i+0], mask_imag); } }
Основной цикл на ассемблере
.L7728: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=4, abs=0, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=4, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=16, disp=0 } .L7242: { loop_mode qppermb,1,sm %b[20], %b[16], %r13, %b[26] stqp,2 %r0, %b[21], %b[25] qppermb,4,sm %b[12], %b[8], %r13, %b[24] stqp,5 %r5, %b[21], %b[22] } { loop_mode qppermb,1,sm %b[17], %b[13], %r12, %b[25] stqp,2 %r2, %b[21], %b[26] shrd,3,sm %b[19], %r11, %b[1] qppermb,4,sm %b[9], %b[5], %r12, %b[22] stqp,5 %r4, %b[21], %b[24] } { loop_mode qppermb,1,sm %b[17], %b[13], %r13, %b[27] stqp,2 %r7, %b[21], %b[25] shld,3,sm %b[1], 0x5, %b[19] qppermb,4,sm %b[9], %b[5], %r13, %b[24] stqp,5 %r10, %b[21], %b[22] movaqp,0 area=0, ind=0, am=1, be=0, %b[12] movaqp,1 area=0, ind=16, am=0, be=0, %b[16] movaqp,2 area=0, ind=0, am=1, be=0, %b[4] movaqp,3 area=0, ind=16, am=0, be=0, %b[8] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[18], %b[14], %r12, %b[23] addd,1,sm %b[2], 0x1, %b[0] stqp,2 %r6, %b[21], %b[27] qppermb,3,sm %b[10], %b[6], %r12, %b[20] bitrevd,4,sm %b[2], %b[17] stqp,5 %r9, %b[21], %b[24] movaqp,0 area=1, ind=0, am=1, be=0, %b[9] movaqp,1 area=1, ind=16, am=0, be=0, %b[13] movaqp,2 area=1, ind=0, am=1, be=0, %b[1] movaqp,3 area=1, ind=16, am=0, be=0, %b[5] }
Теоретическая скорость: 16 комплексных чисел за 4 такта (16/4) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
6. reverse_radix2_vector4_stream8
Чтение в 8 потоков из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector4_stream8(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_000 = (__v2di*)&data_in[0 * count_in/8]; __v2di *in_001 = (__v2di*)&data_in[1 * count_in/8]; __v2di *in_010 = (__v2di*)&data_in[2 * count_in/8]; __v2di *in_011 = (__v2di*)&data_in[3 * count_in/8]; __v2di *in_100 = (__v2di*)&data_in[4 * count_in/8]; __v2di *in_101 = (__v2di*)&data_in[5 * count_in/8]; __v2di *in_110 = (__v2di*)&data_in[6 * count_in/8]; __v2di *in_111 = (__v2di*)&data_in[7 * count_in/8]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/8; ++i) { int64_t offset = 32 * (__builtin_e2k_bitrevd(i) >> shift_out); __v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110}; __v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_000[2*i+1], in_000[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_000[2*i+1], in_000[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_100[2*i+1], in_100[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_100[2*i+1], in_100[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_010[2*i+1], in_010[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_010[2*i+1], in_010[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_110[2*i+1], in_110[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_110[2*i+1], in_110[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_001[2*i+1], in_001[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_001[2*i+1], in_001[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_101[2*i+1], in_101[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_101[2*i+1], in_101[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_011[2*i+1], in_011[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_011[2*i+1], in_011[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_111[2*i+1], in_111[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_111[2*i+1], in_111[2*i+0], mask_imag); } }
Основной цикл на ассемблере
.L8681: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=8, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=3, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=3, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=3, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=3, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=24, disp=0 } .L7803: { loop_mode qppermb,1,sm %b[16], %b[15], %r20, %b[28] stqp,2 %r0, %b[9], %b[5] qppermb,4,sm %b[14], %b[13], %r20, %b[27] stqp,5 %r6, %b[9], %b[4] } { loop_mode qppermb,1,sm %b[26], %b[25], %r21, %b[30] stqp,2 %r2, %b[9], %b[28] qppermb,4,sm %b[24], %b[23], %r21, %b[29] stqp,5 %r5, %b[9], %b[27] movaqp,0 area=3, ind=0, am=1, be=0, %b[5] movaqp,1 area=3, ind=16, am=0, be=0, %b[8] movaqp,2 area=3, ind=0, am=1, be=0, %b[1] movaqp,3 area=3, ind=16, am=0, be=0, %b[4] } { loop_mode qppermb,1,sm %b[26], %b[25], %r20, %b[28] stqp,2 %r9, %b[9], %b[30] qppermb,4,sm %b[24], %b[23], %r20, %b[27] stqp,5 %r11, %b[9], %b[29] movaqp,0 area=0, ind=0, am=1, be=0, %b[13] movaqp,1 area=0, ind=16, am=0, be=0, %b[14] movaqp,2 area=0, ind=0, am=1, be=0, %b[11] movaqp,3 area=0, ind=16, am=0, be=0, %b[12] } { loop_mode qppermb,1,sm %b[22], %b[21], %r20, %b[29] stqp,2 %r7, %b[9], %b[28] qppermb,4,sm %b[22], %b[21], %r21, %b[26] stqp,5 %r10, %b[9], %b[27] movaqp,0 area=2, ind=0, am=1, be=0, %b[19] movaqp,1 area=2, ind=16, am=0, be=0, %b[20] movaqp,2 area=2, ind=0, am=1, be=0, %b[15] movaqp,3 area=2, ind=16, am=0, be=0, %b[16] } { loop_mode addd,0,sm %b[2], 0x1, %b[0] qppermb,1,sm %b[18], %b[17], %r20, %b[28] stqp,2 %r12, %b[9], %b[29] bitrevd,3,sm %b[2], %b[25] qppermb,4,sm %b[18], %b[17], %r21, %b[27] stqp,5 %r13, %b[9], %b[26] movaqp,0 area=1, ind=0, am=1, be=0, %b[23] movaqp,1 area=1, ind=16, am=0, be=0, %b[24] movaqp,2 area=1, ind=0, am=1, be=0, %b[21] movaqp,3 area=1, ind=16, am=0, be=0, %b[22] } { loop_mode qppermb,1,sm %b[10], %b[7], %r20, %b[29] stqp,2 %r14, %b[9], %b[28] shrd,3,sm %b[25], %r22, %b[2] qppermb,4,sm %b[10], %b[7], %r21, %b[26] stqp,5 %r15, %b[9], %b[27] } { loop_mode qppermb,1,sm %b[6], %b[3], %r20, %b[27] stqp,2 %r16, %b[9], %b[29] shld,3,sm %b[2], 0x5, %b[7] qppermb,4,sm %b[6], %b[3], %r21, %b[25] stqp,5 %r17, %b[9], %b[26] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[14], %b[13], %r21, %b[3] stqp,2 %r18, %b[9], %b[27] qppermb,3,sm %b[12], %b[11], %r21, %b[2] stqp,5 %r19, %b[9], %b[25] }
Теоретическая скорость: 32 комплексных числа за 8 тактов (32/8) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
7. reverse_radix2_vector4_stream16
Чтение в 16 потоков из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector4_stream16(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_0000 = (__v2di*)&data_in[ 0 * count_in/16]; __v2di *in_0001 = (__v2di*)&data_in[ 1 * count_in/16]; __v2di *in_0010 = (__v2di*)&data_in[ 2 * count_in/16]; __v2di *in_0011 = (__v2di*)&data_in[ 3 * count_in/16]; __v2di *in_0100 = (__v2di*)&data_in[ 4 * count_in/16]; __v2di *in_0101 = (__v2di*)&data_in[ 5 * count_in/16]; __v2di *in_0110 = (__v2di*)&data_in[ 6 * count_in/16]; __v2di *in_0111 = (__v2di*)&data_in[ 7 * count_in/16]; __v2di *in_1000 = (__v2di*)&data_in[ 8 * count_in/16]; __v2di *in_1001 = (__v2di*)&data_in[ 9 * count_in/16]; __v2di *in_1010 = (__v2di*)&data_in[10 * count_in/16]; __v2di *in_1011 = (__v2di*)&data_in[11 * count_in/16]; __v2di *in_1100 = (__v2di*)&data_in[12 * count_in/16]; __v2di *in_1101 = (__v2di*)&data_in[13 * count_in/16]; __v2di *in_1110 = (__v2di*)&data_in[14 * count_in/16]; __v2di *in_1111 = (__v2di*)&data_in[15 * count_in/16]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/16; ++i) { int64_t offset = 32 * (__builtin_e2k_bitrevd(i) >> shift_out); __v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110}; __v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_0000[2*i+1], in_0000[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_0000[2*i+1], in_0000[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_1000[2*i+1], in_1000[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_1000[2*i+1], in_1000[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_0100[2*i+1], in_0100[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_0100[2*i+1], in_0100[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_1100[2*i+1], in_1100[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_1100[2*i+1], in_1100[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_0010[2*i+1], in_0010[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_0010[2*i+1], in_0010[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_1010[2*i+1], in_1010[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_1010[2*i+1], in_1010[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_0110[2*i+1], in_0110[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_0110[2*i+1], in_0110[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_1110[2*i+1], in_1110[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_1110[2*i+1], in_1110[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 16*16) = __builtin_e2k_qppermb(in_0001[2*i+1], in_0001[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 17*16) = __builtin_e2k_qppermb(in_0001[2*i+1], in_0001[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 18*16) = __builtin_e2k_qppermb(in_1001[2*i+1], in_1001[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 19*16) = __builtin_e2k_qppermb(in_1001[2*i+1], in_1001[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 20*16) = __builtin_e2k_qppermb(in_0101[2*i+1], in_0101[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 21*16) = __builtin_e2k_qppermb(in_0101[2*i+1], in_0101[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 22*16) = __builtin_e2k_qppermb(in_1101[2*i+1], in_1101[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 23*16) = __builtin_e2k_qppermb(in_1101[2*i+1], in_1101[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 24*16) = __builtin_e2k_qppermb(in_0011[2*i+1], in_0011[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 25*16) = __builtin_e2k_qppermb(in_0011[2*i+1], in_0011[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 26*16) = __builtin_e2k_qppermb(in_1011[2*i+1], in_1011[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 27*16) = __builtin_e2k_qppermb(in_1011[2*i+1], in_1011[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 28*16) = __builtin_e2k_qppermb(in_0111[2*i+1], in_0111[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 29*16) = __builtin_e2k_qppermb(in_0111[2*i+1], in_0111[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 30*16) = __builtin_e2k_qppermb(in_1111[2*i+1], in_1111[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 31*16) = __builtin_e2k_qppermb(in_1111[2*i+1], in_1111[2*i+0], mask_imag); } }
Основной цикл на ассемблере
.L10479: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=1, incr=1, ind=0, asz=2, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=15, asz=2, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=14, asz=2, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=13, asz=2, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=12, asz=2, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=11, asz=2, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=10, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=9, asz=2, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=8, asz=2, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=2, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=2, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=2, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=2, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=2, abs=24, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=2, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=28, disp=0 } .L8820: { loop_mode qppermb,1,sm %b[34], %b[33], %r36, %b[32] stqp,2 %r0, %b[4], %b[44] qppermb,4,sm %b[32], %b[31], %r36, %b[31] stqp,5 %r6, %b[4], %b[43] } { loop_mode qppermb,1,sm %b[42], %b[41], %r37, %b[32] stqp,2 %r2, %b[4], %b[32] qppermb,4,sm %b[40], %b[39], %r37, %b[31] stqp,5 %r1, %b[4], %b[31] } { loop_mode qppermb,1,sm %b[42], %b[41], %r36, %b[32] stqp,2 %r9, %b[4], %b[32] qppermb,4,sm %b[40], %b[39], %r36, %b[31] stqp,5 %r11, %b[4], %b[31] } { loop_mode qppermb,1,sm %b[38], %b[37], %r36, %b[32] stqp,2 %r7, %b[4], %b[32] qppermb,4,sm %b[38], %b[37], %r37, %b[31] stqp,5 %r10, %b[4], %b[31] } { loop_mode qppermb,1,sm %b[36], %b[35], %r36, %b[32] stqp,2 %r12, %b[4], %b[32] qppermb,4,sm %b[36], %b[35], %r37, %b[31] stqp,5 %r13, %b[4], %b[31] } { loop_mode qppermb,1,sm %b[30], %b[29], %r36, %b[30] stqp,2 %r14, %b[4], %b[32] qppermb,4,sm %b[30], %b[29], %r37, %b[29] stqp,5 %r15, %b[4], %b[31] movaqp,0 area=7, ind=0, am=1, be=0, %b[5] movaqp,1 area=7, ind=16, am=0, be=0, %b[6] movaqp,2 area=7, ind=0, am=1, be=0, %b[0] movaqp,3 area=7, ind=16, am=0, be=0, %b[1] } { loop_mode qppermb,1,sm %b[26], %b[25], %r36, %b[26] stqp,2 %r16, %b[4], %b[30] qppermb,4,sm %b[26], %b[25], %r37, %b[25] stqp,5 %r17, %b[4], %b[29] movaqp,0 area=6, ind=0, am=1, be=0, %b[13] movaqp,1 area=6, ind=16, am=0, be=0, %b[14] movaqp,2 area=6, ind=0, am=1, be=0, %b[9] movaqp,3 area=6, ind=16, am=0, be=0, %b[10] } { loop_mode qppermb,1,sm %b[28], %b[27], %r36, %b[26] stqp,2 %r18, %b[4], %b[26] qppermb,4,sm %b[28], %b[27], %r37, %b[25] stqp,5 %r19, %b[4], %b[25] movaqp,0 area=5, ind=0, am=1, be=0, %b[21] movaqp,1 area=5, ind=16, am=0, be=0, %b[22] movaqp,2 area=5, ind=0, am=1, be=0, %b[17] movaqp,3 area=5, ind=16, am=0, be=0, %b[18] } { loop_mode qppermb,1,sm %g16, %g17, %r36, %b[28] stqp,2 %r20, %b[4], %b[26] qppermb,4,sm %g16, %g17, %r37, %b[27] stqp,5 %r21, %b[4], %b[25] movaqp,0 area=4, ind=0, am=1, be=0, %b[25] movaqp,1 area=4, ind=16, am=0, be=0, %b[26] movaqp,2 area=4, ind=0, am=1, be=0, %g17 movaqp,3 area=4, ind=16, am=0, be=0, %g16 } { loop_mode qppermb,1,sm %b[24], %b[23], %r36, %b[30] stqp,2 %r22, %b[4], %b[28] qppermb,4,sm %b[24], %b[23], %r37, %b[29] stqp,5 %r23, %b[4], %b[27] movaqp,0 area=3, ind=0, am=1, be=0, %b[27] movaqp,1 area=3, ind=16, am=0, be=0, %b[28] movaqp,2 area=3, ind=0, am=1, be=0, %b[23] movaqp,3 area=3, ind=16, am=0, be=0, %b[24] } { loop_mode qppermb,1,sm %b[20], %b[19], %r36, %b[34] stqp,2 %r24, %b[4], %b[30] qppermb,4,sm %b[20], %b[19], %r37, %b[33] stqp,5 %r25, %b[4], %b[29] movaqp,0 area=0, ind=0, am=1, be=0, %b[31] movaqp,1 area=0, ind=16, am=0, be=0, %b[32] movaqp,2 area=0, ind=0, am=1, be=0, %b[29] movaqp,3 area=0, ind=16, am=0, be=0, %b[30] } { loop_mode qppermb,1,sm %b[16], %b[15], %r36, %b[38] stqp,2 %r26, %b[4], %b[34] qppermb,4,sm %b[16], %b[15], %r37, %b[37] stqp,5 %r27, %b[4], %b[33] movaqp,0 area=2, ind=0, am=1, be=0, %b[35] movaqp,1 area=2, ind=16, am=0, be=0, %b[36] movaqp,2 area=2, ind=0, am=1, be=0, %b[33] movaqp,3 area=2, ind=16, am=0, be=0, %b[34] } { loop_mode addd,0,sm %r5, 0x1, %r5 qppermb,1,sm %b[12], %b[11], %r36, %b[43] stqp,2 %r28, %b[4], %b[38] bitrevd,3,sm %r5, %b[41] qppermb,4,sm %b[12], %b[11], %r37, %b[42] stqp,5 %r29, %b[4], %b[37] movaqp,0 area=1, ind=0, am=1, be=0, %b[39] movaqp,1 area=1, ind=16, am=0, be=0, %b[40] movaqp,2 area=1, ind=0, am=1, be=0, %b[37] movaqp,3 area=1, ind=16, am=0, be=0, %b[38] } { loop_mode qppermb,1,sm %b[8], %b[7], %r36, %b[43] stqp,2 %r30, %b[4], %b[43] shrd,3,sm %b[41], %r38, %b[41] qppermb,4,sm %b[8], %b[7], %r37, %b[42] stqp,5 %r31, %b[4], %b[42] } { loop_mode qppermb,1,sm %b[3], %b[2], %r36, %b[42] stqp,2 %r32, %b[4], %b[43] shld,3,sm %b[41], 0x5, %b[2] qppermb,4,sm %b[3], %b[2], %r37, %b[41] stqp,5 %r33, %b[4], %b[42] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[32], %b[31], %r37, %b[42] stqp,2 %r34, %b[4], %b[42] qppermb,3,sm %b[30], %b[29], %r37, %b[41] stqp,5 %r35, %b[4], %b[41] }
Теоретическая скорость: 64 комплексных числа за 16 тактов (64/16) = 32 Байт/такт
Замеры скорости
Видим замедление по всей длине графика.
8. reverse_radix2_vector4_stream32
Чтение в 32 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix2_vector4_stream32(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_00000 = (__v2di*)&data_in[ 0 * count_in/32]; __v2di *in_00001 = (__v2di*)&data_in[ 1 * count_in/32]; __v2di *in_00010 = (__v2di*)&data_in[ 2 * count_in/32]; __v2di *in_00011 = (__v2di*)&data_in[ 3 * count_in/32]; __v2di *in_00100 = (__v2di*)&data_in[ 4 * count_in/32]; __v2di *in_00101 = (__v2di*)&data_in[ 5 * count_in/32]; __v2di *in_00110 = (__v2di*)&data_in[ 6 * count_in/32]; __v2di *in_00111 = (__v2di*)&data_in[ 7 * count_in/32]; __v2di *in_01000 = (__v2di*)&data_in[ 8 * count_in/32]; __v2di *in_01001 = (__v2di*)&data_in[ 9 * count_in/32]; __v2di *in_01010 = (__v2di*)&data_in[10 * count_in/32]; __v2di *in_01011 = (__v2di*)&data_in[11 * count_in/32]; __v2di *in_01100 = (__v2di*)&data_in[12 * count_in/32]; __v2di *in_01101 = (__v2di*)&data_in[13 * count_in/32]; __v2di *in_01110 = (__v2di*)&data_in[14 * count_in/32]; __v2di *in_01111 = (__v2di*)&data_in[15 * count_in/32]; __v2di *in_10000 = (__v2di*)&data_in[16 * count_in/32]; __v2di *in_10001 = (__v2di*)&data_in[17 * count_in/32]; __v2di *in_10010 = (__v2di*)&data_in[18 * count_in/32]; __v2di *in_10011 = (__v2di*)&data_in[19 * count_in/32]; __v2di *in_10100 = (__v2di*)&data_in[20 * count_in/32]; __v2di *in_10101 = (__v2di*)&data_in[21 * count_in/32]; __v2di *in_10110 = (__v2di*)&data_in[22 * count_in/32]; __v2di *in_10111 = (__v2di*)&data_in[23 * count_in/32]; __v2di *in_11000 = (__v2di*)&data_in[24 * count_in/32]; __v2di *in_11001 = (__v2di*)&data_in[25 * count_in/32]; __v2di *in_11010 = (__v2di*)&data_in[26 * count_in/32]; __v2di *in_11011 = (__v2di*)&data_in[27 * count_in/32]; __v2di *in_11100 = (__v2di*)&data_in[28 * count_in/32]; __v2di *in_11101 = (__v2di*)&data_in[29 * count_in/32]; __v2di *in_11110 = (__v2di*)&data_in[30 * count_in/32]; __v2di *in_11111 = (__v2di*)&data_in[31 * count_in/32]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/32; ++i) { int64_t offset = 32 * (__builtin_e2k_bitrevd(i) >> shift_out); __v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110}; __v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_00000[2*i+1], in_00000[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_00000[2*i+1], in_00000[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_10000[2*i+1], in_10000[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_10000[2*i+1], in_10000[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_01000[2*i+1], in_01000[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_01000[2*i+1], in_01000[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_11000[2*i+1], in_11000[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_11000[2*i+1], in_11000[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_00100[2*i+1], in_00100[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_00100[2*i+1], in_00100[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_10100[2*i+1], in_10100[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_10100[2*i+1], in_10100[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_01100[2*i+1], in_01100[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_01100[2*i+1], in_01100[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_11100[2*i+1], in_11100[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_11100[2*i+1], in_11100[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 16*16) = __builtin_e2k_qppermb(in_00010[2*i+1], in_00010[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 17*16) = __builtin_e2k_qppermb(in_00010[2*i+1], in_00010[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 18*16) = __builtin_e2k_qppermb(in_10010[2*i+1], in_10010[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 19*16) = __builtin_e2k_qppermb(in_10010[2*i+1], in_10010[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 20*16) = __builtin_e2k_qppermb(in_01010[2*i+1], in_01010[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 21*16) = __builtin_e2k_qppermb(in_01010[2*i+1], in_01010[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 22*16) = __builtin_e2k_qppermb(in_11010[2*i+1], in_11010[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 23*16) = __builtin_e2k_qppermb(in_11010[2*i+1], in_11010[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 24*16) = __builtin_e2k_qppermb(in_00110[2*i+1], in_00110[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 25*16) = __builtin_e2k_qppermb(in_00110[2*i+1], in_00110[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 26*16) = __builtin_e2k_qppermb(in_10110[2*i+1], in_10110[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 27*16) = __builtin_e2k_qppermb(in_10110[2*i+1], in_10110[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 28*16) = __builtin_e2k_qppermb(in_01110[2*i+1], in_01110[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 29*16) = __builtin_e2k_qppermb(in_01110[2*i+1], in_01110[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 30*16) = __builtin_e2k_qppermb(in_11110[2*i+1], in_11110[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 31*16) = __builtin_e2k_qppermb(in_11110[2*i+1], in_11110[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 32*16) = __builtin_e2k_qppermb(in_00001[2*i+1], in_00001[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 33*16) = __builtin_e2k_qppermb(in_00001[2*i+1], in_00001[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 34*16) = __builtin_e2k_qppermb(in_10001[2*i+1], in_10001[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 35*16) = __builtin_e2k_qppermb(in_10001[2*i+1], in_10001[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 36*16) = __builtin_e2k_qppermb(in_01001[2*i+1], in_01001[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 37*16) = __builtin_e2k_qppermb(in_01001[2*i+1], in_01001[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 38*16) = __builtin_e2k_qppermb(in_11001[2*i+1], in_11001[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 39*16) = __builtin_e2k_qppermb(in_11001[2*i+1], in_11001[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 40*16) = __builtin_e2k_qppermb(in_00101[2*i+1], in_00101[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 41*16) = __builtin_e2k_qppermb(in_00101[2*i+1], in_00101[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 42*16) = __builtin_e2k_qppermb(in_10101[2*i+1], in_10101[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 43*16) = __builtin_e2k_qppermb(in_10101[2*i+1], in_10101[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 44*16) = __builtin_e2k_qppermb(in_01101[2*i+1], in_01101[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 45*16) = __builtin_e2k_qppermb(in_01101[2*i+1], in_01101[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 46*16) = __builtin_e2k_qppermb(in_11101[2*i+1], in_11101[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 47*16) = __builtin_e2k_qppermb(in_11101[2*i+1], in_11101[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 48*16) = __builtin_e2k_qppermb(in_00011[2*i+1], in_00011[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 49*16) = __builtin_e2k_qppermb(in_00011[2*i+1], in_00011[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 50*16) = __builtin_e2k_qppermb(in_10011[2*i+1], in_10011[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 51*16) = __builtin_e2k_qppermb(in_10011[2*i+1], in_10011[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 52*16) = __builtin_e2k_qppermb(in_01011[2*i+1], in_01011[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 53*16) = __builtin_e2k_qppermb(in_01011[2*i+1], in_01011[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 54*16) = __builtin_e2k_qppermb(in_11011[2*i+1], in_11011[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 55*16) = __builtin_e2k_qppermb(in_11011[2*i+1], in_11011[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 56*16) = __builtin_e2k_qppermb(in_00111[2*i+1], in_00111[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 57*16) = __builtin_e2k_qppermb(in_00111[2*i+1], in_00111[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 58*16) = __builtin_e2k_qppermb(in_10111[2*i+1], in_10111[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 59*16) = __builtin_e2k_qppermb(in_10111[2*i+1], in_10111[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 60*16) = __builtin_e2k_qppermb(in_01111[2*i+1], in_01111[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 61*16) = __builtin_e2k_qppermb(in_01111[2*i+1], in_01111[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 62*16) = __builtin_e2k_qppermb(in_11111[2*i+1], in_11111[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 63*16) = __builtin_e2k_qppermb(in_11111[2*i+1], in_11111[2*i+0], mask_imag); } }
Основной цикл на ассемблере
.L13622: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=17, incr=1, ind=0, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=16, incr=1, ind=0, asz=1, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=15, incr=1, ind=0, asz=1, abs=2, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=14, incr=1, ind=0, asz=1, abs=2, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=13, incr=1, ind=0, asz=1, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=12, incr=1, ind=0, asz=1, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=11, incr=1, ind=0, asz=1, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=10, incr=1, ind=0, asz=1, abs=6, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=9, incr=1, ind=0, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=8, incr=1, ind=0, asz=1, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=7, incr=1, ind=0, asz=1, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=6, incr=1, ind=0, asz=1, abs=10, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=5, incr=1, ind=0, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=4, incr=1, ind=0, asz=1, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=3, incr=1, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=2, incr=1, ind=0, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=1, incr=1, ind=0, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=15, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=14, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=13, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=12, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=11, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=10, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=9, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=8, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=30, disp=0 } .L13394: { loop_mode disp %ctpr1, .L13394 bitrevd,0 %r1, %g20 addd,1 %r1, 0x1, %r1 movaqp,0 area=0, ind=16, am=1, be=0, %g17 movaqp,1 area=0, ind=0, am=0, be=0, %g16 movaqp,2 area=0, ind=16, am=1, be=0, %g19 movaqp,3 area=0, ind=0, am=0, be=0, %g18 } { loop_mode disp %ctpr3, .L12302 shrd,0 %g20, %r6, %g20 movaqp,0 area=1, ind=16, am=1, be=0, %g22 movaqp,1 area=1, ind=0, am=0, be=0, %g21 movaqp,2 area=1, ind=16, am=1, be=0, %g24 movaqp,3 area=1, ind=0, am=0, be=0, %g23 } { loop_mode shld,0 %g20, 0x5, %g20 movaqp,0 area=2, ind=16, am=1, be=0, %g26 movaqp,1 area=2, ind=0, am=0, be=0, %g25 movaqp,2 area=2, ind=16, am=1, be=0, %g28 movaqp,3 area=2, ind=0, am=0, be=0, %g27 } { loop_mode addd,0 %r2, %g20, %r7 movaqp,0 area=3, ind=16, am=1, be=0, %g30 movaqp,1 area=3, ind=0, am=0, be=0, %g29 movaqp,2 area=3, ind=16, am=1, be=0, %r4 movaqp,3 area=3, ind=0, am=0, be=0, %g31 } { loop_mode movaqp,0 area=4, ind=16, am=1, be=0, %r10 movaqp,1 area=4, ind=0, am=0, be=0, %r9 movaqp,2 area=4, ind=16, am=1, be=0, %r12 movaqp,3 area=4, ind=0, am=0, be=0, %r11 } { loop_mode qppermb,0 %g17, %g16, %r0, %r17 qppermb,1 %g17, %g16, %r5, %g16 qppermb,3 %g19, %g18, %r0, %g17 qppermb,4 %g19, %g18, %r5, %g18 movaqp,0 area=5, ind=16, am=1, be=0, %r14 movaqp,1 area=5, ind=0, am=0, be=0, %r13 movaqp,2 area=5, ind=16, am=1, be=0, %r16 movaqp,3 area=5, ind=0, am=0, be=0, %r15 } { loop_mode qppermb,0 %g22, %g21, %r0, %r21 qppermb,1 %g22, %g21, %r5, %g21 stqp,2 0x10, %r7, %g16 qppermb,3 %g24, %g23, %r0, %g22 qppermb,4 %g24, %g23, %r5, %g23 stqp,5 %r7, _f16s,_lts0lo 0x30, %g18 movaqp,0 area=6, ind=16, am=1, be=0, %r18 movaqp,1 area=6, ind=0, am=0, be=0, %g19 movaqp,2 area=6, ind=16, am=1, be=0, %r20 movaqp,3 area=6, ind=0, am=0, be=0, %r19 } { loop_mode qppermb,0 %g26, %g25, %r0, %g17 qppermb,1 %g26, %g25, %r5, %g20 stqp,2 %r2, %g20, %r17 qppermb,3 %g28, %g27, %r0, %g25 qppermb,4 %g28, %g27, %r5, %g26 stqp,5 %r7, _f16s,_lts0lo 0x20, %g17 movaqp,0 area=7, ind=16, am=1, be=0, %g18 movaqp,1 area=7, ind=0, am=0, be=0, %g16 movaqp,2 area=7, ind=16, am=1, be=0, %r22 movaqp,3 area=7, ind=0, am=0, be=0, %g24 } { loop_mode qppermb,0 %g30, %g29, %r0, %r24 qppermb,1 %g30, %g29, %r5, %g29 stqp,2 %r7, _f16s,_lts0lo 0x50, %g21 qppermb,3 %r4, %g31, %r0, %g21 qppermb,4 %r4, %g31, %r5, %g23 stqp,5 %r7, _f16s,_lts0hi 0x70, %g23 movaqp,0 area=8, ind=16, am=1, be=0, %g28 movaqp,1 area=8, ind=0, am=0, be=0, %g27 movaqp,2 area=8, ind=16, am=1, be=0, %r23 movaqp,3 area=8, ind=0, am=0, be=0, %r17 } { loop_mode qppermb,0 %r10, %r9, %r0, %g22 qppermb,1 %r10, %r9, %r5, %r9 stqp,2 %r7, _f16s,_lts0lo 0x40, %r21 qppermb,3 %r12, %r11, %r0, %r10 qppermb,4 %r12, %r11, %r5, %r11 stqp,5 %r7, _f16s,_lts0hi 0x60, %g22 movaqp,0 area=9, ind=16, am=1, be=0, %g31 movaqp,1 area=9, ind=0, am=0, be=0, %g30 movaqp,2 area=9, ind=16, am=1, be=0, %r25 movaqp,3 area=9, ind=0, am=0, be=0, %r4 } { loop_mode qppermb,0 %r14, %r13, %r0, %g17 qppermb,1 %r14, %r13, %r5, %g20 stqp,2 %r7, _f16s,_lts0lo 0x80, %g17 qppermb,3 %r16, %r15, %r0, %r13 qppermb,4 %r16, %r15, %r5, %r14 stqp,5 %r7, _f16s,_lts0hi 0x90, %g20 movaqp,0 area=10, ind=16, am=1, be=0, %r21 movaqp,1 area=10, ind=0, am=0, be=0, %r12 movaqp,2 area=10, ind=16, am=1, be=0, %r27 movaqp,3 area=10, ind=0, am=0, be=0, %r26 } { loop_mode qppermb,0 %r18, %g19, %r0, %r28 qppermb,1 %r18, %g19, %r5, %g19 stqp,2 %r7, _f16s,_lts0lo 0xa0, %g25 qppermb,3 %r20, %r19, %r0, %r18 qppermb,4 %r20, %r19, %r5, %r19 stqp,5 %r7, _f16s,_lts0hi 0xb0, %g26 movaqp,0 area=11, ind=16, am=1, be=0, %r16 movaqp,1 area=11, ind=0, am=0, be=0, %r15 movaqp,2 area=11, ind=16, am=1, be=0, %g26 movaqp,3 area=11, ind=0, am=0, be=0, %g25 } { loop_mode qppermb,0 %g18, %g16, %r0, %r30 qppermb,1 %g18, %g16, %r5, %g16 stqp,2 %r7, _f16s,_lts0lo 0xc0, %r24 qppermb,3 %r22, %g24, %r0, %g18 qppermb,4 %r22, %g24, %r5, %g24 stqp,5 %r7, _f16s,_lts0hi 0xd0, %g29 movaqp,0 area=12, ind=16, am=1, be=0, %r20 movaqp,1 area=12, ind=0, am=0, be=0, %g29 movaqp,2 area=12, ind=16, am=1, be=0, %r29 movaqp,3 area=12, ind=0, am=0, be=0, %r24 } { loop_mode qppermb,0 %g28, %g27, %r0, %r32 qppermb,1 %g28, %g27, %r5, %g27 stqp,2 %r7, _f16s,_lts0lo 0xe0, %g21 qppermb,3 %r23, %r17, %r0, %g28 qppermb,4 %r23, %r17, %r5, %r17 stqp,5 %r7, _f16s,_lts0hi 0xf0, %g23 movaqp,0 area=13, ind=16, am=1, be=0, %g23 movaqp,1 area=13, ind=0, am=0, be=0, %g21 movaqp,2 area=13, ind=16, am=1, be=0, %r31 movaqp,3 area=13, ind=0, am=0, be=0, %r22 } { loop_mode qppermb,0 %g31, %g30, %r0, %r34 qppermb,1 %g31, %g30, %r5, %g30 stqp,2 %r7, _f16s,_lts0lo 0x100, %g22 qppermb,3 %r25, %r4, %r0, %g31 qppermb,4 %r25, %r4, %r5, %r4 stqp,5 %r7, _f16s,_lts0hi 0x110, %r9 movaqp,0 area=14, ind=16, am=1, be=0, %r9 movaqp,1 area=14, ind=0, am=0, be=0, %g22 movaqp,2 area=14, ind=16, am=1, be=0, %r33 movaqp,3 area=14, ind=0, am=0, be=0, %r23 } { loop_mode qppermb,0 %r21, %r12, %r0, %r36 qppermb,1 %r21, %r12, %r5, %r12 stqp,2 %r7, _f16s,_lts0lo 0x120, %r10 qppermb,3 %r27, %r26, %r0, %r21 qppermb,4 %r27, %r26, %r5, %r26 stqp,5 %r7, _f16s,_lts0hi 0x130, %r11 movaqp,0 area=15, ind=16, am=1, be=0, %r11 movaqp,1 area=15, ind=0, am=0, be=0, %r10 movaqp,2 area=15, ind=16, am=1, be=0, %r35 movaqp,3 area=15, ind=0, am=0, be=0, %r25 } { loop_mode qppermb,0 %r16, %r15, %r0, %g17 qppermb,1 %r16, %r15, %r5, %g20 stqp,2 %r7, _f16s,_lts0lo 0x140, %g17 qppermb,3 %g26, %g25, %r0, %r15 qppermb,4 %g26, %g25, %r5, %g25 stqp,5 %r7, _f16s,_lts0hi 0x150, %g20 } { loop_mode qppermb,0 %r20, %g29, %r0, %g26 qppermb,1 %r20, %g29, %r5, %g29 stqp,2 %r7, _f16s,_lts0lo 0x160, %r13 qppermb,3 %r29, %r24, %r0, %r13 qppermb,4 %r29, %r24, %r5, %r14 stqp,5 %r7, _f16s,_lts0hi 0x170, %r14 } { loop_mode qppermb,0 %g23, %g21, %r0, %g19 qppermb,1 %g23, %g21, %r5, %g21 stqp,2 %r7, _f16s,_lts0lo 0x180, %r28 qppermb,3 %r31, %r22, %r0, %g23 qppermb,4 %r31, %r22, %r5, %r16 stqp,5 %r7, _f16s,_lts0hi 0x190, %g19 } { loop_mode qppermb,0 %r9, %g22, %r0, %r18 qppermb,1 %r9, %g22, %r5, %g22 stqp,2 %r7, _f16s,_lts0lo 0x1a0, %r18 qppermb,3 %r33, %r23, %r0, %r9 qppermb,4 %r33, %r23, %r5, %r19 stqp,5 %r7, _f16s,_lts0hi 0x1b0, %r19 } { loop_mode qppermb,0 %r11, %r10, %r0, %g16 qppermb,1 %r11, %r10, %r5, %r10 stqp,2 %r7, _f16s,_lts0lo 0x1c0, %r30 qppermb,3 %r35, %r25, %r0, %r11 qppermb,4 %r35, %r25, %r5, %r20 stqp,5 %r7, _f16s,_lts0hi 0x1d0, %g16 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x1e0, %g18 stqp,5 %r7, _f16s,_lts0hi 0x1f0, %g24 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x200, %r32 stqp,5 %r7, _f16s,_lts0hi 0x210, %g27 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x220, %g28 stqp,5 %r7, _f16s,_lts0hi 0x230, %r17 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x240, %r34 stqp,5 %r7, _f16s,_lts0hi 0x250, %g30 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x260, %g31 stqp,5 %r7, _f16s,_lts0hi 0x270, %r4 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x280, %r36 stqp,5 %r7, _f16s,_lts0hi 0x290, %r12 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x2a0, %r21 stqp,5 %r7, _f16s,_lts0hi 0x2b0, %r26 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x2c0, %g17 stqp,5 %r7, _f16s,_lts0hi 0x2d0, %g20 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x2e0, %r15 stqp,5 %r7, _f16s,_lts0hi 0x2f0, %g25 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x300, %g26 stqp,5 %r7, _f16s,_lts0hi 0x310, %g29 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x320, %r13 stqp,5 %r7, _f16s,_lts0hi 0x330, %r14 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x340, %g19 stqp,5 %r7, _f16s,_lts0hi 0x350, %g21 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x360, %g23 stqp,5 %r7, _f16s,_lts0hi 0x370, %r16 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x380, %r18 stqp,5 %r7, _f16s,_lts0hi 0x390, %g22 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x3a0, %r9 stqp,5 %r7, _f16s,_lts0hi 0x3b0, %r19 } { loop_mode stqp,2 %r7, _f16s,_lts0lo 0x3c0, %g16 stqp,5 %r7, _f16s,_lts0hi 0x3d0, %r10 } { loop_mode ct %ctpr1 ? %NOT_LOOP_END alc alcf=0, alct=1 stqp,2 %r7, _f16s,_lts0lo 0x3e0, %r11 stqp,5 %r7, _f16s,_lts0hi 0x3f0, %r20 } { loop_mode ct %ctpr3 alc alcf=0, alct=1 }
Теоретическая скорость: 128 комплексных чисел за 39 тактов (128/39) = 26.26 Байт/такт
Замеры скорости
Видим замедление по всей длине графика.
При попытке чтения в 64 потока получается резко менее эффективный код.
APB не может читать в 64 потока, поэтому в дополнение к APB компилятор вставляет обычные операции чтения ldqp. В итоге скорость резко проседает.
Итоги по reverse_radix2_vector4
Победителем можно считать reverse_radix2_vector4_stream8.
При реализации Vector-4 Radix-2 FFT будем использовать его.
reverse_radix4_vector2
1. reverse_radix4_vector2_etalon
Эталонный вариант для сравнения на корректность.
Здесь reverseNumber_radix4 вычисляется с помощью цикла.
int reverseNumber_radix4(int number, int bit_count) { int answer = 0; for(int i = 0; i < bit_count/2; ++i) { answer <<= 2; answer |= number & 3; number >>= 2; } return answer; } void reverse_radix4_vector2_etalon(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; // int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; for(int64_t i = 0; i < count_out; ++i) { int index = reverseNumber_radix4(i, bit_count_out); myComplex a = data_in[2*i + 0]; myComplex b = data_in[2*i + 1]; data_out[index] = (myComplex2){.real = {a.real, b.real}, .imag = {a.imag, b.imag}}; } }
2. reverse_radix4_vector2
В процессоре нет готовых инструкций, производящих операцию reverseNumber_radix4.
Поэтому выполним инструкцию bitrevd и переставим соседние биты местами.
Код на Си
void reverse_radix4_vector2(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; // int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in = (__v2di*)data_in; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 16 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555)); __v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504}; *(__v2du*)((void*)data_out + offset) = __builtin_e2k_qppermb(in[i], in[i], mask); } }
Основной цикл на ассемблере
.L4546: { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=5, abs=0, disp=0 } .L4397: { loop_mode bitrevd,0,sm %b[10], %b[1] shr_andd,1,sm %b[3], %r5, %r6, %b[9] ord,2,sm %b[11], %b[7], %b[12] shld,3,sm %b[14], 0x4, %b[15] qppermb,4,sm %b[6], %b[6], %r0, %b[13] addd,5,sm %b[10], 0x1, %b[8] movaqp,1 area=0, ind=0, am=1, be=0, %b[0] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END shr_andd,1,sm %b[1], %r4, %r7, %b[3] stqp,5 %r2, %b[15], %b[13] }
Теоретическая скорость: 2 комплексных числа за 2 такта (2/2) = 8 Байт/такт
Замеры скорости
3. reverse_radix4_vector2_x4
Сделаем ручную раскрутку цикла в 4 раза.
Код на Си
void reverse_radix4_vector2_x4(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; // int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in = (__v2di*) data_in; __v2di *out_0 = (__v2di*)&data_out[0 * count_out/4]; __v2di *out_1 = (__v2di*)&data_out[1 * count_out/4]; __v2di *out_2 = (__v2di*)&data_out[2 * count_out/4]; __v2di *out_3 = (__v2di*)&data_out[3 * count_out/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out; i += 4) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 16 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555)); __v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504}; *(__v2du*)((void*)out_0 + offset) = __builtin_e2k_qppermb(in[i + 0], in[i + 0], mask); *(__v2du*)((void*)out_1 + offset) = __builtin_e2k_qppermb(in[i + 1], in[i + 1], mask); *(__v2du*)((void*)out_2 + offset) = __builtin_e2k_qppermb(in[i + 2], in[i + 2], mask); *(__v2du*)((void*)out_3 + offset) = __builtin_e2k_qppermb(in[i + 3], in[i + 3], mask); } }
Основной цикл на ассемблере
.L4888: { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=32 } .L4593: { loop_mode qppermb,0,sm %b[11], %b[11], %r7, %b[20] shr_andd,1,sm %b[26], %r9, %r11, %b[16] qppermb,3,sm %b[12], %b[12], %r7, %b[23] shr_andd,4,sm %b[26], %r6, %r10, %b[19] addd,5,sm %b[17], 0x4, %b[15] movaqp,0 area=0, ind=16, am=1, be=0, %b[7] movaqp,1 area=0, ind=0, am=0, be=0, %b[8] movaqp,2 area=0, ind=16, am=1, be=0, %b[0] movaqp,3 area=0, ind=0, am=0, be=0, %b[1] } { loop_mode bitrevd,0,sm %b[17], %b[24] qppermb,1,sm %b[4], %b[4], %r7, %b[11] stqp,2 %r5, %b[6], %b[22] ord,3,sm %b[21], %b[18], %b[27] qppermb,4,sm %b[5], %b[5], %r7, %b[12] stqp,5 %r2, %b[6], %b[25] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END stqp,2 %r0, %b[6], %b[13] shld,4,sm %b[27], 0x4, %b[4] stqp,5 %r4, %b[6], %b[14] }
Теоретическая скорость: 8 комплексных чисел за 3 такта (8/3) = 21.33 Байт/такт
Замеры скорости
Видим ускорение в начале графика.
Здесь происходит четыре чтения из одного места памяти и запись в четыре разных места памяти.
4. reverse_radix4_vector2_stream4
Теперь сделаем наоборот: будем читать из четырёх разных мест, а писать рядом.
Чтение в 4 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix4_vector2_stream4(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_0 = (__v2di*)&data_in[0 * count_in/4]; __v2di *in_1 = (__v2di*)&data_in[1 * count_in/4]; __v2di *in_2 = (__v2di*)&data_in[2 * count_in/4]; __v2di *in_3 = (__v2di*)&data_in[3 * count_in/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/4; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 16 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555)); __v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_0[i], in_0[i], mask); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_1[i], in_1[i], mask); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_2[i], in_2[i], mask); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_3[i], in_3[i], mask); } }
Основной цикл на ассемблере
.L5211: { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=4, abs=0, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=4, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=4, abs=16, disp=0 } .L4931: { loop_mode qppermb,0,sm %b[11], %b[11], %r7, %b[20] shr_andd,1,sm %b[26], %r9, %r11, %b[16] qppermb,3,sm %b[12], %b[12], %r7, %b[23] shr_andd,4,sm %b[26], %r6, %r10, %b[19] addd,5,sm %b[17], 0x1, %b[15] movaqp,0 area=1, ind=0, am=1, be=0, %b[1] movaqp,1 area=0, ind=0, am=1, be=0, %b[8] movaqp,2 area=1, ind=0, am=1, be=0, %b[0] movaqp,3 area=0, ind=0, am=1, be=0, %b[7] } { loop_mode bitrevd,0,sm %b[17], %b[24] qppermb,1,sm %b[4], %b[4], %r7, %b[11] stqp,2 %r0, %b[6], %b[22] ord,3,sm %b[21], %b[18], %b[27] qppermb,4,sm %b[5], %b[5], %r7, %b[12] stqp,5 %r2, %b[6], %b[25] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END stqp,2 %r5, %b[6], %b[13] shld,4,sm %b[27], 0x4, %b[4] stqp,5 %r4, %b[6], %b[14] }
Теоретическая скорость: 8 комплексных чисел за 3 такта (8/3) = 21.33 Байт/такт
Замеры скорости
Видим желаемое ускорение по всей длине графика.
5. reverse_radix4_vector2_stream16
Чтение в 16 потоков из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix4_vector2_stream16(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_00 = (__v2di*)&data_in[ 0 * count_in/16]; __v2di *in_01 = (__v2di*)&data_in[ 1 * count_in/16]; __v2di *in_02 = (__v2di*)&data_in[ 2 * count_in/16]; __v2di *in_03 = (__v2di*)&data_in[ 3 * count_in/16]; __v2di *in_10 = (__v2di*)&data_in[ 4 * count_in/16]; __v2di *in_11 = (__v2di*)&data_in[ 5 * count_in/16]; __v2di *in_12 = (__v2di*)&data_in[ 6 * count_in/16]; __v2di *in_13 = (__v2di*)&data_in[ 7 * count_in/16]; __v2di *in_20 = (__v2di*)&data_in[ 8 * count_in/16]; __v2di *in_21 = (__v2di*)&data_in[ 9 * count_in/16]; __v2di *in_22 = (__v2di*)&data_in[10 * count_in/16]; __v2di *in_23 = (__v2di*)&data_in[11 * count_in/16]; __v2di *in_30 = (__v2di*)&data_in[12 * count_in/16]; __v2di *in_31 = (__v2di*)&data_in[13 * count_in/16]; __v2di *in_32 = (__v2di*)&data_in[14 * count_in/16]; __v2di *in_33 = (__v2di*)&data_in[15 * count_in/16]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/16; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 16 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555)); __v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_00[i], in_00[i], mask); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_10[i], in_10[i], mask); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_20[i], in_20[i], mask); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_30[i], in_30[i], mask); *(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_01[i], in_01[i], mask); *(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_11[i], in_11[i], mask); *(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_21[i], in_21[i], mask); *(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_31[i], in_31[i], mask); *(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_02[i], in_02[i], mask); *(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_12[i], in_12[i], mask); *(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_22[i], in_22[i], mask); *(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_32[i], in_32[i], mask); *(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_03[i], in_03[i], mask); *(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_13[i], in_13[i], mask); *(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_23[i], in_23[i], mask); *(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_33[i], in_33[i], mask); } }
Основной цикл на ассемблере
.L6141: { fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=2, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=2, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=2, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=2, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=2, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=2, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=2, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=2, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=2, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=2, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=2, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=2, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=2, abs=24, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=2, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=2, abs=28, disp=0 } .L5350: { loop_mode qppermb,0,sm %b[12], %b[12], %r20, %b[46] shr_andd,1,sm %b[24], %r21, %r24, %b[1] stqp,2 %r0, %b[21], %b[4] qppermb,3,sm %b[17], %b[17], %r20, %b[47] shr_andd,4,sm %b[24], %r22, %r23, %b[5] stqp,5 %r2, %b[21], %b[16] } { loop_mode qppermb,1,sm %b[41], %b[41], %r20, %b[4] stqp,2 %r6, %b[21], %b[46] qppermb,4,sm %b[42], %b[42], %r20, %b[9] stqp,5 %r5, %b[21], %b[47] } { loop_mode qppermb,1,sm %b[40], %b[40], %r20, %b[24] stqp,2 %r10, %b[21], %b[6] qppermb,4,sm %b[39], %b[39], %r20, %b[23] stqp,5 %r7, %b[21], %b[11] movaqp,0 area=7, ind=0, am=1, be=0, %b[16] movaqp,1 area=6, ind=0, am=1, be=0, %b[20] movaqp,2 area=7, ind=0, am=1, be=0, %b[12] movaqp,3 area=6, ind=0, am=1, be=0, %b[17] } { loop_mode qppermb,1,sm %b[38], %b[38], %r20, %b[30] stqp,2 %r9, %b[21], %b[26] qppermb,4,sm %b[37], %b[37], %r20, %b[29] stqp,5 %r11, %b[21], %b[25] movaqp,0 area=5, ind=0, am=1, be=0, %b[11] movaqp,1 area=4, ind=0, am=1, be=0, %b[36] movaqp,2 area=5, ind=0, am=1, be=0, %b[6] movaqp,3 area=4, ind=0, am=1, be=0, %b[35] } { loop_mode qppermb,1,sm %b[13], %b[13], %r20, %b[26] stqp,2 %r12, %b[21], %b[32] qppermb,4,sm %b[8], %b[8], %r20, %b[25] stqp,5 %r13, %b[21], %b[31] movaqp,0 area=3, ind=0, am=1, be=0, %b[39] movaqp,1 area=2, ind=0, am=1, be=0, %b[40] movaqp,2 area=3, ind=0, am=1, be=0, %b[37] movaqp,3 area=2, ind=0, am=1, be=0, %b[38] } { loop_mode addd,0,sm %b[2], 0x1, %b[0] qppermb,1,sm %b[22], %b[22], %r20, %b[42] stqp,2 %r14, %b[21], %b[28] ord,3,sm %b[5], %b[1], %b[45] qppermb,4,sm %b[19], %b[19], %r20, %b[41] stqp,5 %r15, %b[21], %b[27] movaqp,0 area=1, ind=0, am=1, be=0, %b[13] movaqp,1 area=0, ind=0, am=1, be=0, %b[32] movaqp,2 area=1, ind=0, am=1, be=0, %b[8] movaqp,3 area=0, ind=0, am=1, be=0, %b[31] } { loop_mode bitrevd,0,sm %b[2], %b[22] qppermb,1,sm %b[18], %b[18], %r20, %b[5] stqp,2 %r16, %b[21], %b[44] shld,3,sm %b[45], 0x4, %b[19] qppermb,4,sm %b[14], %b[14], %r20, %b[1] stqp,5 %r17, %b[21], %b[43] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[33], %b[33], %r20, %b[2] stqp,2 %r18, %b[21], %b[7] qppermb,3,sm %b[34], %b[34], %r20, %b[14] stqp,5 %r19, %b[21], %b[3] }
Теоретическая скорость: 32 комплексных числа за 8 тактов (32/8) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
При попытке чтения в 64 потока получается резко менее эффективный код.
APB не может читать в 64 потока, поэтому в дополнение к APB компилятор вставляет обычные операции чтения ldqp. В итоге скорость резко проседает.
Но можно сделать чтение в 32 потока. Для этого напишем чтение в 64 потока и обработаем сначала одну половину строк в одном цикле, а потом вторую половину строк во втором цикле. В каждом цикле будут использованы 32 потока чтения APB.
6. reverse_radix4_vector2_stream32
Чтение в 32 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix4_vector2_stream32(int bit_count_in, myComplex *data_in, myComplex2 *data_out) { int bit_count_out = bit_count_in - 1; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_000 = (__v2di*)&data_in[ 0 * count_in/64]; __v2di *in_001 = (__v2di*)&data_in[ 1 * count_in/64]; __v2di *in_002 = (__v2di*)&data_in[ 2 * count_in/64]; __v2di *in_003 = (__v2di*)&data_in[ 3 * count_in/64]; __v2di *in_010 = (__v2di*)&data_in[ 4 * count_in/64]; __v2di *in_011 = (__v2di*)&data_in[ 5 * count_in/64]; __v2di *in_012 = (__v2di*)&data_in[ 6 * count_in/64]; __v2di *in_013 = (__v2di*)&data_in[ 7 * count_in/64]; __v2di *in_020 = (__v2di*)&data_in[ 8 * count_in/64]; __v2di *in_021 = (__v2di*)&data_in[ 9 * count_in/64]; __v2di *in_022 = (__v2di*)&data_in[10 * count_in/64]; __v2di *in_023 = (__v2di*)&data_in[11 * count_in/64]; __v2di *in_030 = (__v2di*)&data_in[12 * count_in/64]; __v2di *in_031 = (__v2di*)&data_in[13 * count_in/64]; __v2di *in_032 = (__v2di*)&data_in[14 * count_in/64]; __v2di *in_033 = (__v2di*)&data_in[15 * count_in/64]; __v2di *in_100 = (__v2di*)&data_in[16 * count_in/64]; __v2di *in_101 = (__v2di*)&data_in[17 * count_in/64]; __v2di *in_102 = (__v2di*)&data_in[18 * count_in/64]; __v2di *in_103 = (__v2di*)&data_in[19 * count_in/64]; __v2di *in_110 = (__v2di*)&data_in[20 * count_in/64]; __v2di *in_111 = (__v2di*)&data_in[21 * count_in/64]; __v2di *in_112 = (__v2di*)&data_in[22 * count_in/64]; __v2di *in_113 = (__v2di*)&data_in[23 * count_in/64]; __v2di *in_120 = (__v2di*)&data_in[24 * count_in/64]; __v2di *in_121 = (__v2di*)&data_in[25 * count_in/64]; __v2di *in_122 = (__v2di*)&data_in[26 * count_in/64]; __v2di *in_123 = (__v2di*)&data_in[27 * count_in/64]; __v2di *in_130 = (__v2di*)&data_in[28 * count_in/64]; __v2di *in_131 = (__v2di*)&data_in[29 * count_in/64]; __v2di *in_132 = (__v2di*)&data_in[30 * count_in/64]; __v2di *in_133 = (__v2di*)&data_in[31 * count_in/64]; __v2di *in_200 = (__v2di*)&data_in[32 * count_in/64]; __v2di *in_201 = (__v2di*)&data_in[33 * count_in/64]; __v2di *in_202 = (__v2di*)&data_in[34 * count_in/64]; __v2di *in_203 = (__v2di*)&data_in[35 * count_in/64]; __v2di *in_210 = (__v2di*)&data_in[36 * count_in/64]; __v2di *in_211 = (__v2di*)&data_in[37 * count_in/64]; __v2di *in_212 = (__v2di*)&data_in[38 * count_in/64]; __v2di *in_213 = (__v2di*)&data_in[39 * count_in/64]; __v2di *in_220 = (__v2di*)&data_in[40 * count_in/64]; __v2di *in_221 = (__v2di*)&data_in[41 * count_in/64]; __v2di *in_222 = (__v2di*)&data_in[42 * count_in/64]; __v2di *in_223 = (__v2di*)&data_in[43 * count_in/64]; __v2di *in_230 = (__v2di*)&data_in[44 * count_in/64]; __v2di *in_231 = (__v2di*)&data_in[45 * count_in/64]; __v2di *in_232 = (__v2di*)&data_in[46 * count_in/64]; __v2di *in_233 = (__v2di*)&data_in[47 * count_in/64]; __v2di *in_300 = (__v2di*)&data_in[48 * count_in/64]; __v2di *in_301 = (__v2di*)&data_in[49 * count_in/64]; __v2di *in_302 = (__v2di*)&data_in[50 * count_in/64]; __v2di *in_303 = (__v2di*)&data_in[51 * count_in/64]; __v2di *in_310 = (__v2di*)&data_in[52 * count_in/64]; __v2di *in_311 = (__v2di*)&data_in[53 * count_in/64]; __v2di *in_312 = (__v2di*)&data_in[54 * count_in/64]; __v2di *in_313 = (__v2di*)&data_in[55 * count_in/64]; __v2di *in_320 = (__v2di*)&data_in[56 * count_in/64]; __v2di *in_321 = (__v2di*)&data_in[57 * count_in/64]; __v2di *in_322 = (__v2di*)&data_in[58 * count_in/64]; __v2di *in_323 = (__v2di*)&data_in[59 * count_in/64]; __v2di *in_330 = (__v2di*)&data_in[60 * count_in/64]; __v2di *in_331 = (__v2di*)&data_in[61 * count_in/64]; __v2di *in_332 = (__v2di*)&data_in[62 * count_in/64]; __v2di *in_333 = (__v2di*)&data_in[63 * count_in/64]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/32/2; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 16 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555)); __v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_000[i], in_000[i], mask); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_100[i], in_100[i], mask); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_200[i], in_200[i], mask); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_300[i], in_300[i], mask); *(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_010[i], in_010[i], mask); *(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_110[i], in_110[i], mask); *(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_210[i], in_210[i], mask); *(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_310[i], in_310[i], mask); *(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_020[i], in_020[i], mask); *(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_120[i], in_120[i], mask); *(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_220[i], in_220[i], mask); *(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_320[i], in_320[i], mask); *(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_030[i], in_030[i], mask); *(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_130[i], in_130[i], mask); *(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_230[i], in_230[i], mask); *(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_330[i], in_330[i], mask); *(__v2du*)((void*)data_out + offset + 16*16) = __builtin_e2k_qppermb(in_001[i], in_001[i], mask); *(__v2du*)((void*)data_out + offset + 17*16) = __builtin_e2k_qppermb(in_101[i], in_101[i], mask); *(__v2du*)((void*)data_out + offset + 18*16) = __builtin_e2k_qppermb(in_201[i], in_201[i], mask); *(__v2du*)((void*)data_out + offset + 19*16) = __builtin_e2k_qppermb(in_301[i], in_301[i], mask); *(__v2du*)((void*)data_out + offset + 20*16) = __builtin_e2k_qppermb(in_011[i], in_011[i], mask); *(__v2du*)((void*)data_out + offset + 21*16) = __builtin_e2k_qppermb(in_111[i], in_111[i], mask); *(__v2du*)((void*)data_out + offset + 22*16) = __builtin_e2k_qppermb(in_211[i], in_211[i], mask); *(__v2du*)((void*)data_out + offset + 23*16) = __builtin_e2k_qppermb(in_311[i], in_311[i], mask); *(__v2du*)((void*)data_out + offset + 24*16) = __builtin_e2k_qppermb(in_021[i], in_021[i], mask); *(__v2du*)((void*)data_out + offset + 25*16) = __builtin_e2k_qppermb(in_121[i], in_121[i], mask); *(__v2du*)((void*)data_out + offset + 26*16) = __builtin_e2k_qppermb(in_221[i], in_221[i], mask); *(__v2du*)((void*)data_out + offset + 27*16) = __builtin_e2k_qppermb(in_321[i], in_321[i], mask); *(__v2du*)((void*)data_out + offset + 28*16) = __builtin_e2k_qppermb(in_031[i], in_031[i], mask); *(__v2du*)((void*)data_out + offset + 29*16) = __builtin_e2k_qppermb(in_131[i], in_131[i], mask); *(__v2du*)((void*)data_out + offset + 30*16) = __builtin_e2k_qppermb(in_231[i], in_231[i], mask); *(__v2du*)((void*)data_out + offset + 31*16) = __builtin_e2k_qppermb(in_331[i], in_331[i], mask); } #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/32/2; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 16 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555)); __v2di mask = {0x0B0A090803020100, 0x0F0E0D0C07060504}; *(__v2du*)((void*)data_out + offset + 32*16) = __builtin_e2k_qppermb(in_002[i], in_002[i], mask); *(__v2du*)((void*)data_out + offset + 33*16) = __builtin_e2k_qppermb(in_102[i], in_102[i], mask); *(__v2du*)((void*)data_out + offset + 34*16) = __builtin_e2k_qppermb(in_202[i], in_202[i], mask); *(__v2du*)((void*)data_out + offset + 35*16) = __builtin_e2k_qppermb(in_302[i], in_302[i], mask); *(__v2du*)((void*)data_out + offset + 36*16) = __builtin_e2k_qppermb(in_012[i], in_012[i], mask); *(__v2du*)((void*)data_out + offset + 37*16) = __builtin_e2k_qppermb(in_112[i], in_112[i], mask); *(__v2du*)((void*)data_out + offset + 38*16) = __builtin_e2k_qppermb(in_212[i], in_212[i], mask); *(__v2du*)((void*)data_out + offset + 39*16) = __builtin_e2k_qppermb(in_312[i], in_312[i], mask); *(__v2du*)((void*)data_out + offset + 40*16) = __builtin_e2k_qppermb(in_022[i], in_022[i], mask); *(__v2du*)((void*)data_out + offset + 41*16) = __builtin_e2k_qppermb(in_122[i], in_122[i], mask); *(__v2du*)((void*)data_out + offset + 42*16) = __builtin_e2k_qppermb(in_222[i], in_222[i], mask); *(__v2du*)((void*)data_out + offset + 43*16) = __builtin_e2k_qppermb(in_322[i], in_322[i], mask); *(__v2du*)((void*)data_out + offset + 44*16) = __builtin_e2k_qppermb(in_032[i], in_032[i], mask); *(__v2du*)((void*)data_out + offset + 45*16) = __builtin_e2k_qppermb(in_132[i], in_132[i], mask); *(__v2du*)((void*)data_out + offset + 46*16) = __builtin_e2k_qppermb(in_232[i], in_232[i], mask); *(__v2du*)((void*)data_out + offset + 47*16) = __builtin_e2k_qppermb(in_332[i], in_332[i], mask); *(__v2du*)((void*)data_out + offset + 48*16) = __builtin_e2k_qppermb(in_003[i], in_003[i], mask); *(__v2du*)((void*)data_out + offset + 49*16) = __builtin_e2k_qppermb(in_103[i], in_103[i], mask); *(__v2du*)((void*)data_out + offset + 50*16) = __builtin_e2k_qppermb(in_203[i], in_203[i], mask); *(__v2du*)((void*)data_out + offset + 51*16) = __builtin_e2k_qppermb(in_303[i], in_303[i], mask); *(__v2du*)((void*)data_out + offset + 52*16) = __builtin_e2k_qppermb(in_013[i], in_013[i], mask); *(__v2du*)((void*)data_out + offset + 53*16) = __builtin_e2k_qppermb(in_113[i], in_113[i], mask); *(__v2du*)((void*)data_out + offset + 54*16) = __builtin_e2k_qppermb(in_213[i], in_213[i], mask); *(__v2du*)((void*)data_out + offset + 55*16) = __builtin_e2k_qppermb(in_313[i], in_313[i], mask); *(__v2du*)((void*)data_out + offset + 56*16) = __builtin_e2k_qppermb(in_023[i], in_023[i], mask); *(__v2du*)((void*)data_out + offset + 57*16) = __builtin_e2k_qppermb(in_123[i], in_123[i], mask); *(__v2du*)((void*)data_out + offset + 58*16) = __builtin_e2k_qppermb(in_223[i], in_223[i], mask); *(__v2du*)((void*)data_out + offset + 59*16) = __builtin_e2k_qppermb(in_323[i], in_323[i], mask); *(__v2du*)((void*)data_out + offset + 60*16) = __builtin_e2k_qppermb(in_033[i], in_033[i], mask); *(__v2du*)((void*)data_out + offset + 61*16) = __builtin_e2k_qppermb(in_133[i], in_133[i], mask); *(__v2du*)((void*)data_out + offset + 62*16) = __builtin_e2k_qppermb(in_233[i], in_233[i], mask); *(__v2du*)((void*)data_out + offset + 63*16) = __builtin_e2k_qppermb(in_333[i], in_333[i], mask); } }
Основной цикл на ассемблере
.L9991: { fapb ct=0, dcd=0, fmt=5, mrng=16, d=17, incr=0, ind=0, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=16, incr=0, ind=0, asz=1, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=15, incr=0, ind=0, asz=1, abs=2, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=14, incr=0, ind=0, asz=1, abs=2, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=13, incr=0, ind=0, asz=1, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=12, incr=0, ind=0, asz=1, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=11, incr=0, ind=0, asz=1, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=10, incr=0, ind=0, asz=1, abs=6, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=9, incr=0, ind=0, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=8, incr=0, ind=0, asz=1, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=7, incr=0, ind=0, asz=1, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=6, incr=0, ind=0, asz=1, abs=10, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=5, incr=0, ind=0, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=4, incr=0, ind=0, asz=1, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=3, incr=0, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=2, incr=0, ind=0, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=1, abs=30, disp=0 } .L6665: { loop_mode qppermb,0,sm %b[41], %b[41], %r0, %b[58] shr_andd,1,sm %b[56], %r1, %r38, %b[54] stqp,2 %r5, %b[34], %b[54] qppermb,3,sm %b[46], %b[46], %r0, %b[60] shr_andd,4,sm %b[56], %r4, %r37, %b[56] stqp,5 %r2, %b[34], %b[58] } { loop_mode qppermb,1,sm %b[76], %b[76], %r0, %b[1] stqp,2 %r7, %b[34], %b[58] qppermb,4,sm %b[77], %b[77], %r0, %b[8] stqp,5 %r6, %b[34], %b[60] } { loop_mode qppermb,1,sm %b[65], %b[65], %r0, %b[10] stqp,2 %r11, %b[34], %b[3] qppermb,4,sm %b[63], %b[63], %r0, %b[3] stqp,5 %r9, %b[34], %b[10] } { loop_mode qppermb,1,sm %b[75], %b[75], %r0, %b[12] stqp,2 %r10, %b[34], %b[12] qppermb,4,sm %b[74], %b[74], %r0, %b[5] stqp,5 %r12, %b[34], %b[5] } { loop_mode qppermb,1,sm %b[73], %b[73], %r0, %b[14] stqp,2 %r13, %b[34], %b[14] qppermb,4,sm %b[72], %b[72], %r0, %b[7] stqp,5 %r14, %b[34], %b[7] } { loop_mode qppermb,1,sm %b[71], %b[71], %r0, %b[16] stqp,2 %r15, %b[34], %b[16] qppermb,4,sm %b[70], %b[70], %r0, %b[9] stqp,5 %r16, %b[34], %b[9] } { loop_mode qppermb,1,sm %b[69], %b[69], %r0, %b[18] stqp,2 %r17, %b[34], %b[18] qppermb,4,sm %b[67], %b[67], %r0, %b[11] stqp,5 %r18, %b[34], %b[11] movaqp,0 area=15, ind=0, am=1, be=0, %b[60] movaqp,1 area=14, ind=0, am=1, be=0, %b[64] movaqp,2 area=15, ind=0, am=1, be=0, %b[58] movaqp,3 area=14, ind=0, am=1, be=0, %b[62] } { loop_mode qppermb,1,sm %b[68], %b[68], %r0, %b[36] stqp,2 %r19, %b[34], %b[20] qppermb,4,sm %b[61], %b[61], %r0, %b[31] stqp,5 %r20, %b[34], %b[13] movaqp,0 area=13, ind=0, am=1, be=0, %b[63] movaqp,1 area=12, ind=0, am=1, be=0, %b[20] movaqp,2 area=13, ind=0, am=1, be=0, %b[61] movaqp,3 area=12, ind=0, am=1, be=0, %b[13] } { loop_mode qppermb,1,sm %b[59], %b[59], %r0, %b[53] stqp,2 %r21, %b[34], %b[38] qppermb,4,sm %b[57], %b[57], %r0, %b[49] stqp,5 %r22, %b[34], %b[33] movaqp,0 area=11, ind=0, am=1, be=0, %b[38] movaqp,1 area=10, ind=0, am=1, be=0, %b[46] movaqp,2 area=11, ind=0, am=1, be=0, %b[33] movaqp,3 area=10, ind=0, am=1, be=0, %b[41] } { loop_mode qppermb,1,sm %b[48], %b[48], %r0, %b[48] stqp,2 %r23, %b[34], %b[55] qppermb,4,sm %b[43], %b[43], %r0, %b[43] stqp,5 %r24, %b[34], %b[51] movaqp,0 area=9, ind=0, am=1, be=0, %b[57] movaqp,1 area=8, ind=0, am=1, be=0, %b[66] movaqp,2 area=9, ind=0, am=1, be=0, %b[55] movaqp,3 area=8, ind=0, am=1, be=0, %b[59] } { loop_mode qppermb,1,sm %b[40], %b[40], %r0, %b[40] stqp,2 %r25, %b[34], %b[50] qppermb,4,sm %b[35], %b[35], %r0, %b[35] stqp,5 %r26, %b[34], %b[45] movaqp,0 area=7, ind=0, am=1, be=0, %b[67] movaqp,1 area=6, ind=0, am=1, be=0, %b[69] movaqp,2 area=7, ind=0, am=1, be=0, %b[65] movaqp,3 area=6, ind=0, am=1, be=0, %b[68] } { loop_mode qppermb,1,sm %b[22], %b[22], %r0, %b[22] stqp,2 %r27, %b[34], %b[42] qppermb,4,sm %b[15], %b[15], %r0, %b[15] stqp,5 %r28, %b[34], %b[37] movaqp,0 area=5, ind=0, am=1, be=0, %b[71] movaqp,1 area=4, ind=0, am=1, be=0, %b[73] movaqp,2 area=5, ind=0, am=1, be=0, %b[70] movaqp,3 area=4, ind=0, am=1, be=0, %b[72] } { loop_mode qppermb,1,sm %b[63], %b[63], %r0, %b[24] stqp,2 %r29, %b[34], %b[24] qppermb,4,sm %b[61], %b[61], %r0, %b[17] stqp,5 %r30, %b[34], %b[17] movaqp,0 area=3, ind=0, am=1, be=0, %b[74] movaqp,1 area=2, ind=0, am=1, be=0, %b[75] movaqp,2 area=3, ind=0, am=1, be=0, %b[61] movaqp,3 area=2, ind=0, am=1, be=0, %b[63] } { loop_mode addd,0,sm %b[2], 0x1, %b[0] qppermb,1,sm %b[64], %b[64], %r0, %b[28] stqp,2 %r31, %b[34], %b[28] ord,3,sm %b[56], %b[54], %b[54] qppermb,4,sm %b[62], %b[62], %r0, %b[21] stqp,5 %r32, %b[34], %b[21] movaqp,0 area=1, ind=0, am=1, be=0, %b[42] movaqp,1 area=0, ind=0, am=1, be=0, %b[50] movaqp,2 area=1, ind=0, am=1, be=0, %b[37] movaqp,3 area=0, ind=0, am=1, be=0, %b[45] } { loop_mode bitrevd,0,sm %b[2], %b[54] qppermb,1,sm %b[60], %b[60], %r0, %b[25] stqp,2 %r33, %b[34], %b[32] shld,3,sm %b[54], 0x4, %b[32] qppermb,4,sm %b[58], %b[58], %r0, %b[2] stqp,5 %r34, %b[34], %b[25] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[47], %b[47], %r0, %b[52] stqp,2 %r35, %b[34], %b[29] qppermb,3,sm %b[52], %b[52], %r0, %b[56] stqp,5 %r36, %b[34], %b[6] } ... .L9337: { fapb ct=0, dcd=0, fmt=5, mrng=16, d=17, incr=0, ind=0, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=16, incr=0, ind=0, asz=1, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=15, incr=0, ind=0, asz=1, abs=2, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=14, incr=0, ind=0, asz=1, abs=2, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=13, incr=0, ind=0, asz=1, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=12, incr=0, ind=0, asz=1, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=11, incr=0, ind=0, asz=1, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=10, incr=0, ind=0, asz=1, abs=6, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=9, incr=0, ind=0, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=8, incr=0, ind=0, asz=1, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=7, incr=0, ind=0, asz=1, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=6, incr=0, ind=0, asz=1, abs=10, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=5, incr=0, ind=0, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=4, incr=0, ind=0, asz=1, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=3, incr=0, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=2, incr=0, ind=0, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=1, incr=0, ind=0, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=15, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=14, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=13, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=12, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=11, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=10, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=9, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=8, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=7, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=6, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=5, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=1, asz=1, abs=30, disp=0 } .L7373: { loop_mode qppermb,0,sm %b[41], %b[41], %r0, %b[58] shr_andd,1,sm %b[56], %r1, %r38, %b[54] stqp,2 %r5, %b[34], %b[54] qppermb,3,sm %b[46], %b[46], %r0, %b[60] shr_andd,4,sm %b[56], %r4, %r37, %b[56] stqp,5 %r2, %b[34], %b[58] } { loop_mode qppermb,1,sm %b[76], %b[76], %r0, %b[1] stqp,2 %r7, %b[34], %b[58] qppermb,4,sm %b[77], %b[77], %r0, %b[8] stqp,5 %r6, %b[34], %b[60] } { loop_mode qppermb,1,sm %b[65], %b[65], %r0, %b[10] stqp,2 %r11, %b[34], %b[3] qppermb,4,sm %b[63], %b[63], %r0, %b[3] stqp,5 %r9, %b[34], %b[10] } { loop_mode qppermb,1,sm %b[75], %b[75], %r0, %b[12] stqp,2 %r10, %b[34], %b[12] qppermb,4,sm %b[74], %b[74], %r0, %b[5] stqp,5 %r12, %b[34], %b[5] } { loop_mode qppermb,1,sm %b[73], %b[73], %r0, %b[14] stqp,2 %r13, %b[34], %b[14] qppermb,4,sm %b[72], %b[72], %r0, %b[7] stqp,5 %r14, %b[34], %b[7] } { loop_mode qppermb,1,sm %b[71], %b[71], %r0, %b[16] stqp,2 %r15, %b[34], %b[16] qppermb,4,sm %b[70], %b[70], %r0, %b[9] stqp,5 %r16, %b[34], %b[9] } { loop_mode qppermb,1,sm %b[69], %b[69], %r0, %b[18] stqp,2 %r17, %b[34], %b[18] qppermb,4,sm %b[67], %b[67], %r0, %b[11] stqp,5 %r18, %b[34], %b[11] movaqp,0 area=15, ind=0, am=1, be=0, %b[60] movaqp,1 area=14, ind=0, am=1, be=0, %b[64] movaqp,2 area=15, ind=0, am=1, be=0, %b[58] movaqp,3 area=14, ind=0, am=1, be=0, %b[62] } { loop_mode qppermb,1,sm %b[68], %b[68], %r0, %b[36] stqp,2 %r19, %b[34], %b[20] qppermb,4,sm %b[61], %b[61], %r0, %b[31] stqp,5 %r20, %b[34], %b[13] movaqp,0 area=13, ind=0, am=1, be=0, %b[63] movaqp,1 area=12, ind=0, am=1, be=0, %b[20] movaqp,2 area=13, ind=0, am=1, be=0, %b[61] movaqp,3 area=12, ind=0, am=1, be=0, %b[13] } { loop_mode qppermb,1,sm %b[59], %b[59], %r0, %b[53] stqp,2 %r21, %b[34], %b[38] qppermb,4,sm %b[57], %b[57], %r0, %b[49] stqp,5 %r22, %b[34], %b[33] movaqp,0 area=11, ind=0, am=1, be=0, %b[38] movaqp,1 area=10, ind=0, am=1, be=0, %b[46] movaqp,2 area=11, ind=0, am=1, be=0, %b[33] movaqp,3 area=10, ind=0, am=1, be=0, %b[41] } { loop_mode qppermb,1,sm %b[48], %b[48], %r0, %b[48] stqp,2 %r23, %b[34], %b[55] qppermb,4,sm %b[43], %b[43], %r0, %b[43] stqp,5 %r24, %b[34], %b[51] movaqp,0 area=9, ind=0, am=1, be=0, %b[57] movaqp,1 area=8, ind=0, am=1, be=0, %b[66] movaqp,2 area=9, ind=0, am=1, be=0, %b[55] movaqp,3 area=8, ind=0, am=1, be=0, %b[59] } { loop_mode qppermb,1,sm %b[40], %b[40], %r0, %b[40] stqp,2 %r25, %b[34], %b[50] qppermb,4,sm %b[35], %b[35], %r0, %b[35] stqp,5 %r26, %b[34], %b[45] movaqp,0 area=7, ind=0, am=1, be=0, %b[67] movaqp,1 area=6, ind=0, am=1, be=0, %b[69] movaqp,2 area=7, ind=0, am=1, be=0, %b[65] movaqp,3 area=6, ind=0, am=1, be=0, %b[68] } { loop_mode qppermb,1,sm %b[22], %b[22], %r0, %b[22] stqp,2 %r27, %b[34], %b[42] qppermb,4,sm %b[15], %b[15], %r0, %b[15] stqp,5 %r28, %b[34], %b[37] movaqp,0 area=5, ind=0, am=1, be=0, %b[71] movaqp,1 area=4, ind=0, am=1, be=0, %b[73] movaqp,2 area=5, ind=0, am=1, be=0, %b[70] movaqp,3 area=4, ind=0, am=1, be=0, %b[72] } { loop_mode qppermb,1,sm %b[63], %b[63], %r0, %b[24] stqp,2 %r29, %b[34], %b[24] qppermb,4,sm %b[61], %b[61], %r0, %b[17] stqp,5 %r30, %b[34], %b[17] movaqp,0 area=3, ind=0, am=1, be=0, %b[74] movaqp,1 area=2, ind=0, am=1, be=0, %b[75] movaqp,2 area=3, ind=0, am=1, be=0, %b[61] movaqp,3 area=2, ind=0, am=1, be=0, %b[63] } { loop_mode addd,0,sm %b[2], 0x1, %b[0] qppermb,1,sm %b[64], %b[64], %r0, %b[28] stqp,2 %r31, %b[34], %b[28] ord,3,sm %b[56], %b[54], %b[54] qppermb,4,sm %b[62], %b[62], %r0, %b[21] stqp,5 %r32, %b[34], %b[21] movaqp,0 area=1, ind=0, am=1, be=0, %b[42] movaqp,1 area=0, ind=0, am=1, be=0, %b[50] movaqp,2 area=1, ind=0, am=1, be=0, %b[37] movaqp,3 area=0, ind=0, am=1, be=0, %b[45] } { loop_mode bitrevd,0,sm %b[2], %b[54] qppermb,1,sm %b[60], %b[60], %r0, %b[25] stqp,2 %r33, %b[34], %b[32] shld,3,sm %b[54], 0x4, %b[32] qppermb,4,sm %b[58], %b[58], %r0, %b[2] stqp,5 %r34, %b[34], %b[25] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[47], %b[47], %r0, %b[52] stqp,2 %r35, %b[34], %b[29] qppermb,3,sm %b[52], %b[52], %r0, %b[56] stqp,5 %r36, %b[34], %b[6] }
Теоретическая скорость: 64 комплексных числа за 16 тактов (64/16) = 32 Байт/такт
Замеры скорости
Видим замедление по всей длине графика.
Итоги по reverse_radix4_vector2
Победителем можно считать reverse_radix4_vector2_stream16.
При реализации Vector-2 Radix-4 FFT будем использовать его.
reverse_radix4_vector4
1. reverse_radix4_vector4_etalon
Эталонный вариант для сравнения на корректность.
Здесь reverseNumber_radix4 вычисляется с помощью цикла.
int reverseNumber_radix4(int number, int bit_count) { int answer = 0; for(int i = 0; i < bit_count/2; ++i) { answer <<= 2; answer |= number & 3; number >>= 2; } return answer; } void reverse_radix4_vector4_etalon(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; // int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; for(int64_t i = 0; i < count_out; ++i) { int index = reverseNumber_radix4(i, bit_count_out); myComplex a = data_in[4*i + 0]; myComplex b = data_in[4*i + 1]; myComplex c = data_in[4*i + 2]; myComplex d = data_in[4*i + 3]; data_out[index] = (myComplex4){.real = {a.real, b.real, c.real, d.real}, .imag = {a.imag, b.imag, c.imag, d.imag}}; } }
2. reverse_radix4_vector4
В процессоре нет готовых инструкций, производящих операцию reverseNumber_radix4.
Поэтому выполним инструкцию bitrevd и переставим соседние биты местами.
Код на Си
void reverse_radix4_vector4(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; // int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in = (__v2di*)data_in; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 32 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555)); __v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110}; __v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in[2*i+1], in[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in[2*i+1], in[2*i+0], mask_imag); } }
Основной цикл на ассемблере
.L6561: { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } .L6351: { loop_mode qppermb,0,sm %b[15], %b[12], %r4, %b[19] shr_andd,1,sm %b[14], %r6, %r10, %b[0] qppermb,3,sm %b[15], %b[12], %r5, %b[20] shr_andd,4,sm %b[16], %r7, %r9, %b[5] ord,5,sm %b[7], %b[4], %b[18] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END bitrevd,0,sm %b[3], %b[12] stqp,2 %r0, %b[17], %b[19] addd,3,sm %b[3], 0x1, %b[1] shld,4,sm %b[18], 0x5, %b[15] stqp,5 %r2, %b[17], %b[20] movaqp,0 area=0, ind=0, am=1, be=0, %b[4] movaqp,1 area=0, ind=16, am=0, be=0, %b[7] }
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
3. reverse_radix4_vector4_x4
Сделаем ручную раскрутку цикла в 4 раза.
Код на Си
void reverse_radix4_vector4_x4(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; // int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in = (__v2di*) data_in; __v2di *out_0 = (__v2di*)&data_out[0 * count_out/4]; __v2di *out_1 = (__v2di*)&data_out[1 * count_out/4]; __v2di *out_2 = (__v2di*)&data_out[2 * count_out/4]; __v2di *out_3 = (__v2di*)&data_out[3 * count_out/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out; i += 4) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 32 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555)); __v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110}; __v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514}; *(__v2du*)((void*)out_0 + offset + 0*16) = __builtin_e2k_qppermb(in[2*(i+0)+1], in[2*(i+0)+0], mask_real); *(__v2du*)((void*)out_0 + offset + 1*16) = __builtin_e2k_qppermb(in[2*(i+0)+1], in[2*(i+0)+0], mask_imag); *(__v2du*)((void*)out_1 + offset + 0*16) = __builtin_e2k_qppermb(in[2*(i+1)+1], in[2*(i+1)+0], mask_real); *(__v2du*)((void*)out_1 + offset + 1*16) = __builtin_e2k_qppermb(in[2*(i+1)+1], in[2*(i+1)+0], mask_imag); *(__v2du*)((void*)out_2 + offset + 0*16) = __builtin_e2k_qppermb(in[2*(i+2)+1], in[2*(i+2)+0], mask_real); *(__v2du*)((void*)out_2 + offset + 1*16) = __builtin_e2k_qppermb(in[2*(i+2)+1], in[2*(i+2)+0], mask_imag); *(__v2du*)((void*)out_3 + offset + 0*16) = __builtin_e2k_qppermb(in[2*(i+3)+1], in[2*(i+3)+0], mask_real); *(__v2du*)((void*)out_3 + offset + 1*16) = __builtin_e2k_qppermb(in[2*(i+3)+1], in[2*(i+3)+0], mask_imag); } }
Основной цикл на ассемблере
.L7192: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=16, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=16, disp=96 } .L6608: { loop_mode qppermb,0,sm %b[23], %b[9], %r11, %b[35] stqp,2 %r0, %b[33], %b[22] qppermb,3,sm %b[23], %b[9], %r13, %b[36] shr_andd,4,sm %b[12], %r14, %r16, %b[1] stqp,5 %r2, %b[33], %b[25] } { loop_mode ord,0,sm %b[17], %b[3], %b[23] qppermb,1,sm %b[15], %b[10], %r11, %b[37] stqp,2 %r4, %b[33], %b[35] qppermb,4,sm %b[28], %b[20], %r13, %b[38] stqp,5 %r10, %b[33], %b[36] movaqp,0 area=1, ind=0, am=1, be=0, %b[14] movaqp,1 area=1, ind=16, am=0, be=0, %b[22] movaqp,2 area=1, ind=0, am=1, be=0, %b[4] movaqp,3 area=1, ind=16, am=0, be=0, %b[9] } { loop_mode shld,0,sm %b[23], 0x5, %b[31] qppermb,1,sm %b[28], %b[20], %r11, %b[36] stqp,2 %r6, %b[33], %b[37] addd,3,sm %b[2], 0x4, %b[0] qppermb,4,sm %b[15], %b[10], %r13, %b[35] stqp,5 %r9, %b[33], %b[38] movaqp,0 area=0, ind=0, am=1, be=0, %b[25] movaqp,1 area=0, ind=16, am=0, be=0, %b[30] movaqp,2 area=0, ind=0, am=1, be=0, %b[3] movaqp,3 area=0, ind=16, am=0, be=0, %b[17] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[34], %b[29], %r11, %b[20] shr_andd,1,sm %b[12], %r12, %r15, %b[15] stqp,2 %r5, %b[33], %b[36] qppermb,3,sm %b[34], %b[29], %r13, %b[23] bitrevd,4,sm %b[2], %b[10] stqp,5 %r7, %b[33], %b[35] }
Теоретическая скорость: 16 комплексных чисел за 4 такта (16/4) = 32 Байт/такт
Замеры скорости
Видим ускорение в начале и замедление в конце графика.
Здесь происходит четыре чтения из одного места памяти и запись в четыре разных места памяти.
4. reverse_radix4_vector4_stream4
Теперь сделаем наоборот: будем читать из четырёх разных мест, а писать рядом.
Чтение в 4 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix4_vector4_stream4(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_0 = (__v2di*)&data_in[0 * count_in/4]; __v2di *in_1 = (__v2di*)&data_in[1 * count_in/4]; __v2di *in_2 = (__v2di*)&data_in[2 * count_in/4]; __v2di *in_3 = (__v2di*)&data_in[3 * count_in/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/4; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 32 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555)); __v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110}; __v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_0[2*i+1], in_0[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_0[2*i+1], in_0[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_1[2*i+1], in_1[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_1[2*i+1], in_1[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_2[2*i+1], in_2[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_2[2*i+1], in_2[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_3[2*i+1], in_3[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_3[2*i+1], in_3[2*i+0], mask_imag); } }
Основной цикл на ассемблере
.L7740: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=4, abs=0, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=4, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=16, disp=0 } .L7235: { loop_mode qppermb,0,sm %b[23], %b[9], %r11, %b[35] stqp,2 %r0, %b[33], %b[22] qppermb,3,sm %b[23], %b[9], %r13, %b[36] shr_andd,4,sm %b[12], %r14, %r16, %b[1] stqp,5 %r2, %b[33], %b[25] } { loop_mode ord,0,sm %b[17], %b[3], %b[23] qppermb,1,sm %b[15], %b[10], %r11, %b[37] stqp,2 %r5, %b[33], %b[35] qppermb,4,sm %b[28], %b[20], %r13, %b[38] stqp,5 %r4, %b[33], %b[36] movaqp,0 area=1, ind=0, am=1, be=0, %b[14] movaqp,1 area=1, ind=16, am=0, be=0, %b[22] movaqp,2 area=1, ind=0, am=1, be=0, %b[4] movaqp,3 area=1, ind=16, am=0, be=0, %b[9] } { loop_mode shld,0,sm %b[23], 0x5, %b[31] qppermb,1,sm %b[28], %b[20], %r11, %b[36] stqp,2 %r10, %b[33], %b[37] addd,3,sm %b[2], 0x1, %b[0] qppermb,4,sm %b[15], %b[10], %r13, %b[35] stqp,5 %r6, %b[33], %b[38] movaqp,0 area=0, ind=0, am=1, be=0, %b[25] movaqp,1 area=0, ind=16, am=0, be=0, %b[30] movaqp,2 area=0, ind=0, am=1, be=0, %b[3] movaqp,3 area=0, ind=16, am=0, be=0, %b[17] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[34], %b[29], %r11, %b[20] shr_andd,1,sm %b[12], %r12, %r15, %b[15] stqp,2 %r7, %b[33], %b[36] qppermb,3,sm %b[34], %b[29], %r13, %b[23] bitrevd,4,sm %b[2], %b[10] stqp,5 %r9, %b[33], %b[35] }
Теоретическая скорость: 16 комплексных чисел за 4 такта (16/4) = 32 Байт/такт
Замеры скорости
Видим желаемое ускорение по всей длине графика.
5. reverse_radix4_vector4_stream16
Чтение в 16 потоков из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix4_vector4_stream16(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_00 = (__v2di*)&data_in[ 0 * count_in/16]; __v2di *in_01 = (__v2di*)&data_in[ 1 * count_in/16]; __v2di *in_02 = (__v2di*)&data_in[ 2 * count_in/16]; __v2di *in_03 = (__v2di*)&data_in[ 3 * count_in/16]; __v2di *in_10 = (__v2di*)&data_in[ 4 * count_in/16]; __v2di *in_11 = (__v2di*)&data_in[ 5 * count_in/16]; __v2di *in_12 = (__v2di*)&data_in[ 6 * count_in/16]; __v2di *in_13 = (__v2di*)&data_in[ 7 * count_in/16]; __v2di *in_20 = (__v2di*)&data_in[ 8 * count_in/16]; __v2di *in_21 = (__v2di*)&data_in[ 9 * count_in/16]; __v2di *in_22 = (__v2di*)&data_in[10 * count_in/16]; __v2di *in_23 = (__v2di*)&data_in[11 * count_in/16]; __v2di *in_30 = (__v2di*)&data_in[12 * count_in/16]; __v2di *in_31 = (__v2di*)&data_in[13 * count_in/16]; __v2di *in_32 = (__v2di*)&data_in[14 * count_in/16]; __v2di *in_33 = (__v2di*)&data_in[15 * count_in/16]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/16; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 32 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555)); __v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110}; __v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_00[2*i+1], in_00[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_00[2*i+1], in_00[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_10[2*i+1], in_10[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_10[2*i+1], in_10[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_20[2*i+1], in_20[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_20[2*i+1], in_20[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_30[2*i+1], in_30[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_30[2*i+1], in_30[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_01[2*i+1], in_01[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_01[2*i+1], in_01[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_11[2*i+1], in_11[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_11[2*i+1], in_11[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_21[2*i+1], in_21[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_21[2*i+1], in_21[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_31[2*i+1], in_31[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_31[2*i+1], in_31[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 16*16) = __builtin_e2k_qppermb(in_02[2*i+1], in_02[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 17*16) = __builtin_e2k_qppermb(in_02[2*i+1], in_02[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 18*16) = __builtin_e2k_qppermb(in_12[2*i+1], in_12[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 19*16) = __builtin_e2k_qppermb(in_12[2*i+1], in_12[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 20*16) = __builtin_e2k_qppermb(in_22[2*i+1], in_22[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 21*16) = __builtin_e2k_qppermb(in_22[2*i+1], in_22[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 22*16) = __builtin_e2k_qppermb(in_32[2*i+1], in_32[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 23*16) = __builtin_e2k_qppermb(in_32[2*i+1], in_32[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 24*16) = __builtin_e2k_qppermb(in_03[2*i+1], in_03[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 25*16) = __builtin_e2k_qppermb(in_03[2*i+1], in_03[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 26*16) = __builtin_e2k_qppermb(in_13[2*i+1], in_13[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 27*16) = __builtin_e2k_qppermb(in_13[2*i+1], in_13[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 28*16) = __builtin_e2k_qppermb(in_23[2*i+1], in_23[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 29*16) = __builtin_e2k_qppermb(in_23[2*i+1], in_23[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 30*16) = __builtin_e2k_qppermb(in_33[2*i+1], in_33[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 31*16) = __builtin_e2k_qppermb(in_33[2*i+1], in_33[2*i+0], mask_imag); } }
Основной цикл на ассемблере
.L9557: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=1, incr=1, ind=0, asz=2, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=15, asz=2, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=14, asz=2, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=13, asz=2, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=12, asz=2, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=11, asz=2, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=10, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=9, asz=2, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=8, asz=2, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=2, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=2, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=2, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=2, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=2, abs=24, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=2, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=28, disp=0 } .L7879: { loop_mode qppermb,0,sm %b[20], %b[15], %r36, %b[73] shr_andd,1,sm %b[69], %r37, %r40, %b[69] stqp,2 %r0, %b[4], %b[71] qppermb,3,sm %b[20], %b[15], %r35, %b[72] shr_andd,4,sm %b[69], %r38, %r39, %b[71] stqp,5 %r2, %b[4], %b[72] } { loop_mode qppermb,1,sm %b[28], %b[23], %r36, %b[72] stqp,2 %r5, %b[4], %b[73] qppermb,4,sm %b[67], %b[64], %r35, %b[74] stqp,5 %r1, %b[4], %b[72] } { loop_mode qppermb,1,sm %b[67], %b[64], %r36, %b[72] stqp,2 %r10, %b[4], %b[72] qppermb,4,sm %b[28], %b[23], %r35, %b[67] stqp,5 %r6, %b[4], %b[74] } { loop_mode qppermb,1,sm %b[63], %b[60], %r36, %b[67] stqp,2 %r7, %b[4], %b[72] qppermb,4,sm %b[63], %b[60], %r35, %b[72] stqp,5 %r9, %b[4], %b[67] } { loop_mode qppermb,1,sm %b[36], %b[31], %r36, %b[67] stqp,2 %r12, %b[4], %b[67] qppermb,4,sm %b[36], %b[31], %r35, %b[72] stqp,5 %r11, %b[4], %b[72] } { loop_mode qppermb,1,sm %b[59], %b[56], %r36, %b[67] stqp,2 %r14, %b[4], %b[67] qppermb,4,sm %b[59], %b[56], %r35, %b[72] stqp,5 %r13, %b[4], %b[72] } { loop_mode qppermb,1,sm %b[44], %b[39], %r36, %b[67] stqp,2 %r16, %b[4], %b[67] qppermb,4,sm %b[44], %b[39], %r35, %b[72] stqp,5 %r15, %b[4], %b[72] movaqp,0 area=7, ind=0, am=1, be=0, %b[7] movaqp,1 area=7, ind=16, am=0, be=0, %b[12] movaqp,2 area=7, ind=0, am=1, be=0, %b[1] movaqp,3 area=7, ind=16, am=0, be=0, %b[6] } { loop_mode qppermb,1,sm %b[55], %b[52], %r36, %b[67] stqp,2 %r18, %b[4], %b[67] qppermb,4,sm %b[55], %b[52], %r35, %b[72] stqp,5 %r17, %b[4], %b[72] movaqp,0 area=6, ind=0, am=1, be=0, %b[23] movaqp,1 area=6, ind=16, am=0, be=0, %b[28] movaqp,2 area=6, ind=0, am=1, be=0, %b[15] movaqp,3 area=6, ind=16, am=0, be=0, %b[20] } { loop_mode qppermb,1,sm %b[51], %b[47], %r36, %b[67] stqp,2 %r20, %b[4], %b[67] qppermb,4,sm %b[51], %b[47], %r35, %b[72] stqp,5 %r19, %b[4], %b[72] movaqp,0 area=5, ind=0, am=1, be=0, %b[39] movaqp,1 area=5, ind=16, am=0, be=0, %b[44] movaqp,2 area=5, ind=0, am=1, be=0, %b[31] movaqp,3 area=5, ind=16, am=0, be=0, %b[36] } { loop_mode qppermb,1,sm %b[48], %b[43], %r36, %b[67] stqp,2 %r22, %b[4], %b[67] qppermb,4,sm %b[48], %b[43], %r35, %b[72] stqp,5 %r21, %b[4], %b[72] movaqp,0 area=4, ind=0, am=1, be=0, %b[48] movaqp,1 area=4, ind=16, am=0, be=0, %b[51] movaqp,2 area=4, ind=0, am=1, be=0, %b[43] movaqp,3 area=4, ind=16, am=0, be=0, %b[47] } { loop_mode qppermb,1,sm %b[40], %b[35], %r36, %b[67] stqp,2 %r24, %b[4], %b[67] qppermb,4,sm %b[40], %b[35], %r35, %b[72] stqp,5 %r23, %b[4], %b[72] movaqp,0 area=3, ind=0, am=1, be=0, %b[52] movaqp,1 area=3, ind=16, am=0, be=0, %b[55] movaqp,2 area=3, ind=0, am=1, be=0, %b[35] movaqp,3 area=3, ind=16, am=0, be=0, %b[40] } { loop_mode qppermb,1,sm %b[32], %b[27], %r36, %b[67] stqp,2 %r26, %b[4], %b[67] qppermb,4,sm %b[32], %b[27], %r35, %b[72] stqp,5 %r25, %b[4], %b[72] movaqp,0 area=2, ind=0, am=1, be=0, %b[56] movaqp,1 area=2, ind=16, am=0, be=0, %b[59] movaqp,2 area=2, ind=0, am=1, be=0, %b[27] movaqp,3 area=2, ind=16, am=0, be=0, %b[32] } { loop_mode qppermb,1,sm %b[24], %b[19], %r36, %b[67] stqp,2 %r28, %b[4], %b[67] qppermb,4,sm %b[24], %b[19], %r35, %b[72] stqp,5 %r27, %b[4], %b[72] movaqp,0 area=1, ind=0, am=1, be=0, %b[60] movaqp,1 area=1, ind=16, am=0, be=0, %b[63] movaqp,2 area=1, ind=0, am=1, be=0, %b[19] movaqp,3 area=1, ind=16, am=0, be=0, %b[24] } { loop_mode addd,0,sm %b[2], 0x1, %b[0] qppermb,1,sm %b[16], %b[11], %r36, %b[69] stqp,2 %r30, %b[4], %b[67] ord,3,sm %b[71], %b[69], %b[67] qppermb,4,sm %b[16], %b[11], %r35, %b[71] stqp,5 %r29, %b[4], %b[72] movaqp,0 area=0, ind=0, am=1, be=0, %b[64] movaqp,1 area=0, ind=16, am=0, be=0, %b[68] movaqp,2 area=0, ind=0, am=1, be=0, %b[11] movaqp,3 area=0, ind=16, am=0, be=0, %b[16] } { loop_mode bitrevd,0,sm %b[2], %b[67] qppermb,1,sm %b[10], %b[5], %r36, %b[69] stqp,2 %r32, %b[4], %b[69] shld,3,sm %b[67], 0x5, %b[2] qppermb,4,sm %b[10], %b[5], %r35, %b[71] stqp,5 %r31, %b[4], %b[71] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qppermb,0,sm %b[70], %b[66], %r36, %b[69] stqp,2 %r34, %b[4], %b[69] qppermb,3,sm %b[70], %b[66], %r35, %b[70] stqp,5 %r33, %b[4], %b[71] }
Теоретическая скорость: 64 комплексных числа за 16 тактов (64/16) = 32 Байт/такт
Замеры скорости
Видим сильное ускорение.
При попытке чтения в 64 потока получается резко менее эффективный код.
APB не может читать в 64 потока, поэтому в дополнение к APB компилятор вставляет обычные операции чтения ldqp. В итоге скорость резко проседает.
Но можно сделать чтение в 32 потока. Для этого напишем чтение в 64 потока и обработаем сначала одну половину строк в одном цикле, а потом вторую половину строк во втором цикле. В каждом цикле будут использованы 32 потока чтения APB.
6. reverse_radix4_vector4_stream32
Чтение в 32 потока из разных мест памяти, запись в одно место памяти.
Код на Си
void reverse_radix4_vector4_stream32(int bit_count_in, myComplex *data_in, myComplex4 *data_out) { int bit_count_out = bit_count_in - 2; int count_in = 1 << bit_count_in; int count_out = 1 << bit_count_out; int shift_out = 64 - bit_count_out; __v2di *in_000 = (__v2di*)&data_in[ 0 * count_in/64]; __v2di *in_001 = (__v2di*)&data_in[ 1 * count_in/64]; __v2di *in_002 = (__v2di*)&data_in[ 2 * count_in/64]; __v2di *in_003 = (__v2di*)&data_in[ 3 * count_in/64]; __v2di *in_010 = (__v2di*)&data_in[ 4 * count_in/64]; __v2di *in_011 = (__v2di*)&data_in[ 5 * count_in/64]; __v2di *in_012 = (__v2di*)&data_in[ 6 * count_in/64]; __v2di *in_013 = (__v2di*)&data_in[ 7 * count_in/64]; __v2di *in_020 = (__v2di*)&data_in[ 8 * count_in/64]; __v2di *in_021 = (__v2di*)&data_in[ 9 * count_in/64]; __v2di *in_022 = (__v2di*)&data_in[10 * count_in/64]; __v2di *in_023 = (__v2di*)&data_in[11 * count_in/64]; __v2di *in_030 = (__v2di*)&data_in[12 * count_in/64]; __v2di *in_031 = (__v2di*)&data_in[13 * count_in/64]; __v2di *in_032 = (__v2di*)&data_in[14 * count_in/64]; __v2di *in_033 = (__v2di*)&data_in[15 * count_in/64]; __v2di *in_100 = (__v2di*)&data_in[16 * count_in/64]; __v2di *in_101 = (__v2di*)&data_in[17 * count_in/64]; __v2di *in_102 = (__v2di*)&data_in[18 * count_in/64]; __v2di *in_103 = (__v2di*)&data_in[19 * count_in/64]; __v2di *in_110 = (__v2di*)&data_in[20 * count_in/64]; __v2di *in_111 = (__v2di*)&data_in[21 * count_in/64]; __v2di *in_112 = (__v2di*)&data_in[22 * count_in/64]; __v2di *in_113 = (__v2di*)&data_in[23 * count_in/64]; __v2di *in_120 = (__v2di*)&data_in[24 * count_in/64]; __v2di *in_121 = (__v2di*)&data_in[25 * count_in/64]; __v2di *in_122 = (__v2di*)&data_in[26 * count_in/64]; __v2di *in_123 = (__v2di*)&data_in[27 * count_in/64]; __v2di *in_130 = (__v2di*)&data_in[28 * count_in/64]; __v2di *in_131 = (__v2di*)&data_in[29 * count_in/64]; __v2di *in_132 = (__v2di*)&data_in[30 * count_in/64]; __v2di *in_133 = (__v2di*)&data_in[31 * count_in/64]; __v2di *in_200 = (__v2di*)&data_in[32 * count_in/64]; __v2di *in_201 = (__v2di*)&data_in[33 * count_in/64]; __v2di *in_202 = (__v2di*)&data_in[34 * count_in/64]; __v2di *in_203 = (__v2di*)&data_in[35 * count_in/64]; __v2di *in_210 = (__v2di*)&data_in[36 * count_in/64]; __v2di *in_211 = (__v2di*)&data_in[37 * count_in/64]; __v2di *in_212 = (__v2di*)&data_in[38 * count_in/64]; __v2di *in_213 = (__v2di*)&data_in[39 * count_in/64]; __v2di *in_220 = (__v2di*)&data_in[40 * count_in/64]; __v2di *in_221 = (__v2di*)&data_in[41 * count_in/64]; __v2di *in_222 = (__v2di*)&data_in[42 * count_in/64]; __v2di *in_223 = (__v2di*)&data_in[43 * count_in/64]; __v2di *in_230 = (__v2di*)&data_in[44 * count_in/64]; __v2di *in_231 = (__v2di*)&data_in[45 * count_in/64]; __v2di *in_232 = (__v2di*)&data_in[46 * count_in/64]; __v2di *in_233 = (__v2di*)&data_in[47 * count_in/64]; __v2di *in_300 = (__v2di*)&data_in[48 * count_in/64]; __v2di *in_301 = (__v2di*)&data_in[49 * count_in/64]; __v2di *in_302 = (__v2di*)&data_in[50 * count_in/64]; __v2di *in_303 = (__v2di*)&data_in[51 * count_in/64]; __v2di *in_310 = (__v2di*)&data_in[52 * count_in/64]; __v2di *in_311 = (__v2di*)&data_in[53 * count_in/64]; __v2di *in_312 = (__v2di*)&data_in[54 * count_in/64]; __v2di *in_313 = (__v2di*)&data_in[55 * count_in/64]; __v2di *in_320 = (__v2di*)&data_in[56 * count_in/64]; __v2di *in_321 = (__v2di*)&data_in[57 * count_in/64]; __v2di *in_322 = (__v2di*)&data_in[58 * count_in/64]; __v2di *in_323 = (__v2di*)&data_in[59 * count_in/64]; __v2di *in_330 = (__v2di*)&data_in[60 * count_in/64]; __v2di *in_331 = (__v2di*)&data_in[61 * count_in/64]; __v2di *in_332 = (__v2di*)&data_in[62 * count_in/64]; __v2di *in_333 = (__v2di*)&data_in[63 * count_in/64]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/32/2; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 32 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555)); __v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110}; __v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514}; *(__v2du*)((void*)data_out + offset + 0*16) = __builtin_e2k_qppermb(in_000[2*i+1], in_000[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 1*16) = __builtin_e2k_qppermb(in_000[2*i+1], in_000[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 2*16) = __builtin_e2k_qppermb(in_100[2*i+1], in_100[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 3*16) = __builtin_e2k_qppermb(in_100[2*i+1], in_100[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 4*16) = __builtin_e2k_qppermb(in_200[2*i+1], in_200[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 5*16) = __builtin_e2k_qppermb(in_200[2*i+1], in_200[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 6*16) = __builtin_e2k_qppermb(in_300[2*i+1], in_300[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 7*16) = __builtin_e2k_qppermb(in_300[2*i+1], in_300[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 8*16) = __builtin_e2k_qppermb(in_010[2*i+1], in_010[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 9*16) = __builtin_e2k_qppermb(in_010[2*i+1], in_010[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 10*16) = __builtin_e2k_qppermb(in_110[2*i+1], in_110[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 11*16) = __builtin_e2k_qppermb(in_110[2*i+1], in_110[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 12*16) = __builtin_e2k_qppermb(in_210[2*i+1], in_210[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 13*16) = __builtin_e2k_qppermb(in_210[2*i+1], in_210[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 14*16) = __builtin_e2k_qppermb(in_310[2*i+1], in_310[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 15*16) = __builtin_e2k_qppermb(in_310[2*i+1], in_310[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 16*16) = __builtin_e2k_qppermb(in_020[2*i+1], in_020[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 17*16) = __builtin_e2k_qppermb(in_020[2*i+1], in_020[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 18*16) = __builtin_e2k_qppermb(in_120[2*i+1], in_120[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 19*16) = __builtin_e2k_qppermb(in_120[2*i+1], in_120[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 20*16) = __builtin_e2k_qppermb(in_220[2*i+1], in_220[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 21*16) = __builtin_e2k_qppermb(in_220[2*i+1], in_220[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 22*16) = __builtin_e2k_qppermb(in_320[2*i+1], in_320[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 23*16) = __builtin_e2k_qppermb(in_320[2*i+1], in_320[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 24*16) = __builtin_e2k_qppermb(in_030[2*i+1], in_030[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 25*16) = __builtin_e2k_qppermb(in_030[2*i+1], in_030[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 26*16) = __builtin_e2k_qppermb(in_130[2*i+1], in_130[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 27*16) = __builtin_e2k_qppermb(in_130[2*i+1], in_130[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 28*16) = __builtin_e2k_qppermb(in_230[2*i+1], in_230[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 29*16) = __builtin_e2k_qppermb(in_230[2*i+1], in_230[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 30*16) = __builtin_e2k_qppermb(in_330[2*i+1], in_330[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 31*16) = __builtin_e2k_qppermb(in_330[2*i+1], in_330[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 32*16) = __builtin_e2k_qppermb(in_001[2*i+1], in_001[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 33*16) = __builtin_e2k_qppermb(in_001[2*i+1], in_001[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 34*16) = __builtin_e2k_qppermb(in_101[2*i+1], in_101[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 35*16) = __builtin_e2k_qppermb(in_101[2*i+1], in_101[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 36*16) = __builtin_e2k_qppermb(in_201[2*i+1], in_201[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 37*16) = __builtin_e2k_qppermb(in_201[2*i+1], in_201[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 38*16) = __builtin_e2k_qppermb(in_301[2*i+1], in_301[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 39*16) = __builtin_e2k_qppermb(in_301[2*i+1], in_301[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 40*16) = __builtin_e2k_qppermb(in_011[2*i+1], in_011[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 41*16) = __builtin_e2k_qppermb(in_011[2*i+1], in_011[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 42*16) = __builtin_e2k_qppermb(in_111[2*i+1], in_111[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 43*16) = __builtin_e2k_qppermb(in_111[2*i+1], in_111[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 44*16) = __builtin_e2k_qppermb(in_211[2*i+1], in_211[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 45*16) = __builtin_e2k_qppermb(in_211[2*i+1], in_211[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 46*16) = __builtin_e2k_qppermb(in_311[2*i+1], in_311[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 47*16) = __builtin_e2k_qppermb(in_311[2*i+1], in_311[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 48*16) = __builtin_e2k_qppermb(in_021[2*i+1], in_021[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 49*16) = __builtin_e2k_qppermb(in_021[2*i+1], in_021[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 50*16) = __builtin_e2k_qppermb(in_121[2*i+1], in_121[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 51*16) = __builtin_e2k_qppermb(in_121[2*i+1], in_121[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 52*16) = __builtin_e2k_qppermb(in_221[2*i+1], in_221[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 53*16) = __builtin_e2k_qppermb(in_221[2*i+1], in_221[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 54*16) = __builtin_e2k_qppermb(in_321[2*i+1], in_321[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 55*16) = __builtin_e2k_qppermb(in_321[2*i+1], in_321[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 56*16) = __builtin_e2k_qppermb(in_031[2*i+1], in_031[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 57*16) = __builtin_e2k_qppermb(in_031[2*i+1], in_031[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 58*16) = __builtin_e2k_qppermb(in_131[2*i+1], in_131[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 59*16) = __builtin_e2k_qppermb(in_131[2*i+1], in_131[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 60*16) = __builtin_e2k_qppermb(in_231[2*i+1], in_231[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 61*16) = __builtin_e2k_qppermb(in_231[2*i+1], in_231[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 62*16) = __builtin_e2k_qppermb(in_331[2*i+1], in_331[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 63*16) = __builtin_e2k_qppermb(in_331[2*i+1], in_331[2*i+0], mask_imag); } #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < count_out/32/2; ++i) { uint64_t rev = __builtin_e2k_bitrevd(i); int64_t offset = 32 * (((rev>>(shift_out-1)) & 0xAAAAAAAAAAAAAAAA) | ((rev>>(shift_out+1)) & 0x5555555555555555)); __v2di mask_real = {0x0B0A090803020100, 0x1B1A191813121110}; __v2di mask_imag = {0x0F0E0D0C07060504, 0x1F1E1D1C17161514}; *(__v2du*)((void*)data_out + offset + 64*16) = __builtin_e2k_qppermb(in_002[2*i+1], in_002[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 65*16) = __builtin_e2k_qppermb(in_002[2*i+1], in_002[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 66*16) = __builtin_e2k_qppermb(in_102[2*i+1], in_102[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 67*16) = __builtin_e2k_qppermb(in_102[2*i+1], in_102[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 68*16) = __builtin_e2k_qppermb(in_202[2*i+1], in_202[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 69*16) = __builtin_e2k_qppermb(in_202[2*i+1], in_202[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 70*16) = __builtin_e2k_qppermb(in_302[2*i+1], in_302[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 71*16) = __builtin_e2k_qppermb(in_302[2*i+1], in_302[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 72*16) = __builtin_e2k_qppermb(in_012[2*i+1], in_012[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 73*16) = __builtin_e2k_qppermb(in_012[2*i+1], in_012[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 74*16) = __builtin_e2k_qppermb(in_112[2*i+1], in_112[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 75*16) = __builtin_e2k_qppermb(in_112[2*i+1], in_112[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 76*16) = __builtin_e2k_qppermb(in_212[2*i+1], in_212[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 77*16) = __builtin_e2k_qppermb(in_212[2*i+1], in_212[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 78*16) = __builtin_e2k_qppermb(in_312[2*i+1], in_312[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 79*16) = __builtin_e2k_qppermb(in_312[2*i+1], in_312[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 80*16) = __builtin_e2k_qppermb(in_022[2*i+1], in_022[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 81*16) = __builtin_e2k_qppermb(in_022[2*i+1], in_022[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 82*16) = __builtin_e2k_qppermb(in_122[2*i+1], in_122[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 83*16) = __builtin_e2k_qppermb(in_122[2*i+1], in_122[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 84*16) = __builtin_e2k_qppermb(in_222[2*i+1], in_222[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 85*16) = __builtin_e2k_qppermb(in_222[2*i+1], in_222[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 86*16) = __builtin_e2k_qppermb(in_322[2*i+1], in_322[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 87*16) = __builtin_e2k_qppermb(in_322[2*i+1], in_322[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 88*16) = __builtin_e2k_qppermb(in_032[2*i+1], in_032[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 89*16) = __builtin_e2k_qppermb(in_032[2*i+1], in_032[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 90*16) = __builtin_e2k_qppermb(in_132[2*i+1], in_132[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 91*16) = __builtin_e2k_qppermb(in_132[2*i+1], in_132[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 92*16) = __builtin_e2k_qppermb(in_232[2*i+1], in_232[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 93*16) = __builtin_e2k_qppermb(in_232[2*i+1], in_232[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 94*16) = __builtin_e2k_qppermb(in_332[2*i+1], in_332[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 95*16) = __builtin_e2k_qppermb(in_332[2*i+1], in_332[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 96*16) = __builtin_e2k_qppermb(in_003[2*i+1], in_003[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 97*16) = __builtin_e2k_qppermb(in_003[2*i+1], in_003[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 98*16) = __builtin_e2k_qppermb(in_103[2*i+1], in_103[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 99*16) = __builtin_e2k_qppermb(in_103[2*i+1], in_103[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 100*16) = __builtin_e2k_qppermb(in_203[2*i+1], in_203[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 101*16) = __builtin_e2k_qppermb(in_203[2*i+1], in_203[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 102*16) = __builtin_e2k_qppermb(in_303[2*i+1], in_303[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 103*16) = __builtin_e2k_qppermb(in_303[2*i+1], in_303[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 104*16) = __builtin_e2k_qppermb(in_013[2*i+1], in_013[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 105*16) = __builtin_e2k_qppermb(in_013[2*i+1], in_013[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 106*16) = __builtin_e2k_qppermb(in_113[2*i+1], in_113[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 107*16) = __builtin_e2k_qppermb(in_113[2*i+1], in_113[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 108*16) = __builtin_e2k_qppermb(in_213[2*i+1], in_213[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 109*16) = __builtin_e2k_qppermb(in_213[2*i+1], in_213[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 110*16) = __builtin_e2k_qppermb(in_313[2*i+1], in_313[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 111*16) = __builtin_e2k_qppermb(in_313[2*i+1], in_313[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 112*16) = __builtin_e2k_qppermb(in_023[2*i+1], in_023[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 113*16) = __builtin_e2k_qppermb(in_023[2*i+1], in_023[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 114*16) = __builtin_e2k_qppermb(in_123[2*i+1], in_123[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 115*16) = __builtin_e2k_qppermb(in_123[2*i+1], in_123[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 116*16) = __builtin_e2k_qppermb(in_223[2*i+1], in_223[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 117*16) = __builtin_e2k_qppermb(in_223[2*i+1], in_223[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 118*16) = __builtin_e2k_qppermb(in_323[2*i+1], in_323[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 119*16) = __builtin_e2k_qppermb(in_323[2*i+1], in_323[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 120*16) = __builtin_e2k_qppermb(in_033[2*i+1], in_033[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 121*16) = __builtin_e2k_qppermb(in_033[2*i+1], in_033[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 122*16) = __builtin_e2k_qppermb(in_133[2*i+1], in_133[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 123*16) = __builtin_e2k_qppermb(in_133[2*i+1], in_133[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 124*16) = __builtin_e2k_qppermb(in_233[2*i+1], in_233[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 125*16) = __builtin_e2k_qppermb(in_233[2*i+1], in_233[2*i+0], mask_imag); *(__v2du*)((void*)data_out + offset + 126*16) = __builtin_e2k_qppermb(in_333[2*i+1], in_333[2*i+0], mask_real); *(__v2du*)((void*)data_out + offset + 127*16) = __builtin_e2k_qppermb(in_333[2*i+1], in_333[2*i+0], mask_imag); } }
Основной цикл на ассемблере
.L15871: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=17, incr=1, ind=0, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=16, incr=1, ind=0, asz=1, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=15, incr=1, ind=0, asz=1, abs=2, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=14, incr=1, ind=0, asz=1, abs=2, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=13, incr=1, ind=0, asz=1, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=12, incr=1, ind=0, asz=1, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=11, incr=1, ind=0, asz=1, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=10, incr=1, ind=0, asz=1, abs=6, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=9, incr=1, ind=0, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=8, incr=1, ind=0, asz=1, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=7, incr=1, ind=0, asz=1, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=6, incr=1, ind=0, asz=1, abs=10, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=5, incr=1, ind=0, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=4, incr=1, ind=0, asz=1, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=3, incr=1, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=2, incr=1, ind=0, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=1, incr=1, ind=0, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=15, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=14, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=13, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=12, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=11, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=10, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=9, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=8, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=30, disp=0 } .L15388: { loop_mode disp %ctpr1, .L15388 bitrevd,0 %r0, %g16 addd,1 %r0, 0x1, %r0 movaqp,0 area=0, ind=16, am=1, be=0, %g18 movaqp,1 area=0, ind=0, am=0, be=0, %g17 movaqp,2 area=0, ind=16, am=1, be=0, %g20 movaqp,3 area=0, ind=0, am=0, be=0, %g19 } { loop_mode disp %ctpr3, .L13219 shrd,0 %g16, %r7, %g25 shrd,1 %g16, %r6, %g16 movaqp,0 area=1, ind=16, am=1, be=0, %g22 movaqp,1 area=1, ind=0, am=0, be=0, %g21 movaqp,2 area=1, ind=16, am=1, be=0, %g24 movaqp,3 area=1, ind=0, am=0, be=0, %g23 } { loop_mode andd,0 %g25, _f64,_lts0 0xaaaaaaaaaaaaaaaa, %g25 andd,1 %g16, _f64,_lts2 0x5555555555555555, %g16 movaqp,0 area=2, ind=16, am=1, be=0, %g27 movaqp,1 area=2, ind=0, am=0, be=0, %g26 movaqp,2 area=2, ind=16, am=1, be=0, %g29 movaqp,3 area=2, ind=0, am=0, be=0, %g28 } { loop_mode ord,0 %g25, %g16, %g16 movaqp,0 area=3, ind=16, am=1, be=0, %g31 movaqp,1 area=3, ind=0, am=0, be=0, %g30 movaqp,2 area=3, ind=16, am=1, be=0, %r9 movaqp,3 area=3, ind=0, am=0, be=0, %r1 } { loop_mode shld,0 %g16, 0x5, %g16 movaqp,0 area=4, ind=16, am=1, be=0, %r10 movaqp,1 area=4, ind=0, am=0, be=0, %g25 movaqp,2 area=4, ind=16, am=1, be=0, %r12 movaqp,3 area=4, ind=0, am=0, be=0, %r11 } { loop_mode qppermb,0 %g18, %g17, %r4, %r17 qppermb,1 %g18, %g17, %r5, %g17 addd,2 %r2, %g16, %g16 qppermb,3 %g20, %g19, %r4, %g18 qppermb,4 %g20, %g19, %r5, %g19 movaqp,0 area=5, ind=16, am=1, be=0, %r14 movaqp,1 area=5, ind=0, am=0, be=0, %r13 movaqp,2 area=5, ind=16, am=1, be=0, %r16 movaqp,3 area=5, ind=0, am=0, be=0, %r15 } { loop_mode qppermb,0 %g22, %g21, %r4, %r21 qppermb,1 %g22, %g21, %r5, %g21 stqp,2 %g16, _f16s,_lts0lo 0x410, %g17 qppermb,3 %g24, %g23, %r4, %g22 qppermb,4 %g24, %g23, %r5, %g23 movaqp,0 area=6, ind=16, am=1, be=0, %r18 movaqp,1 area=6, ind=0, am=0, be=0, %g20 movaqp,2 area=6, ind=16, am=1, be=0, %r20 movaqp,3 area=6, ind=0, am=0, be=0, %r19 } { loop_mode qppermb,0 %g27, %g26, %r4, %r24 qppermb,1 %g27, %g26, %r5, %g26 stqp,2 %g16, _f16s,_lts0lo 0x400, %r17 qppermb,3 %g29, %g28, %r4, %g27 qppermb,4 %g29, %g28, %r5, %g28 stqp,5 %g16, _f16s,_lts0hi 0x430, %g19 movaqp,0 area=7, ind=16, am=1, be=0, %g24 movaqp,1 area=7, ind=0, am=0, be=0, %g17 movaqp,2 area=7, ind=16, am=1, be=0, %r23 movaqp,3 area=7, ind=0, am=0, be=0, %r22 } { loop_mode qppermb,0 %g31, %g30, %r4, %r26 qppermb,1 %g31, %g30, %r5, %g30 stqp,2 %g16, _f16s,_lts0lo 0x420, %g18 qppermb,3 %r9, %r1, %r4, %g31 qppermb,4 %r9, %r1, %r5, %r1 stqp,5 %g16, _f16s,_lts0hi 0x470, %g23 movaqp,0 area=8, ind=16, am=1, be=0, %g29 movaqp,1 area=8, ind=0, am=0, be=0, %g19 movaqp,2 area=8, ind=16, am=1, be=0, %r25 movaqp,3 area=8, ind=0, am=0, be=0, %r17 } { loop_mode qppermb,0 %r10, %g25, %r4, %r28 qppermb,1 %r10, %g25, %r5, %g25 stqp,2 %g16, _f16s,_lts0lo 0x440, %r21 qppermb,3 %r12, %r11, %r4, %r10 qppermb,4 %r12, %r11, %r5, %r11 stqp,5 %g16, _f16s,_lts0hi 0x450, %g21 movaqp,0 area=9, ind=16, am=1, be=0, %g23 movaqp,1 area=9, ind=0, am=0, be=0, %g18 movaqp,2 area=9, ind=16, am=1, be=0, %r27 movaqp,3 area=9, ind=0, am=0, be=0, %r9 } { loop_mode qppermb,0 %r14, %r13, %r4, %r30 qppermb,1 %r14, %r13, %r5, %r13 stqp,2 %g16, _f16s,_lts0lo 0x460, %g22 qppermb,3 %r16, %r15, %r4, %r14 qppermb,4 %r16, %r15, %r5, %r15 stqp,5 %g16, _f16s,_lts0hi 0x490, %g26 movaqp,0 area=10, ind=16, am=1, be=0, %r12 movaqp,1 area=10, ind=0, am=0, be=0, %g21 movaqp,2 area=10, ind=16, am=1, be=0, %r29 movaqp,3 area=10, ind=0, am=0, be=0, %r21 } { loop_mode qppermb,0 %r18, %g20, %r4, %r32 qppermb,1 %r18, %g20, %r5, %g20 stqp,2 %g16, _f16s,_lts0lo 0x480, %r24 qppermb,3 %r20, %r19, %r4, %r18 qppermb,4 %r20, %r19, %r5, %r19 stqp,5 %g16, _f16s,_lts0hi 0x4b0, %g28 movaqp,0 area=11, ind=16, am=1, be=0, %g26 movaqp,1 area=11, ind=0, am=0, be=0, %g22 movaqp,2 area=11, ind=16, am=1, be=0, %r31 movaqp,3 area=11, ind=0, am=0, be=0, %r16 } { loop_mode qppermb,0 %g24, %g17, %r4, %r34 qppermb,1 %g24, %g17, %r5, %g17 stqp,2 %g16, _f16s,_lts0lo 0x4a0, %g27 qppermb,3 %r23, %r22, %r4, %g24 qppermb,4 %r23, %r22, %r5, %r22 stqp,5 %g16, _f16s,_lts0hi 0x4d0, %g30 movaqp,0 area=12, ind=16, am=1, be=0, %r20 movaqp,1 area=12, ind=0, am=0, be=0, %g28 movaqp,2 area=12, ind=16, am=1, be=0, %r33 movaqp,3 area=12, ind=0, am=0, be=0, %r24 } { loop_mode qppermb,0 %g29, %g19, %r4, %r36 qppermb,1 %g29, %g19, %r5, %g19 stqp,2 %g16, _f16s,_lts0lo 0x4c0, %r26 qppermb,3 %r25, %r17, %r4, %g29 qppermb,4 %r25, %r17, %r5, %r17 stqp,5 %g16, _f16s,_lts0hi 0x4f0, %r1 movaqp,0 area=13, ind=16, am=1, be=0, %g30 movaqp,1 area=13, ind=0, am=0, be=0, %g27 movaqp,2 area=13, ind=16, am=1, be=0, %r35 movaqp,3 area=13, ind=0, am=0, be=0, %r23 } { loop_mode qppermb,0 %g23, %g18, %r4, %r38 qppermb,1 %g23, %g18, %r5, %g18 stqp,2 %g16, _f16s,_lts0lo 0x4e0, %g31 qppermb,3 %r27, %r9, %r4, %g23 qppermb,4 %r27, %r9, %r5, %r9 stqp,5 %g16, _f16s,_lts0hi 0x510, %g25 movaqp,0 area=14, ind=16, am=1, be=0, %r25 movaqp,1 area=14, ind=0, am=0, be=0, %r1 movaqp,2 area=14, ind=16, am=1, be=0, %r37 movaqp,3 area=14, ind=0, am=0, be=0, %r26 } { loop_mode qppermb,0 %r12, %g21, %r4, %r40 qppermb,1 %r12, %g21, %r5, %g21 stqp,2 %g16, _f16s,_lts0lo 0x500, %r28 qppermb,3 %r29, %r21, %r4, %r12 qppermb,4 %r29, %r21, %r5, %r21 stqp,5 %g16, _f16s,_lts0hi 0x530, %r11 movaqp,0 area=15, ind=16, am=1, be=0, %g31 movaqp,1 area=15, ind=0, am=0, be=0, %g25 movaqp,2 area=15, ind=16, am=1, be=0, %r39 movaqp,3 area=15, ind=0, am=0, be=0, %r27 } { loop_mode qppermb,0 %g26, %g22, %r4, %r11 qppermb,1 %g26, %g22, %r5, %g22 stqp,2 %g16, _f16s,_lts0lo 0x520, %r10 qppermb,3 %r31, %r16, %r4, %g26 qppermb,4 %r31, %r16, %r5, %r16 stqp,5 %g16, _f16s,_lts0hi 0x550, %r13 } { loop_mode qppermb,0 %r20, %g28, %r4, %r10 qppermb,1 %r20, %g28, %r5, %g28 stqp,2 %g16, _f16s,_lts0lo 0x540, %r30 qppermb,3 %r33, %r24, %r4, %r13 qppermb,4 %r33, %r24, %r5, %r20 stqp,5 %g16, _f16s,_lts0hi 0x570, %r15 } { loop_mode qppermb,0 %g30, %g27, %r4, %r15 qppermb,1 %g30, %g27, %r5, %g27 stqp,2 %g16, _f16s,_lts0lo 0x560, %r14 qppermb,3 %r35, %r23, %r4, %g30 qppermb,4 %r35, %r23, %r5, %r23 stqp,5 %g16, _f16s,_lts0hi 0x590, %g20 } { loop_mode qppermb,0 %r25, %r1, %r4, %g20 qppermb,1 %r25, %r1, %r5, %r1 stqp,2 %g16, _f16s,_lts0lo 0x580, %r32 qppermb,3 %r37, %r26, %r4, %r14 qppermb,4 %r37, %r26, %r5, %r24 stqp,5 %g16, _f16s,_lts0hi 0x5b0, %r19 } { loop_mode qppermb,0 %g31, %g25, %r4, %r19 qppermb,1 %g31, %g25, %r5, %g25 stqp,2 %g16, _f16s,_lts0lo 0x5a0, %r18 qppermb,3 %r39, %r27, %r4, %g31 qppermb,4 %r39, %r27, %r5, %r25 stqp,5 %g16, _f16s,_lts0hi 0x5d0, %g17 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x5c0, %r34 stqp,5 %g16, _f16s,_lts0hi 0x5f0, %r22 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x5e0, %g24 stqp,5 %g16, _f16s,_lts0hi 0x610, %g19 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x600, %r36 stqp,5 %g16, _f16s,_lts0hi 0x630, %r17 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x620, %g29 stqp,5 %g16, _f16s,_lts0hi 0x650, %g18 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x640, %r38 stqp,5 %g16, _f16s,_lts0hi 0x670, %r9 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x660, %g23 stqp,5 %g16, _f16s,_lts0hi 0x690, %g21 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x680, %r40 stqp,5 %g16, _f16s,_lts0hi 0x6b0, %r21 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x6a0, %r12 stqp,5 %g16, _f16s,_lts0hi 0x6d0, %g22 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x6c0, %r11 stqp,5 %g16, _f16s,_lts0hi 0x6f0, %r16 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x6e0, %g26 stqp,5 %g16, _f16s,_lts0hi 0x710, %g28 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x700, %r10 stqp,5 %g16, _f16s,_lts0hi 0x730, %r20 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x720, %r13 stqp,5 %g16, _f16s,_lts0hi 0x750, %g27 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x740, %r15 stqp,5 %g16, _f16s,_lts0hi 0x770, %r23 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x760, %g30 stqp,5 %g16, _f16s,_lts0hi 0x790, %r1 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x780, %g20 stqp,5 %g16, _f16s,_lts0hi 0x7b0, %r24 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x7a0, %r14 stqp,5 %g16, _f16s,_lts0hi 0x7d0, %g25 } { loop_mode stqp,2 %g16, _f16s,_lts0lo 0x7c0, %r19 stqp,5 %g16, _f16s,_lts0hi 0x7f0, %r25 } { loop_mode ct %ctpr1 ? %NOT_LOOP_END alc alcf=0, alct=1 stqp,2 %g16, _f16s,_lts0lo 0x7e0, %g31 } { loop_mode ct %ctpr3 alc alcf=0, alct=1 } .L16599: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=17, incr=1, ind=0, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=16, incr=1, ind=0, asz=1, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=15, incr=1, ind=0, asz=1, abs=2, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=14, incr=1, ind=0, asz=1, abs=2, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=13, incr=1, ind=0, asz=1, abs=4, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=12, incr=1, ind=0, asz=1, abs=4, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=11, incr=1, ind=0, asz=1, abs=6, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=10, incr=1, ind=0, asz=1, abs=6, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=9, incr=1, ind=0, asz=1, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=8, incr=1, ind=0, asz=1, abs=8, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=7, incr=1, ind=0, asz=1, abs=10, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=6, incr=1, ind=0, asz=1, abs=10, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=5, incr=1, ind=0, asz=1, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=4, incr=1, ind=0, asz=1, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=3, incr=1, ind=0, asz=1, abs=14, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=2, incr=1, ind=0, asz=1, abs=14, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=1, incr=1, ind=0, asz=1, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=15, asz=1, abs=16, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=14, asz=1, abs=18, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=13, asz=1, abs=18, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=12, asz=1, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=11, asz=1, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=10, asz=1, abs=22, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=9, asz=1, abs=22, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=8, asz=1, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=7, asz=1, abs=24, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=6, asz=1, abs=26, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=5, asz=1, abs=26, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=4, asz=1, abs=28, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=3, asz=1, abs=28, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=2, asz=1, abs=30, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=30, disp=0 } .L15607: { loop_mode disp %ctpr1, .L15607 bitrevd,0 %r0, %g16 addd,1 %r0, 0x1, %r0 movaqp,0 area=0, ind=16, am=1, be=0, %g18 movaqp,1 area=0, ind=0, am=0, be=0, %g17 movaqp,2 area=0, ind=16, am=1, be=0, %g20 movaqp,3 area=0, ind=0, am=0, be=0, %g19 } { loop_mode disp %ctpr3, .L15094 shrd,0 %g16, %r7, %g25 shrd,1 %g16, %r6, %g16 movaqp,0 area=1, ind=16, am=1, be=0, %g22 movaqp,1 area=1, ind=0, am=0, be=0, %g21 movaqp,2 area=1, ind=16, am=1, be=0, %g24 movaqp,3 area=1, ind=0, am=0, be=0, %g23 } { loop_mode andd,0 %g25, _f64,_lts0 0xaaaaaaaaaaaaaaaa, %g25 andd,1 %g16, _f64,_lts2 0x5555555555555555, %g16 movaqp,0 area=2, ind=16, am=1, be=0, %g27 movaqp,1 area=2, ind=0, am=0, be=0, %g26 movaqp,2 area=2, ind=16, am=1, be=0, %g29 movaqp,3 area=2, ind=0, am=0, be=0, %g28 } { loop_mode ord,0 %g25, %g16, %g16 movaqp,0 area=3, ind=16, am=1, be=0, %g31 movaqp,1 area=3, ind=0, am=0, be=0, %g30 movaqp,2 area=3, ind=16, am=1, be=0, %r14 movaqp,3 area=3, ind=0, am=0, be=0, %r11 } { loop_mode shld,0 %g16, 0x5, %g16 movaqp,0 area=4, ind=16, am=1, be=0, %r18 movaqp,1 area=4, ind=0, am=0, be=0, %g25 movaqp,2 area=4, ind=16, am=1, be=0, %r20 movaqp,3 area=4, ind=0, am=0, be=0, %r19 } { loop_mode qppermb,0 %g18, %g17, %r4, %r31 qppermb,1 %g18, %g17, %r5, %g17 addd,2 %r2, %g16, %g20 qppermb,3 %g20, %g19, %r4, %g18 qppermb,4 %g20, %g19, %r5, %g19 movaqp,0 area=5, ind=16, am=1, be=0, %r25 movaqp,1 area=5, ind=0, am=0, be=0, %r23 movaqp,2 area=5, ind=16, am=1, be=0, %r29 movaqp,3 area=5, ind=0, am=0, be=0, %r28 } { loop_mode qppermb,0 %g22, %g21, %r4, %r50 qppermb,1 %g22, %g21, %r5, %g21 stqp,2 %r2, %g16, %r31 qppermb,3 %g24, %g23, %r4, %g22 qppermb,4 %g24, %g23, %r5, %g23 movaqp,0 area=6, ind=16, am=1, be=0, %r37 movaqp,1 area=6, ind=0, am=0, be=0, %r34 movaqp,2 area=6, ind=16, am=1, be=0, %r46 movaqp,3 area=6, ind=0, am=0, be=0, %r39 } { loop_mode qppermb,0 %g27, %g26, %r4, %r56 qppermb,1 %g27, %g26, %r5, %g26 stqp,2 0x10, %g20, %g17 qppermb,3 %g29, %g28, %r4, %g27 qppermb,4 %g29, %g28, %r5, %g28 stqp,5 %g20, _f16s,_lts0lo 0x30, %g19 movaqp,0 area=7, ind=16, am=1, be=0, %g24 movaqp,1 area=7, ind=0, am=0, be=0, %g16 movaqp,2 area=7, ind=16, am=1, be=0, %r53 movaqp,3 area=7, ind=0, am=0, be=0, %r31 } { loop_mode qppermb,0 %g31, %g30, %r4, %r58 qppermb,1 %g31, %g30, %r5, %g30 stqp,2 %g20, _f16s,_lts0lo 0x20, %g18 qppermb,3 %r14, %r11, %r4, %g31 qppermb,4 %r14, %r11, %r5, %r11 stqp,5 %g20, _f16s,_lts0hi 0x70, %g23 movaqp,0 area=8, ind=16, am=1, be=0, %g19 movaqp,1 area=8, ind=0, am=0, be=0, %g17 movaqp,2 area=8, ind=16, am=1, be=0, %r57 movaqp,3 area=8, ind=0, am=0, be=0, %g29 } { loop_mode qppermb,0 %r18, %g25, %r4, %r61 qppermb,1 %r18, %g25, %r5, %g25 stqp,2 %g20, _f16s,_lts0lo 0x40, %r50 qppermb,3 %r20, %r19, %r4, %r18 qppermb,4 %r20, %r19, %r5, %r19 stqp,5 %g20, _f16s,_lts0hi 0x50, %g21 movaqp,0 area=9, ind=16, am=1, be=0, %g23 movaqp,1 area=9, ind=0, am=0, be=0, %g18 movaqp,2 area=9, ind=16, am=1, be=0, %r59 movaqp,3 area=9, ind=0, am=0, be=0, %r14 } { loop_mode qppermb,0 %r25, %r23, %r4, %r63 qppermb,1 %r25, %r23, %r5, %r23 stqp,2 %g20, _f16s,_lts0lo 0x60, %g22 qppermb,3 %r29, %r28, %r4, %r25 qppermb,4 %r29, %r28, %r5, %r28 stqp,5 %g20, _f16s,_lts0hi 0x90, %g26 movaqp,0 area=10, ind=16, am=1, be=0, %r20 movaqp,1 area=10, ind=0, am=0, be=0, %g21 movaqp,2 area=10, ind=16, am=1, be=0, %r62 movaqp,3 area=10, ind=0, am=0, be=0, %r50 } { loop_mode qppermb,0 %r37, %r34, %r4, %b[1] qppermb,1 %r37, %r34, %r5, %r34 stqp,2 %g20, _f16s,_lts0lo 0x80, %r56 qppermb,3 %r46, %r39, %r4, %r37 qppermb,4 %r46, %r39, %r5, %r39 stqp,5 %g20, _f16s,_lts0hi 0xb0, %g28 movaqp,0 area=11, ind=16, am=1, be=0, %g26 movaqp,1 area=11, ind=0, am=0, be=0, %g22 movaqp,2 area=11, ind=16, am=1, be=0, %b[0] movaqp,3 area=11, ind=0, am=0, be=0, %r29 } { loop_mode qppermb,0 %g24, %g16, %r4, %b[3] qppermb,1 %g24, %g16, %r5, %g16 stqp,2 %g20, _f16s,_lts0lo 0xa0, %g27 qppermb,3 %r53, %r31, %r4, %g24 qppermb,4 %r53, %r31, %r5, %r31 stqp,5 %g20, _f16s,_lts0hi 0xd0, %g30 movaqp,0 area=12, ind=16, am=1, be=0, %r46 movaqp,1 area=12, ind=0, am=0, be=0, %g28 movaqp,2 area=12, ind=16, am=1, be=0, %b[2] movaqp,3 area=12, ind=0, am=0, be=0, %r56 } { loop_mode qppermb,0 %g19, %g17, %r4, %b[5] qppermb,1 %g19, %g17, %r5, %g17 stqp,2 %g20, _f16s,_lts0lo 0xc0, %r58 qppermb,3 %r57, %g29, %r4, %g19 qppermb,4 %r57, %g29, %r5, %g29 stqp,5 %g20, _f16s,_lts0hi 0xf0, %r11 movaqp,0 area=13, ind=16, am=1, be=0, %g30 movaqp,1 area=13, ind=0, am=0, be=0, %g27 movaqp,2 area=13, ind=16, am=1, be=0, %b[4] movaqp,3 area=13, ind=0, am=0, be=0, %r53 } { loop_mode qppermb,0 %g23, %g18, %r4, %b[8] qppermb,1 %g23, %g18, %r5, %g18 stqp,2 %g20, _f16s,_lts0lo 0xe0, %g31 qppermb,3 %r59, %r14, %r4, %g23 qppermb,4 %r59, %r14, %r5, %r14 stqp,5 %g20, _f16s,_lts0hi 0x110, %g25 movaqp,0 area=14, ind=16, am=1, be=0, %r57 movaqp,1 area=14, ind=0, am=0, be=0, %r11 movaqp,2 area=14, ind=16, am=1, be=0, %b[6] movaqp,3 area=14, ind=0, am=0, be=0, %r58 } { loop_mode qppermb,0 %r20, %g21, %r4, %b[10] qppermb,1 %r20, %g21, %r5, %g21 stqp,2 %g20, _f16s,_lts0lo 0x100, %r61 qppermb,3 %r62, %r50, %r4, %r20 qppermb,4 %r62, %r50, %r5, %r50 stqp,5 %g20, _f16s,_lts0hi 0x130, %r19 movaqp,0 area=15, ind=16, am=1, be=0, %g31 movaqp,1 area=15, ind=0, am=0, be=0, %g25 movaqp,2 area=15, ind=16, am=1, be=0, %b[9] movaqp,3 area=15, ind=0, am=0, be=0, %r59 } { loop_mode qppermb,0 %g26, %g22, %r4, %r19 qppermb,1 %g26, %g22, %r5, %g22 stqp,2 %g20, _f16s,_lts0lo 0x120, %r18 qppermb,3 %b[0], %r29, %r4, %g26 qppermb,4 %b[0], %r29, %r5, %r29 stqp,5 %g20, _f16s,_lts0hi 0x150, %r23 } { loop_mode qppermb,0 %r46, %g28, %r4, %r18 qppermb,1 %r46, %g28, %r5, %g28 stqp,2 %g20, _f16s,_lts0lo 0x140, %r63 qppermb,3 %b[2], %r56, %r4, %r23 qppermb,4 %b[2], %r56, %r5, %r46 stqp,5 %g20, _f16s,_lts0hi 0x170, %r28 } { loop_mode qppermb,0 %g30, %g27, %r4, %r28 qppermb,1 %g30, %g27, %r5, %g27 stqp,2 %g20, _f16s,_lts0lo 0x160, %r25 qppermb,3 %b[4], %r53, %r4, %g30 qppermb,4 %b[4], %r53, %r5, %r53 stqp,5 %g20, _f16s,_lts0hi 0x190, %r34 } { loop_mode qppermb,0 %r57, %r11, %r4, %r25 qppermb,1 %r57, %r11, %r5, %r11 stqp,2 %g20, _f16s,_lts0lo 0x180, %b[1] qppermb,3 %b[6], %r58, %r4, %r34 qppermb,4 %b[6], %r58, %r5, %r56 stqp,5 %g20, _f16s,_lts0hi 0x1b0, %r39 } { loop_mode qppermb,0 %g31, %g25, %r4, %r39 qppermb,1 %g31, %g25, %r5, %g25 stqp,2 %g20, _f16s,_lts0lo 0x1a0, %r37 qppermb,3 %b[9], %r59, %r4, %g31 qppermb,4 %b[9], %r59, %r5, %r57 stqp,5 %g20, _f16s,_lts0hi 0x1d0, %g16 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x1c0, %b[3] stqp,5 %g20, _f16s,_lts0hi 0x1f0, %r31 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x1e0, %g24 stqp,5 %g20, _f16s,_lts0hi 0x210, %g17 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x200, %b[5] stqp,5 %g20, _f16s,_lts0hi 0x230, %g29 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x220, %g19 stqp,5 %g20, _f16s,_lts0hi 0x250, %g18 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x240, %b[8] stqp,5 %g20, _f16s,_lts0hi 0x270, %r14 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x260, %g23 stqp,5 %g20, _f16s,_lts0hi 0x290, %g21 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x280, %b[10] stqp,5 %g20, _f16s,_lts0hi 0x2b0, %r50 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x2a0, %r20 stqp,5 %g20, _f16s,_lts0hi 0x2d0, %g22 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x2c0, %r19 stqp,5 %g20, _f16s,_lts0hi 0x2f0, %r29 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x2e0, %g26 stqp,5 %g20, _f16s,_lts0hi 0x310, %g28 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x300, %r18 stqp,5 %g20, _f16s,_lts0hi 0x330, %r46 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x320, %r23 stqp,5 %g20, _f16s,_lts0hi 0x350, %g27 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x340, %r28 stqp,5 %g20, _f16s,_lts0hi 0x370, %r53 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x360, %g30 stqp,5 %g20, _f16s,_lts0hi 0x390, %r11 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x380, %r25 stqp,5 %g20, _f16s,_lts0hi 0x3b0, %r56 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x3a0, %r34 stqp,5 %g20, _f16s,_lts0hi 0x3d0, %g25 } { loop_mode stqp,2 %g20, _f16s,_lts0lo 0x3c0, %r39 stqp,5 %g20, _f16s,_lts0hi 0x3f0, %r57 } { loop_mode ct %ctpr1 ? %NOT_LOOP_END alc alcf=0, alct=1 stqp,2 %g20, _f16s,_lts0lo 0x3e0, %g31 } { loop_mode ct %ctpr3 alc alcf=0, alct=1 }
Теоретическая скорость: 128 комплексных чисел за 40 тактов (128/40) = 25.6 Байт/такт
Замеры скорости
Видим замедление по всей длине графика.
Итоги по reverse_radix4_vector4
Победителем можно считать reverse_radix4_vector4_stream16.
При реализации Vector-4 Radix-4 FFT будем использовать его.
Пишем векторный Stage
stage_radix2_vector2
1. stage_radix2_vector2_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void stage_radix2_vector2_etalon(int data_count, myComplex2 *data_in, myComplex2 *data_out, myComplex2 *coef) { myComplex2 *x_in = &data_in[0]; myComplex2 *y_in = &data_in[1]; myComplex2 *c_in = coef; myComplex2 *out_add = &data_out[0]; myComplex2 *out_sub = &data_out[data_count/2]; #pragma ivdep #pragma unroll(1) // #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { myComplex2 x = x_in[2*i]; myComplex2 y = y_in[2*i]; myComplex2 c = c_in[i]; myComplex2 cy = complex2_mul(c, y); out_add[i] = complex2_add(x, cy); out_sub[i] = complex2_sub(x, cy); } }
Основной цикл на ассемблере
.L917: { fapb ct=1, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=5, abs=0, disp=0 } .L546: { loop_mode pfmuls,0,sm %b[67], %b[6], %b[37] pfsubs,1,sm %b[46], %b[24], %b[57] staad,2 %b[61], %aad1[ %aasti3 + _f32s,_lts0 0x8 ] pfmul_adds,3,sm %b[55], %b[13], %b[43], %b[14] pfadds,4,sm %b[46], %b[24], %b[58] staad,5 %b[62], %aad2[ %aasti4 + _f32s,_lts0 0x8 ] movad,0 area=0, ind=16, am=0, be=0, %b[0] movad,1 area=0, ind=24, am=0, be=0, %b[1] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmuls,0,sm %b[67], %b[7], %b[62] pfsubs,1,sm %b[35], %b[56], %b[69] staad,2 %b[73], %aad1[ %aasti3 ] incr,2 %aaincr3 pfmul_rsubs,3,sm %b[55], %b[12], %b[68], %b[46] pfadds,4,sm %b[35], %b[56], %b[70] staad,5 %b[74], %aad2[ %aasti4 ] incr,5 %aaincr3 movad,0 area=0, ind=0, am=0, be=0, %b[13] movad,1 area=0, ind=8, am=1, be=0, %b[24] movad,2 area=0, ind=8, am=1, be=0, %b[61] movad,3 area=0, ind=0, am=0, be=0, %b[43] }
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
2. stage_radix2_vector2
Прямая векторизация базового алгоритма в 2 раза с помощью инструкций SIMD64.
Код на Си
void stage_radix2_vector2(int data_count, myComplex2 *data_in, myComplex2 *data_out, myComplex2 *coef) { uint64_t *x_real_in = (uint64_t*)&data_in[0].real; uint64_t *x_imag_in = (uint64_t*)&data_in[0].imag; uint64_t *y_real_in = (uint64_t*)&data_in[1].real; uint64_t *y_imag_in = (uint64_t*)&data_in[1].imag; uint64_t *c_real_in = (uint64_t*)&coef[0].real; uint64_t *c_imag_in = (uint64_t*)&coef[0].imag; uint64_t *out_add_real = (uint64_t*)&data_out[0].real; uint64_t *out_add_imag = (uint64_t*)&data_out[0].imag; uint64_t *out_sub_real = (uint64_t*)&data_out[data_count/2].real; uint64_t *out_sub_imag = (uint64_t*)&data_out[data_count/2].imag; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { uint64_t x_real = x_real_in[4*i]; uint64_t x_imag = x_imag_in[4*i]; uint64_t y_real = y_real_in[4*i]; uint64_t y_imag = y_imag_in[4*i]; uint64_t c_real = c_real_in[2*i]; uint64_t c_imag = c_imag_in[2*i]; uint64_t cy_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(c_real, y_real), __builtin_e2k_pfmuls(c_imag, y_imag)); uint64_t cy_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(c_real, y_imag), __builtin_e2k_pfmuls(c_imag, y_real)); out_add_real[2*i] = __builtin_e2k_pfadds(x_real, cy_real); out_add_imag[2*i] = __builtin_e2k_pfadds(x_imag, cy_imag); out_sub_real[2*i] = __builtin_e2k_pfsubs(x_real, cy_real); out_sub_imag[2*i] = __builtin_e2k_pfsubs(x_imag, cy_imag); } }
Основной цикл на ассемблере
.L1379: { fapb ct=1, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=5, abs=0, disp=0 } .L1117: { loop_mode pfmuls,0,sm %b[67], %b[6], %b[37] pfsubs,1,sm %b[46], %b[24], %b[57] staad,2 %b[61], %aad1[ %aasti3 + _f32s,_lts0 0x8 ] pfmul_adds,3,sm %b[55], %b[13], %b[43], %b[14] pfadds,4,sm %b[46], %b[24], %b[58] staad,5 %b[62], %aad2[ %aasti4 + _f32s,_lts0 0x8 ] movad,0 area=0, ind=16, am=0, be=0, %b[0] movad,1 area=0, ind=24, am=0, be=0, %b[1] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmuls,0,sm %b[67], %b[7], %b[62] pfsubs,1,sm %b[35], %b[56], %b[69] staad,2 %b[73], %aad1[ %aasti3 ] incr,2 %aaincr3 pfmul_rsubs,3,sm %b[55], %b[12], %b[68], %b[46] pfadds,4,sm %b[35], %b[56], %b[70] staad,5 %b[74], %aad2[ %aasti4 ] incr,5 %aaincr3 movad,0 area=0, ind=0, am=0, be=0, %b[13] movad,1 area=0, ind=8, am=1, be=0, %b[24] movad,2 area=0, ind=8, am=1, be=0, %b[61] movad,3 area=0, ind=0, am=0, be=0, %b[43] }
Код на ассемблере получился такой же, как в эталонном варианте, то есть компилятор самостоятельно векторизовал эталонный вариант в 2 раза.
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
Из консольного вывода от разных размеров массива видно, что эталонный вариант всегда выполняется на 40–50 тактов быстрее.
Я не смог понять, с чем это связано (особенности расположения кода в памяти?).
Итоги по stage_radix2_vector2
stage_radix2_vector4
1. stage_radix2_vector4_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void stage_radix2_vector4_etalon(int data_count, myComplex4 *data_in, myComplex4 *data_out, myComplex4 *coef) { myComplex4 *x_in = &data_in[0]; myComplex4 *y_in = &data_in[1]; myComplex4 *c_in = coef; myComplex4 *out_add = &data_out[0]; myComplex4 *out_sub = &data_out[data_count/2]; #pragma ivdep #pragma unroll(1) // #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { myComplex4 x = x_in[2*i]; myComplex4 y = y_in[2*i]; myComplex4 c = c_in[i]; myComplex4 cy = complex4_mul(c, y); out_add[i] = complex4_add(x, cy); out_sub[i] = complex4_sub(x, cy); } }
Основной цикл на ассемблере
.L1531: { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=0, disp=16 } { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=4, abs=16, disp=32 } .L926: { loop_mode pfmuls,0,sm %b[77], %b[39], %b[22] pfsubs,1,sm %b[36], %b[69], %b[52] staad,2 %b[54], %aad1[ %aasti3 + _f32s,_lts0 0x8 ] pfmul_adds,3,sm %b[51], %b[43], %b[72], %b[7] pfadds,4,sm %b[36], %b[69], %b[60] staad,5 %b[62], %aad2[ %aasti4 + _f32s,_lts0 0x8 ] movad,2 area=1, ind=16, am=0, be=0, %b[0] movad,3 area=0, ind=0, am=0, be=0, %b[1] } { loop_mode pfmuls,0,sm %b[5], %b[4], %b[62] pfsubs,1,sm %b[23], %b[63], %b[66] staad,2 %b[68], %aad1[ %aasti3 ] pfmul_adds,3,sm %b[44], %b[8], %b[78], %b[54] pfadds,4,sm %b[23], %b[63], %b[69] staad,5 %b[71], %aad2[ %aasti4 ] movad,0 area=1, ind=0, am=0, be=0, %b[36] movad,1 area=1, ind=8, am=1, be=0, %b[43] movad,3 area=1, ind=8, am=0, be=0, %b[51] } { loop_mode pfmuls,0,sm %b[77], %b[55], %b[68] pfsubs,1,sm %b[35], %b[11], %b[71] staad,2 %b[73], %aad1[ %aasti3 + _f32s,_lts0 0x18 ] pfmul_rsubs,3,sm %b[49], %b[57], %b[24], %b[63] pfadds,4,sm %b[35], %b[11], %b[72] staad,5 %b[74], %aad2[ %aasti4 + _f32s,_lts0 0x18 ] movad,0 area=0, ind=16, am=0, be=0, %b[8] movad,1 area=0, ind=24, am=0, be=0, %b[23] movad,3 area=1, ind=0, am=0, be=0, %b[44] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmuls,0,sm %b[5], %b[48], %b[74] pfsubs,1,sm %b[20], %b[58], %b[77] staad,2 %b[79], %aad1[ %aasti3 + _f32s,_lts0 0x10 ] incr,2 %aaincr3 pfmul_rsubs,3,sm %b[42], %b[50], %b[64], %b[57] pfadds,4,sm %b[20], %b[58], %b[78] staad,5 %b[80], %aad2[ %aasti4 + _f32s,_lts0 0x10 ] incr,5 %aaincr3 movad,0 area=0, ind=0, am=0, be=0, %b[11] movad,1 area=0, ind=8, am=1, be=0, %b[24] movad,2 area=1, ind=24, am=1, be=0, %b[35] movad,3 area=0, ind=8, am=1, be=0, %b[73] }
Теоретическая скорость: 8 комплексных чисел за 4 такта (8/4) = 16 Байт/такт
Замеры скорости
2. stage_radix2_vector4
Прямая векторизация базового алгоритма в 4 раза с помощью инструкций SIMD128.
Код на Си
void stage_radix2_vector4(int data_count, myComplex4 *data_in, myComplex4 *data_out, myComplex4 *coef) { __v2di *x_real_in = (__v2di*)&data_in[0].real; __v2di *x_imag_in = (__v2di*)&data_in[0].imag; __v2di *y_real_in = (__v2di*)&data_in[1].real; __v2di *y_imag_in = (__v2di*)&data_in[1].imag; __v2di *c_real_in = (__v2di*)&coef[0].real; __v2di *c_imag_in = (__v2di*)&coef[0].imag; __v2di *out_add_real = (__v2di*)&data_out[0].real; __v2di *out_add_imag = (__v2di*)&data_out[0].imag; __v2di *out_sub_real = (__v2di*)&data_out[data_count/2].real; __v2di *out_sub_imag = (__v2di*)&data_out[data_count/2].imag; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { __v2di x_real = x_real_in[4*i]; __v2di x_imag = x_imag_in[4*i]; __v2di y_real = y_real_in[4*i]; __v2di y_imag = y_imag_in[4*i]; __v2di c_real = c_real_in[2*i]; __v2di c_imag = c_imag_in[2*i]; __v2di cy_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(c_real, y_real), __builtin_e2k_qpfmuls(c_imag, y_imag)); __v2di cy_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(c_real, y_imag), __builtin_e2k_qpfmuls(c_imag, y_real)); out_add_real[2*i] = __builtin_e2k_qpfadds(x_real, cy_real); out_add_imag[2*i] = __builtin_e2k_qpfadds(x_imag, cy_imag); out_sub_real[2*i] = __builtin_e2k_qpfsubs(x_real, cy_real); out_sub_imag[2*i] = __builtin_e2k_qpfsubs(x_imag, cy_imag); } }
Основной цикл на ассемблере
.L1875: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=32 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=4, abs=16, disp=0 } .L1577: { loop_mode qpfmuls,0,sm %b[43], %b[52], %b[45] qpfsubs,1,sm %b[46], %b[24], %b[60] staaqp,2 %b[64], %aad1[ %aasti3 + _f32s,_lts0 0x10 ] qpfmul_adds,3,sm %b[12], %b[13], %b[51], %b[14] qpfadds,4,sm %b[46], %b[24], %b[63] staaqp,5 %b[67], %aad2[ %aasti4 + _f32s,_lts0 0x10 ] movaqp,0 area=1, ind=16, am=1, be=0, %b[37] movaqp,1 area=1, ind=0, am=0, be=0, %b[0] movaqp,3 area=0, ind=16, am=0, be=0, %b[1] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfmuls,0,sm %b[43], %b[7], %b[64] qpfsubs,1,sm %b[35], %b[61], %b[67] staaqp,2 %b[71], %aad1[ %aasti3 ] incr,2 %aaincr3 qpfmul_rsubs,3,sm %b[12], %b[58], %b[70], %b[51] qpfadds,4,sm %b[35], %b[61], %b[72] staaqp,5 %b[76], %aad2[ %aasti4 ] incr,5 %aaincr3 movaqp,0 area=0, ind=0, am=0, be=0, %b[13] movaqp,1 area=0, ind=16, am=1, be=0, %b[24] movaqp,3 area=0, ind=0, am=1, be=0, %b[46] }
Теоретическая скорость: 8 комплексных чисел за 2 такта (8/2) = 32 Байт/такт
Замеры скорости
Компилятор не сумел векторизовать вариант stage_radix2_vector4_etalon в 4 раза.
До этого он успешно векторизовал вариант stage_radix2_vector2_etalon в 2 раза.
Итоги по stage_radix2_vector4
stage_radix4_vector2
Один проход по stage_radix4_vector2 совершает ту же работу, что 2 прохода по stage_radix2. Поэтому скорость stage_radix4_vector2 будем умножать на 2 для удобства сравнения с stage_radix2 (этот факт подписан на оси графика и в выводе консоли).
1. stage_radix4_vector2_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void stage_radix4_vector2_etalon(int data_count, myComplex2 *data_in, myComplex2 *data_out, myComplex2 *coefC, myComplex2 *coefD, myComplex2 *coefE) { myComplex2 *x_in = &data_in[0]; myComplex2 *y_in = &data_in[1]; myComplex2 *z_in = &data_in[2]; myComplex2 *w_in = &data_in[3]; myComplex2 *c_in = coefC; myComplex2 *d_in = coefD; myComplex2 *e_in = coefE; myComplex2 *out_0 = &data_out[0*data_count/4]; myComplex2 *out_1 = &data_out[1*data_count/4]; myComplex2 *out_2 = &data_out[2*data_count/4]; myComplex2 *out_3 = &data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) // #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { myComplex2 x = x_in[4*i]; myComplex2 y = y_in[4*i]; myComplex2 z = z_in[4*i]; myComplex2 w = w_in[4*i]; myComplex2 c = c_in[i]; myComplex2 d = d_in[i]; myComplex2 e = e_in[i]; myComplex2 cy = complex2_mul(c, y); myComplex2 dz = complex2_mul(d, z); myComplex2 ew = complex2_mul(e, w); myComplex2 add02 = complex2_add( x, dz); myComplex2 sub02 = complex2_sub( x, dz); myComplex2 add13 = complex2_add(cy, ew); myComplex2 sub13 = complex2_sub(cy, ew); myComplex2 sub13i = complex2_mul_i(sub13); out_0[i] = complex2_add(add02, add13); out_1[i] = complex2_sub(sub02, sub13i); out_2[i] = complex2_sub(add02, add13); out_3[i] = complex2_add(sub02, sub13i); } }
Основной цикл на ассемблере
.L2695: { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=4, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=3, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=16, disp=0 } .L1896: { loop_mode pfmul_adds,0,sm %b[48], %b[70], %b[42], %b[0] pfsub_rsubs,1,sm %b[37], %b[73], %b[59], %b[4] pfsub_adds,2,sm %b[37], %b[73], %b[59], %b[1] pfmuls,4,sm %b[63], %b[62], %b[40] pfsubs,5,sm %b[78], %b[74], %b[57] } { loop_mode pfmul_rsubs,0,sm %b[48], %b[64], %b[83], %b[70] pfadd_adds,1,sm %b[25], %b[82], %b[90], %b[77] pfadd_rsubs,2,sm %b[25], %b[82], %b[90], %b[73] pfmuls,4,sm %b[63], %b[68], %b[81] pfadds,5,sm %b[78], %b[74], %b[88] movad,0 area=2, ind=0, am=0, be=0, %b[42] movad,1 area=2, ind=8, am=1, be=0, %b[59] movad,2 area=1, ind=0, am=0, be=0, %b[37] movad,3 area=0, ind=16, am=0, be=0, %b[58] } { loop_mode pfmul_rsubs,0,sm %b[69], %b[13], %b[84], %b[74] pfmul_rsubs,1,sm %b[45], %b[34], %b[85], %b[78] staad,2 %b[49], %aad3[ %aasti7 ] pfmuls,3,sm %b[14], %b[53], %b[82] pfmuls,4,sm %b[54], %b[20], %b[83] staad,5 %b[26], %aad1[ %aasti5 ] movad,0 area=1, ind=0, am=0, be=0, %b[63] movad,1 area=0, ind=8, am=0, be=0, %b[25] movad,2 area=1, ind=8, am=1, be=0, %b[48] movad,3 area=0, ind=24, am=0, be=0, %b[64] } { loop_mode pfmul_adds,0,sm %b[67], %b[53], %b[86], %b[34] pfmul_adds,1,sm %b[45], %b[22], %b[87], %b[69] staad,2 %b[10], %aad4[ %aasti8 + _f32s,_lts0 0x8 ] pfmuls,3,sm %b[12], %b[9], %b[84] pfmuls,4,sm %b[54], %b[32], %b[85] staad,5 %b[7], %aad2[ %aasti6 + _f32s,_lts0 0x8 ] movad,0 area=0, ind=0, am=0, be=0, %b[13] movad,1 area=0, ind=24, am=0, be=0, %b[49] movad,2 area=0, ind=0, am=0, be=0, %b[26] movad,3 area=0, ind=8, am=1, be=0, %b[14] } { loop_mode pfsub_rsubs,0,sm %b[23], %b[80], %b[55], %b[45] pfsub_adds,1,sm %b[23], %b[80], %b[55], %b[22] staad,2 %b[6], %aad3[ %aasti7 + _f32s,_lts0 0x8 ] incr,2 %aaincr3 pfsubs,4,sm %b[2], %b[38], %b[53] staad,5 %b[3], %aad1[ %aasti5 + _f32s,_lts0 0x8 ] incr,5 %aaincr3 movad,0 area=1, ind=8, am=1, be=0, %b[10] movad,1 area=0, ind=16, am=1, be=0, %b[7] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfadd_adds,0,sm %b[35], %b[71], %b[56], %b[6] pfadd_rsubs,1,sm %b[35], %b[71], %b[56], %b[3] staad,2 %b[79], %aad4[ %aasti8 ] incr,2 %aaincr3 pfadds,4,sm %b[38], %b[2], %b[54] staad,5 %b[75], %aad2[ %aasti6 ] incr,5 %aaincr3 }
Теоретическая скорость: 8 комплексных чисел за 6 тактов (8/6) = 10.67 Байт/такт
Двойная теоретическая скорость: 21.33 Байт/такт
Замеры скорости
2. stage_radix4_vector2
Прямая векторизация базового алгоритма в 2 раза с помощью инструкций SIMD64.
Код на Си
void stage_radix4_vector2(int data_count, myComplex2 *data_in, myComplex2 *data_out, myComplex2 *coefC, myComplex2 *coefD, myComplex2 *coefE) { uint64_t *x_real_in = (uint64_t*)&data_in[0].real; uint64_t *x_imag_in = (uint64_t*)&data_in[0].imag; uint64_t *y_real_in = (uint64_t*)&data_in[1].real; uint64_t *y_imag_in = (uint64_t*)&data_in[1].imag; uint64_t *z_real_in = (uint64_t*)&data_in[2].real; uint64_t *z_imag_in = (uint64_t*)&data_in[2].imag; uint64_t *w_real_in = (uint64_t*)&data_in[3].real; uint64_t *w_imag_in = (uint64_t*)&data_in[3].imag; uint64_t *c_real_in = (uint64_t*)&coefC[0].real; uint64_t *c_imag_in = (uint64_t*)&coefC[0].imag; uint64_t *d_real_in = (uint64_t*)&coefD[0].real; uint64_t *d_imag_in = (uint64_t*)&coefD[0].imag; uint64_t *e_real_in = (uint64_t*)&coefE[0].real; uint64_t *e_imag_in = (uint64_t*)&coefE[0].imag; uint64_t *out_0_real = (uint64_t*)&data_out[0*data_count/4].real; uint64_t *out_0_imag = (uint64_t*)&data_out[0*data_count/4].imag; uint64_t *out_1_real = (uint64_t*)&data_out[1*data_count/4].real; uint64_t *out_1_imag = (uint64_t*)&data_out[1*data_count/4].imag; uint64_t *out_2_real = (uint64_t*)&data_out[2*data_count/4].real; uint64_t *out_2_imag = (uint64_t*)&data_out[2*data_count/4].imag; uint64_t *out_3_real = (uint64_t*)&data_out[3*data_count/4].real; uint64_t *out_3_imag = (uint64_t*)&data_out[3*data_count/4].imag; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { uint64_t x_real = x_real_in[8*i]; uint64_t x_imag = x_imag_in[8*i]; uint64_t y_real = y_real_in[8*i]; uint64_t y_imag = y_imag_in[8*i]; uint64_t z_real = z_real_in[8*i]; uint64_t z_imag = z_imag_in[8*i]; uint64_t w_real = w_real_in[8*i]; uint64_t w_imag = w_imag_in[8*i]; uint64_t c_real = c_real_in[2*i]; uint64_t c_imag = c_imag_in[2*i]; uint64_t d_real = d_real_in[2*i]; uint64_t d_imag = d_imag_in[2*i]; uint64_t e_real = e_real_in[2*i]; uint64_t e_imag = e_imag_in[2*i]; uint64_t cy_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(c_real, y_real), __builtin_e2k_pfmuls(c_imag, y_imag)); uint64_t cy_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(c_real, y_imag), __builtin_e2k_pfmuls(c_imag, y_real)); uint64_t dz_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(d_real, z_real), __builtin_e2k_pfmuls(d_imag, z_imag)); uint64_t dz_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(d_real, z_imag), __builtin_e2k_pfmuls(d_imag, z_real)); uint64_t ew_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(e_real, w_real), __builtin_e2k_pfmuls(e_imag, w_imag)); uint64_t ew_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(e_real, w_imag), __builtin_e2k_pfmuls(e_imag, w_real)); uint64_t add02_real = __builtin_e2k_pfadds( x_real, dz_real); uint64_t add02_imag = __builtin_e2k_pfadds( x_imag, dz_imag); uint64_t sub02_real = __builtin_e2k_pfsubs( x_real, dz_real); uint64_t sub02_imag = __builtin_e2k_pfsubs( x_imag, dz_imag); uint64_t add13_real = __builtin_e2k_pfadds(cy_real, ew_real); uint64_t add13_imag = __builtin_e2k_pfadds(cy_imag, ew_imag); uint64_t sub13_real = __builtin_e2k_pfsubs(cy_real, ew_real); // uint64_t sub13_imag = __builtin_e2k_pfsubs(cy_imag, ew_imag); // uint64_t sub13i_real = -sub13_imag; uint64_t sub13i_real = __builtin_e2k_pfsubs(ew_imag, cy_imag); uint64_t sub13i_imag = sub13_real; out_0_real[2*i] = __builtin_e2k_pfadds(add02_real, add13_real); out_0_imag[2*i] = __builtin_e2k_pfadds(add02_imag, add13_imag); out_1_real[2*i] = __builtin_e2k_pfsubs(sub02_real, sub13i_real); out_1_imag[2*i] = __builtin_e2k_pfsubs(sub02_imag, sub13i_imag); out_2_real[2*i] = __builtin_e2k_pfsubs(add02_real, add13_real); out_2_imag[2*i] = __builtin_e2k_pfsubs(add02_imag, add13_imag); out_3_real[2*i] = __builtin_e2k_pfadds(sub02_real, sub13i_real); out_3_imag[2*i] = __builtin_e2k_pfadds(sub02_imag, sub13i_imag); } }
Основной цикл на ассемблере
.L3659: { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=4, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=3, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=16, disp=0 } .L3151: { loop_mode pfmul_adds,0,sm %b[48], %b[70], %b[42], %b[0] pfsub_rsubs,1,sm %b[37], %b[73], %b[59], %b[4] pfsub_adds,2,sm %b[37], %b[73], %b[59], %b[1] pfmuls,4,sm %b[63], %b[62], %b[40] pfsubs,5,sm %b[78], %b[74], %b[57] } { loop_mode pfmul_rsubs,0,sm %b[48], %b[64], %b[83], %b[70] pfadd_adds,1,sm %b[25], %b[82], %b[90], %b[77] pfadd_rsubs,2,sm %b[25], %b[82], %b[90], %b[73] pfmuls,4,sm %b[63], %b[68], %b[81] pfadds,5,sm %b[78], %b[74], %b[88] movad,0 area=2, ind=0, am=0, be=0, %b[42] movad,1 area=2, ind=8, am=1, be=0, %b[59] movad,2 area=1, ind=0, am=0, be=0, %b[37] movad,3 area=0, ind=16, am=0, be=0, %b[58] } { loop_mode pfmul_rsubs,0,sm %b[69], %b[13], %b[84], %b[74] pfmul_rsubs,1,sm %b[45], %b[34], %b[85], %b[78] staad,2 %b[49], %aad3[ %aasti7 ] pfmuls,3,sm %b[14], %b[53], %b[82] pfmuls,4,sm %b[54], %b[20], %b[83] staad,5 %b[26], %aad1[ %aasti5 ] movad,0 area=1, ind=0, am=0, be=0, %b[63] movad,1 area=0, ind=8, am=0, be=0, %b[25] movad,2 area=1, ind=8, am=1, be=0, %b[48] movad,3 area=0, ind=24, am=0, be=0, %b[64] } { loop_mode pfmul_adds,0,sm %b[67], %b[53], %b[86], %b[34] pfmul_adds,1,sm %b[45], %b[22], %b[87], %b[69] staad,2 %b[10], %aad4[ %aasti8 + _f32s,_lts0 0x8 ] pfmuls,3,sm %b[12], %b[9], %b[84] pfmuls,4,sm %b[54], %b[32], %b[85] staad,5 %b[7], %aad2[ %aasti6 + _f32s,_lts0 0x8 ] movad,0 area=0, ind=0, am=0, be=0, %b[13] movad,1 area=0, ind=24, am=0, be=0, %b[49] movad,2 area=0, ind=0, am=0, be=0, %b[26] movad,3 area=0, ind=8, am=1, be=0, %b[14] } { loop_mode pfsub_rsubs,0,sm %b[23], %b[80], %b[55], %b[45] pfsub_adds,1,sm %b[23], %b[80], %b[55], %b[22] staad,2 %b[6], %aad3[ %aasti7 + _f32s,_lts0 0x8 ] incr,2 %aaincr3 pfsubs,4,sm %b[2], %b[38], %b[53] staad,5 %b[3], %aad1[ %aasti5 + _f32s,_lts0 0x8 ] incr,5 %aaincr3 movad,0 area=1, ind=8, am=1, be=0, %b[10] movad,1 area=0, ind=16, am=1, be=0, %b[7] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfadd_adds,0,sm %b[35], %b[71], %b[56], %b[6] pfadd_rsubs,1,sm %b[35], %b[71], %b[56], %b[3] staad,2 %b[79], %aad4[ %aasti8 ] incr,2 %aaincr3 pfadds,4,sm %b[38], %b[2], %b[54] staad,5 %b[75], %aad2[ %aasti6 ] incr,5 %aaincr3 }
Код на ассемблере получился такой же, как в эталонном варианте, то есть компилятор самостоятельно векторизовал эталонный вариант в 2 раза.
Теоретическая скорость: 8 комплексных чисел за 6 тактов (8/6) = 10.67 Байт/такт
Двойная теоретическая скорость: 21.33 Байт/такт
Замеры скорости
3. stage_radix4_vector2_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью директивы unroll.
Код на Си
void stage_radix4_vector2_unroll2(int data_count, myComplex2 *data_in, myComplex2 *data_out, myComplex2 *coefC, myComplex2 *coefD, myComplex2 *coefE) { uint64_t *x_real_in = (uint64_t*)&data_in[0].real; uint64_t *x_imag_in = (uint64_t*)&data_in[0].imag; uint64_t *y_real_in = (uint64_t*)&data_in[1].real; uint64_t *y_imag_in = (uint64_t*)&data_in[1].imag; uint64_t *z_real_in = (uint64_t*)&data_in[2].real; uint64_t *z_imag_in = (uint64_t*)&data_in[2].imag; uint64_t *w_real_in = (uint64_t*)&data_in[3].real; uint64_t *w_imag_in = (uint64_t*)&data_in[3].imag; uint64_t *c_real_in = (uint64_t*)&coefC[0].real; uint64_t *c_imag_in = (uint64_t*)&coefC[0].imag; uint64_t *d_real_in = (uint64_t*)&coefD[0].real; uint64_t *d_imag_in = (uint64_t*)&coefD[0].imag; uint64_t *e_real_in = (uint64_t*)&coefE[0].real; uint64_t *e_imag_in = (uint64_t*)&coefE[0].imag; uint64_t *out_0_real = (uint64_t*)&data_out[0*data_count/4].real; uint64_t *out_0_imag = (uint64_t*)&data_out[0*data_count/4].imag; uint64_t *out_1_real = (uint64_t*)&data_out[1*data_count/4].real; uint64_t *out_1_imag = (uint64_t*)&data_out[1*data_count/4].imag; uint64_t *out_2_real = (uint64_t*)&data_out[2*data_count/4].real; uint64_t *out_2_imag = (uint64_t*)&data_out[2*data_count/4].imag; uint64_t *out_3_real = (uint64_t*)&data_out[3*data_count/4].real; uint64_t *out_3_imag = (uint64_t*)&data_out[3*data_count/4].imag; #pragma ivdep #pragma unroll(2) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { uint64_t x_real = x_real_in[8*i]; uint64_t x_imag = x_imag_in[8*i]; uint64_t y_real = y_real_in[8*i]; uint64_t y_imag = y_imag_in[8*i]; uint64_t z_real = z_real_in[8*i]; uint64_t z_imag = z_imag_in[8*i]; uint64_t w_real = w_real_in[8*i]; uint64_t w_imag = w_imag_in[8*i]; uint64_t c_real = c_real_in[2*i]; uint64_t c_imag = c_imag_in[2*i]; uint64_t d_real = d_real_in[2*i]; uint64_t d_imag = d_imag_in[2*i]; uint64_t e_real = e_real_in[2*i]; uint64_t e_imag = e_imag_in[2*i]; uint64_t cy_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(c_real, y_real), __builtin_e2k_pfmuls(c_imag, y_imag)); uint64_t cy_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(c_real, y_imag), __builtin_e2k_pfmuls(c_imag, y_real)); uint64_t dz_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(d_real, z_real), __builtin_e2k_pfmuls(d_imag, z_imag)); uint64_t dz_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(d_real, z_imag), __builtin_e2k_pfmuls(d_imag, z_real)); uint64_t ew_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(e_real, w_real), __builtin_e2k_pfmuls(e_imag, w_imag)); uint64_t ew_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(e_real, w_imag), __builtin_e2k_pfmuls(e_imag, w_real)); uint64_t add02_real = __builtin_e2k_pfadds( x_real, dz_real); uint64_t add02_imag = __builtin_e2k_pfadds( x_imag, dz_imag); uint64_t sub02_real = __builtin_e2k_pfsubs( x_real, dz_real); uint64_t sub02_imag = __builtin_e2k_pfsubs( x_imag, dz_imag); uint64_t add13_real = __builtin_e2k_pfadds(cy_real, ew_real); uint64_t add13_imag = __builtin_e2k_pfadds(cy_imag, ew_imag); uint64_t sub13_real = __builtin_e2k_pfsubs(cy_real, ew_real); // uint64_t sub13_imag = __builtin_e2k_pfsubs(cy_imag, ew_imag); // uint64_t sub13i_real = -sub13_imag; uint64_t sub13i_real = __builtin_e2k_pfsubs(ew_imag, cy_imag); uint64_t sub13i_imag = sub13_real; out_0_real[2*i] = __builtin_e2k_pfadds(add02_real, add13_real); out_0_imag[2*i] = __builtin_e2k_pfadds(add02_imag, add13_imag); out_1_real[2*i] = __builtin_e2k_pfsubs(sub02_real, sub13i_real); out_1_imag[2*i] = __builtin_e2k_pfsubs(sub02_imag, sub13i_imag); out_2_real[2*i] = __builtin_e2k_pfsubs(add02_real, add13_real); out_2_imag[2*i] = __builtin_e2k_pfsubs(add02_imag, add13_imag); out_3_real[2*i] = __builtin_e2k_pfadds(sub02_real, sub13i_real); out_3_imag[2*i] = __builtin_e2k_pfadds(sub02_imag, sub13i_imag); } }
Основной цикл на ассемблере
.L4871: { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=3, abs=0, disp=16 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=64 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=32 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=4, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=16, disp=96 } { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=3, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=3, asz=3, abs=24, disp=0 } .L4115: { loop_mode pfadd_adds,0,sm %b[56], %b[48], %b[97], %b[91] pfadd_rsubs,1,sm %b[56], %b[48], %b[97], %b[88] pfsubs,2,sm %b[83], %b[73], %b[98] pfmul_adds,3,sm %b[35], %b[49], %b[91], %b[0] pfmuls,4,sm %b[88], %b[4], %b[1] pfmuls,5,sm %b[44], %b[72], %b[95] } { loop_mode pfmul_rsubs,0,sm %b[35], %b[29], %g18, %b[11] pfmul_adds,1,sm %b[11], %b[63], %b[96], %b[72] pfadds,2,sm %b[73], %b[83], %b[83] pfmul_adds,3,sm %b[30], %b[72], %b[94], %b[29] pfmuls,4,sm %b[36], %b[47], %g18 pfadds,5,sm %g16, %g17, %b[73] } { loop_mode pfmul_rsubs,0,sm %b[68], %b[14], %b[24], %g17 pfmul_rsubs,1,sm %b[45], %b[34], %b[3], %g16 pfsubs,2,sm %g16, %g17, %b[96] pfmul_adds,3,sm %b[18], %b[60], %b[92], %b[3] pfmuls,4,sm %b[10], %b[61], %b[94] pfsubs,5,sm %b[5], %b[31], %b[92] } { loop_mode pfmul_adds,0,sm %b[45], %b[6], %b[71], %b[71] pfmul_adds,1,sm %b[68], %b[41], %b[85], %b[81] staad,2 %b[86], %aad3[ %aasti7 + _f32s,_lts0 0x10 ] pfmul_rsubs,3,sm %b[18], %b[67], %b[93], %b[24] pfadds,4,sm %b[31], %b[5], %b[85] staad,5 %b[81], %aad1[ %aasti5 + _f32s,_lts0 0x10 ] movad,0 area=3, ind=0, am=0, be=0, %b[14] movad,1 area=3, ind=8, am=1, be=0, %b[18] movad,2 area=3, ind=0, am=0, be=0, %b[5] movad,3 area=3, ind=8, am=0, be=0, %b[6] } { loop_mode pfsub_rsubs,0,sm %b[19], %b[13], %b[98], %b[84] pfsub_adds,1,sm %b[19], %b[13], %b[98], %b[79] staad,2 %b[84], %aad4[ %aasti8 + _f32s,_lts0 0x18 ] pfmul_rsubs,3,sm %b[30], %b[64], %b[95], %b[35] pfsubs,4,sm %b[37], %b[26], %b[93] staad,5 %b[79], %aad2[ %aasti6 + _f32s,_lts0 0x18 ] movad,0 area=2, ind=24, am=0, be=0, %b[86] movad,1 area=1, ind=16, am=0, be=0, %b[30] movad,2 area=3, ind=16, am=0, be=0, %b[31] movad,3 area=3, ind=24, am=1, be=0, %b[34] } { loop_mode pfadd_adds,0,sm %b[25], %b[2], %b[83], %b[82] pfadd_rsubs,1,sm %b[25], %b[2], %b[83], %b[77] staad,2 %b[82], %aad3[ %aasti7 + _f32s,_lts0 0x18 ] pfmuls,3,sm %g19, %b[12], %b[83] pfadds,4,sm %b[37], %b[26], %b[95] staad,5 %b[77], %aad1[ %aasti5 + _f32s,_lts0 0x18 ] movad,0 area=2, ind=16, am=0, be=0, %b[41] movad,1 area=2, ind=0, am=0, be=0, %b[26] movad,2 area=2, ind=24, am=0, be=0, %b[37] movad,3 area=0, ind=8, am=0, be=0, %g19 } { loop_mode pfsub_rsubs,0,sm %b[25], %b[2], %b[96], %b[80] pfsub_adds,1,sm %b[25], %b[2], %b[96], %b[75] staad,2 %b[80], %aad4[ %aasti8 + _f32s,_lts0 0x10 ] pfmul_rsubs,3,sm %b[9], %b[40], %b[94], %b[44] pfmuls,4,sm %b[10], %b[40], %b[94] staad,5 %b[75], %aad2[ %aasti6 + _f32s,_lts0 0x10 ] movad,0 area=2, ind=8, am=1, be=0, %b[40] movad,1 area=1, ind=24, am=0, be=0, %b[2] movad,2 area=2, ind=0, am=0, be=0, %b[25] movad,3 area=2, ind=16, am=0, be=0, %b[10] } { loop_mode pfadd_adds,0,sm %b[19], %b[13], %b[73], %b[78] pfadd_rsubs,1,sm %b[19], %b[13], %b[73], %b[73] staad,2 %b[78], %aad3[ %aasti7 ] pfmuls,4,sm %b[36], %b[27], %b[89] staad,5 %b[89], %aad1[ %aasti5 ] movad,0 area=1, ind=8, am=1, be=0, %b[19] movad,1 area=1, ind=0, am=0, be=0, %b[13] movad,2 area=2, ind=8, am=1, be=0, %b[45] movad,3 area=1, ind=0, am=0, be=0, %b[36] } { loop_mode pfsub_rsubs,0,sm %b[54], %b[46], %b[92], %b[76] pfsub_adds,1,sm %b[54], %b[46], %b[92], %b[87] staad,2 %b[76], %aad4[ %aasti8 + _f32s,_lts0 0x8 ] pfmuls,4,sm %b[42], %b[62], %b[92] staad,5 %b[87], %aad2[ %aasti6 + _f32s,_lts0 0x8 ] movad,0 area=0, ind=8, am=0, be=0, %b[49] movad,1 area=0, ind=0, am=0, be=0, %b[48] movad,2 area=1, ind=8, am=0, be=0, %b[57] movad,3 area=1, ind=24, am=0, be=0, %b[56] } { loop_mode pfadd_adds,0,sm %b[55], %b[72], %b[85], %b[74] pfadd_rsubs,1,sm %b[55], %b[72], %b[85], %b[85] staad,2 %b[90], %aad3[ %aasti7 + _f32s,_lts0 0x8 ] incr,2 %aaincr3 pfmuls,3,sm %b[86], %b[30], %b[67] pfmuls,4,sm %b[20], %b[65], %b[90] staad,5 %b[74], %aad1[ %aasti5 + _f32s,_lts0 0x8 ] incr,5 %aaincr3 movad,0 area=0, ind=24, am=1, be=0, %b[68] movad,1 area=0, ind=16, am=0, be=0, %b[60] movad,2 area=1, ind=16, am=1, be=0, %b[63] movad,3 area=0, ind=0, am=1, be=0, %b[64] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfsub_rsubs,0,sm %b[55], %b[72], %b[93], %b[88] pfsub_adds,1,sm %b[55], %b[72], %b[93], %b[72] staad,2 %b[91], %aad4[ %aasti8 ] incr,2 %aaincr3 pfmuls,3,sm %g19, %b[37], %b[20] pfmuls,4,sm %b[20], %b[58], %b[91] staad,5 %b[88], %aad2[ %aasti6 ] incr,5 %aaincr3 }
Теоретическая скорость: 16 комплексных чисел за 11 тактов (16/11) = 11.64 Байт/такт
Двойная теоретическая скорость: 23.27 Байт/такт
Замеры скорости
4. stage_radix4_vector2_unroll3
Здесь происходит раскрутка цикла в 3 раза с помощью директивы unroll.
Код на Си
void stage_radix4_vector2_unroll3(int data_count, myComplex2 *data_in, myComplex2 *data_out, myComplex2 *coefC, myComplex2 *coefD, myComplex2 *coefE) { uint64_t *x_real_in = (uint64_t*)&data_in[0].real; uint64_t *x_imag_in = (uint64_t*)&data_in[0].imag; uint64_t *y_real_in = (uint64_t*)&data_in[1].real; uint64_t *y_imag_in = (uint64_t*)&data_in[1].imag; uint64_t *z_real_in = (uint64_t*)&data_in[2].real; uint64_t *z_imag_in = (uint64_t*)&data_in[2].imag; uint64_t *w_real_in = (uint64_t*)&data_in[3].real; uint64_t *w_imag_in = (uint64_t*)&data_in[3].imag; uint64_t *c_real_in = (uint64_t*)&coefC[0].real; uint64_t *c_imag_in = (uint64_t*)&coefC[0].imag; uint64_t *d_real_in = (uint64_t*)&coefD[0].real; uint64_t *d_imag_in = (uint64_t*)&coefD[0].imag; uint64_t *e_real_in = (uint64_t*)&coefE[0].real; uint64_t *e_imag_in = (uint64_t*)&coefE[0].imag; uint64_t *out_0_real = (uint64_t*)&data_out[0*data_count/4].real; uint64_t *out_0_imag = (uint64_t*)&data_out[0*data_count/4].imag; uint64_t *out_1_real = (uint64_t*)&data_out[1*data_count/4].real; uint64_t *out_1_imag = (uint64_t*)&data_out[1*data_count/4].imag; uint64_t *out_2_real = (uint64_t*)&data_out[2*data_count/4].real; uint64_t *out_2_imag = (uint64_t*)&data_out[2*data_count/4].imag; uint64_t *out_3_real = (uint64_t*)&data_out[3*data_count/4].real; uint64_t *out_3_imag = (uint64_t*)&data_out[3*data_count/4].imag; #pragma ivdep #pragma unroll(3) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { uint64_t x_real = x_real_in[8*i]; uint64_t x_imag = x_imag_in[8*i]; uint64_t y_real = y_real_in[8*i]; uint64_t y_imag = y_imag_in[8*i]; uint64_t z_real = z_real_in[8*i]; uint64_t z_imag = z_imag_in[8*i]; uint64_t w_real = w_real_in[8*i]; uint64_t w_imag = w_imag_in[8*i]; uint64_t c_real = c_real_in[2*i]; uint64_t c_imag = c_imag_in[2*i]; uint64_t d_real = d_real_in[2*i]; uint64_t d_imag = d_imag_in[2*i]; uint64_t e_real = e_real_in[2*i]; uint64_t e_imag = e_imag_in[2*i]; uint64_t cy_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(c_real, y_real), __builtin_e2k_pfmuls(c_imag, y_imag)); uint64_t cy_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(c_real, y_imag), __builtin_e2k_pfmuls(c_imag, y_real)); uint64_t dz_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(d_real, z_real), __builtin_e2k_pfmuls(d_imag, z_imag)); uint64_t dz_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(d_real, z_imag), __builtin_e2k_pfmuls(d_imag, z_real)); uint64_t ew_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(e_real, w_real), __builtin_e2k_pfmuls(e_imag, w_imag)); uint64_t ew_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(e_real, w_imag), __builtin_e2k_pfmuls(e_imag, w_real)); uint64_t add02_real = __builtin_e2k_pfadds( x_real, dz_real); uint64_t add02_imag = __builtin_e2k_pfadds( x_imag, dz_imag); uint64_t sub02_real = __builtin_e2k_pfsubs( x_real, dz_real); uint64_t sub02_imag = __builtin_e2k_pfsubs( x_imag, dz_imag); uint64_t add13_real = __builtin_e2k_pfadds(cy_real, ew_real); uint64_t add13_imag = __builtin_e2k_pfadds(cy_imag, ew_imag); uint64_t sub13_real = __builtin_e2k_pfsubs(cy_real, ew_real); // uint64_t sub13_imag = __builtin_e2k_pfsubs(cy_imag, ew_imag); // uint64_t sub13i_real = -sub13_imag; uint64_t sub13i_real = __builtin_e2k_pfsubs(ew_imag, cy_imag); uint64_t sub13i_imag = sub13_real; out_0_real[2*i] = __builtin_e2k_pfadds(add02_real, add13_real); out_0_imag[2*i] = __builtin_e2k_pfadds(add02_imag, add13_imag); out_1_real[2*i] = __builtin_e2k_pfsubs(sub02_real, sub13i_real); out_1_imag[2*i] = __builtin_e2k_pfsubs(sub02_imag, sub13i_imag); out_2_real[2*i] = __builtin_e2k_pfsubs(add02_real, add13_real); out_2_imag[2*i] = __builtin_e2k_pfsubs(add02_imag, add13_imag); out_3_real[2*i] = __builtin_e2k_pfadds(sub02_real, sub13i_real); out_3_imag[2*i] = __builtin_e2k_pfadds(sub02_imag, sub13i_imag); } }
Основной цикл на ассемблере
.L6432: { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=64 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=96 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=128 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=160 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=4, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=3, asz=2, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=2, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=4, asz=3, abs=16, disp=32 } { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=3, asz=3, abs=24, disp=32 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=3, abs=24, disp=32 } .L4979: { loop_mode pfsub_rsubs,0,sm %b[40], %g19, %g20, %g21 pfadd_adds,1,sm %b[77], %g16, %g17, %g18 pfsubs,2,sm %g22, %g23, %g24 pfmuls,3,sm %b[6], %b[32], %g27 pfmuls,4,sm %b[56], %b[65], %g26 pfmuls,5,sm %b[72], %b[48], %g25 } { loop_mode pfadd_adds,0,sm %b[68], %g28, %g29, %g31 pfadd_rsubs,1,sm %b[68], %g28, %g29, %g30 pfadds,2,sm %g23, %g22, %r0 pfmuls,3,sm %b[55], %b[61], %r3 pfmuls,4,sm %b[56], %b[102], %r2 pfmuls,5,sm %b[52], %b[99], %r1 } { loop_mode pfsub_rsubs,0,sm %b[68], %g28, %r4, %r7 pfsub_adds,1,sm %b[68], %g28, %r4, %r5 pfmuls,2,sm %b[20], %b[78], %r8 pfmuls,3,sm %b[18], %b[90], %r11 pfmuls,4,sm %b[51], %b[95], %r10 pfmuls,5,sm %b[41], %b[83], %r9 } { loop_mode pfadd_adds,0,sm %b[40], %g19, %r12, %r14 pfadd_rsubs,1,sm %b[40], %g19, %r12, %r13 pfmuls,2,sm %b[35], %b[112], %r18 pfmuls,3,sm %b[55], %b[116], %r20 pfmuls,4,sm %b[35], %b[113], %r19 pfadds,5,sm %r15, %r16, %r17 } { loop_mode pfsub_rsubs,0,sm %b[77], %g16, %g24, %g24 pfsub_adds,1,sm %b[77], %g16, %g24, %r22 staad,2 %b[119], %aad3[ %aasti7 + _f32s,_lts0 0x10 ] pfmuls,3,sm %b[70], %b[104], %b[0] pfsubs,4,sm %r23, %r24, %r25 staad,5 %r21, %aad1[ %aasti5 + _f32s,_lts0 0x10 ] } { loop_mode pfadd_adds,0,sm %b[14], %b[103], %r0, %b[118] pfadd_rsubs,1,sm %b[14], %b[103], %r0, %r27 staad,2 %b[118], %aad4[ %aasti8 + _f32s,_lts0 0x18 ] pfmul_adds,3,sm %b[23], %b[107], %r1, %b[20] pfadds,4,sm %r24, %r23, %r0 staad,5 %r26, %aad2[ %aasti6 + _f32s,_lts0 0x18 ] movad,0 area=5, ind=0, am=0, be=0, %b[1] movad,1 area=5, ind=8, am=1, be=0, %b[14] } { loop_mode pfmul_adds,0,sm %b[69], %b[106], %g25, %r24 pfmul_rsubs,1,sm %b[7], %b[92], %r8, %g16 staad,2 %r29, %aad3[ %aasti7 + _f32s,_lts0 0x18 ] pfmul_adds,3,sm %b[45], %b[65], %r2, %r23 pfsubs,4,sm %r15, %r16, %r1 staad,5 %r28, %aad1[ %aasti5 + _f32s,_lts0 0x18 ] movad,0 area=4, ind=0, am=0, be=0, %b[40] movad,1 area=4, ind=16, am=0, be=0, %b[41] movad,2 area=5, ind=0, am=0, be=0, %b[7] movad,3 area=5, ind=8, am=1, be=0, %b[35] } { loop_mode pfmul_rsubs,0,sm %b[45], %b[102], %g26, %r16 pfmul_rsubs,1,sm %b[19], %b[113], %r18, %g19 staad,2 %r31, %aad3[ %aasti7 + _f32s,_lts0 0x28 ] pfmul_rsubs,3,sm %b[23], %b[99], %r34, %b[55] pfsubs,4,sm %r32, %r33, %g25 staad,5 %r30, %aad1[ %aasti5 + _f32s,_lts0 0x28 ] movad,0 area=4, ind=8, am=1, be=0, %b[51] movad,1 area=4, ind=24, am=0, be=0, %b[52] movad,2 area=4, ind=0, am=0, be=0, %b[23] movad,3 area=4, ind=8, am=1, be=0, %b[45] } { loop_mode pfmul_adds,0,sm %b[29], %b[89], %r10, %g23 pfmul_adds,1,sm %b[13], %b[86], %r9, %g22 staad,2 %r36, %aad2[ %aasti6 + _f32s,_lts1 0x10 ] pfmul_rsubs,3,sm %b[11], %b[81], %r37, %r33 pfadds,4,sm %r32, %r33, %g17 staad,5 %r35, %aad2[ %aasti6 + _f32s,_lts0 0x20 ] movad,0 area=3, ind=0, am=0, be=0, %b[56] movad,1 area=3, ind=16, am=0, be=0, %b[65] movad,2 area=3, ind=0, am=0, be=0, %b[13] movad,3 area=3, ind=8, am=0, be=0, %b[29] } { loop_mode pfmul_rsubs,0,sm %b[69], %b[48], %b[2], %r15 pfmul_adds,1,sm %b[19], %b[112], %r19, %g28 staad,2 %r39, %aad1[ %aasti5 ] pfmul_rsubs,3,sm %b[27], %b[93], %r41, %r32 pfsubs,4,sm %r40, %b[80], %g20 staad,5 %r38, %aad4[ %aasti8 + _f32s,_lts0 0x10 ] movad,0 area=3, ind=8, am=1, be=0, %b[2] movad,1 area=3, ind=24, am=0, be=0, %b[68] movad,2 area=3, ind=16, am=0, be=0, %b[19] movad,3 area=3, ind=24, am=1, be=0, %b[48] } { loop_mode pfsub_rsubs,0,sm %b[28], %b[57], %r25, %b[117] pfsub_adds,1,sm %b[28], %b[57], %r25, %r21 staad,2 %g21, %aad3[ %aasti7 ] pfmul_adds,3,sm %b[60], %b[117], %g27, %b[78] pfmuls,4,sm %b[6], %b[117], %g18 staad,5 %g18, %aad4[ %aasti8 + _f32s,_lts0 0x20 ] movad,0 area=2, ind=0, am=0, be=0, %b[69] movad,1 area=2, ind=8, am=0, be=0, %b[6] movad,2 area=2, ind=8, am=0, be=0, %b[72] movad,3 area=2, ind=16, am=0, be=0, %b[77] } { loop_mode pfadd_adds,0,sm %b[98], %b[22], %r0, %b[116] pfadd_rsubs,1,sm %b[98], %b[22], %r0, %r26 staad,2 %g31, %aad4[ %aasti8 + _f32s,_lts0 0x8 ] pfmul_adds,3,sm %b[44], %b[116], %r3, %r40 pfadds,4,sm %b[80], %r40, %g29 staad,5 %g30, %aad2[ %aasti6 + _f32s,_lts0 0x8 ] movad,0 area=2, ind=16, am=0, be=0, %b[89] movad,1 area=2, ind=24, am=1, be=0, %b[83] movad,2 area=2, ind=0, am=1, be=0, %b[86] movad,3 area=2, ind=24, am=0, be=0, %b[80] } { loop_mode pfsub_rsubs,0,sm %b[98], %b[22], %r1, %r29 pfsub_adds,1,sm %b[98], %b[22], %r1, %r28 staad,2 %r7, %aad3[ %aasti7 + _f32s,_lts0 0x8 ] pfmul_adds,3,sm %b[5], %b[76], %r11, %b[99] pfsubs,4,sm %r42, %r43, %r4 staad,5 %r5, %aad1[ %aasti5 + _f32s,_lts0 0x8 ] movad,0 area=1, ind=8, am=0, be=0, %b[92] movad,1 area=1, ind=0, am=0, be=0, %b[22] movad,2 area=1, ind=16, am=0, be=0, %b[98] movad,3 area=1, ind=0, am=0, be=0, %b[95] } { loop_mode pfsub_rsubs,0,sm %b[12], %b[101], %g25, %r31 pfsub_adds,1,sm %b[12], %b[101], %g25, %r30 staad,2 %r14, %aad4[ %aasti8 ] pfmul_rsubs,3,sm %b[44], %b[61], %r20, %r43 pfadds,4,sm %r42, %r43, %r12 staad,5 %r13, %aad2[ %aasti6 ] movad,0 area=1, ind=16, am=0, be=0, %b[44] movad,1 area=1, ind=24, am=1, be=0, %b[102] movad,2 area=1, ind=24, am=0, be=0, %b[61] movad,3 area=1, ind=8, am=1, be=0, %b[103] } { loop_mode pfadd_rsubs,0,sm %b[28], %b[57], %r17, %r36 pfadd_rsubs,1,sm %b[75], %g16, %g17, %r35 staad,2 %g24, %aad3[ %aasti7 + _f32s,_lts0 0x20 ] incr,2 %aaincr3 pfmul_rsubs,3,sm %b[60], %b[32], %g18, %r42 pfmuls,4,sm %b[37], %b[82], %r37 staad,5 %r22, %aad1[ %aasti5 + _f32s,_lts0 0x20 ] incr,5 %aaincr3 movad,0 area=0, ind=8, am=0, be=0, %b[60] movad,1 area=0, ind=0, am=0, be=0, %b[32] movad,2 area=0, ind=0, am=0, be=0, %b[107] movad,3 area=0, ind=8, am=0, be=0, %b[106] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfsub_adds,0,sm %b[38], %g19, %g20, %r39 pfadd_adds,1,sm %b[28], %b[57], %r17, %r38 staad,2 %b[118], %aad4[ %aasti8 + _f32s,_lts0 0x28 ] incr,2 %aaincr3 pfmuls,3,sm %b[47], %b[85], %r41 pfmuls,4,sm %b[50], %b[105], %r34 staad,5 %r27, %aad2[ %aasti6 + _f32s,_lts0 0x28 ] incr,5 %aaincr3 movad,0 area=0, ind=24, am=1, be=0, %b[113] movad,1 area=0, ind=16, am=0, be=0, %b[28] movad,2 area=0, ind=24, am=1, be=0, %b[112] movad,3 area=0, ind=16, am=0, be=0, %b[57] }
Теоретическая скорость: 24 комплексных числа за 16 тактов (24/16) = 12 Байт/такт
Двойная теоретическая скорость: 24 Байт/такт
Замеры скорости
Итоги по stage_radix4_vector2
stage_radix4_vector4
Один проход по stage_radix4_vector4 совершает ту же работу, что 2 прохода по stage_radix2. Поэтому скорость stage_radix4_vector4 будем умножать на 2 для удобства сравнения с stage_radix2 (этот факт подписан на оси графика и в выводе консоли).
1. stage_radix4_vector4_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void stage_radix4_vector4_etalon(int data_count, myComplex4 *data_in, myComplex4 *data_out, myComplex4 *coefC, myComplex4 *coefD, myComplex4 *coefE) { myComplex4 *x_in = &data_in[0]; myComplex4 *y_in = &data_in[1]; myComplex4 *z_in = &data_in[2]; myComplex4 *w_in = &data_in[3]; myComplex4 *c_in = coefC; myComplex4 *d_in = coefD; myComplex4 *e_in = coefE; myComplex4 *out_0 = &data_out[0*data_count/4]; myComplex4 *out_1 = &data_out[1*data_count/4]; myComplex4 *out_2 = &data_out[2*data_count/4]; myComplex4 *out_3 = &data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) // #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { myComplex4 x = x_in[4*i]; myComplex4 y = y_in[4*i]; myComplex4 z = z_in[4*i]; myComplex4 w = w_in[4*i]; myComplex4 c = c_in[i]; myComplex4 d = d_in[i]; myComplex4 e = e_in[i]; myComplex4 cy = complex4_mul(c, y); myComplex4 dz = complex4_mul(d, z); myComplex4 ew = complex4_mul(e, w); myComplex4 add02 = complex4_add( x, dz); myComplex4 sub02 = complex4_sub( x, dz); myComplex4 add13 = complex4_add(cy, ew); myComplex4 sub13 = complex4_sub(cy, ew); myComplex4 sub13i = complex4_mul_i(sub13); out_0[i] = complex4_add(add02, add13); out_1[i] = complex4_sub(sub02, sub13i); out_2[i] = complex4_sub(add02, add13); out_3[i] = complex4_add(sub02, sub13i); } }
Основной цикл на ассемблере
.L4263: { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=3, abs=0, disp=16 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=64 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=32 } { fapb ct=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=4, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=3, abs=16, disp=96 } { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=3, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=0, d=0, incr=2, ind=3, asz=3, abs=24, disp=0 } .L3418: { loop_mode pfsub_rsubs,0,sm %b[68], %b[38], %b[93], %b[98] pfsub_adds,1,sm %b[68], %b[38], %b[93], %b[93] pfsubs,2,sm %b[87], %b[77], %b[102] pfmul_adds,3,sm %b[19], %b[59], %b[98], %b[0] pfmuls,4,sm %b[47], %b[53], %b[1] pfmuls,5,sm %b[26], %b[11], %b[100] } { loop_mode pfmul_adds,0,sm %b[22], %b[32], %b[91], %b[26] pfmul_adds,1,sm %b[18], %b[55], %b[97], %b[32] pfadds,2,sm %b[87], %b[77], %b[91] pfmul_adds,3,sm %b[14], %b[45], %b[99], %b[38] pfmuls,4,sm %b[26], %b[15], %b[87] pfadds,5,sm %g16, %g17, %b[77] } { loop_mode pfmul_rsubs,0,sm %b[22], %b[72], %b[6], %g17 pfmul_rsubs,1,sm %b[18], %b[76], %b[3], %g16 pfsubs,2,sm %g16, %g17, %b[99] pfmul_adds,3,sm %b[10], %b[23], %b[101], %b[23] pfadds,4,sm %b[34], %b[28], %b[76] pfsubs,5,sm %b[25], %b[40], %b[97] movad,0 area=3, ind=8, am=1, be=0, %b[6] movad,1 area=3, ind=0, am=0, be=0, %b[18] movad,2 area=3, ind=0, am=0, be=0, %b[3] movad,3 area=3, ind=16, am=0, be=0, %b[22] } { loop_mode pfmul_rsubs,0,sm %b[10], %b[71], %b[56], %b[75] pfmul_rsubs,1,sm %b[14], %b[75], %b[31], %b[85] staad,2 %b[90], %aad3[ %aasti7 + _f32s,_lts0 0x18 ] pfmul_rsubs,3,sm %b[19], %b[51], %b[95], %b[31] pfadds,4,sm %b[40], %b[25], %b[90] staad,5 %b[85], %aad1[ %aasti5 + _f32s,_lts0 0x18 ] movad,0 area=2, ind=0, am=0, be=0, %b[14] movad,1 area=2, ind=8, am=0, be=0, %b[10] movad,2 area=2, ind=24, am=0, be=0, %b[19] movad,3 area=0, ind=8, am=0, be=0, %b[25] } { loop_mode pfsub_rsubs,0,sm %b[39], %b[2], %b[102], %b[88] pfsub_adds,1,sm %b[39], %b[2], %b[102], %b[83] staad,2 %b[89], %aad4[ %aasti8 + _f32s,_lts0 0x8 ] pfmul_adds,3,sm %b[7], %b[15], %b[100], %b[40] pfmuls,4,sm %b[83], %b[70], %b[89] staad,5 %b[88], %aad2[ %aasti6 + _f32s,_lts0 0x8 ] movad,0 area=2, ind=16, am=0, be=0, %b[45] movad,1 area=2, ind=24, am=1, be=0, %b[44] movad,2 area=3, ind=8, am=0, be=0, %b[15] movad,3 area=1, ind=24, am=0, be=0, %b[41] } { loop_mode pfadd_adds,0,sm %b[52], %b[33], %b[91], %b[87] pfadd_rsubs,1,sm %b[52], %b[33], %b[91], %b[86] staad,2 %b[86], %aad3[ %aasti7 + _f32s,_lts0 0x10 ] pfmul_rsubs,3,sm %b[7], %b[11], %b[87], %b[34] pfsubs,4,sm %b[28], %b[34], %b[91] staad,5 %b[81], %aad1[ %aasti5 + _f32s,_lts0 0x10 ] movad,0 area=1, ind=16, am=0, be=0, %b[11] movad,1 area=1, ind=0, am=0, be=0, %b[7] movad,2 area=2, ind=16, am=0, be=0, %b[28] movad,3 area=0, ind=0, am=1, be=0, %b[81] } { loop_mode pfsub_rsubs,0,sm %b[65], %b[42], %b[99], %b[84] pfsub_adds,1,sm %b[65], %b[42], %b[99], %b[79] staad,2 %b[84], %aad4[ %aasti8 ] pfmuls,4,sm %b[47], %b[74], %b[95] staad,5 %b[79], %aad2[ %aasti6 ] movad,0 area=1, ind=24, am=0, be=0, %b[55] movad,1 area=1, ind=8, am=1, be=0, %b[47] movad,2 area=3, ind=24, am=1, be=0, %b[56] movad,3 area=1, ind=16, am=0, be=0, %b[51] } { loop_mode pfadd_adds,0,sm %b[66], %b[36], %b[77], %b[82] pfadd_rsubs,1,sm %b[66], %b[36], %b[77], %b[77] staad,2 %b[96], %aad3[ %aasti7 + _f32s,_lts0 0x8 ] pfmuls,4,sm %b[58], %b[49], %b[96] staad,5 %b[82], %aad1[ %aasti5 + _f32s,_lts0 0x8 ] movad,0 area=0, ind=0, am=0, be=0, %b[60] movad,1 area=0, ind=16, am=0, be=0, %b[59] movad,2 area=2, ind=8, am=1, be=0, %b[67] movad,3 area=2, ind=0, am=0, be=0, %b[68] } { loop_mode pfsub_rsubs,0,sm %b[52], %b[33], %b[97], %b[94] pfsub_adds,1,sm %b[52], %b[33], %b[97], %b[80] staad,2 %b[94], %aad2[ %aasti6 + _f32s,_lts1 0x10 ] pfmuls,3,sm %b[25], %b[19], %b[52] pfmuls,4,sm %b[46], %b[73], %b[97] staad,5 %b[80], %aad2[ %aasti6 + _f32s,_lts0 0x18 ] incr,5 %aaincr3 movad,0 area=0, ind=8, am=1, be=0, %b[46] movad,1 area=0, ind=24, am=0, be=0, %b[33] movad,2 area=1, ind=8, am=1, be=0, %b[71] movad,3 area=1, ind=0, am=0, be=0, %b[72] } { loop_mode pfadd_rsubs,0,sm %b[65], %b[42], %b[76], %b[92] pfadd_rsubs,1,sm %b[39], %b[2], %b[90], %b[78] staad,2 %b[92], %aad4[ %aasti8 + _f32s,_lts1 0x10 ] pfmuls,3,sm %b[44], %b[41], %b[27] pfmuls,4,sm %b[27], %b[69], %b[99] staad,5 %b[78], %aad4[ %aasti8 + _f32s,_lts0 0x18 ] incr,5 %aaincr3 } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfadd_adds,0,sm %b[65], %b[42], %b[76], %b[90] pfadd_adds,1,sm %b[39], %b[2], %b[90], %b[76] staad,2 %b[98], %aad3[ %aasti7 ] incr,2 %aaincr3 pfmuls,3,sm %b[81], %b[28], %b[2] pfmuls,4,sm %b[58], %b[57], %b[93] staad,5 %b[93], %aad1[ %aasti5 ] incr,5 %aaincr3 }
Теоретическая скорость: 16 комплексных чисел за 11 тактов (16/11) = 11.64 Байт/такт
Двойная теоретическая скорость: 23.27 Байт/такт
Замеры скорости
2. stage_radix4_vector4
Прямая векторизация базового алгоритма в 4 раза с помощью инструкций SIMD128.
Код на Си
void stage_radix4_vector4(int data_count, myComplex4 *data_in, myComplex4 *data_out, myComplex4 *coefC, myComplex4 *coefD, myComplex4 *coefE) { __v2di *x_real_in = (__v2di*)&data_in[0].real; __v2di *x_imag_in = (__v2di*)&data_in[0].imag; __v2di *y_real_in = (__v2di*)&data_in[1].real; __v2di *y_imag_in = (__v2di*)&data_in[1].imag; __v2di *z_real_in = (__v2di*)&data_in[2].real; __v2di *z_imag_in = (__v2di*)&data_in[2].imag; __v2di *w_real_in = (__v2di*)&data_in[3].real; __v2di *w_imag_in = (__v2di*)&data_in[3].imag; __v2di *c_real_in = (__v2di*)&coefC[0].real; __v2di *c_imag_in = (__v2di*)&coefC[0].imag; __v2di *d_real_in = (__v2di*)&coefD[0].real; __v2di *d_imag_in = (__v2di*)&coefD[0].imag; __v2di *e_real_in = (__v2di*)&coefE[0].real; __v2di *e_imag_in = (__v2di*)&coefE[0].imag; __v2di *out_0_real = (__v2di*)&data_out[0*data_count/4].real; __v2di *out_0_imag = (__v2di*)&data_out[0*data_count/4].imag; __v2di *out_1_real = (__v2di*)&data_out[1*data_count/4].real; __v2di *out_1_imag = (__v2di*)&data_out[1*data_count/4].imag; __v2di *out_2_real = (__v2di*)&data_out[2*data_count/4].real; __v2di *out_2_imag = (__v2di*)&data_out[2*data_count/4].imag; __v2di *out_3_real = (__v2di*)&data_out[3*data_count/4].real; __v2di *out_3_imag = (__v2di*)&data_out[3*data_count/4].imag; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { __v2di x_real = x_real_in[8*i]; __v2di x_imag = x_imag_in[8*i]; __v2di y_real = y_real_in[8*i]; __v2di y_imag = y_imag_in[8*i]; __v2di z_real = z_real_in[8*i]; __v2di z_imag = z_imag_in[8*i]; __v2di w_real = w_real_in[8*i]; __v2di w_imag = w_imag_in[8*i]; __v2di c_real = c_real_in[2*i]; __v2di c_imag = c_imag_in[2*i]; __v2di d_real = d_real_in[2*i]; __v2di d_imag = d_imag_in[2*i]; __v2di e_real = e_real_in[2*i]; __v2di e_imag = e_imag_in[2*i]; __v2di cy_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(c_real, y_real), __builtin_e2k_qpfmuls(c_imag, y_imag)); __v2di cy_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(c_real, y_imag), __builtin_e2k_qpfmuls(c_imag, y_real)); __v2di dz_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(d_real, z_real), __builtin_e2k_qpfmuls(d_imag, z_imag)); __v2di dz_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(d_real, z_imag), __builtin_e2k_qpfmuls(d_imag, z_real)); __v2di ew_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(e_real, w_real), __builtin_e2k_qpfmuls(e_imag, w_imag)); __v2di ew_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(e_real, w_imag), __builtin_e2k_qpfmuls(e_imag, w_real)); __v2di add02_real = __builtin_e2k_qpfadds( x_real, dz_real); __v2di add02_imag = __builtin_e2k_qpfadds( x_imag, dz_imag); __v2di sub02_real = __builtin_e2k_qpfsubs( x_real, dz_real); __v2di sub02_imag = __builtin_e2k_qpfsubs( x_imag, dz_imag); __v2di add13_real = __builtin_e2k_qpfadds(cy_real, ew_real); __v2di add13_imag = __builtin_e2k_qpfadds(cy_imag, ew_imag); __v2di sub13_real = __builtin_e2k_qpfsubs(cy_real, ew_real); // __v2di sub13_imag = __builtin_e2k_qpfsubs(cy_imag, ew_imag); // __v2di sub13i_real = -sub13_imag; __v2di sub13i_real = __builtin_e2k_qpfsubs(ew_imag, cy_imag); __v2di sub13i_imag = sub13_real; out_0_real[2*i] = __builtin_e2k_qpfadds(add02_real, add13_real); out_0_imag[2*i] = __builtin_e2k_qpfadds(add02_imag, add13_imag); out_1_real[2*i] = __builtin_e2k_qpfsubs(sub02_real, sub13i_real); out_1_imag[2*i] = __builtin_e2k_qpfsubs(sub02_imag, sub13i_imag); out_2_real[2*i] = __builtin_e2k_qpfsubs(add02_real, add13_real); out_2_imag[2*i] = __builtin_e2k_qpfsubs(add02_imag, add13_imag); out_3_real[2*i] = __builtin_e2k_qpfadds(sub02_real, sub13i_real); out_3_imag[2*i] = __builtin_e2k_qpfadds(sub02_imag, sub13i_imag); } }
Основной цикл на ассемблере
.L5045: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=3, abs=24, disp=0 } .L4371: { loop_mode qpfmul_adds,0,sm %b[6], %b[69], %b[57], %b[1] qpfsub_rsubs,1,sm %b[30], %b[74], %b[58], %b[47] qpfsub_adds,2,sm %b[30], %b[74], %b[58], %b[5] qpfmuls,4,sm %b[62], %b[61], %b[55] qpfsubs,5,sm %b[77], %b[73], %b[56] movaqp,1 area=3, ind=0, am=0, be=0, %b[0] } { loop_mode qpfmul_rsubs,0,sm %b[6], %b[63], %b[84], %b[69] qpfadd_adds,1,sm %b[29], %b[81], %b[89], %b[78] qpfadd_rsubs,2,sm %b[29], %b[81], %b[89], %b[74] qpfmuls,4,sm %b[62], %b[67], %b[82] qpfadds,5,sm %b[77], %b[73], %b[87] movaqp,0 area=3, ind=16, am=1, be=0, %b[58] movaqp,1 area=2, ind=0, am=0, be=0, %b[30] movaqp,3 area=1, ind=0, am=0, be=0, %b[57] } { loop_mode qpfmul_rsubs,0,sm %b[36], %b[17], %b[83], %b[73] qpfmul_rsubs,1,sm %b[14], %b[70], %b[86], %b[77] staaqp,2 %b[46], %aad3[ %aasti7 ] qpfmuls,3,sm %b[18], %b[50], %b[81] qpfmuls,4,sm %b[42], %b[35], %b[84] staaqp,5 %b[41], %aad1[ %aasti5 ] movaqp,0 area=1, ind=0, am=0, be=0, %b[62] movaqp,1 area=1, ind=16, am=1, be=0, %b[29] movaqp,2 area=2, ind=0, am=0, be=0, %b[6] movaqp,3 area=1, ind=16, am=1, be=0, %b[63] } { loop_mode qpfmul_adds,0,sm %b[34], %b[50], %b[85], %b[41] qpfmul_adds,1,sm %b[14], %b[37], %b[88], %b[70] staaqp,2 %b[53], %aad4[ %aasti8 + _f32s,_lts0 0x10 ] qpfmuls,3,sm %b[16], %b[13], %b[83] qpfmuls,4,sm %b[42], %b[68], %b[86] staaqp,5 %b[11], %aad2[ %aasti6 + _f32s,_lts0 0x10 ] movaqp,0 area=0, ind=16, am=1, be=0, %b[18] movaqp,1 area=0, ind=0, am=0, be=0, %b[17] movaqp,2 area=2, ind=16, am=1, be=0, %b[36] movaqp,3 area=0, ind=16, am=0, be=0, %b[46] } { loop_mode qpfsub_rsubs,0,sm %b[27], %b[79], %b[52], %b[42] qpfsub_adds,1,sm %b[27], %b[79], %b[52], %b[37] staaqp,2 %b[49], %aad3[ %aasti7 + _f32s,_lts0 0x10 ] incr,2 %aaincr3 qpfsubs,4,sm %b[3], %b[45], %b[50] staaqp,5 %b[7], %aad1[ %aasti5 + _f32s,_lts0 0x10 ] incr,5 %aaincr3 movaqp,1 area=2, ind=16, am=1, be=0, %b[14] movaqp,3 area=0, ind=0, am=1, be=0, %b[11] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfadd_adds,0,sm %b[28], %b[72], %b[54], %b[49] qpfadd_rsubs,1,sm %b[28], %b[72], %b[54], %b[7] staaqp,2 %b[80], %aad4[ %aasti8 ] incr,2 %aaincr3 qpfadds,4,sm %b[45], %b[3], %b[52] staaqp,5 %b[76], %aad2[ %aasti6 ] incr,5 %aaincr3 }
Теоретическая скорость: 16 комплексных чисел за 6 тактов (16/6) = 21.33 Байт/такт
Двойная теоретическая скорость: 42.67 Байт/такт
Замеры скорости
3. stage_radix4_vector4_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью директивы unroll.
Код на Си
void stage_radix4_vector4_unroll2(int data_count, myComplex4 *data_in, myComplex4 *data_out, myComplex4 *coefC, myComplex4 *coefD, myComplex4 *coefE) { __v2di *x_real_in = (__v2di*)&data_in[0].real; __v2di *x_imag_in = (__v2di*)&data_in[0].imag; __v2di *y_real_in = (__v2di*)&data_in[1].real; __v2di *y_imag_in = (__v2di*)&data_in[1].imag; __v2di *z_real_in = (__v2di*)&data_in[2].real; __v2di *z_imag_in = (__v2di*)&data_in[2].imag; __v2di *w_real_in = (__v2di*)&data_in[3].real; __v2di *w_imag_in = (__v2di*)&data_in[3].imag; __v2di *c_real_in = (__v2di*)&coefC[0].real; __v2di *c_imag_in = (__v2di*)&coefC[0].imag; __v2di *d_real_in = (__v2di*)&coefD[0].real; __v2di *d_imag_in = (__v2di*)&coefD[0].imag; __v2di *e_real_in = (__v2di*)&coefE[0].real; __v2di *e_imag_in = (__v2di*)&coefE[0].imag; __v2di *out_0_real = (__v2di*)&data_out[0*data_count/4].real; __v2di *out_0_imag = (__v2di*)&data_out[0*data_count/4].imag; __v2di *out_1_real = (__v2di*)&data_out[1*data_count/4].real; __v2di *out_1_imag = (__v2di*)&data_out[1*data_count/4].imag; __v2di *out_2_real = (__v2di*)&data_out[2*data_count/4].real; __v2di *out_2_imag = (__v2di*)&data_out[2*data_count/4].imag; __v2di *out_3_real = (__v2di*)&data_out[3*data_count/4].real; __v2di *out_3_imag = (__v2di*)&data_out[3*data_count/4].imag; #pragma ivdep #pragma unroll(2) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { __v2di x_real = x_real_in[8*i]; __v2di x_imag = x_imag_in[8*i]; __v2di y_real = y_real_in[8*i]; __v2di y_imag = y_imag_in[8*i]; __v2di z_real = z_real_in[8*i]; __v2di z_imag = z_imag_in[8*i]; __v2di w_real = w_real_in[8*i]; __v2di w_imag = w_imag_in[8*i]; __v2di c_real = c_real_in[2*i]; __v2di c_imag = c_imag_in[2*i]; __v2di d_real = d_real_in[2*i]; __v2di d_imag = d_imag_in[2*i]; __v2di e_real = e_real_in[2*i]; __v2di e_imag = e_imag_in[2*i]; __v2di cy_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(c_real, y_real), __builtin_e2k_qpfmuls(c_imag, y_imag)); __v2di cy_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(c_real, y_imag), __builtin_e2k_qpfmuls(c_imag, y_real)); __v2di dz_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(d_real, z_real), __builtin_e2k_qpfmuls(d_imag, z_imag)); __v2di dz_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(d_real, z_imag), __builtin_e2k_qpfmuls(d_imag, z_real)); __v2di ew_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(e_real, w_real), __builtin_e2k_qpfmuls(e_imag, w_imag)); __v2di ew_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(e_real, w_imag), __builtin_e2k_qpfmuls(e_imag, w_real)); __v2di add02_real = __builtin_e2k_qpfadds( x_real, dz_real); __v2di add02_imag = __builtin_e2k_qpfadds( x_imag, dz_imag); __v2di sub02_real = __builtin_e2k_qpfsubs( x_real, dz_real); __v2di sub02_imag = __builtin_e2k_qpfsubs( x_imag, dz_imag); __v2di add13_real = __builtin_e2k_qpfadds(cy_real, ew_real); __v2di add13_imag = __builtin_e2k_qpfadds(cy_imag, ew_imag); __v2di sub13_real = __builtin_e2k_qpfsubs(cy_real, ew_real); // __v2di sub13_imag = __builtin_e2k_qpfsubs(cy_imag, ew_imag); // __v2di sub13i_real = -sub13_imag; __v2di sub13i_real = __builtin_e2k_qpfsubs(ew_imag, cy_imag); __v2di sub13i_imag = sub13_real; out_0_real[2*i] = __builtin_e2k_qpfadds(add02_real, add13_real); out_0_imag[2*i] = __builtin_e2k_qpfadds(add02_imag, add13_imag); out_1_real[2*i] = __builtin_e2k_qpfsubs(sub02_real, sub13i_real); out_1_imag[2*i] = __builtin_e2k_qpfsubs(sub02_imag, sub13i_imag); out_2_real[2*i] = __builtin_e2k_qpfsubs(add02_real, add13_real); out_2_imag[2*i] = __builtin_e2k_qpfsubs(add02_imag, add13_imag); out_3_real[2*i] = __builtin_e2k_qpfadds(sub02_real, sub13i_real); out_3_imag[2*i] = __builtin_e2k_qpfadds(sub02_imag, sub13i_imag); } }
Основной цикл на ассемблере
.L6226: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=128 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=160 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=12, disp=192 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=12, disp=224 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=2, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=2, abs=16, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=2, abs=20, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=2, abs=20, disp=32 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=3, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=3, abs=24, disp=32 } .L5153: { loop_mode qpfadd_adds,0,sm %b[32], %b[21], %b[94], %b[65] qpfadd_rsubs,1,sm %b[32], %b[21], %b[94], %b[88] qpfsubs,2,sm %b[61], %b[65], %b[92] qpfmuls,3,sm %b[15], %b[40], %b[0] qpfmuls,4,sm %b[91], %b[48], %b[1] qpfmuls,5,sm %b[17], %b[9], %b[90] } { loop_mode qpfmul_rsubs,0,sm %b[22], %b[60], %g16, %b[21] qpfmul_adds,1,sm %b[8], %b[64], %b[95], %b[22] qpfsubs,2,sm %b[81], %b[70], %b[94] qpfmul_adds,3,sm %b[22], %b[57], %b[83], %b[17] qpfmuls,4,sm %b[35], %b[55], %g16 qpfsubs,5,sm %b[16], %b[10], %b[83] } { loop_mode qpfmul_rsubs,0,sm %b[14], %b[51], %b[3], %b[8] qpfmul_rsubs,1,sm %b[46], %b[52], %b[96], %b[14] qpfadds,2,sm %b[81], %b[70], %b[84] qpfmul_rsubs,3,sm %b[8], %b[38], %b[84], %b[3] qpfmuls,4,sm %b[53], %b[50], %b[70] qpfmuls,5,sm %b[91], %b[49], %b[81] } { loop_mode qpfmul_rsubs,0,sm %b[13], %b[9], %b[2], %b[68] qpfmul_rsubs,1,sm %b[56], %b[33], %b[93], %b[79] staaqp,2 %b[82], %aad3[ %aasti7 ] qpfmul_adds,3,sm %b[56], %b[68], %b[89], %b[2] qpfadds,4,sm %b[16], %b[10], %b[82] staaqp,5 %b[79], %aad1[ %aasti5 ] } { loop_mode qpfsub_rsubs,0,sm %b[29], %b[5], %b[92], %b[80] qpfsub_adds,1,sm %b[29], %b[5], %b[92], %b[77] staaqp,2 %b[80], %aad2[ %aasti6 + _f32s,_lts1 0x10 ] qpfmul_adds,3,sm %b[13], %b[42], %b[90], %b[16] qpfsubs,4,sm %b[18], %b[4], %b[90] staaqp,5 %b[77], %aad1[ %aasti5 + _f32s,_lts0 0x30 ] movaqp,0 area=6, ind=0, am=0, be=0, %b[10] movaqp,1 area=6, ind=16, am=1, be=0, %b[89] movaqp,2 area=6, ind=0, am=0, be=0, %b[9] movaqp,3 area=6, ind=16, am=1, be=0, %b[13] } { loop_mode qpfadd_rsubs,0,sm %b[47], %b[24], %b[87], %b[78] qpfsub_adds,1,sm %b[30], %b[19], %b[94], %b[75] staaqp,2 %b[75], %aad3[ %aasti7 + _f32s,_lts1 0x30 ] qpfmuls,3,sm %g17, %b[66], %b[91] qpfadds,4,sm %b[4], %b[18], %b[92] staaqp,5 %b[78], %aad2[ %aasti6 + _f32s,_lts0 0x20 ] movaqp,0 area=5, ind=0, am=0, be=0, %b[4] movaqp,1 area=5, ind=16, am=1, be=0, %b[32] movaqp,2 area=5, ind=0, am=0, be=0, %b[18] movaqp,3 area=5, ind=16, am=1, be=0, %b[33] } { loop_mode qpfsub_rsubs,0,sm %b[30], %b[19], %b[94], %b[73] qpfadd_rsubs,1,sm %b[41], %b[23], %b[84], %b[76] staaqp,2 %b[76], %aad4[ %aasti8 + _f32s,_lts0 0x10 ] qpfmuls,4,sm %b[34], %b[36], %b[93] staaqp,5 %b[73], %aad1[ %aasti5 + _f32s,_lts0 0x10 ] movaqp,0 area=4, ind=0, am=0, be=0, %b[42] movaqp,1 area=4, ind=16, am=1, be=0, %b[51] movaqp,2 area=3, ind=16, am=0, be=0, %b[38] movaqp,3 area=1, ind=16, am=0, be=0, %b[46] } { loop_mode qpfadd_adds,0,sm %b[47], %b[24], %b[87], %b[74] qpfsub_adds,1,sm %b[47], %b[24], %b[83], %b[71] staaqp,2 %b[74], %aad3[ %aasti7 + _f32s,_lts1 0x10 ] qpfmuls,3,sm %g17, %b[31], %b[87] qpfmuls,4,sm %b[53], %b[67], %b[94] staaqp,5 %b[71], %aad4[ %aasti8 + _f32s,_lts0 0x20 ] movaqp,0 area=3, ind=0, am=0, be=0, %b[56] movaqp,1 area=3, ind=16, am=1, be=0, %b[53] movaqp,2 area=4, ind=0, am=0, be=0, %b[52] movaqp,3 area=4, ind=16, am=1, be=0, %g17 } { loop_mode qpfsub_rsubs,0,sm %b[47], %b[24], %b[83], %b[72] qpfadd_adds,1,sm %b[41], %b[23], %b[84], %b[69] staaqp,2 %b[72], %aad4[ %aasti8 ] qpfmul_adds,3,sm %b[12], %b[48], %b[81], %b[57] qpfmuls,4,sm %b[35], %b[58], %b[81] staaqp,5 %b[69], %aad2[ %aasti6 ] movaqp,0 area=2, ind=0, am=0, be=0, %b[35] movaqp,1 area=2, ind=16, am=1, be=0, %b[24] movaqp,2 area=1, ind=0, am=1, be=0, %b[47] movaqp,3 area=0, ind=0, am=0, be=0, %b[48] } { loop_mode qpfadd_adds,0,sm %b[29], %b[5], %b[82], %b[70] qpfadd_rsubs,1,sm %b[29], %b[5], %b[82], %b[67] staaqp,2 %b[85], %aad3[ %aasti7 + _f32s,_lts0 0x20 ] incr,2 %aaincr3 qpfmul_adds,3,sm %b[44], %b[67], %b[70], %b[61] qpfmuls,4,sm %b[34], %b[62], %b[82] staaqp,5 %b[86], %aad1[ %aasti5 + _f32s,_lts0 0x20 ] incr,5 %aaincr3 movaqp,0 area=1, ind=16, am=1, be=0, %b[60] movaqp,1 area=1, ind=0, am=0, be=0, %b[34] movaqp,2 area=3, ind=0, am=1, be=0, %b[5] movaqp,3 area=2, ind=0, am=0, be=0, %b[29] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfsub_rsubs,0,sm %b[41], %b[23], %b[90], %b[83] qpfsub_adds,1,sm %b[41], %b[23], %b[90], %b[84] staaqp,2 %b[65], %aad4[ %aasti8 + _f32s,_lts0 0x30 ] incr,2 %aaincr3 qpfadds,4,sm %b[63], %b[59], %b[85] staaqp,5 %b[88], %aad2[ %aasti6 + _f32s,_lts0 0x30 ] incr,5 %aaincr3 movaqp,0 area=0, ind=16, am=1, be=0, %b[41] movaqp,1 area=0, ind=0, am=0, be=0, %b[23] movaqp,2 area=2, ind=16, am=1, be=0, %b[64] movaqp,3 area=0, ind=16, am=1, be=0, %b[65] }
Теоретическая скорость: 32 комплексных числа за 11 тактов (32/11) = 23.27 Байт/такт
Двойная теоретическая скорость: 46.55 Байт/такт
Замеры скорости
4. stage_radix4_vector4_unroll3
Здесь происходит раскрутка цикла в 3 раза с помощью директивы unroll.
Код на Си
void stage_radix4_vector4_unroll3(int data_count, myComplex4 *data_in, myComplex4 *data_out, myComplex4 *coefC, myComplex4 *coefD, myComplex4 *coefE) { __v2di *x_real_in = (__v2di*)&data_in[0].real; __v2di *x_imag_in = (__v2di*)&data_in[0].imag; __v2di *y_real_in = (__v2di*)&data_in[1].real; __v2di *y_imag_in = (__v2di*)&data_in[1].imag; __v2di *z_real_in = (__v2di*)&data_in[2].real; __v2di *z_imag_in = (__v2di*)&data_in[2].imag; __v2di *w_real_in = (__v2di*)&data_in[3].real; __v2di *w_imag_in = (__v2di*)&data_in[3].imag; __v2di *c_real_in = (__v2di*)&coefC[0].real; __v2di *c_imag_in = (__v2di*)&coefC[0].imag; __v2di *d_real_in = (__v2di*)&coefD[0].real; __v2di *d_imag_in = (__v2di*)&coefD[0].imag; __v2di *e_real_in = (__v2di*)&coefE[0].real; __v2di *e_imag_in = (__v2di*)&coefE[0].imag; __v2di *out_0_real = (__v2di*)&data_out[0*data_count/4].real; __v2di *out_0_imag = (__v2di*)&data_out[0*data_count/4].imag; __v2di *out_1_real = (__v2di*)&data_out[1*data_count/4].real; __v2di *out_1_imag = (__v2di*)&data_out[1*data_count/4].imag; __v2di *out_2_real = (__v2di*)&data_out[2*data_count/4].real; __v2di *out_2_imag = (__v2di*)&data_out[2*data_count/4].imag; __v2di *out_3_real = (__v2di*)&data_out[3*data_count/4].real; __v2di *out_3_imag = (__v2di*)&data_out[3*data_count/4].imag; #pragma ivdep #pragma unroll(3) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { __v2di x_real = x_real_in[8*i]; __v2di x_imag = x_imag_in[8*i]; __v2di y_real = y_real_in[8*i]; __v2di y_imag = y_imag_in[8*i]; __v2di z_real = z_real_in[8*i]; __v2di z_imag = z_imag_in[8*i]; __v2di w_real = w_real_in[8*i]; __v2di w_imag = w_imag_in[8*i]; __v2di c_real = c_real_in[2*i]; __v2di c_imag = c_imag_in[2*i]; __v2di d_real = d_real_in[2*i]; __v2di d_imag = d_imag_in[2*i]; __v2di e_real = e_real_in[2*i]; __v2di e_imag = e_imag_in[2*i]; __v2di cy_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(c_real, y_real), __builtin_e2k_qpfmuls(c_imag, y_imag)); __v2di cy_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(c_real, y_imag), __builtin_e2k_qpfmuls(c_imag, y_real)); __v2di dz_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(d_real, z_real), __builtin_e2k_qpfmuls(d_imag, z_imag)); __v2di dz_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(d_real, z_imag), __builtin_e2k_qpfmuls(d_imag, z_real)); __v2di ew_real = __builtin_e2k_qpfsubs(__builtin_e2k_qpfmuls(e_real, w_real), __builtin_e2k_qpfmuls(e_imag, w_imag)); __v2di ew_imag = __builtin_e2k_qpfadds(__builtin_e2k_qpfmuls(e_real, w_imag), __builtin_e2k_qpfmuls(e_imag, w_real)); __v2di add02_real = __builtin_e2k_qpfadds( x_real, dz_real); __v2di add02_imag = __builtin_e2k_qpfadds( x_imag, dz_imag); __v2di sub02_real = __builtin_e2k_qpfsubs( x_real, dz_real); __v2di sub02_imag = __builtin_e2k_qpfsubs( x_imag, dz_imag); __v2di add13_real = __builtin_e2k_qpfadds(cy_real, ew_real); __v2di add13_imag = __builtin_e2k_qpfadds(cy_imag, ew_imag); __v2di sub13_real = __builtin_e2k_qpfsubs(cy_real, ew_real); // __v2di sub13_imag = __builtin_e2k_qpfsubs(cy_imag, ew_imag); // __v2di sub13i_real = -sub13_imag; __v2di sub13i_real = __builtin_e2k_qpfsubs(ew_imag, cy_imag); __v2di sub13i_imag = sub13_real; out_0_real[2*i] = __builtin_e2k_qpfadds(add02_real, add13_real); out_0_imag[2*i] = __builtin_e2k_qpfadds(add02_imag, add13_imag); out_1_real[2*i] = __builtin_e2k_qpfsubs(sub02_real, sub13i_real); out_1_imag[2*i] = __builtin_e2k_qpfsubs(sub02_imag, sub13i_imag); out_2_real[2*i] = __builtin_e2k_qpfsubs(add02_real, add13_real); out_2_imag[2*i] = __builtin_e2k_qpfsubs(add02_imag, add13_imag); out_3_real[2*i] = __builtin_e2k_qpfadds(sub02_real, sub13i_real); out_3_imag[2*i] = __builtin_e2k_qpfadds(sub02_imag, sub13i_imag); } }
Основной цикл на ассемблере
.L7982: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=2, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=2, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=4, disp=128 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=4, disp=160 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=6, disp=192 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=6, disp=224 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=8, disp=256 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=288 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=1, abs=10, disp=320 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=12, disp=352 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=2, abs=16, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=2, abs=16, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=2, abs=20, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=2, abs=20, disp=32 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=2, abs=24, disp=64 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=2, abs=24, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=2, abs=28, disp=32 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=2, abs=28, disp=64 } .L6334: { loop_mode qpfsub_rsubs,0,sm %b[105], %b[23], %g16, %g18 qpfsub_adds,1,sm %b[105], %b[23], %g16, %g17 qpfsubs,2,sm %g19, %g20, %g21 qpfmuls,3,sm %b[38], %b[113], %g24 qpfmuls,4,sm %b[46], %b[81], %g23 qpfmuls,5,sm %b[38], %b[64], %g22 } { loop_mode qpfadd_adds,0,sm %b[102], %b[49], %g25, %g27 qpfadd_rsubs,1,sm %b[102], %b[49], %g25, %g26 qpfadds,2,sm %g19, %g20, %g28 qpfmuls,3,sm %b[45], %b[109], %g31 qpfmuls,4,sm %b[42], %b[31], %g30 qpfmuls,5,sm %b[90], %b[94], %g29 } { loop_mode qpfsub_rsubs,0,sm %b[30], %r0, %r1, %r3 qpfsub_adds,1,sm %b[30], %r0, %r1, %r2 qpfmuls,2,sm %b[90], %b[97], %r7 qpfmuls,3,sm %b[48], %b[104], %b[0] qpfmuls,4,sm %b[82], %b[18], %r5 qpfmuls,5,sm %b[19], %b[76], %r4 } { loop_mode qpfadd_adds,0,sm %b[63], %r9, %r10, %r12 qpfadd_rsubs,1,sm %b[63], %r9, %r10, %r11 qpfmuls,2,sm %b[13], %b[93], %r14 qpfmuls,3,sm %b[11], %b[85], %r16 qpfmuls,4,sm %b[82], %b[70], %r15 qpfadds,5,sm %b[118], %b[119], %r13 } { loop_mode qpfsub_rsubs,0,sm %b[63], %r9, %g21, %g21 qpfsub_adds,1,sm %b[63], %r9, %g21, %r19 staaqp,2 %r18, %aad3[ %aasti7 ] qpfmuls,3,sm %b[17], %b[73], %b[118] qpfsubs,4,sm %b[119], %b[118], %r20 staaqp,5 %r17, %aad1[ %aasti5 ] } { loop_mode qpfadd_adds,0,sm %b[30], %r0, %g28, %g28 qpfadd_rsubs,1,sm %b[30], %r0, %g28, %r23 staaqp,2 %r22, %aad2[ %aasti6 + _f32s,_lts1 0x10 ] qpfmul_rsubs,3,sm %b[86], %b[97], %g29, %b[19] qpfsubs,4,sm %b[116], %r24, %r25 staaqp,5 %r21, %aad1[ %aasti5 + _f32s,_lts0 0x30 ] movaqp,0 area=10, ind=0, am=0, be=0, %b[1] movaqp,1 area=10, ind=16, am=1, be=0, %b[13] } { loop_mode qpfmul_adds,0,sm %b[12], %b[113], %g22, %b[49] qpfmul_adds,1,sm %b[86], %b[94], %r7, %b[45] staaqp,2 %r27, %aad3[ %aasti7 + _f32s,_lts1 0x30 ] qpfmul_rsubs,3,sm %b[27], %b[35], %g30, %r24 qpfadds,4,sm %b[116], %r24, %b[113] staaqp,5 %r26, %aad2[ %aasti6 + _f32s,_lts0 0x20 ] movaqp,0 area=9, ind=0, am=0, be=0, %b[30] movaqp,1 area=9, ind=16, am=1, be=0, %b[46] movaqp,2 area=9, ind=0, am=0, be=0, %b[23] movaqp,3 area=9, ind=16, am=1, be=0, %b[38] } { loop_mode qpfmul_adds,0,sm %b[34], %b[106], %r30, %b[117] qpfmul_adds,1,sm %b[41], %b[114], %g23, %b[116] staaqp,2 %b[117], %aad4[ %aasti8 + _f32s,_lts0 0x10 ] qpfmul_rsubs,3,sm %b[22], %b[109], %r31, %b[114] qpfsubs,4,sm %r29, %b[37], %g22 staaqp,5 %r28, %aad1[ %aasti5 + _f32s,_lts0 0x10 ] movaqp,0 area=8, ind=0, am=0, be=0, %b[82] movaqp,1 area=8, ind=16, am=1, be=0, %b[86] movaqp,2 area=8, ind=0, am=0, be=0, %b[63] movaqp,3 area=8, ind=16, am=1, be=0, %b[76] } { loop_mode qpfmul_rsubs,0,sm %b[7], %b[75], %r4, %g20 qpfmul_rsubs,1,sm %b[8], %b[87], %r14, %g19 staaqp,2 %r33, %aad3[ %aasti7 + _f32s,_lts1 0x10 ] qpfmul_rsubs,3,sm %b[34], %b[110], %b[2], %b[35] qpfmuls,4,sm %b[42], %b[35], %g23 staaqp,5 %r32, %aad4[ %aasti8 + _f32s,_lts0 0x20 ] movaqp,0 area=7, ind=0, am=0, be=0, %b[2] movaqp,1 area=7, ind=16, am=1, be=0, %b[7] movaqp,2 area=7, ind=0, am=0, be=0, %b[8] movaqp,3 area=7, ind=16, am=1, be=0, %b[34] } { loop_mode qpfmul_adds,0,sm %b[69], %b[70], %r5, %r9 qpfmul_rsubs,1,sm %b[69], %b[18], %r15, %r0 staaqp,2 %r35, %aad4[ %aasti8 ] qpfmul_rsubs,3,sm %b[41], %b[81], %r36, %r29 qpfadds,4,sm %r29, %b[37], %g29 staaqp,5 %r34, %aad2[ %aasti6 ] movaqp,0 area=6, ind=0, am=0, be=0, %b[37] movaqp,1 area=6, ind=16, am=1, be=0, %b[42] movaqp,2 area=6, ind=0, am=0, be=0, %b[18] movaqp,3 area=6, ind=16, am=1, be=0, %b[41] } { loop_mode qpfsub_rsubs,0,sm %b[54], %b[77], %r20, %r18 qpfsub_adds,1,sm %b[54], %b[77], %r20, %r17 staaqp,2 %g18, %aad3[ %aasti7 + _f32s,_lts0 0x20 ] qpfmul_rsubs,3,sm %b[12], %b[64], %g24, %b[75] qpfsubs,4,sm %r37, %r38, %g16 staaqp,5 %g17, %aad1[ %aasti5 + _f32s,_lts0 0x20 ] movaqp,0 area=5, ind=16, am=1, be=0, %b[64] movaqp,1 area=5, ind=0, am=0, be=0, %b[12] movaqp,2 area=5, ind=16, am=1, be=0, %b[70] movaqp,3 area=5, ind=0, am=0, be=0, %b[69] } { loop_mode qpfadd_rsubs,0,sm %b[60], %b[51], %r13, %r22 qpfsub_adds,1,sm %b[100], %b[47], %r25, %r21 staaqp,2 %g27, %aad4[ %aasti8 + _f32s,_lts0 0x30 ] qpfmul_adds,3,sm %b[22], %b[55], %g31, %r38 qpfadds,4,sm %r38, %r37, %g25 staaqp,5 %g26, %aad2[ %aasti6 + _f32s,_lts0 0x30 ] movaqp,0 area=4, ind=16, am=1, be=0, %b[55] movaqp,1 area=4, ind=0, am=0, be=0, %b[22] movaqp,2 area=4, ind=16, am=1, be=0, %b[87] movaqp,3 area=4, ind=0, am=0, be=0, %b[81] } { loop_mode qpfsub_rsubs,0,sm %b[100], %b[47], %r25, %r27 qpfadd_rsubs,1,sm %b[103], %b[21], %b[113], %r26 staaqp,2 %r3, %aad3[ %aasti7 + _f32s,_lts0 0x40 ] qpfmul_adds,3,sm %b[27], %b[31], %g23, %r37 qpfsubs,4,sm %b[115], %r39, %r1 staaqp,5 %r2, %aad1[ %aasti5 + _f32s,_lts0 0x40 ] movaqp,0 area=3, ind=0, am=0, be=0, %b[93] movaqp,1 area=3, ind=16, am=1, be=0, %b[90] movaqp,2 area=3, ind=0, am=0, be=0, %b[31] movaqp,3 area=3, ind=16, am=1, be=0, %b[27] } { loop_mode qpfadd_adds,0,sm %b[60], %b[51], %r13, %b[115] qpfsub_adds,1,sm %b[60], %b[51], %g22, %r28 staaqp,2 %r12, %aad4[ %aasti8 + _f32s,_lts0 0x50 ] qpfmul_adds,3,sm %b[6], %b[91], %r16, %r39 qpfadds,4,sm %r39, %b[115], %r10 staaqp,5 %r11, %aad2[ %aasti6 + _f32s,_lts0 0x50 ] movaqp,0 area=2, ind=0, am=0, be=0, %b[97] movaqp,1 area=2, ind=16, am=1, be=0, %b[94] movaqp,2 area=2, ind=0, am=0, be=0, %b[105] movaqp,3 area=1, ind=16, am=0, be=0, %b[102] } { loop_mode qpfsub_rsubs,0,sm %b[60], %b[51], %g22, %r33 qpfadd_adds,1,sm %b[103], %b[21], %b[113], %r32 staaqp,2 %g21, %aad3[ %aasti7 + _f32s,_lts0 0x50 ] incr,2 %aaincr3 qpfmul_adds,3,sm %b[5], %b[74], %b[118], %b[113] qpfmuls,4,sm %b[43], %b[53], %r31 staaqp,5 %r19, %aad1[ %aasti5 + _f32s,_lts0 0x50 ] incr,5 %aaincr3 movaqp,0 area=1, ind=16, am=1, be=0, %b[109] movaqp,1 area=1, ind=0, am=0, be=0, %b[60] movaqp,2 area=2, ind=16, am=1, be=0, %b[51] movaqp,3 area=1, ind=0, am=1, be=0, %b[106] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfadd_adds,0,sm %b[54], %b[77], %g29, %r35 qpfadd_rsubs,1,sm %b[54], %b[77], %g29, %r34 staaqp,2 %g28, %aad4[ %aasti8 + _f32s,_lts0 0x40 ] incr,2 %aaincr3 qpfmuls,3,sm %b[44], %b[112], %r36 qpfmuls,4,sm %b[48], %b[108], %r30 staaqp,5 %r23, %aad2[ %aasti6 + _f32s,_lts0 0x40 ] incr,5 %aaincr3 movaqp,0 area=0, ind=16, am=1, be=0, %b[54] movaqp,1 area=0, ind=0, am=0, be=0, %b[48] movaqp,2 area=0, ind=0, am=0, be=0, %b[77] movaqp,3 area=0, ind=16, am=1, be=0, %b[110] }
Теоретическая скорость: 48 комплексных чисел за 16 тактов (48/16) = 24 Байт/такт
Двойная теоретическая скорость: 48 Байт/такт
Замеры скорости
Компилятор не сумел векторизовать вариант stage_radix4_vector4_etalon в 4 раза.
До этого он успешно векторизовал вариант stage_radix4_vector2_etalon в 2 раза.
Итоги по stage_radix4_vector4
Пишем LastStage
В случае выполнения прямой векторизации в 2 раза нужно использовать lastStage_radix2.
В случае выполнения прямой векторизации в 4 раза нужно использовать lastStage_radix4 (или сначала stage_radix2_vector2 и затем lastStage_radix2).
lastStage_radix2
1. lastStage_radix2_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void lastStage_radix2_etalon(int data_count, myComplex2 *data_in, myComplex *data_out, myComplex *coef) { float *x_real_in = &data_in[0].real[0]; float *x_imag_in = &data_in[0].imag[0]; float *y_real_in = &data_in[0].real[1]; float *y_imag_in = &data_in[0].imag[1]; float *c_real_in = &coef[0].real; float *c_imag_in = &coef[0].imag; float *out_add_real = &data_out[0].real; float *out_add_imag = &data_out[0].imag; float *out_sub_real = &data_out[data_count/2].real; float *out_sub_imag = &data_out[data_count/2].imag; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { float x_real = x_real_in[4*i]; float x_imag = x_imag_in[4*i]; float y_real = y_real_in[4*i]; float y_imag = y_imag_in[4*i]; float c_real = c_real_in[2*i]; float c_imag = c_imag_in[2*i]; float cy_real = c_real*y_real - c_imag*y_imag; float cy_imag = c_real*y_imag + c_imag*y_real; out_add_real[2*i] = x_real + cy_real; out_add_imag[2*i] = x_imag + cy_imag; out_sub_real[2*i] = x_real - cy_real; out_sub_imag[2*i] = x_imag - cy_imag; } }
Основной цикл на ассемблере
.L479: { fapb ct=1, dcd=0, fmt=3, mrng=16, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=2, asz=5, abs=0, disp=0 } .L182: { loop_mode fmuls,0,sm %b[67], %b[6], %b[37] fsubs,1,sm %b[46], %b[24], %b[58] staaw,2 %b[62], %aad1[ %aasti3 + _f32s,_lts0 0x4 ] fmul_adds,3,sm %b[55], %b[13], %b[43], %b[14] fadds,4,sm %b[46], %b[24], %b[57] staaw,5 %b[61], %aad2[ %aasti4 + _f32s,_lts0 0x4 ] movaw,0 area=0, ind=4, am=0, be=0, %b[0] movaw,1 area=0, ind=12, am=0, be=0, %b[1] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END fmuls,0,sm %b[67], %b[7], %b[62] fsubs,1,sm %b[35], %b[56], %b[70] staaw,2 %b[74], %aad1[ %aasti3 ] incr,2 %aaincr3 fmul_rsubs,3,sm %b[55], %b[12], %b[68], %b[46] fadds,4,sm %b[35], %b[56], %b[69] staaw,5 %b[73], %aad2[ %aasti4 ] incr,5 %aaincr3 movaw,0 area=0, ind=0, am=0, be=0, %b[13] movaw,1 area=0, ind=8, am=1, be=0, %b[24] movaw,2 area=0, ind=4, am=1, be=0, %b[61] movaw,3 area=0, ind=0, am=0, be=0, %b[43] }
Теоретическая скорость: 2 комплексных числа за 2 такта (2/2) = 8 Байт/такт
Замеры скорости
2. lastStage_radix2_etalon_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью директивы unroll.
Код на Си
void lastStage_radix2_etalon_unroll2(int data_count, myComplex2 *data_in, myComplex *data_out, myComplex *coef) { float *x_real_in = &data_in[0].real[0]; float *x_imag_in = &data_in[0].imag[0]; float *y_real_in = &data_in[0].real[1]; float *y_imag_in = &data_in[0].imag[1]; float *c_real_in = &coef[0].real; float *c_imag_in = &coef[0].imag; float *out_add_real = &data_out[0].real; float *out_add_imag = &data_out[0].imag; float *out_sub_real = &data_out[data_count/2].real; float *out_sub_imag = &data_out[data_count/2].imag; #pragma ivdep #pragma unroll(2) #pragma prefetch for(int64_t i = 0; i < data_count/2; ++i) { float x_real = x_real_in[4*i]; float x_imag = x_imag_in[4*i]; float y_real = y_real_in[4*i]; float y_imag = y_imag_in[4*i]; float c_real = c_real_in[2*i]; float c_imag = c_imag_in[2*i]; float cy_real = c_real*y_real - c_imag*y_imag; float cy_imag = c_real*y_imag + c_imag*y_real; out_add_real[2*i] = x_real + cy_real; out_add_imag[2*i] = x_imag + cy_imag; out_sub_real[2*i] = x_real - cy_real; out_sub_imag[2*i] = x_imag - cy_imag; } }
Основной цикл на ассемблере
.L1357: { fapb ct=0, dcd=0, fmt=3, mrng=12, d=0, incr=1, ind=1, asz=4, abs=0, disp=20 fapb dpl=0, dcd=0, fmt=3, mrng=20, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=16, disp=0 } .L525: { loop_mode pfmuls,0,sm %b[56], %b[16], %b[26] insfd,1,sm %b[52], %r8, %b[47], %b[1] pfmuls,2,sm %b[56], %b[13], %b[31] pshufb,3,sm %b[19], %b[9], %r0, %b[57] insfd,4,sm %b[55], %r8, %b[37], %b[0] pfadds,5,sm %b[42], %b[43], %b[10] } { loop_mode pfmul_rsubs,0,sm %b[5], %b[15], %b[28], %b[42] insfd,1,sm %b[27], %r8, %b[22], %b[32] pfmul_adds,2,sm %b[5], %b[18], %b[33], %b[37] pshufb,4,sm %b[12], %b[23], %r0, %b[55] staad,5 %b[57], %aad1[ %aasti3 + _f32s,_lts0 0x8 ] movad,0 area=1, ind=0, am=1, be=0, %b[43] movad,1 area=1, ind=8, am=0, be=0, %b[48] movaw,3 area=0, ind=12, am=0, be=0, %b[47] } { loop_mode pfsubs,0,sm %b[8], %b[46], %b[5] insfd,1,sm %b[19], %r8, %b[9], %b[58] pfsubs,2,sm %b[40], %b[41], %b[15] insfd,4,sm %b[12], %r8, %b[23], %b[56] staad,5 %b[55], %aad2[ %aasti4 + _f32s,_lts0 0x8 ] movaw,0 area=0, ind=8, am=0, be=0, %b[22] movaw,1 area=0, ind=0, am=0, be=0, %b[28] movaw,2 area=0, ind=8, am=0, be=0, %b[18] movaw,3 area=0, ind=4, am=0, be=0, %b[27] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfadds,0,sm %b[8], %b[46], %b[19] insfd,1,sm %b[30], %r8, %b[29], %b[9] staad,2 %b[58], %aad1[ %aasti3 ] incr,2 %aaincr3 pshufb,3,sm %b[50], %b[45], %r0, %b[52] insfd,4,sm %b[24], %r8, %b[49], %b[12] staad,5 %b[56], %aad2[ %aasti4 ] incr,5 %aaincr3 movaw,1 area=0, ind=4, am=1, be=0, %b[23] movaw,2 area=0, ind=0, am=1, be=0, %b[33] movaw,3 area=0, ind=16, am=0, be=0, %b[51] }
Теоретическая скорость: 4 комплексных числа за 4 такта (4/4) = 8 Байт/такт
Замеры скорости
3. lastStage_radix2_simd64
Модифицированный stage_radix2_simd64 для работы с векторными данными.
Код на Си
void lastStage_radix2_simd64(int data_count, myComplex2 *data_in, myComplex *data_out, myComplex *coef) { uint64_t *xy0_real_in = (uint64_t*)&data_in[0].real; uint64_t *xy0_imag_in = (uint64_t*)&data_in[0].imag; uint64_t *xy1_real_in = (uint64_t*)&data_in[1].real; uint64_t *xy1_imag_in = (uint64_t*)&data_in[1].imag; uint64_t *c0_in = (uint64_t*)&coef[0]; uint64_t *c1_in = (uint64_t*)&coef[1]; uint64_t *out_add0 = (uint64_t*)&data_out[0]; uint64_t *out_add1 = (uint64_t*)&data_out[1]; uint64_t *out_sub0 = (uint64_t*)&data_out[data_count/2 + 0]; uint64_t *out_sub1 = (uint64_t*)&data_out[data_count/2 + 1]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { uint64_t xy0_real = xy0_real_in[4*i]; uint64_t xy0_imag = xy0_imag_in[4*i]; uint64_t xy1_real = xy1_real_in[4*i]; uint64_t xy1_imag = xy1_imag_in[4*i]; uint64_t c0 = c0_in[2*i]; uint64_t c1 = c1_in[2*i]; uint64_t x_real = __builtin_e2k_pshufb(xy1_real, xy0_real, 0x0B0A090803020100); uint64_t x_imag = __builtin_e2k_pshufb(xy1_imag, xy0_imag, 0x0B0A090803020100); uint64_t y_real = __builtin_e2k_pshufb(xy1_real, xy0_real, 0x0F0E0D0C07060504); uint64_t y_imag = __builtin_e2k_pshufb(xy1_imag, xy0_imag, 0x0F0E0D0C07060504); uint64_t c_real = __builtin_e2k_pshufb(c1, c0, 0x0B0A090803020100); uint64_t c_imag = __builtin_e2k_pshufb(c1, c0, 0x0F0E0D0C07060504); uint64_t cy_real = __builtin_e2k_pfsubs(__builtin_e2k_pfmuls(c_real, y_real), __builtin_e2k_pfmuls(c_imag, y_imag)); uint64_t cy_imag = __builtin_e2k_pfadds(__builtin_e2k_pfmuls(c_real, y_imag), __builtin_e2k_pfmuls(c_imag, y_real)); uint64_t add_real = __builtin_e2k_pfadds(x_real, cy_real); uint64_t add_imag = __builtin_e2k_pfadds(x_imag, cy_imag); uint64_t sub_real = __builtin_e2k_pfsubs(x_real, cy_real); uint64_t sub_imag = __builtin_e2k_pfsubs(x_imag, cy_imag); out_add0[2*i] = __builtin_e2k_pshufb(add_imag, add_real, 0x0B0A090803020100); out_add1[2*i] = __builtin_e2k_pshufb(add_imag, add_real, 0x0F0E0D0C07060504); out_sub0[2*i] = __builtin_e2k_pshufb(sub_imag, sub_real, 0x0B0A090803020100); out_sub1[2*i] = __builtin_e2k_pshufb(sub_imag, sub_real, 0x0F0E0D0C07060504); } }
Основной цикл на ассемблере
.L1929: { fapb ct=1, dcd=0, fmt=4, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=5, abs=0, disp=0 } .L1613: { loop_mode pfadds,0,sm %b[26], %b[50], %b[68] pshufb,1,sm %b[58], %b[61], %r6, %b[1] pfadds,2,sm %b[10], %b[49], %b[9] pshufb,3,sm %b[43], %b[44], %r5, %b[0] pshufb,4,sm %b[71], %b[57], %r6, %b[72] pfsubs,5,sm %b[10], %b[49], %b[69] } { loop_mode pshufb,0,sm %b[53], %b[54], %r5, %b[10] pfmul_adds,1,sm %b[34], %b[66], %b[33], %b[43] pfmul_rsubs,2,sm %b[34], %b[23], %b[67], %b[44] pshufb,3,sm %b[60], %b[63], %r5, %b[26] pshufb,4,sm %b[71], %b[57], %r5, %b[73] staad,5 %b[72], %aad1[ %aasti3 + _f32s,_lts0 0x8 ] movad,0 area=0, ind=0, am=0, be=0, %b[50] movad,1 area=0, ind=16, am=0, be=0, %b[49] } { loop_mode pfmuls,0,sm %b[1], %b[13], %b[23] pshufb,1,sm %b[11], %b[70], %r6, %b[71] pfsubs,2,sm %b[24], %b[48], %b[53] pshufb,4,sm %b[11], %b[70], %r5, %b[72] staad,5 %b[73], %aad1[ %aasti3 ] incr,5 %aaincr3 movad,0 area=0, ind=8, am=1, be=0, %b[34] movad,1 area=0, ind=24, am=0, be=0, %b[33] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pshufb,0,sm %b[51], %b[52], %r6, %b[11] pfmuls,1,sm %b[7], %b[62], %b[63] staad,2 %b[71], %aad2[ %aasti4 + _f32s,_lts0 0x8 ] pshufb,4,sm %b[41], %b[42], %r6, %b[60] staad,5 %b[72], %aad2[ %aasti4 ] incr,5 %aaincr3 movad,2 area=0, ind=0, am=0, be=0, %b[57] movad,3 area=0, ind=8, am=1, be=0, %b[54] }
Теоретическая скорость: 4 комплексных числа за 4 такта (4/4) = 8 Байт/такт
Замеры скорости
4. lastStage_radix2_simd128_noConj
Модифицированный stage_radix2_simd128_noConj для работы с векторными данными.
Код на Си
void lastStage_radix2_simd128_noConj(int data_count, myComplex2 *data_in, myComplex *data_out, myComplex *coef) { __v2di *xy0_in = (__v2di*)&data_in[0]; __v2di *xy1_in = (__v2di*)&data_in[1]; __v2di *c_in = (__v2di*)coef; __v2di *out_add = (__v2di*)&data_out[0]; __v2di *out_sub = (__v2di*)&data_out[data_count/2]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { __v2di xy0 = xy0_in[2*i]; __v2di xy1 = xy1_in[2*i]; __v2di c = c_in[i]; __v2di x = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0B0A090803020100, 0x0B0A090803020100}); __v2di y = __builtin_e2k_qpshufb(xy1, xy0, (__v2di){0x0F0E0D0C07060504, 0x0F0E0D0C07060504}); __v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy_real = __builtin_e2k_qpfmuls( c, y); __v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y); __v2di cy_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy_real); __v2di cy_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy_imag); __v2di cy = __builtin_e2k_qppermb(cy_ii, cy_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); out_add[i] = __builtin_e2k_qpfadds(x, cy); out_sub[i] = __builtin_e2k_qpfsubs(x, cy); } }
Основной цикл на ассемблере
.L2215: { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=5, abs=0, disp=0 } .L1954: { loop_mode qpfmul_hsubs,0,sm %b[25], %b[28], %r9, %b[16] qpfmul_hadds,1,sm %b[27], %b[28], %r9, %b[1] qpfsubs,2,sm %b[14], %b[42], %b[37] qpshufb,3,sm %b[35], %b[36], %r6, %b[0] qppermb,4,sm %b[11], %b[26], %r7, %b[38] staaqp,5 %b[43], %aad1[ %aasti3 ] incr,5 %aaincr0 movaqp,0 area=0, ind=0, am=0, be=0, %b[30] movaqp,1 area=0, ind=16, am=1, be=0, %b[29] movaqp,3 area=0, ind=0, am=1, be=0, %b[19] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpshufb,0,sm %b[33], %b[34], %r0, %b[26] qpshufb,1,sm %b[23], %b[23], %r5, %b[25] qpfadds,2,sm %b[14], %b[42], %b[11] staaqp,5 %b[17], %aad2[ %aasti4 ] incr,5 %aaincr0 }
Теоретическая скорость: 4 комплексных числа за 2 такта (4/2) = 16 Байт/такт
Замеры скорости
Итоги по lastStage_radix2
При реализации Vector-2 FFT будем использовать lastStage_radix2_simd128_noConj.
lastStage_radix4
Один проход по lastStage_radix4 совершает ту же работу, что 2 прохода по stage_radix2. Поэтому скорость lastStage_radix4 будем умножать на 2 для удобства сравнения с stage_radix2 (этот факт подписан на оси графика и в выводе консоли).
1. lastStage_radix4_etalon
Эталонный вариант для сравнения на корректность.
Код на Си
void lastStage_radix4_etalon(int data_count, myComplex4 *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE) { float *x_real_in = &data_in[0].real[0]; float *x_imag_in = &data_in[0].imag[0]; float *y_real_in = &data_in[0].real[1]; float *y_imag_in = &data_in[0].imag[1]; float *z_real_in = &data_in[0].real[2]; float *z_imag_in = &data_in[0].imag[2]; float *w_real_in = &data_in[0].real[3]; float *w_imag_in = &data_in[0].imag[3]; float *c_real_in = &coefC[0].real; float *c_imag_in = &coefC[0].imag; float *d_real_in = &coefD[0].real; float *d_imag_in = &coefD[0].imag; float *e_real_in = &coefE[0].real; float *e_imag_in = &coefE[0].imag; float *out_0_real = &data_out[0*data_count/4].real; float *out_0_imag = &data_out[0*data_count/4].imag; float *out_1_real = &data_out[1*data_count/4].real; float *out_1_imag = &data_out[1*data_count/4].imag; float *out_2_real = &data_out[2*data_count/4].real; float *out_2_imag = &data_out[2*data_count/4].imag; float *out_3_real = &data_out[3*data_count/4].real; float *out_3_imag = &data_out[3*data_count/4].imag; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { float x_real = x_real_in[8*i]; float x_imag = x_imag_in[8*i]; float y_real = y_real_in[8*i]; float y_imag = y_imag_in[8*i]; float z_real = z_real_in[8*i]; float z_imag = z_imag_in[8*i]; float w_real = w_real_in[8*i]; float w_imag = w_imag_in[8*i]; float c_real = c_real_in[2*i]; float c_imag = c_imag_in[2*i]; float d_real = d_real_in[2*i]; float d_imag = d_imag_in[2*i]; float e_real = e_real_in[2*i]; float e_imag = e_imag_in[2*i]; float cy_real = c_real*y_real - c_imag*y_imag; float cy_imag = c_real*y_imag + c_imag*y_real; float dz_real = d_real*z_real - d_imag*z_imag; float dz_imag = d_real*z_imag + d_imag*z_real; float ew_real = e_real*w_real - e_imag*w_imag; float ew_imag = e_real*w_imag + e_imag*w_real; float add02_real = x_real + dz_real; float add02_imag = x_imag + dz_imag; float sub02_real = x_real - dz_real; float sub02_imag = x_imag - dz_imag; float add13_real = cy_real + ew_real; float add13_imag = cy_imag + ew_imag; float sub13_real = cy_real - ew_real; float sub13_imag = cy_imag - ew_imag; float sub13i_real = -sub13_imag; float sub13i_imag = sub13_real; out_0_real[2*i] = add02_real + add13_real; out_0_imag[2*i] = add02_imag + add13_imag; out_1_real[2*i] = sub02_real - sub13i_real; out_1_imag[2*i] = sub02_imag - sub13i_imag; out_2_real[2*i] = add02_real - add13_real; out_2_imag[2*i] = add02_imag - add13_imag; out_3_real[2*i] = sub02_real + sub13i_real; out_3_imag[2*i] = sub02_imag + sub13i_imag; } }
Основной цикл на ассемблере
.L973: { fapb ct=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=4, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=5, abs=0, disp=0 } { fapb ct=0, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=3, asz=3, abs=8, disp=0 } { fapb ct=1, dcd=0, fmt=3, mrng=8, d=0, incr=2, ind=2, asz=4, abs=16, disp=0 } .L394: { loop_mode fmul_adds,0,sm %b[66], %b[72], %b[54], %b[1] fsub_rsubs,1,sm %b[12], %b[71], %b[55], %b[5] fsub_adds,2,sm %b[12], %b[71], %b[55], %b[45] fmuls,4,sm %b[59], %b[58], %b[52] fsubs,5,sm %b[79], %b[75], %b[53] movaw,3 area=0, ind=16, am=0, be=0, %b[0] } { loop_mode fmul_rsubs,0,sm %b[66], %b[60], %b[82], %b[71] fadd_adds,1,sm %b[24], %b[83], %b[90], %b[72] fadd_rsubs,2,sm %b[24], %b[83], %b[90], %b[76] fmuls,4,sm %b[59], %b[70], %b[80] fadds,5,sm %b[79], %b[75], %b[88] movaw,1 area=2, ind=4, am=0, be=0, %b[55] movaw,2 area=0, ind=0, am=0, be=0, %b[12] movaw,3 area=0, ind=12, am=0, be=0, %b[54] } { loop_mode fmul_rsubs,0,sm %b[42], %b[17], %b[84], %b[75] fmul_rsubs,1,sm %b[32], %b[67], %b[85], %b[79] staaw,2 %b[36], %aad3[ %aasti7 ] fmuls,3,sm %b[29], %b[46], %b[82] fmuls,4,sm %b[35], %b[23], %b[83] staaw,5 %b[39], %aad1[ %aasti5 ] movaw,0 area=2, ind=0, am=1, be=0, %b[60] movaw,1 area=1, ind=0, am=0, be=0, %b[24] movaw,2 area=0, ind=8, am=0, be=0, %b[59] movaw,3 area=0, ind=28, am=0, be=0, %b[66] } { loop_mode fmul_adds,0,sm %b[40], %b[46], %b[86], %b[39] fmul_adds,1,sm %b[32], %b[25], %b[87], %b[67] staaw,2 %b[11], %aad4[ %aasti8 + _f32s,_lts0 0x4 ] fmuls,3,sm %b[27], %b[13], %b[84] fmuls,4,sm %b[35], %b[65], %b[85] staaw,5 %b[51], %aad2[ %aasti6 + _f32s,_lts0 0x4 ] movaw,0 area=1, ind=4, am=1, be=0, %b[29] movaw,1 area=0, ind=0, am=0, be=0, %b[36] movaw,2 area=0, ind=24, am=0, be=0, %b[17] movaw,3 area=0, ind=20, am=0, be=0, %b[42] } { loop_mode fsub_rsubs,0,sm %b[22], %b[81], %b[48], %b[32] fsub_adds,1,sm %b[22], %b[81], %b[48], %b[35] staaw,2 %b[7], %aad3[ %aasti7 + _f32s,_lts0 0x4 ] incr,2 %aaincr3 fsubs,4,sm %b[3], %b[43], %b[46] staaw,5 %b[47], %aad1[ %aasti5 + _f32s,_lts0 0x4 ] incr,5 %aaincr3 movaw,1 area=0, ind=4, am=1, be=0, %b[25] movaw,3 area=0, ind=4, am=1, be=0, %b[11] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END fadd_adds,0,sm %b[10], %b[69], %b[50], %b[7] fadd_rsubs,1,sm %b[10], %b[69], %b[50], %b[47] staaw,2 %b[74], %aad4[ %aasti8 ] incr,2 %aaincr3 fadds,4,sm %b[43], %b[3], %b[48] staaw,5 %b[78], %aad2[ %aasti6 ] incr,5 %aaincr3 }
Теоретическая скорость: 4 комплексных числа за 6 тактов (4/6) = 5.33 Байт/такт
Двойная теоретическая скорость: 10.67 Байт/такт
Замеры скорости
2. lastStage_radix4_etalon_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью директивы unroll.
Код на Си
void lastStage_radix4_etalon_unroll2(int data_count, myComplex4 *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE) { float *x_real_in = &data_in[0].real[0]; float *x_imag_in = &data_in[0].imag[0]; float *y_real_in = &data_in[0].real[1]; float *y_imag_in = &data_in[0].imag[1]; float *z_real_in = &data_in[0].real[2]; float *z_imag_in = &data_in[0].imag[2]; float *w_real_in = &data_in[0].real[3]; float *w_imag_in = &data_in[0].imag[3]; float *c_real_in = &coefC[0].real; float *c_imag_in = &coefC[0].imag; float *d_real_in = &coefD[0].real; float *d_imag_in = &coefD[0].imag; float *e_real_in = &coefE[0].real; float *e_imag_in = &coefE[0].imag; float *out_0_real = &data_out[0*data_count/4].real; float *out_0_imag = &data_out[0*data_count/4].imag; float *out_1_real = &data_out[1*data_count/4].real; float *out_1_imag = &data_out[1*data_count/4].imag; float *out_2_real = &data_out[2*data_count/4].real; float *out_2_imag = &data_out[2*data_count/4].imag; float *out_3_real = &data_out[3*data_count/4].real; float *out_3_imag = &data_out[3*data_count/4].imag; #pragma ivdep #pragma unroll(2) #pragma prefetch for(int64_t i = 0; i < data_count/4; ++i) { float x_real = x_real_in[8*i]; float x_imag = x_imag_in[8*i]; float y_real = y_real_in[8*i]; float y_imag = y_imag_in[8*i]; float z_real = z_real_in[8*i]; float z_imag = z_imag_in[8*i]; float w_real = w_real_in[8*i]; float w_imag = w_imag_in[8*i]; float c_real = c_real_in[2*i]; float c_imag = c_imag_in[2*i]; float d_real = d_real_in[2*i]; float d_imag = d_imag_in[2*i]; float e_real = e_real_in[2*i]; float e_imag = e_imag_in[2*i]; float cy_real = c_real*y_real - c_imag*y_imag; float cy_imag = c_real*y_imag + c_imag*y_real; float dz_real = d_real*z_real - d_imag*z_imag; float dz_imag = d_real*z_imag + d_imag*z_real; float ew_real = e_real*w_real - e_imag*w_imag; float ew_imag = e_real*w_imag + e_imag*w_real; float add02_real = x_real + dz_real; float add02_imag = x_imag + dz_imag; float sub02_real = x_real - dz_real; float sub02_imag = x_imag - dz_imag; float add13_real = cy_real + ew_real; float add13_imag = cy_imag + ew_imag; float sub13_real = cy_real - ew_real; float sub13_imag = cy_imag - ew_imag; float sub13i_real = -sub13_imag; float sub13i_imag = sub13_real; out_0_real[2*i] = add02_real + add13_real; out_0_imag[2*i] = add02_imag + add13_imag; out_1_real[2*i] = sub02_real - sub13i_real; out_1_imag[2*i] = sub02_imag - sub13i_imag; out_2_real[2*i] = add02_real - add13_real; out_2_imag[2*i] = add02_imag - add13_imag; out_3_real[2*i] = sub02_real + sub13i_real; out_3_imag[2*i] = sub02_imag + sub13i_imag; } }
Основной цикл на ассемблере
.L2296: { fapb ct=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=3, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=4, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=3, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=4, mrng=16, d=0, incr=2, ind=2, asz=4, abs=16, disp=0 } .L1081: { loop_mode pfmul_rsubs,0,sm %g17, %g18, %g19, %b[62] pshufb,1,sm %b[90], %b[87], %r0, %g16 pfmul_adds,2,sm %g17, %b[101], %b[117], %b[73] insfd,3,sm %b[107], %r7, %b[98], %b[27] insfd,4,sm %b[49], %r7, %b[97], %b[49] pfmuls,5,sm %b[91], %b[29], %b[117] movad,0 area=1, ind=0, am=0, be=0, %b[1] movad,1 area=1, ind=8, am=1, be=0, %b[0] } { loop_mode pfsub_adds,0,sm %b[118], %b[64], %b[108], %b[85] pfmuls,1,sm %b[115], %b[99], %g19 pfsub_adds,2,sm %b[119], %b[75], %g20, %b[88] insfd,3,sm %b[32], %r7, %b[37], %b[92] insfd,4,sm %b[78], %r7, %b[68], %b[81] pfadds,5,sm %b[28], %b[33], %b[91] movad,0 area=2, ind=0, am=0, be=0, %b[78] movad,1 area=2, ind=8, am=1, be=0, %b[68] movad,2 area=1, ind=0, am=0, be=0, %b[37] movad,3 area=1, ind=8, am=1, be=0, %b[32] } { loop_mode pfadd_rsubs,0,sm %b[118], %b[64], %b[93], %b[98] pfsubs,1,sm %b[55], %b[44], %b[106] pfadd_rsubs,2,sm %b[119], %b[75], %b[113], %b[102] insfd,3,sm %b[50], %r7, %b[61], %b[97] insfd,4,sm %b[15], %r7, %b[14], %g18 pfmuls,5,sm %b[114], %b[81], %b[101] movaw,0 area=0, ind=0, am=0, be=0, %b[15] movaw,1 area=0, ind=16, am=0, be=0, %b[61] movaw,2 area=0, ind=0, am=0, be=0, %b[50] movaw,3 area=0, ind=16, am=0, be=0, %b[14] } { loop_mode pfsub_rsubs,0,sm %b[118], %b[64], %b[108], %b[107] pfadds,1,sm %b[44], %b[55], %b[111] pfsub_rsubs,2,sm %b[119], %b[75], %g20, %b[108] insfd,3,sm %b[6], %r7, %b[7], %g22 pshufb,4,sm %b[104], %b[100], %r0, %g21 pfmuls,5,sm %b[115], %g18, %b[115] movaw,0 area=0, ind=8, am=0, be=0, %b[6] movaw,1 area=0, ind=24, am=0, be=0, %b[55] movaw,2 area=0, ind=8, am=0, be=0, %b[7] movaw,3 area=0, ind=24, am=0, be=0, %b[44] } { loop_mode pfadd_adds,0,sm %b[118], %b[64], %b[93], %b[75] pshufb,1,sm %b[38], %b[43], %r0, %b[113] pfadd_adds,2,sm %b[119], %b[75], %b[113], %b[84] insfd,3,sm %b[74], %r7, %b[84], %g24 pshufb,4,sm %b[110], %b[109], %r0, %g23 pfsubs,5,sm %b[28], %b[33], %g20 movaw,0 area=0, ind=20, am=0, be=0, %b[33] movaw,1 area=0, ind=4, am=0, be=0, %b[64] movaw,2 area=0, ind=20, am=0, be=0, %b[28] movaw,3 area=0, ind=4, am=0, be=0, %b[74] } { loop_mode pfmul_adds,0,sm %g22, %b[94], %b[103], %b[40] pshufb,1,sm %b[2], %b[3], %r0, %b[112] staad,2 %g16, %aad1[ %aasti5 + _f32s,_lts0 0x8 ] insfd,3,sm %b[40], %r7, %b[45], %g17 pshufb,4,sm %b[86], %b[77], %r0, %g16 staad,5 %g23, %aad3[ %aasti7 + _f32s,_lts0 0x8 ] movaw,0 area=0, ind=12, am=1, be=0, %b[94] movaw,1 area=0, ind=28, am=0, be=0, %b[93] movaw,2 area=0, ind=12, am=1, be=0, %b[103] movaw,3 area=0, ind=28, am=0, be=0, %b[45] } { loop_mode pfmul_adds,0,sm %g24, %b[51], %b[117], %b[51] pshufb,1,sm %b[70], %b[80], %r0, %b[87] staad,2 %g21, %aad2[ %aasti6 + _f32s,_lts0 0x8 ] insfd,3,sm %b[90], %r7, %b[87], %b[117] insfd,4,sm %b[110], %r7, %b[109], %g16 staad,5 %g16, %aad4[ %aasti8 + _f32s,_lts0 0x8 ] } { loop_mode pfmul_rsubs,0,sm %g24, %b[29], %b[116], %b[29] pfmuls,1,sm %b[114], %b[92], %b[77] staad,2 %b[117], %aad1[ %aasti5 ] incr,2 %aaincr3 insfd,3,sm %b[86], %r7, %b[77], %b[114] insfd,4,sm %b[104], %r7, %b[100], %b[116] staad,5 %g16, %aad3[ %aasti7 ] incr,5 %aaincr3 } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END pfmul_rsubs,0,sm %g22, %b[83], %b[79], %b[24] pfmuls,1,sm %b[89], %b[49], %b[114] staad,2 %b[114], %aad4[ %aasti8 ] incr,2 %aaincr3 insfd,3,sm %b[60], %r7, %b[25], %b[116] insfd,4,sm %b[24], %r7, %b[71], %b[117] staad,5 %b[116], %aad2[ %aasti6 ] incr,5 %aaincr3 }
Теоретическая скорость: 8 комплексных чисел за 9 тактов (8/9) = 7.11 Байт/такт
Двойная теоретическая скорость: 14.22 Байт/такт
Замеры скорости
3. lastStage_radix4_simd128_noConj
Модифицированный stage_radix4_simd128_noConj для работы с векторными данными.
Код на Си
void lastStage_radix4_simd128_noConj(int data_count, myComplex4 *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE) { __v2di *xyzw0_real_in = (__v2di*)&data_in[0].real; __v2di *xyzw0_imag_in = (__v2di*)&data_in[0].imag; __v2di *xyzw1_real_in = (__v2di*)&data_in[1].real; __v2di *xyzw1_imag_in = (__v2di*)&data_in[1].imag; __v2di *c_in = (__v2di*)coefC; __v2di *d_in = (__v2di*)coefD; __v2di *e_in = (__v2di*)coefE; __v2di *out_0 = (__v2di*)&data_out[0*data_count/4]; __v2di *out_1 = (__v2di*)&data_out[1*data_count/4]; __v2di *out_2 = (__v2di*)&data_out[2*data_count/4]; __v2di *out_3 = (__v2di*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(1) #pragma prefetch for(int64_t i = 0; i < data_count/8; ++i) { __v2di xyzw0_real = xyzw0_real_in[4*i]; __v2di xyzw0_imag = xyzw0_imag_in[4*i]; __v2di xyzw1_real = xyzw1_real_in[4*i]; __v2di xyzw1_imag = xyzw1_imag_in[4*i]; __v2di c = c_in[i]; __v2di d = d_in[i]; __v2di e = e_in[i]; __v2di xy_real = __builtin_e2k_qpshufb(xyzw1_real, xyzw0_real, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di zw_real = __builtin_e2k_qpshufb(xyzw1_real, xyzw0_real, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di xy_imag = __builtin_e2k_qpshufb(xyzw1_imag, xyzw0_imag, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di zw_imag = __builtin_e2k_qpshufb(xyzw1_imag, xyzw0_imag, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x = __builtin_e2k_qppermb(xy_imag, xy_real, (__v2di){0x1312111003020100, 0x1B1A19180B0A0908}); __v2di y = __builtin_e2k_qppermb(xy_imag, xy_real, (__v2di){0x1716151407060504, 0x1F1E1D1C0F0E0D0C}); __v2di z = __builtin_e2k_qppermb(zw_imag, zw_real, (__v2di){0x1312111003020100, 0x1B1A19180B0A0908}); __v2di w = __builtin_e2k_qppermb(zw_imag, zw_real, (__v2di){0x1716151407060504, 0x1F1E1D1C0F0E0D0C}); __v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d = __builtin_e2k_qpshufb(d, d, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e = __builtin_e2k_qpshufb(e, e, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy_real = __builtin_e2k_qpfmuls( c, y); __v2di dz_real = __builtin_e2k_qpfmuls( d, z); __v2di ew_real = __builtin_e2k_qpfmuls( e, w); __v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y); __v2di dz_imag = __builtin_e2k_qpfmuls(swap_d, z); __v2di ew_imag = __builtin_e2k_qpfmuls(swap_e, w); __v2di cy_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy_real); __v2di dz_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz_real); __v2di ew_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew_real); __v2di cy_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy_imag); __v2di dz_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz_imag); __v2di ew_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew_imag); __v2di cy = __builtin_e2k_qppermb(cy_ii, cy_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz = __builtin_e2k_qppermb(dz_ii, dz_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew = __builtin_e2k_qppermb(ew_ii, ew_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di add02 = __builtin_e2k_qpfadds( x, dz); __v2di sub02 = __builtin_e2k_qpfsubs( x, dz); __v2di add13 = __builtin_e2k_qpfadds(cy, ew); __v2di sub13 = __builtin_e2k_qpfsubs(cy, ew); //__v2di conj_sub13 = __builtin_e2k_qpxor(sub13, (__v2di){1LL<<63, 1LL<<63}); //__v2di sub13i = __builtin_e2k_qpshufb(conj_sub13, conj_sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13 = __builtin_e2k_qpshufb(sub13, sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di sub13i = __builtin_e2k_qpxor(swap_sub13, (__v2di){1LL<<31, 1LL<<31}); out_0[i] = __builtin_e2k_qpfadds(add02, add13); out_1[i] = __builtin_e2k_qpfsubs(sub02, sub13i); out_2[i] = __builtin_e2k_qpfsubs(add02, add13); out_3[i] = __builtin_e2k_qpfadds(sub02, sub13i); } }
Основной цикл на ассемблере
.L2983: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=4, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=4, asz=3, abs=8, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=3, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=0, ind=2, asz=4, abs=16, disp=0 } .L2350: { loop_mode qpfmul_hsubs,0,sm %b[21], %b[49], %r14, %b[4] qpshufb,1,sm %b[21], %b[21], %r10, %b[11] qpfsub_rsubs,2,sm %b[42], %b[56], %b[54], %b[1] qppermb,4,sm %b[24], %b[25], %r12, %b[9] qpfadds,5,sm %b[48], %b[45], %b[0] } { loop_mode qpfmul_hadds,0,sm %b[11], %b[49], %r14, %b[21] qppermb,1,sm %b[51], %b[55], %r12, %b[33] qpfsub_adds,2,sm %b[42], %b[56], %b[54], %b[12] qpshufb,4,sm %b[46], %b[46], %r10, %b[32] qpfsubs,5,sm %b[48], %b[45], %b[16] } { loop_mode qppermb,1,sm %b[26], %b[27], %r0, %b[34] qpshufb,3,sm %b[39], %b[39], %r10, %b[48] qppermb,4,sm %b[53], %b[57], %r0, %b[45] staaqp,5 %b[37], %aad4[ %aasti8 ] incr,5 %aaincr0 movaqp,1 area=1, ind=0, am=1, be=0, %b[42] movaqp,3 area=1, ind=0, am=1, be=0, %b[11] } { loop_mode qpfmul_hsubs,0,sm %b[41], %b[33], %r14, %b[27] qppermb,1,sm %b[23], %b[6], %r9, %b[54] qpfmul_hadds,2,sm %b[50], %b[33], %r14, %b[26] qpshufb,3,sm %b[52], %b[59], %r11, %b[53] qpshufb,4,sm %b[43], %b[58], %r11, %b[49] staaqp,5 %b[22], %aad2[ %aasti6 ] incr,5 %aaincr0 movaqp,1 area=2, ind=0, am=1, be=0, %b[37] } { loop_mode qpfmul_hsubs,0,sm %b[46], %b[9], %r14, %b[6] qpshufb,1,sm %b[18], %b[18], %r10, %b[60] qpfadd_adds,2,sm %b[40], %b[54], %b[2], %b[33] qpshufb,3,sm %b[52], %b[59], %r13, %b[23] qpshufb,4,sm %b[43], %b[58], %r13, %b[22] staaqp,5 %b[3], %aad3[ %aasti7 ] incr,5 %aaincr0 movaqp,0 area=0, ind=0, am=0, be=0, %b[57] movaqp,1 area=0, ind=16, am=1, be=0, %b[56] movaqp,2 area=0, ind=0, am=0, be=0, %b[50] movaqp,3 area=0, ind=16, am=1, be=0, %b[41] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfmul_hadds,0,sm %b[32], %b[9], %r14, %b[3] qpxor,1,sm %b[60], %r7, %b[52] qpfadd_rsubs,2,sm %b[40], %b[54], %b[2], %b[18] qppermb,3,sm %b[30], %b[31], %r9, %b[43] qppermb,4,sm %b[7], %b[10], %r9, %b[46] staaqp,5 %b[14], %aad1[ %aasti5 ] incr,5 %aaincr0 }
Теоретическая скорость: 8 комплексных чисел за 6 тактов (8/6) = 10.67 Байт/такт
Двойная теоретическая скорость: 21.33 Байт/такт
Замеры скорости
4. lastStage_radix4_simd128_noConj_unroll2
Здесь происходит раскрутка цикла в 2 раза с помощью директивы unroll.
Код на Си
void lastStage_radix4_simd128_noConj_unroll2(int data_count, myComplex4 *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE) { __v2di *xyzw0_real_in = (__v2di*)&data_in[0].real; __v2di *xyzw0_imag_in = (__v2di*)&data_in[0].imag; __v2di *xyzw1_real_in = (__v2di*)&data_in[1].real; __v2di *xyzw1_imag_in = (__v2di*)&data_in[1].imag; __v2di *c_in = (__v2di*)coefC; __v2di *d_in = (__v2di*)coefD; __v2di *e_in = (__v2di*)coefE; __v2di *out_0 = (__v2di*)&data_out[0*data_count/4]; __v2di *out_1 = (__v2di*)&data_out[1*data_count/4]; __v2di *out_2 = (__v2di*)&data_out[2*data_count/4]; __v2di *out_3 = (__v2di*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(2) #pragma prefetch for(int64_t i = 0; i < data_count/8; ++i) { __v2di xyzw0_real = xyzw0_real_in[4*i]; __v2di xyzw0_imag = xyzw0_imag_in[4*i]; __v2di xyzw1_real = xyzw1_real_in[4*i]; __v2di xyzw1_imag = xyzw1_imag_in[4*i]; __v2di c = c_in[i]; __v2di d = d_in[i]; __v2di e = e_in[i]; __v2di xy_real = __builtin_e2k_qpshufb(xyzw1_real, xyzw0_real, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di zw_real = __builtin_e2k_qpshufb(xyzw1_real, xyzw0_real, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di xy_imag = __builtin_e2k_qpshufb(xyzw1_imag, xyzw0_imag, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di zw_imag = __builtin_e2k_qpshufb(xyzw1_imag, xyzw0_imag, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x = __builtin_e2k_qppermb(xy_imag, xy_real, (__v2di){0x1312111003020100, 0x1B1A19180B0A0908}); __v2di y = __builtin_e2k_qppermb(xy_imag, xy_real, (__v2di){0x1716151407060504, 0x1F1E1D1C0F0E0D0C}); __v2di z = __builtin_e2k_qppermb(zw_imag, zw_real, (__v2di){0x1312111003020100, 0x1B1A19180B0A0908}); __v2di w = __builtin_e2k_qppermb(zw_imag, zw_real, (__v2di){0x1716151407060504, 0x1F1E1D1C0F0E0D0C}); __v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d = __builtin_e2k_qpshufb(d, d, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e = __builtin_e2k_qpshufb(e, e, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy_real = __builtin_e2k_qpfmuls( c, y); __v2di dz_real = __builtin_e2k_qpfmuls( d, z); __v2di ew_real = __builtin_e2k_qpfmuls( e, w); __v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y); __v2di dz_imag = __builtin_e2k_qpfmuls(swap_d, z); __v2di ew_imag = __builtin_e2k_qpfmuls(swap_e, w); __v2di cy_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy_real); __v2di dz_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz_real); __v2di ew_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew_real); __v2di cy_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy_imag); __v2di dz_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz_imag); __v2di ew_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew_imag); __v2di cy = __builtin_e2k_qppermb(cy_ii, cy_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz = __builtin_e2k_qppermb(dz_ii, dz_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew = __builtin_e2k_qppermb(ew_ii, ew_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di add02 = __builtin_e2k_qpfadds( x, dz); __v2di sub02 = __builtin_e2k_qpfsubs( x, dz); __v2di add13 = __builtin_e2k_qpfadds(cy, ew); __v2di sub13 = __builtin_e2k_qpfsubs(cy, ew); //__v2di conj_sub13 = __builtin_e2k_qpxor(sub13, (__v2di){1LL<<63, 1LL<<63}); //__v2di sub13i = __builtin_e2k_qpshufb(conj_sub13, conj_sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13 = __builtin_e2k_qpshufb(sub13, sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di sub13i = __builtin_e2k_qpxor(swap_sub13, (__v2di){1LL<<31, 1LL<<31}); out_0[i] = __builtin_e2k_qpfadds(add02, add13); out_1[i] = __builtin_e2k_qpfsubs(sub02, sub13i); out_2[i] = __builtin_e2k_qpfsubs(add02, add13); out_3[i] = __builtin_e2k_qpfadds(sub02, sub13i); } }
Основной цикл на ассемблере
.L3893: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=3, abs=8, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=4, abs=16, disp=0 } { fapb ct=1, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=3, abs=24, disp=0 } .L3037: { loop_mode qpfsub_adds,0,sm %b[16], %b[71], %b[66], %b[57] qpshufb,1,sm %b[32], %b[32], %r14, %b[71] qpfsub_rsubs,2,sm %b[16], %b[71], %b[66], %b[64] qppermb,3,sm %b[38], %b[42], %r11, %b[0] qppermb,4,sm %b[73], %b[76], %r11, %b[1] qpfadds,5,sm %b[64], %b[57], %b[66] } { loop_mode qpfmul_hsubs,0,sm %b[32], %b[83], %r15, %b[74] qpshufb,1,sm %b[10], %b[10], %r14, %b[73] qpfmul_hadds,2,sm %b[71], %b[83], %r15, %b[71] qpshufb,4,sm %b[21], %b[25], %r9, %b[32] qpfadds,5,sm %b[1], %b[0], %b[16] } { loop_mode qpfmul_hsubs,0,sm %b[31], %b[75], %r15, %b[38] qppermb,1,sm %b[69], %b[34], %r13, %b[31] qpfmul_hadds,2,sm %b[72], %b[75], %r15, %b[34] qpshufb,3,sm %b[19], %b[19], %r14, %b[54] qppermb,4,sm %b[54], %b[55], %r13, %b[42] } { loop_mode qpfmul_hsubs,0,sm %b[10], %b[65], %r15, %g17 qpshufb,1,sm %b[26], %b[26], %r14, %b[65] qpfmul_hadds,2,sm %b[73], %b[65], %r15, %g16 qppermb,3,sm %g16, %g17, %r11, %b[55] qppermb,4,sm %b[62], %b[63], %r11, %b[62] qpfsubs,5,sm %b[3], %b[2], %b[63] } { loop_mode qpfmul_hsubs,0,sm %b[41], %b[82], %r15, %b[61] qppermb,1,sm %b[80], %b[78], %r11, %b[72] qpfmul_hadds,2,sm %b[61], %b[82], %r15, %b[60] qpshufb,3,sm %b[43], %b[46], %r9, %b[67] qppermb,4,sm %b[60], %b[67], %r11, %b[69] qpfsubs,5,sm %b[62], %b[55], %b[2] } { loop_mode qpfmul_hsubs,0,sm %b[26], %b[35], %r15, %b[65] qppermb,1,sm %b[58], %b[70], %r13, %b[3] qpfmul_hadds,2,sm %b[65], %b[35], %r15, %b[58] qpshufb,3,sm %b[29], %b[29], %r14, %b[70] qppermb,4,sm %b[67], %b[32], %r12, %b[73] staaqp,5 %b[77], %aad4[ %aasti8 ] } { loop_mode qpfmul_hsubs,0,sm %b[19], %b[44], %r15, %b[76] qppermb,1,sm %b[56], %b[68], %r13, %b[10] qpfadd_adds,2,sm %b[7], %b[72], %b[66], %b[75] qpshufb,3,sm %b[49], %b[53], %r0, %b[68] qpshufb,4,sm %b[11], %b[52], %r0, %b[56] staaqp,5 %b[79], %aad2[ %aasti6 ] } { loop_mode qpfmul_hadds,0,sm %b[54], %b[44], %r15, %b[78] qpshufb,1,sm %b[4], %b[4], %r14, %b[79] qpfadd_rsubs,2,sm %b[7], %b[72], %b[66], %b[77] qpshufb,3,sm %b[23], %b[27], %r0, %b[66] qpshufb,4,sm %b[45], %b[48], %r0, %b[54] staaqp,5 %b[81], %aad2[ %aasti6 + _f32s,_lts0 0x10 ] incr,5 %aaincr3 movaqp,0 area=3, ind=0, am=0, be=0, %b[4] movaqp,1 area=1, ind=0, am=0, be=0, %b[23] movaqp,3 area=1, ind=0, am=0, be=0, %b[19] } { loop_mode qpfadd_adds,0,sm %b[14], %b[69], %b[18], %g18 qpxor,1,sm %b[79], %r10, %b[82] qpfadd_rsubs,2,sm %b[14], %b[69], %b[18], %b[79] qppermb,3,sm %b[56], %b[68], %r12, %b[80] qppermb,4,sm %b[54], %b[66], %r12, %b[81] staaqp,5 %g18, %aad4[ %aasti8 + _f32s,_lts0 0x10 ] incr,5 %aaincr3 movaqp,0 area=3, ind=16, am=1, be=0, %b[27] movaqp,1 area=2, ind=16, am=0, be=0, %b[26] movaqp,3 area=2, ind=16, am=0, be=0, %b[18] } { loop_mode qpfsub_adds,0,sm %b[7], %b[72], %b[82], %g19 qpshufb,1,sm %b[63], %b[63], %r14, %b[59] staaqp,2 %g19, %aad1[ %aasti5 ] qpshufb,3,sm %b[49], %b[53], %r9, %b[53] qpshufb,4,sm %b[11], %b[52], %r9, %b[52] staaqp,5 %b[59], %aad3[ %aasti7 ] movaqp,0 area=2, ind=0, am=1, be=0, %b[35] movaqp,1 area=1, ind=16, am=1, be=0, %b[44] movaqp,2 area=2, ind=0, am=1, be=0, %b[11] movaqp,3 area=1, ind=16, am=1, be=0, %b[41] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfsub_rsubs,0,sm %b[7], %b[72], %b[82], %b[57] qpxor,1,sm %b[59], %r10, %b[64] staaqp,2 %b[57], %aad1[ %aasti5 + _f32s,_lts0 0x10 ] incr,2 %aaincr3 qpshufb,3,sm %b[39], %b[39], %r14, %b[59] qppermb,4,sm %b[52], %b[53], %r12, %b[63] staaqp,5 %b[64], %aad3[ %aasti7 + _f32s,_lts0 0x10 ] incr,5 %aaincr3 movaqp,0 area=0, ind=0, am=0, be=0, %b[49] movaqp,1 area=0, ind=16, am=1, be=0, %b[48] movaqp,2 area=0, ind=0, am=0, be=0, %b[45] movaqp,3 area=0, ind=16, am=1, be=0, %b[7] }
Теоретическая скорость: 16 комплексных чисел за 11 тактов (16/11) = 11.64 Байт/такт
Двойная теоретическая скорость: 23.27 Байт/такт
Замеры скорости
5. lastStage_radix4_simd128_noConj_unroll3
Здесь происходит раскрутка цикла в 3 раза с помощью директивы unroll.
Код на Си
void lastStage_radix4_simd128_noConj_unroll3(int data_count, myComplex4 *data_in, myComplex *data_out, myComplex *coefC, myComplex *coefD, myComplex *coefE) { __v2di *xyzw0_real_in = (__v2di*)&data_in[0].real; __v2di *xyzw0_imag_in = (__v2di*)&data_in[0].imag; __v2di *xyzw1_real_in = (__v2di*)&data_in[1].real; __v2di *xyzw1_imag_in = (__v2di*)&data_in[1].imag; __v2di *c_in = (__v2di*)coefC; __v2di *d_in = (__v2di*)coefD; __v2di *e_in = (__v2di*)coefE; __v2di *out_0 = (__v2di*)&data_out[0*data_count/4]; __v2di *out_1 = (__v2di*)&data_out[1*data_count/4]; __v2di *out_2 = (__v2di*)&data_out[2*data_count/4]; __v2di *out_3 = (__v2di*)&data_out[3*data_count/4]; #pragma ivdep #pragma unroll(3) #pragma prefetch for(int64_t i = 0; i < data_count/8; ++i) { __v2di xyzw0_real = xyzw0_real_in[4*i]; __v2di xyzw0_imag = xyzw0_imag_in[4*i]; __v2di xyzw1_real = xyzw1_real_in[4*i]; __v2di xyzw1_imag = xyzw1_imag_in[4*i]; __v2di c = c_in[i]; __v2di d = d_in[i]; __v2di e = e_in[i]; __v2di xy_real = __builtin_e2k_qpshufb(xyzw1_real, xyzw0_real, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di zw_real = __builtin_e2k_qpshufb(xyzw1_real, xyzw0_real, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di xy_imag = __builtin_e2k_qpshufb(xyzw1_imag, xyzw0_imag, (__v2di){0x0706050403020100, 0x0706050403020100}); __v2di zw_imag = __builtin_e2k_qpshufb(xyzw1_imag, xyzw0_imag, (__v2di){0x0F0E0D0C0B0A0908, 0x0F0E0D0C0B0A0908}); __v2di x = __builtin_e2k_qppermb(xy_imag, xy_real, (__v2di){0x1312111003020100, 0x1B1A19180B0A0908}); __v2di y = __builtin_e2k_qppermb(xy_imag, xy_real, (__v2di){0x1716151407060504, 0x1F1E1D1C0F0E0D0C}); __v2di z = __builtin_e2k_qppermb(zw_imag, zw_real, (__v2di){0x1312111003020100, 0x1B1A19180B0A0908}); __v2di w = __builtin_e2k_qppermb(zw_imag, zw_real, (__v2di){0x1716151407060504, 0x1F1E1D1C0F0E0D0C}); __v2di swap_c = __builtin_e2k_qpshufb(c, c, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_d = __builtin_e2k_qpshufb(d, d, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_e = __builtin_e2k_qpshufb(e, e, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di cy_real = __builtin_e2k_qpfmuls( c, y); __v2di dz_real = __builtin_e2k_qpfmuls( d, z); __v2di ew_real = __builtin_e2k_qpfmuls( e, w); __v2di cy_imag = __builtin_e2k_qpfmuls(swap_c, y); __v2di dz_imag = __builtin_e2k_qpfmuls(swap_d, z); __v2di ew_imag = __builtin_e2k_qpfmuls(swap_e, w); __v2di cy_rr = __builtin_e2k_qpfhsubs((__v2di){0}, cy_real); __v2di dz_rr = __builtin_e2k_qpfhsubs((__v2di){0}, dz_real); __v2di ew_rr = __builtin_e2k_qpfhsubs((__v2di){0}, ew_real); __v2di cy_ii = __builtin_e2k_qpfhadds((__v2di){0}, cy_imag); __v2di dz_ii = __builtin_e2k_qpfhadds((__v2di){0}, dz_imag); __v2di ew_ii = __builtin_e2k_qpfhadds((__v2di){0}, ew_imag); __v2di cy = __builtin_e2k_qppermb(cy_ii, cy_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di dz = __builtin_e2k_qppermb(dz_ii, dz_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di ew = __builtin_e2k_qppermb(ew_ii, ew_rr, (__v2di){0x1B1A19180B0A0908, 0x1F1E1D1C0F0E0D0C}); __v2di add02 = __builtin_e2k_qpfadds( x, dz); __v2di sub02 = __builtin_e2k_qpfsubs( x, dz); __v2di add13 = __builtin_e2k_qpfadds(cy, ew); __v2di sub13 = __builtin_e2k_qpfsubs(cy, ew); //__v2di conj_sub13 = __builtin_e2k_qpxor(sub13, (__v2di){1LL<<63, 1LL<<63}); //__v2di sub13i = __builtin_e2k_qpshufb(conj_sub13, conj_sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di swap_sub13 = __builtin_e2k_qpshufb(sub13, sub13, (__v2di){0x0302010007060504, 0x0B0A09080F0E0D0C}); __v2di sub13i = __builtin_e2k_qpxor(swap_sub13, (__v2di){1LL<<31, 1LL<<31}); out_0[i] = __builtin_e2k_qpfadds(add02, add13); out_1[i] = __builtin_e2k_qpfsubs(sub02, sub13i); out_2[i] = __builtin_e2k_qpfsubs(add02, add13); out_3[i] = __builtin_e2k_qpfadds(sub02, sub13i); } }
Основной цикл на ассемблере
.L5077: { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=0, disp=32 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=64 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=4, disp=96 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=128 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=1, ind=1, asz=2, abs=8, disp=160 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=4, asz=2, abs=12, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=3, asz=2, abs=12, disp=0 } { fapb ct=0, dcd=0, fmt=5, mrng=0, d=0, incr=2, ind=2, asz=3, abs=16, disp=0 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=4, asz=3, abs=16, disp=32 } { fapb ct=1, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=3, asz=3, abs=24, disp=32 fapb dpl=0, dcd=0, fmt=5, mrng=16, d=0, incr=2, ind=2, asz=3, abs=24, disp=32 } .L3947: { loop_mode qpfmul_hadds,0,sm %b[93], %b[76], %r15, %b[75] qpshufb,1,sm %b[85], %b[85], %r13, %b[91] qpfadd_adds,2,sm %b[32], %b[87], %b[75], %b[76] qppermb,3,sm %b[18], %b[17], %r10, %b[86] qppermb,4,sm %b[91], %b[86], %r10, %b[85] qpfsubs,5,sm %b[88], %b[89], %b[0] } { loop_mode qpfmul_hsubs,0,sm %b[52], %b[94], %r15, %b[93] qpxor,1,sm %b[91], %r12, %b[105] qpfadd_adds,2,sm %b[80], %b[77], %b[83], %b[91] qppermb,3,sm %b[44], %b[69], %r11, %b[103] qppermb,4,sm %b[22], %b[27], %r11, %b[104] qpfsubs,5,sm %b[85], %b[86], %b[83] } { loop_mode qpfmul_hadds,0,sm %b[97], %b[94], %r15, %b[85] qpxor,1,sm %b[102], %r12, %b[97] qpfsub_adds,2,sm %b[80], %b[77], %b[105], %b[82] qppermb,3,sm %b[101], %b[100], %r10, %b[94] qppermb,4,sm %b[82], %b[81], %r10, %b[86] qpfadds,5,sm %b[85], %b[86], %b[81] } { loop_mode qpfmul_hadds,0,sm %b[95], %b[72], %r15, %b[100] qpshufb,1,sm %b[2], %b[2], %r13, %b[101] qpfsub_adds,2,sm %b[73], %b[78], %b[97], %b[95] qpshufb,3,sm %b[39], %b[65], %r0, %b[1] qpshufb,4,sm %b[68], %b[60], %r0, %b[2] qpfsubs,5,sm %b[86], %b[94], %b[72] movaqp,1 area=4, ind=16, am=0, be=0, %b[8] movaqp,3 area=5, ind=0, am=1, be=0, %b[7] } { loop_mode qpfsub_rsubs,0,sm %b[80], %b[77], %b[105], %b[78] qpxor,1,sm %b[101], %r12, %b[88] qpfsub_rsubs,2,sm %b[73], %b[78], %b[97], %b[77] qpshufb,3,sm %b[99], %b[98], %r0, %b[18] qppermb,4,sm %b[2], %b[1], %r11, %b[80] qpfadds,5,sm %b[88], %b[89], %b[73] movaqp,0 area=5, ind=0, am=1, be=0, %b[17] movaqp,1 area=1, ind=0, am=0, be=0, %b[23] movaqp,3 area=1, ind=0, am=0, be=0, %b[26] } { loop_mode qpfsub_adds,0,sm %b[32], %b[87], %b[88], %b[87] qpshufb,1,sm %b[61], %b[61], %r13, %b[89] qpfsub_rsubs,2,sm %b[32], %b[87], %b[88], %b[86] qpshufb,3,sm %b[99], %b[98], %r9, %b[39] qpshufb,4,sm %b[96], %b[92], %r9, %b[45] qpfadds,5,sm %b[86], %b[94], %b[88] movaqp,0 area=4, ind=0, am=1, be=0, %b[31] movaqp,1 area=1, ind=16, am=1, be=0, %b[32] movaqp,2 area=4, ind=0, am=1, be=0, %b[40] movaqp,3 area=1, ind=16, am=1, be=0, %b[36] } { loop_mode qpfmul_hsubs,0,sm %b[61], %b[104], %r15, %b[84] qpshufb,1,sm %b[35], %b[35], %r13, %b[94] qpfmul_hadds,2,sm %b[89], %b[104], %r15, %b[89] qpshufb,3,sm %b[96], %b[92], %r0, %b[60] qppermb,4,sm %b[45], %b[39], %r11, %b[92] staaqp,5 %b[84], %aad4[ %aasti8 ] movaqp,0 area=3, ind=0, am=0, be=0, %b[52] movaqp,1 area=3, ind=16, am=1, be=0, %b[57] movaqp,2 area=3, ind=0, am=0, be=0, %b[46] movaqp,3 area=3, ind=16, am=1, be=0, %b[51] } { loop_mode qpfmul_hsubs,0,sm %b[35], %b[103], %r15, %b[98] qpshufb,1,sm %b[56], %b[56], %r13, %b[103] qpfmul_hadds,2,sm %b[94], %b[103], %r15, %b[99] qpshufb,3,sm %b[42], %b[42], %r13, %b[101] qppermb,4,sm %b[60], %b[18], %r11, %b[102] staaqp,5 %b[90], %aad2[ %aasti6 ] movaqp,0 area=2, ind=16, am=1, be=0, %b[90] movaqp,1 area=2, ind=0, am=0, be=0, %b[96] movaqp,2 area=2, ind=16, am=1, be=0, %b[94] movaqp,3 area=2, ind=0, am=0, be=0, %b[97] } { loop_mode qpfmul_hsubs,0,sm %b[56], %b[80], %r15, %b[79] qppermb,1,sm %b[11], %b[12], %r11, %b[103] qpfmul_hadds,2,sm %b[103], %b[80], %r15, %b[80] qpshufb,3,sm %b[7], %b[7], %r13, %b[65] qpshufb,4,sm %b[8], %b[8], %r13, %b[68] staaqp,5 %b[79], %aad2[ %aasti6 + _f32s,_lts0 0x20 ] movaqp,0 area=0, ind=16, am=1, be=0, %b[56] movaqp,1 area=0, ind=0, am=0, be=0, %b[61] movaqp,2 area=0, ind=16, am=1, be=0, %b[64] movaqp,3 area=0, ind=0, am=0, be=0, %b[35] } { loop_mode qpfmul_hsubs,0,sm %b[10], %b[103], %r15, %b[13] qppermb,1,sm %b[47], %b[41], %r14, %b[70] qpfmul_hadds,2,sm %b[70], %b[103], %r15, %b[14] qpshufb,3,sm %b[26], %b[23], %r9, %b[10] qppermb,4,sm %b[13], %b[14], %r14, %b[74] staaqp,5 %b[74], %aad2[ %aasti6 + _f32s,_lts0 0x10 ] incr,5 %aaincr3 } { loop_mode qpfmul_hsubs,0,sm %b[9], %b[92], %r15, %b[41] qppermb,1,sm %b[85], %b[93], %r10, %b[76] qpfmul_hadds,2,sm %b[67], %b[92], %r15, %b[47] qpshufb,3,sm %b[36], %b[32], %r9, %b[9] qpshufb,4,sm %b[37], %b[63], %r9, %b[67] staaqp,5 %b[76], %aad4[ %aasti8 + _f32s,_lts0 0x20 ] } { loop_mode qpfmul_hsubs,0,sm %b[42], %b[102], %r15, %b[3] qppermb,1,sm %b[4], %b[3], %r14, %b[71] qpfmul_hadds,2,sm %b[101], %b[102], %r15, %b[4] qpshufb,3,sm %b[66], %b[58], %r9, %b[42] qppermb,4,sm %b[75], %b[71], %r10, %b[75] staaqp,5 %b[91], %aad4[ %aasti8 + _f32s,_lts0 0x10 ] incr,5 %aaincr3 } { loop_mode qpfmul_hsubs,0,sm %b[21], %b[70], %r15, %g16 qppermb,1,sm %b[100], %g16, %r10, %b[85] qpfadd_adds,2,sm %b[71], %b[76], %b[88], %b[82] qpshufb,3,sm %b[28], %b[25], %r0, %b[25] qpshufb,4,sm %b[55], %b[55], %r13, %b[91] staaqp,5 %b[82], %aad1[ %aasti5 + _f32s,_lts0 0x10 ] } { loop_mode qpfmul_hsubs,0,sm %b[55], %b[74], %r15, %b[69] qppermb,1,sm %b[62], %b[20], %r14, %b[28] qpfadd_rsubs,2,sm %b[71], %b[76], %b[88], %b[88] qpshufb,3,sm %b[38], %b[34], %r0, %b[20] qppermb,4,sm %b[44], %b[69], %r14, %b[92] staaqp,5 %b[95], %aad1[ %aasti5 ] } { loop_mode qpfadd_rsubs,0,sm %b[30], %b[85], %b[73], %b[77] qppermb,1,sm %b[24], %b[29], %r14, %b[78] staaqp,2 %b[78], %aad3[ %aasti7 + _f32s,_lts0 0x10 ] qpshufb,3,sm %b[21], %b[21], %r13, %b[93] qpshufb,4,sm %b[50], %b[50], %r13, %b[95] staaqp,5 %b[77], %aad3[ %aasti7 ] } { loop_mode alc alcf=1, alct=1 abn abnf=1, abnt=1 ct %ctpr1 ? %NOT_LOOP_END qpfadd_rsubs,0,sm %b[78], %b[75], %b[81], %b[72] qpshufb,1,sm %b[72], %b[72], %r13, %b[100] staaqp,2 %b[87], %aad1[ %aasti5 + _f32s,_lts0 0x20 ] incr,2 %aaincr3 qppermb,3,sm %b[49], %b[43], %r10, %b[87] qppermb,4,sm %b[6], %b[5], %r10, %b[86] staaqp,5 %b[86], %aad3[ %aasti7 + _f32s,_lts0 0x20 ] incr,5 %aaincr3 }
Теоретическая скорость: 24 комплексных числа за 16 тактов (24/16) = 12 Байт/такт
Двойная теоретическая скорость: 24 Байт/такт
Замеры скорости
Итоги по lastStage_radix4
При реализации Vector-4 FFT будем использовать lastStage_radix4_simd128_noConj_unroll2.
Собираем FFT
fft_radix2_vector2
Собираем fft_radix2_vector2 из reverse_radix2_vector2_stream16, всех вариантов stage_radix2_vector2 и lastStage_radix2_simd128_noConj.
fft_radix2_vector4
Собираем fft_radix2_vector4 из reverse_radix2_vector4_stream8, всех вариантов stage_radix2_vector4 и lastStage_radix4_simd128_noConj_unroll2.
fft_radix4_vector2
Собираем fft_radix4_vector2 из reverse_radix4_vector2_stream16, всех вариантов stage_radix4_vector2 и lastStage_radix2_simd128_noConj.
fft_radix4_vector4
Собираем fft_radix4_vector4 из reverse_radix4_vector4_stream16, всех вариантов stage_radix4_vector4 и lastStage_radix4_simd128_noConj_unroll2.
Дальнейшие улучшения имеющегося кода
Что изменить в текущем коде:
Перемежить коэффициенты.
Сейчас коэффициенты c/d/e размещаются в нескольких массивах.
Нужно разместить их в одном массиве, перемежив между собой.
Это повысит эффективность чтения памяти.
Скорость вне кэша должна вырасти.Заменить типы на векторные для упрощения внешнего вида кода.
Сейчас везде используются интринсики вместе с типамиuint64_tи__v2di.
Нужно заменитьuint64_tи__v2diна__v2sfи__v4sfсоответственно.
Тогда можно будет заменить интринсики на операции «+», «-», «*».
Код станет выглядеть менее громоздко.
Дальнейшие направления создания кода
Что делать дальше (в дополнение к указанному в предыдущей статье):
Рассмотреть все Stage в качестве LastStage.
Сейчас в качестве LastStage рассмотрены только некоторые Stage из прошлой статьи. Для полноты картины нужно рассмотреть остальные варианты Stage. Небольшое ускорение внутри кэша и замедление вне кэша ожидается от вариантов"_readConjSwap".Оптимизация для малых размеров входных данных (как в EML).
На графиках скорости FFT можно заметить выделяющийся угол у версии из EML.
Скорее всего, в EML сделана оптимизация для малых длин массивов (до 64 элементов). Оптимизация может состоять в хранении промежуточных результатов вычислений (между последовательными Stage) прямо в регистрах, а не в памяти. Если размеры входных данных достаточно малы, то такие данные можно целиком разместить в регистрах. В этом случае этап Reverse не нужен. Вместо него нужно просто переименовать регистры в первом Stage.
Заключение
Прямая векторизация проще для написания, понятнее для чтения и работает быстрее, чем тот код, что был написан в прошлой статье. Однако, нельзя сказать, что та работа была сделана напрасно. Ведь LastStage написан на основе кода из прошлой статьи.
Судя по графикам, версия из EML аналогична fft_radix2_vector4 с добавлением перемежения коэффициентов и оптимизации для малых размеров входных данных.
Пока я писал этот код, возникло ощущение, что было бы удобно создать классы myComplex, myComplex2 и myComplex4. Но сейчас думаю, что если заменить myComplex на родной float complex, то классы уже и не нужны.
