Запускаем на телефоне нейронку LLaMa 7B -13B 4 bit от фейсбука с настройкой Alpaca-LoRA

[teff8083]

любопытненько, стоит попробовать... может быть, и в federated режиме в будущем так и будет работать, а пока поиграть

[123Zero]

Прикольная модель, на компьютере запустил, отлично генерирует чушь) Но на Андроиде low memory killer подвёл(

main: seed = 1680000931
llama_model_load: loading model from 'ggml-alpaca-7b-q4.bin' - please wait ...
llama_model_load: ggml ctx size = 6065.34 MB
llama_model_load: memory_size =  2048.00 MB, n_mem = 65536
llama_model_load: loading model part 1/1 from 'ggml-alpaca-7b-q4.bin'
llama_model_load: ............................Killed

[Freeartist85]

Здравствуйте, при компиляции на android столкнулся с проблемой

~/.../alpaca.cpp/build $ cmake --build . --config Release
[ 12%] Building C object CMakeFiles/ggml.dir/ggml.c.o
/data/data/com.termux/files/home/storage/alpaca-llama/alpaca.cpp/ggml.c:1778:35: error: call to undeclared function 'vzip1_s8'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
const int8x8_t vxlt = vzip1_s8(vxls, vxhs);
^
/data/data/com.termux/files/home/storage/alpaca-llama/alpaca.cpp/ggml.c:1778:28: error: initializing 'const int8x8_t' (vector of 8 'int8_t' values) with an expression of incompatible type 'int'
const int8x8_t vxlt = vzip1_s8(vxls, vxhs);
^ ~~~~~~~~~~~~~~~~~~~~
/data/data/com.termux/files/home/storage/alpaca-llama/alpaca.cpp/ggml.c:1779:35: error: call to undeclared function 'vzip2_s8'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
const int8x8_t vxht = vzip2_s8(vxls, vxhs);
^
/data/data/com.termux/files/home/storage/alpaca-llama/alpaca.cpp/ggml.c:1779:28: error: initializing 'const int8x8_t' (vector of 8 'int8_t' values) with an expression of incompatible type 'int'
const int8x8_t vxht = vzip2_s8(vxls, vxhs);
^ ~~~~~~~~~~~~~~~~~~~~
/data/data/com.termux/files/home/storage/alpaca-llama/alpaca.cpp/ggml.c:1798:37: error: call to undeclared function 'vfmaq_f32'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
const float32x4_t vr0 = vfmaq_f32(vy0, vx0, vd);
^
/data/data/com.termux/files/home/storage/alpaca-llama/alpaca.cpp/ggml.c:1798:31: error: initializing 'const float32x4_t' (vector of 4 'float32_t' values) with an expression of incompatible type 'int'
const float32x4_t vr0 = vfmaq_f32(vy0, vx0, vd);
^ ~~~~~~~~~~~~~~~~~~~~~~~
/data/data/com.termux/files/home/storage/alpaca-llama/alpaca.cpp/ggml.c:1799:31: error: initializing 'const float32x4_t' (vector of 4 'float32_t' values) with an expression of incompatible type 'int'
const float32x4_t vr1 = vfmaq_f32(vy1, vx1, vd);
^ ~~~~~~~~~~~~~~~~~~~~~~~
/data/data/com.termux/files/home/storage/alpaca-llama/alpaca.cpp/ggml.c:1800:31: error: initializing 'const float32x4_t' (vector of 4 'float32_t' values) with an expression of incompatible type 'int'
const float32x4_t vr2 = vfmaq_f32(vy2, vx2, vd);
^ ~~~~~~~~~~~~~~~~~~~~~~~
/data/data/com.termux/files/home/storage/alpaca-llama/alpaca.cpp/ggml.c:1801:31: error: initializing 'const float32x4_t' (vector of 4 'float32_t' values) with an expression of incompatible type 'int'
const float32x4_t vr3 = vfmaq_f32(vy3, vx3, vd);
^ ~~~~~~~~~~~~~~~~~~~~~~~
9 errors generated.
make[2]: *** [CMakeFiles/ggml.dir/build.make:76: CMakeFiles/ggml.dir/ggml.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:139: CMakeFiles/ggml.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

[freQuensy23]

Эта же штука не используют все нейроные ускорители, встроенные в процессоры, а грузит все на CPU