mirror of
https://github.com/m5stack/StackFlow.git
synced 2026-05-20 11:32:11 -07:00
dianjixz
GitHub
@@ -14,6 +14,7 @@
|
||||
|
||||
* [特性](#特性)
|
||||
* [Demo](#demo)
|
||||
* [模型列表](#模型列表)
|
||||
* [环境要求](#环境要求)
|
||||
* [编译](#编译)
|
||||
* [安装](#安装)
|
||||
@@ -54,6 +55,53 @@ StackFlow 语音助手的主要工作模式：
|
||||
- [StackFlow yolo 视觉检测](https://github.com/Abandon-ht/ModuleLLM_Development_Guide/tree/dev/ESP32/cpp)
|
||||
- [StackFlow VLM 图片描述](https://github.com/Abandon-ht/ModuleLLM_Development_Guide/tree/dev/ESP32/cpp)
|
||||
|
||||
## 模型列表
|
||||
| 模型名 | 模型类型 | 模型大小 | 模型能力 | 模型配置文件 | 计算单元 |
|
||||
| :----: | :----: | :----: | :----: | :----: | :----: |
|
||||
| [silero-vad](https://github.com/snakers4/silero-vad) | VAD | 3.3M | 语音活动检测 | [mode_silero-vad.json](projects/llm_framework/main_vad/mode_silero-vad.json) | CPU |
|
||||
| [sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01](https://github.com/k2-fsa/sherpa-onnx/releases/download/kws-models/sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01.tar.bz2) | KWS | 6.4M | 关键词识别 | [mode_sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01.json](projects/llm_framework/main_kws/mode_sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01.json) | CPU |
|
||||
| [sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01](https://github.com/k2-fsa/sherpa-onnx/releases/download/kws-models/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01.tar.bz2) | KWS | 5.7M | 关键词识别 | [mode_sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01.json](projects/llm_framework/main_kws/mode_sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01.json) | CPU |
|
||||
| [sherpa-ncnn-streaming-zipformer-20M-2023-02-17](https://huggingface.co/desh2608/icefall-asr-librispeech-pruned-transducer-stateless7-streaming-small) | ASR | 40M | 语音识别 | [mode_sherpa-ncnn-streaming-zipformer-20M-2023-02-17.json](projects/llm_framework/main_asr/mode_sherpa-ncnn-streaming-zipformer-20M-2023-02-17.json) | CPU |
|
||||
| [sherpa-ncnn-streaming-zipformer-zh-14M-2023-02-23](https://github.com/k2-fsa/icefall/tree/master/egs/librispeech/ASR/pruned_transducer_stateless7_streaming) | ASR | 24M | 语音识别 | [mode_sherpa-ncnn-streaming-zipformer-zh-14M-2023-02-23.json](projects/llm_framework/main_asr/mode_sherpa-ncnn-streaming-zipformer-zh-14M-2023-02-23.json) | CPU |
|
||||
| [whisper-tiny](https://huggingface.co/openai/whisper-tiny) | ASR | 201M | 语音识别 | [mode_whisper-tiny.json](projects/llm_framework/main_whisper/mode_whisper-tiny.json) | NPU |
|
||||
| [whisper-base](https://huggingface.co/openai/whisper-base) | ASR | 309M | 语音识别 | [mode_whisper-base.json](projects/llm_framework/main_whisper/mode_whisper-base.json) | NPU |
|
||||
| [whisper-small](https://huggingface.co/openai/whisper-small) | ASR | 725M | 语音识别 | [mode_whisper-small.json](projects/llm_framework/main_whisper/mode_whisper-small.json) | NPU |
|
||||
| [single-speaker-fast](https://github.com/huakunyang/SummerTTS) | TTS | 77M | 语音生成 | [mode_whisper-tiny.json](projects/llm_framework/main_tts/mode_single-speaker-fast.json) | CPU |
|
||||
| [single-speaker-english-fast](https://github.com/huakunyang/SummerTTS) | TTS | 60M | 语音生成 | [mode_whisper-tiny.json](projects/llm_framework/main_tts/mode_single-speaker-english-fast.json) | CPU |
|
||||
| [melotts-en-au](https://huggingface.co/myshell-ai/MeloTTS-English) | TTS | 102M | 语音生成 | [mode_melotts-en-au.json](projects/llm_framework/main_melotts/mode_melotts-en-au.json) | NPU |
|
||||
| [melotts-en-br](https://huggingface.co/myshell-ai/MeloTTS-English) | TTS | 102M | 语音生成 | [mode_melotts-en-au.json](projects/llm_framework/main_melotts/mode_melotts-en-br.json) | NPU |
|
||||
| [melotts-en-default](https://huggingface.co/myshell-ai/MeloTTS-English) | TTS | 102M | 语音生成 | [mode_melotts-en-india.json](projects/llm_framework/main_melotts/mode_melotts-en-default.json) | NPU |
|
||||
| [melotts-en-us](https://huggingface.co/myshell-ai/MeloTTS-English) | TTS | 102M | 语音生成 | [mode_melotts-en-au.json](projects/llm_framework/main_melotts/mode_melotts-en-us.json) | NPU |
|
||||
| [melotts-es-es](https://huggingface.co/myshell-ai/MeloTTS-Spanish) | TTS | 83M | 语音生成 | [mode_melotts-es-es.json](projects/llm_framework/main_melotts/mode_melotts-es-es.json) | NPU |
|
||||
| [melotts-ja-jp](https://huggingface.co/myshell-ai/MeloTTS-Japanese) | TTS | 83M | 语音生成 | [mode_melotts-ja-jp.json](projects/llm_framework/main_melotts/mode_melotts-ja-jp.json) | NPU |
|
||||
| [melotts-zh-cn](https://huggingface.co/myshell-ai/MeloTTS-Chinese) | TTS | 86M | 语音生成 | [mode_melotts-zh-cn.json](projects/llm_framework/main_melotts/mode_melotts-zh-cn.json) | NPU |
|
||||
| [deepseek-r1-1.5B-ax630c](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B) | LLM | 2.0G | 文本生成 | [mode_deepseek-r1-1.5B-ax630c.json](projects/llm_framework/main_llm/models/mode_deepseek-r1-1.5B-ax630c.json) | NPU |
|
||||
| [deepseek-r1-1.5B-p256-ax630c](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B) | LLM | 2.0G | 文本生成 | [mode_deepseek-r1-1.5B-p256-ax630c.json](projects/llm_framework/main_llm/models/mode_deepseek-r1-1.5B-p256-ax630c.json) | NPU |
|
||||
| [llama3.2-1B-p256-ax630c](https://huggingface.co/meta-llama/Llama-3.2-1B) | LLM | 1.7G | 文本生成 | [mode_llama3.2-1B-p256-ax630c.json](projects/llm_framework/main_llm/models/mode_llama3.2-1B-p256-ax630c.json) | NPU |
|
||||
| [llama3.2-1B-prefill-ax630c](https://huggingface.co/meta-llama/Llama-3.2-1B) | LLM | 1.7G | 文本生成 | [mode_llama3.2-1B-prefill-ax630c.json](projects/llm_framework/main_llm/models/mode_llama3.2-1B-prefill-ax630c.json) | NPU |
|
||||
| [openbuddy-llama3.2-1B-ax630c](https://huggingface.co/OpenBuddy/openbuddy-llama3.2-1b-v23.1-131k) | LLM | 1.7G | 文本生成 | [mode_openbuddy-llama3.2-1B-ax630c.json](projects/llm_framework/main_llm/models/mode_openbuddy-llama3.2-1B-ax630c.json) | NPU |
|
||||
| [qwen2.5-0.5B-Int4-ax630c](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GPTQ-Int4) | LLM | 626M | 文本生成 | [mode_qwen2.5-0.5B-Int4-ax630c.json](projects/llm_framework/main_llm/models/mode_qwen2.5-0.5B-Int4-ax630c.json) | NPU |
|
||||
| [qwen2.5-0.5B-p256-ax630c](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) | LLM | 760M | 文本生成 | [mode_qwen2.5-0.5B-p256-ax630c.json](projects/llm_framework/main_llm/models/mode_qwen2.5-0.5B-p256-ax630c.json) | NPU |
|
||||
| [qwen2.5-0.5B-prefill-20e](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) | LLM | 758M | 文本生成 | [mode_qwen2.5-0.5B-prefill-20e.json](projects/llm_framework/main_llm/models/mode_qwen2.5-0.5B-prefill-20e.json) | NPU |
|
||||
| [qwen2.5-1.5B-Int4-ax630c](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GPTQ-Int4) | LLM | 1.5G | 文本生成 | [mode_qwen2.5-1.5B-Int4-ax630c.json](projects/llm_framework/main_llm/models/mode_qwen2.5-1.5B-Int4-ax630c.json) | NPU |
|
||||
| [qwen2.5-1.5B-p256-ax630c](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) | LLM | 2.0G | 文本生成 | [mode_qwen2.5-1.5B-p256-ax630c.json](projects/llm_framework/main_llm/models/mode_qwen2.5-1.5B-p256-ax630c.json) | NPU |
|
||||
| [qwen2.5-1.5B-ax630c](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) | LLM | 2.0G | 文本生成 | [mode_qwen2.5-1.5B-ax630c.json](projects/llm_framework/main_llm/models/mode_qwen2.5-1.5B-ax630c.json) | NPU |
|
||||
| [qwen2.5-coder-0.5B-ax630c](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct) | LLM | 756M | 文本生成 | [mode_qwen2.5-coder-0.5B-ax630c.json](projects/llm_framework/main_llm/models/mode_qwen2.5-coder-0.5B-ax630c.json) | NPU |
|
||||
| [qwen3-0.6B-ax630c](https://huggingface.co/AXERA-TECH/InternVL2_5-1B) | LLM | 917M | 文本生成 | [mode_qwen3-0.6B-ax630c.json](projects/llm_framework/main_llm/models/mode_qwen3-0.6B-ax630c.json) | NPU |
|
||||
| [mode_internvl2.5-1B-364-ax630c](https://huggingface.co/Qwen/Qwen3-0.6B) | VLM | 1.2G | 多模态文本生成 | [mode_internvl2.5-1B-364-ax630c.json](projects/llm_framework/main_vlm/models/mode_internvl2.5-1B-364-ax630c.json) | NPU |
|
||||
| [smolvlm-256M-ax630c](https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Instruct) | VLM | 330M | 多模态文本生成 | [mode_smolvlm-256M-ax630c.json](projects/llm_framework/main_vlm/models/mode_smolvlm-256M-ax630c.json) | NPU |
|
||||
| [smolvlm-500M-ax630c](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct) | VLM | 605M | 多模态文本生成 | [mode_smolvlm-256M-ax630c.json](projects/llm_framework/main_vlm/models/mode_smolvlm-500M-ax630c.json) | NPU |
|
||||
| [yolo11n](https://github.com/ultralytics/ultralytics) | CV | 2.8M | 目标检测 | [mode_yolo11n.json](projects/llm_framework/main_yolo/mode_yolo11n.json) | NPU |
|
||||
| [yolo11n-npu1](https://github.com/ultralytics/ultralytics) | CV | 2.8M | 目标检测 | [mode_yolo11n-npu1.json](projects/llm_framework/main_yolo/mode_yolo11n-npu1.json) | NPU |
|
||||
| [yolo11n-seg](https://github.com/ultralytics/ultralytics) | CV | 3.0M | 实例分割 | [mode_yolo11n-seg.json](projects/llm_framework/main_yolo/mode_yolo11n-seg.json) | NPU |
|
||||
| [yolo11n-seg-npu1](https://github.com/ultralytics/ultralytics) | CV | 3.0M | 实例分割 | [mode_yolo11n-seg-npu1.json](projects/llm_framework/main_yolo/mode_yolo11n-seg-npu1.json) | NPU |
|
||||
| [yolo11n-pose](https://github.com/ultralytics/ultralytics) | CV | 3.1M | 姿态检测 | [mode_yolo11n-pose.json](projects/llm_framework/main_yolo/mode_yolo11n-pose.json) | NPU |
|
||||
| [yolo11n-pose-npu1](https://github.com/ultralytics/ultralytics) | CV | 3.1M | 姿态检测 | [mode_yolo11n-pose-npu1.json](projects/llm_framework/main_yolo/mode_yolo11n-pose-npu1.json) | NPU |
|
||||
| [yolo11n-hand-pose](https://github.com/ultralytics/ultralytics) | CV | 3.2M | 姿态检测 | [mode_yolo11n-hand-pose.json](projects/llm_framework/main_yolo/mode_yolo11n-hand-pose.json) | NPU |
|
||||
| [yolo11n-hand-pose-npu1](https://github.com/ultralytics/ultralytics) | CV | 3.2M | 姿态检测 | [mode_yolo11n-hand-pose-npu1.json](projects/llm_framework/main_yolo/mode_yolo11n-hand-pose-npu1.json) | NPU |
|
||||
| [depth-anything-ax630c](https://github.com/DepthAnything/Depth-Anything-V2) | CV | 29M | 单目深度估计 | [mode_depth-anything-ax630c.json](projects/llm_framework/main_depth_anything/mode_depth-anything-ax630c.json) | NPU |
|
||||
| [depth-anything-npu1-ax630c](https://github.com/DepthAnything/Depth-Anything-V2) | CV | 29M | 单目深度估计 | [mode_depth-anything-npu1-ax630c.json](projects/llm_framework/main_depth_anything/mode_depth-anything-npu1-ax630c.json) | NPU |
|
||||
|
||||
## 环境要求 ##
|
||||
当前 StackFlow 的 AI 单元是建立在 AXERA 加速平台之上的，主要的芯片平台为 ax630c、ax650n。系统要求为 ubuntu。
|
||||
|
||||
|
||||
@@ -37,6 +37,7 @@ Send JSON:
|
||||
- enoutput: Whether to enable user result output. If you do not need to obtain camera images, do not enable this parameter, as the video stream will increase the communication pressure on the channel.
|
||||
- enable_webstream: Whether to enable webstream output, webstream will listen on tcp:8989 port, and once a client connection is received, it will push jpeg images in HTTP protocol multipart/x-mixed-replace type.
|
||||
- rtsp: Whether to enable rtsp stream output, rtsp will establish an RTSP TCP server at rtsp://{DevIp}:8554/axstream0, and you can pull the video stream from this port using the RTSP protocol. The video stream format is 1280x720 H265. Note that this video stream is only valid on the AX630C MIPI camera, and the UVC camera cannot use RTSP.
|
||||
- VinParam.bAiispEnable: Whether to enable AI-ISP, enabled by default. Set to 0 to disable, only valid when using AX630C MIPI camera.
|
||||
|
||||
Response JSON:
|
||||
|
||||
|
||||
@@ -37,6 +37,7 @@
|
||||
- enoutput：是否起用用户结果输出。如果不需要获取摄像头图片，请不要开启该参数，视频流会增加信道的通信压力。
|
||||
- enable_webstream：是否启用 webstream 流输出，webstream 会监听 tcp:8989 端口，一但收到客户端连接，将会以 HTTP 协议 multipart/x-mixed-replace 类型推送 jpeg 图片。
|
||||
- rtsp：是否启用 rtsp 流输出，rtsp 会建立一个 rtsp://{DevIp}:8554/axstream0 RTSP TCP 服务端，可使用RTSP 协议向该端口拉取视频流。视频流的格式为 1280x720 H265。注意，该视频流只在 AX630C MIPI 摄像头上有效，UVC 摄像头无法使用 RTSP。
|
||||
- VinParam.bAiispEnable：是否开启 AI-ISP，默认开启。关闭为 0，仅在使用 AX630C MIPI 摄像头时有效。
|
||||
|
||||
响应 json：
|
||||
|
||||
|
||||
@@ -0,0 +1,223 @@
|
||||
# llm_cosy_voice
|
||||
|
||||
使用 npu 加速的文字转语音单元，用于提供文字转语音服务，可使用语音克隆，用于提供多语言转语音服务。
|
||||
|
||||
## setup
|
||||
|
||||
配置单元工作。
|
||||
|
||||
发送 json：
|
||||
|
||||
```json
|
||||
cosy_voice
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "cosy_voice",
|
||||
"action": "setup",
|
||||
"object": "cosy_voice.setup",
|
||||
"data": {
|
||||
"model": "CosyVoice2-0.5B-ax650",
|
||||
"response_format": "file",
|
||||
"input": "tts.utf-8",
|
||||
"enoutput": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
- request_id：参考基本数据解释。
|
||||
- work_id：配置单元时，为 `cosy_voice`。
|
||||
- action：调用的方法为 `setup`。
|
||||
- object：传输的数据类型为 `cosy_voice.setup`。
|
||||
- model：使用的模型为 `CosyVoice2-0.5B-ax650` 模型。
|
||||
- prompt_files：要克隆的音频信息文件。
|
||||
- response_format：返回结果为 `sys.pcm`, 系统音频数据，并直接发送到 llm-audio 模块进行播放。返回结果为 `file`, 生成的音频写 wav 文件，可用 `prompt_dir` 指定路径或文件名。
|
||||
- input：输入的为 `tts.utf-8`,代表的是从用户输入。
|
||||
- enoutput：是否起用用户结果输出。
|
||||
|
||||
响应 json：
|
||||
|
||||
```json
|
||||
{
|
||||
"created": 1761791627,
|
||||
"data": "None",
|
||||
"error": {
|
||||
"code": 0,
|
||||
"message": ""
|
||||
},
|
||||
"object": "None",
|
||||
"request_id": "2",
|
||||
"work_id": "cosy_voice.1000"
|
||||
}
|
||||
```
|
||||
|
||||
- created：消息创建时间，unix 时间。
|
||||
- work_id：返回成功创建的 work_id 单元。
|
||||
|
||||
## inference
|
||||
|
||||
### 流式输入
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "cosy_voice.1000",
|
||||
"action": "inference",
|
||||
"object": "cosy_voice.utf-8.stream",
|
||||
"data": {
|
||||
"delta": "今天天气真好！",
|
||||
"index": 0,
|
||||
"finish": true
|
||||
}
|
||||
}
|
||||
```
|
||||
- object：传输的数据类型为 `cosy_voice.utf-8.stream` 代表的是从用户 utf-8 的流式输入
|
||||
- delta：流式输入的分段数据
|
||||
- index：流式输入的分段索引
|
||||
- finish:流式输入是否完成的标志位
|
||||
|
||||
### 非流式输入
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "cosy_voice.1000",
|
||||
"action": "inference",
|
||||
"object": "cosy_voice.utf-8",
|
||||
"data": "今天天气真好！"
|
||||
}
|
||||
```
|
||||
- object：传输的数据类型为 `cosy_voice.utf-8` 代表的是从用户 utf-8 的非流式输入
|
||||
- data：非流式输入的数据
|
||||
|
||||
## pause
|
||||
|
||||
暂停单元工作。
|
||||
|
||||
发送 json：
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "5",
|
||||
"work_id": "cosy_voice.1000",
|
||||
"action": "pause"
|
||||
}
|
||||
```
|
||||
|
||||
响应 json：
|
||||
|
||||
```json
|
||||
{
|
||||
"created": 1761791706,
|
||||
"data": "None",
|
||||
"error": {
|
||||
"code": 0,
|
||||
"message": ""
|
||||
},
|
||||
"object": "None",
|
||||
"request_id": "5",
|
||||
"work_id": "cosy_voice.1000"
|
||||
}
|
||||
```
|
||||
|
||||
error::code 为 0 表示执行成功。
|
||||
|
||||
## exit
|
||||
|
||||
单元退出。
|
||||
|
||||
发送 json：
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "7",
|
||||
"work_id": "cosy_voice.1000",
|
||||
"action": "exit"
|
||||
}
|
||||
```
|
||||
|
||||
响应 json：
|
||||
|
||||
```json
|
||||
{
|
||||
"created": 1761791854,
|
||||
"data": "None",
|
||||
"error": {
|
||||
"code": 0,
|
||||
"message": ""
|
||||
},
|
||||
"object": "None",
|
||||
"request_id": "7",
|
||||
"work_id": "cosy_voice.1000"
|
||||
}
|
||||
```
|
||||
|
||||
error::code 为 0 表示执行成功。
|
||||
|
||||
## taskinfo
|
||||
|
||||
获取任务列表。
|
||||
|
||||
发送 json：
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "cosy_voice",
|
||||
"action": "taskinfo"
|
||||
}
|
||||
```
|
||||
|
||||
响应 json：
|
||||
|
||||
```json
|
||||
{
|
||||
"created": 1761791739,
|
||||
"data": [
|
||||
"cosy_voice.1000"
|
||||
],
|
||||
"error": {
|
||||
"code": 0,
|
||||
"message": ""
|
||||
},
|
||||
"object": "llm.tasklist",
|
||||
"request_id": "2",
|
||||
"work_id": "cosy_voice"
|
||||
}
|
||||
```
|
||||
|
||||
获取任务运行参数。
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "cosy_voice.1000",
|
||||
"action": "taskinfo"
|
||||
}
|
||||
```
|
||||
|
||||
响应 json：
|
||||
|
||||
```json
|
||||
{
|
||||
"created": 1761791761,
|
||||
"data": {
|
||||
"enoutput": false,
|
||||
"inputs": [
|
||||
"tts.utf-8"
|
||||
],
|
||||
"model": "CosyVoice2-0.5B-ax650",
|
||||
"response_format": "sys.pcm"
|
||||
},
|
||||
"error": {
|
||||
"code": 0,
|
||||
"message": ""
|
||||
},
|
||||
"object": "cosy_voice.taskinfo",
|
||||
"request_id": "2",
|
||||
"work_id": "cosy_voice.1000"
|
||||
}
|
||||
```
|
||||
|
||||
> **注意：work_id 是按照单元的初始化注册顺序增加的，并不是固定的索引值。**
|
||||
> **同类型单元不能配置多个单元同时工作，否则会产生未知错误。例如 tts 和 melo tts 不能同时拍起用工作。**
|
||||
@@ -34,7 +34,7 @@ Send JSON:
|
||||
- response_format: The result returned is in `kws.bool` format.
|
||||
- input: The input is `sys.pcm`, representing system audio.
|
||||
- enoutput: Whether to enable user result output.
|
||||
- kws: The Chinese wake-up word is `"你好你好"`.
|
||||
- kws: The English wake-up word is `"HELLO"`. It must be capital letters.
|
||||
- enwake_audio: Whether to enable wake-up audio output. Default is true.
|
||||
|
||||
Response JSON:
|
||||
|
||||
@@ -15,7 +15,7 @@ Send the following JSON:
|
||||
"action": "setup",
|
||||
"object": "vlm.setup",
|
||||
"data": {
|
||||
"model": "internvl2.5-1B-ax630c",
|
||||
"model": "internvl2.5-1B-364-ax630c",
|
||||
"response_format": "vlm.utf-8.stream",
|
||||
"input": "vlm.utf-8",
|
||||
"enoutput": true,
|
||||
@@ -29,7 +29,7 @@ Send the following JSON:
|
||||
- work_id: Set to `vlm` when configuring the unit.
|
||||
- action: The method being called is `setup`.
|
||||
- object: Data type being transferred is `vlm.setup`.
|
||||
- model: The model used is `internvl2.5-1B-ax630c`, a multimodal model.
|
||||
- model: The model used is `internvl2.5-1B-364-ax630c`, a multimodal model.
|
||||
- response_format: The output is in `vlm.utf-8.stream`, a UTF-8 stream format.
|
||||
- input: The input is `vlm.utf-8`, representing user input.
|
||||
- enoutput: Specifies whether to enable user output.
|
||||
@@ -250,7 +250,7 @@ Example:
|
||||
"action": "setup",
|
||||
"object": "vlm.setup",
|
||||
"data": {
|
||||
"model": "internvl2.5-1B-ax630c",
|
||||
"model": "internvl2.5-1B-364-ax630c",
|
||||
"response_format": "vlm.utf-8.stream",
|
||||
"input": [
|
||||
"vlm.utf-8",
|
||||
@@ -264,6 +264,38 @@ Example:
|
||||
}
|
||||
```
|
||||
|
||||
Linking the Output of the llm-camera Unit.
|
||||
|
||||
Sending JSON:
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "3",
|
||||
"work_id": "vlm.1003",
|
||||
"action": "link",
|
||||
"object": "work_id",
|
||||
"data": "camera.1000"
|
||||
}
|
||||
```
|
||||
|
||||
Response JSON:
|
||||
|
||||
```json
|
||||
{
|
||||
"created": 1750992545,
|
||||
"data": "None",
|
||||
"error": {
|
||||
"code": 0,
|
||||
"message": ""
|
||||
},
|
||||
"object": "None",
|
||||
"request_id": "3",
|
||||
"work_id": "vlm.1003"
|
||||
}
|
||||
```
|
||||
|
||||
> **Ensure that the camera is properly configured and ready for operation when performing the link action. If using the AX630C MIPI camera, configure it in AI-ISP disabled mode during the initialization of llm-camera.**
|
||||
|
||||
## unlink
|
||||
|
||||
Unlink units.
|
||||
@@ -447,7 +479,7 @@ Response JSON:
|
||||
"vlm.utf-8",
|
||||
"kws.1000"
|
||||
],
|
||||
"model": "internvl2.5-1B-ax630c",
|
||||
"model": "internvl2.5-1B-364-ax630c",
|
||||
"response_format": "vlm.utf-8.stream"
|
||||
},
|
||||
"error": {
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
"action": "setup",
|
||||
"object": "vlm.setup",
|
||||
"data": {
|
||||
"model": "internvl2.5-1B-ax630c",
|
||||
"model": "internvl2.5-1B-364-ax630c",
|
||||
"response_format": "vlm.utf-8.stream",
|
||||
"input": "vlm.utf-8",
|
||||
"enoutput": true,
|
||||
@@ -29,7 +29,7 @@
|
||||
- work_id：配置单元时，为 `vlm`。
|
||||
- action：调用的方法为 `setup`。
|
||||
- object：传输的数据类型为 `vlm.setup`。
|
||||
- model：使用的模型为 `internvl2.5-1B-ax630c` 多模态模型。
|
||||
- model：使用的模型为 `internvl2.5-1B-364-ax630c` 多模态模型。
|
||||
- response_format：返回结果为 `vlm.utf-8.stream`, utf-8 的流式输出。
|
||||
- input：输入的为 `vlm.utf-8`,代表的是从用户输入。
|
||||
- enoutput：是否起用用户结果输出。
|
||||
@@ -248,7 +248,7 @@ error::code 为 0 表示执行成功。
|
||||
"action": "setup",
|
||||
"object": "vlm.setup",
|
||||
"data": {
|
||||
"model": "internvl2.5-1B-ax630c",
|
||||
"model": "internvl2.5-1B-364-ax630c",
|
||||
"response_format": "vlm.utf-8.stream",
|
||||
"input": [
|
||||
"vlm.utf-8",
|
||||
@@ -262,6 +262,38 @@ error::code 为 0 表示执行成功。
|
||||
}
|
||||
```
|
||||
|
||||
链接 llm-camera 单元的输出。
|
||||
|
||||
发送 json：
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "3",
|
||||
"work_id": "vlm.1003",
|
||||
"action": "link",
|
||||
"object": "work_id",
|
||||
"data": "camera.1000"
|
||||
}
|
||||
```
|
||||
|
||||
响应 json：
|
||||
|
||||
```json
|
||||
{
|
||||
"created": 1750992545,
|
||||
"data": "None",
|
||||
"error": {
|
||||
"code": 0,
|
||||
"message": ""
|
||||
},
|
||||
"object": "None",
|
||||
"request_id": "3",
|
||||
"work_id": "vlm.1003"
|
||||
}
|
||||
```
|
||||
|
||||
> **link 时必须保证 camera 此时已经配置好进入工作状态。当同时使用 AX630C MIPI 摄像头，需要在 llm-camera 初始化的时候配置为 AI-ISP 关闭模式。**
|
||||
|
||||
## unlink
|
||||
|
||||
取消链接。
|
||||
@@ -445,7 +477,7 @@ error::code 为 0 表示执行成功。
|
||||
"vlm.utf-8",
|
||||
"kws.1000"
|
||||
],
|
||||
"model": "internvl2.5-1B-ax630c",
|
||||
"model": "internvl2.5-1B-364-ax630c",
|
||||
"response_format": "vlm.utf-8.stream"
|
||||
},
|
||||
"error": {
|
||||
|
||||
@@ -117,6 +117,11 @@ int llm_channel_obj::send_raw_to_pub(const std::string &raw)
|
||||
return zmq_[-1]->send_data(raw);
|
||||
}
|
||||
|
||||
int llm_channel_obj::send_raw_to_pub(const char *data, int size)
|
||||
{
|
||||
return zmq_[-1]->send_data(data, size);
|
||||
}
|
||||
|
||||
int llm_channel_obj::send_raw_to_usr(const std::string &raw)
|
||||
{
|
||||
if (zmq_[-2]) {
|
||||
|
||||
@@ -106,6 +106,7 @@ public:
|
||||
void stop_subscriber(const std::string &zmq_url);
|
||||
int check_zmq_errno(void *ctx, void *com, int code);
|
||||
int send_raw_to_pub(const std::string &raw);
|
||||
int send_raw_to_pub(const char *data, int size);
|
||||
int send_raw_to_usr(const std::string &raw);
|
||||
template <typename T, typename U>
|
||||
int output_data(const std::string &object, const T &data, const U &error_msg)
|
||||
|
||||
@@ -127,7 +127,7 @@ public:
|
||||
if ((url[0] != 'i') && (url[1] != 'p')) {
|
||||
rpc_url_head_.clear();
|
||||
}
|
||||
if (mode_ != ZMQ_RPC_FUN) creat(url, raw_call);
|
||||
creat(url, raw_call);
|
||||
}
|
||||
void set_timeout(int ms)
|
||||
{
|
||||
@@ -179,8 +179,6 @@ public:
|
||||
if (zmq_fun_.empty()) {
|
||||
std::string url = rpc_url_head_ + rpc_server_;
|
||||
mode_ = ZMQ_RPC_FUN;
|
||||
zmq_fun_["list_action"] =
|
||||
std::bind(&pzmq::_rpc_list_action, this, std::placeholders::_1, std::placeholders::_2);
|
||||
ret = creat(url);
|
||||
}
|
||||
zmq_fun_[action] = raw_call;
|
||||
@@ -330,6 +328,7 @@ public:
|
||||
inline int creat_rep(const std::string &url, const msg_callback_fun &raw_call)
|
||||
{
|
||||
int ret = zmq_bind(zmq_socket_, url.c_str());
|
||||
zmq_fun_["list_action"] = std::bind(&pzmq::_rpc_list_action, this, std::placeholders::_1, std::placeholders::_2);
|
||||
flage_ = false;
|
||||
zmq_thread_ = std::make_unique<std::thread>(std::bind(&pzmq::zmq_event_loop, this, raw_call));
|
||||
return ret;
|
||||
@@ -382,7 +381,7 @@ public:
|
||||
std::unique_lock<std::mutex> lock(zmq_fun_mtx_);
|
||||
retval = zmq_fun_.at(msg_ptr->string())(this, msg1_ptr);
|
||||
} catch (...) {
|
||||
retval = "NotAction";
|
||||
retval = msg_ptr->string() + " NotAction";
|
||||
}
|
||||
zmq_send(zmq_socket_, retval.c_str(), retval.length(), 0);
|
||||
msg1_ptr.reset();
|
||||
@@ -394,6 +393,8 @@ public:
|
||||
}
|
||||
void close_zmq()
|
||||
{
|
||||
int linger = 1000;
|
||||
zmq_setsockopt(zmq_socket_, ZMQ_LINGER, &linger, sizeof(linger));
|
||||
zmq_close(zmq_socket_);
|
||||
zmq_ctx_destroy(zmq_ctx_);
|
||||
if ((mode_ == ZMQ_PUB) || (mode_ == ZMQ_PULL) || (mode_ == ZMQ_RPC_FUN)) {
|
||||
|
||||
@@ -12,8 +12,8 @@ menuconfig AX_MSP_ENABLED
|
||||
bool "enable ax620e bsp"
|
||||
help
|
||||
build by AXERA!
|
||||
config AX_650N_MSP_ENABLED
|
||||
bool "enable ax650n bsp"
|
||||
config AX_650C_MSP_ENABLED
|
||||
bool "enable ax650C bsp"
|
||||
help
|
||||
build by AXERA!
|
||||
config AX_520_MSP_ENABLED
|
||||
@@ -25,17 +25,23 @@ menuconfig AX_MSP_ENABLED
|
||||
choice
|
||||
prompt "choice common version"
|
||||
default MSP_V2_0_0_ENABLED
|
||||
depends on AX_MSP_ENABLED
|
||||
config MSP_V3_0_0_ENABLED
|
||||
bool "enable v3.0.0 bsp"
|
||||
help
|
||||
build by m5stack!
|
||||
depends on AX_MSP_ENABLED && AX_620E_MSP_ENABLED
|
||||
config MSP_V2_0_0_ENABLED
|
||||
bool "enable v2.0.0 bsp"
|
||||
help
|
||||
build by m5stack!
|
||||
endchoice
|
||||
|
||||
|
||||
choice
|
||||
prompt "choice common version"
|
||||
default MSP_V3_6_2_ENABLED
|
||||
depends on AX_MSP_ENABLED && AX_650C_MSP_ENABLED
|
||||
config MSP_V3_6_2_ENABLED
|
||||
bool "enable v3.6.2 bsp"
|
||||
help
|
||||
build by m5stack!
|
||||
endchoice
|
||||
|
||||
config SAMPLE_COMMON_ENABLED
|
||||
bool "Enable sample common"
|
||||
default n
|
||||
|
||||
@@ -29,6 +29,7 @@ if 'CONFIG_AX_620E_MSP_ENABLED' in os.environ:
|
||||
for dirn in third_party:
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'third-party',dirn,'include'))
|
||||
LINK_SEARCH_PATH.append(os.path.join(MSP_PATH,'third-party',dirn,'lib/arm64/glibc'))
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'third-party/tinyalsa/include/tinyalsa'))
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'third-party/live/out/arm64/glibc/include'))
|
||||
LINK_SEARCH_PATH.append(os.path.join(MSP_PATH,'third-party/live/out/arm64/glibc/lib'))
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'third-party/openssl/arm64/include'))
|
||||
@@ -54,3 +55,60 @@ if 'CONFIG_AX_620E_MSP_ENABLED' in os.environ:
|
||||
'REGISTER':'static'
|
||||
})
|
||||
|
||||
if 'CONFIG_AX_650C_MSP_ENABLED' in os.environ:
|
||||
MSP_PATH = ''
|
||||
if 'CONFIG_MSP_V3_6_2_ENABLED' in os.environ:
|
||||
MSP_PATH = os.environ.get('EXT_MSP_PATH', check_wget_down("https://m5stack.oss-cn-shenzhen.aliyuncs.com/resource/linux/llm/m5stack_ax650n_msp.tar.gz", 'm5stack_ax650n_msp.tar.gz'))
|
||||
|
||||
SRCS=[]
|
||||
INCLUDE=[]
|
||||
PRIVATE_INCLUDE=[]
|
||||
REQUIREMENTS=[]
|
||||
STATIC_LIB=[]
|
||||
DYNAMIC_LIB=[]
|
||||
DEFINITIONS=[]
|
||||
DEFINITIONS_PRIVATE=['']
|
||||
LDFLAGS=[]
|
||||
LINK_SEARCH_PATH=[]
|
||||
|
||||
env["MSP_PATH"] = MSP_PATH
|
||||
# , os.path.join(MSP_PATH, 'sample/common')
|
||||
INCLUDE += [os.path.join(MSP_PATH, 'out/include'), os.path.join(MSP_PATH, 'sample'), os.path.join(MSP_PATH, 'sample/rtsp')]
|
||||
LINK_SEARCH_PATH += [os.path.join(MSP_PATH, 'out/lib')]
|
||||
REQUIREMENTS += ['dl', 'm', 'pthread']
|
||||
third_party = ['drm','faac','fdk-aac','ffmpeg','libsamplerate','mp4','opencv','opus','tinyalsa']
|
||||
for dirn in third_party:
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'third-party',dirn,'include'))
|
||||
LINK_SEARCH_PATH.append(os.path.join(MSP_PATH,'third-party',dirn,'lib/arm64/glibc'))
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'third-party/tinyalsa/include/tinyalsa'))
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'third-party/live/out/include/liveMedia'))
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'third-party/live/out/include/groupsock'))
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'third-party/live/out/include/UsageEnvironment'))
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'third-party/live/out/include/BasicUsageEnvironment'))
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'component/isp_proton/sensor/platform/tcm'))
|
||||
LINK_SEARCH_PATH.append(os.path.join(MSP_PATH,'third-party/libsamplerate/lib'))
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'third-party/openssl/include'))
|
||||
LINK_SEARCH_PATH.append(os.path.join(MSP_PATH,'third-party/openssl/arm64/glibc/lib'))
|
||||
LINK_SEARCH_PATH.append(os.path.join(MSP_PATH,'third-party/tinyalsa/lib'))
|
||||
LINK_SEARCH_PATH.append(os.path.join(MSP_PATH,'third-party/fdk-aac/lib'))
|
||||
LINK_SEARCH_PATH.append(os.path.join(MSP_PATH,'third-party/opus/lib'))
|
||||
LINK_SEARCH_PATH.append(os.path.join(MSP_PATH,'app/lib'))
|
||||
|
||||
if 'CONFIG_SAMPLE_COMMON_ENABLED' in os.environ:
|
||||
if 'CONFIG_MSP_V3_0_0_ENABLED' in os.environ:
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'sample/common'))
|
||||
SRCS += AGlob(os.path.join(MSP_PATH,'sample/common/*.c*'))
|
||||
|
||||
env['COMPONENTS'].append({'target':os.path.basename(env['component_dir']),
|
||||
'SRCS':SRCS,
|
||||
'INCLUDE':INCLUDE,
|
||||
'PRIVATE_INCLUDE':PRIVATE_INCLUDE,
|
||||
'REQUIREMENTS':REQUIREMENTS,
|
||||
'STATIC_LIB':STATIC_LIB,
|
||||
'DYNAMIC_LIB':DYNAMIC_LIB,
|
||||
'DEFINITIONS':DEFINITIONS,
|
||||
'DEFINITIONS_PRIVATE':DEFINITIONS_PRIVATE,
|
||||
'LDFLAGS':LDFLAGS,
|
||||
'LINK_SEARCH_PATH':LINK_SEARCH_PATH,
|
||||
'REGISTER':'static'
|
||||
})
|
||||
@@ -0,0 +1,5 @@
|
||||
menuconfig AX_TOKENIZER_ENABLED
|
||||
bool "Enable tokenizer support"
|
||||
default n
|
||||
help
|
||||
enable tokenizer support
|
||||
@@ -0,0 +1,52 @@
|
||||
# component2/SConscript
|
||||
Import("env")
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
with open(env["PROJECT_TOOL_S"]) as f:
|
||||
exec(f.read())
|
||||
|
||||
_SDK_PATH = os.path.normpath(
|
||||
os.environ.get("SDK_PATH", str(Path(os.getcwd()) / ".." / ".."))
|
||||
)
|
||||
|
||||
env["GIT_REPO_LISTS"]["tokenizer"] = {
|
||||
"url": "https://github.com/ZHEQIUSHUI/tokenizer.git",
|
||||
"commit": "83f41d4b5b9a135c167d44fcdf2a0c56ebacca6d",
|
||||
"path": str(Path(_SDK_PATH) / "github_source" / "tokenizer"),
|
||||
}
|
||||
|
||||
if "CONFIG_AX_TOKENIZER_ENABLED" in os.environ:
|
||||
check_component("tokenizer")
|
||||
SRCS = []
|
||||
INCLUDE = []
|
||||
PRIVATE_INCLUDE = []
|
||||
REQUIREMENTS = []
|
||||
STATIC_LIB = []
|
||||
DYNAMIC_LIB = []
|
||||
DEFINITIONS = []
|
||||
DEFINITIONS_PRIVATE = []
|
||||
LDFLAGS = []
|
||||
LINK_SEARCH_PATH = []
|
||||
|
||||
INCLUDE += [
|
||||
os.path.join(env["GIT_REPO_LISTS"]["tokenizer"]["path"], "include"),
|
||||
]
|
||||
print("AX-TOKENIZER INCLUDE:", INCLUDE)
|
||||
|
||||
env["COMPONENTS"].append(
|
||||
{
|
||||
"target": os.path.basename(env["component_dir"]),
|
||||
"SRCS": SRCS,
|
||||
"INCLUDE": INCLUDE,
|
||||
"PRIVATE_INCLUDE": PRIVATE_INCLUDE,
|
||||
"REQUIREMENTS": REQUIREMENTS,
|
||||
"STATIC_LIB": STATIC_LIB,
|
||||
"DYNAMIC_LIB": DYNAMIC_LIB,
|
||||
"DEFINITIONS": DEFINITIONS,
|
||||
"DEFINITIONS_PRIVATE": DEFINITIONS_PRIVATE,
|
||||
"LDFLAGS": LDFLAGS,
|
||||
"LINK_SEARCH_PATH": LINK_SEARCH_PATH,
|
||||
"REGISTER": "static",
|
||||
}
|
||||
)
|
||||
@@ -98,7 +98,7 @@ send :
|
||||
"action": "setup",
|
||||
"object": "melotts.setup",
|
||||
"data": {
|
||||
"model": "melotts_zh-cn",
|
||||
"model": "melotts-zh-cn",
|
||||
"response_format": "sys.pcm",
|
||||
"input": "tts.utf-8",
|
||||
"enoutput": false
|
||||
|
||||
@@ -5,7 +5,7 @@ import shutil
|
||||
os.environ['SDK_PATH'] = os.path.normpath(str(Path(os.getcwd())/'..'/'..'/'SDK'))
|
||||
os.environ['EXT_COMPONENTS_PATH'] = os.path.normpath(str(Path(os.getcwd())/'..'/'..'/'ext_components'))
|
||||
|
||||
version = 'v0.0.7'
|
||||
version = 'v0.1.3'
|
||||
static_lib = 'static_lib'
|
||||
update = False
|
||||
|
||||
|
||||
@@ -23,3 +23,4 @@ CONFIG_SINGLE_HEADER_LIBS_ENABLED=y
|
||||
CONFIG_AX_SAMPLES_ENABLED=y
|
||||
CONFIG_SIMDJSON_COMPENENT_ENABLED=y
|
||||
CONFIG_SAMPLE_COMMON_ENABLED=y
|
||||
CONFIG_MODBUS_ENABLED=y
|
||||
@@ -1,64 +0,0 @@
|
||||
// Copyright 2017 The Abseil Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
//
|
||||
// -----------------------------------------------------------------------------
|
||||
// File: algorithm.h
|
||||
// -----------------------------------------------------------------------------
|
||||
//
|
||||
// This header file contains Google extensions to the standard <algorithm> C++
|
||||
// header.
|
||||
|
||||
#ifndef ABSL_ALGORITHM_ALGORITHM_H_
|
||||
#define ABSL_ALGORITHM_ALGORITHM_H_
|
||||
|
||||
#include <algorithm>
|
||||
#include <iterator>
|
||||
#include <type_traits>
|
||||
|
||||
#include "absl/base/config.h"
|
||||
|
||||
namespace absl {
|
||||
ABSL_NAMESPACE_BEGIN
|
||||
|
||||
// equal()
|
||||
// rotate()
|
||||
//
|
||||
// Historical note: Abseil once provided implementations of these algorithms
|
||||
// prior to their adoption in C++14. New code should prefer to use the std
|
||||
// variants.
|
||||
//
|
||||
// See the documentation for the STL <algorithm> header for more information:
|
||||
// https://en.cppreference.com/w/cpp/header/algorithm
|
||||
using std::equal;
|
||||
using std::rotate;
|
||||
|
||||
// linear_search()
|
||||
//
|
||||
// Performs a linear search for `value` using the iterator `first` up to
|
||||
// but not including `last`, returning true if [`first`, `last`) contains an
|
||||
// element equal to `value`.
|
||||
//
|
||||
// A linear search is of O(n) complexity which is guaranteed to make at most
|
||||
// n = (`last` - `first`) comparisons. A linear search over short containers
|
||||
// may be faster than a binary search, even when the container is sorted.
|
||||
template <typename InputIterator, typename EqualityComparable>
|
||||
bool linear_search(InputIterator first, InputIterator last,
|
||||
const EqualityComparable& value) {
|
||||
return std::find(first, last, value) != last;
|
||||
}
|
||||
|
||||
ABSL_NAMESPACE_END
|
||||
} // namespace absl
|
||||
|
||||
#endif // ABSL_ALGORITHM_ALGORITHM_H_
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user