mirror of
https://github.com/m5stack/StackFlow.git
synced 2026-05-20 11:32:11 -07:00
[update] README camera
This commit is contained in:
dianjixz
Vendored
+26
@@ -0,0 +1,26 @@
|
||||
{
|
||||
// 使用 IntelliSense 了解相关属性。
|
||||
// 悬停以查看现有属性的描述。
|
||||
// 欲了解更多信息，请访问: https://go.microsoft.com/fwlink/?linkid=830387
|
||||
"version": "0.2.0",
|
||||
"configurations": [
|
||||
"name": "gdb 启动",
|
||||
"type": "cppdbg",
|
||||
"request": "launch",
|
||||
"program": "/home/nihao/w2T/axera/StackFlow/projects/test_kit_demo3/dist/test_kit_demo3",
|
||||
"args": [],
|
||||
"stopAtEntry": false,
|
||||
"cwd": "${workspaceFolder}",
|
||||
"miDebuggerPath": "/opt/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gdb",
|
||||
"miDebuggerServerAddress": "localhost:1234",
|
||||
"environment": [],
|
||||
"externalConsole": true,
|
||||
"MIMode": "gdb",
|
||||
"setupCommands":[{
|
||||
"description": "Enable pretty-printing for gdb",
|
||||
"text": "-enable-pretty-printing",
|
||||
"ignoreFailures": true
|
||||
}]
|
||||
|
||||
]
|
||||
}
|
||||
+76
-83
@@ -1,100 +1,93 @@
|
||||
# Module-LLM
|
||||
# StackFlow
|
||||
|
||||
<div class="product_pic"><img class="pic" src="https://m5stack.oss-cn-shenzhen.aliyuncs.com/resource/docs/products/module/Module%20LLM/4.webp" width="25%">
|
||||
<p align="center"><img src="https://static-cdn.m5stack.com/resource/public/assets/m5logo2022.svg" alt="basic" width="300" height="300"></p>
|
||||
|
||||
## 描述
|
||||
|
||||
**Module LLM**是一款集成化的离线大语言模型 (LLM) 推理模块，专为需要高效，智能交互的终端设备设计，无论是在智能家居，语音助手，还是在工业控制中，Module LLM 都能为您带来流畅，自然的 AI 体验，无需依赖云端，确保隐私安全与稳定性。集成 **StackFlow** 框架，配合 **Arduino/UiFlow** 库，几行代码就可轻松实现端侧智能。<br>
|
||||
搭载爱芯 **AX630C** SoC 先进处理器，集成 3.2 TOPs 高能效 NPU，原生支持 Transformer 模型，轻松应对复杂 AI 任务。配备 **4GB LPDDR4** 内存（其中1GB供用户使用，3GB专用于硬件加速）和**32GB eMMC**存储，支持多模型并行加载与串联推理，确保多任务处理流畅无阻。运行功耗仅约 1.5W，远低于同类产品，节能高效，适合长时间运行。<br>
|
||||
集成麦克风，扬声器，TF存储卡，**USB OTG** 及 RGB状态灯，满足多样化应用需求，轻松实现语音交互与数据传输。灵活扩展：板载 SD 卡槽支持固件冷/热升级，**UART** 通信接口简化连接与调试，确保模块功能持续优化与扩展。USB 口支持主从自动切换，既可以做调试口，也可以外接更多 USB 设备如摄像头。搭配 LLM 调试套件，用于扩展百兆以太网口，及内核串口，作为 SBC 使用。<br>
|
||||
多模型兼容，出厂预装 **Qwen2.5-0.5B** 大语言模型，内置**KWS**（唤醒词），**ASR**（语音识别），**LLM**（大语言模型）及**TTS**（文本转语音）功能，且支持分开调用或 **pipeline** 自动流转，方便开发。后续将支持Qwen2.5-1.5B、Llama3.2-1B及InternVL2-1B等多种端侧LLM/VLM模型，支持热更新模型，紧跟社区潮流，适应不同复杂度的AI任务；视觉识别能力：支持CLIP、YoloWorld等Open world模型，未来将持续更新DepthAnything、SegmentAnything等先进模型，赋能智能识别与分析；<br>
|
||||
即插即用，搭配**M5主机**，Module LLM 实现即插即用的AI交互体验。用户无需繁琐设置，即可将其集成到现有智能设备中，快速启用智能化功能，提升设备智能水平。该产品适用于离线语音助手，文本语音转换，智能家居控制，互动机器人等领域。
|
||||
<p align="center">
|
||||
StackFlow 是一个简单、疾速、优雅的面向嵌入式开发者一站式 AI 服务基础设施项目，目的是让 Maker 和 Haker 们能够快速的在当前的嵌入式设备中获取到强大的 AI 加速能力。StackFlow 能够为各类人机交互设备注入智慧的灵魂。
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://opencollective.com/wukong-robot/contribute/tier/8131-sponsor" target="_blank"><img src="https://avatars.githubusercontent.com/u/17420673?s=48&v=4"></a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
## Table of Contents
|
||||
|
||||
## 产品特性
|
||||
|
||||
- 离线推理,3.2T@INT8精度算力
|
||||
|
||||
- 集成KWS（唤醒词），ASR（语音识别），LLM（大语言模型），TTS（文本生成语音）
|
||||
|
||||
- 多模型并行
|
||||
|
||||
- 板载32GB eMMC存储和4GB LPDDR4内存
|
||||
|
||||
- 板载麦克风及扬声器
|
||||
|
||||
- 串口通信
|
||||
|
||||
- SD卡固件升级
|
||||
|
||||
- 支持ADB调试
|
||||
|
||||
- RGB提示灯
|
||||
|
||||
- 内置ubuntu系统
|
||||
|
||||
- 支持OTG功能
|
||||
|
||||
- 支持Arduino/UIFlow
|
||||
|
||||
|
||||
* [特性](#特性)
|
||||
* [Demo](#demo)
|
||||
* [环境要求](#环境要求)
|
||||
* [安装](#安装)
|
||||
* [升级](#升级)
|
||||
* [运行](#运行)
|
||||
* [配置](#配置)
|
||||
* [API接口](#api-接口)
|
||||
* [贡献](#贡献)
|
||||
|
||||
|
||||
## 应用
|
||||
## 特性
|
||||
|
||||
* 分布式通信架构。每个单元都能够单独工作或者和其他单元配合工作。
|
||||
* 多模型支持。包括但不限于语音识别、语音合成、图像识别、自然语言处理、LLM 大模型助手推理等。
|
||||
* 数据内部流转。可根据需求配置不同单元协同工作，避免复杂的数据处理流程。
|
||||
* 简单易用。通过标准 json 交换数据，快速实现 AI 服务。
|
||||
* 离线化运行。无需联网即可实现本地 AI 服务。
|
||||
* 多平台支持。包括但不限于 Module LLM、LLM630 Compute Kit 等。
|
||||
* 灵活配置。所有单元均可完全配置运行参数，在相同的数据流处理情况下，随意更换模型和修改模型运行参数。
|
||||
* 简单易用。开发者只需关注模型和硬件平台，无需关注底层通信和数据处理细节，即可快速实现 AI 服务。
|
||||
* 高效稳定。通过 ZMQ 信道传输，数据传输效率高，延迟低，稳定性强。
|
||||
* 开源免费。StackFlow 采用 MIT 许可证，开发者可以自由使用和修改。
|
||||
* 多语言支持。单元主体由 C++ 实现，性能极致优化，可扩展至多变成语言支持。（需要支持 ZMQ 编程）
|
||||
|
||||
- 离线语音助手
|
||||
- 文本语音转换
|
||||
- 智能家居控制
|
||||
- 互动机器人
|
||||
StackFlow 还在不断优化和迭代，在框架更加完善的同时，会持续增加更多功能，敬请期待。
|
||||
|
||||
## 规格参数
|
||||
|
||||
| 规格 | 参数 |
|
||||
| ------------ | --------------------------------------------------------------------------- |
|
||||
| 处理器SoC | AX630C@Dual Cortex A53 1.2 GHz <br> MAX.12.8 TOPS @INT4,and 3.2 TOPS @INT8 |
|
||||
| 内存 | 4GB LPDDR4(1GB系统内存 + 3GB 硬件加速专用内存) |
|
||||
| 存储 | 32GB eMMC5.1 |
|
||||
| 通信接口 | 串口通信 波特率默认 115200@8N1（可调） |
|
||||
| 麦克风 | MSM421A |
|
||||
| 音频驱动 | AW8737 |
|
||||
| 扬声器 | 8Ω@1W,尺寸:2014 腔体喇叭 |
|
||||
| 内置功能单元 | KWS（唤醒词），ASR（语音识别），LLM（大语言模型），TTS（文本生成语音） |
|
||||
| RGB灯 | 3x RGB LED@2020 由LP5562驱动 (状态指示) |
|
||||
| 功耗 | 空载：5V@0.5W，满载:5V@1.5W |
|
||||
| 按键 | 用于升级固件进入下载模式 |
|
||||
| 升级接口 | SD卡/Type C口 |
|
||||
| 工作温度 | 0-40°C |
|
||||
| 产品尺寸 | 54x54x13mm |
|
||||
| 包装尺寸 | 133x95x16mm |
|
||||
| 产品重量 | 17.4g |
|
||||
| 包装重量 | 32.0g |
|
||||
|
||||
## 相关链接
|
||||
|
||||
- [AX630C](https://m5stack.oss-cn-shenzhen.aliyuncs.com/resource/docs/products/module/Module%20LLM/AX630C.pdf)
|
||||
|
||||
## PinMap
|
||||
|
||||
| Module LLM | RXD | TXD |
|
||||
| ------------ | --- | --- |
|
||||
| Core (Basic) | G16 | G17 |
|
||||
| Core2 | G13 | G14 |
|
||||
| CoreS3 | G18 | G17 |
|
||||
|
||||
>LLM Module引脚切换| LLM Module预留了引脚切换焊盘, 一些出现引脚复用冲突的情况下, 可以通过切割PCB线路然后跳线连接至其他组引脚。
|
||||
|
||||
<img alt="module size" src="https://m5stack.oss-cn-shenzhen.aliyuncs.com/resource/docs/products/module/Module%20LLM/03.jpg" width="25%" />
|
||||
|
||||
> 以`CoreS3`为例子，第一列（左绿色框）是串口通讯的TX引脚，用户根据需要四选一，（从上到下分别代表引脚G18 G7 G14 G10）,默认是IO18引脚，如需要切换其他引脚，切断焊盘连接线（红线处）（建议使用刀片切割），然后连接下面三个引脚之间的一个即可；第二列（右绿色框）是RX引脚切换，如上所述，也是四选一操作。
|
||||
StackFlow 语音助手的主要工作模式：
|
||||
|
||||
|
||||
开机后，KWS、ASR、LLM、TTS、AUDIO 被配置成协同工作状态。当 KWS 从 AUDIO 单元获取的音频中检测出关键词后，发出唤醒信号。此时，ASR 开始工作，识别 AUDIO 的音频数据，并将结果发布到自己的输出信道中。LLM 接收到 ASR 转换后的文本数据后，开始进行推理，并将结果发布到自己的输出信道中。TTS 接收到 LLM 的结果后，开始进行语音合成，合成完成后根据配置来播放合成好的音频数据。
|
||||
|
||||
## 视频
|
||||
|
||||
- Module LLM 产品介绍以及例子展示 [Module_LLM_Video.mp4](https://m5stack.oss-cn-shenzhen.aliyuncs.com/resource/docs/products/module/Module%20LLM/Module_LLM_Video.mp4)
|
||||
## Demo
|
||||
- [StackFlow 连续语音识别]()
|
||||
- [StackFlow LLM 大模型唤醒对话]()
|
||||
- [StackFlow TTS 语音合成播放]()
|
||||
- [StackFlow yolo 视觉检测]()
|
||||
- [StackFlow VLM 图片描述]()
|
||||
|
||||
## AI Benchmark 对比
|
||||
## 环境要求 ##
|
||||
当前 StackFlow 的 AI 单元是建立在 AXERA 加速平台之上的，主要的芯片平台为 ax630c、ax650n。系统要求为 ubuntu。
|
||||
|
||||
<img alt="compare" src="https://m5stack.oss-cn-shenzhen.aliyuncs.com/resource/docs/products/module/Module%20LLM/Benchmark%E5%AF%B9%E6%AF%94.png" width="100%" />
|
||||
## 安装 ##
|
||||
StackFlow 的程序和模型数据是分开的，程序安装完成后，需要单独下载模型数据，并配置到程序中。安装是先安装程序包，然后再安装模型包。
|
||||
裸机环境安装见[StackFlow 安装教程]() 。
|
||||
|
||||
## 升级
|
||||
升级时可单独升级 AI 单元，或者升级整个 StackFlow 框架。
|
||||
升级单个单元时可通过 SD 卡升级或者手动 `dpkg` 命令进行安装。
|
||||
## 运行 ##
|
||||
相关 AI 服务会在开机时自动运行，也可以通过手动命令启动。
|
||||
sys 单元运行状态查询：
|
||||
```bash
|
||||
systemctl status llm-sys
|
||||
```
|
||||
相关命令可参考 systemd 服务命令。
|
||||
|
||||
## 配置 ##
|
||||
StackFlow 的配置分为两类，一类是单元工作参数配置，一类是模型工作参数配置。
|
||||
两类配置文件均采用 json 格式，配置文件位于多个目录下，目录如下：
|
||||
```
|
||||
/opt/m5stack/data/models/
|
||||
/opt/m5stack/share/
|
||||
```
|
||||
## API 接口 ##
|
||||
StackFlow 可通过 UART 和 TCP 端口进行访问。
|
||||
## 贡献
|
||||
|
||||
* 喜欢本项目请先打一颗星；
|
||||
* 提 bug 请到 [issue 页面](https://github.com/m5stack/StackFlow/issues)；
|
||||
* 要贡献代码，欢迎 fork 之后再提 pull request；
|
||||
|
||||
## Star 历史
|
||||
|
||||
[](https://star-history.com/#m5stack/StackFlow&Date)
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
digraph G {
|
||||
SYS -> KWS [color="purple"];
|
||||
SYS -> ASR [color="purple"];
|
||||
SYS -> LLM [color="purple"];
|
||||
SYS -> TTS [color="purple"];
|
||||
SYS -> AUDIO [color="purple"];
|
||||
SYS -> CAMERA [color="purple"];
|
||||
SYS -> VAD [color="purple"];
|
||||
SYS -> VLM [color="purple"];
|
||||
SYS -> YOLO [color="purple"];
|
||||
SYS -> DEPTH [color="purple"];
|
||||
KWS -> SYS [color="green"];
|
||||
ASR -> SYS [color="green"];
|
||||
LLM -> SYS [color="green"];
|
||||
TTS -> SYS [color="green"];
|
||||
AUDIO -> SYS [color="green"];
|
||||
CAMERA -> SYS [color="green"];
|
||||
VAD -> SYS [color="green"];
|
||||
VLM -> SYS [color="green"];
|
||||
YOLO -> SYS [color="green"];
|
||||
DEPTH -> SYS [color="green"];
|
||||
AUDIO -> KWS [color="blue"];
|
||||
AUDIO -> ASR [color="blue"];
|
||||
AUDIO -> VAD [color="blue"];
|
||||
CAMERA -> YOLO [color="blue"];
|
||||
CAMERA -> DEPTH [color="blue"];
|
||||
CAMERA -> VLM [color="blue"];
|
||||
ASR -> VLM [color="blue"];
|
||||
VLM -> TTS [color="blue"];
|
||||
KWS -> ASR [color="blue"];
|
||||
KWS -> LLM [color="blue"];
|
||||
KWS -> AUDIO [color="blue"];
|
||||
ASR -> LLM [color="blue"];
|
||||
LLM -> TTS [color="blue"];
|
||||
TTS -> AUDIO [color="blue"];
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 115 KiB |
@@ -6,13 +6,82 @@ internally handling port resource allocation and a simple in-memory database.
|
||||
## External API
|
||||
|
||||
- sys.ping: Test if communication with the LLM is possible.
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "ping"
|
||||
}
|
||||
```
|
||||
- sys.lsmode: Models that have existed in the system in the past.
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "lsmode"
|
||||
}
|
||||
```
|
||||
- sys.bashexec: Execute bash commands.
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "bashexec",
|
||||
"object": "sys.utf-8.stream",
|
||||
"data" : {
|
||||
"index": 0,
|
||||
"delta": "ls",
|
||||
"finish": true
|
||||
}
|
||||
}
|
||||
```
|
||||
- sys.hwinfo: Retrieve onboard CPU, memory, and temperature parameters of the LLM.
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "hwinfo"
|
||||
}
|
||||
```
|
||||
- sys.uartsetup: Set serial port parameters, effective for a single session.
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "uartsetup",
|
||||
"object": "sys.uartsetup",
|
||||
"data" : {
|
||||
"baud": 115200,
|
||||
"data_bits": 8,
|
||||
"stop_bits": 1,
|
||||
"parity": 110
|
||||
}
|
||||
}
|
||||
```
|
||||
- sys.reset: Reset the entire LLM framework application.
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "reset"
|
||||
}
|
||||
```
|
||||
- sys.reboot: Restart the system.
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "reboot"
|
||||
}
|
||||
```
|
||||
- sys.version: Get the version of the LLM framework program.
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "version"
|
||||
}
|
||||
```
|
||||
|
||||
## Internal API
|
||||
|
||||
|
||||
@@ -5,13 +5,82 @@ StackFlow 的基本服务单元，对外提供串口和 TCP 外部信道和一
|
||||
## 外部 API
|
||||
|
||||
- sys.ping：测试是否能够和 LLM 进行通信。
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "ping"
|
||||
}
|
||||
```
|
||||
- sys.lsmode：过去系统中存在的模型。
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "lsmode"
|
||||
}
|
||||
```
|
||||
- sys.bashexec：执行 bash 命令。
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "bashexec",
|
||||
"object": "sys.utf-8.stream",
|
||||
"data" : {
|
||||
"index": 0,
|
||||
"delta": "ls",
|
||||
"finish": true
|
||||
}
|
||||
}
|
||||
```
|
||||
- sys.hwinfo：获取 LLM 板载 cpu、内存、温度参数。
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "hwinfo"
|
||||
}
|
||||
```
|
||||
- sys.uartsetup：设置串口参数，单次生效。
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "uartsetup",
|
||||
"object": "sys.uartsetup",
|
||||
"data" : {
|
||||
"baud": 115200,
|
||||
"data_bits": 8,
|
||||
"stop_bits": 1,
|
||||
"parity": 110
|
||||
}
|
||||
}
|
||||
```
|
||||
- sys.reset：复位整个 LLM 框架的应用程序。
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "reset"
|
||||
}
|
||||
```
|
||||
- sys.reboot：重启系统。
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "reboot"
|
||||
}
|
||||
```
|
||||
- sys.version：获取 LLM 框架程序版本。
|
||||
```json
|
||||
{
|
||||
"request_id": "2",
|
||||
"work_id": "sys",
|
||||
"action": "version"
|
||||
}
|
||||
```
|
||||
|
||||
## 内部 API：
|
||||
|
||||
|
||||
@@ -16,4 +16,26 @@ menuconfig AX_MSP_ENABLED
|
||||
bool "enable ax520 bsp"
|
||||
help
|
||||
build by AXERA!
|
||||
endchoice
|
||||
endchoice
|
||||
|
||||
choice
|
||||
prompt "choice common version"
|
||||
default MSP_V2_0_0_ENABLED
|
||||
depends on AX_MSP_ENABLED
|
||||
config MSP_V3_0_0_ENABLED
|
||||
bool "enable v3.0.0 bsp"
|
||||
help
|
||||
build by m5stack!
|
||||
config MSP_V2_0_0_ENABLED
|
||||
bool "enable v2.0.0 bsp"
|
||||
help
|
||||
build by m5stack!
|
||||
endchoice
|
||||
|
||||
config SAMPLE_COMMON_ENABLED
|
||||
bool "Enable sample common"
|
||||
default n
|
||||
depends on AX_MSP_ENABLED
|
||||
help
|
||||
enable sample common
|
||||
|
||||
|
||||
@@ -5,7 +5,10 @@ with open(env['PROJECT_TOOL_S']) as f:
|
||||
exec(f.read())
|
||||
|
||||
if 'CONFIG_AX_620E_MSP_ENABLED' in os.environ:
|
||||
MSP_PATH = os.environ.get('EXT_MSP_PATH', check_wget_down("https://m5stack.oss-cn-shenzhen.aliyuncs.com/resource/linux/llm/m5stack_msp.tar.gz", 'm5stack_msp.tar.gz'))
|
||||
MSP_PATH = ''
|
||||
if 'CONFIG_MSP_V2_0_0_ENABLED' in os.environ:
|
||||
MSP_PATH = os.environ.get('EXT_MSP_PATH', check_wget_down("https://m5stack.oss-cn-shenzhen.aliyuncs.com/resource/linux/llm/m5stack_msp.tar.gz", 'm5stack_msp.tar.gz'))
|
||||
|
||||
SRCS=[]
|
||||
INCLUDE=[]
|
||||
PRIVATE_INCLUDE=[]
|
||||
@@ -18,7 +21,8 @@ if 'CONFIG_AX_620E_MSP_ENABLED' in os.environ:
|
||||
LINK_SEARCH_PATH=[]
|
||||
|
||||
env["MSP_PATH"] = MSP_PATH
|
||||
INCLUDE += [os.path.join(MSP_PATH, 'out/arm64_glibc/include'), os.path.join(MSP_PATH, 'sample'), os.path.join(MSP_PATH, 'sample/common'), os.path.join(MSP_PATH, 'sample/rtsp')]
|
||||
# , os.path.join(MSP_PATH, 'sample/common')
|
||||
INCLUDE += [os.path.join(MSP_PATH, 'out/arm64_glibc/include'), os.path.join(MSP_PATH, 'sample'), os.path.join(MSP_PATH, 'sample/rtsp')]
|
||||
LINK_SEARCH_PATH += [os.path.join(MSP_PATH, 'out/arm64_glibc/lib')]
|
||||
REQUIREMENTS += ['dl', 'm', 'pthread']
|
||||
third_party = ['drm','faac','fdk-aac','ffmpeg','libsamplerate','mp4','opencv','opus','tinyalsa']
|
||||
@@ -31,6 +35,11 @@ if 'CONFIG_AX_620E_MSP_ENABLED' in os.environ:
|
||||
LINK_SEARCH_PATH.append(os.path.join(MSP_PATH,'third-party/openssl/arm64/glibc/lib'))
|
||||
LINK_SEARCH_PATH.append(os.path.join(MSP_PATH,'app/lib'))
|
||||
|
||||
if 'CONFIG_SAMPLE_COMMON_ENABLED' in os.environ:
|
||||
if 'CONFIG_MSP_V2_0_0_ENABLED' in os.environ:
|
||||
INCLUDE.append(os.path.join(MSP_PATH,'sample/common'))
|
||||
SRCS += AGlob(os.path.join(MSP_PATH,'sample/common/*.c*'))
|
||||
|
||||
env['COMPONENTS'].append({'target':os.path.basename(env['component_dir']),
|
||||
'SRCS':SRCS,
|
||||
'INCLUDE':INCLUDE,
|
||||
|
||||
@@ -23,3 +23,4 @@ CONFIG_UTILITIES_BASE64_ENABLED=y
|
||||
CONFIG_SINGLE_HEADER_LIBS_ENABLED=y
|
||||
CONFIG_AX_SAMPLES_ENABLED=y
|
||||
CONFIG_SIMDJSON_COMPENENT_ENABLED=y
|
||||
CONFIG_SAMPLE_COMMON_ENABLED=y
|
||||
|
||||
@@ -22,7 +22,17 @@ LDFLAGS+=['-Wl,-rpath=/opt/m5stack/lib', '-Wl,-rpath=/usr/local/m5stack/lib', '-
|
||||
# INCLUDE += [ADir('../include'), ADir('../include/cppzmq'), ADir('../include/libzmq')]
|
||||
LINK_SEARCH_PATH += [ADir('../static_lib')]
|
||||
|
||||
|
||||
# INCLUDE.append(ADir('ax_sample'))
|
||||
# SRCS += Glob('ax_sample/*.c*')
|
||||
|
||||
REQUIREMENTS += ['ax_sys', 'ax_interpreter']
|
||||
REQUIREMENTS += ['ax_engine', 'ax_interpreter', 'ax_sys', 'ax_vo']
|
||||
REQUIREMENTS += ['ax_opal', 'ax_ae', 'ax_skel', 'ax_opal', 'ax_venc', 'ax_mipi', 'ax_ives', 'ax_ivps', 'ax_proton', 'ax_audio', 'mpr', 'tinyalsa']
|
||||
REQUIREMENTS += ['ax_audio_3a','samplerate', 'ax_fdk', 'fdk-aac', 'ax_awb', 'ax_af', 'ax_nt_stream', 'ax_nt_ctrl']
|
||||
|
||||
|
||||
|
||||
|
||||
INCLUDE += [ADir('../include/opencv4')]
|
||||
static_file = []
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,35 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: 2024 M5Stack Technology CO LTD
|
||||
*
|
||||
* SPDX-License-Identifier: MIT
|
||||
*/
|
||||
#ifndef AXERA_CAMERA_H
|
||||
#define AXERA_CAMERA_H
|
||||
|
||||
#if __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
/**
|
||||
* Open the axera_camera
|
||||
* @pdev_name Device node
|
||||
* Return value: NULL for failure
|
||||
*/
|
||||
camera_t* axera_camera_open(const char* pdev_name, int width, int height, int fps);
|
||||
|
||||
/**
|
||||
* Open the axera_camera from config
|
||||
* @pDevName Device node
|
||||
* Return value: 0 for success, -1 for failure
|
||||
*/
|
||||
int axera_camera_open_from(camera_t* camera);
|
||||
|
||||
/**
|
||||
* Close the axera_camera
|
||||
* Return value: 0 for success, -1 for failure
|
||||
*/
|
||||
int axera_camera_close(camera_t* camera);
|
||||
|
||||
#if __cplusplus
|
||||
}
|
||||
#endif
|
||||
#endif
|
||||
@@ -14,7 +14,7 @@
|
||||
#if __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
typedef void (*vcamera_frame_get)(uint8_t* pdata, uint32_t width, uint32_t height, uint32_t length, void* ctx);
|
||||
typedef void (*vcamera_frame_get)(void* pdata, uint32_t width, uint32_t height, uint32_t length, void* ctx);
|
||||
typedef enum { CAMERA_STATE_CLOSE, CAMERA_SATTE_OPEN, CAMERA_SATTE_CAP, CAMERA_STATE_ERR } CAMERA_STATE_t;
|
||||
|
||||
typedef struct {
|
||||
@@ -32,7 +32,7 @@ typedef struct camera_t {
|
||||
uint32_t height_;
|
||||
CAMERA_Buffer_t* pcamera_buffer_;
|
||||
vcamera_frame_get pcallback_;
|
||||
CAMERA_STATE_t state_; /* 0 Not open, 1 Turn on the camera, 2 Start capturing,
|
||||
int state_; /* 0 Not open, 1 Turn on the camera, 2 Start capturing,
|
||||
3 Error */
|
||||
pthread_t capture_thread_id_;
|
||||
void* ctx_;
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
#include <iostream>
|
||||
#include "../../../../SDK/components/utilities/include/sample_log.h"
|
||||
#include "camera.h"
|
||||
#include "axera_camera.h"
|
||||
#include <glob.h>
|
||||
#include <opencv2/opencv.hpp>
|
||||
|
||||
@@ -25,6 +26,10 @@ static void __sigint(int iSigNo)
|
||||
|
||||
typedef std::function<void(const void *, int)> task_callback_t;
|
||||
|
||||
typedef camera_t* (*hal_camera_open_fun)(const char* pdev_name, int width, int height, int fps);
|
||||
typedef int (*hal_camera_close_fun)(camera_t* camera);
|
||||
|
||||
|
||||
#define CONFIG_AUTO_SET(obj, key) \
|
||||
if (config_body.contains(#key)) \
|
||||
mode_config_.key = config_body[#key]; \
|
||||
@@ -34,7 +39,8 @@ typedef std::function<void(const void *, int)> task_callback_t;
|
||||
class llm_task {
|
||||
private:
|
||||
camera_t *cam;
|
||||
|
||||
hal_camera_open_fun hal_camera_open;
|
||||
hal_camera_close_fun hal_camera_close;
|
||||
public:
|
||||
std::string response_format_;
|
||||
task_callback_t out_callback_;
|
||||
@@ -49,7 +55,7 @@ public:
|
||||
int frame_height_;
|
||||
cv::Mat yuv_dist_;
|
||||
|
||||
static void on_cap_fream(uint8_t *pData, uint32_t width, uint32_t height, uint32_t Length, void *ctx)
|
||||
static void on_cap_fream(void *pData, uint32_t width, uint32_t height, uint32_t Length, void *ctx)
|
||||
{
|
||||
llm_task *self = static_cast<llm_task *>(ctx);
|
||||
int src_offsetX;
|
||||
@@ -127,6 +133,13 @@ public:
|
||||
}
|
||||
enstream_ = (response_format_.find("stream") != std::string::npos);
|
||||
yuv_dist_ = cv::Mat(frame_height_, frame_width_, CV_8UC2);
|
||||
if(devname_.find("/dev/video") != std::string::npos){
|
||||
hal_camera_open = camera_open;
|
||||
hal_camera_close = camera_close;
|
||||
}else if(devname_.find("axera_") != std::string::npos){
|
||||
hal_camera_open = axera_camera_open;
|
||||
hal_camera_close = axera_camera_close;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
@@ -136,7 +149,7 @@ public:
|
||||
return -1;
|
||||
}
|
||||
try {
|
||||
cam = camera_open(devname_.c_str(), frame_width_, frame_height_, 30);
|
||||
cam = hal_camera_open(devname_.c_str(), frame_width_, frame_height_, 30);
|
||||
if (cam == NULL) {
|
||||
printf("Camera open failed \n");
|
||||
return -1;
|
||||
@@ -166,7 +179,7 @@ public:
|
||||
{
|
||||
if (cam) {
|
||||
cam->camera_capture_stop(cam);
|
||||
camera_close(cam);
|
||||
hal_camera_close(cam);
|
||||
cam = NULL;
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user