Merge pull request #15 from m5stack/dev

Dev

.clang-format

@@ -163,5 +163,4 @@ StatementMacros:
   - QT_REQUIRE_VERSION
 TabWidth:        4
 UseCRLF:         false
-UseTab:          Never
-...
+UseTab:          Never

.github/workflows/benchmark.yml

@@ -0,0 +1,18 @@
+name: Benchmark Test
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - dev
+jobs:
+  build:
+    runs-on: [self-hosted, linux, arm64]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Start Benchmark Test
+        run: |
+          echo "This job runs on a self-hosted runner!"
+          echo "Running benchmark test..."
+          python3 benchmark/benchmodulellm.py

benchmark/README.md

@@ -0,0 +1,10 @@
+benchmodulellm can be used to test llm unit inference performance
+
+Only the llm unit definition files (model json) are required.
+
+If no model specified, it would benchmark default list. More model networks may be added later.
+
+Usage
+```shell
+python benchmodulellm.py --host 192.168.20.100 --port 10001 --test-items default.yaml
+```

benchmark/RESULTS.md

@@ -0,0 +1,39 @@
+# Results
+
+## ModuleLLM (AX630C)
+
+### LLM
+| model                           | ttft (ms)  | avg-token/s | model version | llm version |
+|---------------------------------|------------|-------------|---------------|-------------|
+| qwen2.5-0.5B-prefill-20e        | 359.8      | 10.32       | v0.2          | v1.8        |
+| qwen2.5-0.5B-p256-ax630c        | 1126.19    | 10.30       | v0.4          | v1.8        |
+| qwen2.5-0.5B-Int4-ax630c        | 442.95     | 12.52       | v0.4          | v1.8        |
+| qwen2.5-coder-0.5B-ax630c       | 361.81     | 10.28       | v0.2          | v1.8        |
+| qwen2.5-1.5B-ax630c             | 1029.41    | 3.59        | v0.3          | v1.8        |
+| qwen2.5-1.5B-p256-ax630c        | 3056.54    | 3.57        | v0.4          | v1.8        |
+| qwen2.5-1.5B-Int4-ax630c        | 1219.54    | 4.63        | v0.4          | v1.8        |
+| deepseek-r1-1.5B-ax630c         | 1075.04    | 3.57        | v0.3          | v1.8        |
+| deepseek-r1-1.5B-p256-ax630c    | 3056.86    | 3.57        | v0.4          | v1.8        |
+| llama3.2-1B-prefill-ax630c      | 891.00     | 4.48        | v0.2          | v1.8        |
+| llama3.2-1B-p256-ax630c         | 2601.11    | 4.49        | v0.4          | v1.8        |
+| openbuddy-llama3.2-1B-ax630c    | 891.02     | 4.52        | v0.2          | v1.8        |
+
+`The input text used by the llm test is "hello!“`
+
+### VLM
+| model                           | ttft (ms)  | avg-token/s | image encode (ms) | model version | vlm version |
+|---------------------------------|------------|-------------|-------------------|---------------|-------------|
+| internvl2.5-1B-364-ax630c       | 1117.27    | 10.56       | 1164.61           | v0.4          | v1.7        |
+| smolvlm-256M-ax630c             | 185.75     | 30.16       | 799.11            | v0.4          | v1.7        |
+| smolvlm-500M-ax630c             | 365.69     | 13.14       | 838.30            | v0.4          | v1.7        |
+
+`The image encoding test uses a jpg image with a size of 810*1080`
+
+### STT
+| model              | encode (ms) | avg-decode (ms) | model version | whisper version |
+|--------------------|-------------|-----------------|---------------|-----------------|
+| whisper-tiny       | 248.0       | 32.54           | v0.4          | v1.7            |
+| whisper-base       | 660.31      | 51.11           | v0.4          | v1.7            |
+| whisper-small      | 1606.08     | 148.92          | v0.4          | v1.7            |
+
+`The STT test uses a 30-second wav English audio`

benchmark/benchmodulellm.py

@@ -0,0 +1,126 @@
+import argparse
+import os
+import sys
+
+import yaml
+import logging
+
+from pathlib import Path
+
+from utils import LLMClient
+
+FILE = Path(__file__).resolve()
+ROOT = FILE.parents[0]
+if str(ROOT) not in sys.path:
+    sys.path.append(str(ROOT))
+ROOT = Path(os.path.relpath(ROOT, Path.cwd()))
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s - %(levelname)s - %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
+)
+
+def parse_opt(known=False):
+    """
+    Parse command-line options.
+    """
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--host", type=str, default="127.0.0.1", help="ModuleLLM IP Address")
+    parser.add_argument("--port", type=int, default=10001, help="ModuleLLM TCP Port")
+    parser.add_argument("--test-items", type=str, default=ROOT / "default.yaml", help="testitems.yaml path")
+
+    args = parser.parse_known_args()[0] if known else parser.parse_args()
+
+    return args
+
+def read_yaml(file_path):
+    """
+    Read a YAML file and return its content.
+    """
+    if not os.path.exists(file_path):
+        logging.error(f"YAML file '{file_path}' does not exist.")
+        sys.exit(1)
+    
+    try:
+        with open(file_path, "r") as file:
+            data = yaml.safe_load(file)
+            if data is None:
+                logging.warning(f"YAML file '{file_path}' is empty.")
+                return {}
+            
+            logging.info(f"YAML file '{file_path}' read successfully.")
+            
+            if "items" in data:
+                return data["items"]
+            else:
+                logging.warning(f"'items' not found in YAML file.")
+                return []
+    except Exception as e:
+        logging.error(f"Failed to read YAML file '{file_path}': {e}")
+        sys.exit(1)
+
+def write_yaml(file_path, data):
+    """
+    Write data to a YAML file.
+    """
+    try:
+        with open(file_path, "w") as file:
+            yaml.safe_dump(data, file)
+            logging.info(f"YAML file '{file_path}' written successfully.")
+    except Exception as e:
+        logging.error(f"Failed to write YAML file '{file_path}': {e}")
+        sys.exit(1)
+
+def categorize_and_deduplicate(items):
+    """
+    Categorize items by 'type' and remove duplicate 'model_name'.
+    """
+    categorized = {}
+    for item in items:
+        item_type = item.get("type")
+        model_name = item.get("model_name")
+        if not item_type or not model_name:
+            continue
+        
+        if item_type not in categorized:
+            categorized[item_type] = set()
+        
+        categorized[item_type].add(model_name)
+    
+    # Convert sets back to lists for easier usage
+    return {key: list(value) for key, value in categorized.items()}
+
+def main(opt):
+    items = read_yaml(opt.test_items)
+    if not items:
+        logging.warning(f"No items found in YAML file '{opt.test_items}'.")
+        return
+    
+    categorized_items = categorize_and_deduplicate(items)
+    
+    logging.info("Categorized items:")
+    for item_type, models in categorized_items.items():
+        logging.info(f"Type: {item_type}, Models: {models}")
+        
+        if item_type == "llm":
+            logging.info("Initializing LLMClient...")
+            llm_client = LLMClient(opt.host, opt.port)
+            
+            for model_name in models:
+                logging.info(f"Testing model: {model_name}")
+                input_text = "Tell me an adventure story."
+                try:
+                    result = llm_client.test(model_name, input_text)
+                    logging.info(f"Test result for model '{model_name}': {result}")
+                except Exception as e:
+                    logging.error(f"Error testing model '{model_name}': {e}")
+            
+            del llm_client
+            logging.info("LLMClient deleted successfully.")
+    
+    return categorized_items
+
+if __name__ == "__main__":
+    opt = parse_opt()
+    main(opt)

benchmark/default.yaml

@@ -0,0 +1,31 @@
+items:
+- model_name: qwen2.5-0.5B-p256-ax630c
+  type: llm
+- model_name: internvl2.5-1B-364-ax630c
+  type: vlm
+- model_name: whisper-tiny
+  type: whisper
+- model_name: whisper-base
+  type: whisper
+- model_name: whisper-small
+  type: whisper
+- model_name: sherpa-ncnn-streaming-zipformer-20M-2023-02-17
+  type: asr
+- model_name: sherpa-ncnn-streaming-zipformer-zh-14M-2023-02-23
+  type: asr
+- model_name: sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01
+  type: kws
+- model_name: sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01
+  type: kws
+- model_name: melotts-zh-cn
+  type: melotts
+- model_name: single_speaker_english_fast
+  type: tts
+- model_name: single_speaker_fast
+  type: tts
+- model_name: yolo11n
+  type: yolo
+- model_name: yolo11n-seg
+  type: yolo
+- model_name: yolo11n-pose
+  type: yolo

benchmark/utils/__init__.py

@@ -0,0 +1,3 @@
+from .llm import LLMClient
+
+__all__ = ["LLMClient"]

benchmark/utils/llm.py

@@ -0,0 +1,168 @@
+import socket
+import json
+import time
+import logging
+import uuid
+# from .token_calc import calculate_token_length
+
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+
+class LLMClient:
+    def __init__(self, host, port):
+        self.host = host
+        self.port = port
+        self.work_id = None
+        self.response_format = None
+        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        self.sock.connect((self.host, self.port))
+
+    def generate_request_id(self):
+        return str(uuid.uuid4())
+
+    def send_request_stream(self, request):
+        self.sock.sendall(json.dumps(request).encode('utf-8'))
+        response = b""
+        parsed_responses = []
+        output_text = ""
+        token_count = 0
+
+        start_time = time.time()
+        first_packet_time = None
+
+        while True:
+            chunk = self.sock.recv(4096)
+            response += chunk
+
+            while b'\n' in response:
+                line, response = response.split(b'\n', 1)
+                try:
+                    parsed_response = json.loads(line.decode('utf-8'))
+                    parsed_responses.append(parsed_response)
+
+                    if "data" in parsed_response and "delta" in parsed_response["data"]:
+                        if first_packet_time is None:
+                            first_packet_time = time.time()
+                        output_text += parsed_response["data"]["delta"]
+                        token_count += 3
+
+                    if "data" in parsed_response and parsed_response["data"].get("finish", False):
+                        end_time = time.time()
+                        total_time = end_time - start_time
+                        first_packet_latency = first_packet_time - start_time if first_packet_time else None
+
+                        # token_count = calculate_token_length(output_text)
+                        token_speed = token_count / total_time if total_time > 0 else 0
+
+                        logging.info("Stream reception completed.")
+                        logging.info("First packet latency: %.2f seconds", first_packet_latency if first_packet_latency else 0)
+                        logging.info("Total reception time: %.2f seconds", total_time)
+                        logging.info("Total tokens received: %d", token_count)
+                        logging.info("Token reception speed: %.2f tokens/second", token_speed)
+                        logging.info("Total output text length: %d characters", len(output_text))
+
+                        return {
+                            "responses": parsed_responses,
+                            "output_text": output_text,
+                            "token_count": token_count,
+                            "first_packet_latency": first_packet_latency,
+                            "total_time": total_time,
+                            "token_speed": token_speed
+                        }
+                except json.JSONDecodeError:
+                    logging.warning("Failed to decode JSON, skipping line.")
+                    continue
+
+    def send_request_non_stream(self, request):
+        self.sock.sendall(json.dumps(request).encode('utf-8'))
+        response = b""
+        while True:
+            chunk = self.sock.recv(4096)
+            response += chunk
+            if b'\n' in chunk:
+                break
+        return json.loads(response.decode('utf-8'))
+
+    def setup(self, model):
+        setup_request = {
+            "request_id": self.generate_request_id(),
+            "work_id": "llm",
+            "action": "setup",
+            "object": "llm.setup",
+            "data": {
+                "model": model,
+                "response_format": "llm.utf-8.stream",
+                "input": "llm.utf-8",
+                "enoutput": True,
+                "max_token_len": 256,
+                "prompt": "You are a knowledgeable assistant capable of answering various questions and providing information."
+            }
+        }
+        response = self.send_request_non_stream(setup_request)
+        self.work_id = response.get("work_id")
+        self.response_format = setup_request["data"]["response_format"]
+        return response
+
+    def inference(self, input_text):
+        if not self.work_id:
+            raise ValueError("work_id is not set. Please call setup() first.")
+        
+        inference_request = {
+            "request_id": self.generate_request_id(),
+            "work_id": self.work_id,
+            "action": "inference",
+            "object": self.response_format,
+            "data": {
+                "delta": input_text,
+                "index": 0,
+                "finish": True
+            }
+        }
+        if "stream" in self.response_format:
+            logging.info("Sending stream request...")
+            result = self.send_request_stream(inference_request)
+            return {
+                "output_text": result["output_text"],
+                "token_count": result["token_count"],
+                "first_packet_latency": result["first_packet_latency"],
+                "total_time": result["total_time"],
+                "token_speed": result["token_speed"]
+            }
+        else:
+            logging.info("Sending non-stream request...")
+            response = self.send_request_non_stream(inference_request)
+            return {
+                "output_text": response.get("data", ""),
+                "token_count": len(response.get("data", "").split())
+            }
+
+    def exit(self):
+        if not self.work_id:
+            raise ValueError("work_id is not set. Please call setup() first.")
+        
+        exit_request = {
+            "request_id": self.generate_request_id(),
+            "work_id": self.work_id,
+            "action": "exit"
+        }
+        response = self.send_request_non_stream(exit_request)
+        return response
+
+    def test(self, model, input_text):
+        logging.info("Setting up...")
+        setup_response = self.setup(model)
+
+        logging.info("Running inference...")
+        inference_result = self.inference(input_text)
+
+        logging.info("Exiting...")
+        exit_response = self.exit()
+
+        return {}
+
+if __name__ == "__main__":
+    host = "192.168.20.186"
+    port = 10001
+    client = LLMClient(host, port)
+    model_name = "qwen2.5-0.5B-p256-ax630c"
+    input_text = "This is a test input for the LLM."
+    client.test(model_name, input_text)

benchmark/utils/token_calc.py

@@ -0,0 +1,20 @@
+import tiktoken
+
+def calculate_token_length(input_string: str) -> int:
+    """
+    Calculate the token length of a given string using tiktoken.
+
+    Args:
+        input_string (str): The input string to calculate token length for.
+
+    Returns:
+        int: The length of the tokens.
+    """
+    # Initialize the tokenizer (you can specify a model if needed, e.g., 'gpt-4')
+    tokenizer = tiktoken.get_encoding("cl100k_base")
+    
+    # Encode the input string to tokens
+    tokens = tokenizer.encode(input_string)
+    
+    # Return the length of the tokens
+    return len(tokens)

doc/projects_llm_framework_doc/llm_asr_en.md

@@ -16,7 +16,7 @@ Send JSON:
   "action": "setup",
   "object": "asr.setup",
   "data": {
-    "model": "sherpa-ncnn-streaming-zipformer-zh-14M-2023-02-23",
+    "model": "sherpa-ncnn-streaming-zipformer-20M-2023-02-17",
     "response_format": "asr.utf-8.stream",
     "input": "sys.pcm",
     "enoutput": true,
@@ -34,7 +34,7 @@ Send JSON:
 - work_id: For configuration units, it is `asr`.
 - action: The method to be called is `setup`.
 - object: The type of data being transmitted is `asr.setup`.
-- model: The model used is the Chinese model `sherpa-ncnn-streaming-zipformer-zh-14M-2023-02-23`.
+- model: The model used is the Chinese model `sherpa-ncnn-streaming-zipformer-20M-2023-02-17`.
 - response_format: The result format is `asr.utf-8.stream`, a UTF-8 stream output.
 - input: The input is `sys.pcm`, representing system audio.
 - enoutput: Whether to enable user result output.
@@ -109,7 +109,7 @@ Example:
   "action": "setup",
   "object": "asr.setup",
   "data": {
-    "model": "sherpa-ncnn-streaming-zipformer-zh-14M-2023-02-23",
+    "model": "sherpa-ncnn-streaming-zipformer-20M-2023-02-17",
     "response_format": "asr.utf-8.stream",
     "input": [
       "sys.pcm",
@@ -119,9 +119,9 @@ Example:
     "endpoint_config.rule1.min_trailing_silence": 2.4,
     "endpoint_config.rule2.min_trailing_silence": 1.2,
     "endpoint_config.rule3.min_trailing_silence": 30.1,
-    "endpoint_config.rule1.must_contain_nonsilence": false,
-    "endpoint_config.rule2.must_contain_nonsilence": false,
-    "endpoint_config.rule3.must_contain_nonsilence": false
+    "endpoint_config.rule1.must_contain_nonsilence": true,
+    "endpoint_config.rule2.must_contain_nonsilence": true,
+    "endpoint_config.rule3.must_contain_nonsilence": true
   }
 }
 ```
@@ -310,7 +310,7 @@ Response JSON:
     "inputs_": [
       "sys.pcm"
     ],
-    "model": "sherpa-ncnn-streaming-zipformer-zh-14M-2023-02-23",
+    "model": "sherpa-ncnn-streaming-zipformer-20M-2023-02-17",
     "response_format": "asr.utf-8-stream"
   },
   "error": {

doc/projects_llm_framework_doc/llm_asr_zh.md

@@ -117,9 +117,9 @@ error::code 为 0 表示执行成功。
     "endpoint_config.rule1.min_trailing_silence": 2.4,
     "endpoint_config.rule2.min_trailing_silence": 1.2,
     "endpoint_config.rule3.min_trailing_silence": 30.1,
-    "endpoint_config.rule1.must_contain_nonsilence": false,
-    "endpoint_config.rule2.must_contain_nonsilence": false,
-    "endpoint_config.rule3.must_contain_nonsilence": false
+    "endpoint_config.rule1.must_contain_nonsilence": true,
+    "endpoint_config.rule2.must_contain_nonsilence": true,
+    "endpoint_config.rule3.must_contain_nonsilence": true
   }
 }
 ```

doc/projects_llm_framework_doc/llm_camera_en.md

@@ -15,25 +15,28 @@ Send JSON:
   "action": "setup",
   "object": "camera.setup",
   "data": {
-    "response_format": "camera.raw",
+    "response_format": "image.yuyv422.base64",
     "input": "/dev/video0",
     "enoutput": false,
     "frame_width": 320,
-    "frame_height": 320
+    "frame_height": 320,
+    "enable_webstream":false,
+    "rtsp":"rtsp.1280x720.h265"
   }
 }
 ```
 
-- request_id: Reference basic data explanation.
+- request_id: Refer to the basic data explanation.
 - work_id: When configuring the unit, it is `camera`.
-- action: The method being called is `setup`.
-- object: The data type being transmitted is `camera.setup`.
-- response_format: The returned result is `camera.raw`, which is in YUV422 format.
-- input: The name of the device being read.
-- frame_width: The output video frame width.
-- frame_height: The output video frame height.
-- enoutput: Whether to enable the user result output. If camera images are not needed, do not enable this parameter, as
-  the video stream will increase the communication load of the channel.
+- action: The method called is `setup`.
+- object: The type of data transmitted is `camera.setup`.
+- response_format: The output format is `image.yuyv422.base64`, which is in yuyv422 format. An optional format is image.jpeg.base64.
+- input: The device name to be read. Example: "/dev/video0", "axera_single_sc850sl"
+- frame_width: The width of the video frame output.
+- frame_height: The height of the video frame output.
+- enoutput: Whether to enable user result output. If you do not need to obtain camera images, do not enable this parameter, as the video stream will increase the communication pressure on the channel.
+- enable_webstream: Whether to enable webstream output, webstream will listen on tcp:8989 port, and once a client connection is received, it will push jpeg images in HTTP protocol multipart/x-mixed-replace type.
+- rtsp: Whether to enable rtsp stream output, rtsp will establish an RTSP TCP server at rtsp://{DevIp}:8554/axstream0, and you can pull the video stream from this port using the RTSP protocol. The video stream format is 1280x720 H265. Note that this video stream is only valid on the AX630C MIPI camera, and the UVC camera cannot use RTSP.
 
 Response JSON:
 
@@ -137,7 +140,7 @@ Response JSON:
   "created": 1731652344,
   "data": {
     "enoutput": false,
-    "response_format": "camera.raw",
+    "response_format": "image.yuyv422.base64",
     "input": "/dev/video0",
     "frame_width": 320,
     "frame_height": 320

doc/projects_llm_framework_doc/llm_camera_zh.md

@@ -15,11 +15,13 @@
   "action": "setup",
   "object": "camera.setup",
   "data": {
-    "response_format": "camera.raw",
+    "response_format": "image.yuyv422.base64",
     "input": "/dev/video0",
     "enoutput": false,
     "frame_width": 320,
-    "frame_height": 320
+    "frame_height": 320,
+    "enable_webstream":false,
+    "rtsp":"rtsp.1280x720.h265"
   }
 }
 ```
@@ -28,11 +30,13 @@
 - work_id：配置单元时，为 `camera`。
 - action：调用的方法为 `setup`。
 - object：传输的数据类型为 `camera.setup`。
-- response_format：返回结果为 `camera.raw`，是 yuv422 格式。
-- input：读取的设备名。
+- response_format：返回结果为 `image.yuyv422.base64`，是 yuyv422 格式。可选 image.jpeg.base64 格式输出。
+- input：读取的设备名。示例："/dev/video0", "axera_single_sc850sl"
 - frame_width：输出的视频帧宽。
 - frame_height：输出的视频帧高。
 - enoutput：是否起用用户结果输出。如果不需要获取摄像头图片，请不要开启该参数，视频流会增加信道的通信压力。
+- enable_webstream：是否启用 webstream 流输出，webstream 会监听 tcp:8989 端口，一但收到客户端连接，将会以 HTTP 协议 multipart/x-mixed-replace 类型推送 jpeg 图片。
+- rtsp：是否启用 rtsp 流输出，rtsp 会建立一个 rtsp://{DevIp}:8554/axstream0 RTSP TCP 服务端，可使用RTSP 协议向该端口拉取视频流。视频流的格式为 1280x720 H265。注意，该视频流只在 AX630C MIPI 摄像头上有效，UVC 摄像头无法使用 RTSP。
 
 响应 json：
 
@@ -136,7 +140,7 @@ error::code 为 0 表示执行成功。
   "created": 1731652344,
   "data": {
     "enoutput": false,
-    "response_format": "camera.raw",
+    "response_format": "image.yuyv422.base64",
     "input": "/dev/video0",
     "frame_width": 320,
     "frame_height": 320
@@ -151,4 +155,32 @@ error::code 为 0 表示执行成功。
 }
 ```
 
+获取本机的摄像头列表。
+
+发送 json：
+
+```json
+{
+  "request_id": "2",
+  "work_id": "camera",
+  "action": "list_camera"
+}
+```
+
+响应 json：
+
+```json
+{
+  "created":1746515639,
+  "data":[],
+  "error":{
+    "code":0,
+    "message":""
+    },
+  "object":"camera.devices",
+  "request_id":"2",
+  "work_id":"camera"
+}
+```
+
 > **注意：work_id 是按照单元的初始化注册顺序增加的，并不是固定的索引值。**  

doc/projects_llm_framework_doc/llm_depth_anything_en.md

@@ -27,7 +27,7 @@ Send JSON:
 - work_id: When configuring the unit, it is `depth_anything`.
 - action: The method called is `setup`.
 - object: The data type being transmitted is `depth_anything.setup`.
-- model: The model used is the `depth_anything` model.
+- model: The model used is the `depth-anything-ax630c` model.
 - response_format: The return result is `jpeg.base64.stream`.
 - input: The input is `camera.1001`, which refers to the input from the camera unit, as detailed in the camera unit
   documentation.

doc/projects_llm_framework_doc/llm_depth_anything_zh.md

@@ -27,7 +27,7 @@ depth_anything 视觉单元，用于提供图片深度信息。
 - work_id：配置单元时，为 `depth_anything`。
 - action：调用的方法为 `setup`。
 - object：传输的数据类型为 `depth_anything.setup`。
-- model：使用的模型为 `depth_anything` 模型。
+- model：使用的模型为 `depth-anything-ax630c` 模型。
 - response_format：返回结果为 `jpeg.base64.stream`。
 - input：输入的为 `camera.1001`，代表的是从 camera 单元内部输入，详见 camera 单位文档。
 - enoutput：是否启用用户结果输出。

doc/projects_llm_framework_doc/llm_kws_en.md

@@ -16,11 +16,12 @@ Send JSON:
   "action": "setup",
   "object": "kws.setup",
   "data": {
-    "model": "sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01",
+    "model": "sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01",
     "response_format": "kws.bool",
     "input": "sys.pcm",
     "enoutput": true,
-    "kws": "你好你好"
+    "kws": "HELLO",
+    "enwake_audio": true
   }
 }
 ```
@@ -29,11 +30,12 @@ Send JSON:
 - work_id: When configuring the unit, it is `kws`.
 - action: The method called is `setup`.
 - object: The type of data being transmitted is `kws.setup`.
-- model: The model used is the Chinese model `sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01`.
+- model: The model used is the Chinese model `sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01`.
 - response_format: The result returned is in `kws.bool` format.
 - input: The input is `sys.pcm`, representing system audio.
 - enoutput: Whether to enable user result output.
 - kws: The Chinese wake-up word is `"你好你好"`.
+- enwake_audio: Whether to enable wake-up audio output. Default is true.
 
 Response JSON:
 
@@ -204,7 +206,7 @@ Response JSON:
     "inputs_": [
       "sys.pcm"
     ],
-    "model": "sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01",
+    "model": "sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01",
     "response_format": "kws.bool"
   },
   "error": {

doc/projects_llm_framework_doc/llm_kws_zh.md

@@ -19,7 +19,8 @@
     "response_format": "kws.bool",
     "input": "sys.pcm",
     "enoutput": true,
-    "kws": "你好你好"
+    "kws": "你好你好",
+    "enwake_audio": true
   }
 }
 ```
@@ -33,6 +34,7 @@
 - input：输入的为 `sys.pcm`,代表的是系统音频。
 - enoutput：是否起用用户结果输出。
 - kws：中文唤醒词为 `"你好你好"`。
+- enwake_audio：是否起用唤醒音频输出。默认是 true
 
 响应 json：
 

doc/projects_llm_framework_doc/llm_melotts_en.md

@@ -16,7 +16,7 @@ Send JSON:
   "action": "setup",
   "object": "melotts.setup",
   "data": {
-    "model": "melotts_zh-cn",
+    "model": "melotts-en-us",
     "response_format": "sys.pcm",
     "input": "tts.utf-8",
     "enoutput": false
@@ -28,7 +28,7 @@ Send JSON:
 - work_id: For configuration, it is `melotts`.
 - action: The method to be called is `setup`.
 - object: The data type being transmitted is `melotts.setup`.
-- model: The model being used is the Chinese model `melotts_zh-cn`.
+- model: The model being used is the English model `melotts-en-us`.
 - response_format: The result is returned as `sys.pcm`, system audio data, which is directly sent to the llm-audio
   module for playback.
 - input: The input is `tts.utf-8`, representing user input.
@@ -139,7 +139,7 @@ Example:
   "action": "setup",
   "object": "melotts.setup",
   "data": {
-    "model": "melotts_zh-cn",
+    "model": "melotts-en-us",
     "response_format": "sys.pcm",
     "input": [
       "tts.utf-8",
@@ -335,7 +335,7 @@ Response JSON:
     "inputs_": [
       "tts.utf-8"
     ],
-    "model": "melotts_zh-cn",
+    "model": "melotts-en-us",
     "response_format": "sys.pcm"
   },
   "error": {

doc/projects_llm_framework_doc/llm_melotts_zh.md

@@ -15,7 +15,7 @@
   "action": "setup",
   "object": "melotts.setup",
   "data": {
-    "model": "melotts_zh-cn",
+    "model": "melotts-zh-cn",
     "response_format": "sys.pcm",
     "input": "tts.utf-8",
     "enoutput": false
@@ -27,7 +27,7 @@
 - work_id：配置单元时，为 `melotts`。
 - action：调用的方法为 `setup`。
 - object：传输的数据类型为 `melotts.setup`。
-- model：使用的模型为 `melotts_zh-cn` 中文模型。
+- model：使用的模型为 `melotts-zh-cn` 中文模型。
 - response_format：返回结果为 `sys.pcm`, 系统音频数据，并直接发送到 llm-audio 模块进行播放。
 - input：输入的为 `tts.utf-8`,代表的是从用户输入。
 - enoutput：是否起用用户结果输出。
@@ -134,7 +134,7 @@ error::code 为 0 表示执行成功。
   "action": "setup",
   "object": "melotts.setup",
   "data": {
-    "model": "melotts_zh-cn",
+    "model": "melotts-zh-cn",
     "response_format": "sys.pcm",
     "input": [
       "tts.utf-8",
@@ -328,7 +328,7 @@ error::code 为 0 表示执行成功。
     "inputs_": [
       "tts.utf-8"
     ],
-    "model": "melotts_zh-cn",
+    "model": "melotts-zh-cn",
     "response_format": "sys.pcm"
   },
   "error": {

doc/projects_llm_framework_doc/llm_tts_en.md

@@ -16,7 +16,7 @@ Send JSON:
   "action": "setup",
   "object": "tts.setup",
   "data": {
-    "model": "single_speaker_fast",
+    "model": "single-speaker-english-fast",
     "response_format": "sys.pcm",
     "input": "tts.utf-8",
     "enoutput": false
@@ -28,7 +28,7 @@ Send JSON:
 - work_id: For configuring the unit, it is `tts`.
 - action: The method to call is `setup`.
 - object: The type of data being transmitted is `tts.setup`.
-- model: The model used is the `single_speaker_fast` Chinese model.
+- model: The model used is the `single-speaker-english-fast` English model.
 - response_format: The returned result is `sys.pcm`, system audio data, which is directly sent to the llm-audio module
   for playback.
 - input: Input is `tts.utf-8`, representing user input.
@@ -46,13 +46,50 @@ Response JSON:
   },
   "object": "None",
   "request_id": "2",
-  "work_id": "llm.1003"
+  "work_id": "tts.1003"
 }
 ```
 
 - created: Message creation time, in Unix time.
 - work_id: The successfully created work_id unit.
 
+## inference
+
+### streaming input
+
+```json
+{
+    "request_id": "2",
+    "work_id": "tts.1003",
+    "action": "inference",
+    "object": "tts.utf-8.stream",
+    "data": {
+        "delta": "What's ur name?",
+        "index": 0,
+        "finish": true
+    }
+}
+```
+- object: The data type transmitted is tts.utf-8.stream, indicating a streaming input from the user's UTF-8.
+- delta: Segment data of the streaming input.
+- index: Index of the segment in the streaming input.
+- finish: A flag indicating whether the streaming input has completed.
+
+### non-streaming input
+
+```json
+{
+    "request_id": "2",
+    "work_id": "tts.1003",
+    "action": "inference",
+    "object": "tts.utf-8",
+    "data": "What's ur name?"
+}
+```
+
+- object: The data type transmitted is tts.utf-8, indicating a non-streaming input from the user's UTF-8.
+- data: Data for non-streaming input.
+
 ## link
 
 Link the output of the upper unit.
@@ -102,7 +139,7 @@ Example:
   "action": "setup",
   "object": "tts.setup",
   "data": {
-    "model": "single_speaker_fast",
+    "model": "single-speaker-fast",
     "response_format": "sys.pcm",
     "input": [
       "tts.utf-8",
@@ -298,7 +335,7 @@ Response JSON:
     "inputs_": [
       "tts.utf-8"
     ],
-    "model": "single_speaker_fast",
+    "model": "single-speaker-fast",
     "response_format": "sys.pcm"
   },
   "error": {