Compare commits

...
Author SHA1 Message Date
Justin Mitchell a39feb4a88 Add support for wrapped subtitles
Update freeink-sdk submodule from d6d40be to 58d34b3.
2026-08-13 05:35:47 -04:00
Justin Mitchell b1dec6491d Remove unused BoardPaperS3 include from HalPowerManager 2026-08-13 05:28:55 -04:00
Justin Mitchell 1915457562 Add server-side search support to plugin catalogs
Implements search functionality for plugin catalogs with configurable search URL and body templates. Adds search icon to browsing header, keyboard entry for queries, and URL encoding for query parameters. Search results use the same item format as browse results. Includes navigation improvements to prevent button press fall-through and proper authentication gating for search requests.
2026-08-13 05:27:02 -04:00
Justin Mitchell 54561e8df3 Fix plugin job lease expiration and completion retry
Separate job lease expiration from slot reuse logic. Jobs that exceed the lease timeout are now requeued as PENDING instead of being treated as stale slots. Add retry logic for job completion requests - if completion fails, it will be retried on the next poll. Also allow completion endpoint to be idempotent by accepting already-completed jobs.
2026-08-13 03:47:13 -04:00
Justin Mitchell 06f6ecc27b Report download progress even when total size is unknown
Remove the check for sink.total > 0 before calling progress callback. This allows progress to be reported for downloads where the total size is not known in advance (e.g., chunked transfer encoding).
2026-08-12 16:00:24 -04:00
Justin Mitchell 58932354bd Handle short writes when saving plugin tokens
Check that the full token was written to storage. If a short write occurs (e.g., SD card full or I/O fault), remove the truncated token file and return false to prevent pollAuth from reporting success while later loadToken() calls would fail.
2026-08-12 15:52:05 -04:00
Justin Mitchell 47021955d4 Unify confirm button labels for pager and fetch actions
Standardize the footer hint behavior across OPDS browser and plugin catalog. Pager rows (prev/next) and book/item entries now consistently show 'Fetch' for server operations and 'Open' for local navigation. Track pager row presence with boolean flags to determine correct label.
2026-08-12 15:31:46 -04:00
Justin Mitchell 7a5f12863c Remove arrow symbols from pagination strings
Remove the « and » arrow symbols from STR_NEXT_PAGE and STR_PREV_PAGE translation strings across all languages. This simplifies the pagination UI text to just the page navigation labels without directional indicators.
2026-08-12 15:25:31 -04:00
Justin Mitchell c64b0ceec2 Add shared UI screen builders for catalog activities
Introduces reusable FreeInkUI screen components for OPDS book browser and SD-plugin catalogs. Provides functions for rendering themed headers with optional action buttons, centered message blocks, download progress screens, and list bodies with viewport syncing.
2026-08-12 15:15:42 -04:00
Justin Mitchell 69d7323888 Merge remote-tracking branch 'origin/develop' into feat-sd-web-plugins 2026-08-12 14:58:08 -04:00
Julia e970abae89 feat: skip boot screen on startup (#2989) 2026-08-12 21:45:21 +03:00
Justin Mitchell 9e3cea2e54 feat: convert all lists and tabs to FUI (#2957) 2026-08-12 20:40:09 +03:00
Justin Mitchell 640e421e5d Refactor plugin catalog and webserver IO
Consolidates repeated IO, HTTP and error-handling logic across plugin/catalog and webserver code. Key changes:
- lib/Epub/Epub.cpp: use extractItemToFile for cover extraction instead of manual write/read.
- src/activities/plugins/PluginCatalogActivity.*: add FsHelpers, pick(), use decodeUriEscapes, introduce fail(), beginLoading(), substitutedHeaders(), openApiClient(), and centralize apiRequest; replace many duplicated header builds and error-state code.
- src/activities/plugins/PluginInfoActivity.cpp: reserve() paragraphs/lines to reduce reallocations.
- src/network/CrossPointWebServer.*: add streamFileToClient() and readJsonBody() to centralize file streaming and JSON body parsing; include ArduinoJson in header.
Reduces duplication, improves robustness and minor perf gains.
2026-08-11 23:59:48 -04:00
Justin Mitchell 94400ed2ea Limit random allocation; ensure plugin dirs
Prevent runaway allocations and improve plugin FS handling.

- src/network/CrossPointWebServer.cpp: In handleCrypto() clamp requested random byte length to a new kMaxRandomBytes (4096) and allocate exactly n bytes to avoid large/unbounded allocations. Also, in handlePluginFs() ensure the parent directory for the requested path is created by calling Storage.ensureDirectoryExists(path.substr(0, lastSlash)) instead of only /.crosspoint, covering nested plugin paths.

- src/network/html/shared/PluginHost.js.inc: Replace innerHTML error string with safe DOM construction and container.replaceChildren(...) to avoid HTML injection and ensure consistent DOM updates.
2026-08-11 23:40:32 -04:00
Justin Mitchell b5a9411a8d Merge remote-tracking branch 'origin/develop' into feat-sd-web-plugins 2026-08-11 18:01:17 -04:00
Justin Mitchell 096223f808 Reformat code for improved readability
Break long lines and switch cases onto multiple lines to improve code readability. No functional changes.
2026-08-11 17:59:36 -04:00
Justin Mitchell ec0cd48b76 Improve plugin catalog navigation and labels
Replace hardcoded '> ' prefix with direction markers from translated strings. Change 'Download' button to 'Fetch' for consistency. Add up/down navigation labels when multiple items exist. This allows better localization of navigation UI elements.
2026-08-11 17:05:50 -04:00
Justin Mitchell 45abc21f6d Stream catalog responses to temp file to reduce RAM
Browse responses (JSON/XML) now stream to an SD temp file instead of being buffered in memory, allowing large catalog pages (60+ KB) to work on low-heap devices. XML parsing reads the file in 2KB chunks. Also adds optional browse.lists feature for named sub-catalogs with a picker UI, and keeps TLS session alive across requests to avoid heap fragmentation.
2026-08-11 16:49:51 -04:00
Thiago Kenji Okada 86eb1cbde2 fix: properly link AGENTS.md (#2976) 2026-08-11 22:24:24 +03:00
Uri Tauber 2cac597143 fix symlink AGENTS.md 2026-08-11 08:24:13 +03:00
Julia 7212129318 fix: ignore empty watchdog reports (#2968)
## Summary

* **What is the goal of this PR?**
* Prevent reset-only watchdog events from opening the crash-report flow
with an empty panic reason.

* **What changes are included?**
* Record a marker in RTC memory only when panic diagnostics are
captured.
  * Continue reporting explicit panics and CPU lockups.
* Require that marker before treating watchdog resets as crashes, and
clear it with the existing panic state.


## Scope Check

CrossPoint is intentionally narrow. See
[SCOPE.md](../blob/master/SCOPE.md) and
[ROADMAP.md](../blob/master/ROADMAP.md).
Please confirm:

- [x] I have read SCOPE.md and ROADMAP.md.
- [x] This PR is **not** a new built-in theme (themes are temporarily
closed pending the move to SD-loaded themes).
- [x] This PR is **not** a new external network connector (sync engine,
cloud storage, remote file access, etc.).
- [x] This PR is **not** an interactive app, writing tool,
RSS/news/browser, media playback, or PDF feature.
- [x] The stock firmware does not already handle this well, **and** no
other popular CrossPoint fork already does
      (or, if one does, I explain why CrossPoint still needs it below).
- [x] If this PR touches `freeink-sdk/`, `lib/hal/`, the bootloader,
OTA, or recovery code, I have coordinated with
      the relevant maintainer.

**If this PR was opened against the previous (broader) scope and was
already in flight under Phase 0, link the
relevant Discussion or issue so reviewers can see the history.**

## Additional Context

* This is a focused port of CrossInk commit `c96d4275`.
* The change adds 4 bytes of RTC no-init state; it does not affect
storage formats, normal boot routing, or real panic reporting.
* Hardware follow-up: on an X3/X4, confirm a reset-only watchdog boots
normally without an empty crash report, while a genuine panic still
reaches the crash-report screen.
---

### AI Usage

While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.

Did you use AI tools to help write this code? _**<YES>**_
2026-08-11 03:02:56 +00:00
Justin Mitchell 9220e5a301 Refactor plugin discovery with structured scanning
Replace inline findPluginDir with scanPlugins() that returns structured Entry objects containing plugin metadata (plugin.js, device.json, manifest.json markers). This centralizes plugin discovery logic and provides a single source of truth for what constitutes a valid plugin across all roots.
2026-08-10 20:32:12 -04:00
Justin Mitchell 28302714df Add multi-file bundle download support to plugin catalog
Enables downloading multiple files from a catalog item instead of just one. Each item can specify a base URL and an array of relative file paths. All files are fetched into a subdirectory with path traversal protection. This supports use cases like plugin installers and theme packs that consist of multiple files.
2026-08-10 19:31:29 -04:00
Justin Mitchell cb22cb5291 Merge remote-tracking branch 'origin/develop' into feat-sd-web-plugins 2026-08-10 16:36:27 -04:00
Justin Mitchell 9b090266cf fix: MAC address reading when WiFi is off (#2960) 2026-08-10 15:38:58 -04:00
William Floyd e9c3a166f5 feat: StarDict .syn synonym lookup support (#2696) 2026-08-10 20:01:28 +03:00
Justin Mitchell d720891b2f feat: Adds dark mode with image polarity preservation (#2705) 2026-08-10 19:58:22 +03:00
e4aae7baef feat: retain custom sleep screen during wake (#2943)
Co-authored-by: Bridger Brundy <bridgerborpcc@gmail.com>
Co-authored-by: Julia <julia@uxj.io>
2026-08-10 17:02:20 +03:00
Uri Tauber 7f8b005ae1 chore: update AI agents instructions 2026-08-10 16:06:51 +03:00
Serena 81028d5849 fix(ui): wrap button hint labels that overflow their button (#2928) 2026-08-10 12:05:27 +03:00
Justin Mitchell 0423276d2c fix: accept any 2xx from KOSync-compatible servers (#2945) 2026-08-10 09:26:37 +03:00
Leopoldo Pla Sempere 596f0d3f71 fix: invalidate stale footnote caches (#2935) 2026-08-09 19:27:48 +03:00
hooligan333andErica Jensen 6af9a04934 perf: cache cumulative spine sizes in RAM (#2441)
Co-authored-by: Erica Jensen <erica@mailershaven.com>
2026-08-09 17:31:42 +03:00
François Chaix 1582e70ef4 fix(epub): bump FOOTNOTE_HREF_LEN 96→256 for long calibre paths (#2722) 2026-08-09 17:23:40 +03:00
Uri Tauber d4733bfc65 chore: split GH build action (#2901)
## Summary

Build action take too much time.
Lets see if splitting it to two seperate action would speed things up.

---

### AI Usage

While CrossPoint doesn't have restrictions on AI tools in contributing,
please be transparent about their usage as it
helps set the right context for reviewers.

Did you use AI tools to help write this code? _**< PARTIALLY >**_
2026-08-09 03:55:33 -04:00
Justin Mitchell ad30652f52 Add download completion state to plugin catalog
Introduces a DONE state to show download completion message with confirmation before returning to browsing. Refactors layout calculations into a reusable CatLayout structure that uses themed metrics for consistent list geometry across screens. Removes hardcoded visibleRows() method in favor of layout-based page item calculation.
2026-08-07 03:45:53 -04:00
Justin Mitchell dbcdace3cf Add plugin menu to home screen when plugins installed
The home screen now shows a Plugins menu item (instead of OPDS Browser) when any plugin is installed. The library slot dynamically switches between Plugins and OPDS Browser based on plugin presence. Also refactors PluginInfoActivity to use theme-based geometry and defers text wrapping until render time when font metrics are available.
2026-08-07 03:20:54 -04:00
Justin Mitchell 41b5c2e498 Add plugin discovery and info screen documentation
Documents the new plugin discovery system where all plugins appear in Settings → System → Plugins with title/description, and selecting one shows an info screen with README and optional Open action. Also adds guidance on choosing between on-device (device.json) vs browser (plugin.js) implementations with concrete examples for each approach.
2026-08-07 02:32:29 -04:00
Justin Mitchell cec96908c2 Add password grant OAuth flow for plugins
Adds support for silent password-based OAuth authentication alongside the existing device code flow. Password grant automatically mints tokens from stored credentials and refreshes them on 401/403 responses, enabling standalone reader operation after initial web-based credential setup.
2026-08-07 01:57:12 -04:00
Justin Mitchell 529c8a6398 Update freeink-sdk submodule
Update freeink-sdk submodule from e52d480 to a485dc4.
2026-08-07 01:37:12 -04:00
Justin Mitchell e7da410a10 Add XML browse format support for SD plugins
Introduces an alternative XML browse format alongside the existing JSON format. XML mode supports folder navigation, WebDAV/OPDS feeds, and custom element selectors. Includes config file templating for user credentials, URL resolution, and file extension filtering. XML responses are limited to 200 entries per folder due to 48KB RAM constraint.
2026-08-07 01:01:41 -04:00
Justin Mitchell 217def55bf Add utility for locating plugins in multiple SD roots
Introduces PluginLocations namespace with findPluginDir() function that searches for plugins across multiple SD card directories. Supports legacy /.crosspoint/plugins path and user-friendly /plugins and /.plugins locations, with earlier roots taking precedence on name collisions.
2026-08-06 22:40:30 -04:00
Justin Mitchell 8bffe852fd Add plugin catalog browser activity
Implements a generic on-device catalog browser driven by plugin device.json manifests. Allows SD card plugins to define browsing/downloading behavior via declarative JSON templates (URL/header/body with token/pagination support) without requiring firmware changes. Includes WiFi checking, catalog browsing, and download states.
2026-08-06 22:34:25 -04:00
Justin Mitchell 000824f5ca Add plugin job queue system with REST API
Implements a fixed-size job queue for SD-plugin execution via external systems. Jobs are submitted via REST API, claimed by browser-hosted plugin runners, and results posted back. Includes stale job detection (10min timeout) and automatic slot recycling for oldest finished jobs. Adds /plugins-run headless executor page and five API endpoints for job lifecycle management.
2026-08-06 21:53:33 -04:00
Justin Mitchell 66ec7dbe4c Update freeink-sdk submodule
Update freeink-sdk submodule from a485dc4 to e52d480.
2026-08-06 21:42:42 -04:00
Justin Mitchell 506ce26d05 Fix line formatting in LOG_ERR call
Consolidate multi-line LOG_ERR statement into a single line for better code readability.
2026-08-05 13:14:05 -04:00
Justin Mitchell 2a5f3d2a54 Merge remote-tracking branch 'origin/develop' into feat-sd-web-plugins
# Conflicts:
#	src/network/html/SettingsPage.html
2026-08-05 13:08:13 -04:00
Justin Mitchell 5acd49fb10 Merge remote-tracking branch 'origin/develop' into feat-sd-web-plugins 2026-07-31 12:33:27 -04:00
Justin Mitchell b4b4e8fef2 Add chunked resumable downloads for fetch API
Implement segmented file downloads to prevent mobile browser timeouts on large files. The server now supports offset-based resumption and segment size limits (4MB max). The client automatically chunks requests into 2MB segments and resumes from the last written offset if interrupted. This prevents browsers from terminating long-running downloads while allowing the device to continue fetching in manageable chunks.
2026-07-29 18:46:13 -04:00
Justin Mitchell af688c56ed Reduce heap logging frequency in web server
Increase heap log interval from 64KB to 1MB to reduce logging overhead and improve performance during data transmission.
2026-07-29 18:15:28 -04:00
Justin Mitchell a41c4ae02e Add keep-alive mechanism for long-running fetch requests
Prevents mobile browsers from discarding HTTP responses during long upstream downloads by sending periodic whitespace chunks. Implements chunked transfer encoding that starts after 5 seconds and sends heartbeat data every 5 seconds to maintain the browser connection. Refactors header handling and response sending into helper lambdas for cleaner code.
2026-07-29 18:14:32 -04:00
Justin Mitchell 0ca93a8653 Release WebServer request memory after handling
Introduce CrossPointHttpServer subclass to expose memory release for parsed request arguments and POST bodies. This prevents large JSON request bodies from remaining resident between requests, freeing memory for outbound TLS operations. Add explicit cleanup calls before TLS handshakes and improve heap logging to track max allocatable block size.
2026-07-29 17:49:52 -04:00
Justin Mitchell f3d4702fa8 Merge remote-tracking branch 'origin/develop' into feat-sd-web-plugins 2026-07-29 16:28:26 -04:00
Justin Mitchell 2e9e643ab0 Refactor watchdog pause/restore into class methods
Extract lambda functions for pausing and restoring the web task watchdog into proper class methods. This improves code reusability and adds error checking for watchdog operations. The watchdog is paused during HTTP transactions to prevent timeouts when NetworkClient::write() blocks for up to 10 seconds, which exceeds the 5-second watchdog window.
2026-07-29 15:29:57 -04:00
Justin Mitchell d992bc0ff4 Add methods to suspend/resume transfer services
Introduces suspendTransferServices() and resumeTransferServices() methods to temporarily halt WebSocket server and discovery UDP during fetch operations. This frees up buffer memory for TLS operations, which require approximately 4.4KB of heap space during large transfers.
2026-07-27 23:45:23 -04:00
Justin Mitchell 27ec21dbb8 Feed task watchdog during slow HTTP requests
Add watchdog feeding callback to prevent WDT timeouts when blocking on slow peer responses (e.g., account sign-in >5s). The callback is passed to SecureHttpClient which polls it in wait loops to keep the task watchdog fed without aborting the request.
2026-07-27 23:36:11 -04:00
Justin Mitchell d6600ed0a8 Improve download resilience with smarter retry logic
Replace single retry counter with separate stalled-attempt and total-attempt limits. This allows downloads to recover from early truncations (common in first ~100KB during heap settling) while still protecting against dead servers. Also removes HAVE_MAX_FRAGMENT flag due to memory corruption issues with ADEPT servers and removes unused decrypt method from ContentProtection.
2026-07-27 23:19:26 -04:00
Justin Mitchell ac74a60efb Enable TLS max fragment length negotiation
Configure wolfSSL to negotiate 2KB TLS records instead of default 16KB to avoid memory allocation failures during large SD downloads when heap is constrained to ~20KB free.
2026-07-27 22:32:41 -04:00
Justin Mitchell 7a89475c80 Increase HTTP timeout for fulfillment server downloads
Fulfillment servers can stall mid-body while assembling books on the fly. The default 15s timeout was truncating these downloads, so increase to 60s to allow packaging to complete.
2026-07-27 22:00:01 -04:00
Justin Mitchell 41922b55c5 Detect and report truncated file downloads
Add validation to detect incomplete HTTP responses after successful status codes. When a download is truncated due to transport issues or timeouts, delete the partial file and return a 502 error with details about whether it was an SD write failure or download truncation. This prevents partial files from appearing as successful downloads.
2026-07-27 21:59:12 -04:00
Justin Mitchell 31f2bb71ae Add memory safety checks to prevent OOM aborts
Implement safeguards against out-of-memory conditions in DRM decryption and HTTP relay. Cap rights document size at 64KB with heap check before allocation. Limit HTTP relay responses to 32KB with streaming validation to prevent unbounded memory growth. Use makeUniqueNoThrow for safer allocation. These changes prevent abort() calls from bad_alloc under -fno-exceptions.
2026-07-27 14:29:58 -04:00
Justin Mitchell c405c223c2 Pre-scan ZIP before initializing crypto for EPUBs
Perform ZIP scan upfront to check for encryption.xml before initializing crypto or loading credentials. This allows plain EPUBs to return immediately through the normal reader path, while protected EPUBs still only scan once by reusing the same ZIP index via openFromScan().
2026-07-27 00:38:48 -04:00
Justin Mitchell b95e0bcd48 Remove redundant ZIP scan in ContentProtection
Delegate encryption detection to ProtectedBook class instead of performing a separate preliminary ZIP scan. ProtectedBook now handles the authoritative scan and determines protection status, including distinguishing plain EPUBs from font-obfuscation-only cases.
2026-07-26 23:28:49 -04:00
Justin Mitchell 90581d1ba4 Stream decryption to avoid buffering in memory
Replace in-memory vector buffering with sink-based streaming for decryption. Adds decryptedSize() and decryptToSink() methods to allow writing decrypted content directly to output buffers or streams, reducing memory usage for large encrypted files.
2026-07-26 23:00:42 -04:00
Justin Mitchell 5987594b2d Remove plugin host allowlist security mechanism
Removes the allowedHosts manifest validation that restricted which external hosts plugins could access via the relay endpoint. Also updates comments throughout to use more generic 'encrypted-entry' terminology instead of 'protected content', and adds NO_WOLFSSL_ESP32_CRYPT_RSA_PRI flag to work around ESP32-C3 hardware RSA limitations with 4096-bit keys.
2026-07-26 22:49:33 -04:00
Justin Mitchell cd09c9c748 Disable TLS peer verification for SecureNet transport
SecureNet transport does not ship with a CA bundle, causing peer verification to fail with wolfSSL error -188. Set connections to insecure mode to maintain TLS encryption without authentication
2026-07-26 21:17:53 -04:00
Justin Mitchell 41f17f1b5c Reformat code and reorganize include statements
Alphabetize include statements and apply consistent formatting to if-else blocks. No functional changes.
2026-07-26 18:56:11 -04:00
Justin Mitchell afd6162e6a Add dedicated color variables for plugin controls
Introduce plugin-control-bg and plugin-control-border CSS variables for both light and dark themes to replace generic card-bg and border-color. Add subtle inset box-shadow for depth.
2026-07-26 18:44:40 -04:00
Justin Mitchell e017d099e2 Allow opening EPUBs with font obfuscation only
Previously, any EPUB with encryption.xml would fail to open without a credential, even if it only used font obfuscation (not DRM). Now credentials are optional until actual content protection is detected. Added user-facing error message for genuinely DRM-protected books and styling for SD-card plugin UI in File Manager.
2026-07-26 18:43:06 -04:00
Justin Mitchell ca8e135276 Add HTML include directive resolution
Implements a function to inline <!--#include file="path" --> directives in HTML files. The function resolves includes relative to a base directory and supports up to 8 levels of nesting. Must be called before HTML minification since minification strips comments.
2026-07-26 18:22:44 -04:00
Justin Mitchell c6a968d0e1 Add content protection SDK binding for SD storage
Implements HAL binding layer that connects the storage-agnostic ContentProtection SDK to the device's SD card storage. Provides SdByteSource for reading encrypted files, credential lookup from provisioned keys, and openProtectedBook() entry point for the reader. This glue code lives in firmware to keep the portable SDK library HAL-independent.
2026-07-26 17:29:02 -04:00
Justin Mitchell cb056d50b6 Add browser-side plugin system with SD card support
Implements a plugin architecture that allows JavaScript bundles stored on SD card (/.crosspoint/plugins/) to extend the web UI. Plugins can mount to Settings or File Manager pages and access device capabilities including HTTPS relay, crypto primitives, SD file operations, and URL downloads. Includes path validation and file serving with proper content types.
2026-07-26 17:17:10 -04:00
179 changed files with 12021 additions and 4454 deletions

No files matched your search

+52 -8
View File
@@ -63,7 +63,8 @@ jobs:
- name: Run cppcheck
run: pio check --fail-on-defect low --fail-on-defect medium --fail-on-defect high
build:
build-default:
name: Build default
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
@@ -90,14 +91,10 @@ jobs:
key: pio-${{ runner.os }}-${{ hashFiles('platformio.ini') }}
restore-keys: pio-${{ runner.os }}-
- name: Build CrossPoint
# Both envs in ONE pio invocation: a second `pio run` can wipe the whole
# .pio/build tree (project-checksum mismatch after the S3 toolchain
# installs into the restored cache), deleting the default firmware.bin
# before the artifact upload.
- name: Build default firmware
run: |
set -euo pipefail
pio run -e default -e sticky | tee pio.log
pio run -e default | tee pio.log
- name: Extract firmware stats
@@ -120,6 +117,52 @@ jobs:
path: .pio/build/default/firmware.bin
if-no-files-found: error
build-sticky:
name: Build sticky
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
submodules: recursive
- uses: actions/setup-python@v6
with:
python-version: '3.14'
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
version: "latest"
enable-cache: false
- name: Install PlatformIO Core
run: uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.19.zip
- name: Cache PlatformIO packages
uses: actions/cache@v4
with:
path: ~/.platformio
key: pio-${{ runner.os }}-${{ hashFiles('platformio.ini') }}
restore-keys: pio-${{ runner.os }}-
- name: Build sticky firmware
run: |
set -euo pipefail
pio run -e sticky | tee pio.log
- name: Extract firmware stats
run: |
set -euo pipefail
ram_line="$(grep -E "RAM:\\s" -m1 pio.log || true)"
flash_line="$(grep -E "Flash:\\s" -m1 pio.log || true)"
echo "ram_line=${ram_line}" >> "$GITHUB_OUTPUT"
echo "flash_line=${flash_line}" >> "$GITHUB_OUTPUT"
{
echo "## Sticky firmware build stats"
if [ -n "$ram_line" ]; then echo "- ${ram_line}"; else echo "- RAM: not found"; fi
if [ -n "$flash_line" ]; then echo "- ${flash_line}"; else echo "- Flash: not found"; fi
} >> "$GITHUB_STEP_SUMMARY"
unit-tests:
runs-on: ubuntu-latest
steps:
@@ -152,7 +195,8 @@ jobs:
test-status:
name: Test Status
needs:
- build
- build-default
- build-sticky
- clang-format
- cppcheck
- unit-tests
-47
View File
@@ -1,47 +0,0 @@
name: Compile Release
on:
push:
tags:
- '*'
jobs:
build-release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
submodules: recursive
- uses: actions/setup-python@v6
with:
python-version: '3.14'
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
version: "latest"
enable-cache: false
- name: Install PlatformIO Core
run: uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.19.zip
- name: Cache PlatformIO packages
uses: actions/cache@v4
with:
path: ~/.platformio
key: pio-${{ runner.os }}-${{ hashFiles('platformio.ini') }}
restore-keys: pio-${{ runner.os }}-
- name: Build CrossPoint
run: pio run -e gh_release
- name: Upload Artifacts
uses: actions/upload-artifact@v4
with:
name: CrossPoint-${{ github.ref_name }}
path: |
.pio/build/gh_release/bootloader.bin
.pio/build/gh_release/firmware.bin
.pio/build/gh_release/firmware.elf
.pio/build/gh_release/firmware.map
.pio/build/gh_release/partitions.bin
+2
View File
@@ -31,3 +31,5 @@ lib/EpdFont/scripts/output/
/dependencies.lock
/sdkconfig.*
sdkconfig.sticky
sdkconfig.x4pro
flake.*
+179 -68
View File
File diff suppressed because it is too large. Load diff
Symlink
+1
View File
@@ -0,0 +1 @@
.skills/SKILL.md
+1 -1
View File
@@ -136,7 +136,7 @@ Conversion runs the firmware repo's `lib/EpdFont/scripts/fontconvert_sdcard.py`
- [Web server endpoints](./docs/webserver-endpoints.md)
- [Project scope](./SCOPE.md)
- [Contributing docs](./docs/contributing/README.md)
- [Touch and UI development](./docs/contributing/touch-and-ui.md) - FreeInkUI components for new screens, the touch bridge for existing ones, and build envs for the non-Xteink touch devices
- [Touch and UI development](./docs/contributing/touch-and-ui.md) - how to build new screens on the FreeInkUI activity bases (UiListActivity and friends), plus build envs for the non-Xteink touch devices
---
+94 -113
View File
@@ -1,23 +1,88 @@
# Touch and UI Development
CrossPoint now runs on touch devices (Seeed Sticky, M5Paper, LilyGo T5) alongside the button-only Xteink X3/X4. Every screen must work with both input styles. There are two supported ways to get there:
CrossPoint runs on touch devices (Seeed Sticky, M5Paper, LilyGo T5) alongside the button-only Xteink X3/X4. Every screen must work with both input styles.
1. **New screens: build them with FreeInkUI components.** Touch hit-testing, tap highlighting, long-press, and button focus navigation come with the component; you never hand-roll coordinate math.
2. **Existing screens and in-flight features: use the MappedInputManager touch bridge.** A small set of helpers adds tap/hold/swipe support to hand-rolled rendering without restructuring the activity.
**There is one supported way to build a new screen: FreeInkUI, hosted through the firmware base classes below.** Touch hit-testing, tap highlighting, long-press, swipe scrolling, and button focus navigation all come from the shared stack; you never hand-roll coordinate math.
If you are starting a new activity, use FreeInkUI. If you have a feature branch with a hand-rolled screen already working on buttons, use the bridge; do not rewrite mid-flight.
The old bridge helpers (`rowTouch`, `colTouch`, `wasTapInRect`, manual rect `contains()` checks) are legacy. They survive only for the two remaining hand-rolled surfaces (the theme-driven home screen and the reader page) and must not appear in new code. PRs that add new uses will be asked to convert.
---
## Path 1: New screens with FreeInkUI
## Picking the right host
FreeInkUI (`freeink-sdk/libs/ui/FreeInkUI`, namespace `freeink::ui`) is an immediate-mode component library. The core idea:
| Your screen is... | Use | In-tree reference |
|---|---|---|
| A single list of rows | subclass `UiListActivity` | [`LanguageSelectActivity`](../../src/activities/settings/LanguageSelectActivity.cpp) (minimal), [`RecentBooksActivity`](../../src/activities/home/RecentBooksActivity.cpp) (long-press) |
| Tabbed lists | subclass `UiTabListActivity` | [`SettingsActivity`](../../src/activities/settings/SettingsActivity.cpp) |
| Custom FUI layout (sliders, prompts, state machines) | inherit `UiAppHost` directly | [`EpubReaderPercentSelectionActivity`](../../src/activities/reader/EpubReaderPercentSelectionActivity.cpp), [`WifiSelectionActivity`](../../src/activities/network/WifiSelectionActivity.cpp) |
| A modal picker or confirm inside a legacy activity | `OptionPopup` (or push `ConfirmationActivity`) | [`OtaUpdateActivity`](../../src/activities/settings/OtaUpdateActivity.cpp) |
| A Yes/No prompt inside a FUI screen | build `fui::optionDialog` into the screen | `WifiSelectionActivity::buildPromptDialog` |
- While a component renders, it registers its tappable areas ("hit rects") into the frame's interaction buffer via `Frame::hit(rect, action, value, inputMask, state)`.
- Each loop you build an `InputSnapshot` from the input manager and route it against that buffer. If a tap (or mapped button press) lands in a registered rect, the routed `ActionEvent` tells you which action fired and with what value.
- Components query `frame.stateFor(action, value)` while painting, so touch-down highlight and focus states render correctly without any per-activity code.
All four paths route input through the same SDK interaction table, so touch and physical buttons fire the same actions with no per-screen coordinate code.
You get touch, hold highlighting, long-press, minimum touch-target sizing, and orientation-aware coordinates for free. The same action IDs fire from physical buttons, so one code path serves both input styles.
## The hosting stack
`UiAppHost` ([src/components/UiAppHost.h](../../src/components/UiAppHost.h)) owns what every FUI screen shares: the font-bound render target, the `FreeInkApp`, and the `uiReady` handshake that lets the loop task route touch snapshots against the interaction table the render task rebuilds. Never re-implement that handshake; it is a cross-task protocol and lives in exactly one place.
`UiListActivity` ([src/activities/UiListActivity.h](../../src/activities/UiListActivity.h)) layers the list protocol on top: swipes scroll the viewport without moving the selection, buttons move the selection and pull the viewport along (`fui::ListNav`), plus the render skeleton (header chrome, app, footer hints). `UiTabListActivity` is the tab-bar sibling; its selection ring puts the tab bar at position 0 and the rows after it.
### New list screen skeleton
Subclasses supply the data; the base owns the loop. The whole contract:
```cpp
class MyListActivity final : public UiListActivity {
public:
MyListActivity(GfxRenderer& r, MappedInputManager& in) : UiListActivity("MyList", r, in) {}
private:
std::vector<Entry> entries; // the activity's data, loaded in onEnter
std::vector<fui::ListItem> rows; // activity-owned row cache: rebuilt only when entries changes, reused every render
int listCount() const override { return entries.size(); }
const char* headerTitle() const override { return tr(STR_MY_TITLE); }
void onEnter() override {
UiListActivity::onEnter();
entries = /* ... load from wherever ... */;
rebuildRows();
}
// Called whenever entries changes (here, only onEnter; a mutable list would
// also call this after any add/remove). NOT called from buildScreen().
void rebuildRows() {
rows.clear();
rows.reserve(entries.size());
// ... one fui::ListItem per entry (label, actionValue = index) ...
}
void buildScreen(UiScreen& screen) override {
// set content margin from the theme safe area, then:
fui::ListProps props;
props.items = rows.data();
props.count = rows.size();
props.action = ACTION_ROW;
props.inputMask = fui::InputTouch; // physical buttons stay in the base loop
syncListViewport(screen, props); // selection/viewport handoff, always right before list()
screen.list(props);
}
void activateIndex(int index) override {
app.clearTapFlash(); // leaving the screen: a lingering flash would gray the next render
// ... open the thing ...
}
};
```
See [`FileBrowserActivity`](../../src/activities/home/FileBrowserActivity.cpp)'s `rebuildRowItems()` for this pattern applied to a directory listing that can run into the hundreds of entries, and `LanguageSelectActivity.cpp` for the rest of the skeleton (content-margin math, footer, etc.) — note that file still builds its row vector locally inside `buildScreen()`; match `FileBrowserActivity`, not that file, for the row cache. Optional overrides: `onRowLongPress(index)`, `drawFooter()`, `handleButtons()` for extra physical-button handling, and `ACTION_USER`-and-up action ids for non-row elements (register handlers in `onEnter` after the base's).
### Rules that apply to every FUI screen
- Register actions with `app.on(...)` in `onEnter` (after `resetUi()` for direct `UiAppHost` users). Handlers receive a `void* user` you cast back to the activity.
- Handlers that leave the current screen call `app.clearTapFlash()` first.
- Theme tokens are shared and bound by `resetUi()`; never call `app.setTheme` yourself. Metrics flow from the active UITheme through [`UIThemeTokens.h`](../../src/components/UIThemeTokens.h), including the per-board bezel insets that keep scrollbars visible.
- `TextStyle.maxLines` defaults to 1 and truncates with an ellipsis. Set `maxLines` explicitly on any dialog headline or message that can wrap.
- Everything stays allocation-free in steady state. A local `std::vector` inside `buildScreen()` is **not** allocation-free even with `reserve()` first: it starts at zero capacity on every call, `reserve()` allocates, and the destructor frees that storage before the call returns — real allocator work and fragmentation risk on every repaint (cursor move, tap flash, ...), not just on data changes. Build `ListItem` rows into activity-owned storage instead, reserved once when the underlying data loads (`onEnter()`/a `load*()` — see the skeleton above and `FileBrowserActivity::rebuildRowItems()`), and reused unchanged by every `buildScreen()` call. Use a fixed-capacity array (e.g. `ListItem rows[MAX]`, as `OptionPopup` and `KOReaderSyncActivity`'s action rows do) when the count is small and bounded. Do not hold FUI `props` across renders — only the row storage they point into.
### Component inventory
@@ -33,106 +98,11 @@ All under `freeink-sdk/libs/ui/FreeInkUI/include/components/`:
| Media | `book-card`, `cover-grid`, `cover-carousel`, `metric-card` |
| Text | `text-field`, `text-area` |
### Integration skeleton
The in-tree reference is [`src/activities/util/KeyboardEntryActivity.cpp`](../../src/activities/util/KeyboardEntryActivity.cpp); it drives the FreeInkUI keyboard component inside a normal Activity. The shape:
```cpp
#include <FreeInkUIGfxRenderer.h>
namespace fui = freeink::ui;
// Render step: draw the component and (re)register its hit rects.
fui::GfxRendererTarget target(renderer); // adapts GfxRenderer to FreeInkUI's DrawTarget
target.setFont(fui::GfxRendererTarget::FONT_BODY, UI_12_FONT_ID);
const fui::DeviceContext device = target.deviceContext(); // orientation, safe area, touch sizing
fui::Frame<48> frame(target, device, fui::InputSnapshot{}, interactions); // 48 = max hit rects
fui::KeyboardProps props;
props.layout = &currentLayout();
props.keyAction = ACTION_KEY; // one action id; the key is the event value
props.inputMask = fui::InputTouch | fui::InputLongPress;
fui::keyboard(frame, kbRect, props); // draws AND registers hit rects
```
```cpp
// Input step (each loop): feed touch state, act on routed events.
int tx, ty, tapX, tapY, hx, hy;
const bool pressedDown = mappedInput.wasScreenTouchDown(tx, ty);
const bool tapped = mappedInput.wasScreenTapped(tapX, tapY);
const bool inContact = mappedInput.isScreenTouchHeld(hx, hy);
const auto result = touchRouter.update(interactions, pressedDown, tx, ty,
tapped, tapX, tapY, inContact, millis());
if (result.event) {
activateValue(result.event.value, result.event.longPress);
requestUpdate();
} else if (result.activeChanged) {
requestUpdate(); // repaint touch-down highlight
}
```
Notes:
- `Frame<N>` is templated on the max number of hit rects; the buffer is stack/static, no heap.
- `fui::TouchHoldRouter` (see `components/keyboard/keyboard.h`) synthesizes long-press while the finger is still down and swallows the eventual release. Use it whenever a component distinguishes tap from long-press.
- For simpler screens without long-press, `FreeInkApp` / `Screen<N>` (`FreeInkApp.h`) is the ergonomic layer: it owns the interaction buffer, offers a top-to-bottom builder (`takeTop`, footer helpers, etc.), and its `render(input)` call routes the snapshot and dispatches registered action callbacks in one step. `snapshotFrom(input, device)` in `FreeInkUIInputManager.h` builds the `InputSnapshot` with orientation-aware touch mapping.
- Everything stays allocation-free and works on the button-only devices unchanged: physical buttons route through the same interaction table via each hit rect's `inputMask`.
One exception to "always go through a host": `KeyboardEntryActivity` drives the keyboard component with a raw `fui::Frame` and `TouchHoldRouter` because per-key hold repeat needs its own routing. If you think your screen needs that, raise it in the PR first.
---
## Path 2: Adding touch to existing hand-rolled screens
For activities that draw their own rows, menus, and buttons, `MappedInputManager` ([src/MappedInputManager.h](../../src/MappedInputManager.h)) exposes bridge helpers. They all return **logical screen coordinates** (orientation already applied via `GfxRenderer::tapToLogical`); never touch raw normalized panel coordinates or the SDK `InputManager` directly.
| Helper | Use for |
|---|---|
| `wasScreenTapped(x, y)` | A completed tap (press + release), with logical coords |
| `wasScreenTouchDown(x, y)` | Touch-down (held > 90 ms, not yet released): draw selection highlight |
| `isScreenTouchHeld(x, y)` | Live contact position while the finger is down (drag tracking) |
| `wasTapInRect(x, y, w, h)` | One-off hit test on a rectangle (a single button, a banner) |
| `wasListItemTapped(index, count, selected, listTop, listHeight, hasSubtitle)` | Taps on a standard UITheme list; does the row/paging math for you |
| `wasListItemTouchedDown(...)` | Same geometry, touch-down phase (highlight before activate) |
| `rowTouch(row, top, rowStep, rowCount, xStart, xEnd, rowHeight)` | Any custom band of equal-height rows; returns `RowTouch::None/Down/Tap` |
| `colTouch(col, left, colStep, colCount, yStart, yEnd, colWidth)` | Horizontal button bands (dialogs, prompts) |
| `wasSwipe()` | Raw swipe direction if you need one beyond the global gestures |
| `hasTouch()` | True when the device has a touch panel (rarely needed; helpers simply never fire without one) |
### Pattern A: standard themed list
One call per loop; UITheme owns the row geometry ([EpubReaderBookmarksActivity.cpp:123](../../src/activities/reader/EpubReaderBookmarksActivity.cpp)):
```cpp
int tapped = -1;
if (mappedInput.wasListItemTapped(tapped, bookmarks.size(), selectorIndex, listY, listHeight, true)) {
selectorIndex = tapped;
openBookmark();
return;
}
```
### Pattern B: custom rows with hold highlight
For non-theme row layouts, use `rowTouch` and distinguish `Down` (highlight) from `Tap` (activate), as in [EpubReaderFootnotesActivity.cpp](../../src/activities/reader/EpubReaderFootnotesActivity.cpp):
```cpp
int row = -1;
const auto touch = mappedInput.rowTouch(row, listTop, lineHeight, visibleCount,
contentX, contentX + contentWidth);
if (touch != MappedInputManager::RowTouch::None) {
const int touched = scrollOffset + row;
if (touch == MappedInputManager::RowTouch::Down) {
if (selectedIndex != touched) { selectedIndex = touched; requestUpdate(); }
} else { // RowTouch::Tap
selectedIndex = touched;
activateSelection();
}
return;
}
```
The `Down` state exists because e-ink repaints are slow: highlight on touch-down gives immediate feedback, activation happens on release.
### Global gestures: do not reimplement these
## Global gestures: do not reimplement these
Three gestures are handled once, for every screen. Activities must not add their own edge-swipe handling:
@@ -142,14 +112,25 @@ Three gestures are handled once, for every screen. Activities must not add their
| Home | Up-swipe starting in the bottom 14% | `ActivityManager::loop()`; pops to Home (activities can override via `handleHomeGesture()`) |
| Menu | Down-swipe starting in the top 14% | Activities that have a menu check `wasMenuGesture()` themselves (the reader does this) |
Because the back gesture arrives as `Button::Back`, most button-era activities gain back-swipe support with zero changes. That is the bar to aim for: bridge helpers should make touch an additive layer over the button flow, not a second input state machine.
Because the back gesture arrives as `Button::Back`, most button-era activities gain back-swipe support with zero changes.
### Bridge rules
---
- Handle touch in `loop()` next to the existing button handling, one helper call per interaction zone, and `return` after consuming an event (mirrors the button pattern).
- Never call the SDK `InputManager` or read GPIO directly; the HAL rule from the main guide applies to touch too.
- Coordinates from the helpers are logical and orientation-correct on all four rotations; test at least Portrait and one Landscape mode before PR.
- Nothing to clean up in `onExit()`; the helpers are stateless from the activity's point of view.
## Legacy bridge helpers (do not use in new code)
`MappedInputManager` still exposes raw touch accessors. The FUI stack consumes them internally (via `touchSnapshotFrom` in [`UiAppHelpers.h`](../../src/components/UiAppHelpers.h)); activities should not.
| Helper | Status |
|---|---|
| `wasScreenTapped` / `wasScreenTouchDown` / `isScreenTouchHeld` | Consumed by the FUI snapshot builder. Direct use only in the two legacy surfaces |
| `wasTapInRect(x, y, w, h)` | Legacy one-off hit test |
| `rowTouch` / `colTouch` | Legacy row/column band math. Sole remaining user: `HomeActivity` (theme-driven layout) |
| `wasSwipe()` | Raw swipe direction, for behavior beyond the global gestures (reader page turns) |
| `hasTouch()` | Still fine anywhere: gate touch-only chrome (on-screen Cancel/OK pairs) on it |
The `wasListItemTapped` / `wasListItemTouchedDown` helpers from the button-era bridge have been removed; every themed list is a `UiListActivity` now. If you are porting a branch that still calls them, convert the screen to `UiListActivity` rather than resurrecting the helpers.
As with all input: never call the SDK `InputManager` or read GPIO directly. The HAL rule from the main guide applies to touch too.
---
+6 -4
View File
@@ -10,9 +10,10 @@ A dictionary folder must contain:
- `.idx` — word index (required, **must be uncompressed** — a `.idx.gz` will not work; decompress it on your computer with `gzip -d` first)
- `.dict` or `.dict.dz` — definition data (`.dict.dz` is supported as-is; entries are decompressed on the fly during lookup)
- `.syn` — synonym index (optional; maps alternate spellings and irregular forms to their headword)
- `.ifo` — metadata (optional)
Not supported: `.syn` synonym files (ignored), dictionaries with 64-bit index offsets (`idxoffsetbits=64` in the `.ifo` — rare, and rejected with an error), and HTML-formatted definitions render as raw markup rather than styled text.
Not supported: dictionaries with 64-bit index offsets (`idxoffsetbits=64` in the `.ifo` — rare, and rejected with an error), and HTML-formatted definitions render as raw markup rather than styled text.
## Setting Up a Dictionary
@@ -35,13 +36,14 @@ One word on the page becomes highlighted:
2. Press **Confirm** to look up the highlighted word.
3. Press **Back** to return to the reader.
On the very first lookup with a dictionary (and again if the dictionary file changes), the reader shows *"Indexing dictionary…"* while it builds a small `.qidx` sidecar file next to the `.idx`. This takes a few seconds for large dictionaries and makes all subsequent lookups fast. The sidecar can be deleted safely at any time — it will simply be rebuilt.
On the very first lookup with a dictionary (and again whenever the `.idx` or `.syn` source file changes), the reader shows *"Indexing dictionary…"* while it builds small sidecar files next to them — a `.qidx` for the word index, and a `.sidx` when a `.syn` synonym file is present. Each sidecar is rebuilt independently, only when its own source changes. This takes a few seconds for large dictionaries and makes all subsequent lookups fast. The sidecars can be deleted safely at any time — they will simply be rebuilt.
### How Lookup Works
1. **Direct match** — the word is found as-is (case-insensitive) in the dictionary index. Surrounding punctuation is ignored.
2. **Stemming** — on a miss, common English word forms are retried automatically: possessives and plurals (`dogs` → `dog`, `stories` → `story`) and verb endings (`walked` → `walk`, `running` → `run`, `making` → `make`).
3. **Not found** — a short popup appears and you return to word selection.
2. **Synonyms** — on a miss, if the dictionary ships a `.syn` file, alternate spellings and irregular forms recorded there are resolved to their headword (e.g. `oxen` → `ox`, `colour` → `color`). This step is skipped if the `.sidx` sidecar could not be built (e.g. transient low memory during indexing); the dictionary otherwise stays usable, and the build is retried the next time it is opened.
3. **Stemming** — still no match: common English word forms are retried automatically: possessives and plurals (`dogs` → `dog`, `stories` → `story`) and verb endings (`walked` → `walk`, `running` → `run`, `making` → `make`).
4. **Not found** — a short popup appears and you return to word selection.
## The Definition Screen
+10 -3
View File
@@ -90,12 +90,19 @@ if (parsedSize != fileSize) {
## `section.bin`
### Version 35
### Version 37
Each file in `sections/*.bin` stores one laid-out spine section. The header is
also the cache-busting key: if any layout-affecting setting differs from the
current reader settings, the section is discarded and rebuilt.
Version 37 increases the fixed-size footnote href field from 96 to 256 bytes.
This changes each serialized footnote record from 128 to 288 bytes, so older
section caches must be discarded and rebuilt.
Version 36 invalidates cached word positions after ruby and CJK justification
layout changes.
Version 35 adds a header offset and a `uint32_t` entry per page for the
visible-text offset LUT. The other section LUTs remain unchanged.
@@ -135,10 +142,10 @@ import std.mem;
import std.string;
import std.core;
#define EXPECTED_VERSION 35
#define EXPECTED_VERSION 37
#define MAX_STRING_LENGTH 65535
#define FOOTNOTE_NUMBER_LEN 32
#define FOOTNOTE_HREF_LEN 96
#define FOOTNOTE_HREF_LEN 256
struct String {
u32 length [[hidden, comment("String byte length")]];
+351
View File
@@ -0,0 +1,351 @@
# SD-card plugins
CrossPoint keeps service integrations off the firmware. A plugin is a folder on
the SD card; the firmware provides three generic, scheme-neutral surfaces that
plugins compose. Adding or updating a plugin never requires a firmware build,
and the firmware carries no vendor names, URLs, or file-format knowledge.
```
<root>/<name>/
manifest.json web UI card metadata (optional): title, description, mount
plugin.js browser-side plugin (optional)
device.json on-device catalog screen (optional): title, description, ...
README.md usage instructions, shown on-device (optional)
...assets
```
**Discovery and the on-device list.** Every plugin folder (anything holding a
`manifest.json`, `plugin.js`, or `device.json`) appears under **Settings →
System → Plugins** on the reader, showing its `title` and one-line
`description`. Selecting a plugin opens an info screen with the description and
the plugin's `README.md` as scrollable usage instructions — so even a
browser-only plugin (no `device.json`) is listed and can explain how to use it
from the web UI. A plugin that ships a `device.json` also gets an **Open**
action there to launch its on-device catalog. `title`/`description` are read
from `manifest.json`, with `device.json` overriding when present.
`<root>` is any of `/.crosspoint/plugins`, `/plugins`, or `/.plugins` — the
first two-dot-free options exist so plugins are easy to copy onto the card
from a computer. All three roots are scanned; on a name collision the earlier
root in that order wins.
A plugin can ship any combination: `plugin.js` alone (web-only),
`device.json` alone (on-device only), or both (e.g. sign in from either side,
browse on the reader).
The plugin sources and examples live in the separate `sd-plugins` repository,
which also documents the browser-side `plugin.js` API. This document covers the
firmware surfaces: the job queue and the `device.json` on-device screens.
## Surface 1: browser plugins (`plugin.js`)
JS loaded into the File Manager or Settings web page, backed by generic device
endpoints (`/api/relay`, `/api/crypto`, `/api/fetch`, `/api/plugin-fs`). See
`docs/webserver-endpoints.md` for the endpoints and the sd-plugins repository
for the JS contract.
## Surface 2: the plugin job queue (external automation)
External systems (a companion app, a script) can trigger plugin actions
without a human clicking the web UI. The firmware stores small opaque
`{plugin, action, args}` JSON blobs in a fixed 6-slot pool; it never interprets
them. Any open page hosting the plugin — including the headless runner at
`GET /plugins-run` — claims jobs, executes the plugin's registered handler in
the browser context, and posts the result back.
| Endpoint | Purpose |
|---|---|
| `POST /api/plugin-jobs` | enqueue `{plugin, action, args?}` → `{id}` (503 when the pool is full) |
| `GET /api/plugin-jobs/claim?plugin=` | executor claims the next pending job |
| `POST /api/plugin-jobs/complete` | executor posts `{id, ok, result?}` |
| `GET /api/plugin-jobs/status?id=` | caller polls: `{id, state, result}`; states: `pending`, `running`, `done`, `error`, `unknown` (recycled) |
| `GET /plugins-run` | headless page that loads every plugin (UI hidden) and executes jobs while open |
Flow for an external system:
```sh
# 1. upload whatever the action needs (e.g. a file, via /upload)
# 2. enqueue
curl -X POST http://crosspoint.local/api/plugin-jobs \
-d '{"plugin":"<name>","action":"<action>","args":{"path":"/folder/file"}}'
# -> {"id":7}
# 3. open http://crosspoint.local/plugins-run in a browser tab or hidden
# webview (jobs only execute while a hosting page is open)
# 4. poll
curl "http://crosspoint.local/api/plugin-jobs/status?id=7"
# -> {"id":7,"state":"done","result":{...}}
```
Limits: plugin/action names < 24 chars, args and result < 192 bytes of JSON
each. A job claimed by a runner that dies is reclaimable after 10 minutes.
Results persist only until their slot is recycled — poll promptly.
Plugins register handlers in `plugin.js`:
```js
api.registerAction('myaction', async (args) => {
// runs in the hosting page; throw -> state "error" with {error: message}
return { anything: 'small' }; // -> state "done" with this result
});
```
## Surface 3: on-device catalog screens (`device.json`)
A declarative manifest the firmware's generic `PluginCatalogActivity` renders
under **Settings → System → Plugins**. It expresses "authenticated JSON
catalog: sign in, browse, download, sidecar" — enough for most book services —
without any code running on the device. Anything beyond this vocabulary
belongs in `plugin.js`.
The firmware pieces (all service-agnostic, in `src/activities/plugins/`):
`PluginListActivity` (the Plugins menu), `PluginInfoActivity` (per-plugin
description + README + Open action), `PluginCatalogActivity` (browse / download
/ sign-in), and `discoverPlugins()` (folder scan when the menu opens; nothing
stays resident).
### Schema
Every string field is a template. Available substitutions:
| Variable | Meaning | Available in |
|---|---|---|
| `{token}` | contents of the token file at `token.path` | everywhere |
| `{cfg.KEY}` | value of `KEY` in the flat JSON config file at `config.file` | everywhere |
| `{page}` | current 1-based page | browse |
| `{limit}` | `page_size + 1` (the extra row detects "more pages") | browse |
| `{id}` `{title}` `{author}` `{url}` | fields of the selected item | download, sidecar |
| `{md5}` | MD5 hex of the destination file path | sidecar |
| `{device_code}` | from the auth request response | auth poll |
Two browse formats:
- **`"json"`** (default) — a paged JSON list, using `items`/`fields` below.
- **`"xml"`** — walks a repeating XML element with optional folder navigation
(Confirm opens a folder, Back goes up). No paging; the JSON `items`/
`page_size` are ignored. The firmware knows no specific protocol: a WebDAV
`PROPFIND` multistatus, an OPDS/Atom feed, or any XML list is all manifest
config. XML field selectors are `"elem"` (child element text), `"elem@attr"`
(child element attribute), or `"@attr"` (attribute on the item element).
```jsonc
{
"title": "Service Name", // menu label; defaults to folder name
"token": { // omit for token-less catalogs
"file": "/.crosspoint/<name>.json", // written by auth (either side)
"path": "token" // dotted JSON path inside the file
},
"config": { // optional: flat JSON of {cfg.KEY} values,
"file": "/.crosspoint/<name>-cfg.json" // e.g. a user-entered server URL + credentials
},
"browse": { // required
"format": "json", // or "xml"
"url": "https://.../search",
"method": "POST", // default GET
"headers": { "Authorization": "Bearer {token}" },
"body": "{\"page\":{page},\"per_page\":{limit}}",
// --- json format ---
"items": "", // dotted path to the item array; "" = response root
"fields": { // json: dotted paths (numeric = array index)
"title": "title", // required (items without one are dropped)
"author": "authors.0.name",
"id": "id",
"url": "download_url" // when the item carries a direct file URL
},
"page_size": 8, // 1..16; response should honor {limit}
"lists": [ // optional named sub-catalogs (json only):
{ "title": "All Books" }, // a picker screen precedes browsing;
{ "title": "Favorites", // each entry may override url and/or
"body": "{\"page\":{page},\"per_page\":{limit},\"list\":\"favorites\"}" }
], // omitted keys fall back to browse's
// --- xml format (ignore the json fields above; fields become selectors) ---
"item": "response", // local-name of the repeating element
"container_element": "collection", // presence marks a navigable folder (omit = flat)
"skip_self": true, // drop the entry equal to the request URL
"resolve_urls": true, // resolve url field against the request origin
"extensions": [".epub", ".pdf"] // allowed file extensions (omit = all)
},
"download": {
"url": "https://.../books/{id}/download",
"method": "POST",
"headers": { "Authorization": "Bearer {token}" }, // sent on the file GET too
"body": "{}",
"url_path": "url", // response field with the file URL;
// omit to treat "url" itself as the file URL
"username": "{cfg.user}", // optional HTTP Basic creds for the file GET
"password": "{cfg.pass}", // omit for token/header auth
"dest_dir": "/ServiceName", // created if missing; falls back to SD root
"filename": "{title}.epub", // {title} is filesystem-sanitized here
"sidecar": { // optional per-book metadata file
"path": "/.crosspoint/<name>_{md5}.json",
"body": "{\"book_id\":{id}}"
}
},
"auth": { // optional on-device sign-in
"type": "device_code", // or "password"
// device_code (interactive: shows a code + QR, polls until authorized):
"request": { "url": "...", "method": "POST", "headers": { ... }, "body": "..." },
"poll": { "url": "...", "method": "POST", "headers": { ... },
"body": "{...\"device_code\":\"{device_code}\"}" },
// device_code response field paths, with their defaults:
"code_path": "user_code", "verify_url_path": "verification_uri",
"device_code_path": "device_code", "interval_path": "interval",
"expires_path": "expires_in", "token_path": "access_token",
"error_path": "error"
// password (silent: mints a token from stored config credentials — for
// OAuth2 password grants and login-returns-token APIs). Only "request" and
// "token_path" apply; no user interaction:
// "type": "password",
// "request": { "url": "{cfg.url}/oauth/v2/token", "method": "POST",
// "headers": {...}, "body": "...{cfg.username}...{cfg.password}..." },
// "token_path": "access_token"
}
}
```
### Behavior
- **Sign-in (`device_code`)**: the not-signed-in screen offers Sign in on the
device: it shows the verification URL (text + QR) and user code, then polls
until authorized, honoring `authorization_pending`, `slow_down`,
`expired_token`, and `access_denied`. The token is written to `token.file`
in the shape `token.path` expects, so the web plugin and the device share
one sign-in.
- **Sign-in (`password`)**: no on-device UI — the device silently POSTs the
stored config credentials to the token endpoint before browsing, and re-mints
a fresh token automatically after any 401 (these tokens are short-lived).
Credentials come from a browser `plugin.js` writing the `config.file`. Set it
up once on the web page; the reader is then standalone.
- Without an `auth` block the not-signed-in screen directs the user to the web
plugin.
- **Stale tokens**: a 401/403 from browse returns to the sign-in screen rather
than an error.
- **Pagination** (json): the browse request should return up to `{limit}`
items; the firmware displays `page_size` and uses the extra row to know
another page exists. Navigation matches the OPDS browser: a "Previous
page" row precedes the items past page 1 and a "Next page" row follows
them while more pages exist — tappable and button-reachable like any row.
- **Lists** (json): with `browse.lists`, opening the catalog shows a picker
of the named entries first; each entry browses with its own url/body
overrides (server-side categories, shelves, sort orders). Back from the
book list returns to the picker.
- **XML navigation**: Confirm on a folder (an item carrying `container_element`)
descends into it, Back climbs out (Back at the root leaves the screen).
With `resolve_urls`, server-relative URLs resolve against the browse URL's
origin. A `config.file` lets values like a server URL and credentials be
user-entered (via a browser `plugin.js`) rather than baked into the manifest.
- **After download**: the book's layout cache is invalidated and the optional
sidecar is written (e.g. a service book id keyed by path MD5, for a future
progress-sync stage).
### Heap budget (ESP32-C3, ~380KB total)
- Manifest ≤ 8KB, token file ≤ 2KB, small API responses (auth, download-url
hop) ≤ 48KB (hard caps).
- Browse responses (json and xml) stream to an SD temp file and are parsed
from it, so the raw body never occupies DRAM — services that inline heavy
per-item metadata (a page of BookFusion JSON runs 60+ KB) work unmodified.
A 1MB cap bounds a misbehaving server.
- Catalog responses are parsed with a dynamically built ArduinoJson filter
containing only the declared field paths, so a page parses into a few KB
no matter the body size.
- One TLS session is kept alive across browse requests: repeated handshakes
permanently fragment the heap.
- The book download streams to SD via `HttpDownloader`; file size is
unconstrained.
- An XML list is parsed from the temp file in 2KB chunks; the engine stops
after 200 entries per folder/feed. Split very large libraries into
subfolders.
### Testing a new manifest
1. Copy the plugin folder to `/.crosspoint/plugins/<name>/` on the SD card.
2. Settings → System → Plugins → your title. With no token and no `auth`
block you should see the not-signed-in screen; with `auth`, the code/QR
screen.
3. Watch serial (`[PCAT]` tag) for request/parse failures — the log includes
HTTP status and truncation flags.
4. Token-less services (public JSON APIs) work by omitting the `token` block
entirely; that is the quickest way to validate `items`/`fields` paths.
## Ideas to build
The two planes decide *how* you build something, so pick by what the idea needs:
- **On-device (`device.json`)** — standalone on the reader, but only the
declarative vocabulary above: an authenticated JSON or XML/OPDS-style catalog,
browse, download to SD, optional device-code or password sign-in and per-book
sidecar. No arbitrary logic.
- **Browser (`plugin.js`)** — anything at all (any protocol, auth, format), but
it runs in a connected browser/app. Use it when the idea needs request
signing, HTML→EPUB conversion, multi-file writes, or a real UI. The job queue
(`/api/plugin-jobs` + `/plugins-run`) lets an external app trigger it.
Rule of thumb: if the service is "log in, list an authenticated JSON/XML
catalog, download files," it's on-device. If it needs computed request bodies
(OAuth 1.0a signing, canonicalized signatures) or format conversion, it's
browser-side.
### On-device (`device.json`) ideas
- **Self-hosted library presets** — Kavita, Komga, Audiobookshelf, Calibre-Web,
COPS. Most expose OPDS (native OPDS browsing already covers those — a plugin
adds value mainly for their JSON APIs or token auth). Build: `browse.format`
`json` against the REST API, or `xml` against the OPDS feed; `download.url`
from the entry; bearer/Basic/`{cfg.*}` auth. See `bookfusion` (JSON + token)
and `webdav` (XML + config creds).
- **Dictionary store** — browse a JSON index of dictionary files, download to
the SD dictionary dir. No native equivalent. Build: token-less `json` catalog,
`download.dest_dir` = the dictionary folder, no sidecar. ~15 lines.
- **Theme store / font store** — same shape as the dictionary store, pointed at
the theme or font directory. Asset catalogs OPDS can't express.
- **More read-later that exports EPUB** — anything with a token API and an
EPUB export URL fits like `wallabag`: `auth.type` `password` (or a stored
bearer), `browse` the entries JSON, `download` the export endpoint with a
Bearer header.
- **A curated free-books catalog** — Standard Ebooks / Gutenberg via their OPDS
feed (`xml` format, token-less). Overlaps native OPDS, but ships as a
zero-config preset.
### Browser (`plugin.js`) ideas
- **Plugin store** — install other plugins from a hosted catalog. Build: fetch
a catalog with `api.relay`, `/mkdir` the target folder, `api.writeFile` each
file. See `plugin-store/`.
- **AO3 / fanfiction downloader** — the site serves per-work EPUB downloads.
Build: take a work URL, `api.fetchToSd` (or `api.relay` + `api.writeFile`) the
EPUB to SD. Browsing/search is HTML, so parse it in the browser.
- **RSS → EPUB digest** — build a "morning edition." Fetch feeds with
`api.relay`, assemble an EPUB in the browser with the JSZip the File Manager
page already loads, then `api.writeFile` it to SD.
- **Read-any-article** — paste a URL, run it through a readability service via
`api.relay`, wrap the result as an EPUB, write to SD.
- **Metadata / cover editor** — a File Manager plugin: open an EPUB with JSZip,
edit the OPF (title/author/cover), rewrite it in place. See
`organize-by-author/` for the File-Manager-endpoints pattern.
- **Library tidy tools** — dedupe, bulk rename, move-by-metadata. Same
same-origin endpoints (`/api/files`, `/mkdir`, `/move`, `/delete`), no device
API needed.
- **Signed-API services** (Instapaper OAuth 1.0a, etc.) — must be browser-side:
do the per-request signing in JS with `crypto.subtle`, relay the calls, and
convert the article HTML to EPUB before writing to SD.
### Building blocks reference
| Need | Use |
|---|---|
| Outbound HTTP(S), any method (CORS-free) | `api.relay(method, url, headers, body)` |
| Download a URL straight to SD | `api.fetchToSd(url, dest, headers)` |
| Write a small file to SD | `api.writeFile(path, base64)` |
| Crypto (hash, HMAC via SHA, AES, RSA) | `api.crypto(op, fields)` or browser `crypto.subtle` |
| Create / delete / move SD files | same-origin `/mkdir`, `/delete`, `/move` |
| On-device catalog/browse/download | `device.json` (this document) |
| External app triggers an action | `api.registerAction(name, fn)` + `/api/plugin-jobs` |
+40
View File
@@ -504,3 +504,43 @@ The final field is the WebSocket upload port.
Calibre Wireless starts the same web server in STA mode and displays setup
instructions plus WebSocket upload progress on the device screen.
## Plugin Job Queue
External systems trigger SD-plugin actions without the web UI. The firmware
stores opaque `{plugin, action, args}` blobs (fixed 6-slot pool; args/result
< 192 bytes JSON); an open page hosting the plugin executes them. See
`docs/sd-plugins.md` for the full contract.
### `POST /api/plugin-jobs`
```bash
curl -X POST http://crosspoint.local/api/plugin-jobs \
-d '{"plugin":"<name>","action":"<action>","args":{"path":"/Books/somefile"}}'
# -> {"id":3} (503 {"error":"job queue full"} when all slots busy)
```
### `GET /api/plugin-jobs/claim?plugin=<name>`
Executor-side (used by the plugin host page): returns the next pending job
`{"id":3,"action":"<action>","args":{...}}` and marks it running, or `{"id":0}`.
### `POST /api/plugin-jobs/complete`
Executor-side: `{"id":3,"ok":true,"result":{...}}` -> `{"ok":true}`.
### `GET /api/plugin-jobs/status?id=<n>`
```bash
curl "http://crosspoint.local/api/plugin-jobs/status?id=3"
# -> {"id":3,"state":"done","result":{"title":"...","dest":"/Books/Book.epub"}}
```
States: `pending`, `running`, `done`, `error`, `unknown` (slot recycled —
poll promptly after completion).
### `GET /plugins-run`
Headless executor page: loads every plugin with its UI hidden so registered
actions run. Keep it open (browser tab or embedded webview) while jobs are
queued; jobs enqueued with no page open stay pending until one opens.
+146
View File
@@ -0,0 +1,146 @@
// SD/HAL binding for the content-protection read path.
//
// The ContentProtection SDK lib is storage-agnostic (it works against a
// ByteSource). This file is the firmware-side glue that backs that seam with
// the device's SD storage: a HalStorage-backed ByteSource, the credential
// lookup, and the openProtectedBook() entry point the reader calls. It lives in
// the firmware — not the SDK lib — so the portable lib carries no HAL dependency.
#include <ByteSource.h>
#include <ContentProtection.h>
#include <Credential.h>
#include <HalStorage.h>
#include <Memory.h>
#include <ProtectedBook.h>
#include <WolfsslCrypto.h>
#include <Zip.h>
#include <esp_heap_caps.h>
#include <time.h>
namespace freeink {
namespace content {
namespace {
// The access credential is provisioned off-device and dropped here.
// Generic path — the reader carries no scheme name.
constexpr const char* kCredentialPath = "/.crosspoint/content.key";
// One shared crypto backend for the whole read path.
WolfsslCrypto& crypto() {
static WolfsslCrypto instance;
return instance;
}
// ByteSource over an SD file (read-only). One open handle per instance.
class SdByteSource : public ByteSource {
public:
explicit SdByteSource(std::string path) : path_(std::move(path)) {}
bool open() {
file_ = Storage.open(path_.c_str(), O_RDONLY);
return file_ && file_.isOpen();
}
int32_t readAt(uint64_t offset, void* dst, uint32_t len) override {
if (!file_ || !file_.seek64(offset)) return -1;
return file_.read(dst, len);
}
uint64_t size() const override { return file_ ? file_.fileSize64() : 0; }
private:
std::string path_;
mutable HalFile file_;
};
// Adapts an opened ProtectedBook to the reader-facing access interface.
class ProtectedBookDecryptor : public ContentDecryptor {
public:
ProtectedBookDecryptor(std::string epubPath, std::unique_ptr<ProtectedBook> book)
: epubPath_(std::move(epubPath)), book_(std::move(book)) {}
bool isEncrypted(const std::string& itemPath) const override { return book_->isEncrypted(itemPath); }
size_t decryptedSize(const std::string& itemPath) const override { return book_->decryptedSize(itemPath); }
bool decryptToSink(const std::string& itemPath, ContentChunkSink sink, void* context) override {
SdByteSource source(epubPath_);
if (!source.open()) return false;
return book_->decryptEntryToSink(source, crypto(), itemPath, sink, context);
}
private:
std::string epubPath_;
std::unique_ptr<ProtectedBook> book_;
};
} // namespace
std::unique_ptr<ContentDecryptor> openProtectedBook(const std::string& epubPath, std::string& err) {
err.clear();
if (!Storage.exists(epubPath.c_str())) return nullptr;
SdByteSource source(epubPath);
if (!source.open()) return nullptr;
// Classify before initializing crypto or loading credentials, then transfer
// this same ZIP index into ProtectedBook. Protected EPUBs still scan only
// once; plain EPUBs return immediately through the normal reader path.
ZipScan scan;
if (!scan.open(source) || !scan.find("META-INF/encryption.xml")) return nullptr;
// A book carrying encryption.xml may only obfuscate its embedded fonts
// (not content-protected). The SDK demands the credential only after parsing
// the manifest and finding genuinely encrypted entries.
SdByteSource credSource(kCredentialPath);
Credential credential;
const bool haveCredential = credSource.open() && parseCredential(credSource, &credential);
auto book = makeUniqueNoThrow<ProtectedBook>();
if (!book) {
err = "out of memory";
return nullptr;
}
// Prefer an out-of-band rights document delivered as a sidecar next to the
// EPUB ("<book>.epub.rights"), so the EPUB on disk stays byte-identical to
// what the server sent. Falls back to a rights.xml injected into the zip.
std::string rightsOverride;
{
// A real rights document is a few KB; 64KB is a generous ceiling. The
// largest-block check keeps the resize below from aborting on OOM (string
// growth is a bare allocation under -fno-exceptions).
constexpr uint64_t kMaxRightsSize = 64 * 1024;
SdByteSource rightsSource(epubPath + ".rights");
if (rightsSource.open()) {
const uint64_t rsize = rightsSource.size();
if (rsize > 0 && rsize <= kMaxRightsSize &&
heap_caps_get_largest_free_block(MALLOC_CAP_8BIT) > static_cast<size_t>(rsize) + 8 * 1024) {
rightsOverride.resize(static_cast<size_t>(rsize));
const int32_t rn = rightsSource.readAt(0, rightsOverride.data(), static_cast<uint32_t>(rsize));
if (rn <= 0)
rightsOverride.clear();
else
rightsOverride.resize(static_cast<size_t>(rn));
}
}
}
if (!book->openFromScan(source, crypto(), credential, std::move(scan), rightsOverride)) {
err = haveCredential ? ("cannot open protected content: " + book->lastError())
: "no content access key on this device";
return nullptr;
}
// An encryption manifest containing only font obfuscation does not require
// this read path; let the reader open it normally.
if (!book->isProtected()) return nullptr;
const int64_t now = static_cast<int64_t>(time(nullptr));
if (book->isExpired(now)) {
err = "access expired";
return nullptr;
}
auto decryptor = makeUniqueNoThrow<ProtectedBookDecryptor>(epubPath, std::move(book));
if (!decryptor) err = "out of memory";
return decryptor;
}
} // namespace content
} // namespace freeink
+78 -29
View File
@@ -8,6 +8,10 @@
#include <Utf8.h>
#include <ZipFile.h>
#include <cstdlib>
#include <cstring>
#include <vector>
#include "Epub/parsers/ContainerParser.h"
#include "Epub/parsers/ContentOpfParser.h"
#include "Epub/parsers/TocNavParser.h"
@@ -357,6 +361,22 @@ void Epub::parseCssFiles() const {
bool Epub::load(const bool buildIfMissing, const bool skipLoadingCss) {
LOG_DBG("EBP", "Loading ePub: %s", filepath.c_str());
// Open the optional encrypted-entry accessor. A null result without an error
// means normal ZIP reads should be used. A hard error refuses the open with
// a user-presentable reason.
{
std::string err;
decryptor = freeink::content::openProtectedBook(filepath, err);
if (!err.empty()) {
LOG_ERR("EBP", "protected content unavailable: %s", err.c_str());
protectionError = err;
return false;
}
if (decryptor) {
LOG_DBG("EBP", "protected content; on-read access path open");
}
}
// Initialize spine/TOC cache
bookMetadataCache.reset(new BookMetadataCache(cachePath));
// Always create CssParser - needed for inline style parsing even without CSS files
@@ -580,15 +600,9 @@ bool Epub::generateCoverBmp(bool cropped) const {
if (FsHelpers::hasJpgExtension(coverImageHref)) {
LOG_DBG("EBP", "Generating BMP from JPG cover image (%s mode)", cropped ? "cropped" : "fit");
const auto coverJpgTempPath = getCachePath() + "/.cover.jpg";
if (!extractItemToFile(coverImageHref, coverJpgTempPath)) return false;
HalFile coverJpg;
if (!Storage.openFileForWrite("EBP", coverJpgTempPath, coverJpg)) {
return false;
}
readItemContentsToStream(coverImageHref, coverJpg, 1024);
// Explicitly close() file before reopening for reading
coverJpg.close();
if (!Storage.openFileForRead("EBP", coverJpgTempPath, coverJpg)) {
return false;
}
@@ -614,15 +628,9 @@ bool Epub::generateCoverBmp(bool cropped) const {
if (FsHelpers::hasPngExtension(coverImageHref)) {
LOG_DBG("EBP", "Generating BMP from PNG cover image (%s mode)", cropped ? "cropped" : "fit");
const auto coverPngTempPath = getCachePath() + "/.cover.png";
if (!extractItemToFile(coverImageHref, coverPngTempPath)) return false;
HalFile coverPng;
if (!Storage.openFileForWrite("EBP", coverPngTempPath, coverPng)) {
return false;
}
readItemContentsToStream(coverImageHref, coverPng, 1024);
// Explicitly close() file before reopening for reading
coverPng.close();
if (!Storage.openFileForRead("EBP", coverPngTempPath, coverPng)) {
return false;
}
@@ -669,15 +677,9 @@ bool Epub::generateThumbBmp(int height) const {
} else if (FsHelpers::hasJpgExtension(coverImageHref)) {
LOG_DBG("EBP", "Generating thumb BMP from JPG cover image");
const auto coverJpgTempPath = getCachePath() + "/.cover.jpg";
if (!extractItemToFile(coverImageHref, coverJpgTempPath)) return false;
HalFile coverJpg;
if (!Storage.openFileForWrite("EBP", coverJpgTempPath, coverJpg)) {
return false;
}
readItemContentsToStream(coverImageHref, coverJpg, 1024);
// Explicitly close() file before reopening for reading
coverJpg.close();
if (!Storage.openFileForRead("EBP", coverJpgTempPath, coverJpg)) {
return false;
}
@@ -706,15 +708,9 @@ bool Epub::generateThumbBmp(int height) const {
} else if (FsHelpers::hasPngExtension(coverImageHref)) {
LOG_DBG("EBP", "Generating thumb BMP from PNG cover image");
const auto coverPngTempPath = getCachePath() + "/.cover.png";
if (!extractItemToFile(coverImageHref, coverPngTempPath)) return false;
HalFile coverPng;
if (!Storage.openFileForWrite("EBP", coverPngTempPath, coverPng)) {
return false;
}
readItemContentsToStream(coverImageHref, coverPng, 1024);
// Explicitly close() file before reopening for reading
coverPng.close();
if (!Storage.openFileForRead("EBP", coverPngTempPath, coverPng)) {
return false;
}
@@ -756,6 +752,38 @@ uint8_t* Epub::readItemContentsToBytes(const std::string& itemHref, size_t* size
const std::string path = FsHelpers::normalisePath(itemHref);
// Decode encrypted entries on demand in memory.
if (decryptor && decryptor->isEncrypted(path)) {
const size_t plainSize = decryptor->decryptedSize(path);
if (plainSize > SIZE_MAX - (trailingNullByte ? 1 : 0)) return nullptr;
const size_t total = plainSize + (trailingNullByte ? 1 : 0);
uint8_t* content = static_cast<uint8_t*>(malloc(total > 0 ? total : 1));
if (!content) {
LOG_ERR("EBP", "insufficient memory for %s (%u bytes)", path.c_str(), static_cast<unsigned>(total));
return nullptr;
}
struct BufferSink {
uint8_t* data;
size_t capacity;
size_t written;
} state{content, plainSize, 0};
auto append = [](void* context, const uint8_t* data, size_t size) {
auto* target = static_cast<BufferSink*>(context);
if (size > target->capacity - target->written) return false;
memcpy(target->data + target->written, data, size);
target->written += size;
return true;
};
if (!decryptor->decryptToSink(path, append, &state) || state.written != plainSize) {
free(content);
LOG_ERR("EBP", "content read failed for %s", path.c_str());
return nullptr;
}
if (trailingNullByte) content[plainSize] = 0;
if (size) *size = plainSize;
return content;
}
const auto content = ZipFile(filepath).readFileToMemory(path.c_str(), size, trailingNullByte);
if (!content) {
LOG_DBG("EBP", "Failed to read item %s", path.c_str());
@@ -773,6 +801,22 @@ bool Epub::readItemContentsToStream(const std::string& itemHref, Print& out, con
}
const std::string path = FsHelpers::normalisePath(itemHref);
if (decryptor && decryptor->isEncrypted(path)) {
struct PrintSink {
Print* output;
} state{&out};
auto append = [](void* context, const uint8_t* data, size_t size) {
auto* target = static_cast<PrintSink*>(context);
return target->output->write(data, size) == size;
};
if (!decryptor->decryptToSink(path, append, &state)) {
LOG_ERR("EBP", "content read failed for %s", path.c_str());
return false;
}
return true;
}
return ZipFile(filepath).readFileToStream(path.c_str(), out, chunkSize, allowEarlyStop);
}
@@ -802,7 +846,12 @@ int Epub::getSpineItemsCount() const {
return bookMetadataCache->getSpineCount();
}
size_t Epub::getCumulativeSpineItemSize(const int spineIndex) const { return getSpineItem(spineIndex).cumulativeSize; }
size_t Epub::getCumulativeSpineItemSize(const int spineIndex) const {
if (!bookMetadataCache || !bookMetadataCache->isLoaded()) {
return 0;
}
return bookMetadataCache->getCumulativeSize(spineIndex);
}
BookMetadataCache::SpineEntry Epub::getSpineItem(const int spineIndex) const {
if (!bookMetadataCache || !bookMetadataCache->isLoaded()) {
+9
View File
@@ -1,5 +1,6 @@
#pragma once
#include <ContentProtection.h>
#include <Print.h>
#include <memory>
@@ -29,6 +30,11 @@ class Epub {
std::unique_ptr<CssParser> cssParser;
// CSS files
std::vector<std::string> cssFiles;
// Optional encrypted-entry accessor. Entries are decoded in memory and stay
// encrypted at rest. Null when the accessor is not needed or unavailable.
std::unique_ptr<freeink::content::ContentDecryptor> decryptor;
// User-presentable reason the encrypted-entry accessor could not be opened.
std::string protectionError;
bool findContentOpfFile(std::string* contentOpfFile) const;
bool parseContentOpf(BookMetadataCache::BookMetadata& bookMetadata, bool writeSpineEntries = true);
@@ -49,6 +55,9 @@ class Epub {
void setupCacheDir() const;
const std::string& getCachePath() const;
const std::string& getPath() const;
// Empty unless the encrypted-entry accessor failed to open.
const std::string& getProtectionError() const { return protectionError; }
bool isProtected() const { return decryptor != nullptr; }
const std::string& getTitle() const;
const std::string& getAuthor() const;
const std::string& getLanguage() const;
+19
View File
@@ -481,11 +481,30 @@ bool BookMetadataCache::load() {
serialization::readString(bookFile, coreMetadata.coverItemHref);
serialization::readString(bookFile, coreMetadata.textReferenceHref);
// Cache cumulative spine sizes in RAM. The progress bar (every render) and percent
// jumps otherwise pay 2 seeks + a heap-allocating SpineEntry read per access. Spine
// entries are stored contiguously in index order immediately after the LUTs, so read
// them in a single sequential pass.
cumulativeSizes.clear();
cumulativeSizes.reserve(spineCount);
const uint32_t lutSize = (static_cast<uint32_t>(spineCount) + tocCount) * sizeof(uint32_t);
bookFile.seek(lutOffset + lutSize);
for (uint16_t i = 0; i < spineCount; i++) {
cumulativeSizes.push_back(readSpineEntry(bookFile).cumulativeSize);
}
loaded = true;
LOG_DBG("BMC", "Loaded cache data: %d spine, %d TOC entries", spineCount, tocCount);
return true;
}
uint32_t BookMetadataCache::getCumulativeSize(const int index) const {
if (index < 0 || index >= static_cast<int>(cumulativeSizes.size())) {
return 0;
}
return cumulativeSizes[index];
}
BookMetadataCache::SpineEntry BookMetadataCache::getSpineEntry(const int index) {
if (!loaded) {
LOG_ERR("BMC", "getSpineEntry called but cache not loaded");
+9
View File
@@ -7,6 +7,7 @@
#include <deque>
#include <memory>
#include <string>
#include <vector>
class BookMetadataCache {
public:
@@ -62,6 +63,11 @@ class BookMetadataCache {
// wrapper serves whichever pass is active (spine, then toc).
std::unique_ptr<serialization::BufferedFileWriter> passOut;
// Cumulative spine sizes, cached in RAM at load() so progress/percent lookups are
// O(1) instead of 2 seeks + a heap-allocating SpineEntry read per access (4 bytes
// per spine item; <1KB for typical books).
std::vector<uint32_t> cumulativeSizes;
// Index for fast href→spineIndex lookup (used only for large EPUBs)
struct SpineHrefIndexEntry {
uint64_t hrefHash; // FNV-1a 64-bit hash
@@ -113,6 +119,9 @@ class BookMetadataCache {
bool load();
SpineEntry getSpineEntry(int index);
TocEntry getTocEntry(int index);
// Cumulative byte size up to and including the given spine item (0 if out of range
// or not loaded). Backed by the in-RAM cumulativeSizes cache populated in load().
uint32_t getCumulativeSize(int index) const;
int getSpineCount() const { return spineCount; }
int getTocCount() const { return tocCount; }
bool isLoaded() const { return loaded; }
+4 -1
View File
@@ -3,7 +3,10 @@
#include <cstring>
#define FOOTNOTE_NUMBER_LEN 32
#define FOOTNOTE_HREF_LEN 96
#define FOOTNOTE_HREF_LEN 256
// ponytail: bumped from 96 to 256; calibre-generated EPUBs with long filenames
// and URL-encoded characters routinely exceed 96 chars (e.g.
// "Author-Title_split_NNN.html#_ftnN" encoded is ~150 chars).
struct FootnoteEntry {
char number[FOOTNOTE_NUMBER_LEN];
+3 -1
View File
@@ -31,7 +31,9 @@ namespace {
// match. Keeps <br>-per-paragraph books (common CJK formatting) from
// re-adding container spacing at every paragraph.
// v35: Persist a uint32_t visible-text start offset for every page.
constexpr uint8_t SECTION_FILE_VERSION = 36;
// v36: Ruby and CJK justification layout changes invalidate cached word positions.
// v37: Footnote href records grew from 96 to 256 bytes.
constexpr uint8_t SECTION_FILE_VERSION = 37;
// Written into the version field while a build is in progress; patched to
// SECTION_FILE_VERSION only when the build is finalized. An abandoned /
// crash-interrupted .bin therefore carries version 0, which loadSectionFile rejects
+2
View File
@@ -357,6 +357,7 @@ void ImageBlock::render(GfxRenderer& renderer, const int x, const int y) {
// Try to render from cache first
std::string cachePath = getCachePath(imagePath);
if (renderFromCache(renderer, cachePath, x, y, width, height)) {
renderer.preserveImagePolarity(x, y, width, height);
return; // Successfully rendered from cache
}
@@ -419,6 +420,7 @@ void ImageBlock::render(GfxRenderer& renderer, const int x, const int y) {
return;
}
renderer.preserveImagePolarity(x, y, width, height);
LOG_DBG("IMG", "Decode successful");
}
Loaded 20 of 179 files, more files were not shown because too many files have changed in this diff. Show more