Compare commits

...

39 Commits

Author SHA1 Message Date
Dave Allie b1763821b5 Cut release 0.10.0 2025-12-29 02:30:27 +11:00
Dave Allie c0b83b626e Use a JSON filter to avoid crashes when checking for updates (#141)
## Summary

* The JSON release data from Github contains the entire release
description which can be very large
  * The 0.9.0 release was especially bad
* Use a JSON filter to avoid deserializing anything but the necessary
fields

## Additional Context

*
https://arduinojson.org/v7/how-to/deserialize-a-very-large-document/#filtering
* Fixes https://github.com/daveallie/crosspoint-reader/issues/124
2025-12-29 02:29:41 +11:00
Dave Allie f8c0b1acea Use confirmation release on home screen to detect action 2025-12-29 02:00:42 +11:00
Eunchurn Park f9b604f04e Add XTC/XTCH ebook format support (#135)
## Summary

* **What is the goal of this PR?**

Add support for XTC (XTeink X4 native) ebook format, which contains
pre-rendered 480x800 1-bit bitmap pages optimized for e-ink displays.

* **What changes are included?**

- New `lib/Xtc/` library with XtcParser for reading XTC files
- XtcReaderActivity for displaying XTC pages on e-ink display
- XTC file detection in FileSelectionActivity
- Cover BMP generation from first XTC page
- Correct XTG page header structure (22 bytes) and bit polarity handling

## Additional Context

- XTC files contain pre-rendered bitmap pages with embedded status bar
(page numbers, progress %)
- XTG page header: 22 bytes (magic + dimensions + reserved fields +
bitmap size)
- Bit polarity: 0 = black, 1 = white
- No runtime text rendering needed - pages display directly on e-ink
- Faster page display compared to EPUB since no parsing/rendering
required
- Memory efficient: loads one page at a time (48KB per page)
- Tested with XTC files generated from https://x4converter.rho.sh/
- Verified correct page alignment and color rendering
- Please report any issues if you test with XTC files from other
sources.

---------

Co-authored-by: Dave Allie <dave@daveallie.com>
2025-12-29 01:56:05 +11:00
Dave Allie 3dc5f6fec4 Avoid jumping straight into chapter selection screen 2025-12-28 23:49:51 +11:00
Dave Allie 41c93e4eba Use font ascender height for baseline offset (#139)
## Summary

* Use font ascender height for baseline offset
* Previously was using font height, but when rendering the font (even
from y = 0), there would be a lot of top margin
* Font would also go below the "bottom of the line" as we were using the
full font height as the baseline

## Additional Context

* This caused some text to move around, I've fixed everything I can
* Notably it moves the first line of font a little closer to the top of
the page
2025-12-28 22:30:01 +11:00
Dave Allie 1c33162368 Fix rendering issue with entering keyboard from wifi screen 2025-12-28 21:50:45 +11:00
Dave Allie 27d42fbef3 Allow entering into chapter select screen correctly 2025-12-28 21:50:36 +11:00
Tannay dd280bdc97 Rotation Support (#77)
•  What is the goal of this PR?  
Implement a horizontal EPUB reading mode so books can be read in
landscape orientation (both 90° and 270°), while keeping the rest of the
UI in portrait.

•  What changes are included?
â—¦  Rendering / Display
â–Ş Added an orientation model to GfxRenderer (Portrait, LandscapeNormal,
LandscapeFlipped) and made:
â–Ş drawPixel, drawImage, displayWindow map logical coordinates
differently depending on orientation.
▪ getScreenWidth() / getScreenHeight() return orientation‑aware logical
dimensions (480Ă—800 in portrait, 800Ă—480 in landscape).
â—¦  Settings / Configuration
â–Ş  Extended CrossPointSettings with:
â–Ş  landscapeReading (toggle for portrait vs. landscape EPUB reading).
▪ landscapeFlipped (toggle to flip landscape 180° so both horizontal
holding directions are supported).
â–Ş Updated settings serialization/deserialization to persist these fields
while remaining backward‑compatible with existing settings files.
â–Ş  Updated SettingsActivity to expose two new toggles:
▪  “Landscape Reading”
▪  “Flip Landscape (swap top/bottom)”
â—¦  EPUB Reader
â–Ş  In EpubReaderActivity:
â–Ş On onEnter, set GfxRenderer orientation based on the new settings
(Portrait, LandscapeNormal, or LandscapeFlipped).
â–Ş On onExit, reset orientation back to Portrait so Home, WiFi, Settings,
etc. continue to render as before.
â–Ş Adjusted renderStatusBar to position the status bar and battery
indicator relative to GfxRenderer::getScreenHeight() instead of
hard‑coded Y coordinates, so it stays correctly at the bottom in both
portrait and landscape.
â—¦  EPUB Caching / Layout
â–Ş Extended Section cache metadata (section.bin) to include the logical
screenWidth and screenHeight used when pages were generated; bumped
SECTION_FILE_VERSION.
â–Ş  Updated loadCacheMetadata to compare:
â–Ş font/margins/line compression/extraParagraphSpacing and screen
dimensions; mismatches now invalidate and clear the cache.
â–Ş Updated persistPageDataToSD and all call sites in EpubReaderActivity
to pass the current GfxRenderer::getScreenWidth() / getScreenHeight() so
portrait and landscape caches are kept separate and correctly sized.



Additional Context

•  Cache behavior / migration
â—¦ Existing section.bin files (old SECTION_FILE_VERSION) will be detected
as incompatible and their caches cleared and rebuilt once per chapter
when first opened after this change.
â—¦ Within a given orientation, caches will be reused as before. Switching
orientation (portrait ↔ landscape) will cause a one‑time re‑index of
each chapter in the new orientation.
•  Scope and risks
â—¦ Orientation changes are scoped to the EPUB reader; the Home screen,
Settings, WiFi selection, sleep screens, and web server UI continue to
assume portrait orientation.
◦ The renderer’s orientation is a static/global setting; if future code
uses GfxRenderer outside the reader while a reader instance is active,
it should be aware that orientation is no longer implicitly fixed.
◦ All drawing primitives now go through orientation‑aware coordinate
transforms; any code that previously relied on edge‑case behavior or
out‑of‑bounds writes might surface as logged “Outside range” warnings
instead.
•  Testing suggestions / areas to focus on
â—¦  Verify in hardware:
â–Ş Portrait mode still renders correctly (boot, home, settings, WiFi,
reader).
â–Ş  Landscape reading in both directions:
â–Ş  Landscape Reading = ON, Flip Landscape = OFF.
â–Ş  Landscape Reading = ON, Flip Landscape = ON.
â–Ş Status bar (page X/Y, % progress, battery icon) is fully visible and
aligned at the bottom in all three combinations.
â—¦  Open the same book:
â–Ş  In portrait first, then switch to landscape and reopen it.
â–Ş  Confirm that:
â–Ş Old portrait caches are rebuilt once for landscape (you should see the
“Indexing…” page).
â–Ş Progress save/restore still works (resume opens to the correct page in
the current orientation).
â—¦ Ensure grayscale rendering (the secondary pass in
EpubReaderActivity::renderContents) still looks correct in both
orientations.

---------

Co-authored-by: Dave Allie <dave@daveallie.com>
2025-12-28 21:33:20 +11:00
Dave Allie bf031fd999 Fix exiting WifiSelectionActivity renderer early 2025-12-28 19:27:00 +11:00
Dave Allie 02350c6a9f Fix underscore on keyboard and standardize activity (#138)
## Summary

* Fix underscore on keyboard
  * Remove special handling of special row characters
* Fix navigating between special row items
* Standardize keyboard activity to use standard loop
  * Fix issue with rendering keyboard non-stop

Fixes https://github.com/daveallie/crosspoint-reader/issues/131
2025-12-28 18:57:06 +11:00
Dave Allie 9023b262a1 Fix issue where pressing back from chapter select would leave book (#137)
## Summary

* Fix issue where pressing back from chapter select would leave book
* Rely on `wasReleased` checks instead
2025-12-28 17:06:18 +11:00
Eunchurn Park eabd149371 Add retry logic and progress bar for chapter indexing (#128)
## Summary

* **What is the goal of this PR?**

Improve reliability and user experience during chapter indexing by
adding retry logic for SD card operations and a visual progress bar.

* **What changes are included?**

- **Retry logic**: Add 3 retry attempts with 50ms delay for ZIP to SD
card streaming to handle timing issues after display refresh
- **Progress bar**: Display a visual progress bar (0-100%) during
chapter indexing based on file read progress, updating every 10% to
balance responsiveness with e-ink display limitations

## Additional Context

* **Problem observed**: When navigating quickly through books with many
chapters (before chapter titles finish rendering), the "Indexing..."
screen would appear frozen. Checking the serial log revealed the
operation had silently failed, but the UI showed no indication of this.
Users would likely assume the device had crashed. Pressing the next
button again would resume operation, but this behavior was confusing and
unexpected.

* **Solution**:
- Retry logic handles transient SD card timing failures automatically,
so users don't need to manually retry
- Progress bar provides visual feedback so users know indexing is
actively working (not frozen)

* **Why timing issues occur**: After display refresh operations, there
can be timing conflicts when immediately starting SD card write
operations. This is more likely to happen when rapidly navigating
through chapters.

* **Progress bar design**: Updates every 10% to avoid excessive e-ink
refreshes while still providing meaningful feedback during long indexing
operations (especially for large chapters with CJK characters).

* **Performance**: Minimal overhead - progress calculation is simple
byte counting, and display updates use `FAST_REFRESH` mode.
2025-12-28 15:59:44 +11:00
1991AcuraLegend 838246d147 Add setting to enable status bar display options (#111)
Add setting toggle that allows status bar display options in EpubReader.

Supported options would be as follows: 

- FULL: display as is today
- PROGRESS: display progress bar only
- BATTERY: display battery only
- NONE: hide status bar

---------

Co-authored-by: Dave Allie <dave@daveallie.com>
2025-12-28 10:48:27 +11:00
Eunchurn Park f96b6ab29c Improve EPUB cover image quality with pre-scaling and Atkinson dithering (#116)
## Summary

* **What is the goal of this PR?**

Replace simple threshold-based grayscale quantization with ordered
dithering using a 4x4 Bayer matrix. This eliminates color banding
artifacts and produces smoother gradients on e-ink display.

* **What changes are included?**

- Add 4x4 Bayer dithering matrix for 16-level threshold patterns
- Modify `grayscaleTo2Bit()` function to accept pixel coordinates and
apply position-based dithering
- Replace simple `grayscale >> 6` threshold with ordered dithering
algorithm that produces smoother gradients

## Additional Context

* Bayer matrix approach: The 4x4 Bayer matrix creates a repeating
pattern that distributes quantization error spatially, effectively
simulating 16 levels of gray using only 4 actual color levels (black,
dark gray, light gray, white).

* Cache invalidation: Existing cached `cover.bmp` files will need to be
deleted to see the improved rendering, as the converter only runs when
the cache is missing.
2025-12-28 10:38:14 +11:00
Brendan O'Leary e3d0201365 Add 'Open' button hint to File Selection page (#136)
## Summary

In using my build of
https://github.com/daveallie/crosspoint-reader/pull/130 I realized that
we need a "open" button hint above the second button in the File browser

## Additional Context

* Add any other information that might be helpful for the reviewer
(e.g., performance implications, potential risks, specific areas to
focus on).
2025-12-28 10:36:26 +11:00
Eunchurn Park 286b47f489 fix(parser): remove MAX_LINES limit that truncates long chapters (#132)
## Summary

* **What is the goal of this PR?** Fixes a bug where text disappears
after approximately 25 pages in long chapters during EPUB indexing.

* **What changes are included?**
- Removed the `MAX_LINES = 1000` hard limit in
`ParsedText::computeLineBreaks()`
- Added safer infinite loop prevention by checking if `nextBreakIndex <=
currentWordIndex` and forcing advancement by one word when stuck

## Additional Context

* **Root cause:** The `MAX_LINES = 1000` limit was introduced to prevent
infinite loops, but it truncates content in long chapters. For example,
a 93KB chapter that generates ~242 pages (~9,680 lines) gets cut off at
~1000 lines, causing blank pages after page 25-27.

* **Solution approach:** Instead of a hard line limit, I now detect when
the line break algorithm gets stuck (when `nextBreakIndex` doesn't
advance) and force progress by moving one word at a time. This preserves
the infinite loop protection while allowing all content to be rendered.

* **Testing:** Verified with a Korean EPUB containing a 93KB chapter -
all 242 pages now render correctly without text disappearing.
2025-12-28 10:35:45 +11:00
Dave Allie aff4dc6628 Fix QRCode import attempt 2 2025-12-26 11:33:41 +10:00
Dave Allie 98a39374e8 Fix QRCode import 2025-12-26 11:29:27 +10:00
Jonas Diemer e8c0fb42d4 Network details QR code (#113)
Using QRCode library from pio to generate the QR code.

Done:
- Display QR code for URL in network mode
- minor fixes of layout
- Display QR for URL in AP mode
- Display QR for AP in AP mode

---------

Co-authored-by: Dave Allie <dave@daveallie.com>
2025-12-26 12:13:40 +11:00
Eunchurn Park b77af16caa Add Continue Reading menu and remember last book folder (#129)
## Summary

* **What is the goal of this PR?**

Add a "Continue Reading" feature to improve user experience when
returning to a previously opened book.

* **What changes are included?**

- Add dynamic "Continue: <book name>" menu item in Home screen when a
book was previously opened

- File browser now starts from the folder of the last opened book
instead of always starting from root directory
- Menu dynamically shows 3 or 4 items based on reading history:
  - Without history: `Browse`, `File transfer`, `Settings`
- With history: `Continue: <book>`, `Browse`, `File transfer`,
`Settings`

## Additional Context

* This feature leverages the existing `APP_STATE.openEpubPath` which
already persists the last opened book path
* The Continue Reading menu only appears if the book file still exists
on the SD card
* Book name in the menu is truncated to 25 characters with "..." suffix
if too long
* If the last book's folder was deleted, the file browser gracefully
falls back to root directory
* No new dependencies or significant memory overhead - reuses existing
state management
2025-12-26 11:55:23 +11:00
Brendan O'Leary e3c1e28b8f Normalize button hints (#130)
## Summary

This creates a `renderer.drawButtonHints` to make all of the "hints"
over buttons to match the home screen.

## Additional Context

* Add any other information that might be helpful for the reviewer
(e.g., performance implications, potential risks, specific areas to
focus on).

---------

Co-authored-by: Dave Allie <dave@daveallie.com>
2025-12-26 11:54:02 +11:00
Eunchurn Park dc7544d944 Optimize glyph lookup with binary search (#125)
Replace linear O(n) search with binary search O(log n) for unicode
interval lookup. Korean fonts have many intervals (~30,000+ glyphs), so
this improves text rendering performance during page navigation.

## Summary

* **What is the goal of this PR?** (e.g., Fixes a bug in the user
authentication module, Implements the new feature for
  file uploading.)

Replace linear `O(n)` glyph lookup with binary search `O(log n)` to
improve text rendering performance during page navigation.

* **What changes are included?**

- Modified `EpdFont::getGlyph()` to use binary search instead of linear
search for unicode interval lookup
- Added early return for empty interval count

## Additional Context

* Add any other information that might be helpful for the reviewer
(e.g., performance implications, potential risks, specific areas to
focus on).

- Performance implications: Fonts with many unicode intervals benefit
the most. Korean fonts have ~30,000+ glyphs across multiple intervals,
but any font with significant glyph coverage (CJK, extended Latin,
emoji, etc.) will see improvement.
- Complexity: from `O(n)` to `O(log n)` where n = number of unicode
intervals. For fonts with 10+ intervals, this reduces lookup iterations
significantly.
- Risk: Low - the binary search logic is straightforward and the
intervals are already sorted by unicode codepoint (required for the
original early-exit optimization).
2025-12-26 11:46:17 +11:00
Dave Allie 504c7b307d Cut release 0.9.0 2025-12-24 21:49:47 +10:00
Dave Allie b6bc1f7ed3 New book.bin spine and table of contents cache (#104)
## Summary

* Use single unified cache file for book spine, table of contents, and
core metadata (title, author, cover image)
* Use new temp item store file in OPF parsing to store items to be
rescaned when parsing spine
  * This avoids us holding these items in memory
* Use new toc.bin.tmp and spine.bin.tmp to build out partial toc / spine
data as part of parsing content.opf and the NCX file
  * These files are re-read multiple times to ultimately build book.bin

## Additional Context

* Spec for file format included below as an image
* This should help with:
  * #10 
  * #60 
  * #99
2025-12-24 22:36:13 +11:00
Dave Allie ea0abaf351 Prevent SD card error causing boot loop (#122)
## Summary

* Prevent SD card error causing boot loop
* We need the screen and fonts to be initialized to show the full screen
error message
* Prior to this change, trying to render the font would crash the
firmware and boot loop it
2025-12-24 22:33:21 +11:00
Dave Allie 2771579007 Add support for blockquote, strong, and em tags (#121)
## Summary

* Add support for blockquote, strong, and em tags
2025-12-24 22:33:17 +11:00
Dave Allie 27035b2b91 Handle 16x16 MCU blocks in JPEG decoding (#120)
## Summary

* Handle 16x16 MCU blocks in JPEG decoding
* We were only correctly handling 8x8 blocks, which means that we did
not correctly support a lot of JPGs leading to an interlacing style on
the images

## Additional Context

* Fixes https://github.com/daveallie/crosspoint-reader/issues/118
2025-12-24 22:21:41 +11:00
Dave Allie 1107590b56 Standardize File handling with FsHelpers (#110)
## Summary

* Standardize File handling with FsHelpers
* Better central place to manage to logic of if files exist/open for
reading/writing
2025-12-23 14:14:10 +11:00
Dave Allie 66ddb52103 Pin espressif32 platform version 2025-12-23 12:17:12 +11:00
Brendan O'Leary 9f4f71fabe Add AP mode option for file transfers (#98)
## Summary

* **What is the goal of this PR?** Adds WiFi Access Point (AP) mode
support for File Transfer, allowing the device to create its own WiFi
network that users can connect to directly - useful when no existing
WiFi network is available. And in my experience is faster when the
device is right next to your laptop (but maybe further from your wifi)

* **What changes are included?**
- New `NetworkModeSelectionActivity` - an interstitial screen asking
users to choose between:
- "Join a Network" - connects to an existing WiFi network (existing
behavior)
- "Create Hotspot" - creates a WiFi access point named
"CrossPoint-Reader"
  - Modified `CrossPointWebServerActivity` to:
    - Launch the network mode selection screen before proceeding
- Support starting an Access Point with mDNS (`crosspoint.local`) and
DNS server for captive portal behavior
    - Display appropriate connection info for both modes
- Modified `CrossPointWebServer` to support starting when WiFi is in AP
mode (not just STA connected mode)

## Additional Context

* **AP Mode Details**: The device creates an open WiFi network named
"CrossPoint-Reader". Once connected, users can access the file transfer
page at `http://crosspoint.local/` or `http://192.168.4.1/`
* **DNS Captive Portal**: A DNS server redirects all domain requests to
the device's IP, enabling captive portal behavior on some devices
* **mDNS**: Hostname resolution via `crosspoint.local` is enabled for
both AP and STA modes
* **No breaking changes**: The "Join a Network" option preserves the
existing WiFi connection flow
* **Memory impact**: Minimal - the AP mode uses roughly the same
resources as STA mode
2025-12-22 17:24:14 +11:00
Dave Allie d23020e268 OTA updates (#96)
## Summary

* Adds support for OTA
  * Gets latest firmware bin from latest GitHub release
* I have noticed it be a little flaky unpacking the JSON and
occasionally failing to start
2025-12-22 17:16:46 +11:00
Dave Allie f4491875ab Thoroughly deinitialise expat parsers before freeing them (#103)
## Summary

* Thoroughly deinitialise expat parsers before freeing them
* Spotted a few crashes when de-initing expat parsers
2025-12-22 17:16:39 +11:00
Dave Allie 6fe28da41b Cut release 0.8.1 2025-12-22 03:20:22 +11:00
Dave Allie 689b539c6b Stream CrossPointWebServer data over JSON APIs (#97)
## Summary

* HTML files are now static, streamed directly to the client without
modification
* For any dynamic values, load via JSON APIs
* For files page, we stream the JSON content as we scan the directory to
avoid holding onto too much data

## Additional details

* We were previously building up a very large string all generated on
the X4 directly, we should be leveraging the browser
* Fixes https://github.com/daveallie/crosspoint-reader/issues/94
2025-12-22 03:19:49 +11:00
Jonas Diemer ce37c80c2d Improve power button hold measurement for boot (#95)
Improves the duration for which the power button needs to be held - see
#53.

I left the measurement code for the calibration value in, as it will
likely change if we move the settings to NVS.
2025-12-22 00:53:55 +11:00
Dave Allie b39ce22e54 Cleanup of activities 2025-12-22 00:48:16 +11:00
Dave Allie 77c655fcf5 Give activities names and log when entering and exiting them (#92)
## Summary

* Give activities name and log when entering and exiting them
* Clearer logs when attempting to debug, knowing where users are coming
from/going to helps
2025-12-21 21:17:00 +11:00
Dave Allie 246afae6ef Start power off sequence as soon as hold duration for the power button is reached (#93)
## Summary

* Swap from `wasReleased` to `isPressed` when checking power button
duration
  * In practice it makes the power down experience feel a lot snappier
* Remove the unnecessary 1000ms delay when powering off

## Additional Context

* A little discussion in here:
https://github.com/daveallie/crosspoint-reader/discussions/53#discussioncomment-15309707
2025-12-21 21:16:41 +11:00
86 changed files with 6410 additions and 2118 deletions
-6
View File
@@ -12,12 +12,6 @@ jobs:
- uses: actions/checkout@v6
with:
submodules: recursive
- uses: actions/cache@v5
with:
path: |
~/.cache/pip
~/.platformio/.cache
key: ${{ runner.os }}-pio
- uses: actions/setup-python@v6
with:
python-version: '3.14'
+20 -6
View File
@@ -59,14 +59,28 @@ bool EpdFont::hasPrintableChars(const char* string) const {
const EpdGlyph* EpdFont::getGlyph(const uint32_t cp) const {
const EpdUnicodeInterval* intervals = data->intervals;
for (int i = 0; i < data->intervalCount; i++) {
const EpdUnicodeInterval* interval = &intervals[i];
if (cp >= interval->first && cp <= interval->last) {
const int count = data->intervalCount;
if (count == 0) return nullptr;
// Binary search for O(log n) lookup instead of O(n)
// Critical for Korean fonts with many unicode intervals
int left = 0;
int right = count - 1;
while (left <= right) {
const int mid = left + (right - left) / 2;
const EpdUnicodeInterval* interval = &intervals[mid];
if (cp < interval->first) {
right = mid - 1;
} else if (cp > interval->last) {
left = mid + 1;
} else {
// Found: cp >= interval->first && cp <= interval->last
return &data->glyph[interval->offset + (cp - interval->first)];
}
if (cp < interval->first) {
return nullptr;
}
}
return nullptr;
}
+173 -158
View File
@@ -1,13 +1,11 @@
#include "Epub.h"
#include <FsHelpers.h>
#include <HardwareSerial.h>
#include <JpegToBmpConverter.h>
#include <SD.h>
#include <ZipFile.h>
#include <map>
#include "Epub/FsHelpers.h"
#include "Epub/parsers/ContainerParser.h"
#include "Epub/parsers/ContentOpfParser.h"
#include "Epub/parsers/TocNcxParser.h"
@@ -44,7 +42,15 @@ bool Epub::findContentOpfFile(std::string* contentOpfFile) const {
return true;
}
bool Epub::parseContentOpf(const std::string& contentOpfFilePath) {
bool Epub::parseContentOpf(BookMetadataCache::BookMetadata& bookMetadata) {
std::string contentOpfFilePath;
if (!findContentOpfFile(&contentOpfFilePath)) {
Serial.printf("[%lu] [EBP] Could not find content.opf in zip\n", millis());
return false;
}
contentBasePath = contentOpfFilePath.substr(0, contentOpfFilePath.find_last_of('/') + 1);
Serial.printf("[%lu] [EBP] Parsing content.opf: %s\n", millis(), contentOpfFilePath.c_str());
size_t contentOpfSize;
@@ -53,7 +59,9 @@ bool Epub::parseContentOpf(const std::string& contentOpfFilePath) {
return false;
}
ContentOpfParser opfParser(getBasePath(), contentOpfSize);
ContentOpfParser opfParser(getCachePath(), getBasePath(), contentOpfSize, bookMetadataCache.get());
Serial.printf("[%lu] [MEM] Free: %d bytes, Total: %d bytes, Min Free: %d bytes\n", millis(), ESP.getFreeHeap(),
ESP.getHeapSize(), ESP.getMinFreeHeap());
if (!opfParser.setup()) {
Serial.printf("[%lu] [EBP] Could not setup content.opf parser\n", millis());
@@ -66,26 +74,20 @@ bool Epub::parseContentOpf(const std::string& contentOpfFilePath) {
}
// Grab data from opfParser into epub
title = opfParser.title;
if (!opfParser.coverItemId.empty() && opfParser.items.count(opfParser.coverItemId) > 0) {
coverImageItem = opfParser.items.at(opfParser.coverItemId);
}
bookMetadata.title = opfParser.title;
// TODO: Parse author
bookMetadata.author = "";
bookMetadata.coverItemHref = opfParser.coverItemHref;
if (!opfParser.tocNcxPath.empty()) {
tocNcxItem = opfParser.tocNcxPath;
}
for (auto& spineRef : opfParser.spineRefs) {
if (opfParser.items.count(spineRef)) {
spine.emplace_back(spineRef, opfParser.items.at(spineRef));
}
}
Serial.printf("[%lu] [EBP] Successfully parsed content.opf\n", millis());
return true;
}
bool Epub::parseTocNcxFile() {
bool Epub::parseTocNcxFile() const {
// the ncx file should have been specified in the content.opf file
if (tocNcxItem.empty()) {
Serial.printf("[%lu] [EBP] No ncx file specified\n", millis());
@@ -95,13 +97,18 @@ bool Epub::parseTocNcxFile() {
Serial.printf("[%lu] [EBP] Parsing toc ncx file: %s\n", millis(), tocNcxItem.c_str());
const auto tmpNcxPath = getCachePath() + "/toc.ncx";
File tempNcxFile = SD.open(tmpNcxPath.c_str(), FILE_WRITE);
File tempNcxFile;
if (!FsHelpers::openFileForWrite("EBP", tmpNcxPath, tempNcxFile)) {
return false;
}
readItemContentsToStream(tocNcxItem, tempNcxFile, 1024);
tempNcxFile.close();
tempNcxFile = SD.open(tmpNcxPath.c_str(), FILE_READ);
if (!FsHelpers::openFileForRead("EBP", tmpNcxPath, tempNcxFile)) {
return false;
}
const auto ncxSize = tempNcxFile.size();
TocNcxParser ncxParser(contentBasePath, ncxSize);
TocNcxParser ncxParser(contentBasePath, ncxSize, bookMetadataCache.get());
if (!ncxParser.setup()) {
Serial.printf("[%lu] [EBP] Could not setup toc ncx parser\n", millis());
@@ -130,9 +137,7 @@ bool Epub::parseTocNcxFile() {
tempNcxFile.close();
SD.remove(tmpNcxPath.c_str());
this->toc = std::move(ncxParser.toc);
Serial.printf("[%lu] [EBP] Parsed %d TOC items\n", millis(), this->toc.size());
Serial.printf("[%lu] [EBP] Parsed TOC items\n", millis());
return true;
}
@@ -140,48 +145,79 @@ bool Epub::parseTocNcxFile() {
bool Epub::load() {
Serial.printf("[%lu] [EBP] Loading ePub: %s\n", millis(), filepath.c_str());
std::string contentOpfFilePath;
if (!findContentOpfFile(&contentOpfFilePath)) {
Serial.printf("[%lu] [EBP] Could not find content.opf in zip\n", millis());
// Initialize spine/TOC cache
bookMetadataCache.reset(new BookMetadataCache(cachePath));
// Try to load existing cache first
if (bookMetadataCache->load()) {
Serial.printf("[%lu] [EBP] Loaded ePub: %s\n", millis(), filepath.c_str());
return true;
}
// Cache doesn't exist or is invalid, build it
Serial.printf("[%lu] [EBP] Cache not found, building spine/TOC cache\n", millis());
setupCacheDir();
// Begin building cache - stream entries to disk immediately
if (!bookMetadataCache->beginWrite()) {
Serial.printf("[%lu] [EBP] Could not begin writing cache\n", millis());
return false;
}
Serial.printf("[%lu] [EBP] Found content.opf at: %s\n", millis(), contentOpfFilePath.c_str());
contentBasePath = contentOpfFilePath.substr(0, contentOpfFilePath.find_last_of('/') + 1);
if (!parseContentOpf(contentOpfFilePath)) {
// OPF Pass
BookMetadataCache::BookMetadata bookMetadata;
if (!bookMetadataCache->beginContentOpfPass()) {
Serial.printf("[%lu] [EBP] Could not begin writing content.opf pass\n", millis());
return false;
}
if (!parseContentOpf(bookMetadata)) {
Serial.printf("[%lu] [EBP] Could not parse content.opf\n", millis());
return false;
}
if (!bookMetadataCache->endContentOpfPass()) {
Serial.printf("[%lu] [EBP] Could not end writing content.opf pass\n", millis());
return false;
}
// TOC Pass
if (!bookMetadataCache->beginTocPass()) {
Serial.printf("[%lu] [EBP] Could not begin writing toc pass\n", millis());
return false;
}
if (!parseTocNcxFile()) {
Serial.printf("[%lu] [EBP] Could not parse toc\n", millis());
return false;
}
initializeSpineItemSizes();
Serial.printf("[%lu] [EBP] Loaded ePub: %s\n", millis(), filepath.c_str());
return true;
}
void Epub::initializeSpineItemSizes() {
Serial.printf("[%lu] [EBP] Calculating book size\n", millis());
const size_t spineItemsCount = getSpineItemsCount();
size_t cumSpineItemSize = 0;
const ZipFile zip("/sd" + filepath);
for (size_t i = 0; i < spineItemsCount; i++) {
std::string spineItem = getSpineItem(i);
size_t s = 0;
getItemSize(zip, spineItem, &s);
cumSpineItemSize += s;
cumulativeSpineItemSize.emplace_back(cumSpineItemSize);
if (!bookMetadataCache->endTocPass()) {
Serial.printf("[%lu] [EBP] Could not end writing toc pass\n", millis());
return false;
}
Serial.printf("[%lu] [EBP] Book size: %lu\n", millis(), cumSpineItemSize);
// Close the cache files
if (!bookMetadataCache->endWrite()) {
Serial.printf("[%lu] [EBP] Could not end writing cache\n", millis());
return false;
}
// Build final book.bin
if (!bookMetadataCache->buildBookBin(filepath, bookMetadata)) {
Serial.printf("[%lu] [EBP] Could not update mappings and sizes\n", millis());
return false;
}
if (!bookMetadataCache->cleanupTmpFiles()) {
Serial.printf("[%lu] [EBP] Could not cleanup tmp files - ignoring\n", millis());
}
// Reload the cache from disk so it's in the correct state
bookMetadataCache.reset(new BookMetadataCache(cachePath));
if (!bookMetadataCache->load()) {
Serial.printf("[%lu] [EBP] Failed to reload cache after writing\n", millis());
return false;
}
Serial.printf("[%lu] [EBP] Loaded ePub: %s\n", millis(), filepath.c_str());
return true;
}
bool Epub::clearCache() const {
@@ -217,7 +253,14 @@ const std::string& Epub::getCachePath() const { return cachePath; }
const std::string& Epub::getPath() const { return filepath; }
const std::string& Epub::getTitle() const { return title; }
const std::string& Epub::getTitle() const {
static std::string blank;
if (!bookMetadataCache || !bookMetadataCache->isLoaded()) {
return blank;
}
return bookMetadataCache->coreMetadata.title;
}
std::string Epub::getCoverBmpPath() const { return cachePath + "/cover.bmp"; }
@@ -227,24 +270,42 @@ bool Epub::generateCoverBmp() const {
return true;
}
if (coverImageItem.empty()) {
if (!bookMetadataCache || !bookMetadataCache->isLoaded()) {
Serial.printf("[%lu] [EBP] Cannot generate cover BMP, cache not loaded\n", millis());
return false;
}
const auto coverImageHref = bookMetadataCache->coreMetadata.coverItemHref;
if (coverImageHref.empty()) {
Serial.printf("[%lu] [EBP] No known cover image\n", millis());
return false;
}
if (coverImageItem.substr(coverImageItem.length() - 4) == ".jpg" ||
coverImageItem.substr(coverImageItem.length() - 5) == ".jpeg") {
if (coverImageHref.substr(coverImageHref.length() - 4) == ".jpg" ||
coverImageHref.substr(coverImageHref.length() - 5) == ".jpeg") {
Serial.printf("[%lu] [EBP] Generating BMP from JPG cover image\n", millis());
File coverJpg = SD.open((getCachePath() + "/.cover.jpg").c_str(), FILE_WRITE, true);
readItemContentsToStream(coverImageItem, coverJpg, 1024);
const auto coverJpgTempPath = getCachePath() + "/.cover.jpg";
File coverJpg;
if (!FsHelpers::openFileForWrite("EBP", coverJpgTempPath, coverJpg)) {
return false;
}
readItemContentsToStream(coverImageHref, coverJpg, 1024);
coverJpg.close();
coverJpg = SD.open((getCachePath() + "/.cover.jpg").c_str(), FILE_READ);
File coverBmp = SD.open(getCoverBmpPath().c_str(), FILE_WRITE, true);
if (!FsHelpers::openFileForRead("EBP", coverJpgTempPath, coverJpg)) {
return false;
}
File coverBmp;
if (!FsHelpers::openFileForWrite("EBP", getCoverBmpPath(), coverBmp)) {
coverJpg.close();
return false;
}
const bool success = JpegToBmpConverter::jpegFileToBmpStream(coverJpg, coverBmp);
coverJpg.close();
coverBmp.close();
SD.remove((getCachePath() + "/.cover.jpg").c_str());
SD.remove(coverJpgTempPath.c_str());
if (!success) {
Serial.printf("[%lu] [EBP] Failed to generate BMP from JPG cover image\n", millis());
@@ -259,45 +320,9 @@ bool Epub::generateCoverBmp() const {
return false;
}
std::string normalisePath(const std::string& path) {
std::vector<std::string> components;
std::string component;
for (const auto c : path) {
if (c == '/') {
if (!component.empty()) {
if (component == "..") {
if (!components.empty()) {
components.pop_back();
}
} else {
components.push_back(component);
}
component.clear();
}
} else {
component += c;
}
}
if (!component.empty()) {
components.push_back(component);
}
std::string result;
for (const auto& c : components) {
if (!result.empty()) {
result += "/";
}
result += c;
}
return result;
}
uint8_t* Epub::readItemContentsToBytes(const std::string& itemHref, size_t* size, bool trailingNullByte) const {
uint8_t* Epub::readItemContentsToBytes(const std::string& itemHref, size_t* size, const bool trailingNullByte) const {
const ZipFile zip("/sd" + filepath);
const std::string path = normalisePath(itemHref);
const std::string path = FsHelpers::normalisePath(itemHref);
const auto content = zip.readFileToMemory(path.c_str(), size, trailingNullByte);
if (!content) {
@@ -310,7 +335,7 @@ uint8_t* Epub::readItemContentsToBytes(const std::string& itemHref, size_t* size
bool Epub::readItemContentsToStream(const std::string& itemHref, Print& out, const size_t chunkSize) const {
const ZipFile zip("/sd" + filepath);
const std::string path = normalisePath(itemHref);
const std::string path = FsHelpers::normalisePath(itemHref);
return zip.readFileToStream(path.c_str(), out, chunkSize);
}
@@ -321,103 +346,93 @@ bool Epub::getItemSize(const std::string& itemHref, size_t* size) const {
}
bool Epub::getItemSize(const ZipFile& zip, const std::string& itemHref, size_t* size) {
const std::string path = normalisePath(itemHref);
const std::string path = FsHelpers::normalisePath(itemHref);
return zip.getInflatedFileSize(path.c_str(), size);
}
int Epub::getSpineItemsCount() const { return spine.size(); }
size_t Epub::getCumulativeSpineItemSize(const int spineIndex) const {
if (spineIndex < 0 || spineIndex >= static_cast<int>(cumulativeSpineItemSize.size())) {
Serial.printf("[%lu] [EBP] getCumulativeSpineItemSize index:%d is out of range\n", millis(), spineIndex);
int Epub::getSpineItemsCount() const {
if (!bookMetadataCache || !bookMetadataCache->isLoaded()) {
return 0;
}
return cumulativeSpineItemSize.at(spineIndex);
return bookMetadataCache->getSpineCount();
}
std::string& Epub::getSpineItem(const int spineIndex) {
static std::string emptyString;
if (spine.empty()) {
Serial.printf("[%lu] [EBP] getSpineItem called but spine is empty\n", millis());
return emptyString;
size_t Epub::getCumulativeSpineItemSize(const int spineIndex) const { return getSpineItem(spineIndex).cumulativeSize; }
BookMetadataCache::SpineEntry Epub::getSpineItem(const int spineIndex) const {
if (!bookMetadataCache || !bookMetadataCache->isLoaded()) {
Serial.printf("[%lu] [EBP] getSpineItem called but cache not loaded\n", millis());
return {};
}
if (spineIndex < 0 || spineIndex >= static_cast<int>(spine.size())) {
if (spineIndex < 0 || spineIndex >= bookMetadataCache->getSpineCount()) {
Serial.printf("[%lu] [EBP] getSpineItem index:%d is out of range\n", millis(), spineIndex);
return spine.at(0).second;
return bookMetadataCache->getSpineEntry(0);
}
return spine.at(spineIndex).second;
return bookMetadataCache->getSpineEntry(spineIndex);
}
EpubTocEntry& Epub::getTocItem(const int tocTndex) {
static EpubTocEntry emptyEntry = {};
if (toc.empty()) {
Serial.printf("[%lu] [EBP] getTocItem called but toc is empty\n", millis());
return emptyEntry;
}
if (tocTndex < 0 || tocTndex >= static_cast<int>(toc.size())) {
Serial.printf("[%lu] [EBP] getTocItem index:%d is out of range\n", millis(), tocTndex);
return toc.at(0);
BookMetadataCache::TocEntry Epub::getTocItem(const int tocIndex) const {
if (!bookMetadataCache || !bookMetadataCache->isLoaded()) {
Serial.printf("[%lu] [EBP] getTocItem called but cache not loaded\n", millis());
return {};
}
return toc.at(tocTndex);
if (tocIndex < 0 || tocIndex >= bookMetadataCache->getTocCount()) {
Serial.printf("[%lu] [EBP] getTocItem index:%d is out of range\n", millis(), tocIndex);
return {};
}
return bookMetadataCache->getTocEntry(tocIndex);
}
int Epub::getTocItemsCount() const { return toc.size(); }
int Epub::getTocItemsCount() const {
if (!bookMetadataCache || !bookMetadataCache->isLoaded()) {
return 0;
}
return bookMetadataCache->getTocCount();
}
// work out the section index for a toc index
int Epub::getSpineIndexForTocIndex(const int tocIndex) const {
if (tocIndex < 0 || tocIndex >= toc.size()) {
if (!bookMetadataCache || !bookMetadataCache->isLoaded()) {
Serial.printf("[%lu] [EBP] getSpineIndexForTocIndex called but cache not loaded\n", millis());
return 0;
}
if (tocIndex < 0 || tocIndex >= bookMetadataCache->getTocCount()) {
Serial.printf("[%lu] [EBP] getSpineIndexForTocIndex: tocIndex %d out of range\n", millis(), tocIndex);
return 0;
}
// the toc entry should have an href that matches the spine item
// so we can find the spine index by looking for the href
for (int i = 0; i < spine.size(); i++) {
if (spine[i].second == toc[tocIndex].href) {
return i;
}
const int spineIndex = bookMetadataCache->getTocEntry(tocIndex).spineIndex;
if (spineIndex < 0) {
Serial.printf("[%lu] [EBP] Section not found for TOC index %d\n", millis(), tocIndex);
return 0;
}
Serial.printf("[%lu] [EBP] Section not found\n", millis());
// not found - default to the start of the book
return 0;
return spineIndex;
}
int Epub::getTocIndexForSpineIndex(const int spineIndex) const {
if (spineIndex < 0 || spineIndex >= spine.size()) {
Serial.printf("[%lu] [EBP] getTocIndexForSpineIndex: spineIndex %d out of range\n", millis(), spineIndex);
return -1;
}
// the toc entry should have an href that matches the spine item
// so we can find the toc index by looking for the href
for (int i = 0; i < toc.size(); i++) {
if (toc[i].href == spine[spineIndex].second) {
return i;
}
}
Serial.printf("[%lu] [EBP] TOC item not found\n", millis());
return -1;
}
int Epub::getTocIndexForSpineIndex(const int spineIndex) const { return getSpineItem(spineIndex).tocIndex; }
size_t Epub::getBookSize() const {
if (spine.empty()) {
if (!bookMetadataCache || !bookMetadataCache->isLoaded() || bookMetadataCache->getSpineCount() == 0) {
return 0;
}
return getCumulativeSpineItemSize(getSpineItemsCount() - 1);
}
// Calculate progress in book
uint8_t Epub::calculateProgress(const int currentSpineIndex, const float currentSpineRead) {
size_t bookSize = getBookSize();
uint8_t Epub::calculateProgress(const int currentSpineIndex, const float currentSpineRead) const {
const size_t bookSize = getBookSize();
if (bookSize == 0) {
return 0;
}
size_t prevChapterSize = (currentSpineIndex >= 1) ? getCumulativeSpineItemSize(currentSpineIndex - 1) : 0;
size_t curChapterSize = getCumulativeSpineItemSize(currentSpineIndex) - prevChapterSize;
size_t sectionProgSize = currentSpineRead * curChapterSize;
const size_t prevChapterSize = (currentSpineIndex >= 1) ? getCumulativeSpineItemSize(currentSpineIndex - 1) : 0;
const size_t curChapterSize = getCumulativeSpineItemSize(currentSpineIndex) - prevChapterSize;
const size_t sectionProgSize = currentSpineRead * curChapterSize;
return round(static_cast<float>(prevChapterSize + sectionProgSize) / bookSize * 100.0);
}
+10 -19
View File
@@ -1,38 +1,29 @@
#pragma once
#include <Print.h>
#include <memory>
#include <string>
#include <unordered_map>
#include <vector>
#include "Epub/EpubTocEntry.h"
#include "Epub/BookMetadataCache.h"
class ZipFile;
class Epub {
// the title read from the EPUB meta data
std::string title;
// the cover image
std::string coverImageItem;
// the ncx file
std::string tocNcxItem;
// where is the EPUBfile?
std::string filepath;
// the spine of the EPUB file
std::vector<std::pair<std::string, std::string>> spine;
// the file size of the spine items (proxy to book progress)
std::vector<size_t> cumulativeSpineItemSize;
// the toc of the EPUB file
std::vector<EpubTocEntry> toc;
// the base path for items in the EPUB file
std::string contentBasePath;
// Uniq cache key based on filepath
std::string cachePath;
// Spine and TOC cache
std::unique_ptr<BookMetadataCache> bookMetadataCache;
bool findContentOpfFile(std::string* contentOpfFile) const;
bool parseContentOpf(const std::string& contentOpfFilePath);
bool parseTocNcxFile();
void initializeSpineItemSizes();
bool parseContentOpf(BookMetadataCache::BookMetadata& bookMetadata);
bool parseTocNcxFile() const;
static bool getItemSize(const ZipFile& zip, const std::string& itemHref, size_t* size);
public:
@@ -54,14 +45,14 @@ class Epub {
bool trailingNullByte = false) const;
bool readItemContentsToStream(const std::string& itemHref, Print& out, size_t chunkSize) const;
bool getItemSize(const std::string& itemHref, size_t* size) const;
std::string& getSpineItem(int spineIndex);
BookMetadataCache::SpineEntry getSpineItem(int spineIndex) const;
BookMetadataCache::TocEntry getTocItem(int tocIndex) const;
int getSpineItemsCount() const;
size_t getCumulativeSpineItemSize(const int spineIndex) const;
EpubTocEntry& getTocItem(int tocIndex);
int getTocItemsCount() const;
int getSpineIndexForTocIndex(int tocIndex) const;
int getTocIndexForSpineIndex(int spineIndex) const;
size_t getCumulativeSpineItemSize(int spineIndex) const;
size_t getBookSize() const;
uint8_t calculateProgress(const int currentSpineIndex, const float currentSpineRead);
uint8_t calculateProgress(const int currentSpineIndex, const float currentSpineRead) const;
};
+326
View File
@@ -0,0 +1,326 @@
#include "BookMetadataCache.h"
#include <HardwareSerial.h>
#include <SD.h>
#include <Serialization.h>
#include <ZipFile.h>
#include <vector>
#include "FsHelpers.h"
namespace {
constexpr uint8_t BOOK_CACHE_VERSION = 1;
constexpr char bookBinFile[] = "/book.bin";
constexpr char tmpSpineBinFile[] = "/spine.bin.tmp";
constexpr char tmpTocBinFile[] = "/toc.bin.tmp";
} // namespace
/* ============= WRITING / BUILDING FUNCTIONS ================ */
bool BookMetadataCache::beginWrite() {
buildMode = true;
spineCount = 0;
tocCount = 0;
Serial.printf("[%lu] [BMC] Entering write mode\n", millis());
return true;
}
bool BookMetadataCache::beginContentOpfPass() {
Serial.printf("[%lu] [BMC] Beginning content opf pass\n", millis());
// Open spine file for writing
return FsHelpers::openFileForWrite("BMC", cachePath + tmpSpineBinFile, spineFile);
}
bool BookMetadataCache::endContentOpfPass() {
spineFile.close();
return true;
}
bool BookMetadataCache::beginTocPass() {
Serial.printf("[%lu] [BMC] Beginning toc pass\n", millis());
// Open spine file for reading
if (!FsHelpers::openFileForRead("BMC", cachePath + tmpSpineBinFile, spineFile)) {
return false;
}
if (!FsHelpers::openFileForWrite("BMC", cachePath + tmpTocBinFile, tocFile)) {
spineFile.close();
return false;
}
return true;
}
bool BookMetadataCache::endTocPass() {
tocFile.close();
spineFile.close();
return true;
}
bool BookMetadataCache::endWrite() {
if (!buildMode) {
Serial.printf("[%lu] [BMC] endWrite called but not in build mode\n", millis());
return false;
}
buildMode = false;
Serial.printf("[%lu] [BMC] Wrote %d spine, %d TOC entries\n", millis(), spineCount, tocCount);
return true;
}
bool BookMetadataCache::buildBookBin(const std::string& epubPath, const BookMetadata& metadata) {
// Open all three files, writing to meta, reading from spine and toc
if (!FsHelpers::openFileForWrite("BMC", cachePath + bookBinFile, bookFile)) {
return false;
}
if (!FsHelpers::openFileForRead("BMC", cachePath + tmpSpineBinFile, spineFile)) {
bookFile.close();
return false;
}
if (!FsHelpers::openFileForRead("BMC", cachePath + tmpTocBinFile, tocFile)) {
bookFile.close();
spineFile.close();
return false;
}
constexpr size_t headerASize =
sizeof(BOOK_CACHE_VERSION) + /* LUT Offset */ sizeof(size_t) + sizeof(spineCount) + sizeof(tocCount);
const size_t metadataSize =
metadata.title.size() + metadata.author.size() + metadata.coverItemHref.size() + sizeof(uint32_t) * 3;
const size_t lutSize = sizeof(size_t) * spineCount + sizeof(size_t) * tocCount;
const size_t lutOffset = headerASize + metadataSize;
// Header A
serialization::writePod(bookFile, BOOK_CACHE_VERSION);
serialization::writePod(bookFile, lutOffset);
serialization::writePod(bookFile, spineCount);
serialization::writePod(bookFile, tocCount);
// Metadata
serialization::writeString(bookFile, metadata.title);
serialization::writeString(bookFile, metadata.author);
serialization::writeString(bookFile, metadata.coverItemHref);
// Loop through spine entries, writing LUT positions
spineFile.seek(0);
for (int i = 0; i < spineCount; i++) {
auto pos = spineFile.position();
auto spineEntry = readSpineEntry(spineFile);
serialization::writePod(bookFile, pos + lutOffset + lutSize);
}
// Loop through toc entries, writing LUT positions
tocFile.seek(0);
for (int i = 0; i < tocCount; i++) {
auto pos = tocFile.position();
auto tocEntry = readTocEntry(tocFile);
serialization::writePod(bookFile, pos + lutOffset + lutSize + spineFile.position());
}
// LUTs complete
// Loop through spines from spine file matching up TOC indexes, calculating cumulative size and writing to book.bin
const ZipFile zip("/sd" + epubPath);
size_t cumSize = 0;
spineFile.seek(0);
for (int i = 0; i < spineCount; i++) {
auto spineEntry = readSpineEntry(spineFile);
tocFile.seek(0);
for (int j = 0; j < tocCount; j++) {
auto tocEntry = readTocEntry(tocFile);
if (tocEntry.spineIndex == i) {
spineEntry.tocIndex = j;
break;
}
}
// Not a huge deal if we don't fine a TOC entry for the spine entry, this is expected behaviour for EPUBs
// Logging here is for debugging
if (spineEntry.tocIndex == -1) {
Serial.printf("[%lu] [BMC] Warning: Could not find TOC entry for spine item %d: %s\n", millis(), i,
spineEntry.href.c_str());
}
// Calculate size for cumulative size
size_t itemSize = 0;
const std::string path = FsHelpers::normalisePath(spineEntry.href);
if (zip.getInflatedFileSize(path.c_str(), &itemSize)) {
cumSize += itemSize;
spineEntry.cumulativeSize = cumSize;
} else {
Serial.printf("[%lu] [BMC] Warning: Could not get size for spine item: %s\n", millis(), path.c_str());
}
// Write out spine data to book.bin
writeSpineEntry(bookFile, spineEntry);
}
// Loop through toc entries from toc file writing to book.bin
tocFile.seek(0);
for (int i = 0; i < tocCount; i++) {
auto tocEntry = readTocEntry(tocFile);
writeTocEntry(bookFile, tocEntry);
}
bookFile.close();
spineFile.close();
tocFile.close();
Serial.printf("[%lu] [BMC] Successfully built book.bin\n", millis());
return true;
}
bool BookMetadataCache::cleanupTmpFiles() const {
if (SD.exists((cachePath + tmpSpineBinFile).c_str())) {
SD.remove((cachePath + tmpSpineBinFile).c_str());
}
if (SD.exists((cachePath + tmpTocBinFile).c_str())) {
SD.remove((cachePath + tmpTocBinFile).c_str());
}
return true;
}
size_t BookMetadataCache::writeSpineEntry(File& file, const SpineEntry& entry) const {
const auto pos = file.position();
serialization::writeString(file, entry.href);
serialization::writePod(file, entry.cumulativeSize);
serialization::writePod(file, entry.tocIndex);
return pos;
}
size_t BookMetadataCache::writeTocEntry(File& file, const TocEntry& entry) const {
const auto pos = file.position();
serialization::writeString(file, entry.title);
serialization::writeString(file, entry.href);
serialization::writeString(file, entry.anchor);
serialization::writePod(file, entry.level);
serialization::writePod(file, entry.spineIndex);
return pos;
}
// Note: for the LUT to be accurate, this **MUST** be called for all spine items before `addTocEntry` is ever called
// this is because in this function we're marking positions of the items
void BookMetadataCache::createSpineEntry(const std::string& href) {
if (!buildMode || !spineFile) {
Serial.printf("[%lu] [BMC] createSpineEntry called but not in build mode\n", millis());
return;
}
const SpineEntry entry(href, 0, -1);
writeSpineEntry(spineFile, entry);
spineCount++;
}
void BookMetadataCache::createTocEntry(const std::string& title, const std::string& href, const std::string& anchor,
const uint8_t level) {
if (!buildMode || !tocFile || !spineFile) {
Serial.printf("[%lu] [BMC] createTocEntry called but not in build mode\n", millis());
return;
}
int spineIndex = -1;
// find spine index
spineFile.seek(0);
for (int i = 0; i < spineCount; i++) {
auto spineEntry = readSpineEntry(spineFile);
if (spineEntry.href == href) {
spineIndex = i;
break;
}
}
if (spineIndex == -1) {
Serial.printf("[%lu] [BMC] addTocEntry: Could not find spine item for TOC href %s\n", millis(), href.c_str());
}
const TocEntry entry(title, href, anchor, level, spineIndex);
writeTocEntry(tocFile, entry);
tocCount++;
}
/* ============= READING / LOADING FUNCTIONS ================ */
bool BookMetadataCache::load() {
if (!FsHelpers::openFileForRead("BMC", cachePath + bookBinFile, bookFile)) {
return false;
}
uint8_t version;
serialization::readPod(bookFile, version);
if (version != BOOK_CACHE_VERSION) {
Serial.printf("[%lu] [BMC] Cache version mismatch: expected %d, got %d\n", millis(), BOOK_CACHE_VERSION, version);
bookFile.close();
return false;
}
serialization::readPod(bookFile, lutOffset);
serialization::readPod(bookFile, spineCount);
serialization::readPod(bookFile, tocCount);
serialization::readString(bookFile, coreMetadata.title);
serialization::readString(bookFile, coreMetadata.author);
serialization::readString(bookFile, coreMetadata.coverItemHref);
loaded = true;
Serial.printf("[%lu] [BMC] Loaded cache data: %d spine, %d TOC entries\n", millis(), spineCount, tocCount);
return true;
}
BookMetadataCache::SpineEntry BookMetadataCache::getSpineEntry(const int index) {
if (!loaded) {
Serial.printf("[%lu] [BMC] getSpineEntry called but cache not loaded\n", millis());
return {};
}
if (index < 0 || index >= static_cast<int>(spineCount)) {
Serial.printf("[%lu] [BMC] getSpineEntry index %d out of range\n", millis(), index);
return {};
}
// Seek to spine LUT item, read from LUT and get out data
bookFile.seek(lutOffset + sizeof(size_t) * index);
size_t spineEntryPos;
serialization::readPod(bookFile, spineEntryPos);
bookFile.seek(spineEntryPos);
return readSpineEntry(bookFile);
}
BookMetadataCache::TocEntry BookMetadataCache::getTocEntry(const int index) {
if (!loaded) {
Serial.printf("[%lu] [BMC] getTocEntry called but cache not loaded\n", millis());
return {};
}
if (index < 0 || index >= static_cast<int>(tocCount)) {
Serial.printf("[%lu] [BMC] getTocEntry index %d out of range\n", millis(), index);
return {};
}
// Seek to TOC LUT item, read from LUT and get out data
bookFile.seek(lutOffset + sizeof(size_t) * spineCount + sizeof(size_t) * index);
size_t tocEntryPos;
serialization::readPod(bookFile, tocEntryPos);
bookFile.seek(tocEntryPos);
return readTocEntry(bookFile);
}
BookMetadataCache::SpineEntry BookMetadataCache::readSpineEntry(File& file) const {
SpineEntry entry;
serialization::readString(file, entry.href);
serialization::readPod(file, entry.cumulativeSize);
serialization::readPod(file, entry.tocIndex);
return entry;
}
BookMetadataCache::TocEntry BookMetadataCache::readTocEntry(File& file) const {
TocEntry entry;
serialization::readString(file, entry.title);
serialization::readString(file, entry.href);
serialization::readString(file, entry.anchor);
serialization::readPod(file, entry.level);
serialization::readPod(file, entry.spineIndex);
return entry;
}
+87
View File
@@ -0,0 +1,87 @@
#pragma once
#include <SD.h>
#include <string>
class BookMetadataCache {
public:
struct BookMetadata {
std::string title;
std::string author;
std::string coverItemHref;
};
struct SpineEntry {
std::string href;
size_t cumulativeSize;
int16_t tocIndex;
SpineEntry() : cumulativeSize(0), tocIndex(-1) {}
SpineEntry(std::string href, const size_t cumulativeSize, const int16_t tocIndex)
: href(std::move(href)), cumulativeSize(cumulativeSize), tocIndex(tocIndex) {}
};
struct TocEntry {
std::string title;
std::string href;
std::string anchor;
uint8_t level;
int16_t spineIndex;
TocEntry() : level(0), spineIndex(-1) {}
TocEntry(std::string title, std::string href, std::string anchor, const uint8_t level, const int16_t spineIndex)
: title(std::move(title)),
href(std::move(href)),
anchor(std::move(anchor)),
level(level),
spineIndex(spineIndex) {}
};
private:
std::string cachePath;
size_t lutOffset;
uint16_t spineCount;
uint16_t tocCount;
bool loaded;
bool buildMode;
File bookFile;
// Temp file handles during build
File spineFile;
File tocFile;
size_t writeSpineEntry(File& file, const SpineEntry& entry) const;
size_t writeTocEntry(File& file, const TocEntry& entry) const;
SpineEntry readSpineEntry(File& file) const;
TocEntry readTocEntry(File& file) const;
public:
BookMetadata coreMetadata;
explicit BookMetadataCache(std::string cachePath)
: cachePath(std::move(cachePath)), lutOffset(0), spineCount(0), tocCount(0), loaded(false), buildMode(false) {}
~BookMetadataCache() = default;
// Building phase (stream to disk immediately)
bool beginWrite();
bool beginContentOpfPass();
void createSpineEntry(const std::string& href);
bool endContentOpfPass();
bool beginTocPass();
void createTocEntry(const std::string& title, const std::string& href, const std::string& anchor, uint8_t level);
bool endTocPass();
bool endWrite();
bool cleanupTmpFiles() const;
// Post-processing to update mappings and sizes
bool buildBookBin(const std::string& epubPath, const BookMetadata& metadata);
// Reading phase (read mode)
bool load();
SpineEntry getSpineEntry(int index);
TocEntry getTocEntry(int index);
int getSpineCount() const { return spineCount; }
int getTocCount() const { return tocCount; }
bool isLoaded() const { return loaded; }
};
-10
View File
@@ -1,10 +0,0 @@
#pragma once
#include <string>
struct EpubTocEntry {
std::string title;
std::string href;
std::string anchor;
uint8_t level;
};
+56
View File
@@ -2,6 +2,26 @@
#include <SD.h>
#include <vector>
bool FsHelpers::openFileForRead(const char* moduleName, const std::string& path, File& file) {
file = SD.open(path.c_str(), FILE_READ);
if (!file) {
Serial.printf("[%lu] [%s] Failed to open file for reading: %s\n", millis(), moduleName, path.c_str());
return false;
}
return true;
}
bool FsHelpers::openFileForWrite(const char* moduleName, const std::string& path, File& file) {
file = SD.open(path.c_str(), FILE_WRITE, true);
if (!file) {
Serial.printf("[%lu] [%s] Failed to open file for writing: %s\n", millis(), moduleName, path.c_str());
return false;
}
return true;
}
bool FsHelpers::removeDir(const char* path) {
// 1. Open the directory
File dir = SD.open(path);
@@ -34,3 +54,39 @@ bool FsHelpers::removeDir(const char* path) {
return SD.rmdir(path);
}
std::string FsHelpers::normalisePath(const std::string& path) {
std::vector<std::string> components;
std::string component;
for (const auto c : path) {
if (c == '/') {
if (!component.empty()) {
if (component == "..") {
if (!components.empty()) {
components.pop_back();
}
} else {
components.push_back(component);
}
component.clear();
}
} else {
component += c;
}
}
if (!component.empty()) {
components.push_back(component);
}
std::string result;
for (const auto& c : components) {
if (!result.empty()) {
result += "/";
}
result += c;
}
return result;
}
+6
View File
@@ -1,6 +1,12 @@
#pragma once
#include <FS.h>
#include <string>
class FsHelpers {
public:
static bool openFileForRead(const char* moduleName, const std::string& path, File& file);
static bool openFileForWrite(const char* moduleName, const std::string& path, File& file);
static bool removeDir(const char* path);
static std::string normalisePath(const std::string& path);
};
+26 -24
View File
@@ -7,48 +7,50 @@ namespace {
constexpr uint8_t PAGE_FILE_VERSION = 3;
}
void PageLine::render(GfxRenderer& renderer, const int fontId) { block->render(renderer, fontId, xPos, yPos); }
void PageLine::serialize(std::ostream& os) {
serialization::writePod(os, xPos);
serialization::writePod(os, yPos);
// serialize TextBlock pointed to by PageLine
block->serialize(os);
void PageLine::render(GfxRenderer& renderer, const int fontId, const int xOffset, const int yOffset) {
block->render(renderer, fontId, xPos + xOffset, yPos + yOffset);
}
std::unique_ptr<PageLine> PageLine::deserialize(std::istream& is) {
void PageLine::serialize(File& file) {
serialization::writePod(file, xPos);
serialization::writePod(file, yPos);
// serialize TextBlock pointed to by PageLine
block->serialize(file);
}
std::unique_ptr<PageLine> PageLine::deserialize(File& file) {
int16_t xPos;
int16_t yPos;
serialization::readPod(is, xPos);
serialization::readPod(is, yPos);
serialization::readPod(file, xPos);
serialization::readPod(file, yPos);
auto tb = TextBlock::deserialize(is);
auto tb = TextBlock::deserialize(file);
return std::unique_ptr<PageLine>(new PageLine(std::move(tb), xPos, yPos));
}
void Page::render(GfxRenderer& renderer, const int fontId) const {
void Page::render(GfxRenderer& renderer, const int fontId, const int xOffset, const int yOffset) const {
for (auto& element : elements) {
element->render(renderer, fontId);
element->render(renderer, fontId, xOffset, yOffset);
}
}
void Page::serialize(std::ostream& os) const {
serialization::writePod(os, PAGE_FILE_VERSION);
void Page::serialize(File& file) const {
serialization::writePod(file, PAGE_FILE_VERSION);
const uint32_t count = elements.size();
serialization::writePod(os, count);
serialization::writePod(file, count);
for (const auto& el : elements) {
// Only PageLine exists currently
serialization::writePod(os, static_cast<uint8_t>(TAG_PageLine));
el->serialize(os);
serialization::writePod(file, static_cast<uint8_t>(TAG_PageLine));
el->serialize(file);
}
}
std::unique_ptr<Page> Page::deserialize(std::istream& is) {
std::unique_ptr<Page> Page::deserialize(File& file) {
uint8_t version;
serialization::readPod(is, version);
serialization::readPod(file, version);
if (version != PAGE_FILE_VERSION) {
Serial.printf("[%lu] [PGE] Deserialization failed: Unknown version %u\n", millis(), version);
return nullptr;
@@ -57,14 +59,14 @@ std::unique_ptr<Page> Page::deserialize(std::istream& is) {
auto page = std::unique_ptr<Page>(new Page());
uint32_t count;
serialization::readPod(is, count);
serialization::readPod(file, count);
for (uint32_t i = 0; i < count; i++) {
uint8_t tag;
serialization::readPod(is, tag);
serialization::readPod(file, tag);
if (tag == TAG_PageLine) {
auto pl = PageLine::deserialize(is);
auto pl = PageLine::deserialize(file);
page->elements.push_back(std::move(pl));
} else {
Serial.printf("[%lu] [PGE] Deserialization failed: Unknown tag %u\n", millis(), tag);
+10 -8
View File
@@ -1,4 +1,6 @@
#pragma once
#include <FS.h>
#include <utility>
#include <vector>
@@ -15,8 +17,8 @@ class PageElement {
int16_t yPos;
explicit PageElement(const int16_t xPos, const int16_t yPos) : xPos(xPos), yPos(yPos) {}
virtual ~PageElement() = default;
virtual void render(GfxRenderer& renderer, int fontId) = 0;
virtual void serialize(std::ostream& os) = 0;
virtual void render(GfxRenderer& renderer, int fontId, int xOffset, int yOffset) = 0;
virtual void serialize(File& file) = 0;
};
// a line from a block element
@@ -26,16 +28,16 @@ class PageLine final : public PageElement {
public:
PageLine(std::shared_ptr<TextBlock> block, const int16_t xPos, const int16_t yPos)
: PageElement(xPos, yPos), block(std::move(block)) {}
void render(GfxRenderer& renderer, int fontId) override;
void serialize(std::ostream& os) override;
static std::unique_ptr<PageLine> deserialize(std::istream& is);
void render(GfxRenderer& renderer, int fontId, int xOffset, int yOffset) override;
void serialize(File& file) override;
static std::unique_ptr<PageLine> deserialize(File& file);
};
class Page {
public:
// the list of block index and line numbers on this page
std::vector<std::shared_ptr<PageElement>> elements;
void render(GfxRenderer& renderer, int fontId) const;
void serialize(std::ostream& os) const;
static std::unique_ptr<Page> deserialize(std::istream& is);
void render(GfxRenderer& renderer, int fontId, int xOffset, int yOffset) const;
void serialize(File& file) const;
static std::unique_ptr<Page> deserialize(File& file);
};
+20 -7
View File
@@ -18,14 +18,14 @@ void ParsedText::addWord(std::string word, const EpdFontStyle fontStyle) {
}
// Consumes data to minimize memory usage
void ParsedText::layoutAndExtractLines(const GfxRenderer& renderer, const int fontId, const int horizontalMargin,
void ParsedText::layoutAndExtractLines(const GfxRenderer& renderer, const int fontId, const int viewportWidth,
const std::function<void(std::shared_ptr<TextBlock>)>& processLine,
const bool includeLastLine) {
if (words.empty()) {
return;
}
const int pageWidth = renderer.getScreenWidth() - horizontalMargin;
const int pageWidth = viewportWidth;
const int spaceWidth = renderer.getSpaceWidth(fontId);
const auto wordWidths = calculateWordWidths(renderer, fontId);
const auto lineBreakIndices = computeLineBreaks(pageWidth, spaceWidth, wordWidths);
@@ -106,21 +106,34 @@ std::vector<size_t> ParsedText::computeLineBreaks(const int pageWidth, const int
ans[i] = j; // j is the index of the last word in this optimal line
}
}
// Handle oversized word: if no valid configuration found, force single-word line
// This prevents cascade failure where one oversized word breaks all preceding words
if (dp[i] == MAX_COST) {
ans[i] = i; // Just this word on its own line
// Inherit cost from next word to allow subsequent words to find valid configurations
if (i + 1 < static_cast<int>(totalWordCount)) {
dp[i] = dp[i + 1];
} else {
dp[i] = 0;
}
}
}
// Stores the index of the word that starts the next line (last_word_index + 1)
std::vector<size_t> lineBreakIndices;
size_t currentWordIndex = 0;
constexpr size_t MAX_LINES = 1000;
while (currentWordIndex < totalWordCount) {
if (lineBreakIndices.size() >= MAX_LINES) {
break;
size_t nextBreakIndex = ans[currentWordIndex] + 1;
// Safety check: prevent infinite loop if nextBreakIndex doesn't advance
if (nextBreakIndex <= currentWordIndex) {
// Force advance by at least one word to avoid infinite loop
nextBreakIndex = currentWordIndex + 1;
}
size_t nextBreakIndex = ans[currentWordIndex] + 1;
lineBreakIndices.push_back(nextBreakIndex);
currentWordIndex = nextBreakIndex;
}
+1 -1
View File
@@ -34,7 +34,7 @@ class ParsedText {
TextBlock::BLOCK_STYLE getStyle() const { return style; }
size_t size() const { return words.size(); }
bool isEmpty() const { return words.empty(); }
void layoutAndExtractLines(const GfxRenderer& renderer, int fontId, int horizontalMargin,
void layoutAndExtractLines(const GfxRenderer& renderer, int fontId, int viewportWidth,
const std::function<void(std::shared_ptr<TextBlock>)>& processLine,
bool includeLastLine = true);
};
+74 -55
View File
@@ -1,22 +1,23 @@
#include "Section.h"
#include <FsHelpers.h>
#include <SD.h>
#include <Serialization.h>
#include <fstream>
#include "FsHelpers.h"
#include "Page.h"
#include "parsers/ChapterHtmlSlimParser.h"
namespace {
constexpr uint8_t SECTION_FILE_VERSION = 5;
}
constexpr uint8_t SECTION_FILE_VERSION = 6;
} // namespace
void Section::onPageComplete(std::unique_ptr<Page> page) {
const auto filePath = cachePath + "/page_" + std::to_string(pageCount) + ".bin";
std::ofstream outputFile("/sd" + filePath);
File outputFile;
if (!FsHelpers::openFileForWrite("SCT", filePath, outputFile)) {
return;
}
page->serialize(outputFile);
outputFile.close();
@@ -25,36 +26,30 @@ void Section::onPageComplete(std::unique_ptr<Page> page) {
pageCount++;
}
void Section::writeCacheMetadata(const int fontId, const float lineCompression, const int marginTop,
const int marginRight, const int marginBottom, const int marginLeft,
const bool extraParagraphSpacing) const {
std::ofstream outputFile(("/sd" + cachePath + "/section.bin").c_str());
void Section::writeCacheMetadata(const int fontId, const float lineCompression, const bool extraParagraphSpacing,
const int viewportWidth, const int viewportHeight) const {
File outputFile;
if (!FsHelpers::openFileForWrite("SCT", cachePath + "/section.bin", outputFile)) {
return;
}
serialization::writePod(outputFile, SECTION_FILE_VERSION);
serialization::writePod(outputFile, fontId);
serialization::writePod(outputFile, lineCompression);
serialization::writePod(outputFile, marginTop);
serialization::writePod(outputFile, marginRight);
serialization::writePod(outputFile, marginBottom);
serialization::writePod(outputFile, marginLeft);
serialization::writePod(outputFile, extraParagraphSpacing);
serialization::writePod(outputFile, viewportWidth);
serialization::writePod(outputFile, viewportHeight);
serialization::writePod(outputFile, pageCount);
outputFile.close();
}
bool Section::loadCacheMetadata(const int fontId, const float lineCompression, const int marginTop,
const int marginRight, const int marginBottom, const int marginLeft,
const bool extraParagraphSpacing) {
if (!SD.exists(cachePath.c_str())) {
return false;
}
bool Section::loadCacheMetadata(const int fontId, const float lineCompression, const bool extraParagraphSpacing,
const int viewportWidth, const int viewportHeight) {
const auto sectionFilePath = cachePath + "/section.bin";
if (!SD.exists(sectionFilePath.c_str())) {
File inputFile;
if (!FsHelpers::openFileForRead("SCT", sectionFilePath, inputFile)) {
return false;
}
std::ifstream inputFile(("/sd" + sectionFilePath).c_str());
// Match parameters
{
uint8_t version;
@@ -66,20 +61,18 @@ bool Section::loadCacheMetadata(const int fontId, const float lineCompression, c
return false;
}
int fileFontId, fileMarginTop, fileMarginRight, fileMarginBottom, fileMarginLeft;
int fileFontId, fileViewportWidth, fileViewportHeight;
float fileLineCompression;
bool fileExtraParagraphSpacing;
serialization::readPod(inputFile, fileFontId);
serialization::readPod(inputFile, fileLineCompression);
serialization::readPod(inputFile, fileMarginTop);
serialization::readPod(inputFile, fileMarginRight);
serialization::readPod(inputFile, fileMarginBottom);
serialization::readPod(inputFile, fileMarginLeft);
serialization::readPod(inputFile, fileExtraParagraphSpacing);
serialization::readPod(inputFile, fileViewportWidth);
serialization::readPod(inputFile, fileViewportHeight);
if (fontId != fileFontId || lineCompression != fileLineCompression || marginTop != fileMarginTop ||
marginRight != fileMarginRight || marginBottom != fileMarginBottom || marginLeft != fileMarginLeft ||
extraParagraphSpacing != fileExtraParagraphSpacing) {
if (fontId != fileFontId || lineCompression != fileLineCompression ||
extraParagraphSpacing != fileExtraParagraphSpacing || viewportWidth != fileViewportWidth ||
viewportHeight != fileViewportHeight) {
inputFile.close();
Serial.printf("[%lu] [SCT] Deserialization failed: Parameters do not match\n", millis());
clearCache();
@@ -114,31 +107,58 @@ bool Section::clearCache() const {
return true;
}
bool Section::persistPageDataToSD(const int fontId, const float lineCompression, const int marginTop,
const int marginRight, const int marginBottom, const int marginLeft,
const bool extraParagraphSpacing) {
const auto localPath = epub->getSpineItem(spineIndex);
// TODO: Should we get rid of this file all together?
// It currently saves us a bit of memory by allowing for all the inflation bits to be released
// before loading the XML parser
bool Section::persistPageDataToSD(const int fontId, const float lineCompression, const bool extraParagraphSpacing,
const int viewportWidth, const int viewportHeight,
const std::function<void()>& progressSetupFn,
const std::function<void(int)>& progressFn) {
constexpr size_t MIN_SIZE_FOR_PROGRESS = 50 * 1024; // 50KB
const auto localPath = epub->getSpineItem(spineIndex).href;
const auto tmpHtmlPath = epub->getCachePath() + "/.tmp_" + std::to_string(spineIndex) + ".html";
File f = SD.open(tmpHtmlPath.c_str(), FILE_WRITE, true);
bool success = epub->readItemContentsToStream(localPath, f, 1024);
f.close();
// Retry logic for SD card timing issues
bool success = false;
size_t fileSize = 0;
for (int attempt = 0; attempt < 3 && !success; attempt++) {
if (attempt > 0) {
Serial.printf("[%lu] [SCT] Retrying stream (attempt %d)...\n", millis(), attempt + 1);
delay(50); // Brief delay before retry
}
// Remove any incomplete file from previous attempt before retrying
if (SD.exists(tmpHtmlPath.c_str())) {
SD.remove(tmpHtmlPath.c_str());
}
File tmpHtml;
if (!FsHelpers::openFileForWrite("SCT", tmpHtmlPath, tmpHtml)) {
continue;
}
success = epub->readItemContentsToStream(localPath, tmpHtml, 1024);
fileSize = tmpHtml.size();
tmpHtml.close();
// If streaming failed, remove the incomplete file immediately
if (!success && SD.exists(tmpHtmlPath.c_str())) {
SD.remove(tmpHtmlPath.c_str());
Serial.printf("[%lu] [SCT] Removed incomplete temp file after failed attempt\n", millis());
}
}
if (!success) {
Serial.printf("[%lu] [SCT] Failed to stream item contents to temp file\n", millis());
Serial.printf("[%lu] [SCT] Failed to stream item contents to temp file after retries\n", millis());
return false;
}
Serial.printf("[%lu] [SCT] Streamed temp HTML to %s\n", millis(), tmpHtmlPath.c_str());
Serial.printf("[%lu] [SCT] Streamed temp HTML to %s (%d bytes)\n", millis(), tmpHtmlPath.c_str(), fileSize);
const auto sdTmpHtmlPath = "/sd" + tmpHtmlPath;
// Only show progress bar for larger chapters where rendering overhead is worth it
if (progressSetupFn && fileSize >= MIN_SIZE_FOR_PROGRESS) {
progressSetupFn();
}
ChapterHtmlSlimParser visitor(sdTmpHtmlPath.c_str(), renderer, fontId, lineCompression, marginTop, marginRight,
marginBottom, marginLeft, extraParagraphSpacing,
[this](std::unique_ptr<Page> page) { this->onPageComplete(std::move(page)); });
ChapterHtmlSlimParser visitor(
tmpHtmlPath, renderer, fontId, lineCompression, extraParagraphSpacing, viewportWidth, viewportHeight,
[this](std::unique_ptr<Page> page) { this->onPageComplete(std::move(page)); }, progressFn);
success = visitor.parseAndBuildPages();
SD.remove(tmpHtmlPath.c_str());
@@ -147,19 +167,18 @@ bool Section::persistPageDataToSD(const int fontId, const float lineCompression,
return false;
}
writeCacheMetadata(fontId, lineCompression, marginTop, marginRight, marginBottom, marginLeft, extraParagraphSpacing);
writeCacheMetadata(fontId, lineCompression, extraParagraphSpacing, viewportWidth, viewportHeight);
return true;
}
std::unique_ptr<Page> Section::loadPageFromSD() const {
const auto filePath = "/sd" + cachePath + "/page_" + std::to_string(currentPage) + ".bin";
if (!SD.exists(filePath.c_str() + 3)) {
Serial.printf("[%lu] [SCT] Page file does not exist: %s\n", millis(), filePath.c_str());
const auto filePath = cachePath + "/page_" + std::to_string(currentPage) + ".bin";
File inputFile;
if (!FsHelpers::openFileForRead("SCT", filePath, inputFile)) {
return nullptr;
}
std::ifstream inputFile(filePath);
auto page = Page::deserialize(inputFile);
inputFile.close();
return page;
+8 -6
View File
@@ -1,4 +1,5 @@
#pragma once
#include <functional>
#include <memory>
#include "Epub.h"
@@ -12,8 +13,8 @@ class Section {
GfxRenderer& renderer;
std::string cachePath;
void writeCacheMetadata(int fontId, float lineCompression, int marginTop, int marginRight, int marginBottom,
int marginLeft, bool extraParagraphSpacing) const;
void writeCacheMetadata(int fontId, float lineCompression, bool extraParagraphSpacing, int viewportWidth,
int viewportHeight) const;
void onPageComplete(std::unique_ptr<Page> page);
public:
@@ -26,11 +27,12 @@ class Section {
renderer(renderer),
cachePath(epub->getCachePath() + "/" + std::to_string(spineIndex)) {}
~Section() = default;
bool loadCacheMetadata(int fontId, float lineCompression, int marginTop, int marginRight, int marginBottom,
int marginLeft, bool extraParagraphSpacing);
bool loadCacheMetadata(int fontId, float lineCompression, bool extraParagraphSpacing, int viewportWidth,
int viewportHeight);
void setupCacheDir() const;
bool clearCache() const;
bool persistPageDataToSD(int fontId, float lineCompression, int marginTop, int marginRight, int marginBottom,
int marginLeft, bool extraParagraphSpacing);
bool persistPageDataToSD(int fontId, float lineCompression, bool extraParagraphSpacing, int viewportWidth,
int viewportHeight, const std::function<void()>& progressSetupFn = nullptr,
const std::function<void(int)>& progressFn = nullptr);
std::unique_ptr<Page> loadPageFromSD() const;
};
+38 -17
View File
@@ -4,11 +4,18 @@
#include <Serialization.h>
void TextBlock::render(const GfxRenderer& renderer, const int fontId, const int x, const int y) const {
// Validate iterator bounds before rendering
if (words.size() != wordXpos.size() || words.size() != wordStyles.size()) {
Serial.printf("[%lu] [TXB] Render skipped: size mismatch (words=%u, xpos=%u, styles=%u)\n", millis(),
(uint32_t)words.size(), (uint32_t)wordXpos.size(), (uint32_t)wordStyles.size());
return;
}
auto wordIt = words.begin();
auto wordStylesIt = wordStyles.begin();
auto wordXposIt = wordXpos.begin();
for (int i = 0; i < words.size(); i++) {
for (size_t i = 0; i < words.size(); i++) {
renderer.drawText(fontId, *wordXposIt + x, y, wordIt->c_str(), true, *wordStylesIt);
std::advance(wordIt, 1);
@@ -17,27 +24,27 @@ void TextBlock::render(const GfxRenderer& renderer, const int fontId, const int
}
}
void TextBlock::serialize(std::ostream& os) const {
void TextBlock::serialize(File& file) const {
// words
const uint32_t wc = words.size();
serialization::writePod(os, wc);
for (const auto& w : words) serialization::writeString(os, w);
serialization::writePod(file, wc);
for (const auto& w : words) serialization::writeString(file, w);
// wordXpos
const uint32_t xc = wordXpos.size();
serialization::writePod(os, xc);
for (auto x : wordXpos) serialization::writePod(os, x);
serialization::writePod(file, xc);
for (auto x : wordXpos) serialization::writePod(file, x);
// wordStyles
const uint32_t sc = wordStyles.size();
serialization::writePod(os, sc);
for (auto s : wordStyles) serialization::writePod(os, s);
serialization::writePod(file, sc);
for (auto s : wordStyles) serialization::writePod(file, s);
// style
serialization::writePod(os, style);
serialization::writePod(file, style);
}
std::unique_ptr<TextBlock> TextBlock::deserialize(std::istream& is) {
std::unique_ptr<TextBlock> TextBlock::deserialize(File& file) {
uint32_t wc, xc, sc;
std::list<std::string> words;
std::list<uint16_t> wordXpos;
@@ -45,22 +52,36 @@ std::unique_ptr<TextBlock> TextBlock::deserialize(std::istream& is) {
BLOCK_STYLE style;
// words
serialization::readPod(is, wc);
serialization::readPod(file, wc);
// Sanity check: prevent allocation of unreasonably large lists (max 10000 words per block)
if (wc > 10000) {
Serial.printf("[%lu] [TXB] Deserialization failed: word count %u exceeds maximum\n", millis(), wc);
return nullptr;
}
words.resize(wc);
for (auto& w : words) serialization::readString(is, w);
for (auto& w : words) serialization::readString(file, w);
// wordXpos
serialization::readPod(is, xc);
serialization::readPod(file, xc);
wordXpos.resize(xc);
for (auto& x : wordXpos) serialization::readPod(is, x);
for (auto& x : wordXpos) serialization::readPod(file, x);
// wordStyles
serialization::readPod(is, sc);
serialization::readPod(file, sc);
wordStyles.resize(sc);
for (auto& s : wordStyles) serialization::readPod(is, s);
for (auto& s : wordStyles) serialization::readPod(file, s);
// Validate data consistency: all three lists must have the same size
if (wc != xc || wc != sc) {
Serial.printf("[%lu] [TXB] Deserialization failed: size mismatch (words=%u, xpos=%u, styles=%u)\n", millis(), wc,
xc, sc);
return nullptr;
}
// style
serialization::readPod(is, style);
serialization::readPod(file, style);
return std::unique_ptr<TextBlock>(new TextBlock(std::move(words), std::move(wordXpos), std::move(wordStyles), style));
}
+3 -2
View File
@@ -1,5 +1,6 @@
#pragma once
#include <EpdFontFamily.h>
#include <FS.h>
#include <list>
#include <memory>
@@ -35,6 +36,6 @@ class TextBlock final : public Block {
// given a renderer works out where to break the words into lines
void render(const GfxRenderer& renderer, int fontId, int x, int y) const;
BlockType getType() override { return TEXT_BLOCK; }
void serialize(std::ostream& os) const;
static std::unique_ptr<TextBlock> deserialize(std::istream& is);
void serialize(File& file) const;
static std::unique_ptr<TextBlock> deserialize(File& file);
};
+54 -24
View File
@@ -1,5 +1,6 @@
#include "ChapterHtmlSlimParser.h"
#include <FsHelpers.h>
#include <GfxRenderer.h>
#include <HardwareSerial.h>
#include <expat.h>
@@ -10,13 +11,16 @@
const char* HEADER_TAGS[] = {"h1", "h2", "h3", "h4", "h5", "h6"};
constexpr int NUM_HEADER_TAGS = sizeof(HEADER_TAGS) / sizeof(HEADER_TAGS[0]);
const char* BLOCK_TAGS[] = {"p", "li", "div", "br"};
// Minimum file size (in bytes) to show progress bar - smaller chapters don't benefit from it
constexpr size_t MIN_SIZE_FOR_PROGRESS = 50 * 1024; // 50KB
const char* BLOCK_TAGS[] = {"p", "li", "div", "br", "blockquote"};
constexpr int NUM_BLOCK_TAGS = sizeof(BLOCK_TAGS) / sizeof(BLOCK_TAGS[0]);
const char* BOLD_TAGS[] = {"b"};
const char* BOLD_TAGS[] = {"b", "strong"};
constexpr int NUM_BOLD_TAGS = sizeof(BOLD_TAGS) / sizeof(BOLD_TAGS[0]);
const char* ITALIC_TAGS[] = {"i"};
const char* ITALIC_TAGS[] = {"i", "em"};
constexpr int NUM_ITALIC_TAGS = sizeof(ITALIC_TAGS) / sizeof(ITALIC_TAGS[0]);
const char* IMAGE_TAGS[] = {"img"};
@@ -151,7 +155,7 @@ void XMLCALL ChapterHtmlSlimParser::characterData(void* userData, const XML_Char
if (self->currentTextBlock->size() > 750) {
Serial.printf("[%lu] [EHP] Text block too long, splitting into multiple pages\n", millis());
self->currentTextBlock->layoutAndExtractLines(
self->renderer, self->fontId, self->marginLeft + self->marginRight,
self->renderer, self->fontId, self->viewportWidth,
[self](const std::shared_ptr<TextBlock>& textBlock) { self->addLineToPage(textBlock); }, false);
}
}
@@ -214,48 +218,75 @@ bool ChapterHtmlSlimParser::parseAndBuildPages() {
return false;
}
XML_SetUserData(parser, this);
XML_SetElementHandler(parser, startElement, endElement);
XML_SetCharacterDataHandler(parser, characterData);
FILE* file = fopen(filepath, "r");
if (!file) {
Serial.printf("[%lu] [EHP] Couldn't open file %s\n", millis(), filepath);
File file;
if (!FsHelpers::openFileForRead("EHP", filepath, file)) {
XML_ParserFree(parser);
return false;
}
// Get file size for progress calculation
const size_t totalSize = file.size();
size_t bytesRead = 0;
int lastProgress = -1;
XML_SetUserData(parser, this);
XML_SetElementHandler(parser, startElement, endElement);
XML_SetCharacterDataHandler(parser, characterData);
do {
void* const buf = XML_GetBuffer(parser, 1024);
if (!buf) {
Serial.printf("[%lu] [EHP] Couldn't allocate memory for buffer\n", millis());
XML_StopParser(parser, XML_FALSE); // Stop any pending processing
XML_SetElementHandler(parser, nullptr, nullptr); // Clear callbacks
XML_SetCharacterDataHandler(parser, nullptr);
XML_ParserFree(parser);
fclose(file);
file.close();
return false;
}
const size_t len = fread(buf, 1, 1024, file);
const size_t len = file.read(static_cast<uint8_t*>(buf), 1024);
if (ferror(file)) {
if (len == 0) {
Serial.printf("[%lu] [EHP] File read error\n", millis());
XML_StopParser(parser, XML_FALSE); // Stop any pending processing
XML_SetElementHandler(parser, nullptr, nullptr); // Clear callbacks
XML_SetCharacterDataHandler(parser, nullptr);
XML_ParserFree(parser);
fclose(file);
file.close();
return false;
}
done = feof(file);
// Update progress (call every 10% change to avoid too frequent updates)
// Only show progress for larger chapters where rendering overhead is worth it
bytesRead += len;
if (progressFn && totalSize >= MIN_SIZE_FOR_PROGRESS) {
const int progress = static_cast<int>((bytesRead * 100) / totalSize);
if (lastProgress / 10 != progress / 10) {
lastProgress = progress;
progressFn(progress);
}
}
done = file.available() == 0;
if (XML_ParseBuffer(parser, static_cast<int>(len), done) == XML_STATUS_ERROR) {
Serial.printf("[%lu] [EHP] Parse error at line %lu:\n%s\n", millis(), XML_GetCurrentLineNumber(parser),
XML_ErrorString(XML_GetErrorCode(parser)));
XML_StopParser(parser, XML_FALSE); // Stop any pending processing
XML_SetElementHandler(parser, nullptr, nullptr); // Clear callbacks
XML_SetCharacterDataHandler(parser, nullptr);
XML_ParserFree(parser);
fclose(file);
file.close();
return false;
}
} while (!done);
XML_StopParser(parser, XML_FALSE); // Stop any pending processing
XML_SetElementHandler(parser, nullptr, nullptr); // Clear callbacks
XML_SetCharacterDataHandler(parser, nullptr);
XML_ParserFree(parser);
fclose(file);
file.close();
// Process last page if there is still text
if (currentTextBlock) {
@@ -270,15 +301,14 @@ bool ChapterHtmlSlimParser::parseAndBuildPages() {
void ChapterHtmlSlimParser::addLineToPage(std::shared_ptr<TextBlock> line) {
const int lineHeight = renderer.getLineHeight(fontId) * lineCompression;
const int pageHeight = GfxRenderer::getScreenHeight() - marginTop - marginBottom;
if (currentPageNextY + lineHeight > pageHeight) {
if (currentPageNextY + lineHeight > viewportHeight) {
completePageFn(std::move(currentPage));
currentPage.reset(new Page());
currentPageNextY = marginTop;
currentPageNextY = 0;
}
currentPage->elements.push_back(std::make_shared<PageLine>(line, marginLeft, currentPageNextY));
currentPage->elements.push_back(std::make_shared<PageLine>(line, 0, currentPageNextY));
currentPageNextY += lineHeight;
}
@@ -290,12 +320,12 @@ void ChapterHtmlSlimParser::makePages() {
if (!currentPage) {
currentPage.reset(new Page());
currentPageNextY = marginTop;
currentPageNextY = 0;
}
const int lineHeight = renderer.getLineHeight(fontId) * lineCompression;
currentTextBlock->layoutAndExtractLines(
renderer, fontId, marginLeft + marginRight,
renderer, fontId, viewportWidth,
[this](const std::shared_ptr<TextBlock>& textBlock) { addLineToPage(textBlock); });
// Extra paragraph spacing if enabled
if (extraParagraphSpacing) {
+13 -14
View File
@@ -15,9 +15,10 @@ class GfxRenderer;
#define MAX_WORD_SIZE 200
class ChapterHtmlSlimParser {
const char* filepath;
const std::string& filepath;
GfxRenderer& renderer;
std::function<void(std::unique_ptr<Page>)> completePageFn;
std::function<void(int)> progressFn; // Progress callback (0-100)
int depth = 0;
int skipUntilDepth = INT_MAX;
int boldUntilDepth = INT_MAX;
@@ -31,11 +32,9 @@ class ChapterHtmlSlimParser {
int16_t currentPageNextY = 0;
int fontId;
float lineCompression;
int marginTop;
int marginRight;
int marginBottom;
int marginLeft;
bool extraParagraphSpacing;
int viewportWidth;
int viewportHeight;
void startNewTextBlock(TextBlock::BLOCK_STYLE style);
void makePages();
@@ -45,20 +44,20 @@ class ChapterHtmlSlimParser {
static void XMLCALL endElement(void* userData, const XML_Char* name);
public:
explicit ChapterHtmlSlimParser(const char* filepath, GfxRenderer& renderer, const int fontId,
const float lineCompression, const int marginTop, const int marginRight,
const int marginBottom, const int marginLeft, const bool extraParagraphSpacing,
const std::function<void(std::unique_ptr<Page>)>& completePageFn)
explicit ChapterHtmlSlimParser(const std::string& filepath, GfxRenderer& renderer, const int fontId,
const float lineCompression, const bool extraParagraphSpacing, const int viewportWidth,
const int viewportHeight,
const std::function<void(std::unique_ptr<Page>)>& completePageFn,
const std::function<void(int)>& progressFn = nullptr)
: filepath(filepath),
renderer(renderer),
fontId(fontId),
lineCompression(lineCompression),
marginTop(marginTop),
marginRight(marginRight),
marginBottom(marginBottom),
marginLeft(marginLeft),
extraParagraphSpacing(extraParagraphSpacing),
completePageFn(completePageFn) {}
viewportWidth(viewportWidth),
viewportHeight(viewportHeight),
completePageFn(completePageFn),
progressFn(progressFn) {}
~ChapterHtmlSlimParser() = default;
bool parseAndBuildPages();
void addLineToPage(std::shared_ptr<TextBlock> line);
@@ -16,6 +16,8 @@ bool ContainerParser::setup() {
ContainerParser::~ContainerParser() {
if (parser) {
XML_StopParser(parser, XML_FALSE); // Stop any pending processing
XML_SetElementHandler(parser, nullptr, nullptr); // Clear callbacks
XML_ParserFree(parser);
parser = nullptr;
}

Some files were not shown because too many files have changed in this diff Show More