Cut release 0.8.1

## Summary

* HTML files are now static, streamed directly to the client without
modification
* For any dynamic values, load via JSON APIs
* For files page, we stream the JSON content as we scan the directory to
avoid holding onto too much data

## Additional details

* We were previously building up a very large string all generated on
the X4 directly, we should be leveraging the browser
* Fixes https://github.com/daveallie/crosspoint-reader/issues/94

.github/workflows/ci.yml

@@ -34,7 +34,7 @@ jobs:
           sudo apt-get install -y clang-format-21
 
       - name: Run cppcheck
-        run: pio check --fail-on-defect medium --fail-on-defect high
+        run: pio check --fail-on-defect low --fail-on-defect medium --fail-on-defect high
 
       - name: Run clang-format
         run: PATH="/usr/lib/llvm-21/bin:$PATH" ./bin/clang-format-fix && git diff --exit-code || (echo "Please run 'bin/clang-format-fix' to fix formatting issues" && exit 1)

.gitignore

@@ -3,3 +3,4 @@
 .DS_Store
 .vscode
 lib/EpdFont/fontsrc
+*.generated.h

README.md

@@ -23,18 +23,23 @@ CrossPoint Reader aims to:
 
 This project is **not affiliated with Xteink**; it's built as a community project.
 
-## Features
+## Features & Usage
 
 - [x] EPUB parsing and rendering
+- [ ] Image support within EPUB
 - [x] Saved reading position
-- [ ] File explorer with file picker
+- [x] File explorer with file picker
   - [x] Basic EPUB picker from root directory
   - [x] Support nested folders
   - [ ] EPUB picker with cover art
-- [ ] Image support within EPUB
+- [x] Custom sleep screen
+  - [ ] Cover sleep screen
+- [x] Wifi book upload
+- [ ] Wifi OTA updates
 - [ ] Configurable font, layout, and display options
-- [ ] WiFi connectivity
-- [ ] BLE connectivity
+- [ ] Screen rotation
+
+See [the user guide](./USER_GUIDE.md) for instructions on operating CrossPoint.
 
 ## Installing
 
@@ -59,10 +64,6 @@ back to the other partition using the "Swap boot partition" button here https://
 
 See [Development](#development) below.
 
-## Usage
-
-See [the user guide](./USER_GUIDE.md) for instructions on operating CrossPoint.
-
 ## Development
 
 ### Prerequisites

USER_GUIDE.md

@@ -19,24 +19,72 @@ The device utilises the standard buttons on the Xtink X4 in the same layout:
 
 ### Power On / Off
 
-To turn the device on or off, **press and hold the Power button for 1 full second**.
+To turn the device on or off, **press and hold the Power button for half a second**. In **Settings** you can configure
+the power button to trigger on a short press instead of a long one.
 
 ### First Launch
 
-Upon turning the device on for the first time, you will be placed on the **Book Selection Screen** (File Browser).
+Upon turning the device on for the first time, you will be placed on the **Home** screen.
 
 > **Note:** On subsequent restarts, the firmware will automatically reopen the last book you were reading.
 
 ---
 
-## 3. Book Selection
+## 3. Screens
 
-The Home Screen acts as a folder and file browser.
+### 3.1 Home Screen
+
+The Home Screen is the main entry point to the firmware. From here you can navigate to the **Book Selection** screen,
+**Settings** screen, or **File Upload** screen.
+
+### 3.2 Book Selection (Read)
+
+The Book Selection acts as a folder and file browser.
 
 * **Navigate List:** Use **Left** (or **Volume Up**), or **Right** (or **Volume Down**) to move the selection cursor up 
   and down through folders and books.
 * **Open Selection:** Press **Confirm** to open a folder or read a selected book.
 
+### 3.3 Reading Screen
+
+See [4. Reading Mode](#4-reading-mode) below for more information.
+
+### 3.4 File Upload Screen
+
+The File Upload screen allows you to upload new e-books to the device. When you enter the screen you'll be prompted with
+a WiFi selection dialog and then your X4 will start hosting a web server.
+
+See the [webserver docs](./docs/webserver.md) for more information on how to connect to the web server and upload files.
+
+### 3.5 Settings
+
+The Settings screen allows you to configure the device's behavior. There are a few settings you can adjust:
+- **Sleep Screen**: Which sleep screen to display when the device sleeps, options are:
+  - "Dark" (default) - The default dark sleep screen
+  - "Light" - The same default sleep screen, on a white background
+  - "Custom" - Custom images from the SD card, see [3.6 Sleep Screen](#36-sleep-screen) below for more information
+  - "Cover" - The book cover image (Note: this is experimental and may not work as expected)
+- **Extra Paragraph Spacing**: If enabled, vertical space will be added between paragraphs in the book, if disabled,
+  paragraphs will not have vertical space between them, but will have first word indentation.
+- **Short Power Button Click**: Whether to trigger the power button on a short press or a long press.
+
+### 3.6 Sleep Screen
+
+You can customize the sleep screen by placing custom images in specific locations on the SD card:
+
+- **Single Image:** Place a file named `sleep.bmp` in the root directory.
+- **Multiple Images:** Create a `sleep` directory in the root of the SD card and place any number of `.bmp` images
+  inside. If images are found in this directory, they will take priority over the `sleep.png` file, and one will be
+  randomly selected each time the device sleeps.
+
+> [!NOTE]
+> You'll need to set the **Sleep Screen** setting to **Custom** in order to use these images.
+
+> [!TIP]
+> For best results:
+> - Use uncompressed BMP files with 24-bit color depth
+> - Use a resolution of 480x800 pixels to match the device's screen resolution.
+
 ---
 
 ## 4. Reading Mode
@@ -76,3 +124,4 @@ are planned for future updates:
 
 * **Images:** Embedded images in e-books will not render.
 * **Text Formatting:** There are currently no settings to adjust font type, size, line spacing, or margins.
+* **Rotation**: Different rotation options are not supported.

docs/webserver.md

@@ -0,0 +1,272 @@
+# Web Server Guide
+
+This guide explains how to connect your CrossPoint Reader to WiFi and use the built-in web server to upload EPUB files from your computer or phone.
+
+## Overview
+
+CrossPoint Reader includes a built-in web server that allows you to:
+
+- Upload EPUB files wirelessly from any device on the same WiFi network
+- Browse and manage files on your device's SD card
+- Create folders to organize your ebooks
+- Delete files and folders
+
+## Prerequisites
+
+- Your CrossPoint Reader device
+- A WiFi network
+- A computer, phone, or tablet connected to the **same WiFi network**
+
+---
+
+## Step 1: Accessing the WiFi Screen
+
+1. From the main menu or file browser, navigate to the **Settings** screen
+2. Select the **WiFi** option
+3. The device will automatically start scanning for available networks
+
+---
+
+## Step 2: Connecting to WiFi
+
+### Viewing Available Networks
+
+Once the scan completes, you'll see a list of available WiFi networks with the following indicators:
+
+- **Signal strength bars** (`||||`, `|||`, `||`, `|`) - Shows connection quality
+- **`*` symbol** - Indicates the network is password-protected (encrypted)
+- **`+` symbol** - Indicates you have previously saved credentials for this network
+
+<img src="./images/wifi/wifi_networks.jpeg" height="500">
+
+### Selecting a Network
+
+1. Use the **Left/Right** (or **Volume Up/Down**) buttons to navigate through the network list
+2. Press **Confirm** to select the highlighted network
+
+### Entering Password (for encrypted networks)
+
+If the network requires a password:
+
+1. An on-screen keyboard will appear
+2. Use the navigation buttons to select characters
+3. Press **Confirm** to enter each character
+4. When complete, select the **Done** option on the keyboard
+
+<img src="./images/wifi/wifi_password.jpeg" height="500">
+
+**Note:** If you've previously connected to this network, the saved password will be used automatically.
+
+### Connection Process
+
+The device will display "Connecting..." while establishing the connection. This typically takes 5-10 seconds.
+
+### Saving Credentials
+
+If this is a new network, you'll be prompted to save the password:
+
+- Select **Yes** to save credentials for automatic connection next time (NOTE: These are stored in plaintext on the device's SD card. Do not use this for sensitive networks.)
+- Select **No** to connect without saving
+
+---
+
+## Step 3: Connection Success
+
+Once connected, the screen will display:
+
+- **Network name** (SSID)
+- **IP Address** (e.g., `192.168.1.102`)
+- **Web server URL** (e.g., `http://192.168.1.102/`)
+
+<img src="./images/wifi/wifi_connected.jpeg" height="500">
+
+**Important:** Make note of the IP address - you'll need this to access the web interface from your computer or phone.
+
+---
+
+## Step 4: Accessing the Web Interface
+
+### From a Computer
+
+1. Ensure your computer is connected to the **same WiFi network** as your CrossPoint Reader
+2. Open any web browser (Chrome is recommended)
+3. Type the IP address shown on your device into the browser's address bar
+   - Example: `http://192.168.1.102/`
+4. Press Enter
+
+### From a Phone or Tablet
+
+1. Ensure your phone/tablet is connected to the **same WiFi network** as your CrossPoint Reader
+2. Open your mobile browser (Safari, Chrome, etc.)
+3. Type the IP address into the address bar
+   - Example: `http://192.168.1.102/`
+4. Tap Go
+
+---
+
+## Step 5: Using the Web Interface
+
+### Home Page
+
+The home page displays:
+
+- Device status and version information
+- WiFi connection status
+- Current IP address
+- Available memory
+
+Navigation links:
+
+- **Home** - Returns to the status page
+- **File Manager** - Access file management features
+
+<img src="./images/wifi/webserver_homepage.png" width="600">
+
+### File Manager
+
+Click **File Manager** to access file management features.
+
+#### Browsing Files
+
+- The file manager displays all files and folders on your SD card
+- **Folders** are highlighted in yellow with a 📁 icon
+- **EPUB files** are highlighted in green with a 📗 icon
+- Click on a folder name to navigate into it
+- Use the breadcrumb navigation at the top to go back to parent folders
+
+<img src="./images/wifi/webserver_files.png" width="600">
+
+#### Uploading EPUB Files
+
+1. Click the **+ Add** button in the top-right corner
+2. Select **Upload eBook** from the dropdown menu
+3. Click **Choose File** and select an `.epub` file from your device
+4. Click **Upload**
+5. A progress bar will show the upload status
+6. The page will automatically refresh when the upload is complete
+
+**Note:** Only `.epub` files are accepted. Other file types will be rejected.
+
+<img src="./images/wifi/webserver_upload.png" width="600">
+
+#### Creating Folders
+
+1. Click the **+ Add** button in the top-right corner
+2. Select **New Folder** from the dropdown menu
+3. Enter a folder name (letters, numbers, underscores, and hyphens only)
+4. Click **Create Folder**
+
+This is useful for organizing your ebooks by genre, author, or series.
+
+#### Deleting Files and Folders
+
+1. Click the **🗑️** (trash) icon next to any file or folder
+2. Confirm the deletion in the popup dialog
+3. Click **Delete** to permanently remove the item
+
+**Warning:** Deletion is permanent and cannot be undone!
+
+**Note:** Folders must be empty before they can be deleted.
+
+---
+
+## Troubleshooting
+
+### Cannot See the Device on the Network
+
+**Problem:** Browser shows "Cannot connect" or "Site can't be reached"
+
+**Solutions:**
+
+1. Verify both devices are on the **same WiFi network**
+   - Check your computer/phone WiFi settings
+   - Confirm the CrossPoint Reader shows "Connected" status
+2. Double-check the IP address
+   - Make sure you typed it correctly
+   - Include `http://` at the beginning
+3. Try disabling VPN if you're using one
+4. Some networks have "client isolation" enabled - check with your network administrator
+
+### Connection Drops or Times Out
+
+**Problem:** WiFi connection is unstable
+
+**Solutions:**
+
+1. Move closer to the WiFi router
+2. Check signal strength on the device (should be at least `||` or better)
+3. Avoid interference from other devices
+4. Try a different WiFi network if available
+
+### Upload Fails
+
+**Problem:** File upload doesn't complete or shows an error
+
+**Solutions:**
+
+1. Ensure the file is a valid `.epub` file
+2. Check that the SD card has enough free space
+3. Try uploading a smaller file first to test
+4. Refresh the browser page and try again
+
+### Saved Password Not Working
+
+**Problem:** Device fails to connect with saved credentials
+
+**Solutions:**
+
+1. When connection fails, you'll be prompted to "Forget Network"
+2. Select **Yes** to remove the saved password
+3. Reconnect and enter the password again
+4. Choose to save the new password
+
+---
+
+## Security Notes
+
+- The web server runs on port 80 (standard HTTP)
+- **No authentication is required** - anyone on the same network can access the interface
+- The web server is only accessible while the WiFi screen shows "Connected"
+- The web server automatically stops when you exit the WiFi screen
+- For security, only use on trusted private networks
+
+---
+
+## Technical Details
+
+- **Supported WiFi:** 2.4GHz networks (802.11 b/g/n)
+- **Web Server Port:** 80 (HTTP)
+- **Maximum Upload Size:** Limited by available SD card space
+- **Supported File Format:** `.epub` only
+- **Browser Compatibility:** All modern browsers (Chrome, Firefox, Safari, Edge)
+
+---
+
+## Tips and Best Practices
+
+1. **Organize with folders** - Create folders before uploading to keep your library organized
+2. **Check signal strength** - Stronger signals (`|||` or `||||`) provide faster, more reliable uploads
+3. **Upload multiple files** - You can upload files one at a time; the page refreshes after each upload
+4. **Use descriptive names** - Name your folders clearly (e.g., "SciFi", "Mystery", "Non-Fiction")
+5. **Keep credentials saved** - Save your WiFi password for quick reconnection in the future
+6. **Exit when done** - Press **Back** to exit the WiFi screen and save battery
+
+---
+
+## Exiting WiFi Mode
+
+When you're finished uploading files:
+
+1. Press the **Back** button on your CrossPoint Reader
+2. The web server will automatically stop
+3. WiFi will disconnect to conserve battery
+4. You'll return to the previous screen
+
+Your uploaded files will be immediately available in the file browser!
+
+---
+
+## Related Documentation
+
+- [User Guide](../USER_GUIDE.md) - General device operation
+- [README](../README.md) - Project overview and features

lib/Epub/Epub.cpp

@@ -1,6 +1,7 @@
 #include "Epub.h"
 
 #include <HardwareSerial.h>
+#include <JpegToBmpConverter.h>
 #include <SD.h>
 #include <ZipFile.h>
 
@@ -30,24 +31,22 @@ bool Epub::findContentOpfFile(std::string* contentOpfFile) const {
   // Stream read (reusing your existing stream logic)
   if (!readItemContentsToStream(containerPath, containerParser, 512)) {
     Serial.printf("[%lu] [EBP] Could not read META-INF/container.xml\n", millis());
-    containerParser.teardown();
     return false;
   }
 
   // Extract the result
   if (containerParser.fullPath.empty()) {
     Serial.printf("[%lu] [EBP] Could not find valid rootfile in container.xml\n", millis());
-    containerParser.teardown();
     return false;
   }
 
   *contentOpfFile = std::move(containerParser.fullPath);
-
-  containerParser.teardown();
   return true;
 }
 
 bool Epub::parseContentOpf(const std::string& contentOpfFilePath) {
+  Serial.printf("[%lu] [EBP] Parsing content.opf: %s\n", millis(), contentOpfFilePath.c_str());
+
   size_t contentOpfSize;
   if (!getItemSize(contentOpfFilePath, &contentOpfSize)) {
     Serial.printf("[%lu] [EBP] Could not get size of content.opf\n", millis());
@@ -63,7 +62,6 @@ bool Epub::parseContentOpf(const std::string& contentOpfFilePath) {
 
   if (!readItemContentsToStream(contentOpfFilePath, opfParser, 1024)) {
     Serial.printf("[%lu] [EBP] Could not read content.opf\n", millis());
-    opfParser.teardown();
     return false;
   }
 
@@ -84,8 +82,6 @@ bool Epub::parseContentOpf(const std::string& contentOpfFilePath) {
   }
 
   Serial.printf("[%lu] [EBP] Successfully parsed content.opf\n", millis());
-
-  opfParser.teardown();
   return true;
 }
 
@@ -96,37 +92,53 @@ bool Epub::parseTocNcxFile() {
     return false;
   }
 
-  size_t tocSize;
-  if (!getItemSize(tocNcxItem, &tocSize)) {
-    Serial.printf("[%lu] [EBP] Could not get size of toc ncx\n", millis());
-    return false;
-  }
+  Serial.printf("[%lu] [EBP] Parsing toc ncx file: %s\n", millis(), tocNcxItem.c_str());
 
-  TocNcxParser ncxParser(contentBasePath, tocSize);
+  const auto tmpNcxPath = getCachePath() + "/toc.ncx";
+  File tempNcxFile = SD.open(tmpNcxPath.c_str(), FILE_WRITE);
+  readItemContentsToStream(tocNcxItem, tempNcxFile, 1024);
+  tempNcxFile.close();
+  tempNcxFile = SD.open(tmpNcxPath.c_str(), FILE_READ);
+  const auto ncxSize = tempNcxFile.size();
+
+  TocNcxParser ncxParser(contentBasePath, ncxSize);
 
   if (!ncxParser.setup()) {
     Serial.printf("[%lu] [EBP] Could not setup toc ncx parser\n", millis());
     return false;
   }
 
-  if (!readItemContentsToStream(tocNcxItem, ncxParser, 1024)) {
-    Serial.printf("[%lu] [EBP] Could not read toc ncx stream\n", millis());
-    ncxParser.teardown();
+  const auto ncxBuffer = static_cast<uint8_t*>(malloc(1024));
+  if (!ncxBuffer) {
+    Serial.printf("[%lu] [EBP] Could not allocate memory for toc ncx parser\n", millis());
     return false;
   }
 
+  while (tempNcxFile.available()) {
+    const auto readSize = tempNcxFile.read(ncxBuffer, 1024);
+    const auto processedSize = ncxParser.write(ncxBuffer, readSize);
+
+    if (processedSize != readSize) {
+      Serial.printf("[%lu] [EBP] Could not process all toc ncx data\n", millis());
+      free(ncxBuffer);
+      tempNcxFile.close();
+      return false;
+    }
+  }
+
+  free(ncxBuffer);
+  tempNcxFile.close();
+  SD.remove(tmpNcxPath.c_str());
+
   this->toc = std::move(ncxParser.toc);
 
   Serial.printf("[%lu] [EBP] Parsed %d TOC items\n", millis(), this->toc.size());
-
-  ncxParser.teardown();
   return true;
 }
 
 // load in the meta data for the epub file
 bool Epub::load() {
   Serial.printf("[%lu] [EBP] Loading ePub: %s\n", millis(), filepath.c_str());
-  ZipFile zip("/sd" + filepath);
 
   std::string contentOpfFilePath;
   if (!findContentOpfFile(&contentOpfFilePath)) {
@@ -155,44 +167,20 @@ bool Epub::load() {
 }
 
 void Epub::initializeSpineItemSizes() {
-  setupCacheDir();
+  Serial.printf("[%lu] [EBP] Calculating book size\n", millis());
 
-  size_t spineItemsCount = getSpineItemsCount();
+  const size_t spineItemsCount = getSpineItemsCount();
   size_t cumSpineItemSize = 0;
-  if (SD.exists((getCachePath() + "/spine_size.bin").c_str())) {
-    File f = SD.open((getCachePath() + "/spine_size.bin").c_str());
-    uint8_t data[4];
-    for (size_t i = 0; i < spineItemsCount; i++) {
-      f.read(data, 4);
-      cumSpineItemSize = data[0] | (data[1] << 8) | (data[2] << 16) | (data[3] << 24);
-      cumulativeSpineItemSize.emplace_back(cumSpineItemSize);
-      // Serial.printf("[%lu] [EBP] Loading item %d size %u to %u %u\n", millis(),
-      //     i, cumSpineItemSize, data[1], data[0]);
-    }
-    f.close();
-  } else {
-    File f = SD.open((getCachePath() + "/spine_size.bin").c_str(), FILE_WRITE);
-    uint8_t data[4];
-    // determine size of spine items
-    for (size_t i = 0; i < spineItemsCount; i++) {
-      std::string spineItem = getSpineItem(i);
-      size_t s = 0;
-      getItemSize(spineItem, &s);
-      cumSpineItemSize += s;
-      cumulativeSpineItemSize.emplace_back(cumSpineItemSize);
+  const ZipFile zip("/sd" + filepath);
 
-      // and persist to cache
-      data[0] = cumSpineItemSize & 0xFF;
-      data[1] = (cumSpineItemSize >> 8) & 0xFF;
-      data[2] = (cumSpineItemSize >> 16) & 0xFF;
-      data[3] = (cumSpineItemSize >> 24) & 0xFF;
-      // Serial.printf("[%lu] [EBP] Persisting item %d size %u to %u %u\n", millis(),
-      //     i, cumSpineItemSize, data[1], data[0]);
-      f.write(data, 4);
-    }
-
-    f.close();
+  for (size_t i = 0; i < spineItemsCount; i++) {
+    std::string spineItem = getSpineItem(i);
+    size_t s = 0;
+    getItemSize(zip, spineItem, &s);
+    cumSpineItemSize += s;
+    cumulativeSpineItemSize.emplace_back(cumSpineItemSize);
   }
+
   Serial.printf("[%lu] [EBP] Book size: %lu\n", millis(), cumSpineItemSize);
 }
 
@@ -231,7 +219,45 @@ const std::string& Epub::getPath() const { return filepath; }
 
 const std::string& Epub::getTitle() const { return title; }
 
-const std::string& Epub::getCoverImageItem() const { return coverImageItem; }
+std::string Epub::getCoverBmpPath() const { return cachePath + "/cover.bmp"; }
+
+bool Epub::generateCoverBmp() const {
+  // Already generated, return true
+  if (SD.exists(getCoverBmpPath().c_str())) {
+    return true;
+  }
+
+  if (coverImageItem.empty()) {
+    Serial.printf("[%lu] [EBP] No known cover image\n", millis());
+    return false;
+  }
+
+  if (coverImageItem.substr(coverImageItem.length() - 4) == ".jpg" ||
+      coverImageItem.substr(coverImageItem.length() - 5) == ".jpeg") {
+    Serial.printf("[%lu] [EBP] Generating BMP from JPG cover image\n", millis());
+    File coverJpg = SD.open((getCachePath() + "/.cover.jpg").c_str(), FILE_WRITE, true);
+    readItemContentsToStream(coverImageItem, coverJpg, 1024);
+    coverJpg.close();
+
+    coverJpg = SD.open((getCachePath() + "/.cover.jpg").c_str(), FILE_READ);
+    File coverBmp = SD.open(getCoverBmpPath().c_str(), FILE_WRITE, true);
+    const bool success = JpegToBmpConverter::jpegFileToBmpStream(coverJpg, coverBmp);
+    coverJpg.close();
+    coverBmp.close();
+    SD.remove((getCachePath() + "/.cover.jpg").c_str());
+
+    if (!success) {
+      Serial.printf("[%lu] [EBP] Failed to generate BMP from JPG cover image\n", millis());
+      SD.remove(getCoverBmpPath().c_str());
+    }
+    Serial.printf("[%lu] [EBP] Generated BMP from JPG cover image, success: %s\n", millis(), success ? "yes" : "no");
+    return success;
+  } else {
+    Serial.printf("[%lu] [EBP] Cover image is not a JPG, skipping\n", millis());
+  }
+
+  return false;
+}
 
 std::string normalisePath(const std::string& path) {
   std::vector<std::string> components;
@@ -291,17 +317,31 @@ bool Epub::readItemContentsToStream(const std::string& itemHref, Print& out, con
 
 bool Epub::getItemSize(const std::string& itemHref, size_t* size) const {
   const ZipFile zip("/sd" + filepath);
-  const std::string path = normalisePath(itemHref);
+  return getItemSize(zip, itemHref, size);
+}
 
+bool Epub::getItemSize(const ZipFile& zip, const std::string& itemHref, size_t* size) {
+  const std::string path = normalisePath(itemHref);
   return zip.getInflatedFileSize(path.c_str(), size);
 }
 
 int Epub::getSpineItemsCount() const { return spine.size(); }
 
-size_t Epub::getCumulativeSpineItemSize(const int spineIndex) const { return cumulativeSpineItemSize.at(spineIndex); }
+size_t Epub::getCumulativeSpineItemSize(const int spineIndex) const {
+  if (spineIndex < 0 || spineIndex >= static_cast<int>(cumulativeSpineItemSize.size())) {
+    Serial.printf("[%lu] [EBP] getCumulativeSpineItemSize index:%d is out of range\n", millis(), spineIndex);
+    return 0;
+  }
+  return cumulativeSpineItemSize.at(spineIndex);
+}
 
 std::string& Epub::getSpineItem(const int spineIndex) {
-  if (spineIndex < 0 || spineIndex >= spine.size()) {
+  static std::string emptyString;
+  if (spine.empty()) {
+    Serial.printf("[%lu] [EBP] getSpineItem called but spine is empty\n", millis());
+    return emptyString;
+  }
+  if (spineIndex < 0 || spineIndex >= static_cast<int>(spine.size())) {
     Serial.printf("[%lu] [EBP] getSpineItem index:%d is out of range\n", millis(), spineIndex);
     return spine.at(0).second;
   }
@@ -310,7 +350,12 @@ std::string& Epub::getSpineItem(const int spineIndex) {
 }
 
 EpubTocEntry& Epub::getTocItem(const int tocTndex) {
-  if (tocTndex < 0 || tocTndex >= toc.size()) {
+  static EpubTocEntry emptyEntry = {};
+  if (toc.empty()) {
+    Serial.printf("[%lu] [EBP] getTocItem called but toc is empty\n", millis());
+    return emptyEntry;
+  }
+  if (tocTndex < 0 || tocTndex >= static_cast<int>(toc.size())) {
     Serial.printf("[%lu] [EBP] getTocItem index:%d is out of range\n", millis(), tocTndex);
     return toc.at(0);
   }
@@ -322,6 +367,11 @@ int Epub::getTocItemsCount() const { return toc.size(); }
 
 // work out the section index for a toc index
 int Epub::getSpineIndexForTocIndex(const int tocIndex) const {
+  if (tocIndex < 0 || tocIndex >= toc.size()) {
+    Serial.printf("[%lu] [EBP] getSpineIndexForTocIndex: tocIndex %d out of range\n", millis(), tocIndex);
+    return 0;
+  }
+
   // the toc entry should have an href that matches the spine item
   // so we can find the spine index by looking for the href
   for (int i = 0; i < spine.size(); i++) {
@@ -336,6 +386,11 @@ int Epub::getSpineIndexForTocIndex(const int tocIndex) const {
 }
 
 int Epub::getTocIndexForSpineIndex(const int spineIndex) const {
+  if (spineIndex < 0 || spineIndex >= spine.size()) {
+    Serial.printf("[%lu] [EBP] getTocIndexForSpineIndex: spineIndex %d out of range\n", millis(), spineIndex);
+    return -1;
+  }
+
   // the toc entry should have an href that matches the spine item
   // so we can find the toc index by looking for the href
   for (int i = 0; i < toc.size(); i++) {
@@ -348,13 +403,21 @@ int Epub::getTocIndexForSpineIndex(const int spineIndex) const {
   return -1;
 }
 
-size_t Epub::getBookSize() const { return getCumulativeSpineItemSize(getSpineItemsCount() - 1); }
+size_t Epub::getBookSize() const {
+  if (spine.empty()) {
+    return 0;
+  }
+  return getCumulativeSpineItemSize(getSpineItemsCount() - 1);
+}
 
 // Calculate progress in book
 uint8_t Epub::calculateProgress(const int currentSpineIndex, const float currentSpineRead) {
+  size_t bookSize = getBookSize();
+  if (bookSize == 0) {
+    return 0;
+  }
   size_t prevChapterSize = (currentSpineIndex >= 1) ? getCumulativeSpineItemSize(currentSpineIndex - 1) : 0;
   size_t curChapterSize = getCumulativeSpineItemSize(currentSpineIndex) - prevChapterSize;
-  size_t bookSize = getBookSize();
   size_t sectionProgSize = currentSpineRead * curChapterSize;
   return round(static_cast<float>(prevChapterSize + sectionProgSize) / bookSize * 100.0);
 }

lib/Epub/Epub.h

@@ -33,6 +33,7 @@ class Epub {
   bool parseContentOpf(const std::string& contentOpfFilePath);
   bool parseTocNcxFile();
   void initializeSpineItemSizes();
+  static bool getItemSize(const ZipFile& zip, const std::string& itemHref, size_t* size);
 
  public:
   explicit Epub(std::string filepath, const std::string& cacheDir) : filepath(std::move(filepath)) {
@@ -47,7 +48,8 @@ class Epub {
   const std::string& getCachePath() const;
   const std::string& getPath() const;
   const std::string& getTitle() const;
-  const std::string& getCoverImageItem() const;
+  std::string getCoverBmpPath() const;
+  bool generateCoverBmp() const;
   uint8_t* readItemContentsToBytes(const std::string& itemHref, size_t* size = nullptr,
                                    bool trailingNullByte = false) const;
   bool readItemContentsToStream(const std::string& itemHref, Print& out, size_t chunkSize) const;

lib/Epub/Epub/EpubTocEntry.h

@@ -2,12 +2,9 @@
 
 #include <string>
 
-class EpubTocEntry {
- public:
+struct EpubTocEntry {
   std::string title;
   std::string href;
   std::string anchor;
-  int level;
-  EpubTocEntry(std::string title, std::string href, std::string anchor, const int level)
-      : title(std::move(title)), href(std::move(href)), anchor(std::move(anchor)), level(level) {}
+  uint8_t level;
 };

lib/Epub/Epub/ParsedText.cpp

@@ -19,20 +19,35 @@ void ParsedText::addWord(std::string word, const EpdFontStyle fontStyle) {
 
 // Consumes data to minimize memory usage
 void ParsedText::layoutAndExtractLines(const GfxRenderer& renderer, const int fontId, const int horizontalMargin,
-                                       const std::function<void(std::shared_ptr<TextBlock>)>& processLine) {
+                                       const std::function<void(std::shared_ptr<TextBlock>)>& processLine,
+                                       const bool includeLastLine) {
   if (words.empty()) {
     return;
   }
 
-  const size_t totalWordCount = words.size();
   const int pageWidth = renderer.getScreenWidth() - horizontalMargin;
   const int spaceWidth = renderer.getSpaceWidth(fontId);
-  // width of 1em to indent first line of paragraph if Extra Spacing is enabled
-  const int indentWidth = (!extraParagraphSpacing) ? 1 * renderer.getTextWidth(fontId, "m", REGULAR) : 0;
+  const auto wordWidths = calculateWordWidths(renderer, fontId);
+  const auto lineBreakIndices = computeLineBreaks(pageWidth, spaceWidth, wordWidths);
+  const size_t lineCount = includeLastLine ? lineBreakIndices.size() : lineBreakIndices.size() - 1;
+
+  for (size_t i = 0; i < lineCount; ++i) {
+    extractLine(i, pageWidth, spaceWidth, wordWidths, lineBreakIndices, processLine);
+  }
+}
+
+std::vector<uint16_t> ParsedText::calculateWordWidths(const GfxRenderer& renderer, const int fontId) {
+  const size_t totalWordCount = words.size();
 
   std::vector<uint16_t> wordWidths;
   wordWidths.reserve(totalWordCount);
 
+  // add em-space at the beginning of first word in paragraph to indent
+  if (!extraParagraphSpacing) {
+    std::string& first_word = words.front();
+    first_word.insert(0, "\xe2\x80\x83");
+  }
+
   auto wordsIt = words.begin();
   auto wordStylesIt = wordStyles.begin();
 
@@ -43,6 +58,13 @@ void ParsedText::layoutAndExtractLines(const GfxRenderer& renderer, const int fo
     std::advance(wordStylesIt, 1);
   }
 
+  return wordWidths;
+}
+
+std::vector<size_t> ParsedText::computeLineBreaks(const int pageWidth, const int spaceWidth,
+                                                  const std::vector<uint16_t>& wordWidths) const {
+  const size_t totalWordCount = words.size();
+
   // DP table to store the minimum badness (cost) of lines starting at index i
   std::vector<int> dp(totalWordCount);
   // 'ans[i]' stores the index 'j' of the *last word* in the optimal line starting at 'i'
@@ -53,7 +75,7 @@ void ParsedText::layoutAndExtractLines(const GfxRenderer& renderer, const int fo
   ans[totalWordCount - 1] = totalWordCount - 1;
 
   for (int i = totalWordCount - 2; i >= 0; --i) {
-    int currlen = -spaceWidth + indentWidth;
+    int currlen = -spaceWidth;
     dp[i] = MAX_COST;
 
     for (size_t j = i; j < totalWordCount; ++j) {
@@ -102,70 +124,59 @@ void ParsedText::layoutAndExtractLines(const GfxRenderer& renderer, const int fo
     currentWordIndex = nextBreakIndex;
   }
 
-  // Initialize iterators for consumption
-  auto wordStartIt = words.begin();
-  auto wordStyleStartIt = wordStyles.begin();
-  size_t wordWidthIndex = 0;
+  return lineBreakIndices;
+}
+
+void ParsedText::extractLine(const size_t breakIndex, const int pageWidth, const int spaceWidth,
+                             const std::vector<uint16_t>& wordWidths, const std::vector<size_t>& lineBreakIndices,
+                             const std::function<void(std::shared_ptr<TextBlock>)>& processLine) {
+  const size_t lineBreak = lineBreakIndices[breakIndex];
+  const size_t lastBreakAt = breakIndex > 0 ? lineBreakIndices[breakIndex - 1] : 0;
+  const size_t lineWordCount = lineBreak - lastBreakAt;
+
+  // Calculate total word width for this line
+  int lineWordWidthSum = 0;
+  for (size_t i = lastBreakAt; i < lineBreak; i++) {
+    lineWordWidthSum += wordWidths[i];
+  }
 
-  size_t lastBreakAt = 0;
-  for (const size_t lineBreak : lineBreakIndices) {
-    const size_t lineWordCount = lineBreak - lastBreakAt;
+  // Calculate spacing
+  const int spareSpace = pageWidth - lineWordWidthSum;
 
-    // Calculate end iterators for the range to splice
-    auto wordEndIt = wordStartIt;
-    auto wordStyleEndIt = wordStyleStartIt;
-    std::advance(wordEndIt, lineWordCount);
-    std::advance(wordStyleEndIt, lineWordCount);
+  int spacing = spaceWidth;
+  const bool isLastLine = breakIndex == lineBreakIndices.size() - 1;
 
-    // Calculate total word width for this line
-    int lineWordWidthSum = 0;
-    for (size_t i = 0; i < lineWordCount; ++i) {
-      lineWordWidthSum += wordWidths[wordWidthIndex + i];
-    }
+  if (style == TextBlock::JUSTIFIED && !isLastLine && lineWordCount >= 2) {
+    spacing = spareSpace / (lineWordCount - 1);
+  }
 
-    // Calculate spacing
-    int spareSpace = pageWidth - lineWordWidthSum;
-    if (wordWidthIndex == 0) {
-      spareSpace -= indentWidth;
-    }
+  // Calculate initial x position
+  uint16_t xpos = 0;
+  if (style == TextBlock::RIGHT_ALIGN) {
+    xpos = spareSpace - (lineWordCount - 1) * spaceWidth;
+  } else if (style == TextBlock::CENTER_ALIGN) {
+    xpos = (spareSpace - (lineWordCount - 1) * spaceWidth) / 2;
+  }
 
-    int spacing = spaceWidth;
-    const bool isLastLine = lineBreak == totalWordCount;
-
-    if (style == TextBlock::JUSTIFIED && !isLastLine && lineWordCount >= 2) {
-      spacing = spareSpace / (lineWordCount - 1);
-    }
+  // Pre-calculate X positions for words
+  std::list<uint16_t> lineXPos;
+  for (size_t i = lastBreakAt; i < lineBreak; i++) {
+    const uint16_t currentWordWidth = wordWidths[i];
+    lineXPos.push_back(xpos);
+    xpos += currentWordWidth + spacing;
+  }
 
-    // Calculate initial x position
-    uint16_t xpos = (wordWidthIndex == 0) ? indentWidth : 0;
+  // Iterators always start at the beginning as we are moving content with splice below
+  auto wordEndIt = words.begin();
+  auto wordStyleEndIt = wordStyles.begin();
+  std::advance(wordEndIt, lineWordCount);
+  std::advance(wordStyleEndIt, lineWordCount);
 
-    if (style == TextBlock::RIGHT_ALIGN) {
-      xpos = spareSpace - (lineWordCount - 1) * spaceWidth;
-    } else if (style == TextBlock::CENTER_ALIGN) {
-      xpos = (spareSpace - (lineWordCount - 1) * spaceWidth) / 2;
-    }
+  // *** CRITICAL STEP: CONSUME DATA USING SPLICE ***
+  std::list<std::string> lineWords;
+  lineWords.splice(lineWords.begin(), words, words.begin(), wordEndIt);
+  std::list<EpdFontStyle> lineWordStyles;
+  lineWordStyles.splice(lineWordStyles.begin(), wordStyles, wordStyles.begin(), wordStyleEndIt);
 
-    // Pre-calculate X positions for words
-    std::list<uint16_t> lineXPos;
-    for (size_t i = 0; i < lineWordCount; ++i) {
-      const uint16_t currentWordWidth = wordWidths[wordWidthIndex + i];
-      lineXPos.push_back(xpos);
-      xpos += currentWordWidth + spacing;
-    }
-
-    // *** CRITICAL STEP: CONSUME DATA USING SPLICE ***
-    std::list<std::string> lineWords;
-    lineWords.splice(lineWords.begin(), words, wordStartIt, wordEndIt);
-    std::list<EpdFontStyle> lineWordStyles;
-    lineWordStyles.splice(lineWordStyles.begin(), wordStyles, wordStyleStartIt, wordStyleEndIt);
-
-    processLine(
-        std::make_shared<TextBlock>(std::move(lineWords), std::move(lineXPos), std::move(lineWordStyles), style));
-
-    // Update pointers/indices for the next line
-    wordStartIt = wordEndIt;
-    wordStyleStartIt = wordStyleEndIt;
-    wordWidthIndex += lineWordCount;
-    lastBreakAt = lineBreak;
-  }
+  processLine(std::make_shared<TextBlock>(std::move(lineWords), std::move(lineXPos), std::move(lineWordStyles), style));
 }

lib/Epub/Epub/ParsedText.h

@@ -2,11 +2,11 @@
 
 #include <EpdFontFamily.h>
 
-#include <cstdint>
 #include <functional>
 #include <list>
 #include <memory>
 #include <string>
+#include <vector>
 
 #include "blocks/TextBlock.h"
 
@@ -18,6 +18,12 @@ class ParsedText {
   TextBlock::BLOCK_STYLE style;
   bool extraParagraphSpacing;
 
+  std::vector<size_t> computeLineBreaks(int pageWidth, int spaceWidth, const std::vector<uint16_t>& wordWidths) const;
+  void extractLine(size_t breakIndex, int pageWidth, int spaceWidth, const std::vector<uint16_t>& wordWidths,
+                   const std::vector<size_t>& lineBreakIndices,
+                   const std::function<void(std::shared_ptr<TextBlock>)>& processLine);
+  std::vector<uint16_t> calculateWordWidths(const GfxRenderer& renderer, int fontId);
+
  public:
   explicit ParsedText(const TextBlock::BLOCK_STYLE style, const bool extraParagraphSpacing)
       : style(style), extraParagraphSpacing(extraParagraphSpacing) {}
@@ -26,7 +32,9 @@ class ParsedText {
   void addWord(std::string word, EpdFontStyle fontStyle);
   void setStyle(const TextBlock::BLOCK_STYLE style) { this->style = style; }
   TextBlock::BLOCK_STYLE getStyle() const { return style; }
+  size_t size() const { return words.size(); }
   bool isEmpty() const { return words.empty(); }
   void layoutAndExtractLines(const GfxRenderer& renderer, int fontId, int horizontalMargin,
-                             const std::function<void(std::shared_ptr<TextBlock>)>& processLine);
+                             const std::function<void(std::shared_ptr<TextBlock>)>& processLine,
+                             bool includeLastLine = true);
 };

lib/Epub/Epub/Section.h

@@ -21,9 +21,10 @@ class Section {
   int currentPage = 0;
 
   explicit Section(const std::shared_ptr<Epub>& epub, const int spineIndex, GfxRenderer& renderer)
-      : epub(epub), spineIndex(spineIndex), renderer(renderer) {
-    cachePath = epub->getCachePath() + "/" + std::to_string(spineIndex);
-  }
+      : epub(epub),
+        spineIndex(spineIndex),
+        renderer(renderer),
+        cachePath(epub->getCachePath() + "/" + std::to_string(spineIndex)) {}
   ~Section() = default;
   bool loadCacheMetadata(int fontId, float lineCompression, int marginTop, int marginRight, int marginBottom,
                          int marginLeft, bool extraParagraphSpacing);

lib/Epub/Epub/parsers/ChapterHtmlSlimParser.cpp

@@ -75,6 +75,18 @@ void XMLCALL ChapterHtmlSlimParser::startElement(void* userData, const XML_Char*
     return;
   }
 
+  // Skip blocks with role="doc-pagebreak" and epub:type="pagebreak"
+  if (atts != nullptr) {
+    for (int i = 0; atts[i]; i += 2) {
+      if (strcmp(atts[i], "role") == 0 && strcmp(atts[i + 1], "doc-pagebreak") == 0 ||
+          strcmp(atts[i], "epub:type") == 0 && strcmp(atts[i + 1], "pagebreak") == 0) {
+        self->skipUntilDepth = self->depth;
+        self->depth += 1;
+        return;
+      }
+    }
+  }
+
   if (matches(name, HEADER_TAGS, NUM_HEADER_TAGS)) {
     self->startNewTextBlock(TextBlock::CENTER_ALIGN);
     self->boldUntilDepth = min(self->boldUntilDepth, self->depth);
@@ -131,6 +143,17 @@ void XMLCALL ChapterHtmlSlimParser::characterData(void* userData, const XML_Char
 
     self->partWordBuffer[self->partWordBufferIndex++] = s[i];
   }
+
+  // If we have > 750 words buffered up, perform the layout and consume out all but the last line
+  // There should be enough here to build out 1-2 full pages and doing this will free up a lot of
+  // memory.
+  // Spotted when reading Intermezzo, there are some really long text blocks in there.
+  if (self->currentTextBlock->size() > 750) {
+    Serial.printf("[%lu] [EHP] Text block too long, splitting into multiple pages\n", millis());
+    self->currentTextBlock->layoutAndExtractLines(
+        self->renderer, self->fontId, self->marginLeft + self->marginRight,
+        [self](const std::shared_ptr<TextBlock>& textBlock) { self->addLineToPage(textBlock); }, false);
+  }
 }
 
 void XMLCALL ChapterHtmlSlimParser::endElement(void* userData, const XML_Char* name) {