Add EPUB optimizer for CrossPoint Reader devices

Port the client-side EPUB optimizer from CrossPoint web server to Python. Implements device-specific image optimization (auto-crop, scaling, grayscale conversion, JPEG re-encoding) and EPUB container rewriting (SVG unwrapping, metadata fixes, proper ZIP structure). Supports X3 and X4 device profiles with configurable JPEG quality. Uses Qt's QImage and numpy for image processing with graceful degradation when numpy is unavailable.

README.md

@@ -8,7 +8,9 @@ Calibre plugins for [CrossPoint Reader](https://github.com/crosspoint-reader).
 
 A wireless device plugin that uploads EPUB files to CrossPoint Reader over WebSocket. The plugin auto-discovers devices on the local network via UDP broadcast.
 
-See [crosspoint_reader/README.md](crosspoint_reader/README.md) for protocol details and configuration.
+It can also **optimize EPUBs before transfer** — mirroring the optimizer built into the CrossPoint web server — by resizing images to the device screen (X4 480×800 / X3 528×792, auto-detected), converting them to grayscale JPEG, and rewriting the container. Enable it in the plugin's settings; see the [usage instructions](crosspoint_reader/README.md#using-the-optimizer).
+
+See [crosspoint_reader/README.md](crosspoint_reader/README.md) for protocol details, the optimizer, and configuration.
 
 ## Installation
 

crosspoint_reader/README.md

@@ -16,6 +16,53 @@ Default settings:
 - Port: 81
 - Upload path: /
 
+## Optimizer
+
+The plugin can optimize EPUBs before transfer, mirroring the optimizer built into
+the CrossPoint web server. When enabled (Preferences > Plugins > device config >
+"Optimize EPUBs before transfer"), each EPUB is processed before upload:
+
+- Every image is scaled to fit the device screen, converted to grayscale, and
+  re-encoded as JPEG (quality configurable, default 85). Optional auto-crop trims
+  uniform page margins.
+- The container is rewritten: raster images are renamed to `.jpg`, stale `<img>`
+  width/height are stripped, SVG covers/wrapped images are unwrapped, OPF
+  media-types and cover meta are fixed, the NCX identifier is synced, a small
+  defensive stylesheet is injected, and the archive is re-zipped mimetype-first.
+
+The target screen size comes from the device profile — **X4 = 480×800**,
+**X3 = 528×792** — which is auto-detected from the device's `/api/status`
+endpoint on connect (matching the web UI), or can be set manually
+(Auto / X4 / X3) in the plugin settings.
+
+After each transfer a summary dialog lists what changed per book (before→after
+size, images processed, fixes, per-image steps). If optimization fails for a
+book, the original is sent unchanged so a transfer is never blocked.
+
+### Using the optimizer
+
+1. Open **Preferences → Plugins**, expand **Device Interface plugins**, select
+   **CrossPoint Reader**, and click **Customize plugin**.
+2. Check **Optimize EPUBs before transfer**.
+3. Set the options below it (they enable once the box is checked):
+   - **Device target** — leave on **Auto-detect** to read X3/X4 from the device
+     on connect, or force **X4** / **X3**. Auto-detect falls back to X4 if the
+     device can't be queried.
+   - **JPEG quality** — 1–100 (default 85). Lower = smaller files.
+   - **Convert images to grayscale** — on by default (recommended for e-ink).
+   - **Auto-crop uniform margins** — off by default; trims solid page borders.
+4. Click **OK**, then **restart Calibre** if it was already running so the new
+   settings take effect.
+5. Send a book to the device as usual (right-click → *Send to device*, or the
+   **Send to device** toolbar button). The EPUB is optimized just before upload.
+6. When the transfer finishes, a **summary dialog** opens showing, per book, the
+   before→after size, how many images were processed/cropped, and the fixes
+   applied. The same steps are also written to the plugin **Log** (visible in the
+   config dialog when **Enable debug logging** is on).
+
+To turn the feature off, uncheck **Optimize EPUBs before transfer** — books are
+then sent exactly as Calibre exports them.
+
 Install:
 1. Download the latest release from the [releases page](https://github.com/crosspoint-reader/calibre-plugins/releases) (or zip the contents of this directory).
 2. In Calibre: Preferences > Plugins > Load plugin from file.

crosspoint_reader/__init__.py

@@ -3,3 +3,12 @@ from .driver import CrossPointDevice
 
 class CrossPointReaderDevice(CrossPointDevice):
     pass
+
+
+# Create the optimization-summary bridge on the main thread at plugin load, so
+# that the post-transfer dialog can be shown safely from the device thread.
+try:
+    from . import summary as _summary
+    _summary.ensure_bridge()
+except Exception:
+    pass

crosspoint_reader/config.py

@@ -1,9 +1,11 @@
 from calibre.utils.config import JSONConfig
 from qt.core import (
     QCheckBox,
+    QComboBox,
     QDialog,
     QDialogButtonBox,
     QFormLayout,
+    QFrame,
     QHBoxLayout,
     QLabel,
     QLineEdit,
@@ -25,6 +27,12 @@ PREFS.defaults['chunk_size'] = 2048
 PREFS.defaults['debug'] = False
 PREFS.defaults['fetch_metadata'] = False
 PREFS.defaults['send_to_root'] = False
+# Optimizer settings (mirrors the CrossPoint web server optimizer).
+PREFS.defaults['optimize'] = False
+PREFS.defaults['optimize_grayscale'] = True
+PREFS.defaults['optimize_auto_crop'] = False
+PREFS.defaults['optimize_quality'] = 85
+PREFS.defaults['device_target'] = 'auto'  # 'auto' | 'X4' | 'X3'
 
 
 class CrossPointConfigWidget(QWidget):
@@ -41,6 +49,18 @@ class CrossPointConfigWidget(QWidget):
         self.fetch_metadata = QCheckBox('Fetch metadata (slower device list)', self)
         self.send_to_root = QCheckBox('Send to root (ignore folder template)', self)
 
+        # Optimizer controls.
+        self.optimize = QCheckBox('Optimize EPUBs before transfer', self)
+        self.optimize_grayscale = QCheckBox('Convert images to grayscale', self)
+        self.optimize_auto_crop = QCheckBox('Auto-crop uniform margins', self)
+        self.optimize_quality = QSpinBox(self)
+        self.optimize_quality.setRange(1, 100)
+        self.optimize_quality.setSuffix('%')
+        self.device_target = QComboBox(self)
+        self.device_target.addItem('Auto-detect', 'auto')
+        self.device_target.addItem('X4 (480×800)', 'X4')
+        self.device_target.addItem('X3 (528×792)', 'X3')
+
         self.host.setText(PREFS['host'])
         self.port.setValue(PREFS['port'])
         self.path.setText(PREFS['path'])
@@ -48,6 +68,12 @@ class CrossPointConfigWidget(QWidget):
         self.debug.setChecked(PREFS['debug'])
         self.fetch_metadata.setChecked(PREFS['fetch_metadata'])
         self.send_to_root.setChecked(PREFS['send_to_root'])
+        self.optimize.setChecked(PREFS['optimize'])
+        self.optimize_grayscale.setChecked(PREFS['optimize_grayscale'])
+        self.optimize_auto_crop.setChecked(PREFS['optimize_auto_crop'])
+        self.optimize_quality.setValue(PREFS['optimize_quality'])
+        idx = self.device_target.findData(PREFS['device_target'])
+        self.device_target.setCurrentIndex(idx if idx >= 0 else 0)
 
         layout.addRow('Host', self.host)
         layout.addRow('Port', self.port)
@@ -63,6 +89,28 @@ class CrossPointConfigWidget(QWidget):
         layout.addRow('', self.fetch_metadata)
         layout.addRow('', self.send_to_root)
 
+        sep = QFrame(self)
+        sep.setFrameShape(QFrame.Shape.HLine)
+        sep.setFrameShadow(QFrame.Shadow.Sunken)
+        layout.addRow(sep)
+
+        opt_heading = QLabel('<b>Optimizer</b>')
+        layout.addRow(opt_heading)
+        opt_notice = QLabel('Mirrors the CrossPoint web optimizer: resizes images to the '
+                            'screen, converts to grayscale and re-encodes as JPEG, then '
+                            'rewrites the EPUB. A summary is shown after each transfer.')
+        opt_notice.setWordWrap(True)
+        opt_notice.setStyleSheet('color: gray; font-style: italic;')
+        layout.addRow('', opt_notice)
+        layout.addRow('', self.optimize)
+        layout.addRow('Device target', self.device_target)
+        layout.addRow('JPEG quality', self.optimize_quality)
+        layout.addRow('', self.optimize_grayscale)
+        layout.addRow('', self.optimize_auto_crop)
+
+        self.optimize.toggled.connect(self._sync_optimizer_enabled)
+        self._sync_optimizer_enabled(self.optimize.isChecked())
+
         self.log_view = QPlainTextEdit(self)
         self.log_view.setReadOnly(True)
         self.log_view.setPlaceholderText('Discovery log will appear here when debug is enabled.')
@@ -84,6 +132,16 @@ class CrossPointConfigWidget(QWidget):
         PREFS['debug'] = bool(self.debug.isChecked())
         PREFS['fetch_metadata'] = bool(self.fetch_metadata.isChecked())
         PREFS['send_to_root'] = bool(self.send_to_root.isChecked())
+        PREFS['optimize'] = bool(self.optimize.isChecked())
+        PREFS['optimize_grayscale'] = bool(self.optimize_grayscale.isChecked())
+        PREFS['optimize_auto_crop'] = bool(self.optimize_auto_crop.isChecked())
+        PREFS['optimize_quality'] = int(self.optimize_quality.value())
+        PREFS['device_target'] = self.device_target.currentData()
+
+    def _sync_optimizer_enabled(self, enabled):
+        for w in (self.optimize_grayscale, self.optimize_auto_crop,
+                  self.optimize_quality, self.device_target):
+            w.setEnabled(enabled)
 
     def _refresh_logs(self):
         self.log_view.setPlainText(get_log_text())

crosspoint_reader/driver.py

@@ -21,7 +21,7 @@ class CrossPointDevice(DeviceConfig, DevicePlugin):
     description = 'CrossPoint Reader wireless device'
     supported_platforms = ['windows', 'osx', 'linux']
     author = 'CrossPoint Reader'
-    version = (0, 1, 4)
+    version = (0, 2, 0)
 
     # Invalid USB vendor info to avoid USB scans matching.
     VENDOR_ID = [0xFFFF]
@@ -44,6 +44,7 @@ class CrossPointDevice(DeviceConfig, DevicePlugin):
         self.is_connected = False
         self.device_host = None
         self.device_port = None
+        self.device_model = None  # 'X3' | 'X4' from /api/status
         self.last_discovery = 0.0
         self.report_progress = lambda x, y: x
         self._debug_enabled = False
@@ -86,11 +87,26 @@ class CrossPointDevice(DeviceConfig, DevicePlugin):
             self.device_host = host
             self.device_port = port
             self.is_connected = True
+            self._detect_device_model()
             return self
         if debug:
             self._log('[CrossPoint] discovery failed')
         return None
 
+    def _detect_device_model(self):
+        """Query /api/status for the device model (X3/X4), like the web UI."""
+        try:
+            status = self._http_get_json('/api/status', timeout=4)
+            model = (status or {}).get('device')
+            if model in ('X3', 'X4'):
+                self.device_model = model
+                self._log(f'[CrossPoint] detected device model: {model}')
+            else:
+                self._log('[CrossPoint] /api/status returned no device model')
+        except Exception as exc:
+            self._log(f'[CrossPoint] device model detection failed: {exc}')
+        return self.device_model
+
     def open(self, connected_device, library_uuid):
         if not self.is_connected:
             raise ControlError(desc='Attempt to open a closed device')
@@ -145,6 +161,12 @@ class CrossPointDevice(DeviceConfig, DevicePlugin):
             raise ControlError(desc=f'HTTP request failed: {exc}')
 
     def config_widget(self):
+        # Runs on the GUI thread; ensure the summary bridge exists (idempotent).
+        try:
+            from . import summary as summary_ui
+            summary_ui.ensure_bridge()
+        except Exception:
+            pass
         return CrossPointConfigWidget()
 
     def save_settings(self, config_widget):
@@ -293,7 +315,23 @@ class CrossPointDevice(DeviceConfig, DevicePlugin):
         if base_path != '/' and base_path.endswith('/'):
             base_path = base_path[:-1]
 
+        optimize_enabled = bool(PREFS['optimize'])
+        opt_profile = None
+        summary_ui = None
+        if optimize_enabled:
+            from .optimizer import resolve_profile
+            _, opt_profile = resolve_profile(PREFS['device_target'], self.device_model)
+            # Open the live optimizer dialog up-front so the user sees steps stream.
+            try:
+                from . import summary as summary_ui
+                summary_ui.begin('Optimizing %d book(s) for %s…' % (
+                    len(files), opt_profile['label']))
+            except Exception as exc:
+                summary_ui = None
+                self._log(f'[CrossPoint] could not open optimizer dialog: {exc}')
+
         paths = []
+        summaries = []
         total = len(files)
         for i, (infile, name) in enumerate(zip(files, names)):
             if hasattr(infile, 'read'):
@@ -317,27 +355,99 @@ class CrossPointDevice(DeviceConfig, DevicePlugin):
             else:
                 lpath = target_dir + '/' + filename
 
+            # Optionally optimize the EPUB to a temp file before uploading.
+            send_path = filepath
+            opt_temp = None
+            if optimize_enabled and filepath.lower().endswith('.epub'):
+                step_cb = summary_ui.step if summary_ui is not None else None
+                opt_temp, summary = self._optimize_book(filepath, opt_profile,
+                                                        step_cb=step_cb)
+                if opt_temp is not None:
+                    send_path = opt_temp
+                if summary is not None:
+                    summaries.append(summary)
+                if summary_ui is not None:
+                    summary_ui.step('SEND', 'Uploading %s …' % filename)
+
             def _progress(sent, size):
                 if size > 0:
                     self.report_progress((i + sent / float(size)) / float(total),
                                          'Transferring books to device...')
 
-            ws_client.upload_file(
-                host,
-                port,
-                target_dir,
-                filename,
-                filepath,
-                chunk_size=chunk_size,
-                debug=debug,
-                progress_cb=_progress,
-                logger=self._log,
-            )
-            paths.append((lpath, os.path.getsize(filepath)))
+            try:
+                ws_client.upload_file(
+                    host,
+                    port,
+                    target_dir,
+                    filename,
+                    send_path,
+                    chunk_size=chunk_size,
+                    debug=debug,
+                    progress_cb=_progress,
+                    logger=self._log,
+                )
+                paths.append((lpath, os.path.getsize(send_path)))
+            finally:
+                if opt_temp is not None:
+                    try:
+                        os.remove(opt_temp)
+                    except OSError:
+                        pass
 
         self.report_progress(1.0, 'Transferring books to device...')
+
+        if summary_ui is not None:
+            try:
+                summary_ui.finish({
+                    'profile': opt_profile['label'] if opt_profile else '?',
+                    'books': summaries,
+                })
+            except Exception as exc:
+                self._log(f'[CrossPoint] could not finalize optimizer dialog: {exc}')
+
         return paths
 
+    def _optimize_book(self, filepath, profile, step_cb=None):
+        """Optimize an EPUB to a temp file. Returns (temp_path_or_None, summary_or_None).
+
+        ``step_cb(tag, message)`` (optional) streams each step to the live dialog.
+        On any failure the original file is used (temp_path is None) so a transfer
+        is never blocked by optimization.
+        """
+        from calibre.ptempfile import PersistentTemporaryFile
+        from .optimizer import optimize_epub, Options
+
+        opts = Options(
+            quality=PREFS['optimize_quality'],
+            grayscale=PREFS['optimize_grayscale'],
+            auto_crop=PREFS['optimize_auto_crop'],
+        )
+
+        def _step(tag, message):
+            self._log(f'[CrossPoint][opt] {tag}: {message}')
+            if step_cb is not None:
+                try:
+                    step_cb(tag, message)
+                except Exception:
+                    pass
+
+        out_path = None
+        try:
+            tf = PersistentTemporaryFile(suffix='.epub')
+            out_path = tf.name
+            tf.close()
+            summary = optimize_epub(filepath, out_path, profile, opts, log_fn=_step)
+            return out_path, summary
+        except Exception as exc:
+            self._log(f'[CrossPoint] optimization failed for {os.path.basename(filepath)}: '
+                      f'{exc} (sending original)')
+            if out_path:
+                try:
+                    os.remove(out_path)
+                except OSError:
+                    pass
+            return None, None
+
     def add_books_to_metadata(self, locations, metadata, booklists):
         self._log(f'[CrossPoint] add_books_to_metadata: {len(locations)} locations, '
                   f'{len(booklists)} booklists')

crosspoint_reader/summary.py

@@ -0,0 +1,189 @@
+"""Live optimization progress dialog (shown during transfer).
+
+The device driver's ``upload_books`` runs on Calibre's device-manager thread, not
+the GUI thread, and Qt widgets may only be created/shown from the GUI thread.
+
+Cross-thread delivery only works reliably when the receiving QObject was *created
+on the GUI thread* (moving a worker-thread object to the GUI thread does not make
+queued delivery work). So the bridge is created once, eagerly, at plugin import
+time — which Calibre performs on the main thread, with the QApplication already
+running — and the worker thread merely emits queued signals to it:
+
+    begin(title)        -> open the dialog and show "working"
+    step(tag, message)  -> append a line as each image/fix is processed
+    finish(payload)     -> append the totals and enable the Close button
+
+The dialog is modeless, so it appears as soon as optimization starts and updates
+live while books are processed and uploaded.
+"""
+
+from qt.core import QObject, QApplication, QThread, Qt, pyqtSignal
+
+from .log import add_log
+from .optimizer import _human
+
+
+def _summary_lines(payload):
+    books = payload.get('books', [])
+    total_orig = sum(b['orig_size'] for b in books)
+    total_new = sum(b['new_size'] for b in books)
+    total_imgs = sum(b['images'] for b in books)
+    total_fixes = sum(b['fixes'] for b in books)
+    total_err = sum(b['errors'] for b in books)
+    saved = total_orig - total_new
+    pct = (saved / float(total_orig) * 100.0) if total_orig else 0.0
+    lines = [
+        '',
+        '──────────────────────────────',
+        'Done: %d book(s), %d image(s), %d fix(es)%s' % (
+            len(books), total_imgs, total_fixes,
+            ('   %d error(s)' % total_err) if total_err else ''),
+        'Total size: %s → %s  (%+.0f%%)' % (
+            _human(total_orig), _human(total_new), -pct),
+    ]
+    return lines
+
+
+class _ProgressDialog(object):
+    """Wraps the Qt widgets; all methods run on the GUI thread."""
+
+    def __init__(self, title):
+        from qt.core import (
+            QDialog, QVBoxLayout, QPlainTextEdit, QLabel, QProgressBar,
+            QDialogButtonBox,
+        )
+        try:
+            from calibre.gui2.ui import get_gui
+            parent = get_gui()
+        except Exception:
+            parent = None
+
+        self.dlg = QDialog(parent)
+        self.dlg.setWindowTitle('CrossPoint optimizer')
+        self.dlg.setModal(False)
+        self.dlg.resize(660, 480)
+        layout = QVBoxLayout(self.dlg)
+
+        self.header = QLabel(title)
+        self.header.setWordWrap(True)
+        layout.addWidget(self.header)
+
+        self.bar = QProgressBar(self.dlg)
+        self.bar.setRange(0, 0)  # indeterminate until finished
+        layout.addWidget(self.bar)
+
+        self.view = QPlainTextEdit(self.dlg)
+        self.view.setReadOnly(True)
+        self.view.setLineWrapMode(QPlainTextEdit.LineWrapMode.NoWrap)
+        try:
+            from qt.core import QFontDatabase
+            self.view.setFont(
+                QFontDatabase.systemFont(QFontDatabase.SystemFont.FixedFont))
+        except Exception:
+            pass
+        layout.addWidget(self.view)
+
+        self.buttons = QDialogButtonBox(QDialogButtonBox.StandardButton.Close)
+        self.buttons.rejected.connect(self.dlg.reject)
+        self.buttons.accepted.connect(self.dlg.accept)
+        self.buttons.button(QDialogButtonBox.StandardButton.Close).setEnabled(False)
+        layout.addWidget(self.buttons)
+
+    def show(self):
+        self.dlg.show()
+        self.dlg.raise_()
+        self.dlg.activateWindow()
+
+    def append(self, line):
+        self.view.appendPlainText(line)
+        sb = self.view.verticalScrollBar()
+        sb.setValue(sb.maximum())
+
+    def finish(self, payload):
+        from qt.core import QDialogButtonBox
+        for line in _summary_lines(payload):
+            self.view.appendPlainText(line)
+        self.bar.setRange(0, 1)
+        self.bar.setValue(1)
+        self.header.setText('Optimization complete.')
+        self.buttons.button(QDialogButtonBox.StandardButton.Close).setEnabled(True)
+        self.dlg.raise_()
+
+
+class _Bridge(QObject):
+    """Lives on the GUI thread; drives the dialog when signalled from any thread."""
+
+    begin_signal = pyqtSignal(object)
+    step_signal = pyqtSignal(object)
+    finish_signal = pyqtSignal(object)
+
+    def __init__(self):
+        super().__init__()
+        q = Qt.ConnectionType.QueuedConnection
+        self.begin_signal.connect(self._on_begin, type=q)
+        self.step_signal.connect(self._on_step, type=q)
+        self.finish_signal.connect(self._on_finish, type=q)
+        self._dialog = None
+
+    def _on_begin(self, title):
+        try:
+            self._dialog = _ProgressDialog(title)
+            self._dialog.show()
+        except Exception as exc:
+            self._dialog = None
+            add_log(f'[CrossPoint] failed to open optimizer dialog: {exc}')
+
+    def _on_step(self, line):
+        if self._dialog is not None:
+            try:
+                self._dialog.append(line)
+            except Exception:
+                pass
+
+    def _on_finish(self, payload):
+        if self._dialog is not None:
+            try:
+                self._dialog.finish(payload)
+            except Exception as exc:
+                add_log(f'[CrossPoint] optimizer dialog finish failed: {exc}')
+
+
+_bridge = None
+
+
+def ensure_bridge():
+    """Create the GUI-thread bridge. Must be called on the GUI (main) thread.
+
+    Called at plugin import (startup, main thread) and again from config_widget()
+    as a belt-and-suspenders. No-op off the main thread or without a QApplication.
+    """
+    global _bridge
+    if _bridge is not None:
+        return _bridge
+    app = QApplication.instance()
+    if app is None:
+        return None
+    if QThread.currentThread() != app.thread():
+        return None  # cannot safely create a QObject for the GUI thread here
+    _bridge = _Bridge()
+    return _bridge
+
+
+# --- API called from the device-manager (worker) thread ---------------------
+
+def begin(title):
+    b = _bridge or ensure_bridge()
+    if b is not None:
+        b.begin_signal.emit(title)
+
+
+def step(tag, message):
+    b = _bridge
+    if b is not None:
+        b.step_signal.emit('[%s] %s' % (tag, message))
+
+
+def finish(payload):
+    b = _bridge
+    if b is not None:
+        b.finish_signal.emit(payload)