Merge pull request #1269 from Dasharo/tos-gate-base64-urls

Tos gate base64 urls feature with TDD and new CI workflow

.github/workflows/test-tos-gate.yml

@@ -0,0 +1,38 @@
+---
+name: ToS gate tests
+
+on:
+  push:
+    branches:
+      - master
+      - main
+    paths:
+      - main.py
+      - docs/javascripts/tos-gate.js
+      - overrides/main.html
+      - mkdocs.yml
+      - docs/**
+      - tests/test_tos_gate.py
+      - requirements.txt
+  pull_request:
+    paths:
+      - main.py
+      - docs/javascripts/tos-gate.js
+      - overrides/main.html
+      - mkdocs.yml
+      - docs/**
+      - tests/test_tos_gate.py
+      - requirements.txt
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.x"
+      - name: Install dependencies
+        run: pip install -r requirements.txt
+      - name: Run ToS gate tests
+        run: pytest tests/test_tos_gate.py -v

README.md

@@ -121,37 +121,73 @@ instead. To embed an video simply type the following in markdown:
 
 ### Embedding ToS checkbox
 
-A ToS acceptance checkbox can be added to any release page to gate binary
-downloads behind a user acknowledgement. The feature is opt-in per page.
+A ToS acceptance checkbox can be added to any page to gate binary downloads
+and/or prose content behind a user acknowledgement. The feature is opt-in per
+page.
 
-**1. Add the checkbox macro** above the content you want to protect:
+#### Gating download buttons
+
+Use the `tos_gated_downloads` macro and pass a unique `section_id` together
+with the list of files. Use the **group format** to preserve subsection
+headings matching the original release layout:
 
 ```md
-{{ tos_checkbox("unique-section-id") }}
+{{ tos_gated_downloads("vendor-board-vXYZ-binaries", [
+  {"group": "Raw firmware image", "note": "(vendor_board_vX.Y.Z.rom)", "items": [
+    {"label": "sha256",     "url": "https://dl.3mdeb.com/.../file.rom.sha256"},
+    {"label": "sha256.sig", "url": "https://dl.3mdeb.com/.../file.rom.sha256.sig"}
+  ]},
+  {"group": "SBOM CycloneDX", "items": [
+    {"label": "sbom.json",   "url": "https://dl.3mdeb.com/.../file.sbom.json"},
+    {"label": "sha256",      "url": "https://dl.3mdeb.com/.../file.sbom.json.sha256"},
+    {"label": "sha256.sig",  "url": "https://dl.3mdeb.com/.../file.sbom.json.sha256.sig"}
+  ]}
+]) }}
 ```
 
-**2. Wrap the protected content** in an HTML div with a matching `id`:
+Download links are **not** emitted into the HTML source - they are
+base64-encoded in a `data-payload` attribute and injected by `tos-gate.js` only
+after the user checks the box. This keeps URLs out of page source and prevents
+them from being indexed.
+
+Download buttons with groups example:
+[`docs/variants/gigabyte_mz33-ar1/releases.md`][tos-example]
+
+#### Optional argument (`tos_url`)
+
+Overrides the default Terms of Service link:
 
 ```md
-<div id="unique-section-id" class="tos-gate-content" markdown="1" style="display: none">
+{{ tos_gated_downloads("...", [...], tos_url="https://example.com/terms") }}
+```
 
-...binary download buttons and related content...
+#### Optional hiding prose content
+
+Pass `prose_section_id` to make the same checkbox also show/hide one or more
+blocks of descriptive text. Wrap each prose block in a div with a matching
+`data-prose-group` attribute:
+
+```md
+{{ tos_gated_downloads("vendor-board-vXYZ-blobs",
+  [{"label": "blobs.zip", "url": "https://dl.3mdeb.com/.../blobs.zip"}],
+  prose_section_id="vendor-board-vXYZ-blobs-prose") }}
+
+<div data-prose-group="vendor-board-vXYZ-blobs-prose"
+     class="tos-gate-content" markdown="1" style="display: none">
+
+Further instructions that only make sense after downloading...
 
 </div>
 ```
 
-`tos_checkbox` accepts an optional `tos_url` argument to override the default
-Terms of Service link:
+Multiple divs sharing the same `data-prose-group` value are all toggled by
+the single checkbox, so one gate can reveal several separate prose sections.
 
-```md
-{{ tos_checkbox("unique-section-id", tos_url="https://example.com/terms") }}
-```
-
-For pages with multiple release sections, use a distinct `section-id` per
-release (e.g. `vendor-board-v100-binaries`, `vendor-board-v090-binaries`). See
-[`docs/variants/gigabyte_mz33-ar1/releases.md`][tos-example] for example.
+Combined buttons + prose example:
+[`docs/variants/gigabyte_mz33-ar1/building-manual.md`][tos-prose-example]
 
 [tos-example]: docs/variants/gigabyte_mz33-ar1/releases.md
+[tos-prose-example]: docs/variants/gigabyte_mz33-ar1/building-manual.md
 
 ### Embedding subscribe forms
 

docs/javascripts/tos-gate.js

@@ -0,0 +1,28 @@
+function revealGated(cb) {
+  const el = document.getElementById(cb.dataset.section);
+  const proseEls = cb.dataset.proseSection
+    ? document.querySelectorAll(`[data-prose-group="${cb.dataset.proseSection}"]`)
+    : [];
+
+  if (!cb.checked) {
+    el.innerHTML = "";
+    proseEls.forEach(p => { p.style.display = "none"; });
+    return;
+  }
+
+  const files = JSON.parse(atob(el.dataset.payload));
+  // rel="noreferrer" is intentionally absent — it would strip the Referer
+  // header and break the MinIO bucket policy gate
+  el.innerHTML = files.map(f => {
+    if (f.group) {
+      const btns = f.items.map(item =>
+        `<a href="${item.url}" class="md-button">${item.label}</a>`
+      ).join(" ");
+      const note = f.note ? ` <span>${f.note}</span>` : "";
+      return `<h4>${f.group}</h4><p>${btns}${note}</p>`;
+    }
+    return `<a href="${f.url}" class="md-button">${f.label}</a>`;
+  }).join("\n");
+
+  proseEls.forEach(p => { p.style.display = ""; });
+}

docs/variants/gigabyte_mz33-ar1/building-manual.md

@@ -46,15 +46,15 @@ then follow the steps below:
 
 3. Download the package with necessary blobs and put it in proper directory:
 
-{{ tos_checkbox("gigabyte-mz33-ar1-v090-blobs") }}
+{{ tos_gated_downloads("gigabyte-mz33-ar1-v090-blobs",
+  [{"label": "Turin.zip", "url": "https://dl.3mdeb.com/open-source-firmware/Dasharo/gigabyte_mz33_ar1/uefi/v0.9.0/Turin.zip"}],
+  prose_section_id="gigabyte-mz33-ar1-v090-blobs-prose") }}
 
-<div id="gigabyte-mz33-ar1-v090-blobs"
+<div data-prose-group="gigabyte-mz33-ar1-v090-blobs-prose"
      class="tos-gate-content" markdown="1"
      style="display: none">
 
-- Download the [blobs
-package](https://dl.3mdeb.com/open-source-firmware/Dasharo/gigabyte_mz33_ar1/uefi/v0.9.0/Turin.zip).
-- Then unzip it to the following directory:
+- Unzip the package to the following directory:
 
     ```bash
     unzip Turin.zip -d 3rdparty/blobs/soc/amd/

main.py

@@ -1,24 +1,38 @@
+import base64
+import json
+
+
 def define_env(env):
     """ Define macros for MkDocs """
 
     @env.macro
-    def tos_checkbox(section_id, tos_url="https://www.dasharo.com/pages/terms/"):
-        """Render a ToS acceptance checkbox that reveals the section with the given id."""
+    def tos_gated_downloads(section_id, files, tos_url="https://www.dasharo.com/pages/terms/",
+                            prose_section_id=None):
+        """Render a ToS checkbox that reveals base64-encoded download links on acceptance.
+
+        files: flat list of {"label": str, "url": str} dicts, or grouped list of
+               {"group": str, "items": [...], "note": str (optional)} dicts.
+               URLs are not emitted into HTML source — base64-encoded in a data
+               attribute and injected by tos-gate.js when the checkbox is checked.
+
+        prose_section_id: when set, tos-gate.js also toggles all elements carrying
+               data-prose-group="{prose_section_id}". Multiple elements may share
+               the same value, allowing one checkbox to reveal several prose blocks.
+        """
+        payload = base64.b64encode(json.dumps(files).encode()).decode()
         checkbox_id = f"tos-cb-{section_id}"
-        onchange = (
-            f"document.getElementById('{section_id}').style.display="
-            f"this.checked?'':'none'"
-        )
+        prose_attr = f' data-prose-section="{prose_section_id}"' if prose_section_id else ""
         return (
             '<div class="tos-gate">'
             f'<label class="tos-gate__label" for="{checkbox_id}">'
-            f'<input type="checkbox" id="{checkbox_id}" '
-            f'class="tos-gate__checkbox" onchange="{onchange}"> '
+            f'<input type="checkbox" id="{checkbox_id}" class="tos-gate__checkbox" '
+            f'data-section="{section_id}"{prose_attr} onchange="revealGated(this)"> '
             'I confirm that I have read and agree to the '
             f'<a href="{tos_url}" target="_blank" rel="noopener noreferrer">'
             'Dasharo Terms of Service</a> and all applicable open-source '
             'licensing terms governing this firmware release.'
             '</label>'
+            f'<div id="{section_id}" class="tos-gate-content" data-payload="{payload}"></div>'
             '</div>'
         )
 

mkdocs.yml

@@ -35,6 +35,7 @@ extra:
 extra_css:
   - stylesheets/extra.css
 extra_javascript:
+  - javascripts/tos-gate.js
   - https://cdn.jsdelivr.net/npm/vega@5
   - https://cdn.jsdelivr.net/npm/vega-lite@5
   - https://cdn.jsdelivr.net/npm/vega-embed@6

overrides/main.html

@@ -0,0 +1,6 @@
+{% extends "base.html" %}
+
+{% block extrahead %}
+  {{ super() }}
+  <meta name="referrer" content="strict-origin-when-cross-origin">
+{% endblock %}

requirements.txt

@@ -46,3 +46,4 @@ watchdog==4.0.2
 markdown-grid-tables==0.4.0
 mkdocs-macros-plugin==1.3.7
 mkdocs-include-markdown-plugin==7.1.5
+pytest==8.3.2

tests/test_tos_gate.py

@@ -0,0 +1,206 @@
+"""
+Red/green tests for the ToS gate feature (Tasks 0 & 1).
+
+Covers:
+  - tos_gated_downloads macro unit tests
+  - tos-gate.js static assertions
+  - Markdown source files: no plain dl.3mdeb.com links in gated sections
+  - mkdocs.yml / overrides configuration
+"""
+import base64
+import json
+import re
+import sys
+from pathlib import Path
+
+import pytest
+
+REPO_ROOT = Path(__file__).parent.parent
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _load_macros():
+    """Collect macro functions from main.py by simulating define_env."""
+    sys.path.insert(0, str(REPO_ROOT))
+    import main  # noqa: PLC0415
+
+    macros: dict = {}
+
+    class _Env:
+        def macro(self, fn):
+            macros[fn.__name__] = fn
+            return fn
+
+    main.define_env(_Env())
+    return macros
+
+
+@pytest.fixture(scope="session")
+def macros():
+    return _load_macros()
+
+
+@pytest.fixture(scope="session")
+def tos_gate_js():
+    return (REPO_ROOT / "docs/javascripts/tos-gate.js").read_text()
+
+
+# ---------------------------------------------------------------------------
+# tos_gated_downloads – unit tests
+# ---------------------------------------------------------------------------
+
+class TestTosGatedDownloads:
+    def test_url_not_in_html_source(self, macros):
+        """URLs must be absent from rendered HTML — only encoded in data-payload."""
+        url = "https://dl.3mdeb.com/secret/firmware.zip"
+        html = macros["tos_gated_downloads"]("s", [{"label": "f", "url": url}])
+        assert url not in html
+
+    def test_payload_is_valid_base64_json(self, macros):
+        files = [{"label": "sha256", "url": "https://dl.3mdeb.com/file.sha256"}]
+        html = macros["tos_gated_downloads"]("s", files)
+        m = re.search(r'data-payload="([^"]+)"', html)
+        assert m, "data-payload attribute missing from output"
+        decoded = json.loads(base64.b64decode(m.group(1)))
+        assert decoded == files
+
+    def test_group_format_encodes_correctly(self, macros):
+        files = [{"group": "Raw image", "note": "(fw.rom)", "items": [
+            {"label": "sha256", "url": "https://dl.3mdeb.com/fw.sha256"},
+        ]}]
+        html = macros["tos_gated_downloads"]("s", files)
+        assert "dl.3mdeb.com" not in html
+        m = re.search(r'data-payload="([^"]+)"', html)
+        decoded = json.loads(base64.b64decode(m.group(1)))
+        assert decoded[0]["group"] == "Raw image"
+        assert decoded[0]["note"] == "(fw.rom)"
+        assert decoded[0]["items"][0]["url"] == "https://dl.3mdeb.com/fw.sha256"
+
+    def test_checkbox_required_attributes(self, macros):
+        html = macros["tos_gated_downloads"]("my-section", [])
+        assert 'data-section="my-section"' in html
+        assert 'onchange="revealGated(this)"' in html
+        assert 'class="tos-gate__checkbox"' in html
+
+    def test_section_content_div_present(self, macros):
+        html = macros["tos_gated_downloads"]("my-section", [])
+        assert 'id="my-section"' in html
+        assert "tos-gate-content" in html
+
+    def test_prose_section_id_emits_data_attribute(self, macros):
+        html = macros["tos_gated_downloads"]("s", [], prose_section_id="my-prose")
+        assert 'data-prose-section="my-prose"' in html
+
+    def test_no_prose_section_id_omits_data_attribute(self, macros):
+        html = macros["tos_gated_downloads"]("s", [])
+        assert "data-prose-section" not in html
+
+    def test_default_tos_url(self, macros):
+        html = macros["tos_gated_downloads"]("s", [])
+        assert 'href="https://www.dasharo.com/pages/terms/"' in html
+
+    def test_custom_tos_url_override(self, macros):
+        html = macros["tos_gated_downloads"]("s", [], tos_url="https://example.com/tos")
+        assert 'href="https://example.com/tos"' in html
+        assert "dasharo.com/pages/terms" not in html
+
+
+# ---------------------------------------------------------------------------
+# tos-gate.js – static assertions
+# ---------------------------------------------------------------------------
+
+class TestTosGateJs:
+    def test_file_exists(self):
+        assert (REPO_ROOT / "docs/javascripts/tos-gate.js").exists()
+
+    def test_reveal_gated_function_defined(self, tos_gate_js):
+        assert "function revealGated" in tos_gate_js
+
+    def test_noreferrer_absent_from_link_template(self, tos_gate_js):
+        """noreferrer on injected download links would strip the Referer header,
+        breaking the MinIO bucket policy gate."""
+        for line in tos_gate_js.splitlines():
+            if "md-button" in line:
+                assert "noreferrer" not in line, (
+                    f"noreferrer must not appear in the injected link template: {line!r}"
+                )
+
+    def test_noreferrer_warning_comment_present(self, tos_gate_js):
+        """The intentional absence of noreferrer should be documented."""
+        assert "noreferrer" in tos_gate_js
+
+    def test_prose_group_queryselector_present(self, tos_gate_js):
+        """Multiple prose divs must be toggled via data-prose-group selector."""
+        assert "data-prose-group" in tos_gate_js
+
+    def test_group_rendering_branch_present(self, tos_gate_js):
+        assert "f.group" in tos_gate_js
+
+    def test_prose_elements_toggled_on_uncheck(self, tos_gate_js):
+        """Unchecking must hide prose sections, not just clear download buttons."""
+        assert 'display = "none"' in tos_gate_js or "display='none'" in tos_gate_js or \
+               'style.display = "none"' in tos_gate_js
+
+
+# ---------------------------------------------------------------------------
+# Static file tests – variant markdown files (parameterized, opt-in)
+#
+# Tests run against every *.md under docs/variants/.
+# Files that do not use tos_gated_downloads are skipped automatically,
+# so adding the gate to a new variant is covered without touching this file.
+# ---------------------------------------------------------------------------
+
+_VARIANT_MD_FILES = sorted((REPO_ROOT / "docs/variants").rglob("*.md"))
+_VARIANT_IDS = [str(f.relative_to(REPO_ROOT)) for f in _VARIANT_MD_FILES]
+
+
+@pytest.mark.parametrize("md_file", _VARIANT_MD_FILES, ids=_VARIANT_IDS)
+def test_no_plain_dl_links_in_gated_file(md_file):
+    """Any file using tos_gated_downloads must not expose dl.3mdeb.com URLs
+    as plain markdown links (inline or reference-style)."""
+    content = md_file.read_text()
+    if "tos_gated_downloads" not in content:
+        pytest.skip("file does not use tos_gated_downloads")
+    assert not re.search(r"^\[.*\]:\s+https://dl\.3mdeb\.com/", content, re.MULTILINE), \
+        f"Reference-style link to dl.3mdeb.com found in {md_file.name}"
+    assert not re.search(r"\]\(https://dl\.3mdeb\.com/", content), \
+        f"Inline markdown link to dl.3mdeb.com found in {md_file.name}"
+
+
+@pytest.mark.parametrize("md_file", _VARIANT_MD_FILES, ids=_VARIANT_IDS)
+def test_prose_section_id_has_matching_div(md_file):
+    """Any file using prose_section_id must have at least one data-prose-group div."""
+    content = md_file.read_text()
+    if "prose_section_id" not in content:
+        pytest.skip("file does not use prose_section_id")
+    assert "data-prose-group" in content, \
+        f"prose_section_id used but no data-prose-group div found in {md_file.name}"
+
+
+def test_at_least_one_variant_uses_tos_gated_downloads():
+    """Sanity check: ensure the feature is actually in use and hasn't been
+    accidentally removed from all variant files."""
+    gated = [f for f in _VARIANT_MD_FILES if "tos_gated_downloads" in f.read_text()]
+    assert gated, "No variant file uses tos_gated_downloads — feature may have been removed"
+
+
+# ---------------------------------------------------------------------------
+# Static file tests – infrastructure configuration
+# ---------------------------------------------------------------------------
+
+class TestInfrastructure:
+    def test_mkdocs_yml_includes_tos_gate_js(self):
+        content = (REPO_ROOT / "mkdocs.yml").read_text()
+        assert "javascripts/tos-gate.js" in content
+
+    def test_mkdocs_yml_permalink_enabled(self):
+        content = (REPO_ROOT / "mkdocs.yml").read_text()
+        assert "permalink: true" in content
+
+    def test_overrides_main_html_referrer_meta(self):
+        content = (REPO_ROOT / "overrides/main.html").read_text()
+        assert 'name="referrer"' in content
+        assert "strict-origin-when-cross-origin" in content