qdl cannot be packaged for EPEL 9, because RHEL 9 ships meson 0.63.3
and the build stops at the version check:
meson.build:1:0: ERROR: Meson version is 0.63.3 but project
requires >=1.1.0
The only thing in the build actually requiring meson 1.1 is the
meson.options file name. The build rules themselves need nothing newer
than project_source_root() and project_build_root() from 0.56. Rename
the options file to the older meson_options.txt spelling, which every
meson version understands, and lower the floor to what the rules
require.
Verified by building with meson 0.56.2 (the new floor), 0.63.3 (EL9)
and 1.10.1, each with the zip container both enabled and disabled.
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
_FILE_OFFSET_BITS was defined in five source files individually,
but off_t appears in shared types - struct firehose_op carries an
off_t sparse_offset, and qdl_file_seek() takes an off_t - so files
that did not define it (file.c, read.c, patch.c, qdl.c) disagreed
on the width of off_t on 32-bit builds, giving mismatched struct
layouts and a corrupt ABI between translation units. Define it once
via add_project_arguments() so every unit agrees, and drop the
per-file definitions.
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
qdl unconditionally links against libzip, although zip archives are
just one of the supported input formats: both the flash-from-zip path
and the create-zip subcommand are conveniences on top of the plain
contents.xml / flashmap.json flows.
That unconditional dependency gets in the way of integrating qdl into
minimal embedded distributions. In Yocto, for example, libzip lives in
meta-openembedded rather than oe-core, so packaging qdl means pulling
in a whole extra layer for a feature many products never use.
Introduce a zip-container feature option. Rather than switching code
paths with preprocessor conditionals, select whole files in the build
system: the libzip implementation moves out of file.c into file_zip.c,
which is compiled together with zipper.c (the create-zip backend) only
when the feature is enabled. Builds without it compile file_nozip.c
instead, which provides the same entry points as inert stubs, so the
headers declare a single unconditional interface and file.c keeps just
the plain POSIX file handling.
The stubbed qdl_zip_open() reports "not a zip archive" for every
input, which keeps the contents.xml and flashmap.json flows working
unchanged, while create-zip and a failed flash of a zip archive report
that this qdl was built without zip-container support. As real symbols
in a single compilation unit, the stubs also remain replaceable by the
mock definitions the unit tests link in.
The feature defaults to enabled rather than auto, so a missing libzip
is a hard configure error instead of a silently skipped dependency:
distro builds cannot accidentally lose zip support, and producing a
leaner qdl requires an explicit -Dzip-container=disabled opt-out.
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Inspecting or editing a device's flash previously meant dumping partitions
to files and reflashing them; there was no way to treat the live storage as
a random-access block device. Exposing a LUN over nbdkit lets ordinary host
tools - parted, mount, dd, imaging tools - operate on it directly.
Add an nbdkit plugin that uploads the firehose programmer and serves a
physical partition (LUN) as a read/write block device:
$ modprobe nbd
$ nbdkit -fv ./nbdkit-qdl-plugin.so programmer=prog_firehose_ddr.elf lun=0
meson builds it only when the nbdkit dependency is present (pass
-Dnbdkit=enabled to require it), producing nbdkit-qdl-plugin.so from the
shared qdl sources; the qdl binary is unaffected.
An EDL device can only be programmed once per session - uploading the
programmer is a one-shot transition and a reset reboots it out of firehose
- but nbdkit calls .open/.close several times per client (NBD_OPT_INFO to
probe, then NBD_OPT_GO to transfer). Programming per open and resetting per
close therefore made the first close reboot the board and every later open
time out. To solve this the plugin sets the device up once, on the first
open, keeps it configured across connections, and resets it only when the
plugin unloads.
Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org>
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
With kickstart now available as `qdl ks`, the standalone qdl-ks binary
is the same redundancy qdl-ramdump was: a second entry point, meson
target and manpage for code the main binary already carries. Removing
it completes the consolidation onto a single qdl binary.
Drop the binary and its build/doc scaffolding, and rewrite the README
kickstart section to use `qdl ks`. The description is kept, not
removed -- the feature lives on, only its entry point changed.
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
qdl-ramdump exposed crash-dump collection as its own binary, but the
exact same code is already reachable as `qdl ramdump`. Keeping a second
entry point for it costs a meson target, a manpage and README real
estate, and gives users a second name to learn, all for functionality
the main binary already provides. Consolidating on a single qdl binary
(as suggested in review) removes that duplication.
Drop the standalone binary and everything that only existed to build
and document it; the `qdl ramdump` subcommand is unchanged.
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Every source change funnells through the single ~200-line
root meson.build that enumerated the sources for all binaries,
so touching one area means editing a file that mostly described
unrelated code.
Declare the source lists in src/meson.build and the tests in
tests/meson.build and pull them in with subdir(). The root meson.build
is then left to compose (dependencies, version header, targets) while
the list of files to build lives next to the files themselves, so
adding or removing a source is a one-line edit in one obvious place.
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
The build system determines the version from the output of
`git describe` then allowing it to be overwritten from the `VERSION`
meson configuration option.
This is the wrong way around for build environments where the git
binary (or even the upstream VCS) is unavailable, such as Debian where
packages are built from source tarballs.
To allow build environments without git or VCS metadata to inject their
own version, or even just build without needing git, rework the order
of the version determinism logic in the build system:
1. Use the value of the `VERSION` meson configuration option (if set).
2. Use the output of `git describe` (if the git binary is found).
3. Fallback to `unknown-version` in all other cases.
This avoids calling a potentially missing git binary and still lets the
build system pick the version up from the VERSION option when a git
checkout may not be possible.
Signed-off-by: Christopher Obbard <obbardc@debian.org>
Add a meta-backend QDL_DEVICE_AUTO that defers the transport choice
to qdl_open(): instead of picking libusb or QUD upfront, its open()
runs a unified wait loop and binds whichever transport first reaches
an EDL device. The "auto" --backend token (and the default) maps to
it.
Each 250 ms tick the loop does:
1. usb_open_attempt() - one libusb scan-and-open. On success the
libusb device is bound as the inner transport.
2. On Windows, if libusb saw the device but could not open it
(-EBUSY: the QDLoader 9008 driver has claimed the interface)
or SetupAPI sees a Qualcomm COM port libusb missed, calls
qud_open(). On success the QUD device is bound as the inner.
3. Otherwise sleeps and retries. No timeout: the loop waits
indefinitely, same cadence as the prior libusb-only loop.
qdl_read/write/close on the outer auto wrapper forward to the inner.
Pre-open out_chunk_size is stashed on the wrapper and applied at
bind time; vip_data and slot live on the outer where firehose
accesses them, so no forwarding is needed for those.
usb_open_attempt() (factored out of usb_open()) returns
0 / -ENODEV / -EBUSY / -EIO. The --backend usb path loops over it
with visibility-transition messaging. The auto loop owns its own
messaging and never emits the -EBUSY diagnostic because that
condition triggers the QUD attempt in the same tick.
qdl-ramdump now uses QDL_DEVICE_AUTO so Windows users with the
official driver can collect crash dumps without re-binding the USB
driver to WinUSB.
QDL_DEVICE_AUTO is explicitly a meta-backend, not a peer of the USB
and QUD backends: its job is to defer the transport choice to the
moment a probe reaches a device. The libusb and QUD backends remain
standalone and selectable via --backend usb / --backend qud.
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Wire up an actual transport for the QUD backend on Windows. The
official QDLoader 9008 driver exposes a Qualcomm EDL device as a
serial port (\\.\COMx). The Sahara/Firehose framing on top is
identical to the libusb backend; this patch only deals with how
user space reaches the underlying bulk pipes.
Non-Windows hosts keep the stub from the previous patch.
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Introduce QDL_DEVICE_QUD as a third device-transport flavour alongside
the libusb-based USB backend and the SIM backend used by --dry-run.
After this patch --backend=qud parses successfully but errors out at
init time on every host. The actual transport is added in a follow-up.
The intent is to support kernel-mode drivers that expose the EDL
device as a character file (e.g. the official QDLoader 9008 driver on
Windows, where the device shows up as \\.\COMx), so libusb is not
required on hosts where claiming the USB interface from user space is
inconvenient.
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
The single-file format of the flashmap zip lends itself very well for
distributing builds, but the tools for generating one isn't broadly
available.
Implement a new "create-zip" operation in QDL, which parses a build from
a contents.xml, generates a flashmap.json and packages all the files
into a flashmap zip file.
The internal structure is flattened and file names are adjusted to avoid
conflicts.
Sparse files are written to the file by storing the relevant chunks in
their decoded form, rather than storing the sparse file (or full
unsparsed file). From an implementation point of view this is easier, as
we don't keep the sparsed program entry around, but it has the two
benefits of us not storing DONT_CARE entries, and it avoids the need for
seeking as we flash the zip file.
Signed-off-by: Bjorn Andersson <bjorn.andersson@oss.qualcomm.com>
Introduce two initial set of test cases for the newly introduced
contents.xml handling.
One is used to validate the storage and flavor selector logic, and the
other more broadly the loading of contents.xml files.
Extract the common operation of creating a temporary directory to a new
file that is shared across tests.
Signed-off-by: Bjorn Andersson <bjorn.andersson@oss.qualcomm.com>
The typical usage of QDL so far has been for the user to manually
specify a programmer, as well as one or more program and patch XML
files, a so called "flat build". Some targets are built around multiple
storage medias (e.g. spinor + nvme), and some targets have a plethora of
sahara programmer files.
This puts a burden on the user to invoke qdl with the right combination
of storage selector, programmer(s), program and patch XML files. It also
adds an extra step to the build, in form of flattening the Qualcomm
builds.
The Qualcomm builds are described by a toplevel contents.xml file, which
describes to the various tools (e.g. the flattening tool) what
programmer, rawprogram, and patch files to use, and it provides mapping
of each referred file into the build directory structure.
The format also allow multiple "flavors" of flashable content to be
described, across multiple storage types.
Introduce support for parsing contents.xml files to allow the user to
flash non-flattened builds, across one or more storage devices and
filtered by select flavors.
The "qdl flash" operation is overloaded, to create a coherent user
experience. The filter/specifier interpretation is extended to allow
each, comma-separated, entry to describe storage types and flavors.
Signed-off-by: Bjorn Andersson <bjorn.andersson@oss.qualcomm.com>
There is a growing number of path concatenation cases, each built on
some memory allocation scheme and some variant of sprintf.
Introduce a pathbuf type and a few primitive operations to reset the
pathbuf, clone a pathbuf, push a path/filename to the pathbuf, get the
path, and truncate the path by dirname().
Also provide a convenient qdl_path_exist() operation to test if the
pathbuf refers to an existing file.
Signed-off-by: Bjorn Andersson <bjorn.andersson@oss.qualcomm.com>
The parsing of rawprogram files as well as the fallback and precedence
rules, is complex and subject for upcoming changes as part of the
introduction of contents XML support.
Introduce cmocka and a set of test cases to validate the <program>
entries in the rawprogram files are handled as expected.
Signed-off-by: Bjorn Andersson <bjorn.andersson@oss.qualcomm.com>
Both the qdl_common static library and each of the
qdl/qdl-ramdump/qdl-ks executables were declared with the same
`dependencies: common_dep` (libusb-1.0, libxml-2.0, libzip,
plus ws2_32 on Windows). Meson propagates a static library's
dependencies to its consumers via `link_with`, so listing those
dependencies again on each executable caused every common library to
appear twice on the final link line. Apple's ld surfaced this as:
ld: warning: ignoring duplicate libraries: '-lxml2'
The other dependencies are passed by absolute dylib path on macOS
(libusb-1.0, libzip), so the linker silently deduplicated those while
still warning about libxml2, which is referenced as `-lxml2` from the
system path.
Fix it by introducing a compile-only view of the common dependencies
using `partial_dependency(compile_args: true, includes: true)` and
attaching that to the executables instead of the full deps. The
executables still see the headers needed to compile their own sources
(e.g. usb.c -> libusb.h, file.c -> zip.h), while link arguments now
flow to them exactly once via the static library's automatic
propagation.
Fixes: ff189b1a12 ("build: migrate from GNU Make to Meson build system")
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Until now checkpatch had two divergent code paths: the local meson
'check' target ran 'checkpatch.pl --no-tree -f' against every tracked
source file, while the CI workflow used the webispy/checkpatch-action
action, which runs checkpatch in patch mode against each commit in the
PR. The two modes report different defects -- in particular, file mode
flags style issues in long-untouched code and skips commit-log checks,
while patch mode validates the commit message (Signed-off-by, subject
length, ...) and only inspects added/changed lines. Contributors could
end up with a green local 'check' but a red CI run, and vice versa.
Introduce a 'check-range' mode in scripts/checkpatch_wrapper.sh that
mirrors what the GitHub Action does: iterate over every commit in
CHECKPATCH_BASE..CHECKPATCH_HEAD (defaulting to origin/master..HEAD),
feed each commit through 'git format-patch -1 --stdout' and pipe the
result into checkpatch.pl. All commits are processed even on failure
so the user sees every offending commit in one go.
Expose the new mode as a meson run_target ('meson compile check-range
-C build') and switch the CI workflow to invoke the wrapper directly
with the PR base/head SHAs, so local and CI invocations exercise the
exact same code path. Document the new target in README under the
Contributing section.
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
The markdown lint check previously only ran in CI via the
markdownlint-cli2 GitHub Action, which meant developers had no
straightforward way to reproduce the check locally before pushing. As a
result, contributors only discovered lint errors after opening a PR.
Introduce scripts/markdownlint_wrapper.sh that drives mdl on the same
set of files the CI workflow lints. mdl is shipped as a Debian/Ubuntu
package, so contributors can install it with 'sudo apt install mdl'
without pulling in a Node.js toolchain. Wire the wrapper up as a meson
run_target so it can be invoked with:
meson compile markdown-lint -C build
Switch the markdown-lint workflow to install mdl from apt and invoke
the wrapper directly, making the script the single source of truth for
both local and CI runs. Document the usage in README under the
Contributing section.
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
The flashmap support saves the user from having to specify programmer,
program and patch XML files, it allows reading the flashmap.json either
among the flat files, or as part of a zip-container of all the parts.
Add dry-run tests for the two cases, by introducing a flashmap.json and
zipping up all the files, then run qdl on the two versions.
Signed-off-by: Bjorn Andersson <bjorn.andersson@oss.qualcomm.com>
The flashmap and related files form a package, suitable for distribution
and one-step installation, but having to unzip the archive in order to
flash/install its content is not convenient for the user.
Extend the qdl_file helper code to also provide access to files within a
zip file, then extend the flashmap code to attempt to decode the package
file as such.
Pass the optional zip object around to the various places where we might
read files from the package.
With this we can now "flash" the installer.zip directly:
qdl flash installer.zip
The selective flashing mechanism is carried into the zip handling,
allowing the user to flash specific memories only, such as:
qdl flash installer.zip::spinor
Signed-off-by: Bjorn Andersson <bjorn.andersson@oss.qualcomm.com>
When distributing builds the user/developer is forced to determine which
programmer, program, patch files should be passed to QDL, to which
storage device each set should be programmed. This isn't user friendly
and it's prone of various forms of mistakes.
A solution to this problem can be found in the build artifacts of a
Qualcomm Windows-releases where a "flashmap" file describes the very
aspects we're looking for.
Implement support for parsing this json blob, populate the Sahara images
list based on the "programmer" information, parse the "program" and
"patch" files for each memory type and populate the firehose operations
queue with "configure", "program", and "patch" operations.
Introduce a new "flash" subcommand to trigger this new implementation:
qdl flash flashmap.json
In certain cases the user might want to only flash the subset of images
pertaining to a particular storage media - such as recovering their
spinor content. The user can through a suffix operation select which
storage media to flash, e.g.:
qdl flash flashmap.json::spinor,ufs
The json parser is derived from the implementation previously in use in
the pd-mapper userspace service, to avoid adding additional library
dependencies.
Signed-off-by: Bjorn Andersson <bjorn.andersson@oss.qualcomm.com>
Today there are a few different places through the code where files are
opened for reading, either in a oneshot fashion or for streaming (e.g.
during programming).
In order to allow these operations to read files packaged into
compressed zip files, and to avoid duplication of the load operation,
introduce the "qdl_file" file reading abstraction for these use cases.
It's not yet known how VIP is handled in these archives and to reduce
the impact of the change this time, the scope is limited to reading the
files used during normal flashing - this can obviously be considered for
extension later.
Signed-off-by: Bjorn Andersson <bjorn.andersson@oss.qualcomm.com>
The alias_target('manpages', manpage_targets) call lived outside the
'if help2man.found()' / 'can_run_exe' block, but manpage_targets is
only populated inside it. When help2man is absent (or this is a cross
build), manpage_targets is undefined and meson setup fails. The same
call would also fail if reached with an empty list, since alias_target
requires at least one dependency.
Move the alias_target declaration inside the same guard so it is only
emitted when there are actual manpage targets to alias.
Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Rename the ks executable to qdl-ks to be consistent with the naming
convention used by the other tools in the project such as qdl-ramdump.
Signed-off-by: Christopher Obbard <christopher.obbard@linaro.org>
Igor Opaniuk
Bjorn Andersson
Christopher Obbard
Bjorn Andersson
Christopher Obbard