As suggested by Baruch Siach, using "git rev-parse HEAD" is a lot
simpler than playing around with "git log" to just retrieve the commit
id corresponding to the current HEAD.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
pkg-stats extracts the Buildroot commit id from which the package
information was collected. However, when doing so, it always assumes
we're using the master branch, by running "git log master".
But in fact, pkg-stats can be run from any branch/tag, so it makes a
lot more sense to use "git log HEAD".
Cc: victor.huesca@bootlin.com
Cc: Baruch Siach <baruch@tkos.co.il>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Now that we can order packages from biggest to smallest, it makes sense
to assign the most aggressive colours to the biggest packages.
As such, reorder the current colours so that we have, in order:
- red-ish
- orange-ish
- yellow-ish
- purple-ish
- eggplant-ish (is that even a colour? :-] )
- some-indeterminate-blue-ish
- dark-green-ish
- light-green-ish
For the previous, smallest-first ordering, it does not matter much what
the ordering is: the actual colours are still somewhat-unpredictably
assigned to packages, depending on the cut-off limit...
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, the packages are sorted smallest first, and biggest last
(with unknown and others second-to-last and last, resp.).
Add an option to invert the ordering (but keeping unknown and others at
their current positions).
This has the nice side effect that we can now control the colours
assigned to the biggest package(s), as the colours are cycled from the
first to the last. Currently, the biggest packages gets a redish colour,
which is appropriate, but the second gets a greenish one, which is not
as appropriate (but changing that can come later).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
When dealing with embedded devices, storage is more often than not some
kind of flash device, on which the memory is usually counted as powers
of 1024 instead of powers of 1000. As such, people may prefer reports
using IEC prefixes [0] instead of the SI prefixes.
Add an option to that effect.
We use argparse's ability to use custom actions [1] [2], to provide a
set of options that act on a boolean, but has a single help entry and
internally ensures consistency of the settings. We could have been using
the more conventional store_true/store_false actions instead, but that
would have meant either two help entries, one for each set of options,
and/or some logic after parse_args() to check the validity of the
settings.
[0] https://en.wikipedia.org/wiki/Binary_prefix
[1] https://docs.python.org/2/library/argparse.html#action
[2] https://docs.python.org/2/library/argparse.html#argparse.Action
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, we group packages that contribute less then 1%, into the
"Other" category.
However, in some cases, there can be a lot of very comparatively small
packages, and they may not exceed this limit, and so only the "Others"
category would be displayed, which is not nice.
Conversely, if there are a lot of packages, most of which only so
slightly exceeding this limit, then we get all of them in the graph,
which is not nice either.
Add a way for the developers to pass a different cut-off limit. As for
the dependency graph which has BR2_GRAPH_DEPS_OPTS, add the environment
variable BR2_GRAPH_SIZE_OPTS to carry those extra option (in preparation
for more to come, later).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[Arnout:
- remove empty base class definition from Config;
- use parser.error instead of ValueError for invalid argument.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, we forcibly report sizes in multiple of Kilobytes. In some
big configurations, the sizes of the system as a whole, as well as that
of individual packages, may exceed megabytes, and when some artistic
assets get used, even the gigabyte may get exceed.
These big sizes are not easy to read when expressed in kilobytes.
Additionally, some very small packages might have sizes below the
kilobyte (and when we can specify the cut-off grouping size, they may
get reported), and thus the size displayed for those would be 0 kB.
Add a helper function that can format a floating-point size into a
string with all the appropriate formatting:
- there are at least 3 meaningfull digits visible, i.e. we display
"3.14" or "10.4" instead of just "3" or "10", but for big number we
don't care about too many precision either, so we report "100" or
"1000", not "100.42" or "1000.27";
- the proper SI prefix is appended, if needed.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, the "unknown" category may be reported anywhere, so it does
not really stand out when there are a lot of packages in the graph.
Move it towards the end, but right before the "other" category, so that
it is a bit more visible. Like for Others, don't report it if its size
is zero.
Also, make it title case (i.e. "Unknown" instead of "unknown").
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
It is nicer overall to have a main() function, like all our other
scripts tend to have too.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Similar to toolchains and jpeg, we now offer a way for br2-external
trees to provide their openssl implementation, which gets included in
the openssl choice.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Similar to toolchains, we now offer a way for br2-external trees to
provide their libjpeg implementation, which gets included in the jpeg
choice.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since we have a choice for the pre-configured pre-built toolchains,
there is no possbility for a br2-external to provide its own. The
only solution so far for defconfigs in br2-external trees is to use
BR2_TOOLCHAIN_EXTERNAL_CUSTOM and define all the bits by itself...
This is not so convemient, so offer a way for br2-external trees to
provide such pre-configured toolchains.
To allow for this, we now scan each br2-external tree and look for a
specific file, provides.toolchains.in. We generate a kconfig file that
sources each such file, and that generated file is sourced from within
the toolchain choice, thus making the toolchains from a br2-external
tree possible and available in the same location as the ones known to
Buildroot:
Toolchain --->
Toolchain type (External toolchain) --->
Toolchain --->
(X) Arm ARM 2019.03
( ) Linaro ARM 2018.05
( ) Custom toolchain
*** Toolchains from my-br2-ext-tree: ***
( ) My custom ARM toolchain
*** Toolchains from another-br2-ext-tree: ***
( ) Another custom ARM toolchain
( ) A third custom ARM toolchain
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, the kconfig part contains two things: the kconfig option
with the paths to br2-external trees, and the kconfig menus for the
br2-external trees.
When we want to include more kconfig files from the br2-external tree
(e.g. to get definitions for pre-built toolchains), we will need to
have the paths defined earlier, so they can be used from the br2-external
tree to include files earlier than the existing menus.
Split the generated kconfig file in two: one to define the paths, which
gets included early in our main Config.in, and one to actually define
the existing menus, which still gets included at the same place they
currently are.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
We currently redirect the output of each helper function. This was nice
as long as we were generating single .mk and .in fragments.
But we are soon to need more .in fragments.
So, do the redirection inside the .in helpers.
We do not (currently) need to generate more than one .mk fragment, but
for consistency, do the redirection in the .mk helper too.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When we introduced support for multiple br2-external trees, we
introduced two files, one on the Makefile side, needed very early,
and one on the kconfig side, needed later in the configuration
process. We naturally introduced a two-step generation, as it looked
like the simplest and most obvious way.
But now, we are on the verge of generating more files on the kconfig
side, and it does not make sense to add even more steps to generate
them.
And even better yet, we can generate both the Makefile-side and
kconfig-side files at the same time, in fact.
Make it so.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
We do not usually provide help for our internal scripts. Besides, such
help has a tendency to bitrot pretty quickly anyway.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Commit b14b02698 (core/br2-external: restore compatibility with old
distros) switched to using 'eval' to emulate associative arrays, for
those distros too old to have bash-4+.
In so doing, it forgot to declare the new local variables in the
respective helper functions.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The major bottleneck in pkg-stats is the time spent waiting for
answers from remote servers. Two functions involve such communication
with remote servers:
- 'check_package_urls' which checks that each package upstream website
is up, it is efficient due to the use of process-pools thanks to
Matt Weber.
- 'check_package_latest_version' which fetches the latest package
version from release-monitoring, it uses a http-pool but runs
sequentially.
This patch extends the use of process-pools to 'check_latest_version'.
Due to some limitations of multiprocess callbacks, this patch loses
the overall progress of packages in favour of just the current package
name.
Runtimes for this function are ~3m vs ~25m for the linear version.
Tested on an i7 7500U (2/4 cores/threads @3.5GHz) with 15ms ping.
Note: There have already been work trying to parallelize this function
using threads but there were a failure on some configurations [1].
This implementation rely on a dedicated module already in use on this
script, so it's unlikely to see failure with this version.
[1] http://lists.busybox.net/pipermail/buildroot/2018-March/215368.html
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Fixes:
- blank space before ':'
- unused 'o' variable left from a previous patch
- bad continuous alignment
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The pkg-stats calls 3 times `make` to get a bunch of variables. These
variables can be obtained in only one make invocation. This patch
replaces the three calls by just one and adjusts the parsing logic
accordingly.
Note: another option suggested by Arnout would be to run `make
show-info` that produces a json with the necessary variables. This
would avoid the duplicated effort done in pkg-stats and pkg-utils and
allow to add other infos to pkg-stats like dependencies, reversed
dependencies or if the package is virtual.
In order to use this method, the following changes are required in
pkg-generic's show-info:
- include license_files;
- have an option to run it on *all* packages, not just the selected
ones.
This patch take the simplest approach of only factorizing the make
calls as it requires less changes.
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since it's used only for the HTML output, and all other functions used
for HTML output are prefixed by dump_html, let's do so for
dump_gen_info() as well by renaming it to dump_html_gen_info().
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The 'dump_html' and 'dump_json' both include commit infos as well as the
current date. It make more sense to retrieve these information once.
This patch simply does this factorization.
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Pkg-stats is a great script that get a lot of interesting info from
buildroot packages. Unfortunately it is currently designed to output a
static HTML page only. While this is great to include on the
buildroot's website, the HTML is not designed to be easily parsable and
thus it is difficult to reuse it in other scripts.
This patch provide a new option to output a JSON file in addition to the
HTML one.
The old 'output' option has been renamed to 'html' to distinguish from
the new 'json' option.
Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Thomas Petazzoni
Yann E. MORIN
Victor Huesca