[libcamera-devel] [PATCH] libcamera: Rework automatic version generation to avoid rebuilds
Kieran Bingham
kieran.bingham at ideasonboard.com
Fri Jul 5 13:18:31 CEST 2019
Hi Laurent,
On 05/07/2019 09:29, Laurent Pinchart wrote:
> Commit b817bcec6b53 ("libcamera: Auto generate version information")
> generates version information in order to automatically include it
> various locations (Sphinx and Doxygen documentation, libcamera::version
> variable available at runtime, and version.h available at compile time).
> Unfortunately this causes lots of unnecessary rebuilds when modifying
> the git tree state, which hinders development.
Did you perform any measurements here?
You are right, that due to the version string changing, any time the
git-version returns a different string, extra steps are required to
re-link that information.
for testing below, the following command is used to show all actual
tasks performed, and remove parallelisation from the equation:
/usr/bin/time -v ninja -j1 -v -d explain -d stats
A previous no-op rebuild which did not consider the version string took:
(taken at 021af795c298d10317c2931dc5ba05d887c49ab6)
Command being timed: "ninja -j1 -v -d explain -d stats"
User time (seconds): 0.01
System time (seconds): 0.01
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.02
Certainly fast, - because it's a full no-op.
With my series, A no-op rebuild with the same version string (taken at
0de1a9f318ecf9f3d069b8a4a4a02ef0391cfb04) takes:
Command being timed: "ninja -j1 -v -d explain -d stats"
User time (seconds): 0.16
System time (seconds): 0.02
Percent of CPU this job got: 101%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.18
Slower, yes, but not IMO a severe penalty or development hindrance.
The only executed command is:
[1/37] meson --internal vcstagger version.h.in version.h v0.0 @VCS_TAG@
'(.*)' utils/gen-version.sh
A 'no-op-version-changed' (again at 0de1a9f, which excludes this patch)
takes:
# 'git commit --amend' executed to change the git-version
without any code change
Command being timed: "ninja -j1 -v -d explain -d stats"
User time (seconds): 2.81
System time (seconds): 0.34
Percent of CPU this job got: 103%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.05
7 commands were executed: <output reduced for brevity here>
[1/37] meson --internal vcstagger version.h.in version.h v0.0 @VCS_TAG@
'(.*)' utils/gen-version.sh
[2/37] c++ -c src/libcamera/control_types.cpp # An autogenerated file
[3/37] c++ -c camera_manager.cpp # Due to version str
[4/37] c++ -o src/libcamera/libcamera.so # Relink libcamera.so
[5/37] meson --internal symbolextractor libcamera.so.symbols
[6/7] c++ -c ../src/qcam/main_window.cpp
[7/7] c++ -o src/qcam/qcam # Regenerated because it references the
version.h which includes the direct string which has changed.
When run without the -j1 I have an 8-core CPU, so 3.05 linear seconds
becames 1.87 elapsed. Perhaps this is still noticeable on slower build
machines. Of course - this is only if the *git version* changes, so some
development change has likely occurred and other objects are probably
being rebuilt too.
But nothing there seems unreasonable, perhaps excepting the
re-generation of the control_types.cpp.
With *this* patch,
a 'no-op' rebuild with the same version string takes:
Command being timed: "ninja -j1 -v -d explain -d stats"
User time (seconds): 0.17
System time (seconds): 0.02
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.20
and has indeed generated exactly one command generation: <cmd edited for
brevity>
[1/36] meson --internal vcstagger version.cpp.in version.cpp v0.0
@VCS_TAG@ '(.*)' gen-version.sh
Not that there was one less check performed internally, so this is now
[1/36] where before it was [1/37]
and a 'no-op-version-changed' rebuild takes:
# git commit --amend; edit message, quit to generate new sha1.
Command being timed: "ninja -j1 -v -d explain -d stats"
User time (seconds): 0.94
System time (seconds): 0.19
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.16
so 3.05 elapsed time for non-parallel build is down to 1.16. Certainly
can't fault that as an improvement. For parallel builds, this patch
brings Elapsed time down to 1.07 from 1.87.
Again for completeness, here's the commands which were now executed:
[1/36] meson --internal vcstagger version.cpp.in
src/libcamera/version.cpp v0.0 src/libcamera @VCS_TAG@ '(.*)'
src/libcamera/gen-version.sh /home/linuxembedded/iob/libcamera/libcamera
[2/36] c++ -c src/libcamera/version.cpp # Our autogenerated version.cpp
[3/36] c++ -o src/libcamera/libcamera.so # Re-link
[4/36] meson --internal symbolextractor libcamera.so.symbols
[5/5] /usr/bin/doxygen Documentation/Doxyfile
> The problem is caused by the generated version.h being listed as a
> dependency for the whole libcamera. This is required as meson (to the
> best of my knowledge) doesn't provide a way to explicitly specify the
> dependency of a single object file (camera_manager.o in this case, as
Single objects determine their dependencies through the compiler parsing
their includes, so that tells them what needs to be rebuilt - but
doesn't link to the vcs_tag object indeed. Perhaps this is a limitation
in meson that could be updated, but I don't think it's necessary.
The dependencies are checked in sequence to ensure that the version tag
is updated before it is used. As far as I can see, no full rebuild
occurs, (with the exception of the controls file which is also
auto-generated).
> camera_manager.cpp is the only consumer of the generated version string)
> on the custom target used to generate version.h. The dependency can't be
> automatically detected at build time, like dependencies on normal
> headers that are generated by parsing the source, because the version.h
> header may not exist yet. The build could then fail in a racy way.
Correct, the normal dependency generation can't queue the execution of
the vcs_tag command, but it does mean that only objects which include
version.h directly are rebuilt, which I think is as expected?
> This change attempts at solving the issue by generating a version.cpp
> instead of a version.h. This minimises the number of files that need to
> be rebuild when then git tree state changes, while retaining the main
> purpose of the original automatic version generation, the ability to
> access the git-based version string at runtime.
I'm fine with this, it's more akin to what I originally proposed.
By removing the version string from the header, it means that qcam does
not get rebuilt each time as it now only gets the runtime version string.
> The Sphinx and Doxygen documentation however lose git-based version
> information, to prevent a full documentation rebuild for every commit.
> We may want to investigate how to preserve detailed version information
> there, as well as how to make it available in a header file for
> applications. For the time being, this commit should be a good
> compromise to avoid unnecessary recompilation, and additional features
> can be built on top of it after taking the time to consider and test
> them carefully.
Having the build information directly linked into the documentation was
a key point of my series. I wanted the documentation to specify the
version that it represents.
I agree that there is a meson issue there that because the version given
to Doxygen is only generated at configure time it could get stale.
So there's a policy point to consider there, As anything which generates
the version for public consumption should do a full build - we can
ensure that it does do any necessary reconfiguration stage. In fact I
believe you can "ninja reconfigure" without requiring a full rebuild too...
>
> Fixes: b817bcec6b53 ("libcamera: Auto generate version information")
'Fixes' seems a strong choice to me.
I don't seem to have the same concerns as you regarding b817, but lets
move on.
Acked-by: Kieran Bingham <kieran.bingham at ideasonboard.com>
> Signed-off-by: Laurent Pinchart <laurent.pinchart at ideasonboard.com>
> ---
> Documentation/Doxyfile.in | 4 ++-
> meson.build | 8 +-----
> {version.h.in => version.h} | 4 ---
> meson.build | 4 +--
> src/libcamera/camera_manager.cpp | 5 ----
> {utils => src/libcamera}/gen-version.sh | 0
> src/libcamera/meson.build | 8 +++++-
> src/libcamera/version.cpp.in | 25 +++++++++++++++++++
> 8 files changed, 37 insertions(+), 21 deletions(-)
> rename {version.h.in => version.h} (79%)
> rename {utils => src/libcamera}/gen-version.sh (100%)
> create mode 100644 src/libcamera/version.cpp.in
>
> diff --git a/Documentation/Doxyfile.in b/Documentation/Doxyfile.in
> index cad85ff979f8..ed111909b603 100644
> --- a/Documentation/Doxyfile.in
> +++ b/Documentation/Doxyfile.in
> @@ -791,7 +791,9 @@ WARN_LOGFILE =
> # spaces. See also FILE_PATTERNS and EXTENSION_MAPPING
> # Note: If this tag is empty the current directory is searched.
>
> -INPUT = "@TOP_SRCDIR@/include/libcamera" "@TOP_SRCDIR@/src/libcamera"
> +INPUT = "@TOP_SRCDIR@/include/libcamera" \
> + "@TOP_SRCDIR@/src/libcamera" \
> + "@TOP_BUILDDIR@/src/libcamera"
>
> # This tag can be used to specify the character encoding of the source files
> # that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses
> diff --git a/meson.build b/meson.build
> index 6f81f1117318..bafa103d141b 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -14,15 +14,9 @@ libcamera_api = files([
> 'signal.h',
> 'stream.h',
> 'timer.h',
> + 'version.h',
> ])
>
> -gen_version = join_paths(meson.source_root(), 'utils', 'gen-version.sh')
> -
> -version_h = vcs_tag(command : [gen_version, meson.current_source_dir()],
> - input : 'version.h.in',
> - output : 'version.h',
> - fallback : 'v0.0')
> -
> gen_header = files('gen-header.sh')
>
> libcamera_h = custom_target('gen-header',
> diff --git a/version.h.in b/version.h
> similarity index 79%
> rename from version.h.in
> rename to version.h
> index e49b36962aed..75d57d48af0d 100644
> --- a/version.h.in
> +++ b/version.h
> @@ -3,16 +3,12 @@
> * Copyright (C) 2019, Google Inc.
> *
> * version.h - Library version information
> - *
> - * This file is auto-generated. Do not edit.
> */
> #ifndef __LIBCAMERA_VERSION_H__
> #define __LIBCAMERA_VERSION_H__
>
> #include <string>
>
> -#define LIBCAMERA_VERSION "@VCS_TAG@"
> -
> namespace libcamera {
>
> extern const std::string version;
> diff --git a/meson.build b/meson.build
> index 342b3cc76a93..271b538bdae8 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -1,8 +1,6 @@
> project('libcamera', 'c', 'cpp',
> meson_version : '>= 0.40',
> - version : run_command('utils/gen-version.sh',
> - '@0@'.format(meson.source_root()),
> - check : true).stdout().strip(),
> + version : 'v0.0',
I'm unhappy that this patch changes the project version, but you seem to
be very keen to remove this, so I concede.
In the long run, I'd perhaps envisage having an option on gen-version
which prepares a 'simple' version, which will return the latest known
release without extra labels or commit specific information, that is
provided to the meson.project_version.
> default_options : [
> 'werror=true',
> 'warning_level=2',
> diff --git a/src/libcamera/camera_manager.cpp b/src/libcamera/camera_manager.cpp
> index c5da46b4062d..69503c6e8726 100644
> --- a/src/libcamera/camera_manager.cpp
> +++ b/src/libcamera/camera_manager.cpp
> @@ -26,11 +26,6 @@ namespace libcamera {
>
> LOG_DEFINE_CATEGORY(Camera)
>
> -/**
> - * \brief The library global version string
> - */
> -const std::string version(LIBCAMERA_VERSION);
> -
> /**
> * \class CameraManager
> * \brief Provide access and manage all cameras in the system
> diff --git a/utils/gen-version.sh b/src/libcamera/gen-version.sh
> similarity index 100%
> rename from utils/gen-version.sh
> rename to src/libcamera/gen-version.sh
> diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build
> index 336f4f066fac..c35ecb988d72 100644
> --- a/src/libcamera/meson.build
> +++ b/src/libcamera/meson.build
> @@ -79,8 +79,14 @@ control_types_cpp = custom_target('control_types_cpp',
>
> libcamera_sources += control_types_cpp
>
> +version_cpp = vcs_tag(command : ['gen-version.sh', meson.source_root()],
> + input : 'version.cpp.in',
> + output : 'version.cpp',
> + fallback : meson.project_version())
> +
> +libcamera_sources += version_cpp
> +
> libcamera_deps = [
> - declare_dependency(sources : version_h),
> cc.find_library('dl'),
> libudev,
> ]
> diff --git a/src/libcamera/version.cpp.in b/src/libcamera/version.cpp.in
> new file mode 100644
> index 000000000000..055dc0b5c5ad
> --- /dev/null
> +++ b/src/libcamera/version.cpp.in
> @@ -0,0 +1,25 @@
> +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> +/*
> + * Copyright (C) 2019, Google Inc.
> + *
> + * version.cpp - libcamera version
> + *
> + * This file is auto-generated. Do not edit.
> + */
> +
> +#include <libcamera/version.h>
> +
> +/**
> + * \file version.h
> + * \brief libcamera version
> + */
> +
> +namespace libcamera {
> +
> +/**
> + * \var libcamera::version
> + * \brief The library global version string
> + */
> +const std::string version("@VCS_TAG@");
> +
> +} /* namespace libcamera */
>
--
Regards
--
Kieran
More information about the libcamera-devel
mailing list