[libcamera-devel] [PATCH] libcamera: Rework automatic version generation to avoid rebuilds

[Laurent Pinchart]

Hi Kieran,

On Fri, Jul 05, 2019 at 12:18:31PM +0100, Kieran Bingham wrote:
> On 05/07/2019 09:29, Laurent Pinchart wrote:
> > Commit b817bcec6b53 ("libcamera: Auto generate version information")
> > generates version information in order to automatically include it
> > various locations (Sphinx and Doxygen documentation, libcamera::version
> > variable available at runtime, and version.h available at compile time).
> > Unfortunately this causes lots of unnecessary rebuilds when modifying
> > the git tree state, which hinders development.
> 
> Did you perform any measurements here?

Only subjective measurement when developing on top of the automatic
version generation. I've now measured the actual time. The test case is
a moving between a base commit and a commit directly on top that only
touches a .cpp file (thread.cpp in this case, which I still have to
post). Running plain 'time ninja' on my machine (2 cores with
hyperthreading), I went from

[35/35] Linking target test/timer.

real    0m1.737s
user    0m3.342s
sys     0m0.584s

to

[39/39] Linking target src/qcam/qcam.

real    0m3.376s
user    0m8.669s
sys     0m0.898s

The increase is most probably not as dramatic as it felt when I realised
it was an undesired side effect I hadn't foreseen, but I would still
avoid increasing build time if possible.

> You are right, that due to the version string changing, any time the
> git-version returns a different string, extra steps are required to
> re-link that information.
> 
> for testing below, the following command is used to show all actual
> tasks performed, and remove parallelisation from the equation:
> 
>    /usr/bin/time -v ninja -j1 -v -d explain -d stats
> 
> 
> A previous no-op rebuild which did not consider the version string took:
>  (taken at 021af795c298d10317c2931dc5ba05d887c49ab6)
> 
> 	Command being timed: "ninja -j1 -v -d explain -d stats"
> 	User time (seconds): 0.01
> 	System time (seconds): 0.01
> 	Percent of CPU this job got: 100%
> 	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.02
> 
> 
> Certainly fast, - because it's a full no-op.
> 
> 
> With my series, A no-op rebuild with the same version string (taken at
> 0de1a9f318ecf9f3d069b8a4a4a02ef0391cfb04) takes:
> 
> 	Command being timed: "ninja -j1 -v -d explain -d stats"
> 	User time (seconds): 0.16
> 	System time (seconds): 0.02
> 	Percent of CPU this job got: 101%
> 	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.18
> 
> 
> Slower, yes, but not IMO a severe penalty or development hindrance.
> 
> The only executed command is:
> 
> [1/37] meson --internal vcstagger version.h.in version.h v0.0  @VCS_TAG@
> '(.*)' utils/gen-version.sh
> 
> 
> 
> A 'no-op-version-changed' (again at 0de1a9f, which excludes this patch)
> takes:
> 
> 	# 'git commit --amend' executed to change the git-version
>            without any code change
> 
> 	Command being timed: "ninja -j1 -v -d explain -d stats"
> 	User time (seconds): 2.81
> 	System time (seconds): 0.34
> 	Percent of CPU this job got: 103%
> 	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.05
> 
> 
> 7 commands were executed: <output reduced for brevity here>
> 
> [1/37] meson --internal vcstagger version.h.in version.h v0.0  @VCS_TAG@
> '(.*)' utils/gen-version.sh
> [2/37] c++ -c src/libcamera/control_types.cpp # An autogenerated file
> [3/37] c++ -c camera_manager.cpp # Due to version str
> [4/37] c++ -o src/libcamera/libcamera.so # Relink libcamera.so
> [5/37] meson --internal symbolextractor libcamera.so.symbols
> [6/7] c++ -c ../src/qcam/main_window.cpp
> [7/7] c++ -o src/qcam/qcam # Regenerated because it references the
> version.h which includes the direct string which has changed.
> 
> 
> When run without the -j1 I have an 8-core CPU, so 3.05 linear seconds
> becames 1.87 elapsed. Perhaps this is still noticeable on slower build
> machines. Of course - this is only if the *git version* changes, so some
> development change has likely occurred and other objects are probably
> being rebuilt too.
> 
> 
> But nothing there seems unreasonable, perhaps excepting the
> re-generation of the control_types.cpp.
> 
> With *this* patch,
> 
> a 'no-op' rebuild with the same version string takes:
> 
> 	Command being timed: "ninja -j1 -v -d explain -d stats"
> 	User time (seconds): 0.17
> 	System time (seconds): 0.02
> 	Percent of CPU this job got: 99%
> 	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.20
> 
> and has indeed generated exactly one command generation: <cmd edited for
> brevity>
> 
> [1/36] meson --internal vcstagger version.cpp.in version.cpp v0.0
> @VCS_TAG@ '(.*)' gen-version.sh
> 
> 
> Not that there was one less check performed internally, so this is now
> [1/36] where before it was [1/37]
> 
> 
> and a 'no-op-version-changed' rebuild takes:
> 
> 	# git commit --amend; edit message, quit to generate new sha1.
> 
> 	Command being timed: "ninja -j1 -v -d explain -d stats"
> 	User time (seconds): 0.94
> 	System time (seconds): 0.19
> 	Percent of CPU this job got: 98%
> 	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.16
> 
> 
> so 3.05 elapsed time for non-parallel build is down to 1.16. Certainly
> can't fault that as an improvement. For parallel builds, this patch
> brings Elapsed time down to 1.07 from 1.87.
> 
> Again for completeness, here's the commands which were now executed:
> 
> [1/36] meson --internal vcstagger version.cpp.in
> src/libcamera/version.cpp v0.0 src/libcamera @VCS_TAG@ '(.*)'
> src/libcamera/gen-version.sh /home/linuxembedded/iob/libcamera/libcamera
> [2/36] c++ -c src/libcamera/version.cpp # Our autogenerated version.cpp
> [3/36] c++ -o src/libcamera/libcamera.so # Re-link
> [4/36] meson --internal symbolextractor libcamera.so.symbols
> [5/5] /usr/bin/doxygen Documentation/Doxyfile
> 
> > The problem is caused by the generated version.h being listed as a
> > dependency for the whole libcamera. This is required as meson (to the
> > best of my knowledge) doesn't provide a way to explicitly specify the
> > dependency of a single object file (camera_manager.o in this case, as
> 
> Single objects determine their dependencies through the compiler parsing
> their includes, so that tells them what needs to be rebuilt - but
> doesn't link to the vcs_tag object indeed. Perhaps this is a limitation
> in meson that could be updated, but I don't think it's necessary.

I also believe it's a limitation of meson, yes. They may be a way to
declare additional dependencies at the object level, but I haven't found
one.

> The dependencies are checked in sequence to ensure that the version tag
> is updated before it is used. As far as I can see, no full rebuild
> occurs, (with the exception of the controls file which is also
> auto-generated).

I'm not sure to follow you here. What ensures that the version tag is
updated before it is used ?

> > camera_manager.cpp is the only consumer of the generated version string)
> > on the custom target used to generate version.h. The dependency can't be
> > automatically detected at build time, like dependencies on normal
> > headers that are generated by parsing the source, because the version.h
> > header may not exist yet. The build could then fail in a racy way.
> 
> Correct, the normal dependency generation can't queue the execution of
> the vcs_tag command, but it does mean that only objects which include
> version.h directly are rebuilt, which I think is as expected?

The problem, which I encountered before when generating libcamera.h and
controls, is that the missing dependency can cause a build failure due
to a race condition. Initially version.h doesn't exist, so the automatic
source dependency generation mechanism will not pick it up, and will not
cause it to be built. If it gets build before the source files that
include it, it's all fine. Otherwise compilation will fail as the header
doesn't exist. This can cause a transient build failure too, as the
version.h generation could run in parallel to the build of a file using
it. Re-running ninja will then work.

> > This change attempts at solving the issue by generating a version.cpp
> > instead of a version.h. This minimises the number of files that need to
> > be rebuild when then git tree state changes, while retaining the main
> > purpose of the original automatic version generation, the ability to
> > access the git-based version string at runtime.
> 
> I'm fine with this, it's more akin to what I originally proposed.
> 
> By removing the version string from the header, it means that qcam does
> not get rebuilt each time as it now only gets the runtime version string.
> 
> > The Sphinx and Doxygen documentation however lose git-based version
> > information, to prevent a full documentation rebuild for every commit.
> > We may want to investigate how to preserve detailed version information
> > there, as well as how to make it available in a header file for
> > applications. For the time being, this commit should be a good
> > compromise to avoid unnecessary recompilation, and additional features
> > can be built on top of it after taking the time to consider and test
> > them carefully.
> 
> Having the build information directly linked into the documentation was
> a key point of my series. I wanted the documentation to specify the
> version that it represents.

I would like that too, I actually think it's more important than
displaying the version number in qcam (and is of similar importance to
displaying it in the libcamera log). I would however make a distinction
between the Sphinx and the Doxygen documentation, I think it's more
important for the latter than the former (at least at this time).

> I agree that there is a meson issue there that because the version given
> to Doxygen is only generated at configure time it could get stale.
> 
> So there's a policy point to consider there, As anything which generates
> the version for public consumption should do a full build - we can
> ensure that it does do any necessary reconfiguration stage. In fact I
> believe you can "ninja reconfigure" without requiring a full rebuild too...

Regarding documentation, I think having version information is useful,
but I would like to know what use cases you care about. The
documentation we currently publish on libcamera.org is certainly not
based on a stable branch, but on the other hand I don't think anyone
expects it to be. When we'll start creating releases for libcamera I
foresee a need to have multiple versions of the API documentation on the
website, one for each release (possibly with a link named latest
pointing to the latest one). We can also keep publishing a nightly
build, and that's probably the only one for which I can think of a full
version string being useful. For that we will do a full build in any
case, so the correct version will be included.

I have a feeling that vcs_tag() is currently a half-baked feature in
meson, without any best practice rules on how to use it, and in general
on how to generate and use version information (this part isn't specific
to meson). That's why I believe we should take the time required to
think about what we need. In particular, we expose no version
information that can be used at build time (the preprocessor can't
really make use of a version string in a meaningful way), so I think we
should expose the classic LIBCAMERA_VERSION_MAJOR,
LIBCAMERA_VERSION_MINOR and LIBCAMERA_VERSION_PATCH in a header file
generated at meson configure time, without including any git SHA1 or
patch count in there. A "plain" version string can easily be generated
from that, and a "git" version string should be exposed through a
runtime API only (if at all) and be printed to the libcamera log (that
part I believe is important for debugging purpose).

> > Fixes: b817bcec6b53 ("libcamera: Auto generate version information")
> 
> 'Fixes' seems a strong choice to me.
> I don't seem to have the same concerns as you regarding b817, but lets
> move on.
> 
> Acked-by: Kieran Bingham <kieran.bingham at ideasonboard.com>
> 
> > Signed-off-by: Laurent Pinchart <laurent.pinchart at ideasonboard.com>
> > ---
> >  Documentation/Doxyfile.in                     |  4 ++-
> >  meson.build                 |  8 +-----
> >  {version.h.in => version.h} |  4 ---
> >  meson.build                                   |  4 +--
> >  src/libcamera/camera_manager.cpp              |  5 ----
> >  {utils => src/libcamera}/gen-version.sh       |  0
> >  src/libcamera/meson.build                     |  8 +++++-
> >  src/libcamera/version.cpp.in                  | 25 +++++++++++++++++++
> >  8 files changed, 37 insertions(+), 21 deletions(-)
> >  rename {version.h.in => version.h} (79%)
> >  rename {utils => src/libcamera}/gen-version.sh (100%)
> >  create mode 100644 src/libcamera/version.cpp.in
> > 
> > diff --git a/Documentation/Doxyfile.in b/Documentation/Doxyfile.in
> > index cad85ff979f8..ed111909b603 100644
> > --- a/Documentation/Doxyfile.in
> > +++ b/Documentation/Doxyfile.in
> > @@ -791,7 +791,9 @@ WARN_LOGFILE           =
> >  # spaces. See also FILE_PATTERNS and EXTENSION_MAPPING
> >  # Note: If this tag is empty the current directory is searched.
> >  
> > -INPUT                  = "@TOP_SRCDIR@/include/libcamera" "@TOP_SRCDIR@/src/libcamera"
> > +INPUT                  = "@TOP_SRCDIR@/include/libcamera" \
> > +			 "@TOP_SRCDIR@/src/libcamera" \
> > +			 "@TOP_BUILDDIR@/src/libcamera"
> >  
> >  # This tag can be used to specify the character encoding of the source files
> >  # that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses
> > diff --git a/meson.build b/meson.build
> > index 6f81f1117318..bafa103d141b 100644
> > --- a/meson.build
> > +++ b/meson.build
> > @@ -14,15 +14,9 @@ libcamera_api = files([
> >      'signal.h',
> >      'stream.h',
> >      'timer.h',
> > +    'version.h',
> >  ])
> >  
> > -gen_version = join_paths(meson.source_root(), 'utils', 'gen-version.sh')
> > -
> > -version_h = vcs_tag(command : [gen_version, meson.current_source_dir()],
> > -                    input : 'version.h.in',
> > -                    output : 'version.h',
> > -                    fallback : 'v0.0')
> > -
> >  gen_header = files('gen-header.sh')
> >  
> >  libcamera_h = custom_target('gen-header',
> > diff --git a/version.h.in b/version.h
> > similarity index 79%
> > rename from version.h.in
> > rename to version.h
> > index e49b36962aed..75d57d48af0d 100644
> > --- a/version.h.in
> > +++ b/version.h
> > @@ -3,16 +3,12 @@
> >   * Copyright (C) 2019, Google Inc.
> >   *
> >   * version.h - Library version information
> > - *
> > - * This file is auto-generated. Do not edit.
> >   */
> >  #ifndef __LIBCAMERA_VERSION_H__
> >  #define __LIBCAMERA_VERSION_H__
> >  
> >  #include <string>
> >  
> > -#define LIBCAMERA_VERSION "@VCS_TAG@"
> > -
> >  namespace libcamera {
> >  
> >  extern const std::string version;
> > diff --git a/meson.build b/meson.build
> > index 342b3cc76a93..271b538bdae8 100644
> > --- a/meson.build
> > +++ b/meson.build
> > @@ -1,8 +1,6 @@
> >  project('libcamera', 'c', 'cpp',
> >      meson_version : '>= 0.40',
> > -    version : run_command('utils/gen-version.sh',
> > -                          '@0@'.format(meson.source_root()),
> > -                          check : true).stdout().strip(),
> > +    version : 'v0.0',
> 
> I'm unhappy that this patch changes the project version, but you seem to
> be very keen to remove this, so I concede.

I've thought about it since our last discussion on this topic, and from
my side I have the following requirements for the base version number:

- It shall not be stored in multiple files (canonical source)
- For tarball releases, tt shall be stored in a file in the root
  directory of the project
- Under git control, if it is stored in a file, it shall be stored in
  the same file as for the tarball releases

meson.build is an obvious candidate. If we decide to store the version
number in a file under git control, it should be meson.build. The
advantage in my opinion is that it gives an easy way to quickly check
the base version.

However, I'm not completely opposed to storing the version number in a
file not under git control, which would be generated when creating a
tarball release. There are pros and cons for this, and I think I could
be convinced if most of the concerns can be addressed.

So going back to storing the version information in meson.build in this
patch is not an attempt to set this in stone in the long term, but
really an interim measure.

> In the long run, I'd perhaps envisage having an option on gen-version
> which prepares a 'simple' version, which will return the latest known
> release without extra labels or commit specific information, that is
> provided to the meson.project_version.
> 
> >      default_options : [
> >          'werror=true',
> >          'warning_level=2',
> > diff --git a/src/libcamera/camera_manager.cpp b/src/libcamera/camera_manager.cpp
> > index c5da46b4062d..69503c6e8726 100644
> > --- a/src/libcamera/camera_manager.cpp
> > +++ b/src/libcamera/camera_manager.cpp
> > @@ -26,11 +26,6 @@ namespace libcamera {
> >  
> >  LOG_DEFINE_CATEGORY(Camera)
> >  
> > -/**
> > - * \brief The library global version string
> > - */
> > -const std::string version(LIBCAMERA_VERSION);
> > -
> >  /**
> >   * \class CameraManager
> >   * \brief Provide access and manage all cameras in the system
> > diff --git a/utils/gen-version.sh b/src/libcamera/gen-version.sh
> > similarity index 100%
> > rename from utils/gen-version.sh
> > rename to src/libcamera/gen-version.sh
> > diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build
> > index 336f4f066fac..c35ecb988d72 100644
> > --- a/src/libcamera/meson.build
> > +++ b/src/libcamera/meson.build
> > @@ -79,8 +79,14 @@ control_types_cpp = custom_target('control_types_cpp',
> >  
> >  libcamera_sources += control_types_cpp
> >  
> > +version_cpp = vcs_tag(command : ['gen-version.sh', meson.source_root()],
> > +                      input : 'version.cpp.in',
> > +                      output : 'version.cpp',
> > +                      fallback : meson.project_version())
> > +
> > +libcamera_sources += version_cpp
> > +
> >  libcamera_deps = [
> > -    declare_dependency(sources : version_h),
> >      cc.find_library('dl'),
> >      libudev,
> >  ]
> > diff --git a/src/libcamera/version.cpp.in b/src/libcamera/version.cpp.in
> > new file mode 100644
> > index 000000000000..055dc0b5c5ad
> > --- /dev/null
> > +++ b/src/libcamera/version.cpp.in
> > @@ -0,0 +1,25 @@
> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> > +/*
> > + * Copyright (C) 2019, Google Inc.
> > + *
> > + * version.cpp - libcamera version
> > + *
> > + * This file is auto-generated. Do not edit.
> > + */
> > +
> > +#include <libcamera/version.h>
> > +
> > +/**
> > + * \file version.h
> > + * \brief libcamera version
> > + */
> > +
> > +namespace libcamera {
> > +
> > +/**
> > + * \var libcamera::version
> > + * \brief The library global version string
> > + */
> > +const std::string version("@VCS_TAG@");
> > +
> > +} /* namespace libcamera */

-- 
Regards,

Laurent Pinchart