[libcamera-ci] [RFC PATCH v2 4/4] Add job to run lc-compliance on the virtual pipeline handler

[Laurent Pinchart]

On Tue, Dec 17, 2024 at 11:41:57AM +0100, Barnabás Pőcze wrote:
> 2024. 12. 17. 1:17 keltezéssel, Laurent Pinchart írta:
> > Hi Barnabás,
> > 
> > Thank you for the patch.
> > 
> > On Mon, Dec 16, 2024 at 06:28:27PM +0100, Barnabás Pőcze wrote:
> >> Add a new job named `lc-compliance:virtual` that uses the
> >> build artifacts created by the `build-package:debug` job
> >> on amd64 to run `lc-compliance` on the "Virtual0" camera
> >> in a virtual machine.
> >>
> >> The `force_fallback_for=gtest` option is needed because `cpp_debustl`
> > 
> > s/debustl/debugstl/
> > 
> >> makes ABI incompatible changes in the STL containers, so googletest
> >> also needs to be compiled with these changes.
> > 
> > Downloading sources at build time in CI isn't great, I'd like to avoid
> > it when possible. If it can't be avoided, so be it. There are
> > precedents, we download and install the packages that are not multi-arch
> > compatible at build time, and we also force a fallback for gtest when
> > compiling with clang and libc++.
> 
> I believe libyuv is also downloaded.

Aarrrghhhhh. Why can't google get their projects packaged by
distributions ? :-(

> > On a side note, I'm wondering if we shouldn't just disable lc-compliance
> > when building with clang and libc++, and download and cache the .deb
> > packages at container build time. If anyone wants to give it a try...
> > :-)
> > 
> > Back to the topic, does cpp_debugstl give use significant benefits
> > compared to ASan and UBSan ?
> 
> Well, there are certainly things that it recognizes that ASAN/UBSAN
> do not, for example:
> 
>    std::vector<int> v; ...
>    auto it = /* into `v` */;
>    v.erase(/* before `it` */);
>    *it; // error

That's good to catch indeed. Is the error caught at runtime or compile
time ?

> >> Signed-off-by: Barnabás Pőcze <barnabas.pocze at ideasonboard.com>
> >> ---
> >>
> >> Changes in v2:
> >>    * publish lc-compliance results to gitlab
> >>    * add and use script to install libcamera binary package
> >>
> >> ---
> >>   .gitlab-ci/test-lc-compliance.sh  | 37 +++++++++++++++++++++++++++++++
> >>   .gitlab-ci/unpackage-libcamera.sh | 17 ++++++++++++++
> >>   gitlab-ci.yml                     | 24 +++++++++++++++++++-
> >>   3 files changed, 77 insertions(+), 1 deletion(-)
> >>   create mode 100755 .gitlab-ci/test-lc-compliance.sh
> >>   create mode 100755 .gitlab-ci/unpackage-libcamera.sh
> >>
> >> diff --git a/.gitlab-ci/test-lc-compliance.sh b/.gitlab-ci/test-lc-compliance.sh
> >> new file mode 100755
> >> index 0000000..a927f1d
> >> --- /dev/null
> >> +++ b/.gitlab-ci/test-lc-compliance.sh
> >> @@ -0,0 +1,37 @@
> >> +#!/bin/bash
> >> +
> >> +# SPDX-License-Identifier: GPL-2.0-or-later
> >> +# SPDX-FileCopyrightText: © 2024 Google Inc.
> >> +
> >> +set -e
> >> +
> >> +source "$(dirname "$0")/lib.sh"
> >> +
> >> +libcamera_compliance() {
> >> +	echo "Running libcamera compliance tests in a qemu VM"
> >> +
> >> +	virtme-ng \
> >> +		--verbose \
> >> +		--skip-modules \
> >> +		--force-9p \
> >> +		--rwdir "$PWD/build" \
> >> +		--run /opt/linux/bzImage \
> >> +		--exec "\
> > 
> > s/"/" /
> > 
> >> +			LIBCAMERA_LOG_LEVELS=*:DEBUG \
> >> +			ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 \
> >> +			UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 \
> >> +			GTEST_OUTPUT=xml:./build/lc-compliance-report.xml \
> >> +			lc-compliance -c Virtual0; \
> >> +			echo \\\$? > ./build/.test-status \
> >> +		"
> >> +
> >> +	local status=$(cat build/.test-status)
> >> +	echo "Test result exit state: $status"
> >> +	rm build/.test-status
> >> +
> >> +	if [[ $status != 0 ]] ; then
> >> +		exit $status
> >> +	fi
> >> +}
> >> +
> >> +run libcamera_compliance
> >> diff --git a/.gitlab-ci/unpackage-libcamera.sh b/.gitlab-ci/unpackage-libcamera.sh
> >> new file mode 100755
> >> index 0000000..9eb3192
> >> --- /dev/null
> >> +++ b/.gitlab-ci/unpackage-libcamera.sh
> >> @@ -0,0 +1,17 @@
> >> +#!/bin/bash
> >> +
> >> +# SPDX-License-Identifier: GPL-2.0-or-later
> >> +# SPDX-FileCopyrightText: © 2024 Google Inc.
> >> +
> >> +set -e
> >> +
> >> +source "$(dirname "$0")/lib.sh"
> >> +
> >> +libcamera_unpackage() {
> >> +	echo "Unpackage libcamera binaries"
> >> +
> >> +	tar -xvf libcamera-${CI_COMMIT_SHA}.tar.xz -C /
> >> +	ldconfig -v
> >> +}
> >> +
> >> +run libcamera_unpackage
> >> diff --git a/gitlab-ci.yml b/gitlab-ci.yml
> >> index 0a3eab3..8ea1d12 100644
> >> --- a/gitlab-ci.yml
> >> +++ b/gitlab-ci.yml
> >> @@ -328,11 +328,13 @@ build-package:debug:
> >>       BUILD_TYPE: debug
> >>       MESON_OPTIONS: >-
> >>         -D auto_features=disabled
> >> +      -D lc-compliance=enabled
> >>         -D test=false
> >>         -D v4l2=false
> >>         -D b_sanitize=address,undefined
> >>         -D cpp_debugstl=true
> >> -      -D pipelines=[]
> >> +      -D pipelines=['virtual']
> >> +      -D force_fallback_for=['gtest']
> >>     parallel:
> >>       matrix:
> >>         - ARCH: amd64
> >> @@ -440,3 +442,23 @@ test-unit:
> >>     # artifacts:
> >>     #   reports:
> >>     #     junit: build/meson-logs/testlog.junit.xml
> >> +
> >> +lc-compliance:virtual:
> > 
> > Nitpicking on the naming, we could name this test-lc-compliance:virtual
> > or test-compliance:virtual to have the same "test-" prefix as
> > "test-unit". Maybe "test:unit" and "test:compliance:virtual" could also
> > be an option ?
> > 
> >> +  extends:
> >> +    - .fdo.distribution-image at debian
> >> +    - .libcamera-ci.debian:12
> >> +    - .libcamera-ci.scripts
> >> +  stage: test
> >> +  needs:
> >> +    - job: build-package:debug
> >> +      parallel:
> >> +        matrix:
> >> +          - ARCH: amd64
> >> +  tags:
> >> +    - kvm
> >> +  script:
> >> +    - $CI_PROJECT_DIR/.gitlab-ci/unpackage-libcamera.sh
> >> +    - $CI_PROJECT_DIR/.gitlab-ci/test-lc-compliance.sh
> >> +  artifacts:
> >> +    reports:
> >> +      junit: build/lc-compliance-report.xml

-- 
Regards,

Laurent Pinchart