[libcamera-ci] [RFC PATCH v2 4/4] Add job to run lc-compliance on the virtual pipeline handler

[Laurent Pinchart]

Hi Barnabás,

Thank you for the patch.

On Mon, Dec 16, 2024 at 06:28:27PM +0100, Barnabás Pőcze wrote:
> Add a new job named `lc-compliance:virtual` that uses the
> build artifacts created by the `build-package:debug` job
> on amd64 to run `lc-compliance` on the "Virtual0" camera
> in a virtual machine.
> 
> The `force_fallback_for=gtest` option is needed because `cpp_debustl`

s/debustl/debugstl/

> makes ABI incompatible changes in the STL containers, so googletest
> also needs to be compiled with these changes.

Downloading sources at build time in CI isn't great, I'd like to avoid
it when possible. If it can't be avoided, so be it. There are
precedents, we download and install the packages that are not multi-arch
compatible at build time, and we also force a fallback for gtest when
compiling with clang and libc++.

On a side note, I'm wondering if we shouldn't just disable lc-compliance
when building with clang and libc++, and download and cache the .deb
packages at container build time. If anyone wants to give it a try...
:-)

Back to the topic, does cpp_debugstl give use significant benefits
compared to ASan and UBSan ?

> Signed-off-by: Barnabás Pőcze <barnabas.pocze at ideasonboard.com>
> ---
> 
> Changes in v2:
>   * publish lc-compliance results to gitlab
>   * add and use script to install libcamera binary package
> 
> ---
>  .gitlab-ci/test-lc-compliance.sh  | 37 +++++++++++++++++++++++++++++++
>  .gitlab-ci/unpackage-libcamera.sh | 17 ++++++++++++++
>  gitlab-ci.yml                     | 24 +++++++++++++++++++-
>  3 files changed, 77 insertions(+), 1 deletion(-)
>  create mode 100755 .gitlab-ci/test-lc-compliance.sh
>  create mode 100755 .gitlab-ci/unpackage-libcamera.sh
> 
> diff --git a/.gitlab-ci/test-lc-compliance.sh b/.gitlab-ci/test-lc-compliance.sh
> new file mode 100755
> index 0000000..a927f1d
> --- /dev/null
> +++ b/.gitlab-ci/test-lc-compliance.sh
> @@ -0,0 +1,37 @@
> +#!/bin/bash
> +
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# SPDX-FileCopyrightText: © 2024 Google Inc.
> +
> +set -e
> +
> +source "$(dirname "$0")/lib.sh"
> +
> +libcamera_compliance() {
> +	echo "Running libcamera compliance tests in a qemu VM"
> +
> +	virtme-ng \
> +		--verbose \
> +		--skip-modules \
> +		--force-9p \
> +		--rwdir "$PWD/build" \
> +		--run /opt/linux/bzImage \
> +		--exec "\

s/"/" /

> +			LIBCAMERA_LOG_LEVELS=*:DEBUG \
> +			ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 \
> +			UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 \
> +			GTEST_OUTPUT=xml:./build/lc-compliance-report.xml \
> +			lc-compliance -c Virtual0; \
> +			echo \\\$? > ./build/.test-status \
> +		"
> +
> +	local status=$(cat build/.test-status)
> +	echo "Test result exit state: $status"
> +	rm build/.test-status
> +
> +	if [[ $status != 0 ]] ; then
> +		exit $status
> +	fi
> +}
> +
> +run libcamera_compliance
> diff --git a/.gitlab-ci/unpackage-libcamera.sh b/.gitlab-ci/unpackage-libcamera.sh
> new file mode 100755
> index 0000000..9eb3192
> --- /dev/null
> +++ b/.gitlab-ci/unpackage-libcamera.sh
> @@ -0,0 +1,17 @@
> +#!/bin/bash
> +
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# SPDX-FileCopyrightText: © 2024 Google Inc.
> +
> +set -e
> +
> +source "$(dirname "$0")/lib.sh"
> +
> +libcamera_unpackage() {
> +	echo "Unpackage libcamera binaries"
> +
> +	tar -xvf libcamera-${CI_COMMIT_SHA}.tar.xz -C /
> +	ldconfig -v
> +}
> +
> +run libcamera_unpackage
> diff --git a/gitlab-ci.yml b/gitlab-ci.yml
> index 0a3eab3..8ea1d12 100644
> --- a/gitlab-ci.yml
> +++ b/gitlab-ci.yml
> @@ -328,11 +328,13 @@ build-package:debug:
>      BUILD_TYPE: debug
>      MESON_OPTIONS: >-
>        -D auto_features=disabled
> +      -D lc-compliance=enabled
>        -D test=false
>        -D v4l2=false
>        -D b_sanitize=address,undefined
>        -D cpp_debugstl=true
> -      -D pipelines=[]
> +      -D pipelines=['virtual']
> +      -D force_fallback_for=['gtest']
>    parallel:
>      matrix:
>        - ARCH: amd64
> @@ -440,3 +442,23 @@ test-unit:
>    # artifacts:
>    #   reports:
>    #     junit: build/meson-logs/testlog.junit.xml
> +
> +lc-compliance:virtual:

Nitpicking on the naming, we could name this test-lc-compliance:virtual
or test-compliance:virtual to have the same "test-" prefix as
"test-unit". Maybe "test:unit" and "test:compliance:virtual" could also
be an option ?

> +  extends:
> +    - .fdo.distribution-image at debian
> +    - .libcamera-ci.debian:12
> +    - .libcamera-ci.scripts
> +  stage: test
> +  needs:
> +    - job: build-package:debug
> +      parallel:
> +        matrix:
> +          - ARCH: amd64
> +  tags:
> +    - kvm
> +  script:
> +    - $CI_PROJECT_DIR/.gitlab-ci/unpackage-libcamera.sh
> +    - $CI_PROJECT_DIR/.gitlab-ci/test-lc-compliance.sh
> +  artifacts:
> +    reports:
> +      junit: build/lc-compliance-report.xml

-- 
Regards,

Laurent Pinchart