[libcamera-ci] [RFC PATCH v2 4/4] Add job to run lc-compliance on the virtual pipeline handler

[Barnabás Pőcze]

2024. 12. 17. 1:17 keltezéssel, Laurent Pinchart írta:
> Hi Barnabás,
> 
> Thank you for the patch.
> 
> On Mon, Dec 16, 2024 at 06:28:27PM +0100, Barnabás Pőcze wrote:
>> Add a new job named `lc-compliance:virtual` that uses the
>> build artifacts created by the `build-package:debug` job
>> on amd64 to run `lc-compliance` on the "Virtual0" camera
>> in a virtual machine.
>>
>> The `force_fallback_for=gtest` option is needed because `cpp_debustl`
> 
> s/debustl/debugstl/
> 
>> makes ABI incompatible changes in the STL containers, so googletest
>> also needs to be compiled with these changes.
> 
> Downloading sources at build time in CI isn't great, I'd like to avoid
> it when possible. If it can't be avoided, so be it. There are
> precedents, we download and install the packages that are not multi-arch
> compatible at build time, and we also force a fallback for gtest when
> compiling with clang and libc++.

I believe libyuv is also downloaded.

> 
> On a side note, I'm wondering if we shouldn't just disable lc-compliance
> when building with clang and libc++, and download and cache the .deb
> packages at container build time. If anyone wants to give it a try...
> :-)
> 
> Back to the topic, does cpp_debugstl give use significant benefits
> compared to ASan and UBSan ?

Well, there are certainly things that it recognizes that ASAN/UBSAN
do not, for example:

   std::vector<int> v; ...
   auto it = /* into `v` */;
   v.erase(/* before `it` */);
   *it; // error


> 
>> Signed-off-by: Barnabás Pőcze <barnabas.pocze at ideasonboard.com>
>> ---
>>
>> Changes in v2:
>>    * publish lc-compliance results to gitlab
>>    * add and use script to install libcamera binary package
>>
>> ---
>>   .gitlab-ci/test-lc-compliance.sh  | 37 +++++++++++++++++++++++++++++++
>>   .gitlab-ci/unpackage-libcamera.sh | 17 ++++++++++++++
>>   gitlab-ci.yml                     | 24 +++++++++++++++++++-
>>   3 files changed, 77 insertions(+), 1 deletion(-)
>>   create mode 100755 .gitlab-ci/test-lc-compliance.sh
>>   create mode 100755 .gitlab-ci/unpackage-libcamera.sh
>>
>> diff --git a/.gitlab-ci/test-lc-compliance.sh b/.gitlab-ci/test-lc-compliance.sh
>> new file mode 100755
>> index 0000000..a927f1d
>> --- /dev/null
>> +++ b/.gitlab-ci/test-lc-compliance.sh
>> @@ -0,0 +1,37 @@
>> +#!/bin/bash
>> +
>> +# SPDX-License-Identifier: GPL-2.0-or-later
>> +# SPDX-FileCopyrightText: © 2024 Google Inc.
>> +
>> +set -e
>> +
>> +source "$(dirname "$0")/lib.sh"
>> +
>> +libcamera_compliance() {
>> +	echo "Running libcamera compliance tests in a qemu VM"
>> +
>> +	virtme-ng \
>> +		--verbose \
>> +		--skip-modules \
>> +		--force-9p \
>> +		--rwdir "$PWD/build" \
>> +		--run /opt/linux/bzImage \
>> +		--exec "\
> 
> s/"/" /
> 
>> +			LIBCAMERA_LOG_LEVELS=*:DEBUG \
>> +			ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 \
>> +			UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 \
>> +			GTEST_OUTPUT=xml:./build/lc-compliance-report.xml \
>> +			lc-compliance -c Virtual0; \
>> +			echo \\\$? > ./build/.test-status \
>> +		"
>> +
>> +	local status=$(cat build/.test-status)
>> +	echo "Test result exit state: $status"
>> +	rm build/.test-status
>> +
>> +	if [[ $status != 0 ]] ; then
>> +		exit $status
>> +	fi
>> +}
>> +
>> +run libcamera_compliance
>> diff --git a/.gitlab-ci/unpackage-libcamera.sh b/.gitlab-ci/unpackage-libcamera.sh
>> new file mode 100755
>> index 0000000..9eb3192
>> --- /dev/null
>> +++ b/.gitlab-ci/unpackage-libcamera.sh
>> @@ -0,0 +1,17 @@
>> +#!/bin/bash
>> +
>> +# SPDX-License-Identifier: GPL-2.0-or-later
>> +# SPDX-FileCopyrightText: © 2024 Google Inc.
>> +
>> +set -e
>> +
>> +source "$(dirname "$0")/lib.sh"
>> +
>> +libcamera_unpackage() {
>> +	echo "Unpackage libcamera binaries"
>> +
>> +	tar -xvf libcamera-${CI_COMMIT_SHA}.tar.xz -C /
>> +	ldconfig -v
>> +}
>> +
>> +run libcamera_unpackage
>> diff --git a/gitlab-ci.yml b/gitlab-ci.yml
>> index 0a3eab3..8ea1d12 100644
>> --- a/gitlab-ci.yml
>> +++ b/gitlab-ci.yml
>> @@ -328,11 +328,13 @@ build-package:debug:
>>       BUILD_TYPE: debug
>>       MESON_OPTIONS: >-
>>         -D auto_features=disabled
>> +      -D lc-compliance=enabled
>>         -D test=false
>>         -D v4l2=false
>>         -D b_sanitize=address,undefined
>>         -D cpp_debugstl=true
>> -      -D pipelines=[]
>> +      -D pipelines=['virtual']
>> +      -D force_fallback_for=['gtest']
>>     parallel:
>>       matrix:
>>         - ARCH: amd64
>> @@ -440,3 +442,23 @@ test-unit:
>>     # artifacts:
>>     #   reports:
>>     #     junit: build/meson-logs/testlog.junit.xml
>> +
>> +lc-compliance:virtual:
> 
> Nitpicking on the naming, we could name this test-lc-compliance:virtual
> or test-compliance:virtual to have the same "test-" prefix as
> "test-unit". Maybe "test:unit" and "test:compliance:virtual" could also
> be an option ?
> 
>> +  extends:
>> +    - .fdo.distribution-image at debian
>> +    - .libcamera-ci.debian:12
>> +    - .libcamera-ci.scripts
>> +  stage: test
>> +  needs:
>> +    - job: build-package:debug
>> +      parallel:
>> +        matrix:
>> +          - ARCH: amd64
>> +  tags:
>> +    - kvm
>> +  script:
>> +    - $CI_PROJECT_DIR/.gitlab-ci/unpackage-libcamera.sh
>> +    - $CI_PROJECT_DIR/.gitlab-ci/test-lc-compliance.sh
>> +  artifacts:
>> +    reports:
>> +      junit: build/lc-compliance-report.xml
>