[libcamera-devel] [PATCH v9 0/5] lc-compliance: Refactor using Googletest
Nícolas F. R. A. Prado
nfraprado at collabora.com
Fri Jun 25 21:39:20 CEST 2021
(We're now running lc-compliance on KernelCI, more details below)
This is a refactoring of lc-compliance using Googletest as the framework.
Patches 1 and 2 are just to simplify the cleanup path from the tests. Patch 3
adds an Environment singleton to provide access to the camera from all tests.
Patch 4 does the actual refactor. Patch 5 adds two new parameters, --list and
--filter.
lc-compliance on KernelCI
-------------------------
lc-compliance is already being run on KernelCI [1] for mt8173-elm-hana
(Chromebook), which has an USB camera. All test runs of lc-compliance so far can
be seen on the Tests tab [2] by searching for "lc-compliance". I'm currently
looking into adding Rock Pi 4 boards to the lab so that the tests can run on
those too.
[1] https://linux.kernelci.org/test/job/tip/branch/master/kernel/v5.13-rc6-239-g0b42677e2e5d/plan/lc-compliance/
[2] https://linux.kernelci.org/test/
I wrote a blog post on Collabora's blog [3] going over the process of adding
lc-compliance to KernelCI, so check it out for more details on that. At the
moment lc-compliance is being compiled from my tree [4], but as soon as this
series get merged we can easily change KernelCI to compile from upstream.
[3] https://www.collabora.com/news-and-blog/blog/2021/06/15/testing-cameras-with-lc-compliance-on-kernelci/
[4] https://gitlab.collabora.com/nfraprado/libcamera/-/tree/lcc-gtest-v5
lc-compliance Usage
-------------------
lc-compliance is built around testing a single camera (and possibly optional
secondary cameras in the future), which is passed through the -c flag. The list
of tests can be queried with:
lc-compliance --list
By default all tests are run, but it's possible to select which tests to run by
using:
lc-compliance -c '<camera_name>' --filter '<test_filter>'
where the test_filter is in the same format as Googletest's --gtest_filter.
Quoting its help:
--gtest_filter=POSTIVE_PATTERNS[-NEGATIVE_PATTERNS]
Run only the tests whose name matches one of the positive patterns but
none of the negative patterns. '?' matches any single character; '*'
matches any substring; ':' separates two patterns.
This is a snippet of a successful run:
[==========] Running 120 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 120 tests from RolesAndRequests/FixedRequestsTest
[ RUN ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_1
[31:53:24.860985336] [271719] INFO Camera camera.cpp:915 configuring streams: (0) 1280x720-MJPEG
Camera needs 4 requests, can't test only 1
[ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_1 (64 ms)
[...]
[ RUN ] RolesAndRequests/FixedRequestsTest.Unbalanced/Viewfinder_89
[31:56:18.607757616] [271719] INFO Camera camera.cpp:915 configuring streams: (0) 1280x720-MJPEG
[ OK ] RolesAndRequests/FixedRequestsTest.Unbalanced/Viewfinder_89 (3148 ms)
[----------] 120 tests from RolesAndRequests/FixedRequestsTest (176896 ms total)
[----------] Global test environment tear-down
[==========] 120 tests from 1 test suite ran. (176896 ms total)
[ PASSED ] 96 tests.
[ SKIPPED ] 24 tests, listed below:
[ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_1
[ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_2
[ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_3
[ SKIPPED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/StillCapture_1
[...]
And how a failure looks like:
[ RUN ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_5
[31:59:59.528082413] [272962] INFO Camera camera.cpp:915 configuring streams: (0) 1280x720-MJPEG
../src/lc-compliance/simple_capture.cpp:117: Failure
Expected equality of these values:
captureCount_
Which is: 5
captureLimit_
Which is: 6
[ FAILED ] RolesAndRequests/FixedRequestsTest.BalancedSingleCycle/Raw_5, where GetParam() = (0, 5) (317 ms)
The current complete list of tests is shown below:
RolesAndRequests/FixedRequestsTest.
BalancedSingleCycle/Raw_1 # GetParam() = (0, 1)
BalancedSingleCycle/Raw_2 # GetParam() = (0, 2)
BalancedSingleCycle/Raw_3 # GetParam() = (0, 3)
BalancedSingleCycle/Raw_5 # GetParam() = (0, 5)
BalancedSingleCycle/Raw_8 # GetParam() = (0, 8)
BalancedSingleCycle/Raw_13 # GetParam() = (0, 13)
BalancedSingleCycle/Raw_21 # GetParam() = (0, 21)
BalancedSingleCycle/Raw_34 # GetParam() = (0, 34)
BalancedSingleCycle/Raw_55 # GetParam() = (0, 55)
BalancedSingleCycle/Raw_89 # GetParam() = (0, 89)
BalancedSingleCycle/StillCapture_1 # GetParam() = (1, 1)
BalancedSingleCycle/StillCapture_2 # GetParam() = (1, 2)
BalancedSingleCycle/StillCapture_3 # GetParam() = (1, 3)
BalancedSingleCycle/StillCapture_5 # GetParam() = (1, 5)
BalancedSingleCycle/StillCapture_8 # GetParam() = (1, 8)
BalancedSingleCycle/StillCapture_13 # GetParam() = (1, 13)
BalancedSingleCycle/StillCapture_21 # GetParam() = (1, 21)
BalancedSingleCycle/StillCapture_34 # GetParam() = (1, 34)
BalancedSingleCycle/StillCapture_55 # GetParam() = (1, 55)
BalancedSingleCycle/StillCapture_89 # GetParam() = (1, 89)
BalancedSingleCycle/VideoRecording_1 # GetParam() = (2, 1)
BalancedSingleCycle/VideoRecording_2 # GetParam() = (2, 2)
BalancedSingleCycle/VideoRecording_3 # GetParam() = (2, 3)
BalancedSingleCycle/VideoRecording_5 # GetParam() = (2, 5)
BalancedSingleCycle/VideoRecording_8 # GetParam() = (2, 8)
BalancedSingleCycle/VideoRecording_13 # GetParam() = (2, 13)
BalancedSingleCycle/VideoRecording_21 # GetParam() = (2, 21)
BalancedSingleCycle/VideoRecording_34 # GetParam() = (2, 34)
BalancedSingleCycle/VideoRecording_55 # GetParam() = (2, 55)
BalancedSingleCycle/VideoRecording_89 # GetParam() = (2, 89)
BalancedSingleCycle/Viewfinder_1 # GetParam() = (3, 1)
BalancedSingleCycle/Viewfinder_2 # GetParam() = (3, 2)
BalancedSingleCycle/Viewfinder_3 # GetParam() = (3, 3)
BalancedSingleCycle/Viewfinder_5 # GetParam() = (3, 5)
BalancedSingleCycle/Viewfinder_8 # GetParam() = (3, 8)
BalancedSingleCycle/Viewfinder_13 # GetParam() = (3, 13)
BalancedSingleCycle/Viewfinder_21 # GetParam() = (3, 21)
BalancedSingleCycle/Viewfinder_34 # GetParam() = (3, 34)
BalancedSingleCycle/Viewfinder_55 # GetParam() = (3, 55)
BalancedSingleCycle/Viewfinder_89 # GetParam() = (3, 89)
BalancedMultiCycle/Raw_1 # GetParam() = (0, 1)
BalancedMultiCycle/Raw_2 # GetParam() = (0, 2)
BalancedMultiCycle/Raw_3 # GetParam() = (0, 3)
BalancedMultiCycle/Raw_5 # GetParam() = (0, 5)
BalancedMultiCycle/Raw_8 # GetParam() = (0, 8)
BalancedMultiCycle/Raw_13 # GetParam() = (0, 13)
BalancedMultiCycle/Raw_21 # GetParam() = (0, 21)
BalancedMultiCycle/Raw_34 # GetParam() = (0, 34)
BalancedMultiCycle/Raw_55 # GetParam() = (0, 55)
BalancedMultiCycle/Raw_89 # GetParam() = (0, 89)
BalancedMultiCycle/StillCapture_1 # GetParam() = (1, 1)
BalancedMultiCycle/StillCapture_2 # GetParam() = (1, 2)
BalancedMultiCycle/StillCapture_3 # GetParam() = (1, 3)
BalancedMultiCycle/StillCapture_5 # GetParam() = (1, 5)
BalancedMultiCycle/StillCapture_8 # GetParam() = (1, 8)
BalancedMultiCycle/StillCapture_13 # GetParam() = (1, 13)
BalancedMultiCycle/StillCapture_21 # GetParam() = (1, 21)
BalancedMultiCycle/StillCapture_34 # GetParam() = (1, 34)
BalancedMultiCycle/StillCapture_55 # GetParam() = (1, 55)
BalancedMultiCycle/StillCapture_89 # GetParam() = (1, 89)
BalancedMultiCycle/VideoRecording_1 # GetParam() = (2, 1)
BalancedMultiCycle/VideoRecording_2 # GetParam() = (2, 2)
BalancedMultiCycle/VideoRecording_3 # GetParam() = (2, 3)
BalancedMultiCycle/VideoRecording_5 # GetParam() = (2, 5)
BalancedMultiCycle/VideoRecording_8 # GetParam() = (2, 8)
BalancedMultiCycle/VideoRecording_13 # GetParam() = (2, 13)
BalancedMultiCycle/VideoRecording_21 # GetParam() = (2, 21)
BalancedMultiCycle/VideoRecording_34 # GetParam() = (2, 34)
BalancedMultiCycle/VideoRecording_55 # GetParam() = (2, 55)
BalancedMultiCycle/VideoRecording_89 # GetParam() = (2, 89)
BalancedMultiCycle/Viewfinder_1 # GetParam() = (3, 1)
BalancedMultiCycle/Viewfinder_2 # GetParam() = (3, 2)
BalancedMultiCycle/Viewfinder_3 # GetParam() = (3, 3)
BalancedMultiCycle/Viewfinder_5 # GetParam() = (3, 5)
BalancedMultiCycle/Viewfinder_8 # GetParam() = (3, 8)
BalancedMultiCycle/Viewfinder_13 # GetParam() = (3, 13)
BalancedMultiCycle/Viewfinder_21 # GetParam() = (3, 21)
BalancedMultiCycle/Viewfinder_34 # GetParam() = (3, 34)
BalancedMultiCycle/Viewfinder_55 # GetParam() = (3, 55)
BalancedMultiCycle/Viewfinder_89 # GetParam() = (3, 89)
Unbalanced/Raw_1 # GetParam() = (0, 1)
Unbalanced/Raw_2 # GetParam() = (0, 2)
Unbalanced/Raw_3 # GetParam() = (0, 3)
Unbalanced/Raw_5 # GetParam() = (0, 5)
Unbalanced/Raw_8 # GetParam() = (0, 8)
Unbalanced/Raw_13 # GetParam() = (0, 13)
Unbalanced/Raw_21 # GetParam() = (0, 21)
Unbalanced/Raw_34 # GetParam() = (0, 34)
Unbalanced/Raw_55 # GetParam() = (0, 55)
Unbalanced/Raw_89 # GetParam() = (0, 89)
Unbalanced/StillCapture_1 # GetParam() = (1, 1)
Unbalanced/StillCapture_2 # GetParam() = (1, 2)
Unbalanced/StillCapture_3 # GetParam() = (1, 3)
Unbalanced/StillCapture_5 # GetParam() = (1, 5)
Unbalanced/StillCapture_8 # GetParam() = (1, 8)
Unbalanced/StillCapture_13 # GetParam() = (1, 13)
Unbalanced/StillCapture_21 # GetParam() = (1, 21)
Unbalanced/StillCapture_34 # GetParam() = (1, 34)
Unbalanced/StillCapture_55 # GetParam() = (1, 55)
Unbalanced/StillCapture_89 # GetParam() = (1, 89)
Unbalanced/VideoRecording_1 # GetParam() = (2, 1)
Unbalanced/VideoRecording_2 # GetParam() = (2, 2)
Unbalanced/VideoRecording_3 # GetParam() = (2, 3)
Unbalanced/VideoRecording_5 # GetParam() = (2, 5)
Unbalanced/VideoRecording_8 # GetParam() = (2, 8)
Unbalanced/VideoRecording_13 # GetParam() = (2, 13)
Unbalanced/VideoRecording_21 # GetParam() = (2, 21)
Unbalanced/VideoRecording_34 # GetParam() = (2, 34)
Unbalanced/VideoRecording_55 # GetParam() = (2, 55)
Unbalanced/VideoRecording_89 # GetParam() = (2, 89)
Unbalanced/Viewfinder_1 # GetParam() = (3, 1)
Unbalanced/Viewfinder_2 # GetParam() = (3, 2)
Unbalanced/Viewfinder_3 # GetParam() = (3, 3)
Unbalanced/Viewfinder_5 # GetParam() = (3, 5)
Unbalanced/Viewfinder_8 # GetParam() = (3, 8)
Unbalanced/Viewfinder_13 # GetParam() = (3, 13)
Unbalanced/Viewfinder_21 # GetParam() = (3, 21)
Unbalanced/Viewfinder_34 # GetParam() = (3, 34)
Unbalanced/Viewfinder_55 # GetParam() = (3, 55)
Unbalanced/Viewfinder_89 # GetParam() = (3, 89)
Changes in v9:
- Thanks to Jacopo:
- Removed the Harness class, substituting methods with static helper functions
- Added EXPECT() for the number of buffers allocated in SimpleCapture::start()
- Changed ASSERT_FALSE() to ASSERT_EQ(..., 0) in SetUp()
- Changed tests naming
- Renamed single_stream.cpp to capture_test.cpp
- Fixed some style issues
- Added to --help that further Googletest options can be passed as environment
variables
- Thanks to Paul:
- Fixed compiling error on clang
v8: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021398.html
Changes in v8:
- Thanks to Laurent:
- Fixed lc-compliance's meson.build to disable when gtest is not installed
- Fixed compiling errors in Clang
- Thanks to Kieran:
- Fixed coding style issues
- Switched from malloc/free to new/delete in gtest parameter allocation
v7: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021253.html
Changes in v7:
- Thanks to Jacopo:
- Fixed style issue
- Made CameraManager a unique_ptr and passed to Environment as raw pointer
- Moved the check for buffers allocated to the beginning of
SimpleCapture::stop()
- Thanks to Niklas:
- Removed intermediary filter string variable in Harness::initGtestParameters()
v6: https://lists.libcamera.org/pipermail/libcamera-devel/2021-June/021137.html
Changes in v6:
- Thanks to Niklas:
- Made return of Environment::cameraId() be a const reference
- Changed buildGtestParameters() into initGtestParameters() and removed the
need for Harness::gtestArgc_ and Harness::gtestArgv_
v5: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020759.html
Changes in v5:
- Thanks to Laurent:
- Fixed style issues
- Stored CameraManager and cameraId instead of Camera in Environment, and
acquire and release the camera for each test
- Thanks to Niklas:
- Moved buildGtestParameters() inside run()
- Thanks to Laurent & Niklas:
- Improved Environment singleton class instantiation and destruction
v4: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020594.html
Changes in v4:
- Removed old lc-compliance results classes
- Thanks to Niklas:
- Added patch 3 with the environment singleton in its own files
- Improved naming of tests
- Thanks to Jacopo:
- Added \todo in Camera::stop()
v3: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020382.html
Changes in v3:
- Thanks to Niklas:
- Went back to static test registration, and created a Singleton Environment
class to store the camera global variable
- Added patch 4 implementing list and filter parameters
- Added a nameParameters() function to give meaningful names to the test
parameters, removing the need to register each test suite for every role
v2: https://lists.libcamera.org/pipermail/libcamera-devel/2021-May/020056.html
Changes in v2:
- Added patch 1 to make Camera::stop() idempotent as well from [5].
- Made test registration dynamic in patch 3
- Some other minor fixes
[5] https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019735.html
v1: https://lists.libcamera.org/pipermail/libcamera-devel/2021-April/019702.html
Nícolas F. R. A. Prado (5):
libcamera: camera: Make stop() idempotent
lc-compliance: Make SimpleCapture::stop() idempotent
lc-compliance: Add Environment singleton
lc-compliance: Refactor using Googletest
lc-compliance: Add list and filter parameters
src/lc-compliance/capture_test.cpp | 127 ++++++++++++++++++++
src/lc-compliance/environment.cpp | 20 ++++
src/lc-compliance/environment.h | 31 +++++
src/lc-compliance/main.cpp | 170 +++++++++++++++++----------
src/lc-compliance/meson.build | 8 +-
src/lc-compliance/results.cpp | 75 ------------
src/lc-compliance/results.h | 47 --------
src/lc-compliance/simple_capture.cpp | 95 ++++++---------
src/lc-compliance/simple_capture.h | 9 +-
src/lc-compliance/single_stream.cpp | 97 ---------------
src/lc-compliance/tests.h | 16 ---
src/libcamera/camera.cpp | 20 +++-
12 files changed, 346 insertions(+), 369 deletions(-)
create mode 100644 src/lc-compliance/capture_test.cpp
create mode 100644 src/lc-compliance/environment.cpp
create mode 100644 src/lc-compliance/environment.h
delete mode 100644 src/lc-compliance/results.cpp
delete mode 100644 src/lc-compliance/results.h
delete mode 100644 src/lc-compliance/single_stream.cpp
delete mode 100644 src/lc-compliance/tests.h
--
2.32.0
More information about the libcamera-devel
mailing list