[libcamera-devel] [PATCH] lc-compliance: Add a libcamera compliance tool
Niklas Söderlund
niklas.soderlund at ragnatech.se
Fri Jan 29 17:53:36 CET 2021
Add a compliance tool to ease testing of cameras on target. In contrast
to the unit-tests under test/ that aims to test the internal components
of libcamera the compliance aims to test application use-cases and to
some extend the public API.
This change adds the boilerplate code for a simple framework to write
tests in as well as two simple test. The tests aims both to demonstrate
the tool and to catch real problems. The tests added are:
- Test that if one queues exactly N requests to a camera exactly N
requests are eventually completed.
- Test that a configured camera can be started and stopped multiple
times in an attempt to exercise cleanup code paths otherwise not
often tested with 'cam' for example.
Example pass run on Raspberry Pi:
$ LIBCAMERA_LOG_LEVELS="*:ERROR" lc-compliance -c "/base/soc/i2c0mux/i2c at 1/imx219 at 10"
[2:36:17.672447527] [11896] ERROR V4L2 v4l2_device.cpp:191 'imx219 10-0010': Control 0x009a0922 not found
[2:36:17.673095289] [11896] ERROR V4L2 v4l2_subdevice.cpp:285 'imx219 10-0010': Unable to get rectangle 2 on pad 0: Invalid argument
Using camera /base/soc/i2c0mux/i2c at 1/imx219 at 10
Test single capture cycles
- SKIP - Camera needs 4 requests, can't test only 1
- SKIP - Camera needs 4 requests, can't test only 2
- SKIP - Camera needs 4 requests, can't test only 3
- PASS - Balanced capture of 5 requests with 1 start cycles
- PASS - Balanced capture of 8 requests with 1 start cycles
- PASS - Balanced capture of 13 requests with 1 start cycles
- PASS - Balanced capture of 21 requests with 1 start cycles
- PASS - Balanced capture of 34 requests with 1 start cycles
- PASS - Balanced capture of 55 requests with 1 start cycles
- PASS - Balanced capture of 89 requests with 1 start cycles
Test multiple start/stop cycles
- SKIP - Camera needs 4 requests, can't test only 1
- SKIP - Camera needs 4 requests, can't test only 2
- SKIP - Camera needs 4 requests, can't test only 3
- PASS - Balanced capture of 5 requests with 3 start cycles
- PASS - Balanced capture of 8 requests with 3 start cycles
- PASS - Balanced capture of 13 requests with 3 start cycles
- PASS - Balanced capture of 21 requests with 3 start cycles
- PASS - Balanced capture of 34 requests with 3 start cycles
- PASS - Balanced capture of 55 requests with 3 start cycles
- PASS - Balanced capture of 89 requests with 3 start cycles
20 tests executed, 14 tests passed, 6 tests skipped and 0 tests failed
Example fail run on IPU3:
- LIBCAMERA_LOG_LEVELS="*:ERROR" lc-compliance -c "\_SB_.PCI0.I2C2.CAM0"
- [0:31:36.992763102] [6420] ERROR V4L2 v4l2_device.cpp:191 'ov13858 8-0010': Control 0x009a0922 not found
- [0:31:36.992815436] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 2 on pad 0: Inappropriate ioctl for device
- [0:31:36.992847049] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 1 on pad 0: Inappropriate ioctl for device
- [0:31:36.992869029] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device
- [0:31:36.993152932] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device
- [0:31:36.993172833] [6420] ERROR CameraSensor camera_sensor.cpp:678 'ov13858 8-0010': The analogue crop rectangle has been defaulted to the active area size
- [0:31:36.993208149] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov13858 8-0010': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device
- [0:31:36.993224502] [6420] ERROR CameraSensor camera_sensor.cpp:678 'ov13858 8-0010': The analogue crop rectangle has been defaulted to the active area size
- [0:31:36.993354770] [6420] ERROR V4L2 v4l2_device.cpp:191 'ov5670 10-0036': Control 0x009a0922 not found
- [0:31:36.993377622] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 2 on pad 0: Inappropriate ioctl for device
- [0:31:36.993398093] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 1 on pad 0: Inappropriate ioctl for device
- [0:31:36.993415611] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device
- [0:31:36.993660192] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device
- [0:31:36.993678492] [6420] ERROR CameraSensor camera_sensor.cpp:678 'ov5670 10-0036': The analogue crop rectangle has been defaulted to the active area size
- [0:31:36.993724648] [6420] ERROR V4L2 v4l2_subdevice.cpp:285 'ov5670 10-0036': Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device
- [0:31:36.993741032] [6420] ERROR CameraSensor camera_sensor.cpp:678 'ov5670 10-0036': The analogue crop rectangle has been defaulted to the active area size
- Using camera \_SB_.PCI0.I2C2.CAM0
- Test single capture cycles
- - SKIP - Camera needs 4 requests, can't test only 1
- - SKIP - Camera needs 4 requests, can't test only 2
- - SKIP - Camera needs 4 requests, can't test only 3
- - PASS - Balanced capture of 5 requests with 1 start cycles
- - PASS - Balanced capture of 8 requests with 1 start cycles
- - PASS - Balanced capture of 13 requests with 1 start cycles
- - PASS - Balanced capture of 21 requests with 1 start cycles
- - PASS - Balanced capture of 34 requests with 1 start cycles
- [0:31:41.969783243] [6420] ERROR IPU3 cio2.cpp:264 CIO2 buffer underrun
<Application lockup here as the pipeline error is not propagated to the application>
Signed-off-by: Niklas Söderlund <niklas.soderlund at ragnatech.se>
---
Hi,
I have tested this on RkISP1, RPi and IPU3 on latest master [1]. I plan
to post separate patches which starts to address some of the problems
found while working. A fix for 'cam --captur=N' is coming shortly for
example.
I still have to find a good fix for a races found in the
IPU3 pipeline between the CIO2 and IMGU (as showed above). What I don't
(yet) have a plan for is how to fix that request queueing errors are not
propagated in our application facing API leading the test application to
wait forever to a request the library knows have failed, ideas welcome.
1. 958c80a4f1c28301 ("android: camera_device: Set AE precapture trigger according to request")
---
src/lc-compliance/main.cpp | 139 ++++++++++++++++++
src/lc-compliance/meson.build | 24 ++++
src/lc-compliance/results.cpp | 75 ++++++++++
src/lc-compliance/results.h | 45 ++++++
src/lc-compliance/single_stream.cpp | 210 ++++++++++++++++++++++++++++
src/lc-compliance/tests.h | 16 +++
src/meson.build | 2 +
7 files changed, 511 insertions(+)
create mode 100644 src/lc-compliance/main.cpp
create mode 100644 src/lc-compliance/meson.build
create mode 100644 src/lc-compliance/results.cpp
create mode 100644 src/lc-compliance/results.h
create mode 100644 src/lc-compliance/single_stream.cpp
create mode 100644 src/lc-compliance/tests.h
diff --git a/src/lc-compliance/main.cpp b/src/lc-compliance/main.cpp
new file mode 100644
index 0000000000000000..e1cbce7eac3df2bc
--- /dev/null
+++ b/src/lc-compliance/main.cpp
@@ -0,0 +1,139 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (C) 2020, Google Inc.
+ *
+ * main.cpp - lc-compliance - The libcamera compliance tool
+ */
+
+#include <iomanip>
+#include <iostream>
+#include <string.h>
+
+#include <libcamera/libcamera.h>
+
+#include "../cam/options.h"
+#include "tests.h"
+
+using namespace libcamera;
+
+class Harness
+{
+public:
+ Harness();
+ ~Harness();
+
+ int exec(int argc, char **argv);
+
+private:
+ enum {
+ OptCamera = 'c',
+ OptHelp = 'h',
+ };
+
+ int parseOptions(int argc, char **argv);
+ int init(int argc, char **argv);
+
+ OptionsParser::Options options_;
+ std::unique_ptr<CameraManager> cm_;
+ std::shared_ptr<Camera> camera_;
+};
+
+Harness::Harness()
+{
+ cm_ = std::make_unique<CameraManager>();
+}
+
+Harness::~Harness()
+{
+ if (camera_) {
+ camera_->release();
+ camera_.reset();
+ }
+
+ cm_->stop();
+}
+
+int Harness::exec(int argc, char **argv)
+{
+ int ret = init(argc, argv);
+ if (ret)
+ return ret;
+
+ std::vector<Results> results;
+
+ results.push_back(testSingleStream(camera_));
+
+ for (const Results &result : results) {
+ ret = result.summary();
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+int Harness::init(int argc, char **argv)
+{
+ int ret = parseOptions(argc, argv);
+ if (ret < 0)
+ return ret;
+
+ ret = cm_->start();
+ if (ret) {
+ std::cout << "Failed to start camera manager: "
+ << strerror(-ret) << std::endl;
+ return ret;
+ }
+
+ if (!options_.isSet(OptCamera)) {
+ std::cout << "No camera specified, available cameras:" << std::endl;
+ for (const std::shared_ptr<Camera> &cam : cm_->cameras())
+ std::cout << "- " << cam.get()->id() << std::endl;
+ return -ENODEV;
+ }
+
+ const std::string &cameraId = options_[OptCamera];
+ camera_ = cm_->get(cameraId);
+ if (!camera_) {
+ std::cout << "Camera " << cameraId << " not found, available cameras:" << std::endl;
+ for (const std::shared_ptr<Camera> &cam : cm_->cameras())
+ std::cout << "- " << cam.get()->id() << std::endl;
+ return -ENODEV;
+ }
+
+ if (camera_->acquire()) {
+ std::cout << "Failed to acquire camera" << std::endl;
+ return -EINVAL;
+ }
+
+ std::cout << "Using camera " << cameraId << std::endl;
+
+ return 0;
+}
+
+int Harness::parseOptions(int argc, char **argv)
+{
+ OptionsParser parser;
+ parser.addOption(OptCamera, OptionString,
+ "Specify which camera to operate on, by id", "camera",
+ ArgumentRequired, "camera");
+ parser.addOption(OptHelp, OptionNone, "Display this help message",
+ "help");
+
+ options_ = parser.parse(argc, argv);
+ if (!options_.valid())
+ return -EINVAL;
+
+ if (options_.empty() || options_.isSet(OptHelp)) {
+ parser.usage();
+ return options_.empty() ? -EINVAL : -EINTR;
+ }
+
+ return 0;
+}
+
+int main(int argc, char **argv)
+{
+ Harness harness;
+ return harness.exec(argc, argv) ? EXIT_FAILURE : 0;
+}
diff --git a/src/lc-compliance/meson.build b/src/lc-compliance/meson.build
new file mode 100644
index 0000000000000000..d700a307c71405b4
--- /dev/null
+++ b/src/lc-compliance/meson.build
@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: CC0-1.0
+
+libevent = dependency('libevent_pthreads', required : false)
+
+if not libevent.found()
+ warning('libevent_pthreads not found, \'lc-compliance\' application will not be compiled')
+ subdir_done()
+endif
+
+lc_compliance_sources = files([
+ '../cam/event_loop.cpp',
+ '../cam/options.cpp',
+ 'main.cpp',
+ 'results.cpp',
+ 'single_stream.cpp',
+])
+
+lc_compliance = executable('lc-compliance', lc_compliance_sources,
+ dependencies : [
+ libatomic,
+ libcamera_dep,
+ libevent,
+ ],
+ install : true)
diff --git a/src/lc-compliance/results.cpp b/src/lc-compliance/results.cpp
new file mode 100644
index 0000000000000000..fb4242bf49a3268b
--- /dev/null
+++ b/src/lc-compliance/results.cpp
@@ -0,0 +1,75 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (C) 2020, Google Inc.
+ *
+ * results.cpp - Test result aggregator
+ */
+
+#include "results.h"
+
+#include <iostream>
+
+void Results::add(const Result &result)
+{
+ if (result.first == Pass)
+ passed_++;
+ else if (result.first == Fail)
+ failed_++;
+ else if (result.first == Skip)
+ skipped_++;
+
+ printResult(result);
+}
+
+void Results::add(Status status, const std::string &message)
+{
+ add({ status, message });
+}
+
+void Results::fail(const std::string &message)
+{
+ add(Fail, message);
+}
+
+void Results::pass(const std::string &message)
+{
+ add(Pass, message);
+}
+
+void Results::skip(const std::string &message)
+{
+ add(Skip, message);
+}
+
+int Results::summary() const
+{
+ if (failed_ + passed_ + skipped_ != planned_) {
+ std::cout << "Planned and executed numer of tests differ "
+ << failed_ + passed_ + skipped_ << " executed "
+ << planned_ << " planned" << std::endl;
+
+ return -EINVAL;
+ }
+
+ std::cout << planned_ << " tests executed, "
+ << passed_ << " tests passed, "
+ << skipped_ << " tests skipped and "
+ << failed_ << " tests failed " << std::endl;
+
+ return 0;
+}
+
+void Results::printResult(const Result &result)
+{
+ std::string prefix;
+
+ /* \todo Make parsable as TAP. */
+ if (result.first == Pass)
+ prefix = "PASS";
+ else if (result.first == Fail)
+ prefix = "FAIL";
+ else if (result.first == Skip)
+ prefix = "SKIP";
+
+ std::cout << "- " << prefix << " - " << result.second << std::endl;
+}
diff --git a/src/lc-compliance/results.h b/src/lc-compliance/results.h
new file mode 100644
index 0000000000000000..a02fd5ab46edd62c
--- /dev/null
+++ b/src/lc-compliance/results.h
@@ -0,0 +1,45 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (C) 2020, Google Inc.
+ *
+ * results.h - Test result aggregator
+ */
+#ifndef __LC_COMPLIANCE_RESULTS_H__
+#define __LC_COMPLIANCE_RESULTS_H__
+
+#include <string>
+
+class Results
+{
+public:
+ enum Status {
+ Fail,
+ Pass,
+ Skip,
+ };
+
+ using Result = std::pair<Status, std::string>;
+
+ Results(unsigned int planned)
+ : planned_(planned), passed_(0), failed_(0), skipped_(0)
+ {
+ }
+
+ void add(const Result &result);
+ void add(Status status, const std::string &message);
+ void fail(const std::string &message);
+ void pass(const std::string &message);
+ void skip(const std::string &message);
+
+ int summary() const;
+
+private:
+ void printResult(const Result &result);
+
+ unsigned int planned_;
+ unsigned int passed_;
+ unsigned int failed_;
+ unsigned int skipped_;
+};
+
+#endif /* __LC_COMPLIANCE_RESULTS_H__ */
diff --git a/src/lc-compliance/single_stream.cpp b/src/lc-compliance/single_stream.cpp
new file mode 100644
index 0000000000000000..d31b6f8d06487a85
--- /dev/null
+++ b/src/lc-compliance/single_stream.cpp
@@ -0,0 +1,210 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (C) 2020, Google Inc.
+ *
+ * single_stream.cpp - Test a single camera stream
+ */
+
+#include <iostream>
+
+#include "../cam/event_loop.h"
+#include "tests.h"
+
+using namespace libcamera;
+
+class SimpleCapture
+{
+public:
+ SimpleCapture(std::shared_ptr<Camera> camera)
+ : camera_(camera), allocator_(std::make_unique<FrameBufferAllocator>(camera))
+ {
+ }
+
+ Results::Result configure(StreamRole role);
+ Results::Result start();
+ Results::Result capture(unsigned int numRequests);
+ Results::Result stop();
+
+private:
+ int queueRequest(Request *request);
+ void requestComplete(Request *request);
+
+ std::shared_ptr<libcamera::Camera> camera_;
+ std::unique_ptr<FrameBufferAllocator> allocator_;
+ std::unique_ptr<libcamera::CameraConfiguration> config_;
+
+ EventLoop *loop_;
+ unsigned int queueCount_;
+ unsigned int captureCount_;
+ unsigned int captureLimit_;
+};
+
+Results::Result SimpleCapture::configure(StreamRole role)
+{
+ config_ = camera_->generateConfiguration({ role });
+
+ if (config_->validate() != CameraConfiguration::Valid) {
+ config_.reset();
+ return { Results::Fail, "Configuration not valid" };
+ }
+
+ if (camera_->configure(config_.get())) {
+ config_.reset();
+ return { Results::Fail, "Failed to configure camera" };
+ }
+
+ return { Results::Pass, "Configure camera" };
+}
+
+Results::Result SimpleCapture::start()
+{
+ Stream *stream = config_->at(0).stream();
+ if (allocator_->allocate(stream) < 0)
+ return { Results::Fail, "Failed to allocate buffers camera" };
+
+ if (camera_->start())
+ return { Results::Fail, "Failed to start camera" };
+
+ camera_->requestCompleted.connect(this, &SimpleCapture::requestComplete);
+
+ return { Results::Pass, "Started camera" };
+}
+
+Results::Result SimpleCapture::capture(unsigned int numRequests)
+{
+ Stream *stream = config_->at(0).stream();
+ const std::vector<std::unique_ptr<FrameBuffer>> &buffers = allocator_->buffers(stream);
+
+ /* No point in testing less requests then the camera depth. */
+ if (buffers.size() > numRequests)
+ return { Results::Skip, "Camera needs " + std::to_string(buffers.size()) + " requests, can't test only " + std::to_string(numRequests) };
+
+ queueCount_ = 0;
+ captureCount_ = 0;
+ captureLimit_ = numRequests;
+
+ /* Queue the camera recommended number of reqeuests. */
+ std::vector<std::unique_ptr<libcamera::Request>> requests;
+ for (const std::unique_ptr<FrameBuffer> &buffer : allocator_->buffers(stream)) {
+ std::unique_ptr<Request> request = camera_->createRequest();
+ if (!request)
+ return { Results::Fail, "Can't create request" };
+
+ if (request->addBuffer(stream, buffer.get()))
+ return { Results::Fail, "Can't set buffer for request" };
+
+ if (queueRequest(request.get()) < 0)
+ return { Results::Fail, "Failed to queue request" };
+
+ requests.push_back(std::move(request));
+ }
+
+ /* Run capture session. */
+ loop_ = new EventLoop();
+ loop_->exec();
+ delete loop_;
+
+ if (captureCount_ != captureLimit_)
+ return { Results::Fail, "Got " + std::to_string(captureCount_) + " request, wanted " + std::to_string(captureLimit_) };
+
+ return { Results::Pass, "Balanced capture of " + std::to_string(numRequests) + " requests" };
+}
+
+Results::Result SimpleCapture::stop()
+{
+ Stream *stream = config_->at(0).stream();
+
+ camera_->stop();
+
+ camera_->requestCompleted.disconnect(this, &SimpleCapture::requestComplete);
+
+ allocator_->free(stream);
+
+ return { Results::Pass, "Stopped camera" };
+}
+
+int SimpleCapture::queueRequest(Request *request)
+{
+ queueCount_++;
+ if (queueCount_ > captureLimit_)
+ return 0;
+
+ return camera_->queueRequest(request);
+}
+
+void SimpleCapture::requestComplete(Request *request)
+{
+ captureCount_++;
+ if (captureCount_ >= captureLimit_) {
+ loop_->exit(0);
+ return;
+ }
+
+ request->reuse(Request::ReuseBuffers);
+ if (queueRequest(request))
+ loop_->exit(-EINVAL);
+}
+
+Results::Result testRequestBalance(std::shared_ptr<Camera> camera,
+ unsigned int startCycles,
+ unsigned int numRequests)
+{
+ SimpleCapture capture(camera);
+ Results::Result ret;
+
+ ret = capture.configure(Viewfinder);
+ if (ret.first != Results::Pass)
+ return ret;
+
+ for (unsigned int starts = 0; starts < startCycles; starts++) {
+ ret = capture.start();
+ if (ret.first != Results::Pass)
+ return ret;
+
+ ret = capture.capture(numRequests);
+ if (ret.first != Results::Pass) {
+ capture.stop();
+ return ret;
+ }
+
+ ret = capture.stop();
+ if (ret.first != Results::Pass)
+ return ret;
+ }
+
+ return { Results::Pass, "Balanced capture of " + std::to_string(numRequests) + " requests with " + std::to_string(startCycles) + " start cycles" };
+}
+
+Results testSingleStream(std::shared_ptr<Camera> camera)
+{
+ const std::vector<unsigned int> numRequests = { 1, 2, 3, 5, 8, 13, 21, 34, 55, 89 };
+
+ Results results(numRequests.size() * 2);
+
+ if (!camera)
+ return results;
+
+ /*
+ * Test single capture cycles
+ *
+ * Makes sure the camera completes the exact number of requests queued.
+ * Example failure is a camera that needs N+M requests queued to
+ * complete N requests to the application.
+ */
+ std::cout << "Test single capture cycles" << std::endl;
+ for (unsigned int num : numRequests)
+ results.add(testRequestBalance(camera, 1, num));
+
+ /*
+ * Test multiple start/stop cycles
+ *
+ * Makes sure the camera supports multiple start/stop cycles.
+ * Example failure is a camera that does not clean up correctly in its
+ * error path but is only tested by single-capture applications.
+ */
+ std::cout << "Test multiple start/stop cycles" << std::endl;
+ for (unsigned int num : numRequests)
+ results.add(testRequestBalance(camera, 3, num));
+
+ return results;
+}
diff --git a/src/lc-compliance/tests.h b/src/lc-compliance/tests.h
new file mode 100644
index 0000000000000000..396605214e4b8980
--- /dev/null
+++ b/src/lc-compliance/tests.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (C) 2020, Google Inc.
+ *
+ * tests.h - Test modules
+ */
+#ifndef __LC_COMPLIANCE_TESTS_H__
+#define __LC_COMPLIANCE_TESTS_H__
+
+#include <libcamera/libcamera.h>
+
+#include "results.h"
+
+Results testSingleStream(std::shared_ptr<libcamera::Camera> camera);
+
+#endif /* __LC_COMPLIANCE_TESTS_H__ */
diff --git a/src/meson.build b/src/meson.build
index 4b75f05878bcb702..a8e1af7adf2ca9c8 100644
--- a/src/meson.build
+++ b/src/meson.build
@@ -18,6 +18,8 @@ subdir('android')
subdir('libcamera')
subdir('ipa')
+subdir('lc-compliance')
+
subdir('cam')
subdir('qcam')
--
2.30.0
More information about the libcamera-devel
mailing list