[libcamera-devel] [PATCH v3 0/2] lc-compliance: Add a libcamera compliance tool

[Kieran Bingham]

Hi Niklas,

On 10/03/2021 15:44, Niklas Söderlund wrote:
> Hello,
> 
> This series adds a compliance tool to libcamera. It was developed out of
> necessity while extending and debugging the IPU3 pipeline hander. Each
> test in the tool so far have at one point triggered fatal issues in one
> pipeline or another. All known issues are however either fixed or have
> patches on ML to address them.

There's more (or at least now) but this tool has helped in flushing out
a few.

The raspberry pi pipeline handler seems to fail a few of these tests.
But all the more reason to integrate this.

I can't say I've done a detailed review but I can say this:


For both:

Acked-by: Kieran Bingham <kieran.bingham at ideasonboard.com>
Tested-by: Kieran Bingham <kieran.bingham at ideasonboard.com>


On IPU3 (with my stability branch):
120 tests executed, 96 tests passed, 24 tests skipped and 0 tests failed

On vivid:
120 tests executed, 96 tests passed, 24 tests skipped and 0 tests failed

On UVC:
120 tests executed, 96 tests passed, 24 tests skipped and 0 tests failed


On RPi:
120 tests executed, 78 tests passed, 12 tests skipped and 30 tests failed

Naush - I had to do a quick fudge fix in the configure path to get this
to run at all - so it might be interesting for you to try running this.


> The tool is simple at the moment and only tests a single stream. As our
> use-cases grow more complex I hope to find time to extend the tool to
> cover more areas.

Future feature request
  --fail-fast

So if it fails, it halts? (To help see what to fix)

> 
> Niklas Söderlund (2):
>   lc-compliance: Add a libcamera compliance tool
>   lc-compliance: Add test stopping single stream with requests queued
> 
>  src/lc-compliance/main.cpp           | 139 ++++++++++++++++++
>  src/lc-compliance/meson.build        |  25 ++++
>  src/lc-compliance/results.cpp        |  75 ++++++++++
>  src/lc-compliance/results.h          |  45 ++++++
>  src/lc-compliance/simple_capture.cpp | 212 +++++++++++++++++++++++++++
>  src/lc-compliance/simple_capture.h   |  68 +++++++++
>  src/lc-compliance/single_stream.cpp  |  98 +++++++++++++
>  src/lc-compliance/tests.h            |  16 ++
>  src/meson.build                      |   2 +
>  9 files changed, 680 insertions(+)
>  create mode 100644 src/lc-compliance/main.cpp
>  create mode 100644 src/lc-compliance/meson.build
>  create mode 100644 src/lc-compliance/results.cpp
>  create mode 100644 src/lc-compliance/results.h
>  create mode 100644 src/lc-compliance/simple_capture.cpp
>  create mode 100644 src/lc-compliance/simple_capture.h
>  create mode 100644 src/lc-compliance/single_stream.cpp
>  create mode 100644 src/lc-compliance/tests.h
> 

-- 
Regards
--
Kieran