[RFC] libcamera: lc-compliance: Add initial set of per frame control tests

Stefan Klug stefan.klug at ideasonboard.com
Mon Mar 4 17:27:29 CET 2024


Hi David,

Am 04.03.24 um 12:35 schrieb David Plowman:
> Hi Stefan
> 
> On Fri, 1 Mar 2024 at 13:22, Stefan Klug <stefan.klug at ideasonboard.com> wrote:
>>
>> Hi David,
>>
>> Am 01.03.24 um 13:04 schrieb David Plowman:
>>> Hi Stefan
>>>
>>> On Fri, 1 Mar 2024 at 11:22, Stefan Klug <stefan.klug at ideasonboard.com> wrote:
>>>>
>>>> Hi David,
>>>>
>>>> thanks for your comment. This is the same message I sent before, but with proper
>>>> linewrapping - sorry about that.
>>>>
>>>> Am 01.03.24 um 08:56 schrieb David Plowman:
>>>>> Hi Stefan
>>>>>
>>>>> Thanks for posting this and re-starting the discussion of per-frame controls.
>>>>>
>>>>> Could you perhaps just summarise exactly what you mean by per-frame
>>>>> controls, that is to say, what is being tested here?
>>>>>
>>>>> Eye-balling the code, I think I understood that:
>>>>>
>>>>> * You are setting controls in a specific request.
>>>>> * When that request completes, you are expecting the controls to have
>>>>> taken effect in the images that were returned with that request (but
>>>>> not earlier).
>>>>>
>>>>> But I wasn't totally sure - have I understood that correctly?
>>>>>
>>>>> Also, do we know if any other pipeline handlers implement this
>>>>> behaviour? Do folks think that all pipeline handlers should implement
>>>>> this behaviour?
>>>>>
>>>>> Thanks again!
>>>>> David
>>>>>
>>>> You are completely right. This needs a bit more context.
>>>>
>>>> It all started on my side with implementing metadata support in SimplePipeline
>>>> handler. I soon hit some corner cases where I expected things to behave
>>>> differently in cooperation with DelayedControls. So either I hit a bug in
>>>> DelayedControls or I didn't fully understand the concept behind it.
>>>>
>>>> I ended up changing several things in DelayedControls which I believe where
>>>> correct. The question then was how to prove correctness as all current users of
>>>> DelayedControls where ISP implementations. There are some unittests, but the
>>>> order of calls in these tests also felt counterintuitive (might well be, that
>>>> it's just missing knowledge on my side).
>>>>
>>>> In that area there were also no tests in lc-compliance which tested the
>>>> behaviour of an actual sensor/isp combination. So I started to write these tests
>>>> with my personal expectation of how I believe things should work. These tests
>>>> pass on my SimplePipeline. I also tried to massage the rkisp pipeline to pass
>>>> these tests and hit some corners where work would be required.
>>>>
>>>> Before digging deeper into that I believe it makes sense to see if my
>>>> expectations are correct and if a broader audience agrees to them. So here we
>>>> are, that's the reason for the RFC.
>>>>
>>>> Now on to the technical details:
>>>> Yes my expectation on per-frame-controls would be: If I queue something for
>>>> frame x, I expect that the system tries to fullfill exacty that request (and
>>>> never earlier). I'm well aware that this will not be possible in every case when
>>>> an ISP is involved, so the ISP should do the best it can, but the metadata
>>>> should reflect what was actually achieved. All the tests are currently for
>>>> manual mode, so no regulation is involved (at least for the params I test).
>>>>
>>>> Do you agree with these assumptions? Looking forward to your opinion.
>>>
>>> Thanks for the clarification, and glad that I've understood!
>>>
>>> A bit of context from our side... We looked at and implemented a
>>> per-frame controls solution for the Raspberry Pi back in summer 2022.
>>> I would say there were a couple of differences:
>>>
>>> 1. We included a mechanism for reporting failure cases. The only real
>>> failure case we have is failing to update camera exposure/gain
>>> settings in time because there is quite a short window in which to
>>> service the camera frame interrupts (especially when the system is
>>> busy and the framerate is fast).
>>
>> That's interesting. Could you explain the mechanics you used for that
>> feedback or point me to a patchset? Would be interesting to see which
>> failure cases you modelled. How did you test these?
> 
> The last time we did any work on this was last summer for a libcamera
> F2F in Prague. I think this
> https://github.com/naushir/libcamera/tree/pfc_update was the code we
> were running at that time. Obviously things have moved on since then,
> not least the existence of Pi 5, which will need handling as well.
> 
> The way the feedback worked was that you got a sequence number every
> time you submitted a request with controls in it (distinct from the
> request sequence number), which we called the "submit id". Every
> request that completed contained one of these sequence numbers
> indicating which the most recent controls were that had been applied,
> and this was known as the "sync id". So to ask "does this request
> implement the controls that I set in it when I queued it?" you would
> check "requset->syncId == request->submitId".
> 
>>>
>>> The mechanism allowed the application to check whether controls had
>>> been applied as expected, or whether they'd been delayed to the next
>>> frame. It told you whether they'd got "merged" with the next set of
>>> controls, or whether all the controls are now running "a frame late".
>>>
>>> 2. We implemented our scheme in the same way as you've described,
>>> where the request where you set the controls has the first images that
>>> fulfill the request. We called this "Android mode" though tbh I'm not
>>> sure whether or not this really is what Android does!!!
>>>
>>> Although in the end we decided we didn't like this behaviour so much
>>> because controls get delayed to the back of all the requests that have
>>> been queued (and we normally queue quite a lot so as to avoid the risk
>>
>> Why was it necessary to move them to the back of all request? I guess in
>> a typical usecase most requests would not contain controls (at least not
>> from user side). So there would be a way to apply them as early in the
>> queue as possible. I guess I'm missing somethimng here.
> 
> Sorry, I'm not being super clear! We never "moved" controls to the
> back of the requests, the problem is that when you queue a request
> with some controls in it, those controls are necessarily behind all
> the other requests that you queued previously. So you have potentially
> many extra frames of latency.
> 
> But you're exactly right, most requests don't contain any controls, so
> we used to move the controls up the request queue so that they would
> happen earlier.

Now I got it! Thanks. That makes perfect sense now.

> 
>>
>>> of frame drops). So we also added "Raspberry Pi mode" where controls
>>> are applied earlier.
>>>
>>> In principle I would be happy to let the Pi run in either mode
>>> ("Android" or "Pi"), so long as we can choose "Pi mode" for our
>>> applications.  In both cases the same reporting mechanism told the
>>> application exactly when the controls had been applied, so that
>>> applications could be "mode" agnostic.
>>>
>>> Since Summer 2022, per-frame controls have not really progressed so
>>> far as I know. We are intending to resurrect our per-frame controls
>>> implementation again once we've got over the Pi 5 libcamera
>>> integration, so I'm very happy to be resuming this discussion.
>>
>> Sure. Would be great if we could come up with a set of tests that wee
>> all agree on.
> 
> I agree! Though the last 18 months suggests to me that we're not there yet.
> 
> There's currently work going on that splits buffers out of requests.
> To me, the problem with the control lists suggests that possibly
> controls should have their own queues and be removed from requests as
> well. At which point this all becomes an even bigger discussion.
> 
> So I'm kind of sorry to be listing all the ways in which per-frame
> controls are an unresolved problem, but not really knowing where it
> goes from here. Nonetheless, we are wanting to move forward once the
> Pi 5 integration is out of the way.

Don't feel sorry. At least I'm aware of some of the nasty details now :-)

Stefan

> 
> David
> 
>>
>> Cheers,
>> Stefan
>>
>>>
>>> Thanks!
>>> David
>>>
>>>>
>>>> Best regards,
>>>> Stefan
>>>>
>>>>>
>>>>> On Thu, 29 Feb 2024 at 17:01, Stefan Klug <stefan.klug at ideasonboard.com> wrote:
>>>>>>
>>>>>> These tests check if controls (only exposure time and analogue gain at
>>>>>> the moment) get applied on the frame they were requested for.
>>>>>>
>>>>>> This is tested by looking at the metadata and additionally by
>>>>>> calculating a mean brightness on a centered rect of 20x20 pixels.
>>>>>>
>>>>>> Until today, these tests where only run on a project specific branch
>>>>>> with a modified simple pipeline. In theory they should pass on a
>>>>>> current master :-)
>>>>>>
>>>>>> Current test setup: imx219 with simple pipeline on an imx8mp.
>>>>>> Modifications of either the exposure delay or the gain delay in
>>>>>> the camera_sensor class resulted in test failures.
>>>>>> Which is exactly what this test shall proove.
>>>>>>
>>>>>> Signed-off-by: Stefan Klug <stefan.klug at ideasonboard.com>
>>>>>> ---
>>>>>>  src/apps/lc-compliance/capture_test.cpp       |  39 ++++
>>>>>>  src/apps/lc-compliance/meson.build            |   2 +
>>>>>>  src/apps/lc-compliance/per_frame_controls.cpp | 214 ++++++++++++++++++
>>>>>>  src/apps/lc-compliance/per_frame_controls.h   |  41 ++++
>>>>>>  src/apps/lc-compliance/simple_capture.cpp     |   4 +-
>>>>>>  src/apps/lc-compliance/simple_capture.h       |   2 +-
>>>>>>  src/apps/lc-compliance/time_sheet.cpp         | 135 +++++++++++
>>>>>>  src/apps/lc-compliance/time_sheet.h           |  53 +++++
>>>>>>  8 files changed, 487 insertions(+), 3 deletions(-)
>>>>>>  create mode 100644 src/apps/lc-compliance/per_frame_controls.cpp
>>>>>>  create mode 100644 src/apps/lc-compliance/per_frame_controls.h
>>>>>>  create mode 100644 src/apps/lc-compliance/time_sheet.cpp
>>>>>>  create mode 100644 src/apps/lc-compliance/time_sheet.h
>>>>>>
>>>>>> diff --git a/src/apps/lc-compliance/capture_test.cpp b/src/apps/lc-compliance/capture_test.cpp
>>>>>> index 1dcfcf92..43fe59f3 100644
>>>>>> --- a/src/apps/lc-compliance/capture_test.cpp
>>>>>> +++ b/src/apps/lc-compliance/capture_test.cpp
>>>>>> @@ -11,6 +11,7 @@
>>>>>>  #include <gtest/gtest.h>
>>>>>>
>>>>>>  #include "environment.h"
>>>>>> +#include "per_frame_controls.h"
>>>>>>  #include "simple_capture.h"
>>>>>>
>>>>>>  using namespace libcamera;
>>>>>> @@ -133,3 +134,41 @@ INSTANTIATE_TEST_SUITE_P(CaptureTests,
>>>>>>                          testing::Combine(testing::ValuesIn(ROLES),
>>>>>>                                           testing::ValuesIn(NUMREQUESTS)),
>>>>>>                          SingleStream::nameParameters);
>>>>>> +
>>>>>> +/*
>>>>>> + * Test Per frame controls
>>>>>> + */
>>>>>> +TEST_F(SingleStream, testFramePreciseExposureChange)
>>>>>> +{
>>>>>> +       PerFrameControls capture(camera_);
>>>>>> +       capture.configure(StreamRole::Viewfinder);
>>>>>> +       capture.testFramePreciseExposureChange();
>>>>>> +}
>>>>>> +
>>>>>> +TEST_F(SingleStream, testFramePreciseGainChange)
>>>>>> +{
>>>>>> +       PerFrameControls capture(camera_);
>>>>>> +       capture.configure(StreamRole::Viewfinder);
>>>>>> +       capture.testFramePreciseGainChange();
>>>>>> +}
>>>>>> +
>>>>>> +TEST_F(SingleStream, testExposureGainIsAppliedOnFirstFrame)
>>>>>> +{
>>>>>> +       PerFrameControls capture(camera_);
>>>>>> +       capture.configure(StreamRole::Viewfinder);
>>>>>> +       capture.testExposureGainIsAppliedOnFirstFrame();
>>>>>> +}
>>>>>> +
>>>>>> +TEST_F(SingleStream, testExposureGainFromFirstRequestGetsApplied)
>>>>>> +{
>>>>>> +       PerFrameControls capture(camera_);
>>>>>> +       capture.configure(StreamRole::Viewfinder);
>>>>>> +       capture.testExposureGainFromFirstRequestGetsApplied();
>>>>>> +}
>>>>>> +
>>>>>> +TEST_F(SingleStream, testExposureGainFromFirstAndSecondRequestGetsApplied)
>>>>>> +{
>>>>>> +       PerFrameControls capture(camera_);
>>>>>> +       capture.configure(StreamRole::Viewfinder);
>>>>>> +       capture.testExposureGainFromFirstAndSecondRequestGetsApplied();
>>>>>> +}
>>>>>> diff --git a/src/apps/lc-compliance/meson.build b/src/apps/lc-compliance/meson.build
>>>>>> index c792f072..2a6f52af 100644
>>>>>> --- a/src/apps/lc-compliance/meson.build
>>>>>> +++ b/src/apps/lc-compliance/meson.build
>>>>>> @@ -15,7 +15,9 @@ lc_compliance_sources = files([
>>>>>>      'capture_test.cpp',
>>>>>>      'environment.cpp',
>>>>>>      'main.cpp',
>>>>>> +    'per_frame_controls.cpp',
>>>>>>      'simple_capture.cpp',
>>>>>> +    'time_sheet.cpp',
>>>>>>  ])
>>>>>>
>>>>>>  lc_compliance  = executable('lc-compliance', lc_compliance_sources,
>>>>>> diff --git a/src/apps/lc-compliance/per_frame_controls.cpp b/src/apps/lc-compliance/per_frame_controls.cpp
>>>>>> new file mode 100644
>>>>>> index 00000000..70fc44ac
>>>>>> --- /dev/null
>>>>>> +++ b/src/apps/lc-compliance/per_frame_controls.cpp
>>>>>> @@ -0,0 +1,214 @@
>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */
>>>>>> +/*
>>>>>> + * Copyright (C) 2024, Ideas on Board Oy
>>>>>> + *
>>>>>> + * per_frame_controls.cpp - Tests for per frame controls
>>>>>> + */
>>>>>> +#include "per_frame_controls.h"
>>>>>> +
>>>>>> +#include <gtest/gtest.h>
>>>>>> +
>>>>>> +#include "time_sheet.h"
>>>>>> +
>>>>>> +using namespace libcamera;
>>>>>> +
>>>>>> +PerFrameControls::PerFrameControls(std::shared_ptr<Camera> camera)
>>>>>> +       : SimpleCapture(camera)
>>>>>> +{
>>>>>> +}
>>>>>> +
>>>>>> +std::shared_ptr<TimeSheet> PerFrameControls::startCaptureWithTimeSheet(unsigned int framesToCapture, const ControlList *controls)
>>>>>> +{
>>>>>> +       ControlList ctrls(camera_->controls().idmap());
>>>>>> +       /* Ensure defined default values */
>>>>>> +       ctrls.set(controls::AeEnable, false);
>>>>>> +       ctrls.set(controls::AeExposureMode, controls::ExposureCustom);
>>>>>> +       ctrls.set(controls::ExposureTime, 10000);
>>>>>> +       ctrls.set(controls::AnalogueGain, 1.0);
>>>>>> +
>>>>>> +       if (controls) {
>>>>>> +               ctrls.merge(*controls, true);
>>>>>> +       }
>>>>>> +
>>>>>> +       start(&ctrls);
>>>>>> +
>>>>>> +       queueCount_ = 0;
>>>>>> +       captureCount_ = 0;
>>>>>> +       captureLimit_ = framesToCapture;
>>>>>> +
>>>>>> +       auto timeSheet = std::make_shared<TimeSheet>(captureLimit_, camera_->controls().idmap());
>>>>>> +       timeSheet_ = timeSheet;
>>>>>> +       return timeSheet;
>>>>>> +}
>>>>>> +
>>>>>> +int PerFrameControls::queueRequest(Request *request)
>>>>>> +{
>>>>>> +       queueCount_++;
>>>>>> +       if (queueCount_ > captureLimit_)
>>>>>> +               return 0;
>>>>>> +
>>>>>> +       auto ts = timeSheet_.lock();
>>>>>> +       if (ts) {
>>>>>> +               ts->prepareForQueue(request, queueCount_ - 1);
>>>>>> +       }
>>>>>> +
>>>>>> +       return camera_->queueRequest(request);
>>>>>> +}
>>>>>> +
>>>>>> +void PerFrameControls::requestComplete(Request *request)
>>>>>> +{
>>>>>> +       auto ts = timeSheet_.lock();
>>>>>> +       if (ts) {
>>>>>> +               ts->handleCompleteRequest(request);
>>>>>> +       }
>>>>>> +
>>>>>> +       captureCount_++;
>>>>>> +       if (captureCount_ >= captureLimit_) {
>>>>>> +               loop_->exit(0);
>>>>>> +               return;
>>>>>> +       }
>>>>>> +
>>>>>> +       request->reuse(Request::ReuseBuffers);
>>>>>> +       if (queueRequest(request))
>>>>>> +               loop_->exit(-EINVAL);
>>>>>> +}
>>>>>> +
>>>>>> +void PerFrameControls::runCaptureSession()
>>>>>> +{
>>>>>> +       Stream *stream = config_->at(0).stream();
>>>>>> +       const std::vector<std::unique_ptr<FrameBuffer>> &buffers = allocator_->buffers(stream);
>>>>>> +
>>>>>> +       /* Queue the recommended number of reqeuests. */
>>>>>> +       for (const std::unique_ptr<FrameBuffer> &buffer : buffers) {
>>>>>> +               std::unique_ptr<Request> request = camera_->createRequest();
>>>>>> +               request->addBuffer(stream, buffer.get());
>>>>>> +               queueRequest(request.get());
>>>>>> +               requests_.push_back(std::move(request));
>>>>>> +       }
>>>>>> +
>>>>>> +       /* Run capture session. */
>>>>>> +       loop_ = new EventLoop();
>>>>>> +       loop_->exec();
>>>>>> +       stop();
>>>>>> +       delete loop_;
>>>>>> +}
>>>>>> +
>>>>>> +void PerFrameControls::testFramePreciseExposureChange()
>>>>>> +{
>>>>>> +       auto timeSheet = startCaptureWithTimeSheet(10);
>>>>>> +       auto &ts = *timeSheet;
>>>>>> +
>>>>>> +
>>>>>> +       ts[3].controls().set(controls::ExposureTime, 5000);
>>>>>> +       //wait a few frames to settle
>>>>>> +       ts[6].controls().set(controls::ExposureTime, 20000);
>>>>>> +       ts.printAllInfos();
>>>>>> +
>>>>>> +       runCaptureSession();
>>>>>> +
>>>>>> +       EXPECT_NEAR(ts[5].metadata().get(controls::ExposureTime).value(), 5000, 20);
>>>>>> +       EXPECT_NEAR(ts[6].metadata().get(controls::ExposureTime).value(), 20000, 20);
>>>>>> +
>>>>>> +       /* No increase just before setting exposure */
>>>>>> +       EXPECT_NEAR(ts[5].getBrightnessChange(), 1.0, 0.05) << "Brightness changed too much before the expected time of change (control delay too high?).";
>>>>>> +       /*
>>>>>> +     * Todo: The change is brightness was a bit low (Exposure time increase by 4x resulted in a brightness increase of < 2).
>>>>>> +     * This should be investigated.
>>>>>> +    */
>>>>>> +       EXPECT_GT(ts[6].getBrightnessChange(), 1.3) << "Brightness in frame " << 6 << " did not increase as expected (reference: "
>>>>>> +                                                   << ts[3].getSpotBrightness() << " current: " << ts[6].getSpotBrightness() << " )" << std::endl;
>>>>>> +
>>>>>> +       /* No increase just after setting exposure */
>>>>>> +       EXPECT_NEAR(ts[7].getBrightnessChange(), 1.0, 0.05) << "Brightness changed too much after the expected time of change (control delay too low?).";
>>>>>> +
>>>>>> +       /* No increase just after setting exposure */
>>>>>> +       EXPECT_NEAR(ts[8].getBrightnessChange(), 1.0, 0.05) << "Brightness changed too much2 frames after the expected time of change (control delay too low?).";
>>>>>> +}
>>>>>> +
>>>>>> +void PerFrameControls::testFramePreciseGainChange()
>>>>>> +{
>>>>>> +       auto timeSheet = startCaptureWithTimeSheet(10);
>>>>>> +       auto &ts = *timeSheet;
>>>>>> +
>>>>>> +       ts[3].controls().set(controls::AnalogueGain, 1.0);
>>>>>> +       //wait a few frames to settle
>>>>>> +       ts[6].controls().set(controls::AnalogueGain, 4.0);
>>>>>> +
>>>>>> +       runCaptureSession();
>>>>>> +
>>>>>> +       EXPECT_NEAR(ts[5].metadata().get(controls::AnalogueGain).value(), 1.0, 0.1);
>>>>>> +       EXPECT_NEAR(ts[6].metadata().get(controls::AnalogueGain).value(), 4.0, 0.1);
>>>>>> +
>>>>>> +       /* No increase just before setting gain */
>>>>>> +       EXPECT_NEAR(ts[5].getBrightnessChange(), 1.0, 0.05) << "Brightness changed too much before the expected time of change (control delay too high?).";
>>>>>> +       /*
>>>>>> +     * Todo: I see a brightness change of roughly half the expected one. This is not yet understood and needs investigation
>>>>>> +    */
>>>>>> +       EXPECT_GT(ts[6].getBrightnessChange(), 1.7) << "Brightness in frame " << 6 << " did not increase as expected (reference: "
>>>>>> +                                                   << ts[5].getSpotBrightness() << " current: " << ts[6].getSpotBrightness() << " )" << std::endl;
>>>>>> +
>>>>>> +       /* No increase just after setting gain */
>>>>>> +       EXPECT_NEAR(ts[7].getBrightnessChange(), 1.0, 0.05) << "Brightness changed too much after the expected time of change (control delay too low?).";
>>>>>> +}
>>>>>> +
>>>>>> +void PerFrameControls::testExposureGainFromFirstRequestGetsApplied()
>>>>>> +{
>>>>>> +       auto timeSheet = startCaptureWithTimeSheet(5);
>>>>>> +       auto &ts = *timeSheet;
>>>>>> +
>>>>>> +       ts[0].controls().set(controls::ExposureTime, 10000);
>>>>>> +       ts[0].controls().set(controls::AnalogueGain, 4.0);
>>>>>> +
>>>>>> +       runCaptureSession();
>>>>>> +
>>>>>> +       /* We expect it to be applied after 3 frames, the latest*/
>>>>>> +       EXPECT_NEAR(ts[4].metadata().get(controls::ExposureTime).value(), 10000, 20);
>>>>>> +       EXPECT_NEAR(ts[4].metadata().get(controls::AnalogueGain).value(), 4.0, 0.1);
>>>>>> +}
>>>>>> +
>>>>>> +void PerFrameControls::testExposureGainFromFirstAndSecondRequestGetsApplied()
>>>>>> +{
>>>>>> +       auto timeSheet = startCaptureWithTimeSheet(5);
>>>>>> +       auto &ts = *timeSheet;
>>>>>> +
>>>>>> +       ts[0].controls().set(controls::ExposureTime, 8000);
>>>>>> +       ts[0].controls().set(controls::AnalogueGain, 2.0);
>>>>>> +       ts[1].controls().set(controls::ExposureTime, 10000);
>>>>>> +       ts[1].controls().set(controls::AnalogueGain, 4.0);
>>>>>> +
>>>>>> +       runCaptureSession();
>>>>>> +
>>>>>> +       /* We expect it to be applied after 3 frames, the latest*/
>>>>>> +       EXPECT_NEAR(ts[4].metadata().get(controls::ExposureTime).value(), 10000, 20);
>>>>>> +       EXPECT_NEAR(ts[4].metadata().get(controls::AnalogueGain).value(), 4.0, 0.1);
>>>>>> +}
>>>>>> +
>>>>>> +void PerFrameControls::testExposureGainIsAppliedOnFirstFrame()
>>>>>> +{
>>>>>> +       ControlList startValues;
>>>>>> +       startValues.set(controls::ExposureTime, 5000);
>>>>>> +       startValues.set(controls::AnalogueGain, 1.0);
>>>>>> +
>>>>>> +       auto ts1 = startCaptureWithTimeSheet(3, &startValues);
>>>>>> +
>>>>>> +       runCaptureSession();
>>>>>> +
>>>>>> +       EXPECT_NEAR((*ts1)[0].metadata().get(controls::ExposureTime).value(), 5000, 20);
>>>>>> +       EXPECT_NEAR((*ts1)[0].metadata().get(controls::AnalogueGain).value(), 1.0, 0.01);
>>>>>> +
>>>>>> +       /* Second capture with different values to ensure we don't hit default/old values */
>>>>>> +
>>>>>> +       startValues.set(controls::ExposureTime, 15000);
>>>>>> +       startValues.set(controls::AnalogueGain, 4.0);
>>>>>> +
>>>>>> +       auto ts2 = startCaptureWithTimeSheet(3, &startValues);
>>>>>> +
>>>>>> +       runCaptureSession();
>>>>>> +
>>>>>> +       EXPECT_NEAR((*ts2)[0].metadata().get(controls::ExposureTime).value(), 15000, 20);
>>>>>> +       EXPECT_NEAR((*ts2)[0].metadata().get(controls::AnalogueGain).value(), 4.0, 0.01);
>>>>>> +
>>>>>> +       /* with 3x exposure and 4x gain we could expect a brightness increase of 2x */
>>>>>> +       double brightnessChange = ts2->get(1).getSpotBrightness() / ts1->get(1).getSpotBrightness();
>>>>>> +       EXPECT_GT(brightnessChange, 2.0);
>>>>>> +}
>>>>>> diff --git a/src/apps/lc-compliance/per_frame_controls.h b/src/apps/lc-compliance/per_frame_controls.h
>>>>>> new file mode 100644
>>>>>> index 00000000..e783f024
>>>>>> --- /dev/null
>>>>>> +++ b/src/apps/lc-compliance/per_frame_controls.h
>>>>>> @@ -0,0 +1,41 @@
>>>>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */
>>>>>> +/*
>>>>>> + * Copyright (C) 2024, Ideas on Board Oy
>>>>>> + *
>>>>>> + * per_frame_controls.h - Tests for per frame controls
>>>>>> + */
>>>>>> +
>>>>>> +#pragma once
>>>>>> +
>>>>>> +#include <memory>
>>>>>> +
>>>>>> +#include <libcamera/libcamera.h>
>>>>>> +
>>>>>> +#include "../common/event_loop.h"
>>>>>> +
>>>>>> +#include "simple_capture.h"
>>>>>> +#include "time_sheet.h"
>>>>>> +
>>>>>> +class PerFrameControls : public SimpleCapture
>>>>>> +{
>>>>>> +public:
>>>>>> +       PerFrameControls(std::shared_ptr<libcamera::Camera> camera);
>>>>>> +
>>>>>> +       std::shared_ptr<TimeSheet> startCaptureWithTimeSheet(unsigned int framesToCapture, const libcamera::ControlList *controls = nullptr);
>>>>>> +       void runCaptureSession();
>>>>>> +
>>>>>> +       void testFramePreciseExposureChange();
>>>>>> +       void testFramePreciseGainChange();
>>>>>> +       void testExposureGainIsAppliedOnFirstFrame();
>>>>>> +       void testExposureGainFromFirstRequestGetsApplied();
>>>>>> +       void testExposureGainFromFirstAndSecondRequestGetsApplied();
>>>>>> +
>>>>>> +       int queueRequest(libcamera::Request *request);
>>>>>> +       void requestComplete(libcamera::Request *request) override;
>>>>>> +
>>>>>> +       unsigned int queueCount_;
>>>>>> +       unsigned int captureCount_;
>>>>>> +       unsigned int captureLimit_;
>>>>>> +
>>>>>> +       std::weak_ptr<TimeSheet> timeSheet_;
>>>>>> +};
>>>>>> diff --git a/src/apps/lc-compliance/simple_capture.cpp b/src/apps/lc-compliance/simple_capture.cpp
>>>>>> index cf4d7cf3..56680a83 100644
>>>>>> --- a/src/apps/lc-compliance/simple_capture.cpp
>>>>>> +++ b/src/apps/lc-compliance/simple_capture.cpp
>>>>>> @@ -42,7 +42,7 @@ void SimpleCapture::configure(StreamRole role)
>>>>>>         }
>>>>>>  }
>>>>>>
>>>>>> -void SimpleCapture::start()
>>>>>> +void SimpleCapture::start(const ControlList *controls)
>>>>>>  {
>>>>>>         Stream *stream = config_->at(0).stream();
>>>>>>         int count = allocator_->allocate(stream);
>>>>>> @@ -52,7 +52,7 @@ void SimpleCapture::start()
>>>>>>
>>>>>>         camera_->requestCompleted.connect(this, &SimpleCapture::requestComplete);
>>>>>>
>>>>>> -       ASSERT_EQ(camera_->start(), 0) << "Failed to start camera";
>>>>>> +       ASSERT_EQ(camera_->start(controls), 0) << "Failed to start camera";
>>>>>>  }
>>>>>>
>>>>>>  void SimpleCapture::stop()
>>>>>> diff --git a/src/apps/lc-compliance/simple_capture.h b/src/apps/lc-compliance/simple_capture.h
>>>>>> index 2911d601..54b1d54b 100644
>>>>>> --- a/src/apps/lc-compliance/simple_capture.h
>>>>>> +++ b/src/apps/lc-compliance/simple_capture.h
>>>>>> @@ -22,7 +22,7 @@ protected:
>>>>>>         SimpleCapture(std::shared_ptr<libcamera::Camera> camera);
>>>>>>         virtual ~SimpleCapture();
>>>>>>
>>>>>> -       void start();
>>>>>> +       void start(const libcamera::ControlList *controls = nullptr);
>>>>>>         void stop();
>>>>>>
>>>>>>         virtual void requestComplete(libcamera::Request *request) = 0;
>>>>>> diff --git a/src/apps/lc-compliance/time_sheet.cpp b/src/apps/lc-compliance/time_sheet.cpp
>>>>>> new file mode 100644
>>>>>> index 00000000..9a0e6544
>>>>>> --- /dev/null
>>>>>> +++ b/src/apps/lc-compliance/time_sheet.cpp
>>>>>> @@ -0,0 +1,135 @@
>>>>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */
>>>>>> +/*
>>>>>> + * Copyright (C) 2024, Ideas on Board Oy
>>>>>> + *
>>>>>> + * time_sheet.cpp
>>>>>> + */
>>>>>> +#include "time_sheet.h"
>>>>>> +
>>>>>> +#include <sstream>
>>>>>> +#include <libcamera/libcamera.h>
>>>>>> +
>>>>>> +#include "libcamera/internal/formats.h"
>>>>>> +#include "libcamera/internal/mapped_framebuffer.h"
>>>>>> +
>>>>>> +using namespace libcamera;
>>>>>> +
>>>>>> +double calcPixelMeanNV12(const uint8_t *data)
>>>>>> +{
>>>>>> +       return (double)*data;
>>>>>> +}
>>>>>> +
>>>>>> +double calcPixelMeanRAW10(const uint8_t *data)
>>>>>> +{
>>>>>> +       return (double)*((const uint16_t *)data);
>>>>>> +}
>>>>>> +
>>>>>> +double calculateMeanBrightnessFromCenterSpot(libcamera::Request *request)
>>>>>> +{
>>>>>> +       const Request::BufferMap &buffers = request->buffers();
>>>>>> +       for (const auto &[stream, buffer] : buffers) {
>>>>>> +               MappedFrameBuffer in(buffer, MappedFrameBuffer::MapFlag::Read);
>>>>>> +               if (in.isValid()) {
>>>>>> +                       auto data = in.planes()[0].data();
>>>>>> +                       auto streamConfig = stream->configuration();
>>>>>> +                       auto formatInfo = PixelFormatInfo::info(streamConfig.pixelFormat);
>>>>>> +
>>>>>> +                       std::function<double(const uint8_t *data)> calcPixelMean;
>>>>>> +                       int pixelStride;
>>>>>> +
>>>>>> +                       switch (streamConfig.pixelFormat) {
>>>>>> +                       case formats::NV12:
>>>>>> +                               calcPixelMean = calcPixelMeanNV12;
>>>>>> +                               pixelStride = 1;
>>>>>> +                               break;
>>>>>> +                       case formats::SRGGB10:
>>>>>> +                               calcPixelMean = calcPixelMeanRAW10;
>>>>>> +                               pixelStride = 2;
>>>>>> +                               break;
>>>>>> +                       default:
>>>>>> +                               std::stringstream s;
>>>>>> +                               s << "Unsupported Pixelformat " << formatInfo.name;
>>>>>> +                               throw std::invalid_argument(s.str());
>>>>>> +                       }
>>>>>> +
>>>>>> +                       double sum = 0;
>>>>>> +                       int w = 20;
>>>>>> +                       int xs = streamConfig.size.width / 2 - w / 2;
>>>>>> +                       int ys = streamConfig.size.height / 2 - w / 2;
>>>>>> +
>>>>>> +                       for (auto y = ys; y < ys + w; y++) {
>>>>>> +                               auto line = data + y * streamConfig.stride;
>>>>>> +                               for (auto x = xs; x < xs + w; x++) {
>>>>>> +                                       sum += calcPixelMean(line + x * pixelStride);
>>>>>> +                               }
>>>>>> +                       }
>>>>>> +                       sum = sum / (w * w);
>>>>>> +                       return sum;
>>>>>> +               }
>>>>>> +       }
>>>>>> +       return 0;
>>>>>> +}
>>>>>> +
>>>>>> +TimeSheetEntry::TimeSheetEntry(const ControlIdMap &idmap)
>>>>>> +       : controls_(idmap)
>>>>>> +{
>>>>>> +}
>>>>>> +
>>>>>> +void TimeSheetEntry::handleCompleteRequest(libcamera::Request *request, const TimeSheetEntry *previous)
>>>>>> +{
>>>>>> +       metadata_ = request->metadata();
>>>>>> +
>>>>>> +       spotBrightness_ = calculateMeanBrightnessFromCenterSpot(request);
>>>>>> +       if (previous) {
>>>>>> +               brightnessChange_ = spotBrightness_ / previous->getSpotBrightness();
>>>>>> +       }
>>>>>> +       sequence_ = request->sequence();
>>>>>> +}
>>>>>> +
>>>>>> +void TimeSheetEntry::printInfo()
>>>>>> +{
>>>>>> +       std::cout << "=== Frame " << sequence_ << std::endl;
>>>>>> +       std::cout << "Brightness: " << spotBrightness_ << std::endl;
>>>>>> +
>>>>>> +       if (!metadata_.empty()) {
>>>>>> +               std::cout << "Metadata:" << std::endl;
>>>>>> +               auto idMap = metadata_.idMap();
>>>>>> +               assert(idMap);
>>>>>> +               for (const auto &[id, value] : metadata_) {
>>>>>> +                       std::cout << "  " << idMap->at(id)->name() << " : " << value.toString() << std::endl;
>>>>>> +               }
>>>>>> +       }
>>>>>> +}
>>>>>> +
>>>>>> +TimeSheetEntry &TimeSheet::get(size_t pos)
>>>>>> +{
>>>>>> +       auto &entry = entries_[pos];
>>>>>> +       if (!entry)
>>>>>> +               entry = std::make_shared<TimeSheetEntry>(idmap_);
>>>>>> +       return *entry;
>>>>>> +}
>>>>>> +
>>>>>> +void TimeSheet::prepareForQueue(libcamera::Request *request, uint32_t sequence)
>>>>>> +{
>>>>>> +       request->controls() = get(sequence).controls();
>>>>>> +}
>>>>>> +
>>>>>> +void TimeSheet::handleCompleteRequest(libcamera::Request *request)
>>>>>> +{
>>>>>> +       uint32_t sequence = request->sequence();
>>>>>> +       auto &entry = get(sequence);
>>>>>> +       TimeSheetEntry *previous = nullptr;
>>>>>> +       if (sequence >= 1) {
>>>>>> +               previous = entries_[sequence - 1].get();
>>>>>> +       }
>>>>>> +
>>>>>> +       entry.handleCompleteRequest(request, previous);
>>>>>> +}
>>>>>> +
>>>>>> +void TimeSheet::printAllInfos()
>>>>>> +{
>>>>>> +       for (auto entry : entries_) {
>>>>>> +               if (entry)
>>>>>> +                       entry->printInfo();
>>>>>> +       }
>>>>>> +}
>>>>>> diff --git a/src/apps/lc-compliance/time_sheet.h b/src/apps/lc-compliance/time_sheet.h
>>>>>> new file mode 100644
>>>>>> index 00000000..c155763c
>>>>>> --- /dev/null
>>>>>> +++ b/src/apps/lc-compliance/time_sheet.h
>>>>>> @@ -0,0 +1,53 @@
>>>>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */
>>>>>> +/*
>>>>>> + * Copyright (C) 2024, Ideas on Board Oy
>>>>>> + *
>>>>>> + * time_sheet.h
>>>>>> + */
>>>>>> +
>>>>>> +#pragma once
>>>>>> +
>>>>>> +#include <future>
>>>>>> +#include <vector>
>>>>>> +
>>>>>> +#include <libcamera/libcamera.h>
>>>>>> +
>>>>>> +class TimeSheetEntry
>>>>>> +{
>>>>>> +public:
>>>>>> +       TimeSheetEntry(const libcamera::ControlIdMap &idmap);
>>>>>> +       TimeSheetEntry(TimeSheetEntry &&other) noexcept = default;
>>>>>> +       TimeSheetEntry(const TimeSheetEntry &) = delete;
>>>>>> +
>>>>>> +       libcamera::ControlList &controls() { return controls_; };
>>>>>> +       libcamera::ControlList &metadata() { return metadata_; };
>>>>>> +       void handleCompleteRequest(libcamera::Request *request, const TimeSheetEntry *previous);
>>>>>> +       void printInfo();
>>>>>> +       double getSpotBrightness() const { return spotBrightness_; };
>>>>>> +       double getBrightnessChange() const { return brightnessChange_; };
>>>>>> +
>>>>>> +private:
>>>>>> +       double spotBrightness_ = 0.0;
>>>>>> +       double brightnessChange_ = 0.0;
>>>>>> +       libcamera::ControlList controls_;
>>>>>> +       libcamera::ControlList metadata_;
>>>>>> +       uint32_t sequence_ = 0;
>>>>>> +};
>>>>>> +
>>>>>> +class TimeSheet
>>>>>> +{
>>>>>> +public:
>>>>>> +       TimeSheet(int count, const libcamera::ControlIdMap &idmap)
>>>>>> +               : idmap_(idmap), entries_(count){};
>>>>>> +
>>>>>> +       void prepareForQueue(libcamera::Request *request, uint32_t sequence);
>>>>>> +       void handleCompleteRequest(libcamera::Request *request);
>>>>>> +       void printAllInfos();
>>>>>> +
>>>>>> +       TimeSheetEntry &operator[](size_t pos) { return get(pos); };
>>>>>> +       TimeSheetEntry &get(size_t pos);
>>>>>> +
>>>>>> +private:
>>>>>> +       const libcamera::ControlIdMap &idmap_;
>>>>>> +       std::vector<std::shared_ptr<TimeSheetEntry>> entries_;
>>>>>> +};
>>>>>> --
>>>>>> 2.40.1
>>>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Stefan Klug
>>>>
>>
>> --
>> Regards,
>>
>> Stefan Klug
>>

-- 
Regards,

Stefan Klug



More information about the libcamera-devel mailing list