[libcamera-devel] Use and behavior of delayed sensor controls

[Kieran Bingham]

Hi Matthias,

Quoting Matthias Fend via libcamera-devel (2022-12-23 09:01:25)
> Hello,
> 
> I have some questions about delayed sensor controls introduced with 
> 'libcamera: pipeline: rkisp1: Use delayed controls'.
> 
> Referring to these lines of code:
> 
> std::unordered_map<uint32_t, DelayedControls::ControlParams> params = {
>         { V4L2_CID_ANALOGUE_GAIN, { 1, false } },
>         { V4L2_CID_EXPOSURE, { 2, false } },
> };


Yes. These are not correctly set per-sensor in the rkisp1. This should
be fixed.

Instead of this single table, the values should be determined on a
per-sensor basis from the camera sensor class, or camera sensor helpers.


Different sensors have different delays.

> If I see things correctly, the RKISP1 pipeline is supposed to write the 
> analog gain to the sensor with a one frame delay and the exposure with a 
> two frame delay.
> I know there is a hint that these values are somehow guessed for a 
> generic use case and this might need some fine tuning, but still I got 
> some questions.

Yes, these 'default's might cover 'most' sensors - but in no means all.
Take a look at the RPi camera helpers, which is really the only place
this is implemented 'correctly'.

CamHelperImx477::getDelays for instance shows delays for exposure, gain,
hblank, and vblank. Each control takes a different number of frames to
'apply' ... so the delays are there to ensure that when the IPA tries to
set the controls for a frame, they are all aligned.



> - Since gain and exposure somehow belong together and if an algorithm 
> changes both values at the same time, they should be taken into account 
> synchronously in the sensor, I expected these values to be written in 
> the same frame. Is there a reason why there is a frame delay between 
> these controls?

Because it takes a different number of frames for the sensor to 'apply'
that action depending on the control.


> 
> - Why are these controls even delayed by one frame (I would expect zero 
> frames delay for both controls)?
> 

That's down to the hardware. It takes the sensor 'that amount of time'
to make the change.


> - Since the parameter is called delay, I was expecting that a higher 
> number would mean a longer delay, but it looks like the exposure (with a 
> delay of 2) is written to the sensor before the gain (with a delay of 
> 1). I also had the impression that the control with the longest delay is 
> always written as soon as possible and the others are delay by the 
> number of frames resulting from the difference of their delay parameter.
> Is my observation correct and is this intentional?
> 

Yes, that sounds about right. The highest delay of the set is the delay
of applying 'all' controls, and the remaining controls are shifted so
that all of the controls 'take effect' on the same captured image frame.

> Of course, it's possible that there might be sensors that work exactly 
> as intended with this configuration. But for a simple generic case, at 
> least from my current understanding, that doesn't seem to fit that well.
> 

What is the 'simple case' you have in mind? 


> Maybe someone knows about this and would be so kind and give me some 
> background information or correct my observations and guesses.

We've tried to spend quite a bit of time looking at this area to work
out how to handle 'per frame controls', where each request should
complete with the exact properties asked for in the control list. To do
that we have to ensure that the controls are set 'early enough', which
then leads into all sorts of other areas of complication such as how to
deal with frame drops and not enough requests in the pipeline queue.

David at Raspberry Pi also has a proposal on this which they're working
on too.

--
Kieran


> Thanks!
>   ~Matthias