[libcamera-devel] Use and behavior of delayed sensor controls

[Matthias Fend]

Hi Kieran,

Am 23.12.2022 um 12:22 schrieb Kieran Bingham:
> Hi Matthias,
> 
> Quoting Matthias Fend via libcamera-devel (2022-12-23 09:01:25)
>> Hello,
>>
>> I have some questions about delayed sensor controls introduced with
>> 'libcamera: pipeline: rkisp1: Use delayed controls'.
>>
>> Referring to these lines of code:
>>
>> std::unordered_map<uint32_t, DelayedControls::ControlParams> params = {
>>          { V4L2_CID_ANALOGUE_GAIN, { 1, false } },
>>          { V4L2_CID_EXPOSURE, { 2, false } },
>> };
> 
> 
> Yes. These are not correctly set per-sensor in the rkisp1. This should
> be fixed.
> 
> Instead of this single table, the values should be determined on a
> per-sensor basis from the camera sensor class, or camera sensor helpers.
> 
> 
> Different sensors have different delays.

Since sensor helpers are unlikely to be present in various places, I 
assume that there are plans to consolidate them.
Will these sensor helpers then be in a library that can be used by the 
IPA modules (and would allow custom or modified helpers to be used there 
as well) or should this remain internal to libcamera and handled in the 
pipelines?
This seems to be a similar topic as with the optics.

> 
>> If I see things correctly, the RKISP1 pipeline is supposed to write the
>> analog gain to the sensor with a one frame delay and the exposure with a
>> two frame delay.
>> I know there is a hint that these values are somehow guessed for a
>> generic use case and this might need some fine tuning, but still I got
>> some questions.
> 
> Yes, these 'default's might cover 'most' sensors - but in no means all.
> Take a look at the RPi camera helpers, which is really the only place
> this is implemented 'correctly'.
> 
> CamHelperImx477::getDelays for instance shows delays for exposure, gain,
> hblank, and vblank. Each control takes a different number of frames to
> 'apply' ... so the delays are there to ensure that when the IPA tries to
> set the controls for a frame, they are all aligned.
> 
> 
> 
>> - Since gain and exposure somehow belong together and if an algorithm
>> changes both values at the same time, they should be taken into account
>> synchronously in the sensor, I expected these values to be written in
>> the same frame. Is there a reason why there is a frame delay between
>> these controls?
> 
> Because it takes a different number of frames for the sensor to 'apply'
> that action depending on the control.
> 
> 
>>
>> - Why are these controls even delayed by one frame (I would expect zero
>> frames delay for both controls)?
>>
> 
> That's down to the hardware. It takes the sensor 'that amount of time'
> to make the change.
> 
> 
>> - Since the parameter is called delay, I was expecting that a higher
>> number would mean a longer delay, but it looks like the exposure (with a
>> delay of 2) is written to the sensor before the gain (with a delay of
>> 1). I also had the impression that the control with the longest delay is
>> always written as soon as possible and the others are delay by the
>> number of frames resulting from the difference of their delay parameter.
>> Is my observation correct and is this intentional?
>>
> 
> Yes, that sounds about right. The highest delay of the set is the delay
> of applying 'all' controls, and the remaining controls are shifted so
> that all of the controls 'take effect' on the same captured image frame.

Ahhhh, I think I've just spotted my misunderstanding!
The delay specified is not the number of frames libcamera delays the 
control, but the number of frames it takes for the sensor to apply the 
control.
Now the behavior makes a lot more sense to me and most of the questions 
are superfluous.

> 
>> Of course, it's possible that there might be sensors that work exactly
>> as intended with this configuration. But for a simple generic case, at
>> least from my current understanding, that doesn't seem to fit that well.
>>
> 
> What is the 'simple case' you have in mind?

I assumed that 'normally' both exposure and analog gain take effect in 
the next frame. But maybe the view is because I'm biased by the sensor 
I'm currently testing ;)


Thanks for your helpful explanations!

  ~Matthias

> 
> 
>> Maybe someone knows about this and would be so kind and give me some
>> background information or correct my observations and guesses.
> 
> We've tried to spend quite a bit of time looking at this area to work
> out how to handle 'per frame controls', where each request should
> complete with the exact properties asked for in the control list. To do
> that we have to ensure that the controls are set 'early enough', which
> then leads into all sorts of other areas of complication such as how to
> deal with frame drops and not enough requests in the pipeline queue.
> 
> David at Raspberry Pi also has a proposal on this which they're working
> on too.
> 
> --
> Kieran
> 
> 
>> Thanks!
>>    ~Matthias