[libcamera-devel] [PATCH 0/4] ipa: ipu3: af: Temporarily suspend AWB and AE during AF scanning

[Jean-Michel Hautbois]

Hi Kate !

On 22/03/2022 03:41, Kate Hsuan via libcamera-devel wrote:
> The floating results of AWB and AE significantly impact the AF
> performance and the AF scene variance estimation. If the exposure value
> and color settings are changed by the algorithm, the AF scene variance will
> be affected and the algorithm may find an incorrect local maximum value.
> Consequently, a wrong lens position is determined and we could only get
> a blurred image.

I agree with you, at least on one assumption: AWB can have an impact on 
AF. AE is more debatable I think :-).
I think we should consider something a bit different from what you did, 
if you don't mind, as I don't think AWB should know anything out of its 
own scope, same for AGC, same for AF, same for... you get it :-).
This should be taken into account at the IPA level, the controller, 
which could set a state for an algorithm.

I think that, while autofocus is starting, or in coarse search mode, we 
should not change the awb gains. When is gets into the fine search mode, 
then AWB could be started as it should not interfere anymore. So, I 
think AF could have a 'status' field in its frameContext, and the 
algorithms (AWB, AGC, probably AF) should be "pausable". David sent a 
patch lately for RPi [1] and this is something we should consider.

That beeing said, and I hope I am clear in my explanations, before 
calling the process() call for the algorithms, the IPU3IPU could set the 
status of AWB and AGC based on the AF status. I would start AF after AGC 
is stable enough (could be based on a number of frames for a first try) 
and I would enable AWB while AGC is starting.

Once AGC is stable, AF is entering the coarse search, and in this phase 
AWB is paused. Once it enters the fine search, we can resume AWB.

I think it should be enough, as we are not zooming, so illumination is 
not changing a lot with focus change AFAIK.

I don't really have time to test it so if you want/can I would greatly 
appreciate :-).

Thanks !
JM

[1]: https://patchwork.libcamera.org/patch/15343/

> 
> This patch proposed an AWB and AE suspension mechanism while AF is
> searching for the focus. A lock flag is used to identify that AWB and
> AE should be suspended or not. AGC and AE algorithm test the flag and
> determine that they have to perform the algorithm or suspend itself.
> 
> Kate Hsuan (4):
>    ipa: ipu3: af: Introduce AWB and AE lock when AF scanning
>    ipa: ipu3: af: AE and AWE lock request flow when performing AF
>    ipa: ipu3: awb: AWB lock for AF scan
>    ipa: ipu3: agc: AE lock for AF scanning
> 
>   src/ipa/ipu3/algorithms/af.cpp  | 25 +++++++++++++++++++++++++
>   src/ipa/ipu3/algorithms/af.h    |  2 ++
>   src/ipa/ipu3/algorithms/agc.cpp | 14 ++++++++++++++
>   src/ipa/ipu3/algorithms/agc.h   |  1 +
>   src/ipa/ipu3/algorithms/awb.cpp | 13 +++++++++++++
>   src/ipa/ipu3/algorithms/awb.h   |  1 +
>   src/ipa/ipu3/ipa_context.h      |  1 +
>   7 files changed, 57 insertions(+)
>