[PATCH v3 2/8] Documentation: design: ae: Document the design for AE controls
Paul Elder
paul.elder at ideasonboard.com
Fri Nov 15 14:13:02 CET 2024
On Thu, Nov 14, 2024 at 03:36:41PM +0100, Stefan Klug wrote:
> Hi Paul,
>
> Thank you for the patch.
>
> On Wed, Nov 13, 2024 at 10:12:50PM +0900, Paul Elder wrote:
> > Document the design and rationale for the AE-related controls.
> > Also add documentation for the controls.
> >
> > Signed-off-by: Paul Elder <paul.elder at ideasonboard.com>
> > Signed-off-by: Jacopo Mondi <jacopo at jmondi.org>
> >
> > ---
> > Changes in v3:
> > - merge the control documentation into the same document (including the
> > patch)
> > - this is because it was a bit unwieldy to put it in
> > control_ids.cpp.in, now that it's used generically to generate
> > control ids of all namespaces
> > ---
> > Documentation/design/ae.rst | 348 ++++++++++++++++++++++++++++++++++++
> > 1 file changed, 348 insertions(+)
> > create mode 100644 Documentation/design/ae.rst
> >
> > diff --git a/Documentation/design/ae.rst b/Documentation/design/ae.rst
> > new file mode 100644
> > index 000000000000..f121afecea5a
> > --- /dev/null
> > +++ b/Documentation/design/ae.rst
> > @@ -0,0 +1,348 @@
> > +.. SPDX-License-Identifier: CC-BY-SA-4.0
> > +
> > +Design of Exposure and Gain controls
> > +====================================
> > +
> > +This document explains the design and rationale of the controls related to
> > +exposure and gain. This includes the all-encompassing auto-exposure (AE), the
> > +manual exposure control, and the manual gain control.
> > +
> > +Description of the problem
> > +--------------------------
> > +
> > +Sub controls
> > +^^^^^^^^^^^^
> > +
> > +There are more than one control that make up exposure: exposure, gain, and
> > +aperture (though for now we will not consider aperture). We already had
> > +individual controls for setting the values of manual exposure and manual gain,
> > +but for switching between auto mode and manual mode we only had a high-level
> > +boolean AeEnable control that would set *both* exposure and gain to auto mode
> > +or manual mode; we had no way to set one to auto and the other to manual.
> > +
> > +So, we need to introduce two new controls to act as "levers" to indicate
> > +individually for exposure and gain if the value would come from AEGC or if it
> > +would come from the manual control value.
> > +
> > +Aperture priority
> > +^^^^^^^^^^^^^^^^^
> > +
> > +We eventually will support aperture, and so whatever our solution is for having
> > +only some controls on auto and the others on manual needs to be extensible.
> > +
> > +Flickering when going from auto to manual
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +When a manual exposure or gain value is requested by the application, it costs
> > +a few frames worth of time for them to take effect. This means that during a
> > +transition from auto to manual, there would be flickering in the control values
> > +and the transition won't be smooth.
> > +
> > +Take for instance the following flow, where we start on auto exposure (which
> > +for the purposes of the example increments by 1 each frame) and we want to
> > +switch seamlessly to manual exposure, which involves copying the exposure value
> > +computed by the auto exposure algorithm:
> > +
> > +::
> > +
> > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> > + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |
> > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> > +
> > + Mode requested: Auto Auto Auto Manual Manual Manual Manual
> > + Exp requested: N/A N/A N/A 2 2 2 2
> > + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8
> > +
> > + Mode used: Auto Auto Auto Auto Auto Manual Manual
> > + Exp used: 0 1 2 3 4 2 2
> > +
> > +As we can see, after frame N+2 completes, we copy the exposure value that was
> > +used for frame N+2 (which was computed by AE algorithm), and queue that value
> > +into request N+3 with manual mode on. However, as it takes two frames for the
> > +exposure to be set, the exposure still changes since it is set by AE, and we
> > +get a flicker in the exposure during the switch from auto to manual.
> > +
> > +A solution is to *not submit* any exposure value when manual mode is enabled,
> > +and wait until the manual mode as been "applied" before copying the exposure
> > +value:
> > +
> > +::
> > +
> > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> > + | N | | N+1 | | N+2 | | N+3 | | N+4 | | N+5 | | N+6 |
> > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> > +
> > + Mode requested: Auto Auto Auto Manual Manual Manual Manual
> > + Exp requested: N/A N/A N/A None None None 5
> > + Set in Frame: N+2 N+3 N+4 N+5 N+6 N+7 N+8
> > +
> > + Mode used: Auto Auto Auto Auto Auto Manual Manual
> > + Exp used: 0 1 2 3 4 5 5
> > +
> > +In practice, this works. However, libcamera has a policy where once a control
> > +is submitted, its value is saved and does not need to be resubmitted (even
> > +though this isn't implemented yet). So if the manual exposure value was set
> > +while auto mode was on, in theory the value would be saved, so when manual mode
> > +is enabled, the exposure value that was previously set would immediately be
> > +used. Clearly this solution isn't correct, but it can serve as the basis for a
> > +proper solution, with some more rigorous rules.
> > +
> > +Existing solutions
> > +------------------
> > +
> > +Raspberry Pi
> > +^^^^^^^^^^^^
> > +
> > +The raspberry pi IPA got around the lack of individual AeEnable controls for
> > +exposure and gain by using magic values. When AeEnable was false, if one of the
> > +manual control values was set to 0 then the value computed by AEGC would be
> > +used for just that control. This solution isn't desirable, as it prevents
> > +that magic value from being used as a valid value.
> > +
> > +To get around the flickering issue, when AeEnable was false, the raspberry pi
> > +AEGC would simply stop updating the values to be set. As mentioned above, since
> > +the value retention mechanism hasn't actually been implemented yet, this
> > +worked. But, it's not a proper solution.
> > +
> > +Android
> > +^^^^^^^
> > +
> > +The Android HAL specification requires that exposure and gain (sensitivity)
> > +must both be manual or both be auto. It cannot be that one is manual while the
> > +other is auto, so they simply don't support sub controls.
> > +
> > +For the flickering issue, the Android HAL has an AeLock control. To transition
> > +from auto to manual, the application would keep AE on auto, and turn on the
> > +lock. Once the lock has propagated through, then the value can be copied from
> > +the result into the request and the lock disabled and the mode set to manual.
> > +
> > +The problem with this solution is, besides the extra complexity, that it is
> > +ambiguous what happens if there is a state transition from manual to locked
> > +(even though it's a state transition that doesn't make sense). If locked is
> > +defined to "use the last automatically computed values" then it could use the
> > +values from the last time it AE was set to auto, or it would be undefined if AE
> > +was never auto (eg. it started out as manual), or if AE is implemented to run
> > +in the background it could just use the current values that are computed. If
> > +locked is defined to "use the last value that was set" there would be less
> > +ambiguity. Still, it's better if we can make it impossible to execute this
> > +nonsensical state transition, and if we can reduce the complexity of having
> > +this extra control or extra setting on a lever.
> > +
> > +Summary of goals
> > +----------------
> > +
> > + - We need a lock of some sort, to instruct the AEGC to not update output
> > + results
> > +
> > + - We need manual modes, to override the values computed by the AEGC
> > +
> > + - We need to support seamless transitions from auto to manual, and do so
> > + without flickering
> > +
> > + - We need custom minimum values for the manual controls; that is, no magic
> > + values for enabling/disabling auto
> > +
> > + - All of these need to be done with AE sub-controls (exposure time, analogue
> > + gain) and be extensible to aperture in the future
> > +
> > +Our solution
> > +------------
> > +
> > +A diagram of our solution:
> > +
> > +::
> > +
> > + +----------------------------+-------------+------------------+-----------------+
> > + | INPUT | ALGORITHM | RESULT | OUTPUT |
> > + +----------------------------+-------------+------------------+-----------------+
> > +
> > + ExposureTimeMode ExposureTimeMode
> > + ---------------------+----------------------------------------+----------------->
> > + 0: Auto | |
> > + 1: Manual | V
> > + | |\
> > + | | \
> > + | /----------------------------------> | 1| ExposureTime
> > + | | +-------------+ exposure time | | -------------->
> > + \--)--> | | --------------> | 0|
> > + ExposureTime | | | | /
> > + ------------------------+--> | | |/
> > + | | AeState
> > + | AEGC | ----------------------------------->
> > + AnalogueGain | |
> > + ------------------------+--> | | |\
> > + | | | | \
> > + /--)--> | | --------------> | 0| AnalogueGain
> > + | | +-------------+ analogue gain | | -------------->
> > + | \----------------------------------> | 1|
> > + | | /
> > + | |/
> > + | ^
> > + AnalogueGainMode | | AnalogueGainMode
> > + ---------------------+----------------------------------------+----------------->
> > + 0: Auto
> > + 1: Manual
> > +
> > +
> > +The diagram is divided in four sections horizontally:
> > +
> > + - Input: The values received from the request controls
> > +
> > + - Algorithm: The algorithm itself
> > +
> > + - Result: The values calculated by the algorithm
> > +
> > + - Output: The values that sent in result metadata and applied to the device
> > +
> > +The four input controls are divided between manual values (ExposureTime and
> > +AnalogueGain), and operation modes (ExposureTimeMode and AnalogueGainMode). The
> > +former are the manual values, the latter control how they're applied. The two
> > +modes are independent from each other, and each can take one of two values:
> > +
> > + - Auto (0): The AGC computes the value normally. The AGC result is applied
> > + to the output. The manual value is ignored *and is not retained*.
> > +
> > + - Manual (1): The AGC uses the manual value internally. The corresponding
> > + manual control from the request is applied to the output. The AGC result
> > + is ignored.
> > +
> > +The AeState control reports the state of the unified AEGC block. If both
> > +ExposureTimeMode and AnalogueGainMode are set to disabled then it will report
> > +Idle. If at least one of the two is set to auto, then AeState will report
> > +if the AEGC has Converged or not (Searching). This control replaces the old
> > +AeLocked control, as it was insufficient for reporting the AE state.
> > +
> > +There is a caveat to the disabled mode: the manual control value is not
> > +retained if it is set during auto mode. This means that if the disabled mode is
> > +entered without also setting the manual value, then it will enter a state
> > +similar to "locked", where the last automatically computed value while the mode
> > +was auto will be used. Once the manual value is set, then that will be used and
> > +retained as usual.
> > +
> > +This simulates an auto -> locked -> manual or auto -> manual state transition,
> > +and makes it impossible to do the nonsensical manual -> locked state
> > +transition.
> > +
> > +We specifically do not have a "master AE control" like the old AeEnable. This
> > +is because we have the individual mode controls, and if we had a master AE
> > +control it would be a "control that sets other controls", which could easily
> > +get out of control.
> > +
> > +With this solution, the earlier example would become:
> > +
> > +::
> > +
> > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> > + | N+2 | | N+3 | | N+4 | | N+5 | | N+6 | | N+7 | | N+8 | | N+9 | | N+10|
> > + +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
> > + Mode requested: Auto Disab Disab Disab Disab Disab Disab Disab Disab
> > + Exp requested: N/A None None None None 10 None 10 10
> > + Set in Frame: N+4 N+5 N+6 N+7 N+8 N+9 N+10 N+10 N+10
> > +
> > + Mode used: Auto Auto Auto Disab Disab Disab Disab Disab Disab
> > + Exp used: 2 3 4 5 5 5 5 10 10
> > +
> > +This example is extended by a few frames to exhibit the simulated "locked"
> > +state. At frame N+5 the application has confirmed that the auto mode has been
> > +disabled, but does not provide a manual value until request N+7. Thus, the
> > +value that is used in requests N+5 and N+6 (where the mode is disabled), comes
> > +from the last value that was used when the mode was auto, which comes from
> > +frame N+4.
> > +
> > +Then, in N+7, a manual value of 10 is supplied. It takes until frame N+9 for
> > +the exposure to be applied. N+8 does not supply a manual value, but the last
> > +supplied value is retained, so a manual value of 10 is still used and set in
> > +frame N+10.
> > +
> > +Although this behavior is the same as what we had with waiting for the manual
> > +mode to propagate (in the section "Description of the problem"), this time it
> > +is correct as we have defined specifically that if a manual value was specified
> > +while the mode was auto, it will not be retained.
> > +
> > +Description of the controls
> > +===========================
> > +
> > + libcamera offers the following controls related to exposure and gain:
> > +
> > + - AnalogueGain
> > +
> > + - AnalogueGainMode
> > +
> > + - ExposureTime
> > +
> > + - ExposureTimeMode
> > +
> > + - AeState
> > +
> > + Auto-exposure and auto-gain can be enabled and disabled separately using the
> > + ExposureTimeMode and AnalogueGainMode controls respectively. There is no
> > + overarching AeEnable control.
> > +
> > + For each of exposure and gain, we can model it with three states: auto,
> > + locked, and manual. Note that AnalogueGainMode and ExposureTimeMode only
> > + have two values, as the locked state is simulated.
> > +
> > + ::
> > +
> > + /---------------------------------\
> > + | |
> > + V |
> > + +--------+ +--------+
> > + | | ---------------------> | |
> > + | Auto | +--------+ | Manual |
> > + | | ----> | Locked | ----> | |
> > + +--------+ +--------+ +--------+
> > + ^ |
> > + | |
> > + \----------------/
> > +
> > + Notice from the state diagram that locked to manual is a one-way state
> > + change, as the reverse direction is nonsensical (see the design document for
> > + more details on this topic).
> > +
> > + The exposure/gain is in the Auto state when
> > + ExposureTimeMode/AnalogueGainMode is set to Auto. In this state, the value
> > + that is computed by the AE algorithm is applied to the image sensor. Any
> > + value that is supplied in the ExposureTime/AnalogueGain control is ignored
> > + and is not retained.
> > +
> > + If ExposureTimeMode/AnalogueGainMode is set to Manual, it can put us in
> > + either the Locked or Manual state. The difference is in if
> > + ExposureTime/AnalogueGain has been supplied. If it has not yet been
> > + supplied, then we are in the Locked state. If it has been supplied, then we
> > + are in the Manual state.
> > +
> > + In both the Locked state and the Manual state the exposure/gain value does
> > + not come from the AE algorithm. In the Locked state the value comes from the
> > + last value computed by the AE algorithm while the state was Auto, or if the
> > + state was never Auto (e.g. we started in Locked, or the camera doesn't
> > + support Auto), then the value should be a best-effort default value. In the
> > + Manual state the value comes from the value supplied in the
> > + ExposureTime/AnalogueGain control.
> > +
> > + To transition from the Locked state to the Manual state, a value needs to be
> > + submitted in ExposureTime/AnalogueGain. Once the state has transitioned to
> > + Manual, then this value will be retained, and doesn't need to be resubmitted
> > + if it doesn't change.
> > +
> > + To transition to the Auto state, simply set
> > + ExposureTimeMode/AnalogueGainMode to Auto.
> > +
> > +
> > + The AeState metadata reports the state of the AE algorithm. As AE cannot
> > + compute exposure and gain separately, the state of the AE component is
> > + unified. There are three states: Idle, Searching, and Converged.
> > +
> > + The state shall be Idle if both ExposureTimeMode and AnalogueGainMode
> > + are set to Manual. If the camera only supports one of the two controls,
> > + then the state shall be Idle if that one control is set to Manual. If
> > + the camera does not support Manual for at least one of the two controls,
> > + then the state will never be Idle, as AE will always be running.
> > +
> > + The state shall be Searching if at least one of exposure or gain calculated
> > + by the AE algorithm is used (that is, at least one of the two modes is Auto),
> > + *and* the value(s) have not converged yet.
> > +
> > + The state shall be Converged if at least one of exposure or gain calculated
> > + by the AE algorithm is used (that is, at least one of the two modes is Auto),
> > + *and* the value(s) have converged.
>
> That is a lot of stuff to go through. I like the model. One thing that
\o/
> came to my mind: What is the reason to use a enum for the ExposureTimeMode
> control instead of a bool? Are there further extensions planned? GenICam
tbh I can't recall any specific reason. Maybe that it's more explicit
than AutoExposureTime/AutoAnalogueGain == true/false?
> devices often have an additional mode "Once" that runs the control loop
> until converged and then snaps back to off/manual.
Oh neat! Maybe in the future...
>
> But overall I believe we should get that it.
> Reviewed-by: Stefan Klug <stefan.klug at ideasonboard.com>
Thanks,
Paul
More information about the libcamera-devel
mailing list