[libcamera-devel] [PATCH v4 1/3] controls: Reorganize the AE-related controls
Jacopo Mondi
jacopo at jmondi.org
Wed May 18 21:26:28 CEST 2022
Hi Paul,
On Wed, May 18, 2022 at 03:47:26PM +0200, Paul Elder via libcamera-devel wrote:
> We have multiple goals:
> - we need a lock of some sort, to instruct the AEGC to not update output
> results
> - we need manual modes, to override the values computed by the AEGC
> - we need to support seamless transitions from auto -> manual, and do so
> without flickering
> - we need custom minimum values for the manual controls, that is no
> magic values for enabling/disabling auto
> - all of these need to be done with AE sub-controls (exposure time,
> analogue gain)
>
> To achieve these goals, we introduce mode controls for the AE
> sub-controls: ExposureTimeMode and AnalogueGainMode. These have an auto
> state, and a disabled state. The disabled state has an internal one-way
> state change from locked to manual, triggered by the presence of the
> value-controls (ExposureTime and AnalogueGain).
>
> We then remove the AeEnable control, as it is a redundant control in the
> face of these two mode controls.
>
> We also remove AeLocked, as it is insufficient for reporting the AE
> state, and we promote AeState to non-draft to fill its role. Notably,
> the locked state is removed, since this information can be obtained from
> the aforementioned mode controls.
>
> Bug: https://bugs.libcamera.org/show_bug.cgi?id=42
> Bug: https://bugs.libcamera.org/show_bug.cgi?id=43
> Bug: https://bugs.libcamera.org/show_bug.cgi?id=47
> Signed-off-by: Paul Elder <paul.elder at ideasonboard.com>
>
Be prepared: a lot of bikeshedding ahead
I think the content is good and these are the last refinements. Take
them as suggestions.
> ---
> Changes in v4:
> - remove FlashRequired and Precapture from AeState
> - upgrade documentation of all the controls
>
> Changes in v3:
> - improve wording of the control descriptions
> - make more succinct and clear
> - add description of how to do a flickerless transition
>
> Changes in v2:
> - No changes, just resubmitting at the head of this series so that it's
> together and so that /people will actually see it/
>
> Initial version:
> Still RFC as I haven't updated the users of the control yet, and I want
> to check that these are the controls and docs that we want.
>
> We've decided that the "master AE control" will be implemented by a
> helper... but looking at uvcvideo and the V4L2 controls I'm wondering if
> such helper should come earlier than later?
> ---
> src/libcamera/control_ids.yaml | 262 +++++++++++++++++++++++++--------
> 1 file changed, 200 insertions(+), 62 deletions(-)
>
> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> index 9d4638ae..9f5ce5e8 100644
> --- a/src/libcamera/control_ids.yaml
> +++ b/src/libcamera/control_ids.yaml
> @@ -7,23 +7,46 @@
> # Unless otherwise stated, all controls are bi-directional, i.e. they can be
> # set through Request::controls() and returned out through Request::metadata().
> controls:
> - - AeEnable:
> - type: bool
> + - AeState:
> + type: int32_t
> description: |
> - Enable or disable the AE.
> + Control to report the AE algorithm state associated with the capture
> + result.
I know you added the last part to highlight that this control is only
reported in metadata. To be honest I would defer this to a later
series where the 'direction' of controls will be more clearly defined
and just define this
"Control to report the AE algorithm state'
>
> - \sa ExposureTime AnalogueGain
> + The state is still reported even if ExposureTimeMode or
> + AnalogueGainMode is set to Disabled.
>
> - - AeLocked:
> - type: bool
> - description: |
> - Report the lock status of a running AE algorithm.
> + \sa AnalogueGain
> + \sa AnalogueGainMode
> + \sa ExposureTime
> + \sa ExposureTimeMode
>
> - If the AE algorithm is locked the value shall be set to true, if it's
> - converging it shall be set to false. If the AE algorithm is not
> - running the control shall not be present in the metadata control list.
> + enum:
The AF controls use the term "Idle" instead of inactive. I think it
would be nice if we use the same term to express the same concept.
> + - name: AeStateInactive
> + value: 0
> + description: |
> + The AE algorithm is inactive.
>
> - \sa AeEnable
> + This state should be returned if both AnalogueGainMode and
> + ExposureTimeMode are set to disabled (or one, if the camera only
> + supports one of the two controls).
Do you think it would be better to write this slightly differently, to
tell what it means to application, instead of suggesting to IPA
developers what they have to report (this part can be kept as well,
maybe implicitly suggested).
To explain what I mean, I would make this along the lines of:
This state is returned when the AE algorithm is not
actively computing new values for the exposure time and
analogue gain. This happens when both AnalogueGainMode
and ExposureTimeMode are set to disabled or when the
algorithm has converged and has not yet initiated a new
scan.
If either ExposureTimeMode or AnalogueGainMode are set
to auto, the AE algorithm might spontaneously initiate
a new scan in which case the state is moved to be
AeStateSearching.
> + - name: AeStateSearching
> + value: 1
> + description: |
> + The AE algorithm has not converged yet.
The AE algorithm is actively computing new values and
has not converged yet.
> + This state should be returned if at least one of AnalogueGainMode
> + or ExposureTimeMode is set to auto, and the AE algorithm hasn't
> + converged yet. If the AE algorithm converges, the state shall go to
> + AeStateConverged.
This state is only returned when at least one of
AnalogueGainMode or ExposureTimeMode is set to auto and
the AE algorithm has spontaneously started a scan and
has not convergd yet. If the AE algorithm converges,
the state is moved to AeStateConverged.
For the same reason of telling applications how they should interepet
it instead of telling IPA developers how they should behave.
Do we need a AeStateFailed state to report failed scans or it
shouldn't happen at all ?
> + - name: AeStateConverged
> + value: 2
> + description: |
> + The AE algorithm has converged.
> +
> + This state should be returned if at least one of AnalogueGainMode
> + or ExposureTimeMode is set to auto, and the AE algorithm has
> + converged.
Same "This state is returned if .." instead of "should be"
I hope my reasoning is right here and I'm not trying to correct a
native speaker with my bad english.
>
> # AeMeteringMode needs further attention:
> # - Auto-generate max enum value.
> @@ -93,6 +116,13 @@ controls:
> how the desired total exposure is divided between the shutter time
> and the sensor's analogue gain. The exposure modes are platform
> specific, and not all exposure modes may be supported.
> +
> + When one of AnalogueGainMode or ExposureTimeMode is set to Disabled,
> + the fixed values will override any choices made by AeExposureMode.
> +
> + \sa AnalogueGainMode
> + \sa ExposureTimeMode
> +
> enum:
> - name: ExposureNormal
> value: 0
> @@ -111,13 +141,15 @@ controls:
> type: float
> description: |
> Specify an Exposure Value (EV) parameter. The EV parameter will only be
> - applied if the AE algorithm is currently enabled.
> + applied if the AE algorithm is currently enabled, that is, at least one
> + of AnalogueGainMode and ExposureTimeMode are auto.
>
> By convention EV adjusts the exposure as log2. For example
> EV = [-2, -1, 0.5, 0, 0.5, 1, 2] results in an exposure adjustment
> of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].
>
> - \sa AeEnable
> + \sa AnalogueGainMode
> + \sa ExposureTimeMode
>
> - ExposureTime:
> type: int32_t
> @@ -125,17 +157,85 @@ controls:
> Exposure time (shutter speed) for the frame applied in the sensor
> device. This value is specified in micro-seconds.
>
> - Setting this value means that it is now fixed and the AE algorithm may
> - not change it. Setting it back to zero returns it to the control of the
> - AE algorithm.
> + This control will only take effect if ExposureTimeMode is Disabled. If
> + this control is set when ExposureTimeMode is Auto, the value will be
> + ignored and will not be retained.
> +
> + When reported in metadata, this control indicates what exposure time
> + was used for the current request, regardless of ExposureTimeMode.
s/for the current request//
metadata are always associated to requests :)
> + ExposureTimeMode will indicate the source of the exposure time value,
> + whether it came from the AE algorithm or not.
> +
> + \sa AnalogueGain
> + \sa ExposureTimeMode
> +
> + - ExposureTimeMode:
> + type: int32_t
> + description: |
> + Controls the source of the exposure time that is applied to the image
> + sensor. When set to Auto, the AE algorithm computes the exposure time
> + and configures the image sensor accordingly. When set to Disabled,
> + exposure time specified in ExposureTime is applied to the image sensor.
> + If ExposureTime is not set, then the value last computed by the AE
I think you mean "When transitioning from Auto to Manual mode and no
ExposureTime is provided, then..."
> + algorithm when the mode was Auto will be used.
> +
> + If ExposureTime is not set and the mode is ExposureTimeModeDisabled and
> + AE was never Auto (either because the camera started in Disabled mode,
> + or Auto is not supported by the camera), the camera should use a
> + best-effort default value.
> +
> + When ExposureTimeMode is set Auto, the value set in ExposureTime is
> + ignored and is not retained. This means that if ExposureTimeMode is set
"is later set to Disabled"
to make it clear it is a state transition
> + to Disabled and ExposureTime is not also set, the exposure time that
> + was last computed by the AE algorithm while the mode was Auto will be
> + applied to the sensor.
Ah, isn't this also described above ?
> +
> + If ExposureTimeModeDisabled is supported, the ExposureTime control must
> + also be supported.
This is a note for implementer, it doesn't hurt though, but I would
rather move it to the design document you have prepared.
> +
> + The set of ExposureTimeMode modes that are supported by the camera must
> + have an intersection with the supported set of AnalogueGainMode modes.
>
> - \sa AnalogueGain AeEnable
> + As it takes a few frames to apply the exposure time, there is a period of
> + time between submitting a request with ExposureTimeMode set to Disabled
> + and the exposure time component of the AE actually being disabled,
> + during which the AE algorithm can still update the exposure time. If an
> + application is switching from automatic and manual control and wishes
> + to eliminate any flicker during the switch, the following procedure is
> + recommended.
>
> - \todo Document the interactions between AeEnable and setting a fixed
> - value for this control. Consider interactions with other AE features,
> - such as aperture and aperture/shutter priority mode, and decide if
> - control of which features should be automatically adjusted shouldn't
> - better be handled through a separate AE mode control.
> + 1. Start with ExposureTimeMode set to Auto
> +
> + 2. Set ExposureTimeMode to Disabled
> +
> + 3. Wait for the first request to be output that has ExposureTimeMode
> + set to Disabled
> +
> + 4. Copy the value reported in ExposureTime into a new request, and
> + submit it
> +
> + 5. Proceed to run manual exposure time
I dare to repropose the text I suggested on v2:
Applications that transition from auto ExposureTimeMode
to the direct control of the exposure time should
aim to do so by selecting an ExposureTime value as close
as possible to the last value computed by the auto
exposure algorithm in order to avoid any visible
flickering.
To select the correct value to use as ExposureTime value,
applications should accommodate the natural delay in
applying controls caused by the capture pipeline frame
depth.
When switching to manual exposure mode, applications
should not immediately specify an ExposureTime value in
the same request where ExposureTimeMode is set to
Disabled. They should instead wait for the first Request
where ExposureTimeMode is reported as Disabled in the
Request metadata, and use the there reported exposure to
populate the ExposureTime control value in the next
Request to be queued to the Camera.
The implementation of the auto-exposure algorithm
should equally try to minimize flickering and when
transitioning from manual exposure mode to auto
exposure use the last value provided by the application
as starting point.
This is not alternative to the list of steps you have proposed but
provides a more detailed introduction to them. Do you think it's not
necessary ?
> +
> + \sa ExposureTime
> + enum:
> + - name: ExposureTimeModeAuto
> + value: 0
> + description: |
> + The exposure time will be calculated automatically and set by the
> + AE algorithm. If ExposureTime is set while this mode is active, it
> + will be ignored, and it will also not be retained.
> + - name: ExposureTimeModeDisabled
> + value: 1
> + description: |
> + The exposure time will not be updated by the AE algorithm. It will
> + come from the last calculated value when the mode was Auto, or from
> + the value specified in ExposureTime.
> +
> + When transitioning from Auto to Disabled mode, the last computed
> + exposure value is used until a new value is specified through the
> + ExposureTime control. If an ExposureTime value is specified in the
> + same request where the ExposureTimeMode is changed from Auto to
> + Disabled, the provided ExposureTime is applied.
"is applied immediately" ?
Thanks and sorry for being picky, we're almost there :)
>
> - AnalogueGain:
> type: float
> @@ -144,17 +244,85 @@ controls:
> The value of the control specifies the gain multiplier applied to all
> colour channels. This value cannot be lower than 1.0.
>
> - Setting this value means that it is now fixed and the AE algorithm may
> - not change it. Setting it back to zero returns it to the control of the
> - AE algorithm.
> + This control will only take effect if AnalogueGainMode is Disabled. If
> + this control is set when AnalogueGainMode is Auto, the value will be
> + ignored and will not be retained.
> +
> + When reported in metadata, this control indicates what analogue gain
> + was used for the current request, regardless of AnalogueGainMode.
> + AnalogueGainMode will indicate the source of the analogue gain value,
> + whether it came from the AE algorithm or not.
> +
> + \sa ExposureTime
> + \sa AnalogueGainMode
> +
> + - AnalogueGainMode:
> + type: int32_t
> + description: |
> + Controls the source of the analogue gain that is applied to the image
> + sensor. When set to Auto, the AE algorithm computes the analogue gain
> + and configures the image sensor accordingly. When set to Disabled,
> + analogue gain specified in AnalogueGain is applied to the image sensor.
> + If AnalogueGain is not set, then the value last computed by the AE
> + algorithm when the mode was Auto will be used.
> +
> + If AnalogueGain is not set and the mode is AnalogueGainModeDisabled and
> + AE was never Auto (either because the camera started in Disabled mode,
> + or Auto is not supported by the camera), the camera should use a
> + best-effort default value.
> +
> + When AnalogueGainMode is set Auto, the value set in AnalogueGain is
> + ignored and is not retained. This means that if AnalogueGainMode is set
> + to Disabled and AnalogueGain is not also set, the analogue gain that
> + was last computed by the AE algorithm while the mode was Auto will be
> + applied to the sensor.
>
> - \sa ExposureTime AeEnable
> + If AnalogueGainModeDisabled is supported, the AnalogueGain control must
> + also be supported.
> +
> + The set of AnalogueGainMode modes that are supported by the camera must
> + have an intersection with the supported set of ExposureTimeMode modes.
> +
> + As it takes a few frames to apply the analogue gain, there is a period of
> + time between submitting a request with AnalogueGainMode set to Disabled
> + and the analogue gain component of the AE actually being disabled,
> + during which the AE algorithm can still update the analogue gain. If an
> + application is switching from automatic and manual control and wishes
> + to eliminate any flicker during the switch, the following procedure is
> + recommended.
> +
> + 1. Start with AnalogueGainMode set to Auto
> +
> + 2. Set AnalogueGainMode to Disabled
> +
> + 3. Wait for the first request to be output that has AnalogueGainMode
> + set to Disabled
> +
> + 4. Copy the value reported in AnalogueGain into a new request, and
> + submit it
> +
> + 5. Proceed to run manual analogue gain
> +
> + \sa AnalogueGain
> + enum:
> + - name: AnalogueGainModeAuto
> + value: 0
> + description: |
> + The analogue gain will be calculated automatically and set by the
> + AE algorithm. If AnalogueGain is set while this mode is active, it
> + will be ignored, and it will also not be retained.
> + - name: AnalogueGainModeDisabled
> + value: 1
> + description: |
> + The analogue gain will not be updated by the AE algorithm. It will
> + come from the last calculated value when the mode was Auto, or from
> + the value specified in AnalogueGain.
>
> - \todo Document the interactions between AeEnable and setting a fixed
> - value for this control. Consider interactions with other AE features,
> - such as aperture and aperture/shutter priority mode, and decide if
> - control of which features should be automatically adjusted shouldn't
> - better be handled through a separate AE mode control.
> + When transitioning from Auto to Disabled mode the last computed
> + gain value is used until a new value is specified through the
> + AnalogueGain control. If an AnalogueGain value is specified in the
> + same request where the AnalogueGainMode is set to Disabled, the
> + provided AnalogueGain is applied.
>
> - Brightness:
> type: float
> @@ -477,36 +645,6 @@ controls:
> High quality aberration correction which might reduce the frame
> rate.
>
> - - AeState:
> - type: int32_t
> - draft: true
> - description: |
> - Control to report the current AE algorithm state. Currently identical to
> - ANDROID_CONTROL_AE_STATE.
> -
> - Current state of the AE algorithm.
> - enum:
> - - name: AeStateInactive
> - value: 0
> - description: The AE algorithm is inactive.
> - - name: AeStateSearching
> - value: 1
> - description: The AE algorithm has not converged yet.
> - - name: AeStateConverged
> - value: 2
> - description: The AE algorithm has converged.
> - - name: AeStateLocked
> - value: 3
> - description: The AE algorithm is locked.
> - - name: AeStateFlashRequired
> - value: 4
> - description: The AE algorithm would need a flash for good results
> - - name: AeStatePrecapture
> - value: 5
> - description: |
> - The AE algorithm has started a pre-capture metering session.
> - \sa AePrecaptureTrigger
> -
> - AfState:
> type: int32_t
> draft: true
> --
> 2.30.2
>
More information about the libcamera-devel
mailing list