[PATCH v5 1/8] controls: Introduce AEGC-related controls
Laurent Pinchart
laurent.pinchart at ideasonboard.com
Mon Dec 16 15:20:44 CET 2024
Hi Paul,
Thank you for the patch.
On Mon, Dec 16, 2024 at 01:39:47PM +0900, Paul Elder wrote:
> Introduce the AeState, ExposureTimeMode and AnalogueGainMode controls
> to model the AEGC algorithm block.
>
> The three controls allow applications to select the exposure time and
> analogue gain computation calculation mode (auto vs manual)
> independently from one another, while the AeState control reports the
> global state for the AEGC algorithm.
>
> The new controls are meant to replace the existing AeEnable and AeLocked
> controls, which are momentarily kept not to break compilation of
> platforms making use of them.
>
> Bug: https://bugs.libcamera.org/show_bug.cgi?id=42
> Bug: https://bugs.libcamera.org/show_bug.cgi?id=43
> Bug: https://bugs.libcamera.org/show_bug.cgi?id=47
> Signed-off-by: Paul Elder <paul.elder at ideasonboard.com>
> Signed-off-by: Jacopo Mondi <jacopo at jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart at ideasonboard.com>
> ---
> Changes in v5:
> - improve wordings
>
> Changes in v4:
> - fix wordings
> - add mention on transition from manual to auto
> - add rules regarding the allowed combinations of control info
>
> Changes in v3:
> - recovered from 2-year-old bitrot
> ---
> src/libcamera/control_ids_core.yaml | 262 +++++++++++++++++++++++++--
> src/libcamera/control_ids_draft.yaml | 29 ---
> 2 files changed, 242 insertions(+), 49 deletions(-)
>
> diff --git a/src/libcamera/control_ids_core.yaml b/src/libcamera/control_ids_core.yaml
> index d45cf8e56187..a64d567791c3 100644
> --- a/src/libcamera/control_ids_core.yaml
> +++ b/src/libcamera/control_ids_core.yaml
> @@ -26,6 +26,70 @@ controls:
>
> \sa AeEnable
>
> + - AeState:
> + type: int32_t
> + description: |
> + Report the AEGC algorithm state.
> +
> + The AEGC algorithm computes the exposure time and the analogue gain
> + to be applied to the image sensor.
> +
> + The AEGC algorithm behaviour is controlled by the ExposureTimeMode and
> + AnalogueGainMode controls, which allow applications to decide how
> + the exposure time and gain are computed, in Auto or Manual mode,
> + independently from one another.
> +
> + The AeState control reports the AEGC algorithm state through a single
> + value and describes it as a single computation block which computes
> + both the exposure time and the analogue gain values.
> +
> + When both the exposure time and analogue gain values are configured to
> + be in Manual mode, the AEGC algorithm is quiescent and does not actively
> + compute any value and the AeState control will report AeStateIdle.
> +
> + When at least the exposure time or analogue gain are configured to be
> + computed by the AEGC algorithm, the AeState control will report if the
> + algorithm has converged to stable values for all of the controls set
> + to be computed in Auto mode.
> +
> + \sa AnalogueGainMode
> + \sa ExposureTimeMode
> +
> + enum:
> + - name: AeStateIdle
> + value: 0
> + description: |
> + The AEGC algorithm is inactive.
> +
> + This state is returned when both AnalogueGainMode and
> + ExposureTimeMode are set to Manual and the algorithm is not
> + actively computing any value.
> + - name: AeStateSearching
> + value: 1
> + description: |
> + The AEGC algorithm is actively computing new values, for either the
> + exposure time or the analogue gain, but has not converged to a
> + stable result yet.
> +
> + This state is returned if at least one of AnalogueGainMode or
> + ExposureTimeMode is auto and the algorithm hasn't converged yet.
> +
> + The AEGC algorithm converges once stable values are computed for
> + all of the controls set to be computed in Auto mode. Once the
> + algorithm converges the state is moved to AeStateConverged.
> + - name: AeStateConverged
> + value: 2
> + description: |
> + The AEGC algorithm has converged.
> +
> + This state is returned if at least one of AnalogueGainMode or
> + ExposureTimeMode is Auto, and the AEGC algorithm has converged to a
> + stable value.
> +
> + If the measurements move too far away from the convergence point
> + then the AEGC algorithm might start adjusting again, in which case
> + the state is moved to AeStateSearching.
> +
> # AeMeteringMode needs further attention:
> # - Auto-generate max enum value.
> # - Better handling of custom types.
> @@ -104,6 +168,13 @@ controls:
> The exposure modes specify how the desired total exposure is divided
> between the exposure time and the sensor's analogue gain. They are
> platform specific, and not all exposure modes may be supported.
> +
> + When one of AnalogueGainMode or ExposureTimeMode is set to Manual,
> + the fixed values will override any choices made by AeExposureMode.
> +
> + \sa AnalogueGainMode
> + \sa ExposureTimeMode
> +
> enum:
> - name: ExposureNormal
> value: 0
> @@ -124,13 +195,15 @@ controls:
> Specify an Exposure Value (EV) parameter.
>
> The EV parameter will only be applied if the AE algorithm is currently
> - enabled.
> + enabled, that is, at least one of AnalogueGainMode and ExposureTimeMode
> + are in Auto mode.
>
> By convention EV adjusts the exposure as log2. For example
> EV = [-2, -1, -0.5, 0, 0.5, 1, 2] results in an exposure adjustment
> of [1/4x, 1/2x, 1/sqrt(2)x, 1x, sqrt(2)x, 2x, 4x].
>
> - \sa AeEnable
> + \sa AnalogueGainMode
> + \sa ExposureTimeMode
>
> - ExposureTime:
> type: int32_t
> @@ -139,17 +212,107 @@ controls:
>
> This value is specified in micro-seconds.
>
> - Setting this value means that it is now fixed and the AE algorithm may
> - not change it. Setting it back to zero returns it to the control of the
> - AE algorithm.
> + This control will only take effect if ExposureTimeMode is Manual. If
> + this control is set when ExposureTimeMode is Auto, the value will be
> + ignored and will not be retained.
> +
> + When reported in metadata, this control indicates what exposure time
> + was used for the current frame, regardless of ExposureTimeMode.
> + ExposureTimeMode will indicate the source of the exposure time value,
> + whether it came from the AE algorithm or not.
> +
> + \sa AnalogueGain
> + \sa ExposureTimeMode
> +
> + - ExposureTimeMode:
> + type: int32_t
> + description: |
> + Controls the source of the exposure time that is applied to the image
> + sensor.
> +
> + When set to Auto, the AE algorithm computes the exposure time and
> + configures the image sensor accordingly. When set to Manual, the value
> + of the ExposureTime control is used.
> +
> + When transitioning from Auto to Manual mode and no ExposureTime control
> + is provided by the application, the last value computed by the AE
> + algorithm when the mode was Auto will be used. If the ExposureTimeMode
> + was never set to Auto (either because the camera started in Manual mode,
> + or Auto is not supported by the camera), the camera should use a
> + best-effort default value.
>
> - \sa AnalogueGain AeEnable
> + If ExposureTimeModeManual is supported, the ExposureTime control must
> + also be supported.
>
> - \todo Document the interactions between AeEnable and setting a fixed
> - value for this control. Consider interactions with other AE features,
> - such as aperture and aperture/shutter priority mode, and decide if
> - control of which features should be automatically adjusted shouldn't
> - better be handled through a separate AE mode control.
> + Cameras that support manual control of the sensor shall support manual
> + mode for both ExposureTimeMode and AnalogueGainMode, and shall expose
> + the ExposureTime and AnalogueGain controls. If the camera also has an
> + AEGC implementation, both ExposureTimeMode and AnalogueGainMode shall
> + support both manual and auto mode. If auto mode is available, it shall
> + be the default mode. These rules do not apply to black box cameras
> + such as UVC cameras, where the available gain and exposure modes are
> + completely dependent on what the device exposes.
> +
> + \par Flickerless exposure mode transitions
> +
> + Applications that wish to transition from ExposureTimeModeAuto to direct
> + control of the exposure time without causing extra flicker can do so by
> + selecting an ExposureTime value as close as possible to the last value
> + computed by the auto exposure algorithm in order to avoid any visible
> + flickering.
> +
> + To select the correct value to use as ExposureTime value, applications
> + should accommodate the natural delay in applying controls caused by the
> + capture pipeline frame depth.
> +
> + When switching to manual exposure mode, applications should not
> + immediately specify an ExposureTime value in the same request where
> + ExposureTimeMode is set to Manual. They should instead wait for the
> + first Request where ExposureTimeMode is reported as
> + ExposureTimeModeManual in the Request metadata, and use the reported
> + ExposureTime to populate the control value in the next Request to be
> + queued to the Camera.
> +
> + The implementation of the auto-exposure algorithm should equally try to
> + minimize flickering and when transitioning from manual exposure mode to
> + auto exposure use the last value provided by the application as starting
> + point.
> +
> + 1. Start with ExposureTimeMode set to Auto
> +
> + 2. Set ExposureTimeMode to Manual
> +
> + 3. Wait for the first completed request that has ExposureTimeMode
> + set to Manual
> +
> + 4. Copy the value reported in ExposureTime into a new request, and
> + submit it
> +
> + 5. Proceed to run manual exposure time as desired
> +
> + \sa ExposureTime
> + enum:
> + - name: ExposureTimeModeAuto
> + value: 0
> + description: |
> + The exposure time will be calculated automatically and set by the
> + AE algorithm.
> +
> + If ExposureTime is set while this mode is active, it will be
> + ignored, and its value will not be retained.
> +
> + When transitioning from Manual to Auto mode, the AEGC should start
> + its adjustments based on the last set manual ExposureTime value.
> + - name: ExposureTimeModeManual
> + value: 1
> + description: |
> + The exposure time will not be updated by the AE algorithm.
> +
> + When transitioning from Auto to Manual mode, the last computed
> + exposure value is used until a new value is specified through the
> + ExposureTime control. If an ExposureTime value is specified in the
> + same request where the ExposureTimeMode is changed from Auto to
> + Manual, the provided ExposureTime is applied immediately.
>
> - AnalogueGain:
> type: float
> @@ -159,17 +322,76 @@ controls:
> The value of the control specifies the gain multiplier applied to all
> colour channels. This value cannot be lower than 1.0.
>
> - Setting this value means that it is now fixed and the AE algorithm may
> - not change it. Setting it back to zero returns it to the control of the
> - AE algorithm.
> + This control will only take effect if AnalogueGainMode is Manual. If
> + this control is set when AnalogueGainMode is Auto, the value will be
> + ignored and will not be retained.
> +
> + When reported in metadata, this control indicates what analogue gain
> + was used for the current request, regardless of AnalogueGainMode.
> + AnalogueGainMode will indicate the source of the analogue gain value,
> + whether it came from the AEGC algorithm or not.
> +
> + \sa ExposureTime
> + \sa AnalogueGainMode
> +
> + - AnalogueGainMode:
> + type: int32_t
> + description: |
> + Controls the source of the analogue gain that is applied to the image
> + sensor.
> +
> + When set to Auto, the AEGC algorithm computes the analogue gain and
> + configures the image sensor accordingly. When set to Manual, the value
> + of the AnalogueGain control is used.
> +
> + When transitioning from Auto to Manual mode and no AnalogueGain control
> + is provided by the application, the last value computed by the AEGC
> + algorithm when the mode was Auto will be used. If the AnalogueGainMode
> + was never set to Auto (either because the camera started in Manual mode,
> + or Auto is not supported by the camera), the camera should use a
> + best-effort default value.
> +
> + If AnalogueGainModeManual is supported, the AnalogueGain control must
> + also be supported.
> +
> + For cameras where we have control over the ISP, both ExposureTimeMode
> + and AnalogueGainMode are expected to support manual mode, and both
> + controls (as well as ExposureTimeMode and AnalogueGain) are expected to
> + be present. If the camera also has an AEGC implementation, both
> + ExposureTimeMode and AnalogueGainMode shall support both manual and
> + auto mode. If auto mode is available, it shall be the default mode.
> + These rules do not apply to black box cameras such as UVC cameras,
> + where the available gain and exposure modes are completely dependent on
> + what the hardware exposes.
> +
> + The same procedure described for performing flickerless transitions in
> + the ExposureTimeMode control documentation can be applied to analogue
> + gain.
> +
> + \sa ExposureTimeMode
> + \sa AnalogueGain
> + enum:
> + - name: AnalogueGainModeAuto
> + value: 0
> + description: |
> + The analogue gain will be calculated automatically and set by the
> + AEGC algorithm.
> +
> + If AnalogueGain is set while this mode is active, it will be
> + ignored, and it will also not be retained.
>
> - \sa ExposureTime AeEnable
> + When transitioning from Manual to Auto mode, the AEGC should start
> + its adjustments based on the last set manual AnalogueGain value.
> + - name: AnalogueGainModeManual
> + value: 1
> + description: |
> + The analogue gain will not be updated by the AEGC algorithm.
>
> - \todo Document the interactions between AeEnable and setting a fixed
> - value for this control. Consider interactions with other AE features,
> - such as aperture and aperture/shutter priority mode, and decide if
> - control of which features should be automatically adjusted shouldn't
> - better be handled through a separate AE mode control.
> + When transitioning from Auto to Manual mode, the last computed
> + gain value is used until a new value is specified through the
> + AnalogueGain control. If an AnalogueGain value is specified in the
> + same request where the AnalogueGainMode is changed from Auto to
> + Manual, the provided AnalogueGain is applied immediately.
>
> - AeFlickerMode:
> type: int32_t
> diff --git a/src/libcamera/control_ids_draft.yaml b/src/libcamera/control_ids_draft.yaml
> index 1b284257f601..32c2b3feaf65 100644
> --- a/src/libcamera/control_ids_draft.yaml
> +++ b/src/libcamera/control_ids_draft.yaml
> @@ -77,35 +77,6 @@ controls:
> High quality aberration correction which might reduce the frame
> rate.
>
> - - AeState:
> - type: int32_t
> - description: |
> - Control to report the current AE algorithm state. Currently identical to
> - ANDROID_CONTROL_AE_STATE.
> -
> - Current state of the AE algorithm.
> - enum:
> - - name: AeStateInactive
> - value: 0
> - description: The AE algorithm is inactive.
> - - name: AeStateSearching
> - value: 1
> - description: The AE algorithm has not converged yet.
> - - name: AeStateConverged
> - value: 2
> - description: The AE algorithm has converged.
> - - name: AeStateLocked
> - value: 3
> - description: The AE algorithm is locked.
> - - name: AeStateFlashRequired
> - value: 4
> - description: The AE algorithm would need a flash for good results
> - - name: AeStatePrecapture
> - value: 5
> - description: |
> - The AE algorithm has started a pre-capture metering session.
> - \sa AePrecaptureTrigger
> -
> - AwbState:
> type: int32_t
> description: |
--
Regards,
Laurent Pinchart
More information about the libcamera-devel
mailing list