[libcamera-devel] [PATCH v3 1/1] libcamera: controls: Controls for driving AF (autofocus) algorithms
David Plowman
david.plowman at raspberrypi.com
Thu Apr 21 14:59:53 CEST 2022
Hi Jacopo
Thanks for the quick reply!
On Thu, 21 Apr 2022 at 10:25, Jacopo Mondi <jacopo at jmondi.org> wrote:
>
> Ah, forgot one thing!
>
> On Thu, Apr 21, 2022 at 11:24:16AM +0200, Jacopo Mondi wrote:
> > Hi David,
> > thanks for this new version
> >
> > On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
> > > This patch describes a series of controls that allow applications to
> > > drive AF algorithms:
> > >
> > > AfMode - manual, auto or continuous
> > > AfRange - full, macro or normal
> > > AfSpeed - fast or slowwith
> > > AfWindows - AF window locations
> > > AfTrigger - start (trigger) an AF scan or cancel
> > > AfPause - pause continuous AF
> > > LensPosition - set or retrieve position of lens
> > > AfState - reports whether scanning/success/failure
> > > AfPauseState - reports whether continuous AF paused or not
> > >
> > > Signed-off-by: David Plowman <david.plowman at raspberrypi.com>
> > > ---
> > > src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
> > > 1 file changed, 258 insertions(+), 60 deletions(-)
> > >
> > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > > index 9d4638ae..eccf18bc 100644
> > > --- a/src/libcamera/control_ids.yaml
> > > +++ b/src/libcamera/control_ids.yaml
> > > @@ -381,6 +381,264 @@ controls:
> > > \todo Define how the sensor timestamp has to be used in the reprocessing
> > > use case.
> > >
> > > + - AfMode:
> > > + type: int32_t
> > > + description: |
> > > + Control to set the mode of the AF (autofocus) algorithm. Applications
> > > + are allowed to set a new mode, and to send additional controls for
> > > + that new mode, in the same request.
> > > +
> > > + An implementation may choose not to implement all the modes.
> > > +
> > > + enum:
> > > + - name: AfModeManual
> > > + value: 0
> > > + description: |
> > > + The AF algorithm is in manual mode. In this mode it will never
> > > + perform any action nor move the lens of its own accord, but an
> > > + application can set controls to move the lens.
> > > +
> >
> > If, as I understand it, the lens can only be moved with 'LensPosition'
> > when in manual mode, I would say so explicitly
> >
> > "... but an application can specify the desired lens
> > position using the LensPosition control."
Yes, will do.
> >
> > > + In this mode the AfState will always report AfStateReset.
> >
> > I'm tempted to say AfState is useless and can not be reported at all
> > in this case. For simplicity (for ph/ipa) I'm fine reporting it
> > regardless, even if seeing 'reset' might be slightly confusing. I
> > don't have better alternatives to propose though.
Don't really mind either. I think reporting stuff all the time might
be slightly easier, at least it's always obvious what to expect...
> >
> > Reading the AfStateReset description
> >
> > - name: AfStateReset
> > value: 0
> > description: |
> > The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > scan was cancelled.
> >
> > I wonder if this isn't better named AfStateIdle
Yes, also fine with that.
> >
> >
> > > + - name: AfModeAuto
> > > + value: 1
> > > + description: |
> > > + The AF algorithm is in auto mode. This means that the algorithm
> > > + will never move the lens or change state unless the AfTrigger
> > > + control is used. The AfTrigger control can be used to initiate a
> > > + focus scan, the results of which will be reported by AfState.
> > > +
> > > + If the autofocus algorithm is moved from AfModeAuto to another
> > > + mode while a scan is in progress, the scan is cancelled
> > > + immediately, without waiting for the scan to finish.
> > > +
> > > + When first entering this mode the AfState will report
> > > + AfStateReset. When a trigger control is sent, AfState will
> > > + report AfStateScanning for a period before spontaneously
> > > + changing to AfStateFocused or AfStateFailed, depending on
> > > + the outcome of the scan. It will remain in this state until
> > > + another scan is initiated by the AfTrigger control. If a scan is
> > > + cancelled (without changing to another mode), AfState will return
> > > + to AfStateReset.
> > > + - name: AfModeContinuous
> > > + value: 2
> > > + description: |
> > > + The AF algorithm is in continuous mode. This means that the lens
> > > + can re-start a scan spontaneously at any moment, without any user
> > > + intervention. The AfState still reports whether the algorithm is
> > > + currently scanning or not, though the application has no ability
> > > + to initiate or cancel scans (though it can "pause" them), nor to
> > > + move the lens for itself.
> > > +
> > > + When set to AfModeContinuous, the system will immediately initiate
> > > + a scan so AfState will report AfStateScanning, and will settle on
> > > + one of AfStateFocused or AfStateFailed, depending on the scan result.
> > > +
> > > + - AfRange:
> > > + type: int32_t
> > > + description: |
> > > + Control to set the range of focus distances that is scanned. An
> > > + implementation may choose not to implement all the options here.
> > > + enum:
> > > + - name: AfRangeNormal
> > > + value: 0
> > > + description: |
> > > + A wide range of focus distances is scanned, all the way from
> > > + infinity down to close distances, though depending on the
> > > + implementation, possibly not including the very closest macro
> > > + positions.
> > > + - name: AfRangeMacro
> > > + value: 1
> > > + description: Only close distances are scanned.
> > > + - name: AfRangeFull
> > > + value: 2
> > > + description: |
> > > + The full range of focus distances is scanned just as with
> > > + AfRangeNormal but this time including the very closest macro
> > > + positions.
> > > +
> > > + - AfSpeed:
> > > + type: int32_t
> > > + description: |
> > > + Control that determines whether the AF algorithm is to move the lens
> > > + as quickly as possible or more steadily. For example, during video
> > > + recording it may be desirable not to move the lens too abruptly, but
> > > + when in a preview mode (waiting for a still capture) it may be
> > > + helpful to move the lens as quickly as is reasonably possible.
> > > + enum:
> > > + - name: AfSpeedNormal
> > > + value: 0
> > > + description: Move the lens at its usual speed.
> > > + - name: AfSpeedFast
> > > + value: 1
> > > + description: Move the lens more quickly.
> > > +
> > > + - AfWindows:
> > > + type: Rectangle
> > > + description: |
> > > + Sets the focus windows used by the AF algorithm. The units used express
> > > + a proportion of the ScalerCrop control (or if unavailable, of the entire
> > > + image), as u0.16 format numbers.
> >
> > I don't like too much the "or if unavailable", as it means that the
> > reference rectangle will change depending on the platform
> > capabilities.
I guess what I want is (like in the case of Android below), to specify
AF regions relative to the actual output image. If a platform supports
digital zoom (through ScalerCrop, I assume), the regions must be
relative to that part of the image. If there is no ScalerCrop, then do
we assume the platform doesn't support digital zoom? But the basic
idea is the same, the numbers supplied should be relative to the
output image in this case too.
Is there a better way to explain this, do you think?
> >
> > I understand your argument in reply to Jean-Michel comment on the
> > previous version of the series (quoted here for reference)
> >
> > "The thing that bothers me a bit about the full pixel array is that, if
> > you use digital zoom, then you will have to update your AfWindows as
> > they might otherwise lie outside your final image! If it's defined as
> > being a region of the scaler crop, then it automatically stays the
> > same (to the application, at least, the pipeline handler probably has
> > some work to do).
> >
> > I note that the Android link you give below says "Focus areas are
> > relative to the current field of view (getZoom())", which I guess is
> > partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
> > is easier to work with than the u0.16 format that I specified?"
> >
> > Let's reason on the IPA point of view. If the AF algorithm works by
> > estimating the contrast, will it do so before or after the ScalerCrop
> > rectangle is applied ? If a sensor provides PDAF statistics, will it
> > do so on the full pixel array size, the portion of if that gets
> > processed or on the final image ?
In the case of PDAF, we can presumably use information from whatever
part of the image that the sensor is outputting.
For CDAF (contrast detect AF) I guess it's possible that focus
statistics could be computed either before or after any cropping is
done in the ISP. (Maybe there are some platforms where cropping could
be delegated to the sensor?) So I'd guess that using any focus
statistics from outside the final FoV may actually be impossible,
depending on the platform. But even where you can do this, I don't
believe it's what anyone would want (and Android seems to behave like
this).
> >
> > Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
> > rectangle, so that they are applied to the actual FOV before being
> > passed to the AF algorithm ?
Not quite sure what you meant here. I'd expect the ISP to generate
focus statistics from the raw Bayer pixels that it gets, without too
much processing applied to them. So the job will be to take rectangles
defined in the final FoV and translate them into the image coming
directly from the sensor.
> >
> > One last point about the unit: is the below representation correct,
> > when it comes to use a Rectangle to represent a proportion ?
> > Regardless of the reference rectangle, if we have a 4x3 grid, will the
> > pipeline report 12 rectangles like the ones below, for applications to
> > chose from ?
> >
> > (0,0)
> > +-----+-----+-----+-----+
> > | | | | |
> > +-----+-----+-----+-----+
> > | | | | |
> > +-----+-----+-----+-----+
> > | | | | |
> > +-----+-----+-----+-----+
> >
> > AfWindows = {
> > {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
> > {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
> > {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
> > };
> >
> > My understanding is that the grid is fixed and depends on the platform
> > capabilities. I wonder if we shouldn't allow applications to select
> > grids by index.
I think there is a problem about determining the AF capabilities of a
platform. I could imagine some platforms will only accept a regular
grid of regions, but others might let you put them in arbitrary
places. Or a mixture of both. To be honest, I'm not sure how we handle
that sort of thing. Anyone have any good ideas?
> >
> > Also, do we need a property to report the grid structure and allows
> > application to select them from, or do you see this conveyed through a
> > ControlInfo ?
I wasn't quite sure what the question was here. Are you talking about
selecting the configuration for the AfWindows, or selecting focus
statistics from particular regions? Perhaps an example would help me
here!
> >
> >
> > > +
> > > + In order to be activated, a rectangle must be programmed with non-zero
> > > + width and height. If no rectangles are programmed in this way, then the
> > > + system will choose its own single default window in the centre of the
> > > + image.
> > > +
> > > + The details of how the windows are used are platform dependent. We note
> > > + that when there is more than one AF window, a typical implementation
> > > + might find the optimal focus position for each one and finally select
> > > + the window closest to the camera.
> >
> > What window is more close to the camera ? My understanding is that
> > windows apply to the sensor produced frame, hence, when it comes to distance,
> > they all sit on the same focal plan :)
Sorry, by "the window closest to the camera" I mean "the window that
is showing objects in the real world closest to the camera". I'll try
to re-word that a bit.
> >
> > > +
> > > + size: [n]
> > > +
> > > + - AfTrigger:
> > > + type: int32_t
> > > + description: |
> > > + This control starts an autofocus scan when AfMode is set to AfModeAuto,
> > > + and can also be used to terminate a scan early.
> > > +
> > > + It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> > > +
> > > + enum:
> > > + - name: AfTriggerStart
> > > + value: 0
> > > + description: Start an AF scan. Ignored if a scan is in progress.
> > > + - name: AfTriggerCancel
> > > + value: 1
> > > + description: Cancel an AF scan. This does not cause the lens to move
> > > + anywhere else. Ignored if no scan is in progress.
> > > +
> > > + - AfPause:
> > > + type: int32_t
> > > + description: |
> > > + This control has no effect except when in continuous autofocus mode
> > > + (AfModeContinuous). It can be used to pause any lens movements while
> > > + (for example) images are captured. The algorithm remains inactive
> > > + until it is instructed to resume.
> > > +
> > > + enum:
> > > + - name: AfPauseImmediate
> > > + value: 0
> > > + description: |
> > > + Pause the continuous autofocus algorithm immediately, whether or
> > > + not any kind of scan is underway. AfPauseState will subsequently
> > > + report AfPauseStatePaused. AfState may report any of AfStateScanning,
> > > + AfStateFocused or AfStateFailed, depending on the algorithm's state
> > > + when it received this control.
> > > + - name: AfPauseDeferred
> > > + value: 1
> > > + description: |
> > > + This is similar to AfPauseImmediate, and if the AfState is currently
> > > + reporting AfStateFocused or AfStateFailed it will remain in that
> > > + state and AfPauseState will report AfPauseStatePaused.
> > > +
> > > + However, if the algorithm is scanning (AfStateScanning), AfPauseState
> > > + will report AfPauseStatePausing until the scan is finished, at which
> > > + point AfState will report one of AfStateFocused or AfStateFailed, and
> > > + AfPauseState will change to AfPauseStatePaused.
> >
> > From here on I see lines getting over 80 cols by a few chars. Can we
> > align it back ?
Sure, I'll have a look!
> >
> > > + - name: AfPauseResume
> > > + value: 2
> > > + description: |
> > > + Resume continuous autofocus operation. The algorithm starts again
> > > + from exactly where it left off, and AfPauseState will report
> > > + AfPauseStateRunning.
> > > +
> > > + - LensPosition:
> > > + type: float
> > > + description: |
> > > + Acts as a control to instruct the lens to move to a particular position
> > > + and also reports back the position of the lens for each frame.
> > > +
> > > + The LensPosition control is ignored unless the AfMode is set to
> > > + AfModeManual.
> > > +
> > > + The units are dioptres divided by the hyperfocal distance. Non-integer
> > > + values are permitted. For example:
> > > + 0 moves the lens to infinity.
> > > + 0.5 moves the lens to twice the hyperfocal distance.
> > > + 1 moves the lens to the hyperfocal position.
> > > + And larger values will focus the lens ever closer.
> > > +
> >
> > This seems good to me, albeit my understanding of the issue is limited :)
> >
>
> Do we need a property to report the lens hyperfocal distance ?
Yes, sometimes. Android distinguishes between calibrated and
uncalibrated cameras. For a calibrated camera an application should be
able to set the focal distance to (for example) 1m, and given the
units I've proposed, you would need to know the hyperfocal distance to
calculate the right number.
For an uncalibrated camera I don't think it matters. Many applications
would simply set the lens position to 1.0 (meaning hyperfocal) and
you'll get the "best overall" focal distance (largest depth of field),
and that's it.
There's also a need to know the allowable range of lens movement. This
will depend on the type of module (knowing the lens driver isn't
enough). But I don't see that applications need to know this, I think
the pipeline handler and AF implementation should take care of it.
>
>
> > > + - AfState:
> > > + type: int32_t
> > > + description: |
> > > + Reports the current state of the AF algorithm in conjunction with the
> > > + reported AfMode value and (in continuous AF mode) the AfPauseState value.
> > > + The possible state changes are described below, though we note the following
> > > + state transitions that occur when the AfMode is changed.
> > > +
> > > + If the AfMode is set to AfModeManual then the AfState will report
> > > + AfStateReset. This action on its own does not cause the lens to move; the
> > > + LensPosition control must subsequently be used to achieve this.
> > > +
> > > + If the AfMode is set to AfModeAuto then the AfState will report
> > > + AfStateReset, unless AfTriggerStart is sent at the same time in which
> >
> > s/unless/until ?
> > s/at the same time// ?
I'm trying to describe what happens to the AfState when you send
AfMode = AfModeAuto. It will go to AfStateIdle (even if you later send
AfTriggerStart). The only occasion when you won't see AfStateIdle will
be if you send AfMode = AfModeAuto and AfTriggerStart together. So I
think this reads OK to me, but I'll see if I can make the wording any
clearer.
> >
> > > + case it will (start a scan and) report AfStateScanning.
> > > +
> > > + If the AfMode is set to AfModeContinuous then the AfState will initially
> > > + report AfStateScanning.
> > > +
> > > + enum:
> > > + - name: AfStateReset
> > > + value: 0
> > > + description: |
> > > + The AF algorithm is in manual mode (AfModeManual) or in auto mode
> > > + (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> > > + scan was cancelled.
> > > + - name: AfStateScanning
> > > + value: 1
> > > + description: |
> > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> > > + started using the AfTrigger control. The scan can be cancelled by
> > > + sending AfTriggerCancel at which point the algorithm will either
> > > + move back to AfStateReset or, if the scan actually completes
> > > + before the cancel request is processed, to one of
> > > + AfStateFocused or AfStateFailed.
> > > +
> > > + Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
> >
> > s/be be/is ?
Yes, thanks!
> >
> > > + at which point it may enter this state spontaneously whenever it
> > > + determines that a rescan is needed.
> > > + - name: AfStateFocused
> > > + value: 2
> > > + description: |
> > > + The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > + mode and a scan has completed with the result that the algorithm believes
> > > + the image is now in focus.
> > > + - name: AfStateFailed
> > > + value: 3
> > > + description: |
> > > + The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> > > + mode and a scan has completed with the result that the algorithm did not
> > > + find a good focus position.
> > > +
> > > + - AfPauseState:
> > > + type: int32_t
> > > + description: |
> > > + Only applicable in continuous (AfModeContinuous) mode, this reports whether
> > > + the algorithm is currently running, paused or pausing (that is, will pause
> > > + as soon as any in-progress scan completes).
> > > +
> > > + Any change to AfMode will cause AfPauseStateRunning to be reported.
> > > +
> > > + enum:
> > > + - name: AfPauseStateRunning
> > > + value: 0
> > > + description: |
> > > + Continuous AF is running and the algorithm may restart a scan
> > > + spontaneously.
> > > + - name: AfPauseStatePausing
> > > + value: 1
> > > + description: |
> > > + Continuous AF has been sent an AfPauseDeferred control, and will pause
> > > + as soon as any in-progress scan completes (and then report
> > > + AfPauseStatePaused). No new scans will be start spontaneously until
> > > + the AfPauseResume control is sent.
> > > + - name: AfPauseStatePaused
> > > + value: 2
> > > + description: |
> > > + Continuous AF is paused. No further state changes or lens movements
> > > + will occur until the AfPauseResume control is sent.
> > > +
> >
> > Very nice overall! With a few more clarifications I think this is good
> > to go!
> >
> > Thanks
> > j
Thanks!
David
> >
> > > # ----------------------------------------------------------------------------
> > > # Draft controls section
> > >
> > > @@ -406,27 +664,6 @@ controls:
> > > The camera will cancel any active or completed metering sequence.
> > > The AE algorithm is reset to its initial state.
> > >
> > > - - AfTrigger:
> > > - type: int32_t
> > > - draft: true
> > > - description: |
> > > - Control for AF trigger. Currently identical to
> > > - ANDROID_CONTROL_AF_TRIGGER.
> > > -
> > > - Whether the camera device will trigger autofocus for this request.
> > > - enum:
> > > - - name: AfTriggerIdle
> > > - value: 0
> > > - description: The trigger is idle.
> > > - - name: AfTriggerStart
> > > - value: 1
> > > - description: The AF routine is started by the camera.
> > > - - name: AfTriggerCancel
> > > - value: 2
> > > - description: |
> > > - The camera will cancel any active trigger and the AF routine is
> > > - reset to its initial state.
> > > -
> > > - NoiseReductionMode:
> > > type: int32_t
> > > draft: true
> > > @@ -507,45 +744,6 @@ controls:
> > > The AE algorithm has started a pre-capture metering session.
> > > \sa AePrecaptureTrigger
> > >
> > > - - AfState:
> > > - type: int32_t
> > > - draft: true
> > > - description: |
> > > - Control to report the current AF algorithm state. Currently identical to
> > > - ANDROID_CONTROL_AF_STATE.
> > > -
> > > - Current state of the AF algorithm.
> > > - enum:
> > > - - name: AfStateInactive
> > > - value: 0
> > > - description: The AF algorithm is inactive.
> > > - - name: AfStatePassiveScan
> > > - value: 1
> > > - description: |
> > > - AF is performing a passive scan of the scene in continuous
> > > - auto-focus mode.
> > > - - name: AfStatePassiveFocused
> > > - value: 2
> > > - description: |
> > > - AF believes the scene is in focus, but might restart scanning.
> > > - - name: AfStateActiveScan
> > > - value: 3
> > > - description: |
> > > - AF is performing a scan triggered by an AF trigger request.
> > > - \sa AfTrigger
> > > - - name: AfStateFocusedLock
> > > - value: 4
> > > - description: |
> > > - AF believes has focused correctly and has locked focus.
> > > - - name: AfStateNotFocusedLock
> > > - value: 5
> > > - description: |
> > > - AF has not been able to focus and has locked.
> > > - - name: AfStatePassiveUnfocused
> > > - value: 6
> > > - description: |
> > > - AF has completed a passive scan without finding focus.
> > > -
> > > - AwbState:
> > > type: int32_t
> > > draft: true
> > > --
> > > 2.30.2
> > >
More information about the libcamera-devel
mailing list