[libcamera-devel] [PATCH v3 1/1] libcamera: controls: Controls for driving AF (autofocus) algorithms
Jean-Michel Hautbois
jeanmichel.hautbois at ideasonboard.com
Thu Apr 28 17:43:43 CEST 2022
Hello Hanlin,
Thanks for your remarks !
On 28/04/2022 14:45, Hanlin Chen wrote:
> Hi everyone,
>
> On Tue, Apr 26, 2022 at 9:16 PM Jacopo Mondi <jacopo at jmondi.org> wrote:
>>
>> Hi David,
>>
>> On Mon, Apr 25, 2022 at 04:50:06PM +0100, David Plowman wrote:
>>> Hi Jacopo
>>>
>>> Thanks for the message. Some replies and more questions below!
>>>
>>> On Thu, 21 Apr 2022 at 15:31, Jacopo Mondi <jacopo at jmondi.org> wrote:
>>>>
>>>> Hi David,
>>>>
>>>> On Thu, Apr 21, 2022 at 01:59:53PM +0100, David Plowman wrote:
>>>>> Hi Jacopo
>>>>>
>>>>> Thanks for the quick reply!
>>>>>
>>>>> On Thu, 21 Apr 2022 at 10:25, Jacopo Mondi <jacopo at jmondi.org> wrote:
>>>>>>
>>>>>> Ah, forgot one thing!
>>>>>>
>>>>>> On Thu, Apr 21, 2022 at 11:24:16AM +0200, Jacopo Mondi wrote:
>>>>>>> Hi David,
>>>>>>> thanks for this new version
>>>>>>>
>>>>>>> On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
>>>>>>>> This patch describes a series of controls that allow applications to
>>>>>>>> drive AF algorithms:
>>>>>>>>
>>>>>>>> AfMode - manual, auto or continuous
>>>>>>>> AfRange - full, macro or normal
>>>>>>>> AfSpeed - fast or slowwith
>>>>>>>> AfWindows - AF window locations
>>>>>>>> AfTrigger - start (trigger) an AF scan or cancel
>>>>>>>> AfPause - pause continuous AF
>>>>>>>> LensPosition - set or retrieve position of lens
>>>>>>>> AfState - reports whether scanning/success/failure
>>>>>>>> AfPauseState - reports whether continuous AF paused or not
>>>>>>>>
>>>>>>>> Signed-off-by: David Plowman <david.plowman at raspberrypi.com>
>>>>>>>> ---
>>>>>>>> src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
>>>>>>>> 1 file changed, 258 insertions(+), 60 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
>>>>>>>> index 9d4638ae..eccf18bc 100644
>>>>>>>> --- a/src/libcamera/control_ids.yaml
>>>>>>>> +++ b/src/libcamera/control_ids.yaml
>>>>>>>> @@ -381,6 +381,264 @@ controls:
>>>>>>>> \todo Define how the sensor timestamp has to be used in the reprocessing
>>>>>>>> use case.
>>>>>>>>
>>>>>>>> + - AfMode:
>>>>>>>> + type: int32_t
>>>>>>>> + description: |
>>>>>>>> + Control to set the mode of the AF (autofocus) algorithm. Applications
>>>>>>>> + are allowed to set a new mode, and to send additional controls for
>>>>>>>> + that new mode, in the same request.
>>>>>>>> +
>>>>>>>> + An implementation may choose not to implement all the modes.
>>>>>>>> +
>>>>>>>> + enum:
>>>>>>>> + - name: AfModeManual
>>>>>>>> + value: 0
>>>>>>>> + description: |
>>>>>>>> + The AF algorithm is in manual mode. In this mode it will never
>>>>>>>> + perform any action nor move the lens of its own accord, but an
>>>>>>>> + application can set controls to move the lens.
>>>>>>>> +
>>>>>>>
>>>>>>> If, as I understand it, the lens can only be moved with 'LensPosition'
>>>>>>> when in manual mode, I would say so explicitly
>>>>>>>
>>>>>>> "... but an application can specify the desired lens
>>>>>>> position using the LensPosition control."
>>>>>
>>>>> Yes, will do.
>>>>>
>>>>>>>
>>>>>>>> + In this mode the AfState will always report AfStateReset.
>>>>>>>
>>>>>>> I'm tempted to say AfState is useless and can not be reported at all
>>>>>>> in this case. For simplicity (for ph/ipa) I'm fine reporting it
>>>>>>> regardless, even if seeing 'reset' might be slightly confusing. I
>>>>>>> don't have better alternatives to propose though.
>>>>>
>>>>> Don't really mind either. I think reporting stuff all the time might
>>>>> be slightly easier, at least it's always obvious what to expect...
>>>>>
>>>>>>>
>>>>>>> Reading the AfStateReset description
>>>>>>>
>>>>>>> - name: AfStateReset
>>>>>>> value: 0
>>>>>>> description: |
>>>>>>> The AF algorithm is in manual mode (AfModeManual) or in auto mode
>>>>>>> (AfModeAuto) and a scan has not yet been triggered, or an in-progress
>>>>>>> scan was cancelled.
>>>>>>>
>>>>>>> I wonder if this isn't better named AfStateIdle
>>>>>
>>>>> Yes, also fine with that.
>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> + - name: AfModeAuto
>>>>>>>> + value: 1
>>>>>>>> + description: |
>>>>>>>> + The AF algorithm is in auto mode. This means that the algorithm
>>>>>>>> + will never move the lens or change state unless the AfTrigger
>>>>>>>> + control is used. The AfTrigger control can be used to initiate a
>>>>>>>> + focus scan, the results of which will be reported by AfState.
>>>>>>>> +
>>>>>>>> + If the autofocus algorithm is moved from AfModeAuto to another
>>>>>>>> + mode while a scan is in progress, the scan is cancelled
>>>>>>>> + immediately, without waiting for the scan to finish.
>>>>>>>> +
>>>>>>>> + When first entering this mode the AfState will report
>>>>>>>> + AfStateReset. When a trigger control is sent, AfState will
>>>>>>>> + report AfStateScanning for a period before spontaneously
>>>>>>>> + changing to AfStateFocused or AfStateFailed, depending on
>>>>>>>> + the outcome of the scan. It will remain in this state until
>>>>>>>> + another scan is initiated by the AfTrigger control. If a scan is
>>>>>>>> + cancelled (without changing to another mode), AfState will return
>>>>>>>> + to AfStateReset.
>>>>>>>> + - name: AfModeContinuous
>>>>>>>> + value: 2
>>>>>>>> + description: |
>>>>>>>> + The AF algorithm is in continuous mode. This means that the lens
>>>>>>>> + can re-start a scan spontaneously at any moment, without any user
>>>>>>>> + intervention. The AfState still reports whether the algorithm is
>>>>>>>> + currently scanning or not, though the application has no ability
>>>>>>>> + to initiate or cancel scans (though it can "pause" them), nor to
>>>>>>>> + move the lens for itself.
>>>>>>>> +
>>>>>>>> + When set to AfModeContinuous, the system will immediately initiate
>>>>>>>> + a scan so AfState will report AfStateScanning, and will settle on
>>>>>>>> + one of AfStateFocused or AfStateFailed, depending on the scan result.
>>>>>>>> +
>>>>>>>> + - AfRange:
>>>>>>>> + type: int32_t
>>>>>>>> + description: |
>>>>>>>> + Control to set the range of focus distances that is scanned. An
>>>>>>>> + implementation may choose not to implement all the options here.
>>>>>>>> + enum:
>>>>>>>> + - name: AfRangeNormal
>>>>>>>> + value: 0
>>>>>>>> + description: |
>>>>>>>> + A wide range of focus distances is scanned, all the way from
>>>>>>>> + infinity down to close distances, though depending on the
>>>>>>>> + implementation, possibly not including the very closest macro
>>>>>>>> + positions.
>>>>>>>> + - name: AfRangeMacro
>>>>>>>> + value: 1
>>>>>>>> + description: Only close distances are scanned.
>>>>>>>> + - name: AfRangeFull
>>>>>>>> + value: 2
>>>>>>>> + description: |
>>>>>>>> + The full range of focus distances is scanned just as with
>>>>>>>> + AfRangeNormal but this time including the very closest macro
>>>>>>>> + positions.
>>>>>>>> +
>>>>>>>> + - AfSpeed:
>>>>>>>> + type: int32_t
>>>>>>>> + description: |
>>>>>>>> + Control that determines whether the AF algorithm is to move the lens
>>>>>>>> + as quickly as possible or more steadily. For example, during video
>>>>>>>> + recording it may be desirable not to move the lens too abruptly, but
>>>>>>>> + when in a preview mode (waiting for a still capture) it may be
>>>>>>>> + helpful to move the lens as quickly as is reasonably possible.
>>>>>>>> + enum:
>>>>>>>> + - name: AfSpeedNormal
>>>>>>>> + value: 0
>>>>>>>> + description: Move the lens at its usual speed.
>>>>>>>> + - name: AfSpeedFast
>>>>>>>> + value: 1
>>>>>>>> + description: Move the lens more quickly.
>>>>>>>> +
>>>>>>>> + - AfWindows:
>>>>>>>> + type: Rectangle
>>>>>>>> + description: |
>>>>>>>> + Sets the focus windows used by the AF algorithm. The units used express
>>>>>>>> + a proportion of the ScalerCrop control (or if unavailable, of the entire
>>>>>>>> + image), as u0.16 format numbers.
>>>>>>>
>>>>>>> I don't like too much the "or if unavailable", as it means that the
>>>>>>> reference rectangle will change depending on the platform
>>>>>>> capabilities.
>>>>>
>>>>> I guess what I want is (like in the case of Android below), to specify
>>>>> AF regions relative to the actual output image. If a platform supports
>>>>> digital zoom (through ScalerCrop, I assume), the regions must be
>>>>> relative to that part of the image. If there is no ScalerCrop, then do
>>>>> we assume the platform doesn't support digital zoom? But the basic
>>>>> idea is the same, the numbers supplied should be relative to the
>>>>> output image in this case too.
>>>>>
>>>>
>>>> Thing is that there's no "output image size" as two streams produced
>>>> from the same camera might have different output sizes ?
>>>
>>> You're right that there might be several outputs. So what do I really
>>> mean? I'm sort of looking for the rectangle in the sensor image that
>>> is required to make all the outputs, but I think the ScalerCrop
>>> encapsulates that idea reasonably well. Where there isn't a
>>> ScalerCrop, I think we use the active pixel area instead.
>>>
>>
>> Gotcha! Indeed scaler crop defines the area of the sensor's output
>> frame used to generate all streams... Thanks for sticking to it and
>> help me straight my thoughts.
>>
>> I still don't like the "if not available part" :(
>>
>> I'll try to rope in others to get their opinions. What concerns me is
>> that applications will behave differently on platforms where scaler
>> crop is available compared to ones where it is not. Maybe I'm just
>> over-concerned ?
>>
>>
>>>>
>>>>> Is there a better way to explain this, do you think?
>>>>>
>>>>>>>
>>>>>>> I understand your argument in reply to Jean-Michel comment on the
>>>>>>> previous version of the series (quoted here for reference)
>>>>>>>
>>>>>>> "The thing that bothers me a bit about the full pixel array is that, if
>>>>>>> you use digital zoom, then you will have to update your AfWindows as
>>>>>>> they might otherwise lie outside your final image! If it's defined as
>>>>>>> being a region of the scaler crop, then it automatically stays the
>>>>>>> same (to the application, at least, the pipeline handler probably has
>>>>>>> some work to do).
>>>>>>>
>>>>>>> I note that the Android link you give below says "Focus areas are
>>>>>>> relative to the current field of view (getZoom())", which I guess is
>>>>>>> partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
>>>>>>> is easier to work with than the u0.16 format that I specified?"
>>>>>>>
>>>>>>> Let's reason on the IPA point of view. If the AF algorithm works by
>>>>>>> estimating the contrast, will it do so before or after the ScalerCrop
>>>>>>> rectangle is applied ? If a sensor provides PDAF statistics, will it
>>>>>>> do so on the full pixel array size, the portion of if that gets
>>>>>>> processed or on the final image ?
>>>>>
>>>>> In the case of PDAF, we can presumably use information from whatever
>>>>> part of the image that the sensor is outputting.
>>>>>
>>>>> For CDAF (contrast detect AF) I guess it's possible that focus
>>>>> statistics could be computed either before or after any cropping is
>>>>> done in the ISP. (Maybe there are some platforms where cropping could
>>>>> be delegated to the sensor?) So I'd guess that using any focus
>>>>> statistics from outside the final FoV may actually be impossible,
>>>>> depending on the platform. But even where you can do this, I don't
>>>>> believe it's what anyone would want (and Android seems to behave like
>>>>> this).
>>>>>
>>>>>>>
>>>>>>> Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
>>>>>>> rectangle, so that they are applied to the actual FOV before being
>>>>>>> passed to the AF algorithm ?
>>>>>
>>>>> Not quite sure what you meant here. I'd expect the ISP to generate
>>>>> focus statistics from the raw Bayer pixels that it gets, without too
>>>>> much processing applied to them. So the job will be to take rectangles
>>>>> defined in the final FoV and translate them into the image coming
>>>>> directly from the sensor.
>>>>>
>>>>
>>>> What I meant is: if we specify the AfWindows relative to the full
>>>> pixel array, can't we then re-scale it to final cropped output ? or if
>>>> no copping on the sensor's output frame size.
>>>>
>>>> This would be similar to what you do with ScalerCrop already ?
>>>>
>>>> void RPiCameraData::applyScalerCrop(const ControlList &controls) {
>>>>
>>>> Rectangle nativeCrop = controls.get<Rectangle>(controls::ScalerCrop);
>>>>
>>>> /* Create a version of the crop scaled to ISP (camera mode) pixels. */
>>>> Rectangle ispCrop = nativeCrop.translatedBy(-sensorInfo_.analogCrop.topLeft());
>>>> ispCrop.scaleBy(sensorInfo_.outputSize, sensorInfo_.analogCrop.size());
>>>> }
>>>>
>>>> But I guess I'm reasoning in absolute coordinates, not in proportions
>>>> like you have proposed here.
>>>
>>> I think the question here is whether to use some kind of fractional
>>> coordinates for describing the part of the ScalerCrop that we want AF
>>> to use, or if we simply use pixel coordinates within the ScalerCrop
>>> (or active pixel array), which would then look rather like the code
>>> above.
>>>
>>> I suppose I don't particularly mind. If we use actual pixel
>>> coordinates it's maybe a bit less clear that some other scaling
>>> (shrinking to lie within the ScalerCrop) might be necessary, but if
>>> we're happy that this is going to happen anyway then I think that
>>> would work for me.
>>>
>>
>> Let's hear from others as well!
>
> I think the focusAreas() is deprecated on Android.
> https://developer.android.com/reference/android/hardware/Camera.Parameters#getFocusAreas()
> Currently it should be AF_REGIONS.
> https://developer.android.com/reference/android/hardware/camera2/CaptureRequest#CONTROL_AF_REGIONS
>
Thank you, it is interesting !
> It uses the pixel array as the coordinate system of both ScalarCrop
> and AFWindows.
> For the problem that the AFWindow may be outside of the ScalarCrop
> region, it adds an intersection rule:
> Only the intersection of AFWindows and ScalarCrop takes effect.
>
> To be honest, the intersection rule is a little weird. I think
> limiting the AFWindow to ScalarCrop looks good to me. The Android
> adapter can do the intersection.
> On the other hand, I think the benefit of using pixel arrays as
> coordinate is that the precision can be at pixel level.
> Not sure if it matters for any other algorithms.
>
> Another thing I'd like to mention is a rule for ScalarCrop in Android, I quote:
> "The crop region is applied after the RAW to other color space (e.g.
> YUV) conversion. Since raw streams (e.g. RAW16) don't have the
> conversion stage, they are not croppable. The crop region will be
> ignored by raw streams."
>
> Since the statistics is usually generated from the RAW, the input for
> IPA would be the full pixel array's statistics.
> In result, the IPA needs to translate the coordinate to match the
> pixel array anyway.
> It might be the reason that both ScalarCrop and AFWindows are in the
> pixel array coordinate, since they will be in the end.
>
> This is also a weird rule, because it somehow forbids the device to
> crop on sensor.
>
>>
>>>>
>>>>>>>
>>>>>>> One last point about the unit: is the below representation correct,
>>>>>>> when it comes to use a Rectangle to represent a proportion ?
>>>>>>> Regardless of the reference rectangle, if we have a 4x3 grid, will the
>>>>>>> pipeline report 12 rectangles like the ones below, for applications to
>>>>>>> chose from ?
>>>>>>>
>>>>>>> (0,0)
>>>>>>> +-----+-----+-----+-----+
>>>>>>> | | | | |
>>>>>>> +-----+-----+-----+-----+
>>>>>>> | | | | |
>>>>>>> +-----+-----+-----+-----+
>>>>>>> | | | | |
>>>>>>> +-----+-----+-----+-----+
>>>>>>>
>>>>>>> AfWindows = {
>>>>>>> {0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
>>>>>>> {0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
>>>>>>> {0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
>>>>>>> };
>>>>>>>
>>>>>>> My understanding is that the grid is fixed and depends on the platform
>>>>>>> capabilities. I wonder if we shouldn't allow applications to select
>>>>>>> grids by index.
>>>>>
>>>>> I think there is a problem about determining the AF capabilities of a
>>>>> platform. I could imagine some platforms will only accept a regular
>>>>> grid of regions, but others might let you put them in arbitrary
>>>>> places. Or a mixture of both. To be honest, I'm not sure how we handle
>>>>> that sort of thing. Anyone have any good ideas?
>>>>>
>>>>
>>>> Exactly, that's why having proportions as the ones I tried to sketch
>>>> out above doesn't really fit as a model in my head. Each platform
>>>> would have its own 'grid', do we need to expose it to applications ?
>>>>
>>>> Can't we let application specify Rectangles with absolute coordinates
>>>> (relative to PixelArray or else) and let the IPA/PH map those areas on
>>>> their grids ?
>>>
>>> I really don't know how to deal with this. Some ISPs will have grids
>>> of some sort, some may have windows in arbitrary locations, different
>>> numbers and sizes of each depending on the platform, and some may do
>>> both. Typically I would imagine many applications wanting to use grids
>>> much of the time, but perhaps arbitrary rectangles for following
>>> objects, people and faces.
>
> It's actually happening. GCam is using its own face detection
> algorithm and passes the region to HAL.
> ChromeOS is also integrating face algorithms to choose ROI for 3A.
> Although they are for AE mostly, I think the windows for 3A should
> have the same coordinate system.
> I guess (I'm not sure) this is why the grid based focusAreas() is
> deprecated on Android, and moves to pixel array coordinates, since the
> face algorithm usually works on pixel coordinates. I think the grid of
> ISPs could be left as implementation details for the pipeline handler?
>
I agree with this, we should rely on the pixel array coordinates, and
let the PH do the conversion it wants to find the intersecting grid
given its statistics format.
I find the object MeteringRectangle very interesting [1] and I wonder if
this could be an internal object for 3A algorithms to use (could be
useful for AGC and AF at least) ?
[1]:
https://developer.android.com/reference/android/hardware/camera2/params/MeteringRectangle
JM
>>
>> Indeed. This is an advanced feature for which it might be expected
>> application to know what the platform can provide maybe ? Describing
>> all possible options seems very complex, and we'll never be able to
>> express all of them.
>>
>> I would be fine stabilizing on a simpler grid-based version of AfWindows
>> for now and improve on top, maybe ?
>>
>>>
>>> Perhaps having an AfWindowGrid control and an AfWindows array control
>>> of Rectangles could work? For the former, perhaps we leave the
>>> decisions on the size of the grid to the platform/algorithm - I'd be
>>> worried about having so many parameters that it would be difficult to
>>> use. (You'd have to think about the size of the grid, the spacing of
>>> rectangles in the grid etc... but perhaps Android requires us to
>>> expose this sort of thing?) Perhaps someone knows what Android would
>>> require here?
>>>
>>>>
>>>>>>>
>>>>>>> Also, do we need a property to report the grid structure and allows
>>>>>>> application to select them from, or do you see this conveyed through a
>>>>>>> ControlInfo ?
>>>>>
>>>>> I wasn't quite sure what the question was here. Are you talking about
>>>>> selecting the configuration for the AfWindows, or selecting focus
>>>>> statistics from particular regions? Perhaps an example would help me
>>>>> here!
>>>>>
>>>>
>>>> I'm talking about how would an application know how many spots it can
>>>> select from, in other words, how do we let an application know that,
>>>> looking at the above example, each grid is 1/4 in width and 1/3 in
>>>> height and not 1/3 and 1/2 in case you have a 3x2 grid instead of a
>>>> 4x3 one.
>>>>
>>>> Or maybe I got it all wrong, and you where implying already that the
>>>> here specified Rectangles cover arbitrary parts of the image and it's
>>>> the IPA that has to match them on their grids.
>>>
>>> Or maybe we treat grids as a whole different thing? (as per the
>>> comments just above)
>>>
>>>>
>>>>>>>
>>>>>>>
>>>>>>>> +
>>>>>>>> + In order to be activated, a rectangle must be programmed with non-zero
>>>>>>>> + width and height. If no rectangles are programmed in this way, then the
>>>>>>>> + system will choose its own single default window in the centre of the
>>>>>>>> + image.
>>>>>>>> +
>>>>>>>> + The details of how the windows are used are platform dependent. We note
>>>>>>>> + that when there is more than one AF window, a typical implementation
>>>>>>>> + might find the optimal focus position for each one and finally select
>>>>>>>> + the window closest to the camera.
>>>>>>>
>>>>>>> What window is more close to the camera ? My understanding is that
>>>>>>> windows apply to the sensor produced frame, hence, when it comes to distance,
>>>>>>> they all sit on the same focal plan :)
>>>>>
>>>>> Sorry, by "the window closest to the camera" I mean "the window that
>>>>> is showing objects in the real world closest to the camera". I'll try
>>>>> to re-word that a bit.
>>>>>
>>>>
>>>> Yeah sorry, I could have realized that myself :)
>>>>
>>>>>>>
>>>>>>>> +
>>>>>>>> + size: [n]
>>>>>>>> +
>>>>>>>> + - AfTrigger:
>>>>>>>> + type: int32_t
>>>>>>>> + description: |
>>>>>>>> + This control starts an autofocus scan when AfMode is set to AfModeAuto,
>>>>>>>> + and can also be used to terminate a scan early.
>>>>>>>> +
>>>>>>>> + It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
>>>>>>>> +
>>>>>>>> + enum:
>>>>>>>> + - name: AfTriggerStart
>>>>>>>> + value: 0
>>>>>>>> + description: Start an AF scan. Ignored if a scan is in progress.
>>>>>>>> + - name: AfTriggerCancel
>>>>>>>> + value: 1
>>>>>>>> + description: Cancel an AF scan. This does not cause the lens to move
>>>>>>>> + anywhere else. Ignored if no scan is in progress.
>>>>>>>> +
>>>>>>>> + - AfPause:
>>>>>>>> + type: int32_t
>>>>>>>> + description: |
>>>>>>>> + This control has no effect except when in continuous autofocus mode
>>>>>>>> + (AfModeContinuous). It can be used to pause any lens movements while
>>>>>>>> + (for example) images are captured. The algorithm remains inactive
>>>>>>>> + until it is instructed to resume.
>>>>>>>> +
>>>>>>>> + enum:
>>>>>>>> + - name: AfPauseImmediate
>>>>>>>> + value: 0
>>>>>>>> + description: |
>>>>>>>> + Pause the continuous autofocus algorithm immediately, whether or
>>>>>>>> + not any kind of scan is underway. AfPauseState will subsequently
>>>>>>>> + report AfPauseStatePaused. AfState may report any of AfStateScanning,
>>>>>>>> + AfStateFocused or AfStateFailed, depending on the algorithm's state
>>>>>>>> + when it received this control.
>>>>>>>> + - name: AfPauseDeferred
>>>>>>>> + value: 1
>>>>>>>> + description: |
>>>>>>>> + This is similar to AfPauseImmediate, and if the AfState is currently
>>>>>>>> + reporting AfStateFocused or AfStateFailed it will remain in that
>>>>>>>> + state and AfPauseState will report AfPauseStatePaused.
>>>>>>>> +
>>>>>>>> + However, if the algorithm is scanning (AfStateScanning), AfPauseState
>>>>>>>> + will report AfPauseStatePausing until the scan is finished, at which
>>>>>>>> + point AfState will report one of AfStateFocused or AfStateFailed, and
>>>>>>>> + AfPauseState will change to AfPauseStatePaused.
>>>>>>>
>>>>>>> From here on I see lines getting over 80 cols by a few chars. Can we
>>>>>>> align it back ?
>>>>>
>>>>> Sure, I'll have a look!
>>>>>
>>>>>>>
>>>>>>>> + - name: AfPauseResume
>>>>>>>> + value: 2
>>>>>>>> + description: |
>>>>>>>> + Resume continuous autofocus operation. The algorithm starts again
>>>>>>>> + from exactly where it left off, and AfPauseState will report
>>>>>>>> + AfPauseStateRunning.
>>>>>>>> +
>>>>>>>> + - LensPosition:
>>>>>>>> + type: float
>>>>>>>> + description: |
>>>>>>>> + Acts as a control to instruct the lens to move to a particular position
>>>>>>>> + and also reports back the position of the lens for each frame.
>>>>>>>> +
>>>>>>>> + The LensPosition control is ignored unless the AfMode is set to
>>>>>>>> + AfModeManual.
>>>>>>>> +
>>>>>>>> + The units are dioptres divided by the hyperfocal distance. Non-integer
>>>>>>>> + values are permitted. For example:
>>>>>>>> + 0 moves the lens to infinity.
>>>>>>>> + 0.5 moves the lens to twice the hyperfocal distance.
>>>>>>>> + 1 moves the lens to the hyperfocal position.
>>>>>>>> + And larger values will focus the lens ever closer.
>>>>>>>> +
>>>>>>>
>>>>>>> This seems good to me, albeit my understanding of the issue is limited :)
>>>>>>>
>>>>>>
>>>>>> Do we need a property to report the lens hyperfocal distance ?
>>>>>
>>>>> Yes, sometimes. Android distinguishes between calibrated and
>>>>> uncalibrated cameras. For a calibrated camera an application should be
>>>>> able to set the focal distance to (for example) 1m, and given the
>>>>> units I've proposed, you would need to know the hyperfocal distance to
>>>>> calculate the right number.
>>>>
>>>> Right, so we need a LensHyperfocalDistance or something for calibrated
>>>> lenses ?
>>>
>>> I think so.
>>>
>>>>
>>>>>
>>>>> For an uncalibrated camera I don't think it matters. Many applications
>>>>> would simply set the lens position to 1.0 (meaning hyperfocal) and
>>>>> you'll get the "best overall" focal distance (largest depth of field),
>>>>> and that's it.
>>>>
>>>> I think this part would be relevant to be captured in the control
>>>> description.
>>>
>>> Yes, I can add something about that.
>>>
>>>>
>>>>>
>>>>> There's also a need to know the allowable range of lens movement. This
>>>>> will depend on the type of module (knowing the lens driver isn't
>>>>> enough). But I don't see that applications need to know this, I think
>>>>> the pipeline handler and AF implementation should take care of it.
>>>>>
>>>>
>>>> I think so. When it comes to the actual lens movement range I don't
>>>> think it's relevant for applications. But do the actual lens movement
>>>> range maps to a range of LensPosition values that can be specified ? I
>>>> guess it's hard to map them on un-calibrated lenses, but for
>>>> calibrated ones a LensPositionRanges whose ControlInfo informs
>>>> application on the range's min and max might make sense ? (Please note
>>>> that this could also be used for the use case we discussed about
>>>> wanting to limit the lens range search scope)
>>>
>>> I think I would expect a calibrated lens would have to have some kind
>>> of property that says what the nearest and furthest focal distances
>>> are (the furthest usually being zero, meaning infinity). For an
>>> uncalibrated lens things will only ever be "approximately correct".
>>
>>
>> Care to add the property in next version, or can we do it on top with
>> a separate patch ?
>>
>> Thanks
>> j
>>
>>>
>>> I could imagine this being used in conjunction with the hypothetical
>>> AfRangeCustom to construct custom search ranges for particular
>>> applications (in future, perhaps).
>>>
>>> Thanks again!
>>> David
>>>
>>>>
>>>>>>
>>>>>>
>>>>>>>> + - AfState:
>>>>>>>> + type: int32_t
>>>>>>>> + description: |
>>>>>>>> + Reports the current state of the AF algorithm in conjunction with the
>>>>>>>> + reported AfMode value and (in continuous AF mode) the AfPauseState value.
>>>>>>>> + The possible state changes are described below, though we note the following
>>>>>>>> + state transitions that occur when the AfMode is changed.
>>>>>>>> +
>>>>>>>> + If the AfMode is set to AfModeManual then the AfState will report
>>>>>>>> + AfStateReset. This action on its own does not cause the lens to move; the
>>>>>>>> + LensPosition control must subsequently be used to achieve this.
>>>>>>>> +
>>>>>>>> + If the AfMode is set to AfModeAuto then the AfState will report
>>>>>>>> + AfStateReset, unless AfTriggerStart is sent at the same time in which
>>>>>>>
>>>>>>> s/unless/until ?
>>>>>>> s/at the same time// ?
>>>>>
>>>>> I'm trying to describe what happens to the AfState when you send
>>>>> AfMode = AfModeAuto. It will go to AfStateIdle (even if you later send
>>>>> AfTriggerStart). The only occasion when you won't see AfStateIdle will
>>>>> be if you send AfMode = AfModeAuto and AfTriggerStart together. So I
>>>>> think this reads OK to me, but I'll see if I can make the wording any
>>>>> clearer.
>>>>
>>>> I see. I didn't get it, sorry. If it's fine with you, keep this
>>>> version!
>>>>
>>>>>
>>>>>>>
>>>>>>>> + case it will (start a scan and) report AfStateScanning.
>>>>>>>> +
>>>>>>>> + If the AfMode is set to AfModeContinuous then the AfState will initially
>>>>>>>> + report AfStateScanning.
>>>>>>>> +
>>>>>>>> + enum:
>>>>>>>> + - name: AfStateReset
>>>>>>>> + value: 0
>>>>>>>> + description: |
>>>>>>>> + The AF algorithm is in manual mode (AfModeManual) or in auto mode
>>>>>>>> + (AfModeAuto) and a scan has not yet been triggered, or an in-progress
>>>>>>>> + scan was cancelled.
>>>>>>>> + - name: AfStateScanning
>>>>>>>> + value: 1
>>>>>>>> + description: |
>>>>>>>> + The AF algorithm is in auto mode (AfModeAuto), and a scan has been
>>>>>>>> + started using the AfTrigger control. The scan can be cancelled by
>>>>>>>> + sending AfTriggerCancel at which point the algorithm will either
>>>>>>>> + move back to AfStateReset or, if the scan actually completes
>>>>>>>> + before the cancel request is processed, to one of
>>>>>>>> + AfStateFocused or AfStateFailed.
>>>>>>>> +
>>>>>>>> + Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
>>>>>>>
>>>>>>> s/be be/is ?
>>>>>
>>>>> Yes, thanks!
>>>>>
>>>>>>>
>>>>>>>> + at which point it may enter this state spontaneously whenever it
>>>>>>>> + determines that a rescan is needed.
>>>>>>>> + - name: AfStateFocused
>>>>>>>> + value: 2
>>>>>>>> + description: |
>>>>>>>> + The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
>>>>>>>> + mode and a scan has completed with the result that the algorithm believes
>>>>>>>> + the image is now in focus.
>>>>>>>> + - name: AfStateFailed
>>>>>>>> + value: 3
>>>>>>>> + description: |
>>>>>>>> + The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
>>>>>>>> + mode and a scan has completed with the result that the algorithm did not
>>>>>>>> + find a good focus position.
>>>>>>>> +
>>>>>>>> + - AfPauseState:
>>>>>>>> + type: int32_t
>>>>>>>> + description: |
>>>>>>>> + Only applicable in continuous (AfModeContinuous) mode, this reports whether
>>>>>>>> + the algorithm is currently running, paused or pausing (that is, will pause
>>>>>>>> + as soon as any in-progress scan completes).
>>>>>>>> +
>>>>>>>> + Any change to AfMode will cause AfPauseStateRunning to be reported.
>>>>>>>> +
>>>>>>>> + enum:
>>>>>>>> + - name: AfPauseStateRunning
>>>>>>>> + value: 0
>>>>>>>> + description: |
>>>>>>>> + Continuous AF is running and the algorithm may restart a scan
>>>>>>>> + spontaneously.
>>>>>>>> + - name: AfPauseStatePausing
>>>>>>>> + value: 1
>>>>>>>> + description: |
>>>>>>>> + Continuous AF has been sent an AfPauseDeferred control, and will pause
>>>>>>>> + as soon as any in-progress scan completes (and then report
>>>>>>>> + AfPauseStatePaused). No new scans will be start spontaneously until
>>>>>>>> + the AfPauseResume control is sent.
>>>>>>>> + - name: AfPauseStatePaused
>>>>>>>> + value: 2
>>>>>>>> + description: |
>>>>>>>> + Continuous AF is paused. No further state changes or lens movements
>>>>>>>> + will occur until the AfPauseResume control is sent.
>>>>>>>> +
>>>>>>>
>>>>>>> Very nice overall! With a few more clarifications I think this is good
>>>>>>> to go!
>>>>>>>
>>>>>>> Thanks
>>>>>>> j
>>>>>
>>>>> Thanks!
>>>>> David
>>>>>
>>>>>>>
>>>>>>>> # ----------------------------------------------------------------------------
>>>>>>>> # Draft controls section
>>>>>>>>
>>>>>>>> @@ -406,27 +664,6 @@ controls:
>>>>>>>> The camera will cancel any active or completed metering sequence.
>>>>>>>> The AE algorithm is reset to its initial state.
>>>>>>>>
>>>>>>>> - - AfTrigger:
>>>>>>>> - type: int32_t
>>>>>>>> - draft: true
>>>>>>>> - description: |
>>>>>>>> - Control for AF trigger. Currently identical to
>>>>>>>> - ANDROID_CONTROL_AF_TRIGGER.
>>>>>>>> -
>>>>>>>> - Whether the camera device will trigger autofocus for this request.
>>>>>>>> - enum:
>>>>>>>> - - name: AfTriggerIdle
>>>>>>>> - value: 0
>>>>>>>> - description: The trigger is idle.
>>>>>>>> - - name: AfTriggerStart
>>>>>>>> - value: 1
>>>>>>>> - description: The AF routine is started by the camera.
>>>>>>>> - - name: AfTriggerCancel
>>>>>>>> - value: 2
>>>>>>>> - description: |
>>>>>>>> - The camera will cancel any active trigger and the AF routine is
>>>>>>>> - reset to its initial state.
>>>>>>>> -
>>>>>>>> - NoiseReductionMode:
>>>>>>>> type: int32_t
>>>>>>>> draft: true
>>>>>>>> @@ -507,45 +744,6 @@ controls:
>>>>>>>> The AE algorithm has started a pre-capture metering session.
>>>>>>>> \sa AePrecaptureTrigger
>>>>>>>>
>>>>>>>> - - AfState:
>>>>>>>> - type: int32_t
>>>>>>>> - draft: true
>>>>>>>> - description: |
>>>>>>>> - Control to report the current AF algorithm state. Currently identical to
>>>>>>>> - ANDROID_CONTROL_AF_STATE.
>>>>>>>> -
>>>>>>>> - Current state of the AF algorithm.
>>>>>>>> - enum:
>>>>>>>> - - name: AfStateInactive
>>>>>>>> - value: 0
>>>>>>>> - description: The AF algorithm is inactive.
>>>>>>>> - - name: AfStatePassiveScan
>>>>>>>> - value: 1
>>>>>>>> - description: |
>>>>>>>> - AF is performing a passive scan of the scene in continuous
>>>>>>>> - auto-focus mode.
>>>>>>>> - - name: AfStatePassiveFocused
>>>>>>>> - value: 2
>>>>>>>> - description: |
>>>>>>>> - AF believes the scene is in focus, but might restart scanning.
>>>>>>>> - - name: AfStateActiveScan
>>>>>>>> - value: 3
>>>>>>>> - description: |
>>>>>>>> - AF is performing a scan triggered by an AF trigger request.
>>>>>>>> - \sa AfTrigger
>>>>>>>> - - name: AfStateFocusedLock
>>>>>>>> - value: 4
>>>>>>>> - description: |
>>>>>>>> - AF believes has focused correctly and has locked focus.
>>>>>>>> - - name: AfStateNotFocusedLock
>>>>>>>> - value: 5
>>>>>>>> - description: |
>>>>>>>> - AF has not been able to focus and has locked.
>>>>>>>> - - name: AfStatePassiveUnfocused
>>>>>>>> - value: 6
>>>>>>>> - description: |
>>>>>>>> - AF has completed a passive scan without finding focus.
>>>>>>>> -
>>>>>>>> - AwbState:
>>>>>>>> type: int32_t
>>>>>>>> draft: true
>>>>>>>> --
>>>>>>>> 2.30.2
>>>>>>>>
More information about the libcamera-devel
mailing list