[libcamera-devel] [PATCH 3/3] ipa: ipu3: af: AfMode and LensPosition control implementation
Kate Hsuan
hpa at redhat.com
Thu Sep 29 08:46:37 CEST 2022
On Wed, Sep 28, 2022 at 3:07 PM Jacopo Mondi <jacopo at jmondi.org> wrote:
>
> Hi Kate,
>
> On Fri, Sep 16, 2022 at 06:37:13PM +0800, Kate Hsuan via libcamera-devel wrote:
> > AfMode controls include manual, auto, and continuous modes. For auto
> > and continuous modes, both of the algorithms try to keep the image
> > focused. The difference between them is that continuous mode has higher
> > sensitivity to the change of image variance than auto mode. So, the AF
> > is frequently triggered in continuous mode. The user-defined lens
> > position can only be set in manual mode.
>
> We discussed in the past with David and Naush from RPi if contrast-based
> CAF without PDAF statistics from the sensor is a good idea or not. I
> cannot tell how it performs yet, and if we want it or not, but for
> sake of simplicity can we split this patch in two, one that adds
> manual mode support and another one that adds CAF support ?
Okay. No problem.
>
> The introduction of Manual mode alone would also help with aligning
> this design with the other algorithms that has been ported to use the
> FrameContext queue in:
> https://patchwork.libcamera.org/project/libcamera/list/?series=3506
> on which this series would be rebased.
>
> I would:
> 1) Make use of the frame context and active state for auto mode (if
> necessary)
> 2) Make use of frame context for manual mode like it's done in that
> series for AWB on RkISP1 to see if the concepts apply to AF as
> well. It won't be straight-forward as the active state and frame
> context are still evolvng and each algorithms might have different
> requirements, but to validate them we should try to port all
> algorithms to use them
You mean all the controls should use framecontext. Okay, I would base
on this series
to propose my v2 patch.
> 3) Eventually implement CAF
>
> A question remains: how to express lens movement ranges. This requires
> a larger discussion and I think you're free to do what's best for your
> setup for this first implementation, including using directly the range
> of the lens used in your setup.
This is a good question for deciding whether to use the percentage or
the range of the lens.
A normalized value such as a floating point based percentage or 0-1000
integer (basically, it is 1000 times of percentage) is good for the
application implementation.
So, here, I would like to use normalized values to pass the value
between the application and the algorithm.
>
> >
> > Signed-off-by: Kate Hsuan <hpa at redhat.com>
> > ---
> > src/ipa/ipu3/algorithms/af.cpp | 106 ++++++++++++++++++++++++++++-----
> > src/ipa/ipu3/algorithms/af.h | 15 +++++
> > 2 files changed, 105 insertions(+), 16 deletions(-)
> >
> > diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp
> > index 4835a034..c57d3e18 100644
> > --- a/src/ipa/ipu3/algorithms/af.cpp
> > +++ b/src/ipa/ipu3/algorithms/af.cpp
> > @@ -21,6 +21,8 @@
> >
> > #include <libcamera/base/log.h>
> >
> > +#include <libcamera/control_ids.h>
> > +
> > #include <libcamera/ipa/core_ipa_interface.h>
> >
> > #include "libipa/histogram.h"
> > @@ -109,7 +111,8 @@ static struct ipu3_uapi_af_filter_config afFilterConfigDefault = {
> > */
> > Af::Af()
> > : focus_(0), bestFocus_(0), currentVariance_(0.0), previousVariance_(0.0),
> > - coarseCompleted_(false), fineCompleted_(false)
> > + coarseCompleted_(false), fineCompleted_(false), maxChange_(kMaxChange),
> > + afMode_(controls::AfModeAuto)
> > {
> > }
> >
> > @@ -194,6 +197,69 @@ int Af::configure(IPAContext &context, const IPAConfigInfo &configInfo)
> > return 0;
> > }
> >
> > +/**
> > + * \brief AF controls handler
> > + *
> > + * Put the control parameter to the corresponding variables when receiving the controls
> > + * from the user.
>
> Documentation comes after the parameters documentation.
> Also you can use
Okay
>
> * \copydoc libcamera::ipa::Algorithm::queueRequest
>
> > + * \param[in] context The shared IPA context
> > + * \param[in] frame Frame number
> > + * \param[in] controls control list of the request
> > + */
> > +void Af::queueRequest([[maybe_unused]] IPAContext &context,
> > + [[maybe_unused]] const uint32_t frame,
> > + const ControlList &controls)
> > +{
> > + for (auto const &ctrl : controls) {
>
> You can
> const auto &afMode = controls.get(controls::AfMode;
> if (afMode && *afMode == controls::afModeAuto...
>
> if it helps you
>
> > + unsigned int ctrlEnum = ctrl.first;
> > + const ControlValue &ctrlValue = ctrl.second;
> > +
> > + switch (ctrlEnum) {
> > + case controls::AF_MODE:
> > + afModeSet(ctrlValue.get<int32_t>());
> > + break;
> > + case controls::LENS_POSITION:
> > + lensPosition_ = ctrlValue.get<float>();
>
> We need a bit more of logic here.
>
> If AfModeAuto then controls::lensPosition should be ignored.
>
> In AfModeAuto auto-focus scans are driven by the AfTrigger control.
> It is specified in control_ids.yaml as:
>
> - name: AfModeAuto
> value: 1
> description: |
> The AF algorithm is in auto mode. This means that the algorithm
> will never move the lens or change state unless the AfTrigger
> control is used. The AfTrigger control can be used to initiate a
> focus scan, the results of which will be reported by AfState.
>
> If the autofocus algorithm is moved from AfModeAuto to another
> mode while a scan is in progress, the scan is cancelled
> immediately, without waiting for the scan to finish.
>
> When first entering this mode the AfState will report
> AfStateIdle. When a trigger control is sent, AfState will
> report AfStateScanning for a period before spontaneously
> changing to AfStateFocused or AfStateFailed, depending on
> the outcome of the scan. It will remain in this state until
> another scan is initiated by the AfTrigger control. If a scan is
> cancelled (without changing to another mode), AfState will return
> to AfStateIdle.
>
> You also need to populate AfState which if I'm not mistaken is
> currently not reported.
>
> What do you think about implementing AfTrigger and AfState support on
> the existing Auto mode implementation (using the
> FrameContext/ActiveState) before adding Manual/CAF modes ?
Okay. I'll try implementing them in AutoMode first.
>
> > + }
> > + }
> > +}
> > +
> > +/**
> > + * \brief AF Mode set
> > + *
> > + * Set AF mode, including manual, auto, and continuous.
> > + *
> > + * \param[in] AF operation mode 0, 1, 2 are manual, auto, and continuous, respectively.
>
> Same comments, parameters documentation first
>
> > + */
> > +
> > +void Af::afModeSet(uint32_t mode)
> > +{
> > + switch (mode) {
> > + case controls::AfModeManual:
> > + case controls::AfModeAuto:
> > + afMode_ = mode;
> > + break;
> > + case controls::AfModeContinuous:
> > + afMode_ = mode;
> > + maxChange_ = 0.05;
> > + break;
> > + default:
> > + afMode_ = controls::AfModeAuto;
>
> The current afMode (auto or manual or CAF) should in example be part
> of the active state probably
Okay.
>
> > + }
> > +
> > + LOG(IPU3Af, Debug) << "AfMode set " << mode;
> > +}
> > +
> > +/**
> > + * \brief Set lens position
> > + *
> > + * Set user-defined lens position to the focus steps.
> > + */
> > +void Af::afLensPositionSet(IPAContext &context)
> > +{
> > + context.activeState.af.focus = kMaxFocusSteps * lensPosition_;
> > +}
> > +
> > /**
> > * \brief AF coarse scan
> > *
> > @@ -397,7 +463,7 @@ bool Af::afIsOutOfFocus(IPAContext context)
> > << " Current VCM step: "
> > << context.activeState.af.focus;
> >
> > - if (var_ratio > kMaxChange)
> > + if (var_ratio > maxChange_)
> > return true;
> > else
> > return false;
> > @@ -432,21 +498,29 @@ void Af::process(IPAContext &context, [[maybe_unused]] IPAFrameContext *frameCon
> > Span<const y_table_item_t> y_items(reinterpret_cast<const y_table_item_t *>(&stats->af_raw_buffer.y_table),
> > afRawBufferLen);
> >
> > - /*
> > - * Calculate the mean and the variance of AF statistics for a given grid.
> > - * For coarse: y1 are used.
> > - * For fine: y2 results are used.
> > - */
> > - currentVariance_ = afEstimateVariance(y_items, !coarseCompleted_);
> > + switch (afMode_) {
> > + case controls::AfModeManual:
> > + afLensPositionSet(context);
> > + break;
> > + case controls::AfModeContinuous:
> > + case controls::AfModeAuto:
> > + default:
> > + /*
> > + * Calculate the mean and the variance of AF statistics for a given grid.
> > + * For coarse: y1 are used.
> > + * For fine: y2 results are used.
> > + */
> > + currentVariance_ = afEstimateVariance(y_items, !coarseCompleted_);
> >
> > - if (!context.activeState.af.stable) {
> > - afCoarseScan(context);
> > - afFineScan(context);
> > - } else {
> > - if (afIsOutOfFocus(context))
> > - afReset(context);
> > - else
> > - afIgnoreFrameReset();
> > + if (!context.activeState.af.stable) {
> > + afCoarseScan(context);
> > + afFineScan(context);
> > + } else {
> > + if (afIsOutOfFocus(context))
> > + afReset(context);
> > + else
> > + afIgnoreFrameReset();
> > + }
> > }
> > }
> >
> > diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h
> > index ccf015f3..77bab2a7 100644
> > --- a/src/ipa/ipu3/algorithms/af.h
> > +++ b/src/ipa/ipu3/algorithms/af.h
> > @@ -26,6 +26,7 @@ class Af : public Algorithm
> > uint16_t y1_avg;
> > uint16_t y2_avg;
> > } y_table_item_t;
> > +
> > public:
> > Af();
> > ~Af() = default;
> > @@ -34,6 +35,9 @@ public:
> > int configure(IPAContext &context, const IPAConfigInfo &configInfo) override;
> > void process(IPAContext &context, IPAFrameContext *frameContext,
> > const ipu3_uapi_stats_3a *stats) override;
> > + void queueRequest([[maybe_unused]] IPAContext &context,
> > + [[maybe_unused]] const uint32_t frame,
> > + [[maybe_unused]] const ControlList &controls) override;
> >
> > private:
> > void afCoarseScan(IPAContext &context);
> > @@ -46,6 +50,11 @@ private:
> >
> > bool afIsOutOfFocus(IPAContext context);
> >
> > + void afModeSet(uint32_t mode);
> > + void afModeGet();
> > +
> > + void afLensPositionSet(IPAContext &context);
> > +
> > /* VCM step configuration. It is the current setting of the VCM step. */
> > uint32_t focus_;
> > /* The best VCM step. It is a local optimum VCM step during scanning. */
> > @@ -62,6 +71,12 @@ private:
> > bool coarseCompleted_;
> > /* If the fine scan completes, it is set to true. */
> > bool fineCompleted_;
> > + /* Max focus change ratio to determine */
> > + double maxChange_;
> > + /* Relative lens position in percentage. */
> > + double lensPosition_;
> > + /* Af operation mode. */
> > + uint32_t afMode_;
> > };
> >
> > } /* namespace ipa::ipu3::algorithms */
> > --
> > 2.37.3
> >
>
--
BR,
Kate
More information about the libcamera-devel
mailing list