[libcamera-devel] [RFC v2 1/1] ipa: ipu3: af: Auto focus for dw9719 Surface Go2 VCM
Kate Hsuan
hpa at redhat.com
Fri Nov 26 08:23:33 CET 2021
Hi Jean-Michel,
On Wed, Nov 24, 2021 at 9:40 PM Jean-Michel Hautbois
<jeanmichel.hautbois at ideasonboard.com> wrote:
>
> Hi Kate,
>
> Thanks for the patch.
>
> On 24/11/2021 11:34, Kate Hsuan wrote:
> > Since VCM for surface Go 2 (dw9719) had been successfully
> > driven, this Af module can be used to control the VCM and
> > determine the focus value based on the IPU3 AF state.
> >
> > The variance of each focus step is determined and a greedy
> > approah is used to find the maximum variance of the AF
> > state and a appropriate focus value.
> >
> > Changes since v2:
> > 1. Add grid configuration interface.
> > 2. Add AF accelerator configuration.
> > 3. Move default focus area to the center of the image.
> >
> > Signed-off-by: Kate Hsuan <hpa at redhat.com>
> > ---
> > src/ipa/ipu3/algorithms/af.cpp | 284 ++++++++++++++++++++++++++++
> > src/ipa/ipu3/algorithms/af.h | 54 ++++++
> > src/ipa/ipu3/algorithms/agc.cpp | 2 +-
> > src/ipa/ipu3/algorithms/meson.build | 3 +-
> > src/ipa/ipu3/ipa_context.cpp | 26 +++
> > src/ipa/ipu3/ipa_context.h | 11 ++
> > src/ipa/ipu3/ipu3.cpp | 2 +
> > 7 files changed, 380 insertions(+), 2 deletions(-)
> > create mode 100644 src/ipa/ipu3/algorithms/af.cpp
> > create mode 100644 src/ipa/ipu3/algorithms/af.h
> >
> > diff --git a/src/ipa/ipu3/algorithms/af.cpp b/src/ipa/ipu3/algorithms/af.cpp
> > new file mode 100644
> > index 00000000..b0359721
> > --- /dev/null
> > +++ b/src/ipa/ipu3/algorithms/af.cpp
> > @@ -0,0 +1,284 @@
> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> > +/*
> > + * Copyright (C) 2021, Red Hat
> > + *
> > + * af.cpp - IPU3 auto focus control
> > + */
> > +
> > +#include "af.h"
> > +
> > +#include <algorithm>
> > +#include <chrono>
> > +#include <cmath>
> > +#include <fcntl.h>
> > +#include <numeric>
> > +#include <sys/ioctl.h>
> > +#include <sys/stat.h>
> > +#include <sys/types.h>
> > +#include <unistd.h>
> > +
> > +#include <linux/videodev2.h>
> > +
> > +#include <libcamera/base/log.h>
> > +
> > +#include <libcamera/ipa/core_ipa_interface.h>
> > +
> > +#include "libipa/histogram.h"
> > +
> > +/**
> > + * \file af.h
> > + */
> > +
> > +namespace libcamera {
> > +
> > +using namespace std::literals::chrono_literals;
> > +
> > +namespace ipa::ipu3::algorithms {
> > +
> > +/**
> > + * \class Af
> > + * \brief A IPU3 auto-focus accelerator based auto focus algorthim
> > + *
> > + * This algorithm is used to determine the position of the lens and get a
> > + * focused image. The IPU3 AF accelerator computes the statistics, composed
> > + * by high pass and low pass filtered value and stores in a AF buffer.
> > + * Typically, for a focused image, it has relative high contrast than a
> > + * blurred image, i.e. an out of focus image. Therefore, if an image with the
> > + * highest contrast can be found from the AF scan, the lens' position is the
> > + * best step of the focus.
> > + *
> > + */
> > +
> > +LOG_DEFINE_CATEGORY(IPU3Af)
> > +
> > +/**
> > + * Maximum focus value of the VCM control
> > + * \todo should be obtained from the VCM driver
> > + */
> > +static constexpr uint32_t MaxFocusSteps_ = 1023;
> > +
> > +/* Minimum focus step for searching appropriate focus*/
> > +static constexpr uint32_t MinSearchStep_ = 5;
> > +
> > +/* max ratio of variance change, 0.0 < MaxChange_ < 1.0*/
> > +static constexpr double MaxChange_ = 0.8;
> > +
> > +Af::Af()
> > + : focus_(0), currentVariance_(0.0)
> > +{
> > + /**
> > + * For surface Go 2 back camera VCM (dw9719)
> > + * \todo move to control class
> > + */
> > + vcmFd_ = open("/dev/v4l-subdev8", O_RDWR);
>
> This shouldn't be here :-).
>
> > +}
> > +
> > +Af::~Af()
> > +{
> > + if (vcmFd_ != -1)
> > + close(vcmFd_);
> > +}
> > +
> > +void Af::prepare(IPAContext &context, ipu3_uapi_params *params)
> > +{
> > + /* AF grid config */
> > + params->acc_param.af.grid_cfg.width = 16;
> > + params->acc_param.af.grid_cfg.height = 16;
> > + params->acc_param.af.grid_cfg.block_height_log2 = 3;
> > + params->acc_param.af.grid_cfg.block_width_log2 = 3;
> > + params->acc_param.af.grid_cfg.height_per_slice = 2;
> > + /* Start position of AF area */
> > + params->acc_param.af.grid_cfg.x_start = context.configuration.af.start_x;
> > + params->acc_param.af.grid_cfg.y_start = context.configuration.af.start_y | IPU3_UAPI_GRID_Y_START_EN;
>
> That is interesting... and probably not right :-).
> You are setting the ImgU to use the same default grid it is using if you
> are not setting params->use.acc_af.
>
> Meaning, you could simplify this by using the default table here:
> https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-tables.c#L9579
>
> You can use what is done in IPU3::Awb for instance, we are using the
> default BNR table from
> https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-tables.c#L9313
>
> And then you can only change the desired start_x and start_y (I have
> comments on those later).
OK, Thanks for the information, I'll try this BNR table.
>
> > +
> > + params->acc_param.af.filter_config.y1_sign_vec = 0;
> > + params->acc_param.af.filter_config.y2_sign_vec = 0;
> > +
> > + /* b + gb + gr + r = 32 */
> > + params->acc_param.af.filter_config.y_calc.y_gen_rate_b = 8;
> > + params->acc_param.af.filter_config.y_calc.y_gen_rate_gb = 8;
> > + params->acc_param.af.filter_config.y_calc.y_gen_rate_gr = 8;
> > + params->acc_param.af.filter_config.y_calc.y_gen_rate_r = 8;
> > +
> > + /* 2^7 = 128, a1 + a2 + ... + a12 = 128, log2 of sum of a1 to a12*/
> > + params->acc_param.af.filter_config.nf.y1_nf = 7;
> > + params->acc_param.af.filter_config.nf.y2_nf = 7;
> > +
> > + /* Low pass filter configuration (y1_coeff_n) */
> > + params->acc_param.af.filter_config.y1_coeff_0.a1 = 0;
> > + params->acc_param.af.filter_config.y1_coeff_0.a2 = 0;
> > + params->acc_param.af.filter_config.y1_coeff_0.a3 = 0;
> > + params->acc_param.af.filter_config.y1_coeff_0.a4 = 0;
> > +
> > + params->acc_param.af.filter_config.y1_coeff_1.a5 = 0;
> > + params->acc_param.af.filter_config.y1_coeff_1.a6 = 0;
> > + params->acc_param.af.filter_config.y1_coeff_1.a7 = 0;
> > + params->acc_param.af.filter_config.y1_coeff_1.a8 = 0;
> > +
> > + params->acc_param.af.filter_config.y1_coeff_2.a9 = 0;
> > + params->acc_param.af.filter_config.y1_coeff_2.a10 = 0;
> > + params->acc_param.af.filter_config.y1_coeff_2.a11 = 0;
> > + params->acc_param.af.filter_config.y1_coeff_2.a12 = 128;
> > +
> > + /* High pass filter configuration (y2_coeff_n) */
> > + params->acc_param.af.filter_config.y2_coeff_0.a1 = 0;
> > + params->acc_param.af.filter_config.y2_coeff_0.a2 = 0;
> > + params->acc_param.af.filter_config.y2_coeff_0.a3 = 0;
> > + params->acc_param.af.filter_config.y2_coeff_0.a4 = 0;
> > +
> > + params->acc_param.af.filter_config.y2_coeff_1.a5 = 0;
> > + params->acc_param.af.filter_config.y2_coeff_1.a6 = 0;
> > + params->acc_param.af.filter_config.y2_coeff_1.a7 = 0;
> > + params->acc_param.af.filter_config.y2_coeff_1.a8 = 0;
> > +
> > + params->acc_param.af.filter_config.y2_coeff_2.a9 = 0;
> > + params->acc_param.af.filter_config.y2_coeff_2.a10 = 0;
> > + params->acc_param.af.filter_config.y2_coeff_2.a11 = 0;
> > + params->acc_param.af.filter_config.y2_coeff_2.a12 = 128;
> > +
> > + /* Enable AF accelerator */
> > + params->use.acc_af = 1;
> > +}
> > +
> > +/**
> > + * \brief Configure the Af given a configInfo
> > + * \param[in] context The shared IPA context
> > + * \param[in] configInfo The IPA configuration data
> > + *
> > + * \return 0
> > + */
> > +int Af::configure(IPAContext &context, [[maybe_unused]] const IPAConfigInfo &configInfo)
> > +{
> > + /* Determined focus value i.e. current focus value */
> > + context.frameContext.af.focus = 0;
> > + /* Maximum variance of the AF statistics */
> > + context.frameContext.af.maxVariance = 0;
> > + /* is focused? if it is true, the AF should be in a stable state. */
> > + context.frameContext.af.stable = false;
> > + /* Frame to be ignored before start to estimate AF variance. */
> > + ignoreFrame_ = 10;
> > +
> > + /*
> > + * AF default area configuration
> > + * Move AF area to the center of the image.
> > + */
> > + /* AF width is 16x8 = 128 */
> > + context.configuration.af.start_x = (1280 / 2) - 64;
> > + context.configuration.af.start_y = (720 / 2) - 64;
>
> This is good, only if the frame output is set to 1280x720.
> If I call:
> 'cam -c2 -C -s width=1920,height=1280'
>
> Then you won't be centered.
> You can get the configuration directly from configInfo.bdsOutputSize (I
> think af is using the BDS output) and then remove the [[maybe_unused]].
Thank you for this, I'll use the bdsOutputSize to determine the centre
of the image.
>
> > +
> > + return 0;
> > +}
> > +
> > +/**
> > + * \brief Send focus step to the VCM.
> > + * \param[in] value Set lens position.
> > + * \todo It is hardcoded here for the dw9717 VCM and will be moved to the
> > + * subdev control in the future.
> > + */
> > +int Af::vcmSet(int value)
> > +{
> > + int ret;
> > + struct v4l2_control ctrl;
> > + if (vcmFd_ == -1)
> > + return -EINVAL;
> > + memset(&ctrl, 0, sizeof(struct v4l2_control));
> > + ctrl.id = V4L2_CID_FOCUS_ABSOLUTE;
> > + ctrl.value = value;
> > + ret = ioctl(vcmFd_, VIDIOC_S_CTRL, &ctrl);
> > + return ret;
> > +}
>
> This does not belong to the algorithm.
Sure :)
>
> > +
> > +/**
> > + * \brief Determine the max contrast image and lens position. y_table is the
> > + * statictic data from IPU3 and is composed of low pass and high pass filtered
> > + * value. High pass filtered value also represents the sharpness of the image.
> > + * Based on this, if the image with highest variance of the high pass filtered
> > + * value (contrast) during the AF scan, the position of the len should be the
> > + * best focus.
> > + * \param[in] context The shared IPA context.
> > + * \param[in] stats The statistic buffer of 3A from the IPU3.
> > + */
> > +void Af::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)
> > +{
> > + uint32_t total = 0;
> > + double mean;
> > + uint64_t var_sum = 0;
> > + y_table_item_t *y_item;
> > + int z = 0;
> > +
> > + y_item = (y_table_item_t *)stats->af_raw_buffer.y_table;
> > +
> > + /**
> > + * Calculate the mean of each non-zero AF statistics, since IPU3 only determine the AF value
> > + * for a given grid.
> > + */
> > + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4; z++) {
> > + printf("%d, ", y_item[z].y2_avg);
> > + total = total + y_item[z].y2_avg;
> > + if (y_item[z].y2_avg == 0)
> > + break;
> > + }
> > + mean = total / z;
> > +
> > + /* Calculate the variance of every AF statistic value. */
> > + for (z = 0; z < (IPU3_UAPI_AF_Y_TABLE_MAX_SIZE) / 4 && y_item[z].y2_avg != 0; z++) {
> > + var_sum = var_sum + ((y_item[z].y2_avg - mean) * (y_item[z].y2_avg - mean));
> > + if (y_item[z].y2_avg == 0)
> > + break;
> > + }
> > +
> > + /* Determine the average variance of the frame. */
> > + currentVariance_ = static_cast<double>(var_sum) / static_cast<double>(z);
> > + LOG(IPU3Af, Debug) << "variance: " << currentVariance_;
> > +
> > + if (context.frameContext.af.stable == true) {
> > + const uint32_t diff_var = std::abs(currentVariance_ - context.frameContext.af.maxVariance);
> > + const double var_ratio = diff_var / context.frameContext.af.maxVariance;
> > + LOG(IPU3Af, Debug) << "Change ratio: "
> > + << var_ratio
> > + << " current focus: "
> > + << context.frameContext.af.focus;
> > + /**
> > + * If the change ratio of contrast is over Maxchange_ (out of focus),
> > + * trigger AF again.
> > + */
> > + if (var_ratio > MaxChange_) {
> > + if (ignoreFrame_ == 0) {
> > + context.frameContext.af.maxVariance = 0;
> > + context.frameContext.af.focus = 0;
> > + focus_ = 0;
> > + context.frameContext.af.stable = false;
> > + ignoreFrame_ = 60;
> > + } else
> > + ignoreFrame_--;
> > + } else
> > + ignoreFrame_ = 10;
> > + } else {
> > + if (ignoreFrame_ != 0)
> > + ignoreFrame_--;
> > + else {
> > + /* Find the maximum variance during the AF scan using a greedy strategy */
> > + if (currentVariance_ > context.frameContext.af.maxVariance) {
> > + context.frameContext.af.maxVariance = currentVariance_;
> > + context.frameContext.af.focus = focus_;
> > + }
> > +
> > + if (focus_ > MaxFocusSteps_) {
> > + /* If reach the max step, move lens to the position and set "focus stable". */
> > + context.frameContext.af.stable = true;
> > + vcmSet(context.frameContext.af.focus);
> > + } else {
> > + focus_ += MinSearchStep_;
> > + vcmSet(focus_);
> > + }
> > + LOG(IPU3Af, Debug) << "Focus searching max variance is: "
> > + << context.frameContext.af.maxVariance
> > + << " Focus step is "
> > + << context.frameContext.af.focus;
> > + }
> > + }
> > +}
>
> I won't comment the algorithm yet, as you already have some changes to
> make according to Kieran's comments ;-).
>
> > +
> > +} /* namespace ipa::ipu3::algorithms */
> > +
> > +} /* namespace libcamera */
> > diff --git a/src/ipa/ipu3/algorithms/af.h b/src/ipa/ipu3/algorithms/af.h
> > new file mode 100644
> > index 00000000..b5c11874
> > --- /dev/null
> > +++ b/src/ipa/ipu3/algorithms/af.h
> > @@ -0,0 +1,54 @@
> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> > +/*
> > + * Copyright (C) 2021, Red Hat
> > + *
> > + * af.h - IPU3 Af control
> > + */
> > +#ifndef __LIBCAMERA_IPU3_ALGORITHMS_AF_H__
> > +#define __LIBCAMERA_IPU3_ALGORITHMS_AF_H__
> > +
> > +#include <linux/intel-ipu3.h>
> > +
> > +#include <libcamera/base/utils.h>
> > +
> > +#include <libcamera/geometry.h>
> > +
> > +#include "algorithm.h"
> > +
> > +namespace libcamera {
> > +
> > +namespace ipa::ipu3::algorithms {
> > +
> > +class Af : public Algorithm
> > +{
> > + /* The format of y_table. From ipu3-ipa repo */
> > + typedef struct y_table_item {
> > + uint16_t y1_avg;
> > + uint16_t y2_avg;
> > + } y_table_item_t;
> > +
> > +public:
> > + Af();
> > + ~Af();
> > +
> > + void prepare(IPAContext &context, ipu3_uapi_params *params) override;
> > + int configure(IPAContext &context, const IPAConfigInfo &configInfo) override;
> > + void process(IPAContext &context, const ipu3_uapi_stats_3a *stats) override;
> > +
> > +private:
> > + int vcmSet(int value);
> > +
> > + int vcmFd_;
> > + /* Used for focus scan. */
> > + uint32_t focus_;
> > + /* Recent AF statistic variance. */
> > + double currentVariance_;
> > + /* The frames to be ignore before starting measuring. */
> > + uint32_t ignoreFrame_;
> > +};
> > +
> > +} /* namespace ipa::ipu3::algorithms */
> > +
> > +} /* namespace libcamera */
> > +
> > +#endif /* __LIBCAMERA_IPU3_ALGORITHMS_AF_H__ */
> > diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp
> > index b5d736c1..70ae4e59 100644
> > --- a/src/ipa/ipu3/algorithms/agc.cpp
> > +++ b/src/ipa/ipu3/algorithms/agc.cpp
> > @@ -138,7 +138,7 @@ void Agc::measureBrightness(const ipu3_uapi_stats_3a *stats,
> > }
> >
> > /* Estimate the quantile mean of the top 2% of the histogram */
> > - iqMean_ = Histogram(Span<uint32_t>(hist)).interQuantileMean(0.98, 1.0);
> > + iqMean_ = Histogram(Span<uint32_t>(hist)).interQuantileMean(0.7, 1.0);
>
> Why are you changing this ? It is the top 30% of the cumulative histogram ?
>
Sorry about this. It's my fault. AE connot work properly on my surface
Go 2 back camera. I try to make it more stable for the exposure. So I
expend the mean from the top 2% to 30%. It works for the front camera
before but doesn't work for the back camera.
I'll remove this experimental code in the v3.
> > }
> >
> > /**
> > diff --git a/src/ipa/ipu3/algorithms/meson.build b/src/ipa/ipu3/algorithms/meson.build
> > index 3ec42f72..8574416e 100644
> > --- a/src/ipa/ipu3/algorithms/meson.build
> > +++ b/src/ipa/ipu3/algorithms/meson.build
> > @@ -1,9 +1,10 @@
> > # SPDX-License-Identifier: CC0-1.0
> >
> > ipu3_ipa_algorithms = files([
> > + 'af.cpp',
> > 'agc.cpp',
> > 'algorithm.cpp',
> > 'awb.cpp',
> > 'blc.cpp',
> > - 'tone_mapping.cpp',
> > + 'tone_mapping.cpp'
> > ])
> > diff --git a/src/ipa/ipu3/ipa_context.cpp b/src/ipa/ipu3/ipa_context.cpp
> > index 2355a9c7..1064d62d 100644
> > --- a/src/ipa/ipu3/ipa_context.cpp
> > +++ b/src/ipa/ipu3/ipa_context.cpp
> > @@ -69,6 +69,17 @@ namespace libcamera::ipa::ipu3 {
> > * \brief Number of cells on one line including the ImgU padding
> > */
> >
> > +/**
> > + * \var IPASessionConfiguration::af
> > + * \brief AF parameters configuration of the IPA
> > + *
> > + * \var IPASessionConfiguration::af.start_x
> > + * \brief The start X position of the AF area
> > + *
> > + * \var IPASessionConfiguration::af.start_y
> > + * \brief The start Y position of the AF area
> > + */
> > +
> > /**
> > * \var IPASessionConfiguration::agc
> > * \brief AGC parameters configuration of the IPA
> > @@ -86,6 +97,21 @@ namespace libcamera::ipa::ipu3 {
> > * \brief Maximum analogue gain supported with the configured sensor
> > */
> >
> > +/**
> > + * \var IPAFrameContext::af
> > + * \brief Context for the Automatic Focus algorithm
> > + *
> > + * \struct IPAFrameContext::af
> > + * \var IPAFrameContext::af.focus
> > + * \brief Current position of the lens
> > + *
> > + * \var IPAFrameContext::af.maxVariance
> > + * \brief The maximum variance of the current image.
> > + *
> > + * \var IPAFrameContext::af.stable
> > + * \brief is the image focused?
> > + */
> > +
> > /**
> > * \var IPAFrameContext::agc
> > * \brief Context for the Automatic Gain Control algorithm
> > diff --git a/src/ipa/ipu3/ipa_context.h b/src/ipa/ipu3/ipa_context.h
> > index 1e46c61a..a7ffcbed 100644
> > --- a/src/ipa/ipu3/ipa_context.h
> > +++ b/src/ipa/ipu3/ipa_context.h
> > @@ -31,6 +31,11 @@ struct IPASessionConfiguration {
> > double minAnalogueGain;
> > double maxAnalogueGain;
> > } agc;
> > +
> > + struct {
> > + uint32_t start_x;
> > + uint32_t start_y;
> > + } af;
>
> Those are __u16 in the kernel header file, maybe can we keep the same type ?
Sure, I'll modify it in my v3.
>
> > };
> >
> > struct IPAFrameContext {
> > @@ -47,6 +52,12 @@ struct IPAFrameContext {
> > } gains;
> > } awb;
> >
> > + struct {
> > + uint32_t focus;
> > + double maxVariance;
> > + bool stable;
> > + } af;
> > +
> > struct {
> > double gamma;
> > struct ipu3_uapi_gamma_corr_lut gammaCorrection;
> > diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp
> > index 5c51607d..f19d0059 100644
> > --- a/src/ipa/ipu3/ipu3.cpp
> > +++ b/src/ipa/ipu3/ipu3.cpp
> > @@ -30,6 +30,7 @@
> >
> > #include "libcamera/internal/mapped_framebuffer.h"
> >
> > +#include "algorithms/af.h"
> > #include "algorithms/agc.h"
> > #include "algorithms/algorithm.h"
> > #include "algorithms/awb.h"
> > @@ -298,6 +299,7 @@ int IPAIPU3::init(const IPASettings &settings,
> > }
> >
> > /* Construct our Algorithms */
> > + algorithms_.push_back(std::make_unique<algorithms::Af>());
> > algorithms_.push_back(std::make_unique<algorithms::Agc>());
> > algorithms_.push_back(std::make_unique<algorithms::Awb>());
> > algorithms_.push_back(std::make_unique<algorithms::BlackLevelCorrection>());
> >
>
Thank you :)
--
BR,
Kate
More information about the libcamera-devel
mailing list