[libcamera-devel] [PATCH v6 06/10] ipa: rkisp1: Add AF algorithm based on AfHillClimbing

Daniel Semkowicz dse at thaumatec.com
Tue Apr 4 08:09:06 CEST 2023


Hi Jacopo,

On Mon, Apr 3, 2023 at 11:25 AM Jacopo Mondi
<jacopo.mondi at ideasonboard.com> wrote:
>
> Hi Daniel
>
> On Fri, Mar 31, 2023 at 10:19:26AM +0200, Daniel Semkowicz via libcamera-devel wrote:
> > Add contrast based auto focus implementation for rkisp1 platform using
> > the common AfHillClimbing algorithm.
> >
> > Rockchip ISP AF block allows calculation of sharpness and luminance
> > in up to three user defined windows. If no windows are set, there are
> > some default settings applied for the first window and exposed through
> > the driver. For each frame, use the sharpness value calculated for this
> > default window and feed the hill climbing algorithm with them. Then set
> > the lens position to value calculated by the algorithm.
> >
> > Signed-off-by: Daniel Semkowicz <dse at thaumatec.com>
> > ---
> >  src/ipa/rkisp1/algorithms/af.cpp      | 121 ++++++++++++++++++++++++++
> >  src/ipa/rkisp1/algorithms/af.h        |  43 +++++++++
> >  src/ipa/rkisp1/algorithms/meson.build |   1 +
> >  src/ipa/rkisp1/ipa_context.cpp        |  13 +++
> >  src/ipa/rkisp1/ipa_context.h          |   5 ++
> >  5 files changed, 183 insertions(+)
> >  create mode 100644 src/ipa/rkisp1/algorithms/af.cpp
> >  create mode 100644 src/ipa/rkisp1/algorithms/af.h
> >
> > diff --git a/src/ipa/rkisp1/algorithms/af.cpp b/src/ipa/rkisp1/algorithms/af.cpp
> > new file mode 100644
> > index 00000000..fde924d4
> > --- /dev/null
> > +++ b/src/ipa/rkisp1/algorithms/af.cpp
> > @@ -0,0 +1,121 @@
> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> > +/*
> > + * Copyright (C) 2023, Theobroma Systems
> > + *
> > + * af.cpp - RkISP1 AF hill climbing based control algorithm
> > + */
> > +
> > +#include "af.h"
> > +
> > +/**
> > + * \file af.h
> > + */
> > +
> > +namespace libcamera {
> > +
> > +namespace ipa::rkisp1::algorithms {
> > +
> > +/**
> > + * \class Af
> > + * \brief AF contrast based hill climbing control algorithm for RkISP platforms
> > + *
> > + * Auto focus algorithm for RkISP platforms, based on the common hill climbing
> > + * auto focus implementation (libcamera::ipa::algorithms::AfHillClimbing).
> > + *
> > + * This is the platform layer of the algorithm.
> > + *
> > + * Tuning file parameters:
> > + * - **wait-frames-lens:** Number of frames that should be skipped when lens
> > + *   position is changed. Lens movement takes some time and statistics measured
> > + *   during the lens movement are unstable. Currently there is no way to know
> > + *   when lens movement finished and this is a workaround for this. Wait a fixed
> > + *   amount of time on each movement. This parameter should be set according
> > + *   to the worst case  - the number of frames it takes to move lens between
> > + *   limit positions.
> > + * .
> > + * \sa libcamera::ipa::algorithms::AfHillClimbing for additional tuning
> > + * parameters.
> > + *
> > + * \todo Model the lens delay as number of frames required for the lens position
> > + * to stabilize in the CameraLens class.
> > + */
> > +
> > +LOG_DEFINE_CATEGORY(RkISP1Af)
> > +
> > +/**
> > + * \copydoc libcamera::ipa::Algorithm::init
> > + */
> > +int Af::init(IPAContext &context, const YamlObject &tuningData)
> > +{
> > +     const auto &lensConfig = context.configuration.lens;
> > +     if (!lensConfig) {
> > +             LOG(RkISP1Af, Error) << "Lens not found";
> > +             return 1;
>
> As we discussed in v5, I think it's sane to fail at init() time
> if the user has instantiated the Af algorithm in config file, but the
> platform does not expose a movable lens.
>
> Maybe a return -EINVAL instead of just 1, but that's a minor I can fix
> when applying.

Ah, yes the -EINVAL is a better choice than "1" :)

>
> Looking at the diff with v5, my tag still stands
> Reviewed-by: Jacopo Mondi <jacopo.mondi at ideasonboard.com>
>
> Thanks
>   j
>
> > +     }
> > +
> > +     waitFramesLens_ = tuningData["wait-frames-lens"].get<uint32_t>(1);
> > +
> > +     LOG(RkISP1Af, Debug) << "waitFramesLens_: " << waitFramesLens_;
> > +
> > +     return af.init(lensConfig->minFocusPosition,
> > +                    lensConfig->maxFocusPosition, tuningData);
> > +}
> > +
> > +/**
> > + * \copydoc libcamera::ipa::Algorithm::configure
> > + */
> > +int Af::configure(IPAContext &context,
> > +               const IPACameraSensorInfo &configInfo)
> > +{
> > +     /*
> > +      * Lens position is unknown at the startup, so initialize
> > +      * the current position to something out of range.
> > +      */
> > +     context.activeState.af.lensPosition =
> > +             context.configuration.lens->maxFocusPosition + 1;
> > +
> > +     af.configure(configInfo.outputSize);
> > +     return 0;
> > +}
> > +
> > +/**
> > + * \copydoc libcamera::ipa::Algorithm::queueRequest
> > + */
> > +void Af::queueRequest([[maybe_unused]] IPAContext &context,
> > +                   [[maybe_unused]] const uint32_t frame,
> > +                   [[maybe_unused]] IPAFrameContext &frameContext,
> > +                   const ControlList &controls)
> > +{
> > +     af.queueRequest(controls);
> > +}
> > +
> > +/**
> > + * \copydoc libcamera::ipa::Algorithm::process
> > + */
> > +void Af::process(IPAContext &context, [[maybe_unused]] const uint32_t frame,
> > +              [[maybe_unused]] IPAFrameContext &frameContext,
> > +              const rkisp1_stat_buffer *stats,
> > +              [[maybe_unused]] ControlList &metadata)
> > +{
> > +     uint32_t sharpness = stats->params.af.window[0].sum;
> > +     uint32_t luminance = stats->params.af.window[0].lum;
> > +
> > +     LOG(RkISP1Af, Debug)
> > +             << "lensPosition: " << context.activeState.af.lensPosition
> > +             << ", Sharpness: " << sharpness
> > +             << ", Luminance: " << luminance;
> > +
> > +     int32_t newLensPosition = af.process(sharpness);
> > +
> > +     if (newLensPosition != context.activeState.af.lensPosition) {
> > +             context.activeState.af.lensPosition = newLensPosition;
> > +             context.activeState.af.applyLensCtrls = true;
> > +             af.skipFrames(waitFramesLens_);
> > +     }
> > +}
> > +
> > +REGISTER_IPA_ALGORITHM(Af, "Af")
> > +
> > +} /* namespace ipa::rkisp1::algorithms */
> > +
> > +} /* namespace libcamera */
> > diff --git a/src/ipa/rkisp1/algorithms/af.h b/src/ipa/rkisp1/algorithms/af.h
> > new file mode 100644
> > index 00000000..3ba66d38
> > --- /dev/null
> > +++ b/src/ipa/rkisp1/algorithms/af.h
> > @@ -0,0 +1,43 @@
> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> > +/*
> > + * Copyright (C) 2023, Theobroma Systems
> > + *
> > + * af.h - RkISP1 AF hill climbing based control algorithm
> > + */
> > +
> > +#pragma once
> > +
> > +#include <linux/rkisp1-config.h>
> > +
> > +#include "libipa/algorithms/af_hill_climbing.h"
> > +
> > +#include "algorithm.h"
> > +
> > +namespace libcamera {
> > +
> > +namespace ipa::rkisp1::algorithms {
> > +
> > +class Af : public Algorithm
> > +{
> > +public:
> > +     int init(IPAContext &context, const YamlObject &tuningData) override;
> > +     int configure(IPAContext &context,
> > +                   const IPACameraSensorInfo &configInfo) override;
> > +     void queueRequest(IPAContext &context, uint32_t frame,
> > +                       IPAFrameContext &frameContext,
> > +                       const ControlList &controls) override;
> > +     void process(IPAContext &context, uint32_t frame,
> > +                  IPAFrameContext &frameContext,
> > +                  const rkisp1_stat_buffer *stats,
> > +                  ControlList &metadata) override;
> > +
> > +private:
> > +     ipa::algorithms::AfHillClimbing af;
> > +
> > +     /* Wait number of frames after changing lens position */
> > +     uint32_t waitFramesLens_ = 0;
> > +};
> > +
> > +} /* namespace ipa::rkisp1::algorithms */
> > +
> > +} /* namespace libcamera */
> > diff --git a/src/ipa/rkisp1/algorithms/meson.build b/src/ipa/rkisp1/algorithms/meson.build
> > index 93a48329..ab7e44f3 100644
> > --- a/src/ipa/rkisp1/algorithms/meson.build
> > +++ b/src/ipa/rkisp1/algorithms/meson.build
> > @@ -1,6 +1,7 @@
> >  # SPDX-License-Identifier: CC0-1.0
> >
> >  rkisp1_ipa_algorithms = files([
> > +    'af.cpp',
> >      'agc.cpp',
> >      'awb.cpp',
> >      'blc.cpp',
> > diff --git a/src/ipa/rkisp1/ipa_context.cpp b/src/ipa/rkisp1/ipa_context.cpp
> > index aea99299..d80a7b1b 100644
> > --- a/src/ipa/rkisp1/ipa_context.cpp
> > +++ b/src/ipa/rkisp1/ipa_context.cpp
> > @@ -139,6 +139,19 @@ namespace libcamera::ipa::rkisp1 {
> >   * member may be read by any algorithm, but shall only be written by its owner.
> >   */
> >
> > +/**
> > + * \var IPAActiveState::af
> > + * \brief State for the Automatic Focus Control algorithm
> > + *
> > + * \var IPAActiveState::af.lensPosition
> > + * \brief Lens position calculated by the AF algorithm
> > + *
> > + * \var IPAActiveState::af.applyLensCtrls
> > + * \brief Whether the lens position should be applied
> > + *
> > + * If true, IPA should send new controls to the PH to set new lens position.
> > + */
> > +
> >  /**
> >   * \var IPAActiveState::agc
> >   * \brief State for the Automatic Gain Control algorithm
> > diff --git a/src/ipa/rkisp1/ipa_context.h b/src/ipa/rkisp1/ipa_context.h
> > index 65b3fbfe..3dcc5aa0 100644
> > --- a/src/ipa/rkisp1/ipa_context.h
> > +++ b/src/ipa/rkisp1/ipa_context.h
> > @@ -62,6 +62,11 @@ struct IPASessionConfiguration {
> >  };
> >
> >  struct IPAActiveState {
> > +     struct {
> > +             int32_t lensPosition;
> > +             bool applyLensCtrls;
> > +     } af;
> > +
> >       struct {
> >               struct {
> >                       uint32_t exposure;
> > --
> > 2.39.2
> >

Best regards
Daniel


More information about the libcamera-devel mailing list