[libcamera-devel] [RFC PATCH v2 2/4] ipa: raspberrypi: Introduce an autofocus algorithm
David Plowman
david.plowman at raspberrypi.com
Fri Mar 25 12:15:21 CET 2022
Hi Jean-Michel
Thanks for this patch, great to see progress on the AF question!
On Wed, 23 Mar 2022 at 16:01, Jean-Michel Hautbois via libcamera-devel
<libcamera-devel at lists.libcamera.org> wrote:
>
> Now that the ancillary links are plumbed and we can set the lens
> position, implement a contrast-based algorithm for RPi. This algorithm
> is adapted from the one proposed for the IPU3 IPA.
>
> It is currently taking all the regions and tries to make the focus on
> the global scene in a first attempt.
>
> Signed-off-by: Jean-Michel Hautbois <jeanmichel.hautbois at ideasonboard.com>
> ---
> .../raspberrypi/controller/af_algorithm.hpp | 20 ++
> src/ipa/raspberrypi/controller/af_status.h | 31 +++
> src/ipa/raspberrypi/controller/focus_status.h | 3 +
> src/ipa/raspberrypi/controller/iob/af.cpp | 231 ++++++++++++++++++
> src/ipa/raspberrypi/controller/iob/af.h | 55 +++++
> src/ipa/raspberrypi/meson.build | 1 +
> 6 files changed, 341 insertions(+)
> create mode 100644 src/ipa/raspberrypi/controller/af_algorithm.hpp
> create mode 100644 src/ipa/raspberrypi/controller/af_status.h
> create mode 100644 src/ipa/raspberrypi/controller/iob/af.cpp
> create mode 100644 src/ipa/raspberrypi/controller/iob/af.h
>
> diff --git a/src/ipa/raspberrypi/controller/af_algorithm.hpp b/src/ipa/raspberrypi/controller/af_algorithm.hpp
> new file mode 100644
> index 00000000..553a37e1
> --- /dev/null
> +++ b/src/ipa/raspberrypi/controller/af_algorithm.hpp
> @@ -0,0 +1,20 @@
> +/* SPDX-License-Identifier: BSD-2-Clause */
> +/*
> + * Copyright (C) 2019, Raspberry Pi (Trading) Limited
> + *
> + * af_algorithm.hpp - autofocus control algorithm interface
> + */
> +#pragma once
> +
> +#include "algorithm.hpp"
> +
> +namespace RPiController {
> +
> +class AfAlgorithm : public Algorithm
> +{
> +public:
> + AfAlgorithm(Controller *controller) : Algorithm(controller) {}
> + // An af algorithm must provide the following:
Yes, we have a bit of a chicken-or-egg thing going on here at the
moment, don't we? Should we try and make this look a bit more like an
interface that implements the AF Controls proposal (even though that's
not agreed yet). Or do we commit this first on the understanding that
we'll come back and patch it all up later? I don't think I really
mind, I guess it would just be good to be clear on the plan!
For the record, the kinds of methods we'd want in here would include
things like:
SetMode(AfMode mode); // set the AF mode (manual, auto, continuous)
Trigger(); // start a cycle (in auto mode)
Cancel(); // cancel a cycle (in auto mode)
SetWindows(...); // set AF windows
SetRange(....) ; // set AF range
etc.
> +};
> +
> +} // namespace RPiController
> diff --git a/src/ipa/raspberrypi/controller/af_status.h b/src/ipa/raspberrypi/controller/af_status.h
> new file mode 100644
> index 00000000..835e1e2f
> --- /dev/null
> +++ b/src/ipa/raspberrypi/controller/af_status.h
> @@ -0,0 +1,31 @@
> +/* SPDX-License-Identifier: BSD-2-Clause */
> +/*
> + * Copyright (C) 2020, Raspberry Pi (Trading) Limited
> + * Copyright (C) 2022, Ideas On Board
> + *
> + * af_status.h - autofocus measurement status
> + */
> +#pragma once
> +
> +#include <linux/bcm2835-isp.h>
> +
> +/*
> + * The focus algorithm should post the following structure into the image's
> + * "af.status" metadata.
> + */
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +struct AfStatus {
> + unsigned int num;
> + uint32_t focus_measures[FOCUS_REGIONS];
> + bool stable;
> + uint32_t focus;
> + double maxVariance;
> +};
Exactly what to include here is an interesting question. The basic
principles are that we should include:
* Anything the pipeline handler needs in order to control hardware or
other parts of the system. In this case that would probably mean we
pass out "the next lens position to ask for".
* Anything that we need to fill in the libcamera metadata. So this
would include the AfState (which is mostly just
scanning/focused/failed).
* We also like to pass out any settings that have been applied. This
is so that you know for sure whether the algorithm has seen the
settings you asked for. In this case it might include the mode, the AF
windows, the range, the speed etc. We do tend to do this even when
there's no libcamera metadata for the information - we take the view
that it's a thing we might need to consider adding at some point in
the future.
* Anything else that might reasonably be thought to be "useful". This
is obviously where things become more debatable!
Then the final principle would be that we mostly try to exclude other
things because (a) they might be algorithm specific and (b) it just
makes for more stuff to get wrong unless you really need them.
Looking specifically at the above:
focus_measures: I'm not convinced that these are necessary here,
particularly when they're in the focus algorithm metadata already.
stable: I guess this is a focused/not-focused kind of flag? Maybe a
scanning/focused/failed enum would be better, and look a bit more like
a future "full" AF algorithm?
focus: is that the next lens position to ask for?
maxVariance: that's feeling a bit implementation specific to me. The
question is always: what might the pipeline handler do with this?
> +
> +#ifdef __cplusplus
> +}
> +#endif
> diff --git a/src/ipa/raspberrypi/controller/focus_status.h b/src/ipa/raspberrypi/controller/focus_status.h
> index ace2fe2c..8122df4b 100644
> --- a/src/ipa/raspberrypi/controller/focus_status.h
> +++ b/src/ipa/raspberrypi/controller/focus_status.h
> @@ -19,6 +19,9 @@ extern "C" {
> struct FocusStatus {
> unsigned int num;
> uint32_t focus_measures[FOCUS_REGIONS];
> + bool stable;
> + uint32_t focus;
> + double maxVariance;
I'm guessing these should have been removed again when the AfStatus was added?
> };
>
> #ifdef __cplusplus
> diff --git a/src/ipa/raspberrypi/controller/iob/af.cpp b/src/ipa/raspberrypi/controller/iob/af.cpp
> new file mode 100644
> index 00000000..dc5258ba
> --- /dev/null
> +++ b/src/ipa/raspberrypi/controller/iob/af.cpp
> @@ -0,0 +1,231 @@
> +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> +/*
> + * Copyright (C) 2021, Red Hat
> + * Copyright (C) 2022, Ideas On Board
> + *
> + * af.cpp - automatic contrast-based focus algorithm
> + */
> +#include <cmath>
> +
> +#include <stdint.h>
> +
> +#include <libcamera/base/log.h>
> +
> +#include "af.h"
> +
> +using namespace RPiController;
> +using namespace libcamera;
> +
> +LOG_DEFINE_CATEGORY(IoBAf)
> +
> +#define NAME "iob.af"
> +
> +/*
> + * Maximum focus steps of the VCM control
> + * \todo should be obtained from the VCM driver
> + */
> +static constexpr uint32_t kMaxFocusSteps = 1023;
> +
> +/* Minimum focus step for searching appropriate focus */
> +static constexpr uint32_t kCoarseSearchStep = 30;
> +static constexpr uint32_t kFineSearchStep = 1;
> +
> +/* Max ratio of variance change, 0.0 < kMaxChange < 1.0 */
> +static constexpr double kMaxChange = 0.5;
> +
> +/* The numbers of frame to be ignored, before performing focus scan. */
> +static constexpr uint32_t kIgnoreFrame = 10;
> +
> +/* Fine scan range 0 < kFineRange < 1 */
> +static constexpr double kFineRange = 0.05;
> +
> +Af::Af(Controller *controller)
> + : AfAlgorithm(controller), focus_(0), bestFocus_(0), ignoreCounter_(0),
> + currentVariance_(0.0), previousVariance_(0.0), maxStep_(0),
> + coarseCompleted_(false), fineCompleted_(false)
> +{
> +}
> +
> +char const *Af::Name() const
> +{
> + return NAME;
> +}
> +
> +void Af::Initialise()
> +{
> + status_.focus = 0.0;
> + status_.maxVariance = 0.0;
> + status_.stable = false;
> +}
> +
> +void Af::Prepare(Metadata *image_metadata)
> +{
> + image_metadata->Set("af.status", status_);
> +}
> +
> +double Af::estimateVariance()
> +{
> + unsigned int i;
> + double mean;
> + uint64_t total = 0;
> + double var_sum = 0.0;
> +
> + /* Compute the mean value. */
> + for (i = 0; i < FOCUS_REGIONS; i++)
> + total += status_.focus_measures[i];
> + mean = total / FOCUS_REGIONS;
> +
> + /* Compute the sum of the squared variance. */
> + for (i = 0; i < FOCUS_REGIONS; i++)
> + var_sum += std::pow(status_.focus_measures[i] - mean, 2);
> +
> + return var_sum / FOCUS_REGIONS;
> +}
I'm interested in the choice of variance across the regions of the
per-pixel FoM for measuring the "goodness" of focus, rather than just
the per-pixel FoM. But maybe this kind of question can be left until
the algorithm is running and then we can examine the focus measure
curves and see what works best.
> +
> +bool Af::afNeedIgnoreFrame()
> +{
> + if (ignoreCounter_ == 0)
> + return false;
> + else
> + ignoreCounter_--;
> + return true;
> +}
> +
> +void Af::afCoarseScan()
> +{
> + if (coarseCompleted_)
> + return;
> +
> + if (afNeedIgnoreFrame())
> + return;
> +
> + if (afScan(kCoarseSearchStep)) {
> + coarseCompleted_ = true;
> + status_.maxVariance = 0;
> + focus_ = status_.focus - (status_.focus * kFineRange);
> + status_.focus = focus_;
> + previousVariance_ = 0;
> + maxStep_ = std::clamp(focus_ + static_cast<uint32_t>((focus_ * kFineRange)),
> + 0U, kMaxFocusSteps);
> + }
> +}
> +
> +void Af::afFineScan()
> +{
> + if (!coarseCompleted_)
> + return;
> +
> + if (afNeedIgnoreFrame())
> + return;
> +
> + if (afScan(kFineSearchStep)) {
> + status_.stable = true;
> + fineCompleted_ = true;
> + }
> +}
> +
> +bool Af::afScan(uint32_t minSteps)
> +{
> + if (focus_ > maxStep_) {
> + /* If the max step is reached, move lens to the position. */
> + status_.focus = bestFocus_;
> + return true;
> + } else {
> + /*
> + * Find the maximum of the variance by estimating its
> + * derivative. If the direction changes, it means we have passed
> + * a maximum one step before.
> + */
> + if ((currentVariance_ - status_.maxVariance) >=
> + -(status_.maxVariance * 0.1)) {
> + /*
> + * Positive and zero derivative:
> + * The variance is still increasing. The focus could be
> + * increased for the next comparison. Also, the max
> + * variance and previous focus value are updated.
> + */
> + bestFocus_ = focus_;
> + focus_ += minSteps;
> + status_.focus = focus_;
> + status_.maxVariance = currentVariance_;
> + } else {
> + /*
> + * Negative derivative:
> + * The variance starts to decrease which means the maximum
> + * variance is found. Set focus step to previous good one
> + * then return immediately.
> + */
> + status_.focus = bestFocus_;
> + return true;
> + }
> + }
> +
> + previousVariance_ = currentVariance_;
> + LOG(IoBAf, Debug) << " Previous step is "
> + << bestFocus_
> + << " Current step is "
> + << focus_;
> + return false;
> +}
> +
> +void Af::afReset()
> +{
> + if (afNeedIgnoreFrame())
> + return;
> +
> + status_.maxVariance = 0;
> + status_.focus = 0;
> + focus_ = 0;
> + status_.stable = false;
> + ignoreCounter_ = kIgnoreFrame;
> + previousVariance_ = 0.0;
> + coarseCompleted_ = false;
> + fineCompleted_ = false;
> + maxStep_ = kMaxFocusSteps;
> +}
> +
> +bool Af::afIsOutOfFocus()
> +{
> + const uint32_t diff_var = std::abs(currentVariance_ -
> + status_.maxVariance);
> + const double var_ratio = diff_var / status_.maxVariance;
> + LOG(IoBAf, Debug) << "Variance change rate: "
> + << var_ratio
> + << " Current VCM step: "
> + << status_.focus;
> + if (var_ratio > kMaxChange)
> + return true;
> + else
> + return false;
> +}
> +
> +void Af::Process(StatisticsPtr &stats, Metadata *image_metadata)
> +{
> + unsigned int i;
> + image_metadata->Get("af.status", status_);
> +
> + /* Use the second filter results only, and cache those. */
> + for (i = 0; i < FOCUS_REGIONS; i++)
> + status_.focus_measures[i] = stats->focus_stats[i].contrast_val[1][1]
> + / stats->focus_stats[i].contrast_val_num[1][1];
> + status_.num = i;
> +
> + currentVariance_ = estimateVariance();
> +
> + if (!status_.stable) {
> + afCoarseScan();
> + afFineScan();
> + } else {
> + if (afIsOutOfFocus())
> + afReset();
> + else
> + ignoreCounter_ = kIgnoreFrame;
> + }
> +}
> +
> +/* Register algorithm with the system. */
> +static Algorithm *Create(Controller *controller)
> +{
> + return new Af(controller);
> +}
> +static RegisterAlgorithm reg(NAME, &Create);
> diff --git a/src/ipa/raspberrypi/controller/iob/af.h b/src/ipa/raspberrypi/controller/iob/af.h
> new file mode 100644
> index 00000000..45c9711f
> --- /dev/null
> +++ b/src/ipa/raspberrypi/controller/iob/af.h
> @@ -0,0 +1,55 @@
> +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> +/*
> + * Copyright (C) 2021, Red Hat
> + * Copyright (C) 2022, Ideas On Board
> + *
> + * af.h - automatic contrast-based focus algorithm
> + */
> +#pragma once
> +
> +#include "../af_algorithm.hpp"
> +#include "../af_status.h"
> +#include "../metadata.hpp"
> +
> +namespace RPiController {
I think the namespace thing has been commented on elsewhere.
> +
> +class Af : public AfAlgorithm
> +{
> +public:
> + Af(Controller *controller);
> + char const *Name() const override;
> + void Initialise() override;
> + void Prepare(Metadata *image_metadata) override;
> + void Process(StatisticsPtr &stats, Metadata *image_metadata) override;
> +private:
> + double estimateVariance();
> + bool afNeedIgnoreFrame();
> + void afCoarseScan();
> + void afFineScan();
> + bool afScan(uint32_t minSteps);
> + void afReset();
> + bool afIsOutOfFocus();
> +
> + AfStatus status_;
> +
> + /* VCM step configuration. It is the current setting of the VCM step. */
> + uint32_t focus_;
> + /* The best VCM step. It is a local optimum VCM step during scanning. */
> + uint32_t bestFocus_;
> +
> + /* The frames ignored before starting measuring. */
> + uint32_t ignoreCounter_;
> +
> + /* Current AF statistic variance. */
> + double currentVariance_;
> + /* It is used to determine the derivative during scanning */
> + double previousVariance_;
> + /* The designated maximum range of focus scanning. */
> + uint32_t maxStep_;
> + /* If the coarse scan completes, it is set to true. */
> + bool coarseCompleted_;
> + /* If the fine scan completes, it is set to true. */
> + bool fineCompleted_;
> +};
> +
> +} /* namespace RPiController */
> diff --git a/src/ipa/raspberrypi/meson.build b/src/ipa/raspberrypi/meson.build
> index 32897e07..37068ecc 100644
> --- a/src/ipa/raspberrypi/meson.build
> +++ b/src/ipa/raspberrypi/meson.build
> @@ -28,6 +28,7 @@ rpi_ipa_sources = files([
> 'controller/controller.cpp',
> 'controller/histogram.cpp',
> 'controller/algorithm.cpp',
> + 'controller/iob/af.cpp',
> 'controller/rpi/alsc.cpp',
> 'controller/rpi/awb.cpp',
> 'controller/rpi/sharpen.cpp',
> --
> 2.32.0
>
One other thing I'd like to understand better is how an application
knows what lens positions to ask for in "manual mode". The most common
use case, I think, would be to set the lens to "hyperfocal position"
when the system starts, or after capturing an image following an AF
cycle.
How would we know what lens position corresponds to hyperfocal? Would
it be the default value of the V4L2 focus position control? The catch
here is that the lens actuator cannot know - in fact it can't even
know the valid usable range for any given type of module.
Does anyone have any thoughts on this?
Thanks again for all this work!
Best regards
David
More information about the libcamera-devel
mailing list