[libcamera-devel] [PATCH 07/22] ipa: ipu3: agc: Improve gain calculation

[Kieran Bingham]

Quoting Jean-Michel Hautbois (2021-11-08 13:13:35)
> When an image is partially saturated, its brightness is not increasing
> linearly when the shutter time or gain increases. It is a big issue with
> a backlight as the algorithm is fading to darkness right now.
> 
> Introduce a function to estimate the brightness of the frame, based on
> the current exposure/gain and loop on it several times to estimate it
> again and approach the non linear function.
> 
> Inspired-by: 7de5506c30b3 ("libcamera: src: ipa: raspberrypi: agc: Improve gain update calculation for partly saturated images")
> Signed-off-by: Jean-Michel Hautbois <jeanmichel.hautbois at ideasonboard.com>
> ---
>  src/ipa/ipu3/algorithms/agc.cpp | 77 ++++++++++++++++++++++++++++++++-
>  src/ipa/ipu3/algorithms/agc.h   |  6 ++-
>  2 files changed, 80 insertions(+), 3 deletions(-)
> 
> diff --git a/src/ipa/ipu3/algorithms/agc.cpp b/src/ipa/ipu3/algorithms/agc.cpp
> index 119a7938..e4048d40 100644
> --- a/src/ipa/ipu3/algorithms/agc.cpp
> +++ b/src/ipa/ipu3/algorithms/agc.cpp
> @@ -186,8 +186,9 @@ void Agc::filterExposure()
>   * \brief Estimate the new exposure and gain values
>   * \param[inout] exposure The exposure value reference as a number of lines
>   * \param[inout] gain The gain reference to be updated
> + * \param[in] currentYGain The gain calculated on the current brightness level
>   */
> -void Agc::computeExposure(uint32_t &exposure, double &analogueGain)
> +void Agc::computeExposure(uint32_t &exposure, double &analogueGain, double currentYGain)
>  {
>         /* Estimate the gain needed to have the proportion wanted */
>         double evGain = kEvGainTarget * knumHistogramBins / iqMean_;
> @@ -206,6 +207,14 @@ void Agc::computeExposure(uint32_t &exposure, double &analogueGain)
>                             << " Gain " << analogueGain
>                             << " Needed ev gain " << evGain;
>  
> +       if (evGain < currentYGain)
> +               evGain = currentYGain;
> +
> +       if (std::abs(evGain - 1.0) < 0.01) {
> +               LOG(IPU3Agc, Debug) << "We are well exposed (iqMean = "
> +                                   << iqMean_ << ")";
> +       }

{ } are not needed for a single line statement.


> +
>         /*
>          * Calculate the current exposure value for the scene as the latest
>          * exposure value applied multiplied by the new estimated gain.
> @@ -253,6 +262,48 @@ void Agc::computeExposure(uint32_t &exposure, double &analogueGain)
>         prevExposureValue_ = shutterTime * analogueGain;
>  }
>  
> +/**
> + * \brief Estimate the average brightness of the frame
> + * \param[in] context The shared IPA context
> + * \param[in] grid The grid used to store the statistics in the IPU3
> + * \param[in] stats The IPU3 statistics and ISP results
> + * \param[in] currentYGain The gain calculated on the current brightness level
> + */
> +double Agc::computeInitialY(IPAFrameContext &frameContext,
> +                           const ipu3_uapi_grid_config &grid,
> +                           const ipu3_uapi_stats_3a *stats,
> +                           double currentYGain)
> +{
> +       double redSum = 0, greenSum = 0, blueSum = 0;
> +
> +       for (unsigned int cellY = 0; cellY < grid.height; cellY++) {
> +               for (unsigned int cellX = 0; cellX < grid.width; cellX++) {
> +                       uint32_t cellPosition = cellY * stride_ + cellX;
> +
> +                       const ipu3_uapi_awb_set_item *cell =
> +                               reinterpret_cast<const ipu3_uapi_awb_set_item *>(
> +                                       &stats->awb_raw_buffer.meta_data[cellPosition]
> +                               );
> +
> +                       redSum += cell->R_avg * currentYGain;
> +                       greenSum += (cell->Gr_avg + cell->Gb_avg) / 2 * currentYGain;
> +                       blueSum += cell->B_avg * currentYGain;
> +               }
> +       }
> +
> +       /*
> +        * Estimate the sum of the brightness values, weighted with the gains
> +        * applied on the channels in AWB.
> +        */
> +       double Y_sum = redSum * frameContext.awb.gains.red * .299 +
> +                      greenSum * frameContext.awb.gains.green * .587 +
> +                      blueSum * frameContext.awb.gains.blue * .114;

Is this some RGB to HSL/YUV calcuation? Can we make it clearer somehow?

It currently looks like 'magic'.

> +
> +       /* And return the average brightness */
> +       return Y_sum / (grid.height * grid.width);
> +}
> +
> +
>  /**
>   * \brief Process IPU3 statistics, and run AGC operations
>   * \param[in] context The shared IPA context
> @@ -267,7 +318,29 @@ void Agc::process(IPAContext &context, const ipu3_uapi_stats_3a *stats)
>         uint32_t &exposure = context.frameContext.agc.exposure;
>         double &analogueGain = context.frameContext.agc.gain;
>         measureBrightness(stats, context.configuration.grid.bdsGrid);
> -       computeExposure(exposure, analogueGain);
> +
> +       double currentYGain = 1.0;
> +       /* \todo: the target Y needs to be grabbed from a configuration */

Can we quantify what '60' means at all yet? Is it 60%? 60/255? something
else? Does it have units or a meaning at all?

> +       double targetY = 60;

line break here.

> +       /*
> +        * Do this calculation a few times as brightness increase can be
> +        * non-linear when there are saturated regions.
> +        */
> +       for (int i = 0; i < 8; i++) {
> +               double initialY = computeInitialY(context.frameContext,
> +                                                 context.configuration.grid.bdsGrid,
> +                                                 stats, currentYGain);

does this value change? Should it be outside of the loop - that's a lot
of iterations over every cell.

Hrm ... in fact, it looks like it is dependant upon the currentYGain
value ... so maybe this is required.


> +               double extra_gain = std::min(10.0, targetY / (initialY + .001));
> +
> +               currentYGain *= extra_gain;
> +               LOG(IPU3Agc, Debug) << "Initial Y " << initialY
> +                                   << " target " << targetY
> +                                   << " gives gain " << currentYGain;
> +               if (extra_gain < 1.01)
> +                       break;
> +       }
> +
> +       computeExposure(exposure, analogueGain, currentYGain);
>         frameCount_++;
>  }
>  
> diff --git a/src/ipa/ipu3/algorithms/agc.h b/src/ipa/ipu3/algorithms/agc.h
> index 69e0b831..0a9152a9 100644
> --- a/src/ipa/ipu3/algorithms/agc.h
> +++ b/src/ipa/ipu3/algorithms/agc.h
> @@ -34,7 +34,11 @@ private:
>         void measureBrightness(const ipu3_uapi_stats_3a *stats,
>                                const ipu3_uapi_grid_config &grid);
>         void filterExposure();
> -       void computeExposure(uint32_t &exposure, double &gain);
> +       void computeExposure(uint32_t &exposure, double &gain, double currentYGain);
> +       double computeInitialY(IPAFrameContext &frameContext,
> +                              const ipu3_uapi_grid_config &grid,
> +                              const ipu3_uapi_stats_3a *stats,
> +                              double currentYGain);
>  
>         uint64_t frameCount_;
>         uint64_t lastFrame_;
> -- 
> 2.32.0
>