[PATCH 2/5] libipa: histogram: Fix quantile() calculation for fractional results

[Laurent Pinchart]

Hi Stefan,

Thank you for the patch.

On Mon, Mar 24, 2025 at 06:07:37PM +0100, Stefan Klug wrote:
> The calculation of the frac variable is based solely on integers and
> therefore results in the fractional part being either 0 or 1.

Oops.

> In the original code from RaspberryPi this is mitigated by casting the
> nominator to a double. This works for most cases, but fails when q is
> very small because of the quantization introduced by item being an
> integer.
> 
> Fix both issues by doing the full calculation in double.
> 
> Signed-off-by: Stefan Klug <stefan.klug at ideasonboard.com>
> ---
>  src/ipa/libipa/histogram.cpp | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/src/ipa/libipa/histogram.cpp b/src/ipa/libipa/histogram.cpp
> index 10e44b54a0cf..c19a4cbbf3cd 100644
> --- a/src/ipa/libipa/histogram.cpp
> +++ b/src/ipa/libipa/histogram.cpp
> @@ -130,7 +130,8 @@ double Histogram::quantile(double q, uint32_t first, uint32_t last) const
>  	if (cumulative_[first + 1] == cumulative_[first])
>  		frac = 0;
>  	else
> -		frac = (item - cumulative_[first]) / (cumulative_[first + 1] - cumulative_[first]);
> +		frac = (q * total() - cumulative_[first]) /
> +		       (cumulative_[first + 1] - cumulative_[first]);

Maybe

		frac = (q * total() - cumulative_[first])
		     / (cumulative_[first + 1] - cumulative_[first]);

Reviewed-by: Laurent Pinchart <laurent.pinchart at ideasonboard.com>

>  	return first + frac;
>  }
>  

-- 
Regards,

Laurent Pinchart