[libcamera-devel] [PATCH] pipeline: raspberrypi: Fix rounding issue in findBestFormat()

[David Plowman]

Hi Naush

Oops, yes we should have spotted that! Thanks!

On Tue, 2 Nov 2021 at 12:47, Naushir Patuck <naush at raspberrypi.com> wrote:

> The aspect ratio calculation divides two integer values then casts to a
> double.
> This might reduce precision when scoring for aspect rato differences.
>
> Fix this by casting the integer to a double before the division.
>
> Reported-by: Coverity CID=361652
> Signed-off-by: Naushir Patuck <naush at raspberrypi.com>
>

Reviewed-by: David Plowman <david.plowman at raspberrypi.com>

Thanks
David


> ---
>  src/libcamera/pipeline/raspberrypi/raspberrypi.cpp | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> index 44760093f7ab..5e1f227398a9 100644
> --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> @@ -141,7 +141,7 @@ V4L2SubdeviceFormat findBestFormat(const SensorFormats
> &formatsMap, const Size &
>
>                 for (const Size &size : iter.second) {
>                         double reqAr = static_cast<double>(req.width) /
> req.height;
> -                       double fmtAr = size.width / size.height;
> +                       double fmtAr = static_cast<double>(size.width) /
> size.height;
>
>                         /* Score the dimensions for closeness. */
>                         score = scoreFormat(req.width, size.width);
> --
> 2.25.1
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20211102/b0df255f/attachment.htm>