[libcamera-devel] [PATCH] pipeline: raspberrypi: Fix rounding issue in findBestFormat()

[Naushir Patuck]

The aspect ratio calculation divides two integer values then casts to a double.
This might reduce precision when scoring for aspect rato differences.

Fix this by casting the integer to a double before the division.

Reported-by: Coverity CID=361652
Signed-off-by: Naushir Patuck <naush at raspberrypi.com>
---
 src/libcamera/pipeline/raspberrypi/raspberrypi.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
index 44760093f7ab..5e1f227398a9 100644
--- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
+++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
@@ -141,7 +141,7 @@ V4L2SubdeviceFormat findBestFormat(const SensorFormats &formatsMap, const Size &
 
 		for (const Size &size : iter.second) {
 			double reqAr = static_cast<double>(req.width) / req.height;
-			double fmtAr = size.width / size.height;
+			double fmtAr = static_cast<double>(size.width) / size.height;
 
 			/* Score the dimensions for closeness. */
 			score = scoreFormat(req.width, size.width);
-- 
2.25.1