[libcamera-devel] Suboptimal resolution for digital zoom (controls::ScalerCrop)

[Laurent Pinchart]

Hi Elias,

On Mon, Apr 24, 2023 at 08:04:22PM -0600, Elias Naur via libcamera-devel wrote:
> Hi,
> 
> I'd like to squeeze out as much resolution as possible from my Raspberry Pi camera v1
> for scanning QR codes. The camera  has a fixed focus at ~50cm from the lens, so the
> resolution gain from moving the QR codes closer is lost to blur from the loss of focus.
> 
> My current strategy is to scan codes at a distance, but use the considerable resolution
> of the sensor (~2500x1900) to gain enough resolution for successful scans.
> 
> However, I fail to find any way to achieve that with libcamera. I tried three options, all
> suboptimal:
> 
> 1. Configure the camera for full resolution and crop to a smaller section in my program.
> That works surprisingly well, except that the white-balance etc. algorithms run on the full
> image and not just the cropped part.
> 2. Configure the camera for full resolution and set controls::ScalerCrop to the area
> of interest. This seems to fix the white-balance algorithms, but results in the cropped
> area being *upscaled* to full resolution when I receive them. Scanning for QR codes
> at full resolution is way too slow on a RPi Zero.
> 3. Configure the camera for the same resolution of the area of interest, and set
> controls::ScalerCrop. This is efficient, but the sensor is configured for the low
> resolution and thus ruins the resolution gain.
> 
> Questions:
> 1. Is there another way?

How about capturing full resolution frames, without cropping in the
capture pipeline, and scanning for the QR code in the area of interest
only ?

> 2. The following hack works for me,
> 
> diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> index 00600441..6ed81dc1 100644
> --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> @@ -842,7 +842,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)
>          * request, and apply it to the sensor with the cached transform, if
>          * any.
>          */
> -       V4L2SubdeviceFormat sensorFormat = findBestFormat(data->sensorFormats_, rawStream ? sensorSize : maxSize, bitDepth);
> +       V4L2SubdeviceFormat sensorFormat = findBestFormat(data->sensorFormats_, Size{2592,1944}, bitDepth);
>         const RPiCameraConfiguration *rpiConfig = static_cast<const RPiCameraConfiguration *>(config);
>         ret = data->sensor_->setFormat(&sensorFormat, rpiConfig->combinedTransform_);
>         if (ret)
> 
> Would a similar change that *always* uses the full sensor resolution be acceptable?
> I can't discern any performance issues between full resolution and lower resolution.

You can't hardcode a specific resolution, as this wouldn't work properly
with other sensors. Always selecting the sensor full resolution is also
not acceptable, as that will limit the frame rate on high resolution
sensors (e.g. the Raspberry Pi HQ camera will be limited to 15fps).

> 3. If the above fix is unacceptable, what is the right way to fix this
> issue in libcamera?

The current way to select the sensor resolution is a bit of a hack. You
need to add a raw stream to your camera configuration, set to the full
sensor resolution. You don't need to submit buffers for that stream when
queuing requests, so you can just capture from the processed YUV stream
as you do now.

We will likely implement in the future an easier API to select the
sensor resolution.

-- 
Regards,

Laurent Pinchart