[libcamera-devel] Suboptimal resolution for digital zoom (controls::ScalerCrop)

[Elias Naur]

On Tue, 25 Apr 2023 at 05:24, David Plowman
<david.plowman at raspberrypi.com> wrote:
>
> Hi
>
> It should also be possible to request a raw stream at the full sensor
> resolution, but set the output size to the crop that you really want.
> Initially you'll obviously find that the full image is being
> downscaled to your output resolution. But then set the ScalerCrop to
> match the output size that you asked for, and you should be able to
> get a 1:1 crop from the full resolution image. Does that help?
>
> David

This sounds similar to Laurent's idea below. It worked, thank you.
For future reference, I spent some time figuring out that my dummy raw
stream needed two properties to function as a sensor resolution setting,
only:

  rawConfig.colorSpace = ColorSpace::Raw;
  rawConfig.bufferCount = 0;

Without the colorSpace setting, Camera::configure fails even with a prior
CameraConfiguration::validate. Without the bufferCount setting, you
won't get any request callbacks.

>
> On Tue, 25 Apr 2023 at 07:34, Laurent Pinchart via libcamera-devel
> <libcamera-devel at lists.libcamera.org> wrote:
> >
> > Hi Elias,
> >
> > On Mon, Apr 24, 2023 at 08:04:22PM -0600, Elias Naur via libcamera-devel wrote:
> > > Hi,
> > >
> > > I'd like to squeeze out as much resolution as possible from my Raspberry Pi camera v1
> > > for scanning QR codes. The camera  has a fixed focus at ~50cm from the lens, so the
> > > resolution gain from moving the QR codes closer is lost to blur from the loss of focus.
> > >
> > > My current strategy is to scan codes at a distance, but use the considerable resolution
> > > of the sensor (~2500x1900) to gain enough resolution for successful scans.
> > >
> > > However, I fail to find any way to achieve that with libcamera. I tried three options, all
> > > suboptimal:
> > >
> > > 1. Configure the camera for full resolution and crop to a smaller section in my program.
> > > That works surprisingly well, except that the white-balance etc. algorithms run on the full
> > > image and not just the cropped part.
> > > 2. Configure the camera for full resolution and set controls::ScalerCrop to the area
> > > of interest. This seems to fix the white-balance algorithms, but results in the cropped
> > > area being *upscaled* to full resolution when I receive them. Scanning for QR codes
> > > at full resolution is way too slow on a RPi Zero.
> > > 3. Configure the camera for the same resolution of the area of interest, and set
> > > controls::ScalerCrop. This is efficient, but the sensor is configured for the low
> > > resolution and thus ruins the resolution gain.
> > >
> > > Questions:
> > > 1. Is there another way?
> >
> > How about capturing full resolution frames, without cropping in the
> > capture pipeline, and scanning for the QR code in the area of interest
> > only ?
> >

I believe this is my option 1. above. It works, except that auto-white balance
etc. runs on the full field of view and not just the cropped subimage.

> > > 2. The following hack works for me,
> > >
> > > diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> > > b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> > > index 00600441..6ed81dc1 100644
> > > --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> > > +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> > > @@ -842,7 +842,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)
> > >          * request, and apply it to the sensor with the cached transform, if
> > >          * any.
> > >          */
> > > -       V4L2SubdeviceFormat sensorFormat = findBestFormat(data->sensorFormats_, rawStream ? sensorSize : maxSize, bitDepth);
> > > +       V4L2SubdeviceFormat sensorFormat = findBestFormat(data->sensorFormats_, Size{2592,1944}, bitDepth);
> > >         const RPiCameraConfiguration *rpiConfig = static_cast<const RPiCameraConfiguration *>(config);
> > >         ret = data->sensor_->setFormat(&sensorFormat, rpiConfig->combinedTransform_);
> > >         if (ret)
> > >
> > > Would a similar change that *always* uses the full sensor resolution be acceptable?
> > > I can't discern any performance issues between full resolution and lower resolution.
> >
> > You can't hardcode a specific resolution, as this wouldn't work properly
> > with other sensors. Always selecting the sensor full resolution is also
> > not acceptable, as that will limit the frame rate on high resolution
> > sensors (e.g. the Raspberry Pi HQ camera will be limited to 15fps).
> >

I thought so.

> > > 3. If the above fix is unacceptable, what is the right way to fix this
> > > issue in libcamera?
> >
> > The current way to select the sensor resolution is a bit of a hack. You
> > need to add a raw stream to your camera configuration, set to the full
> > sensor resolution. You don't need to submit buffers for that stream when
> > queuing requests, so you can just capture from the processed YUV stream
> > as you do now.
> >
> > We will likely implement in the future an easier API to select the
> > sensor resolution.
> >

This idea worked, thank you! As you may have expected it took me a
while to figure out
the right incantations, so a more intuitive way of setting the sensor resolution
would be greatly appreciated.

Elias