[libcamera-devel] Cropping (aka. digital zoom)

[Jacopo Mondi]

Hello,
  just a few questions that I have after reading the whole thread

On Fri, Jun 19, 2020 at 12:36:45PM +0100, David Plowman wrote:
> Hi everyone
>
> Another thing I'd like to discuss is the ability to crop a region of
> interest from a camera image. This is another feature that will be
> available in many camera systems but which is not currently possible
> within libcamera.
>
> Just to recap, cropping specifies a window in the input image, and
> everything outside this window is discarded. However, the used portion
> of the image is still scaled to the same final output size, making
> this a natural mechanism for implementing digital zoom.
>
> I would therefore like to propose a new control consisting of four
> numbers:
>
> * The first two are the x and y offsets of the crop window within the
>   input image. Having these parameters allows an application to pan
>   around within the input image.
>
> * The next two would be the width and height of the crop window.

How does this play with the current stream dimensions ?

I assume that the stream desired size is the final up-scaled
image size (or down-scaled, why not), while the proposed control
specifies the image sensor pixel array portion to process, and
up(down)-scale through the ISP.

If we consider the control to select the pixel array portion to
process, I agree it should apply to the pixel array active matrix
size, being it expressed in pixels, or as a fraction of the active
area size.

If we consider this a sensor-related control, Naush' question on how to
take "hidden" crops to maintain the aspect ratio would be answered by
respecting the sequence of operation that actually take place, and it
will be responsibility of the ISP to further crop the image received
from the sensor before up(down)-scaling it to the user requested size.

Making out of this a Controls::Sensor::digitalCrop control (we don't
have control and properties namespaces, yet, but you got the idea)
what would happes is the application will have to know the active
pixel array size dimension (we have a property for that), select the
portion to extract through the control, and request the final
(up-scaled) image size through the StreamConfiguration.size parameter
as it would usually do.

In this case, the Sensor::digitalCrop control would limit the size of
the available raw frames (ie frames captured before ISP processing at
the CSI-2 receiver output), but I think that's expected ?

More references on how this handled on Android:
https://jmondi.org/android_metadata_tags/docs.html#controls_android.scaler.cropRegion

Thanks
  j


>
> I believe it's sensible for these numbers to be ratios, rather than
> (for example) pixels. This makes it much easier for an application to
> use. To specify that you want the middle quarter of an image ("2x
> digital zoom"), you'd pass the numbers 0.25, 0.25, 0.5 and 0.5. Note
> how setting width == height guarantees that you don't mess up your
> aspect ratio.
>
> Questions
>
> 1. How to represent the numbers?
>
> I see an existing Rectangle class that gets used within controls, so
> maybe that makes sense. I'd perhaps go with fixed point values such
> that 65536 is equivalent to 1.0? (Though that doesn't leave huge scope
> for sub-pixel cropping, where platforms support this...)
>
> 2. Are there devices that can't pan within the input image?
>
> How would they signal this? I guess reporting min and max
> rectangles with identical x and y values would cover this.
>
> 3. Minimum and maximum values?
>
> A platform could also report "reasonable" minimum width and height
> values that indicate roughly what is supported. Values of 1.0 here
> (65536) would mean that there is no zoom capability.
>
> Valid maximum widths and heights of course depend on x and y (or
> vice versa). Perhaps we just report 1.0 here.
>
> 4. How to handle awkward cases or requests?
>
> I think we have to allow implementations silently to coerce the
> requested values into the "closest" thing that works. This might
> involve changing offsets (for example) to accommodate the requested
> width/height, or even adjusting the crop slightly to satisfy
> things like pixel alignment requirements.
>
> Perhaps the metadata returned in a request can indicate a value closer
> to what was actually used.
>
> Finally, I'm sure there are other questions and things I've
> overlooked. What do people think?
>
> Best regards
> David
> _______________________________________________
> libcamera-devel mailing list
> libcamera-devel at lists.libcamera.org
> https://lists.libcamera.org/listinfo/libcamera-devel