[libcamera-devel] Custom automatic image processing

[Marco Felsch]

Hi Laurent,

thanks for your fast response :)

On 20-04-29 19:33, Laurent Pinchart wrote:
> Hi Marco,
> 
> On Wed, Apr 29, 2020 at 04:49:51PM +0200, Marco Felsch wrote:
> > Hi all,
> > 
> > first of all I'm very new to this project. I noticed the project since
> > the ELCE2018 but never got intouch with it until now. A customer of us
> > wants to implement a custom automatic image processing unit within the
> > gstreamer pipeline. This isn't the right place for it and we proposed
> > libcamera but I don't know how much work we need to implement it. Here
> > are the facts:
> >   - Platform: imx6
> 
> Which i.MX6 is that ? There are important variations in the camera
> hardware between different SoCs in the i.MX6 family.

It's an i.MX6 Dual.

> >   - Camera: AR0237 (I will send a driver in the next weeks)
> 
> Please make sure to support at least the following features in that
> driver:
> 
> - .get_selection() with the V4L2_SEL_TGT_CROP_BOUNDS,
>   V4L2_SEL_TGT_NATIVE_SIZE and V4L2_SEL_TGT_CROP targets
>   (V4L2_SEL_TGT_NATIVE_SIZE isn't required today but we're considering
>   switching from V4L2_SEL_TGT_CROP_BOUNDS to V4L2_SEL_TGT_NATIVE_SIZE)
> - V4L2_CID_PIXEL_RATE in read-only mode
> - V4L2_CID_HBLANK in read-only mode (read-write is fine too)

Thanks for this list :)

> The driver should also support the following features, even if they're
> not mandatory at the moment:
> 
> - Direct control of binning and cropping through the .set_selection()
>   and .set_fmt() operations (no hardcoded list of modes with register
>   lists)

Luckily I went that way.

> - V4L2_CID_VBLANK in read-write mode
> 
> >   - Gstreamer as media framework
> 
> Thanks to Nicolas we have GStreamer support for libcamera :-)

And that is working fine? Just saw a longer ToDo, therfore I'm asking
here :)

> >   - Custom embedded OS (no Android)
> > 
> > The sensor can embed sensor statistics into the frame at the beginning
> > and the end. We need to extract those data, calc all necessary values
> > (like gain, exposure, ...) and adjust the sensor using v4l2-ctrls if
> > necessary.
> > 
> > Pls, can someone provide me some information what we can expect from the
> > current master state. What is working? How much work would you expect
> > for the above use-case?
> 
> I few questions before providing you with answers.
> 
> - Unless I'm mistaken, the i.MX6 doesn't support colour interpolation
>   (de-bayering) in hardware, and the AR0237 can only produce raw Bayer
>   data, right ?

You're right the iMX6 has no de-bayer unit. Therefore we wanna do the
de-bayering within the GPU using the GStreamer gl plugins.

> - Is the i.MX6 you're using capable of capturing the embedded data and
>   embedded statistics to a different buffer than the image data
>   (splitting the incoming frame based on line numbers to different DMA
>   engines) ?

As far as I know no. The i.MX6 IPU is not capable to split the incoming
stream. So we need to extract them manually.

Regards,
  Marco