[libcamera-devel] Custom automatic image processing

[Laurent Pinchart]

Hi Marco,

On Thu, Apr 30, 2020 at 11:07:53AM +0200, Marco Felsch wrote:
> On 20-04-29 19:33, Laurent Pinchart wrote:
> > On Wed, Apr 29, 2020 at 04:49:51PM +0200, Marco Felsch wrote:
> > > Hi all,
> > > 
> > > first of all I'm very new to this project. I noticed the project since
> > > the ELCE2018 but never got intouch with it until now. A customer of us
> > > wants to implement a custom automatic image processing unit within the
> > > gstreamer pipeline. This isn't the right place for it and we proposed
> > > libcamera but I don't know how much work we need to implement it. Here
> > > are the facts:
> > >   - Platform: imx6
> > 
> > Which i.MX6 is that ? There are important variations in the camera
> > hardware between different SoCs in the i.MX6 family.
> 
> It's an i.MX6 Dual.

IPUv3 based then, OK. Not supported in libcamera yet, but that would be
a nice addition :-)

> > >   - Camera: AR0237 (I will send a driver in the next weeks)
> > 
> > Please make sure to support at least the following features in that
> > driver:
> > 
> > - .get_selection() with the V4L2_SEL_TGT_CROP_BOUNDS,
> >   V4L2_SEL_TGT_NATIVE_SIZE and V4L2_SEL_TGT_CROP targets
> >   (V4L2_SEL_TGT_NATIVE_SIZE isn't required today but we're considering
> >   switching from V4L2_SEL_TGT_CROP_BOUNDS to V4L2_SEL_TGT_NATIVE_SIZE)
> > - V4L2_CID_PIXEL_RATE in read-only mode
> > - V4L2_CID_HBLANK in read-only mode (read-write is fine too)
> 
> Thanks for this list :)
> 
> > The driver should also support the following features, even if they're
> > not mandatory at the moment:
> > 
> > - Direct control of binning and cropping through the .set_selection()
> >   and .set_fmt() operations (no hardcoded list of modes with register
> >   lists)
> 
> Luckily I went that way.

Nice :-)

> > - V4L2_CID_VBLANK in read-write mode
> > 
> > >   - Gstreamer as media framework
> > 
> > Thanks to Nicolas we have GStreamer support for libcamera :-)
> 
> And that is working fine? Just saw a longer ToDo, therfore I'm asking
> here :)

It has been tested and is working, but as you noted there's a todo list,
so everything isn't perfect (yet). It largely depends on what your
particular needs are on the gstreamer side, but I don't see any
particular problem that couldn't be solved by some more work on the
libcamera gstreamer element.

> > >   - Custom embedded OS (no Android)
> > > 
> > > The sensor can embed sensor statistics into the frame at the beginning
> > > and the end. We need to extract those data, calc all necessary values
> > > (like gain, exposure, ...) and adjust the sensor using v4l2-ctrls if
> > > necessary.
> > > 
> > > Pls, can someone provide me some information what we can expect from the
> > > current master state. What is working? How much work would you expect
> > > for the above use-case?
> > 
> > I few questions before providing you with answers.
> > 
> > - Unless I'm mistaken, the i.MX6 doesn't support colour interpolation
> >   (de-bayering) in hardware, and the AR0237 can only produce raw Bayer
> >   data, right ?
> 
> You're right the iMX6 has no de-bayer unit. Therefore we wanna do the
> de-bayering within the GPU using the GStreamer gl plugins.
> 
> > - Is the i.MX6 you're using capable of capturing the embedded data and
> >   embedded statistics to a different buffer than the image data
> >   (splitting the incoming frame based on line numbers to different DMA
> >   engines) ?
> 
> As far as I know no. The i.MX6 IPU is not capable to split the incoming
> stream. So we need to extract them manually.

OK. As you de-bayer with the GPU then I assume the GPU will skip the
embedded data lines.

I don't see any particular obstacle. We would need to implement an IPA
module for the algorithms (which ones are you interested in by the way,
will you configure other sensor parameters than exposure time and gain
?). The IPA would be given the image buffer, and extract embedded data.
This is already what happens today when embedded data is captured to a
separate buffer, having a large image in the buffer between embedded
data and embedded statistics isn't a big deal.

The only point that could require an API extension in libcamera is
reporting the amount of embedded data lines at the beginning and end of
the image to applications, for the GPU to properly skip them. I would
however like to explore the possibility of implementing GPU de-bayering
directly inside libcamera instead of using a GStreamer gl plugin, as
that's a component that could be reused between multiple devices.

-- 
Regards,

Laurent Pinchart