[libcamera-devel] Raw streams and video

[Jacopo Mondi]

Hi David, Niklas,

On Wed, Sep 23, 2020 at 01:07:40PM +0200, Niklas Söderlund wrote:
> Hi David,
>
> Thanks for bringing this up. I have wrestled with similar questions
> myself for this API, my thoughts are however note yet mature. I will
> just fill in some extra information to yours to hopefully help the
> discussion.
>
> On 2020-09-23 09:45:43 +0100, David Plowman wrote:
> > Hi everyone
> >
> > I wanted to raise a topic that I've run into recently. One of the
> > features I want to implement is the ability to capture raw video
> > (meaning the raw Bayer frames from the sensor). In this use case we
> > might typically be capturing binned frames from the sensor.
> >
> > Originally I'd thought that all I need is Naush's set of raw stream
> > patches, however, now that they've appeared I'm not so sure.
>
> I think you can do what you wish with the current API can you not?
> Create a stream with a RAW format and the desired resolution and
> configure() wold have all the information it could possibly have from a
> stream created with generateConfiguration() with any current or future
> role. As you outline bellow, after generatConfuguration() returns a
> configuration the role no longer exists, roles are only used as hints to
> generate a configuration containing a set of streams with default
> formats and resolutions.

Just to add that the generated CameraConfiguration are not only about
sizes and formats, but they will transport a set of Controls which are
adjusted according to the desired role. I'm not sure at this point if
there's any control that would change between Raw and VideoRaw (I'm
thinking about the IPU3 that has a video stabilization feature, but it
applies on the ISP, not to the raw frames) but for several other use
cases I expect we might have roles which generate configurations
with the same size/format but different sets of 'default' controls.

>
> >
> > 1. The current StillCaptureRaw role seems inappropriate. It forces the
> > use of the full sensor resolution (I realise this is something I
> > wanted at the time!) and requests a small number of buffers.
> >
> > 2. I wonder whether it would be appropriate to introduce a "VideoRaw"
> > role. This could default to a larger number of buffers and not force
> > the maximum resolution. In fact, we would need to treat its resolution
> > quite differently - it would have no effect on selecting a resolution
> > (other streams would determine that), but would have its details
> > filled in after configure() has chosen the camera mode.
>
> We are still working on defining which roles we should have and maybe
> more important what applications can expect if they request a role. For
> raw our current direction is to rename StillCaptureRaw to Raw.  The idea
> is that it should be used by generateConfugiration() to create a stream
> in the configuration for the RAW buffers produced by the sensor taking
> the other requested roles into account for resolution. A patch to this
> effect [1] have been posted.
>
> >
> > One minor point is that the stream configurations don't "know" what
> > role they have, is that right? That would make it difficult for
> > configure() to distinguish "StillCaptureRaw" streams from "VideoRaw"
> > streams, so maybe there's something to think about there.

Also here I expect that knwoing which 'role' a configuration is
assigned to might help adjusting the pipeline configuration depending
on the desired use case, and again I think this mostly relates to the
default control values. David how would you use the role information
to distinguish between VideoRaw and Raw in your pipeline ?

>
> Here is where my concerns and thought have been focused. I think our
> current configuration API is a bit blunt and hard for pipelines to
> implement while still not being able to truly express the capabilities
> and limitations of a combinations of streams and current configuration.
>
> I like the idea that we create a CameraConfiguration object that is
> disconnected from the camera and owned by the application which it can
> tweak and validate, and when it wish to configure the camera it hands it
> an config object it knows the camera will accept.
>
> I don't like that it's hard for applications to enumerate the
> CameraConfiguration. After generating the configuration object the
> application IMHO should be able to reconfigure it in a more discoverable
> way. Say it changes the format on stream1 to A which makes it impossible
> for stream2 to also be configured with format A, as stream1 have higher
> priority then stream2 the application should then when enumerating
> formats for stream2 not see format A.
>

validate() partially helps, but I agree this can quickly become a
try-and-fail approach

> I'm envisioning a console test application that could interactively asks
> it user to configure it's camera, in a very simplistic form for some
> theoretical pipeline which only can support a single RAW stream and a
> ISP that supports 2 outputs, but it can only downscale
>
>
>     How many streams do you want (1-3):
>     > 3
>
>     For stream 0 which of the supported pixel formats do you want:
>     1: RAW
>     2: NV16
>     3: NV12
>     > 1
>
>     For stream 0 with format RAW which resolution do you want:
>     1: 640x480
>     2: 800x600
>     3: 1024x768
>     > 2
>
>     For stream 1 which of the supported pixel formats do you want:
>     1: NV16
>     2: NV12
>     > 1
>
>     For stream 1 with format NV16 which resolution do you want:
>     1: 640x480
>     2: 800x600
>     > 2
>
>     For stream 2 which of the supported pixel formats do you want:
>     1: NV16
>     2: NV12
>     > 2
>
>     For stream 2 with format NV12 which resolution do you want:
>     1: 640x480
>     2: 800x600
>     > 1
>
>     Stream 0 RAW 800x600
>     Stream 1 NV16 800x600
>     Stream 2 NV12 640x480
>
> I know Laurent is working on a new configuration API but I'm unfamiliar
> with the specifics. But for this reason I have put the shortcomings I
> see in the current interface low on my agenda as I'm sure the API will
> change with his work.
>
> On an related topic I'm also not found of exposing Camera::streams() to
> applications ;-)
>
> >
> > Anyway, apologies if I've started another meta-discussion. Thoughts
> > welcome, as always!
> >
> > Best regards
> > David
>
> 1. [PATCH] libcamera: stream: Rename StillCaptureRaw to Raw
>
> --
> Regards,
> Niklas Söderlund
> _______________________________________________
> libcamera-devel mailing list
> libcamera-devel at lists.libcamera.org
> https://lists.libcamera.org/listinfo/libcamera-devel