[libcamera-devel] Raw streams and video
David Plowman
david.plowman at raspberrypi.com
Wed Sep 23 16:53:30 CEST 2020
Hi Niklas, Jacopo
Well, I'm delighted to have found a receptive audience to my
raw-related angst! I had indeed missed the patch about renaming
StillCaptureRaw to just Raw, so thanks for pointing that out.
On Wed, 23 Sep 2020 at 12:32, Jacopo Mondi <jacopo at jmondi.org> wrote:
>
> Hi David, Niklas,
>
> On Wed, Sep 23, 2020 at 01:07:40PM +0200, Niklas Söderlund wrote:
> > Hi David,
> >
> > Thanks for bringing this up. I have wrestled with similar questions
> > myself for this API, my thoughts are however note yet mature. I will
> > just fill in some extra information to yours to hopefully help the
> > discussion.
> >
> > On 2020-09-23 09:45:43 +0100, David Plowman wrote:
> > > Hi everyone
> > >
> > > I wanted to raise a topic that I've run into recently. One of the
> > > features I want to implement is the ability to capture raw video
> > > (meaning the raw Bayer frames from the sensor). In this use case we
> > > might typically be capturing binned frames from the sensor.
> > >
> > > Originally I'd thought that all I need is Naush's set of raw stream
> > > patches, however, now that they've appeared I'm not so sure.
> >
> > I think you can do what you wish with the current API can you not?
> > Create a stream with a RAW format and the desired resolution and
> > configure() wold have all the information it could possibly have from a
> > stream created with generateConfiguration() with any current or future
> > role. As you outline bellow, after generatConfuguration() returns a
> > configuration the role no longer exists, roles are only used as hints to
> > generate a configuration containing a set of streams with default
> > formats and resolutions.
>
> Just to add that the generated CameraConfiguration are not only about
> sizes and formats, but they will transport a set of Controls which are
> adjusted according to the desired role. I'm not sure at this point if
> there's any control that would change between Raw and VideoRaw (I'm
> thinking about the IPU3 that has a video stabilization feature, but it
> applies on the ISP, not to the raw frames) but for several other use
> cases I expect we might have roles which generate configurations
> with the same size/format but different sets of 'default' controls.
>
> >
> > >
> > > 1. The current StillCaptureRaw role seems inappropriate. It forces the
> > > use of the full sensor resolution (I realise this is something I
> > > wanted at the time!) and requests a small number of buffers.
> > >
> > > 2. I wonder whether it would be appropriate to introduce a "VideoRaw"
> > > role. This could default to a larger number of buffers and not force
> > > the maximum resolution. In fact, we would need to treat its resolution
> > > quite differently - it would have no effect on selecting a resolution
> > > (other streams would determine that), but would have its details
> > > filled in after configure() has chosen the camera mode.
> >
> > We are still working on defining which roles we should have and maybe
> > more important what applications can expect if they request a role. For
> > raw our current direction is to rename StillCaptureRaw to Raw. The idea
> > is that it should be used by generateConfugiration() to create a stream
> > in the configuration for the RAW buffers produced by the sensor taking
> > the other requested roles into account for resolution. A patch to this
> > effect [1] have been posted.
> >
> > >
> > > One minor point is that the stream configurations don't "know" what
> > > role they have, is that right? That would make it difficult for
> > > configure() to distinguish "StillCaptureRaw" streams from "VideoRaw"
> > > streams, so maybe there's something to think about there.
So I like the idea of renaming StillCaptureRaw to just Raw, and I'm
certainly of the opinion that one raw stream shouldn't really be
different from any other. I only proposed "VideoRaw" so as to be
distinct from "StillCaptureRaw" as the latter has some problematic
behaviours, but I'll come onto that in a moment. I'd much rather have
just a "Raw" stream type.
So what I'd hope for is something like this:
* The application can request the raw stream, and a configuration for
it will be returned by generateConfiguration().
* By default the raw stream shouldn't influence the camera mode chosen
- that's determined by the other streams, so they do have to be
handled differently in the pipeline handler.
* However the application can, if it wants, fill in the raw stream
resolution at which point it will be respected (i.e. we'll get raw
frames of that size or bigger, if possible). But if you don't, you get
whatever suits the other streams, and the size and format of the raw
stream are all filled in for you by validate() or configure() or
someone. (So just as a note, we need to know when the user has updated
the stream size and when they haven't.)
Now, StillCaptureRaw is a bit problematic because, in the RPi pipeline
handler at least, it forces the full resolution camera mode to be
chosen, and moving to a single type of Raw stream, without this
behaviour, will break the existing qcam behaviour. We could:
1. Change the qcam app to request the full resolution in the raw
stream, irrespective of any other stream resolutions.
2. Ditch this behaviour completely. As you know we have our libcamera
version of "raspistill" ("libcamera-still") which does raw captures
without this problem. So anyway, this gives us an alternative way to
get those full resolution raws.
>
> Also here I expect that knwoing which 'role' a configuration is
> assigned to might help adjusting the pipeline configuration depending
> on the desired use case, and again I think this mostly relates to the
> default control values. David how would you use the role information
> to distinguish between VideoRaw and Raw in your pipeline ?
If we've resolved the difficulties with the existing StillCaptureRaw
streams as explained above, then I don't think we'd have any need to
distinguish.
Just finally - is this change actively being worked on, or is it
mostly just discussion at the moment?
Thanks everyone!
David
>
> >
> > Here is where my concerns and thought have been focused. I think our
> > current configuration API is a bit blunt and hard for pipelines to
> > implement while still not being able to truly express the capabilities
> > and limitations of a combinations of streams and current configuration.
> >
> > I like the idea that we create a CameraConfiguration object that is
> > disconnected from the camera and owned by the application which it can
> > tweak and validate, and when it wish to configure the camera it hands it
> > an config object it knows the camera will accept.
> >
> > I don't like that it's hard for applications to enumerate the
> > CameraConfiguration. After generating the configuration object the
> > application IMHO should be able to reconfigure it in a more discoverable
> > way. Say it changes the format on stream1 to A which makes it impossible
> > for stream2 to also be configured with format A, as stream1 have higher
> > priority then stream2 the application should then when enumerating
> > formats for stream2 not see format A.
> >
>
> validate() partially helps, but I agree this can quickly become a
> try-and-fail approach
>
> > I'm envisioning a console test application that could interactively asks
> > it user to configure it's camera, in a very simplistic form for some
> > theoretical pipeline which only can support a single RAW stream and a
> > ISP that supports 2 outputs, but it can only downscale
> >
> >
> > How many streams do you want (1-3):
> > > 3
> >
> > For stream 0 which of the supported pixel formats do you want:
> > 1: RAW
> > 2: NV16
> > 3: NV12
> > > 1
> >
> > For stream 0 with format RAW which resolution do you want:
> > 1: 640x480
> > 2: 800x600
> > 3: 1024x768
> > > 2
> >
> > For stream 1 which of the supported pixel formats do you want:
> > 1: NV16
> > 2: NV12
> > > 1
> >
> > For stream 1 with format NV16 which resolution do you want:
> > 1: 640x480
> > 2: 800x600
> > > 2
> >
> > For stream 2 which of the supported pixel formats do you want:
> > 1: NV16
> > 2: NV12
> > > 2
> >
> > For stream 2 with format NV12 which resolution do you want:
> > 1: 640x480
> > 2: 800x600
> > > 1
> >
> > Stream 0 RAW 800x600
> > Stream 1 NV16 800x600
> > Stream 2 NV12 640x480
> >
> > I know Laurent is working on a new configuration API but I'm unfamiliar
> > with the specifics. But for this reason I have put the shortcomings I
> > see in the current interface low on my agenda as I'm sure the API will
> > change with his work.
> >
> > On an related topic I'm also not found of exposing Camera::streams() to
> > applications ;-)
> >
> > >
> > > Anyway, apologies if I've started another meta-discussion. Thoughts
> > > welcome, as always!
> > >
> > > Best regards
> > > David
> >
> > 1. [PATCH] libcamera: stream: Rename StillCaptureRaw to Raw
> >
> > --
> > Regards,
> > Niklas Söderlund
> > _______________________________________________
> > libcamera-devel mailing list
> > libcamera-devel at lists.libcamera.org
> > https://lists.libcamera.org/listinfo/libcamera-devel
More information about the libcamera-devel
mailing list