[libcamera-devel] Raw streams and video

David Plowman david.plowman at raspberrypi.com
Thu Sep 24 15:13:36 CEST 2020 

Ah OK, so maybe just set the raw stream dimensions to match my video
output and see what happens? I'll try that.

David

On Thu, 24 Sep 2020 at 13:42, Naushir Patuck <naush at raspberrypi.com> wrote:
>
> Hi David,
>
> On Thu, 24 Sep 2020 at 10:22, David Plowman
> <david.plowman at raspberrypi.com> wrote:
> >
> > Hi again
> >
> > Just one more update on this. I was playing around with it yesterday
> > and altered the behaviour of raw streams in the RPi pipeline handler a
> > little. I wouldn't propose this change exactly as it is, but maybe it
> > informs the discussion.
> >
> > 1. When you request a raw stream, a full sensor-resolution raw stream
> > config is returned by generateConfiguration(), just as now.
> >
> > 2. If the application doesn't do anything special then that raw stream
> > will force the full sensor resolution, and qcam exactly maintains its
> > current behaviour.
> >
> > 3. However, the application can set the size in the raw stream config
> > to zero. When it sees this, the pipeline handler ignores the raw
> > stream when figuring out the sensor mode, but then updates all those
> > details in the config once the sensor mode has been chosen.
>
> Having a (very) cursory look at the code,
> PipelineHandlerRPi::configure() in particular, it seems like the
> application should be able to request any RAW stream size in
> configure(), and the pipeline handler will match it to one of the
> camera modes.  So forcing the RAW stream size to 0 should not be
> necessary.  Is that not what you are seeing?  Of course we do end up
> in a situation where the pipeline selected RAW stream size will not be
> the same as the application requested RAW stream size, but that would
> be the case with setting it to 0 as well.
>
> Regards,
> Naush
>
>
>
> >
> > Working in this way, there is no difference between "still" or "video"
> > raw streams, it's merely a matter of how the application chooses to
> > configure it, and renaming the stream role to just "Raw" makes sense.
> > Setting the raw stream config size to zero seems a bit hacky though, I
> > don't know how anyone else feels about that!
> >
> > With the zero-copy raw patch it runs really well. I can run my 12MP
> > imx477 in full resolution 12-bit mode at the maximum rate (that's only
> > 10fps though), write all the raw frames straight to disk, and I don't
> > appear to drop a single one. Mind you, I'm running off an SSD which
> > makes it much easier to keep up with the 185MB/s data rate!!
> >
> > Best regards
> > David
> >
> > On Wed, 23 Sep 2020 at 15:53, David Plowman
> > <david.plowman at raspberrypi.com> wrote:
> > >
> > > Hi Niklas, Jacopo
> > >
> > > Well, I'm delighted to have found a receptive audience to my
> > > raw-related angst! I had indeed missed the patch about renaming
> > > StillCaptureRaw to just Raw, so thanks for pointing that out.
> > >
> > > On Wed, 23 Sep 2020 at 12:32, Jacopo Mondi <jacopo at jmondi.org> wrote:
> > > >
> > > > Hi David, Niklas,
> > > >
> > > > On Wed, Sep 23, 2020 at 01:07:40PM +0200, Niklas Söderlund wrote:
> > > > > Hi David,
> > > > >
> > > > > Thanks for bringing this up. I have wrestled with similar questions
> > > > > myself for this API, my thoughts are however note yet mature. I will
> > > > > just fill in some extra information to yours to hopefully help the
> > > > > discussion.
> > > > >
> > > > > On 2020-09-23 09:45:43 +0100, David Plowman wrote:
> > > > > > Hi everyone
> > > > > >
> > > > > > I wanted to raise a topic that I've run into recently. One of the
> > > > > > features I want to implement is the ability to capture raw video
> > > > > > (meaning the raw Bayer frames from the sensor). In this use case we
> > > > > > might typically be capturing binned frames from the sensor.
> > > > > >
> > > > > > Originally I'd thought that all I need is Naush's set of raw stream
> > > > > > patches, however, now that they've appeared I'm not so sure.
> > > > >
> > > > > I think you can do what you wish with the current API can you not?
> > > > > Create a stream with a RAW format and the desired resolution and
> > > > > configure() wold have all the information it could possibly have from a
> > > > > stream created with generateConfiguration() with any current or future
> > > > > role. As you outline bellow, after generatConfuguration() returns a
> > > > > configuration the role no longer exists, roles are only used as hints to
> > > > > generate a configuration containing a set of streams with default
> > > > > formats and resolutions.
> > > >
> > > > Just to add that the generated CameraConfiguration are not only about
> > > > sizes and formats, but they will transport a set of Controls which are
> > > > adjusted according to the desired role. I'm not sure at this point if
> > > > there's any control that would change between Raw and VideoRaw (I'm
> > > > thinking about the IPU3 that has a video stabilization feature, but it
> > > > applies on the ISP, not to the raw frames) but for several other use
> > > > cases I expect we might have roles which generate configurations
> > > > with the same size/format but different sets of 'default' controls.
> > > >
> > > > >
> > > > > >
> > > > > > 1. The current StillCaptureRaw role seems inappropriate. It forces the
> > > > > > use of the full sensor resolution (I realise this is something I
> > > > > > wanted at the time!) and requests a small number of buffers.
> > > > > >
> > > > > > 2. I wonder whether it would be appropriate to introduce a "VideoRaw"
> > > > > > role. This could default to a larger number of buffers and not force
> > > > > > the maximum resolution. In fact, we would need to treat its resolution
> > > > > > quite differently - it would have no effect on selecting a resolution
> > > > > > (other streams would determine that), but would have its details
> > > > > > filled in after configure() has chosen the camera mode.
> > > > >
> > > > > We are still working on defining which roles we should have and maybe
> > > > > more important what applications can expect if they request a role. For
> > > > > raw our current direction is to rename StillCaptureRaw to Raw.  The idea
> > > > > is that it should be used by generateConfugiration() to create a stream
> > > > > in the configuration for the RAW buffers produced by the sensor taking
> > > > > the other requested roles into account for resolution. A patch to this
> > > > > effect [1] have been posted.
> > > > >
> > > > > >
> > > > > > One minor point is that the stream configurations don't "know" what
> > > > > > role they have, is that right? That would make it difficult for
> > > > > > configure() to distinguish "StillCaptureRaw" streams from "VideoRaw"
> > > > > > streams, so maybe there's something to think about there.
> > >
> > > So I like the idea of renaming StillCaptureRaw to just Raw, and I'm
> > > certainly of the opinion that one raw stream shouldn't really be
> > > different from any other. I only proposed "VideoRaw" so as to be
> > > distinct from "StillCaptureRaw" as the latter has some problematic
> > > behaviours, but I'll come onto that in a moment. I'd much rather have
> > > just a "Raw" stream type.
> > >
> > > So what I'd hope for is something like this:
> > >
> > > * The application can request the raw stream, and a configuration for
> > > it will be returned by generateConfiguration().
> > >
> > > * By default the raw stream shouldn't influence the camera mode chosen
> > > - that's determined by the other streams, so they do have to be
> > > handled differently in the pipeline handler.
> > >
> > > * However the application can, if it wants, fill in the raw stream
> > > resolution at which point it will be respected (i.e. we'll get raw
> > > frames of that size or bigger, if possible). But if you don't, you get
> > > whatever suits the other streams, and the size and format of the raw
> > > stream are all filled in for you by validate() or configure() or
> > > someone. (So just as a note, we need to know when the user has updated
> > > the stream size and when they haven't.)
> > >
> > > Now, StillCaptureRaw is a bit problematic because, in the RPi pipeline
> > > handler at least, it forces the full resolution camera mode to be
> > > chosen, and moving to a single type of Raw stream, without this
> > > behaviour, will break the existing qcam behaviour. We could:
> > >
> > > 1. Change the qcam app to request the full resolution in the raw
> > > stream, irrespective of any other stream resolutions.
> > >
> > > 2. Ditch this behaviour completely. As you know we have our libcamera
> > > version of "raspistill" ("libcamera-still") which does raw captures
> > > without this problem.  So anyway, this gives us an alternative way to
> > > get those full resolution raws.
> > >
> > > >
> > > > Also here I expect that knwoing which 'role' a configuration is
> > > > assigned to might help adjusting the pipeline configuration depending
> > > > on the desired use case, and again I think this mostly relates to the
> > > > default control values. David how would you use the role information
> > > > to distinguish between VideoRaw and Raw in your pipeline ?
> > >
> > > If we've resolved the difficulties with the existing StillCaptureRaw
> > > streams as explained above, then I don't think we'd have any need to
> > > distinguish.
> > >
> > > Just finally - is this change actively being worked on, or is it
> > > mostly just discussion at the moment?
> > >
> > > Thanks everyone!
> > > David
> > >
> > > >
> > > > >
> > > > > Here is where my concerns and thought have been focused. I think our
> > > > > current configuration API is a bit blunt and hard for pipelines to
> > > > > implement while still not being able to truly express the capabilities
> > > > > and limitations of a combinations of streams and current configuration.
> > > > >
> > > > > I like the idea that we create a CameraConfiguration object that is
> > > > > disconnected from the camera and owned by the application which it can
> > > > > tweak and validate, and when it wish to configure the camera it hands it
> > > > > an config object it knows the camera will accept.
> > > > >
> > > > > I don't like that it's hard for applications to enumerate the
> > > > > CameraConfiguration. After generating the configuration object the
> > > > > application IMHO should be able to reconfigure it in a more discoverable
> > > > > way. Say it changes the format on stream1 to A which makes it impossible
> > > > > for stream2 to also be configured with format A, as stream1 have higher
> > > > > priority then stream2 the application should then when enumerating
> > > > > formats for stream2 not see format A.
> > > > >
> > > >
> > > > validate() partially helps, but I agree this can quickly become a
> > > > try-and-fail approach
> > > >
> > > > > I'm envisioning a console test application that could interactively asks
> > > > > it user to configure it's camera, in a very simplistic form for some
> > > > > theoretical pipeline which only can support a single RAW stream and a
> > > > > ISP that supports 2 outputs, but it can only downscale
> > > > >
> > > > >
> > > > >     How many streams do you want (1-3):
> > > > >     > 3
> > > > >
> > > > >     For stream 0 which of the supported pixel formats do you want:
> > > > >     1: RAW
> > > > >     2: NV16
> > > > >     3: NV12
> > > > >     > 1
> > > > >
> > > > >     For stream 0 with format RAW which resolution do you want:
> > > > >     1: 640x480
> > > > >     2: 800x600
> > > > >     3: 1024x768
> > > > >     > 2
> > > > >
> > > > >     For stream 1 which of the supported pixel formats do you want:
> > > > >     1: NV16
> > > > >     2: NV12
> > > > >     > 1
> > > > >
> > > > >     For stream 1 with format NV16 which resolution do you want:
> > > > >     1: 640x480
> > > > >     2: 800x600
> > > > >     > 2
> > > > >
> > > > >     For stream 2 which of the supported pixel formats do you want:
> > > > >     1: NV16
> > > > >     2: NV12
> > > > >     > 2
> > > > >
> > > > >     For stream 2 with format NV12 which resolution do you want:
> > > > >     1: 640x480
> > > > >     2: 800x600
> > > > >     > 1
> > > > >
> > > > >     Stream 0 RAW 800x600
> > > > >     Stream 1 NV16 800x600
> > > > >     Stream 2 NV12 640x480
> > > > >
> > > > > I know Laurent is working on a new configuration API but I'm unfamiliar
> > > > > with the specifics. But for this reason I have put the shortcomings I
> > > > > see in the current interface low on my agenda as I'm sure the API will
> > > > > change with his work.
> > > > >
> > > > > On an related topic I'm also not found of exposing Camera::streams() to
> > > > > applications ;-)
> > > > >
> > > > > >
> > > > > > Anyway, apologies if I've started another meta-discussion. Thoughts
> > > > > > welcome, as always!
> > > > > >
> > > > > > Best regards
> > > > > > David
> > > > >
> > > > > 1. [PATCH] libcamera: stream: Rename StillCaptureRaw to Raw
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Niklas Söderlund
> > > > > _______________________________________________
> > > > > libcamera-devel mailing list
> > > > > libcamera-devel at lists.libcamera.org
> > > > > https://lists.libcamera.org/listinfo/libcamera-devel

More information about the libcamera-devel mailing list