[libcamera-devel] using libcamerasrc gst element

[Laurent Pinchart]

Hi Nicolas,

On Sun, Sep 05, 2021 at 10:06:56PM -0400, Nicolas Dufresne wrote:
> Le 5 sept. 2021 19 h 40, Laurent Pinchart a écrit :
> > On Fri, Sep 03, 2021 at 09:28:10AM +0200, Martin Kepplinger wrote:
> > > Am Donnerstag, dem 02.09.2021 um 17:48 +0300 schrieb Laurent Pinchart:
> > > > On Thu, Sep 02, 2021 at 10:56:41AM +0200, Martin Kepplinger wrote:
> > > > > hi all,
> > > > >
> > > > > I'm starting to use libcamera with the hi846 camera on imx8mq. (The
> > > > > following all works on my laptop btw, where there's only 1 camera.)
> > > > >
> > > > > I test using the "libaperture" gstreamer app with the libcamera gst src element
> > > > > (https://gitlab.gnome.org/jwestman/libaperture/-/issues/14 ).
> > > > > But it doesn't work on imx8mq where I have 2 cameras (and 1 is working
> > > > > with libcamera currently, but that shouldn't matter I think).
> > > > >
> > > > > Here's the end of the logs. earlier logs seem to be the same as the
> > > > > successful case with qcam, see below.
> > > > >
> > > > >
> > > > > [1:57:12.541751844] [3187] DEBUG Camera camera_manager.cpp:150
> > > > > Found registered pipeline handler 'PipelineHandlerUVC'
> > > > > [1:57:12.541817127] [3187] DEBUG Camera camera_manager.cpp:150
> > > > > Found registered pipeline handler 'PipelineHandlerVimc'
> > > > > [1:57:12.542646484] [3180] DEBUG Camera camera.cpp:878 streams
> > > > > configuration: (0) 1632x1224-SGBRG16
> > > > > [1:57:12.543146546] [3187] DEBUG V4L2 v4l2_videodevice.cpp:1423
> > > > > /dev/video0[32:cap]: Releasing buffers
> > > > > [1:57:12.543212069] [3187] DEBUG V4L2 v4l2_videodevice.cpp:1160
> > > > > /dev/video0[32:cap]: 0 buffers requested.
> > >
> > > [snip]
> > >
> > > > > what does libcamerasrc do wrong? Why isn't it configuring the stream?
> > > > > What config options do I have for the gstreamer element to play with?
> > > >
> > > > Have you tried using libcamerasrc with a simple gst-launch pipeline
> > > > first ? That could give us additional feedback from GStreamer itself.
> > >
> > > thanks. first, verified that the gstreamer test src works:
> > >
> > > LANG=C LIBCAMERA_LOG_FILE=/home/purism/lc.log LIBCAMERA_LOG_LEVELS=0
> > > GST_PLUGIN_PATH=/usr/local/lib/aarch64-linux-gnu/gstreamer-1.0/
> > > gst-launch-1.0 videotestsrc ! glimagesink
> > >
> > > one appearent problem: gstreamer and libcamerasrc don't know about the
> > > SGBRG16 format I'm receiving. Tried to workaround that like so (?):
> > >
> > > --- a/src/gstreamer/gstlibcamera-utils.cpp
> > > +++ b/src/gstreamer/gstlibcamera-utils.cpp
> > > @@ -30,6 +30,8 @@ static struct {
> > >         { GST_VIDEO_FORMAT_YUY2, formats::YUYV },
> > >         { GST_VIDEO_FORMAT_YVYU, formats::YVYU },
> > >         /* \todo NV42 is used in libcamera but is not mapped in
> > > GStreamer yet. */
> > > +       { GST_VIDEO_FORMAT_GRAY16_BE, formats::SGBRG16 },
> >
> > I think this should be GST_VIDEO_FORMAT_GRAY16_LE.
> >
> > We should also add support for the other Bayer patterns, but it won't be
> > as trivial as adding the SBGGR, SGRBG and SRGGB entries here, as
> > gst_format_to_pixel_format() won't necessarily pick the right format in
> > that case.
> >
> > This reminds me that I think we should decouple the CFA pattern from the
> > format in libcamera, but that's for later. Nicolas, do I understand
> > correctly that GStreamer uses x-bayer instead of x-raw for Bayer formats
> > ? I can't see a bit depth being specified by x-bayer, does it mean it
> > supports 8bpp only ?
> 
> Correct, x-bayer and all defined formats are currently 8 bits. The bayer
> support is also missing stride support (missing equivalent of GstVideoFromat).
> 
> Bayer formats do not fit generically with x-raw, as the concept of pixels is
> very different. This is why it's not integrated into libgstvideo.
> 
> There is very little users for Bayer in GStreamer, but also in most
> applications. No one actually even spent the time or had the need for a proper
> integration. I wonder how usable this is, I know this SoC is missing an ISP,
> but it might be better to use a software or GPU ISP inside libcamera.

That's the goal, it will just take some time :-) I was wondering if
Bayer capture with GStreamer was possible in the meantime for testing.

> > > +       /* 16bit bayer not supported by gstreamer yet (?) */
> > >  };
> 
> Correct, but in absence of supporting library, adding support is just about
> adding the format string for it. Of course having minimal debayering support
> for that would be a plus.
> 
> > > but that results in:
> > >
> > > purism at pureos:~$ GST_DEBUG=3 LANG=C
> > > LIBCAMERA_LOG_FILE=/home/purism/lc.log LIBCAMERA_LOG_LEVELS=0
> > > GST_PLUGIN_PATH=/usr/local/lib/aarch64-linux-gnu/gstreamer-1.0/
> > > gst-launch-1.0 libcamerasrc ! videoconvert ! glimagesink
> > > Setting pipeline to PAUSED ...
> > > Pipeline is live and does not need PREROLL ...
> > > 0:00:01.793370564  3386 0xaaaaddf97760 FIXME                default gstutils.c:4025:gst_pad_create_stream_id_internal:<libcamerasrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
> > > Got context from element 'sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayWayland\)\gldisplaywayland0";
> > > Pipeline is PREROLLED ...
> > > Setting pipeline to PLAYING ...
> > > New clock: GstSystemClock
> > >
> > > (gst-launch-1.0:3386): GStreamer-CRITICAL **: 09:20:50.114: gst_value_set_int_range_step: assertion 'start < end' failed
> > >
> > > (gst-launch-1.0:3386): GStreamer-CRITICAL **: 09:20:50.116: gst_value_set_int_range_step: assertion 'start < end' failed
> 
> This is an assertion, don't expect anything to work when such assertion pops.
> (They are only there for crash avoidance). Can you generate a backtrack with
> env G_DEBUG=fatal_criticals and share ?
> 
> > > ^Chandling interrupt.
> > > Interrupt: Stopping pipeline ...
> > > Execution ended after 0:00:05.128408250
> > > Setting pipeline to NULL ...
> > > Freeing pipeline ...
> > >
> > >
> > > the clock is *not* running. it just shows how long it ran after killing
> > > it.
> > >
> > > not sure what my specific question is but in case you've seen this
> > > before and have an idea, just say so :)
> > 
> > I haven't tried to capture raw data using libcamerasrc yet, so I'm not
> > sure what to advise :-S

-- 
Regards,

Laurent Pinchart