[libcamera-devel] gstreamer libcamerasrc negotiation woes

[Kieran Bingham]

Hi Thorsten,

Sorry to revive an old thread, but it was still sat on my todo list, and
recent discussions relating to this have probably cleared up what the
issue you faced was...


Quoting Kieran Bingham (2021-09-21 11:33:41)
> Hi Thorsten,
> 
> On 14/09/2021 18:31, Thorsten von Eicken wrote:
> > I'm trying to get libcamerasrc to work on an rPi with the standard V2
> > camera module (imx219) and I have some behavior I cannot explain. I'm
> > trying to stream h264 from the camera to rtsp, but am starting using
> > just a filesink for testing. After toying around, the following
> > commandline works but I don't understand why the v4l2convert is necessary:
> > 
> > gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=640,height=480' !
> > v4l2convert ! v4l2h264enc ! filesink location=test.h264

The V4L2convert component is currently 'required' because our
gstlibcamerasrc is not exposing colorimetry or framerates correctly.


https://bugs.libcamera.org/show_bug.cgi?id=75#c8 shows an example
capsfilter that will allow the components to negotiate without requiring
a v4l2convert.

gst-launch-1.0 -vvvv libcamerasrc \
	! video/x-raw,width=1280,height=720,format=NV12,colorimetry=bt601,framerate=30/1,interlace-mode=progressive \
	! v4l2h264enc extra-controls="controls,repeat_sequence_header=1" \
	! 'video/x-h264,level=(string)4' ! h264parse ! matroskamux \
	! filesink location=foo.mkv


> > 
> > Looking at the debug output (reproduced at the end of this message)
> > libcamerasrc ends up outputting video/x-raw in NV21 format, then
> > v4l2convert converts it to YUY2, and then v4l2h264enc encodes to h.264.
> 
> We certainly don't want unnecessary converts occurring.
> 
> 
> > But looking at `gst-inspect-1.0 v4l2h264enc` I see that v4l2h264enc
> > accepts NV21:
> > 
> >   SINK template: 'sink'
> >     Availability: Always
> >     Capabilities:
> >       video/x-raw
> >                  format: { (string)I420, (string)YV12, (string)NV12,
> > (string)NV21, (string)RGB16, (s
> > tring)RGB, (string)BGR, (string)BGRx, (string)BGRA, (string)YUY2,
> > (string)YVYU, (string)UYVY }
> >                   width: [ 1, 32768 ]
> >                  height: [ 1, 32768 ]
> >               framerate: [ 0/1, 2147483647/1 ]
> > 
> > So I should be able to do without the v4l2convert, right? Removing it
> > results in failure:
> 
> Yes, I would have expected so...
> 
> So the question we need to solve is - why is the negotiation failing.
> 
> 
> > 
> > ERROR: from element
> > /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0: Internal data
> > stream error.
> > Additional debug info:
> > ../src/gstreamer/gstlibcamerasrc.cpp(323): gst_libcamera_src_task_run
> > (): /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0:streaming
> > stopped, reason not-negotiated (-4)
> 
> 
> If the formats are compatible, the sizes are compatible ...
> The only other item listed there is the framerate. I wonder if that's a
> suspect.
> 
> I don't have a RPi available to operate on currently, but this would be
> the first area I'd check.
> 
> 
> > even if I change the caps filter to add a "format=NV21" I get the same
> > result. What is going on?
> > 
> > NB: why do I always get the following critical error that seems
> > completely benign when I use libcamerasrc?
> > 
> > ** (gst-launch-1.0:907): CRITICAL **: 10:09:51.212:
> > gst_video_format_to_string: assertion 'format !=
> >  GST_VIDEO_FORMAT_UNKNOWN' failed
> 
> 
> I guess somewhere a pixel format conversion (just the type, not the
> actual data) is occuring between libcamera and gstreamer.
> 
> I see you've referenced a bug elsewhere showing that the I420 type is
> not mapped, so perhaps it's that when trying to do negotiation.
> 
> It looks like this is handled in src/gstreamer/gstlibcamera-utils.cpp.
> 
> There is a format map that likely needs extending at a minimum.
> 
> GST_VIDEO_FORMAT_I420 would need adding to the appropriate libcamera format.
> 
> I see you referenced this in :
> 
> https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=319571&p=1914817&sid=79614f1638dd6b781ab6e14dc68c8f1a#p1914865
> 
> but you reported that it didn't fix your main issue. (Though it looks
> like it did make the warning disappear?)
> 
> 
> Would you like to submit that as a patch? Or would you prefer that I
> handle it ?

This was added in :

https://git.libcamera.org/libcamera/libcamera.git/commit/?id=f9ca897c7fe2846f0a2163563014ca02d3c1624a

So should now also be resolved.

--
Kieran