[libcamera-devel] gstreamer libcamerasrc negotiation woes
Kieran Bingham
kieran.bingham at ideasonboard.com
Tue Sep 21 12:33:41 CEST 2021
Hi Thorsten,
On 14/09/2021 18:31, Thorsten von Eicken wrote:
> I'm trying to get libcamerasrc to work on an rPi with the standard V2
> camera module (imx219) and I have some behavior I cannot explain. I'm
> trying to stream h264 from the camera to rtsp, but am starting using
> just a filesink for testing. After toying around, the following
> commandline works but I don't understand why the v4l2convert is necessary:
>
> gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=640,height=480' !
> v4l2convert ! v4l2h264enc ! filesink location=test.h264
>
> Looking at the debug output (reproduced at the end of this message)
> libcamerasrc ends up outputting video/x-raw in NV21 format, then
> v4l2convert converts it to YUY2, and then v4l2h264enc encodes to h.264.
We certainly don't want unnecessary converts occurring.
> But looking at `gst-inspect-1.0 v4l2h264enc` I see that v4l2h264enc
> accepts NV21:
>
> SINK template: 'sink'
> Availability: Always
> Capabilities:
> video/x-raw
> format: { (string)I420, (string)YV12, (string)NV12,
> (string)NV21, (string)RGB16, (s
> tring)RGB, (string)BGR, (string)BGRx, (string)BGRA, (string)YUY2,
> (string)YVYU, (string)UYVY }
> width: [ 1, 32768 ]
> height: [ 1, 32768 ]
> framerate: [ 0/1, 2147483647/1 ]
>
> So I should be able to do without the v4l2convert, right? Removing it
> results in failure:
Yes, I would have expected so...
So the question we need to solve is - why is the negotiation failing.
>
> ERROR: from element
> /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0: Internal data
> stream error.
> Additional debug info:
> ../src/gstreamer/gstlibcamerasrc.cpp(323): gst_libcamera_src_task_run
> (): /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0:streaming
> stopped, reason not-negotiated (-4)
If the formats are compatible, the sizes are compatible ...
The only other item listed there is the framerate. I wonder if that's a
suspect.
I don't have a RPi available to operate on currently, but this would be
the first area I'd check.
> even if I change the caps filter to add a "format=NV21" I get the same
> result. What is going on?
>
> NB: why do I always get the following critical error that seems
> completely benign when I use libcamerasrc?
>
> ** (gst-launch-1.0:907): CRITICAL **: 10:09:51.212:
> gst_video_format_to_string: assertion 'format !=
> GST_VIDEO_FORMAT_UNKNOWN' failed
I guess somewhere a pixel format conversion (just the type, not the
actual data) is occuring between libcamera and gstreamer.
I see you've referenced a bug elsewhere showing that the I420 type is
not mapped, so perhaps it's that when trying to do negotiation.
It looks like this is handled in src/gstreamer/gstlibcamera-utils.cpp.
There is a format map that likely needs extending at a minimum.
GST_VIDEO_FORMAT_I420 would need adding to the appropriate libcamera format.
I see you referenced this in :
https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=319571&p=1914817&sid=79614f1638dd6b781ab6e14dc68c8f1a#p1914865
but you reported that it didn't fix your main issue. (Though it looks
like it did make the warning disappear?)
Would you like to submit that as a patch? Or would you prefer that I
handle it ?
>
> Setup:
>
> - rPi3B+
> - Linux rpi3 5.10.60-v7+ #1449 SMP Wed Aug 25 15:00:01 BST 2021 armv7l
> GNU/Linux
> - libcamera compiled from source (#4cf7a8f)
>
I'm trying to get my RPi4 up to be able to test locally.
It seems a few people have hit this issue recently, so we'll try to get
to the bottom of it.
--
Regards
Kieran
> - /boot/config.txt:
> dtoverlay=vc4-fkms-v3d
> dtoverlay=imx219
> gpu_mem=128
> start_x=1
> max_framebuffers=0
>
> ---------------------------
>
> Log from working gst-launch:
>
> tve at rpi3 ~> gst-launch-1.0 -vvv libcamerasrc !
> 'video/x-raw,width=640,height=480' ! v4l2con[120/410]
> l2h264enc ! filesink location=test.h264
> Setting pipeline to PAUSED ...
> [0:04:22.812479146] [907] INFO Camera camera_manager.cpp:294 libcamera
> v0.0.0+3032-4cf7a8fc
> Pipeline is live and does not need PREROLL ...
> Setting pipeline to PLAYING ...
> New clock: GstSystemClock
>
> ** (gst-launch-1.0:907): CRITICAL **: 10:09:51.212:
> gst_video_format_to_string: assertion 'format !=
> GST_VIDEO_FORMAT_UNKNOWN' failed
> /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0.GstLibcameraPad:src: caps
> = video/x-raw, format
> =(string)NV21, width=(int)640, height=(int)480
> /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps =
> video/x-raw, format=(string)NV21
> , width=(int)640, height=(int)480
> /GstPipeline:pipeline0/v4l2convert:v4l2convert0.GstPad:src: caps =
> video/x-raw, format=(string)YUY2,
> width=(int)640, height=(int)480, framerate=(fraction)0/1,
> interlace-mode=(string)progressive, color
> imetry=(string)bt709
> /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0.GstPad:src: caps =
> video/x-h264, stream-format=(stri
> ng)byte-stream, alignment=(string)au, profile=(string)high,
> level=(string)4, width=(int)640, height=
> (int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1,
> interlace-mode=(string)progress
> ive, colorimetry=(string)bt709
> /GstPipeline:pipeline0/GstFileSink:filesink0.GstPad:sink: caps =
> video/x-h264, stream-format=(string
> )byte-stream, alignment=(string)au, profile=(string)high,
> level=(string)4, width=(int)640, height=(i
> nt)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1,
> interlace-mode=(string)progressiv
> e, colorimetry=(string)bt709
> Redistribute latency...
> /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0.GstPad:sink: caps =
> video/x-raw, format=(string)YUY2
> , width=(int)640, height=(int)480, framerate=(fraction)0/1,
> interlace-mode=(string)progressive, colo
> rimetry=(string)bt709
> /GstPipeline:pipeline0/v4l2convert:v4l2convert0.GstPad:sink: caps =
> video/x-raw, format=(string)NV21
> , width=(int)640, height=(int)480
> /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps =
> video/x-raw, format=(string)NV2
> 1, width=(int)640, height=(int)480
> [0:04:23.273836423] [911] INFO Camera camera.cpp:937 configuring
> streams: (0) 640x480-NV21
> [0:04:23.274589971] [908] INFO RPI raspberrypi.cpp:122 Mode: 3280x2464
> fmt RG10 Score: 2662.49 (bes
> t 2662.49)
> [0:04:23.274793410] [908] INFO RPI raspberrypi.cpp:122 Mode: 1920x1080
> fmt RG10 Score: 2136.67 (bes
> t 2136.67)
> [0:04:23.274908515] [908] INFO RPI raspberrypi.cpp:122 Mode: 1640x1232
> fmt RG10 Score: 1944.49 (bes
> t 1944.49)
> [0:04:23.275018256] [908] INFO RPI raspberrypi.cpp:122 Mode: 640x480
> fmt RG10 Score: 1500 (best 150
> 0)
> [0:04:23.275138725] [908] INFO RPI raspberrypi.cpp:122 Mode: 3280x2464
> fmt pRAA Score: 2162.49 (bes
> t 1500)
> [0:04:23.275250185] [908] INFO RPI raspberrypi.cpp:122 Mode: 1920x1080
> fmt pRAA Score: 1636.67 (bes
> t 1500)
> [0:04:23.275358884] [908] INFO RPI raspberrypi.cpp:122 Mode: 1640x1232
> fmt pRAA Score: 1444.49 (bes
> t 1444.49)
> [0:04:23.275468989] [908] INFO RPI raspberrypi.cpp:122 Mode: 640x480
> fmt pRAA Score: 1000 (best 100
> 0)
> [0:04:23.275605500] [908] INFO RPI raspberrypi.cpp:122 Mode: 3280x2464
> fmt RGGB Score: 3662.49 (bes
> t 1000)
> [0:04:23.275722168] [908] INFO RPI raspberrypi.cpp:122 Mode: 1920x1080
> fmt RGGB Score: 3136.67 (bes
> t 1000)
> [0:04:23.275834669] [908] INFO RPI raspberrypi.cpp:122 Mode: 1640x1232
> fmt RGGB Score: 2944.49 (bes
> t 1000)
> [0:04:23.275943576] [908] INFO RPI raspberrypi.cpp:122 Mode: 640x480
> fmt RGGB Score: 2500 (best 100
> 0)
> [0:04:23.276445872] [908] INFO RPI raspberrypi.cpp:620 Sensor:
> /base/soc/i2c0mux/i2c at 1/imx219 at 10 -
> Selected mode: 640x480-pRAA
> [0:04:23.298140323] [908] INFO RPISTREAM rpi_stream.cpp:122 No buffers
> available for ISP Output0
> [0:04:23.298277980] [908] INFO RPISTREAM rpi_stream.cpp:122 No buffers
> available for ISP Output0
> [0:04:23.298366731] [908] INFO RPISTREAM rpi_stream.cpp:122 No buffers
> available for ISP Output0
> [0:04:23.358951818] [908] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA (6
> left)
> [0:04:23.391732142] [908] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA (5
> left)
> [0:04:23.424765802] [908] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA (4
> left)
> [0:04:23.456638254] [908] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA ($
> left)
> [0:04:23.489703946] [908] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA (2
> left)
> [0:04:23.523865480] [908] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA (1
> left)
> [0:04:23.555756161] [908] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA (0
> left)
> ^Chandling interrupt. {that's me hitting ctrl-c to stop the recording}
> Interrupt: Stopping pipeline ...
> Execution ended after 0:00:28.749092992
> Setting pipeline to PAUSED ...
> Setting pipeline to READY ...
> Setting pipeline to NULL ...
> Freeing pipeline ...
>
> ---------------------------
>
> Log from failing gst-launch:
>
> tve at rpi3 ~> gst-launch-1.0 -vvv libcamerasrc !
> 'video/x-raw,format=NV21,width=640,height=480' ! v4l2
> h264enc ! filesink location=test.h264
> Setting pipeline to PAUSED ...
> [0:22:34.494879960] [1057] INFO Camera camera_manager.cpp:294 libcamera
> v0.0.0+3032-4cf7a8fc
> Pipeline is live and does not need PREROLL ...
> Setting pipeline to PLAYING ...
> New clock: GstSystemClock
>
> ** (gst-launch-1.0:1057): CRITICAL **: 10:28:02.891:
> gst_video_format_to_string: assertion 'format !
> = GST_VIDEO_FORMAT_UNKNOWN' failed
> /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0.GstLibcameraPad:src: caps
> = video/x-raw, format
> =(string)NV21, width=(int)640, height=(int)480
> [0:22:34.886945683] [1061] INFO Camera camera.cpp:937 configuring
> streams: (0) 640x480-NV21
> [0:22:34.887579223] [1058] INFO RPI raspberrypi.cpp:122 Mode: 3280x2464
> fmt RG10 Score: 2662.49 (be
> st 2662.49)
> [0:22:34.887769482] [1058] INFO RPI raspberrypi.cpp:122 Mode: 1920x1080
> fmt RG10 Score: 2136.67 (be
> st 2136.67)
> [0:22:34.887883284] [1058] INFO RPI raspberrypi.cpp:122 Mode: 1640x1232
> fmt RG10 Score: 1944.49 (be
> st 1944.49)
> [0:22:34.887994482] [1058] INFO RPI raspberrypi.cpp:122 Mode: 640x480
> fmt RG10 Score: 1500 (best 15
> 00)
> [0:22:34.888113127] [1058] INFO RPI raspberrypi.cpp:122 Mode: 3280x2464
> fmt pRAA Score: 2162.49 (be
> st 1500)
> [0:22:34.888223439] [1058] INFO RPI raspberrypi.cpp:122 Mode: 1920x1080
> fmt pRAA Score: 1636.67 (be
> st 1500)
> [0:22:34.888332293] [1058] INFO RPI raspberrypi.cpp:122 Mode: 1640x1232
> fmt pRAA Score: 1444.49 (be
> st 1444.49)
> [0:22:34.888441668] [1058] INFO RPI raspberrypi.cpp:122 Mode: 640x480
> fmt pRAA Score: 1000 (best 10
> 00)
> [0:22:34.888575886] [1058] INFO RPI raspberrypi.cpp:122 Mode: 3280x2464
> fmt RGGB Score: 3662.49 (be
> st 1000)
> [0:22:34.888684844] [1058] INFO RPI raspberrypi.cpp:122 Mode: 1920x1080
> fmt RGGB Score: 3136.67 (b$
> st 1000)
> [0:22:34.888790729] [1058] INFO RPI raspberrypi.cpp:122 Mode: 1640x1232
> fmt RGGB Score: 2944.49 (be
> st 1000)
> [0:22:34.888899531] [1058] INFO RPI raspberrypi.cpp:122 Mode: 640x480
> fmt RGGB Score: 2500 (best 10
> 00)
> [0:22:34.889397602] [1058] INFO RPI raspberrypi.cpp:620 Sensor:
> /base/soc/i2c0mux/i2c at 1/imx219 at 10 -
> Selected mode: 640x480-pRAA
> [0:22:34.943958581] [1058] INFO RPISTREAM rpi_stream.cpp:122 No buffers
> available for ISP Output0
> [0:22:34.944094934] [1058] INFO RPISTREAM rpi_stream.cpp:122 No buffers
> available for ISP Output0
> [0:22:34.944183320] [1058] INFO RPISTREAM rpi_stream.cpp:122 No buffers
> available for ISP Output0
> [0:22:35.004862560] [1058] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA (
> 6 left)
> [0:22:35.037365065] [1058] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA (
> 5 left)
> [0:22:35.070214287] [1058] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA (
> 4 left)
> [0:22:35.104224183] [1058] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA (
> 3 left)
> [0:22:35.136588771] [1058] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA (
> 2 left)
> [0:22:35.169885857] [1058] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA (
> 1 left)
> [0:22:35.204680594] [1058] INFO RPI raspberrypi.cpp:1635 Dropping frame
> at the request of the IPA (
> 0 left)
> ERROR: from element
> /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0: Internal data
> stream error.
> Additional debug info:
> ../src/gstreamer/gstlibcamerasrc.cpp(323): gst_libcamera_src_task_run
> (): /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0:
> streaming stopped, reason not-negotiated (-4)
> Execution ended after 0:00:00.358389578
> Setting pipeline to PAUSED ...
> Setting pipeline to READY ...
> Setting pipeline to NULL ...
> Freeing pipeline ...
>
More information about the libcamera-devel
mailing list