[libcamera-devel] gstreamer libcamerasrc negotiation woes

Tue Sep 21 14:51:56 CEST 2021

Le mardi 14 septembre 2021 à 17:31 +0000, Thorsten von Eicken a écrit :
> I'm trying to get libcamerasrc to work on an rPi with the standard V2 
> camera module (imx219) and I have some behavior I cannot explain. I'm 
> trying to stream h264 from the camera to rtsp, but am starting using 
> just a filesink for testing. After toying around, the following 
> commandline works but I don't understand why the v4l2convert is necessary:
> 
> gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=640,height=480' ! 
> v4l2convert ! v4l2h264enc ! filesink location=test.h264
> 
> Looking at the debug output (reproduced at the end of this message) 
> libcamerasrc ends up outputting video/x-raw in NV21 format, then 
> v4l2convert converts it to YUY2, and then v4l2h264enc encodes to h.264.
> 
> But looking at `gst-inspect-1.0 v4l2h264enc` I see that v4l2h264enc 
> accepts NV21:
> 
>    SINK template: 'sink'
>      Availability: Always
>      Capabilities:
>        video/x-raw
>                   format: { (string)I420, (string)YV12, (string)NV12, 
> (string)NV21, (string)RGB16, (s
> tring)RGB, (string)BGR, (string)BGRx, (string)BGRA, (string)YUY2, 
> (string)YVYU, (string)UYVY }
>                    width: [ 1, 32768 ]
>                   height: [ 1, 32768 ]
>                framerate: [ 0/1, 2147483647/1 ]
> 
> So I should be able to do without the v4l2convert, right? Removing it 
> results in failure:
> 
> ERROR: from element 
> /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0: Internal data 
> stream error.
> Additional debug info:
> ../src/gstreamer/gstlibcamerasrc.cpp(323): gst_libcamera_src_task_run 
> (): /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0:streaming 
> stopped, reason not-negotiated (-4)

Most drivers are multi-format, I remember we had a bug in gst, were we'd list
the raw format without setting the encoded format. Perhaps you are hitting this
issue and a update of GStreamer would fix it.

Other then that, there zillion of ways negotiation can fail in the late process,
since HW is not only about width/height/format. You can enable some debug
category to try and gather more information. Though, before we get into that, it
would seem that this discussion would make more sense of gstreamer-devel mailing
list.

Also, make sure that you encoder works by itself with NV21:

  videotestsrc ! video/x-raw,format=NV21 ! v4l2h264enc ! fakesink

> 
> even if I change the caps filter to add a "format=NV21" I get the same 
> result. What is going on?
> 
> NB: why do I always get the following critical error that seems 
> completely benign when I use libcamerasrc?
> 
> ** (gst-launch-1.0:907): CRITICAL **: 10:09:51.212: 
> gst_video_format_to_string: assertion 'format !=
>   GST_VIDEO_FORMAT_UNKNOWN' failed

This does not look benign to me. This error can be trapped in a debugger using
G_DEBUG=fatal_criticals env. A backtrace would give us a lot of information
about what went wrong. Assertions in GLib code are design for crash safety,
while sometimes the code will recover (failsafe) the results afterward are
mostly undefined.

> 
> Setup:
> 
> - rPi3B+
> - Linux rpi3 5.10.60-v7+ #1449 SMP Wed Aug 25 15:00:01 BST 2021 armv7l 
> GNU/Linux
> - libcamera compiled from source (#4cf7a8f)
> 
> - /boot/config.txt:
> dtoverlay=vc4-fkms-v3d
> dtoverlay=imx219
> gpu_mem=128
> start_x=1
> max_framebuffers=0
> 
> ---------------------------
> 
> Log from working gst-launch:
> 
> tve at rpi3 ~> gst-launch-1.0 -vvv libcamerasrc ! 
> 'video/x-raw,width=640,height=480' ! v4l2con[120/410]
> l2h264enc ! filesink location=test.h264
> Setting pipeline to PAUSED ...
> [0:04:22.812479146] [907]  INFO Camera camera_manager.cpp:294 libcamera 
> v0.0.0+3032-4cf7a8fc
> Pipeline is live and does not need PREROLL ...
> Setting pipeline to PLAYING ...
> New clock: GstSystemClock
> 
> ** (gst-launch-1.0:907): CRITICAL **: 10:09:51.212: 
> gst_video_format_to_string: assertion 'format !=
>   GST_VIDEO_FORMAT_UNKNOWN' failed
> /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0.GstLibcameraPad:src: 
> caps = video/x-raw, format
> =(string)NV21, width=(int)640, height=(int)480
> /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = 
> video/x-raw, format=(string)NV21
> , width=(int)640, height=(int)480
> /GstPipeline:pipeline0/v4l2convert:v4l2convert0.GstPad:src: caps = 
> video/x-raw, format=(string)YUY2,
>   width=(int)640, height=(int)480, framerate=(fraction)0/1, 
> interlace-mode=(string)progressive, color
> imetry=(string)bt709
> /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0.GstPad:src: caps = 
> video/x-h264, stream-format=(stri
> ng)byte-stream, alignment=(string)au, profile=(string)high, 
> level=(string)4, width=(int)640, height=
> (int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1, 
> interlace-mode=(string)progress
> ive, colorimetry=(string)bt709
> /GstPipeline:pipeline0/GstFileSink:filesink0.GstPad:sink: caps = 
> video/x-h264, stream-format=(string
> )byte-stream, alignment=(string)au, profile=(string)high, 
> level=(string)4, width=(int)640, height=(i
> nt)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1, 
> interlace-mode=(string)progressiv
> e, colorimetry=(string)bt709
> Redistribute latency...
> /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0.GstPad:sink: caps = 
> video/x-raw, format=(string)YUY2
> , width=(int)640, height=(int)480, framerate=(fraction)0/1, 
> interlace-mode=(string)progressive, colo
> rimetry=(string)bt709
> /GstPipeline:pipeline0/v4l2convert:v4l2convert0.GstPad:sink: caps = 
> video/x-raw, format=(string)NV21
> , width=(int)640, height=(int)480
> /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = 
> video/x-raw, format=(string)NV2
> 1, width=(int)640, height=(int)480
> [0:04:23.273836423] [911]  INFO Camera camera.cpp:937 configuring 
> streams: (0) 640x480-NV21
> [0:04:23.274589971] [908]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 
> fmt RG10 Score: 2662.49 (bes
> t 2662.49)
> [0:04:23.274793410] [908]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 
> fmt RG10 Score: 2136.67 (bes
> t 2136.67)
> [0:04:23.274908515] [908]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 
> fmt RG10 Score: 1944.49 (bes
> t 1944.49)
> [0:04:23.275018256] [908]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 
> fmt RG10 Score: 1500 (best 150
> 0)
> [0:04:23.275138725] [908]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 
> fmt pRAA Score: 2162.49 (bes
> t 1500)
> [0:04:23.275250185] [908]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 
> fmt pRAA Score: 1636.67 (bes
> t 1500)
> [0:04:23.275358884] [908]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 
> fmt pRAA Score: 1444.49 (bes
> t 1444.49)
> [0:04:23.275468989] [908]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 
> fmt pRAA Score: 1000 (best 100
> 0)
> [0:04:23.275605500] [908]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 
> fmt RGGB Score: 3662.49 (bes
> t 1000)
> [0:04:23.275722168] [908]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 
> fmt RGGB Score: 3136.67 (bes
> t 1000)
> [0:04:23.275834669] [908]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 
> fmt RGGB Score: 2944.49 (bes
> t 1000)
> [0:04:23.275943576] [908]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 
> fmt RGGB Score: 2500 (best 100
> 0)
> [0:04:23.276445872] [908]  INFO RPI raspberrypi.cpp:620 Sensor: 
> /base/soc/i2c0mux/i2c at 1/imx219 at 10 -
> Selected mode: 640x480-pRAA
> [0:04:23.298140323] [908]  INFO RPISTREAM rpi_stream.cpp:122 No buffers 
> available for ISP Output0
> [0:04:23.298277980] [908]  INFO RPISTREAM rpi_stream.cpp:122 No buffers 
> available for ISP Output0
> [0:04:23.298366731] [908]  INFO RPISTREAM rpi_stream.cpp:122 No buffers 
> available for ISP Output0
> [0:04:23.358951818] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA (6
>   left)
> [0:04:23.391732142] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA (5
>   left)
> [0:04:23.424765802] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA (4
>   left)
> [0:04:23.456638254] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA ($
>   left)
> [0:04:23.489703946] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA (2
>   left)
> [0:04:23.523865480] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA (1
>   left)
> [0:04:23.555756161] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA (0
>   left)
> ^Chandling interrupt. {that's me hitting ctrl-c to stop the recording}
> Interrupt: Stopping pipeline ...
> Execution ended after 0:00:28.749092992
> Setting pipeline to PAUSED ...
> Setting pipeline to READY ...
> Setting pipeline to NULL ...
> Freeing pipeline ...
> 
> ---------------------------
> 
> Log from failing gst-launch:
> 
> tve at rpi3 ~> gst-launch-1.0 -vvv libcamerasrc ! 
> 'video/x-raw,format=NV21,width=640,height=480' ! v4l2
> h264enc ! filesink location=test.h264
> Setting pipeline to PAUSED ...
> [0:22:34.494879960] [1057]  INFO Camera camera_manager.cpp:294 libcamera 
> v0.0.0+3032-4cf7a8fc
> Pipeline is live and does not need PREROLL ...
> Setting pipeline to PLAYING ...
> New clock: GstSystemClock
> 
> ** (gst-launch-1.0:1057): CRITICAL **: 10:28:02.891: 
> gst_video_format_to_string: assertion 'format !
> = GST_VIDEO_FORMAT_UNKNOWN' failed
> /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0.GstLibcameraPad:src: 
> caps = video/x-raw, format
> =(string)NV21, width=(int)640, height=(int)480
> [0:22:34.886945683] [1061]  INFO Camera camera.cpp:937 configuring 
> streams: (0) 640x480-NV21
> [0:22:34.887579223] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 
> fmt RG10 Score: 2662.49 (be
> st 2662.49)
> [0:22:34.887769482] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 
> fmt RG10 Score: 2136.67 (be
> st 2136.67)
> [0:22:34.887883284] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 
> fmt RG10 Score: 1944.49 (be
> st 1944.49)
> [0:22:34.887994482] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 
> fmt RG10 Score: 1500 (best 15
> 00)
> [0:22:34.888113127] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 
> fmt pRAA Score: 2162.49 (be
> st 1500)
> [0:22:34.888223439] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 
> fmt pRAA Score: 1636.67 (be
> st 1500)
> [0:22:34.888332293] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 
> fmt pRAA Score: 1444.49 (be
> st 1444.49)
> [0:22:34.888441668] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 
> fmt pRAA Score: 1000 (best 10
> 00)
> [0:22:34.888575886] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 
> fmt RGGB Score: 3662.49 (be
> st 1000)
> [0:22:34.888684844] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 
> fmt RGGB Score: 3136.67 (b$
> st 1000)
> [0:22:34.888790729] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 
> fmt RGGB Score: 2944.49 (be
> st 1000)
> [0:22:34.888899531] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 
> fmt RGGB Score: 2500 (best 10
> 00)
> [0:22:34.889397602] [1058]  INFO RPI raspberrypi.cpp:620 Sensor: 
> /base/soc/i2c0mux/i2c at 1/imx219 at 10 -
>   Selected mode: 640x480-pRAA
> [0:22:34.943958581] [1058]  INFO RPISTREAM rpi_stream.cpp:122 No buffers 
> available for ISP Output0
> [0:22:34.944094934] [1058]  INFO RPISTREAM rpi_stream.cpp:122 No buffers 
> available for ISP Output0
> [0:22:34.944183320] [1058]  INFO RPISTREAM rpi_stream.cpp:122 No buffers 
> available for ISP Output0
> [0:22:35.004862560] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA (
> 6 left)
> [0:22:35.037365065] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA (
> 5 left)
> [0:22:35.070214287] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA (
> 4 left)
> [0:22:35.104224183] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA (
> 3 left)
> [0:22:35.136588771] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA (
> 2 left)
> [0:22:35.169885857] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA (
> 1 left)
> [0:22:35.204680594] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
> at the request of the IPA (
> 0 left)
> ERROR: from element 
> /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0: Internal data 
> stream error.
> Additional debug info:
> ../src/gstreamer/gstlibcamerasrc.cpp(323): gst_libcamera_src_task_run 
> (): /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0:
> streaming stopped, reason not-negotiated (-4)
> Execution ended after 0:00:00.358389578
> Setting pipeline to PAUSED ...
> Setting pipeline to READY ...
> Setting pipeline to NULL ...
> Freeing pipeline ...
> 