[libcamera-devel] gstreamer libcamerasrc negotiation woes

[Thorsten von Eicken]

I'm trying to get libcamerasrc to work on an rPi with the standard V2 
camera module (imx219) and I have some behavior I cannot explain. I'm 
trying to stream h264 from the camera to rtsp, but am starting using 
just a filesink for testing. After toying around, the following 
commandline works but I don't understand why the v4l2convert is necessary:

gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=640,height=480' ! 
v4l2convert ! v4l2h264enc ! filesink location=test.h264

Looking at the debug output (reproduced at the end of this message) 
libcamerasrc ends up outputting video/x-raw in NV21 format, then 
v4l2convert converts it to YUY2, and then v4l2h264enc encodes to h.264.

But looking at `gst-inspect-1.0 v4l2h264enc` I see that v4l2h264enc 
accepts NV21:

   SINK template: 'sink'
     Availability: Always
     Capabilities:
       video/x-raw
                  format: { (string)I420, (string)YV12, (string)NV12, 
(string)NV21, (string)RGB16, (s
tring)RGB, (string)BGR, (string)BGRx, (string)BGRA, (string)YUY2, 
(string)YVYU, (string)UYVY }
                   width: [ 1, 32768 ]
                  height: [ 1, 32768 ]
               framerate: [ 0/1, 2147483647/1 ]

So I should be able to do without the v4l2convert, right? Removing it 
results in failure:

ERROR: from element 
/GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0: Internal data 
stream error.
Additional debug info:
../src/gstreamer/gstlibcamerasrc.cpp(323): gst_libcamera_src_task_run 
(): /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0:streaming 
stopped, reason not-negotiated (-4)

even if I change the caps filter to add a "format=NV21" I get the same 
result. What is going on?

NB: why do I always get the following critical error that seems 
completely benign when I use libcamerasrc?

** (gst-launch-1.0:907): CRITICAL **: 10:09:51.212: 
gst_video_format_to_string: assertion 'format !=
  GST_VIDEO_FORMAT_UNKNOWN' failed

Setup:

- rPi3B+
- Linux rpi3 5.10.60-v7+ #1449 SMP Wed Aug 25 15:00:01 BST 2021 armv7l 
GNU/Linux
- libcamera compiled from source (#4cf7a8f)

- /boot/config.txt:
dtoverlay=vc4-fkms-v3d
dtoverlay=imx219
gpu_mem=128
start_x=1
max_framebuffers=0

---------------------------

Log from working gst-launch:

tve at rpi3 ~> gst-launch-1.0 -vvv libcamerasrc ! 
'video/x-raw,width=640,height=480' ! v4l2con[120/410]
l2h264enc ! filesink location=test.h264
Setting pipeline to PAUSED ...
[0:04:22.812479146] [907]  INFO Camera camera_manager.cpp:294 libcamera 
v0.0.0+3032-4cf7a8fc
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

** (gst-launch-1.0:907): CRITICAL **: 10:09:51.212: 
gst_video_format_to_string: assertion 'format !=
  GST_VIDEO_FORMAT_UNKNOWN' failed
/GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0.GstLibcameraPad:src: 
caps = video/x-raw, format
=(string)NV21, width=(int)640, height=(int)480
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = 
video/x-raw, format=(string)NV21
, width=(int)640, height=(int)480
/GstPipeline:pipeline0/v4l2convert:v4l2convert0.GstPad:src: caps = 
video/x-raw, format=(string)YUY2,
  width=(int)640, height=(int)480, framerate=(fraction)0/1, 
interlace-mode=(string)progressive, color
imetry=(string)bt709
/GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0.GstPad:src: caps = 
video/x-h264, stream-format=(stri
ng)byte-stream, alignment=(string)au, profile=(string)high, 
level=(string)4, width=(int)640, height=
(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1, 
interlace-mode=(string)progress
ive, colorimetry=(string)bt709
/GstPipeline:pipeline0/GstFileSink:filesink0.GstPad:sink: caps = 
video/x-h264, stream-format=(string
)byte-stream, alignment=(string)au, profile=(string)high, 
level=(string)4, width=(int)640, height=(i
nt)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1, 
interlace-mode=(string)progressiv
e, colorimetry=(string)bt709
Redistribute latency...
/GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0.GstPad:sink: caps = 
video/x-raw, format=(string)YUY2
, width=(int)640, height=(int)480, framerate=(fraction)0/1, 
interlace-mode=(string)progressive, colo
rimetry=(string)bt709
/GstPipeline:pipeline0/v4l2convert:v4l2convert0.GstPad:sink: caps = 
video/x-raw, format=(string)NV21
, width=(int)640, height=(int)480
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = 
video/x-raw, format=(string)NV2
1, width=(int)640, height=(int)480
[0:04:23.273836423] [911]  INFO Camera camera.cpp:937 configuring 
streams: (0) 640x480-NV21
[0:04:23.274589971] [908]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 
fmt RG10 Score: 2662.49 (bes
t 2662.49)
[0:04:23.274793410] [908]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 
fmt RG10 Score: 2136.67 (bes
t 2136.67)
[0:04:23.274908515] [908]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 
fmt RG10 Score: 1944.49 (bes
t 1944.49)
[0:04:23.275018256] [908]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 
fmt RG10 Score: 1500 (best 150
0)
[0:04:23.275138725] [908]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 
fmt pRAA Score: 2162.49 (bes
t 1500)
[0:04:23.275250185] [908]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 
fmt pRAA Score: 1636.67 (bes
t 1500)
[0:04:23.275358884] [908]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 
fmt pRAA Score: 1444.49 (bes
t 1444.49)
[0:04:23.275468989] [908]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 
fmt pRAA Score: 1000 (best 100
0)
[0:04:23.275605500] [908]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 
fmt RGGB Score: 3662.49 (bes
t 1000)
[0:04:23.275722168] [908]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 
fmt RGGB Score: 3136.67 (bes
t 1000)
[0:04:23.275834669] [908]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 
fmt RGGB Score: 2944.49 (bes
t 1000)
[0:04:23.275943576] [908]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 
fmt RGGB Score: 2500 (best 100
0)
[0:04:23.276445872] [908]  INFO RPI raspberrypi.cpp:620 Sensor: 
/base/soc/i2c0mux/i2c at 1/imx219 at 10 -
Selected mode: 640x480-pRAA
[0:04:23.298140323] [908]  INFO RPISTREAM rpi_stream.cpp:122 No buffers 
available for ISP Output0
[0:04:23.298277980] [908]  INFO RPISTREAM rpi_stream.cpp:122 No buffers 
available for ISP Output0
[0:04:23.298366731] [908]  INFO RPISTREAM rpi_stream.cpp:122 No buffers 
available for ISP Output0
[0:04:23.358951818] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA (6
  left)
[0:04:23.391732142] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA (5
  left)
[0:04:23.424765802] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA (4
  left)
[0:04:23.456638254] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA ($
  left)
[0:04:23.489703946] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA (2
  left)
[0:04:23.523865480] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA (1
  left)
[0:04:23.555756161] [908]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA (0
  left)
^Chandling interrupt. {that's me hitting ctrl-c to stop the recording}
Interrupt: Stopping pipeline ...
Execution ended after 0:00:28.749092992
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

---------------------------

Log from failing gst-launch:

tve at rpi3 ~> gst-launch-1.0 -vvv libcamerasrc ! 
'video/x-raw,format=NV21,width=640,height=480' ! v4l2
h264enc ! filesink location=test.h264
Setting pipeline to PAUSED ...
[0:22:34.494879960] [1057]  INFO Camera camera_manager.cpp:294 libcamera 
v0.0.0+3032-4cf7a8fc
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

** (gst-launch-1.0:1057): CRITICAL **: 10:28:02.891: 
gst_video_format_to_string: assertion 'format !
= GST_VIDEO_FORMAT_UNKNOWN' failed
/GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0.GstLibcameraPad:src: 
caps = video/x-raw, format
=(string)NV21, width=(int)640, height=(int)480
[0:22:34.886945683] [1061]  INFO Camera camera.cpp:937 configuring 
streams: (0) 640x480-NV21
[0:22:34.887579223] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 
fmt RG10 Score: 2662.49 (be
st 2662.49)
[0:22:34.887769482] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 
fmt RG10 Score: 2136.67 (be
st 2136.67)
[0:22:34.887883284] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 
fmt RG10 Score: 1944.49 (be
st 1944.49)
[0:22:34.887994482] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 
fmt RG10 Score: 1500 (best 15
00)
[0:22:34.888113127] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 
fmt pRAA Score: 2162.49 (be
st 1500)
[0:22:34.888223439] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 
fmt pRAA Score: 1636.67 (be
st 1500)
[0:22:34.888332293] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 
fmt pRAA Score: 1444.49 (be
st 1444.49)
[0:22:34.888441668] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 
fmt pRAA Score: 1000 (best 10
00)
[0:22:34.888575886] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 
fmt RGGB Score: 3662.49 (be
st 1000)
[0:22:34.888684844] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 
fmt RGGB Score: 3136.67 (b$
st 1000)
[0:22:34.888790729] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 
fmt RGGB Score: 2944.49 (be
st 1000)
[0:22:34.888899531] [1058]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 
fmt RGGB Score: 2500 (best 10
00)
[0:22:34.889397602] [1058]  INFO RPI raspberrypi.cpp:620 Sensor: 
/base/soc/i2c0mux/i2c at 1/imx219 at 10 -
  Selected mode: 640x480-pRAA
[0:22:34.943958581] [1058]  INFO RPISTREAM rpi_stream.cpp:122 No buffers 
available for ISP Output0
[0:22:34.944094934] [1058]  INFO RPISTREAM rpi_stream.cpp:122 No buffers 
available for ISP Output0
[0:22:34.944183320] [1058]  INFO RPISTREAM rpi_stream.cpp:122 No buffers 
available for ISP Output0
[0:22:35.004862560] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA (
6 left)
[0:22:35.037365065] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA (
5 left)
[0:22:35.070214287] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA (
4 left)
[0:22:35.104224183] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA (
3 left)
[0:22:35.136588771] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA (
2 left)
[0:22:35.169885857] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA (
1 left)
[0:22:35.204680594] [1058]  INFO RPI raspberrypi.cpp:1635 Dropping frame 
at the request of the IPA (
0 left)
ERROR: from element 
/GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0: Internal data 
stream error.
Additional debug info:
../src/gstreamer/gstlibcamerasrc.cpp(323): gst_libcamera_src_task_run 
(): /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.358389578
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...