Gstreamer assertion failed after a short period when running two cameras simultaneously

Fri Oct 18 03:44:37 CEST 2024

 Hi Nicolas,Thanks for the reply.
First of all,
> Note thats libcamera project track bufs like this one at https://bugs.libcamera.org/
Sorry for  the disturbance, I'll open a ticket at the proper bugzilla site.
> This assertion is what needs to be debugged. Preferably, it would be nice to take this as an opportunity to turn this assertion into a crash safe format (see g_return_if* glib assertions). The best way to debug GLib assertion is to ask GLib to abort on it. This can be done using environment variable:>
>  G_DEBUG=fatal_criticals
>
> From there, you will be able to check the backtrace of the current and other threads running, and also inspect local variables state. This information will be key in understand what is going on, and what could be the fix. Please try and provide similar information. 
I'll try it and post relevant information at the Bugzilla ticket.

Best regards,Ewing Kang2024.10.18

    在 2024年10月17日 星期四 下午09:05:06 [GMT+8]， Nicolas Dufresne<nicolas at ndufresne.ca> 寫道：  

 Hi,
Le vendredi 11 octobre 2024 à 03:10 +0000, 康 心奕 a écrit :
Hello, everyone
This is a re-post of the issue at the Raspberry PI branch: https://github.com/raspberrypi/libcamera/issues/186?notification_referrer_id=NT_kwDOAH7EvLMxMjc0NDgwODE0NTo4MzA3OTAw. The admin redirected me here. I'd really appreciate it if anyone could point me some directions on how to resolve or further debug the issue. Thanks .

Symptom
I'm experiencing the following assertion fail error when running two RPi cameras simultaneously.

void GstLibcameraSrcState::requestCompleted(libcamera::Request*): assertion 'wrap->request_.get() == request' failed 

This assertion is what needs to be debugged. Preferably, it would be nice to take this as an opportunity to turn this assertion into a crash safe format (see g_return_if* glib assertions). The best way to debug GLib assertion is to ask GLib to abort on it. This can be done using environment variable:

  G_DEBUG=fatal_criticals

>From there, you will be able to check the backtrace of the current and other threads running, and also inspect local variables state. This information will be key in understand what is going on, and what could be the fix. Please try and provide similar information. Note thats libcamera project track bufs like this one at https://bugs.libcamera.org/

Some additional tests:

   - Setting both cameras down to 5 FPS can avoid the problem.
   - No problem at any FPS if we run one camera only

Spec/ Platform

   - Hardware: Raspberry PI-5 4GB/ 8GB
   - Camera: Camera module V2

If someone from RPi could provide me a second module, I would then be able to reproduce.
cheers,Nicolas

   - OS: Bookworm (Linux gmpi6 6.6.51+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.51-1+rpt2 (2024-10-01) aarch64 GNU/Linux)
   - Version: libcamera0.3/stable,now 0.3.2+rpt20240927-1 arm64
   - Camera debug file with camera-bug-report: bug-report.txt

Detail

I have two IMX219 (Pi camera V2) connected to two Raspberry Pi-5. I'm launching the two cameras with the gstreamer element libcamerasrc:
gst-launch-1.0 -v libcamerasrc camera-name="/base/axi/pcie at 120000/rp1/i2c at 80000/imx219 at 10" ! video/x-raw,colorimetry=bt709,format=NV12,width=1664,height=1232,framerate=10/1 ! x264enc tune=zerolatency speed-preset=fast pass=qual quantizer=23 bitrate=16384 ! video/x-h264,profile=high,stream-format=avc ! rtph264pay pt=96 ! udpsink host=$IP port=8600
gst-launch-1.0 -v libcamerasrc camera-name="/base/axi/pcie at 120000/rp1/i2c at 88000/imx219 at 10" ! video/x-raw,colorimetry=bt709,format=NV12,width=1664,height=1232,framerate=10/1 ! x264enc tune=zerolatency speed-preset=fast pass=qual quantizer=23 bitrate=16384 ! video/x-h264,profile=high,stream-format=avc ! rtph264pay pt=96 ! udpsink host=$IP port=8601
For a short period, I can receive streaming from both cameras on the client side. However, the first launched camera will always hangs after a few 10's of seconds with following error:
0:01:37.9 / 99:99:99.
** (gst-launch-1.0:4162): CRITICAL **: 15:26:46.151: void GstLibcameraSrcState::requestCompleted(libcamera::Request*): assertion 'wrap->request_.get() == request' failed
Caught SIGSEGV
#0  0x00007fffbc08ba58 in gst_pad_query_default at plt () from /lib/aarch64-linux-gnu/gstreamer-1.0/libgstrtp.so
#1  0x00007fffbf11bb9c in gst_pad_query () from /lib/aarch64-linux-gnu/libgstreamer-1.0.so.0
#2  0x00007fffbf11c36c in gst_pad_peer_query () from /lib/aarch64-linux-gnu/libgstreamer-1.0.so.0
#3  0x00007fffbe248d8c in ?? () from /lib/aarch64-linux-gnu/libgstbase-1.0.so.0
#4  0x00007fffbf0f0b98 in gst_element_query () from /lib/aarch64-linux-gnu/libgstreamer-1.0.so.0
#5  0x00007fffbf0c2ee4 in ?? () from /lib/aarch64-linux-gnu/libgstreamer-1.0.so.0
#6  0x00007fffbf1050f8 in gst_iterator_fold () from /lib/aarch64-linux-gnu/libgstreamer-1.0.so.0
#7  0x00007fffbf0c2fc0 in ?? () from /lib/aarch64-linux-gnu/libgstreamer-1.0.so.0
#8  0x00007fffbf0c9140 in ?? () from /lib/aarch64-linux-gnu/libgstreamer-1.0.so.0
#9  0x00007fffbf0f0b98 in gst_element_query () from /lib/aarch64-linux-gnu/libgstreamer-1.0.so.0
#10 0x00007fffbf15d350 in gst_element_query_duration () from /lib/aarch64-linux-gnu/libgstreamer-1.0.so.0
#11 0x00005555922c3aa4 in ?? ()
#12 0x00007fffbef782ac in ?? () from /lib/aarch64-linux-gnu/libglib-2.0.so.0
#13 0x00007fffbef77614 in g_main_context_dispatch () from /lib/aarch64-linux-gnu/libglib-2.0.so.0
#14 0x00007fffbef779e0 in ?? () from /lib/aarch64-linux-gnu/libglib-2.0.so.0
#15 0x00007fffbef77d04 in g_main_loop_run () from /lib/aarch64-linux-gnu/libglib-2.0.so.0
#16 0x00005555922c60d4 in ?? ()
#17 0x00007fffbed07740 in __libc_start_call_main (main=main at entry=0x5555922c38c0, argc=argc at entry=22, argv=argv at entry=0x7fffe8059bc8) at ../sysdeps/nptl/libc_start_call_main.h:58
#18 0x00007fffbed07818 in __libc_start_main_impl (main=0x5555922c38c0, argc=22, argv=0x7fffe8059bc8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:360
#19 0x00005555922c3930 in ?? ()
Spinning.  Please run 'gdb gst-launch-1.0 4162' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.
^\Quit8.3 / 99:99:99.

Additional background info:

   - I intended to have the resolution of 1640x1232, which is the default binning mode for the IMX219. However, due to some presumably alignment issue, I can only set it to 1664x1232
   - I cannot use the rpicam-apps since at such low FPS, the 8-frame delay become a huge delay
   - I did enlarge the UDP memroy size beforehand by: sudo sysctl -w net.core.wmem_max=134217728

Best regards,Ewing Kang2024.10.11

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20241018/e1659742/attachment.htm>