XISP : Creating new Pipeline Handler for AMD/Xilinx HLS-based ISP

Tue Jan 14 15:55:03 CET 2025

Hi

2025. január 14., kedd 15:43 keltezéssel, Mario Bergeron <grouby177 at gmail.com> írta:

> Hello,
> 
> I have a design with four (4) capture pipelines implemented in the PL of a Zynq-UltraScale+ device. The capture pipelines include the following components:
> - CAM : imx219, imx708, imx500, imx477
> - CSI : MIPI CSI-2 receiver
> - XISP : HLS-based ISP
> - VPSS : for color space conversion and image resizing
> - VCAP : Video Frame Buffer Write (DMA)
> 
> These come up with the following names:
> 
> // Additional context to what is being searched
> // driver = "xilinx-video"
> //
> // Capture Pipeline 0
> // V4l2VideoDevice = "vcap_mipi_0_v_proc output 0"
> // V4l2Subdevice = "imx219 1-0010"
> // V4l2Subdevice = "80050000.mipi_csi2_rx_subsystem"
> // V4l2Subdevice = "a0010000.ISPPipeline_accel"
> // V4l2Subdevice = "a0040000.v_proc_ss"
> //
> // Capture Pipeline 1
> // V4l2VideoDevice = "vcap_mipi_1_v_proc output 0"
> // V4l2Subdevice = "imx708"
> // V4l2Subdevice = "dw9807 2-000c"
> // V4l2Subdevice = "80051000.mipi_csi2_rx_subsystem"
> // V4l2Subdevice = "a0030000.ISPPipeline_accel"
> // V4l2Subdevice = "a00c0000.v_proc_ss"
> //
> // Capture Pipeline 2
> // V4l2VideoDevice = "vcap_mipi_2_v_proc output 0"
> // V4l2Subdevice = "imx500 3-001a"
> // V4l2Subdevice = "80052000.mipi_csi2_rx_subsystem"
> // V4l2Subdevice = "a0090000.ISPPipeline_accel"
> // V4l2Subdevice = "a0100000.v_proc_ss"
> //
> // Capture Pipeline 3
> // V4l2VideoDevice = "vcap_mipi_3_v_proc output 0"
> // V4l2Subdevice = "imx477 4-001a"
> // V4l2Subdevice = "80053000.mipi_csi2_rx_subsystem"
> // V4l2Subdevice = "a00b0000.ISPPipeline_accel"
> // V4l2Subdevice = "a0180000.v_proc_ss"
> 
> I am able to match the V4L2VideoDevice and V4L2Subdevices during "match" phase, and enumerate my camera pipelines:
> 
> # LIBCAMERA_LOG_LEVELS=XISP:0 cam -l
> [43:24:27.675100583] [5666] INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/home/root/libcamera.xisp_take01/build/src/ipa' to the IPA search path
> [43:24:27.675839535] [5666] WARN IPAManager ipa_manager.cpp:148 No IPA found in '/usr/local/lib/libcamera'
> [43:24:27.675941856] [5666] INFO Camera camera_manager.cpp:327 libcamera v0.4.0+19-91de5502-dirty (2025-01-14T14:36:27UTC)
> [43:24:27.745595071] [5667] DEBUG XISP xisp.cpp:1153 [PipelineHandlerXISP::match] Looking for capture pipeline
> [43:24:27.745819631] [5667] DEBUG XISP xisp.cpp:1234 VCAP : vcap_mipi_0_v_proc output 0
> [43:24:27.745914892] [5667] DEBUG XISP xisp.cpp:1221 CSI : 80050000.mipi_csi2_rx_subsystem
> [43:24:27.745987572] [5667] DEBUG XISP xisp.cpp:1185 CAM : imx219 1-0010
> [43:24:27.746046542] [5667] DEBUG XISP xisp.cpp:1190 IMX219 : 1920x1080-SRGGB10_1X10
> [43:24:27.746110092] [5667] DEBUG XISP xisp.cpp:1226 XISP : a0010000.ISPPipeline_accel
> [43:24:27.746171352] [5667] DEBUG XISP xisp.cpp:1230 VPSS : a0040000.v_proc_ss
> [43:24:27.755114607] [5667] DEBUG XISP xisp.cpp:1153 [PipelineHandlerXISP::match] Looking for capture pipeline
> [43:24:27.755278268] [5667] DEBUG XISP xisp.cpp:1234 VCAP : vcap_mipi_1_v_proc output 0
> [43:24:27.755350048] [5667] DEBUG XISP xisp.cpp:1221 CSI : 80051000.mipi_csi2_rx_subsystem
> [43:24:27.755412538] [5667] DEBUG XISP xisp.cpp:1185 CAM : imx708
> [43:24:27.755471358] [5667] DEBUG XISP xisp.cpp:1197 IMX708 : 1536x864-SRGGB10_1X10
> [43:24:27.755533129] [5667] DEBUG XISP xisp.cpp:1217 VCM : dw9807 2-000c
> [43:24:27.755596879] [5667] DEBUG XISP xisp.cpp:1226 XISP : a0030000.ISPPipeline_accel
> [43:24:27.755657829] [5667] DEBUG XISP xisp.cpp:1230 VPSS : a00c0000.v_proc_ss
> [43:24:27.765771027] [5667] DEBUG XISP xisp.cpp:1153 [PipelineHandlerXISP::match] Looking for capture pipeline
> [43:24:27.765926408] [5667] DEBUG XISP xisp.cpp:1234 VCAP : vcap_mipi_2_v_proc output 0
> [43:24:27.765996008] [5667] DEBUG XISP xisp.cpp:1185 CAM : imx500 3-001a
> [43:24:27.766056888] [5667] DEBUG XISP xisp.cpp:1211 IMX500 : 2028x1520-SRGGB10_1X10
> [43:24:27.766120928] [5667] DEBUG XISP xisp.cpp:1221 CSI : 80052000.mipi_csi2_rx_subsystem
> [43:24:27.766183828] [5667] DEBUG XISP xisp.cpp:1226 XISP : a0090000.ISPPipeline_accel
> [43:24:27.766244829] [5667] DEBUG XISP xisp.cpp:1230 VPSS : a0100000.v_proc_ss
> [43:24:27.773600649] [5667] WARN CameraSensorProperties camera_sensor_properties.cpp:463 No static properties available for 'imx500'
> [43:24:27.773693479] [5667] WARN CameraSensorProperties camera_sensor_properties.cpp:465 Please consider updating the camera sensor properties database
> [43:24:27.773874120] [5667] DEBUG XISP xisp.cpp:1153 [PipelineHandlerXISP::match] Looking for capture pipeline
> [43:24:27.774010170] [5667] DEBUG XISP xisp.cpp:1234 VCAP : vcap_mipi_3_v_proc output 0
> [43:24:27.774081241] [5667] DEBUG XISP xisp.cpp:1221 CSI : 80053000.mipi_csi2_rx_subsystem
> [43:24:27.774146091] [5667] DEBUG XISP xisp.cpp:1185 CAM : imx477 4-001a
> [43:24:27.774205011] [5667] DEBUG XISP xisp.cpp:1204 IMX477 : 1332x990-SRGGB10_1X10
> [43:24:27.774268071] [5667] DEBUG XISP xisp.cpp:1226 XISP : a00b0000.ISPPipeline_accel
> [43:24:27.774331881] [5667] DEBUG XISP xisp.cpp:1230 VPSS : a0180000.v_proc_ss
> [43:24:27.782596404] [5667] DEBUG XISP xisp.cpp:1153 [PipelineHandlerXISP::match] Looking for capture pipeline
> Available cameras:
> 1: 'imx219' (/base/pl-bus/i2c at 80040000/sensor at 10)
> 2: 'imx708' (/base/pl-bus/i2c at 80070000/sensor at 10)
> 3: 'imx500' (/base/pl-bus/i2c at 80090000/sensor at 1a)
> 4: 'imx477' (/base/pl-bus/i2c at 800b0000/sensor at 1a)
> 
> I have unexpected "Defer due to missing dependencies" messages, which is the V4L2VideoDevice entity:
> 
> ...
> [32:28:31.627641623] [4979] DEBUG Camera camera_manager.cpp:73 Starting camera manager
> [32:28:31.678492579] [4979] DEBUG DeviceEnumerator device_enumerator.cpp:242 New media device "xilinx-video" created from /dev/media0
> [32:28:31.681069857] [4979] DEBUG DeviceEnumerator device_enumerator_udev.cpp:96 Defer media device /dev/media0 due to 1 missing dependencies
> [32:28:31.681175337] [4979] DEBUG DeviceEnumerator device_enumerator_udev.cpp:105 Missing dependency = vcap_mipi_0_v_proc output 0
> [32:28:31.682177024] [4979] DEBUG DeviceEnumerator device_enumerator_udev.cpp:330 All dependencies for media device /dev/media0 found
> [32:28:31.682256265] [4979] DEBUG DeviceEnumerator device_enumerator.cpp:270 Added device /dev/media0: xilinx-video
> [32:28:31.682849299] [4979] DEBUG DeviceEnumerator device_enumerator.cpp:242 New media device "xilinx-video" created from /dev/media1
> [32:28:31.686341484] [4979] DEBUG DeviceEnumerator device_enumerator_udev.cpp:96 Defer media device /dev/media1 due to 1 missing dependencies
> [32:28:31.686433004] [4979] DEBUG DeviceEnumerator device_enumerator_udev.cpp:105 Missing dependency = vcap_mipi_1_v_proc output 0
> [32:28:31.687418801] [4979] DEBUG DeviceEnumerator device_enumerator_udev.cpp:330 All dependencies for media device /dev/media1 found
> [32:28:31.687493362] [4979] DEBUG DeviceEnumerator device_enumerator.cpp:270 Added device /dev/media1: xilinx-video
> [32:28:31.688111086] [4979] DEBUG DeviceEnumerator device_enumerator.cpp:242 New media device "xilinx-video" created from /dev/media2
> [32:28:31.690556563] [4979] DEBUG DeviceEnumerator device_enumerator_udev.cpp:96 Defer media device /dev/media2 due to 1 missing dependencies
> [32:28:31.690670334] [4979] DEBUG DeviceEnumerator device_enumerator_udev.cpp:105 Missing dependency = vcap_mipi_2_v_proc output 0
> [32:28:31.691687431] [4979] DEBUG DeviceEnumerator device_enumerator_udev.cpp:330 All dependencies for media device /dev/media2 found
> [32:28:31.691797452] [4979] DEBUG DeviceEnumerator device_enumerator.cpp:270 Added device /dev/media2: xilinx-video
> [32:28:31.692409806] [4979] DEBUG DeviceEnumerator device_enumerator.cpp:242 New media device "xilinx-video" created from /dev/media3
> [32:28:31.694842583] [4979] DEBUG DeviceEnumerator device_enumerator_udev.cpp:96 Defer media device /dev/media3 due to 1 missing dependencies
> [32:28:31.694955104] [4979] DEBUG DeviceEnumerator device_enumerator_udev.cpp:105 Missing dependency = vcap_mipi_3_v_proc output 0
> [32:28:31.695980181] [4979] DEBUG DeviceEnumerator device_enumerator_udev.cpp:330 All dependencies for media device /dev/media3 found
> [32:28:31.696112512] [4979] DEBUG DeviceEnumerator device_enumerator.cpp:270 Added device /dev/media3: xilinx-video
> [32:28:31.697657323] [4979] DEBUG Camera camera_manager.cpp:140 Found registered pipeline handler 'xisp'
> ...

See the messages after: "All dependencies for media device /dev/mediaX found".
So it seems to me that everything is in order, four media devices successfully discovered.

> 
> I also get these unexpected messages:
> 
> ...
> [32:28:31.709995969] [4979] DEBUG CameraSensor camera_sensor_raw.cpp:210 imx219 1-0010: unsupported number of sinks (0) or sources (2)
> ...
> [32:28:31.728432108] [4979] DEBUG CameraSensor camera_sensor_raw.cpp:210 imx708: unsupported number of sinks (0) or sources (2)
> ...
> [32:28:31.746255953] [4979] DEBUG CameraSensor camera_sensor_raw.cpp:210 imx500 3-001a: unsupported number of sinks (0) or sources (2)
> [32:28:31.750244771] [4979] WARN CameraSensorProperties camera_sensor_properties.cpp:463 No static properties available for 'imx500'
> [32:28:31.750356602] [4979] WARN CameraSensorProperties camera_sensor_properties.cpp:465 Please consider updating the camera sensor properties database
> ...
> [32:28:31.762326825] [4979] DEBUG CameraSensor camera_sensor_raw.cpp:210 imx477 4-001a: unsupported number of sinks (0) or sources (2)
> ...
> 
> 
> I need to know if the previous messages are normal, or need to be resolved.
> 
> 
> When I try to capture images, I can see the "generateConfigration" and "validate" phases being run, but get a "Failed to Acquire" error before the "configure" phase ...
> 
> # LIBCAMERA_LOG_LEVELS=XISP:0 cam --camera /base/pl-bus/i2c at 80040000/sensor at 10 -c1
> ...
> [43:26:44.685016474] [5673] DEBUG XISP xisp.cpp:1153 [PipelineHandlerXISP::match] Looking for capture pipeline
> ...
> [43:26:44.723683858] [5672] DEBUG XISP xisp.cpp:949 [PipelineHandlerXISP::generateConfiguration] Generate Configuration
> [43:26:44.723784408] [5672] DEBUG XISP xisp.cpp:634 [PipelineHandlerXISP::validate] Validating Configuration
> [43:26:44.723851918] [5672] DEBUG XISP xisp.cpp:678 Computed sensor configuration: 1920x1080-UYVY8_1X16
> [43:26:44.723925599] [5672] DEBUG XISP xisp.cpp:730 Selected sensor format: 1920x1080-UYVY8_1X16
> [43:26:44.724095059] [5672] DEBUG XISP xisp.cpp:634 [PipelineHandlerXISP::validate] Validating Configuration
> [43:26:44.724171159] [5672] DEBUG XISP xisp.cpp:678 Computed sensor configuration: 1920x1080-UYVY8_1X16
> [43:26:44.724235059] [5672] DEBUG XISP xisp.cpp:730 Selected sensor format: 1920x1080-UYVY8_1X16
> Using camera /base/pl-bus/i2c at 80040000/sensor at 10 as cam0
> [43:26:44.724404510] [5672] ERROR Camera camera.cpp:690 Camera in Acquired state trying acquire() requiring state Available
> Failed to acquire camera 1
> Failed to create camera session

The camera "--camera /base/pl-bus/i2c at 80040000/sensor at 10" and "-c1" arguments designate
the same camera. As the error says "Camera in Acquired state" because it has
been already acquired: "Using camera /base/pl-bus/i2c at 80040000/sensor at 10 as cam0".
What exactly are you trying to achieve here?

Regards,
Barnabás Pőcze

> 
> 
> I have not modified the source code for the cam application.
> I built the libcamera repo as follows:
> 
> # meson setup build --buildtype=release -Dpipelines=xisp -Dv4l2=true -Dgstreamer=enabled -Dtest=false -Dlc-compliance=disabled -Dcam=enabled -Dqcam=disabled -Ddocumentation=disabled -Dpycamera=enabled
> # ninja -C build install
> 
> Then launch the application as follows:
> 
> # meson devenv -C build
> # cam ...
> # LIBCAMERA_LOG_LEVELS=*:0 cam ...
> 
> Any help and/or collaboration would be greatly appreciated :)
> 
> Cheers !
> 
> Mario Bergeron (AlbertaBeef)