XISP : Creating new Pipeline Handler for AMD/Xilinx HLS-based ISP
Mario Bergeron
grouby177 at gmail.com
Tue Jan 14 15:43:27 CET 2025
Hello,
I have a design with four (4) capture pipelines implemented in the PL of a
Zynq-UltraScale+ device. The capture pipelines include the following
components:
- CAM : imx219, imx708, imx500, imx477
- CSI : MIPI CSI-2 receiver
- XISP : HLS-based ISP
- VPSS : for color space conversion and image resizing
- VCAP : Video Frame Buffer Write (DMA)
These come up with the following names:
// Additional context to what is being searched
// driver = "xilinx-video"
//
// Capture Pipeline 0
// V4l2VideoDevice = "vcap_mipi_0_v_proc output 0"
// V4l2Subdevice = "imx219 1-0010"
// V4l2Subdevice = "80050000.mipi_csi2_rx_subsystem"
// V4l2Subdevice = "a0010000.ISPPipeline_accel"
// V4l2Subdevice = "a0040000.v_proc_ss"
//
// Capture Pipeline 1
// V4l2VideoDevice = "vcap_mipi_1_v_proc output 0"
// V4l2Subdevice = "imx708"
// V4l2Subdevice = "dw9807 2-000c"
// V4l2Subdevice = "80051000.mipi_csi2_rx_subsystem"
// V4l2Subdevice = "a0030000.ISPPipeline_accel"
// V4l2Subdevice = "a00c0000.v_proc_ss"
//
// Capture Pipeline 2
// V4l2VideoDevice = "vcap_mipi_2_v_proc output 0"
// V4l2Subdevice = "imx500 3-001a"
// V4l2Subdevice = "80052000.mipi_csi2_rx_subsystem"
// V4l2Subdevice = "a0090000.ISPPipeline_accel"
// V4l2Subdevice = "a0100000.v_proc_ss"
//
// Capture Pipeline 3
// V4l2VideoDevice = "vcap_mipi_3_v_proc output 0"
// V4l2Subdevice = "imx477 4-001a"
// V4l2Subdevice = "80053000.mipi_csi2_rx_subsystem"
// V4l2Subdevice = "a00b0000.ISPPipeline_accel"
// V4l2Subdevice = "a0180000.v_proc_ss"
I am able to match the V4L2VideoDevice and V4L2Subdevices during "match"
phase, and enumerate my camera pipelines:
# LIBCAMERA_LOG_LEVELS=XISP:0 cam -l
[43:24:27.675100583] [5666] INFO IPAManager ipa_manager.cpp:137 libcamera
is not installed. Adding '/home/root/libcamera.xisp_take01/build/src/ipa'
to the IPA search path
[43:24:27.675839535] [5666] WARN IPAManager ipa_manager.cpp:148 No IPA
found in '/usr/local/lib/libcamera'
[43:24:27.675941856] [5666] INFO Camera camera_manager.cpp:327 libcamera
v0.4.0+19-91de5502-dirty (2025-01-14T14:36:27UTC)
[43:24:27.745595071] [5667] DEBUG XISP xisp.cpp:1153
[PipelineHandlerXISP::match] Looking for capture pipeline
[43:24:27.745819631] [5667] DEBUG XISP xisp.cpp:1234 VCAP :
vcap_mipi_0_v_proc output 0
[43:24:27.745914892] [5667] DEBUG XISP xisp.cpp:1221 CSI :
80050000.mipi_csi2_rx_subsystem
[43:24:27.745987572] [5667] DEBUG XISP xisp.cpp:1185 CAM : imx219 1-0010
[43:24:27.746046542] [5667] DEBUG XISP xisp.cpp:1190 IMX219 :
1920x1080-SRGGB10_1X10
[43:24:27.746110092] [5667] DEBUG XISP xisp.cpp:1226 XISP :
a0010000.ISPPipeline_accel
[43:24:27.746171352] [5667] DEBUG XISP xisp.cpp:1230 VPSS :
a0040000.v_proc_ss
[43:24:27.755114607] [5667] DEBUG XISP xisp.cpp:1153
[PipelineHandlerXISP::match] Looking for capture pipeline
[43:24:27.755278268] [5667] DEBUG XISP xisp.cpp:1234 VCAP :
vcap_mipi_1_v_proc output 0
[43:24:27.755350048] [5667] DEBUG XISP xisp.cpp:1221 CSI :
80051000.mipi_csi2_rx_subsystem
[43:24:27.755412538] [5667] DEBUG XISP xisp.cpp:1185 CAM : imx708
[43:24:27.755471358] [5667] DEBUG XISP xisp.cpp:1197 IMX708 :
1536x864-SRGGB10_1X10
[43:24:27.755533129] [5667] DEBUG XISP xisp.cpp:1217 VCM : dw9807 2-000c
[43:24:27.755596879] [5667] DEBUG XISP xisp.cpp:1226 XISP :
a0030000.ISPPipeline_accel
[43:24:27.755657829] [5667] DEBUG XISP xisp.cpp:1230 VPSS :
a00c0000.v_proc_ss
[43:24:27.765771027] [5667] DEBUG XISP xisp.cpp:1153
[PipelineHandlerXISP::match] Looking for capture pipeline
[43:24:27.765926408] [5667] DEBUG XISP xisp.cpp:1234 VCAP :
vcap_mipi_2_v_proc output 0
[43:24:27.765996008] [5667] DEBUG XISP xisp.cpp:1185 CAM : imx500 3-001a
[43:24:27.766056888] [5667] DEBUG XISP xisp.cpp:1211 IMX500 :
2028x1520-SRGGB10_1X10
[43:24:27.766120928] [5667] DEBUG XISP xisp.cpp:1221 CSI :
80052000.mipi_csi2_rx_subsystem
[43:24:27.766183828] [5667] DEBUG XISP xisp.cpp:1226 XISP :
a0090000.ISPPipeline_accel
[43:24:27.766244829] [5667] DEBUG XISP xisp.cpp:1230 VPSS :
a0100000.v_proc_ss
[43:24:27.773600649] [5667] WARN CameraSensorProperties
camera_sensor_properties.cpp:463 No static properties available for 'imx500'
[43:24:27.773693479] [5667] WARN CameraSensorProperties
camera_sensor_properties.cpp:465 Please consider updating the camera sensor
properties database
[43:24:27.773874120] [5667] DEBUG XISP xisp.cpp:1153
[PipelineHandlerXISP::match] Looking for capture pipeline
[43:24:27.774010170] [5667] DEBUG XISP xisp.cpp:1234 VCAP :
vcap_mipi_3_v_proc output 0
[43:24:27.774081241] [5667] DEBUG XISP xisp.cpp:1221 CSI :
80053000.mipi_csi2_rx_subsystem
[43:24:27.774146091] [5667] DEBUG XISP xisp.cpp:1185 CAM : imx477 4-001a
[43:24:27.774205011] [5667] DEBUG XISP xisp.cpp:1204 IMX477 :
1332x990-SRGGB10_1X10
[43:24:27.774268071] [5667] DEBUG XISP xisp.cpp:1226 XISP :
a00b0000.ISPPipeline_accel
[43:24:27.774331881] [5667] DEBUG XISP xisp.cpp:1230 VPSS :
a0180000.v_proc_ss
[43:24:27.782596404] [5667] DEBUG XISP xisp.cpp:1153
[PipelineHandlerXISP::match] Looking for capture pipeline
Available cameras:
1: 'imx219' (/base/pl-bus/i2c at 80040000/sensor at 10)
2: 'imx708' (/base/pl-bus/i2c at 80070000/sensor at 10)
3: 'imx500' (/base/pl-bus/i2c at 80090000/sensor at 1a)
4: 'imx477' (/base/pl-bus/i2c at 800b0000/sensor at 1a)
I have unexpected "*Defer due to missing dependencies*" messages, which is
the V4L2VideoDevice entity:
...
[32:28:31.627641623] [4979] DEBUG Camera camera_manager.cpp:73 Starting
camera manager
[32:28:31.678492579] [4979] DEBUG DeviceEnumerator
device_enumerator.cpp:242 New media device "xilinx-video" created from
/dev/media0
[32:28:31.681069857] [4979] DEBUG DeviceEnumerator
device_enumerator_udev.cpp:96 Defer media device /dev/media0 due to 1
missing dependencies
[32:28:31.681175337] [4979] DEBUG DeviceEnumerator
device_enumerator_udev.cpp:105 *Missing dependency = vcap_mipi_0_v_proc
output 0*
[32:28:31.682177024] [4979] DEBUG DeviceEnumerator
device_enumerator_udev.cpp:330 All dependencies for media device
/dev/media0 found
[32:28:31.682256265] [4979] DEBUG DeviceEnumerator
device_enumerator.cpp:270 Added device /dev/media0: xilinx-video
[32:28:31.682849299] [4979] DEBUG DeviceEnumerator
device_enumerator.cpp:242 New media device "xilinx-video" created from
/dev/media1
[32:28:31.686341484] [4979] DEBUG DeviceEnumerator
device_enumerator_udev.cpp:96 Defer media device /dev/media1 due to 1
missing dependencies
[32:28:31.686433004] [4979] DEBUG DeviceEnumerator
device_enumerator_udev.cpp:105 * Missing dependency = vcap_mipi_1_v_proc
output 0*
[32:28:31.687418801] [4979] DEBUG DeviceEnumerator
device_enumerator_udev.cpp:330 All dependencies for media device
/dev/media1 found
[32:28:31.687493362] [4979] DEBUG DeviceEnumerator
device_enumerator.cpp:270 Added device /dev/media1: xilinx-video
[32:28:31.688111086] [4979] DEBUG DeviceEnumerator
device_enumerator.cpp:242 New media device "xilinx-video" created from
/dev/media2
[32:28:31.690556563] [4979] DEBUG DeviceEnumerator
device_enumerator_udev.cpp:96 Defer media device /dev/media2 due to 1
missing dependencies
[32:28:31.690670334] [4979] DEBUG DeviceEnumerator
device_enumerator_udev.cpp:105 *Missing dependency = vcap_mipi_2_v_proc
output 0*
[32:28:31.691687431] [4979] DEBUG DeviceEnumerator
device_enumerator_udev.cpp:330 All dependencies for media device
/dev/media2 found
[32:28:31.691797452] [4979] DEBUG DeviceEnumerator
device_enumerator.cpp:270 Added device /dev/media2: xilinx-video
[32:28:31.692409806] [4979] DEBUG DeviceEnumerator
device_enumerator.cpp:242 New media device "xilinx-video" created from
/dev/media3
[32:28:31.694842583] [4979] DEBUG DeviceEnumerator
device_enumerator_udev.cpp:96 Defer media device /dev/media3 due to 1
missing dependencies
[32:28:31.694955104] [4979] DEBUG DeviceEnumerator
device_enumerator_udev.cpp:105 *Missing dependency = vcap_mipi_3_v_proc
output 0*
[32:28:31.695980181] [4979] DEBUG DeviceEnumerator
device_enumerator_udev.cpp:330 All dependencies for media device
/dev/media3 found
[32:28:31.696112512] [4979] DEBUG DeviceEnumerator
device_enumerator.cpp:270 Added device /dev/media3: xilinx-video
[32:28:31.697657323] [4979] DEBUG Camera camera_manager.cpp:140 Found
registered pipeline handler 'xisp'
...
I also get these unexpected messages:
...
[32:28:31.709995969] [4979] DEBUG CameraSensor camera_sensor_raw.cpp:210
imx219 1-0010: *unsupported number of sinks (0) or sources (2)*
...
[32:28:31.728432108] [4979] DEBUG CameraSensor camera_sensor_raw.cpp:210
imx708: *unsupported number of sinks (0) or sources (2)*
...
[32:28:31.746255953] [4979] DEBUG CameraSensor camera_sensor_raw.cpp:210
imx500 3-001a:* unsupported number of sinks (0) or sources (2)*
[32:28:31.750244771] [4979] WARN CameraSensorProperties
camera_sensor_properties.cpp:463 No static properties available for 'imx500'
[32:28:31.750356602] [4979] WARN CameraSensorProperties
camera_sensor_properties.cpp:465 Please consider updating the camera sensor
properties database
...
[32:28:31.762326825] [4979] DEBUG CameraSensor camera_sensor_raw.cpp:210
imx477 4-001a: *unsupported number of sinks (0) or sources (2)*
...
I need to know if the previous messages are normal, or need to be resolved.
When I try to capture images, I can see the "generateConfigration" and
"validate" phases being run, but get a "*Failed to Acquire*" error before
the "configure" phase ...
# LIBCAMERA_LOG_LEVELS=XISP:0 cam --camera /base/pl-bus/i2c at 80040000
/sensor at 10 -c1
...
[43:26:44.685016474] [5673] DEBUG XISP xisp.cpp:1153
[PipelineHandlerXISP::match] Looking for capture pipeline
...
[43:26:44.723683858] [5672] DEBUG XISP xisp.cpp:949
[PipelineHandlerXISP::generateConfiguration] Generate Configuration
[43:26:44.723784408] [5672] DEBUG XISP xisp.cpp:634
[PipelineHandlerXISP::validate] Validating Configuration
[43:26:44.723851918] [5672] DEBUG XISP xisp.cpp:678 Computed sensor
configuration: 1920x1080-UYVY8_1X16
[43:26:44.723925599] [5672] DEBUG XISP xisp.cpp:730 Selected sensor format:
1920x1080-UYVY8_1X16
[43:26:44.724095059] [5672] DEBUG XISP xisp.cpp:634
[PipelineHandlerXISP::validate] Validating Configuration
[43:26:44.724171159] [5672] DEBUG XISP xisp.cpp:678 Computed sensor
configuration: 1920x1080-UYVY8_1X16
[43:26:44.724235059] [5672] DEBUG XISP xisp.cpp:730 Selected sensor format:
1920x1080-UYVY8_1X16
Using camera /base/pl-bus/i2c at 80040000/sensor at 10 as cam0
[43:26:44.724404510] [5672] ERROR Camera camera.cpp:690 Camera in Acquired
state trying acquire() requiring state Available
Failed to acquire camera 1
Failed to create camera session
I have not modified the source code for the cam application.
I built the libcamera repo as follows:
# meson setup build --buildtype=release -Dpipelines=xisp -Dv4l2=true
-Dgstreamer=enabled -Dtest=false -Dlc-compliance=disabled -Dcam=enabled
-Dqcam=disabled -Ddocumentation=disabled -Dpycamera=enabled
# ninja -C build install
Then launch the application as follows:
# meson devenv -C build
# cam ...
# LIBCAMERA_LOG_LEVELS=*:0 cam ...
Any help and/or collaboration would be greatly appreciated :)
Cheers !
Mario Bergeron (AlbertaBeef)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20250114/d8334feb/attachment.htm>
More information about the libcamera-devel
mailing list