[libcamera-devel] [RFC PATCH] utils: ipu3-process: Stream multiple frames using v4l2-ctl

[Umang Jain]

Hi Laurent,

On 7/12/22 03:45, Laurent Pinchart wrote:
> Hi Umang,
>
> Thank you for the patch.
>
> On Mon, Jul 11, 2022 at 10:14:13PM +0530, Umang Jain via libcamera-devel wrote:
>> Stream IMGU using v4l2-ctl as it provides more granular controls
>> on the video nodes. The goal here is to stream a single input
>> file containing multiple frames in 10-bit IPU3 packed bayer format.
>> Such a input file can be created using the ipu3-pack utility.
> s/a input/an input/
>
>> This patch enables the IMGU's parameter node as well in
>> configure_pipeline() without which, the input node won't stream
> s/ without which,/, without which/
>
>> successive input frames.
>>
>> The viewfinder and output nodes will still stream using yavta, so
>> no changes required there.
> Would it be difficult to use v4l2-ctl everywhere ? It would be nice to
> avoid depending on multiple tools.


Ok, if you prefer not to use multiple tools, I can live with that. yavta 
seems to be more concise and clear to me so I preferred keeping those parts.

>
>> It's is quite tricky to stream input and parameters nodes synchronously
>> with v4l2-ctl commands for each. The patch overcomes this short-coming
>> by streaming both the nodes endlessly until both of them are killed.
>> A sleep of 1 second is introduced to make sure $frame_count buffers
>> are captured on the output and vf nodes.
> What's the issue, can't we get v4l2-ctl to stream a given number of
> frames ?


So the commands in the script runs sequentially so if you stream nodes 
one after the other for a given set of frame_count - only (frame_count - 
X) frames get captured on output / vf nodes. So for frame_count=20, I 
got somewhere between first 7-15 outputs only(this number various as 
well). We had a brief discussion on this on IRC - seems whichever node 
calls "stream-off" will make the entire script halt/waiting.

In my observation, the parameters node can stream independently. But the 
input node, waits for parameter node to start streaming.

I tried multiple combinations to capture frame_count deterministically 
but didn't get much success, except the approach I adopted in this patch.

I should have provided exact commands here with issues in each so I need 
to re-construct the scripts first and post the excerpts here. I'll play 
around with v4l2-ctl (since you prefer not having yavta+v4l2-ctl 
combination) and see if I can replicate the goal as achieved here.

>
>> Another short-coming of this patch is when specific parameter-buffer(s)
>> (from a file) are needed to streamed on the parameters node, this might
>> require some adjusting. For now, the parameters node streams with
>> un-assigned buffers. It depends on the decision how this script is
>> expanded further down the line.
> Any proposal ? :-)


Not yet. I need to design a test case to develop it. Would you like to 
suggest any?

>
>> Signed-off-by: Umang Jain <umang.jain at ideasonboard.com>
>> ---
>>   utils/ipu3/ipu3-process.sh | 29 ++++++++++++++++++++++++-----
>>   1 file changed, 24 insertions(+), 5 deletions(-)
>>
>> diff --git a/utils/ipu3/ipu3-process.sh b/utils/ipu3/ipu3-process.sh
>> index bb4abbe8..48466d70 100755
>> --- a/utils/ipu3/ipu3-process.sh
>> +++ b/utils/ipu3/ipu3-process.sh
>> @@ -37,11 +37,13 @@ configure_pipeline() {
>>   	local enable_3a=1
>>   	local enable_out=1
>>   	local enable_vf=1
>> +	local enable_param=1
>>   	local mode=0
>>   
>>   	# Configure the links
>>   	$mediactl -r
>>   	$mediactl -l "\"$imgu_entity input\":0 -> \"$imgu_entity\":0[1]"
>> +	$mediactl -l "\"$imgu_entity parameters\":0 -> \"$imgu_entity\":1[$enable_param]"
>>   	$mediactl -l "\"$imgu_entity\":2 -> \"$imgu_entity output\":0[$enable_out]"
>>   	$mediactl -l "\"$imgu_entity\":3 -> \"$imgu_entity viewfinder\":0[$enable_vf]"
>>   	$mediactl -l "\"$imgu_entity\":4 -> \"$imgu_entity 3a stat\":0[$enable_3a]"
>> @@ -76,9 +78,26 @@ process_frames() {
>>   	$yavta $($mediactl -e "$imgu_entity 3a stat") &
>>   	sleep 0.5
>>   
>> -	# Feed the IMGU input.
>> -	$yavta -f $IMGU_IN_PIXELFORMAT -s $in_size "-F$in_file" \
>> -		$($mediactl -e "$imgu_entity input")
>> +	echo "Feeding IMGU $in_size"
>> +	# Start stream parameters node first
>> +	v4l2-ctl -d$($mediactl -e "$imgu_entity parameters") --stream-out-mmap &
>> +	local param_streaming=$!
>> +
>> +	# Start streaming on input node
>> +	local width=$(echo $in_size | awk -F 'x' '{print $1}')
>> +	local height=$(echo $in_size | awk -F 'x' '{print $2}')
>> +	v4l2-ctl -d $($mediactl -e "$imgu_entity input") \
>> +                --set-fmt-video-out=width=$width,height=$height,pixelformat=$IMGU_IN_PIXELFORMAT \
>> +                --stream-out-mmap --stream-from=$in_file --stream-loop &
>> +	local input_streaming=$!
>> +
>> +	# Sleep for 1 second and then kill the IMGU streaming. By then $frame_count output and vf
>> +	# buffers should have been captured.
>> +	sleep 1
>> +	kill $input_streaming
>> +	kill $param_streaming
>> +
>> +	echo "IMGU FED"
>>   }
>>   
>>   # Convert captured files to ppm
>> @@ -98,7 +117,7 @@ convert_files() {
>>   }
>>   
>>   run_test() {
>> -	IMGU_IN_PIXELFORMAT=IPU3_SGRBG10
>> +	IMGU_IN_PIXELFORMAT=ip3G
>>   	IMGU_OUT_PIXELFORMAT=NV12
>>   	IMGU_VF_PIXELFORMAT=NV12
>>   
>> @@ -193,6 +212,6 @@ mediactl="media-ctl -d $mdev"
>>   echo "Using device $mdev"
>>   
>>   output_dir="/tmp"
>> -frame_count=5
>> +frame_count=10
>>   nbufs=7
>>   run_test