Non centered scaler-crop on imx8mp

[Stefan Klug]

Hi Johannes,

Am 23.01.25 um 14:21 schrieb Johannes Kirchmair - SKIDATA:
> Dear libcamera folks,
> 
> I was trying out this new scaler-crop feature because I wanted a 4:3 video stream from a 16:9 camera.
> I am working on an imx8mp.
> 
> So, I tried to get jpegs like this:
> gst-launch-1.0 libcamerasrc scaler-crop="<0,0,1440,1080>" ! video/x-raw,format=NV12,width=1440,height=1080 ! jpegenc ! multifilesink location=/tmp/test.jpeg
> 
> The focus is to be able to vary the x coordinate of the scaler crop, to choose the correct viewing angle for our application.
> Unfortunately, it seems like this does not work.
> 
> Debug information reveals the following:
> [0:08:48.011393125] [2108]  INFO Camera camera.cpp:1202 configuring streams: (0) 1440x1080-NV12
> [0:08:48.011711000] [2107] DEBUG RkISP1 rkisp1.cpp:1231 Enabling link from sensor 'imx462 6-001a' to ISP
> [0:08:48.011798500] [2107] DEBUG RkISP1 rkisp1.cpp:819 Configuring sensor with 1920x1080-SRGGB10_1X10
> [0:08:48.012114000] [2107] DEBUG RkISP1 rkisp1.cpp:831 Sensor configured with 1920x1080-SRGGB10_1X10
> [0:08:48.012159625] [2107] DEBUG RkISP1 rkisp1.cpp:848 ISP input pad configured with 1920x1080-SRGGB10_1X10 crop (0, 0)/1920x1080
> [0:08:48.012399750] [2107] DEBUG RkISP1 rkisp1.cpp:880 Configuring ISP output pad with 1440x1080-YUYV8_2X8 crop (240, 0)/1440x1080
> [0:08:48.012452500] [2107] DEBUG RkISP1 rkisp1.cpp:893 ISP output pad configured with 1440x1080-YUYV8_2X8 crop (240, 0)/1440x1080
> [0:08:48.012673000] [2107] DEBUG RkISP1 rkisp1_path.cpp:436 Configured main resizer input pad with 1440x1080-YUYV8_2X8 crop (0, 0)/1440x1080
> [0:08:48.012718000] [2107] DEBUG RkISP1 rkisp1_path.cpp:442 Configuring main resizer output pad with 1440x1080-YUYV8_2X8
> [0:08:48.012949500] [2107] DEBUG RkISP1 rkisp1_path.cpp:456 Configured main resizer output pad with 1440x1080-YUYV8_1_5X8
> [0:08:48.249117750] [2107] DEBUG RkISP1 rkisp1.cpp:1573 Applied rectangle (240, 0)/1440x1080 differs from requested (0, 0)/1440x1080
> 
> It seems like, the camera setup crops the stream already to a centric view angle on ISP out, so the resizer is not able to crop it to the view angle that I prefer.
> Wouldn't it be the best to respect the scaler-crop parameters on the ISP output already?

Your analysis is correct. The cropping to the output aspect ratio 
happens on the output stage of the ISP. In your case, as the output 
height is the same as the sensor height you are left with no freedom in 
x-direction.

It would be possible to split the ScalerCrop parameters and apply parts 
of it on the ISP output (Offset) and the rest in the dewarper (scaling). 
We didn't implement this because there is a pending patch set that 
supports arbitrary cropping in the dewarper. In that patchset the full 
sensor frame gets forwarded to the dewarper and all cropping (+rotate) 
happens pixel perfect in the dewarper without having to reconfigure the 
ISP output during streaming. So there is code on the way that fully 
supports your usecase.

> 
> To be honest I tried to implement it but was not able to access the scaler-crop data in the function that sets the ISP output. I am not (yet) very familiar with the inner workings of libcamera :-D
> 
> What is your opinion about this.

Unfortunately you need to wait a bit :-). If you are eager to hack on 
it, the correct place would be in src/pipeline/rkisp1/rkisp1.cpp

In PipelineHandlerRkISP1::queueRequestDevice() you would need to update
isp_->setSelection(2, V4L2_SEL_TGT_CROP, ...); and make sure that gets 
honored in imageBufferReady() (e.g. only remaining crop get passed to 
the dewarper). But to be honest I don't know how the isp reacts on 
SEL_TGT_CROP during streaming. So that might be a bumpy ride.

Best regards,
Stefan

> 
> Best regards
> Johannes