libcamera - Development of Open Source Embedded Linux Wildlife Camera System

[w.robertson at cairnwater.com]

Hi Laurent,

> Possibly. Training an NN model for this may be lots of work though.
> The traditional AEGC algorithm approach seems less costly to me.

I think the AEGC work of an IPU may be something for which an extremely 
simple neural network (NN) could be written by hand instead of being 
trained - the NN might only contain one neuron which very rapidly adds 
together the values from a selection of the pixels and outputs the 
average to be used to adjust autoexposure in the next frame - or it 
could be a little more complex and weight central pixels more strongly 
before averaging - other simple CNNs could be programmed to handle 
things like autofocus - but the NNs would still be simple compared to 
NNs for image recognition, etc.

The advantage of a simple NN is that it could be coded once in one of 
the widely used open source frameworks for working with NNs then run on 
the NPU on any manufacturer's SoC (with some manufacturers offering 
software to optimise NNs before being run on their NPUs) - this might 
give it an advantage over IPU code which has to be manually written for 
each different SoC manufacturer's ISP.

This would have been irrelevant a few years ago when very few SoC 
manufacturers included NPUs on their devices but more recent SoCs seem 
to include some inexpensive, powerful and energy-efficient NPUs.

A task like adding together a large number of pixels for AEGC would take 
a large number of clock cycles to do sequentially on a CPU but might be 
much faster done in parallel on an NPU.

That's all blue-sky thinking - I could be wrong.

> That's one of the things you would be able to experiment with more
> easily with libcamera compared to closed-source camera stacks :-)

:-)

> libcamera supports the RK3399, which is getting quite old now (but may
> still be an interesting platform to consider). Newer Rockchip SoCs are
> not supported yet. I know that some people expressed interest in 
> working
> on them, but no code has been contributed yet. I'm am not aware of any
> schedule.

Yes - that was my hesitation with the RK3399 - the documentation dated 
2017 suggests it's 8 years old now.

I guess the important thing is that NXP seems to be vastly more 
committed to libcamera and to open source than Rockchip is...

Will

On 2025-01-07 13:56, Laurent Pinchart wrote:
> Hi Will,
> 
> On Tue, Jan 07, 2025 at 12:23:36PM +0000, w.robertson at cairnwater.com 
> wrote:
>> Hi Laurent,
>> 
>> Thank you very much.
>> 
>> > Roughly speaking, AEGC (auto-exposure and gain control) involves
>> > computing statistics on the image (it can be as simple as measuring the
>> > average pixel intensity on a small region of interest in software, but
>> > to obtain good results more complex types of statistics are needed, such
>> > as dividing the image in a grid and computing averages and histograms
>> > for each grid element), and using those statistics to compute the
>> > exposure time and analog gain for the next frame. The process needs to
>> > be repeated for every frame. Some sensors implement AEGC internally,
>> > which could be useful for your use cases. Otherwise, statistics are
>> > computed by the ISP, and libcamera runs an AEGC algorithm to process the
>> > statistics, compute the exposure time and gain, and apply them to the
>> > sensor.
>> 
>> > Note that the AECG algorithm will take several frames to converge, which
>> > means that the first few frames captured from the camera won't be
>> > properly exposed. Convergence may be faster when AEGC is implemented by
>> > the sensor (no guarantee though, it would need to be tested).
>> 
>> With wildlife cameras it may be possible to use tricks like a 
>> photodiode
>> or photodiode array and ADC used as a light level meter for AEGC - 
>> that
>> wouldn't work so well for other applications but for wildlife cameras 
>> it
>> might.
> 
> That's one of the things you would be able to experiment with more
> easily with libcamera compared to closed-source camera stacks :-)
> 
>> It could be that NPUs can do this sort of task more efficiently
>> than CPUs - it might be possible to implement AEGC as a simple neural
>> network running on an NPU - I'm not sure.
> 
> Possibly. Training an NN model for this may be lots of work though.
> The traditional AEGC algorithm approach seems less costly to me.
> 
>> > It would be interesting to port libcamera to those smaller SoCs, running
>> > real time operating systems, but it's a large project.
>> 
>> How would this be done? Would it be a matter of finding one funder or 
>> a
>> consortium of funders to fund the work or of starting with a subset of
>> the simplest functionality and progressing from there?
> 
> It first requires finding someone, or a set of people, with an interest
> to make this happen. Those people could contribute time by performing
> the work, our budget to outsource it.
> 
>> One of the problems might be choosing an RTOS - different players seem
>> to be favouring different RTOSs so I became cautious about investing 
>> too
>> much in any one RTOS.
>> 
>> What's the situation like with Rockchip SoCs? I've heard different
>> conflicting things about them recently so I became cautious.
> 
> libcamera supports the RK3399, which is getting quite old now (but may
> still be an interesting platform to consider). Newer Rockchip SoCs are
> not supported yet. I know that some people expressed interest in 
> working
> on them, but no code has been contributed yet. I'm am not aware of any
> schedule.
> 
>> On 2025-01-06 23:21, Laurent Pinchart wrote:
>> > On Mon, Jan 06, 2025 at 04:41:03PM +0000, w.robertson at cairnwater.com  wrote:
>> >> Hi Laurent,
>> >>
>> >> > That looks fine.
>> >>
>> >> That's great - that's an enormous help to know that I'm on the right
>> >> track. The DEBIX Model A SBC (NXP i.MX 8M Plus) and DEBIX I/O Board
>> >> arrived :-)
>> >>
>> >> > I have no experience with those chips, so I can't comment on this.
>> >>
>> >> I think hardly anyone has experience with them - the idea of a μC with
>> >> CSI-2 and ISP so new - STM32N6 was only launched in December.
>> >>
>> >> > I'm not sure what they expect their customers to use for the time being,
>> >> > I believe (and hope) libcamera is the solution they will push forward.
>> >>
>> >> Thanks - I'll try to see if I can give them some encouragement re.
>> >> libcamera gently.
>> >>
>> >> > It's theoretically possible, assuming again that auto-exposure is
>> >> > implemented on the device. That, and auto-focus if needed, are the
>> >> > only parts that can't be implemented in post-processing. You would
>> >> > however need to store the data uncompressed, which will require a large
>> >> > amount of storage, and high bandwidth to the storage medium.
>> >>
>> >> Auto-exposure is something that I don't know about - some image
>> >> sensors
>> >> seem to claim that they implement it and others don't.
>> >
>> > Roughly speaking, AEGC (auto-exposure and gain control) involves
>> > computing statistics on the image (it can be as simple as measuring the
>> > average pixel intensity on a small region of interest in software, but
>> > to obtain good results more complex types of statistics are needed, such
>> > as dividing the image in a grid and computing averages and histograms
>> > for each grid element), and using those statistics to compute the
>> > exposure time and analog gain for the next frame. The process needs to
>> > be repeated for every frame. Some sensors implement AEGC internally,
>> > which could be useful for your use cases. Otherwise, statistics are
>> > computed by the ISP, and libcamera runs an AEGC algorithm to process the
>> > statistics, compute the exposure time and gain, and apply them to the
>> > sensor.
>> >
>> > Note that the AECG algorithm will take several frames to converge, which
>> > means that the first few frames captured from the camera won't be
>> > properly exposed. Convergence may be faster when AEGC is implemented by
>> > the sensor (no guarantee though, it would need to be tested).
>> >
>> >> Usually autofocus isn't present on wildlife cameras - in commercial
>> >> wildlife cameras the focus is fixed in manufacture and can't be changed
>> >> without dismantling the wildlife camera - including watertight seals -
>> >> or by using glue or blue-tack (I'm not joking!) to attach simple +/- 1,
>> >> 2 or 3 dioptre lenses in front of the wildlife camera.
>> >>
>> >> Just by offering the ability to set focus by manually adjusting the lens
>> >> or set focus in software configuration CritterCam would be a big step
>> >> forward - autofocus (for example with the RPi Camera v3) would be an
>> >> optional nice-to-have.
>> >>
>> >> The high bandwidth to the storage medium might be a problem - I'm not
>> >> sure what the limits are for that. Commercial wildlife cameras seem to
>> >> vomit enormous uncompressed photo and video files onto the SD card and
>> >> leave them there.
>> >>
>> >> This is out of scope for libcamera: The STM32N6 Series claims "MIPI
>> >> CSI-2 interface and image signal processing (ISP)" and "an H264 hardware
>> >> encoder" and the NXP S32V claims "Embedded image sensor processing (ISP)
>> >> for HDR, color conversion, tone mapping, etc.", " Image signal processor
>> >> (ISP), supporting 2x1 or 1x2 megapixel @ 30 fps and 4x2 megapixel for
>> >> subset of functions (exposure control, gamma correction)" and "H.264
>> >> video encode" so it looks like ISP and compression and encoding are
>> >> there on some μCs in theory but the technology seems very new and the
>> >> number of pixels and frame rate supported is limited.
>> >
>> > It would be interesting to port libcamera to those smaller SoCs, running
>> > real time operating systems, but it's a large project.
>> >
>> >> On 2025-01-06 08:20, Laurent Pinchart wrote:
>> >> > Hi Will,
>> >> >
>> >> > On Fri, Jan 03, 2025 at 09:52:13PM +0000, w.robertson at cairnwater.com
>> >> > wrote:
>> >> >> Hi Laurent,
>> >> >>
>> >> >> Thank you very much. I've redrafted the README.md file to get rid of the
>> >> >> largely irrelevant information about CPU cores on SoCs and include the
>> >> >> much more relevant information about ISP instead. Also reduced down the
>> >> >> list of SoCs to SoCs with current or planned libcamera support. In the
>> >> >> low power category I got rid of the NXP i.MX RT1170 and replaced it with
>> >> >> the S32V2 which has an ISP. Let me know if I got it right and I'll
>> >> >> redraft the 6. Development & Prototyping section:
>> >> >>
>> >> >> - **Candidate Processors**
>> >> >> 	- **High Quality, High Flexibility**
>> >> >> 		- **Supported by libcamera**
>> >> >> 			- **ST STM32MP2 Series** CSI-2 #1 (5 Mpixels @30 fps with ISP), #2 (1
>> >> >> Mpixels @15 fps no ISP)
>> >> >> 			- **NXP i.MX 8M Plus** Dual ISPs: up to 12MP and 375MPixels/s
>> >> >> 			- **NXP i.MX95** Preproduction: 500 Mpixel/s MIPI-CSI and ISP (2x
>> >> >> 4-lane, 2.5 Gbps/lane)
>> >> >> 			- **Chips Integrating Mali-C55 ISP**
>> >> >> 				- **Renesas RZ/V2H** Mali-C55 ISPC
>> >> >> 			- Chips Integrating **Mali C52 ISP**
>> >> >>     				- Possibly in the longer therm
>> >> >> 			- **Chips Integrating Amlogic C3 ISP**
>> >> >>     				- Currently in development by Amlogic
>> >> >
>> >> > That looks fine.
>> >> >
>> >> >> 	- **Low Power**
>> >> >> 		- **ST STM32N6 Series μCs** ISP, MIPI CSI-2, H264 hardware
>> >> >> 		- **NXP S32V2** Embedded ISP for HDR, color conversion, tone mapping,
>> >> >> etc. enabled by ISP graph tool
>> >> >
>> >> > I have no experience with those chips, so I can't comment on this.
>> >> >
>> >> >> > I think that would be a good first target. Using a raw sensor with an
>> >> >> > SoC that integrates an ISP gives you a very wide range of usable camera
>> >> >> > sensors, with a wide range of prices and performances. It is the most
>> >> >> > flexible solution, especially if you want a modular system that allows
>> >> >> > users to pick a sensor based on their needs and budget. Adding support
>> >> >> > for a new sensor is relative easy. The downside is the limited choice of
>> >> >> > SoCs, and the cost of bringing up a new ISP. As libcamera evolves, you
>> >> >> > can expect support for more ISPs and SoCs, so the situation will improve
>> >> >> > over time.
>> >> >>
>> >> >> Thank you very much - that's an enormous help - I'd got mixed up and had
>> >> >> thought that the big cost was in the image sensor driver software not in
>> >> >> the software for the ISP - it's an enormous help to me to know that the
>> >> >> software for the ISP is the expensive part.
>> >> >>
>> >> >> Commercial wildlife camera manufacturers often seem to try to make money
>> >> >> by price gouging researchers and conservationists but between NXP and ST
>> >> >> there seems to be healthy competition and they seem to try to aim to
>> >> >> make money through excellence and by winning market share in large,
>> >> >> highly competitive and fast-growing markets - they've no interest in
>> >> >> trying to price gouge conservationists.
>> >> >>
>> >> >> All the real intellectual heavy lifting in CritterCam is done by the
>> >> >> libcamera team and open source so safe into the future.
>> >> >>
>> >> >> I don't have the specialist experience or funding needed to design a
>> >> >> board for a modern SoC but prototyping using an evaluation board for a
>> >> >> SoM or SBC then designing a carrier board for a SoM or SBC seems
>> >> >> feasible. Maybe if CritterCam becomes widely used someone could find
>> >> >> separate funding to reduce production costs by designing a board based
>> >> >> on a reference design for a SoC without a SoM but that would be some
>> >> >> time in the future.
>> >> >>
>> >> >> > Using a smart sensor with a simpler SoC gives you a much wider range of
>> >> >> > SoCs to choose from, and a lower price point, and possibly with better
>> >> >> > low power capabilities. The drawback is the limited choice of camera
>> >> >> > sensors. If you want to provide very very low power consumption on some
>> >> >> > models, this is an architecture that may still be worth investigating.
>> >> >> >
>> >> >> > In any case, I would recommend picking one of the two options to start
>> >> >> > with, and exploring the second option later as an extension to the
>> >> >> > product range if needed.
>> >> >>
>> >> >> It's difficult. I think fundamentally solar panels are becoming fairly
>> >> >> cheap now but a software engineer's time is very expensive so there's an
>> >> >> economic case for having the flexibility of an OS there and keeping
>> >> >> firmware development costs to a minimum even if it comes at the price of
>> >> >> relatively higher power consumption and relatively large solar panels.
>> >> >>
>> >> >> I kind-of get the feeling that a highly power efficient μC solution
>> >> >> might be something to look at separately in the future if funding
>> >> >> becomes available - STM32N6 and NXP S32V2 were the only μCs I could find
>> >> >> with an ISP and STM32N6 was only launched in mid-December so the
>> >> >> technology seems very new.
>> >> >>
>> >> >> > Yes, the list of pipeline handlers is the best place to check. STM32MP2
>> >> >> > support will come (I don't know ST's roadmap, so I can't tell when).
>> >> >>
>> >> >> Do ST expect customers to use the V4L2 and Media Controller APIs for the
>> >> >> time being until libcamera support is implemented?
>> >> >
>> >> > I'm not sure what they expect their customers to use for the time being,
>> >> > I believe (and hope) libcamera is the solution they will push forward.
>> >> >
>> >> >> > Amlogic is working on support for their C3 ISP
>> >> >>
>> >> >> Is there a way of working out which of the Amlogic chips use their C3
>> >> >> ISP or do all Amlogic chips with an ISP use either their C3 ISP or an
>> >> >> ARM Mali ISP? (I was trying to work out if the Amlogic C308X's ISP is a
>> >> >> C3 ISP but I couldn't work it out 🙈)
>> >> >
>> >> > Kieran, do you have more information about this ?
>> >> >
>> >> >> > There are V2H versions without the C55 ISP but with a simpler "ISP
>> >> >> > lite" processing pipeline that could also be considered. They should be
>> >> >> > cheaper, may consume less power, and could still provide acceptable
>> >> >> > image quality.
>> >> >>
>> >> >> I don't think much about minimising the cost of the SoC - I reckon a lot
>> >> >> of the modern SoCs with a camera pipeline offer exceptional value for
>> >> >> money - there seems to be a big price increase and increase in power
>> >> >> consumption between the SoC and the SoM or SBC so I tend to worry more
>> >> >> about the cost and power consumption of the SoM or SBC. I tend to worry
>> >> >> about power consumption in sleep mode a lot more than power consumption
>> >> >> in active mode because wildlife cameras only tend to spend a very small
>> >> >> proportion of their time in an active state.
>> >> >>
>> >> >> > Note that, to use an SoC without an ISP, you will need a full ISP in the
>> >> >> > camera sensor. As far as I can tell from the data brief, the ISP in the
>> >> >> > VD1940 doesn't implement a full processing pipeline, and still outputs
>> >> >> > bayer data.
>> >> >>
>> >> >> Thanks - I should be cautious of that.
>> >> >>
>> >> >> > That being said, it also depends what you want to record. If your goal
>> >> >> > is to capture still images, you could possibly get away without a real
>> >> >> > ISP and use software processing on still images. You would still need to
>> >> >> > implement AGC and AWB, and some SoCs (in the i.MX7 family for instance)
>> >> >> > include a statistics computation engine that should be enough to do so.
>> >> >> > Some camera sensors also implement AGC and AWB in the sensor itself,
>> >> >> > albeit with less control on the algorithm, so it may or may not work for
>> >> >> > your use cases.
>> >> >>
>> >> >> Goedele Verbeylen strongly prefers videos to still images.
>> >> >>
>> >> >> Would it be possible to stream raw data from an image sensor to a raw
>> >> >> data file in real time then for ISP tasks to be carried out on the raw
>> >> >> data at a later date when handling data in real time isn't important
>> >> >> (this could be done after download to a more powerful computer or by a
>> >> >> μC core gradually during the day when solar energy is plentiful)?
>> >> >
>> >> > It's theoretically possible, assuming again that auto-exposure is
>> >> > implemented on the device. That, and auto-focus if needed, are the
>> >> > only parts that can't be implemented in post-processing. You would
>> >> > however need to store the data uncompressed, which will require a large
>> >> > amount of storage, and high bandwidth to the storage medium.
>> >> >
>> >> >> On 2025-01-02 22:55, Laurent Pinchart wrote:
>> >> >> > On Thu, Jan 02, 2025 at 11:53:07AM +0000, w.robertson at cairnwater.com wrote:
>> >> >> >> Hi Laurent,
>> >> >> >>
>> >> >> >> Yes - I'm very worried about the ambitiousness of it and the risk of
>> >> >> >> scope creep wrecking the project - I was thinking about limiting the
>> >> >> >> scope to SoCs and image sensors supported by libcamera - do you think
>> >> >> >> that would work? Anyone wanting a SoC or image sensor added that isn't
>> >> >> >> supported by libcamera would have to estimate feasibility and cost for
>> >> >> >> that separately.
>> >> >> >
>> >> >> > I think that would be a good first target. Using a raw sensor with an
>> >> >> > SoC that integrates an ISP gives you a very wide range of usable camera
>> >> >> > sensors, with a wide range of prices and performances. It is the most
>> >> >> > flexible solution, especially if you want a modular system that allows
>> >> >> > users to pick a sensor based on their needs and budget. Adding support
>> >> >> > for a new sensor is relative easy. The downside is the limited choice of
>> >> >> > SoCs, and the cost of bringing up a new ISP. As libcamera evolves, you
>> >> >> > can expect support for more ISPs and SoCs, so the situation will improve
>> >> >> > over time.
>> >> >> >
>> >> >> > Using a smart sensor with a simpler SoC gives you a much wider range of
>> >> >> > SoCs to choose from, and a lower price point, and possibly with better
>> >> >> > low power capabilities. The drawback is the limited choice of camera
>> >> >> > sensors. If you want to provide very very low power consumption on some
>> >> >> > models, this is an architecture that may still be worth investigating.
>> >> >> >
>> >> >> > In any case, I would recommend picking one of the two options to start
>> >> >> > with, and exploring the second option later as an extension to the
>> >> >> > product range if needed.
>> >> >> >
>> >> >> >> > Most (if not all) of the camera sensors you list are raw camera sensors,
>> >> >> >> > and most of the SoCs you list either don't include an ISP (SAMA7G54,
>> >> >> >> > R2/G2, AM625, STM32N6, RT1170) or have an ISP that is not supported yet
>> >> >> >> > but upstream kernel drivers and by libcamera (AM67A, TDA4VM, Genio 510).
>> >> >> >> > It isn't clear to me how you plan to address that issue.
>> >> >> >>
>> >> >> >> Thank you very much. That's an extremely important point.
>> >> >> >>
>> >> >> >> What's the best way to work out which SoCs are supported by libcamera?
>> >> >> >> Is it best to look in the git repository here?:
>> >> >> >>
>> >> >> >> https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/pipeline
>> >> >> >>
>> >> >> >> with NXP i.MX8M Plus SoC having full support and that ST planning to add
>> >> >> >> full support for STM32MP2 Series during 2025 ( currently partial support
>> >> >> >> https://wiki.st.com/stm32mpu/wiki/How_to_use_libcamera )?
>> >> >> >
>> >> >> > Yes, the list of pipeline handlers is the best place to check. STM32MP2
>> >> >> > support will come (I don't know ST's roadmap, so I can't tell when).
>> >> >> > There's work underway by NXP on i.MX95 support too. We are working on
>> >> >> > the Mali C55 ISP, which is integrated in the Renesas RZ/V2H. As a
>> >> >> > stretch goal we want to support the C52, found in at least some Amlogic
>> >> >> > SoCs, but that's more of a spare time project at this point. Amlogic is
>> >> >> > working on support for their C3 ISP (see
>> >> >> > https://lore.kernel.org/r/20241227-c3isp-v5-0-c7124e762ff6@amlogic.com
>> >> >> > and
>> >> >> > https://patchwork.libcamera.org/project/libcamera/list/?series=4926), I
>> >> >> > don't know when it will be ready for production. There will be more.
>> >> >> >
>> >> >> >> I think it would be an enormous help for CritterCam to focus only on
>> >> >> >> SoCs and image sensors supported by libcamera with any SoCs and image
>> >> >> >> sensors that aren't supported by libcamera being treated as optional
>> >> >> >> add-on functionality for which costs and feasibility would have to be
>> >> >> >> estimated separately.
>> >> >> >>
>> >> >> >> If I abandon all the following because they do not have a hardware ISP:
>> >> >> >>
>> >> >> >> SAMA7G54
>> >> >> >> i.MX RT1170
>> >> >> >> Renesas RZ/G2
>> >> >> >>
>> >> >> >> However Renesas RZ/G2 could be replaced by Renesas RZ/V2H which has an
>> >> >> >> ARM Mali-C55 ISP since libcamera is working on Mali-C55 support
>> >> >> >> https://www.renesas.com/en/software-tool/rzv2h-isp-support-package
>> >> >> >> https://libcamera.org/entries/2024-01-31.html
>> >> >> >
>> >> >> > Indeed. I should have read your mail fully before writing the text above
>> >> >> > :-) There are V2H versions without the C55 ISP but with a simpler "ISP
>> >> >> > lite" processing pipeline that could also be considered. They should be
>> >> >> > cheaper, may consume less power, and could still provide acceptable
>> >> >> > image quality.
>> >> >> >
>> >> >> >> AM67A, TDA4VM and Genio 510 would all be put out of scope until they
>> >> >> >> support libcamera.
>> >> >> >
>> >> >> > Those are SoCs we would really like to support.
>> >> >> >
>> >> >> >> STM32N6 can't run Linux but does have a hardware ISP "A dedicated
>> >> >> >> computer vision pipeline with a MIPI CSI-2 interface and image signal
>> >> >> >> processing (ISP) ensures compatibility with a wide range of cameras. The
>> >> >> >> STM32N6 also features an H264 hardware encoder".
>> >> >> >>
>> >> >> >> I put together a list of the ST BrightSense image sensors and whether
>> >> >> >> they do or do not have an integrated ISP or similar (Google Sheets
>> >> >> >> https://docs.google.com/spreadsheets/d/1S1-Ji6rsUPARtQFwsfhblWhzQ-97r-zk/edit?usp=sharing&ouid=102590517794814819664&rtpof=true&sd=true
>> >> >> >> or tab separated below):
>> >> >> >>
>> >> >> >> VG5761, VG5661, VD5761, VD5661 data sheet
>> >> >> >> Block diagram of Vx5y61 Shows ISP however web pages don't all seem to
>> >> >> >> reflect this
>> >> >> >>
>> >> >> >> VB56G4A	No ISP	Embedded autoexposure
>> >> >> >> VD16GZ	No ISP	in-sensor AE and various corrections.
>> >> >> >> VD1940	On-chip bayerization ISP
>> >> >> >
>> >> >> > Note that, to use an SoC without an ISP, you will need a full ISP in the
>> >> >> > camera sensor. As far as I can tell from the data brief, the ISP in the
>> >> >> > VD1940 doesn't implement a full processing pipeline, and still outputs
>> >> >> > bayer data.
>> >> >> >
>> >> >> > That being said, it also depends what you want to record. If your goal
>> >> >> > is to capture still images, you could possibly get away without a real
>> >> >> > ISP and use software processing on still images. You would still need to
>> >> >> > implement AGC and AWB, and some SoCs (in the i.MX7 family for instance)
>> >> >> > include a statistics computation engine that should be enough to do so.
>> >> >> > Some camera sensors also implement AGC and AWB in the sensor itself,
>> >> >> > albeit with less control on the algorithm, so it may or may not work for
>> >> >> > your use cases.
>> >> >> >
>> >> >> > Lots of options...
>> >> >> >
>> >> >> >> VD55G0	No ISP	Embedded auto-exposure
>> >> >> >> VD55G1	No ISP	Various on-chip features to optimize image quality, such
>> >> >> >> as autoexposure, noise reduction, or defect correction
>> >> >> >> VD5661	No ISP	Embedded 16-bit video processing pipe with pixel defect
>> >> >> >> correction, high dynamic range (HDR) merge with ghost removal, and
>> >> >> >> programmable compression of dynamic
>> >> >> >> VD56G3	No ISP	in-sensor autoexposure and various corrections.
>> >> >> >> VD5761	No ISP	Embedded 16-bit video processing pipe with pixel defect
>> >> >> >> correction, high dynamic range (HDR) merge with ghost removal, and
>> >> >> >> programmable compression of dynamic
>> >> >> >> VD66GY	No ISP	in-sensor autoexposure and various corrections.
>> >> >> >> VG5761	No ISP	Embedded 16-bit video processing pipe with pixel defect
>> >> >> >> correction, high dynamic range (HDR) merge with ghost removal, and
>> >> >> >> programmable compression of dynamic
>> >> >> >> VG6640	No ISP	An embedded Bayer and monochrome data pre-processor
>> >> >> >> VB1740	On-chip bayerization ISP
>> >> >> >> VB1940	On-chip bayerization ISP
>> >> >> >>
>> >> >> >> For Sony I couldn't find the data but as far as I could work out the
>> >> >> >> IMX708 doesn't have an integrated ISP.
>> >> >> >>
>> >> >> >> > Wake up latency is also something that will need to be considered very
>> >> >> >> > carefully. If your power consumption requirements require powering down
>> >> >> >> > the SoC and DRAM completely, the system will need to cold-boot when
>> >> >> >> > woken up, which will probably take too long. Some workarounds are
>> >> >> >> > possible but may require very extensive software development efforts,
>> >> >> >> > and those workaround may not be portable across different SoCs.
>> >> >> >>
>> >> >> >> Power consumption is a major factor - at the moment I'm looking at using
>> >> >> >> a 120 W nominal solar panel - guessing that real output would be 5 % of
>> >> >> >> that on average would give 6 W - using that to charge a c. 500 Wh PBa
>> >> >> >> battery to run a SoM that consumes several hundred mW in sleep / suspend
>> >> >> >> mode should give enough margin for safety during spring, summer and
>> >> >> >> autumn - and perhaps winter - the measurement equipment hasn't arrived
>> >> >> >> yet to test this though.
>> >> >> >>
>> >> >> >> > One last (smaller) technical comment: you mention ACPI system states (S0
>> >> >> >> > to S4), while none the the SoCs you list use ACPI.
>> >> >> >>
>> >> >> >> Thanks - I'd just been asking the SoM and SoC manufacturers about a "low
>> >> >> >> power sleep / suspend mode" not about ACPI specifically and I'd mixed
>> >> >> >> the two up. I'm cautious about committing to this until I've checked
>> >> >> >> whether the measured power consumption matches the manufacturer's stated
>> >> >> >> power consumption.
>> >> >> >
>> >> >> > Real life measurement are definitely needed.
>> >> >> >
>> >> >> >> On 2025-01-01 23:08, Laurent Pinchart wrote:
>> >> >> >> > On Tue, Dec 31, 2024 at 02:45:44PM +0000, w.robertson at cairnwater.com
>> >> >> >> > wrote:
>> >> >> >> >> Hi Laurent, Jacopo and Kieran,
>> >> >> >> >>
>> >> >> >> >> All my best wishes to you for 2025!
>> >> >> >> >>
>> >> >> >> >> I put together this overview summary of CritterCam that hopefully
>> >> >> >> >> explains how the engineering and economics of the situation make it
>> >> >> >> >> feasible - let me know if it would be of any help to you as an example
>> >> >> >> >> use case (it might be a good one because I'm not trying to sell anything
>> >> >> >> >> or make a sales pitch but I am trying to take the speed and aggression
>> >> >> >> >> of modern software engineering to an until-now far-too-sleepy area of
>> >> >> >> >> electronics):
>> >> >> >> >>
>> >> >> >> >> https://bitbucket.org/WillRobertRobertson/crittercam/src/main/README.md
>> >> >> >> >
>> >> >> >> > I went through the document. The project sounds very ambitious, I think
>> >> >> >> > you will need to carefully plan intermediate steps on your way to the
>> >> >> >> > end goal.
>> >> >> >> >
>> >> >> >> > One point that you may have overlooked is the design of the camera
>> >> >> >> > pipeline. Roughly speaking, to capture processed image, you can use
>> >> >> >> >
>> >> >> >> > - A raw camera sensor connected to an SoC with an ISP (ideally hardware,
>> >> >> >> >   possibly software with GPU offload at the cost of higher power
>> >> >> >> >   consumption).
>> >> >> >> >
>> >> >> >> > - A raw camera sensor connected to a standalone ISP, itself connected to
>> >> >> >> >   an SoC without an ISP (but with a camera capture interface, typically
>> >> >> >> >   a CSI-2 or parallel receiver). This increases the cost and power
>> >> >> >> >   consumption due to the external ISP. Wake up latencies can also be
>> >> >> >> >   affected.
>> >> >> >> >
>> >> >> >> > - A camera sensor that integrates an ISP (a.k.a. smart sensor or YUV
>> >> >> >> >   sensor) connected to an SoC without an ISP (but with a camera capture
>> >> >> >> >   interface, typically a CSI-2 or parallel receiver).
>> >> >> >> >
>> >> >> >> > Most (if not all) of the camera sensors you list are raw camera sensors,
>> >> >> >> > and most of the SoCs you list either don't include an ISP (SAMA7G54,
>> >> >> >> > R2/G2, AM625, STM32N6, RT1170) or have an ISP that is not supported yet
>> >> >> >> > but upstream kernel drivers and by libcamera (AM67A, TDA4VM, Genio510).
>> >> >> >> > It isn't clear to me how you plan to address that issue.
>> >> >> >> >
>> >> >> >> > Wake up latency is also something that will need to be considered very
>> >> >> >> > carefully. If your power consumption requirements require powering down
>> >> >> >> > the SoC and DRAM completely, the system will need to cold-boot when
>> >> >> >> > woken up, which will probably take too long. Some workarounds are
>> >> >> >> > possible but may require very extensive software development efforts,
>> >> >> >> > and those workaround may not be portable across different SoCs.
>> >> >> >> >
>> >> >> >> > One last (smaller) technical comment: you mention ACPI system states (S0
>> >> >> >> > to S4), while none the the SoCs you list use ACPI.
>> >> >> >> >
>> >> >> >> > Please see below for more comments.
>> >> >> >> >
>> >> >> >> >> The in-depth analysis behind that is in separate files (far too long to
>> >> >> >> >> include in the summary) that're in a private repository for now because
>> >> >> >> >> they have email addresses, etc.  (Think I'm developing a preference for
>> >> >> >> >> GitHub over BitBucket...)
>> >> >> >> >>
>> >> >> >> >> Also translated Goedele's most recent article to English :-)
>> >> >> >> >>
>> >> >> >> >> https://new-homes-for-old-friends.cairnwater.com/carved-tree-cavities-for-endangered-dormice/
>> >> >> >> >>
>> >> >> >> >> All the best for 2025!
>> >> >> >> >>
>> >> >> >> >> Will
>> >> >> >> >>
>> >> >> >> >> On 2024-12-22 19:22, w.robertson at cairnwater.com wrote:
>> >> >> >> >> > Hi Laurent, Jacopo and Kieran,
>> >> >> >> >> >
>> >> >> >> >> > I was trying to think of ways that I could help out. All my current
>> >> >> >> >> > experience is server side Python/.Net/SQL ( www.faclair.com is one
>> >> >> >> >> > project) and as a climbing arborist so I don't have any kernel
>> >> >> >> >> > experience.
>> >> >> >> >> >
>> >> >> >> >> > I'm compiling a list of SoMs, SiPs and SBCs that should in theory work
>> >> >> >> >> > with libcamera with relevant information from their technical data
>> >> >> >> >> > sheets and questions in emails to their manufacturers - at the moment
>> >> >> >> >> > this is in CritterCam's git repository but I could make this available
>> >> >> >> >> > for other libcamera users and testers if this would be a help?
>> >> >> >> >> >
>> >> >> >> >> > I looked at using a μC with a MIPI CSI-2 interface.
>> >> >> >> >> >
>> >> >> >> >> > From a conservation perspective, the research has been very successful
>> >> >> >> >> > but trying to fund it from my commercial work has been financially
>> >> >> >> >> > disastrous - I'm moving out of my flat and into my van to save money in
>> >> >> >> >> > the hope that enough funding to survive might become available next
>> >> >> >> >> > year - if I tried to port libcamera to a μC I'd go bankrupt long before
>> >> >> >> >> > the work could be completed.
>> >> >> >> >> >
>> >> >> >> >> > The need for CritterCam to outlast all currently available silicon -
>> >> >> >> >> > keeping up with very rapidly evolving processor and sensor technology -
>> >> >> >> >> > and to support multiple contributors contributing both software
>> >> >> >> >> > features and add-on off-the-shelf and custom electronic modules to a
>> >> >> >> >> > single open source repository also encouraged me to look at a μP and
>> >> >> >> >> > embedded Linux instead of a μC.
>> >> >> >> >> >
>> >> >> >> >> > The only ways I can see at the moment of having any possibility of
>> >> >> >> >> > getting a high quality camera working on a μC would be to work together
>> >> >> >> >> > with Raspberry Pi or ST.
>> >> >> >> >> >
>> >> >> >> >> > A talk by Naushir Patuck from Raspberry Pi in November 2024 mentioned
>> >> >> >> >> > that Raspberry Pi were considering the possibility of a μC with MIPI
>> >> >> >> >> > CSI-2 support:
>> >> >> >> >> >
>> >> >> >> >> > https://www.digikey.com/en/blog/webinar-introduction-raspberry-pi-imaging-and-computer-vision-tools
>> >> >> >> >> >
>> >> >> >> >> > ST seem to have very recently unveiled the STM32N6x7 (datasheet v1
>> >> >> >> >> > November 2024) and STM32N6x5 (datasheet link gives 404 error)
>> >> >> >> >> > Cortex-M55 μC with MIPI CSI-2 support:
>> >> >> >> >> >
>> >> >> >> >> > https://www.st.com/en/microcontrollers-microprocessors/stm32n6-series.html
>> >> >> >> >> >
>> >> >> >> >> >> the i.MX8MP, for instance, has a Cortex M core
>> >> >> >> >> >
>> >> >> >> >> > The STM32MP25 also includes a Cortex M33 - so the i.MX8MP and STM32MP25
>> >> >> >> >> > raise the theoretical possibility of having both a low-power μC
>> >> >> >> >> > solution and a more power hungry but much more flexible μP solution
>> >> >> >> >> > that both use the same silicon - avoiding the potentially costly
>> >> >> >> >> > headache of CritterCam maintaining seperate low power μC and high
>> >> >> >> >> > flexibility μP forks - Cortex M can't support CSI-2 so would be limited
>> >> >> >> >> > to simpler camera modules.
>> >> >> >> >
>> >> >> >> > Whether or not you can control the camera hardware (CSI-2 receiver, ISP,
>> >> >> >> > DMA engines, ...) from a Cortex M core depends on the integration of
>> >> >> >> > those peripherals in a particular SoC, and thus varies between SoC
>> >> >> >> > models. It's not an intrinsic property of Cortex M that it can't support
>> >> >> >> > CSI-2.
>> >> >> >> >
>> >> >> >> >> >> I presume you would have more luck asking on the rpi forums.
>> >> >> >> >> >
>> >> >> >> >> > Every time the need for a low power sleep or suspend mode has been
>> >> >> >> >> > raised on the RPi forums the answer has been a very decisive "No.
>> >> >> >> >> > Switch it off and reboot." - this is a massive problem that renders RPi
>> >> >> >> >> > SBCs effectively useless for any power sensitive application - it lead
>> >> >> >> >> > me to look at silicon from ST and NXP instead. I keep hoping that
>> >> >> >> >> > something might appear from RPi but nothing has.
>> >> >> >> >> >
>> >> >> >> >> >> Yes, to realize and tune a camera system you need knowledge of both
>> >> >> >> >> >> the ISP, the image sensor and the optics.
>> >> >> >> >> >>
>> >> >> >> >> >> If a vendor instead provide tuning data pre-calculated for
>> >> >> >> >> >> a specific set of camera modules designed to work with their platforms
>> >> >> >> >> >> then yes, you would have a choice of pre-tuned cameras to pick from.
>> >> >> >> >> >
>> >> >> >> >> > Thinking about this - I used to work in the physics, chemistry and
>> >> >> >> >> > nonlinear optics of optoelectronic materials - I don't know if this
>> >> >> >> >> > could be any help or not - the thought went through my mind of
>> >> >> >> >> > developing a small optical bench on which an image sensor and any
>> >> >> >> >> > associated optics could be placed then red, green, blue and IR lasers
>> >> >> >> >> > modulated and scanned over both it's surface and possibly a known
>> >> >> >> >> > reference image sensor while a computer reads data from it so that
>> >> >> >> >> > precise calibration and tuning data for that particular make and model
>> >> >> >> >> > of image sensor and any associated optics could be determined and open
>> >> >> >> >> > sourced?
>> >> >> >> >
>> >> >> >> > Calibration and tuning for camera modules requires a test bench, but I
>> >> >> >> > don't think lasers would help. See chapter 6 of
>> >> >> >> > https://datasheets.raspberrypi.com/camera/raspberry-pi-camera-guide.pdf
>> >> >> >> > for an example of a tuning procedure and the corresponding tools and
>> >> >> >> > environment.
>> >> >> >> >
>> >> >> >> >> > I don't know if this would be of use or not (perhaps using
>> >> >> >> >> > this data to train a simple NN could be a way of implementing
>> >> >> >> >> > calibration and tuning algorithms if NPUs or GPUs become fast enough -
>> >> >> >> >> > NPUs and GPUs might be good at handling the simple but highly parallel
>> >> >> >> >> > tensor operations required without proprietory tie-in)?
>> >> >> >> >> >
>> >> >> >> >> > This might allow some flaws and limitations inherent in the
>> >> >> >> >> > manufacturing of the sensor and optics to be compensated for through
>> >> >> >> >> > firmware tuning and calibration?
>> >> >> >> >
>> >> >> >> > Flaws such as defective pixels, or intrinsic physical properties of the
>> >> >> >> > camera module that degrate the image quality (noise, lens shading, ...)
>> >> >> >> > are typically corrected by the ISP, based on calibration and tuning
>> >> >> >> > data.
>> >> >> >> >
>> >> >> >> >> >> The implementation of
>> >> >> >> >> >> the algorithms, how they use and combine features of different IPs on
>> >> >> >> >> >> the platforms (gpu, other accelerators etc) and the tuning of the whole
>> >> >> >> >> >> camera pipeline is usually what SoC vendors compete on.
>> >> >> >> >> >
>> >> >> >> >> >> Sensor driver manufacturers are not involved when it comes to ISP
>> >> >> >> >> >> algorithms development. Knowledge of the ISP, its design and
>> >> >> >> >> >> documentation are usually only accessible to the SoC vendors (again
>> >> >> >> >> >> apart from RPi that fully documents their ISP publicly).
>> >> >> >> >> >
>> >> >> >> >> > Thank you ver much - it's a big help for me to understand that - so
>> >> >> >> >> > basically one way that businesses like ST and NXP compete with each
>> >> >> >> >> > other is by competing to provide the best ISP algorithms as well as the
>> >> >> >> >> > best silicon?
>> >> >> >> >> >
>> >> >> >> >> >> The RPi pipeline handler implements a selection logic to pick the
>> >> >> >> >> >> "best"
>> >> >> >> >> > sensor configuration given a desired output streams configuration:
>> >> >> >> >> > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/pipeline/rpi/common/pipeline_base.cpp#n937
>> >> >> >> >> >
>> >> >> >> >> >> libcamera offers an API to override the pipeline handler choice, so I
>> >> >> >> >> >> would say your application is free to chose any configuration
>> >> >> >> >> >> offered by the sensor's driver.
>> >> >> >> >> >
>> >> >> >> >> > Thank you very much - that's an enormous help - I'd been confused by
>> >> >> >> >> > that. Typically commercial wildlife cameras boast the number of pixels
>> >> >> >> >> > - often including nonexistent "interpolated pixels" - whereas the big
>> >> >> >> >> > limiting factor on the image quality is the signal to noise ratio on
>> >> >> >> >> > those pixels so binning may be very valuable - particularly in low
>> >> >> >> >> > light conditions or when processing power and storage are limited.
>> >> >> >> >> >
>> >> >> >> >> >> Some vendors
>> >> >> >> >> >> decide to distribute their "advanced" algorithms as binaries through
>> >> >> >> >> >> different channels and support their platforms in mainline libcamera with
>> >> >> >> >> >> a reduced number of features in the source implementation.
>> >> >> >> >> >
>> >> >> >> >> > Why don't they contribute their "advanced" proprietory features as
>> >> >> >> >> > obfuscated and compiled binary plugins for libcamera?
>> >> >> >> >
>> >> >> >> > They can do that, and some vendors are looking at this for their newest
>> >> >> >> > SoCs. I believe ST will have a closed-source plugin (we call those IPA
>> >> >> >> > modules in libcamera, for Image Processing Algorithms) for the STM32MP2
>> >> >> >> > (don't quote me on that though). There will be more vendors adopting
>> >> >> >> > that approach.
>> >> >> >> >
>> >> >> >> >> >> Maybe that's why I've never seen a flying squirrel in Nuuksio :-(
>> >> >> >> >> >
>> >> >> >> >> > The bad news is that their numbers have been dropping very rapidly. The
>> >> >> >> >> > good news is that research into Pteromys volans by Ralf Wistbacka and
>> >> >> >> >> > others and into Red Squirrels, Hazel Dormice and Garden Dormice by
>> >> >> >> >> > Goedele Verbeylen, Thomas Briner and others has identified lack of
>> >> >> >> >> > suitable nesting holes as a significant cause for this decline and that
>> >> >> >> >> > we now have very fast, precise and scalable chainsaw and rotary carving
>> >> >> >> >> > techniques to create nest holes for them. The other good news is that
>> >> >> >> >> > Goedele Verbeylen and her colleagues have brought Garden Dormice back
>> >> >> >> >> > from the brink of extinction in Belgium, proving that it can be done.
>> >> >> >> >> >
>> >> >> >> >> >> If you need to
>> >> >> >> >> >> operate with really low power, keeping the Cortex A cores and DRAM
>> >> >> >> >> >> powered, even in low-power mode, may go over your power budget.
>> >> >> >> >> >
>> >> >> >> >> > Yup. I'm keeping a close eye on reported power consumption for SoMs and
>> >> >> >> >> > I'll measure power consumption. I'm also trying to keep everything
>> >> >> >> >> > small to fit into a small amount of RAM and tentatively rejecting any
>> >> >> >> >> > board that uses DDR4 instead of LPDDR4.
>> >> >> >> >> >
>> >> >> >> >> > The power available is limited by the amount of sun, size of solar
>> >> >> >> >> > panels, size of rechargeable batteries and battery chemistry (Li and
>> >> >> >> >> > LiFePO4 batteries work well in summer but only PbA batteries work below
>> >> >> >> >> > a few °C). It may be that tiny wind generators could be added in
>> >> >> >> >> > difficult conditions. Like any tree dwelling creature, CritterCam has
>> >> >> >> >> > to be very light, flexible and adaptable to be successful.
>> >> >> >> >> >
>> >> >> >> >> > Simply by allowing the battery, solar panel and the μP to be in
>> >> >> >> >> > seperate boxes, allowing videos and photos to be optionally compressed
>> >> >> >> >> > and allowing WiFi or Ethernet to be switched on to download data,
>> >> >> >> >> > CritterCam will remove the need for conservationists and researchers to
>> >> >> >> >> > risk their lives on wobbly ladders changing batteries and SD cards.
>> >> >> >> >
>> >> >> >> > Modular designs are always tempting, but they come at a cost. I would
>> >> >> >> > advice, for the first version, focussing on the components where
>> >> >> >> > modularity would bring the best added value, and keeping the other
>> >> >> >> > components in a more monolithic design. An external solar panel seems to
>> >> >> >> > make sense, and making the connection waterproof shouldn't be too
>> >> >> >> > difficult. An external camera module, on the other hand, would be more
>> >> >> >> > difficult, with (in my opinion) less added value. You could still make
>> >> >> >> > the design within the main enclosure modular to let users pick among
>> >> >> >> > different camera sensors, or decide what communication module(s) they
>> >> >> >> > want to install.
>> >> >> >> >
>> >> >> >> >> > An energy and power meter I ordered should arrive in a week or two and
>> >> >> >> >> > I'll start measuring how much power a solar panel can give in the
>> >> >> >> >> > typical worst case of a forest floor.
>> >> >> >> >> >
>> >> >> >> >> > When things get really desperate an optional GSM module could be
>> >> >> >> >> > switched on to send a simple SMS or MQTT message to ask someone to come
>> >> >> >> >> > and change a low battery for a fully charged one and consider adding
>> >> >> >> >> > more solar panels.
>> >> >> >> >> >
>> >> >> >> >> >> C-PHY is more recent, it can provide the same bandwidth with a lower
>> >> >> >> >> >> number of signals, but isn't as widely supported (I expect that to
>> >> >> >> >> >> improve over time).
>> >> >> >> >> >
>> >> >> >> >> > That's good to know - C-PHY's 3 level signalling must be giving big
>> >> >> >> >> > headaches for silicon designers used to working with digital signals.
>> >> >> >> >> > Good to know that for the time being D-PHY is the best supported.
>> >> >> >> >> >
>> >> >> >> >> >> CSI-3 was killed by Intel and Qualcomm before it reached the market
>> >> >> >> >> >
>> >> >> >> >> > That's good to know. One less thing to keep track of. CSI-1 and CSI-3
>> >> >> >> >> > RIP.
>> >> >> >> >> >
>> >> >> >> >> > Will
>> >> >> >> >> >
>> >> >> >> >> > On 2024-12-22 10:30, Laurent Pinchart wrote:
>> >> >> >> >> >> Hi Will,
>> >> >> >> >> >>
>> >> >> >> >> >> On Fri, Dec 20, 2024 at 11:04:09AM +0000, w.robertson at cairnwater.com wrote:
>> >> >> >> >> >>> Hello Laurent and Kieran,
>> >> >> >> >> >>>
>> >> >> >> >> >>> > This sounds very interesting. We rarely get contacted by people working
>> >> >> >> >> >>> > on projects related to animals or nature. I live in Finland, so I would
>> >> >> >> >> >>> > be happy to help the flying squirrels from Nuuksio :-)
>> >> >> >> >> >>>
>> >> >> >> >> >>> That's wonderful! Dormouse species hibernate but the flying squirrels
>> >> >> >> >> >>> can't hibernate and so must have nest holes which provide protection
>> >> >> >> >> >>> from weather and allow them to build well insulated nests to survive the
>> >> >> >> >> >>> winter, they also ideally need nest holes with entrances which are the
>> >> >> >> >> >>> right size to let them in but keep the larger red squirrels out. Past
>> >> >> >> >> >>> forestry practice was to remove dead trees and trees with holes (which
>> >> >> >> >> >>> were seen as unsightly or unstable) and this has left a drastic shortage
>> >> >> >> >> >>> of suitable nest holes for the flying squirrels.
>> >> >> >> >> >>
>> >> >> >> >> >> Maybe that's why I've never seen a flying squirrel in Nuuksio :-(
>> >> >> >> >> >>
>> >> >> >> >> >>> Most of my research is unpaid (with occasional small sponsorship here
>> >> >> >> >> >>> and there) so resources have to be used very efficiently.
>> >> >> >> >> >>>
>> >> >> >> >> >>> By developing new rotary boring and precision chainsaw carving
>> >> >> >> >> >>> techniques we can carve nest holes entirely through a single narrow,
>> >> >> >> >> >>> very precise entrance with a much larger nesting chamber behind.
>> >> >> >> >> >>>
>> >> >> >> >> >>> > I'll try to help, in my (unfortunately limited) free time. The best
>> >> >> >> >> >>> > option, if compatible with your needs, would be to have open
>> >> >> >> >> >>> > discussions
>> >> >> >> >> >>> > either on the libcamera development mailing list, or on the project's
>> >> >> >> >> >>> > IRC channel (you can find contact information in
>> >> >> >> >> >>> > https://libcamera.org/contributing.html). That way other developers
>> >> >> >> >> >>> > will also be able to chime in.
>> >> >> >> >> >>>
>> >> >> >> >> >>> That would be wonderful!
>> >> >> >> >> >>>
>> >> >> >> >> >>> > libcamera doesn't currently support the STM32MP2, but ST is working on
>> >> >> >> >> >>> > it. They showcased a working prototype at Embedded World in April this
>> >> >> >> >> >>> > year, and I assume they will provide a version usable in production
>> >> >> >> >> >>> > soon. While they haven't started upstreaming much of their code in
>> >> >> >> >> >>> > libcamera, I believe they plan to do so. The platform is therefore
>> >> >> >> >> >>> > interesting for low-power applications.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Thank you very much - I'd been unsure about that - that's very helpful
>> >> >> >> >> >>> to know - I'll maybe try to find a way to give ST some gentle
>> >> >> >> >> >>> encouragement in the spring - they're pushing aggressively to sell both
>> >> >> >> >> >>> their image sensors and STM32MP2 so libcamera support should be a high
>> >> >> >> >> >>> strategic priority for them. I get the feeling that having launched
>> >> >> >> >> >>> STM32MP2 into both industrial control and domestic appliance markets
>> >> >> >> >> >>> they've got a fairly heavy workload at the moment and the STM32MP257F-DK
>> >> >> >> >> >>> dev. board is due to be launched later than planned.
>> >> >> >> >> >>>
>> >> >> >> >> >>> > The i.MX8MP is much better supported in libcamera at the moment, and we
>> >> >> >> >> >>> > are actively improving its support. There are low-cost development
>> >> >> >> >> >>> > boards, such as the Debix Model A that we use for development (note that
>> >> >> >> >> >>> > its camera connector is not standard, but they have a Debix I/O board
>> >> >> >> >> >>> > that can interface the Model A camera connector to a Raspberry Pi 15
>> >> >> >> >> >>> > pins connector).
>> >> >> >> >> >>>
>> >> >> >> >> >>> Thank you very much! That's wonderful to have a good dev. board! I'd
>> >> >> >> >> >>> been looking at i.MX8MP SoMs and SiPs but some of them still seem to be
>> >> >> >> >> >>> in the design stage with some technical information missing from data
>> >> >> >> >> >>> sheets. Will Debix Model A's LPDDR4 allow it to quickly enter and waken
>> >> >> >> >> >>> from a low power sleep mode?
>> >> >> >> >> >>
>> >> >> >> >> >> I haven't experienced with low power sleep and wake up time myself, so I
>> >> >> >> >> >> can't tell.
>> >> >> >> >> >>
>> >> >> >> >> >> Generally speaking, it all depends on your power budget. If you need to
>> >> >> >> >> >> operate with really low power, keeping the Cortex A cores and DRAM
>> >> >> >> >> >> powered, even in low-power mode, may go over your power budget. Other
>> >> >> >> >> >> options are possible, such as turning power to the Cortex A and DRAM
>> >> >> >> >> >> off, and handling fast wake up tasks in a smaller core (the i.MX8MP, for
>> >> >> >> >> >> instance, has a Cortex M core that could be used for this). The wake up
>> >> >> >> >> >> time gets significantly increased, as Linux will need to boot from
>> >> >> >> >> >> scratch. You may be able to start powering up the camera sensor in
>> >> >> >> >> >> parallel from the Cortex M core to save some time, or even drive the
>> >> >> >> >> >> camera completely from the Cortex M, but that would require porting
>> >> >> >> >> >> libcamera (and drivers) to whatever OS you would be running there, which
>> >> >> >> >> >> would require really significant effort.
>> >> >> >> >> >>
>> >> >> >> >> >>> The 4 lane MIPI CSI seems potentially more
>> >> >> >> >> >>> flexible than the STM32MP2's 2 lane CSI-2 interface.
>> >> >> >> >> >>
>> >> >> >> >> >> 4 lanes will give you more bandwidth, but unless you target really high
>> >> >> >> >> >> resolutions at high frame rates, that shouldn't be necessary.
>> >> >> >> >> >>
>> >> >> >> >> >>> I guess the best way for me to get a Debix Model A or B is just to order
>> >> >> >> >> >>> it from RS?
>> >> >> >> >> >>
>> >> >> >> >> >> I believe so.
>> >> >> >> >> >>
>> >> >> >> >> >>> (One thing that's confusing me - when modern data sheets like the Debix
>> >> >> >> >> >>> Model A data sheet say "MIPI CSI 4-lane" I guess they mean "MIPI CSI-2"?
>> >> >> >> >> >>> I'd heard of CSI-1 which supports only 1 lane and seems old and rarely
>> >> >> >> >> >>> used now, of CSI-2 which supports multiple lanes and theoretically a
>> >> >> >> >> >>> choice of C-PHY of D-PHY physical layers (though I guess many CSI-2
>> >> >> >> >> >>> implementations maybe don't implement C-PHY because of its complexity?)
>> >> >> >> >> >>> for the physical layer and CSI-3 which is based on UniPro and rarely
>> >> >> >> >> >>> implemented at the moment.)
>> >> >> >> >> >>
>> >> >> >> >> >> Today the terms "MIPI CSI" (or sometimes just "MIPI camera") refers to
>> >> >> >> >> >> CSI-2. CSI-1 has retired a very long time ago, and CSI-3 was killed by
>> >> >> >> >> >> Intel and Qualcomm before it reached the market (there were some
>> >> >> >> >> >> prototype implementations, but nothing in mass production).
>> >> >> >> >> >>
>> >> >> >> >> >> D-PHY is the most common PHY for CSI-2, and should be supported
>> >> >> >> >> >> everywhere. C-PHY is more recent, it can provide the same bandwidth with
>> >> >> >> >> >> a lower number of signals, but isn't as widely supported (I expect that
>> >> >> >> >> >> to improve over time). I don't think the choice of PHY matters too much
>> >> >> >> >> >> in your case, whatever is supported by both the camera sensor and the
>> >> >> >> >> >> SoC will be fine.
>> >> >> >> >> >>
>> >> >> >> >> >>> Thank you very much for all your help!
>> >> >> >> >> >>>
>> >> >> >> >> >>> Will
>> >> >> >> >> >>>
>> >> >> >> >> >>> On 2024-12-19 23:17, Laurent Pinchart wrote:
>> >> >> >> >> >>> > Hello Will,
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > (CC'ing my colleague Kieran)
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > On Thu, Dec 19, 2024 at 07:46:53PM +0000, w.robertson at cairnwater.com wrote:
>> >> >> >> >> >>> >> Hi Laurent
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Thank you very much for your video "Giving Linux a Camera Stack:
>> >> >> >> >> >>> >> libcamera's 3 Years Journey and Exciting Future" and the wonderful work
>> >> >> >> >> >>> >> of you and the libcamera developers in bringing such a powerful and
>> >> >> >> >> >>> >> structured approach to such an extremely difficult and complex
>> >> >> >> >> >>> >> problem.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > You're more than welcome.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >> I work as a climbing arborist in practice and research and software
>> >> >> >> >> >>> >> engineer. My research is mostly voluntary and focused on new minimally
>> >> >> >> >> >>> >> invasive techniques to carve critically needed nest holes for endangered
>> >> >> >> >> >>> >> tree dwelling dormouse species (Hazel Dormouse, Garden Dormouse, Forest
>> >> >> >> >> >>> >> Dormouse), bat species and in Finland the European Flying Squirrel
>> >> >> >> >> >>> >> (Pteromys volans).
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> To overcome problems and limitations with commercial wildlife cameras
>> >> >> >> >> >>> >> for small mammals I started work on CritterCam which - if I can get it
>> >> >> >> >> >>> >> to work - will be a fully open source wildlife camera system which is
>> >> >> >> >> >>> >> vastly better, vastly more flexible, vastly more independent in the
>> >> >> >> >> >>> >> field and significantly cheaper than anything commercially available at
>> >> >> >> >> >>> >> the moment. Working with embedded Linux and with several different
>> >> >> >> >> >>> >> manufacturers for the microprocessor and image sensors for CritterCam
>> >> >> >> >> >>> >> means that CritterCam users can never be price gouged and CritterCam
>> >> >> >> >> >>> >> will be able to adapt to update all of its critical components with
>> >> >> >> >> >>> >> newer alternatives well into the future.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> This is an article with some videos of Garden Dormice exploring our
>> >> >> >> >> >>> >> carved nest holes published in Flemish yesterday by Goedele Verbeylen:
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> https://www.natuurpunt.be/nieuws/boomholtes-kerven-voor-bedreigde-eikelmuizen
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> This is a short summary that I wrote in English and German:
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> https://drive.google.com/file/d/1jkFA-PgDr-O9-nGSgODCUFkukj63F4fc/view?usp=sharing
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> We have a small website at:
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> new-homes-for-old-friends.cairnwater.com
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > This sounds very interesting. We rarely get contacted by people working
>> >> >> >> >> >>> > on projects related to animals or nature. I live in Finland, so I would
>> >> >> >> >> >>> > be happy to help the flying squirrels from Nuuksio :-)
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >> My background is mainly server side so I don't have kernel experience
>> >> >> >> >> >>> >> and I was wondering if you or another libcamera might be willing to
>> >> >> >> >> >>> >> give
>> >> >> >> >> >>> >> me any pointers to help me get started using supported MIPI CSI-2
>> >> >> >> >> >>> >> cameras with embedded Linux or ANdroid?
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > I'll try to help, in my (unfortunately limited) free time. The best
>> >> >> >> >> >>> > option, if compatible with your needs, would be to have open
>> >> >> >> >> >>> > discussions
>> >> >> >> >> >>> > either on the libcamera development mailing list, or on the project's
>> >> >> >> >> >>> > IRC channel (you can find contact information in
>> >> >> >> >> >>> > https://libcamera.org/contributing.html). That way other developers
>> >> >> >> >> >>> > will
>> >> >> >> >> >>> > also be able to chime in.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >> I'm hoping to build a first prototype using the STM32MP257F-DK dev.
>> >> >> >> >> >>> >> board based on the STM32MP257 μP when it becomes available in January
>> >> >> >> >> >>> >> but I'm also looking at SoMs and SiPs based on other μPs like the NXP
>> >> >> >> >> >>> >> i.MX 8M Plus that support a CSI-2 interface and a low power sleep mode
>> >> >> >> >> >>> >> -
>> >> >> >> >> >>> >> I can also do some initial testing using a Raspberry Pi SBC.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > libcamera doesn't currently support the STM32MP2, but ST is working on
>> >> >> >> >> >>> > it. They showcased a working prototype at Embedded World in April this
>> >> >> >> >> >>> > year, and I assume they will provide a version usable in production
>> >> >> >> >> >>> > soon. While they haven't started upstreaming much of their code in
>> >> >> >> >> >>> > libcamera, I believe they plan to do so. The platform is therefore
>> >> >> >> >> >>> > interesting for low-power applications.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > The i.MX8MP is much better supported in libcamera at the moment, and we
>> >> >> >> >> >>> > are actively improving its support. There are low-cost development
>> >> >> >> >> >>> > boards, such as the Debix Model A that we use for development (note
>> >> >> >> >> >>> > that
>> >> >> >> >> >>> > its camera connector is not standard, but they have a Debix I/O board
>> >> >> >> >> >>> > that can interface the Model A camera connector to a Raspberry Pi 15
>> >> >> >> >> >>> > pins connector).
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > Raspberry Pi platforms are also well supported, with the work being
>> >> >> >> >> >>> > performed by Raspberry Pi.