[libcamera-devel] [RFC] Android 3A interface and libcamera design
Hanlin Chen
hanlinchen at chromium.org
Thu Feb 17 14:54:53 CET 2022
Hello,
This mail is to discuss libcamera and Android 3A interface, inspired by
David's AF and Paul's AE series. And maybe as a preparation for the design
for AWB. There was a great discussion on the mailing list about how it can
be translated back to Android. Because Android's 3A definition is somehow
obscure and inconsistent among 3A. I think it might be good to have an
overview of its problem and implications, for easier discussion and
reference.
The article will try to have an overview among 3A in Android Camera3,
especially its ambiguity, by categorizing the use cases of 3A in Android.
And then to discuss possible design for libcamera as a super set, having a
natural translation to Android, and to see if there is something mandatory
or missing.
Note that there are tables in text which need a fixed-width font for a
better view.
I will refer to the 3A definition in Android from the following documents.
https://android.googlesource.com/platform/system/media/+/master/camera/docs
https://source.android.com/devices/camera/camera3_3Amodes
In the Android Camera3, there seems to be 4 kinds of use cases that need
interaction with the application, and have extensions on controls and
states to take care of them. To help further the discussion, I will try to
list and describe them briefly, and refer to them with A), B), C), D) for
later context. The terms used in Android are sometimes ambiguous. I will
try to introduce names to clarify if needed.
A) Avoid flickering when changing between Auto mode and Manual mode:
I quote Android's definition for AE here: "Since the camera device has a
pipeline of in-flight requests, the settings that get locked do not
necessarily correspond to the settings that were present in the latest
capture result received from the camera device, since additional captures
and AE updates may have occurred even before the result was sent out. If an
application is switching between automatic and manual control and wishes to
eliminate any flicker during the switch, the following procedure is
recommended:"
1. Starting in auto-AE mode:
2. Lock AE
3. Wait for the first result to be output that has the AE locked
4. Copy exposure settings from that result into a request, set the request
to manual AE
5. Submit the capture request, proceed to run manual AE as desired.
One may guess it's similar when changing from Manual to Auto. However, it's
not defined clearly in the document:
1. Starting in Manual-AE mode:
2. Set a request to Auto-AE mode and AE lock on.
3. Wait for the first result to be output that has the AE locked
4. Unlock the AE, so AE can resume from where it was.
Note that AWB has a similar definition, but no corresponding definition for
AF.
In the definition, the word "Lock" means the "locking down" the result of
AE/AWB in Auto mode.
The following context will use the word as such. Sometimes Android uses
"Lock" as a different meaning.
I will use "Metering Done" for that case to avoid ambiguity. See use case
B) for details.
B) Metering a good state in Auto mode quickly and fix done for capture
The use case is defined by the usage of AEPrecaptureTrigger for AE-Auto
mode and AFTrigger for AF-ContinuosVideo/Still mode. The application
assumes the algorithm is in a mode, which updates its results continuously
for preview. When the application needs a stable state to take a picture,
it requests the algorithm to find a good state quickly (good exposure or
lens position), and stay in the state. Application will read the frame, and
then ask the algorithm to resume to continuous mode for preview. Note that
the searching should avoid doing a full scan, but take advantage of current
internal states, i.e., if the algorithm is already doing a search, it
should follow the momentum to find the good state quickly.
Take AEPrecapture as an example. The application uses the following
sequence:
1. Starting in auto-AE mode:
2. Set AEPrecaptureTrigger as START
3. Wait for the metering process done, and output AEState as Converged or
FLASH_REQUIRED
4. Take a picture
5. Set AEPrecaptureTrigger as CANCEL to resume the AE process
For AF, AFTrigger with AF-ContinuosVideo/Still mode is the same use case.
The kind of use case implies an internal state for the metering progress.
When the progress is done, the algorithm fixes the result of the algorithm,
and waits for a resume.
The word and definition for the progress are very unclear. In AF, Android
extends the AFState to indicate that the progress is done and result is
fixed, by two special states: FOCUSED_LOCKED, and NOT_FOCUSED_LOCKED.
However, the meaning of "Lock" is different from the case A), which is an
immediate lock from application, instead of a state reported from the
algorithm.
For AE, it's even worse. The document defines an internal lock which is not
reporting back to the application, and assumes the application will
understand and cancel the process. There are three ways to cancel the
internal lock. I quote them them here:
"When a precapture metering sequence is finished, the camera device may
lock the auto-exposure routine internally to be able to accurately expose
the subsequent still capture image (android.control.captureIntent ==
STILL_CAPTURE). For this case, the AE may not resume normal scan if no
subsequent still capture is submitted. To ensure that the AE routine
restarts normal scan, the application should submit a request with
android.control.aeLock == true, followed by a request with
android.control.aeLock == false, if the application decides not to submit a
still capture request after the precapture sequence completes.
Alternatively, for API level 23 or newer devices, the CANCEL can be used to
unlock the camera device internally locked AE if the application doesn't
submit a still capture request after the AE precapture trigger. Note that
the CANCEL was added in API level 23, and must not be used in devices that
have earlier API levels."
To clarify the ambiguity, I will use the word "Metering" for this kind of
process. And use "Metering Done" for the fixed state at the end of the
metering, to distinguish "Lock" from the use case A).
Note that AWB doesn't have a similar definition.
C) Trigger a full scan, and fix done
For AF only, AFTrigger defines the use case with AF-Auto/Macro mode. The
case is similar to B). However, the algorithm doesn't run continuously, but
only does a full scan when the application triggers it. Like the use case
B), it will fix at a good state for the application to take pictures, and
resume its state when the application cancels the progress.
Take AFTrigger as an example. The application uses the following sequence:
1. Starting in AF-Auto mode:
2. Set AFTrigger as START for a single request
3. Wait for the scan process done, and output AFState as FOCUSED_LOCKED or
NOT_FOCUSED_LOCKED
4. Take a picture
5. Set AFTrigger as CANCEL to restore its state to AF_STATE_INACTIVE
Like use case B), Android extends the AFState to indicate that the scan
progress is done and result is fixed, by two special states:
FOCUSED_LOCKED, and NOT_FOCUSED_LOCKED. Moreover, it adds another AFState
AF_STATE_ACTIVE_SCAN to indicate AF is performing the full scan, instead of
quick metering of use case B).
Note that AE and AWB don't have the use case.
D) Combine of Lock and Metering/Fullscan
Although it sounds strange, Android allows a combination of Lock and
Metering/Fullscan.
It's tricky, and implied in the AELock definition. I quote it from the
document:
"If AE precapture is triggered (see android.control.aePrecaptureTrigger)
when AE is already locked, the camera device will not change the exposure
time (android.sensor.exposureTime) and sensitivity
(android.sensor.sensitivity) parameters. The flash may be fired if the
android.control.aeMode is ON_AUTO_FLASH/ON_AUTO_FLASH_REDEYE and the scene
is too dark."
It implies if Lock and metering both exist, the Lock has higher priority
and will lock the result controls (exposure, sensitivity) from the
algorithm. On the other hand, the metering process will still continue
until fixing the state as in B), even if it cannot change the controls.
However, the metering process can change things which are not locked, like
flash for AE in Android.
Also, according to "The Great AE Series", we can lock (in its term,
Disabled) exposure time and analog gain independently.
Thus, I imagine there is a more generic way for the use case, for example.
The application can lock the analog gain, but the exposure time, and then
start a metering process. Thus, the AE algorithm can only change the
exposure time to find a good brightness as such:
1. Starting in AE-Manual mode
2. Lock analog gain with a fixed value
3. Start the AE Metering process
4. Wait for the metering process done, and output AEState as Converged
5. Take a picture
6. Unlock the analog gain
7. Cancel metering process to resume the AE for preview
I will refer to the four use cases as A), B), C). D), and I conclude them
as the following table:
+--------------------+-----------+--------------------------+---------------+----------------+
| Algorithm/Use Case | (A) | (B) | (C)
| (D) |
+--------------------+-----------+--------------------------+---------------+----------------+
| AE | AELock + | AEPrecapture + | None
| AELock + |
| | AE-Auto | AE-Auto mode |
| AEPrecapture + |
| | mode | |
| AE-Auto mode |
+--------------------+-----------+--------------------------+---------------+----------------+
| AF | None | AFTrigger + | AFTrigger +
| None |
| | | AF-ContinuousVideo/Still | AF-Auto/Macro
| |
| | | mode | mode
| |
+--------------------+-----------+--------------------------+---------------+----------------+
| AWB | AWBLock + | None | None
| None |
| | AWB-Auto | |
| |
| | mode | |
| |
+--------------------+-----------+--------------------------+---------------+----------------+
I can see the supported use cases are not consistent with all algorithms.
For example, AF doesn't have lock function, AWB doesn't have Metering
function, AE and AWB doesn't have full scan function, etc. The reason seems
to be that Android doesn't think all of them are useful, and historically,
each function is added incrementally.
In theory, the corresponding design for libcamera should be a superset for
that of the Android Camera3, and covers all supported cases above. This
left me with a question: whether it's useful to define an interface
consistent to all algorithms, covering missing cases in the table? In more
detail, the questions to ask are as follows, and some thoughts.
1. Should we add Lock for AF to support A) and D)?
This seems to be reasonable to me. If we'd like to avoid flickers in AE
and AWB, we'd like to avoid it in AF too.
2. Should we add Full Scan for AE to support C)? Should we add Metering and
FullScan for AWB to support B) C) D)?
This seems to depend on the implementation of AE and AWB. Need some
advice here. In theory, the interface can define them and leave the
implementation detail to 3A provider, i.e., Metering and Full Scan may have
the same implementation internally.
3. Metering and FullScan are similar. Should they share the same interface?
If so, how to differentiate Metering and FullScan? How would this affect
the translation to Android?
In Android, it differentiates Metering and FullScan by extending AFMode.
"ContinuousVideo" and "ContinuousStill" will do Metering when AFTrigger is
started, but "Auto" and "Macro" will do Full Scan instead. In David's
design, it is further divided into sub modes. I think it's good to follow
the approach. Another way is to add an extra flag to distinguish them.
Another problem is that Android extends AFState to express the progress
of Metering/FullScan. AF_STATE_ACTIVE_SCAN for FullScanning, FOCUSED_LOCKED
and NOT_FOCUSED_LOCKED for "Metering Done". Since in B) and C), it's a
state for application to wait, the corresponding state should be added.
However, the information seems to be orthogonal to the AFState. Maybe we
can add an extra control for it, instead of extending AFState.
For AFState translation to Android. If we merge definitions of Metering
and FullScan, and differentiate them by modes or a flag, there will be a
special case for translating AF_STATE_ACTIVE_SCAN and
AF_STATE_PASSIVE_SCAN. See the AFState combination table in later context.
Some Pros and Cons of adding missing use cases to all 3A:
Pros:
1. The interface can be consistent for all algorithms, since we don't
need to rule out a function in AF, but another one in AE/AWB.
2. Easier to understand and document common behavior.
3. Application can treat 3A in a consistent way.
Cons:
1. Maybe not all cases are meaningful, and the details would be left to
the 3A implementor. For example, what's the difference between Metering and
Full scan on AWB? Could they be the same implementation internally?
I think they are up for discussion. Still, based on the above, I draw out a
most generic definition to cover everything, and check if there is anything
missing. And then to see the influence on simplifying each case. The
following are the controls I think needed.
# Input Controls:
1. We need controls to choose Manual or Auto mode in A), for each algorithm
enum AFMode: {Manual, Auto}
enum AEMode: {Manual, Auto}
enum AWBMode: {Manual, Auto}
2. We need controls to hint the internal behavior for each algorithm
For AF:
enum AFHint: {ContinuousVideo, ContinuousStill, OnTrigger, MacroOnTrigger,
...}
Or a sub mode version:
enum AFHintMode: {Continuous, OnTrigger}
enum AFHintRange: {Full, Macro, Normal}
enum AFHintSpeed: {Fast, Normal}
For AWB:
enum AWBHint: {Normal, Incandescent, Fluorescent, ...}
For AE:
enum AEHintTargetFPS: Lower and upper limits of target FPS
3. We need controls to lock outputs of each algorithm
bool AFFocusLock
bool AWBCCMLock
bool AEExposureTimeLock
bool AEAnalogGainLockk
4. Controls to trigger Metering or Full scan
enum AFMeterMethod: {Meter, FullScan}
enum AEMeterMethod: {Meter, FullScan}
enum AWBMeterMethod: {Meter, FullScan}
enum AFMeteringTrigger: {Idle, MeterStart, Cancel}
enum AEMeteringTrigger: {Idle, MeterStart, Cancel}
enum AWBMeteringTrigger: {Idle, MeterStart, Cancel}
# Output Controls from the camera
1. Controls reporting the 3A calculation state.
AFState: {Inactive, Searching, Converged, Fail}
AEState: {Inactive, Searching, Converged, Fail}
AWBState: {Inactive, Searching, Converged, Fail}
2. Controls reporting the Metering/FullScan state.
For all algorithms, when a Metering/FullScan is triggered, the
corresponding MeteringState will change to Metering immediately, and the
corresponding {AF/AE/AWB}State will change to Searching immediately.
When the Metering progress is done, the MeteringState is set to Done and
{AF/AE/AWB}State will be set to Converged or Fail. The MeteringState will
change back to None, when the Metering/FullScan is canceled by the
application.
enum AFMeteringState: {None, Metering, Done}
enum AEMeteringState: {None, Metering, Done}
enum AWBMeteringState: {None, Metering, Done}
3. We need to define valid combinations of a 3A calculation status and a
Metering State.
+--------------------------------+----------+-----------+-----------+------+
| MeteringState/{AF/AE/AWB}State | Inactive | Searching | Converged | Fail |
+--------------------------------+----------+-----------+-----------+------+
| None | Yes | Yes | Yes | Yes |
+--------------------------------+----------+-----------+-----------+------+
| Metering | No | Yes | No | No |
+--------------------------------+----------+-----------+-----------+------+
| Done | No | No | Yes | Yes |
+--------------------------------+----------+-----------+-----------+------+
With the above definition. I can redefine the sequence for the application
to do four use cases above.
I use AE for example, but they should be consistent to 3A.
A) Avoid flickering when changing between Auto mode and Manual mode:
1. Starting in AEMode as "Auto":
2. Set AEExposureTimeLock and AEAnalogGainLock to "true"
3. Wait for the first result to be output that has the locks
4. Copy exposure/gain settings from that result into a request
5. Set the request with AEMode as "Manual"
5. Submit the capture request, proceed to run manual AE as desired.
B) Metering a good state in Auto mode quickly and fix done for capture
1. Starting AEMode as "Auto":
2. Set AEMeterMode as "Meter" and AEMeteringTrigger as "MeterStart"
3. Wait for the metering process done, and output MeteringState as "Done"
4. Take a picture
5. Set AEMeteringTrigger as "Cancel" to resume the AE process
C) Trigger a Full Scan, and fix done for capture
1. Starting AEMode as "Auto":
2. Set AEMeterMode as "FullScan" and AEMeteringTrigger as "MeterStart"
3. Wait for the metering process done, and output MeteringState as "Done"
4. Take a picture
5. Set AEMeteringTrigger as "Cancel" to resume the AE process
D) Combine of Lock and Metering/Fullscan
1. Starting in AEMode as "Manual" with a fixed value for analog gain
2. Set AEAnalogGainLock as "true"
3. Setting AEMode as "Auto", and AEMeteringTrigger as "MeteringStart"
4. Wait for the metering process done, and output AEMeteringState as "Done"
5. Take a picture
2. Set AEAnalogGainLock as "false"
6. Set AEMeteringTrigger as "Cancel" to resume the AE process
# Finally we need to verify, in the defintion, if all states can be
translated to Android without ambiguity, based on Android's requirements.
1. Translate AFState back to Android's version.
Note that the "X" in the table means an invalid combination of
MeteringState and AFState.
Since Android doesn't define Lock for AF, we can AFLock in the table. Note
that when MeteringState is "Metering", we need to check the AFMeterMode to
differentiate AF_STATE_ACTIVE_SCAN and AF_STATE_PASSIVE_SCAN.
+-----------------------------+-------------------+-----------------------+--------------------------+----------------------------+
| MeteringState/AFState | Inactive | Searching |
Converged | Fail |
+-----------------------------+-------------------+-----------------------+--------------------------+----------------------------+
| None | AF_STATE_INACTIVE | AF_STATE_PASSIVE_SCAN |
AF_STATE_PASSIVE_FOCUSED | AF_STATE_PASSIVE_UNFOCUSED |
+-----------------------------+-------------------+-----------------------+--------------------------+----------------------------+
| Metering with FullScan mode | X | AF_STATE_ACTIVE_SCAN |
X | X |
+-----------------------------+-------------------+-----------------------+--------------------------+----------------------------+
| Metering with Meter Mode | X | AF_STATE_PASSIVE_SCAN |
X | X |
+-----------------------------+-------------------+-----------------------+--------------------------+----------------------------+
| Done | X | X |
AF_STATE_FOCUSED_LOCKED | AF_STATE_UNFOCUSED_LOCKED |
+-----------------------------+-------------------+-----------------------+--------------------------+----------------------------+
2. Translate AEState back to Android's version.
Note that the "X" in the table means an invalid combination of
MeteringState and AEState.
Since Android doesn't define Full Scan for AE, we can assume AEMeterMode is
alway "Meter"
The AELock will override the AEState according to Android's definition.
+-------------------------+-------------------+---------------------+--------------------+--------------------+
| MeteringState/AEState | Inactive | Searching |
Converged | Fail |
+-------------------------+-------------------+---------------------+--------------------+--------------------+
| None + Locked | AE_STATE_LOCKED | AE_STATE_LOCKED |
AE_STATE_LOCKED | AE_STATE_LOCKED |
+-------------------------+-------------------+---------------------+--------------------+--------------------+
| None + Unlocked | AE_STATE_INACTIVE | AE_STATE_SEARCHING |
AE_STATE_CONVERGED | AE_STATE_CONVERGED |
+-------------------------+-------------------+---------------------+--------------------+--------------------+
| Metering + Locked | AE_STATE_LOCKED | AE_STATE_LOCKED |
AE_STATE_LOCKED | AE_STATE_LOCKED |
+-------------------------+-------------------+---------------------+--------------------+--------------------+
| Metering + Unlocked | X | AE_STATE_PRECAPTURE | X
| X |
+-------------------------+-------------------+---------------------+--------------------+--------------------+
| Done + Locked | AE_STATE_LOCKED | AE_STATE_LOCKED |
AE_STATE_LOCKED | AE_STATE_LOCKED |
+-------------------------+-------------------+---------------------+--------------------+--------------------+
| Done + Unlocked | X | X |
AE_STATE_CONVERGED | AE_STATE_CONVERGED |
+-------------------------+-------------------+---------------------+--------------------+--------------------+
3. Translate AWBState back to Android's version.
Since Android doesn't define Metering/FullScan for AWB, we can assume
AWBMetering is never triggered.
The AWBLock will override the AWBState according to Android's definition.
+-------------------------+-------------------+--------------------+---------------------+---------------------+
| MeteringState/AWBState | Inactive | Searching |
Converged | Fail |
+-------------------------+-------------------+--------------------+---------------------+---------------------+
| Locked | AWB_STATE_LOCKED | AWB_STATE_LOCKED |
AWB_STATE_LOCKED | AWB_STATE_LOCKED |
+-------------------------+-------------------+--------------------+---------------------+---------------------+
| Unlocked | AE_STATE_INACTIVE | AE_STATE_SEARCHING |
AWB_STATE_CONVERGED | AWB_STATE_CONVERGED |
+-------------------------+-------------------+--------------------+---------------------+---------------------+
I think all Android states are covered. Base on the table, something to
think when simplifying the definition:
1. The {AF/AE}MeteringState is a key to translate {AF/AE}State back to
Android naturally, otherwise, some states will need special care:
AE_STATE_PRECAPTURE, AF_STATE_PASSIVE_FOCUSED, AF_STATE_PASSIVE_UNFOCUSED,
AF_STATE_ACTIVE_SCAN, etc.
2. The {AE/AWB}Lock will override the {AE/AWB}State to its locked state. If
we'd like to remove the lock semantics, the translation of {AE/AWB}State
needs special care. And also need a way to take care of the use case D).
3. Some states/flags can be safely removed without affecting the
translation, for example: Fail state in {AE/AWB}State, and the FullScan
mode for AE and AWB. As mentioned, I really have two minds on whether to
reserve them and let the 3A provider decide the implementation details.
Hope this mail leads to something helpful. Please give me some comments if
you will.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20220217/af862328/attachment-0001.htm>
More information about the libcamera-devel
mailing list