Reliability Requirements

In this page, we first provide more details on the parameter estimation method, then to make the page self contain we remind our procedures 1 and 2 used to generate the requirements, next we provide the proof of the Theorem 1, and finally, we show how Procedure 1 can be adapted for compound decomposable evaluation metrics, such as mAP.

Parameter Estimation

To generate reliability requirements for c-tasks, Procedure 1 takes a list of estimated threshold values ($t_c, t_p$) for each subtask $\mathbf{v}_i$ from the human performance data. The thresholds are estimated following ICRAF estimation method. Specifically, we begin with an estimation of the threshold values ($t_c, t_p$) for each subtask $v_i$ from the human performance data, following the state-of-the-art estimation method (see Section Background). To compute the human-tolerated range of visual changes, state-of-the-art uses the binomial statistical test, which is specific to the task metric. E.g., for the metric Precision-Recall curve $PR$ (used for object detection and instance segmentation), obtained with the human experiment data, the binomial test is performed on each point of the curve, checking if $PR$ for transformed images is below that of the original images with sufficient statistical significance. Since in the empirical studies one does not have the data of all the points of the $PR$ curve, the binomial test is performed only in the curve locations where sufficient data is available. To define binomial test in a point of a $PR$ curve, we need to define two binomial tests, for precision and recall at that point, respectively. Specifically, for each individual binomial test, we follow the procedure of state-of-the-art to estimate the human-tolerated range of visual changes, $\Delta_v$ (see Section Background), and pick the minimum range for the $PR$ curve binomial test, i.e., $\Delta^{PR}_v = min{(\Delta^{prec}_v, \Delta^{rec}_v})$. The resulting $t_c$ and $t_p$ thresholds for the entire $PR$ curve are obtained by considering the visual change $\Delta^{PR}_v$ value at the curve point that has been obtained with the largest amount of human data.

Although not shown in the paper, we also conducted experiments with the transformation brightness and obtained thresholds shown below:

Procedure 1

Procedure 2

Procedure generating subtask requirements

Correctness of our requirement composition

Given a c-task $\mathbf{V} = \mathbf{v}_n \odot ...\mathbf{v}_2 \odot \mathbf{v}_1$ and a decomposable performance metric $M_\mathbf{V}$, such that $M_\mathbf{V} = F(M'_\mathbf{V})$ and $M'_\mathbf{V} = \prod_{i=1}^n m_{\mathbf{v}_i}$, let reqs$=\{$ req$_{\mathbf{v}_n}\}$ be the list of subtask requirements generated by Procedure 1, where req$_{\mathbf{v}_i}$ is defined with $m_{\mathbf{v}_i}$. Let req$_\mathbf{V}$ be the composed c-task requirement defined using $M'_\mathbf{V}$. The theorem states that if an MVC satisfies subtask requirements reqs, it also satisfies c-task composed requirements req:math:_mathbf{V}`.

Each req can either be correctness-preservation (cp) or prediction-preservation ($pp$). Let an MVC $f_V$ performing a vision task $\mathbf{V}$, a distribution of input images $P_X$ and a image transformation $T_X$ be given. For a $cp$ requirement defined with a metric $\psi^{cp}$, the required condition is $\psi_{cp}( f_V , f_V^*, P_{T_{X,t_c}})\geq \psi_{cp}(f_V,f_V^*,P_X)$, where $f_\mathbf{V}^*$ is the ground truth function and $t_c$ is the human tolerated threshold. For a $pp$ requirement defined with metric $\psi^{pp}$, the required condition is $\psi_{pp}(f_V, P_{T_{X,t_p}})\geq \psi_{pp}(f_V,P_{X,\epsilon})$, where $t_p$ is the human-tolerated threshold and $\epsilon$ is a small value such that $P_{X,\epsilon}$ represents minimally transformed images.

Theorem 1

Theorem 1: If all subtask requirements req$_{\mathbf{v}_i} \in \textit{reqs}$ are satisfied then so is the composed c-task requirement req$_{\mathbf{V}}$.

Proof. Depending on the type of the requirement, req$_{\mathbf{v}_i}$ can be $cp_{\mathbf{v}_i}$ or $pp_{\mathbf{v}_i}$. We prove this theorem for both types.

For $cp$, we want to show that if $cp_{\mathbf{v}_i}$ is satisfied for all $\mathbf{v}_i \in V$, the condition $M'_\mathbf{V}(f_\mathbf{V}, f_\mathbf{V}^*, P_{T_{X,t_\mathbf{V}}}) \geq M'_\mathbf{V}(f_\mathbf{V},f_\mathbf{V}^*,P_X)$ in $cp'_{\mathbf{V}}$ is also satisfied, where $M_\mathbf{V}$ takes as input an MVC $f_V$ performing a vision task $\mathbf{V}$, $f_\mathbf{V}^*$ is a comparing function, a distribution of input images $P_X$, and a distribution $P_{T_{X,t_\mathbf{V}}}$ of original and transformed images with visual change $\leq t_\mathbf{V}$. The threshold $t_\mathbf{V}$ for req$_{\mathbf{V}}$ is defined as $t_{\mathbf{V}} = \min(\bar{t})$ (Procedure 1,L:9), where $\bar{t}$ is the list of subtasks’ thresholds. Because each subtask requirement $cp_{\mathbf{v}_i}$ includes a requirement on smaller thresholds (Procedure 1,LL:5-7), each $cp_{\mathbf{v}_i}$ includes the condition $m_{\mathbf{v}_i}(f_{\mathbf{v}_i}, f_{\mathbf{v}_i}^*, P_{T_{X,t_\mathbf{V}}}) \geq m_{\mathbf{v}_i}(f_{\mathbf{v}_i},f_{\mathbf{v}_i}^*,P_X)$, where the subtask metric $m_{\mathbf{v}_i}$ takes as input an MVC $f_V$ performing the subtask $\mathbf{v}_i$, $f_{\mathbf{v}_i}^*$ is a comparing function, and distribution $P_{T_{X,t_\mathbf{V}}}$. Specifically, conditions for different thresholds are connected by conjunction in $cp_{\mathbf{v}_i}$ (Procedure 1), thus, satisfying every $cp_{\mathbf{v}_i}$ implies $m_{\mathbf{v}_i}(f_{\mathbf{v}_i}, f_{\mathbf{v}_i}^*, P_{T_{X,t_\mathbf{V}}}) \geq m_{\mathbf{v}_i}(f_{\mathbf{v}_i},f_{\mathbf{v}_i}^*,P_X)$ is satisfied for all ${\mathbf{v}_i} \in \mathbf{V}$. Then, we have $\prod_{{\mathbf{v}_i} \in \mathbf{V}} m_{\mathbf{v}_i}(f_{\mathbf{v}_i}, f_{\mathbf{v}_i}^*, P_{T_{X,t_\mathbf{V}}}) \geq \prod_{{\mathbf{v}_i} \in \mathbf{V}} m_{\mathbf{v}_i}(f_{\mathbf{v}_i}, f_{\mathbf{v}_i}^*, P_X)$ since all metric values are positive; therefore, $M'_\mathbf{V}(f_\mathbf{V}, f_\mathbf{V}^*, P_{T_{X,t_\mathbf{V}}}) \geq M'_\mathbf{V}(f_\mathbf{V},f_\mathbf{V}^*,P_X)$ since $M'_\mathbf{V}$ is defined as $M'_\mathbf{V} = \prod_{{\mathbf{v}_i} \in \mathbf{V}} m_{\mathbf{v}_i}$ (see Metric Decomposition). As a result, $cp'_{\mathbf{V}}$ is satisfied if all $cp_{\mathbf{v}_i}$ are satisfied.

Similarly for $pp$, we want to show that if $pp_{\mathbf{v}_i}$ is satisfied for all $\mathbf{v}_i \in V$, the condition $M'_\mathbf{V}(f_\mathbf{V}, P_{T_{X,t_\mathbf{V}}}) \geq M'_\mathbf{V}(f_\mathbf{V},P_{T_{X,\epsilon}})$ in $pp'_{\mathbf{V}}$ is also satisfied. $pp_{\mathbf{v}_i}$ include the condition $m_{\mathbf{v}_i}(f_{\mathbf{v}_i}, P_{T_{X,t_\mathbf{V}}}) \geq m_{\mathbf{v}_i}(f_{\mathbf{v}_i},P_{T_{X,\epsilon}})$. Specifically, conditions for different thresholds are connected by a conjunction in $pp_{\mathbf{v}_i}$ (see Procedure 2); thus, satisfying every $pp_{\mathbf{v}_i}$ implies $m_{\mathbf{v}_i}(f_{\mathbf{v}_i}, P_{T_{X,t_\mathbf{V}}}) \geq m_{\mathbf{v}_i}(f_{\mathbf{v}_i},P_{T_{X,\epsilon}})$ is satisfied for all ${\mathbf{v}_i} \in \mathbf{V}$. Then, we have $\prod_{{\mathbf{v}_i} \in \mathbf{V}} m_{\mathbf{v}_i}(f_{\mathbf{v}_i}, P_{T_{X,t_\mathbf{V}}}) \geq \prod_{{\mathbf{v}_i} \in \mathbf{V}} m_{\mathbf{v}_i}(f_{\mathbf{v}_i}, P_{T_{X,\epsilon}})$ since all metric values are positive; therefore, $M'_\mathbf{V}(f_\mathbf{V}, P_{T_{X,t_\mathbf{V}}}) \geq M'_\mathbf{V}(f_\mathbf{V},P_{T_{X,\epsilon}})$ since $M'_\mathbf{V}$ is defined as $M'_\mathbf{V} = \prod_{{\mathbf{v}_i} \in \mathbf{V}} m_{\mathbf{v}_i}$ (see Metric Decomposition). As a result, $pp'_{\mathbf{V}}$ is satisfied if all $pp_{\mathbf{v}_i}$ are satisfied.

Therefore, req$_{\mathbf{V}}$ is satisfied if all req$_{\mathbf{v}_i}$ are satisfied.

Procedure for Compound Decomposable Metrics

We provide the following Procedure 3 for generating the reliability requirements for the c-task $\mathbf{V}$ and its subtasks using the compound decomposable metrics $M^k_\mathbf{V}$ (see Metric Decomposition). Note that the difference with Procedure 1 are highlighted in purple.

See the table above for examples of generated correctness-preservation requirements with compound decomposable metrics.

Example requirements with complex metrics

For the correctness of the requirement composition using the compound decomposable metrics, we prove the following theorem.

Theorem 2

Theorem 2: Let $\mathbf{V}$ be a c-task $\mathbf{V}= \mathbf{v}_n \odot ...\mathbf{v}_2 \odot \mathbf{v}_1$ and a $M_\mathbf{V}$ be a compound decomposable performance metric, such that $M_\mathbf{V} = F(M^1_\mathbf{V}, ..., M^k_\mathbf{V})$ and each $M^k_\mathbf{V} = \prod_{i=1}^n m^k_{\mathbf{v}_i}$ is directly decomposable. Let req$^k_\mathbf{V}$ be the composed requirement generated by Procedure for Compound Decomposable Metrics using $M^k_\mathbf{V}$, where $k \in [1, K]$, and req$_\mathbf{V}$ be the c-task requirement defined using $M_\mathbf{V}$. If all composed requirements req$^k_\mathbf{V}, k \in [1, N]$ are satisfied, then so is the c-task requirement req$_\mathbf{V}$.

Proof. Depending on the type of the requirement, req$^k_{\mathbf{V}}$ can be $cp^k_{\mathbf{V}}$ or $pp^k_{\mathbf{V}}$. We prove the theorem for both types.

For $cp$, assume all $cp_{\mathbf{V}}^k$ are satisfied. We show the required condition $M_V( f_\mathbf{V} , f_\mathbf{V}^*, P_{T_{X,t_\mathbf{V}}}) \geq M_V(f_\mathbf{V},f_\mathbf{V}^*,P_X)$ in $cp_{\mathbf{V}}$ is satisfied. Since $M_\mathbf{V} = F(M^1, ..., M^K)$ is positively correlated with each $M^k \in \{M^1, ..., M^K\}$, if the value of each $M^k$ increases, the value of $M_\mathbf{V}$ increases as well. Satisfying all $cp_{\mathbf{V}}^k$ suggests $M^k( f_\mathbf{V} , f_\mathbf{V}^*, P_{T_{X,t_\mathbf{V}^k}}) \geq M^k(f_\mathbf{V},f_\mathbf{V}^*,P_X)$ for each $M^k$ and therefore, $F(M^1( f_\mathbf{V} , f_\mathbf{V}^*, P_{T_{X,t_\mathbf{V}}}), ..., M^K( f_\mathbf{V} , f_\mathbf{V}^*, P_{T_{X,t_\mathbf{V}}})) \geq F(M^1(f_\mathbf{V},f_\mathbf{V}^*,P_X), ..., M^K(f_\mathbf{V},f_\mathbf{V}^*,P_X))$ which is $M_V( f_\mathbf{V} , f_\mathbf{V}^*, P_{T_{X,t_\mathbf{V}}}) \geq M_V(f_\mathbf{V},f_\mathbf{V}^*,P_X)$. As a result, $cp_\mathbf{V}$ is satisfied.

For $pp$, assume all $pp_{\mathbf{V}}^k$ are satisfied. We show the required condition $M_V( f_\mathbf{V}, P_{T_{X,t_\mathbf{V}}}) \geq M_V(f_\mathbf{V},P_{T_{X,\epsilon}})$ in $pp_{\mathbf{V}}$ is satisfied. Since $M_\mathbf{V} = F(M^1, ..., M^K)$ is positively correlated with each $M^k \in \{M^1, ..., M^K\}$, if the value of each $M^k$ increases, the value of $M_\mathbf{V}$ increases as well. Satisfying all $pp_{\mathbf{V}}^k$ suggests $M^k( f_\mathbf{V}, P_{T_{X,t_\mathbf{V}^k}}) \geq M^k(f_\mathbf{V}, P_{T_{X,\epsilon}})$ for each $M^k$ and therefore, $F(M^1( f_\mathbf{V} , P_{T_{X,t_\mathbf{V}}}), ..., M^K( f_\mathbf{V}, P_{T_{X,t_\mathbf{V}}})) \geq F(M^1(f_\mathbf{V},P_{T_{X,\epsilon}}), ..., M^K(f_\mathbf{V},P_{T_{X,\epsilon}}))$ which is $M_V( f_\mathbf{V}, P_{T_{X,t_\mathbf{V}}}) \geq M_V(f_\mathbf{V},P_{T_{X,t_\mathbf{V}}})$. As a result, $pp_\mathbf{V}$ is satisfied.

Therefore, satisfying all $req_{V}^k$ implies satisfying $req_\mathbf{V}$.