Introduction#
Maps are essential tools for environmental monitoring, resource management, and sustainable development planning. As global resources become increasingly scarce due to population growth and environmental pressures, their role has widened further, into urban planning, emergency response, and habitat conservation. They serve as critical instruments for communication, navigation, and decision support, providing a symbolic representation of reality [Awange and Kyalo Kiema, 2013]. By supplying precise and timely spatial information, maps enable us to measure, analyze, and understand the extent and distribution of resources, empowering decision-makers to take informed actions [Congalton and Green, 2019].
Since maps are models or generalizations of reality, they inherently contain errors due to factors such as map projections, construction techniques, and data symbolization [Lightfoot and Butler, 1987, Maling, 2013]. That single fact has two distinct consequences for anyone who uses a thematic map:
You do not know how good the map is — which classes are reliable and which are not.
You do not know how much area each class really covers — because the mapped extent of a class is not the same as its true extent.
AcATaMa answers both questions from the same set of labeled sample points. The two sections below explain why each matters; the rest of the documentation describes how the plugin delivers them.
Assessing Map Quality#
The primary goal of quantitative accuracy assessment is to quantify the error (or correctness) of mapped labels or attributes by comparing them with true reference data, providing critical insights into the reliability of classification results [Congalton and Green, 2019]. It is, in effect, the quality control step of map production: it establishes whether a map is fit for the use it is intended to serve, and makes the limits of that fitness explicit.
A rigorous assessment of map accuracy is necessary to either derive scientifically valid data or ensure the reliability of data for decision-making [Stehman and Czaplewski, 1998, Olofsson et al., 2014, Strahler et al., 2006]:
For researchers: the construction of scientific knowledge requires a structured and robust analytical framework with appropriate statistical methods to make valid inferences; consequently, maps without associated information on the accuracy of the predictions essentially remain untested hypotheses [Strahler et al., 2006, McRoberts, 2011, Mitchell et al., 2018].
For policymakers: errors in geospatial data can lead to flawed conclusions and misguided actions with potentially serious consequences; assessing the accuracy of thematic maps is essential to incorporate realistic expectations into the planning processes and environmental monitoring, supporting informed decision-making and effective resource allocation [Estes and Mooneyhan, 1994, McKendry, 2000, Foody, 2001, Foody, 2015].
Because accuracy is reported per class — not only as a single overall figure — quality control also tells you where a map can be trusted. A map may be highly accurate for a dominant, stable class and unreliable for the rare change class that a study actually depends on.
Estimating Area#
The accuracy measures alone do not answer the second question. Knowing that a class has 85 % user’s accuracy tells you how often that label is correct, but not how much of the class actually exists on the ground.
The intuitive answer — count the pixels labeled as a class and multiply by the pixel size — is biased whenever the map contains classification error, which in practice is always [Olofsson et al., 2014, Gallego, 2004, Olofsson et al., 2013]. Omission and commission errors do not cancel out, so the mapped area of a class departs systematically from its true area. Gallego [2004] characterizes this as a measurement bias rather than an estimator bias: the map is a complete census of the territory, but the instrument doing the counting is imperfect. This is why the problem cannot be solved by mapping more carefully at the margins or by collecting a larger sample — the bias is a property of the map itself.
The distortion is most severe for rare classes, which are usually the ones that matter most in change monitoring. A class such as deforestation may occupy a very small fraction of the territory, so even a low commission rate among the dominant stable classes introduces false-positive pixels that are large relative to the rare class’s true extent [Olofsson et al., 2014, Olofsson et al., 2020, Stehman and Olofsson, 2024]. Reporting a raw pixel count in that situation can misstate the very change being monitored.
Good practice is therefore to derive the area from the reference sample, using the map only to define the strata and their weights [Olofsson et al., 2014, Olofsson et al., 2013, Stehman, 2013]. Expressing the error matrix in terms of area proportions instead of sample counts yields a design-based estimator of each class’s true proportion, and hence its area. Just as importantly, that estimator carries a variance — so the output is not merely a corrected number but a number with a confidence interval, which a pixel count can never provide.
That property is what makes an area figure reportable. Good-practice guidance for forest monitoring and greenhouse-gas accounting requires activity data to be accompanied by quantified uncertainty [Global Forest Observations Initiative, 2020, Food and Agriculture Organization of the United Nations (FAO), 2016, Finegold and Ortmann, 2016]. AcATaMa produces the area-adjusted estimate, its standard error, the confidence interval, the uncertainty and the coefficient of variation [McMurray et al., 2017] in the same operation that produces the accuracy metrics.
Two design decisions make the plugin practical for this purpose:
The estimator follows the sampling design. Stratum weights are taken from the thematic map under assessment, and the estimator is selected to match the design in use — simple random, stratified, systematic, or post-stratified — so the area estimate is always statistically consistent with the sample that produced it.
The estimate stays tied to the assessment. The error matrix, the accuracy metrics and the area estimates come from one sample, one map and one configuration. This removes the manual transfer step between separate accuracy and area tools, which is where inconsistent assumptions typically enter.
See also
A worked application of both outputs is described in Real-World Case Study.
Why a Dedicated Tool#
Despite the widespread acceptance of accuracy assessment and the extensive literature on the topic, conducting rigorous assessments remains operationally challenging. Practitioners must translate statistical guidance into sampling designs, define response protocols, interpret reference data consistently, calculate design-based accuracy and area estimators, and report uncertainty in a reproducible way. Each step demands specialized expertise, which can itself pose a barrier to adoption [Foody, 2001].
These steps are also frequently distributed across separate GIS, spreadsheet, scripting, and visual interpretation tools, each covering only part of the workflow. Every hand-off between them is an opportunity for inconsistent assumptions or incomplete reporting. Several existing tools and platforms support important parts of the workflow — visual reference-data interpretation, sampling design, or area estimation — but none brings all three stages together in a single, open-source desktop environment. Some rely on persistent internet connectivity or cloud platforms, others require programming expertise, and proprietary options impose licensing costs that limit accessibility.
AcATaMa (Accuracy Assessment of Thematic Maps) was developed to close that gap: a QGIS plugin providing a complete, open-source, and offline-capable workflow that integrates sampling design, response design, and analysis in a single, user-friendly environment, within a design-based inference framework.
Theoretical Foundation#
AcATaMa is designed and developed based on validated methodologies from foundational research studies in thematic map accuracy assessment. Its structure and protocols follow the fundamental principles outlined by Stehman and Czaplewski [1998] for the design and analysis of accuracy assessment for raster thematic maps. The sampling methods, statistical techniques, and estimation procedures are implemented as specified in the literature for each sampling design by Cochran [1977] and Särndal and Swensson [1992].
Additionally, AcATaMa incorporates other key recommendations for good practice in accuracy assessment from Stehman and Foody [2009], Olofsson et al. [2013], Olofsson et al. [2014], Stehman [2014], Finegold and Ortmann [2016], Food and Agriculture Organization of the United Nations (FAO) [2016], Stehman and Foody [2019], and Global Forest Observations Initiative [2020].
These protocols and best practices for accuracy assessment are internationally recognized, widely accepted, and extensively applied in numerous studies and real-world applications across various disciplines. Only well-validated and widely trusted methods have been integrated into AcATaMa to ensure reliable, reproducible, and statistically rigorous accuracy assessments.
Equations, Measures and Estimators#
The following table presents the equations, measures, and estimators incorporated in AcATaMa’s sampling design and analysis protocol, which are used for sampling design, accuracy assessment, and area estimation. The estimator applied depends on the sampling design selected in the plugin, so make sure the design reported in the analysis window matches the design you intend to report.
Sampling design |
Measures and estimators |
Reference |
Key formulas (compact notation) |
|---|---|---|---|
Simple random sampling (SRS) |
Accuracy (overall, user’s, producer’s) |
Stehman and Foody [2009], Table 21.3 |
\[\hat{O} = \frac{\sum_{i=1}^{q} n_{ii}}{n}, \quad \hat{U}_i = \frac{n_{ii}}{n_{i+}}, \quad \hat{P}_j = \frac{n_{jj}}{n_{+j}}\]
\[\hat{V}(\hat{O}) = \frac{\hat{O}(1-\hat{O})}{n-1}, \quad \hat{V}(\hat{U}_i) = \frac{\hat{U}_i(1-\hat{U}_i)}{n_{i+}-1}, \quad \hat{V}(\hat{P}_j) = \frac{\hat{P}_j(1-\hat{P}_j)}{n_{+j}-1}\]
|
Area estimation and confidence interval |
Stehman [2013], Eq. 18; variance estimator unbiased for SRS with replacement [Cochran, 1977] |
\[\hat{p}_{+j} = \frac{n_{+j}}{n}, \quad \hat{A}_j = A_T\, \hat{p}_{+j}\]
\[\hat{V}\left(\hat{p}_{+j}\right) = \frac{\hat{p}_{+j}\left(1 - \hat{p}_{+j}\right)}{n - 1}\]
|
|
Stratified random sampling (STR) |
Accuracy (overall, user’s, producer’s) |
Olofsson et al. [2013], Eqs. 1 and 6–8; variances Olofsson et al. [2014], Eqs. 5–7 |
\[\hat{q}_{ij} = \frac{n_{ij}}{n_{i+}}, \quad \hat{p}_{ij} = W_i\, \hat{q}_{ij}, \quad \hat{O} = \sum_i \hat{p}_{ii}\]
\[\hat{U}_i = \frac{\hat{p}_{ii}}{\hat{p}_{i+}}, \quad \hat{P}_j = \frac{\hat{p}_{jj}}{\hat{p}_{+j}}\]
\[\hat{V}(\hat{O}) = \sum_i \frac{W_i^{2}\,\hat{U}_i(1-\hat{U}_i)}{n_{i+}-1}, \quad \hat{V}(\hat{U}_i) = \frac{\hat{U}_i(1-\hat{U}_i)}{n_{i+}-1}\]
\[\hat{V}(\hat{P}_j) = \frac{1}{\hat{p}_{+j}^{2}}\left[\frac{W_j^{2}(1-\hat{P}_j)^{2}\,\hat{U}_j(1-\hat{U}_j)}{n_{j+}-1} + \hat{P}_j^{2}\sum_{i \ne j}\frac{W_i^{2}\,\hat{q}_{ij}(1-\hat{q}_{ij})}{n_{i+}-1}\right]\]
|
Area estimation and confidence interval |
Olofsson et al. [2014], Eqs. 9–11 |
\[\hat{p}_{+j} = \sum_i W_i\, \hat{q}_{ij}, \quad \hat{A}_j = A_T\, \hat{p}_{+j}\]
\[\hat{V}\left(\hat{p}_{+j}\right) = \sum_i \frac{W_i^{2}\, \hat{q}_{ij}\left(1 - \hat{q}_{ij}\right)}{n_{i+} - 1}\]
|
|
Sample size and allocation |
Olofsson et al. [2014], Eq. 13 |
\[n = \frac{\left(\sum_i W_i S_i\right)^{2}}{\left[S\left(\hat{O}\right)\right]^{2} + \frac{1}{N}\sum_i W_i S_i^{2}} \approx \left(\frac{\sum_i W_i S_i}{S\left(\hat{O}\right)}\right)^{2}\]
\[S_i = \sqrt{U_i\left(1 - U_i\right)}, \quad n_i \approx W_i\, n\]
with a minimum allocation per stratum |
|
Systematic sampling (SYS) |
Accuracy (overall, user’s, producer’s) |
Stehman and Foody [2009], Table 21.3 |
\[\hat{O} = \frac{\sum_{i=1}^{q} n_{ii}}{n}, \quad \hat{U}_i = \frac{n_{ii}}{n_{i+}}, \quad \hat{P}_j = \frac{n_{jj}}{n_{+j}}\]
SYS uses the simple (SRS) estimators; their variances are approximations that usually overestimate the variance of systematic samples [Olofsson et al., 2014] |
Area estimation and confidence interval |
Stehman [2013], Eq. 18; variance [Cochran, 1977] |
\[\hat{p}_{+j} = \frac{n_{+j}}{n}, \quad \hat{A}_j = A_T\, \hat{p}_{+j}\]
\[\hat{V}\left(\hat{p}_{+j}\right) = \frac{\hat{p}_{+j}\left(1 - \hat{p}_{+j}\right)}{n - 1}\]
SYS uses the simple (SRS) area estimator |
|
Post stratified (SRS and SYS) |
Accuracy (overall, user’s, producer’s) |
Olofsson et al. [2014], Eqs. 1–7 |
The stratified estimators and variances above, with the map classes as post-strata [Olofsson et al., 2014] |
Area estimation and confidence interval |
Olofsson et al. [2014], Eq. 9; approximate (large sample) variance Cochran [1977], Eq. 5A.42 and Stehman [2013], Eq. 48, for sampling with replacement |
\[\hat{p}_{+j}^{\mathrm{PS}} = \sum_h W_h^{\mathrm{PS}}\left(\frac{n_{hj}}{n_{h+}}\right), \quad \hat{A}_j^{\mathrm{PS}} = A_T\, \hat{p}_{+j}^{\mathrm{PS}}\]
\[\hat{V}\left(\hat{p}_{+j}^{\mathrm{PS}}\right) = \frac{1}{n}\sum_{h=1}^{H} W_h s_{yh}^{2} + \frac{1}{n^{2}}\sum_{h=1}^{H}\left(1 - W_h\right) s_{yh}^{2}\]
\[s_{yh}^{2} = \frac{n_{h+}\, \hat{q}_{hj}\left(1 - \hat{q}_{hj}\right)}{n_{h+} - 1}\]
the sample variance in post-stratum \(h\) of \(y_u = 1\) if the reference class of unit \(u\) is \(j\) (0 otherwise) |
|
SRS and SYS |
Sample size |
Cochran [1977], Eq. 4.2 |
\[n = \frac{Z^{2}\, p\,(1 - p)}{d^{2}}\]
rounded up, where \(p\) is the anticipated overall accuracy, \(d\) the desired half-width of the confidence interval and \(Z\) the standard normal quantile. For SYS the point spacing is \(\sqrt{A/n}\), \(A\) being the area with valid data |
All designs |
Uncertainty and coefficient of variation |
McMurray et al. [2017], p. 17 |
\[U(\%) = \frac{100\, z\, \mathrm{SE}\left(\hat{A}_j\right)}{\hat{A}_j}, \quad \mathrm{CV}(\%) = \frac{100\, \mathrm{SE}\left(\hat{A}_j\right)}{\hat{A}_j}\]
|
Notation: \(n\) = total sample size; \(N\) = total number of map units/pixels in the analysis area; \(n_{ij}\) = sample count in map/stratum row \(i\) and reference-class column \(j\); \(n_{i+}\) and \(n_{+j}\) = row and column totals; \(W_i = A_i/A_T\); \(A_T\) = total map area; \(\hat{q}_{ij}\) = within-stratum sample proportion; \(\hat{p}_{ij}\) = area-adjusted proportion; \(U_i\) = conjectured user’s accuracy for stratum \(i\); \(S(\hat{O})\) = target standard error of overall accuracy; \(z\) = normal critical value for the selected confidence level. The standard error is the square root of the estimated variance, \(\mathrm{SE} = \sqrt{\hat{V}}\). All the variances are the ones of sampling with replacement, without the finite population correction \((1 - n/N)\) of sampling without replacement, as the random sampling of AcATaMa: its sampling points are drawn over the map, so several of them can fall in the same pixel. For samples without replacement, omitting the correction makes the variance estimates larger, negligibly when the sampling fractions are small, also within each stratum (\(n_{i+} \ll N_i\)) [Cochran, 1977, Stehman, 2012]. The accuracy variances of the post-stratified estimator are the stratified ones [Olofsson et al., 2014], while its area variance also accounts for the random sample sizes of the post-strata [Cochran, 1977]. The variance of the post-stratified area is a large sample approximation [Stehman, 2013], as the variance of the producer’s accuracy, a ratio estimator (linearization). The variances of systematic samples are approximations that usually overestimate the variance [Olofsson et al., 2014], but they can underestimate it if the map has periodic patterns aligned with the grid [Cochran, 1977].
Note
The population of the simple estimator is the whole thematic map (its pixels with data), including the map classes without samples. The stratified and post-stratified estimators need samples in every stratum (map class): a map class with mapped area but no samples is excluded from the estimation (the total area and all the estimates then refer to the remaining map classes), and a stratum with a single sample has no estimable variance, so the standard errors that depend on it (overall and producer’s accuracy and all the area estimates) are not reported. AcATaMa reports these situations as warnings in the analysis results.
Tip
The minimum-per-stratum allocation prevents rare but policy-relevant classes (for example deforestation) from receiving too few samples to yield comparably precise estimates. See Sampling Design for how to set this in the plugin.
Important
AcATaMa has been rigorously tested using multiple real-world examples across various use cases to ensure the reliability of its results. This involved manually calculating several test cases from scratch and systematically comparing them with AcATaMa’s outputs to validate its computational accuracy.