Influence of Validation Sample Selection In Ecological Modeling.

Spatially-explicit models that capture the distribution of ecological properties (feature space) and spatial patterns (across spatial space) allow addressing many critical questions in the realms of mixed-use agricultural, forest, urban, and wetland ecosystems. To evaluate the quality and uncertainty of ecological model predictions ideally independent validation is used. In practice, the criteria to select an independent validation set may introduce bias and may even derive incorrect conclusions. This study investigated the behavior of criteria used for selection of validation sets on large ecological datasets (n: 1000 randomly selected samples).  We used three ecological datasets (solar radiation, vegetation and soil), generated over the state of Florida, to investigate the effect of the size and the distribution (in terms of both spatial and feature space) of validation datasets on the outcomes (RMSE, R² and RPD) of spatial model validation on large scale studies.   These effects were tested taking into consideration the different population distributions (bimodal, normal and skewed) displayed in the ecological datasets. The effects of selecting validation sets on spatially-explicit relationships were evaluated using models developed by applying Kriging (ordinary and universal – method dependent on the dataset properties).  It was found that as the validation dataset was reduced in size, from 30% to 5% of the overall dataset, the variation in the validation outcomes increased.  Unexpectedly it was found that selecting a validation dataset with a distribution statistically similar to the calibration/overall distribution did not ensure that the validation results (RMSE, R² and RPD) were significantly lower than a validation dataset differing significantly from the calibration/overall distribution.  Our findings indicate that, in large scale ecological studies, irrespective of the population distribution a validation dataset should contain between 20-30% of the total original dataset, and these validation datasets should be maximized in terms of their spread across both spatial and feature space.