Bootstrap simulations to estimate relationships between Type I error, power, effect size, and appropriate sample numbers for bioassessments of aquatic ecosystems

References

• Garamszegi, L. Z. A Simple Statistical Guide for the Analysis of Behavior When Data Are Constrained Due to Practical or Ethical Reasons. Anim. Behav. 2016, 120, 223–234. DOI: 10.1016/j.anbehav.2015.11.009.
   Web of Science ®Google Scholar
• Johnson, P. C. D.; Barry, S. J. E.; Ferguson, H. M.; Muller, P. Power Analysis for Generalized Linear Mixed Models in Ecology and Evolution. Methods Ecol. Evol. 2015, 6, 133–142. DOI: 10.1111/2041-210X.12306.
   PubMed Web of Science ®Google Scholar
• Garman, S. L.; Schweiger, E. W.; Manier, D. J. Simulation Future Uncertainty to Guide the Selection of Survey Designs for Long-Term Monitoring. In Design and Analysis of Long-Term Ecological Monitoring Studies; Gitzen, R. A, Millspaugh, J. J., Cooper, A. B., Licht, D. S., Eds.; Cambridge University Press: Cambridge, UK, 2012; pp 228–249.
   Google Scholar
• Fox, D. R. Environmental Power Analysis – A New Perspective. Environmetrics 2001, 12, 437–449. DOI: 10.1002/env.470.
   Web of Science ®Google Scholar
• Lenth, R. V. Practical Guidelines for Effective Sample Size Determination. Am. Statistician 2001, 55, 187–193. DOI: 10.1198/000313001317098149.
   Web of Science ®Google Scholar
• Murphy, K. R.; Myors, B.; Wolach, A. Statistical Power Analysis, 4th ed.; Routledge, Taylor and Francis Group: East Sussex, UK, 2014.
   Google Scholar
• Edgington, E.; Onghena, P. Randomization Tests; Chapman and Hall/CRC: New York, 2007. 10.1201/9781420011814.
   Google Scholar
• Manly, B. F. J., ed. Randomization, Bootstrap and Monte Carlo Methods in Biology, 3rd ed. Chapman & Hall/CRC Taylor & Francis Group: New York, 2007.
   Google Scholar
• Scheiner, S.M.; Gurevitch, J., eds. Design and Analysis of Ecological Experiments, 2nd ed. Oxford University Press: New York, 2001.
   Google Scholar
• Westfall, P. H.; Young, S. S. Resampling-Based Multiple Testing, John Wiley and Sons, Inc.: New York, 1993.
   Google Scholar
• Cohen, J. Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Psychology Press, Taylor & Francis Group: New York, 1988.
   Google Scholar
• Jennions, M. D.; Moller, A. P. A Survey of the Statistical Power of Research in Behavioral Ecology and Animal Behavior. Behav. Ecol. 2003, 14, 438–445. DOI: 10.1093/beheco/14.3.438.
   Web of Science ®Google Scholar
• Manly, B. F. J. A Review of Computer Intensive Multivariate Methods in Ecology. Chapter. 15 In Multivariate Environmental Statistics; Patil, G. P., Rao, C. R., Eds.; Elsevier Science Publishers, B.V: Amsterdam, Netherlands, 1993, pp. 307–346.
   Google Scholar
• Crowley, P. H. Resampling Methods for Computation-Intensive Data Analysis in Ecology and Evolution. Annu. Rev. Ecol. Syst. 1992, 23, 405–447. DOI: 10.1146/annurev.es.23.110192.002201.
   Google Scholar
• Cohen, J. The Statistical Power of Abnormal-Social Psychological Research: A Review. J. Abnorm. Soc. Psychol. 1962, 65, 145–153. DOI: 10.1037/h0045186.
   PubMed Web of Science ®Google Scholar
• Cohen, J. Some Statistical Issues in Psychological Research. In Handbook of Clinical Psychology; Wolman, B.B., Ed.; McGraw-Hill: New York, 1965; pp. 95–121.
   Google Scholar
• Cohen, J. Statistical Power Analysis and Research Results. Am. Ed. Res. J. 1973, 10, 225–229. DOI: 10.2307/1161884.
   Web of Science ®Google Scholar
• Green, R. H. Power Analysis and Practical Strategies for Environmental Monitoring. Environ. Res. 1989, 50, 195–205. DOI: 10.1016/S0013-9351(89)80058-1.
   PubMed Web of Science ®Google Scholar
• Moller, A. P.; Jennions, M. D. How Much Variance Can Be Explained by Ecologists and Evolutionary Biologists? Oecologia 2002, 132, 492–500. DOI: 10.1007/s00442-002-0952-2.
   PubMed Web of Science ®Google Scholar
• Hoenig, J. M.; Heisey, D. M. The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis. Am. Stat. Assoc. 2001, 55, 1–6.
   Web of Science ®Google Scholar
• Hegyi, G.; Laczi, M. Using Full Models, Stepwise Regression and Model Selection in Ecological Data Sets: Monte Carlo Simulations. Ann. Zool. Fenn. 2015, 52, 257–279. DOI: 10.5735/086.052.0502.
   Web of Science ®Google Scholar
• Levine, C. R.; Yanai, R. D.; Lampman, G. G.; Burns, D. A.; Driscoll, C. T.; Lawrence, G. B.; Lynch, J. A.; Schoch, N. Evaluating the Efficiency of Environmental Monitoring Programs. Ecol. Indic. 2014, 39, 94–101. DOI: 10.1016/j.ecolind.2013.12.010.
   Web of Science ®Google Scholar
• Zuur, A. F.; Ieno, E. N.; Elphick, C. S. A Protocol for Data Exploration to Avoid Common Statistical Problems. Methods Ecol. Evol. 2010, 1, 3–14. DOI: 10.1111/j.2041-210X.2009.00001.x.
   Web of Science ®Google Scholar
• Kenkel, N. C. On Selecting an Appropriate Multivariate Analysis. Can. J. Plant Sci. 2006, 86, 663–676. DOI: 10.4141/P05-164.
   Web of Science ®Google Scholar
• Olden, J. D.; Jackson, D. A. Torturing Data for the Sake of Generality: How Valid Are Our Regression Models? Ecoscience 2000, 7, 501–510. DOI: 10.1080/11956860.2000.11682622.
   Web of Science ®Google Scholar
• Fletcher, D.; Mackenzie, D.; Villouta, E. Modelling Skewed Data with Many Zeros: A Simple Approach Combining Ordinary and Logistic Regression. Environ. Ecol. Stat. 2005, 12, 45–54. DOI: 10.1007/s10651-005-6817-1.
   Web of Science ®Google Scholar
• Wheeler, J. R.; Grist, E. P. M.; Leung, K. M. Y.; Morritt, D.; Crane, M. Species Sensitivity Distributions: Data and Model Choice. Mar. Poll. Bull. 2002, 45, 192–202. DOI: 10.1016/S0025-326X(01)00327-7.
   PubMed Web of Science ®Google Scholar
• Rodgers, J. L. The Bootstrap, the Jackkife, and the Randomization Test: A Sampling Taxonomy. Multivar. Behav. Res. 1999, 34, 441–456. DOI: 10.1207/S15327906MBR3404_2.
   PubMed Web of Science ®Google Scholar
• Efron, B. Bootstrap Methods: Another Look at the Jackknife. Ann. Stat.. 1979, 7, 1–26. DOI: 10.1214/aos/1176344552.
   Web of Science ®Google Scholar
• Diaconis, P.; Efron, B. Computer Intensive Methods in Statistics. Sci. Am. 1983, 248, 96–108.
   Web of Science ®Google Scholar
• Nores, M. L.; Diaz, M. P. Bootstrap Hypothesis Testing in Generalized Additive Models for Comparing Curves of Treatments in Longitudinal Studies. J. Appl. Stat. 2016, 43, 810–826. DOI: 10.1080/02664763.2015.1078301.
   Web of Science ®Google Scholar
• Yuan, K. H.; Hayashi, K. Bootstrap Approach to Inference and Power Analysis Based on Three Test Statistics for Covariance Structure Models. Br. J. Math. Stat. Psychol. 2003, 56, 93–110. DOI: 10.1348/000711003321645368.
   PubMed Web of Science ®Google Scholar
• Winter, J. G.; Duthie, H. C. Epilithic Diatoms as Indicators of Stream Total N and Total P Concentration. J. N. Am. Benthol. Soc 2000, 19, 32–49. DOI: 10.2307/1468280.
   Web of Science ®Google Scholar
• Thomas, L. Retrospective Power Analysis. Conserv. Biol. 1997, 11, 276–280. DOI: 10.1046/j.1523-1739.1997.96102.x.
   Web of Science ®Google Scholar
• Hu, F.; Zidek, J. V. A Bootstrap Based on the Estimating Equations of the Linear Model. Biometrika 1995, 82, 263–275. DOI: 10.1093/biomet/82.2.263.
   Web of Science ®Google Scholar
• ter Braak, C. J. F. Permutation versus Bootstrap Significance Tests in Multiple Regression and ANOVA. In Bootstrapping and Related Techniques; K. H. Jöckel, Ed.; Springer-Verlag: Berlin, 1992; pp. 79–86.
   Google Scholar
• Romano, J. P. Bootstrap and Randomization Tests of Some Nonparametric Hypotheses. Ann. Stat. 1989, 17, 141–159. DOI: 10.1214/aos/1176347007.
   Web of Science ®Google Scholar
• Beran, R.; Srivastava, M. S. Bootstrap Tests and Confidence Regions for Functions of a Covariance Matrix. Ann. Stat. 1985, 13, 95–115. DOI: 10.1214/aos/1176346579.
   Web of Science ®Google Scholar
• Hanson, N. Using Biological Data from Field Studies with Multiple Reference Sites as a Basis for Environmental Management: The Risks for False Positives and False Negatives. J. Environ. Manage. 2011, 92, 610–619. DOI: 10.1016/j.jenvman.2010.09.026.
   PubMed Web of Science ®Google Scholar
• Grist, E. P. M.; Leung, K. M. Y.; Wheeler, J. R.; Crane, M. Better Bootstrap Estimation of Hazardous Concentration Thresholds for Aquatic Assemblages. Environ. Toxicol. Chem. 2002, 21, 1515–1524. DOI: 10.1002/etc.5620210725.
   PubMed Web of Science ®Google Scholar
• Johnson, R. K. Spatiotemporal Variability of Temperate Lake Macroinvertebrate Communities: Detection of Impact. Ecol. Appl. 1998, 8, 61–70. DOI: 10.1890/1051-0761(1998)008[0061:SVOTLM]2.0.CO;2.
   Web of Science ®Google Scholar
• Alden, R. W.; III, Dauer, D. M.; Ranasinghe, J. A.; Scott, L. C.; Llansó, R. J. Statistical Verification of the Chesapeake Bay Benthic Index of Biotic Integrity. Environmetrics 2002, 13, 473–498. DOI: 10.1002/env.548.
   Web of Science ®Google Scholar
• Alden, R. W.; Weisberg, S. B.; Ranasinghe, J. A.; Dauer, D. M. III. Optimizing Temporal Sampling Strategies for Benthic Environmental Monitoring Programs. Mar. Poll. Bull. 1997, 34, 913–922. DOI: 10.1016/S0025-326X(97)00049-0.
   Web of Science ®Google Scholar
• Alden, R. W. III Uncertainty and Sediment Quality Assessments: I. Confidence Limits for the Triad. Environ. Toxicol. Chem. 1992, 11, 637–644. DOI: 10.1002/etc.5620110507.
   Web of Science ®Google Scholar
• Alden, R. W.; III.; Rule, J. H. Uncertainty and Sediment Quality Assessments: II. Effects of Correlations between Contaminants on the Interpretation of Apparent Effects Threshold Data. Environ. Toxicol. Chem. 1992, 11, 645–651. DOI: 10.1002/etc.5620110508.
   Web of Science ®Google Scholar
• Valdecasas, A. G.; Baltanas, A. Jackknife and Bootstrap Estimation of Biological Index of Water Quality. Wat. Res. 1990, 24, 1279–1283.
   Web of Science ®Google Scholar
• Leonardsson, K.; Blomqvist, M.; Rosenberg, R. Theoretical and Practical Aspects on Benthic Quality Assessment according to the EU-Water Framework Directive-examples from Swedish waters. Mar. Pollut. Bull. 2009, 58, 1286–1296. DOI: 10.1016/j.marpolbul.2009.05.007.
   PubMed Web of Science ®Google Scholar
• Vaz, S.; Martin, C. S.; Eastwood, P. D.; Ernande, B.; Carpentier, A.; Meaden, G. J.; Coppin, F. Modelling Species Distributions Using Regression Quantiles. J. App. Ecol. 2007, 45, 204–217. DOI: 10.1111/j.1365-2664.2007.01392.x.
   Web of Science ®Google Scholar
• O’Gorman, T. W. The Performance of Randomization Tests That Use Permutations of Independent Variables. Commun. Statist. Simul. Comput. 2005, 34, 895–908.
   Web of Science ®Google Scholar
• Peres-Neto, P. R.; Jackson, D. A.; Somers, K. M. Giving Meaningful Interpretation to Ordination Axes Assessing Loading Significance in Principal Components Analysis. Ecology 2003, 84, 2347–2363. DOI: 10.1890/00-0634.
   Web of Science ®Google Scholar
• McKenna, J. E. Jr. An Enhanced Cluster Analysis Program with Bootstrap Significance Testing for Ecological Community Analysis. Environ. Model. Software 2003, 18, 205–220. DOI: 10.1016/S1364-8152(02)00094-4.
   Web of Science ®Google Scholar
• Alden, R. W. III. The Evaluation of Minimum Detectable Impacts in Environmental Assessment Studies. In Ports and Harbors: Our Link to the Water; Coastal Society: Bethesda, MD, 1990; pp. 327–339.
   Google Scholar
• Manly, B. F. J. Bootstrapping for Determining Sample Sizes in Biological Studies. J. Exp. Mar. Biol. Ecol. 1992, 158, 189–196. DOI: 10.1016/0022-0981(92)90226-Z.
   Web of Science ®Google Scholar
• Bros, W. E.; Cowell, B. C. A Technique for Optimizing Sample Size (Replication). J. Exp. Mar. Biol. Ecol. 1987, 114, 63–71.
   Web of Science ®Google Scholar
• Hall, P.; Titterington, D. M. The Effect of Simulation Order on Level Accuracy and Power of Monte Carlo Tests. J.R. Statist. Soc. B 1989, 51, 459–467.
   Google Scholar
• Anderson, M. J.; Legendre, P. An Empirical Comparison of Permutation Methods for Tests of Partial Regression Coefficients in a Linear Model. J. Statist. Comp. Simul. 1999, 62, 271–303. DOI: 10.1080/00949659908811936.
   Web of Science ®Google Scholar
• Hall, L. W.; Jr.; Alden R, W.; III; Anderson, R. D.; Killen, W. D. Ranking the Importance of Benthic Metrics and Environmental Stressors from over a Decade of Bioassessment Multiple Stressor Studies in Five California Waterbodies. J. Environ. Sci. Health, Part A 2019, 54, 1364–1386. DOI: 10.1080/10934529.2019.1651590.
   PubMedGoogle Scholar
• Hall, L. W.; Jr.; Alden R, W.; III; Anderson, R. D.; Killen, W. D. Multivariate Spatial Patterns Analysis of Environmental Variables and Benthic Metrics in Five California Waterbodies. J. Environ. Sci. Health, Part A 2019, 54, 1387–1399. DOI: 10.1080/10934529.2019.1656968.
   PubMedGoogle Scholar
• Hall, L. W.; Jr.; Anderson, R. D.; Killen, W. D.; Alden, R. W. III. An Analysis of Multiple Stressors on Benthic Communities in a California Agricultural Stream. Air Soil Water Res. 2018, 11, 1–18. DOI: 10.1177/11786221877761.
   Web of Science ®Google Scholar
• Hall, L. W.; Jr.; Killen, W. D.; Anderson, R. D.; Alden, R. W. III. A Three Year Assessment of the Influence of Physical Habitat, Pyrethroids and Metals on Benthic Communities in Two Urban California Streams. J. Ecosyst. Ecograph 2013, 3, 133–150. DOI: 10.4172/2157-7625.1000133.
   Google Scholar
• Hall, L. W.; Jr.; Killen, W. D.; Anderson, R. D.; Alden, R. W. III. The Influence of Physical Habitat, Pyrethroids, and Metals on Benthic Community Condition in an Urban and Residential Stream in California. Human Ecol.. Risk Assess. 2009, 15, 526–553. DOI: 10.1080/10807030902892521.
   Web of Science ®Google Scholar
• Harrington, J.; Born, M. Measuring the Health of California Streams and Rivers. – A Methods Manual for Water Resource Professionals, Citizen Monitors and Natural Resource Students; Technical Report; Sustainable Land Stewardship International Institute: Sacramento, CA, 2000.
   Google Scholar
• SAS Institute Inc. Statistical Analysis Program Version 9.4. Cary, NC, 2016.
   Google Scholar
• Hall, L. W.; Jr.; Killen, W. D. Temporal and Spatial Assessment of Water Quality, Physical Habitat, and Benthic Communities in an Impaired Agricultural Stream in California's San Joaquin Valley. J. Environ. Sci. Health A Tox. Hazard. Subst. Environ. Eng. 2005, 40, 959–989. DOI: 10.1081/ESE-200056123.
   PubMed Web of Science ®Google Scholar
• Graham, M. H. Confronting Multicollinearity in Ecological Multiple Regression. Ecology 2003, 84, 2809–2815. DOI: 10.1890/02-3114.
   Web of Science ®Google Scholar
• Dormann, C. F.; Elith, J.; Bacher, S.; Buchmann, C.; Carl, G.; Carré, G.; Marquéz, J. R. G.; Gruber, B.; Lafourcade, B.; Leitão, P. J.; et al. Collinearity: A Review of Methods to Deal with It and a Simulation Study Evaluating Their Performance. Ecography 2013, 36, 27–46. 07348. DOI: 10.1111/j.1600-0587.2012.
   Web of Science ®Google Scholar