Power Analysis, Sample Size, and Assessment of Statistical Assumptions—Improving the Evidential Value of Lighting Research

Figures & data

Table 1. Summary of basic findings from review of recent lighting papers and papers relating to spatial brightness research since 2000.

Variable	Lighting Research & Technology and LEUKOS, 2017	Spatial brightness papers since 2000
Total number of journal papers in 2017	83	N/A
Number of papers included in final review	37	13
Research design	84% (31) within subjects	54% (7) within subjects
	11% (4) between subjects	15% (2) between subjects
	5% (2) mixed (within and between subjects)	31% (4) mixed (within and between subjects)
Median sample size	23 for within subjects	35 for within subjects
	30 for individual groups in between subjects^a	21 for individual groups in between subjects^a
Statistical tests used	ANOVA (including MANOVA) = 62%	ANOVA (including MANOVA) = 46%
	t-Test = 22%	t-Test = 23%
	Regression = 27%	Regression = 8%
	Correlation = 32%	Wilcoxon signed rank/Friedman test/Kruskal-Wallis = 8%
	Wilcoxon signed rank = 11%	Other^b = 23%
	Kruskal-Wallis/Friedman test = 11%	No inferential statistics reported = 23%
	Other^b = 19%
	No inferential statistics reported = 8%
Assessment of assumptions of statistical test(s) used	24% of papers (9)	15% of papers (2)
Report measure of effect size	30% of papers (11)	8% of papers (1)

^aIn studies with unequal group sizes, the mean group sample size was used in the calculation of the median group size across all studies.

^bIncludes McNemar’s test, Cochran’s Q, post hoc Tukey’s tests, standardized residual sum of squares, variance stable rank sums, binomial test, Dunn-Rankin test.

Table 2. Assumptions of parametric statistical tests.

Assumption	Description
Data are measured at least at interval level	The response or property being measured should be recorded using a dependent variable on an interval scale, minimum, or on a continuous scale. The intervals on the scale should represent differences of equal magnitude. For example, if a 1–5 rating scale is used to measure a participant’s perceived brightness of a space, the difference in perceived brightness between ratings of 1 and 2 should be the same as it is between ratings of 4 and 5
Data are independent	Data from one participant should not influence data from another participant, which can be addressed through randomization in experimental design. In within-subjects designs, we do not expect the responses from the same participant to be independent, but responses between different participants in within-subjects designs should be independent. In regression analysis, the errors in the regression model should also be uncorrelated
Data are normally distributed	The raw data within each condition approximate a normal distribution or the residuals (individual minus the mean value) approximate a normal distribution, depending on the type of test being carried out
Variance is the same throughout the data	When comparing more than one group of participants, each of these groups should have approximately equal variance. If carrying out a correlation, the variance of one of your variables should be stable at all levels of the other variable. This is known as homogeneity of variance, or homoscedasticity, particularly in relation to regression analysis. In within-subjects designs with three or more conditions, an assumption of sphericity is also made. Sphericity refers to the variances of the differences between pairs of conditions being equal across all combinations of conditions

Fig. 1. Histograms of simulated data with (left) normal and (right) nonnormal distributions.

Fig. 2. Quantile–quantile plots of simulated data with (left) normal and (right) nonnormal distributions.

Fig. 3. Box plot visualizations of the simulated (left) normal and (right) nonnormal data sets.

Fig. 4. Probability that the Shapiro-Wilk test will be significant for a sample taken from marginally normal population, by sample size.

Fig. 5. Example plots of fitted values compared with residuals for linear regression model, showing data with (left) equal variance and (right)unequal variance.

Fig. 6. Density plots showing distributions of two populations of simulated reaction time data, generated using the “pastecs” package in R.

Table 3. Nonparametric alternatives to commonly used parametric tests (see Motulsky 1995).

Assessment being made	Parametric test	Nonparametric test
Compare one group to a hypothetical value	One-sample t-test	Wilcoxon signed rank test
Compare two independent groups	Independent t-test	Mann-Whitney test
Compare three or more independent groups	One-way ANOVA	Kruskal-Wallis test
Compare two dependent groups (within subjects)	Dependent t-test	Wilcoxon signed rank test
Compare three or more dependent groups	Repeated measures ANOVA	Friedman test
Association between two variables	Pearson correlation	Spearman correlation
Predict value based on another value	Linear/nonlinear regression	Nonparametric regression/logistic regression

Table 4. Small, medium, and large effect size criteria conventions, for different effect size measures, based on statistical tests used in studies published in Lighting Research & Technology and LEUKOS in 2017.

Effect size statistic	Statistical test(s)	Small effect	Medium effect	Large effect
Cohen’s d	One-sample t-test	0.2	0.5	0.8
Cohen’s dz	Dependent t-test; Wilcoxon signed rank test	0.2	0.5	0.8
Cohen’s f	Repeated measures ANOVA; between-subjects ANOVA	0.1	0.25	0.40
Cohen’s f^2	Regression	0.02	0.15	0.35
Odds ratio	McNemar’s test	1.22	1.86	3.00
Kendall’s w	Friedman ANOVA	0.1	0.3	0.5
g	Binomial test	0.05	0.15	0.25

Fig. 7. Calculated power of one-tailed independent t-test for between-subjects designs (left) and dependent t-test for within-subjects designs (right), by sample size and effect size (Cohen’s d for between subjects, Cohen’s d_z for within subjects), assuming an alpha of 0.05. Independent t-test uses group sample size, not total sample size.

Fig. 8. Estimated power of statistical tests used in reviewed papers, based on sample size and other parameters such as number of measurements (in within-subjects methods), for small (top), medium, (center) and large (bottom) effect sizes. Vertical dashed line indicates conventional minimum recommended power of 0.8 (Cohen 1988).

Motulsky H. 1995. Intuitive biostatistics. New York (NY): Oxford University Press.

 Google Scholar

Cohen J. 1988. Statistical power analysis for the beahavioral sciences. New York (NY): Lawrence Erlbaum Associates.

 Google Scholar