Toward Evidence-Based Urban Planning

Figures & data

Table 1 Typical methods in four types of literature reviews common in urban planning.

	Critical reviews	Scoping reviews	Systematic reviews	Meta-analyses
Search strategy	Will often identify most significant articles	Comprehensiveness determined by constraints	Exhaustive and comprehensive search	Exhaustive and comprehensive search
Appraisal	No formal appraisal; may consider impact of articles	No formal appraisal	Quality assessments	Appraisals can be used to determine inclusion
Synthesis	Usually narrative; can be conceptual	Tabular and narrative synthesis	Narrative and tabular synthesis	Numerical data with narrative synthesis
Analysis	Develop conceptual contribution; derive new theory	Characterize literature by key features	Determine what is known/gaps, certainty of evidence, recommendations for practice	Numerical analysis

Source: Adapted from Table 1 in Grant and Booth (2009).

Table 2 Sample risk of bias analysis criteria.

Criteria	Guiding questions	Evaluation
Components rating
Selection bias	(Q1) Are the individuals selected to participate in the study likely to be representative of the target population? Very likely Somewhat likely Not likely Can’t tell (Q2) What percentage of selected individuals agreed to participate?	Strong = Sample’s sociodemographic characteristics compared with the general population (e.g., through the census) and little discrepancy is identified OR 60% or more agreed to participate Moderate = Sample characteristics compared with the general population and some discrepancy is identified OR use multiple data sources, some of which are highly representative (e.g., the census) OR representative sampling strategy Weak = No comparison with general population and very small sample (< 100)
Study design	Indicate the study design Was the study described as randomized? No/Yes If Yes, was the method of randomization described? No/Yes If Yes, was the method appropriate? No/Yes	Strong = Natural experiment design Moderate = Randomized cross-sectional design Weak = Cross-sectional design with no randomization
Confounders	(Q1) Were there important differences between groups prior to the intervention? Yes/No/Can’t tell (Q2) If yes, indicate the percentage of relevant confounders that were controlled (either in the design [e.g., stratification, matching] or analysis)? 80%–100% (most)/ 60%–79% (some)/less than 60% (few or none)/Can’t tell	Strong = Compared groups and found minimal differences Moderate = Compared groups and found some differences OR controlled well for confounders Weak = Did not compare groups OR did not control well for confounders
Blinding	(Q1) Was (were) the outcome assessor(s) aware of the intervention or exposure status of participants? Yes/No/Can’t tell (Q2) Were the study participants aware of the research question? Yes/No/Can’t tell	Strong = Authors state the participants were not aware of research question during data collection Moderate = Unclear whether participants were not aware of research question during data collection Weak = Authors state participants were aware of research question during data collection
Data-collection methods	(Q1) Were data-collection tools shown to be valid? Yes/No/Can’t tell (Q2) Were data collection tools shown to be reliable? Yes/No/Can’t tell	Strong = data-collection tools result in measured data (accelerometry, GPS) Moderate = self-reported but rigorous survey Weak = Self-reported through poorly described survey OR data originate from different sources
Withdrawals and dropouts	(Q1) Were withdrawals and dropouts reported in terms of numbers and/or reasons per group? Yes/No/Can’t tell/Not applicable (i.e., one-time surveys or interviews) (Q2) Indicate the percentage of participants completing the study. (If the percentage differs by groups, record the lowest.) 80%–100%/60%–79%/Less than 60%/Can’t tell/Not applicable (i.e., retrospective case-control)	Strong = Withdrawal and dropouts reported (numbers and reasons), minimal differences identified between withdrawals and sample Moderate = Withdrawal and dropouts reported (numbers and reasons) with some differences identified OR 60% or more participants completed the study Weak = Withdrawals and dropouts not reported. Less than 60% completed the study Not applicable = All cross-sectional research designs
Intervention integrity	(Q1) What percentage of participants received the allocated intervention or exposure of interest? 80%–100%/60%–79%/Less than 60%/Can’t tell (Q2) Was the consistency of the intervention measured? Yes/No/Can’t tell (Q3) Is it likely that subjects received an unintended intervention (contamination or co-intervention) that may influence the results? Yes/No/Can’t tell	Strong = Consistency of intervention measured AND no reason to believe participants received an unintended intervention beyond regular daily life Moderate = No reason to believe participants received an unintended intervention beyond regular daily life Weak = Reason to believe participants received an unintended intervention beyond regular daily life Not applicable = All cross-sectional research designs
Analyses	(Q1) What statistical methods are used? (Q2) Are the statistical methods appropriate for the study design? Yes/No/Can’t tell	Strong = Uses statistical modeling Moderate = Uses descriptive statistics Weak = Presents results with no statistical analysis or unclear analysis
Global rating
List all component ratings		Strong = no weak ratings on any component Moderate = 1 weak rating on a component Weak = 2+ weak ratings on any components

Adapted by the authors from Effective Public Healthcare Panacea Project (2021). Criteria and guiding questions are from original EPHPP tool (Effective Public Healthcare Panacea Project, 2021). Specific evaluation criteria were developed by the authors to appraise survey-based urban planning research.

Figure 1. Sample risk of bias assessment from Figure 4 (p. 17) in Desrosiers et al. (2020), which is licensed under Creative Commons Attribution 4.0 International License: https://creativecommons.org/licenses/by/4.0/. Half of the original Figure 4 is presented herein.

Table 3 Sample evaluation of certainty of evidence criteria.

Domain	Judgment	Scoring	Criteria
Risk of bias (review quality)	No serious RoB	0	≥ 75% of the reviews were considered high quality as assessed by the AMSTAR2^a
	Not serious, borderline RoB	–0.5 points	≥ 75% of the reviews were considered low/moderate quality as assessed by the AMSTAR2^a < 75% of the reviews were considered critically low quality as assessed by the AMSTAR2^a
	Serious RoB	–1 point	≥ 75% of the reviews were considered critically low quality as assessed by the AMSTAR2^a
Inconsistency	No serious inconsistency	0	The direction and magnitude of the effect was consistent across reviews (> 75% in same direction) Heterogeneity could be explained by measurement of AT and PA (self-report vs. device)
	Not serious, borderline inconsistency	–0.5 points	The direction and magnitude of effect was inconsistent across reviews Heterogeneity could be partially (but not completely) explained by measurement of AT and PA (self-report vs. device)
	Serious inconsistency	–1 point	The direction and magnitude of the effect was inconsistent across reviews Heterogeneity could not be explained by measurement of AT and PA (self-report vs. device)
Indirectness	No serious indirectness	0	There was good global representation of primary studies included in the reviews There was little-to-no inclusion of other age groups captured in the reviews
	Not serious, borderline indirectness	–0.5 points	There was good global representation of primary studies included in the reviews There was considerable inclusion of other age groups captured in the reviews
	Serious indirectness	–1 point	There was limited global representation of primary studies included in the reviews There was considerable inclusion of other age groups captured in the reviews
Imprecision	No serious imprecision	0	The total number of participants across all studies was greater than 10,000^b
	Serious imprecision	–1 point	The total number of participants across all studies was less than 10,000^b
Publication Bias	No serious risk of publication bias	0	Due to the systematic and comprehensive search strategy including a scan of the gray literature, an a priori decision was made not to downgrade for risk of publication bias.

Notes: AT: active transportation; GRADE: Grading of Recommendations Assessment, Development and Evaluation; PA: physical activity; RoB: risk of bias. The quality of the evidence was upgraded if there is not cause to downgrade, and there was evidence of a large magnitude of effect from meta-analyses. Half-points were combined across domains to yield a total score; if the final scoring included a half-point, we conservatively rounded down (i.e., –0.5 points = –1 point). The quality of the evidence can be interpreted as follows: High—we are confident that the true direction of association between AT and PA lies close to the association we have estimated, and further research is very unlikely to change our confidence in the effect; Moderate—we are moderately confident that the true direction of association between AT and PA is likely to be close to the association we have estimated, but there is a possibility that it substantially different; further research is likely to have an important impact on the confidence in the direction of association and may change the direction of association; Low—we have limited confidence; the true direction of association between AT and PA may be substantially different from our estimate; Very low—we have very little confidence; the true direction of association between AT and PA may be substantially different from our estimate. ^aECE based off the AMSTAR2 RoB from Shea et al. (2017). ^bFrom Daniel (2012). Source: Supplemental Table 2 in Prince et al. (2021).

Figure 2. Sample portion of evaluation of certainty of evidence. Source: Prince et al. (2021, p. 16).

Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26(2), 91–108.

 PubMed Web of Science ®Google Scholar

Effective Public Healthcare Panacea Project. (2021). Quality assessment tool for quantitative studies. https://www.ephpp.ca/quality-assessment-tool-for-quantitative-studies/

 Google Scholar

Desrosiers, A., Betancourt, T., Kergoat, Y., Servilli, C., Say, L., & Kobeissi, L. (2020). A systematic review of sexual and reproductive health interventions for young people in humanitarian and lower-and-middle-income country settings. BMC Public Health, 20(1), 666. https://doi.org/10.1186/s12889-020-08818-y

 PubMed Web of Science ®Google Scholar

Shea, B., Reeves, B., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., & Henry, D. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. British Medical Journal, 358, j4008. https://doi.org/10.1136/bmj.j4008

 PubMed Web of Science ®Google Scholar

Daniel, J. (2012). Choosing the size of the sample. In J. Daniel (Ed.), Sampling essentials: Practical guidelines for making sampling choices. SAGE Publications, Inc.

 Google Scholar

Prince, S., Lancione, S., Lang, J., Amankwah, N., de Groh, M., Garcia, A., Merucci, K., & Geneau, R. (2021). Are people who use active modes of transportation more physically active? An overview of reviews across the life course. Transport Reviews. Advance online publication. https://doi.org/10.1080/01441647.2021.2004262

 Web of Science ®Google Scholar