Mixed Messages about Mixed Tenure: Do Reviews Tell the Real Story?

Abstract

Mixed tenure is the predominant development and regeneration strategy and is a key component of UK housing and urban policy. It is purported to provide wide-ranging social, environmental and economic benefits to residents. While there is a large literature on mixed tenure, policy makers are likely to rely on reviews and summaries of the evidence rather than primary studies. But can they rely on such reviews? Using systematic review methods this paper critically appraises recent reviews for the evidence that mixed tenure policies and strategies have achieved any of these expected benefits. Of the six UK reviews of primary studies, most drew on less than half the available primary studies, none provided a critical appraisal of individual studies and made no comment on conflicting evidence between and within studies. While the reviews gave indications of the deficiencies of the evidence base, rather than focus on the implications of these deficiencies, four of the six reviews emphasised the positive effects of tenure mix.

Keywords:

• Housing policy
• housing tenure
• social housing
• mixed tenure
• systematic review

Introduction

Mixed tenure has become the predominant development and regeneration strategy over the past 15 years or so, and is a key component of housing, planning, regeneration and urban policy in the UK and elsewhere. Looking across West European countries, Galster has described social diversity and tenure mixing as a ‘widespread policy thrust’ (2007, p. 20), even though it is pursued in different ways in different places.

Although these are recent developments, it should be noted that social mix has a long history in UK housing and planning policy, although at different periods of time. Cole & Goodchild (2001) remarked that ‘social mix and balance are a recurrent theme in the history of housing, planning and urban policy in Britain’ (p. 351). Indeed Sarkissian (1976) demonstrates the role that social mix played in the Victorian model villages of Buckingham, Cadbury and Rowntree and in the Garden City Movement at the start of the 20th century, although she remarks that it was a ‘limited form of residential social mix’ whereby, certainly in the case of the garden city developments, people were ‘segregated according to class and income on the micro-level’ (p. 235). Cole & Goodchild (2001) continue the story, identifying the hey-day of policies of social mix and balance in the council housing and New Town programmes of the 1940s and 1950s. They describe how both of these programmes failed in their objectives, as the former did not promote social interaction between socio-economic groups and the latter allowed clustering by social class and the development of ‘a complex hierarchy of status differentials’ within larger developments, following which ‘social balance slipped down the urban policy agenda from the 1960s onwards’ (p. 353).

However, after a period of renewed expansion of owner-occupation and retrenchment of council housing under successive Conservative Governments of the 1980s and 1990s in the UK, social mix has again received policy attention. Following a debate prompted by studies of ‘unbalanced communities’ on social housing estates in the 1990s (see Page, 1993; Page & Boughton, 1997), New Labour Governments have repeatedly stated their preference and requirement for mixed tenure communities. This appears in broad statements of urban policy, for example in the final report of the Rogers' Urban Task Force: ‘Whether we are talking about new settlements or expanding the capacity of existing urban areas, a good mix of incomes and tenure is important for a number of reasons …’ (Urban Task Force, 1999).

Following this, urban policy objectives were set out in the government's Sustainable Communities plan (ODPM, 2003), a ‘long-term programme of action for delivering sustainable communities’ through new developments in southern areas of housing shortage, and the redevelopment of areas of abandoned housing in the north of England. One of the stipulated key requirements for these sustainable communities was: ‘A well-integrated mix of decent homes of different types and tenures to support a range of household sizes, ages and incomes’ (p. 5). This included both ‘promoting mixed communities’ in areas of council housing, and ‘encouraging a better mix of housing in new [private] developments’ (p. 37).

The emphasis on housing and social diversity has also informed regeneration policy and major programmes such as the New Deal for Communities launched early in New Labour's first term to tackle the social exclusion of deprived areas. This programme is expected to achieve the ‘mixed communities’ (Social Exclusion Unit, 2001) and ‘broad social mix’ (Social Exclusion Unit, 2000) advocated in overarching statements of neighbourhood renewal policy. Cole & Goodchild (2001) argue that in its aims for neighbourhood regeneration, the more recent policy emphasis on social mix is different to that of the post-war period; whereas the latter aimed to reduce class differences through the mechanism of the ‘melting pot’ (although they also say this was not achieved), New Labour aims to reduce social exclusion and promote social mobility through cross-tenure bridging social capital, rather than through social mixing within housing tenures and neighbourhoods. In the authors' view, New Labour policies are ambivalent about the desired scale and extent of social mixing to be achieved in more- or less-deprived communities.

Mixed tenure objectives can also be seen in statements of housing policy (e.g. Department of Environment, 1995), and associated planning guidance:

The Government believes that it is important to help create mixed and inclusive communities, which offer a choice of housing and lifestyle. It does not accept that different types of housing and tenures make bad neighbours. (Department of the Environment, 2000)

Later in a planning statement on housing: ‘The Government is seeking … to create sustainable, inclusive, mixed communities in all areas, both urban and rural’ (ODPM, 2006).

In its last major statement of housing policy, the New Labour Government aimed to boost the development of affordable housing due to a housing supply deficiency and pricing problems for young households. This was to be done through the ‘wider delivery of affordable housing in mixed communities’ (DCLG, 2007, p. 11). In the context of more government funding for an expansion of social housing supply, local authorities were said to be aiming for ‘a better mix of quality, type, distribution and tenure of housing in localities’ and to develop some estates into ‘more sustainable mixed communities’ (p. 78). Thus, both council estates and other areas, were to become more mixed in tenure terms.

Mixed tenure communities are purported to provide wide ranging benefits to residents, in particular addressing issues such as better neighbourhood reputation, better facilities and services, increased social cohesion and community participation, role models for work and education, more job opportunities, and residential sustainability. A fuller overview of alleged mixed community benefits and hypothesised mechanisms is given in Kearns & Mason (2007). Over a decade after the notion of mixed communities has once again become prominent, and started to achieve the status of conventional wisdom in housing, planning and urban policy circles, a question might be asked: what is the evidence that mixed tenure achieves any of these benefits?

Why Review Reviews?

There is a diverse, global literature on tenure mix ranging from commentaries, analysis of policy, good practice guidelines, descriptive primary studies and reviews of these. This array of literature can make it difficult to easily or quickly extract what evidence there is that mixed tenure policies ‘work’, for whom and in what contexts. For policy to be evidence-informed, evidence must be timely and needs to be presented to policy makers succinctly (Nutley et al., 2007; Petticrew et al., 2004). Review articles present an obvious vehicle for providing a synthesis of the evidence in a field such as this, and have become more popular in recent years. However it is known that traditional, non-systematic literature reviews have been criticised for being biased, with little or no transparency of why studies are included or excluded from the review, little or no information about their scope and little critical appraisal of the quality of the studies reviewed (Chalmers et al., 2002; Petticrew, 2001).

Thus, reviews constitute critical material as a result of the fact that evidence-based policy has ‘emerged as central to policy making and governance in Britain’ (David, 2002, p. 213). However, according to Powell & Maynard (2007), one of the results of the recent ‘modernisation’ of public policy making is that ‘Since 1997, New Labour has placed a great deal of stress on different types of review’ (p. 157). Yet, although the government's own internal guidance on policy evaluation and analysis, ‘The magenta book’ (PMSU, 2003), ‘tends to favour systematic reviews’ as the means of synthesising what is already known about a policy and its effects, in reality government receives review evidence of various forms including narrative reviews which ‘almost always involve selection and/or publication bias’ (Powell & Maynard, 2007, p. 160) and more often ‘rapid reviews’ which are ‘less comprehensive than systematic reviews’ (p. 161). It seems likely, therefore, that the reviews used by many policy makers could be less thorough or rigorous than they might be.

In the light of these developments, the aim of this paper was to identify and critically appraise recent reviews for the evidence that mixed tenure policies and strategies have achieved any or all of the expected benefits outlined above. This review of reviews examined the quality of the reviews by taking a systematic review approach. The systematic review approach involves attempting to locate and include all relevant published and unpublished studies to reduce the impact of publication bias, an explicit description of inclusion and exclusion criteria to reduce the likelihood of selection bias by the reviewers, a transparent and replicable assessment of the methods and quality of the studies being reviewed; and production of a summary using the evidence from those studies evaluated as being the most methodologically robust (Petticrew, 2001; Petticrew & Roberts, 2006; Powell & Maynard, 2007). Thus systematic reviews are transparent in what evidence has been collected, the quality of the evidence, how it has been synthesised and conclusions have been drawn.

To the authors' knowledge such critical appraisals of literature reviews in this field have not been done, although a number of systematic reviews of primary studies have been undertaken examining the evidence for the effects on health and wellbeing of housing refurbishment/urban renewal (e.g. Thomson et al., 2009).

In particular, the study wanted to know the following, in line with best practice in systematic reviewing:

• how comprehensive or inclusive the reviews were;

• whether they described and appraised the primary studies in terms of their methods, samples and analyses, and compared these across studies;

• whether study limitations were appraised;

• whether the reviewers' conclusions about the presence or absence of evidence were justified by the findings of the review itself and the quality of the studies examined; and

• whether needs for future research were identified.

This paper also summarises the reviewers' overview of the evidence on the effects of mixed tenure across a range of impact domains (described later).

Methods

The Search Procedure

This review sought to include all reviews of primary UK studies published between 1995 ¹ and February 2009, and focused on the effects of mixed tenure on social cohesion and social capital, social norms (attitudes and expectations), area reputation, health and health related behaviour, economic effects and environmental effects; these are the main domains of effects claimed by advocates of mixed tenure and/or focused upon in studies of mixed tenure. The search conducted was part of a broader strategy, which aimed to identify all UK evidence on the effects of mixed tenure. The search for primary studies and for reviews was conducted simultaneously and used the same methods.

Twelve databases were searched: ISI Web of Knowledge; International Bibliography of the Social Sciences (IBSS); SocIndex sociology research database; Copac National, Academic, and Specialist Library Catalogue; The Directory of Open Access Repositories – OpenDOAR; Social Care Institute for Excellence database (SCIE); Sociological Abstracts; Applied Social Sciences Index and Abstracts (Assia); FRANCIS Humanities and Social Sciences database; Opensigle System for Information on Grey Literature; OCLC WorldCat Dissertations and Theses; Planex-IDOX (covering all the subject areas of local public policy and governance), using the following search terms: mixed tenure; tenure mix*; mix of tenure*; mixed income communit*; mixed communit*; balanced communit*; mixed income new communit* (MINCs); mixed income*; tenure diversification; dispersed tenure*. The search was restricted by date (1995 to 2009) and geography (UK).

In addition, 12 UK experts were asked for their knowledge of existing reviews the search may have missed. With the same purpose, the reference lists from the papers identified were also examined.

Screening and Reviewing Procedures

A reference was defined as a review if the term ‘review’ was in the title, and/or if the stated aim or focus of the paper or report was to ‘review’ or ‘summarise’ the evidence for mixed tenure effects. Titles and abstracts of all documents and, where these did not contain sufficient information, full papers were screened independently by two authors (LB, ES).

Two authors (LB, ES) evaluated the reviews independently, addressing the research questions outlined above. Disagreement between the reviewers was resolved by discussion between all authors. Disagreement tended to be due to the sometimes vague or convoluted statements about evidence, for example:

• The use of ‘little evidence’ being interpreted as meaning either there is ‘little (no) evidence’ or there is ‘a little bit of evidence’, e.g. ‘There was also little evidence of inter-tenure socialising’ (see Wood, 2003, p. 49);

• Understanding what is meant by the use of ‘claim of evidence is exaggerated’, e.g. ‘Overall the researchers also found that the kinds of claims made in relation to mixed tenure have been exaggerated’ (see Atkinson, 2005, p. 17).

• The use of double negatives, e.g. ‘Rowlands et al., 2006 … found no evidence that mixed tenure had a negative impact on property values’ (see Bailey & Manzi, 2008, p. 6);

• A lack of clarity in reporting and interpreting studies' outcomes, e.g. ‘Allen et al., 2005 … found that there was general satisfaction with all the communities, but mixed tenure was only one element’ (see Holmes, 2006, p. 5).

In some cases, it was necessary to refer to the texts of primary studies in order to clarify questions raised by the ways in which reviews described primary evidence.

Search Results

A total of 240 references were identified from searching the databases. Three additional references were found through hand searching reference lists. No further reviews were identified following expert consultation.

Of the total 243 references, 108 were excluded as being not relevant to mixed tenure (e.g. they referred to mixed use or ethnic mix rather than to social or tenure mix). For this review of reviews, from the remaining 135 citations 127 references were excluded as being either: (a) primary studies (n = 22) and secondary studies (n = 4) focusing on effects or correlates of tenure mix; or (b) policy reviews, policy discussions and practice guides which did not explicitly review the evidence on effects of mixed tenure, mixed tenure community descriptions, or studies on housing issues which did not directly focus on effects of mixed tenure (n = 101).

Eight reviews were examined in detail (Atkinson, 2005; Bailey & Manzi, 2008; Berube, 2005; Holmes, 2006; Kleinhans, 2004; Tunstall, 2003; Wood, 2003; Tunstall & Coulter, 2006) but at this point excluded two reviews (Berube, 2005 and Tunstall, 2003), both of which were more focused on reviewing mixed tenure policy than on the evidence of the effects of mixed tenure, although they did include some discussion of the latter.

Six papers were therefore included in the review: four reports (Atkinson, 2005; Bailey & Manzi, 2008; Holmes, 2006; Tunstall & Coulter, 2006) and two peer-reviewed journal articles (Kleinhans, 2004; Wood, 2003). Two reviews included studies from other countries as well as from the UK (Kleinhans: Netherlands and Wood: Australia).

Table 1 provides a summary of the stated focus of the reviews and details of the breadth of the review, inclusion/exclusion criteria, timeframe and sources of data. The last row of the Table records the fact that none of the reviews appraised the quality of the primary studies, and this is an issue returned to later. A generous interpretation might be that the authors have appraised study quality and chosen to omit studies deemed to be ‘weak’. This is thought to be unlikely to have been the case, as it would be atypical of narrative reviews to do this in a thorough manner, as the full quotation from Powell & Maynard (2007) states:

Major limitations of the “traditional literature review” or narrative review are that they almost always involve selection and/or publication bias, and do not involve a systematic, rigorous, transparent and exhaustive search of all the literature' (p. 160)

Table 1. Summary of the six reviews

	Wood (2003)	Kleinhans (2004)	Atkinson (2005)	Holmes (2006)	Tunstall & Fenton (2006)	Bailey & Manzi (2008)
Journal article/report	Journal article	Journal article	Report	Report	Report	Report
Type of review	Narrative	Narrative	Systematic (search)	Narrative	Narrative	Narrative
Stated purpose	‘To test the robustness of … [MT] policy … by examining studies that present contrary evidence’	‘Review of the actual consequences of diversification in UK & Netherlands’	‘To look at existing research literature’	‘Summarise studies to inform future policies towards the promotion of successful mixed communities’	‘Present a summary of evidence about the promotion of mixed communities’	‘To review some of the recent research’
Inclusion criteria	Not stated	Not stated	‘Focus directly relevant to social mix in neighbourhood & robust methodological approach’	Not stated	Not stated	‘Range of studies’
Exclusion criteria	Not stated	Not stated	‘Low quality’	Not stated	Not stated	Not stated
Timeframe for search	Not stated	Not stated	Not stated	Not stated	Not stated	Not stated
Search terms	Not specified	Not specified	Specified	Not specified	Not specified	Not specified
Data sources	Not specified	Not specified	Specified	Not specified	Not specified	Not specified
Languages/countries included	Not stated	Yes	Yes	Yes	Not stated	Not stated
Tabular summary	No	No	No	No	No	No
Number of articles reviewed	Not stated	Not stated	Not stated	7	Not stated	Not stated
Study quality appraised	No	No	No	No	No	No

Appraisal of the Reviews

This section assesses the reviews in terms of their search and citation of primary studies, and their presentation of the evidence, to see for example whether, using the same UK evidence base, reviewers reached similar conclusions on issues, and whether their summary of the evidence matched the detailed findings of their reviews.

The Reviews' Search Methods and Citations of Primary Studies

Only one review was found that took a systematic approach to searching and retrieving the literature, providing details of search terms and databases searched, and methods such as inclusion and exclusion criteria (Atkinson, 2005). However, the description of inclusion and exclusion criteria was not sufficient to allow replication of the search strategy used in the review, namely: ‘… focusing on material directly relevant to the question of social mix in neighbourhoods …[and]…identifying literature based on a robust methodological approach’ (Atkinson, 2005, p. 3). Nor did this review state how many studies were identified from the search or specifically which studies were included in the review. No other reviews provided inclusion or exclusion criteria and only one review (Holmes, 2006) specifically stated which studies were reviewed. The lack of detail with respect to search methods and inclusion/exclusion criteria severely limits any readers' assessment of how well the field of interest has been reviewed. It is unclear whether the review from which conclusions about evidence are drawn is founded on a comprehensive look across the research base, or based on a selective reading of only part of the research base. Therefore, regarding search methods, the quality of the identified reviews was poor.

It is standard practice in reviewing to identify the included items, yet only one of the reviews provided a list the primary studies they included. Nevertheless, from the authors' own appraisal, Table 2 lists the primary UK studies cited in the body of the reviews (i.e. excluding those cited only in the Introduction or Discussion sections) where the reviews explicitly related the reference(s) to evidence of mixed tenure. Thus, this list does not necessarily represent all the existing primary UK studies providing evidence for mixed tenure effects. This Table excludes publications referred to by the reviewers as providing evidence of the effects of mixed tenure, but which on examination were either primary studies of other topics (e.g. gentrification), non-UK primary studies, commentaries, policy reviews or policy guidelines; approximately 30 citations across the reviews fitted these exclusion criteria. Excluding these, Table 2 presents a count of how many UK primary studies of mixed tenure were included in each review and the frequency each study was included across the reviews. Studies were cited one to five times across the reviews with Jupp (1999) being the primary study cited by most. The number of primary studies included in each review ranged from 5 to 12. Kleinhans (2004), Tunstall & Fenton (2006) and Bailey & Manzi (2008) included the greatest number of primary studies in their reviews. Obviously, the more recent reviews had the opportunity to include the most studies. Nevertheless, the proportion of primary studies identified from the reviews' citations, published up to one year before the publication of the respective review, ranged from 24 per cent to 73 per cent, but was about 50 per cent or less in four of the six reviews.

Table 2. Primary studies of mixed tenure cited to support claims of evidence of effect, no effect or absence of evidence in the six reviews

Year	UK Primary studies of mixed tenure	Wood (2003)	Kleinhans (2004)	Atkinson (2005)	Holmes (2006)	Tunstall & Fenton (2006)	Bailley & Manzi (2008)	Cited by Reviews
1997	Cole et al. (1997)		✓					1
	Page & Boughton (1997)		✓			✓		2
1998	Atkinson & Kintrea (1998)		✓			✓		2
	Harding (1998)					✓	✓	2
	Ramwell & Saltburn (1998)					✓		1
1999	Jupp, B. (1999)	✓	✓	✓		✓	✓	5
	Wood & Vamplew (1999)	✓						1
	Scottish Homes Research (1999)	✓						1
2000	Atkinson & Kintrea (2000)	✓	✓	✓		✓		4
	Pawson et al. (2000)	✓	✓	✓			✓	4
2001	Beekman et al. (2001)		✓	✓		✓	✓	4
	Atkinson & Kintrea (2001)	✓	✓	✓				3
2002	Knox & Alcock (2002)		✓			✓		1
2003	Groves et al. (2003)	*	✓			✓	✓	3
	Martin & Watkinson (2003)	*	✓		✓	✓		3
2004	Manzi & Smith-Bowers (2004)	*	*				✓	1
2005	Allen et al. (2005)	*	*	✓	✓	✓	✓	4
	Andrews & Reardon Smith (2005)	*	*	*		✓	✓	2
	Meen et al. (2005)	*	*	*	✓			1
	Silverman et al. (2005)	*	*	*	✓	✓	✓	3
2006	Rowlands et al. (2006)	*	*	*	✓		✓	2
	Tunstall & Coulter (2006)	*	*	*	*	*	✓	1
2007	Bailey et al. (2007)	*	*	*	*	*	✓	1
	Number of primary studies reviewed	6	11	6	5	13	12
	% possible studies cited	46%	73%	35%	24%	62%	52%
This list was generated from the UK mixed tenure studies cited in these reviews. This may not be an exhaustive list as there may be mixed tenure primary studies that none of the reviewers cited. Other studies not about mixed tenure and reviews were cited in these reviews but are not included in this list.*Primary study published after the Review publication date

Thus, whether or not systematic or narrative review methods were adopted, in most cases only a minority of available primary studies were examined by reviewers; bear in mind, as already mentioned, that most reviews do not state their inclusion or exclusion criteria, nor give any idea of the size or breadth of field from which they are drawing studies for review purposes.

Description of Primary Studies' Methods and Presentation of Evidence

Only one review (Atkinson, 2005) was found that provided brief descriptions of the studies reviewed, their aims, some indication of the methods used (diaries, surveys etc.) and size of the studies in terms of numbers of case studies included and/or numbers of participants providing data. These were not summarised in a Table but presented as part of the narrative. The lack of study details in most of the reviews prevents the reader from assessing the nature or robustness of the available evidence being reviewed. To say whether there is evidence for something is not enough; the reader also needs to be told whether that finding is based upon a small qualitative study (if so, of what kind?) or on a large quantitative survey (if so, conducted how and when?).

Table 3 summarises the evidence for the effects of tenure mix as presented by the authors of each of the reviews, and indicates which primary studies the review authors cited to support their statements. From the reviews, seven outcome domains were identified for potential impacts of mixed tenure, namely: social cohesion; social capital; residents' attitudes and outsiders' views; crime, anti-social behaviour and fear of crime; economic impacts; environment and amenities; and residential sustainability. The reviews' findings have been interpreted in the categories given below:

• an absence of evidence

  Table 3. Evidence of mixed tenure as reported in the six reviews assessed summarised as: no effect, positive or negative effects, mixed evidence or absence of evidence

  	Wood (2003)	Kleinhans (2004)	Atkinson (2005)	Holmes (2006)	Tunstall & Fenton (2006)	Bailey & Manzi (2008)	Summary of evidence (number of studies cited)
  Social cohesion
  Promotes social interaction/cohesion	No effect^[5, 9]	No effect ^[3, 4, 5, 7, 9]	No effect ^[1, 4, 5, 9]	No effect ^[1, 12]	No effect ^[1, 2, 4, 6, 9, 14, 17]	No effect ^[1, 9]	No effect (8 no effect, 3 negative)
  	Negative effect ^[19]	Negative effect ^[6, 7]
  Social capital
  Creates social capital, encourages norms & values, provides role models	No effect*	No effect ^[9]	No effect ^[1]	No effect ^[1]		No effect ^[1]	No effect (2 no effect)
  Residents' attitudes & outsiders' views
  Augments neighbourhood satisfaction		No effect ^[6, 9] Positive effect ^[9, 13]		No effect ^[1, 12] Positive effect ^[15]		Positive effect ^[1,17]	Mixed evidence (4 positive, 4 no effect)
  Improves area desirability/popularity			Positive effect ^[1]	Positive effect ^[1, 11]	Positive effect ^[1]	Positive effect^[1]	Positive effect (2 positive)
  Property values rise	Negative effect ^[19]			Positive effect ^[11], No effect ^[15]	Positive effect ^[11, 17]	No effect^[11, 15, 17]	Mixed evidence (3 no effect, 2 positive, 1 negative)
  Overcomes place-based stigma	Positive effect ^[4] No effect ^[19]	Positive effect ^[3, 11] No effect ^[7, 14]	Positive effect ^[4, 5], No effect ^[1]	Positive effect ^[1, 11]	Positive effect ^[1, 18]	Positive effect ^[1]	Mixed evidence (6 positive, 4 no effect)
  Crime
  Diminishes crime, ASB and fear of crime	Negative effect ^[19]	No effect ^[14]	Mixed evidence ^[14]	Mixed evidence ^[11]	Positive effect ^[14]	Positive effect ^[6]	Mixed evidence (4 negative, 2 positive, 2 no effect, 2 mixed)
  					No effect ^[6]	No effect ^[1]
  					Negative effect^[2, 17, 10]
  Economic impacts
  Increases income mix, reduces inequalities		No effect ^[2, 3, 4]		Mixed evidence ^[17]	Mixed evidence ^[17]	No effect ^[15]	No effect (5 no effect, 1 mixed)
  				No effect ^[1, 15]
  Creates job opportunities	No effect ^[16]			No effect ^[1]			No effect (2 no effect)
  Increases employment rates		No effect ^[6, 14]			No effect^[6, 8, 9, 14]	No effect ^[6, 8, 9]	No effect (6 no effect)
  Environment & amenities
  Improves physical environment**		Positive effect ^[3, 4, 6, 9, 14]		Positive effect ^[1, 12]		Positive effect ^[1]	Positive effect (7 positive)
  Attracts additional services	Absence of evidence*		Absence of evidence ^[14]		No effect ^[6, 14]	No effect ^[6]	No effect (3 no effect, 2 absence of evidence, 1 negative)
  						Absence of evidence ^[17]
  						Negative effect ^[18]
  Residential sustainability
  Renters can buy, supports kinship networks, reduces turnover		Positive effect ^[11]	Positive effect ^[1, 14]	Positive effect ^[11]	Positive effect^[1, 14]	Positive effect ^[1, 14, 17]	Mixed evidence (4 positive, 4 negative)
  				Negative effect ^[11, 15, 17]	Negative effect ^[18]	Negative effect ^[15, 17]
  Notes: * Indicates statement is made without referring to specific primary studies
  **Although most reviewers say this is due to planning and regeneration initiatives rather than Mixed Tenure per se; some may be due to owners' higher standards [1, 6, 9, 11, 13, 14]
  References cited by reviews for specific outcomes of mixed tenure:
  1. Allen et al. (2005)
  2. Andrews & Reardon Smith (2005)
  3. Atkinson & Kintrea (1998)
  4. Atkinson & Kintrea (2000)
  5. Atkinson & Kintrea (2001)
  6. Beekman et al. (2001)
  7. Cole et al. (1997)
  8. Harding (1998)
  9. Jupp (1999)
  10. Manzi & Smith-Bowers (2004)
  11. Martin & Watkinson (2003)
  12. Meen et al. (2005)
  13. Page & Boughton (1997)
  14. Pawson et al. (2000)
  15. Rowlands et al. (2006)
  16. Scottish Homes Research (1999)
  17. Silverman et al. (2005)
  18. Tunstall & Coulter (2006)
  19. Wood & Vamplew (1999)

• evidence of no effect

• evidence of a positive effect

• evidence of a negative effect

• mixed evidence (studies report a mixture of positive, negative, no effect or an absence of evidence).

In order to be fully informed about the nature of the evidence base, it is important to be able to distinguish between these types of findings, however, it was not always easy from the presentation of the evidence to classify the evidence within these categories.

The last column of Table 3 presents a summary of the review evidence and the number of studies cited for each category of evidence. Thus, for Social Cohesion, eight studies were cited across the six reviews as indicating there was no effect of mixed tenure on this outcome, and three studies indicated a negative effect. An overall summary emerging from the reviews of ‘no effect’ on this issue has been provided, although it could be argued that perhaps the evidence here is ‘mixed’. If the reviews had provided an indication of the quality of the studies it would be possible to be more certain about the evidence as more weight could have been given to the better or more robust studies. A further constraint here is that the reviewers do not comment on conflicting evidence between studies or even within one study. However, the balance of evidence as presented across the reviews is mostly clear to see and this is stated in the last column of the Table.

The remainder of this section presents an overview of the evidence for mixed tenure effects, as reported by the reviews themselves. Key summaries of what the reviews tell us in each domain are given in italics for ease of identification. It should be noted that from the reviews that no judgement can yet be made on the quality of the evidence base from which these summaries were drawn, but the reader should note that the authors intend to do this soon, in a separate examination of the primary study evidence base using systematic review methods. Rather, the purpose here is to examine consistency within and across reviews, not to directly examine the primary studies.

Social Cohesion and Social Capital

The evidence reported by the reviewers indicated that there was either no effect or negative effects of mixed tenure on social cohesion and therefore that tenure mix did not have any effect on social capital. Drawing on 11 studies (Allen et al., 2005; Andrews & Reardon Smith, 2005; Atkinson & Kintrea, 1998, 2000, 2001; Beekman et al., 2001; Cole et al., 1997; Jupp, 1999; Meen et al., 2005; Pawson et al., 2000; Silverman et al., 2005), all the reviews reported that there was no evidence of an effect of mixed tenure on social interaction and cohesion, although how this was expressed varied from ‘evidence of little interaction’ to “strong evidence that interactions are limited”. Wood (2003) and Kleinhans (2004) also reported there was evidence of negative effects such as tensions between tenures (Beekman et al., 2001; Cole et al., 1997; Wood & Vamplew, 1999). There was mixed evidence as to whether mechanisms that might facilitate social interaction such as schools and communal facilities are effective (data not shown in Table).

Citing two studies (Allen et al., 2005; Jupp, 1999), four reviews reported that there was no evidence that mixed tenure promoted social capital, or stated that ‘any claim that mixed tenure leads to enhanced social capital or creates role models is exaggerated’ (Atkinson, 2005, p. 17). Wood (2003) reported the absence of effect without referring to any primary study in particular.

Resident Attitudes and Outsiders' Views

There was mixed evidence from the reviews that mixed tenure changed residents' attitudes and outsiders' views about the area. Neighbourhood satisfaction in mixed-tenure areas was reported variously as having improved (Allen et al., 2005; Jupp, 1999; Page & Boughton, 1997; Rowlands et al., 2006; Silverman et al., 2005), staying unchanged (Beekman et al., 2001; Jupp, 1999; Meen et al., 2005), or being influenced by other factors, e.g. quality of physical environment and services (Allen et al., 2005).

There was mixed evidence that mixed tenure increases area desirability and popularity: two studies only (Allen et al., 2005 and Martin & Watkinson, 2003) were cited by four reviewers.

Evidence for change in property values, as a financial indicator of mixed-tenure neighbourhood popularity, varied throughout the reviews. Values were reported as having risen (Martin & Watkinson, 2003; Silverman et al., 2005), having fallen (Wood & Vamplew, 1999), or remaining unchanged (Rowlands et al., 2006), while it ‘it is not clear whether [any] increase was due to the social mix or to other factors’ (Bailey & Manzi, 2008, p. 5 referring to studies of Martin & Watkinson, 2003 and Silverman et al., 2005).

Evidence for change in neighbourhood reputation was mixed in the reviews. Mixed tenure communities, it was argued, could overcome place-based stigma (Allen et al., 2005; Atkinson & Kintrea, 1998, 2000, 2001; Martin & Watkinson, 2003), but this was a long-term process according to Tunstall & Coulter (2006). Reviews also reported absence of effects (Allen et al., 2001; Wood & Vamplew, 1999) or that tenure mix per se was an insufficient condition for change in area reputation (Cole et al., 1997; Pawson et al., 2000).

Crime, ASB, Fear of Crime and Other Behaviours or Perceptions

Reviews reported mixed evidence for effects on the incidence or perceptions of crime and anti-social behaviour (ASB): crime and ASB might increase (Andrews & Reardon Smith, 2005; Manzi & Smith-Bowers, 2004; Wood & Vamplew, 1999; Silverman et al., 2005); actual crime (Pawson et al., 2000) and fear of crime might reduce (Beekman et al., 2001; Pawson et al., 2000), but this depended on the type of crime (Pawson et al., 2000) and location (Martin & Watkinson, 2003). Absence of effect was also mentioned with references to Allen et al. (2005), Beekman et al. (2001), Pawson et al. (2000).

None of the reviews provided evidence for or against changes in health or well-being, health behaviours (drinking, exercise, diet) or psychosocial benefits (pride in area; sense of progress; feeling calm and peaceful; positive identity; raised status or self-esteem), indicating a likely absence of evidence for these effects.

Economic Effects

According to the reviews, mixed tenure did not produce change in income mix and inequalities, in job opportunities or employment rates for social tenants. Reviews referred to the fact that tenure mix did not produce any change in social disadvantage, inequalities and deprivation (Allen et al., 2005; Andrews & Reardon Smith, 2005; Atkinson & Kintrea, 1998, 2000, Rowlands et al., 2006), or that tenure mix did not translate into income mix except in ‘high demand areas’ and some new developments (Silverman et al., 2005). Nor, it appeared from the reviews, did tenure mix create job opportunities (Allen et al., 2005; Scottish Homes Research, 1999) or increase employment rates (Beekman et al., 2001; Harding, 1998; Jupp, 1999; Pawson et al., 2000). According to Kleinhans (2004, p. 382), tenure diversification only ‘contributed to a smaller concentration of unemployed people … by attracting economically active households’ (Beekman et al., 2001; Pawson et al., 2000).

Environmental Change and Local Amenities

The reviews reported that mixed tenure, and associated housing diversification, had a positive impact on the physical characteristics of housing estates (Atkinson & Kintrea, 1998, 2000; Beekman et al., 2001; Jupp, 1999; Pawson et al., 2000) with residents reporting high levels of satisfaction with the quality of the physical environment and range of services (Allen et al., 2005; Meen et al., 2005). Resident satisfaction was, however, attributed to these improvements rather than to tenure mix (Jupp, 1999).

There was no effect or an absence of evidence of an effect of mixed tenure on improving or attracting additional services. Any additional provision was reported as being due to planning policies rather than a consequence of mixed tenure (Beekman et al., 2001; Pawson et al., 2000). Other reported impacts were a negative effect, resulting in less targeted provision of services, presumably to areas which were less poor as a result of mixed tenure (Tunstall & Coulter, 2006), or a concern that it was unclear whether all residents could benefit from any newly provided facilities (Silverman et al., 2005).

Residential Sustainability

Some of the reviews reported that mixed tenure contributed to residential sustainability within an area; however negative effects akin to gentrification were also described. Five reviews reported positive effects of tenure mix such as: support of kinship networks and the possibility for families to stay in the area (Allen et al., 2005; Pawson et al., 2000); the opportunity for social renters to stay in the area and become owners due to Right to Buy (Pawson et al., 2000); and a reduction in residential turnover (Allen et al., 2005; Martin & Watkinson, 2003; Pawson et al., 2000, Silverman et al., 2005). With that, three of the reviews referred to the fact that tenure mix might lead to pricing resident families and less affluent residents out of the area (Martin & Watkinson, 2003; Rowlands et al., 2006; Silverman et al., 2005; Tunstall & Coulter, 2006) and to the proliferation of buy-to-let (Rowlands et al., 2006, Tunstall & Coulter, 2006).

Summarising the Evidence: Did the Reviewers' Conclusions Match their Reviews of the Evidence?

Twenty-three primary studies were cited overall (Table 2), with 19 cited to support statements of the presence or absence of evidence by the reviewers. One-third of the studies were included at most twice with respect to providing evidence on mixed tenure effects. Thus, as can be seen from Table 3, the evidence for or against most of the above outcomes is drawn primarily from a subset of studies. Allen et al. (2005) is referred to 29 times, Jupp (1999) 17 times, Beekman et al. (2001) 14 times, Silverman et al. (2005) 14 times, Pawson et al. (2005) 13 times and Martin & Watkinson (2003) 11 times. Thus, the evidence base for mixed tenure would appear to be less well populated by useable studies than the aggregate number of published studies would suggest. Indeed, for three outcome domains—social capital, employment and area popularity—only two primary studies are cited in each case, across all the reviews, so that it can at least be said that in these areas the evidence base is ‘thin’ (irrespective of whether the relevant research is robust or not).

It is also clear from Table 3 that what little evidence there is would indicate that mixed tenure has not achieved most of the expected outcomes; where there is evidence, it tends to be limited or mixed at best. Furthermore, it appears that any positive benefits have been achieved as a by-product of mixed tenure (i.e. provision of better quality housing and physical environment) rather than as a direct result of tenure mixing per se. However, given that the reviewers do not critically appraise the quality of the studies they reviewed, it may well be that in fact there is an absence of evidence for any effects of mixed tenure.

Overall, without the detail of breadth of search, inclusion/exclusion criteria, and information about the quality of the evidence presented in the primary studies being reviewed, the review methods are not transparent and their conclusions regarding presence or absence of evidence difficult to verify. What the study here can do is assess the degree of consistency maintained within the reviews, albeit given their limitations. Table 4 compares the reviewers' conclusions presented in the body of their paper/report, their abstract or executive summary, and the research evidence presented in the body of the review as summarised in Table 3. There was concordance between these for two reviews (Kleinhans, 2004; Wood, 2003), both peer-reviewed journal articles. For the other reviews, the summaries and conclusions were more positive than the evidence presented in the body of the reviews. For example, reviews stated that the evidence for tenure mix effects was weak or contradictory, but concluded that tenure mix has beneficial social and environmental impacts (Tunstall & Fenton, 2006, Atkinson, 2005, Holmes, 2006, Bailey & Manzi, 2008). The starkest contrast between concluding statements and the review evidence from primary studies occurred in the case of social interaction or integration effects, with two reviews (Holmes, 2006; Bailey & Manzi, 2008) making strong statements about the wholly positive impacts of mixed tenure without demonstrating the existence of any evidence to support such conclusions (compare the entries for these two reviews in the first row of Table 3 with their conclusions quoted in Table 4).

Table 4. Comparing reviewers' concluding statements in the body of the papers/reports to abstract or executive summaries

	Summary of citations in the review	Conclusion in abstract or executive summary	Conclusion or summary in review	Our assessment of match
Wood (2003)	Mixed evidence (5 no effect, 3 negative effect, 1 positive, 1 absence of evidence)	‘much of the research evidence that exists … questions the effectiveness of the strategy [of tenure diversification]’.	‘Much of the empirical evidence …, while limited, appears … to be weighted against these outcomes … The literature examined here demonstrates that most of the arguments put forward for the practice of diversification are at best inadequate’.	Summary, evidence in the body of review and conclusions match
Kleinhans (2004)	Mixed evidence (16 no effect, 10 positive, 2 negative)	‘The review shows ambivalent results that necessitate modest expectations, especially with regard to area reputation, cross-tenure social interaction and residential attitudes. This ambivalence is partly due to unclear policy goals and policy terms as well as vagueness about the relevant spatial level … positive role-model effects in neighbourhoods have not yet been adequately studied and therefore remain based on conviction’.	‘The main conclusion is that the evidence base is not as insubstantial as has often been suggested. It is not the lack of empirical findings but their ambivalent nature that is probably the reason for many authors to claim a small evidence base … most research literature partly or completely refutes several policy assumptions. The only exceptions are improvements in the quality of housing and the physical living environment’.	Summary, evidence in the body of review and conclusions match
Atkinson (2005)	Mixed evidence (6 no effect, 5 positive, 1 mixed, 1 absence of evidence)	No abstract or executive summary	‘MT areas seem to deliver lower management costs, reduced or non-existent stigma and a broader range of services. However,… social networks of such areas are no more likely to be integrated between owners and renters than many other communities … and this has not been interpreted as a failure of these policies … [MT] has been an effective antidote to the problems associated with areas of mono-tenure social renting … elevat[ing] some areas to relatively unremarkable normality’, signalling ‘a success’.	Conclusions are more positive than evidence in the body of review. Summary is not available
Holmes (2006)	Mixed evidence (9 positive, 9 no effect, 3 negative, 2 mixed)	Mixed income and MT communities were ‘overwhelmingly judged successful’, ‘non-issues to residents’, ‘can attract young families’. ‘Tenure is not fixed, and it alters in a community’.	‘Mixed income communities can give low-income residents access to successful environments. … The research studies show that they are a tested way of delivering high quality, popular neighbourhoods which achieve socio-economic integration’	Summary and conclusions are more positive than evidence in the body of review
Tunstall & Fenton (2006)	Mixed evidence (14 no effect, 8 positive, 4 negative, 1 mixed)	‘A review of the evidence suggests that some of these goals [of MT] are more likely to be achieved than others’	There is a ‘measured optimism about the benefits of [MT] … Benefits that rely on there being increased mix, such as more and more varied services, are well evidenced. Where the benefits of mix rely on less measurable attitudes and behaviours and on interaction between residents, there is evidence, but… less decisive’	Summary and conclusions are more positive than evidence in the body of review
Bailey & Manzi (2008)	Mixed evidence (12 no effect, 9 positive, 3 negative, 1 absence of evidence)	‘MT communities have the potential to facilitate social interaction’ and ‘counteract social exclusion and adverse neighbourhood effects associated with mono-tenure…’. ‘Residents are not normally aware of tenure’, ‘The quality of design … has proven to be a major influence on social interaction’. ‘There is no evidence that MT … affects house prices’.	TM is ‘not a universal remedy for the problems of neighbourhoods’, ‘a necessary but not a sufficient precondition for sustainable communities’. ‘Attention needs to be paid to the design and layout of homes and their surroundings, the provision of the full range of facilities, as well as accessibility and integration into the wider locality’.	Summary and conclusions are more positive than evidence in the body of review

Discussion

This section will highlight some of the most important findings of this review of reviews, as they represent key messages for both policy makers and, more importantly, researchers in this field.

Unsystematic, Uncritical Reviews

A major limitation with all the reviews was that none provided sufficient information on, or critical appraisal of, the primary studies (e.g. the study methods, samples/participants or analyses undertaken) for the reader to judge the quality of the studies and therefore the quality of the evidence reviewed. Where reviewers were critical of the evidence they did not present clear reasons as to why the evidence was poor. No reviews discussed or presented possible explanations for the contradictory findings that they reported either within a study or across studies, and this is a crucial omission. It is a simple point, but a review of the evidence in any field must include a critical appraisal of the research itself, not simply comprise a short reporting of the stated findings of the research, albeit packaged together across a number of studies.

However, the reviewers did make general comments about the limitations of the current evidence base. Kleinhans (2004) noted that much of the evidence is based on case studies, thus limiting generalisability, and moreover examines the effects of tenure mix at the neighbourhood level when much impact might more likely occur at a micro or street level. It was also noted that there was a lack of longitudinal data, a need of quasi-experimental designs (Holmes, 2006; Kleinhans, 2004) and to control for potential confounders (Atkinson, 2005; Holmes, 2006; Tunstall, 2003). Interestingly, Tunstall & Fenton (2006) noted there was ‘a large literature on what “works” but too little on why it works’ (p. 45).

Thus, within the reviews there are indications of the deficiencies of the evidence base, but it is interesting that rather than focus on the implications of the quite considerable weaknesses, except for the two journal articles, the reviews mostly sought to emphasise (indeed, over-emphasise) what they could say about the positive effects of tenure mix. It is suspected that this is a function often of their raison d'être, namely to provide evidence for policy, and yet such a purpose could equally justify a more critical stance.2

It may be unrealistic to have expected these reviews to have taken a systematic approach to reviewing the literature, although it would not have been inappropriate for them to have done so. Systematic review methods are not the sole preserve of medical health research as is widely believed. They were developed by the social sciences (Chalmers et al., 2002) and have been used widely to assess the evidence for policy and social interventions (see for example of breadth Petticrew, 2001). Because systematic reviews require at least two reviewers and can take more time than traditional reviews, they may not fit into the time scales of commissioned research. However, it is certainly within the scope of any review to provide (1) a clear aim of the review; (2) clarity that the review is a review of evidence of the effects or outcomes of a policy OR a review of evidence providing the rationale of a policy; (3) a critique of the evidence being reviewed; and (4) a sense of the coverage of the review.

The lack of a systematic approach and perhaps more importantly the lack of critical appraisal of the evidence in these reviews brings into question their utility and therefore, their conclusions. With almost no details presented about the primary case studies being reviewed, readers cannot draw their own conclusions about the evidence for mixed tenure; nothing is known of the breadth of studies reviewed, nor the quality of them. The one paper that did take a systematic approach to identifying studies (Atkinson, 2005) failed to enumerate the studies found with this approach and did not provide a critical appraisal of the studies that were included in his review. It can only be assumed from the call for more robust study designs and longitudinal data (Holmes, 2006; Kleinhans, 2004; Tunstall, 2003), that the studies presented in these reviews have less robust designs, but with lack of detail it is not possible to even discriminate between them to determine if any are better or worse than others.

Most importantly, without explicit critical appraisal within the reviews themselves, and full coverage of relevant primary studies (good or bad), it is difficult from these reviews to determine if there is an absence of evidence that mixed tenure meets policy objectives or whether the evidence is that it does not meet these objectives. As stated earlier, this is an important distinction for policy makers and other readers to be informed about. For example, there appears to be a consensus that mixed tenure does not promote social interaction and social cohesion and is therefore unlikely to create social capital, but as Kleinhans (2004) states, ‘not one of the studies reviewed has succeeded in developing a framework that enables a real test …’ of this (p. 381).

As identified by some reviewers, the difficulty of determining if there is evidence that mixed tenure ‘works’ is compounded by the vagueness of policy objectives (Berube, 2005; Tunstall, 2003; Wood, 2003), and the unanswered question of ‘how mixed is “mixed”?’ (Atkinson, 2005; Bailey & Manzi, 2008; Holmes, 2006; Kleinhans, 2004; Tunstall & Fenton, 2006; Wood, 2003)? It would seem necessary, if more robust evaluations were to be undertaken that some attention be paid to more clearly operationalising policy objectives, so that their achievement can be assessed.

Non-specific or Absent Framework for Presenting the Evidence

In this review of the reviews it was quite difficult to map the presentation of the evidence with the language used (e.g. ‘there is little evidence’, or the evidence is ‘greatly exaggerated’) to clearly assess whether there was evidence of: positive effects, negative effects, no effect or an absence of evidence. The evidence from the same studies was interpreted differently across reviews (e.g. three reviewers citing Allen (2005) concluded there was a positive effect of mixed tenure on overcoming place-based stigma, a fourth review, again citing Allen, concluded there was no effect (Atkinson, 2005). The study by Pawson et al. (2000) is cited as providing evidence of no effect (Kleinhans, 2004), positive effect (Tunstall & Fenton, 2006) and mixed evidence (Atkinson, 2005) on crime. Further, not only was it difficult at times to determine whether there was evidence of positive effects, negative effects, evidence of no effect or inadequate evidence, the reviewers made no comment, let alone judgement, where the evidence from the studies they reviewed appeared to provide conflicting evidence and sometimes uncritically presented conflicting evidence within the same study (e.g. Kleinhans (2004) stating that resident attitudes to mixed tenure are ‘positive’ or ‘ambivalent’ citing Page & Boughton (1997) for both, or ‘unconcerned’ or ‘negative’ citing Beekman et al. (2001); and Holmes (2006) reporting Martin & Watkins (2003) study as providing both positive and negative evidence for residential sustainability).

Small Number of Studies Cited

Another concern for the development of the evidence-base or what has been established as evidence is the reliance on a subset of studies to provide that evidence. The evidence for (lack of) social interaction between social renters and owner occupiers, and the conclusion that mixed tenure is not ‘an issue’ for residents, rests on just four frequently cited studies by Jupp (1999), Atkinson & Kintrea (1998, 2000) and Allen et al., 2005. Again, because these reviews provide no detail on the quality of these studies, it is difficult to determine from reading the reviews whether these conclusions are justified or not.

Researchers, from other disciplines, have drawn attention to the problem of how a small number of highly cited studies may ‘create unfounded authority’ (Greenberg, 2009), have undue influence in the development of or conceptualisation of theory (Moore et al., 2006), be over interpreted or misinterpreted (Cummins & Macintyre, 2002; Gebel et al., 2007), and through this process of interpretation and reproduction, provide a somewhat spurious evidence base in support of current policies or development of new policies. This has happened in urban studies before, most notably in the case of research on gentrification: a critical re-reading of Atkinson's review article (2004) shows that, despite the extent and volume of debate about the displacement impacts of gentrification, from over 1000 items published over four decades, from 1964–2002, he identified only seven primary empirical studies that had measured the nature and extent of ‘gentrification-induced displacement’ and even then ‘the measurements provided are not without caveats and problems’ (p. 114). There is clearly a risk, compounded by the replication of reviews such as studied here, that the mixed tenure policy evidence base could suffer a similar malaise and become the ‘new gentrification’, only this time, unlike gentrification, the casual repetition of headline findings from insufficiently critical reviews may lead people to conclude (either wrongly or more strongly than they ought) that mixed tenure policies are definitely, and wholly, beneficial, whereas they might have little effect or even negative impacts, but the oft-quoted evidence base would not show this.

Little Evidence or No Evidence Translated to Success or at Least ‘No Harm’

Perhaps of even more concern than that much of the evidence for mixed tenure is drawn from a small number or subset of the studies, is the mismatch between executive summaries and conclusions and the reviewers' findings of little or no evidence for mixed tenure achieving any policy objectives. Executive summaries or abstracts are quite likely to be more influential than academic papers or long reports; policy makers with little time want succinct presentations of evidence (Nutley et al., 2007). Clearly research needs to be summarised, and clear messages produced for policy makers. Unfortunately, the conclusions presented in many of these reviews offer a more positive view (or ‘it doesn't do any harm’ view) of the effects of mixed tenure than the evidence warrants. It is felt this was true for four of the six reviews examined, a not inconsiderable swaying of the main messages conveyed through the reviews.

Conflating Evidence Types: Policy Problems, Policy Approaches and Policy Effectiveness

To select the reviews for this paper a review was defined as a report or paper with ‘review’ in the title or with a stated aim to ‘review’ or ‘summarise’ the evidence of effects of mixed tenure. As was found, not only can it be difficult to identify whether a particular publication is indeed a review, the reader can also be confused as to what is being reviewed: social problems, policies themselves or policy evaluations. The two items omitted from this review of reviews—the article by Tunstall (2003) and the report by Berube (2005)—are interesting as illustrations of this conflation of review types. Although not formally reviewing the evidence, both studies reported that there was ‘limited and equivocal’ evidence (Tunstall, 2003) or that it was ‘thin’ (Berube, 2005) for the effects of mixed tenure. Tunstall's piece is clearly pitched as a review of policy itself; however, one-third of the space is devoted to reviewing the evidence for the ‘rationales’ for mixed tenure policy, namely ‘beneficial neighbourhood effects’. Citing less than a quarter of primary studies available at the time of the review, the review part of the article is limited in its coverage, yet nonetheless concludes that there is a ‘need for… robust supporting evidence’ (p. 158).3

The Berube report is more strongly presented as a research review, described in the accompanying short briefing paper as ‘a review of research evidence’ and as having ‘involved a review of relevant literature’ (Joseph Rowntree Foundation, 2005). Described as a ‘perspective’ on ‘mixed communities’, the report contains less evidence about mixed tenure and its effects than it does about the effects of ‘concentrated poverty’ (the US term for deprived areas in UK parlance). While this may be a rationale for mixed tenure, it is not the same thing as reviewing the evidence that mixed tenure will cure some of the identified problems. Although in some of its ‘lead-in’ comments the report says that ‘mixed income development is not a silver bullet’ (p. 26) and that the evidence on its achievements is ‘more thin than we might hope’ (p. 25), and despite only citing four primary studies of mixed tenure communities in the UK, the report describes the evidence about mixed tenure's effects and mixed tenure developments themselves in terms such as ‘successful’ (p. 44) and ‘impressive’ (p. 45).4 The report also recommends how mixed tenure strategies should be implemented to have greatest effect, based on what the ‘evidence suggests’ (JRF, 2005, p. 1). These strong statements of ‘fact’, and other similar statements about the potentially beneficial effects of mixed tenure in the future, are based on very little or no evidence and certainly not evidence that is critically appraised. Yet they demand attention and attract credibility by virtue of being the product of a so-called ‘review of the evidence’, however incomplete that review is.

The issue of reviewing evidence about a policy problem as if it provides evidence for a particular policy solution is not uncommon, although it is very problematic. Both these pieces illustrate how the review of evidence can sometimes be the poor relation to a review of policy, but that this does not prevent authors and publishers from giving the impression that a thorough review of evidence of effects has taken place. Some review publications then, on the basis of a limited review of evidence of effects, discuss mixed tenure policies as if they quite correctly and obviously are the ‘way to go’ in policy terms to deal with the problems of concern. Given the crucial importance of review documents, there is a need for greater care and clarity in conveying the nature and type of review being presented.

Conclusion

To achieve evidence-informed policy, policy makers need to know what the evidence is, the quality of the evidence and where there is no evidence. Relying on reviews such as those reviewed here would make that nearly impossible with respect to knowing whether or not mixed tenure policies achieve the desired social, environmental, or economic outcomes. Both policy makers and researchers can contribute to improving this situation.

Policy makers need to be clearer about what they mean by mixed-tenure, or at least perhaps clearer about the parameters of mixed communities (scale, nature and extent of mixing) which they would like to receive evidence about. They also need to treat the reviews they receive in a circumspect manner, asking questions of the reviewers and publishers, especially where the publisher is an organisation from within the policy community which might wish to push a particular policy line. In other words, the term ‘review’ should not grant a written piece any special privileges as to its veracity, even though the term ‘review’ can itself suggest a higher level of independence or rigour. The message is ‘don't take anything for granted!’ Policy makers also need to be more demanding of the evidence, rather than receiving limited and weak evidence as sufficient if it can possibly be presented as supportive of government policy. Policy makers have to want to hear the ‘inconvenient truth’ rather than the ‘comfortable spin’ on the evidence.

Researchers in the field of housing and urban studies have to be more systematic and critical, i.e. making more use of systematic review methods, and at minimum, taking a critical approach to the evidence from primary studies and reporting this critical appraisal in full. Being able to succinctly summarise the evidence for where mixed tenure works or does not, and identify the research gaps would be a valuable step forward. Both reviews and primary studies would benefit from greater clarification and more realistic expectations of what mixed tenure might achieve, including likely adverse effects. Again, conventional wisdoms such as that mixed tenure communities will resolve problems of ‘concentrated poverty’ need to be questioned. Too many researchers and reviewers appear to wish to please policy makers by presenting their summaries of evidence in the best possible light in support of the predominant policy direction: this cannot be healthy in the longer-term both in terms of the evidence base or the nature of policy actions. Alternatively, it may be that the demand from policy makers for ‘clear messages’ from academics has led to a reluctance to say that the evidence is ‘weak’ or ‘mixed’; yet it ought to be possible to present a varied and mixed picture of the evidence (if that is what it is) in a clear and balanced way. While there is no firm evidence of the extent to which policy makers make use of evidence, whether derived from primary studies or reviews (although the current public policy paradigm suggests that they use evidence more than in the recent past), it nonetheless behoves researchers to clearly provide an assessment of the quality of evidence and to be clear whether there is evidence of effect or an absence of evidence.

This review of the reviews of the evidence about the effects of mixed tenure has highlighted some of the current weaknesses in the evidence-based policy arena in the field of housing and urban policy. It is to be hoped that these reflect a current immaturity of the review paradigm that can be improved before too long. Others have already identified the need to improve the skills of policy makers, for example, in making ‘the analysis and use of evidence’ a core skill for civil servants (Bulmer et al., 2007, p. 99), in order to make ‘evidence-based’, ‘evidence-informed’ or ‘evidence-inspired’ policy (Duncan, 2005) stronger. It is argued that researchers also need to ‘up their game’ by being more robust in their review methods, more critical in their appraisals, and more independent in their presentation of the evidence.

Acknowledgements

This research was conducted as part of the GoWell Glasgow Community Health and Wellbeing Research and Learning Programme (www.gowellonline.com). GoWell is a collaborative partnership between the Glasgow Centre for Population Health, the University of Glasgow and the Medical Research Council Social and Public Health Sciences Unit, sponsored by Glasgow Housing Association, the Scottish Government, NHS Health Scotland and NHS Greater Glasgow and Clyde.

We would like to acknowledge the contribution of the Information scientist Candida Fenton, thank Hilary Thomson for her advice on systematic reviews and thank her and Matt Egan for their comments on an earlier draft of the paper. LB is funded by the Chief Scientist Office at the Scottish Government Health Directorate as part of the Evaluating the Health Effects of Social Interventions programme at the MRC/CSO Social and Public Health Science Unit (U130059812). ES is part of the Evaluating the Health Effects of Social Interventions programme at the MRC/CSO Social and Public Health Science Unit and funded by GCPH as part of GoWell. AK's input to the GoWell programme is funded by the University of Glasgow.

Notes

¹ 1995 was chosen as the start point for the search for reviews for the following reasons. The recent policy interest in mixed communities was prompted by studies of (un)balanced communities, starting with Page's (1993) study. Shortly thereafter, Atkinson's (1995) review, as far as is known, was the first such review of the evidence undertaken in the UK. A start date in the mid-1990s also made it possible to include all reviews produced in the period of ‘evidence-based policy’ promoted under the New Labour Government, in addition pre-dating the prioritisation of mixed-tenure solutions by New Labour's Urban Task Force, and subsequent similar policy statements.

² Again, the two journal articles that were more probably aimed at an academic audience rather than policy makers would be excluded.

³ Interestingly, this is the only example found where a limited review was used to reach a negative conclusion about the evidence for beneficial effects of mixed tenure; such scepticism was rarely exhibited in other works.

⁴ The report continues in similar positive vein stating that mixed income or mixed tenure developments are said to: ‘provide an important platform for addressing social, economic and health inequalities over the longer term’ (p. 26); ‘offer a sustainable model for development’ (p. 26); ‘provide far better opportunities for low income families’ (p. 26); ‘stem the cycle of decline’, ‘upgrade community conditions over time’ (p. 39); and ‘promote healthier communities on declining estates’ (p. 51).