Why Scrum works in new product development: the role of social capital in managing complexity

Abstract

Major changes are currently underway across new product development (NPD) practice, and a number of new NPD management methods and processes are emerging. Managers are faced with an array of possible process models and methods to choose from, including the formal Stage-Gate method as well as multiple emerging variants of Agile. The claimed benefits of Agile methods make it attractive, but its suitability is uncertain. In safety-critical organizations and environments a well-controlled, waterfall-based project model would likely be expected. In an empirical study of an R&D department in a large organization creating and adapting complex air traffic management systems we investigate the use and effects of Scrum, the leading Agile method. Since project coordination is a social phenomenon, we apply social capital and project complexity as theoretical lenses for evaluating the effects of Scrum. We find that Scrum and social capital provide reciprocal benefits, and that the stakeholders found Scrum to be an effective and valuable way of working, mitigating the effects of complexity.

Keywords:

• Agile
• Scrum
• complexity
• project management
• new product development

1. Introduction

A key issue for managers of new product development (NPD) projects is deciding which management process to apply, and one important factor is the degree of uncertainty. Company practice is rather varied (Cooper 2014) and the most common NPD method, the stage-gate process, has been criticised for some time for being too slow (Takeuchi and Nonaka 1986) and too rigid (Smith 2008). If the world is becoming increasingly volatile, uncertain, complex and ambiguous (VUCA, see, e.g. Millar, Groth, and Mahon 2018) then this is a pressing issue, especially since the traditional approach to NPD ‘is based on the assumption of a low uncertainty environment’ (Sommer, Dukovska-Popovska, and Steger-Jensen 2014, 970). Recently, the adoption of ‘Agile’ (a more iterative and incremental way of working compared to more detailed, plan-driven, ‘waterfall’ approaches) has been recommended when the environment is uncertain (Munthe et al. 2014; Olausson and Berggren 2010) or turbulent (Conforto et al. 2016; Cooper and Sommer 2016a), implying that agility is a response to the complexity anticipated within the work. A widely held view is that plan-driven methods suit well-known markets with well-known technology, and that Agile methods suit projects with uncertainty: unknown or changing elements (MacCormack et al. 2012). In this study we investigate a counter-case, in which Agile methods are applied within a relatively stable (but complex and safety-critical) environment.

The adoption of Agile development in NPD is largely due to learning from the software industry (Smith 2008) where it is claimed that more organisations were using Agile development than traditional ‘waterfall’ processes by as early as 2009 (Schwaber 2010). Agile development now represents ‘best practice’ in software development (Cañete-Valdeón 2013; Smith 2008) and has been shown to improve a wide range of project performance metrics including productivity and quality (Tarhan and Yilmaz 2014) and project time and cost (Bianchi, Marzi, and Guerini 2020). Scrum is by far the most widely applied Agile method, with 75% of respondents in the State of Agile report using Scrum or a hybrid that includes Scrum (Digital.ai 2020), and we discuss this shortly. However, Agile development is still considered to be an emerging topic in NPD (Cooper and Sommer 2016a) and so guidance for managers is lacking, including why it works and when it is suitable. Further, in the wider Agile development literature theoretical analysis is lacking (Niederman, Lechler, and Petit 2018).

Tensions may arise when trying to implement an Agile approach, not least the challenge of applying Agile within an organisation that currently applies plan-driven methods such as stage-gate or waterfall. Agile is culturally different as a way of working (Thorgren and Caiman 2019), and integrating Agile with more structured methods is notably difficult. This ‘hybrid’ approach is a key focus of recent research (e.g. Lichtenthaler 2020; Zasa, Patrucco, and Pellizzoni 2021). A more controlled ‘waterfall’ approach has been the widely-applied ‘standard’ model, especially in settings perceived as relatively stable. Implementation of a new system may thus be perceived as risky, and although it offers (indeed, demands) greater cooperation between the NPD organisation and its customers, such a change in working may be a significant shock to established ways of working for many. A corresponding difficulty can be the contractual arrangements between developer and client, since clear criteria on duration, specification and cost are less feasible to establish in advance (e.g. Gajanayaka 2016).

One gap in the literature is exactly ‘how’ Agile delivers the benefits its proponents contend, given the challenges identified. One promising theoretical perspective which could help to understand how and why Agile development is effective is social capital, an umbrella construct representing the nature and effects of social relations (Nahapiet and Ghoshal 1998). In innovation and NPD, high social capital improves the capacity of firms to innovate (Landry, Amara, and Lamari 2002; Laursen, Masciarelli, and Prencipe 2012; Subramaniam and Youndt 2005). Whilst social capital has not been applied a great deal to the study of Agile development it does seem to have clear potential. Previous studies have linked the success of Agile development to improved social factors such as coordination and communication (Dingsoyr et al. 2016; Law and Larusdottir 2015), and this also aligns with recent work investigating the role of social capital with regard to interorganizational risk (e.g. Daghar, Alinaghian, and Turner 2021).

In this paper we also investigate the roles of and relationships between social capital and complexity in an implementation of Scrum (an Agile process framework) in an R&D department. We offer two contributions. First, we identify that the Scrum framework enhances social capital, and that social capital enhances the Scrum framework: they are mutually reinforcing over time. We unpack which dimensions of social capital are involved, and discuss how the benefits are enabled. Second, we identify that the Scrum framework reduces sources of project complexity, and that the enhanced social capital improves responses to complexity. We use these insights to argue that the use of Agile development should be further investigated in complex but low turbulence projects in other domains. We note also that such an NPD approach is highly dependent upon social factors, and that these must be a focus of management attention.

2. Literature review

This paper investigates the broad question of ‘how Agile works’, with a specific focus on the role of social mechanisms in managing the complexity inherent in NPD. This section discusses the history of Agile innovation, introduces Scrum, and then presents project complexity and social capital theory as ways to investigate the case and analyse the resulting data.

2.1. Agile development

The word ‘agile’ has a great many uses and meanings. In the Oxford dictionary agile is an adjective meaning ‘able to move quickly and easily’. The recent interest in Agile development has stemmed from the software industry (Cooper 2014) which was influenced by the Manifesto for Agile Software Development, created in 2001 by a group of software developers seeking an alternative to documentation driven, ‘heavyweight’ software development processes (Beck et al. 2001). In software development a number of terms are used, including agile methodologies (Dikert, Paasivaara, and Lassenius 2016), agile development (Cooper and Sommer 2016b), agile innovation (Rigby, Sutherland, and Takeuchi 2016), agile project management (Conforto and Amaral 2016) or just (capital-A) Agile (Bianchi, Marzi, and Guerini 2020). In this paper we will distinguish between ‘agile’ as a label applied to a wide range of methods and practices, and ‘Agile’ when describing the principles, practices and methods derived from the Agile Manifesto.

Before the software industry adopted the term Agile there were also a variety of agile practices in the manufacturing and supply chain literatures. Agile manufacturing has been studied for many years (Preiss 1994), and was later extended in scope to enterprise agility (Sherehiy, Karwowski, and Layer 2007). There is an emerging body of work on agile business models (Loss and Crave 2011). Agile supply chains remains a significant research topic, addressing ‘the abilities to sense changes and to rapidly and flexibly respond to changes’ (Eckstein et al. 2015, 3029).

In the emerging New Product Development literature Agile development is referred to by its application as ‘a specific style of development commonly used in the software industry’ (MacCormack et al. 2012, 36) and ‘the rapid development system developed by the software industry’ (Cooper 2014, 21). It is also positioned in terms of its foundation: ‘based on the Agile Manifesto crafted by IT industry leaders in 2001’ (Cooper and Sommer 2016a, 513). In terms of its characteristics, Agile development is described as being incremental and iterative (Lei et al. 2017) and flexible (Bianchi, Marzi, and Guerini 2020). Agile accommodates uncertainty (Riis and Pedersen 2003; Williams et al. 2019) and allows multiple stakeholders to collaborate, engage with the work, and shape the outcomes (Chipulu et al. 2019; Ojiako et al. 2015; Ollus et al. 2011).

Historically, Agile development has its roots in the application of plan-do-study-act cycles in the 1930s (Rigby, Sutherland, and Takeuchi 2016) and in the ‘rugby’ approach to product development ‘where a team tries to go the distance as a unit’ (Takeuchi and Nonaka 1986, 137), which was argued to offer much improved flexibility and speed compared to the traditional ‘relay race’ phased approach.

The Agile Manifesto itself is a set of values and principles, and is not a methodology. Indeed the first expressed value is ‘Individuals and interactions over processes and tools’ (Beck et al. 2001), expressing a distinctly process agnostic position. Agile now embodies a number of development methodologies including Scrum, Kanban, Lean, and their hybrids (Rigby, Sutherland, and Takeuchi 2016), but Agile is not itself a method.

2.2. Scrum

Scrum is a structured method for enabling small self-organising teams to choose how best to accomplish their work. Scrum was first introduced at a conference in 1995 (Schwaber 1997), and was later refined and adapted in the Scrum Guide (Schwaber and Sutherland 2013). It was influenced by the Agile Manifesto (Beck et al. 2001). The twelve principles prioritised satisfying the customer and welcoming change, with an emphasis on delivering working software frequently. The manifesto was therefore not originally intended to be applied directly to other domains.

In Scrum, projects are broken down into defined time periods (‘sprints’) of generally between 1 and 4 weeks. The Product Owner defines the highest priority work that is required (the product backlog). During the sprint planning meeting, the team selects from that list what they will do, and who will do it. Team members then decide how they will do the work. Daily stand-up meetings are used to report progress and problems. Review meetings occur at the end of each sprint, where stakeholders are invited to give feedback. This temporal regularity allows participants to base their actions on recent previous experience, and also their expectations of future circumstances.

2.3. Project complexity

Persistent low levels of project performance in both the public and private sectors are an ongoing cause of concern (Maylor and Turner 2017), despite major investment in processes, tools and professional bodies of knowledge (e.g. APM 2019; PMI 2017). An argument within the literature is that the inherent complexity of project management work in reality precludes ‘straightforward’ solutions. Many authors have sought to understand the nature and impact of this complexity (Baccarini 1996; Jaafari 2003; Pich, Loch, and Meyer 2002; Shenhar and Dvir 2007; Turner, Aitken, and Bozarth 2018; Williams 2005; Xia and Lee 2005). Cicmil et al. (2009) differentiate between the complexity in projects (taking a complexity science approach) and the complexity of projects, which relies on a subjective view based on the ‘lived experience’ of the managers involved.

Building on the systematic review of Geraldi, Maylor, and Williams (2011), Maylor, Turner, and Murray-Webster (2013) created the Complexity Assessment Tool (CAT), which identified the major barriers to successful project completion. They identified three categories of complexity:

1. Structural complexity: increases with the number of people involved, financial scale, number of interdependencies within and without, variety of work being performed, pace, breadth of scope, number of specialist disciplines involved, number of locations and time-zones.

2. Socio-political complexity: increases with the divergence of people involved, level of politics or power-play to which the project is subjected, lack of stakeholder/sponsor commitment, degree of resistance to work being undertaken, lack of shared understanding of the project goals, lack of fit with strategic goals, hidden agendas, conflicting priorities of stakeholders.

3. Emergent complexity: increases with novelty of project, lack of technological and commercial maturity, lack of clarity of vision/goals, lack of clear success criteria/benefits, lack of previous experience, failure to disclose information, rising to prominence of previously unidentified stakeholders, any changes imposed on or by the project.’ (Maylor and Turner 2017, 1080)

Maylor and Turner (2017) subsequently built on the Maylor, Turner, and Murray-Webster (2013) complexities and proposed three response mechanisms. They suggested that structural complexities could benefit from a ‘planning and control’ approach (e.g. project management tools and techniques), socio-political complexities via a focus on relationship-building with key stakeholders, and emergent complexities by enabling flexibility (include deviations from normal processes, and novel approaches based on context-specific managerial judgement). This model was used in our data analysis, as it enables a clearer understanding and classification of both the particular complexities faced, and the responses implemented.

The model also implies that a specific kind of complexity could be matched with a corresponding response. Their data, though, showed that managers’ responses were not aligned so neatly, and that each of the nine complexity/response options could in fact be identified. Illustrative examples they give of the nine complexity responses are indicated in Table 1.

Table 1. Complexities and Responses (From Maylor and Turner, 2017, 1086).

		Complexity
		Structural	Socio-political	Emergent
Response	Planning and control	Initiating, planning, monitoring (e.g. applying earned value systems). Using an Integrated Master Schedule.	Develop a communications plan. Establish project board of stakeholders.	Apply risk management and change control processes.
	Relationship development	Prioritize communications with stakeholders. Conduct project outreach activities.	Engage in teambuilding activities. Invest in social capital.	Socialize changes. Increase informal communications.
	Flexibility	Embrace changes from process. Anticipate change. Enable parallel development.	Manage expectations of change. Engage in joint look-ahead planning with major stakeholders.	Use Agile PM approaches. Encourage entrepreneurial PM

2.4. Theoretical framework: social capital (SC)

We adopted a social capital perspective since the effectiveness of Scrum is inherently facilitated by the social interaction that it promotes. Social capital also overlaps with complexity management in projects, which has both technical and social elements (Maylor and Turner 2017). As an illustration, in Table 1 the socio-political column and relationship development row (i.e. five of the nine grid elements) explicitly focus on social dimensions. The socio-political dimension of complexity includes people, power and politics. Social capital represents the resources available through social networks, which can significantly enhance the ability to navigate socio-political challenges in projects. Relationship development is also presented as a key response dimension, and this also has a very clear crossover with social capital, since building relationships would be expected to increase social capital.

It has also been argued that informal social factors are necessary in complex project environments because rational management systems can be ineffective (Hobday and Brady 1998). Agile methods are reported to improve project outcomes in part because they enhance social elements such as coordination and cohesion (Dingsoyr et al. 2016), team spirit, commitment, and communication (Law and Larusdottir 2015) and working closely together (Moe, Dingsøyr, and Dybå 2010). NPD projects rely heavily on social factors for their effective operation, therefore social capital is an appropriate theoretical perspective to investigate the effectiveness of Scrum in NPD. In seeking to understand how the project participants operate within the project, considering both the complexity and social capital lenses should offer clear insight into the operation of Scrum.

SC facilitates innovation because the engendered goodwill supports resource exchange: individuals feel comfortable sharing their knowledge, time and other resources (Adler and Kwon 2002). This is important in Scrum, where regular face-to-face contact is part of the process, and co-location is advocated. Kwon and Adler (2014) describe the ‘propinquity effect’ and argue that ‘solidarity and cooperation are often intensified by face-to-face interaction, and actors who are located closer together in physical space are more likely to interact and form ties’ (Kwon and Adler 2014, 415). This can enhance ‘bonding’ SC between group members who work together, and ‘bridging’ SC to connect to others outside the immediate group (Kwon and Adler 2014). These internal linkages are essential in new product development (Cometto et al. 2016).

Through improved knowledge and resource sharing, SC directly supports the creation of new intellectual capital (Nahapiet and Ghoshal 1998). This relationship has been tested empirically: SC improves knowledge exchange (Subramaniam and Youndt 2005) and knowledge integration (Sargis Roussel and Deltour 2012). It has also been shown to be an enabler of innovation networks (Hansson, Husted, and Vestergaard 2005; Rothschild and Darr 2005). Accordingly, numerous authors have discussed the link between SC and innovation (Kaasa 2009; Laursen, Masciarelli, and Prencipe 2012; Subramaniam and Youndt 2005; Tsai and Ghoshal 1998). Payne et al. (2011) advocate a greater focus on the role of time and suggest that further research is needed to study the role of SC in innovation.

Nahapiet and Ghoshal (1998) propose a model of SC directly addressing its contribution to the creation of new intellectual capital in firms. This is shown in Figure 1, and is recognised as a foundational contribution within the field (e.g. Lee 2009). SC ‘comprises both the network and the assets that may be mobilised through that network’ (Nahapiet and Ghoshal 1998, 243). They also consider that tacit collective knowledge is a critical element of team performance, and that it is generated through social interaction and coactivity. The structural dimension of SC is defined as: ‘the overall pattern of connections between actors – that is, who you reach and how you reach them’ (Nahapiet and Ghoshal 1998, 244). The cognitive dimension is: ‘those resources providing shared representations, interpretations, and systems of meaning among parties’ (244), including shared codes, language, and shared narratives. The relational dimension is defined as: ‘those assets created and leveraged through relationships’ (244), and include trust, norms and sanctions, obligations and expectations, and identity.

Figure 1. Social capital in the creation of intellectual capital (Nahapiet and Ghoshal 1998).

2.5. Literature summary and research question

We identified that the effectiveness of Scrum (as a leading Agile method) is intimately related to social dynamics, but that empirical support for this claim is limited. We selected social capital as a suitable theoretical framework for evaluating this relationship in the context of Agile projects and their particular complexities.

The research question for our empirical study was:

What is the role of social capital in coordinating Scrum projects?

3. Research method

This study sought to examine the relationships between Scrum and social capital in an NPD project. Agile methods remain an emerging research topic in innovation and R&D (Cooper and Sommer 2016a). The rationale for applying Agile methods is often presented in terms of environmental change (Conforto et al. 2016; Cooper 2016), yet a great deal of NPD activity is not exclusively software and is not taking place in a turbulent environment. Since project complexity is one factor driving the choice to adopt Agile methods (MacCormack et al. 2012) we sought a complex NPD case with medium-to-low environmental turbulence but still exhibiting a range of challenges. Of particular interest is the fact that the case is within a safety-critical context. This is unusual, as iterative design and the accommodation of emergent requirements is not usually associated with safety critical systems.

We undertook a case study in a major technical organisation using Scrum. As discussed shortly, this was selected as an appropriate context for our particular study (Eisenhardt 1989), enabling us to understand the dynamics of product development. Our aim was to address the research question by examining the role of social capital in this context, and ‘to understand as fully as possible the phenomena being studied’ (Meredith 1998, 433). We sought to analyse the data in detail with a view to seeking a sense of generality from the findings (Ketokivi and Choi 2014) and ‘reconciliation of the general with the particular’ (236). Qualitative research is proffered as a way of extending our understanding of the mature subject of operations management (Narasimhan 2014), and the case shown here, that of implementing a new way of working, is an example of ‘changing the way operations are managed’ (Childe 2017, 1). A single case can be persuasive (Siggelkow 2007), and the value of the Maylor and Turner (2017) complexity framework in analysing single-case innovation projects has previously been established (Boehme et al. 2021).

Our case study organisation is in the air traffic management industry in the UK, hereafter referred to as AIRCO. Air traffic is a safety critical domain, and air traffic management systems have historically represented record levels of complexity. The AIRCO R&D team are responsible for the development and testing of new air traffic management concepts and tools. This highly regulated environment has a low potential for rapid, radical, changes. The AIRCO R&D team represents an ideal case study for investigating the role of SC in Scrum within a highly complex but low turbulence NPD environment.

We used multiple methods of data collection (e.g. Shibin et al. 2018), using both individual interviews and observations of meetings. In order to address the research question, we carried out 19 individual interviews with R&D personnel. The interview protocol sought examples and explanations within Scrum projects of each of the three dimensions of social capital in Figure 1. Interviews lasted from 48 to 73 min. We also observed 4 Scrum meetings, ranging from 7 min for one daily stand-up to 52 min for a sprint planning meeting.

Our analysis is summarised in Figure 2. We sought to understand the dynamics of product development by focussing on the complexities being worked with, and the ways in which social capital interacts with the Scrum framework.

Figure 2. Analysis framework.

All interviews and meetings were audio recorded. Recordings were transcribed verbatim and then coded using NVivo software. The three dimensions of social capital were used as an initial template for data analysis (King 2012). One new top-level category (feedback) emerged from the data, and lower level codes were identified as we analysed the transcripts. These codes were subsequently revised and modified according to the first and second cycle coding methods described by Saldaña (2013).

4. Results

We first discuss the roles of social capital, and then show how these enable the reduction of complexity within the development work.

4.1. Social Capital dimensions

4.1.1. The structural dimension

The structural dimension of social capital includes the patterns of connections, the presence of ties and the overall configuration of the social network (Nahapiet and Ghoshal 1998). Before Scrum was implemented there was a natural tendency for some of the technical staff to communicate infrequently. Scrum meetings significantly increased the amount of interaction between team members, promoting additional informal knowledge-sharing:

So a lot more talking, a lot more regularly kind of going in and looking at each other’s work, commenting on each other’s work and things like going for a ten minute stand-up and then going for a cup of coffee afterwards. So, you know, there’s the extra time to chat and to kind of exchange knowledge. R7

Beyond the immediate team, adopting Scrum increased the amount of contact between the project team members and their customers and internal stakeholders, primarily via the review meeting held every 2 weeks where participants are invited to attend and make comments. This enabled customer needs to be understood better, and promoted a deeper consideration of interfaces and interactions between system components.

Basically you’re meeting with the customers every two weeks… they come in with the requirement, seeing what we’ve done, and we’re getting feedback on what we’ve done, what works and what doesn’t work for them. R14

More frequent formal contact also led to additional informal contact. This social interaction is closely linked to the relational dimension of social capital. We include it here because the Scrum meetings promote the social interaction that leads to deeper relationships. Respondents commented on social chat taking place within and following the Scrum meetings. This made future informal contact much more likely. An additional contributor to this was the office geography. The physical proximity of the staff, known to support innovation (e.g. Wilson and Doz 2012), also aided communication and knowledge sharing.

In summary, the structural dimension of social capital is enhanced by the regularity of the Scrum meetings. These meetings increase informal and social interaction, which supports knowledge integration. We note that there are strong links to the cognitive and relational aspects, and thinking of the three dimensions as fully separate is not realistic.

4.1.2. The cognitive dimension

The cognitive dimension of social capital includes shared language, codes, narratives and, in this case study, different knowledge domains. The output of the R&D process represents the integrated knowledge of different specialists. The ability to understand how others worked and how the ‘big picture’ came together was identified as being central to successful working. This is challenging given the complexity of the work and the specificity of language. Mathematicians, software developers and systems engineers all need to develop an understanding of the air traffic control domain. Staff develop this understanding over time, although it is a problem for new starters:

Because a lot of people have been in the business for a long time there is just an expectation that you understand what people are saying. A lot of acronyms, a lot of specific terminology is used, and trying to understand that when we're a newbie can be a bit awkward sometimes. R16

Meetings enabled much better understanding than documentation alone, allowing clarity to be obtained swiftly and giving the participants confidence in their understanding. Short-hand stories and narratives are valuable in sharing complex concepts, since a great deal of communication relies on contextual knowledge, and the ability to communicate in a clear and concise way develops over time. User and flight scenarios are used to guide developments and concept testing. As an example:

You can’t tell the aircraft to stand still so the traditional way of that is just flying in a circuit, and that’s known as a holding pattern. So now, I had to start modelling these holding trajectories and there’s a whole load of stuff associated. So the word ‘hold’ now, to me now that means a whole lot of things… I can talk to the guys now and just use, ‘I’m modelling a hold now,’ we don’t have to go into all that detail. R2

In summary, the cognitive dimension of social capital is influenced a great deal by the Scrum framework. Understanding is improved and shared narratives are created through regular communication as a team, discussing and understanding the project as a multi-disciplinary whole.

4.1.3. The relational dimension

The relational dimension of social capital refers to the type of personal relationships. Concepts such as trust and respect are included in this category (Nahapiet and Ghoshal 1998). Our analysis also identified context-specific elements which we now discuss.

‘Obligations and expectations’ was by far the most frequently coded aspect that we identified. Although we cannot state that this means it is the most significant, its prevalence is noteworthy and, we believe, instructive. Regular meetings and commitments drive expectations that people will work hard to achieve their goals. The fundamental obligations and expectations were unchanged by Scrum. Safety has always been a key driver, which means that extremely robust, thoroughly tested systems are required. In our study, even in the results-driven environment encouraged by Scrum, project scope, requirements and timings might flex, but safety standards would not.

Trust within projects was reported to have improved significantly since Scrum was introduced. This was due to a combination of closer interaction, responsiveness, and improved project outcomes. Some previous projects were characterised as ‘hostile’ in terms of individual interactions and difficult meetings, and Scrum was thought to be beneficial in this respect. Specifically, the frequent interactions were reported to improve mutual understanding:

There’s a lot of trust between us so we work very closely together and I know that I can give very - I’ll use the term ‘loose’ requirements - in terms of what functionality we need and I have a great amount of respect and trust in his capabilities and his abilities, for him to implement it however he thinks is best because I know we don’t have to babysit him. R6

The transparency and frequency of commitment and delivery provides a foundation for building trust, which relies on delivered promises.

…because there is a daily stand-up where you report on what you’ve achieved and what you promise to achieve, then you do really become very focused on those goals. R12

Conversely, others identified that small steps and the expectation of regular feedback also ‘removes the need for trust’. However, the prevailing view was that regular feedback leads to better products, which further improves trust. The communication norms and working relationships were reported to be improved by Scrum, reducing tensions and barriers to conversations. In particular, working closely together was reported to have reduced some of the hierarchical and inter-group tensions that existed previously, and increased the recognition of the value of other professions. Collaboration was thought to have improved as a result.

Together, the three elements of social capital enabled staff to request help from people within and outside the organisation to provide access to knowledge, informal support, and context-specific action. Well-developed relationships helped them achieve progress more smoothly than may be expected just through a formal process.

Rather than waiting for the formal process and agreeing this is what we’re going to handover, things like that. It just means you can bypass some of the red tape and bureaucracy. R6

4.1.4. Feedback and reciprocity between social capital and the Scrum framework

One new category that emerged from the data was Feedback. Although this was not one of the dimensions of social capital, it has implications for the model in Figure 1, which we discuss shortly. The Scrum framework integrates a regular feedback cycle of plan, deliver, review. This occurs in short ‘sprint’ cycles, typically 2 weeks, and is critical for two reasons. First, the outcome of an NPD project is, by definition, uncertain. As such, fully specifying the interrelationships between the constituent knowledge domains is not possible in advance. Second, the customer is not typically aware of the specific capability or potential of the enabling technology. Consequently, their vision cannot be specified in precise detail in advance and may evolve as new options are identified. Regular feedback can serve to overcome these gaps.

Before the Scrum process was implemented, a detailed specification would require several months of up-front work to prepare. This planning phase would be followed by a major development process that also took several months. The lack of communication between the teams during this period could lead to significant tension if (or when) expectations were not aligned. The rapid review cycles in the Scrum project allows all stakeholders to develop a shared understanding of the project aims or requirements. The feedback mechanism is critical to this model:

…there’s very much a back-and-forth conversation attitude where it’s allowed R&D to have more flexible requirements or more high-level requirements rather than getting down to very specific engineering requirements. R6

Scrum provides the opportunity to clarify, resolve or explain the requirements more often, in response to the actual development. This flexibility is supported by the social capital within the group, as one respondent candidly acknowledged:

We don’t know exactly what we’re going to do, and we can sort of change it as we go along. R16

In terms of what Scrum delivers, the internal customer perspective is that ‘Essentially, we get far more of what we want by the end of the process’ (R9). Feedback is a critical enabler.

Overall, AIRCO reported that Scrum delivered significant performance benefits. Whereas a previously-observed NPD project performance level was a 50% delivery rate for concept requirements, Scrum improved project delivery substantially:

Then we started this Agile process. We started with 200 concept requirements and delivered 190 concept requirements in seven months. As soon as we did it once, we’d never go back. R8

4.2. Complexity analysis

Here we apply a framework from the complexity literature in order to analyse elements of project complexity in the AIRCO case, and the responses to complexity.

Table 2 shows example complexities and responses identified in the AIRCO case study, using the Maylor and Turner framework (Table 1). Aspects of the framework which are expected to be reliant on social capital include the central column (socio-political complexities) and the middle row (relationship development responses). Interestingly, however, we find that the ‘corners’ are also meaningfully influenced by social capital. Structural complexities are aided by the scheduled meetings (top-left), which are also a key component of structural social capital. These meetings rely on relational social capital to be effective, and we identified in our case study that the social connections formed in and around scheduled meetings is fundamental to Agile working.

Table 2. Examples of complexities and responses in the case.

		Complexity
		Structural	Socio-political	Emergent
Response	Planning and Control	Regular scheduled meetings to coordinate development activity.	Regular meetings force social interaction which can reduce tensions.	The regularity of back-and-forth discussions enables requirements to be evolved in a controlled manner.
	Relationship Development	Schedule and technical issues can be addressed through open dialogue.	Client and team can coordinate and reach agreement to prevent problems growing.	‘Red tape’ can be bypassed to smooth the product development process.
	Flexibility	Resource allocation problems were simplified by making smaller scale requests.	Co-location enables face-to-face discussions more easily, enabling rapid informal issue-reduction.	Uncertainty is accommodated within the process through loosely-defined requirements.

Table 2 indicates that a key objective of the scheduled meetings is to coordinate the development activity, which is important in a complex project. Agile methods improved the capacity of the team to coordinate their efforts as their understanding developed and changed the final deliverable. In the case study this is very closely related to the response to emergent issues (top-right). The regularity of meetings (the structural dimension in the Nahapiet and Ghoshal (1998) model) enables regular communication, but the cognitive and relational dimensions support the behavioural aspects that lead to the co-creation of viable and effective outcomes as perceived by the participants. Short-term goals, frequent stakeholder reviews and the steady demonstration of value simplified some of the project governance issues that had been faced in the past, particularly resource allocation. Finally, flexibility was embedded in the project through loosely defined requirements (bottom-right). Resource allocation was identified as a major barrier to project initiation, and a major driver of behaviour. This had a significant impact but remained one of the hardest aspects to bring into a traditionally managed environment, where a heavily ‘process-based’ way of working remained the norm.

We note that the elements in Table 2 are not discrete complexities with linear responses. Although the Maylor and Turner (2017) analysis technique is powerful in understanding and classifying actors’ actions, the evidence in the case shows that the nine elements should not necessarily be thought of as wholly separate from each other. Especially when viewed through the lens of social capital, SC permeates the Scrum way of working and disentangling specific response mechanisms may not be feasible. This line of argument is similar to that made by Nahapiet and Ghoshal (1998) when introducing their model shown in Figure 1. Although they suggest that the sub-components of SC can be identified, it is the totality of the elements that enable performance benefits. As part of our analysis we considered whether elements of social capital could be linked to the complexity model. For instance, does the structural dimension link to structural complexity or the planning and control response, and does the relational dimension link to socio-political complexity or the relationship development response? This may be a pleasing idea, but such simplification is not supported by the data and a more holistic view of both complexity and social capital is warranted. We now discuss this further.

5 Discussion

This paper examines the role of social capital in the application of the Scrum framework within an R&D department. The literature identifies that Scrum is expected to deliver faster speed to market (Sommer et al. 2015), higher productivity (Bianchi, Marzi, and Guerini 2020; Kautz, Johanson, and Uldahl 2014; Tarhan and Yilmaz 2014) and, more generally, better outcomes (Scott et al. 2016) compared to traditional or plan-driven methods. Within the limits of our qualitative method, our study supports these findings although, as indicated in Figure 2, we were not looking for definitive outcome measurements as part of the case. We now discuss our findings in terms of the significance of the feedback path over time, and the benefits for overlapping knowledge domains.

5.1. Scrum and social capital are mutually reinforcing

In our case study, we identified that the regular planning, daily, and review Scrum meetings significantly increased social interaction. These meetings change the patterns of connections between actors, enhancing the structural dimension of social capital. In the cognitive dimension, face-to-face meetings improve the degree of shared understanding of specialist domain knowledge and project issues. In the relational dimension, additional social interaction enhanced relationship development, which improves trust and commitment to the Scrum framework and to the project. In our observations, the Scrum framework was enhanced by social capital, and social capital enhanced the effectiveness of the Scrum framework. We illustrate this reciprocal arrangement in Figure 3.

Figure 3. Reciprocity between social capital and the Scrum framework.

Building on the feedback cycle shown in Figure 3 with examples from our case study, structural social capital relates to the network configuration and access to people. In the Scrum project the primary network is the team, and the meeting structure significantly enhanced the amount of team communication. Further, since the daily meetings were primarily face-to-face, team members would frequently go for a coffee afterwards. This change in the structural context then helped to develop the relational dimension, which included team member expectations about delivery, but also the level of trust. This trust was not only about ‘delivery on time’, but in delivering what was actually required, sometimes based on loose initial requirements. This trust, as a relational component, relied on and included the cognitive dimension within which ‘there is just an expectation that you understand what people are saying’ (R16). The regular meetings and personal contact enabled this understanding to be developed in a shorter cycle that had been experienced previously, when team meetings were less frequent and people were left alone to get on with their work. The face-to-face communication also made the level of understanding visible to team members through verbal and nonverbal cues, which made developmental feedback much more likely.

The reciprocity between the dimensions of social capital and the Scrum framework has a major effect on the progression of the project. Previous research found that the Scrum framework can improve knowledge sharing (Li, Moe, and Dybå 2010; Moe, Dingsøyr, and Dybå 2010; Paasivaara, Durasiewicz, and Lassenius 2008; Vlaanderen et al. 2011; Sungkur and Ramasawmy 2014), and the interaction of the three dimensions of SC could explain why. We have therefore begun to unpack the feedback path in Figure 1, and our work explains how this pathway helps to build social capital. Not only are the structural, cognitive and relational elements mutually reinforcing, but they support, and are in turn supported by, the Scrum framework. The feedback path in the Nahapiet and Ghoshal (1998) model is underexplored within the social capital literature, and through this work we have examined the mechanisms by which feedback strengthens the three dimensions of social capital.

5.2. Scrum and social capital reinforce each other to reduce complexity

Our findings indicate that the Scrum framework transforms both the sources of, and responses to, project complexity. In particular, the Scrum framework enabled the major process change from a detailed up-front specification to ‘more flexible requirements’ (R6). This meant that the very long (and unreliable) early planning activities were largely avoided. Unpacking the sources and responses to project complexity, we observe a very high level of reciprocity between the elements of the framework discussed in Table 2. The sources of complexity in projects (illustrated in Tables 1 and 2) include structural, socio-political and emergent factors, and the Scrum operation influences each of these.

In terms of responses to complexity, the ‘planning and control’ complexity response was not to increase the detail of the plan and to tighten project control, but rather to enhance communication and feedback through a variety of communication mechanisms (i.e. planning meetings, daily stand-up, sprint reviews and retrospectives). The ‘relationship development’ complexity responses were both structural (regular meetings) and informal, and, as discussed in Section 5.1, relationships served as a key enabler of the ‘flexibility’ complexity response of loosely defined requirements and reliance on professional expertise to solve problems.

5.3. Agile methods reduce complexity in stable environments

Our literature review identified that Agile methods are primarily beneficial for emergent complexity, as expected under conditions of environmental turbulence (Conforto et al. 2016; Cooper and Sommer 2016a). Plan-driven methods are said to be suitable in stable environments (MacCormack et al. 2012), including complex product development (Olausson and Berggren 2010). In contrast, other studies have argued that plan-driven methods are fundamentally unsuitable for complex projects (Hobday and Brady 1998). Our case study organisation developed complex air traffic systems, and the participants reported that, in their view, the move away from a plan-driven (waterfall) method to Scrum, an Agile method, had positive effects on project performance.

We would not characterise the environmental turbulence of AIRCO as high, and so the question of when and why Agile methods should be employed is of special interest in this case. Our respondents discussed the inability of the planning process to predict accurately the detailed requirements of a complex air traffic system, even at the relatively small scale of developing new system features to build and test in a simulated setting. In this case it appears that complexity shares important features with uncertainty, specifically the inability to predict future outcomes (Olausson and Berggren 2010).

As discussed in Section 5.1, an important contribution of Scrum is its ability to improve the cognitive dimension of social capital, which increases the ability of individuals to understand and integrate the contribution of other specialists. As an example of the limits of shared knowledge, mathematicians did not fully understand the nuances of air traffic control procedures or its software, and in isolation from real-world knowledge of these complex constraints could specify mathematically optimal solutions which did not suit controller preferences, air traffic legislation or flight physics. Air traffic controllers also do not have a complete grasp of the opportunities and limits of the software, and so lack the conceptual understanding that would enable them to express their operational issues in a sufficiently precise way that accounts for the actual software design and architecture. This illustrates how the problems and solutions exist in multiple knowledge domains. The scope of knowledge to be integrated dramatically increases the level of difficulty in a project (Grant 1996). In terms of scope, a maths PhD might require 8 or more years of study. Air traffic controllers might require 2 years of training followed by several years of experiential learning. Software developers might also require multiple years of training. Learning about the organisation itself might take 12 months or more to achieve a workable (but still shallow) level. All three knowledge domains have very broad scope, and Figure 4 graphically illustrates the knowledge overlap problem.

Figure 4. Partially overlapping knowledge domains.

In our case study, AIRCO has for some time used cross-training to increase the degree of knowledge overlap. However, the large scope of knowledge within each domain means that a more complete overlap is neither realistic nor desirable. Small gaps between the domains can cause major difficulties for projects. Examples include the colour-coding of on-screen objects (critical to an air traffic controller) and the precise requirements for the process and sequence applied to emergency situations, such as an aircraft conflict avoidance scenario. Revising software to rectify errors (that are common knowledge for a specialist) is time-consuming and expensive. Designing the system within a short build-and-demonstrate cycle reduces this problem. Scrum, through frequent feedback and close social coordination reduces the knowledge overlap problem significantly. AIRCO reported substantially improved project performance having adopted Scrum.

The knowledge overlap problem means that new design features will always bring about unexpected results and interactions. Whilst the air traffic domain is well known overall, the degree to which any single discipline is able to create meaningful innovation in a predictable way is low, due to the broad scope of each specialist discipline and the complexity inherent in their combination. Agile methods reduce the sources of complexity (in part by reducing reliance on initial specification), and also improve the responses to complexity. They are therefore well suited to complex NPD projects.

5.4. Limitations and areas of future research

Although this study is necessarily limited as it is a single case, it offers opportunities for future research. As businesses strive to achieve growth in their market share through introducing new products (Zhao and Chadwick 2014), the capability of managing NPD (Schilke 2014) is often touted as one of the major potential sources of competitive advantage. The exact methods by which this can be achieved, and the underlying factors supporting it, are, though, less well understood. Acur et al. (2010) argue that organisations can leverage their technological competence to influence positively their NPD programmes. Further work is required to understand the relationships between IT and technology capability and the development of NPD capability (Aljumah, Nuseir, and Alam 2021; Pavlou and El Sawy 2006), and this seems particularly relevant in the study of Agile methods since they are so prominent in IT development. As this study was within one organisation only, future work should also investigate further the role of alliances in developing such NPD capability (e.g. Rothaermel and Deeds 2006), especially since alliances are underexplored with regard to Agile implementation. Although we have highlighted the social factors aiding product development in this case, as Agile development becomes far more widespread, it may be possible to gain a greater understanding of the most beneficial implementation. Big data analytics (e.g. Cappa et al. 2021; Johnson, Friend, and Lee 2017; Zhan et al. 2018) might offer the opportunity to determine how best to support NPD under different conditions.

6. Conclusions and implications

This paper examined the role of social capital in Scrum implementation, using the well-established Nahapiet and Ghoshal (1998) framework as the basis for our analysis and considering the complexity of the work based on Maylor and Turner (2017). All three sub-components of the social capital model (structural, cognitive and relational elements) were evident and contributed to Scrum effectiveness, and we identified the significant role of social capital in managing the three forms of complexity evident (structural, socio-political and emergent). We report two key contributions.

First, we identified the importance and inter-relation of each of the dimensions of social capital, and the reciprocal effects between social capital and the implementation of Scrum. We discussed the mechanisms underpinning this reciprocity in the day-to-day activities of the Scrum teams in terms of the feedback path in the Nahapiet and Ghoshal (1998) model. This does not appear to have been identified in the literature previously.

Second, we identify the relationships between the Scrum framework and project complexity, finding that the sources of project complexity are reduced and that responses to project complexity are improved. We use these insights to offer an explanation for why Scrum improves project success, and to argue that Agile methods should be further investigated in complex but low turbulence projects in other domains. Literature to date has proposed that Agile methods are suited to projects with unknown, changing or ‘turbulent’ environments (Cooper and Sommer 2016a; Conforto et al. 2016). In contrast, we suggest that Agile methods are also applicable in stable but complex systems. Environmental changes are taking place in the air traffic management industry, but their R&D environment would not be characterised (by outsiders, at least) as turbulent. The benefit of the Scrum framework was in enabling specialists with disparate knowledge domains to engage effectively and to create collaborative solutions that neither one alone could have predicted. Specifically, the multiple complexities identified by the participants were aided by both the regularity of the Scrum meetings and the corresponding growth of social capital. These factors are intertwined, and we showed that social capital enables complexity to be managed more effectively (including by pre-empting problems) than may be inferred by the nine elements in the Maylor and Turner (2017) model.

The application of Scrum within a complex (and therefore uncertain) but non-turbulent setting was shown to be highly valuable. In complex projects, Agile methods provide the opportunity to develop and refine the specifications in an iterative manner, which allows the emergence of novel and unexpected solutions that combine several knowledge domains. Agile supports, and is in turn supported by, the development of social capital. Managers must be cognizant of this and work to foster the structural, cognitive and relational dimensions of social capital to aid their teams’ success in NPD.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

David Baxter

David Baxter is Associate Professor of Innovation at Southampton Business School. His current research is broadly related to the innovation process, studying the question ‘how do firms innovate’? This has included themes on Agile, knowledge management and knowledge integration, ontology, Product-Service Systems, and customer insight.

Neil Turner

Neil Turner is a Professor at Cranfield School of Management, having joined from a previous career as an engineering manager in a major telecommunications firm. His current research activities involve organizational knowledge and learning in the context of complex projects, and how managerial practices and organizational strategic choices can improve both delivery performance and resilience. His interests lie primarily in how managers deal with the organizational realities they face, including supply chain challenges in both the public and private sectors. He has published extensively in a wide range of academic journals.