An expert-based taxonomy of ERP implementation activities

ABSTRACT

ERP implementation projects are complex and expensive projects. Generally, the complexity is managed by splitting the project into phases. However, splitting the project into phases seems not to enhance the understanding of the underlying processes sufficiently. Therefore, this research aims at enhancing the understanding of these underlying processes through an expert-based taxonomy of implementation activities, independent of time and phasing. This taxonomy has been developed by retrieval of 205 ERP implementation activities from literature, a grouping of these activities by 11 ERP implementation experts, and a comparison with a previous similar study. The method used for grouping was Delphi card sorting that was supported by Websort (https://www.optimalworkshop.com/optimalsort) as a web-based card sorting tool. The proposed taxonomy provides a structured list of 205 identified activities and can serve as a base for further research into ERP implementation projects and can support the planning and resource allocation of ERP projects.

KEYWORDS:

• ERP
• enterprise resource planning
• ERP activities
• ERP complexity
• ERP implementation

Introduction

Implementing enterprise resource planning (ERP) systems is considered to be a complex matter^1–3, because in most cases, it influences large parts of an organization deeply.

Implementing an ERP system is also a very expensive affair. The costs of software, hardware, maintenance, and especially the implementation process itself are high. The implementation may also cause significant risks to the organization.⁴ Therefore, researchers are interested in the implementation process of ERP systems. Research can provide practice with useful insights and tools for improved planning and management of this process. In general, the focus of an ERP implementation project is set by introducing phases or stages in an implementation project, which will provide some overview.⁵ However, phasing such large projects still doesn’t appear to be adequate, since often the projects are too late, are over budget, are not embraced by the users, or don’t realise the expected benefits.⁶

Therefore, this research aims to make planning and resource allocation more effective by aiding the understanding of these underlying processes through the introduction of an expert-based taxonomy for implementation activities, which is expected to be more trustworthy than individual efforts can be. In short, we focus on the “what” rather than the “when” aspect of planning and control. In addition, such an expert-based taxonomy can be used to add a new perspective to research into and concrete planning of ERP implementation projects. For instance, this taxonomy can serve as a starting point for definition and planning of subprojects.

This paper describes the methods and results for the design of an expert-based taxonomy. Basically, a collection of activities was identified by means of a literature review of scientific publications concerning ERP implementations. Then, 11 experts grouped this collection into coherent collections of activities that in turn form the basis for the taxonomy.

First, we will explain why this research is relevant, followed by a discussion on the way in which implementation activities referenced in literature have been extracted and cleaned up for homonyms and synonyms. Then, we will discuss how a grouping method and online supportive tool for this method and a group of experts have been selected. Next, we will discuss the results of the grouping by the experts and the comparison of these results with previous research.⁷ Finally, we will draw conclusions from the results of this research and the comparison with our previous research and propose a first taxonomy. This taxonomy will provide a structured list of identified activities. It can serve as a base for further research into ERP implementation projects and can be of immediate use to support the planning and resource allocation of ERP projects.

Research goal

To manage the complexity of an ERP implementation, it is important to have a thorough understanding of the most important implementation issues. Extensive research has been done on the identification of critical success factors for implementing ERP systems. There is a vast amount of research, which provides an overview of critical success factors.^8–12 These overviews all contain several critical success factors refering to project management topics like: “Teamwork and Composition,” “Education and Training,” “Project Management,” “Definition of Scope and Goals,” and “Champion.” This indicates that proper project management is believed to be an important topic.

To effectively manage an ERP project, it is a custom to decompose a project into meaningful phases, which determine in what order the activities should be undertaken to reach an intermediate or end goal of the project. Research into ERP implementation aspects also strongly focuses on these various phases in an ERP implementation project. Rajagopal⁵ collected several phase models and argued that the model from Kwon and Zmud¹³ seemed appropriate. Somers¹⁴ also agrees with Rajagopal. The phases are very similar and are often only distinguished by the amount of detail. These phasings serve the project manager well by cutting the complex total implementation into less complex parts, which the project manager can plan separately and which are easier to oversee. However, research in this area discusses these phases mainly focused on time and sequence of all project activities and less on the detailed understanding of the underlying activities and purposes themselves. Also, generally, every phase is depicted by a large amount of activities, which in some cases have no or only minimal mutual relationships. For instance, “training of end users” will have only a minimal relationship with the implementation of the “technical infrastructure.” The main purpose of phasing is to cut the project in time into smaller and better plannable and controllable parts.¹⁵

Robey et al.¹⁶ states that “Stage theories allow participants to anticipate future challenges, but they do not provide an understanding of the underlying process.”

Division of ERP implementation in phases, which are individually planned¹⁶, ignores that activities, and hence, the intermediate products and needed resources are often performed through the phases. In the example of user training, in the initiation phase, the number of users in need of training will be determined. In the implementation phase of the system itself, the training will be designed and executed, and lastly, in the phase following the implementation, additional training and support must be offered to users. Consequently, the training dimension is thus relevant through the entire project cycle and does not belong to one phase alone. Still, most effort for training, in most cases, will be needed in the adaptation phase. Figure 1 shows the difference between the phase viewpoint and the viewpoint on various meaningful collections of activities (illustrated by some examples) independent of phases.

Figure 1. Phases of ERP implementation projects versus meaningful process collections.

Our research tries to improve the plannability of these underlying processes in ERP implementation projects by retrieving meaningful collections of coherent and more detailed activities or processes, independent of phases and viewed over the entire project. We submit that knowledge of more detailed activities helps in reducing the complexity of an ERP implementation project. We verified this assumption by conducting a limited survey among ERP implementation experts.⁷ The survey revealed that the experts agree with the assumption and that perhaps eight to 10 meaningful collections of activities might be reasonable to manage from a planning and resource allocation point of view.

To retrieve these collections of coherent activities, we intend to form collections of ERP project activities that have a common purpose in the implementation route. For instance, a common purpose might be to reach a state that users become sufficiently trained for the ERP system or a state that the ERP software is implemented and up and running.

The current research seeks to retrieve the various collections of activities of an ERP implementation, but more concrete: an expert-based taxonomy of ERP implementation activities. The premise behind this is that if it is possible to determine these collections in an unambiguous way, it will enhance the understanding of the implementation processes in projects and therefore in time can reduce the complexity of planning and managing ERP implementation projects in general. For instance, the taxonomy can be used to support project managers in the design of the necessary subprojects for the ERP implementation project. Also, the collections of activities can be used for research and practical application in various domains of ERP implementation topics (e.g., planning, management of implementation, stakeholder analysis, communication with consultants and ERP suppliers). The assumption is that these collections can be retrieved by determining which groups of activities in an ERP implementation are strongly interrelated throughout the complete project. In this context, strong interrelations mean that these activities focus on the same intermediate product within the project. For example, a product could be a sufficiently trained group of users.

The aim of this research is to determine an expert-based taxonomy of all major activities needed to be performed to achieve an implementation of an ERP system.

This taxonomy will demonstrate what activities are closely dependent on each order. If these dependencies are ignored, a high risk of sub-optimization is incurred.

Research approach

An important consideration in designing a new taxonomy is its required level of abstraction. Obviously, the abstraction level is dependent on its purpose. This research aims at a rather high level of abstraction since the taxonomy does not yet exist and also the purpose of this research is to form an initial base for further research. Furthermore, based on our orientating survey¹⁷ and previous research⁷, a taxonomy indicates that 8–20 categories would be an appropriate level of abstraction. On the other hand, it can be expected that it is impossible to determine an unambiguous level of abstraction as various purposes require various levels of abstraction. These various levels of abstraction are for example also the case for the stage theories mentioned before⁵, which are very similar but vary in detail, that is, level of abstraction. Naturally, abstraction levels may vary in detail, but still, the result should be expressed in mutually exclusive terms.

Firstly, these ERP implementation activities had to be identified and listed to initialize the taxonomy building process. Since we did not encounter a comprehensive collection of these activities in literature suited for our purpose, we extracted a collection of from the literature concerning the implementation of ERP systems ourselves. We assume that the activities that appear in literature must be of significance in ERP implementation projects. Of course, this may not guarantee a complete list, but at least the items in the list should be known to scientists in the field and should have been validated properly.

Secondly, the retrieved activities had to be scanned manually for possible duplicates, as activities can appear in the literature as synonyms and homonyms and can have different wording.

Thirdly, these refined activities had to be grouped into meaningful collections. In this research, experts were chosen as an information source for these collections. The experts used a card sorting method for this purpose.

In the next sections, the three steps are further elaborated.

Collection of ERP implementation activities

Firstly, an expert researcher selected appropriate keywords for the literature search. These keywords were checked by two other expert researchers. After that, the first researcher performed the search and enhanced the keywords further, if the results of that literature search indicated even more relevant other keywords. In total, 3860 search results from 15 scientific literature databases were retrieved and scanned for relevance (Econ papers, Springerlink, Science direct, Emerald, Blackwell, DOAJ, Ebsco host, Wiley, JSTOR, Swetswise, ACM Digital library, IEEE Computer society, Google scholar, Darenet, Citeseer). The databases where selected for their expected relevant ERP research content. Of course, a large number of these 3860 papers overlapped in the search results of these databases. The literature and extracted activities from the previous research⁷ were also included in the results.

The full text of literature that seemed relevant after a first check was scanned in detail for relevancy for this research. Finally, 42 papers were selected that include relevant information on ERP implementation activities, that is, the paper did describe and mention concrete ERP implementation activities.

From the results of this literature search, it can be concluded that ERP activities are described in these types of papers:

1. Papers that used accepted ERP concepts from previous research for designing new theoretical models, for instance, critical success factors, stage theories and supplier implementation methods

2. Papers that portrayed and analysed real implementation cases

3. Papers that combined theoretical models with empirical data

4. Papers that contained ERP implementation activities but where the origin of the list of activities was not always stated

5. Papers that showed activities for implementation of enterprise systems

From these 42 relevant papers, a list of 484 ERP implementation activities was retrieved. A list of these 42 relevant papers and the list of the 484 ERP implementation activities can be obtained from the corresponding author.

Refining the collection of ERP implementation activities

In the extracted activities from scientific papers, of course synonyms and homonyms occurred. For example, one paper would mention “training of users,” whereas other papers would mention “user training.” Also, homonyms exist; for example, “redesign” might stand for “business process redesign” or for “infrastructure redesign.” To deduplicate the list of activities from synonyms and homonyms, five voluntary ERP field experts were selected and received formal criteria, instructions, and rules on how to detect synonyms and homonyms.

The five field experts consisted of two researchers and three field experts working with ERP systems and implementations routinely. The detailed explanation of the used procedure can be obtained from the corresponding author.

The field experts received the total list of 484 activities and received the instructions to indicate which activities should be deleted because of being synonymous with another activity or activities.

One of the researchers processed the results of the five participants as an extra check-up. The results of the five participants led to a condensed list of 232 activities. Finally, the same researcher checked in detail this list of 232 activities and removed a further 27 clearly incorrect activities, which led to a net list of 205 activities suitable for grouping. Also, these activities were renamed in a consistent formulation, that is, verb + noun(s); for instance, “definition of scope” would become “define scope.” Table 1 shows an extract from the collection of these 205 activities. The full list can be obtained from the corresponding author.

Table 1. Extract from the collection of 205 ERP activities.

.
..
…
9.	Test system
10.	Carry out cultural and structural changes
11.	Define project objectives
12.	Collect vendor consultant information
13.	Find out need analysis
14.	Evaluate vendor and consultant alternatives
15.	Evaluate IT infrastructure
16.	Perform feasibility study
17.	Finalize contracts
18.	Define project scope
…
..
.

Grouping of the collection of ERP implementation activities

Information about the dependency between the ERP implementation activities can be retrieved from three sources: literature, documents from ERP projects, and persons who have sufficient knowledge and experience of ERP implementation projects.

As in the researched literature, only documents in ERP projects and persons with sufficient knowledge are in this case appropriate sources of information.

Documentation can be handled by detecting activities (as collected from literature) in the actual project documentation of completed ERP implementation projects and analyzing the relationships between these activities, for instance, by taking the viewpoint of subprojects. The advantage of using such documents is transparency of the process and therefore the ability to reproduce results. However, disadvantages are: a large number of projects needed, the necessity of constructing and verifying a proper method for analyzing the project documentation, the difficulty in access to these projects, and the large time expenditure and duration needed to perform the research itself.

In fields where knowledge in the decision-making processes is rare and incomplete, expert consultation is often used.¹⁸ However, it can be expected that no single expert exists with every necessary knowledge needed to form the collections of the retrieved ERP implementation activities. Even if this expert would exist, the only way to detect this expert is by testing the expert against the knowledge that is yet to be retrieved, which of course is a paradox. On the other hand, a sufficient number of experts will have overlapping knowledge¹⁹ and therefore can provide the necessary input if an appropriate consultation method is used. Gustafsson and Ollila²⁰ showed the characteristics of the consultation methods for groups of experts in light of the communication media theory. These consultation methods are: questionnaire, interview, workshop, and Delphi. They also designed an application typology for these consultation methods. In the case of topics, which relate to multiple and sometimes ambiguous disciplines, such as in our case, they recommend as consultation methods to use Delphi or workshops. They also conclude that if the topic is uncertain, Delphi would be the preferred method.

In contrast to analyzing project documents, in our research, the use of a group of experts has the advantages of easier access, of a smaller duration of the consultation itself and less effort. By proper selection of experts and use of an adequate workshop or Delphi consultation method, the quality of the results will be sufficient to form a first taxonomy. Therefore, in this research, experts were invited to group the ERP implementation activities into meaningful collections which form the taxonomy. A second reason for choosing experts to form the collections is that we tested in our previous study⁷ a method for detecting activities that normally occur in an ERP implementation project and a method for grouping these activities into collections of ERP project activities. This study shows that both the selection of activities as well as the used method of grouping is appropriate and relevant results can be obtained.

To be able to group coherent collections of these activities, it was necessary to select an approved consultation method by which experts could model the groups of activities. As suggested by Gustafsson and Ollila²⁰, a Delphi method would be appropriate. From the outcomes of the previous research⁷, it was concluded that the metaplan technique, which is a form of Delphi, is a suitable technique for grouping ERP activities but unfortunately also has some practical limitations. The number of 205 activities in this present research and the fact that experts are hard to persuade to participate in a group session like metaplan indicated that the metaplan technique would cause practical limitations. The grouping of these activities is dependent on time and place, and it can be expected that the metaplan session would be very hard to organize and have an unacceptable long duration for the participating experts, let alone the difficulty of overseeing such a large number of activities. It is expected that, if experts could perform the grouping whenever they want and wherever they want, the willingness to participate would be higher. As shown by Howard [include reference] as well, support of this process by a Group Decision Support System (GDSS), which can support grouping in different locations and/or at different times, leads to the same quality of results.²¹ Also in our research, the Delphi aspect should be integrated into the GDDS.

The number of 205 established activities implies the need for a formal technique. For this type of grouping, a card sorting technique, which will be described in detail further on, is appropriate, as card sorting is a simple method for establishing a taxonomy that cannot be inferred from objective sources of information. Card sorting has proven its usefulness in many concept mapping studies.²² If one human individual does a card sorting, bias and limited knowledge will influence the result. Judgment by several individuals and group interaction will improve the quality of the results. Unfortunately, members of freely interactive groups are often dissatisfied with group interaction.²¹ According to Howard, a Nominal Group Technique (NGT) improves the output and satisfaction of the group members.²¹ Therefore, in our previous research⁷, the metaplan technique for the grouping was chosen. The metaplan technique uses card sorting and can be considered a Nominal Group Technique (NGT). However, card sorting as a regular technique does not contain the Delphi aspect. Fortunately, Paul²³ combined the Delphi method with the card sorting method. She showed that the combination of the Delphi method with the card sorting method results in better grouping quality when compared to regular card sorting. Paul also showed that the experts needed less effort, which in this research is very relevant as the number of activities to be grouped is large in comparison to regular card sorting. Therefore, this research adopted Paul’s Delphi card sorting method and used Websort as a supporting GDDS.

In the next sections, the selection of the experts, the Delphi card sorting technique, and the selected tool for the card sorting are discussed.

Experts

To perform the Delphi card sorting, about 8–10 expert participants are necessary.²³ The experts had to meet the following requirements:

• Minimal five years of experience as a manager in ERP implementation projects

• Knowledge and experience of the complete ERP implementation issues and not only on a special issue

• Sufficient English knowledge to understand the descriptions of the activities from the activity collection and to be able to name categories

• A professional reflection level indicated by at least a completed Bachelor degree

• Experience in ERP implementations in the Netherlands, because of the risk that Paul recognized, that too heterogeneous a group could lead to an unstable model²³

• Knowledge of and experience in implementation of large ERP applications like SAP, Oracle, BAAN of Peoplesoft, whereby several modules were implemented

• Originating from different organizations

We approached one expert individually, and the other experts have been selected and approached using LinkedIn (http://www.linkedin.com). This provided access to a sufficient number appropriate experts.

Ultimately, 11 experts that met the requirements agreed to participate in the Delphi card sorting study.²³ This number satisfies the criterion from Paul²³ that 8–10 experts are needed to perform a Delphi card sorting study.

This group of 11 experts consisted of six managers and five consultants. In this group, three experts worked for Dutch ministries and eight for business organisations. Six experts had 5 to 10 years’ experience in ERP implementation projects, and five had 10 or more years of experience. All experts worked in the Netherlands.

Delphi card sorting

Recently, Paul²³ proposed a new open card sorting method. She named this variation “modified Delphi card sorting method.” This method deviates from the closed card sorting method in that the participants, except for the first participant, do not start with a pile of unsorted cards but receive already sorted and named grouped lists from their predecessor. The first participant performs the initial sorting of cards and provides each category with its initial name. The next participant and subsequent participants will continue improving this initial sorting and categorization of their predecessor as they please. This means that they can move cards from one pile to another, start new piles, delete piles, and make changes to the naming of piles. The idea behind this adapted method of card sorting is that toward the end, fewer changes will be applied by the participants, and they only need to reflect on the difficult issues. The model toward the end will stabilize. The final result of the sorting will be the result of the last participant. There is no further analysis necessary. Furthermore, it is not necessary to “subjectively” decide on what the final name of each category should be, as the participants have changed the names if they thought they were inappropriate. In her research, Paul ²³ showed that the quality of the final model by Delphi card sorting was better than the quality of a model through the regular card sorting method.

Naturally, this new method also bears some disadvantages:

• The lead time of the sorting will be longer compared to regular card sorting, because participants need to sort sequentially.

• Most likely, the sequence in which the participants participate in the sorting procedure may influence the outcome.

• If the last, or one of the last, participants has a complete deviating opinion on how the cards should be sorted, then the result of that final sorting might not be optimal or even useless. However, this can be easily checked and the results from the previous rounds can be used instead.

In this research, we tried to cope with these last two disadvantages by analyzing the results from every participant and comparing them to each other.

Tool

In the field of ERP implementations, it is difficult to encourage experts to participate in scientific research. Therefore, important requirements for the method and supporting tools are the minimization of time and effort to be spent by experts and also independence of place to perform the sorting. Taking these requirements into account, an Internet-based card sorting tool is potentially a suitable solution. Hence, Websort (http://www.websort.net) has been selected, which supports card sorting and specifically Delphi card sorting. The tool is easily accessible, and the functionality is user-friendly. After using Websort, the experts reported no problems using it. Also, the results of the sorting (per expert) could be easily exported and further processed in a spreadsheet.

Grouping

The 11 independent experts used Websort to group the 205 ERP activities. The order in which the experts have executed the grouping, except for the first one, was random. The first expert was individually approached by one of the authors to be able to motivate and instruct this expert, as this expert had to perform the initial sorting of the 205 activities that is, of course, a time-consuming and intense task.

The first expert received the complete set of unsorted activities and was asked to group this set into relevant groups (of activities) and label those groups with an appropriate name. The second expert received the anonymous result of the first expert and was invited to improve the grouping regarding relocation of activities between groups and changing group names and/or creating new groups. The 3rd to the 11th expert received the anonymous outcome of their predecessor and the same instructions as the 2nd expert. Websort provided the information required to fully trace all individual changes made by the experts. Figures 2 and 3 show a graphical representation of the changes between rounds.

Figure 2. Movements of items in categories per round.

Results

Apart from the first expert, none of the experts knew in which turn he was. Remarkably, the results clearly indicate a quick stabilisation of the sorting between rounds 1 through 5 and even quicker between 6 to 11, as shown in Figure 2. Figure 3 shows the same stabilization for the number of categories. The stabilization seems to confirm the claim of the Delphi card sorting method that each round will improve the model. Although the experts did not know in when they had their turn, each following expert needed to improve less on the results of his predecessor.

Figure 3. Net changes in number of categories in respect to previous round.

Figure 2 shows a quick stabilization of the sorting in rounds 1 through 5. The relocation of activities between the groups and the changes to the number and the naming of the groups decreases. However, the expert in round 6 made a considerable change to the model. Round 7 to 11 again show increased stability. The experts 7 to 11 apparently accepted the significant changes made by expert 6 and only made improvements upon these changes. The graph in Figure 2 might lead to the conclusion that the sixth expert has made drastic changes to the model of his predecessors, but detailed analysis of his changes shows that the 6th expert has just refined some groups. This expert kept nine groups the same, split two and combined two into one. Apparently, this expert changed the level of abstraction. This change of abstraction also can explain why in turn 7 until 11 show increased stability again. Apparently, the experts in these rounds accepted this more detailed level of abstraction from the sixth expert and made only small changes to it.

All things considered, the card sorting procedure we performed, as described by Paul²³, went perfectly well. Therefore, we feel encouraged to accept the outcome.

Comparing results with a previous grouping

In our exploratory research⁷ also, activities within an ERP implementation project were extracted from scientific papers and grouped using the metaplan method. In this exploratory research, the extraction and sorting of activities was performed by three scientists (one also with practical ERP project manager experience), which resulted in a different abstraction level and grouping than what we have seen in this research. However, the activities and groups retrieved in that former study are still largely similar to the activities and groups in this study. This finding further enhances confidence in our card sorting results. Therefore, the activities from the first research⁷ were matched with the activities in this research. Two authors conducted this matching independently of each other and checked their matching with each other afterwards. Next, it was determined to what extent activities ended up in the same collection of activities, regardless of the name of the group.

Figure 4 shows the results of the comparison of the groups with the exploratory research⁷ and the outcome of the 11th round. The results from the 5th round were not used, because the model of the 11th round is a more detailed model than the 5th round and not significantly different from the 5th round. Figure 4 shows on the y-axis the groups from the exploratory research. The x-axis shows the groups from this. The cells in the matrix represent the number of the 192 activities that were assigned to a group from the exploratory research and to a group from this research. For example, from the 17 activities from the metaplan group “System configuration,” 15 activities were classified in the “Configuration” group in round 11, one in the “Technical implementation” group, and one in the “Blue printing” group.

Figure 4. Comparison sorting round 11 with metaplan sorting.

To further explore the overall similarity of the grouping between the two sorts, the matrix has been sorted in such a way that the cells with the largest numbers of matching activities were moved to the diagonal sorted in decreasing order, that is, decreasing from the top left to the bottom right in the matrix. After that, borders were drawn around adjacent groups of cells on the diagonal that contained the most activities to form groups of groups between the two groupings that are closely related to each other, or in other words, they are very similar. As shown in the matrix, 12 groups were formed containing 74% of all activities, which indicates that there is a major similarity between the grouping of our exploratory research and this research. Table 2 shows a list of these similar groups of groups with a proposed name by the authors, taking into account the nomenclature in the two studies.

Table 2. Proposed taxonomy of ERP implementation activities.

A	Software and vendor selection
B	Project initialization
C	System configuration
D	Implementation and support
E	Testing
F	Training
G	Change management
H	Technical implementation
I	Project communication
J	Business Process Design
K	Data conversion
L	Blueprinting

Table 2 represents the expert-based taxonomy of activities that represents the implementation of ERP systems, regardless of any formal phasing, to increase the understanding and planning of ERP implementations.

Discussion

Every expert, except the first one, was influenced by his predecessor although unaware who sorted before him and the round in which he was participating. On the one hand, this is the intention of Delphi card sorting and improves the quality of the model, while on the other hand, the influence of the predecessor narrows down the idea for a solution for the successor expert.

Although in our research, experts designed the taxonomy, we could not specify to the experts what level of abstraction was needed other than by indicating in the expert’s instructions the required level by examples of intermediate and end products of an ERP implementation. In a previous study⁷, we already discovered the potential issue of shifts in detail in activities and the disruption it may cause to subsequent sortings. In this case, this shift of detail didn’t produce any problems, that is, subsequent card sorters simply accepted the level of detail they were presented with, without questioning or interruption. Therefore, we were also interested in the levels of abstraction that experts would find relevant, that is, according to which level of abstraction the experts would group the activities. Besides that, we were interested in whether experts would reach consensus about the level of abstraction or whether this would be an issue in the grouping process.

The purpose of this research was to find the main stable cores of activities by which the final groups derive their legitimacy. If different experts allocated some activities to different groups, that is, the experts did not come to an agreement, this would not influence the outcome. Of course, this should be, and in our case, this was only the case for a small percentage of the total number of activities. Arguably, this small number of activities can be considered as borderline activities, which depend on the view of the expert, formed by his/her experience, should be sorted in a specific category. Also, the abstraction level that an expert considered may have an influence on the allocation. Although the experts made no remarks, whether the activities were clear for them, it is possible that some experts interpreted a specific activity different from other experts. However, the general convergence in the results shows that these effect were minor and that the results can be considered sufficiently reliable.

Conclusions and further research

Our consulted experts agree upon coherent groups of activities that occur in ERP implementation projects.

Though very similar, the results of our previous exploratory research and the results of this research from round 5 and 11 provided different groups of activities. Analysis of the data shows that participant number 6 changed the level of abstraction into a more detailed one. Also, the comparison of the grouping from our previous research with the results of round 11 shows that there is a great similarity, dependent on the level of abstraction. Therefore, we assume that the adopted level of abstraction by a participating expert is an important factor. We assume that there is no single universal correct grouping of ERP implementation activities. Nevertheless, the combined high-level grouping from the exploratory research and this research is a first by experts verified grouping of activities in highly related activities within an ERP implementation project independent of phases.

As a result of the expert grouping and the comparison with the previous research, Table 2 can be considered as a first proposal for an expert-based taxonomy of ERP implementation activities at a fairly high level of abstraction. This taxonomy can serve as a base for further research into ERP implementation projects and can support the management of these projects (see below).

The rapid stabilization in round 1 to 5 and 6 to 11 seems to confirm the claim of the Modified Delphi card sorting method that each round will improve the model. Although the experts did not know when they had their turn, each following expert needed to improve less on the results of his predecessor. As shown in Figure 2 in round 5 and 11 groupings exist with stable cores as only a few percent of the activities still move during round 3 to 5 and even less during round 6 to 11.

Although some experts commented that they missed a particular detailed activity in the set of 205 activities, they did not indicate that they, therefore, were unable to form a needed group. Apparently, all expected groups could be formed using the available 205 ERP activities.

The results, participation, and comments from the experts show that Delphi card sorting appears to be a practical method for retrieving this type of information and Websort is a suitable supportive tool. The willingness of the invited experts to participate in this online Delphi card sorting was high. All experts who were willing to participate also finished the sorting. This willing participation might confirm the assumption that an appropriate method and tool would stimulate the participation of experts and the actual sorting. The possibility for the experts to sort whenever and wherever they wanted, and the user friendliness of the tool might be important factors. Websort also provided functionality to an expert to comment on his sorting and the tool itself. In these comments, none of the experts complained about the method or used tool for the sorting. Paul²³ also observed that performing Delphi card sorting required less effort from the experts than regular card sorting. In this research, we had no opportunity to validate this observation, but it might have been a factor that influenced the willingness of the experts to finish the sorting.

The resulting taxonomy in this research is a taxonomy based on extensive literature search, structured by the careful application of expert judgment. We feel the taxonomy is therefore of value, specifically for usage in practice. Nevertheless, some further validation of its usefulness in practice is a logical next step.

The resulting taxonomy in this research is a taxonomy solely based on expert judgment. This taxonomy should, therefore, be confirmed and enhanced by the use of empirical data from ERP implementation projects.

The taxonomy can be used in practice to support project managers in the design and planning of the necessary subprojects and the allocation of relevant resources to them for the ERP implementation project.

The collections of activities can be used for research and practical application in various domains of ERP implementation topics (e.g., planning, management of implementation, stakeholder analysis, communication with consultants and ERP suppliers). Project managers can use the list of 205 activities as a reminder when designing the detailed project plan for the ERP implementation project. Research can further complement and elaborate this list. For instance, research can examine every activity in further detail and provide practice with guidelines for estimation of time, resources, and deliverables.

Research can use the taxonomy as a framework independent from the variety in stage models to confine research into well-defined topics within an ERP implementation project. Even when also using the corresponding list of 205 activities to precisely delineate the research area for ERP implementation projects.