Development of a twenty-first century skills scale for agri varsities

Abstract

A new twenty-first century skills framework and a 19-item scale was developed to bridge the gap between science and practice. A self-reporting 19-item scale was developed to measure twenty-first century skills of faculty and non-faculty staff of agri varsities. A total of 49 items were generated using deduction approach for four skills: communication and collaboration, leadership and responsibility, problem-solving and decision-making, and creativity and innovation. Two content validity techniques were used that reduced the number of items to 29. Exploratory factor analysis was conducted in Mplus and items with poor loadings and cross-loadings were removed in each round. After five rounds of EFA, a promising 19-item scale was identified that fit the data well based on multiple model fit indices. Confirmatory factor analysis further confirmed the construct validity of the items and a four construct scale was identified where creativity was the only skill that failed to load. The reliability of the scale was measured using Cronbach’s alpha which was above 0.70 for all the factors. Convergent and discriminant validity was analyzed using AVE, shared variance, and factor loadings. The implication and limitations of the scale were discussed.

Keywords:

• twenty-first century skills
• twenty-first century skills framework
• twenty-first century skills scale
• twenty-first century competencies
• twenty-first century skills

PUBLIC INTEREST STATEMENT

There has been a lot of debate on the changing skills demand in recent years. Businesses need new skills (known as twenty-first century skills) to get today’s job done from their employees. Unfortunately, a lot of work on the twenty-first century skills was done by the education institutions and the needs of businesses were somewhat ignored. This created a gap between what education institutions are preparing their students for and what businesses need from their workforce. This widening gap leaves several graduates unemployed because they don’t possess the right skills. This gap was addressed by the current paper by developing a new twenty-first century skills scale for agri varsities so they can hire and train the right people for the right jobs by making sense of the twenty-first century skills. The paper identified, defined, grouped, and ranked twenty-first century skills for businesses empirically.

1. Introduction

The twenty-first century has reshaped the world in unimaginable ways. Computers and IT have revolutionized the skills needed by a twenty-first century worker (Autor, Levy, & Murnane, 2003; Burrus, Jackson, Xi, & Steinberg, 2013; Levy & Murnane, 2005; Neubert, Mainert, Kretzschmar, & Greiff, 2015). The decline in the routine tasks (that can be accompanied by a machine or a computer, e.g., monitoring level of fluid in the furnace after regular intervals) and a rise in abstract tasks took place in the past few decades. A nonroutine (abstract) task is one that has no clear rules defined so it cannot be performed by a machine because it’s hard to create standard rules for such tasks (Autor et al., 2003). Employees need twenty-first century skills (essential skills needed in the twenty-first century to be an effective employee) to perform abstract tasks of the twenty-first century and a framework is used to identify and assess these twenty-first century skills. The frameworks for twenty-first century skills (e.g. P21, enGauge, DeSeCo, and ATC21S) are used by education systems to prepare students for the new era (Burrus et al., 2013; Finegold & Notabartolo, 2010; Jia, Oh, Sibuma, LaBanca, & Lorentson, 2016).

The twenty-first century frameworks have multiple demerits such as lack of empirical evidence, lack of standard definition and construct definitions, and lack of integration with I-O psychology (Burrus et al., 2013; Dass, 2014; Finegold & Notabartolo, 2010; Jia et al., 2016; Neubert et al., 2015; Su, Golubovich, & Robbins, 2015). There is a need to bridge the science-practice gap by establishing empirical evidence for the usefulness and effectiveness of these skills and frameworks. Human resource practitioners need a framework with skills that are ranked, defined, and are measurable so they can hire and train right people for the right jobs. The current study addressed these challenges by developing a new scale with clearly defined and ranked skills so businesses and HR practitioners can recruit and train right people.

2. Overview of the twenty-first century skills frameworks

We reviewed existing literature to identify existing frameworks of twenty-first century skills.Table 1 demonstrates the list of all the skills identified from the review of the eight frameworks.

Table 1. List of twenty-first century skills

Skills	SCANS	DeSeCo	enGauge	P21	ATC21S	NML	HP	O*NET
Reading	✓
Writing	✓
Arithmetic and Mathematics	✓
Listening	✓
Speaking	✓
Creative thinking	✓
Decision making	✓					✓	✓
Problem solving	✓					✓	✓	✓
Learning to learn	✓
Responsibility	✓		✓	✓
Self-esteem	✓
Sociability	✓
Integrity	✓
Interpersonal	✓		✓
Resource management	✓
Information processing	✓
Systems thinking	✓
Technology savvy	✓
Autonomy		✓
Interactive use of tools		✓	✓
Being part of groups		✓
Basic literacy			✓
Scientific literacy			✓
Technological literacy			✓
Economic literacy			✓
Information literacy			✓
Visual literacy			✓
Multicultural literacy			✓
Global awareness			✓	✓
Creativity			✓
Self-direction			✓	✓
Managing complexity			✓
Adaptability			✓	✓		✓
Risk taking			✓
Curiosity			✓
Higher-order thinking			✓
Collaboration			✓		✓		✓
Interactive communication			✓		✓		✓	✓
Social responsibility			✓			✓
Planning for results			✓
High-quality product production			✓	✓
Civic literacy				✓
Environmental literacy				✓
Health literacy				✓
Financial, business, economic, and entrepreneurial literacy				✓			✓
Flexibility				✓		✓	✓
Social skills				✓
Cross-cultural skills				✓
Initiative skills				✓
Leadership				✓
Accountability				✓
Productivity				✓			✓
Critical thinking and problem solving				✓		✓	✓
Creativity and innovation				✓	✓	✓	✓
Communication and collaboration				✓	✓	✓	✓
Media literacy				✓		✓	✓
Information literacy				✓	✓	✓	✓
ICT literacy				✓	✓	✓	✓
Critical thinking, problem solving, and decision making					✓			✓
Learning to learn and metacognition					✓	✓
Citizenship					✓
Life and career					✓
Personal and social responsibility					✓	✓
Teamwork						✓		✓
Research and inquiry						✓	✓
Leadership and responsibility							✓
Initiative and self-direction							✓
Digital citizenship						✓	✓
Adaptive learning							✓
Fluid intelligence								✓
Innovation								✓

The review of the existing frameworks reveals an interesting trend in the shift of skills from basic literacy skills to more complex skills (see Table 1). The skills identified by The Secretary’s Commission on Achieving Necessary Skills (SCANS) (Scans, 1991; Skills, 1992) were more focused on basic reading, writing, speaking, and mathematics. Several skills relevant to twenty-first century were identified such as creative thinking, problem-solving, and interpersonal. It was developed in 1991 and it did a decent job of identifying relevant skills. In fact, most of these skills are still relevant today and were used in several frameworks that were developed after 2000.

The Definition and Selection of Key Competencies (DeSeCo) (Salganik, Rychen, Moser, & Konstant, 1999) identified similar skills but with different names. For example, technology savvy skill was replaced by interactive use of tools and interpersonal skill was replaced by being part of groups. The skills in both the frameworks were somewhat common and overlapping. This confirms the criticism by Burrus et al. (2013) that the skills in the frameworks are overlapping. The DeSeCo wasn’t entirely based on SCANS rather it identified several new skills and most of the old skills were renamed and redefined.

The enGauge (Burkhardt et al., 2003) identified and defined a list of both literacy and workplace skills. Several literacy skills were identified including basic literacy, scientific literacy, technology literacy, and others. In terms of literacy skills, enGauge was more comprehensive than SCANS. It had all the literacy skills from SCANS and added several others to prepare students for life and work. It also introduced several workplace skills (e.g., creativity, collaboration, and self-direction) most of which were similar to SCANS and DeSeCo. The enGauge, however, was more comprehensive and updated as compared to previous frameworks. The skills were more of general skills and weren’t properly defined constructs and this is what creates an integration into practice issue. Skills like planning for results, product production, and managing complexity aren’t well-known constructs.

The Partnership for twenty-first Century Skills (P21) (Greenhill, 2009; P21, 2002) is one of the most famous frameworks that identified several crucial skills including a few skills from SCANS and enGauge. Several literacy and workplace skills were included in the framework that weren’t part of any previous framework such as leadership, critical thinking, and communication and collaboration. The skills identified by P21 were well-defined constructs in the management and I-O psychology literature like critical thinking, leadership, productivity, and innovation. This was a much appreciated approach that made it possible for practitioners to relate to these skills.

The Assessment & Teaching of twenty-first Century Skills (ATC21S) (Binkley et al., 2012) identified limited but essential skills such as, problem-solving, critical thinking, decision-making, collaboration, and innovation. The skills, however, were regrouped and categorized differently than P21, enGauge, DeSeCo, and SCANS. Problem-solving, decision-making, and critical thinking were grouped together as a single skill. Literacy skills were similar to the ones identified in P21, however, only two literacy skills were identified: ICT and information. The long list of literacy skills was shortened by ATC21S.

The New Millennium Learners (NML) (Ananiadou & Claro, 2009) was primarily based on P21 as most of the skills were the ones that were already identified by P21; however, three new skills were introduced: Teamwork, research and inquiry, and digital citizenship. The focus of the NML was on students; however, it also identified workplace skills that were considered critical for success.

The Hewlett Packard (HP) framework (Finegold & Notabartolo, 2010) regrouped skills from the existing frameworks primarily from P21. Interestingly, both HP and NML identified somewhat similar skills from P21 but HP identified more skills as compared to NML. Only three new skills were identified while all the other skills were the ones that had been identified in one of the previous frameworks.

The Occupational Information Network (O*NET) framework (Burrus et al., 2013) used data to identify five skills. Fluid intelligence was the only new skill while other four skills were already identified by previous frameworks. It was a data-driven framework so its primary purpose was to identify skills that employers were looking for instead of identifying any new skills.

The review of these frameworks reveals that the frameworks are overlapping. Even when new skills are identified, they aren’t new rather renamed skills. When a new skill is identified, its definition is more important than its name. Inclusion of new skill with a new name doesn’t mean that it is a “new” skill rather it means that the skill has been renamed. This clearly reveals that there is an overlap of skills and there is a lack of consensus on definition (Burrus et al., 2013; Jang, 2016; Su et al., 2015). Second, three frameworks did the primary work of identifying skills: SCANS, enGauge, and P21. All the other frameworks used skills from these three, however, skills identified by other frameworks were rarely re-identified by other frameworks. The core skills were identified and defined by these three frameworks which truly represent twenty-first century skills. There are less than 10 workplace skills that were identified by these frameworks with different names but similar definitions. If all the workplace skills (not literacy skills) from these three (and other frameworks) were to be grouped based on their definitions (and not names), there are only eight skills that will explain the whole story (see Table 2). We renamed and grouped skills based on their definitions and created one single standard framework for businesses as suggested by Su et al. (2015) to arrive at common names, definitions, and categorization of the twenty-first century skills.

Table 2. Regrouped workplace skills from existing frameworks

Regrouped Workplace Skills	Skills Included
Communication and Collaboration	Sociability, Interpersonal, Listening, Speaking, Being part of groups, Collaboration, Interactive communication, Social skills, Communication and collaboration, Teamwork
Problem Solving and Critical Thinking	Decision making, Problem solving, Managing complexity, Higher-order thinking, Planning for results, Critical thinking, Critical thinking, problem solving, and decision making
Leadership and Responsibility	Responsibility, Resource management, Autonomy, Initiative skills, Leadership, Accountability, Leadership and responsibility, Initiative and self-direction
Creativity and Innovation	Creative thinking, Creativity, Curiosity, Creativity and innovation, Innovation
Self-management	Self-esteem, Integrity, Learning to learn, Risk taking, High-quality product production, Productivity, Self-direction
Information processing	Information processing, Cross-cultural skills
Flexibility and Adaptability	Adaptability, Flexibility, Adaptive learning, Fluid intelligence
General skills	Interactive use of tools, Systems thinking, Technology savvy

3. The twenty-first century skills framework development and skills ranking

The twenty-first century skills for businesses were identified from the regrouped list and a new framework was developed as a preliminary step for scale development. The skills in the framework were included based on two criteria. First, the skill has to be relevant to business and/or organizational settings. All the academic skills were ignored such as financial literacy, mathematics skills, etc. Second, the skill should have appeared in at least half of the frameworks. Any skill having frequency of 4 or above was included in the new framework. This limit was set to add most used and proven twenty-first century skills that were identified by multiple experts in different frameworks. Self-management, information processing, flexibility and adaptability, and general skills were, therefore, not included in the preliminary framework. Table 3 demonstrates the preliminary twenty-first century skills framework.

Table 3. Preliminary framework of twenty-first century skills

Twenty-first Century Skills Framework
1. Communication and Collaboration
2. Problem Solving and Critical Thinking
3. Creativity and Innovation
4. Leadership and Responsibility

3.1. Ranking

The skills in the preliminary framework were ranked by experts. A self-administered four-item questionnaire was developed to rank skills in terms of their importance for job performance. The respondents included directors, head of departments, deans, chairpersons, and individuals on top positions in the agri varsities of Pakistan. The respondents ranked skills from 1 to 5 where 1 was for most important skill and 5 for least important skill for three job categories: Administration and coordination, academic, and research staff. The survey was distributed by-hand in three universities, via email in two universities, and by post in one university. A total of 73 questionnaires were distributed. A maximum of three follow up emails and up to three personal visits were made for the collection of the questionnaires. The response rate was 42%. A total of 31 filled questionnaires were received. SPSS 20 was used for data analysis.

Table 4 represents the ranked framework with communication and collaboration skill ranked highest, followed by leadership and responsibility, then problem solving and critical thinking, and finally creativity and innovation.

Table 4. Descriptive statistics

	N	Mean	Std. Deviation
Communication and Collaboration	31	1.81	1.11
Leadership and Responsibility	31	1.96	0.98
Problem Solving and Critical Thinking	31	1.98	0.95
Creativity and Innovation	31	2.03	1.13

4. Scale development

The literature on twenty-first century skills and frameworks is driven by and originates from the education sector because twenty-first century movement is educational in nature. However, the other side of the page is mostly blank. Businesses and I-O psychologists haven’t played their role (actively) of making use of, integrating, measuring, or even defining twenty-first century skills. In their interdisciplinary review of twenty-first century skills, Finegold and Notabartolo (2010) concluded that despite consensus among researchers, policymakers, and educationists on the importance of twenty-first century skills, there is little evidence on how these skills impact individual and/or organizational outcomes. One of the reasons they identified was unavailability or shortage of appropriate measures. They further added that it will be beneficial for businesses if they know what skills boost individual performance and how to develop those skills in their workforce. Consistent with these findings, Su et al. (2015) argued that there is a need to develop a standard twenty-first century skills framework that will integrate these skills into I-O psychology so businesses and HR practitioners can use these skills to improve individual and organizational performance. They further argued that defining each skill in the framework clearly is the only way to move ahead.

Researchers use different assessments and scales to measure twenty-first century skills such as self-reporting scales, global rating scales, assessments, and observations (Lai & Viering, 2012). Educators use advanced assessments to measure twenty-first century skills such as Watson-Glaser to measure critical thinking. Not just educators but employers also use most of these measures to assess their employee’s skills. The review of the existing scales and measures on twenty-first century skills reveal that there is a lack of scales for businesses. Most of the scales are developed for educational institutions where they’re used to measure skills of students or teachers. Businesses have limited option in terms of measuring twenty-first century skills of their employees. There are two ways a firm can measure twenty-first century skills of its employees: Use skill-specific measures or use a single scale to measure all the skills. A single scale to measure twenty-first century skills is easy-to-administer and cost-effective. This is one way of bridging the science-practice gap—by providing employers with right measurement tools that they can use. Developing a scale for the framework will establish empirical evidence at the same time.

5. Item generation

We used deductive scale development approach for item generation (see Table 5 for definitions). The skills were grouped (e.g., communication and collaboration) but were treated separately (yet related) constructs during the item generation process. A total of 49 items were generated and it was ensured that each skill had at least four items as recommended by Hinkin (1998) and DeVellis (2016).

Table 5. Definitions of twenty-first century skills

Twenty-first Century Skill	Definition
Communication	Language competency in mother tongue and any other languages including reading, writing, speaking, listening, developing arguments, and using additional materials that will aid communication (Binkley et al., 2012)
Collaboration	The ability to interact effectively with others, work effectively in teams, guide and lead peers, and manage projects (Binkley et al., 2012)
Responsibility	The ability to understand personal and organizational interests and goals, and be responsible for one’s own outcomes (Greenhill, 2009)
Leadership	The ability to guide, influence, and lead others towards a goal, leverage the skills of others, show high levels of integrity and behavior, and the ability to ethically use power (Greenhill, 2009)
Problem Solving	An ability to engage, understand, and resolve unknown and complex problematic situations (Peña-López, 2012)
Critical Thinking	A thinking process that is goal-directed, reasoned, logical, and one that helps in solving problems and helps achieving desirable outcomes (Halpern, 2013)
Creativity	The ability to generate novel and useful ideas (Amabile, 1988)
Innovation	The ability to successfully implement creative ideas (Amabile, 1988)

6. Content validity

We employed two content validity techniques. First, we used expert ratings from 10 experts using Lawshe (1975) content validity ratio (CVR). The experts included human resource professors and PhD candidates. Each item was rated on a 3-point scale (Essential, Useful but not essential, and Not necessary). CVR for each item was calculated using the following formula:

$\mathrm{C}\mathrm{V}\mathrm{R}=\left[\left(\mathrm{E}-\left(\mathrm{N}/2\right)\right)/\left(\mathrm{N}/2\right)\right]$

N refers to the total number of experts and E refers to the experts who rated the items Essential. The CVR value ranges between −1 and 1. All the items having CVR value 0.60 or above were retained which is the acceptable value according to CVR table (Lawshe, 1975). All the items retained were rated “Essential” by at least eight out of 10 experts. This resulted in 32 items.

These 32 items were subjected to another content validity technique by MacKenzie, Podsakoff, and Fetter (1991). The items were randomly written on a paper and a total of nine categories were created—one for each skill and an “others” category. Respondents (graduate students N = 14) were asked to match each item to an appropriate category. According to Hinkin (1998), 75% is the minimum acceptable index. Likert scale format was used and the items were scaled on five-point scale. Out of 32 items, 29 were matched to the correct category more than 75% of the time.

7. Exploratory factor analysis

7.1. Participants

Data were collected from two agriculture universities which are located in two different cities: Rawalpindi and Peshawar. Purposive sampling technique was used for data collection but it was ensured that each department (faculty and non-faculty) was contacted during the process. The questionnaires were distributed personally as well as via email. The respondents included all the faculty and non-faculty staff. The total filled questionnaires received were 231 out of 480. A sample size of 231 for EFA is considered sufficient, according to Guadagnoli and Velicer (1988), who recommended that a sample size of 150 is sufficient to conduct EFA with accuracy. The participants filled a questionnaire having 29 items on a five-point Likert scale. The participants included faculty members (53%), administration and coordination staff (38%), and research staff (9%). The majority of the respondents were males (64%). The mean age of the respondents was 39.7 years.

7.2. Factorability

To analyze data for factorability and sample size adequacy, Bartlett’s test of sphericity and Kaiser-Meyer-Olkin (KMO) was conducted in SPSS. According to Tabachnick and Fidell (2007), a significant Bartlett’s test (p < 0.05) means the factor analysis should be accurate and KMO index of 0.6 or above warrants a decent factor analysis. The KMO was 0.93 and Bartlett’s test was significant which makes sample (N = 231) adequate for factor analysis.

7.3. Method

Mplus 7 was used to conduct EFA with Maximum likelihood (ML) estimation and oblique rotation. Multiple rounds of EFA were conducted in order to remove poorly loaded items. The analysis was repeated until an appropriate model was identified with clear factor loadings and acceptable model fit indices. Internal consistency was measured using Cronbach’s alpha. For newly developed measures, an alpha of 0.70 is considered minimum (Cortina, 1993). An alpha value greater than 0.70 for EFA is desirable which shows that the domain has been captured successfully (Nunally, 1978).

7.4. EFA

The decision to retain or remove an item was based on eigenvalues, scree plot, item loadings, and model fit indices. Factors were identified by inspecting eigenvalues and all the factors having eigenvalues greater than one were retained and inspected further. The first round of the EFA was conducted with 1–8 factor models to identify the best fitting model based on eigenvalues and model fit indices. Goodness of fit indices showed a lack of model fitting for all the models. Since eigenvalues revealed a five-factor model, therefore, loadings of the five-factor model were analyzed further. All the items clearly loaded on their appropriate factors as expected except creativity and innovation which loaded on two different factors. The items didn’t load on appropriate factors in the four-factor model as expected. Items having loading less than .40 and items that were loaded on multiple factors were removed. A total of six items were removed and the second round of EFA was conducted. Eigenvalues revealed a five-factor model and model fit indices showed a lack of model fitting. Analysis revealed that all the items loaded on their appropriate latent variable including creativity and innovation items that loaded on separate factors in Round 1. However, only one creativity item loaded on a separate factor (5th factor). All the items loaded on their appropriate latent variables in the four-factor model. Further analysis of the loadings of 4-factor model revealed that one item (Generating new ideas is normally not an issue for me) was loading on two factors so it was removed. Another item (A poor idea doesn’t stop me from generating new ideas) was loading on two factors but it had a loading of more than .4 on its home factor so it wasn’t removed. Only a single factor indicator was removed in this round.

Round 3 was conducted with 22 indicators where eigenvalues revealed a clear four-factor model with acceptable fit. Analysis of the factor loadings revealed that one item (A poor idea doesn’t stop me from generating new ideas) was cross loading and had a poor loading (0.39) on its home latent variable so it was removed and analysis repeated. The Round 4 was conducted with 21 indicators, eigenvalues represented a four-factor model with acceptable fit. All the values of model fit indices were in the acceptable range. All the items loaded clearly on their relevant latent variable with strong loadings. Only two items (I hold myself accountable for work I do and I try doing traditional work in a new way) had loadings below 0.50. All the other indicators had loadings greater than 0.60. These two items were deleted and another round of EFA was carried out.

A powerful four-factor model revealed with strong loadings on appropriate latent variables in the fifth round (see Table 6). The model scored reasonably better than the previous models on all the indices. The four-factor model fit the data well: χ² (101) = 198.69, RMSEA = 0.07, SRMR = 0.03, CFI = 0.96, TLI = 0.93. All the factor loadings were above 0.50 with no cross-loadings (see Table 7). The 19-item scale had four constructs. The first factor represented communication and collaboration with seven items, the second factor represented problem-solving and critical thinking with five items, leadership and responsibility emerged at the third factor with three items, and innovation was loaded on fourth factor with four items. All the factors were positively correlated and the correlation ranged from 0.40 to 0.60 revealing that the factors were separate but related constructs.

Internal consistency of the 19-item scale was measured using Cronbach’s alpha. The reliability of each latent variable was above 0.70. The alpha of the 19-item scale was 0.93 and three variables had alpha above 0.80, leadership and responsibility was the only construct with alpha 0.783, primarily because it only had three items. The final scale after EFA was internally consistent and each factor was measuring the construct.

8. Confirmatory factor analysis

8.1. Participants

CFA was conducted to confirm the factor structure and validity of the scale. Data were collected for the 19-item scale identified after EFA from four agri varsities located in different cities: Faisalabad, Multan, Tandojam, and Lasbela. A total of 520 self-administered questionnaires were distributed via email. Purposive sampling technique was used and all the faculty and non-faculty departments in all the four universities were contacted to ensure participation of all the departments. A total of 289 filled questionnaires were received at the end of the third follow up, which was way over the expected (N = 200) sample size. After treating incomplete questionnaires, 281 responses were used for analysis.

8.2. Method

MPlus 7 was used to conduct CFA with Maximum likelihood estimation. Reliability was measured through Cronbach’s alpha in SPSS 20. The quality of the model was analyzed using factor loadings, variation score, and factor determinacies.

8.3. CFA

The CFA revealed that all the factors loaded strongly on their appropriate latent variables (Figure 1). All the factor loadings were above 0.60 and factor determinacies were above 0.90. No significantly high, low, or negative correlations were found. The goodness of fit indices revealed a good model fit. All the indices were well within the acceptable range: χ2 (146) = 327.58, RMSEA = 0.07, SRMR = 0.05, CFI = 0.92, TLI = 0.90. Cronbach’s Alpha was in the acceptable range (α = 0.90). All the subscales had alpha values above 0.80 except leadership and responsibility which had alpha value 0.78, the lowest among all the other subscales.

Figure 1. CFA factor loadings.

9. Convergent and discriminant validity

The factor loadings were significant and higher than 0.50 which indicates all the loadings converge (Brown, 2014; Hair, Black, Babin, & Anderson, 2009). This clearly demonstrates convergent validity of the new scale. Further, the AVE estimate was also used to verify convergent validity. Table 8shows the AVE values along the diagonal for each construct. All the AVE values are above 0.50 except communication and collaboration. There could be two reasons. First, the number of items is high and the overall average loadings of the items are 0.65 which resulted in low AVE. Since individual factor loadings are well over 0.60 for each item which consequently represents convergent validity, therefore, an AVE value of less than 0.50 is acceptable in this case. Second, the AVE value is higher than squared correlation estimate (shared variance) of the construct with the other three constructs (see values above the diagonal in Table 8), this indicates that the communication and collaboration accounts for more variance in its individual items as compared to other constructs—and that the remaining portion of variance is accounted for both by other constructs and error estimates which is still lower than the AVE. Also, the reliability of this scale measured by alpha is 0.83 which indicates that the variance captured by the construct is greater than the variance captured by error (Fornell & Larcker, 1981; Hair et al., 2009; Raykov, 1997). Therefore, overall all the subscales demonstrated convergent validity.

Table 6. Goodness of fit indices

Round	Factors	χ^2	df	χ^2/df	RMSEA	SRMR	CFI	TLI	AIC
1	1	1327.87	377	3.52	0.1	0.08	0.74	0.72	15,313.9
	2	958.79	349	2.75	0.09	0.05	0.83	0.81	15,000.9
	3	791.58	322	2.46	0.08	0.05	0.87	0.84	14,887.6
	4	654.42	296	2.21	0.07	0.04	0.9	0.87	14,802.5
	5	522.88	271	1.93	0.06	0.03	0.93	0.9	14,720.9
2	1	1031.88	230	4.49	0.12	0.08	0.72	0.69	12,282.3
	2	711.57	208	3.42	0.1	0.06	0.82	0.78	12,006
	3	549.6	187	2.94	0.09	0.05	0.87	0.83	11,886
	4	426.22	167	2.55	0.08	0.04	0.91	0.86	11,802.6
	5	312.78	148	2.11	0.07	0.03	0.94	0.9	11,727.2
3	1	918.31	209	4.39	0.12	0.08	0.73	0.7	11,758
	2	596.71	188	3.17	0.1	0.06	0.85	0.81	11,478.4
	3	436.55	168	2.6	0.08	0.05	0.9	0.86	11,358.2
	4	317.23	149	2.13	0.07	0.03	0.94	0.9	11,276.9
4	1	864.76	189	4.58	0.12	0.09	0.73	0.7	11,261.3
	2	543.55	169	3.22	0.1	0.06	0.85	0.81	10,980.1
	3	386.12	150	2.57	0.08	0.05	0.91	0.87	10,860.6
	4	269.88	132	2.04	0.07	0.03	0.95	0.91	10,780.4
5	1	760.71	152	5	0.13	0.09	0.73	0.7	10,170.5
	2	453.93	134	3.39	0.1	0.06	0.86	0.82	9899.71
	3	297.62	117	2.54	0.08	0.04	0.92	0.88	9777.4
	4	198.69	101	1.97	0.07	0.03	0.96	0.93	9710.47

Table 7. Factor loadings

Item	Round 1 Factors					Round 2 Factors				Round 3 Factors
	1	2	3	4	5	1	2	3	4	1	2	3	4
1. I feel comfortable communicating in my mother tongue.	0.640*	−0.08	0.214*	−0.09	0.184*	0.637*	0	0.13	0.04	0.632*	−0.04	0.17	0.06
2. I can write and express myself through written material.	0.769*	0.06	−0.03	−0.01	0.11	0.810*	0.08	−0.07	0.03	0.798*	0.07	−0.06	0.04
3. I’m able to speak my mother tongue.	0.743*	0.11	−0.09	0.06	0.03	0.810*	0.09	−0.11	0.02	0.801*	0.11	−0.11	0.01
4. People don’t have issues in communicating with me.	0.580*	0.214*	0.05	0	0.02	0.638*	0.171*	0.05	−0.05	0.625*	0.17	0.06	−0.03
5. I prefer working in a team.	0.628*	0.05	0.01	0.14	−0.03	0.708*	0.04	0	−0.01	0.701*	0.05	0	−0.02
6. I help team members when working in a group.	0.640*	−0.10	0.08	0.221*	−0.07	0.732*	−0.167*	0.12	0	0.735*	−0.14	0.11	−0.01
7. I easily mix with new people.	0.565*	0.03	0	0.222*	−0.05	0.659*	−0.03	0.04	0.03	0.659*	−0.02	0.05	0.02
8. My peers approach me for help.	0.037	0.412*	0.271*	−0.01	0.192*	\\	\\	\\	\\	\\	\\	\\	\\
9. I feel good when others rely on my expertise for completing their work.	0.041	0.878*	−0.04	−0.06	−0.03	0.09	0.737*	0.03	0.01	0.06	0.776*	−0.01	0.01
10. I lead others to achieve a common goal.	−0.062	0.742*	0.05	0.06	−0.07	0	0.631*	0.13	0.01	−0.03	0.693*	0.09	−0.01
11. I know my personal goals.	0.072	0.535*	0.07	−0.06	0.10	0.07	0.587*	0.07	0.08	0.05	0.600*	0.06	0.07
12. I hold myself accountable for work I do.	−02	0.462*	0.01	0.02	0.18	0.01	0.460*	0.06	0.18	0	0.450*	0.06	0.18
13. I accept my failures.	0.056	0.377*	−0.01	0.13	0.20	\\	\\	\\	\\	\\	\\	\\	\\
14. I take responsibility of my work.	−0.038	0.309*	0.03	0.13	0.233*	\\	\\	\\	\\	\\	\\	\\	\\
15. Complex tasks don’t bother me.	−0.04	0.359*	0.04	0.287*	0.12	\\	\\	\\	\\	\\	\\	\\	\\
16. I solve problems systematically.	0.049	0.274*	0.09	0.425*	0.07	\\	\\	\\	\\	\\	\\	\\	\\
17. I identify multiple solutions to a problem.	0.047	0.275*	0.200*	0.293*	0.12	\\	\\	\\	\\	\\	\\	\\	\\
18. I try a different solution if the first one doesn’t solve the problem.	−0.018	0.14	0.585*	0.204*	−0.07	0.01	0.08	0.661*	0.03	0.02	0.09	0.644*	0.02
19. I don’t hesitate to make tough decisions.	−0.043	−0.02	0.783*	−0.05	0.03	−0.10	0.07	0.763*	−0.04	−0.10	0.04	0.782*	−0.03
20. I come up with logical solutions to problems.	0.085	0	0.687*	0.02	0.08	0.06	0.07	0.683*	0.03	0.06	0.03	0.717*	0.04
21. My solution to a problem is not influenced by my emotions.	0.011	0.08	0.511*	0.192*	0.03	0.03	0.04	0.620*	0.09	0.04	0.03	0.620*	0.08
22. My solution often leads to the desired outcome.	0.109	0.06	0.572*	0.182*	−0.01	0.13	0.03	0.641*	0.05	0.14	0.03	0.638*	0.04
23. Generating new ideas is normally not an issue for me.	0.092	−0.01	0.02	0.719*	0	0.224*	−0.11	0.20	0.381*	\\	\\	\\	\\
24. I try doing traditional work in a new way.	−0.021	−0.01	−0.06	0.802*	0.13	0.14	−0.12	0.15	0.519*	0.173*	−0.07	0.14	0.446*
25. A poor idea doesn’t stop me from generating new ideas.	0.056	−0.02	0.10	0.515*	0.18	0.14	−0.09	0.246*	0.441*	0.161*	−0.05	0.239*	0.389*
26. My ideas often help in the development of new products.	0.062	0.01	−0.01	0.01	0.705*	0.02	0.07	−0.08	0.734*	0.03	0.04	−0.06	0.734*
27. Practically implementing my ideas is not a challenge for me.	05	0.01	0.09	0.02	0.737*	−0.03	0.04	0.05	0.739*	−0.02	−0.01	0.07	0.765*
28. I don’t stop from coming up with new ideas even if I’m unable to implement them.	0.019	0.06	−0.02	0.10	0.733*	−0.01	0.11	−0.09	0.846*	0	0.08	−0.07	0.839*
29. I try to convert my idea into a useful product.	−0.019	0.06	0.16	0.06	0.583*	−0.04	0.12	0.12	0.619*	−0.03	0.07	0.14	0.625*
Item	Round 4 Factors				Round 5 Factors
	1	2	3	4	1	2	3	4
1. I feel comfortable communicating in my mother tongue.	0.635*	−0.06	0.18	0.06	0.642*	0.197*	−0.10	0.07
2. I can write and express myself through written material.	0.799*	0.07	−0.06	0.03	0.805*	−0.04	0.03	0.05
3. I’m able to speak my mother tongue.	0.802*	0.10	−0.11	0.01	0.801*	−0.11	0.10	0.02
4. People don’t have issues in communicating with me.	0.625*	0.16	0.05	−0.03	0.629*	0.06	0.15	−0.02
5. I prefer working in a team.	0.703*	0.04	0	−0.01	0.703*	0.01	0.04	−0.02
6. I help team members when working in a group.	0.742*	−0.15	0.11	−0.01	0.736*	0.09	−0.10	−0.03
7. I easily mix with new people.	0.664*	−0.02	0.05	0.02	0.661*	0.04	0	0
8. My peers approach me for help.	\\	\\	\\	\\	\\	\\	\\	\\
9. I feel good when others rely on my expertise for completing their work.	0.05	0.786*	−0.01	0.01	0.04	−0.01	0.824*	0.01
10. I lead others to achieve a common goal.	−0.03	0.696*	0.08	−0.01	−0.04	0.11	0.678*	0.01
11. I know my personal goals.	0.04	0.604*	0.06	0.07	0.05	0.11	0.520*	0.09
12. I hold myself accountable for work I do.	0	0.461*	0.06	0.17	\\	\\	\\	\\
13. I accept my failures.	\\	\\	\\	\\	\\	\\	\\	\\
14. I take responsibility of my work.	\\	\\	\\	\\	\\	\\	\\	\\
15. Complex tasks don’t bother me.	\\	\\	\\	\\	\\	\\	\\	\\
16. I solve problems systematically.	\\	\\	\\	\\	\\	\\	\\	\\
17. I identify multiple solutions to a problem.	\\	\\	\\	\\	\\	\\	\\	\\
18. I try a different solution if the first one doesn’t solve the problem.	0.03	0.09	0.642*	0.02	0.02	0.628*	0.13	0.01
19. I don’t hesitate to make tough decisions.	−0.09	0.02	0.794*	−0.03	−0.10	0.796*	0.02	−0.03
20. I come up with logical solutions to problems.	0.07	0.02	0.724*	0.04	0.07	0.743*	−0.01	0.03
21. My solution to a problem is not influenced by my emotions.	0.05	0.03	0.613*	0.08	0.05	0.615*	0.04	0.06
22. My solution often leads to the desired outcome.	0.150*	0.03	0.633*	0.03	0.150*	0.633*	0.04	0.03
23. Generating new ideas is normally not an issue for me.	\\	\\	\\	\\	\\	\\	\\	\\
24. I try doing traditional work in a new way.	0.192*	−0.04	0.13	0.407*	\\	\\	\\	\\
25. A poor idea doesn’t stop me from generating new ideas.	\\	\\	\\	\\	\\	\\	\\	\\
26. My ideas often help in the development of new products.	0.04	0.02	−0.05	0.732*	0.04	−0.03	0.01	0.739*
27. Practically implementing my ideas is not a challenge for me.	−0.01	−0.04	0.09	0.771*	0	0.10	−0.04	0.755*
28. I don’t stop from coming up with new ideas even if I’m unable to implement them.	0.01	0.06	−0.05	0.837*	0.01	−0.03	0.05	0.829*
29. I try to convert my idea into a useful product.	−0.03	0.05	0.16	0.630*	−0.02	0.185*	0.03	0.607*

Table 8. AVE, correlations, and shared variance of the constructs

	Items	1	2	3	4
1. Communication and Collaboration	7	0.42	0.29	0.30	0.28
2. Leadership and Responsibility	3	0.54	0.55	0.49	0.39
3. Problem Solving and Critical Thinking	5	0.55	0.70	0.50	0.35
4. Innovation	4	0.53	0.63	0.59	0.54

Table 9. Final twenty-first century skills framework

Final twenty-first century skills framework for businesses
1. Communication and Collaboration
2. Responsibility and Leadership
3. Problem Solving and Critical Thinking
4. Innovation

Discriminant validity was established by comparing AVE and shared variance. AVE estimates greater than shared variance estimates are indicative of discriminant validity (Fornell & Larcker, 1981). Each construct’s AVE is greater than the shared variance of that construct with the other three constructs, that is, estimates on the diagonal are higher than their corresponding estimates above the diagonal. This demonstrates discriminant validity of the new scale.

AVE along the diagonal, correlations are below and shared variance above the diagonal.

10. Discussion

The aim of the study was to develop a standard twenty-first century skills framework for practitioners and to develop a scale to measure the skills. The skills were identified from existing frameworks and only those skills were identified that were most relevant to businesses. The skills in the framework were then ranked by heads and directors of agri varsities in Pakistan. Each construct in the framework was defined clearly and a new scale was developed so agri varsities can use it for practical purposes.

The skills were paired together and were considered different but related constructs. Communication and collaboration skill was the most common skill that was a part of every twenty-first century skill framework reviewed in the study. It was ranked most important by the university heads which is consistent with the international twenty-first century skills frameworks. Leadership and responsibility was ranked second surprisingly as problem-solving and critical thinking was hypothesized to be more important. Logically, one cannot be a good leader or exhibit responsibility without being a critical thinker. Theoretically, problem-solving and critical thinking should be ranked higher than leadership and responsibility but results showed otherwise. Innovation was ranked last exactly as expected and in accordance with its frequency in the frameworks. Creativity was the only skill that was removed from the framework.

The scale development process was initiated by item generation. Initially, 49 items were generated using a deductive approach from the definitions of the skills. Two content validity techniques reduced the number of items to 29. This 29 item scale was further reduced to 19 items with multiple rounds of EFA. Finally, CFA further confirmed the 19 items with strong loadings and revealed the validity of the new scale. The convergent and divergent validity of the scale was demonstrated for construct validity. The final scale was not the same as hypothesized. We hypothesized a scale with all the skills from the framework. Creativity is part of the innovation and innovation is the practical-side of creativity, it can be argued that implementing ideas is more important than generating creative ideas. And creativity itself isn’t an identifiable twenty-first century skill for agri varsities.

All the other skills in the framework are the ones that are considered essential for twenty-first century workplace. According to Gray (2016), the top 10 skills that individuals will need in 2020 were significantly different than the top 10 skills in 2015. The top 10 skills that are expected to be essential in 2020 include skills like problem-solving, critical thinking, creativity, people management, and coordination. Out of the five most needed skills, four are included in the framework identified in the current study (see Table 9). The only skill that isn’t included is creativity; however, the new framework includes innovation which, according to several researchers (Findlay & Lumsden, 1988; Kanter, 1983; Van de Ven, 1986; West, 2002; West & Rickards, 1999) includes creativity but it isn’t a part of the framework explicitly.

The removal of creativity was consistent with O*NET framework (Burrus et al., 2013) where only innovation was identified as a skill. It was a data-driven framework that used data from O*NET which means employers are interested in innovation and not creativity. The removal of creativity from the scale and thus framework based on empirical data is a confirmation that creativity is not an essential skill that employers need rather innovation is. The twenty-first century-scale for teachers developed by Jia et al. (2016) revealed similar pattern. For their pre-service teacher scale, the items from “innovation and creativity” and “problem-solving” categories all loaded on a single construct.

11. Implications

This is the first study that explored the business aspect of the twenty-first century skills and developed a framework for agri varsities in Pakistan. Burrus et al. (2013) developed a data-driven framework but it didn’t include a supporting scale, and Jia et al. (2016) created a twenty-first century skills scale for teachers. We developed a standard framework and created a scale which will advance research in this field. To date, most of the research on twenty-first century skills was (and is) focused on curriculum and developing these skills in students. The gap between education and business research is filled by the current study empirically. The current study laid the foundation of research on twenty-first century skills in business sector and especially in the agri varsities of Pakistan. To our knowledge, there hasn’t been any prior study on this subject in Pakistan. It will give direction to the new researchers as to how they can move ahead and add more to this research avenue. Internationally, it is the first study that linked twenty-first century skills to I-O psychology and bridged the gap between science and practice as pointed by Su et al. (2015).

The scale provides guiding principles to the education sector of Pakistan in terms of what skills agri varsities specifically and non-agri varsities in general prefer to see in their employees. This can be used to develop curriculum and equip college graduates with relevant skills based on their career priorities. At the same time, it will guide research in education field. Educationists can develop and/or research on creating localized twenty-first century skills framework. In fact, an educational framework derived from a business framework makes more sense since it will not create a science-practice gap.

12. Limitations and future research directions

Despite practical and theoretical implications, the current study has some limitations. First, the scale’s scope is limited. It can only be used in agri varsities or varsities in Pakistan to measure employee skills. The purposive sampling technique was used which makes generalizability a challenge, however, since data were collected from all the six agri varsities in Pakistan, generalizability in the agri varsities will not be an issue. Second, self-reporting was the only data collection method used which adds to the limitations of the study. A possible area of future research is data collection on skills and performance from superiors. Researchers should expand the scale by collecting data from supervisors or perhaps a 360-degree data collection technique can be used to overcome the weakness of self-reporting data collection technique.

Future research should focus on several areas. First, validate the scale for other sectors and use other sampling techniques preferably probability sampling. Second, Multitrait-Multimethod (MTMM) matrix wasn’t used which is a recommended method to conduct convergent and discriminant validity (DeVellis, 2016; Hinkin, 1998). MTMM is a time-consuming multi-trait and multi-method technique that sure helps a lot with improving scale’s validity. Future studies can conduct the scale’s validity using MTMM and by comparing it with similar constructs. This will further ensure scale’s construct validity.

correction

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Acknowledgements

This study is sponsored by the Higher Education Commission (HEC) Pakistan. The HEC had no involvement in study design, data collection, analysis, interpretation, write-up, and article submission.

Additional information

Funding

The authors received no direct funding for this research.

Notes on contributors

Muhammad Sabih Javed

Muhammad Sabih Javed is a full-time PhD scholar of Management Sciences in the University Institute of Management Sciences, Arid Agriculture University Rawalpindi, Pakistan. He developed a new 21^st century skills scale for agri varsities as part of this PhD work. His research interest includes 21^st century skills, skills and competences, and individual performance.