Recalling Extra Data: A Replication Study of Finding Missing Markets

Abstract

We re-examine some of the strongest evidence supporting agricultural commercialisation, a highly touted yet under-researched development intervention. Our replication study re-examines Ashraf, Giné, and Karlan’s ‘Finding Missing Markets’ paper. Using the previous paper’s raw data, our research generally reproduces the original findings. We explore the evaluation’s theory of change, focusing on the result that first time export crop adopters benefit more from agricultural commercialisation than previous adopters. We also examine recall bias questions and provide sample size guidance for future researchers. Similar to the original paper, we find that the intervention mostly benefits households just entering the agricultural production value-chain.

1. Introduction

Our replication study reconstructs the paper ‘Finding missing markets (and a disturbing epilogue): Evidence from an export crop adoption and marketing intervention in Kenya’ (Ashraf, Giné, & Karlan, 2009a) using the original raw data. This innovative study by Nava Ashraf, Xavier Giné, and Dean Karlan (henceforth referred to as AGK) captures the much promoted yet under-researched concept of agricultural commercialisation and diversification through export-oriented crop promotion in the developing world.¹ Specifically, AGK evaluate the household-level income effects of efforts to encourage Kenyan farmers to adopt new export crops. AGK’s experimental design and package of commercialisation tools allow for a clean causal interpretation of the intervention’s impact.

Development professionals generally present agricultural commercialisation as a path to poverty alleviation, but little impact evaluation evidence exists to support this claim. Previous research shows households that adopt cash crops in which they maintain a comparative advantage to reach consistently higher income levels. However, many of these studies use village, region, or even nationally representative cross-sectional data.² Stewart et al.’s (2015) systematic review determines ‘Finding Missing Markets’ to be the only impact evaluation with low risk of bias in the area of agricultural practice innovation.

Agricultural commercialisation is a highly policy-relevant topic. International agencies continue to draw attention to agribusiness development opportunities because of both their ability to support rural households and the possibility of averting future food crises. World Bank (2013) highlight the importance of unlocking the potential of agricultural commercialisation, while describing some of the market failures that prevented previous efforts from being fully realised. Fan, Brzeska, and Olofinbiyi (2015) explain how properly incentivised agricultural commercialisation supports economic development.

Policy-makers recommend farmers transition from consumption farming to income farming (Fan et al., 2015). Agricultural commercialisation often presents opportunities for rural households to exploit their comparative advantage by growing labour-intensive agricultural commodities. Generally, commercialisation is advocated for as a means of agricultural diversification into high-value crops, while allowing for the purchasing of required food. Nevertheless, to reap the benefits of commercialisation, farmers must overcome a number of potential market failures, from unfamiliar export crops to capital constraints.

1.1. Reviewing the intervention

AGK evaluate an intervention designed to increase agricultural commercialisation by providing a package of extension and marketing services to treatment smallholder farmers. After identifying high-value international crops with strong local growth potential, the intervention targeted smallholder farmers with a package of services including current price information for those crops, linkages to a transportation supply chain, and, for one treatment arm, linkages with commercial banks. These services were provided to overcome general constraints faced by smallholder farmers engaged in higher value agriculture.

DrumNet, the project implementers, specifically targeted smallholder farmers through existing self-help groups (SHGs). They based intervention eligibility on agricultural household membership in a SHG registered with the Kenyan government; SHG interest in growing export crops; household access to irrigated land; and household ability to make a minimum payment of about US$10. DrumNet gave intervention recipients a month-long course in Good Agricultural Practices and instructions on opening a local bank account. They also gave households in the credit arm access to microcredit, with a minimum deposit required of those households to guarantee the loans they received.

The researchers based their paper on a plausible causal chain. Farmers gain knowledge and skills in growing high-value export crops. The adoption of these crops, along with a reduction in transportation costs, leads to increases in household income. In the end, it is hoped that this increase in household income will lead to general welfare benefits.³ The evaluation also examines possible heterogeneous impacts based on households that previously produced export crops versus first-time adopters, as first-time adopters may have more capacity for increasing their household income.

1.2. Introducing the replication

The original study uses a randomised evaluation framework to test the impact of providing a package of agricultural commercialisation services to SHGs. AGK conclude that the intervention only increases the household income of first-time export crop adopters. Possible recall bias and sample size constraints may influence the strength of the original results. Following our posted replication plan, our study aims to test the robustness of the existing agricultural commercialisation evidence and highlight the effectiveness of this package of interventions.

Following Brown, Cameron, and Wood (2014), we include three main sections: pure replication, measurement and estimation analysis, and theory of change analysis. In the pure replication section, we explain the data, methods, and assumptions we use to re-evaluate the intervention. Our measurement and estimation analysis examines the different datasets from the original analysis and includes an analysis of the original study’s sample size. Our theory of change analysis briefly explores alternative methods of analysing heterogeneous impacts by considering whether the types of crop being planted or the method of entering the commercialisation market is of more relevance to this intervention.

2. The pure replication

Our pure replication uses the raw data to reassess the intervention (Ashraf, Giné, & Karlan, 2013). We reconstruct the original paper using these data, the publication, and the survey instruments as a pseudo-codebook. While we see strengths in recoding the entire paper, ensuring our results to be independent of the original research findings, that independence comes at the cost of not necessarily following AGK’s evaluation path.

2.1. The data

The original study includes three datasets: the baseline, the follow-up, and the ‘extra 500’.⁴ AGK shared with us the survey instruments, the raw data, and two codebooks that generally describe the data but do not contain detailed explanations of how they generated the study variables.⁵

While most of the data record time of survey statuses, the extra 500 dataset contains different types of data. Budget constraints forced AGK initially to interview a relatively small sample. To address possibly inadequate sample sizes, AGK expanded their sample in the middle of the evaluation.⁶ As they did not have baseline information for these new households, AGK collected the information retrospectively through recall questions at the time of the follow-up.

2.2. Assignment to treatment, Table 1

We begin our pure replication by reproducing the SHG-level balance statistics reported in the first table of the original paper. These summary statistics results compare the pre-intervention SHG characteristics between control and treatment groups, with the treatment groups separated into credit and no-credit arms. Ashraf, Giné, and Karlan (2009a) note that they randomly assigned the 36 SHGs into three groups of 12 SHGs before implementing the intervention.⁷

Table 1. Replication results of pre-intervention balance with original and ‘updated’ group treatment assignment, AGK Table 1 reproduction

		Means				Means
		All	Control	Combined Treatment	p-Value	Credit	No Credit	p-Value of F-test
	Number of Observations	(1)	(2)	(3)	(4)	(5)	(6)	(7)
Panel A: Original group treatment assignments
Current number of members	36	28.7	31.4	27.3	0.51	25.3	29.3	0.70
		(17.5)	(19.6)	(16.6)		(10.9)	(21.2)
Age of SHG (months)	36	4.77	4.99	4.66	0.85	3.61	5.71	0.58
		(4.89)	(3.9)	(5.39)		(2.23)	(7.3)
SHG has social activities (1=yes)	36	0.53	0.75	0.42	0.06*	0.42	0.42	0.18
		(0.51)	(0.45)	(0.5)		(0.51)	(0.51)
Fee contribution to the SHG per member	36	103	87.5	111	0.54	112	109	0.83
		(106)	(56.9)	(124)		(133)	(120)
SHG has an account in the bank (1=yes)	36	0.64	0.67	0.63	0.81	0.58	0.67	0.90
		(0.49)	(0.49)	(0.49)		(0.51)	(0.5)
Main road paved (1=yes)	36	0.86	1.00	0.79	0.09*	0.75	0.83	0.21
		(0.35)	(0)	(0.41)		(0.45)	(0.4)
Distance to main market (km)	36	5.82	5.08	6.19	0.39	5.79	6.58	0.61
		(3.6)	(3.2)	(3.79)		(2.92)	(4.6)
Time to the main market (minutes)	36	41.5	22.5	51.0	0.09*	68.8	33.3	0.04**
		(47.1)	(16)	(54.6)		(70.3)	(24.5)
Panel B: ‘Updated’ group treatment assignments
Current number of members	36	28.7	31.4	27.3	0.51	24.2	31.0	0.52
		(17.5)	(19.6)	(16.6)		(11.3)	(21.3)
Age of SHG (months)	36	4.77	4.99	4.66	0.85	5.24	3.97	0.81
		(4.89)	(3.9)	(5.39)		(6.24)	(4.37)
SHG has social activities (1=yes)	36	0.53	0.75	0.42	0.06*	0.46	0.36	0.16
		(0.51)	(0.45)	(0.5)		(0.52)	(0.5)
Fee contribution to the SHG per member	36	103	87.5	111	0.54	111	110	0.83
		(106)	(56.9)	(124)		(128)	(126)
SHG has an account in the bank (1=yes)	36	0.64	0.67	0.63	0.81	0.62	0.64	0.97
		(0.49)	(0.49)	(0.49)		(0.51)	(0.5)
Main road paved (1=yes)	36	0.86	1.00	0.79	0.09*	0.69	0.91	0.07*
		(0.35)	(0)	(0.41)		(0.48)	(0.3)
Distance to main market (km)	36	5.82	5.08	6.19	0.39	5.42	7.09	0.37
		(3.6)	(3.2)	(3.79)		(3.09)	(4.46)
Time to the main market (minutes)	36	41.5	22.5	51.0	0.09*	65.0	34.5	0.06*
		(47.1)	(16)	(54.6)		(68.6)	(25.3)

Notes: Statistically significant differences between the original findings and replication results set in bold within the table. Throughout the tables, results’ statistical significance is noted as *p < 0.01; ** p< 0.05; ***p < 0.01. ‘Updated’ treatment assignments refer to alternative treatment statuses. Panel B exactly reproduces AGK Table 1.

The first table comprises eight variables: current number of members, age of SHG, existence of social activities, fee contributions to the SHG per member, existence of SHG bank account, paved main road, distance to the main market, and time to the main market. Due to the unavailability of raw SHG-level data, we use AGK’s pre-constructed variables to analyse the pre-intervention balance between the different SHGs.

The differences in our pre-intervention summary statistic finding in panel A of Table 1 leads us to explore alternative possible assignments to treatment. Table 1 panel B exactly replicates AGK’s first table including their treatment arm post-estimation tests on each baseline characteristic.⁸

As treatment status influences all of the evaluation results, we cross-reference our results with the original .do files. It appears that there are 12 control SHGs, 13 credit SHGs, and 11 no-credit SHGs.⁹ We use the updated treatment assignment for the remainder of our replication study, making the treatment assignment question moot because we combine all treatment SHGs into one group.

2.3. Reproducing the summary statistics

Our study focuses on the intent-to-treat (ITT) analysis for the combined treatment arms. In Table 2, we present the combined treatment group baseline summary statistics.¹⁰ The baseline statistically significant differences between the control and treatment households in Table 2 appear to bias the impact evaluation results downwards. While we control for any reported differences in the ITT analysis, these factors suggest the analysis may understate the influence of the intervention evaluation results.

Table 2. Updated baseline summary statistics, AGK Table 2 partial reproduction

	Means			p-Value on t-Test of Difference	AGK Original Means			AGK’s p-Value on t-Test of Difference
	All	Control	Treatment	(2) and (3)	All	Control	Treatment	(6) and (7)
	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
Member
Age of member	42.32	40.20	43.34	0.00***	41.2	39.3	42.2	0.17
	(12.62)	(12.46)	(12.57)		(12.2)	(11.9)	(12.2)
Literacy	0.91	0.91	0.91	0.91	0.90	0.89	0.90	0.79
	(0.29)	(0.29)	(0.29)		(0.30)	(0.30)	(0.29)
Risk tolerance	0.37	0.37	0.37	0.85	0.38	0.39	0.38	0.89
	(0.42)	(0.42)	(0.42)		(0.42)	(0.42)	(0.42)
Months as member in SHG	53.13	58.33	50.20	0.01***	52.51	57.2	49.8	0.51
	(40.07)	(44.68)	(36.96)		(39.7)	(44.4)	(36.5)
(i) Member of SHG is an officer (1=yes)	0.14	0.14	0.14	0.75	0.16	0.16	0.16	0.92
	(0.35)	(0.34)	(0.35)		(0.37)	(0.36)	(0.37)
Deposit in a formal bank (1=yes)	0.70	0.73	0.69	0.25	0.69	0.70	0.69	0.77
	(0.46)	(0.45)	(0.46)		(0.46)	(0.46)	(0.46)
Loan from formal institution (1=yes)	0.04	0.06	0.03	0.05**	0.04	0.06	0.03	0.03**
	(0.19)	(0.23)	(0.17)		(0.19)	(0.23)	(0.17)
Log of total annual household income	3.53	3.63	3.47	0.10*	3.49	3.59	3.44	0.30
	(1.17)	(1.16)	(1.18)		(1.20)	(1.19)	(1.20)
Number of household members	4.59	4.52	4.63	0.50	4.59	4.55	4.61	0.79
	(2.09)	(2.10)	(2.08)		(2.09)	(2.12)	(2.08)
Land
Harvest yield per acre	1.02	0.80	1.15	0.33	0.29	0.33	0.27	0.30
(in Ksh 100,000)	(4.63)	(6.02)	(3.62)		(0.62)	(0.65)	(0.60)
Proportion of land that is irrigated	0.37	0.37	0.36	0.84	0.40	0.39	0.40	0.87
	(0.31)	(0.29)	(0.32)		(0.31)	(0.29)	(0.32)
Total landholdings	2.13	2.20	2.09	0.36	1.80	1.90	1.75	0.56
(Acres)	(1.93)	(1.91)	(1.93)		(2.05)	(2.36)	(1.89)
Proportion of land devoted to cash crops	0.53	0.55	0.51	0.03**	0.58	0.59	0.57	0.54
	(0.28)	(0.28)	(0.28)		(0.25)	(0.24)	(0.26)
Production
Grows export crop	0.48	0.58	0.44	0.00***	0.46	0.55	0.41	0.15
(1=yes)	(0.50)	(0.49)	(0.50)		(0.50)	(0.50)	(0.49)
Sells to market	0.34	0.36	0.33	0.53	0.39	0.41	0.38	0.54
(1=yes)	(0.47)	(0.48)	(0.47)		(0.49)	(0.49)	(0.49)
Used hired labour	0.32	0.32	0.32	0.88	0..34	0.34	0.34	0.99
(1=yes)	(0.47)	(0.47)	(0.47)		(0.45)	(0.44)	(0.46)
Uses machinery and/or animal force (1=yes)	0.05	0.07	0.03	0.00***	0.06	0.09	0.04	0.06*
	(0.21)	(0.26)	(0.18)		(0.23)	(0.28)	(0.19)
Value of harvested produce	201.21	126.06	241.01	0.11	44.27	48.1	42.1	0.37
	(911.69)	(1228.56)	(685.55)		(72.7)	(73.1)	(72.6)
Production of French beans (in 1,000 kg)	0.75	0.54	0.85	0.40	3.40	2.89	3.65	0.61
	(5.55)	(1.20)	(6.66)		(14.3)	(13.1)	(14.9)
Production of baby corn (in kg)	4.10	10.70	1.06	0.13	13.3	21.0	9.48	0.34
	(94.60)	(167.01)	(14.95)		(114.1)	(162.1)	(80.6)
Total spent in marketing	1.30	0.88	1.53	0.16	1.00	0.36	1.36	0.06*
(in Ksh 1,000)	(5.95)	(3.64)	(6.91)		(8.18)	(2.13)	(10.1)
Use of inputs	0.98	0.98	0.97	0.21	0.95	0.95	0.95	0.89
	(0.15)	(0.12)	(0.17)		(0.23)	(0.22)	(0.23)

Notes: Columns 1–4 report our results, and columns 5–8 report AGK’s findings.

Overall, the original paper and the replication study report very similar baseline sample sizes for each variable. Small differences exist between the number of household baseline observations in the original and the replication, which become a bit larger when looking at the number of households in the specific treatment arms.¹¹

2.4. Reproducing the main results

Although AGK report a number of results, they present their key findings on the intervention’s effectiveness mainly in their fourth table.¹² In this difference-in-difference table they present results from controlling for SHG fixed effects and some of the variables highlighted in their balance tables. AGK includes both ITT and treatment-on-the-treated instrumental variable results. We only reproduce the ITT results, as AGK express strong validity concerns with their instrumental variable approach and present their ITT estimates as their preferred results.

AGK examine the causal relationship between the DrumNet intervention and an increase in household income. This ultimate outcome of interest first requires households to grow export crops and devote more land to cash crop production. As DrumNet targeted baby corn and French beans, we expect to see increases in the production of these cash crops in the treatment SHGs. The intervention simultaneously relieves marketing and transportation constraints, which should correspond to supply-side cost decreases for farmers in the treatment SHGs. Each of these intermediate steps along the intervention’s causal chain is designed to increase the household income.

Shown side-by-side in our Table 3, which partially reproduces AGK’s Table 4, the replication study results generally support the original findings. AGK highlight that treatment households are 19.2 per cent more likely to adopt export crops in the follow-up period. Our 24.6 per cent result for the same outcome of interest is very similar and of an equal statistical significance level as the original finding. Similar to AGK, we do not find any significant changes in French bean production, so we only present the baby corn results.

Table 3. Impacts of DrumNet: ITT, OLS, AGK Table 4 partial reproduction

	Export Crop	Export Crop AGK	Proportion of Land Devoted to Cash Crops	Proportion of Land Devoted to Cash Crops AGK	Production of Baby Corn (kg)	Production of Baby Corn (kg) AGK	Total Spent in Marketing	Total Spent in Marketing AGK	Logarithm of Household Income	Logarithm of Household Income AGK
	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	(10)
Panel A: intent-to-treat estimates, OLS
Post	−0.005	−0.004	−0.096	−0.078	21.455	11,133	0.801	3,567	−0.253	−0.107
	[0.053]	[0.059]	[0.014]***	[0.019]***	[19.496]	[34.775]	[0.791]	[2.133]	[0.099]**	[0.097]
Post x	0.246	0.192	0.061	0.043	86.465	396,735	−1.825	−3,528	0.176	0.089
Treatment	[0.067]***	[0.067]***	[0.017]***	[0.024]*	[41.947]**	[99.607]***	[0.943]*	[1.781]*	[0.113]	[0.110]
Number	1983	1983	1847	1779	1983	1981	1674	1653	1581	1566
of observations
R^2 squared	0.203	0.27	0.157	0.13	0.022	0.07	0.026	0.02	0.158	0.16
Mean	0.563	0.526	0.515	0.568	43.872	144.6	1.053	1.4	3.498	3.495
dependent variable

Notes: Odd columns contain replication results and even columns contain AGK’s original results.

Table 4. Impacts of DrumNet (prior exporters versus new adopters): ITT, OLS, AGK Table 7 partial reproduction

	Proportion of Land Devoted to Cash Crops		Proportion of Land Devoted to Cash Crops AGK		Production of Baby Corn (kg)		Production of Baby Corn (kg) AGK
Grows Export Crops at Baseline	Yes (1)	No (2)	Yes (3)	No (4)	Yes (5)	No (6)	Yes (7)	No (8)
Post	−0.117	−0.059	−0.102	−0.052	13.235	40.277	−18.175	64.590
	[0.020]***	[0.022]**	[0.017]***	[0.034]	[19.918]	[23.447]*	[31.051]	[48.654]
Post x Treat	−0.016	0.104	−0.019	0.086	150.759	30.338	489.112	338.607
	[0.029]	[0.029]***	[0.031]	[0.041]***	[84.862]*	[33.668]	[128.097]**	[104.410]**
Number of observations	895	940	818	909	957	1014	894	1027
R^2	0.19	0.15	0.19	0.14	0.03	0.05	0.10	0.08
Mean dependent variable	0.601	0.432	0.653	0.496	58.274	29.971	147.642	156.560

Notes: Columns 1, 2, 5 and 6 contain replication results. Columns 3, 4, 7 and 8 contain original results. Based on our earlier discussion and Ashraf, Giné, and Karlan (2008) we revert some of the publication coefficients to their working paper levels in columns 7 and 8.

Table 5. Impacts of DrumNet (prior exporters versus new adopters): ITT, OLS, AGK Table 7 partial reproduction

	Total Spent in Marketing (in Ksh1,000)		Total Spent in Marketing (in Ksh1,000) AGK		Logarithm of Household Income		Logarithm of Household Income AGK
Grows Export Crops atBaseline	Yes (9)	No (10)	Yes (11)	No (12)	Yes (13)	No (14)	Yes (15)	No (16)
Post	0.698	1.512	4.974	2.535	−0.143	−0.364	−0.127	−0.132
	[0.966]	[2.112]	[3.344]	[2.153]	[0.093]	[0.223]	[0.094]	[0.176]
Post x Treat	−2.037	−2.270	−6.488	−1.494	−0.077	0.494	−0.028	0.319
	[1.132]*	[2.481]	[3.319]*	[1.913]	[0.133]	[0.234]**	[0.119]	[0.182]*
Number of observations	883	779	800	793	842	727	764	744
R^2	0.03	0.08	0.03	0.10	0.16	0.20	0.20	0.19
Mean dependent	1.085	1.006	1.979	0.768	3.603	3.374	3.641	3.354
variable

Notes: Columns 9, 10, 13 and 14 contain replication results. Columns 11, 12, 15 and 16 contain original results. Based on our earlier discussion and Ashraf, Giné, and Karlan (2008) we revert some of the publication coefficients to their working paper levels in columns 11 and 12.

Table 6. Wood and Dong versus AGK research decisions

	Issues	Revision by Wood & Dong	Revision by AGK
1	SHG Treatment Status	Moved one SHG from non-credit to credit group.	Same as Wood & Dong
2	Households without IDs	Filled in IDs by matching respondent’s name with administrative dataset	Used on-the-ground information
3	Double-entry	Used most ‘reasonable’ value	Used on-the-ground information
4A	Land size (unit - stem)	Dropped values when reported in stem	Separately calculated fruit tree production using stem
4B	Land size (missing values)	Imputed based on control variables	Assumed extreme outliers as missing and substituted missing values with the mean
4C	Land size outliers in extra 500 sample	Multiplied the outliers by 100	Multiplied the outliers by 100
5	Intercropping	Did not account for this possibility	Accounted for fruit tree intercropping with stems
6	Outliers of French bean production in extra 500 sample	Divided the outliers by 100
7	Assumption on fertiliser	Chemical & organic fertiliser are the same	Chemical & organic fertiliser are the same
8	When total harvest produced exceeds total harvest sold	Decreased sold value to 100% of production	Increased produced value to 100% of sold
9	Cap on number of sales	We cap at 100
10	Transportation cost reported but number of sales missing	Assumed one sale	Imputed using amount sold and amount sold in typical transaction
11A	Missing crop quantities	Imputed based on crop code & amount planted	Imputed based on baseline control variables
11B	Missing control variables	Imputed based on other control variables	Imputed based on baseline control variables
11C	Missing household member information in the follow-up	Used baseline household size	Imputed based on baseline control variables
11D	Missing number of sales	Assumed no sales	Imputed using amount sold and amount sold in typical transaction
11E	Missing household income data	Assumed no household income	Imputed based on baseline control variables

Table 7. Updated baseline DrumNet summary statistics by dataset: replication results

	Original Data Means				Extra 500 Data Means
	All	Control	Treatment	p-Value on t-Test of Difference (2) and (3)	All	Control	Treatment	p-Value on t-Test of Difference (6) and (7)	p-Value on t-Test of Difference (1) and (5)
	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)
Member
Age of member	43.17	41.98	43.83	0.07*	40.71	35.56	42.52	0.00***	0.00***
	(13.22)	(13.20)	(13.20)		(11.23)	(8.77)	(11.45)
Literacy	0.89	0.88	0.89	0.60	0.95	0.99	0.93	0.03**	0.00***
	(0.32)	(0.33)	(0.31)		(0.22)	(0.10)	(0.25)
(i) Member of SHG is an officer (1=yes)	0.17	0.16	0.18	0.38	0.08	0.08	0.08	0.94	0.00***
	(0.38)	(0.36)	(0.39)		(0.27)	(0.27)	(0.27)
Land
Proportion of land that is irrigated	0.32	0.35	0.30	0.04**	0.45	0.41	0.46	0.14	0.00***
	(0.31)	(0.29)	(0.32)		(0.30)	(0.31)	(0.30)
Total landholdings	2.28	2.37	2.23	0.33	1.83	1.75	1.86	0.61	0.00***
(Acres)	(1.95)	(2.09)	(1.87)		(1.84)	(1.27)	(2.01)
Proportion of land devoted to cash crops	0.53	0.55	0.52	0.26	0.52	0.57	0.50	0.04**	0.44
	(0.27)	(0.26)	(0.27)		(0.30)	(0.32)	(0.29)
Production
Grows export crop	0.57	0.65	0.53	0.00***	0.30	0.36	0.28	0.13	0.00***
(1=yes)	(0.50)	(0.48)	(0.50)		(0.46)	(0.48)	(0.45)
Used hired labour	0.29	0.29	0.29	0.98	0.39	0.40	0.38	0.73	0.00***
(1=yes)	(0.45)	(0.45)	(0.45)		(0.49)	(0.49)	(0.49)
Uses machinery and/or animal force	0.07	0.10	0.05	0.01***	0.00	0.00	0.00	.	0.00***
(1=yes)	(0.26)	(0.31)	(0.23)		(0.00)	(0.00)	(0.00)
Production of French beans (in 1,000 kg)	1.02	0.66	1.21	0.31	0.20	0.20	0.21	0.94	0.03**
	(6.71)	(1.34)	(8.24)		(1.06)	(0.53)	(1.18)
Production of baby corn (in kg)	4.84	13.64	0.18	0.14	2.57	2.35	2.65	0.92	0.72
	(114.06)	(193.73)	(3.74)		(23.79)	(21.69)	(24.49)
Use of inputs	0.98	0.98	0.98	0.63	0.97	0.99	0.96	0.21	0.22
	(0.14)	(0.13)	(0.15)		(0.18)	(0.11)	(0.20)

Notes: We break results out by data source, with all results coming from the replication study. Column 9 compares all of the ‘original’ data to all of the extra 500 data among select variables at the baseline time period.

Our reproduction of AGK’s baby corn results, while similar in statistical significance level, differ quite substantially in coefficient size. We find average increases of an extra 86 kilograms in treatment households. These results are closer to the findings in AGK’s working paper (Ashraf, Giné, & Karlan, 2008), where treatment households average an increase of 397 kilograms production of baby corn. Throughout the rest of the paper, we correct for misplaced commas in AGK’s production of baby corn, production of French beans, and total spent in marketing results.

A few notable differences exist between the original findings and the replication results in some of the secondary outcomes in our Table 3.¹³ Overall, outside of the difference in coefficient size for a few of the outcomes of interest, the original results are generally robust to our replication study.

2.5. Exploring heterogeneous impacts

AGK explore how the DrumNet interventions influence previous crop exporters versus new adopters. The researchers determine that the intervention mainly benefits producers who had not previously grown the recommended export-oriented crops. We focus on these results.

The heterogeneous impact findings in our Table 4 and Table 5 mostly support the original results. Our results show statistically significant increases in baby corn production, on average, for previous export crop adopters. However, that statistical significance does not carry over to the new export crop adopters in our results. These findings generally reinforce AGK’s claim that the intervention caused, on average, a statistically significant increase in the household income of newly commercialised agricultural households.

The possible presence of endogenous sorting may bias these heterogeneous impacts, in that households more prone to selling their crops may select themselves into the adoption group. A number of factors may influence crop adoption patterns, including a host of unobservable household and individual attributes. We do not directly address these concerns, although we find little correlational evidence to suggest that richer households sell export crops.

2.6. Pure replication challenges

Our decision to reconstruct independently the original evaluation from the raw data limits our replication study. As seen in our reproduced summary statistics, although the number of observations in the surveys remains very similar, in some instances the magnitudes of the variables of interest vary quite substantially with the published results.

2.6.1. Data documentation limitations

We encounter difficulties identifying households within and across the survey rounds, partly because of inconsistent household identification numbers. The multi-dataset format requires extensive merging throughout the analysis process, further complicating household identification.

We make a number of coding assumptions in our replication process. We assume the extra 500 survey instrument documentation reverses the crop production results, that many of the extra 500 observations are missing decimal points (and thus need to be divided by a factor of 100), and that missing variables need to be imputed.

We find it easier to reproduce some variables than others. Variables prove difficult to reproduce for a number of reasons. For example, without knowing the date of the interview, we approximate the number of months as member in SHG. Total spent in marketing is the hardest variable for us to reproduce, as it requires household transport costs and assumptions of the number of typical transactions for each household.¹⁴

2.6.2. Conversion factors

AGK’s survey instruments include difficult to calculate local measurement units.¹⁵ We convert production and planting area to standard units of measurement using AGK’s agricultural production conversion tables.¹⁶ We were unable to convert production measured in some units, like ‘gorogoro’ (ERS, 1992). By not accounting for planting measurements recorded in ‘stems’, which apparently represent fruit trees, we could not capture around 9 per cent of the recorded responses.

The use of non-standard units of measure for agricultural reporting in the original study touches on a much broader debate about these measurements in the literature (Diskin, 1999; Fermont & Benson, 2011). Although conversion factors contribute to the difficulties we encounter with reproducing the original evaluation results, we ultimately manage to capture most of the original observations in our analysis.

2.6.3. Pure replication conclusions

Our overall results are similar, but not identical to the original publication. Researchers make numerous decisions during an evaluation. In Table 6 we document a number of the decisions me made during our replication process, and how they differed from the original authors. Highlights of these differences include imputation methodologies, methods for working with outliers, and data assumptions. We consider the general robustness of our pure replication supportive of continued interest in future agricultural commercialisation projects.

3. Measurement and estimation analysis: sample size

Our measurement and estimation analysis generally follows our replication plan in examining the evaluation findings from sample size requirements and data perspectives. The one major caveat is our analysis of the extra 500 sample, which we did not pre-specify due to a lack of understanding as to the extent of recall sampling before we undertook our replication study. First, we separate the extra 500 households from the original survey participants to check the within- and between-sample balance for both datasets. We then calculate sample size requirements for future work.

The publication results rely on pooling the baseline and the extra 500 surveys to measure the impact of the intervention. Enumerators collected the extra 500 baseline information at the time of the follow-up survey. These extra 500 data may include courtesy bias, as the intervention had already begun, or recall bias, as enumerators asked respondents to remember information from the previous year.

Given the differences in the baseline data collection timelines, we examine the datasets separately. Table 7 shows the balance between the original baseline data and the extra 500 recall data. Age continues to be statistically significantly imbalanced across all of our datasets. The other variables highlighted in Table 2, export crops and uses of machinery/animal force, are only significantly imbalanced statistically in the original dataset. Literacy and proportion of land devoted to cash crops are only significantly imbalanced in the extra 500 data.

We identify a number of statistically significant imbalances when comparing the balance between the two datasets. Following Bruhn and McKenzie (2009), we combine the data and conduct an F-test on the joint orthogonality of the variables in relation to treatment. With an F value of 4.13, the test finds the collective contribution of the variables in the balance tables to be significantly different from each other at the 1 per cent level when comparing the control and the treatment households.

Of all of the evaluation’s outcome variables, we focus our analysis on the one arguably hardest to precisely recall, proportion of land devoted to cash crops. To understand differences between the two surveys better, we separate the responses by the dataset and treatment status in the histogram plots in Figure 1. As the recall period refers to the baseline period, we only present the baseline data results.

Figure 1. Baseline data distribution of land devoted to cash crops, by dataset.

The distribution of the baseline responses for proportion of land devoted to cash crops differ quite substantially between the two datasets. We find a wide distribution of answers in the original dataset. Less variation exists in the extra 500 dataset. The clumpier distribution of the extra 500 data of around one third and one half suggests recall approximations. We formally test the equality of the distributions with a Kolmogorov–Smirnov test, finding a statistically significant difference between the two groups at a 1 per cent level (Stephens, 1992).

Because of our concerns with the recall data, we reproduce the main evaluation results excluding the extra 500 data in Table 8. Although we continue to find similar coefficient size between the combined and original samples, some of the key results lack statistical significance with this subset of the sample. The extra 500 data appear to drive at least some of the main results, especially in the proportion of land devoted to cash crop variable coefficient, which switches signs after excluding the extra 500 data.

Table 8. Impacts of DrumNet: ITT, OLS by dataset

	Export crop, original only	Export crop, entire sample	Land devoted to cash crops, original only	Land devoted to cash crops, entire sample	Production of baby corn (kg), original only	Production of baby corn (kg), entire sample	Total spent in marketing, original, original only	Total spent in marketing, entire sample	Log of household income, original only	Log of household income, entire sample
	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	(10)
Panel A: intent-to-treat estimates, OLS
Post	−0.089	−0.005	−0.144	−0.096	21.894	21.455	1.535	0.801	−0.222	−0.253
	[0.082]	[0.053]	[0.020]***	[0.014]***	[24.784]	[19.496]	[1.201]	[0.791]	[0.100]**	[0.099]**
Post x	0.227	0.246	−0.006	0.061	114.220	86.465	−2.518	−1.825	0.099	0.176
Treatment	[0.106]**	[0.067]***	[0.027]	[0.017]***	[70.791]	[41.947]**	[1.376]*	[0.943]*	[0.125]	[0.113]
Number of	1252	1983	1117	1847	1252	1983	1279	1674	1186	1581
Observations
R^2	0.290	0.203	0.244	0.157	0.036	0.022	0.030	0.026	0.171	0.158
Mean	0.613	0.563	0.497	0.515	50.887	43.872	1.252	1.053	3.497	3.498
dependent variable

Notes: We present the main outcomes of interest from the main ITT analysis excluding the extra 500 data in columns 1, 3, 5, 7 and 9. For comparison purposes, we also reproduce the entire sample results from our Table 3 above in columns 2, 4, 6, 8 and 10.

Sample size constraints restrict our ability to make definite statements about the results from the separate datasets. AGK specifically collected the extra 500 data to offset their power concerns with the original size of the sample. We calculate sample size requirements from the baseline sample to determine the necessary size of future samples for researchers hoping to detect statistically significant change in theses outcomes of interest. We also calculate the actual intra-cluster correlation coefficient (ICC) for each outcome of interest, as a reference for future research.¹⁷

We focus our interest on the log of household income variables in Table 9. Our calculation results show, unsurprisingly given the published findings, that the log of household income variable does not contain enough observations to detect statistical significance. After calculating the effect size, we determine the required size of future evaluations to be able to test for statistical significance in log of household income. We perform a back-of-the-envelope calculation assuming the same number of SHGs, the same average number of interviewees per SHG, and that our replication results are the true change in household income. Under these assumptions, we estimate that future evaluations would need to include approximately 2,500 people to be able to detect a statistically significant increase in the log of household income among treatment households.

Table 9. Post-intervention sample size calculations, baseline by dataset

Outcome of interest	Including extra 500 dataset (1)	Intra-cluster correlation coefficient (2)	Minimum detectable effect size (3)	Change in outcome of interest, our replication study (4)	Change in outcome of interest, AGK publication (5)
Proportion of land devoted to cash crops	Yes	0.101	0.099	0.061	0.043
	No	0.105	0.105	−0.006	n/a
Baby corn production	Yes	0.013	20.441	86.465	396.735
	No	0.018	28.192	114.220	n/a
French bean production	Yes	0.047	1.488	2.232	1.611
	No	0.037	1.892	3.119	n/a
Total spent in marketing	No	0.024	1.545	−1.825	−3.528
Log of household income	No	0.050	0.329	0.176	0.089

Notes: We use Djimeu and Houndolo (2016) to conduct these sample size calculations, assuming a standard statistical significance level of give per cent and a .80 power of the test. We separate the results by inclusion of either baseline samples or only the original data set when possible. Based on our earlier coefficient size discussion and Ashraf, Giné, and Karlan (2008) we revert some of the publication coefficients to their working paper levels.

4. Theory of change analysis: reanalysis and alternative heterogeneity analysis

AGK’s heterogeneous impact results may lead to striking policy implications, notably that policy-makers should focus on subsistence farmers unfamiliar with export crops. We explore the hypothesis that the DrumNet intervention encourages formerly isolated farmers to work within established markets. Under this theory, the specific export crop encouragement is less important than the commercialisation of smallholder farmers. This alternative hypothesis would alter the policy recommendations somewhat, away from encouraging specific export crop production and towards supporting farmers to engage with markets.

In our opinion, DrumNet’s success with subsistence farmers may stem from the intervention’s focus on factors that had previously prevented these farmers from growing crops that are more valuable. A large body of literature finds subsistence crop production a typically inefficient use of land, with labour-intensive commercial crops potentially allowing for significant welfare gains through comparative advantage in lower wage costs.¹⁸ Relating to market failures, subsistence farmers may not adopt export crops due to a lack of established relationships with traders. In addition, these same farmers may fear unknown adoption effects, due to a need for knowledge of advanced agricultural practices or uncertainty around increased household dependence on outside markets for food purchases.

Defining previous adopter of export crops is difficult. As seen in Figure 2, large differences exist between the per cent of previous adopters of export crops in the original baseline data and the extra 500 baseline recall data. The marked difference in the baseline status of the two samples questions the interpretability of the pooled heterogeneous impacts.

Figure 2. Previous adopters of export crops, by dataset.

In addition to our concerns over differences between the two datasets, we also explore the robustness of the heterogeneous outcomes findings in relation to alternative definitions of previous adopters. AGK define previous adopters rather narrowly as those farmers who previously grew the three DrumNet target crops. We explore an alternative approach to defining previous adopter.

Rerunning the original heterogeneous impact estimations with our alternative previous market participant definition creates different results. First, in panel A of Tables 10 and 11, we separate the original heterogeneous impact results into the full sample and the sub-sample that only includes the original baseline households. Then we replace the before-intervention cash crop adopter category with our sold crops at a market variable in panels B of these tables.¹⁹

Table 10. Impacts of DrumNet, ITT, OLS heterogeneous outcomes, by baseline characteristics and dataset

	Proportion of Land Devoted to Cash Crops, entire sample		Proportion of Land Devoted to Cash Crops, excluding extra 500		Production of Baby Corn (kg), entire sample		Production of Baby Corn (kg), excluding extra 500
	Yes	No	Yes	No	Yes	No	Yes	No
	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
Panel A: Binary baseline heterogeneous variable, Previously grew export crop
Post	−0.117	−0.059	−0.136	−0.140	13.275	40.562	7.941	64.338
	[0.020]***	[0.022]**	[0.026]***	[0.038]***	[19.923]	[23.491]*	[19.921]	[38.429]
Post x Treat	−0.016	0.104	−0.053	0.042	151.484	30.233	219.141	−8.010
	[0.029]	[0.029]***	[0.038]	[0.044]	[84.854]*	[33.686]	[126.742]*	[37.074]
Number of	895	940	646	471	957	1014	707	545
observations
R^2	0.19	0.15	0.22	0.2	0.03	0.05	0.06	0.05
Mean	0.601	0.432	0.582	0.381	58.473	30.070	1.956	3.598
dependent variable
Panel B: Binary baseline heterogeneous variable, Previously sold to market
Post	n/a	n/a	−0.122	−0.158	n/a	n/a	5.789	32.479
			[0.029]***	[0.025]***			[25.597]	[35.312]
Post x Treat			−0.008	−0.001			63.329	131.449
			[0.055]	[0.033]			[31.948]*	[100.596]
Number of			390	390			390	390
observations
R^2			0.29	0.28			0.18	0.04
Mean dependent variable			0.501	0.495			34.630	59.573

Notes: All columns are replication results. Export crop refers to households who sold French beans and baby corn before the intervention began. Sold to market refers to households who report selling crops to a village or distant markets in the baseline period.

Table 11. Impacts of DrumNet, ITT, OLS heterogeneous outcomes, by baseline characteristics and dataset

	Total Spent in Marketing (in Ksh 1,000), entire sample		Total Spent in Marketing (in Ksh 1,000), excluding extra 500		Logarithm of Household Income, entire sample		Logarithm of Household Income, excluding extra 500
	Yes	No	Yes	No	Yes	No	Yes	No
	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
Panel A: Binary baseline heterogeneous variable, Previously grew export crop
Post	0.698	1.512	1.306	2.220	−0.143	−0.364	−0.117	−0.379
	[0.966]	[2.112]	[1.200]	[3.089]	[0.093]	[0.223]	[0.101]	[0.211]*
Post x Treat	−2.037	−2.270	−2.775	−2.837	−0.077	0.494	−0.113	0.427
	[1.132]*	[2.481]	[1.421]*	[3.528]	[0.133]	[0.234]**	[0.141]	[0.237]*
Number of	883	779	734	545	842	727	693	493
observations
R^2	0.03	0.08	0.04	0.09	0.16	0.2	0.17	0.24
Mean	1.085	1.006	1.233	1.279	3.603	3.374	3.622	3.320
dependent variable
Panel B: Binary baseline heterogeneous variable, Previously sold to market
Post	n/a	n/a	1.462	1.564	n/a	n/a	−0.001	−0.325
			[2.811]	[1.499]			[0.174]	[0.167]*
Post x Treat			−3.671	−1.954			−0.377	0.321
			[2.805]	[1.758]			[0.210]*	[0.187]*
Number of			390	390			390	390
observations
R^2			0.05	0.08			0.25	0.19
Mean dependent variable			2.011	0.859			3.553	3.467

Notes: All columns are replication results. Export crop refers to households who sold French beans and baby corn before the intervention began. Sold to market refers to households who report selling crops to a village or distant markets in the baseline period

Focusing on panel A, we find that limiting the sample to only the households included in the original baseline survey generally reduces the magnitude of the coefficients on the outcomes of interest. These results support a hypothesis that the extra 500 data represent a different subsample of households than the original baseline households. The findings would also support the possibility of a courtesy bias in the extra 500 household responses (Crawford, 1997). In the courtesy bias scenario, treatment households, having been exposed to the intervention before being asked recall baseline questions, might tailor their answers to demonstrate the effectiveness of the intervention.

When we change the binary previous adopter baseline variable to previously sold to market, we find very similar results with the original heterogeneous impact analysis. As we note above, the original researchers did not ask the extra 500 households to recall this question, so we cannot present entire sample heterogeneous impact results.

Specifically, we see average increases in baby corn production by treatment households, and marketing costs tend to reduce. Of particular interest, we continue to find dichotomous log of household income results. Following this alternative approach to measuring heterogeneous impacts, treatment households that previously sold to markets average statistically significant decreases in the income measurement, whereas we find the opposite result for treatment households who had previously not sold to a market.

The relationship between different channels of entering the agricultural production value chain remains an area for future research. We believe previously selling crops to market to be more relevant than producing specific export crops, as markets should dictate the highest prices for the crops with the most demand. However, incomplete markets may interrupt this natural process. Fan, Brzeska, Keyzer, and Halsema’s (2013) recent work on helping smallholder farmers transition from subsistence farming to more profitable agricultural strategies provides policy options to encourage smallholder participation in the commercial food chain.

5. Conclusion and limitations

Our study suggests, similar to previous findings, that the intervention may benefit households by encouraging them to participate in market-based economies. Nevertheless, we suggest some degree of intervention targeting, as our results show offering assistance to farmers already participating in the value-chain is less productive.

Regarding our robustness checks, we find our data pooling and future sample size requirements results most striking. The differences between the two baselines make us question the interpretability of the study’s findings. Our analysis suggests future evaluation needs to substantially increase sample sizes to detect possibly statistically significant results in household income.

We encounter a number of difficulties with reproducing the original evaluation. Data limitations, both in the original data and the additional datasets we hoped to incorporate into the analysis, prevent us from completing all our replication plan activities. The unavailability of exact geographic locations prevents us from conducting potential control contamination analysis. In addition, data limitations around gender and assets restrict our planned analyses in those areas.

Our replication study provides a number of avenues for further research. One area is disentangling the relationship between different channels of entering the agricultural production value chain. Another connected area is the potential for targeting agricultural commercialisation interventions. We consider the necessity of working with smallholder groups or exclusively with subsistence farmers within agricultural commercialisation interventions to be open research questions. Finally, space exists for more research on the use of cash crop proceeds. Beyond household income, and in the vein of Lundberg, Pollak, and Wales (1997) and Meinzen-Dick et al. (2010), we encourage future researcher on gender specific questions in relation to adoption and gains from agricultural commercialisation. We hope this replication study will encourage future work in this field.

Acknowledgements

We would like to thank Annette Brown, Howard White, Heather Lanthorn, Eric Djimeu, and four additional anonymous external reviewers for helpful comments on this research. Thanks also to Alan de Brauw, our external project advisor, who’s sharp eye greatly improved our paper. We would also like to thank Sebastian Insfran Moreno for his early-stage research assistance and Courtney Soderberg for sharing her statistician perspective on power calculations in this after-evaluation context. Finally, we extend our gratitude to Nara Ashraf, Xavier Giné, and Dean Karlan, the authors of the original study, for providing us with the original data and survey instruments that made this replication study possible. The Stata statistical code to reproduce our results is available here: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/OCPDXG.

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplementary Materials

Supplementary Materials are available for this article which can be accessed via the online version of this journal available at https://doi.org/10.1080/00220388.2018.1506574

Additional information

Funding

This work was mainly conducted as in-house research when Wood and Dong worked at 3ie. We thank 3ie for their generous support.

Notes

1. See World Bank (2013) and Kherallah, Delgado, Gabre-Madhin, Minot, and Johnson (2002) for examples of export crop promotion efforts.

2. Obare (2000) uses a small district-level survey to discuss some of the impediments to agricultural commercialisation in a Kenyan context. While this research contextualises the research space, it does not answer the general effectiveness question.

3. As noted in the epilogue of the original paper, changes in Europe’s food import certification system prevent the evaluation, and us, from estimating the long-term effects of this intervention.

4. A reviewer noted that the ‘extra 500’ data do not actually contain 500 household observations. We follow AGK’s naming convention in this regard. An additional 500 households were interviewed but AGK only include a subsample of them in the analysis due to intervention eligibility.

5. AGK could not locate the formal follow-up survey instrument, but the composite version they provided us is relatively accurate in relation to the dataset.

6. Ashraf, Giné, and Karlan (2009a) detail their data collection process on page 976 and in footnote 8. We did not focus on recall in our replication plan, as we did not understand the extent AGK relied on this technique. See Beegle, Carletto, and Himelein (2012) for a greater discussion of issues with recall bias in agricultural data.

7. Targeting treatment at the SHG level allows for greater effectiveness because the intervention reaches a larger group of people with a lower cost. Focusing on SHGs also alleviates some unobserved-characteristic concerns among smallholder farmers. Future researchers should note the SHG intervention participation requirement when considering the generalisability of the findings.

8. We keep the SHG that switches treatment arm status within our balance analysis, although no members from that SHG remain in the follow up surveys after data cleaning. We follow AGK’s lead in referring to infrastructure in reference to the roads variable discussed in Table 1A and 1B.

9. We contacted AGK about our assignment concerns and they said we had discovered an ‘error’ in their .do file and that the assignment was equal between the control and two treatment arms. Our do-file is publicly available here (doi:10.7910/DVN/OCPDXG).

10. The treatment arms differ only in access to credit. AGK did not find any significant differences between the two arms in their main analysis. Copies of all tables with alternative treatment assignments are available upon request. Supplementary Materials Table 1(b) includes a breakdown of the numbers of observations by treatment assignment rule. See online appendix of Ashraf, Giné, and Karlan (2009b) for an overview of their observations per treatment arm. The replication study’s version of Table 2(a) is presented in its entirety in Supplementary Materials Table 2.

11. A table comparing the number of observations between the original publication and the replication study is available upon request.

12. As determinants of DrumNet participation is not central to AGK’s argument, we present those results in Supplementary Materials Table 4. We made a similar determination about Tables 6, 8, and 9 from the original publication, which we present in Supplementary Materials Tables 7, 8, and 10.

13. As these findings are not central to the theory of change argument we explore, we relegate them to Supplementary Materials Table 6.

14. A table documenting how we generated the variables is available upon request. Imputation ultimately played a role in recreating the sample sizes, which were partially guided by the original paper.

15. For an example of the difficulty around conversion factors, see the World Bank researchers’ note on determining how best to convert non-standard measurements in a Malawian household survey (World Bank, n.d.). And an alternative approach proposed by Verduzco‐Gallo, Ecker, and Pauw (2014).

16. AGK made these tables available to us upon request. They noted that they created the tables in conjunction with the Tegemeo Institute at Egerton University in Nairobi.

17. Our purpose in running post-evaluation power calculations is to determine required sample sizes. See Hoenig and Heisey (2001) and Lenth (2007) for a wider discussion of issues with post-hoc power analysis. See Wood and Djimeu (2014) for more information on the importance of documenting power calculations.

18. See Wood and Djimeu (2014) for a literature review around subsistence versus cash crop production.

19. We also planned to explore distance to market differences, but privacy concerns prevented AGK from sharing household geographic location data about the sample populations. https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/OCPDXG.