Harman and Lorandos’ false critique of Meier et al.’s family court study

Figures & data

Table 1. Response to Harmon & Lorando’s List of “30 Methodological Flaws” in FCO Study. NB: Items with an asterisk are those which are variations on a single theme – lack of information. This is not a flaw in methodology.

ALLEGED FLAW	REBUTTAL
*1 Hypotheses were not explicitly detailed prior to the presentation of the results	False: All hypotheses were in Proposal to NIJ, which was subjected to peer review before approval; did not belong in Final Summary Overview.
2 Several hypotheses were tested without any theoretical or practical rationale detailed in the introduction.	False: All rationales were described in Proposal; theoretical framework also contained in published article.
*3 Failure to provide information about the different search engines and databases used to “test” search strings	False: Contained in archived documentation.
*4 Failure to provide in the paper the database search string of “over 10 lines of search terms” that were “constructed and applied” (p. 5.a)	False: Harman and Lorandos accessed the search string (which was posted) but decided it was too unwieldy for them to utilize. It did not belong in the summary overview, which doesn’t have methodological details.
*5 No information was provided about the coders who “triaged” the initial search cases.	Minor. Coders were law graduates. Harman and Lorandos obscure the fact that their coding was performed not only by undergraduates but also by members of Lorandos’ Psych Law office.
6 Cases in which the parents were of the same sex were excluded from the database	Not a flaw. These cases are in the database, but hypotheses focused on mothers and fathers; therefore, it was necessary to exclude these cases to avoid confounding.
7 Only cases that had three core outcomes were included in the sample: crediting of abuse, custody outcomes, and “wins.”	False. All electronically published cases with abuse or alienation claims over 10-years were included in the study; they were not selected by outcomes, but by allegations. In our first round of analyses those are the outcomes we analyzed.
8 Cases with “third party victims” such as a new or old romantic partner, “mutual abuse” cases, “nonspecific” abuse claims, and “AKA” cases, which were cases involving negative parenting behaviors, which are “similar” in analysis to PA cases, were excluded.	False. As report makes clear, these cases are in the dataset but excluded from the first tranche of analyses, which compare cases with alienation crossclaims to cases without such crossclaims. They are included in the “all abuse” analyses which look at outcomes in all abuse cases. AKA cases are analyzed separately as they could be argued not to be alienation cases.
*9 No information was provided about the coders who excluded cases at the inclusion or exclusion stage	Duplicative. See answer to #5 above.
*10 An expanded data set of all abuse claims was created, without details of the search terms or search engine used.	False. Same dataset, different filters to answer different hypotheses, as detailed in archived documentation.
11 Extrafamilial abuse cases were included in the “all abuse” data set but were excluded in the “analytic data set.”	Not a flaw. Reasoning explained in posted documentation.
*12 The degree of overlap between the “all abuse” and “analytic database” was not described.	False. The overlap is irrelevant; the two “datasets” are merely different filters applied to the same dataset.
*13 There was a lack of clarity about how “non-alienation” and “pure alienation” cases were identified and how the final numbers for each were determined.	False. The Coding Manual (appendix to User Guide) details how each category/code is defined.
14 Trial-level cases were included in the full data set	Not a flaw. Since the Study analyzes trial court decisions, inclusion of published opinions which had not been appealed provides a valuable basis for comparing the opinions that were and were not appealed, and renders the overall dataset a more accurate reflection of court practices.
*15 Details about the coders of the data sets are not provided	Duplicative. See answer to # 5 above.
*16 Details about how coding discrepancies were resolved are not provided, nor are interrater reliabilities	Not a flaw. This information was reported to the funder in regular Progress Reports and did not belong in the Final Summary Overview. Interrater reliability was not necessary where coding was merely objective reporting from court opinions; Harman and Lorandos concluded the same for their own study.
*17 No details about the 45 specific codes and options for them that were applied to the cases were provided in the paper.	False. All of the codes, definitions, and how they were applied is detailed in the extensive Coding Manual contained in the archived and posted User Guide.
*18 Readers are referred to “Appendix B” (Meier et al., 2019, p. 8) to obtain details about the codes used.	Not a flaw. Appendix B was included in the User Guide posted by the NIJ Archives.
*19 There was a lack of clarity in the paper concerning how codes were defined.	False/duplicative. See answer to # 17 above.
20 There did not appear to be consistent application of codes across the two datasets.	False. There is only one dataset, with two filters.
21 Change in custody is consistently worded throughout the report as whether the mother or father “loses custody” after making an abuse allegation.	Not a flaw. Custody loss and custody change are used to describe the same thing, which is explicitly defined in the report: removal of custody from a parent who started with primary physical care of the children. We looked at how this outcome was correlated with different variables, not only abuse allegations.
22 Readers are referred to “Appendix C” (Meier et al., 2019, p. 8) to obtain details about the analytic plan and statistical codes.	Not a flaw. Appendix C is part of the User Guide posted by the NIJ Archives.
*23 Control variables were not clearly described or justified	Not a flaw. No control variables were used for the frequencies and descriptive statistics, nor the odds ratios, which were based on the frequencies.
24 Gender is reported to be included as a control variable (p. 7) in the statistical models.	Minor/Not a flaw. Regression analyses with gender as a control variable were mentioned in passing as part of the report to NIJ but not reported in any specificity because a future paper will provide detailed regression analyses and results.
*25 The types of variables used in the model (e.g., continuous, dichotomous, ordinal) were not described.	Not a flaw. Such variable characteristics were not reported because they are not used for frequencies and odds ratios. See answer to #26 below.
*26 The ways variables were dummy-coded was not clearly described.	False. All coding is clearly labeled in the archived documentation. Because regression analysis was not used to analyze the hypotheses, this critique is not relevant.
*27 No model fit statistics were provided for any of the logistic regressions that were conducted	Not a flaw. Not relevant to descriptive statistics and odds ratios. See answers to 23–26 above.
*28 Adjustments to p values for multiple comparisons (e.g., p< .01) and effect sizes were not provided	Minor. Given the census nature of the data, a p-value of .05 represents a conservative approach to significance testing. This level was referenced in the report. Because regression analysis was not employed, there are no effect sizes to report.
29 Odds ratios were described as likelihoods (e.g., “mothers are 2.48 times as likely to lose custody when an evaluator is present than not”; p. 24) when odds and likelihood are not the same concepts.	Minor. It is conventional to use “likelihood” in connection with odds ratios; a convention Harman and Lorandos also adopt in their article.
30 Proportions of cases were continually reported as “rates.”	Minor. “Relative frequencies” may be more accurate; however, use of the word “rates” does not mislead readers, and is not a methodology flaw.

Meier, J., Dickson, S., Rosen, R., O’Sullivan, C., & Hayes, J. (2019). Child custody outcomes in cases involving parental alienation and abuse allegations. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3448062

 Google Scholar