This essay tested an application of the standard integrity to a case: JBI's response to Bergdahl. We, therefore, have two levels of result: substantive and methodological. These are discussed in this order as the substantive results are relevant in presenting our methodological results.

3.1 Substantive results1

To what extent does JBI recognize bergdahl's definitions of key terms?

Sound argument requires participants to properly recognize the claims each are making. This capacity, as noted above, is consistent with JBI's expectation that reviewers be able correctly to understand the texts they examine. In the exchange between Bergdahl and JBI the terms meta-aggregation, meta-synthesis, and the qualitative sciences figure prominently so are used to test JBI's recognition of Bergdahl's definitions of those terms.

JBI is consistent with Bergdahl in recognizing that meta-aggregation is data-grounded/inductive; based on interpretation; and aims to generate a common objective meaning. JBI misrecognizes Bergdahl, however, on every point that is relevant to practice. Where Bergdahl presents meta-aggregation as reducing findings; sorting away explanations; stripping meaning; and of no use to practice, JBI works with a meta-aggregation that synthesizes findings; generates explanatory statements; inclusively recognizes the meaning of findings; and provides useful guidance to practice.

JBI is consistent in recognizing meta-synthesis as a synthesis of qualitative findings. However, as JBI do not provide a substantive discussion of their understanding of meta-synthesis, it is not possible to test their agreement with the detailed description proposed by Bergdahl.

JBI is consistent with Bergdahl in their recognition that the qualitative sciences aim to: (a) clarify meaning and (b) serve understanding of findings. JBI, however, silently rejects several aspects of Bergdahl's definition. JBI, for instance, proposes that qualitative sciences: (a) facilitate general conclusions to serve understanding of findings (emphasis “general”) and (b) consider findings, which is not consistent with Bergdahl who sees them to (emphasis “consider”): (a) create specific theory within context to serve interpretation and understanding of findings (emphasis “specific”) and (b) construct and critically question instead of merely consider findings (emphasis “construct” “critically question”).

Where JBI provides sufficient data, it appears that JBI does not adequately recognize key claims made by Bergdahl.

To what extent does the content of JBI's rebuttals match the content of the matching critiques?

To what extent do JBI's rebuttals reference recognizable backing?

One of the capacities required to test the congruence of a qualitative study is the ability to determine which components are related, how they are related; and how well they fit each other. The parallel in argumentation is that rebuttals must relate and respond to their intended critique. Taking the next step, the selection of each of the components used in a qualitative study must be acceptably justified. As is the case in review of studies, in this essay both logic and evidence provide acceptable backing.

JBI contends that it “…is unclear why Bergdahl has chosen to contrast meta-aggregation solely with meta-ethnography” (p. 2) and backs this argument with the assertion that the “critique provided by Bergdahl is overshadowed by a lack of clarity on the terms used across the qualitative synthesis community” (p. 3). Bergdahl, however, did not solely contrast meta-aggregation with meta-ethnography and she argued that researchers “must also take a critical stance towards the methods used” (p. 5). In this case, JBI has neither correctly recognized the relevant components of Bergdahl's critique nor provided recognizable backing.

JBI contends that “The claim that the endpoint of a qualitative evidence synthesis ends with a theory positions Bergdahl with the general belief that only one philosophical paradigm is appropriate for all types of qualitative syntheses” (p. 2). JBI backs this claim with the assertion that “Methodological diversity should be encouraged to acknowledge and meet the broad range of knowledge needs associated with scholarship and theory generation” (p. 2) and backs this with the assertion that “this perspective is aligned with pragmatism” (p. 3).

Bergdahl does claim that an aim of the qualitative review is the theory which is justified by the assertion, never countered by JBI, that theory is a necessary condition for the interpretation of results. In this response, JBI binds advocacy for theory creation to the belief that only one philosophical paradigm is appropriate for all types of qualitative synthesis, and then falsely attributes this position to Bergdahl. At no point does JBI: (a) identify the paradigm trapping Bergdahl; (b) explain how advocacy for theory generation necessarily traps scholars within a single paradigm; or (c) help the reader understand how JBI's immediate endorsement of pragmatism is anything other than entrapment within a single paradigm. Furthermore, the “alignment” deployed to back the legitimacy of JBI's aggregation requires that the target to which their practice is aligned—pragmatism—is itself coherent, compatible with the goals of review, and sympathetic to atheoretical aggregation across heterogeneous studies. The same source cited by JBI to back endorsement of pragmatism, Cherryholmes (1992), states that “there are many versions of pragmatism” (p. 13). This would suggest that there may be no coherent referent to support alignment. Cherryholmes goes on to assert that the revolutionary contribution of pragmatism, which distinguishes it from scientific realism, is a commitment to providing the reader with a diversity of mutually distinct means. These means serve to reach their stated objective (e.g., improving literacy) that may then be assessed according to values, esthetics, politics; and social and normative preferences (e.g., a desirable community). In this connection, Cherryholmes attributes to Dewey the statement "Not everything that works is desirable, not every belief that is 'true' is to be acted upon" (p. 14). The declared interest of this pragmatism is to afford policy makers with a diversity of effective options that vary in their coincident effects. It is not clear in JBI's text how retention of the detail required to predict coincident effects is compatible with the decontextualizing aggregation they practice, nor is it clear that the purpose of JBI review is to identify mutually distinct equally effective means from which practitioners may choose based on preferred coincident effects.

JBI has thus incorrectly recognized and invented portions of Bergdahl's critique, as well as provided backing that either fails to rise to the level of refutation, or merely contradicts the point they are attempting to make.

With respect to meta-synthesis, JBI states that they “…agree that the transfer of these presuppositions to methodology for the health professions is worthwhile. However, we are concerned with the notion that it is the only perspective of value for the health care sector” (p. 2). Bergdahl does not claim that meta-synthesis is the only valuable perspective for the health-care sector. She states that “meta-synthesis could serve to re-interpret, compare and translate different qualitative studies, using a different conceptual apparatus, into a consolidated knowledge of fundamental importance to nursing care practitioners” (p. 7). Bergdahl goes on to state that all results from meta-research always require epistemological discussion—especially within the health-care sector—for the results to acquire value. JBI misreads Bergdahl (“Meta-synthesis could serve…” [[emphasis “could”]) and JBI does not address Bergdahl's claim that meta-research always requires epistemological consideration. Taking the next step, none of the arguments provided by JBI are accompanied by recognizable backing. For example, to back the argument that sensitivity to theory and methods is not required, JBI states that the standards suggested by Bergdahl, sensitivity to theory and methods by the reviewer, are not appropriate:

Theory building as the primary pursuit of knowledge synthesis requires caution when extrapolating to practising health professions whose primary interests are in the delivery of care … expert clinical practice is founded on theory and evidenced through ‘thinking in action.' (p. 2)

The claim that the primary purpose meta-synthesis is theory building, which JBI attributes to Bergdahl, cannot be found in her intervention, and the assertion that thinking in action is what counts as sound evidence for clinical practice is not accompanied by a description or backing. JBI has thus incorrectly recognized and invented portions of Bergdahl's critique. They then fail to provide sound argument or recognizable backing to support their response.

JBI states that the “adoption of a pragmatist perspective for meta-aggregation is compatible with the epistemological, philosophical, and ethical traditions of qualitative research and is therefore justified as a proper scientific method” (p. 2). This assertion is presented as a rebuttal of Bergdahl's argument that “aggregation seems incompatible with the qualitative tradition” (p. 2), which is backed by the cited assertions both that “leading voices in the interpretivist qualitative research tradition are explicitly against generalization” (p. 4) and that JBIs approach to meta-aggregation “stands in contradiction to what is understood as good, post-positivistic, scientific practice” (p. 4) since “interpretive methods reject the notion of an objective truth that can be found, and verified, by empirical methods” (p. 4). Furthermore, Bergdahl does not claim that meta-aggregation is not a proper scientific method. Her claim is that it is “a different approach that is in contrast to meta-synthesis and should not be considered a form of meta-synthesis. If that is correct, meta-aggregation cannot be justified as a proper scientific practice in terms of being a form of meta-synthesis” (p. 1).

JBI does not correctly read Bergdahl where she states that “meta-aggregation seems incompatible with the qualitative research traditions” (emphasis “seems”) and that “meta-aggregation is not a proper scientific method in terms of being a form of meta-synthesis” (emphasis “in terms of being a form of meta-synthesis") and JBI fails to address the supporting discussion of postpositivist science. Even granting that JBI's reading was adequate, they would not have refuted Bergdahl. JBI does not explain why adoption of a pragmatist perspective for meta-aggregation is compatible with the qualitative research tradition as described by JBI, with reference to postpositivism and construction, so their response can only be recognized as a counter. JBI has not correctly understood or fully addressed Bergdahl, and the responses made by JBI are not adequately backed.

With reference to a study mentioned by Bergdahl, Lamb et al. (2008) (hereafter Lamb) as exemplifying how meta-aggregation “moves knowledge backward, from specific and actionable knowledge to broad generalization” (Bergdahl, p. 5), JBI states “We credit Bergdahl for the comment that the example review cited from Lamb and colleagues would have had different results if it were conducted from an interpretive perspective. We doubt, however, whether it should have had different results” (p. 2). Bergdahl does not claim that the example review of Lamb would or should have had different results if it were conducted from an interpretative perspective (emphasis “would,” “should”). Instead, Bergdahl claims that the example review of Lamb could have had different results if it were conducted from an interpretive, thus meta-synthesis perspective (emphasis “could”). Using logic, Bergdahl suggests that meta-synthesis could produce something more than generalization because it is sensitive to theory and context. This supports Bergdahl's argument that meta-synthesis could “serve to re-interpret, compare and translate different qualitative studies, using a different conceptual apparatus, into a consolidated knowledge of fundamental importance to nursing care practitioners” (p. 7) (emphasis “could”). JBI misreads the claims made by Bergdahl concerning the review paper of Lamb they fail to see the difference between could and should and they fail to address the arguments Bergdahl uses to support her claims. Even if JBI correctly read Bergdahl on Lamb, their response would only amount to a contradiction.

JBI asserts that in their approach to meta-aggregation “findings are grouped based on similarity of meaning (as Bergdahl rightly highlights) into categories, from which synthesised findings are constructed. Bergdahl takes issue with this concept of ‘similarity in meaning’; however, that phrase does not just mean simply matching like with like” (p. 3). JBI is mistaken in their interpretation of Bergdahl's “similarity in meaning.” Bergdahl does not say that “similarity in meaning” in meta-aggregation is the matching of like with like. She likens the JBI approach to “generating findings using the highest common denominator in mathematics: the more cases one adds, the fewer common factors there are going to be. Thus, very few and common abstract attributes are left, and any context-dependent attribute or property is sorted away” (p. 4). Bergdahl does claim this search for “similarity in meaning” in meta-aggregation to be a weakness of the method as interpretation “towards common meaning will naturally evolve from specific terms to less specific and more general, less informative and abstract terms” (p. 4), which brings the risk that the searching for common meaning advocated by JBI results in “neglecting, crucial contradictory findings in specific contexts.” As such “results run the risk of becoming meaningless abstractions that do not say anything profound about how things work” (p. 4).

JBI has not correctly read the claims Bergdahl, addressed all important supporting arguments or provided adequate backing for their own claims.

3.2 Methodological results

While our substantive results demonstrate that: those who conduct review of qualitative research may declare standards that are relevant for consideration of their contributions to debates over methods; and that these standards allow to produce results that are possible to interpret and defend. We did encounter several challenges.

In our analysis, we started from JBI's rebuttals and, then selected for each rebuttal from Bergdahl only those data that were immediately relevant. There were several dimensions to Bergdahl's critique that JBI did not mention. This limited the claims we could make to those individual rebuttals and not to the extent to which JBI answers the entire critique.

In our examination of both Bergdahl and JBI, we found that components of claims were not consistently clustered. Identifying all the relevant components of a claim required careful repetitive reading of the entire text. A side-effect of the scattering of components was that the relationships between components were not clear at times. While we did resolve this ambiguity, the path we chose, of making very charitable assumptions, biases our results.

We found that JBI's rebuttals at times did not structurally match Bergdahl's critiques. This meant, for instance, that while Bergdahl provided evidence or logic to back all the arguments that supported a particular critique, JBI's response at times missed branches, added branches, or operated at a different depth. The lack of structural identity between the critique and response introduced substantial missing data. While the third research question, which queried the presence and nature of backing, was sensitive to depth (i.e., are claims backed?), it was not sensitive to missing branches (i.e., Are all component claims in Bergdahl addressed?) so, our method does not provide comprehensive coverage of data.

Finally, our analysis focuses entirely on the behavior of JBI. We were constrained to the examination of JBI, as we could not justify using JBI's standards to assess Bergdahl and as we could not find an equivalent set of standards that covered Bergdahl's behavior. In this case, where we found standards to cover only one party's practice and where the practice of that party was found to be severely wanting, and despite several methodological choices that bias results in JBI's favor, our efforts may appear biased against JBI.