Writing a clinical research article for Pediatric Critical Care Medicine (PCCM) involves reproducing for readers the sequence from ideas to measurements, observations, and evidence, to reasoning and conclusions (1,2). In academic medical writing a Citation (i.e., the act of referring to others) involves three steps: 1) identifying some aspect of other’s work that is useful for readers; 2) quoting a Reference to the source; and 3) providing bibliographic details (3). The references highlight articles that substantiate the topicality of a particular question in the field, the provenance of an idea, and the appropriateness and validity of experimental methods, as well as studies that serve as comparisons to the results (4).

At PCCM, we recommend that a 3,000-word Structured Clinical Research Report include up to 40 references: 10–15 in the Introduction, possibly some in the Methods, leaving 20 for the Discussion (1). Table 1 summarizes the traditional rules when compiling a Reference list and it is a given that for every citation the PCCM author acts as the intermediary between the content of the primary source and our readers.

TABLE 1. - Rules for Citations to References Section Rules for Citations to References Introduction 10–15 references:  Cite references for statements of fact; not necessary if it is common knowledge  Cite your other publications that precede or overlap with the current work  Cite essential background references, but no more than five studies  Cite a recent review if it is a better summary of the field Methods 5–10 references:  Cite references for measurements using an assay, instrument, or questionnaire  Cite references that define diagnostic criteria used in the study  Do not cite a reference for standard or commonly used statistical tests; citing the software package will suffice  Cite a reference for complex or less-known statistical analyses that will be unfamiliar to readers Results No references:  If you think references are needed, they probably belong to the Methods or Discussion sections Discussion 15–20 references (total of 40 in a Clinical Research Report):  Cite references to prior studies that are used as comparison for the current work  Cite references that support possible explanations or implications  Cite references that contextualizes the study
WHAT IS ON THE HORIZON FOR CITATIONS TO REFERENCES?

The International Committee of Medical Journal Editors describes the responsibilities of authors and journals for cited references Table 2 (5). That guidance is followed by PCCM, but changes in 2023 have meant that some recalibration is needed.

TABLE 2. - Responsibilities for Publication of Citations to References Citation to References Responsibilities Reference list review Journals (by some, but not all):  Risk of citation errors in published articles Accuracy of references Authors:  Verify source and content of original article by reading the entire article  Check that no reference cites retracted articles Identification of retracted articles Authors:  Verify identity in PubMed before the proofing stage

In June 2023, two academics produced a narrative and statistical analysis about the impact of fruit and vegetable consumption and physical activity on diabetes risk in adults (6,7). The authors started with the 2015 survey results from the U.S. Centers for Disease Control and Prevention Behavioral Risk Factor Surveillance System. Then, a data-to-paper tool served as an assistant acting between themselves and the artificial intelligence (AI) language model, the Chatbot Generative Pre-trained Transformer (ChatGPT). The final “writing” took an hour and shows how any research dataset with access to literature search engines could be used to generate a semiautonomous, AI, data-to-paper manuscript. There are many questions that now arise for journals when we think about future reviewing and quality control. Does semi-supervised statistical testing in dataset research risk misleading readers (8)? Have ethical principles guiding AI use in health research been followed (9)? Is clinical science at threat (10)? Is there a risk of plagiarism (11)? How are the citations to references chosen (12)?

So far, two critical care journals (Critical Care and Anaesthesia Critical Care & Pain Medicine) have published material about AI and academic medical writing (13–15). One perspective is “…consensus on how to regulate the use of chatbots in scientific writing will soon be required” (13). Such an agreement will take time. However, both critical care journals emphasize that readers should check citations to references, because of two problems. First, ChatGPT has limited access to literature databases, finding works only up to September 2021. (As an illustration, the discussion in the diabetes data-to-paper manuscript does not have literature references after 2019 [7]; find the up-to-date papers in PubMed.) Second, ChatGPT may generate random, false, or nonexistent references, called “AI hallucinations” (16).

Taking all of the above into account, my conclusion is that authors will use ChatGPT and AI as an efficient way of gaining insight into a topic—but writing articles is a step too far until we understand more. I know that university or college students—even high schools—are embracing this approach. However, until ChatGPT plug-ins to databases such as PubMed are developed, there will remain a blind spot to new material, items without Abstracts, and articles that are not Open Access. In response to these possibilities, and problems, the editorial in Anaesthesia Critical Care & Pain Medicine said “…ChatGPT and other language models should make us, researchers, realize that only we can elevate scientific writing” (14). My paraphrase for PCCM is that we can use the opportunity enabled by ChatGPT and raise our standard of writing to a new level.

PCCM IN THE NEW ERA: A FRAMEWORK FOR ENGAGING WITH REFERENCES

This next item in the PCCM Notes, Methods, and Statistics series—already seven to date (1,2,17–21)—is therefore focused on a new framework for engaging with references in our Journal articles. The purpose is to elevate writing for PCCM, and in no way is it a criticism of past publications and our exemplary authors. Rather, we can use the opportunity of the ChatGPT-era to have a new approach that supports a reader-focused, structured engagement of cited references. This framework also adds to the 2023 initiative of the Society of Critical Care Medicine (SCCM) Reviewer Academy (22).

There are five key questions about citations to references that should be thought about by authors submitting to PCCM (Table 3). These questions are an extension to our editorial principles (i.e., Read, Rigor, Relevant, Responsive, and Reach [23]) and the purpose is to help writers produce a more informative manuscript for readers, so that reference content is better appreciated. Briefly, PCCM will be asking authors to include details about the content of references that help readers to understand answers to questions such as: What is? What was? Why now? How? What are the appropriate comparisons and ongoing narrative?

TABLE 3. - Questions to Ask When Including Citations to References in a Pediatric Critical Care Medicine Research Report Questions Citations to References No. 1: Does it describe “What is?” Introduction to prospective work:  References about “what is?” are dated to before the start of cohort enrollment Introduction to retrospective work:  Use any references about “what is?” dated before the start of analysis No. 2: If not “What is?,” then how about “What was?” Introduction to either prospective or retrospective work:  References about rates and prevalence: Let “what is?” vs “what was?” be guided by epidemiology  References about practice: Let “what is?” vs “what was?” be guided by timing of clinical practice guidelines or professional standards  Other references: Let “what is?” vs “what was?” be guided by timing of cohort and not year of publication No. 3: Does it describe “Why now?” Introduction to retrospective work:  Use references that describe the progression in ideas  Refer to any previous or overlapping publications with the cohort No. 4: Describe “How?” methods apply to the cohort Methods in prospective work:  References to a method, score, or definition predates the period of data collection Methods in retrospective work:  References to a method, score, or definition must be available at the time of analysis. In essence, a “what if?” study in a convenience sample No. 5: What comparisons and narrative?” Discussion of comparison in prospective work:  Include references to works since cohort enrollment Discussion of comparison in retrospective work:  Ensure work in historical references are held to the appropriate standard Discussion on extending the narrative in either prospective or retrospective work:  Check post 2021 (i.e., post-Chatbot Generative Pre-trained Transformer range) references in the pediatric critical care literature
QUESTION NO. 1: DOES THE REPORT ADEQUATELY DESCRIBE “WHAT IS?”

The Introduction to a clinical research or brief report in PCCM has two paragraphs (1,2). The first paragraph provides setting and context with enough information for readers to understand the reasoning presented in the second paragraph, which is about aims and hypotheses tested. Those citations to references are used to describe ideas in two types of study.

Prospective Recruitment

Consider, for example, an imaginary preplanned, prospective study carried out January 2020 to December 2022. Here, readers may need to be able to read references that explain the “what is?” of the clinical setting and context. In this example, the references should have been published in 2019, or before; references from 2021 to 2023, although more up to date, are misplaced in this Introduction because that information or ideas conveyed would not have been known at the time of study planning. Citations to 2021–2023 references should appear in the Discussion (see below, Question No. 5).

Retrospective Cohort or Database Research

Now consider a post hoc analysis of a dataset or single-center patient chart review. Here, the citations to references used to explain “what is?” can be from any time up to the timing of the analysis. That is, authors have had an idea about something important now, and it is being tested in a convenience sample. The first paragraph should establish how the idea arose.

QUESTION NO. 2: IF NOT “WHAT IS?,” THEN HOW ABOUT “WHAT WAS?”

When is “what was?” more accurate than “what is?” in the opening paragraph of the Introduction? Here, three possible explanations with added information from the references will help readers of both “prospective” and “retrospective” studies.

First, if a reference is used to describe rate of admission, mortality or morbidity, or prevalence of some disease or complication, then year(s) of the cohort in the reference will be useful in the text. For example, consider the proportional use of PICU beds in the U.S. Pediatric Health Information System dataset of 43 children’s hospitals; admission rates increased during 2014–2019 (24). Now, we also know that U.S. school closures to mitigate the spread of COVID-19 in early 2020 were associated with significant decreases in standardized admission rates to 81 hospitals (in the U.S. Virtual Pediatric Systems LLC database; https://myvps.org) for bronchiolitis, asthma, and pneumonia when compared with 2018–2019 (25). So, when needing to use a citation about respiratory epidemiology, the “what is versus what was?” may now need to be defined in relation to before versus after COVID-19 by also providing for readers the chronology of the cohort in the cited reference. These same principles apply to other diseases or diagnoses.

Second, whether a reference is describing the “what is versus what was?” consequence of a particular practice or management is best defined in relation to before versus after some widely accepted professional standard. For example, let readers know whether the reference about transfusion, or sedation, or pediatric acute respiratory distress syndrome (PARDS) practice in a cohort relate to a period before or after the introduction of the new guidance on transfusion (26), sedation (27), or PARDS (28), respectively.

Third, if the reference is just used to describe some other type of cohort, the distinction between “what is versus what was?” may be down to the chronology of the cohort in the reference. The age of the cohort may be somewhere between 7 and 10 years, but it will depend on the subject area and the uniqueness of available data. Therefore, for readers, a dataset that closed recruitment 2013–2016 may not be important to understanding their PICU population managed in 2023/2024. Remember also that the year of publication of a reference does not determine whether its content can be categorized as “recent” or described as occurring “recently.” What matters are the dates of cohort enrollment. For example, a reference published in 2018, reporting a cohort from 2001 to 2010, is not about recent or contemporary practice. If that is the best information available, the solution for authors is to organize citations to references according to the timeline of the cohorts, rather than by year of publication. Help readers put this information into context by using the references to show how the chronology justifies the study plan, and it may even highlight the gap in what we know about “what is?”

QUESTION NO. 3: CAN YOU DESCRIBE “WHY NOW?”

Typically, the question of “why now?” is answered using text and citations to references that describe a progression in ideas. However, when reporting a “retrospective study” using a carefully curated and previously published clinical research dataset it is important to help readers with the evolution of the research plan, and how a published report has led to the current report.

Consider an imaginary project in which a national acute kidney injury (AKI) dataset has been used to examine an association between fluid balance and the endpoint of, say, ventilator-free days, and the findings are published. Now, the same dataset is used in an unpublished report that examines an association and interaction between the diagnosis (or not) of infection or sepsis, fluid balance, and ventilator-free days. In the Introduction to the new report, our expert PCCM readers will not need a physiological account of sepsis, inflammation, and alveolar capillary membrane fluid leak research. Rather, our readers should be helped with an Introduction that describes findings in the published report and the reasoning that resulted in the work of the current report.

Alternatively, an idea may have arisen from recently published research and the new submission is a test or validation of that idea in a different dataset. The citations to references in this case will cover both “Why now?” and “What was?”

QUESTION NO. 4: CAN YOU PROVIDE DETAILS OF “HOW?” METHODS APPLY TO THE COHORT?

The Methods section describes data collection and handling, methods of measurement, and the definitions of states, terms, and outcomes (1,2). Citations to references about methods used for measurement, and meaning ascribed to certain values, should also provide details of how they apply to the dataset in the report. For example, more explanation will be needed if a measurement has been validated only in older than 1-year-olds but that method is now being used in a PICU population of younger than 6-month-olds. Simply inserting the cited reference will not be helpful to readers who will want to understand potential limitations in the work. It may be the case that professionals working in the field use this methodology in all age groups even without validation; that is fine, readers just need to be told. Definitions also change. For example, during 2021–2023, PCCM provided updates to definitions for PARDS (28), AKI (29), and prolonged mechanical ventilation (30).

Prospective Studies

These studies, whether observational or interventional, have a defined period of recruitment. Citations to references about a method, scoring system, or definition used in the research must predate the period of data collection or cohort recruitment.

Retrospective Studies

Citations to references about a method, scoring system, or definition must have been available before the time of analyzing a retrospective cohort. Readers will also need to know that the work is a post hoc examination, or “what if?” study in a convenience sample.

QUESTION NO. 5: WHAT ARE THE APPROPRIATE COMPARISONS AND ONGOING NARRATIVE?

The Discussion provides an opportunity to present two or three main observations in the context of the “what is?” or “what was?” literature identified in the Introduction, as well as any newer reports that have been published. In this case, there are two writing formats that are helpful.

Making Comparisons

An observation is often generalized using a percentage. However, when making comparisons with percentages in cited references, it is helpful for readers to know that any potential differences being suggested have been checked with a statistical test of proportions. Invariably, there is no difference, which may suggest the new observation is no worse or no better than prior data.

The other aspect of making comparisons is when considering the importance of some standard of care against which a dataset has been evaluated; it is useful to know that the cited references are appropriate and informative. As an example, consider the 2017 SCCM and American Society for Parenteral and Enteral Nutrition (ASPEN) guidelines for the provision and assessment of nutrition support therapy in the pediatric critically ill patient (31). The guidelines recommended a minimum protein intake of 1.5 g/kg/d (moderate quality of evidence, strong recommendation). Now, imagine a study cohort from 2018 or later; it is certainly appropriate to inform readers of an analysis comparing protein intake in this cohort with the 2017 SCCM/ASPEN standard (31). However, a comparison in protein intake between the new cohort and a reference cohort predating 2017 has no meaning. The authors of the pre-2017 work (or cohort) cannot be expected to have met a standard that did not exist at the time of publication or patient care.

Extending the Narrative

The overall goal of PCCM is to promote knowledge that accelerates change in our discipline and impacts and improves patients’ lives in our field (23). The articles in PCCM are published for clinical practitioners, for the SCCM Pediatric Section and the World Federation of Pediatric Intensive and Critical Care Societies, and for researchers. In 2020, PCCM published 30% of all original articles in pediatric critical care, and almost all the editorials and commentaries (32).

In this context, paragraphs 2 to 4 of the Discussion should contain evidence of extending the narrative in our literature (1). This means checking search engines for references after 2021, (i.e., beyond the current scope of ChatGPT). It also means that to substantiate the topicality of a particular question in the field, and likely importance to PCCM readers, post-2021 references from the main pediatric critical care literature sources should be evident in the citations (e.g., PCCM, Frontiers in Pediatrics, Journal of Pediatric Intensive Care, Critical Care Medicine, and Pediatric Pulmonology [33]). These requirements are not intended to limit academic freedom or autonomy, but to help authors more efficiently identify whether PCCM has the right readers for the work submitted. It may be that the work is better suited to a more specialist journal. Alternatively, the report may have been written for another journal (and readership); now, it just needs to be rewritten for PCCM readers (1,2).

CONCLUSIONS

In this ChatGPT-era of AI-generated reference lists, readers of PCCM can be reassured that the Journal will rise to the challenge to “elevate scientific writing” (14). PCCM’s new framework for engaging with citations to references is now an amalgamation of rules, responsibilities, and questions (Tables 1–3). My three goals are, first, to encourage authors to provide details from cited references that cover the essential questions about What is? What was? Why now? How? What are the appropriate comparisons and ongoing narrative? Second, to ensure that PCCM will continue to seek out expert reviewers for submitted reports who will help us with the most contemporary knowledge. Finally, to use the pre-acceptance editorial review to fact check the reference list for accuracy, which will guard against publishing citations to retracted material and the risk of AI hallucination (34).

ACKNOWLEDGMENT

Chatbot Generative Pre-trained Transformer was not used to author any part of this PCCM Notes, Methods, and Statistics article about the new era of artificial intelligence-generated citations to references.

REFERENCES 1. Tasker RC: Writing for PCCM: The 3,000-word structured clinical research report. Pediatr Crit Care Med. 2021; 22:312–317 2. Tasker RC: Writing for PCCM: Instructions for authors. Pediatr Crit Care Med. 2022; 23:651–655 3. Bahadoran Z, Mirmiran P, Kashfi K, et al.: The principles of biomedical scientific writing: Citaton. Int J Endocrinol Metab. 2020; 18:e102622 4. Browner WS: Publishing and Presenting Clinical Research. Third Edition. Philadelphia, PA, Lippincott Williams & Wilkins, 2012 5. International Committee of Medical Journal Editors: Manuscript Sections. References. Preparing a Manuscript for Submission to a Medical Journal. Available at: https://www.icmje.org/recommendations/browse/manuscript-preparation/preparing-for-submission.html#g. Accessed July 21, 2023 6. Conroy G: Scientists used ChatGPT to generate an entire paper from scratch–but is it any good? Nature. 2023; 619:443–444 7. Kishony R, Ifargan T: The Impact of Fruit and Vegetable Consumption and Physical Activity on Diabetes Risk Among Adults: Data to Paper. 2023. Available at: https://t.co/DsEZfg0JQp. Accessed July 23, 2023 8. Gupta A, Bosco F: Tempest in a teacup: An analysis of p-hacking in organizational research. PLoS One. 2023; 18:e0281938 9. Al-Hwsali A, Alsaadi B, Abdi N, et al.: Scoping review: Legal and ethical principles of artificial intelligence in public health. Stud Health Technol Inform. 2023; 305:640–643 10. Tools such as ChatGPT threaten transparent science; here are our ground rules for their use. Nature. 2023; 613:612 11. Gao CA, Howard FM, Markov NS, et al.: Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. NPJ Digit Med. 2023; 6:75 12. Hutson M: Could AI help you to write your next paper? Nature. 2022; 611:192–193 13. Salvagno M, Taccone FS, Gerli AG: Can artificial intelligence help for scientific writing? Crit Care. 2023; 27:75 14. Pozencwajg S, Kantor E: Elevating scientific writing with ChatGPT: A guide for reviewers, editors, and authors. Anaesth Crit Care Pain Med. 2023; 42:101209 15. Blanchard F, Assefi M, Gatulle N, et al.: ChatGPT in the world of medical research: From how it works to how to use it. Anaesth Crit Care Pain Med. 2023; 42:101231 16. De Cassai DB: Concerns regarding the uncritical use of ChatGPT: A critical analysis of AI-generated references in the context of regional anesthesia. Reg Anesth Pain Med. 2023 Jun 27. [online ahead of print] 17. Tasker RC: PCCM narratives, letters, and correspondence. Pediatr Crit Care Med. 2021; 22:426–427 18. Horvat C: Statistical note: Confounding and causality in observational studies. Pediatr Crit Care Med. 2021; 22:496–498 19. Toh TSW, Lee JH: Statistical note: Using scoping and systematic reviews. Pediatr Crit Care Med. 2021; 22:572–575 20. Bartman T, Brilli RJ: Quality improvement studies in pediatric critical care medicine. Pediatr Crit Care Med. 2021; 22:662–668 21. Kreif N: Learning from an association analysis using propensity scores. Pediatr Crit Care Med. 2021; 22:1088–1092 22. Alexander PMA, Aslakson RA, Barreto EF, et al.: The reviewer academy of the Society of Critical Care Medicine: Key principles and strategic plan. Crit Care Med. 2023; 51:1111–1123 23. Tasker RC: Pediatric Critical Care Medicine 2021: The five Rs. Pediatr Crit Care Med. 2021; 22:1–2 24. Heneghan JA, Rogerson C, Goodman DM, et al.: Epidemiology of pediatric critical care admissions in 43 United States children’s hospitals, 2014-2019. Pediatr Crit Care Med. 2022; 23:484–492 25. Rogerson CM, Lin A, Klein MJ, et al.: School closures in the United States and severe respiratory illnesses in children: A normalized nationwide sample. Pediatr Crit Care Med. 2022; 23:535–543 26. Nellis ME, Karam O, Valentine SL, et al.; Pediatric Critical Care Transfusion and Anemia EXpertise Initiative—Control/Avoidance of Bleeding (TAXI-CAB), in collaboration with the Pediatric Critical Care Blood Research Network (BloodNet), and the Pediatric Acute Lung Injury and Sepsis Investigators (PALISI) Network: Executive summary of recommendations and expert consensus for plasma and platelet transfusion practice in critically ill children: From the Transfusion and Anemia eXpertise Initiative–Control/Avoidance of Bleeding (TAXI-CAB). Pediatr Crit Care Med. 2022; 23:34–51 27. Smith HAB, Besunder JB, Betters KA, et al.: 2022 Society of Critical Care Medicine clinical practice guidelines on prevention and management of pain, agitation, neuromuscular blockade, and delirium in critically ill pediatric patients with consideration of the ICU environment and early mobility. Pediatr Crit Care Med. 2022; 23:e74–e110 28. Emeriaud G, Lopez-Fernandez YM, Iyer NP, et al.; Second Pediatric Acute Lung Injury Consensus Conference (PALICC-2) Group on behalf of the Pediatric Acute Lung Injury and Sepsis Investigators (PALISI) Network: Executive summary of the second international guidelines for the diagnosis and management of pediatric acute respiratory distress syndrome (PALICC-2). Pediatr Crit Care Med. 2023; 24:143–168 29. Deep A: The ever-changing paradigm of acute kidney injury definition: Does duration and trajectory of acute kidney injury add another dimension? Pediatr Crit Care Med. 2021; 22:676–679 30. Colleti J Jr, Azevedo RT, de Oliveira Caino FR, et al.: Prolonged mechanical ventilation in children: Review of the definition. Pediatr Crit Care Med. 2021; 22:e588–e593 31. Mehta NM, Skillman HE, Irving SY, et al.: Guidelines for the provision and assessment of nutrition support therapy in the pediatric critically ill patient: Society of Critical Care Medicine and American Society for Parenteral and Enteral Nutrition. Pediatr Crit Care Med. 2017; 18:675–715 32. Tasker RC: 2021 in review. Pediatr Crit Care Med. 2021; 22:1009–1010 33. Tasker RC: 2022 in review. Pediatr Crit Care Med. 2022; 23:961–963 34. Bakker C, Boughton S, Faggion CM, et al.: Reducing the residue of retractions in evidence synthesis: Ways to minimise inappropriate citation and use of retracted data. BMJ Evid Based Med. 2023 Jul 18. [online ahead of print]