Remember me
ARTICLE
Year : 2023 | Volume
: 21
| Issue : 1 | Page : 30-46
Development and Psychometric Testing of the Student Learning in Emergencies Checklist (SLEC): Measuring Promotors of Academic Functioning and Wellbeing in Palestinian Youth Affected by War and Conflict
June T Forsberg1, Carly Tubbs Dolan2, Jon-Haakon Schultz1
1 The Arctic University of, Norway
2 NYU Global TIES for Children, New York, USA
Correspondence Address:
June T Forsberg
UiT The Arctic University of Norway, Faculty of Health Sciences, Sykehusveien 44, 9019 TROMSØ
Norway
Source of Support: None, Conflict of Interest: None
DOI: 10.4103/intv.intv_17_22
Youth living with war and ongoing conflict are frequently exposed to potentially traumatic events and cumulative stress which can result in temporarily impaired academic functioning and reduced wellbeing in school. In Palestine, the Ministry of Education has included psychosocial support programmes in the schools for the purpose of reducing stress reactions and improving school functioning amongst students. The Student Learning in Emergencies Checklist (SLEC) was developed in a psychometric testing study for monitoring and evaluating school-based psychosocial support in the Palestinian context—self-perceived academic functioning and wellbeing in particular. The study included 789 Palestinian students (12–16 years of age) living in Gaza, the West Bank and East Jerusalem, using an instrument that monitors five theoretical constructs considered to promote school functioning in emergencies. The analysis revealed a different factor structure than hypothesised: 1) safety and adaptability, 2) emotion regulation, 3) school support, 4) family support and 5) wellbeing and hope. The SLEC satisfies monitoring and evaluation purposes for fieldwork regarding programme planning, design and evaluation. Further, it appears well-suited for use by teachers for raising awareness of the need for and guiding their delivery of psychosocial support. The present study discusses practical implications and limitations of the SLEC.
Keywords: education in emergencies, monitoring and evaluation (M&E), psychometrics, psychosocial support, traumatic stress
Key implications for practice:
MHPSS practitioners should monitor and evaluate with context specific and evidence-based instruments.MHPSS instruments are not only necessary for programme quality control but can also be helpful for teachers to assess the need for and their students’ perceived experience of MHPSS.NGOs and researchers must collaborate closely in development of new instruments to secure valid research methods and field experience from the specific context. Introduction
Growing up with war or in conflict-affected areas has major negative impacts on the wellbeing of children and young people. These groups are more vulnerable than adults and more likely to be impaired and severely affected by traumatic exposure and cumulative stress (Norris et al., 2002). Given their lack of life experience and not fully developed cognitive capacity, this affects their ability to handle acute helplessness or comprehend what is happening around them. This can cause loss of perceived safety, increasing their need for support from significant adults in their lives. Moreover, exposure to terrifying events and cumulative stress increases the risk of cognitive impairment, academic difficulties and social-emotional-behavioural problems (Perfect et al., 2016). These groups have been shown to be at high risk of developing symptoms associated with posttraumatic stress disorder (PTSD) (Charlson et al., 2019). A review of research conducted in Palestine found PTSD in up to 70% of the young population (Dimitry, 2012).
The negative impacts of traumatic exposure and cumulative stress on cognition are increasingly recognised (Blair et al., 2013; Bücker et al., 2012; Fernando et al., 2010; Lupien et al., 2018; Stene et al., 2019; Strøm et al., 2016) and several studies have shown that symptoms associated with PTSD impair attention span, memory capacity, executive functioning and speed of information processing (Malarbi et al., 2016; Perfect et al., 2016). As these cognitive functions are also important in learning, one consequence of traumatic exposure and cumulative stress is often impaired school functioning (Perfect et al., 2016).
School Functioning and Support in Emergency Areas
School functioning is commonly defined by three components: (a) academic functioning, (b) wellbeing at school and (c) academic performance. “Academic functioning” refers to the cognitive process of learning, whereas “academic performance” concerns the extent to which the student has achieved his or her academic and educational goals. “Wellbeing at school” is defined as feeling safe and secure during the school day, as well as having positive peer relationships (Wentzel, 2003; Wentzel, 2009).
In emergencies, schools are important not only for educational purposes: they are also essential arenas for mental health and psychosocial support (MHPSS). The Interagency Standing Committee Reference Group on MHPSS in Emergency Settings defines MHPSS as a type of local or support outside the health-care system that aims to protect and promote psychosocial wellbeing and/or prevent or treat mental disorders (IASC, 2007). Schools can play key roles in both preventive and post disaster efforts, providing support and care to reduce negative reactions and cumulative stress, as well as enhancing learning outcomes and wellbeing. As mental health professionals (like psychologists and counsellors) are often lacking (Marie et al., 2016), teachers represent a more available resource. They can help their students stabilize and recover after stressful events and have been shown to be a valuable asset in large scale school-based MHPSS programmes (Forsberg & Schultz, 2022).
There exist various school-based MHPSS programmes for the prevention of symptoms and the treatment of distressed children and youth exposed to war and war-related terrifying events. Systematic reviews and meta-analysis have explored the effect of programmes provided after disasters and terrorism (Fu & Underwood, 2015; Pfefferbaum et al., 2014), in war and conflict (Jordans et al., 2016), for reduction of PTSD symptoms (Rolfsnes & Idsoe, 2011), and also what mechanisms influence psychosocial wellbeing and learning outcomes (Lasater et al., 2022). Overall, such interventions show promising results on mental health indicators and PTSD symptoms. Mechanisms for strengthening coping skills, the ability to regulate negative emotions and the quality of interpersonal relationships are found to improve learning outcomes and wellbeing.
Essential Elements of Mass Trauma Intervention as Promotors of School Functioning
The five essential elements of immediate and midterm mass trauma intervention presented by Hobfoll et al. (2007) are widely accepted, empirically supported principles that are used to inform prevention efforts, both immediately after terrifying events and up to 3 months thereafter. These principles are (1) reestablishing a sense of safety, (2) promote calming, (3) promote self- and community-efficacy, (4) connectedness and (5) establish hope for the future. These five are included in the guidelines for best practices in MHPSS after potential traumatic events, such as Psychological First Aid (Brymer et al., 2013), European Network for Traumatic Stress Guidelines (Bisson et al., 2010) and Skills for Psychological Recovery (Berkowitz et al., 2010). They are also used in school-based delivery of post disaster psychosocial care, as with the Better Learning Programme (BLP) (NRC, 2017).
Given the known impacts of psychological distress on learning abilities and wellbeing (Lasater et al., 2022; Perfect et al., 2016), the five elements developed by Hobfoll et al. (2007) can also be described as promotors of school functioning. This finds support in several educational and psychological theories (e.g., cognitive-affective theory of learning, theory of self-efficacy); and there is strong empirical evidence of the importance of connectedness/social support, and wellbeing as protecting factors for PTSD, learning abilities and performance in school (Aydin et al., 2016; Bandura, 1997; Becker & Luthar, 2002; Hascher, 2010; Knörzer et al., 2016; Münzer et al., 2017; Richardson et al., 2012; Robbins et al., 2004; Vedder et al., 2005; Wentzel et al., 2017).
The elements of safety, calming and hope can all be seen as promotors of school functioning based on the cognitive-affective theory of learning (Hascher, 2010; Knörzer et al., 2016). This theory holds that the physiological reactions in the brain that occur when negative emotions (e.g., fear, anxiety, depression and acute/cumulative stress) are experienced inhibit the cognitive processes associated with learning. By contrast, positive emotions like hope, gratitude, reward and motivation release neurotransmitters such as serotonin, dopamine and endorphins that facilitate the same cognitive processes (Hascher, 2010; Lupien et al., 2018). Moreover, elevated levels of stress and feelings of hopelessness for the future can affect student motivation and behaviour in connection with educational tasks, academic learning/performance and general functioning in school (Hascher, 2010). A recent study found that after participating in a school-based MHPSS programme (BLP) (NRC, 2017), a group of conflict-affected students in Gaza self-reported improved academic functioning, wellbeing and academic performance, as measured in two subjects (Arabic and math) (Forsberg & Schultz, 2022). The aim of BLP was to reduce stress reactions, provide a sense of safety, the ability to self-regulate negative emotions and foster greater hope. This indicates that the cognitive-affective theory of learning applies also in conflict areas, where it can help to promote school functioning amongst conflict-affected youth.
Future hope appears particularly relevant in school functioning. Snyder et al. (2002) found that children and youth with high hopes for the future were better able to plan appropriate cognitive and motivational strategies to achieve and accomplish desired educational goals. Other studies have also found that high levels of hope are related to higher academic performance (Chang, 1998; Worrell & Hale, 2001). A comprehensive review that explored the role of hope in academic and work environments found hope to be positively correlated with several variables related to task accomplishment and wellbeing (Yotsidi et al., 2018).
Self- and community-efficacy can be defined as promotors of school functioning: two comprehensive meta-analyses, exploring more than 7,200 articles, found self-efficacy to be a particularly strong predictor of academic performance (Richardson et al., 2012; Robbins et al., 2004). Self-efficacy refers to a person’s general belief that his or her actions have positive outcomes (Bandura, 1997) through regulation of negative thoughts/emotions and through goal-oriented behaviour (Carver & Scheier, 1998). This can also be extended to the surrounding community (such as the school environment, teachers/counsellors, and school–home collaboration) and the belief that one belongs to a community that is likely to achieve positive outcomes (Benight, 2004). Supplementary analyses conducted by Robbins et al. (2004) revealed that socioeconomic status was an important covariate in the link between self-efficacy and academic performance. However, living in emergency areas is associated with low-income countries and low socioeconomic status (Jordans et al., 2016), which serves to underscore the importance of community efficacy together with self-efficacy in crisis-affected areas (Benight, 2004).
The essential element of connectedness is conceptualized as support from family, peers and other close attachments, such as a specific teacher, along with reconnecting with the community after experiencing terrifying events. Connectedness is related to better emotional wellbeing and general daily functioning (Hobfoll et al., 2007). For children and youth, returning to school is very important for reconnecting with the community. Social support at school serves as a resilience factor for young people in high-risk settings or living with cumulative stress (Malecki & Demaray, 2006). Becker and Luthar (2002) found social support from parents and teachers to be important predictors for the academic performance of children living with stress; further, that peer support can also be a buffer factor, depending on the closeness of the peer relationships. These findings indicate that connectedness can also be seen as a promotor of school functioning, together with other essential elements of immediate and midterm mass trauma intervention as described by Hobfoll et al. (2007).
The Student Learning in Emergencies Checklist
The MHPSS reviews/meta-studies referred to in this article report a wide range of standardized instruments and programme-specific, self-made/cultural-adapted instruments that have yielded comprehensive amounts of data and findings that are, however, difficult to compare beyond the descriptions of the programmes (Fu & Underwood, 2015; Jordans et al., 2016; Lasater et al., 2022; Pfefferbaum et al., 2014; Rolfsnes & Idsoe, 2011). Nongovernmental organizations (NGOs) also collect considerable amounts of data as part of their monitoring and evaluation (M&E) processes, but they often lack formal procedures and evidence-based measurement tools. MHPSS interventions that target PTSD symptoms can choose amongst a range of standardized instruments, e.g., the UCLA PTSD Index, Child PTSD Symptom Scale, Child PTSD Reaction Index, CRIES 13 or CPSS-V SR, all of which have been validated for use in emergency contexts. However, MHPSS programmes that also focus on learning abilities should include additional instruments for measuring outcomes on factors such as wellbeing, academic functioning/performance, self-efficacy, emotion regulation and social support (such as the WHO Wellbeing Index, Self-efficacy Scale, or Social Support Scale for Children). However, such comprehensive questionnaires are impractical in M&E processes and difficult to manage for the young people involved in the programmes.
To address the gap in evidence-based instruments for M&E of school-based MHPSS, this study aimed to develop and test one such instrument: the Student Learning in Emergency Checklist (SLEC). Further, this study responds to the needs of the Norwegian Refugee Council (NRC) for evidence-based, culturally adapted M&E instruments for the Palestinian context. Here, the NRC has conducted large-scale educational programmes that deliver school-based MHPSS for conflict-affected students. What was needed was a practical instrument for teachers to monitor self-perceived improvement amongst their students, as well as an instrument for MHPSS programme quality control.
The SLEC is a result of a unique research/practice-based collaboration between two research facilities, the Arctic University of Norway (UiT) and New York University (NYU), and an NGO, the Norwegian Refugee Council (NRC). The project combined established knowledge, valid research methods and field experience from the Palestinian context in accordance with international guidelines for best practices (INEE, 2010). The collaboration involving UiT, NYU and NRC was one of eight partnerships between research facilities and NGOs in the Evidence to Action: Education in Emergencies Measurement Consortium (inee.org). The aims of the Measurement Consortium were to (1) develop, adapt and rigorously test measures concerning the holistic learning and development of children and youth, and (2) evaluate programme implementation and quality outcomes in the Middle East, North Africa and Turkey. The SLEC is a response to the first purpose of the Measurement Consortium. This checklist aims to (a) establish a baseline report in wellbeing, self-reported academic functioning and promotors for school functioning prior to the implementation of systematic MHPSS efforts, (b) provide demographic information about students and (c) measure improvement after MHPSS have been carried out.
Objectives for the Present Study
The main objective of this study was to evaluate the psychometric properties of SLEC. The proposed constructs for this checklist have been inspired by and based on the essential elements of mass trauma intervention (Hobfoll et al., 2007), and educational and psychological theories on academic performance (Aydin et al., 2016; Bandura, 1997; Becker & Luthar, 2002; Carver & Scheier, 1998; Hascher, 2010; Knörzer et al., 2016; Malecki & Demaray, 2006; Münzer et al., 2017; Richardson et al., 2012; Robbins et al., 2004; Vedder et al., 2005; Wentzel et al., 2017). The proposed constructs were sense of safety, self-regulation, self-efficacy, connectedness, hope and self-perceived academic functioning and wellbeing.
MethodsThe Process of Developing the SLEC
SLEC is the result of a thorough development process and a psychometric testing study conducted in the Occupied Palestinian Territories (the Gaza Strip, East Jerusalem and the West Bank). The process took place over a period of 8 months between September 2018 and April 2019 and included scale development, a pilot study and psychometric testing study. Content validity was continuously assessed throughout the development process to ensure correct functioning of the items. Assessment was performed separately for each section of the checklist. The SLEC development was based on the guidelines of scale development of DeVellis (2011).
Step 1: Determine clearly what it is you want to measure. The constructs were defined and operationalized based on empirical principles of recovery after trauma exposure, and educational and psychological theories that underpin academic performance, such as emotion regulation, self-efficacy and support. Appendix 1 presents an overview of the hypothesized constructs and all items that were included in the psychometric testing. Sections A and C included items that targeted the elements of trauma intervention (Hobfoll et al., 2007). Items for measuring self-efficacy were retrieved from the General Self-Efficacy Scale (Schwarzer & Jerusalem, 1995). Section B (academic functioning and wellbeing) was designed by the authors and was based on the pedagogical principles and goals of the BLP (NRC, 2017). All items were designed to capture the proposed constructs. As the SLEC is a self-report measurement instrument for students’ self-assessment, academic performance is not included as an element. Measures of academic performance (grades) can be collected additionally to fully assess school functioning.
Step 2: Generate an item pool. Several concerns were considered in generating the item pool. (a) Number of items, which had to be enough to cover the necessary constructs but few enough so they would be manageable for students (ages 12–16) in a timeframe of 30–45 minutes; (b) Choice of wording, to accommodate language complexity for the targeted age range; (c) Including both positively and negatively worded items; and (d) Context, the items had to be sensitive to the political situation and ongoing Israeli–Palestinian conflict. The item pool resulted in 35 items.
Step 3: Determine the format of the measure. A four-point Likert scale was chosen as response format. The Likert scale is commonly used in scales that explore behaviour and are broadly validated in scales for children and youth, e.g., Child Revised Impact of Events Scale (CRIES) (e.g., Dyregrov et al., 1996; Smith et al., 2001; Stallard et al., 1999), Children’s Hope Scale (Snyder et al., 1997) and KINDL (Ravens-Sieberer & Bullinger, 1998a; Ravens-Sieberer and Bullinger (1998b)).
Step 4: Have the initial item pool reviewed by an expert panel. The operators of the instrument (NRC staff, school staff and students) were the experts in the developmental process. All items were first reviewed by the NRC, which considered context sensitivity. The instrument was then administered to 50 randomly selected students at NRC-operated schools in Gaza and West Bank for an instrument test. Thereafter, the NRC conducted two focus group interviews (FGI) involving a total of five teachers and 10 students, who also participated in the instrument test. The FGIs were conducted by an experienced M&E consultant, and the focus was on evaluating the items. Language complexity was assessed, and participants provided feedback on what they thought the items measured and what they intended to measure. Based on feedback from the instrument test and the FGIs, some items were rephrased, and expository text was included to clarify the purpose of the instrument. Further, the instrument was piloted, amongst other instruments, in an evaluation study (N = 300) for a school-based MHPSS programme (BPL-2) (Forsberg & Schultz, 2022). The results indicated that Cronbach’s alpha did not achieve the accepted cut-off level (0.70) for any of the proposed constructs, and step 5 was initiated to increase the reliability.
Step 5: Consider inclusion of validation items and revision of the instrument. After the pilot/evaluation study was conducted, the researchers visited the schools to gather experiences with the instrument. The researchers conducted nonstructured interviews with three school principals, nine teachers and two school counsellors. The interviews covered instrument design, constructs/items, choice of wording, number of items and their experiences from helping the students to manage the instrument. After these interviews, six items were revised and adjusted to more appropriate language complexity for the age group, five new items were added to the instrument, and a fifth neutral response option was added to the Likert scale. When the revision was finalized, the instrument was translated into Arabic by NRC staff and back-translated into English by an independent translation agency. Back-translation is a well-established method for ensuring the reliability of an item (Guillemin et al., 1993). In this case, the back-translation revealed three items that needed minor changes in wording in order to match the English version. An Arabic version of the instrument is available upon request to the authors.
The next sections present Steps 6–8 of the scale development guidelines (DeVellis, 2011) that cover development sample, procedure, evaluation of the items and analytic plan. The development process is illustrated in [Figure 1].
Participants/Sample
Step 6: Administer items to a development sample. The initial recruits for the psychometric testing study (PTS) were 825 Palestinian youth. The age range for the sample was set to 12−16 years. From the initial recruits, 36 were excluded either because they were not within the age range or because they had not recorded their age on the instrument. The final sample consisted of 789 young people (358 males and 431 females; mean age = 14.29, SD = 1.29).
Participants came from 17 NRC-operated schools in the Gaza Strip (N = 397), East Jerusalem (N = 150), Hebron H2 (N = 114) and other areas of the West Bank (N = 126). The Ministry of Education (MoE) had identified these schools for participation on the basis of demographic characteristics. All geographic areas within Gaza (north, east and west), East Jerusalem, Hebron H2 and other areas of the West Bank were represented, and both boys’ and girls’ schools were included (nine boys’ schools, five girls’ schools and three mixed schools). All the schools had experience with implementing BLP and collaborated closely with the NRC. These were basic schools, grades 1−10, with an average of 595 students each. In total, 55 schools in Gaza (44 Gaza MoE) and 11 United Nations Relief and Work Agency (UNRWA) and 78 schools in the West Bank (all MoE) have implemented the BLP.
Procedure
The participating schools each had one visit from NRC staff, and the instrument was administered, and data collection was conducted. As the schools were already enrolled in the BLP, the parents had already provided written consent for their children to participate in research studies relating to the programme. The informed consent form included information about the purpose of the study. It stated that the data would be used for a research study, that all information would be anonymous, and that respondents could withdraw consent at any point without having to justify their decision. Participation was voluntary, and the youth gave additional informed oral consent prior to their involvement in this specific study. Ethical approval of the study was given by the Palestinian Health Research Council and the Norwegian Centre for Research Data. The MoE in Gaza approved the protocol and procedure.
All students (aged 12 to 16) present on the day of data collection were invited to participate. Fewer than 10 did not want to participate and left the classroom; the remaining students were guided through the questionnaire. The NRC staff who conducted the data collection stayed together with the students during the session, read the items out loud, explained what they meant and what the response options meant. This group-based data collection was a practical approach to enable as many students as possible to participate. The classroom sessions took approximately 45 minutes.
Analytic Plan
Step 7: Evaluation of the items; Step 8: Optimising the scale length. The evaluation of the items and validity assessment of the instrument were performed based on an analytic plan from Kline (2011). The length of the scale was based on reliability of the constructs and dropping items that reduced the Cronbach’s alpha.
Construct Validity
The construct validity and internal structure of sections A, B and C were assessed by a series of exploratory and confirmatory factor analyses (CFAs) using MPlus version 7.2. The students in the sample were nested in schools, which violated the assumption of the independence of standard errors required in the application of factor analysis techniques. Therefore, the TYPE=COMPLEX command in MPlus was used to estimate robust standard errors (Asparouhov & Muthèn, 2014). The analyses proceeded in three steps.
The first step was to split the sample randomly. The full sample was first randomly divided in half and stratified on gender and age to create exploratory and confirmatory samples. The exploratory samples were used to examine multiple versions of data-driven models, from which a final proposed solution was selected based on conceptual and empirical considerations. The confirmatory samples were used to test hypothesized and proposed factor structures, thereby building confidence in the stability of empirically derived exploratory factor analytical estimates (Osborn & Fitzpatrick, 2012).
The second step was to conduct exploratory analyses by hypothesized construct. Using the exploratory sample, the first test was whether there was evidence that correlations amongst items within each hypothesized construct of sections A, B and C were consistent with a single latent construct. The second test was fitting one-factor confirmatory models for (a) a sense of safety, (b) self-regulation, (c) self-efficacy, (d) social support, (e) academic functioning and wellbeing at school and (f) hope. To assess the goodness of fit of the models, the following two criteria were used (Hu & Bentler, 1999): a root mean squared error of approximation (RMSEA) value below .08 provided an acceptable fit to the data, whereas an RMSEA of less than .05 provided a good fit, and a comparative fit index (CFI) value above .90 provided an acceptable fit to the data, whereas a CFI value above .95 provided a good fit (Kline, 2011).
The third step involved conducting analyses to identify empirically derived constructs and confirm the solutions out-of-sample. Although some of the confirmatory models provided a good fit to the data, many of the resulting subscales did not show evidence of satisfying reliability (Yang & Green, 2011). Thus, an exploratory factor analysis was conducted using items from sections A, B and E to examine whether it was possible to empirically derive constructs that aligned with the theory but resulted in scores with better reliability. Analyses were conducted using the exploratory sample with an oblimin rotation. Through a combination of the overall goodness-of-fit statistics, item specificity and strength, targeted misspecification indices, and the face validity of the models, a final factor structure was selected to test using the confirmatory sample.
Concurrent validity
To assess concurrent validity, we calculated sum scores for each of the empirically derived constructs that emerged from the analysis (see [Table 2]). Such a method assumes that all items have equal weight in the composite scores; however, unit-weighted scores are the most common scoring method used by practitioners in humanitarian contexts (Kline, 2011). Thus, we examined zero-order Pearson correlation coefficients between the empirically derived sum scores and demographic information (gender and age) on the participants.
ResultsConstruct Validity
Confirmatory Factor Analysis (CFA) by Hypothesized Construct
A CFA was conducted to ascertain whether the intercorrelations amongst items within each hypothesized construct were consistent with a single latent factor. [Table 1] presents the model fit statistics for the one-factor models and Cronbach’s alpha for all the hypothesized constructs. Although correlations amongst items in the hypothesized “academic functioning and wellbeing at school” and “hope” constructs were consistent with a single latent construct, a one-factor model did not provide a good fit to the data from the hypothesized sense of safety, self-regulation, self-efficacy, or connectedness items. In addition, the internal consistency of all scores calculated according to the hypothesized scale structure was below 0.70, the commonly accepted cut-off value for Cronbach’s alpha (Yang & Green, 2011).
Table 1 Summary of Goodness-of-Fit Statistics for Hypothesized Factor ModelExploratory Factor Analyses
Using a random half sample of the data, two types of exploratory factor analyses (EFAs) were conducted. First, to establish whether a reduced number of constructs could be used to represent the items in the instrument, the scree plot was examined for eigenvalues > 2 cut-off criteria and the model fit statistics of 1- through 7-factor exploratory factor analytic solutions. Based on these results and following an examination of the factor structure at the item level, a five-factor structure emerged. [Table 2] presents the factor structure.
Cross-loading and low-loading items indicated by the exploratory analysis were removed before the proposed final model was confirmed. We also conducted a set of exploratory analyses within each originally hypothesized construct to find whether the model fit could be improved by removing certain items with low factor loadings or high model residual correlations. Although these modifications would probably not have improved the internal consistency of the constructs, they may guide future revisions of the measure if the user is specifically interested in measuring the six core constructs hypothesized to be assessed in the instrument. [Table 3] presents the overview of the modifications. [Table 4] presents the overview of the items, and factor loading.
CFA of Empirically Derived Constructs
Based on the results of the exploratory analyses, two sets of CFAs were conducted with the other random half of the sample. First, the CFA model with the five main factors identified through the EFA ([Table 3]) provided a good fit to the data (c2 (288) = 359.60, p = 0.003, RMSEA [90% CI] = 0.025 [0.016-0.033], CFI = 0.946). Items showed moderate-to-high (> 0.40) loadings for their factors, except for item A10. This item was negatively worded and reverse-coded; future researchers may want to revise it. Correlations between factors were all in the expected positive direction. They were low-to-moderate, except for the high correlation between the safety and adaptability factor and the current and future wellbeing factor.
Second, a CFA model with the eight factors identified by the EFAs of the hypothesized constructs ([Table 3]) was fitted to the data. This model also provided a good fit to the data (c2 (405) = 475.80, p = 0.008, RMSEA [90% CI] = 0.021 [0.011-0.029], CFI = 0.949). Items showed moderate–to-high (> 0.40) loadings on their factor, except for A4 and A5. These items were negatively worded and reverse-coded; again, future researchers may want to change their wording. Correlations between factors were all in the expected direction. They were low-to-moderate, except for that between safety and school functioning (r =.81).
Concurrent Validity
Bivariate correlations between empirically derived SLEC constructs and the participants’ gender and age were examined using sum scores of each construct ([Table 5]). There were two significant and moderate correlations between gender and SLEC constructs: Females perceived less support from school staff than males but reported greater perceived current and future wellbeing. Correlations between SLEC constructs and age were small but significant.
Reliability
Cronbach’s alpha coefficients of the constructs’ sum scores were calculated from the empirically derived five- and eight-factor models. The reliability of three of the five construct sum scores (safety and adaptability, school support and family support) calculated according to the empirically derived five-factor model was above 0.70, which is the commonly accepted cut-off value (Yang & Green, 2011) ([Table 2]). Although this was an improvement in reliability over the scores from the hypothesized eight-factor model, the construct scores on emotion regulation and current and future wellbeing remained below the commonly accepted criteria. Moreover, the reliability of most of the construct sum scores calculated according to the empirically derived eight-factor model was below 0.70, except for the scores from school and family support scales ([Table 3]).
DiscussionThe aim of the study reported here was to collect evidence for a psychometric instrument for the M&E of school-based MHPSS in Palestine. The data analysis revealed a different factor structure than expected, which resulted in adjustments involving the proposed constructs. Given that the CFA fit indices of the five- and eight-factor models were acceptable, we found conflicting evidence regarding the extent to which the empirically derived scoring strategy was supported by the data. Such a situation can arise when scales are too short, as calculation of the alpha coefficient depends on the number of items in the scale; or when the underlying construct is defined and operationalized broadly, which results in item responses being less closely linked (Kline, 2011; Osborn & Fitzpatrick, 2012). We thus recommend that future iterations of either the SLEC (the five-factor model: Appendix 2) or of the tested instrument (Appendix 1) should contain constructs that are clearer and more narrowly defined (as with safety and self-efficacy), and that extra items should be added (e.g., in the hope construct) to ensure reliability. As the overall factor model was good and most of the reliabilities were acceptable or nearly acceptable, the SLEC can be said to satisfy the M&E requirements of NGOs. Thus, our instrument can be used to measure the constructs it is intended to measure.
Since the analysis revealed inconsistencies in the hypothesized constructs that were tested, a new five-factor solution was generated. The new constructs were 1) safety and adaptability, 2) emotion regulation, 3) school support, 4) family support and 5) wellbeing and hope. Internal consistency was satisfactory in three of these, and the overall factor structure model was good.
The two new constructs combine items from the original ones. The first new construct, Safety and adaptability, combines several items from the original constructs of sense of safety and self-efficacy, as well as one item from academic functioning and wellbeing. This suggests that feeling safe is a premise for children and young people to be able to believe that their actions have positive outcomes and that they can achieve their goals by concentrating on their school work. The second new construct, Wellbeing and hope, combines the items from the original constructs of academic functioning and wellbeing, and hope, indicating that a sense of wellbeing and a level of hope are related. The high correlation between the two new constructs of Safety and adaptability and Wellbeing and hope indicates that these constructs provide a high degree of overlapping information, so future researchers should consider reviewing and adjusting the items in them, in order to clarify and distill the constructs.
The five constructs have the advantage of being relatively easy to include in teacher training as promotors for school functioning − and thereby integrated parts of delivering psychosocial support for conflict-affected students to improve learning.
A further revision of the SLEC would imply the need for a new PTS. A revision of the instrument implies that items with low-factor loadings could be exchanged with new items within each of the constructs of the empirically derived five-factor model. New items should be similar to already existing items with high factor loadings. This should improve the internal consistency of all constructs (Kline, 2011). The SLEC was developed with a specific sample in mind: children and youth (both genders) aged 12 to 16 who were living in occupied Palestinian territories (the Gaza Strip, Jerusalem and the West Bank). Adjusting the SLEC for other populations and contexts will require careful consideration. Just as in the present study, adjustment will require thorough consideration of the age, gender and background (e.g., country/area, and cultural context) of potential respondents.
For use of SLEC beyond the Palestinian context, we recommend a systematic validation approach to ensure that the instrument is be contextually relevant. Especially important is to follow steps 4 and 5 of DeVellis (2011), presented in the method section. Step 4: Have the initial item pool reviewed by an expert panel. The initial item pool of SLEC can be found in Appendix 1. The expert panel, local NGO staff and some user representatives from those who will operate the instrument (e.g., teachers and students), can provide valuable insights. Contextual adaptation requires considering in two main areas: (1) The native language in the region and (2) cultural and contextual characteristics of the region. First, the SLEC is currently available in Arabic and English. If SLEC is to be used in a context with a different native language, the instrument should be translated and then back-translated via a professional translation service. As to the second consideration, national/local NGO staff with experience in the region should review and assess the instrument items, which may then be adjusted or rephrased as necessary, with regard to what is considered sensitive or inappropriate in the specific context. User representatives can provide insights on, e.g., language complexity for the targeted age group and the practicality of conducting data collection using the SLEC. Step 5: Consider inclusion of validation items and revision of the instrument. The context might require that additional items are included in the SLEC, in which case the expert panel can provide feedback and advice. After a provisional contextual adaptation of the instrument, a small scale pilot study should be conducted amongst a small sample drawn from the target population. Feedback from the pilot should be reviewed by the expert panel, and SLEC adjusted accordingly if necessary.
Strengths and Limitations of the Study
Two strengths of this study are the number of participants and the careful selection of participating schools in terms of demographic characteristics. The schools were all defined as schools in conflict-affected areas by the MoE in Gaza and West Bank, and UNRWA. This ensured a representative sample for this specific context. Furthermore, the analytic plan followed was as described by Kline (2011), and the analysis was conducted by a researcher with extensive experience from similar analyses.
In emergency contexts, such as the Occupied Palestinian Territories, it is challenging to conduct research and ensure reliable and valid results because of the difficulties involved in maintaining scientific quality control throughout the process. The PTS was conducted in February and March 2019, which was a turbulent time in the ongoing conflict in the area. This may be regarded as both a strength and a limitation in terms of the study. The children and youth who participated in the study were all personally affected by the conflict. They were exposed to traumatic experiences and cumulative stress, which may influence how they responded, thus affecting the validity of this study. However, the SLEC was developed in the context in which it was employed—in an emergency area and under challenging circumstances. Another strength was the collaboration between research facilities and an NGO. This collaboration ensured research quality and context-specific expertise and is in line with the general guidelines for best practice (WHO, 2019), thereby, enhancing the validity of the results.
However, one limitation of the study is the level of reliability within the empirically derived constructs of the SLEC. The reliability level does limit the research purposes of the instrument; however, both the five-factor model and eight-factor model showed a good fit with the data. Thus, the SLEC can be utilized for M&E purposes and provide empirical evidence for further research towards an evidence-based validated instrument. Other limitations are that predictive and discriminate validity were not assessed. Comparators better suited for assessing concurrent validity than demographic variables of age and gender should have been included—like social/economic determinants, and measures of academic performance (grades). Social and economic determinants are important factors in predicting wellbeing (Dimitry, 2012). It would also have been worthwhile to include academic performance as a correlate for construct validity; this would have enabled full assessment of school functioning, not just self-perceived academic functioning and perception of wellbeing.
There are also certain biases associated with reliance on self-reported measures alone. First, the students must be able to understand the items; second, their ability to assess their own academic functioning accurately. Third, there is a possibility that students answered what they felt to be most socially accepted, rather than being truthful. These biases can be addressed by assisting the students when they fill in the instrument, providing age-appropriate explanations of the constructs, highlighting the importance of answering truthfully, and announcing short breaks when the students lose their focus and concentration.
ConclusionsThe study reported here resulted in a 26-item instrument for monitoring and evaluation of school-based MHPSS in the Palestinian context. The instrument measured self-perceived academic functioning, precepted wellbeing and promotors associated with school functioning in conflict-affected areas. The constructs of SLEC are 1) safety and adaptability, 2) emotion regulation, 3) school support, 4) family support and 5) wellbeing and hope. Comprehensive psychometric testing showed that these all had satisfactory validity, and that the instrument was well-suited for M&E purposes in Palestine. The SLEC can be utilized for research purposes, with a few considerations in the constructs with reliability below 0.70.
The SLEC has a potential of serving as a practical instrument for NGOs in quality control concerning educational programme planning, design and evaluation of school-based MHPSS amongst conflict-affected students. Further, the instrument could be systematically used by teachers at school and class level, in monitoring students’ perceptions of received MHPSS. SLEC results can be discussed in teacher dialogues as well as student dialogues for raising awareness of the importance of psychosocial support and how this can strengthen school functioning during times of conflict and severe stress.
More research is needed before the SLEC can be sufficiently validated as an evidence-based psychometric instrument for research purposes in the Palestinian context. Revision of the instrument and a further validation study should generate higher reliability within each construct. Including objective measures of academic performance would increase the ability to assess construct, concurrent, predictive, and discriminate validity of the instrument. Thus, a further validation study should include measures of academic performance from the participants. For the purpose of validating the SLEC for other contexts, a natural next step could be to include participants from other countries in the Middle Eastern region.
Acknowledgements
This study was a collaboration between the Arctic University of Norway and New York University, and an NGO, the Norwegian Refugee Council and Education in Emergencies Evidence for Action (3EA) Measurement Consortium. Special thanks to the other partnerships in the Measurement Consortium, Roxane Caires, Sarah Capper, Asad Ashour and Camilla Lodi. The charity organisation “Porticus” has, together with 3EA, provided financial support for this study.
Financial support and sponsorship
Nil.
Declaration of competing interest
The authors report no conflict of interest.
Appendix 1Section A:
Hypothesised domain: Sense of safety
I feel safe at school.I feel that the teachers and school staff respect me.I feel calm at school.I am afraid when I am at school.I am scared to travel to school or home because the trip can be dangerous.I feel safe at home.Hypothesised domain: Self-regulation
When I feel sad, I know what to do to feel better.I can control my temper.When I am scared, I can calm myself down.When I get angry, I hit other people or things.When I feel angry, I can calm myself down.Hypothesised domain: Self-efficacy
I can manage to solve difficult problems if I try hard enough.It is easy for me to stick to my aims.It is easy for me to accomplish my goals.If I am in trouble, I can think of a solution.I can handle whatever comes my way.I am confident that I can adapt to events I had not predicted.Hypothesised domain: Connectedness
I can talk to someone in the school staff (a teacher, the principal, a counsellor) about my worries.Someone in the school staff (a teacher, the principal, a counsellor) ask me how I am doing.Someone in the school staff (a teacher, the principal, a counsellor) support me when I feel scared.I can talk to my parents about my worries.When I feel scared, I can tell my parents.My parents ask me how I am doing.I have friends to play with at school.I can talk to my friends about my worries.My friends support me when I feel scared.Section B:
Hypothesised domain: Academic functioning and wellbeing
I can easily concentrate when doing school work.I am able to do my best in school.I am satisfied with my life.I like being at school.Section C:
Hypothesised domain: Hope
Things will turn out well in the future.I will graduate school.I will get a job when I grow up.I will get a family when I grow up.I will live a meaningful life when I grow up.
References
Comments (0)