Remember me
FIELD REPORT
Year : 2023 | Volume
: 21
| Issue : 1 | Page : 3-8
Lessons Learned in Using the Score of Perceived Outcomes (ScoPeO) Tool in Mental Health and Psychosocial Support Programmes to Measure Quality of Life
Blanche Mattern1, Maximilien Zimmerman2, Lise Archambaud3
1 Independent Community-based Strategy Expert, Senegal
2 MHPSS Global Specialist, Humanity & Inclusion, Belgium
3 Humanity & Inclusion, Belgium
Correspondence Address:
MPhil Blanche Mattern
Independent Community-based Strategy Expert
Senegal
Source of Support: None, Conflict of Interest: None
DOI: 10.4103/intv.intv_8_22
Following the launch of the Impact and Quality of Life project in December 2012 by the Luxembourg Ministry of Foreign and European Affairs, Humanity & Inclusion − formerly Handicap International (HI) − developed a new monitoring tool, the score of perceived outcomes (ScoPeO), which is now widely used across HI programmes to measure the quality of life. The tool was used to evaluate service-users’ quality of life in four countries (Lebanon, Madagascar, Sierra Leone and Togo) in the Touching Minds, Raising Dignity (TMRD) programme which aims to end the stigmatisation of people living with mental health issues and to improve their quality of life. This field report documents lessons learned in using the ScoPeO tool, specifically in relation to quality of life. It describes the methods used to survey the interventions in the four different settings and discusses the challenges and limitations in using the ScoPeO tool.
Keywords: assessment, comparative study, discernment, family quality of life scale, mental health, quality of life, score of perceived outcomes
Based in Lyon since its foundation in 1982, Handicap International/Humanity & Inclusion (HI) targets vulnerable populations on the global stage, notably people with disabilities. HI responds to their specific needs (food, shelter, clothes, financial resources, mobility aids), improves their living conditions and contributes to their social participation. Beyond this, HI works towards a more inclusive society. HI beneficiaries include people with mental health issues and/or dealing with psychosocial distress.
Beyond following the WHO understanding of mental health (“mental health is more than the absence of mental disorders” (WHO, 2022), HI has an engaged, resolutely social definition of mental health (MH) in line with the Lyon Declaration from the Five Continents Conference in October 2011. Accordingly, HI defines a reasonably good mental health as “the ability to live with oneself and others in the pursuit of pleasure, happiness, and meaning in life, in a given yet not unchanging environment which can be transformed by the activities of people and groups of people, without destructiveness but not without a sense of revolt” (Orspere, 2012). Through this definition and through its mode of operation, HI places the person back into the centre of the therapeutic process as an active individual.
The Touching Minds, Raising Dignity (TMRD) programme has been implemented in Madagascar, Lebanon, Togo and Sierra Leone since 2014. It aims to improve the quality of life and the wellbeing/mental health of people facing psychosocial distress and/or mental health issues by developing community-based strategies for prevention and response. Indeed, building the mechanisms of community resilience is a key driver of change towards an improved psychosocial wellbeing and an increased respect for the target population’s basic rights.
Phase 2 of the TMRD programme began in 2018 with funding from the Agence Française de Développement (AFD) and the French Ministry of Foreign Affairs. It has three axes: (1) enhancing quality in collaboration with users who were key project stakeholders; (2) capacity-building, leading to improved consistency and efficiency; (3) conducting research, enabling the development of more effective interventions. It should be noted that all the intervention countries faced the health and economic crises of the COVID-19 pandemic during this time. Moreover, Madagascar and Sierra Leone lie in disaster-prone areas and Lebanon has been facing an acute economic crisis.
Phase 2 focussed on monitoring and evaluating users’ quality of life. HI sought to measure the quality of life defined by beneficiaries’ perception of their physical and mental health (PMH), social and personal relationships (SPR), subjective wellbeing (SWB), basic needs (BN), safety, material wellbeing (MWB) and participation in society and family life. An operational study was, therefore, planned which aimed at generating evidence and demonstrating the relevance and impact (or absence thereof) of the TMRD project. It was implemented from May 2019 to December 2021.
Within the scope of the TMRD project, ScoPeO was defined as an objectively verifiable indicator (OVI). This tool rests on a subjective approach: interviewees express their perception of the different dimensions that constitute their quality of life. HI has been using ScoPeO for years, but its implementation as a part of the TMRD project was tested for the first time on mental health (MH) interventions. Of particular interest to the study, therefore, were the challenges encountered both while implementing this tool in this area of intervention and while determining which complementary methods could be used to enhance the collected data. The study showed the degree of relevance, conditions of use, limits and challenges in the implementation of ScoPeO within interventions on MH-related issues. It also identified avenues for further reflection into strengthening data collection and analysis for this type of intervention. This field report presents these findings and reflections.
MethodThe ScoPeO Tool
The ScoPeO tool was developed to measure the effects of HI interventions on project beneficiaries’ quality of life, both for children (ScoPeO-Kids version, for beneficiaries aged 5–17 years (Brus et al., 2019) and for adults (ScoPeO-Adults version, for beneficiaries aged 18 and over (Brus et al., 2020). The ScoPeO-Adults tool development was based on several activities carried out internally: a literature review on definitions and tools for measuring quality of life, wellbeing, social participation and living conditions; summary analysis of experiences, and in particular, of tools already developed by different teams; consultation of a group of 14 partners in seven intervention countries on the dimensions constituting quality of life. This served as a basis for the following stages, carried out via an academic partnership between the French National Centre for Scientific Research (CNRS), the Impact Monitoring Evaluation Unit from HI and HI’s representatives from the head office and HI field teams. These stages included interviews with HI teams and testing of different versions of the form in Senegal. Based on this experience and on knowing that children under 18 years of age constitute an important part of the population targeted by HI’s projects, a new round of research was launched to allow the measurement of quality of life among children. ScoPeO-Kids is not, however, a simple adaptation of the original adult version but is the result of independent research focussing on the specificities of younger audiences and appropriate administration. The design of ScoPeO-Kids followed an 8-step process: a literature search; a workshop with HI professionals; focus groups with children and their parents in two countries, Bangladesh and Thailand; production of a set of items and selection of response options; a quantitative survey (testing of the first version of the questionnaire); checking the validity of the tool; cognitive debriefing of the children; and revision of the tool to the version we are currently using. Thus, the dimensions, number of questions, response categories and administration procedures are not the same for ScoPeO-Adults and ScoPeO-Kids. However, HI has endeavoured to maintain consistency between the two tools so that they can be used together in the same project.
Both tools are based on the same founding principles:
conceptual principles: quality of life is subjective, multidimensional and dynamic; operational principles: the tool is generic, applicable regardless of the gender, health status or disability situation of the people studied in the survey; it serves the same purpose of monitoring a project (measuring the effects of interventions on the quality of life of beneficiaries); methodological principles: the same data collection methods (direct interview with beneficiaries) and analysis (score and partial scores) are recommended.The ScoPeO tool comprises a guideline, four questionnaires (baseline and endline for adults, baseline and endline for children above 5 years old) and a matrix for analysis.
ScoPeO-Adults focusses on the following dimensions: PMH, SPR, SWB, BN, MWB, and participation in society and family life. It has 27 questions. ScoPeO-Kids focusses on these dimensions: physical wellbeing, emotional wellbeing, autonomy and self-realisation, perception of safety, and social wellbeing. It has 21 questions. Both questionnaires collect quantitative data related to the mentioned dimensions using Likert scales. The data are analysed through a data analysis matrix developed on Excel. ScoPeO being a comparative tool, a score is calculated each at the start and at the end of the project and an overall analysis is run at the end of the project to measure an increase or a decrease in the ScoPeO score (overall and by dimension). In addition, both questionnaires collect data using the Washington Group Short Set of questions (Washington Group on Disability, 2022) to ventilate the result according to the type of functional difficulties.
Planning and Training
The operational study was launched remotely, with one field visit in each of the four countries except Madagascar1 and in close cooperation with the coordination team based in Lomé, Togo and with the project team. Preliminary work was carried out with all the mission teams, the coordination team, and the technical team from the HI headquarters during a seminar in Freetown, Sierra Leone in May 2019 as well as remotely and/or during field visits.
This preliminary work made it possible to calibrate and support the implementation of ScoPeO but also to anticipate a few issues. First, it enabled the teams to decide which activities would be measured, how, and why, but also to define the focus of the study, its duration and its training and translation plans. Second, it also raised the issue of discernment of how to identify its hypothetical limitations and of anticipating by developing mitigation and documentation procedures. A constant collaboration with the teams from the initial analysis to the implementation of the endline yielded complementary proposals for data collection and analysis. However, some of them could not be explored due to the COVID-19 pandemic. These proposed complementary procedures were both qualitative and quantitative. As part of the ScoPeO-Adults endline, a questionnaire and group discussions were also tested in Madagascar, Sierra Leone and Togo to collect secondary data on the impacts of the Covid-19 crisis on quality of life. In Lebanon, another tool, Family Quality of Life (FQoL), was used whenever ScoPeO-Kids could not be administered. Finally, a secondary analysis was carried out to investigate perception trends for each dimension and question.
In 2018, a training on the use of ScoPeO was organised in Dakar, Senegal. In 2019, a field visit served to strengthen skills and train those mission teams that were still untrained. The training participants were field teams likely to be involved in the development and the implementation of ScoPeO studies in the future. A part of the training focussed on how, why and when to administer the questionnaire. A second part focussed on anticipating difficulties that could arise while administering the questionnaire with persons with a mental health condition, and how to mitigate those difficulties. Finally, an important part of the training aimed to train the instructors in charge of training survey administrators. In turn, the trainees went on to train survey administrators.
The study’s implementation protocols resulted from a collaboration between an external consultant, the technical team at the headquarters and the coordination team in Lomé, Togo. The procedures for the survey’s ethical approval, implementation and monitoring were devised by the mission teams with the consultant’s technical support. The latter was also in charge of analysing the data and of writing reports as well as the lessons learned document. She was also responsible for finding technical solutions to the limitations of the study.
Data Collection
A baseline survey was conducted in 2020 to collect data on the initial situation, and an endline survey was conducted at the end of the project in 2021 to collect data on the outcome. These assessments provided a basis for comparison between beneficiaries’ situations before and after intervention, based on calculating scores to bring out changes over time within a comparative framework. The same participants were included in both baseline and endline surveys and questions were administered by trained surveyors.
Data were collected on smartphones stored in SurveyCTO and analysed using Excel. In terms of confidentiality, data on SurveyCTO were accessible only to specific mission members. Information such as participants’ names was collected to be able to find them at the end of the study. The data had been anonymised using a unique ID before being transferred to Excel analysis sheet. Therefore, all accessible data are anonymised.
Key Country InformationLebanon
In Lebanon, the target population comprised all users of the programme, that is, children aged 0 to 17 with mental health issues, both with and without physical disabilities.The study was conducted between August 2019 and January 2021 in Aarsal, Bar Elias, and Baalbek, Lebanon. In Lebanon, TMRD provides a holistic course of treatment based on a set of activities specific to each individual child.2 While other missions start activities with a group of beneficiaries, in Lebanon each child entering the programme follow an individual activities plan. Given the setup of the programme, a baseline and an endline could not be run as initially planned. A procedure was, therefore, tested where the questionnaire was systematically administered whenever a child entered or left the programme. All the initial surveys were entered into the baseline, and all the exit surveys, into the endline.
Since ScoPeO-Kids is not suitable for children under 5, a means of interviewing younger children had to be found. It was also found that some children over 5 were not able to answer the questionnaire due to severe cognitive disabilities. Another tool − FQoL (Beach Center of Disability, 2015) − was identified for use with these groups. All the questionnaires were translated into Arabic by a translation agency using a double-blind translation method (two blind translations from English into Arabic, 2 back translations from Arabic into English, comparison and validation by a translation committee).
For the baseline study, 60 children responded to ScoPeO-Kids, and 79 children, to FQoL. For the endline, 34 children responded to ScoPeO-Kids and 42 children, to FQoL. The measurement of improvement or decrease in the quality of life was done on those who had responded to the baseline and the endline study.
Madagascar
In Madagascar, there are limited human resources and care and support structures. The gap in provision is largely filled by Toby which are religious structures that offer religion-based care. The ScoPeO survey was, therefore, administered in three large Toby in the city of Mahajunga. Since Toby have a stable take-up over time, the survey targeted adults experiencing psychosocial distress and/or MH issues or those participating in TMRD activities either directly through HI or through its operational partner(s) and living in a Toby.
The team translated the questionnaire into Sakalava, the main local language spoken in the study area. The Madagascar ScoPeO study was a comprehensive, quantitative and transversal survey (all adult participants in the Toby were interviewed at time t). It was expected that the number of service users would not differ between the baseline and the endline surveys. This study design was selected as the majority of participants were anticipated to take part in both studies. Indeed, having the same participants from start to end was crucial in terms of reporting to the project funding agency. Unfortunately, of the 124 users participating in the baseline, only 37 were identified at the end of the study (approx. 29.84% of the baseline participants).
Although, participants at the Toby usually remained stable, this drop in numbers was due to a combination of circumstances including the activity setup, the impact of the Covid-19 crisis (leading HI to suspend its activities for some time), and thus, the time that elapsed between the baseline and endline surveys. On reflection, it was concluded that it might have been more fruitful to use ScoPeO as an individual monitoring tool, given these very conditions.
Sierra Leone
Unfortunately, Sierra Leone has regularly faced various crises over recent years (civil war, Ebola outbreak and natural disasters) and its population has been deeply affected, particularly, in relation to their mental health. The TMRD project in Sierra Leone provides services related to mental health to user groups over a set period of time. Beneficiaries include those with mental health issues or in need of psychological support. This may include individuals with addictions (alcohol and drug misuse), victims of sexual assault and other kind of physical and psychological abuse. Activities organised as part of the support group involve individual and group psychosocial counselling as well as income-generating activities.
The survey was conducted among support group users in Susan’s Bay, Magazine, Kuntorlor, and Looking Town. The teams translated the questionnaires into Krio. However, responses from Susan’s Bay beneficiaries had to be removed from the comparative study as − very sadly − a fire devastated the shanty town in 2021, and conditions were such that endline questionnaires could not be administered. Data had to be re-analysed to exclude Susan’s Bay inhabitants, leaving 64 questionnaires to be considered for the final baseline results and comparison analysis.
Togo
In Togo, HI launched its first mental health project (TMRD) in 2014 with the combined roles of supporting existing structures, uniting local and national stakeholders and reinforcing beneficiaries’ autonomy. The study targeted users over 18 facing psychosocial distress and/or MH issues who were beneficiaries of TMRD interventions and were being treated either directly by the mobile team or by one of HI’s two local partners, the Regional Hospital of Dapaong and the Yendubé Mental Health Centre in Dapaong (both being mental health services). Depending on their psychosocial and psychiatric needs, users offered individual appointments, support groups, home visits by peer support workers and medical treatment.
The baseline and endline studies were implemented in the Barkoissi Canton in the Savannah region, in the Oti Health District. The questionnaires were translated from French into the Ben language during the trainers’ training (double-blind translation from French into Ben by two translators, back translation from Ben into French, consultation between the translators and testing with the technical team). This was a longitudinal survey, with the same users between the baseline and endline studies. A total of 87 participants were interviewed during the baseline study. Sixty five participants were contacted for the endline study and subsequent comparative analysis.
Lessons LearnedThe Issue of Discernment
Discerning whether interviewees understand the purpose and instructions in an evaluation such as this and are capable of answering questions concerning their quality of life is crucial for studies evaluating interventions; for example, there are several tools to assess patients’ capacity in the contexts of treatment take-up or end-of-life decisions in medical settings (Silberfeld, 1993; Swiss Academy of Medical Sciences, 2018). However, there appears to be no standardised tool for assessing whether participants are able to take part in a survey.
The team considered whether a protocol could be written to guide decision-making in excluding anyone from the study. The idea of assessing each individual user was ruled out due to a lack of resources (such as funding, time, team members with relevant expertise in conducting and supervising the survey work, etc.). In addition, the possibility was considered of checking users’ informed consent prior to the study at certain points during the evaluation process. This was also discounted mainly as the informed consent given in advance does not necessarily reflect the capacity of discernment of the individual at a later time. The decision was eventually made to allocate time to raising awareness of this issue in the survey administrators’ training. Teams were instructed to notify their supervisor and document the procedure whenever they decided to exclude a user.
Mitigating the Limitations Stemming from the Context of Intervention
The experience of implementing this study has highlighted potential limitations when using ScoPeO within MH programmes. However, it has proved possible to use the tool provided that measures are put into place to take account of users’ needs, and therefore, reduce the risk of users being excluded; for example, the training programme for survey administrators devoted time to practising relating to users in a respectful, positive way, given that they may have different needs. The mission teams and the consultant pinpointed different possible scenarios for each country context and then turned them into role plays that were used as part of trainers’ training. These exercises were then also used to train survey administrators; for example, the training in Sierra Leone had a roleplay about relating to a person under the influence of drugs.
The lack of staff with a good working knowledge and capacity in MHPSS creates considerable challenges. Staff members who are untrained and not fully sensitised to people’s needs and capacities may feel apprehensive about some users and then act inappropriately. In this study, two strategies were used to mitigate these risks: (1) organising refresher training for survey administrators (2) adding a module on mental health and psychological distress that can be adapted to different contexts in the ScoPeO training.
Another issue was related to the use of ScoPeO-Kids tool. In Lebanon, for example, the target population included children with a range of cognitive difficulties. Some of the children struggled with the use of Likert scales in the first questionnaire. Several possible solutions were considered, such as using objects or multimedia tools. These methods were ruled out because they were even more complex than using the Likert scales, requiring understanding of shape, colour, and size of objects used, for example. Instead, the mission used a scale made of cardstock circles and organised incrementally to convey the concept of progression.
Using the Family Quality-of-Life Scale
Within the framework of this study, establishing communication with some users turned out to be extremely difficult, to the extent of making data collection impossible. The hypothesis was formulated that within this framework, interventions towards reinforcing support and patient care for people with disabilities also contributed to improving the quality of life of their caregivers, that is, their families. In Lebanon, the FQoL was, therefore, used to test this hypothesis. Within the scope of this study, the FQoL did not seem to raise major difficulties of appropriation or administration. It seems to be an interesting tool to employ in contexts where ScoPeO cannot be applied. However, a strict protocol should be established, with precise criteria for the use, implementation and administration of the FQoL questionnaire.
Conclusion and Next StepsThe ScoPeO tool developed by HI has been widely used by missions in many different contexts but was yet to be tested in mental health interventions. Yet, assessing quality of life of mental health interventions’ users constitutes a real challenge at several levels, including gaining informed consent.
The second phase of the TMRD project in Lebanon, Madagascar, Sierra Leone and Togo, was therefore taken as an opportunity to test the implementation and limits of the tool. The study did not face unexpected implementing issues and overall gave interesting methodological insights, yet, several challenges due to Covid-19 had to be overcome. We found that using the adult version of ScoPeO tool was feasible as long as comprehensive training was offered for survey team members and that questionnaires are provided in various formats to take account of users’ needs. The ScoPeO-Kids version faced some difficulties in relation to children with serious cognitive delays.
Additional activities had been planned to improve the tool and capture the quality of life as seen and felt by the users in a better way. However, mainly due to the Covid-19 pandemic, some of them had to be cancelled. For example, a qualitative component was planned. This mixed methodology was meant to collect data on maximising the impact of activities and improving service provision.
The study highlighted the limitations of the tool in terms of inclusiveness. In Lebanon, ScoPeO-Kids could not be administered to all the children (notably due to severe cases of cerebral aplasia). A more inclusive method is needed to continue ensuring a person-centred approach. The FQoL, as used here, is one of the possibilities under consideration. Further research had been implemented in Mali (September–December 2022) to identify and test the most appropriate tool and led to the identification of a family level tool associated to an analysis matrix that can be deployed. The current approach gives a quality-of-life general score and individual dimensions’ score. Going forward, there is scope for psychometric analyses to establish the reliability and validity of the ScoPeO tool, regarding both overall quality of life scores and that of individual dimensions of the scale.
1Due to the health context related to Covid 19 pandemic, this field visit was cancelled. The Madagascar field mission had already benefitted from a training and had implemented a ScoPeO study; therefore, implementation was done following a virtual refresher.
2Each child is assessed when he or she enters the programme, and an individual rehabilitation programme is then defined, with varied activities such as physiotherapy, speech therapy, and/or psychological counselling sessions.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
References
Comments (0)