Adapting the Australian Public Mental Health Performance Indicators to the contextual needs of a remote area child and youth mental health service

Original article (peer reviewed)

Published in the HealthBulletin

Posted on:

5 February, 2009

Issue

Vol 9 No 1, January 2009 - March 2009

Introduction

Until relatively recently, Child and Youth Mental Health Services in Far North Queensland have operated on an inadequate and ad hoc basis, with no available data to describe or evaluate service provision. A Queensland Health review of Child and Youth Mental Health Services operating in Far North Queensland noted that these “operate within an environment where there are few alternative health services with less participation by the NGO sector compared to Brisbane and fewer local General Practitioners, or great distances to access services” [1]. The issues raised by Queensland Health are further accentuated in remote areas where factors such as long distance and extreme weather and road conditions markedly limit access and hence provision of external mental health services. Furthermore, remote service areas of Far North Queensland are characterised by complex, politically and culturally sensitive settings, high levels of poverty and unemployment, poor educational outcomes and housing problems [2]. In order to address the unique needs of remote service, the establishment of a specific Remote Area Child and Youth Mental Health (RACYMH) service in Far North Queensland was identified by Queensland Health as a key priority in 2001.

RACYMH Service Development Background

Commencing November 2001, a project partnership between Queensland Health and the University of Queensland was initiated with the aim of establishing a RACYMH best practice service model for Aboriginal and Torres Strait Islander families living in Far North Queensland. The project objectives were to evaluate and restructure CYMHS in three Health Service Districts: Cape York (Kowanyama, Pormpuraaw, Lockhart River, Aurukun, Napranum, Weipa and Coen), the Torres Strait including Thursday Island and the Northern Peninsula Area (Bamaga, with the satellite communities of Injinoo, New Mapoon, Seisia and Umagico) and Remote Cairns (Cooktown, Hopevale and Wujal Wujal). These sixteen communities vary in population from 200 to 2000.

Under the auspices of the Cairns district the first phase of the project (2002) systematically mapped resources and needs across a range of child services in each of the service communities and included community-identified need, infrastructure, functional access, existing local mental health resources and capacity. The second phase, activated between 2002 and 2004, involved the restructure and reforming of RACYMH as an independent service. The current RACYMH is an outreach service which commenced operation in 2004; its establishment being the key outcome of the project’s second phase [2]. The adaptation of Performance Indicators (PI) from the Key Performance Indicators for Australian Public Mental Health Services framework [3] occurred as part of the third phase (2005 – 2006) of the ongoing RACYMH service evaluation and restructuring project.

Information development from the first two phases of the project informed the third phase activities; specifically, the need for the development of PIs for the specific needs of a remote area service. The project’s first two phases indicated that to capture the essential and unique aspects of child and youth mental health service provision in remote Aboriginal and Torres Strait Islander settings, modifications to the Key Performance Indicators for Australian Public Mental Health Services framework [3], from here on referenced as the national document, were imperative. See Figure 1.
Specifically then, the purpose of this paper is to describe the third phase of this action research project – the process of developing PIs adapted from the national document [3] to meet the operational needs of a RACYMH service functioning in Aboriginal and Torres Strait Islander settings in Cape York, Remote Cairns and the Torres Strait. See Figure 1.

Figure 1. Project timeline (PDF – 77KB) Methodological Rationale

Theoretical background

In 2005, RACYMH service explored the theoretical underpinnings for the ongoing restructuring and evaluation project through Flyvberg’s [4] concept of ‘phronetic social science’. Flyvberg’s thesis espouses that the natural and social sciences make different, though complementary contributions to challenges facing society. Within this context Flyvberg explains that value-rational questions require ‘phronesis’ – prudence or practical wisdom – involving “the analysis of values…as a point of departure for action” [4, p. 57]. The implication for social research is that, in Aristotelian terms, a phronetic researcher aims for pragmatism, focuses on context, the variable or particular, as opposed to the universal and context-independent. Additionally Flyvberg asserts, a contemporary conception of phronesis cannot “be adequate…unless it confronts an analysis of power” [4, p. 88].

Flyvberg’s theoretical insights influenced the project’s third phase in several ways; Flyvberg’s theory provided some of the philosophical basis for the RACYMH project. In terms of value-rational questions, the service was asking “where to now?” and sought to elicit information related to the situational ethics of remote area practice. Flyvberg’s insights also afforded explicit recognition of the complex power dynamics facing health practitioners when engaging with Aboriginal and Torres Strait Islander people residing in remote settings. These dynamics include those operating both within remote communities and those which characterise the ‘health system’ as a bureaucratic institution.

Flyvberg’s theory, expressing the interrelatedness of Episteme (scientific knowledge), Techne (craft/art) and Phronesis (ethics and values) encouraged a conscious critique of the single-pronged bio-medical approach to service delivery and re-conceptualised what a health service might look like where the express aim is to maintain cultural relevance to families in remote Aboriginal and Torres Strait Islander communities. However, it is important to note that adopting Flyvberg’s approach does not involve discounting the role of RACYMH as a clinical service. Indeed, the capacity of the service to provide evidence based therapeutic interventions, such as family therapy, pharmacological therapy and/or behavioural management, rests upon Episteme. The assumption therefore, is not that we need either phronesis or episteme or techne – rather, in the area of healthcare, all three Aristotelian ‘intellectual virtues’ are required in varying degrees in any clinical situation [4], thus providing sound justification for an interdisciplinary and inter-sectorial holistic approach to health service delivery.

Current Policy context

This initiative’s developmental process was guided by Values and Ethics: Guidelines for Ethical Conduct in Aboriginal and Torres Strait Islander Health Research [5]. Two policy documents informed RACYMH development of PIs during 2005 to 2006 – Queensland Health’s Strategic Policy for Aboriginal and Torres Strait Islander Children and Young People 2005 – 2010 [6] and the Commonwealth Department of Health and Ageing’s Key Performance Indicators for Australian Public Mental Health Services [3], as mentioned above in the introduction.

1) Strategic Policy for Aboriginal and Torres Strait Islander Children and Young People 2005 – 2010 [6].
Several principles articulated in this strategic document are of particular relevance to both RACYMH service aims and objectives and the regional practice context and are summarised as follows:

Cultural respect and reconciliation
Rights of children to health care
Holistic community approach
Population-based approach
Building the capacity of health services and communities
Needs-based resourcing

Data obtained from questionnaires and interviews conducted with primary health care staff, district management staff, collaborative partners, stakeholders and RACYMH service team members reflected the above principles. RACYMH service aims to observe these principles in its engagement with families and communities within its capacity, scope and resources.

2) Key Performance Indicators for Australian Public Mental Health Services Framework [3]
This document is based on the National Health Performance Framework [3] and is linked to the strategic directions of the National Mental Health Plan 2003 – 2008 [3]. It outlines a new performance indicator framework for evaluation of public mental health services and suggests that organisational service should be: effective, appropriate, efficient, accessible, continuous, responsive, capable, safe and sustainable. RACYMH adopted this document as the key template for developing PIs that are locally and contextually appropriate for RACYMH service.

Domains

The national document [3] offers a comprehensive model for monitoring and organising the performance of mental health services. It has nine essential domains, each representing a broad area of concern applicable to health service performance. Essentially, each domain requires further specification oriented towards the most significant issues of concern – sub-domains. For example, in Australia’s National Health Performance framework [3], the domain of ‘Effective’ covers the concepts of care, intervention or action. Each concept may be the precursor to one or more indicators [3]. All nine domains of the national document were deemed essential and relevant for RACYMH context and practice.

Sub-domains & Indicators

The national PI document identifies key sub-domains relevant to the delivery of mental health services. Each sub-domain can be regarded as describing a topic of concern, or the most salient aspects of organisation performance. The sub-domains and indicators focus on inpatient care; the service dimensions assume a bio-medical approach to health care provision with a focus on specialised public clinical mental health [3]. The following factors informed the type of sub-domains and indicators that would be feasible for the RACYMH service:

RACYMH is at an early stage of service development.
Service activity monitoring in Queensland is currently limited to the Consumer Event Service Application data (although there is ongoing work to integrate outcome data).
The need to reflect a commitment to service collaboration and partnerships in communities, across government and non-government agencies and research institutes.
Identifying items that were too complex to measure at this stage, for example quantifying community activity outcomes.

Methods:

The development of RACYMH PIs involved five interrelated processes and specifically focused upon service development and service providers. The first step involved identifying a theoretical framework for evaluation – the Key Performance Indicators for Australian Public Mental Health Services document [3]; recognising the current integrating policy context; and establishing and seeking direction from steering and advisory committees comprised of service providers and health service researchers. The Steering Committee acted like an overseeing regulatory body that provided strategic advice and helped disseminate information from this project to wider networks. The Evaluation Committee was set up to provide support and advice on research questions and methodological issues.

The second step concerned the conduct of questionnaires with stakeholders in other health services (for example, Education Queensland), District Managers and Primary Health Care Centre staff and interviews with all RACYMH team members [7]. In the third step, feedback obtained from stakeholder questionnaires and practitioner interviews was utilised to assess RACYMH service performance in line with drafted indicators. A critical examination and modification of the national document followed. Specialised public clinical mental health is recognised in the document. Therefore, the challenge for RACYMH service was to develop indicators that would capture the broader type of service provision required in remote Aboriginal and Torres Strait Islander settings with a comparable level of rigour and reliability. In this respect the national document was helpful in providing a commentary of intended interpretation of each domain, sub-domain and service indicator.

Questionnaires

Questionnaires were distributed to three groups: Stakeholders, District Management and Primary Health Care Centre Staff [7]. While the national domains were not hierarchical, some appeared more relevant than others in terms of a newly established remote area service and therefore more likely to elicit feedback. For practical purposes, the Evaluation Committee recommended that questionnaires should focus on four domains: effective, sustainable, accessible and appropriate. Thirty-one questionnaires were distributed across the three groups; 18 questionnaires (58%) were completed and returned (2 from District Contacts, 8 from Primary Health Care and 8 from Collaborative Partners).

Interviews

Seven RACYMH staff, i.e., practitioners working with the service, consented to do a reflective narrative interview on their clinical practice, service issues and community dilemmas. Interviews [7], thus recorded and transcribed provided detailed narratives of the current model of service, ethical dimensions of remote area child and youth mental health practice, personal values, professional and service development issues, needs and concerns.

Questionnaire Analysis

Questionnaire data analysis involved sorting responses according to key themes under the domains: effective, sustainable, accessible and appropriate. Themes not included under existing domains were also recorded. For example, respondents were asked to indicate essential competencies for RACYMH workers in remote areas. In accordance with respondents’ responses, one of the questionnaire themes became “essential competencies”.

Interview Analysis

Individual interviews were analysed in detail and with a particular lens to articulate the key aspects of service activity that would define Performance Indicator measures for RACYMH service. For example, interview data highlighted the crucial role of Aboriginal and/or Torres Strait Islander mental health workers and the need to strengthen collaborative practice: “RACYMH Indigenous worker is regularly involved in consumer/family assessment, management and care”. Consequently the PIs for RACYMH were adapted to reflect these findings.

The key value of the interview data was its contribution to clarifying and refining the current service model. It also ensured that directions for service development were constructed through a collective, participatory process of people working in the service and stakeholders in the community. Critical themes arising from interview analysis included: the role of the Aboriginal and/or Torres Strait Islander health workers; professional development needs; ethical challenges faced by RACYMH workers; mentorship and team cohesion; community engagement; worker motivation and longevity. Subsequently, a specific supervision framework was developed and introduced into the RACYMH service model. This involved reflective practice, in which the practitioner’s daily practice and decisions are examined and explored in order to identify assumptions which may be limiting skills development and to draw upon qualities such as creativity and intuition in practice. Critically, the approach also acknowledges and follows Flyvberg’s thinking [4] – that ethical practice must take into consideration issues of social, cultural and political power and recognise the health practitioner’s role within such dynamics.

Results

The most significant results arising from the process of developing of Performance Indicators were:

Introduction of two service dimensions – Community Engagement and Therapeutic Care
Two concurrent intersecting dimensions of practice, Community Engagement (CE) and Therapeutic Care (TC) were established for RACYMH service. To capture the breadth and scope of service activity in communities informed by community consultation, practitioner interviews and stakeholders’ feedback, an appreciation of community engagement practice was vital. Understanding the relevance of defining, developing, measuring and evaluating ‘therapeutic care’ and ‘community engagement’ practices as essential service components was one of the unique contributions of this project.

Refinement of the RACYMH model
The action research process of observation, data collection, restructuring, evaluation, reflection and review led to conceptual shifts and clarification of the service and what was achievable with current resources. Thus the inclusion of ‘Community Engagement’ vis-à-vis ‘community development’ as a core activity within the RACYMH service model and the expansion of clinical intervention to include the broader concept of ‘Therapeutic Care’ were imperative additions. The dimensions of CE and TC are additional to the prescribed categories in the national document and sit between each of the nine national ‘Domain’ and ‘Sub-domain’ levels of performance measurement. The intersecting area between CE and TC represents the ‘space’ from which RACYMH operates. Illustrated in Figure 2.

Figure 2: Positioning Community Engagement and Therapeutic Care dimension between the domain and sub-domain (PDF – 56KB)

Discussion:

The outcome of the project’s third phase activities was the modification of the national document [3] to incorporate the unique contextual needs of RACYMH service. The fundamental adaptations were the establishment of two additional dimensions of practice within the RACYMH model: Community Engagement (CE) and Therapeutic Care (TC). See Figure 2.

There were several reasons for the incorporation of CE and TC into the RACYMH service performance framework. Firstly, the current RACYMH service model of care was established in response to a process of broad scoping analyses which included community consultations during 2002, 2003 and 2004. The need for a culturally oriented and contextually sensitive, yet comprehensive, service model to include early intervention, group work and community capacity strengthening activities, in addition to quality clinical interventions, emerged as critical themes in the feedback both from community members and key community organisations. In response, the current model of care integrates non-clinical therapeutic and relationship building activities aimed at facilitating community participation in service development and responses to community needs as they arise. One example is RACYMH service work done in collaboration with justice group elders and other community members in times of crisis, such as the trauma a community experiences following a suicide or a riot. Adding CE and TC also served to represent differences between what ‘service performance’ might mean within a secondary or tertiary health care setting and what ‘service performance’ might mean for a health care service operating in remote Aboriginal and/or Torres Strait Islander settings. See Figure 2. The difference between these contexts is the manner in which the notion of holism is integrated into health care practice. Although hospital based care and indicators generate meaningful measurements while focusing upon the individual, this can be conceptually problematic in relation to the delivery of health care services in remote Aboriginal and/or Torres Strait Islander communities.
In the current Social and Emotional Well Being Framework [8] the concept of ‘connectedness’ and ‘community’ are fundamental to Aboriginal and/or Torres Strait Islander concepts of individual health and healing. Wellness of the individual is related to ‘wholeness’ that is, the totality of the person’s experience and environment. This holistic notion of health and wellbeing has particular resonance when considering children and young people in whom health, illness, growth and development are most tangibly determined by connectedness to significant others. In Flyvberg’s [4] framework the critical emphasis is on the interrelatedness between multiple sets of knowledge or practices i.e., scientific knowledge; technical knowledge and ethical knowledge. These practices then set the scenario for social sciences research to contribute to a context’s capacity by outlining value based goals. Thus CE and TC also link to the principle of a ‘holistic community approach’ expressed in the Queensland Health Strategic Policy for Aboriginal and Torres Strait Islander Children and Young People 2005 – 2010 [6].

Limitations of the project:

A limitation of the research was that the action research cycle in this phase did not aim to generate consumer data of the service. This was primarily because the stability, regularity and predictability of the service in the communities were just beginning to be established. The third phase of the project focused upon service development and service providers and articulation of practice frameworks and performance indicators. The consumer/care dimensions (i.e. client and family outcomes and community perceptions) will form the focus of the next research cycle of evaluation.

The other limitation of this project was the relatively small number of informants; data was provided by 25 people working actively with children and families in Aboriginal and/or Torres Strait Islander settings in remote area mental health in Far North Queensland. However, to some extent, the small number of informants, difficult to access in remote work, was countered by the in-depth nature of the interviews and by the quality of the data generated.

Conclusion:
RACYMH service capacity and practice frameworks have been restructured and reformed since Dec 2001. The ongoing action research has helped build the rigor of a small, but significant service operating in remote Aboriginal and/or Torres Strait Islander settings. This paper outlined the phase that relates to the development of performance indicators for RACYMH service based on the national document [3]. For RACYMH service to be locally responsive, contextually sensitive and culturally oriented it is critical that community engagement and therapeutic care act as concurrent complimentary streams on which performance of the service can be measured.

References

1 Rogan K (2005) Child and youth mental health plan, Northern Zone. Cairns: Queensland Health
2 Santhanam R, Hunter E, Wilkinson Y, Whiteford H, McEwan A (2006) Care, community, capacity: rethinking mental health services in remote Indigenous settings. Australian Journal of Primary Health;12:51-56
3 National Mental Health Working Group Information Strategy Committee Performance Indicator Drafting Group (2005) Key performance indicators for Australian public health services. (Discussion paper 5) Canberra: Department of Health and Ageing. Australian Government
4 Flyvberg B (2001) Making social science matter: why social inquiry fails and how it can succeed again. Cambridge: Cambridge University Press
5 National Health and Medical Research Council (2003) Values and ethics: guidelines for ethical conduct in Aboriginal and Torres Strait Islander health research. Canberra: National Health and Medical Research Council
6 Queensland Health (2005) Strategic policy for Aboriginal and Torres Strait Islander children and young people’s health 2005-2010. Brisbane: State of Queensland, Queensland Health
7 Santhanam R, McEwan A (2007) Remote area child and youth mental health service evaluation report: January 2005 – January 2007. Cairns: Mental Health Branch Queensland Health
8 Social Health Reference Group (2004) Social and emotional well being framework: a national strategic framework for Aboriginal and Torres Strait Islander mental health and social and emotional well being 2004-2009. Canberra: Australian Government