Assessing compliance with Aboriginal and Torres Strait Islander health research guidelines within systematic reviews

MacLean S^1,2, Ritte R¹, Thorpe A¹, Ewen S³, Arabena K¹ (2015) Assessing compliance with Aboriginal and Torres Strait Islander health research guidelines within systematic reviews. Australian Indigenous HealthBulletin 15(4). Retrieved [access date] from http://healthbulletin.org.au/articles/assessing-compliance-with-aboriginal-and-torres-strait-islander-health-research-guidelines-within-systematic-reviews

¹ Indigenous Health Equity Unit, Melbourne School of Population and Global Health, University of Melbourne, Victoria, 3010

² Department of Community and Clinical Allied Health, La Trobe University, Victoria, 3086

³ Melbourne Poche Centre for Indigenous Health, University of Melbourne, Victoria, 3010

Correspondence to: Dr Sarah MacLean, Department of Community and Clinical Allied Health, La Trobe University, Victoria, 3086 ph: (03) 94791374, email: S.MacLean@latrobe.edu.au

Acknowledgements

We are grateful to Priscilla Robinson for constructive comments on an earlier draft. Sarah MacLean’s work on this paper was supported by a fellowship from VicHealth.

Abstract

Objective: Australian research guidelines emphasise that high quality research into Australian Aboriginal and Torres Strait Islander (henceforth Indigenous) health should be conducted in accordance with specific principles. These include: involvement of Indigenous people; making findings accessible; and ensuring that communities benefit from studies in which they participate. Nonetheless, these features are not measured by existing quality appraisal / risk of bias assessment tools that are designed for use in systematic reviews. We report here on the development and trial of a tool for use in systematic reviews of Indigenous health interventions.

Method: Our tool includes measures of compliance with Indigenous health guidelines as well as questions about study design that are common to systematic review tools. The tool was trialled in relation to 13 studies of Indigenous health interventions.

Results: The tool supported a comprehensive assessment of studies. We found strong consistency between reviewers’ evaluations of studies, with only two disagreements on our scales.

Conclusion: Tools used to measures of quality and risk of bias in studies of Indigenous health should assess for rigor in accordance with Indigenous health research principles. Our tool represents an attempt to develop a methodological approach that is acceptable within both non-Indigenous and Indigenous research traditions.

Implications: We hope that researchers will adapt the tool for use in their own reviews of studies concerning the health and wellbeing of Australian Indigenous peoples.

Related to: systematic review, Indigenous health, Indigenous methodologies, quality assessment, risk of bias

View PDF version (PDF 424KB)

 Introduction

Australian Aboriginal and Torres Strait Islander (henceforth Indigenous) peoples have emphasised the importance of ethical and culturally appropriate research conduct. Research involving Indigenous people can be harmful and perpetuate the dynamics of colonisation when undertaken without due involvement from Indigenous peoples or in a manner which is seen as inappropriate by the communities concerned (1-4). Studies can also produce inaccurate findings when Indigenous perspectives and knowledge are not considered (2, 5). Guidelines produced by the Australian National Health and Medical Research Council (NHMRC) (1) recommend that Indigenous health research be underpinned by the values of ‘spirit and integrity’,’ reciprocity’, ‘respect’, ‘equality’, ‘survival and protection’ and ‘responsibility’. Suggestions are provided about how each of these values may be implemented by researchers (1).

We are a team of Indigenous and non-Indigenous researchers working together to conduct a systematic review of studies in Indigenous health. In undertaking a systematic review, it is necessary to identify or develop procedures to ensure that all researchers involved in data extraction ask standardised questions about each included study that are relevant to the research question (6). This enables research teams to reach consistent assessments about the weight that should be given to research findings.

To find a suitable tool for assessing quality and risk of bias in studies using quantitative designs identified through our systematic search, we reviewed resources discussed in a recent paper on systematic reviews in public health research (7). None of these included measures for assessing compliance with key principles enshrined in Indigenous health research guidelines (see Table 1 below).

Table 1. Key components of risk of bias/quality assessment tools for quantitative studies

Drawing on a range of resources and after extensive discussion within the research team, we designed and trialled a new data extraction and quality assessment / risk of bias tool, which we named the ‘Cultural Identity Interventions Systematic Review Proforma’ (12). Our tool incorporates measures assessing reliability, consistency and reporting bias which were adapted from existing systematic review tools (6, 8-10). These tools were selected as sources for relevant measures because they were identified in a paper cited above (7) as appropriate for use in systematic reviews of public health interventions. In addition to these measures, our tool includes a range of questions that our team developed to assess the implementation of key principles identified by the NHMRC as critical to good practice in conducting Indigenous health research (1). Thus, the tool enabled us to assess studies in terms of scientific rigour and also in relation to meeting ethical and methodological standards that are specific to Indigenous health research.

Methods

Our tool includes questions that members of our research team saw as essential to making decisions about quality appraisal and risk of bias assessment. The first section of the tool includes questions about study design that are common to data extraction and quality assessment / risk of bias forms utilised in systematic reviews. We tried to express these questions in non-technical language so they could be answered by researchers with extensive statistical expertise and also those who were less confident in this research method. The second section addresses Indigenous people’s involvement in the research, for example identifying whether Indigenous involvement and ownership of the study extended beyond participation as research subjects, including questions about involvement in development of research question, study design, analyses, interpretation and reporting of results. The third section seeks similar information in relation to Indigenous people’s input to interventions described in the research. To respect the diversity of Indigenous communities we also included questions about contextual factors that may have impacted on study results and the transferability of the intervention described (the fourth section).

In trialling the tool we assigned two members from a team of five researchers to review each of 13 studies of Indigenous health interventions. Papers were divided evenly so that team members reviewed at least two papers with each colleague. Our intention here was to assess whether consensus could be reached on study assessments using the tool between all possible pairings of researchers within our team.

Results

We found strong consistency between team members’ evaluations of studies, with only two disagreements on our five point scale for global assessment of studies. One disagreement was due to a difference in the interpretation of the tool, which we subsequently clarified thorough redrafting our wording. The other inconsistency reflected a difference in research background and cultural worldview. The latter discrepancy was resolved through discussion.

Conclusions

Like other researchers working in Indigenous health, we have found the process of undertaking a systematic literature review problematic and confronting (13, 14). Systematic reviews tend to focus on biomedical disease rather than social and cultural health determinants and are deeply rooted in a quantitative paradigm. They frequently exclude qualitative and other primary studies considered to be of lower evidentiary value than randomised controlled trials. A lack of good quality primary intervention studies according to scientific criteria means that reviews are often inconclusive. Further, systematic reviews tend to overlook the influence of the contexts in which interventions are implemented (15). This is particularly important in Indigenous communities where unique opportunities strongly influence the transferability of interventions or programs described in studies. For example, an intervention in an Indigenous community may owe its success to the availability and willingness of Elders with cultural knowledge to contribute to activities. Lesser support from Elders for the activity might strongly impede the implementation of the intervention elsewhere. For these reasons, and also because of the particular concerns that Indigenous peoples have expressed about research, we did not consider it appropriate to simply replicate tools used with other populations when conducting our systematic review of studies of Indigenous health.

Our tool represents an attempt to navigate some of the challenges that are implicit to undertaking systematic reviews of studies in Indigenous health. We have aimed to develop a methodological approach that is acceptable within both non-Indigenous and Indigenous research traditions.

Implications

Tools used to measures of quality and risk of bias in studies of Indigenous health should assess for rigour in accordance with scientific and also Indigenous health research principles and values. The tool that we developed is available on the website of the Indigenous Health Equity Unit at the University of Melbourne: http://go.unimelb.edu.au/68ra. We hope that other researchers will adapt the tool for their own systematic reviews and that they will find it useful conducting comprehensive assessments of studies of interventions addressing health and wellbeing for Australian Indigenous peoples.

References

1. National Health and Medical Research Council. Values and Ethics: Guidelines for ethical conduct in Aboriginal and Torres Strait Islander Health Research. Canberra: Commonwealth of Australia; 2003.
2. Laycock A, with, Walker D, Harrison N, Brands J. Researching Indigenous Health: a practical guide to researchers. Carlton South, Victoria: The Lowitja Institute; 2011.
3. McDonald E, Priest N, Doyle J, Bailie R, Anderson I, Waters E. Issues and challenges for systematic reviews in indigenous health. Journal of Epidemiology and Community Health. 2010;64(7):643-4.
4. Stewart P, Anderson I, Dunbar T, Shibasaki S, Pyett P, Devitt J. Aboriginal and Torres Strait Islander Participation in the Ethical Review of Health Research. Aboriginal and Islander Health Worker Journal. 2006;30(6):22.
5. Walter M, Andersen C. Indigenous Statistics. Walnut Creek, California: Left Coast Press; 2013.
6. Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 [updated September 2009]. The Cochrane Collaboration; 2009.
7. McCall J, Connor J. Systematic reviews in public health research. Australian & New Zealand Journal of Public Health. 2010;34:343-4.
8. Centre for Reviews and Dissemination. Systematic Reviews: CRD’s guidance for undertaking reviews in health care. Yorik: CRD, University of York; 2009.
9. Scottish Intercollegiate Guidelines Network. Critical appraisal: notes and checklists Edinburgh: Healthcare Improvement Scotland; 2014 [updated 13 October, 201430 September, 2014]. Available from: http://www.sign.ac.uk/methodology/checklists.html.
10. Effective Public Health Practice Project. Quality assessment tool for quantitative studies2009 10 June 2014. Available from: http://www.ephpp.ca/PDF/Quality%20Assessment%20Tool_2010_2.pdf.
11. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews. 2015;4(1).
12. Ritte R, Thorpe A, Ewen S, Arabena K, MacLean S. Cultural Identity Interventions Systematic Review Proforma2015. Available from: http://go.unimelb.edu.au/68ra.
13. McDonald E, Priest N, Doyle J, Bailie R, Anderson I, Waters E. Issues and challenges for systematic reviews in Indigenous health. Journal of Epidemiology & Community Health. 2010;64(7):643-4.
14. Johnston L, Doyle J, Morgan B, Atkinson-Briggs S, Firebrace B, Marika M, et al. A review of programs that targeted environmental determinants of Aboriginal and Torres Strait Islander health. International Journal of Environmental Research and Public Health. 2013;10(8):3518-42.
15. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review – a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005;10:21-34.