Evaluation of the SMARTACOP score for stratifying pneumonia risk in a Torres Strait Islander population

McIver L¹, Berghofer H¹, Tongpao L¹, Kippin A¹, Buettner P² (2013)
Evaluation of the SMARTACOP score for stratifying pneumonia risk in a Torres Strait Islander population. Australian Indigenous HealthBulletin 14(1). Retrieved [access date] from http://healthbulletin.org.au/articles/evaluation-of-the-smartacop-score-for-stratifying-pneumonia-risk-in-a-torres-strait-islander-population

¹Torres Strait and Northern Peninsula Area Health Service, QLD. Contact details: ph: 07 4069 0200; fax: 07 4069 0219; email: lachlan.mciver@gmail.com
²James Cook University, QLD

Abstract

Objective: To review the accuracy and utility of the SMARTACOP score in classifying pneumonia risk in a Torres Strait Islander population.

Methods: Retrospective chart audit in the setting of a remote hospital in northern Australia with a majority Indigenous population. Eligible for inclusion in the study were 70 adult Torres Strait Islander patients admitted to Thursday Island Hospital with pneumonia between 1998 and 2010. The main outcome measure was correlation between retrospectively-generated SMARTACOP scores and patient outcomes.

Results: The SMARTACOP score, in its current form, appeared to correlate poorly with patient outcomes in this population, with a positive predictive value of 19.0% (95%-CI = 5.5, 41.9), but does appear to be useful in excluding serious disease, given the negative predictive value of 100% (95%-CI = 92.8, 100).

Conclusion: The SMARTACOP score does not appear to accurately predict severity of pneumonia in Torres Strait Islander patients in the Torres Strait.

Implications: A more useful and practical score in the Torres Strait islands would combine the need for patient retrieval along with the need for invasive ventilation or inotropic support in stratifying pneumonia risk in this population; this may also be a useful modification in other rural and remote hospital settings.

View PDF version (PDF – 709KB)

Introduction

Community-acquired pneumonia is a major cause of morbidity and mortality in Australia, with tens of thousands of hospitalisations for lower respiratory tract infections occurring each year [1]. A disproportionate burden of the disease is borne by Aboriginal and Torres Strait Islanders, who in recent years have experienced significantly higher rates of hospitalisation (rate ratio, RR 3.5) and deaths (RR 1.5) due to lower respiratory tract infections than non-Indigenous Australians [1]. The causes of this excess burden in the Australian Indigenous population include (but are not limited to) higher rates of health risk behaviours (such as smoking and alcohol abuse), co-morbidities (particularly chronic airways disease, diabetes, chronic kidney disease and chronic liver disease), and exposure to different pathogens in remote and/or tropical areas [2]. In an effort to improve outcomes in the management of pneumonia, significant attention has been focussed recently on the use of evidence-based scoring systems for classifying disease severity.

The CORB score, based on four clinical markers of pneumonia severity (confusion, oxygenation, respiratory rate and blood pressure), was developed for use in the emergency department setting to identify patients at risk of acute deterioration [3]. The CURB65 score combines confusion, uraemia, respiratory rate and blood pressure with age to give an overall severity score and the Pneumonia Severity Index (PSI) incorporates several more demographic and clinical variables as well as co-morbid illnesses; both scores are used to predict 30-day mortality [4,5]. The SMART-COP score, based on clinical and radiographic indicators of disease severity (systolic blood pressure, multilobar consolidation, serum albumin level, respiratory rate, tachycardia, confusion, oxygenation and arterial pH), aims to identify patients with pneumonia who may require intensive respiratory or vasopressor support (IRVS) [6]. The SMART-COP score divides pneumonia patients into four groups based on their likelihood of needing IRVS – low risk (negligible likelihood of needing IRVS), moderate risk, (1 in 8), high risk (1 in 3) and very high risk (2 in 3). A recent modification of the SMART-COP score, developed in the Northern Territory for use in tropical regions of Australia and known as the SMARTACOP score, doubles the relative weight of the serum albumin score and adds Indigenous ethnicity as a risk factor (see Table 1) [7]. Each of these systems is included in version 14 of the Therapeutic Guidelines: Antibiotic, which provides an evidence basis for the pharmacological management of pneumonia in Australia. This commonly-used set of Guidelines suggests that the CORB and SMART-COP scores may be the preferable systems for use in this country, including the modified SMARTACOP score for tropical regions [8].
Table 1: SMARTACOP score for predicting pneumonia severity in Indigenous patients in Australia

*Intensive respiratory/vasopressor support

The rationale for this study was based on the following four key premises: the epidemiology of pneumonia is different in tropical areas of Australia, including the Torres Strait; despite both being classed as ‘Indigenous’, Torres Strait Islanders are ethnically distinct from mainland Aboriginal people, thus clinical scoring systems such as the SMARTACOP score developed for use in the Aboriginal population cannot be assumed to apply to Torres Strait Islanders; empirical observations by medical officers working in the Torres Strait suggested that the SMARTACOP score did not correlate well with outcomes for Torres Strait Islander patients with pneumonia; and in the Torres Strait region, as in other rural and remote hospital and primary care settings, the absence of an intensive care unit mandates that any patient requiring assisted ventilation or inotropic support requires retrieval to tertiary-level care in Cairns. Thus, an additional important point when considering patient outcomes is that transfer (out of Thursday Island Hospital) is itself a significant clinical end-point for patients with pneumonia.

With these premises in mind, we undertook a retrospective evaluation of the SMARTACOP score for patients with pneumonia in the Torres Strait Islands, in order to examine how accurately this tool predicted outcomes for this unique Indigenous population in a remote geographical location.

Methods

We conducted a retrospective audit of all adult patients admitted to Thursday Island Hospital with a diagnosis of ‘pneumonia’ or ‘pneumonia unspecified’ between November 1998 and October 2010. Thursday Island Hospital is the referral hospital for the Torres Strait and Northern Peninsula Area Health Service District. Permission to conduct the study was granted by Meriba Dhoeynidhay Yabu – the Torres Strait and Northern Peninsula Area Community Health Council – and ethics approval provided by the Cairns and Hinterland Health Service District Human Research Ethics Committee.

The audit took the form of a chart review, with basic patient demographic information recorded along with clinical details and the results of radiology and pathology tests, in particular positive microbial cultures from sputum and blood. SMARTACOP scores were retrospectively generated for each patient based on their clinical data at the time of admission (see Table 1 for SMARTACOP score criteria), and these scores were compared with the outcome for each patient (death, discharge from Thursday Island Hospital, transfer to tertiary-level care at Cairns Base Hospital, requirements for intensive ventilatory and/or circulatory support). Inclusion criteria were the patient identifying themselves as being of Torres Strait Islander origin, a diagnosis of pneumonia based on at least two suggestive symptoms (cough, dyspnoea, fever, rigors, chest discomfort) and supportive radiological evidence of pneumonia; exclusion criteria were pre-morbid immunosuppression and/or active orders limiting life-sustaining treatment. These criteria, with the exception of specifying Indigenous ethnicity, conformed to the previous studies that developed the SMART-COP and SMARTACOP scores.

Results

There were 140 patients admitted to Thursday Island Hospital with a diagnosis of pneumonia over the study period, of which 70 were adults who met the criteria described above.

The demographic characteristics, co-morbidities and clinical outcomes of the Torres Strait Islander patients are listed in Table 2.
Table 2: Descriptive analysis of study cohort (adult Torres Strait Islander patients with pneumonia; n=70)

*SD = standard deviation

Note:

1. With respect to the size of the sample population and calculations of appropriate statistical power, as this was a retrospective chart audit, a power calculation would have had no effect on the design of the study. An a posteriori power calculation would have required specifying acceptable values for sensitivity and specificity for such a screening tool. If we take a sensitivity of 100% (which is actually what we observed if we collate low and moderate risk and high and very high risk) then a sample size of 36 IRVS patients would result in a 95% confidence interval with a width less than 0.1. If we consider a specificity of 75% (which was approximately observed with the collated risk groups) then a sample size of 306 non-IRVS patients would have been required for a 95% confidence interval with a width less than 0.1. These are therefore useful parameters for follow-up studies to further test the hypotheses that existing pneumonia risk scores are inappropriate for Torres Strait Islander patients and the remote hospital context.

Table 3 lists the clinical characteristics and outcomes of the study patients in terms of their retrospectively-generated SMARTACOP categories.
Table 3: Association between SMARTACOP score and characteristics of patients (n=70)

*SD = standard deviation

Note:

1. ** p-values indicate significance at the ≤0.05 level

2. The SMARTACOP scores in this sample population showed a symmetrical distribution. The median score was 5.5 (inter-quartile range = 3 to 7; range 1 to 10). The distribution of the outcome variables can be seen in Table 3. Experimental modifications to the SMARTACOP scores for this sample population were not deemed useful, as our small sample size with ‘only’ 4 positive outcome would not have allowed a conclusive result (see note in Discussion below).

As can be seen from Table 3, the SMARTACOP scores bore little relationship with the eventual patient outcomes. In particular, it is worth highlighting that only five patients required IRVS; all of these patients were in the ‘very high risk’ group. Based on the SMARTACOP scores, approximately two of the ‘medium risk’ group, six of the ‘high risk’ group and 14 of the ‘very high risk’ group would have been expected to require IRVS.

An additional interpretation of these results in terms of the predictive value of the SMARTACOP scores with respect to clinical outcomes for this study population suggests that the score has a low positive predictive value (PPV) but very high negative predictive value (NPV), of which more below. The clinical implications of this, as well as the effects of modifying the score to include ‘transfer to referral hospital’ on its predictive value are discussed in more detail below.

Discussion

The key finding from this study is that, on retrospective analysis, SMARTACOP scores do not appear to accurately predict the need for IRVS for this study population (i.e. Torres Strait Islander patients admitted to Thursday Island Hospital with pneumonia). These findings suggest that SMARTACOP scores overestimate the risk of Torres Strait Islander patients with pneumonia requiring IRVS by a significant margin; however the high NPV of the score suggests that it nevertheless may have a useful role in ‘ruling out’ serious illness.

There may be several possible causes of the apparent discrepancy between the accuracy of the score in predicting outcomes in this population compared to elsewhere in Australia (e.g. the Northern Territory, where the modified SMARTACOP score was developed and tested). The first is, of course, error due to the limitations of this study – namely the relatively small numbers of patients eligible for inclusion in the statistical analysis. A second reason for why these results may differ from studies elsewhere in Australia is the epidemiology of the disease, in particular the pathogenic organisms causing pneumonia. While once again our analysis was necessarily limited by the data collected at the time of each admission, and for the majority of cases (47/70, 67.1%) there was either normal respiratory flora alone on sputum culture or no microbiological result at all, nevertheless the most common bacteria cultured from either blood or sputum were Streptococcus pneumonia, Burkholderia pseudomallei (the soil saprophyte that causes melioidosis which is known to occur in the Torres Strait Islands),[9] Mycoplasma pneumonia, Staphylococcus aureus and Haemophilus influenzae – roughly the same suite of organisms implicated in large pneumonia studies in Far North Queensland, the Northern Territory and tropical north of Western Australia [2,10,11]. These results almost certainly underestimate the true incidence of each pneumonia-causing pathogen (due to the low rate of blood and sputum sample collection and culture yields) and indicate the need for more enthusiastic pursuit of microbiological diagnoses in order to guide antimicrobial therapy and limit the use of broad-spectrum antibiotics at the study site. The role of serum and urinary antigen testing in rural hospitals such as Thursday Island is yet to be firmly established, given the time taken for the specimen to be transported and processed and a result generated, but as such tests become less expensive and available on-site they may well find a place as a useful adjunct to bacterial culture in the primary and secondary care context [12].

Given that, in the rural hospital setting, the need to evacuate patients to tertiary-level care is an important clinical end-point in itself, consideration was given to the potential usefulness of a modified SMARTACOP score which incorporated ‘need for evacuation’ as an outcome (need for IRVS notwithstanding). Were this outcome to be included, an additional two patients in this group (one from each of the ‘high’ and ‘very high’ risk SMARTACOP categories) could be said to have suffered significant or severe illness, requiring intensive respiratory support, intensive vasopressor support and/or evacuation.

The impact of this modification – including ‘transfer to referral hospital’ as a clinical endpoint – on the predictive value of the score (comparing the ‘very high risk’ group to the three other groups (‘low’, ‘medium’ and ‘high’ risk) improves the PPV of the score from 19.0% (95%-confidence interval = 5.5, 41.9) to 23.8% (95%-CI = 8.2, 47.2), with a slight depreciation in the NPV from 100% (95%-CI = 92.8, 100) to 98% (95%-CI = 89.2, 100). Table 4 displays the associations between the original and ‘modified’ scores and outcomes by risk group, from which these predictive values were derived.
Table 3: Association between SMARTACOP score and characteristics of patients (n=70)

*Significant at the ≤0.05 level

For a rural hospital such as Thursday Island, this hypothetical modified score is perhaps a more simple and useful way of stratifying patients with pneumonia. Using the SMARTACOP criteria, with the same relative weights for each component of the score but with different categories, such as ‘low risk’ (SMARTACOP score ≤4: likely to be managed entirely within the rural hospital context) or ‘high risk’ (SMARTACOP score ≥4: likely to need ventilatory and/or circulatory support and/or retrieval to tertiary-level care) may enable clinicians in rural and remote areas to anticipate the need for retrieval at an earlier stage in the patient’s management. It is pertinent to note that the authors of the initial SMART-COP study did propose an alternative system for use in the primary care setting (ie. office-based general practice) which excluded the measurements of serum albumin, arterial oxygen saturation and pH, resulting in a modified SMRT-CO score [6].

The rural hospital context is different from both the traditional ‘primary care’ setting (for which the SMRT-CO score was developed) and the tertiary hospital setting (in which the CORB score was devised) in that it generally incorporates both acute care and inpatient management capabilities, as well as variable levels of laboratory and radiology support, but generally lacks intensive care facilities as previously discussed. There is thus a strong argument to be made for the use of a pneumonia risk stratification system specific to this context, which incorporates outcomes such as evacuation of a patient rather than just the likelihood of needing assisted ventilation or inotropes. While it is self-evident that rural hospitals are a heterogeneous group with different capabilities and retrieval of a patient to a tertiary centre is dependent on many factors; nevertheless it must be recognised that clinical scoring systems developed in primary care and tertiary hospital settings do not necessarily apply to the rural hospital patient or clinician.

Conclusion

This study demonstrates that the SMARTACOP score does not accurately predict outcomes for Torres Strait Islander patients admitted to Thursday Island Hospital with pneumonia, although it may still be of value in excluding severe illness. A simplified, modified score, based on the same SMARTACOP clinical criteria, but that classifies patients as either ‘high risk’ or ‘low risk’ based on the probability of a patient requiring either ventilatory support, inotropes or evacuation to tertiary-level care may be more appropriate for settings such as Thursday Island.

While the results of this study are applicable only to Torres Strait Islander patients with pneumonia admitted to Thursday Island Hospital and could not necessarily be extrapolated to include other rural hospitals or other ethnic groups, this model of evaluation is eminently repeatable for other, similar facilities. In the longer term, and noting the small population that was reviewed in this study to generate the above hypothesis of the utility of a modified score, a prospective, multi-centre trial of rural hospitals could examine the utility of a modified scoring system which indicates the likelihood of a patient requiring assisted ventilation, inotropes or evacuation, these being the most significant clinical end-points for patients who are to survive severe pneumonia in the rural hospital setting.

Acknowledgements

The authors would like to acknowledge Meriba Dhoeynidhay Yabu – the Torres Strait and Northern Peninsula Area Community Health Council – for their support of this project.

References

1. Australian Government Department of Health and Ageing. Vaccine preventable diseases and vaccination coverage in Aboriginal and Torres Strait Islander people, Australia, 2003 to 2006. Commun Dis Intell 2008; 32(Supplement): 18-20
2. Elliott JH, Anstey NM, Jacups SP, Fisher DA, Currie BJ. Community-acquired pneumonia in northern Australia: low mortality in a tropical region using locally-developed treatment guidelines. Int J Infect Dis 2005; 9: 15-20
3. Buising KL, Thursky KA, Black JF, MacGregor L, Street AC, Kennedy MP, Brown GV. Identifying severe community-acquired pneumonia: a simple clinical prediction tool. Emerg Med Australas 2007; 19(5): 418-426
4. Lim WS, van der Eerden MM, Laing R et al. Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax 2003; 58: 377-382
5. Fine MJ, Auble TE, Yealy DM et al. A prediction rule to identify low risk patients with community acquired pneumonia. N Engl J Med 1997; 336: 243-250
6. Charles PG, Wolfe R, Whitby M, Fine MJ, Fuller AJ, Stirling R, et al. SMART-COP: a tool for predicting the need for intensive respiratory or vasopressor support in community-acquired pneumonia. Clin Infect Dis 2008; 47(3): 375-384
7. Davis JS, Cross GB, Charles PGP, Currie BJ, Anstey NM, Cheng AC. Pneumonia risk stratification in tropical Australia: does the SMART-COP score apply? Med J Aust 2010; 192(3): 133-136
8. Therapeutic Guidelines Limited. Community-acquired pneumonia in adults: patient assessment in Therapeutic Guidelines: Antibiotic (Version 14). Melbourne: Therapeutic Guidelines Limited; 2010
9. Faa AG, Holt PJ. Melioidosis in the Torres Strait Islands of Far North Queensland. Comm Dis Intell 2002; 26: 279-283
10. Rémond MGW, Ralph AP, Brady SJ, Martin J, Tikoft E, Maguire GP. Community-acquired pneumonia in the central desert and north-western tropics of Australia. Intern Med J 2010; 40: 37-44
11. Thompson JE. Community acquired pneumonia in north eastern Australia – a hospital based study of Aboriginal and non-Aboriginal patients. Aust N Z J Med 1997; 27(1): 59-61
12. Sordé R, Falcó V, Lowak M, Domingo E, Ferrer A, Burgos J et al. Current and potential usefulness of pneumococcal urinary antigen detection in hospitalized patients with community-acquired pneumonia to guide antimicrobial therapy. Arch Intern Med 2011; 171(2): 166-172