PANSS item reliability: Can standardized rater training improve negative subscale reliability?

Background: Paloa et al. (1994) identify the assessment of negative symptoms as important in the assessment of change in overall psychopathology in schizophrenia. The accurate assessment of negative symptoms is also critical in the development of treatment for this debilitating, often under treated and recalcitrant feature of schizophrenia. However the assessment of these symptoms is difficult and numerous studies (e.g., Möller, 2007; Betsen et al, 1996; Norman et al, 1996) have found that negative subscale items in the PANSS are somewhat less reliable than those in the positive and general subscales. Rater error, the subtlety of these symptoms, and the subjective nature of many of these items have been identified as probable culprits. Standardized rater training has been identified as one way to improve overall PANSS reliability. Here we asked if this applied equally to the negative subscale and if these results were durable.

Methods: Results from several large standardized rater training events were analyzed. Training included both applied and didactic components. Raters were asked to rate the PANSS from a video-taped assessment before and after training. The inter-rater reliability was determined using intra-class correlation coefficient (ICC) for pre and post training scores on the negative subscale of the PANSS.  

Results: There appeared to be an improvement in the reliable assessment of negative subscale items after training in all cases examined. The ICC range for the PANSS negative subscale before training ranged from .358 (n=11, p<.001) to .615 (n=30, p<.001). The range of post training ICCs were .764 (n=9, p< .001) to .945 (n=30, p<.001).

Conclusions: Based on the improvement in inter-rater reliability it appears that standardized rater training has a significant impact on the assessment of negative symptoms. It is posited that this effect is related to the framework to discuss and engage with the instrument as well as the exposure to clinical examples. Although every rater has unique clinical experience with the target population, the goal of standardized rater training is to encourage the consistent conceptualization of subscale items in the PANSS instrument.