Advertisement

Reliability assessment and validation of the Melasma Area and Severity Index (MASI) and a new modified MASI scoring method

Published:April 16, 2010DOI:https://doi.org/10.1016/j.jaad.2009.10.051

      Background

      The Melasma Area and Severity Index (MASI), the most commonly used outcome measure for melasma, has not been validated.

      Objective

      We sought to determine the reliability and validity of the MASI.

      Methods

      After standardized training, 6 raters independently rated 21 patients with mild to severe melasma once daily over a period of 2 days to determine intrarater and interrater reliability. Validation was performed by comparing the MASI with the melasma severity scale. The darkness component of the MASI was validated by comparing it with the difference between mexameter scores for affected versus adjacent normal-appearing skin. The area component of the MASI was validated by comparing it with the area of each section of the face determined by computer-based measurement software.

      Results

      The MASI score showed good reliability within and between raters and was found to be valid when compared with the melasma severity scale, mexameter scores, and area measurements. Homogeneity assessment by raters showed the least agreement and can be removed from the MASI score without any loss of reliability.

      Limitations

      Patients were limited to Hispanic, African, and Asian backgrounds.

      Conclusion

      The MASI is a reliable measure of melasma severity. Area of involvement and darkness are sufficient for accurate measurement of the severity of melasma and homogeneity can be eliminated.

      Key words

      Abbreviations used:

      ANOVA (analysis of variance), MASI (Melasma Area and Severity Index), MSS (melasma severity scale)
      • The Melasma Area and Severity Index (MASI) is a reliable measure of the severity of melasma, showing temporal stability and internal consistency.
      • The melasma severity scale, colorimetric readings of pigmentation intensity, and area measurement using computer software established validation of the MASI.
      • Measurement of darkness and area of involvement is sufficient in determining severity of melasma, whereas homogeneity is unreliable and should be eliminated from the MASI.
      • A new modified MASI score is recommended as a result of this study.
      Melasma is a common, persistent disorder of hyperpigmentation affecting a significant portion of the population, particularly women with skin types IV to VI living in areas with intense ultraviolet radiation.
      • Grimes P.E.
      Melasma: etiologic and therapeutic considerations.
      In all, 8.8% of the female Hispanic population was found to have melasma in one study and an additional 4% had it in the past.
      • Werlinger K.D.
      • Guevara I.L.
      • Gonzalez C.M.
      • Rincon E.T.
      • Caetano R.
      • Haley R.W.
      • et al.
      Prevalence of self-diagnosed melasma among premenopausal Latino women in Dallas and Fort Worth, Tex.
      A significant impact on quality of life has also been demonstrated, including a negative effect on the patient's emotional well-being and social life.
      • Balkrishnan R.
      • McMichael A.J.
      • Camacho F.T.
      • Saltzberg F.
      • Housman T.S.
      • Grummer S.
      • et al.
      Development and validation of a health-related quality of life instrument for women with melasma.
      • Dominguez A.R.
      • Balkrishnan R.
      • Ellzey A.R.
      • Pandya A.G.
      Melasma in Latina patients: cross-cultural adaptation and validation of a quality-of-life questionnaire in Spanish language.
      Conventional treatment of melasma includes elimination of any causative factors coupled with the use of sunscreens and hypopigmenting agents, often in combination with other therapies. Despite the availability of these measures, melasma is often recalcitrant to treatment and frustrating for both patients and clinicians. To develop new agents for melasma, valid and reliable outcome measures are critical to determining clinical severity of disease and the significance of treatment results when performing clinical trials. These outcome measures should be easy to learn, responsive, inexpensive, and applicable worldwide. Unfortunately, no such measure exists for melasma.
      The Melasma Area and Severity Index (MASI), an outcome measure developed to provide a more accurate quantification of the severity of melasma and changes during therapy, was developed by Kimbrough-Green et al,
      • Kimbrough-Green C.K.
      • Griffiths C.E.
      • Finkel L.J.
      • Hamilton T.A.
      • Bulengo-Ransby S.M.
      • Ellis C.N.
      • et al.
      Topical retinoic acid for melasma in black patients: a vehicle-controlled clinical trial.
      who based it on a similar scoring system devised for psoriasis (Fig 1). The MASI score is calculated by subjective assessment of 3 factors: area (A) of involvement, darkness (D), and homogeneity (H), with the forehead (f), right malar region (rm), left malar region (lm), and chin (c), corresponding to 30%, 30%, 30%, and 10% of the total face, respectively. The area of involvement in each of these 4 areas is given a numeric value of 0 to 6 (0 = no involvement; 1 = <10%; 2 = 10%-29%; 3 = 30%-49%; 4 = 50%-69%; 5 = 70%-89%; and 6 = 90%-100%). Darkness and homogeneity are rated on a scale from 0 to 4 (0 = absent; 1 = slight; 2 = mild; 3 = marked; and 4 = maximum). The MASI score is calculated by adding the sum of the severity ratings for darkness and homogeneity, multiplied by the value of the area of involvement, for each of the 4 facial areas:
      MASItotalscore=0.3A(f)[D(f)+H(f)]+0.3A(lm)[D(lm)+H(lm)]+0.3A(rm)[D(rm)+H(rm)]+0.1A(c)[D(c)+H(c)]


      Figure thumbnail gr1
      Fig 1Melasma Area and Severity Index (MASI). A, Area; D, darkness, H, homogeneity; L, left, R, right.
      Reprinted with permission from Kimbrough-Green et al.
      • Kimbrough-Green C.K.
      • Griffiths C.E.
      • Finkel L.J.
      • Hamilton T.A.
      • Bulengo-Ransby S.M.
      • Ellis C.N.
      • et al.
      Topical retinoic acid for melasma in black patients: a vehicle-controlled clinical trial.
      The total score range is 0 to 48. The MASI is the most commonly used outcome measure for melasma trials but has never been validated. The purpose of this prospectively designed study was to determine the reliability and validity of the MASI score.

      Methods

      This prospective study was approved by the institutional review board of the University of Texas Southwestern Medical Center, Dallas, and all patients gave written informed consent. Six raters were recruited for the study, including 5 dermatologists and one dermatology resident. Two raters had extensive experience, two had minimal experience, and two had no experience scoring the MASI. The raters were given a 30-minute lecture and slide presentation describing the MASI; shown examples of different levels of area, darkness, and homogeneity; and shown 4 practice images of patients with melasma. These images were scored by raters, after which the answers were discussed. The raters were also shown an image of each level of the melasma severity scale (MSS) (Fig 2), which is a nonvalidated, simple, easy-to-use categorical scale (range: 0 = none, 1 = mild, 2 = moderate, and 3 = severe) developed for a large multicenter trial using a combination cream for patients with melasma.
      • Taylor S.C.
      • Torok H.
      • Jones T.
      • Lowe N.
      • Rich P.
      • Tschen E.
      • et al.
      Efficacy and safety of a new triple-combination agent for the treatment of facial melasma.
      Figure thumbnail gr2
      Fig 2Melasma severity scale. A, 0 = Clear. B, 1 = Mild. C, 2 = Moderate. D, 3 = Severe.
      Reprinted with permission from Cutis 2003;72:67-72.
      • Taylor S.C.
      • Torok H.
      • Jones T.
      • Lowe N.
      • Rich P.
      • Tschen E.
      • et al.
      Efficacy and safety of a new triple-combination agent for the treatment of facial melasma.
      In all, 21 patients with mild to severe melasma (MSS score 1-3) were recruited from local dermatology clinics. Inclusion criteria included bilateral epidermal or mixed melasma and age 18 years or older. The 6 raters examined the patients independently and rated the patient's degree of melasma using the MASI and the MSS. The following day, the same patients were seated in different rooms and the raters scored each patient again.
      All patients completed a short demographic questionnaire and were photographed using a stereotactic face device (Canfield Scientific, Fairfield, NJ). A narrowband reflectance spectrophotometer (Mexameter MX-16, Courage and Khazaka, Cologne, Germany) was used to measure degree of pigmentation of involved and adjacent uninvolved skin in all affected areas. The percent of involved skin on digital images was measured using computer imaging software (Scion Image for Windows, Scion Corp, Frederick, MD).
      Weighted kappa statistics were computed to assess the intrarater reliability for each rater by comparing MASI scores on all patients from day 1 to 2, so that ratings that were directly adjacent to one another received a higher weight than ratings spaced further apart. Strength of agreement of kappas was as follows: fair = 0.21 to 0.40, moderate = 0.41 to 0.60, and substantial = 0.61 to 0.80.
      • Landis J.R.
      • Koch G.G.
      The measurement of observer agreement for categorical data.
      MASI scores were compared by location (forehead, right malar region, left malar region, and chin) combined with area of involvement, darkness of melasma, and homogeneity of hyperpigmentation for intrarater reliability assessment. Interrater reliability was assessed using Spearman rank order correlations comparing MASI darkness ratings between raters separately on days 1 and 2, by location. Both intrarater and interrater reliabilities were also assessed using intraclass correlation coefficients, in which a reliability of 0.6 or greater is considered good.
      • Shrout P.E.
      • Fleiss J.L.
      Intraclass correlations: uses in assessing rater reliability.

      Hamer RM. INTRACC.SAS. 90. Cary, NC: SAS Institute, 2002. %INTRACC macro. Available from: URL:http://support.sas.com/kb/25/031.html. Accessed September 1, 2007.

      • Hartmann D.P.
      Considerations in the choice of interobserver reliability estimates.
      Validation of the MASI was performed comparing MSS (ratings for all subjects performed by all raters), mexameter (darkness of affected skin compared with normal-appearing adjacent skin), and size, based on computer imaging (area of skin affected through digital images). The MASI total score was compared with the MSS using analysis of variance (ANOVA) models combined across raters for each day with MASI total score as the continuous response variable and MSS score as the grouping variable. Validity was considered good if the ANOVA models were significant and the mean MASI score increased with increasing category scores for MSS using the linear contrast for trend; Tukey-Kramer adjustments for multiple pairwise mean comparisons were also conducted. Additional analyses (results not reported) compared MASI total score by levels of MSS for each rater separately by day using ANOVA. Pairwise differences between MASI and MSS scores by each rater were also examined using Spearman correlations. The darkness subcomponent of MASI was correlated with the difference in mexameter scores between normal-appearing and involved skin for each of the 4 facial locations. The area subcomponent was correlated with area calculation of involved skin on digital images using software (Scion Image for Windows, Scion Corp). Validity was considered good if the Spearman correlation between these measures was high (≥0.40) and significant.
      • Anastasi A.
      • Urbina S.
      Psychological testing.
      In all analyses, two-sided P values less than .05 were considered statistically significant. Data were analyzed using software (SAS, Version 9.1.3, SAS Institute Inc, Cary, NC; and SPSS 15.0, SPSS Inc, Chicago, IL).

      Results

      A total of 21 patients were recruited for this study. Demographic information is presented in Table I. The majority of the patients identified themselves as Hispanic, whereas two patients self-identified as Asian and one as African. An item analysis of the ratings was performed, which showed that all levels for all 3 components of the MASI in all 4 areas of the face were used, indicating a wide spectrum of melasma severity in the patient sample.
      Table IPatient demographics
      No. of patients (%) (N = 21)
      Age, y
       20-302 (9.5)
       31-4012 (57)
       41-506 (28.5)
       51-601 (5)
      Race/ethnicity
       Hispanic18 (86)
       African1 (5)
       Asian2 (9)
      Fitzpatrick skin type
       III5 (24)
       IV13 (62)
       V3 (14)
      Pattern of melasma
       Centrofacial16 (76)
       Malar5 (24)
       Mandibular0
      Duration of melasma, y
       0-1015 (71)
       11-206 (29)
      History of melasma in a first-degree relative12 (57)
      Aggravating factors
       Pregnancy5 (24)
       Hormonal therapy2 (9.5)
       Sun exposure17 (81)
       Cosmetic use0

       Reliability

      The average results across raters for intrarater reliability (weighted kappa) presented in Table II (available online at www.eblue.org) show moderate agreement for the area, darkness, and homogeneity values for the forehead and darkness values for the malar region. There was fair agreement for all chin values and area and homogeneity values for the malar region. Average interrater reliability (Spearman) was greater than 0.4 in all 4 locations of the face on both days (Table III, available online at www.eblue.org). Average correlations improved from day 1 to 2 and met the minimum criterion for good reliability except for chin. Overall, the average intrarater reliability comparing the MASI total score on days 1 and 2 (intraclass correlation coefficient = 0.75) (Table IV, available online at www.eblue.org) exceeded the minimum criterion of 0.6.

       Validity

      Validity analyses using ANOVA models showed that mean MASI total scores increased significantly with increasing categories of MSS levels across raters on both days (P < .0001 for linear trend on both days) (Fig 3). All pairwise comparisons for MASI total score between MSS categories were significant (maximum P < .0001) on both days, separately, except for mild versus moderate on day 2 (P = .061). These results were robust for each individual rater when examining linear trend test and pairwise comparisons among mild, moderate, and severe MSS categories and day. Validity using Spearman correlation for MASI total score versus MSS score revealed good agreement overall for all raters on both days (average rho = .73) (Table V, available online at www.eblue.org). A similar analysis of correlations between MASI darkness ratings and the mexameter difference between normal-appearing and uninvolved skin showed poor agreement (rho = 0.14 and 0.25) for the forehead but good agreement on the malar areas and the chin (rho = 0.45-0.61) (Table VI, available online at www.eblue.org). Validity analysis between the area of involvement calculated by computer imaging software (Scion Image for Windows, Scion Corp) and MASI area ratings by rater revealed good correlation for most raters and locations (Table VII, available online at www.eblue.org). The forehead and chin were low on day 1 and rater 3 (average day 1 rho = 0.2653), however, this improved on day 2 (average day 2 rho = 0.4184). The MASI total score was substantially higher when the MSS category was severe, and the severe melasma category could be distinguished clearly from the mild and moderate categories. When the MSS rating was changed from mild to moderate for these subjects, the corresponding change in the mean MASI total score was 0.7 (±2.5) points; therefore, the MASI scale being a continuous scale is able to provide more precise numeric estimates of the melasma severity than the categorical scale of MSS.
      Figure thumbnail gr3
      Fig 3Validity: means and 95% confidence intervals for Melasma Area and Severity Index (MASI) total score by melasma severity scale (MSS) levels across raters. P value < .0001 for test of linear trend for days 1 and 2.

      Discussion

      Until now, the MASI has never been tested for reliability or validity. Although a simple outcome, such as number of patients who cleared, is optimal, melasma is frequently a chronic, recalcitrant disorder; therefore, a measure of disease severity of all degrees is needed. Multiple instruments to measure hyperpigmentation have been developed; however, these instruments are expensive, and measurements can be time-consuming. We were able to use such instruments to validate clinical measures of melasma in the current study, indicating that the MASI measures what it is designed to measure.
      Outcome measures should have internal consistency, have construct validity, and demonstrate sensitivity to change.
      • Ashcroft D.M.
      • Wan Po A.L.
      • Williams H.C.
      • Griffiths C.E.M.
      Clinical measures of disease severity and outcome in psoriasis: a critical appraisal of their quality.
      • Cook D.A.
      • Beckman T.J.
      Current concepts in validity and reliability for psychometric instruments: theory and application.
      The MASI has face validity, as it attempts to measure the size and darkness of pigmentation associated with melasma, which are the most frequent symptoms of patients with this disorder. Experts in melasma have used the MASI score as the predominant outcome measure for almost 20 years, suggesting that it has content validity as well.
      • Balkrishnan R.
      • McMichael A.J.
      • Camacho F.T.
      • Saltzberg F.
      • Housman T.S.
      • Grummer S.
      • et al.
      Development and validation of a health-related quality of life instrument for women with melasma.
      • Dominguez A.R.
      • Balkrishnan R.
      • Ellzey A.R.
      • Pandya A.G.
      Melasma in Latina patients: cross-cultural adaptation and validation of a quality-of-life questionnaire in Spanish language.
      • Kimbrough-Green C.K.
      • Griffiths C.E.
      • Finkel L.J.
      • Hamilton T.A.
      • Bulengo-Ransby S.M.
      • Ellis C.N.
      • et al.
      Topical retinoic acid for melasma in black patients: a vehicle-controlled clinical trial.
      • Balkrishnan R.
      • Kelly A.P.
      • McMichael A.
      • Torok H.
      Improved quality of life with effective treatment of facial melasma: the pigment trial.
      This study demonstrated that the overall MASI is reliable in the measurement of melasma and that raters were consistent in their assessments between days 1 and 2, demonstrating temporal stability. Interrater reliability testing showed consistency in the use of the scale, however, individual components of the MASI were problematic, particularly assessment of homogeneity and the chin. Homogeneity was the most difficult component to reliably assess in our patients. Removal of homogeneity did not alter reliability or validity measures, therefore, we recommend removal of homogeneity from the MASI score. The chin may be more difficult to assess than other components of the face because of its small size and less involvement with melasma than other areas. Nevertheless, we recommend that the evaluation of the chin be retained, as it is an integral part of evaluation of the face, represents only 10% of the overall score, and variations in assessment cause little change to the MASI score. The modified MASI we propose is easy to learn and perform and is scored as follows:
      ModifiedMASItotalscore=0.3A(f)D(f)+0.3A(lm)D(lm)+0.3A(rm)D(rm)+0.1A(c)D(c)


      The range of the total score is 0 to 24. Area and darkness are scored as follows: area of involvement: 0 = absent, 1 = <10%, 2 = 10%-29%, 3 = 30%-49%, 4 = 50%-69%, 5 = 70%-89%, and 6 = 90%-100%; darkness: 0 = absent, 1 = slight, 2 = mild, 3 = marked, and 4 = severe.
      Validity testing showed that the MASI scale being a continuous scale is able to provide more precise numeric estimates of the melasma severity than the categorical scale of MSS. Importantly, the mean MASI scores presented in Fig 3 may be useful as threshold scores for entry of patients with moderate and severe melasma in future studies, as there is currently no consensus on threshold levels for entry into a melasma study. For the modified MASI proposed in the current study, the values should be reduced by half.
      In summary, the MASI score shows good reliability within and between raters and good validity when compared with the MSS, mexameter readings, and computer-assisted area measurements. Removing chin and homogeneity measurements does not improve intraclass correlation coefficient reliability but because kappa statistics were poor for homogeneity and because it is a difficult assessment to teach and perform, we recommend removal of homogeneity from the MASI score. We also recommend that chin assessments are preserved in the MASI as it is an important part of facial assessment and because it only represents 10% of the whole index. Practice of raters is important, as more agreement was noted on day 2 versus day 1. The validation and reliability testing of the modified MASI score required a training module, including practice images, before initiating patient assessments, therefore, this training should be given to raters to ensure reliability in future studies. If desired, readers may contact the corresponding author for access to this training module. Future studies assessing the modified MASI over time in individual patients should be performed to determine the sensitivity to change of this outcome measure.
      The authors would like to thank Christina Carrigan, RN, for assistance in conducting this study.

      Table II.

      Tabled 1Intrarater reliability: weighted kappa statistics comparing Melasma Area and Severity Index ratings across days
      Rater 1Rater 2Rater 3Rater 4Rater 5Rater 6Average
      LocationKappaSEKappaSEKappaSEKappaSEKappaSEKappaSEKappa
      Forehead (area)0.350.120.630.100.510.160.510.100.500.150.430.120.49
      Forehead (darkness)0.370.150.630.130.190.180.580.110.350.130.360.170.41
      Forehead (homogeneity)0.400.120.410.150.440.150.410.140.280.170.440.160.40
      Left malar (area)0.610.130.110.160.370.170.210.130.170.170.450.140.32
      Left malar (darkness)0.710.140.500.140.640.160.430.150.560.16−0.060.240.46
      Left malar (homogeneity)0.510.140.030.130.220.200.290.160.490.120.270.170.30
      Right malar (area)0.510.130.270.160.140.150.300.150.330.150.580.140.36
      Right malar (darkness)0.570.140.600.140.530.160.550.150.720.130.230.220.53
      Right malar (homogeneity)0.440.150.280.160.000.190.330.180.220.160.230.180.25
      Chin (area)0.140.170.590.110.210.110.480.140.420.120.350.150.37
      Chin (darkness)0.170.200.420.170.290.160.500.130.210.130.300.180.31
      Chin (homogeneity)0.250.17−0.070.130.210.150.480.130.200.130.270.150.22
      N = 21.
      Benchmarks for kappa
      • Landis J.R.
      • Koch G.G.
      The measurement of observer agreement for categorical data.
      : <0.00 = poor; 0.0-0.2 = slight; 0.2-0.4 = fair; 0.4-0.6 = moderate; 0.6-0.8 = substantial; 0.8-1.0 = almost perfect.

      Table III.

      Tabled 1Interrater reliability for Melasma Area and Severity Index darkness: average Spearman correlations across raters
      LocationDay 1Day 2
      Forehead0.550.61
      Right malar0.560.62
      Left malar0.440.62
      Chin0.430.50

      Table IV.

      Tabled 1Interrater and intrarater reliability for Melasma Area and Severity Index total score: intraclass correlations
      Reliabilities
      InterraterIntrarater
      RaterDay 1Day 2Day 1 vs 2
      10.720.670.79
      20.750.710.67
      30.660.740.59
      40.550.740.66
      50.740.690.79
      60.660.680.78
      Average0.680.700.75

      Table V.

      Tabled 1Validity: Spearman correlations for Melasma Area and Severity Index total score versus melasma severity scale by rater and day
      RaterDay 1Day 2
      10.820.71
      20.820.77
      30.770.80
      40.550.67
      50.700.63
      60.690.75
      Average0.730.73

      Table VI.

      Tabled 1Validity: Spearman correlations for average difference between normal-appearing and involved skin using mexameter versus Melasma Area and Severity Index darkness ratings by rater
      Day 1Day 2
      RaterForeheadLeft malarRight malarChinForeheadLeft malarRight malarChin
      1−0.070.370.470.610.110.670.480.36
      2−0.200.400.540.630.080.540.660.75
      30.510.490.840.23−0.100.680.590.46
      4−0.110.550.370.510.090.520.450.66
      5−0.090.590.570.600.200.550.580.24
      60.200.140.500.410.230.670.340.20
      Average0.250.450.570.520.140.610.530.49

      Table VII.

      Tabled 1Validity: Spearman correlations for area measurement using computer software
      Scion Image for Windows, Scion Corp, Frederick, MD.
      versus Melasma Area and Severity Index area of involvement ratings by rater
      Day 1Day 2
      RaterForeheadLeft malarRight malarChinForeheadLeft malarRight malarChin
      10.360.730.720.090.440.580.600.57
      20.400.540.570.460.490.670.320.47
      30.150.450.21−0.090.470.510.410.21
      40.190.380.420.410.420.370.480.10
      50.460.470.370.350.54−0.180.400.37
      60.530.560.630.310.720.180.400.72
      Average0.370.540.520.320.520.460.440.46
      Scion Image for Windows, Scion Corp, Frederick, MD.

      References

        • Grimes P.E.
        Melasma: etiologic and therapeutic considerations.
        Arch Dermatol. 1995; 131: 1453-1457
        • Werlinger K.D.
        • Guevara I.L.
        • Gonzalez C.M.
        • Rincon E.T.
        • Caetano R.
        • Haley R.W.
        • et al.
        Prevalence of self-diagnosed melasma among premenopausal Latino women in Dallas and Fort Worth, Tex.
        Arch Dermatol. 2007; 143: 424-425
        • Balkrishnan R.
        • McMichael A.J.
        • Camacho F.T.
        • Saltzberg F.
        • Housman T.S.
        • Grummer S.
        • et al.
        Development and validation of a health-related quality of life instrument for women with melasma.
        Br J Dermatol. 2003; 149: 572-577
        • Dominguez A.R.
        • Balkrishnan R.
        • Ellzey A.R.
        • Pandya A.G.
        Melasma in Latina patients: cross-cultural adaptation and validation of a quality-of-life questionnaire in Spanish language.
        J Am Acad Dermatol. 2006; 55: 59-66
        • Kimbrough-Green C.K.
        • Griffiths C.E.
        • Finkel L.J.
        • Hamilton T.A.
        • Bulengo-Ransby S.M.
        • Ellis C.N.
        • et al.
        Topical retinoic acid for melasma in black patients: a vehicle-controlled clinical trial.
        Arch Dermatol. 1994; 130: 727-733
        • Taylor S.C.
        • Torok H.
        • Jones T.
        • Lowe N.
        • Rich P.
        • Tschen E.
        • et al.
        Efficacy and safety of a new triple-combination agent for the treatment of facial melasma.
        Cutis. 2003; 72: 67-72
        • Landis J.R.
        • Koch G.G.
        The measurement of observer agreement for categorical data.
        Biometrics. 1977; 33: 159-174
        • Shrout P.E.
        • Fleiss J.L.
        Intraclass correlations: uses in assessing rater reliability.
        Psychol Bull. 1979; 86: 420-428
      1. Hamer RM. INTRACC.SAS. 90. Cary, NC: SAS Institute, 2002. %INTRACC macro. Available from: URL:http://support.sas.com/kb/25/031.html. Accessed September 1, 2007.

        • Hartmann D.P.
        Considerations in the choice of interobserver reliability estimates.
        J Appl Behav Anal. 1977; 10: 103-116
        • Anastasi A.
        • Urbina S.
        Psychological testing.
        Prentice Hall, Upper Saddle River (NJ)1997 (p. 144)
        • Ashcroft D.M.
        • Wan Po A.L.
        • Williams H.C.
        • Griffiths C.E.M.
        Clinical measures of disease severity and outcome in psoriasis: a critical appraisal of their quality.
        Br J Dermatol. 1999; 141: 185-191
        • Cook D.A.
        • Beckman T.J.
        Current concepts in validity and reliability for psychometric instruments: theory and application.
        Am J Med. 2006; 119 (e7-16): 166
        • Balkrishnan R.
        • Kelly A.P.
        • McMichael A.
        • Torok H.
        Improved quality of life with effective treatment of facial melasma: the pigment trial.
        J Drugs Dermatol. 2004; 3: 377-381