Observer Variability in BI-RADS Ultrasound Features and Its Influence on Computer-Aided Diagnosis of Breast Masses
Author(s)
Laith R. Sultan1*,
Ghizlane Bouzghar1,
Benjamin J. Levenback1,
Nauroze A. Faizi1,
Santosh S. Venkatesh2,
Emily F. Conant1,
Chandra M. Sehgal1
Affiliation(s)
1 Department of Radiology, University of Pennsylvania, Philadelphia, USA. 2 Department of Electrical Engineering, University of Pennsylvania, Philadelphia, USA.
1 Department of Radiology, University of Pennsylvania, Philadelphia, USA. 2 Department of Electrical Engineering, University of Pennsylvania, Philadelphia, USA.
ABSTRACT
Objective: Computer classification of sonographic BI-RADS features can aid differentiation of the malignant and benign masses. However, the variability in the diagnosis due to the differences in the observed features between the observations is not known. The goal of this study is to measure the variation in sonographic features between multiple observations and determine the effect of features variation on computer-aided diagnosis of the breast masses. Materials and Methods: Ultrasound images of biopsy proven solid breast masses were analyzed in three independent observations for BI-RADS sonographic features. The BI-RADS features from each observation were used with Bayes classifier to determine probability of malignancy. The observer agreement in the sonographic features was measured by kappa coefficient and the difference in the diagnostic performances between observations was determined by the area under the ROC curve, Az, and interclass correlation coefficient. Results: While some features were repeatedly observed, κ = 0.95, other showed a significant variation, κ = 0.16. For all features, combined intra-observer agreement was substantial, κ = 0.77. The agreement, however, decreased steadily to 0.66 and 0.56 as time between the observations increased from 1 to 2 and 3 months, respectively. Despite the variation in features between observations the probabilities of malignancy estimates from Bayes classifier were robust and consistently yielded same level of diagnostic performance, Az was 0.772-0.817 for sonographic features alone and 0.828-0.849 for sonographic features and age combined. The difference in the performance, ΔAz, between the observations for the two groups was small (0.003-0.044) and was not statistically significant (p < 0.05). Interclass correlation coefficient for the observations was 0.822 (CI: 0.787-0.853) for BI-RADS sonographic features alone and for those combined with age was 0.833 (CI: 0.800-0.862). Conclusion: Despite the differences in the BI-RADS sonographic features between different observations, the diagnostic performance of computer-aided analysis for differentiating breast masses did not change. Through continual retraining, the computer-aided analysis provides consistent diagnostic performance independent of the variations in the observed sonographic features.
Objective: Computer classification of sonographic BI-RADS features can aid differentiation of the malignant and benign masses. However, the variability in the diagnosis due to the differences in the observed features between the observations is not known. The goal of this study is to measure the variation in sonographic features between multiple observations and determine the effect of features variation on computer-aided diagnosis of the breast masses. Materials and Methods: Ultrasound images of biopsy proven solid breast masses were analyzed in three independent observations for BI-RADS sonographic features. The BI-RADS features from each observation were used with Bayes classifier to determine probability of malignancy. The observer agreement in the sonographic features was measured by kappa coefficient and the difference in the diagnostic performances between observations was determined by the area under the ROC curve, Az, and interclass correlation coefficient. Results: While some features were repeatedly observed, κ = 0.95, other showed a significant variation, κ = 0.16. For all features, combined intra-observer agreement was substantial, κ = 0.77. The agreement, however, decreased steadily to 0.66 and 0.56 as time between the observations increased from 1 to 2 and 3 months, respectively. Despite the variation in features between observations the probabilities of malignancy estimates from Bayes classifier were robust and consistently yielded same level of diagnostic performance, Az was 0.772-0.817 for sonographic features alone and 0.828-0.849 for sonographic features and age combined. The difference in the performance, ΔAz, between the observations for the two groups was small (0.003-0.044) and was not statistically significant (p < 0.05). Interclass correlation coefficient for the observations was 0.822 (CI: 0.787-0.853) for BI-RADS sonographic features alone and for those combined with age was 0.833 (CI: 0.800-0.862). Conclusion: Despite the differences in the BI-RADS sonographic features between different observations, the diagnostic performance of computer-aided analysis for differentiating breast masses did not change. Through continual retraining, the computer-aided analysis provides consistent diagnostic performance independent of the variations in the observed sonographic features.
Cite this paper
Sultan, L. , Bouzghar, G. , Levenback, B. , Faizi, N. , Venkatesh, S. , Conant, E. and Sehgal, C. (2015) Observer Variability in BI-RADS Ultrasound Features and Its Influence on Computer-Aided Diagnosis of Breast Masses. Advances in Breast Cancer Research, 4, 1-8. doi: 10.4236/abcr.2015.41001.
Sultan, L. , Bouzghar, G. , Levenback, B. , Faizi, N. , Venkatesh, S. , Conant, E. and Sehgal, C. (2015) Observer Variability in BI-RADS Ultrasound Features and Its Influence on Computer-Aided Diagnosis of Breast Masses. Advances in Breast Cancer Research, 4, 1-8. doi: 10.4236/abcr.2015.41001.
References
[1] Kopans, D.B. (1992) The Positive Predictive Value of Mammography. American Journal of Roentgenology, 158, 521-526. http://dx.doi.org/10.2214/ajr.158.3.1310825 [2] Jiang, Y.L., Nishikawa, R.M., Schmidt, R.A., Metz, C.E., Giger, M.L. and Doi, K. (1999) Improving Breast Cancer Diagnosis with Computer-Aided Diagnosis. Academic Radiology, 6, 22-33.
http://dx.doi.org/10.1016/S1076-6332(99)80058-0 [3] Shen, W.C., Chang, R.F., Moon, W.K., Chou, Y.H. and Huang, C.S. (2007) Breast Ultrasound Computer-Aided Diagnosis Using BI-RADS Features. Academic Radiology, 14, 928-939.
http://dx.doi.org/10.1016/j.acra.2007.04.016 [4] Shen, W.C., Chang, R.F. and Moon, W.K. (2007) Computer Aided Classification System for Breast Ultrasound Based on Breast Imaging Reporting and Data System (BI-RADS). Ultrasound in Medicine & Biology, 33, 1688-1698. http://dx.doi.org/10.1016/j.ultrasmedbio.2007.05.016 [5] Moon, W.K., Lo, C.M., Chang, J.M., Huang, C.S., Chen, J.H. and Chang, R.F. (2012) Computer-Aided Classification of Breast Masses Using Speckle Features of Automated Breast Ultrasound Images. Medical Physics, 39, 6465-6473. http://dx.doi.org/10.1118/1.4754801 [6] Moon, W.K., Lo, C.-M., Chang, J.M., Huang, C.-S., Chen, J.-H. and Chang, R.-F. (2013) Quantitative Ultrasound Analysis for Classification of BI-RADS Category 3 Breast Masses. Journal of Digital Imaging, 26, 1091-1098. http://dx.doi.org/10.1007/s10278-013-9593-8 [7] Bouzghar, G., Levenback, B.J., Sultan, L.R., Venkatesh, S.S., Cwanger, A., Conant, E.F. and Sehgal, C.M. (2014) Bayesian Probability of Malignancy with Breast Ultrasound BI-RADS Features. Journal of Ultrasound in Medicine, 33, 641-648. http://dx.doi.org/10.7863/ultra.33.4.641 [8] American College of Radiology (2013) Breast Imaging Reporting and Data System: BI-RADS Atlas. 5th Edition, American College of Radiology, Reston. [9] Stavros, A.T., Thickman, D., Rapp, C.L., Dennis, M.A., Parker, S.H. and Sisney, G.A. (1995) Solid Breast Nodules: Use of Sonography to Distinguish between Benign and Malignant Lesions. Radiology, 196, 123-134. http://dx.doi.org/10.1148/radiology.196.1.7784555 [10] Cohen, J. (1960) A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20, 37-46. http://dx.doi.org/10.1177/001316446002000104 [11] Landis, J.R. and Koch, G.G. (1977) The Measurement of Observer Agreement for Categorical Data. Biometrics, 33, 159-174. http://dx.doi.org/10.2307/2529310 [12] Cary, T.W., Cwanger, A., Venkatesh, S.S., Conant, E.F. and Sehgal, C.M. (2012) Comparison of Naive Bayes and Logistic Regression for Computer-Aided Diagnosis of Breast Masses Using Ultrasound Imaging. In: Bosch, J.G. and Doyley, M.M., Eds., Medical Imaging: Ultrasonic Imaging, Tomography, and Therapy, SPIE, Bellingham. [13] DeLong, E.R., DeLong, D.M. and Clarke-Pearson, D.L. (1988) Comparing the Areas under Two or More Correlated ROC Curves: A Nonparametric Approach. Biometrics, 44, 837-845.
http://dx.doi.org/10.2307/2531595 [14] Abdullah, N., Mesurolle, B., El-Khoury, M. and Kao, E. (2009) Breast Imaging Reporting and Data System Lexicon for US: Interobserver Agreement for Assessment of Breast Masses. Radiology, 252, 665-672. [15] Calas, M.J., Almeida, R.M., Gutfilen, B. and Pereira, W.C. (2009) Intra-Observer Interpretation of Breast Ultrasonography Following the BI-RADS Classification. European Journal of Radiology, 74, 525-528. http://dx.doi.org/10.1016/j.ejrad.2009.04.015 [16] Park, C.S., Lee, J.H., Yim, H.W., Kang, B.J., Kim, H.S., Jung, J.I., Jung, N.Y. and Kim, S.H. (2007) Observer Agreement Using the ACR Breast Imaging Reporting and Data System (BI-RADS)-Ultrasound. Korean Journal of Radiology, 8, 397-402. [17] Lee, H.J., Kim, E.K., Kim, M.J., Youk, J.H., Lee, J.Y., Kang, D.R. and Oh, K.K. (2008) Observer Variability of Breast Imaging Reporting and Data System (BI-RADS) for Breast Ultrasound. European Journal of Radiology, 65, 293-298. http://dx.doi.org/10.1016/j.ejrad.2007.04.008 [18] Ryan, J.T., Haygood, T.M., Yamal, J.M., Evanoff, M., O’Sullivan, P., McEntee, M. and Brennan, P.C. (2011) The “Memory Effect” for Repeated Radiologic Observations. American Journal of Roent- genology, 197, W985-W991. http://dx.doi.org/10.2214/AJR.10.5859
[1] Kopans, D.B. (1992) The Positive Predictive Value of Mammography. American Journal of Roentgenology, 158, 521-526. http://dx.doi.org/10.2214/ajr.158.3.1310825 [2] Jiang, Y.L., Nishikawa, R.M., Schmidt, R.A., Metz, C.E., Giger, M.L. and Doi, K. (1999) Improving Breast Cancer Diagnosis with Computer-Aided Diagnosis. Academic Radiology, 6, 22-33.
http://dx.doi.org/10.1016/S1076-6332(99)80058-0 [3] Shen, W.C., Chang, R.F., Moon, W.K., Chou, Y.H. and Huang, C.S. (2007) Breast Ultrasound Computer-Aided Diagnosis Using BI-RADS Features. Academic Radiology, 14, 928-939.
http://dx.doi.org/10.1016/j.acra.2007.04.016 [4] Shen, W.C., Chang, R.F. and Moon, W.K. (2007) Computer Aided Classification System for Breast Ultrasound Based on Breast Imaging Reporting and Data System (BI-RADS). Ultrasound in Medicine & Biology, 33, 1688-1698. http://dx.doi.org/10.1016/j.ultrasmedbio.2007.05.016 [5] Moon, W.K., Lo, C.M., Chang, J.M., Huang, C.S., Chen, J.H. and Chang, R.F. (2012) Computer-Aided Classification of Breast Masses Using Speckle Features of Automated Breast Ultrasound Images. Medical Physics, 39, 6465-6473. http://dx.doi.org/10.1118/1.4754801 [6] Moon, W.K., Lo, C.-M., Chang, J.M., Huang, C.-S., Chen, J.-H. and Chang, R.-F. (2013) Quantitative Ultrasound Analysis for Classification of BI-RADS Category 3 Breast Masses. Journal of Digital Imaging, 26, 1091-1098. http://dx.doi.org/10.1007/s10278-013-9593-8 [7] Bouzghar, G., Levenback, B.J., Sultan, L.R., Venkatesh, S.S., Cwanger, A., Conant, E.F. and Sehgal, C.M. (2014) Bayesian Probability of Malignancy with Breast Ultrasound BI-RADS Features. Journal of Ultrasound in Medicine, 33, 641-648. http://dx.doi.org/10.7863/ultra.33.4.641 [8] American College of Radiology (2013) Breast Imaging Reporting and Data System: BI-RADS Atlas. 5th Edition, American College of Radiology, Reston. [9] Stavros, A.T., Thickman, D., Rapp, C.L., Dennis, M.A., Parker, S.H. and Sisney, G.A. (1995) Solid Breast Nodules: Use of Sonography to Distinguish between Benign and Malignant Lesions. Radiology, 196, 123-134. http://dx.doi.org/10.1148/radiology.196.1.7784555 [10] Cohen, J. (1960) A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20, 37-46. http://dx.doi.org/10.1177/001316446002000104 [11] Landis, J.R. and Koch, G.G. (1977) The Measurement of Observer Agreement for Categorical Data. Biometrics, 33, 159-174. http://dx.doi.org/10.2307/2529310 [12] Cary, T.W., Cwanger, A., Venkatesh, S.S., Conant, E.F. and Sehgal, C.M. (2012) Comparison of Naive Bayes and Logistic Regression for Computer-Aided Diagnosis of Breast Masses Using Ultrasound Imaging. In: Bosch, J.G. and Doyley, M.M., Eds., Medical Imaging: Ultrasonic Imaging, Tomography, and Therapy, SPIE, Bellingham. [13] DeLong, E.R., DeLong, D.M. and Clarke-Pearson, D.L. (1988) Comparing the Areas under Two or More Correlated ROC Curves: A Nonparametric Approach. Biometrics, 44, 837-845.
http://dx.doi.org/10.2307/2531595 [14] Abdullah, N., Mesurolle, B., El-Khoury, M. and Kao, E. (2009) Breast Imaging Reporting and Data System Lexicon for US: Interobserver Agreement for Assessment of Breast Masses. Radiology, 252, 665-672. [15] Calas, M.J., Almeida, R.M., Gutfilen, B. and Pereira, W.C. (2009) Intra-Observer Interpretation of Breast Ultrasonography Following the BI-RADS Classification. European Journal of Radiology, 74, 525-528. http://dx.doi.org/10.1016/j.ejrad.2009.04.015 [16] Park, C.S., Lee, J.H., Yim, H.W., Kang, B.J., Kim, H.S., Jung, J.I., Jung, N.Y. and Kim, S.H. (2007) Observer Agreement Using the ACR Breast Imaging Reporting and Data System (BI-RADS)-Ultrasound. Korean Journal of Radiology, 8, 397-402. [17] Lee, H.J., Kim, E.K., Kim, M.J., Youk, J.H., Lee, J.Y., Kang, D.R. and Oh, K.K. (2008) Observer Variability of Breast Imaging Reporting and Data System (BI-RADS) for Breast Ultrasound. European Journal of Radiology, 65, 293-298. http://dx.doi.org/10.1016/j.ejrad.2007.04.008 [18] Ryan, J.T., Haygood, T.M., Yamal, J.M., Evanoff, M., O’Sullivan, P., McEntee, M. and Brennan, P.C. (2011) The “Memory Effect” for Repeated Radiologic Observations. American Journal of Roent- genology, 197, W985-W991. http://dx.doi.org/10.2214/AJR.10.5859