A Simulation Based Comparison of Correlation Coefficients with Regard to Type I Error Rate and Power
Affiliation(s)
1 Canakkale Onsekiz Mart University, Canakkale, Turkey. 2 University of Tennessee Health Science Center, Memphis, USA. 3 Yüzüncüyil University, Van, Turkey.
1 Canakkale Onsekiz Mart University, Canakkale, Turkey. 2 University of Tennessee Health Science Center, Memphis, USA. 3 Yüzüncüyil University, Van, Turkey.
ABSTRACT
In this simulation study, five correlation coefficients, namely, Pearson, Spearman, Kendal Tau, Permutation-based, and Winsorized were compared in terms of Type I error rate and power under different scenarios where the underlying distributions of the variables of interest, sample sizes and correlation patterns were varied. Simulation results showed that the Type I error rate and power of Pearson correlation coefficient were negatively affected by the distribution shapes especially for small sample sizes, which was much more pronounced for Spearman Rank and Kendal Tau correlation coefficients especially when sample sizes were small. In general, Permutation-based and Winsorized correlation coefficients are more robust to distribution shapes and correlation patterns, regardless of sample size. In conclusion, when assumptions of Pearson correlation coefficient are not satisfied, Permutation-based and Winsorized correlation coefficients seem to be better alternatives.
In this simulation study, five correlation coefficients, namely, Pearson, Spearman, Kendal Tau, Permutation-based, and Winsorized were compared in terms of Type I error rate and power under different scenarios where the underlying distributions of the variables of interest, sample sizes and correlation patterns were varied. Simulation results showed that the Type I error rate and power of Pearson correlation coefficient were negatively affected by the distribution shapes especially for small sample sizes, which was much more pronounced for Spearman Rank and Kendal Tau correlation coefficients especially when sample sizes were small. In general, Permutation-based and Winsorized correlation coefficients are more robust to distribution shapes and correlation patterns, regardless of sample size. In conclusion, when assumptions of Pearson correlation coefficient are not satisfied, Permutation-based and Winsorized correlation coefficients seem to be better alternatives.
KEYWORDS
Correlation Coefficient, Robustness, Type I Error Rate, Test Power, Non-Normality, Simulation
Correlation Coefficient, Robustness, Type I Error Rate, Test Power, Non-Normality, Simulation
Cite this paper
Tuğran, E. , Kocak, M. , Mirtagioğlu, H. , Yiğit, S. and Mendes, M. (2015) A Simulation Based Comparison of Correlation Coefficients with Regard to Type I Error Rate and Power. Journal of Data Analysis and Information Processing, 3, 87-101. doi: 10.4236/jdaip.2015.33010.
Tuğran, E. , Kocak, M. , Mirtagioğlu, H. , Yiğit, S. and Mendes, M. (2015) A Simulation Based Comparison of Correlation Coefficients with Regard to Type I Error Rate and Power. Journal of Data Analysis and Information Processing, 3, 87-101. doi: 10.4236/jdaip.2015.33010.
References
[1] Edgell, S. and Noon, S. (1984) Effect of Violation of Normality on the t Test of the Correlation Coefficient. Psychological Bulletin, 95, 576-583.
http://dx.doi.org/10.1037/0033-2909.95.3.576 [2] Hayes, A.F. (1996) PERMUSTAT: Randomization Tests for the MacIntosh. Behavior Research Methods, Instruments, & Computers, 28, 473-475.
http://dx.doi.org/10.3758/BF03200530 [3] Bonett, D.G. and Wright, T.A. (2000) Sample Size Requirements for Estimating Pearson, Kendall and Spearman Correlations. Psychometrics, 65, 23-28.
http://dx.doi.org/10.1007/BF02294183 [4] Zimmerman, D.W., Zumbo, B.D. and Williams, R.H. (2003) Bias in Estimation and Hypothesis Testing of Correlation. Psychologica, 24, 133-158. [5] Wilcox, R.R. (2001) Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy. Springer, New York.
http://dx.doi.org/10.1007/978-1-4757-3522-2 [6] Warner, R. (2008) Applied Statistics: From Bivariate through Multivariate Techniques. Sage Publications, Inc., Thousand Oaks. [7] Triola, M. (2010) Elementary Statistics. 11th Edition, Addison-Wesley/Pearson Education, Boston. [8] Witte, R. and Witte, J. (2010) Statistics. 9th Edition, Wiley, New York. [9] Blair, R. and Lawson, S. (1982) Another Look at the Robustness of the Product-Moment Correlation Coefficient to Population Non-Normality. Florida Journal of Educational Research, 24, 11-15. [10] Fowler, R. (1987) Power and Robustness in Product-Moment Correlation. Applied Psychological Measurement, 11, 419-428.
http://dx.doi.org/10.1177/014662168701100407 [11] Zimmerman, D. and Zumbo, B. (1993) Significance Testing of Correlation Using Scores, Ranks, and Modified Ranks. Educational and Psychological Measurement, 53, 897-904.
http://dx.doi.org/10.1177/0013164493053004003 [12] Bishara, A.J. and Hittner, J.B. (2012) Testing the Significance of a Correlation with Non-Normal Data: Comparison of Pearson, Spearman, Transformation, and Resampling Approaches. Psychological Methods, 17, 399-417. [13] Osborne, J. (2002) Notes on the Use of Data Transformations. Practical Assessment, Research & Evaluation, 8.
http://pareonline.net/getvn.asp?v=8&n=6 [14] Tabachnick, B. and Fidell, L. (2007) Using Multivariate Statistics. 5th Edition, Allyn & Bacon/Pearson Education, Boston. [15] Rasmussen, J. (1989) Data Transformation, Type I Error Rate and Power. British Journal of Mathematical and Statistical Psychology, 42, 203-213.
http://dx.doi.org/10.1111/j.2044-8317.1989.tb00910.x [16] Dunlap, W., Burke, M. and Greer, T. (1995) The Effect of Skew on the Magnitude of Product-Moment Correlations. Journal of General Psychology, 122, 365-377.
http://dx.doi.org/10.1080/00221309.1995.9921248 [17] Calkins, D.S. (1974) Some Effects of Non-Normal Distribution Shape on the Magnitude of the Pearson Product Moment Correlation Coefficient. Interamerican Journal of Psychology, 8, 261-288. [18] Wilcox, R.R. (1993) Some Results on a Winsorized Correlation Coefficient. British Journal of Mathematical and Statistical Psychology, 46, 339-349.
http://dx.doi.org/10.1111/j.2044-8317.1993.tb01020.x [19] Good, P. (2009) Robustness of Pearson Correlation. Interstat, 15, 1-6. [20] Kotz, S., Balakrishnan, N. and Johnson, N.L. (2000) Continuous Multivariate Distributions: Volume 1: Models and Applications. 2nd Edition, John Wiley & Sons, Inc., New York.
http://dx.doi.org/10.1002/0471722065 [21] Ruscio, J. and Kaczetow, W. (2008) Simulating Multivariate Non-Normal Data Using an Iterative Algorithm. Multivariate Behavioral Research, 43, 335-381.
http://dx.doi.org/10.1080/00273170802285693 [22] Balakrishnan, N. and Lai, C.D. (2009) Continuous Bivariate Distributions. 2nd Edition, Springer, New York. [23] Chok, N.S. (2010) Pearson’s versus Spearman’s and Kendall’s Correlation Coefficients for Continuous Data. Master’s Thesis, University of Pittsburgh, Pittsburgh. [24] Sheskin, D.J. (2007) Handbook of Parametric and Nonparametric Statistical Procedures. 4th Edition, Chapman & Hall/CRC, Boca Raton. [25] Tanizaki, H. (2004) On Small Sample Properties of Permutation Tests: Independence between Two Samples. International Journal of Pure and Applied Mathematics, 13, 235-243. [26] Ernst, M.D. (2004) Permutation Methods: A Basis for Exact Inference. Statistical Science, 19, 676-685.
http://dx.doi.org/10.1214/088342304000000396 [27] Knight, W. (1966) A Computer Method for Calculating Kendall’s Tau with Ungrouped Data. Journal of the American Statistical Association, 61, 436-439.
http://dx.doi.org/10.1080/01621459.1966.10480879 [28] Kowalski, C.J. and Tarter, M.E. (1969) Co-Ordinate Transformations to Normality and the Power of Normal Tests for Independence. Biometrika, 56, 139-148.
http://dx.doi.org/10.1093/biomet/56.1.139 [29] Rasmussen, J. and Dunlap, W. (1991) Dealing with Non-Normal Data: Parametric Analysis of Transformed Data vs. Nonparametric Analysis. Educational and Psychological Measurement, 51, 809-820.
http://dx.doi.org/10.1177/001316449105100402 [30] Siegel, S. (1957) Nonparametric Statistics. The American Statistician, 11, 13-19. [31] Lieberson, S. (1964) Limitations in the Application of Non-Parametric Coefficients of Correlation. American Socio-logical Review, 29, 744-746.
http://dx.doi.org/10.2307/2091428 [32] Khamis, H. (2008) Measures of Association: How to Choose? Journal of Diagnostic Medical Sonography, 24, 155-162.
http://dx.doi.org/10.1177/8756479308317006 [33] Good, P. (2005) Permutation, Parametric and Bootstrap Tests of Hypotheses. 3rd Edition, Springer-Verlag, New York. [34] Mielke, P.W. and Berry, K.J. (2007) Permutation Methods. A Distance Function Approach. 2nd Edition, Springer, New York. [35] Keller-McNulty, S. and Higgins, J.J. (1987) Effect of Tail Weight and Outliers on Power and Type-I Error of Robust Permutation Tests for Location. Communications in Statistics B-Simulation and Computation, 16, 17-35.
http://dx.doi.org/10.1080/03610918708812575 [36] Akkartal, E., Mendes, M. and Mendes, E. (2010) Determination of Suitable Permutation Numbers in Comparing Independent Group Means: A Monte Carlo Simulation Study. Journal of Scientific & Industrial Research, 69, 422-425. [37] Hayes, A.F. (1998) SPSS Procedures for Approximate Randomization Tests. Behavior Research Methods, Instruments, & Computers, 30, 536-543.
http://dx.doi.org/10.3758/BF03200687 [38] Hayes, A.F. (2000) Randomization Tests and the Equality of Variance Assumption When Comparing Group Means. Animal Behaviour, 59, 653-656.
http://dx.doi.org/10.1006/anbe.1999.1366
[1] Edgell, S. and Noon, S. (1984) Effect of Violation of Normality on the t Test of the Correlation Coefficient. Psychological Bulletin, 95, 576-583.
http://dx.doi.org/10.1037/0033-2909.95.3.576 [2] Hayes, A.F. (1996) PERMUSTAT: Randomization Tests for the MacIntosh. Behavior Research Methods, Instruments, & Computers, 28, 473-475.
http://dx.doi.org/10.3758/BF03200530 [3] Bonett, D.G. and Wright, T.A. (2000) Sample Size Requirements for Estimating Pearson, Kendall and Spearman Correlations. Psychometrics, 65, 23-28.
http://dx.doi.org/10.1007/BF02294183 [4] Zimmerman, D.W., Zumbo, B.D. and Williams, R.H. (2003) Bias in Estimation and Hypothesis Testing of Correlation. Psychologica, 24, 133-158. [5] Wilcox, R.R. (2001) Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy. Springer, New York.
http://dx.doi.org/10.1007/978-1-4757-3522-2 [6] Warner, R. (2008) Applied Statistics: From Bivariate through Multivariate Techniques. Sage Publications, Inc., Thousand Oaks. [7] Triola, M. (2010) Elementary Statistics. 11th Edition, Addison-Wesley/Pearson Education, Boston. [8] Witte, R. and Witte, J. (2010) Statistics. 9th Edition, Wiley, New York. [9] Blair, R. and Lawson, S. (1982) Another Look at the Robustness of the Product-Moment Correlation Coefficient to Population Non-Normality. Florida Journal of Educational Research, 24, 11-15. [10] Fowler, R. (1987) Power and Robustness in Product-Moment Correlation. Applied Psychological Measurement, 11, 419-428.
http://dx.doi.org/10.1177/014662168701100407 [11] Zimmerman, D. and Zumbo, B. (1993) Significance Testing of Correlation Using Scores, Ranks, and Modified Ranks. Educational and Psychological Measurement, 53, 897-904.
http://dx.doi.org/10.1177/0013164493053004003 [12] Bishara, A.J. and Hittner, J.B. (2012) Testing the Significance of a Correlation with Non-Normal Data: Comparison of Pearson, Spearman, Transformation, and Resampling Approaches. Psychological Methods, 17, 399-417. [13] Osborne, J. (2002) Notes on the Use of Data Transformations. Practical Assessment, Research & Evaluation, 8.
http://pareonline.net/getvn.asp?v=8&n=6 [14] Tabachnick, B. and Fidell, L. (2007) Using Multivariate Statistics. 5th Edition, Allyn & Bacon/Pearson Education, Boston. [15] Rasmussen, J. (1989) Data Transformation, Type I Error Rate and Power. British Journal of Mathematical and Statistical Psychology, 42, 203-213.
http://dx.doi.org/10.1111/j.2044-8317.1989.tb00910.x [16] Dunlap, W., Burke, M. and Greer, T. (1995) The Effect of Skew on the Magnitude of Product-Moment Correlations. Journal of General Psychology, 122, 365-377.
http://dx.doi.org/10.1080/00221309.1995.9921248 [17] Calkins, D.S. (1974) Some Effects of Non-Normal Distribution Shape on the Magnitude of the Pearson Product Moment Correlation Coefficient. Interamerican Journal of Psychology, 8, 261-288. [18] Wilcox, R.R. (1993) Some Results on a Winsorized Correlation Coefficient. British Journal of Mathematical and Statistical Psychology, 46, 339-349.
http://dx.doi.org/10.1111/j.2044-8317.1993.tb01020.x [19] Good, P. (2009) Robustness of Pearson Correlation. Interstat, 15, 1-6. [20] Kotz, S., Balakrishnan, N. and Johnson, N.L. (2000) Continuous Multivariate Distributions: Volume 1: Models and Applications. 2nd Edition, John Wiley & Sons, Inc., New York.
http://dx.doi.org/10.1002/0471722065 [21] Ruscio, J. and Kaczetow, W. (2008) Simulating Multivariate Non-Normal Data Using an Iterative Algorithm. Multivariate Behavioral Research, 43, 335-381.
http://dx.doi.org/10.1080/00273170802285693 [22] Balakrishnan, N. and Lai, C.D. (2009) Continuous Bivariate Distributions. 2nd Edition, Springer, New York. [23] Chok, N.S. (2010) Pearson’s versus Spearman’s and Kendall’s Correlation Coefficients for Continuous Data. Master’s Thesis, University of Pittsburgh, Pittsburgh. [24] Sheskin, D.J. (2007) Handbook of Parametric and Nonparametric Statistical Procedures. 4th Edition, Chapman & Hall/CRC, Boca Raton. [25] Tanizaki, H. (2004) On Small Sample Properties of Permutation Tests: Independence between Two Samples. International Journal of Pure and Applied Mathematics, 13, 235-243. [26] Ernst, M.D. (2004) Permutation Methods: A Basis for Exact Inference. Statistical Science, 19, 676-685.
http://dx.doi.org/10.1214/088342304000000396 [27] Knight, W. (1966) A Computer Method for Calculating Kendall’s Tau with Ungrouped Data. Journal of the American Statistical Association, 61, 436-439.
http://dx.doi.org/10.1080/01621459.1966.10480879 [28] Kowalski, C.J. and Tarter, M.E. (1969) Co-Ordinate Transformations to Normality and the Power of Normal Tests for Independence. Biometrika, 56, 139-148.
http://dx.doi.org/10.1093/biomet/56.1.139 [29] Rasmussen, J. and Dunlap, W. (1991) Dealing with Non-Normal Data: Parametric Analysis of Transformed Data vs. Nonparametric Analysis. Educational and Psychological Measurement, 51, 809-820.
http://dx.doi.org/10.1177/001316449105100402 [30] Siegel, S. (1957) Nonparametric Statistics. The American Statistician, 11, 13-19. [31] Lieberson, S. (1964) Limitations in the Application of Non-Parametric Coefficients of Correlation. American Socio-logical Review, 29, 744-746.
http://dx.doi.org/10.2307/2091428 [32] Khamis, H. (2008) Measures of Association: How to Choose? Journal of Diagnostic Medical Sonography, 24, 155-162.
http://dx.doi.org/10.1177/8756479308317006 [33] Good, P. (2005) Permutation, Parametric and Bootstrap Tests of Hypotheses. 3rd Edition, Springer-Verlag, New York. [34] Mielke, P.W. and Berry, K.J. (2007) Permutation Methods. A Distance Function Approach. 2nd Edition, Springer, New York. [35] Keller-McNulty, S. and Higgins, J.J. (1987) Effect of Tail Weight and Outliers on Power and Type-I Error of Robust Permutation Tests for Location. Communications in Statistics B-Simulation and Computation, 16, 17-35.
http://dx.doi.org/10.1080/03610918708812575 [36] Akkartal, E., Mendes, M. and Mendes, E. (2010) Determination of Suitable Permutation Numbers in Comparing Independent Group Means: A Monte Carlo Simulation Study. Journal of Scientific & Industrial Research, 69, 422-425. [37] Hayes, A.F. (1998) SPSS Procedures for Approximate Randomization Tests. Behavior Research Methods, Instruments, & Computers, 30, 536-543.
http://dx.doi.org/10.3758/BF03200687 [38] Hayes, A.F. (2000) Randomization Tests and the Equality of Variance Assumption When Comparing Group Means. Animal Behaviour, 59, 653-656.
http://dx.doi.org/10.1006/anbe.1999.1366