Skip to main content
SpringerLink
Log in

Multivariate Response and Parsimony for Gaussian Cluster-Weighted Models

  • Published:
Journal of Classification Aims and scope Submit manuscript

Abstract

A family of parsimonious Gaussian cluster-weighted models is presented. This family concerns a multivariate extension to cluster-weighted modelling that can account for correlations between multivariate responses. Parsimony is attained by constraining parts of an eigen-decomposition imposed on the component covariance matrices. A sufficient condition for identifiability is provided and an expectation-maximization algorithm is presented for parameter estimation. Model performance is investigated on both synthetic and classical real data sets and compared with some popular approaches. Finally, accounting for linear dependencies in the presence of a linear regression structure is shown to offer better performance, vis-à-vis clustering, over existing methodologies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • AITKEN, A.C. (1926), “A Series Formula for the Roots of Algebraic and Transcendental Equations, in Proceedings of the Royal Society of Edinburgh, 45, 14–22.

  • ANDERLUCCI, L., and VIROLI, C. (2015), “Covariance Pattern Mixture Models for Multivariate Longitudinal Data”, The Annals of Applied Statistics, 9(2), 777–800.

    Article  MathSciNet  MATH  Google Scholar 

  • ANDERSON, E. (1935), “The Irises of the Gaspé Peninsula”, Bulletin of the American Iris Society, 59, 2–5.

    Google Scholar 

  • ANDERSON, J.A. (1972), “Separate Sample Logistic Discrimination”, Biometrikam, 59(1), 19–35.

    Article  MathSciNet  MATH  Google Scholar 

  • ANDREWS, J.L., and MCNICHOLAS, P.D. (2012), “Model-Based Clustering, Classification, and Discriminant Analysis via Mixtures of Multivariate t-Distributions”, Statistics and Computing, 22(5), 1021–1029.

    Article  MathSciNet  MATH  Google Scholar 

  • ANDREWS, J.L., and MCNICHOLAS, P.D. (2014), “Variable Selection for Clustering and Classification”, Journal of Classification, 31(2), 136–153.

    Article  MathSciNet  MATH  Google Scholar 

  • ANDREWS, J.L., MCNICHOLAS, P.D., and SUBEDI, S. (2011), “Model-Based Classification via Mixtures of Multivariate t-Distributions”, Computational Statistics and Data Analysis, 55(1), 520–529.

    Article  MathSciNet  MATH  Google Scholar 

  • AZZALINI, A. (2013), sn: The Skew-Normal and Skew-t Distributions, R Package Version 0.4-18.

  • BAUDRY, J.-P., and CELEUX, G. (2015), “EM for Mixtures”, Statistics and Computing, 25(4), 713–726.

    Article  MathSciNet  MATH  Google Scholar 

  • BIERNACKI, C., and LOURME, A. (2014), “Stable and Visualizable Gaussian Parsimonious Clustering Models”, Statistics and Computing, 24(6), 953–969.

    Article  MathSciNet  MATH  Google Scholar 

  • BIERNACKI, C., CELEUX, G., and GOVAERT, G. (2000), “Assessing a Mixture Model for Clustering with the Integrated Completed Likelihood”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7), 719–725.

    Article  Google Scholar 

  • BÖHNING, D. (1995), “A Review of Reliable Algorithms for the Semi-Parametric Maximum Likelihood Estimator of a Mixture Distribution”, Journal of Statistical Planning and Inference, 47, 5–28.

    Article  MathSciNet  MATH  Google Scholar 

  • BÖHNING, D. (2000), Computer-Assisted Analysis of Mixtures and Applications: Metaanalysis, Disease Mapping and Others, London: Chapman & Hall/CRC.

  • BÖHNING, D. (2003), “The EM Algorithm with Gradient Function Update for Discrete Mixtures with Known (Fixed) Number of Components”, Statistics and Computing, 13(3), 257–265.

    Article  MathSciNet  Google Scholar 

  • BÖHNING, D., DIETZ, E., SCHAUB, R., SCHLATTMANN, P., and LINDSAY, B.G. (1994), “The Distribution of the Likelihood Ratio for Mixtures of Densities from the One-Parameter Exponential Family”, Annals of the Institute of Statistical Mathematics, 46(2), 373–388.

    Article  MATH  Google Scholar 

  • BOUVEYRON, C., and BRUNET-SAUMARD, C. (2014), “Model-Based Clustering of High-Dimensional Data: A Review”, Computational Statistics and Data Analysis, 71, 52–78.

    Article  MathSciNet  MATH  Google Scholar 

  • BREIMAN, L., and FRIEDMAN, J.H. (1997), “Predicting Multivariate Responses in Multiple Linear Regression”, Journal of the Royal Statistical Society: Series B, 59(1), 3–54.

    Article  MathSciNet  MATH  Google Scholar 

  • BROWNE, R.P., and MCNICHOLAS, P.D. (2014a), “Estimating Common Principal Components in High Dimensions”, Advances in Data Analysis and Classification, 8(2), 217–226.

  • BROWNE, R.P., and MCNICHOLAS, P.D. (2014b), “Orthogonal Stiefel Manifold Optimization for Eigen-Decomposed Covariance Parameter Estimation in Mixture Models”, Statistics and Computing, 24(2), 203–210.

    Article  MathSciNet  MATH  Google Scholar 

  • BROWNE, R.P., and MCNICHOLAS, P.D. (2015), mixture: Mixture Models for Clustering and Classification, R Package Version 1.1.

  • BROWNE, R.P., SUBEDI, S., and MCNICHOLAS, P.D. (2013), “Constrained Optimization for a Subset of the Gaussian Parsimonious Clustering Models”, arXiv preprint arXiv:1306.5824.

  • CAMPBELL, N.A., and MAHON, R.J. (1974), “A Multivariate Study of Variation in Two Species of Rock Crab of the Genus Leptograpsus”, Australian Journal of Zoology, 22(3), 417–425.

    Article  Google Scholar 

  • CELEUX, G., and DIEBOLT, J. (1988), “A Random Imputation Principle: The Stochastic EM Algorithm”, in Rapports de Recherche 901, INRIA.

  • CELEUX, G., and GOVAERT, G. (1995), “Gaussian Parsimonious Clustering Models”, Pattern Recognition, 28(5), 781–793.

    Article  Google Scholar 

  • COOK, R.D., and WEISBERG, S. (1994), An Introduction to Regression Graphics, John Wiley & Sons.

  • DANG, U.J., and MCNICHOLAS, P.D. (2015), “Families of Parsimonious Finite Mixtures of Regression Models”, in Advances in Statistical Models for Data Analysis, Studies in Classification, Data Analysis and Knowledge Organization, eds. I. Morlini, T. Minerva, and M. Vichi, Switzerland: Springer International Publishing, pp. 73–84.

  • DANG, U.J., BROWNE, R.P., and MCNICHOLAS, P.D. (2015), “Mixtures of Multivariate Power Exponential Distributions”, Biometrics, 71(4), 1081–1089.

    Article  MathSciNet  MATH  Google Scholar 

  • DASGUPTA, A., and RAFTERY, A.E. (1998), “Detecting Features in Spatial Point Processes with Clutter via Model-Based Clustering”, Journal of the American Statistical Association, 93(441), 294–302.

    Article  MATH  Google Scholar 

  • DEMPSTER, A., LAIRD, N., and RUBIN, D. (1977), “Maximum Likelihood from Incomplete Data via the EM Algorithm”, Journal of the Royal Statistical Society: Series B, 39(1), 1–38.

    MathSciNet  MATH  Google Scholar 

  • DESARBO, W.S., and CRON, W.L. (1988), “A Maximum Likelihood Methodology for Clusterwise Linear Regression”, Journal of Classification, 5(2), 249–282.

    Article  MathSciNet  MATH  Google Scholar 

  • FISHER, R.A. (1936), “The Use of Multiple Measurements in Taxonomic Problems”, Annals of Eugenics, 7(2), 179–188.

    Article  Google Scholar 

  • FLURY, B.N., and GAUTSCHI, W. (1986), “An Algorithm for Simultaneous Orthogonal Transformation of Several Positive Definite Matrices to Nearly Diagonal Form”, SIAM Journal on Scientific and Statistical Computing, 7(1), 169–184.

    Article  MathSciNet  MATH  Google Scholar 

  • FRALEY, C., and RAFTERY, A.E. (2002), “Model-Based Clustering, Discriminant Analysis, and Density Estimation”, Journal of the American Statistical Association, 97(458), 611–631.

    Article  MathSciNet  MATH  Google Scholar 

  • FRALEY, C., RAFTERY, A.E., MURPHY, T.B., and SCRUCCA, L. (2012), “Mclust Version 4 for R: Normal Mixture Modeling for Model-Based Clustering, Classification, and Density Estimation”, Technical Report 597, Department of Statistics, University of Washington, Seattle, WA.

  • FRANCZAK, B.C., TORTORA, C., BROWNE, R.P., and MCNICHOLAS, P.D. (2015), “Unsupervised Learning viaMixtures of Skewed Distributions with Hypercube Contours” Pattern Recognition Letters, 58(1), 69–76.

    Article  Google Scholar 

  • GALIMBERTI, G., and SOFFRITTI,G. (2013), “A Multivariate Linear Regression Analysis Using Finite Mixtures of t Distributions”, Computational Statistics and Data Analysis, 71, 138–150.

    Article  MathSciNet  MATH  Google Scholar 

  • GALIMBERTI, G., SCARDOVI, E., and SOFFRITTI, G. (2015), “Using Mixtures in Seemingly Unrelated Linear Regression Models with Non-Normal Errors”, Statistics and Computing, 1–14.

  • GERSHENFELD,N. (1997), “Nonlinear Inference and Cluster-Weighted Modeling”, Annals of the New York Academy of Sciences, 808(1), 18–24.

    Article  Google Scholar 

  • GRÜN, B., and LEISCH, F. (2008), “Flexmix Version 2: Finite Mixtures with Concomitant Variables and Varying and Constant Parameters”, R package version 2.3.13, Journal of Statistical Software, 28(4), 1–35.

    Article  Google Scholar 

  • HARTIGAN, J.A., and WONG, M.A. (1979), “A k-Means Clustering Algorithm”, Journal of the Royal Statistical Society: Series C, 28(1), 100–108.

    MATH  Google Scholar 

  • HASTIE, T., and TIBSHIRANI, R. (1996), “Discriminant Analysis by Gaussian Mixtures”, Journal of the Royal Statistical Society: Series B, 58(1), 155–176.

    MathSciNet  MATH  Google Scholar 

  • HENNIG, C. (2000), “Identifiablity of Models for Clusterwise Linear Regression”, Journal of Classification, 17(2), 273–296.

    Article  MathSciNet  MATH  Google Scholar 

  • HENNIG, C., and LIAO, T.F. (2013), “How to Find an Appropriate Clustering for Mixed-Type Variables with Application to Socio-Economic Stratification (With Discussion)”, Journal of the Royal Statistical Society: Series C, 62(3), 309–369.

    Article  MathSciNet  Google Scholar 

  • HUBERT, L., and ARABIE, P. (1985), “Comparing Partitions”, Journal of Classification, 2(1), 193–218.

    Article  MATH  Google Scholar 

  • INGRASSIA, S. (2004), “A Likelihood-Based Constrained Algorithm for Multivariate Normal Mixture Models”, Statistical Methods and Applications, 13(2), 151–166.

    Article  MathSciNet  Google Scholar 

  • INGRASSIA, S., and PUNZO, A. (2016), “Decision Boundaries for Mixtures of Regressions”, Journal of the Korean Statistical Society, DOI: 10.1016/j.jkss.2015.11.005.

  • INGRASSIA, S., and ROCCI, R. (2007), “Constrained Monotone EM Algorithms for Finite Mixture of Multivariate Gaussians”, Computational Statistics and Data Analysis, 51(11), 5339–5351.

    Article  MathSciNet  MATH  Google Scholar 

  • INGRASSIA, S., MINOTTI, S.C., and VITTADINI, G. (2012), “Local Statistical Modeling via a Cluster-Weighted Approach with Elliptical Distributions”, Journal of Classification, 29(3), 363–401.

    Article  MathSciNet  MATH  Google Scholar 

  • INGRASSIA, S., MINOTTI, S.C., and PUNZO, A. (2014), “Model-Based Clustering via Linear Cluster-Weighted Models”, Computational Statistics and Data Analysis, 71, 159–182.

    Article  MathSciNet  Google Scholar 

  • INGRASSIA, S., PUNZO, A., VITTADINI, G., and MINOTTI, S.C. (2015), “The Generalized Linear Mixed Cluster-Weighted Model”, Journal of Classification, 32(1), 85–113.

    Article  MathSciNet  MATH  Google Scholar 

  • KERIBIN, C. (1998), “Estimation Consistante de l’Ordre de Modèles deMélange”, Comptes Rendus de l’Académie des Sciences-Series I-Mathematics, 326(2), 243–248.

  • KERIBIN, C. (2000), “Consistent Estimation of the Order of Mixture Models”, Sankhyā: The Indian Journal of Statistics, Series A, 62(1), 49–66.

  • LEBRET, R., IOVLEFF, S., LANGROGNET, F., BIERNACKI, C., CELEUX, G., and GOVAERT, G. (2015), “Rmixmod: The R Package of the Model-Based Unsupervised, Supervised, and Semi-Supervised Classification Mixmod Library”, Journal of Statistical Software, 67(6), 1–29.

    Article  Google Scholar 

  • LEE, S.X., and MCLACHLAN, G.J. (2013), “On Mixtures of Skew Normal and Skew t-Distributions”, Advances in Data Analysis and Classification, 7(3), 241–266.

    Article  MathSciNet  MATH  Google Scholar 

  • LEISCH, F. (2004), “FlexMix: A General Framework for Finite Mixture Models and Latent Class Regression in R”, Journal of Statistical Software, 11(8), 1–18.

    Article  Google Scholar 

  • LIN, T.-I., MCNICHOLAS, P.D., and HSIU, J.H. (2014), “Capturing Patterns via Parsimonious t Mixture Models”, Statistics and Probability Letters, 88, 80–87.

    Article  MathSciNet  MATH  Google Scholar 

  • LINDSAY, B.G. (1995), “Mixture Models: Theory, Geometry and Applications”, in NSFCBMS Regional Conference Series in Probability and Statistics, pp. 1–163.

  • MACQUEEN, J. (1967), “Some Methods for Classification and Analysis of Multivariate Observations”, in Proceedings of the Fifth Berkeley Symposium onMathematical Statistics and Probability, Vol. 1, Oakland CA: University of California Press, pp. 281–297.

  • MCLACHLAN, G.J., and BASFORD, K.E. (1988), Mixture Models: Inference and Applications to Clustering, New York: Marcel Dekker.

  • MCLACHLAN, G.J., and PEEL, D. (2000), Finite Mixture Models, New York: John Wiley & Sons Inc.

    Book  MATH  Google Scholar 

  • MCNICHOLAS, P.D. (2010), “Model-Based Classification Using Latent Gaussian Mixture Models”, Journal of Statistical Planning and Inference, 140(5), 1175–1181.

    Article  MathSciNet  MATH  Google Scholar 

  • MCNICHOLAS, P.D. (2016a), Mixture Model-Based Classification, Boca Raton FL: Chapman & Hall/CRC Press.

  • MCNICHOLAS, P.D. (2016b),“Model-Based Clustering”, Journal of Classification 33, in press.

  • MCNICHOLAS, P.D., MURPHY, T.B., MCDAID, A.F., and FROST, D. (2010), “Serial and Parallel Implementations of Model-Based Clustering via Parsimonious Gaussian Mixture Models”, Computational Statistics and Data Analysis, 54(3), 711–723.

    Article  MathSciNet  MATH  Google Scholar 

  • MURRAY, P.M., BROWNE, R.P., and MCNICHOLAS, P.D. (2014), “ Mixtures of Skew-t Factor Analyzers”, Computational Statistics and Data Analysis, 77, 326–335.

  • O’HAGAN, A., MURPHY, T.B., GORMLEY, I.C., MCNICHOLAS, P.D., and KARLIS, D. (2016), “Clustering with the Multivariate Normal Inverse Gaussian Distribution”, Computational Statistics and Data Analysis, 93, 18–30.

    Article  MathSciNet  Google Scholar 

  • PUNZO, A. (2014), “Flexible Mixture Modeling with the Polynomial Gaussian Cluster-Weighted Model”, Statistical Modelling, 14(3), 257–291.

    Article  MathSciNet  Google Scholar 

  • PUNZO, A., and INGRASSIA, S. (2013), “On the Use of the Generalized Linear Exponential Cluster-Weighted Model to Assess Local Linear Independence in Bivariate Data”, QdS -Journal of Methodological and Applied Statistics, 15, 131–144.

    Google Scholar 

  • PUNZO, A., and INGRASSIA, S. (2015), “Clustering Bivariate Mixed-Type Data via the Cluster-Weighted Model”, Computational Statistics, 31(3), 989–1013.

    Article  MathSciNet  MATH  Google Scholar 

  • PUNZO, A., and MCNICHOLAS, P.D. (2016), “Parsimonious Mixtures of Multivariate Contaminated Normal Distributions”, Biometrical Journal, in press.

  • PUNZO, A., BROWNE, R.P., and MCNICHOLAS, P.D. (2016), “Hypothesis Testing for MixtureModelling Selection”, Journal of Statistical Computation and Simulation, 86(14), 2797–2818.

    Article  MathSciNet  Google Scholar 

  • R CORE TEAM(2015), R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing.

  • RAND,W.M. (1971). “Objective Criteria for the Evaluation of ClusteringMethods”, Journal of the American Statistical Association, 66(336), 846–850.

    Article  Google Scholar 

  • SCHWARZ, G. (1978), “Estimating the Dimension of a Model”, Annals of Statistics, 6(2), 461–464.

    Article  MathSciNet  MATH  Google Scholar 

  • SOFFRITTI, G., and GALIMBERTI, G. (2011), “Multivariate Linear Regression with Non-Normal Errors: A Solution Based on Mixture Models”, Statistics and Computing, 21(4), 523–536.

    Article  MathSciNet  MATH  Google Scholar 

  • STEINLEY, D. (2004), “Properties of the Hubert-Arabie Adjusted Rand Index”, Psychological Methods, 9(3), 386–396.

    Article  Google Scholar 

  • SUBEDI, S., PUNZO, A., INGRASSIA, S., and MCNICHOLAS, P.D. (2013), “Clustering and Classification via Cluster-Weighted Factor Analyzers”, Advances in Data Analysis and Classification, 7(1), 5–40.

    Article  MathSciNet  MATH  Google Scholar 

  • SUBEDI, S., PUNZO, A., INGRASSIA, S., and MCNICHOLAS, P.D. (2015), “Cluster-Weighted t-Factor Analyzers for Robust Model-Based Clustering and Dimension Reduction”, Statistical Methods and Applications, 24(4), 623–649.

    Article  MathSciNet  MATH  Google Scholar 

  • TEICHER, H. (1963), “Identifiability of Finite Mixtures”, The Annals of Mathematical Statistics, 34(4), 1265–1269.

    Article  MathSciNet  MATH  Google Scholar 

  • TITTERINGTON, D.M., SMITH, A.F.M., and MAKOV, U.E. (1985), Statistical Analysis of Finite Mixture Distributions, New York: Wiley.

    MATH  Google Scholar 

  • VENABLES, W.N., and RIPLEY, B.D. (2002), Modern Applied Statistics with S (4th ed.), New York: Springer.

  • VRBIK, I., and MCNICHOLAS, P.D. (2014), “Parsimonious Skew Mixture Models for Model-Based Clustering and Classification”, Computational Statistics and Data Analysis, 71, 196–210.

    Article  MathSciNet  Google Scholar 

  • WEDEL, M. (2002), “Concomitant Variables in Finite Mixture Models”, Statistica Neerlandica, 56(3), 362–375.

    Article  MathSciNet  MATH  Google Scholar 

  • WEI, Y., and MCNICHOLAS, P.D. (2015), “Mixture Model Averaging for Clustering”, Advances in Data Analysis and Classification, 9(2), 197–217.

    Article  MathSciNet  Google Scholar 

  • YAKOWITZ, S.J., and SPRAGINS, J.D. (1968), “On the Identifiability of Finite Mixtures”, The Annals of Mathematical Statistics, 39(1), 209–214.

    Article  MathSciNet  MATH  Google Scholar 

  • ZHOU, H., and LANGE, K.L. (2010), “On the Bumpy Road to the Dominant Mode”, Scandinavian Journal of Statistics, 37(4), 612–631.

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematical Sciences, Binghamton University, State University of New York, 4400 Vestal Parkway East, Binghamton, New York, 13902, USA

    Utkarsh J. Dang

  2. University of Catania, Catania, Italy

    Antonio Punzo & Salvatore Ingrassia

  3. McMaster University, Ontario, Canada

    Paul D. McNicholas

  4. University of Waterloo, Ontario, Canada

    Ryan P. Browne

Authors
  1. Utkarsh J. Dang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antonio Punzo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul D. McNicholas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Salvatore Ingrassia
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ryan P. Browne
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Utkarsh J. Dang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dang, U.J., Punzo, A., McNicholas, P.D. et al. Multivariate Response and Parsimony for Gaussian Cluster-Weighted Models. J Classif 34, 4–34 (2017). https://doi.org/10.1007/s00357-017-9221-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00357-017-9221-2

Keywords

Search

Navigation