Abstract
The Questions About Behavioral Function (QABF) has a high degree of convergent validity, but there is still a lack of agreement between the results of the assessment and the results of experimental function analysis. Machine learning (ML) may improve the validity of assessments by using data to build a mathematical model for more accurate predictions. We used published QABF and subsequent functional analyses to train ML models to identify the function of behavior. With ML models, predictions can be made from indirect assessment results based on learning from results of past experimental functional analyses. In Experiment 1, we compared the results of five algorithms to the QABF criteria using a leave-one-out cross-validation approach. All five outperformed the QABF assessment on multilabel accuracy (i.e., percentage of predictions with the presence or absence of each function indicated correctly), but false negatives remained an issue. In Experiment 2, we augmented the data with 1,000 artificial samples to train and test an artificial neural network. The artificial network outperformed other models on all measures of accuracy. The results indicated that ML could be used to inform conditions that should be present in a functional analysis. Therefore, this study represents a proof-of-concept for the application of machine learning to functional assessment.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aggarwal, C. C. (2015). Data classification: Algorithms and applications. Boca Raton, FL: CRC Press.
Alter, P. J., Conroy, M. A., Mancil, G. R., & Haydon, T. (2008). A comparison of functional behavior assessment methodologies with young children: Descriptive methods and functional analysis. Journal of Behavioral Education, 17(2), 200–219. https://doi.org/10.1007/s10864-008-9064-3.
Beavers, G. A., Iwata, B. A., & Lerman, D. C. (2013). Thirty years of research on the functional analysis of problem behavior. Journal of Applied Behavior Analysis, 46(1), 1–21. https://doi.org/10.1002/jaba.30.
Behavior Analyst Certification Board. (2014). Professional and ethical compliance code for behavior analysts. Retrieved from https://www.bacb.com/wp-content/uploads/2020/05/BACB-Compliance-Code-english_190318.pdf.
Beleites, C., Neugebauer, U., Bocklitz, T., Krafft, C., & Popp, J. (2013). Sample size planning for classification models. Analytica Chimica Acta, 760, 25–33. https://doi.org/10.1016/j.aca.2012.11.007.
Bell, J. (2014). Machine learning: hands-on for developers and technical professionals. Wiley.
Beyer, K., Goldstein, J., Ramakrishnan, R., & Shaft, U. (1999). When is “Nearest Neighbor” meaningful? Lecture Notes in Computer Science Database Theory–ICDT’99 (pp. 217–235). https://doi.org/10.1007/3-540-49257-7_15
Biran, O., & McKeown, K. (2017). Human-centric justification of machine learning predictions. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. https://doi.org/10.24963/ijcai.2017/202
Bischl, B., Lang, M., Kotthoff, L., Schiffner, J., Richter, J., Studerus, E., et al. (2016). mlr: Machine Learning in R. Journal of Machine Learning Research, 17, 1–5.
Breiman, L. (2001). Random forests. Machine Learning, 45, 5–32. https://doi.org/10.1023/A:1010933404324.
Burgos, J. E. (2003). Theoretical note: Simulating latent inhibition with selection ANNs. Behavioural Processes, 62(1–3), 183–192. https://doi.org/10.1016/s0376-6357(03)00025-1.
Burgos, J. E. (2007). Autoshaping and automaintenance: A neural-network approach. Journal of the Experimental Analysis of Behavior, 88(1), 115–130. https://doi.org/10.1901/jeab.2007.75-04.
Bzdok, D., Altman, N., & Krzywinski, M. (2018). Statistics versus machine learning. Nature Methods, 15(4), 233–234. https://doi.org/10.1038/nmeth.4642.
Castelvecchi, D. (2016). Can we open the black box of AI? Nature News, 538, 20–23. https://doi.org/10.1038/538020a.
Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 2(4), 303–314. https://doi.org/10.1007/bf02551274.
Deo, R. C. (2015). Machine learning in medicine. American Heart Association Circulation, 132(20), 1920–1930. https://doi.org/10.1161/circulationaha.115.001593.
Donahoe, J. W., & Palmer, D. C. (2004). Learning and complex behavior. Richmond, MA: Ledgetop Publishing.
Donahoe, J. W., Palmer, D. C., & Burgos, J. E. (1997). The S-R issue: Its status in behavior analysis and in Donahoe and Palmers learning and complex behavior. Journal of the Experimental Analysis of Behavior, 67(2), 193–211. https://doi.org/10.1901/jeab.1997.67-193.
Gravestock, I., & Held, L. (2017). Adaptive power priors with empirical Bayes for clinical trials. Pharmaceutical Statistics, 16(5), 349–360.
Fee, A., Schieber, E., Noble, N., & Valdovinos, M. G. (2016). Agreement between questions about behavior function, the motivation assessment scale, functional assessment interview, and brief functional analysis of children’s challenging behaviors. Behavior Analysis: Research & Practice, 16(2), 94–102. https://doi.org/10.1037/bar0000040.
Fisher, W. W., Kelley, M. E., & Lomas, J. E. (2003). Visual aids and structured criteria for improving visual inspection and interpretation of single-case designs. Journal of Applied Behavior Analysis, 36(3), 387–406. https://doi.org/10.1901/jaba.2003.36-387.
Friedman, J. H. (1997). On bias, variance, 0/1—loss, and the curse-of-dimensionality. Data Mining and Knowledge Discovery, 1(1), 55–77. https://doi.org/10.1023/a:1009778005914.
Fryling, M. J., & Baires, N. A. (2016). The practical importance of the distinction between open and closed-ended indirect assessments. Behavior Analysis in Practice, 9(2), 146–151. https://doi.org/10.1007/s40617-016-0115-2.
Grodzicki, R., Mańdziuk, J., & Wang, L. (2008). Improved multi-label classification with ANNs. Parallel Problem Solving from Nature: PPSN X Lecture Notes in Computer Science (pp. 409–416). https://doi.org/10.1007/978-3-540-87700-4_41
Hagan, M. T., Demuth, H. B., Beale, M. H., & Jesús, O. D. (2016). ANN design. [eBook]. Retrieved from https://hagan.okstate.edu/NNDesign.pdf
Hall, S. S. (2005). Comparing descriptive, experimental and informant-based assessments of problem behaviors. Research in Developmental Disabilities, 26(6), 514–526. https://doi.org/10.1016/j.ridd.2004.11.004.
Hanley, G. P. (2012). Functional assessment of problem behavior: Dispelling myths, overcoming implementation obstacles, and developing new lore. Behavior Analysis in Practice, 5(1), 54–72. https://doi.org/10.1007/bf03391818.
Healy, O., Brett, D., & Leader, G. (2013). A comparison of experimental functional analysis and the Questions About Behavioral Function (QABF) in the assessment of challenging behavior of individuals with autism. Research in Autism Spectrum Disorders, 7(1), 66–81. https://doi.org/10.1016/j.rasd.2012.05.006.
Individuals with Disabilities Education Act, 20 U.S.C. §1400 (2004)
Iwata, B. A., Dorsey, M. F., Slifer, K. J., Bauman, K. E., & Richman, G. S. (1982/1994). Toward a functional analysis of self-injury. Journal of Applied Behavior Analysis, 27(2), 197–209. https://doi.org/10.1901/jaba.1994.27-197.
Iwata, B. A., & Dozier, C. L. (2008). Clinical application of functional analysis methodology. Behavior Analysis in Practice, 1(1), 3–9. https://doi.org/10.1007/bf03391714.
Iwata, B. A., Pace, G. M., Dorsey, M. F., Zarcone, J. R., Vollmer, T. R., Smith, R. G., et al. (1994). The functions of self-injurious behavior: an experimental-epidemiological analysis. Journal of Applied Behavior Analysis, 27(2), 215–240. https://doi.org/10.1901/jaba.1994.27-215.
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning: With applications in R. New York, NY: Springer.
Kittler, J. (1997). Statistical classification. Vistas in Astronomy, 41(3), 405–410. https://doi.org/10.1016/s0083-6656(97)00045-7.
Krstajic, D., Buturovic, L. J., Leahy, D. E., & Thomas, S. (2014). Cross-validation pitfalls when selecting and assessing regression and classification models. Journal of Cheminformatics, 6(1), 10. https://doi.org/10.1186/1758-2946-6-10.
Lanovaz, M. J., Argumedes, M., Roy, D., Duquette, J. R., & Watkins, N. (2013). Using ABC narrative recording to identify the function of problem behavior: A pilot study. Research in Developmental Disabilities, 34(9), 2734–2742. https://doi.org/10.1016/j.ridd.2013.05.038.
Lanovaz, M. J., Giannakakos, A. R., & Destras, O. (2020). Machine learning to analyze single-case data: A proof of concept. Perspectives on Behavior Science, 43(1), 21–38. https://doi.org/10.1007/s40614-020-00244-0.
Lawrence, S., Giles, C. L., & Tsoi, A. C. (1997). Lessons in neural network training: overfitting may be harder than expected [Paper presentation]. Proceedings of the Fourteenth National Conference on Artificial Intelligence and Ninth Innovative Applications of Artificial Intelligence Conference, AAAI 97, IAAI Providence, RI, United States
Lemoine, N. P. (2019). Moving beyond noninformative priors: Why and how to choose weakly informative priors in Bayesian analyses. Oikos, 128(7), 912–928. https://doi.org/10.1111/oik.05985.
Lipton, Z. C., Elkan, C., Naryanaswamy, B. (2014) Optimal thresholding of classifiers to maximize f1 measure. In Calders T., Esposito F., Hüllermeier E., Meo R. (Eds.), Machine learning and knowledge discovery in databases (vol. 8725, pp. 225–239). Berlin, Heidelberg: Springer. https://doi.org/10.1007/978-3-662-44851-9_15.
Mahmood, M., Al-Khateeb, B., & Alwash, W. M. (2020). A review on neural networks approach on classifying cancers. IAES International Journal of Artificial Intelligence, 9(2), 317. https://doi.org/10.11591/ijai.v9.i2.pp317-326.
Matson, J. L., & Vollmer, T. R. (1995). User’s guide: Questions About Behavioral Function (QABF). Baton Rouge, LA: Disability Consultants.
Mohammed, M., Khan, M. B., & Bashier, E. B. (2017). Machine learning: Algorithms and applications. Boca Raton, FL: CRC Press.
Mueller, M. M., Nkosi, A., & Hine, J. F. (2011). Functional analysis in public schools: A summary of 90 functional analyses. Journal of Applied Behavior Analysis, 44(4), 807–818. https://doi.org/10.1901/jaba.2011.44-807.
Paclawskyj, T. R. (1998). Questions about behavioral function (QABF): A behavioral checklist for functional assessment of aberrant behavior (Unpublished doctoral dissertation). Retrieved from https://digitalcommons.lsu.edu/gradschool_disstheses/6855
Paclawskyj, T. R., Matson, J. L., Rush, K. S., Smalls, Y., & Vollmer, T. R. (2001). Assessment of the convergent validity of the Questions About Behavioral Function scale with analogue functional analysis and the Motivation Assessment Scale. Journal of Intellectual Disability Research, 45(6), 484–494. https://doi.org/10.1046/j.1365-2788.2001.00364.x.
Perez, L., & Wang, J. (2017). The effectiveness of data augmentation in image classification using deep learning. arXiv. Preprint arXiv:1712.04621.
Qian, Y., Zhou, W., Yan, J., Li, W., & Han, L. (2014). Comparing machine learning classifiers for object-based land cover classification using very high resolution imagery. Remote Sensing, 7(1), 153–168. https://doi.org/10.3390/rs70100153.
R Core Team. (2019). R: A language and environment for statistical computing (Version 3.6.1). [Computer software]. Vienna, Austria: R Foundation for Statistical Computing.
Raudys, S., & Jain, A. (1991). Small sample size effects in statistical pattern recognition: Recommendations for practitioners. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(3), 252–264. https://doi.org/10.1109/34.75512.
Rooker, G. W., DeLeon, I. G., Borrero, C. S., Frank-Crawford, M. A., & Roscoe, E. M. (2015). Reducing ambiguity in the functional assessment of problem behavior. Behavioral Interventions, 30(1), 1–35. https://doi.org/10.1002/bin.1400.
Safdar, S., Zafar, S., Zafar, N., & Khan, N. F. (2017). Machine learning based decision support systems (DSS) for heart disease diagnosis: A review. Artificial Intelligence Review, 50(4), 597–623. https://doi.org/10.1007/s10462-017-9552-8.
Sarle, W. S. (1995). Stopped training and other remedies for overfitting. Proceedings of the 27th Symposium on the Interface of Computing Science & Statistics. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.42.3920
Smith, J. E., Atif Tahir, M., Sannen, D., & Van Brussel, H. (2012). In Sayed-Mouchaweh, M., & Lughofer, E. (Eds.), Learning in non-stationary environments: Methods and applications (pp. 125–151). Springer. https://doi.org/10.1007/978-1-4419-8020-5.
Shalev-Shwartz, S., & Ben-David, S. (2017). Understanding machine learning: From theory to algorithms. Cambridge, UK: Cambridge University Press.
Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1), 1–48. https://doi.org/10.1186/s40537-019-0197-0.
Sordo M., Zeng Q. (2005) On sample size and classification accuracy: a performance comparison. In: Oliveira J.L., Maojo V., Martín-Sánchez F., Pereira A.S. (Eds.) ISBMDA 2005: Biological and Medical Data Analysis (pp. 193–201). Springer, Berlin, Heidelberg. https://doi.org/10.1007/11573067_20.
Spitzer, J. (2018, July 25). IBM’s Watson recommended “unsafe and incorrect” cancer treatments, STAT report finds. Retrieved from https://www.beckershospitalreview.com/artificial-intelligence/ibm-s-watson-recommended-unsafe-and-incorrect-cancer-treatments-stat-report-finds.html
Strohmeier, C. W., Murphy, A., & O’Connor, J. T. (2016). Parent-informed test-control functional analysis and treatment of problem behavior related to combined establishing operations. Developmental Neurorehabilitation, 20(4), 247–252. https://doi.org/10.3109/17518423.2015.1133723.
Tarbox, J., Wilke, A. E., Najdowski, A. C., Findel-Pyles, R. S., Balasanyan, S., Caveney, A. C., & Tia, B. (2009). Comparing indirect, descriptive, and experimental functional assessments of challenging behavior in children with autism. Journal of Developmental and Physical Disabilities., 21(6), 493–514. https://doi.org/10.1007/s10882-009-9154-8.
Tetko, I. V., Livingstone, D. J., & Luik, A. I. (1995). Neural network studies. 1. Comparison of overfitting and overtraining. Journal of Chemical Information and Computer Sciences, 35(5), 826–833.
Utgoff, P. E. (1989). Incremental induction of decision trees. Machine Learning, 4(2), 161–186. https://doi.org/10.1023/a:1022699900025.
Watkins, N., & Rapp, J. T. (2013). The convergent validity of the Questions About Behavioral Function scale and functional analysis for problem behavior displayed by individuals with autism spectrum disorder. Research in Developmental Disabilities, 34(1), 11–16. https://doi.org/10.1016/j.ridd.2012.08.003.
Wong, T.-T. (2015). Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation. Pattern Recognition, 48(9), 2839–2846. https://doi.org/10.1016/j.patcog.2015.03.009.
Zhang, J., & Shields, M. D. (2018). The effect of prior probabilities on quantification and propagation of imprecise probabilities resulting from small datasets. Computer Methods in Applied Mechanics and Engineering, 334, 483–506. https://doi.org/10.1016/j.cma.2018.01.045.
Zhang, M.-L., Li, Y.-K., Liu, X.-Y., & Geng, X. (2018). Binary relevance for multi-label learning: an overview. Frontiers of Computer Science, 12(2), 191–202. https://doi.org/10.1007/s11704-017-7031-7.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethical Approval
Only published data sets were used in this manuscript
Informed Consent
Informed consent was not necessary because participants were not directly used in the study
Conflicts of Interest
There are no conflicts of interest to disclose
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
Rights and permissions
About this article
Cite this article
Bailey, J.D., Baker, J.C., Rzeszutek, M.J. et al. Machine Learning for Supplementing Behavioral Assessment. Perspect Behav Sci 44, 605–619 (2021). https://doi.org/10.1007/s40614-020-00273-9
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40614-020-00273-9