Skip to main content
SpringerLink
Log in

Recognition of sketching from surface electromyography

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The main objective of this study is to recognize sketching precisely and effectively in human–computer interaction. A surface electromyography (sEMG)-based sketching recognition method is proposed. We conducted an experiment in which we recorded the sEMG signals from the forearm muscles of two participants who were instructed to sketch seven basic one-stroke shapes. Subsequently, seven features of the sEMG time domain were extracted. After reducing data dimensionality using principal component analysis, these features were used as input vectors to a sketching recognition model based on support vector machines (SVMs). The performance of this model was compared to two other recognition models based on multilayer perceptron (MLP) neural networks and self-organization feature map (SOFM) neural networks. The average recognition rates for the seven basic one-stroke shapes of two participants achieved by the SVM-based and MLP-based models were both 98.5% in the test set, which were slightly superior to the performance of the SOFM classifier. Our results demonstrate the feasibility of converting forearm sEMG signals into sketching patterns.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Landay J, Myers B et al (2001) Sketching interfaces: Toward more human interface design. Computer 34(3):56–64

    Article  Google Scholar 

  2. Gross MD, Do EY-L (1996) Ambiguous intentions: a paper-like interface for creative design. In: Proceedings of the 9th annual ACM symposium on user interface software and technology, ACM, pp 183–192

  3. Pu J, Gur D (2009) Automated freehand sketch segmentation using radial basis functions. Comput Aided Des 41(12):857–864

    Article  Google Scholar 

  4. Schön DA (1983) The reflective practitioner: how professionals think in action, vol 5126. Basic books, New York

    Google Scholar 

  5. Sun L, Xiang W, Chai C, Wang C, Huang Q (2014) Creative segment: a descriptive theory applied to computer-aided sketching. Des Stud 35(1):54–79

    Article  Google Scholar 

  6. Olsen L, Samavati FF, Sousa MC, Jorge JA (2009) Sketch-based modeling: a survey. Comput Graph 33(1):85–103

    Article  Google Scholar 

  7. Li Y (2003) Incremental sketch understanding for intention extraction in sketch-based user interfaces. Citeseer

  8. Saponas TS, Tan DS, Morris D, Balakrishnan R (2008) Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, pp 515–524

  9. Chen Y, Liu J, Tang X (2008) Sketching in the air: a vision-based system for 3d object design. In: IEEE Conference on computer vision and pattern recognition, 2008, CVPR 2008, IEEE, pp 1–6

  10. Zhao Y (2012) Human emotion recognition from body language of the head using soft computing techniques. Ph.D. thesis, University of Ottawa

  11. Chen Y, Yang Z, Wang J (2015) Eyebrow emotional expression recognition using surface EMG signals. Neurocomputing 168:871–879

    Article  Google Scholar 

  12. Chowdhury A, Ramadas R, Karmakar S (2013) Muscle computer interface: a review. In: ICoRD’13, Springer, pp 411–421

  13. Kiguchi K, Hayashi Y (2012) An EMG-based control for an upper-limb power-assist exoskeleton robot. IEEE Trans Syst Man Cybern Part B Cybern 42(4):1064–1071

    Article  Google Scholar 

  14. Andreasen D, Gabbert D (2006) Electromyographic switch navigation of power wheelchairs. In: Annual conference of the rehabilitation engineering and assistive technology society of North America

  15. Disselhorst-Klug C, Schmitz-Rode T, Rau G (2009) Surface electromyography and muscle force: limits in sEMG-force relationship and new approaches for applications. Clin Biomech 24(3):225–235

    Article  Google Scholar 

  16. Park E, Meek SG (1995) Adaptive filtering of the electromyographic signal for prosthetic control and force estimation. IEEE Trans Bio-med Eng 42(10):1048–1052

    Article  Google Scholar 

  17. Hernandez Arieta A, Katoh R, Yokoi H, Wenwei Y (2006) Development of a multi-dof electromyography prosthetic system using the adaptive joint mechanism. Appl Bion Biomech 3(2):101–111

    Article  Google Scholar 

  18. Subasi A, Kiymik MK (2010) Muscle fatigue detection in EMG using time-frequency methods, ICA and neural networks. J Med Syst 34(4):777–785

    Article  Google Scholar 

  19. Singh VP, Kumar DK, Polus B, Fraser S (2007) Strategies to identify changes in sEMG due to muscle fatigue during cycling. J Med Eng Technol 31(2):144–151

    Article  Google Scholar 

  20. Chen Y, Yang Z, Wang J, Gong H (2016) Physiological and subjective responses to breathing resistance of N95 filtering facepiece respirators in still-sitting and walking. Int J Ind Ergon 53:93–101

    Article  Google Scholar 

  21. Yang Z, Chen Y, Tang Z, Wang, J (2016) Surface EMG based handgrip force predictions using gene expression programming. Neurocomputing 207:568–579

    Article  Google Scholar 

  22. Linderman M, Lebedev MA, Erlichman JS (2009) Recognition of handwriting from electromyography. PLoS One 4(8):e6791

    Article  Google Scholar 

  23. Ahsan MR, Ibrahimy MI, Khalifa OO et al (2009) Emg signal classification for human computer interaction: a review. Eur J Sci Res 33(3):480–501

    Google Scholar 

  24. Kim J, Mastnik S, André E (2008) EMG-based hand gesture recognition for realtime biosignal interfacing. In: Proceedings of the 13th international conference on intelligent user interfaces, ACM, pp 30–39

  25. Naik GR, Kumar DK, Palaniswami M (2008) Multi run ICA and surface EMG based signal processing system for recognising hand gestures. In: 8th IEEE international conference on computer and information technology, 2008. CIT 2008, IEEE, pp 700–705

  26. Chen Y, Wang J, Yang Z (2014) An EMG-based attitude recognition method of nodding and head-shaking for interactive optimization. J Comput Inf Syst 10(18):7939–7948

    Google Scholar 

  27. Lansari A, Bouslama F, Khasawneh M, Al-Rawi A (2003) A novel electromyography (EMG) based classification approach for Arabic handwriting. In: Proceedings of the international joint conference on neural networks, 2003, pp 2193–2196

  28. Huang G, Zhang D, Zheng X, Zhu X (2010) An EMG-based handwriting recognition through dynamic time warping. In: 2010 Annual international conference of the IEEE engineering in medicine and biology society (EMBC), pp 4902–4905

  29. Li C, Ma Z, Yao L, Zhang D (2013) Improvements on EMG-based handwriting recognition with DTW algorithm. In: 2013 35th Annual international conference of the IEEE engineering in medicine and biology society (EMBC), IEEE, pp 2144–2147

  30. Okorokova E, Lebedev M, Linderman M, Ossadtchi A (2015) A dynamical model improves reconstruction of handwriting from multichannel electromyographic recordings. Front Neurosci 9:389

    Article  Google Scholar 

  31. McKeague IW (2005) A statistical model for signature verification. J Am Stat Assoc 100(469):231–241

    Article  MathSciNet  Google Scholar 

  32. Li Y, Hospedales TM, Song YZ, Gong S. Free-hand sketch recognition by multi-kernel feature learning. Comput Vis Image Underst

  33. De Luca CJ (1997) The use of surface electromyography in biomechanics. J Applied Biomech 13:135–163

    Article  Google Scholar 

  34. Schmieder P, Plimmer B, Blagojevic R (2009) Automatic evaluation of sketch recognizers. In: Proceedings of the 6th eurographics symposium on sketch-based interfaces and modeling, ACM, pp 85–92

  35. Ren H, Xu G, Kee S (2004) Subject-independent natural action recognition. In: Sixth IEEE international conference on automatic face and gesture recognition, 2004. Proceedings, IEEE, pp 523–528

  36. Huang G (2013) Modeling, analysis of surface bioelectric signal and its application in human computer interaction. Ph.D. thesis, Shanghai Jiao Tong University

  37. Baspinar U, Varol HS, Senyurek VY (2013) Performance comparison of artificial neural network and Gaussian mixture model in classifying hand motions by using sEMG signals. Biocybern Biomed Eng 33(1):33–45

    Article  Google Scholar 

  38. Potvin J (1997) Effects of muscle kinematics on surface EMG amplitude and frequency during fatiguing dynamic contractions. J Appl Physiol 82(1):144–151

    Article  Google Scholar 

  39. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    Article  MathSciNet  Google Scholar 

  40. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297

    MATH  Google Scholar 

  41. Zhang K, Sun S (2013) Web music emotion recognition based on higher effective gene expression programming. Neurocomputing 105:100–106

    Article  Google Scholar 

  42. Schölkopf B, Smola A J (2002) Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, Cambridge

    Google Scholar 

  43. Tang QY, Feng MG. DPS data processing system: experimental design, statistical analysis and data mining. Science, Beijing

  44. Kocyigit Y, Alkan A, Erol H (2008) Classification of EEG recordings by using fast independent component analysis and artificial neural network. J Med Syst 32(1):17–20

    Article  Google Scholar 

  45. Übeyli ED (2009) Combined neural network model employing wavelet coefficients for EEG signals classification. Digit Signal Process 19(2):297–308

    Article  Google Scholar 

  46. Tsenov G, Zeghbib A, Palis F, Shoylev N, Mladenov V (2006) Neural networks for online classification of hand and finger movements using surface EMG signals. In: 8th Seminar on neural network applications in electrical engineering, 2006. NEUREL 2006, IEEE, pp 167–171

  47. Gardner M, Dorling S (1998) Artificial neural networks (the multilayer perceptron) a review of applications in the atmospheric sciences. Atmos Environ 32(14):2627–2636

    Article  Google Scholar 

  48. Kohonen T (2012) Self-organization and associative memory, vol 8. Springer, Berlin

    MATH  Google Scholar 

  49. Sadowski L, Nikoo M (2014) Corrosion current density prediction in reinforced concrete by imperialist competitive algorithm. Neural Comput Appl 25(7–8):1627–1638

    Article  Google Scholar 

  50. Nikoo M, Zarfam P, Sayahpour H (2015) Determination of compressive strength of concrete using self organization feature map (sofm). Eng Comput 31(1):113–121

    Article  Google Scholar 

  51. Sadowski Ł, Nikoo M, Nikoo M (2015) Principal component analysis combined with a self organization feature map to determine the pull-off adhesion between concrete layers. Constr Build Mater 78:386–396

    Article  Google Scholar 

  52. NeuroDimension I, Neurosolutions user’s guide. http://www.neurosolutions.com/neurosolutions/help/index.html?NormalizationFile.html

  53. Xiong J, Zhang G, Hu J, Wu L (2014) Bead geometry prediction for robotic gmaw-based rapid manufacturing through a neural network and a second-order regression analysis. J Intell Manuf 25(1):157–163

    Article  Google Scholar 

  54. Chen X, Zhang X, Zhao ZY, Yang JH, Lantz V, Wang KQ (2007) Hand gesture recognition research based on surface EMG sensors and 2d-accelerometers. In: 2007 11th IEEE international symposium on wearable computers, IEEE, pp 11–14

  55. Goel V (1995) Sketches of thought. MIT Press, Cambridge

    Google Scholar 

  56. Taura T, Yamamoto E, Fasiha MYN, Goka M, Mukai F, Nagai Y, Nakashima H (2012) Constructive simulation of creative concept generation process in design: a research method for difficult-to-observe design-thinking processes. J Eng Des 23(4):297–321

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the participants of the experiment. They also thank the editors and anonymous referees for useful comments. This study was partly supported by the National Natural Science Foundation of China (No. 51305077), the Fundamental Research Funds for the Central Universities (No. CUSF-DH-D-2016068), the Zhejiang Provincial Key Laboratory of integration of healthy smart kitchen system (2014E10014), and the China Scholarship Council (CSC). Grant Nos. 201506630036, 201506635030.

Author information

Authors and Affiliations

  1. Fashion Institute, Donghua University, Shanghai, 200051, China

    Yumiao Chen

  2. College of Mechanical Engineering, Donghua University, Shanghai, 201620, China

    Zhongliang Yang

  3. School of Materials, The University of Manchester, Manchester, M13 9PL, UK

    Hugh Gong

  4. College of Mechanical Engineering, Donghua University, Shanghai, 201620, China

    Shengze Wang

Authors
  1. Yumiao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhongliang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hugh Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shengze Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhongliang Yang.

Ethics declarations

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Yang, Z., Gong, H. et al. Recognition of sketching from surface electromyography. Neural Comput & Applic 30, 2725–2737 (2018). https://doi.org/10.1007/s00521-017-2857-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-017-2857-3

Keywords

Search

Navigation