Skip to main content
SpringerLink
Log in

An open source integrated framework for rapid prototyping of multimodal affective applications in digital entertainment

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Multimodal interfaces are becoming more and more an emerging leading-edge area of interest in the Digital Art and Entertainment Domain. This paper aims at proposing a new approach to develop applications relying on integration of multimodal modules which can capture and collect the emotional states of the audience, process them and provide a final emotional output. A concrete framework has been designed and is being developed to provide application designers with a powerful and easy-to-use set of tools for building up multimodal affective interfaces. The proposed approach has been conceived inside the FP6 EU co-funded CALLAS (Conveying Affectiveness in Leading-edge Living Adaptive Systems) Project.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. CALLAS Project, http://www.callas-newmedia.eu/

  2. Myllys M (2002) Context Toolkit, Research Seminar on Middleware for Mobile Computing, Helsinki, University of Helsinki, Department of Computer Science 20th February, 2002

  3. Dragicevic P, Fekete JD (2001) Input device selection and interaction configuration with ICON. In: Proc IHM-HCI. Springer, Berlin, pp 543–548. http://www.dgp.toronto.edu/~dragice/papers/ICon_IHM-HCI.pdf

    Google Scholar 

  4. Sinha AK (2003) Informally prototyping multimodal, multidevice user interfaces. PhD dissertation, Electrical Engineering and Computer Sciences Department, University of California, Berkeley, CA, Fall, 2003

  5. Huot S, Dumas C, Dragicevic P, Fekete JD, Hégron G (2004) The MaggLite Post-WIMP Toolkit: draw it, connect it and run it. In: ACM proceedings: 17th ACM symposium on user interface software and technologies (UIST 2004), Santa Fe, New Mexico, USA, October, 2004, pp 257–266. http://www.emn.fr/x-info/magglite/papers/Huot-uist2004MaggLite.pdf

  6. Nigay L, Coutaz J (1997) The CARE properties and their impact on software design. In: Intelligence and multimodality in multimedia interfaces

  7. Alexandersson J, Becker T (2001) Overlay as the basic operation for discourse processing in a multimodal dialogue system. In: Proc of the 2nd IJCAI workshop on knowledge and reasoning in practical dialogue systems, Seattle, August, 2001

  8. Heider T, Jantz D, Kirste T (2001) Embassi. Comput Graph Top 13(2):15–17

    Google Scholar 

  9. Elting C, Rapp S, Möhler G, Strube M (2003) Architecture and implementation of multimodal plug and play. In: Proceedings of the 5th international conference on multimodal interfaces, Vancouver, British Columbia, Canada, November 5–7, 2003

  10. Galaxy Communicator, http://communicator.sourceforge.net/index.shtml

  11. Turunen M, Hakulinen J, Räihä K-J, Salonen E-P, Kainulainen A, Prusi P (2005) An architecture and applications for speech-based accessibility systems, IBM Syst J 44(3)

  12. openFrameworks, http://openframeworks.cc/

  13. Reas B, Maeda J (2007) Processing: A programming handbook for visual designers and artists, 1st edn. MIT Press, Cambridge. ISBN 0262182629, http://processing.org

    Google Scholar 

  14. Greenberg I (2007) Processing creative coding and computational art (foundation), 1st edn. friends of ED, New York, ISBN 159059617X

    Google Scholar 

  15. Functional requirements & CHIL cooperative information system software design, deliverable D2.1 (part 1: cooperative information system software design) of the project CHIL (Computers in the human interaction loop), IP 506909, 14 July, 2004

  16. Bouchet J, Nigay L (2004) ICARE: a component-based approach for the design and development of multimodal interfaces. In: CHI 2004, Vienna, Austria, April 24–29, 2004

  17. Nicolas V, Trevisan DG, Wamy Wapet S (2004) OpenInterface-Multimodal library for multimodal applications development specification. SIMILAR network of excellence, project no FP6-507609, deliverable D4, 1st version, Communication and Remote Sensing Laboratory, Université Catholique de Louvain, 1348 Louvain-la-Neuve, Belgique, May, 2004

  18. Ghigny C, Lawson JY, Macq B, Nigay L, Trevisan DG (2005) OpenInterface-Multimodal library for multimodal applications development specification. SIMILAR network of excellence, project no. FP6-507609, deliverable D58, Final version, CLIPS-IMAG, Université Joseph Fourier, Grenoble, France TELE, Université Catholique de Louvain, Belgium, August, 2005

  19. Lawson L, Macq B, Mansoux B, Nicolas V, Nigay L, Trevisan DG (2005) OpenInterface platform, SIMILAR network of excellence, project no. FP6-507609, deliverable D60, final version, platform CLIPS-IMAG, Université Joseph Fourier, Grenoble, France, TELE, Université Catholique de Louvain, Belgium, November, 2005

  20. OSC, http://opensoundcontrol.org/

  21. Feigenbaum EA (1988) Book foreword. In: Engelmore R.S., Morgan A. (eds) Blackboard Systems. Addison-Wesley, Reading, pp 13–14

    Google Scholar 

  22. Eugster P, Guerraoui R, Damm C (2001) On objects and events. In: Proceedings for OOPSLA 2001, Tampa Bay, Florida, October 2001

  23. Banavar G, Chandra T, Mukherjee B, Nagarajarao J, Strom RE, Sturman DC (1999) An efficient multicast protocol for content based publish-subscribe systems. In: Proceedings of the 19th international conference on distributed computing systems (ICDCS’99), 1999

  24. Carzaniga A, Rosenblum DS, Wolf AL (2000) Achieving scalability and expressiveness in an Internet-scale event notification service. In: proceedings nineteenth annual ACM symposium on principles of distributed computing (PODC 2000), Portland, Oregon, July 2000, pp. 219–227

  25. Segall B, Arnold D, Boot J, Henderson M, Phelps T (2000) Content based routing with Elvin4. In: AUUG2K, Canberra, Australia, June 2000

  26. Cugola G, Di Nitto E, Fugetta A (2001) The Jedi event-based infrastructure and its application to the development of the opss wfms. IEEE Trans Softw Eng 27(9):827–850

    Article  Google Scholar 

  27. Hapner M, Burridge R, Sharma R, Fialli J, Stout K (2000) Java message service. Sun Microsystems Inc, April 2002

  28. Henning M (2004) Massively multiplayer middleware. ACM Queue 1(10):40–45

    Article  Google Scholar 

  29. Henning M, Spruiell M Distributed programming with Ice, http://www.zeroc.com/Ice-Manual.pdf

  30. Sleepycat Software, Inc (2003) Berkeley DB technical articles. Lincoln, MA: Sleepycat Software, Inc. http://www.sleepycat.com/company/technical.shtml

  31. Henning M (2004) A new approach to object-oriented middleware. IEEE Internet Comput 8(1):66–75

    Article  Google Scholar 

  32. Max/MSP, http://www.cycling74.com/products/max5

  33. Pure Data, http://puredata.info/

  34. Quartz Composer, http://developer.apple.com/graphics-imaging/quartz/quartzcomposer.html

  35. EyesWeb, http://www.infomus.org/EywMain.html

  36. Processing, http://processing.org/

  37. The CALLAS Framework, http://www.callas-newmedia.eu/insights/the-framework.html

  38. Charles F, Lemercier S, Cavazza M, Vogt T, Bee N, André E, Mancini M, Pélachaud C, Urbain J, Price M (2007) Affective interactive narrative in the CALLAS project. In: ICVS2007, Saint-Malo, France, 5–7, Dec, 2007

  39. Gilroy SW, Cavazza M, Chaignon R, Makela SM, Niranen M, Andre E, Vogt T, Urbain J, Billinghurst M, Seichter H, Benayoun M E-tree: emotionally driven augmented reality art. In: Proc ACM multimedia, Vancouver, BC, Canada, 2008, pp 945–948

  40. Cavazza M, Gilroy SW, Niiranen M, André E, Vogt T, Urbain J, Benayoun M, Seichter H, Billinghurst M (2009) PAD-based multimodal affective fusion. In: Proc. ACII 2009, Amsterdam, The Netherlands, September 2009

  41. Asteriadis S, Tzouveli P, Karpouzis K, Kollias S (2007) Non-verbal feedback on user interest based on gaze direction and head pose. In: 2nd international workshop on semantic media adaptation and personalization (SMAP 2007), London, United Kingdom, December, 2007

  42. Niiranen M, Vehkaperä J, Mäkelä S-M, Peltola J, Räty T (2008) Fusion of sound source localization and face detection for supporting human behavior analysis. In: Proc 4th international mobile multimedia communications conference, MobiMedia 2008, ICST, Oulu, Finland, Jul 7–9, 2008

  43. Vogt T, André E (2006) Improving automatic emotion recognition from speech via gender differentiation. In: Proc language resources and evaluation conference (LREC 2006), Genoa, Italy, 2006

  44. Guile 3D Studio—Nicole Visual Character, http://www.guile3d.com.br/nicole_page.asp

Download references

Author information

Authors and Affiliations

  1. Engineering Ingegneria Informatica, viale della Regione Siciliana 7275, Palermo, Italy

    Diego Arnone & Alessandro Rossi

  2. Engineering Ingegneria Informatica, via San Martino della Battaglia 56, Rome, Italy

    Massimo Bertoncini

Authors
  1. Diego Arnone
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alessandro Rossi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Massimo Bertoncini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Diego Arnone.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Arnone, D., Rossi, A. & Bertoncini, M. An open source integrated framework for rapid prototyping of multimodal affective applications in digital entertainment. J Multimodal User Interfaces 3, 227–236 (2010). https://doi.org/10.1007/s12193-010-0035-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-010-0035-3

Keywords

Search

Navigation