Skip to main content
SpringerLink
Log in

Testing IoT systems using a hybrid simulation based testing approach

  • Published:
Computing Aims and scope Submit manuscript

Abstract

This paper presents an extensive overview of the challenges that arise when testing large IoT applications at the system level. In order do that we start from analyzing behavior of local entities such as IoT devices or people interacting with the IoT system. The interactions of these local entities eventually leads to an emergent behavior. Both the emergent behavior and the local behavior need to be taken into account when testing IoT systems. Therefore, we present a novel hybrid simulation based testing approach that is able to effectively facilitate interactions of these local entities. Furthermore, we introduce various solutions to the challenges that arise when implementing this hybrid methodology. These challenges are mainly related to the IoT development pipeline, synchronization between real-life and simulation environment and the scalability constraints of modern simulation techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Arora A, Ertin E, Ramnath R, Nesterenko M, Leal W (2006) Kansei: a high-fidelity sensing testbed. IEEE Internet Comput 10(2):35–47

    Article  Google Scholar 

  2. Beizer B (1995) Black-box testing: techniques for functional testing of software and systems. Wiley, Hoboken

    Google Scholar 

  3. Bertolino A (2007) Software testing research: achievements, challenges, dreams. In: 2007 Future of software engineering. IEEE Computer Society, pp 85–103

  4. Bononi L, Bracuto M, D’Angelo G, Donatiello L (2006) Proximity detection in distributed simulation of wireless mobile systems. In: Proceedings of the 9th ACM international symposium on modeling analysis and simulation of wireless and mobile systems. ACM, pp 44–51

  5. Bormann C, Castellani AP, Shelby Z (2012) Coap: an application protocol for billions of tiny internet nodes. IEEE Internet Comput 16(2):62–67

    Article  Google Scholar 

  6. Bosmans S, Mercelis S, Hellinckx P, Denil J (2018) Towards evaluating emergent behavior of the internet of things using large scale simulation techniques (wip). In: Proceedings of the theory of modeling and simulation symposium. Society for Computer Simulation International, p 4

  7. Carneiro G (2010) Ns-3: network simulator 3. In: UTM Lab Meeting, vol 20

  8. Caughlin D, Sisti AF (1997) Summary of model abstraction techniques. Enabling Technol Simul Sci Int Soc Opt Photon 3083:2–14

    Google Scholar 

  9. Crisan DA, Radoi IE, Arvind D (2013) Coap-mediated hybrid simulation and visualisation environment for specknets. In: Proceedings of the 1st ACM sigsim conference on principles of advanced discrete simulation. ACM, pp 285–294

  10. DAngelo G (2017) The simulation model partitioning problem: an adaptive solution based on self-clustering. Simul Model Pract Theory 70:1–20

    Article  Google Scholar 

  11. D’Angelo G, Ferretti S, Ghini V (2016) Simulation of the internet of things. In: High performance computing & simulation (HPCS), 2016 International conference on IEEE, pp 1–8

  12. Dunkels A, Gronvall B, Voigt T (2004) Contiki-a lightweight and flexible operating system for tiny networked sensors. In: Local computer networks, 2004. 29th Annual IEEE international conference. IEEE, pp 455–462

  13. Fortino G, Gravina R, Russo W, Savaglio C (2017) Modeling and simulating internet-of-things systems: a hybrid agent-oriented approach. Comput Sci Eng 19(5):68–76

    Article  Google Scholar 

  14. Frantz FK (1995) A taxonomy of model abstraction techniques. In: Proceedings of the 27th conference on winter simulation. IEEE Computer Society, pp 1413–1420

  15. Fujimoto RM (2000) Parallel and distributed simulation systems, vol 300. Wiley, New York

    Google Scholar 

  16. Gupta H, Vahid Dastjerdi A, Ghosh SK, Buyya R (2017) iFogSim: A toolkit for modeling and simulation of resource management techniques in the internet of things, edge and fog computing environments. Software: Pract Exp 47(9):1275–1296

    Google Scholar 

  17. Hunkeler U, Truong HL, Stanford-Clark A (2008) Mqtt-sa publish/subscribe protocol for wireless sensor networks. In: Communication systems software and middleware and workshops, 2008. Comsware 2008. 3rd International conference on IEEE, pp 791–798

  18. Latre S, Leroux P, Coenen T, Braem B, Ballon P, Demeester P (2016) City of things: an integrated and multi-technology testbed for iot smart city experiments. In: Smart cities conference (ISC2), 2016 IEEE international. IEEE, pp 1–8

  19. Levis P, Lee N, Welsh M, Culler D (2003) TOSSIM: accurate and scalable simulation of entire tinyOS applications. In: Proceedings of the 1st international conference on embedded networked sensor systems. ACM, pp 126–137

  20. Marjanović M, Antonić A, Žarko IP (2018) Edge computing architecture for mobile crowdsensing. IEEE Access 6:10662–10674

    Article  Google Scholar 

  21. Mataric MJ (1993) Designing emergent behaviors: from local interactions to collective intelligence. In: Proceedings of the 2nd international conference on simulation of adaptive behavior

  22. Murray JA, Sasani M, Shao X (2015) Hybrid simulation for system-level structural response. Eng Struct 103:228–238

    Article  Google Scholar 

  23. Nunes DS, Zhang P, Silva JS (2015) A survey on human-in-the-loop applications towards an internet of all. IEEE Commun Surv Tutor 17(2):944–965

    Article  Google Scholar 

  24. Roca D, Nemirovsky D, Nemirovsky M, Milito R, Valero M (2016) Emergent behaviors in the internet of things: the ultimate ultra-large-scale system. IEEE Micro 36(6):36–44

    Article  Google Scholar 

  25. Rodriguez JD, Bauer Jr KW, Miller JO, Neher Jr RE (2008) Building prediction models of large hierarchical simulation models with artificial neural networks and other statistical techniques. In: Visual information processing XVII, vol 6978. International Society for Optics and Photonics, p 69780

  26. Sanchez L, Muñoz L, Galache JA, Sotres P, Santana JR, Gutierrez V (2014) Smartsantander: Iot experimentation over a smart city testbed. Comput Netw 61:217–238

    Article  Google Scholar 

  27. Van Tendeloo Y, Vangheluwe H (2014) Activity in pythonpdevs. In: ITM Web of conferences, vol 3. EDP Sciences, p 01002

  28. Varga A, Hornig R (2008) An overview of the omnet++ simulation environment. In: Proceedings of the 1st international conference on simulation tools and techniques for communications, networks and systems & workshops, ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), p 60

Download references

Acknowledgements

SeRGIo is a project realized in collaboration with imec. Project partners are bpost, imec, Joyn and Nokia Bell Labs, with project support from VLAIO (Flanders Innovation & Entrepreneurship).

Author information

Authors and Affiliations

  1. imec - IDLab, University of Antwerp, Groenenborgerlaan 171, Antwerp, Belgium

    Stig Bosmans, Siegfried Mercelis & Peter Hellinckx

  2. Flanders Make, University of Antwerp, Groenenborgerlaan 171, Antwerp, Belgium

    Joachim Denil

Authors
  1. Stig Bosmans
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Siegfried Mercelis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joachim Denil
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Peter Hellinckx
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stig Bosmans.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bosmans, S., Mercelis, S., Denil, J. et al. Testing IoT systems using a hybrid simulation based testing approach. Computing 101, 857–872 (2019). https://doi.org/10.1007/s00607-018-0650-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-018-0650-5

Keywords

Mathematics Subject Classification

Search

Navigation