Review of Model-Based Testing Approaches in Production Automation and Adjacent Domains—Current Challenges and Research Gaps
Affiliation(s)
1 Institute of Automation and Information Systems, Technische Universit?t München, München, Germany . 2 Assistant Professorship for Safe Embedded Systems, Technische Universit?t München, München, Germany .
1 Institute of Automation and Information Systems, Technische Universit?t München, München, Germany . 2 Assistant Professorship for Safe Embedded Systems, Technische Universit?t München, München, Germany .
ABSTRACT
As production automation systems have been and are becoming more and more complex, the task of quality assurance is increasingly challenging. Model-based testing is a research field addressing this challenge and many approaches have been suggested for different applications. The goal of this paper is to review these approaches regarding their suitability for the domain of production automation in order to identify current trends and research gaps. The different approaches are classified and clustered according to their main focus which is either testing and test case generation from some form of model automatons, test case generation from models used within the development process of production automation systems, test case generation from fault models or test case selection and regression testing.
As production automation systems have been and are becoming more and more complex, the task of quality assurance is increasingly challenging. Model-based testing is a research field addressing this challenge and many approaches have been suggested for different applications. The goal of this paper is to review these approaches regarding their suitability for the domain of production automation in order to identify current trends and research gaps. The different approaches are classified and clustered according to their main focus which is either testing and test case generation from some form of model automatons, test case generation from models used within the development process of production automation systems, test case generation from fault models or test case selection and regression testing.
KEYWORDS
Model-Based Testing, Automated Production Systems, Conformance Testing, Regression Testing, Fault Injection, Survey
Model-Based Testing, Automated Production Systems, Conformance Testing, Regression Testing, Fault Injection, Survey
Cite this paper
Rösch, S. , Ulewicz, S. , Provost, J. and Vogel-Heuser, B. (2015) Review of Model-Based Testing Approaches in Production Automation and Adjacent Domains—Current Challenges and Research Gaps. Journal of Software Engineering and Applications, 8, 499-519. doi: 10.4236/jsea.2015.89048.
Rösch, S. , Ulewicz, S. , Provost, J. and Vogel-Heuser, B. (2015) Review of Model-Based Testing Approaches in Production Automation and Adjacent Domains—Current Challenges and Research Gaps. Journal of Software Engineering and Applications, 8, 499-519. doi: 10.4236/jsea.2015.89048.
References
[1] (2010) ISO/IEC/IEEE 24765:2010(E), Systems and Software Engineering—Vocabulary. [2] Vogel-Heuser, B., Diedrich, C., Fay, A., Jeschke, S., Kowalewski, S., Wollschlaeger, M. and Gohner, P. (2014) Challenges for Software Engineering in Automation. Journal of Software Engineering and Applications, 7, 440-451.
http://dx.doi.org/10.4236/jsea.2014.75041 [3] Wehrmeister, M.A., Pereira, C.E. and Rammig, F.J. (2013) Aspect-Oriented Model-Driven Engineering for Embedded Systems Applied to Automation Systems. IEEE Transactions on Industrial Informatics, 9, 2373-2386.
http://dx.doi.org/10.1109/TII.2013.2240308 [4] IEC 60880 (2006) Nuclear Power Plants—Instrumentation and Control Systems Important to Safety—Software Aspects for Computer-Based Systems Performing Category A Functions. International Electrotechnical Commission, 2nd Edition. [5] IEC 61850-10 (2005) Communications Networks and Systems in Substations—Part 10: Conformance Testing. International Electrotechnical Commission, 2nd Edition. [6] Dubey, A. (2011) Evaluating Software Engineering Methods in the Context of Automation Applications. 2011 9th IEEE International Conference on Industrial Informatics (INDIN), Caparica, 26-29 July 2011, 585-590.
http://dx.doi.org/10.1109/INDIN.2011.6034944 [7] Utting, M., Pretschner, A. and Legeard, B. (2012) A Taxonomy of Model-Based Testing Approaches. Software Testing, Verification and Reliability, 22, 297-312.
http://dx.doi.org/10.1002/stvr.456 [8] Vyatkin, V. (2013) Software Engineering in Industrial Automation: State-of-the-Art Review. IEEE Transactions on Industrial Informatics, 9, 1234-1249.
http://dx.doi.org/10.1109/TII.2013.2258165 [9] Thramboulidis, K. and Frey, G. (2011) Towards a Model-Driven IEC 61131-Based Development Process in Industrial Automation. Journal of Software Engineering and Applications, 4, 217-226.
http://dx.doi.org/10.4236/jsea.2011.44024 [10] Corriveau, J.-P. and Shi, W. (2013) Traceability in Acceptance Testing. Journal of Software Engineering and Applications, 6, 36-46.
http://dx.doi.org/10.4236/jsea.2013.610A005 [11] Roussel, J.-M. and Lesage, J.-J. (1996) Validation and Verification of Grafcets Using Finite State Machine. Proceedings of IMACS-IEEE, CESA’96, Lille, 9-12 July 1996, 1-6. [12] Frey, G. and Litz, L. (2000) Formal Methods in PLC Programming. 2000 IEEE International Conference on Systems, Man, and Cybernetics, 4, 2431-2436.
http://dx.doi.org/10.1109/icsmc.2000.884356 [13] Younis, M.B. and Frey, G. (2003) Formalization of Existing PLC Programs: A Survey. Proceedings of IMACS-IEEE, CESA’03, Lille, 9-11 July 2003, 234-239. [14] Girbea, A., Suciu, C., Nechifor, S. and Sisak, F. (2014) Design and Implementation of a Service-Oriented Architecture for the Optimization of Industrial Applications. IEEE Transactions on Industrial Informatics, 10, 185-196.
http://dx.doi.org/10.1109/TII.2013.2253112 [15] Verein Deutscher Ingenieure (2005) Classification and Evaluation of Description Methods in Automation and Control Technology. [16] Van Veenendaal, E. (2014) Standard Glossary of Terms Used in Software Testing. International Software Testing Qualifications Board, Version 2.3, 1-53. [17] Hsueh, M.-C., Tsai, T.K. and Iyer, R.K. (1997) Fault Injection Techniques and Tools. Computer, 30, 75-82.
http://dx.doi.org/10.1109/2.585157 [18] Barth, M. and Fay, A. (2013) Automated Generation of Simulation Models for Control Code Tests. Control Engineering Practice, 21, 218-230.
http://dx.doi.org/10.1016/j.conengprac.2012.09.022 [19] Carlsson, H., Svensson, B., Danielsson, F. and Lennartson, B. (2012) Methods for Reliable Simulation-Based PLC Code Verification. IEEE Transactions on Industrial Informatics, 8, 267-278.
http://dx.doi.org/10.1109/TII.2011.2182653 [20] Gu, F., Harrison, W.S., Tilbury, D.M. and Yuan, C. (2007) Hardware-in-the-Loop for Manufacturing Automation Control: Current Status and Identified Needs. 2007 IEEE International Conference on Automation Science and Engineering, CASE 2007, Scottsdale, 22-25 September 2007, 1105-1110.
http://dx.doi.org/10.1109/COASE.2007.4341787 [21] Myers, G.J., Sandler, C., Badgett, T. and Thomas, T.M. (2004) The Art of Software Testing. Second Edition, John Wiley & Sons, Hoboken. [22] Thramboulidis, K. (2012) IEC 61499: Back to the Well Proven Practice of IEC 61131? 2012 IEEE 17th Conference on Emerging Technologies & Factory Automation (ETFA), Krakow, 17-21 September 2012, 1-8.
http://dx.doi.org/10.1109/etfa.2012.6489672 [23] Zoitl, A. and Prahofer, H. (2013) Guidelines and Patterns for Building Hierarchical Automation Solutions in the IEC 61499 Modeling Language. IEEE Transactions on Industrial Informatics, 9, 2387-2396.
http://dx.doi.org/10.1109/TII.2012.2235449 [24] Avizienis, A., Laprie, J.-C., Randell, B. and Landwehr, C. (2004) Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Transactions on Dependable and Secure Computing, 1, 11-33.
http://dx.doi.org/10.1109/TDSC.2004.2 [25] Fernandez, J.-C., Jard, C., Jéron, T. and Viho, C. (1996) Using On-the-Fly Verification Techniques for the Generation of Test Suites. In: Alur, R. and Henzinger, T.A., Eds., Computer Aided Verification, Springer, Berlin, 348-359.
http://dx.doi.org/10.1007/3-540-61474-5_82 [26] Lee, D. and Yannakakis, M. (1996) Principles and Methods of Testing Finite State Machines—A Survey. Proceedings of the IEEE, 84, 1090-1123.
http://dx.doi.org/10.1109/5.533956 [27] Tretmans, J. (1996) Test Generation with Inputs, Outputs and Repetitive Quiescence. Software, Concepts and Tools, 17, 103-120. [28] Tretmans, J. (2008) Model Based Testing with Labelled Transition Systems. In: Hierons, R.M., Bowen, J.P. and Harman, M., Eds., Formal Methods and Testing, Lecture Notes in Computer Science, Vol. 4949, Springer, Berlin, 1-38.
http://dx.doi.org/10.1007/978-3-540-78917-8_1 [29] Pickin, S., Jard, C., Jéron, T., Jézéquel, J.-M. and Le Traon, Y. (2007) Test Synthesis from UML Models of Distributed Software. IEEE Transactions on Software Engineering, 33, 252-269.
http://dx.doi.org/10.1109/TSE.2007.39 [30] Krichen, M. and Tripakis, S. (2009) Conformance Testing for Real-Time Systems. Formal Methods in System Design, 34, 238-304.
http://dx.doi.org/10.1007/s10703-009-0065-1 [31] Pocci, M., Demongodin, I., Giambiasi, N. and Giua, A. (2014) Testing Experiments on Synchronized Petri Nets. IEEE Transactions on Automation Science and Engineering, 11, 125-138.
http://dx.doi.org/10.1109/TASE.2013.2290774 [32] Guignard, A. and Faure, J.-M. (2014) A Conformance Relation for Model-Based Testing of PLC. Proceedings of the 12th International Workshop on Discrete Event Systems-WODES 2014, Cachan, 14-16 May 2014, 412-419. [33] Fujiwara, S., von Bochmann, G., Khendek, F., Amalou, M. and Ghedamsi, A. (1991) Test Selection Based on Finite State Models. IEEE Transactions on Software Engineering, 17, 591-603.
http://dx.doi.org/10.1109/32.87284 [34] Dorofeeva, R., El-Fakih, K., Maag, S., Cavalli, A.R. and Yevtushenko, N. (2010) FSM-Based Conformance Testing Methods: A Survey Annotated with Experimental Evaluation. Information and Software Technology, 52, 1286-1297.
http://dx.doi.org/10.1016/j.infsof.2010.07.001 [35] Endo, A.T. and Simao, A. (2013) Evaluating Test Suite Characteristics, Cost, and Effectiveness of FSM-Based Testing Methods. Information and Software Technology, 55, 1045-1062.
http://dx.doi.org/10.1016/j.infsof.2013.01.001 [36] Petrenko, A., Simao, A. and Yevtushenko, N. (2012) Generating Checking Sequences for Nondeterministic Finite State Machines. 2012 IEEE 5th International Conference on Software Testing, Verification and Validation (ICST), Montreal, 17-21 April 2012, 310-319.
http://dx.doi.org/10.1109/ICST.2012.111 [37] Hierons, R.M. and Türker, U.C. (2014) Distinguishing Sequences for Partially Specified FSMs. In: Badger, J.M. and Rozier, K.Y., Eds., NASA Formal Methods, Springer, Cham, 62-76.
http://dx.doi.org/10.1007/978-3-319-06200-6_5 [38] Pocci, M., Demongodin, I., Giambiasi, N. and Giua, A. (2013) A New Algorithm to Compute Synchronizing Sequences for Synchronized Petri Nets. 2013 IEEE Region 10 Conference (31194), TENCON 2013, Xi’an, 22-25 October 2013, 1-6.
http://dx.doi.org/10.1109/tencon.2013.6718970 [39] Kaner, C., Bach, J. and Pettichord, B. (2008) Lessons Learned in Software Testing. John Wiley & Sons, Hoboken. [40] Simao, A., Petrenko, A. and Yevtushenko, N. (2009) Generating Reduced Tests for FSMs with Extra States. In: Núnez, M., Baker, P. and Merayo, M.G., Eds., Testing of Software and Communication Systems, Springer, Berlin, 129-145.
http://dx.doi.org/10.1007/978-3-642-05031-2_9 [41] El-Fakih, K., Yevtushenko, N. and Fouchal, H. (2009) Testing Timed Finite State Machines with Guaranteed Fault Coverage. In: Núnez, M., Baker, P. and Merayo, M.G., Eds., Testing of Software and Communication Systems, Springer, Berlin, 66-80.
http://dx.doi.org/10.1007/978-3-642-05031-2_5 [42] White, L.J. and Cohen, E.I. (1980) A Domain Strategy for Computer Program Testing. IEEE Transactions on Software Engineering, SE-6, 247-257.
http://dx.doi.org/10.1109/TSE.1980.234486 [43] Andrade, W.D.L., Machado, P.D., Jeron, T. and Marchand, H. (2011) Abstracting Time and Data for Conformance Testing of Real-Time Systems. IEEE 4th International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Berlin, 21-25 March 2011, 9-17.
http://dx.doi.org/10.1109/icstw.2011.82 [44] Andrade, W.D.L. and Machado, P.D. (2013) Generating Test Cases for Real-Time Systems Based on Symbolic Models. IEEE Transactions on Software Engineering, 39, 1216-1229.
http://dx.doi.org/10.1109/TSE.2013.13 [45] Cadar, C. and Sen, K. (2013) Symbolic Execution for Software Testing: Three Decades Later. Communications of the ACM, 56, 82-90.
http://dx.doi.org/10.1145/2408776.2408795 [46] Petrenko, A. and Yevtushenko, N. (2005) Testing from Partial Deterministic FSM Specifications. IEEE Transactions on Computers, 54, 1154-1165.
http://dx.doi.org/10.1109/TC.2005.152 [47] Shahbaz, M. and Groz, R. (2014) Analysis and Testing of Black-Box Component-Based Systems by Inferring Partial Models. Software Testing, Verification and Reliability, 24, 253-288.
http://dx.doi.org/10.1002/stvr.1491 [48] Constant, C., Jeron, T., Marchand, H. and Rusu, V. (2007) Integrating Formal Verification and Conformance Testing for Reactive Systems. IEEE Transactions on Software Engineering, 33, 558-574.
http://dx.doi.org/10.1109/TSE.2007.70707 [49] Armando, A., Pellegrino, G., Carbone, R., Merlo, A. and Balzarotti, D. (2012) From Model-Checking to Automated Testing of Security Protocols: Bridging the Gap. In: Brucker, A.D. and Julliand, J., Eds., Tests and Proofs, Springer, Berlin, 3-18.
http://dx.doi.org/10.1007/978-3-642-30473-6_3 [50] Enoiu, E.P., Sundmark, D. and Pettersson, P. (2013) Model-Based Test Suite Generation for Function Block Diagrams Using the UPPAAL Model Checker. 2013 IEEE 6th International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Luxembourg, 18-22 March 2013, 158-167.
http://dx.doi.org/10.1109/ICSTW.2013.27 [51] Gaudel, M.-C. (2011) Checking Models, Proving Programs, and Testing Systems. In: Gogolla, M. and Wolff, B., Eds., Tests and Proofs, Springer, Berlin, 1-13.
http://dx.doi.org/10.1007/978-3-642-21768-5_1 [52] Gaudel, M.-C., Lassaigne, R., Magniez, F. and de Rougemont, M. (2013) Some Approximations in Model Checking and Testing. arXiv: 1304.5199. [53] Hanisch, H.-M., Thieme, J., Luder, A. and Wienhold, O. (1997) Modeling of PLC Behavior by Means of Timed Net Condition/Event Systems. 1997 6th International Conference on Emerging Technologies and Factory Automation Proceedings, ETFA’97, Los Angeles, 9-12 September 1997, 391-396.
http://dx.doi.org/10.1109/ETFA.1997.616302 [54] Heiner, M. and Menzel, T. (1998) A Petri Net Semantics for the PLC Language Instruction List. 4th Workshop on Discrete Event Systems (WODES’98), Cagliari, 26-28 August 1998, 161-166. [55] Mader, A. and Wupper, H. (1999) Timed Automaton Models for Simple Programmable Logic Controllers. Proceedings of the 11th Euromicro Conference on Real-Time Systems, York, 9-11 June 1999, 106-113.
http://dx.doi.org/10.1109/emrts.1999.777456 [56] Canet, G., Couffin, S., Lesage, J.-J., Petit, A. and Schnoebelen, P. (2000) Towards the Automatic Verification of PLC Programs Written in Instruction List. 2000 IEEE International Conference on Systems, Man, and Cybernetics, 4, 2449-2454.
http://dx.doi.org/10.1109/icsmc.2000.884359 [57] Jiménez-Fraustro, F. and Rutten, E. (2001) A Synchronous Model of IEC 61131 PLC Languages in SIGNAL. 13th Euromicro Conference on Real-Time Systems, Delft, 13-15 June 2001, 135-142.
http://dx.doi.org/10.1109/EMRTS.2001.934016 [58] Gourcuff, V., De Smet, O. and Faure, J. (2006) Efficient Representation for Formal Verification of PLC Programs. 2006 8th International Workshop on Discrete Event Systems, Ann Arbor, 10-12 July 2006, 182-187.
http://dx.doi.org/10.1109/WODES.2006.1678428 [59] Zoubek, B., Roussel, J.-M. and Kwiatkowska, M. (2003) Towards Automatic Verification of Ladder Logic Programs. Proceedings of IMACS-IEEE CESA’03: Computational Engineering in Systems Applications, Lille, 9-11 July 2003. [60] Bender, D.F., Combemale, B., Crégut, X., Farines, J.M., Berthomieu, B. and Vernadat, F. (2008) Ladder Metamodeling and PLC Program Validation through Time Petri Nets. In: Schieferdecker, I. and Hartman, A., Eds., Model Driven Architecture—Foundations and Applications, Springer, Berlin, 121-136.
http://dx.doi.org/10.1007/978-3-540-69100-6_9 [61] Bel Mokadem, H., Berard, B., Gourcu, V., De Smet, O. and Roussel, J.-M. (2010) Verification of a Timed Multitask System with UPPAAL. IEEE Transactions on Automation Science and Engineering, 7, 921-932.
http://dx.doi.org/10.1109/TASE.2010.2050199 [62] Pavlovic, O. and Ehrich, H.-D. (2010) Model Checking PLC Software Written in Function Block Diagram. 2010 3rd International Conference on Software Testing, Verification and Validation (ICST), Paris, 6-10 April 2010, 439-448.
http://dx.doi.org/10.1109/icst.2010.10 [63] Soliman, D., Thramboulidis, K. and Frey, G. (2012) Transformation of Function Block Diagrams to UPPAAL Timed Automata for the Verification of Safety Applications. Annual Reviews in Control, 36, 338-345.
http://dx.doi.org/10.1016/j.arcontrol.2012.09.015 [64] Wardana, A., Folmer, J. and Vogel-Heuser, B. (2009) Automatic Program Verification of Continuous Function Chart Based on Model Checking. 35th Annual Conference of IEEE Industrial Electronics, Porto, 3-5 November 2009, 2422-2427.
http://dx.doi.org/10.1109/iecon.2009.5415231 [65] Yoong, L.H., Roop, P.S., Vyatkin, V. and Salcic, Z. (2009) A Synchronous Approach for IEC 61499 Function Block Implementation. IEEE Transactions on Computers, 58, 1599-1614.
http://dx.doi.org/10.1109/TC.2009.128 [66] Prahofer, H. and Zoitl, A. (2013) Verification of Hierarchical IEC 61499 Component Systems with Behavioral Event Contracts. 2013 11th IEEE International Conference on Industrial Informatics (INDIN), Bochum, 29-31 July 2013, 578-585.
http://dx.doi.org/10.1109/INDIN.2013.6622948 [67] L’Her, D., Le Parc, P. and Marce, L. (1999) Proving Sequential Function Chart Programs Using Automata. In: Champarnaud, J.-M., Ziadi, D. and Maurel, D., Eds., Automata Implementation, Springer, Berlin, 149-163.
http://dx.doi.org/10.1007/3-540-48057-9_13 [68] Remelhe, M., Lohmann, S., Stursberg, O., Engell, S. and Bauer, N. (2004) Algorithmic Verification of Logic Controllers Given as Sequential Function Charts. 2004 IEEE International Symposium on Computer Aided Control Systems Design, Taipei, 4 September 2004, 53-58.
http://dx.doi.org/10.1109/CACSD.2004.1393850 [69] Bauer, N., Engell, S., Huuck, R., Lohmann, S., Lukoschus, B., Remelhe, M. and Stursberg, O. (2004) Verification of PLC Programs Given as Sequential Function Charts. In: Ehrig, H., Damm, W., Desel, J., Große-Rhode, M., Reif, W., Schnieder, E. and Westkamper, E., Eds., Integration of Software Specification Techniques for Applications in Engineering, Lecture Notes in Computer Science, Vol. 3147, Part V: Verification, Springer, Berlin, 517-540.
http://dx.doi.org/10.1007/978-3-540-27863-4_28 [70] Bauer, N., Huuck, R., Lukoschus, B. and Engell, S. (2004) A Unifying Semantics for Sequential Function Charts. In: Ehrig, H., Damm, W., Desel, J., Große-Rhode, M., Reif, W., Schnieder, E., Westkamper, E., Eds., Integration of Software Specification Techniques for Applications in Engineering, Springer, Berlin, 400-418.
http://dx.doi.org/10.1007/978-3-540-27863-4_22 [71] Provost, J., Roussel, J.-M. and Faure, J.-M. (2011) Translating Grafcet Specifications into Mealy Machines for Conformance Test Purposes. Control Engineering Practice, 19, 947-957.
http://dx.doi.org/10.1016/j.conengprac.2010.10.001 [72] Provost, J., Roussel, J.-M. and Faure, J.-M. (2011) A Formal Semantics for Grafcet Specifications. Proceedings of the IEEE 7th International Conference on Automation Science and Engineering (CASE 2011), Trieste, 24-27 August 2011, 488-494.
http://dx.doi.org/10.1109/CASE.2011.6042457 [73] Provost, J., Roussel, J.-M. and Faure, J.-M. (2014) Generation of Single Input Change Test Sequences for Conformance Test of Programmable Logic Controllers. IEEE Transactions on Industrial Informatics, 10, 1696-1704.
http://dx.doi.org/10.1109/TII.2014.2315972 [74] Schumacher, F., Schrok, S. and Fay, A. (2013) Transforming Hierarchical Concepts of GRAFCET into a Suitable Petri Net Formalism. Manufacturing Modelling, Management, and Control, 7, 295-300. [75] Schumacher, F. and Fay, A. (2013) Transforming Time Constraints of a GRAFCET Graph into a Suitable Petri Net Formalism. 2013 IEEE International Conference on Industrial Technology (ICIT), Cape Town, 25-28 February 2013, 210-218.
http://dx.doi.org/10.1109/ICIT.2013.6505674 [76] Hamon, G. and Rushby, J. (2007) An Operational Semantics for Stateflow. International Journal on Software Tools for Technology Transfer, 9, 447-456.
http://dx.doi.org/10.1007/s10009-007-0049-7 [77] Beer, A., Georgiev, T., Leitner-Fischer, F. and Leue, S. (2013) Model-Based Quantitative Safety Analysis of Matlab Simulink/Stateow Models. Tagungsband des Dagstuhl-Workshops, Dagstuhl, 24-26 April 2013. [78] Jouault, F., Allilaire, F., Bezivin, J. and Kurtev, I. (2008) ATL: A Model Transformation Tool. Science of Computer Programming, 72, 31-39.
http://dx.doi.org/10.1016/j.scico.2007.08.002 [79] Schetinin, N., Moriz, N., Kumar, B., Maier, A., Faltinski, S. and Niggemann, O. (2013) Why Do Verification Approaches in Automation Rarely Use HIL-Test? 2013 IEEE International Conference on Industrial Technology (ICIT), Cape Town, 25-28 February 2013, 1428-1433.
http://dx.doi.org/10.1109/ICIT.2013.6505881 [80] Obermeier, M., Braun, S. and Vogel-Heuser, B. (2014) A Model-Driven Approach on Object Oriented PLC Programming for Manufacturing Systems with Regard to Usability. IEEE Transactions on Industrial Informatics, 11, 790-800.
http://dx.doi.org/10.1109/TII.2014.2346133 [81] Hametner, R., Winkler, D., Ostreicher, T., Bi, S. and Zoitl, A. (2010) The Adaptation of Test-Driven Software Processes to Industrial Automation Engineering. 2010 8th IEEE International Conference on Industrial Informatics (INDIN), Osaka, 13-16 July 2010, 921-927.
http://dx.doi.org/10.1109/indin.2010.5549620 [82] Hussain, T. and Frey, G. (2006) UML-Based Development Process for IEC 61499 with Automatic Test-Case Generation. IEEE Conference on Emerging Technologies and Factory Automation (ETFA’06), Prague, 20-22 September 2006, 1277-1284.
http://dx.doi.org/10.1109/etfa.2006.355407 [83] Hametner, R., Kormann, B., Vogel-Heuser, B., Winkler, D. and Zoitl, A. (2011) Test Case Generation Approach for Industrial Automation Systems. 2011 5th International Conference on Automation, Robotics and Applications, Wellington, 6-8 December 2011, 57-62.
http://dx.doi.org/10.1109/ICARA.2011.6144856 [84] Krause, J., Herrmann, A. and Diedrich, C. (2008) Test Case Generation from Formal System Specifications Based on UML State Machine. ATP-International, 1, 47-54. [85] Kormann, B., Tikhonov, D. and Vogel-Heuser, B. (2012) Automated PLC Software Testing Using Adapted UML Sequence Diagrams. 14th IFAC Symposium of Information Control Problems in Manufacturing, 14, 1615-1621. [86] Zander, J., Dai, Z.R., Schieferdecker, I. and Din, G. (2005) From U2TP Models to Executable Tests with TTCN-3— An Approach to Model Driven Testing. In: Khendek, F. and Dssouli, R., Eds., Testing of Communicating Systems, Springer, Berlin, 289-303.
http://dx.doi.org/10.1007/11430230_20 [87] Kumar, B., Czybik, B. and Jasperneite, J. (2011) Model Based TTCN-3 Testing of Industrial Automation Systems— First Results. 2011 IEEE 16th Conference on Emerging Technologies and Factory Automation, Toulouse, 5-9 September 2011, 1-4.
http://dx.doi.org/10.1109/etfa.2011.6059146 [88] DeTommasi, G., Vitelli, R., Boncagni, L. and Neto, A.C. (2013) Modeling of MARTe-Based Real-Time Applications with SysML. IEEE Transactions on Industrial Informatics, 9, 2407-2415.
http://dx.doi.org/10.1109/TII.2012.2235073 [89] Bayrak, G. and Vogel-Heuser, B. (2014) Evaluation of Programming Languages for a Flexible Programming of Thermo-Mechanical Processes by Means of Cognitive Effectiveness. IEEE. [90] Bayrak, G., Murr, P., Ulewicz, S. and Vogel-Heuser, B. (2012) Comparison of a Transformed Matlab/Simulink Model into the Programming Language CFC on Different IEC 61131-3 PLC Environments. 2012 IEEE 17th Conference on Emerging Technologies & Factory Automation (ETFA), Krakow, 17-21 September 2012, 1-8.
http://dx.doi.org/10.1109/ETFA.2012.6489667 [91] Petrenko, A., Dury, A., Ramesh, S. and Mohalik, S. (2013) A Method and Tool for Test Optimization for Automotive Controllers. 2013 IEEE 6th International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Luxembourg, 18-22 March 2013, 198-207.
http://dx.doi.org/10.1109/ICSTW.2013.31 [92] Zelenov, S., Silakov, D., Petrenko, A., Conrad, M. and Fey, I. (2006) Automatic Test Generation for Model-Based Code Generators. ISoLA 2006, 2nd International Symposium on Leveraging Applications of Formal Methods, Verification and Validation, Paphos, 15-19 November 2006, 75-81.
http://dx.doi.org/10.1109/ISoLA.2006.70 [93] Estévez, E. and Marcos, M. (2012) Model-Based Validation of Industrial Control Systems. IEEE Transactions on Industrial Informatics, 8, 302-310.
http://dx.doi.org/10.1109/TII.2011.2174248 [94] Svenningsson, R., Vinter, J., Eriksson, H. and Torngren, M. (2010) MODIFI: A MODel-Implemented Fault Injection Tool. In: Schoitsch, E., Ed., Computer Safety, Reliability, and Security, Lecture Notes in Computer Science, Vol. 6351, Springer, Berlin, 210-222.
http://dx.doi.org/10.1007/978-3-642-15651-9_16 [95] Arlat, J., Aguera, M., Amat, L., Crouzet, Y., Fabre, J.-C., Laprie, J.-C., Martins, E. and Powell, D. (1990) Fault Injection for Dependability Validation: A Methodology and Some Applications. IEEE Transactions on Software Engineering, 16, 166-182.
http://dx.doi.org/10.1109/32.44380 [96] Aidemark, J., Vinter, J., Folkesson, P. and Karlsson, J. (2001) GOOFI: Generic Object-Oriented Fault Injection Tool. 2001 International Conference on Dependable Systems and Networks, DSN 2001, Goteborg, 1-4 July 2001, 83-88.
http://dx.doi.org/10.1109/DSN.2001.941394 [97] Ziade, H., Ayoubi, R. and Velazco, R. (2004) A Survey on Fault Injection Techniques. The International Arab Journal of Information Technology, 1, 171-186. [98] Baraza, J., Gracia, J., Gil, D. and Gil, P. (2005) Improvement of Fault Injection Techniques Based on VHDL Code Modification. 10th IEEE International High-Level Design Validation and Test Workshop, 30 November-2 December 2005, 19-26.
http://dx.doi.org/10.1109/hldvt.2005.1568808 [99] Schlingloff, H. and Vulinovic, S. (2005) Model Based Dependability Evaluation for Automotive Control Functions. In: Fritzson, P., Ed., Modeling and Simulation for Public Safety. [100] Vinterl, J., Bromander, L., Raistrick, P. and Edlerl, H. (2007) FISCADE—A Fault Injection Tool for SCADE Models. 2007 3rd Institution of Engineering and Technology Conference on Automotive Electronics, Warwick, 28-29 June 2007, 1-9. [101] Powell, D., Arlat, J., Chu, H.N., Ingrand, F. and Killijian, M. (2012) Testing the Input Timing Robustness of Real-Time Control Software for Autonomous Systems. 2012 9th European Dependable Computing Conference, Sibiu, 8-11 May 2012, 73-83.
http://dx.doi.org/10.1109/EDCC.2012.16 [102] Kormann, B. and Vogel-Heuser, B. (2011) Automated Test Case Generation Approach for PLC Control Software Exception Handling Using Fault Injection. IECON 2011—37th Annual Conference on IEEE Industrial Electronics Society, Melbourne, 7-10 November 2011, 365-372.
http://dx.doi.org/10.1109/IECON.2011.6119280 [103] Rosch, S., Tikhonov, D., Schutz, D. and Vogel-Heuser, B. (2014) Model-Based Testing of PLC Software: Test of Plants’ Reliability by Using Fault Injection on Component Level. IFAC World Conference, Cape Town, 24-29 August 2014. [104] Sung, A., Choi, B., Wong, W.E. and Debroy, V. (2011) Mutant Generation for Embedded Systems Using Kernel-Based Software and Hardware Fault Simulation. Information and Software Technology, 53, 1153-1164.
http://dx.doi.org/10.1016/j.infsof.2011.03.010 [105] Arlat, J. and Crouzet, Y. (2010) Physical Fault Models and Fault Tolerance. In: Wunderlich, H.-J., Ed., Models in Hardware Testing, Frontiers in Electronic Testing, Vol. 43, Springer Netherlands, Dordrecht, 217-255.
http://dx.doi.org/10.1007/978-90-481-3282-9_8 [106] Schludermann, H., Kirchmair, T. and Vorderwinkler, M. (2000) Soft-Commissioning: Hardware-in-the-Loop-Based Verification of Controller Software. Proceedings of the 2000 Winter Simulation Conference, 1, 893-899.
http://dx.doi.org/10.1109/wsc.2000.899889 [107] Carreira, J., Madeira, H. and Silva, J.G. (1998) Xception: Software Fault Injection and Monitoring in Processor Functional Units. Dependable Computing and Fault Tolerant Systems, 10, 245-266. [108] Yuste, P., Ruiz, J.C., Lemus, L. and Gil, P. (2003) Non-Intrusive Software-Implemented Fault Injection in Embedded Systems. In: de Lemos, R., Weber, T.S. and Camargo Jr., J.B., Eds., Dependable Computing, Springer, Berlin, 23-38.
http://dx.doi.org/10.1007/978-3-540-45214-0_5 [109] (1998) IEEE Std 1219-1998, IEEE Standard for Software Maintenance. The Institute of Electrical and Electronics Engineers, Inc., Los Alamos. [110] Rothermel, G. and Harrold, M. (1996) Analyzing Regression Test Selection Techniques. IEEE Transactions on Software Engineering, 22, 529-551.
http://dx.doi.org/10.1109/32.536955 [111] Yoo, S. and Harman, M. (2010) Regression Testing Minimization, Selection and Prioritization: A Survey. Software Testing, Verification and Reliability, 22, 67-120.
http://dx.doi.org/10.1002/stv.430 [112] Bohner, S.A. and Arnold, R.S. (1996) Software Change Impact Analysis. The Institute of Electrical and Electronic Engineers, Inc., Los Alamos. [113] Farooq, Q., Iqbal, M.Z.Z., Malik, Z.I. and Nadeem, A. (2007) An Approach for Selective State Machine Based Regression Testing. Proceedings of the 3rd International Workshop on Advances in Model-Based Testing (A-MOST’07), ACM Press, New York, 44-52.
http://dx.doi.org/10.1145/1291535.1291540 [114] Briand, L., Labiche, Y. and Soccar, G. (2002) Automating Impact Analysis and Regression Test Selection Based on UML Designs. International Conference on Software Maintenance, Montreal, 3-6 October 2002, 252-261.
http://dx.doi.org/10.1109/icsm.2002.1167775 [115] Le Traon, Y., Jéron, T., Jézéquel, J.-M. and Morel, P. (2000) Efficient Object-Oriented Integration and Regression Testing. IEEE Transactions on Reliability, 49, 12-25.
http://dx.doi.org/10.1109/24.855533 [116] Caliebe, P., Herpel, T. and German, R. (2012) Dependency-Based Test Case Selection and Prioritization in Embedded Systems. 2012 IEEE 5th International Conference on Software Testing, Verification and Validation, Montreal, 17-21 April 2012, 731-735.
http://dx.doi.org/10.1109/ICST.2012.164 [117] Chen, Y., Probert, R.L. and Ural, H. (2009) Regression Test Suite Reduction Based on SDL Models of System Requirements. Journal of Software Maintenance and Evolution: Research and Practice, 21, 379-405.
http://dx.doi.org/10.1002/smr.415 [118] Korel, B., Tahat, L. and Vaysburg, B. (2002) Model Based Regression Test Reduction Using Dependence Analysis. 2002 International Conference on Software Maintenance, Montreal, 3-6 October 2002, 214-223.
http://dx.doi.org/10.1109/icsm.2002.1167768 [119] Ryder, B. and Tip, F. (2001) Change Impact Analysis for Object-Oriented Programs. Proceedings of the 2001 ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering, 46-53.
http://dx.doi.org/10.1145/379605.379661 [120] Leung, H. and White, L. (1990) A Study of Integration Testing and Software Regression at the Integration Level. 1990 International Conference on Software Maintenance, San Diego, 26-29 November 1990, 290-301.
http://dx.doi.org/10.1109/icsm.1990.131377 [121] Ren, X., Shah, F., Tip, F., Ryder, B.G. and Chesley, O. (2004) Chianti: A Tool for Change Impact Analysis of Java Programs Categories and Subject Descriptors. ACM Sigplan Notices, 39, 432-448.
http://dx.doi.org/10.1145/1028976.1029012 [122] Orso, A., Apiwattanapong, T. and Harrold, M.J. (2003) Leveraging Field Data for Impact Analysis and Regression Testing. ACM SIGSOFT Software Engineering Notes, 28, 128-137.
http://dx.doi.org/10.1145/949952.940089 [123] Rothermel, G. and Harrold, M.J. (1997) A Safe, Efficient Regression Test Selection Technique. ACM Transactions on Software Engineering and Methodology, 6, 173-210.
http://dx.doi.org/10.1145/248233.248262 [124] Chen, Y.F., Rosenblum, D. and Vo, K. (1994) TESTTUBE: A System for Selective Regression Testing. Proceedings of 16th International Conference on Software Engineering, (ICSE’94), Sorrento, 16-21 May 1994, 211-220.
http://dx.doi.org/10.1109/ICSE.1994.296780 [125] Ulewicz, S., Schütz, D. and Vogel-Heuser, B. (2014) Software Changes in Factory Automation—Towards Automatic Change Based Regression Testing. IECON 2014, 40th Annual Conference of the IEEE Industrial Electronics Society, Dallas, 29 October-1 November 2014, 2617-2623.
http://dx.doi.org/10.1109/IECON.2014.7048875
[1] (2010) ISO/IEC/IEEE 24765:2010(E), Systems and Software Engineering—Vocabulary. [2] Vogel-Heuser, B., Diedrich, C., Fay, A., Jeschke, S., Kowalewski, S., Wollschlaeger, M. and Gohner, P. (2014) Challenges for Software Engineering in Automation. Journal of Software Engineering and Applications, 7, 440-451.
http://dx.doi.org/10.4236/jsea.2014.75041 [3] Wehrmeister, M.A., Pereira, C.E. and Rammig, F.J. (2013) Aspect-Oriented Model-Driven Engineering for Embedded Systems Applied to Automation Systems. IEEE Transactions on Industrial Informatics, 9, 2373-2386.
http://dx.doi.org/10.1109/TII.2013.2240308 [4] IEC 60880 (2006) Nuclear Power Plants—Instrumentation and Control Systems Important to Safety—Software Aspects for Computer-Based Systems Performing Category A Functions. International Electrotechnical Commission, 2nd Edition. [5] IEC 61850-10 (2005) Communications Networks and Systems in Substations—Part 10: Conformance Testing. International Electrotechnical Commission, 2nd Edition. [6] Dubey, A. (2011) Evaluating Software Engineering Methods in the Context of Automation Applications. 2011 9th IEEE International Conference on Industrial Informatics (INDIN), Caparica, 26-29 July 2011, 585-590.
http://dx.doi.org/10.1109/INDIN.2011.6034944 [7] Utting, M., Pretschner, A. and Legeard, B. (2012) A Taxonomy of Model-Based Testing Approaches. Software Testing, Verification and Reliability, 22, 297-312.
http://dx.doi.org/10.1002/stvr.456 [8] Vyatkin, V. (2013) Software Engineering in Industrial Automation: State-of-the-Art Review. IEEE Transactions on Industrial Informatics, 9, 1234-1249.
http://dx.doi.org/10.1109/TII.2013.2258165 [9] Thramboulidis, K. and Frey, G. (2011) Towards a Model-Driven IEC 61131-Based Development Process in Industrial Automation. Journal of Software Engineering and Applications, 4, 217-226.
http://dx.doi.org/10.4236/jsea.2011.44024 [10] Corriveau, J.-P. and Shi, W. (2013) Traceability in Acceptance Testing. Journal of Software Engineering and Applications, 6, 36-46.
http://dx.doi.org/10.4236/jsea.2013.610A005 [11] Roussel, J.-M. and Lesage, J.-J. (1996) Validation and Verification of Grafcets Using Finite State Machine. Proceedings of IMACS-IEEE, CESA’96, Lille, 9-12 July 1996, 1-6. [12] Frey, G. and Litz, L. (2000) Formal Methods in PLC Programming. 2000 IEEE International Conference on Systems, Man, and Cybernetics, 4, 2431-2436.
http://dx.doi.org/10.1109/icsmc.2000.884356 [13] Younis, M.B. and Frey, G. (2003) Formalization of Existing PLC Programs: A Survey. Proceedings of IMACS-IEEE, CESA’03, Lille, 9-11 July 2003, 234-239. [14] Girbea, A., Suciu, C., Nechifor, S. and Sisak, F. (2014) Design and Implementation of a Service-Oriented Architecture for the Optimization of Industrial Applications. IEEE Transactions on Industrial Informatics, 10, 185-196.
http://dx.doi.org/10.1109/TII.2013.2253112 [15] Verein Deutscher Ingenieure (2005) Classification and Evaluation of Description Methods in Automation and Control Technology. [16] Van Veenendaal, E. (2014) Standard Glossary of Terms Used in Software Testing. International Software Testing Qualifications Board, Version 2.3, 1-53. [17] Hsueh, M.-C., Tsai, T.K. and Iyer, R.K. (1997) Fault Injection Techniques and Tools. Computer, 30, 75-82.
http://dx.doi.org/10.1109/2.585157 [18] Barth, M. and Fay, A. (2013) Automated Generation of Simulation Models for Control Code Tests. Control Engineering Practice, 21, 218-230.
http://dx.doi.org/10.1016/j.conengprac.2012.09.022 [19] Carlsson, H., Svensson, B., Danielsson, F. and Lennartson, B. (2012) Methods for Reliable Simulation-Based PLC Code Verification. IEEE Transactions on Industrial Informatics, 8, 267-278.
http://dx.doi.org/10.1109/TII.2011.2182653 [20] Gu, F., Harrison, W.S., Tilbury, D.M. and Yuan, C. (2007) Hardware-in-the-Loop for Manufacturing Automation Control: Current Status and Identified Needs. 2007 IEEE International Conference on Automation Science and Engineering, CASE 2007, Scottsdale, 22-25 September 2007, 1105-1110.
http://dx.doi.org/10.1109/COASE.2007.4341787 [21] Myers, G.J., Sandler, C., Badgett, T. and Thomas, T.M. (2004) The Art of Software Testing. Second Edition, John Wiley & Sons, Hoboken. [22] Thramboulidis, K. (2012) IEC 61499: Back to the Well Proven Practice of IEC 61131? 2012 IEEE 17th Conference on Emerging Technologies & Factory Automation (ETFA), Krakow, 17-21 September 2012, 1-8.
http://dx.doi.org/10.1109/etfa.2012.6489672 [23] Zoitl, A. and Prahofer, H. (2013) Guidelines and Patterns for Building Hierarchical Automation Solutions in the IEC 61499 Modeling Language. IEEE Transactions on Industrial Informatics, 9, 2387-2396.
http://dx.doi.org/10.1109/TII.2012.2235449 [24] Avizienis, A., Laprie, J.-C., Randell, B. and Landwehr, C. (2004) Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Transactions on Dependable and Secure Computing, 1, 11-33.
http://dx.doi.org/10.1109/TDSC.2004.2 [25] Fernandez, J.-C., Jard, C., Jéron, T. and Viho, C. (1996) Using On-the-Fly Verification Techniques for the Generation of Test Suites. In: Alur, R. and Henzinger, T.A., Eds., Computer Aided Verification, Springer, Berlin, 348-359.
http://dx.doi.org/10.1007/3-540-61474-5_82 [26] Lee, D. and Yannakakis, M. (1996) Principles and Methods of Testing Finite State Machines—A Survey. Proceedings of the IEEE, 84, 1090-1123.
http://dx.doi.org/10.1109/5.533956 [27] Tretmans, J. (1996) Test Generation with Inputs, Outputs and Repetitive Quiescence. Software, Concepts and Tools, 17, 103-120. [28] Tretmans, J. (2008) Model Based Testing with Labelled Transition Systems. In: Hierons, R.M., Bowen, J.P. and Harman, M., Eds., Formal Methods and Testing, Lecture Notes in Computer Science, Vol. 4949, Springer, Berlin, 1-38.
http://dx.doi.org/10.1007/978-3-540-78917-8_1 [29] Pickin, S., Jard, C., Jéron, T., Jézéquel, J.-M. and Le Traon, Y. (2007) Test Synthesis from UML Models of Distributed Software. IEEE Transactions on Software Engineering, 33, 252-269.
http://dx.doi.org/10.1109/TSE.2007.39 [30] Krichen, M. and Tripakis, S. (2009) Conformance Testing for Real-Time Systems. Formal Methods in System Design, 34, 238-304.
http://dx.doi.org/10.1007/s10703-009-0065-1 [31] Pocci, M., Demongodin, I., Giambiasi, N. and Giua, A. (2014) Testing Experiments on Synchronized Petri Nets. IEEE Transactions on Automation Science and Engineering, 11, 125-138.
http://dx.doi.org/10.1109/TASE.2013.2290774 [32] Guignard, A. and Faure, J.-M. (2014) A Conformance Relation for Model-Based Testing of PLC. Proceedings of the 12th International Workshop on Discrete Event Systems-WODES 2014, Cachan, 14-16 May 2014, 412-419. [33] Fujiwara, S., von Bochmann, G., Khendek, F., Amalou, M. and Ghedamsi, A. (1991) Test Selection Based on Finite State Models. IEEE Transactions on Software Engineering, 17, 591-603.
http://dx.doi.org/10.1109/32.87284 [34] Dorofeeva, R., El-Fakih, K., Maag, S., Cavalli, A.R. and Yevtushenko, N. (2010) FSM-Based Conformance Testing Methods: A Survey Annotated with Experimental Evaluation. Information and Software Technology, 52, 1286-1297.
http://dx.doi.org/10.1016/j.infsof.2010.07.001 [35] Endo, A.T. and Simao, A. (2013) Evaluating Test Suite Characteristics, Cost, and Effectiveness of FSM-Based Testing Methods. Information and Software Technology, 55, 1045-1062.
http://dx.doi.org/10.1016/j.infsof.2013.01.001 [36] Petrenko, A., Simao, A. and Yevtushenko, N. (2012) Generating Checking Sequences for Nondeterministic Finite State Machines. 2012 IEEE 5th International Conference on Software Testing, Verification and Validation (ICST), Montreal, 17-21 April 2012, 310-319.
http://dx.doi.org/10.1109/ICST.2012.111 [37] Hierons, R.M. and Türker, U.C. (2014) Distinguishing Sequences for Partially Specified FSMs. In: Badger, J.M. and Rozier, K.Y., Eds., NASA Formal Methods, Springer, Cham, 62-76.
http://dx.doi.org/10.1007/978-3-319-06200-6_5 [38] Pocci, M., Demongodin, I., Giambiasi, N. and Giua, A. (2013) A New Algorithm to Compute Synchronizing Sequences for Synchronized Petri Nets. 2013 IEEE Region 10 Conference (31194), TENCON 2013, Xi’an, 22-25 October 2013, 1-6.
http://dx.doi.org/10.1109/tencon.2013.6718970 [39] Kaner, C., Bach, J. and Pettichord, B. (2008) Lessons Learned in Software Testing. John Wiley & Sons, Hoboken. [40] Simao, A., Petrenko, A. and Yevtushenko, N. (2009) Generating Reduced Tests for FSMs with Extra States. In: Núnez, M., Baker, P. and Merayo, M.G., Eds., Testing of Software and Communication Systems, Springer, Berlin, 129-145.
http://dx.doi.org/10.1007/978-3-642-05031-2_9 [41] El-Fakih, K., Yevtushenko, N. and Fouchal, H. (2009) Testing Timed Finite State Machines with Guaranteed Fault Coverage. In: Núnez, M., Baker, P. and Merayo, M.G., Eds., Testing of Software and Communication Systems, Springer, Berlin, 66-80.
http://dx.doi.org/10.1007/978-3-642-05031-2_5 [42] White, L.J. and Cohen, E.I. (1980) A Domain Strategy for Computer Program Testing. IEEE Transactions on Software Engineering, SE-6, 247-257.
http://dx.doi.org/10.1109/TSE.1980.234486 [43] Andrade, W.D.L., Machado, P.D., Jeron, T. and Marchand, H. (2011) Abstracting Time and Data for Conformance Testing of Real-Time Systems. IEEE 4th International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Berlin, 21-25 March 2011, 9-17.
http://dx.doi.org/10.1109/icstw.2011.82 [44] Andrade, W.D.L. and Machado, P.D. (2013) Generating Test Cases for Real-Time Systems Based on Symbolic Models. IEEE Transactions on Software Engineering, 39, 1216-1229.
http://dx.doi.org/10.1109/TSE.2013.13 [45] Cadar, C. and Sen, K. (2013) Symbolic Execution for Software Testing: Three Decades Later. Communications of the ACM, 56, 82-90.
http://dx.doi.org/10.1145/2408776.2408795 [46] Petrenko, A. and Yevtushenko, N. (2005) Testing from Partial Deterministic FSM Specifications. IEEE Transactions on Computers, 54, 1154-1165.
http://dx.doi.org/10.1109/TC.2005.152 [47] Shahbaz, M. and Groz, R. (2014) Analysis and Testing of Black-Box Component-Based Systems by Inferring Partial Models. Software Testing, Verification and Reliability, 24, 253-288.
http://dx.doi.org/10.1002/stvr.1491 [48] Constant, C., Jeron, T., Marchand, H. and Rusu, V. (2007) Integrating Formal Verification and Conformance Testing for Reactive Systems. IEEE Transactions on Software Engineering, 33, 558-574.
http://dx.doi.org/10.1109/TSE.2007.70707 [49] Armando, A., Pellegrino, G., Carbone, R., Merlo, A. and Balzarotti, D. (2012) From Model-Checking to Automated Testing of Security Protocols: Bridging the Gap. In: Brucker, A.D. and Julliand, J., Eds., Tests and Proofs, Springer, Berlin, 3-18.
http://dx.doi.org/10.1007/978-3-642-30473-6_3 [50] Enoiu, E.P., Sundmark, D. and Pettersson, P. (2013) Model-Based Test Suite Generation for Function Block Diagrams Using the UPPAAL Model Checker. 2013 IEEE 6th International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Luxembourg, 18-22 March 2013, 158-167.
http://dx.doi.org/10.1109/ICSTW.2013.27 [51] Gaudel, M.-C. (2011) Checking Models, Proving Programs, and Testing Systems. In: Gogolla, M. and Wolff, B., Eds., Tests and Proofs, Springer, Berlin, 1-13.
http://dx.doi.org/10.1007/978-3-642-21768-5_1 [52] Gaudel, M.-C., Lassaigne, R., Magniez, F. and de Rougemont, M. (2013) Some Approximations in Model Checking and Testing. arXiv: 1304.5199. [53] Hanisch, H.-M., Thieme, J., Luder, A. and Wienhold, O. (1997) Modeling of PLC Behavior by Means of Timed Net Condition/Event Systems. 1997 6th International Conference on Emerging Technologies and Factory Automation Proceedings, ETFA’97, Los Angeles, 9-12 September 1997, 391-396.
http://dx.doi.org/10.1109/ETFA.1997.616302 [54] Heiner, M. and Menzel, T. (1998) A Petri Net Semantics for the PLC Language Instruction List. 4th Workshop on Discrete Event Systems (WODES’98), Cagliari, 26-28 August 1998, 161-166. [55] Mader, A. and Wupper, H. (1999) Timed Automaton Models for Simple Programmable Logic Controllers. Proceedings of the 11th Euromicro Conference on Real-Time Systems, York, 9-11 June 1999, 106-113.
http://dx.doi.org/10.1109/emrts.1999.777456 [56] Canet, G., Couffin, S., Lesage, J.-J., Petit, A. and Schnoebelen, P. (2000) Towards the Automatic Verification of PLC Programs Written in Instruction List. 2000 IEEE International Conference on Systems, Man, and Cybernetics, 4, 2449-2454.
http://dx.doi.org/10.1109/icsmc.2000.884359 [57] Jiménez-Fraustro, F. and Rutten, E. (2001) A Synchronous Model of IEC 61131 PLC Languages in SIGNAL. 13th Euromicro Conference on Real-Time Systems, Delft, 13-15 June 2001, 135-142.
http://dx.doi.org/10.1109/EMRTS.2001.934016 [58] Gourcuff, V., De Smet, O. and Faure, J. (2006) Efficient Representation for Formal Verification of PLC Programs. 2006 8th International Workshop on Discrete Event Systems, Ann Arbor, 10-12 July 2006, 182-187.
http://dx.doi.org/10.1109/WODES.2006.1678428 [59] Zoubek, B., Roussel, J.-M. and Kwiatkowska, M. (2003) Towards Automatic Verification of Ladder Logic Programs. Proceedings of IMACS-IEEE CESA’03: Computational Engineering in Systems Applications, Lille, 9-11 July 2003. [60] Bender, D.F., Combemale, B., Crégut, X., Farines, J.M., Berthomieu, B. and Vernadat, F. (2008) Ladder Metamodeling and PLC Program Validation through Time Petri Nets. In: Schieferdecker, I. and Hartman, A., Eds., Model Driven Architecture—Foundations and Applications, Springer, Berlin, 121-136.
http://dx.doi.org/10.1007/978-3-540-69100-6_9 [61] Bel Mokadem, H., Berard, B., Gourcu, V., De Smet, O. and Roussel, J.-M. (2010) Verification of a Timed Multitask System with UPPAAL. IEEE Transactions on Automation Science and Engineering, 7, 921-932.
http://dx.doi.org/10.1109/TASE.2010.2050199 [62] Pavlovic, O. and Ehrich, H.-D. (2010) Model Checking PLC Software Written in Function Block Diagram. 2010 3rd International Conference on Software Testing, Verification and Validation (ICST), Paris, 6-10 April 2010, 439-448.
http://dx.doi.org/10.1109/icst.2010.10 [63] Soliman, D., Thramboulidis, K. and Frey, G. (2012) Transformation of Function Block Diagrams to UPPAAL Timed Automata for the Verification of Safety Applications. Annual Reviews in Control, 36, 338-345.
http://dx.doi.org/10.1016/j.arcontrol.2012.09.015 [64] Wardana, A., Folmer, J. and Vogel-Heuser, B. (2009) Automatic Program Verification of Continuous Function Chart Based on Model Checking. 35th Annual Conference of IEEE Industrial Electronics, Porto, 3-5 November 2009, 2422-2427.
http://dx.doi.org/10.1109/iecon.2009.5415231 [65] Yoong, L.H., Roop, P.S., Vyatkin, V. and Salcic, Z. (2009) A Synchronous Approach for IEC 61499 Function Block Implementation. IEEE Transactions on Computers, 58, 1599-1614.
http://dx.doi.org/10.1109/TC.2009.128 [66] Prahofer, H. and Zoitl, A. (2013) Verification of Hierarchical IEC 61499 Component Systems with Behavioral Event Contracts. 2013 11th IEEE International Conference on Industrial Informatics (INDIN), Bochum, 29-31 July 2013, 578-585.
http://dx.doi.org/10.1109/INDIN.2013.6622948 [67] L’Her, D., Le Parc, P. and Marce, L. (1999) Proving Sequential Function Chart Programs Using Automata. In: Champarnaud, J.-M., Ziadi, D. and Maurel, D., Eds., Automata Implementation, Springer, Berlin, 149-163.
http://dx.doi.org/10.1007/3-540-48057-9_13 [68] Remelhe, M., Lohmann, S., Stursberg, O., Engell, S. and Bauer, N. (2004) Algorithmic Verification of Logic Controllers Given as Sequential Function Charts. 2004 IEEE International Symposium on Computer Aided Control Systems Design, Taipei, 4 September 2004, 53-58.
http://dx.doi.org/10.1109/CACSD.2004.1393850 [69] Bauer, N., Engell, S., Huuck, R., Lohmann, S., Lukoschus, B., Remelhe, M. and Stursberg, O. (2004) Verification of PLC Programs Given as Sequential Function Charts. In: Ehrig, H., Damm, W., Desel, J., Große-Rhode, M., Reif, W., Schnieder, E. and Westkamper, E., Eds., Integration of Software Specification Techniques for Applications in Engineering, Lecture Notes in Computer Science, Vol. 3147, Part V: Verification, Springer, Berlin, 517-540.
http://dx.doi.org/10.1007/978-3-540-27863-4_28 [70] Bauer, N., Huuck, R., Lukoschus, B. and Engell, S. (2004) A Unifying Semantics for Sequential Function Charts. In: Ehrig, H., Damm, W., Desel, J., Große-Rhode, M., Reif, W., Schnieder, E., Westkamper, E., Eds., Integration of Software Specification Techniques for Applications in Engineering, Springer, Berlin, 400-418.
http://dx.doi.org/10.1007/978-3-540-27863-4_22 [71] Provost, J., Roussel, J.-M. and Faure, J.-M. (2011) Translating Grafcet Specifications into Mealy Machines for Conformance Test Purposes. Control Engineering Practice, 19, 947-957.
http://dx.doi.org/10.1016/j.conengprac.2010.10.001 [72] Provost, J., Roussel, J.-M. and Faure, J.-M. (2011) A Formal Semantics for Grafcet Specifications. Proceedings of the IEEE 7th International Conference on Automation Science and Engineering (CASE 2011), Trieste, 24-27 August 2011, 488-494.
http://dx.doi.org/10.1109/CASE.2011.6042457 [73] Provost, J., Roussel, J.-M. and Faure, J.-M. (2014) Generation of Single Input Change Test Sequences for Conformance Test of Programmable Logic Controllers. IEEE Transactions on Industrial Informatics, 10, 1696-1704.
http://dx.doi.org/10.1109/TII.2014.2315972 [74] Schumacher, F., Schrok, S. and Fay, A. (2013) Transforming Hierarchical Concepts of GRAFCET into a Suitable Petri Net Formalism. Manufacturing Modelling, Management, and Control, 7, 295-300. [75] Schumacher, F. and Fay, A. (2013) Transforming Time Constraints of a GRAFCET Graph into a Suitable Petri Net Formalism. 2013 IEEE International Conference on Industrial Technology (ICIT), Cape Town, 25-28 February 2013, 210-218.
http://dx.doi.org/10.1109/ICIT.2013.6505674 [76] Hamon, G. and Rushby, J. (2007) An Operational Semantics for Stateflow. International Journal on Software Tools for Technology Transfer, 9, 447-456.
http://dx.doi.org/10.1007/s10009-007-0049-7 [77] Beer, A., Georgiev, T., Leitner-Fischer, F. and Leue, S. (2013) Model-Based Quantitative Safety Analysis of Matlab Simulink/Stateow Models. Tagungsband des Dagstuhl-Workshops, Dagstuhl, 24-26 April 2013. [78] Jouault, F., Allilaire, F., Bezivin, J. and Kurtev, I. (2008) ATL: A Model Transformation Tool. Science of Computer Programming, 72, 31-39.
http://dx.doi.org/10.1016/j.scico.2007.08.002 [79] Schetinin, N., Moriz, N., Kumar, B., Maier, A., Faltinski, S. and Niggemann, O. (2013) Why Do Verification Approaches in Automation Rarely Use HIL-Test? 2013 IEEE International Conference on Industrial Technology (ICIT), Cape Town, 25-28 February 2013, 1428-1433.
http://dx.doi.org/10.1109/ICIT.2013.6505881 [80] Obermeier, M., Braun, S. and Vogel-Heuser, B. (2014) A Model-Driven Approach on Object Oriented PLC Programming for Manufacturing Systems with Regard to Usability. IEEE Transactions on Industrial Informatics, 11, 790-800.
http://dx.doi.org/10.1109/TII.2014.2346133 [81] Hametner, R., Winkler, D., Ostreicher, T., Bi, S. and Zoitl, A. (2010) The Adaptation of Test-Driven Software Processes to Industrial Automation Engineering. 2010 8th IEEE International Conference on Industrial Informatics (INDIN), Osaka, 13-16 July 2010, 921-927.
http://dx.doi.org/10.1109/indin.2010.5549620 [82] Hussain, T. and Frey, G. (2006) UML-Based Development Process for IEC 61499 with Automatic Test-Case Generation. IEEE Conference on Emerging Technologies and Factory Automation (ETFA’06), Prague, 20-22 September 2006, 1277-1284.
http://dx.doi.org/10.1109/etfa.2006.355407 [83] Hametner, R., Kormann, B., Vogel-Heuser, B., Winkler, D. and Zoitl, A. (2011) Test Case Generation Approach for Industrial Automation Systems. 2011 5th International Conference on Automation, Robotics and Applications, Wellington, 6-8 December 2011, 57-62.
http://dx.doi.org/10.1109/ICARA.2011.6144856 [84] Krause, J., Herrmann, A. and Diedrich, C. (2008) Test Case Generation from Formal System Specifications Based on UML State Machine. ATP-International, 1, 47-54. [85] Kormann, B., Tikhonov, D. and Vogel-Heuser, B. (2012) Automated PLC Software Testing Using Adapted UML Sequence Diagrams. 14th IFAC Symposium of Information Control Problems in Manufacturing, 14, 1615-1621. [86] Zander, J., Dai, Z.R., Schieferdecker, I. and Din, G. (2005) From U2TP Models to Executable Tests with TTCN-3— An Approach to Model Driven Testing. In: Khendek, F. and Dssouli, R., Eds., Testing of Communicating Systems, Springer, Berlin, 289-303.
http://dx.doi.org/10.1007/11430230_20 [87] Kumar, B., Czybik, B. and Jasperneite, J. (2011) Model Based TTCN-3 Testing of Industrial Automation Systems— First Results. 2011 IEEE 16th Conference on Emerging Technologies and Factory Automation, Toulouse, 5-9 September 2011, 1-4.
http://dx.doi.org/10.1109/etfa.2011.6059146 [88] DeTommasi, G., Vitelli, R., Boncagni, L. and Neto, A.C. (2013) Modeling of MARTe-Based Real-Time Applications with SysML. IEEE Transactions on Industrial Informatics, 9, 2407-2415.
http://dx.doi.org/10.1109/TII.2012.2235073 [89] Bayrak, G. and Vogel-Heuser, B. (2014) Evaluation of Programming Languages for a Flexible Programming of Thermo-Mechanical Processes by Means of Cognitive Effectiveness. IEEE. [90] Bayrak, G., Murr, P., Ulewicz, S. and Vogel-Heuser, B. (2012) Comparison of a Transformed Matlab/Simulink Model into the Programming Language CFC on Different IEC 61131-3 PLC Environments. 2012 IEEE 17th Conference on Emerging Technologies & Factory Automation (ETFA), Krakow, 17-21 September 2012, 1-8.
http://dx.doi.org/10.1109/ETFA.2012.6489667 [91] Petrenko, A., Dury, A., Ramesh, S. and Mohalik, S. (2013) A Method and Tool for Test Optimization for Automotive Controllers. 2013 IEEE 6th International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Luxembourg, 18-22 March 2013, 198-207.
http://dx.doi.org/10.1109/ICSTW.2013.31 [92] Zelenov, S., Silakov, D., Petrenko, A., Conrad, M. and Fey, I. (2006) Automatic Test Generation for Model-Based Code Generators. ISoLA 2006, 2nd International Symposium on Leveraging Applications of Formal Methods, Verification and Validation, Paphos, 15-19 November 2006, 75-81.
http://dx.doi.org/10.1109/ISoLA.2006.70 [93] Estévez, E. and Marcos, M. (2012) Model-Based Validation of Industrial Control Systems. IEEE Transactions on Industrial Informatics, 8, 302-310.
http://dx.doi.org/10.1109/TII.2011.2174248 [94] Svenningsson, R., Vinter, J., Eriksson, H. and Torngren, M. (2010) MODIFI: A MODel-Implemented Fault Injection Tool. In: Schoitsch, E., Ed., Computer Safety, Reliability, and Security, Lecture Notes in Computer Science, Vol. 6351, Springer, Berlin, 210-222.
http://dx.doi.org/10.1007/978-3-642-15651-9_16 [95] Arlat, J., Aguera, M., Amat, L., Crouzet, Y., Fabre, J.-C., Laprie, J.-C., Martins, E. and Powell, D. (1990) Fault Injection for Dependability Validation: A Methodology and Some Applications. IEEE Transactions on Software Engineering, 16, 166-182.
http://dx.doi.org/10.1109/32.44380 [96] Aidemark, J., Vinter, J., Folkesson, P. and Karlsson, J. (2001) GOOFI: Generic Object-Oriented Fault Injection Tool. 2001 International Conference on Dependable Systems and Networks, DSN 2001, Goteborg, 1-4 July 2001, 83-88.
http://dx.doi.org/10.1109/DSN.2001.941394 [97] Ziade, H., Ayoubi, R. and Velazco, R. (2004) A Survey on Fault Injection Techniques. The International Arab Journal of Information Technology, 1, 171-186. [98] Baraza, J., Gracia, J., Gil, D. and Gil, P. (2005) Improvement of Fault Injection Techniques Based on VHDL Code Modification. 10th IEEE International High-Level Design Validation and Test Workshop, 30 November-2 December 2005, 19-26.
http://dx.doi.org/10.1109/hldvt.2005.1568808 [99] Schlingloff, H. and Vulinovic, S. (2005) Model Based Dependability Evaluation for Automotive Control Functions. In: Fritzson, P., Ed., Modeling and Simulation for Public Safety. [100] Vinterl, J., Bromander, L., Raistrick, P. and Edlerl, H. (2007) FISCADE—A Fault Injection Tool for SCADE Models. 2007 3rd Institution of Engineering and Technology Conference on Automotive Electronics, Warwick, 28-29 June 2007, 1-9. [101] Powell, D., Arlat, J., Chu, H.N., Ingrand, F. and Killijian, M. (2012) Testing the Input Timing Robustness of Real-Time Control Software for Autonomous Systems. 2012 9th European Dependable Computing Conference, Sibiu, 8-11 May 2012, 73-83.
http://dx.doi.org/10.1109/EDCC.2012.16 [102] Kormann, B. and Vogel-Heuser, B. (2011) Automated Test Case Generation Approach for PLC Control Software Exception Handling Using Fault Injection. IECON 2011—37th Annual Conference on IEEE Industrial Electronics Society, Melbourne, 7-10 November 2011, 365-372.
http://dx.doi.org/10.1109/IECON.2011.6119280 [103] Rosch, S., Tikhonov, D., Schutz, D. and Vogel-Heuser, B. (2014) Model-Based Testing of PLC Software: Test of Plants’ Reliability by Using Fault Injection on Component Level. IFAC World Conference, Cape Town, 24-29 August 2014. [104] Sung, A., Choi, B., Wong, W.E. and Debroy, V. (2011) Mutant Generation for Embedded Systems Using Kernel-Based Software and Hardware Fault Simulation. Information and Software Technology, 53, 1153-1164.
http://dx.doi.org/10.1016/j.infsof.2011.03.010 [105] Arlat, J. and Crouzet, Y. (2010) Physical Fault Models and Fault Tolerance. In: Wunderlich, H.-J., Ed., Models in Hardware Testing, Frontiers in Electronic Testing, Vol. 43, Springer Netherlands, Dordrecht, 217-255.
http://dx.doi.org/10.1007/978-90-481-3282-9_8 [106] Schludermann, H., Kirchmair, T. and Vorderwinkler, M. (2000) Soft-Commissioning: Hardware-in-the-Loop-Based Verification of Controller Software. Proceedings of the 2000 Winter Simulation Conference, 1, 893-899.
http://dx.doi.org/10.1109/wsc.2000.899889 [107] Carreira, J., Madeira, H. and Silva, J.G. (1998) Xception: Software Fault Injection and Monitoring in Processor Functional Units. Dependable Computing and Fault Tolerant Systems, 10, 245-266. [108] Yuste, P., Ruiz, J.C., Lemus, L. and Gil, P. (2003) Non-Intrusive Software-Implemented Fault Injection in Embedded Systems. In: de Lemos, R., Weber, T.S. and Camargo Jr., J.B., Eds., Dependable Computing, Springer, Berlin, 23-38.
http://dx.doi.org/10.1007/978-3-540-45214-0_5 [109] (1998) IEEE Std 1219-1998, IEEE Standard for Software Maintenance. The Institute of Electrical and Electronics Engineers, Inc., Los Alamos. [110] Rothermel, G. and Harrold, M. (1996) Analyzing Regression Test Selection Techniques. IEEE Transactions on Software Engineering, 22, 529-551.
http://dx.doi.org/10.1109/32.536955 [111] Yoo, S. and Harman, M. (2010) Regression Testing Minimization, Selection and Prioritization: A Survey. Software Testing, Verification and Reliability, 22, 67-120.
http://dx.doi.org/10.1002/stv.430 [112] Bohner, S.A. and Arnold, R.S. (1996) Software Change Impact Analysis. The Institute of Electrical and Electronic Engineers, Inc., Los Alamos. [113] Farooq, Q., Iqbal, M.Z.Z., Malik, Z.I. and Nadeem, A. (2007) An Approach for Selective State Machine Based Regression Testing. Proceedings of the 3rd International Workshop on Advances in Model-Based Testing (A-MOST’07), ACM Press, New York, 44-52.
http://dx.doi.org/10.1145/1291535.1291540 [114] Briand, L., Labiche, Y. and Soccar, G. (2002) Automating Impact Analysis and Regression Test Selection Based on UML Designs. International Conference on Software Maintenance, Montreal, 3-6 October 2002, 252-261.
http://dx.doi.org/10.1109/icsm.2002.1167775 [115] Le Traon, Y., Jéron, T., Jézéquel, J.-M. and Morel, P. (2000) Efficient Object-Oriented Integration and Regression Testing. IEEE Transactions on Reliability, 49, 12-25.
http://dx.doi.org/10.1109/24.855533 [116] Caliebe, P., Herpel, T. and German, R. (2012) Dependency-Based Test Case Selection and Prioritization in Embedded Systems. 2012 IEEE 5th International Conference on Software Testing, Verification and Validation, Montreal, 17-21 April 2012, 731-735.
http://dx.doi.org/10.1109/ICST.2012.164 [117] Chen, Y., Probert, R.L. and Ural, H. (2009) Regression Test Suite Reduction Based on SDL Models of System Requirements. Journal of Software Maintenance and Evolution: Research and Practice, 21, 379-405.
http://dx.doi.org/10.1002/smr.415 [118] Korel, B., Tahat, L. and Vaysburg, B. (2002) Model Based Regression Test Reduction Using Dependence Analysis. 2002 International Conference on Software Maintenance, Montreal, 3-6 October 2002, 214-223.
http://dx.doi.org/10.1109/icsm.2002.1167768 [119] Ryder, B. and Tip, F. (2001) Change Impact Analysis for Object-Oriented Programs. Proceedings of the 2001 ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering, 46-53.
http://dx.doi.org/10.1145/379605.379661 [120] Leung, H. and White, L. (1990) A Study of Integration Testing and Software Regression at the Integration Level. 1990 International Conference on Software Maintenance, San Diego, 26-29 November 1990, 290-301.
http://dx.doi.org/10.1109/icsm.1990.131377 [121] Ren, X., Shah, F., Tip, F., Ryder, B.G. and Chesley, O. (2004) Chianti: A Tool for Change Impact Analysis of Java Programs Categories and Subject Descriptors. ACM Sigplan Notices, 39, 432-448.
http://dx.doi.org/10.1145/1028976.1029012 [122] Orso, A., Apiwattanapong, T. and Harrold, M.J. (2003) Leveraging Field Data for Impact Analysis and Regression Testing. ACM SIGSOFT Software Engineering Notes, 28, 128-137.
http://dx.doi.org/10.1145/949952.940089 [123] Rothermel, G. and Harrold, M.J. (1997) A Safe, Efficient Regression Test Selection Technique. ACM Transactions on Software Engineering and Methodology, 6, 173-210.
http://dx.doi.org/10.1145/248233.248262 [124] Chen, Y.F., Rosenblum, D. and Vo, K. (1994) TESTTUBE: A System for Selective Regression Testing. Proceedings of 16th International Conference on Software Engineering, (ICSE’94), Sorrento, 16-21 May 1994, 211-220.
http://dx.doi.org/10.1109/ICSE.1994.296780 [125] Ulewicz, S., Schütz, D. and Vogel-Heuser, B. (2014) Software Changes in Factory Automation—Towards Automatic Change Based Regression Testing. IECON 2014, 40th Annual Conference of the IEEE Industrial Electronics Society, Dallas, 29 October-1 November 2014, 2617-2623.
http://dx.doi.org/10.1109/IECON.2014.7048875