Testing

The Testing Research Group is one of the largest groups of its kind in the UK developing innovative approaches to software testing and quality assurance.

Digital hand shake with human
Off

Drawing equally on theory and practice, the aim of the research group is to produce practical software engineering solutions for industry, improving the productivity of developers and testers, and improving the reliability of software systems. 

The group investigates a number of areas, including model-based testing, search-based testing, security testing, reverse engineering, model-driven engineering, multi-agent modelling and XML data processing.

Crossing the boundaries between the recovery and verification of designs, to the generation of economical test-suites that exercise software completely, our research has created innovative tools for industry, including: EvoSuite, IGUANA, JWalk, Broker@Cloud, HOLTestGen (test generation), StateChum, SUMO (reverse engineering) and FLAME (agent-based modelling). Our work is funded by the EU, EPSRC, InnovateUK and industry.


Research themes

Model-based Testing

This formal testing method generates complete functional test suites from a model/specification expressed as an extended finite state machine; and offers guarantees of correctness once testing is complete. The work has been used at Daimler for testing automotive systems designed using Harel Statecharts, applied to testing the gate-logic on low-power ARM chips, used for unit testing Java and also used to test software services for SAP and SingularLogic in the Cloud. Several software testing tools have grown out of this work, including StateTest, JWalk and Broker@Cloud.

Leaders: Prof HieronsDr SimonsDr Bogdanov

Search-based Testing

Search-based testing uses a problem-specific fitness function to guide a search process (such as a genetic algorithm) to automatically generate test cases or select from a given set of test cases. Fitness is judged by instrumenting the tested code to measure what is covered by the tests. The approach has been applied to large C programs and also to object-oriented programs in Java. Several software tools have grown out of this work, including EvoSuite for Java, and IGUANA for C.

Leaders: Prof HieronsDr WalkinshawProf McMinn

Property-based Testing

Property-based testing provides a high-level approach to testing and is a complementary technique to model-based testing. Rather than focusing on individual test cases to encapsulate the behaviour of a system, in property-based testing this behaviour is specified by properties, expressed in a logical form. The system is then tested by checking whether it has the required properties for randomly generated data, which may be inputs to functions, sequences of API calls, or other representations of test cases. This work has resulted in a code coverage tool (Smother) and a mutation testing tool (Mu2) for the Erlang language.

Leaders: Prof DerrickDr BogdanovDr Taylor

Security Testing

Security encompasses information security, software engineering, security engineering, and formal methods. Our research in this area investigates all security aspects of distributed and service-oriented systems. This includes applied security aspects, such as access control or business-process modelling, as well as fundamental aspects, such as novel static and dynamic approaches for ensuring the security of applications. We participate in the development of interactive theorem proving environments for Z (HOL-Z) and UML/OCL (HOL-OCL, which is integrated into a formal MDE tool-chain) and a model-based test-case generator (HOL-TestGen).

Leaders: Professor Clark

Model-Driven Engineering

The aim of this approach is to create systems not by writing program code, but rather by generating it from high-level abstract models that are closer to end-user requirements. What initial models and languages should be chosen? How should models be checked? How should they be folded together to create more detailed system specifications? How should the transformation rules be verified? Our work so far has investigated dependently-typed languages as a means of verifying model transformations, has generated simple information systems from requirements and has also generated platform-specific test suites for SOAP or REST-based software services in the Cloud.

Leaders: Dr SimonsProf Derrick

Reverse Engineering

This novel approach recovers specifications from legacy software systems. The reverse engineering method collects traces of the system's execution and performs grammar inference on the traces, to detect behavioural regularity. From this, finite state models are constructed, which allow further hypotheses about the specification to be generated and tested. The approach has been applied to recover both flat and nested state specifications, and used for the supervised re-modularisation of software systems. Software tools include StateChum (for reverse engineering) and SUMO (for supervised re-modularisation).

Leaders: Dr WalkinshawDr Bogdanov, Dr Hall, Dr Taylor

XML Data Processing

The rise of distributed data processing in the Cloud has led to a resurgence of non-relational key-value and tree-structured data formats like XML. This has opened up new research areas in XML data compression, distribution and storage, with associated issues of data indexing and query-processing. We have developed algorithms for fast searching in distributed XML databases using sparse binary matrix indexing; and for trust-based access control to XML data with dynamic learning. The technology has been applied in Botswana for the distributed mobile phone hosting of compressed XML databases.

Leaders: Dr North

Software Engineering

Why do software development projects frequently fail, and what can be done about it? This strand of our research has investigated both traditional and agile software development methods, to see how they work in practice. We have published internationally-famous critiques of the UML notation and associated development process, and also of the Agile method known as XP (eXtreme Programming). We have collaborated with work psychologists to monitor the behaviour and effectiveness of developer teams in our own software company Genesys Solutions, revealing how the constitution of teams affects how well they work.

Leaders: Dr CowlingDr Simons

Core members

Academic staff

Research staff

Affiliated academics

Publications

Academic articles

Here you can find research publications for the Testing Research Group, listed by academic.  The head link navigates to the official web page for the relevant academic (with highlighted favourite publications).  The remaining links navigate to their DBLP author page, their Google Scholar citations page and optionally a self-maintained publications page.

Academic staff

Prof Rob Hierons DBLP GoogleScholar Web
Dr Kirill E Bogdanov DBLP GoogleScholar Web
Dr Achim D Brucker DBLP GoogleScholar Web
Prof John Derrick DBLP GoogleScholar  
Prof Phil McMinn DBLP GoogleScholar Web
Dr Siobhán D North DBLP GoogleScholar Web
Dr Anthony J H Simons DBLP GoogleScholar Web
Dr Neil Walkinshaw DBLP GoogleScholar Web
Dr Ramsay Taylor DBLP GoogleScholar  

Affiliated academics

Prof Graham Birtwistle DBLP GoogleScholar  
Eur Ing Dr Anthony J Cowling DBLP GoogleScholar  
Dr Dimitris Dranidis DBLP GoogleScholar  
Dr George Eleftherakis DBLP GoogleScholar  
Prof Gordon Fraser DBLP GoogleScholar Web
Prof W Michael L Holcombe DBLP GoogleScholar  
Prof Petros Kefalas DBLP GoogleScholar  
Dr Mariam Kiran DBLP GoogleScholar  
Dr Pietro S Oliveto DBLP GoogleScholar Web
Dr Ioanna Stamatopoulou DBLP GoogleScholar  
Dr Jose Miguel Rojas Siles DBLP GoogleScholar Web

Research staff

Dr Raluca-Elena Lefticaru DBLP GoogleScholar
Dr Krishna Patel DBLP  

Affiliated researchers 

Dr Mike Croucher GoogleScholar    
Dr Paul Richmond DBLP GoogleScholar Web

Visiting academics

Dr Mathew Hall
Research theses

Here you can find recently-published PhD (and MPhil) theses, which have been deposited in the White Rose eTheses Online repository.  Follow links to the abstract, and then to the full thesis (if public) or to a request form (if a time-embargo restriction has been placed on public release).

Recently published theses

Dr Eidah Alzahrani 2021 The impact of minimising data movement on the overall performance of the simulation of complex systems applied to FLAME GPU
Dr Nasser Albunian 2020 An Investigation of Search Behaviour in Search-Based Unit Test Generation
Dr Abdullah Alsharif 2020 Automated Software Testing of Relational Database Schemas
Dr Michael Foster 2020 Reverse Engineering Systems to Identify Flaws and Understand Behaviour
Dr Thomas Weripuo Gyeera 2019 Monitoring and Adaptation of Pooled Cloud Computing Resources
Dr Michael Herzberg 2019 Formal Foundations for Provably Safe Web Components
Dr David Paterson 2019 Improvements to Test Case Prioritisation considering Efficiency and Effectiveness on Real Faults
Dr Michal Soucha 2019 Testing and Active Learning of Resettable Finite-State Machines
Dr Thomas White 2019 Guiding Random Graphical and Natural User Interface Testing Through Domain Knowledge
Dr Michal Soucha 2019 Testing and Active Learning of Resettable Finite-State Machines
Dr Sadeen Alharbi 2018 Automatic Framework to Aid Therapists to Diagnose Children who Stutter
Dr Krenare Pireva 2018 Cloud eLearning - Personalisation of Learning using Resources from the Cloud
Dr Thomas Walsh 2018 Automatic Identification of Presentation Failures in Responsive Web Pages
Dr Ermira Daka 2018 Improving Readability in Automatic Unit Test Generation
Dr Shtwai Alsubai 2018 Child Prime Label Approaches to Evaluate XML Structured Queries
Dr José Carlos Medeiros de Campos 2017 Search-based Unit Test Generation for Evolving Software
Dr Hanaa Al Zadjali 2017 Compressing Labels of Dynamic XML Data using Base-9 Scheme and Fibonacci Encoding
Dr Dimitrios Kourtesis 2017 Policy-Driven Governance in Cloud Service Ecosystems
Dr Maria Ulfah Siregar 2016 Support for Model Checking Z Specifications
Mr Samer Al Khazraji 2016 A Labelling Technique Comparison for Indexing Large XML Database
Dr Sina Shamshiri 2016 Automated Unit Testing of Evolving Software
Mr Konstantinos Rousis 2016 Disciplined Exploitation of Emergent Properties
Dr Abdullah Alsaeedi 2016 Improving Software Model Inference by Combining State Merging and Markov Models
Dr Rustem Dautov 2016 EXCLAIM Framework: a Monitoring and Analysis Framework to Support Self-Governance in Cloud Application Platforms
Dr Fotios Gonidis 2016 A Framework Enabling the Cross-Platform Development of Service-based Cloud Applications
Dr Christopher Wright 2016 Mutation Analysis of Relational Database Schemas
Dr Alaa Almelibari 2015 Labelling Dynamic XML Documents: a Group Based Approach
Dr Ahmad Subahi 2015 A Business User Model-Driven Engineering Method for Developing Information Systems
Dr Othlapile Dinakenyane 2014 SIQXC: Schema Independent Queryable XML Compression for Smartphones
Ms Isidora Petreska 2014 Biologically Inspired Agents: A Framework for Formal Modelling and Simulation of Agents with Elementary Spatial Attributes
Dr Joseph Vella 2014 An Object-Oriented Data and Query Model
Dr Andrea Corbett 2013 Agent-Based Modelling of Transactive Memory Systems and Knowledge Processes in Agile versus Traditional Software Development Teams
Dr Sheeva Afshan 2013 Search-Based Generation of Human Readable Test Data and Its Impact on Human Oracle Costs
Dr Norah Farooqi 2013 Applying Dynamic Trust Based Access Control to Improve XML Databases Security
Dr Mathew Hall 2013 Improving Software Remodularisation
Dr Ognen Paunovski 2013 Exploring Emergent Phenomena: Towards Analysis and Synthesis of Emergent Formations in Complex Systems
Dr Ervin Ramollari 2013 Automated Verification and Testing of Third-Party Web Services

Seminars

The Testing Research Group seminar series is a public-facing forum which gives details of forthcoming seminars by members of the group and by our research visitors. 

This forum may be viewed by the general public, but only authorised members of the Testing Research Group may post new messages.  Messages emailed to the testing-seminar-group@sheffield.ac.uk alias will appear in the newsfeed below. Authorised members can also interact with the newsfeed directly to post new messages or extend existing threads with new content.

Flagship institutes

The University’s four flagship institutes bring together our key strengths to tackle global issues, turning interdisciplinary and translational research into real-world solutions.