This deliverable ‘6.1 Evaluation Plan’ starts with a general chapter on the background of evaluation in DORA and its purpose. It shows the relation of the DORA project to Horizon 2020 as the biggest EU Research and Innovation programme of the European Union and to the specific call ‘MG 1.3-2014: Seamless Air Mobility’.
It continues with describing the purpose of the deliverable which is to set the plan for all evaluation activities that will be carried out within the DORA project. It, moreover, describes the approach to the evaluation, presents a working plan for the evaluation activities and sets the frame for the responsibilities of the partners involved. Besides this clearly structured delineation of the evaluation framework, the evaluation plan also aims at providing explanations on methods and indicators to be used, as well as presenting guidelines for the application of specific methods by the partners involved in evaluation.
The following chapter summarises the aims of the project as they were described in the ‘Description of Work’ document. Aims are being understood as the realisation of single components of the DORA service and their final integration. The components being described are: “Design and Implementation of a DORA Service Platform (PLA) and a Long Term Door-to-Door Journey Planner (DJP)”, “Design and Implementation of End-User-Applications for Seamless Mobility Information (APP / SWA / WEB)”, “Implementation of Personal Information Services (FIS / ILR / SRS / TMS)”, “Implementation of an Incident and Information Management for Airports (MIM)”, “Design and Implementation of a System for Detection of Waiting Time in Airports (WTS)” and “Design and Implementation of an Indoor Location and Routing at the Terminal (IP / IR)”.
The following chapter (1.5) identifies and describes all objectives that will be evaluated within this project. The objectives being described are ordered along their corresponding task in Work Package 6. Together with the indications of the following chapter, that describe the frame conditions in the implementation sites in Berlin and Palma de Mallorca, the before mentioned objectives provide the basis for the description of those methods that will be applied to measure if – or to which extend - the objectives were reached.
Chapter 2 is dedicated to the description of the iterative DORA evaluation approach. It describes the evaluation activities and their relations to each other, and presents methods and information on how they can be applied. It contains detailed information on the four evaluation activities that will be carried out in DORA:
The Technical evaluation chapter (2.3.1) demonstrates how the evaluation of the technical features of DORA will be evaluated. The DORA developments are divided into three main categories: Services, Applications and Technology. Each category follows a different development methodology that is being described by giving information on methods and timelines for accomplishing them by orienting on the ‘Evaluation Roadmap’.
The chapter Usability evaluation (2.3.2) describes the main goal of the Usability Evaluation, which is to assess usability criteria in order to identify possible problems with the design of the interfaces at an early stage during the conceptual phase and during its integration phase. It also provides a description on how to realize three usability evaluations at each of the three development steps of the DORA system: Usability Evaluation of the Concept for the Web and App GUI / Usability Evaluation of the DORA Prototype / Usability Evaluation of the Integrated Final Product (DORA system Alpha Version). The end of this chapter consists of an overview on the operationalization of these evaluation activities, including information on methods to be applied, partners who are involved, or times for their conducts.
The chapter Results evaluation (2.3.3) focuses on the description of the evaluation activities related to the assessment of the results of the project. It comprises information related to Impact Evaluation and Evaluation of User Satisfaction. It, moreover, includes information on the application of impact evaluation, shows which indicators are being applied for the assessment of the objectives, describes why a control site approach is planned to be applied for two indicators, and shows which methods (user tests / surveys with questionnaire and interviews) will be applied for the assessment of ‘User Satisfaction’. Methods and remarks concerning the operationalization are summarized in tables.
The chapter Process evaluation (2.3.4) starts with an explanation of the intention of this evaluation activity, which is to understand what influences the project process in an either positive or negative way and how experiences gathered during the early phase of the project have supported the further process of the project. It looks at the strengths and weaknesses (drivers and barriers), thereby focusing on HOW an outcome or product is being realized rather than on how the impact from the project performs. This chapter also summarizes information concerning the operationalization of the three phases of this activity at the end of the chapter.
A major ambition of the DORA project, is to develop a system that is easily transferable to other cities and their airports in Europe. Thus, the following chapter 3, provides information on the relevance of the results from all evaluation activities that are being carried out within DORA for other cities that are considering to implement the system in their cities.
Being aware of the fact that every development or implementation process is constantly subject of changes and also risks, chapter 4 describes ‘Issues that can jeopardise the evaluation’. Since these risks are mainly related to the technical development of the components or to the integration of them (e.g.: incompatibilities of systems or a lack of information that is necessary for the development of the system), this chapter tries to foresee a range of possible risks in order to prepare for the case of their occurrence. Thus, also mechanisms to avoid or minimise the risks are being described.The Evaluation Plan ends with chapter 5 “Operationalization / Roles and Responsibilities”. It provides a quick overview on the DORA objectives and the related Work Packages, shows the involvement of the partners in the tasks of Works Package 6 and the timeline for the deliverables of the single tasks as described in the DOW. It presents a table with an overview on all evaluation tasks and sub-tasks, the related methods for assessment and time frames for their accomplishment.