These are the protocols used both for competitors (C) and evaluation committees (EC) during the competition:
Track 1 on Location for AAL (you can download the protocol in pdf here)
1. Welcome and briefing
For each of the following, it is sufficient that one person from the EC carries out the tasks (to allow other EC members to carry out tasks in parallel, related to a previous competitor):
2. Infrastructure Preparation
3. Evaluation of Installation Process
Note 1: If the competitor needs to take a break during installation for any reason: inform EC, who will pause the timer.
Note 2: The installation time includes any measurements, software configuration etc. that the competitor may need to carry out, in addition to the physical installation itself.
Note 3: The recording of device locations (after timing has stopped) is not part of the competition itself and does not influence the score for the competitor. The data is collected for scientific interest, and documentation afterwards.
4. Integration of EvAAL and Competitor’s Software
EvAAL provides a software infrastructure for gathering data generated by the competitor’s software, comparing this with benchmark data on the user’s location, and providing the basis for scores. It is necessary to be sure that the systems communicate properly and have synchronized clocks before the location evaluation can begin.
5. Preparation for benchmark tests
6. Benchmark Testing
Benchmark testing takes place in three phases, and is based on three pre-defined paths through the Living Lab (referred to as P1, P2 and P3 below).
The phases are:1. Accuracy Test: one person.
EC+C: Carry out tests twice, for phase 1.
EC+C: Carry out tests twice, for phase 2.
EC: Ask the competitor to check whether logging of internal data etc. has been successful. If not, fix the problem and repeat the tests for the tests where it was missing.
EC+C: Carry out tests twice, for phase 3.
EC: Collect all log data from the competitor. If some is missing, repeat the affected tests.
EC: Stop video recording.
EC: Provided that all log data has been successfully collected, inform the competitor that it is OK to start dismantling equipment.
C: Start dismantling and packing equipment. (If the competitor has several staff, this may proceed in parallel with the de-briefing described below; otherwise it can be started here, then completed after de-briefing).
EC: Inform the competitor of the score attained for each test, as calculated by the EvAAL infrastructure software. Make it clear that the score may be adjusted afterwards, if a detailed analysis of all information stored in logs indicates that there may be a reason to do so.
Note that the recorded score for each phase will be based on whichever of the two tests provides the most favourable score.
7. De-briefing and wrap-up
The main purpose of this phase is to make some simple conclusions about key aspects of the competitor’s system, both in its current form and by judging its future potential. A “checklist” of questions with simple yes/no/not applicable answers is used for this purpose.
For each of the tasks below related to the checklist of questions, there should be at least 3 members of the EC present, to discuss points where there may be diverging views. The EC member responsible for data management should not be included in the minimum set of 3 members, because the duty of the data manager at this time is to make sure that metadata etc. is collected. Other EC members are free to carry out tasks in preparation for the following competitor.
Track 2 on Activity Recognition for AAL (you can download the protocol in pdf here)
1. Welcome and preparation:
When the competitor arrives the EC welcome warmly and explain the protocol.
One person will be the responsable of checking this document twice so that nothing is missed
2. Installation and configuration/calibration:
The team have 60 minutes (it corresponds to X value in the Technical Annex) to install and calibrate the system. Calibration include any training that actor must do with the system.
When installation starts, no one a part from the competitor's team must be in the place.
One measures the time for the installation time.
Installation ends when the competitor tells it. With the calculus explained in Technical Annex we can obtain the result for Installation complexity evaluation criteria.
Two people measure (metadata) the devices and 1 person makes some pictures (if competitor agrees).
3. Evaluation system set up:
Competitor is invited to launch his system and to check if the server receives correctly the information sent:
One checks that competitor's clock and the EvAAL system clocks are really synchronized (competitor should indicate which ntp server is using, we are using pool.ntp.org)
One checks the integration of the competitor's system with EvAAL system
If some problema occurs a máximum of 20 minutes (it corresponds to Y value in the Technical Annex) is given to fix integration, otherwise the log backup solution will be used. Log backup correspond to a txt file created by the competitor system with the same information required through sockets (milliseconds from 1970, integer of activity) i.e.:
Any problem related to the integration is logged on a note (for feedback to universAAL)
4. Actor performance:
Only the actor and one EC member (responsable to annotate ground truth activities) will be inside the living lab during the competition, the competitor as well as the evaluation committee will be outside.
One is responsable for recording all the video (if the competitor agrees)
The evaluator starts the mp3 file that will guide the actor.
The actor moves and performs activities using the mp3 file. The evaluator uses a client to mark activities and transitions.
Once the mp3 finish, the actor will continue during 1 minute performing the last activity in order to capture the delayed events recognized by the competitor system. If any competitor has a delay greater than 1 minute he must indicate us before the performance begin.
After that time the evaluation software will run in order to obtain the performance criteria and the delay criteria. The software will select the recognition delay that maximizes the performance criteria.
The performance will be repeated again to obtain a second mark in performance and delay criteria and we will select the best performance.
Competitor will be interviewed about Interoperability and User acceptance criteria. To avoid subjective values in user acceptance and interoperability we will use the yes/no questions in annex I and II. However E.C. can ask any other questions.
Interview will be annotated in a scoring sheet.
If any vote must be taken by the EC it is taken AFTER the competitor has left the room
The competitor is invited to answer a feedback questionnaire
The competitor is asked for the intermediate data produced during the competition, one person from the EC is responsible for collecting the data
Competitor is asked to provide the logs of the EvAAL system
All the documents are stapled together, ALL files including photos, logs, etc.
The competitor is asked to remove installation
The competitor is thanked warmly and given a small present.