The EvAAL Framework

The EvAAL Framework

The EvAAL Framework was conceived with the purpose of Evaluating AAL systems through competitive benchmarking. The idea is to gather together working systems, both prototypal and mature, and independently compare their performance in one or several specific areas, with the long- term objective of creating a set of evaluation benchmarks for indoor pervasive systems.

IPIN Competition 2024 is on our new site

homepic

EvAAL Framework's core criteria are:

Natural movement of an actor: The agent testing a system moves spontaneously with a predefined script, including walking, resting or other actions.
Realistic environment: The scene is a realistic setting.
Realistic measurement resolution: The minimum time and space error considered are relative to the actor’s movement. The space resolution for a person is defined by the diameter of the body projection on the ground, which is set to 50 cm. The time resolution is defined by the time a person takes to walk a distance equal to the space resolution. In an indoor environment, considering a maximum speed of 1 m/s, the time resolution is 0.5 s.
Third quartile of point Euclidean error: The accuracy score is based on the third quartile of the error, which is defined as the 2D Euclidean distance between the measurement points and the estimated points for single-floor settings.

EvAAl Framework's extended criteria additionally adopted by the first EvAAL competitions are the following:

Secret path: The final path is disclosed immediately before the test starts, and only to the competitor whose system is under test. This prevents competitors from designing systems exploiting specific features of the path.
Independent actor: The actor is an agent not trained to use the localization system.
Independent logging system: The competitor system estimates the position twice per second, and sends the estimates to a logging application provided by the testing committee. This prevents any malicious actions from the competitors. The logging system is free software.
Identical path and timing: The actor walks along the same identical path with the same identical timing for all competitors, within time and space errors within the above-defined resolutions.

EvAAL competitions were organized yearly during the lifespan of the universAAL FP7 project, from 2011 to 2013. In 2014, the International Conference on Indoor Positioning and Indoor Navigation (IPIN) organised the first IPIN competition, building on EvAAL's experience. Since then, the IPIN conference has hosted a competition based on the EvAAL framework, with both on-site and off-site Tracks.