The EvAAL Framework

The EvAAL Framework was conceived with the purpose of Evaluating AAL systems through competitive benchmarking in 2011. The idea was to gather together working systems, both prototypal and mature, and independently compare their performance in one or several specific areas, with the long- term objective of creating a set of evaluation benchmarks for indoor pervasive systems.

homepic

The core EvAAL framwework criteria are:
  1. Natural movement of an actor: The agent testing a localization system walks with a regular pace along a predefined path. The actor can rest in a few points and walk again until the end of the path.
  2. Realistic environment: The path the actor walks is defined in a realistic setting.
  3. Realistic measurement resolution: The minimum time and space error considered are relative to people’s movement. The space resolution for a person is defined by the diameter of the body projection on the ground, which is set to 50 cm. The time resolution is defined by the time a person takes to walk a distance equal to the space resolution. In an indoor environment, considering a maximum speed of 1 m/s, the time resolution is 0.5 s.
  4. Third quartile of point Euclidean error: The accuracy score is based on the third quartile of the error, which is defined as the 2D Euclidean distance between the measurement points and the estimated points.
The extended criteria additionally adopted by the first EvAAL competitions are the following:
  1. Secret path: The final path is disclosed immediately before the test starts, and only to the competitor whose system is under test. This prevents competitors from designing systems exploiting specific features of the path.
  2. Independent actor: The actor is an agent not trained to use the localization system.
  3. Independent logging system: The competitor system estimates the position at a rate of twice per second, and sends the estimates to a logging application provided by the EvAAL committee. This prevents any malicious actions from the competitors. The source code of the logging system is publicly available.
  4. Identical path and timing: The actor walks along the same identical path with the same identical timing for all competitors, within time and space errors within the above-defined resolutions.

EvAAL competitions were organized yearly during the lifespan of the universAAL project, until 2013. In 2014, the International Conference on Indoor Positioning and Indoor Navigation (IPIN Conference) decided to organize an indoor competition on its own, building on EvAAL's experience. Since that, the IPIN conference has hosted a competition based on the EvAAL framework, which in 2015 was called EvAAL-ETRI competition. In 2016, the IPIN Conference formally adopted the "IPIN Competition" name and the EvAAL framework.

This year, the IPIN Competition takes place in the facilities of the Consiglio Nazionale delle Ricerche (Pisa, Italy), where the participants could compete in 4 challenging positionig tracks (2 on-site and 2 off-site). Moreover, an additional off-site track is offered where competitors can evaluate their xDR methods in two challenging scenarios, a restaurant and a factory. The five tracks will be assessed using the EvAAL framework.

 
Joomla SEO powered by JoomSEF