[evaal contest] IMPORTANT: A second clarification about timings

[Juan Pablo Lazaro Ramos]

Dear competitors,

As you may know in the technical annex we say that "each localization system should
produce localization data with a frequency of 1 new item of data every
half a second (this will be also used to evaluate availability)".

In a previous email sent yesterday the Committee informed about how to manage samples that arrive following a different pattern than 1 each half a second, for instance arriving in bursts. In that email, we mentioned the proposal to consider only the first sample of a burst and not consider any new sample before 500ms, both for accuracy and availability.

After some discussions with one of the competitors we decided that instead of only taking the first sample of a series of samples that are very close one to each other and invalid the rest, we will consider all the samples for the calculation of the accuracy but only the first one for the calculation of the availability.

Following the example in the previous email below, accuracy will be based on every sample received however you continue getting 60% of the score on availability. This rule promotes those systems that are able to produce correct data on time, over those that produce correct data with delay (during the time is delayed there is no data available).

This rule is not in contradiction with the previous rules and we hope it helps to have a fair environment where competitors could take as much score as possible.

Regards,
Juan Pablo Lázaro on behalf of Francesco Potorti (currently starting his trip to Valencia).





-----Mensaje original-----
De: contest-bounces at evaal.aaloa.org [mailto:contest-bounces at evaal.aaloa.org] En nombre de Francesco Potortì
Enviado el: domingo, 24 de julio de 2011 2:21
Para: contest at evaal.aaloa.org
Asunto: [evaal contest] a clarification about timings

Dear competitors,

in the technical annex we say that "each localization system should
produce localization data with a frequency of 1 new item of data every
half a second (this will be also used to evaluate availability)".

This provision is intended to measure the capability of your
localization system to provid up-to-date info, that is, to work in real
time.  The timings considered will be those when your system calls the
Activator.locate method.

If your system internally generates readings at the correct timings, but
is late in calling the Activator.locate method, then the availability
score may be lowered, as some samples will be discarded.  For example,
if your system generates samples # 0, 1, 2, 3, 4 at times (ms) 0, 500,
1000, 1500, 2000 and calls the Activator.locate method at times 0, 500,
1980, 1990, 2000, then samples # 2, 3 will be considered as missing, and
the two last readings will be discarded (considered invalid).  Accuracy
will be computed based on the valid samples only.  So you may get
a high score on accuracy, but you will get 60% score on availability.

If your system internally generates reading once per second, but calls
the Activator.locate method at the correct timings by providing the same
sample twice, you will get 100% score on availability, and a lower score
on accuracy (because one sample in two is equal to the previous one).

-- 
Francesco Potortì (ricercatore)        Voice:  +39.050.315.3058 (op.2111)
ISTI - Area della ricerca CNR          Mobile: +39.348.8283.107
via G. Moruzzi 1, I-56124 Pisa         Fax:    +39.050.315.2040	 
(entrance 20, 1st floor, room C71)     Web:    http://fly.isti.cnr.it

_______________________________________________
Contest mailing list
Contest at evaal.aaloa.org
http://evaal.aaloa.org/mailman/listinfo/contest