[evaal contest] Evaluation / Time synchronisation

Thu Sep 22 14:23:33 CEST 2011

Dear Dante and Stefan and other members of Competition,

We have received Dante’s and Stefan’s concerns, identified them, and we are very near (a couple of hours) to provide you with an “official” answer to both of them.

Both concerns are not connected. 

-          For the case of Stefan it is related to synchronization aspects.

-          For the case of Dante it is related to a random pattern of series of two identical lines in the input file.

We wanted to publish everything and applying the same rules for everybody and considering the samples “as is”, “as they arrive to us”. So then, when you check the published algorithms (java project) and samples we verify together that everything is ok.

We would should insist in the fact that we want to be as fair as possible according to the rules and procedures we established. Any change to the way to proceed with one competitor, may affect the others, and should be analyzed very carefully.

@Dante, @Stefan: Thanks for your effort in checking and reporting!

Juan Pablo

De: contest-bounces at evaal.aaloa.org [mailto:contest-bounces at evaal.aaloa.org] En nombre de Dante Tapia
Enviado el: jueves, 22 de septiembre de 2011 12:44
Para: Stefan Knauth
CC: Stefano Chessa; contest at evaal.aaloa.org
Asunto: Re: [evaal contest] Evaluation / Time synchronisation

Dear EvAAL organizers,

We have also found important inconsistencies on the results provided. The most important problems we have found are related to randomly duplicated samples in your input file for each test which lead to truncated test samples on our side.

For this reason, we are recalculating the results and we will provide you the report asap.

Best regards,

_________________________
Dante I. Tapia
Grupo de Investigación BISITE / BISITE Research Group
Universidad de Salamanca (España) / University of Salamanca (Spain)
http://bisite.usal.es

On Thu, Sep 22, 2011 at 11:45 AM, Stefan Knauth <stefan.knauth at hft-stuttgart.de> wrote:

Dear EvAAL competition evaluators,
Dear Competitors

(via Contest mailing list)

Yesterday we (the iLoc team) received the data used for evaluation, together with the evaluation code.

When running the evaluation using the timestamps given in the files provided by the email, obtained scores were more or less zero for all
tests.

The reason is a time mismatch between our system and the evaal system of 50 seconds.
This can be easily seen for example in the AOI data, where the iLoc reported AOIs appear 50 secs later than the expected AOIs.

We request to take into account a time shift of 50 seconds,
with respect to the data received in the mail from Sept 21, i.e. to rerun the evaluation and add 50 seconds (i.e. 50000 msec) to the "Start timestamp" and "Stop timestamp" values.

I don't know how the obtained time mismatch was induced.

The system is installed for 2 years now at the iHomeLab and runs there with a typical delay of less than 0.5 seconds. Also during the setup in Valencia, we tested the system and the AOI detection, and a delay of 50 seconds would obviously have been realized by us.

Best regards,

Stefan Knauth / iLoc Team

-----------------------------------
Prof. Dr. Stefan Knauth
Stuttgart University of Applied Sciences - HFT Stuttgart
Faculty of Geomatics, Computer Science and Mathematics
Schellingstr. 24
D-70174 Stuttgart
Germany

mail  : stefan.knauth at hft-stuttgart.de
phone : +49 711 8926 2966 <tel:%2B49%20711%208926%202966> 
cell  : +49 179 2932 009 <tel:%2B49%20179%202932%20009> 

Am 21.09.11 15:15, schrieb Stefano Chessa:
...
>
>
> I send you the logfiles and the software used to perform the evaluation.
> With these data you can, if you wish, evaluate your own score regarding
> availability and accuracy.
>
>
>
> I apologize for sending this material so late, our initial plan was deliver
> all of this material much earlier than the EvAAL workshop, so that each
> competitor would have had time to check his own score. However many issues
> (included even software bugs) delayed us too much.
>
>
>
> Please note that each competitor is, in this phase, granted access to its
> own data, but we will put all the logs of everybody on the EvAAL website
> soon after the workshop and the prize awarding.
>
>
>
> If you test the program and you find bugs or other issue I ask you to let us
> know as soon as possible.
>
...

_______________________________________________
Contest mailing list
Contest at evaal.aaloa.org
http://evaal.aaloa.org/mailman/listinfo/contest

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://evaal.aaloa.org/pipermail/contest/attachments/20110922/d54eead0/attachment.html>