[evaal contest] Status report about Concerns of iLoc and nCore

[Juan Pablo Lázaro Ramos]

Dear Competitor and other people involved in the competition,

 

Regarding the two concerns expressed by Dante and Stefan the situation is
the following:

 

Stefan, about starting and stopping timestamp (so, synchronization):

-          Stefan detected for his system that a delay of 50 seconds in the
timestamp improves a lot the total calculation of accuracy.

-          We could verify, without changing the code of the evaluation
application that this is true.

-          We have two ways to calculate the first sample, so the
TIMESTAMP-START of each competitor. And both ways should provide the same
value (you can check at the excel file all this calculations):

o   The first is: Timestamp of evAAL PC when the experiment finishes –
(duration of the experiment) = Timestamp of evAAL PC for the START +
(difference of Times between evAAL PC and Competitor PC) = TimeStamp START
of the Competitor’s clock -> TIMESTAMP-START.

o   The second: We go to the video server and collect the timestamp of the
moment when Francesco Portotì starts (human perception), then we apply a
fixed offset between video server and evAAL PC = TimeStamp of evAAL PC for
the START. Then we follow the last two steps of the previous method, and we
got TIMESTAMP-START 2. 

The two way calculation, TIMESTAMP-START and TIMESTAMP-START2, provide the
same number aprox., but no difference of 50 seconds, just 1 or half a second
(because of human perception in the second method).

-          Therefore, for us it is mystery why 50 delay is introduced
considering the information that we have.

-          Then we started a process to try to find out what is happening
and we did the following actions:

o   Let’s assume that this 50 sec delay affects every competitor (This is
not something that we can demonstrate but we made that assumption).

o   Based on that assumption apply 50 seconds delay in all timestamps for
all competitors.

o   The results were: Dramatic improvement for 3 out of 4 competitors.

o   The next question is then: why 50s and not 35, 37, 40? And why this
delay should be the same for all? And why for all tests? 

-          Then we continued exploring:

o   We decided to make some tests: take competitors and calculate the scores
with 30,31,32
 68, 69, 70 seconds of delay (+/- 20 considering 50sec as a
reference) and see if for all of them there is a maximum at 50s.

§   If this is true, somehow we can conclude that there is an offset of 50s
for all of them.

§  If this is not true and the different competitors experience a maximum in
different times, to  be honest, the Steering Committee should meet and reach
a consensus on how to proceed.

o   The experiments said it is not true, but SIMILAR, which means that given
one competitor we can find a maximum value for First Experiment in Path 1
and find a maximum value for Second Experiment in Path 1, but the difference
is around 1-2 seconds if not in the same. But it changes from experiment to
experiment and from competitor to competitor. However, most of them are in
the surroundings of 50s (except for path 3 which is the surroundings of
40sec).

-          Our suggestion is that we take the initial timestamp that
provides the Maximum score for each competitor, and take this score as the
final one to be computed.

 

Dante about random repetition of lines at the input file:

-          In order to avoid any problem we are going to create a simple
program that removes repeated lines, in order to get a new Samples file for
all the competitors, and do the calculations using these new files. 

-          We’ve made a test and some changes were observed though, as
expected, the improvement is not sensible (a change of +/- 0,5 in scores
with 10 as a maximum). However, 0,5 points can be ciritical, and we decided
to eliminate any doubt and simply remove repeated lines.

 

Please, consider the samples and excel file sent by Stefano Chessa this week
as an old version. We’ll try to provide an updated version considering the
actions described by the end of business day today.

We hope this satisfies everybody, because competitors will have the
opportunity to bright as they deserve.  J

 

Juan Pablo and Angel on behalf of evAAL Steeting Committee.

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://evaal.aaloa.org/pipermail/contest/attachments/20110923/d5d23134/attachment.html>