[evaal contest] Status report about Concerns of iLoc and nCore

[Stefan Knauth]

Dear Juan Pablo and Angel,
(via mailing list)

Thank you for the fast analysis and clarification,

 > -          Our suggestion is that we take the initial timestamp that
 > provides the Maximum score for each competitor, and take this score 
as the
 > final one to be computed.

The iLoc team agrees to that procedure,

best regards,
looking forward seeing you at lecce,

Stefan Knauth

Am 23.09.11 11:10, schrieb Juan Pablo Lázaro Ramos:
> Dear Competitor and other people involved in the competition,
>
>
>
> Regarding the two concerns expressed by Dante and Stefan the situation is
> the following:
>
>
>
> Stefan, about starting and stopping timestamp (so, synchronization):
>
> -          Stefan detected for his system that a delay of 50 seconds in the
> timestamp improves a lot the total calculation of accuracy.
>
> -          We could verify, without changing the code of the evaluation
> application that this is true.
>
> -          We have two ways to calculate the first sample, so the
> TIMESTAMP-START of each competitor. And both ways should provide the same
> value (you can check at the excel file all this calculations):
>
> o   The first is: Timestamp of evAAL PC when the experiment finishes –
> (duration of the experiment) = Timestamp of evAAL PC for the START +
> (difference of Times between evAAL PC and Competitor PC) = TimeStamp START
> of the Competitor’s clock ->  TIMESTAMP-START.
>
> o   The second: We go to the video server and collect the timestamp of the
> moment when Francesco Portotì starts (human perception), then we apply a
> fixed offset between video server and evAAL PC = TimeStamp of evAAL PC for
> the START. Then we follow the last two steps of the previous method, and we
> got TIMESTAMP-START 2.
>
> The two way calculation, TIMESTAMP-START and TIMESTAMP-START2, provide the
> same number aprox., but no difference of 50 seconds, just 1 or half a second
> (because of human perception in the second method).
>
> -          Therefore, for us it is mystery why 50 delay is introduced
> considering the information that we have.
>
> -          Then we started a process to try to find out what is happening
> and we did the following actions:
>
> o   Let’s assume that this 50 sec delay affects every competitor (This is
> not something that we can demonstrate but we made that assumption).
>
> o   Based on that assumption apply 50 seconds delay in all timestamps for
> all competitors.
>
> o   The results were: Dramatic improvement for 3 out of 4 competitors.
>
> o   The next question is then: why 50s and not 35, 37, 40? And why this
> delay should be the same for all? And why for all tests?
>
> -          Then we continued exploring:
>
> o   We decided to make some tests: take competitors and calculate the scores
> with 30,31,32… 68, 69, 70 seconds of delay (+/- 20 considering 50sec as a
> reference) and see if for all of them there is a maximum at 50s.
>
> §   If this is true, somehow we can conclude that there is an offset of 50s
> for all of them.
>
> §  If this is not true and the different competitors experience a maximum in
> different times, to  be honest, the Steering Committee should meet and reach
> a consensus on how to proceed.
>
> o   The experiments said it is not true, but SIMILAR, which means that given
> one competitor we can find a maximum value for First Experiment in Path 1
> and find a maximum value for Second Experiment in Path 1, but the difference
> is around 1-2 seconds if not in the same. But it changes from experiment to
> experiment and from competitor to competitor. However, most of them are in
> the surroundings of 50s (except for path 3 which is the surroundings of
> 40sec).
>
> -          Our suggestion is that we take the initial timestamp that
> provides the Maximum score for each competitor, and take this score as the
> final one to be computed.
>
>
>
> Dante about random repetition of lines at the input file:
>
> -          In order to avoid any problem we are going to create a simple
> program that removes repeated lines, in order to get a new Samples file for
> all the competitors, and do the calculations using these new files.
>
> -          We’ve made a test and some changes were observed though, as
> expected, the improvement is not sensible (a change of +/- 0,5 in scores
> with 10 as a maximum). However, 0,5 points can be ciritical, and we decided
> to eliminate any doubt and simply remove repeated lines.
>
>
>
> Please, consider the samples and excel file sent by Stefano Chessa this week
> as an old version. We’ll try to provide an updated version considering the
> actions described by the end of business day today.
>
> We hope this satisfies everybody, because competitors will have the
> opportunity to bright as they deserve.  J
>
>
>
> Juan Pablo and Angel on behalf of evAAL Steeting Committee.
>
>
>
>
>
>
>
> _______________________________________________
> Contest mailing list
> Contest at evaal.aaloa.org
> http://evaal.aaloa.org/mailman/listinfo/contest