[evaal contest] [localization] status of integration and timestamp computation

[Fet, Ngewi Awa]

Hello Dario,
I understood that the Socket integration server will be running on the EvAAL hosts 
and we will be provided with a hostname to bind to and send our location coordinates. 
Your mail seems to mean that we will be running the server on our machines and then reporting the location
to the instance on our machines. Am I correctly interpreting this?
If so, will the log files from my machine then be copied for evaluation. Or how are the EvAAL systems going to evaluate our system?
I would appreciate some clarification please.

What about the NTP server? I was under the assumption that we would be provided also
a host with which to synchronize. Or are we just supposed to connect with a public NTP server
like the NIST time servers and synchronize? Please advice. thanks.

Regards,
Ngewi Fet

On 01.07.2012, at 17:34, <dsalvi at lst.tfo.upm.es>
 wrote:

> Dear Julia, dear all,
> 
>> With regards to the status of integration I have to say a couple of
>> things: in general our system successfully communicates with the
>> provided socket server and receives events from the device simulator
>> as well. However, I have observed some weird situations when the the
>> device simulator client cannot connect to the server. More
>> specifically, these situations occur only when I have to restart the
>> device client after my own client has already been connected to the
>> server, and the server has generated a thread for my client. At this
>> point, if I restart the device simulator, the server fails to register
>> this new client (which is confirmed by the server log - there is no
>> indication of a new client connection). I am wondering whether other
>> competitors have encountered a similar problem.
> 
> The device simulator is not to be used during the benchmark, this piece
> of software was exclusively designed so that competitors are able to perform
> tests generating their own context events. Without going in to too much
> detail context events will be generated in two nodes, one for all the
> switches and another for the bicycle.
> The reported issue about the device simulator is known and explained
> through the way it closes combined with the configuration of the running
> environment: When the device simulator is closed using the x on the GUI
> window, the node does not unregister, therefore the other nodes ignore
> the new register requests from the node. The correct way to close the
> device manager is to type "stop 0" in the console, but this shouldn't be
> an issue during the competition since we will not use the device
> simulator. Nevertheless we will report this issue to the development
> team of the universAAL platform.
> 
> 
>> I understand that during the competition the device simulator will not
>> be used as it is, that is, domotic bus events generation and the
>> socket server might be incorporated in one processing unit and
>> therefore there won't be a problem of reconnecting clients. However,
>> if this is not the case, and every physical device connects to the
>> server as a separate client, I would like to be reassured that our
>> system won't be missing any contextual events due to the issue with
>> clients reconnections.
> 
> The context event generation nodes will not be restarted through out the
> entire process of evaluation per competitor; they will be on when the
> competitors arrive, before their installation, and should not be
> restarted until the competitor uninstalls their system (except for
> unexpected behaviour, then the competitors will be informed). Through
> out all of our testing the context events are reaching the competitors
> node, although there is some expected delay, that's why context events
> are timestamped. This, once again, stresses the need of clock
> synchronization.
> 
> Before starting the benchmarks, in any case, the connection among all
> these nodes will be tested.
> I recommend therefore that you provide some output to the user when
> context events are recevied at your system (a print on the screen or a log
> file) so that we can assure that you are receving data before starting.
> 
> 
>> Another question relevant to the integration: will we be given the
>> unique IDs of the domotic bus devices before the competition? I am
>> asking because in order to make the integration of our system with the
>> domotic bus more or less transparent, we need to register these
>> additional devices in our system in compliance with the specifications
>> used for our own devices. At the very least, we need to make sure that
>> all the ids are unique so if we don't know the ids of your devices in
>> advance, we might have collisions. If you choose not to disclose the
>> ids of your devices since you have already provided the coordinates,
>> it's fine with us too because we can assign our own ids and upon
>> receiving events from the domotic bus, we'll simply infer those ids
>> from the provided coordinates.
> 
> 
> Context events' IDs are unique, the integration package also gives the
> relevant information about the devices (coordinates, state and timestamp).
> 
> These are the official IDs of the domotic bus appliances:
> 
>  * bicycle
>  * switch1
>  * switch2
>  * switch3
>  * switch4
>  * switch5
>  * switch6
>  * switch7
>  * switch8
>  * switch9
> 
> 
> and, yes, you can just trust the coordinates you receive together with the
> event and ignore the IDs.
> 
> 
> 
>> After reading one of the recent digests, I am not sure I understand
>> what timestamp is meant here:
>> 
>> 
>>    the time stamp is really used in the metrics is the one that is
>>    created
>>    where the integration program is running.
>>    This timestamp is automatically computed locally using the computer's
>>    clock. For this reason, the only machine that must be synchronised
>>    with
>>    the NTP client is the one runnning the integration program (the socket
>>    server or the itnegrated java package).
>> 
>> 
>> I believe that this information applies to java-based systems only,
>> otherwise it's rather confusing. We are using a second option for
>> integration - communication over sockets. As far as I understand,
>> competitors do NOT run the socket server in their systems during the
>> competition - our system acts as a telnet client which publishes
>> localization tuples to/reads contextual events from the server
>> provided by the organizers at the given ip address. I did not read any
>> details of java integration but the above explanation sounds like the
>> java package calculates its own timestamps when sending a tuple to the
>> socket server. This is NOT the case for our system because the
>> timestamps reported by our system will be delayed compared to the time
>> when they are actually published. Our computer which acts as a telnet
>> client and publishes the tuples will be ntp synchronized prior to the
>> beginning of the tests and the timestamps will be reported in epoch
>> time with milliseconds so that there is no issue of conversion from
>> relative time. Please let me know if my understanding is correct.
> 
> The software that we have provided you sends two timestamps: the one you
> explicitly declare and the one that is calcualted when the sample is
> created, using the clock of the computer.
> This is true for both Java and socket-based integration packages.
> 
> The socket-based integration package is meant to run on YOUR machine. This
> means that you will have to run the executable and set the socket IP
> address to localhost.
> This way we avoid network delays when computing the timestamps.
> It is also possible to run the server on a different machine (e.g. on our
> server), but this should be considered as a backup solution in case of
> problems.
> 
> 
> Now in the evaluations we will use the timestamp that is automatically
> created on your computer. If you have any known delay unfortunately you
> will be penalized. The reason is because we expect these localization
> systems to be almost real-time in an AAL scenario: i.e. the more the
> delay, the worse the final scoring will be (even if you had a
> super-accurate system).
> 
> Last year we had cases where competitors used to buffer their measurements
> and then send all of them in a sort of "burst". This should be avoided, as
> timestamps will be incorrect.
> Please try to make your software as "reactive" as possible.
> 
> 
>> One more thing I would like to know is how the "half a second"
>> parameter is exactly measured. In the case of our system where the
>> delayed timestamps are reported, there are two options: 1) to keep the
>> log of when the tuples are published to the server socket and measure
>> the time difference between each two consecutive published tuples;
> 
> Everything is generated at your machine (running our socket server) and at
> our machines is logged with all timestamps: the one declared by you, the
> one when the sample has been created, and even the one when we receive the
> packet at our machine.
> A part from these complete logs, we also compute the differences between
> these times in separate log so that any delay is explicitly visible.
> Nonetheless, as I told you, the timestamp of the creation of the sample is
> the one that really counts.
> 
>> 2) measure the difference between two consecutive timestamps taken
>> from the actual tuple. Depending on which calculation scheme is used,
>> we need to make appropriate arrangements in the code. Also, how strict
>> is this requirement, is it expected to be 500ms sharp or some
>> deviations within reasonable number of milliseconds are allowed?
> 
> deviations are allowed, we don't expect perfect synchronization. In
> principle you are allowed to send more samples then expected, but in this
> case the extra samples will be ignored, or less samples, but in this case
> you will get lower scores. Just do your best at producing a sample every
> 500 milliseconds and everything will be fine.
> 
> 
>> Thank you,
> 
> you are welcome, and please if you have any other doubt contact us !
> 
> 
> best regards,
> 
> 
> Dario
> 
> 
> 
> 
> _______________________________________________
> Contest mailing list
> Contest at evaal.aaloa.org
> http://evaal.aaloa.org/mailman/listinfo/contest