[evaal contest] [localization] status of integration and timestamp computation

[dsalvi at lst.tfo.upm.es]

Dear Julia, dear all,

> With regards to the status of integration I have to say a couple of
> things: in general our system successfully communicates with the
> provided socket server and receives events from the device simulator
> as well. However, I have observed some weird situations when the the
> device simulator client cannot connect to the server. More
> specifically, these situations occur only when I have to restart the
> device client after my own client has already been connected to the
> server, and the server has generated a thread for my client. At this
> point, if I restart the device simulator, the server fails to register
> this new client (which is confirmed by the server log - there is no
> indication of a new client connection). I am wondering whether other
> competitors have encountered a similar problem.

The device simulator is not to be used during the benchmark, this piece
of software was exclusively designed so that competitors are able to perform
tests generating their own context events. Without going in to too much
detail context events will be generated in two nodes, one for all the
switches and another for the bicycle.
The reported issue about the device simulator is known and explained
through the way it closes combined with the configuration of the running
environment: When the device simulator is closed using the x on the GUI
window, the node does not unregister, therefore the other nodes ignore
the new register requests from the node. The correct way to close the
device manager is to type "stop 0" in the console, but this shouldn't be
an issue during the competition since we will not use the device
simulator. Nevertheless we will report this issue to the development
team of the universAAL platform.


> I understand that during the competition the device simulator will not
> be used as it is, that is, domotic bus events generation and the
> socket server might be incorporated in one processing unit and
> therefore there won't be a problem of reconnecting clients. However,
> if this is not the case, and every physical device connects to the
> server as a separate client, I would like to be reassured that our
> system won't be missing any contextual events due to the issue with
> clients reconnections.

The context event generation nodes will not be restarted through out the
entire process of evaluation per competitor; they will be on when the
competitors arrive, before their installation, and should not be
restarted until the competitor uninstalls their system (except for
unexpected behaviour, then the competitors will be informed). Through
out all of our testing the context events are reaching the competitors
node, although there is some expected delay, that's why context events
are timestamped. This, once again, stresses the need of clock
synchronization.

Before starting the benchmarks, in any case, the connection among all
these nodes will be tested.
I recommend therefore that you provide some output to the user when
context events are recevied at your system (a print on the screen or a log
file) so that we can assure that you are receving data before starting.


> Another question relevant to the integration: will we be given the
> unique IDs of the domotic bus devices before the competition? I am
> asking because in order to make the integration of our system with the
> domotic bus more or less transparent, we need to register these
> additional devices in our system in compliance with the specifications
> used for our own devices. At the very least, we need to make sure that
> all the ids are unique so if we don't know the ids of your devices in
> advance, we might have collisions. If you choose not to disclose the
> ids of your devices since you have already provided the coordinates,
> it's fine with us too because we can assign our own ids and upon
> receiving events from the domotic bus, we'll simply infer those ids
> from the provided coordinates.


Context events' IDs are unique, the integration package also gives the
relevant information about the devices (coordinates, state and timestamp).

These are the official IDs of the domotic bus appliances:

  * bicycle
  * switch1
  * switch2
  * switch3
  * switch4
  * switch5
  * switch6
  * switch7
  * switch8
  * switch9


and, yes, you can just trust the coordinates you receive together with the
event and ignore the IDs.



> After reading one of the recent digests, I am not sure I understand
> what timestamp is meant here:
>
>
>     the time stamp is really used in the metrics is the one that is
>     created
>     where the integration program is running.
>     This timestamp is automatically computed locally using the computer's
>     clock. For this reason, the only machine that must be synchronised
>     with
>     the NTP client is the one runnning the integration program (the socket
>     server or the itnegrated java package).
>
>
> I believe that this information applies to java-based systems only,
> otherwise it's rather confusing. We are using a second option for
> integration - communication over sockets. As far as I understand,
> competitors do NOT run the socket server in their systems during the
> competition - our system acts as a telnet client which publishes
> localization tuples to/reads contextual events from the server
> provided by the organizers at the given ip address. I did not read any
> details of java integration but the above explanation sounds like the
> java package calculates its own timestamps when sending a tuple to the
> socket server. This is NOT the case for our system because the
> timestamps reported by our system will be delayed compared to the time
> when they are actually published. Our computer which acts as a telnet
> client and publishes the tuples will be ntp synchronized prior to the
> beginning of the tests and the timestamps will be reported in epoch
> time with milliseconds so that there is no issue of conversion from
> relative time. Please let me know if my understanding is correct.

The software that we have provided you sends two timestamps: the one you
explicitly declare and the one that is calcualted when the sample is
created, using the clock of the computer.
This is true for both Java and socket-based integration packages.

The socket-based integration package is meant to run on YOUR machine. This
means that you will have to run the executable and set the socket IP
address to localhost.
This way we avoid network delays when computing the timestamps.
It is also possible to run the server on a different machine (e.g. on our
server), but this should be considered as a backup solution in case of
problems.


Now in the evaluations we will use the timestamp that is automatically
created on your computer. If you have any known delay unfortunately you
will be penalized. The reason is because we expect these localization
systems to be almost real-time in an AAL scenario: i.e. the more the
delay, the worse the final scoring will be (even if you had a
super-accurate system).

Last year we had cases where competitors used to buffer their measurements
and then send all of them in a sort of "burst". This should be avoided, as
timestamps will be incorrect.
Please try to make your software as "reactive" as possible.


> One more thing I would like to know is how the "half a second"
> parameter is exactly measured. In the case of our system where the
> delayed timestamps are reported, there are two options: 1) to keep the
> log of when the tuples are published to the server socket and measure
> the time difference between each two consecutive published tuples;

Everything is generated at your machine (running our socket server) and at
our machines is logged with all timestamps: the one declared by you, the
one when the sample has been created, and even the one when we receive the
packet at our machine.
A part from these complete logs, we also compute the differences between
these times in separate log so that any delay is explicitly visible.
Nonetheless, as I told you, the timestamp of the creation of the sample is
the one that really counts.

> 2) measure the difference between two consecutive timestamps taken
> from the actual tuple. Depending on which calculation scheme is used,
> we need to make appropriate arrangements in the code. Also, how strict
> is this requirement, is it expected to be 500ms sharp or some
> deviations within reasonable number of milliseconds are allowed?

deviations are allowed, we don't expect perfect synchronization. In
principle you are allowed to send more samples then expected, but in this
case the extra samples will be ignored, or less samples, but in this case
you will get lower scores. Just do your best at producing a sample every
500 milliseconds and everything will be fine.


> Thank you,

you are welcome, and please if you have any other doubt contact us !


best regards,


Dario