[time]
2020-11-26T12:49:26+01:00

[team_name]
NLE

[team_institution]
Naver Labs Europe

[logolink]
https://europe.naverlabs.com/

[system_name]
MAD PDR (Magnetic Assisted Deep PDR)

[website]


[track]
3

[reference_person]
Leonid Antsfeld

[email]
leonid.antsfeld@naverlabs.com

[description]
==== Short description ====
In order to tackle Track 3 of this year IPIN challenge we have used a stack of complementary localization components that were intelligently fused together to provide a final actor’s itinerary. 
The main components are: floor detection, user activity recognition, speed/step length estimation, PDR, WiFi based localization and, finally, a novel component of magnetic field based localization.
Floor detection was done using classic classification methods based on WiFi and barometer data. We have applied spectral analysis on the accelerometer data in order to identify user's activity (walking, standing, going up or down the stairs). Next, we have used peaks detection of accelerometer data in order to detect steps. We extract acceleration features in order to learn the step length / speed in a given sliding window. Together with the orientation sensor it allowed us to build a PDR which was a first order approximation of the itinerary.
The PDR is known for being accurate locally but drifting over time. In order to compensate for this drift, we have used global localization techniques based on WiFi VAE that was presented at IPIN’19 and recently developed LSTM based magnetic field localization algorithm. 
Finally, the itinerary was adjusted using a map that was extracted from the training data. 


===== Longer description ======


\section{Project description}

Our participation in IPIN 2020 challenge is based on extending the localization pipeline we developed for IPIN 2019 challenge with new components.

Our pipeline is a sensor fusion framework for accurate indoor positioning and tracking using the smartphone inertial sensors, WiFi measurements, magnetic field data and landmarks. The main components include floor detection, user activity recognition, speed/step length estimation, PDR, WiFi based localization and a novel component of magnetic field based localization. We briefly describe these components below:

\begin{description}
\item [Floor detection.]
Floor detection was done using classic classification methods based on WiFi and barometer data.
\item[Activity detection.]
 We have applied spectral analysis on the accelerometer data in order to identify user's activity (walking, standing, going up or down the stairs).
\item[PDR.]
We have used peaks detection of accelerometer data in order to detect steps. We extract acceleration features in order to learn the step length / speed in a given sliding window. Together with the orientation sensor it allowed us to build a PDR which was a first order approximation of the itinerary.
  
\item [Deep PDR.]
Applying deep learning to PDR is inspired by using deep learning for user's activity detection~\cite{deng-2016-continuous,lima19human}. We pre-process and reshape sensor data streams as images and use convolutional (CNN) and recurrent (RNN) neural networks to extract underlying hidden correlations between different sensors and modalities to learn a model of user local displacement. 

This allows to cope with sensor noise and replace the manual feature extraction which is a frequent subject to data noise and sophisticated thresholding, including tuning to different pedestrian profiles, depending on gender, age, height etc.

Unlike classification for activity recognition, we predict relative $x$ and $y$ displacements, and deploy the regression loss for learning the relative user displacement model. 

The PDR is known for being accurate locally but drifting over time. We compensate for this drift by using global localization techniques based on WiFi VAE and magnetic field based localization.
 
\item [Landmarks and pseudo labels.]
CNN/RNN models require a large annotated dataset for training, while genuine ground truth annotations are sparse and available for a limited number of landmarks. On the other hand, raw sensor data are massively generated at a high rate. So we annotate sensor data with pseudo labels and generate a large annotated set for training CNN/RNNs. It is based on simpler tasks of user walking and landmark detection and a interpolation of user's behaviour between the landmarks.

\item [Semi-supervised VAE for WiFi.] 
A radiomap is constructed from the WiFi data provided in training and validation data. Recorded data provides a WiFi scan reading every 4 seconds approximately, however without an exact position where this scan was taken. 
Using the provided timestamped landmarks and inertial sensor data, we can infer the approximate position where the WiFi fingerprint was taken and build a radiomap with this information~\cite{chidlovskii19}.

\item [Magnetic filed based localization.]
This new component for indoor localization uses magnetic field data provided by mobile phone magnetometer. We extend the state of the art approaches, in particular landmark-based classification~\cite{amid_article} in order to benefit from the presence of magnetic anomalies in indoor environment created by different ferromagnetic objects.  
First, we capture changes of the Eurth's magnetic field due to indoor magnetic anomalies and transform them in multi-variate times series. 
We use a number of techniques to convert temporal patterns in visual ones. We use methods of Recurrent plots, Gramian Angular Fields and Markov Transition Fields to represent magnetic field time series as image sequences. This permits to deploy convolutional layers to extract different patterns in magnetic data stream. 
We train a deep regression on the user's position and combine convolutional and recurrent layers in the deep network.

\item [Prediction fusion and map projection.]
Relative predictions provided by deep PDR and absolute predictions provided by WiFi and magnetic field data are combined using an Extended Kalman Filter. The output of the filter is then fine-tuned, by projecting it on the paths that were traversed while collecting training and validation sets as well, as additional possible walking paths, 
to make sure that the final result lies within the layout of the building.

\end{description}


[references]
https://dl.acm.org/doi/abs/10.1145/3384419.3430419
https://arxiv.org/abs/2011.10799
https://ieeexplore.ieee.org/abstract/document/8911825
https://ieeexplore.ieee.org/document/9253514