#+OPTIONS: ':nil *:t -:t ::t <:t H:3 \n:nil ^:nil arch:headline
#+OPTIONS: author:t broken-links:nil c:nil creator:nil
#+OPTIONS: d:(not "LOGBOOK") date:t e:t email:t f:t inline:t num:t
#+OPTIONS: p:nil pri:nil prop:nil stat:t tags:t tasks:t tex:t
#+OPTIONS: timestamp:t title:t toc:t todo:t |:t
#+OPTIONS: html-postamble:nil
#+STYLE: <style>table.center {margin-left:auto; margin-right:auto;}</style> 
#+TITLE: IPIN2022 Competition Track2 Scoring Summary
#+DATE: <2022-09-06 Wed>
#+AUTHOR: etri
#+EMAIL: 
#+LANGUAGE: en
#+SELECT_TAGS: export
#+EXCLUDE_TAGS: noexport

* IPIN2022 Competition Track2

** Teams

    - Team1: *SZUSCRI*, Shenzhen University, Smart City Research Institute
    - Team2: *CamLoc*, Beijing University of Posts and Telecommunications

** Scoring trials

*** Scoring trials - ground truth
    
    - Trial1 file:trials_set3_cut750_gt/trial_21_0.csv
    - Trial2 file:trials_set3_cut750_gt/trial_41_0.csv

#+attr_html: :class="center"
| Scoring | Length | Number of |        Number of |
| trial   |        |    images | reference points |
|---------+--------+-----------+------------------|
| Trial1  | 170m   |       735 |               80 |
| Trial2  | 136m   |       750 |               64 |


    
*** Scoring trials - log files

    - Trial1
      - Team1 file:trials_logs/team1/evlog_user1_2022-09-02_09-07-30.145158_trial_21_0_2022-09-02T090730.536.csv
      - Team2 file:trials_logs/team2/evlog_user2_2022-09-02_10-30-08.647129_trial_21_0_2022-09-02T103009.466.csv

    - Trial2
      - Team1 file:trials_logs/team1/evlog_user1_2022-09-02_18-04-49.398353_trial_41_0_2022-09-02T180449.798.csv
      - Team2 file:trials_logs/team2/evlog_user2_2022-09-02_16-02-16.436026_trial_41_0_2022-09-02T160217.442.csv


** Evaluation scripts

    - Evaluation script for Team1 file:track2_evaluation_team1.py
    - Sample run for Team1:
      #+begin_src bash
# Make sure that the following python packages are installed: numpy, pandas, matplotlib, pyproj
python3 track2_evaluation_team1.py
# expected output:
#  plots: plots/team1_errcdf.png, plots/team1_trial1_trj.png, plots/team1_trial2_trj.png
# errors: ['trial1_team1: err75 = 28.151m', 'trial2_team1: err75 = 3.225m']
      #+end_src

    - Evaluation script for Team2 file:track2_evaluation_team2.py
    - Sample run for Team2:
      #+begin_src bash
# Make sure that the following python packages are installed: numpy, pandas, matplotlib, pyproj
python3 track2_evaluation_team2.py
# expected output:
#  plots: plots/team2_errcdf.png, plots/team2_trial1_trj.png, plots/team2_trial2_trj.png
# errors: ['trial1_team2: err75 = 2.055m', 'trial2_team2: err75 = 2.844m']
      #+end_src


*** Results summary

    Errors at $75^{th}$ percentile:

#+attr_html: :class="center"
| Trial  | Team  | 75% error, m |
|--------+-------+--------------|
| Trial1 | Team1 |       28.151 |
| Trial1 | Team2 |      *2.055* |
| Trial2 | Team1 |        3.225 |
| Trial2 | Team2 |        2.844 |

*** Plots Team1
    
   #+attr_html: :width 640px
   file:plots/team1_errcdf.png
   #+attr_html: :width 640px
   file:plots/team1_trial1_trj.png
   #+attr_html: :width 640px
   file:plots/team1_trial2_trj.png
   
*** Plots Team2
   
   #+attr_html: :width 640px
   file:plots/team2_errcdf.png
   #+attr_html: :width 640px
   file:plots/team2_trial1_trj.png
   #+attr_html: :width 640px
   file:plots/team2_trial2_trj.png