Submission and Evaluation
The evaluation process will be conducted using an automatic grader that will test the submitted programs using a collection of test cases in a batch mode. The automatic grader will measure both the accuracy of the generated output and the execution time of the program.
What to submit?
Each participant is expected to submit:
- A single .zip file that contains the original Source code and any dependencies. Please include a readme.txt file for any special instructions on how to compile the submitted code. Submission of the source code is mandatory to ensure originality of the submitted work.
A single executable file named: “mapmatch.exe”. The mapmatch.exe accepts three command line parameters. The usage of the mapmatch.exe program is as follows:
mapmatch [RoadNetworkInfo_Path] [Input_Path] [Output_Path]
RoadNetworkInfo_Path: Specifies the directory that contains three text files (WA_Nodes.txt, WA_Edges.txt and WA_EdgeGeometry.txt). These files contain the road network of Washington State in the U.S. The format of these files is detailed under the Road Network Information section.
Input_Path: Specifies the directory that contains n text files. Each file is a single test case and contains a series of recorded GPS readings recorded in Washington State in the U.S. The files are named as: input_01.txt to input_n.txt, where input_i.txt is the file that contains the ith test case. Examples of input test cases can be found in the Training Data Sets section.
Output_Path: Specifies the directory where the program is expected to place the output files. For each input file “input_i.txt” the mapmatch.exe program has to generate a corresponding output file named “output_i.txt”. Examples of the output files can be also found in the Training Data Sets section.
Note that the number of rows in the output file output_i.txt must be the same as the input file inut_i.txt. That is for every single row in the input file, there exists a row in the output file that has the same
<Time>values and indicates the
<EdgeId>that the GPS reading is believed to be on. Also, the
<EdgeId>must be annotated with a confidence value. Note that all test cases contain real world GPS data recorded using a GPS logger at a variable sampling rate. The sampling rate varies between one second and 30 seconds. This variable sampling rate is intended to test the resiliency of the submitted map matching algorithm under sparse GPS traces. The grading formula is best described by the following algorithm:
For each output file Output_i.txt
For each row in Output_i.txt
If <EdgeId> is correct
Grade = Grade + <Confidence>
Grade = Grade - <Confidence>
Grade = Grade/ExecutionTime
In plain English, if the result of a map matched GPS reading is correct, the participant earns one point weighted by the program’s declared confidence about this result. If the result of a map matched GPS reading is incorrect, the participant loses one point, again after being weighted by the confidence value. Considering the confidence value in the grading formula encourages participants to do their best effort in estimating the confidence value. A high confidence value for an incorrect result would result in a higher deduction in the grade. Finally, the grade is weighted by the total execution time of the program.
All submissions must be compatible with the Microsoft Windows Operating System and will be evaluated using machines running Microsoft Windows 7.
How to submit?
Submissions must be uploaded through the contest submission site at: https://cmt.research.microsoft.com/ACMGIS2012/