SLAM benchmark

The following metrics are evaluated (for these metrics, lower is better):

• ATE RMSE [cm]: Absolute trajectory root-mean-square-error in centimeters. This is the RMSE of the estimated pose translations vs. the ground truth pose translations.
• Rel. translation [%] for different evaluation thresholds t: Relative trajectory translation error in percent. This is the average percentual translation error after proceeding t meters.
• Rel. rotation [deg/m] for different evaluation thresholds t: Relative trajectory rotation error in degrees per meter. This is the average rotation error per meter after proceeding t meters.
• Time [s]: The runtime of the method.

The error metrics are evaluated for different ways of aligning the estimated trajectory and the ground truth trajectory:

• SE(3) alignment: The trajectories are aligned with a rigid-body transformation. SLAM methods that do not return results in metric scale (such as most monocular SLAM methods) are excluded from these evaluations.
• Sim(3) alignment: The trajectories are aligned with a similarity transformation.

The graph on the top is a cumulative error visualization for all datasets. For a given error threshold on the x axis, the plots show for how many datasets a method achieves a lower error. The table on the bottom gives the error values for each dataset. Results larger than 10 (centimeters, degrees per meter, ...) are shown as "fail".

The method ranking is determined as the area-under-curve measure for the graph on the top (up to the selected maximum error). In contrast to the other metrics, for this metric, larger is better.

Please note that every choice of a single error metric implies some kind of weighting between different quality aspects: Robustness (on how many datasets a SLAM method works) and accuracy (how well it works on those datasets where it does not fail). The chosen metric weights robustness highly, since having an additional result on a dataset usually adds much more area under the curve than slightly improving the results on many datasets.

Click a dataset result cell in the table to open a 3D visualization. Ground truth visualization is only available for training datasets.

			all	cables 1	cables 2	cables 3	camera shake 1	camera shake 2	camera shake 3	ceiling 1	ceiling 2	desk 3	desk changing 1	einstein 1	einstein 2	einstein dark	einstein flashlight	einstein global light changes 1	einstein global light changes 2	einstein global light changes 3	kidnap 1	kidnap dark	large loop 1	mannequin 1	mannequin 3	mannequin 4	mannequin 5	mannequin 7	mannequin face 1	mannequin face 2	mannequin face 3	mannequin head	motion 1	planar 2	planar 3	plant 1	plant 2	plant 3	plant 4	plant 5	plant dark	plant scene 1	plant scene 2	plant scene 3	reflective 1	repetitive	sfm bench	sfm garden	sfm house loop	sfm lab room 1	sfm lab room 2	sofa 1	sofa 2	sofa 3	sofa 4	sofa dark 1	sofa dark 2	sofa dark 3	sofa shake	table 3	table 4	table 7	vicon light 1	vicon light 2
	Method	Info
	Anonv2		8687.62 1	142.42 7	142.42 7	142.42 7	142.42 8	142.42 6	142.42 7	142.42 7	142.42 6	142.42 4	142.42 3	142.42 7	142.42 7	142.42 4	142.42 4	142.42 7	142.42 7	142.42 4	142.42 7	142.42 5	142.42 6	142.42 7	142.42 7	142.42 7	142.42 6	142.42 7	142.42 7	142.42 7	142.42 7	142.42 7	142.42 3	142.42 7	142.42 7	142.42 7	142.42 7	142.42 7	142.42 7	142.42 7	142.42 4	142.42 7	142.42 7	142.42 7	142.42 1	142.42 6	142.42 6	142.42 6	142.42 6	142.42 6	142.42 6	142.42 6	142.42 6	142.42 6	142.42 6	142.42 4	142.42 5	142.42 4	142.42 6	142.42 7	142.42 7	142.42 6	142.42 7	142.42 7
	Anon8		8687.62 1	142.42 7	142.42 7	142.42 7	142.42 8	142.42 6	142.42 7	142.42 7	142.42 6	142.42 4	142.42 3	142.42 7	142.42 7	142.42 4	142.42 4	142.42 7	142.42 7	142.42 4	142.42 7	142.42 5	142.42 6	142.42 7	142.42 7	142.42 7	142.42 6	142.42 7	142.42 7	142.42 7	142.42 7	142.42 7	142.42 3	142.42 7	142.42 7	142.42 7	142.42 7	142.42 7	142.42 7	142.42 7	142.42 4	142.42 7	142.42 7	142.42 7	142.42 1	142.42 6	142.42 6	142.42 6	142.42 6	142.42 6	142.42 6	142.42 6	142.42 6	142.42 6	142.42 6	142.42 4	142.42 5	142.42 4	142.42 6	142.42 7	142.42 7	142.42 6	142.42 7	142.42 7
	BADSLAM2		6675.16 3	59.51 4	69.27 6	17.21 3	21.22 5	39.80 5	33.74 4	120.04 4	112.26 4	205.54 6	276.77 5	21.37 3	64.32 3	122.08 3	78.81 3	39.15 4	20.82 3	168.55 6	43.74 4	104.01 4	154.67 8	29.16 3	31.40 3	41.10 5	200.76 8	46.32 4	18.94 3	20.12 3	53.14 5	88.76 5	795.00 7	26.69 3	38.68 3	6.87 5	8.59 4	10.77 4	7.57 4	10.55 3	57.24 3	31.45 3	36.80 5	26.60 3	2484.56 7	87.11 3	66.32 5	69.26 5	58.04 4	31.27 4	19.88 5	38.90 2	28.22 2	23.07 2	32.72 2	68.02 2	15.04 2	49.77 2	21.97 2	46.67 4	40.77 4	71.90 4	39.51 4	22.75 3
	DVI-SLAM		5805.65 4	71.18 5	8.29 3	25.16 5	74.37 7	251.60 9	52.82 5	126.73 5	174.54 8	238.74 7	378.29 7	21.50 4	84.41 5	334.03 6	279.14 7	81.70 6	25.30 5	244.39 7	59.87 5	75.90 3	79.77 4	76.26 6	63.42 6	49.73 6	99.76 5	91.15 6	24.43 5	30.28 6	68.65 6	105.17 6	139.66 2	34.82 5	60.57 5	3.78 2	5.85 2	10.46 2	4.85 2	11.23 4	169.27 6	39.57 5	32.46 3	37.18 5	691.46 6	133.60 5	52.37 3	59.25 4	51.14 3	25.97 3	10.72 3	50.25 4	39.02 4	37.69 4	64.65 5	243.12 6	30.34 4	169.98 6	37.77 4	35.87 2	34.74 2	97.74 5	48.65 5	45.10 6
	DROID-SLAM		5406.04 5	113.89 6	10.61 4	25.75 6	70.99 6	242.40 8	68.00 6	129.59 6	132.29 5	253.15 8	635.43 8	34.24 6	118.97 6		154.00 6	75.16 5	35.10 6	250.39 8	73.47 6		114.04 5	50.89 5	52.40 5	40.40 4	83.60 4	51.70 5	56.86 6	30.16 5	48.06 4	52.48 4	188.26 6	52.97 6	86.23 6	6.56 4	12.26 5	17.09 6	9.53 5	16.63 6		53.33 6	49.50 6	46.97 6	596.79 5	191.22 8	56.43 4	55.13 3	131.08 5	51.98 5	12.22 4	87.23 5	52.29 5	43.24 5	58.47 4				90.88 5	88.91 6	78.72 6	165.01 8	64.70 6	38.38 5
	fbadslam		3027.47 6	51.36 3	12.19 5	19.74 4	19.80 4	29.67 3	18.63 3	82.72 3	99.65 3	110.52 3	328.92 6	25.76 5	64.41 4	99.10 2	73.24 2	39.14 3	24.94 4	116.70 3	39.61 3	26.98 2	62.80 3	31.33 4	31.41 4	29.37 3	55.01 3	31.38 3	23.26 4	24.12 4	31.37 3	33.33 3	183.69 5	30.56 4	40.22 4	11.24 6	13.09 6	15.40 5	11.84 6	14.51 5	49.22 2	34.44 4	33.99 4	30.25 4	276.51 4	105.30 4						42.87 3	32.30 3	27.72 3	36.20 3	75.27 3	18.46 3	53.26 3	26.20 3	50.17 5	43.96 5	70.61 3	37.05 3	26.69 4
	badslam_opensource		2211.37 7	44.46 2	7.53 2	14.78 2	14.93 3	33.66 4	13.90 2	58.68 2	56.47 2	74.95 2	139.50 2	20.95 2	56.34 2	73.74 1	66.81 1	33.60 2	20.58 2	88.42 2	31.61 2	22.14 1	54.48 2	26.23 2	25.96 2	23.06 2	35.76 2	25.72 2	18.53 2	19.58 2	25.69 2	26.49 2	46.16 1	25.88 2	35.25 2	6.51 3	8.34 3	10.62 3	6.97 3	9.74 2	42.37 1	29.28 2	28.63 2	25.09 2	158.43 3	70.23 2	27.25 2	27.44 2	23.25 2	13.32 2	8.55 2	36.96 1	27.11 1	22.79 1	31.27 1	58.12 1	14.04 1	44.51 1	21.29 1	44.41 3	38.55 3	60.52 2	32.26 2	21.68 2
	ORB-SLAM3		498.87 8	22.12 1	1.66 1	5.00 1	3.46 1	8.60 1		15.78 1	16.41 1	29.59 1	49.72 1	7.72 1	23.85 1			5.52 1	7.66 1	24.57 1	11.28 1		16.22 1	6.78 1	6.31 1	6.25 1	8.97 1	6.01 1	5.68 1	6.22 1	6.06 1	7.00 1		9.54 1	11.67 1	0.84 1	1.39 1	2.41 1	0.98 1	2.15 1		6.34 1	6.92 1	7.79 1		24.36 1	9.89 1	10.62 1	7.58 1	3.80 1	1.28 1									19.33 1	15.40 1	26.28 1	13.82 1	8.04 1
	ROSEFusion		44.90 9				9.36 2	27.04 2	8.50 1
	ORB-SLAM2		0.00 10
	ElasticFusion		0.00 10
	DVO-SLAM		0.00 10
	BundleFusion		0.00 10
	BAD SLAM		0.00 10