Spatial Simulators
Assessing the quality of spatial transcriptomics simulators
Computational methods for spatially resolved transcriptomics (SRT) are frequently developed and assessed through data simulation. The effectiveness of these evaluations relies on the simulation methods' ability to accurately reflect experimental data. However, a systematic evaluation framework for spatial simulators is lacking. Here, we present SpatialSimBench, a comprehensive evaluation framework that assesses 13 simulation methods using 10 distinct STR datasets.
The research goal of this benchmark is to systematically evaluate and compare the performance of various simulation methods for spatial transcriptomics (ST) data. It aims to address the lack of a comprehensive evaluation framework for spatial simulators and explore the feasibility of leveraging existing single-cell simulators for ST data. The experimental setup involves collecting public spatial transcriptomics datasets and corresponding scRNA-seq datasets. The spatial and scRNA-seq datasets can originate from different study but should consist of similar cell types from similar tissues.
Leaderboard
Methods ranked by scaled overall mean. Each cell encodes a score from 0 to 1 by size and intensity.
QC: Normalisation Visualisation 11 plots
Per metric: points placed by control-anchored scaled score (x); dashed lines mark scaled 0 and 1 (worst/best control); the lower axis shows the raw score. Points beyond [-0.2, 1.2] are clamped to the edge as triangles. Hover a dot or line to highlight it and read details.
- clustering_arihigher better
- clustering_nmihigher better
- crosscor_cosinehigher better
- crosscor_mantelhigher better
- ctdeconvolute_jsdlower better
- ctdeconvolute_rmselower better
- L statisticslower better
- Moran's Ilower better
- Nearest-neighbour correlationlower better
- svg_precisionhigher better
- svg_recallhigher better
QC: Indicator table 102 errors2 warnings
Automated checks on the benchmark run and its results: missing values, score scaling, metric ranges and similar. Errors are high-severity issues that usually need a maintainer's attention; warnings are lower-severity signals. Findings that are expected for this task are listed separately as silenced.
102 high-severity issues need review. 303 of 407 checks passed.
- error Raw results Task number of results
Number of results should be equal to #datasets × #methods × #metrics Task: spatial_simulators Number of results: 1004 Number of datasets: 10 Number of methods: 11 Number of metrics: 39 Expected number of results: 4290
- error Raw results Dataset 'osteosarcoma' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Dataset: osteosarcoma Number of results: 106 Expected number of results: 429 Percentage missing: 75%
- error Raw results Dataset 'cortex' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Dataset: cortex Number of results: 107 Expected number of results: 429 Percentage missing: 75%
- error Raw results Dataset 'breast' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Dataset: breast Number of results: 91 Expected number of results: 429 Percentage missing: 79%
- error Raw results Dataset 'prostate' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Dataset: prostate Number of results: 101 Expected number of results: 429 Percentage missing: 76%
- error Raw results Dataset 'gastrulation' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Dataset: gastrulation Number of results: 108 Expected number of results: 429 Percentage missing: 75%
- error Raw results Dataset 'brain' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Dataset: brain Number of results: 89 Expected number of results: 429 Percentage missing: 79%
- error Raw results Dataset 'pdac' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Dataset: pdac Number of results: 103 Expected number of results: 429 Percentage missing: 76%
- error Raw results Dataset 'olfactorybulb' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Dataset: olfactorybulb Number of results: 108 Expected number of results: 429 Percentage missing: 75%
- error Raw results Dataset 'fibrosarcoma' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Dataset: fibrosarcoma Number of results: 105 Expected number of results: 429 Percentage missing: 76%
- error Raw results Dataset 'hindlimbmuscle' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Dataset: hindlimbmuscle Number of results: 86 Expected number of results: 429 Percentage missing: 80%
- error Raw results Method 'scdesign2' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Method: scdesign2 Number of results: 100 Expected number of results: 390 Percentage missing: 74%
- error Raw results Method 'scdesign3_nb' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Method: scdesign3_nb Number of results: 102 Expected number of results: 390 Percentage missing: 74%
- error Raw results Method 'scdesign3_poisson' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Method: scdesign3_poisson Number of results: 95 Expected number of results: 390 Percentage missing: 76%
- error Raw results Method 'sparsim' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Method: sparsim Number of results: 100 Expected number of results: 390 Percentage missing: 74%
- error Raw results Method 'splatter' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Method: splatter Number of results: 104 Expected number of results: 390 Percentage missing: 73%
- error Raw results Method 'srtsim' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Method: srtsim Number of results: 100 Expected number of results: 390 Percentage missing: 74%
- error Raw results Method 'symsim' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Method: symsim Number of results: 100 Expected number of results: 390 Percentage missing: 74%
- error Raw results Method 'zinbwave' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Method: zinbwave Number of results: 0 Expected number of results: 390 Percentage missing: 100%
- error Raw results Method 'positive' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Method: positive Number of results: 106 Expected number of results: 390 Percentage missing: 73%
- error Raw results Method 'negative_shuffle' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Method: negative_shuffle Number of results: 99 Expected number of results: 390 Percentage missing: 75%
- error Raw results Method 'negative_normal' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Method: negative_normal Number of results: 98 Expected number of results: 390 Percentage missing: 75%
- error Raw results Metric 'svg_precision' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: svg_precision Number of results: 65 Expected number of results: 110 Percentage missing: 41%
- error Raw results Metric 'crosscor_cosine' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: crosscor_cosine Number of results: 68 Expected number of results: 110 Percentage missing: 38%
- error Raw results Metric 'ks_statistic_frac_zero_genes_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_frac_zero_genes_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_frac_zero_cells_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_frac_zero_cells_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_lib_size_cells_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_lib_size_cells_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_efflib_size_cells_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_efflib_size_cells_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_tmm_cells_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_tmm_cells_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_scaled_var_cells_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_scaled_var_cells_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_scaled_mean_cells_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_scaled_mean_cells_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_lib_fraczero_cells_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_lib_fraczero_cells_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_pearson_cells_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_pearson_cells_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_scaled_var_genes_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_scaled_var_genes_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_scaled_mean_genes_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_scaled_mean_genes_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_pearson_genes_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_pearson_genes_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_mean_var_genes_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_mean_var_genes_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_mean_fraczero_genes_zstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_mean_fraczero_genes_zstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_frac_zero_genes_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_frac_zero_genes_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_frac_zero_cells_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_frac_zero_cells_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_lib_size_cells_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_lib_size_cells_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_efflib_size_cells_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_efflib_size_cells_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_tmm_cells_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_tmm_cells_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_scaled_var_cells_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_scaled_var_cells_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_scaled_mean_cells_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_scaled_mean_cells_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_lib_fraczero_cells_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_lib_fraczero_cells_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_pearson_cells_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_pearson_cells_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_scaled_var_genes_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_scaled_var_genes_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_scaled_mean_genes_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_scaled_mean_genes_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_pearson_genes_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_pearson_genes_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_mean_var_genes_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_mean_var_genes_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Metric 'ks_statistic_mean_fraczero_genes_tstat' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: ks_statistic_mean_fraczero_genes_tstat Number of results: 0 Expected number of results: 110 Percentage missing: 100%
- error Raw results Method 'zinbwave' % failed
Percentage of failed processes should be less than 10% Task: spatial_simulators Method: zinbwave Succeeded processes: 0 Attempted processes: 10 Percentage failed: 100%
- error Raw results Metric 'svg_recall' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: svg_recall Control method scores: 24 Expected control method scores: 30 Percentage succeeded: 80%
- error Raw results Metric 'svg_precision' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: svg_precision Control method scores: 9 Expected control method scores: 30 Percentage succeeded: 30%
- error Raw results Metric 'ks_statistic_frac_zero_genes_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_frac_zero_genes_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_frac_zero_cells_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_frac_zero_cells_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_lib_size_cells_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_lib_size_cells_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_efflib_size_cells_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_efflib_size_cells_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_tmm_cells_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_tmm_cells_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_scaled_var_cells_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_scaled_var_cells_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_scaled_mean_cells_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_scaled_mean_cells_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_lib_fraczero_cells_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_lib_fraczero_cells_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_pearson_cells_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_pearson_cells_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_scaled_var_genes_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_scaled_var_genes_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_scaled_mean_genes_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_scaled_mean_genes_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_pearson_genes_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_pearson_genes_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_mean_var_genes_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_mean_var_genes_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_mean_fraczero_genes_zstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_mean_fraczero_genes_zstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_frac_zero_genes_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_frac_zero_genes_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_frac_zero_cells_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_frac_zero_cells_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_lib_size_cells_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_lib_size_cells_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_efflib_size_cells_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_efflib_size_cells_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_tmm_cells_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_tmm_cells_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_scaled_var_cells_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_scaled_var_cells_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_scaled_mean_cells_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_scaled_mean_cells_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_lib_fraczero_cells_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_lib_fraczero_cells_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_pearson_cells_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_pearson_cells_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_scaled_var_genes_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_scaled_var_genes_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_scaled_mean_genes_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_scaled_mean_genes_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_pearson_genes_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_pearson_genes_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_mean_var_genes_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_mean_var_genes_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Raw results Metric 'ks_statistic_mean_fraczero_genes_tstat' number of control methods
Number of metric scores for control methods should be equal to #datasets × #control_methods Task: spatial_simulators Metric: ks_statistic_mean_fraczero_genes_tstat Control method scores: 0 Expected control method scores: 30 Percentage succeeded: 0%
- error Scaling Metric 'svg_precision' % outside range
Percentage of scaled scores outside control range should be less than 10% Task: spatial_simulators Metric: svg_precision Inside range: NA Scaled scores: 65 Percentage outside: NA%
- error Scaling Worst 'svg_precision' score for 'negative_shuffle'
Method 'negative_shuffle' performs much worse than controls for metric' svg_precision' Task: spatial_simulators Method: negative_shuffle Metric: svg_precision Worst score: NA Percentage outside range: 0%
- error Scaling Best 'svg_precision' score for 'negative_shuffle'
Method 'negative_shuffle' performs much better than controls for metric 'svg_precision' Task: spatial_simulators Method: negative_shuffle Metric: svg_precision Best score: NA Percentage outside range: 0%
- error Scaling Worst 'svg_precision' score for 'positive'
Method 'positive' performs much worse than controls for metric' svg_precision' Task: spatial_simulators Method: positive Metric: svg_precision Worst score: NA Percentage outside range: 0%
- error Scaling Best 'svg_precision' score for 'positive'
Method 'positive' performs much better than controls for metric 'svg_precision' Task: spatial_simulators Method: positive Metric: svg_precision Best score: NA Percentage outside range: 0%
- error Scaling Worst 'svg_precision' score for 'scdesign2'
Method 'scdesign2' performs much worse than controls for metric' svg_precision' Task: spatial_simulators Method: scdesign2 Metric: svg_precision Worst score: NA Percentage outside range: 0%
- error Scaling Best 'svg_precision' score for 'scdesign2'
Method 'scdesign2' performs much better than controls for metric 'svg_precision' Task: spatial_simulators Method: scdesign2 Metric: svg_precision Best score: NA Percentage outside range: 0%
- error Scaling Worst 'svg_precision' score for 'scdesign3_nb'
Method 'scdesign3_nb' performs much worse than controls for metric' svg_precision' Task: spatial_simulators Method: scdesign3_nb Metric: svg_precision Worst score: NA Percentage outside range: 0%
- error Scaling Best 'svg_precision' score for 'scdesign3_nb'
Method 'scdesign3_nb' performs much better than controls for metric 'svg_precision' Task: spatial_simulators Method: scdesign3_nb Metric: svg_precision Best score: NA Percentage outside range: 0%
- error Scaling Worst 'svg_precision' score for 'scdesign3_poisson'
Method 'scdesign3_poisson' performs much worse than controls for metric' svg_precision' Task: spatial_simulators Method: scdesign3_poisson Metric: svg_precision Worst score: NA Percentage outside range: 0%
- error Scaling Best 'svg_precision' score for 'scdesign3_poisson'
Method 'scdesign3_poisson' performs much better than controls for metric 'svg_precision' Task: spatial_simulators Method: scdesign3_poisson Metric: svg_precision Best score: NA Percentage outside range: 0%
- error Scaling Worst 'svg_precision' score for 'sparsim'
Method 'sparsim' performs much worse than controls for metric' svg_precision' Task: spatial_simulators Method: sparsim Metric: svg_precision Worst score: NA Percentage outside range: 0%
- error Scaling Best 'svg_precision' score for 'sparsim'
Method 'sparsim' performs much better than controls for metric 'svg_precision' Task: spatial_simulators Method: sparsim Metric: svg_precision Best score: NA Percentage outside range: 0%
- error Scaling Worst 'svg_precision' score for 'splatter'
Method 'splatter' performs much worse than controls for metric' svg_precision' Task: spatial_simulators Method: splatter Metric: svg_precision Worst score: NA Percentage outside range: 0%
- error Scaling Best 'svg_precision' score for 'splatter'
Method 'splatter' performs much better than controls for metric 'svg_precision' Task: spatial_simulators Method: splatter Metric: svg_precision Best score: NA Percentage outside range: 0%
- error Scaling Worst 'svg_precision' score for 'srtsim'
Method 'srtsim' performs much worse than controls for metric' svg_precision' Task: spatial_simulators Method: srtsim Metric: svg_precision Worst score: NA Percentage outside range: 0%
- error Scaling Best 'svg_precision' score for 'srtsim'
Method 'srtsim' performs much better than controls for metric 'svg_precision' Task: spatial_simulators Method: srtsim Metric: svg_precision Best score: NA Percentage outside range: 0%
- error Scaling Worst 'svg_precision' score for 'symsim'
Method 'symsim' performs much worse than controls for metric' svg_precision' Task: spatial_simulators Method: symsim Metric: svg_precision Worst score: NA Percentage outside range: 0%
- error Scaling Best 'svg_precision' score for 'symsim'
Method 'symsim' performs much better than controls for metric 'svg_precision' Task: spatial_simulators Method: symsim Metric: svg_precision Best score: NA Percentage outside range: 0%
Show 2 warnings
- warning Raw results Metric 'svg_recall' % missing
Percentage of missing results should be less than 10% Task: spatial_simulators Metric: svg_recall Number of results: 79 Expected number of results: 110 Percentage missing: 28%
- warning Raw results Dataset 'breast' % failed
Percentage of failed processes should be less than 10% Task: spatial_simulators Dataset: breast Succeeded processes: 9 Attempted processes: 11 Percentage failed: 18%
Method info 8
A transparent simulator that generates high-fidelity single-cell gene expression count data with gene correlations captured
scDesign2 is a transparent simulator that achieves all three goals (preserving genes, capturing gene correlations, and generating any number of cells with varying sequencing depths) and generates high-fidelity synthetic data for multiple single-cell gene expression count-based technologies.
A probabilistic model that unifies the generation and inference for single-cell and spatial omics data
scDesign3 offers a probabilistic model that unifies the generation and inference for single-cell and spatial omics data. The model's interpretable parameters and likelihood enable scDesign3 to generate customized in silico data and unsupervisedly assess the goodness-of-fit of inferred cell latent structures (for example, clusters, trajectories and spatial locations).
A probabilistic model that unifies the generation and inference for single-cell and spatial omics data
scDesign3 offers a probabilistic model that unifies the generation and inference for single-cell and spatial omics data. The model's interpretable parameters and likelihood enable scDesign3 to generate customized in silico data and unsupervisedly assess the goodness-of-fit of inferred cell latent structures (for example, clusters, trajectories and spatial locations).
SPARSim single cell is a count data simulator for scRNA-seq data.
SPARSim is a scRNA-seq count data simulator based on a Gamma-Multivariate Hypergeometric model. It allows to generate count data that resembles real data in terms of count intensity, variability and sparsity.
A single cell RNA-seq data simulator based on a gamma-Poisson distribution.
The Splat model is a gamma-Poisson distribution used to generate a gene by cell matrix of counts. Mean expression levels for each gene are simulated from a gamma distribution and the Biological Coefficient of Variation is used to enforce a mean-variance trend before counts are simulated from a Poisson distribution.
An SRT-specific simulator for scalable, reproducible, and realistic SRT simulations.
A key benefit of srtsim is its ability to maintain location-wise and gene-wise SRT count properties and preserve spatial expression patterns, enabling evaluation of SRT method performance using synthetic data.
Simulating multiple faceted variability in single cell RNA sequencing
SymSim is a simulator for modeling single-cell RNA-Seq data, accounting for three primary sources of variation: intrinsic transcription noise, extrinsic variation from different cell states, and technical variation from measurement noise and bias.
A general and flexible method for signal extraction from single-cell RNA-seq data
ZINB-WaVE is a general and flexible zero-inflated negative binomial model, which leads to low-dimensional representations of the data that account for zero inflation (dropouts), over-dispersion, and the count nature of the data.
Control method info 3
A negative control which generates normal distributed data.
This control method generates normally distributed data as a negative control, using a fixed mean of 3 and standard deviation of 1.
A negative control method which shuffles the input data.
This control method shuffles the input data as a negative control.
A positive control method.
Metric info 39
Adjusted rand index (ARI) measures the similarity between two clusters in real and simulated datasets.
Adjusted Rand Index used in spatial clustering to measure the similarity between two data clusterings, adjusted for chance.
Normalized mutual information (NMI) measures of the mutual dependence between the real and simulated spatial clusters.
Normalized Mutual Information used in spatial clustering to measure the agreement between two different clusterings, scaled to [0, 1].
Cosine similarity measures similarity between bivariate Moran’s I of real dataset and that of in simulation dataset.
Cosine similarity used in spatial cross-correlation to measure the cosine of the angle between two non-zero vectors.
Mantel statistic is the test statistic for the Mantel test, which is a correlation coefficient calculated between bivariate Moran’s I of real dataset and that of in simulation dataset.
Mantel statistic used in spatial cross-correlation to test the correlation between two distance matrices.
Jensen-Shannon divergence (JSD) is calculated between the true and predicted proportion per cell type in all spots.
Jensen-Shannon Divergence used in cell type deconvolution to measure the similarity between two probability distributions.
Root Mean Square deviation is calculated between the true and predicted proportion of per cell type.
Root Mean Squared Error used in cell type deconvolution to measure the difference between observed and predicted values.
KS statistic of the effective library size.
The Kolmogorov-Smirnov statistic comparing the effective library size of the real datasets versus the effective library size of the simulated datasets.
KS statistic of the effective library size.
The Kolmogorov-Smirnov statistic comparing the effective library size of the real datasets versus the effective library size of the simulated datasets.
KS statistic of the fraction of zeros per spot (cell).
The Kolmogorov-Smirnov statistic comparing the fraction of zeros per spot (cell) in the real datasets versus the fraction of zeros per spot (cell) in the simulated datasets.
KS statistic of the fraction of zeros per spot (cell).
The Kolmogorov-Smirnov statistic comparing the fraction of zeros per spot (cell) in the real datasets versus the fraction of zeros per spot (cell) in the simulated datasets.
KS statistic of the fraction of zeros per gene.
The Kolmogorov-Smirnov statistic comparing the fraction of zeros per gene in the real datasets versus the fraction of zeros per gene in the simulated datasets.
KS statistic of the fraction of zeros per gene.
The Kolmogorov-Smirnov statistic comparing the fraction of zeros per gene in the real datasets versus the fraction of zeros per gene in the simulated datasets.
KS statistic of the gene Pearson correlation.
The Kolmogorov-Smirnov statistic comparing the gene Pearson correlation of the real datasets versus the gene Pearson correlation of the simulated datasets.
KS statistic of the gene Pearson correlation.
The Kolmogorov-Smirnov statistic comparing the gene Pearson correlation of the real datasets versus the gene Pearson correlation of the simulated datasets.
KS statistic of the L statistics
The Kolmogorov-Smirnov statistic comparing the L statistics in the real datasets versus the L statistics in the simulated datasets.
KS statistic of the library size.
The Kolmogorov-Smirnov statistic comparing the total sum of UMI counts across all genes in the real datasets versus the total sum of UMI counts across all genes in the simmulated datasets.
KS statistic of the library size.
The Kolmogorov-Smirnov statistic comparing the total sum of UMI counts across all genes in the real datasets versus the total sum of UMI counts across all genes in the simmulated datasets.
KS statistic of the relationship between library size and the proportion of zeros per spot (cell).
The Kolmogorov-Smirnov statistic comparing the relationship between library size and the proportion of zeros per spot (cell) in the real datasets versus the simulated datasets.
KS statistic of the relationship between library size and the proportion of zeros per spot (cell).
The Kolmogorov-Smirnov statistic comparing the relationship between library size and the proportion of zeros per spot (cell) in the real datasets versus the simulated datasets.
KS statistic of the relationship between mean expression and the proportion of zero per gene.
The Kolmogorov-Smirnov statistic comparing the relationship between mean expression and the proportion of zero per gene in the real datasets versus the simulated datasets.
KS statistic of the relationship between mean expression and the proportion of zero per gene.
The Kolmogorov-Smirnov statistic comparing the relationship between mean expression and the proportion of zero per gene in the real datasets versus the simulated datasets.
KS statistic of the relationship between mean expression and variance expression.
The Kolmogorov-Smirnov statistic comparing the relationship between mean expression and variance expression in the real datasets versus the simulated datasets.
KS statistic of the relationship between mean expression and variance expression.
The Kolmogorov-Smirnov statistic comparing the relationship between mean expression and variance expression in the real datasets versus the simulated datasets.
KS statistic of Moran's I.
The Kolmogorov-Smirnov statistic comparing the Moran's I of the real datasets versus the Moran's I of the simulated datasets.
KS statistic of the nearest-neighbour correlation.
The Kolmogorov-Smirnov statistic comparing the nn correlation in the real datasets versus the nn correlation in the simulated datasets.
KS statistic of the sample Pearson correlation.
The Kolmogorov-Smirnov statistic comparing the sample Pearson correlation of the real datasets versus the sample Pearson correlation of the simulated datasets.
KS statistic of the sample Pearson correlation.
The Kolmogorov-Smirnov statistic comparing the sample Pearson correlation of the real datasets versus the sample Pearson correlation of the simulated datasets.
KS statistic of the spot- (or cell-) level scaled mean of the expression matrix.
The Kolmogorov-Smirnov statistic comparing the z-score standardization of the mean of expression matrix in terms of log2(CPM) in the real datasets versus the simulated datasets.
KS statistic of the spot- (or cell-) level scaled mean of the expression matrix.
The Kolmogorov-Smirnov statistic comparing the z-score standardization of the mean of expression matrix in terms of log2(CPM) in the real datasets versus the simulated datasets.
KS statistic of the gene-level scaled mean of the expression matrix.
The Kolmogorov-Smirnov statistic comparing the gene-level z-score standardization of the mean of expression matrix in terms of log2(CPM) in the real datasets versus the simulated datasets.
KS statistic of the gene-level scaled mean of the expression matrix.
The Kolmogorov-Smirnov statistic comparing the gene-level z-score standardization of the mean of expression matrix in terms of log2(CPM) in the real datasets versus the simulated datasets.
KS statistic of the spot- (or cell-) level scaled variance of the expression matrix.
The Kolmogorov-Smirnov statistic comparing the spot-level z-score standardization of the variance of expression matrix in terms of log2(CPM) in the real datasets versus the simulated datasets.
KS statistic of the spot- (or cell-) level scaled variance of the expression matrix.
The Kolmogorov-Smirnov statistic comparing the spot-level z-score standardization of the variance of expression matrix in terms of log2(CPM) in the real datasets versus the simulated datasets.
KS statistic of the gene-level scaled variance of the expression matrix.
The Kolmogorov-Smirnov statistic comparing the gene-level z-score standardization of the variance of expression matrix in terms of log2(CPM) in the real datasets versus the simulated datasets.
KS statistic of the gene-level scaled variance of the expression matrix.
The Kolmogorov-Smirnov statistic comparing the gene-level z-score standardization of the variance of expression matrix in terms of log2(CPM) in the real datasets versus the simulated datasets.
Precision measures the proportion of correctly identified items in simulated datasets.
Precision used in identifying spatial variable genes, measuring the accuracy of positive predictions.
Recall measures the proportion of real SVG correctly identified in the simulated dataset.
Recall used in identifying spatial variable genes, measuring the true positive rate.
KS statistic of the weight trimmed mean of M-values normalization factor (TMM).
The Kolmogorov-Smirnov statistic comparing the weight trimmed mean of M-values normalization factor for the real datasets versus the weight trimmed mean of M-values normalization factor for the simulated datasets.
KS statistic of the weight trimmed mean of M-values normalization factor (TMM).
The Kolmogorov-Smirnov statistic comparing the weight trimmed mean of M-values normalization factor for the real datasets versus the weight trimmed mean of M-values normalization factor for the simulated datasets.
Dataset info 10
10X Visium spatial RNA-seq from adult mouse brain sections paired to single-nucleus RNA-seq
This datasets were generated matched single nucleus (sn, this submission) and Visium spatial RNA-seq (10X Genomics) profiles of adjacent mouse brain sections that contain multiple regions from the telencephalon and diencephalon.
A spatially resolved atlas of human breast cancers
This study presents a spatially resolved transcriptomics analysis of human breast cancers.
Scripts and source data for image processing, barcode calling, and cell type annotations in a seqFISH+ experiment.
The dataset includes processed image data, cell type annotations with Louvain clusters, gene IDs for transcript locations, and mRNA point locations, with additional data available on Zenodo.
Multi-resolution deconvolution of spatial transcriptomics data reveals continuous patterns of Tumor A1 of Tissue 1
Spatial transcriptomics of Tumor A1 of Tissue 1.
single-cell and spatial transcriptomic molecular map of mouse gastrulation
Single-Cell omics Data across Mouse Gastrulation and Highly multiplexed spatially resolved gene expression profiling of Early Organogenesis.
Spatial RNA sequencing of regenerating mouse hindlimb muscle
The spatial transcriptomics datasets regenerates mouse muscle tissue generated with the 10x Genomics Visium platform.
Single-cell and spatial transcriptomic of mouse olfactory bulb
Spatial profiling of human osteosarcoma cells.
Spatial transcriptome profiling by MERFISH reveals subcellular RNA compartmentalization and cell cycle-dependent gene expression.
Integrating microarray-based spatial transcriptomics and single-cell RNA-seq reveals tissue architecture in pancreatic ductal adenocarcinomas
We developed a multimodal intersection analysis method combining scRNA-seq with spatial transcriptomics to map and characterize the spatial organization and interactions of distinct cell subpopulations in complex tissues, such as primary pancreatic tumors..
Spatially resolved gene expression of human protate tissue slices treated with steroid hormones for 8 hours
Spatially resolved gene expression was prepard by dissociated hman prostate tissue to single cells, and collected & prepped for RNA-seq using the Visium Spatial Gene Expression kit.
- Journal of Machine Learning Technologies. (n.d.). 10.9735/2229-3981 ↗
- Baruzzo, G., Patuzzi, I., & Di Camillo, B. (2019). SPARSim single cell: a count data simulator for scRNA-seq data. 10.1093/bioinformatics/btz752 ↗
- Chacón, J. E., & Duong, T. (2018). Multivariate Kernel Smoothing and its Applications. 10.1201/9780429485572 ↗
- Drost, H.-G. (2018). Philentropy: Information Theory and Distance Quantification with R. 10.21105/joss.00765 ↗
- Eng, C.-H. L., Lawson, M., Zhu, Q., Dries, R., Koulena, N., Takei, Y., Yun, J., Cronin, C., Karp, C., Yuan, G.-C., & Cai, L. (2019). Transcriptome-scale super-resolved imaging in tissues by RNA seqFISH+. 10.1038/s41586-019-1049-y ↗
- Hodson, T. O. (2022). Root-mean-square error (RMSE) or mean absolute error (MAE): when to use them or not. 10.5194/gmd-15-5481-2022 ↗
- Kleshchevnikov, V., Shmatko, A., Dann, E., Aivazidis, A., King, H. W., Li, T., Elmentaite, R., Lomakin, A., Kedlian, V., Gayoso, A., Jain, M. S., Park, J. S., Ramona, L., Tuck, E., Arutyunyan, A., Vento-Tormo, R., Gerstung, M., James, L., Stegle, O., & Bayraktar, O. A. (2022). Cell2location maps fine-grained cell types in spatial transcriptomics. Nature Biotechnology, 40(5), 661–671. 10.1038/s41587-021-01139-4 ↗
- Legendre, P., Fortin, M., & Borcard, D. (2015). Should the Mantel test be used in spatial analysis? 10.1111/2041-210x.12425 ↗
- Leydesdorff, L. (2005). Similarity measures, author cocitation analysis, and information theory. 10.1002/asi.20130 ↗
- Liang, X., Cao, Y., & Hwa Yang, J. Y. (2024). Multi-task benchmarking of spatially resolved gene expression simulation models. 10.1101/2024.05.29.596418 ↗
- Lohoff, T., Ghazanfar, S., Missarova, A., Koulena, N., Pierson, N., Griffiths, J. A., Bardot, E. S., Eng, C.-H. L., Tyser, R. C. V., Argelaguet, R., Guibentif, C., Srinivas, S., Briscoe, J., Simons, B. D., Hadjantonakis, A.-K., Göttgens, B., Reik, W., Nichols, J., Cai, L., & Marioni, J. C. (2021). Integration of spatial and single-cell transcriptomic data elucidates mouse organogenesis. 10.1038/s41587-021-01006-2 ↗
- Lopez, R., Li, B., Keren-Shaul, H., Boyeau, P., Kedmi, M., Pilzer, D., Jelinski, A., Yofe, I., David, E., Wagner, A., Ergen, C., Addadi, Y., Golani, O., Ronchese, F., Jordan, M. I., Amit, I., & Yosef, N. (2022). DestVI identifies continuums of cell types in spatial transcriptomics data. Nature Biotechnology, 40(9), 1360–1369. 10.1038/s41587-022-01272-8 ↗
- McCray, T., Pacheco, J. V., Loitz, C. C., Garcia, J., Baumann, B., Schlicht, M. J., Valyi-Nagy, K., Abern, M. R., & Nonn, L. (2021). Vitamin D sufficiency enhances differentiation of patient-derived prostate epithelial organoids. 10.1016/j.isci.2021.102640 ↗
- McKellar, D. W., Walter, L. D., Song, L. T., Mantri, M., Wang, M. F. Z., De Vlaminck, I., & Cosgrove, B. D. (2020). Strength in numbers: Large-scale integration of single-cell transcriptomic data reveals rare, transient muscle progenitor cell states in muscle regeneration. 10.1101/2020.12.01.407460 ↗
- Risso, D., Perraudeau, F., Gribkova, S., Dudoit, S., & Vert, J.-P. (2018). A general and flexible method for signal extraction from single-cell RNA-seq data. 10.1038/s41467-017-02554-5 ↗
- Song, D., Wang, Q., Yan, G., Liu, T., Sun, T., & Li, J. J. (2023). scDesign3 generates realistic in silico data for multimodal single-cell and spatial omics. 10.1038/s41587-023-01772-1 ↗
- Ståhl, P. L., Salmén, F., Vickovic, S., Lundmark, A., Navarro, J. F., Magnusson, J., Giacomello, S., Asp, M., Westholm, J. O., Huss, M., Mollbrink, A., Linnarsson, S., Codeluppi, S., Borg, Å., Pontén, F., Costea, P. I., Sahlén, P., Mulder, J., Bergmann, O., … Frisén, J. (2016). Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. 10.1126/science.aaf2403 ↗
- Sun, T., Song, D., Li, W. V., & Li, J. J. (2021). scDesign2: a transparent simulator that generates high-fidelity single-cell gene expression count data with gene correlations captured. 10.1186/s13059-021-02367-2 ↗
- Vinh, N. X., Epps, J., & Bailey, J. (2009). Information theoretic measures for clusterings comparison. 10.1145/1553374.1553511 ↗
- Wu, S. Z., Al-Eryani, G., Roden, D. L., Junankar, S., Harvey, K., Andersson, A., Thennavan, A., Wang, C., Torpy, J. R., Bartonicek, N., Wang, T., Larsson, L., Kaczorowski, D., Weisenfeld, N. I., Uytingco, C. R., Chew, J. G., Bent, Z. W., Chan, C.-L., Gnanasambandapillai, V., … Swarbrick, A. (2021). A single-cell and spatially resolved atlas of human breast cancers. Nature Genetics, 53(9), 1334–1347. 10.1038/s41588-021-00911-1 ↗
- Xia, C., Fan, J., Emanuel, G., Hao, J., & Zhuang, X. (2019). Spatial transcriptome profiling by MERFISH reveals subcellular RNA compartmentalization and cell cycle-dependent gene expression. 10.1073/pnas.1912459116 ↗
- Zappia, L., Phipson, B., & Oshlack, A. (2017). Splatter: simulation of single-cell RNA sequencing data. 10.1186/s13059-017-1305-0 ↗
- Zhang, X., Xu, C., & Yosef, N. (2019). Simulating multiple faceted variability in single cell RNA sequencing. 10.1038/s41467-019-10500-w ↗
- Zhu, J., Shang, L., & Zhou, X. (2023). SRTsim: spatial pattern preserving simulations for spatially resolved transcriptomics. 10.1186/s13059-023-02879-z ↗