Fig. 2

Evaluation of spatial regularization effects on deconvolution accuracy using simulated data. In the first simulation, doughnut-shaped spatial transcriptomic datasets were generated from a scRNA-seq reference [27] on a 50 × 50 grid. We assigned 15 cell types of high (n = 5) and low (n = 10) densities to overlapping spatial compartments (a-d). In the second simulation, spatial transcriptomic datasets of the mouse embryo were generated from the sci-Space dataset [36] by pooling single cells of eight types barcoded at the same spot (approximately 200-µm pitch) (e–h). a Relative abundance of two typical cell types in a single replicate. b Deconvolution accuracy of different methods as measured by Pearson correlation, with results aggregated over ten independent replicates. Error bars denote standard errors of the mean (50 high-density and 100 low-density cell types in total). c Binary presence status (proportion >  = 0.05) of two typical cell types in a single replicate. d Deconvolution accuracy of different methods, similar to b, but measured by binary prediction average precision score. e Relative abundance of the lateral plate mesoderm in one sci-Space slide. f Deconvolution accuracy of different methods as measured by Pearson correlation, with results aggregated over 14 biological slides. Error bars denote standard errors of the mean (112 cell types in total). g Binary presence status (proportion >  = 0.05) of the myocyte in one sci-Space slide. h Deconvolution accuracy of different methods, similar to f, but measured by binary prediction average precision score. Smoother-guided deconvolution models: NNLS (nonnegative least squares), DWLS (dampened weighted least squares, modified implementation), SVR (support vector regression), and LNR (log-normal regression. The CARD model features its own implementation of spatial regularization