Fig. 2

A Overview of data simulation: To simulate realistic ST data, we take a real ST dataset and perform deconvolution to calculate the expected expression vector for each spot \({X}_{s}\). To generate data in the absence of niche effects, we simulate expression vectors from a negative binomial distribution with mean \({X}_{s}\) and overdispersion parameter 1. To generate data with niche effects, we specify \({\beta }_{i,n}\) for all index-niche pairs \((i,n)\) and calculate the new expected expression vector for each spot \({Y}_{s}\) based on the niche-DE model. We then simulate expression vectors from a negative binomial distribution with mean \({Y}_{s}\) and overdispersion parameter 1. We also simulate ST data in the presence of spatial bleeding by calculating new expression vectors based on the SpotClean model with local bleeding parameter 0.25. Afterwards, we calculate the type 1 error rate, power, and runtime of niche-DE. B Gene level, cell-type level, and interaction-level null p-value QQ plots when performing niche-DE on the simulated data. The empirical quantiles are based on those generated by niche-DE on the simulated data. The theoretical quantiles are based on the uniform distribution. C Power calculation when performing niche-DE on the simulated data with niche effects of varying sizes. D Runtime of nice-DE across the number of genes, the number of cells/spots, and the number of unique cell types present in the data. E Pseudo-spot data simulation overview: To simulate spot-level data from single cell level data, we created pseudo spots by partitioning the field of view into equal-sized squares. Counts are aggregated within each square, to create a pseudo-spot. Spot size is defined as the average number of cells in a pseudo-spot. We applied niche-DE to these lower-resolution datasets, and using the niche-DE results from the original high-resolution datasets as the gold standard, we computed the sensitivity and specificity of the niche genes found at each spot size. F Gene level, cell-type level, and interaction-level sensitivity and specificity vs spot size in both Slide-seq cerebellum, Xenium breast cancer, and CosMX SMI NSCLC data