Fig. 1

A Schematic of effective niche calculation: We aim to quantify the cell-type composition of each cell’s neighborhood. For each index cell, we calculate the pairwise kernel distance similarity between itself and each other cell in the sample. We use a Gaussian kernel with bandwidth \(\sigma\). The effective niche for the index cell is a vector with dimension equal to the number of unique cell types in the sample where index \(i\) represents the sum of kernel similarities between the index cell and the cells of type \(i\). B Schematic of niche-DE pipeline: To perform niche-DE, we first perform deconvolution/cell-type identification of our data. We then calculate the effective niche using a Gaussian kernel of bandwidth \(\sigma\). We then apply the regression-based niche-DE model using the effective niche calculated in the previous step. If one desires, they can repeat this step for multiple bandwidths. Using the Cauchy combination test across the different kernels used, we can calculate a p-value for testing whether gene \(g\) is an \((i,n)\) niche gene for all genes \(g\) and index-niche cell-type pairs. C FDR control: To guarantee correct FDR control, we utilize the hierarchical Benjamini-Hochberg procedure [33]. We first test if a gene shows evidence of being a niche-DE gene in any index-niche pair. This results in a p-value for each gene. We then apply the BH procedure to this set of genes. For a gene whose adjusted p-value is below the cutoff value, we test if it a niche gene in with index cell type \(i\). Testing across all \(T\) unique cell types in our sample, we get \(T\) cell-type-specific p-values for each gene \(g\) that is tested at this level. We then apply the Benjamini-Hochberg correction at level \(\alpha\) across these p-values for each gene. For all gene, index cell-type pairs \((g,i)\) that are significant after correction, we proceed to test if gene \(g\) is an \((i,n)\) niche gene for each niche cell type \(n\). After applying the Benjamini-Hochberg correction at level \(\alpha\) across all \(T\)p-values for each \((g,i)\) pair, if a \(\left(g,i,n\right)\) set has a p-value below the cutoff value, we conclude that gene \(g\) is an \((i,n)\) niche gene