in_silico_perturber runtime issue
I am using the in_silico_perturber to evaluate the impact of gene deletions. This is my current setup running on an NVIDIA A100 40GB GPU:
perturb_type="delete",
perturb_rank_shift=None,
genes_to_perturb="all",
combos=0,
anchor_gene=None,
model_type="CellClassifier",
num_classes=4,
emb_mode="cls",
cell_emb_style="mean_pool",
cell_states_to_model=states_to_model,
state_embs_dict=state_embs_dict,
max_ncells=1500,
emb_layer=0,
forward_batch_size=256,
nproc=16,
clear_mem_ncells=200
additionally, states_to_model = {
"state_key": "condition",
"start_state": "r1000",
"goal_state": "control",
"alt_states": ["r10", "r100"]
}
This setup took ~60 hours to complete. I aim to perturb 15K cells, however, this is not feasible with the current runtime.
Is there any method to speed this up?
I noticed that using multiple GPUs could help solve this problem. However, I am wondering if distributing the perturbation into 4 GPUs processing 3.75K cells each would yield different results than one GPU processing 15K cells.
Thanks
Thanks for your question! Keep in mind that perturbing all genes in the cell essentially runs a simulated perturbed cell for each perturbation. So, if there are 4096 max genes per cell per the input size, this is 4096*1500+original 1500=6,145,500 cells running through the model over your 60 hours. To increase the speed, you can increase the batch size if your memory allows. Or, you can run in parallel on multiple GPUs. The argument "cell_inds_to_perturb" allows you to do this, running a separate set of cells for each job, defined by the cell indices. There will be no difference in the results as this is all performed as inference and doesn't involve training.