One of my Python scripts is taking too long to run, which exceeds the short queue. I decided to use MPI, and mpi4py seems to be a good choice. But there are many issue when I first used it.
After many attempts, I found at least one way to use it.
Some ideas are from here: https://researchcomputing.princeton.edu/mpi4py
- ssh into the cluster
- load anaconda into system
- create a conda environment for this task mpienv
- activate the environment
- load compiler, I used gcc
- load openmpi
- export MPICC=”$(which mpicc)”
- pip install mpi4py, you should not use conda-forge channel because it will install mpi again.
- test with python -c “import mpi4py”
So earlier I had issues and there are several lessons: You should load before installing mpi4py Because we need to use system mpi, we need to load mpi after activate the environment.