Because of Python’s Global Interpreter Lock (GIL)<\/a>, the threads within each python process cannot truly run in parallel, unlike threads in other programming languages such as Java, C\/C++, and Go. For parallelism you have to create multiple processes, for this python comes with the multiprocessing module. Also note that Python’s modules are often written in C and have efficient multithreading, such as numpy<\/a><\/span>.<\/p>\n Note: To follow with these examples, both the jobscript and the python script will be used together.<\/em><\/p>\n Sample jobscript:<\/p>\n Sample Python script:<\/p>\n To execute this code, you would have to copy the job script into a file named multi_proc.sh<\/span>, the python script into multi_proc.py<\/span>, then run:<\/p>\n In most situations you need to pass the number of CPUs per task to the Pool<\/span> constructor. This argument tells the Pool<\/span> object how many “worker processes” to create. If I create more processes than I have CPUs then the processes will be fighting for resources. By default the Pool<\/span> object would create as many processes as there are CPUs on the system, which may not line up with your job allocation.<\/p>\n You’ll also want to prevent these processes from spawning tons of threads. The numpy library uses multithreading by default, and so parallelizing a python function that uses numpy may create a huge number of threads. If the number of running threads exceeds the number of cores this could bottleneck important system processes on our compute nodes. By default you should set these four environment variables in your job script, before executing any python code that uses the multiprocessing module.<\/p>\n These first two environment variables tell any OpenMP code to use one thread (per process). The second two environment variables tell numpy to use one thread (per process).<\/p>\nMultiprocessing Example:<\/h2>\n
#!\/bin\/bash\r\n#SBATCH --job-name=multiproc-test\r\n#SBATCH --output=multiproc-output.txt\r\n#SBATCH -c 4\r\n#SBATCH --time=15:00\r\n#SBATCH --mem=100M\r\n\r\nsrun .\/multi_proc.py 40 1\r\n<\/code><\/pre>\n#!\/usr\/bin\/env python3\r\n\r\nimport multiprocessing\r\nimport os\r\nimport sys\r\n\r\ndef call_stress(mins):\r\n os.system(f\"timeout {mins}m stress -c 1\")\r\nif __name__ == '__main__':\r\n sub_tasks = int(sys.argv[1])\r\n mins = int(sys.argv[2])\r\n cores = int(os.environ['SLURM_CPUS_PER_TASK'])\r\n with multiprocessing.Pool(cores) as pool:\r\n pool.map(call_stress,\r\n [mins for i in range(sub_tasks)])\r\n<\/code><\/pre>\n[abc123@wind ~]$ sbatch .\/multi_proc.sh<\/code><\/pre>\n[abc123@wind ~]$ export OMP_NUM_THREADS=1\r\n[abc123@wind ~]$ export MPI_NUM_THREADS=1\r\n[abc123@wind ~]$ export MKL_NUM_THREADS=1\r\n[abc123@wind ~]$ export OPENBLAS_NUM_THREADS=1<\/code><\/pre>\n