Multiprocessing with SARComp#

SARComp leverages OpenMP (OMP) for multiprocessing capabilities, allowing your computations to be distributed across multiple threads. This section provides detailed instructions on how to enable and utilize the multiprocessing features of SARComp.

Enabling Multiprocessing Support#

To enable multiprocessing in SARComp, follow the steps below:

  1. Configure the Build

    Run the configure script with the –enable-omp flag to enable OpenMP support. This script sets up the necessary parameters for the build.

    ./configure --enable-python-api --enable-omp

    This command generates a setup file with all the relevant parameters for building SARComp with OpenMP support.

  2. Build and Install SARComp

    After configuring the build, compile and install the library with the following commands:

    make pylibsarcomp
make pyinstall

    These commands will build the SARComp library and install it with Python bindings.

Using SARComp with Multiple Threads#

Once SARComp is installed with OpenMP support, you can run its functions using multiple threads. By default, SARComp will utilize the available CPU cores. However, you can specify the number of threads manually.

  1. Setting the Number of Threads

    You can set the number of threads by defining the SC_NUM_THREADS environment variable. For example, to use 4 threads:

    export SC_NUM_THREADS=4

  2. Setting Cache Usage with `SC_CACHE_FRACTION`

    SARComp allows you to control the fraction of cache usage for optimal performance on memory-bound tasks. The SC_CACHE_FRACTION environment variable sets the proportion of the L2 cache that SARComp can utilize per thread. This helps balance cache usage and avoid conflicts with other applications or system processes.

    For example, to set SARComp to use 90% of the L2 cache per thread:

    export SC_CACHE_FRACTION=0.90

    By default, SC_CACHE_FRACTION is set to 0.8 (80%), which reserves some cache for other processes, reducing potential cache contention. Increasing this value can improve performance on large data operations but may cause cache thrashing if other applications are also heavily utilizing the cache. Experiment with different values to find the best balance for your workload.

  3. Running SARComp Functions

    After setting the SC_NUM_THREADS environment variable, any function call to SARComp will utilize the specified number of threads. This allows you to optimize performance based on your specific computational needs.

    Here is an example of how you might use SARComp in a Python script:

    import sarcomp as sc

# Example function call
result = sc.fft.fft()

    By setting the environment variable as described, some_function will execute using the specified number of threads.

Important Note on Thread Management#

It is very important not to enable OpenMP when you intend to use any other multiprocessing or multi-threading Python libraries together with SARComp functions. The reason behind this is that initiating too many threads will lead to competition for resources, ultimately slowing down the overall operation. To ensure optimal performance, avoid combining OpenMP with other threading libraries in your Python scripts.

Best Practices and Tips#

  • Environment Variable Scope: The SC_NUM_THREADS environment variable should be set in the same shell session where you plan to run your SARComp scripts. If you are running a script from an integrated development environment (IDE), make sure to set the environment variable within the IDE’s configuration.

  • Performance Tuning: The optimal number of threads can vary based on your hardware and the specific computational task. Experiment with different values of SC_NUM_THREADS to find the best performance for your use case.

  • Thread Safety: Ensure that your code is thread-safe when using multiple threads, particularly if you are managing resources or data structures shared between threads.

Troubleshooting#

If you encounter issues when enabling or using multiprocessing with SARComp, consider the following troubleshooting steps:

  • Check Configuration: Ensure that the configure script was run with the correct flags and that no errors occurred during the configuration process.

  • Verify Environment Variable: Confirm that the SC_NUM_THREADS environment variable is set correctly in your shell session.

  • Review Logs: Check the build and runtime logs for any warnings or errors related to OpenMP or thread management.

For further assistance, refer to the SARComp documentation or contact us.

Additional Resources#

For more information on OpenMP and parallel computing, consider the following resources: