Building From Source on Windows

Overview

Compared to OSX and Linux, building NumPy and SciPy on Windows is more difficult, largely due to the lack of compatible, open-source libraries like LAPACK or ATLAS that are necessary to build both libraries and have them perform relatively well. It is not possible to just call a one-liner on the command prompt as you would on other platforms via sudo apt-get install machinery.

This document describes one option to build OpenBLAS and SciPy from source that was validated for scipy 1.0.0. However, in light of all the work currently being done, do not expect these instructions to be accurate in the long-run and be sure to check up on any of the open source projects mentioned for the most up-to-date information. For more information on all of these projects, the Mingwpy website is an excellent source of in-depth information than this document will provide.

Building the Released SciPy

This section provides the step-by-step process to build the released scipy. If you want to build completely from source, you should estimate at least three hours to build all libraries and compile SciPy. Feel free to stop and inspect any step at any time, but for this section, we’ll just mention the steps without providing an in-depth explanation for the reasons behind them. If you have further questions about what we’re doing, more in-depth documentation is provided in the sections below. Also, please make sure to read this section before proceeding, as the presence or absence of error messages in general is not a good indication of whether you’ve completed a step correctly. Each step creates particular files, and what ultimately matters is whether you have built the required files rather than whether error messages appeared in your terminal.

Building OpenBLAS

First, we need to install the software required to build OpenBLAS, which is the BLAS library that we’re going to use. Because the software to build OpenBLAS is different than that required to build SciPy and because OpenBLAS takes a long time to build, we’re going to start building OpenBLAS first and then explain what to do next while the OpenBLAS build is running. Alternatively, if you’d rather download a pre-built OpenBLAS, download the one of the pre-built zip files and skip to the Installing OpenBLAS section below.

Otherwise, install MSYS2 using these instructions including the pacman update instructions. Occasionally during the updates the terminal might ask you to close the terminal but then might refuse to be closed and hang. If this happens you can kill it via Task Manager and continue with the instructions. Make sure to install the correct architecture for the SciPy that you want to build (eg. 32 or 64 bit). Now, you have three options for opening a terminal which are MSYS2, MINGW (32 or 64 bit). After updating all the packages, now we are ready to install some more package bundles that we will need. Open a MSYS2 terminal and type the following depending on the architecture of your choice; run the following for a 32-bit build

pacman -S --needed base-devel mingw-w64-i686-toolchain mingw-w64-i686-cmake git

and for 64-bit

pacman -S --needed base-devel mingw-w64-x86_64-toolchain mingw-w64-x86_64-cmake git

It will prompt to whether install everything in these packages and you can simply accept all via hitting enter key at each step.

We should be aware of the fact that these tools also install Python2, very similar to a virtual environment, which is only usable within an MSYS2 terminal and we are not going to use it at any point.

If you already have a GitHub repository folder where you keep your own repos, it is better to use that location to keep things nice and tidy since we are going to clone yet another repository to obtain the source code, hence

cd /c/<wherever the GitHub repo folder is>/GitHub

You don’t necessarily need to build in that particular location, but it should be somewhere convenient. To make sure that we’re ready to build, type the following in the terminal:

make
gfortran
gcc

These commands should give errors as we have not provided any arguments to them. However an error also implies that they are accessible on the path. Now clone the repository required to build OpenBLAS:

git clone https://github.com/matthew-brett/build-openblas.git
cd build-openblas
git submodule update --init --recursive

If any of these commands fail, you’re not ready to build. Go back and make sure that MSYS2 is installed correctly and has the required packages enabled. Now, let’s set some environment variables. In the MSYS2 terminal, type the following.

export OPENBLAS_COMMIT=5f998ef
export OPENBLAS_ROOT="C:\\opt"
export BUILD_BITS=64

Please check these variables’ purpose for a moment. More specifically, make sure that you have read/write access to the path that OPENBLAS_ROOT points to. The output of the OpenBLAS build will be collected in this folder. Make sure that the OPENBLAS_COMMIT points to the correct OpenBLAS commit that you want to build in the cloned repo. In the future, build_openblas repository might get updated and you might want to get those updates by changing the commit. Make sure that the architecture is correctly set to either 32 or 64 bit. And after you’ve made sure of that, start the OpenBLAS build with:

./build_openblas.sh

Building OpenBLAS is challenging. The build may fail with an error after a few hours but may also fail silently and produce an incorrect binary. Please, if you have any issues, report them so that we can save the next person’s time.

While you’re waiting on OpenBLAS to finish building, go ahead and install build tools from Microsoft, since these take a while to install and you’ll need them later.

After the build_openblas.sh script has completed (probably with an error), there should be an openblas.a file somewhere on your system. If OPENBLAS_ROOT was set to C:\\opt, then you might see a line like this in the MSYS2 terminal:

Copying the static library to /c/opt/64/lib

Installing OpenBLAS

If you see that line, then you might have OpenBLAS correctly built, even if other failures occurred. Look in that folder for openblas.a. If you find a file called something like libopenblas_5f998ef_gcc7_2_0.a, just rename it to openblas.a and continue. If the file isn’t there, then poke around and try to find the file elsewhere in OPENBLAS_ROOT. If you don’t have that file, you’ll probably need to find out what happened and then build OpenBLAS again. But if you have that file, we’ll assume that you’ve completed this step correctly. Proceeding on that assumption, let’s build SciPy.

Before continuing, make sure that you don’t have other copies of either openblas.lib or libopenblas.lib on your computer elsewhere. Multiple copies could result in later build errors that will be difficult to debug. You may verifiy that the openblas library was correctly picked up by looking for the following in your build log:

FOUND:
   libraries = ['openblas']
   library_dirs = ['C:\opt\64\lib']
   language = c
   define_macros = [('HAVE_CBLAS', None)]

Building SciPy

Once you have built OpenBLAS, it’s time to build SciPy. Before continuing make sure to install the following software for building on the latest Python version. For building on other Python versions, see the WindowsCompilers page.

1. Install Microsoft Visual Studio 2015 or 2017 Community Edition (use the build tools from Microsoft)

2. Finally, install Python from https://python.org/ (make sure to check the box to install pip)

After you’ve installed the required software, open an MSYS2 terminal, change to a good location to build, and clone SciPy.

cd C:\Users\MyUser\Downloads
git clone https://github.com/scipy/scipy.git
cd scipy

Now we need to copy the openblas.a file that we’ve built earlier to the correct location. If your Python is installed somewhere like the following:

C:\Users\<user name>\AppData\Local\Programs\Python\Python36\python.exe

Then you’ll need to put the openblas.a file somewhere like the following:

C:\Users\<user name>\AppData\Local\Programs\Python\Python36\Lib

Adjust the location accordingly based on where python.exe is located. Now for a sanity check. Type the following and press enter.

gfortran

If you see an error with the above command, gfortran is not correctly installed. Go back to the “Building OpenBLAS” section and make sure that you have installed the correct tools.

Now install the dependencies that we need to build and test SciPy. It’s important that you specify the full path to the native Python interpreter so that the built-in MSYS2 Python will not be used. Attempting to build with the MSYS2 Python will not work correctly.

18 October 2017 Until NumPy 1.14 is officially released, we have to work with the latest development version of the NumPy repository. See the NumPy documentation….

/c/Users/<user name>/AppData/Local/Programs/Python/Python36/python.exe \
     -m pip install numpy cython pytest pytest-xdist pytest-faulthandler

Please note that this is a simpler procedure than what is used for the official binaries. Your binaries will only work with the latest NumPy (v1.14.0dev and higher). For building against older Numpy versions, see Building Against an Older Numpy Version. Make sure that you are in the same directory where setup.py is (you should be if you have not changed directories):

ls setup.py

Assuming that you have set up everything correctly, you should be ready to build. Run the following commands:

/c/Users/<user name>/AppData/Local/Programs/Python/Python36/python.exe \
     -m pip wheel -v -v -v .
/c/Users/<user name>/AppData/Local/Programs/Python/Python36/python.exe \
     runtests.py --mode full

Congratulatations, you’ve built SciPy!

Building Against an Older Numpy Version

If you want to build SciPy to work with an older numpy version, then you will need to replace the Numpy “distutils” folder with the folder from the latest numpy. The following powershell snippet can upgrade Numpy distutils while retaining an older Numpy ABI.

$NumpyDir = $((python -c 'import os; import numpy; print(os.path.dirname(numpy.__file__))') | Out-String).Trim()
rm -r -Force "$NumpyDir\distutils"
$tmpdir = New-TemporaryFile | %{ rm $_; mkdir $_ }
git clone -q --depth=1 -b master https://github.com/numpy/numpy.git $tmpdir
mv $tmpdir\numpy\distutils $NumpyDir

Additional Resources

As discussed in the overview, this document is not meant to provide extremely detailed explanations on how to build NumPy and SciPy on Windows. This is largely because currently, there is no single superior way to do so and because the process for building these libraries on Windows is under development. It is likely that any information will go out of date relatively soon. If you wish to receive more assistance, please reach out to the NumPy and SciPy mailing lists, which can be found here. There are many developers out there, working on this issue right now, and they would certainly be happy to help you out! Google is also a good resource, as there are many people out there who use NumPy and SciPy on Windows, so it would not be surprising if your question or problem has already been addressed.