conda build Recipe Reference

Building a package requires a recipe. A recipe is flat directory which contains the following files:

  • meta.yaml (metadata file)
  • build.sh (Unix build script which is executed using bash)
  • bld.bat (Windows build script which is executed using cmd)
  • run_test.py (optional Python test file)
  • patches to the source (optional, see below)
  • other resources, which are not included in the source and cannot be generated by the build scripts.

The same recipe should be used to build a package on all platforms.

When building a package, the following steps are invoked:

  1. read the metadata
  2. download the source (into a cache)
  3. extract the source in a source directory
  4. apply the patches
  5. create a build environment (build dependencies are installed here)
  6. run the actual build script. The current working directory is the source directory with environment variables set. The build script installs into the build environment
  7. do some necessary post processing steps: shebang, rpath, etc.
  8. add conda metadata to the build environment
  9. package up the new files in the build environment into a conda package
  10. test the new conda package: - create a test environment with the package (and its dependencies) - run the test scripts

There are example recipes for many conda packages in the conda-recipes repo.

The conda skeleton command can help to make skeleton recipes for common repositories, such as PyPI.

The meta.yaml file

All the metadata in the recipe is specified in the meta.yaml file. All sections are optional except for package/name and package/version.

package:
  name: bsdiff4       # lower case name of package, may contain '-' but no spaces
  version: "1.1.4"    # version of package. Should use the PEP-386 verlib
                      # conventions. Note that YAML will interpret
                      # versions like 1.0 as floats, meaning that 1.0 will
                      # be the same as 1. To avoid this, always put the
                      # version in quotes, so that it will be interpreted
                      # as a string.

                      # The version cannot contain a dash '-' character.

source:
  # The source section specifies where the source code of the package is
  # coming from, it may be coming from a source tarball like:
  fn: bsdiff-1.1.14.tar.gz
  url: https://pypi.python.org/packages/source/b/bsdiff4/bsdiff4-1.1.4.tar.gz
  md5: 29f6089290505fc1a852e176bd276c43
  sha1: f0a2c9a30073449cfb7d171c57552f3109d93894
  sha256: 5a022ff4c1d1de87232b1c70bde50afbb98212fd246be4a867d8737173cf1f8f

  # or from git:
  # git_url can also be a relative path to the recipe directory
  git_url: git@github.com:ilanschnell/bsdiff4.git
  git_tag: 1.1.4

  # or from hg:
  hg_url: ssh://hg@bitbucket.org/ilanschnell/bsdiff4
  hg_tag: 1.1.4

  # or from svn:
  svn_url: https://github.com/ilanschnell/bsdiff
  svn_rev: 1.1.4
  svn_ignore_externals: yes # (defaults to no)

  # or from a local path:
  # if the path is relative it is taken relative to the recipe
  # directory. The source is copied to the work directory before building.
  path: ../src

  # Patches may optionally be applied to the source
  patches:
    - my.patch    # the patch file is expected to be found in the recipe

  #########################################################################
  # Note, the source section is optional. If you want to specify a source #
  # location locally, the easiest way is to not specify the source here,  #
  # but to just add something like                                        #
  #                                                                       #
  # cp -r $RECIPE_DIR/../src .                                            #
  # cd src                                                                #
  # ...                                                                   #
  #                                                                       #
  # in build.sh (and similarly in bld.bat). This assumes the source is    #
  # shipped alongside the recipe in src.                                  #
  #########################################################################

build:
  # The build number should be incremented for new builds of the same version
  number: 1       # (defaults to 0)
  string: abc     # (defaults to default conda build string plus the build
                  # number)
                  # The build string cannot contain a dash '-' character

  # Optional Python entry points
  entry_points:
    # This creates an entry point named bsdiff4 that calls
    # bsdiff4.cli.main_bsdiff4()
    - bsdiff4 = bsdiff4.cli:main_bsdiff4
    - bspatch4 = bsdiff4.cli:main_bspatch4

  # If osx_is_app is set, entry points will use python.app instead of
  # python in Mac OS X
  osx_is_app: yes # (defaults to no)

  # See the Features section below for more information on features

  # Defines what features a package has
  features:
    - feature1

  # Indicates that installing this package should enable (track) the given
  # features. It is generally a good idea to use a separate metapackage to
  # track a feature, which does not have the feature. See the Features
  # section below for more information about features.
  track_features:
    - feature2

  # Preserve the Python egg directory. This is needed for some packages
  # that use setuptools specific features.
  preserve_egg_dir: yes # (default no)

  # A list of globs for files that should not be linked (soft or
  # hard). They will always be copied.
  no_link:
    - bin/*.py # Don't link any .py files in bin/

  # Used instead of build.sh or bld.bat. For short build scripts, this can
  # be more convenient. You may need to use selectors (see below) to use
  # different scripts for different platforms.
  script: python setup.py install

  #########################################################################
  # binary_relocation, has_prefix_files, binary_has_prefix_files, and     #
  # detect_binary_files_with_prefix may be required to relocate files     #
  # from the build environment to the installation environment.  See      #
  # "Making Packages Relocatable" below.                                  #
  #########################################################################

  # Whether binary files should be made relocatable using install_name_tool
  # on OS X or patchelf on Linux.
  binary_relocation: false # (defaults to true)

  # Binary files may contain the build prefix and need it replaced with the
  # install prefix at installation time.  Conda can automatically identify
  # and register such files:
  detect_binary_files_with_prefix: true # (defaults to false)
  # or you may elect to specify such files individually:
  binary_has_prefix_files:
    - bin/binaryfile1
    - lib/binaryfile2

  # Text files (containing no NULL bytes) may contain the build prefix and
  # need it replaced with the install prefix at installation time.  Conda
  # will automatically register such files.  Binary files that contain the
  # build prefix are generally handled differently (see
  # binary_has_prefix_files or detect_binary_files_with_prefix), but there
  # may be cases where such a binary file needs to be treated as an
  # ordinary text file, in which case they need to be identified:
  has_prefix_files:
    - bin/file1
    - lib/file2

# the build and runtime requirements. Dependencies of these requirements
# are included automatically.
requirements:
  # Packages required to build the package. python and numpy must be
  # listed explicitly if they are required.
  build:
    - python
  # Packages required to run the package. These are the dependencies that
  # will be installed automatically whenever the package is installed.
  run:
    - python
    - argparse # [py26]

test:
  # files which are copied from the recipe into the (temporary) test
  # directory which are needed during testing
  files:
    - test-data.txt
  # in addition to the run-time requirements, you can specify requirements
  # needed during testing. The run time requirements specified above are
  # included automatically.
  requires:
    - nose
  # commands we want to make sure they work, which are expected to get
  # installed by the package
  commands:
    - bsdiff4 -h
    - bspatch4 -h
  # Python imports
  imports:
    - bsdiff4

  # The script run_test.py will be run automatically if it is part of the
  # recipe

about:
  home: https://github.com/ilanschnell/bsdiff4
  license: BSD
  summary: binary diff and patch using the BSDIFF4-format

# If the app key is present, the package will be an app, meaning it will
# appear in the Anaconda launcher.
app:
  # The command that is called to launch the app
  entry: ipython notebook
  # Icon file contained in the recipe
  icon: icon_64x64.png
  summary: Summary of the package
  # If own_environment is true, installing the app through the launcher
  # will install into its own environment. The default is false.
  own_environment: true

Specifying versions in requirements

The requirements versions should follow the Conda match specification. See Package Match Specifications.

Preprocessing selectors

In addition, you can add selectors to any line, which are used as part of a preprocessing stage. Before the yaml file is read, each selector is evaluated, and if it is False, the line that it is on is removed. A selector is of the form # [<selector>] at the end of a line.

For example

source:
  url: http://path/to/unix/source    # [not win]
  url: http://path/to/windows/source # [win]

A selector is just a valid Python statement, that is executed. The following variables are defined. Unless otherwise stated, the variables are booleans.

linux True if the platform is Linux
linux32 True if the platform is Linux and the Python architecture is 32-bit
linux64 True if the platform is Linux and the Python architecture is 64-bit
armv6 True if the platform is Linux and the Python architecture is armv6l
osx True if the platform is OS X
unix True if the platform is Unix (OS X or Linux)
win True if the platform is Windows
win32 True if the platform is Windows and the Python architecture is 32-bit
win64 True if the platform is Windows and the Python architecture is 64-bit
py The Python version as a two digit string (like '27'). See also the CONDA_PY environment variable below.
py3k True if the Python major version is 3
py2k True if the Python major version is 2
py26 True if the Python version is 2.6
py27 True if the Python version is 2.7
py33 True if the Python version is 3.3
py34 True if the Python version is 3.4
np The NumPy version as a two digit string (like '17'). See also the CONDA_NPY environment variable below.

Because the selector is any valid Python expression, complicated logic is possible.

source:
  url: http://path/to/windows/source      # [win]
  url: http://path/to/python2/unix/source # [unix and py2k]
  url: http://path/to/python3/unix/source # [unix and py3k]

Note that the selectors delete only they line that they are on, so you may need to put the same selector on multiple lines.

source:
  url: http://path/to/windows/source     # [win]
  md5: 30fbf531409a18a48b1be249052e242a  # [win]
  url: http://path/to/unix/source        # [unix]
  md5: 88510902197cba0d1ab4791e0f41a66e  # [unix]

Environment variables set during the build process

The following environment variables are set, both on Unix (build.sh) and on Windows (bld.bat) during the build process:

ARCH Either 32 or 64, to specify whether the build is 32-bit or 64-bit. The value depends on the ARCH environment variable, and defaults to the architecture the interpreter running conda was compiled with.
CONDA_BUILD=1 Always set.
SRC_DIR Path to where source is unpacked (or cloned). If the source file is not a recognized file type (right now, .zip, .tar, .tar.bz2, .tar.xz, and .tar), this is a directory containing a copy of the source file.
PREFIX Build prefix where build script should install to.
RECIPE_DIR Directory of recipe.
PKG_NAME Name of the package being built.
PKG_VERSION Version of the package being built.
PKG_BUILDNUM Build number of the package being built.
PATH Prepended by the build prefix bin directory.
PYTHON Path to python executable in build prefix (note that python is only installed in the build prefix when it is listed as a build requirement).
PY3K 1 when Python 3 is installed in build prefix, else 0.
STDLIB_DIR Python standard library location
SP_DIR Python’s site-packages location
PY_VER Python version building against
CPU_COUNT Number of CPUs on the system, as reported by multiprocessing.cpu_count()

When building “unix-style” packages on Windows, which are then usually statically linked to executables, we do this in a special Library directory under the build prefix. The following environment variables are only defined in Windows:

LIBRARY_PREFIX <build prefix>\Library
LIBRARY_BIN <build prefix>\Library\bin
LIBRARY_INC <build prefix>\Library\include
LIBRARY_LIB <build prefix>\Library\lib
SCRIPTS <build prefix>\Scripts

On non-Windows (Linux and Mac OS X), we have:

PKG_CONFIG_PATH Path to pkgconfig directory.
HOME Standard $HOME environment variable.
INCLUDE_PATH <build prefix>/include
LIBRARY_PATH <build prefix>/lib

On Mac OS X, we have:

OSX_ARCH i386 or x86_64, depending on Python build
CFLAGS -arch flag.
CXXFLAGS Same as CFLAGS.
LDFLAGS Same as CFLAGS.
MACOSX_DEPLOYMENT_TARGET Same as the Anaconda Python. Currently 10.5.

On Linux, we have:

LD_RUN_PATH <build prefix>/lib

When the source is a git repository, the following variables are defined:

GIT_DESCRIBE_TAG string denoting the most recent tag from the current commit (based on the output of git describe --tags)
GIT_DESCRIBE_NUMBER string denoting the number of commits since the most recent tag
GIT_DESCRIBE_HASH the current commit short-hash as displayed from git describe --tags
GIT_BUILD_STR a string that joins GIT_DESCRIBE_NUMBER and GIT_DESCRIBE_HASH by an underscore.

These can be used in conjunction with templated meta.yaml files to set things like the build string based on the state of the git repository.

For example, here’s a meta.yaml that would work with these values. In this example, the recipe is included at the base directory of the git repository, so the git_url is ../:

package:
  name: mypkg
  version: {{ environ['GIT_DESCRIBE_TAG'] }}

build:
  number: {{ environ.get('GIT_DESCRIBE_NUMBER', 0) }}

  # Note that this will override the default build string with the Python
  # and NumPy versions
  string: {{ environ.get('GIT_BUILD_STR', '') }}

source:
  git_url: ../

All of the above environment variables are also set during the test process, except with the test prefix instead of the build prefix everywhere.

Note that build.sh is run with bash -x -e (the -x makes it echos each command that is run, and the -e makes it exit whenever a command in the script returns nonzero exit status). You can revert this in the script if you need to by using the set command.

Environment variables that affect the build process

CONDA_PY Should be 26, 27, 33, or 34. This is the Python version used to build the package.
CONDA_NPY Should be either 16 or 17. This is the NumPy version used to build the package.

Pre/Post link/unlink scripts

You can add scripts pre-link.sh, post-link.sh, or pre-unlink.sh (or .bat for Windows) to the recipe, which will be run before the package is installed, after it is installed, and before it is removed, respectively. If these scripts exit nonzero the installation/removal will fail.

Environment variables are set in these scripts:

PREFIX The install prefix.
PKG_NAME The name of the package.
PKG_VERSION The version of the package.
PKG_BUILDNUM The build number of the package.

No output is shown from the build script, but it may write to $PREFIX/.messages.txt, which is shown after conda completes all actions.

Post-build version

In some cases, you may not know the version, build number, or build string of the package until after it is built. In this case, you can write files named __conda_version__.txt, __conda_buildnum__.txt, or __conda_buildstr__.txt to the source directory, and the contents of the file will be used as the version, build number, or build string, respectively (and the respective metadata from the meta.yaml will be ignored).

Features

Features are a way to track differences in two packages that have the same name and version. For example, a feature might indicate a specialized compiler or runtime, or a fork of a package. The canonical example of a feature is the mkl feature in Anaconda Accelerate. Packages that are compiled against MKL, such as NumPy, have the mkl feature set. The mkl metapackage has the mkl feature set in track_features, so that installing it installs the mkl feature (the fact that the name of this metapackage matches the name of the feature is a coincidence).

Features should be thought of as features of the environment the package is installed into, not the package itself. The reason is that when a feature is installed, conda will automatically change to a package with that feature if it exists, for instance, when the mkl feature is installed, regular numpy is removed and the numpy package with the mkl feature is installed. Enabling a feature does not install any packages that are not already installed, but it all future packages with that feature that are installed into that environment will be preferred.

Feature names are independent of package names—it is a coincidence that mkl is both the name of a package and the feature that it tracks.

To install a feature, install a package that tracks it. To remove a feature, use conda remove --features.

It’s a good idea to create a metapackage for track_features. If you add track_features to a package that also has versions without that feature, then the versions without that feature will never be selected, because conda will always add the feature when it is installed from the track_features specification if your package with the feature.

Instead, it is a good idea to create a separate metapackage. For instance, if you want to create some packages with the feature debug, you would create several packages with

build:
  features:
    - debug

and then create a special metapackage

package:
  # This name doesn't have to be the same as the feature, but can avoid confusion if it is
  name: debug
  # This need not relate to the version of any of the packages with the
  # feature. It is just a version for this metapackage.
  version: 1.0

build:
  track_features:
    - debug

Making Packages Relocatable

Often, the most difficult thing about building a conda package is making it relocatable. Relocatable means that the package can be installed into any prefix. Otherwise, the package would only be usable in the same environment in which it was built.

Conda build does the following things automatically to make packages relocatable:

  • Binary object files are converted to use relative paths using install_name_tool on Mac OS X and patchelf on Linux.
  • Any text file (containing no NULL bytes) containing the build prefix or the placeholder prefix /opt/anaconda1anaconda2anaconda3 is registered in the info/has_prefix file in the package metadata. When conda installs the package, any files in info/has_prefix will have the registered prefix replaced with the install prefix. See Package metadata for more information.
  • Any binary file containing the build prefix can automatically be registered in info/has_prefix using build/detect_binary_files_with_prefix in meta.yaml. Alternatively, individual binary files can be registered by listing them in build/binary_has_prefix_files in meta.yaml. The registered files will have their build prefix replaced with the install prefix at install time. This works by padding the install prefix with null terminators, such that the length of the binary file remains the same. The build prefix must therefore be long enough to accommodate any reasonable installation prefix. Whenever the build/binary_has_prefix_files list is not empty or build/detect_binary_files_with_prefix is set, conda will pad the build prefix (appending _placehold‘s to the end of the build directory name) to 80 characters.
  • There may be cases where conda identified a file as binary, but it needs to have the build prefix replaced as if it were text (no padding with null terminators). Such files can be listed in build/has_prefix_files in meta.yaml.