The package processing pipelines consist of a series of steps, the handlers of which are determined as the pipeline is processed. This enables us to defer the build type of a package (e.g. a Python distribution) until after it has been downloaded and extracted.
All pipelines must start with an “init” step, which must be indempodent, and may normalize user-specified data on the package (e.g. normalize the URL).
The install pipeline is the primary pipeline of VEE, and is reponsible for installing the packages that are used in the default runtime. It’s steps are:
Normalizes user-specified data. This step MUST be indempodent, as it may run more than once in common usage. It may also set the package name/path.
The package is retrieved and placed at
This step should be idempotent (and so is assumed to cache its results and
may freely be called multiple times).
The package’s contents (“source”) are placed into
(which is usually a temporary directory).
An opportunity to check meta-data and determine self-described dependencies. This step may also set build and install names/paths.
The source is built into a build “artifact”.
Shared libraries are relocated to link against existant libraries (in case they are not already relocatable, and their dependencies are not in the same location in all environments).
Same as above.
Prepare the package for running in the development environment. Prepare any generated scripts, perhaps perform a build, and identify any environment variables to set in order to include this package in the runtime environment.
Names and Paths¶
There are a series of
*_name attribute of a
Package. They are
Requirement attributes, or self-determined on request via
There are a series of
*_path properties on a
Package. They usually
incorporate the corresponding name, but don’t have it. They are set from
It is very important that an API consumer only every assert the existence of
names or paths that they are about to use. This allows for the determination
of some of the names (especially
install_path) to be
deferred as long as possible so that they may use information revealed during
the earlier of the build pipeline.
*_name attributes exist only for the construction of paths; API consumers
should only ever use the
The location of the package (e.g. archive or git work tree) on disk. This must always be correct and never change. Therefore it can only derive from the requirement’s specification.
A (usually temporary) directory for building. This must not change once the package has been extracted.
The final location of a built artifact. May be
Noneif it cannot be determined. This must not change once installed.
Where within the build_path to install from. Good for selecting a sub directory that the package build itself into.
Where within the install_path to install into. Good for installing packages into the correct place within the standard tree.
Most packages are inspected to determine which style of build to use. Unless otherwise stated, they will also use an automatic install process as well. The base styles (in order of inspection) are:
vee-build.sh file exists, it will be sourced and is expected to build
the package. A few environment variables are passed to assist it:
The script may export a few environment variables to modify the install process:
python setup.py build¶
setup.py file exists, the package is assumed to be a standard
distutils-style Python package. The build process is to call:
python setup.py build
and the install process will be (essentially) to call:
python setup.py install --skip-build --single-version-externally-managed
*.dist-info directory exists, the package is
assumed to be a prepared Python package (an Egg or Wheel, respectively), and no
further build steps are taken. The install process will be modified to install
the package contents into
configure file exists, it will be executed and passed the install path:
This continues onto the next step...
Makefile file exists (which may have been constructed by running
make will be called.
Unless overridden (either by the package type, or the discovered build type (e.g. Python packages have their own install process)), the contents of the build path are copied to the install path, like:
shutils.copytree( os.path.join(pkg.build_path, pkg.build_subdir)), os.path.join(pkg.install_path, pkg.install_prefix)) )
--hard-link flag indicates that the build and install should
be hard-linked, instead of copied. This results in massive time and space
savings, but requires the packages to be well behaved.
Since we cannot trust that the standard
make; make install pattern will
actually install into a prefix provided to
./configure, we do not run
--make-install flag signals that it is safe to do so.
python setup.py install¶
Instead of running
python setup.py install, we break it into
python setup.py build and
python setup.py install --skip-build.
Some packages may not like this much.