Artifact Definition Reference

Artifact definitions are required to build artifact with Fatbuildr. Artifact definition is basically a set of artifact metadata (ex: versions, checksums, releases, etc) and build rules (ex: packaging code).

This sections explains how to write these artifact definitions.

Artifact Repository

While Fatbuildr can naturally be used to build only one artifact, it is also designed to help maintaining a full set of artifacts in multiple formats in a consistent manner. In this case, it is recommended to maintain all artifacts definitions in one global directory designated as the artifacts repository.

In Fatbuildr, artefacts definitions are designed to be composed of mostly small text files. This makes source code management systems (such as Git) especially appropriate to track changes in artifacts repositories.

In this artifacts repository, it is recommended to create one subdirectory per artifact. This subdirectory is intended to fully contain the definition of this artifact. For example, to maintain artifact definitions of foo, bar and baz artifacts, create the following layout of subdirectories:

repository
├── foo
├── bar
└── baz

While this setup in recommended in Fatbuildr artifact repositories, it is just a convention to simplify usage of fatbuildrctl command. This layout is not enforced. You can adopt the layout of your choice, you may just have to provide fatbuildrctl base directory and subdirectory options more often.

Artifact Definition

An artifact definition directory are expected to contain multiple files and directories. These are described in the following subsections.

Meta

An artifact definition directory must contain a definition file formatted in YAML and named artifact.yml.

As an alternative, the file can also be named artifact.yaml, depending upon your preference. The filename meta.yml is also supported for historical reason and it is deprecated.

This file can contain the following parameters:

source

Remote URL of the software source archive. This URL is used to download the software source. The value can use {{ version }} placeholder. This is dynamically replaced by Fatbuildr with the target source version number (ex: 1.2.3). This parameter is required as soon as version, versions or derivatives is defined.

Example

source: https://download.software.org/foo-{{ version }}.tar.xz

Fatbuildr natively supports all compressed tarballs (ex: *.tar.gz, *.tar.xz) and zip files.

sources

This is an alternative to source parameter for artifacts with multiple sources. The value must be a hash with sources identifiers as keys and remote URL of the software source tarball as values. The first source identifier is considered as the main source of the artifact and must match the artifact name.

Example

sources:
  foo: https://download.software.org/foo-{{ version }}.tar.xz
  bar: https://download.software.org/bar-{{ version }}.tar.xz

Fatbuildr downloads the source archives targeted by these URL and fills a cache. As long as the archive is available in the cache (typically starting from the 2^nd build), Fatbuildr does not try to access the URL anymore. This notably makes Fatbuildr resilient to resources removed on remote servers and saves network bandwidth.

version

A source version number (ex: 1.2.3). This is the source version used when building the artifact, as released by upstream developers. This straightforward parameter can be used for mono-source artifacts when the artifact is only intended to be built for default main derivative. For multi-sources artifacts, please refer to versions parameter. For artifacts targeting other derivatives, please refer to derivatives parameter.

Example

version: 1.2.3

versions

This is an alternative to version parameter for artifacts with multiple sources for default main derivative. For artifacts targeting other derivatives, please refer to derivatives key. The value must be a hash with sources identifiers as keys and target source version numbers as values.

Example

versions:
  foo: 1.2.3
  bar: 4.5.6

derivatives

This is an alternative to version and versions parameters for artifacts with multiple derivatives. The value must be a hash whose keys are the targeted derivatives names. The format of values depends on the number of artifact sources. For mono-source artifacts, the values are the version numbers targeted for the derivative. For artifacts with multiple sources, the values are hashes with source identifier as keys and the version number for this source in the associated derivative as values.

Example for mono-source artifact

derivatives:
  foo2: 2.1.0
  foo1: 1.3.5

Example for artifact with multiple sources

derivatives:
  foo2:
    foo: 2.1.0
    bar: 4.5.6
  foo1:
    foo: 1.3.5
    bar: 4.2.1

Either version, versions or derivatives parameters must be defined in most cases, unless users want to always specify the version at build time using the dedicated option of fatbuildrctl build command (ex: for sofware never released by upstream developers).

checksums

The checksums of software source archives. The value is a hash whose keys format of values depends on the number of artifact sources. For mono-source artifacts, the keys are the version numbers (ex: 1.2.3) and the values are hashes with hash function as keys (ex: sha256) and the expected hash function result as values. For artifacts with multiple sources, there is an additional top-level key for the source identifiers.

Example for mono-source artifact

checksums:
  1.2.3:
    sha256: 45b593694ffbea195d7d6830f5ea6e6e647d9a109f237e84cbaf40d028caddd4
  2.0.1:
    sha256: 7dce312cb278a3ec4288371f09ebb4717f1facf52793ed55147f14306273d9f4

Example for artifact with multiple sources

checksums:
  foo:
    1.3.5:
      sha256: 0b9f6f215ef228f1c5548c0299d35aece348d0f17d0f6819857c9e6b7ebf1549
    2.1.0:
      sha256: 085bbca73f5452f6e782369596d86ff46c836a89dd8512b2e5d1a782cbe2aff3
  bar:
    4.2.1:
      sha256: d2d00481c19b8413a8267a07c098224e608a12bd7198c1c8b8fac148c2705880
    4.5.6:
      sha256: 56a36126dfa93df6617077da842b869d60834aa09c84335be989295db607a086

Then, one parameter must be declared per format supported for this artifact. If one format is not declared in this file, Fatbuildr refuses to build the artifact in this missing format.

The formats parameters contain a hash of parameters. Each format supports a different set of parameters:

deb

release: The release number of deb package, suffixed to the source version number to compose the package number. This parameter is required. Ex: 1 or 2+beta1.

rpm

release: The release number of deb package, suffixed with the distribution name to the source version number to compose the package number. This parameter is required. Ex: 1 or 2+beta1.
buildargs: The build arguments provided to rpmbuild for the spec file. This parameter is optional.

osi

The OSI format does support any additional parameter. It must be declared with an empty hash ({}).

Example

rpm:
  release: 1
  buildargs: --with mysql
deb:
  release: 1
osi: {}

For reference, here are some examples of full artifact definition files artifact.yml:

Artifact (for main derivative only) in RPM format:

source: https://download.software.org/foo-{{ version }}.tar.xz
version: 1.2.3
checksums:
  1.2.3:
    sha256: 4355a46b19d348dc2f57c046f8ef63d4538ebb936000f3c9ee954a27460dd865
rpm:
  release: 1

Artifact with multiple sources foo and bar (for main derivative only) in deb format:

sources:
  foo: https://download.software.org/foo-{{ version }}.tar.xz
  bar: https://download.software.org/bar-{{ version }}.tar.xz
versions:
  foo: 1.2.3
  bar: 4.5.6
checksums:
  foo:
    1.2.3:
      sha256: 45b593694ffbea195d7d6830f5ea6e6e647d9a109f237e84cbaf40d028caddd4
  bar:
    4.5.6:
      sha256: 56a36126dfa93df6617077da842b869d60834aa09c84335be989295db607a086
deb:
  release: 1

Artifact with two versions for foo1 and foo2 derivatives in all supported formats:

source: https://download.software.org/foo-{{ version }}.tar.xz
derivatives:
  foo1: 1.2.3
  foo2: 2.0.1
checksums:
  1.2.3:
    sha256: 45b593694ffbea195d7d6830f5ea6e6e647d9a109f237e84cbaf40d028caddd4
  2.0.1:
    sha256: 7dce312cb278a3ec4288371f09ebb4717f1facf52793ed55147f14306273d9f4
rpm:
  release: 1
deb:
  release: 1
osi: {}

Artifact with multiple sources foo and bar targeting two derivatives foo1 and foo2 in all supported formats:

sources:
  foo: https://download.software.org/foo-{{ version }}.tar.xz
  bar: https://download.software.org/bar-{{ version }}.tar.xz
derivatives:
  foo2:
    foo: 2.1.0
    bar: 4.5.6
  foo1:
    foo: 1.3.5
    bar: 4.2.1
checksums:
  foo:
    1.3.5:
      sha256: 0b9f6f215ef228f1c5548c0299d35aece348d0f17d0f6819857c9e6b7ebf1549
    2.1.0:
      sha256: 085bbca73f5452f6e782369596d86ff46c836a89dd8512b2e5d1a782cbe2aff3
  bar:
    4.2.1:
      sha256: d2d00481c19b8413a8267a07c098224e608a12bd7198c1c8b8fac148c2705880
    4.5.6:
      sha256: 56a36126dfa93df6617077da842b869d60834aa09c84335be989295db607a086
rpm:
  release: 1
deb:
  release: 1
osi: {}

Deb packages

For Deb packages, packaging code must be located in deb/ subdirectory of artifact definition directory. The content of this directory is basically the content of the debian/ subdirectory of Deb source package, with some notable exceptions.

When the artifact archives contains an upstream debian/ directory, it is automatically removed by Fatbuildr during Deb packages builds. It is replaced by the debian/ directory generated by Fatbuildr based on the content of deb/ subdirectory of artifact definition.

The changelog file is not required. Fatbuildr generates this file dynamically at build time. If the artifact is already published for the targeted build distribution. If the changelog file is present in deb subdirectory, it is replaced by the file generated at build time.

The patches directory is not required. If patches are present in artifact definition directory or if prescript produces a patch, Fatbuildr generates the patches directory accordingly. If patches directory already exists in deb subdirectory, it is replaced by the generated directory.

All files whose name is suffixed by .j2 is considered as a Jinja2 template and is processed to generate the same file without the suffix. For example, deb/control.j2 generates deb/control.

The following variables are available in templates:

version

the artifact version, as an ArtifactVersion object. It has the following attributes:

version.main: The upstream version of the software. For example, for package version number 1.2-3.deb11, version.main is 1.2.
version.release: The release number of the package. For example, for package version number 1.2-3.deb11, version.release is 3.
version.major: The major component of the main version. For example, for package version number 1.2-3.deb11, version.major is 1.
version.full: The full version number of the package. For example, for package version number 1.2-3.deb11, version.full is 1.2-3.deb11.
version.dist: The distribution tag of the package, as defined in build pipelines for the targeted distribution. For example, for package version number 1.2-3.deb11, version.dist is deb11.

RPM packages

For RPM packages, a rpm/ subdirectory must be present in artifact definition directory. It must contain a spec file named <artifact>.spec where <artefact> is the name of the artifact. For example, the file rpm/foo.spec is expected to build RPM packages for artifact foo.

The spec file is processed as a Jinja2 template. The following variables are available in the template:

pkg

The artifact ArtifactBuildRpm object. It has many attributes such as:

distribution: The name of targeted RPM distribution (ex: el8)
derivative: The name of targeted RPM distribution derivative
architectures: The list of targeted CPU architectures for the build
version: The artifact version, as an ArtifactVersion object
env_name: The name of the build environment associated to the targeted RPM distribution (ex: rocky-8)
tarball_url: The full URL to the upstream tarball (optional)
tarball_filename: The filename of the uptream tarball (optional)

version

The upstream version number of the software. For example, for package version number 1.2-3.el8, version is 1.2.

This is a short alias for pkg.version.main.

release

The release number of the artifact. For example, for package version number 1.2-3.el8, release is 3.el8.

This is a short alias for pkg.version.fullrelease.

sources

The Source* tags to declare the sources input for the RPM packages (ex: Source0: foo-1.2.tar.gz).

prep_sources

The %setup macros to declare in the %prep section of the spec file.

The source and prep_sources variables are dynamically generated by Fatbuildr at build time and handles proper declaration of optional supplementary tarballs generated by artifact prescript. This provides solution to write and maintain generic spec file compatible with all prescripts.

patches: The Patch* to declare the upstream source patch queue. It is generated dynamically by Fatbuildr based on optional set of artifact patches and optional patch generated by artifact prescript.
prep_patches: The %autopatch macro to declare in the %prep section of the spec file.

The patches and prep_patches variables are dynamically generated by Fatbuildr at build time including the optional set of artifact patches and the optional patch generated by artifact prescript. This provides a solution to write and maintain generic spec files compatible with all patchs queues and all prescripts.

prep: Shorthand for prep_sources and prep_patches successively.
changelog: The %changelog section of the spec file dynamically generated by Fatbuildr at build time by concatenating the changelog of the same artifact currently available in managed repository for the targeted distribution, or none if absent, and the changelog entry generated with current build metadata.

The rpm/ subdirectory can also contain plain files. These plain files are automatically declared as additional sources (eg. Source1, Source2, etc) in the spec file for the source RPM by the sources variable.

The additional plain files can be manipulated in the %install and %files sections with %{_sourcedir} macro. For example, use %{_sourcedir}/foo for the plain file foo in the rpm/ subdirectory. Usage of the traditional %{SOURCE1}, … %{SOURCEn} macros are not recommended as other sources can be added dynamically at build time by Fatbuildr (ex: multi-sources packages, prescript tarballs, etc) and the source offset can be shifted.

OSI images

For mkosi OS images, an osi/ subdirectory must be present in artifact definition directory. It must contain a mkosi default settings file named <artifact>.mkosi where <artefact> is the name of the artifact. For example, the file osi/foo.mkosi is expected to build OS images for artifact foo.

The user-provided mkosi default settings file is used by Fatbuildr without modification. Some mkosi parameters are ignored in the file, as they are overriden at build time in mkosi command line. This applies to the following parameters:

OutputDirectory
ImageId
ImageVersion
Checksum

Renaming Rules

Artifact definition directory can contain a file named rename.j2 with renaming rules. These renaming rules are applied at build time before all other operations on the artifact definition directory. This feature can be useful to maintain packaging files with generic names that are renamed depending on artifact version.

The rename.j2 is a plain text file formatted with one renaming rule per line. Empty lines are ignored.

Renaming rule are formatted with the relative path to the source file within the artifact definition directory, a space separator, and the relative path to the destination file.

The file is processed as a Jinja2 template. The following variables are available in the template:

version

the artifact version, as an ArtifactVersion object. It has the following attributes:

version.main: The upstream version of the software. For example, for package version number 1.2-3.deb11, version.main is 1.2.
version.release: The release number of the package. For example, for package version number 1.2-3.deb11, version.release is 3.
version.major: The major component of the main version. For example, for package version number 1.2-3.deb11, version.major is 1.
version.full: The full version number of the package. For example, for package version number 1.2-3.deb11, version.full is 1.2-3.deb11.
version.dist: The distribution tag of the package, as defined in build pipelines for the targeted distribution. For example, for package version number 1.2-3.deb11, version.dist is deb11.

Example content of rename.j2 renaming rules file:

deb/foo-NM.install deb/foo-{{ version.main }}.install
deb/bar-N.install deb/bar-{{ version.major }}.install

Patches

Users can develop patches to apply on external upstream sources, for additional bug fixes or features.

Artifact patches are used only for Deb and RPM packages artifact formats.

These patches must be saved in the patches/<version>/ subdirectory of the artifact definition directory, where <version> is the uptream version of the software. For example, patches for foo version 1.2 must be saved into patches/1.2/ subdirectory of foo artifact definition directory. This layout gives the possibility to maintain patches for multiple versions.

It is also possible to write and save generic patches in patches/generic/ subdirectory of the artifact definition directory. These patches are applied for all versions of the artifact. This can be useful for example to create an additional file in artifact source tree, without need to maintain the same patch for multiple versions.

The patches are assembled into a serie by Fatbuildr and integrated in the source packages (SRPM or dsc). They are applied by respective build tools at the beginning of the build phase.

Generic patches are applied before version specific patches. When multiples patches are present in a subdirectory, they are applied in lexicographic order.

Even if it is not a requirement, it is recommended to fill header of patches files with metadata in deb822 format, following Debian DEP-3 patch tagging guidelines. In particular, these fields are particulary useful to help maintainance of patches over time:

Description: This […] field contains at least a short description on the first line, […] it allows for a more verbose explanation of the patch and its history. This field should explain why the patch is vendor-specific (ex: branding patch) when that is the case. If the patch has been submitted upstream but has been rejected, the description should also document why it’s kept and what were the reasons for the reject.

Last-Update: This field can be used to record the date when the meta-information was last updated. It should use the ISO date format YYYY-MM-DD.

Forwarded: Any value other than "no" or "not-needed" means that the patch has been forwarded upstream. Ideally the value is an URL proving that it has been forwarded and where one can find more information about its inclusion status. If the field is missing, its implicit value is "yes" if the "Bug" field is present, otherwise it’s "no". The field is really required only if the patch is vendor specific, in that case its value should be "not-needed" to indicate that the patch must not be forwarded upstream (whereas "no" simply means that it has not yet been done).

Author: This field can be used to record the name and email of the patch author (ex: "John Bear <foo@example.com>"). […] It’s also a good idea to add this contact information when the patch needs to be maintained over time because it has very little chance of being integrated upstream. This field can be used multiple times if several people authored the patch.

Source: Debian DEP-3

Additionally to these standard fields, Fatbuildr supports the following custom fields:

Distributions: This field can be used to restrict patch application to specific distributions. For example, the value Distributions: el8 el9 makes the patch apply during builds targeting el8 and el9 distributions only.
Formats: This field can be used to restrict patch application to specific artifact format. For example, the value Formats: deb makes the patch apply during builds of deb packages only.
Template: When set to yes, Fatbuildr considers this patch as a Jinja2 template to process. The following variables are available in the template:

version

the artifact version, as an ArtifactVersion object. It has the following attributes:

version.main

The upstream version of the software. For example, for package version number 1.2-3.deb11, version.main is 1.2.

version.release

The release number of the package. For example, for package version number 1.2-3.deb11, version.release is 3.

version.major

The major component of the main version. For example, for package version number 1.2-3.deb11, version.major is 1.

version.full

The full version number of the package. For example, for package version number 1.2-3.deb11, version.full is 1.2-3.deb11.

version.dist

The distribution tag of the package, as defined in build pipelines for the targeted distribution. For example, for package version number 1.2-3.deb11, version.dist is deb11.

The fatbuildrctl patches command provides a convenient way to maintain artifact patches subdirectory using Git version control system. It manages the files names, patches metadata and ensure patches are properly formatted without offset or context error.

With this command, the git commit messages are significant. The first line of the message is the resulting filename of the patch in the artifact definition tree, without the index prefix (ex: 0001-) and the .patch suffix. Fatbuildr expects to find Debian DEP-3 metadata in the other lines of the commit message.

The Author field is automatically generated by fatbuildrctl patches when exporting patches based on the author of the corresponding Git commit. During patches import in Git repository history, the Author field is extracted from patch header to set the author of the commit.

The following field can also be defined in Git commit messages:

Generic: When set to yes, Fatbuildr exports this patch in the patches/generic/ subdirectory, instead of version specific subdirectory. This way, the patch is applied for all upstream versions of the artifact. In the other way, fatbuildrctl patches defines this field in commit messages for all patches imported from the patches/generic/ subdirectory.

Prescript

The artifact definition directory can contain a script file named pre.sh, referred as the artifact prescript. This prescript is actually executed by Fatbuildr at build time, before the actual artifact build, in the targeted build environment, inside the artifact source tree. For more details about the functional aspects of prescripts, please refer to prescript feature description.

Artifact prescripts are used only for Deb and RPM packages artifact formats.

The prescript must be a valid bash (Bourne Again SHell) script.

The #!/bin/bash shebang is not required, although it is recommended because it gives hint to text editor and other tools guessing the file format, typically for syntax highlighting.

The prescript is executed in the source tree with artifact patches applied, if present.

As opposed to the actual build phase, the prescript get access to external networks and potentially Internet if permitted by your network policy.

Some bash functions are available in prescripts:

DL

Download HTTP URL and save file. The first argument is the URL, the second argument is the relative path to the saved file in the source tree.

Exemple:

DL https://host/bootstrap.min.css assets/bootstrap.min.css

Prescript rules can be declared in comments:

#PRESCRIPT_DEPS: A space separated list of packages to install temporarily in the build environment before running the prescript. This list of packages is concatenated with the basic set of prescript dependencies declared in main Fatbuildr configuration file.

Fatbuildr supports installation of DNF modules as prescript dependencies when building RPM packages. The module names must be specified with module: prefix in the list of packages.

#PRESCRIPT_TARBALLS: A space separated list of subdirectories used to generate supplementary tarballs for the artifact.

When #PRESCRIPT_TARBALLS rule is not defined, Fatbuildr generates a patch with all modifications performed by the prescript after its execution. This patch is automatically integrated in the source package, after the potential assembled serie of artifact patches.

When #PRESCRIPT_TARBALLS rule is defined, Fatbuildr generates supplementary tarballs with the content of the provided subdirectories, after prescript execution. One supplementary tarball is generated per subdirectory. These supplementary tarballs are then used in combination with the main upstream source tarball to compose the initial source tree of the package.

Generally, supplementary tarballs should be preferred over the patch approach when the prescript modifications are quite large (over 1k lines of modification).

Supplementary tarballs are the only solution to apply binary blob modification on upstream source tarball, as it is not supported in standard patches format.

The subdirectories used for supplementary tarballs are actually renamed by Fatbuildr to get unique names. This renaming is performed to avoid conflict in packages repositories with previously generated supplementary tarballs and potentially different content. The unique names are composed of the build timestamp and part of the build task ID. Symlinks from the initial name to the unique name are installed by Fatbuildr with an additional patch integrated in the source package, then packaging code can rely on initial names without impact.

It is possible to define distribution and format specific values of prescript rules, with @distributions:<dists> and @formats:<formats> suffixes respectively. The matching distributions and formats are specified as a comma-separated list of values.

This distribution specific value takes precedence over the format specific value, which takes precedence over the generic value. Only the first matching rule is considered.

Exemple 1

#!/bin/bash

mkdir assets
DL https://host/bootstrap.min.css assets/bootstrap.min.css

In this example, the DL function is used to download CSS file on external host and save this file in a new assets/ subdirectory. As the #PRETARBALLS rule is not defined, a patch will be generated by Fatbuildr and integrated in source package patches serie to install the assets/ subdirectory with the CSS file.

Example 2

#!/bin/bash

#PRESCRIPT_DEPS@distributions:el8,el9 rpm-build-golang module:python:39
#PRESCRIPT_DEPS@format:deb dh-golang
#PRESCRIPT_DEPS golang
#PRESCRIPT_TARBALLS vendor

mkdir vendor
go mod vendor

In this example:

rpm-build-golang package and python:39 module are temporarily installed by Fatbuildr in build environments for el8 and el9 distributions.
dh-golang package is installed in build environments for deb format.
golang in other build environments.

Then, go command is used to do vendoring of all dependencies. After the execution, Fatbuildr will generate one supplementary tarball with the content of the vendor/ subdirectory.

Template filters

Additionally to Jinja2 built-in filters, Fatbuildr also provides some filters available to use in your templates:

Name Description

Name	Description
`gittag`	Replace in a string characters disallowed in Git tags. For reference, see `git check-ref-format` manpage. It is notably useful to transform a complex version number into a valid Git tag in tarball URL. Ex: `1.2.3~beta1` → `1.2.3-beta1`
`rpm_version`	Replace in a string illegal characters in RPM spec Version field. More specifically, it replaces `-` by `~`. Ex: `1.2-0rc1` → `1.2~0rc1`
`timestamp_rpmdate`	Transform an integer timestamp into a date formatted as expected for RPM spec changelog. Ex: `949547106` → `Thu Feb 03 2000`
`timestamp_iso`	Transform an integer timestamp into a date in ISO format. Ex: `949547106` → `2000-02-03 04:05:06`

gittag

Replace in a string characters disallowed in Git tags. For reference, see git check-ref-format manpage. It is notably useful to transform a complex version number into a valid Git tag in tarball URL.

Ex: 1.2.3~beta1 → 1.2.3-beta1

rpm_version

Replace in a string illegal characters in RPM spec Version field. More specifically, it replaces - by ~.

Ex: 1.2-0rc1 → 1.2~0rc1

timestamp_rpmdate

Transform an integer timestamp into a date formatted as expected for RPM spec changelog.

Ex: 949547106 → Thu Feb 03 2000

timestamp_iso

Transform an integer timestamp into a date in ISO format.

Ex: 949547106 → 2000-02-03 04:05:06