Rust-Debrepbuild: debrep — APT repository tool for building and managing an APT repo

Debian Repository Builder

A simple utility for constructing and maintaining Debian repositories. Configuration of a repo is based on the directory hierarchy, and a TOML configuration file. Real world repos to demonstrate may be found bellow.

Key Features

  • Can import pre-built Debian archives from external sources
  • Can repackage pre-built Debian archives
  • Can build packages from various sources, using sbuild
  • Supports multiple components within a suite
  • Supports migrating packages between components
  • Processes Debian archives within the pool in parallel.
  • Generates distribution file archives in parallel
  • Generates Packages, Sources, Contents, and Release dist files.
  • Repos will be apt-file-compatible

Todo

  • Generate Appstream + Icon archives

Directory Structure

The root directory of a debrep-based repo will contain the following directories:

  • assets/: where files that need to be linked at build-time are stored
    • cache/: files which debrep downloads from external sources, and should be cached between runs
    • share/: files that can be shared across packages, and are specified in the TOML config
    • packages/: files which are automatically linked to the build before building
    • replace/${suite}/${component}/${arch}/package/files/: Repackage prepackaged archives
      • DEBIAN: control archive files to replace
      • data: data archive files to replace
  • build/${suite}/: debrep performs all builds within this directory.
    • Every file is linked / sourced here at build time.
    • After each successful build, files are moved into the repo.
  • debian/${suite}/: contains the debian configuration for each source package that needs one.
    • The directories within must have the same name as the source package they reference.
    • Each package directory contains the entire contents of the debian directory for that package.
  • metapackages/${suite/: place your metapackage.cfg equivs files in here.
    • On build, they'll be generated and placed into the repo.
  • record/${suite}/: keeps tabs on what source packages have been built
  • repo/: Contains the archive & associated dist and pool directories for each
  • suites/${suite}.toml: Configuration files for each repo to build.

Highly Parallel Distribution File Generation

Since this tool is written in Rust, one of the key focuses has been on making it do as much as it can in parallel, as fast as it can do it. It uses thread pools, parallel iterators, and state machines to achieve that goal. Each component of a suit; each architecture in those components; and each package in those architectures are all processed in parallel. Data from each archive is also processed in parallel, and the final stage of processing that data into information and writing it to various compressed archives is done in parallel as well. Our tool should be fast with large archives.

Source Building Support

Packages can be generated from sources so long as you provide the debian files necessary -- either by using existing debian files in the upstream archive or git repository, or by providing your own through a variety of means.

Components Support

Managing components are supported by this utility! There's currently a default_component variable for the config, which will designate where packages will be stored by default. The migrate subcommand can be used to move packages between components. After moving packages, the dist files will be re-generated.

Contents Generation

Tools like apt-file require the the repository stores Contents archives, which it will download and read from to find which packages contain what files in a repository. This tool will process and generate these files in parallel as it is also processing the Packages archives.

Repo Structure

This is what you can expect to see after a successful build. You may sync the dists and pool directories to your package server to make your repository accessible to other machines.

repo/
  dists/
    cosmic/
    bionic/
      Contents-amd64
      Contents-amd64.gz
      Contents-amd64.xz
      InRelease
      proposed/
        binary-amd64/
          Packages
          Packages.gz
          Packages.xz
          Release
      main/
        binary-amd64/
          Packages
          Packages.gz
          Packages.xz
          Release
        source/
          Sources
          Sources.gz
          Sources.xz
        Release
        Release.gpg
  pool/
    cosmic/
    bionic/
      proposed/
        binary-amd64/
          p/
            package/
              package...
      main/
        binary-amd64/
          p/
            package/
              package_version_amd64.buildinfo
              package_version_amd64.changes
              package_version_amd64.deb
              package-dbgsym_version_amd64.ddeb
        source/
          p/
            package/
              package_version.dsc
              package_version.tar.xz

Usage

Create / update a Debian repository

debrep build [ -f | --force ]
debrep build packages <PACKAGES>... [ -f | --force ]
debrep build pool
debrep build dist

Migrate packages between components

debrep migrate package1 package2 pacakge3 --from proposed --to main

Clean up old packages

debrep clean

Remove packages

debrep remove <PACKAGES>...

Comments

  • Incremental update of Packages and Content files
    Incremental update of Packages and Content files

    Oct 22, 2018

    [ ] Support capability to determine what packages were changed between the last dists update [ ] Insert details into the dists file at the correct location, rather than regenerating them from scratch

    enhancement help wanted bite-sized 
    Reply
  • Support quilt patches in custom debian entries
    Support quilt patches in custom debian entries

    Oct 22, 2018

    Simply run quilt push until no more patches can be applied, if the debian details are using quilt.

    enhancement good first issue bite-sized 
    Reply
  • Improve apt mirroring support
    Improve apt mirroring support

    Jan 18, 2019

    It should be possible to:

    • Specify the base URL of a repo
    • Specify the suite to pull from
    • Fetch packages from the packages list(s) for that suite

    The implementation belongs in apt-fetcher.

    enhancement 
    Reply
  • Git-based Continuous Packaging Integration
    Git-based Continuous Packaging Integration

    Mar 5, 2019

    • [ ] Provide a GitHub / GitLab repo, or a list of individual repos
    • [ ] Provide a destination repo which we can compare against / submit to
    • [ ] Fetch each repo and their branches to the "CPI" system
    • [ ] Watch for new branches, and updated branches
    • [ ] Generate a list of repos that need to be built
    • [ ] Generate packages, and submit packages
    enhancement 
    Reply
  • Optional Launchpad Integration
    Optional Launchpad Integration

    Mar 5, 2019

    The CPI will require launchpad support to fetch package versions / submit packages.

    enhancement bite-sized 
    Reply
  • Run `cargo update` and fix some clippy lints
    Run `cargo update` and fix some clippy lints

    Jul 2, 2020

                                                                                                                                                                                                           
    Reply
  • Filter duplicate package entries from repo info archives
    Filter duplicate package entries from repo info archives

    Aug 14, 2018

    Closes #19

                                                                                                                                                                                                           
    Reply
  • Sync packages from third party repos
    Sync packages from third party repos

    Oct 5, 2018

    • Refactoring the config module to separate the source, repo, and direct structures.
    • Added a new repos field for handling repo-based sources.
    • Added a new repos module to repo::download for fetching sources from a repo.
    • Split the web crawling functionality out to a new url-scraper, url-crawler, and apt-repo-crawler crate

    Example

    [[repos]]
    repo = "http://ppa.launchpad.net/js-reynaud/kicad-5/ubuntu/pool/main/"
    version = { is = "ubuntu18.04" }
    arch = { not = "i386" }
    
    [[repos]]
    repo = "http://ppa.launchpad.net/freecad-maintainers/freecad-stable/ubuntu/pool/main/"
    version = { is = "ubuntu18.04" }
    arch = { not = "i386" }
    
    # Peek GTK application
    [[repos]]
    repo = "http://ppa.launchpad.net/peek-developers/stable/ubuntu/pool/main/"
    version = { is = "ubuntu18.04" }
    arch = { not = "i386"}
    
    # Updated thunderbird packages, and firefox-esr
    [[repos]]
    repo = "http://ppa.launchpad.net/mozillateam/ppa/ubuntu/pool/main/"
    version = { is = "18.04" }
    arch = { is = "amd64|all" }
    
    Reply
  • Configurable default branch
    Configurable default branch

    Aug 7, 2018

    When adding a new package to a repo, we should support placing these packages in a branch other than "main", such as "proposed".

    enhancement 
    Reply
  • Update file locations for records, logs, and debian packages to support suites
    Update file locations for records, logs, and debian packages to support suites

    Oct 16, 2018

    Currently testing locally

    • Built source packages will now be recorded at /records/${suite}/package-name
    • Builds now occur at /build/${suite}/
    • Metapackages are now moved to /metapackages/${suite}/
    • Package logs will be retained for each suite at /logs/${suite}/package-name
    • Debian packaging data will now be separated at /debian/${suite}/package-name

    This will fix source builds on cosmic so that the CUDA packages will build on both bionic and cosmic separately.

    Reply
  • support multi-architecture cross-compiling
    support multi-architecture cross-compiling

    May 16, 2019

    hi @mmstick, first up thanks for a sweet tool for Debian packaging! :100:

    i'd like to use debrep for building Debian packages for a project PeachCloud :peach: :cloud: , which will run on (Raspberry Pi) ARM devices, so i'm keen to add cross-compilation support to deprep.

    here's a pull request that seems to work:

    • adds architectures field to suite config, which defaults to [amd64, i386]
    • runs sbuild for each architecture (in a iter.try_for_each)
    • architecture matches support all official Debian architectures: https://www.debian.org/ports/

    here's a demo using this to build a basic Rust package for amd64, i386, arm64, and armhf: https://github.com/peachcloud/peach-packages.

    happy to implement this feature however you want, let me know if anything is not okay, or if you want something done differently, or if i'm missing a better way to do this, etc. i kept the changes pretty minimal to get it working and start the conversation, but for example at the moment this compiles every architecture in series and then writes the record object without any architecture information, maybe you have a better idea.

    cheers! :smiley_cat:

    Reply
  • Fix VS Code packaging (missing fields in Packages archive)
    Fix VS Code packaging (missing fields in Packages archive)

    Oct 3, 2018

    There are some optional fields that weren't being added to the Packages archive. This would cause packages that have those fields to give repeated update messages.

    Reply