Go-Nydus snapshotter: A containerd snapshotter with capability of on-demand read

Nydus Snapshotter

Nydus-snapshotter is a non-core sub-project of containerd.

Pulling and unpacking OCI container image are time consuming when start a container. Nydus is a vendor-neutral project aiming at mitigating the problem. It designs a new container image oriented and optimized file system format with capability of on-demand read. For instructions on how to build nydus container image, please refer to nydusify conversion tool.

Building

Just invoke make and find output binary ./bin/containerd-nydus-grpc

make

Integrate Nydus-snapshotter into Containerd

Containerd provides a general mechanism to exploit different types of snapshotters. Please ensure your containerd's version is beyond 1.4.0. Add nydus as a proxy plugin into containerd's configuration file which may be located at /etc/containerd/config.toml.

# The `address` field specifies through which socket snapshotter and containerd communicate.
[proxy_plugins]
  [proxy_plugins.nydus]
    type = "snapshot"
    address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

Restart your containerd service making the change take effect. Assume that your node is systemd based, restart the service as below:

systemctl restart containerd

Get Nydusd Binary

Find a suitable nydusd release for you from nydus releases page.

nydusd-fusedev is FUSE userspace daemon handling linux kernel fuse requests from /dev/fuse frontend. nydusd-virtiofs is a virtiofs daemon handling guest kernel fuse requests.

Configure Nydus

Nydus is configured by a json file which is required now. Because nydus container images are likely stored in a registry, where auth has to be provided. Please follow instructions to configure nydus configure nydus making it work properly in your environment.

Start Nydus Snapshotter

Nydus-snapshotter is implemented as a proxy plugin (containerd-nydus-grpc) for containerd.

A example of starting nydus-snapshotter:

# `nydusd-path` is the path to nydusd binary
# `address` is the domain socket that you configured in containerd configuration file
# `root` is the path to nydus snapshotter
# `config-path` is the path to nydus configuration file
$ ./containerd-nydus-grpc \
    --config-path /etc/nydusd-config.json \
    --shared-daemon \
    --log-level info \
    --root /var/lib/containerd/io.containerd.snapshotter.v1.nydus \
    --cache-dir /var/lib/nydus/cache \
    --address /run/containerd/containerd-nydus-grpc.sock \
    --nydusd-path /usr/local/bin/nydusd \
    --nydusimg-path /usr/local/bin/nydus-image \
    --log-to-stdout

Validate Nydus-snapshotter Setup

Utilize containerd's ctr CLI command to validate if nydus-snapshotter is set up successfully.

$ ctr -a /run/containerd/containerd.sock plugin ls
TYPE                            ID                       PLATFORMS      STATUS
io.containerd.snapshotter.v1    nydus                    -              ok

Quickly Start Container with Lazy Pulling

Start Container on Node

Containerd can start container with specified snapshotter, so legacy method like nerdctl or ctr needs to specify the nydus snapshotter when start container. A CLI tool ctr-remote is alongside. Use nydus ctr-remote to pull nydus image or start container based on nydus image.

$ sudo ctr-remote image rpull ghcr.io/dragonflyoss/image-service/nginx:nydus-latest
fetching sha256:75002dfe... application/vnd.oci.image.manifest.v1+json
fetching sha256:5a42e21c... application/vnd.oci.image.config.v1+json
fetching sha256:eb1af2e1... application/vnd.oci.image.layer.v1.tar+gzip

# Start container by `ctr`
$ sudo ctr-remote run --snapshotter nydus ghcr.io/dragonflyoss/image-service/nginx:nydus-latest

# Start container by `nerdctl`
nerdctl --snapshotter nydus run ghcr.io/dragonflyoss/image-service/nginx:nydus-latest

Start Container in Kubernetes

NOTE: A potential drawback using CRI is that we can hardly specify snapshotter to nydus-snapshotter. So we have to change containerd's default snapshotter in its configuration file like below:

[plugins."io.containerd.grpc.v1.cri".containerd]
   snapshotter = "nydus"

Use crictl to debug starting container via Kubernetes CRI. Dry run steps of using ctrctl can be found in documents.

Community

Nydus aims to form a vendor-neutral opensource image distribution solution to all communities. Questions, bug reports, technical discussion, feature requests and contribution are always welcomed!

Join our Slack workspace

Comments

  • Add/build nydus-snapshotter container image
    Add/build nydus-snapshotter container image

    Jan 13, 2022

    So that users can easily deploy and run it.

    Reply
  • The blobs annotation in manifest should be deprecated
    The blobs annotation in manifest should be deprecated

    Jan 18, 2022

    Currently, nydus image puts an annotation to the manifest to track all referenced blobs in bootstrap: nydus

    This will cause the label kv size limitation to be exceeded in containerd when acceld/buildkit write a nydus manifest included a large number of blobs into the content store: containerd

    I noticed that the blobs annotation only be used for blob cache gc in nydus-snapshotter: nydus-snapshotter

    A feasible workaround is to use nydusd sock API to get the blobs in use, instead of using the list in manifest annotation, so we can remove the annotation from acceld/buildkit.

    Reply
  • Add end-to-end CI
    Add end-to-end CI

    Jan 18, 2022

    The current CI is missing e2e tests. We should run end-to-end tests to make sure change works.

    Reply
  • Golangci lint runs twice in github action
    Golangci lint runs twice in github action

    Jan 18, 2022

    There is no need to run twice there.

    Reply
  • nydus-snapshotter should look for nydusd/nydus-image in $PATH
    nydus-snapshotter should look for nydusd/nydus-image in $PATH

    Jan 18, 2022

    Instead of hardcoding it or just relying on a fixed parameter.

    enhancement 
    Reply
  • mod: update containerd and runc dependencies
    mod: update containerd and runc dependencies

    Jan 19, 2022

    To bring in upstream CVE fixes.

    Reply
  • action: build snapshotter image
    action: build snapshotter image

    Jan 18, 2022

    null

                                                                                                                                                                                                           
    Reply
  • action: add release workflow definition
    action: add release workflow definition

    Jan 18, 2022

    Thus to automatically build binary and upload output artifacts.

    Reply
  • doc: rename documentation folder to docs
    doc: rename documentation folder to docs

    Jan 11, 2022

    Docs is a more conventional name.

    Signed-off-by: Changwei Ge [email protected]

    Reply
  • actions: add go mod cache
    actions: add go mod cache

    Jan 19, 2022

    null

                                                                                                                                                                                                           
    Reply
  • improve makefile
    improve makefile

    Jan 18, 2022

    null

                                                                                                                                                                                                           
    Reply