Jump to content

Aptly

From Wikitech

Aptly is a simple debian repository management tool. It can be used for maintaining a simple, per-project debian repository.

Setting up

You need a packages server that'll host your packages, and then apply a role to all the instances in your project.

Server

Just apply the role role::aptly::server to any Debian Jessie (or Ubuntu Trusty, although that is not recommended) instance in your project. Since the server doesn't need much resources, it's ok to put this on a multi-purpose machine.

Client

You now need to point all the instances in your project to the aptly server instance. You can do this by editing your Hiera config (https://wikitech.wikimedia.org/wiki/Hiera:$projectname) to include: 

classes:  
  - role::aptly::client

By default, they look for an instance named $projectname-packages (eg. tools-packages, ores-packages, etc). You can point his to any instance by setting:

role::aptly::client::servername to the name of the instance with role::aptly::server applied.

Inspecting the content of the repo

Some examples on how to check the current content of the repos. 

user@server:~$ sudo aptly repo list
List of local repos:
 * [jessie-tools] (packages: 65)
 * [precise-tools] (packages: 102)
 * [stretch-tools]: Toolforge packages for Debian 9 (Stretch) (packages: 57)
 * [stretch-toolsbeta] (packages: 53)
 * [trusty-tools] (packages: 148)

To get more information about local repository, run `aptly repo show <name>`.

List packages in a repo: 

user@server:~$ sudo aptly repo show --with-packages=true jessie-tools
Name: jessie-tools
Comment: 
Default Distribution: jessie-tools
Default Component: main
Number of packages: 65
Packages:
  jobutils_1.35_all
  toollabs-webservice_0.43_all
  tools-manifest_0.16_all
  docker-engine_1.12.6-0~debian-jessie_amd64
  kubernetes-client_1.4.6-6_amd64
  kubernetes-master_1.4.6-6_amd64
  kubernetes-node_1.4.6-6_amd64
  misctools_1.33_amd64
  prometheus-blackbox-exporter_0.2.0_amd64
[...]

Adding Packages

Adding packages is fairly easy, once you've built them.

  1. ssh on to the aptly server instance
  2. Build your package
  3. do sudo aptly repo add <repo-name> <path-to-package>
  4. do sudo aptly publish --skip-signing update <repo-name>
  5. Tada!

<repo-name> is <distribution>-<projectname>, where distribution is the Debian distribution you are publishing packages for (stretch, jessie, and trusty are currently supported), and <projectname> is just the name of your project!

The full functionality documented at aptly.info is available. By default aptly is restricted to users with root only.

From the command line, you can add the same package to multiple repos with something like this: 

user@tools-services-05:~$ for d in "bookworm" "bullseye" "buster" ; do for p in "tools" "toolsbeta" ; do sudo aptly repo add ${d}-${p} toolforge-webservice_0.103.7_all.deb ; done ; done
Loading packages...
[+] toolforge-webservice_0.103.7_all added
Loading packages...
[+] toolforge-webservice_0.103.7_all added
Loading packages...
[+] toolforge-webservice_0.103.7_all added
Loading packages...
[+] toolforge-webservice_0.103.7_all added
Loading packages...
[+] toolforge-webservice_0.103.7_all added
Loading packages...
[+] toolforge-webservice_0.103.7_all added
user@tools-services-05:~$ for d in "bookworm" "bullseye" "buster" ; do for p in "tools" "toolsbeta" ; do sudo aptly publish --skip-signing update ${d}-${p} ; done ; done
[..]

Adding a new repo

Adding a new repo is a matter of updating the file modules/role/manifests/aptly/server.pp.

This is running all the required individual commands, see modules/aptly/manifests/repo.pp.

Individual commands are, for example for a new Debian stable release: 

$ sudo aptly repo create -comment="Toolforge packages for Debian 9 (Stretch)" -component="main" -distribution="stretch-tools" stretch-tools

Local repo [stretch-tools]: Toolforge packages for Debian 9 (Stretch) successfully added.

The new repo is now created: 

$ sudo aptly repo show stretch-tools
Name: stretch-tools
Comment: Toolforge packages for Debian 9 (Stretch)
Default Distribution:
Default Component: main
Number of packages: 0

Note that running individual commands isn't persisting anything in puppet, so do this via puppet instead.

You then probably want to do something like:

$ sudo aptly publish --architectures="all,amd64" --skip-signing repo stretch-tools
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...

Local repo stretch-tools has been successfully published.
Please setup your webserver to serve directory '/srv/packages/public' with autoindexing.
Now you can add following line to apt sources:
  deb http://your-server/ stretch-tools main
Don't forget to add your GPG key to apt with apt-key.

You can also use `aptly serve` to publish your repositories over HTTP quickly.

FAQ

What happens when the instance hosting the package dies?

Packages become unavailable. There's no failover mechanism for this at the moment. You can backup your /srv/aptly directory to NFS / other means if you wish. We do this for Tools (backup at /data/project/.system/aptly/).

In general, we do not have a backup solution for labs that does not involve NFS (yet). This should change in the near future.

Oops, I added two different builds of a universal package

The key to noticing this is file already exists and is different when trying to publish. Just remove one of the debs and then add the remaining one back for the other dist target. 

$ sudo aptly publish --skip-signing update precise-tools
Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: error linking file to /srv/packages/public/pool/main/t/toollabs/jobutils_1.13_all.deb: file already exists and is different
$ sudo aptly repo remove precise-tools 'jobutils (1.13)'
Loading packages...
[-] jobutils_1.13_all removed
$ sudo aptly repo add precise-tools /srv/packages/pool/5d/da/jobutils_1.13_all.deb
Loading packages...
[+] jobutils_1.13_all added
$ sudo aptly publish --skip-signing update precise-tools
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
Cleaning up prefix "." components main...

Publish for local repo ./precise-tools [all, amd64] publishes {main: [precise-tools]} has been successfully updated.

Oops, I rm'd a file in the pool

Got trigger happy while trying to fix the different universal builds problem? Remove the metadata pointing the now removed file from the index. 

$ sudo aptly publish --skip-signing update precise-tools
Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: stat /srv/packages/pool/90/5c/tools-manifest_0.6_all.deb: no such file or directory
$ sudo aptly repo remove precise-tools 'tools-manifest (0.6)'
Loading packages...
[-] tools-manifest_0.6_all removed
[-] tools-manifest_0.6_source removed
$ sudo aptly publish --skip-signing update precise-tools
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
Cleaning up prefix "." components main...

Publish for local repo ./precise-tools [all, amd64] publishes {main: [precise-tools]} has been successfully updated.

"unable to find control.tar.gz part in package <path-to-package>"

This can be a problem on aptly versions before 1.2.0 (as of writing we have 0.9.7 in stretch, but 1.3.0 is in stretch-backports). You probably have control.tar.xz instead (you can check with ar t package.deb) see https://github.com/aptly-dev/aptly/issues/655#issuecomment-405911043

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Aptly&oldid=2197912"