Andrey Smirnov's Blog (devops)

aptly 0.7 - S3 publishing, complex queries

Andrey — Sat, 04 Oct 2014 20:57:38 GMT

aptly 0.8 has been released the day before yesterday. All news about aptly are now published in aptly blog: http://www.aptly.info/post/aptly-0-8/

aptly 0.7 - S3 publishing, complex queries

Andrey — Tue, 29 Jul 2014 17:22:04 GMT

aptly 0.7 has been released today. aptly is a Debian repository management tool, it allows to mirror remote repositories, create local package repositories, manage repositories snapshots and publish them back as Debian repository. aptly main idea is "owning your own repository": you can mix and match official repos, 3rd-party repositories, your own packages, creating your own stable/testing/whatever repositories, allowing reproducible package installations along with controlled upgrades. It is available for download as binary executables or from Debian repository:

deb http://repo.aptly.info/ squeeze main

When installing from repository, don't forget to import key used to sign the release:

$ gpg --keyserver keys.gnupg.net --recv-keys 2A194991
$ gpg -a --export 2A194991 | sudo apt-key add -

Aptly has new logo, soon I'm going to launch new website:

Most important new features are:

Publishing to Amazon S3

aptly can publish repositories directly to Amazon S3.

First, create new S3 bucket using AWS console. Let it be aptly-repo. Remember Amazon region you have used to create, I'll be using us-west-1 in this example. If you're going to have public repository, enable website hosting for that bucket.

Go to IAM page, create new user, save access key ID and secret access key and create bash script aws.creds.sh:

# Access Key ID:
# AKIAISHG7G3H8AWBCFG
# Secret Access Key:
# E7aujXChaMZwp3ghfk45+Zabd55

export AWS_ACCESS_KEY_ID="AKIAISHG7G3H8AWBCFG" AWS_SECRET_ACCESS_KEY="E7aujXChaMZwp3ghfk45+Zabd55"

In IAM console, attach new custom policy for that user:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1405592139000",
      "Effect": "Allow",
      "Action": [
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": [
        "arn:aws:s3:::aptly-repo/*", "arn:aws:s3:::aptly-repo"
      ]
    }
  ]
}

This user would have limited access only to the bucket you've created.

Now, configure aptly, edit configuration file ~/.aptly.conf and add key S3PublishEndpoints:

"S3PublishEndpoints": {
  "aptly-repo": {
    "region": "us-west-1",
    "bucket": "aptly-repo",
    "acl": "public-read"
  }
}

If you're going to have public repository, set acl to public-read, otherwise set acl to private. Now you're ready to do your first publish. For example, to publish snapshot my-snapshot to the mentioned bucket, run:

aptly publish snapshot my-snapshot s3:aptly-repo:

As with publishes to local filesystem, you can publish under prefix:

aptly publish snapshot my-snapshot s3:aptly-repo:debian/

All regular publish commands are supported: you can switch between snapshots (atomically), update published local repositories, drop published repos, etc. aptly would do its best to upload package files only once to package pool in S3.

In order to use published repository, for public repositories use regular HTTP protocol in /etc/apt/sources.list:

deb http://s3-us-west-1.amazonaws.com/aptly-repo wheezy main

For private repositories you would need special apt s3 transport, after installing transport you can use it like that:

deb s3://AWS_ACCESS_ID:[AWS_SECRET_KEY_IN_BRACKETS]@s3-us-west-1.amazonaws.com/aptly-repo wheezy main

Package Queries

Before 0.7, aptly supported only Debian dependency-like package queries. In version 0.7, complex queries were introduced. Query syntax matches reprepro query language, reference could be found in the docs. I'll give some examples.

Now you can filter mirrors to include only packages with limited priorities:

aptly mirror create -filter="Priority (required)" wheezy-required http://mirror.yandex.ru/debian/ wheezy main

Or download single packages and all its dependencies:

aptly mirror create -filter="nginx" -filter-with-deps wheezy-required http://mirror.yandex.ru/debian/ wheezy main

Pull packages with complex conditions:

aptly snapshot pull snapshot1 source snapshot2 '!Name (% *-dev), $Version (>= 3.5)'

Or remove packages based on query:

aptly repo remove local-repo 'Name (% http-*) | $Source (webserver)'

In the next version, package queries would be used to filter snapshots, search for packages in repos/snapshots and local repos, and do whole "world" package searching.

Other Features

aptly can now pull all matching packages with aptly snapshot pull command using flag -all-matches, e.g. one can pull subset of versions from 0.7 to 0.9:

aptly snapshot pull stable1 foo-snapsot stable2 'foo (>= 0.7), foo (<= 0.9)'

Download speed could be limited while mirroring using config option downloadSpeedLimit, so that aptly won't consume all bandwidth.

All Changes

Full ist of changes since 0.7:

direct publishing to Amazon S3 (#15)
support for new, powerful query language in many commands: aptly snapshot pull, aptly repo move, aptly repo copy, aptly repo import and aptly repo remove
bug fix: files from conflicting packages might override each other while publishing (#65)
partial mirrors: filter package lists when mirroring (#64)
new commands: mirrors, local repositories and snapshots can be renamed (#63)
new command: aptly mirror edit allows to change mirror filtering (#63)
download transfer rate could be limited either via configuration file parameter downloadSpeedLimit or with flag -download-limit for command aptly mirror update (#62)
new flag: -all-matches for aptly snapshot pull enables pulling of all matching packages (#70), thanks to Simon Aquino
when matching single package in aptly snapshot pull latest version would be pulled (#67), thanks to Simon Aquino
new flag: -sort for aptly snapshot list allows to change order of snapshots in the list (#73), thanks to Simon Aquino
bug fix: publish update fails on empty multi-component repo (#66)
bug fix: aptly snapshot pull might remove already pulled packages (#78)
bug fix: aptly package was missing bzip2 dependency (#84)
aptly binary packages are built with go1.3

aptly 0.6

Andrey — Sun, 08 Jun 2014 20:57:38 GMT

aptly 0.6 has been released on June, 7th. It is available for download as binary executables or from Debian repository:

deb http://repo.aptly.info/ squeeze main

When installing from repository, don't forget to import key used to sign the release:

$ gpg --keyserver keys.gnupg.net --recv-keys 2A194991
$ gpg -a --export 2A194991 | sudo apt-key add -

Most important new features are:

Multi-Component Repository Publishing

aptly is based on concept of list of packages. Snapshots, mirrors and local repositories are list of packages (more precisely, list of references to packages). When merging, pulling, copying or moving packages might move from one list into another. Component is a way to break down packages into groups, usually these groups make sense only in published repository. At the same time mapping from package to component is not universal, there's Debian way to group packages into main, contrib and non-free components, Ubuntu uses different schema of components, some 3rd party repositories use components in place of different distributions (like squeeze, wheezy etc.) or to separate stable and testing versions of software.

In order to keep aptly simple, I decided that there's no mapping from package to component and package lists internally aren't split by component. Each list (snapshot, mirror and local repository) is mono-component (actually there's no component at all). When publishing repository, several lists could be published as separate components.

By default, aptly mirrors all components from remote repository and merges them into one "single component". If we'd like to preserve package split by components, individual mirrors should be created for each component:

aptly mirror create wheezy-main http://ftp.ru.debian.org/debian/ wheezy main
aptly mirror create wheezy-contrib http://ftp.ru.debian.org/debian/ wheezy main
aptly mirror create wheezy-non-free http://ftp.ru.debian.org/debian/ wheezy non-free

aptly mirror list -raw | xargs -n 1 aptly mirror update

We can create snapshots from each of the mirrors:

aptly snapshot create wheezy-main-7.5 from mirror wheezy-main
aptly snapshot create wheezy-contrib-7.5 from mirror wheezy-contrib
aptly snapshot create wheezy-non-free-7.5 from mirror wheezy-non-free

And publish all snapshots as single repository preserving component structure (publishing distribution wheezy under prefix upstream):

aptly publish snapshot -component=main,contrib,non-free -distribution=wheezy wheezy-main-7.5 wheezy-contrib-7.5 wheezy-non-free-7.5  upstream

aptly is smart enough to figure out component names and distribution from the mirrors, so I can omit them (commas left to identify number of components):

aptly publish snapshot -component=,, wheezy-main-7.5 wheezy-contrib-7.5 wheezy-non-free-7.5 upstream

Of course we could do all regular aptly operations: merging snapshots, pulling packages, etc.

Handling Package Conflicts

Package in Debian universe is identified by triple (architecture, name, version). If two packages have the same (architecture, name, version) but different content, they are called conflicting packages. Debian guidelines prohibit including conflicting packages in repositories that could be used together (which could be present in one apt.sources file). Unfortunately, in real word there are conflicting packages, one such package has been reported in squeeze + security updates, another example was puppet repository which contains packages with the same triple but for different Debian distributions in several components.

Before 0.6, aptly would complain when it detects such conflicts and stop processing. In this version special handling has been added that considers packages with same (architecture, name, version) and different files as different package entires. There's one restriction though: you can't put packages with duplicate (architecture, name, version) into one list (one mirror, snapshot, local repo, published repository). This is in line with Debian guidelines that one repository shouldn't contain duplicate packages.

This feature works transparently when upgrading from older versions of aptly: conflicts would be just gone. In the background aptly would be updating references to packages when you update mirrors, create new snapshots, etc.

Empty Repository Publishing

Many people are using aptly to handle package repository from various automation tools, e.g. configuration management systems. For such usage it is convenient to create local repository (empty initially), publish it, and then add packages and update published repository.

Before 0.6, aptly would refuse to publish empty repositories. Now this is possible, but correct architecture list should be supplied when publishing (as aptly can't figure out architecture list automatically from package list). List of architectures can't be changed when published repository is updated, you would have drop published repository and create new one if required.

Merging Snapshots: 3rd Strategy

There's a feature in aptly that allows to merge two snapshots: this is useful to combine for example main repository and security updates or main repository and 3rd-party repository. With 0.6, three merge strategies are available:

for packages with same (architecture, name) package which comes from latest snapshot on the command line wins (default);
for packages with same (architecture, name) package with latest version wins (-latest);
all versions of packages are preserved (-no-remove, new in 0.6).

All Changes

Full list of changes in 0.6:

support for multi-component published repositories (#36)
handling duplicate packages with different content gracefully (#60)
repositories published by aptly now can be consumed by debian-installer (#61)
new flag: -no-remove for aptly snapshot merge to merge snapshots with all package versions preserved (#57)
publishing of empty snapshots/repositories is possible (#55)
aptly repo add now exists with 1 if any of files failed to add (#53)
bug fix: Package: line comes first in package metadata (#49)
bug fix: when command parsing fails, aptly returns exit code 2 (#52)
bug fix: pulling more than 128 packates at once (#53)
bug fix: aptly graph may get confused with package pull requests (#58)

aptly 0.5

Andrey — Thu, 24 Apr 2014 20:12:18 GMT

aptly 0.5 has been released today. It is available for download as binary executables or from Debian repository:

deb http://repo.aptly.info/ squeeze main

When installing from repository, don't forget to import key used to sign the release:

$ gpg --keyserver keys.gnupg.net --recv-keys 2A194991
$ gpg -a --export 2A194991 | sudo apt-key add -

Most important new features are:

Local Repository Publishing

Local repositories could be used in two ways:

test new versions of software
provide stable distribution of new versions

For the second case, it is best to create snapshots of local repositories and publish them. However, when testing out new versions, there isn't much sense in creating snapshot each time repository is updated. So aptly since version 0.5 supports direct publishing of repositories. Moreover, when local repository is updated, published repository could be updated as well in one step.

When local repository is created, default publishing options (distribution and component) could be specified, so that these options don't need to be specified when publishing:

aptly repo create -distribution=wheezy testing-wheezy
aptly repo add -remove-files testing-wheezy incoming/*.deb
aptly publish repo testing-wheezy
...
aptly repo add -remove-files testing-wheezy incoming/*.deb
aptly publish update wheezy

aptly 0.4

Andrey — Tue, 11 Mar 2014 12:14:03 GMT

aptly version 0.4 has been released today. Major feature in this version is local package repository management which allows to manage collection of your own packages, publish, take snapshots, mix with upstream repositories mirrors. Please download it or install from source, raise issues, disscuss in aptly-discuss group, follow me (@smira) to get information about updates.

Other features in 0.4 worth mentioning are: support for source packages for mirrors and local repositories, ability to delete unused package files and DB entries, and memory usage optimizations.

Full list of changes in this version:

local package repositories are supported
aptly supports mirroring remote repos with source packages and publishing repositories with sources
new command: aptly db cleanup to remove unreferenced DB entries and files
aptly peak memory usage has been reduced by factor of 3x
new flags: -keyring & -secret-keyring for aptly snapshot publish command
new config: downloadSourcePackages to enable source package downloading
new flag: -with-sources for aptly mirror create command
new config & flag: dependencyFollowSource & -dep-follow-source to follow Source: dependencies
new commands in aptly repo family: add, copy, create, drop, import, list, move, remove and show
command aptly snapshot create supports creation of snapshots from local repos
new flag`` -no-remove`` for aptly snapshot pull: don't remove other version of packages when pulling (e.g. keep old versions)
command aptly mirror create supports shorthand PPA url: ppa:user/project
new config: ppaDistributorID & ppaCodename to specify PPA url expansion rules
packages are printed in lists with underscores instead of dashes, e.g. pkg_1.3-3_amd64 instead of pkg-1.3-3-amd64

With addition of local package repositories, schema of aptly entities and transitions looks like that:

aptly 0.3

Andrey — Mon, 10 Feb 2014 19:20:40 GMT

Today I've released aptly version 0.3. It's the first version I would recommend for production usage. Please download it or install from source, raise issues, disscuss in aptly-discuss group, follow me (@smira) to get information about updates.

New features:

using aptly serve command you can quickly serve your published repositories over HTTP, aptly would even advise right settings for apt sources;
aptly checks signatures and verifies checksums for downloaded files while mirroring remote repositories, if you don't have key that was used to sign the mirror in your trusted GnuPG keychain, aptly would give some hints, some hints;
flat format of Debian repositories is now supported (e.g. OBS creates repositories in such format);
now you can drop mirrors and snapshots;
aptly can draw graph of relationships between your mirros, snapshots and published repositories;
bash completion is available for aptly, try it out, it's amazing!
aptly gained ability to create empty snapshot, it could be useful if you'd like to extract part of repository by pulling packages;
custom config location could be given with flag -config.

Nice picture (actually it's output of aptly graph command):

aptly 0.2, Moscow DevOps Meetup

Andrey — Wed, 29 Jan 2014 08:24:00 GMT

Two great things have happened recently:

I've released aptly version 0.2, which is alpha-quality.
I've presented aptly at DevOps Meetup Moscow (in Russian).

Slides from my talk about aptly can be downloaded in PDF (English version).

aptly: Debian repository management tool from Andrey Smirnov

Budapest Devops Meetup Nov 18, 2013

Andrey — Tue, 10 Dec 2013 12:44:14 GMT

I've neen invited to join Budapest Devops Meetup by Péter Halácsy and Gábor Vészi. I've been talking about DevOps transformation in Qik, how different teams were successful or not in changing the way the deployment and infrastructure changes are handled. I've presented the challenges they faced with monitoring, implementing DRY principle, making teams truly cross-functional and DevOps.Dev and Ops in huge organizations like Skype.

Presentation is available in PDF.