### Setting up a Devuan Package mirror -- walkthrough ###
This document includes a simple walkthrough to help setting up Devuan
package mirrors. We assume that you know how to create and use an ssh
key pair, how to use rsync, and how to setup a web server.
(onefang -- 20230524 -- added note about the ISO mirrors)
(onefang -- 20230524 -- update the apt-panopticon servers)
(onefang -- 20230524 -- added stuff about CC.deb.devuan.org support)
(onefang -- 20230524 -- update the expected size requirement)
(onefang -- 20220722 -- added note about rsync compression)
(onefang -- 20220721 -- some tweaks and white space removal)
(onefang -- 20220721 -- added note about redirecting HTTP to HTTPS)
(onefang -- 20220721 -- added the extra info we now require)
(onefang -- 20220721 -- added the apt-panopticon checking stuff)
(Evilham -- 20190522 -- added instructions for self-checks)
(Evilham -- 20190505 -- added lighttpd instructions)
(KatolaZ -- 20171224 -- included information about deb.devuan.org)
(KatolaZ -- 20171031 -- first edition)
== INDEX ==
0) INTRODUCTION
1) YOU NEED AN SSH KEY PAIR
2) LET US KNOW ABOUT YOUR MIRROR, AND STAY INFORMED
3) EVERYBODY: PULL!
4) WHAT USERS WILL LOOK FOR
5) SETTING UP THE NEEDED REWRITE RULES
6) ENTERING THE deb.devuan.org DNS ROUND-ROBIN
7) CHECKING YOUR MIRROR
======
0) INTRODUCTION
The full Devuan package repository is available at pkgmaster.devuan.org
All the mirrors will rsync from that main repo. The repo is contained
in a folder "devuan/" which contains two more folders:
merged/ : this is the folder containing the merged repo, i.e., the list
of packages available in Devuan, where forked Devuan packages
superseed the corresponding Debian package (if available)
devuan/ : this folder contains the repos that are specific to Devuan,
and the Devuan's /pool/ folder (i.e., the actual Devuan
packages).
At the time of initial writing (Julyy 2019) the full Devuan package
mirror requires about 60 GB of space. Notice that this size is much
smaller than the size of a full Debian repo (which is about 2TB). The
reason is that, thanks to amprolla, Devuan can use all the standard
Debian packages (coming from a standard Debian repo) together with
Devuan-specific packages. This is achieved by rewriting the Filename:
property of each package description, and implemented on the
repository side by appropriately rewriting the URLs required by the
client. Hence, any mirror needs to provide a few rewrite rules in
order to be able to serve Devuan packages. See SECTION 5 below for
further details.
NOTE: The size of Devuan package repositories will most probably keep
increasing in the future, so please take that into account when you
decide to host a package mirror. In 2019 it was 50GB, in 2022 60 GB.
On the other hand, it can decrease when old suites are archived, so in
2023 it's 17 GB.
We now support CC codes, so particular mirrors can choose to respond
to CC.deb.devuan.org, where CC is the country code. For example, the
country code for Netherlands is NL, so a mirror in Netherlands could
also respond to nl.deb.devuan.org as well as deb.devuan.org, and it's
own domain name.
For details about ISO mirrors (hosting the various boot images) see
https://files.devuan.org/MIRRORS.txt
1) YOU NEED AN SSH KEY PAIR
rsync access to pkgmaster.devuan.org is through ssh with public key.
You can use an existing ssh key pair or instead create an ad-hoc
key pair for your new Devuan mirror using:
$ ssh-keygen
2) LET US KNOW ABOUT YOUR MIRROR, AND STAY INFORMED
In order to be able to pull from pkgmaster.devuan.org, you need to have
your ssh public key added to the list of authorised keys. Please send
the public ssh key you wish to use to mirrors@devuan.org, together with
the following information:
- Name of the mirror contact person.
- Contact person email.
- Mirror IP or IPs, IPv4 and IPv6 if you have both.
- Mirror FQDN.
- Base URL for Devuan.
- Bandwidth limits (if any).
- Disk limits (if any).
- Update rate, how often you will update. We much prefer everyone does every 30 minutes.
- Mirror Country.
- Any CC codes for countries you are happy to support.
- Supported protocols (HTTP/HTTPS/RSYNC/FTP).
- If you want to be part of the DNS round robin, so your mirror can be accessed at deb.devuan.org.
- If you want to be part of the CC DNS round robin, so your mirror can be accessed at CC.deb.devuan.org.
We try to guarantee a quick setup of new mirrors, and we will get back
to you as soon as possible. Just please be patient :)
In addition, we require that the mirror contacts subscribe the
devuan-mirrors mailing list at the following address:
devuan-mirrors@lists.dyne.org
This is a very low-traffic mailing list where all the important
announcements about mirror configuration and issues are posted. The
current mirror herder will subscribe you.
3) EVERYBODY: PULL!
Once your ssh key has been set up, you should be able to rsync
the whole package repository using:
$ cd MIRROR_DIR
$ rsync --delete -avzX vesta@pkgmaster.devuan.org:~/devuan ./
where MIRROR_DIR is the directory on your mirror where the "devuan/"
folder will be placed. By doing so, you will have the new folders:
${MIRROR_DIR}/devuan/devuan
${MIRROR_DIR}/devuan/merged
The same command should be run periodically (e.g., as a cron job), and
normally once every half hour. Each mirror might be asked to set
slightly different update rates, if the need arises.
(NOTE: upon setting up your mirror, and only then, you might need to
run rsync twice, since amprolla3 is merging every few minutes and in
theory some files could have been changed during your first rsync
of the whole stuff.)
(NOTE: the rsync -z option will compress files during transfer, by
default it wont compress files with certain file extensions. That
includes .deb, .gz, and .xz, all of which apply to most of the files.)
4) WHAT USERS WILL LOOK FOR
apt users will connect to your mirror via http/https. So you need to
have a web server setup at your mirror FQDN. If ${MIRROR_DIR}/devuan/
is the ServerRoot of:
your.dev.uan.doma.in
the users of your mirror will look for stuff into:
your.dev.uan.doma.in/merged
your.dev.uan.doma.in/devuan
In most of the cases, Devuan is not the only thing mirrored by a
mirror site, hence your ServerRoot might be set to the parent folder
of "devuan/" (i.e., ${MIRROR_DIR}). In these cases, the users of your
mirror will look for stuff into:
your.dev.uan.doma.in/devuan/merged
your.dev.uan.doma.in/devuan/devuan
Your configuration might differ a bit from either of the two above.
But if that is the case, you know what we are talking about and you
should be able to find a way around it. If you are unsure, just ask
(e.g., either on the devuan-mirrors ML, or on the devuan-dev ML, or on
#devuan-dev IRC channel.
NOTE: that older versions of Debian and Devuan defaulted to using
HTTP, and could break if that gets redirected to HTTPS and they don't
have the apt HTTPS transport module installed. Mirrors serving
deb.devuan.org in the DNS round robin wont have the deb.devuan.org
certificate, so no HTTPS for that.
5) SETTING UP THE NEEDED REWRITE RULES
Now you have to configure your webserver to rewrite some of the
destinations under ${MIRROR_DIR}/devuan/ so that users can find the
packages. There are two cases: either your mirror has also a local Debian
mirror under '${MIRROR_DIR}/debian/', or it does not.
5a) IF YOU HAVE A LOCAL DEBIAN MIRROR AT ${MIRROR_DIR}/debian/
If you have a local DEBIAN mirror and it is placed at $MIRROR_DIR/debian,
and ${MIRROR_DIR} is your ServerRoot, then you will need to rewrite the
following destinations:
/devuan/merged/pool/DEVUAN/(.*) -> /devuan/devuan/pool/$1
/devuan/merged/pool/DEBIAN/(.*) -> /debian/pool/$1
/devuan/merged/pool/DEBIAN-SECURITY/(.*) -> /debian-security/pool/$1
For nginx, you will want to add the following section to the relevant
server config:
location /devuan/merged {
autoindex on;
rewrite /devuan/merged/pool/DEVUAN/(.*) /devuan/pool/$1;
rewrite /devuan/merged/pool/DEBIAN-SECURITY/(.*) /debian-security/pool/$1;
rewrite /devuan/merged/pool/DEBIAN/(.*) /debian/pool/$1;
}
For apache2, you will want to use something like the following instead:
RewriteEngine on
RewriteRule /devuan/merged/pool/DEVUAN/(.*) /devuan/pool/$1
RewriteRule /devuan/merged/pool/DEBIAN-SECURITY/(.*) /debian-security/pool/$1
RewriteRule /devuan/merged/pool/DEBIAN/(.*) /debian/pool/$1
(NOTE: These rewrite rules are just indicative. If you have already a Debian
mirror, you know better than us where the stuff is to be found in your
server. If you are unsure, just shout).
5b) IF YOU DON'T HAVE A LOCAL DEBIAN MIRROR
If you don't have a local DEBIAN mirror, you will need to rewrite all the
DEBIAN destinations to an appropriate DEBIAN MIRROR. We assume below that
the "devuan/" folder is directly under your ServerRoot. We also assume that
you choose "deb.debian.org" as Debian mirror. This is the recommended
choice, since "deb.debian.org" is the entry point of the Debian mirror
infrastructure, and should automatically redirect to the appropriate local
Debian mirror. You need the following rewrites:
/merged/pool/DEVUAN/(.*) -> /devuan/pool/$1
/merged/pool/DEBIAN/(.*) -> http://deb.debian.org/debian/pool/$1
/merged/pool/DEBIAN-SECURITY/(.*) -> http://deb.debian.org/debian-security/pool/$1
In nginx you will want to add the following section to the relevant
server config:
location /merged {
autoindex on;
rewrite /merged/pool/DEVUAN/(.*) /devuan/pool/$1;
rewrite /merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1;
rewrite /merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1;
}
For apache, instead:
RewriteEngine on
RewriteRule /devuan/merged/pool/DEVUAN/(.*) /devuan/pool/$1
RewriteRule /devuan/merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1
RewriteRule /devuan/merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1
For lighttpd, please check section 6.3) of this file.
Again, those rewrites are just indicative. Similar instructions hold
for setting up https mirrors.
6) ENTERING THE deb.devuan.org DNS ROUND-ROBIN
We have put in place a DNS Round-Robin for the domain deb.devuan.org,
which points to all the available package mirrors which can serve
requests for the domanin "deb.devuan.org". We have a similar setup for
the "CC.deb.devuan.org" country codes.
The easiest to have your mirror added to the Round-Robin is to add a
named VirtualHost to your web server conf to serve files for
deb.devuan.org. Sample configuration files for apache and nginx are
available under Section 6.1) and 6.2) below. Please amend them as
necessary, and incorporate them in your webserver configuration. In
particular, be careful in setting the document root and rewrite rules
appropriately.
*** IMPORTANT: THE DNS ROUND-ROBIN WILL NOT WORK FOR HTTPS ***
It is nevertheless recommended to keep your mirror reachable in
*both* ways, i.e., directly through your own URL and via deb.devuan.org,
since we will also advertise a list of existing mirrors with their
corresponding URL (and HTTPS is not supported through deb.devuan.org).
As usual, please shout if you need help with this configuration.
6.1) Sample apache conf for a deb.devuan.org named virtual host
<VirtualHost *:80>
ServerName deb.devuan.org
#### the root must be the folder containing "amprolla.txt"
DocumentRoot /home/mirror/devuan
RewriteEngine on
RewriteRule /merged/pool/DEVUAN/(.*) /devuan/pool/$1
RewriteRule /merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1
RewriteRule /merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1
</VirtualHost>
6.2) Sample nginx conf for a deb.devuan.org named virtual host
server {
listen 80;
listen [::]:80;
server_name deb.devuan.org;
### the root must be the folder containing "amprolla.txt"
root /home/mirror/devuan;
default_type "text/plain";
location / {
autoindex on;
}
location /merged {
autoindex on;
rewrite /merged/pool/DEVUAN/(.*) /devuan/pool/$1;
rewrite /merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1;
rewrite /merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1;
}
}
6.3) Sample lighttpd conf for a deb.devuan.org named virtual host
$HTTP["scheme"] == "http" {
$HTTP["host"] == "deb.devuan.org" {
server.document-root = "/home/mirror/devuan"
dir-listing.activate = "enable"
url.redirect = (
"^/merged/pool/DEVUAN/(.*)" => "/devuan/pool/$1",
"^/merged/pool/DEBIAN/(.*)" => "http://deb.debian.org/debian/pool/$1",
"^/merged/pool/DEBIAN-SECURITY/(.*)" => "http://deb.debian.org/debian-security/pool/$1"
)
}
}
7) CHECKING YOUR MIRROR
Check that your rewrite rules are working properly by using the provided
scripts/test_rewrite.sh script.
7.a) onefang wrote apt-panopticon, which tests all our official package
mirrors. We run it on several servers, they each update every ten minutes -
http://veritas.devuan.org/apt-panopticon/results/Report-web.html
https://sledjhamr.org/apt-panopticon/results/Report-web.html
https://ap.in.devuan.org/apt-panopticon/results/Report-web.html
https://mishka.snork.ca/apt-panopticon/results/Report-web.html
It is highly recommended that you keep an eye on one or more of those,
and fix any problems it shows, or complain to onefang if you find a bug
in apt-panopticon. At the bottom of the page is a link to the main
source repo, which in turn links to the bug / issue tracker.
Note the URL sanity test, which is there due to old bugs in apt and
potential user error.
7.b) UNDER YOUR OWN HOSTNAME
# If $HOSTNAME contains both /devuan and /merged directories:
$ scripts/test_rewrite.sh $HOSTNAME
# It is also common to serve the Devuan mirror under a /devuan directory
# that contains both devuan and merged directories:
$ scripts/test_rewrite.sh $HOSTNAME/$BASEURL
# To test HTTPS support, run as above making the protocol explicit:
$ scripts/test_rewrite.sh https://$HOSTNAME[$BASEURL]
7.c) AS deb.devuan.org (ROUND ROBIN)
# Regardless of how you serve the mirror under your hostname,
# test as follows:
$ scripts/test_rewrite.sh $HOSTNAME deb.devuan.org
7.d) TESTING RSYNC
# Output of
$ rsync $HOSTNAME::devuan
# Should list this file along with the devuan, merged and scripts dirs.