### Setting up a Devuan Package mirror -- walkthrough ### This document includes a simple walkthrough to help setting up Devuan package mirrors. We assume that you know how to create and use an ssh key pair, how to use rsync, and how to setup a web server. (onefang -- 20230524 -- added note about the ISO mirrors) (onefang -- 20230524 -- update the apt-panopticon servers) (onefang -- 20230524 -- added stuff about CC.deb.devuan.org support) (onefang -- 20230524 -- update the expected size requirement) (onefang -- 20220722 -- added note about rsync compression) (onefang -- 20220721 -- some tweaks and white space removal) (onefang -- 20220721 -- added note about redirecting HTTP to HTTPS) (onefang -- 20220721 -- added the extra info we now require) (onefang -- 20220721 -- added the apt-panopticon checking stuff) (Evilham -- 20190522 -- added instructions for self-checks) (Evilham -- 20190505 -- added lighttpd instructions) (KatolaZ -- 20171224 -- included information about deb.devuan.org) (KatolaZ -- 20171031 -- first edition) == INDEX == 0) INTRODUCTION 1) YOU NEED AN SSH KEY PAIR 2) LET US KNOW ABOUT YOUR MIRROR, AND STAY INFORMED 3) EVERYBODY: PULL! 4) WHAT USERS WILL LOOK FOR 5) SETTING UP THE NEEDED REWRITE RULES 6) ENTERING THE deb.devuan.org DNS ROUND-ROBIN 7) CHECKING YOUR MIRROR ====== 0) INTRODUCTION The full Devuan package repository is available at pkgmaster.devuan.org All the mirrors will rsync from that main repo. The repo is contained in a folder "devuan/" which contains two more folders: merged/ : this is the folder containing the merged repo, i.e., the list of packages available in Devuan, where forked Devuan packages superseed the corresponding Debian package (if available) devuan/ : this folder contains the repos that are specific to Devuan, and the Devuan's /pool/ folder (i.e., the actual Devuan packages). At the time of initial writing (Julyy 2019) the full Devuan package mirror requires about 60 GB of space. Notice that this size is much smaller than the size of a full Debian repo (which is about 2TB). The reason is that, thanks to amprolla, Devuan can use all the standard Debian packages (coming from a standard Debian repo) together with Devuan-specific packages. This is achieved by rewriting the Filename: property of each package description, and implemented on the repository side by appropriately rewriting the URLs required by the client. Hence, any mirror needs to provide a few rewrite rules in order to be able to serve Devuan packages. See SECTION 5 below for further details. NOTE: The size of Devuan package repositories will most probably keep increasing in the future, so please take that into account when you decide to host a package mirror. In 2019 it was 50GB, in 2022 60 GB. On the other hand, it can decrease when old suites are archived, so in 2023 it's 17 GB. We now support CC codes, so particular mirrors can choose to respond to CC.deb.devuan.org, where CC is the country code. For example, the country code for Netherlands is NL, so a mirror in Netherlands could also respond to nl.deb.devuan.org as well as deb.devuan.org, and it's own domain name. For details about ISO mirrors (hosting the various boot images) see https://files.devuan.org/MIRRORS.txt 1) YOU NEED AN SSH KEY PAIR rsync access to pkgmaster.devuan.org is through ssh with public key. You can use an existing ssh key pair or instead create an ad-hoc key pair for your new Devuan mirror using: $ ssh-keygen 2) LET US KNOW ABOUT YOUR MIRROR, AND STAY INFORMED In order to be able to pull from pkgmaster.devuan.org, you need to have your ssh public key added to the list of authorised keys. Please send the public ssh key you wish to use to mirrors@devuan.org, together with the following information: - Name of the mirror contact person. - Contact person email. - Mirror IP or IPs, IPv4 and IPv6 if you have both. - Mirror FQDN. - Base URL for Devuan. - Bandwidth limits (if any). - Disk limits (if any). - Update rate, how often you will update. We much prefer everyone does every 30 minutes. - Mirror Country. - Any CC codes for countries you are happy to support. - Supported protocols (HTTP/HTTPS/RSYNC/FTP). - If you want to be part of the DNS round robin, so your mirror can be accessed at deb.devuan.org. - If you want to be part of the CC DNS round robin, so your mirror can be accessed at CC.deb.devuan.org. We try to guarantee a quick setup of new mirrors, and we will get back to you as soon as possible. Just please be patient :) In addition, we require that the mirror contacts subscribe the devuan-mirrors mailing list at the following address: devuan-mirrors@lists.dyne.org This is a very low-traffic mailing list where all the important announcements about mirror configuration and issues are posted. The current mirror herder will subscribe you. 3) EVERYBODY: PULL! Once your ssh key has been set up, you should be able to rsync the whole package repository using: $ cd MIRROR_DIR $ rsync --delete -avzX vesta@pkgmaster.devuan.org:~/devuan ./ where MIRROR_DIR is the directory on your mirror where the "devuan/" folder will be placed. By doing so, you will have the new folders: ${MIRROR_DIR}/devuan/devuan ${MIRROR_DIR}/devuan/merged The same command should be run periodically (e.g., as a cron job), and normally once every half hour. Each mirror might be asked to set slightly different update rates, if the need arises. (NOTE: upon setting up your mirror, and only then, you might need to run rsync twice, since amprolla3 is merging every few minutes and in theory some files could have been changed during your first rsync of the whole stuff.) (NOTE: the rsync -z option will compress files during transfer, by default it wont compress files with certain file extensions. That includes .deb, .gz, and .xz, all of which apply to most of the files.) 4) WHAT USERS WILL LOOK FOR apt users will connect to your mirror via http/https. So you need to have a web server setup at your mirror FQDN. If ${MIRROR_DIR}/devuan/ is the ServerRoot of: your.dev.uan.doma.in the users of your mirror will look for stuff into: your.dev.uan.doma.in/merged your.dev.uan.doma.in/devuan In most of the cases, Devuan is not the only thing mirrored by a mirror site, hence your ServerRoot might be set to the parent folder of "devuan/" (i.e., ${MIRROR_DIR}). In these cases, the users of your mirror will look for stuff into: your.dev.uan.doma.in/devuan/merged your.dev.uan.doma.in/devuan/devuan Your configuration might differ a bit from either of the two above. But if that is the case, you know what we are talking about and you should be able to find a way around it. If you are unsure, just ask (e.g., either on the devuan-mirrors ML, or on the devuan-dev ML, or on #devuan-dev IRC channel. NOTE: that older versions of Debian and Devuan defaulted to using HTTP, and could break if that gets redirected to HTTPS and they don't have the apt HTTPS transport module installed. Mirrors serving deb.devuan.org in the DNS round robin wont have the deb.devuan.org certificate, so no HTTPS for that. 5) SETTING UP THE NEEDED REWRITE RULES Now you have to configure your webserver to rewrite some of the destinations under ${MIRROR_DIR}/devuan/ so that users can find the packages. There are two cases: either your mirror has also a local Debian mirror under '${MIRROR_DIR}/debian/', or it does not. 5a) IF YOU HAVE A LOCAL DEBIAN MIRROR AT ${MIRROR_DIR}/debian/ If you have a local DEBIAN mirror and it is placed at $MIRROR_DIR/debian, and ${MIRROR_DIR} is your ServerRoot, then you will need to rewrite the following destinations: /devuan/merged/pool/DEVUAN/(.*) -> /devuan/devuan/pool/$1 /devuan/merged/pool/DEBIAN/(.*) -> /debian/pool/$1 /devuan/merged/pool/DEBIAN-SECURITY/(.*) -> /debian-security/pool/$1 For nginx, you will want to add the following section to the relevant server config: location /devuan/merged { autoindex on; rewrite /devuan/merged/pool/DEVUAN/(.*) /devuan/pool/$1; rewrite /devuan/merged/pool/DEBIAN-SECURITY/(.*) /debian-security/pool/$1; rewrite /devuan/merged/pool/DEBIAN/(.*) /debian/pool/$1; } For apache2, you will want to use something like the following instead: RewriteEngine on RewriteRule /devuan/merged/pool/DEVUAN/(.*) /devuan/pool/$1 RewriteRule /devuan/merged/pool/DEBIAN-SECURITY/(.*) /debian-security/pool/$1 RewriteRule /devuan/merged/pool/DEBIAN/(.*) /debian/pool/$1 (NOTE: These rewrite rules are just indicative. If you have already a Debian mirror, you know better than us where the stuff is to be found in your server. If you are unsure, just shout). 5b) IF YOU DON'T HAVE A LOCAL DEBIAN MIRROR If you don't have a local DEBIAN mirror, you will need to rewrite all the DEBIAN destinations to an appropriate DEBIAN MIRROR. We assume below that the "devuan/" folder is directly under your ServerRoot. We also assume that you choose "deb.debian.org" as Debian mirror. This is the recommended choice, since "deb.debian.org" is the entry point of the Debian mirror infrastructure, and should automatically redirect to the appropriate local Debian mirror. You need the following rewrites: /merged/pool/DEVUAN/(.*) -> /devuan/pool/$1 /merged/pool/DEBIAN/(.*) -> http://deb.debian.org/debian/pool/$1 /merged/pool/DEBIAN-SECURITY/(.*) -> http://deb.debian.org/debian-security/pool/$1 In nginx you will want to add the following section to the relevant server config: location /merged { autoindex on; rewrite /merged/pool/DEVUAN/(.*) /devuan/pool/$1; rewrite /merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1; rewrite /merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1; } For apache, instead: RewriteEngine on RewriteRule /devuan/merged/pool/DEVUAN/(.*) /devuan/pool/$1 RewriteRule /devuan/merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1 RewriteRule /devuan/merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1 For lighttpd, please check section 6.3) of this file. Again, those rewrites are just indicative. Similar instructions hold for setting up https mirrors. 6) ENTERING THE deb.devuan.org DNS ROUND-ROBIN We have put in place a DNS Round-Robin for the domain deb.devuan.org, which points to all the available package mirrors which can serve requests for the domanin "deb.devuan.org". We have a similar setup for the "CC.deb.devuan.org" country codes. The easiest to have your mirror added to the Round-Robin is to add a named VirtualHost to your web server conf to serve files for deb.devuan.org. Sample configuration files for apache and nginx are available under Section 6.1) and 6.2) below. Please amend them as necessary, and incorporate them in your webserver configuration. In particular, be careful in setting the document root and rewrite rules appropriately. *** IMPORTANT: THE DNS ROUND-ROBIN WILL NOT WORK FOR HTTPS *** It is nevertheless recommended to keep your mirror reachable in *both* ways, i.e., directly through your own URL and via deb.devuan.org, since we will also advertise a list of existing mirrors with their corresponding URL (and HTTPS is not supported through deb.devuan.org). As usual, please shout if you need help with this configuration. 6.1) Sample apache conf for a deb.devuan.org named virtual host ServerName deb.devuan.org #### the root must be the folder containing "amprolla.txt" DocumentRoot /home/mirror/devuan RewriteEngine on RewriteRule /merged/pool/DEVUAN/(.*) /devuan/pool/$1 RewriteRule /merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1 RewriteRule /merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1 6.2) Sample nginx conf for a deb.devuan.org named virtual host server { listen 80; listen [::]:80; server_name deb.devuan.org; ### the root must be the folder containing "amprolla.txt" root /home/mirror/devuan; default_type "text/plain"; location / { autoindex on; } location /merged { autoindex on; rewrite /merged/pool/DEVUAN/(.*) /devuan/pool/$1; rewrite /merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1; rewrite /merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1; } } 6.3) Sample lighttpd conf for a deb.devuan.org named virtual host $HTTP["scheme"] == "http" { $HTTP["host"] == "deb.devuan.org" { server.document-root = "/home/mirror/devuan" dir-listing.activate = "enable" url.redirect = ( "^/merged/pool/DEVUAN/(.*)" => "/devuan/pool/$1", "^/merged/pool/DEBIAN/(.*)" => "http://deb.debian.org/debian/pool/$1", "^/merged/pool/DEBIAN-SECURITY/(.*)" => "http://deb.debian.org/debian-security/pool/$1" ) } } 7) CHECKING YOUR MIRROR Check that your rewrite rules are working properly by using the provided scripts/test_rewrite.sh script. 7.a) onefang wrote apt-panopticon, which tests all our official package mirrors. We run it on several servers, they each update every ten minutes - http://veritas.devuan.org/apt-panopticon/results/Report-web.html https://sledjhamr.org/apt-panopticon/results/Report-web.html https://ap.in.devuan.org/apt-panopticon/results/Report-web.html https://mishka.snork.ca/apt-panopticon/results/Report-web.html It is highly recommended that you keep an eye on one or more of those, and fix any problems it shows, or complain to onefang if you find a bug in apt-panopticon. At the bottom of the page is a link to the main source repo, which in turn links to the bug / issue tracker. Note the URL sanity test, which is there due to old bugs in apt and potential user error. 7.b) UNDER YOUR OWN HOSTNAME # If $HOSTNAME contains both /devuan and /merged directories: $ scripts/test_rewrite.sh $HOSTNAME # It is also common to serve the Devuan mirror under a /devuan directory # that contains both devuan and merged directories: $ scripts/test_rewrite.sh $HOSTNAME/$BASEURL # To test HTTPS support, run as above making the protocol explicit: $ scripts/test_rewrite.sh https://$HOSTNAME[$BASEURL] 7.c) AS deb.devuan.org (ROUND ROBIN) # Regardless of how you serve the mirror under your hostname, # test as follows: $ scripts/test_rewrite.sh $HOSTNAME deb.devuan.org 7.d) TESTING RSYNC # Output of $ rsync $HOSTNAME::devuan # Should list this file along with the devuan, merged and scripts dirs.