### Setting up a Devuan Package mirror -- walkthrough ###


This document includes a simple walkthrough to help setting up Devuan
package mirrors. We assume that you know how to create and use an ssh 
key pair, how to use rsync, and how to setup a web server.  

(onefang -- 20230524 -- added note about the ISO mirrors)
(onefang -- 20230524 -- update the apt-panopticon servers)
(onefang -- 20230524 -- added stuff about CC.deb.devuan.org support)
(onefang -- 20230524 -- update the expected size requirement)
(onefang -- 20220722 -- added note about rsync compression)
(onefang -- 20220721 -- some tweaks and white space removal)
(onefang -- 20220721 -- added note about redirecting HTTP to HTTPS)
(onefang -- 20220721 -- added the extra info we now require)
(onefang -- 20220721 -- added the apt-panopticon checking stuff)
(Evilham -- 20190522 -- added instructions for self-checks)
(Evilham -- 20190505 -- added lighttpd instructions)
(KatolaZ -- 20171224 -- included information about deb.devuan.org)
(KatolaZ -- 20171031 -- first edition)


== INDEX ==

0) INTRODUCTION
1) YOU NEED AN SSH KEY PAIR
2) LET US KNOW ABOUT YOUR MIRROR, AND STAY INFORMED
3) EVERYBODY: PULL!
4) WHAT USERS WILL LOOK FOR
5) SETTING UP THE NEEDED REWRITE RULES
6) ENTERING THE deb.devuan.org DNS ROUND-ROBIN
7) CHECKING YOUR MIRROR

======

0) INTRODUCTION

   The full Devuan package repository is available at pkgmaster.devuan.org
   All the mirrors will rsync from that main repo. The repo is contained
   in a folder "devuan/" which contains two more folders:

     merged/ : this is the folder containing the merged repo, i.e., the list
               of packages available in Devuan, where forked Devuan packages
               superseed the corresponding Debian package (if available)

     devuan/ : this folder contains the repos that are specific to Devuan,
               and the Devuan's /pool/ folder (i.e., the actual Devuan
               packages).


   At the time of initial writing (Julyy 2019) the full Devuan package
   mirror requires about 60 GB of space. Notice that this size is much
   smaller than  the size of a full Debian repo (which is about 2TB). The
   reason is that,  thanks to amprolla, Devuan can use all the standard
   Debian packages  (coming from a standard Debian repo) together with
   Devuan-specific  packages. This is achieved by rewriting the Filename:
   property of each  package description, and implemented on the
   repository side by  appropriately rewriting the URLs required by the
   client. Hence, any  mirror needs to provide a few rewrite rules in
   order to be able to serve Devuan packages. See SECTION 5 below for
   further details. 

   NOTE: The size of Devuan package repositories will most probably keep
   increasing in the future, so please take that into account when you 
   decide to host a package mirror. In 2019 it was 50GB, in 2022 60 GB. 
   On the other hand, it can decrease when old suites are archived, so in
   2023 it's 17 GB.

   We now support CC codes, so particular mirrors can choose to respond
   to CC.deb.devuan.org, where CC is the country code.  For example, the
   country code for Netherlands is NL, so a mirror in Netherlands could
   also respond to nl.deb.devuan.org as well as deb.devuan.org, and it's
   own domain name.

   For details about ISO mirrors (hosting the various boot images) see
   https://files.devuan.org/MIRRORS.txt

1) YOU NEED AN SSH KEY PAIR

   rsync access to pkgmaster.devuan.org is through ssh with public key.  
   You can use an existing ssh key pair or instead create an ad-hoc 
   key pair for your new Devuan mirror using:

      $ ssh-keygen
  

2) LET US KNOW ABOUT YOUR MIRROR, AND STAY INFORMED

   In order to be able to pull from pkgmaster.devuan.org, you need to have
   your ssh public key added to the list of authorised keys. Please send 
   the public ssh key you wish to use to mirrors@devuan.org, together with 
   the following information:

   - Name of the mirror contact person.
   - Contact person email.
   - Mirror IP or IPs, IPv4 and IPv6 if you have both.
   - Mirror FQDN.
   - Base URL for Devuan.
   - Bandwidth limits (if any).
   - Disk limits (if any).
   - Update rate, how often you will update.  We much prefer everyone does every 30 minutes.
   - Mirror Country.
   - Any CC codes for countries you are happy to support.
   - Supported protocols (HTTP/HTTPS/RSYNC/FTP).
   - If you want to be part of the DNS round robin, so your mirror can be accessed at deb.devuan.org.
   - If you want to be part of the CC DNS round robin, so your mirror can be accessed at CC.deb.devuan.org.
 
   We try to guarantee a quick setup of new mirrors, and we will get back 
   to you as soon as possible. Just please be patient :)

   In addition, we require that the mirror contacts subscribe the 
   devuan-mirrors mailing list at the following address:

     devuan-mirrors@lists.dyne.org 

   This is a very low-traffic mailing list where all the important
   announcements about mirror configuration and issues are posted. The
   current mirror herder will subscribe you.


3) EVERYBODY: PULL!

   Once your ssh key has been set up, you should be able to rsync
   the whole package repository using:

     $ cd MIRROR_DIR
     $ rsync --delete -avzX vesta@pkgmaster.devuan.org:~/devuan ./

   where MIRROR_DIR is the directory on your mirror where the "devuan/"
   folder will be placed. By doing so, you will have the new folders:

       ${MIRROR_DIR}/devuan/devuan
       ${MIRROR_DIR}/devuan/merged

   The same command should be run periodically (e.g., as a cron job), and 
   normally once every half hour. Each mirror might be asked  to set
   slightly different update rates, if the need arises. 

   (NOTE: upon setting up your mirror, and only then, you might need to 
    run rsync twice, since amprolla3 is merging every few minutes and in 
    theory some files could have been changed during your first rsync
    of the whole stuff.)

   (NOTE: the rsync -z option will compress files during transfer, by
   default it wont compress files with certain file extensions.  That
   includes .deb, .gz, and .xz, all of which apply to most of the files.)

4) WHAT USERS WILL LOOK FOR

   apt users will connect to your mirror via http/https. So you need to 
   have a web server setup at your mirror FQDN. If ${MIRROR_DIR}/devuan/
   is the ServerRoot of: 

      your.dev.uan.doma.in

   the users of your mirror will look for stuff into:

      your.dev.uan.doma.in/merged
      your.dev.uan.doma.in/devuan

   In most of the cases, Devuan is not the only thing mirrored by a
   mirror site, hence your ServerRoot might be set to the parent folder
   of "devuan/" (i.e., ${MIRROR_DIR}). In these cases, the users of your
   mirror will look  for stuff into: 

      your.dev.uan.doma.in/devuan/merged
      your.dev.uan.doma.in/devuan/devuan

   Your configuration might differ a bit from either of the two above.
   But  if that is the case, you know what we are talking about and you
   should be  able to find a way around it. If you are unsure, just ask
   (e.g., either on the devuan-mirrors ML, or on the devuan-dev ML, or on
   #devuan-dev IRC channel.

   NOTE: that older versions of Debian and Devuan defaulted to using
   HTTP, and could break if that gets redirected to HTTPS and they don't
   have the apt HTTPS transport module installed.  Mirrors serving
   deb.devuan.org in the DNS round robin wont have the deb.devuan.org
   certificate, so no HTTPS for that.

5) SETTING UP THE NEEDED REWRITE RULES

   Now you have to configure your webserver to rewrite some of the 
   destinations under ${MIRROR_DIR}/devuan/ so that users can find the 
   packages. There are two cases: either your mirror has also a local Debian 
   mirror under '${MIRROR_DIR}/debian/', or it does not.

5a) IF YOU HAVE A LOCAL DEBIAN MIRROR AT ${MIRROR_DIR}/debian/

    If you have a local DEBIAN mirror and it is placed at $MIRROR_DIR/debian, 
    and ${MIRROR_DIR} is your ServerRoot, then you will need to rewrite the 
    following destinations:

      /devuan/merged/pool/DEVUAN/(.*)           ->  /devuan/devuan/pool/$1
      /devuan/merged/pool/DEBIAN/(.*)           ->  /debian/pool/$1
      /devuan/merged/pool/DEBIAN-SECURITY/(.*)  ->  /debian-security/pool/$1

   For nginx, you will want to add the following section to the relevant 
   server config:

      location /devuan/merged {
                  autoindex on;
                  rewrite /devuan/merged/pool/DEVUAN/(.*)          /devuan/pool/$1;
                  rewrite /devuan/merged/pool/DEBIAN-SECURITY/(.*) /debian-security/pool/$1;
                  rewrite /devuan/merged/pool/DEBIAN/(.*)          /debian/pool/$1;
          }
 
   For apache2, you will want to use something like the following instead:

        RewriteEngine on
	RewriteRule /devuan/merged/pool/DEVUAN/(.*) /devuan/pool/$1
	RewriteRule /devuan/merged/pool/DEBIAN-SECURITY/(.*) /debian-security/pool/$1
    	RewriteRule /devuan/merged/pool/DEBIAN/(.*) /debian/pool/$1
 
   (NOTE: These rewrite rules are just indicative. If you have already a Debian 
    mirror, you know better than us where the stuff is to be found in your 
    server. If you are unsure, just shout). 
 
5b) IF YOU DON'T HAVE A LOCAL DEBIAN MIRROR

    If you don't have a local DEBIAN mirror, you will need to rewrite all the 
    DEBIAN destinations to an appropriate DEBIAN MIRROR. We assume below that
    the "devuan/" folder is directly under your ServerRoot. We also assume that 
    you choose "deb.debian.org" as Debian mirror. This is the recommended 
    choice, since "deb.debian.org" is the entry point of the Debian mirror 
    infrastructure, and should automatically redirect to the appropriate local
    Debian mirror. You need the following rewrites:

      /merged/pool/DEVUAN/(.*)           ->  /devuan/pool/$1
      /merged/pool/DEBIAN/(.*)           ->  http://deb.debian.org/debian/pool/$1
      /merged/pool/DEBIAN-SECURITY/(.*)  ->  http://deb.debian.org/debian-security/pool/$1

    In nginx you will want to add the following section to the relevant
    server config:

    location /merged {
                autoindex on;
                rewrite /merged/pool/DEVUAN/(.*)          /devuan/pool/$1;
                rewrite /merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1;
                rewrite /merged/pool/DEBIAN/(.*)          http://deb.debian.org/debian/pool/$1;
        }

   For apache, instead:

	RewriteEngine on
	RewriteRule /devuan/merged/pool/DEVUAN/(.*) /devuan/pool/$1
	RewriteRule /devuan/merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1
    	RewriteRule /devuan/merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1

   For lighttpd, please check section 6.3) of this file.
 
   Again, those rewrites are just indicative. Similar instructions hold 
   for setting up https mirrors. 

6) ENTERING THE deb.devuan.org DNS ROUND-ROBIN

We have put in place a DNS Round-Robin for the domain deb.devuan.org, 
which points to all the available package mirrors which can serve
requests for the domanin "deb.devuan.org". We have a similar setup for
the "CC.deb.devuan.org" country codes.

The easiest to have your mirror added to the Round-Robin is to add a 
named VirtualHost to your web server conf to serve files for 
deb.devuan.org. Sample configuration files for apache and nginx are 
available under Section 6.1) and 6.2) below. Please amend them as
necessary, and incorporate them in your webserver configuration. In 
particular, be careful in setting the document root and rewrite rules
appropriately.

  *** IMPORTANT: THE DNS ROUND-ROBIN WILL NOT WORK FOR HTTPS ***

It is nevertheless recommended to keep your mirror reachable in 
*both* ways, i.e., directly through your own URL and via deb.devuan.org,
since we will also advertise a list of existing mirrors with their
corresponding URL (and HTTPS is not supported through deb.devuan.org).

As usual, please shout if you need help with this configuration.

6.1) Sample apache conf for a deb.devuan.org named virtual host


<VirtualHost *:80>

        ServerName deb.devuan.org
#### the root must be the folder containing "amprolla.txt"
        DocumentRoot /home/mirror/devuan

        RewriteEngine on

        RewriteRule /merged/pool/DEVUAN/(.*) /devuan/pool/$1
        RewriteRule /merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1
        RewriteRule /merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1

</VirtualHost>



6.2) Sample nginx conf for a deb.devuan.org named virtual host

server {
        listen 80;
        listen [::]:80;

        server_name deb.devuan.org;

### the root must be the folder containing "amprolla.txt"
        root /home/mirror/devuan;
        default_type "text/plain";

        location / {
                autoindex on;
        }

        location /merged {
                autoindex on;
                rewrite /merged/pool/DEVUAN/(.*)          /devuan/pool/$1;
                rewrite /merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1;
                rewrite /merged/pool/DEBIAN/(.*)          http://deb.debian.org/debian/pool/$1;
        }
}



6.3) Sample lighttpd conf for a deb.devuan.org named virtual host

$HTTP["scheme"] == "http" {
    $HTTP["host"] == "deb.devuan.org" {
        server.document-root = "/home/mirror/devuan"
        dir-listing.activate = "enable"
        url.redirect = (
            "^/merged/pool/DEVUAN/(.*)" => "/devuan/pool/$1",
            "^/merged/pool/DEBIAN/(.*)" => "http://deb.debian.org/debian/pool/$1",
            "^/merged/pool/DEBIAN-SECURITY/(.*)" => "http://deb.debian.org/debian-security/pool/$1"
        )
    }
}

7) CHECKING YOUR MIRROR

Check that your rewrite rules are working properly by using the provided
scripts/test_rewrite.sh script.

7.a) onefang wrote apt-panopticon, which tests all our official package
mirrors.  We run it on several servers, they each update every ten minutes - 

http://veritas.devuan.org/apt-panopticon/results/Report-web.html
https://sledjhamr.org/apt-panopticon/results/Report-web.html
https://ap.in.devuan.org/apt-panopticon/results/Report-web.html
https://mishka.snork.ca/apt-panopticon/results/Report-web.html


It is highly recommended that you keep an eye on one or more of those,
and fix any problems it shows, or complain to onefang if you find a bug
in apt-panopticon.  At the bottom of the page is a link to the main
source repo, which in turn links to the bug / issue tracker.

Note the URL sanity test, which is there due to old bugs in apt and
potential user error.

7.b) UNDER YOUR OWN HOSTNAME

  # If $HOSTNAME contains both /devuan and /merged directories:
  $ scripts/test_rewrite.sh $HOSTNAME

  # It is also common to serve the Devuan mirror under a /devuan directory
  # that contains both devuan and merged directories:
  $ scripts/test_rewrite.sh $HOSTNAME/$BASEURL

  # To test HTTPS support, run as above making the protocol explicit:
  $ scripts/test_rewrite.sh https://$HOSTNAME[$BASEURL]

7.c) AS deb.devuan.org (ROUND ROBIN)

  # Regardless of how you serve the mirror under your hostname,
  # test as follows:
  $ scripts/test_rewrite.sh $HOSTNAME deb.devuan.org

7.d) TESTING RSYNC

  # Output of
  $ rsync $HOSTNAME::devuan
  # Should list this file along with the devuan, merged and scripts dirs.