# Advanced deployment

## Note

Please make sure that you have experienced the installation process on single node. This deployment method is *NOT* recommended on first try.

It would be easy for you to understand what we are going to do if you have some experience in using `docker` and `frp`.

## Goal

The goal of this advanced deployment is to deploy the CTFd and challenge containers on seperate machines for better experiences.

Overall, `ctfd-whale` can be decomposed into three compnents: `CTFd`, challenge containers along with frpc, and frps itself. The three components can be deployed seperately or together to satisfy different needs.

For example, if you're in a school or an organization that has a number of high-performance dedicated server *BUT* no public IP for public access, you can refer to this tutorial.

Here are some options:

* deploy frps on a server with public access
* deploy challenge containers on a seperate sever by joining the server into the swarm you created earlier
* deploy challenge containers on *rootless* docker
* deploy challenge containers on a remote server with public access, *securely*

You could achieve the first option with little effort by deploying the frps on the server and configure frpc with a different `server_addr`.  
In a swarm with multiple nodes, you can configure CTFd to start challenge containers on nodes you specifies randomly. Just make sure the node `whale` controlls is a `Leader`. This is not covered in this guide. You'll find it rather simple, even if you have zero experience on docker swarm.  
The [Docker docs](https://docs.docker.com/engine/security/rootless/) have a detailed introduction on how to set up a rootless docker, so it's also not covered in this guide.

In following paragraphs, the last option is introduced.

## Architecture

In this tutorial, we have 2 separate machines which we'll call them `web` and `target` server later. We will deploy CTFd on `web` and challenge containers (along with frp) on `target`.

This picture shows a brief glance.

![architecture](imgs/arch.png)

---

### Operate on `target` server

> root user is NOT recommended
> if you want to expose your docker deployment, you might also want to use [rootless docker](https://docs.docker.com/engine/security/rootless/)

Please read the [Docker docs](https://docs.docker.com/engine/security/protect-access/#use-tls-https-to-protect-the-docker-daemon-socket) thoroughly before continuing.

Setup docker swarm and clone this repo as described in [installation](./install.md), then follow the steps described in the Docker docs to sign your certificates.

> protect your certificates carefully
> one can take over the user running `dockerd` effortlessly with them
> and in most cases, the user is, unfortunately, root.

You can now create a network for your challenges by executing

```bash
docker network create --driver overlay --attachable challenges
```

Then setup frp on this machine. You might want to setup frps first:

```bash
# change to the version you prefer
wget https://github.com/fatedier/frp/releases/download/v0.37.0/frp_0.37.0_linux_amd64.tar.gz
tar xzvf frp_0.37.0_linux_amd64.tar.gz
cd frp_0.37.0_linux_amd64
mkdir /etc/frp
configure_frps frps.ini  # refer to [installation](./install.md)
cp systemd/frps.service /etc/systemd/system
systemctl daemon-reload
systemctl enable frps
systemctl start frps
```

Then frpc. Frpc should be running in the same network with the challenge containers, so make sure you connect frpc to the network you just created.

```bash
docker run -it --restart=always -d  --network challenges -p 7400:7400 frankli0324/frp:frpc \
    --server_addr=host_ip:host_port \
    --server_port=7000 \
    --admin_addr=7400 \
    --admin_port=7400 \
    --admin_user=username \
    --admin_pwd=password \
    --token=your_token
```

You could use `docker-compose` for better experience.

Here are some pitfalls or problems you might run into:

#### working with `systemd`

Copy the systemd service file to `/etc/systemd` in order to prevent it from being overwritten by future updates.

```bash
cp /lib/systemd/system/docker.service /etc/systemd/system/docker.service
```

Locate `ExecStart` in the file and change it into something like this:

```systemd
ExecStart=/usr/bin/dockerd \
    --tlsverify \
    --tlscacert=/etc/docker/certs/ca.pem \
    --tlscert=/etc/docker/certs/server-cert.pem \
    --tlskey=/etc/docker/certs/server-key.pem \
    -H tcp://0.0.0.0:2376 \
    -H unix:///var/run/docker.sock
```

Remember to reload `systemd` before restarting `docker.service`

```bash
systemctl daemon-reload
systemctl restart docker
```

#### cloud service providers

Most service providers provides you with a basic virus scanner in their system images, for example, AliCloud images comes with `YunDun`. You might want to disable it. The challenge containers often comes with backdoors, and is often accessed in a way cloud providers don't like (they are obviously attacks).

#### certificate security

Please follow the best practices when signing your certificates. If you gets used to signing both the client and server certicates on a single machine, you might run into troubles in the future.

If you feel inconvenient, at least sign them on your personal computer, and transfer only the needed files to client/server.

#### challenge networks and frpc

You could create an internal network for challenges, but you have to connect frpc to a different network *with* internet in order to map the ports so that CTFd can access the admin interface. Also, make sure frps is accessible by frpc.

### Operate on `web` server

Map your client certificates into docker. You might want to use `docker secrets`. Remember where the files are *inside the container*. In the case which you use `docker secrets`, the directory is `/run/secrets`.

You may also delete everything related to `frp` like `frp_network` since we are not going to run challenge containers on `web` server anymore. But if you just has one public IP for `web` server, you can leave `frps` service running.

Then recreate your containers:

```bash
docker-compose down  # needed for removing unwanted networks
docker-compose up -d
```

Now you can configure CTFd accordingly.
Sample configurations:

![whale-config1](imgs/whale-config1.png)
![whale-config2](imgs/whale-config2.png)
![whale-config3](imgs/whale-config3.png)

refer to [installation](./install.md) for explanations.

---

Now you can add a challenge to test it out.
