月度归档:2022年03月

kubeadm 更新证书


2022年3月03日 11:33:35   2,070 次浏览

安装 go 语言环境

Golang 官网下载地址:golang官网
打开官网下载地址选择对应的系统版本, 复制下载链接
这里我选择的是
go1.16.5.linux-amd64.tar.gz

下载解压

下载安装包

wget https://dl.google.com/go/go1.16.5.linux-amd64.tar.gz

解压到/usr/loacl目录下

tar -C /usr/local -zxvf  go1.16.5.linux-amd64.tar.gz

添加环境变量

添加/usr/loacl/go/bin 目录到 PATH 变量中。添加到 /etc/profile

vim /etc/profile 
# 在最后一行添加 
export GOROOT=/usr/local/go 
export PATH=$PATH:$GOROOT/bin 
# 保存退出后source一下 
source /etc/profile

验证

执行go version,如果现实版本号,则Go环境安装成功。

[root@master ~]# go version 
go version go1.16.5 linux/amd64

查看当前的证书时间

执行命令 查看当前证书时间

kubeadm alpha certs check-expiration

 

下载源码

打开github kubernetes 选择对应的版本下载
下载并解压
因为我是 v1.20.6 版本所以下载对应的

wget https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.20.6.zip 
unzip v1.20.6.zip

修改 constants.go 文件
vim cmd/kubeadm/app/constants/constants.go 找到 CertificateValidity ,修改如下

cd kubernetes-1.20.6
vim cmd/kubeadm/app/constants/constants.go
....
const (
        // KubernetesDir is the directory Kubernetes owns for storing various configuration files
        KubernetesDir = "/etc/kubernetes"
        // ManifestsSubDirName defines directory name to store manifests
        ManifestsSubDirName = "manifests"
        // TempDirForKubeadm defines temporary directory for kubeadm
        // should be joined with KubernetesDir.
        TempDirForKubeadm = "tmp"

        // CertificateValidity defines the validity for all the signed certificates generated by kubeadm
        CertificateValidity = time.Hour * 24 * 365 * 100  #  修改此内容
....

编译 kubeadm

make WHAT=cmd/kubeadm

返回如下

[root@master kubernetes-1.20.6]# make WHAT=cmd/kubeadm

+++ [0624 10:59:21] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
+++ [0624 10:59:25] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0624 10:59:33] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0624 10:59:44] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0624 11:00:04] Building go targets for linux/amd64:
    ./vendor/k8s.io/kube-openapi/cmd/openapi-gen
+++ [0624 11:00:19] Building go targets for linux/amd64:
    ./vendor/github.com/go-bindata/go-bindata/go-bindata
+++ [0624 11:00:20] Building go targets for linux/amd64:
    cmd/kubeadm

编译完生成如下目录和二进制文件

[root@master kubernetes-1.20.6]#  ll _output/bin/
总用量 75680
-rwxr-xr-x. 1 root root  5943296 6月  24 10:59 conversion-gen
-rwxr-xr-x. 1 root root  5689344 6月  24 10:59 deepcopy-gen
-rwxr-xr-x. 1 root root  5709824 6月  24 10:59 defaulter-gen
-rwxr-xr-x. 1 root root  3555111 6月  24 10:59 go2make
-rwxr-xr-x. 1 root root  1966080 6月  24 11:00 go-bindata
-rwxr-xr-x. 1 root root 39325696 6月  24 11:01 kubeadm
-rwxr-xr-x. 1 root root  9650176 6月  24 11:00 openapi-gen
-rwxr-xr-x. 1 root root  5656576 6月  24 10:59 prerelease-lifecycle-gen

备份文件

备份 kubeadm 和证书文件

cp /usr/bin/kubeadm{,.bak20210624} 
cp -r /etc/kubernetes/pki{,.bak20210624}

查看备份文件

[root@master kubernetes-1.20.6]# ll /usr/bin/kubeadm*
-rwxr-xr-x. 1 root root 39325696 6月  24 11:05 /usr/bin/kubeadm
-rwxr-xr-x. 1 root root 39210880 6月  24 11:02 /usr/bin/kubeadm.bak20210624

[root@master kubernetes-1.20.6 ll /etc/kubernetes/pki*
/etc/kubernetes/pki:
总用量 56
-rw-r--r--. 1 root root 1289 6月  24 11:05 apiserver.crt
-rw-r--r--. 1 root root 1139 6月  24 11:05 apiserver-etcd-client.crt
-rw-------. 1 root root 1675 6月  24 11:05 apiserver-etcd-client.key
-rw-------. 1 root root 1679 6月  24 11:05 apiserver.key
-rw-r--r--. 1 root root 1147 6月  24 11:05 apiserver-kubelet-client.crt
-rw-------. 1 root root 1675 6月  24 11:05 apiserver-kubelet-client.key
-rw-r--r--. 1 root root 1066 6月  22 15:01 ca.crt
-rw-------. 1 root root 1675 6月  22 15:01 ca.key
drwxr-xr-x. 2 root root  162 6月  22 15:01 etcd
-rwxr-xr-x. 1 root root 1078 6月  22 15:01 front-proxy-ca.crt
-rw-------. 1 root root 1675 6月  22 15:01 front-proxy-ca.key
-rw-r--r--. 1 root root 1103 6月  24 11:05 front-proxy-client.crt
-rw-------. 1 root root 1679 6月  24 11:05 front-proxy-client.key
-rw-------. 1 root root 1675 6月  22 15:01 sa.key
-rw-------. 1 root root  451 6月  22 15:01 sa.pub

/etc/kubernetes/pki.bak20210624:
总用量 56
-rw-r--r--. 1 root root 1289 6月  24 11:04 apiserver.crt
-rw-r--r--. 1 root root 1135 6月  24 11:04 apiserver-etcd-client.crt
-rw-------. 1 root root 1675 6月  24 11:04 apiserver-etcd-client.key
-rw-------. 1 root root 1679 6月  24 11:04 apiserver.key
-rw-r--r--. 1 root root 1143 6月  24 11:04 apiserver-kubelet-client.crt
-rw-------. 1 root root 1675 6月  24 11:04 apiserver-kubelet-client.key
-rw-r--r--. 1 root root 1066 6月  24 11:04 ca.crt
-rw-------. 1 root root 1675 6月  24 11:04 ca.key
drwxr-xr-x. 2 root root  162 6月  24 11:04 etcd
-rwxr-xr-x. 1 root root 1078 6月  24 11:04 front-proxy-ca.crt
-rw-------. 1 root root 1675 6月  24 11:04 front-proxy-ca.key
-rw-r--r--. 1 root root 1103 6月  24 11:04 front-proxy-client.crt
-rw-------. 1 root root 1679 6月  24 11:04 front-proxy-client.key
-rw-------. 1 root root 1675 6月  24 11:04 sa.key
-rw-------. 1 root root  451 6月  24 11:04 sa.pub

替换 kubeadm

将新生成的 kubeadm 进行替换

cp _output/bin/kubeadm /usr/bin/kubeadm

生成新的证书

cd /etc/kubernetes/pki kubeadm alpha certs renew all

返回内容

[root@master pki]# kubeadm alpha certs renew all
Command "all" is deprecated, please use the same command under "kubeadm certs"
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

 

验证结果

到这里,证书就替换完成了。接下来验证下证书时间是否延长。

kubeadm alpha certs check-expiration

返回信息

[root@master pki]# kubeadm alpha certs check-expiration
Command "check-expiration" is deprecated, please use the same command under "kubeadm certs"
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 May 31, 2121 03:05 UTC   99y                                     no      
apiserver                  May 31, 2121 03:05 UTC   99y             ca                      no      
apiserver-etcd-client      May 31, 2121 03:05 UTC   99y             etcd-ca                 no      
apiserver-kubelet-client   May 31, 2121 03:05 UTC   99y             ca                      no      
controller-manager.conf    May 31, 2121 03:05 UTC   99y                                     no      
etcd-healthcheck-client    May 31, 2121 03:05 UTC   99y             etcd-ca                 no      
etcd-peer                  May 31, 2121 03:05 UTC   99y             etcd-ca                 no      
etcd-server                May 31, 2121 03:05 UTC   99y             etcd-ca                 no      
front-proxy-client         May 31, 2121 03:05 UTC   99y             front-proxy-ca          no      
scheduler.conf             May 31, 2121 03:05 UTC   99y                                     no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jun 20, 2031 07:01 UTC   9y              no      
etcd-ca                 Jun 20, 2031 07:01 UTC   9y              no      
front-proxy-ca          Jun 20, 2031 07:01 UTC   9y              no      

查看 node 状态

[root@master pki]#  kubectl get node
NAME     STATUS   ROLES                  AGE   VERSION
master   Ready    control-plane,master   44h   v1.20.6
node1    Ready    <none>                 43h   v1.20.6

 

Zabbix历史数据处理


2022年3月03日 11:19:55   2,489 次浏览

问题描述

zabbix server 平稳运行有一段时间了,但是最近问题却来了,今天早上收到zabbixserver磁盘空间不足的告警。通过查看之后发现是大部分数据是zabbix 库的的数据 在进一步查看发现是history表和history_uint数据太多导致磁盘占用过多。

问题分析

history_uint

该表存储的是监控项的无符号整型的数据。
该数据的保存时长,取决于在监控项设置的 历史数据保留时长。

history

这个表保存的是浮点型的。
像 history_str 等保存的是 字符型数据。这些都是我们在设置监控项的对应的信息类型决定的。
该数据的保存时长,取决于在监控项设置的 历史数据保留时长

针对这个问题,我打算删除 history_uint 和 history 的一些历史数据。
要删除history_uint里的数据,还需要注意一点,由于数据量比较多,我建议可以分多次少量数据进行删除,因为我一次删除90天的时候CPU已经吃不消了…
这样可以避免一次性删除数据过多导致数据库的负载比较大。(或者可以使用limit 10000)

处理过程

我这里需要删除90天以前的数据下面是我的操作过程

获取时间戳

#通过如下命令进行获取90天以前的时间戳

[root@zabbix-server ~]# date -d $(date -d "-90 day" +%Y%m%d) +%s 1590105600

登录数据库操作

[root@zabbix-server ~]# mysql -uzabbix -p 
Enter password: mysql> use zabbix; 
Database changed 

#delete history_uint 
mysql> delete from history_uint where clock < 1590105600 LIMIT 10000; 
Query OK, 1653 rows affected (1 min 45.42 sec) 

#delete history 
mysql> delete from history where clock < 1590105600 LIMIT 10000; 
Query OK, 0 rows affected (24.72 sec)

释放空间

上面执行删除后,数据的存储空间是没有减少的,因为对于delete from table_name where xxx 带条件的删除,不管是innodb还是MyISAM都不会释放空间,需要进行OPTIMIZE TABLE操作,进行释放空间。

注意:在optimize table ‘表名’ 运行过程中,MySQL会进行锁表。
optimize table history_uin

mysql>  optimize table history_uint;
+---------------------+----------+----------+-------------------------------------------------------------------+
| Table               | Op       | Msg_type | Msg_text                                                          |
+---------------------+----------+----------+-------------------------------------------------------------------+
| zabbix.history_uint | optimize | note     | Table does not support optimize, doing recreate + analyze instead |
| zabbix.history_uint | optimize | status   | OK                                                                |
+---------------------+----------+----------+-------------------------------------------------------------------+
2 rows in set (5 min 33.76 sec)

optimize table history

mysql>  optimize table history;
+----------------+----------+----------+-------------------------------------------------------------------+
| Table          | Op       | Msg_type | Msg_text                                                          |
+----------------+----------+----------+-------------------------------------------------------------------+
| zabbix.history | optimize | note     | Table does not support optimize, doing recreate + analyze instead |
| zabbix.history | optimize | status   | OK                                                                |
+----------------+----------+----------+-------------------------------------------------------------------+
2 rows in set (1 min 39.51 sec)

问题解决

待以上步骤都完成以后,检查磁盘可以看到问题解决 。
不过想要一劳永益的话的话 还是需要写一个脚本来处理这个问题

#!/bin/bash
User="zabbix"
Passwd="zabbix"
Date=`date -d $(date -d "-90 day" +%Y%m%d) +%s`
$(which mysql) -u${User} -p${Passwd} -e "
use zabbix;
DELETE FROM history WHERE 'clock' < '$Date' LIMIT 10000;
optimize table history;
DELETE FROM history_str WHERE 'clock' < '$Date' LIMIT 10000;
optimize table history_str;
DELETE FROM history_uint WHERE 'clock' < '$Date' LIMIT 10000;
optimize table history_uint;
DELETE FROM history_text WHERE 'clock' < $Date' LIMIT 10000;
optimize table history_text;
DELETE FROM  trends WHERE 'clock' < '$Date' LIMIT 10000;
optimize table  trends;
DELETE FROM trends_uint WHERE 'clock' < '$Date' LIMIT 10000;
optimize table trends_uint;
DELETE FROM events WHERE 'clock' < '$Date' LIMIT 10000;
optimize table events;
"

 

另外历史数据过多是由于我们保存的历史数据的时间所致,我们可以根据需求设置历史数据的保留时长,例如一些相对不太重要的数据,我们可以将该值设置的更短一些,这样数据量也就随着减少了。

CentOS 7 OpenVPN一键安装脚本


2022年3月03日 11:06:46   6,989 次浏览

为了避免数据库服务器等内网应用资源暴露到公网中,打算利用VPN 技术实现链接到内网。
本文主要介绍CentOS 7 服务器上安装与配置OpenVPN服务器,以及如何编写客户端连接到新建立的OpenVPN服务器上所需的配置文件

OpenVPN的介绍

OpenVPN是一个开源的应用程序,它允许您通过公共互联网创建一个安全的专用网络。OpenVPN实现一个虚拟专用网(VPN)来创建一个安全连接。OpenVPN使用OpenSSL库提供加密,它提供了几种身份验证机制,如基于证书的、预共享密钥和用户名/密码身份验证。

openvpn 有两种模式

数据包(TUN模式)或数据帧(TAP模式)

  • TUN模式:TUN模拟了网络层设备,第三层数据包如IP封包,底层数据隧道数据
  • TAP模式等同于一个设备,第二操作层数据包如扩展数据帧,创建一个相对桥接接,复杂T

接口接口的好处可见,客户端优化VPN服务子网的IP(忽然忽隐忽现)物理上的区别,可以完全将客户端看做完全与VPN服务器相关的时间,而TUN接口下所有的客户端则出现一个独立的子网内,与VPN服务器相关的子网没有关系,这种使用比较好,和公司的网络区分开,完全是一个虚拟的网络

脚本内容

openvpn 一键安装脚本

#!/bin/bash
#
# https://github.com/Nyr/openvpn-install
#
# Copyright (c) 2013 Nyr. Released under the MIT License.


# Detect Debian users running the script with "sh" instead of bash
if readlink /proc/$$/exe | grep -q "dash"; then
    echo 'This installer needs to be run with "bash", not "sh".'
    exit
fi

# Discard stdin. Needed when running from an one-liner which includes a newline
read -N 999999 -t 0.001

# Detect OpenVZ 6
if [[ $(uname -r | cut -d "." -f 1) -eq 2 ]]; then
    echo "The system is running an old kernel, which is incompatible with this installer."
    exit
fi

# Detect OS
# $os_version variables aren't always in use, but are kept here for convenience
if grep -qs "ubuntu" /etc/os-release; then
    os="ubuntu"
    os_version=$(grep 'VERSION_ID' /etc/os-release | cut -d '"' -f 2 | tr -d '.')
    group_name="nogroup"
elif [[ -e /etc/debian_version ]]; then
    os="debian"
    os_version=$(grep -oE '[0-9]+' /etc/debian_version | head -1)
    group_name="nogroup"
elif [[ -e /etc/centos-release ]]; then
    os="centos"
    os_version=$(grep -oE '[0-9]+' /etc/centos-release | head -1)
    group_name="nobody"
elif [[ -e /etc/fedora-release ]]; then
    os="fedora"
    os_version=$(grep -oE '[0-9]+' /etc/fedora-release | head -1)
    group_name="nobody"
else
    echo "This installer seems to be running on an unsupported distribution.
Supported distributions are Ubuntu, Debian, CentOS, and Fedora."
    exit
fi

if [[ "$os" == "ubuntu" && "$os_version" -lt 1804 ]]; then
    echo "Ubuntu 18.04 or higher is required to use this installer.
This version of Ubuntu is too old and unsupported."
    exit
fi

if [[ "$os" == "debian" && "$os_version" -lt 9 ]]; then
    echo "Debian 9 or higher is required to use this installer.
This version of Debian is too old and unsupported."
    exit
fi

if [[ "$os" == "centos" && "$os_version" -lt 7 ]]; then
    echo "CentOS 7 or higher is required to use this installer.
This version of CentOS is too old and unsupported."
    exit
fi

# Detect environments where $PATH does not include the sbin directories
if ! grep -q sbin <<< "$PATH"; then
    echo '$PATH does not include sbin. Try using "su -" instead of "su".'
    exit
fi

if [[ "$EUID" -ne 0 ]]; then
    echo "This installer needs to be run with superuser privileges."
    exit
fi

if [[ ! -e /dev/net/tun ]] || ! ( exec 7<>/dev/net/tun ) 2>/dev/null; then
    echo "The system does not have the TUN device available.
TUN needs to be enabled before running this installer."
    exit
fi

new_client () {
    # Generates the custom client.ovpn
    {
    cat /etc/openvpn/server/client-common.txt
    echo "<ca>"
    cat /etc/openvpn/server/easy-rsa/pki/ca.crt
    echo "</ca>"
    echo "<cert>"
    sed -ne '/BEGIN CERTIFICATE/,$ p' /etc/openvpn/server/easy-rsa/pki/issued/"$client".crt
    echo "</cert>"
    echo "<key>"
    cat /etc/openvpn/server/easy-rsa/pki/private/"$client".key
    echo "</key>"
    echo "<tls-crypt>"
    sed -ne '/BEGIN OpenVPN Static key/,$ p' /etc/openvpn/server/tc.key
    echo "</tls-crypt>"
    } > ~/"$client".ovpn
}

if [[ ! -e /etc/openvpn/server/server.conf ]]; then
    clear
    echo 'Welcome to this OpenVPN road warrior installer!'
    # If system has a single IPv4, it is selected automatically. Else, ask the user
    if [[ $(ip -4 addr | grep inet | grep -vEc '127(\.[0-9]{1,3}){3}') -eq 1 ]]; then
        ip=$(ip -4 addr | grep inet | grep -vE '127(\.[0-9]{1,3}){3}' | cut -d '/' -f 1 | grep -oE '[0-9]{1,3}(\.[0-9]{1,3}){3}')
    else
        number_of_ip=$(ip -4 addr | grep inet | grep -vEc '127(\.[0-9]{1,3}){3}')
        echo
        echo "Which IPv4 address should be used?"
        ip -4 addr | grep inet | grep -vE '127(\.[0-9]{1,3}){3}' | cut -d '/' -f 1 | grep -oE '[0-9]{1,3}(\.[0-9]{1,3}){3}' | nl -s ') '
        read -p "IPv4 address [1]: " ip_number
        until [[ -z "$ip_number" || "$ip_number" =~ ^[0-9]+$ && "$ip_number" -le "$number_of_ip" ]]; do
            echo "$ip_number: invalid selection."
            read -p "IPv4 address [1]: " ip_number
        done
        [[ -z "$ip_number" ]] && ip_number="1"
        ip=$(ip -4 addr | grep inet | grep -vE '127(\.[0-9]{1,3}){3}' | cut -d '/' -f 1 | grep -oE '[0-9]{1,3}(\.[0-9]{1,3}){3}' | sed -n "$ip_number"p)
    fi
    # If $ip is a private IP address, the server must be behind NAT
    if echo "$ip" | grep -qE '^(10\.|172\.1[6789]\.|172\.2[0-9]\.|172\.3[01]\.|192\.168)'; then
        echo
        echo "This server is behind NAT. What is the public IPv4 address or hostname?"
        # Get public IP and sanitize with grep
        get_public_ip=$(grep -m 1 -oE '^[0-9]{1,3}(\.[0-9]{1,3}){3}$' <<< "$(wget -T 10 -t 1 -4qO- "http://ip1.dynupdate.no-ip.com/" || curl -m 10 -4Ls "http://ip1.dynupdate.no-ip.com/")")
        read -p "Public IPv4 address / hostname [$get_public_ip]: " public_ip
        # If the checkip service is unavailable and user didn't provide input, ask again
        until [[ -n "$get_public_ip" || -n "$public_ip" ]]; do
            echo "Invalid input."
            read -p "Public IPv4 address / hostname: " public_ip
        done
        [[ -z "$public_ip" ]] && public_ip="$get_public_ip"
    fi
    # If system has a single IPv6, it is selected automatically
    if [[ $(ip -6 addr | grep -c 'inet6 [23]') -eq 1 ]]; then
        ip6=$(ip -6 addr | grep 'inet6 [23]' | cut -d '/' -f 1 | grep -oE '([0-9a-fA-F]{0,4}:){1,7}[0-9a-fA-F]{0,4}')
    fi
    # If system has multiple IPv6, ask the user to select one
    if [[ $(ip -6 addr | grep -c 'inet6 [23]') -gt 1 ]]; then
        number_of_ip6=$(ip -6 addr | grep -c 'inet6 [23]')
        echo
        echo "Which IPv6 address should be used?"
        ip -6 addr | grep 'inet6 [23]' | cut -d '/' -f 1 | grep -oE '([0-9a-fA-F]{0,4}:){1,7}[0-9a-fA-F]{0,4}' | nl -s ') '
        read -p "IPv6 address [1]: " ip6_number
        until [[ -z "$ip6_number" || "$ip6_number" =~ ^[0-9]+$ && "$ip6_number" -le "$number_of_ip6" ]]; do
            echo "$ip6_number: invalid selection."
            read -p "IPv6 address [1]: " ip6_number
        done
        [[ -z "$ip6_number" ]] && ip6_number="1"
        ip6=$(ip -6 addr | grep 'inet6 [23]' | cut -d '/' -f 1 | grep -oE '([0-9a-fA-F]{0,4}:){1,7}[0-9a-fA-F]{0,4}' | sed -n "$ip6_number"p)
    fi
    echo
    echo "Which protocol should OpenVPN use?"
    echo "   1) UDP (recommended)"
    echo "   2) TCP"
    read -p "Protocol [1]: " protocol
    until [[ -z "$protocol" || "$protocol" =~ ^[12]$ ]]; do
        echo "$protocol: invalid selection."
        read -p "Protocol [1]: " protocol
    done
    case "$protocol" in
        1|"") 
        protocol=udp
        ;;
        2) 
        protocol=tcp
        ;;
    esac
    echo
    echo "What port should OpenVPN listen to?"
    read -p "Port [1194]: " port
    until [[ -z "$port" || "$port" =~ ^[0-9]+$ && "$port" -le 65535 ]]; do
        echo "$port: invalid port."
        read -p "Port [1194]: " port
    done
    [[ -z "$port" ]] && port="1194"
    echo
    echo "Select a DNS server for the clients:"
    echo "   1) Current system resolvers"
    echo "   2) Google"
    echo "   3) 1.1.1.1"
    echo "   4) OpenDNS"
    echo "   5) Quad9"
    echo "   6) AdGuard"
    read -p "DNS server [1]: " dns
    until [[ -z "$dns" || "$dns" =~ ^[1-6]$ ]]; do
        echo "$dns: invalid selection."
        read -p "DNS server [1]: " dns
    done
    echo
    echo "Enter a name for the first client:"
    read -p "Name [client]: " unsanitized_client
    # Allow a limited set of characters to avoid conflicts
    client=$(sed 's/[^0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_-]/_/g' <<< "$unsanitized_client")
    [[ -z "$client" ]] && client="client"
    echo
    echo "OpenVPN installation is ready to begin."
    # Install a firewall in the rare case where one is not already available
    if ! systemctl is-active --quiet firewalld.service && ! hash iptables 2>/dev/null; then
        if [[ "$os" == "centos" || "$os" == "fedora" ]]; then
            firewall="firewalld"
            # We don't want to silently enable firewalld, so we give a subtle warning
            # If the user continues, firewalld will be installed and enabled during setup
            echo "firewalld, which is required to manage routing tables, will also be installed."
        elif [[ "$os" == "debian" || "$os" == "ubuntu" ]]; then
            # iptables is way less invasive than firewalld so no warning is given
            firewall="iptables"
        fi
    fi
    read -n1 -r -p "Press any key to continue..."
    # If running inside a container, disable LimitNPROC to prevent conflicts
    if systemd-detect-virt -cq; then
        mkdir /etc/systemd/system/openvpn-server@server.service.d/ 2>/dev/null
        echo "[Service]
LimitNPROC=infinity" > /etc/systemd/system/openvpn-server@server.service.d/disable-limitnproc.conf
    fi
    if [[ "$os" = "debian" || "$os" = "ubuntu" ]]; then
        apt-get update
        apt-get install -y openvpn openssl ca-certificates $firewall
    elif [[ "$os" = "centos" ]]; then
        yum install -y epel-release
        yum install -y openvpn openssl ca-certificates tar $firewall
    else
        # Else, OS must be Fedora
        dnf install -y openvpn openssl ca-certificates tar $firewall
    fi
    # If firewalld was just installed, enable it
    if [[ "$firewall" == "firewalld" ]]; then
        systemctl enable --now firewalld.service
    fi
    # Get easy-rsa
    easy_rsa_url='https://github.com/OpenVPN/easy-rsa/releases/download/v3.0.8/EasyRSA-3.0.8.tgz'
    mkdir -p /etc/openvpn/server/easy-rsa/
    { wget -qO- "$easy_rsa_url" 2>/dev/null || curl -sL "$easy_rsa_url" ; } | tar xz -C /etc/openvpn/server/easy-rsa/ --strip-components 1
    chown -R root:root /etc/openvpn/server/easy-rsa/
    cd /etc/openvpn/server/easy-rsa/
    # Create the PKI, set up the CA and the server and client certificates
    ./easyrsa init-pki
    ./easyrsa --batch build-ca nopass
    EASYRSA_CERT_EXPIRE=3650 ./easyrsa build-server-full server nopass
    EASYRSA_CERT_EXPIRE=3650 ./easyrsa build-client-full "$client" nopass
    EASYRSA_CRL_DAYS=3650 ./easyrsa gen-crl
    # Move the stuff we need
    cp pki/ca.crt pki/private/ca.key pki/issued/server.crt pki/private/server.key pki/crl.pem /etc/openvpn/server
    # CRL is read with each client connection, while OpenVPN is dropped to nobody
    chown nobody:"$group_name" /etc/openvpn/server/crl.pem
    # Without +x in the directory, OpenVPN can't run a stat() on the CRL file
    chmod o+x /etc/openvpn/server/
    # Generate key for tls-crypt
    openvpn --genkey --secret /etc/openvpn/server/tc.key
    # Create the DH parameters file using the predefined ffdhe2048 group
    echo '-----BEGIN DH PARAMETERS-----
MIIBCAKCAQEA//////////+t+FRYortKmq/cViAnPTzx2LnFg84tNpWp4TZBFGQz
+8yTnc4kmz75fS/jY2MMddj2gbICrsRhetPfHtXV/WVhJDP1H18GbtCFY2VVPe0a
87VXE15/V8k1mE8McODmi3fipona8+/och3xWKE2rec1MKzKT0g6eXq8CrGCsyT7
YdEIqUuyyOP7uWrat2DX9GgdT0Kj3jlN9K5W7edjcrsZCwenyO4KbXCeAvzhzffi
7MA0BM0oNC9hkXL+nOmFg/+OTxIy7vKBg8P+OxtMb61zO7X8vC7CIAXFjvGDfRaD
ssbzSibBsu/6iGtCOGEoXJf//////////wIBAg==
-----END DH PARAMETERS-----' > /etc/openvpn/server/dh.pem
    # Generate server.conf
    echo "local $ip
port $port
proto $protocol
dev tun
ca ca.crt
cert server.crt
key server.key
dh dh.pem
auth SHA512
tls-crypt tc.key
topology subnet
server 10.8.0.0 255.255.255.0" > /etc/openvpn/server/server.conf
    # IPv6
    if [[ -z "$ip6" ]]; then
        echo 'push "redirect-gateway def1 bypass-dhcp"' >> /etc/openvpn/server/server.conf
    else
        echo 'server-ipv6 fddd:1194:1194:1194::/64' >> /etc/openvpn/server/server.conf
        echo 'push "redirect-gateway def1 ipv6 bypass-dhcp"' >> /etc/openvpn/server/server.conf
    fi
    echo 'ifconfig-pool-persist ipp.txt' >> /etc/openvpn/server/server.conf
    # DNS
    case "$dns" in
        1|"")
            # Locate the proper resolv.conf
            # Needed for systems running systemd-resolved
            if grep -q '^nameserver 127.0.0.53' "/etc/resolv.conf"; then
                resolv_conf="/run/systemd/resolve/resolv.conf"
            else
                resolv_conf="/etc/resolv.conf"
            fi
            # Obtain the resolvers from resolv.conf and use them for OpenVPN
            grep -v '^#\|^;' "$resolv_conf" | grep '^nameserver' | grep -oE '[0-9]{1,3}(\.[0-9]{1,3}){3}' | while read line; do
                echo "push \"dhcp-option DNS $line\"" >> /etc/openvpn/server/server.conf
            done
        ;;
        2)
            echo 'push "dhcp-option DNS 8.8.8.8"' >> /etc/openvpn/server/server.conf
            echo 'push "dhcp-option DNS 8.8.4.4"' >> /etc/openvpn/server/server.conf
        ;;
        3)
            echo 'push "dhcp-option DNS 1.1.1.1"' >> /etc/openvpn/server/server.conf
            echo 'push "dhcp-option DNS 1.0.0.1"' >> /etc/openvpn/server/server.conf
        ;;
        4)
            echo 'push "dhcp-option DNS 208.67.222.222"' >> /etc/openvpn/server/server.conf
            echo 'push "dhcp-option DNS 208.67.220.220"' >> /etc/openvpn/server/server.conf
        ;;
        5)
            echo 'push "dhcp-option DNS 9.9.9.9"' >> /etc/openvpn/server/server.conf
            echo 'push "dhcp-option DNS 149.112.112.112"' >> /etc/openvpn/server/server.conf
        ;;
        6)
            echo 'push "dhcp-option DNS 94.140.14.14"' >> /etc/openvpn/server/server.conf
            echo 'push "dhcp-option DNS 94.140.15.15"' >> /etc/openvpn/server/server.conf
        ;;
    esac
    echo "keepalive 10 120
cipher AES-256-CBC
user nobody
group $group_name
persist-key
persist-tun
verb 3
crl-verify crl.pem" >> /etc/openvpn/server/server.conf
    if [[ "$protocol" = "udp" ]]; then
        echo "explicit-exit-notify" >> /etc/openvpn/server/server.conf
    fi
    # Enable net.ipv4.ip_forward for the system
    echo 'net.ipv4.ip_forward=1' > /etc/sysctl.d/99-openvpn-forward.conf
    # Enable without waiting for a reboot or service restart
    echo 1 > /proc/sys/net/ipv4/ip_forward
    if [[ -n "$ip6" ]]; then
        # Enable net.ipv6.conf.all.forwarding for the system
        echo "net.ipv6.conf.all.forwarding=1" >> /etc/sysctl.d/99-openvpn-forward.conf
        # Enable without waiting for a reboot or service restart
        echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
    fi
    if systemctl is-active --quiet firewalld.service; then
        # Using both permanent and not permanent rules to avoid a firewalld
        # reload.
        # We don't use --add-service=openvpn because that would only work with
        # the default port and protocol.
        firewall-cmd --add-port="$port"/"$protocol"
        firewall-cmd --zone=trusted --add-source=10.8.0.0/24
        firewall-cmd --permanent --add-port="$port"/"$protocol"
        firewall-cmd --permanent --zone=trusted --add-source=10.8.0.0/24
        # Set NAT for the VPN subnet
        firewall-cmd --direct --add-rule ipv4 nat POSTROUTING 0 -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j SNAT --to "$ip"
        firewall-cmd --permanent --direct --add-rule ipv4 nat POSTROUTING 0 -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j SNAT --to "$ip"
        if [[ -n "$ip6" ]]; then
            firewall-cmd --zone=trusted --add-source=fddd:1194:1194:1194::/64
            firewall-cmd --permanent --zone=trusted --add-source=fddd:1194:1194:1194::/64
            firewall-cmd --direct --add-rule ipv6 nat POSTROUTING 0 -s fddd:1194:1194:1194::/64 ! -d fddd:1194:1194:1194::/64 -j SNAT --to "$ip6"
            firewall-cmd --permanent --direct --add-rule ipv6 nat POSTROUTING 0 -s fddd:1194:1194:1194::/64 ! -d fddd:1194:1194:1194::/64 -j SNAT --to "$ip6"
        fi
    else
        # Create a service to set up persistent iptables rules
        iptables_path=$(command -v iptables)
        ip6tables_path=$(command -v ip6tables)
        # nf_tables is not available as standard in OVZ kernels. So use iptables-legacy
        # if we are in OVZ, with a nf_tables backend and iptables-legacy is available.
        if [[ $(systemd-detect-virt) == "openvz" ]] && readlink -f "$(command -v iptables)" | grep -q "nft" && hash iptables-legacy 2>/dev/null; then
            iptables_path=$(command -v iptables-legacy)
            ip6tables_path=$(command -v ip6tables-legacy)
        fi
        echo "[Unit]
Before=network.target
[Service]
Type=oneshot
ExecStart=$iptables_path -t nat -A POSTROUTING -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j SNAT --to $ip
ExecStart=$iptables_path -I INPUT -p $protocol --dport $port -j ACCEPT
ExecStart=$iptables_path -I FORWARD -s 10.8.0.0/24 -j ACCEPT
ExecStart=$iptables_path -I FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
ExecStop=$iptables_path -t nat -D POSTROUTING -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j SNAT --to $ip
ExecStop=$iptables_path -D INPUT -p $protocol --dport $port -j ACCEPT
ExecStop=$iptables_path -D FORWARD -s 10.8.0.0/24 -j ACCEPT
ExecStop=$iptables_path -D FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT" > /etc/systemd/system/openvpn-iptables.service
        if [[ -n "$ip6" ]]; then
            echo "ExecStart=$ip6tables_path -t nat -A POSTROUTING -s fddd:1194:1194:1194::/64 ! -d fddd:1194:1194:1194::/64 -j SNAT --to $ip6
ExecStart=$ip6tables_path -I FORWARD -s fddd:1194:1194:1194::/64 -j ACCEPT
ExecStart=$ip6tables_path -I FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
ExecStop=$ip6tables_path -t nat -D POSTROUTING -s fddd:1194:1194:1194::/64 ! -d fddd:1194:1194:1194::/64 -j SNAT --to $ip6
ExecStop=$ip6tables_path -D FORWARD -s fddd:1194:1194:1194::/64 -j ACCEPT
ExecStop=$ip6tables_path -D FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT" >> /etc/systemd/system/openvpn-iptables.service
        fi
        echo "RemainAfterExit=yes
[Install]
WantedBy=multi-user.target" >> /etc/systemd/system/openvpn-iptables.service
        systemctl enable --now openvpn-iptables.service
    fi
    # If SELinux is enabled and a custom port was selected, we need this
    if sestatus 2>/dev/null | grep "Current mode" | grep -q "enforcing" && [[ "$port" != 1194 ]]; then
        # Install semanage if not already present
        if ! hash semanage 2>/dev/null; then
            if [[ "$os_version" -eq 7 ]]; then
                # Centos 7
                yum install -y policycoreutils-python
            else
                # CentOS 8 or Fedora
                dnf install -y policycoreutils-python-utils
            fi
        fi
        semanage port -a -t openvpn_port_t -p "$protocol" "$port"
    fi
    # If the server is behind NAT, use the correct IP address
    [[ -n "$public_ip" ]] && ip="$public_ip"
    # client-common.txt is created so we have a template to add further users later
    echo "client
dev tun
proto $protocol
remote $ip $port
resolv-retry infinite
nobind
persist-key
persist-tun
remote-cert-tls server
auth SHA512
cipher AES-256-CBC
ignore-unknown-option block-outside-dns
block-outside-dns
verb 3" > /etc/openvpn/server/client-common.txt
    # Enable and start the OpenVPN service
    systemctl enable --now openvpn-server@server.service
    # Generates the custom client.ovpn
    new_client
    echo
    echo "Finished!"
    echo
    echo "The client configuration is available in:" ~/"$client.ovpn"
    echo "New clients can be added by running this script again."
else
    clear
    echo "OpenVPN is already installed."
    echo
    echo "Select an option:"
    echo "   1) Add a new client"
    echo "   2) Revoke an existing client"
    echo "   3) Remove OpenVPN"
    echo "   4) Exit"
    read -p "Option: " option
    until [[ "$option" =~ ^[1-4]$ ]]; do
        echo "$option: invalid selection."
        read -p "Option: " option
    done
    case "$option" in
        1)
            echo
            echo "Provide a name for the client:"
            read -p "Name: " unsanitized_client
            client=$(sed 's/[^0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_-]/_/g' <<< "$unsanitized_client")
            while [[ -z "$client" || -e /etc/openvpn/server/easy-rsa/pki/issued/"$client".crt ]]; do
                echo "$client: invalid name."
                read -p "Name: " unsanitized_client
                client=$(sed 's/[^0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_-]/_/g' <<< "$unsanitized_client")
            done
            cd /etc/openvpn/server/easy-rsa/
            EASYRSA_CERT_EXPIRE=3650 ./easyrsa build-client-full "$client" nopass
            # Generates the custom client.ovpn
            new_client
            echo
            echo "$client added. Configuration available in:" ~/"$client.ovpn"
            exit
        ;;
        2)
            # This option could be documented a bit better and maybe even be simplified
            # ...but what can I say, I want some sleep too
            number_of_clients=$(tail -n +2 /etc/openvpn/server/easy-rsa/pki/index.txt | grep -c "^V")
            if [[ "$number_of_clients" = 0 ]]; then
                echo
                echo "There are no existing clients!"
                exit
            fi
            echo
            echo "Select the client to revoke:"
            tail -n +2 /etc/openvpn/server/easy-rsa/pki/index.txt | grep "^V" | cut -d '=' -f 2 | nl -s ') '
            read -p "Client: " client_number
            until [[ "$client_number" =~ ^[0-9]+$ && "$client_number" -le "$number_of_clients" ]]; do
                echo "$client_number: invalid selection."
                read -p "Client: " client_number
            done
            client=$(tail -n +2 /etc/openvpn/server/easy-rsa/pki/index.txt | grep "^V" | cut -d '=' -f 2 | sed -n "$client_number"p)
            echo
            read -p "Confirm $client revocation? [y/N]: " revoke
            until [[ "$revoke" =~ ^[yYnN]*$ ]]; do
                echo "$revoke: invalid selection."
                read -p "Confirm $client revocation? [y/N]: " revoke
            done
            if [[ "$revoke" =~ ^[yY]$ ]]; then
                cd /etc/openvpn/server/easy-rsa/
                ./easyrsa --batch revoke "$client"
                EASYRSA_CRL_DAYS=3650 ./easyrsa gen-crl
                rm -f /etc/openvpn/server/crl.pem
                cp /etc/openvpn/server/easy-rsa/pki/crl.pem /etc/openvpn/server/crl.pem
                # CRL is read with each client connection, when OpenVPN is dropped to nobody
                chown nobody:"$group_name" /etc/openvpn/server/crl.pem
                echo
                echo "$client revoked!"
            else
                echo
                echo "$client revocation aborted!"
            fi
            exit
        ;;
        3)
            echo
            read -p "Confirm OpenVPN removal? [y/N]: " remove
            until [[ "$remove" =~ ^[yYnN]*$ ]]; do
                echo "$remove: invalid selection."
                read -p "Confirm OpenVPN removal? [y/N]: " remove
            done
            if [[ "$remove" =~ ^[yY]$ ]]; then
                port=$(grep '^port ' /etc/openvpn/server/server.conf | cut -d " " -f 2)
                protocol=$(grep '^proto ' /etc/openvpn/server/server.conf | cut -d " " -f 2)
                if systemctl is-active --quiet firewalld.service; then
                    ip=$(firewall-cmd --direct --get-rules ipv4 nat POSTROUTING | grep '\-s 10.8.0.0/24 '"'"'!'"'"' -d 10.8.0.0/24' | grep -oE '[^ ]+$')
                    # Using both permanent and not permanent rules to avoid a firewalld reload.
                    firewall-cmd --remove-port="$port"/"$protocol"
                    firewall-cmd --zone=trusted --remove-source=10.8.0.0/24
                    firewall-cmd --permanent --remove-port="$port"/"$protocol"
                    firewall-cmd --permanent --zone=trusted --remove-source=10.8.0.0/24
                    firewall-cmd --direct --remove-rule ipv4 nat POSTROUTING 0 -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j SNAT --to "$ip"
                    firewall-cmd --permanent --direct --remove-rule ipv4 nat POSTROUTING 0 -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j SNAT --to "$ip"
                    if grep -qs "server-ipv6" /etc/openvpn/server/server.conf; then
                        ip6=$(firewall-cmd --direct --get-rules ipv6 nat POSTROUTING | grep '\-s fddd:1194:1194:1194::/64 '"'"'!'"'"' -d fddd:1194:1194:1194::/64' | grep -oE '[^ ]+$')
                        firewall-cmd --zone=trusted --remove-source=fddd:1194:1194:1194::/64
                        firewall-cmd --permanent --zone=trusted --remove-source=fddd:1194:1194:1194::/64
                        firewall-cmd --direct --remove-rule ipv6 nat POSTROUTING 0 -s fddd:1194:1194:1194::/64 ! -d fddd:1194:1194:1194::/64 -j SNAT --to "$ip6"
                        firewall-cmd --permanent --direct --remove-rule ipv6 nat POSTROUTING 0 -s fddd:1194:1194:1194::/64 ! -d fddd:1194:1194:1194::/64 -j SNAT --to "$ip6"
                    fi
                else
                    systemctl disable --now openvpn-iptables.service
                    rm -f /etc/systemd/system/openvpn-iptables.service
                fi
                if sestatus 2>/dev/null | grep "Current mode" | grep -q "enforcing" && [[ "$port" != 1194 ]]; then
                    semanage port -d -t openvpn_port_t -p "$protocol" "$port"
                fi
                systemctl disable --now openvpn-server@server.service
                rm -rf /etc/openvpn/server
                rm -f /etc/systemd/system/openvpn-server@server.service.d/disable-limitnproc.conf
                rm -f /etc/sysctl.d/99-openvpn-forward.conf
                if [[ "$os" = "debian" || "$os" = "ubuntu" ]]; then
                    apt-get remove --purge -y openvpn
                else
                    # Else, OS must be CentOS or Fedora
                    yum remove -y openvpn
                fi
                echo
                echo "OpenVPN removed!"
            else
                echo
                echo "OpenVPN removal aborted!"
            fi
            exit
        ;;
        4)
            exit
        ;;
    esac
fi