Kubespray로 쿠버네티스 설치하기

 

Kubespray를 이용하여 쿠버네티스를 설치하는 가이드

 

• Ansible의 playbook과 inventory 설정으로 Kubernetes 클러스터 설치
• Kubernetes는 3개의 노드로 구성하기로함(control-pannel, kube-node(2))
• OS : Linux ubuntu 22.04

ansible을 이용하므로 모든 vm끼리는 ssh 통신이 되어야 함.

1. ssh key 생성 및 복사

ssh-keygen 명령어를 터미널에서 날리게 되면 prompt에 물어보는게 너무 많은데 이런거 매번 엔터치기 귀찮으니까

한방에 키 생성

ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa <<< y > /dev/null 2>&1

 

shell script에 IP 정보를 미리 입력해 놓는다. 이렇게 해놓으면 굳이 계속해서 IP를 찾을 필요가 없다. 미리 하나의 파일로 관리하자

cat << EOF | tee ~/preinstall.sh
#!/bin/bash

##########################################################################

# IPs
export IP1="192.168.56.100"
export IP2="192.168.56.101"
export IP3="192.168.56.102"

# ClusterDir
export CLUSTER_DIR="mycluster"

##########################################################################
EOF

source ~/preinstall.sh

cat << EOF | sudo tee -a /etc/hosts

##########################################################################
${IP1}    master
${IP2}    worker1
${IP3}    worker2
EOF

 

host를 등록했으니 이제 키를 등록해주자

ssh-copy-id master
ssh-copy-id worker1
ssh-copy-id worker2

 

ssh 등록이 완료됐으니 이제 kubespray를 시작해보자.

git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray

kubespray를 다운 받고 그 폴더로 들어가자.

 

먼저 python 의존성을 설치해야하는데, pip3, python3 혹은 python이 3.xx버전인지 확인해야한다. 그리고 나서

pip 이 없으면 pip 을깔자

# pip 안깔려있을 때
# service는 재시작하자
sudo apt update && sudo apt install python3-pip -y
pip3 install -r requirements.txt

그리고 파이썬 의존성을 다운받자. 

 

의존성을 다운 받았으면 이제 ansible의 inventory를 만들 차례다. ansible의 inventory는 그냥 hosts관리 모음이라 생각ㅎ면 편하다.

cp -rfp inventory/sample inventory/${CLUSTER_DIR}
declare -a IPS=($IP1 $IP2 $IP3)
CONFIG_FILE=inventory/${CLUSTER_DIR}/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

# hosts.yaml 파일이 만들어졌는지 확인
ll inventory/mycluster/ | grep hosts
# -rw-rw-r-- 1 vagrant vagrant  654 Jan 31 15:58 hosts.yaml

 

이제 hosts파일을 수정하자.

vim inventory/mycluster/hosts.yaml

 

기본적으로 이렇게 되어 있는데

all:
  hosts:
    node1:
      ansible_host: 192.168.56.100
      ip: 192.168.56.100
      access_ip: 192.168.56.100
    node2:
      ansible_host: 192.168.56.101
      ip: 192.168.56.101
      access_ip: 192.168.56.101
    node3:
      ansible_host: 192.168.56.102
      ip: 192.168.56.102
      access_ip: 192.168.56.102
....

이 파일을 다음과 같이 수정하자.

 

자동화파일인데 이정도는 감수하자.

all:
  hosts:
    master:
      ansible_host: 192.168.56.100
      ip: 192.168.56.100
      access_ip: 192.168.56.100

      # user 이름 변경
      ansible_ssh_user: vagrant

      # ssh key파일
      ansible_ssh_private_key_file: ~/.ssh/id_rsa
    worker1:
      ansible_host: 192.168.56.101
      ip: 192.168.56.101
      access_ip: 192.168.56.101

      # user 이름 변경
      ansible_ssh_user: vagrant

      # ssh key파일
      ansible_ssh_private_key_file: ~/.ssh/id_rsa
    worker2:
      ansible_host: 192.168.56.102
      ip: 192.168.56.102
      access_ip: 192.168.56.102

      # user 이름 변경
      ansible_ssh_user: vagrant

      # ssh key파일
      ansible_ssh_private_key_file: ~/.ssh/id_rsa
  children:
    kube_control_plane:
      hosts:
        master:
    kube_node:
      hosts:
        worker1:
        worker2:
    etcd:
      hosts:
        master:
    k8s_cluster:
      children:
        kube_control_plane:
        kube_node:
        calico_rr:
    calico_rr:

 

$ ansible all -m ping -i inventory/mycluster/hosts.yaml 

[WARNING]: Skipping callback plugin 'ara_default', unable to load
worker2 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
worker1 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
master | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

위명령어로 ping 이 튕기는지 확인하자 제대로 success뜨면 이제 준비가 된거다.

 

kubespray 폴더에서 다음을 명령어로 실행한다.

cat <<EOF | tee preinstall.yml
---
- name: PreInstall Kubernetes
  ansible.builtin.import_playbook: playbooks/preinstall.yml
EOF

cat <<EOF | tee playbooks/preinstall.yml
---
- hosts: all
  tasks:
    - name: "Disabling Firewall"
      become: true
      community.general.ufw:
        state: disabled

    - name: "Disabling Swap on all nodes"
      become: true
      shell:
        cmd: swapoff -a

    - name: "Commenting Swap entries in /etc/fstab"
      become: true
      replace:
        path: /etc/fstab
        regexp: "(.*swap*)"
        replace: '#\1'

    - name: "Installing NTP"
      become: true
      ansible.builtin.apt:
        name: ntp
        update_cache: true
        state: latest

    - name: "Syncronize date"
      become: true
      shell:
        cmd: ntpq -p

    - name: "Restarting NTP service"
      become: true
      service:
        name: ntp
        state: restarted
        enabled: true

    - name: "Set /proc/sys/net/ipv4/ip_forward to 1"
      become: true
      shell:
        cmd: echo '1' > /proc/sys/net/ipv4/ip_forward

    - name: "Configuring container runtime using containerd"
      become: true
      file:
        path: /etc/modules-load.d/containerd.conf
        state: touch

    - name: "Adding Configuring container runtime using containerd"
      become: true
      blockinfile:
        path: /etc/modules-load.d/containerd.conf
        block: |
          overlay
          br_netfilter

    - name: "Register the modules required by modprobe in order to the kernel"
      become: true
      shell:
        cmd: |
          modprobe overlay
          modprobe br_netfilter

    - name: "Create /etc/sysctl.d/99-kubernetes-cri.conf"
      become: true
      file:
        path: /etc/sysctl.d/99-kubernetes-cri.conf
        state: touch

    - name: "Setting /etc/sysctl.d/99-kubernetes-cri.conf"
      become: true
      blockinfile:
        path: /etc/sysctl.d/99-kubernetes-cri.conf
        block: |
          net.bridge.bridge-nf-call-iptables  = 1
          net.ipv4.ip_forward                 = 1
          net.bridge.bridge-nf-call-ip6tables = 1

    - name: "Create /etc/modules-load.d/k8s.conf"
      become: true
      file:
        path: /etc/modules-load.d/k8s.conf
        state: touch

    - name: "Setting /etc/modules-load.d/k8s.conf"
      become: true
      blockinfile:
        path: /etc/modules-load.d/k8s.conf
        block: br_netfilter

    - name: "Create /etc/sysctl.d/k8s.conf"
      become: true
      file:
        path: /etc/sysctl.d/k8s.conf
        state: touch

    - name: "Setting /etc/sysctl.d/k8s.conf"
      become: true
      blockinfile:
        path: /etc/sysctl.d/k8s.conf
        block: |
          net.bridge.bridge-nf-call-ip6tables = 1
          net.bridge.bridge-nf-call-iptables  = 1
          net.ipv4.ip_forward                 = 1
EOF

cat <<EOF | tee -a playbooks/cluster.yml

- name: "Copy File /etc/kubernetes/admin.conf"
  hosts: kube_control_plane
  become: true
  tasks:
    - name: "/etc/kubernetes/admin.conf -> "
      copy:
        src: "/etc/kubernetes/admin.conf"
        dest: "{{ ansible_user_dir }}/.kube/config"
        owner: "{{ ansible_user }}"
        group: "{{ ansible_user }}"
        remote_src: true
EOF

 

이제  yaml파일 추가가 끝났으니 실행해보자

$ ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root preinstall.yml 
$ ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml

수분이 지난 후에 완료된다.!!