Failed to add control-plane node #10973
Comments
|
I have the same problem when trying to replace a first-controlplane node. The docs https://github.com/kubernetes-sigs/kubespray/blob/master/docs/nodes.md just dont work. |
|
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
k8s-ci-robot
added
lifecycle/rotten
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close not-planned |
|
@k8s-triage-robot: Closing this issue, marking it as "Not Planned". In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
What happened?
Playbook cluster.yml fails when adding a new control plane node to an existing cluster.
What did you expect to happen?
Playbook cluster.yml succeeds.
How can we reproduce it (as minimally and precisely as possible)?
In a cluster deployed with Kubespray :
OS
Linux 6.1.0-18-amd64 x86_64
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
Version of Ansible
ansible [core 2.16.2]
config file = /home/nicolas/git/kub/k8s-leno/kubespray/ansible.cfg
configured module search path = ['/home/nicolas/git/kub/k8s-leno/kubespray/library']
ansible python module location = /home/nicolas/ansible/lib/python3.11/site-packages/ansible
ansible collection location = /home/nicolas/.ansible/collections:/usr/share/ansible/collections
executable location = /home/nicolas/ansible/bin/ansible
python version = 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] (/home/nicolas/ansible/bin/python3)
jinja version = 3.1.3
libyaml = True
Version of Python
Python 3.11.2
Version of Kubespray (commit)
8760abf
Network plugin used
calico
Full inventory with variables
localhost | SUCCESS =>
{ "hostvars[inventory_hostname]": { "access_ip": "{{ ansible_host }}", "additional_sysctl": [ { "name": "fs.inotify.max_user_instances", "value": 512 } ], "ansible_check_mode": false, "ansible_config_file": "/home/nicolas/git/kub/k8s-leno/ansible.cfg", "ansible_connection": "local", "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 10, "ansible_inventory_sources": [ "/home/nicolas/git/kub/k8s-leno/inventory/leno" ], "ansible_playbook_python": "/home/nicolas/ansible/bin/python3", "ansible_python_interpreter": "/home/nicolas/ansible/bin/python3", "ansible_user": "core", "ansible_verbosity": 0, "ansible_version": { "full": "2.16.2", "major": 2, "minor": 16, "revision": 2, "string": "2.16.2" }, "coreos_zincati_disable": true, "download_container": false, "download_localhost": true, "download_run_once": true, "group_names": [ "ungrouped" ], "groups": { "all": [ "localhost", "lenop", "leno1", "leno2", "leno3", "leno4" ], "calico_rr": [], "etcd": [ "lenop", "leno3", "leno4" ], "k8s_cluster": [ "lenop", "leno3", "leno4", "leno1", "leno2" ], "kube_control_plane": [ "lenop", "leno3", "leno4" ], "kube_node": [ "leno1", "leno2", "leno3", "leno4", "lenop" ], "kube_storage_nodes": [ "leno1", "leno2", "leno3", "leno4" ], "metadata": [], "ungrouped": [ "localhost" ] }, "if_name": "eth0", "inventory_dir": "/home/nicolas/git/kub/k8s-leno/inventory/leno", "inventory_file": "/home/nicolas/git/kub/k8s-leno/inventory/leno/hosts.yml", "inventory_hostname": "localhost", "inventory_hostname_short": "localhost", "ip": "{{ ansible_host }}", "is_fedora_coreos": true, "kube_vip_address": "192.168.1.250", "kube_vip_services": "{{ kube_lb_addresses }}", "ntp_manage_config": true, "ntp_servers": [ "0.fr.pool.ntp.org iburst", "1.fr.pool.ntp.org iburst", "2.fr.pool.ntp.org iburst", "3.fr.pool.ntp.org iburst" ], "omit": "__omit_place_holder__6d1590d3cd4c0ff5954bce6dd24b6053512da6ee", "ping_access_ip": false, "playbook_dir": "/home/nicolas/git/kub/k8s-leno", "service_pools": { "default": [ "{{ kube_lb_addresses }}" ] }, "sysctl_file_path": "/etc/sysctl.d/99-kubernetes.conf", "upstream_dns_servers": [ "192.168.1.1" ] } }leno1 | SUCCESS =>
{ "hostvars[inventory_hostname]": { "access_ip": "192.168.1.251", "additional_sysctl": [ { "name": "vm.max_map_count", "value": 262144 } ], "ansible_check_mode": false, "ansible_config_file": "/home/nicolas/git/kub/k8s-leno/ansible.cfg", "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 10, "ansible_host": "192.168.1.251", "ansible_inventory_sources": [ "/home/nicolas/git/kub/k8s-leno/inventory/leno" ], "ansible_playbook_python": "/home/nicolas/ansible/bin/python3", "ansible_user": "core", "ansible_verbosity": 0, "ansible_version": { "full": "2.16.2", "major": 2, "minor": 16, "revision": 2, "string": "2.16.2" }, "audit_log_maxage": 7, "audit_log_maxbackups": 10, "audit_log_maxsize": 100, "audit_log_path": "/var/log/audit/kube-apiserver-audit.json", "authorization_modes": [ "Node", "RBAC" ], "auto_renew_certificates": true, "coreos_zincati_disable": true, "dashboard_enabled": false, "dns_mode": "manual", "download_container": false, "download_localhost": true, "download_run_once": true, "drain_grace_period": 300, "drain_retries": 3, "drain_timeout": "600s", "enable_nodelocaldns": false, "etcd_deployment_type": "kubeadm", "etcd_metrics_port": 2381, "etcd_metrics_service_labels": { "app": "kube-prometheus-stack-kube-etcd", "app.kubernetes.io/managed-by": "Kubespray", "k8s-app": "etcd", "release": "prometheus-stack" }, "group_names": [ "k8s_cluster", "kube_node", "kube_storage_nodes" ], "groups": { "all": [ "localhost", "lenop", "leno1", "leno2", "leno3", "leno4" ], "calico_rr": [], "etcd": [ "lenop", "leno3", "leno4" ], "k8s_cluster": [ "lenop", "leno3", "leno4", "leno1", "leno2" ], "kube_control_plane": [ "lenop", "leno3", "leno4" ], "kube_node": [ "leno1", "leno2", "leno3", "leno4", "lenop" ], "kube_storage_nodes": [ "leno1", "leno2", "leno3", "leno4" ], "metadata": [], "ungrouped": [ "localhost" ] }, "if_name": "eth0", "inventory_dir": "/home/nicolas/git/kub/k8s-leno/inventory/leno", "inventory_file": "/home/nicolas/git/kub/k8s-leno/inventory/leno/hosts.yml", "inventory_hostname": "leno1", "inventory_hostname_short": "leno1", "ip": "192.168.1.251", "is_fedora_coreos": true, "is_storage_node": true, "kube_apiserver_admission_control_config_file": true, "kube_apiserver_admission_event_rate_limits": { "limit_1": { "burst": 1000, "cache_size": 4000, "qps": 100, "type": "Namespace" }, "limit_2": { "burst": 1000, "qps": 100, "type": "User" } }, "kube_apiserver_enable_admission_plugins": [ "EventRateLimit", "AlwaysPullImages", "ServiceAccount", "NamespaceLifecycle", "NodeRestriction", "LimitRanger", "ResourceQuota", "MutatingAdmissionWebhook", "ValidatingAdmissionWebhook", "PodNodeSelector", "PodSecurity" ], "kube_apiserver_request_timeout": "120s", "kube_apiserver_service_account_lookup": true, "kube_cert_group": "root", "kube_controller_feature_gates": [ "RotateKubeletServerCertificate=true" ], "kube_controller_manager_bind_address": "127.0.0.1", "kube_controller_terminated_pod_gc_threshold": 50, "kube_encrypt_secret_data": true, "kube_encryption_algorithm": "secretbox", "kube_encryption_resources": [ "secrets" ], "kube_lb_addresses": "192.168.1.160/27", "kube_master_cpu_reserved": "200m", "kube_master_memory_reserved": "128Mi", "kube_network_node_prefix": 24, "kube_network_plugin": "cni", "kube_network_plugin_multus": false, "kube_owner": "root", "kube_pod_security_default_enforce": "restricted", "kube_pod_security_use_default": true, "kube_pods_subnet": "10.20.64.0/18", "kube_profiling": false, "kube_proxy_remove": true, "kube_proxy_strict_arp": true, "kube_read_only_port": 0, "kube_scheduler_bind_address": "127.0.0.1", "kube_service_addresses": "10.20.0.0/18", "kube_vip_address": "192.168.1.250", "kube_vip_arp_enabled": true, "kube_vip_controlplane_enabled": true, "kube_vip_enabled": true, "kube_vip_services": "192.168.1.160/27", "kube_vip_services_enabled": false, "kubeadm_feature_gates": [ "PublicKeysECDSA=true", "EtcdLearnerMode=true" ], "kubeadm_scale_down_coredns_enabled": false, "kubeconfig_localhost": true, "kubectl_localhost": true, "kubelet_authentication_token_webhook": true, "kubelet_authorization_mode_webhook": true, "kubelet_csr_approver_enabled": false, "kubelet_event_record_qps": 1, "kubelet_feature_gates": [ "RotateKubeletServerCertificate=true", "SeccompDefault=true" ], "kubelet_image_gc_high_threshold": 90, "kubelet_image_gc_low_threshold": 60, "kubelet_make_iptables_util_chains": false, "kubelet_max_pods": 250, "kubelet_protect_kernel_defaults": true, "kubelet_rotate_certificates": true, "kubelet_rotate_server_certificates": true, "kubelet_streaming_connection_idle_timeout": "5m", "kubernetes_audit": false, "loadbalancer_apiserver": { "address": "192.168.1.250", "port": 6443 }, "loadbalancer_apiserver_localhost": false, "manual_dns_server": "10.20.0.10", "node_labels": { "node-role.kubernetes.io/storage": "true", "node.longhorn.io/create-default-disk": "true" }, "ntp_manage_config": true, "ntp_servers": [ "0.fr.pool.ntp.org iburst", "1.fr.pool.ntp.org iburst", "2.fr.pool.ntp.org iburst", "3.fr.pool.ntp.org iburst" ], "omit": "__omit_place_holder__6d1590d3cd4c0ff5954bce6dd24b6053512da6ee", "ping_access_ip": false, "playbook_dir": "/home/nicolas/git/kub/k8s-leno", "resolvconf_mode": "none", "resolved_conf_customization": [ "LLMNR=no", "MulticastDNS=no", "ReadEtcHosts=yes" ], "service_pools": { "default": [ "192.168.1.160/27" ] }, "sysctl_file_path": "/etc/sysctl.d/99-kubernetes.conf", "system_cpu_reserved": "200m", "system_memory_reserved": "128Mi", "tls_cipher_suites": [ "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305" ], "tls_min_version": "VersionTLS12", "upstream_dns_servers": [ "192.168.1.1" ] } }lenop | SUCCESS =>
{ "hostvars[inventory_hostname]": { "access_ip": "192.168.1.254", "additional_sysctl": [ { "name": "vm.max_map_count", "value": 262144 } ], "ansible_check_mode": false, "ansible_config_file": "/home/nicolas/git/kub/k8s-leno/ansible.cfg", "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 10, "ansible_host": "192.168.1.254", "ansible_inventory_sources": [ "/home/nicolas/git/kub/k8s-leno/inventory/leno" ], "ansible_playbook_python": "/home/nicolas/ansible/bin/python3", "ansible_user": "core", "ansible_verbosity": 0, "ansible_version": { "full": "2.16.2", "major": 2, "minor": 16, "revision": 2, "string": "2.16.2" }, "audit_log_maxage": 7, "audit_log_maxbackups": 10, "audit_log_maxsize": 100, "audit_log_path": "/var/log/audit/kube-apiserver-audit.json", "authorization_modes": [ "Node", "RBAC" ], "auto_renew_certificates": true, "coreos_zincati_disable": true, "dashboard_enabled": false, "dns_etchosts": "{% for item in (groups['k8s_cluster'] + groups['etcd']|default([]) + groups['calico_rr']|default([]))|unique -%}\n{% if 'access_ip' in hostvars[item] or 'ip' in hostvars[item] or 'ansible_default_ipv4' in hostvars[item] -%}\n{{ hostvars[item]['access_ip'] | default(hostvars[item]['ip'] | default(hostvars[item]['ansible_default_ipv4']['address'])) }}\n{%- if ('ansible_hostname' in hostvars[item] and item != hostvars[item]['ansible_hostname']) %} {{ hostvars[item]['ansible_hostname'] }}.{{ dns_domain }} {{ hostvars[item]['ansible_hostname'] }} {% else %} {{ item }}.{{ dns_domain }} {{ item }} {% endif %}\n\n{% endif %}\n{% endfor %}", "dns_mode": "manual", "download_container": false, "download_localhost": true, "download_run_once": true, "drain_grace_period": 300, "drain_retries": 3, "drain_timeout": "600s", "enable_nodelocaldns": false, "etcd_deployment_type": "kubeadm", "etcd_metrics_port": 2381, "etcd_metrics_service_labels": { "app": "kube-prometheus-stack-kube-etcd", "app.kubernetes.io/managed-by": "Kubespray", "k8s-app": "etcd", "release": "prometheus-stack" }, "group_names": [ "etcd", "k8s_cluster", "kube_control_plane", "kube_node" ], "groups": { "all": [ "localhost", "lenop", "leno1", "leno2", "leno3", "leno4" ], "calico_rr": [], "etcd": [ "lenop", "leno3", "leno4" ], "k8s_cluster": [ "lenop", "leno3", "leno4", "leno1", "leno2" ], "kube_control_plane": [ "lenop", "leno3", "leno4" ], "kube_node": [ "leno1", "leno2", "leno3", "leno4", "lenop" ], "kube_storage_nodes": [ "leno1", "leno2", "leno3", "leno4" ], "metadata": [], "ungrouped": [ "localhost" ] }, "if_name": "eth0", "inventory_dir": "/home/nicolas/git/kub/k8s-leno/inventory/leno", "inventory_file": "/home/nicolas/git/kub/k8s-leno/inventory/leno/hosts.yml", "inventory_hostname": "lenop", "inventory_hostname_short": "lenop", "ip": "192.168.1.254", "is_fedora_coreos": true, "kube_apiserver_admission_control_config_file": true, "kube_apiserver_admission_event_rate_limits": { "limit_1": { "burst": 1000, "cache_size": 4000, "qps": 100, "type": "Namespace" }, "limit_2": { "burst": 1000, "qps": 100, "type": "User" } }, "kube_apiserver_enable_admission_plugins": [ "EventRateLimit", "AlwaysPullImages", "ServiceAccount", "NamespaceLifecycle", "NodeRestriction", "LimitRanger", "ResourceQuota", "MutatingAdmissionWebhook", "ValidatingAdmissionWebhook", "PodNodeSelector", "PodSecurity" ], "kube_apiserver_request_timeout": "120s", "kube_apiserver_service_account_lookup": true, "kube_cert_group": "root", "kube_controller_feature_gates": [ "RotateKubeletServerCertificate=true" ], "kube_controller_manager_bind_address": "127.0.0.1", "kube_controller_terminated_pod_gc_threshold": 50, "kube_encrypt_secret_data": true, "kube_encryption_algorithm": "secretbox", "kube_encryption_resources": [ "secrets" ], "kube_lb_addresses": "192.168.1.160/27", "kube_master_cpu_reserved": "200m", "kube_master_memory_reserved": "128Mi", "kube_network_node_prefix": 24, "kube_network_plugin": "cni", "kube_network_plugin_multus": false, "kube_owner": "root", "kube_pod_security_default_enforce": "restricted", "kube_pod_security_use_default": true, "kube_pods_subnet": "10.20.64.0/18", "kube_profiling": false, "kube_proxy_remove": true, "kube_proxy_strict_arp": true, "kube_read_only_port": 0, "kube_scheduler_bind_address": "127.0.0.1", "kube_service_addresses": "10.20.0.0/18", "kube_vip_address": "192.168.1.250", "kube_vip_arp_enabled": true, "kube_vip_controlplane_enabled": true, "kube_vip_enabled": true, "kube_vip_services": "192.168.1.160/27", "kube_vip_services_enabled": false, "kubeadm_feature_gates": [ "PublicKeysECDSA=true", "EtcdLearnerMode=true" ], "kubeadm_scale_down_coredns_enabled": false, "kubeconfig_localhost": true, "kubectl_localhost": true, "kubelet_authentication_token_webhook": true, "kubelet_authorization_mode_webhook": true, "kubelet_csr_approver_enabled": false, "kubelet_event_record_qps": 1, "kubelet_feature_gates": [ "RotateKubeletServerCertificate=true", "SeccompDefault=true" ], "kubelet_image_gc_high_threshold": 90, "kubelet_image_gc_low_threshold": 60, "kubelet_make_iptables_util_chains": false, "kubelet_max_pods": 250, "kubelet_protect_kernel_defaults": true, "kubelet_rotate_certificates": true, "kubelet_rotate_server_certificates": true, "kubelet_streaming_connection_idle_timeout": "5m", "kubernetes_audit": false, "loadbalancer_apiserver": { "address": "192.168.1.250", "port": 6443 }, "loadbalancer_apiserver_localhost": false, "manual_dns_server": "10.20.0.10", "ntp_manage_config": true, "ntp_servers": [ "0.fr.pool.ntp.org iburst", "1.fr.pool.ntp.org iburst", "2.fr.pool.ntp.org iburst", "3.fr.pool.ntp.org iburst" ], "omit": "__omit_place_holder__6d1590d3cd4c0ff5954bce6dd24b6053512da6ee", "ping_access_ip": false, "playbook_dir": "/home/nicolas/git/kub/k8s-leno", "resolvconf_mode": "none", "resolved_conf_customization": [ "LLMNR=no", "MulticastDNS=no", "ReadEtcHosts=yes" ], "service_pools": { "default": [ "192.168.1.160/27" ] }, "sysctl_file_path": "/etc/sysctl.d/99-kubernetes.conf", "system_cpu_reserved": "200m", "system_memory_reserved": "128Mi", "tls_cipher_suites": [ "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305" ], "tls_min_version": "VersionTLS12", "upstream_dns_servers": [ "192.168.1.1" ] } }leno2 | SUCCESS =>
{ "hostvars[inventory_hostname]": { "access_ip": "192.168.1.252", "additional_sysctl": [ { "name": "vm.max_map_count", "value": 262144 } ], "ansible_check_mode": false, "ansible_config_file": "/home/nicolas/git/kub/k8s-leno/ansible.cfg", "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 10, "ansible_host": "192.168.1.252", "ansible_inventory_sources": [ "/home/nicolas/git/kub/k8s-leno/inventory/leno" ], "ansible_playbook_python": "/home/nicolas/ansible/bin/python3", "ansible_user": "core", "ansible_verbosity": 0, "ansible_version": { "full": "2.16.2", "major": 2, "minor": 16, "revision": 2, "string": "2.16.2" }, "audit_log_maxage": 7, "audit_log_maxbackups": 10, "audit_log_maxsize": 100, "audit_log_path": "/var/log/audit/kube-apiserver-audit.json", "authorization_modes": [ "Node", "RBAC" ], "auto_renew_certificates": true, "coreos_zincati_disable": true, "dashboard_enabled": false, "dns_mode": "manual", "download_container": false, "download_localhost": true, "download_run_once": true, "drain_grace_period": 300, "drain_retries": 3, "drain_timeout": "600s", "enable_nodelocaldns": false, "etcd_deployment_type": "kubeadm", "etcd_metrics_port": 2381, "etcd_metrics_service_labels": { "app": "kube-prometheus-stack-kube-etcd", "app.kubernetes.io/managed-by": "Kubespray", "k8s-app": "etcd", "release": "prometheus-stack" }, "group_names": [ "k8s_cluster", "kube_node", "kube_storage_nodes" ], "groups": { "all": [ "localhost", "lenop", "leno1", "leno2", "leno3", "leno4" ], "calico_rr": [], "etcd": [ "lenop", "leno3", "leno4" ], "k8s_cluster": [ "lenop", "leno3", "leno4", "leno1", "leno2" ], "kube_control_plane": [ "lenop", "leno3", "leno4" ], "kube_node": [ "leno1", "leno2", "leno3", "leno4", "lenop" ], "kube_storage_nodes": [ "leno1", "leno2", "leno3", "leno4" ], "metadata": [], "ungrouped": [ "localhost" ] }, "if_name": "eth0", "inventory_dir": "/home/nicolas/git/kub/k8s-leno/inventory/leno", "inventory_file": "/home/nicolas/git/kub/k8s-leno/inventory/leno/hosts.yml", "inventory_hostname": "leno2", "inventory_hostname_short": "leno2", "ip": "192.168.1.252", "is_fedora_coreos": true, "is_storage_node": true, "kube_apiserver_admission_control_config_file": true, "kube_apiserver_admission_event_rate_limits": { "limit_1": { "burst": 1000, "cache_size": 4000, "qps": 100, "type": "Namespace" }, "limit_2": { "burst": 1000, "qps": 100, "type": "User" } }, "kube_apiserver_enable_admission_plugins": [ "EventRateLimit", "AlwaysPullImages", "ServiceAccount", "NamespaceLifecycle", "NodeRestriction", "LimitRanger", "ResourceQuota", "MutatingAdmissionWebhook", "ValidatingAdmissionWebhook", "PodNodeSelector", "PodSecurity" ], "kube_apiserver_request_timeout": "120s", "kube_apiserver_service_account_lookup": true, "kube_cert_group": "root", "kube_controller_feature_gates": [ "RotateKubeletServerCertificate=true" ], "kube_controller_manager_bind_address": "127.0.0.1", "kube_controller_terminated_pod_gc_threshold": 50, "kube_encrypt_secret_data": true, "kube_encryption_algorithm": "secretbox", "kube_encryption_resources": [ "secrets" ], "kube_lb_addresses": "192.168.1.160/27", "kube_master_cpu_reserved": "200m", "kube_master_memory_reserved": "128Mi", "kube_network_node_prefix": 24, "kube_network_plugin": "cni", "kube_network_plugin_multus": false, "kube_owner": "root", "kube_pod_security_default_enforce": "restricted", "kube_pod_security_use_default": true, "kube_pods_subnet": "10.20.64.0/18", "kube_profiling": false, "kube_proxy_remove": true, "kube_proxy_strict_arp": true, "kube_read_only_port": 0, "kube_scheduler_bind_address": "127.0.0.1", "kube_service_addresses": "10.20.0.0/18", "kube_vip_address": "192.168.1.250", "kube_vip_arp_enabled": true, "kube_vip_controlplane_enabled": true, "kube_vip_enabled": true, "kube_vip_services": "192.168.1.160/27", "kube_vip_services_enabled": false, "kubeadm_feature_gates": [ "PublicKeysECDSA=true", "EtcdLearnerMode=true" ], "kubeadm_scale_down_coredns_enabled": false, "kubeconfig_localhost": true, "kubectl_localhost": true, "kubelet_authentication_token_webhook": true, "kubelet_authorization_mode_webhook": true, "kubelet_csr_approver_enabled": false, "kubelet_event_record_qps": 1, "kubelet_feature_gates": [ "RotateKubeletServerCertificate=true", "SeccompDefault=true" ], "kubelet_image_gc_high_threshold": 90, "kubelet_image_gc_low_threshold": 60, "kubelet_make_iptables_util_chains": false, "kubelet_max_pods": 250, "kubelet_protect_kernel_defaults": true, "kubelet_rotate_certificates": true, "kubelet_rotate_server_certificates": true, "kubelet_streaming_connection_idle_timeout": "5m", "kubernetes_audit": false, "loadbalancer_apiserver": { "address": "192.168.1.250", "port": 6443 }, "loadbalancer_apiserver_localhost": false, "manual_dns_server": "10.20.0.10", "node_labels": { "node-role.kubernetes.io/storage": "true", "node.longhorn.io/create-default-disk": "true" }, "ntp_manage_config": true, "ntp_servers": [ "0.fr.pool.ntp.org iburst", "1.fr.pool.ntp.org iburst", "2.fr.pool.ntp.org iburst", "3.fr.pool.ntp.org iburst" ], "omit": "__omit_place_holder__6d1590d3cd4c0ff5954bce6dd24b6053512da6ee", "ping_access_ip": false, "playbook_dir": "/home/nicolas/git/kub/k8s-leno", "resolvconf_mode": "none", "resolved_conf_customization": [ "LLMNR=no", "MulticastDNS=no", "ReadEtcHosts=yes" ], "service_pools": { "default": [ "192.168.1.160/27" ] }, "sysctl_file_path": "/etc/sysctl.d/99-kubernetes.conf", "system_cpu_reserved": "200m", "system_memory_reserved": "128Mi", "tls_cipher_suites": [ "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305" ], "tls_min_version": "VersionTLS12", "upstream_dns_servers": [ "192.168.1.1" ] } }leno4 | SUCCESS =>
{ "hostvars[inventory_hostname]": { "access_ip": "192.168.1.247", "additional_sysctl": [ { "name": "vm.max_map_count", "value": 262144 } ], "ansible_check_mode": false, "ansible_config_file": "/home/nicolas/git/kub/k8s-leno/ansible.cfg", "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 10, "ansible_host": "192.168.1.247", "ansible_inventory_sources": [ "/home/nicolas/git/kub/k8s-leno/inventory/leno" ], "ansible_playbook_python": "/home/nicolas/ansible/bin/python3", "ansible_user": "core", "ansible_verbosity": 0, "ansible_version": { "full": "2.16.2", "major": 2, "minor": 16, "revision": 2, "string": "2.16.2" }, "audit_log_maxage": 7, "audit_log_maxbackups": 10, "audit_log_maxsize": 100, "audit_log_path": "/var/log/audit/kube-apiserver-audit.json", "authorization_modes": [ "Node", "RBAC" ], "auto_renew_certificates": true, "coreos_zincati_disable": true, "dashboard_enabled": false, "dns_etchosts": "{% for item in (groups['k8s_cluster'] + groups['etcd']|default([]) + groups['calico_rr']|default([]))|unique -%}\n{% if 'access_ip' in hostvars[item] or 'ip' in hostvars[item] or 'ansible_default_ipv4' in hostvars[item] -%}\n{{ hostvars[item]['access_ip'] | default(hostvars[item]['ip'] | default(hostvars[item]['ansible_default_ipv4']['address'])) }}\n{%- if ('ansible_hostname' in hostvars[item] and item != hostvars[item]['ansible_hostname']) %} {{ hostvars[item]['ansible_hostname'] }}.{{ dns_domain }} {{ hostvars[item]['ansible_hostname'] }} {% else %} {{ item }}.{{ dns_domain }} {{ item }} {% endif %}\n\n{% endif %}\n{% endfor %}", "dns_mode": "manual", "download_container": false, "download_localhost": true, "download_run_once": true, "drain_grace_period": 300, "drain_retries": 3, "drain_timeout": "600s", "enable_nodelocaldns": false, "etcd_deployment_type": "kubeadm", "etcd_metrics_port": 2381, "etcd_metrics_service_labels": { "app": "kube-prometheus-stack-kube-etcd", "app.kubernetes.io/managed-by": "Kubespray", "k8s-app": "etcd", "release": "prometheus-stack" }, "group_names": [ "etcd", "k8s_cluster", "kube_control_plane", "kube_node", "kube_storage_nodes" ], "groups": { "all": [ "localhost", "lenop", "leno1", "leno2", "leno3", "leno4" ], "calico_rr": [], "etcd": [ "lenop", "leno3", "leno4" ], "k8s_cluster": [ "lenop", "leno3", "leno4", "leno1", "leno2" ], "kube_control_plane": [ "lenop", "leno3", "leno4" ], "kube_node": [ "leno1", "leno2", "leno3", "leno4", "lenop" ], "kube_storage_nodes": [ "leno1", "leno2", "leno3", "leno4" ], "metadata": [], "ungrouped": [ "localhost" ] }, "if_name": "eth0", "inventory_dir": "/home/nicolas/git/kub/k8s-leno/inventory/leno", "inventory_file": "/home/nicolas/git/kub/k8s-leno/inventory/leno/hosts.yml", "inventory_hostname": "leno4", "inventory_hostname_short": "leno4", "ip": "192.168.1.247", "is_fedora_coreos": true, "is_storage_node": true, "kube_apiserver_admission_control_config_file": true, "kube_apiserver_admission_event_rate_limits": { "limit_1": { "burst": 1000, "cache_size": 4000, "qps": 100, "type": "Namespace" }, "limit_2": { "burst": 1000, "qps": 100, "type": "User" } }, "kube_apiserver_enable_admission_plugins": [ "EventRateLimit", "AlwaysPullImages", "ServiceAccount", "NamespaceLifecycle", "NodeRestriction", "LimitRanger", "ResourceQuota", "MutatingAdmissionWebhook", "ValidatingAdmissionWebhook", "PodNodeSelector", "PodSecurity" ], "kube_apiserver_request_timeout": "120s", "kube_apiserver_service_account_lookup": true, "kube_cert_group": "root", "kube_controller_feature_gates": [ "RotateKubeletServerCertificate=true" ], "kube_controller_manager_bind_address": "127.0.0.1", "kube_controller_terminated_pod_gc_threshold": 50, "kube_encrypt_secret_data": true, "kube_encryption_algorithm": "secretbox", "kube_encryption_resources": [ "secrets" ], "kube_lb_addresses": "192.168.1.160/27", "kube_master_cpu_reserved": "200m", "kube_master_memory_reserved": "128Mi", "kube_network_node_prefix": 24, "kube_network_plugin": "cni", "kube_network_plugin_multus": false, "kube_owner": "root", "kube_pod_security_default_enforce": "restricted", "kube_pod_security_use_default": true, "kube_pods_subnet": "10.20.64.0/18", "kube_profiling": false, "kube_proxy_remove": true, "kube_proxy_strict_arp": true, "kube_read_only_port": 0, "kube_scheduler_bind_address": "127.0.0.1", "kube_service_addresses": "10.20.0.0/18", "kube_vip_address": "192.168.1.250", "kube_vip_arp_enabled": true, "kube_vip_controlplane_enabled": true, "kube_vip_enabled": true, "kube_vip_services": "192.168.1.160/27", "kube_vip_services_enabled": false, "kubeadm_feature_gates": [ "PublicKeysECDSA=true", "EtcdLearnerMode=true" ], "kubeadm_scale_down_coredns_enabled": false, "kubeconfig_localhost": true, "kubectl_localhost": true, "kubelet_authentication_token_webhook": true, "kubelet_authorization_mode_webhook": true, "kubelet_csr_approver_enabled": false, "kubelet_event_record_qps": 1, "kubelet_feature_gates": [ "RotateKubeletServerCertificate=true", "SeccompDefault=true" ], "kubelet_image_gc_high_threshold": 90, "kubelet_image_gc_low_threshold": 60, "kubelet_make_iptables_util_chains": false, "kubelet_max_pods": 250, "kubelet_protect_kernel_defaults": true, "kubelet_rotate_certificates": true, "kubelet_rotate_server_certificates": true, "kubelet_streaming_connection_idle_timeout": "5m", "kubernetes_audit": false, "loadbalancer_apiserver": { "address": "192.168.1.250", "port": 6443 }, "loadbalancer_apiserver_localhost": false, "manual_dns_server": "10.20.0.10", "node_labels": { "node-role.kubernetes.io/storage": "true", "node.longhorn.io/create-default-disk": "true" }, "ntp_manage_config": true, "ntp_servers": [ "0.fr.pool.ntp.org iburst", "1.fr.pool.ntp.org iburst", "2.fr.pool.ntp.org iburst", "3.fr.pool.ntp.org iburst" ], "omit": "__omit_place_holder__6d1590d3cd4c0ff5954bce6dd24b6053512da6ee", "ping_access_ip": false, "playbook_dir": "/home/nicolas/git/kub/k8s-leno", "resolvconf_mode": "none", "resolved_conf_customization": [ "LLMNR=no", "MulticastDNS=no", "ReadEtcHosts=yes" ], "service_pools": { "default": [ "192.168.1.160/27" ] }, "sysctl_file_path": "/etc/sysctl.d/99-kubernetes.conf", "system_cpu_reserved": "200m", "system_memory_reserved": "128Mi", "tls_cipher_suites": [ "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305" ], "tls_min_version": "VersionTLS12", "upstream_dns_servers": [ "192.168.1.1" ] } }leno3 | SUCCESS =>
{ "hostvars[inventory_hostname]": { "access_ip": "192.168.1.253", "additional_sysctl": [ { "name": "vm.max_map_count", "value": 262144 } ], "ansible_check_mode": false, "ansible_config_file": "/home/nicolas/git/kub/k8s-leno/ansible.cfg", "ansible_diff_mode": false, "ansible_facts": {}, "ansible_forks": 10, "ansible_host": "192.168.1.253", "ansible_inventory_sources": [ "/home/nicolas/git/kub/k8s-leno/inventory/leno" ], "ansible_playbook_python": "/home/nicolas/ansible/bin/python3", "ansible_user": "core", "ansible_verbosity": 0, "ansible_version": { "full": "2.16.2", "major": 2, "minor": 16, "revision": 2, "string": "2.16.2" }, "audit_log_maxage": 7, "audit_log_maxbackups": 10, "audit_log_maxsize": 100, "audit_log_path": "/var/log/audit/kube-apiserver-audit.json", "authorization_modes": [ "Node", "RBAC" ], "auto_renew_certificates": true, "coreos_zincati_disable": true, "dashboard_enabled": false, "dns_etchosts": "{% for item in (groups['k8s_cluster'] + groups['etcd']|default([]) + groups['calico_rr']|default([]))|unique -%}\n{% if 'access_ip' in hostvars[item] or 'ip' in hostvars[item] or 'ansible_default_ipv4' in hostvars[item] -%}\n{{ hostvars[item]['access_ip'] | default(hostvars[item]['ip'] | default(hostvars[item]['ansible_default_ipv4']['address'])) }}\n{%- if ('ansible_hostname' in hostvars[item] and item != hostvars[item]['ansible_hostname']) %} {{ hostvars[item]['ansible_hostname'] }}.{{ dns_domain }} {{ hostvars[item]['ansible_hostname'] }} {% else %} {{ item }}.{{ dns_domain }} {{ item }} {% endif %}\n\n{% endif %}\n{% endfor %}", "dns_mode": "manual", "download_container": false, "download_localhost": true, "download_run_once": true, "drain_grace_period": 300, "drain_retries": 3, "drain_timeout": "600s", "enable_nodelocaldns": false, "etcd_deployment_type": "kubeadm", "etcd_metrics_port": 2381, "etcd_metrics_service_labels": { "app": "kube-prometheus-stack-kube-etcd", "app.kubernetes.io/managed-by": "Kubespray", "k8s-app": "etcd", "release": "prometheus-stack" }, "group_names": [ "etcd", "k8s_cluster", "kube_control_plane", "kube_node", "kube_storage_nodes" ], "groups": { "all": [ "localhost", "lenop", "leno1", "leno2", "leno3", "leno4" ], "calico_rr": [], "etcd": [ "lenop", "leno3", "leno4" ], "k8s_cluster": [ "lenop", "leno3", "leno4", "leno1", "leno2" ], "kube_control_plane": [ "lenop", "leno3", "leno4" ], "kube_node": [ "leno1", "leno2", "leno3", "leno4", "lenop" ], "kube_storage_nodes": [ "leno1", "leno2", "leno3", "leno4" ], "metadata": [], "ungrouped": [ "localhost" ] }, "if_name": "eth0", "inventory_dir": "/home/nicolas/git/kub/k8s-leno/inventory/leno", "inventory_file": "/home/nicolas/git/kub/k8s-leno/inventory/leno/hosts.yml", "inventory_hostname": "leno3", "inventory_hostname_short": "leno3", "ip": "192.168.1.253", "is_fedora_coreos": true, "is_storage_node": true, "kube_apiserver_admission_control_config_file": true, "kube_apiserver_admission_event_rate_limits": { "limit_1": { "burst": 1000, "cache_size": 4000, "qps": 100, "type": "Namespace" }, "limit_2": { "burst": 1000, "qps": 100, "type": "User" } }, "kube_apiserver_enable_admission_plugins": [ "EventRateLimit", "AlwaysPullImages", "ServiceAccount", "NamespaceLifecycle", "NodeRestriction", "LimitRanger", "ResourceQuota", "MutatingAdmissionWebhook", "ValidatingAdmissionWebhook", "PodNodeSelector", "PodSecurity" ], "kube_apiserver_request_timeout": "120s", "kube_apiserver_service_account_lookup": true, "kube_cert_group": "root", "kube_controller_feature_gates": [ "RotateKubeletServerCertificate=true" ], "kube_controller_manager_bind_address": "127.0.0.1", "kube_controller_terminated_pod_gc_threshold": 50, "kube_encrypt_secret_data": true, "kube_encryption_algorithm": "secretbox", "kube_encryption_resources": [ "secrets" ], "kube_lb_addresses": "192.168.1.160/27", "kube_master_cpu_reserved": "200m", "kube_master_memory_reserved": "128Mi", "kube_network_node_prefix": 24, "kube_network_plugin": "cni", "kube_network_plugin_multus": false, "kube_owner": "root", "kube_pod_security_default_enforce": "restricted", "kube_pod_security_use_default": true, "kube_pods_subnet": "10.20.64.0/18", "kube_profiling": false, "kube_proxy_remove": true, "kube_proxy_strict_arp": true, "kube_read_only_port": 0, "kube_scheduler_bind_address": "127.0.0.1", "kube_service_addresses": "10.20.0.0/18", "kube_vip_address": "192.168.1.250", "kube_vip_arp_enabled": true, "kube_vip_controlplane_enabled": true, "kube_vip_enabled": true, "kube_vip_services": "192.168.1.160/27", "kube_vip_services_enabled": false, "kubeadm_feature_gates": [ "PublicKeysECDSA=true", "EtcdLearnerMode=true" ], "kubeadm_scale_down_coredns_enabled": false, "kubeconfig_localhost": true, "kubectl_localhost": true, "kubelet_authentication_token_webhook": true, "kubelet_authorization_mode_webhook": true, "kubelet_csr_approver_enabled": false, "kubelet_event_record_qps": 1, "kubelet_feature_gates": [ "RotateKubeletServerCertificate=true", "SeccompDefault=true" ], "kubelet_image_gc_high_threshold": 90, "kubelet_image_gc_low_threshold": 60, "kubelet_make_iptables_util_chains": false, "kubelet_max_pods": 250, "kubelet_protect_kernel_defaults": true, "kubelet_rotate_certificates": true, "kubelet_rotate_server_certificates": true, "kubelet_streaming_connection_idle_timeout": "5m", "kubernetes_audit": false, "loadbalancer_apiserver": { "address": "192.168.1.250", "port": 6443 }, "loadbalancer_apiserver_localhost": false, "manual_dns_server": "10.20.0.10", "node_labels": { "node-role.kubernetes.io/storage": "true", "node.longhorn.io/create-default-disk": "true" }, "ntp_manage_config": true, "ntp_servers": [ "0.fr.pool.ntp.org iburst", "1.fr.pool.ntp.org iburst", "2.fr.pool.ntp.org iburst", "3.fr.pool.ntp.org iburst" ], "omit": "__omit_place_holder__6d1590d3cd4c0ff5954bce6dd24b6053512da6ee", "ping_access_ip": false, "playbook_dir": "/home/nicolas/git/kub/k8s-leno", "resolvconf_mode": "none", "resolved_conf_customization": [ "LLMNR=no", "MulticastDNS=no", "ReadEtcHosts=yes" ], "service_pools": { "default": [ "192.168.1.160/27" ] }, "sysctl_file_path": "/etc/sysctl.d/99-kubernetes.conf", "system_cpu_reserved": "200m", "system_memory_reserved": "128Mi", "tls_cipher_suites": [ "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305" ], "tls_min_version": "VersionTLS12", "upstream_dns_servers": [ "192.168.1.1" ] } }Command used to invoke ansible
ansible-playbook playbooks/cluster.yml --limit=kube_control_plane --become
Output of ansible run
Only put then end, full log is too long.
Anything else we need to know
When adding a new control-plane, the task
kubernetes/control-plane : Create kubeadm token for joining nodes with 24h expiration (default)is delegated tofirst_kube_control_plane(leno4in the run).The task
Parse certificate key if not setin kubeadm-secondary sets facts fromgroups['kube_control_plane'][0]which is not the same node (lenopin the run).For this reason
kubeadm_certificate_keyis not valid and taskkubernetes/control-plane : Create kubeadm ControlPlane configfails.This patch resolves the bug in my situation :
The text was updated successfully, but these errors were encountered: