Respan Dataset Explorer

vagrant@Master:~$ kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter
NAME                                        READY   STATUS    RESTARTS   AGE
prometheus-prometheus-node-exporter-6l6lr   0/1     Pending   0          12h
prometheus-prometheus-node-exporter-dj4hj   0/1     Pending   0          12h
prometheus-prometheus-node-exporter-ns4fv   0/1     Pending   0          12h
vagrant@Master:~$ 

The node-exporter pods status are `Pending`, which means they have been created but aren’t scheduled onto any node yet. This usually happens because:

- No suitable node matches the pod's scheduling requirements
- Insufficient resources on nodes
- Node taints prevent the pods from being scheduled

---

### Steps to troubleshoot and fix:

---

### 1. Check why pods are pending

Use `kubectl describe` on one of the pending pods to see scheduling events and reasons:

```bash
kubectl describe pod prometheus-prometheus-node-exporter-6l6lr -n prometheus
```

Look for the **Events** section at the bottom. Common messages include:

- `0/3 nodes are available: 3 Insufficient cpu.`
- `0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.`
- Or other scheduling errors.

---

### 2. Check node status and resources

Check your nodes status and resource availability:

```bash
kubectl get nodes
kubectl describe node <node-name>
```

Look at resource allocation:

```bash
kubectl top nodes
```

If `kubectl top` is not installed, you may need to enable metrics-server, but you have metrics-server running already.

---

### 3. Check taints on nodes

List taints for your nodes:

```bash
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, taints: .spec.taints}'
```

Or simpler if no `jq`:

```bash
kubectl describe node <node-name> | grep -i taint
```

If taints exist (like `node-role.kubernetes.io/master:NoSchedule`), your node-exporter pods need tolerations to be scheduled there.

---

### 4. Check node-exporter DaemonSet tolerations

If your nodes have taints (common on control-plane nodes), and your node-exporter pods don't tolerate them, pods will stay pending.

Inspect DaemonSet:

```bash
kubectl describe ds prometheus-prometheus-node-exporter -n prometheus
```

Look for **Tolerations** section and confirm it matches taints on your nodes.

---

### 5. Fix scenario examples

- **If nodes have taints and your DaemonSet lacks tolerations:**

  Edit the DaemonSet to add appropriate tolerations:

  ```yaml
  tolerations:
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"
  ```

  You can patch DaemonSet:

  ```bash
  kubectl patch ds prometheus-prometheus-node-exporter -n prometheus --patch '{
    "spec": {
      "template": {
        "spec": {
          "tolerations": [
            {
              "key": "node-role.kubernetes.io/master",
              "operator": "Exists",
              "effect": "NoSchedule"
            }
          ]
        }
      }
    }
  }'
  ```

- **If nodes do not have enough resources:**

  Consider adding more resources or reducing resource requests in node-exporter DaemonSet pod spec.

- **If no nodes available at all**

  Check node status and fix accordingly.

---

### Summary

Run

```bash
kubectl describe pod prometheus-prometheus-node-exporter-6l6lr -n prometheus
```

Check for scheduling error messages, then adjust DaemonSet tolerations or node state accordingly.

---

Let me know the **Events** section output of `kubectl describe pod ...` if you want more detailed advice!

vagrant@Master:~$ kubectl describe pod prometheus-prometheus-node-exporter-6l6lr -n prometheus
Name:             prometheus-prometheus-node-exporter-6l6lr
Namespace:        prometheus
Priority:         0
Service Account:  prometheus-prometheus-node-exporter
Node:             <none>
Labels:           app.kubernetes.io/component=metrics
                  app.kubernetes.io/instance=prometheus
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=prometheus-node-exporter
                  app.kubernetes.io/part-of=prometheus-node-exporter
                  app.kubernetes.io/version=1.9.1
                  controller-revision-hash=5cb7cc78f7
                  helm.sh/chart=prometheus-node-exporter-4.45.2
                  pod-template-generation=1
Annotations:      cluster-autoscaler.kubernetes.io/safe-to-evict: true
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    DaemonSet/prometheus-prometheus-node-exporter
Containers:
  node-exporter:
    Image:      quay.io/prometheus/node-exporter:v1.9.1
    Port:       9100/TCP
    Host Port:  9100/TCP
    Args:
      --path.procfs=/host/proc
      --path.sysfs=/host/sys
      --path.rootfs=/host/root
      --path.udev.data=/host/root/run/udev/data
      --web.listen-address=[$(HOST_IP)]:9100
    Liveness:   http-get http://:9100/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:9100/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      HOST_IP:  0.0.0.0
    Mounts:
      /host/proc from proc (ro)
      /host/root from root (ro)
      /host/sys from sys (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  proc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:  
  sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:  
  root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  
QoS Class:         BestEffort
Node-Selectors:    kubernetes.io/os=linux
Tolerations:       :NoSchedule op=Exists
                   node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                   node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                   node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                   node.kubernetes.io/not-ready:NoExecute op=Exists
                   node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                   node.kubernetes.io/unreachable:NoExecute op=Exists
                   node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  11m (x12 over 66m)  default-scheduler  0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't satisfy plugin(s) [NodeAffinity]. preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling.
vagrant@Master:~$ 

The scheduling failure event you see:

```
0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't satisfy plugin(s) [NodeAffinity].
```

means:

- **1 node already has port 9100 in use** (the port `node-exporter` tries to bind with hostPort=9100).
- **2 nodes are blocked due to NodeAffinity constraints**, so pods cannot be scheduled there.

---

## What is happening?

- Your Prometheus Node Exporter pods are `DaemonSet` pods that try to run on every node.
- Each pod uses `hostPort: 9100`, so only **one pod per node can bind port 9100**.
- One node already has something (likely another node-exporter pod or another process) using port 9100.
- The other two nodes don’t satisfy the DaemonSet's **node affinity rules**, so the Pod can't be scheduled there.

---

## Steps to fix

### 1. Check if another process or pod is already using port 9100 on the node with "port in use"

On the problematic node (you can check nodes with `kubectl get nodes`), ssh into it and run:

```bash
sudo lsof -i :9100
```

or

```bash
sudo netstat -tulpn | grep 9100
```

If something else is bound to 9100, you must either:

- Stop that process; or
- Change the node-exporter DaemonSet to use a different, free `hostPort`.

---

### 2. Inspect NodeAffinity rules on your DaemonSet

From the DaemonSet description, the nodeAffinity is not shown, but the error clearly complains about NodeAffinity.

Run:

```bash
kubectl get ds prometheus-prometheus-node-exporter -n prometheus -o yaml | grep -A 10 affinity
```

Or view the whole affinity section in the pod template:

```bash
kubectl get ds prometheus-prometheus-node-exporter -n prometheus -o yaml | less
```

Look for an `affinity` block under `.spec.template.spec.affinity`.

Possible common affinity could be something like scheduling only on nodes with a label like:

```yaml
nodeAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
      - key: node-role.kubernetes.io/worker
        operator: Exists
```

Check if your nodes have the labels needed (e.g. `node-role.kubernetes.io/worker`):

```bash
kubectl get nodes --show-labels
```

---

### 3. Fix the NodeAffinity or label nodes accordingly

- **Option A: Label nodes to match affinity**

If node affinity expects labels like `node-role.kubernetes.io/worker` but nodes are missing that label,

add it:

```bash
kubectl label node <node-name> node-role.kubernetes.io/worker=""
```

Do this for the nodes that should run node-exporter.

- **Option B: Remove or modify affinity**

If you want node-exporters on all nodes, modify the DaemonSet affinity to remove the affinity or relax it.

You can edit the DaemonSet:

```bash
kubectl edit ds prometheus-prometheus-node-exporter -n prometheus
```

Look for the `.spec.template.spec.affinity` and comment out or adjust `nodeAffinity`.

---

### 4. If port 9100 conflicts persist but you still want node-exporter on nodes

- Consider disabling the `hostPort` in the DaemonSet pod spec (depends on your monitoring setup).
- Or select a different `hostPort` if your setup allows and you adjust scrape configs.

---

## Summary of commands you may want to try:

Check nodes and labels:

```bash
kubectl get nodes --show-labels
```

Check affinity on DaemonSet:

```bash
kubectl get ds prometheus-prometheus-node-exporter -n prometheus -o yaml | grep -A 15 affinity
```

Label nodes if missing worker label (replace `<node-name>`):

```bash
kubectl label node <node-name> node-role.kubernetes.io/worker=""
```

Check port 9100 usage on node with conflict (ssh into that node):

```bash
sudo lsof -i :9100
```

---

### After changes, the pods should get scheduled and start running.

Verify:

```bash
kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter
kubectl get ep -n prometheus prometheus-prometheus-node-exporter
```

---

If you want, share your DaemonSet affinity yaml and `kubectl get nodes --show-labels` output here, and I can help give more precise fix instructions.

how to fix

To fix your `prometheus-prometheus-node-exporter` DaemonSet pods stuck in **Pending** due to:

- **Port 9100 conflict on one node**
- **NodeAffinity preventing scheduling on two nodes**

Follow these step-by-step instructions:

---

## Step 1: Check your nodes and their labels

Run:

```bash
kubectl get nodes --show-labels
```

You will see your nodes with their labels, for example:

```
NAME       STATUS   ROLES    AGE   VERSION   LABELS
node1      Ready    master   ...   ...       ...
node2      Ready    worker   ...   ...       node-role.kubernetes.io/worker=""
node3      Ready    worker   ...   ...       node-role.kubernetes.io/worker=""
```

---

## Step 2: Check your DaemonSet's affinity rules

Run:

```bash
kubectl get ds prometheus-prometheus-node-exporter -n prometheus -o yaml | grep -A 15 affinity
```

(or edit to inspect full yaml)

You will see something like this (example):

```yaml
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: node-role.kubernetes.io/worker
          operator: Exists
```

---

## Step 3: Fix NodeAffinity by labeling nodes or editing DaemonSet

### Option A - Label nodes to match affinity

If your DaemonSet requires `node-role.kubernetes.io/worker` label but some nodes do not have it (like your master/control-plane), add it with:

```bash
kubectl label node <node-name> node-role.kubernetes.io/worker=""
```

Do it for any node you'd like node-exporter to run on (probably masters missing the label).

---

### Option B - Remove/relax affinity rules in your DaemonSet

If you want node-exporter running on all nodes without affinity restrictions:

```bash
kubectl edit ds prometheus-prometheus-node-exporter -n prometheus
```

Locate (or add) this block in `.spec.template.spec`:

```yaml
affinity: {}
```

Or remove the existing `nodeAffinity` section under affinity in the DaemonSet spec.

Save and exit the editor. This removes the affinity restriction.

---

## Step 4: Fix port conflict on node(s)

### Find which node has port 9100 busy:

Login to nodes (usually master and worker nodes):

```bash
ssh <node-ip-or-hostname>
sudo lsof -i :9100
```

(or)

```bash
sudo netstat -tulpn | grep 9100
```

- If another process or node-exporter pod is binding port 9100, identify it.

- You cannot run two pods on the same node with `hostPort: 9100` because of port conflict.

- Either stop that conflicting process/pod or change the node-exporter DaemonSet to not use that `hostPort` or use another port (less common).

---

## Step 5 (optional): Remove/modify `hostPort` in DaemonSet

If port conflict persists or you don't require `hostPort`, you can modify your DaemonSet:

```bash
kubectl edit ds prometheus-prometheus-node-exporter -n prometheus
```

Find the container spec `ports` section, remove or comment out lines like:

```yaml
hostPort: 9100
```

Save and exit.

**Note:** Removing `hostPort` means that node-exporter inside the pod listens on a container port only and is accessible only via cluster IP. This may require changes in Prometheus scrape configs.

---

## Step 6: Wait and check pods

After applying fixes, check pods:

```bash
kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter
```

They should transition from Pending to Running.

Also check endpoints:

```bash
kubectl get ep prometheus-prometheus-node-exporter -n prometheus
```

You should see pods IPs listed there.

---

# Summary in commands:

Label missing nodes (replace `<node-name>`):

```bash
kubectl label node <node-name> node-role.kubernetes.io/worker=""
```

Or remove nodeAffinity:

```bash
kubectl edit ds prometheus-prometheus-node-exporter -n prometheus
```

And delete or adjust `.spec.template.spec.affinity.nodeAffinity`

Check port usage on your nodes:

```bash
ssh <node-name>
sudo lsof -i :9100
```

Remove conflicting process or change DaemonSet to avoid using `hostPort`.

---

If you share output of:

```bash
kubectl get nodes --show-labels
kubectl get ds prometheus-prometheus-node-exporter -n prometheus -o yaml | grep -A 15 affinity
```

I can help with specific fix commands.

vagrant@Master:~$ kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter
kubectl get ep -n prometheus prometheus-prometheus-node-exporter
NAME                                        READY   STATUS    RESTARTS   AGE
prometheus-prometheus-node-exporter-cvkm7   0/1     Pending   0          3m16s
prometheus-prometheus-node-exporter-l46nw   0/1     Pending   0          3m16s
prometheus-prometheus-node-exporter-p8fzz   0/1     Pending   0          3m16s
NAME                                  ENDPOINTS   AGE
prometheus-prometheus-node-exporter   <none>      12h
vagrant@Master:~$ kubectl get ds prometheus-prometheus-node-exporter -n prometheus -o yaml | grep -A 15 affinity
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: eks.amazonaws.com/compute-type
                operator: NotIn
                values:
                - fargate
              - key: type
                operator: NotIn
                values:
                - virtual-kubelet
      automountServiceAccountToken: false
      containers:
      - args:
vagrant@Master:~$ 

Thanks for the info!

Your DaemonSet's **nodeAffinity** looks like this (relevant excerpt):

```yaml
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: eks.amazonaws.com/compute-type
          operator: NotIn
          values:
          - fargate
        - key: type
          operator: NotIn
          values:
          - virtual-kubelet
```

### What this means:
- Pods **cannot** run on nodes labeled `eks.amazonaws.com/compute-type=fargate`  
- Pods **cannot** run on nodes labeled `type=virtual-kubelet`

Your nodes likely either have one or both labels **OR** are missing required labels for these affinity rules.

---

## **How to fix your pending pods**

### Step 1: Check your node labels

Run this to see your node labels:

```bash
kubectl get nodes --show-labels
```

Look for any nodes with labels:

- `eks.amazonaws.com/compute-type=fargate`  
- `type=virtual-kubelet`

If **your nodes have these labels**, the node affinity will prevent node-exporter pods from scheduling on them (ie: pods are excluded from nodes marked as fargate or virtual-kubelet).

If **your nodes do NOT have these labels** and pods still don't schedule, the affinity is likely not the cause blocking everything — but usually these labels are only present in AWS EKS (managed Kubernetes) environments.

---

### Step 2: Temporarily remove or relax affinity to test

Edit the DaemonSet to **remove the affinity block** so scheduling restrictions are removed:

```bash
kubectl edit ds prometheus-prometheus-node-exporter -n prometheus
```

Delete or comment out the entire `affinity:` section under `.spec.template.spec`.

Save and exit.

---

### Step 3: Check for port conflicts (as from previous problem)

Pods stuck pending can also be due to **`hostPort: 9100` conflicts**.

Check all your nodes:

```bash
kubectl get nodes
```

SSH into each node and check if port 9100 is in use:

```bash
sudo lsof -i :9100
```
or
```bash
sudo netstat -tulpn | grep 9100
```

If port 9100 is busy on a node, either kill the process occupying it, or modify the DaemonSet to not use `hostPort` (advanced, requires changing Prometheus scrape config).

---

### Step 4: After removing affinity, check pods status

Wait a minute, then check:

```bash
kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter
```

Pods should move from Pending to Running.

Also check endpoints:

```bash
kubectl get ep prometheus-prometheus-node-exporter -n prometheus
```

---

## Summary of what you probably want to do right now:

1. **Edit DaemonSet to remove affinity:**

```bash
kubectl edit ds prometheus-prometheus-node-exporter -n prometheus
```
Remove the entire `affinity` block or set it to `{}`.

2. **Check port usage on nodes to avoid conflicts.**

3. **After changes, monitor pod scheduling and readiness:**

```bash
kubectl get pods -n prometheus -w
```

---

If you want, after trying this, share the output of `kubectl get nodes --show-labels` and the new pod status so I can assist further!

    spec:                                                     
      affinity:                                               
        nodeAffinity:                                         
          requiredDuringSchedulingIgnoredDuringExecution:     
            nodeSelectorTerms:                                
            - matchExpressions:                               
              - key: eks.amazonaws.com/compute-type           
                operator: NotIn                               
                values:                                       
                - fargate                                     
              - key: type                                     
                operator: NotIn                               
                values:                                       
                - virtual-kubelet                             
      automountServiceAccountToken: false                     
      containers:                                             
      - args:                                                 
        - --path.procfs=/host/proc                            
        - --path.sysfs=/host/sys                              
        - --path.rootfs=/host/root                            
        - --path.udev.data=/host/root/run/udev/data           
        - --web.listen-address=[$(HOST_IP)]:9100              
        env:       

Your current `affinity` configuration restricts scheduling the pods on nodes labeled with:

- `eks.amazonaws.com/compute-type=fargate`
- `type=virtual-kubelet`

Since you're likely running a local or on-prem cluster (not EKS Fargate), this affinity might unnecessarily restrict pods from scheduling if labels don't match or exist on your nodes.

---

## How to fix

### Option 1 — **Remove or comment out the affinity block**

Edit the DaemonSet to remove the affinity section so pods can schedule on all nodes:

```bash
kubectl edit ds prometheus-prometheus-node-exporter -n prometheus
```

Delete or comment out the entire `affinity:` block (from your snippet it starts at `affinity:` and includes all below it until under containers).

After editing, save and exit.

---

### Option 2 — Replace with empty affinity block (if you want to keep syntax intact)

Replace the block with:

```yaml
affinity: {}
```

---

### After editing:

Give Kubernetes some time and check pods status:

```bash
kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter -w
```

They should move from `Pending` to `Running`.

---

### Check endpoints afterward:

```bash
kubectl get ep prometheus-prometheus-node-exporter -n prometheus
```

Should no longer say `<none>` but have pod IPs listed.

---

### Bonus: verify node labels (optional)

To confirm nodes don’t have unexpected labels:

```bash
kubectl get nodes --show-labels
```

---

### Why this helps

The current affinity:

```yaml
requiredDuringSchedulingIgnoredDuringExecution:
  nodeSelectorTerms:
  - matchExpressions:
    - key: eks.amazonaws.com/compute-type
      operator: NotIn
      values:
      - fargate
    - key: type
      operator: NotIn
      values:
      - virtual-kubelet
```

means:

- Pod **can only run on nodes NOT labeled** with `fargate` or `virtual-kubelet`.
- If your nodes don't have these labels at all, the selector might be effectively excluding them because they don't **explicitly** not match (depends on your kube scheduler version).

Removing this block lets pods schedule anywhere.

---

If after removal you still see pods stuck:

- Check for port 9100 conflicts on nodes (`sudo lsof -i :9100`)
- Check nodeReady status (`kubectl get nodes`)
- Check pod events (`kubectl describe pod <pod-name> -n prometheus`)

---

Feel free to ask if you need help on editing or troubleshooting after this!

vagrant@Master:~$ kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter -w
NAME                                        READY   STATUS    RESTARTS   AGE
prometheus-prometheus-node-exporter-cvkm7   0/1     Pending   0          9m47s
prometheus-prometheus-node-exporter-l46nw   0/1     Pending   0          9m47s
prometheus-prometheus-node-exporter-p8fzz   0/1     Pending   0          9m47s
^[[A^[[A^[[A^[[A



^Cvagrant@Master:~kubectl get ep prometheus-prometheus-node-exporter -n prometheusus
NAME                                  ENDPOINTS   AGE
prometheus-prometheus-node-exporter   <none>      12h
vagrant@Master:~$ kubectl edit ds prometheus-prometheus-node-exporter -n prometheus
error: daemonsets.apps "prometheus-prometheus-node-exporter" is invalid
A copy of your changes has been stored to "/tmp/kubectl-edit-1500215338.yaml"
error: Edit cancelled, no valid changes were saved.

The error message:
```
error: daemonsets.apps "prometheus-prometheus-node-exporter" is invalid
A copy of your changes has been stored to "/tmp/kubectl-edit-1500215338.yaml"
error: Edit cancelled, no valid changes were saved.
```
indicates that the edits you made to the DaemonSet were syntactically invalid (YAML formatting or structure issues) and so Kubernetes rejected them.

---

## How to fix this edit error and correctly remove the affinity block

### Steps:

1. **Get the current DaemonSet YAML to a file**:

```bash
kubectl get ds prometheus-prometheus-node-exporter -n prometheus -o yaml > node-exporter-ds.yaml
```

2. **Open the file with your preferred editor (vim/nano/micro)**:

```bash
nano node-exporter-ds.yaml
```

3. **Find the `affinity:` block under `.spec.template.spec` and completely remove it**

For example, remove this section (including indentation):

```yaml
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: eks.amazonaws.com/compute-type
          operator: NotIn
          values:
          - fargate
        - key: type
          operator: NotIn
          values:
          - virtual-kubelet
```

Make sure you remove exactly the whole `affinity` key and its children.

---

**Important:** Be careful **not** to remove or break any other required sections in the file (like `metadata`, `spec`, `containers`, etc.).

---

4. **Save the file**

5. **Apply the fixed YAML**

```bash
kubectl apply -f node-exporter-ds.yaml
```

6. **Check DaemonSet and Pods**

```bash
kubectl get ds prometheus-prometheus-node-exporter -n prometheus
kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter -w
```

Wait to see if pods start scheduling and become `Running`.

---

## If you prefer patching instead of editing full YAML:

You can delete the affinity with a strategic patch:

```bash
kubectl patch ds prometheus-prometheus-node-exporter -n prometheus --type=json -p='[{"op":"remove","path":"/spec/template/spec/affinity"}]'
```

This command **removes the affinity section** safely.

Then check pods:

```bash
kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter -w
```

---

## Notes:

- Always validate YAML syntax (e.g. online YAML linters) if editing manually.
- Make sure indentation matches properly, Kubernetes YAML is indentation sensitive.

---

Let me know if you want me to review the YAML snippet you tried to apply!

vagrant@Master:~$ kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter -w
NAME                                        READY   STATUS    RESTARTS   AGE
prometheus-prometheus-node-exporter-cvkm7   0/1     Pending   0          9m35s
prometheus-prometheus-node-exporter-l46nw   0/1     Pending   0          9m35s
prometheus-prometheus-node-exporter-p8fzz   0/1     Pending   0          9m35s
^[[A
^[[A^[[A^[[A^Cvagrant@Master:~$ kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter -w
NAME                                        READY   STATUS    RESTARTS   AGE
prometheus-prometheus-node-exporter-cvkm7   0/1     Pending   0          9m47s
prometheus-prometheus-node-exporter-l46nw   0/1     Pending   0          9m47s
prometheus-prometheus-node-exporter-p8fzz   0/1     Pending   0          9m47s
^[[A^[[A^[[A^[[A



^Cvagrant@Master:~kubectl get ep prometheus-prometheus-node-exporter -n prometheusus
NAME                                  ENDPOINTS   AGE
prometheus-prometheus-node-exporter   <none>      12h
vagrant@Master:~$ kubectl edit ds prometheus-prometheus-node-exporter -n prometheus
error: daemonsets.apps "prometheus-prometheus-node-exporter" is invalid
A copy of your changes has been stored to "/tmp/kubectl-edit-1500215338.yaml"
error: Edit cancelled, no valid changes were saved.
vagrant@Master:~$ kubectl edit ds prometheus-prometheus-node-exporter -n prometheus
Edit cancelled, no changes made.
vagrant@Master:~$ kubectl patch ds prometheus-prometheus-node-exporter -n prometheus --type=json -p='[{"op":"remove","path":"/spec/template/spec/affinity"}]'
daemonset.apps/prometheus-prometheus-node-exporter patched
vagrant@Master:~$ kubectl get ep prometheus-prometheus-node-exporter -n prometheus
NAME                                  ENDPOINTS   AGE
prometheus-prometheus-node-exporter   <none>      12h
vagrant@Master:~$ kubectl get ep prometheus-prometheus-node-exporter -n prometheus
NAME                                  ENDPOINTS   AGE
prometheus-prometheus-node-exporter   <none>      12h
vagrant@Master:~$ kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter -w
NAME                                        READY   STATUS    RESTARTS   AGE
prometheus-prometheus-node-exporter-f7hbc   0/1     Pending   0          42s
prometheus-prometheus-node-exporter-gnbfb   0/1     Pending   0          42s
prometheus-prometheus-node-exporter-tq8f8   0/1     Pending   0          42s
^Cvagrant@Master:~$ kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter -w
NAME                                        READY   STATUS    RESTARTS   AGE
prometheus-prometheus-node-exporter-f7hbc   0/1     Pending   0          87s
prometheus-prometheus-node-exporter-gnbfb   0/1     Pending   0          87s
prometheus-prometheus-node-exporter-tq8f8   0/1     Pending   0          87s





^Cvagrant@Master:~$ kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter -w
NAME                                        READY   STATUS    RESTARTS   AGE
prometheus-prometheus-node-exporter-f7hbc   0/1     Pending   0          2m11s
prometheus-prometheus-node-exporter-gnbfb   0/1     Pending   0          2m11s
prometheus-prometheus-node-exporter-tq8f8   0/1     Pending   0          2m11s
^Cvagrant@Master:~$ kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter -w
NAME                                        READY   STATUS    RESTARTS   AGE
prometheus-prometheus-node-exporter-f7hbc   0/1     Pending   0          2m16s
prometheus-prometheus-node-exporter-gnbfb   0/1     Pending   0          2m16s
prometheus-prometheus-node-exporter-tq8f8   0/1     Pending   0          2m16s
^[[A^[[A^[[A^Cvagrant@Mastedit ds prometheus-prometheus-node-exporter -n prometheus
Edit cancelled, no changes made.
vagrant@Master:~$ kubectl get pods -n prometheus -l app.kubernetes.io/name=prometheus-node-exporter -w
NAME                                        READY   STATUS    RESTARTS   AGE
prometheus-prometheus-node-exporter-f7hbc   0/1     Pending   0          2m53s
prometheus-prometheus-node-exporter-gnbfb   0/1     Pending   0          2m53s
prometheus-prometheus-node-exporter-tq8f8   0/1     Pending   0          2m53s
^Cvagrant@Master:~$ 

Your node-exporter pods are still stuck in **Pending**, even after successfully removing the affinity rule, so the problem is elsewhere.

---

### We know so far:
- The affinity blocking scheduling has been removed (`kubectl patch ... remove affinity`).
- Pods remain **Pending** with no endpoints.
- No errors editing the DaemonSet.
- DaemonSet is recreating pods (different pod names).
- The usual cause in this situation is **port conflicts or node conditions**.

---

# Next troubleshooting steps — Find why pods stay Pending

Run `kubectl describe` on one of the **pending** pods to see detailed scheduling failure reasons and events:

```bash
kubectl describe pod prometheus-prometheus-node-exporter-f7hbc -n prometheus
```

Pay very close attention to the **Events** section at bottom. Look for reasons like:

- `FailedScheduling`
- `Insufficient cpu/memory`
- `node(s) had taints that the pod didn't tolerate`
- `node(s) port conflicts`

---

### Common scenario here:  
**"1 node(s) didn't have free ports for the requested pod ports"**

Because `hostPort: 9100` is used by the node-exporter DaemonSet pod spec, only one pod can bind the node's port 9100.

If:

- You have **a previous node-exporter or other app using port 9100 on that node**, or
- The port is reserved or used by other process,

the **scheduler cannot place the new pod**, leaving it Pending.

---

### How to check port conflicts on nodes:

On each node, SSH and run:

```bash
sudo ss -tulpn | grep 9100
```

or

```bash
sudo lsof -i :9100
```

- Identify if any process (including old node-exporter pods or other services) is already binding port 9100.
- If yes, stop/kill the process or pod.

---

### How to remove problematic pods or processes that occupy port 9100:

- List all pods, including possible old node-exporter pods in kube-system or other namespaces:

```bash
kubectl get pods --all-namespaces -o wide | grep node-exporter
```

- If found, delete duplicates:

```bash
kubectl delete pod <pod-name> -n <namespace>
```

---

### Optional: Remove hostPort (if port conflicts are persistent)

You can edit the DaemonSet and **remove hostPort 9100** from container ports:

```bash
kubectl edit ds prometheus-prometheus-node-exporter -n prometheus
```

In the container spec, find and **remove** the line:

```yaml
hostPort: 9100
```

**Warning:** Removing `hostPort` means the node-exporter pods will no longer be exposed on the node's network interface on port 9100. This might break Prometheus scraping unless your setup supports scraping pods directly or via a Service.

---

### Summary of commands to run next:

1. Check pod details & events:

```bash
kubectl describe pod prometheus-prometheus-node-exporter-f7hbc -n prometheus
```

2. On all worker/master nodes, check port usage:

```bash
sudo ss -tulpn | grep 9100
```

3. List all node-exporter pods everywhere:

```bash
kubectl get pods --all-namespaces -o wide | grep node-exporter
```

4. Delete any duplicate or conflicting pods/processes.

---

Please paste here the **Events** section from `kubectl describe pod ...` if you'd like help interpreting it!

Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  3m41s  default-scheduler  0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't satisfy plugin(s) [NodeAffinity]. preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling.
vagrant@Master:~$ 

The key error message in the pod event:

```
0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't satisfy plugin(s) [NodeAffinity].
```

means:

- **1 node is blocked because port 9100 on the host is already in use**, so the pod can't bind its required `hostPort`.
- **2 other nodes are still blocked by NodeAffinity**, meaning the affinity problem **was not fully removed or the nodes still don't satisfy the affinity**.

---

### Action Plan to fix this step-by-step:

---

## 1. Check port usage on all nodes to fix port conflict

On each node, SSH and run:

```bash
sudo ss -tulpn | grep :9100
```

or

```bash
sudo lsof -i :9100
```

If any process is listening on port 9100, identify and stop it.

It's common you might have an old node-exporter instance or some other process holding that port.

---

## 2. Check nodes' labels and affinity issues

Run:

```bash
kubectl get nodes --show-labels
```

Verify if the nodes have labels conflicting with DaemonSet affinity rules.

Since you patched affinity away in the DaemonSet, check if pod spec truly has no affinity:

```bash
kubectl get ds prometheus-prometheus-node-exporter -n prometheus -o yaml | grep -A10 affinity
```

If affinity section is still present, patch it again to remove it:

```bash
kubectl patch ds prometheus-prometheus-node-exporter -n prometheus --type=json -p='[{"op":"remove","path":"/spec/template/spec/affinity"}]'
```

---

## 3. Can your node-exporter tolerate node taints? Check for taints:

```bash
kubectl describe nodes | grep -i taint
```

If nodes have taints and node-exporter does not have matching tolerations, scheduler will refuse to schedule pods there.

If taints exist, you may need to add tolerations to the DaemonSet pod spec.

---

## 4. Optional: Remove `hostPort` to avoid port conflicts (if acceptable)

Removing hostPort avoids port conflict, but means node exporter pod isn't exposed on node IP:9100.

Edit DaemonSet:

```bash
kubectl edit ds prometheus-prometheus-node-exporter -n prometheus
```

Remove this line from container port spec:

```yaml
hostPort: 9100
```

---

## Summary:

- Fix the port conflict on the one node blocking scheduling.
- Ensure affinity is removed completely.
- Verify taints and add tolerations if needed.

---

If you want, paste the output of:

```bash
kubectl get nodes --show-labels
kubectl get ds prometheus-prometheus-node-exporter -n prometheus -o yaml | grep -A15 affinity
kubectl describe nodes | grep -i taint
```

and any port usage info from your nodes (`ss`/`lsof`), I can help you with exact next commands.

antix@antix:~/worker$ vagrant ssh
[fog][WARNING] Unrecognized arguments: libvirt_ip_command
Linux worker 6.12.22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.22-1 (2025-04-10) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Apr 24 19:23:25 2025 from 192.168.121.1
vagrant@worker:~$ sudo ss -tulpn | grep :9100
vagrant@worker:~$ kubectl describe nodes | grep -i taint
E0425 09:23:19.283394    6555 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server could not find the requested resource"
E0425 09:23:19.294294    6555 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server could not find the requested resource"
E0425 09:23:19.300624    6555 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server could not find the requested resource"
E0425 09:23:19.310081    6555 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server could not find the requested resource"
E0425 09:23:19.319136    6555 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server could not find the requested resource"
Error from server (NotFound): the server could not find the requested resource
vagrant@worker:~$ 

You are running `kubectl describe nodes | grep -i taint` on the worker node shell, but:

- The error `Error from server (NotFound): the server could not find the requested resource` indicates **kubectl cannot connect properly to the API server or lacks permissions**.
- Similarly the `memcache.go` errors indicate the client can’t get the API group info — probably your `kubectl` config on the worker node isn't set up or is broken.

---

### What this means:

- You need to run **kubectl commands on the control plane/master node**, where kubeconfig and permissions are correctly set.
- Running `kubectl` from the worker node shell without proper kubeconfig configured will result in these errors.

---

## What to do now

### 1. SSH or log into your Kubernetes **master node**

From your host:

```bash
vagrant ssh master
```

(Or however you access your master VM)

### 2. Run the commands on the master node to check node taints, labels, and affinity again:

```bash
kubectl get nodes --show-labels
kubectl describe nodes | grep -i taint
kubectl get ds prometheus-prometheus-node-exporter -n prometheus -o yaml | grep -A 15 affinity
```

### 3. Also double-check port 9100 usage on master and worker nodes

On each node (master and workers):

```bash
sudo ss -tulpn | grep :9100
```

---

## Summary

- The `kubectl` command on the worker does not have correct context/permissions to query kube API.
- Run `kubectl` on the master node or on machine with proper kubeconfig.
- Investigate taints, node labels, affinity from there.
- Check port 9100 conflicts on all nodes.

---

If you want, share outputs of those commands from the master node and from port checks, and I can help you with the next steps to get node-exporter pods scheduled.

vagrant@worker2:~$ sudo ss -tulpn | grep :9100
vagrant@worker2:~$ sudo ss -tulpn | grep :9100
vagrant@worker2:~$ 

On `worker2` node, port **9100 is not in use**, so no port conflict there.

---

### Next steps:

1. **Check port 9100 usage on *all* other nodes (including master and other workers):**

Make sure no node has port 9100 occupied:

```bash
ssh <node>
sudo ss -tulpn | grep :9100
```

2. **Check node labels and taints on master node**

Connect to your **master node** where kubectl works properly:

```bash
vagrant ssh master
```

Run:

```bash
kubectl get nodes --show-labels
kubectl describe nodes | grep -i taint
kubectl get ds prometheus-prometheus-node-exporter -n prometheus -o yaml | grep -A 15 affinity
```

3. **Verify that DaemonSet affinity is removed**

Verify affinity block is gone:

```bash
kubectl get ds prometheus-prometheus-node-exporter -n prometheus -o yaml | grep -A 15 affinity
```

It should return no affinity block or empty `{}`.

---

### Explanation:

- One node is refusing to schedule node-exporter pods because port 9100 is busy.
- Other two nodes are rejected due to affinity or taints.
- So far on worker2, port 9100 free — good.
- You need to check other nodes for port usage and taints.

---

### If all nodes have free port 9100 and no taints preventing scheduling, pods should start running.

If after fixing that pods are still pending:

- Paste output of `kubectl describe pod <node-exporter-pod> -n prometheus` to see fresh events.

---

### Summary:

- **No port 9100 conflict on worker2** (good).
- Check other nodes for conflict.
- Check master node for node labels and taints.
- Confirm DaemonSet affinity removed.

---

Let me know if you want help interpreting outputs when you get them!