PRD: External API Access Issue

Problem Statement

After deploying Kube-OVN with external gateway configuration, the Kubernetes API server becomes unreachable from external networks. The public IP address (213.111.154.233) is moved to an OVS bridge (br-ext-cloud), breaking the network path for incoming API connections.

Current Behavior

Before Kube-OVN deployment: API accessible via https://213.111.154.233:6443
After Kube-OVN deployment: Connection refused/reset on port 6443

Root Cause

When Kube-OVN configures the external gateway:

Creates OVS bridge br-ext-cloud on VLAN 884
Moves the public IP from physical interface eno1.884 to br-ext-cloud
The API server binds to 0.0.0.0:6443 but network path is disrupted

# IP is now on OVS bridge, not physical interface
$ ip addr show br-ext-cloud
    inet 213.111.154.233/25 brd 213.111.154.255 scope global br-ext-cloud

Current Workaround

Using SSH tunnel to forward API server port:

# Create tunnel
ssh -N -L 16443:192.168.0.2:6443 root@213.111.154.233 &

# Use tunneled kubeconfig
export KUBECONFIG=/home/voa/.kube/cloud_kubeconfig_tunnel
# server: https://localhost:16443

Requirements

Must Have

External API access without SSH tunnel
No disruption to Kube-OVN networking functionality
Compatible with RKE2 and kube-vip

Should Have

Automatic recovery after node reboot
Works with all master nodes (HA setup)

Could Have

Separate VIP for API server independent of node IPs

Proposed Solutions

Option A: Configure kube-vip for External Access

Use kube-vip to maintain a floating VIP on the physical interface while Kube-OVN manages the OVS bridge.

Pros:

Standard HA pattern
Works with RKE2 manifest-based kube-vip

Cons:

Requires additional IP address
May conflict with OVS bridge management

Option B: OVS Bridge Port Configuration

Configure OVS to properly route traffic destined for port 6443 to the API server.

# Add flow rule to forward API traffic
ovs-ofctl add-flow br-ext-cloud "priority=100,tcp,tp_dst=6443,actions=normal"

Pros:

No additional IP needed
Works within existing OVS infrastructure

Cons:

Complex OVS configuration
May need to be reapplied after updates

Option C: Dedicated API Endpoint Network

Keep API server on private VLAN (885) and expose via LoadBalancer or Ingress.

Pros:

Clean separation of concerns
More secure (API not on public network)

Cons:

Requires additional infrastructure
Changes access patterns

Affected Files

/home/voa/projects/kube-dc.cloud/installer/kube-dc/templates/kube-dc/kube-ovn/setup-external-route.sh
/home/voa/projects/kube-dc.cloud/installer/kube-dc/templates/kube-dc/template.yaml (external route commented out)
/home/voa/projects/kube-dc.cloud/installer/kube-dc/stack.yaml (kubeconfig path)

Environment Details

Public IPs: 213.111.154.233, 213.111.154.229, 213.111.154.223
Private IPs: 192.168.0.2, 192.168.0.3, 192.168.0.4
VLAN 884: External/cloud network
VLAN 885: Kubernetes internal network
Kube-OVN version: v1.15.0

Current Network Topology

Physical Layer

┌─────────────────────────────────────────────────────────────────────────────┐
│                           Physical Network                                   │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐              │
│  │ ams1-blade179-8 │  │ ams1-blade184-5 │  │ ams1-blade58-2  │              │
│  │ (control-plane) │  │ (control-plane) │  │ (control-plane) │              │
│  └────────┬────────┘  └────────┬────────┘  └────────┬────────┘              │
│           │                    │                    │                        │
│  ┌────────┴────────┐  ┌────────┴────────┐  ┌────────┴────────┐              │
│  │ enp94s0f0np0    │  │ enp94s0f0np0    │  │ enp94s0f0np0    │              │
│  │ 213.111.154.233 │  │ 213.111.154.229 │  │ 213.111.154.223 │              │
│  └────────┬────────┘  └────────┴────────┘  └────────┴────────┘              │
│           │                                                                  │
│           ├── vlan885 ──► 192.168.0.2/18 (K8s internal)                     │
│           │                                                                  │
│           └── br-ext-cloud ──► 213.111.154.233/23 (OVS bridge, VLAN 884)    │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Nodes

Node	Role	Internal IP	External IP
ams1-blade179-8	control-plane,etcd	192.168.0.2	213.111.154.233
ams1-blade184-5	control-plane,etcd	192.168.0.3	213.111.154.229
ams1-blade58-2	control-plane,etcd	192.168.0.4	213.111.154.223
ams1-blade187-2	worker	192.168.0.10	-

Network Interfaces (Master Node 1)

enp94s0f0np0     UP    213.111.154.233/23    # Physical NIC (public)
vlan885          UP    192.168.0.2/18        # VLAN for K8s internal
ovn0             UP    172.30.0.2/22         # OVN overlay (join network)
br-ext-cloud     UP    213.111.154.233/23    # OVS bridge for external

Issue: Same IP (213.111.154.233) on both enp94s0f0np0 AND br-ext-cloud

Routing Table

default via 213.111.154.1 dev br-ext-cloud proto static      # Default route (OVS)
default via 213.111.154.1 dev enp94s0f0np0 proto static      # Default route (physical) - DUPLICATE
100.0.0/16 via 172.30.0.1 dev ovn0                        # Pod network via OVN
30.0.0/22 dev ovn0                                       # Join network
168.0.0/18 dev vlan885                                   # K8s internal (VLAN 885)
111.154.0/23 dev br-ext-cloud                            # External via OVS bridge
111.154.0/23 dev enp94s0f0np0                            # External via physical - DUPLICATE

Kube-OVN Resources

VPCs:

Name	External	Subnets
ovn-cluster	false	ext-cloud, join, ovn-default
shalb-demo	true	shalb-demo-default

Subnets:

Name	CIDR	Gateway	VLAN
ext-cloud	100.65.0.0/16	100.65.0.1	vlan884
join	172.30.0.0/22	172.30.0.1	-
ovn-default	10.100.0.0/16	10.100.0.1	-
shalb-demo-default	10.0.10.0/24	10.0.10.1	-

Provider Network:

Name: ext-cloud
Default Interface: enp94s0f0np0
VLAN ID: 884

Traffic Flow

External Client                    Pod/VM
      │                               │
      ▼                               ▼
213.111.154.233:443           10.0.10.x (project subnet)
      │                               │
      ▼                               ▼
┌─────────────┐               ┌───────────────┐
│ br-ext-cloud│◄─────────────►│   ovn0        │
│ (OVS bridge)│   NAT/SNAT    │ (OVN overlay) │
└─────────────┘               └───────────────┘
      │                               │
      ▼                               ▼
   VLAN 884                    OVN Logical Router
 (external)                    (ovn-cluster)

Known Issues in Topology

Duplicate IP Assignment: 213.111.154.233 assigned to both physical NIC and OVS bridge
Duplicate Default Routes: Two default routes via same gateway on different interfaces
OVS Bridge Init Failed: ams1-blade58-2 shows InitOVSBridgeFailed status

Kube-OVN 1.15 VLAN Subinterface Features

Kube-OVN v1.15.0 (released 2026-01-01) introduces new features for automatic VLAN interface management that could resolve our topology issues:

New ProviderNetwork Spec Fields

apiVersion: kubeovn.io/v1
kind: ProviderNetwork
metadata:
  name: ext-cloud
spec:
  defaultInterface: enp94s0f0np0.884    # VLAN-style interface name
  autoCreateVlanSubinterfaces: true     # NEW: Auto-create missing VLAN subinterfaces
  preserveVlanInterfaces: true          # NEW: Preserve existing VLAN configs
  vlanInterfaces:                       # NEW: Explicit list of VLAN interfaces to manage
    - enp94s0f0np0.884
    - enp94s0f0np0.885

Feature 1: Auto Create VLAN Sub-interfaces (PR #5966)

When autoCreateVlanSubinterfaces: true and the interface name is VLAN-style (e.g., eth0.10):

Kube-OVN will automatically create the missing VLAN subinterface on the parent NIC
No manual ip link add commands needed on nodes
Works during node initialization

Use Case: Specify defaultInterface: enp94s0f0np0.884 and Kube-OVN creates it automatically.

Feature 2: Auto Move VLAN Interfaces to OVS Bridges (PR #5949)

New fields preserveVlanInterfaces and vlanInterfaces enable:

Automatic detection of existing VLAN interfaces
IP address transfer: Moves IPs from kernel VLAN interfaces to OVS internal ports
Route preservation: Transfers routing table entries to OVS ports
Cleanup/Restoration: Properly restores kernel VLAN interfaces on deletion

This solves our duplicate IP issue: Instead of having IPs on both physical NIC and OVS bridge, Kube-OVN properly migrates the IP to OVS-managed internal ports.

Proposed Configuration for Cloud Cluster

apiVersion: kubeovn.io/v1
kind: ProviderNetwork
metadata:
  name: ext-cloud
spec:
  defaultInterface: enp94s0f0np0
  autoCreateVlanSubinterfaces: true
  preserveVlanInterfaces: true
  vlanInterfaces:
    - enp94s0f0np0.884   # Will be created and managed by OVS

Testing Plan

Upgrade to Kube-OVN 1.15.0:
```
# Update stack.yaml
kube_ovn: "v1.15.0"
```

Update ProviderNetwork with new fields:

kubectl patch providernetwork ext-cloud --type=merge -p '
{
  "spec": {
    "autoCreateVlanSubinterfaces": true,
    "preserveVlanInterfaces": true
  }
}'

Verify VLAN interface creation:

# On each node, check if VLAN subinterface was created
ip link show | grep "enp94s0f0np0\."

# Check OVS bridge has the VLAN interface
ovs-vsctl show | grep -A5 "br-ext-cloud"

Verify IP migration (should have IP on OVS port only, not physical):

ip addr show enp94s0f0np0      # Should NOT have 213.111.154.x
ip addr show br-ext-cloud      # Should have 213.111.154.x

Verify routes are correct (single default route):

ip route | grep default        # Should show only br-ext-cloud

Benefits

Eliminates duplicate IP/route issues - Clean network state
Automatic VLAN management - No manual configuration on nodes
Proper IP migration - IPs move cleanly to OVS bridges
Route preservation - Existing routes work with OVS

References

Success Criteria

kubectl get nodes works from external machine without SSH tunnel
API server accessible on all master nodes
Kube-OVN external gateway functions correctly
No manual intervention required after node reboot

Priority

High - Blocks production deployment and operator access

Kube-OVN external gateway configuration
RKE2 bootstrap network setup

Problem Statement​

Current Behavior​

Root Cause​

Current Workaround​

Requirements​

Must Have​

Should Have​

Could Have​

Proposed Solutions​

Option A: Configure kube-vip for External Access​

Option B: OVS Bridge Port Configuration​

Option C: Dedicated API Endpoint Network​

Affected Files​

Environment Details​

Current Network Topology​

Physical Layer​

Nodes​

Network Interfaces (Master Node 1)​

Routing Table​

Kube-OVN Resources​

Traffic Flow​

Known Issues in Topology​

Kube-OVN 1.15 VLAN Subinterface Features​

New ProviderNetwork Spec Fields​

Feature 1: Auto Create VLAN Sub-interfaces (PR #5966)​

Feature 2: Auto Move VLAN Interfaces to OVS Bridges (PR #5949)​

Proposed Configuration for Cloud Cluster​

Testing Plan​

Benefits​

References​

Success Criteria​

Priority​

Related Issues​

Problem Statement

Current Behavior

Root Cause

Current Workaround

Requirements

Must Have

Should Have

Could Have

Proposed Solutions

Option A: Configure kube-vip for External Access

Option B: OVS Bridge Port Configuration

Option C: Dedicated API Endpoint Network

Affected Files

Environment Details

Current Network Topology

Physical Layer

Nodes

Network Interfaces (Master Node 1)

Routing Table

Kube-OVN Resources

Traffic Flow

Known Issues in Topology

Kube-OVN 1.15 VLAN Subinterface Features

New ProviderNetwork Spec Fields

Feature 1: Auto Create VLAN Sub-interfaces (PR #5966)

Feature 2: Auto Move VLAN Interfaces to OVS Bridges (PR #5949)

Proposed Configuration for Cloud Cluster

Testing Plan

Benefits

References

Success Criteria

Priority

Related Issues