Azure K8S runbook

发表于 更新于 分类于 ， 阅读次数：

Az Setup for a oc cluster in single node

Az Login

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
PS C:\Users\loginid> az login
A web browser has been opened at https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize. Please continue the login in the web browser. If no web browser is available or if the web browser fails to open, use device code flow with `az login --use-device-code`.
[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "85cb24e4-0bcf-4767-b44b-39eae544706e",
    "id": "bd435e85-b401-48c5-90ad-96dbefac1503",
    "isDefault": true,
    "managedByTenants": [],
    "name": "Visual Studio Professional",
    "state": "Enabled",
    "tenantId": "85cb24e4-0bcf-4767-b44b-39eae544706e",
    "user": {
      "name": "sunny_hll@hotmail.com",
      "type": "user"
    }
  }
]

Az Terraform example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
PS C:\Users\loginid> az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/bd435e85-b401-48c5-90ad-96dbefac1503"
Creating 'Contributor' role assignment under scope '/subscriptions/bd435e85-b401-48c5-90ad-96dbefac1503'
The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
{
  "appId": "414b5739-2c30-4cea-adff-d5c8ff1c6470",
  "displayName": "azure-cli-2022-12-09-06-36-24",
  "password": "if48Q~Wd6aUHekpKNLrgJWeC3Pr-sstzZHRXfa90",
  "tenant": "85cb24e4-0bcf-4767-b44b-39eae544706e"
}

PS C:\Users\loginid> $Env:ARM_CLIENT_ID = "414b5739-2c30-4cea-adff-d5c8ff1c6470"
PS C:\Users\loginid> $Env:ARM_CLIENT_SECRET = "if48Q~Wd6aUHekpKNLrgJWeC3Pr-sstzZHRXfa90"
PS C:\Users\loginid> $Env:ARM_SUBSCRIPTION_ID = "bd435e85-b401-48c5-90ad-96dbefac1503"
PS C:\Users\loginid> $Env:ARM_TENANT_ID = "85cb24e4-0bcf-4767-b44b-39eae544706e"

Az 手动创建VM

查询VM配置的地区Spot资源不太可用

1
az vm list-skus --location centralus --resource-type virtualMachines --zone --all --output table

VM Login

1
ssh -i ~/.ssh/oc_key.pem loginid@20.100.44.77

Az 部署oc

CRC setup guide: https://crc.dev/crc/getting_started/getting_started/using/

Maybe install for ubuntu OS:

1
2
3
sudo apt install gnome-keyring
sudo apt install dbus-x11
sudo snap install kubectl --classic

下载CRC

1
2
wget https://developers.redhat.com/content-gateway/rest/mirror/pub/openshift-v4/clients/crc/latest/crc-linux-amd64.tar.xz -o /var/tmp/crc-linux-amd64.tar.xz

VPN env network setting:

https://github.com/crc-org/crc/wiki/VPN-support--with-an--userland-network-stack

!!reset Disk size t0 128G as crcbundle took more than 30G space.

启动CRC

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#Started the OpenShift cluster.
sudo service haproxy stop # previously broken haproxy bind 443 port....Orz
crc start

The server is accessible via web console at:        
  https://console-openshift-console.apps-crc.testing

Log in as administrator:
  Username: kubeadmin
  Password: i4GM8-VqA42-8Eqix-pAg7h

Log in as user:
  Username: developer
  Password: developer

Use the 'oc' command line interface:
  $ eval $(crc oc-env)
  $ oc login -u developer https://api.crc.testing:6443

创建oc hello项目

环境变量

1
2
3
4
5
6
# crc 命令
echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc
# oc 命令
echo 'export PATH="/home/loginid/.crc/bin/oc:$PATH"' >> ~/.bashrc
echo 'alias k=kubectl' >> ~/.bashrc
echo 'oc login -u developer https://api.crc.testing:6443' >> ~/.bashrc
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# create project 
oc new-project my-project

# create http server
kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost  netexec --tls-cert-file /home/loginid/test.crt --tls-private-key-file /home/loginid/test.key

# expose as service
oc expose deployment hello-node --port=8080

# https://www.baeldung.com/openssl-self-signed-cert
# prepare self-signed key remove -des3
openssl genrsa -out test.key 2048

# create csr
openssl req -key test.key -new -out test.csr

# sign
openssl x509 -signkey test.key -in test.csr -req -days 365 -out test.crt

# upload
scp -i ~/.ssh/oc_key.pem /mnt/d/Git/mypages/test.key loginid@hostip:/home/loginid/test.key

scp -i ~/.ssh/oc_key.pem /mnt/d/Git/mypages/test.crt loginid@hostip:/home/loginid/test.crt

# create secret https://www.baeldung.com/openssl-self-signed-cert
oc create secret generic my-cert --from-file=/home/loginid/test.crt --from-file=/home/loginid/test.key

# mount secret for deployment pods
oc apply -f deploy.yaml

# create route on haproxy from hostName=oc-dev.centralus.cloudapp.azure.com
oc create route passthrough test-pass --service=hello-node --port=8080 --hostname=oc-dev.centralus.cloudapp.azure.com.crc.testing

本地服务测试

1
2
3
4
5
6
7
8
9
10
11
12
13

# pod DNS

curl https://hello-node-7556f558d-psbrp:8080 -k

# service DNS

curl https://hello-node-svc.optimus.svc.cluster.local -k

# cluster DNS

curl https://oc-dev.centralus.cloudapp.azure.com.crc.testing -k -v

远程连接admin console

https://www.redhat.com/en/blog/accessing-codeready-containers-on-a-remote-server

在服务器端，需要利用haproxy做外部网卡IP的端口转发到内部地址127.0.0.1，建立外网网卡与内网之间的网桥.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
sudo apt install haproxy

sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak

# 加上一下端口转发规则

frontend apps
    bind SERVER_IP:80
    bind SERVER_IP:443
    option tcplog
    mode tcp
    default_backend apps

backend apps
    mode tcp
    balance roundrobin
    option ssl-hello-chk
    server webserver1 CRC_IP check

frontend api
    bind SERVER_IP:6443
    option tcplog
    mode tcp
    default_backend api

backend api
    mode tcp
    balance roundrobin
    option ssl-hello-chk
    server webserver1 CRC_IP:6443 check

# 更改IP地址

export SERVER_IP=$(hostname --ip-address)
export CRC_IP=$(crc ip)
sudo sed -i "s/SERVER_IP/$SERVER_IP/g" haproxy.cfg
sudo sed -i "s/CRC_IP/$CRC_IP/g" haproxy.cfg

# 启动服务haproxy
sudo ststemctl start haproxy

# 诊断

sudo apt install net-tools

Java API创建Route资源

Useful Code example

https://github.com/kubernetes-client/java/wiki/3.-Code-Examples

Leader Election Example

https://github.com/kubernetes-client/java/blob/master/examples/examples-release-18/src/main/java/io/kubernetes/client/examples/LeaderElectionExample.java

安装 Helm 到 oc-dev的crc cluster上

1
2
3
4
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

huangsun@oc-dev:~$ helm version
version.BuildInfo{Version:"v3.18.1", GitCommit:"f6f8700a539c18101509434f3b59e6a21402a1b2", GitTreeState:"clean", GoVersion:"go1.24.3"}

helm chart 开源仓库

1
2
helm repo add stable https://charts.helm.sh/stable
helm search repo stable

helm 创建一个release

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# 创建一个空release
helm create gsm

# 生成目录

huangsun@oc-dev:~/helm$ cd gsm/
huangsun@oc-dev:~/helm/gsm$ ls
Chart.yaml  charts  templates  values.yaml

# intall release

huangsun@oc-dev:~/helm$ helm install myapp gsm
NAME: myapp
LAST DEPLOYED: Sun Jun  1 10:07:23 2025
NAMESPACE: optimus
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace optimus -l "app.kubernetes.io/name=gsm,app.kubernetes.io/instance=myapp" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace optimus $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace optimus port-forward $POD_NAME 8080:$CONTAINER_PORT
#查看pod状态
huangsun@oc-dev:~/helm$ k get po
NAME                         READY   STATUS    RESTARTS      AGE
hello-node-7556f558d-psbrp   1/1     Running   4             537d
myapp-gsm-7f8b95f4-g856s     0/1     Error     3 (29s ago)   50s
#查看release状态
huangsun@oc-dev:~/helm$ helm list
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
myapp   optimus         1               2025-06-01 10:07:23.835586443 +0000 UTC deployed        gsm-0.1.0       1.16.0