一.ingress简介
1.简介
Ingress 是 k8s 官方提供的用于对外暴露服务的方式,也是在生产环境用的比较多的方式,一般情况是 LB + Ingress Ctroller 方式对外提供服务,这样就可以在一个 LB 的情况下根据域名路由到对应后端的 Service。
实际上,Ingress相当于一个七层的负载均衡器,是kubernetes对反向代理的一个抽象,它的工作原理类似于Nginx,可以理解为Ingress里面建立了诸多映射规则,Ingress Controller通过监听这些配置规则并转化为Nginx的反向代理配置,然后对外提供服务。
Ingress:kubernetes中的一个对象,作用是定义请求如何转发到Service的规则。
Ingress Controller:具体实现反向代理及负载均衡的程序,对Ingress定义的规则进行解析,根据配置的规则来实现请求转发,实现的方式有很多,比如Nginx,Contour,Haproxy等。
2.ingress解决的问题
2.1Pod 漂移问题
Kubernetes 具有强大的副本控制能力,能保证在任意副本(Pod)挂掉时自动从其他机器启动一个新的,还可以动态扩容等,通俗地说,这个 Pod 可能在任何时刻出现在任何节点上,也可能在任何时刻死在任何节点上;那么自然随着 Pod 的创建和销毁,Pod IP 肯定会动态变化;那么如何把这个动态的 Pod IP 暴露出去?这里借助于 Kubernetes 的 Service 机制,Service 可以以标签的形式选定一组带有指定标签的 Pod,并监控和自动负载他们的 Pod IP,那么我们向外暴露只暴露 Service IP 就行了;这就是 NodePort 模式:即在每个节点上开起一个端口,然后转发到内部 Pod IP 上,如下图所示:此时的访问方式:http://nodeip:nodeport/
2.2端口的管理问题
采用 NodePort 方式暴露服务面临问题是,服务一旦多起来,NodePort 在每个节点上开启的端口会及其庞大,而且难以维护;这时,我们可以能否使用一个Nginx直接对内进行转发呢?众所周知的是,Pod与Pod之间是可以互相通信的,而Pod是可以共享宿主机的网络名称空间的,也就是说当在共享网络名称空间时,Pod上所监听的就是Node上的端口。那么这又该如何实现呢?简单的实现就是使用 DaemonSet 在每个 Node 上监听 80,然后写好规则,因为 Nginx 外面绑定了宿主机 80 端口(就像 NodePort),本身又在集群内,那么向后直接转发到相应Service IP就行了 。如下图所示:
2.3域名分配及动态更新问题
用户编写Ingress规则,说明那个域名对应kubernetes集群中的那个Service。
Ingress控制器动态感知Ingress服务规则的变化,然后生成一段对应的Nginx的反向代理配置。
Ingress控制器会将生成的Nginx配置写入到一个运行着的Nginx服务中,并动态更新。
到此为止,其实真正在工作的就是一个Nginx了,内部配置了用户定义的请求规则。
二.ingress的配置
准备环境,一个deployment,service,ingress,secret域名证书
1.域名重定向
#重定向,访问nginx.zhouruihao.com则会重定向到https://cloudnative-blog.com/地址
annotations:
nginx.ingress.kubernetes.io/permanent-redirect: 'https://cloudnative-blog.com/'
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-view
namespace: view
annotations:
nginx.ingress.kubernetes.io/permanent-redirect: 'https://cloudnative-blog.com/'
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-svc
port:
number: 80
2.单主机ingress和多主机ingress
2.1单主机
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-view
namespace: view
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-svc
port:
number: 80
2.2多主机
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-view
namespace: view
spec:
rules:
- host: nginx1.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx1-svc
port:
number: 80
- host: nginx2.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx2-svc
port:
number: 80
3.基于url的ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-view
namespace: view
spec:
rules:
- host: nginx1.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/nginx1"
backend:
service:
name: nginx1-svc
port:
number: 80
- host: nginx2.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/nginx2"
backend:
service:
name: nginx2-svc
port:
number: 80
4.配置https证书
4.1创建证书
kubectl create secret tls zhouruihao --key zhouruihao.com.key --cert zhouruihao.com.crt -n zhouruihao
4.2创建ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress
namespace: view
annotations:
#如果设置为true,那么http请求会强制跳转https,默认也是强制跳转https,设置为false是希望也能够访问到http请求
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
tls:
- hosts:
- nginx.zhouruihao.com
secretName: zhouruihao
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-svc
port:
number: 80
5.白名单和黑名单
5.1白名单
#白名单建议配置单域名,可以是网段也可以是单ip
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: 1.1.1.1/32,2.2.2.0/24
#想要全局配置白名单,则直接在ingress-nginx里面修改配置文件即可
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-view
namespace: view
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: 1.1.1.1/32,2.2.2.0/24
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-svc
port:
number: 80
5.2黑名单
#在ingress使用nginx配置文件,只在当前ingress生效
nginx.ingress.kubernetes.io/server-snippet
#配置黑名单,不允许1.1.1.1的ip地址访问当前域名
nginx.ingress.kubernetes.io/server-snippet: |-
deny 1.1.1.1;
#想要全局配置黑名单,则直接在ingress-nginx里面修改配置文件即可
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-view
namespace: view
annotations:
nginx.ingress.kubernetes.io/server-snippet: |-
deny 1.1.1.1;
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-svc
port:
number: 80
6.限流
参数 | 注释 |
---|---|
nginx.ingress.kubernetes.io/limit-connections | #单个IP地址允许的并发连接数。超出此限制时,默认将返回503错误。 |
nginx.ingress.kubernetes.io/limit-rps | 每秒从给定IP接受的请求数。突发限制设置为此限制乘以突发乘数,默认乘数为5。当客户端超过此限制时,将返回limit-req-status-code默认值:503。 |
nginx.ingress.kubernetes.io/limit-rpm | 每分钟从给定IP接受的请求数。突发限制设置为此限制乘以突发乘数,默认乘数为5。当客户端超过此限制时,将 返回limit-req-status-code默认值: 503。 |
nginx.ingress.kubernetes.io/limit-burst-multiplier | 突发大小限制速率的倍数。默认的脉冲串乘数为5,此注释将覆盖默认的乘数。当客户端超过此限制时,将 返回limit-req-status-code默认值: 503。 |
nginx.ingress.kubernetes.io/limit-rate-after | 最初的千字节数,在此之后,对给定连接的响应的进一步传输将受到速率的限制。必须在启用代理缓冲的情况下使用此功能。 |
nginx.ingress.kubernetes.io/limit-rate | 每秒允许发送到给定连接的千字节数。零值禁用速率限制。必须在启用代理缓冲的情况下使用此功能。 |
nginx.ingress.kubernetes.io/limit-whitelist | 每秒允许发送到给定连接的千字节数。零值禁用速率限制。必须在启用代理缓冲的情况下使用此功能。 |
nginx.ingress.kubernetes.io/limit-whitelist | 客户端IP源范围要从速率限制中排除。该值是逗号分隔的CIDR列表。 |
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-view
namespace: view
annotations:
nginx.ingress.kubernetes.io/limit-connections: "20"
nginx.ingress.kubernetes.io/limit-rps: ”10“
nginx.ingress.kubernetes.io/limit-rpm: “100”
nginx.ingress.kubernetes.io/limit-burst-multiplier: ”5“
nginx.ingress.kubernetes.io/global-rate-limit: ”100“
nginx.ingress.kubernetes.io/global-rate-limit-window: 1s
nginx.ingress.kubernetes.io/limit-rate: ”10240“
nginx.ingress.kubernetes.io/limit-rate-after: ”10240“
nginx.ingress.kubernetes.io/limit-whitelist: “1.1.1.1”
#配置限速超出后的响应码为429,默认是503
nginx.ingress.kubernetes.io/configuration-snippet: |
limit_req_status 429;
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-svc
port:
number: 80
7.爬虫
#下面示例是单个ingress域名配置反爬虫,也可以全局配置,全局配置在ingress-nginx
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-view
namespace: view
annotations:
nginx.ingress.kubernetes.io/server-snippet: |-
#禁止Scrapy等工具的抓取
if ($http_user_agent ~* (Scrapy|Curl|HttpClient)) {
return 403;
}
#禁止指定UA及UA为空的访问
if ($http_user_agent ~ "WinHttp|WebZIP|FetchURL|node-superagent|java/|FeedDemon|Jullo|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|Java|Feedly|Apache-HttpAsyncClient|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms|BOT/0.1|YandexBot|FlightDeckReports|Linguee Bot|^$" ) {
return 403;
}
#禁止非GET|HEAD|POST|OPTIONS方式的抓取
if ($request_method !~ ^(GET|HEAD|POST|OPTIONS)$) {
return 403;
}
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-svc
port:
number: 80
8.auth认证
8.1创建用户
htpasswd -c auth zrh
8.2配置secret
kubectl create secret generic zrh --from-file=auth -n view
8.3配置ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-view
namespace: view
nginx.ingress.kubernetes.io/auth-realm: Need to longin
nginx.ingress.kubernetes.io/auth-secret: zrh
nginx.ingress.kubernetes.io/auth-type: basic
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-svc
port:
number: 80
9.配置连接超时和数据大小
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-view
namespace: view
annotations:
nginx.ingress.kubernetes.io/proxy-connect-timeout: "300" ##连接超时时间,默认为5s
nginx.ingress.kubernetes.io/proxy-send-timeout: "600" ##后端服务器回转数据超时时间,默认为60s
nginx.ingress.kubernetes.io/proxy-read-timeout: "300" ##后端服务器响应超时时间,默认为60s
nginx.ingress.kubernetes.io/proxy-body-size: "500m" ##客户端上传文件,最大大小,默认为20m
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-svc
port:
number: 80
10.配置单域名日志位置
ingress-nginx默认打印所有域名日志到控制台,可以通过配置文件将每个域名日志打印到本地(需提前创建好目录,否则会报错)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-view
namespace: view
annotations:
nginx.ingress.kubernetes.io/server-snippet: |-
access_log /var/log/ingress/nginx-view/access.log upstreaminfo if=$loggable;
error_log /var/log/ingress/nginx-view/error.log;
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-svc
port:
number: 80
11.限制访问目录
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-view
namespace: view
annotations:
nginx.ingress.kubernetes.io/server-snippet: |-
location = /nginx {
return 403;
}
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-svc
port:
number: 80
12.灰度发布
环境
环境旧pod: nginx-v1
新pod: nginx-v2
nginx-v1
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-v1
namespace: view
spec:
replicas: 1
selector:
matchLabels:
app: nginx
version: v1
template:
metadata:
labels:
app: nginx
version: v1
spec:
containers:
- name: nginx1
image: nginx
---
apiVersion: v1
kind: Service
metadata:
name: nginx-v1
namespace: view
spec:
type: ClusterIP
selector:
app: nginx
version: v1
ports:
- port: 80
targetPort: 80
nginx-v2
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-v2
namespace: view
spec:
replicas: 1
selector:
matchLabels:
app: nginx
version: v2
template:
metadata:
labels:
app: nginx
version: v2
spec:
containers:
- name: nginx2
image: nginx
imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
name: nginx-v2
namespace: view
spec:
type: ClusterIP
selector:
app: nginx
version: v2
ports:
- port: 80
targetPort: 80
1.基于header请求头的流量切分
默认流量会进入nginx-v1
如果请求头中携带version: v2的信息,那么流量会进入nginx-v2
1.1nginx-v1的ingress(默认流量)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-v1
namespace: view
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-v1
port:
number: 80
1.2nginx-v2的ingress(匹配请求头)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-v2
namespace: view
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "version" #基于Request Header的流量切分
nginx.ingress.kubernetes.io/canary-by-header-pattern: "v2" #Header信息中带有 version=v2 的转发到ingress
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-v2
port:
number: 80
1.3流量验证
[root@master01]# curl https://zhouruihao.com
nginx-v1
[root@master01]# curl -H "version: v2" https://zhouruihao.com
nginx-v2
2.基于Cookie的流量切分
默认流量会进入nginx-v1
如果cookie中的信息带有version,那么流量会进入nginx-v2
2.1nginx-v1的ingress(默认流量)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-v1
namespace: view
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-v1
port:
number: 80
2.2nginx-v2的ingress(匹配cookie)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-v2
namespace: view
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-cookie: "version" #匹配cookie携带version,使用该ingress规则
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-v2
port:
number: 80
2.3流量验证
[root@master01]# curl https://zhouruihao.com
nginx-v1
[root@master01]# curl -s --cookie "version=always" https://zhouruihao.com
nginx-v2
3.基于权重的流量切分
90%流量会进入nginx-v1
10%流量会进入nginx-v2
3.1nginx-v1的ingress(权重90%)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-v1
namespace: view
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-v1
port:
number: 80
3.2nginx-v2的ingress(权重10%)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-v2
namespace: view
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "10" #10%的流量会转发到这个ingress
spec:
rules:
- host: nginx.zhouruihao.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nginx-v2
port:
number: 80
3.3流量验证
[root@master01]# for i in {1..10}; do curl https://nginx.zhouruihao.com; done;
nginx-v1
nginx-v1
nginx-v1
nginx-v1
nginx-v1
nginx-v1
nginx-v1
nginx-v1
nginx-v1
nginx-v2
评论区