第1章 CDN与反向代理缓存
1.1 CDN
CDN解决了用户访问最后一公里的问题。
参考 腾讯CDN白皮书
1.1.1 CDN关键技术
- 内容调度:智能dns(根据你的localDNS的IP来判断)
- 监测系统:监测各个节点的链路状态和健康状态
- 内容分发:预缓存(PUSH(分发)、PULL(回源))
- 过期刷新:缓存刷新
- 数据存储:文件的存储(内容源存储、Cache节点存储)
- 内容管理:鉴黄等不良内容管理
- 用户配置中心:saltstack/ansible
- 计费系统:商业CDN功能
- 数据可视化:日志分析,日志下载
- 防攻击:WAF
1.1.2 CDN请求流程
1.1.3 腾讯云CDN基本架构
1.2 Nginx反向代理缓存
1.2.1 编写缓存配置文件
[root@linux-node1 ~]# mkdir -p /data/cdn_cache/{proxy_temp_dir,proxy_cache_dir} [root@linux-node1 ~]# chown -R nginx.nginx /data/cdn_cache/ [root@linux-node1 ~]# vim /etc/nginx/conf.d/proxy.conf # CDN proxy_temp_path /data/cdn_cache/proxy_temp_dir; proxy_cache_path /data/cdn_cache/proxy_cache_dir levels=1:2 keys_zone=cache_one:50m inactive=1d max_size=1g; proxy_connect_timeout 5; proxy_read_timeout 60; proxy_send_timeout 5; proxy_buffer_size 16k; proxy_buffers 4 64k; proxy_busy_buffers_size 128k; proxy_temp_file_write_size 128k; proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_404;
- 配置解析:
- proxy_temp_path:缓存的临时目录
- proxy_cache_path:缓存的目录
- levels=1:2:缓存级别1:2,最后一位做一级目录,倒数第二和第三位做二级目录
- keys_zone=cache_one:50m:缓存名称cache_one,内存缓存大小50M
- inactive=1d:自动清除1天没访问的文件
- max_size=1g:硬盘缓存大小
- proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_404:后端服务器出现500,502等异常后,可以分配到下一台服务器程序继续处理,提高平台访问成功率
1.2.2 模拟反向代理环境
1.2.2.1 编辑nginx负载均衡和反向代理配置文件
[root@linux-node1 ~]# vim /etc/nginx/conf.d/upstream.conf upstream www.leonshadow.com.pool { server 10.10.10.102:80 weight=10 max_fails=3; } [root@linux-node1 ~]# vim /etc/nginx/conf.d/www.leonshadow.com.conf server { listen 80; server_name www.leonshadow.com; access_log /var/log/nginx/www.leonshadow.com-access.log main; location ~ .*\.(gif|jpg|png|html|htm|css|js|ico|swf|pdf)$ { #Proxy proxy_redirect off; proxy_next_upstream http_502 http_504 http_404 error timeout invalid_header; proxy_set_header Host $host; proxy_set_header X-real-ip $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://www.leonshadow.com.pool; # Use Proxy Cache proxy_cache cache_one; proxy_cache_key "$host$request_uri"; add_header Cache "$upstream_cache_status"; proxy_cache_valid 200 304 301 302 8h; proxy_cache_valid 404 1m; proxy_cache_valid any 2d; } location / { proxy_redirect off; proxy_next_upstream http_502 http_504 http_404 error timeout invalid_header; proxy_set_header Host $host; proxy_set_header X-real-ip $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://www.leonshadow.com.pool; client_max_body_size 40m; client_body_buffer_size 128k; proxy_connect_timeout 60; proxy_send_timeout 60; proxy_read_timeout 60; proxy_buffer_size 64k; proxy_buffers 4 32k; proxy_busy_buffers_size 64k; } }
1.2.2.2 修改web服务器(10.10.102)配置文件
[root@linux-node2 ~]# vim /etc/nginx/nginx.conf …… server { listen 80; listen [::]:80; server_name www.leonshadow.com; root /usr/share/nginx/html; ……
1.2.2.3 修改本机及反向代理服务器(10.10.10.101)host文件
[root@linux-node1 ~]# vim /etc/hosts 10.10.10.102 leonshadow.com
1.2.3 测试运行结果
[root@linux-node1 ~]# ps -ef |grep nginx root 1737 1 0 12:43 00:00:00 nginx: master process /usr/sbin/nginx nginx 1738 1737 0 12:43 00:00:00 nginx: worker process nginx 1739 1737 0 12:43 00:00:00 nginx: worker process # 出现了cache管理和调度的进程 nginx 1740 1737 0 12:43 00:00:00 nginx: cache manager process nginx 1741 1737 0 12:43 00:00:00 nginx: cache loader process root 1748 1411 0 12:43 pts/0 00:00:00 grep --color=auto nginx [root@linux-node1 ~]# tree /data/cdn_cache/ /data/cdn_cache/ ├── proxy_cache_dir │ ├── 1 │ │ └── 11 │ │ └── 42169aa7a6fa07b22938ae0a038f1111 │ ├── 5 │ │ └── a6 │ │ └── fbd9701fec88c134cbaa678f61797a65 │ ├── a │ │ └── 47 │ │ └── 6ad8d83c76f7e7682c4cf2281de3c47a │ └── e │ └── 13 │ └── 7d5301564cfea4a616a8b5c5e4d4c13e └── proxy_temp_dir 10 directories, 4 files
1.2.4 Nginx CDN缓存规则
Nginx根据proxy_cache_key "$host$request_uri";参数设置缓存文件的KEY,再根据KEY值计算MD5值,将MD5值根据levels=1:2的设置将最后一位设置为一级目录,将倒数第二位和倒数第三位设置为二级目录:
[root@linux-node1 ~]# tree /data/cdn_cache/ /data/cdn_cache/ ├── proxy_cache_dir │ ├── 1 │ │ └── 11 │ │ └── 42169aa7a6fa07b22938ae0a038f1111 │ ├── 5 │ │ └── a6 │ │ └── fbd9701fec88c134cbaa678f61797a65 │ ├── a │ │ └── 47 │ │ └── 6ad8d83c76f7e7682c4cf2281de3c47a │ └── e │ └── 13 │ └── 7d5301564cfea4a616a8b5c5e4d4c13e └── proxy_temp_dir 10 directories, 4 files [root@linux-node1 ~]# head /data/cdn_cache/proxy_cache_dir/1/11/42169aa7a6fa07b22938ae0a038f1111 \N _ N _ KEY: www.leonshadow.com/index.htm HTTP/1.1 404 Not Found Server: nginx/1.16.1 Date: Sat, 11 Jul 2020 05:29:04 GMT Content-Type: text/html Content-Length: 3650 Connection: close ETag: "5d958342-e42" [root@linux-node1 ~]# echo -n "www.leonshadow.com/index.htm" | md5sum 42169aa7a6fa07b22938ae0a038f1111 -
1.2.5 删除Nginx的CDN缓存
1.2.5.1 编写删除脚本
[root@linux-node1 ~]# vim /server/scripts/nginx_purge.sh #!/bin/bash cache_purge(){ PURGE_URL=$1 URL_NAME=$(echo -n $PURGE_URL | md5sum | awk '{print $1}') FILE_NAME=$(echo $URL_NAME | awk '{print "/data/cdn_cache/proxy_cache_dir/"substr($0,length($0),1)"/"substr($0,length($0 )-2,2)"/"$0}') rm -rf $FILE_NAME } purge_file(){ PURGE_FILE=$1 for url in $(cat $PURGE_FILE);do cache_purge $url done } purge_url(){ PURGE_URL=$1 cache_purge $PURGE_URL } usage(){ echo $"Usage: $0 <url_file | 'url'>" } main (){ if [ "$#" -ne 1 ];then usage; else if [ -f $1 ];then purge_file $1; else purge_url $1; fi fi } main $1
1.2.5.2 脚本使用规则
[root@linux-node1 scripts]# ./nginx_purge.sh Usage: ./nginx_purge.sh <url_file | 'url'>
- ./nginx_purge.sh url:删除单个url的缓存
- ./nginx_purge.sh url_file:根据文件中列出的url批量删除
1.2.5.3 测试脚本
- 未删除缓存之前:
- 执行命令删除后:
[root@linux-node1 scripts]# ./nginx_purge.sh www.leonshadow.com/index.html


我的微信
如果有技术上的问题可以扫一扫我的微信