11-26 18,875 views
很多时候需要对nginx的日志进行分析,首先需要做nginx的日志回滚,不然一个日志文件会无限增长,最终会占满整个硬盘空间,导致nginx挂起,开启了nginx日志就必须做好日志回滚,
一般用系统自带的日志回滚,如下:
[root@localhost ~]# ll /etc/logrotate.d/
total 24
-rw-r–r–. 1 root root 103 Feb 22 2013 dracut
-rw-r–r– 1 root root 185 Aug 15 14:57 httpd
-rw-r–r– 1 root root 121 Nov 25 11:03 nginx
首先需要写一个nginx的配置文件:
/usr/local/nginx/logs/test.access.log #需要切割日志的文件位置
{
nocompress
daily
copytruncate
create
ifempty
olddir /home/nginx/logs #切割后文件的存放位置
rotate 0
}
然后需要写一个日志回滚脚本,如下:
#!/bin/bash
/usr/sbin/logrotate -vf /etc/logrotate.d/nginx
time=$(date -d “yesterday” +”%Y-%m-%d”)
times=$(date -d “-2 day” +”%Y-%m-%d”)
cd /home/nginx/logs # 根据nginx中定义存放日志目录来打开这个目录
for i in $(ls ./ | grep “^\(.*\)\.[[:digit:]]$”)
do
mv ${i} ./$(echo ${i}|sed -n ‘s/^\(.*\)\.\([[:digit:]]\)$/\1/p’)-$(echo $time)
done
for name in {www,down}
do
if [ -f “$name.test.cn_access.log-$(echo $times)” ];then
tar zcvf /home/nginx/logs/$name.test.cn_access.log-$(echo $times).tar.gz ./$name.test.cn_access.log-$(echo $times)
rm -rf ./$name.test.cn_access.log-$(echo $times)
else
echo “The file don’t exist.”
fi
done
find /home/nginx/logs/* -name “*.tar.gz” -mtime 30 -type f -exec rm -rf {} \;
根据自己的配置更改相应的目录即可,运行测试正常后,最后就是加入计划任务:
0 0 * * * /bin/sh log-nginx.sh >/dev/null 2>&1
注意:这种日志回滚的方案需要nginx日志文件和回滚后的nginx日志文件要在同一设备中,否则会报错。
这样切割后的日志就会保存在/home/nginx/logs下面,nginx日志回滚完成,下面开始搭建Awstats日志分析,
首先nginx的日志应该遵循以下格式,才可以被awstats识别,如定义日志格式
log_format main ‘$remote_addr – $remote_user [$time_local] “$request” ‘
‘$status $body_bytes_sent “$http_referer” ‘
‘”$http_user_agent” “$http_x_forwarded_for”‘;
access_log logs/access.log main;
在server中加入
access_log logs/test.com_access.log main;
上面计划任务中nginx日志是0点开始切割,切割昨天的日志(交由awstats分析),压缩前天的日志(压缩日志可减小存储空间,为防止awstats没有分析完就被压缩,所以只压缩前天的日志)。如果你的nginx和log文件放的路径和我的不一样,请对应修改。
好了,开始安装awstats了,下载awstats-7.0.tar.gz,如附件:awstats-7.0.tar
tar -zxvf awstats-7.0.tar.gz
mv awstats-7.0 /usr/local/awstats
修改权限,wget下载下来的包中权限是非root的,赋予过权限之后,.pl的文件也就可以运行了。
chown -R root:root /usr/local/awstats
chmod +x /usr/local/awstats/tools/*.pl
chmod +x /usr/local/awstats/wwwroot/cgi-bin/*.pl
然后执行 tools 目录中的 awstats_configure.pl 配置向导,创建一个新的统计
运行(注意这里要在当前目录运行。否则会有一些关于标准目录的提示。)
cd /usr/local/awstats/tools
./awstats_configure.pl
将会有如下一些提示:
—–> Running OS detected: Linux, BSD or Unix
—–> Check for web server install
Enter full config file path of your Web server.
Example: /etc/httpd/httpd.conf
Example: /usr/local/apache2/conf/httpd.conf
Example: c:\Program files\apache group\apache\conf\httpd.conf
Config file path (‘none’ to skip web server setup):
>none #这里添none并回车,因为我们没有使用apache
回车之后下一个选项
Your web server config file(s) could not be found.
You will need to setup your web server manually to declare AWStats
script as a CGI, if you want to build reports dynamically.
See AWStats setup documentation (file docs/index.html)
—–> Update model config file ‘/usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf’
File awstats.model.conf updated.
—–> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ?
这里选Y,创建一个新的配置文件
—–> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
>test.com #这里输入你要分析的域名,或是随便一个你易记的配置名并回车
接下来要定义你的配置文件存放的路径,可用默认
—–> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):
> #直接回车,使用默认路径/etc/awstats
回车后的提示
—–> Create config file ‘/etc/awstats/awstats.akii.org.conf’
Config file /etc/awstats/awstats.akii.org.conf created.
—–> Add update process inside a scheduler
Sorry, configure.pl does not support automatic add to cron yet.
You can do it manually by adding the following command to your cron:
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=akii.org
Or if you have several config files and prefer having only one command:
/usr/local/awstats/tools/awstats_updateall.pl now
Press ENTER to continue… #按回车继续
A SIMPLE config file has been created: /etc/awstats/awstats.akii.org.conf
You should have a look inside to check and change manually main parameters.
You can then manually update your statistics for ‘yuyuanchun.com’ with command:
> perl awstats.pl -update -config=akii.org
You can also build static report pages for ‘akii.org’ with command:
> perl awstats.pl -output=pagetype -config=akii.org
Press ENTER to finish… #回车完成配置文件的创建
完成配置文件的创建后,我们还要修改一下。因为我们是按天切割的日志,切割完成后交由awstats去分析。并不是让awstats去分时正在时时增长的也就是正在被写入的日志,这样的好处是不至于遗漏数据,并且分析已经切割完成的日志,更不用担心会有冲突。坏处是我一天切割一次日志,你要等第二天才能看昨天的一些详细数据。
以上步骤默认会产生配置文件 ==>> /etc/awstats/awstats.test.com.conf
awstats 配置文件的命名规则:awstats.website.conf
修改配置文件
修改/etc/awstats/awstats.test.com.conf,执行:
vi /etc/awstats/awstats.test.com.conf
找到
LogFile=”/var/log/httpd/mylog.log”
修改为:
LogFile=”/home/nginx/logs/www.test.com_access.log-%YYYY-24-%MM-24-%DD-24″ 服务器的配置
如果你的日志路径和我的不一样,请修改成对应的日志文件名。以上的完整路径是切割后保存的nginx日志文件。其中%YYYY-24/%MM-24/%DD-24表示年月日都减去24小时,也就是昨天的日志目录。修改完成后按:wq保存退出。
接下来可以测试一下awstats分析日志了(前提是你已经有了切割过的日志,没有的话可以先退行一下切割日志的脚本/root/log-nginx.sh)
首先,还要创建一个awstats用于记录数据的目录
mkdir -p /var/lib/awstats
[root@localhost ~]# cd /var/lib/awstats/
[root@localhost awstats]# ls
awstats.www.test.com.txt
然后运行awstats的wwwroot目录中的awatsts.pl来测试一下
[root@test]# /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=test.com
你如果看到类似下面的提示就说明配置文件都正确了。
awstats.pl 会去读取-config中的参数,把test.com扩展成 awstats.test.com.conf 去 /etc/awstats 目录或者当期目录下搜索,根据读取到的配置文件运行程序。
分析日志:运行后将这样的日志统计结果归档到一个AWStats的数据库(纯文本)里;
然后是输出:分两种形式
1. 一种是通过cgi程序读取统计结果数据库输出;
2. 一种是运行后台脚本将输出导出成静态文件;
统计分析完成后,结果还在 Awstats 的数据库中。在 Apache 上,可以直接打开 Perl 程序的网页查看统计。 但本文开始时已经提到,Nginx 对 Perl 支持并不好,所以我们要换个方法,利用 awstats 的工具将统计的结果生成静态文件,具体的步骤如下:
awstats_buildstaticpages.pl 会根据 -config 的参数去读取里面的 DirData 路径下数据库配置文件,然后生成静态的html文件,生成的文件重定向到/var/www/HduISA/awstats/douxie.com。
最后是nginx的配置文件,根据自己路径配置
* 首先在 webroot 目录下创建一个文件夹。例:/home/www/awstats
* 写一个脚本,定期执行让 Awstats 把静态页面生成到该目录中
先生成存放awstats生成的静态文件的目录,我这里用的是/home/www/awstats
mkdir -p /home/www/awstats
我们来写一个脚本
vim /root/awstats.sh
然后输入以下内容
#!/bin/bash
mkdir -p /home/www/awstats/akii.org
/usr/local/awstats/tools/awstats_buildstaticpages.pl -update \
-config=test.com -lang=cn -dir=/home/www/awstats/test.com \
-awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats.pl
上述命令的具体意思如下:
* /usr/local/awstats/tools/awstats_buildstaticpages.pl Awstats 静态页面生成工具
* -update -config=test.com 更新配置项
* -lang=cn 语言为中文
* -dir=/home/www/awstats 统计结果输出目录
* -awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats.pl Awstats 日志更新程序路径。
然后在你的nginx的配置文件中,在你想要安置awstats或默认的ip或域名的server段中,加入关于awstats和icon的两个目录配置。如下:
接下来可以测试一下脚本是否可以正确执行
还是别忘了给它可执行权限
chmod +x /root/awstats.sh
/root/awstats.sh
如果你看到它生成了一堆网页,那就说明成功了。
输出信息部分例如
然后可以把它加入自动运行了。
配置awstats脚本自动运行
crontab -e 加入
00 1 * * * /root/awstats.sh
然后保存退出。
这样就可以每天在凌晨自动分割日志,并且开始自动用awstats分析nginx的日志了。
解决awstats用以搜索的关键词都乱码了的问题:
Awstats是一套非常好用的免费的日志分析软件,他是用perl实现的,支持web log、ftp log和mail log;而且它还能自动根据你浏览器的字符设置来选取语言(支持中文). 但是缺省安装的话有个问题,就是用来搜索的关键字如果是中文的话显示出来是乱码的.之所以搜索的关键字句会变成乱码的原因,主要是因为现在的搜索引擎都是使用UTF8,而Awstats是使用decodeUTFkeys这个plugin来处理搜索引擎的UTF8关键字,默认是没有打开的,所以在显示上会出现乱码。
要解决中文乱码问题,方法也很简单.
1>修改配置文件awstats.xxx.conf
在配置文件中把decodeutfkeys这个plugin打开就可以了。在配置文件中找到:
#LoadPlugin=”decodeutfkeys”
去掉前面的#就可以了。然后直接运行sh awstats.sh。
不过打开这个plugin 需要perl模块儿Encode和URI::Escape的支持,一般情况下perl是默认支持的。
配置 awstats后,如果打开AWStats会显示:
Error: Plugin load for plugin ‘decodeutfkeys’ failed with return code: Error: Can’t locate URI/Escape.pm in @INC (@INC contains: /usr/lib/perl5/5.8.7/i686-linux /usr/lib/perl5/5.8.7 /usr/lib/perl5/site_perl/5.8.7/i686-linux /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl . ./lib ./plugins) at (eval 4) line 1.
這是因為缺少 Perl 的 URI:Escape模块
2>下载安装URI:Escape模块
[root@test]#wget http://download.librewrt.org/sources/URI-1.36.tar.gz
解压缩文件
tar zxvf URI-1.36.tar.gz
进入解压缩后目录
cd URI-1.36
然后 perl Makefile.PL
[root@test URI-1.36]# perl Makefile.PL
Checking if your kit is complete…
Looks good
Writing Makefile for URI
说明正常,可以直接make && make install安装完成
[root@AY140224161243222ad2Z URI-1.36]# perl Makefile.PL 如果出现
Can’t locate ExtUtils/MakeMaker.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at Makefile.PL line 2.
BEGIN failed–compilation aborted at Makefile.PL line 2.
解决方法:
yum -y install perl-devel
调试后一下,分析出来的搜索关键字就可以正常显示了。
直接 sh awstats.sh 后在查看页面。
补充:nginx认证配置,首先需要安装httpd。
在需要创建密码的地址执行:
htpasswd -b -c site_pass username password
创建你的密码文件site_pass ,里面包含你的用户名和密码。
Pingback: roshe run mid id
Pingback: bayern trikots 眉bersicht
Pingback: oakley m frame with helo kit
Pingback: 胶管接头
Pingback: maillot psg 2014 pas cher
Pingback: ray ban 2151 vs 2132
Pingback: asics gel pulse
Pingback: asics gel nimbus 14 review
Pingback: asics gel kinsei 5
Pingback: jordan 11 Low Bred online
Pingback: jordan 11 Low Bred
Pingback: Air Jordan 11 Low Bred
Pingback: authentic jordan 11 Low Bred
Pingback: Jordan 11 Low
Pingback: jordan 11 Low Bred
Pingback: jordan 11 Low Bred 2015
Pingback: cheap Jordan 11 Bred
Pingback: jordan 11 Low Bred 2015
Pingback: Low Bred 11s
Pingback: Low Bred 11s
Pingback: Jordan 11 Low
Pingback: Low Bred 11s for sale
Pingback: Low Bred 11s online
Pingback: cheap Jordan 11 Bred
Pingback: air jordan 11 Low Bred
Pingback: Low Bred 11s
Pingback: order Low Bred 11s
Pingback: Low Bred 11s 2015
Pingback: jordan Low Bred 11s
Pingback: Low Bred 11s
Pingback: air jordan 11 retro Low Bred
Pingback: cheap Jordan 11 Low
Pingback: Jordan 11 Low for sale
Pingback: asics womens running shoes
Pingback: air jordan 11 retro Low Bred
Pingback: jordan 11 bred
Pingback: nike free shoes for kids
Pingback: Low Bred 11s
Pingback: Jordan 11 Low
Pingback: cheap asics
Pingback: 石膏板价格
Pingback: asics gel lyte 3 for sale
Pingback: asics gel blur
Pingback: 涂料厂家
Pingback: asics gel nimbus 14 womens
Pingback: nike roshe pronunciation
Pingback: Tiffany And Co Vegas
Pingback: Tiffany Lamp Designer
Pingback: Tiffany Co Mens Watches
Pingback: 塑料网
Pingback: 水泥发泡保温板
Pingback: Cheap Tiffany And Co Jewellery
Pingback: Tiffany York Uk
Pingback: authenticsaleshoes
Pingback: Tiffany Jewelry Outlet Sale
Pingback: Tiffany Diamonds
Pingback: Retired Tiffany Jewelry
Pingback: safest wow gold
Pingback: cheap new balance
Pingback: Tiffany Lamps Rochester Ny
Pingback: Tiffany Rings Hk
Pingback: Tiffany Key Chain
Pingback: Tiffany And Co Australia Opening Times
Pingback: Tiffany And Co Venetian Link Bracelet
Pingback: Tiffany And Co Outlet Philippines
Pingback: Tiffany Antique Rings
Pingback: Tiffany Bracelet Tarnish
Pingback: Tiffany Jewelry Virginia Beach
Pingback: Tiffany Tf2061 Australia
Pingback: Tiffany Armband Silber Preis
Pingback: roshe run leopard red
Pingback: Tiffany Bracelet On Ebay
Pingback: nimbus asics
Pingback: 橡胶止水带
Pingback: 压滤机
Pingback: 硝化棉
Pingback: 勾花网
Pingback: 铁路护栏网
Pingback: 不锈钢网
Pingback: 烧纸
Pingback: 金刚网
Pingback: Tiffany & Co Xo Ring
Pingback: Tiffany Ford
Pingback: nike roshe bewertung
Pingback: Discount Tiffany Wedding Bands
Pingback: Tiffany T Knockoff
Pingback: Breakfast At Tiffany Uk Quad
Pingback: nike roshe running sneaker
Pingback: Tiffany Jewellery Perth Wa
Pingback: Tiffany And Co Discontinued Rings
Pingback: toms shoes for sale
Pingback: buy cheap toms shoes
Pingback: cheap sunglasses
Pingback: cheap sunglasses
Pingback: 肉牛
Pingback: Tiffany Dunk High Australia
Pingback: Breakfast At Tiffany Uk Quad
Pingback: asics gel pulse
Pingback: Tiffany Set
Pingback: buy wow gold
Pingback: Tiffany And Co Great Gatsby Collection
Pingback: Tiffany E Co Infinity
Pingback: Tiffany Outlet Europa
Pingback: Tiffany Edmonton
Pingback: Tiffany 7 Co
Pingback: Tiffany Engagement Ring Guarantee
Pingback: Tiffany Wishbone Necklace
Pingback: Tiffany Signature X Ring
Pingback: Tiffany Necklace Blue Heart
Pingback: Tiffany Return
Pingback: Tiffany Official
Pingback: Tiffany Rings Atlas
Pingback: Tiffany And Co Flower Ring
Pingback: Tiffany & Co Canada Calgary
Pingback: Tiffany Alvord Quiz
Pingback: Tiffany T Ring
Pingback: Tiffany Palm Springs
Pingback: Tiffany Earrings Pinterest
Pingback: Tiffany Engagement Ring Finder App
Pingback: Tiffany And Co Ring Price In Malaysia
Pingback: Tiffany Armband Mit Herz
Pingback: Tiffany Jewellery Wedding Rings
Pingback: Tiffany Bracelet Las Vegas
Pingback: Bracciale Tiffany Titanio
Pingback: Tiffany Dunk High Australia
Pingback: Tiffany Earrings Under 200
Pingback: Tiffany Alvord Outfits
Pingback: Tiffany Jewellery Dundrum
Pingback: Tiffany Hotel Nyc
Pingback: Tiffany Sunglasses Crystals
Pingback: Tiffany Inspired Jewelry Cheap
Pingback: Tiffany Snsd
Pingback: Tiffany Jewellery Newcastle
Pingback: Bracciale Tiffany Dove Comprarlo
Pingback: Tiffany Alvord Wiki
Pingback: replica watches for sale
Pingback: Tiffany Jewellery Gumtree
Pingback: Tiffany Padlock Necklace
Pingback: Tiffany & Co Outlet Fake
Pingback: Tiffany Rose Us
Pingback: Tiffany Uk Lamps
Pingback: Tiffany And Co Ring Insurance
Pingback: Tiffany Rings Usa
Pingback: Tiffany And Company Dresses
Pingback: robe de mariée avec chapeau
Pingback: Tiffany Alvord Young Volcanoes Mp3
Pingback: Tiffany Kiss Necklace
Pingback: Tiffany Outlet
Pingback: cheap wow gold
Pingback: Jordan Uprising 2013
Pingback: My Hermes Glasgow
Pingback: Hermes Tie App
Pingback: Hermes Lindy Lime
Pingback: Oakley Zoning Map
Pingback: Hermes Herbag Instructions
Pingback: Nike Free Run Womens Zumba
Pingback: Tyler Oakley On The Talk
Pingback: Air Jordan 11 Gamma Blue Ebay
Pingback: Hermes God Age
Pingback: Oakley Frogskins Forum
Pingback: Cheap Nike Running Shoes Philippines
Pingback: Air Max 90 Infrared 2015
Pingback: Oakley Canada Twitter
Pingback: Oakley Frogskins Emerald Green
Pingback: Buy Nike Free Run 3 Tropical Twist
Pingback: Hermes Scarves At Saks
Pingback: Jordan 28 In 2K13
Pingback: Oakley Canada Twitter
Pingback: Hermes Scarves Real And Fake
Pingback: Nike Kd 6 Gs Camo
Pingback: toms Sko,toms Sko outlet,toms Sko for sale,toms Sko cheap,toms Sko online,toms Sko,toms Sko for sale,cheap toms Sko for sale,toms Sko for cheap,pre order toms Sko,pre order cheap toms Sko,toms Sko,cheap toms Sko,buy cheap toms Sko,authentic toms Sko,authe
Pingback: Air Jordan 5 June 6
Pingback: Nike Air Max 95 Black Anthracite
Pingback: Oakley Quarter Jacket Youth
Pingback: Nike Air Max 90 Hyperfuse Mint
Pingback: Jordan 28 Houston Release
Pingback: toms Sko,toms Sko outlet,toms Sko for sale,toms Sko cheap,toms Sko online,toms Sko,toms Sko for sale,cheap toms Sko for sale,toms Sko for cheap,pre order toms Sko,pre order cheap toms Sko,toms Sko,cheap toms Sko,buy cheap toms Sko,authentic toms Sko,authe
Pingback: Hermes Birkin Canada Price
Pingback: Nike Air Max Womens Zappos
Pingback: Air Jordan 1 Retro High Strap A Tribe Called Quest
Pingback: Oakley Silver Sunglasses
Pingback: Nike Free 4.0 V3 Shoes - Ho13
Pingback: toms Sko,toms Sko outlet,toms Sko for sale,toms Sko cheap,toms Sko online,toms Sko,toms Sko for sale,cheap toms Sko for sale,toms Sko for cheap,pre order toms Sko,pre order cheap toms Sko,toms Sko,cheap toms Sko,buy cheap toms Sko,authentic toms Sko,authe
Pingback: Oakley Glasses Frames Uk
Pingback: toms Sko,toms Sko outlet,toms Sko for sale,toms Sko cheap,toms Sko online,toms Sko,toms Sko for sale,cheap toms Sko for sale,toms Sko for cheap,pre order toms Sko,pre order cheap toms Sko,toms Sko,cheap toms Sko,buy cheap toms Sko,authentic toms Sko,authe
Pingback: cheap wow gold
Pingback: toms Sko,toms Sko outlet,toms Sko for sale,toms Sko cheap,toms Sko online,toms Sko,toms Sko for sale,cheap toms Sko for sale,toms Sko for cheap,pre order toms Sko,pre order cheap toms Sko,toms Sko,cheap toms Sko,buy cheap toms Sko,authentic toms Sko,authe
Pingback: toms Sko,toms Sko outlet,toms Sko for sale,toms Sko cheap,toms Sko online,toms Sko,toms Sko for sale,cheap toms Sko for sale,toms Sko for cheap,pre order toms Sko,pre order cheap toms Sko,toms Sko,cheap toms Sko,buy cheap toms Sko,authentic toms Sko,authe
Pingback: toms Sko,toms Sko outlet,toms Sko for sale,toms Sko cheap,toms Sko online,toms Sko,toms Sko for sale,cheap toms Sko for sale,toms Sko for cheap,pre order toms Sko,pre order cheap toms Sko,toms Sko,cheap toms Sko,buy cheap toms Sko,authentic toms Sko,authe
Pingback: fastest wow gold
Pingback: fastest wow gold
Pingback: 水泥板
Pingback: 荷兰网