I was looking for a quick and painless way to determine what IP addresses used our web app the most and stumbled across this excellent post. I got reminded of the power of the command-line interface and that often, I underestimate the way these commands are supposed to make a sysad's life easier.

So here's a quick tip for counting unique visitors to your web app. Locate your web app's log file and do a quick scan of the last few lines. In my case, this is what I saw:

112.198.79.223 - - [10/Jul/2010:00:57:29 -0400] "POST /v2/api/execute.php?method=upload_time&ver=win1.1.15 HTTP/1.1" 200 116 "-" "Mozilla/5.0"
117.241.112.254 - - [10/Jul/2010:00:57:30 -0400] "POST /v2/api/execute.php?method=timestat&ver=win1.1.15 HTTP/1.1" 200 288 "-" "Mozilla/5.0"
80.249.84.105 - - [10/Jul/2010:00:57:29 -0400] "POST /v2/api/execute.php?method=offtime&ver=win1.1.15 HTTP/1.1" 200 19 "-" "Mozilla/5.0"
117.241.112.254 - - [10/Jul/2010:00:57:32 -0400] "POST /v2/api/execute.php?method=get_defaults&ver=win1.1.15 HTTP/1.1" 200 77 "-" "Mozilla/5.0"

In this log format, the IP address appears first, so we will only need to get the first set of characters, separated by a space. In order to count instances of an IP address, we sort the IP addresses so similar IP addresses are grouped together, then count them. Then we sort them again so that the sums are arranged. The number of unique IPs may give you pages of standard output, so optionally, if you only want to see which IP addresses accessed the web app the most, we look at the top results only.

So these are the commands used, piped one after the other, with the output:

[root@server2 ~]# cat /var/log/httpd/access_ssl.log | awk '{print $1}' | sort | uniq -c | sort -nr | head

71612 216.157.78.237
34094 120.28.205.244
30091 119.93.97.54
25078 120.28.214.104
18949 120.28.195.151
17929 58.107.64.40
14877 212.98.174.235
14752 117.241.112.143
14429 120.28.247.74
12864 117.241.113.183
[root@server2 ~]#

Happy counting, err reporting!

Permalink | Leave a comment  »

we do a lot of performance tuning on mysql, and as a result, we almost always manage to get plesk mangled. not its fault, it just can't run without mysql, doh.

not of preference, but plesk happens to be there to make handling domains easier, as a single editing point when you want to change something and not worry about missing something out.

so there's a custom /etc/my.cnf that we use, and with changes to limits done (both on mysql and via ulimit), you'd think that a reload of mysqld would do. (we use InnoDB btw)

normally, that's all you need to have mysqld read off the new settings. if you've grown so dependent on plesk (like i have), you would not notice that mysqld didn't actually start, and you will get an error when you load plesk, thinking that it's a plesk error:


ERROR: PleskMainDBException
MySQL query failed: Incorrect information in file: './psa/misc.frm'

0: /usr/local/psa/admin/plib/common_func.php3:218
db_query(string 'select param, val from misc')
1: /usr/local/psa/admin/plib/common_func.php3:497
get_param(string 'mysql41_compatible')
2: /usr/local/psa/admin/plib/common_func.php3:394
db_set_names()
3: /usr/local/psa/admin/plib/common_func.php3:373
db_connect_real(string 'localhost', string 'admin', string '********', string 'psa')
4: /usr/local/psa/admin/plib/common_func.php3:353
db_connect()
5: /usr/local/psa/admin/auto_prepend/auth.php3:82


even a restart of the plesk service will not work:


[root@jedi]# /etc/init.d/psa restart
Unable to query parameter by query select val from misc where param='mysql41_compatible': Incorrect information in file: './psa/misc.frm'
Unable to query parameter by query select val from misc where param='mysql41_compatible': Incorrect information in file: './psa/misc.frm'
Mysql error: Incorrect information in file: './psa/key_history.frm'
Processing config directory: /usr/local/psa/admin/conf/httpsd.*.include
Syntax OK
/usr/local/psa/admin/bin/httpsdctl restart: httpd restarted
ERROR 1033 (HY000) at line 1: Incorrect information in file: './psa/sessions.frm'


so this is a mysqld failing to start properly problem. to fix this, go to your mysql datadir and see if the sizes of your innodb log files match those in your .cnf. we usually set innodb_log_file_size = 256M and innodb_log_buffer_size = 256M in /etc/my.cnf.

if they don't match, stop mysqld, remove the ib_logfile0 and ib_logfile1 files, and start mysqld. be careful not to delete the ibdata1 or ibdata2 files or you lose your data. avoid using rm -r ib* (yea, wildcards and fast fingers give sysads their most stupefying moments)

innodb will recreate the log files to match your .cnf and mysqld now becomes available for plesk.

Permalink | Leave a comment  »

So the hosting company eventually understood what was happening and started asking the right questions. The bug is that the system allowed network administrators to assign to new clients, the IP addresses that have already been assigned to a current client.

We spent crazy hours figuring out what happened. Since the domains were not loading our sites, and instead were redirecting to some domains we don’t own (real bad sites btw), we checked if .htaccess was modified and tested if it was working at all. I also did double checks on virtual host configurations and they all looked clean.

I started thinking that it was a routing issue. We were not using the publicly routable IPs for the virtual hosts; instead, we used internal IPs as mapped to the firewall hardware. A lookup on the domain/s that we were being redirected to reported different IP addresses, which didn’t readily confirm my suspicions, but stressed the possibility of IP-hijacking.

It’s not supposed to be taken in the context of BGP (border gateway protocol), as this is beyond me. Instead, think of your assumption over IP addresses that we’re using, to serve your websites. Sounds clear? I hope it does. I got pretty lost myself and it isn’t after deep digging at the hosting company’s end that their errors became apparent.

So in the light of this realization, I’ve already moved the domains to some other hosting. Way too much work and way uncool.

Permalink | Leave a comment  »

I was not certain of the performance implications to this. I heard from a fellow Linux junkie here that there may be a slight penalties in terms of access speed, since /home was meant for generic read/write, whereas /var was accessed in a very quick manner. Gotta verify this. But for now, I dwell in the power of symbolic links.

Permalink | Leave a comment  »