An Ode to memcached

When your servers are hurting
And your database needs some relief
Something small, simple and efficient
Can make your response times brief

Sometimes, you don't need anything fancy
You just need something good
My friend is lightweight and speedy
And he does exactly what he should

Keys and values
Hashes and slabs
K.I.S.S.
And viola! it all just works

So go forth, apply caching
And let your servers rest.
Because any sysadmin will tell you,
memcached is the best.

cacti + C = graphing goodness

I recently setup cacti at work and at home. The rpmforge yum repository made it easy to install and a snap to setup. At work, with an assist from the other guys on the Systems team, there are 143 hosts - over 1500 data sources - being polled and the setup was pretty damn quick. However, it initially scaled like a pig - it brought a dual-core Opteron 242 with 2GB of RAM to it's knees. However, moving to using cactid and tweaking the proc/threads/php max memory helped a ton. What I especially liked is that when php crapped out due to the default memory restrictions, I didn't lose all the data for the hour or so it took me to debug the problem. It was "cached" in the MySQL database and then flushed to the RRD files once php was working again.

Additionally, I wrote an "apache_stats" plugin in C that polls apache's server-status page using libcurl and easyregex. I'd like to release the code to the public, but I'll have to discuss it with my new employers.

At home, I'm seeing some odd results when polling the net-snmp agent on my wireless routers. The old WRT54Gv2.2 is just fine, but my WRTSL54GS is very inconsistent when returning data... whole hours of the day are missing. I'll have to investigate more.

bad timing

Previously, I had my fileserver sync time with my primary wireless router and some pool.ntp.org servers. Then I noticed this:

Jul  7 08:41:10 remote  /usr/sbin/ntpd[521]: skew change 255.181 exceeds limit
Jul  7 08:56:48 remote  /usr/sbin/ntpd[521]: skew change -34.707 exceeds limit
Jul  7 09:24:28 remote  /usr/sbin/ntpd[521]: adjusting local clock by -0.159868s
Jul  7 09:24:28 remote  /usr/sbin/ntpd[521]: skew change -32.121 exceeds limit
Jul  7 09:39:06 remote  /usr/sbin/ntpd[521]: adjusting local clock by 0.465410s
Jul  7 09:39:06 remote  /usr/sbin/ntpd[521]: skew change 132.634 exceeds limit
Jul  7 09:44:23 remote  /usr/sbin/ntpd[521]: adjusting local clock by 0.727679s

According to the documentation, there's no real-time clock on most OpenWrt compatible hardware. I guess when you consider the minimalistic setup of these devices, you can't expect everything.

I've now switched it around. My routers now sync with my fileserver and us.pool.ntp.org. My fileserver ignores the access points and syncs with:

server timex.cs.columbia.edu
server ntp-2.cso.uiuc.edu
server ntppub.tamu.edu
server ntp-1.vt.edu
server ntp3.cs.wisc.edu
server 0.pool.ntp.org
server 1.north-america.pool.ntp.org
server 2.us.pool.ntp.org

Those are the same NTP servers I use at work, with 3 pool.ntp.org servers thrown in. It just annoys me that OpenNTPD doesn't have the equivalent of "ntpq -p" to check the status of the NTP sync.

giving xubuntu a try

I'm nuking my crappy WindowsXP install on ye olde notebook - a Toshiba Satellite 4090XDVD. Since I fixed the VPN at work with OpenVPN, I don't need it to connect to the stupid Firebox PPTP (pronounced "PoPToP") VPN anymore. So I'm giving Xubuntu a try because it's supposed to be a lighter desktop.

One gotcha that I encountered on the install is that it didn't initially like my Xircom 10/100+56k modem network card. But I found a tip buried deep in a message board - eject the card, re-insert in and then tell the installer to redetect the network, and it works fine.

Another gotcha was the software dependencies. I am not exactly sure what went wrong, but it failed to install all the dependencies correctly on the first pass. I just told it to re-install again and it was fine. (shrug)

Ye olde datacenter 2 - this time it's enterprise-class gear

Ye olde datacenter

view of my old office datacenter, from the entrance. trying out go.blogger.com. Looks like the photo worked, but the comment didn't. Oh well.

rtg cgis

I posted my RTG CGIs to the rtg mailing list today. It's more of a work-thing as opposed to a home-project-thing, but since they're released under the GPL and it's a giving-back-to-the-community-thing, it figured it was worth mentioning. I'm still a little annoyed that the RTG database desperately needs normalization, but I understand the performance considerations and realize it's a design decision.