I was just doing some routine maintenance on our servers when I had to use some command-line judo so solve a small issue and thought I'd share it.  Not for the particular solution, but for the path of thought that led to the solution; that is, demonstrating the power of the unix/linux command shell.

On a server running CentOS 5.6 (a linux distribution), I ran the command to perform routine maintenance and got an error upgrading a package named "shadow-utils-4.0.17-18.el5_6.1.x86_64".  To make a long story short, I realized it was failing because a file it wanted to update was set to be 'immutable'...  that is, a security tool I ran modified a bunch of files that are routinely hijacked by viruses and made them unmodifiable.  This makes the machine a little more secure, but this also means the updater can't modify them either.

I found the unmodifiable file and using the 'chattr' command, made the file modifiable (with the intent to set it back to be unmodifiable later).  I ran the updater again, and it failed on yet another file.  I did the same to that file, tried to update, and ran into a third.

I stopped and thought to myself "there could be dozens of files I'll have to manually set and restore for this process... and it'll be prone to error... there has to be a better way".

I realized all three files I had to modify so far were in the /usr/sbin directory.  This makes sense - all the shadow utilities I was updating were tools for the system administrator to run.  So I made the assumption that all the files that were going to need to be set and reset were in this directory.

So I reset the files I had manually modified so far, and dumped the contents of the extended permissions to a file:

lsattr /usr/sbin > sbin-results.txt

That uses a feature of the shell called 'redirection'...  the '>' symbol says "take whatever this would normally dump to the screen, and dump it into a text file with this name instead".  So now I had a record of the settings I was going to have to restore after my update.

Next, I ran this command to reset all the permissions to a modifiable state:

sudo chattr -i /usr/sbin/*

'chattr' is the tool to 'change attributes', and '-i' means 'remove the immutable setting'

I was then able to run 'yum update' and have my update succeed.

At this point I had some work to do.  I could have manually gone through that text file, found every immutable file, and manually applied the immutable setting back...  but that might take a long time *and* be error-prone.  Is there a way I can have the computer do that for me?

First, lets use 'grep' to get a list of all the files that were originally set to be immutable:

grep '\-i\-' sbin-results.txt

and we get the output:

----i-------- /usr/sbin/irdadump
----i-------- /usr/sbin/pwunconv
----i-------- /usr/sbin/grpconv
----i-------- /usr/sbin/clockdiff
----i-------- /usr/sbin/useradd
----i-------- /usr/sbin/sys-unconfig
----i-------- /usr/sbin/tracepath6
----i-------- /usr/sbin/irdaping
----i-------- /usr/sbin/adduser
----i-------- /usr/sbin/grpck
----i-------- /usr/sbin/pwck
----i-------- /usr/sbin/usermod
----i-------- /usr/sbin/findchip
----i-------- /usr/sbin/irnetd
----i-------- /usr/sbin/groupdel
----i-------- /usr/sbin/irattach
----i-------- /usr/sbin/ping6
----i-------- /usr/sbin/chpasswd
----i-------- /usr/sbin/arping
----i-------- /usr/sbin/usernetctl
----i-------- /usr/sbin/pwconv
----i-------- /usr/sbin/dongle_attach
----i-------- /usr/sbin/tracepath
----i-------- /usr/sbin/groupmod
----i-------- /usr/sbin/grpunconv
----i-------- /usr/sbin/newusers
----i-------- /usr/sbin/userdel
----i-------- /usr/sbin/groupadd

looks good... thats not a huge list as I feared, but that is still going to be a lot of copy and pasting, and error prone to do by hand.  In order to get closer to automating this, I need to extract just the filenames from that list.  Here a tool called 'awk' is invaluable:

grep '\-i\-' sbin-results.txt | awk '{ print $2 }'

gives me the output:

/usr/sbin/irdadump
/usr/sbin/pwunconv
/usr/sbin/grpconv
/usr/sbin/clockdiff
/usr/sbin/useradd
/usr/sbin/sys-unconfig
/usr/sbin/tracepath6
/usr/sbin/irdaping
/usr/sbin/adduser
/usr/sbin/grpck
/usr/sbin/pwck
/usr/sbin/usermod
/usr/sbin/findchip
/usr/sbin/irnetd
/usr/sbin/groupdel
/usr/sbin/irattach
/usr/sbin/ping6
/usr/sbin/chpasswd
/usr/sbin/arping
/usr/sbin/usernetctl
/usr/sbin/pwconv
/usr/sbin/dongle_attach
/usr/sbin/tracepath
/usr/sbin/groupmod
/usr/sbin/grpunconv
/usr/sbin/newusers
/usr/sbin/userdel
/usr/sbin/groupadd


looking good...  notice the "|" symbol in the command - this is called the 'pipe' symbol, and with it, I can take the output of one command and feed it to the input of another.  the command above now means "take the file with the previous directory permissions in it, find me every line that contains a file set to be immutable, and print out just the filename".

We are getting very close now...  for every one of those files, we now need to run the command:

sudo chattr +i <filename>

so we can restore the immutable setting.  We can use the command "xargs" to do that, again, with the pipe command:

grep '\-i\-' sbin-results.txt | awk '{ print $2 }' | xargs -l sudo chattr +i


and voila!  All directory settings are restored.  One line, no manual mistakes, and it took less than a second to run.  that saved me enough time to write this blog entry.


This is such a useful mechanism I'm now going to turn it into some command line utilities.  Imagine running into this situation again and being able to say:

sudo unlock-immutable /usr/sbin
yum update
sudo relock-immutable /usr/sbin

and having that just work.  By passing in the directory, I could choose to reuse this on directories like 'usr/bin' should that be the cause of my pain next time.  Maybe thats my next blog entry, or maybe I'll just leave that as an exercise for the reader.

I'm hardly the first person to use grep, awk, and xargs in a command line - I do things like this a lot, and so do many sysadmins...  but I see far too many developers who don't think this way - they look for a button in their IDE, and when there isn't one, they spend far too much time and effort on manual, error-prone attempts.  Spend the time to learn a half dozen command line tools that can be chained this way... make the command line your IDE!

Dear Restaurant owner,

I want to eat at your restaurant, but I need to find you first. You need a website. And not just any website. It has to work on a mobile device. If not, you might as well not exist anymore.

In the past few months, I have been out with my friends several times, spontaneously decided to get a lunch, dinner, etc. and had a bad experience with a restaurant's website.

Today's typical dining experience starts with a bit of research on a phone, looking for a menu. I start with the maps app on my iPhone, search for something like "Indian" or "Italian" around my current location, find a restaurant, go to the website, and I'd greeted with a message like "you must install Flash version 9 or higher" if I'm lucky enough to find your website at all.

You know what the cool kids call that? EPIC FAIL.

Even if I can see the website, I'm more likely to find a picture of the owner's family as if they are slaving over a hot stove, the history of the restaurant starting with the owner's mother immigrating in 1977, or a stock photo of a slice of pizza. If there is a menu at all, its an out of date pdf file I'm struggling to view on my tiny screen.

If you care about your restaurant, you should want your customers to find you.

What information should be at their fingertips?

When visiting a restaurants website from my phone, I want:

1. location (including phone number)
2. hours
3. menu

Anything else is just noise on a mobile phone... If I'm planning a catered event, I'll do that from my computer.

But I bought a package deal...

Yes, I'm aware that ground-up website design can be expensive, and restaurants often fall for the "Website for $29.95 a month" marketing ploy, where you get a "billboard-at-a-url" and a scanned menu. You are losing out. You need a real website.

Here's why:

Customers can find what they are looking for

You wouldn't imagine not having a sign outside your restaurant - a website is the same thing to the person actively looking for you. Restaurants think nothing of printing thousands of folded menus and sticking them on every car in a parking lot or every door in a heighborhood. Your website can have a far greater reach than that, and with immediate impact.

you can have a CMS to keep the menu up to date

For most restaurants, the menu on the website is like a time capsule. Dishes come and go... prices change... why shouldn't your website reflect that? With a content management system behind your menu, you can log in to update prices, add specials, etc. Why shouldn't your website be able to tell people what the soup of the day is?

You can tailor your SEO strategy

Try this: Put the name of your city and the type of food your restaurant serves into Google. (If you are in a big city, try your zip code). Is your restaurant's website on the first page? It should be. If you put in the name of your restaurant, are you the first result? If you're not, the first result is probably something like a restaurant review - and you have no control over that! Your worst customer could be the first thing people learn about your restaurant.

With your own website, you can easily influence how search engines view you. That is a topic larger than this blog entry though - its called 'Search Engine Optimization' or 'Search Engine Marketing'.

Further, by having your website as a real website (as opposed to flash or pdf), all your dishes become keywords Google can index. Maybe I'll find you because I searched on "Vindaloo" or "Pad Thai". All those words are pure SEO gold.

You can target your market more effectively

Visitors to your website can be tracked more closely than cattle with tags in their ears. Are most of your visitors coming from a particular neighborhood? Maybe thats the place to target with your door hangers. Is there a neighborhood thats not visiting? Send them a coupon for a free cheesy bread.

Ok, so how do I do it?

I'm not trying to sell you anything here - in fact, our typical projects are much larger than restaurant websites. I just want a better dining experience, so I'm offering you some technical advice:

Use 'Open Technology' standards

Make sure your website firm designs with html/css/javascript. Anything that requires a browser plugin to use (flash, Java, and most likely pdf), are non-starters for your marketing purposes.

Make sure your website looks good on small devices

I'll spare you most of the technical details, but with the kinds of tool we have today, a good website design scales down to work on phones, and scales up to work on big screens. On big screens, sure - show me your family in the kitchen. On my cell phone, echo the design with logos, fonts, and colors, but just give me the menu, hours, and location.

Search Engines are your best marketing tool

You don't need to hire an expensive SEO consultant - in fact, I think that most SEO consultants are the equivalent of Internet Snake Oil. Instead, just make sure your design firm eats and breathes the advice in Google's SEO starter guide. From your business' perspective, you should read a book like Search Engine Optimization (SEO): An Hour a Day or Internet Marketing: An Hour a Day to figure out how you can best market your restaurant.

Thats it! I hope you find this advice useful. Remember... I want to eat your food, I just need to be able to find you when you're just down the street.

Fullcalendar is an impressive calendar widget for JQuery, showing you events on a month, week, or day view, and letting you drag them around to reschedule them.

It needed a good sample app showing how to use it in Rails 3, so I wrote one.

Rails3_fullcalendar is a rails application that tries to live up to the rails3 notion of restful ideals.  It does this thanks to the power of Adam Lassek's jquery.rest plugin, which handles all of the wrapping of the jquery.ajax method into create, read, update, and delete methods, and also deals with the complexity of the token's rails uses to avoid cross-site scripting attacks.

Download it from github, run the migrations, and fire up the app.  You'll see an empty calendar, and scaffolding for adding events.  When looking at the calendar view, you can drag events around to change their date and time.  Dig into the code to see how it works.

I'm going to need to prepare a longer rant regarding timezones, but fullcalendar does something nice - it respects timezones (as long as you send it the right time format in the json), and normalizes everything to the local timezone on display...  Now if I schedule something for 12:00 noon in Washington DC during Eartern Daylight Time, it'll end up on the server as 16:00:00 UTC on the server, and if viewed by someone in California will show up on their calendar at 9:00am.

I have some future plans for this demo - including having multiple schedules, and then showing how to overlap them on the same calendar, with the events from each schedule showing up in their own color.  With a view like this, someone can have a calendar 'command center' view, dragging around events from different people's schedules until conflicts are resolved.

Last night we performed some upgrades and restarted a server with nearly a year of uptime.  This morning I was checking on the status of the server, and saw this graph.  It is a beautiful example of the kinds of things you can see with Munin.

You are looking at a graph from munin, showing the usage of memcache of a large Ruby on Rails application.

After the server restarted, the cache was empty.  As the server traffic picked up this morning, more and more database calls and page fragments were generated by users hitting the site.  They were tucked away in memcache so they could be served up faster to the next visitor.  As we get more traffic, this cache fills up, and the website actually gets faster.

The performance gains aren't the subject of this post though - I thought this was a beautiful representation of a day in the life of a server process.  This represents one day period, with time going from left to right, and the three lines representing the amount of data in bytes (green), the number of objects (whether that be objects from the database or pre-rendered pieces of html - red), and the number of connections our rails processes are making to the memcached server (blue).

What does this show me?  I can see that memcache is doing its job - that the rails app is putting data in and getting it out.  I can also see that the cache is filling up... based on the 'hockey stick' curve, I can see we have allocated enough memory to hold the data we are putting in there.  If we hadn't allocated enough memory, I would visually be able to see it 'hit the ceiling'.

I can also tell things about the system overall by comparing to other graphs from munin and other tools.  Comparing this to graphs of mysql queries, I can see the impact of using memcached on mysql (by the number of reduced database queries) By comparing it to processor utilization, I can see the amount that pre-rendering reduces processor load (since we don't spend as much time rendering).

It is a lot of hard work putting on a conference like RubyNation; Three CodeSherpas are on the organization committee, out of a total of 7.  I'd like to thank Gray Herter for being a driving force behind this conference... it's his baby, and without him, there would be a gaping hole in the Ruby Community in the Northern Virginia/DC Metro area.  I'd also like to thank all of our sponsors.  Without them, this conference would still just be a dream.

Every year the conference gets better and better.  With a live event, there are always issues that have to be solved in real-time (like the programs being delivered late), and each year we hold a retrospective after the conference to figure out what we need to improve.  This year it seems like it was only minor things - nothing that couldn't be handled on the spot with our great staff of volunteers.

If you attended, I'd lke to remind you to please rate the talks you attended.  Nothing helps us improve the conference like real feedback.  If you didn't attend, you missed a great event; don't miss it next year!  Make sure to visit the RubyNation website early and often.  We'll have a signup for next year's announcements available soon.