The gval function should be edited to reflect your own region or university. It has been written and tested on Bash 3.2.48.

#!/bin/sh
#
# Bash GPA calculator
#
# Isam | r0cketjump@yahoo.com | biodegradablegeek.com
# 05/21/2009 - Just another 4 AM project

function usage {
  echo -e "\nBASH GPA Calculator"
  echo
  echo -e "\tAccepts an even # of arguments in the form of C G C G C G ..."
  echo -e "\t (C = number of credits, G = grade for the course)"
  echo
  echo -e "\tExample: You got a B+ in a 4 credit course, "
  echo -e "\t         an A in a 3 credit course, etc.."
  echo
  echo -e "\tUSAGE: $0 4 B+ 3 A 3 F 3 B-"
  echo
  echo "Acceptable grades are A B C D F WU (eq to F)"
  echo
}

function calc {
  echo `echo "scale=3; $1" | bc`
}

function gval {
  grade=`echo "$1" | tr [a-z] [A-Z]`
  case $grade in
    A+ ) echo '4.3';;
    A ) echo '4';;
    A- ) echo '3.7';;

    B+ ) echo '3.3';;
    B ) echo '3.00';;
    B- ) echo '2.7';;

    C+ ) echo '2.3';;
    C ) echo '2.0';;
    C- ) echo '1.7';;

    D+ ) echo '1.3';;
    D ) echo '1.0';;
    D- ) echo '0.7';;

    F ) echo '0';;
    WF ) echo '0';;
    WU ) echo '0';;
  esac
}

# check # of arguments. is it even?
let MOD=$#%2
if [ ! $MOD -eq 0 ]; then
  usage
  exit
elif [ $# -eq 0 ]; then
  usage
  exit
fi

args=($@)
n=${#args[@]}

points=0
credits=0

for ((i=0;i<$n-1;i+=2)); do
  k=${i}

  creds=${args[$k]}
  cgrade=${args[$k+1]}

  # convert cgrade (C-) to a number
  grade=`gval $cgrade`
  pts=`calc $grade*$creds`

  echo "$creds * $cgrade ($grade) = $pts"

  points=`calc $points+$pts`
  credits=`calc $credits+$creds`
done

gpa=`calc $points/$credits`
echo "------------"
echo "Total points  = $points"
echo "Total credits = $credits"
echo "------------"
echo "** GPA (pts/crd) = $gpa"
echo "------------"

(Script uses bc as the calculator. Change that in the calc function if you need to.)

I’ll never get used to Bash’s ugly ass syntax. … esac?

“AOL System Msg: Your screen name (mrEman) is now signed into AOL(R) Instant Messenger (TM) in 2 locations. Click here for more information.”

To use, paste code in file, save file as blockaolsystemmsg.pl in ~/.purple/plugins/ and then open (or re-open) Pidgin and go to Tools -> Plugins (or press CTRL+U), and enable “Block AOLSystemMsg.” That should be it!

If you’re having any trouble, try going to Help -> Debug to open up Pidgin’s debug console.

#!/usr/bin/perl
# BlockAOLSystemMsg plugin tested on Pidgin 2.5.5. Put in ~/.purple/plugins/ and enable
use Purple;
our $target = 'AOL System Msg'; # case-insensitive
our $plugin_name = 'Block AOLSystemMsg'; 

%PLUGIN_INFO = (
  perl_api_version => 2,
  name => $plugin_name,
  version => "0.1",
  summary => "Blocks the screen name 'AOL System Msg'",
  description => "Ignore annoying 'your SN has signed on at 2 locations' AIM message",
  author => "Isam ",
  url => "http://biodegradablegeek.com",
  load => "plugin_load",
  unload => "plugin_unload"
);

sub loginfo { Purple::Debug::info($plugin_name, " @_\n"); }
sub minimize {
  my $r = lc($_[0]);
  $r =~ s/ //g;
  return $r;
}

sub plugin_init { return %PLUGIN_INFO; }

sub plugin_load {
  my $plugin = shift;
  $target = minimize($target);
  loginfo("Sight set on '$target'");
  Purple::Signal::connect(Purple::Conversations::get_handle(),
                          'receiving-im-msg', $plugin, \&callback, '');
}

sub plugin_unload {
  my $plugin = shift;
  loginfo('Block AOLSystemMsg Unloaded.');
}

sub callback {
  my ($acc, $sender, $msg, $flags) = @_;
  if (minimize($sender) eq $target) {
    loginfo("(BLOCKED) <$sender> $msg");
    return 1
  };
}

update: Fixed the botched code. Thanks.

Jekyll tackles all the nuisances involved in creating static pages (I used to add just enough PHP to make a layout). It works by running your prototype through some parsers and outputs plain static HTML/XML (RSS feeds) etc. It’s perfect for lightweight sites that would be impractical on WordPress, like a few static pages of information, landing pages, portfolio/resume pages, and parked domains.

Git takes care of keeping your development (local) and production (remote) environments synced. Git might be a little confusing if you’re learning it with the mindset that it works like Subversion.

I’ll update this post when the guide is done. For now, the following will assume you’re familiar with Jekyll (or at least have an empty file in the prototype directory) and git. This Bash script simplifies creating the remote git repository:

** please read through the code and make sure you know what this does, and what you’re doing. As of now, this is bias towards my own Apache/vhost setup. It’s trivial to edit for your specific needs. You’re using this at your own risk.

(direct link – repogen.sh)

#!/bin/sh
#
# 04/01/2009 | http://biodegradablegeek.com | GPL
#
# You should be in site (NOT public) root (be in same dir as public/ log/ etc)
# proto/ is created and will house the jekyll prototype
# public/ will be the generated static site
# the public/ folder will be REMOVED and regenerated on every push
# 

if [ -z "$1" ]; then
  echo "Usage: ./repogen.sh domain.comn"
  exit
fi

# optional. will make it easier to copy/paste cmd to clone repo
SSHURL="ssh.domain.com"
URL="$1"

echo "** creating tmp repo"
mkdir proto
cd proto
git init
touch INITIAL
git add INITIAL
git commit -a -m "Initial Commit"

echo "** creating bare repo"
cd ..
git clone --bare proto proto.git
mv proto proto.old
git clone proto.git
rm -rf proto.old

echo "** generating hook"
HOOK=proto.git/hooks/post-update

mv $HOOK /tmp
echo '#!/bin/sh' >> $HOOK
echo '# To enable this hook, make this file executable by "chmod +x post-update".' >> $HOOK
echo '#exec git-update-server-info' >> $HOOK
echo '' >> $HOOK
echo '' >> $HOOK
echo 'URL='"$URL" >> $HOOK
echo 'PROTO="/home/$USER/www/$URL/proto"' >> $HOOK
echo 'PUBLIC="/home/$USER/www/$URL/public"' >> $HOOK
echo  '' >> $HOOK
echo 'export GIT_DIR="$PROTO/.git"' >> $HOOK
echo 'pushd $PROTO > /dev/null' >> $HOOK
echo 'git pull' >> $HOOK
echo 'popd > /dev/null' >> $HOOK
echo '' >> $HOOK
echo "echo -----------------------------" >> $HOOK
echo "echo '** Pushing changes to '$URL" >> $HOOK
echo "echo '** Moving current public to /tmp'" >> $HOOK
echo 'mv "$PUBLIC" "/tmp/'$URL'public-`date '+%m%d%Y'`"' >> $HOOK
echo 'echo "** Generating new public"' >> $HOOK
echo 'jekyll "$PROTO" "$PUBLIC"' >> $HOOK

echo "** enabling hook"
chmod a+x $HOOK 

echo "** clone repo on local machina. example:"
echo "git clone ssh://$USER@$SSHURL/~$USER/www/$SSHURL/proto.git"

Usage

Your site structure might be different. repogen.sh is made by pasting the above code in a new file, and then chmod a+x to make it executable. This should be done on the remote server.

cd www/domain.com/

ls
public/ private/ log/ cgi-bin/

./repogen.sh domain.com

Now on your local machine, clone the new repo, move your files in, and push:

git clone ssh://[username]@ssh.domain.com/~[username]/www/domain.com/proto.git
cd proto/
cat "hello, world" > index.htm
git add index.htm
git commit -a -m 'first local commit'
git push

After you push your changes, the post-update hook will delete the public/ directory (the root of the site). This dir and its contents are automatically generated and will get wiped out on EVERY push. Keep this in mind. All your changes and content should reside in proto/.

The proto/ repo will pull in the new changes, and then Jekyll will be invoked to generate the updated site in public/ from the prototype.

Should you need to edit it, the post-update hook is in the bare git repo (proto.git/hooks/)

Thanks to the authors in the posts below for sharing ideas. I first read this git method on dmiessler’s site.

Resources:
dmiessler.com – using git to maintain static pages
toroid.org – using git to manage a web site
Jekyll @ GitHub
git info
more git info

On a technical note, it’s using mechanize, hpricot, tempfile (for the cache). A lot of this is just copy & paste programming from the earlier anime scraper.

To grab the gems (rdoc takes 10x as long as the gem to fetch and install):

sudo gem install mechanize --no-rdoc
sudo gem install hpricot --no-rdoc

#!/usr/bin/env ruby
# biodegradablegeek.com
# public domain
# 

require 'rubygems'
require 'hpricot'
require 'tempfile'
require 'mechanize'
#require 'highline/import'
#HighLine.track_eof = false

$mech = WWW::Mechanize.new
$mech.user_agent_alias = 'Mac Safari'
$master = []

def puts2(txt=''); puts "*** #{txt}"; end

class Cache
  def initialize
    # Setup physical cache location
    @path = 'cache'
    Dir.mkdir @path unless File.exists? @path

    # key/val = url/filename (of fetched data)
    @datafile = "#{@path}/cache.data"
    @cache = load @datafile
  end

  def put key, val
    tf = Tempfile.new('googletrends', @path)
    path = tf.path
    tf.close! # important!

    puts2 "Saving to cache (#{path})"
    open(path, 'w') { |f|
      f.write(val)
      @cache[key] = path
    }

    save @datafile
  end

  def get key
    return nil unless exists?(key) && File.exists?(@cache[key])
    open(@cache[key], 'r') { |f| f.read }
  end

  def files
    @cache.values
  end

  def first
    @cache.first
  end

  def exists? key
    @cache.has_key? key
  end

private
  # Load saved cache
  def load file
    return File.exists?(file) ? YAML.load(open(file).read) : {}
  end

  # Save cache
  def save path
    open(path, 'w') { |f|
      f.write @cache.to_yaml
    }
  end
end

$cache = Cache.new

def fetch(url)
  body = $mech.get(url).body()
  $cache.put(url, body)
  body
end

def getPage(url)
  body = $cache.get(url) 

  if body.nil?
    puts "Not cached. Fetching from site..."
    body = fetch url
  end
  body
end

def loadState
  mf = 'cache/master.data'
  $master = File.exists?(mf) ? YAML.load(open(mf).read) : {}
  $master = {} if $master==false
end

def saveState
  open('cache/master.data', 'w+') { |f|
    f.write $master.to_yaml
  }
end

def main
  #loadState

  # Grab top 100 Google Trends (today)
  #date = Time.now.strftime '%Y-%m-%d'
  date = '2009-01-21'

  puts2 "Getting Google's top 100 search trends for #{date}"
  url = "http://www.google.com/trends/hottrends?sa=X&date=#{date}"
  puts2 url

  begin
    body = getPage(url)
  rescue WWW::Mechanize::ResponseCodeError
    puts2 "Couldn't fetch URL. Invalid date..?"
    exit 5
  end

  puts2 "Fetched page (#{body.size} bytes)"

  if body['There is no data on date']
    puts2 'No data available for this date.'
    puts2 'Date might be too old or too early for report, or just invalid'
    exit 3
  end

  doc = Hpricot(body)

  (doc/"td[@class='hotColumn']/table[@class='Z2_list']//tr").each do |tr|
    td = (tr/:td)
    num = td[0].inner_text.sub('.','').strip
    kw = td[1].inner_text
    url = (td[1]/:a).first[:href]
    Keyword.find_or_new(kw) << Occurance.new(num, date, url)
  end
  puts "Got info on #{$master.size} keywords for #{date}"
  puts "keyword '#{$master.first.name}' occured #{$master.first.occurances} times"
end

class Occurance
  attr_accessor :pos, :date, :url
  def initialize(pos, date, url)
    @pos = pos
    @date = date
    @url = url
  end
end

class Keyword
  attr_accessor :name, :occurances
  def initialize(name)
    @name = name
    @occurances = []
    @position_average = nil
    @count = nil
    $master << self
  end

  def self.find_or_new(name)
    x = $master.find { |m| name==m.name }
    x || Keyword.new(name)
  end

  def << occurance
    @occurances << occurance
  end

  def occured_on? datetime
    raise 'implement me'
  end

  def occured_between? datetime
    raise 'implement me'
  end

  def occurances datetime=nil
    raise 'implement me' if datetime
    @occurances.size
  end

  def occurances_between datetime
    raise 'implement me'
  end

  def pos_latest
    @occurances.last.date
  end

  def pos_average
    @position_average
  end

  def pos_average_between datetime
    raise 'implement me'
  end
end

#   Instance= [num, date, url]
#   Keyword=[Instance, Intance, Instance]
#   Methods for keywords:
#   KW.occured_on? date
#   KW.occured_between? d1, d2
#   KW.occurances
#   KW.occurances_between? d1, d2
#   KW.pos_latest
#   KW.pos_average
#   KW.pos_average_between

#   KW has been on the top 100 list KW.occurances.size times
#   The #1 keywords for the month of January: Master.sort_by KW.occurances_between? Jan1,Jan31.pos_average_between Jan1,Jan31
#
#   Top keywords: sort by KW.occurances.size = N keyword was listed the most.
#   Top keywords for date D: Master.sort_by KW.occured_on (x).num

main

This scraper needs some initial setup and won’t work out of the box, but I’m including it here in the hopes that it will serve as a decent example of a small real world scraper, if you’re looking to learn the basics of scraping with Hpricot and Mechanize. Let me know if you find any use for it. I will update the posted code later this week when I have time to complete it and add some more features.

There’s one major problem with the organization of episodes on AnimeCrazy, and it’s the fact that some episodes are glued together into one post. Right now the scraper stops and asks you how to proceed when it comes across such a post. You basically need to tell the scraper if a post (page) contains 1 episode (video) or multiple. If there’s 1, it proceeds on its own, but if there’s two, it requires that you give it the names and links of each individual episode (part1 and part2 usually). Sometimes 2 episodes are together in 1 video. Sorta like those music albums on KaZaA or LimeWire that are basically ripped as one huge mp3 instead of individual songs.

This only accounts for maybe 30-40 out of 6000 videos, and it’s not that big of a deal because the amount of work needed to proceed with the scraping is small, but it IS work, and is a bitch slap to the entire concept of automation, but coding around the issue is a major hassle and there would still be a high chance that some inconsistencies will still come through. It would be far less work to just find another anime site which is far more consistent, though the reason animecrazy is good is because it’s active, and the site IS updated manually these days, as far as I can tell.

BTW, Why The Lucky Stiff rocks, and Hpricot is amazing. But the serious scrapologist should consider scrAPI or sCRUBYt (uses Hpricot) for big projects.

#!/usr/bin/env ruby
# License: Public domain. Go sell it to newbs on DigitalPoint.

require 'rubygems'
require 'hpricot'
require 'mechanize'
require 'tempfile'
require 'highline/import'
HighLine.track_eof = false

$mech = WWW::Mechanize.new
$mech.user_agent_alias = 'Mac Safari'

###############################
$skip_until = false
DEBUG=false
###############################

def debug?
  DEBUG
end

def puts2(txt='')
  puts "*** #{txt}"
end

#  Anime has: title, type (series, movie), series
#  Episode has name/#, description, parts (video code)

class Episode
  attr_accessor :name, :src, :desc, :cover
  def initialize(title, page)
    @src = page # parts (megavideo, youtube etc)
    @name = title
    @desc = nil # episode description
    @cover = nil # file path
  end
end

class Anime
  attr_accessor :name, :page, :completed, :anime_type, :episodes
  def initialize(title, page)
    @name = title
    @page = page
    @episodes = []
    @anime_type = 'series'
    @completed = false
  end

  def complete!
    @completed = true
  end

  def episode! episode
    @episodes << episode
  end
end

class Cache
  def initialize
    # Setup physical cache location
    @path = 'cache'
    Dir.mkdir @path unless File.exists? @path

    # key/val = url/filename (of fetched data)
    @datafile = "#{@path}/cache.data"
    @cache = load @datafile
    #puts @cache.inspect
  end

  def put key, val
    tf = Tempfile.new('animecrazy', @path)
    path = tf.path
    tf.close! # important!

    puts2 "Saving to cache (#{path})"
    open(path, 'w') { |f|
      f.write(val)
      @cache[key] = path
    }

    save @datafile
  end

  def get key
    return nil unless exists?(key) && File.exists?(@cache[key])
    open(@cache[key], 'r') { |f| f.read }
  end

  def exists? key
    @cache.has_key? key
  end

private
  # Load saved cache
  def load file
    return File.exists?(file) ? YAML.load(open(file).read) : {}
  end

  # Save cache
  def save path
    open(path, 'w') { |f|
      f.write @cache.to_yaml
    }
  end
end

$cache = Cache.new

def fetch(url)
  body = $mech.get(url).body()
  $cache.put(url, body)
  body
end

def getPage(url)
  # First let's see if this is cached already.
  body = $cache.get(url) 

  if body.nil?
    puts "Not cached. Fetching from site..."
    body = fetch url
  end
  body
end

def main
  # Open anime list (anime_list = saved HTML of
...

sidebar from animecrazy.net)
  anime_list = Hpricot(open('anime_list', 'r') { |f| f.read })
  puts2 "Anime list open"

  # Read in the URL to every series
  masterlist = []

  (anime_list/:li/:a).each do |series|
    anime = Anime.new(series.inner_text, series[:href])
    masterlist << anime
    puts2 "Built structure for #{anime.name}..."
  end

  puts2

  puts2 "Fetched #{masterlist.size} animes. Now fetching episodes..."
  masterlist.each do |anime|
    puts2 "Fetching body (#{anime.name})"
    body = getPage(anime.page)
    puts2 "Snatched that bitch (#{body.size} bytes of Goku Goodness)"
    puts2

    doc = Hpricot(body)
    (doc/"h1/a[@rel='bookmark']").each do |episode|
      name = clean(episode.inner_text)

      if $skip_until
        #$skip_until = !inUrl(episode[:href], 'basilisk-episode-2')
        #$skip_until = nil == name['Tsubasa Chronicles']
        puts2 "Resuming from #{episode[:href]}" if !$skip_until
        next
      end

      # Here it gets tricky. This is a major source of inconsistencies in the site.
      # They group episodes into 1 post sometimes, and the only way to find
      # out from the title of the post is by checking for the following patterns
      # (7 and 8 are example episode #s)
      # X = 7+8, 7 + 8, 7 and 8, 7and8, 7 & 8, 7&8

      # If an episode has no X then it is 1 episode.
      # If it has multiple parts, they are mirrors.
      if single_episode? name
        begin

          puts2 "Adding episode #{name}..."
          ep = Episode.new(name, episode[:href])
          ep.src = getPage(episode[:href])
          anime.episode! ep
        rescue WWW::Mechanize::ResponseCodeError
          puts2 "ERROR: Page not found? Skipping..."
          puts name
          puts2 episode[:href]
        end
      else
        # If an episode DOES have X, it *may* have 2 episodes (but may have mirrors, going up to 4 parts/vids per page).
        # Multiple parts will be the individual episodes in chronological order.
        puts2 "Help me! I'm confused @ '#{name}'"
        puts2 "This post might contain multiple episodes..."

        puts2 "Please visit this URL and verify the following:"
        puts episode[:href]

        if agree("Is this 1 episode? yes/no ")
          begin
            puts2 "Adding episode #{name}..."
            ep = Episode.new(name, episode[:href])
            ep.src = getPage(episode[:href])
            anime.episode! ep
          rescue WWW::Mechanize::ResponseCodeError
            puts2 "ERROR: Page not found? Skipping..."
            puts name
            puts2 episode[:href]
          end
        else
          more = true
          while more
            ename = ask("Enter the name of an episode: ")
            eurl =  ask("Enter the URL of an episode: ")

            begin
              puts2 "Adding episode #{ename}..."
              ep = Episode.new(name, episode[:href])
              ep.src = getPage(episode[:href])
              anime.episode! ep
            rescue WWW::Mechanize::ResponseCodeError
              puts2 "ERROR: Page not found? Skipping..."
              puts name
              puts2 episode[:href]
            end
            more = agree("Add another episode? Y/N")
          end
          puts2 "Added episodes manually... moving on"
        end
      end
    end
    anime.complete!
    # XXX save the entire anime object, instead of just cache
  end
end

def inTitle(document, title)
  return (document/:title).inner_text[title]
end

def inUrl(url, part)
  return url[part]
end

def single_episode?(name)
  !(name =~ /[0-9] ?([+&]|and) ?[0-9]/)
end

def clean(txt)
  # This picks up most of them, but some are missing. Like *Final* and just plain "Final"
  txt[' (Final)']='' if txt[' (Final)']
  txt[' (Final Episode)']='' if txt[' (Final Episode)']
  txt[' (FINAL)']='' if txt[' (FINAL)']
  txt[' (FINAL EPISODE)']='' if txt[' (FINAL EPISODE)']

  txt['(Final)']='' if txt['(Final)']
  txt['(Final Episode)']='' if txt['(Final Episode)']
  txt['(FINAL)']='' if txt[' (FINAL)']
  txt['(FINAL EPISODE)']='' if txt[' (FINAL EPISODE)']

  txt
end

main

If you’re writing your own scraper and would like to use the minimal caching functionality present below, you can gut everything in main() out and put in your own code. Feel free to contact me for assistance.

Here is some sample output:

*** Adding episode Initial D: Episode 1 (Stage 2)...
Not cached. Fetching from site...
*** Saving to cache (cache/animecrazy20090111-12300-mbdpcl-0)
*** Fetching body (Initial D: Third Stage)
*** Snatched that bitch (77695 bytes of Goku Goodness)
***
*** Adding episode Initial D: Third Stage...
Not cached. Fetching from site...
*** Saving to cache (cache/animecrazy20090111-12300-ea69nr-0)
*** Fetching body (Kaiji)
*** Snatched that bitch (87553 bytes of Goku Goodness)
***
*** Adding episode Basilisk Episode 4...
Not cached. Fetching from site...
*** Saving to cache (cache/animecrazy20090110-14992-fomoh0-0)
*** Adding episode Basilisk Episode 3...
Not cached. Fetching from site...
*** Saving to cache (cache/animecrazy20090110-14992-1dx9xm-0)
*** Adding episode Basilisk Episode 2...
Not cached. Fetching from site...
*** Saving to cache (cache/animecrazy20090110-14992-5xt774-0)
*** Adding episode Basilisk Episode 1...
Not cached. Fetching from site...
*** Saving to cache (cache/animecrazy20090110-14992-br5fxd-0)
*** Adding episode Tsubasa Chronicles: Tokyo Revelations Episode 3...
Not cached. Fetching from site...
*** Saving to cache (cache/animecrazy20090110-14992-zmuwix-0)
*** Adding episode Tsubasa Chronicles: Tokyo Revelations Episode 2...
Not cached. Fetching from site...
*** Saving to cache (cache/animecrazy20090110-14992-1ah20eg-0)
*** Adding episode Tsubasa Chronicles: Tokyo Revelations Episode 1...
Not cached. Fetching from site...

This was written for fun, but primarily profit, and not for my own viewing pleasure. The only anime I’ve seen was Akira a decade or so ago, and only because the cover looked cool, but feel free to recommend your favorites.

(be warned that it dumps ALL databases. This can get huge uncompressed)

#!/bin/sh
# Isam (Biodegradablegeek.com) public domain 12/28/2008
# Basic BASH script to dump and compress a MySQL dump

out=sequel_`date +'%m%d%Y_%M%S'`.sql
dest=/bx/

function e {
  echo -e "n** $1"
}

e "Dumping SQL file ($out). May take awhile..."
#echo "oh snap" > $out
sudo mysqldump -u root -p --all-databases > $out
if [ $? -ne 0 ]; then
  e "MySQL dump failed. Check that server is up and your username/pass"
  exit 7
fi

e "Uncompressed SQL file size"
du -hs $out

e "Compressing SQL file"
gz=$out.tar.gz
tar -zvvcf $gz $out
rt=$?

if [ $rt -ne 0 ]; then
  e "tar failed (error=$rt). Will NOT remove uncompressed SQL file"
else
  e "Removing uncompressed SQL file"
  rm -f $out
  out=$gz

  e "Compressed SQL file size"
  du -hs $out
fi

e "Moving shit to '$dest'"
sudo mv $out $dest

BackThatSqlUp.sh

To eliminate having find(params[:id]) in multiple methods, you can use before_filter, like this:

class Admin::PostsController < Admin::ApplicationController
  before_filter :find_post, :only => [:show, :edit, :update, :destroy]
  rescue_from(ActiveRecord::RecordNotFound) { |e| render :text => "
Post not found

" }

  def index
    @posts = Post.find(:all)
  end

  def show
  end

  def new
    @post = Post.new
  end

  def create
    @post = Post.new
  end

  def edit
  end

  def update
  end

  def destroy
  end

protected
  def find_post(id = params[:id])
    @post = Post.find(id)
  end
end

(Thanks Jon)

<% if ENV['RAILS_ENV']=='development' -%>


<% end -%>

A huge benefit is the massive amount of information you learn and get accustomed to in a short period of time. There’s no way to download O’reilly PDFs into your brain just yet, but grokking source code written by those much more experienced than you is one of the fastest ways to see and practice everything you’re been learning in theory (books, sites, classes).

It’s certainly overwhelming to jump head first into a huge app trying to understand every line. I think it’s common for people to open up some code, read it for a few minutes and then never touch it again because they don’t understand it. This was the case with me when I began programming. Here are some ways i used to justify putting off the need to read third party code.

Their code style didn’t suit my taste, i.e., they add the opening curly bracket under the function definition, and I would find myself changing their brackets and formatting more than I spent time actually looking at the logic.

I told myself I would learn much more by re-inventing the wheel, or have more control over my app if I built it from scratch. This is only partial true, but the cons outweight the pros. Reinventing the wheel means diviating from writing program logic and having to learn something that might not even remotely be related to the project I intended to start or finish. Here’s an example that used to be common.

I would happily hack away and then realize that I need a string library (I <3 C). At this point I could have downloaded one, and continued hacking at my app. But instead I would start a new project aimed at writing an efficient string library. Before I knew it, I’m hours into my string lib and totally abandoned my main project. I usually  anyway. The funny thing is that sometimes I’ll just give up and use a third party lib, but most of the time I just ended up scrapping both projects. What a waste of time.

Everyone has their own way of learning other people’s code. I found that what works best for me is to download an app and just browse the source indifferently. It’s not good, it’s not bad, it’s not ugly or inefficient or godly. I just try to get a basic idea of where everything is, what plugins are being used, etc. Layout stuff is in this general area, settings here, plugins are all in there, etc. This step is probably useless and I only do it as a guilt-free way to avoid working for 10 minutes. Or, maybe it does help.

Then I begin by setting up the environment. Put in my DB settings, make sure I have all the requirements installed, etc. Logs are your best lead at this point. I usually open up a fullscreen console and tail -f the log file(s), then watch them like a hawk every time you shift a pixel in the code. It’s not as tedious as it sounds.

Another thing that helps is knowing the language/framework the code is written in. You should know at least how to read the language – though if you’re dedicated, I’m sure you can muddle through with a crash course/reference sheet. By knowing the lang/framework, you can find or create some functions to help you debug and examine objects (even just raise object.inspect is great) in detail. You can also examine other aspects of the code. One thing I do early on is rake routes > ROUTES. This saves all the routes to a file which I can glance at if necessary. And rake routes takes forever on my ancient machine.

After this, the code is compiled/started and I’m ready when I can interact with it live. Now I begin actually reading the code. If I have a specific update I want to make, be it adding a new feature, fixing a bug, or seeing how something works, I work backwards from the live app tracing everything going on around the component I’d like to edit/update.

This is not easy, not quick, and very error prone when you’re making updates. Things breaking == You learning, and all you need to do is be persistent. It’s like reading a calculus book. Nothing will make sense until you turn over the page and see how the problems and answers are laid out, and begin doing them on your own. It’s a great feeling when 2-3 days later you realize that you have a pretty good view of the app’s design (unless it’s something like the Kernel).

No matter how cryptic the code looks, don’t be turned off. Is there some sort of force that will completely stop you from understanding what the logic says? Can other people look at this code and instantly tell you what it’s doing? (well, they can form a great guess if they have the experience). It’s time consuming but the concepts apply to other things in life. You can understand how to change your oil without knowing how the engine works, or how to make simple CSS/HTML changes to a layout to change the background or font size without knowing every single CSS tag and technique.

define:persistent: Refusing to give up, especially when faced with opposition or difficulty; continuing firmly or steadily

Most of my favorite quotes are about it. It’s unfortunate my memory isn’t very persistent.

Do you have any special techniques/advice for learning to read third party code?

This will be updated periodically as I refactor and add support for more apps (notably Joomla and Clipshare – though this would violate their terms unless you have the unlimited license). This was written primarily because I couldn’t stand setting up another vhost and WordPress installation. There are plenty of existing deployers but I plan on adding very specific features and tweaking this for in-house work. I also wanted to try Rio (Ruby-IO). GPL license. Go nuts.

* As of 10/11, the apicore.rb file on the site has some syntactic errors in the domainResourceSave method. I sent an email out to the author about it. Problems aren’t major. You can get my apicore.rb here.

This won’t run unless you create the appropriate folder structure in /etc/mksite/. I’ll get going on this in a bit. See the code below:

#!/usr/bin/env ruby 

########################################################
# Isam M.
# http://biodegradablegeek.com
# MKSITE.RB (0.2) Last updated Oct 19th, 2008
#
# mksite makes it quicker to setup sites and web
# apps by doing most of the tedious work.
#
#
#                     UNSTABLE
#           Run it in your imagination,
#              not on your system!
#
#
######################################################## 

require 'rubygems'
require 'rio'
require 'yaml'
require 'mysql'
require 'highline/import'
HighLine.track_eof = false

begin
  require 'apicore'
rescue LoadError
  puts "NOTICE: Unable to load apicore.rb - domains will not be added automatically"
end

Apache_sites = '/etc/apache2/sites-available/'
Subdomains = ['', 'www', 'mail', 'blog', "dev#{(rand*10).floor}"]
Testing = false
Homedir = ENV['HOME']
Username = ENV['USER']
Applications = [:Skeleton, :Clipshare, :Joomla, :PHPMotion, 'PmWiki (N/A)', :Wordpress]
$config = nil
$log = nil

def say(txt)
  super txt
#  $log << txt if $log
end

# flash('message', :notice || :error) to output msg
def flash(msg, message_type = :notice)
 if message_type.eql? :notice
    say("INFORMATION: #{msg}")
  elsif message_type.eql? :emphasize
    say("***************************************************")
    say("NOTICE: #{msg}n")
    say("***************************************************")
  elsif message_type.eql? :error
    STDERR.puts("FATAL ERROR: #{msg}n")
  end
end

# DZone 2111
def genAlpha(size=64)
  s=''
  size.times {
    s << (i = Kernel.rand(62); i += ((i < 10) ? 48 : ((i < 36) ? 55 : 61 ))).chr
  }
  s
end

if defined? ApiCore
  class DNSAPI < ApiCore
    # This depends on the Linode API Ruby bindings (apicore.rb)
    def initialize(key, debug=false, batching=false)
      super
      @batching=false
    end

    # Add domain. Return ID on success, nil on failure
    def addDomain domain
      return nil if !domain
      params = {}
      params[:DomainID] = 0
      params[:Domain] = domain
      params[:Type] = 'master'
      params[:Status] = 1
      params[:SOA_Email] = getVal('email') || ask("Enter SOA email for the domain: ")
      domainSave params
    end

    def addDomainResource(domain, resource, target, record_type = 'A')
      return nil if (did = getDomainIdByName domain).nil?
      params = {}
      params[:ResourceID] = 0
      params[:DomainID] = did
      params[:Name] = resource
      params[:Type] = record_type
      params[:Target] = target
      domainResourceSave params
    end

    def getDomainIdByName domain
      domainList.find do |dom|
        return dom["DOMAINID"] if dom["DOMAIN"].downcase == domain.downcase
      end
    end
  end
else
  flash('ApiCore not loaded. Skipping DNS stuff', :notice)
end

class App
  class << self; attr_reader :message, :name, :version, :description, :vhost; end
  @message = nil
  @name = nil
  @version = nil
  @description = 'Just Another Web App'
  @vhost = 'generic'

  def initialize(rootdir, domain, db)
    flash("Initializing application (dir=#{rootdir}, domain=#{domain})")
    @rootdir = (rootdir[-1].chr.eql? '/') ? rootdir.chop : rootdir
    @domain = domain
    @db = db # hash of database info 

    # System stuff
    #if Testing
      #@templates = '/home/kiwi/Code/mksite/templates'
      #@configs = '/home/kiwi/Code/mksite/configs'
      #@vhosts = '/home/kiwi/Code/mksite/vhosts'
    #else
      @templates = '/etc/mksite/templates'
      @configs = '/etc/mksite/configs'
      @vhosts = '/etc/mksite/vhosts'
    #end
  end

  # This generally does not need to be overriden.
  # It does 'generic shit' like creating the rootdir
  # and setting permissions
  def envSetup
    if !rio(@rootdir).exist?
      flash("Creating directory #{@rootdir}")
      rio(@rootdir).mkpath
    end

    rio(@rootdir).chdir do |root|
      # Copy the generic public/private/log apache
      # structure to rootdir
      flash("Changing working dir to #{@rootdir}")
      flash("Working inside '#{root.to_s}'")
      rio(@templates,'skeleton.www').each { |df|
        # Overwrite existing files? .. yes.
        #while rio(root, df).exist? do
        df > root
        #flash("Copied #{df.to_s} to #{root.to_s}")
      } 

      # Set permissions (a+w on logs, etc)
      flash('Setting permissions...')
      flash('666 ./log/*.log')
      rio('./log/access.log').chmod(00666)
      rio('./log/error.log').chmod(00666)
      flash('700 ./private')
      rio('./private/').chmod(00700)
    end
  end

  def setup
    # envSetup()
    # databaseSetup()
    raise 'OVERRIDE ME'
    #flash("setup() template function invoked. 'OVERRIDE ME'", :log)
  end

  def databaseSetup
    # Create database if it doesn't already exist
    # This usually doesn't need to be overriden
    #Mysql.server_connect(@db['name'])
    #  flash('Checking database connection... ')
    begin
      dbo = Mysql.real_connect(@db['host'], @db['user'], @db['pass'])
      flash("Creating database '#{@db['name']}'");
      res = dbo.query("CREATE DATABASE IF NOT EXISTS #{@db['name']};")
      flash("Database server returned #{res}") if res
    rescue Mysql::Error => err
      flash('Unable to connect to access/create database', :error)
      flash("Error returned (#{err.errno}) = '#{err.error}'", :error)
      exit 1
    ensure
      dbo.close if dbo
    end
  end

  def dnsSetup
    return if !defined? ApiCore

    # Setup DNS - currently uses Linode API
    flash("Setting up DNS for '#{@domain}'")
    flash("This requires a Linode API keynLogin to linode.com and find it under 'My Profile'", :emphasize)
    dns = nil
    api_key = getVal('apikey')
    loop do
      api_key = ask('Paste your Linode API key (or 'skip'): ') if api_key.nil? || api_key.empty?
      break if api_key.downcase.eql? 'skip'

      # Check API key
      begin
        dns = DNSAPI.new api_key
        dns.domainList
        break
      rescue RuntimeError
        flash("API key invalid (or service down?). Learn to paste and try again.n")
        api_key = nil
      end
    end

    unless api_key.nil? or api_key.downcase.eql? 'skip'
      flash("Adding master domain '#{@domain}'")
      begin
        begin
          dns.addDomain(@domain)
        rescue RuntimeError
          flash('Unable to add domain (exists already?). Attempting to add subdomains...', :error)
        end

        server = getVal('server') ||
                   ask("Enter IP subdomains should point to (or 'skip'): ") { |q| q.default = 'skip' } 

        unless server.downcase.eql? 'skip'
          Subdomains.each do |sub|
            flash("Adding subdomain '#{sub}.#{@domain}' (points to #{server})")
            flash('Could not add subdomain', :error) if !dns.addDomainResource(@domain, sub, server)
          end
        end
     rescue
       raise
       flash("Unable to add domain/subdomain. Skipping", :error)
     end
    end
  end

  def serverSetup
    # Set Apache vhost
    vhost_file = (defined? self.class.vhost) ? self.class.vhost : 'generic'
    flash("Generating vhost file (#{@vhosts}/#{vhost_file}) for Apache 2.x")
    vhost = ''
    email = getVal('email') || ask("Enter a valid email for tech support: ")
    rio(@vhosts, vhost_file) > vhost
    vhost.gsub!('_MKS_DOMAIN_', @domain)
    vhost.gsub!('_MKS_EMAIL_', email)
    vhost.gsub!('_MKS_ROOT_', "#{@rootdir}")
    vhost.gsub!('_MKS_PUBLIC_', "#{@rootdir}/public")
    #rio(Testing ? '/tmp/' : Apache_sites, @domain).puts(vhost)
    rio('/tmp/', @domain).puts(vhost) 

    # Your enemies should not read this
    rio('/tmp/', @domain).chmod(00600)
    flash("vhost file has been generated as /tmp/#{@domain}n
          It is YOUR responsibility to move this to #{Apache_sites}n
          Site will not work until you 'a2ensite && apache2ctl restart'", :emphasize)
  end

  def postInstall()
    flash("Finished!", :emphasize)
  end
end

class Skeleton < App
  @name = 'Skeleton'
  @description = 'Generic WWW directory structure'
  @message = ''
  @vhost = 'generic'

  def setup
    self.envSetup()
    self.dnsSetup()
    self.serverSetup()
  end
end

class WordPress < App
  @name = 'Wordpress'
  @version = '2.6.2'
  @description = 'A popular blogging platform'
  @message = 'Trying to make monies on the Internets?'
  @vhost = 'generic'

  def setup
    # This sets up the initial environment / permissions
    self.envSetup() 

    # Copy the wordpress skeleton directory to the new dir
    wproot = rio(@rootdir,'public')
    flash("Wordpress root will be #{wproot}")
    flash('Copying WordPress data over... (may take awhile)')
    rio(@templates,'wordpress').each { |df| df > wproot } 

    rio(wproot).chdir do
      rio('wp-config-sample.php').rm()
      # Generate and output wp config
      flash('Generating wp-config.php based on your DB settings...')
      wpcfg = rio(@configs, 'wordpress.cfg')
      if !wpcfg.exist? || !wpcfg.readable?
        flash("Wordpress config template missing or unreadable, quitting", :error)
        exit 2
      end

      # Copy the config into a string, do things with it and then write it to disk
      config = ''
      rio(@configs, 'wordpress.cfg') > config
      flash("Warning: config file '#{@configs}/wordpress.cfg' is empty") if config.empty?
      config['_MKS_DB_HOST_'] = @db['host']
      config['_MKS_DB_USER_'] = @db['user']
      config['_MKS_DB_PASS_'] = @db['pass']
      config['_MKS_DB_NAME_'] = @db['name']
      config.gsub!('_MKS_SECRET_', genAlpha())
      flash("Writing #{wpcfg} data")
      rio('wp-config.php').w!.puts(config)
    end

    # Setup the database
    self.databaseSetup() 

    # Add DNS info
    self.dnsSetup() 

    # Setup the server/vhost
    self.serverSetup()
  end
end

class Clipshare < App
  def setup
    raise 'Clipshare support is currently not available. sowwie'
  end
end

class Joomla < App
  @name = 'Joomla'
  @version = '1.5.7'
  @description = 'A widely used Content Management System'
  @message = ''
  @vhost = 'generic'

  def setup
    # This sets up the initial environment / permissions
    self.envSetup() 

    # Copy the wordpress skeleton directory to the new dir
    flash('Copying Joomla data over... (may take awhile)')
    jooroot = rio(@rootdir,'public')
    flash("Joomla root will be #{jooroot}")
    rio(@templates,'joomla-1.5.7').each { |df| df > jooroot } 

    rio(jooroot).chdir do
      flash('Removing stock Joomla config...')
      rio('configuration.php-dist').rm()

      # Generate and output joomla config
      flash('Generating configuration.php based on provided settings...')
      joocfg = rio(@configs, 'joomla.cfg')
      if !joocfg.exist? || !joocfg.readable?
        flash("Joomla config template missing or unreadable, quitting", :error)
        exit 2
      end

      # Copy the config into a string, do things with it and then write it to disk
      config = ''
      rio(@configs, 'joomla.cfg') > config
      flash("Warning: config file '#{@configs}/joomla.cfg' is empty") if config.empty?
      config['_MKS_DB_HOST_'] = @db['host']
      config['_MKS_DB_USER_'] = @db['user']
      config['_MKS_DB_PASS_'] = @db['pass']
      config['_MKS_DB_NAME_'] = @db['name']
      config.gsub!('_MKS_DOMAIN_', @domain)
      config.gsub!('_MKS_SECRET_', genAlpha()) 

      flash("Writing ./configuration.php")
      rio('configuration.php').w!.puts(config) 

      flash("Making configuration.php world writable (0666)")
      rio('configuration.php').chmod(0666)
    end

    # Setup the database
    self.databaseSetup() 

    # Add DNS info
    self.dnsSetup() 

    # Setup the server/vhost
    self.serverSetup()
  end

  def postInstall()
    flash('Joomla is ready to be setup using the web interface', :emphasize)
    flash("Go to #{@domain} where the Joomla! web based installer will
           guide you through the rest of the installation")
    flash("Here's the database information:n
           USERNAME: #{@db['user']}n
           DB HOST : #{@db['host']}n
           DB NAME : #{@db['name']}") 

    flash("nAdmin panel is located @ #{@domain}/administrator ")
    flash("You can log into Admin using the username 'admin' along with the
    password that was generated or you chose during the web based install.")
    super
  end
end

class PHPMotion < App
  def setup
    raise 'PHPMotion support is currently not available. sowwie'
  end
end

class Log
  def initialize(filename="/tmp/#{Username}_mksite.log")
    @log = rio(filename)
    n=0; @log = rio("#{filename}.#{n+=1}") while @log.exist?
    @log.puts("# GENERATED BY MKS - BEGAN @ #{Time.now.to_i}")
    @log.chmod(00600)
    flash("Log file generated (important): #{@log.to_s} (no worries, set to 600)", :emphasize)
    return @log
  end

  # append to log
  def <<(data)
    #super <<(data)
    @log.puts(data)
    @log.puts("n")
    #@log << data << "n"
  end
end

def loadConfig()
  cfgpath = "#{ENV['HOME']}/.mksite"
  return YAML.load(rio(cfgpath).read()) if rio(cfgpath).exist?
  nil
end

def getVal(key, default=nil)
  return $config[key] if $config and defined? $config[key]
  default
end

def main
  exit 1 if 'root'==ENV['USER']
  if ($config = loadConfig())
    flash('~/.mksite config loaded')
  else
    flash('~/.mksite not found. It's fine, I'll annoy you with questions.')
  end

  puts $config.inspect if Testing
  flash('Press ^C (CTRL+C) at any time quit', :emphasize)
  # Let's ask neutral questions about the new site.
  if Testing
    rootdir = '/tmp/sandbox9/'
    domain = 'domain.cxm'
  else
    domain = ask('Enter site's domain name (no http:// or www): ') #{ |d| d.validate = !! /^www.|^http:/// }
    rootdir = ask('Root site directory (leave blank for default): ') { |q|
      q.default = "/home/#{Username}/www/#{domain}/"
      q.validate = /^/home/#{Username}//
    }
  end

  flash("Domain has been set to "#{domain}"")
  #$log = Log.new("/tmp/mks_#{domain}.log") 

  flash("Site will reside in "#{rootdir}", and the index/script")
  flash(" files (index.php, .htaccess etc) will go in "#{rootdir}/public/"nn")

  # Ask the user what app she wants to install and specific questions about that app
  app = nil
  choose do |menu|
    menu.prompt = 'Choose the software for your new site: '
    Applications.each do |app|
      menu.choice app do |a|
        app = a
      end
    end
  end

  flash("You chose #{app}. #{Kernel.const_get(app).message}n") 

  # Fetch DB info if this app needs a database
  # XXX Should just have a db flag in the App's class
  db = nil
  unless app.eql? :Skeleton
    flash('This app requires a database (only MySQL supported)')
    flash('It will be created if it doesn't exist')

    db = {}
    db['host'] = 'localhost'
    loop do
      if Testing
        db['user'] = 'kiwi'
        db['pass'] = ''
        db['name'] = 'kiwi_sandbox9'
      else
        db['user'] = getVal('db_user') || ask('Enter MySQL username: ')
        db['pass'] = getVal('db_pass') || ask('Enter MySQL password: ') { |q| q.echo = '*'}
        db['name'] = ask('Enter database name (should begin w/ your 'username_')') { |dbn| dbn.validate = /^#{Username}_/}
        puts db.inspect
      end
      flash("Please double check: mysql://#{db['user']}:#{'*' * (db['pass']).size}@#{db['host']}/#{db['name']}")
      break if Testing || agree('Is this correct?')
      flash("Please re-enter database infon")
    end
  end

  # Check permissions and database login
  flash('Doing a preliminary check', :emphasize)
  flash('Checking installation environment')
  # Quit if the directory exists and the user does not want to go ahead
  unless Testing
    exit 1 if rio(rootdir).exist? && !agree("Directory '#{rootdir}' exists! Continue? (Y/N)")
  end

  # Is the parent directory available and writable?
  parentdir = rio(rootdir).dirname
  unless parentdir.exist? && parentdir.writable?
    flash("Directory '#{parentdir.to_s}' either doesn't exist or is not writable. Check permissions", :error)
    exit 1
  end

  unless db.nil? || false==getVal('db_confirm')
    # Check DB connection - but this does not check if user has any privileges
    flash('Checking database connection')
    begin
      dbo = Mysql.real_connect(db['host'], db['user'], db['pass'])
      flash("Successfully connected to '#{dbo.get_server_info}'")
    rescue Mysql::Error => err
      flash('Unable to connect to database server', :error)
      flash("Error returned (#{err.errno}) = '#{err.error}'", :error)
      exit 1
    ensure
      dbo.close if dbo
    end
  end

  # Begin installation
  flash("Word on the server racks is... you're good to go", :emphasize)
  klass = Kernel.const_get(app)
  webapp = klass.new(rootdir, domain, db)
  webapp.setup()
  webapp.postInstall()
end

main()

Download the ASCII mksite.rb file here.