Blogger index

I have been a little frustrated with the template limitations and design of blogger. I have checked out and started a Vox account and have been very impressed by what they have to offer but the customisation is even more restricted even though you have way more gizmos to play with. I still like Vox and will probably maintain the blog because of the community feel to it.

My main beef with Blogger is that it does not currently have a way to list all of the titles of your blog postings on one page. To get to old articles you either have to know the date you published and/or explore the archive or use the search engine. I believe this will have negative effects on users browsing your articles and possibly search engines that see the only way to get to the old articles is to follow multiple links down. When people ask me questions I want to find a posting really quickly, sure search is OK, but it feels messy and may miss, so having an index allows me to copy and send the link on very quickly.

To solve my needs I created a Python program to read the blog pages, scan the archives and create my own index page. I then encased it all in a script that would add headers and footers then upload the page to the right spot.

The more I use Python, the more I find it a very powerful programming language that is both easy and quick to write as well as read. For your programming pleasure I provide the code below. Feel free to copy/alter and make suggestions as my experience with Python is rather limited.

blogdex.py

# BLOGDEX
# Uses the latest template and requires the Blog Archive widget in flat list format
# - Simon Forsyth 20 Feb 2007, updated 11 Jul 2008

import urllib, sys, re

srcURL = "http://neoporcupine.blogspot.com"
outFileName = "blogdex.out"
logFileName = "blogdex.log"

log = open(logFileName,"w")
outFile =  open(outFileName,"w")


def ExtractYM (thisURL):
 # Returns the Year and Month specified in the URL in string format 2004/08
 # http://neoporcupine.blogspot.com/2004/08/microsoft.html
 myRE = re.compile (r'(\d\d\d\d/\d\d)')
 thisMatch = myRE.search(thisURL)
 if thisMatch:
  return thisMatch.group(1)
 else:
  return "----/--"

def ProcessArchive (thisURL):
        # This is the monthly/weekly/daily page with possibly multiple blog entries
        print "Processing " + thisURL
 print "****"
 log.write ("Opening %s " % (thisURL))
 
 # Read in source URL
 f = urllib.urlopen(thisURL)
 MainPage = f.read()
 f.close
 log.write ("success\n")
 
 # Get titles
 log.write ("Searching for titles\n")
 myRE = re.compile (r'<h3 class=\'post-title entry-title\'>(.*?)</h3>', re.DOTALL)
 myTitles = myRE.findall(MainPage)
 myRE = re.compile(r'<a href=.*?>(.*)</a>', re.DOTALL)
 # Remove any link from titles
 for i in range(0,len(myTitles)):
  myCheck = myRE.search(myTitles[i])
  if myCheck:
   myTitles[i] = myCheck.group(1).strip()
  else:
   myTitles[i] = myTitles[i].strip()
  log.write ("\t%s\n" % (myTitles[i]))

 # Get direct link to the article page by processing the link at the bottom of each article
 log.write ("Extracting URL details\n")
 myRE = re.compile (r"<span class='post-timestamp'>(.*?)</span>", re.DOTALL)
 myURLs = myRE.findall(MainPage,re.DOTALL)
 
 # Extract URL info
 myRE = re.compile (r'href=\'(.*?)\'')
 for i in range(0,len(myURLs)):
  myURLs[i] = myRE.search(myURLs[i]).group(1)
 
 if len(myURLs) != len(myTitles):
  #Error of some sorts
  log.write ("ERROR: number of titles and urls different")
  for i in range(0,len(myTitles)):
   log.write  ( "%d %s\n" % (i,myTitles[i]) )
  for i in range(0,len(myURLs)):
   log.write  ( "%d %s\n" % (i,myURLs[i]) )
 else:
  # Write date, and all title-URL to output
  log.write ("Writing output\n")
  outFile.write ( "<P><b>%s</b>\n" % (ExtractYM(myURLs[0])) )
  for i in range(0,len(myTitles)):
   outFile.write ( "<li><a href='%s'>%s</a></li>\n" % (myURLs[i],myTitles[i]) )

def processBlog(thissrcURL):
 # Read in source URL
 log.write ("**********************************************")
 log.write ("Opening %s " % (thissrcURL))
 f = urllib.urlopen(srcURL)
 MainPage = f.read()
 f.close
 log.write ("success\n")

 # Extract arhive URL section from the "Blog Archive" widget in flat format
 myRE = re.compile (r'<div id=\'BlogArchive1_ArchiveList\'>(.*?)</div>', re.DOTALL)
 myRawList = myRE.search(MainPage)
 if myRawList:
  log.write ("Found archive list in %s\n" % (thissrcURL))
  # print myRawList.group(1)
 else:
  print "Could not find archive-list in main blog"
  log.write ("Could not find archive-list in %s\n" % (thissrcURL))
  sys.exit

 # Convert the archive url section into an array of URLs
 myRE = re.compile(r'href=\'(.*)\'>')
 myArchiveURLs = myRE.findall(myRawList.group(1))
 
 # Process each Archive URL
 log.write('Found %i archive urls.\n' % (len(myArchiveURLs)))
 for i in range(0,len(myArchiveURLs)):
  ProcessArchive (myArchiveURLs[i])
  

processBlog (srcURL)
log.close
outFile.close

Then a nice little shell script (works with OS X) to scoop it all together and ftp off to your favourite website. This assumes that you have already created the header.html and footer.html that surrounds the blogdex.out output.

python blogdex.py
cat header.html > blogdex.html
cat blogdex.out >> blogdex.html
cat footer.html >> blogdex.html
# login to ftp assigning out put to END SCRIPT
ftp -n my.server.net << cp2web.log
user myusername mypassword
passive
put ~/myScripts/blogdex/blogdex.html /blogdex.html
quit

Europe: my guide

Long time no post - well I went to Europe for a while. This is my second trip to Europe, but this time we were determined to see more places over a longer period than the first two week trip allowed. I thought I would share some tips. You may want to check out my photo collection hosted by the wonderful people at Flickr or on my .Mac website while it still lasts.

European capitals are expensive

But that still does not mean you have to be ripped off. Expect to pay more for food and accommodation, however, do some investigation. We travelled during low season so we were able to haggle prices down. Other things like food can get you though, people really want you to pay top euro for cola. Grab some in larger bottles from grocery stores before you go out. Fill up bottles with water, apparently it is safe - I had no problems doing this and I even used a public fountain in Venice. Don't buy food or drink around tourist attractions. Certainly don't get your money changed under the Eiffel tower where an obnoxious little man sits in his booth, pretending not to speak any English but apparently understanding everything we said, adding extra charges for no readily apparent reason. I wish I took a pic of his grey bearded angry red face for the blog. Just to balance it out, everyone else we met in France was extremely friendly and helpful.

If you have a Eurail pass make sure you investigate time tables on the internet before you go. Also, check out alternative times and train types. There are a couple private train companies that like to think their trains are so much better than the standard ones that they can charge a very large premium above the Eurail ticket. Don't go on these, there is always an alternative and every standard train that we went on was fine to at least as good as the private company trains.

Accommodation

My partner found a great site called Trip Advisor which shows reviews posted by the general public about hotels. We were initially very concerned as many of the hotels we were looking at had some poor reviews in amongst some glowing reviews. Then after looking at some five star accommodation I realised that some people complain about anything. You have to read the negative review with suspicion to try to determine the psychological state of the person who wrote it. Some people will tell you the hotel is a write off because the desk clerk wouldn't pay them enough attention. Phht, welcome to Europe you self indulgent idiot. Others find a stain on the carpet or a crack in the wall and think they whole place should be condemned. Or perhaps the bathroom isn't an acre wide for their fat American backsides to fit in. Some are legitimate, like the one I am going write about the appaling Brodies Hostel in Edinburgh.

Again, low season is great. In Edinburgh we abruptly left the horror of Brodies Hostel and I decided to splurge. I knew the train station was close by, which had a "last minute" hotel booking place. They mentioned a couple places and I asked if she could get us in for 20 pounds less, a quick call confirmed that this was fine and we were in a very pleasant four star hotel rather than the barely better than a dumpster zero star Brodies Hostel.

To save ourselves a night of accommodation we decided to try a sleeper cabin from Zurich to Venice. I had never been in a sleeper before and was quite happy that we were trying it out. I was surprised that it was 100 euro over the eurail pass, but thought that was reasonable ... until he explained that it was 100 euro each. I was hesitant, I still wanted to experience a sleeper. Then the attendant said "I would not regret it, the sleepers are wonderful." So I got the sleeper tickets. He lied. They were crap. It would have been much better to travel in on an earlier train and get an extra night in Venice. Still, now I know. Maybe there are better sleeper cabins around but I won't be game to try them.

Time of year

Low season has fewer and smaller crowds, more free seats at events and on trains, you don't get hot and sweaty while trudging around, most things are cheaper or negotiable, you get to see snow (except global warming seems to have prevented that for our trip) or at least frosts and fogs. The downside is that some places are closed (the chocolate museum in Belgium) or close early, you have a lot less light to work with, fountains are empty in the north (Venice and Rome were fine), many trees are leafless and no flowers (although Amsterdam still had heaps), it can get bloody cold and windy. Despite the cold, we had an awesome time! Frankly I gladly accept the chill to avoid the crowds.

Travel guides

Travel guides are hugely useful. Lonely planet, Frommers and Let's Go seem to be the common ones about. I got the "Let's Go" guide and picked up plenty of great tips on what to see, eat and places to stay. Not all the places to stay are that good, but a little research on your own and you will find plenty of accommodation bookable over the internet.

One of the tips in the guides was about pick pockets and how they worked. Very interesting. So when we encountered them I knew exactly what they were and how they were going to operate. We actually encountered them in Rome at about the exact place that the guide told us we would. I was so amused that I stopped in my tracks and said to my partner "Oh! They're pick pockets! Cool!" and then realised that there were about to make a grab for my pocketed camera so I launched into a run, knocking them out of the way. I reported them to a nearby policewoman who only spoke Italian but understood my mime. So rather than be the event where I lost my camera, it was the event where I foiled pick pockets thanks to the warning from the guide books.

Europe is enormous and there is lots of good information to put in these guides making them quite bulky. You can get pocket guides which are specific for only one city which have the same sort of information, usually in more detail, as the large guides, as well as containing really good quality maps. Certainly the pocket books fit into my pocket, but I was wearing a nice warm winter jacket with deep pockets. During summer they would be more of a pain. You would have to carry them around in your backpack, but they are far easier to carry than the complete Europe guide. Each hotel usually has a good tourist map for free which is usually pamphlet size and often the hotel clerk can offer you great tips about the local area.

Non-smoker's guide to Europe

Only go to Italy, everywhere else stinks. Switzerland was particularly nasty because of the combined effect of the very smelly cheese fondue. In each country we visited we were told that they were considering or about to ban smoking in public places / bars and restaurants, but nobody could name a specific date. In Italy this had already taken place. We also noted on the news that Hong Kong had gone smoke free. Really, Europe would have you believe they are all about high culture, but I think they are an oldish culture that plays catch-up on many issues. They have their fair share of rude, ugly, ignorant, aggressive, unhelpful people like any other culture.

If you have to visit France, Amsterdam, Germany, Switzerland or what ever country that has not yet been enlightened about their responsibility of keeping the local air breathable for others then your best bet is to have dinner early and ask the restaurant about the non-smoking area. If you get placed out of the way in an ugly spot near the toilet or the kitchen doors then just get up and leave - there are plenty of restaurants about. One particularly lovely Indonesian restaurant in Amsterdam put us way at the back almost in our own private quiet softly lit room - it was excellent, as was the meal. Alternatively, a restaurant in Wales had the non smoking area next to the kitchen surrounded by the smoking area, at which we just chuckled at the waiter, politely informed him of our disapproval and left.

You should not be impolite as these people are not as well educated on the issues of smoking so you must make allowances for their immature cultural differences.

You must go!

Apart from the caveats ... just go to Europe, have a big budget, make sure you check out the hotels somehow before you go. It is awesome. Europe is a romantic setting - the perfect place to pop the question. She said yes :)