Warning: fopen() [function.fopen]: open_basedir restriction in effect. File(/f5/virtuous/public/wp-content/backup/.htaccess) is not within the allowed path(s): (/fs1c/virtuous/:/nfsn/apps/php53/lib/php/:/nfsn/apps/php5/lib/php/:/nfsn/apps/php/lib/php/) in /fs1c/virtuous/public/wp-content/plugins/backupwordpress/functions.php on line 377

Warning: fopen(/f5/virtuous/public/wp-content/backup/.htaccess) [function.fopen]: failed to open stream: Operation not permitted in /fs1c/virtuous/public/wp-content/plugins/backupwordpress/functions.php on line 377
Cannot open file (/f5/virtuous/public/wp-content/backup/.htaccess)
Warning: fwrite() expects parameter 1 to be resource, boolean given in /fs1c/virtuous/public/wp-content/plugins/backupwordpress/functions.php on line 381
Cannot write to file (/f5/virtuous/public/wp-content/backup/.htaccess)
Warning: fclose() expects parameter 1 to be resource, boolean given in /fs1c/virtuous/public/wp-content/plugins/backupwordpress/functions.php on line 385
rss « Virtuous Programmer

Virtuous Programmer Adventures of an Autodidact

15Nov/100

Python 3/4: Context Managers, XML and Networking

Posted by Frank Berthold

This is the third article in a series, to read the entire series go here.

Writing an RSS Aggregator with Python

For this article I'll be putting together a simple RSS aggregator. This will shed a some light on Python's XML and web libraries and introduce a way to minimize boilerplate code with context managers.

The final product will read through a file containing a list of RSS feeds and print the names of the channels and the articles they contain to screen.

To use it, type: python RssReader.py feeds.txt

Files Used in this Project

  • RssReader.txt: The sourcecode for this project, rename it to "RssReader.py" after downloading.
  • feeds.txt: A sample input file.

Libraries

  1. import sys
  2. import urllib2
  3. import xml.sax
  4. import xml.sax.handler
  5.  
  • sys: A library for accessing information passed into or maintained by Python's interpreter.
  • urllib2: A library for manipulating URL's and opening HTTP streams.
  • xml.sax: Read from and write to XML files using SAX.
    • Tutorial: A good overview of parsing XML using the SAX library.

Parsing XML

The RSS feed parser extracts the names of the channels and the titles of each of the articles.

The handler below gets most of its functionality from its parent class. There are four methods added: startDocument, startElement, endElement and characters. These each fire as an XML document is being read. Because the methods themselves are generic enough to apply to any XML document, you need a lot of if/else statements inside to deal with specific kinds of XML document.

  1. class RssHandler(xml.sax.handler.ContentHandler):
  2.   def startDocument(self):
  3.     self.inItem = False
  4.     self.inTitle = False
  5.     self.channels = {}
  6.     self.currentChannel = None
  7.     self.str = ""
  8.  
  9. def startElement(self, name, attrs):
  10.     lname = name.lower()
  11.  
  12.     if lname == "item":
  13.       self.inItem = True
  14.     elif lname == "title":
  15.       self.inTitle = True
  16.  
  17. def endElement(self, name):
  18.     lname = name.lower()
  19.  
  20.     if lname == "item":
  21.       self.inItem = False
  22.     elif lname == "title":
  23.       self.inTitle = False
  24.       if self.inItem:
  25.         self.channels[self.currentChannel] += [self.str]
  26.       else:
  27.         self.currentChannel = self.str
  28.         if self.currentChannel not in self.channels.keys():
  29.           self.channels[self.currentChannel] = []
  30.       self.str = ""
  31.  
  32.   def characters(self, content):
  33.     if self.inTitle:
  34.       self.str += content
  35.  

This class reads through an XML document and accumulates a relational list of the form: {"Channel 1":["Article 1", "Article 2"], "Channel 2":["Article 3"]}

Context Managers and Downloading Webpages

Context Managers

Files, network connections and other information that require streams of data generally require some amount of boilerplate to open and close those streams. This both adds uninteresting noise to your code and can cause unexpected bugs when you open a stream but fail to close it. Python deals with this through context managers.

Without context managers printing the lines of a file to screen looks like this:

  1. file = open("infile.txt", "r")
  2. print file.read()
  3. file.close()
  4.  

Though this is clear and obvious in a small example, in real code there can be problems between opening and closing the file, which cause it not to be closed properly. The standard fix is to wrap this with a try/finally to deal with any exceptions that are thrown in the process:

  1. file = open("infile.txt", "r")
  2. try:
  3.   print file.read()
  4. finally:
  5.   file.close()
  6.  

Where finally makes certain that, regardless of what happens in the middle, the file gets closed. This is effective, but contains a fair amount of boilerplate for a simple action. Context managers allow us to do this:

  1. with open("infile.txt", "r") as file:
  2.   print file.read()
  3.  

The open function returns a file object which contains the context manager. Any object can become a context manager object by adding an __enter__ and __exit__ method for creation and cleanup. Many of Python's standard stream objects have these methods added already.

Downloading Webpages

Unfortunately the urlopen function from urllib2 does not come with a context manager, so I had to write one myself. Fortunately, it's easy.

  1. class Url():
  2.   def __init__(self, url):
  3.     self.url = url
  4.  
  5.   def __enter__(self):
  6.     self.stream = urllib2.urlopen(self.url)
  7.     return self.stream
  8.  
  9.   def __exit__(self, type, value, traceback):
  10.     self.stream.close()
  11.  

Tying it All Together

Using the XML Parser

  1. def generateRsses(feedFile):
  2.   with open(feedFile, "r") as file:
  3.     urls = [url.strip() for url in file.readlines()]
  4.  
  5.   for url in urls:
  6.     with Url(url) as rss:
  7.       handler = RssHandler()
  8.       parser = xml.sax.make_parser()
  9.       parser.setContentHandler(handler)
  10.       parser.parse(rss)
  11.       yield handler.channels
  12.  
  13. def printFeed(rss):
  14.   for channelName in rss.keys():
  15.     print "*** " + channelName + " ***"
  16.     for title in rss[channelName]:
  17.       print "\t" + title
  18.  

The most complex part of this is the actual parsing of the xml, I'll walk you through it line by line.

  1. Create the RssHandler (the class created above). This defines how the parser will work.
  2. make_parser creates an object that does the XML heavy lifting.
  3. Attach the handler to the parser.
  4. Do the actual parsing.
  5. Extract the resulting data.

Commandline Arguments

  1. if __name__ == "__main__":
  2.   [scriptName,feedFileName] = sys.argv
  3.   for rss in generateRsses(feedFileName):
  4.       printFeed(rss)
  5.  

The only new element here is sys.argv. sys.argv contains all of the arguments handed into the python interpreter. The first is the name of the script itself, so the program ignores it, the second is the name of the file that contains a list of RSS feeds.

Resources:

Filed under: python, rss, xml No Comments
20Dec/100

Scala 3/4: XML

Posted by Frank Berthold

This is the third article in a series, to read the whole series go here.

Writing an RSS Aggregator with Scala

One of the cooler things about Scala is its XML processing and production features. Scala has several powerful mechanisms for creating DSL's (Domain Specific Languages) which are essentially special purpose languages inside the parent language. The details of how it goes about this are beyond the scope of this article, but I'll show you one way you can enjoy some of the results.

To illustrate how Scala can be used to process and generate XML, I'll be putting together a simple RSS aggregator. It will print to the screen and write an html file with the channel titles and article titles.

Files Used in this Project

Libraries

  1. import scala.xml._
  2. import java.net.URL
  3.  
  • scala.xml: Scala's XML library has a vast number of features which I've only begun to sample.
  • java.net.URL: Java's library for URL's and HTTP connections.

I've mentioned that Java libraries can be called as though they were Scala libraries elsewhere, this is the first time I've actually done it. It's painless as advertised.

Main Functions, Arrays and Command Line Arguments

  1. object RssReader {
  2.   def main(args : Array[String]) : Unit = {
  3.     val rssFeeds = combineFeeds(getRssFeeds(args(0)))
  4.     printFeeds(rssFeeds)
  5.     writeFeeds(rssFeeds)
  6.   }
  7.  

One piece of syntax that is especially confusing in Scala is how it indicates that one type contains another, in this case Array[String]. This syntax looks a lot like what you'll see for array indexing in most other languages. In Scala arrays are treated as functions that receive an integer argument to return their values, eg args(0) returns the first argument of args. This is something of a simplification, but it helps me remember what the syntax is.

Anonymous Functions, Maps and Folds

  1.   def getRssFeeds(fileName : String) : List[(String, Seq[String])] = {
  2.     // Given a config file containing a list of rss feed URL's,
  3.     // returns a list with ('channel name', ['article 1', 'article 2'])
  4.     var baseList : List[(String, Seq[String])] = List()
  5.     getFileLines(fileName).map(extractRss).foldLeft(baseList)
  6.         { (l, r) => l ::: r.toList }
  7.   }
  8.  

Anonymous Functions

The function on line 4, (l, r) => l ::: r.toList is a case of an anonymous function in Scala, I'll break it down: 1. (l, r) =>: The arguments section, if I wanted to be pedantic I could specify their type, but anonymous functions already clutter code and their purpose should usually be pretty obvious. 2. l ::: r.toList: l is an already existing list object which is being prepended to r after r is turned into a list.

Higher-order functions

Scala's for syntax is extraordinarily powerful, but sometimes it's more than you need. The three most common higher-order functions are:

  • map: Apply a function to each element in a sequence.
  • filter: Use a boolean function to filter the elements of a sequence.
  • fold(generally split into foldLeft and foldRight): Use a 2-arity function to combine all of the elements in a sequence into 1 element.

The for syntax combines the functionality of map and filter, so is most useful when you need to do both to a sequence. It does not cover the functionality of fold. Here I'm:
1. Getting a sequence which contains the URL's of several RSS feeds. 2. Mapping a extractRss over them to get the RSS data. 3. Folding the anonymous function (l, r) => l ::: r.toList over them to change them from immutable Seq values to Lists and combine them.

Filter and Some Sugar

  1.   def combineFeeds(feeds : List[(String, Seq[String])])
  2.       : List[(String, Seq[String])] = {
  3.      // Cleanup function, given a list of feeds it combines duplicate channels.
  4.     def combinedFeed(feedName : String) : Seq[String] = {
  5.       feeds.filter(_._1 == feedName).map(_._2).flatten
  6.     }
  7.  
  8.     val feedNames = feeds.map(_._1).distinct
  9.  
  10.     feedNames.map(x => (x, combinedFeed(x)))
  11.   }
  12.  

The filter function in combineFeed makes use of the fact that it is in combineFeeds's namespace and doesn't need to have feeds passed to it. It filters through feeds, keeping those with the same name. It then passes the result to map which extracts the article lists, and flatten combines each list. Generally you would either pass a named function with a single argument to filter and map, or an anonymous function with the format (x) => x == feedName. This pattern comes up so often that Scala allows us to ignore the usual anonymous function syntax and use the underscore as a placeholder for a single variable.

Brackets are Optional in Single Liners

  1.   def getFileLines(fileName : String) : Array[String] =
  2.     // Extracts the URL's from a file and breaks them up into individual strings.
  3.     scala.io.Source.fromFile(fileName).mkString.split("\n")
  4.  

Opening Network Connections and Reading XML

  1.   def extractRss(urlStr : String) : Seq[(String, Seq[String])] = {
  2.     // Given a URL, returns a Seq of RSS data in the form:
  3.     // ("channel", ["article 1", "article 2"])
  4.     val url = new URL(urlStr)
  5.     val conn = url.openConnection
  6.     val xml = XML.load(conn.getInputStream)
  7.  
  8.     for (channel <- xml \\ "channel")
  9.       yield {
  10.         val channelTitle = (channel \ "title").text
  11.         val titles = extractRssTitles(channel)
  12.         (channelTitle, titles)
  13.     }
  14.   }
  15.  

Opening Network Connections

For practical purposes Lines 2-4 are straight Java. They format a URL, open a connection, download the stream and convert it into the XML library's internal format.

Reading the XML

Line 6 is where I start making good on my promise of easy XML processing. xml \\ "channel" descends through the XML tree and extracts every element named "channel", we then use the for expression to take each of those elements and treat them each as XML trees in their own right. The \\ will descend through the entire tree, while \ only examines those nodes that are immediately under the given root.

  1.   def extractRssTitles(channel : Node) : Seq[String] = {
  2.     // Given a channel node from an RSS feed, returns all of the article names
  3.     for (title <- (channel \\ "item") \\ "title") yield title.text
  4.   }
  5.  

In extractRssTitles you can see how \\ can be nested to do more complex dives into an XML document.

Display

To Screen

  1.   def printFeeds(feeds : List[(String, Seq[String])]) : List[Seq[Unit]] = {
  2.     // Given a list of ("channel", ["article 1", "article 2"]), prints
  3.     //  them each to screen.
  4.     for (feed <- feeds) yield {
  5.           println("*** " + feed._1 + " ***")
  6.           for (title <- feed._2) yield println("\t" + title)
  7.     }
  8.   }
  9.  

printFeeds takes a list of the feed objects and displays them on screen. There are a couple of ways to extract data from a tuple. If you're dealing with a complex set of structures, you can use match/case. I knew I'd always be dealing with a 2-tuple, so I chose to use the _# syntax, where _1 will give you the first value in the tuple, etc.

To HTML

  1.   def writeFeeds(feeds : List[(String, Seq[String])]) = {
  2.     // Given a list of ("channel", ["article 1", "article 2"]), generates
  3.     //  and writes an HTML document listing the articles with channel names
  4.     //  as headers.
  5.     val html =
  6.       <html><title>Rss Feeds</title><body>
  7.         {for (feed <- feeds) yield
  8.           <h2>{feed._1}</h2>
  9.           <ul>
  10.             {for (title <- feed._2) yield <li>{title}</li>}
  11.           </ul>}
  12.         </body> </html>
  13.    
  14.     XML.saveFull("rss.html", html, "UTF-8", true, null)
  15.   }
  16. }
  17.  

When you produce XML in most languages you have to choose between using hard to read code that produces guaranteed well formed XML, or generating your XML out of strings that are easy to read but you can't be sure is well formed.

This is what has had me excessively excited about Scala for the past week. It seems that the majority of projects I get involved in sooner or later involve doing some kind of work with XML, and it's usually one of the ugliest parts of the job. Here you can see I have XML freely interspersed with the rest of my code, and was by far the easies part of what I wrote this week. At any time inside the XML, if I need to break back out into Scala I just surround the statement I want to use in curly braces. With it I've been able to clearly produce XML that I can be confident is well formed.

My Experience

Arrays

An unpleasant surprise I got while I was working on this program was that, unlike other sequence types, Arrays in Scala have a simple way to convert to Lists.

Functional Programming

Reading over my own code, I realize that functional style has become a habit. It'll be interesting to see how well I cope when I try to pick up a language where functional style isn't a realistic option. To anyone reading, if there are places in my Scala code where it would be more clear if I used a more object oriented style, please let me know.

XML

One more time. The XML processing capabilities of this language are beautiful. Scala is going to end up being part of my standard arsenal for this reason alone.

Resources:

Filed under: rss, scala, xml No Comments
17Jan/110

Prolog 3/4: XML Parsing

Posted by Frank Berthold

Reading, Parsing and Displaying RSS Data in Prolog

This is the third article in a series, to read it from the beginning go here.

The Plan

This week is when I really put the assertion that Prolog is a general purpose language to the test. Most decently rational Prolog programmers will tell you to do things like XML parsing and display through another language, then use Prolog dll's to do the interesting logical stuff. But, it does have the libraries, so let's see how they work.

What to Expect

When this article is complete:

  • You will have:
    • A command-line RSS aggregator.
  • You will know:
    • How to read a text file.
    • How to read from an HTTP stream.
    • How to parse an XML file.
    • How to pretty print text to the screen.

Files Used in this Project

Compiling

To compile "xml.pl", navigate to the directory that contains it on the commandline, then execute:

  1. swipl -g main -o xml.exe -c xml.pl
  2.  

The Code

Libraries Used

  1. :- consult(library(sgml)).
  2. :- consult(library(http/http_open)).
  3.  
  • sgml: The SGML library, which also handles XML.
  • http/http_open: The HTTP library handles server and client code, I just used the small bit that let's me treat an HTTP stream like a file.

Reading a File

  1. readConfig(Lines) :-
  2.   open('feeds.txt', read, ConfigStream),
  3.   read_stream(ConfigStream, Lines).
  4.  
  5. read_stream(ReadStream, Lines) :-
  6.   read_line_to_codes(ReadStream, Line),
  7.   read_stream(ReadStream, Line, Lines).
  8. read_stream(ReadStream, end_of_file, []) :- close(ReadStream), !.
  9. read_stream(ReadStream, OldLine, [Atom | Lines]) :-
  10.   read_line_to_codes(ReadStream, NewLine),
  11.   string_to_atom(OldLine, Atom),
  12.   read_stream(ReadStream, NewLine, Lines).
  13.  

read_stream is a helper function to read multiple lines from a stream. It reads a stream in one line a time and returns/unifies its contents to Lines. Most of it is similar to what you saw in the previous article, recursively pulling off one line at a time, testing if it's the last line and adding it to the list. The two key elements that are worth noting are: the use of the atom end_of_file, which is what the file stream returns when the stream is empty and the use of ! to indicate that if execution makes it to closing the stream, then there's no need to backtrack.

readConfig's functionality is pretty obvious. Open a file using an atom with the file's name. Remember, single-quotes are for atoms, double quotes are for strings. This opens a read only stream and unifies it with ConfigStream, then read_stream pulls the text in the file out into Lines.

Reading From HTTP

  1. rssFetch(URL, XML) :-
  2.   http_open(URL, XmlStream, []),
  3.   load_xml_file(XmlStream, XML),
  4.   close(XmlStream).
  5.  

The pattern here is similar to the one for readConfig, except that it extracts the XmlStream straight to a Prolog XML structure. There's quite a bit to it, but the short version is, it's a nested linked list. There are a variety of functors in that list, but the one that interests us is element which has the form element(<element name>, <attributes>, <sub elements>). For example when given the xml:

  1. <testOuter>
  2.   <inside inAtt="hello">in text</inside>
  3. </testOuter>
  4.  

load_xml_file returns:

  1. [element(testOuter, [],
  2.     ['\n  ', element(inside, [inAtt=hello], ['in text']), '\n'])].
  3.  

Parsing XML

  1. rssRead([], []).
  2. rssRead([element(channel, _, Elements) | _], [Rss | Rsses]) :-
  3.   channelRead(Elements, Rss), rssRead(Elements, Rsses).
  4. rssRead([element(rss, _, Elements) | _], Rss) :-
  5.   rssRead(Elements, Rss).
  6. rssRead([_ | Elements], Rss) :-
  7.   rssRead(Elements, Rss).
  8.  
  9. channelRead(Elements, channel(Name, Titles)) :-
  10.   titleRead(Elements, Name), itemsRead(Elements, Titles).
  11.  
  12. itemsRead([], []).
  13. itemsRead([element(item, _, Elements) | Items], [ Title | Titles ]) :-
  14.   titleRead(Elements, Title), itemsRead(Items, Titles).
  15. itemsRead([_ | Items], Titles) :-
  16.   itemsRead(Items, Titles).
  17.  
  18. titleRead([element(title, _, [Text | _]) | _], Text) :- !.
  19. titleRead([_ | Elements], Text) :-
  20.   titleRead(Elements, Text).
  21.  

Strictly speaking Prolog is a weakly typed language, you have only a couple of basic types which all reduce to atoms and functors. But with those you can do the same tricks that are generally associated to algebraic type systems. Here is one example where I use the element functor to recurse through the XML structure and extract the titles.

Once you know how Prolog's XML structures work, it's just a matter of figuring out what information you need and recursing until you get it. For this purpose I've created my own functor channel(<channel name>, <title list>).

In Prolog functors and atoms are created simply by using them. This makes writing the code feel a lot like freeform sketching a picture. I like it. But I'm pretty sure that unless I made an effort to document my code well, it would make large projects unwieldy.

Pretty Printing Text

  1. displayRss(Channels) :-
  2.   foreach(member(channel(Name, Titles), Channels),
  3.       (writef("*** %w ***\n", [Name]), displayTitles(Titles), nl)).
  4.  
  5. displayTitles(Titles) :-
  6.   foreach(member(Title, Titles), writef("\t%w\n", [Title])).
  7.  

writef works similarly to printf in nearly every language I've worked in. It has a few quirks that are detailed in its docs.

Main And Helper Text

  1. readAndDisplayRss(URL) :-
  2.   catch((rssFetch(URL, XML),
  3.         rssRead(XML, RSS),
  4.         displayRss(RSS)), _, readAndDisplayRss(URL)).
  5.  
  6. main :-
  7.   readConfig(FeedURLs),
  8.   foreach(member(URL, FeedURLs), readAndDisplayRss(URL)),
  9.   halt.  
  10.  

The above is glue code. It's been written to continue retrying until it succeeds or runs out of stack, so make sure your URL's are valid.

Final Summary

When I was reading up on Prolog, I heard predicates described as a generalization on functions. This week has driven that one home. Working in Prolog has felt a great deal like when I was first learning Haskell. I'll spend hours fighting with something that I could do in my sleep in another language, then when I figure it out it is extraordinarily clear, obvious and feels cleaner than anything else I've worked with.

Hard Lessons Learned

foreach is not the same as a for comprehension/list comprehension, no matter how much it looks like one. The key difference is that the second term in the statement does not contain its own namespace. What happens in there happens throughout the same namespace that contains the foreach. For example:

  1. main :-
  2.   foreach(member(Number, [1, 2, 3, 4]),
  3.       (Square is Number * Number,
  4.        write(Square), nl)).
  5.  

Does not print the square of the numbers 1 through 4. It first asserts that Square is equal to 1 * 1, then that it is also equal to 2 * 2, 3 * 3, and 4 * 4, which is clearly false and ends execution of the foreach. It is possible to do this, the safe and idiomatic way is to break the second term out into its own predicate:

  1. showSquare(Number) :-
  2.   Square is Number * Number,
  3.   write(Square), nl.
  4.  
  5. main :-
  6.   foreach(member(Number, [1, 2, 3, 4]), showSquare(Number)).
  7.  

That lesson took me half a day for me to learn. You're welcome.

Coming Up

Next week I'll be working out how to use XPCE, Prolog's native GUI library.

Resources

Filed under: prolog, rss, xml No Comments