Yes I noticed the timeouts in the logs. I'm going to look into that.
Arc language forum wasn't in the feeds I crawled (I should have checked this one term, d'uh!), so the program doesn't know anything about, hence the 'this may suck' message.
Peace: I'm trying to be smarter than just matching on keywords. That didn't pay off here.
with google.com/finance, it tried to find finance news. It's smart enough to know that that URL isn't about google.. will you try that station again, vote on a few stories, tell me what you thought?
Will you try out a few more blog/site titles? I'd love to get the coverage of my crawler stress-tested like this :) Thanks for trying it out like this.
