Changes

Spiders, in this context, are things that index the web. So you might also call them indexers. Once you index the web, there are a lot of interesting things you can do. One of the use cases is to "scrape" data. Technologies such as [[Search]] rely on indexing.

A long time ago I wrote a spider. If I ever get around to digging up that old code, here is where I might find it. But that was old school, and not really useful for anything non-trivial these days. Lots of other people are making interesting spiders that you can use. [http://scrapy.org/ Scrapy] is a Python spider. There is a visual frontend, [https://github.com/scrapinghub/portia Portia], from the folks over in Cork, Ireland at [http://scrapinghub.com/ Scrapinghub].

== Use Case ==I want to scrape questions and answers from OKCupid. The OKC content is dynamically generated on the client-side (browser) using JavaScript. This means we are NOT dealing with a "rendered" page to scrape, but rather their application sends a bunch of JavaScript to the browser (client). The user interacts via clicks to instruct the JavaScript to perform actions and render new results. Thus we need a browser equivalent that can speak JavaScript and can load additional JavaScript for execution on the "page". Portia doesn't do this. Instead, there are a couple options to integrate with Portia or to modify the middleware<ref>[http://doc.scrapy.org/en/latest/topics/spider-middleware.html Middleware component of Scrapy]</ref> of Portia.  Even the OKC [http://pastebin.com/7B1TgUB9 login] page is scripted.  [https://github.com/scrapinghub/portia Portiasplash Splash] is one example from the folks over a JavaScript rendering service implemented in Cork, Ireland at Python using Twisted and QT. Splash can be [http://scrapinghubsplash.readthedocs.comorg/en/latest/ Scrapinghubscripting-tutorial.html scripted]. I won't repeat their documentation here needlesslySo, but I will note my experiences using Splash with the tools. I wanted Portia, we should be able to visually scrape questions and answers from OKCupid, but so far Portia can't handle the JavaScript login. I need to deconstruct it more to find out what the solution might beOKC. {{References}}