PyQuery-based scraping micro-framework. Supports Python 2.x and 3.x.
Source code: https://github.com/matiasb/demiurge/
- Added docs.
- Added RelatedItem.
- Added field clean support.
- Initial release.
pip install demiurge
You can define items to be scraped using a declarative (Django-inspired) syntax:
import demiurge class Torrent(demiurge.Item): url = demiurge.AttributeValueField( selector='td:eq(2) a:eq(1)', attr='href') name = demiurge.TextField(selector='td:eq(2) a:eq(2)') size = demiurge.TextField(selector='td:eq(3)') class Meta: selector = 'table.maintable:gt(0) tr:gt(0)' base_url = 'http://www.mininova.org'
At the moment, there are only two available fields: TextField and AttributeValueField. A TextField expects an optional selector argument, meanwhile AttributeValueField possible arguments are selector and attr.
selector specifies the PyQuery selector for the element, relative to the Item element (determined by the Meta selector attribute). If not specified, the current Item element is assumed.
On the other hand, attr parameter allows to retrieve an element attribute value instead of its text content.
In the example above, the Item selector is any row but not the first one, from the table with CSS class maintable (also ignoring the first ocurrence, sponsored results).
Each field selector is relative to the Item element (in this case, a table row). Then, name refers to the second anchor in the second cell.
New in version 0.2: Added clean_<fieldname> method support.
If you need an extra cleanup or processing for a field (for instance, to apply a regex), you can define a method clean_<fieldname> in your class. This method will receive the retrieved value for the field and should return the cleaned one. For example:
class Torrent(demiurge.Item): ... size = demiurge.TextField(selector='td:eq(3)') def clean_size(self, value): return value.replace('Mb', 'MB')
New in version dev: Added coerce parameter to fields.
You can also specify a coerce function as argument when defining a field. This function will be applied to the cleaned value (also after a clean_<fieldname> call). For example, if value needed to be an int, you could do:
value = demiurge.TextField(coerce=int)
to use Python’s built-in int and get an integer value.
Once you defined your items, there are a couple of useful methods you can use, both expecting as argument a relative path to the Item base_url if it was defined in the Item.Meta class, or a full URL (if base_url was not specified):
>>> t = Torrent.one('/search/ubuntu/seeds') >>> t.name 'Ubuntu 7.10 Desktop Live CD' >>> t.size u'695.81\xa0MB' >>> t.url '/get/1053846' >>> t.html u'<td>19\xa0Dec\xa007</td><td><a href="/cat/7">Software</a></td><td>...' >>> results = Torrent.all('/search/ubuntu/seeds') >>> len(results) 116 >>> for t in results[:3]: ... print t.name, t.size ... Ubuntu 7.10 Desktop Live CD 695.81 MB Super Ubuntu 2008.09 - VMware image 871.95 MB Portable Ubuntu 9.10 for Windows 559.78 MB ...
Any extra attributes defined in the Item.Meta class will be passed to PyQuery when doing the URL request (i.e. you could add, for example, encoding or method; if python-requests is available, there is a bunch of extra parameters you could use, such as: auth, data, headers, verify, cert, config, hooks, proxies).
Alternatively, there is an all_from method that will retrieve all items from a PyQuery object created from the given arguments (i.e. it will directly pass all specified parameters to PyQuery and scrap items from there).
Plato, as the speaker Timaeus, refers to the Demiurge frequently in the Socratic dialogue Timaeus, c. 360 BC. The main character refers to the Demiurge as the entity who “fashioned and shaped” the material world. Timaeus describes the Demiurge as unreservedly benevolent, and hence desirous of a world as good as possible. The world remains imperfect, however, because the Demiurge created the world out of a chaotic, indeterminate non-being.