Components
Unobtanium is made up of:
unobtanium(lib-unobtanium)- Main application library that implements most data-structures and database logic.
criterium- Query framework for matching Data in memory and in the DB against criteria.
- API Documentation
unobtanium-crawler- Web Crawling and summarizing application of Unobtanium.
- Manual
unobtanium-viewer- Web frontend for querying an Unobtanium summary database.
- Manual
- crawler database
- Database schema optimized for crawling.
- Schema Documentation
- summary database
- Database schema optimized for querying/searching.
- Schema Documentation
Data Pipeline
The main Unobtanium data pipeline consists of three steps:
- Crawling (Web to crawler database)
- Summarizing (crawler database to summary database)
- Querying/Searching (summary database to curious Creature) (Query Syntax)
Each step in independent of the previous, so no huge setup is needed to get Unobtanium working.
Crawling and summarizing are decoupled to make iterating on code and configuration easier, as summarizing is a quite complex step.