Unobtanium Documentation

Your first search engine

2026-02-01T00:00:00+00:00

This is a step by step guide on setting up your first unobtanium search instance from installing the dependencies to getting first search results.

This guide assumes you are running on Linux and know how to navigate on the terminal.

!note: In case you get stuck: You getting stuck while following this guide really shouldn't happen, if you do get stuck please open on issue on codeberg.org/unobtanium/unobtanium-documentation</a>.

!note: Resource requirements: This tutorial requires almost 2GB of disk space, make sure you have that much free.

Installing dependencies</h2>
You need the following packages:

git - The version control system</li>
rustc - The Rust compiler (sometimes just called rust)</li>
cargo - The Rust build system</li>
SQLite development files - The SQLite database</li>
OpenSSL development files</li>
A text editor for editing files</li>
A web-browser for viewing the final result</li> </ul>
These packages don't always have the same names, but they should be available for every Linux distribution. Operating systems other than Linux are currently not supported, consider running this inside a Linux virtual machine.
On alpine Linux:
`apk add git rust cargo sqlite-dev openssl-dev </code></pre> On Debian trixie:`
`apt install git rustc cargo libsqlite3-dev libssl-dev </code></pre> On Void-Linux:`
`xbps-install git rust cargo sqlite-devel openssl-devel </code></pre>`
Setting up</h2>
To set up create a folder `my-first-unobtanium</code>, everything in this tutorial will happen inside it. (The exact name isn't important, but this tutorial is going to reference it a lot)`
`Now to set up run the following commands:`
# Navigate to the folder you just created cd my-first-unobtanium # Download the sourcecode using git git clone https://codeberg.org/unobtanium/unobtanium # Navigate inside the sourcecode cd unobtanium # Use a known working version of unobtanium # that doesn't require extra steps git checkout v3.0.0 # Git will complain about something it calls 'detached HEAD' state. # This is okay since we won't be doing any development. # Run the rust compiler to build release optimized versions # This will take a while ... cargo build --release # Create a folder outside the source code # where we can put the resulting binaries mkdir ../bin # Copy the binaries to the bin folder we just created cp target/release/unobtanium-viewer ../bin/ cp target/release/unobtanium-crawler ../bin/ # Free up some space cargo clean # Back to the my-first-unobtanium folder cd .. # Tell your shell that there are additional commands # in the bin folder so you can use them by typing their name. # This is not permanent: # If you come back later remember to repeat this step. export PATH="$PATH:$PWD/bin/" # Make sure the crawler is there unobtanium-crawler --help # Make sure the viewer is there unobtanium-viewer --help </code></pre> You now have an environment that will work for the rest of the tutorial.
Your first crawl</h2>
To search something you need an index, to build an index you need raw data.
In this step we will:

Create a first configuration file for the crawler</li>
Run the crawler to get data from the web</li>
Summarise the data to get a searchable index</li> </ul>
Creating a configuration file</h3>
Inside the `my-first-unobtanium</code> folder create a text file example_config.toml</code>:`
# This is just a human readable name name = "Unobtanium example index" # Wait one second between each requst to the same URL origin default_delay_ms = 1000 # The number of requests to attempt when running the crawler command once. max_commands_per_run = 100 # Only crawl pages that haven't been crawled within a week recrawl_interval = "1 week" # The http `User-Agent`, in this case a placeholder for the tutorial user_agent = "unobtanium-tutorial-crawler" # The entry points of the sites that unobtanium shoudl crawl. seeds = [ "https://doc.unobtanium.rocks/", "https://slatecave.net/", ] </code></pre> Please don't change the file for now, you can mix it up after you've gotten it working. I know you're curious. Running the crawler</h3> This step will collect the crawl database from the web. !note: Crawl Database The crawl database contains raw web pages along with information on when and how they were fetched, other search engines call this their "repository". Back in the Terminal, inside the my-first-unobtanium</code> folder run: unobtanium-crawler crawl \ --config example_config.toml \ --database example_crawl.db </code></pre> This will run for about a minute and will collect slightly less than 100 documents from doc.unobtanium.rocks</code> and slatecave.net</code> in what is roughly a 50:50 split. Running the crawler command again will fetch another (almost) 100 documents. !note: Interrupting the crawler: You can stop the crawler like any other well behaved command line program through Ctrl</key>+C</key></key>. When you start it again, it will continue where it left off. The rest of this section is explanation of what is going on. To break the command down: unobtanium-crawl</code> is the unobtanium crawler, it is your go to multitool that implements almost everything of unobtanium that isn't part of the search interface.</li> crawl</code> is a subcommand to tell it to crawl websites</li> --config example_config.toml</code> tells it where the configuration file for crawling is, here it is the example_config.toml</code></li> --database example_crawl.db</code> tells it to create the file example_crawl.db</code> and use it as the database to store the crawling results in.</li> </ul> While the crawler is running you can observe it doing a number of things: Initialising it's database</li> Running the crawler tasks ("Crawl loop") fetching robots.txt for …</code> the crawler always fetches the robots.txt</code> of a website first so it doesn't go where it isn't welcome.</li> [ file_ingested ]</code> The crawler successfully downloaded a document and stored it</li> Scheduling …</code> This is the crawler telling you what it plans to do next, everything except the seed URLs needs to be scheduled before being crawled.</li> [ file_of_unknown_type ] …</code> The crawler tried to download a file, but then found out that it doesn't know the format. This happens every few requests and usually is nothing to worry about.</li> [ redirect ] …</code> This is the same ay your browser being redirected, the crawler notes those redirects and will at some point schedule the URL the redirect pointed it to.</li> [ blocked_at_request_of_remote ] …</code> looks scary, but isn't. This is a page requesting not to be indexed and the crawler just told you that it respected that.</li> Ran out of crawl commands on …</code> at some point the limit on requests ("crawl commands") you set in the configuration file will be reached.</li> </ul> </li> [Crawl_Statistics_Report]</code> This is some nice numbers so you know what the crawler did while you weren't looking. out_of_patience_origins</code> in case any origins (from the seeds</code> in the configuration file) throws so many errors that the crawler gives up they'll be listed there. For you this should just be an empty list.</li> total_requests</code> this is the total number of requests that were sent.</li> ingested_files</code> how many files you now have ready for building your search index.</li> </ul> </li> Optimizing and vacuuming the database ...</code> this is the step where the database runs some heavy optimisation and cleanup, for larger databases this usually takes a while, for this tutorial it shouldn't take much longer than a second.</li> </ul> Running the summarizer</h2> The crawler gave you raw data from the web, which is as searchable as a pile of random papers someone dumped on your desk without explanation. The summarizer takes this pile of pages and generates the summary database. !note: Summary Database: The summary database is the search index of unobtanium and contains data and metadata in a way that is easily searchable. The summarizer is also built into the crawler, you can run it with the following command in the my-first-unobtanium</code> folder: unobtanium-crawler summarize \ --crawler-db example_crawl.db \ --summary-db example_summary.db </code></pre> This will run for a few seconds and generate the file example_summary.db</code>. !note: Interrupting the summarizer: Like the crawler the summarizer can be interrupted through Ctrl</key>+C</key></key>. It will also resume where it was interrupted. The rest of this section is explanation of what is going on. To break down the command: unobtanium-crawler</code> is again the crawler binary that all the tools are built into</li> summarize</code> is the subcommand that starts the summarizer.</li> --crawler-db example_crawl.db</code> tells the summarizer where to find the data that the crawler collected.</li> --summary-db example_summary.db</code> tells the summarizer to put its output in the file example_summary.db</code>.</li> </ul> The summarizer will do a few things: Initialise its database</li> Summarizing Page 1 …</code> This is the summarization loop running batches of up to 1000 documents called pages. Found 0 self duplicates.</code> Self duplicates are pages where the metadata from the server indicated that they have changed, but unobtanium detected that their main constant didn't change. On the first run there of course aren't any of these</li> Found … exact duplicates.</code> Sometimes the same document is available on two different addresses when unobtanium notices this it picks one of the addresses as the canonical one and flags the others as "exact duplicates".</li> </ul> </li> (Re)generating full text index ...</code> in unobtanium 3.0.0 this rebuilds the part of the index that provides the actual search engine where query words go in and results come out.</li> Optimizing and vacuuming the summary database ...</code> Cleaning up the database again, this will take longer the larger the database grows, but for the tutorial this should not take much longer than a second.</li> </ul> Running the search frontend</h2> Now that you have an index you want to search it, you can start it using the following command in the my-first-unobtanium</code> folder: unobtanium-viewer \ --summary-db example_summary.db \ --template-location unobtanium/viewer/templates/ </code></pre> You should see it starting up some search workers, starting the templating engine and then telling you Web interface on: 127.0.0.1:3000</code>. You can now open http://127.0.0.1:3000</a> in your local web-browser and you'll be greeted by a search box. !note: Access from the local Network: In case you want/need to open the search engine for your local network pass in --listen 0.0.0.0:3000</code>. In a real deployment on the internet you want to use a reverse proxy. Some queries for you to try: unobtanium</code></li> slatians hideout</code></li> sql</code></li> your first search engine</code></li> </ul> You can stop the viewer with Ctrl</key>+C</key></key>. File overview</h2> An overview of all files created in this guide: 📁 my-first-unobtanium</code> 📁 unobtanium</code> The unobtanium source code from Codeberg</li> </ul> </li> 📁 bin</code> ⚙️ unobtanium-crawler</code></li> ⚙️ unobtanium-viewer</code></li> </ul> </li> 📄 example_config.toml</code> - Configuration file</li> 📚 example_crawl.db</code> - Crawler database</li> 📚 example_summary.db</code> - Summary database</li> </ul> </li> </ul> What now?</h2> Congratulations, you now have a working search engine! To build this from a tutorial-example into a real search engine, whether it be on your own network or for the internet the next steps are: Change the user_agent</code> in the configuration file (User-Agent manual</a>)</li> Raise the max_commands_per_run</code> limit (multiplying by 10 for each step works well)</li> Have a look at the crawler configuration manual</a> to know what is possible.</li> Add the shared policies file</a> to your crawler by downloading it and adding the --policy-file</code> option</a> to your crawler configuration.</li> Add your own sites to the seeds</code> list.</li> Rerun crawling and summarizing, if the database grow too large you can always throw them away and start over.</li> See the selfhosting guide</a></li> Put unobtanium on a server behind a reverse proxy.</li> </ul>



Text Pile
2025-11-28T00:00:00+00:00
The text pile as a concept is that the search engine stores text independent of where it found that text so that it can reuse the storage instace in case of a page being reachable via more than one address (exact duplicates</a>).</p>
The text piles store the plain text from the pages it was scraped from in a cleaned up form along with some metadata about the semantics of the text and information directly derived from the text.</p>
Implementations</h2>

Text Pile Generation 1 ("Legacy")</a></li>
Text Pile Generation 2 ("NG")</a></li>
</ul>



Text Pile Gen2
2025-11-28T00:00:00+00:00
The Text Pile Gen2</abbr> (also known as Text Pile v2 or "NG") is a datastructure for storing text content along with limited formatting and optional segmenting information. It is extracted from pages during the summarizing step</a>.</p>
It is implemented in the unobtanium-text-pile</code> crate</a>.</p>
!note: This replaced the Text Pile Gen1:</b> The Text Pile</a> was used in unobtanium release 3.0.0 and older in place of the datastructure described on this page.</p>
The Text Pile NG stores the following data:</p>

The raw page text in a cleaned up form</li>
Metadata:

Semantic markers (roughly: HTML tag semantics)</li>
The language indicated in the document</li>
</ul>
</li>
Optional segmentation information ("segmentation cache"):

Segment lengths</li>
A relevance marker for each segment</li>
Sentence lengths</li>
The language guessed by the segmenter</li>
</ul>
</li>
</ul>
The segmentation information is intended to be updated along with the segmentation algorithm.</p>
Digest</h2>
The digest of the Text Pile Ng is calculated using the Blake2b</a> 512 bit algorithm. It is used for detection of exact duplicates</a>.</p>
The Digest is calculated across the shortest possible postcard serialization</a> of the text and metadata sections of the text pile ng.</p>
The segmentation cache is not</strong> included as it is derived data and may change over time to allow the segmentation algorithm to be updated.</p>
Comparison with the previous Text Pile</h2>
The old Text Pile</a> was optimized for use with the sqlite fts5 extension and had multiple strings, depending on the semantics the text was appended to one of them.</p>
This had the following drawbacks:</p>

It couldn't recall the original text in original order for previews</li>
No segmentation cache means running expensive segmentation algorithm over and over</li>
No preservation of language metadata, which is important for segmentation</li>
Only limited preservation of document semantics</li>
A lot of duplicate storage as text could be appended to multiple fields</li>
</ul>
See Also</h2>

Initial tracking issue "TextPile v2 format (TextPileNg) #37"</a></li>
The unobtanium-text-pile</code> on docs.rs</a></li>
The unobtanium-text-pile</code> on Codeberg</a></li>
</ul>


Text Pile (Disambiguation)
2025-11-28T00:00:00+00:00
You are probably here because you followed an old link or Bookmark.</p>
Unobtanium now supports multiple text piles:</p>

Text Pile Generation 1 ("Legacy")</a></li>
Text Pile Generation 2 ("NG")</a></li>
</ul>
There is also a concept page for text piles in general</a>.</p>
Note for documentation editors:</b> Do not link to this page, prefer the concept page instead.</p>


Selfhosting Unobtanium
2025-08-15T00:00:00+00:00
Warning:</b> This guide is incomplete, see the Your first search engine guide</a> instead.</p>
I'll assume you already know a few things:</p>

Linux administration in general</li>
Configuring a web reverse proxy</li>
HTML</li>
SQL</li>
TOML</li>
How to use cargo build</code> to build your own binaries</li>
Shell scripting (for your own convenience)</li>
</ul>
If you haven't yet please have a look at the overview page</a>.</p>
Resource requirements</h2>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
		
	
		
			
			
		
	
		
			
		
	
		
			
			
		
	
		
			
			
				
			
		
	




	
		
		CPU
		</dt>
	

	
		

		No special requirements, it should have at least 2 cores.
		</dd>
	

	
		

		RAM
		</dt>
	

	
		

		Plan at least 2GB for unobtanium alone, this includes some buffer for caching and making sure nothing runs out of memory. In practice its very likely you'll use less, but not having the RAM free for the Kernel to use as cache will noticeably slow down searching.
		</dd>
	

	
		

		Disk
		</dt>
	

	
		

		For a small index of one medium size blog 100MB shoukd be enough.
		</dd>
	

	
		
		For large deployments plan ~10GB per 100K searchable pages for the crawler database and another 6GB per 100K pages for one summary database (in practice you'll probably want two of them to get new data in without downtime).
		</dd>
	

</dl>

What you need to do, overview</h2>
Bootstrapping a search engine:</p>

Writing and testing an initial crawler configuration</li>
Expanding the configuration</li>
Doing a full initial crawl and summary</li>
Setting up the frontend</li>
</ol>
Maintainence:</p>

Semi regular recrawls</li>
Updating the crawler configuration</li>
Keeping unobtanium updated</li>
</ul>
Your first crawler configuration</h2>
Have the crawler crawl configuration manual</a> ready.</p>
This file will define where the crawler is allowed to collect pages from.</p>
You may find the configuration of unobtanium.rocks</a> useful. Use the shared policies file</a> with the --policy-file</code> option, it avoids some common parts of websites that the crawler should avoid.</p>
!note: Hint:</b> Start with a small configuration of one to three seeds and a crawler command limit of 1000. This will quickly give you a feel for how the crawler behaves</a> and what it tries to collect.</p>
After the first successful crawl try running the summarizer</a> and the pointing a locally running viewer</a> at the resulting summary database. Try typing in some keywords you'd expect to find results for.</p>
Once that works you can expand the crawling configuration and raise the crawler command limit.</p>


Crawl Data Tree
2025-08-14T00:00:00+00:00
The crawl data tree is a construct in the crawler database</a> sourrounding the crawl log</a>.</p>
Each crawl log entry may</strong> have:</p>

one mandatory crawl log entry</a></li>
multiple requests</li>
multiple files

one file text</li>
</ul>
</li>
multiple redirects</li>
</ul>
Files and redirects reference the requst they came from if they were derived from a network request.</p>


Token Index
2025-08-14T00:00:00+00:00
The token index is an experimental way to store a full text index in SQL tables in the summary database</a>.</p>
A token in the sense of the token index is rouhly equivalent to a word after splitting up compound words</a> and applying normalization.</p>
In its current state there is a statistics table that keeps track of how often a token appears in a given text pile</a>.</p>
Open Problems</h2>
At least the following problems need to be solved for the token index to exit its experimental state:</p>

Page previews must be generated ouside of fts5. See pull request #36</a>.</li>
It must support at least BM25 or another result weighting mechanism.</li>
</ul>
How to access the token index</h2>
The index being experimental isn't built by default and can be (re)build using unobtanium-crawler regenerate-token</code> command</a>.</p>
It can be queried using the token:</code> filter</a>.</p>


Crawler Database
2025-08-13T00:00:00+00:00
The crawler database schema is implemented on top of the base database schema</a> and mainly holds data surrounding the crawl log</a> (see also: crawl data tree</a>) and the crawl candidates.</p>
Overview</h2>
Tables in the crawler database are:</p>




	




	
		
		agent</code>
		</dt>
	

	
		

		Information about crawling agents
		</dd>
	

	

	
		

		crawl_log</code>
		</dt>
	

	
		
		Crawl log entries</a>
		</dd>
	

	

	
		

		request</code>
		</dt>
	

	
		

		Requests that belong to a crawl log entry
		</dd>
	

	

	
		

		file</code>
		</dt>
	

	
		

		File metadata obtained from a request
		</dd>
	

	

	
		

		file_text</code>
		</dt>
	

	
		

		Text content belonging to an entry in the file</code> table
		</dd>
	

	

	
		

		redirect</code>
		</dt>
	

	
		

		Redirect resulting from a request
		</dd>
	

	

	
		

		crawl_candidate</code>
		</dt>
	

	
		

		A URL that was discovered in a context that makes it a potential crawling target
		</dd>
	

</dl>

Tables</h2>
agent</code></h3>
The agent</code> table has the following fields:</p>




	




	
		
		agent_id</code>
		</dt>
	

	
		
		Text</a> Primary Key</span>
		</dd>
	

	

	
		

		time_started_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a>  When the agent started its work.
		</dd>
	

	

	
		

		time_finished_unitx_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a> Null</span> When the agent finished its work.
		</dd>
	

	
		

		Null means that the agent is currently running or was forcefully terminated.
		</dd>
	

	

	
		

		agent_uuid</code>
		</dt>
	

	
		

		UUID</a> External Key</span>
		</dd>
	

	

	
		

		name</code>
		</dt>
	

	
		

		Text</a>  The name of the crawler as specified using the --worker-name</code> option</a>.
		</dd>
	

	

	
		

		http_user_agent</code>
		</dt>
	

	
		

		Text</a> Null</span> The HTTP user agent that was used by the crawler.
		</dd>
	

	
		

		Null means that the the concept of a user agent isn't applicable to the crawler.
		</dd>
	

</dl>

crawl_log</code></h3>
The crawl log table gets written after a crawl command has been finished to log the outcome.</p>
The crawl_log</code> table stores crawl log entries</a>, it has the following fields:</p>




	




	
		
		crawl_log_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>
		</dd>
	

	

	
		

		agent_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the agent</code> table</a> of which agent is responsible for the entry.
		</dd>
	

	

	
		

		url_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the url</code> table</a>, which URL was crawled.
		</dd>
	

	

	
		

		crawl_type</code>
		</dt>
	

	
		

		Integer / Enumeration</a>  Which crawl type</a> resulted in this entry.
		</dd>
	

	

	
		

		crawl_uuid</code>
		</dt>
	

	
		

		UUID</a> External Key</span>
		</dd>
	

	

	
		

		time_started_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a>  When the action that resulted in this entry started.
		</dd>
	

	

	
		

		time_taken_ms</code>
		</dt>
	

	
		

		Integer / Timestamp</a>  How long the action took.
		</dd>
	

	

	
		

		exit_code</code>
		</dt>
	

	
		

		Integer / Enumeration</a>  Which outcome the action had, see crawl exit code</a>.
		</dd>
	

	

	
		

		message</code>
		</dt>
	

	
		

		Text</a> Null</span> A place to store an error message, this is intended to help humans with debugging.
		</dd>
	

</dl>

Indices:</p>




	




	
		
		crawl_log_quickinfo</code>
		</dt>
	

	
		

		on fields url_id</code>, time_started_unix_utc</code>, exit_code</code>
		</dd>
	

	
		
		For speeding up querying the last exit code of a URL
		</dd>
	

</dl>

request</code></h3>
The request</code> table has the following fields:</p>




	




	
		
		request_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>
		</dd>
	

	

	
		

		crawl_log_id</code>
		</dt>
	

	
		

		Integer</a>  Which entry in the crawl-log</code> table</a> this request belongs to.
		</dd>
	

	

	
		

		url_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the [url</code> table] which URL was requested (this may be different from the crawled URL, i.e. when scraping a CMS API).
		</dd>
	

	

	
		

		time_sent_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a>  When the request was sent.
		</dd>
	

	

	
		

		request_duration_ms</code>
		</dt>
	

	
		

		Integer / Duration</a> Null</span> How long the request took.
		</dd>
	

	
		

		Null means that measuring the time failed.
		</dd>
	

	

	
		

		robotstxt_approved</code>
		</dt>
	

	
		

		Bool</a>  Whether the request was approved by a robots.txt</code> file.
		</dd>
	

	

	
		

		exit_code</code>
		</dt>
	

	
		

		Integer / Enumeration</a>  The crawl exit code</a> of this single request.
		</dd>
	

	

	
		

		server_last_modified_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a> Null</span> When the server claimed that the file was last modified.
		</dd>
	

	
		

		Null means that the header information about the last modification time was missing.
		</dd>
	

	

	
		

		http_status_code</code>
		</dt>
	

	
		

		Integer / Enumeration</a> Null</span> The HTTP Status Code</a> that this request resulted in.
		</dd>
	

	
		

		Null means that no HTTP response was received.
		</dd>
	

	

	
		

		http_etag</code>
		</dt>
	

	
		

		Varchar</a> Null</span> The the content of the ETag</code> header</a>.
		</dd>
	

	
		

		Null means that no ETag header was received.
		</dd>
	

</dl>

file</code></h3>
Note:</b> The file</code> and redirect</code> tables have the following fields in common: crawl_log_id</code>, request_id</code>, url_id</code>, last_modified_unix_utc</code>.</p>
The file</code> table has the following fields:</p>




	




	
		
		file_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>
		</dd>
	

	

	
		

		crawl_log_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the crawl_log</code> table</a> of which crawl log entry resulted in this file. If a request is linked the request must be from the same crawl log entry.
		</dd>
	

	

	
		

		request_id</code>
		</dt>
	

	
		

		Integer</a> Null</span> Reference to the request</code> table</a> from which request this file metadata came from.
		</dd>
	

	
		

		Null means that the file didn't result from a network request. (i.e. by reading from a dump)
		</dd>
	

	

	
		

		url_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the url</code> table</a>, which URL this file is associated with.
		</dd>
	

	

	
		

		last_modified_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a> Null</span> When the file was last modified according to file metadata, archive metadata or response headers, the data here should be from the most reliable source.
		</dd>
	

	
		

		Null means that the file didn't contain any readable metadata on its last modification date.
		</dd>
	

	

	
		

		file_size</code>
		</dt>
	

	
		

		Integer</a> Null</span> The file size in bytes
		</dd>
	

	
		

		Null means that the file size is unknown (i.e. the file wasn't fetched and no metadata was present)
		</dd>
	

	

	
		

		mimetype_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the mimetype</code> table</a>, which MIME-Type (Media Type) the file has.
		</dd>
	

	

	
		

		canonical_url_id</code>
		</dt>
	

	
		

		Integer</a> Null</span> Reference to the url</code> table</a>, which URL the file claims its canonical version to be at. (See also RFC 6596</a>)
		</dd>
	

	
		

		Null means that the file didn't claim the be the non-canonical version of another resource.
		</dd>
	

</dl>

file_text</code></h3>
The file_text</code> table has the following fields:</p>




	




	
		
		file_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span> Reference to the file</code> table</a> entry that holds the metadata
		</dd>
	

	

	
		

		text</code>
		</dt>
	

	
		

		Text</a>  Unparsed text content of the file translated to UTF-8.
		</dd>
	

</dl>

redirect</code></h3>
Note:</b> The file</code> and redirect</code> tables have the following fields in common: crawl_log_id</code>, request_id</code>, url_id</code>, last_modified_unix_utc</code>.</p>
The redirect</code> table has the following fields:</p>




	




	
		
		redirect_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>
		</dd>
	

	

	
		

		crawl_log_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the crawl_log</code> table</a> of which crawl log entry resulted in this redirect. If a request is linked the request must be from the same crawl log entry.
		</dd>
	

	

	
		

		request_id</code>
		</dt>
	

	
		

		Integer</a> Null</span> Reference to the request</code> table</a> from which request this redirect came from.
		</dd>
	

	
		

		Null means that the redirect didn't result from a network request. (i.e. by reading from a dump)
		</dd>
	

	

	
		

		url_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the url</code> table</a>, which URL this redirect is associated with.
		</dd>
	

	

	
		

		last_modified_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a> Null</span> When the file was last modified according to file metadata, archive metadata or response headers, the data here should be from the most reliable source.
		</dd>
	

	
		

		Null means that the file didn't contain any readable metadata on its last modification date.
		</dd>
	

	

	
		

		to_url_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the url</code> table</a>, which URL the redirect points at.
		</dd>
	

	

	
		

		information_source</code>
		</dt>
	

	
		

		Integer / Enumeration</a>  Where the information for this redirect came from, see Information Source</a> for possible values.
		</dd>
	

	

	
		

		is_permanent</code>
		</dt>
	

	
		

		Bool</a>  Whether one can expect future requests to result in the same redirect.
		</dd>
	

	

	
		

		by_security_policy</code>
		</dt>
	

	
		

		Bool</a>  Whether the redirect was because of a security policy (i.e. an automatic http</code> to https</code> upgrade).
		</dd>
	

	
		

		🔧 This field is currently unused</strong> and may be removed in the future. (See Possible future changes</a>).
		</dd>
	

</dl>

crawl_candidate</code></h3>
Note:</b> This table contains information that is also present in the crawl log. This is to make sure that the information doesn't get lost when the crawl log gets cleaned up.</p>
This table attached crawling specific metadata to URLs.</p>
The crawl_candidate</code> table contains the following fields:</p>




	




	
		
		url_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span> Reference to the url</code> table</a> of the URL that this metadata is for.
		</dd>
	

	

	
		

		last_crawl_time_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a> Null</span> When the URL was last crawled
		</dd>
	

	
		

		Null means that the URL has only been discovered as crawlable, but not been crawled yet.
		</dd>
	

	

	
		

		last_crawl_exit_code</code>
		</dt>
	

	
		

		Integer / Timestamp</a> Null</span> The crawl exit code</a> of the last crawl.
		</dd>
	

	
		

		Null means that the exit code is unavailable.
		</dd>
	

	

	
		

		last_contentful_crawl_time_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a> Null</span> The last time the crawler exited with a code from the contentful category of exit codes</a>.
		</dd>
	

	
		

		Null means that there never was a crawl with a contentful exit code.
		</dd>
	

	

	
		

		last_contentful_http_etag</code>
		</dt>
	

	
		

		Text</a> Null</span> Set together with last_contentful_crawl_time_unix_utc</code>, this will contain the ETag header that was returned on the last contentful crawl.
		</dd>
	

	
		

		Null means that there was no last contentful crawl or the last contentful crawl didn't have a ETag header set.
		</dd>
	

</dl>

Possible future changes</h2>

Addition of UUID fields to the tables request</code>, file</code> and redirect</code></li>
The redirect</code></a>.by_security_policy</code> field could be integrated into the information_source</code>. It is currently never set to true by the crawler.</li>
</ul>
Version history</h2>
1.0.0 - De-facto stable</h3>
This schema has been de-facto stable for a while and been assigned the 1.0.0 version with the introduction of database versioning.</p>


Database Data Types
2025-08-13T00:00:00+00:00
This document describes the column datatypes used in the unobtanium databases.</p>
By default unless indicated by a Null</span> keyword in the documentation the datatypes are not nullable and in SQL use the NOT NULL</code> keyword. In general NULL</code> should mean unknown or unavailable data unless documented otherwise. The meaning of a NULL</code> value must</strong> always be documented.</p>
Columns that are marked an Unique</span> only allow unique vlues using the UNIQUE</code> SQL keyword, however they do allow multiple null values</strong>.</p>
Unobtanium uses SQLite as its database, see the SQLite documentation on datatypes</a>.</p>
Note that some advice on this page regarding datatype usage isn't followed. These are probbly the tables where a mistake was made in the past resulting in the advice here.</p>
Bool</h2>
Booleans are used to store true/false values represented using the SQLite BOOL</code> datatype and 1</code>/0</code> for the values.</p>
Column names should be prefixed with is_</code> or has_</code>.</p>
Text</h2>
Text is represented using the TEXT</code> SQLite datatype.</p>
Text must</strong> be encoded as UTF-8.</p>
Advice:</p>

If possible normalize empty strings to a NULL</code>.</li>
Do not use VARCHAR</code>.</li>
Check to see if there is already a table in the base schema</a> that you can reference.</li>
</ul>
Integer</h2>
Integers are always stored using the SQLite INTEGER</code> datatype, the rust equivlent is i64</code>, while always encoded as a signed integer negative values don't always make sense here.</p>
Integers are usually used to encode some other data.</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	




	
		
		Primary Key
		</dt>
	

	
		

		As a database internal identifier, these may not be handed out via external interfaces.
		</dd>
	

	
		

		The SQL for SQLite must be INTEGER NOT NULL PRIMARY KEY</code>, the columns are called {table_name}_id</code>.
		</dd>
	

	
		

		Marked as Primary Key</span>
		</dd>
	

	
		

		To enforce 1:1 relationships this may double as a reference into another table, in this case the colum name follows the reference naming scheme.
		</dd>
	

	

	
		

		Reference / Foregin Key
		</dt>
	

	
		

		To reference another tables primary key.
		</dd>
	

	
		

		The column must be named [(to|from)_]{table_name}[_{purpose}]_id</code>.
		</dd>
	

	

	
		

		Enumeration
		</dt>
	

	
		

		In the case they are uised for enumerations there must</strong> be linked documentation on a list of valid values and what they mean.
		</dd>
	

	

	
		

		Timestamp
		</dt>
	

	
		

		To store a Timestamps in unix time since 1970-01-01 00:00:00 UTC</code> in seconds, this is consistent with how SQLite expects timestamps to be stored.
		</dd>
	

	
		

		Column names must end with _unix_utc</code>
		</dd>
	

	

	
		

		Duration
		</dt>
	

	
		

		To store a length of time in some unit.
		</dd>
	

	
		
		The column name must end with a unit mentioned here: _ms</code>
		</dd>
	

</dl>

UUID</h2>
UUIDs</a> are represented as BLOB(16)</code> types in SQLite. Use the hex()</code> function to get the human reable representation.</p>
UUIDs are used as indentifiers that can be generated without the database, are unique across multiple databases and can be handed out via APIs as external identifiers.</p>
If a UUID colum fulfills a similar role as priamry key, the column name should be {table_name}_uuid</code>, in this case it is marked using External Key</span>.</p>
Advice:</p>

If unsure generate v7</code> (timestmp+random, preferred) or v4</code> (random) UUIDs.</li>
Treat UUIDs as opaque identifiers.</li>
Use UUIDs to hnd out identifiers via interface outside the application.</li>
Do not use UUIDs for database internal references.</li>
</ul>
Blob</h2>
Blobs are represented as BLOB</code> types in SQLite. They can be used to store arbitrary binary information.</p>
The format must</strong> be documented.</p>


Information Source
2025-08-13T00:00:00+00:00
The information source enumeration describes which part of a document a piece of information came from.</p>
Possible values are:</p>
Id</th> Name</th> Description</th></tr></thead>

0</td> unknown</code></td> The source is officially unknown</td></tr>
1</td> response_header</code></td> From a (network) response header</td></tr>
2</td> file_metadata</code></td> From the metadata section of a file</td></tr>
3</td> file_body</code></td> From the content section of a file</td></tr>
4</td> summary</code></td> The information is a summary derived from other data</td></tr>
</tbody></table>


Link destination type
2025-08-13T00:00:00+00:00
The link destination type enumerates whtto expect when follwing a link across all of its redirects.</p>
It is mostly derived from the html element that represents the link.</p>
Available values are:</p>
Id</th> Name</th> Descrioption</th></tr></thead>

0</td> file</code></td> Unknown content</td></tr>
1</td> document</code></td> Browsable document</td></tr>
2</td> feed</code></td> A feed as in RSS, Atom or similar</td></tr>
3</td> stylesheet</code></td> A stylesheet</td></tr>
4</td> script</code></td> Executable script inteded for a browser</td></tr>
100</td> media</code></td> Media or Media related file (image, audio, video, playlist, subitles, etc.)</td></tr>
101</td> media_image</code></td> Image File</td></tr>
102</td> media_audio</code></td> Audio File</td></tr>
103</td> media_video</code></td> Video File</td></tr>
1000</td> communication</code></td> Communication link like mailto</code> or tel</code></td></tr>
</tbody></table>


Summary Database
2025-08-13T00:00:00+00:00
The summary database scheme is implemented on top of the base database schema</a> and mainly houses the entity data tree</a>. It is mainly built by the summarizes algorithm</a>.</p>
Overview</h2>
Tables in the base database are:</p>




	




	
		
		entity_generation</code>
		</dt>
	

	
		
		The root of all summary data, spacetime coordinates for websites. See entity generation</a>.
		</dd>
	

	

	
		

		duplicate_summary</code>
		</dt>
	

	
		

		Information about exact duplicates</a>.
		</dd>
	

	

	
		

		crawl_summary</code>
		</dt>
	

	
		

		Stores the crawl summaries</a> that is derived from the crawl log</a>.
		</dd>
	

	

	
		

		http_summary</code>
		</dt>
	

	
		

		HTTP extension for the crawl_summary</code> table.
		</dd>
	

	

	
		

		file_summary</code>
		</dt>
	

	
		

		File metadata derived from the file</code> table in the crawler database</a>.
		</dd>
	

	

	
		

		redirect_summary</code>
		</dt>
	

	
		

		Redirect metadata derived from the redirect</code> table in the crawler database</a>.
		</dd>
	

	

	
		

		link_summary</code>
		</dt>
	

	
		

		Lists parsed links in files.
		</dd>
	

	

	
		

		document_description</code>
		</dt>
	

	
		

		Document metadata.
		</dd>
	

	

	
		

		text_pile</code>
		</dt>
	

	
		

		Collection of document text indexed by hash (deprecated), see text pile gen1</a>.
		</dd>
	

	

	
		

		text_pile_v0_2</code>
		</dt>
	

	
		

		Contains the Text Pile Gen2</a> as generated by version 0.2 of the unobtanium-text-pile</code> crate.
		</dd>
	

	
		

		Since 0.0.0-textrank-preview.0</code>
		</dd>
	

	

	
		

		token</code>
		</dt>
	

	
		

		Enumerates all possible tokens that can make up a document in this database.
		</dd>
	

	

	
		

		token_statistics</code>
		</dt>
	

	
		

		Maps tokens to text piles, also referred to as the token index</a>, this is experimental.
		</dd>
	

</dl>

Tables</h2>
entity_generation</code></h3>
The entity_generation</code> table stores entity generations</a>, it has the following fields:</p>




	




	
		
		entity_generation_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>
		</dd>
	

	

	
		

		entity_generation_uuid</code>
		</dt>
	

	
		

		UUID</a> External Key</span>
		</dd>
	

	

	
		

		url_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the url</code> table</a>, which URL this entity generation is about.
		</dd>
	

	

	
		

		url_fragment</code>
		</dt>
	

	
		

		Text</a> Null</span> The id of the document element this entity generation is about.
		</dd>
	

	
		

		Null means that this entity generation is about the whole document
		</dd>
	

	

	
		

		first_seen_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a>  First known existence of this generation (may change if better data is integrated)
		</dd>
	

	

	
		

		last_seen_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a>  Last known existence, may equal first_seen_unix_utc</code>.
		</dd>
	

	

	
		

		confirmed_end_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a> Null</span> The time the entity generation is known to be closed. (i.e. the first_seen</code> of the next entity generation)
		</dd>
	

	
		

		Null means that the entity generation is still open.
		</dd>
	

	

	
		

		marked_duplicate</code>
		</dt>
	

	
		

		Bool</a>  Whether this is marked as a duplicate in the duplicate_summary</code> table</a>. This is a cache value intended to accelerate queries by requiring less joining.
		</dd>
	

	

	
		

		text_pile_id</code>
		</dt>
	

	
		

		Integer</a> Null</span> Reference to the text_pile</code> table</a>, that contains the text content that this entity generation represents.
		</dd>
	

	
		

		Null means that this entity generation does not have any searchable text attached.
		</dd>
	

	

	
		

		text_pile_v0_2_id</code>
		</dt>
	

	
		

		Integer</a> Null</span> Reference to the text_pile_v0_2</code> table</a>, that contains the text content this entity generation represents.
		</dd>
	

	
		

		Since 0.0.0-textrank-preview.0</code>
		</dd>
	

</dl>

There is a check that first_seen_unix_utc <= last_seen_unix_utc</code>.</p>
Indices:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	




	
		
		entity_generation_by_uuid</code>
		</dt>
	

	
		

		On field entity_generation_uuid</code>
		</dd>
	

	
		

		Speeds up looking for entity generations by UUID.
		</dd>
	

	

	
		

		entity_generation_by_first_seen</code>
		</dt>
	

	
		

		On fields url_id</code>, url_fragment</code> and first_seen_unix_utc</code>
		</dd>
	

	
		

		Speeds up looking for entity generations based on when they were first seen for a given URL.
		</dd>
	

	

	
		

		entity_generation_by_text_pile</code>
		</dt>
	

	
		

		On fields text_pile_id</code> and confirmed_end_unix_utc</code>
		</dd>
	

	
		

		Speeds up looking for non closed entity generations given a text_pile_id</code>, this is needed for searching.
		</dd>
	

	

	
		

		entity_generation_by_active</code>
		</dt>
	

	
		

		On fields marked_duplicate</code>, confirmed_end_unix_utc</code>
		</dd>
	

	
		
		Speeds up querying for (non) duplicates.
		</dd>
	

</dl>

duplicate_summary</code></h3>
Note:</b> There is no way to directly address the duplicate summaries, they are always referenced as being the attachments of an entity generation.</p>
The duplicate_summary</code> table has the following fields:</p>




	




	
		
		subject_entity_generation_id</code>
		</dt>
	

	
		
		Integer</a>  Reference to the entity_generation</code> table</a>, of the entity generation that is flagged as duplicate.
		</dd>
	

	

	
		

		duplicate_of_entity_generation_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the entity_generation</code> table</a>, of the entity generation that the flagged entity generation is a duplicate of.
		</dd>
	

	

	
		

		duplicate_status_start_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a>  When the duplicate status started (usually the time it was detected)
		</dd>
	

	

	
		

		duplicate_status_end_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a> Null</span> When the duplicate status ended
		</dd>
	

	
		

		Null means that the duplicate status hasn't ended yet.
		</dd>
	

</dl>

Indices:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	




	
		
		duplicate_summary_by_duplicate</code>
		</dt>
	

	
		

		On fields subject_entity_generation_id</code> and duplicate_status_end_unix_utc</code>
		</dd>
	

	
		
		Speeds up looking if some entity generation is a non-expired duplicate.
		</dd>
	

</dl>

crawl_summary</code></h3>
The crawl_summary</code> table has the following fields:</p>




	




	
		
		crawl_summary_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>
		</dd>
	

	

	
		

		entity_generation_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the [entity_generation</code> table], of the entity generation that this crawl resulted in.
		</dd>
	

	

	
		

		was_robotstxt_approved</code>
		</dt>
	

	
		

		Bool</a>  Whether the crawl was approved by a robots.txt</code> file.
		</dd>
	

	

	
		

		crawl_type</code>
		</dt>
	

	
		

		Integer / Enumeration</a>  Which crawl type</a> resulted in this entry.
		</dd>
	

	

	
		

		crawl_uuid</code>
		</dt>
	

	
		

		UUID</a>  The UUID of the corresponding crawl log entry</a>.
		</dd>
	

	

	
		

		agent_uuid</code>
		</dt>
	

	
		

		UUID</a>  The UUID of the crawl agent that made the requests.
		</dd>
	

	

	
		

		time_started_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a>  The time when the crawl action was started.
		</dd>
	

	

	
		

		exit_code</code>
		</dt>
	

	
		

		Integer / Enumeration</a>  The crawl exit code</a> of the crawl action.
		</dd>
	

	

	
		

		request_duration_ms</code>
		</dt>
	

	
		

		Integer / Timestamp</a> Null</span> How long it took to execute the crawl action.
		</dd>
	

	
		

		Null means that the measurement failed.
		</dd>
	

</dl>

Indices:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	




	
		
		crawl_summary_by_crawl_uuid</code>
		</dt>
	

	
		

		On field crawl_uuid</code>
		</dd>
	

	
		
		Speeds up looking for already integrated crawls via test_has_crawl_summary_with_uuid_bulk()</code>.
		</dd>
	

</dl>

http_summary</code></h3>
An entry in this table is only present if the crawl resulted in a HTTP response.</p>
The http_summary</code> table has the following fields:</p>




	




	
		
		crawl_summary_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span> Reference to the crawl_summary</code> table</a> of the extended crawl summary.
		</dd>
	

	

	
		

		status_code</code>
		</dt>
	

	
		

		Integer / Enumeration</a>  The resulting HTTP status code</a>.
		</dd>
	

	

	
		

		etag</code>
		</dt>
	

	
		

		Text</a> Null</span> The resulting HTTP ETag header.
		</dd>
	

	
		

		Null means that no ETag header was returned.
		</dd>
	

</dl>

file_summary</code></h3>
The file_summary</code> table has the following fields:</p>




	




	
		
		entity_generation_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>Unique</span> Reference to the entity_generation</code> table</a> of the extended entity generation. (There can only be one file per entity generation)
		</dd>
	

	

	
		

		file_size</code>
		</dt>
	

	
		

		Integer</a> Null</span> The file size in bytes
		</dd>
	

	
		

		Null means that the file size is not known.
		</dd>
	

	

	
		

		mimetype_id</code>
		</dt>
	

	
		

		Integer</a> Null</span> Reference to the mimetype</code> table</a> of the files mimetype.
		</dd>
	

	
		

		Null means that the mimetype is not known (i.e. when there was a conflict detected between claimed mimetype and file content)
		</dd>
	

	

	
		

		canonical_url_id</code>
		</dt>
	

	
		

		Integer</a> Null</span> Reference to the url</code> table</a>, which URL the file claims its canonical version to be at.
		</dd>
	

	
		

		Null means that the file didn't claim the be the non-canonical version of another resource.
		</dd>
	

</dl>

redirect_summary</code></h3>
The redirect_summary</code> table has the following fields:</p>




	




	
		
		entity_generation_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span> Reference to the entity_generation</code> table</a> of the extended entity generation. (There can only be one redirect per entity generation)
		</dd>
	

	

	
		

		to_url_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the url</code> table</a>, which URL the redirect points at.
		</dd>
	

	

	
		

		information_source</code>
		</dt>
	

	
		

		Integer / Enumeration</a>  Where the information for this redirect came from, see Information Source</a> for possible values.
		</dd>
	

	

	
		

		by_security_policy</code>
		</dt>
	

	
		

		Bool</a>  Whether the redirect was because of a security policy.
		</dd>
	

	
		

		🔧 This field is currently unused</strong> and may be removed in the future.
		</dd>
	

	

	
		

		to_url_fragment</code>
		</dt>
	

	
		

		Text</a> Null</span> Fragment part of the URL that the redirect points at.
		</dd>
	

	
		

		Null means that the destination URL has no fragment part.
		</dd>
	

	
		

		🔧 This field is currently unused</strong> as there is no equivalent in the crawler database, it could be populated from HTML redirects.
		</dd>
	

</dl>

link_summary</code></h3>
The link_summary</code> table has the following fields:</p>




	




	
		
		link_summary_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>
		</dd>
	

	

	
		

		entity_generation_id</code>
		</dt>
	

	
		

		Integer</a> Primary Key</span> Reference to the entity_generation</code> table</a> of the entity generation this link is part of.
		</dd>
	

	

	
		

		link_to_url</code>
		</dt>
	

	
		

		Integer</a>  Reference to the url</code> table</a>, which URL the link points at.
		</dd>
	

	

	
		

		link_to_fragment</code>
		</dt>
	

	
		

		Text</a> Null</span> Fragment part of the URL that the link points at.
		</dd>
	

	
		

		Null means that the destination URL has no fragment part.
		</dd>
	

	

	
		

		rel_nofollow</code>
		</dt>
	

	
		

		Bool</a>  Whether the link was tagged with a rel="nofollow"</code>
		</dd>
	

	

	
		

		rel_me</code>
		</dt>
	

	
		

		Bool</a>  Whether the link was tagged with a rel="me"</code>
		</dd>
	

	

	
		

		rel_tag</code>
		</dt>
	

	
		

		Bool</a>  Whether the link was tagged with a rel="tag"</code>
		</dd>
	

	

	
		

		in_header</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_footer</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_aside</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_nav</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_form</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_main</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_article</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_section</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_table</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_figure</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_address</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_code</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_headline</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_list</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		in_paragraph</code>
		</dt>
	

	
		

		Bool</a>  Part of the location signature</a>.
		</dd>
	

	

	
		

		contains_headline</code>
		</dt>
	

	
		

		Bool</a>  Whether the link contains a headline element.
		</dd>
	

	

	
		

		destination_type</code>
		</dt>
	

	
		

		Integer / Enumeration</a> Null</span> The link destination type</a> of this link
		</dd>
	

	
		

		Null means that the link destination type couldn't be derived for some unknown reason.
		</dd>
	

	

	
		

		link_locality</code>
		</dt>
	

	
		

		Integer / Enumeration</a>  The link locality</a> describing how far reaching this link is.
		</dd>
	

	

	
		

		html_tag_name</code>
		</dt>
	

	
		

		Varchar</a> Null</span> The name of the HTML element the link appeared in.
		</dd>
	

	
		

		Null means that the link wasn't inside an HTML document or that there is no HTML equivalent for the link container (prefer using the destination_type</code>).
		</dd>
	

	

	
		

		text</code>
		</dt>
	

	
		

		Varchar</a> Null</span> Human readable text that describes the link.
		</dd>
	

	
		

		Null means the link didn't have any text to describe itself.
		</dd>
	

</dl>

document_description</code></h3>
Note:</b> All data in this table is derived from the document on a best effort</strong> basis.</p>
The document_description</code> table has the following fields:</p>




	




	
		
		entity_generation_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span> Reference to the entity_generation</code> table</a> of the entity generation extended by this document description.
		</dd>
	

	

	
		

		time_created_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a> Null</span> When the document was created.
		</dd>
	

	
		

		Null means that this information is unavailable.
		</dd>
	

	

	
		

		time_updated_unix_utc</code>
		</dt>
	

	
		

		Integer / Timestamp</a> Null</span> When the document was last updated.
		</dd>
	

	
		

		Null means that this information is unavailable.
		</dd>
	

	

	
		

		primary_language</code>
		</dt>
	

	
		

		Varchar</a> Null</span> The language code that represents the primary document language. should</strong> be a member of the iana language subtag registry</a>.
		</dd>
	

	
		

		Null means that this information is unavailable.
		</dd>
	

	

	
		

		title</code>
		</dt>
	

	
		

		Text</a> Null</span> The title of the document that would be appropriate to display on a browser tab.
		</dd>
	

	
		

		Null means that this information is unavailable.
		</dd>
	

	

	
		

		primary_headline</code>
		</dt>
	

	
		

		Text</a> Null</span> The main headline of the document. This should not be the site name.
		</dd>
	

	
		

		Null means that this information is unavailable.
		</dd>
	

	

	
		

		description</code>
		</dt>
	

	
		

		Text</a> Null</span> A short description of the document.
		</dd>
	

	
		

		Null means that this information is unavailable.
		</dd>
	

	

	
		

		indexiness</code>
		</dt>
	

	
		

		[Integer] The indexiness score</a> of the document.
		</dt>
	

</dl>

text_pile</code></h3>
The text_pile</code> table stores text piles Gen1</a>, it has the following fields:</p>




	




	
		
		text_pile_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>
		</dd>
	

	

	
		

		blake2b512_digest</code>
		</dt>
	

	
		

		Blob</a> Unique</span> The text pile digest</a> that is used for exact duplicate</a> detection, it also makes sure each possible text pile is only stored once.
		</dd>
	

	

	
		

		text</code>
		</dt>
	

	
		

		Text</a>  Main file content, empty string if not present
		</dd>
	

	

	
		

		secondary_text</code>
		</dt>
	

	
		

		Text</a>  Secondary file content, empty string if not present
		</dd>
	

	

	
		

		big_headlines</code>
		</dt>
	

	
		

		Text</a>  Bigger headlines, empty string if not present
		</dd>
	

	

	
		

		small_headlines</code>
		</dt>
	

	
		

		Text</a>  Smaller headlines, empty string if not present
		</dd>
	

	

	
		

		code_text</code>
		</dt>
	

	
		

		Text</a>  Text marked up as code, empty string if not present
		</dd>
	

	

	
		

		quote_text</code>
		</dt>
	

	
		

		Text</a>  Text marked up as quote, empty string if not present
		</dd>
	

</dl>

text_pile_v0_2</code></h3>
The text_pile_v0_2</code> table stores text piles Gen2</a>, it has the following fields:</p>




	




	
		
		text_pile_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>
		</dd>
	

	

	
		

		blake2b512_digest</code>
		</dt>
	

	
		

		Blob</a> Unique</span> The text pile Gen2 digest</a> that is used for exact duplicate</a> detection, it also makes sure each possible text pile Gen2 is only stored once.
		</dd>
	

	

	
		

		text</code>
		</dt>
	

	
		

		Text</a>  The raw text behind the text pile Gen2.
		</dd>
	

	

	
		

		metadata</code>
		</dt>
	

	
		

		Blob</a>  The text pile Gen2 metadata encoded as postcard.
		</dd>
	

	

	
		

		segmentation_cache</code>
		</dt>
	

	
		

		Blob</a>  The segmentation metadata of the text pile Gen2, non optional.
		</dd>
	

</dl>

Note on the segmentation_cache</code>:</b> The segmentation cache in the database is deliberately non optional to make handling of text data easier on the consumer side. This is contrary to the segmentation cache being an optional component of a text pile Gen2 elsewhere.</p>
token</code></h3>
The token</code> table has the following fields:</p>




	




	
		
		token_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>
		</dd>
	

	

	
		

		token_text</code>
		</dt>
	

	
		

		Text</a> Unique</span> The text represented by this token.
		</dd>
	

</dl>

token_statistics</code></h3>
The token_statistics</code> table has the following fields:</p>




	




	
		
		token_id</code>
		</dt>
	

	
		
		Integer</a>  Reference to the token</code> table</a> of the token this is about.
		</dd>
	

	

	
		

		text_pile_id</code>
		</dt>
	

	
		

		Integer</a>  Reference to the text_pile</code> table</a> of the text pile this is about.
		</dd>
	

	

	
		

		occurances</code>
		</dt>
	

	
		

		Integer</a>  How often the token occurs in the text pile.
		</dd>
	

</dl>

The combination of token_id</code> and text_pile_id</code> must be unique.</p>
🔧 This table is work in progress man may be subject to major changes in the future.</p>
full_text_entity_index</code></h3>
This table is backed by the fts5</code> SQLite extension. It is mainly derived from the text_pile</code> and document_description</code> tables.</p>
The full_text_entity_index</code> table has the following fields:</p>




	




	
		
		text_pile_id</code>
		</dt>
	

	
		
		Integer</a>  Reference to the [text_pile</code> table] of the text pile this was derived from.
		</dd>
	

	

	
		

		any</code>
		</dt>
	

	
		

		Text</a>  Always contains the string "any", this is needed to work around limitations of the fts5 NOT</code> operator.
		</dd>
	

	

	
		

		title</code>
		</dt>
	

	
		

		Text</a>  The document title taken from the document_description</code>.
		</dd>
	

	

	
		

		description</code>
		</dt>
	

	
		

		Text</a> Null</span> The description taken from the document_description</code>.
		</dd>
	

	
		

		Null means that no description is available.
		</dd>
	

	

	
		

		text</code>
		</dt>
	

	
		

		Text</a>  The text taken from the text_pile</code>
		</dd>
	

	

	
		

		secondary_text</code>
		</dt>
	

	
		

		Text</a>  The secondary text taken from the text_pile</code>
		</dd>
	

	

	
		

		big_headlines</code>
		</dt>
	

	
		

		Text</a>  The big headlines taken from the text_pile</code>
		</dd>
	

	

	
		

		small_headlines</code>
		</dt>
	

	
		

		Text</a>  The small headlines taken from the text_pile</code>
		</dd>
	

	

	
		

		code_text</code>
		</dt>
	

	
		

		Text</a>  The code text taken from the text_pile</code>
		</dd>
	

	

	
		

		quote_text</code>
		</dt>
	

	
		

		Text</a>  The quote text taken from the text_pile</code>
		</dd>
	

</dl>

Version History</h2>
0.0.0 - The first versioned</h3>
This is the first versioned and documented version of this database.i</p>
This schema is released with unobtanium 3.0.0.</p>
0.0.0-textrank-preview.0</h3>
This is an unstable development version to introduce and test the Text Pile Gen2</a>.</p>


unobtanium-viewer
2025-08-10T00:00:00+00:00
The unobtanium-viewer</code> takes a summary database and makes it acessible as a search frontend.</p>
It exports an HTTP socket you should put it behind a reverse proxy.</p>
Synopsis</h2>
unobtanium-viewer [OPTIONS...]
</code></pre>
Options</h2>
Accepted options are:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	




	
		
		-s</code>, --summary-db</code> <file.db></em>
		</dt>
	

	
		

		Path to the summary database.
		</dd>
	

	
		

		Default is summary.db</code>
		</dd>
	

	

	
		

		-t</code>, --template-location</code> <template-directory></em>
		</dt>
	

	
		
		Path to the template directory</a>.
		</dd>
	

	
		

		Default is /templates</code>
		</dd>
	

	

	
		

		-e</code>, --extra-config</code> <extra.toml></em>
		</dt>
	

	
		

		Path to the template configuration file. Its content depends on the template. An example file for the default template can be found in the source</a>, make sure to keep this one updated.
		</dd>
	

	
		

		Default is extra.toml</code> inside the template location
		</dd>
	

	

	
		

		-l</code>, --listen-on</code> <ip:port></em>
		</dt>
	

	
		

		HTTP Socket to listen on.
		</dd>
	

	
		

		Default is 127.0.0.1:3000</code> (local machine only, tcp port 3000)
		</dd>
	

	

	
		

		--request-workers</code> <n></em>
		</dt>
	

	
		

		How many worker threads to keep available for searching, you should keep this number below the number of available CPU cores. This determines the number of concurrent queries that unobtanium can process.
		</dd>
	

	
		

		Default is 3</code>
		</dd>
	

	

	
		

		--help</code>
		</dt>
	

	
		

		Print help text describing the available options and exit.
		</dd>
	

</dl>

HTTP headers</h2>
It is recommended to set the following security headers in your reverse proxy:</p>
X-Content-Type-Options "nosniff"
Referrer-Policy "strict-origin"
permissions-policy "interest-cohort=()"
Content-Security-Policy "default-src 'none'; script-src 'self' blob: ; style-src 'self'; img-src 'self'; media-src 'self'; base-uri 'none'; form-action 'self' echoip.slatecave.net ; frame-ancestors 'none'"
Cache-Control "max-age=3600, min-fresh=600, stale-if-error=86400, must-revalidate"
</code></pre>
Note:</b> Currently the unobtanium frontend contains no javascript, this may change in the future to enable progresive enhancements or interactive features.</p>
SQLite caching</h2>
Unobtanium itself implements no caching mechanisms, keeping a bunch of memory free for the Linux kernel to keep frequently needed parts of the database in RAM is recommended and will greatly improve the query speed.</p>
Plan keeping at least 10% of the summary database size as "usused" RAM so that the kernel can use it for caching.</p>
Serving static files directly</h2>
Inside the template directory there is a static</code> directory which cn be served by your web server directly by overlaying it on top of the reverse proxy functionality.</p>
Note:</b> If you don't do this you're fine too, the unobtanium-viewer will serve them with the same functionality.</p>
Example configuration for the Caddy web server assuming the template directory is /opt/unobtaniun/default-template/</code>:</p>
your.domain {
	[… other configuration …]

	# Static file matcher
	@isStatic file {
		root /opt/unobtaniun/default-template/static/
	} 

	# Static file handler
	handle @isStatic {
		root * /opt/unobtaniun/default-template/static/
		file_server
	}

	# Reverse proxy
	reverse_proxy * localhost:11006 {
		header_up Host search.slatecave.net
	}
}
</code></pre>
Anubis</h2>
In case you want to put the Anubis scraper blocker</a> in front of the viewer because you're reciving a high volume of automated queries, the default configuration for anubis works fine.</p>
It should not be necessary though, unobtanium can handle high volumes of requests pretty well.</p>


Database Info
2025-08-09T00:00:00+00:00
The unonatanium_database_info</code> table is part of the Base Database Schema</a>, it contains schema versioning metadata about a given database.</p>
Valid keys are:</p>




	




	
		
		unobtanium_base_schema_version</code>
		</dt>
	

	
		

		The version of the base schema i.e. 0.1.0</code>
		</dd>
	

	

	
		

		unobtanium_database_kind</code>
		</dt>
	

	
		

		Which kind of database this is, either crawler</code> or summary</code>
		</dd>
	

	

	
		

		unobtanium_crawler_schema_version</code>
		</dt>
	

	
		

		The version of the crawler database schema i.e. 1.0.0</code> (only if the database is a crawler database)
		</dd>
	

	

	
		

		unobtanium_summary_schema_version</code>
		</dt>
	

	
		

		The version of the summary database schema i.e. 0.0.0</code> (only if the database is a summary database)
		</dd>
	

</dl>

Note:</b> The base schema may increment without the summary or crawler schema getting a version bump.</p>
Manually adding version information</h2>
In case you have an "old" database run the following SQL manually.</p>
For this to work the databases must have the right schema, otherwise they'll fail like they did previously: Error on some random SQL statement that happens to not work anymore.</p>
For crawler</strong> databases:</p>
INSERT INTO unobtanium_database_info(key,value) VALUES
	('unobtanium_base_schema_version', '0.1.0'),
	('unobtanium_database_kind', 'crawler'),
	('unobtanium_summary_schema_version','1.0.0');
</code></pre>
For summary</strong> databases:</p>
INSERT INTO unobtanium_database_info(key,value) VALUES
	('unobtanium_base_schema_version', '0.1.0'),
	('unobtanium_database_kind', 'summary'),
	('unobtanium_summary_schema_version','0.0.0');
</code></pre>


Crawl Delay between Requests
2025-08-06T00:00:00+00:00
The crawl delay is the time between requests the crawler waits to avoid causing too much load on web servers.</p>
Choosing the crawl delay is always a compromise between not causing too much load on a server and getting the crawling done as fast as possible.</p>
Currently unobtanium runs once crawler for each Origin</a>, each with its own crawl loop</a> and its own crawl delay calculation.</p>
The crawl delay is implemented in unobtanium/crawler/src/crawler/crawl_delay.rs</a>.</p>
Calculating the Crawl Delay</h2>
Unobtanium chooses from multiple delays:</p>

A minimum delay</li>
A politeness based delay</li>
</ul>
Of those the longer one will be chosen as the wait time until the next request.</p>
The Minimum Delay</h3>
The minimum delay is either the Crawl-Delay</code> from robots.txt</code> capped at 2 minutes if available or a configured minimum delay</a>.</p>
The Politeness based Delay</h3>
The politeness mechanism is taken from the stract crawler</a>.</p>
In general it is based on taking how long the last request took to respond and multiplying it by 2^{politeness</sup></code>. The politeness factor</i> starts at 2 and then decreases, but not below 0.</p>}
If a HTTP 429</code> reponse is received, the politeness factor will be incremented by 1 (doubling the wait time) and autodecrementing will be turned off until the end of the crawl run.</p>


Boolean
2025-07-26T00:00:00+00:00
Bolean matching criterium from the criterium</code> crate</a>.</p>
The boolean matching criterium has the following variants:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	




	
		
		equals</code>
		</dt>
	

	
		

		Takes a boolean as argument.
		</dd>
	

	
		

		Matches if the boolean to match equals the given value.
		</dd>
	

	

	
		

		is_none</code>
		</dt>
	

	
		

		Takes no argument
		</dd>
	

	
		

		Matches if the boolean to match is a None value.
		</dd>
	

</dl>


Criterium Chain
2025-07-26T00:00:00+00:00
Criterium Chain from the criterium</code> crate</a>.</p>
The criterium chain is used to combine multiple criteria of the same type using boolean logic. Multiple chains can be nested.</p>
Which criterium is used can be found near the documentation that mentions that a criterium chain is accepted for some value as this is usecase dependent.</p>
You can use the variants of the contained criterium directly in place of where the chain would go. Under the hood this will generate a chain of the match</code> variant tha matches when the contained criterium matches.</p>
The number matching criterium has the following variants:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
			
			
				
			
		
	




	
		
		and</code>
		</dt>
	

	
		

		Takes a list of chains of the same type as argument.
		</dd>
	

	
		

		Matches if all of the given chains match.
		</dd>
	

	

	
		

		or</code>
		</dt>
	

	
		

		Takes a list of chains of the same type as argument.
		</dd>
	

	
		

		Matches if at least one of the given chains match.
		</dd>
	

	

	
		

		not</code>
		</dt>
	

	
		

		Takes a single chain as argument.
		</dd>
	

	
		

		Matches if the contained chain does not match.
		</dd>
	

	

	
		

		always</code>
		</dt>
	

	
		

		Takes no argument.
		</dd>
	

	
		

		Always matches.
		</dd>
	

	

	
		

		never</code>
		</dt>
	

	
		

		Takes no argument.
		</dd>
	

	
		

		Never matches.
		</dd>
	

	

	
		

		not_match</code>
		</dt>
	

	
		

		Takes a criterium matching the chain type as argument.
		</dd>
	

	
		

		Matches if the contained criterium does not match.
		</dd>
	

	

	
		

		match</code> (invisible variant)
		</dt>
	

	
		

		Takes a criterium matching the chain type as argument.
		</dd>
	

	
		

		Matches if the contained criterium marches, this one is just a type adaptor.
		</dd>
	

	
		

		This variant is invisible</strong> (untagged), you can use the argument it would take directly in place of this variant and it will be contsructed auutomatically.
		</dd>
	

</dl>


Number
2025-07-26T00:00:00+00:00
Number matching criterium from the criterium</code> crate</a>.</p>
Number matching works th same on integers, floats, signed and unsigned numbers.</p>
The number matching criterium has the following variants:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	




	
		
		equals</code>
		</dt>
	

	
		

		Takes a number as argument.
		</dd>
	

	
		

		Matches if the number to match equals the given number.
		</dd>
	

	

	
		

		less_than</code>
		</dt>
	

	
		

		Takes a number as argument.
		</dd>
	

	
		

		Matches if the number to match is less than the given number.
		</dd>
	

	

	
		

		less_than_or_equal</code>
		</dt>
	

	
		

		Takes a number as argument.
		</dd>
	

	
		

		Matches if the number to match is less than or equal to the given number.
		</dd>
	

	

	
		

		greater_than</code>
		</dt>
	

	
		

		Takes a number as argument.
		</dd>
	

	
		

		Matches if the number to match is greater than the given number.
		</dd>
	

	

	
		

		greater_than_or_equal</code>
		</dt>
	

	
		

		Takes a number as argument.
		</dd>
	

	
		

		Matches if the number to match is greater than or equal to the given number.i
		</dd>
	

	

	
		

		in_list</code>
		</dt>
	

	
		

		Takes a list of numbers as argument.
		</dd>
	

	
		

		Matches if the number to match is equal to one of the given numbers.
		</dd>
	

	

	
		

		is_none</code>
		</dt>
	

	
		

		Takes no argument
		</dd>
	

	
		

		Matches if the number to match is a None value
		</dd>
	

</dl>


Origin
2025-07-26T00:00:00+00:00
This criterium is for matching URL orgins</a> in unobtanium.</p>
The number matching criterium has the following variants:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	




	
		
		scheme</code>
		</dt>
	

	
		
		Takes String Criterium</a> as argument.
		</dd>
	

	
		

		Matches the origins scheme against the given string criterium.
		</dd>
	

	

	
		

		host</code>
		</dt>
	

	
		

		Takes String Criterium</a> as argument.
		</dd>
	

	
		

		Matches the origins host against the given string criterium.
		</dd>
	

	

	
		

		port</code>
		</dt>
	

	
		

		Takes Number Criterium</a> for a u16</code> as argument.
		</dd>
	

	
		

		Matches the origins port against the given number criterium.
		</dd>
	

</dl>


String
2025-07-26T00:00:00+00:00
String matching criterium from the criterium</code> crate</a>.</p>
The String criterium has the following variants:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	




	
		
		equals</code>
		</dt>
	

	
		

		Takes a string as argument.
		</dd>
	

	
		

		Matches if the string to match equals the given string.
		</dd>
	

	

	
		

		has_prefix</code>
		</dt>
	

	
		

		Takes a string as argument.
		</dd>
	

	
		

		Matches if the string to match has the gven string as prefix.
		</dd>
	

	

	
		

		has_suffix</code>
		</dt>
	

	
		

		Takes a string as argument.
		</dd>
	

	
		

		Matches if the string to match has the gven string as suffix.
		</dd>
	

	

	
		

		contains</code>
		</dt>
	

	
		

		Takes a string as argument.
		</dd>
	

	
		

		Matches if the string to match contains the given string as a substring.
		</dd>
	

	

	
		

		length</code>
		</dt>
	

	
		
		Takes a Number Criterium</a> as argument.
		</dd>
	

	
		

		Matches the length of the string in unicode codepoints to match against the given Number Criterium.
		</dd>
	

	

	
		

		is_none</code>
		</dt>
	

	
		

		Takes no argument
		</dd>
	

	
		

		Matches if the string to match is None
		</dd>
	

</dl>


Url
2025-07-26T00:00:00+00:00
A criterium for matchain against URLs</a> in unobtanium.</p>
The URL matching criterium has the following variants:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	




	
		
		user</code>
		</dt>
	

	
		
		Takes a String Criterium</a> as argument.
		</dd>
	

	
		

		Matches the URLs user part against the given string criterium.
		</dd>
	

	
		

		Defaults to an empty string, will never be a none value.
		</dd>
	

	

	
		

		password</code>
		</dt>
	

	
		

		Takes a String Criterium</a> as argument.
		</dd>
	

	
		

		Matches the URLs password part against the given string criterium.
		</dd>
	

	
		

		Defaults to a none value.
		</dd>
	

	

	
		

		path</code>
		</dt>
	

	
		

		Takes a String Criterium</a> as argument.
		</dd>
	

	
		

		Matches the URLs path part against the given string criterium.
		</dd>
	

	
		

		Defaults to an empty string, will never be a none value.
		</dd>
	

	

	
		

		query</code>
		</dt>
	

	
		

		Takes a String Criterium</a> as argument.
		</dd>
	

	
		

		Matches the URLs query part against the given string criterium.
		</dd>
	

	
		

		Defaults to a none value.
		</dd>
	

	

	
		

		fragment</code>
		</dt>
	

	
		

		Takes a String Criterium</a> as argument.
		</dd>
	

	
		

		Matches the URLs fragment part against the given string criterium.
		</dd>
	

	
		

		Defaults to a none value.
		</dd>
	

	

	
		

		scheme</code>
		</dt>
	

	
		

		Same as scheme</code> on the Origin criterium</a>
		</dd>
	

	

	
		

		host</code>
		</dt>
	

	
		

		Same as host</code> on the Origin criterium</a>
		</dd>
	

	

	
		

		port</code>
		</dt>
	

	
		

		Same as port</code> on the Origin criterium</a>
		</dd>
	

	

	
		

		equals</code>
		</dt>
	

	
		

		Takes an URL as argument.
		</dd>
	

	
		

		Matches if the URL to match is equals the given one.
		</dd>
	

	

	
		

		same_document_as</code>
		</dt>
	

	
		

		Takes an URL without a fragment part as argument.
		</dd>
	

	
		

		Matches if the URL to match equals the given URL ignoring the fragment part.
		</dd>
	

</dl>


Crawler crawl configuration
2025-07-26T00:00:00+00:00
This page documents the configuration for the unobtanium-crawler crawl</code> subcommand that is given with the --config</code> option.</p>
!note: New here?</b> A guided tour for your first search engine</a>.</p>
!note: More Examples:</b>
Examples of working configuration files can be found in the unobtanium.
rocks index configuration</a>. (Those file are behind the index on unobtanum.rocks</a>)</p>
Available settings are:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	




	
		
		name</code> (optional)
		</dt>
	

	
		

		Human readable name of the index for documentation purposes
		</dd>
	

	

	
		

		description</code> (optional)
		</dt>
	

	
		

		Human readable description of the index for documentation purposes. When writing rich text here format it as markdown.
		</dd>
	

	

	
		

		database_file</code> (optional)
		</dt>
	

	
		

		If given, the crawler database files path must end with the same path segments as given here, otherwise the crawler will throw an error. This is part of a safety mechanism to prevent mixing up multiple databases.
		</dd>
	

	
		

		The --ignore-db-name-from-config</code> option can be used to ignore this setting.
		</dd>
	

	

	
		

		default_delay_ms</code>
		</dt>
	

	
		

		The default crawl delay in milliseconds between requests to the same site, if the sites robots.txt</code> didn't request a different one. Setting this from 1000</code> to 2000</code> milliseconds should be a sane default. Setting more here is more polite, but it will take longer to crawl.
		</dd>
	

	

	
		

		max_commands_per_run</code>
		</dt>
	

	
		

		The number of crawling commands the crawler will execute before exiting automatically. A crawling command is roughly equivalent to one request. This stops the crawler from running too long. Setting this to 60</code> to 100</code> per indexed site</strong> is a sane default.
		</dd>
	

	
		

		The --max-commands</code> option can be used to override this setting.
		</dd>
	

	

	
		

		recrawl_interval</code>
		</dt>
	

	
		

		How long the crawler will wait until it schedules a site that has already been crawled successfully for being crawled again in case it was updated. A sane default would be 4 week</code> but can be more or less depending on the usecase. Guaranteed supported suffixes are second</code>, minute</code>, hour</code>, day</code>, week</code>, month</code>, year</code>. Other suffixes might work but are not guaranteed to be supported long term.
		</dd>
	

	
		

		The --force-recrawl</code> option can be used to ignore this setting.
		</dd>
	

	

	
		

		user_agent</code> (optional, but highly recommended)
		</dt>
	

	
		
		Configure the user agent that will be used while crawling. See the Crawler User-Agent and robots.txt</a> page for more information.
		</dd>
	

	
		

		Can be overridden using the --user-agent</code> option.
		</dd>
	

	

	
		

		seeds</code>
		</dt>
	

	
		

		List of URLs that are the entry points for the crawler. The crawler will start at these and follow links from there given they point to an origin that is part of one of these seed URLs. See the Crawling Loop algorithm</a> page for more information.
		</dd>
	

	
		

		Can be overridden using the --schedule</code> option.
		</dd>
	

	

	
		

		do_not_crawl</code>
		</dt>
	

	
		

		List of policies to limit which pages are crawled on top of a sites own robots.txt</code>.
		</dd>
	

</dl>

Policies</h2>
Policies are lists of configuration objects. The canonical way to write them is to use thee Array of Tables</a> TOML syntax at the end of the file.</p>
See the Crawling Loop algorithm</a> page on the impact of crawl policies.</p>
do_not_crawl</code></h3>
The do_not_crawl</code> Policy can inhibit crawling when a Crawl Policy Criterium</a> matches.</p>
Settings for the do not crawl policy are:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	




	
		
		reason</code>
		</dt>
	

	
		

		Documents the reason this policy exists, usually a human readable version of the rule.
		</dd>
	

	

	
		

		if</code>
		</dt>
	

	
		
		The serialised criterium chain</a> carrying crawl policy criteria that will prevent crawling if matched. (Example: if.url.path.has_prefix = "login"</code>)
		</dd>
	

</dl>

Example policy that prevents crawling of the issue tracker on osmocom.org</code> that is under the path /issues</code>:</p>
[[do_not_crawl]]
reason="Don't crawl the osmocom issue tracker"
if.and = [
	{ url.host.equals = "osmocom.org" },
	{ url.path.has_prefix = "/issues" },
]
</code></pre>
url_query_paramters</code></h3>
This crawl policy can be used to allow a combination of URL query parameters when a Crawl Policy Criterium</a> matches.</p>
By default URLs with query parameters are ignored as the usually point to page variants that are not useful for search applications.</p>
Not all of the allowed parameters have to be present. If additional parameters are present, that URL won't be crawled unless another rule allows that combination.</p>
Rules do not combine their allow lists. Example: If one rule for a page only allows foo</code> and another only allows bar</code> an URL containing both foo</code> and bar</code> parameters is still not allowed.</p>
Settings for the URL query parameters policy are:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	




	
		
		reason</code>
		</dt>
	

	
		

		The reason this policy exists, usually a why this combination of query parameters is allowed.
		</dd>
	

	

	
		

		allow</code>
		</dt>
	

	
		

		List of URL query parameters to allow. (Example: allow = ["page"]</code>)
		</dd>
	

	

	
		

		if</code>
		</dt>
	

	
		
		The serialised criterium chain</a> carrying crawl policy criteria that will activate this policy when matched.
		</dd>
	

</dl>

Example policy that allows the crawler to use the page</code> parameter on the site theorangeone.net</code> to navigate the paginated index pages:</p>
[[url_query_paramters]]
if.url.host.equals = "theorangeone.net"
allow = ["page"]
reason = "used for pagination"
</code></pre>
Crawl Policy Criterium</h3>
The crawl policy criterium can match page metadata that is available before</strong> the request happened at the scheduling stage.</p>
The crawl policy criterium has the follwing variants</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	




	
		
		url</code>
		</dt>
	

	
		
		Takes a URL Criterium</a> as argument.
		</dd>
	

	
		

		Matches if the URL of the to be crawled page matches the given URL criterium.
		</dd>
	

</dl>


unobtanium-crawler
2025-07-26T00:00:00+00:00
The unobtanium-crawler</code> collects data from the web and summarizes it.</p>
It mainatains both the crawler and summary databases.</p>
Synopsis</h2>
The crawler is split up into many subcommands.</p>

unobtanium-crawler crawl [OPTIONS]...</code></li>
unobtanium-crawler summarize [OPTIONS]...</code></li>
unobtanium-crawler optimize-db --database <database-file></code></li>
unobtanium-crawler regenerate-token-index [OPTIONS]...</code></li>
unobtanium-crawler delete [S-SUBCOMMAND]</code>

old-crawl-log-entries [OPTIONS]...</code></li>
</ul>
</li>
unobtanium-crawler debug [SUBCOMMAND]</code>

indexiness [OPTIONS]...</code></li>
query-crawl-log [OPTIONS]...</code></li>
sqlite-version</code></li>
</ul>
</li>
</ul>
crawl</code></h2>
The crawl</code> subcommand starts the Crawl Loop</a> for a given crawl configuration</a>.</p>
Note:</b> For testing the crawler can be configured with command line options only, however this setup isn't recommended for long term deployments.</p>
Accepted options are:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	




	
		
		-c</code>, --database</code> <file></em>
		</dt>
	

	
		

		The crawler database file to store the crawl results in.
		</dd>
	

	
		

		If the file doesn't exist yet, it will be created.
		</dd>
	

	

	
		

		-u</code>, --user-agent</code> <user_agent></em>
		</dt>
	

	
		
		Set the user agent</a>.
		</dd>
	

	
		

		Overrides the user_agent</code> setting from the configuration file</a>.
		</dd>
	

	

	
		

		-w</code>, --worker-name</code> <name></em>
		</dt>
	

	
		

		Set the worker name to be logged to the database.
		</dd>
	

	
		

		Default is ant</code>
		</dd>
	

	

	
		

		-m</code>, --max-commands</code> <number></em>
		</dt>
	

	
		

		The maximum number of commands to process in this run
		</dd>
	

	
		

		Overrides the max_commands_per_run</code> setting from the configuration file</a>.
		</dd>
	

	

	
		

		-d</code>, --default-delay</code> <milliseconds></em>
		</dt>
	

	
		

		The default wait time between requests.
		</dd>
	

	
		

		Overrides the default_delay_ms</code> setting from the configuration file</a>.
		</dd>
	

	

	
		

		--schedule</code> <url></em>
		</dt>
	

	
		

		Manually schedule a seed URL.
		</dd>
	

	
		

		Overrides the seeds</code> setting from the configuration file</a>.
		</dd>
	

	

	
		

		--config</code> <path></em>
		</dt>
	

	
		

		Specify a path to the configuration file</a>.
		</dd>
	

	

	
		

		--policy-file</code> <path></em>
		</dt>
	

	
		

		Specify a path to a policy configuration file, it can contain additional policies with the same notation as in the configurtion file</a>.
		</dd>
	

	

	
		

		--force-recrawl</code>
		</dt>
	

	
		

		Ignore when a pages were last crawled and scedule them for recrawling immedeately.
		</dd>
	

	
		

		Overrides the recrawl_interval</code> setting from the configuration file</a>.
		</dd>
	

	

	
		

		--ignore-db-name-from-config</code>
		</dt>
	

	
		

		Ignore the database_name</code> setting from the configuration file</a>.
		</dd>
	

</dl>

summarize</code></h2>
The summarize subcommand takes a crawler database and integrates it into a summary database using the Summarizing algorithm</a>.</p>
Accepted options are:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
				
			
		
	




	
		
		-c</code>, --crawler-db</code> <file></em>
		</dt>
	

	
		

		Database file of the crawler database.
		</dd>
	

	

	
		

		-s</code>, --summary-db</code> <file></em>
		</dt>
	

	
		

		Database file of the crawler database.
		</dd>
	

	
		
		If the file doesn't exist yet, it will be created.
		</dd>
	

</dl>

optimize-db</code></h2>
Runs the SQLite internal  analyze and optimize commands on the given crawler or summary  database.</p>
Accepted options are:</p>




	




	
		
		-c</code>, --database</code> <file></em>
		</dt>
	

	
		
		The database file to optimize.
		</dd>
	

</dl>

regenerate-token-index</code></h2>
Regenerates the experimental token based index</a> for use with the token:</code> filter</a>.</p>
Accepted options are:</p>




	




	
		
		-s</code>, --summary-database</code> <file></em>
		</dt>
	

	
		
		The summary database to generate the token index for.
		</dd>
	

</dl>

delete old-crawl-log-entries</code></h2>
Delete old entries from the crawl log</a> in the crawler database along with their associated data.</p>
Accepted options are:</p>




	




	
		
		-c</code>, --crawler-db</code> <file></em>
		</dt>
	

	
		

		The crawler database to delete crawl log entries from.
		</dd>
	

	

	
		

		--keep-latest</code> <n></em>
		</dt>
	

	
		

		How many of the latest entries for each page to keep.
		</dd>
	

	

	
		

		--apply</code>
		</dt>
	

	
		
		Actually apply the deletion instead of running a simulation.
		</dd>
	

</dl>

debug indexiness</code></h2>
Prints a breakdown of the indexiness calculation</a> for a given page.</p>
Accepted options are:</p>




	




	
		
		-c</code>, --database</code> <file></em>
		</dt>
	

	
		

		The crawler database to fetch the source data fir the indexiness calculation from.
		</dd>
	

	

	
		

		-u</code>, --url</code> <url></em>
		</dt>
	

	
		
		The URL to run the calculation for, if there are multiple, the latest instance will be used.
		</dd>
	

</dl>

debug query-crawl-log</code></h2>
Query entries from the crawl log</a> in the crawler database.</p>
Accepted options are:</p>




	




	
		
		-c</code>, --crawler-db</code> <file></em>
		</dt>
	

	
		

		The crawler database to delete crawl log entries from.
		</dd>
	

	

	
		

		--uuid</code> <uuid></em>
		</dt>
	

	
		

		Query by crawl log entry UUID.
		</dd>
	

	

	
		

		--host</code> <host></em>
		</dt>
	

	
		

		Filter the results by hostname.
		</dd>
	

	

	
		

		--url</code> <url></em>
		</dt>
	

	
		

		Filter the results by URL.
		</dd>
	

	

	
		

		--exit-code</code> <exit-code></em>
		</dt>
	

	
		
		Filter the results by exit-code</a>, both name and id are accepted.
		</dd>
	

</dl>

debug sqlite-version</code></h2>
Prints the SQLite version that this version of unobtanium is using.</p>
This command takes no options.</p>


About the unobtanium.rocks Crawler
2025-07-10T00:00:00+00:00
What is the Crawler for?</h2>
The crawler feeds the development instance of a fully stand alone search engine over at [https://unobtanium.rocks]. The resulting data is also used to test development versions of the unobtanium software. You can find the source on Codeberg</a>.</p>
How will the Crawler behave?</h2>
The crawler runs on the same machine as the frontend with public configuration</a>.</p>
The user agent is unobtanium.rocks (for https://unobtanium.rocks, {index} index)</code>, where {index}</code> is the name of the configuration file without the .toml</code> suffix that is resposible foe the request. Each user agent is it's own independent crawler process, the results are combined at the end.</p>
In general the crawler will start at a configured seed page and follow links from there, for subsequent crawls it will usually remember the last crawl.</p>
Crawls happen in runs that are started manually, usually every few weeks, in that time the crawler will try to dicover new pages and recrawl existing ones.</p>
Unobtanium is a fast, but polite crawler. It will try to crawl as fast as possible while still respecting the servers limits.</p>
The delay between requests will be at least the Crawl-Delay</code> set in the robots.txt</code> (capped at 180 seconds) file or a dynamically calculated minimum that is at least the time it took for the Server to respond, this way, if the Server slows down, the crawler also slows down. The crawler will react to HTTP 429 reponses. Details on the crawl delay algorithm page</a>.</p>
In case the connection itself fails for whatever reason the crawler will immedeately try to send a second request. This behaviour is because failing connections are suprisingly common and usually work on the second try. Unontanium calls these Fluke events</a>.</p>
The crawler will stop once it runs into a preconfigured limit of operations/requests per crawl run or when it finds no more crawlable pages.</p>
Why is the Crawler trying to Access Private Resources?</h2>
Most likely this is because it found a link that points there and no robots.txt</code> rule exists to stop it.</p>
Pages that result in a 4xx</code> status code will not be indexed. Though the crawler will keep checking them on every recrawl as if they were dead links.</p>
If the matter seems bigger than some robots.txt entries to you please open an issue on the index repository</a>.</p>
robots.txt</h2>
In general the unobtanium crawler will use the part of its user agent up to the first space for matching the robots.txt</code>, for the unobtanium.rocks crawler this will be unobtanium.rocks</code> independent of whoch index the site is in.</p>
The robots.txt</code> file will be refected in a regular interval (~30 minutes) while crawling which makes it possible for web admins to directly stop or slow down the crawler while in case it is going where it shouldn't or is too fast.</p>
The first request of every crawl will be to fetch the robots.txt file.</p>
Who is Responsible for the Crawler?</h2>
Like the rest of unobtanium.rocks the unobtanium.rocks crawler is operated by Slatian</a>.</p>
See Also</h2>

Details about the crawling algorithm</a></li>
Configuring the User-Agent</code> for crawler Admins</a></li>
The crawler manual for Admins</a></li>
The index configuration for Unontanium.rocks</a></li>
</ul>


Configuring SearxNG to use Unobtanium
2025-02-23T00:00:00+00:00
SearxNG</a> is a metasearch engine that can combine the results of multiple search engines.</p>
To configure an Unobtanium instance for searx one can use the json_engine</code></a> module and the fact that all HTML views of unobtanium can be turned into an API by passing in format=json</code> as an additional URL query parameter.</p>
Add the following to the engines:</code></a> list in the settings.yaml file of SearxNG and replace unobtanium.rocks</code> with the domain name of your preferred instance for the search_url</code> and website</code>.</p>
  - name: unobtanium
    engine: json_engine
    shortcut: uo
    categories: general
    paging: true
    search_url: https://unobtanium.rocks/search?format=json&search={query}&page={pageno}
    first_page_num: 0
    results_query: list
    title_query: title
    url_query: url
    content_query: description
    categories: [general, web]
    timeout: 3
    about:
      website: https://unobtanium.rocks
      use_official_api: true
      require_api_key: false
      results: JSON
</code></pre>
Depending on how big your database and how fast your server is you may want to adjust the timeout</code> option. While in general unobtanium is pretty fast for most queries, it is possible to submit queries that take a long time to complete.</p>


Base Database
2025-02-09T00:00:00+00:00
The base database schema is shared between the crawler database and the summary database and provides basic facilities for storing data that is always needed in the context of a search engine.</p>
Note:</b> While the base database schema is often referred to as "The base database" it never is a standalone database. Its always embedded into another database.</p>
Tables in the base database are:




	




	
		
		unobtanium_database_info</code>
		</dt>
	

	
		

		Used to store global database metadata.
		</dd>
	

	

	
		

		origin</code>
		</dt>
	

	
		

		URL Origins, queryable by component
		</dd>
	

	

	
		

		url</code>
		</dt>
	

	
		

		URLs, queryable by component
		</dd>
	

	

	
		

		mimetype</code>
		</dt>
	

	
		

		Mimetypes/Mediatypes
		</dd>
	

	

	
		

		mime_parameter</code>
		</dt>
	

	
		
		Extends the mimetype</code> table with SQL queryable parameter information
		</dd>
	

</dl>

</p>
Tables</h2>
unobtanium_database_info</code></h3>
Added in version 0.1.0 the unobtanium_database_info</code> table has the following fields:</p>




	




	
		
		key</code>
		</dt>
	

	
		
		Text</a> Primary Key</span>
		</dd>
	

	

	
		

		value</code>
		</dt>
	

	
		

		Text</a>  "Schemaless" configuration data
		</dd>
	

</dl>

The main purpose of this table is storing global configuration information about the database and its schema.</p>
See also: Database Info</a></p>
origin</code></h3>
The origin</code> table has the following fields:</p>




	




	
		
		origin_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>
		</dd>
	

	

	
		

		port</code>
		</dt>
	

	
		

		Integer</a> Null</span> The port as specified in the URL or a well known default
		</dd>
	

	

	
		

		scheme</code>
		</dt>
	

	
		

		Varchar</a>  The scheme as specified in the URL
		</dd>
	

	

	
		

		host</code>
		</dt>
	

	
		

		Varchar</a> Null</span> The hostname as specified in the URL, if present
		</dd>
	

	

	
		

		str_origin</code>
		</dt>
	

	
		

		Text</a> Unique</span> The origin as a string, as if it was an URL consisting only of scheme</code>, host</code> and port</code>, main purpose is deduplication.
		</dd>
	

	

	
		

		crawl_delay_ms</code></s>
		</dt>
	

	
		

		Integer</a> Null</span> Delay in milliseconds that the crawler should wait between requests, taken from robots.txt
		</dd>
	

	
		

		Deprecated</strong> since 0.0.0: This is crawler specific information that shouldn't be part of the base database.
		</dd>
	

	
		

		Removed</strong> in 0.1.0
		</dd>
	

</dl>

There is a unique constraint across the fields port</code>, scheme</code> and host</code>.</s> (Removed in Version 0.1.0)</p>
See the Origin data type</a> for more information.</p>
url</code></h3>
The url</code> table has the following fields:</p>




	




	
		
		url_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>
		</dd>
	

	

	
		

		origin_id</code>
		</dt>
	

	
		

		Integer</a>  Reference into the origin</code> table
		</dd>
	

	

	
		

		path</code>
		</dt>
	

	
		

		Text</a> Null</span> Path part of the URL, if present
		</dd>
	

	

	
		

		query</code>
		</dt>
	

	
		

		Text</a> Null</span> Query part of the URL, if present
		</dd>
	

	

	
		

		username</code>
		</dt>
	

	
		

		Text</a> Null</span> Username part of the URL, if present
		</dd>
	

	

	
		

		password</code>
		</dt>
	

	
		

		Text</a> Null</span> Password part of the URL, if present
		</dd>
	

	

	
		

		str_url</code>
		</dt>
	

	
		

		Text</a> Unique</span> String representation of the whole URL for easy retrieval
		</dd>
	

</dl>

The url</code> table stores URLs that point to documents, notably it does not store the fragment part of an URL, since that points to somewhere inside a document.</p>
Fragment fields are stored together with URL ids where appropriate.</p>
See the URL data type</a> for more information.</p>
mimetype</code></h3>
The mimetype</code> table has the following fields:</p>




	




	
		
		mimetype_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span>
		</dd>
	

	

	
		

		mime_type</code>
		</dt>
	

	
		

		Text</a>  The part before the /</code>
		</dd>
	

	

	
		

		mime_subtype</code>
		</dt>
	

	
		

		Text</a>  The part after the /</code>
		</dd>
	

	

	
		

		mime_suffix</code>
		</dt>
	

	
		

		Text</a> Null</span> The part after the +</code> if present
		</dd>
	

	

	
		

		charset</code>
		</dt>
	

	
		

		Text</a> Null</span> The charset</code> parameter. This is in here since it is by far the most common and webservers love sending it.
		</dd>
	

	

	
		

		str_mimetype</code>
		</dt>
	

	
		

		Text</a> Unique</span> String representation of the whole mimetype with the parameters sorted in alphabetical order.
		</dd>
	

</dl>

mime_parameter</code></h3>
The mime_parameter</code> table has the following fields:</p>




	




	
		
		mime_parameter_id</code>
		</dt>
	

	
		
		Integer</a> Primary Key</span> Internal database id for the mime parameter, not used.
		</dd>
	

	

	
		

		mimetype_id</code>
		</dt>
	

	
		

		Integer</a>  Reference into the mimetype</code> table.
		</dd>
	

	

	
		

		key</code>
		</dt>
	

	
		

		Text</a>  Key of the mimetype parameter
		</dd>
	

	

	
		

		value</code>
		</dt>
	

	
		

		Text</a>  Value of the mimetype parameter
		</dd>
	

</dl>

Version history</h2>
0.0.0 - The last unversioned</h3>
Version 0.0.0 represents the last unversioned version of the database schema.</p>
0.1.0 - Basic cleanup (2025-04-10)</h3>
Version 0.1.0 cleans up some historic design choices.</p>

For the origin</code> table add a unique str_origin</code> and remove the unique constraint from scheme</code>, host</code> and port</code>, which had some edge case problems because of host</code> and port</code> being nullable</a>.</li>
Remove the crawl_delay_ms</code> column from the origin</code> table. It stems from a very early version of the crawler and unused.</li>
</ul>


Crawl Summary
2025-01-06T00:00:00+00:00
This document describes the crawl summary data structure stored in the summary database and is part of the entity data tree</a>.</p>
It is almost directly derived</a> from the crawl log</a>.</p>
A crawl summary has the following fields:</p>




	




	
		
		crawl_type</code>
		</dt>
	

	
		
		Which kind of crawl was performed. (Same as in the crawl log</a>)
		</dd>
	

	

	
		

		crawl_uuid</code>
		</dt>
	

	
		

		UUID to indetify a crawl across databases, taken from crawl log.
		</dd>
	

	

	
		

		crawl_time</code>
		</dt>
	

	
		

		Taken from time_started</code> in the crawl log.
		</dd>
	

	

	
		

		agent_uuid</code>
		</dt>
	

	
		

		The uuid indentifying the agent that did the crawling.
		</dd>
	

	

	
		

		exit_code</code>
		</dt>
	

	
		

		The crawl exit code</a> that summarizes the outcome of the
		</dd>
	

	

	
		

		server_last_modified</code>
		</dt>
	

	
		

		When the resource was last modified according to the server (UTC timestamp)
		</dd>
	

	

	
		

		request_duration_ms</code>
		</dt>
	

	
		

		How long the request took in milliseconds
		</dd>
	

	

	
		

		was_robotstxt_approved</code>
		</dt>
	

	
		

		Wheter the request was approved by robots.txt.
		</dd>
	

	

	
		

		http</code>
		</dt>
	

	
		

		One optional http summary
		</dd>
	

</dl>

The URL is not prt of this data structure and is assumed to match the entity generation</a> URL.</p>
HTTP Summary</h2>
The http summary fields are:</p>




	




	
		
		status_code</code>
		</dt>
	

	
		
		The status code</a> returned by the server
		</dd>
	

	

	
		

		etag</code>
		</dt>
	

	
		

		Optional, ETag</a> retuened by the server
		</dd>
	

</dl>


Useful database queries
2025-01-06T00:00:00+00:00
Count Active entity generations in summary database</h2>
This query counts all entity generations that</a> are not marked as duplicate, have not been marked as EOL yet, have a non-empty title and at least a bit of text or secondary text in their text pile generation 1</a></p>
select count(*)
from entity_generation
inner join document_description
	on entity_generation.entity_generation_id = document_description.entity_generation_id
inner join text_pile
	on entity_generation.text_pile_id = text_pile.text_pile_id
where (
	(NOT entity_generation.marked_duplicate) AND
	entity_generation.confirmed_end_unix_utc is NULL AND
	(document_description.title != '') AND
	(text_pile.text != '' OR text_pile.secondary_text != '')
);
</code></pre>
Count and delete unused tokens in token index</h2>
This is for the experimental token index</a> that is generated using the regenerate-token-index</code> on the crawler.</p>
select count(*)
from token
where not exists (
	select 1
	from token_statistics
	where token_statistics.token_id = token.token_id
	limit 1
);
</code></pre>
And to delete them:</p>
delete
from token
where not exists (
	select 1
	from token_statistics
	where token_statistics.token_id = token.token_id
	limit 1
);
</code></pre>


Crawl exit code
2024-12-22T00:00:00+00:00
Exit codes enumerate the outcome of a crawling reequest on a high level, they have fixed integers. Referring to them by name should be preferred, the integer representation is mainly for copact storage in databases.</p>
Note:</b> Some of those exit codes are from a time when unobtanium had a very different and more database coupled archtecture, they are unused now, but remain here to keep the number reserved.</p>
Id</th> Name</th> Description</th></tr></thead>

-3</td> database_error</code></td> The crawl couldn't be finished because of a database error</td></tr>
-2</td> cancelled</code></td> The crawl was cancelled mid-way (it would also be okay to discard those)</td></tr>
-1</td> someone_stole_my_work</code></td> If a race condition was detected in a task queue</td></tr>
20</td> file_ingested</code></td> A file was crawled and the content ingested</td></tr>
29</td> file_of_unknown_type</code></td> A file was found at the requested location, but the crawler doesn't know what to do</td></tr>
31</td> permanent_redirect</code></td> The server indicated a redirect and hited that it isn't going away soon.</td></tr>
32</td> redirect</code></td> The server responded with a redirect, this may change in hte future</td></tr>
34</td> file_did_not_change</code></td> The server communicated that the file did not change since the last request</td></tr>
40</td> server_blamed_client</code></td> The server respose indicated that the client sent a request wrong, this includes authentication errors</td></tr>
41</td> file_gone</code></td> The Server indicated that the requested resource isn't available and won't come back</td></tr>
42</td> did_not_understand_answer</code></td> The crawler couldn't receive the servers answer because of a protocol error or encoding issue</td></tr>
44</td> file_not_found</code></td> The server communicated that there is no resource at the requested location, this may change in the future</td></tr>
49</td> rate_limited</code></td> The Server communicated that the crawler was going too fast</td></tr>
50</td> server_internal_error</code></td> The Server couldn't answer because of an internal error</td></tr>
100</td> connection_failed</code></td> Could not connect to the server</td></tr>
101</td> request_timeout</code></td> The server tok too long to repond to the request.</td></tr>
102</td> error_reading_response</code></td> A Problem occourred while reading the server response (i.e. unexpected connection termination)</td></tr>
170</td> blocked_by_robots_txt</code></td> At the request of the Servers robots.txt</code> the resource wasn't crawled</td></tr>
171</td> blocked_at_request_of_remote</code></td> The server requested to not crawl the given resource</td></tr>
172</td> blocked_origin_by_local_policy</code></td> Crawling was blocked by a local policy on the origin level</td></tr>
173</td> blocked_url_by_local_policy</code></td> Crawling was blocked by a local policy on the url level</td></tr>
174</td> blocked_by_challenge</code></td> The Server returned a challenge/captcha page of some sort</td></tr>
180</td> not_canonical</code></td> The content was discarded because it marked itself as not being canonical</td></tr>
181</td> duplicate</code></td> The resource was found to be a duplicate by the crawler or a post-processing stage</td></tr>
-999</td> unknown_error</code></td> Placehholder for Errors that don't have an exit code assigned yet.</td></tr>
</tbody></table>
Redirects</h2>
The codes redirect</code> and permanent_redirect</code> represent redirects.</p>
They may have meaningful metadata about the destination of the redirect attached.</p>
Blocked</h2>
The codes blocked_by_robots_txt</code>, blocked_at_request_of_remote</code>, blocked_origin_by_local_policy</code>, blocked_url_by_local_policy</code> represent crawls where either no request was sent or the returned data was discarded because the resource indicated that it didn't want to be indexed.</p>
The blocked_by_challenge</code> indicates that the crawler ran into a challenge/captcha page that is likely intended to keep bad bots out, this is unfortunate and not a clear signal. (The crawler should be stopped robots.txt</code> before letting it run into a challenge page)</p>
Could be a Fluke</h2>
The codes unknown_error</code>, connection_failed</code>, request_timeout</code> and error_reading_response</code> all can have temporary networking problems as a possible cause and are therefore worth retrying immedeately once</strong>.</p>
See the Fluke Event concept for more information.</a></p>
Contentful</h2>
The following codes are considered being contentful</i>:</p>

file_ingested</code></li>
file_of_unknown_type</code></li>
permanent_redirect</code></li>
redirect</code></li>
file_not_found</code></li>
file_gone</code></li>
not_canonical</code></li>
duplicate</code></li>
</ul>
These have in common that from this request the crawler explicitly learned something about a resource or its absence.</p>
The file_did_not_change</code> is excluded as the crawler explicitly didn't learn anything new from that.</p>


Crawl Log
2024-12-22T00:00:00+00:00
The crawl log is an append only table in the crawler database</a>, that stores when which URL was crawled, how long that crawl took and what the outcome was.</p>
Crawl Types</h2>
Id</th> Name</th> Description</th></tr></thead>

0</td> file_crawl</code></td> File crawl for general purpose indexing</td></tr>
1</td> robotstxt_fetch</code></td> Fetching of robots.txt</code></td></tr>
2</td> metadata_crawl</code></td> HEAD</code> only request no content was indexded</td></tr>
</tbody></table>


Unobtanium Overview
2024-12-07T00:00:00+00:00
Components</h2>
Unobtanium is made up of:</p>




	




	
		
		unobtanium</code> (lib-unobtanium)
		</dt>
	

	
		

		Main application library that implements most data-structures and database logic.
		</dd>
	

	

	
		

		criterium</code>
		</dt>
	

	
		

		Query framework for matching Data in memory and in the DB against criteria.
		</dd>
	

	
		

		API Documentation</a>
		</dd>
	

	

	
		

		unobtanium-crawler</code>
		</dt>
	

	
		

		Web Crawling and summarizing application of Unobtanium.
		</dd>
	

	
		

		Manual</a>
		</dd>
	

	

	
		

		unobtanium-viewer</code>
		</dt>
	

	
		

		Web frontend for querying an Unobtanium summary database.
		</dd>
	

	
		

		Manual</a>
		</dd>
	

	

	
		

		crawler database
		</dt>
	

	
		

		Database schema optimized for crawling.
		</dd>
	

	
		

		Schema Documentation</a>
		</dd>
	

	

	
		

		summary database
		</dt>
	

	
		

		Database schema optimized for querying/searching.
		</dd>
	

	
		

		Schema Documentation</a>
		</dd>
	

</dl>

Data Pipeline</h2>
The main Unobtanium data pipeline consists of three steps:</p>

Crawling</a> (Web to crawler database)</li>
Summarizing</a> (crawler database to summary database)</li>
Querying/Searching (summary database to curious Creature) (Query Syntax</a>)</li>
</ol>
Each step in independent of the previous, so no huge setup is needed to get Unobtanium working.</p>
Crawling and summarizing are decoupled to make iterating on code and configuration easier, as summarizing is a quite complex step.</p>


Duplicate Types
2024-11-18T00:00:00+00:00
In general duplicates are different files that have the same main content, that is relevant for indexing.</p>
Exact Duplicates</h2>
Exact duplicates are detected using the Text Pile Digest</a>, they are byte for byte duplicates of the original. Link destinations and Formatting (unless relevant for the Text Pile) are ignored.</p>
Self-Duplicates</h2>
Self-Duplicates are a special case of exact duplicates where a duplication is detected to content that came from the same URL earlier. If that is exactly the previous version unobtanium knows that the main content of the page hasn't changed, even if the Server didn't explicitly say so.</p>
!note: Important quirk:</b>
Self-Duplicates aren't treated as duplicates, but rather as "content didn't change".</p>
They are not recorded as duplicates.</p>


Summarizing
2024-11-14T00:00:00+00:00
Note:</b> The crawl loop is built into the unobtanium-crawler summarize</code></a> command.</p>
The summarizing algorithm takes a crawl database and turns it into the summary database, this algorithm processes all files that are available for search later.</p>
This is done by iterating all crawl log entries</a> and then deciding what to do with them.</p>
For easier understanding the algorithm is described as if every file was processed individually, in reality it is batched</a>.</p>
The algorithm is implemented in crawler/src/summarizer</a> as part of the crawler.</p>
Overview</h2>

Fetch file information from the crawl database</li>
Cross check with the summary database to only integrate non-integrated entities</li>
Turn the raw informtion from the crawler into scrape results</li>
Detect self-duplicates</a> and generate candidates for exact duplicate</a> detection</li>
Turn self duplicates into crawl summaries</a></li>
Generate metatadata for all non self-duplicates:

Crawl summaries</a></li>
Entity generations</a> to create</li>
Entity generations</a> to close</li>
Link summaries</li>
File summaries</li>
</ul>
</li>
Store all data derived from both self-duplicates and non self-duplicates into the summary database.</li>
Flag exact duplicates</a> using the duplicate candidates from earlier</li>
</ul>
Steps in Detail</h2>
Fetch file_info</code> from the crawl database.</p>
Fetch the corresponding crawl_log_entry</code></a></p>
Test if the summary database already has crawl summary</a> matching the the crawl_log_entry.uuid</code>, if yes that file has alredy been summarized in a previous run and the algorithm skips the file</strong>.</p>
Self-Duplicate Detection</h3>
Note:</b> At this point a lot of independet things will be started concurrently. This is mostly done to keep the batch iteration count low.</p>
Derive the file_summary</code> (text_pile</code> (deprecated) + document_desciption</code>), text_pile_ng</code> and link_summaries</code>. See Deriving Content</a>.</p>
If the summary database already contains a self duplicate Entity generation:</b></p>

Use its UUID as the entity_generation_uuid</code></li>
Add it to the mapping from file_id</code> to entity_generation_uuids</code></li>
</ul>
Otherwise, not a self-duplicate:</b></p>

Generate a new entity_generation_uuid</code></li>
Remember URL and text pile digest for duplicate detection later</li>
Remember file_summary</code> and link_summaries</code> for adding to the database later.</li>
</ul>
Endif</b></p>
Generate Entity Generations</h3>
Fetch request_info</code> for the file.</p>
If not a self duplicate:</b></p>
Generate a new entity_generation</code></a> from the data collected so far:</p>




	




	
		
		url</code>
		</dt>
	

	
		

		taken from file_info</code>
		</dd>
	

	

	
		

		uuid</code>
		</dt>
	

	
		

		The entity_generation_uuid</code> generated earlier.
		</dd>
	

	

	
		

		first_seen</code>
		</dt>
	

	
		

		Taken from when the request was started from request_info</code>
		</dd>
	

	

	
		

		last_seen</code>
		</dt>
	

	
		

		Same as first_seen</code>
		</dd>
	

	

	
		

		time_end_confirmed</code>
		</dt>
	

	
		

		Set to None</code>
		</dd>
	

	

	
		

		marked_duplicate</code>
		</dt>
	

	
		
		Set to false</code> (innocent until proven guilty)
		</dd>
	

</dl>

Store entity_generation</code> into the database.</p>
Note:</b> At this point things in the database can be connected with the entity_generation_uuid</code> in the database.</p>
Close any old Entity generation based on URL and the first_seen</code> time.</p>
Endif</b></p>
Store file_summary</code> and link_summaries</code> into the database.</p>
Integrate Crawl Information</h3>
If there is a request, associated with the file:</b></p>
The crawl_summary</code></a> is generated from the information in request_info</code> and crawl_log_entry</code></a>.</p>




	




	
		
		crawl_time</code>
		</dt>
	

	
		

		taken from request_info</code> time sent
		</dd>
	

	

	
		

		crawl_uuid</code>
		</dt>
	

	
		

		taken from request_info</code>
		</dd>
	

	

	
		

		agent_uuid</code>
		</dt>
	

	
		

		resolved from the crawl_log_entry</code>
		</dd>
	

	

	
		

		crawl_type</code>
		</dt>
	

	
		

		taken from the crawl_log_entry</code>
		</dd>
	

	

	
		

		was_robotstxt_approved</code>
		</dt>
	

	
		

		taken from request_info</code>
		</dd>
	

	

	
		

		server_last_modified</code>
		</dt>
	

	
		

		taken from request_info</code>
		</dd>
	

	

	
		

		exit_code</code>
		</dt>
	

	
		

		taken from request_info</code>
		</dd>
	

	

	
		

		request_duration_ms</code>
		</dt>
	

	
		

		taken from request_info</code>
		</dd>
	

	

	
		

		http</code>
		</dt>
	

	
		
		taken from request_info</code>
		</dd>
	

</dl>

Store the crawl_summary</code> into the database.</p>
Endif</b></p>
Finding duplicates</h3>
To find duplicates query the database for entity generations that match the following criteria:</p>

Must not have a confirmed end time.</li>
Must be from the same origin as the original entity_generation</code></a> (as a proxy for being on the same website)</li>
Must have the same  text_pile_ng</code></a> digest as the text_pile_ng</code> of the original.</li>
</ul>
From the results plus the original entity_generation</code> the one with the shortest URL (other criteria are possible if better ones are available) is picked as the one to be marked as the original, all other entity generations in the list get marked as duplicates.</p>
Deriving Content</h2>
The text_pile</code>, text_pile_ng</code>, link_summaries</code> and document_description</code> are assembled from by fetching file content from the database and running an appropriate scraping algorithm.</p>
Todo:</b> Link scraping algorithms here.</p>
The document_desciption</code> indexiness is calculated from the link_summaries</code> and document_desciption</code> using the indexiness algorithm</a>.</p>
The file_summary</code> is generated from the file_info</code>, document_description</code>, and text_pile</code>.</p>
Changes to the Algorithm</h2>
After release 3.0.0</code> the Text Pile (Gen1)</a> has been replaced by the Text Pile Gen2</a>.</p>


Text Pile Gen1
2024-10-31T00:00:00+00:00
The Text Pile Gen1</abbr> ("Legacy") is a datastructure for storing unformatted content text that is extracted from pages during the summarizing step</a>.</p>
!warning: Deprecated:</b> The Text Pile is deprecated after version 3.0.0</code> see the Text Pile Gen2</a> for the datastructure that replaced it.</p>
The purpose of the text pile is to have a data source for the full text search that can also be used to generate preview snippets from any relevant point in the text.</p>
The text pile has the following fields:</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	




	
		
		text</code>
		</dt>
	

	
		

		What has been determined the main file content, roughly one line per paragrph, also includes headlines, code, quotes etc.
		</dd>
	

	

	
		

		secondary_text</code>
		</dt>
	

	
		

		Like text</code>, but for everything that is not marked as the main file content.
		</dd>
	

	

	
		

		big_headlines</code>
		</dt>
	

	
		

		Bigger headlines (down to level 2 or 3, depending on derivation algorithm), headlines are newline seperated.
		</dd>
	

	

	
		

		small_headlines</code>
		</dt>
	

	
		

		Headlines below the level of what goes into big_headlines</code>, same format.
		</dd>
	

	

	
		

		code_text</code>
		</dt>
	

	
		

		Text that was marked up as code.
		</dd>
	

	

	
		

		quote_text</code>
		</dt>
	

	
		
		Text that is marked up as being quoted.
		</dd>
	

</dl>

Note that text</code> and secondary_text</code> together contain all of the pages content, the other fields are effecitively duplicates of specific subsections for the purpose weighting them differently.</p>
If there is no text for a given field it must contain an empty string.</p>
Leading and trailing space characters must be stripped.</p>
Digest</h2>
To quicly detect and eliminate exact duplicates</a> ech Text Pile has a digest that is calculated using the Blake2b</a> 512bit algorithm.</p>
The hash is fed each field of the text pile, with a delimiter made up of three linebreaks between</b> each two fields \n\n\n</code>. If a field is empty it is fed as an empty string.</p>
They are fed to the has function in the following order:</p>

text</code></li>
secondary_text</code></li>
big_headlines</code></li>
small_headlines</code></li>
code_text</code></li>
quote_text</code></li>
</ul>


Calculating Indexiness
2024-10-28T00:00:00+00:00
Indexiness us a numeric score that helps to distinguish between Index and Leaf-pages. Where index pages are navigation heavy and the leaf pages have some content. The score is mostly derived from a pages link elements.</p>
A negative indexiness score means that a document is a leaf page (not very indexi), a positive score means that the document is a navigation page. A higher absolute score correlates with how confident the algorithm is that it is correct. The absolute score height also correlates with the number of links on a page.</p>
Relevant Data-Structures</h2>

Link Locality</a></li>
Location signature</a></li>
</ul>
What makes a Page an Index or a Leaf?</h2>
The following are some stereotype criteria for each page-type:</p>
Index pages have:</p>

Less outlinks</li>
No publishing date</li>
Inlinks in lists</li>
Inlinks in headers</li>
</ul>
Leaf pages have:</p>

More outlinks</li>
Selflinks in headers (to get link-able headers)</li>
Selflinks in lists (table of contents)</li>
A publishing date</li>
</ul>
There are other criteria too, like the paragraph size/count that readability.js</a> uses, but the above seem enough to classify most pages correctly.</p>
Metrics</h2>
To calculate the indexiness the following metrics have to be known about a document.</p>
Metric</th> How to get</th></tr></thead>

has_publishing_date</td> Did the crawler find a publishing date in the page metadata?</td></tr>
selflinks_in_headers</td> How many selflinks either are in a headline or contain a headline?</td></tr>
selflinks_in_lists</td> How many selflinks are in some kind of list?</td></tr>
inlinks_in_headers</td> How many inlinks either are in a headline or contain a headline?</td></tr>
inlinks_in_lists</td> How many inlinks are in some kind of list?</td></tr>
number_of_outlinks</td> How many outlinks are on the page?</td></tr>
</tbody></table>
Counting Links</h3>
For being relevant indexness a link must fullfill all of the following criteria:</p>

It is a link intended for document navigation (HTML a</code>-tag)</li>
Not in_nav</code></li>
Not in_aside</code></li>
One of:

in_main</code></li>
in_article</code></li>
Neither in_footer</code> or in_header</code></li>
</ul>
</li>
</ul>
Each relevant link only gets counted once, being relevant for the _in_headers</code> metrics beats being relevant for the _in_lists</code> metrics if both would apply.</p>
Note: the in nav/aside requirement might be lifted for selflinks to not penalize correctly marked up tables of content.</p>
Metrics to Indexiness</h2>
(selflinks_in_headers * -20 ) +
(selflinks_in_lists   * -5 ) +
(inlinks_in_headers   * 11 ) +
(inlinks_in_lists     * 5 ) +
(number_of_outlinks   * -1 ) +
if has_date_published { -25 } else { 0 } +
-10
</code></pre>
The metrics are weighted by how strong of an indicator they are compared to the weakest indicator.</p>
The headstart of -10</code> is chosen so that the page starts out relatively leafy, but one headline-inlink with no additional clues makes the score barely positive.</p>
Known Misclassifications</h2>
Inside Rust Blog, table based index (found: 2024-08-24)</h3>
https://blog.rust-lang.org/inside-rust/</p>
The index page makes use of tables, which are not recognized by the current algorithm iteration.</p>
A quickfix would be to consider links in tables equivalent to links in lists, though this will clash with other usage modes of tables.</p>
Another way of dealing with this is comparing the amount of table links to the total mount of relevant links and only apply that metric if a significant amount of links is inside tables. This threshold should be pretty high though.</p>
Note that tables are also often used on leafy pages for inlinks to other pages:</p>

https://www.freedesktop.org/software/libqmi/libqmi-glib/latest/libqmi-glib-DMS-Set-FCC-Authentication-response.html (gtk-doc genereated, current  correct negative indexiness)</li>
https://wiki.postmarketos.org/wiki/Qualcomm_Snapdragon_415/615/616_(MSM8929/MSM8939) (Table-heavy wiki page, current semi-correct indexiness of 6)</li>
</ul>
Blogger, multiple issues with blog entries fully replicated on index page (found: 2024-10-20)</h3>
https://www.righto.com/2019/</p>
The index pages basically have the full articles content here, including an <a href="https://static.…"><img …></a></code> construct. The links that make the images interactive shouldn't affect the indexiness algorithm and should be treated as part of the images.</p>
The alt text of the img</code> tags doesn't get counted as text on the link wrapped around the images, so these are basically empty.</p>
Quickfix would be ignoring links without text on them.</p>
Note:</b> After applying the quickfix the page still gets misclassified as a leaf-page with a score of -279 (instead of roughly -380 before).</p>


Crawl Loop
2024-10-28T00:00:00+00:00
Note:</b> The crawl loop is built into the unobtanium-crawler crawl</code></a> command.</p>
This document describes how the unobtanium Crawler works. Discovers new pages and deals with existing pages.</p>
Inputs:</p>

Crawler configuration</a>

Seed URLs</li>
Crawl Policies:

do_not_crawl</code></li>
url_query_paramters</code></li>
</ul>
</li>
</ul>
</li>
From Crawler Database:

Found links and recrawl information (Crawl Candidates)</li>
</ul>
</li>
</ul>
Outputs:</p>

To Crawler Database:

Found links and recrawl information (Crawl Candidates)</li>
Redirect information</li>
File metadata</li>
File content</li>
</ul>
</li>
</ul>
The crawler splits itself up into one crawl loop for each URL origin that is configured (OriginCrawler), this makes internals simpler as the data for one site is neatly kept together.
Which origins are crawled is derived from the seed-URLs. If a found link has the same origin as a configured seed, it will be considered for crawling.</p>
Crawling stops after a preconfigured number of crawling actions have happened or if there are no more URLs that need crawling.</p>
Example: If the crawler is configured for example.org</code> and finds a link to example.com</code> it won't crawl that link with it's example.org</code> crawler. If example.com</code> also happens to be configured, it will be crawled as part of that, otherwise the link will not be used.</p>
Crawl Candidates</h2>
Crawl Candidates are all URLs that are considered for crawling along with optional recrawl information, they are kept in the Crawler Database.</p>
Initial crawl candidates are the seed URLs and candidates from the previous crawls already in the database.</p>
Redirects</h2>
The crawler never follows redirects, it saves the fact that an URL redirected and marks the target as a Crawl Candidate.</p>
Crawl inhibitors</h2>
A URL might be discarded from crawling for a number of reasons:</p>

The URL doesn't share an origin with one of the configured seeds.</li>
The URL is excluded by a do_not_crawl</code> policy in the crawler configuration.</li>
The URL has a combination of query parameters not explicitly allowed by a url_query_parameters</code> policy.</li>
The robots.txt</code> file denies crawling</li>
The URL has already been crawled and is not due for a recrawl.</li>
</ul>
Crawl results will be discarded if:</p>

The crawled resource marks itself as non-canonical, the canonical URL will be marked as a crawl candidate.</li>
The crawled resource marks itself as not wanting to be indexed (using meta</code> robots</code> noindex</code>), the unobtanium crawler respects that.</li>
The page marks itself as not containing links that a crawler should follow (using meta</code> robots</code> nofollow</code>), in this case links aren't saved to the database.</li>
</ul>
The Crawl loop</h2>
All crawlers run simultanously, alternating between doing requests and idling to not overwhelm web servers</a>.</p>
The Origin Crawlers have the following state relevant for crawling:</p>

Their robots.txt information</li>
An expiry time for the robots.txt information</li>
A todo-list - initalized to the seed URLs</li>
A temporary ignore list in the database</li>
The time to wait between requests</li>
Patience counter - initialized to 5</code></li>
</ul>
Each iteration the crawler does one of three things:</p>
[1] Fetch the robots.txt</code> file</h3>
If there is no robots.txt file or after expiry of the old one (after 30 minutes, currently hard coded) the origin crawler tries to fetch the /robots.txt</code> file, parse it and store it. If no robots.txt is found the crawler assumes that it is okay to crawl.</p>
The time to wait between requests is set to the one given in the robots.txt if it contains a crawl delay.</p>
Note:</b> This is implemented by the DomainInformationLibrary</code> struct.</p>
[2] Fill the todo list</h3>
If the todo-list is empty the crawler will query crawl-candidates from the crawl database that:</p>

match the crawlers assigned origin</li>
have either never been crawled or are due for a recrawl</li>
are not on the temporary ignore list</li>
</ul>
All URLs that have been in this crawling stage are saved to a temporary ignore-list that is handled by the database to ensure no URL gets considered twice. This ensures that even if the recrawl interval is set very low the crawler "finishes" a site and only recrawls on it's next invocation, it also allows the crawler to recognize when it is finished.</p>
If the database doesn't have any URLs for crawling the origin crawler signals that it has finished and gets removed from the scheduler.</p>
It applies the do_not_crawl</code> policies, if an URL is denied by such a policy the crawler logs it as a crawl with the BLOCKED_URL_BY_LOCAL_POLICY</code> exit code.</p>
If the URL has a query part, the url_query_parameters</code> policies are evaluated. If no applicable policy allows the combination of URL parameters the crawler also logs the crawl as BLOCKED_URL_BY_LOCAL_POLICY</code>.</p>
It checks if the robots.txt</code>, if the URL shouldn't be crawled it gets logged as a crawl with the BLOCKED_BY_ROBOTS_TXT</code> exit code.</p>
[3] Crawl an URL</h3>
If there is an URL on the todo-list it is taken off and handed to the scraper for fetching and crawl-time scraping, this will extract URLs relevant for the crawler, such as from links (html a</code> elements), redirects and canonical URLs (in case a site marks itself as not canonical) and mark them as crawl candidates.</p>
After that happened the CrawlCandidate will be updated with information necessary for the recrawl to recognize if a resource changed since the last crawl.</p>
If the exit code indicates that the crawl has been rate-limited the tine to wait between requests is increased and the URL un-ignored so that it can be recawled in the same run.</p>
If the error seems like a temporary condition that might have been caused by the network, the URL is un-ignored so that it can be recrawled in the same run.</p>
If the conditions seems like it could persist the patience counter is decreased by one, when the counter reaches zero the origin crawler signals that is is "finished" because crawling with a broken connection or an otherwise offline site or server that speaks gibberish makes little sense.</p>
If a database error is encountered the patience counter is set to zero immediately.</p>
In any case the final crawl result is saved to the crawl log</a>.</p>
Notable Changes</h2>
2025-08-05:</strong> The crawlers now run concurrently as part of the Crawling at the speed of politeness</a> goal, this version introduced the politeness based delay algorithm. Before that the non concurrent crawlers were taking turns in a round robin like scheduling pattern.</p>


Databse Batching (Pattern)
2024-10-28T00:00:00+00:00
Batching up database queries is something you'll see a lot in unobtanium, but mostly in the summarizer wich should, in the best case process as many pages as possible as fast as possible.</p>
The reason why it is faster boils down to SQL parsing and Query overhead, which is the same, no matter if the database query is used once, or 1024 times, so just let the database do the work once and then let the main logic do some work before calling out to the database again.</p>
The methods on the database explicitly made for batching are suffixed _bulk</code>, otherwise they have the same name as the regular version (if there is one). These methods usually take a Vec</code> of data and then return a HashMap</code> of data.</p>
When something is implemented as batched the code alternates between building the list of arguments for the next database queries, possibly filtering them and doing the queries. Where it makes sense there are checks in place that make the algorithm abort early if all wirk has been filtered out (i.e. in the summarizer after checking for already integrated files).</p>
Implementation Considerations</h2>
When btching make sure you know your data-flow and be aware that batching may interfere with how data is selected.</p>
This may result in neccessary checks against the current batch in addition to checks against the database.</p>
A case where batching lead to an undesired edge case</a> is in the summarizing algorithm</a> where the test for self-duplicates</a> relies on testing against all already integrated data, but fails to check against the data that is in the pipeline which causes problems if the crawl results are shorter than the batch-size.</p>


Entity Data Tree
2024-10-28T00:00:00+00:00
Every Entity in the Unobtanium summary database may have multiple kinds of data attached to it, apart from the generation metadata all of that data is optional. Because the database doesn't know any Entity data structured directly they are uniquely identified by their entity generation UUID for organically grown reasons.</p>
An entity may have:</p>

one mandatory Entity Generation</a></li>
multiple Crawl Summaries</a>

one HTTP Summary</a></li>
</ul>
</li>
one File Summary

one Document Description</li>
one Text Pile Gen1</a>

multiple Token Statistics entries</li>
</ul>
</li>
</ul>
</li>
one Text Pile Gen2</a></li>
one Redirect Summary</li>
multiple Link Summaries</li>
</ul>
Note that Text Piles are deduplicated and in the actual database. An entity generation points to the text pile via its ID, this is abstracted away by the database code in lib-unobtanium.</p>


Entity Generation
2024-10-28T00:00:00+00:00
An Entity Generation is a concept and data structure used by Unobtanium to have some kind of space/time coordinate for any given version of queryable resource (an entity).</p>
Entity Generations live in the summary database</a> and are given a UUID for unique and stable identification.</p>
Note:</b> Recommended are UUID v7 or v4 as they both have huge random components. v7 is interesting for data transparency as it in theory allows tracing when the data was generated.</p>
See also:</p>

The Entity Data Tree Concept</a></li>
EntityGeneration on docs.rs</a></li>
</ul>
Open / Closed State</h2>
Each entity generation has a few timestamps:</p>

first_seen</code></li>
last_seen</code></li>
time_end_confirmed</code></li>
</ul>
The first_seen</code> and last_seen</code> timestamps are always set and can be used to determine when the entity generation definitely existed in the recorded form.</p>
As long as the entity can be assumed to exist in the recorded form the entity generation is considered open</i>.</p>
The time_end_confirmed</code> timestamp is set when unobtanium learns that the entity has changed. When that happened the entity generation is considered closed</i>.</p>
Fields</h2>




	




	
		
		url</code>
		</dt>
	

	
		

		Url</code> The url under which the given entity lives
		</dd>
	

	

	
		

		uuid</code>
		</dt>
	

	
		

		Uuid</code> The UUID that identifies the entity generation
		</dd>
	

	

	
		

		first_seen</code>
		</dt>
	

	
		

		UtcTimestamp</code> First known existence of this generation (may change if better data is integrated)
		</dd>
	

	

	
		

		last_seen</code>
		</dt>
	

	
		

		UtcTimestamp</code> Last known existence, may equal first_seen</code>, updated every time a newer request confirms that the entity generation is still open.
		</dd>
	

	

	
		

		marked_duplicate</code>
		</dt>
	

	
		
		bool</code> Caches the current duplicate</a> status to avoid uneccessary queries to the duplicate summary table.
		</dd>
	

	

	
		

		time_end_confirmed</code>
		</dt>
	

	
		

		Option<UtcTimestamp></code> If set the entity generation is considered no longer live, set to the time that this has been confirmed. (Usually the first_seen</code> of the next entity generation) In cases of temorary outages this might reset back to None</code> when the entity comes back.
		</dd>
	

</dl>

Updates</h2>
2024-10-12 Improved Timestamping</h3>
Removed fields:</p>

time_started</code></li>
known_lifetime_seconds</code></li>
</ul>
Added fields:</p>

first_seen</code></li>
last_seen</code></li>
time_end_confirmed</code></li>
</ul>
Before this update every entity generation had a time_started</code> and a known_lifetime_seconds</code>, this was put in place before unobtanium was really able to work with multiple versions of entity generation for one URL. As it turns out these are quite impractical to work with.</p>
They were replaced by start and end times for the range that was observed "seen" and a definitive end time of the entity generation at which it was confiemed to be no longer there.</p>
2024-10-26 Duplicate Marker</h3>
Added fields:</p>

marked_duplicate</code></li>
</ul>
A duplicate marker has been added that follows the current duplicate status for an entity generation as that saves a lot of query time during search.</p>


Link Locality
2024-10-28T00:00:00+00:00
LinkLocality</a> relates the URL of a link of a document to the link destination.</p>
Link locality is a tool for analyzing a sites structure, it is not intended or fit for security purposes</strong>.</p>
Possible Values are:</p>
Id</th> Name</th></tr></thead>

0</td> SelfLink</code></td></tr>
1</td> InLink</code></td></tr>
2</td> OutLink</code></td></tr>
</tbody></table>
Self-links refer to the same document as they are on, they only change the fragment</code>.</p>
In-links stay on the same site, they are allowed to change the path</code>, query</code> and fragment</code>. In addition they are allowed to switch between the encrypted version of a protocol and the non-encrypted one. (The default implementation allows appending or removing an s</code> suffix in the scheme</code> independent of the used protocol)</p>
Out-links are all other links. Links without a hostname are also considered outlinks by the default implementation. They are also the sane default to fall back to in case of doubt or lack of information.</p>
Please note that the default derivation implementation is currently made for websites and is intended to be wrapped by a custom implementation should the need arise.</p>


Location Signature
2024-10-28T00:00:00+00:00
The Location Signature is a datastructure for describing where — semantically speaking — an element is in a (HTML) document. It is implemented as a bunch of flags that get set according to the elements parent-elements.</p>
See LocationSignature on docs.rs</a>.</p>
The fields are named after the corresponding HTML element.</p>
Field</th> HTML-Tags</th> Notes</th></tr></thead>

in_header</code></td> header</code></td> </td></tr>
in_footer</code></td> footer</code></td> </td></tr>
in_aside</code></td> aside</code></td> </td></tr>
in_nav</code></td> nav</code></td> </td></tr>
in_form</code></td> form</code></td> </td></tr>
in_main</code></td> main</code></td> </td></tr>
in_article</code></td> article</code></td> 1</td></tr>
in_section</code></td> section</code></td> 2</td></tr>
in_table</code></td> table</code></td> </td></tr>
in_figure</code></td> figure</code></td> </td></tr>
in_address</code></td> address</code></td> </td></tr>
in_code</code></td> code</code></td> </td></tr>
in_headline</code></td> h1</code> – h6</code></td> </td></tr>
in_list</code></td> ul</code>, ol</code>, dl</code></td> </td></tr>
in_paragraph</code></td> p</code></td> </td></tr>
</tbody></table>
The location signature is mainly used to extract the purpose of a link.</p>




	
		
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	




	
		
		1
		</dt>
	

	
		

		Depending on document structure, if no in_main</code> is present in_article</code> might be a "good enough" alternative.
		</dd>
	

	

	
		

		2
		</dt>
	

	
		

		section</code> has no real semantic meaning, but it might be useful to separate useful content from fluff on sites that have little to no other semantic markup.
		</dd>
	

</dl>

Serialized Representation</h2>
The Location signature is always represented as key-value pairs, fields set to false are not serialized.</p>


URL
2024-10-28T00:00:00+00:00
Unobtanium makes heavy use of the url crate</a> and inherits its way of representing parsed URLs.</p>
Also see: The URL Standard</a></p>
URL Component</th> Datatype</th> If absent</th></tr></thead>

scheme</code></td> str</td> empty</td></tr>
username</code></td> str</td> empty</td></tr>
password</code></td> Option<str></td> empty</td></tr>
host</code></td> Option<str></td> None</td></tr>
port</code></td> Option<u16></td> None</td></tr>
path</code></td> str</td> empty</td></tr>
query</code></td> Option<str></td> None</td></tr>
fragment</code></td> Option<str></td> None</td></tr>
</tbody></table>
All values are taken as is, for "special" URLs (mostly http</code> and https</code>) the hostnames are punycode encoded, otherwise they are percent encoded.</p>
Origin</h2>
Unobtanium does not follow the url crate when it comes to the URL origin implementation. For unobtanium the Origin</a> is the scheme</code>, host</code> and port</code> taken from the URL and put into a struct. For http</code>, https</code>, ws</code>, wss</code> and ftp</code> the default port numbers are automatically filled in if no port is mentioned in the URL using the port_or_known_default() getter</a>. If two origins are represented using equal data, they equal each other, this simplifies matching and comparing logic across the whole codebase.</p>
Criteria</h2>
URLs and Origins can be matched using the Url Criterium</a> and Origin Criterium</a>. Their naming and types follow the above table.</p>


Crawler User-Agent and robots.txt
2024-10-28T00:00:00+00:00
Crawling responsibly on the web, also means giving other a chance to indicate how they want to be crawled, if at all.</p>
Since the unobtanium-crawler can be used for multiple purposes it allows configuring it's user agent, either in the crawl configuration or via a command line option to the run</code> subcommand.</p>
If left unconfigured the crawler uses the user agent unobtanium-crawler-unconfigured-ua</code>, outside of temporary testing, use of this user agent is heavily discouraged</strong>.</p>
When setting you own user agent please follow the following pattern:</p>
{your-project}-crawler (for {url})
</code></pre>
Where {your-project}</code> is a representative name of what you are using the data for and {url}</code> leads to a website describing or showing what you are using the data for.</p>
An example would be:</p>
example-net-crawler (for https://example.net/why-we-crawl-your-website)
</code></pre>
robots.txt</h2>
Everything before the first space in the user agent (i.e. example-net-crawler</code>) will be used to match against the robots.txt</code> file, if you are an operator you can just copy that part over.</p>
If the crawler finds itself not addressed by name in the robots.txt it follows the rules set for all bots (*</code>).</p>
The crawler re-fetches the robots.txt file every 30 minutes, so stopping a rouge crawler is possible within a reasonable timeframe. Sending it 429</code> response codes will also increase the crawl delay slightly every-time the reponse hits. Alternatively a crawler can be stopped by simulating an outage with 503</code> responses, it will give up within a few requests.</p>
Note:</b> Unobtanium is free software, and while responsible use is encouraged, it can be modified to not respect all of the above. Such modifications while not forbidden are heavily discouraged</strong> by the developer.</p>
How to set the user agent for unobtanium?</h2>
There are two ways to set the user agent for the unobtanium crawler:</p>

The recommended way is to use the user_agent</code></a> option in the configuration file.</li>
For testing you can use the --user-agent</code> command line option on the crawl</code> subcommand</a>.</li>
</ul>


Terminology
2024-10-28T00:00:00+00:00
This page describes some terms used across Unobtanium which have a specific meaning within Unobtanium.</p>
Outside of Unobtanium you might see these terms used more broadly.</p>
Terms referring to things outside of Unobtanium</h2>




	
		
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	




	
		
		Homepage
		</dt>
	

	
		

		A webpage that is the entry-point and main navigation hub for a website. Each site may only have one homepage. It is usually linked in the header of each page on a name or icon representing the site.
		</dd>
	

	

	
		

		Webpage
		</dt>
	

	
		

		A single Document with one main file and maybe some auxiliary files that can usually be fetched over HTTP(s).
		</dd>
	

	

	
		

		Website/Site
		</dt>
	

	
		

		A website is a collection of one or more pages that belong together in some sense (even if it is just a collection of random pages). That collection usually is the set of all pages sharing the same origin, though this may not always be the case (i.e. the same site being accessible via HTTP and HTTPS or a public Unix system where each /~user/</code> path has its own website, a big web presence available in many languages may also be considered one site per language.)
		</dd>
	

	

	
		
		URL-Origin/Origin</a>
		</dt>
	

	
		

		With an URLs Origin Unobtanium refers to the combination of the protocol, hostname and port fields. Origins which omit the port and origins which explicitly specify the standard port for an URL are considered equivalent.
		</dd>
	

</dl>

Terms referring to things inside Unobtanium</h2>




	
		
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	




	
		
		Crawl Policy</a>
		</dt>
	

	
		

		A crawl policy is a rulue that configured the Crawl Loop</a>, mostly regarding which pages it is allowed to crawl.
		</dd>
	

	

	
		

		Entity
		</dt>
	

	
		

		An entity refers to a (search-)queryable entry in the Unobtanium database. Each entity is uniquely identified by its entity-generation-UUID.
		</dd>
	

	

	
		

		Entity-Generation
		</dt>
	

	
		

		An entity-generation refers to the generation metadata attached to an entity describing its location (URL) and lifetime. If the result from querying a URL changes in a way that is significant to Unobtanium a new entity-generation is created.
		</dd>
	

	
		

		Since "entity-generation" is pretty long it is abbreviated eg</code> in some places.
		</dd>
	

	

	
		

		Exact Duplicate</a>
		</dt>
	

	
		

		A Page with exactly the same main-content as another
		</dd>
	

	

	
		

		Fluke-Event</a>
		</dt>
	

	
		

		An unlikely and temporary error where immediately retrying is a valid strategy.
		</dd>
	

	

	
		

		Patience</a>
		</dt>
	

	
		

		Patience is a metric in the crawler represented by a countdown, to make the crawler eventually give up on unreachable/broken origins.
		</dd>
	

	

	
		

		Self-Duplicate</a>
		</dt>
	

	
		

		Self-Duplicates are exact duplicates, but they came from the same URL. They are used as an indicator of whether a page has changed or not.
		</dd>
	

</dl>

Terms referring to actions inside Unobtanium</h2>




	
		
			
		
	
		
			
			
				
			
		
	
		
	
		
			
		
	
		
			
			
		
	
		
	
		
			
		
	
		
			
			
		
	




	
		
		Crawling</a>
		</dt>
	

	
		

		Crawling is the process of collecting resurces.
		</dd>
	

	

	
		

		Scraping
		</dt>
	

	
		

		Scraping is the process of extracting useful data from raw data that is mainly aimed at human readers. The raw data usually comes from a crawling step.
		</dd>
	

	

	
		

		Summarizing</a>
		</dt>
	

	
		

		Summarizing is turning a lot of raw information into a usable and searchable summary. This usually involves scraping, but also other steps.
		</dd>
	

</dl>

Unobtanium Documentation

Your first search engine

Setting up</h2> To set up create a folder my-first-unobtanium</code>, everything in this tutorial will happen inside it. (The exact name isn't important, but this tutorial is going to reference it a lot)</p> Now to set up run the following commands:</p>

File overview</h2> An overview of all files created in this guide:</p>

Text Pile

Text Pile Gen2

Text Pile (Disambiguation)

Selfhosting Unobtanium

What you need to do, overview</h2> Bootstrapping a search engine:</p>

Crawl Data Tree

Token Index

Crawler Database

Overview</h2> Tables in the crawler database are:</p> agent</code> </dt> Information about crawling agents </dd> crawl_log</code> </dt>

Tables</h2>

Possible future changes</h2> Addition of UUID fields to the tables request</code>, file</code> and redirect</code></li>

Version history</h2>

1.0.0 - De-facto stable</h3> This schema has been de-facto stable for a while and been assigned the 1.0.0 version with the introduction of database versioning.</p>

Database Data Types

Integer</h2> Integers are always stored using the SQLite INTEGER</code> datatype, the rust equivlent is i64</code>, while always encoded as a signed integer negative values don't always make sense here.</p> Integers are usually used to encode some other data.</p>

Blob</h2> Blobs are represented as BLOB</code> types in SQLite. They can be used to store arbitrary binary information.</p> The format must</strong> be documented.</p>

Information Source

Link destination type

Summary Database

Overview</h2> Tables in the base database are:</p> entity_generation</code> </dt>

Tables</h2>

Version History</h2>

0.0.0 - The first versioned</h3> This is the first versioned and documented version of this database.i</p> This schema is released with unobtanium 3.0.0.</p>

unobtanium-viewer

Synopsis</h2> unobtanium-viewer [OPTIONS...] </code></pre> Options</h2> Accepted options are:</p>

Options</h2> Accepted options are:</p>

HTTP headers</h2> It is recommended to set the following security headers in your reverse proxy:</p>

Database Info

Crawl Delay between Requests

Calculating the Crawl Delay</h2> Unobtanium chooses from multiple delays:</p> A minimum delay</li>

Boolean

Criterium Chain

Number

Origin

String

Url

Crawler crawl configuration

Crawl Policy Criterium</h3> The crawl policy criterium can match page metadata that is available before</strong> the request happened at the scheduling stage.</p> The crawl policy criterium has the follwing variants</p>

unobtanium-crawler

optimize-db</code></h2> Runs the SQLite internal analyze and optimize commands on the given crawler or summary database.</p> Accepted options are:</p> -c</code>, --database</code> <file></em> </dt>

debug sqlite-version</code></h2> Prints the SQLite version that this version of unobtanium is using.</p> This command takes no options.</p>

About the unobtanium.rocks Crawler

Configuring SearxNG to use Unobtanium

Base Database

Tables</h2>

Version history</h2>

0.0.0 - The last unversioned</h3> Version 0.0.0 represents the last unversioned version of the database schema.</p>

0.1.0 - Basic cleanup (2025-04-10)</h3> Version 0.1.0 cleans up some historic design choices.</p>

Crawl Summary

HTTP Summary</h2> The http summary fields are:</p> status_code</code> </dt>

Useful database queries

Crawl exit code

Redirects</h2> The codes redirect</code> and permanent_redirect</code> represent redirects.</p> They may have meaningful metadata about the destination of the redirect attached.</p>

Crawl Log

Unobtanium Overview

Data Pipeline</h2> The main Unobtanium data pipeline consists of three steps:</p>

Duplicate Types

Summarizing

Overview</h2> Fetch file information from the crawl database</li> Cross check with the summary database to only integrate non-integrated entities</li> Turn the raw informtion from the crawler into scrape results</li>

Steps in Detail</h2> Fetch file_info</code> from the crawl database.</p>

Finding duplicates</h3> To find duplicates query the database for entity generations that match the following criteria:</p> Must not have a confirmed end time.</li>

Text Pile Gen1

Calculating Indexiness

1.0.0 - De-facto stable</h3>
This schema has been de-facto stable for a while and been assigned the 1.0.0 version with the introduction of database versioning.</p>

Blob</h2>
Blobs are represented as `BLOB</code> types in SQLite. They can be used to store arbitrary binary information.</p>`
`The format must</strong> be documented.</p>`

0.0.0 - The first versioned</h3>
This is the first versioned and documented version of this database.i</p>
This schema is released with unobtanium 3.0.0.</p>

`debug sqlite-version</code></h2> Prints the SQLite version that this version of unobtanium is using.</p> This command takes no options.</p>`

0.0.0 - The last unversioned</h3>
Version 0.0.0 represents the last unversioned version of the database schema.</p>

HTTP Summary</h2>
The http summary fields are:</p>

`status_code</code> </dt>`
`Thestatus code</a> returned by the server </dd>`
`etag</code> </dt>`
`Optional, ETag</a> retuened by the server </dd> </dl>`

Redirects</h2>
The codes `redirect</code> and permanent_redirect</code> represent redirects.</p>`
`They may have meaningful metadata about the destination of the redirect attached.</p>`