The Text Pile is a datastructure for storing unformatted content text that is extracted from pages during the summarizing step.
The purpose of the text pile is to have a data source for the full text search that can also be used to generate preview snippets from any relevant point in the text.
The text pile has the following fields:
text- What has been determined the main file content, roughly one line per paragrph, also includes headlines, code, quotes etc.
secondary_text- Like
text, but for everything that is not marked as the main file content. big_headlines- Bigger headlines (down to level 2 or 3, depending on derivation algorithm), headlines are newline seperated.
small_headlines- Headlines below the level of what goes into
big_headlines, same format. code_text- Text that was marked up as code.
quote_text- Text that is marked up as being quoted.
Note that text and secondary_text together contain all of the pages content, the other fields are effecitively duplicates of specific subsections for the purpose weighting them differently.
If there is no text for a given field it must contain an empty string.
Leading and trailing space characters must be stripped.