The idea
Extractium™ collects what an organization already publishes into one compendium, which is a searchable collection of its content saved as ordinary files. You can point it at a website, a knowledge base portal, GitHub repositories, a YouTube channel, a library repository, or a folder of files. It collects the content, prepares it for keyword search and meaning-based search, and saves it in several formats.
Inputs on the left, Extractium in the middle, and the four outputs one build writes.
How it works
A vector database needs a server to run. Extractium doesn't need a server, a database, or an API. Every output is a static file that you can host anywhere, including GitHub Pages, and one build writes all of them. Each source and each output is a plug-in.
Outputs
- A compressed JSON search index for keyword and meaning-based search.
- An
llms.txtfile plus an index file per source, for AI assistants that can read web pages but cannot call tools. - A SQLite database for SQL queries and reporting.
- A Markdown folder in the Open Knowledge Format, for reading and editing the content as ordinary files.
Context
Extractium started as the indexing engine in Field Station AI. Field Station AI is still an example of an app that reads the JSON output. I wrote it as part of my work at the Depression Center. The project is university property and open source, with a citable record at DOI 10.5281/zenodo.22754640.