Library
A personal research library that helps AI agents find source passages, inspect their context, and qualify claims across more than 300 documents.
We’ve graduated from the early days of fabricated citations. Today, the headache is misinterpretation of the attributed source. LLMs can cite literature with fluency. Readers need to pinpoint exactly what in the literature supports the claims, identify the boundaries of those claims, and translate them into design specifications.
I built Library so the agents I use can search my own collection as an initial screener. It holds more than 300 papers, books, and reference documents across learning science, measurement, statistics, and related fields. I use it to draft literature reviews, find citations, work through literature questions, and supply source material to Octo pipelines.
A question from the collection
Consider: “What does the literature say about worked examples in math education?” A quick summary might say that showing learners a completed solution helps them learn. A search of Library returns a more useful qualification in Clark and Mayer’s e-Learning and the Science of Instruction (2016).
The book discusses research in which worked examples helped less experienced learners with complex problems. In the study it describes, more experienced learners did better with practice. The chapter also discusses expertise reversal: guidance that helps a novice may cease to help as knowledge grows.
That qualification matters when deciding who needs an example, for which task, and at what stage of learning. The search gives me a place to start reading and a source to check. It does not make the instructional decision for me. The transfer boundary between knowledge and application still lives with the human.
How a question becomes evidence
Library searches papers and books in parallel, combining meaning-based and keyword matches. It gathers up to 80 candidate passages, then reranks them against the question to return eight hits.
The wide search helps find passages that may use different words for the same idea. Reranking then reads each candidate against the question and moves the more relevant ones to the top.
Read beyond the match
A short passage can be enough to locate a claim while leaving out the conditions around it. Each search hit points to a larger section that the agent can retrieve before using the passage. The diagram below illustrates that relationship; its text is an example, not a quotation from the worked-examples search.
The small passage makes the right part of a document easier to find. The larger section gives the reader a chance to check what the author compared, who was studied, and how narrowly the conclusion should be stated.
Getting sources into the library
Documents enter through a local command or a cloud upload. The pipeline extracts their text, separates papers from books, adds metadata, and builds the search index. It preserves section boundaries and filters out reference lists and indexes so a bibliography entry does not crowd out an explanation. Metadata extraction uses DeepSeek Flash, with Mistral Small as a fallback; Semantic Scholar can enrich paper metadata.
Library makes evidence easier to find and inspect. It cannot decide whether a study applies to a particular learner, settle a disagreement in the literature, or turn a retrieved passage into a justified claim on its own. The source still needs to be read.
Stack
| Part | Tool |
|---|---|
| Runtime and interface | AWS Lambda, API Gateway, MCP |
| Source documents | S3 |
| Text extraction | pymupdf4llm |
| Metadata | DeepSeek Flash, Mistral Small fallback, Semantic Scholar |
| Search | Pinecone, dense and sparse vectors |
| Embeddings and reranking | Voyage AI |