← All projects
300+ sources · learning science & methods active

Library

A personal research library that helps AI agents find source passages, inspect their context, and qualify claims across more than 300 documents.

We’ve graduated from the early days of fabricated citations. Today, the headache is misinterpretation of the attributed source. LLMs can cite literature with fluency. Readers need to pinpoint exactly what in the literature supports the claims, identify the boundaries of those claims, and translate them into design specifications.

I built Library so the agents I use can search my own collection as an initial screener. It holds more than 300 papers, books, and reference documents across learning science, measurement, statistics, and related fields. I use it to draft literature reviews, find citations, work through literature questions, and supply source material to Octo pipelines.

A question from the collection

Consider: “What does the literature say about worked examples in math education?” A quick summary might say that showing learners a completed solution helps them learn. A search of Library returns a more useful qualification in Clark and Mayer’s e-Learning and the Science of Instruction (2016).

The book discusses research in which worked examples helped less experienced learners with complex problems. In the study it describes, more experienced learners did better with practice. The chapter also discusses expertise reversal: guidance that helps a novice may cease to help as knowledge grows.

That qualification matters when deciding who needs an example, for which task, and at what stage of learning. The search gives me a place to start reading and a source to check. It does not make the instructional decision for me. The transfer boundary between knowledge and application still lives with the human.

How a question becomes evidence

Library searches papers and books in parallel, combining meaning-based and keyword matches. It gathers up to 80 candidate passages, then reranks them against the question to return eight hits.

"worked examples in math education" Embed the query Question → 1,024 numbers representing its meaning VOYAGE AI Search papers Hybrid dense + sparse Top 40 candidates PINECONE Search books Hybrid dense + sparse Top 40 candidates PINECONE in parallel Pool 80 candidates A wide net — some great, some mediocre, some off-topic Rerank Cross-encoder reads each passage with the query — keeps 8 VOYAGE RERANK Top 8 passages Each with a parent_id for full-section expansion Optional: retrieve_parent expands a hit into its full section

The wide search helps find passages that may use different words for the same idea. Reranking then reads each candidate against the question and moves the more relevant ones to the top.

Read beyond the match

A short passage can be enough to locate a claim while leaving out the conditions around it. Each search hit points to a larger section that the agent can retrieve before using the passage. The diagram below illustrates that relationship; its text is an example, not a quotation from the worked-examples search.

Child chunk What got searched and matched Worked examples reduce extraneous cognitive load by providing complete solutions for learners to study before practice. ~500 tokens High precision, low context retrieve_parent Parent section What the agent reads Before: setup of the cognitive load framework, expertise levels, the problem-solving baseline... Worked examples reduce extraneous cognitive load by providing complete solutions for learners to study before practice. After: experimental conditions, measured outcomes, faded scaffolding implications, comparison to direct practice. ~2-4 KB Same precision, full context

The small passage makes the right part of a document easier to find. The larger section gives the reader a chance to check what the author compared, who was studied, and how narrowly the conclusion should be stated.

Getting sources into the library

Documents enter through a local command or a cloud upload. The pipeline extracts their text, separates papers from books, adds metadata, and builds the search index. It preserves section boundaries and filters out reference lists and indexes so a bibliography entry does not crowd out an explanation. Metadata extraction uses DeepSeek Flash, with Mistral Small as a fallback; Semantic Scholar can enrich paper metadata.

Library makes evidence easier to find and inspect. It cannot decide whether a study applies to a particular learner, settle a disagreement in the literature, or turn a retrieved passage into a justified claim on its own. The source still needs to be read.

Stack

PartTool
Runtime and interfaceAWS Lambda, API Gateway, MCP
Source documentsS3
Text extractionpymupdf4llm
MetadataDeepSeek Flash, Mistral Small fallback, Semantic Scholar
SearchPinecone, dense and sparse vectors
Embeddings and rerankingVoyage AI