Research Pipeline
ResearchRaven
An automated pipeline that turns web research into structured knowledge, and structured knowledge into publishable writing.
Research tools are good at retrieval and bad at what comes after it. They return more material than anyone can read, and leave nothing behind: the next question starts from zero. ResearchRaven is built around the opposite premise — that the durable artifact is the structure, not the search result.
The system collects sources on a topic, organizes what it finds into a navigable knowledge base, and writes from that base in whichever format is needed: an article, a newsletter, a chapter. Each of those steps is a separate stage with a defined boundary, which is what allows any one of them to be rebuilt without disturbing the others.
How it works
The architecture follows one idea: research and reading want opposite shapes, and the system's job is to convert between them.
Flat
Research
Sources are gathered and summarised independently, each on its own terms. Relevance is deliberately not judged here — that decision belongs to the moment a question is asked, not the moment material is collected.
Hierarchical
Knowledge
The collected material is organized into a navigable structure and linked across documents. This is the stage that turns a pile of summaries into something that answers questions it was not collected for.
Linear
Content
A slice of the knowledge base is projected back into a single reading order and written up in the target format. The writing stage reads the knowledge base as a service and never needs to know how it is organized internally.
Flat, then hierarchical, then flat again
Research arrives flat. Sources sit side by side, each one summarised on its own terms, none of them aware of the others. That is the natural shape of collection, and it is a poor shape for understanding.
Knowledge wants hierarchy. Topics contain aspects, aspects contain sub-aspects, and claims connect across them. Building that structure is the step where the actual work happens.
Reading wants flat again. Nobody consumes a graph; a reader follows one line from beginning to end. So the final stage projects a slice of the hierarchy back into a single sequence, adapted to the format.
The round trip is the design. Information theory has described the cost of moving between these forms since Shannon; Miller quantified how little of a hierarchy a reader can hold at once; Sweller showed why gathering and structuring at the same time degrades both. Separating the stages is what those constraints imply.
Why the boundary matters
The interface between organizing and writing is the one seam that has to stay clean. As long as the writing stage asks for knowledge by topic rather than reaching into its internals, the knowledge layer can be rebuilt — and it has been — without touching anything downstream.
How quality is established
A pipeline of language models is easy to demo and hard to trust. The interesting question is not whether it produces plausible output, but what happens when it is measured against a bar that was set before the measurement.
Thresholds are fixed before the run, not after
The acceptance bar for each capability is written into an architecture decision record before any measurement is taken. Committing to the bar in advance is what makes a failure interpretable rather than negotiable — a threshold agreed after seeing the result is not a threshold.
A flattering result turned out to be an artifact
One capability measured as essentially perfect. Repeating the same measurement under an independent blind judge — a fresh model with no access to the reference answers — produced a materially worse result, and the two judges disagreed on a substantial share of the cases. The favourable number had come from a judge that shared context with the system it was grading. The finding was about the evaluation setup rather than the feature, which is the kind of error that quietly validates everything downstream of it.
When the gate failed, the architecture changed
A capability missed its bar at realistic scale. The failure analysis separated two causes: most misses were a matter of how widely the system searched, but some were representation failures that no amount of searching recovers. Combined with the observation that the operation was irreversible — it destroyed the distinction it got wrong — the work moved from happening at write time to happening at read time. The threshold was never lowered to fit the result.
Results have to be reproducible to count
The derived knowledge layer can be deleted and rebuilt from the source files, and the rebuild is byte-identical. Measurements taken against a state that cannot be reconstructed are anecdotes, not evidence.
None of this makes the system correct. It makes its failures visible early, and forces the architecture rather than the acceptance criteria to absorb them.
Where this sits
The design started with a survey rather than a blank page, and returns to one whenever a layer is rebuilt.
Ten existing systems were evaluated before the first line was written — among them Fabric, GPT Researcher, STORM, CrewAI, Khoj, Quivr and Open Deep Research. Most solve retrieval well. Few leave behind an artifact that outlives the query, and that gap defined what this project would be.
The knowledge layer was surveyed again before it was rebuilt: GraphRAG, RAPTOR, hierarchical index structures from the established retrieval frameworks, discourse parsing, and the personal-knowledge-management lineage — Zettelkasten, Obsidian, Logseq, Roam — for how they handle links between notes without the link set exploding.
The conclusion that shaped the current design is a separation the surveyed systems mostly do not make: structuring a single document and linking across documents are different problems with different failure modes, and merging them into one graph is what produces the failures measured above. Keeping them apart is the bet this architecture makes.
Where it is going
ResearchRaven is built around the observation that in knowledge work it is usually not information that is scarce, but structure. Search engines and language models both return more material than anyone can hold; what neither does is leave behind an artifact that gets more useful the more you feed it.
The system is therefore designed around permanence rather than throughput. Knowledge is captured without deciding relevance up front, because relevance is a property of the question asked later, not of the moment of collection. It is stored in plain files under version control rather than in a database, so it outlives the tool that produced it. And its cost is governed explicitly, because a system that cannot be run within a budget is not one you keep running.
The direction of travel is from short-form output toward long-form — from posts and newsletters toward chapters and books. Those are the formats where the difference between accumulated knowledge and retrieved snippets actually shows.