Agentic Research Planner

Shounak Ray and Harrison Delecki. Supported by John Mitchell, Anupam Datta, and Ankur Taly.

Key Takeaways

Specificity more than doubled (0.34 → 0.76) with iterative actor-critic refinement, producing concrete research plans instead of vague handwaving.

Context relevance hit 98% through RAG integration, showing the system is excellent at retrieving pertinent literature.

Low groundedness might be a feature, not a bug. Creative research ideas should push beyond existing literature—pure recombination isn't real research.

Metrics plateau after a few iterations, suggesting the critic's feedback mechanism needs more sophistication to drive continuous improvement.

The Problem

Planning a research project is time-intensive. The process involves surveying literature for weeks, identifying gaps in existing work, formulating hypotheses, designing experiments, and planning evaluations—all before writing a single line of code. Experienced researchers often spend months in this phase. We wondered: could an agentic AI system accelerate this process?

Preliminary studies suggest LLMs can help with ideation and collaboration, but existing tools have limited scope. They search papers, summarize content, or assist with drafting. What's missing is a system that handles the full arc: from initial research question to concrete experimental plan.

We built an actor-critic system that takes a research question and 1-3 seed papers, then iteratively refines a complete research proposal through 10 cycles of generation and critique. The question was whether this iterative approach could outperform direct prompting of ChatGPT.

The Approach

The architecture mirrors how researchers typically conduct literature reviews. An Actor generates research plans by retrieving relevant papers through RAG, synthesizing literature to identify gaps, then proposing hypotheses and experimental designs. A Critic evaluates each plan: is the hypothesis novel? Are the methods specific enough? Is the experimental design complete? This reflects the iterative refinement process that occurs when developing research proposals.

The implementation uses a decorator pattern to elegantly integrate critique into the workflow:

@critic.overwatch(prompt_path="Prompts/design_experiments.txt")
def design_experiments(self, hypotheses: List[Hypothesis]) -> List[str]:
    """Designs experiments for each hypothesis. The critic will evaluate
    and provide feedback to improve prompt quality over iterations."""
    prompt_tmpl = PromptTemplate(template=critic.get_prompt())
    return [self.designer.complete(prompt_tmpl.format(...)) for h in hypotheses]

This @critic.overwatch decorator automatically captures outputs, conducts critiques, and dynamically revises prompts between iterations.

Here's the workflow:

Figure 1: The iterative actor-critic workflow. The system cycles through generation and critique 10 times, refining the research plan at each step until producing a final actionable proposal.

The RAG system builds a vector store from seed papers and uses it to retrieve relevant context:

# Initialize vector store from research papers
self.index = VectorStoreIndex.from_documents(
    documents=[Document(text=ctx.paper_context,
                       metadata={"paper_id": ctx.paper_id})
              for ctx in init_contexts],
    storage_context=self.storage_context
)

# Query with top-k retrieval
query_engine = self.index.as_query_engine(similarity_top_k=5)
response = query_engine.query(prompt).response

To evaluate this, we measured two things. First, the RAG triad (courtesy of Anupam's work at TruLens): answer relevance (does it address the query?), context relevance (are retrieved papers pertinent?), and groundedness (is it grounded in literature?). Second, LLM judges scored creativity, specificity, and completeness of the final plans.

We instrumented the query engines with TruLens to automatically track these metrics:

# Wrap query engine with TruLens feedback functions
f_groundedness = Feedback(provider.groundedness_measure_with_cot_reasons)
    .on(context.collect()).on_output()

f_context_relevance = Feedback(provider.context_relevance_with_cot_reasons)
    .on_input().on(context).aggregate(np.mean)

instrumented = TruLlama(query_engine,
                       feedbacks=[f_groundedness, f_context_relevance])

Every query automatically logs metrics to a database for analysis.

Results

We tested three approaches: prompting ChatGPT directly for a research plan, running a single Actor pass without the Critic, and our full actor-critic loop with 10 iterations.

The most striking improvement was in context relevance—the iterative system got nearly perfect at retrieving pertinent papers (0.92 → 0.98). Answer relevance improved marginally, but groundedness stayed stubbornly low around 0.40. Interestingly, we think low groundedness might actually be a feature, not a bug: creative research ideas should push beyond existing literature rather than just recombining it.

The improvements were more pronounced here. Specificity more than doubled (0.34 → 0.76) and completeness increased from 0.57 to 0.76. The agentic system produces concrete, actionable research plans with well-defined experiments. Creativity remained constant at 0.60—the system improves rigor rather than novelty. Direct ChatGPT showed high variance in quality, with outputs ranging from detailed to vague.

What We Learned

The actor-critic loop works—it transforms vague ideas into concrete, actionable research plans. The 98% context relevance shows our RAG system is excellent at finding relevant literature. We built abstractions that could be reused for other agentic workflows.

One such abstraction is a validated LLM wrapper with automatic retry logic:

def prompt_LLM(client, prompt, desired_format="json",
               validate_func=None, num_retry=3):
    """Call LLM with automatic validation and retries."""
    try:
        response = client.chat.completions.create(...)
        # Built-in format validation
        response = _validate_response_format(response, desired_format)
        # Custom validation (e.g., "must contain 'hypothesis' key")
        if validate_func and not validate_func(response):
            raise ValidationError()
        return response
    except Exception:
        if num_retry > 0:
            return prompt_LLM(..., num_retry=num_retry-1)
        raise

This handles inconsistent LLM outputs gracefully, which is important when chaining multiple LLM calls in a pipeline.

But there are issues. Metrics plateaued after just a few iterations, suggesting our critic's feedback mechanism needs work. The critic revises prompts based on feedback using an LLM:

def chastise(self) -> None:
    """Revise prompts based on accumulated critiques."""
    for func_name, critiques in self._critiques.items():
        old_prompt = self._prompt_mapping[func_name]
        # LLM revises the prompt given the critique
        new_prompt = self.__revise_prompt(old_prompt, critiques[-1])
        self._prompt_mapping[func_name] = new_prompt

This meta-learning approach improves prompts automatically, but we found it plateaus quickly. Maybe we need multi-faceted critics specialized in different aspects (novelty vs. feasibility vs. rigor), or alternative update strategies like diverse sampling or tree search.

The low groundedness presents an interesting question. Is the system generating creative ideas by pushing beyond existing work, or is it producing poorly-grounded outputs? There may be a fundamental tension here: truly novel research ideas should extend beyond existing literature rather than simply recombining known results. The appropriate level of groundedness for creative research planning remains an open question.

Future Directions

Several improvements could enhance the system's performance. More sophisticated critics and live integration with paper databases like arXiv would provide richer feedback and more current literature. Multi-modal inputs—figures, equations, code—would make the system more comprehensive. User studies with researchers are needed to evaluate whether these plans are genuinely useful in practice.

Architecturally, there's room for experimentation. Alternative approaches include tree search instead of linear iteration, Monte Carlo sampling for diverse plan generation, and different LLM backbones. The design space remains largely unexplored.

Conclusion

Our results suggest that agentic systems can improve research planning quality, though with important limitations. The actor-critic approach produces more specific and complete plans than direct prompting, and the RAG integration effectively retrieves relevant literature. However, metrics plateau after a few iterations, indicating the critic mechanism needs further development.

The broader question is whether automated research planning serves researchers' needs. Low groundedness may indicate creative extension beyond existing work, though it could also reflect poor grounding (obvious from the low scores, but a more subjective understanding of what this means is necessary). High specificity improves plan quality, but user studies with researchers are needed to assess practical utility.

As LLMs improve at reasoning and long-horizon planning, systems like this may accelerate early-stage research by helping explore directions, generate hypotheses, and potentially suggest novel approaches that researchers might not have considered.

Acknowledgments

We're grateful to John Mitchell, Anupam Datta, and Ankur Taly for their guidance on agentic AI systems and evaluation methodologies. Their insights on the RAG triad and trustworthiness metrics shaped our experimental design. Thanks to Harrison for thinking together + coding this up with me, was a lot of fun!