The Pitfalls of AI-Led Data Extraction

AI-Led Extraction Isn’t Saving Researchers Time. Here’s What Happened When We Tried.

Artificial intelligence is often celebrated as the ultimate time-saver for researchers. But in a recent experiment, we discovered something unexpected: when it comes to data extraction, AI can actually slow things down.

At PICO Portal, we tested large language models (LLMs) to handle several structured data extraction questions — for example:

“Provide the comparative results for patients with heart failure.”

On paper, it should have been faster. In practice, it wasn’t.

Instead of accelerating our review, AI became a bottleneck. Our methodologists had to meticulously verify every AI-generated answer for accuracy. What we gained in automation, we lost in verification. The process became slower, not faster.

Why AI Slowed Us Down

Reviewing AI outputs isn’t a passive step — it’s a detailed audit. Each AI-generated answer must be cross-checked against study texts to ensure accuracy. That means manually combing through dense, 10-page studies to confirm or correct what the AI suggested.

The problem isn’t that AI can’t read studies. It’s that it doesn’t yet understand scientific reliability. For systematic reviewers and evidence synthesists, precision matters more than speed. When AI skips context, humans must compensate.

A Better Way Forward

There’s a smarter way to apply AI — not as an oracle, but as a guide.

Instead of trying to answer extraction questions directly, AI could highlight key passages within studies, pointing reviewers to the most relevant evidence. This approach maintains scientific rigor while reducing time spent searching.

For tasks like risk of bias assessment, AI could serve as a first reviewer — flagging relevant sections and proposing initial judgments. Since these questions are predefined and structured, the AI can be trained to identify and categorize evidence more precisely. Human reviewers then confirm or adjust, with a third adjudicator resolving any discrepancies — seamlessly fitting into existing workflows.

This is precisely how PICO Portal has designed its AI-assisted Risk of Bias and Data Extraction.

The Takeaway

AI isn’t ready to replace researchers — and it shouldn’t. But it can support them, strategically.

For data extraction and evidence synthesis, the path forward is clear:

• Don’t ask AI to replace human judgment.

• Use it to surface and organize information more intelligently.

By helping researchers focus their expertise where it matters most, AI can save time without compromising scientific integrity.

That’s the vision behind PICO Portal: AI that works with you, not for you.