Skip to content
Drix10 Blog

Poster Session 5: Evaluating Retrieval Robustness of LLMs

, 8 items in AI Organizations and Media, 4 min read

In this digest (8 items)

Shuyang Cao and Karthik Radhakrishnan will present a poster on their paper “Evaluating the Retrieval Robustness of Large Language Models.” The study defines retrieval robustness for RAG and proposes three metrics covering RAG benefit, document count effect, and document order impact. Experiments use 1,891 samples from five datasets, three task categories, and 11 LLMs with sparse and dense retrievers.

Key points

  • Presenters: Shuyang Cao & Karthik Radhakrishnan.

  • Benchmark: 1,891 samples, five datasets, three task categories, 11 LLMs.

Sources

Qwen, GPT robustness drops; Claude gains from docs-as-tool

Qwen and GPT lose robustness when reasoning is disabled, even on simple single-hop questions. Using retrieved documents as tool calls improves Claude models but harms Qwen and GPT with the same documents.

Key points

  • Disabling reasoning tanked Qwen & GPT robustness on simple single-hop questions.

  • Treating retrieved docs as tool calls helped Claude models but hurt Qwen & GPT.

Sources

Lenovo AI Capacity Acceleration Levers

Lenovo outlines two levers to get AI capacity online faster. The first lever is solving problems before execution starts. The second lever is lining up flexible capital early because builds now run into the tens of billions of dollars.

Key points

  • Lever 1: Solve problems before execution starts

  • Lever 2: Line up flexible capital early; builds cost tens of billions of dollars

Sources

@googlegemma Hardware Hackathonb event announced

@googlegemma Hardware Hackathonb is scheduled for October 24. The event is sponsored by @GoogleDeepMind and @ AGI House. It will gather 120 builders to create 100% local applications that run entirely on-device.

Key points

  • Date: October 24.

  • Participants: 120 builders.

Sources

ethy_agent autonomous trading assistant announced for Base

ethy_agent is an autonomous trading assistant. It executes on-chain strategies defined in natural language on Base. A profile poster is linked in the post.

Key points

  • Tool: ethy_agent, an autonomous trading assistant

  • Capability: executes on-chain strategies defined in natural language on Base

Sources

McMasterCarr digital catalog and CAD integration highlighted

McMasterCarr is an industrial supplier. It offers a digital catalog of over 500,000 products. The catalog includes CAD integration for engineers. A profile poster was shared.

Key points

  • Catalog size: over 500,000 products

  • Feature: CAD integration for engineers

Sources

XPONENTIAL 2027 Call for Presentations deadline is Oct 9

XPONENTIAL 2027 is accepting presentation proposals. The deadline is tomorrow, October 9. Submissions must be completed and sent via the provided link.

Key points

Sources

VAPS – Viability‑Aware Policy Selection for Safe Humanoid Acrobatics

VAPS is a decision framework that selects between a tracking, abort, and fall policy for humanoid robots. It predicts short‑horizon viability of the tracking and abort policies at each control step and chooses the least‑sacrificial viable option. Experiments show reduced head and hand contacts on Unitree G1 and LimX Oli robots during flips.

Key points

  • VAPS uses learned predictors to estimate viability of nominal tracking and abort policies over a short horizon.

  • In simulation with randomized disturbances, VAPS sharply reduces head and hand contacts compared to single‑network baselines.

Sources