Perplexity's serving infrastructure is designed to handle large-scale AI workloads, including online and batch embedding. The infrastructure combines three components: Ivy, Tulip, and ROSE. Ivy handles CPU-side request prep, including parsing, tokenization, and templating. Tulip is the inference engine, and ROSE is the optimization layer. The combined infrastructure lowers latency and improves throughput compared to off-the-shelf solutions.
Key points
• Perplexity's serving infrastructure is designed for large-scale AI workloads. • The infrastructure combines Ivy, Tulip, and ROSE components. • Ivy handles CPU-side request prep, including parsing, tokenization, and templating. • Tulip is the inference engine, and ROSE is the optimization layer. • The combined infrastructure lowers latency and improves throughput.
Sources
- Original post - Perplexity's serving infrastructure
- Ivy - CPU-side request prep component
- Tulip - Inference engine component
- ROSE - Optimization layer component
AI Infrastructure - HTTP Gateway for Large-Scale AI
Ivy is the HTTP gateway for Perplexity's serving infrastructure. It handles CPU-side request prep, including parsing, tokenization, templating, and splitting large batches before sending them to Tulip over gRPC. Ivy allows for tuning request formatting and tokenization without touching the heavier inference servers.
Key points
• Ivy is the HTTP gateway for Perplexity's serving infrastructure. • Ivy handles CPU-side request prep, including parsing, tokenization, and templating. • Ivy splits large batches before sending them to Tulip over gRPC. • Ivy allows for tuning request formatting and tokenization.
Sources
- Original post - Ivy HTTP gateway
- Tulip - Inference engine component
AI Infrastructure - Embedding and Ranking Models
Perplexity's embedding and ranking models are designed to pick the most relevant results for a query. The models start with embedding and ranking, and the entire process is optimized for speed and accuracy. Perplexity has published research on how they built their serving infrastructure behind these models.
Key points
• Perplexity's embedding and ranking models pick the most relevant results for a query. • The models start with embedding and ranking. • The entire process is optimized for speed and accuracy. • Perplexity has published research on their serving infrastructure.
Sources
- Original post - Perplexity's embedding and ranking models
- Research paper - Perplexity's serving infrastructure research
AI Infrastructure - Environment Evolution for RL Agents
Terminal RL agents often hit a wall as synthetic CLI tasks become too easy. Environment Evolution is a new approach that levels up the game by growing task difficulty off-policy, generation by generation, along three axes: scenario novelty, action novelty, and reward novelty. This approach allows RL agents to learn more complex tasks.
Key points
• Terminal RL agents often hit a wall as synthetic CLI tasks become too easy. • Environment Evolution grows task difficulty off-policy, generation by generation. • The approach allows RL agents to learn more complex tasks. • Environment Evolution is designed for three axes: scenario novelty, action novelty, and reward novelty.
Sources
- Original post - Environment Evolution
- Research paper - Environment Evolution research
AI Infrastructure - Model Landscape and Architecture
The AI model landscape is changing rapidly, with new models being released every month. If your AI architecture hardwires one provider, you're building technical debt in the long run. August's model landscape is unrecognizable from July, with new models like Muse Glimmer and Muse Image being released.
Key points
• The AI model landscape is changing rapidly. • New models are being released every month. • Hardwiring one provider in your AI architecture is building technical debt. • August's model landscape is unrecognizable from July.
Sources
- Original post - AI model landscape
- Muse Glimmer - New AI model
- Muse Image - New AI model
AI Infrastructure - Open-Weight Models and Model Commoditization
MBZUAI has open-sourced 6 foundation models with full training code, data, and recipes. This move towards model commoditization allows enterprise teams to deploy frontier-quality models with full auditability. No black boxes are involved, and model commoditization is the future of AI.
Key points
• MBZUAI has open-sourced 6 foundation models. • The models include full training code, data, and recipes. • Model commoditization allows for full auditability. • No black boxes are involved in model commoditization.
Sources
- Original post - MBZUAI open-sourced models
- MBZUAI models - Open-sourced foundation models
AI Infrastructure - AI in Education and Student Learning Journals
The first days of school are coming up fast, and saving time is essential. Watching Janet on page templates for building a student learning journal in Book Creator is a great way to get started. This approach allows teachers to create engaging and interactive learning materials for their students.
Key points
• The first days of school are coming up fast. • Saving time is essential for teachers. • Watching Janet on page templates for building a student learning journal is a great resource. • This approach allows teachers to create engaging and interactive learning materials.
Sources
- Original post - Student learning journal
- Book Creator - Learning journal creation tool
AI Infrastructure - Mandatory AI Classes for Kids
The UAE has launched mandatory AI classes for kids as young as 5. This move towards AI education is essential for preparing the next generation of AI professionals. However, the US is limiting AI in early grades, raising questions about who controls the AI infrastructure children interact with.
Key points
• The UAE has launched mandatory AI classes for kids as young as 5. • The US is limiting AI in early grades. • Who controls the AI infrastructure children interact with is a critical question. • Dual-layer content moderation and COPPA compliance are essential for AI education.
Sources
- Original post - Mandatory AI classes
- COPPA compliance - Essential for AI education
AI Infrastructure - EHR-Connected AI and Data Completeness
A physician running EHR-connected AI for a year has reported that records from a single institution are not enough. Patient data lives across 5+ systems, and no single EHR has the full picture. This highlights the importance of data completeness in EHR-connected AI.
Key points
• Records from a single institution are not enough for EHR-connected AI. • Patient data lives across 5+ systems. • No single EHR has the full picture. • Data completeness is essential for EHR-connected AI.
Sources
- Original post - EHR-connected AI
- EHR systems - Essential for EHR-connected AI
AI Infrastructure - AI for Healthcare and Patient Data
A year of running EHR-connected AI has highlighted the importance of data completeness. 29 hospitals have created Chief AI Officer roles in 2026, but the data completeness problem remains unsolved. This highlights the need for more research and development in EHR-connected AI.
Key points
• A year of running EHR-connected AI has highlighted the importance of data completeness. • 29 hospitals have created Chief AI Officer roles in 2026. • The data completeness problem remains unsolved. • More research and development are needed in EHR-connected AI.
Sources
- Original post - EHR-connected AI
- Chief AI Officer roles - Essential for EHR-connected AI