🤖 3D Graphics - Splatting to 3D Displays

This article discusses a novel algorithm that converts triangular and Gaussian splats into 3D display-compatible formats. This technology is being developed to support true defocus and parallax in 3D visualizations.

Key Points:

• Converts splats to 3D display formats for enhanced visualization

• Supports true defocus, providing realistic depth perception

• Enables true parallax for accurate spatial representation

• Leverages existing convex and Gaussian splatting techniques

🔗 Resources:

• Brian C. Chao ↗ - Developer of the novel splatting conversion algorithm

• Tweet by Brian C. Chao ↗ - Original announcement thread

• Jan Held ↗ - Researcher in convex splatting

Image

🤖 Computer Vision - Feed-Forward Camera Localization

This article discusses an approach to feed-forward camera localization from a collection of image features. It highlights an efficient method for 3D point-map estimation by using random subsampling of image data.

Key Points:

• Utilizes a collection of image features for camera localization

• Emphasizes random subsampling over full image pairs for efficiency

• Enables faster computation in 3D point-map estimation

• Allows processing a larger number of images for improved robustness

🚀 Implementation:

Context 3D Point-Map Estimator: Provide relevant contextual data to the estimator for initialization.
Randomly Subsample Image Features: Select a random subset of image features instead of full image pairs.
Process Subsampled Features: Input the selected features into the estimator for accelerated computation.

🔗 Resources:

• Kwang Moo Yi ↗ - Discussing camera localization research

• Tweet by Kwang Moo Yi ↗ - Original discussion on camera localization

Image

🤖 Robotics - VLM2VLA for Multimodal Robot Policies

This article introduces VLM2VLA, a method for adapting Vision-Language Models (VLMs) into robot policies. It maintains multimodal generalization by treating robot actions as language and applying LoRA tuning.

Key Points:

• Adapts Vision-Language Models (VLMs) for robot policy control

• Treats robot actions as language for unified multimodal processing

• Utilizes LoRA tuning to preserve core VLM generalization capabilities

• Enables strong zero-shot generalization across new tasks and languages

🚀 Implementation:

Represent Robot Actions as Language: Define robot actions using a linguistic format or lexicon.
Integrate with a VLM: Connect these language-represented actions to a Vision-Language Model.
Apply LoRA Tuning: Fine-tune the VLM using Low-Rank Adaptation for specific robot tasks.
Deploy for Zero-Shot Generalization: Test the adapted policy on novel tasks and languages.

🔗 Resources:

• Visual AI Lab ↗ - Researching visual AI and robotics

• Cindy Wu ↗ - Discussing VLM2VLA development

• Tweet by Cindy Wu ↗ - Original announcement of VLM2VLA

Image

✨ AI Creativity - Sora for Video Generation

This article explores the capabilities of Sora in generating evocative video content from simple textual prompts. It reflects on how AI can create compelling visual narratives that resonate as true.

Key Points:

• Utilizes Sora for generating video content based on textual prompts

• Demonstrates AI's capacity to create "true-feeling" visual narratives

• Offers a new paradigm for capturing and sharing moments without direct recording

• Highlights the artistic potential of advanced AI video generation

🔗 Resources:

• Create Compute ↗ - Sharing AI art and video creations

• Tweet by Create Compute ↗ - Original post on Sora video generation

Image

🚀 Tools - Tinker API for LLM Fine-tuning

This article introduces Tinker, a flexible API designed for fine-tuning language models. It enables developers to define training loops locally, which are then efficiently executed on distributed GPU infrastructure.

Key Points:

• Provides a flexible API for fine-tuning large language models

• Allows local Python development for custom training loops

• Manages execution on distributed GPU resources automatically

• Supports researchers and developers using cutting-edge open models

🚀 Implementation:

Develop Training Loops: Write Python training code for language models on a local machine.
Utilize Tinker API: Integrate the custom training logic with the Tinker API.
Submit for Distributed Execution: Leverage Tinker to run the training on distributed GPUs.
Monitor and Iterate: Observe results and refine models based on performance.

🔗 Resources:

• Pranab Kumar ↗ - Announcing Tinker API for LLM fine-tuning

• Thinky Machines ↗ - The team behind the Tinker API

• Tweet by Thinky Machines ↗ - Tinker API official launch

Image

🤖 AI Theory - LLMs, Continual Learning, and New Architectures

This article discusses Richard Sutton's perspective on Large Language Models (LLMs) and the fundamental requirement for continual learning architectures. It explores the idea that new architectural paradigms are crucial for effective on-the-job learning in AI.

Key Points:

• Richard Sutton suggests current LLMs may not align with the "bitter lesson" of scaling

• Advocates for new architectures enabling continual, on-the-job learning

• Proposes that continual learning reduces the need for specialized pre-training

• Challenges prevailing views on the future direction of LLM development

🔗 Resources:

• Biruk Abere ↗ - Discussing AI and learning theory

• Dwarkesh Patel ↗ - Curating discussions on AI advancements

• Tweet by Dwarkesh Patel ↗ - Original discussion on Sutton's views

• Richard S. Sutton ↗ - Father of reinforcement learning

🚀 Tools - Tinker API for Accessible LLM Training

This article announces Tinker, a new training API designed to democratize fine-tuning of large language models. It simplifies distributed training, allowing users to focus on data and algorithms without needing deep infrastructure expertise.

Key Points:

• Tinker is presented as an accessible training API for all users

• Abstracts away the complexities of distributed GPU training

• Enables users to concentrate on their data and model algorithms

• Facilitates training custom models without extensive infrastructure knowledge

🚀 Implementation:

Define Model and Data: Prepare your language model and the training dataset.
Specify Algorithms: Outline the training algorithms and required parameters.
Integrate with Tinker API: Use the API to submit your custom training job.
Execute Distributed Training: Let Tinker handle the distributed GPU execution automatically.

🔗 Resources:

• Pranab Kumar ↗ - Co-founder of Thinky Machines

• Dhruv Chaplot ↗ - Announcing Tinker, the new training API

• Thinky Machines ↗ - The company behind the Tinker API

• Tweet by Dhruv Chaplot ↗ - Product launch announcement

• Tweet by Thinky Machines ↗ - Tinker API official launch details

Image

🤖 Robotics - Community Engagement in Robot Events

This article highlights Rerun.io's sponsorship of the REKrobot fight, emphasizing the importance of community engagement in robotics. The event successfully fostered enthusiasm and participation in robot experimentation and development.

Key Points:

• Rerun.io supports community robotics events and initiatives

• Encourages wider participation and experimentation with robots

• Fosters a positive and entertaining environment for robotics enthusiasts

• Demonstrates commitment to the growth of the robotics field

🔗 Resources:

• Rerun.io ↗ - Company supporting robotics community events

• REKrobot ↗ - Robotics fight event organizer

• Tweet by Rerun.io ↗ - Post on sponsoring the robot fight

Image

💡 Event - AutoSens for Automotive Advancements

This article highlights the AutoSens event, offering insights into the latest advancements in Advanced Driver-Assistance Systems (ADAS), Autonomous Vehicles (AV), and in-cabin monitoring. Attendees can engage with innovative demos and a comprehensive technical agenda.

Key Points:

• Features innovative hands-on demonstrations of automotive technologies

• Provides an insightful technical agenda on current automotive advancements

• Covers the latest developments in ADAS technologies

• Explores new solutions in Autonomous Vehicle systems

• Discusses innovations in in-cabin monitoring solutions

🔗 Resources:

• AutoSens ↗ - Organizer of the automotive technology event

• Event Pass ↗ - Link to book passes for the AutoSens event

• Tweet by AutoSens ↗ - Announcement for the upcoming event

Image

🤖 Computer Vision - DA^2 for 360° Depth Estimation

This article introduces DA^2 (Depth Anything in Any Direction), a novel approach to 360° depth estimation. It details the creation of a large-scale training dataset and the subsequent training of a Sphere-aware Vision Transformer for comprehensive depth mapping.

Key Points:

• Focuses on comprehensive 360° depth estimation capabilities

• Involves creating a significantly scaled-up training dataset (10x larger)

• Utilizes a Sphere-aware Vision Transformer (ViT) architecture

• Aims to improve depth accuracy in all directions

🚀 Implementation:

Collect or Scale Existing Data: Expand the training dataset for depth estimation by a factor of ten.
Preprocess Data for 360° Context: Prepare the data to be compatible with spherical perspectives.
Train a Sphere-aware ViT Model: Use the scaled and prepared data to train the Vision Transformer.
Evaluate 360° Depth Estimation: Assess the model's performance across all directions.

🔗 Resources:

• Laks ↗ - Discussing 360° depth estimation advancements

• Haodong Li ↗ - Presenting the DA^2 project

• Akhaliq ↗ - Shared insights on the DA^2 project

• Tweet by Haodong Li ↗ - Original announcement of DA^2

Image

⭐️ Support

If you liked reading this report, please star ⭐️ this repository and follow me on Github ↗, 𝕏 (previously known as Twitter) ↗ to help others discover these resources and regular updates.