🤖 Singular Value Behavior in mclip3 - NS Iterations
This article discusses unexpected singular value dropping behavior observed in the mclip3 model after increasing the number of NS iterations. A solution involving nested msigns is presented.
Key Points:
• Increased NS iterations led to mclip3 dropping singular values below a threshold.
• Nesting msigns resolved the singular value dropping issue.
• The solution improves the model's stability and reliability.
Image
Image
🔗 Resources:
• Twitter Thread ↗ - Original discussion thread
🤖 Muon Model Training Challenges - Weight Decay
This article examines challenges encountered while training a Muon model with weight decay, specifically its failure to "grok" within 1000 steps even after hyperparameter tuning.
Key Points:
• Muon model with weight decay (0.1 and 0.01) failed to converge within 1000 steps.
• Hyperparameter tuning did not resolve the convergence issue.
• Further investigation is needed to understand the underlying cause.
Image
🔗 Resources:
• Twitter Thread ↗ - Original observation
🤖 Muon Model Grokking Challenges - Add-Mod-113 and Mul-Mod-113 Tasks
This article details the failure of a Muon model (without weight decay) to "grok" the Add-Mod-113 and Mul-Mod-113 tasks and discusses techniques that enable grokking and improve model robustness.
Key Points:
• Muon model (without weight decay) failed to grok Add-Mod-113 and Mul-Mod-113 tasks.
• Novel techniques enabled grokking and improved model robustness.
• Improved robustness against small changes in input data.
Image
🔗 Resources:
• Twitter Thread ↗ - Original findings
🤖 Spectral Hardcapping Implementation - Gradient Dualization
This article presents a novel implementation trick for Spectral Hardcapping that outperforms existing methods on larger inputs, albeit with increased computational cost. This approach allows for gradient dualization.
Key Points:
• Novel Spectral Hardcapping implementation.
• Outperforms existing methods on larger inputs.
• Enables gradient dualization.
Image
Image
🔗 Resources:
• Twitter Thread ↗ - Discussion of the implementation
🤖 P-bits and PAOA - Large-Scale Quantum Computing
This article briefly mentions research building upon the PAOA (presumably, Parallel Architecture for Optimization Algorithms) concept, utilizing p-bits for potentially reaching scales beyond current quantum computing capabilities.
Key Points:
• Research builds upon the PAOA concept.
• Utilizes p-bits for scalability.
• Aims to achieve computations beyond the reach of quantum computers.
🔗 Resources:
• Twitter Thread ↗ - Related discussion
🚀 Apple Silicon for Large-Scale Distributed Training - Memory-Optimized AI
This article describes a new approach to efficient large-scale distributed training using Apple Silicon, leveraging its higher memory-to-FLOPS ratio compared to traditional GPUs.
Key Points:
• New approach for efficient large-scale distributed training.
• Utilizes Apple Silicon's high memory capacity.
• Addresses the memory limitations of traditional GPUs.
Image
Image
Image
🔗 Resources:
• Twitter Thread ↗ - Details of the approach
💡 Filing a Complaint - Sexual Harassment and Discrimination in Academia
This article announces the filing of a complaint with the US Equal Employment Opportunity Commission (EEOC) against Ding-Zhu Du and Weili Wu for alleged sexual harassment, retaliation, gender discrimination, and psychological abuse. It encourages others who experienced similar harm to come forward.
Key Points:
• EEOC complaint filed against Ding-Zhu Du and Weili Wu.
• Allegations include sexual harassment, retaliation, and discrimination.
• Encouragement for others to report similar experiences.
Image
🔗 Resources:
• Twitter Thread ↗ - Original announcement
💡 Reporting Sexual Harassment and Discrimination - Academic Resources
This article provides links to resources for reporting sexual harassment and discrimination in academic settings, including the NSF, IEEE, and EEOC. It is related to the previous article announcing a complaint against Ding-Zhu Du and Weili Wu.
Key Points:
• Links to report sexual harassment and discrimination.
• Resources provided for NSF, IEEE, and EEOC.
• Encouragement for victims to seek recourse.
🔗 Resources:
• NSF ↗ - National Science Foundation
• IEEE ↗ - Institute of Electrical and Electronics Engineers
• EEOC ↗ - Equal Employment Opportunity Commission
🤖 Bitcoin Halving - Deflationary Mechanism and Market Impact
This article explains the Bitcoin halving, a deflationary mechanism that reduces the rate of new Bitcoin issuance, impacting miners, network security, and market value.
Key Points:
• Bitcoin halving is a deflationary mechanism.
• Controls the issuance of new bitcoins.
• Impacts miners, network security, and market value.
🔗 Resources:
• Twitter Thread ↗ - Original post
💡 Networking at ICML 2025 - Agentic AI
This article is an invitation to network at ICML 2025 in Vancouver, focusing on discussions around building agentic AI workflows and achieving high reliability.
Key Points:
• Networking opportunity at ICML 2025.
• Focus on agentic AI workflows and reliability.
• Invitation to discuss wins and pain points.
Image
🔗 Resources:
• Twitter Thread ↗ - Original post
⭐️ Support
If you liked reading this report, please star ⭐️ this repository and follow me on Github ↗, 𝕏 (previously known as Twitter) ↗ to help others discover these resources and regular updates.