🤖 AI Training Costs at 1T Scale
AI training costs at 1T scale can be substantial. Mimo V2.6 Pro, a large language model, requires 1.02T total parameters and 42B active parameters, resulting in a daily cost of $493k and a step cost of $70k.
Key Points:
Cost Breakdown: Mimo V2.6 Pro costs $493k per day, with a step cost of $70k and a sample cost of $2.78 per trajectory.
Model Comparison: Mimo V2.6 Flash, a smaller model, costs $247k per day, with a step cost of $21k and a sample cost of $0.86 per trajectory.
Token Cost: Mimo V2.6 Pro costs $33.91 per million tokens, while Mimo V2.6 Flash costs $9.25 per million tokens.
🔗 Resources:
- Original post ↗
- Mimo V2.6 Pro
- Mimo V2.6 Flash
🤖 EAs and Predictive Power
EAs (Evolutionary Algorithms) may not be uniquely good at foresight, as some people may have predicted the success of deep learning before it happened.
Key Points:
EA Limitations: EAs may not be as effective at predicting future trends as some people believe.
Deep Learning: Deep learning was predicted to be revolutionary before it happened, but EAs were not.
Foresight: EAs are not uniquely good at foresight, and other factors may contribute to their success.
🔗 Resources:
- Original post ↗
- EAs
- Deep Learning
🤖 Toxicity in Online Communities
Toxicity in online communities can be a significant problem, and it's essential to address it.
Key Points:
Toxicity: Toxicity in online communities can be a significant problem, and it's essential to address it.
Flawed Reasoning: A research paper with flawed reasoning can lead to toxicity and conflict.
Conflict Resolution: Conflict resolution requires addressing the underlying issues and flaws in reasoning.
🔗 Resources:
- Original post ↗
- Online Communities
- Conflict Resolution
🤖 Founder's Wife and Identity
Being a founder's wife can be similar to being a pastor's wife, where one trades in their identity for a supporting role.
Key Points:
Identity: Being a founder's wife can be similar to being a pastor's wife, where one trades in their identity for a supporting role.
Supporting Role: The supporting role can be appealing, but it's essential to maintain one's identity.
Contributing: Contributing to something bigger than oneself can make the job appealing.
🔗 Resources:
- Original post ↗
- Founder's Wife
- Pastor's Wife
🤖 GenAI in CS Classes
GenAI (General Artificial Intelligence) is being integrated into CS (Computer Science) classes, but it's unclear how it's being taught.
Key Points:
GenAI in CS: GenAI is being integrated into CS classes, but it's unclear how it's being taught.
Python Classes: GenAI is being taught in introductory Python classes, but it's unclear if students are learning basic Python code first.
Integration: GenAI is being integrated from the get-go in some CS 101 classes.
🔗 Resources:
- Original post ↗
- GenAI
- CS Classes
🤖 Encoder-Only Classifiers
Encoder-only classifiers exist and are widely used, but many people are not aware of them.
Key Points:
Encoder-Only Classifiers: Encoder-only classifiers exist and are widely used, but many people are not aware of them.
LLMs: LLMs (Large Language Models) are not the only type of model, and encoder-only classifiers are a valuable alternative.
Reactions: Reactions to encoder-only classifiers suggest that many people are not aware of their existence.
🔗 Resources:
- Original post ↗
- Encoder-Only Classifiers
- LLMs
🤖 Training Multiple Heads
Training multiple heads on a ModernBERT model can be a cost-effective way to get a similar system.
Key Points:
Training Multiple Heads: Training multiple heads on a ModernBERT model can be a cost-effective way to get a similar system.
Cost-Effectiveness: Training multiple heads can be more cost-effective than training a single large model.
Similar System: The resulting system can be similar to a large language model.
🔗 Resources:
- Original post ↗
- ModernBERT
- Training Multiple Heads
🤖 Lisbon AI Tickets
Supabase is offering two free tickets to Lisbon AI, and they want to go to talented builders who would otherwise not be able to afford to attend.
Key Points:
Lisbon AI Tickets: Supabase is offering two free tickets to Lisbon AI, and they want to go to talented builders who would otherwise not be able to afford to attend.
Talented Builders: The tickets are for talented builders who would otherwise not be able to afford to attend.
Application: If you are a young engineer who is serious about building, apply here.
🔗 Resources:
- Original post ↗
- Lisbon AI
- Supabase
🤖 Transparency Laws
Transparency laws should focus on public disclosure, which can improve our collective understanding.
Key Points:
Transparency Laws: Transparency laws should focus on public disclosure, which can improve our collective understanding.
Public Disclosure: Public disclosure can lead to more access to talent, greater diversity of perspectives, and higher likelihood of adversarial testing of ideas.
Volume of Ideas: Public disclosure can lead to a higher volume of ideas and a greater likelihood of surfacing new ones.
🔗 Resources:
- Original post ↗
- Transparency Laws
- Public Disclosure
🤖 Credit Assignment
Credit assignment is a crucial aspect of AI research, and it's essential to address it.
Key Points:
Credit Assignment: Credit assignment is a crucial aspect of AI research, and it's essential to address it.
Navier-Strokes Drama: The Navier-Strokes drama highlighted the importance of credit assignment and attribution.
Co-Authorship: Co-authorship can be a stepping stone to credit assignment.
🔗 Resources:
- Original post ↗
- Credit Assignment
- Co-Authorship
