AI training costs at 1T scale can be substantial. Mimo V2.6 Pro, a large language model, requires 1.02T total parameters and 42B active parameters, resulting in a daily cost of $493k and a step cost of $70k.
Key points
Cost Breakdown: Mimo V2.6 Pro costs $493k per day, with a step cost of $70k and a sample cost of $2.78 per trajectory.
Model Comparison: Mimo V2.6 Flash, a smaller model, costs $247k per day, with a step cost of $21k and a sample cost of $0.86 per trajectory.
Token Cost: Mimo V2.6 Pro costs $33.91 per million tokens, while Mimo V2.6 Flash costs $9.25 per million tokens.
Sources
EAs and Predictive Power
EAs (Evolutionary Algorithms) may not be uniquely good at foresight, as some people may have predicted the success of deep learning before it happened.
Key points
EA Limitations: EAs may not be as effective at predicting future trends as some people believe.
Deep Learning: Deep learning was predicted to be revolutionary before it happened, but EAs were not.
Foresight: EAs are not uniquely good at foresight, and other factors may contribute to their success.
Sources
Toxicity in Online Communities
Toxicity in online communities can be a significant problem, and it's essential to address it.
Key points
Toxicity: Toxicity in online communities can be a significant problem, and it's essential to address it.
Flawed Reasoning: A research paper with flawed reasoning can lead to toxicity and conflict.
Conflict Resolution: Conflict resolution requires addressing the underlying issues and flaws in reasoning.
Sources
Founder's Wife and Identity
Being a founder's wife can be similar to being a pastor's wife, where one trades in their identity for a supporting role.
Key points
Identity: Being a founder's wife can be similar to being a pastor's wife, where one trades in their identity for a supporting role.
Supporting Role: The supporting role can be appealing, but it's essential to maintain one's identity.
Contributing: Contributing to something bigger than oneself can make the job appealing.
Sources
GenAI in CS Classes
GenAI (General Artificial Intelligence) is being integrated into CS (Computer Science) classes, but it's unclear how it's being taught.
Key points
GenAI in CS: GenAI is being integrated into CS classes, but it's unclear how it's being taught.
Python Classes: GenAI is being taught in introductory Python classes, but it's unclear if students are learning basic Python code first.
Integration: GenAI is being integrated from the get-go in some CS 101 classes.
Sources
Encoder-Only Classifiers
Encoder-only classifiers exist and are widely used, but many people are not aware of them.
Key points
Encoder-Only Classifiers: Encoder-only classifiers exist and are widely used, but many people are not aware of them.
LLMs: LLMs (Large Language Models) are not the only type of model, and encoder-only classifiers are a valuable alternative.
Reactions: Reactions to encoder-only classifiers suggest that many people are not aware of their existence.
Sources
Training Multiple Heads
Training multiple heads on a ModernBERT model can be a cost-effective way to get a similar system.
Key points
Training Multiple Heads: Training multiple heads on a ModernBERT model can be a cost-effective way to get a similar system.
Cost-Effectiveness: Training multiple heads can be more cost-effective than training a single large model.
Similar System: The resulting system can be similar to a large language model.
Sources
Lisbon AI Tickets
Supabase is offering two free tickets to Lisbon AI, and they want to go to talented builders who would otherwise not be able to afford to attend.
Key points
Lisbon AI Tickets: Supabase is offering two free tickets to Lisbon AI, and they want to go to talented builders who would otherwise not be able to afford to attend.
Talented Builders: The tickets are for talented builders who would otherwise not be able to afford to attend.
Application: If you are a young engineer who is serious about building, apply here.
Sources
Transparency Laws
Transparency laws should focus on public disclosure, which can improve our collective understanding.
Key points
Transparency Laws: Transparency laws should focus on public disclosure, which can improve our collective understanding.
Public Disclosure: Public disclosure can lead to more access to talent, greater diversity of perspectives, and higher likelihood of adversarial testing of ideas.
Volume of Ideas: Public disclosure can lead to a higher volume of ideas and a greater likelihood of surfacing new ones.
Sources
Credit Assignment
Credit assignment is a crucial aspect of AI research, and it's essential to address it.
Key points
Credit Assignment: Credit assignment is a crucial aspect of AI research, and it's essential to address it.
Navier-Strokes Drama: The Navier-Strokes drama highlighted the importance of credit assignment and attribution.
Co-Authorship: Co-authorship can be a stepping stone to credit assignment.