From Trillions Of Tokens To Every Word Ever Written

At the AI for Impact course at MIT today the trajectory of AI’s evolution was laid out in a manner that’s nothing short of breathtaking. Here’s a glimpse into the future of AI, as seen through the eyes of industry expert:

• Explosive Growth in Resources: We’re witnessing a 10,000-fold increase in computer resources dedicated to AI this decade. This isn’t just growth; it’s a revolution.
• It’s Happened Before: Per Moore’s Law, processing power doubles every two years. Nvidia CEO Jensen Huang has claimed Nvidia’s GPUs had boosted AI processing performance by a factor of no less than one million in the last 10 years. Huang has predicted another million x increase in compute.
• Trillions of Tokens: Modern AI models are being trained on a staggering trillion tokens. To put that in perspective, the entire Library of Congress is around 20TB of text, or 3.6 trillion tokens. For reference, GPT-4 is thought to have 1.75 trillion parameters

• Every Word Ever Written: The ambition doesn’t stop at the Library of Congress. The next training model may have the capacity to encompass effectively every word ever penned in human history.
• Monster Data Sets: The term ‘big data’ feels antiquated when we talk about these monster data sets AI is now grappling with.
• Multimodal Learning: It’s not just about text. The future AI will be multi-modal, processing images, sounds, and more, both for input and output.
• Personal AI: You’ll be able to chat and call your personal AI, which could even be a clone of a person, like yourself.
• Multimodel: AI systems will also be multi-model, bringing together a mixture of models depending on the expertise and information needed.
• Foundational Giants: The AI models of tomorrow are vast. They’re not just big; they’re colossal foundation models that redefine scale.
• A Book’s Worth of Context: Imagine an AI that uses 100k tokens, akin to an entire book, just to provide context for answering a single question. You don’t have to imagine this; Anthropic’s Claude.ai can already do this.
• Context is King: This extensive context isn’t just for show. It’s crucial in helping AI avoid “hallucinations” or erroneous outputs.

The future of AI isn’t just about bigger data or more processing power. It’s about redefining the very fabric of knowledge, understanding, and context. As we stand on the precipice of this new era, one thing is clear: the AI revolution is just getting started.

Here’s an inside look at the coming AI revolution, based on insights shared at MIT’s AI course:

AI is on the cusp of explosive growth that will reshape how we live and work. A point made in class today imagine another revolution on the scale of the steam engine or the microchip or the internet. Students are identifying massive opportunities for AI to transform major industries.

The ideas emerging from this program have the potential to be tomorrow’s breakthrough unicorns. But they are grounded in solving real human needs first, technology second. Students pitched startups in agtech, edtech, waste, and elderly care. Speakers stressed the importance of being bold, but focused, given the timing of the market.

The key is matching these powerful technologies with real market demand. Tony continues, “this technology excites me, but if you don’t start with the market, you will just build something that nobody will use.” The class pushes students to deeply understand their customers, map the ecosystem, and articulate how AI can enable a step-change in value. At MIT’s AI Venture Studio, our focus is on real-world impact.

“Although AI effort seemed to be trapped in the hands of large centralized companies, the future is in millions of mini-models that will engage with each other and usher the field of Decentralized AI”, said Ramesh Raskar, MIT Professor

This was from a conversation with Ramesh Raskar, MIT professor & Dave Blundin, MIT ‘88 and members of the class.