The 2nd Edition

July 15, 2023

👨‍💻 Developers Arena 👨‍💻

⚡️Myth of Context Length -

Recently, we have seen a wave of LLMs for longer contexts: We had

Don't get too excited by claims of 1M or even 1B context tokens. You know what, LSTMs already achieve infinite context length 25 yrs ago!

What truly matters is how well the model actually uses the context. It's easy to make seemingly wild claims, but much harder to solve real problems better.

I highly recommend "Lost in the Middle: How Language Models Use Long Contexts" from Stanford. It's jam-packed with rigorous experiments that put popular long-context models to test.

Key findings:

  • The models are good at using information at the beginning or end of its context, but significantly worse at the middle. This isn't unique to GPT architecture - encoder-decoder like Flan-T5 also suffers in the middle.

  • The models that are natively more extended context do NOT actually use the context better. You can see that the curves of GPT-3.5 (4k vs 16k) almost completely overlap.

  • Model performance substantially decreases as input contexts grow longer, regardless of their native length. We don't need more tokens. We need models that actually pay attention to them.

✨ Last 15 Days in AI ✨

🚀 Elon forms xAI 🚀

Elon Musk has announced the formation of his latest company with an all-star cast of 12, including:

• Igor Babuschkin: Behind DeepMind's StarCraft player that beats humans

• Jimmy Ba: Responsible for building blocks of modern transformers & neural nets

What's the purpose of xAI?

It's going to compete on the foundational model layer. Musk actually founded OpenAI, but he created it as an open-source nonprofit, and they have become a closed-source for-profit.

🚀 Tumor Detection AI 🚀

📌 Consistently identifying subtle genomic changes in tissue samples poses a challenge for the human eye.

📌 However, the recently developed AI system, CHARM, overcomes this obstacle by achieving an impressive 93% accuracy in detecting specific molecular alterations.

📌 The implementation of CHARM holds great potential for assisting neurosurgeons in making crucial decisions regarding the extent of tissue removal and potential on-the-spot treatments.

📌 Although clinical validation and FDA approval are necessary steps, the creation of CHARM marks a significant leap forward in the field of real-time precision oncology.

🚀 AI Wildfire Camera Startup Raises $17M 🚀

📌 Pano, a startup specializing in wildfire detection, secured an additional $17 million in funding as an extension to its series A round.

📌 By leveraging AI technology, Pano aims to establish early detection systems that provide accurate and detailed alerts, with the ultimate goal of minimizing the devastating impact of wildfires.

📌 With a portfolio of over 100 units, each generating $50,000 in annual revenue ($5 million total yearly revenue), the company utilizes 360-degree spinning cameras deployed across various locations including Colorado, Montana, Oregon, California, and Australia.

📌 Governments are willing to invest in Pano's technology as it offers the potential to effectively reduce the occurrence of mega-fires.

🚀 ChatGPT in an E-bike 🚀

📌An upcoming carbon e-bike showcased by Urtopia offers the ability to interact with a ChatGPT voice.

📌 A potential future scenario involves the integration of a ChatGPT-like assistant into every device.

📌 Your imagination is the only limit when it comes to the possible use cases:

  • Engaging in a step-by-step DIY repair conversation.

  • Creating a touring plan with multiple stops using interactive guidance.

  • Receiving guidance on a personalized nutrition plan.

  • Seeking advice on accidents or injuries

**

Author’s Note

The substack recommends the shorter newsletter, that’s why I divided the actual newsletter into three different parts to make it quick.

I will publish the other parts the next week. The 3rd Edition will reach your inbox by Monday.

This is the 2nd Edition, If you have any feedback please don’t hesitate to share it with me, And if you love my work, do share it with your colleagues.

Cheers!!

Raahul

**

Reply

or to participate.