… Hybrid executing, across both the cloud and local hardware, is how they maximize tokens per user. Now it’s demo time. Working on confidential materials with an AI chatbot. The local model flags what is sensitive and what is not, so that only non-sensitive materials are sent to the cloud AI service. …
… As we discussed in a recent Substack post, even after years, the Qualcomm AI100 that you can buy in a new Dell laptop has a challenging software stack to say the least. This feels like a reset from Qualcomm. New hardware, a new software framework, and a focus on connectivity. …
… With all of this said, it is probably worth remembering that there are some enormous systems out there that are not being benchmarked for Linpack and the Top500. What is also happening, however, is that the AI hardware is starting to shed FP64 support to further improve mixed-precision performance.