… Meta GenAI 1.0: AI Incrementalism Compared to pure-play AI labs like OpenAI, companies like Meta and Google have followed an “AI Incrementalism” strategy by enhancing existing products with better recommendation systems and GenAI to improve ad targeting, content tagging, and internal tools. …
… HBM is much more expensive to produce and has a warranted price premium to DDR5, but demand remains strong for HBM. All leading AI accelerators deployed for GenAI training and inference use HBM. …
… GPT-3's total training cost of $162k and 19 households’ annual energy consumption doesn’t sound excessive, but it is the many experiments and many failed training runs that add up to the ballooning energy growth from AI Training we are seeing now in the United States. …
… It is easy see a future in which models are pre-trained via H800s, post-trained on H20s, then served to the population on Ascends and H20s. There is so much demand for AI powered products that should more capacity to serve these models exist, it would immediately be satisfied. …
… Memory: The Story So Far The memory wall has been the most important constraint for AI. Larger memory capacities have been necessary to load larger models into accelerators, while memory bandwidth has been the main gating factor for inference and training token throughput. …
… The isolation boundaries should be on VLANs and each tenant getting their own Kubernetes cluster. https://www.wiz.io/blog/nvidia-ai-vulnerability-deep-dive-cve-2024-0132 https://www.wiz.io/blog/wiz-research-critical-nvidia-ai-vulnerability https://nvidia.custhelp.com/app/answers/detail/a id/5614 ht… …