Server/Cloud Computing/AI Forum - Taiwan

Comprehensive presentation of compute memory used in today’s Data Centers and future challenges and considerations that impact both compute and AI memory subsystems.

Coming soon

Tom will cover the definition of the AI PC, core developments that are occurring to bring it to product, explain some of the software complexities, and how to make AI real. Dell is at the forefront of the AI Transformation so not all can be shared, but this talk will provide the vision and the building blocks to show the path to everyday use of AI.

Introduction to MediaTek - MediaTek datacenter products - Challenges to innovation in the memory space - Summary and Call to action.

As DDR5 move to End of Life, traditional high capacity based on 2 DIMMs keep total system bandwidth within 1-2 speed bin of the 1 DIMM DDR5 system. This talk will examine the challenges of using 2 DIMMs per channel. Then we will explore other alternative capacity DIMM solutions for server system.

The rapid growth of machine learning is accelerating the adoption of AI applications across a range of platforms, including cloud-based servers, edge devices, and endpoint devices. However, deploying AI at the edge and on the end-point devices introduces unique challenges, particularly related to system memory management. These challenges include limited memory resources, power consumption restrictions, bandwidth constraints, and the need for real-time processing capabilities. This presentation will begin by outlining the expected trends in edge AI platforms. It will then offer an overview of the key components essential for edge AI computing, highlighting their relative importance. Also, it will highlight the memory usage demand of the Key Value Cache layer in Large Language Model (LLM). Finally, recent advancements in system memory technology from JEDEC will be discussed and evaluated in the context of addressing the specific challenges and demands of edge AI computing.

We will discuss the transition from traditional AI computing to more sustainable AI computing in data centers, and the contribution and relevance of memory power in the overall sustainability of AI compute in future data centers, with a call for action to focus in future memory technology generations on power reduction innovation as a primary goal.

The DDR5 Multiplex Rank DIMM (MRDIMM) is the newest type of dual in-line memory module (DIMM) specified by the JEDEC standards organization. The MRDIMM enables a 2:1 ratio of data bandwidth between the host channel on the baseboard running at 12.8 Gbps, and two mostly independent “pseudo-channels” of DRAM memory on the DIMM running at 6.4 Gbps. Performance will be evaluated with latency vs. throughput and latency vs. utilization curves, comparing DDR5-8800 RDIMM 2-ranks, MRDIMM-12800 2-ranks and MRDIMM-12800 4-ranks.

The coming generation of memory solutions pose unique challenges for JEDEC. The module solutions are smaller than previous generations, yet device density is higher and power consumption continues to go up. Quantum computing is on the horizon, making the next generation of systems potentially vulnerable to a new variety of malicious attacks. This talk will holistically examine how these challenges tie together from voltage regulation (ala PMIC), registers, and serial presence detect (SPD) with an expansion of the Hub features introduced in DDR5 generation.

As the Wearables (AI Glasses, AR Glasses, and Smart Glasses) market experiences rapid and robust growth, we are confronted with a significant challenge: the absence of smaller capacity NAND (eMMC/UFS) options in the market. The trend is towards increasing capacity, driven by the demands of infrastructure, mobile, and automotive sectors. In this presentation, my objective is to emphasize the necessity of creating wearables-specific NAND standards that not only accommodate smaller capacities but also achieve substantial reductions in peak power consumption and footprint. Establishing these new market requirements is crucial.

Recent gains in DDR5 DIMM performance and capacity have led to an incremental increase in power consumption. Server systems, still dominated by the traditional 19-inch 1U rack, face design constraints as the demand for more memory channels forces a reduction in DIMM pitch. This combination of factors intensifies both power and thermal challenges in current systems. In this presentation, we explore these issues and discuss potential solutions for power and thermal management for future memory standards.

In the post-Moore era, the shrinkage of 2D DRAM technology is approaching the limit, and 3D DRAM has not yet arrived. AI intelligent computing is booming, which has led to explosive growth in the demand for memory bandwidth and capacity. AI training, inference, central inference, edge inference, and different AI computing architectures have very diverse requirements for memory bandwidth & capacity. Existing memory interface standards can no longer meet the requirements of AI computing. Whether future memory solutions are standardized or customized requires rapid response from the memory industry and JEDEC standards organizations, based on advanced packaging and chiplet technology to provide the optimal solution.

Generative AI has unlocked the power of artificial intelligence accelerating productivity, learning, and innovation in our work and personal lives. But AI does not exist without data, and memory is essential to the movement and processing of data. Compute performance is scaling at an ever increasing rate and requires substantial growth in memory capacity and bandwidth as we enter a new era of global intelligence. Scaling memory however creates challenges with energy consumption and energy efficiency is of utmost importance. This presentation will outline the trends driving memory technology and efficiency requirements and the solutions provided to scale future computing for this new computing era..

Server memory solutions had faced challenges at every step forward. 4400Mbps DDR5 SDRAM had been adopted as the 1st generation after 3200Mbs DDR4, and has reached to 6400 today with expectation of expanding to 9200Mbps in coming years. This presentation tries to review brief historical steps of evolutions up to today, and to look ahead to the future.

Coming soon

As the industry continues to push the boundaries of memory system performance, the challenges in system characterization and qualification are increasing in complexity. The rapid evolution of technologies, such as AI, high-performance computing, and mobile devices has placed new demands on our testing capabilities. The ability to probe and measure high-speed signals in modern systems is becoming more difficult, as the bandwidth of our testing equipment struggles to keep up with the speed of signaling. In this presentation we will explore these growing challenges in the context of memory subsystems. We will look into the unique issues faced in qualifying memory modules across different industry segments, focusing on high end server-class memory. Additionally, we will examine the physical limitations of current equipment, specifically the difficulty of mid-bus probing. We will show how emerging advanced testing solutions are helping to address these issues in leading new memory technologies such as RDIMM, MRDIMM and CAMM2 module families.

HBM, LPDDR, GDDR, and DDR DRAM; on-die SRAM; novel nonvolatile memories; and CXL(TM)-based storage devices have all been proposed as the main memory in different AI processors. How should a chip architect choose the correct memory for their AI application? This presentation will focus on the non-Von-Neumann GPU/TPU/NPU architectures used in AI, how they interact with memory, and how that drives the choice of memory for edge, client and datacenter AI chips.

AI and HPC performance is being limited by the memory and IO walls. The performance of HPC & AI architectures can be improved by utilizing HBM4 DRAM (a new specification recently published by JEDEC) along with custom HBM4 solutions.

Over time, as memory interface speeds have increased, the fundamental approach used to move data has had to change. Traditional high speed digital timing and noise with min/typ/max specifications has given way to high speed serial approaches based on eye masks with jitter specifications. Next generation memory must go a step further to deal with distorted eyes using tunable equalization. At each point the need to characterize and measure what’s defined in the spec has made measurement science and design for test (DFT) increasingly important in defining a standard. This session will focus on the Measurement Science as used for both device and system validation.

Mario Martinez is a seasoned Product Marketing & Sales Director with over 15 years of experience in the hardware and semiconductor industry. He has held key roles at Netlist Inc., Sanmina/Viking Interworks, Cypress Semiconductor, SK Hynix, and Toshiba where he led product marketing, strategic planning, applications, and business development efforts. Mario has been instrumental in driving growth and innovation, securing key victories with enterprise server and storage customers, and establishing global standards as a JEDEC Board of Directors representative. His expertise spans product marketing, testing, validation, and market analysis, with a strong focus on memory solutions for servers, storage, and networking environments. Mario holds a Bachelor of Science in Electrical Engineering from California State Polytechnic University-Pomona and has been awarded multiple patents for his contributions to the field.