News
IndustryTSMC’s grip on advanced chipmaking will face competitionIndustryCadence joins European Chip Design Platform to support semiconductor startups and SMEsMaking chipsMarc Hijink on ASML’s Culture, Supply Chain, and GeopoliticsProcessorsIntel's post-Nova Lake CPUs take shape as Razor Lake desktop and laptop variants surfaceAI chipsHow Google’s New AI Chip Software Helps It Battle NvidiaIndiaMore IC Packaging Plants Emerge In IndiaIndustryUS Tightens China AI Chip Access as Nvidia, TSMC and Semiconductor Stocks SlideDesignAxelera AI moves into HPC with TitaniaMaking chipsChinese envoy says chip equipment export curbs threaten EU-China trade thawIndiaSEMICON India 2026 Day 2 Sees 25 MoUs, Strategic Collaborations Across Semiconductor Value ChainIndustryTSMC’s grip on advanced chipmaking will face competitionIndustryCadence joins European Chip Design Platform to support semiconductor startups and SMEsMaking chipsMarc Hijink on ASML’s Culture, Supply Chain, and GeopoliticsProcessorsIntel's post-Nova Lake CPUs take shape as Razor Lake desktop and laptop variants surfaceAI chipsHow Google’s New AI Chip Software Helps It Battle NvidiaIndiaMore IC Packaging Plants Emerge In IndiaIndustryUS Tightens China AI Chip Access as Nvidia, TSMC and Semiconductor Stocks SlideDesignAxelera AI moves into HPC with TitaniaMaking chipsChinese envoy says chip equipment export curbs threaten EU-China trade thawIndiaSEMICON India 2026 Day 2 Sees 25 MoUs, Strategic Collaborations Across Semiconductor Value Chain
EveryChip
AWS Inferentia

AWS Inferentia

AI inference chip designed by Amazon Web Services

AWS Inferentia is a custom silicon family built by Amazon for deep learning inference in the cloud. It powers EC2 Inf1 and Inf2 instances, aiming to lower costs and latency for generative AI and large language models.

Share on WhatsApp

  • Designed by AWS for deep learning inference
  • Inferentia2 offers 32 GB HBM memory per chip
  • Supports FP8 data type for flexible precision
  • Powers Amazon EC2 Inf1 and Inf2 instances
  • Enables scale-out distributed inference across chips

What it is

AWS Inferentia is a series of artificial intelligence chips created by Amazon Web Services. These processors are built specifically for running deep learning models in the cloud, rather than training them. The hardware targets inference tasks, which involve using trained models to make predictions or generate content. Amazon states that these chips offer high performance at a lower cost compared to general-purpose computing instances. They are integrated into the Amazon Elastic Compute Cloud service, allowing developers to access the hardware without managing physical servers.

How it came about

Amazon designed the first-generation Inferentia chip to power its EC2 Inf1 instances. The company claims this initial release provided higher throughput and reduced costs for inference workloads. Following this, Amazon developed Inferentia2 to handle more complex models like large language generators. This second generation supports distributed inference across multiple chips with high-speed connections. Customers such as Leonardo.ai and Deutsche Telekom have adopted the newer Inf2 instances for their generative AI applications.

Inside the chip

The chip architecture relies on custom units called NeuronCores. The first version includes four of these cores per chip, while the second generation has two more powerful cores. Inferentia2 delivers up to 190 tera floating point operations per second in half-precision format. Memory capacity increased significantly from 8 GB of DDR4 in the first chip to 32 GB of high-bandwidth memory in the second. The design supports various data types, including FP16, BF16, INT8, and the newer configurable FP8. It also features hardware optimizations for dynamic input sizes and stochastic rounding to improve accuracy.

How it was made

Amazon Web Services is identified as the designer of the chip family. The physical packaging details and specific clock speeds are not provided in the available information. The focus of the public documentation remains on the logical architecture and performance metrics within the cloud environment rather than the physical fabrication details.

Where it went

These chips power Amazon EC2 Inf1 and Inf2 instances. They are used for deploying deep learning models in production environments. Common applications include natural language processing, text summarization, and language translation. The hardware also supports generative tasks like image and video creation, speech recognition, and fraud detection. Companies use them to run large language models and latent diffusion models at scale. The system aims to reduce latency and cost for real-time data processing and customer experience management tools.

Versions and family

The family consists of two main generations. The first-generation Inferentia chip features four NeuronCores and 8 GB of DDR4 memory. It supports FP16, BF16, and INT8 data formats. Inferentia2 is the successor, offering two second-generation NeuronCores and 32 GB of high-bandwidth memory. It adds support for FP32, TF32, and configurable FP8 types. Inferentia2 provides higher throughput and lower latency than its predecessor. It also enables scale-out distributed inference, a feature not present in the first generation.

Why it matters

Amazon claims the chips significantly lower the cost per inference compared to comparable EC2 instances. Users report cost reductions of up to 80 percent and improved throughput. The technology allows companies to deploy larger and more complex AI models while maintaining budget controls. It supports sustainability goals by offering better performance per watt. The availability of these specialized chips has enabled businesses to process larger data volumes and reduce model latency without retraining models for lower precision.

Problems and reception

Several companies have publicly shared positive experiences with the hardware. Finch Computing reported an 80 percent cost reduction over graphics processing units. Leonardo.ai stated that the chips allowed them to cut costs by 80 percent without losing performance. Qualtrics highlighted the benefits of distributed inference for scaling complex models. Dataminr noted a nine-fold improvement in throughput per dollar. These testimonials suggest the hardware meets the needs of enterprises requiring high-volume, low-latency AI processing.

Questions people ask about AWS Inferentia

What is AWS Inferentia used for?

It is used for deep learning inference in the cloud. This includes running large language models, generating images, translating text, and detecting fraud. It helps companies deploy AI models with lower latency and cost.

How does Inferentia2 differ from the first generation?

Inferentia2 has more memory (32 GB HBM vs 8 GB DDR4) and supports new data types like FP8. It offers higher throughput and lower latency, and it supports distributed inference across multiple chips.

Do I need to rewrite my code to use these chips?

No, the AWS Neuron SDK integrates with PyTorch and TensorFlow. It allows developers to deploy existing models with minimal code changes. The SDK handles optimization and data type casting automatically.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on AWS Inferentia yet, so the facts on this page come from the sources listed above: aws.amazon.com/ai/machine-learning/inferentia/. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryChip wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the chip of the day, new pages on EveryChip, and the conferences and anniversaries of the week ahead.