A New Solution for AI Computation Costs? 'TrnRT', an Open-Source Engine for Running Large Language Models on Amazon Chips, Emerges

A visual illustration of a modern data center symbolizing Amazon Trainium chips and server infrastructure.
AI Summary

Startup RunAnywhere has released 'TrnRT', an open-source inference engine capable of efficiently running the GLM-5.3 flash model on AWS Trainium chips, presenting a new alternative to reduce costly reliance on Nvidia.

Imagine this: If you run a massive artificial intelligence (AI) service used by millions every day, your heart would sink every time server maintenance costs and electricity bills arrive. Today, the vast majority of AI models run on Nvidia’s high-performance graphics cards (GPUs), but their hardware prices and maintenance costs are no small matter. What if there was a way to run this massive AI much more cheaply on proprietary AI-dedicated chips created by Amazon, the world’s top cloud market player?

Recently, a Silicon Valley startup appeared with a clear answer to this question. Meet TrnRT (Trainium RunTime), unveiled through the Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium project [Source 1]. Let’s examine the background and significance of why this new open-source engine is receiving hot attention in the AI industry.


Why is this important?

As the chatbots and AI assistants we use in daily life become smarter, a massive amount of computation takes place in the background. This is called ‘inference’, which is the core process where AI receives user queries and generates answers in real time. Because this inference process requires tremendous amounts of memory and computing power, “how to provide faster answers at a lower cost” is a matter of business survival for companies.

Until now, high-performance chips like Nvidia’s H100 or H200 have monopolized the market. However, as AI demand has exploded, chip purchase costs and cloud rental prices have skyrocketed. Amidst this, Trainium, an AI accelerator chip independently developed by Amazon Web Services (AWS), has been cited as an attractive alternative to Nvidia [Source 5].

However, there was a realistic obstacle. Even with exceptional hardware, there was no suitable ‘software engine’ to efficiently run large language models tailored precisely to the chip’s characteristics. In fact, some of Amazon’s existing native stacks showed limitations in failing to properly support latest models or having development discontinued [Source 1]. Appearing to fill this software gap, TrnRT has opened a breakthrough for companies to reduce their heavy reliance on Nvidia and build high-performance AI services at much more economical costs.


Easy to Understand: The Analogy of a Delivery Truck and a Supercar

Let’s use an analogy so that even those unfamiliar with computer architecture or AI hardware can easily understand.

  • Nvidia GPUs (H100/H200) are like luxury supercars speeding on the road. They are extremely fast and dazzlingly performant, but the vehicle price is exorbitantly expensive and maintenance costs (fuel and insurance) are overwhelmingly huge.
  • Amazon Trainium chips can be compared to large delivery trucks stably carrying massive amounts of cargo. While their potential to process countless data is immense, that truck needs dedicated roads and a sophisticated navigation system to unleash 100% performance.

Here, TrnRT acts as an ‘optimized navigation and high-performance engine control program’ designed to let that large truck safely travel rugged mountain roads and complex city centers at top speed.

Put simply, when a book requested by a reader in a giant library (large language model) must be found and delivered in an instant, TrnRT is like a seasoned and efficient head librarian who fully grasps the library’s bookshelf structure and retrieves books via the fastest route. In particular, according to [Serving GLM-5.3-Flash on one Trainium2 box with NKI… RunAnywhere](https://runanywhere.ai/blog/glm-5-3-flash-trainium2-nki-megakernels), RunAnywhere continues its unique philosophy of overcoming physical hardware limitations through software optimization, having developed MetalRT for Apple Silicon, QHexRT for Qualcomm chips, and now TrnRT for AWS Trainium [Source 4].

Current Status: What Has Been Proved?

The team that developed TrnRT achieved industry-astonishing results in just three weeks of starting the project. They successfully ran the high-performance large language model GLM-5.3 flash (FP8) on AWS’s latest Trainium hardware environment, the trn2.48xlarge instance, proving stable service status [Source 1].

The specific achievements they accomplished are as follows:

  • High-Density Serving: Demonstrated the ability to simultaneously and stably serve two instances of GLM-5.3 flash in a single box (an environment equipped with 8 trn2 chips) using cache-aware router technology [Source 1].
  • Overwhelming Throughput: Recorded tremendous speed and performance, processing a staggering 3.57 million tokens per minute (3.57M TPM) in strict AI benchmark testing environments [Source 1].
  • Outstanding Economic Advantage: Memory cost per terabyte in the trn2.48xlarge instance environment is less than half compared to using 8 H200 chips within the AWS cloud [Source 1].

Previously, there was a gap where AWS’s native stacks (e.g., vLLM Neuron, etc.) failed to timely register latest GLM models, or distributed inference frameworks like NxDI were no longer actively maintained [Source 1]. In this situation, talented developers building this engine themselves as Apache 2.0-licensed open source and releasing it to the world Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium is injecting tremendous vitality into the global developer community [Source 1, Source 4].


What’s Next?

TrnRT, making its debut now, is merely the starting point of a grand journey toward reducing AI infrastructure costs. Based on the initial high-performance and high-efficiency success experience, the development team plans to rapidly expand the types of large language models supported moving forward [Source 1].

In particular, because it is open to a structure where open-source contributors and developers worldwide can freely participate [Source 1], optimization code is expected to accumulate and evolve much faster than the closed software update cycles provided directly by hardware manufacturers. In the future, countless companies will be able to choose high-performance alternative chips like Amazon Trainium alongside Nvidia GPUs, dramatically lowering AI infrastructure maintenance costs and providing more reasonably priced AI services to end users.


MindTickleBytes’ AI Reporter Perspective

As important as the dazzling speed of AI technology advancement is ‘sustainable affordability that anyone can access without burden’. It is time to break away from the past structure where only a handful of mega-corporations monopolized expensive GPUs with massive funding and passed those costs on to consumers. The combination of diverse custom hardware and agile open-source software is tearing down the cost barrier, which is the biggest obstacle to AI popularization. The emergence of TrnRT is a fresh and important milestone demonstrating how powerful synergy is generated when hardware diversity meets software creativity.


References

  1. Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium
  2. [Serving GLM-5.3-Flash on one Trainium2 box with NKI… RunAnywhere](https://runanywhere.ai/blog/glm-5-3-flash-trainium2-nki-megakernels)
  3. NeuronMM: High-Performance Matrix Multiplication for LLM Inference…
AD
Test Your Understanding
Q1. Who developed and open-sourced the TrnRT engine?
  • Amazon Web Services (AWS)
  • RunAnywhere
  • OpenAI
TrnRT is an inference engine for AWS Trainium developed and open-sourced under the Apache 2.0 license by startup RunAnywhere.
Q2. What is the representative large language model currently supported by TrnRT?
  • GPT-4o
  • GLM-5.3 flash
  • Claude 3.5 Sonnet
TrnRT currently serves the GLM-5.3 flash (FP8) model on AWS trn2.48xlarge instances.
Q3. What is one of the biggest economic advantages of choosing Trainium-based hardware?
  • Memory cost per terabyte is less than half that of Nvidia H200-based systems
  • Electricity bills are completely waived
  • Server hardware is rented for free
The memory cost per terabyte for the trn2.48xlarge instance is less than half that of AWS's 8x H200 configuration.
A New Solution for AI Compu...
0:00