Neura Market
FasterTransformer logo

FasterTransformer

Free

NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)

FreeFree tier
Inputs: textOutputs: text
Type
Open Source
Company
NVIDIA
Visit Website GitHub (38 stars)

About FasterTransformer

FasterTransformer is a high-performance library from NVIDIA that provides highly optimized implementations of transformer-based encoder and decoder components for inference. Built on CUDA, cuBLAS, and cuBLASLt, it leverages Tensor Cores on Volta, Turing, and Ampere GPUs for accelerated FP16 inference. The library offers APIs and integration examples for TensorFlow, PyTorch, and Triton Inference Server, and supports a wide range of models including BERT, GPT, GPT-J, BLOOM, T5, and Swin Transformer. Features include support for FP16, INT8 quantization (Turing+), FP8 (Hopper), tensor parallelism, pipeline parallelism, and sparsity. Note: Development of FasterTransformer has transitioned to TensorRT-LLM; the repository remains available but will not receive further updates.

Key Features

Highly optimized encoder and decoder transformer layers for inference
Supports FP16, INT8, FP8, and sparsity (depending on GPU architecture)
Tensor parallelism and pipeline parallelism for large models
APIs for TensorFlow, PyTorch, and Triton Inference Server
Includes example code and performance benchmarks for various models
Built with CUDA, cuBLAS, cuBLASLt, and C++

Pros & Cons

Pros
  • High performance on NVIDIA GPUs via Tensor Core utilization
  • Support for multiple precision levels (FP16, INT8, FP8) for speed or memory trade-offs
  • Seamless integration with popular frameworks like TensorFlow and PyTorch
  • Open source and free to use
  • Well-documented with examples and performance data
Cons
  • No longer actively developed; all future improvements merge into TensorRT-LLM
  • Limited to NVIDIA GPUs (requires CUDA/cuBLAS)
  • May require C++/CUDA knowledge for custom extensions
  • No support for non-NVIDIA hardware

Best For

BERT inference for natural language understanding tasksGPT and GPT-J text generation inferenceT5 and UL2 sequence-to-sequence generationBLOOM and GPT-MoE large language model inferenceSwin Transformer and ViT computer vision inference

FAQ

What is FasterTransformer?
FasterTransformer is a library from NVIDIA that provides highly optimized implementations of transformer-based encoder and decoder components for inference, targeting NVIDIA GPUs.
Which models are supported?
FasterTransformer supports BERT, GPT, GPT-J, BLOOM, T5, UL2, T5-MoE, Swin Transformer, ViT, and more, across TensorFlow, PyTorch, and Triton backends.
Is FasterTransformer still under development?
No, development has transitioned to TensorRT-LLM. The repository remains available for reference but will not receive further updates.
What are the hardware requirements?
FasterTransformer requires NVIDIA GPUs with CUDA support. Tensor Cores on Volta, Turing, Ampere, or Hopper GPUs are used for acceleration.
Does FasterTransformer support quantization?
Yes, it supports INT8 quantization on Turing and later GPUs, FP8 on Hopper, and sparsity on Ampere and later.