
	
		
	
	
		I-BERT base model
	


This model, ibert-roberta-base, is an integer-only quantized version of RoBERTa, and was introduced in this paper.
I-BERT stores all parameters with INT8 representation, and carries out the entire inference using integer-only arithmetic.
In particular, I-BERT replaces all floating point operations in the Transformer architectures (e.g., MatMul, GELU, Softmax, and LayerNorm) with closely approximating integer operations.
This can result in upto 4x inference speed up as compared to floating point counterpart when tested on an Nvidia T4 GPU.
The best model parameters searched via quantization-aware finetuning can be then exported (e.g., to TensorRT) for integer-only deployment of the model.


	
		
	
	
		Finetuning Procedure
	


Finetuning of I-BERT consists of 3 stages: (1) Full-precision finetuning from the pretrained model on a down-stream task, (2) model quantization, and (3) integer-only finetuning (i.e., quantization-aware training) of the quantized model.


	
		
	
	
		Full-precision finetuning
	


Full-precision finetuning of I-BERT is similar to RoBERTa finetuning.
For instance, you can run the following command to finetune on the MRPC text classification task.


	
		
	
	
		Model Quantization
	


Once you are done with full-precision finetuning, open up config.json in your checkpoint directory and set the quantize attribute as true.

Then, your model will automatically run as the integer-only mode when you load the checkpoint.
Also, make sure to delete optimizer.pt, scheduler.pt and trainer_state.json in the same directory.
Otherwise, HF will not reset the optimizer, scheduler, or trainer state for the following integer-only finetuning.


	
		
	
	
		Integer-only finetuning (Quantization-aware training)
	


Finally, you will be able to run integer-only finetuning simply by loading the checkpoint file you modified.
Note that the only difference in the example command below is model_name_or_path.


	
		
	
	
		Citation info
	


If you use I-BERT, please cite our papaer.



ibert-roberta-base is an integer-only quantized version of the RoBERTa model. It stores all parameters in INT8 representation and performs inference using integer-only arithmetic. This results in up to 4x faster inference compared to the floating-point counterpart. The finetuning procedure for I-BERT consists of three stages: full-precision finetuning, model quantization, and integer-only finetuning. After full-precision finetuning, the model can be quantized by setting the quantize attribute in the config.json file. The quantized model can then be finetuned using integer-only operations. If you use I-BERT, please cite the paper.

Platform did not provide a description for this model.

Kssteven

Models by this creator

ibert-roberta-base

ibert-roberta-large-mnli

ibert-roberta-large

Similar creators