Damo-vilab
Rank:Average Model Cost: $0.0000
Number of Runs: 48,489
Models by this creator
text-to-video-ms-1.7b
text-to-video-ms-1.7b
The text-to-video-synthesis model is a multi-stage diffusion-based model that takes a text description as input and generates a video that matches the description. It consists of three sub-networks: a text feature extraction model, a text feature-to-video latent space diffusion model, and a video latent space to video visual space model. The model has approximately 1.7 billion parameters and currently only supports English input. It generates videos through an iterative denoising process from pure Gaussian noise video. The model is meant for research purposes and has limitations and biases related to training data and generating complex compositions. It should not be used for generating content that is demeaning, harmful, pornographic, violent, or contains false information.
$-/run
44.4K
Huggingface
modelscope-damo-text-to-video-synthesis
modelscope-damo-text-to-video-synthesis
Platform did not provide a description for this model.
$-/run
4.0K
Huggingface
text-to-video-ms-1.7b-legacy
text-to-video-ms-1.7b-legacy
Text-to-video-synthesis Model in Open Domain This model is based on a multi-stage text-to-video generation diffusion model, which inputs a description text and returns a video that matches the text description. Only English input is supported. Model description The text-to-video generation diffusion model consists of three sub-networks: text feature extraction model, text feature-to-video latent space diffusion model, and video latent space to video visual space model. The overall model parameters are about 1.7 billion. Currently, it only supports English input. The diffusion model adopts a UNet3D structure, and implements video generation through the iterative denoising process from the pure Gaussian noise video. This model is meant for research purposes. Please look at the model limitations and biases and misuse, malicious use and excessive use sections. Model Details Developed by: ModelScope Model type: Diffusion-based text-to-video generation model Language(s): English License: CC-BY-NC-ND Resources for more information: ModelScope GitHub Repository, Summary. Cite as: Use cases This model has a wide range of applications, and can reason and generate videos based on arbitrary English text descriptions. Usage Let's first install the libraries required: Now, generate a video: Here are some results: Long Video Generation You can optimize for memory usage by enabling attention and VAE slicing and using Torch 2.0. This should allow you to generate videos up to 25 seconds on less than 16GB of GPU VRAM. View results The above code will display the save path of the output video, and the current encoding format can be played with VLC player. The output mp4 file can be viewed by VLC media player. Some other media players may not view it normally. Model limitations and biases The model is trained based on public data sets such as Webvid, and the generated results may have deviations related to the distribution of training data. This model cannot achieve perfect film and television quality generation. The model cannot generate clear text. The model is mainly trained with English corpus and does not support other languages at the moment**. The performance of this model needs to be improved on complex compositional generation tasks. Misuse, Malicious Use and Excessive Use The model was not trained to realistically represent people or events, so using it to generate such content is beyond the model's capabilities. It is prohibited to generate content that is demeaning or harmful to people or their environment, culture, religion, etc. Prohibited for pornographic, violent and bloody content generation. Prohibited for error and false information generation. Training data The training data includes LAION5B, ImageNet, Webvid and other public datasets. Image and video filtering is performed after pre-training such as aesthetic score, watermark score, and deduplication. (Part of this model card has been taken from here)
$-/run
73
Huggingface