free-vc
Maintainer: jagilley
60
| Property | Value |
|---|---|
| Model Link | View on Replicate |
| API Spec | View on Replicate |
| Github Link | View on Github |
| Paper Link | View on Arxiv |
Get summaries of the top AI models delivered straight to your inbox:
Model overview
The free-vc model is a tool developed by jagilley that allows you to change the voice of spoken text. It can be used to convert the audio of one person's voice to sound like another person's voice. This can be useful for applications like voice over, dubbing, or text-to-speech. The free-vc model is similar in capabilities to other voice conversion models like VoiceConversionWebUI, incredibly-fast-whisper, voicecraft, and styletts2.
Model inputs and outputs
The free-vc model takes two inputs: a source audio file containing the words that should be spoken, and a reference audio file containing the voice that the resulting audio should have. The model then outputs a new audio file with the source text spoken in the voice of the reference audio.
Inputs
- Source Audio: The audio file containing the words that should be spoken
- Reference Audio: The audio file containing the voice that the resulting audio should have
Outputs
- Output Audio: The new audio file with the source text spoken in the voice of the reference audio
Capabilities
The free-vc model can be used to change the voice of any spoken audio, allowing you to convert one person's voice to sound like another. This can be useful for a variety of applications, such as voice over, dubbing, or text-to-speech.
What can I use it for?
The free-vc model can be used for a variety of applications, such as:
- Voice Over: Convert the voice in a video or audio recording to sound like a different person.
- Dubbing: Change the voice in a foreign language film or video to match the local language.
- Text-to-Speech: Generate audio of text spoken in a specific voice.
Things to try
Some ideas for things to try with the free-vc model include:
- Experiment with different source and reference audio files to see how the resulting audio sounds.
- Try using the model to create a voice over or dub for a short video or audio clip.
- See if you can use the model to generate text-to-speech audio in a specific voice.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Models
📊
openvoice
6
The openvoice model, developed by the team at MyShell, is a versatile instant voice cloning AI that can accurately clone the tone color and generate speech in multiple languages and accents. It offers flexible control over voice styles, such as emotion and accent, as well as other style parameters like rhythm, pauses, and intonation. The model also supports zero-shot cross-lingual voice cloning, allowing it to generate speech in languages not present in the training dataset. The openvoice model builds upon several excellent open-source projects, including TTS, VITS, and VITS2. It has been powering the instant voice cloning capability of myshell.ai since May 2023 and has been used tens of millions of times by users worldwide, witnessing explosive growth on the platform. Model inputs and outputs Inputs Audio**: The reference audio used to clone the tone color. Text**: The text to be spoken by the cloned voice. Speed**: The speed scale of the output audio. Language**: The language of the audio to be generated. Outputs Output**: The generated audio in the cloned voice. Capabilities The openvoice model excels at accurate tone color cloning, flexible voice style control, and zero-shot cross-lingual voice cloning. It can generate speech in multiple languages and accents, while allowing for granular control over voice styles, including emotion and accent, as well as other parameters like rhythm, pauses, and intonation. What can I use it for? The openvoice model can be used for a variety of applications, such as: Instant voice cloning for audio, video, or gaming content Customized text-to-speech for assistants, chatbots, or audiobooks Multilingual voice acting and dubbing Voice conversion and style transfer Things to try With the openvoice model, you can experiment with different input reference audios to clone a wide range of voices and accents. You can also play with the style parameters to create unique and expressive speech outputs. Additionally, you can explore the model's cross-lingual capabilities by generating speech in languages not present in the training data.
Updated Invalid Date
realistic-voice-cloning
165
The realistic-voice-cloning model, created by zsxkib, is an AI model that can create song covers by cloning a specific voice from audio files. It builds upon the Realistic Voice Cloning (RVC v2) technology, allowing users to generate vocals in the style of any RVC v2 trained voice. This model offers an alternative to similar voice cloning models like create-rvc-dataset, openvoice, free-vc, train-rvc-model, and voicecraft, each with its own unique features and capabilities. Model inputs and outputs The realistic-voice-cloning model takes a variety of inputs that allow users to fine-tune the generated vocals, including the RVC model to use, pitch changes, reverb settings, and more. The output is a generated audio file in either MP3 or WAV format, containing the original song's vocals replaced with the cloned voice. Inputs Song Input**: The audio file to use as the source for the song RVC Model**: The specific RVC v2 model to use for the voice cloning Pitch Change**: Adjust the pitch of the AI-generated vocals Index Rate**: Control the balance between the AI's accent and the original vocals RMS Mix Rate**: Adjust the balance between the original vocal's loudness and a fixed loudness Filter Radius**: Apply median filtering to the harvested pitch results Pitch Detection Algorithm**: Choose between different pitch detection algorithms Protect**: Control the amount of original vocals' breath and voiceless consonants to leave in the AI vocals Reverb Size, Damping, Dryness, and Wetness**: Adjust the reverb settings Pitch Change All**: Change the pitch/key of the background music, backup vocals, and AI vocals Volume Changes**: Adjust the volume of the main AI vocals, backup vocals, and background music Outputs The generated audio file in either MP3 or WAV format, with the original vocals replaced by the cloned voice Capabilities The realistic-voice-cloning model can create high-quality song covers by replacing the original vocals with a cloned voice. Users can fine-tune the generated vocals to achieve their desired sound, adjusting parameters like pitch, reverb, and volume. This model is particularly useful for musicians, content creators, and audio engineers who want to create unique vocal covers or experiments with different voice styles. What can I use it for? The realistic-voice-cloning model can be used to create song covers, remixes, and other audio projects where you want to replace the original vocals with a different voice. This can be useful for musicians who want to experiment with different vocal styles, content creators who want to create unique covers, or audio engineers who need to modify existing vocal tracks. The model's ability to fine-tune the generated vocals also makes it suitable for professional audio production work. Things to try With the realistic-voice-cloning model, you can try creating unique song covers by cloning the voice of your favorite singers or even your own voice. Experiment with different RVC models, pitch changes, and reverb settings to achieve the desired sound. You could also explore using the model to create custom vocal samples or background vocals for your music productions. The versatility of the model allows for a wide range of creative possibilities.
Updated Invalid Date
metavoice
6
MetaVoice-1B is a 1.2 billion parameter base model trained on 100,000 hours of speech, developed by the MetaVoice team. This large-scale speech model can be used for a variety of text-to-speech and audio generation tasks, similar to models like ml-mgie, meta-llama-3-8b-instruct, whisperspeech-small, voicecraft, and whisperx. Model inputs and outputs MetaVoice-1B takes in text as input and generates audio as output. The model can be used for a wide range of text-to-speech and audio generation tasks. Inputs Text**: The text to be converted to speech. Outputs Audio**: The generated audio in a URI format. Capabilities MetaVoice-1B is a powerful foundational audio model capable of generating high-quality speech from text inputs. It can be used for tasks like text-to-speech, audio synthesis, and voice cloning. What can I use it for? The MetaVoice-1B model can be used for a variety of applications, such as creating audiobooks, podcasts, or voice assistants. It can also be used to generate synthetic voices for video games, movies, or other multimedia projects. Additionally, the model can be fine-tuned for specific use cases, such as language learning or accessibility applications. Things to try With MetaVoice-1B, you can experiment with generating speech in different styles, emotions, or languages. You can also explore using the model for tasks like audio editing, voice conversion, or multi-speaker audio generation.
Updated Invalid Date
👁️
vqmivc
6
vqmivc is a one-shot voice conversion model that can perform speech representation disentanglement to convert voices across arbitrary speakers using only a single target-speaker utterance for reference. It uses a vector quantization and mutual information-based approach to learn disentangled representations of speech content, speaker, and pitch. This allows for any-to-any voice conversion without the need for parallel training data. The model was developed by Wendison, a researcher at the Chinese University of Hong Kong. Similar voice conversion models include free-vc which can change the voice of spoken text, openvoice which offers versatile instant voice cloning, and voicecraft which enables zero-shot speech editing and text-to-speech. Model inputs and outputs The vqmivc model takes two audio files as input - the source voice and the reference (target) voice. It then outputs the converted audio, where the content and pitch of the source voice is converted to match the speaker identity of the reference voice. Inputs input_source**: The path to the source voice wav file input_reference**: The path to the reference (target) voice wav file Outputs Output**: The path to the converted voice wav file Capabilities vqmivc can perform one-shot voice conversion, meaning it can convert the voice of a source speaker to match the voice of a target speaker using only a single reference utterance from the target speaker. This is achieved through unsupervised disentanglement of the speech representations, allowing the model to separate content, speaker identity, and pitch. By minimizing the mutual information between these latent factors, the model can effectively convert the voice while preserving the linguistic content. What can I use it for? The vqmivc model can be used for a variety of voice conversion applications, such as dubbing foreign language films, personalizing text-to-speech systems, and creating voice avatars. It could also be used in the entertainment industry to modify actors' voices or create synthetic voices for animated characters. Additionally, the model's ability to convert voices without parallel training data makes it useful for low-resource languages or scenarios where data is scarce. Things to try One interesting thing to try with vqmivc is to experiment with the mutual information minimization (MIM) training approach. By training the model with and without MIM, you can observe how the disentanglement of speech representations affects the quality and fidelity of the voice conversion. Additionally, you could try using the model to convert between a wide variety of speakers, exploring its capabilities for any-to-any voice conversion.
Updated Invalid Date