Vilsonrodrigues

Rank:

Average Model Cost: $0.0000

Number of Runs: 18,518

Models by this creator

falcon-7b-instruct-sharded

The falcon-7b-instruct-sharded model is a resharded version of the falcon-7b-instruct model. It is designed to be used in low RAM environments, such as Colab or Kaggle, and is implemented using safetensors. Falcon-7B-Instruct is a 7B parameter causal decoder-only model developed by TII. It is based on the Falcon-7B model and has been fine-tuned on a mixture of chat and instruct datasets. The model is optimized for inference and features FlashAttention and multiquery. It is recommended for users who are looking for a ready-to-use chat/instruct model based on Falcon-7B. However, it may not be ideal for further finetuning. The model is trained on English and French data and has an Apache 2.0 license. It should be used with appropriate precautions for production use. The model has been fine-tuned on a mixture of instruct and chat datasets and is not optimized for NLP benchmarks. It has a causal decoder-only architecture and was trained on a causal language modeling task. The model was trained on AWS SageMaker using a custom distributed training codebase. The citation and contact information for the model is provided.

$-/run

18.2K

Huggingface

falcon-7b-sharded

vilsonrodrigues

Resharded Resharded version of https://huggingface.co/tiiuae/falcon-7b for low RAM enviroments (e.g. Colab, Kaggle) in safetensors 🚀 Falcon-7B Falcon-7B is a 7B parameters causal decoder-only model built by TII and trained on 1,500B tokens of RefinedWeb enhanced with curated corpora. It is made available under the Apache 2.0 license. Paper coming soon 😊. 🤗 To get started with Falcon (inference, finetuning, quantization, etc.), we recommend reading this great blogpost fron HF! Why use Falcon-7B? It outperforms comparable open-source models (e.g., MPT-7B, StableLM, RedPajama etc.), thanks to being trained on 1,500B tokens of RefinedWeb enhanced with curated corpora. See the OpenLLM Leaderboard. It features an architecture optimized for inference, with FlashAttention (Dao et al., 2022) and multiquery (Shazeer et al., 2019). It is made available under a permissive Apache 2.0 license allowing for commercial use, without any royalties or restrictions. ⚠️ Falcon is now available as a core model in the transformers library! To use the in-library version, please install the latest version of transformers with pip install git+https://github.com/ huggingface/transformers.git, then simply remove the trust_remote_code=True argument from from_pretrained(). ⚠️ This is a raw, pretrained model, which should be further finetuned for most usecases. If you are looking for a version better suited to taking generic instructions in a chat format, we recommend taking a look at Falcon-7B-Instruct. 🔥 Looking for an even more powerful model? Falcon-40B is Falcon-7B's big brother! 💥 Falcon LLMs require PyTorch 2.0 for use with transformers! For fast inference with Falcon, check-out Text Generation Inference! Read more in this blogpost. You will need at least 16GB of memory to swiftly run inference with Falcon-7B. Model Card for Falcon-7B Model Details Model Description Developed by: https://www.tii.ae; Model type: Causal decoder-only; Language(s) (NLP): English and French; License: Apache 2.0. Model Source Paper: coming soon. Uses Direct Use Research on large language models; as a foundation for further specialization and finetuning for specific usecases (e.g., summarization, text generation, chatbot, etc.) Out-of-Scope Use Production use without adequate assessment of risks and mitigation; any use cases which may be considered irresponsible or harmful. Bias, Risks, and Limitations Falcon-7B is trained on English and French data only, and will not generalize appropriately to other languages. Furthermore, as it is trained on a large-scale corpora representative of the web, it will carry the stereotypes and biases commonly encountered online. Recommendations We recommend users of Falcon-7B to consider finetuning it for the specific set of tasks of interest, and for guardrails and appropriate precautions to be taken for any production use. How to Get Started with the Model Training Details Training Data Falcon-7B was trained on 1,500B tokens of RefinedWeb, a high-quality filtered and deduplicated web dataset which we enhanced with curated corpora. Significant components from our curated copora were inspired by The Pile (Gao et al., 2020). The data was tokenized with the Falcon-7B/40B tokenizer. Training Procedure Falcon-7B was trained on 384 A100 40GB GPUs, using a 2D parallelism strategy (PP=2, DP=192) combined with ZeRO. Training happened in early March 2023 and took about two weeks. Evaluation Paper coming soon. See the OpenLLM Leaderboard for early results. Technical Specifications Model Architecture and Objective Falcon-7B is a causal decoder-only model trained on a causal language modeling task (i.e., predict the next token). The architecture is broadly adapted from the GPT-3 paper (Brown et al., 2020), with the following differences: Positionnal embeddings: rotary (Su et al., 2021); Attention: multiquery (Shazeer et al., 2019) and FlashAttention (Dao et al., 2022); Decoder-block: parallel attention/MLP with a single layer norm. Compute Infrastructure Falcon-7B was trained on AWS SageMaker, on 384 A100 40GB GPUs in P4d instances. Falcon-7B was trained a custom distributed training codebase, Gigatron. It uses a 3D parallelism approach combined with ZeRO and high-performance Triton kernels (FlashAttention, etc.) Citation Paper coming soon 😊. In the meanwhile, you can use the following information to cite: To learn more about the pretraining dataset, see the 📓 RefinedWeb paper. License Falcon-7B is made available under the Apache 2.0 license. Contact falconllm@tii.ae