HuggingFaceFW/fineweb-edu

O dataset HuggingFaceFW/fineweb-edu está entre os destaques do Hugging Face — dados que alimentam o treinamento e a avaliação dos modelos do momento.

Ficha do dataset

Sobre o dataset

📚 FineWeb-Edu 1.3 trillion tokens of the finest educational data the 🌐 web has to offer Paper: https://arxiv.org/abs/2406.17557 What is it? 📚 FineWeb-Edu dataset consists of 1.3T tokens and 5.4T tokens (FineWeb-Edu-score-2) of educational web pages filtered from 🍷 FineWeb dataset. This is the 1.3 trillion version. To enhance FineWeb's quality, we developed an educational quality classifier using annotations generated by LLama3-70B-Instruct. We… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu.

Como carregar

Use a biblioteca datasets do Hugging Face:

pip install -U datasets

Como é um dataset grande, vale carregar em modo streaming (sem baixar tudo):

from datasets import load_dataset

ds = load_dataset("HuggingFaceFW/fineweb-edu", split="train", streaming=True)
for exemplo in ds.take(3):
    print(exemplo)

Tags

text-generation

Explorar o dataset no Hugging Face →