Training: unify instruction training through apply_chat_template()
Instead of two separate paths (format files vs Chat Template), all instruction training now uses apply_chat_template() with assistant-only label masking. Users pick a Jinja2 template from the dropdown or use the model's built-in chat template — both work identically.
This commit is contained in:
oobabooga
+38
-75
@@ -7,8 +7,7 @@ The WebUI seeks to make training your own LoRAs as easy as possible. It comes do
|
||||
- What are you training it on? Do you want it to learn real information, a simple format, ...?
|
||||
|
||||
### **Step 2**: Gather a dataset.
|
||||
- If you use a dataset similar to the [Alpaca](https://github.com/gururise/AlpacaDataCleaned/blob/main/alpaca_data_cleaned.json) format, that is natively supported by the `Formatted Dataset` input in the WebUI, with premade formatter options.
|
||||
- If you use a dataset that isn't matched to Alpaca's format, but uses the same basic JSON structure, you can make your own format file by copying `user_data/training/formats/alpaca-format.json` to a new file and [editing its content](#format-files).
|
||||
- For instruction/chat training, prepare a JSON dataset in one of the [supported formats](#instruction-templates) (OpenAI messages or ShareGPT).
|
||||
- For pretraining-style training on raw text, use the `Text Dataset` tab with a JSON file where each row has a `"text"` key.
|
||||
- If you use a structured dataset not in this format, you may have to find an external way to convert it - or open an issue to request native support.
|
||||
|
||||
@@ -38,41 +37,53 @@ The WebUI seeks to make training your own LoRAs as easy as possible. It comes do
|
||||
- If your model isn't learning detailed information but you want it to, you might need to just run more epochs, or you might need a higher Rank.
|
||||
- If your model is enforcing a format you didn't want, you may need to tweak your dataset, or start over and not train as far.
|
||||
|
||||
## Format Files
|
||||
## Instruction Templates
|
||||
|
||||
If using JSON formatted datasets, they are presumed to be in the following approximate format:
|
||||
All instruction/chat training uses `apply_chat_template()` with Jinja2 templates. You have two options in the **Data Format** dropdown:
|
||||
|
||||
- **Chat Template**: Uses the model's built-in chat template from its tokenizer. Works with instruct/chat models that ship with a chat template (Llama 3, Qwen, Mistral, etc.).
|
||||
- **Named template** (e.g. ChatML, Alpaca, Llama-v3, etc.): Loads a Jinja2 template from `user_data/instruction-templates/`. This is useful for base models that don't have a built-in template, or when you want to override the model's default template.
|
||||
|
||||
Both options are functionally identical — the only difference is where the Jinja2 template string comes from. In both cases:
|
||||
- The dataset is tokenized via `apply_chat_template()`
|
||||
- Labels are automatically masked so only assistant responses are trained on
|
||||
- Multi-turn conversations are supported natively
|
||||
- Special tokens are handled correctly by the template
|
||||
|
||||
The WebUI ships with 50+ templates in `user_data/instruction-templates/`. You can also add your own by creating a `.yaml` file with an `instruction_template` key containing a Jinja2 template string, or a plain `.jinja` file.
|
||||
|
||||
**Dataset formats:** Your JSON dataset can use either of these structures:
|
||||
|
||||
OpenAI messages format:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"somekey": "somevalue",
|
||||
"key2": "value2"
|
||||
},
|
||||
{
|
||||
// etc
|
||||
}
|
||||
{
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "What is Python?"},
|
||||
{"role": "assistant", "content": "A programming language."},
|
||||
{"role": "user", "content": "What's it used for?"},
|
||||
{"role": "assistant", "content": "Web dev, data science, scripting, and more."}
|
||||
]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Where the keys (eg `somekey`, `key2` above) are standardized, and relatively consistent across the dataset, and the values (eg `somevalue`, `value2`) contain the content actually intended to be trained.
|
||||
|
||||
For Alpaca, the keys are `instruction`, `input`, and `output`, wherein `input` is sometimes blank.
|
||||
|
||||
A simple format file for Alpaca to be used as a chat bot is:
|
||||
|
||||
ShareGPT format (`conversations` key with `from`/`value` fields):
|
||||
```json
|
||||
{
|
||||
"instruction,output": "User: %instruction%\nAssistant: %output%",
|
||||
"instruction,input,output": "User: %instruction%: %input%\nAssistant: %output%"
|
||||
}
|
||||
[
|
||||
{
|
||||
"conversations": [
|
||||
{"from": "system", "value": "You are a helpful assistant."},
|
||||
{"from": "human", "value": "What is Python?"},
|
||||
{"from": "gpt", "value": "A programming language."},
|
||||
{"from": "human", "value": "What's it used for?"},
|
||||
{"from": "gpt", "value": "Web dev, data science, scripting, and more."}
|
||||
]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Note that the keys (eg `instruction,output`) are a comma-separated list of dataset keys, and the values are a simple string that use those keys with `%%`.
|
||||
|
||||
So for example if a dataset has `"instruction": "answer my question"`, then the format file's `User: %instruction%\n` will be automatically filled in as `User: answer my question\n`.
|
||||
|
||||
If you have different sets of key inputs, you can make your own format file to match it. This format-file is designed to be as simple as possible to enable easy editing to match your needs.
|
||||
|
||||
## Text Dataset
|
||||
|
||||
For pretraining-style training on raw text, use the **Text Dataset** tab. Your dataset should be a JSON file with one document per row, each with a `"text"` key:
|
||||
@@ -90,54 +101,6 @@ Each document is tokenized (with BOS token), concatenated into one long token se
|
||||
|
||||
- `Stride Length` controls the overlap between consecutive chunks in tokens. Set to 0 for non-overlapping chunks (the standard concatenate-and-split approach). Values like 256 or 512 create overlapping chunks that help the model learn context across chunk boundaries, at the cost of more training samples.
|
||||
|
||||
## Chat Template Format
|
||||
|
||||
Select **Chat Template** as the Data Format to use the model's built-in chat template (via `apply_chat_template()`) instead of a format file. This works with instruct/chat models that ship with a chat template in their tokenizer (Llama 3, Qwen, Mistral, etc.).
|
||||
|
||||
**Advantages over format files:**
|
||||
- Special tokens are handled correctly by the tokenizer itself
|
||||
- Multi-turn conversations are supported natively
|
||||
- Labels are automatically masked so only assistant responses are trained on (no need for `Train Only After`)
|
||||
|
||||
**Dataset formats:** Your JSON dataset can use any of these structures:
|
||||
|
||||
OpenAI messages format (multi-turn):
|
||||
```json
|
||||
[
|
||||
{
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "What is Python?"},
|
||||
{"role": "assistant", "content": "A programming language."},
|
||||
{"role": "user", "content": "What's it used for?"},
|
||||
{"role": "assistant", "content": "Web dev, data science, scripting, and more."}
|
||||
]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
The conversation gets tokenized with the model's own chat template (correct special tokens), and the labels are automatically masked so the model only trains on the assistant responses — the system prompt and user turns get `-100` labels and contribute no gradient.
|
||||
|
||||
ShareGPT format (`conversations` key with `from`/`value` fields):
|
||||
```json
|
||||
[
|
||||
{
|
||||
"conversations": [
|
||||
{"from": "system", "value": "You are a helpful assistant."},
|
||||
{"from": "human", "value": "What is Python?"},
|
||||
{"from": "gpt", "value": "A programming language."},
|
||||
{"from": "human", "value": "What's it used for?"},
|
||||
{"from": "gpt", "value": "Web dev, data science, scripting, and more."}
|
||||
]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Simple instruction/output format (auto-converted to a single-turn conversation):
|
||||
```json
|
||||
[{"instruction": "What is 2+2?", "output": "4"}]
|
||||
```
|
||||
|
||||
## Target Modules
|
||||
|
||||
By default, **Target all linear layers** is enabled. This uses peft's `all-linear` mode, which applies LoRA to every `nn.Linear` layer in the model except the output head (`lm_head`). It works for any model architecture.
|
||||
|
||||
+63
-116
@@ -14,6 +14,7 @@ import traceback
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
import gradio as gr
|
||||
|
||||
from modules import shared, ui, utils
|
||||
@@ -25,7 +26,7 @@ from modules.evaluate import (
|
||||
from modules.logging_colors import logger
|
||||
from modules.models import reload_model
|
||||
|
||||
PARAMETERS = ["lora_name", "always_override", "all_linear", "q_proj_en", "v_proj_en", "k_proj_en", "o_proj_en", "gate_proj_en", "down_proj_en", "up_proj_en", "save_steps", "micro_batch_size", "batch_size", "epochs", "learning_rate", "lr_scheduler_type", "lora_rank", "lora_alpha", "lora_dropout", "cutoff_len", "dataset", "eval_dataset", "format", "eval_steps", "text_dataset", "higher_rank_limit", "warmup_steps", "optimizer", "stride_length", "train_only_after", "stop_at_loss", "add_eos_token", "report_to"]
|
||||
PARAMETERS = ["lora_name", "always_override", "all_linear", "q_proj_en", "v_proj_en", "k_proj_en", "o_proj_en", "gate_proj_en", "down_proj_en", "up_proj_en", "save_steps", "micro_batch_size", "batch_size", "epochs", "learning_rate", "lr_scheduler_type", "lora_rank", "lora_alpha", "lora_dropout", "cutoff_len", "dataset", "eval_dataset", "format", "eval_steps", "text_dataset", "higher_rank_limit", "warmup_steps", "optimizer", "stride_length", "stop_at_loss", "add_eos_token", "report_to"]
|
||||
WANT_INTERRUPT = False
|
||||
|
||||
train_log = {}
|
||||
@@ -96,9 +97,8 @@ def create_ui():
|
||||
|
||||
with gr.Column():
|
||||
warmup_steps = gr.Number(label='Warmup Steps', value=100, info='For this many steps at the start, the learning rate will be lower than normal. This helps the trainer prepare the model and precompute statistics to improve the quality of training after the start.')
|
||||
train_only_after = gr.Textbox(label='Train Only After', value='', info='Only consider text *after* this string in any given chunk for training. For Alpaca datasets, use "### Response:" to only train the response and ignore the input.')
|
||||
|
||||
add_eos_token = gr.Checkbox(label='Add EOS token', value=False, info="Adds EOS token for each dataset item. In case of raw text, the EOS will be added at the Hard Cut")
|
||||
add_eos_token = gr.Checkbox(label='Add EOS token', value=False, info="Adds EOS token for each document in text datasets.")
|
||||
|
||||
higher_rank_limit = gr.Checkbox(label='Enable higher ranks', value=False, info='If checked, changes Rank/Alpha slider above to go much higher. This will not work without a datacenter-class GPU.')
|
||||
report_to = gr.Radio(label="Save detailed logs with", value="None", choices=["None", "wandb", "tensorboard"], interactive=True)
|
||||
@@ -106,8 +106,8 @@ def create_ui():
|
||||
with gr.Column():
|
||||
with gr.Tab(label='Formatted Dataset'):
|
||||
with gr.Row():
|
||||
format = gr.Dropdown(choices=['None', 'Chat Template'] + [x for x in utils.get_datasets('user_data/training/formats', 'json') if x != 'None'], value='None', label='Data Format', info='The format file used to decide how to format the dataset input. "Chat Template" uses the model\'s built-in chat template via apply_chat_template().', elem_classes=['slim-dropdown'], interactive=not mu)
|
||||
ui.create_refresh_button(format, lambda: None, lambda: {'choices': ['None', 'Chat Template'] + [x for x in utils.get_datasets('user_data/training/formats', 'json') if x != 'None']}, 'refresh-button', interactive=not mu)
|
||||
format = gr.Dropdown(choices=get_instruction_templates(), value='None', label='Data Format', info='Select an instruction template for formatting the dataset, or "Chat Template" to use the model\'s built-in chat template.', elem_classes=['slim-dropdown'], interactive=not mu)
|
||||
ui.create_refresh_button(format, lambda: None, lambda: {'choices': get_instruction_templates()}, 'refresh-button', interactive=not mu)
|
||||
|
||||
with gr.Row():
|
||||
dataset = gr.Dropdown(choices=utils.get_datasets('user_data/training/datasets', 'json'), value='None', label='Dataset', info='The dataset file to use for training.', elem_classes=['slim-dropdown'], interactive=not mu)
|
||||
@@ -158,7 +158,7 @@ def create_ui():
|
||||
refresh_table = gr.Button('Refresh the table', elem_classes="small-button", interactive=not mu)
|
||||
|
||||
# Training events
|
||||
all_params = [lora_name, always_override, all_linear, q_proj_en, v_proj_en, k_proj_en, o_proj_en, gate_proj_en, down_proj_en, up_proj_en, save_steps, micro_batch_size, batch_size, epochs, learning_rate, lr_scheduler_type, lora_rank, lora_alpha, lora_dropout, cutoff_len, dataset, eval_dataset, format, eval_steps, text_dataset, higher_rank_limit, warmup_steps, optimizer, stride_length, train_only_after, stop_at_loss, add_eos_token, report_to]
|
||||
all_params = [lora_name, always_override, all_linear, q_proj_en, v_proj_en, k_proj_en, o_proj_en, gate_proj_en, down_proj_en, up_proj_en, save_steps, micro_batch_size, batch_size, epochs, learning_rate, lr_scheduler_type, lora_rank, lora_alpha, lora_dropout, cutoff_len, dataset, eval_dataset, format, eval_steps, text_dataset, higher_rank_limit, warmup_steps, optimizer, stride_length, stop_at_loss, add_eos_token, report_to]
|
||||
|
||||
copy_from.change(do_copy_params, [copy_from] + all_params, all_params)
|
||||
start_button.click(do_train, all_params, output)
|
||||
@@ -222,6 +222,29 @@ def clean_path(base_path: str, path: str):
|
||||
return f'{Path(base_path).absolute()}/{path}'
|
||||
|
||||
|
||||
def get_instruction_templates():
|
||||
path = Path('user_data/instruction-templates')
|
||||
names = set()
|
||||
for ext in ['yaml', 'yml', 'jinja']:
|
||||
for f in path.glob(f'*.{ext}'):
|
||||
names.add(f.stem)
|
||||
return ['None', 'Chat Template'] + sorted(names, key=utils.natural_keys)
|
||||
|
||||
|
||||
def load_template(name):
|
||||
"""Load a Jinja2 template string from user_data/instruction-templates/."""
|
||||
path = Path('user_data/instruction-templates')
|
||||
for ext in ['jinja', 'yaml', 'yml']:
|
||||
filepath = path / f'{name}.{ext}'
|
||||
if filepath.exists():
|
||||
if ext == 'jinja':
|
||||
return filepath.read_text(encoding='utf-8')
|
||||
else:
|
||||
data = yaml.safe_load(filepath.read_text(encoding='utf-8'))
|
||||
return data.get('instruction_template', '')
|
||||
return ''
|
||||
|
||||
|
||||
def backup_adapter(input_folder):
|
||||
# Get the creation date of the adapter file (safetensors or bin)
|
||||
try:
|
||||
@@ -269,7 +292,7 @@ def calc_trainable_parameters(model):
|
||||
return trainable_params, all_param
|
||||
|
||||
|
||||
def do_train(lora_name: str, always_override: bool, all_linear: bool, q_proj_en: bool, v_proj_en: bool, k_proj_en: bool, o_proj_en: bool, gate_proj_en: bool, down_proj_en: bool, up_proj_en: bool, save_steps: int, micro_batch_size: int, batch_size: int, epochs: int, learning_rate: str, lr_scheduler_type: str, lora_rank: int, lora_alpha: int, lora_dropout: float, cutoff_len: int, dataset: str, eval_dataset: str, format: str, eval_steps: int, text_dataset: str, higher_rank_limit: bool, warmup_steps: int, optimizer: str, stride_length: int, train_only_after: str, stop_at_loss: float, add_eos_token: bool, report_to: str):
|
||||
def do_train(lora_name: str, always_override: bool, all_linear: bool, q_proj_en: bool, v_proj_en: bool, k_proj_en: bool, o_proj_en: bool, gate_proj_en: bool, down_proj_en: bool, up_proj_en: bool, save_steps: int, micro_batch_size: int, batch_size: int, epochs: int, learning_rate: str, lr_scheduler_type: str, lora_rank: int, lora_alpha: int, lora_dropout: float, cutoff_len: int, dataset: str, eval_dataset: str, format: str, eval_steps: int, text_dataset: str, higher_rank_limit: bool, warmup_steps: int, optimizer: str, stride_length: int, stop_at_loss: float, add_eos_token: bool, report_to: str):
|
||||
|
||||
import torch
|
||||
import transformers
|
||||
@@ -324,47 +347,6 @@ def do_train(lora_name: str, always_override: bool, all_linear: bool, q_proj_en:
|
||||
}.items() if enabled]
|
||||
return target_mods
|
||||
|
||||
def encode(text, add_bos_token):
|
||||
result = shared.tokenizer.encode(text, truncation=True, max_length=cutoff_len)
|
||||
# Check if the first two tokens are BOS
|
||||
if len(result) >= 2 and result[:2] == [shared.tokenizer.bos_token_id, shared.tokenizer.bos_token_id]:
|
||||
result = result[1:]
|
||||
|
||||
if not add_bos_token and result[0] == shared.tokenizer.bos_token_id:
|
||||
result = result[1:]
|
||||
return result
|
||||
|
||||
def tokenize(prompt, append_eos_token=False):
|
||||
|
||||
if train_only_after == '' or train_only_after not in prompt:
|
||||
input_ids = encode(prompt, True)
|
||||
|
||||
if append_eos_token and input_ids[-1] != shared.tokenizer.eos_token_id and len(input_ids) < cutoff_len:
|
||||
input_ids.append(shared.tokenizer.eos_token_id)
|
||||
|
||||
labels = list(input_ids)
|
||||
|
||||
else:
|
||||
ind = prompt.index(train_only_after) + len(train_only_after)
|
||||
before_tokens = encode(prompt[:ind], True)
|
||||
after_tokens = encode(prompt[ind:], False)
|
||||
|
||||
if append_eos_token and len(after_tokens) > 0 and after_tokens[-1] != shared.tokenizer.eos_token_id:
|
||||
after_tokens.append(shared.tokenizer.eos_token_id)
|
||||
|
||||
full_length = len(after_tokens) + len(before_tokens)
|
||||
if full_length > cutoff_len:
|
||||
after_tokens = after_tokens[:cutoff_len - len(before_tokens)]
|
||||
|
||||
input_ids = before_tokens + after_tokens
|
||||
labels = [-100] * len(before_tokens) + list(after_tokens)
|
||||
|
||||
return {
|
||||
"input_ids": input_ids,
|
||||
"labels": labels,
|
||||
"attention_mask": [1] * len(input_ids),
|
||||
}
|
||||
|
||||
def normalize_messages(data_point):
|
||||
"""Convert a dataset row to OpenAI messages format for apply_chat_template()."""
|
||||
if "messages" in data_point:
|
||||
@@ -377,16 +359,8 @@ def do_train(lora_name: str, always_override: bool, all_linear: bool, q_proj_en:
|
||||
for turn in data_point["conversations"]
|
||||
]
|
||||
|
||||
if "instruction" in data_point and "output" in data_point:
|
||||
messages = []
|
||||
if data_point.get("system", "").strip():
|
||||
messages.append({"role": "system", "content": data_point["system"]})
|
||||
messages.append({"role": "user", "content": data_point["instruction"]})
|
||||
messages.append({"role": "assistant", "content": data_point["output"]})
|
||||
return messages
|
||||
|
||||
raise RuntimeError(
|
||||
f'Dataset row must contain "messages", "conversations", or "instruction"/"output" keys. '
|
||||
f'Dataset row must contain "messages" or "conversations" key. '
|
||||
f'Found: {list(data_point.keys())}'
|
||||
)
|
||||
|
||||
@@ -492,73 +466,46 @@ def do_train(lora_name: str, always_override: bool, all_linear: bool, q_proj_en:
|
||||
return
|
||||
|
||||
if format == 'Chat Template':
|
||||
# Use the model's built-in chat template via apply_chat_template()
|
||||
if not getattr(shared.tokenizer, 'chat_template', None):
|
||||
yield "Error: this model's tokenizer does not have a chat template. Use a format file instead, or load an instruct/chat model."
|
||||
yield "Error: this model's tokenizer does not have a chat template. Select an instruction template instead, or load an instruct/chat model."
|
||||
return
|
||||
|
||||
train_template["template_type"] = "chat_template"
|
||||
|
||||
logger.info("Loading JSON dataset with Chat Template format")
|
||||
data = load_dataset("json", data_files=clean_path('user_data/training/datasets', f'{dataset}.json'))
|
||||
|
||||
# Validate the first row
|
||||
try:
|
||||
normalize_messages(data['train'][0])
|
||||
except (RuntimeError, KeyError, IndexError) as e:
|
||||
yield f"Error: {e}"
|
||||
else:
|
||||
# Load custom instruction template and set on tokenizer
|
||||
template_str = load_template(format)
|
||||
if not template_str:
|
||||
yield f"Error: could not load instruction template '{format}'."
|
||||
return
|
||||
shared.tokenizer.chat_template = template_str
|
||||
|
||||
train_data = data['train'].map(
|
||||
# Unified path — both cases use tokenize_conversation()
|
||||
train_template["template_type"] = "chat_template"
|
||||
|
||||
logger.info("Loading JSON dataset with chat template format")
|
||||
data = load_dataset("json", data_files=clean_path('user_data/training/datasets', f'{dataset}.json'))
|
||||
|
||||
# Validate the first row
|
||||
try:
|
||||
normalize_messages(data['train'][0])
|
||||
except (RuntimeError, KeyError, IndexError) as e:
|
||||
yield f"Error: {e}"
|
||||
return
|
||||
|
||||
train_data = data['train'].map(
|
||||
tokenize_conversation,
|
||||
remove_columns=data['train'].column_names,
|
||||
new_fingerprint='%030x' % random.randrange(16**30)
|
||||
)
|
||||
|
||||
if eval_dataset == 'None':
|
||||
eval_data = None
|
||||
else:
|
||||
eval_data = load_dataset("json", data_files=clean_path('user_data/training/datasets', f'{eval_dataset}.json'))
|
||||
eval_data = eval_data['train'].map(
|
||||
tokenize_conversation,
|
||||
remove_columns=data['train'].column_names,
|
||||
remove_columns=eval_data['train'].column_names,
|
||||
new_fingerprint='%030x' % random.randrange(16**30)
|
||||
)
|
||||
|
||||
if eval_dataset == 'None':
|
||||
eval_data = None
|
||||
else:
|
||||
eval_data = load_dataset("json", data_files=clean_path('user_data/training/datasets', f'{eval_dataset}.json'))
|
||||
eval_data = eval_data['train'].map(
|
||||
tokenize_conversation,
|
||||
remove_columns=eval_data['train'].column_names,
|
||||
new_fingerprint='%030x' % random.randrange(16**30)
|
||||
)
|
||||
else:
|
||||
# Use format file for prompt generation
|
||||
train_template["template_type"] = "dataset"
|
||||
|
||||
with open(clean_path('user_data/training/formats', f'{format}.json'), 'r', encoding='utf-8-sig') as formatFile:
|
||||
format_data: dict[str, str] = json.load(formatFile)
|
||||
|
||||
# == store training prompt ==
|
||||
for _, value in format_data.items():
|
||||
prompt_key = f"template_{len(train_template)}"
|
||||
train_template[prompt_key] = value
|
||||
|
||||
def generate_prompt(data_point: dict[str, str]):
|
||||
for options, data in format_data.items():
|
||||
if set(options.split(',')) == set(x[0] for x in data_point.items() if (type(x[1]) is str and len(x[1].strip()) > 0)):
|
||||
for key, val in data_point.items():
|
||||
if type(val) is str:
|
||||
data = data.replace(f'%{key}%', val)
|
||||
return data
|
||||
raise RuntimeError(f'Data-point "{data_point}" has no keyset match within format "{list(format_data.keys())}"')
|
||||
|
||||
def generate_and_tokenize_prompt(data_point):
|
||||
prompt = generate_prompt(data_point)
|
||||
return tokenize(prompt, add_eos_token)
|
||||
|
||||
logger.info("Loading JSON datasets")
|
||||
data = load_dataset("json", data_files=clean_path('user_data/training/datasets', f'{dataset}.json'))
|
||||
train_data = data['train'].map(generate_and_tokenize_prompt, new_fingerprint='%030x' % random.randrange(16**30))
|
||||
|
||||
if eval_dataset == 'None':
|
||||
eval_data = None
|
||||
else:
|
||||
eval_data = load_dataset("json", data_files=clean_path('user_data/training/datasets', f'{eval_dataset}.json'))
|
||||
eval_data = eval_data['train'].map(generate_and_tokenize_prompt, new_fingerprint='%030x' % random.randrange(16**30))
|
||||
|
||||
# == We MUST reload model if it went through any previous training, even failed one ==
|
||||
if shared.model_dirty_from_training:
|
||||
selected_model = shared.model_name
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
[
|
||||
{"messages": [{"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "The capital of France is Paris."}]},
|
||||
{"messages": [{"role": "user", "content": "What is 2+2?"}, {"role": "assistant", "content": "2+2 equals 4."}]},
|
||||
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Name a color."}, {"role": "assistant", "content": "Blue is a color."}, {"role": "user", "content": "Another one?"}, {"role": "assistant", "content": "Red is also a color."}]},
|
||||
{"messages": [{"role": "user", "content": "What is the sun?"}, {"role": "assistant", "content": "The sun is a star at the center of our solar system."}]},
|
||||
{"messages": [{"role": "user", "content": "Say hello."}, {"role": "assistant", "content": "Hello! How can I help you today?"}]},
|
||||
{"messages": [{"role": "user", "content": "What is water?"}, {"role": "assistant", "content": "Water is a chemical compound with the formula H2O."}]},
|
||||
{"messages": [{"role": "user", "content": "Name a fruit."}, {"role": "assistant", "content": "An apple is a fruit."}]},
|
||||
{"messages": [{"role": "user", "content": "What is gravity?"}, {"role": "assistant", "content": "Gravity is a fundamental force that attracts objects with mass toward each other."}]}
|
||||
]
|
||||
@@ -0,0 +1,10 @@
|
||||
[
|
||||
{"conversations": [{"from": "human", "value": "What is the capital of France?"}, {"from": "gpt", "value": "The capital of France is Paris."}]},
|
||||
{"conversations": [{"from": "human", "value": "What is 2+2?"}, {"from": "gpt", "value": "2+2 equals 4."}]},
|
||||
{"conversations": [{"from": "system", "value": "You are a helpful assistant."}, {"from": "human", "value": "Name a color."}, {"from": "gpt", "value": "Blue is a color."}, {"from": "human", "value": "Another one?"}, {"from": "gpt", "value": "Red is also a color."}]},
|
||||
{"conversations": [{"from": "human", "value": "What is the sun?"}, {"from": "gpt", "value": "The sun is a star at the center of our solar system."}]},
|
||||
{"conversations": [{"from": "human", "value": "Say hello."}, {"from": "gpt", "value": "Hello! How can I help you today?"}]},
|
||||
{"conversations": [{"from": "human", "value": "What is water?"}, {"from": "gpt", "value": "Water is a chemical compound with the formula H2O."}]},
|
||||
{"conversations": [{"from": "human", "value": "Name a fruit."}, {"from": "gpt", "value": "An apple is a fruit."}]},
|
||||
{"conversations": [{"from": "human", "value": "What is gravity?"}, {"from": "gpt", "value": "Gravity is a fundamental force that attracts objects with mass toward each other."}]}
|
||||
]
|
||||
@@ -0,0 +1,10 @@
|
||||
[
|
||||
{"text": "The quick brown fox jumps over the lazy dog. This is a simple sentence used for testing purposes. It contains all the letters of the English alphabet."},
|
||||
{"text": "Machine learning is a subset of artificial intelligence that focuses on building systems that learn from data. These systems improve their performance over time without being explicitly programmed."},
|
||||
{"text": "Python is a high-level programming language known for its simplicity and readability. It was created by Guido van Rossum and first released in 1991."},
|
||||
{"text": "The Earth orbits the Sun at an average distance of about 93 million miles. It takes approximately 365.25 days to complete one orbit, which is why we have leap years."},
|
||||
{"text": "Neural networks are computing systems inspired by biological neural networks in the brain. They consist of layers of interconnected nodes that process information using connectionist approaches."},
|
||||
{"text": "Water covers about 71 percent of the Earth's surface. The oceans hold about 96.5 percent of all Earth's water. Only about 2.5 percent of the Earth's water is freshwater."},
|
||||
{"text": "The history of computing dates back to ancient times with devices like the abacus. Modern electronic computing began in the mid-20th century with the development of vacuum tube computers."},
|
||||
{"text": "Photosynthesis is the process by which green plants and some other organisms use sunlight to synthesize nutrients from carbon dioxide and water. It generates oxygen as a byproduct."}
|
||||
]
|
||||
@@ -1 +1 @@
|
||||
to load multiple raw text files create a subdirectory and put them all there
|
||||
Put your training dataset JSON files here.
|
||||
|
||||
@@ -1,4 +0,0 @@
|
||||
{
|
||||
"instruction,output": "<|im_start|>system\n<|im_end|>\n<|im_start|>user\n%instruction%<|im_end|>\n<|im_start|>assistant\n%output%<|im_end|>",
|
||||
"instruction,input,output": "<|im_start|>system\n<|im_end|>\n<|im_start|>user\n%instruction%: %input%<|im_end|>\n<|im_start|>assistant\n%output%<|im_end|>"
|
||||
}
|
||||
@@ -1,4 +0,0 @@
|
||||
{
|
||||
"instruction,output": "User: %instruction%\nAssistant: %output%",
|
||||
"instruction,input,output": "User: %instruction%: %input%\nAssistant: %output%"
|
||||
}
|
||||
@@ -1,4 +0,0 @@
|
||||
{
|
||||
"instruction,output": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n%instruction%\n\n### Response:\n%output%",
|
||||
"instruction,input,output": "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\n%instruction%\n\n### Input:\n%input%\n\n### Response:\n%output%"
|
||||
}
|
||||
@@ -1,4 +0,0 @@
|
||||
{
|
||||
"modelanswer,userprompt,systemprompt": "<s>[INST] <<SYS>>\n%systemprompt%\n<</SYS>>\n\n%userprompt%[/INST] %modelanswer%</s>",
|
||||
"modelanswer,userprompt": "<s>[INST] <<SYS>>\n\n<</SYS>>\n\n%userprompt%[/INST] %modelanswer%</s>"
|
||||
}
|
||||
@@ -1,3 +0,0 @@
|
||||
{
|
||||
"instruction,output": "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\n\nUSER: %instruction%\n\nASSISTANT: %output%"
|
||||
}
|
||||
Reference in New Issue
Block a user