Nvflare Convert Huggingface

NVFLARE Convert Hugging Face

Use When

Use when converting training code built around transformers.Trainer,

Seq2SeqTrainer, TRL SFTTrainer, or another Trainer subclass. Support

full-model and PEFT/LoRA fine-tuning, datasets/tokenizers, Trainer callbacks and

metrics, checkpoint continuity, and replicated torch.distributed training.

Do Not Use When

Do not use for an AutoModel driven by a manual PyTorch loop without a

Hugging Face Trainer (route to nvflare-convert-pytorch), PyTorch Lightning

(route to nvflare-convert-lightning, including Lightning modules that contain

Transformers models), inference-only pipelines, model serving, failed jobs

(route to nvflare-diagnose-job), or federated statistics without training

(route to nvflare-fed-stats). Route a project with active Lightning and

Hugging Face Trainer entrypoints to nvflare-orient to select one training-loop

owner or separate jobs. Route unresolved Trainer ownership, such as a Trainer

factory without a bound owner call, to nvflare-orient; do not patch either Trainer.

Out of scope: DeepSpeed, FSDP, production/POC deployment, controller rewrites,

experiment search, and privacy-protection requests such as HE, encrypted

aggregation, differential privacy, or privacy filters; never substitute an

unprotected recipe or present a disclaimer as implementation.

If a request combines federated statistics and model-training conversion, treat it as two independent jobs and

workflows: do not merge or automatically chain them, do not route the combination to nvflare-orient, and ask which

workflow to run first before generating or running either job. Recommend nvflare-fed-stats first only when the

user's purpose is to understand data distribution; handle conversion later as a separate request.

Workflow

• Load ../nvflare-shared/references/conversion-common.md and apply it for the

whole conversion; this SKILL.md states only the framework-specific deltas.

Load ../nvflare-shared/references/conversion-workflow.md only for a non-standard

case that needs its detailed rerun, data-location, authorization, or

missing-semantics guidance.

• Inspect before editing with nvflare agent inspect source <path> --format json

plus direct source reading. Load references/huggingface-detection.md during

this phase. If inspect recommends nvflare-orient for unresolved Trainer

ownership or active Lightning/Hugging Face owners, stop before editing.

Extract the entrypoint, Trainer subclass, model constructor, tokenizer or

processor, datasets and collator, Trainer arguments, compute_metrics,

callbacks, checkpoint and PEFT settings, precision, local budget,

distributed launcher, site/round counts, data location, and aggregation

intent. Do not import or execute user training modules to discover them.

• Apply the dependency-install ordering rule in ../nvflare-shared/references/conversion-common.md before

any Python command imports user, framework, NVFLARE, or declared dependency

modules.

• Select the recipe from FL intent. For explicit FedAvg, run `nvflare recipe

show fedavg-pt --format json`, then immediately load

../nvflare-shared/references/pytorch-family-recipe-construction.md and use

the returned module, class, and parameters with the required construction

and execution shape in assets/job.py. Import FedAvgRecipe from

nvflare.app_opt.pt.recipes.fedavg, never from nvflare.recipe. Treat

class_path as the public recipe key and path as its normalized exported

representation; do not inspect Recipe source or signatures to reconcile

them. Do not guess adjacent symbols or add per-site recipe config unless

sites genuinely differ. Load

../nvflare-shared/references/pytorch-family-recipe-selection.md only for

ambiguous, evaluation-only, or non-FedAvg requests.

• Convert with references/huggingface-conversion.md and adapt

assets/client_with_eval.py rather than drafting a new round loop. Preserve

model, tokenizer/processor, datasets, collator, Trainer arguments,

callbacks, and metrics. Partition site data per the "Site Data Partitioning"

rule in ../nvflare-shared/references/conversion-common.md. Import the Client API as

import nvflare.client.hf as flare, so flare.init(), flare.patch(), and

flare.is_running() resolve to nvflare.client.hf. Keep

flare.patch(trainer) simple with inferred params_scope="auto" and encode

one per-round budget in

Trainer arguments: requested steps use max_steps, requested epochs use

num_train_epochs, and a silent prompt uses the reported default

max_steps=10 unless source-budget preservation was requested. Do not

duplicate the budget in patch local_steps/local_epochs. When the client

uses HfArgumentParser, construct it with allow_abbrev=False.

• Adapt assets/server_model.py and assets/job.py instead of inventing

server-model, packaging, export, or SimEnv wiring. Keep generated and

packaged project-local modules in the same writable source directory. Never

use .. in train_script, add_server_file(), or add_client_file(); use

an existing resolved absolute path when co-location is impossible. Keep the

server and Trainer model factory and exchange keyspace identical, with

explicit model config rather than a live model. Apply only options confirmed

by the construction reference. Preserve the job asset's recipe-before-parser

ordering, ArgumentParser(allow_abbrev=False), and strict parse_args(); do

not use parse_known_args().

• Only after generated files exist, load

../nvflare-shared/references/validation-evidence.md, then

references/huggingface-validation.md. Follow the shared compile,

construction, export, package-inspection, simulation, and terminal-evidence

ladder; apply only the standard Trainer checks from the HF reference. Stop

at the first failed rung. Review and exercise the maintained assets directly;

do not inspect NVFLARE implementation source, improvise Recipe API probes, or

write one-off AST programs to re-prove them. Use

references/huggingface-state-and-distributed.md

only when inspection found PEFT, DDP, checkpoint/restore overrides,

auxiliary trainable models, or another non-default patch setting.

• Report the recipe, source facts, parameter scope, data partition, changed

files, validation status, and exact artifact paths. When validation produces

metrics, load ../nvflare-shared/references/metrics-and-artifact-reporting.md

before the final response and report each observed primary scalar with its

metric name, numeric value, and artifact or bounded-log source.

Requirements

• Must use flare.patch(trainer) as the sole model-exchange owner. receive()

inside a patched loop may inspect task metadata only; it must not load a

second copy of the global model.

• Must make the client entry's global rank argument required and pass it to

flare.init(rank=rank); never default every process to rank zero. Resolve it

from an initialized process group or global RANK, using explicit zero only

for a verified single-process launch. Client API initialization order

otherwise follows ../nvflare-shared/references/conversion-common.md.

• Must preserve source evaluation. When per-round global-model evaluation is

required, call trainer.evaluate() before trainer.train() on every rank.

Do not invent compute_metrics, label mappings, averaging denominators, or

metric direction.

• Must follow the Best-Model Metric policy in

../nvflare-shared/references/pytorch-family-recipe-construction.md; the

Hugging Face delta is only how the delivered key is named and produced. Must

preserve source metric names when practical: if the generated

trainer.evaluate() emits accuracy, set key_metric="accuracy"; if Trainer

emits a prefixed key such as eval_accuracy, set the server to that exact key

and report the source-to-server mapping. When best-model selection is

requested, every lower-is-better metric, including Trainer-generated

eval_loss, is delivered as an explicitly negated companion and selected by

that key — never as raw loss. When selection is not requested, use

key_metric=""; do not omit it and accidentally activate the recipe default.

• Must preserve PEFT configuration exactly and verify adapter key compatibility

between the server model and patched Trainer. Do not infer LoRA target

modules, silently switch adapter/full-model scope, or solve key mismatches

with non-strict loading.

• Must verify that trainer.model owns all federated trainable state for

Trainer subclasses with reference, reward, value-head, or other auxiliary

models. Ask or fail closed when params_scope="auto" would omit trainable

state required by the algorithm.

• Must preserve model constructor values needed on both server and clients per

../nvflare-shared/references/pytorch-model-exchange.md (State-Dict

Compatibility). Ask one semantic question or fail closed when required values

are not statically available.

• Must patch only one Trainer per Python process. Preserve a single Trainer

lifecycle across rounds when restore_state=True.

• Must use a positive TrainingArguments.max_steps budget for a length-less

iterable training dataset and let flare.patch(trainer) infer it.

• Must reject or report DeepSpeed, FSDP, save_only_model=True with

restore_state=True, load_best_model_at_end=True, prebuilt

optimizer/scheduler instances with restore_state=False, and checkpoint paths

not visible to every distributed rank. Do not rewrite these settings silently.

launch_once is a framework-neutral recipe parameter owned by

../nvflare-shared/references/pytorch-family-recipe-construction.md; the

Hugging Face delta is only that the product rejects explicit

launch_once=False together with restore_state=True.

• Must initialize torch.distributed before patching when rank environment

variables declare multiple ranks. All ranks must call patched Trainer methods

in identical order.

• Must not set trust_remote_code=True, download model/data artifacts unless

requested, or recover from an offline/cache-only miss by going online. Cache

misses, offline errors, remote identifiers, and validation requests do not

authorize online retries. This narrows the authorization rules in

../nvflare-shared/references/conversion-common.md.

• Site partitioning, custom aggregation, the Source Of Truth Boundary, and user

input/authorization follow ../nvflare-shared/references/conversion-common.md.

Always read this converter SKILL.md together with

../nvflare-shared/references/conversion-common.md. Complete each workflow

phase before loading the next phase's reference. Do not preload validation,

state/DDP, broad workflow, dependency, or reporting references. The standard

FedAvg path loads, in order:

../nvflare-shared/references/conversion-common.md,

references/huggingface-detection.md,

../nvflare-shared/references/pytorch-family-recipe-construction.md,

references/huggingface-conversion.md,

../nvflare-shared/references/pytorch-model-exchange.md,

../nvflare-shared/references/validation-evidence.md, and

references/huggingface-validation.md. Load

references/huggingface-state-and-distributed.md and other shared references

only under the triggers above. Do not depend on repository examples.