--- title: - Downloading & Uploading functions for Models keywords: fastai sidebar: home_sidebar nb_path: "nbs/data.loader.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %} {% raw %}
{% endraw %} {% raw %}

find_git_repo[source]

find_git_repo()

{% endraw %} {% raw %}
{% endraw %} {% raw %}

get_registry_api_key[source]

get_registry_api_key()

{% endraw %} {% raw %}
{% endraw %} {% raw %}

class upload_in_chunks[source]

upload_in_chunks(filename, chunksize=16384)

{% endraw %} {% raw %}

class IterableToFileAdapter[source]

IterableToFileAdapter(iterable)

{% endraw %} {% raw %}
{% endraw %} {% raw %}

write_file_to_package_registry[source]

write_file_to_package_registry(project_id, file_path, api_key, version='0.0.1')

{% endraw %} {% raw %}
{% endraw %} {% raw %}

project_id_from_name[source]

project_id_from_name(project_name, api_key, job_token=None)

{% raw %}

get_project_id_from_project_path_unsafe[source]

get_project_id_from_project_path_unsafe(project_path)

write_huggingface_model_to_package_registry[source]

write_huggingface_model_to_package_registry(project_name, model)

{% endraw %} {% raw %}
write_model_to_package_registry(model, project_name="test-1234")

write_model_to_package_registry[source]

write_model_to_package_registry(model, project_name=None)

100.00% [100/100 00:00<00:00]

download_package_file[source]

download_package_file(filename, project_path=None, out_dir=None, package_name='plugin-model-package', package_version='0.0.1', download_if_exists=False)

{% endraw %} {% raw %}
{% endraw %} {% raw %}

download_huggingface_model_for_project[source]

download_huggingface_model_for_project(project_path=None, files=None, download_if_exists=False)

{% endraw %} {% raw %}
{% endraw %} {% raw %}

load_huggingface_model_for_project[source]

load_huggingface_model_for_project(project_path=None, files=None, download_if_exists=False)

Succesfully uploaded /tmp/config.json
writing pytorch_model.bin to package registry of test-1234 with project id 190
uploading /tmp/pytorch_model.bin
# todo: cleanup old package files during testing
filename = "config.json"
out_file = download_package_file(filename, "memri/finetuning-example")
100.00% [100/100 04:23<00:02]
/Users/koen/.memri/projects/finetuning-example/config.json
/Users/koen/.memri/projects/finetuning-example/config.json already exists, and `download_if_exists`==False, using cached version
{% endraw %} {% raw %}
model = load_huggingface_model_for_project(project_path="memri/finetuning-example")
Some weights of the model checkpoint at distilroberta-base were not used when initializing RobertaForSequenceClassification: ['lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.bias', 'roberta.pooler.dense.bias', 'lm_head.dense.weight', 'roberta.pooler.dense.weight']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at distilroberta-base and are newly initialized: ['classifier.dense.weight', 'classifier.out_proj.bias', 'classifier.dense.bias', 'classifier.out_proj.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
{% endraw %} {% raw %}
write_model_to_package_registry(model, project_name="test-1234")
writing config.json to package registry of test-1234 with project id 190
uploading /tmp/config.json
100.00% [100/100 00:00<00:00]

Succesfully uploaded /tmp/config.json
writing pytorch_model.bin to package registry of test-1234 with project id 190
uploading /tmp/pytorch_model.bin
100.00% [100/100 04:24<00:02]

Succesfully uploaded /tmp/pytorch_model.bin
{% endraw %}
model = load_huggingface_model_for_project(project_path="memri/finetuning-example")
/Users/koen/.memri/projects/finetuning-example/config.json
/Users/koen/.memri/projects/finetuning-example/config.json already exists, and `download_if_exists`==False, using cached version
/Users/koen/.memri/projects/finetuning-example/pytorch_model.bin
/Users/koen/.memri/projects/finetuning-example/pytorch_model.bin already exists, and `download_if_exists`==False, using cached version
{% endraw %} {% raw %}
out_dir = download_huggingface_model_for_project(project_path="memri/finetuning-example")
model = AutoModelForSequenceClassification.from_pretrained(out_dir, num_labels=20)
/Users/koen/.memri/projects/finetuning-example/config.json
/Users/koen/.memri/projects/finetuning-example/config.json already exists, and `download_if_exists`==False, using cached version
/Users/koen/.memri/projects/finetuning-example/pytorch_model.bin
/Users/koen/.memri/projects/finetuning-example/pytorch_model.bin already exists, and `download_if_exists`==False, using cached version
{% endraw %}