HyFI Template Example

HyFI Template Example#

This Jupyter Notebook demonstrates the usage of the hyfit package, including initializing a workspace, mounting Google Drive on Colab, and using HyFI to manage configurations.

First, let’s import the necessary functions and classes from the hyfit package.

from hyfit import get_version, HyFI

/home/yj.lee/.venvs/hyfi-template/lib/python3.8/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Check Version#

Now, let’s get the version of the hyfi_template package.

version = get_version()
print("HyFI Template version:", version)

HyFI Template version: 0.6.4

Initialize Project#

We’ll initialize the project using the HyFI.initialize function. The function takes the following parameters:

  • project_name: Name of the project to use.

  • project_description: Description of the project that will be used.

  • project_root: Root directory of the project.

  • project_workspace_name: Name of the project’s workspace directory.

  • global_hyfi_root: Root directory of the global hyfi.

  • global_workspace_name: Name of the global hierachical workspace directory.

  • num_workers: Number of workers to run.

  • logging_level: Log level for the log.

  • autotime: Whether to automatically set time and / or keep track of run times.

  • retina: Whether to use retina or not.

  • verbose: Enables or disables logging

We’ll check if we’re running in Google Colab, and if so, we’ll mount Google Drive.

if HyFI.is_colab():
    HyFI.mount_google_drive()

h = HyFI.initialize(
    project_name="hyfi",
    logging_level="INFO",
    verbose=True,
)

print("project_dir:", h.project.root_dir)
print("project_workspace_dir:", h.project.workspace_dir)

INFO:hyfi.utils.notebooks:Google Colab not detected.
INFO:hyfi.utils.notebooks:Extension autotime not found. Install it first.
INFO:hyfi.joblib.joblib:initialized batcher with <hyfi.joblib.batch.batcher.Batcher object at 0x7fc72dd2b0a0>
INFO:hyfi.main.config:HyFi project [hyfi] initialized

project_dir: /home/yj.lee/.hyfi/projects/hyfi
project_workspace_dir: /home/yj.lee/.hyfi/projects/hyfi/workspace

Compose Configuration#

We can use the HyFI.compose function to load a configuration file. In this example, we’ll use the default configuration by specifying path=__default__.

cfg = HyFI.compose("project")

Display Configuration#

Now, let’s print the loaded configuration using the HyFI.print function.

HyFI.print(cfg)

{'_config_group_': '/project',
 '_config_name_': '__init__',
 'dotenv': {'CACHED_PATH_CACHE_ROOT': None

,
            'CUDA_DEVICE_ORDER': None,
            'CUDA_VISIBLE_DEVICES': None,
            'DOTENV_DIR': '/mnt/nvme1n1p2/home/yj.lee/workspace/projects/hyfi-template/book',
            'DOTENV_FILE': '/mnt/nvme1n1p2/home/yj.lee/workspace/projects/hyfi-template/book/.env',
            'DOTENV_FILENAME': '.env',
            'ECOS_API_KEY': None,
            'FRED_API_KEY': None,
            'HF_USER_ACCESS_TOKEN': None,
            'HUGGING_FACE_HUB_TOKEN': None,
            'HYFI_GLOBAL_ROOT': None,
            'HYFI_GLOBAL_WORKSPACE_NAME': None,
            'HYFI_LOG_LEVEL': 'WARNING',
            'HYFI_NUM_WORKERS': None,
            'HYFI_PROJECT_DESC': None,
            'HYFI_PROJECT_NAME': None,
            'HYFI_PROJECT_ROOT': None,
            'HYFI_PROJECT_WORKSPACE_NAME': None,
            'HYFI_RESOURCE_DIR': None,
            'HYFI_SECRETS_DIR': '/mnt/nvme1n1p2/home/yj.lee/workspace/projects/hyfi-template/book/secrets',
            'HYFI_VERBOSE': False,
            'KMP_DUPLICATE_LIB_OK': None,
            'LABEL_STUDIO_SERVER': None,
            'NASDAQ_API_KEY': None,
            'OPENAI_API_KEY': None,
            'TOKENIZERS_PARALLELISM': None,
            'WANDB_API_KEY': None,
            'WANDB_DIR': None,
            'WANDB_DISABLED': None,
            'WANDB_NOTEBOOK_NAME': None,
            'WANDB_PROJECT': None,
            'WANDB_SILENT': None},
 'global_hyfi_root': '/home/yj.lee',
 'global_workspace_name': '.hyfi',
 'joblib': {'backend': 'joblib',
            'initialize_backend': True,
            'minibatch_size': 1000,
            'num_gpus': 0,
            'num_workers': '1',
            'verbose': False},
 'num_workers': '1',
 'path': {'_config_group_': '/path',
          '_config_name_': '__project__',
          'dirnames': {'archives': 'archives',
                       'cache': '.cache',
                       'config_json': 'config.json',
                       'config_yaml': 'config.yaml',
                       'configs': 'configs',
                       'datasets': 'datasets',
                       'inputs': 'inputs',
                       'library': 'libs',
                       'logs': 'logs',
                       'models': 'models',
                       'modules': 'modules',
                       'outputs': 'outputs',
                       'tmp': 'tmp'},
          'global_hyfi_root': '/home/yj.lee',
          'global_workspace_name': '.hyfi',
          'home': '/home/yj.lee',
          'hyfi': '/home/yj.lee/.venvs/hyfi-template/lib/python3.8/site-packages/hyfi',
          'package': '/mnt/nvme1n1p2/home/yj.lee/workspace/projects/hyfi-template/src/hyfit',
          'project_name': 'hyfi',
          'project_root': '/home/yj.lee/.hyfi/projects/hyfi',
          'project_workspace_name': 'workspace',
          'resources': '/home/yj.lee/.venvs/hyfi-template/lib/python3.8/site-packages/hyfi/resources',
          'runtime': '/mnt/nvme1n1p2/home/yj.lee/workspace/projects/hyfi-template/book'},
 'project_description': '',
 'project_name': 'hyfi',
 'project_root': '/home/yj.lee/.hyfi/projects/hyfi',
 'project_workspace_name': 'workspace',
 'use_huggingface_hub': False,
 'use_wandb': False,
 'verbose': 'True'}

That’s it! This example demonstrated the basic usage of the hyfit package. You can now use this package to manage your own projects and tasks in a structured manner.

Contents