Skip to content

llmcompressor.core.state

Module for managing LLM Compressor state.

Provides classes for holding and updating the state information related to data, hardware, and model compression.

Classes:

  • Data

    A dataclass to hold different data sets for training, validation,

  • Hardware

    A dataclass to hold information about the hardware being used.

  • ModifiedState

    A dataclass to represent a modified model, optimizer, and loss.

  • State

    State class holds information about the current compression state.

Data dataclass

Data(
    train: Optional[Any] = None,
    val: Optional[Any] = None,
    test: Optional[Any] = None,
    calib: Optional[Any] = None,
)

A dataclass to hold different data sets for training, validation, testing, and/or calibration. Each data set is a ModifiableData instance.

Parameters:

  • train

    (Optional[Any], default: None ) –

    The training data set

  • val

    (Optional[Any], default: None ) –

    The validation data set

  • test

    (Optional[Any], default: None ) –

    The testing data set

  • calib

    (Optional[Any], default: None ) –

    The calibration data set

Hardware dataclass

Hardware(
    device: Optional[str] = None,
    devices: Optional[List[str]] = None,
    rank: Optional[int] = None,
    world_size: Optional[int] = None,
    local_rank: Optional[int] = None,
    local_world_size: Optional[int] = None,
    distributed: Optional[bool] = None,
    distributed_strategy: Optional[str] = None,
)

A dataclass to hold information about the hardware being used.

Parameters:

  • device

    (Optional[str], default: None ) –

    The current device being used for training

  • devices

    (Optional[List[str]], default: None ) –

    List of all devices to be used for training

  • rank

    (Optional[int], default: None ) –

    The rank of the current device

  • world_size

    (Optional[int], default: None ) –

    The total number of devices being used

  • local_rank

    (Optional[int], default: None ) –

    The local rank of the current device

  • local_world_size

    (Optional[int], default: None ) –

    The total number of devices being used on the local machine

  • distributed

    (Optional[bool], default: None ) –

    Whether or not distributed training is being used

  • distributed_strategy

    (Optional[str], default: None ) –

    The distributed strategy being used

ModifiedState dataclass

ModifiedState(model, optimizer, loss, modifier_data)

A dataclass to represent a modified model, optimizer, and loss.

Parameters:

  • model

    (Optional[Any]) –

    The modified model

  • optimizer

    (Optional[Any]) –

    The modified optimizer

  • loss

    (Optional[Any]) –

    The modified loss

  • modifier_data

    (Optional[List[Dict[str, Any]]]) –

    The modifier data used to modify the model, optimizer, and loss

Initialize the ModifiedState with the given parameters.

Parameters:

  • model

    (Any) –

    The modified model

  • optimizer

    (Any) –

    The modified optimizer

  • loss

    (Any) –

    The modified loss

  • modifier_data

    (List[Dict[str, Any]]) –

    The modifier data used to modify the model, optimizer, and loss

Source code in llmcompressor/core/state.py
def __init__(self, model, optimizer, loss, modifier_data):
    """
    Initialize the ModifiedState with the given parameters.

    :param model: The modified model
    :type model: Any
    :param optimizer: The modified optimizer
    :type optimizer: Any
    :param loss: The modified loss
    :type loss: Any
    :param modifier_data: The modifier data used to modify the model, optimizer,
        and loss
    :type modifier_data: List[Dict[str, Any]]
    """
    self.model = model
    self.optimizer = optimizer
    self.loss = loss
    self.modifier_data = modifier_data

State dataclass

State(
    model: Any = None,
    teacher_model: Any = None,
    optimizer: Any = None,
    optim_wrapped: bool = None,
    loss: Any = None,
    batch_data: Any = None,
    data: Data = Data(),
    hardware: Hardware = Hardware(),
    loggers: Optional[LoggerManager] = None,
    model_log_cadence: Optional[float] = None,
    _last_log_step: Union[float, int, None] = None,
)

State class holds information about the current compression state.

Parameters:

  • model

    (Any, default: None ) –

    The model being used for compression

  • teacher_model

    (Any, default: None ) –

    The teacher model being used for compression

  • optimizer

    (Any, default: None ) –

    The optimizer being used for training

  • optim_wrapped

    (bool, default: None ) –

    Whether or not the optimizer has been wrapped

  • loss

    (Any, default: None ) –

    The loss function being used for training

  • batch_data

    (Any, default: None ) –

    The current batch of data being used for compression

  • data

    (Data, default: Data() ) –

    The data sets being used for training, validation, testing, and/or calibration, wrapped in a Data instance

  • hardware

    (Hardware, default: Hardware() ) –

    Hardware instance holding info about the target hardware being used

  • loggers

    (Optional[LoggerManager], default: None ) –

    LoggerManager instance holding all the loggers to log

  • model_log_cadence

    (Optional[float], default: None ) –

    The cadence to log model information w.r.t epochs. If 1, logs every epoch. If 2, logs every other epoch, etc. Default is 1.

Methods:

  • update

    Update the state with the given parameters.

Attributes:

  • compression_ready (bool) –

    Check if the model and optimizer are set for compression.

compression_ready property

compression_ready: bool

Check if the model and optimizer are set for compression.

Returns:

  • bool

    True if model and optimizer are set, False otherwise

update

update(
    model: Any = None,
    teacher_model: Any = None,
    optimizer: Any = None,
    attach_optim_callbacks: bool = True,
    train_data: Any = None,
    val_data: Any = None,
    test_data: Any = None,
    calib_data: Any = None,
    copy_data: bool = True,
    start: float = None,
    steps_per_epoch: int = None,
    batches_per_step: int = None,
    loggers: Union[
        None, LoggerManager, List[BaseLogger]
    ] = None,
    model_log_cadence: Optional[float] = None,
    **kwargs
) -> Dict

Update the state with the given parameters.

Parameters:

  • model

    (Any, default: None ) –

    The model to update the state with

  • teacher_model

    (Any, default: None ) –

    The teacher model to update the state with

  • optimizer

    (Any, default: None ) –

    The optimizer to update the state with

  • attach_optim_callbacks

    (bool, default: True ) –

    Whether or not to attach optimizer callbacks

  • train_data

    (Any, default: None ) –

    The training data to update the state with

  • val_data

    (Any, default: None ) –

    The validation data to update the state with

  • test_data

    (Any, default: None ) –

    The testing data to update the state with

  • calib_data

    (Any, default: None ) –

    The calibration data to update the state with

  • copy_data

    (bool, default: True ) –

    Whether or not to copy the data

  • start

    (float, default: None ) –

    The start index to update the state with

  • steps_per_epoch

    (int, default: None ) –

    The steps per epoch to update the state with

  • batches_per_step

    (int, default: None ) –

    The batches per step to update the state with

  • loggers

    (Union[None, LoggerManager, List[BaseLogger]], default: None ) –

    The metrics manager to setup logging important info and milestones to, also accepts a list of BaseLogger(s)

  • model_log_cadence

    (Optional[float], default: None ) –

    The cadence to log model information w.r.t epochs. If 1, logs every epoch. If 2, logs every other epoch, etc. Default is 1.

  • kwargs

    Additional keyword arguments to update the state with

Returns:

  • Dict

    The updated state as a dictionary

Source code in llmcompressor/core/state.py
def update(
    self,
    model: Any = None,
    teacher_model: Any = None,
    optimizer: Any = None,
    attach_optim_callbacks: bool = True,
    train_data: Any = None,
    val_data: Any = None,
    test_data: Any = None,
    calib_data: Any = None,
    copy_data: bool = True,
    start: float = None,
    steps_per_epoch: int = None,
    batches_per_step: int = None,
    loggers: Union[None, LoggerManager, List[BaseLogger]] = None,
    model_log_cadence: Optional[float] = None,
    **kwargs,
) -> Dict:
    """
    Update the state with the given parameters.

    :param model: The model to update the state with
    :type model: Any
    :param teacher_model: The teacher model to update the state with
    :type teacher_model: Any
    :param optimizer: The optimizer to update the state with
    :type optimizer: Any
    :param attach_optim_callbacks: Whether or not to attach optimizer callbacks
    :type attach_optim_callbacks: bool
    :param train_data: The training data to update the state with
    :type train_data: Any
    :param val_data: The validation data to update the state with
    :type val_data: Any
    :param test_data: The testing data to update the state with
    :type test_data: Any
    :param calib_data: The calibration data to update the state with
    :type calib_data: Any
    :param copy_data: Whether or not to copy the data
    :type copy_data: bool
    :param start: The start index to update the state with
    :type start: float
    :param steps_per_epoch: The steps per epoch to update the state with
    :type steps_per_epoch: int
    :param batches_per_step: The batches per step to update the state with
    :type batches_per_step: int
    :param loggers: The metrics manager to setup logging important info and
        milestones to, also accepts a list of BaseLogger(s)
    :type loggers: Union[None, LoggerManager, List[BaseLogger]]
    :param model_log_cadence: The cadence to log model information w.r.t epochs.
        If 1, logs every epoch. If 2, logs every other epoch, etc. Default is 1.
    :type model_log_cadence: Optional[float]
    :param kwargs: Additional keyword arguments to update the state with
    :return: The updated state as a dictionary
    :rtype: Dict
    """
    logger.debug(
        "Updating state with provided parameters: {}",
        {
            "model": model,
            "teacher_model": teacher_model,
            "optimizer": optimizer,
            "attach_optim_callbacks": attach_optim_callbacks,
            "train_data": train_data,
            "val_data": val_data,
            "test_data": test_data,
            "calib_data": calib_data,
            "copy_data": copy_data,
            "start": start,
            "steps_per_epoch": steps_per_epoch,
            "batches_per_step": batches_per_step,
            "loggers": loggers,
            "model_log_cadence": model_log_cadence,
            "kwargs": kwargs,
        },
    )

    if model is not None:
        self.model = model
    if teacher_model is not None:
        self.teacher_model = teacher_model
    if optimizer is not None:
        self.optim_wrapped = attach_optim_callbacks
        self.optimizer = optimizer
    if train_data is not None:
        self.data.train = train_data if not copy_data else deepcopy(train_data)
    if val_data is not None:
        self.data.val = val_data if not copy_data else deepcopy(val_data)
    if test_data is not None:
        self.data.test = test_data if not copy_data else deepcopy(test_data)
    if calib_data is not None:
        self.data.calib = calib_data if not copy_data else deepcopy(calib_data)

    if "device" in kwargs:
        self.hardware.device = kwargs["device"]

    loggers = loggers or []
    if isinstance(loggers, list):
        loggers = LoggerManager(loggers)
    self.loggers = loggers

    if model_log_cadence is not None:
        self.model_log_cadence = model_log_cadence

    return kwargs