Amazon SageMaker - North Mini Code

Use this path if you want a managed AWS endpoint with the Hugging Face SageMaker SDK.

The Hugging Face deploy snippet currently warns that it is not yet compatible with sagemaker >= 3.0.0. If you follow this exact example, pin sagemaker<3.0.0.

Install a compatible SDK

pip install "sagemaker<3.0.0"

Deploy the model

import boto3
import sagemaker
from sagemaker.huggingface import HuggingFaceModel

try:
    role = sagemaker.get_execution_role()
except ValueError:
    iam = boto3.client("iam")
    role = iam.get_role(RoleName="sagemaker_execution_role")["Role"]["Arn"]

hub = {
    "HF_MODEL_ID": "CohereLabs/North-Mini-Code-1.0",
    "HF_TASK": "text-generation",
}

huggingface_model = HuggingFaceModel(
    transformers_version="4.51.3",
    pytorch_version="2.6.0",
    py_version="py312",
    env=hub,
    role=role,
)

predictor = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type="ml.m5.xlarge",
)

response = predictor.predict(
    {
        "inputs": "Write a Python function that removes duplicates from a list while preserving order."
    }
)

print(response)

Right-size the instance

Start with a development-sized instance for bring-up, then size for real traffic and model memory after benchmarking.

HF SageMaker inference docs

Official Hugging Face guide for deploying models on SageMaker.

HF SageMaker training docs

Companion docs for SageMaker-based training and packaging workflows.

Instance sizing guide

AWS guidance on choosing instance types for large model inference.

HF notebook examples

Example notebooks for SageMaker plus Hugging Face workflows.

Use SageMaker when you want AWS-managed endpoints and IAM-centric operations. If your target hardware is Inferentia2 or Trainium, also review /deployment/aws-neuron.