diff --git a/src/content/docs/aws/enterprise/kubernetes/configuration.md b/src/content/docs/aws/enterprise/kubernetes/configuration.md index 8ff330e0..df509e80 100644 --- a/src/content/docs/aws/enterprise/kubernetes/configuration.md +++ b/src/content/docs/aws/enterprise/kubernetes/configuration.md @@ -3,7 +3,7 @@ title: Configuration description: Kubernetes configuration reference for LocalStack running on Kubernetes template: doc sidebar: - order: 6 + order: 7 tags: ["Enterprise"] --- @@ -28,6 +28,18 @@ LOCALSTACK_K8S_LABELS=env=dev,team=platform LOCALSTACK_K8S_ANNOTATIONS=prometheus.io/scrape=true,prometheus.io/port=8080 ``` +### Pod configuration + +`LOCALSTACK_K8S_POD_CONFIG` configures Kubernetes metadata, scheduling, and resource settings for child pods created by supported services such as Lambda and ECS. +Use it to define reusable pod profiles with fields such as `nodeSelector`, `tolerations`, `affinity`, `resources`, `labels`, and `annotations`. +The value must be valid JSON. + +```bash +LOCALSTACK_K8S_POD_CONFIG='{"profiles":{"default":{"nodeSelector":{"pool":"general"}}}}' +``` + +For the full JSON schema, profile resolution order, and examples, see [Pod Configuration](/aws/enterprise/kubernetes/pod-configuration/). + ### Container security context `K8S_CONTAINER_SECURITY_CONTEXT` sets the [container security context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) applied to child pods created by LocalStack. The value should be a JSON object matching the Kubernetes `SecurityContext` spec. @@ -79,6 +91,7 @@ Increase these values if your cluster is under heavy load or if image pulls are | `LOCALSTACK_K8S_NAMESPACE` | Kubernetes namespace for child pods | | `LOCALSTACK_K8S_LABELS` | Comma-separated `key=value` labels applied to child pods | | `LOCALSTACK_K8S_ANNOTATIONS` | Comma-separated `key=value` annotations applied to child pods | +| `LOCALSTACK_K8S_POD_CONFIG` | JSON pod configuration for supported child pods. See [Pod Configuration](/aws/enterprise/kubernetes/pod-configuration/) | | `K8S_CONTAINER_SECURITY_CONTEXT` | JSON security context applied to child pod containers | | `K8S_CURL_INIT_IMAGE` | Init container image used for network readiness checks | | `LAMBDA_K8S_INIT_IMAGE` | Init container image used in Lambda pods | diff --git a/src/content/docs/aws/enterprise/kubernetes/faq.md b/src/content/docs/aws/enterprise/kubernetes/faq.md index 1b9f8cec..01130954 100644 --- a/src/content/docs/aws/enterprise/kubernetes/faq.md +++ b/src/content/docs/aws/enterprise/kubernetes/faq.md @@ -3,7 +3,7 @@ title: FAQ description: FAQ template: doc sidebar: - order: 6 + order: 9 tags: ["Enterprise"] --- diff --git a/src/content/docs/aws/enterprise/kubernetes/limitations.md b/src/content/docs/aws/enterprise/kubernetes/limitations.md index 3dec9bcb..f8a29cb0 100644 --- a/src/content/docs/aws/enterprise/kubernetes/limitations.md +++ b/src/content/docs/aws/enterprise/kubernetes/limitations.md @@ -3,7 +3,7 @@ title: Limitations description: Known limitations when running LocalStack on Kubernetes template: doc sidebar: - order: 7 + order: 8 tags: ["Enterprise"] --- diff --git a/src/content/docs/aws/enterprise/kubernetes/pod-configuration.md b/src/content/docs/aws/enterprise/kubernetes/pod-configuration.md new file mode 100644 index 00000000..3eee0e81 --- /dev/null +++ b/src/content/docs/aws/enterprise/kubernetes/pod-configuration.md @@ -0,0 +1,244 @@ +--- +title: Pod Configuration +description: Configure Kubernetes pods launched by LocalStack services. +template: doc +sidebar: + order: 6 +tags: ["Enterprise"] +--- + +## Introduction + +When LocalStack runs inside Kubernetes with the Kubernetes executor enabled, some services create child pods for workloads such as Lambda invocations and ECS tasks. +In heterogeneous clusters, these child pods may need additional Kubernetes configuration so they are scheduled onto the right node pools, carry the right resource requests, or integrate with cluster policies. + +Use the `LOCALSTACK_K8S_POD_CONFIG` environment variable to configure Kubernetes metadata, scheduling, and resource settings for LocalStack-spawned pods. +The variable accepts a JSON object with reusable `profiles` and optional per-service mappings. +The value must be valid JSON; an invalid value raises an error when the first child pod is created. + +::::note +This feature currently applies to Lambda and ECS pods created by the Kubernetes executor. +Support for additional service integrations is being added incrementally. +:::: + +## Supported fields + +Each profile can define the following fields: + +| Field | Description | +|---|---| +| `tolerations` | Kubernetes tolerations applied to the pod spec | +| `nodeSelector` | Node selector applied to the pod spec | +| `affinity` | Kubernetes affinity configuration applied to the pod spec | +| `topologySpreadConstraints` | Topology spread constraints applied to the pod spec | +| `priorityClassName` | Priority class name applied to the pod spec | +| `resources` | Resource requests and limits applied to every non-init container in the pod | +| `labels` | Labels merged into the pod metadata | +| `annotations` | Annotations merged into the pod metadata | + +Scheduling fields such as `nodeSelector`, `tolerations`, and `affinity` replace the corresponding values on the generated pod spec. +They are not merged with existing spec values. +Resource settings are applied to the pod's application containers; init containers keep their generated defaults. + +## Configure default profiles + +The simplest configuration is to define default profiles. +LocalStack uses the architecture-specific defaults for Lambda and ECS when the workload architecture is known, and falls back to `default` otherwise. + +```json +{ + "profiles": { + "default": { + "nodeSelector": { + "pool": "general" + } + }, + "defaultArm64": { + "nodeSelector": { + "pool": "arm-nodes" + } + }, + "defaultAmd64": { + "nodeSelector": { + "pool": "amd-nodes" + } + } + } +} +``` + +With this configuration: + +- Lambda and ECS ARM workloads use the `defaultArm64` profile. +- Lambda and ECS x86 workloads use the `defaultAmd64` profile. +- Workloads without architecture information use the `default` profile. + +To use this with the Helm chart, pass the JSON as an environment variable in your `values.yaml`: + +```yaml +extraEnvVars: + - name: LOCALSTACK_K8S_POD_CONFIG + value: | + { + "profiles": { + "default": { + "nodeSelector": { + "pool": "general" + } + }, + "defaultArm64": { + "nodeSelector": { + "pool": "arm-nodes" + } + }, + "defaultAmd64": { + "nodeSelector": { + "pool": "amd-nodes" + } + } + } + } +``` + +## Configure per-service profiles + +Use the `services` block when a service needs a dedicated profile. +A service can reference a single profile with `profile`, or it can map `arm64` and `amd64` workloads to different profiles. + +```json +{ + "profiles": { + "default": { + "nodeSelector": { + "pool": "general" + } + }, + "defaultAmd64": { + "nodeSelector": { + "pool": "amd-nodes" + } + }, + "lambda-arm": { + "nodeSelector": { + "pool": "lambda-arm-nodes" + }, + "resources": { + "requests": { + "cpu": "500m", + "memory": "512Mi" + }, + "limits": { + "cpu": "2", + "memory": "2Gi" + } + } + }, + "ecs-tasks": { + "nodeSelector": { + "pool": "ecs-nodes" + } + } + }, + "services": { + "lambda": { + "arm64": "lambda-arm" + }, + "ecs": { + "profile": "ecs-tasks" + } + } +} +``` + +In this example, ARM Lambda pods use `lambda-arm`. +AMD64 Lambda pods use `defaultAmd64`. +ECS pods always use `ecs-tasks`, regardless of architecture, because `services.ecs.profile` bypasses architecture-based routing. +Workloads without architecture information use `default`. + +## Profile resolution + +LocalStack resolves the profile for a child pod in this order: + +1. `services..profile` +2. `services..arm64` or `services..amd64` +3. `profiles.defaultArm64` or `profiles.defaultAmd64` +4. `profiles.default` +5. No scheduling, resource, or metadata profile is applied. System labels are still added. + +`defaultArm64`, `defaultAmd64`, and `default` are reserved profile names. +Use them only for global defaults. + +Architecture values are normalized before lookup. +For example, `ARM64` is treated as `arm64`, and `x86_64` or `X86_64` are treated as `amd64`. +The keys in `LOCALSTACK_K8S_POD_CONFIG` should still be `arm64` and `amd64`. + +If a service references a profile that does not exist, LocalStack logs a warning and applies no pod configuration for that request. +It does not silently fall back to `default`. + +## Labels and annotations + +`labels` and `annotations` in `LOCALSTACK_K8S_POD_CONFIG` are merged into the metadata of the generated child pod. +Profile labels override labels set through `LOCALSTACK_K8S_LABELS`, and profile annotations override annotations set through `LOCALSTACK_K8S_ANNOTATIONS`. + +LocalStack also injects the following system labels: + +```yaml +app.kubernetes.io/managed-by: localstack +localstack.cloud/service: +``` + +System labels are applied last and cannot be overridden. +You can use these labels to target LocalStack-spawned pods from Kubernetes tooling such as network policies, admission controllers, or monitoring agents. + +## Example with scheduling and metadata + +The following example schedules Lambda pods on dedicated nodes, adds tolerations, sets resource limits, and attaches metadata used by platform tooling: + +```json +{ + "profiles": { + "lambda": { + "nodeSelector": { + "workload": "localstack-lambda" + }, + "tolerations": [ + { + "key": "dedicated", + "operator": "Equal", + "value": "localstack", + "effect": "NoSchedule" + } + ], + "resources": { + "requests": { + "cpu": "250m", + "memory": "256Mi" + }, + "limits": { + "cpu": "1", + "memory": "1Gi" + } + }, + "labels": { + "team": "platform" + }, + "annotations": { + "prometheus.io/scrape": "true" + } + } + }, + "services": { + "lambda": { + "profile": "lambda" + } + } +} +``` + +## Related configuration + +- Use `LOCALSTACK_K8S_NAMESPACE` to choose the namespace for child pods. +- Use `LOCALSTACK_K8S_LABELS` and `LOCALSTACK_K8S_ANNOTATIONS` for simple labels and annotations that apply to all child pods. +- Use `K8S_CONTAINER_SECURITY_CONTEXT` to configure the security context for child pod containers. + +For the complete list of Kubernetes executor configuration variables, see the [Kubernetes configuration reference](/aws/enterprise/kubernetes/configuration/).