Skip to content
35 changes: 35 additions & 0 deletions deploy/cloudformation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,41 @@ aws cloudformation create-stack-instances \
- The `CreationPolicy` with `ResourceSignal` ensures the stack only completes when the instance is fully bootstrapped
- Requires `CAPABILITY_NAMED_IAM` due to named IAM roles and users

## VPC Block Public Access exclusion

The stack creates a VPC-wide `allow-bidirectional` VPC Block Public Access exclusion so bootstrap egress and public endpoints keep working when BPA is enabled. This exempts the **entire VPC** for internet ingress and egress, not only the LowKey instance.

| `ExistingVpcId` | `CreateVpcBpaExclusion` | Behavior |
|---|---|---|
| empty (new VPC) | `true` | Stack-owned exclusion, deleted with the stack |
| empty (new VPC) | `false` | Same as `true` — ignored, because a new VPC always needs one |
| set (reused VPC) | `true` | Created with `DeletionPolicy: Retain` / `UpdateReplacePolicy: Retain`, so it survives stack deletion |
| set (reused VPC) | `false` | Nothing created; the VPC must already have a complete `allow-bidirectional` exclusion |

Reusing a VPC that already has one **requires** `CreateVpcBpaExclusion=false`, or the stack fails trying to create a duplicate. UserData revalidates the exclusion before running any pack and aborts if it is missing.

Cleanup, once no deployment needs that VPC exempt:

```bash
aws ec2 describe-vpc-block-public-access-exclusions \
--query 'VpcBlockPublicAccessExclusions[].[ExclusionId,ResourceArn,State]' --output text
aws ec2 delete-vpc-block-public-access-exclusion --exclusion-id <id>
```

> **Warning**
> A **new-VPC** exclusion is stack-owned and deleted with its stack. If another LowKey deployment was later pointed at that same VPC, deleting the first stack removes the exemption the second one depends on. Recreate an exclusion, or redeploy the remaining stack with `CreateVpcBpaExclusion=true`.

### Limitation: first 100 exclusions only

Exclusion discovery is deliberately not paginated. Both the installer and the instance-side check inspect only the **first 100** BPA exclusions in the region (`--max-results 100`). The default quota is well under that, so this is an accepted edge case for now.

If an account holds more than 100 exclusions and the target VPC's exclusion falls outside that first page:

- The installer treats it as absent and passes `CreateVpcBpaExclusion=true`, so CloudFormation attempts a duplicate and the stack fails with a create error.
- The instance-side check likewise does not see it and refuses to start pack bootstrap, so the deployment fails closed rather than running without internet access.

Workaround: none that keeps the deployment working. Setting `CreateVpcBpaExclusion=false` only avoids the duplicate-create failure — the instance-side check reads the same first 100 results, so bootstrap still refuses to start. Deploying into such a region requires bringing the region's exclusion count back under 100, so the target VPC's exclusion appears in the first page of results.

## Next Steps

See [Next Steps After Deployment](../README.md#next-steps-after-deployment) for bootstrap scripts setup.
134 changes: 132 additions & 2 deletions deploy/cloudformation/template.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
AWSTemplateFormatVersion: '2010-09-09'
Description: >
OpenClaw Instance - Deploys a fully configured OpenClaw AI assistant on EC2
within its own VPC. Designed for StackSet deployment across AWS Organization accounts.
LowKey agent host on EC2 - provisions public VPC connectivity and a
bidirectional VPC Block Public Access exclusion for reliable bootstrap and ingress.

Metadata:
AWSToolsMetrics:
AWSAgentToolkit: aws-cloudformation@2
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
Expand Down Expand Up @@ -53,6 +55,7 @@ Metadata:
- ExistingVpcId
- ExistingSubnetId
- ExistingSubnetId2
- CreateVpcBpaExclusion
- SSHAllowedCidr
- KeyPairName
- Label:
Expand Down Expand Up @@ -368,6 +371,12 @@ Parameters:
Description: "Second public subnet ID in a different AZ. Required for KiroCrew pack (ALB needs 2 AZs) when ExistingVpcId is set."
AllowedPattern: '^(subnet-[a-z0-9]+)?$'

CreateVpcBpaExclusion:
Type: String
Default: 'true'
AllowedValues: ['true', 'false']
Description: "Create a VPC-wide allow-bidirectional Block Public Access exclusion. The LowKey wizard sets this to false only when the reused VPC already has an active bidirectional exclusion."

RepoBranch:
Type: String
Default: 'main'
Expand Down Expand Up @@ -463,6 +472,11 @@ Conditions:
IsApiKey: !Equals [!Ref ModelMode, 'api-key']
IsBedrock: !Equals [!Ref ModelMode, 'bedrock']
CreateNewVpc: !Equals [!Ref ExistingVpcId, '']
# New VPCs always get a stack-owned exclusion. When LowKey must create one
# for a reused VPC, retain it independently of this individual deployment.
CreateExistingVpcBpaExclusion: !And
- !Not [!Condition CreateNewVpc]
- !Equals [!Ref CreateVpcBpaExclusion, 'true']
IsBuilder: !Equals [!Ref ProfileName, 'builder']
IsNotBuilder: !Not [!Condition IsBuilder]
IsAccountAssistant: !Equals [!Ref ProfileName, 'account_assistant']
Expand Down Expand Up @@ -506,6 +520,56 @@ Resources:
- Key: loki:pack
Value: !Ref PackName

# New LowKey VPCs own their exclusion: stack deletion removes both in order.
VpcBpaExclusion:
Type: AWS::EC2::VPCBlockPublicAccessExclusion
Condition: CreateNewVpc
Metadata:
com.aws.cloudformation.Context:
why: Keep LowKey's IGW-backed VPC reachable when regional VPC BPA is enabled.
must:
- VPC-wide allow-bidirectional; bootstrap needs internet egress and public endpoints need ingress.
mutable: review-required
Properties:
InternetGatewayExclusionMode: allow-bidirectional
VpcId: !Ref VPC
Tags:
- Key: Name
Value: !Sub '${EnvironmentName}-vpc-bpa-exclusion'
- Key: loki:managed
Value: 'true'
- Key: loki:watermark
Value: !Ref LokiWatermark
- Key: loki:pack
Value: !Ref PackName

# An exclusion on a reused VPC is VPC-wide shared infrastructure. Retain it
# when this deployment is deleted or replaced so another LowKey stack using
# the same VPC cannot lose internet connectivity with the owning stack.
ExistingVpcBpaExclusion:
Type: AWS::EC2::VPCBlockPublicAccessExclusion
Condition: CreateExistingVpcBpaExclusion
DeletionPolicy: Retain
UpdateReplacePolicy: Retain
Metadata:
com.aws.cloudformation.Context:
why: Give a reused VPC a BPA exclusion whose lifetime is independent of one LowKey deployment.
must:
- Retain on stack deletion or replacement because other stacks may share this VPC-wide exclusion.
mutable: review-required
Properties:
InternetGatewayExclusionMode: allow-bidirectional
VpcId: !Ref ExistingVpcId
Tags:
- Key: Name
Value: !Sub '${EnvironmentName}-vpc-bpa-exclusion'
- Key: loki:managed
Value: 'true'
- Key: loki:watermark
Value: !Ref LokiWatermark
- Key: loki:pack
Value: !Ref PackName

InternetGateway:
Type: AWS::EC2::InternetGateway
Condition: CreateNewVpc
Expand Down Expand Up @@ -850,6 +914,16 @@ Resources:
- secretsmanager:GetSecretValue
- secretsmanager:DescribeSecret
Resource: !Sub 'arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${KirocrewTgBotTokenSecret}*'
# Revalidate the VPC-wide prerequisite immediately before bootstrap.
# Inline attachment avoids an IAM propagation race at instance launch.
- PolicyName: !Sub '${EnvironmentName}-vpc-bpa-read'
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: VerifyVpcBpaExclusion
Effect: Allow
Action: ec2:DescribeVpcBlockPublicAccessExclusions
Resource: '*'
Tags:
- Key: Name
Value: !Sub '${EnvironmentName}-role'
Expand Down Expand Up @@ -1838,6 +1912,22 @@ Resources:
# --------------------------------------------------------------------------
Instance:
Type: AWS::EC2::Instance
Metadata:
# Wait for whichever exclusion this stack creates before UserData starts.
# A complete exclusion supplied by a reused VPC needs no stack dependency.
VpcBpaExclusionDependency: !If
- CreateNewVpc
- !Ref VpcBpaExclusion
- !If
- CreateExistingVpcBpaExclusion
- !Ref ExistingVpcBpaExclusion
- existing
# On the new-VPC path the public internet route must exist before UserData
# makes its first AWS call; a reused VPC brings its own routing.
PublicRoutingDependency: !If
- CreateNewVpc
- !Sub '${VPCGatewayAttachment}|${PublicRoute}|${PublicSubnetRouteTableAssociation}'
- existing
CreationPolicy:
ResourceSignal:
Timeout: PT30M
Expand Down Expand Up @@ -1933,6 +2023,46 @@ Resources:
aws cloudformation signal-resource --stack-name "${!STACK_NAME}" --logical-resource-id Instance --unique-id "$_INSTANCE_ID" --status FAILURE --region "$REGION" 2>/dev/null || true
fi
' ERR
# Fail closed before any pack code starts. This revalidates exclusions
# supplied by reused VPCs and protects direct CloudFormation callers
# that bypass the installer's pre-deployment check.
_IMDS_TOKEN=$(curl -sf -X PUT http://169.254.169.254/latest/api/token \
-H "X-aws-ec2-metadata-token-ttl-seconds: 60")
_PRIMARY_MAC=$(curl -sf -H "X-aws-ec2-metadata-token: $_IMDS_TOKEN" \
http://169.254.169.254/latest/meta-data/network/interfaces/macs/ | head -1)
# IMDS directory listings return each entry with a trailing slash;
# leaving it in would build a rejected double-slash vpc-id path.
_PRIMARY_MAC="${!_PRIMARY_MAC%/}"
_TARGET_VPC_ID=$(curl -sf -H "X-aws-ec2-metadata-token: $_IMDS_TOKEN" \
"http://169.254.169.254/latest/meta-data/network/interfaces/macs/$_PRIMARY_MAC/vpc-id")
Comment thread
royosherove marked this conversation as resolved.
if [[ -z "$_TARGET_VPC_ID" ]]; then
echo "FATAL: could not resolve the instance VPC before BPA validation" >&2
# Fail via a failing command, not 'exit': only that triggers the ERR
# trap above, which publishes SSM status and signals CFN promptly.
false
fi

_BPA_READY=false
for _attempt in 1 2 3 4 5 6; do
# Known limitation: only the first 100 exclusions in the region are
# inspected; see the BPA section in deploy/cloudformation/README.md.
_BPA_COUNT=$(aws ec2 describe-vpc-block-public-access-exclusions \
--region "$REGION" --max-results 100 \
--query "length(VpcBlockPublicAccessExclusions[?ends_with(ResourceArn, ':vpc/$_TARGET_VPC_ID') && InternetGatewayExclusionMode == 'allow-bidirectional' && (State == 'create-complete' || State == 'update-complete')])" \
--output text 2>/dev/null || echo 0)
if [[ "$_BPA_COUNT" =~ ^[1-9][0-9]*$ ]]; then
_BPA_READY=true
break
fi
echo "VPC BPA exclusion not ready (attempt $_attempt/6); retrying in 10s..."
sleep 10
done
if [[ "$_BPA_READY" != "true" ]]; then
echo "FATAL: VPC $_TARGET_VPC_ID lacks a complete allow-bidirectional BPA exclusion; refusing to start pack bootstrap" >&2
# Fail via a failing command, not 'exit', so the ERR trap reports it.
false
Comment on lines +2060 to +2063

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid relying on blocked networking to report BPA failure

When CreateVpcBpaExclusion=false is used without a valid exclusion while regional BPA is active, the missing exclusion also blocks the instance's IGW path to the public SSM and CloudFormation APIs. Although false now invokes the ERR trap, both status writes and signal-resource therefore fail over the same unavailable network, leaving CloudFormation waiting for the full PT30M CreationPolicy timeout. The fresh evidence after the prior review is that the reporting trap still uses public AWS API calls and this template provisions no private endpoints; perform this validation outside the instance, provide endpoint connectivity, or otherwise avoid depending on the blocked path for the failure signal.

Useful? React with 👍 / 👎.

fi

# Ensure git is available (not present on all AMIs)
command -v git &>/dev/null || dnf install -y git 2>/dev/null || yum install -y git
# Clone repo with retry (GitHub blips shouldn't kill bootstrap)
Expand Down
6 changes: 6 additions & 0 deletions deploy/test-templates.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,12 @@ check_contains "$CFN_TEMPLATE" "git clone --depth 1" "CFN: UserData uses git clo
check_contains "$CFN_TEMPLATE" "deploy/bootstrap.sh" "CFN: UserData calls bootstrap.sh"
check_contains "$CFN_TEMPLATE" "--pack" "CFN: UserData passes --pack flag"
check_contains "$CFN_TEMPLATE" "Deployed agent pack" "CFN: PackName in Outputs"
check_contains "$CFN_TEMPLATE" "AWS::EC2::VPCBlockPublicAccessExclusion" "CFN: VPC BPA exclusion resource defined"
check_contains "$CFN_TEMPLATE" "InternetGatewayExclusionMode: allow-bidirectional" "CFN: VPC BPA exclusion allows bidirectional internet traffic"
check_contains "$CFN_TEMPLATE" "CreateVpcBpaExclusion" "CFN: BPA exclusion creation parameter defined"
check_contains "$CFN_TEMPLATE" "ExistingVpcBpaExclusion:" "CFN: reused VPC BPA exclusion has independent lifecycle"
check_contains "$CFN_TEMPLATE" "DeletionPolicy: Retain" "CFN: reused VPC BPA exclusion retained on stack deletion"
check_contains "$CFN_TEMPLATE" "UpdateReplacePolicy: Retain" "CFN: reused VPC BPA exclusion retained on replacement"

echo ""

Expand Down
31 changes: 31 additions & 0 deletions docs/reference/cloudformation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ You rarely set these by hand — `install.sh` computes them from your pack + pro
| `EnableConfigRecorder` | `true` | Enable Config recorder |
| `ExistingVpcId` | `vpc-0abc123` | Reuse an existing VPC instead of creating one |
| `ExistingSubnetId` | `subnet-0def456` | Public subnet in the existing VPC |
| `CreateVpcBpaExclusion` | `true` | Create the VPC-wide `allow-bidirectional` Block Public Access exclusion. Set to `false` only when reusing a VPC that already has a complete one |
| `RepoBranch` | `main` | Git branch of the lowkey repo to clone |
| `SSHAllowedCidr` | `127.0.0.1/32` | CIDR allowed to SSH (default disables SSH) |
| `KiroFromSecret` | `/lowkey/kiro-api-key` | Secrets Manager id/arn for Kiro API key (kiro-cli only) |
Expand All @@ -49,6 +50,29 @@ You rarely set these by hand — `install.sh` computes them from your pack + pro
`LiteLLMApiKey` and `ProviderApiKey` are `NoEcho: true` — they won't appear in stack history or describe-stacks output, but they **do** pass through UserData (which is Base64-encoded in stack metadata). For long-lived secrets, prefer the `--kiro-from-secret` / Secrets Manager pattern which flows only a reference.
</Warning>

## VPC Block Public Access exclusion

Every deployment needs its VPC exempt from [VPC Block Public Access](https://docs.aws.amazon.com/vpc/latest/userguide/security-vpc-bpa.html): bootstrap downloads require internet egress, and public endpoints (such as the KiroCrew ALB) require ingress. The stack creates a VPC-wide `allow-bidirectional` exclusion to guarantee this.

<Warning>
`allow-bidirectional` exempts the **entire VPC**, allowing internet ingress *and* egress for every resource in it — not just the LowKey instance. Deploy only into a VPC where that is acceptable.
</Warning>

| `ExistingVpcId` | `CreateVpcBpaExclusion` | Behavior |
|---|---|---|
| empty (new VPC) | `true` | Creates a stack-owned exclusion, deleted with the stack |
| empty (new VPC) | `false` | Same as `true` — the parameter is ignored, because a new VPC always needs an exclusion |
| set (reused VPC) | `true` | Creates an exclusion with `DeletionPolicy: Retain` and `UpdateReplacePolicy: Retain`, so the shared VPC-wide exclusion outlives this stack |
| set (reused VPC) | `false` | Creates nothing; the VPC must already have a complete `allow-bidirectional` exclusion |

Reusing a VPC that already has a complete exclusion **requires** `CreateVpcBpaExclusion=false` — otherwise CloudFormation attempts a duplicate exclusion and the stack fails. `install.sh` detects this and sets the parameter for you.

Before cloning the repo or running any pack, UserData verifies its own VPC has a complete `allow-bidirectional` exclusion and aborts if it does not, so a misconfigured direct deployment fails closed instead of bootstrapping without connectivity.

<Info>
**Limitation — first 100 exclusions only.** Discovery is not paginated: the installer and the instance-side check each inspect only the first 100 BPA exclusions in the region. The default quota is far below that, so this is an accepted edge case. In a region holding more, an exclusion outside that first page reads as absent — the stack fails on a duplicate create, and bootstrap refuses to start rather than running unexempted. Setting `CreateVpcBpaExclusion=false` is **not** a workaround: it avoids the duplicate create, but the instance-side check reads the same first 100 results and still aborts. Deploying into such a region requires bringing the region's exclusion count back under 100.
</Info>

## Deploying the template directly

```bash
Expand Down Expand Up @@ -105,3 +129,10 @@ The stack owns the VPC, EC2, IAM — everything vanishes together. Security serv
<Info>
If you reused an existing VPC via `ExistingVpcId`, the VPC is **not** deleted (you brought it, you keep it).
</Info>

<Warning>
Two BPA exceptions to "everything vanishes together":

- An exclusion this stack created for a **reused** VPC is `Retain`ed and survives deletion. Remove it with `aws ec2 delete-vpc-block-public-access-exclusion --exclusion-id <id>` once no deployment needs that VPC exempt.
- An exclusion created for a **new** VPC is deleted with the stack. If another LowKey deployment was later pointed at that same VPC, it loses the exemption — recreate one, or redeploy that stack with `CreateVpcBpaExclusion=true`.
</Warning>
Loading