diff --git a/.icons/nixos-rainbow.svg b/.icons/nixos-rainbow.svg new file mode 100644 index 000000000..440f3f7a0 --- /dev/null +++ b/.icons/nixos-rainbow.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/.icons/nixos-trans.svg b/.icons/nixos-trans.svg new file mode 100644 index 000000000..0d72d1ae0 --- /dev/null +++ b/.icons/nixos-trans.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/.icons/nixos.svg b/.icons/nixos.svg new file mode 100644 index 000000000..a7b94a69d --- /dev/null +++ b/.icons/nixos.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 22007f657..4796156dc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -109,6 +109,7 @@ output "scripts" { - Use `tf` (not `hcl`) for code blocks in README; use relative icon paths (e.g., `../../../../.icons/`) - **Never include parameter listings or input/output variable tables in module or template READMEs.** This includes workspace parameters declared with `coder_parameter`. The registry automatically parses the Terraform source and displays parameters in a dedicated tab on `registry.coder.com`; input/output documentation is also generated from the source. Duplicating these listings in the README is redundant and creates maintenance drift. - Usage examples (e.g., a `module "..." { }` block) and explanations of parameter behavior are encouraged, but not tables or lists enumerating parameters, inputs, or outputs. +- Script shebangs must be `#!/usr/bin/env bash`, never `#!/bin/bash`. NixOS workspaces have only `/bin/sh`, so the kernel fails the exec before anything runs and the agent reports exit 255 with an empty log — which looks like a broken module, not a missing interpreter. ### Variable and output conventions diff --git a/registry/coder-labs/templates/aws-nixos/README.md b/registry/coder-labs/templates/aws-nixos/README.md new file mode 100644 index 000000000..b193ff8a4 --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/README.md @@ -0,0 +1,61 @@ +--- +display_name: AWS EC2 (NixOS) +description: Provision NixOS EC2 VMs as Coder workspaces from a flake +icon: ../../../../.icons/nixos.svg +verified: true +tags: [vm, linux, aws, nixos, persistent-vm] +--- + +# NixOS workspaces on AWS EC2 + +Boot an EC2 workspace from a Git flake. NixOS rebuilds before the agent starts. + +## Before you start + +- Give the Coder provisioner AWS credentials through the usual provider credential chain. + See the [EC2 policy example](../../../coder/templates/aws-linux/PREREQUISITES.md); its `RunInstances` and `CreateTags` permissions cover more than tagged resources. +- Keep a default VPC/subnet. Allow provisioner egress to AWS/JetBrains and VM egress to Coder/Git/Nix caches. +- Use a supported Git URL: `https://host/org/repo` or `git+ssh://git@host/org/repo`, with optional `?ref=branch`. + `github:` and `?dir=` are not supported. Commit your flake before starting a workspace. + +## Choose a flake + +**Start with the [example flake](https://github.com/coder/nixos-example-flake):** + +1. Fork it, edit `configuration.nix`, and commit your changes and `flake.lock`. +2. Set `flake_ref` to your fork's Git URL. Keep the default `flake_attr = "coder-workspace-ec2-$ARCH"`. +3. Push the template and create a workspace. Its default instance type is `t3.medium`. + +**Bring an existing flake:** + +1. Add `github:coder/nixos-modules` as an input. Import `coder-modules.nixosModules.default` in each workspace host. +2. Include EC2 hardware support; the [example hardware module](https://github.com/coder/nixos-example-flake/blob/main/hardware/ec2.nix) imports the boot-critical NixOS Amazon image module. +3. Set each host's `nixpkgs.hostPlatform` and `coder.flakeAttr` to its own `nixosConfigurations` name. +4. Export hosts for the instance types you offer. Use `$ARCH` in `flake_attr` for paired `x86_64`/`aarch64` names, or a fixed name for one architecture. +5. Set `flake_ref` and `flake_attr`, push the template, then create a workspace. + +Instance type determines AMI, agent architecture, and `$ARCH`. Small sizes may lack build memory. + +## Work with the workspace + +Boot syncs `/etc/nixos` and rebuilds. Dirty trees and local commits stay untouched. To rebuild manually: + +```console +sudo nixos-rebuild switch --flake /etc/nixos#coder-workspace-ec2-x86_64 +``` + +Use your host's attribute instead of the example name. The root disk and Nix store survive stop/start, **not** instance deletion or replacement. A larger root disk can be selected later; EBS cannot shrink it. + +Watch the **NixOS** workspace log or `/var/log/coder-nixos/rebuild-latest.log`. If first boot has no agent, use EC2 console output: + +```console +aws ec2 get-console-output --instance-id --output text +``` + +## Secrets and limitations + +Do not put secrets in Nix expressions: the Nix store is readable on the VM. Workspace facts and optional bootstrap files are not secret storage. The agent token is kept out of Nix, but EC2 user-data and Terraform state contain it; restrict access to both. Processes with instance-metadata access can read user-data. + +Private repos need Git authentication before first boot and separate credentials for private Nix inputs; this template supplies neither. HTTP credentials in `flake_ref` are allowed but exposed in Terraform state, EC2 user-data, Coder metadata, and `/etc/nixos/.git/config`. Prefer root-managed credentials. NixOS scripts need `#!/usr/bin/env bash`; downloaded IDE binaries need `programs.nix-ld`. + +Existing templates may retain a stored legacy `flake_attr`; update that variable explicitly before rebuilding against renamed example-flake hosts. diff --git a/registry/coder-labs/templates/aws-nixos/main.tf b/registry/coder-labs/templates/aws-nixos/main.tf new file mode 100644 index 000000000..454e01a18 --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/main.tf @@ -0,0 +1,247 @@ +terraform { + required_providers { + coder = { + source = "coder/coder" + version = "~> 2.0" + } + aws = { + source = "hashicorp/aws" + } + } +} + +module "aws-region" { + source = "registry.coder.com/coder/aws-region/coder" + version = "~> 1.1" + default = "eu-west-3" +} + +provider "aws" { + region = module.aws-region.value +} + +variable "flake_ref" { + description = <<-EOT + Git reference to the NixOS configuration, in the form `nix` itself + accepts: `https://host/org/repo`, optionally with a `git+` prefix and a + `?ref=` branch. Without `?ref=` the remote's default branch is used. + + The configuration must be committed -- a Git flake reference only ever + sees committed files. + EOT + type = string + default = "https://github.com/coder/nixos-example-flake" + + validation { + condition = can(regex("^(git\\+)?(https?|ssh)://", var.flake_ref)) && !can(regex("[[:cntrl:]]", var.flake_ref)) && !strcontains(var.flake_ref, "#") && (!strcontains(var.flake_ref, "?") || can(regex("\\?ref=[^&#?]+$", var.flake_ref))) + error_message = "Use an http(s) or ssh Git URL without control characters, fragments, or query parameters other than ?ref=." + } +} + +variable "flake_attr" { + description = "`nixosConfigurations` attribute to build. `$ARCH` is replaced with `x86_64` or `aarch64` to match the instance type." + type = string + default = "coder-workspace-ec2-$ARCH" +} + +variable "nixos_release" { + description = <<-EOT + NixOS release series used to select the AMI, matched as + `nixos/*`. Only affects newly created workspaces; packages and + the kernel come from the flake's own nixpkgs pin. + EOT + type = string + default = "26.05" +} + +module "aws-ec2-instance-type" { + source = "registry.coder.com/coder/aws-ec2-instance-type/coder" + version = "~> 1.0" + + default = "t3.medium" + description = trimspace(<<-EOT + t3.medium is the smallest that works: the NixOS AMI configures no swap and + the Nix store shares the root volume, so a rebuild that has to compile + anything will exhaust a 1-2 GiB instance. + EOT + ) + include = [ + "t3", + "t4g", + "m7g", + ] +} + +data "coder_parameter" "root_volume_size" { + name = "root_volume_size" + display_name = "Root volume size (GiB)" + description = "Holds the Nix store as well as /home. Can be increased later." + type = "number" + default = 80 + mutable = true + + validation { + min = 40 + max = 2000 + monotonic = "increasing" + } +} + +data "coder_workspace" "me" {} +data "coder_workspace_owner" "me" {} + +data "aws_ami" "nixos" { + most_recent = true + filter { + name = "name" + values = ["nixos/${var.nixos_release}*"] + } + filter { + name = "architecture" + values = [local.instance.arch] + } + # Restrict the name match to official NixOS images. + owners = ["427812963091"] +} + +resource "coder_agent" "main" { + count = data.coder_workspace.me.start_count + arch = local.instance.coder_arch + os = "linux" + auth = "token" + # The first boot rebuilds NixOS before starting the agent. + connection_timeout = 1200 + + metadata { + key = "cpu" + display_name = "CPU Usage" + interval = 5 + timeout = 5 + script = "coder stat cpu" + } + metadata { + key = "memory" + display_name = "Memory Usage" + interval = 5 + timeout = 5 + script = "coder stat mem" + } + metadata { + key = "disk" + display_name = "Disk Usage" + interval = 600 + timeout = 30 + script = "coder stat disk --path $HOME" + } + # Include staged-but-not-activated generations in the agent metadata. + metadata { + key = "nixos" + display_name = "NixOS version" + interval = 60 + timeout = 10 + script = module.nix.version_command + } +} + +module "code-server" { + count = data.coder_workspace.me.start_count + source = "registry.coder.com/coder/code-server/coder" + version = "~> 1.0" + agent_id = coder_agent.main[0].id + order = 1 +} + +# IDE binaries need nix-ld in the flake; the example enables it. +module "jetbrains-gateway" { + count = data.coder_workspace.me.start_count + source = "registry.coder.com/coder/jetbrains-gateway/coder" + version = "~> 1.2" + agent_id = coder_agent.main[0].id + agent_name = "main" + arch = local.instance.coder_arch + folder = "/home/coder" + # All listed IDEs have both x86 and ARM builds. + jetbrains_ides = ["IU", "PY", "GO", "WS"] + default = "IU" + latest = true + order = 2 +} + +module "git-config" { + count = data.coder_workspace.me.start_count + source = "registry.coder.com/coder/git-config/coder" + version = "~> 1.0" + agent_id = coder_agent.main[0].id +} + +locals { + instance = module.aws-ec2-instance-type.instances[module.aws-ec2-instance-type.value] + nix_arch = local.instance.arch == "arm64" ? "aarch64" : local.instance.arch +} + +module "nix" { + source = "./modules/nix" + + flake_ref = var.flake_ref + flake_attr = var.flake_attr + arch = local.nix_arch +} + +module "amazon-init" { + source = "./modules/amazon-init" + + agent_token = try(coder_agent.main[0].token, "") + agent_init_script = try(coder_agent.main[0].init_script, "") + boot_script = module.nix.boot_script + values = module.nix.values + + log_display_name = "NixOS" + log_icon = "/icon/nix.svg" +} + +resource "aws_instance" "dev" { + ami = data.aws_ami.nixos.id + availability_zone = module.aws-region.default_availability_zone + instance_type = module.aws-ec2-instance-type.value + user_data = module.amazon-init.user_data + + # Rotating the user-data token must not replace the persistent root disk. + user_data_replace_on_change = false + + root_block_device { + volume_size = data.coder_parameter.root_volume_size.value + volume_type = "gp3" + encrypted = true + } + + tags = { + Name = "coder-${data.coder_workspace_owner.me.name}-${data.coder_workspace.me.name}" + Coder_Provisioned = "true" + } + + lifecycle { + # New AMI releases must not replace an existing workspace and its disk. + ignore_changes = [ami] + } +} + +resource "coder_metadata" "workspace_info" { + resource_id = aws_instance.dev.id + item { + key = "AMI" + value = aws_instance.dev.ami + } + item { + key = "Flake URI" + value = module.nix.flake_uri + } + item { + key = "Build logs location" + value = module.nix.log_dir + } +} + +resource "aws_ec2_instance_state" "dev" { + instance_id = aws_instance.dev.id + state = data.coder_workspace.me.transition == "start" ? "running" : "stopped" +} diff --git a/registry/coder-labs/templates/aws-nixos/modules/amazon-init/README.md b/registry/coder-labs/templates/aws-nixos/modules/amazon-init/README.md new file mode 100644 index 000000000..7136b3607 --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/modules/amazon-init/README.md @@ -0,0 +1,25 @@ +# amazon-init + +`amazon-init.service` runs this user-data every boot. It writes the agent +handoff and public facts, runs `boot_script` as root, then starts the agent +even on failure. Do not enable the agent unit at boot. + +```tf +module "amazon-init" { + source = "./modules/amazon-init" + agent_token = coder_agent.main.token + agent_init_script = coder_agent.main.init_script + boot_script = local.my_boot_script +} +``` + +`runtime_dir` must be tmpfs. Root-owned scripts cannot be replaced by the agent; +only `agent.env` (0600) and `init.sh` (0700) pass to it. The token also lives in +Terraform state and EC2 user-data: restrict IMDS access. Never put secrets in +public `values` or `files` (0644). + +The root boot script receives `CODER_RUNTIME_DIR`, `CODER_WORKSPACE_FACTS`, +`CODER_ACCESS_URL`, `CODER_AGENT_TOKEN`, `CODER_LOG_SOURCE_ID`, and `CODER_LOG_LIBRARY`. +Early logs require outbound Coder access and `curl`. The shared 1 MiB +agent log cap is budgeted. `files` paths are trusted admin input: do not +write into `runtime_dir` or through symlinked directories. diff --git a/registry/coder-labs/templates/aws-nixos/modules/amazon-init/main.tf b/registry/coder-labs/templates/aws-nixos/modules/amazon-init/main.tf new file mode 100644 index 000000000..8ab823eb5 --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/modules/amazon-init/main.tf @@ -0,0 +1,229 @@ +terraform { + required_version = ">= 1.0" + + required_providers { + coder = { + source = "coder/coder" + version = ">= 2.5" + } + random = { + source = "hashicorp/random" + version = ">= 3.0" + } + } +} + +# Keep the log source stable across workspace restarts. +resource "random_uuid" "log_source" {} + +data "coder_workspace" "me" {} + +data "coder_workspace_owner" "me" {} + +variable "agent_token" { + description = "Agent token. Written to `agent.env` at 0600 on a tmpfs." + type = string + sensitive = true +} + +variable "agent_init_script" { + description = "`coder_agent.init_script`, run verbatim once the boot script has finished." + type = string + sensitive = true +} + +variable "boot_script" { + description = <<-EOT + Shell script run on every boot, after the handoff and workspace facts are + published and before the agent is started. + + It runs as root, as a child process, with these set: + + | Variable | Meaning | + | ------------------------ | ------------------------------------------ | + | `CODER_LOG_LIBRARY` | path to source for `coder_log` | + | `CODER_RUNTIME_DIR` | the tmpfs this module owns | + | `CODER_WORKSPACE_FACTS` | path to `workspace.json` | + | `CODER_ACCESS_URL` | deployment URL | + | `CODER_AGENT_TOKEN` | agent token | + | `CODER_LOG_SOURCE_ID` | log source to write to | + + Its exit status is reported and propagated, but never suppresses the agent + start: a workspace whose boot script failed still has to be reachable. + EOT + type = string + default = "" +} + +variable "files" { + description = <<-EOT + Files to write before the boot script runs: absolute path to contents. + Written mode 0644, parent directories created. + + Contents are carried gzipped and base64-encoded, so any text is safe -- + but note that user-data is itself compressed, and compressing twice buys + nothing. This is for small files; the size precondition on `user_data` is + what stops it being abused. + EOT + type = map(string) + default = {} + + validation { + condition = alltrue([for path in keys(var.files) : startswith(path, "/")]) + error_message = "File paths must be absolute." + } +} + +variable "values" { + description = <<-EOT + Extra facts to publish in `workspace.json`, merged with the ones this + module writes itself. + + This is the injection point for anything the machine needs to know at + runtime: one map entry rather than a new file and a new variable each + time. Keys this module writes (`workspace`, `owner`, `owner_name`, + `owner_email`, `access_url`, `hostname`, `log_source_id`) win on conflict. + + Not for secrets. The file is world-readable, by design -- an unprivileged + service reads it. + EOT + type = map(string) + default = {} +} + +variable "runtime_dir" { + description = "Directory for the agent handoff and this module's own state. Must be on a tmpfs: it holds the token." + type = string + default = "/run/coder" + + validation { + condition = startswith(var.runtime_dir, "/") && var.runtime_dir != "/" && abspath(var.runtime_dir) == var.runtime_dir && !can(regex("[\\x00-\\x1f\\x7f]", var.runtime_dir)) + error_message = "runtime_dir must be a canonical absolute directory path without control characters." + } +} + +variable "path" { + description = "Prepended to `PATH` for the boot script. amazon-init's own PATH is short." + type = string + default = "/run/current-system/sw/bin" +} + +variable "log_display_name" { + description = "Name of that log source in the workspace UI." + type = string + default = "Boot" +} + +variable "log_icon" { + description = "Icon for that log source." + type = string + default = "/icon/widgets.svg" +} + +variable "log_budget_bytes" { + description = <<-EOT + How many bytes of log this module will push before going quiet. + + Coder caps agent logs at 1 MiB per agent across every source, and + overflowing does not truncate: the agent is flagged overflowed and all + later logs are dropped permanently. The default leaves half the cap for + everything else. + EOT + type = number + default = 524288 +} + +variable "hostname" { + description = "Hostname to set on the instance. Defaults to the workspace name." + type = string + default = "" +} + +locals { + hostname = var.hostname != "" ? var.hostname : lower(data.coder_workspace.me.name) + + files = [for path, content in var.files : { + path = base64encode(path) + content = base64gzip(content) + }] + + # Module-owned identity wins over caller-provided facts. + facts = merge(var.values, { + workspace = data.coder_workspace.me.name + owner = data.coder_workspace_owner.me.name + owner_name = coalesce(data.coder_workspace_owner.me.full_name, data.coder_workspace_owner.me.name) + owner_email = data.coder_workspace_owner.me.email + access_url = data.coder_workspace.me.access_url + hostname = local.hostname + + log_source_id = random_uuid.log_source.result + }) + + bootstrap = templatefile("${path.module}/scripts/bootstrap.sh.tftpl", { + FACTS_JSON = jsonencode(local.facts) + + LOG_SH = file("${path.module}/scripts/log.sh") + FILES = local.files + BOOT_SCRIPT = var.boot_script + INIT_SCRIPT = var.agent_init_script + + ARG_ACCESS_URL = base64encode(data.coder_workspace.me.access_url) + ARG_AGENT_TOKEN = base64encode(var.agent_token) + ARG_RUNTIME_DIR = base64encode(var.runtime_dir) + ARG_PATH = base64encode(var.path) + + ARG_LOG_SOURCE_ID = random_uuid.log_source.result + ARG_LOG_BUDGET = var.log_budget_bytes + ARG_LOG_REGISTRATION_B64 = base64encode(jsonencode({ + id = random_uuid.log_source.result + display_name = var.log_display_name + icon = var.log_icon + })) + + ARG_HOSTNAME = base64encode(local.hostname) + }) + + # EC2 caps user-data at 16 KiB; compress the bootstrap before sending it. + user_data = <<-SH + #!/usr/bin/env bash + set -euo pipefail + runtime_dir=$(printf %s '${base64encode(var.runtime_dir)}' | base64 -d) + install -d -m 0700 -o root -g root -- "$runtime_dir" + base64 -d <<'CODER_PAYLOAD' | gzip -dc | install -m 0700 -o root -g root /dev/stdin "$runtime_dir/bootstrap.sh" + ${base64gzip(local.bootstrap)} + CODER_PAYLOAD + exec bash "$runtime_dir/bootstrap.sh" + SH +} + +output "user_data" { + description = "Rendered EC2 user-data. Sensitive: it carries the agent token." + value = local.user_data + sensitive = true + + # Terraform suppresses error messages derived from sensitive values. + precondition { + condition = nonsensitive(length(local.user_data)) < 16384 + error_message = "Rendered user-data is ${nonsensitive(length(local.user_data))} bytes; EC2 allows at most 16384." + } +} + +output "log_source_id" { + description = "Log source the boot output is streamed to." + value = random_uuid.log_source.result +} + +output "runtime_dir" { + description = "Directory holding the agent handoff, the logging library and the workspace facts." + value = var.runtime_dir +} + +output "workspace_facts_path" { + description = "Path to the workspace identity file written on every boot." + value = "${var.runtime_dir}/workspace.json" +} + +output "bootstrap_path" { + description = "Where the user-data wrapper extracts the real boot script." + value = "${var.runtime_dir}/bootstrap.sh" +} diff --git a/registry/coder-labs/templates/aws-nixos/modules/amazon-init/main.tftest.hcl b/registry/coder-labs/templates/aws-nixos/modules/amazon-init/main.tftest.hcl new file mode 100644 index 000000000..6c0dfb410 --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/modules/amazon-init/main.tftest.hcl @@ -0,0 +1,57 @@ +mock_provider "coder" { + mock_data "coder_workspace" { + defaults = { + name = "test" + access_url = "https://coder.example" + } + } + mock_data "coder_workspace_owner" { + defaults = { + name = "owner" + full_name = "Owner" + email = "owner@example.org" + } + } +} + +mock_provider "random" { + mock_resource "random_uuid" { + defaults = { + result = "11111111-1111-4111-8111-111111111111" + } + } +} + +run "safe_render" { + command = apply + variables { + agent_token = "token" + agent_init_script = "echo init" + files = { "/tmp/quote'$(touch injected) 🐈" = "safe text" } + log_display_name = "quoted \" name 🐈" + } + assert { + condition = length(nonsensitive(output.user_data)) < 16384 && startswith(nonsensitive(output.user_data), "#!/usr/bin/env bash") + error_message = "User-data must remain a runnable, EC2-sized shell script." + } +} + +run "reject_relative_path" { + command = plan + variables { + agent_token = "token" + agent_init_script = "echo init" + files = { "relative/file" = "bad" } + } + expect_failures = [var.files] +} + +run "reject_noncanonical_runtime" { + command = plan + variables { + agent_token = "token" + agent_init_script = "echo init" + runtime_dir = "/run//coder" + } + expect_failures = [var.runtime_dir] +} diff --git a/registry/coder-labs/templates/aws-nixos/modules/amazon-init/scripts/bootstrap.sh.tftpl b/registry/coder-labs/templates/aws-nixos/modules/amazon-init/scripts/bootstrap.sh.tftpl new file mode 100644 index 000000000..13ccc8609 --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/modules/amazon-init/scripts/bootstrap.sh.tftpl @@ -0,0 +1,121 @@ +#!/usr/bin/env bash +# amazon-init runs this on every boot; the agent must start after the boot script. +set -Eeuo pipefail +decode() { printf '%s' "$1" | base64 -d; } +export PATH="$(decode '${ARG_PATH}'):$PATH" +export HOME=/root + +ACCESS_URL=$(decode '${ARG_ACCESS_URL}') +AGENT_TOKEN=$(decode '${ARG_AGENT_TOKEN}') +LOG_SOURCE_ID='${ARG_LOG_SOURCE_ID}' +LOG_BUDGET='${ARG_LOG_BUDGET}' +HOSTNAME_=$(decode '${ARG_HOSTNAME}') +RUNTIME_DIR=$(decode '${ARG_RUNTIME_DIR}') + +# The token stays on tmpfs; only handoff files are owned by the agent. +install -d -m 0700 -- "$RUNTIME_DIR" +chown root:root -- "$RUNTIME_DIR" +chmod 0711 -- "$RUNTIME_DIR" +rm -f "$RUNTIME_DIR/ready" + +umask 077 +printf 'CODER_AGENT_TOKEN=%s\nCODER_AGENT_URL=%s\n' "$AGENT_TOKEN" "$ACCESS_URL" \ + >"$RUNTIME_DIR/agent.env" +umask 022 +cat >"$RUNTIME_DIR/init.sh" <<'CODER_AMAZON_INIT_AGENT_SCRIPT' +${INIT_SCRIPT} +CODER_AMAZON_INIT_AGENT_SCRIPT +chmod 0700 "$RUNTIME_DIR/init.sh" + +chown_handoff() { + local target + target=$(systemctl show coder-agent -p User --value 2>/dev/null || true) + [ -n "$target" ] || return 0 + chown "$target" "$RUNTIME_DIR/agent.env" "$RUNTIME_DIR/init.sh" 2>/dev/null || true +} +chown_handoff +touch "$RUNTIME_DIR/ready" +chown_handoff + +[ -z "$HOSTNAME_" ] || hostnamectl set-hostname "$HOSTNAME_" || true + +export CODER_ACCESS_URL="$ACCESS_URL" +export CODER_AGENT_TOKEN="$AGENT_TOKEN" +export CODER_LOG_SOURCE_ID="$LOG_SOURCE_ID" +export CODER_LOG_STATE_DIR="$RUNTIME_DIR" +export CODER_LOG_BUDGET="$LOG_BUDGET" + +cat >"$RUNTIME_DIR/log.sh" <<'CODER_AMAZON_INIT_LOG_LIBRARY' +${LOG_SH} +CODER_AMAZON_INIT_LOG_LIBRARY +chmod 0644 "$RUNTIME_DIR/log.sh" +export CODER_LOG_LIBRARY="$RUNTIME_DIR/log.sh" +# shellcheck source=/dev/null +. "$CODER_LOG_LIBRARY" + +CODER_LOG_REGISTRATION=$(decode '${ARG_LOG_REGISTRATION_B64}') +export CODER_LOG_REGISTRATION +coder_log_init || true + +# Agent startup follows every boot-script outcome, including failures. +start_agent() { + if systemctl is-active --quiet coder-agent; then + return 0 + fi + if ! systemctl cat coder-agent >/dev/null 2>&1; then + coder_log error "coder-agent.service does not exist; the boot script has never created it." || true + return 1 + fi + + chown_handoff + systemctl start coder-agent || true + + sleep 3 + if systemctl is-active --quiet coder-agent; then + coder_log info "coder-agent is active." || true + return 0 + fi + + coder_log error "coder-agent failed to start." || true + journalctl -u coder-agent --no-pager --lines=30 2>/dev/null | coder_log_pipe error || true + return 1 +} + +on_error() { + local rc=$? + coder_log error "Bootstrap failed (exit $rc)." || true + start_agent || true + exit "$rc" +} +trap on_error ERR + +# Public facts must never contain secrets. +cat >"$RUNTIME_DIR/workspace.json" <<'CODER_AMAZON_INIT_FACTS' +${FACTS_JSON} +CODER_AMAZON_INIT_FACTS +chmod 0644 "$RUNTIME_DIR/workspace.json" +export CODER_WORKSPACE_FACTS="$RUNTIME_DIR/workspace.json" +export CODER_RUNTIME_DIR="$RUNTIME_DIR" + +%{ for file in FILES ~} +file_path=$(decode '${file.path}') +install -d -m 0755 -- "$(dirname -- "$file_path")" +printf '%s' '${file.content}' | base64 -d | gzip -dc >"$file_path" +chmod 0644 -- "$file_path" +%{ endfor ~} + +cat >"$RUNTIME_DIR/boot.sh" <<'CODER_AMAZON_INIT_BOOT_SCRIPT' +${BOOT_SCRIPT} +CODER_AMAZON_INIT_BOOT_SCRIPT +chmod 0700 "$RUNTIME_DIR/boot.sh" + +# A child cannot exit the bootstrap before the agent-start attempt. +boot_rc=0 +bash "$RUNTIME_DIR/boot.sh" || boot_rc=$? + +if [ "$boot_rc" -ne 0 ]; then + coder_log error "Boot script exited $boot_rc; starting the agent anyway." || true +fi + +start_agent || true +exit "$boot_rc" diff --git a/registry/coder-labs/templates/aws-nixos/modules/amazon-init/scripts/log.sh b/registry/coder-labs/templates/aws-nixos/modules/amazon-init/scripts/log.sh new file mode 100644 index 000000000..7f54ae3c2 --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/modules/amazon-init/scripts/log.sh @@ -0,0 +1,141 @@ +# shellcheck shell=bash +# Pre-agent log API client; failures never prevent boot. +CODER_LOG_BUDGET="${CODER_LOG_BUDGET:-524288}" +CODER_LOG_STATE_DIR="${CODER_LOG_STATE_DIR:-/run/coder}" +CODER_LOG_MAX_LINE=2048 +CODER_LOG_FLUSH_SECS="${CODER_LOG_FLUSH_SECS:-5}" +CODER_LOG_READY="${CODER_LOG_READY:-0}" + +_curl() { + if [ -z "${CODER_CURL:-}" ]; then + CODER_CURL=$(command -v curl) || return 1 + export CODER_CURL + fi + "$CODER_CURL" "$@" +} + +# A rejected overflow permanently silences every log source on this agent. +coder_log_budget_left() { + local used + used=$(cat "$CODER_LOG_STATE_DIR/log-budget" 2> /dev/null || echo 0) + echo $((CODER_LOG_BUDGET - used)) +} + +coder_log_budget_add() { + local used + used=$(cat "$CODER_LOG_STATE_DIR/log-budget" 2> /dev/null || echo 0) + echo $((used + $1)) > "$CODER_LOG_STATE_DIR/log-budget" +} + +coder_log_init() { + local attempt=0 code + command -v curl > /dev/null 2>&1 || return 1 + mkdir -p "$CODER_LOG_STATE_DIR" || return 1 + while [ "$attempt" -lt 40 ]; do + code=$( + _curl -sS -o /dev/null -w '%{http_code}' -X POST \ + "$CODER_ACCESS_URL/api/v2/workspaceagents/me/log-source" \ + -H "Coder-Session-Token: $CODER_AGENT_TOKEN" \ + -H 'Content-Type: application/json' \ + --data-binary "$CODER_LOG_REGISTRATION" || echo 000 + ) + case "$code" in + 200 | 201) + export CODER_LOG_READY=1 + return 0 + ;; + 401 | 403 | 000 | 5??) + attempt=$((attempt + 1)) + sleep 15 + ;; + *) return 1 ;; + esac + done + return 1 +} + +coder_log_json() { + local level="$1" line="$2" + [ "${#line}" -le "$CODER_LOG_MAX_LINE" ] || line="${line:0:$CODER_LOG_MAX_LINE}..." + line=$(printf '%s' "$line" | sed -e 's/\\/\\\\/g' -e 's/"/\\"/g' -e 's/\r//g' -e 's/\t/ /g') + printf '{"created_at":"%s","level":"%s","output":"%s"}' \ + "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$level" "$line" +} + +coder_log_send() { + local payload request size lock="$CODER_LOG_STATE_DIR/log-budget.lock" tries=0 + payload=$(paste -sd, -) || return 0 + [ -n "$payload" ] || return 0 + request="{\"log_source_id\":\"$CODER_LOG_SOURCE_ID\",\"logs\":[$payload]}" + size=$(printf '%s' "$request" | LC_ALL=C wc -c) + # mkdir is an atomic cross-process lock; fail closed if a writer is stuck. + while ! mkdir "$lock" 2> /dev/null; do + tries=$((tries + 1)) + [ "$tries" -lt 40 ] || return 0 + sleep 0.05 + done + if [ "$(coder_log_budget_left)" -lt "$size" ] || ! coder_log_budget_add "$size"; then + rmdir "$lock" + return 0 + fi + rmdir "$lock" + _curl -sS -o /dev/null -X PATCH \ + "$CODER_ACCESS_URL/api/v2/workspaceagents/me/logs" \ + -H "Coder-Session-Token: $CODER_AGENT_TOKEN" \ + -H 'Content-Type: application/json' \ + --data-binary "$request" || true +} + +coder_log() { + local level="$1" + shift + [ "$CODER_LOG_READY" = 1 ] || return 0 + coder_log_json "$level" "$*" | coder_log_send +} + +# Flush slow producers as well as full batches. +coder_log_pipe() { + local level="${1:-info}" line batch="" n=0 bytes=0 obj rc now last + [ "$CODER_LOG_READY" = 1 ] || { + cat > /dev/null + return 0 + } + last=$(date +%s) + while :; do + line="" + if IFS= read -r -t "$CODER_LOG_FLUSH_SECS" line; then rc=0; else rc=$?; fi + if [ "$rc" -ne 0 ] && [ "$rc" -le 128 ]; then + if [ -n "$line" ]; then + obj=$(coder_log_json "$level" "$line") + batch="${batch:+$batch +}$obj" + fi + break + fi + if [ -n "$line" ]; then + obj=$(coder_log_json "$level" "$line") + batch="${batch:+$batch +}$obj" + n=$((n + 1)) + bytes=$((bytes + $(printf '%s' "$obj" | LC_ALL=C wc -c))) + fi + now=$(date +%s) + if [ "$n" -ge 50 ] || [ "$bytes" -ge 32768 ] \ + || { [ "$n" -gt 0 ] && [ "$((now - last))" -ge "$CODER_LOG_FLUSH_SECS" ]; }; then + printf '%s\n' "$batch" | coder_log_send + batch="" + n=0 + bytes=0 + last=$now + fi + done + [ -z "$batch" ] || printf '%s\n' "$batch" | coder_log_send + return 0 +} + +coder_log_tail() { + local file="$1" lines="${2:-200}" + [ -f "$file" ] || return 0 + coder_log error "--- last $lines lines of $file ---" + tail -n "$lines" "$file" | coder_log_pipe error +} diff --git a/registry/coder-labs/templates/aws-nixos/modules/nix/README.md b/registry/coder-labs/templates/aws-nixos/modules/nix/README.md new file mode 100644 index 000000000..8426f3534 --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/modules/nix/README.md @@ -0,0 +1,18 @@ +# nix + +This local module renders a root-run `boot_script` before the Coder agent starts. It clones a Git flake and runs `nixos-rebuild switch` when the checkout, attribute, or active generation changes. + +```tf +module "nix" { + source = "./modules/nix" + flake_ref = "git+https://github.com/coder/nixos-example-flake?ref=main" + flake_attr = "coder-workspace-ec2-$ARCH" + arch = "x86_64" +} +``` + +References accept HTTP(S) or SSH Git URLs, optional `git+` and `?ref=`. Without `ref`, Git follows the default branch. `$ARCH` expands to `arch`. HTTP URL userinfo is allowed but leaks into the checkout and `flake_uri` output; prefer root-managed authentication. The instance requires outbound Git and Nix input/substituter access, root privileges, systemd, and NixOS. + +A clean checkout fast-forwards; tracked edits and local commits remain untouched. Untracked files do not trigger builds: Git flakes ignore them. The `state_dir` lock prevents races only with callers that take it. Rebuild transcripts live in `log_dir`. First-boot failures may require AWS logs before the agent exists. + +The caller runs `boot_script` as root before agent startup. Display outputs include `flake_uri`, `flake_attr`, `flake_dir`, `log_dir` and `version_command`; `values` passes through unchanged. diff --git a/registry/coder-labs/templates/aws-nixos/modules/nix/main.tf b/registry/coder-labs/templates/aws-nixos/modules/nix/main.tf new file mode 100644 index 000000000..b5463a81d --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/modules/nix/main.tf @@ -0,0 +1,146 @@ +terraform { + required_version = ">= 1.3" +} + +variable "flake_ref" { + description = "HTTP(S) or SSH Git URL of a committed flake; optional git+ prefix and ?ref= branch. Embedded credentials appear in the output and checkout." + type = string + + validation { + condition = can(regex("^(git\\+)?(https?|ssh)://", var.flake_ref)) && !can(regex("[[:cntrl:]]", var.flake_ref)) && !strcontains(var.flake_ref, "#") && (!strcontains(var.flake_ref, "?") || can(regex("\\?ref=[^&#?]+$", var.flake_ref))) + error_message = "flake_ref must be an http(s) or ssh Git URL with optional ?ref=, without control characters, fragments, or other query parameters." + } +} + +variable "flake_attr" { + description = "`nixosConfigurations` attribute to build. `$ARCH` is replaced with `arch`." + type = string + default = "coder-workspace-ec2-$ARCH" + + validation { + condition = !can(regex("[[:cntrl:]]", var.flake_attr)) + error_message = "flake_attr must not contain control characters." + } +} + +variable "arch" { + description = "Nix architecture name substituted into `flake_attr`." + type = string + default = "x86_64" + + validation { + condition = contains(["x86_64", "aarch64"], var.arch) + error_message = "arch must be x86_64 or aarch64." + } +} + +variable "values" { + description = <<-EOT + Extra facts for the bootstrapper to publish on the instance, passed + straight through to `values` on whatever writes them. + + Routed through this module so the caller has one wire, and so a + Nix-specific runtime fact has an obvious home. There are none today: the + configuration already knows its own checkout, attribute and directories, + because it is the thing that sets them. + EOT + type = map(string) + default = {} +} + +variable "flake_dir" { + description = "Checkout to build. Owned by the workspace user so the configuration can be edited in place." + type = string + default = "/etc/nixos" + + validation { + condition = !can(regex("[[:cntrl:]]", var.flake_dir)) + error_message = "flake_dir must not contain control characters." + } +} + +variable "state_dir" { + description = "Revision marker and rebuild lock." + type = string + default = "/var/lib/coder-nixos" + + validation { + condition = !can(regex("[[:cntrl:]]", var.state_dir)) + error_message = "state_dir must not contain control characters." + } +} + +variable "log_dir" { + description = "Rebuild transcripts." + type = string + default = "/var/log/coder-nixos" + + validation { + condition = !can(regex("[[:cntrl:]]", var.log_dir)) + error_message = "log_dir must not contain control characters." + } +} + +locals { + flake_url = replace(replace(var.flake_ref, "/^git\\+/", ""), "/\\?.*$/", "") + + flake_branch = try(regex("[?&]ref=([^&#]+)", var.flake_ref)[0], "") + + flake_attr = replace(var.flake_attr, "$ARCH", var.arch) + + lifecycle_sh = file("${path.module}/scripts/lifecycle.sh") +} + +output "values" { + description = "Facts to publish on the instance, for the bootstrapper's `values`." + value = var.values +} + +output "boot_script" { + description = "Applies the flake. Hand this to whatever runs a script on every boot; it expects to run as root." + value = templatefile("${path.module}/scripts/boot.sh.tftpl", { + lifecycle_sh = local.lifecycle_sh + flake_url = base64encode(local.flake_url) + flake_branch = base64encode(local.flake_branch) + flake_attr = base64encode(local.flake_attr) + flake_dir = base64encode(var.flake_dir) + state_dir = base64encode(var.state_dir) + log_dir = base64encode(var.log_dir) + }) +} + +output "flake_uri" { + description = "The reference actually built, normalised for display." + value = "${local.flake_url}${local.flake_branch == "" ? "" : "?ref=${local.flake_branch}"}#${local.flake_attr}" +} + +output "flake_attr" { + description = "`nixosConfigurations` attribute after `$ARCH` substitution." + value = local.flake_attr +} + +output "flake_dir" { + description = "Checkout on the instance." + value = var.flake_dir +} + +output "log_dir" { + description = "Where rebuild transcripts are written." + value = var.log_dir +} + +output "version_command" { + description = <<-EOT + Shell that reports the running NixOS version, and whether a generation is + staged but not yet booted. For a `coder_agent` metadata block, which has + to be declared inline on the agent. + EOT + value = <<-EOT + version=$(nixos-version 2>/dev/null || echo unknown) + if [ "$(readlink -f /run/current-system)" = "$(readlink -f /nix/var/nix/profiles/system)" ]; then + echo "$version" + else + echo "$version (restart to apply update)" + fi + EOT +} diff --git a/registry/coder-labs/templates/aws-nixos/modules/nix/main.tftest.hcl b/registry/coder-labs/templates/aws-nixos/modules/nix/main.tftest.hcl new file mode 100644 index 000000000..f093c50ca --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/modules/nix/main.tftest.hcl @@ -0,0 +1,57 @@ +run "safe_render" { + command = plan + + variables { + flake_ref = "git+ssh://git@example.org/flake?ref=feature" + flake_attr = "host-$ARCH'$(touch /tmp/should-not-run)" + flake_dir = "/etc/nixos'$(touch /tmp/should-not-run)" + } + + assert { + condition = output.flake_uri == "ssh://git@example.org/flake?ref=feature#host-x86_64'$(touch /tmp/should-not-run)" + error_message = "SSH userinfo or reference parsing changed." + } + assert { + condition = strcontains(output.boot_script, base64encode("/etc/nixos'$(touch /tmp/should-not-run)")) && !strcontains(output.boot_script, "FLAKE_DIR='/etc/nixos'") + error_message = "The boot script must encode untrusted arguments." + } +} + +run "default_branch" { + command = plan + variables { + flake_ref = "https://example.org/flake" + arch = "aarch64" + } + assert { + condition = output.flake_uri == "https://example.org/flake#coder-workspace-ec2-aarch64" + error_message = "The default branch or ARM attribute changed." + } +} + +run "allow_http_userinfo" { + command = plan + variables { + flake_ref = "git+https://user:token@example.org/flake?ref=main" + } + assert { + condition = output.flake_uri == "https://user:token@example.org/flake?ref=main#coder-workspace-ec2-x86_64" + error_message = "Git URL userinfo should remain available for private clones." + } +} + +run "reject_control_characters" { + command = plan + variables { + flake_ref = "https://example.org/flake?ref=main\nmalicious" + } + expect_failures = [var.flake_ref] +} + +run "reject_unsupported_query" { + command = plan + variables { + flake_ref = "https://example.org/flake?dir=subdir" + } + expect_failures = [var.flake_ref] +} diff --git a/registry/coder-labs/templates/aws-nixos/modules/nix/scripts/boot.sh.tftpl b/registry/coder-labs/templates/aws-nixos/modules/nix/scripts/boot.sh.tftpl new file mode 100644 index 000000000..cdce3bae8 --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/modules/nix/scripts/boot.sh.tftpl @@ -0,0 +1,72 @@ +#!/usr/bin/env bash + +set -Eeuo pipefail + +# The dot sentinel preserves trailing newlines through Bash command substitution. +decode_arg() { local value; value="$(printf '%s' "$1" | base64 -d; printf '.')"; printf '%s' "$${value%.}"; } +FLAKE_URL=$(decode_arg '${flake_url}') +FLAKE_BRANCH=$(decode_arg '${flake_branch}') +FLAKE_ATTR=$(decode_arg '${flake_attr}') +FLAKE_DIR=$(decode_arg '${flake_dir}') +STATE_DIR=$(decode_arg '${state_dir}') +LOG_DIR=$(decode_arg '${log_dir}') + +install -d -m 0755 "$STATE_DIR" "$LOG_DIR" + +if [ -r "$${CODER_LOG_LIBRARY:-}" ]; then + # shellcheck source=/dev/null + . "$CODER_LOG_LIBRARY" +else + coder_log() { + local level="$1" + shift + printf '%s: %s\n' "$level" "$*" + } + coder_log_pipe() { cat; } + coder_log_tail() { + [ -f "$1" ] || return 0 + tail -n "$${2:-200}" "$1" + } +fi + +nix_log() { coder_log "$@" || true; } + +NIX_FLAKE_DIR="$FLAKE_DIR" +NIX_FLAKE_ATTR="$FLAKE_ATTR" +NIX_STATE_DIR="$STATE_DIR" +NIX_LOG_DIR="$LOG_DIR" + +${lifecycle_sh} + +TRANSCRIPT="$LOG_DIR/rebuild-$(date -u +%Y%m%dT%H%M%SZ).log" + +on_error() { + local rc=$? + coder_log error "Boot script failed (exit $rc). Transcript: $TRANSCRIPT" || true + coder_log_tail "$TRANSCRIPT" 200 || true + exit "$rc" +} +trap on_error ERR + +nix_lock +nix_sync_checkout "$FLAKE_URL" "$FLAKE_BRANCH" + +REV=$(nix_needs_rebuild) && NEEDS_REBUILD=1 || NEEDS_REBUILD=0 +if [ "$NEEDS_REBUILD" -eq 0 ]; then + coder_log info "Configuration unchanged ($${REV:0:12}); skipping rebuild." || true + exit 0 +fi + +coder_log info "Applying $FLAKE_DIR#$FLAKE_ATTR ($${REV:0:12})" || true +coder_log info "Transcript: $TRANSCRIPT" || true + +if ! nix_apply switch "$TRANSCRIPT"; then + coder_log error "nixos-rebuild switch failed; the previous generation is still active." || true + coder_log_tail "$TRANSCRIPT" 200 || true + exit 1 +fi + +nix_record_rev "$REV" +nix_own_checkout + +coder_log info "Switch complete." || true diff --git a/registry/coder-labs/templates/aws-nixos/modules/nix/scripts/lifecycle.sh b/registry/coder-labs/templates/aws-nixos/modules/nix/scripts/lifecycle.sh new file mode 100644 index 000000000..6ac03de63 --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/modules/nix/scripts/lifecycle.sh @@ -0,0 +1,216 @@ +# shellcheck shell=bash + +export NIX_CONFIG="experimental-features = nix-command flakes" + +_sudo() { + if [ "$(id -u)" -eq 0 ]; then + "$@" + else + sudo "$@" + fi +} + +command -v nix_log > /dev/null 2>&1 || nix_log() { + shift + printf '%s\n' "$*" +} + +nix_redact_url() { + printf '%s' "$1" | sed -E 's#(https?://)[^/@[:space:]]+@#\1[redacted]@#g' +} + +nix_strip_ansi() { + sed -e 's/\x1b\[[0-9;]*[a-zA-Z]//g' -e 's/\r$//' +} + +nix_filter_log() { + local line prefix + while IFS= read -r line || [ -n "$line" ]; do + line=$(nix_redact_url "$line") + case "$line" in + *$'\033'*) line=$(printf '%s' "$line" | nix_strip_ansi) ;; + esac + case "$line" in + "") continue ;; + " "*/nix/store/*) continue ;; + "these "*" will be "*: | "this "*" will be "*:) + case "$line" in + "these "*) line=${line#these } ;; + "this "*) line="1 ${line#this }" ;; + esac + printf '%s\n' "${line%:}" + continue + ;; + "remote: "* | "From "* | " "*".."*"->"*) continue ;; + *"> "*) + prefix=${line%%> *} + case "$prefix" in + "" | *[[:space:]]*) ;; + *) continue ;; + esac + ;; + esac + + case "${line%% *}" in + [a-z]*[!a-zA-Z:]*) ;; + [a-z]*) line="${line^}" ;; + esac + + printf '%s\n' "$line" + done +} + +nix_run_logged() { + local rc=0 out line + out="$(mktemp)" + "$@" > "$out" 2>&1 || rc=$? + nix_filter_log < "$out" | while IFS= read -r line; do nix_log info "$line"; done + rm -f "$out" + return "$rc" +} + +nix_checkout_dirty() { + [ -n "$(_sudo git -C "$NIX_FLAKE_DIR" status --porcelain --untracked-files=no 2> /dev/null | head -1)" ] +} + +nix_checkout_rev() { + _sudo git -C "$NIX_FLAKE_DIR" rev-parse HEAD 2> /dev/null || true +} + +nix_own_checkout() { + local owner + owner=$(stat -c %U "$NIX_FLAKE_DIR" 2> /dev/null || echo root) + _sudo chown -R "$owner" "$NIX_FLAKE_DIR" 2> /dev/null || true +} + +nix_checkout_branch() { + _sudo git -C "$NIX_FLAKE_DIR" rev-parse --abbrev-ref HEAD 2> /dev/null || true +} + +nix_sync_checkout() { + local url="$1" branch="${2:-}" upstream_rev local_rev current_branch rc + + if [ ! -e "$NIX_FLAKE_DIR/flake.nix" ]; then + nix_log info "Cloning $(nix_redact_url "$url") into $NIX_FLAKE_DIR" + _sudo install -d -m 0755 "$NIX_FLAKE_DIR" + local tmp + tmp="$(_sudo mktemp -d)" + rc=0 + if [ -n "$branch" ]; then + nix_run_logged _sudo git clone --branch "$branch" "$url" "$tmp/repo" || rc=$? + else + nix_run_logged _sudo git clone "$url" "$tmp/repo" || rc=$? + fi + if [ "$rc" -ne 0 ]; then + nix_log error "Could not clone $(nix_redact_url "$url")${branch:+ (branch $branch)} (git exited $rc)" + nix_log error "Check the flake reference the template was pushed with: the repository has to exist and be readable from this instance." + _sudo rm -rf "$tmp" + return 1 + fi + _sudo tar -C "$tmp/repo" -cf - . | _sudo tar -C "$NIX_FLAKE_DIR" -xf - + _sudo rm -rf "$tmp" + nix_own_checkout + return 0 + fi + + current_branch="$(nix_checkout_branch)" + [ -n "$branch" ] || branch="$current_branch" + + if nix_checkout_dirty; then + nix_log info "$NIX_FLAKE_DIR has local changes; building those instead of $branch" + return 0 + fi + + if [ -n "$current_branch" ] && [ "$current_branch" != "$branch" ]; then + nix_log warn "$NIX_FLAKE_DIR is on $current_branch, not $branch; building $current_branch" + nix_log warn "Check out $branch there, or delete $NIX_FLAKE_DIR to start from the remote" + return 0 + fi + + # Root fetch writes into .git even when it fails. + rc=0 + nix_run_logged _sudo git -C "$NIX_FLAKE_DIR" fetch --quiet origin "$branch" || rc=$? + nix_own_checkout + if [ "$rc" -ne 0 ]; then + nix_log warn "Could not reach the remote; building the existing checkout" + return 0 + fi + + local_rev="$(nix_checkout_rev)" + upstream_rev="$(_sudo git -C "$NIX_FLAKE_DIR" rev-parse FETCH_HEAD 2> /dev/null || true)" + [ -n "$upstream_rev" ] || return 0 + [ "$local_rev" != "$upstream_rev" ] || return 0 + + # Fast-forward only. A checkout carrying local commits is left alone. + if _sudo git -C "$NIX_FLAKE_DIR" merge-base --is-ancestor "$local_rev" "$upstream_rev" 2> /dev/null; then + nix_log info "Updating $NIX_FLAKE_DIR to ${upstream_rev:0:12}" + _sudo git -C "$NIX_FLAKE_DIR" reset --hard --quiet "$upstream_rev" + nix_own_checkout + else + nix_log info "$NIX_FLAKE_DIR has local commits; building those instead of $branch" + fi +} + +nix_recorded_rev() { + cat "$NIX_STATE_DIR/flake.rev" 2> /dev/null || true +} + +nix_record_rev() { + _sudo install -d -m 0755 "$NIX_STATE_DIR" + printf '%s\n' "$1" | _sudo tee "$NIX_STATE_DIR/flake.rev" > /dev/null +} + +# Compares against /run/current-system, the *activated* system, and not +nix_pending_generation() { + [ "$(readlink -f /run/current-system)" != "$(readlink -f /nix/var/nix/profiles/system)" ] +} + +nix_needs_rebuild() { + local rev + + if nix_checkout_dirty; then + printf 'dirty#%s' "$NIX_FLAKE_ATTR" + return 0 + fi + + rev="$(nix_checkout_rev)" + [ -n "$rev" ] || rev="unknown" + printf '%s#%s' "$rev" "$NIX_FLAKE_ATTR" + + [ "$rev#$NIX_FLAKE_ATTR" = "$(nix_recorded_rev)" ] || return 0 + nix_pending_generation +} + +# Build the local checkout as-is; no remote override or lock-file suppression. +nix_apply() { + local operation="$1" transcript="$2" rc + + _sudo install -d -m 0755 "$NIX_LOG_DIR" + _sudo install -m 0644 /dev/null "$transcript" + _sudo ln -sfn "$transcript" "$NIX_LOG_DIR/rebuild-latest.log" + + # stdbuf must run under _sudo; it cannot exec a shell function. + set +e + _sudo nixos-rebuild "$operation" \ + --flake "$NIX_FLAKE_DIR#$NIX_FLAKE_ATTR" \ + --print-build-logs \ + 2>&1 | _sudo stdbuf -oL tee -a "$transcript" | nix_filter_log \ + | while IFS= read -r line; do nix_log info "$line"; done + rc=${PIPESTATUS[0]} + set -e + + return "$rc" +} + +nix_lock() { + local lock="$NIX_STATE_DIR/rebuild.lock" + _sudo install -d -m 0755 "$NIX_STATE_DIR" + # Mode 0666 so the workspace user can take the same advisory lock as root. + [ -e "$lock" ] || _sudo install -m 0666 /dev/null "$lock" + exec 9> "$lock" + if [ "${1:-wait}" = "nowait" ]; then + flock -n 9 + else + flock 9 + fi +} diff --git a/registry/coder-labs/templates/aws-nixos/modules/nix/scripts/lifecycle.test.sh b/registry/coder-labs/templates/aws-nixos/modules/nix/scripts/lifecycle.test.sh new file mode 100755 index 000000000..99aa71fb0 --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/modules/nix/scripts/lifecycle.test.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +here=$(cd "$(dirname "$0")" && pwd) +root=$(mktemp -d) +trap 'rm -rf "$root"' EXIT +export GIT_CONFIG_NOSYSTEM=1 +export GIT_CONFIG_GLOBAL=/dev/null + +git init -q -b main "$root/remote" +git -C "$root/remote" config user.email test@example.org +git -C "$root/remote" config user.name Test +printf 'initial\n' > "$root/remote/flake.nix" +git -C "$root/remote" add flake.nix +git -C "$root/remote" commit -qm initial + +# Source the production functions while replacing privileged ownership with a probe. +# shellcheck source=lifecycle.sh +source "$here/lifecycle.sh" +nix_pending_generation() { false; } +_sudo() { "$@"; } +ownership_calls=0 +nix_own_checkout() { ownership_calls=$((ownership_calls + 1)); } +NIX_FLAKE_DIR="$root/checkout" +NIX_STATE_DIR="$root/state" +NIX_FLAKE_ATTR=host-one +mkdir -p "$NIX_FLAKE_DIR" + +nix_sync_checkout "file://$root/remote" main +[ -f "$NIX_FLAKE_DIR/flake.nix" ] +[ "$ownership_calls" -eq 1 ] +rev=$(nix_needs_rebuild) +[ "$rev" = "$(git -C "$NIX_FLAKE_DIR" rev-parse HEAD)#host-one" ] +nix_record_rev "$rev" +if nix_needs_rebuild > /dev/null; then + echo 'unexpected rebuild' >&2 + exit 1 +fi +NIX_FLAKE_ATTR=host-two +nix_needs_rebuild > /dev/null +nix_record_rev "$(nix_needs_rebuild)" +if nix_needs_rebuild > /dev/null; then + echo 'unexpected attribute rebuild' >&2 + exit 1 +fi + +printf 'ignored by Git flake\n' > "$NIX_FLAKE_DIR/untracked" +if nix_needs_rebuild > /dev/null; then + echo 'untracked file caused rebuild' >&2 + exit 1 +fi +printf 'modified\n' > "$NIX_FLAKE_DIR/flake.nix" +nix_checkout_dirty +nix_needs_rebuild > /dev/null +git -C "$NIX_FLAKE_DIR" checkout -q -- flake.nix + +printf 'updated\n' > "$root/remote/flake.nix" +git -C "$root/remote" commit -qam updated +nix_sync_checkout "file://$root/remote" main +[ "$(cat "$NIX_FLAKE_DIR/flake.nix")" = updated ] +[ "$ownership_calls" -eq 3 ] + +before=$ownership_calls +nix_sync_checkout "file://$root/remote" main +[ "$ownership_calls" -eq "$((before + 1))" ] +before=$ownership_calls +# A failed fetch must still restore ownership. +git -C "$NIX_FLAKE_DIR" remote set-url origin file:///does-not-exist +nix_sync_checkout "file://$root/remote" main +[ "$ownership_calls" -eq "$((before + 1))" ] + +redacted=$(printf 'fatal: https://user:token@example.org/repo\n' | nix_filter_log) +[[ "$redacted" == *'https://[redacted]@example.org/repo'* ]] +[[ "$redacted" != *token* ]] +[[ "$(nix_redact_url 'https://user:token@example.org/repo')" != *token* ]] +printf 'lifecycle tests passed\n' diff --git a/registry/coder-labs/templates/aws-nixos/tests/architecture.tftest.hcl b/registry/coder-labs/templates/aws-nixos/tests/architecture.tftest.hcl new file mode 100644 index 000000000..62a947f47 --- /dev/null +++ b/registry/coder-labs/templates/aws-nixos/tests/architecture.tftest.hcl @@ -0,0 +1,130 @@ +mock_provider "coder" { + mock_data "coder_workspace" { + defaults = { + name = "test" + start_count = 1 + transition = "start" + } + } + mock_data "coder_workspace_owner" { + defaults = { + name = "owner" + full_name = "Owner" + email = "owner@example.org" + } + } + mock_data "coder_parameter" { + defaults = { + value = "80" + } + } +} + +mock_provider "aws" { + mock_data "aws_ami" { + defaults = { + id = "ami-example" + name = "nixos/latest" + } + } +} + +mock_provider "http" {} +mock_provider "random" {} + +override_module { + target = module.aws-region + outputs = { + value = "eu-west-3" + default_availability_zone = "eu-west-3a" + } +} + +override_module { + target = module.aws-ec2-instance-type + outputs = { + value = "t3.medium" + instances = { + "t3.medium" = { arch = "x86_64", coder_arch = "amd64" } + "t4g.medium" = { arch = "arm64", coder_arch = "arm64" } + } + } +} + +override_module { + target = module.code-server + outputs = {} +} +override_module { + target = module.jetbrains-gateway + outputs = {} +} +override_module { + target = module.git-config + outputs = {} +} + +run "x86" { + command = plan + assert { + condition = local.nix_arch == "x86_64" && coder_agent.main[0].arch == "amd64" && module.nix.flake_attr == "coder-workspace-ec2-x86_64" + error_message = "The x86 instance, agent and flake attribute must agree." + } + assert { + condition = aws_instance.dev.ami == "ami-example" && coder_metadata.workspace_info.item[0].value == aws_instance.dev.ami + error_message = "AMI metadata must show the instance's actual AMI ID." + } +} + +run "arm" { + command = plan + override_module { + target = module.aws-ec2-instance-type + outputs = { + value = "t4g.medium" + instances = { + "t3.medium" = { arch = "x86_64", coder_arch = "amd64" } + "t4g.medium" = { arch = "arm64", coder_arch = "arm64" } + } + } + } + assert { + condition = local.nix_arch == "aarch64" && coder_agent.main[0].arch == "arm64" && module.nix.flake_attr == "coder-workspace-ec2-aarch64" + error_message = "The ARM instance, agent and flake attribute must agree." + } +} + +run "stop" { + command = plan + override_data { + target = data.coder_workspace.me + values = { + name = "test" + start_count = 0 + transition = "stop" + } + } + assert { + condition = length(coder_agent.main) == 0 && aws_ec2_instance_state.dev.state == "stopped" + error_message = "Stopping must remove the agent and stop, not destroy, the EC2 instance." + } +} + +run "reject_unsupported_query" { + command = plan + variables { + flake_ref = "https://example.org/flake?dir=subdir" + } + expect_failures = [var.flake_ref] +} + +run "allow_http_userinfo" { + command = plan + variables { + flake_ref = "https://user:token@example.org/flake?ref=main" + } + assert { + condition = module.nix.flake_uri == "https://user:token@example.org/flake?ref=main#coder-workspace-ec2-x86_64" + error_message = "Template must allow userinfo for private Git clones." + } +}