• Resources are region-scoped. An EC2 instance in us-east-1 has nothing to do with one in eu-west-1, they're completely separate.
• us-east-1 (N. Virginia) is where AWS launches new services first. Good for learning, but for production choose a region close to your actual users.

• AZ names are randomly mapped per AWS account. Your us-east-1a may point to a different physical data centre than someone else's us-east-1a. This is intentional to spread load.
• Each subnet lives in exactly one AZ. To span AZs, create one subnet per AZ.

• Edge locations are not for full compute. You cannot run EC2 instances or RDS here. However, Lambda@Edge and CloudFront Functions can execute lightweight code at edge locations in response to CloudFront events, useful for auth, redirects, and header manipulation at the CDN layer.

• Every AWS account has a default VPC (172.31.0.0/16) in every region. If you accidentally delete it, you can recreate it from the VPC console (Actions → Create default VPC) or via aws ec2 create-default-vpc, no support ticket required.
• For production and learning, always create a custom VPC. The default VPC has public subnets by default, which is not ideal for production.
• Two VPCs that need to communicate via VPC Peering cannot have overlapping CIDR ranges. Plan CIDR blocks upfront.

• Being in a public subnet does not automatically expose a resource. The resource also needs a public IP address, and the Security Group must allow the traffic.

• Private subnet resources can still initiate outbound connections (to download packages, call external APIs) via NAT Gateway. NAT is outbound-only, it does not allow inbound connections from the internet.

• The local route (e.g., 10.0.0.0/16 → local) is automatically added and cannot be removed. This allows all resources within the VPC to communicate with each other.
• Multiple subnets can share one route table, but a subnet can only be associated with one route table at a time.

• An EIP that is not associated with a running instance costs $0.005/hr (~$3.65/month). AWS charges for idle EIPs to discourage hoarding the scarce IPv4 space. Release any EIP you are not using.
• An EIP associated with a running instance is free. Stop the instance or disassociate the EIP and the charge starts immediately.
• Each AWS account has a default limit of 5 EIPs per region. This is a soft limit; request an increase via Service Quotas if needed.

• Flow Logs do not capture the packet payload, only metadata. They also do not capture traffic to/from the VPC DNS resolver (the base CIDR + 2 address) or DHCP traffic.
• Flow Logs require an IAM role with permission to publish to CloudWatch Logs (or an S3 bucket policy if logging to S3). Forgetting to create this role is the most common setup failure.
• A REJECT entry points to a Security Group or NACL rule. An ACCEPT entry followed by no response points to an application-level issue (the packet arrived but the process did not respond). Knowing this distinction saves significant debugging time.

• One VPC can only have one IGW attached at a time.
• The IGW itself has no cost, charges come from data transfer through it.

• NAT Gateway costs money: ~$0.045/hr per gateway plus data processing charges. For a learning account, delete it when not in use to avoid surprise bills.
• For high availability, deploy one NAT Gateway per AZ and have each AZ's private subnets route to their local NAT Gateway. Routing all AZs through one NAT Gateway means an AZ failure takes down all outbound internet access.

• VPC Peering is non-transitive. If VPC A peers with B, and B peers with C, A cannot reach C through B. You must create a direct A ↔ C peering connection.
• Peered VPCs cannot have overlapping CIDR blocks. Plan your IP address space before peering: two VPCs both using 10.0.0.0/16 cannot be peered.
• The peering connection itself does nothing. You must add routes in both VPCs' route tables: a route in VPC A pointing to VPC B's CIDR via the peering connection ID, and a matching route in VPC B. Forgetting the return route is the most common misconfiguration.
• Cross-region peering charges data transfer at ~$0.01-$0.02/GB depending on regions. Within a region, traffic between AZs across peered VPCs is billed at the standard cross-AZ rate ($0.01/GB each direction).

• TGW costs $0.05/hr per attachment plus $0.02/GB of data processed. For a learning account with 2-3 VPCs, VPC Peering is cheaper. Use TGW when the peering mesh is the actual problem.
• TGW route tables are separate from VPC route tables. A common misconfiguration: the TGW route table has the correct entries but the VPC's route table does not have a route pointing toward the TGW attachment ID. Both sides must be configured.
• TGW is regional. Cross-region connectivity requires inter-region TGW peering (a separate attachment between two TGWs), which adds latency and data transfer cost.

• Access Keys are long-term credentials. If they are leaked (e.g., committed to GitHub), rotate them immediately and assume they were used maliciously.
• Never embed Access Keys in source code or Docker images. Use IAM Roles for EC2/Lambda and environment variables or a secrets manager for external systems.

• A user can belong to multiple groups. Their effective permissions are the union of all policies from all groups they belong to, plus any policies attached directly to the user.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "ec2.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}

• Temporary credentials from a role automatically expire (15 minutes to 12 hours) and are automatically rotated. A leaked set of temporary credentials has a built-in time limit, far safer than long-term Access Keys.
• If you SSH into an EC2 and run aws s3 ls without configuring credentials, it works because the Instance Profile role is used automatically. This is the intended behaviour.

• An explicit Deny always overrides an Allow, regardless of which policy it comes from. If any policy denies an action, that action is denied even if another policy allows it.
• AWS Managed Policies (like AmazonS3ReadOnlyAccess) are maintained by AWS. Prefer these for standard roles. Avoid using AdministratorAccess for anything other than your admin user.

• Temporary credentials from STS expire (15 min-12 hrs depending on role session duration). The AWS SDK's credential provider chain automatically refreshes them before expiry, your application code never needs to handle credential rotation.
• The External ID is an optional parameter used in cross-account scenarios to prevent the "confused deputy" problem, where a malicious actor tricks a trusted service into assuming a role on their behalf. When a third party (e.g., a vendor) assumes a role in your account, always require an External ID in the trust policy.

• Add the S3 Gateway Endpoint to every VPC immediately. It is free and private-subnet instances accessing S3 without it pay unnecessary NAT costs.
• Interface Endpoints use private DNS: when enabled, the SDK resolves secretsmanager.us-east-1.amazonaws.com to a private IP inside your VPC. Confirm enableDnsHostnames and enableDnsSupport are both enabled on the VPC, or the private DNS override won't work.
• When building a PrivateLink Endpoint Service, the NLB must preserve the client source IP if your backend needs it, because PrivateLink terminates the connection at the NLB.

• Security Groups can reference other Security Groups as the traffic source. Example: allow port 5432 from "App-SG", any EC2 in that Security Group can connect to RDS, regardless of its IP address. This is better than using IP-based rules.
• Security Group changes take effect immediately. No instance restart needed.

• Stateless means: if you allow inbound port 80, you must also allow outbound ephemeral ports 1024-65535 so the HTTP response can leave the subnet. Forgetting this is a classic misconfiguration.
• The default NACL allows all traffic in both directions. If you create a custom NACL, it denies everything by default, you must add explicit allow rules.
• In practice, most teams keep the default NACLs and rely on Security Groups. Know the difference for interviews but don't over-engineer NACLs in practice.

• RDS connection fails → check the Security Group, not IAM. Connecting to PostgreSQL on port 5432 is a TCP connection. IAM does not control TCP connections. The RDS Security Group must allow port 5432 inbound from the EC2's Security Group. If IAM is not involved, adding an IAM policy does nothing.
• S3 call fails → check IAM, not the Security Group. Calling S3 is an HTTPS request to an AWS API endpoint. Security Groups allow all outbound traffic by default, so port 443 is already open. The EC2 Instance Profile needs s3:PutObject (or equivalent) on the bucket. If the Security Group is not involved, adding an SG rule does nothing.
• Some resources require both simultaneously. EC2 in a private subnet calling Secrets Manager needs: (1) the Instance Profile with secretsmanager:GetSecretValue (IAM), and (2) either a NAT Gateway or a Secrets Manager VPC Interface Endpoint so the API request can leave the subnet (network). A missing IAM policy gives you an immediate AccessDeniedException. A missing network path gives you a connection timeout after ~20 seconds. Different error, different fix, both required.
• IAM Database Authentication is not a substitute for Security Groups on RDS. This opt-in feature lets your app authenticate to RDS using a short-lived IAM token instead of a password. It requires rds-db:connect in the IAM policy. The TCP connection to port 5432 still must be permitted by the Security Group. IAM handles the authentication step; the Security Group handles whether the packet can arrive at the database at all. Both are required regardless of which authentication method you use.
• The quick mental test: is your code opening a socket to a specific port on a specific host? That is a Security Group question. Is your code calling an AWS service API (S3, SQS, DynamoDB, Secrets Manager, etc.)? That is an IAM question.

• Instance type naming: t3.micro means t = burstable family, 3 = generation, micro = size. Common families: t (burstable), m (general), c (compute-optimised), r (memory-optimised).
• Burstable instances (t-series) earn CPU credits at idle and spend them under load. If credits run out, CPU is throttled to a baseline rate. Fine for dev; watch out for sustained load in production.
• Stop ≠ Terminate. Stopped: instance paused, EBS persists, compute billing stops (EBS still billed). Terminated: instance deleted, EBS deleted by default. You cannot un-terminate.

• AMIs are region-specific. An AMI from us-east-1 cannot be directly used in eu-west-1, you must copy it first.
• Use Amazon Linux 2023 (AL2023), not Amazon Linux 2, which reaches end of support on June 30, 2026. AL2023 uses dnf as its package manager (though yum still works as an alias).

• By default the root EBS volume is deleted when the instance is terminated. Change "Delete on Termination" to false if you want to keep the volume after termination.
• EBS volumes are AZ-specific. An EBS volume in us-east-1a cannot be attached to an instance in us-east-1b.
• You are billed for EBS storage even while the instance is stopped.

• Set permissions immediately after download: chmod 400 key.pem. Without this, SSH refuses to use the key: "WARNING: UNPROTECTED PRIVATE KEY FILE".
• Default usernames: ec2-user (Amazon Linux), ubuntu (Ubuntu), admin (Debian). Not root.
• Connect command: ssh -i /path/to/key.pem ec2-user@<public-ip>

• Every public IPv4 address bills $0.005/hr, including Elastic IPs attached to running instances (a 2024 change; older tutorials still claim public IPs are free). Accounts created before July 15, 2025 keep 750 hours/month of free public IPv4 under the legacy Free Tier; newer accounts pay from the first hour, with credits absorbing it. Check aws.amazon.com/free for current terms, and release any public IPv4 addresses you are not actively using.

• Install Java 21: sudo dnf install java-21-amazon-corretto -y
• Install Git: sudo dnf install git -y
• Verify Java: java -version → should show Corretto-21.x.x
• Copy JAR from laptop: scp -i key.pem app.jar ec2-user@<ip>:~/
• Check disk space: df -h · Check memory: free -h · Check processes: ps aux | grep java

[Unit]
Description=Spring Boot Application
After=network.target

[Service]
User=ec2-user
WorkingDirectory=/home/ec2-user
ExecStart=/usr/bin/java \
  -Xms128m -Xmx256m \
  -XX:+UseG1GC \
  -Dspring.application.name=springapp \
  -jar /home/ec2-user/app.jar
StandardOutput=append:/home/ec2-user/app.log
StandardError=append:/home/ec2-user/app.log
SuccessExitStatus=143
Restart=on-failure
RestartSec=10
TimeoutStopSec=60

[Install]
WantedBy=multi-user.target

• -Xmx is not total JVM memory. The JVM's actual RSS is heap + metaspace (~80-150MB) + thread stacks (~1MB per thread) + GC overhead. With -Xmx512m, expect 650-750MB of physical RAM consumed. On a t3.micro (1GB total, ~916MB available), that leaves under 200MB for the OS, CloudWatch agent (~54MB), and SSM agent (~36MB). The OOM killer hits the JVM, the ALB health check fails, the CW agent stops collecting, and SSM loses connectivity. One oversized heap flag causes all of it. Use -Xmx256m on 1GB instances.
• Configure swap on any instance running multiple agents. Even 512MB prevents hard OOM kills on memory spikes: sudo dd if=/dev/zero of=/swapfile bs=128M count=4 && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile. Persist it by adding /swapfile swap swap defaults 0 0 to /etc/fstab.
• If OOM kills the JVM and you lose both SSH and SSM access, the EC2 system log is still readable from the console without any instance access: EC2 Console → select instance → Actions → Monitor and troubleshoot → Get system log. Look for oom-kill lines. This works even when the instance appears completely unresponsive.

• Never open SSH (port 22) to 0.0.0.0/0. Use "My IP" in the console. Bots scan for open SSH ports constantly.
• Opening port 8080 to 0.0.0.0/0 is acceptable for initial testing, but the tasks below will have you lock this down behind an ALB, which is how production always looks.

• An ALB requires subnets in at least two AZs. Even with one EC2 target, you must specify two subnets in different AZs when creating the ALB.
• Health check path matters. If your Spring Boot app has Spring Actuator enabled, use /actuator/health. If the path returns anything other than HTTP 2xx, targets are marked unhealthy and receive no traffic, your app appears "down" even though it's running.
• The ALB has its own Security Group. Pattern: ALB SG allows 80/443 inbound from internet. EC2 SG allows 8080 inbound only from the ALB SG, not from 0.0.0.0/0. This is the correct production security model.
• Startup timing trap: If Spring Boot takes 15-20 seconds to start and the ALB health check fires every 5 seconds with an unhealthy threshold of 2, the target is flagged unhealthy during boot, and once an ASG manages the fleet, the instance gets replaced mid-startup. Set the health check interval and thresholds so (interval × unhealthy threshold) exceeds your app's startup time, and pair it with the ASG health check grace period. Monitor ALB target health status when a deployment stalls.
• Graceful shutdown: Set ALB's deregistration delay (default 300 seconds) appropriately, it's how long the ALB continues sending in-flight requests to a de-registering target before cutting it off. Your Spring Boot app must have server.shutdown=graceful and a timeout that matches. Without this, rolling deploys cause 502s for active users.

• ACM certificates are free when attached to ALB, CloudFront, or API Gateway. The free certificates cannot be exported for use on your own EC2 server (ACM does sell exportable public certificates as a separate paid option, added in June 2025).
• DNS validation is preferred: it adds a CNAME record to your domain's DNS and auto-renews without human action. Email validation requires you to click a link before the cert expires.
• If you do not own a domain yet, skip the HTTPS task, use the ALB's default DNS name (my-alb-123.us-east-1.elb.amazonaws.com) with HTTP for now. The ALB + Target Group pattern is what matters here.

• Health check type defaults to EC2 (instance is running, not stopped or terminated). After attaching a Target Group, change the health check type to ELB. Without it, the ASG has no visibility into your application's health. A crashed Spring Boot app inside a running instance will never be replaced.
• Use Launch Templates, not Launch Configurations. Launch Configurations are deprecated and AWS does not accept new ones; a tutorial that teaches them is outdated. Launch Templates support versioning, mixed instance types, Graviton (Arm64) instances, and Spot capacity.
• Health check grace period: the ASG ignores ELB health check failures during this window after an instance launches. Set it to your app's worst-case startup time, typically 90-120 seconds for Spring Boot. Too short means the ASG terminates an instance that is still starting. Too long means a genuinely broken instance stays in the fleet.
• Max is the only cost guard. The ASG itself has no cost, but it will launch EC2 instances without hesitation. A misconfigured scaling policy with no upper bound can create hundreds of instances. Set a realistic max and enable AWS Billing alerts before enabling auto scaling in any account.
• Termination policy: the default first picks the AZ with the most instances (to rebalance), then terminates the instance launched from the oldest Launch Template version. Terminating the oldest instance specifically is the opt-in OldestInstance policy. Either way, scale-in terminates instances carrying live traffic: if your Spring Boot app holds in-memory session state, those sessions are silently dropped. Externalise session state to ElastiCache (covered in Phase 10) before enabling scale-in.
• On a scale-out event, new instances do not contribute to the fleet's CloudWatch metrics until their instance warm-up period expires. Set warm-up equal to your app's startup time. Without this, slow-starting JVMs drag down the average CPU metric and prematurely cancel further scale-out before the fleet has absorbed the load.

• Target tracking creates CloudWatch alarms automatically. Do not delete these alarms manually. The ASG owns them. Deleting them silently disables scaling without any error or warning.
• CPU is not always the right metric. A Spring Boot app doing heavy database I/O or waiting on downstream services can be saturated while CPU stays below 10%. Consider publishing custom CloudWatch metrics from your app: active HTTP connections, JVM thread pool queue depth, request latency p99.
• ALB RequestCountPerTarget is a built-in predefined metric available directly in target tracking configuration. For HTTP services, scaling on request count per instance is often more responsive than CPU because it tracks actual traffic load, not processing cost.
• Cooldown period (default 300 seconds) applies to simple scaling policies: after a scaling activity, the ASG waits before evaluating another. Step scaling does not pause between activities; it relies on instance warm-up instead, and target tracking manages its own stabilisation. If you do use simple scaling on a bursty workload, reduce the cooldown to 60-120 seconds or you will lag behind traffic spikes.

aws autoscaling complete-lifecycle-action \
  --lifecycle-hook-name <hook-name> \
  --auto-scaling-group-name <asg-name> \
  --lifecycle-action-result CONTINUE \
  --instance-id <instance-id>

• The default heartbeat timeout is 3600 seconds (one hour). An instance stuck in Terminating:Wait continues to accrue EC2 billing for the full duration. Set the timeout to your expected drain time plus a margin, typically 60-180 seconds for Spring Boot apps.
• If your hook handler is not deployed or crashes, every instance termination hangs for the full timeout. Test hook handling explicitly in staging before enabling in production.
• Spring Boot's server.shutdown=graceful (configured in the systemd tasks above) handles in-flight HTTP requests at the application layer. A lifecycle hook operates at the ASG layer, one level up. Both are needed for a fully clean shutdown under load: the hook pauses instance termination, Spring Boot drains its request threads, then the hook signals CONTINUE.

• If enableDnsHostnames is off on your VPC, Interface Endpoints private DNS override silently stops working. The SDK resolves the service endpoint to the public IP instead of the ENI in your subnet. This is the root cause when an Interface Endpoint exists but traffic still routes to the internet.
• A Private Hosted Zone must be explicitly associated with a VPC before its records resolve inside that VPC. Creating the zone is not enough.
• Route 53 Resolver Endpoints cost $0.125/hr per endpoint plus $0.40 per million queries. For small deployments, consider this cost before adding endpoints purely for convenience.

• KMS key policies are mandatory. Unlike S3 bucket policies, a KMS key without an explicit key policy grants access to nobody. The default key policy AWS creates includes a root-account delegation statement; if you write a custom key policy from scratch and omit that statement, you can permanently lock yourself out of the key.
• For same-account access, identity-based policies alone are usually sufficient. Resource-based policies become necessary for cross-account access and for expressing conditions tied to the resource itself (e.g., deny delete unless MFA is present).
• When both policy types exist for the same account, AWS grants access if either allows it and no explicit Deny overrides it. Across accounts, both the identity-based policy on the caller and the resource-based policy on the target must allow the action.

• IAM Identity Center requires AWS Organizations to be enabled. It is a multi-account service by design.
• Permission Sets become IAM roles in each assigned account under the naming pattern AWSReservedSSO_PermissionSetName_xxx. Do not modify these roles directly in IAM; they are managed by Identity Center and changes get overwritten.
• aws sso login credentials are short-lived (typically 1-12 hours). Automated scripts that run unattended cannot use SSO credentials; they need a dedicated service account role with long-lived credentials stored in Secrets Manager or a dedicated IAM role assumed by the automation service (e.g., a Lambda execution role or a GitHub Actions OIDC role).

• SCPs do not apply to the management account. The management account has full access regardless of SCP configuration. This is why the management account should hold no workloads: it is a privileged account for organization administration only.
• Organizations consolidates billing: all charges across member accounts roll up to the management account, and volume discounts aggregate across the whole org. This benefit alone is worth enabling Organizations even for small teams.
• AWS service-linked roles are exempt from SCPs when they need specific permissions to function on your behalf. This prevents SCPs from accidentally breaking managed services like EKS or RDS.

• Event History shows only management events for the last 90 days. For S3 object access logs or records older than 90 days, you must create a Trail. The first copy of management events per region is free; additional copies and data events are billed.
• CloudTrail logs land in S3 with up to a 15-minute delay. For near-real-time alerting on specific API calls (e.g., alert when anyone calls DeleteTrail), send the Trail to CloudWatch Logs and create a metric filter and alarm.
• Protect the Trail's S3 bucket. An attacker who compromises your account will attempt to delete the CloudTrail logs to cover their tracks. Enable S3 Object Lock on the Trail bucket and restrict cloudtrail:StopLogging and cloudtrail:DeleteTrail with an SCP or a deny policy on all non-admin roles.

• Deleting a CMK has a mandatory 7-30 day waiting period and is irreversible. Any data encrypted with that key and not separately backed up becomes permanently unrecoverable after deletion. AWS cannot help you recover it. Before scheduling deletion, run a last-access audit in the KMS console.
• Cross-account EBS snapshot sharing requires the snapshot to be encrypted with a CMK whose key policy explicitly grants access to the target account. Snapshots encrypted with AWS Managed Keys cannot be shared cross-account.
• KMS is rate-limited; the default in most regions is 30,000 requests/second across all cryptographic operations. High-throughput applications that call GenerateDataKey per database record can approach this limit. The correct pattern is to cache the plaintext data key in memory and call KMS only when the cache expires or the key is rotated.

• When Secrets Manager rotates a secret, the value changes in place. Your application must re-fetch the credential on the next connection attempt rather than caching it indefinitely at startup. The AWS Secrets Manager JDBC driver wrapper for Java handles this automatically for database connections.
• SSM Parameter Store standard tier is throttled at 40 transactions/second by default (soft limit). Applications that fetch many parameters at startup can hit this. Use GetParametersByPath to batch fetches, or use the advanced tier for higher throughput.
• Use Secrets Manager for anything that rotates or is a credential (database passwords, API keys, TLS private keys). Use SSM Parameter Store for non-secret configuration (feature flags, service URLs, environment-specific config values). Mixing them per-secret type is fine; mixing them arbitrarily per-project creates confusion during incident response.

• db.t3.micro is the right size for this phase. On accounts created before July 15, 2025 it is free tier eligible (750 hrs/month for MySQL, PostgreSQL, or MariaDB, not Aurora). Newer accounts pay by the hour (single-digit cents; check the RDS pricing page), drawn from the Free Tier credits.
• RDS must go in private subnets. It should never have a public IP. Access it only from within the VPC via Security Groups.
• The RDS endpoint looks like: my-db.abc123.us-east-1.rds.amazonaws.com. Use this as your JDBC host.

• Use standard RDS PostgreSQL for this phase's hands-on work: a db.t3.micro is the cheapest managed PostgreSQL you can run, and Aurora costs noticeably more at this scale.
• Aurora has its own endpoint types: cluster endpoint (always points to the primary, use for writes), reader endpoint (load-balances across read replicas). Your app should use both if running read replicas.

• RDS Proxy costs ~$0.015 per vCPU-hour of the target database's capacity, with a 2-vCPU minimum for small instances; check the RDS Proxy pricing page. Worth it in Lambda-heavy architectures; overkill for a single EC2 app with HikariCP.
• RDS Proxy requires IAM authentication or Secrets Manager, it does not accept direct password credentials from the JDBC URL. This is a security improvement: no passwords in JDBC strings.

• The standby in a Multi-AZ instance deployment is not readable. It exists purely for failover. You cannot send read traffic to it to reduce load on the primary, that is what Read Replicas are for. A Multi-AZ DB cluster deployment is different: it runs two readable standbys in other AZs and typically fails over in around 35 seconds.
• Multi-AZ roughly doubles the RDS cost. In dev/learning environments, leave it off.

• Replication is asynchronous, replicas may lag behind the primary by seconds. Never use a replica for anything requiring up-to-date data (e.g., immediately after a write).
• Multi-AZ ≠ Read Replica. Multi-AZ is for availability (automatic failover). Read Replicas are for scalability (read offloading). These are separate features and can both be enabled simultaneously.

• Automated backups are deleted when you delete the RDS instance (unless you take a final snapshot). Manual snapshots survive instance deletion.
• Restoring a snapshot creates a new RDS instance, it does not restore in place. Plan for the new endpoint in your app config.

SecretsManagerClient client = SecretsManagerClient.create();
GetSecretValueResponse r = client.getSecretValue(
    GetSecretValueRequest.builder()
        .secretId("prod/myapp/db")
        .build());
// r.secretString() → JSON: {"username":"dbadmin","password":"..."}
// Parse with ObjectMapper, inject into DataSource

• Secrets Manager costs $0.40 per secret per month + $0.05 per 10,000 API calls. Negligible at learning scale.
• IAM permission required: secretsmanager:GetSecretValue on the specific secret ARN, not on *.
• For local development, use environment variables or a local .env file (excluded from Git). Never configure Secrets Manager with hardcoded access keys locally, that defeats the purpose.

spring.datasource.url=jdbc:postgresql://my-db.abc123.us-east-1.rds.amazonaws.com:5432/mydb?ssl=true&sslmode=require
spring.datasource.username=dbadmin
spring.datasource.password=${DB_PASSWORD}
# Pool sizing formula: (core_count × 2) + effective_spindle_count
# For db.t3.micro (2 vCPU): (2 × 2) + 1 = 5; 10 is a safe ceiling for light workloads
spring.datasource.hikari.maximum-pool-size=10
spring.datasource.hikari.minimum-idle=2
spring.datasource.hikari.connection-timeout=3000
spring.datasource.hikari.idle-timeout=600000

• A connection timeout means the Security Group (or routing), an authentication failure means credentials. Diagnose by error type before changing anything.
• HikariCP defaults to 10 connections per app instance, and a db.t3.micro allows roughly 100 total. A few app instances plus stray psql sessions reach that ceiling sooner than you expect; size pools deliberately.

• Bucket names are globally unique across all AWS accounts worldwide. If someone else already has my-app-uploads, you cannot use it. Use a prefix like your company name or account ID.
• Bucket names must be 3-63 characters, lowercase, no underscores.
• Buckets are created in a specific region. Data stays in that region unless you explicitly replicate it.

• Once enabled, versioning cannot be fully disabled, only suspended. Suspended means new uploads no longer create versions, but existing versions are preserved.
• Old versions accumulate storage costs. Pair versioning with a lifecycle rule to expire non-current versions after N days.

• Standard (~$0.023/GB): frequent access, lowest latency. Default.
• Standard-IA (Infrequent Access, ~$0.0125/GB + retrieval fee): for files accessed less than once a month.
• Glacier Instant Retrieval (~$0.004/GB): archival, millisecond retrieval.
• Glacier Flexible Retrieval (~$0.0036/GB): archival, minutes to hours retrieval.
• Glacier Deep Archive (~$0.00099/GB): cheapest, 12-hour retrieval. For compliance archives.
• Intelligent-Tiering: automatically moves objects between tiers based on access patterns. Monitoring fee per object.

• IA classes have a minimum storage duration: 30 days for Standard-IA, 90 days for Glacier Instant Retrieval and Glacier Flexible Retrieval, 180 days for Glacier Deep Archive. Objects deleted before the minimum are still billed for the full duration.

• Objects must sit in Standard for 30 days before a lifecycle rule can transition them to Standard-IA or One Zone-IA. A "transition after 7 days" rule to IA is silently impossible.
• Transitions are billed per request (~$0.01 per 1,000 to IA, more to Glacier tiers). Transitioning millions of tiny objects can cost more than it saves; for small objects, expiry is often the better rule.

• Presigned URLs inherit the permissions of the IAM identity that generated them. If your EC2 IAM Role has s3:GetObject on the bucket, it can generate presigned GET URLs. Without the permission, the URL will fail.
• Maximum expiry: 7 days (604800 seconds), and only when the URL is signed with long-lived IAM user credentials. A URL signed with temporary credentials (an EC2 instance role, ECS task role, Lambda execution role) dies when that session expires, typically within 6 hours, regardless of the expiry you request. If you need long-lived URLs, sign them with a dedicated IAM user.

• Bucket policies still control which IAM identities (your EC2 role, Lambda function) can access objects, Block Public Access only prevents public (unauthenticated) access.
• Never use bucket ACLs, they are a legacy mechanism. Use bucket policies and IAM policies instead.

• S3 Event Notifications are at-least-once, a Lambda or SQS consumer may receive the same event twice on rare occasions. Make your processing idempotent (check if the object was already processed before doing work).
• Use an SQS queue between S3 and Lambda rather than invoking Lambda directly, SQS buffers events during Lambda throttling and provides a DLQ for failed processing.

S3TransferManager manager = S3TransferManager.create();
FileUpload upload = manager.uploadFile(b -> b
    .putObjectRequest(r -> r.bucket("my-bucket").key("large-file.csv"))
    .source(Path.of("/tmp/large-file.csv")));
upload.completionFuture().join();

• Incomplete multipart uploads accumulate storage charges for the uploaded parts. Add a lifecycle rule: "Abort incomplete multipart uploads after 7 days."

<!-- Use the AWS SDK BOM, no individual version needed -->
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>software.amazon.awssdk</groupId>
      <artifactId>bom</artifactId>
      <version>2.46.7</version>  <!-- current at time of writing; check central.sonatype.com/artifact/software.amazon.awssdk/bom -->
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependency>
  <groupId>software.amazon.awssdk</groupId>
  <artifactId>s3</artifactId>
  <!-- version managed by BOM -->
</dependency>

<!-- pom.xml -->
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-cloudwatch2</artifactId>
</dependency>

// MetricsConfig.java
// Spring Boot core does not auto-configure CloudWatch export;
// auto-configuration for it lives in Spring Cloud AWS (io.awspring.cloud).
// This manual bean needs no extra starter and works on any Boot version.
@Configuration
public class MetricsConfig {

    @Bean
    public CloudWatchConfig cloudWatchConfig() {
        return key -> switch (key) {
            case "cloudwatch.namespace" -> "MyApp";
            case "cloudwatch.step"      -> "PT1M";
            default                     -> null;
        };
    }

    @Bean
    public CloudWatchMeterRegistry cloudWatchMeterRegistry(
            CloudWatchConfig config) {
        return new CloudWatchMeterRegistry(
                config, Clock.SYSTEM, CloudWatchAsyncClient.create());
    }
}

// Imports needed:
// import io.micrometer.cloudwatch2.CloudWatchConfig;
// import io.micrometer.cloudwatch2.CloudWatchMeterRegistry;
// import io.micrometer.core.instrument.Clock;   // NOT java.time.Clock
// import software.amazon.awssdk.services.cloudwatch.CloudWatchAsyncClient;

• CloudWatch export is not part of Spring Boot core. Unlike Prometheus or OTLP, CloudWatch is absent from Boot's auto-configured registry list, and management.cloudwatch.metrics.export.* properties take effect only when Spring Cloud AWS (io.awspring.cloud) is on the classpath providing the auto-configuration. Without it, the properties are silently ignored: the app starts fine, metrics never appear. Either add Spring Cloud AWS or wire the manual @Bean above. Diagnose via /actuator/beans (expose it first: management.endpoints.web.exposure.include=health,beans): if there is no cloudWatchMeterRegistry bean, no exporter is wired.
• Micrometer pushes metrics in batches (default every 1 minute). CloudWatch charges per custom metric per month (~$0.30). With many fine-grained tags, costs escalate. Filter which metrics are exported.
• Tag all metrics with the service name for multi-service dashboards. In the manual bean approach, add this to cloudWatchMeterRegistry: registry.config().commonTags("application", appName);
• cloudwatch:ListMetrics denied in CloudTrail or the IAM Policy Simulator looks alarming when debugging metric publishing. It is harmless: the Micrometer CloudWatch exporter only calls PutMetricData and never calls ListMetrics. Do not add cloudwatch:ListMetrics to the role policy based on this denial. It will not fix anything.

<!-- Recommended: OpenTelemetry starter (ADOT-compatible) -->
<dependency>
  <groupId>io.opentelemetry.instrumentation</groupId>
  <artifactId>opentelemetry-spring-boot-starter</artifactId>
  <version>2.16.0</version>  <!-- current at time of writing; check Maven Central -->
</dependency>
<!-- Configure OTEL_EXPORTER_OTLP_ENDPOINT to point at the ADOT collector. -->
<!-- The ADOT collector runs as a sidecar or daemon and forwards traces to X-Ray. -->

• X-Ray uses sampling by default (5% of requests, or 1 req/sec minimum). Don't panic when you can't find a specific trace, it may not have been sampled. Increase the rate in the X-Ray sampling rules for debugging.
• Instrument with OpenTelemetry (ADOT). The X-Ray SDKs and daemon entered maintenance mode in February 2026 and AWS directs all new tracing work to OpenTelemetry; ADOT is AWS's supported distribution of it, and it stays vendor-neutral (the same instrumentation feeds Jaeger or Grafana Tempo). Use the X-Ray SDK only in codebases already wired to it.
• CloudWatch Application Signals is AWS's lead APM offering: ADOT auto-instrumentation plus the CloudWatch agent's OTLP endpoint, surfacing service health, latency SLOs, and dependency maps without custom metrics code. If you are running EKS or ECS, check whether Application Signals covers your requirements before wiring a custom pipeline.

# application.properties (Spring Boot 3.4+)
# alternative format: logstash
logging.structured.format.console=ecs

# For older Boot versions, use logstash-logback-encoder:
# logback-spring.xml with <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>

// In a servlet filter (Java), set the correlation ID per request:
MDC.put("traceId", UUID.randomUUID().toString());
// X-Ray or OTel auto-injects trace IDs into MDC when tracing is enabled

• logging.structured.format.console formats the console appender only. The Phase 2 systemd setup pipes console output to app.log, so it covers you there; if you later add a dedicated file appender, also set logging.structured.format.file.
• Put the traceId into MDC in a filter that runs before everything else (highest precedence), or the first log lines of each request will be missing the field. MDC values must be Strings.

• CPUUtilization, NetworkIn, NetworkOut, DiskReadOps, DiskWriteOps
• NOT included by default: memory usage, disk space used. These require the CloudWatch Agent.
• Detailed monitoring (1-minute intervals): costs extra. Enable per instance in the console.

sudo dnf install amazon-cloudwatch-agent -y
# Use the wizard to generate config:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
# Start the agent:
sudo systemctl enable amazon-cloudwatch-agent
sudo systemctl start amazon-cloudwatch-agent

• A single data point in the CWAgent namespace followed by silence means the agent started once and was then killed. On small instances with a Spring Boot app, the cause is almost always OOM: the kernel kills the largest process (the JVM), which cascades to kill the CW agent as well. Before restarting the agent, confirm there is enough free memory: free -h. If RAM is exhausted, fixing the JVM heap size (or resizing the instance) must come first, otherwise the agent will be killed again immediately.
• The EC2 system log is accessible from the EC2 Console without SSH or SSM: select the instance → Actions → Monitor and troubleshoot → Get system log. It shows kernel OOM kills (oom-kill: task=java), boot errors, and SSM authentication failures. Use it as the first diagnostic step when you cannot connect to an instance.

• Log retention defaults to Never Expire. This accumulates storage costs indefinitely. Always set a retention policy (e.g., 30 or 90 days) on every log group.
• CloudWatch Logs Insights: SQL-like query language for searching and analysing logs. Example: filter @message like /ERROR/ | stats count(*) by bin(5m)

• Set the evaluation period and datapoints carefully. CPU > 80% for 1 out of 1 datapoints at 1-minute intervals will fire on transient spikes. 2 out of 3 datapoints reduces false positives.
• An alarm in INSUFFICIENT_DATA state means CloudWatch is not receiving metric data, which itself can indicate a problem (agent stopped, instance down).

• Dashboards cost $3/month per dashboard (first 3 are free). Not a budget concern at learning scale. Verify current pricing at aws.amazon.com/cloudwatch/pricing, the free tier and rates change.

• Container filesystem is ephemeral. Anything written inside the container is lost when the container stops. Use volumes or external storage (S3, RDS) for persistence.
• Each layer in a Docker image is cached. Put infrequently-changing layers (base image, dependencies) early in the Dockerfile, and your app code last, this speeds up rebuilds significantly.

FROM eclipse-temurin:21-jre-alpine
WORKDIR /app
# Maven names the JAR target/<artifactId>-<version>.jar; the wildcard handles it
COPY target/*.jar app.jar
ENTRYPOINT ["java", "-jar", "app.jar"]

• Use a JRE image (not JDK) in production containers, JDK includes the compiler which you don't need at runtime and adds unnecessary image size.
• Alpine images use musl libc instead of glibc. This is fine for most Spring Boot apps but can cause issues with certain JNI-based libraries or specific DNS configurations. If you hit unexplained runtime issues (DNS failures, TLS oddities), switch to eclipse-temurin:21-jre (Debian slim) as a baseline to rule out musl compatibility.
• Spring Boot's layered JAR feature creates separate Docker layers for dependencies and app code, making rebuilds faster. The spring-boot-maven-plugin repackage goal produces a layered JAR by default; you only add <layers> plugin config to customise the layer list. To benefit, use a multi-stage Dockerfile that extracts the layers (the JAR ships a jarmode for this) and copies dependencies before app code. Worth exploring after basics are solid.

# Authenticate Docker to ECR
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin \
  123456789.dkr.ecr.us-east-1.amazonaws.com

# Tag and push
docker tag my-app:latest 123456789.dkr.ecr.us-east-1.amazonaws.com/my-app:latest
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/my-app:latest

• There is no SSH into Fargate containers. Debugging uses CloudWatch Logs (stdout/stderr from the container) and ECS Exec (an optional feature that provides a shell into a running task).
• ECS tasks are not persistent. A stopped task is replaced by a new one. Any state must be in external storage (RDS, S3, ElastiCache).

• Fargate CPU units: 256 (0.25 vCPU), 512 (0.5 vCPU), 1024 (1 vCPU), up to 16384 (16 vCPU)
• Memory: 512 MB minimum, must be compatible with chosen CPU. E.g., 512 CPU → 1-4 GB memory; 1024 CPU → 2-8 GB memory.
• Spring Boot typically needs at least 512 MB; 1 GB is comfortable for a small service.

• Two roles, constantly confused. The Task Execution Role is used by the ECS agent itself: pulling the ECR image and writing logs to CloudWatch. The Task Role is what your application code receives through the SDK credential chain, the container equivalent of the EC2 instance profile. s3:GetObject for your app belongs on the Task Role, never on the execution role. An app that gets AccessDenied despite "the role having the permission" usually has it on the wrong one of the two.

on:
  push:
    branches: [main]
permissions:
  id-token: write   # required for OIDC; without it role assumption fails
  contents: read
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: arn:aws:iam::ACCOUNT:role/github-deploy-role
        aws-region: us-east-1
    - name: Login to ECR
      id: ecr-login
      uses: aws-actions/amazon-ecr-login@v2
    - name: Build and push
      env:
        ECR_URI: ${{ steps.ecr-login.outputs.registry }}/my-app
      run: |
        IMAGE_TAG=${{ github.sha }}
        docker build -t $ECR_URI:$IMAGE_TAG -t $ECR_URI:latest .
        docker push $ECR_URI:$IMAGE_TAG
        docker push $ECR_URI:latest
    - name: Deploy to ECS
      run: |
        aws ecs update-service --cluster my-ecs-cluster \
          --service my-service --force-new-deployment

• Enable ECR image scanning on push: aws ecr put-image-scanning-configuration --repository-name my-app --image-scanning-configuration scanOnPush=true. It flags known CVEs in your base image layers. Block deployments on CRITICAL findings using the scan results in your pipeline.

• Fargate has a slightly slower cold start than EC2 launch type (~10-30 seconds to start a new task). Not usually a problem for long-running services, but relevant for burst scaling.
• Fargate does not support privileged containers or GPU workloads. For GPUs on ECS, use the EC2 launch type with GPU instances (e.g., g6); the same constraint pushes EKS users to GPU node groups.

• Pods have dynamic IPs. Never hardcode a pod's IP, use a Service to get a stable endpoint.
• When a pod crashes, Kubernetes restarts it (according to the restartPolicy). It does not move to a new IP or name unless it is rescheduled to a different node.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      serviceAccountName: my-app-sa   # for IRSA (AWS permissions)
      containers:
      - name: my-app
        image: 123456789.dkr.ecr.us-east-1.amazonaws.com/my-app:latest
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"
        # readinessProbe: gate traffic until app is ready
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          initialDelaySeconds: 20
          periodSeconds: 5
        # livenessProbe: restart if app is deadlocked
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 40
          periodSeconds: 10

• A LoadBalancer Service in EKS creates an AWS load balancer that bills ~$0.0225/hr (~$16/month) plus capacity units, per Service. For HTTP routing to multiple services, use an Ingress with the AWS Load Balancer Controller instead, it creates a single ALB shared across services.

• Kubernetes Secrets are not encrypted at rest by default in etcd. For production, integrate with AWS Secrets Manager using the External Secrets Operator, or use the AWS Secrets and Configuration Provider (ASCP) with the Secrets Store CSI Driver.
• Base64 is encoding, not encryption. Anyone with kubectl access can decode a Secret trivially: kubectl get secret my-secret -o jsonpath='{.data.password}' | base64 -d

• An Ingress without ingressClassName: alb and the alb.ingress.kubernetes.io/scheme: internet-facing annotation provisions nothing, with no error. Check the controller's logs (kubectl logs -n kube-system deploy/aws-load-balancer-controller) when no ALB appears.
• Default target type is instance (NodePort-based). Set alb.ingress.kubernetes.io/target-type: ip to route straight to pod IPs; it is also the only mode that works for Fargate pods.

helm install my-app ./my-chart --values values-prod.yaml
helm upgrade my-app ./my-chart --set image.tag=v1.2.3
helm rollback my-app 1          # rollback to revision 1
helm list                       # list installed releases

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

• HPA requires the Kubernetes Metrics Server to be running for CPU-based scaling. On EKS, install it first: kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml. Without Metrics Server, kubectl get hpa shows <unknown>/60% in the TARGETS column and HPA never triggers a scale event.

• EKS control plane: $0.10/hour (~$72/month) per cluster while it runs a Kubernetes version in standard support. A cluster left on a version past standard support bills $0.60/hour extended support, six times the price; keep the version current. Plus the EC2 cost of worker nodes. Delete the cluster when not actively learning, this is the biggest cost trap in this roadmap.
• Alternatively, use EKS with Fargate profiles to avoid managing EC2 worker nodes entirely. Pay per pod CPU/memory instead.

# Step 1: Associate OIDC provider with the cluster (once per cluster)
eksctl utils associate-iam-oidc-provider \
  --cluster my-cluster --region us-east-1 --approve

# Step 2: Create an IAM service account in your cluster namespace
eksctl create iamserviceaccount \
  --cluster my-cluster \
  --namespace default \
  --name my-app-sa \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess \
  --approve

# Step 3: Reference it in your Deployment manifest
# spec.template.spec.serviceAccountName: my-app-sa

• The OIDC provider must be associated with the cluster before creating IRSA service accounts. Without this, the IAM role trust policy cannot validate the pod's identity token.
• The AWS SDK inside the pod automatically uses IRSA credentials. On EC2, it uses the Instance Profile. Both flow through the same credential provider chain, your Spring Boot code works identically in both environments without any changes.
• EKS Pod Identity is AWS's recommended approach for new clusters: no OIDC provider association, simpler setup. It requires the EKS Pod Identity Agent add-on running on the cluster, then: eksctl create podidentityassociation --cluster my-cluster --namespace default --service-account-name my-app-sa --role-arn arn:aws:iam::ACCOUNT:role/my-role. Use IRSA for existing clusters or tools whose docs still lead with it; this phase uses IRSA because the Load Balancer Controller setup does.
• When a team member gets "Unauthorized" on kubectl: grant access via EKS Access Entries (AWS Console → EKS cluster → Access → Create access entry), the API-driven approach and the default on current clusters. Clusters still using the legacy aws-auth ConfigMap: kubectl edit configmap aws-auth -n kube-system and add their IAM principal ARN.

# Create cluster (takes ~15-20 minutes)
eksctl create cluster --name my-cluster --region us-east-1 \
  --nodegroup-name workers --node-type t3.small --nodes 2

kubectl get nodes          # verify workers are Ready
kubectl get pods -A        # all pods in all namespaces
kubectl apply -f app.yaml  # deploy from manifest
kubectl get svc            # list services and their external IPs
kubectl logs pod-name -f   # tail pod logs
kubectl rollout undo deployment/my-app  # rollback

• Default concurrent execution limit: 1000 per account per region (soft limit, can be increased). One Lambda function hitting the limit can throttle all other functions in the account. Use reserved concurrency to isolate critical functions.
• Scaling rate: each function scales by up to 1,000 concurrent executions every 10 seconds, independently per function, until the account concurrency limit is reached. (Older tutorials describe a 3,000-burst-plus-500-per-minute regional model; that model is gone.) A function can still throttle during an extreme spike, watch the Throttles metric.
• Lambda is not suitable for long-running processes (batch jobs, video encoding) or anything requiring persistent local state.
• Default timeout is 3 seconds. A Java cold start alone can exceed that, so the first invocation of a freshly deployed function can fail before your code runs. Set the timeout to 30 seconds for API-backed functions (maximum 15 minutes) before debugging anything else.

public class UserHandler
    implements RequestHandler<APIGatewayV2HTTPEvent,
                               APIGatewayV2HTTPResponse> {

  // Initialised ONCE per container, reused on warm starts
  private final DynamoDbClient dynamo = DynamoDbClient.create();

  @Override
  public APIGatewayV2HTTPResponse handleRequest(
      APIGatewayV2HTTPEvent event, Context ctx) {

    String userId = event.getPathParameters().get("id");
    // query dynamo, build response...
    return APIGatewayV2HTTPResponse.builder()
        .withStatusCode(200)
        .withBody("{\"userId\":\"" + userId + "\"}")
        .build();
  }
}

• Do not use Spring Boot as a Lambda runtime. Spring's application context initialises in 3-8 seconds, that's a multi-second cold start on every scale-out event. Use plain Java for Lambda. If your team requires Spring, use Spring Cloud Function + SnapStart, which snapshots the initialised context to cut cold starts to ~1 second.
• Static initialisers (static { ... }) run during the init phase, counted as cold start time. Move heavyweight setup into instance variables (initialised once) or lazy-init them on first invocation.

• Lambda SnapStart (available for Java 11, Java 17, Java 21, Python 3.12+, and .NET 8+): AWS takes a snapshot of the initialised execution environment after the init phase, then restores from it on cold starts. Reduces Java cold starts from seconds to ~1 second. It applies only to published versions and aliases, never to $LATEST. Included at no extra cost for Java; Python and .NET bill for snapshot cache storage and per restore. Most impactful for functions initialising Spring Cloud Function or other heavyweight frameworks.
• Provisioned Concurrency: keeps N containers pre-warmed and ready. Eliminates cold starts completely but you pay for the reserved capacity even when idle. Use for latency-sensitive APIs.
• Avoid heavy Spring Boot in Lambda, Spring's context initialisation is slow. Use plain Java, Quarkus (with native compilation), or Micronaut instead. If you need Spring, use Spring Cloud Function with GraalVM native image.

• @Idempotent requires a DynamoDB table to store idempotency keys. The annotation handles read-before-write and conditional write automatically, but you must provision the table and pass its name via the IDEMPOTENCY_TABLE environment variable (or configure the DynamoDBPersistenceStore explicitly).
• @Metrics uses Embedded Metric Format (EMF): metrics are written to CloudWatch Logs as structured JSON and CloudWatch extracts them asynchronously. This avoids the PutMetricData API call cost, but metrics can lag behind log delivery by a few seconds.

• Function URLs do not support custom domains natively. If you need a vanity domain (api.yourdomain.com), put CloudFront in front of the Function URL or use API Gateway.
• The auth mode NONE makes the URL publicly accessible with no authentication. Only use this for genuinely public endpoints or behind a trusted network boundary.

• The right memory setting is not always the minimum. Use AWS Lambda Power Tuning (an open source Step Functions state machine) to find the optimal memory-to-cost ratio for your function.
• Ephemeral storage (/tmp): 512 MB by default, configurable up to 10 GB. Files in /tmp persist between warm invocations on the same container, useful for caching, but don't assume it's always available.

• SQS: Lambda polls the queue and invokes your handler with a batch of messages (default batch size 10; standard queues go up to 10,000 if you also set a batching window, FIFO queues cap at 10). Built-in retry (message returns to queue on failure), DLQ support.
• SNS: Fan-out pattern, SNS topic → multiple SQS queues → each with its own Lambda. Decouples producer from many consumers.
• Kinesis Data Streams: Lambda processes records in order within a shard. Use for real-time streaming (log processing, IoT data, event sourcing). Each shard gets one concurrent Lambda invocation.
• DynamoDB Streams: Every write to a DynamoDB table appears as an event. Lambda can react to inserts/updates/deletes for change data capture (CDC), cross-table sync, or event sourcing.

• Idempotency is mandatory for SQS and Kinesis consumers. Lambda may invoke your handler more than once for the same message (at-least-once delivery). Your handler must produce the same result on re-processing, use a DynamoDB conditional write or idempotency key table to detect duplicates.
• SQS batch failures: If one message in a batch fails, the entire batch is retried by default (returning all messages to the queue, including the ones that succeeded). Use ReportBatchItemFailures in your Lambda response to return only the failed message IDs for retry.
• A DLQ (Dead Letter Queue) captures messages that fail after all retries. Always configure one, without it, poisoned messages cycle forever and block the queue.
• Lambda Destinations vs DLQ: these are different mechanisms. A DLQ is configured on the SQS queue (or the Lambda function for async invocations) and captures raw failed messages. Lambda Destinations (configured on the Lambda function) route the full invocation record, including the request payload, response, and error details, to SQS, SNS, EventBridge, or another Lambda. Use Destinations when you need the original event payload plus failure context for debugging; use DLQ when the SQS queue owns retry and archiving.
• Event Source Mapping Filters: for SQS, Kinesis, and DynamoDB Streams triggers, you can add a filter pattern so Lambda is only invoked when messages match specific criteria (e.g., only process events where eventType = "ORDER_PLACED"). Filters evaluate before your handler runs. Unmatched messages are discarded (SQS), or skipped (Kinesis/DynamoDB Streams). Configure via "Additional settings" in the trigger or in the FilterCriteria field of the EventSourceMapping API.

• HTTP API: JWT authorisers, Lambda proxy integration, CORS, per-route throttling, lower cost (~$1/million requests). Missing: caching, API keys/usage plans, AWS WAF integration.
• REST API: all HTTP API features plus response caching, usage plans, API keys, resource policies, AWS WAF, mock integrations, request/response transformations. Costs ~$3.50/million requests.
• Default recommendation: use HTTP API unless you need a specific REST API feature.

• The API Gateway endpoint is public by default. Anyone can call it. Add an authoriser (JWT/Cognito for HTTP API, Lambda authoriser for custom logic) or at minimum use an API key to prevent open access in production.
• Payload format version mismatch: HTTP API uses payload format version 2.0 by default; REST API uses 1.0. The Java event types are different. For HTTP API (format 2.0), use APIGatewayV2HTTPEvent (from aws-lambda-java-events): path is at event.getRawPath(). For REST API (format 1.0), use APIGatewayProxyRequestEvent: path is at event.getPath(). Using the wrong event class silently produces null path parameters and empty bodies.

• Choose a high-cardinality partition key (e.g., user ID, order ID, values that are unique and evenly distributed). A low-cardinality partition key (e.g., status = "active"/"inactive") creates hot partitions that throttle performance.
• DynamoDB has no joins. Design your table access patterns upfront, think in queries, not entities. One table design (putting multiple entity types in one table) is the advanced but optimal pattern.
• Scan vs Query: a Scan reads every item in the table and applies a filter after reading. You are billed for every item read, regardless of how many match. On a 10-million-item table, a Scan that returns 1 item still bills for reading 10 million items. Always use Query with a partition key condition. Only use Scan for backfill/migration scripts or admin tooling that runs once.
• Item size limit: 400 KB. If an item exceeds 400 KB (e.g., storing user-uploaded content inline), the PutItem call fails. The standard pattern is to store large payloads in S3 and keep only the S3 object key in DynamoDB.
• Cost at this phase's scale is effectively zero: 25 GB of storage and a provisioned-capacity allowance sit in AWS's always-free offerings, and on-demand requests at learning volume cost cents. Verify current terms at aws.amazon.com/free before building cost assumptions.

• 1 RCU = 1 strongly consistent read per second for items up to 4 KB, or 2 eventually consistent reads per second. 1 WCU = 1 write per second for items up to 1 KB.
• Provisioned mode bills for the capacity you reserve whether you use it or not; under-utilised provisioned tables often cost more than On-Demand would. Compare against your real utilisation before switching.

• Transactions consume 2× the capacity units (reads and writes are doubled). Budget accordingly.
• TransactionConflictException occurs when two concurrent transactions touch the same item. Implement exponential backoff and retry in your Lambda handler.

• GSI reads are eventually consistent, the GSI may lag behind the base table by milliseconds to seconds. Never use a GSI for reads that must reflect the very latest write.

• LSIs share the provisioned throughput of the base table. Under heavy read load against an LSI, it competes directly with base table reads. GSIs have their own independent read/write capacity.
• LSIs have a 10 GB per-partition limit (the combined size of all items sharing the same partition key across the base table and all LSIs). Exceeding this cap causes ItemCollectionSizeLimitExceededException.

• Changing access patterns after launch means adding GSIs or migrating data. Single-table design front-loads the design effort: map every query your application will ever run before writing a line of code.
• Do not start with single-table design on a new project unless the team has DynamoDB experience. Multiple tables is a valid starting point and easier to reason about during initial development.

• TTL deletion is not instantaneous: items can persist up to 48 hours past their expiry timestamp. If your application logic reads the table and must not see expired items, add a filter in your query: FilterExpression: "#exp > :now" using the current epoch time.
• TTL deletions appear in DynamoDB Streams as REMOVE events with a userIdentity of {"type": "Service", "principalId": "dynamodb.amazonaws.com"}. If your Lambda processes the stream, filter these out unless you want to react to expirations.

terraform {
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 6.0" }  # check registry.terraform.io for latest
  }
}

provider "aws" {
  region = var.region
}

variable "region" {
  default = "us-east-1"
}

resource "aws_s3_bucket" "my_bucket" {
  bucket = "my-app-uploads-${var.region}"
}

resource "aws_s3_bucket_versioning" "my_bucket" {
  bucket = aws_s3_bucket.my_bucket.id
  versioning_configuration { status = "Enabled" }
}

• terraform init: download providers and modules, initialise the backend. Run once per project or after backend/provider changes.
• terraform plan: show what will be created, changed, or destroyed. Reads current state and compares to your code. Always review this output before applying.
• terraform apply: execute the plan. Prompts for confirmation. Writes results to state file.
• terraform destroy: destroys all resources managed by the current state. Use carefully.

• Never run terraform apply in CI without first reviewing the plan output. A -replace or -destroy flag in the wrong hands deletes production resources.
• terraform import: bring an existing manually-created resource under Terraform management without recreating it.

terraform {
  backend "s3" {
    bucket       = "my-terraform-state"
    key          = "prod/terraform.tfstate"
    region       = "us-east-1"
    use_lockfile = true
    encrypt      = true
  }
}

• State files can contain sensitive values (RDS passwords, private keys). Enable S3 bucket encryption and restrict access with bucket policies.
• Never edit the state file manually. Use terraform state mv, terraform state rm, or terraform import for state surgery.

• Avoid deeply nested modules, they make debugging hard. Flat module structures are more readable.
• Pin module versions (version = "5.5.0") to avoid unexpected breaking changes from upstream updates.

public class MyStack extends Stack {
  public MyStack(Construct scope, String id) {
    super(scope, id);
    Bucket bucket = Bucket.Builder.create(this, "MyBucket")
        .versioned(true)
        .blockPublicAccess(BlockPublicAccess.BLOCK_ALL)
        .build();
  }
}
// Deploy: cdk synth  →  cdk deploy

• CDK synthesises to CloudFormation, you get Change Sets and rollback protection for free. The downside: CDK's generated templates are verbose and hard to read directly.
• CDK is opinionated and sets non-obvious defaults that vary by construct and version (removal policies, encryption settings, generated IAM policies). Always run cdk synth and inspect the generated CloudFormation template before deploying, it reveals exactly what CDK will create. Use cdk diff to preview changes before every deploy.

AWSTemplateFormatVersion: '2010-09-09'
Description: My application stack
Parameters:
  InstanceType:
    Type: String
    Default: t3.micro
  LatestAmiId:    # resolves the current AL2023 AMI at deploy time; no hardcoded AMI ID
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64
Resources:          # required
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      ImageId: !Ref LatestAmiId
Outputs:
  InstanceId:
    Value: !Ref MyInstance

• If a stack update fails, CloudFormation automatically rolls back to the previous state. This is generally helpful but means you must investigate why the update failed before retrying.
• Deleting a stack deletes all resources in it (by default). Use DeletionPolicy: Retain on critical resources like RDS and S3 to protect them.

• Manual changes to CloudFormation-managed resources cause drift. If you update a resource manually and then run CloudFormation, it may overwrite your manual change or fail with a conflict. Always make changes through CloudFormation or Terraform, never manually in the console for IaC-managed resources.

• ElastiCache Redis does not support all Redis commands. In cluster mode, commands that operate across multiple keys (e.g., MGET with keys in different slots, KEYS *, Lua scripts touching multiple keys) are restricted or behave differently. Always test your Redis command usage against a cluster-mode instance before going to production.
• Memcached on ElastiCache does not support Multi-AZ automatic failover or replication. If a Memcached node fails, all data in that node is lost.
• ElastiCache Serverless exists for Valkey and Redis: no node sizing, pay per GB stored and per request. Convenient for spiky workloads; for a steady learning workload, a cache.t3.micro node is the cheaper option.

• Spring Boot's default Redis client is Lettuce (not Jedis). Lettuce handles cluster topology changes automatically, when a primary fails and a replica is promoted, Lettuce refreshes the cluster view and reconnects without application restart. Jedis requires more manual cluster configuration.
• You cannot switch a cluster between cluster mode disabled and enabled without recreating it. Plan your topology before provisioning.
• In cluster mode enabled, all keys in a MULTI/EXEC transaction or a Lua script must hash to the same slot. Use hash tags (e.g., {user:123}:cart and {user:123}:profile) to force related keys into the same slot.

• TTL is mandatory. Without a TTL, cached entries never expire. If the underlying data changes in the database and the cache is not explicitly evicted, the application will serve stale data indefinitely. Always set a TTL appropriate to your data's rate of change.
• Cache stampede (thundering herd): when a popular key's TTL expires (or on a cold start), dozens or hundreds of simultaneous requests all miss the cache and fire concurrent queries to the database, which can overwhelm it. Mitigate with: (1) adding random jitter to TTLs (e.g., 55-65 seconds instead of exactly 60) so hot keys don't expire simultaneously; (2) probabilistic early expiry (PER), start refreshing a key before it expires based on a probability function; (3) a single-flight/request-coalescing pattern where only one thread fetches from the DB and others wait for the result.

• Write penalty: every database write now requires a second write to Redis. Writes are slower, and the write path now has a dependency on Redis availability.
• Cache pollution: data written to the cache may never be read, wasting memory. For write-heavy datasets with unpredictable read access patterns (e.g., audit logs, bulk imports), write-through wastes cache space.
• Hybrid approach: use write-through for hot, frequently-read data (user sessions, current inventory counts for top products) and cache-aside for cold or unpredictable data. Many production systems combine both patterns by data category.

# Fetch host from environment variable injected via Secrets Manager / EKS secret
spring.data.redis.host=${REDIS_HOST}
spring.data.redis.port=6379
spring.data.redis.password=${REDIS_AUTH_TOKEN}
spring.data.redis.ssl.enabled=true   # required when TLS is enabled on the cluster
# Connection pool (Lettuce)
spring.data.redis.lettuce.pool.max-active=20
spring.data.redis.lettuce.pool.max-idle=10
spring.data.redis.lettuce.pool.min-idle=2

@Configuration
@EnableCaching
public class CacheConfig {

    @Bean
    public RedisCacheConfiguration redisCacheConfiguration() {
        // Generic serializer: embeds the class name in the JSON so reads
        // deserialize back to InventoryItem, not LinkedHashMap
        GenericJackson2JsonRedisSerializer serializer =
            new GenericJackson2JsonRedisSerializer();

        return RedisCacheConfiguration.defaultCacheConfig()
            .entryTtl(Duration.ofSeconds(60))
            .serializeValuesWith(
                RedisSerializationContext.SerializationPair.fromSerializer(serializer));
    }
}

// Service layer
@Service
public class InventoryService {

    @Cacheable(value = "inventory", key = "#id")
    public InventoryItem getInventoryById(Long id) {
        return inventoryRepository.findById(id).orElseThrow();
    }

    @CacheEvict(value = "inventory", key = "#id")
    public void updateInventory(Long id, InventoryItem item) {
        inventoryRepository.save(item);
    }
}

• Default JDK serialization is unreadable. Out of the box, Spring's RedisTemplate uses JDK serialization, cached values appear as binary gibberish in redis-cli and are tied to the exact class structure. Configure a JSON serializer, and pick the right one: Jackson2JsonRedisSerializer typed to Object.class writes JSON fine but reads it back as LinkedHashMap, producing a ClassCastException on the first cache hit. GenericJackson2JsonRedisSerializer embeds the class name in the JSON, so values round-trip to the original type; that is the one to use with @Cacheable on mixed value types.
• When using GenericJackson2JsonRedisSerializer, the serialized JSON includes the fully qualified class name. This means renaming or moving a class will break deserialization of existing cached entries. Plan cache key versioning or use flushdb on deployment when class structure changes.

# ElastiCache Security Group, inbound rule
Type: Custom TCP
Port: 6379
Source: [Application Security Group ID]   # NOT 0.0.0.0/0, NOT the VPC CIDR

# Spring Boot: use rediss:// (double-s) when TLS is enabled
spring.data.redis.url=rediss://:${REDIS_AUTH_TOKEN}@${REDIS_HOST}:6379

• ElastiCache supports IAM authentication on Redis 7.0+ and Valkey clusters, the preferred approach for new clusters: no static password to store or rotate. It has prerequisites: TLS must be enabled and you authenticate as an RBAC user, with short-lived (15-minute) IAM tokens the client library refreshes. Where IAM auth does not fit, use the AUTH token: store it in Secrets Manager and rotate via aws elasticache modify-replication-group --auth-token <new-token>.
• In-transit encryption and at-rest encryption cannot be enabled on an existing cluster, they must be enabled at creation time. Plan for this before provisioning your first cluster.
• Spring's rediss:// URL (double s) enables TLS. Using redis:// against a TLS-enabled cluster will result in a connection error that can be difficult to diagnose.

• Evictions > 0 for 1 datapoint: any eviction means Redis is memory-constrained and is silently invalidating your cache. Action: upgrade the node type or reduce the TTL of lower-priority keys.
• CurrConnections approaching maxclients: maxclients is 65000 on ElastiCache Redis and Valkey nodes. New connections are refused when the limit is hit, causing application errors, and steady growth toward it almost always means a connection leak (Lettuce connection pool misconfiguration), not real load.
• ReplicationLag > 1 second: reads from replicas may return stale data older than 1 second. Investigate write throughput, the replica may be falling behind on replication.

• Counters only go up, never reset them. If you need rate, use rate() in PromQL. Never use a Gauge for something that only increases.
• Histograms require pre-defined buckets. If your p99 latency exceeds your highest bucket value, histogram_quantile() will return +Inf. Configure buckets that cover your expected latency range.

• The chart creates ~50 default Kubernetes alert rules. On first install they may fire immediately, for example, the "Watchdog" alert is intentional and proves alerting is working end-to-end.
• Review and tune the default rules for your cluster size. Some rules (e.g., "KubeMemoryOvercommit") require adjustment for small dev clusters.

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

# application.properties
management.endpoints.web.exposure.include=health,info,metrics,prometheus

• Histogram buckets are opt-in. By default Spring exports only _count and _sum for http.server.requests: no _bucket series exist, and histogram_quantile() silently returns nothing. Enable them with management.metrics.distribution.percentiles-histogram.http.server.requests=true, or define explicit SLO buckets: management.metrics.distribution.slo.http.server.requests=50ms,100ms,200ms,500ms,1s,2s,5s. If your p99 exceeds the highest bucket, the quantile clamps to that bucket's value.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: default
  labels:
    release: monitoring   # must match your Helm release name; see gotcha below
spec:
  selector:
    matchLabels:
      app: my-app         # matches the labels on your Phase 8 Service
  endpoints:
    - port: http          # the Service port NAME, not a number
      path: /actuator/prometheus
      interval: 30s

• The release label is required. kube-prometheus-stack configures Prometheus to select only ServiceMonitors labelled with the Helm release name (release: monitoring if you installed with helm install monitoring ...). A ServiceMonitor without it is silently ignored: no error, no target.
• The ServiceMonitor must be in a namespace Prometheus watches (serviceMonitorNamespaceSelector: {} watches all). Another silent failure mode: the ServiceMonitor exists but Prometheus never picks it up.
• The label selector on the ServiceMonitor must match the labels on your Kubernetes Service object, not the Pod labels. And port: refers to the Service port's name; if your Service defines the port without a name, add name: http to it.

• Request rate: rate(http_server_requests_seconds_count{job="my-app"}[5m])
• Error rate: rate(http_server_requests_seconds_count{status=~"5.."}[5m]) / rate(http_server_requests_seconds_count[5m])
• p99 latency: histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))
• JVM heap used: jvm_memory_used_bytes{area="heap"}
• HikariCP pool usage: hikaricp_connections_active / hikaricp_connections_max

• rate() requires a counter, not a gauge. Using rate() on a gauge gives nonsense. Use deriv() or delta() for gauges instead.
• histogram_quantile() requires the _bucket metric, not _count or _sum. Always use rate() on the bucket series inside the function.

• Grafana stores dashboards in a SQLite database on an emptyDir volume by default, all custom dashboards are lost when the pod restarts. Fix this with one of: (a) configure a PersistentVolumeClaim for Grafana storage, or (b) provision dashboards as Kubernetes ConfigMaps with grafana.sidecar.dashboards.enabled: true in the Helm values, GitOps-friendly, dashboards live in Git as JSON files and survive pod restarts automatically.

• AlertManager has "inhibition", a critical alert can suppress related warning alerts for the same component. Configure inhibition rules before going to production or one outage may page you 40 times for the same root cause.
• Test your alerting pipeline end-to-end with the "Watchdog" alert, it fires continuously and proves your full chain (Prometheus → AlertManager → receiver) works.

• Prometheus requires cluster resources and operational care, StatefulSet storage, backup, capacity planning. CloudWatch is serverless. Factor operational overhead into the decision.
• You can send metrics to both by adding both micrometer-registry-prometheus and micrometer-registry-cloudwatch2, Micrometer fans out to all configured registries simultaneously.

• GitOps does not mean putting secrets in Git. Use External Secrets Operator or Sealed Secrets to store encrypted secret references in Git while the actual values live in AWS Secrets Manager. Committing a plaintext Secret manifest is a critical security failure.
• GitOps replaces cluster state management, not the entire pipeline. GitHub Actions still builds your image and pushes to ECR, GitOps takes over at the point of deploying to the cluster.

• ArgoCD itself is a workload in your cluster, if the cluster is down, ArgoCD is down. This is expected: Kubernetes keeps running whatever was last applied regardless of ArgoCD's state. Don't confuse an ArgoCD outage with an inability to serve traffic.
• ArgoCD stores application state in its own CRDs inside the cluster. Back up your ArgoCD Application resources or manage them via Git (App of Apps pattern) so they survive cluster recreation.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/my-org/manifests.git
    targetRevision: main
    path: apps/my-app
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      selfHeal: true
      prune: true   # delete resources removed from Git

• Start with Manual sync for production, Automatic for dev. With Automatic + selfHeal, a misguided kubectl edit will be silently reverted, which is correct GitOps behaviour but surprises engineers used to imperative workflows.
• prune: true tells ArgoCD to delete cluster resources that no longer exist in Git. Without it, removed manifest files leave orphaned resources running in the cluster indefinitely.

• OutOfSync does not mean broken, it means Git and the cluster differ. OutOfSync + Degraded together means something is both different AND not working. The combination matters more than either status alone.
• ArgoCD's health status for a Deployment is Progressing (not Healthy) during a rolling update. Alert on Degraded, not on Progressing, otherwise every deploy triggers an alert.

• Image Updater needs IAM permissions to read from ECR, use IRSA so no credentials are embedded in the cluster. It also needs write access to your Git repo (via SSH key or GitHub App token stored in a Kubernetes Secret).
• Know what you are adopting: Image Updater is an argoproj-labs project whose own README advises against critical production workloads. It is well suited to this lab and to dev/staging automation; production teams often have CI commit the manifest bump instead, which achieves the same loop with tooling they already trust.
• Configure the update strategy carefully: digest (always pull latest digest of a tag, useful for latest tag workflows) vs semver (update to latest matching semver pattern, e.g., ~1.2 updates to any 1.2.x). For production, semver with explicit constraints is safer than tracking latest.

• Never auto-sync production. Always require a manual PR approval → merge → manual ArgoCD sync for prod. Dev and staging can be auto-synced. This is the most important governance rule in a GitOps workflow.
• Promotion is not ArgoCD's job, it is a Git operation. Promote by opening a PR that updates the prod path. ArgoCD detects the merged commit and either auto-syncs (staging) or waits for a manual sync (prod).

• Rollback in ArgoCD does not revert the Git repo, it deploys an older revision while the repo still has the newer commit. This creates a deliberate sync divergence (the cluster is behind Git). After rollback, fix the root cause, commit a fix forward, and sync, do not leave the cluster in a rolled-back state indefinitely.
• The correct production process: rollback ArgoCD to restore service → open a hotfix PR → merge → sync forward. Never leave the cluster and Git permanently diverged.

• If you accidentally commit a secret to Git, even briefly, rotate the secret immediately. Git history is permanent; the value is exposed even after deletion from the working tree.
• ESO creates a standard Kubernetes Secret from the external source. Configure ArgoCD to ignore ESO-managed secrets in its diff to avoid spurious OutOfSync status, ArgoCD did not create them, so they will always appear as drifted without this exclusion.