• Resources are region-scoped. An EC2 instance in us-east-1 has nothing to do with one in eu-west-1 — they're completely separate.
• us-east-1 (N. Virginia) is where AWS launches new services first. Good for learning, but for production choose a region close to your actual users.

• AZ names are randomly mapped per AWS account. Your us-east-1a may point to a different physical data centre than someone else's us-east-1a. This is intentional to spread load.
• Each subnet lives in exactly one AZ. To span AZs, create one subnet per AZ.

• Edge locations are not for compute. You cannot run EC2 instances or containers here. They only cache CloudFront content and resolve Route 53 DNS queries.

• Every AWS account has a default VPC (172.31.0.0/16) in every region. Do not delete it — getting it back requires an AWS support ticket.
• For production and learning, always create a custom VPC. The default VPC has public subnets by default, which is not ideal for production.
• Two VPCs that need to communicate via VPC Peering cannot have overlapping CIDR ranges. Plan CIDR blocks upfront.

• Being in a public subnet does not automatically expose a resource. The resource also needs a public IP address, and the Security Group must allow the traffic.

• Private subnet resources can still initiate outbound connections (to download packages, call external APIs) via NAT Gateway. NAT is outbound-only — it does not allow inbound connections from the internet.

• The local route (e.g., 10.0.0.0/16 → local) is automatically added and cannot be removed. This allows all resources within the VPC to communicate with each other.
• Multiple subnets can share one route table, but a subnet can only be associated with one route table at a time.

• One VPC can only have one IGW attached at a time.
• The IGW itself has no cost — charges come from data transfer through it.

• NAT Gateway costs money: ~$0.045/hr per gateway plus data processing charges. For a learning account, delete it when not in use to avoid surprise bills.
• For high availability, deploy one NAT Gateway per AZ and have each AZ's private subnets route to their local NAT Gateway. Routing all AZs through one NAT Gateway means an AZ failure takes down all outbound internet access.

• Access Keys are long-term credentials. If they are leaked (e.g., committed to GitHub), rotate them immediately and assume they were used maliciously.
• Never embed Access Keys in source code or Docker images. Use IAM Roles for EC2/Lambda and environment variables or a secrets manager for external systems.

• A user can belong to multiple groups. Their effective permissions are the union of all policies from all groups they belong to, plus any policies attached directly to the user.

• Temporary credentials from a role automatically expire (15 minutes to 12 hours) and are automatically rotated. A leaked set of temporary credentials has a built-in time limit — far safer than long-term Access Keys.
• If you SSH into an EC2 and run aws s3 ls without configuring credentials, it works because the Instance Profile role is used automatically. This is the intended behaviour.

• An explicit Deny always overrides an Allow, regardless of which policy it comes from. If any policy denies an action, that action is denied even if another policy allows it.
• AWS Managed Policies (like AmazonS3ReadOnlyAccess) are maintained by AWS. Prefer these for standard roles. Avoid using AdministratorAccess for anything other than your admin user.

• Temporary credentials from STS expire (15 min–12 hrs depending on role session duration). The AWS SDK's credential provider chain automatically refreshes them before expiry — your application code never needs to handle credential rotation.
• The External ID is an optional parameter used in cross-account scenarios to prevent the "confused deputy" problem — where a malicious actor tricks a trusted service into assuming a role on their behalf. When a third party (e.g., a vendor) assumes a role in your account, always require an External ID in the trust policy.

• Add S3 Gateway Endpoint to every VPC immediately — it is free and private-subnet instances accessing S3 without it pay unnecessary NAT costs.
• Interface Endpoints use DNS resolution: when enabled, the SDK resolves secretsmanager.us-east-1.amazonaws.com to a private IP inside your VPC instead of the public IP. Confirm enableDnsHostnames and enableDnsSupport are enabled on your VPC, or the private DNS won't resolve.

• Security Groups can reference other Security Groups as the traffic source. Example: allow port 5432 from "App-SG" — any EC2 in that Security Group can connect to RDS, regardless of its IP address. This is better than using IP-based rules.
• Security Group changes take effect immediately. No instance restart needed.

• Stateless means: if you allow inbound port 80, you must also allow outbound ephemeral ports 1024–65535 so the HTTP response can leave the subnet. Forgetting this is a classic misconfiguration.
• The default NACL allows all traffic in both directions. If you create a custom NACL, it denies everything by default — you must add explicit allow rules.
• In practice, most teams keep the default NACLs and rely on Security Groups. Know the difference for interviews but don't over-engineer NACLs in practice.

• Instance type naming: t3.micro — t = burstable family, 3 = generation, micro = size. Free tier: t3.micro (2 vCPU, 1 GB RAM, 750 hrs/month). Common families: t (burstable), m (general), c (compute-optimised), r (memory-optimised).
• Burstable instances (t-series) earn CPU credits at idle and spend them under load. If credits run out, CPU is throttled to a baseline rate. Fine for dev; watch out for sustained load in production.
• Stop ≠ Terminate. Stopped: instance paused, EBS persists, compute billing stops (EBS still billed). Terminated: instance deleted, EBS deleted by default. You cannot un-terminate.

• AMIs are region-specific. An AMI from us-east-1 cannot be directly used in eu-west-1 — you must copy it first.
• Use Amazon Linux 2023 (AL2023), not Amazon Linux 2 which is approaching end-of-life. AL2023 uses dnf as its package manager (though yum still works as an alias).

• By default the root EBS volume is deleted when the instance is terminated. Change "Delete on Termination" to false if you want to keep the volume after termination.
• EBS volumes are AZ-specific. An EBS volume in us-east-1a cannot be attached to an instance in us-east-1b.
• You are billed for EBS storage even while the instance is stopped.

• Set permissions immediately after download: chmod 400 key.pem. Without this, SSH refuses to use the key: "WARNING: UNPROTECTED PRIVATE KEY FILE".
• Default usernames: ec2-user (Amazon Linux), ubuntu (Ubuntu), admin (Debian). Not root.
• Connect command: ssh -i /path/to/key.pem ec2-user@<public-ip>

• Elastic IPs are free when associated with a running instance. You are charged ~$0.005/hr when the IP is allocated but not associated (i.e., you're holding an IP without using it). Release IPs you're not using.

• Install Java 17: sudo dnf install java-17-amazon-corretto -y
• Install Git: sudo dnf install git -y
• Verify Java: java -version → should show Corretto-17.x.x
• Copy JAR from laptop: scp -i key.pem app.jar ec2-user@<ip>:~/
• Check disk space: df -h · Check memory: free -h · Check processes: ps aux | grep java

[Unit]
Description=Spring Boot Application
After=network.target

[Service]
User=ec2-user
WorkingDirectory=/home/ec2-user
ExecStart=/usr/bin/java \
  -Xms512m -Xmx1g \
  -XX:+UseG1GC \
  -Dspring.application.name=springapp \
  -jar /home/ec2-user/app.jar
SuccessExitStatus=143
Restart=on-failure
RestartSec=10
TimeoutStopSec=60

[Install]
WantedBy=multi-user.target

• Never open SSH (port 22) to 0.0.0.0/0. Use "My IP" in the console. Bots scan for open SSH ports constantly.
• Opening port 8080 to 0.0.0.0/0 is acceptable for initial testing, but the tasks below will have you lock this down behind an ALB — which is how production always looks.

• An ALB requires subnets in at least two AZs. Even with one EC2 target, you must specify two subnets in different AZs when creating the ALB.
• Health check path matters. If your Spring Boot app has Spring Actuator enabled, use /actuator/health. If the path returns anything other than HTTP 2xx, targets are marked unhealthy and receive no traffic — your app appears "down" even though it's running.
• The ALB has its own Security Group. Pattern: ALB SG allows 80/443 inbound from internet. EC2 SG allows 8080 inbound only from the ALB SG — not from 0.0.0.0/0. This is the correct production security model.
• Startup timing trap: If Spring Boot takes 15–20 seconds to start and the ALB health check fires every 5 seconds with a threshold of 2 failures, the target is marked unhealthy during boot and receives no traffic — even though the app is fine. Set initialDelaySeconds (or ALB health check healthy threshold + interval) to exceed your app's startup time. Monitor ALB target health status when a deployment stalls.
• Graceful shutdown: Set ALB's deregistration delay (default 300 seconds) appropriately — it's how long the ALB continues sending in-flight requests to a de-registering target before cutting it off. Your Spring Boot app must have server.shutdown=graceful and a timeout that matches. Without this, rolling deploys cause 502s for active users.

• ACM certificates are free when attached to ALB, CloudFront, or API Gateway. You cannot export or download them for use on your own EC2 server.
• DNS validation is preferred: it adds a CNAME record to your domain's DNS and auto-renews without human action. Email validation requires you to click a link before the cert expires.
• If you do not own a domain yet, skip the HTTPS task — use the ALB's default DNS name (my-alb-123.us-east-1.elb.amazonaws.com) with HTTP for now. The ALB + Target Group pattern is what matters here.

• db.t3.micro is free tier eligible (750 hrs/month for MySQL, PostgreSQL, or MariaDB — not Aurora). Good enough for this phase.
• RDS must go in private subnets. It should never have a public IP. Access it only from within the VPC via Security Groups.
• The RDS endpoint looks like: my-db.abc123.us-east-1.rds.amazonaws.com. Use this as your JDBC host.

• Aurora is not free tier eligible. Standard RDS PostgreSQL is — use it for this phase's hands-on work.
• Aurora has its own endpoint types: cluster endpoint (always points to the primary, use for writes), reader endpoint (load-balances across read replicas). Your app should use both if running read replicas.

• RDS Proxy costs ~$0.015/vCPU/hour plus $0.001/GB of data transferred. Worth it in Lambda-heavy architectures; overkill for a single EC2 app with HikariCP.
• RDS Proxy requires IAM authentication or Secrets Manager — it does not accept direct password credentials from the JDBC URL. This actually improves security: no passwords in JDBC strings.

• The standby replica is not readable. It exists purely for failover. You cannot send read traffic to it to reduce load on the primary — that is what Read Replicas are for.
• Multi-AZ roughly doubles the RDS cost. In dev/learning environments, leave it off.

• Replication is asynchronous — replicas may lag behind the primary by seconds. Never use a replica for anything requiring up-to-date data (e.g., immediately after a write).
• Multi-AZ ≠ Read Replica. Multi-AZ is for availability (automatic failover). Read Replicas are for scalability (read offloading). These are separate features and can both be enabled simultaneously.

• Automated backups are deleted when you delete the RDS instance (unless you take a final snapshot). Manual snapshots survive instance deletion.
• Restoring a snapshot creates a new RDS instance — it does not restore in place. Plan for the new endpoint in your app config.

SecretsManagerClient client = SecretsManagerClient.create();
GetSecretValueResponse r = client.getSecretValue(
    GetSecretValueRequest.builder()
        .secretId("prod/myapp/db")
        .build());
// r.secretString() → JSON: {"username":"dbadmin","password":"..."}
// Parse with ObjectMapper, inject into DataSource

• Secrets Manager costs $0.40 per secret per month + $0.05 per 10,000 API calls. Negligible at learning scale.
• IAM permission required: secretsmanager:GetSecretValue on the specific secret ARN, not on *.
• For local development, use environment variables or a local .env file (excluded from Git). Never configure Secrets Manager with hardcoded access keys locally — that defeats the purpose.

spring.datasource.url=jdbc:postgresql://my-db.abc123.us-east-1.rds.amazonaws.com:5432/mydb?ssl=true&sslmode=require
spring.datasource.username=dbadmin
spring.datasource.password=${DB_PASSWORD}
# Pool sizing formula: (core_count × 2) + effective_spindle_count
# For db.t3.micro (2 vCPU): (2 × 2) + 1 ≈ 10 connections per app instance
spring.datasource.hikari.maximum-pool-size=10
spring.datasource.hikari.minimum-idle=2
spring.datasource.hikari.connection-timeout=3000
spring.datasource.hikari.idle-timeout=600000

• Bucket names are globally unique across all AWS accounts worldwide. If someone else already has my-app-uploads, you cannot use it. Use a prefix like your company name or account ID.
• Bucket names must be 3–63 characters, lowercase, no underscores.
• Buckets are created in a specific region. Data stays in that region unless you explicitly replicate it.

• Once enabled, versioning cannot be fully disabled — only suspended. Suspended means new uploads no longer create versions, but existing versions are preserved.
• Old versions accumulate storage costs. Pair versioning with a lifecycle rule to expire non-current versions after N days.

• Standard (~$0.023/GB): frequent access, lowest latency. Default.
• Standard-IA (Infrequent Access, ~$0.0125/GB + retrieval fee): for files accessed less than once a month.
• Glacier Instant Retrieval (~$0.004/GB): archival, millisecond retrieval.
• Glacier Flexible Retrieval (~$0.0036/GB): archival, minutes to hours retrieval.
• Glacier Deep Archive (~$0.00099/GB): cheapest, 12-hour retrieval. For compliance archives.
• Intelligent-Tiering: automatically moves objects between tiers based on access patterns. Monitoring fee per object.

• IA classes have a minimum storage duration (30 days for Standard-IA, 90 days for Glacier). Objects deleted before the minimum are still billed for the full duration.

• Presigned URLs inherit the permissions of the IAM identity that generated them. If your EC2 IAM Role has s3:GetObject on the bucket, it can generate presigned GET URLs. Without the permission, the URL will fail.
• Maximum expiry: 7 days (604800 seconds) for IAM user or role credentials. For temporary session credentials (e.g., from STS), the URL expires no later than when the session expires — even if you set a longer duration.

• Bucket policies still control which IAM identities (your EC2 role, Lambda function) can access objects — Block Public Access only prevents public (unauthenticated) access.
• Never use bucket ACLs — they are a legacy mechanism. Use bucket policies and IAM policies instead.

• S3 Event Notifications are at-least-once — a Lambda or SQS consumer may receive the same event twice on rare occasions. Make your processing idempotent (check if the object was already processed before doing work).
• Use an SQS queue between S3 and Lambda rather than invoking Lambda directly — SQS buffers events during Lambda throttling and provides a DLQ for failed processing.

S3TransferManager manager = S3TransferManager.create();
FileUpload upload = manager.uploadFile(b -> b
    .putObjectRequest(r -> r.bucket("my-bucket").key("large-file.csv"))
    .source(Path.of("/tmp/large-file.csv")));
upload.completionFuture().join();

• Incomplete multipart uploads accumulate storage charges for the uploaded parts. Add a lifecycle rule: "Abort incomplete multipart uploads after 7 days."

<dependency>
  <groupId>software.amazon.awssdk</groupId>
  <artifactId>s3</artifactId>
  <version>2.25.x</version>
</dependency>

<!-- pom.xml -->
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-cloudwatch2</artifactId>
</dependency>

# application.properties
management.cloudwatch.metrics.namespace=MyApp
management.cloudwatch.metrics.step=1m
management.endpoints.web.exposure.include=health,info,metrics,prometheus

• Micrometer pushes metrics in batches (default every 1 minute). CloudWatch charges per custom metric per month (~$0.30). With many fine-grained tags, costs can escalate — filter which metrics are exported.
• Use management.metrics.tags.application=${spring.application.name} to tag all metrics with the service name — essential for multi-service dashboards.

<!-- AWS X-Ray SDK -->
<dependency>
  <groupId>com.amazonaws</groupId>
  <artifactId>aws-xray-recorder-sdk-spring</artifactId>
</dependency>

# Alternatively (modern approach): AWS Distro for OpenTelemetry
# ADOT collector runs as a sidecar or agent, sends traces to X-Ray
# Add opentelemetry-spring-boot-starter + configure OTEL_EXPORTER_OTLP_ENDPOINT

• X-Ray uses sampling by default (5% of requests, or 1 req/sec minimum). Don't panic when you can't find a specific trace — it may not have been sampled. Increase the rate in the X-Ray sampling rules for debugging.
• OpenTelemetry (OTEL) with AWS Distro is the modern, vendor-neutral approach. X-Ray is AWS-specific. For teams planning multi-cloud or using Jaeger/Grafana Tempo, OTEL is the better investment.

# application.properties (Spring Boot 3.4+)
logging.structured.format.console=ecs  # or 'logstash'

# For older versions, use logstash-logback-encoder:
# logback-spring.xml with <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>

# Set correlation ID in a filter:
MDC.put("traceId", UUID.randomUUID().toString());
# X-Ray or OTEL auto-injects trace IDs into MDC

• CPUUtilization, NetworkIn, NetworkOut, DiskReadOps, DiskWriteOps
• NOT included by default: memory usage, disk space used. These require the CloudWatch Agent.
• Detailed monitoring (1-minute intervals): costs extra. Enable per instance in the console.

sudo dnf install amazon-cloudwatch-agent -y
# Use the wizard to generate config:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
# Start the agent:
sudo systemctl enable amazon-cloudwatch-agent
sudo systemctl start amazon-cloudwatch-agent

• Log retention defaults to Never Expire. This accumulates storage costs indefinitely. Always set a retention policy (e.g., 30 or 90 days) on every log group.
• CloudWatch Logs Insights: SQL-like query language for searching and analysing logs. Example: filter @message like /ERROR/ | stats count(*) by bin(5m)

• Set the evaluation period and datapoints carefully. CPU > 80% for 1 out of 1 datapoints at 1-minute intervals will fire on transient spikes. 2 out of 3 datapoints reduces false positives.
• An alarm in INSUFFICIENT_DATA state means CloudWatch is not receiving metric data — which itself can indicate a problem (agent stopped, instance down).

• Dashboards cost $3/month per dashboard (first 3 are free). Not a budget concern at learning scale.

• Container filesystem is ephemeral. Anything written inside the container is lost when the container stops. Use volumes or external storage (S3, RDS) for persistence.
• Each layer in a Docker image is cached. Put infrequently-changing layers (base image, dependencies) early in the Dockerfile, and your app code last — this speeds up rebuilds significantly.

FROM eclipse-temurin:17-jre-alpine
WORKDIR /app
COPY target/app.jar app.jar
ENTRYPOINT ["java", "-jar", "app.jar"]

• Use a JRE image (not JDK) in production containers — JDK includes the compiler which you don't need at runtime and adds unnecessary image size.
• Spring Boot's layered JAR feature (spring-boot:build-info) creates separate Docker layers for dependencies and app code, making rebuilds faster. Worth exploring after basics are solid.

# Authenticate Docker to ECR
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin \
  123456789.dkr.ecr.us-east-1.amazonaws.com

# Tag and push
docker tag my-app:latest 123456789.dkr.ecr.us-east-1.amazonaws.com/my-app:latest
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/my-app:latest

• There is no SSH into Fargate containers. Debugging uses CloudWatch Logs (stdout/stderr from the container) and ECS Exec (an optional feature that provides a shell into a running task).
• ECS tasks are not persistent. A stopped task is replaced by a new one. Any state must be in external storage (RDS, S3, ElastiCache).

• Fargate CPU units: 256 (0.25 vCPU), 512 (0.5 vCPU), 1024 (1 vCPU), up to 16384 (16 vCPU)
• Memory: 512 MB minimum, must be compatible with chosen CPU. E.g., 512 CPU → 1–2 GB memory.
• Spring Boot typically needs at least 512 MB; 1 GB is comfortable for a small service.

on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: arn:aws:iam::ACCOUNT:role/github-deploy-role
        aws-region: us-east-1
    - name: Login to ECR
      id: ecr-login
      uses: aws-actions/amazon-ecr-login@v2
    - name: Build and push
      run: |
        IMAGE_TAG=${{ github.sha }}
        docker build -t $ECR_URI:$IMAGE_TAG .
        docker push $ECR_URI:$IMAGE_TAG
    - name: Deploy to ECS
      run: |
        aws ecs update-service --cluster my-cluster \
          --service my-service --force-new-deployment

• Enable ECR image scanning on push: aws ecr put-image-scanning-configuration --repository-name my-app --image-scanning-configuration scanOnPush=true. It flags known CVEs in your base image layers. Block deployments on CRITICAL findings using the scan results in your pipeline.

• Fargate has a slightly slower cold start than EC2 launch type (~10–30 seconds to start a new task). Not usually a problem for long-running services, but relevant for burst scaling.
• Fargate does not support GPU workloads or privileged containers.

• Pods have dynamic IPs. Never hardcode a pod's IP — use a Service to get a stable endpoint.
• When a pod crashes, Kubernetes restarts it (according to the restartPolicy). It does not move to a new IP or name unless it is rescheduled to a different node.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      serviceAccountName: my-app-sa   # for IRSA (AWS permissions)
      containers:
      - name: my-app
        image: 123456789.dkr.ecr.us-east-1.amazonaws.com/my-app:latest
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"
        # readinessProbe: gate traffic until app is ready
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          initialDelaySeconds: 20
          periodSeconds: 5
        # livenessProbe: restart if app is deadlocked
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 40
          periodSeconds: 10

• A LoadBalancer Service in EKS creates an AWS NLB, which costs money. For HTTP routing to multiple services, use an Ingress with the AWS Load Balancer Controller instead — it creates a single ALB shared across services.

• Kubernetes Secrets are not encrypted at rest by default in etcd. For production, integrate with AWS Secrets Manager using the External Secrets Operator, or use the AWS Secrets and Configuration Provider (ASCP) with the Secrets Store CSI Driver.
• Base64 is encoding, not encryption. Anyone with kubectl access can decode a Secret trivially: kubectl get secret my-secret -o jsonpath='{.data.password}' | base64 -d

helm install my-app ./my-chart --values values-prod.yaml
helm upgrade my-app ./my-chart --set image.tag=v1.2.3
helm rollback my-app 1          # rollback to revision 1
helm list                       # list installed releases

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

• EKS control plane: $0.10/hour (~$72/month) per cluster. Plus the EC2 cost of worker nodes. Delete the cluster when not actively learning — this is the biggest cost trap in this roadmap.
• Alternatively, use EKS with Fargate profiles to avoid managing EC2 worker nodes entirely. Pay per pod CPU/memory instead.

# Step 1: Associate OIDC provider with the cluster (once per cluster)
eksctl utils associate-iam-oidc-provider \
  --cluster my-cluster --region us-east-1 --approve

# Step 2: Create an IAM service account in your cluster namespace
eksctl create iamserviceaccount \
  --cluster my-cluster \
  --namespace default \
  --name my-app-sa \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess \
  --approve

# Step 3: Reference it in your Deployment manifest
# spec.template.spec.serviceAccountName: my-app-sa

• The OIDC provider must be associated with the cluster before creating IRSA service accounts. Without this, the IAM role trust policy cannot validate the pod's identity token.
• The AWS SDK inside the pod automatically uses IRSA credentials. On EC2, it uses the Instance Profile. Both flow through the same credential provider chain — your Spring Boot code works identically in both environments without any changes.
• When a second team member tries to kubectl into your cluster and gets "Unauthorized", they need to be added to the aws-auth ConfigMap: kubectl edit configmap aws-auth -n kube-system and add their IAM user ARN.

# Create cluster (takes ~15-20 minutes)
eksctl create cluster --name my-cluster --region us-east-1 \
  --nodegroup-name workers --node-type t3.small --nodes 2

kubectl get nodes          # verify workers are Ready
kubectl get pods -A        # all pods in all namespaces
kubectl apply -f app.yaml  # deploy from manifest
kubectl get svc            # list services and their external IPs
kubectl logs pod-name -f   # tail pod logs
kubectl rollout undo deployment/my-app  # rollback

• Default concurrent execution limit: 1000 per account per region (soft limit, can be increased). One Lambda function hitting the limit can throttle all other functions in the account. Use reserved concurrency to isolate critical functions.
• Lambda is not suitable for long-running processes (batch jobs, video encoding) or anything requiring persistent local state.

public class UserHandler
    implements RequestHandler<APIGatewayProxyRequestEvent,
                               APIGatewayProxyResponseEvent> {

  // Initialised ONCE per container — reused on warm starts
  private final DynamoDbClient dynamo = DynamoDbClient.create();

  @Override
  public APIGatewayProxyResponseEvent handleRequest(
      APIGatewayProxyRequestEvent event, Context ctx) {

    String userId = event.getPathParameters().get("id");
    // query dynamo, build response...
    return new APIGatewayProxyResponseEvent()
        .withStatusCode(200)
        .withBody("{\"userId\":\"" + userId + "\"}");
  }
}

• Do not use Spring Boot as a Lambda runtime. Spring's application context initialises in 3–8 seconds — that's a multi-second cold start on every scale-out event. Use plain Java for Lambda. If your team requires Spring, use Spring Cloud Function + SnapStart, which snapshots the initialised context to cut cold starts to ~1 second.
• Static initialisers (static { ... }) run during the init phase — counted as cold start time. Move heavyweight setup into instance variables (initialised once) or lazy-init them on first invocation.

• Lambda SnapStart (available for Java 11+ managed runtimes): AWS takes a snapshot of the initialised execution environment after the init phase, then restores from it on cold starts. Reduces Java cold starts from seconds to ~1 second. Enable in the function configuration → "Edit" → SnapStart.
• Provisioned Concurrency: keeps N containers pre-warmed and ready. Eliminates cold starts completely but you pay for the reserved capacity even when idle. Use for latency-sensitive APIs.
• Avoid heavy Spring Boot in Lambda — Spring's context initialisation is slow. Use plain Java, Quarkus (with native compilation), or Micronaut instead. If you need Spring, use Spring Cloud Function with GraalVM native image.

• The right memory setting is not always the minimum. Use AWS Lambda Power Tuning (an open source Step Functions state machine) to find the optimal memory-to-cost ratio for your function.
• Ephemeral storage (/tmp): 512 MB by default, configurable up to 10 GB. Files in /tmp persist between warm invocations on the same container — useful for caching, but don't assume it's always available.

• SQS: Lambda polls the queue and invokes your handler with a batch of messages. Built-in retry (message returns to queue on failure), DLQ support, batch window and batch size configuration.
• SNS: Fan-out pattern — SNS topic → multiple SQS queues → each with its own Lambda. Decouples producer from many consumers.
• Kinesis Data Streams: Lambda processes records in order within a shard. Use for real-time streaming (log processing, IoT data, event sourcing). Each shard gets one concurrent Lambda invocation.
• DynamoDB Streams: Every write to a DynamoDB table appears as an event. Lambda can react to inserts/updates/deletes for change data capture (CDC), cross-table sync, or event sourcing.

• Idempotency is mandatory for SQS and Kinesis consumers. Lambda may invoke your handler more than once for the same message (at-least-once delivery). Your handler must produce the same result on re-processing — use a DynamoDB conditional write or idempotency key table to detect duplicates.
• SQS batch failures: If one message in a batch fails, the entire batch is retried by default (returning all messages to the queue — including the ones that succeeded). Use ReportBatchItemFailures in your Lambda response to return only the failed message IDs for retry.
• A DLQ (Dead Letter Queue) captures messages that fail after all retries. Always configure one — without it, poisoned messages cycle forever and block the queue.

• HTTP API: JWT authorisers, Lambda proxy integration, CORS, lower cost (~$1/million requests). Missing: caching, API keys/usage plans, per-method throttling, AWS WAF integration.
• REST API: all HTTP API features plus response caching, usage plans, API keys, resource policies, AWS WAF, mock integrations, request/response transformations. Costs ~$3.50/million requests.
• Default recommendation: use HTTP API unless you need a specific REST API feature.

• The API Gateway endpoint is public by default. Anyone can call it. Add an authoriser (JWT/Cognito for HTTP API, Lambda authoriser for custom logic) or at minimum use an API key to prevent open access in production.

• Choose a high-cardinality partition key (e.g., user ID, order ID — values that are unique and evenly distributed). A low-cardinality partition key (e.g., status = "active"/"inactive") creates hot partitions that throttle performance.
• DynamoDB has no joins. Design your table access patterns upfront — think in queries, not entities. One table design (putting multiple entity types in one table) is the advanced but optimal pattern.
• Free tier: 25 GB storage + 25 RCUs + 25 WCUs per month — always free, not just 12 months.

• 1 RCU = 1 strongly consistent read per second for items up to 4 KB, or 2 eventually consistent reads per second. 1 WCU = 1 write per second for items up to 1 KB.
• On-Demand is ~5–7x more expensive than Provisioned at equivalent sustained throughput. Use On-Demand for unpredictable or new workloads; switch to Provisioned once traffic patterns are known.

• Transactions consume 2× the capacity units (reads and writes are doubled). Budget accordingly.
• TransactionConflictException occurs when two concurrent transactions touch the same item. Implement exponential backoff and retry in your Lambda handler.

• GSI reads are eventually consistent — the GSI may lag behind the base table by milliseconds to seconds. Never use a GSI for reads that must reflect the very latest write.

terraform {
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
  }
}

provider "aws" {
  region = var.region
}

variable "region" {
  default = "us-east-1"
}

resource "aws_s3_bucket" "my_bucket" {
  bucket = "my-app-uploads-${var.region}"
}

resource "aws_s3_bucket_versioning" "my_bucket" {
  bucket = aws_s3_bucket.my_bucket.id
  versioning_configuration { status = "Enabled" }
}

• terraform init: download providers and modules, initialise the backend. Run once per project or after backend/provider changes.
• terraform plan: show what will be created, changed, or destroyed. Reads current state and compares to your code. Always review this output before applying.
• terraform apply: execute the plan. Prompts for confirmation. Writes results to state file.
• terraform destroy: destroys all resources managed by the current state. Use carefully.

• Never run terraform apply in CI without first reviewing the plan output. A -replace or -destroy flag in the wrong hands deletes production resources.
• terraform import: bring an existing manually-created resource under Terraform management without recreating it.

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-state-lock"
    encrypt        = true
  }
}

• State files can contain sensitive values (RDS passwords, private keys). Enable S3 bucket encryption and restrict access with bucket policies.
• Never edit the state file manually. Use terraform state mv, terraform state rm, or terraform import for state surgery.

• Avoid deeply nested modules — they make debugging hard. Flat module structures are more readable.
• Pin module versions (version = "5.5.0") to avoid unexpected breaking changes from upstream updates.

public class MyStack extends Stack {
  public MyStack(Construct scope, String id) {
    super(scope, id);
    Bucket bucket = Bucket.Builder.create(this, "MyBucket")
        .versioned(true)
        .blockPublicAccess(BlockPublicAccess.BLOCK_ALL)
        .build();
  }
}
// Deploy: cdk synth  →  cdk deploy

• CDK synthesises to CloudFormation — you get Change Sets and rollback protection for free. The downside: CDK's generated templates are verbose and hard to read directly.
• CDK is opinionated. It sets sensible defaults (encryption on S3, deletion protection on RDS). Sometimes you need to explicitly opt out of a sensible default, which requires reading the Construct docs carefully.

AWSTemplateFormatVersion: '2010-09-09'
Description: My application stack
Parameters:
  InstanceType:
    Type: String
    Default: t3.micro
Resources:          # required
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      ImageId: ami-0abcdef1234567890
Outputs:
  InstanceId:
    Value: !Ref MyInstance

• If a stack update fails, CloudFormation automatically rolls back to the previous state. This is generally helpful but means you must investigate why the update failed before retrying.
• Deleting a stack deletes all resources in it (by default). Use DeletionPolicy: Retain on critical resources like RDS and S3 to protect them.

• Manual changes to CloudFormation-managed resources cause drift. If you update a resource manually and then run CloudFormation, it may overwrite your manual change or fail with a conflict. Always make changes through CloudFormation or Terraform — never manually in the console for IaC-managed resources.

• ElastiCache Redis does not support all Redis commands. In cluster mode, commands that operate across multiple keys (e.g., MGET with keys in different slots, KEYS *, Lua scripts touching multiple keys) are restricted or behave differently. Always test your Redis command usage against a cluster-mode instance before going to production.
• Memcached on ElastiCache does not support Multi-AZ automatic failover or replication. If a Memcached node fails, all data in that node is lost.

• Spring Boot's default Redis client is Lettuce (not Jedis). Lettuce handles cluster topology changes automatically — when a primary fails and a replica is promoted, Lettuce refreshes the cluster view and reconnects without application restart. Jedis requires more manual cluster configuration.
• You cannot switch a cluster between cluster mode disabled and enabled without recreating it. Plan your topology before provisioning.
• In cluster mode enabled, all keys in a MULTI/EXEC transaction or a Lua script must hash to the same slot. Use hash tags (e.g., {user:123}:cart and {user:123}:profile) to force related keys into the same slot.

• TTL is mandatory. Without a TTL, cached entries never expire. If the underlying data changes in the database and the cache is not explicitly evicted, the application will serve stale data indefinitely. Always set a TTL appropriate to your data's rate of change.
• Cache stampede (thundering herd): when a popular key's TTL expires (or on a cold start), dozens or hundreds of simultaneous requests all miss the cache and fire concurrent queries to the database — which can overwhelm it. Mitigate with: (1) adding random jitter to TTLs (e.g., 55–65 seconds instead of exactly 60) so hot keys don't expire simultaneously; (2) probabilistic early expiry (PER) — start refreshing a key before it expires based on a probability function; (3) a single-flight/request-coalescing pattern where only one thread fetches from the DB and others wait for the result.

• Write penalty: every database write now requires a second write to Redis. Writes are slower, and the write path now has a dependency on Redis availability.
• Cache pollution: data written to the cache may never be read, wasting memory. For write-heavy datasets with unpredictable read access patterns (e.g., audit logs, bulk imports), write-through wastes cache space.
• Hybrid approach: use write-through for hot, frequently-read data (user sessions, current inventory counts for top products) and cache-aside for cold or unpredictable data. Many production systems combine both patterns by data category.

# Fetch host from environment variable injected via Secrets Manager / EKS secret
spring.data.redis.host=${REDIS_HOST}
spring.data.redis.port=6379
spring.data.redis.password=${REDIS_AUTH_TOKEN}
spring.data.redis.ssl.enabled=true   # required when TLS is enabled on the cluster
# Connection pool (Lettuce)
spring.data.redis.lettuce.pool.max-active=20
spring.data.redis.lettuce.pool.max-idle=10
spring.data.redis.lettuce.pool.min-idle=2

@Configuration
@EnableCaching
public class CacheConfig {

    @Bean
    public RedisCacheConfiguration redisCacheConfiguration(ObjectMapper objectMapper) {
        Jackson2JsonRedisSerializer<Object> serializer =
            new Jackson2JsonRedisSerializer<>(objectMapper, Object.class);

        return RedisCacheConfiguration.defaultCacheConfig()
            .entryTtl(Duration.ofSeconds(60))
            .serializeValuesWith(
                RedisSerializationContext.SerializationPair.fromSerializer(serializer));
    }
}

// Service layer
@Service
public class InventoryService {

    @Cacheable(value = "inventory", key = "#id")
    public InventoryItem getInventoryById(Long id) {
        return inventoryRepository.findById(id).orElseThrow();
    }

    @CacheEvict(value = "inventory", key = "#id")
    public void updateInventory(Long id, InventoryItem item) {
        inventoryRepository.save(item);
    }
}

@Autowired
private RedisTemplate<String, Object> redisTemplate;

// Manual cache-aside
public InventoryItem getWithManualCache(Long id) {
    String key = "inventory::" + id;
    InventoryItem cached = (InventoryItem) redisTemplate.opsForValue().get(key);
    if (cached != null) return cached;
    InventoryItem item = inventoryRepository.findById(id).orElseThrow();
    redisTemplate.opsForValue().set(key, item, Duration.ofSeconds(60));
    return item;
}

• Default JDK serialization is unreadable. Out of the box, Spring's RedisTemplate uses JDK serialization — cached values appear as binary gibberish in redis-cli and are tied to the exact class structure. Always configure Jackson2JsonRedisSerializer or GenericJackson2JsonRedisSerializer so cached objects are stored as human-readable JSON and are forward-compatible with class refactoring.
• When using GenericJackson2JsonRedisSerializer, the serialized JSON includes the fully qualified class name. This means renaming or moving a class will break deserialization of existing cached entries. Plan cache key versioning or use flushdb on deployment when class structure changes.

# ElastiCache Security Group — inbound rule
Type: Custom TCP
Port: 6379
Source: [Application Security Group ID]   # NOT 0.0.0.0/0, NOT the VPC CIDR

# Spring Boot: use rediss:// (double-s) when TLS is enabled
spring.data.redis.url=rediss://:${REDIS_AUTH_TOKEN}@${REDIS_HOST}:6379

• ElastiCache does not support IAM authentication — unlike RDS IAM auth, there is no way to use an IAM role to authenticate to Redis. The AUTH token is a static string. Store it in Secrets Manager, inject it into your application at startup via an environment variable or Spring Cloud AWS, and rotate it via Secrets Manager's rotation + aws elasticache modify-replication-group --auth-token <new-token>.
• In-transit encryption and at-rest encryption cannot be enabled on an existing cluster — they must be enabled at creation time. Plan for this before provisioning your first cluster.
• Spring's rediss:// URL (double s) enables TLS. Using redis:// against a TLS-enabled cluster will result in a connection error that can be difficult to diagnose.

• Evictions > 0 for 1 datapoint: any eviction means Redis is memory-constrained and is silently invalidating your cache. Action: upgrade the node type or reduce the TTL of lower-priority keys.
• CurrConnections approaching maxclients: the default maxclients is 65000 but is lower for smaller node types (e.g., cache.t3.micro has a lower limit). New connections will be refused when the limit is hit, causing application errors. Check for connection leaks (Lettuce connection pool misconfiguration).
• ReplicationLag > 1 second: reads from replicas may return stale data older than 1 second. Investigate write throughput — the replica may be falling behind on replication.

• Counters only go up — never reset them. If you need rate, use rate() in PromQL. Never use a Gauge for something that only increases.
• Histograms require pre-defined buckets. If your p99 latency exceeds your highest bucket value, histogram_quantile() will return +Inf. Configure buckets that cover your expected latency range.

• The chart creates ~50 default Kubernetes alert rules. On first install they may fire immediately — for example, the "Watchdog" alert is intentional and proves alerting is working end-to-end.
• Review and tune the default rules for your cluster size. Some rules (e.g., "KubeMemoryOvercommit") require adjustment for small dev clusters.

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

# application.properties
management.endpoints.web.exposure.include=health,info,metrics,prometheus

• Histogram buckets must be configured for latency. By default Spring exports http.server.requests with SLO buckets at 50ms, 100ms, 200ms, etc. If your p99 exceeds 1s, add custom buckets: management.metrics.distribution.slo.http.server.requests=50ms,100ms,200ms,500ms,1s,2s,5s

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-spring-app
  namespace: default
spec:
  selector:
    matchLabels:
      app: my-spring-app
  endpoints:
    - port: http
      path: /actuator/prometheus
      interval: 30s

• The ServiceMonitor must be in the same namespace as Prometheus or the Prometheus resource must have serviceMonitorNamespaceSelector: {} to watch all namespaces. This is the most common reason scraping silently fails — the ServiceMonitor exists but Prometheus never picks it up.
• The label selector on the ServiceMonitor must match the labels on your Kubernetes Service object, not the Pod labels.

• Request rate: rate(http_server_requests_seconds_count{job="my-app"}[5m])
• Error rate: rate(http_server_requests_seconds_count{status=~"5.."}[5m]) / rate(http_server_requests_seconds_count[5m])
• p99 latency: histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))
• JVM heap used: jvm_memory_used_bytes{area="heap"}
• HikariCP pool usage: hikaricp_connections_active / hikaricp_connections_max

• rate() requires a counter, not a gauge. Using rate() on a gauge gives nonsense. Use deriv() or delta() for gauges instead.
• histogram_quantile() requires the _bucket metric, not _count or _sum. Always use rate() on the bucket series inside the function.

• Grafana dashboards are stored in the Grafana pod's SQLite DB by default — it is ephemeral. If the pod restarts, you lose all custom dashboards. Configure a PVC for persistence or provision dashboards as ConfigMaps (GitOps-friendly, dashboards live in Git as JSON files).

• AlertManager has "inhibition" — a critical alert can suppress related warning alerts for the same component. Configure inhibition rules before going to production or one outage may page you 40 times for the same root cause.
• Test your alerting pipeline end-to-end with the "Watchdog" alert — it fires continuously and proves your full chain (Prometheus → AlertManager → receiver) works.

• Prometheus requires cluster resources and operational care — StatefulSet storage, backup, capacity planning. CloudWatch is serverless. Factor operational overhead into the decision.
• You can send metrics to both by adding both micrometer-registry-prometheus and micrometer-registry-cloudwatch2 — Micrometer fans out to all configured registries simultaneously.

• GitOps does not mean putting secrets in Git. Use External Secrets Operator or Sealed Secrets to store encrypted secret references in Git while the actual values live in AWS Secrets Manager. Committing a plaintext Secret manifest is a critical security failure.
• GitOps replaces cluster state management, not the entire pipeline. GitHub Actions still builds your image and pushes to ECR — GitOps takes over at the point of deploying to the cluster.

• ArgoCD itself is a workload in your cluster — if the cluster is down, ArgoCD is down. This is expected: Kubernetes keeps running whatever was last applied regardless of ArgoCD's state. Don't confuse an ArgoCD outage with an inability to serve traffic.
• ArgoCD stores application state in its own CRDs inside the cluster. Back up your ArgoCD Application resources or manage them via Git (App of Apps pattern) so they survive cluster recreation.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/my-org/manifests.git
    targetRevision: main
    path: apps/my-app
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      selfHeal: true
      prune: true   # delete resources removed from Git

• Start with Manual sync for production, Automatic for dev. With Automatic + selfHeal, a misguided kubectl edit will be silently reverted — which is correct GitOps behaviour but surprises engineers used to imperative workflows.
• prune: true tells ArgoCD to delete cluster resources that no longer exist in Git. Without it, removed manifest files leave orphaned resources running in the cluster indefinitely.

• OutOfSync does not mean broken — it means Git and the cluster differ. OutOfSync + Degraded together means something is both different AND not working. The combination matters more than either status alone.
• ArgoCD's health status for a Deployment is Progressing (not Healthy) during a rolling update. Alert on Degraded, not on Progressing — otherwise every deploy triggers an alert.

• Image Updater needs IAM permissions to read from ECR — use IRSA so no credentials are embedded in the cluster. It also needs write access to your Git repo (via SSH key or GitHub App token stored in a Kubernetes Secret).
• Configure the update strategy carefully: digest (always pull latest digest of a tag — useful for latest tag workflows) vs semver (update to latest matching semver pattern, e.g., ~1.2 updates to any 1.2.x). For production, semver with explicit constraints is safer than tracking latest.

• Never auto-sync production. Always require a manual PR approval → merge → manual ArgoCD sync for prod. Dev and staging can be auto-synced. This is the most important governance rule in a GitOps workflow.
• Promotion is not ArgoCD's job — it is a Git operation. Promote by opening a PR that updates the prod path. ArgoCD detects the merged commit and either auto-syncs (staging) or waits for a manual sync (prod).

• Rollback in ArgoCD does not revert the Git repo — it deploys an older revision while the repo still has the newer commit. This creates a deliberate sync divergence (the cluster is behind Git). After rollback, fix the root cause, commit a fix forward, and sync — do not leave the cluster in a rolled-back state indefinitely.
• The correct production process: rollback ArgoCD to restore service → open a hotfix PR → merge → sync forward. Never leave the cluster and Git permanently diverged.

• If you accidentally commit a secret to Git — even briefly — rotate the secret immediately. Git history is permanent; the value is exposed even after deletion from the working tree.
• ESO creates a standard Kubernetes Secret from the external source. Configure ArgoCD to ignore ESO-managed secrets in its diff to avoid spurious OutOfSync status — ArgoCD did not create them, so they will always appear as drifted without this exclusion.