AWS serverless service desigin

  1. AWS serverless app

    • Security: IAM
    • Compute: Lambda, Docker
    • Files and ingest: S3, Kinesis
    • Data: DynamoDB
    • Networking: Key pairs, VPC
    • Data processing: ML

  2. IAM (搜索IAM)

    • create adimin user
    • create admin group (policy type: administrator access)
    • change IAM users sign-in link (自定义)
    • 使用更新后的IAM，新建的admin用户登陆

  3. Cost alarm

    • create budgets (从 AWS Account 进入)
    • create alarm (搜索cloudwatch)

  4. Ingest with S3 (搜索 S3)

    • login in non-root
    • create buckets
      • tag your bucket
      • upload objects with different encrytion

  5. Ingest with Kinesis (搜索 Kinesis)

    • a service creates a pipe to stream files into Amazon
    • to send data in or out, you have to write code with the Kinesis stream
    • shards
      • esimate the number of shards since Amazon storage is auto scale
    • edit your kinesis
      • encytion
      • shard level metrics to monitor the distribution of data throughput

  6. Inget with Kinesis Firehose

    • create a delivery stream and send your data to the steream with a Kinesis agent or the Firehose API

  7. Dynamo DB (搜索 Dynamo)

    • No SQL
    • create table
      • table overview:
        • TTL (time to live)
        • provision read/write capacity units
      • itmes:
        • add key:value
    • reserved capaciy (saving money)
      • if one year with lagre scale

  8. Compute with Lambda

    • ability to run functon and oly billed when the function is called
    • think of Lambda as a method
    • creae a function
      • Runtion: python3.6
      • hello-word-python3 template
    • test your function
    • trigger your function
    • config your function
      • debug
        • x-ray
          • add “x-ray” to a role
      • tag

  9. Data Processing with Rekoginition

    • deep learning image recognition
    • based on Amazon’s huge body of labeled images
    • help you derive meaning out of images
    • try demo

  10. Data Processing with ML

    • machine learning model

  11. Data Processing with Polly

    • converts text to speech

  12. Data Processing with Lex

    • building conversational interfaces
    • chat bot

AWS server-based service desigin

  1. AWS server-based app

    • Security: IAM, EC2 key pairs
    • Compute: EC2
    • Files: S3
    • Data: RDS aurora, Redshift, Elastic MapReduce(EMR)
    • Networking: Elastic IP address(EIP), VPC
    • Data visualization: QuickSight

  2. Network with VPC (类同OCI的VCN)

    • virtual private cloud(VPC) protection
    • VPC is a set of internal IP addresses for your service to connect which underlying VM
    • Lable the objects in VPC
    • GateWay
      • Internet gateway: translate between the public insternet and the internal website resouces
        • Egress
      • Virtual private gateway: a secure tunnel for the traffic going back and forth from your on-premise site up into the cloud
    • Router
      • configured with route tables that route traffic inside of the VPC
    • ACLs
      • to particular subnets and this can further segment the traffic by IP and
      • inbound/outbound rules
    • Elastic IPs
      • static IP that are routable outside of our VPC
      • allocate the public IP and spin up an instance from EC2
    • EndPoints
    • …

  3. EC2

    • IAM:create EC2 role
      • add S3 policy
    • Compute for EC2:
      • key pairs
      • Instance
        • type
        • size: cpur, ram
        • configuration: os, sw, version
        • security: vpc, subnet, key, role
        • purchase method: on demand, RI, spot
      • spot instance
      • reserved instance
    • Availablility for EC2: ?

  4. Data-tier

    • RDS
    • Redshift

  5. Visualizaiton with QuickSight

AWS code tools

  1. CLI
    • aws –version
    • aws configure
      • access key from IAM->user->security credential
      • region
      • format: json
    • aws s3 ls
    • aws ec2 describe-instances
    • aws ec2 stop-instances –instance-ids {instanceID}
    • aws ec2 describe-images –image-ids {imageID}
  2. AWS SDK

AWS apps of Trade-offs

  1. Design (architecture) matters

    • Scalability
      • elasticity: lbr, replicas
      • scalability: monitor, auto
      • integration: alarms
    • HA
      • availability
        • serverless
          • rely on Amazon
        • server-based
          • server redundancy or multi-location
          • Amazon Machine Images(AMIs)
      • fault-tolerance
        • fail-over
        • log and alert
        • DR
          • RTO
          • RPO
      • security
        • work as admin (non-root)
        • reduec the attach surface
        • monitor
    • predictable cost
      • cost of services
        • best fit type of services, eg: serverless vs server-based
        • best fit size of services, eg: EC2 instance size
        • best fit qty of services, eg: number of RDS instances
      • purchase methods
        • on demand/ RI/ spot
        • location
      • human cost
        • training
        • consultants
        • time to build when using new patterns like serverless

  2. EC2 LBR for Availability and Auto Scaling Group Objs

    • LBR
      • type of lbr
        • app level
        • network level
    • Auto Scaling Group Objs
      • group:
        • unit for scaling and managemnt
      • launch configurations
        • template for EC2 instances
      • scaling plans:
        • tell auto scaling group how and when to scale

  3. Scenario-website:

    • Compute
      • EC2 (server-based)
        • instance sizes, instance types, purchase types
      • Lambda serveless)
        • cost of implementation
          • training costs
    • Files services
      • S3
        • location
        • redundacy
        • archving
        • sercurity
    • Data Services
      • RDS
        • licensed vs open source
      • serverless
        • DynamoDB
          • query types and training
      • caching
      • ML
      • data visualization
    • Others:
      • data ingress/egress
      • direct connect
      • CDN

  4. Scenario-data pipeline

    • Compute
      • big dta pipelinse movint to serverless due to economies of scale
      • large-scale big data pipelines magnify serverless cost savings
    • Files Services
      • require archiving
      • working with high-volume data
        • location
        • redundancy
        • compression
        • encryption
    • Data Services
      • no RDS, in other words, S3 becomes the db
      • streaming
        • kinesis
        • firehose
        • open source: Apache Kafka
      • data transformations computationally intensive
        • aws glue
      • caching
        • hosted caching or EC2 Redis
      • ML
        • endpoints or hosted
      • data visualiztion
        • 3rd party: tablo, looker
        • custom
    • Others + data ingress/egress + direct connect: between your on-premise DC and Amazons’s services

  5. Scenario-data lake

    • implementations can be inexpensive
    • high economies of scale
    • no lbr, auto scale, groups, redundance…since Lambda scales automatically
    • serverless can be HA, high scalabiltiy and reduced total cost of ownership

  6. Scenario-IoT app

Useful resources

  1. ServerlessConf
  2. AWS Well-Architected series of whitepapers