Showing posts with label AWS CDK. Show all posts
Showing posts with label AWS CDK. Show all posts

Monday, December 6, 2021

Creating an Aurora Database Instance with AWS CDK (Python) and Secret Manager

 Title: Creating an Aurora Database Instance with AWS CDK (Python) and Secret Manager


Introduction:

In this blog post, we will walk through the process of creating an Amazon Aurora database instance using AWS Cloud Development Kit (CDK) with Python. Additionally, we'll enhance the security of our application by utilizing AWS Secrets Manager to manage and retrieve our database credentials.


Prerequisites:

Before we begin, make sure you have the following prerequisites:


1. AWS CDK installed: [CDK Installation Guide](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html)

2. Python installed: [Python Installation Guide](https://www.python.org/downloads/)


Step 1: Set Up Your CDK Project

Create a new directory for your CDK project and navigate to it in your terminal.


mkdir aurora-cdk

cd aurora-cdk



Initialize your CDK project.


cdk init app --language python



Step 2: Install Required CDK Libraries

Install the necessary CDK libraries for Amazon Aurora and Secrets Manager.



pip install aws-cdk.aws-rds aws-cdk.aws-secretsmanager



Step 3: Import Dependencies in Your CDK App

Open the `app.py` file in your favorite code editor and import the required CDK modules.


from aws_cdk import (

    core,

    aws_rds as rds,

    aws_secretsmanager as secretsmanager

)



 Step 4: Define the CDK Stack

Define your CDK stack by creating a class that inherits from `core.Stack`. Inside the class, define the Aurora database instance and the Secrets Manager secret.


class AuroraCdkStack(core.Stack):


    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:

        super().__init__(scope, id, **kwargs)


        # Create a Secrets Manager secret for database credentials

        secret = secretsmanager.Secret(

            self,

            "AuroraSecret",

            secret_name="AuroraCredentials",

            generate_secret_string=secretsmanager.SecretStringGenerator(

                secret_string_template='{"username": "admin"}',

                generate_string_key="password",

                exclude_characters='"@/',

            )

        )


        # Create an Aurora database instance

        aurora_db = rds.DatabaseInstance(

            self,

            "AuroraDB",

            engine=rds.DatabaseInstanceEngine.aurora_postgres,

            master_username=secret.secret_value_from_json("username").to_string(),

            master_password=secret.secret_value_from_json("password").to_string(),

            instance_class=core.Fn.select(0, ["db.t3.small"]),

            vpc_subnets={"subnet_type": core.SubnetType.PRIVATE},

            removal_policy=core.RemovalPolicy.DESTROY  # WARNING: Do not use in production

        )


Step 5: Deploy Your CDK Stack

Deploy your CDK stack to create the Aurora database instance.



cdk deploy


### Conclusion:

Congratulations! You have successfully created an Amazon Aurora database instance using AWS CDK with Python. By integrating AWS Secrets Manager, you've enhanced the security of your application by securely managing and retrieving your database credentials.


Remember to manage your secrets and credentials responsibly, and never expose sensitive information in your code or configuration files.

Thursday, September 30, 2021

AWS - Create Layers for Lambda for Python

 AWS - Create Layers for Lambda for Python

When working with Python 3 runtimes the layer needs to follow a particular folder structure otherwise you will receive errors as the Python runtime is unable to find the module. 

1. install jwt to a target directory using pip, ie pip3 install jwt -t jwt 

2. Navigate to the jwt directory which contains the dependency files for the JWT module, create a directory named python and copy all the files into this directory 

3. Zip the python directory, the zip file should contain the directory "python" that contains the lambda dependency files 

4. Create the layer to the Lambda console[2] and copy it's ARN 

5. Add the layer to your Lambda function by navigating to the Lambda function, and click on the "Layers" link at the top of the page, copy the layer's ARN into the "Specify an ARN" input box and choose the "specify an ARN" option. 

To test to make sure the Lambda is configured correctly you can add "import jwt" at the top of your Python file and run the function. 

If the layer is incorrectly configured, you will receive an error indicating that the function cannot find the simple-salesforce library. If the function executes OK at this point, then the layer is correctly configured and can be used.

The directory structure will be for e.g.

  /LayersWorkingDirectory/

 1. mkdir python 

2. cd python 

3. python3 -m pip install simple-salesforce -t . 

4. cd .. 

5. zip -r9 salesforce.zip python 

6. Use "salesforce.zip" to create a layer and use it with the function. 

 


Thursday, December 31, 2020

AWS CDK (Python) Create Step Functions for Orchestrating Lambda and Glue Tasks

 AWS Step Functions is a serverless function orchestrator that makes it easy to sequence AWS Lambda functions and multiple AWS services into business-critical applications. Through its visual interface, you can create and run a series of checkpointed and event-driven workflows that maintain the application state. The output of one step acts as an input to the next. Each step in your application executes in order, as defined by your business logic.

Orchestrating a series of individual serverless applications, managing retries, and debugging failures can be challenging. As your distributed applications become more complex, the complexity of managing them also grows. Step Functions automatically manages error handling, retry logic, and state. With its built-in operational controls, Step Functions manages sequencing, error handling, retry logic, and state, removing a significant operational burden from your team.



Setup

The cdk.json file tells the CDK Toolkit how to execute your app.

This project is set up like a standard Python project. The initialization process also creates a virtualenv within this project, stored under the .env directory. To create the virtualenv it assumes that there is a python3 (or python for Windows) executable in your path with access to the venv package. If for any reason the automatic creation of the virtualenv fails, you can create the virtualenv manually.

To manually create a virtualenv on MacOS and Linux:

$ python3 -m venv .env

After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv.

$ source .env/bin/activate

If you are a Windows platform, you would activate the virtualenv like this:

% .env\Scripts\activate.bat

Once the virtualenv is activated, you can install the required dependencies.

$ 

At this point you can now synthesize the CloudFormation template for this code.

$ cdk synth

To add additional dependencies, for example other CDK libraries, just add them to your setup.py file and rerun the pip install -r requirements.txt command.

This thread helps to create an AWS Step Functions StateMachine with the Python language bindings for CDK.


app.py

from aws_cdk import (
    aws_stepfunctions as sfn,
    aws_stepfunctions_tasks as sfn_tasks,
    core,
)


class JobPollerStack(core.Stack):
    def __init__(self, app: core.App, id: str, **kwargs) -> None:
        super().__init__(app, id, **kwargs)

        submit_job_activity = sfn.Activity(
            self, "SubmitJob"
        )
        check_job_activity = sfn.Activity(
            self, "CheckJob"
        )
        do_mapping_activity1 = sfn.Activity(
            self, "MapJOb1"
        )
        do_mapping_activity2 = sfn.Activity(
            self, "MapJOb2"
        )

        submit_job = sfn.Task(
            self, "Submit Job",
            task=sfn_tasks.InvokeActivity(submit_job_activity),
            result_path="$.guid",
        )

        task1 = sfn.Task(
            self, "Task 1 in Mapping",
            task=sfn_tasks.InvokeActivity(do_mapping_activity1),
            result_path="$.guid",
        )

        task2 = sfn.Task(
            self, "Task 2 in Mapping",
            task=sfn_tasks.InvokeActivity(do_mapping_activity2),
            result_path="$.guid",
        )

        wait_x = sfn.Wait(
            self, "Wait X Seconds",
            time=sfn.WaitTime.seconds_path('$.wait_time'),
        )
        get_status = sfn.Task(
            self, "Get Job Status",
            task=sfn_tasks.InvokeActivity(check_job_activity),
            input_path="$.guid",
            result_path="$.status",
        )
        is_complete = sfn.Choice(
            self, "Job Complete?"
        )
        job_failed = sfn.Fail(
            self, "Job Failed",
            cause="AWS Batch Job Failed",
            error="DescribeJob returned FAILED"
        )
        final_status = sfn.Task(
            self, "Get Final Job Status",
            task=sfn_tasks.InvokeActivity(check_job_activity),
            input_path="$.guid",
        )

        definition_map = task1.next(task2)

        process_map = sfn.Map(
            self, "Process_map",
            max_concurrency=10
        ).iterator(definition_map)

        definition = submit_job \
            .next(process_map) \
            .next(wait_x) \
            .next(get_status) \
            .next(is_complete
                  .when(sfn.Condition.string_equals(
                    "$.status", "FAILED"), job_failed)
                  .when(sfn.Condition.string_equals(
                    "$.status", "SUCCEEDED"), final_status)
                  .otherwise(wait_x))

        sfn.StateMachine(
            self, "StateMachine",
            definition=definition,
            timeout=core.Duration.seconds(30),
        )


cdk.json
{
    "app": "python3 app.py"
}

Useful Commands

  • cdk ls list all stacks in the app
  • cdk synth emits the synthesized CloudFormation template
  • cdk deploy deploy this stack to your default AWS account/region
  • cdk diff compare deployed stack with current state
  • cdk docs open CDK documentation

The cdk.json file tells the CDK Toolkit how to execute your app.


To manually create a virtualenv on MacOS and Linux:

$ python  -m venv .venv

After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv.

$ source .venv/Scripts/activate

If you are a Windows platform, you would activate the virtualenv like this:

% .venv\Scripts\activate.bat

Once the virtualenv is activated, you can install the required dependencies.

$ pip install -r requirements.txt

At this point you can now synthesize the CloudFormation template for this code.

$ cdk synth


Saturday, December 26, 2020

Provision AWS ECS & Fargate Load Balanced Service with AWS CDK (Python) Part 1

AWS Fargate is a serverless compute engine for containers that works with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS). Fargate makes it easy for you to focus on building your applications. Fargate removes the need to provision and manage servers, lets you specify and pay for resources per application, and improves security through application isolation by design.

Fargate allocates the right amount of compute, eliminating the need to choose instances and scale cluster capacity. You only pay for the resources required to run your containers, so there is no over-provisioning and paying for additional servers. Fargate runs each task or pod in its own kernel providing the tasks and pods their own isolated compute environment. This enables your application to have workload isolation and improved security by design.  


This thread discusses aws-cdk (Python) to provision VPC/ECS and Fargate.


app.py

from aws_cdk import (

    aws_ec2 as ec2,

    aws_ecs as ecs,

    aws_ecs_patterns as ecs_patterns,

    core,

)

class ProvisionFargate(core.Stack):

      def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:

        super().__init__(scope, id, *kwargs)

        # Create VPC and Fargate Cluster

        # NOTE: Limit AZs to avoid reaching resource quotas

        vpc = ec2.Vpc(

            self, "MindTelligentVpc",

            max_azs=2

        )

        cluster = ecs.Cluster(

            self, 'Ec2Cluster',

            vpc=vpc

        )


        fargate_service = ecs_patterns.NetworkLoadBalancedFargateService(

            self, "FargateService",

            cluster=cluster,

            task_image_options={

                'image': ecs.ContainerImage.from_registry("amazon/amazon-ecs-sample")

            }

        )


        fargate_service.service.connections.security_groups[0].add_ingress_rule(

            peer = ec2.Peer.ipv4(vpc.vpc_cidr_block),

            connection = ec2.Port.tcp(80),

            description="Allow http inbound from VPC"

        )


        core.CfnOutput(

            self, "LoadBalancerDNS",

            value=fargate_service.load_balancer.load_balancer_dns_name

        )


app = core.App()

ProvisionFargate(app, "MindTelligent")

app.synth()


requirements.txt

 aws-cdk.core
aws-cdk.aws_ec2
aws-cdk.aws_ecs
aws-cdk.aws_ecs_patterns

# Work around for jsii#413
aws-cdk.aws-autoscaling-common


Tuesday, May 5, 2020

AWS CDK - Build EC2 Instance (Python)

AWS CDK is a software development framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation.
AWS CloudFormation enables you to:
  • Create and provision AWS infrastructure deployments predictably and repeatedly.
  • Leverage AWS products such as Amazon EC2, Amazon Elastic Block Store, Amazon SNS, Elastic Load Balancing, and Auto Scaling.
  • Build highly reliable, highly scalable, cost-effective applications in the cloud without worrying about creating and configuring the underlying AWS infrastructure.
  • Use a template file to create and delete a collection of resources together as a single unit (a stack).


def create_windows_bastion_server(self, vpc=None):
    if vpc is None:
        vpc = self.vpc
    # The code that defines your stack goes here    host = ec2.Instance(self,windows_bastion_server_name,
                        instance_type=ec2.InstanceType(
                            instance_type_identifier=ec2_micro_type),
                        instance_name='windows_bastion_server',
                        machine_image=windows_ami,
                        vpc=vpc,
                        key_name='mindtelligent_aws_bastion_server_key',
                        vpc_subnets=windows_vpc_subnet
                        )
    # ec2.Instance has no property of BlockDeviceMappings, add via lower layer cdk api:    host.instance.add_property_override("BlockDeviceMappings",[{
        "DeviceName": "/dev/xvda",
        "Ebs": {
            "VolumeSize": "10",
            "VolumeType": "io1",
            "Iops": "150",
            "DeleteOnTermination": "true"        }
    },{
        "DeviceName": "/dev/sdb",
        "Ebs": {"VolumeSize": "30"}
    }
    ])  # by default VolumeType is gp2, VolumeSize 8GB    host.connections.allow_from_any_ipv4(
        ec2.Port.tcp(3389),"Allow RDP from internet")
    host.connections.allow_from_any_ipv4(
        ec2.Port.tcp(80),"Allow ssh from internet")

How IdP Groups Are Tied to Databricks Groups (Unity Catalog)

  🔗 How IdP Groups Are Tied to Databricks Groups (Unity Catalog) 🔑 Key Principle (Read This First) Databricks does NOT “map” IdP groups...