Wednesday, December 23, 2020

Docker Containers for Data Weave: Data WeavePlay Ground

DataWeave (DW) Playground that can be used for both versions of DW: 1.0 and 2.0.

DataWeave version 1.0 is used in Mule 3, and DataWeave version 2.0 is used in Mule 4.

DataWeave is MuleSoft’s scripting language that is mainly used to transform the data in your integration apps. DW Playground has a Web Based Transform Message Component that allows the users to build and test Data Weave Code.



Installation:

1. Download and Install the Docker Desktop app

You can go to this link and select your operating system (Windows / Mac) in order to download the corresponding installer.

Just follow the instructions to install as you would with any other regular application.


2. Run Docker

Double click on the installed application to open it and make sure that Docker is running in the background. To verify this, you can check your taskbar and look for the Docker icon.
3. Pull Docker Imagedocker pull machaval/dw-playground:2.3.1-SNAPSHOT
4. Run Dockerdocker run -d --name DWPlayground2 -p 9999:8080 machaval/dw-playground:2.3.1-SNAPSHOT
or
docker run -it --rm -p9000:8080 machaval/dw-playground:2.3.1-SNAPSHOT
After the Docker image has downloaded and started up, point your browser to localhost port 9000, the playground DataWeave container.








Tuesday, May 5, 2020

AWS CDK - Build EC2 Instance (Python)

AWS CDK is a software development framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation.
AWS CloudFormation enables you to:
  • Create and provision AWS infrastructure deployments predictably and repeatedly.
  • Leverage AWS products such as Amazon EC2, Amazon Elastic Block Store, Amazon SNS, Elastic Load Balancing, and Auto Scaling.
  • Build highly reliable, highly scalable, cost-effective applications in the cloud without worrying about creating and configuring the underlying AWS infrastructure.
  • Use a template file to create and delete a collection of resources together as a single unit (a stack).


def create_windows_bastion_server(self, vpc=None):
    if vpc is None:
        vpc = self.vpc
    # The code that defines your stack goes here    host = ec2.Instance(self,windows_bastion_server_name,
                        instance_type=ec2.InstanceType(
                            instance_type_identifier=ec2_micro_type),
                        instance_name='windows_bastion_server',
                        machine_image=windows_ami,
                        vpc=vpc,
                        key_name='mindtelligent_aws_bastion_server_key',
                        vpc_subnets=windows_vpc_subnet
                        )
    # ec2.Instance has no property of BlockDeviceMappings, add via lower layer cdk api:    host.instance.add_property_override("BlockDeviceMappings",[{
        "DeviceName": "/dev/xvda",
        "Ebs": {
            "VolumeSize": "10",
            "VolumeType": "io1",
            "Iops": "150",
            "DeleteOnTermination": "true"        }
    },{
        "DeviceName": "/dev/sdb",
        "Ebs": {"VolumeSize": "30"}
    }
    ])  # by default VolumeType is gp2, VolumeSize 8GB    host.connections.allow_from_any_ipv4(
        ec2.Port.tcp(3389),"Allow RDP from internet")
    host.connections.allow_from_any_ipv4(
        ec2.Port.tcp(80),"Allow ssh from internet")

Sunday, March 15, 2020

Apache Spark 2.4.5: Installation on Ubuntu on AWS.


Apache Spark 2.4.5: Installation on Ubuntu on AWS.

  • Download the latest release of Spark here.
  • Unpack the archive.
  •  tar -xvf spark-2.4.5-bin-hadoop2.7.tgz
  •  Move the resulting folder and create a symbolic link so that you can have multiple versions of Spark installed.
  • sudo mv spark-2.4.5-bin-hadoop2.7 /usr/local 
  • sudo ln -s /usr/local/spark-2.4.5-bin-hadoop2.7/ /usr/local/spark
  • cd spark-2.4.5-bin-hadoop2.7/ 
  • Also add SPARK_HOME to your environment.
  • export SPARK_HOME=/usr/local/spark 
  • Start a standalone master server. At this point you can browse  to http://localhost:8080/, to view the status
  • $SPARK_HOME/sbin/start-master.sh 
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark/logs/spark-osboxes-org.apache.spark.deploy.master.Master-1-osboxes.out 




Start a Slave Process

$SPARK_HOME/sbin/start-slave.sh spark://osboxes:7077

To get this to work, make an entry in /etc/hosts as


27.0.0.1       localhost
127.0.1.1       osboxes

To check the Logs

vi /usr/local/spark/logs/spark-osboxes-org.apache.spark.deploy.worker.Worker-1-osboxes.out 

Test out the Spark shell. You’ll note that this exposes the native Scala interface to Spark. 
    $SPARK_HOME/bin/spark-shell



  • To use Py Spark
$SPARK_HOME/bin/pyspark




To Stop the Slave:
$SPARK_HOME/sbin/stop-slave.sh

To Stop the master:

$SPARK_HOME/sbin/stop-master.sh























Saturday, February 29, 2020

Ubuntu: How to resolve Err:4 http://security.ubuntu.com/ubuntu cosmic-updates/main amd64 libsnmp-base all 5.7.3+dfsg-1.8ubuntu3.18.10.1 404 Not Found

Ubuntu: How to resolve Err:4 http://security.ubuntu.com/ubuntu cosmic-updates/main amd64 libsnmp-base all 5.7.3+dfsg-1.8ubuntu3.18.10.1   404  Not Found

This happens because you still have somehow the sources.list file for Ubuntu 13.10 on your machine:
You can use sudo -i in terminal and copy & paste the following (complete block) into it:

cat > /etc/apt/sources.list << EOF
deb http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-backports main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial-backports main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse
#deb http://archive.ubuntu.com/ubuntu xenial-proposed restricted main universe multiverse
#deb-src http://archive.ubuntu.com/ubuntu xenial-proposed restricted main universe multiverse
deb http://archive.canonical.com/ubuntu xenial partner
deb-src http://archive.canonical.com/ubuntu xenial partner
EOF

This will overwrite your /etc/apt/sources.list file. After that you should use exit to get out of root.
If you not wish all package sources to be enabled you can change this from the GUI applet called 'Software&Updates' which you find within your settings.
Then you should do the mandatory:
sudo apt clean
sudo apt update
sudo apt dist-upgrade
This will resolve the issue
 
 
 
 

Tuesday, February 4, 2020

Mule SDK to Create Custom Connectors

Mule SDK to Create Custom Connectors

Create your first Mule SDK project in just a few steps:
  • Install required software
  • Generate a project using Maven and explore it
  • Set up your IDE
  • Build the project you generated
  • Develop a connector or module
  • Run the connector or module in Anypoint Studio
The SDK provides a Maven plugin and a Maven archetype to create new projects. The archetype is a project template; use it to ensure your new project has a consistent structure, employs recommended practices, and contains a few elements for you to experiment with.

Install the Required Software

Install and verify that the following tools are working in your local environment:
  • Java Development Kit 8 (JDK 8) to compile and build your Java code.
  • Apache Maven, version 3.3.9 or higher, to manage your project’s build.

Generate a Project Using the Maven Plugin

Run the generate command to create an extension project based on the Maven archetype:
mvn org.mule.extensions:mule-extensions-archetype-maven-plugin:1.2.0:generate
You are prompted to provide required field values, then a new project is generated.

The pom.xml has the following structure:

<parent> <groupId>org.mule.extensions</groupId> <artifactId>mule-modules-parent</artifactId> <version>1.1.3</version> </parent> 

 Once the code is modified to achieve the objective of the Connector, please install the connector in the repository

mvn install:install-file -DgroupId=com.mindtelligent.connect -DartifactId=MindTelligentPayloadProcessorForMule -Dversion=2.1.2 -Dclassifier=mule-plugin -Dpackaging=jar -Dfile=MindTelligentPayloadProcessorForMule-2.1.2-mule-plugin.jar

Replace the JAR file with the name of your JAR file.

 
Once your connector is working, you can try it in Anypoint Studio.
Go to your Mule app pom.xml file and add the connector as a <dependency> with its groupId, artifactId, and version.
Example: Dependency
 
 ... <dependencies> ... <dependency> <groupId>com.mindtelligent.connect</groupId> <artifactId>MindTelligentPayloadProcessorForMule </artifactId> <version>2.1.2</version> <classifier>mule-plugin</classifier> </dependency> ... </dependencies> ...


You are Ready to use the Connector

Oracle Cloud-Containerized development with Docker (Part 2)

Oracle Cloud-Containerized development with Docker (Part 2)

To tread further on this thread, you must have done the following:

  • Create a compartment
  • Create a Virtual Cloud Network. 
Please refer to Part 1 of this thread

Add a Security List
  
A security list provides a virtual firewall for an instance, with ingress and egress rules that specify the types of traffic allowed in and out. Each security list is enforced at the instance level. However, you configure your security lists at the subnet level, which means that all instances in a given subnet are subject to the same set of rules. The security lists apply to a given instance whether it's talking with another instance in the VCN or a host outside the VCN.



  • In the DockerVCN network click on Security Lists
 




  •  Click on Default Security List for DockerVCN
 



  •  Click Add Ingress Rule





  •  In the Pop up dialog Enter the following and then click the Add Ingress Rule button.
 


  • When completed your Ingress Rules should look like:


How IdP Groups Are Tied to Databricks Groups (Unity Catalog)

  🔗 How IdP Groups Are Tied to Databricks Groups (Unity Catalog) 🔑 Key Principle (Read This First) Databricks does NOT “map” IdP groups...