William Dutton's Blog

How to find Private Network Load Balancer (NLB) ALBv2 IP4 addresses for adding to security group to allow health checks to pass

2023-02-02T08:59:00.004+10:00

Use case:

My Target Groups attached to my Load Balancer route to a private internal port that is not on the security group white listing. Due to Network Load Balancer's use of the EC2 connected Security Group. I don't want to whitelist the private internal port to the internet for possible bypass.

This can be done manually in the aws console by finding the ENI by looking up the ELB (type)/(name)/(random string) getting the private ip4 and then adding them manually to the security group.

But this defeats the purpose of having a cloudformation script that automatically stands up everything to work.

in this example we want to get the private ip4 address of a ELBv2 (network). Full working example: https://github.com/qld-gov-au/quickstart-atlassian-bitbucket/blob/d6ebe59b5ccdd204a7edc72ab6f0f89d575ac6f8/templates/quickstart-bitbucket-dc.template.yaml

(non-gist version below)

#Network Load Balancer health checks, need internal ip to approve connectivity

InternalNLBIp4List:
  DependsOn: NetworkLoadBalancerELB2
  Type: Custom::InternalNLBIp4ListCollector
  Version: 1.0
  Properties:
    ServiceToken: !GetAtt InternalNLBIp4ListCollector.Arn
    ELBv2Arn: !Ref NetworkLoadBalancerELB2
    StackName: !Ref 'AWS::StackName'
InternalNLBIp4ListCollector:
  Type: "AWS::Lambda::Function"
Properties:
    Handler: index.lambda_handler
    Role: !GetAtt InternalNLBIp4ListCollectorExecutionRole.Arn
    Runtime: python3.7
    Timeout: 120
    Code:
      ZipFile: |
        import cfnresponse
        import boto3

        def lambda_handler(event, context):
          elbv2 = boto3.client('elbv2')
          ec2 = boto3.client('ec2')
          elb2arn = event['ResourceProperties']['ELBv2Arn']
          response = elbv2.describe_load_balancers(LoadBalancerArns=[elb2arn])
          name = response['LoadBalancers'][0]['LoadBalancerName']
          elbtype = response['LoadBalancers'][0]['Type']
          filters = [{'Name': 'description', 'Values': ['ELB '+ elbtype[0:3] + '/' + name + '*']}]
          eni_response = ec2.describe_network_interfaces(Filters=filters)
          ip_addresses = [eni['PrivateIpAddress'] for eni in eni_response['NetworkInterfaces']]
          ip_addresses_cidr = [eni['PrivateIpAddress'] + '/32' for eni in eni_response['NetworkInterfaces']]
          print (ip_addresses)
          responseData = {}
          responseData['PrivateIpAddresses'] = ip_addresses
          responseData['PrivateIpCidrAddresses'] = ip_addresses_cidr
          cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData)

InternalNLBIp4ListCollectorExecutionRole:
  Type: "AWS::IAM::Role"
Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
Statement:
        - Effect: Allow
          Principal:
            Service:
              - lambda.amazonaws.com
          Action:
            - sts:AssumeRole
    Path: "/"
Policies:
      - PolicyName: root
        PolicyDocument:
          Version: '2012-10-17'
Statement:
            - Effect: Allow
              Action:
                - logs:CreateLogGroup
                - logs:CreateLogStream
                - logs:PutLogEvents
              Resource: !Sub "arn:${AWS::Partition}:logs:*:${AWS::AccountId}:log-group:/aws/lambda/*InternalNLBIp4ListCollector*"
- Effect: Allow
              Action:
                - "elasticloadbalancing:DescribeLoadBalancers"
- "ec2:DescribeNetworkInterfaces"
Resource: "*"

#NLB ip's need to be whitelisted to allow health checks to pass
SecurityGroupIngressNLB:
  DependsOn:
    - InternalNLBIp4List
    - SecurityGroup
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !Ref SecurityGroup
    IpProtocol: "-1"
FromPort: -1
    ToPort: -1
    CidrIp: !Select [ 0, !GetAtt InternalNLBIp4List.PrivateIpCidrAddresses ]

SecurityGroupIngressNLB2: #ELB in 2 subnets, will have 2 ip's
DependsOn:
    - InternalNLBIp4List
    - SecurityGroup
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !Ref SecurityGroup
    IpProtocol: "-1"
FromPort: -1
    ToPort: -1
    CidrIp: !Select [ 1, !GetAtt InternalNLBIp4List.PrivateIpCidrAddresses ]

AWS Java Lambda package ways

2021-05-13T14:00:00.007+10:00

There is two ways to make Aws Java Lambda packages, one is to make a uber FAT jar with the shade plugin.

Many people are in two camps on this, as well as it causing major problems with duplicates and needing custom code to make things 'happy'.

An example of this is below

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <configuration>
        <createDependencyReducedPom>false</createDependencyReducedPom>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <transformers>
                    <transformer implementation="com.github.edwgiz.maven_shade_plugin.log4j2_cache_transformer.PluginsCacheFileTransformer">
                    </transformer>
                </transformers>
            </configuration>
        </execution>
    </executions>
</plugin>

The second way is to make a zip, it like a .war file in a way and still works, there is more maven configuration of setting up. This also allows a split to occur with the use of Lambda Layers (playing the lib's as a layer)

<!-- alternative way without needing to use shader -->
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <configuration>
        <finalName>${project.build.finalName}</finalName>
        <archive>
            <manifest>
                <addClasspath>true</addClasspath>
                <classpathPrefix>lib/</classpathPrefix>
                <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
                <addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
            </manifest>
        </archive>
    </configuration>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>3.1.2</version>
    <executions>
        <execution>
            <id>copy-dependencies</id>
            <phase>package</phase>
            <goals>
                <goal>copy-dependencies</goal>
            </goals>
            <configuration>
                <includeScope>runtime</includeScope>
                <outputDirectory>${project.build.directory}/lib</outputDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-assembly-plugin</artifactId>
    <version>2.5.4</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>single</goal>
            </goals>
            <configuration>
                <descriptors>
                    <descriptor>src/main/assembly/bin.xml</descriptor>
                </descriptors>
                <attach>false</attach>
            </configuration>
        </execution>
    </executions>
</plugin>

<!-- END alternative way without needing to use shader -->

This does require an assembly xml file placed at  src/main/assembly/bin.xml

<assembly>
    <id>aws-lambda-package</id>
    <formats>
        <format>zip</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <fileSets>
        <fileSet>
            <directory>${project.build.directory}${file.separator}lib</directory>
            <outputDirectory>lib</outputDirectory>
            <useDefaultExcludes>true</useDefaultExcludes>
        </fileSet>
        <!-- copy all classes -->
        <fileSet>
            <directory>${project.build.directory}${file.separator}classes</directory>
            <includes>
                <include>**</include>
            </includes>
            <outputDirectory>${file.separator}</outputDirectory>
        </fileSet>
    </fileSets>

</assembly>

Upload the zip instead of jar. All is good.

See https://github.com/awslabs/aws-serverless-java-container/commit/22edc6e65dcdefab46145f80c80dcb274b0eedd4

Junit 4 to Junit 5 upgrade

2021-05-04T10:04:00.005+10:00

So you started to upgrade from old junit 4 to the new junit 5 api spec.

Sadly the move the packages around. Its best to follow other docs out on the web for this like

https://www.baeldung.com/junit-filtering-tests

Now heres a gotcha, if you are upgrading your springboot tests from junit4 to junit 5 and are trying to use tags to drop external tests or slow tests and it is not working.

It might actually be the import Test class that is breaking it.

Replace import org.junit.Test; with import org.junit.jupiter.api.Test;

This should make everything work correctly again.

Then the command below can work.

mvn test -DexcludedGroups=ExternalIntegrationTest

Major change between spring-boot 2.2.x to 2.4.x with spring-cloud and spring-cloud-starter-aws-parameter-store-config (bootstrap.yml, profiles and more)

2021-04-20T08:47:00.001+10:00

So with the change in how versioning works. A change to how config is loaded by default and a split between cloud agnostic spring-cloud and vendor specific integration types. There has been a major shake up on how to upgrade to the next minor version of spring-boot.

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-parent</artifactId>

- <version>2.2.11.RELEASE</version>

+ <version>2.4.4</version>

</parent>

-<spring-cloud.version>Hoxton.SR5</spring-cloud.version>

+<spring-cloud.version>2020.0.2</spring-cloud.version>

We now need to include io.awspring.cloud:spring-cloud-aws-dependencies as its now not included in the upstream org.springframework.cloud:spring-cloud-dependencies

<dependency><scope>import</scope><groupId>org.springframework.cloud</groupId><artifactId>spring-cloud-dependencies</artifactId><version>${spring-cloud.version}</version><type>pom</type></dependency>

+ <dependency><scope>import</scope><groupId>io.awspring.cloud</groupId><artifactId>spring-cloud-aws-dependencies</artifactId><version>2.3.1</version><type>pom</type></dependency>

With dependencies we don't include spring-cloud-starter any more but with bootstrap.yml not being the 'default' way for loading we now need to include spring-cloud-starter-bootstrap to re-enable that functionality

- <dependency><groupId>org.springframework.cloud</groupId><artifactId>spring-cloud-starter</artifactId></dependency>

- <dependency><groupId>org.springframework.cloud</groupId><artifactId>spring-cloud-starter-aws-parameter-store-config</artifactId></dependency>

+

+ <dependency><groupId>org.springframework.cloud</groupId><artifactId>spring-cloud-starter-bootstrap</artifactId></dependency>

+

+ <dependency><groupId>io.awspring.cloud</groupId><artifactId>spring-cloud-starter-aws-parameter-store-config</artifactId></dependency>

You now can't have spring profiles load other profiles, you can do profile groups, but that is limited if you wanted something dynamic like enabling proxy settings. (this is maven profile adding a spring profile)

<id>local-proxy</id>

<name>env.http_proxy</name>

</property>

</activation>

- <springBootRunArguments>--spring.profiles.include=PROXY,</springBootRunArguments>

+ <springBootRunArguments>--spring.config.import=classpath:application-PROXY.yml</springBootRunArguments>

</properties>

<build>

<artifactId>maven-surefire-plugin</artifactId>

<version>${maven-surefire-plugin.version}</version>

- <spring.profiles.include>PROXY</spring.profiles.include>

+ <spring.config.import>classpath:application-PROXY.yml</spring.config.import>

</systemPropertyVariables>

</configuration>

</plugin>

<artifactId>maven-failsafe-plugin</artifactId>

- <spring.profiles.include>PROXY</spring.profiles.include>

+ <spring.config.import>classpath:application-PROXY.yml</spring.config.import>

</systemPropertyVariables>

</configuration>

</plugin>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-maven-plugin</artifactId>

- <jvmArguments>-Dspring-boot.run.jvmArguments='-Dspring.profiles.include="PROXY"'</jvmArguments>

+ <jvmArguments>-Dspring-boot.run.jvmArguments='-Dspring.config.import="classpath:application-PROXY.yml"'</jvmArguments>

</configuration>

</plugin>

</plugins>

</build>

</profile>

Any properties files you may have loaded that used "spring.profiles.include" can't be used any more with 2.4.x+ (unless you enabled legacy which will be going away after 2.6.x? version i believe)

-spring.profiles.include: defaults

+spring.config.import: classpath:application-defaults.yml

Also ensure that your bootstrap.yml has

aws.paramstore.enabled: true

It is on by default (But as its nowvin your config you can set it to false for local run's)

For more info, see:

https://spring.io/blog/2020/08/14/config-file-processing-in-spring-boot-2-4

https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#boot-features-external-config-files-profile-specific

https://github.com/awspring/spring-cloud-aws/blob/2.3.x/docs/src/main/asciidoc/parameter-store.adoc

https://github.com/awspring/spring-cloud-aws/blob/2.3.x/spring-cloud-starter-aws-parameter-store-config/src/test/java/io/awspring/cloud/autoconfigure/paramstore/AwsParamStoreBootstrapConfigurationTest.java

https://stackoverflow.com/questions/64907675/including-profiles-in-spring-boot-2-4-0-version

https://stackoverflow.com/questions/64994034/bootstrap-yml-configuration-not-processed-anymore-with-spring-cloud-2020-0

https://stackoverflow.com/questions/65063402/why-bootstrap-properties-is-ignored-by-spring-cloud-starter-config

https://docs.awspring.io/spring-cloud-aws/docs/2.3.0/reference/html/index.html#integrating-your-spring-cloud-application-with-the-aws-parameter-store

Something I want to look into is setting up something like https://github.com/localstack/localstack within a maven project so that param store loading can be tested in pdev instead of being caught in an aws dev/test environment.

Spring cloud param store Hoxton.SR6 to Hoxton.SR8 how to run locally

2020-10-07T15:23:00.004+10:00

So your using aws param store to configure your application when deployed to docker/elastic beanstalk but ran into issues on doing testing after updates to remove cve issues.

before hand

at or before Hoxton.SR6 you only needed in test/resources/bootstrap.yml

aws.paramstore.enabled: false

But after updating to

<spring-cloud.version>Hoxton.SR8</spring-cloud.version>

<dependency><scope>import</scope><groupId>org.springframework.cloud</groupId>

<artifactId>spring-cloud-dependencies</artifactId>

<version>${spring-cloud.version}</version><type>pom</type></dependency>

<dependency><groupId>org.springframework.cloud</groupId>

<artifactId>spring-cloud-starter</artifactId></dependency>
<dependency><groupId>org.springframework.cloud</groupId>

<artifactId>spring-cloud-starter-aws-parameter-store-config</artifactId></dependency>

it now throws logs of errors and fails to boot, this sucks :'(

errors are:

see Log file output

lets work out where it broke. We first need to see what jar's were imported by our spring-cloud-dependencies

we do this by looking at

https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-dependencies/Hoxton.SR6 to https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-dependencies/Hoxton.SR8

And we notice that that the aws cloud version incremented from 2.2.2 to 2.2.4

to see what changed we can do this on github by visiting the link below

https://github.com/spring-cloud/spring-cloud-aws/compare/v2.2.2.RELEASE...v2.2.4.RELEASE

What was added in the doc's which looked like it might be our problem.

|aws.paramstore.region | | If region value is not null or empty it will be used in creation of AWSSimpleSystemsManagement.

|aws.secretsmanager.region | | If region value is not null or empty it will be used in creation of AWSSecretsManager.

On application startup, for its internal purposes Spring Cloud AWS performs a check if application runs in AWS cloud environment

by using `EC2MetadataUtils` class provided by AWS SDK. Starting from version 1.11.678, AWS SDK logs a warning message with exception when this check is made outside of AWS environment.

This warning message can be hidden by setting `ERROR` logging level on `com.amazonaws.util.EC2MetadataUtils` class.

so it seems we now need to set a region to block auto region lookup even when we have the paramstore disabled, we also need to do it for the stack and region lookup outside of paramstore.

test/resources/bootstrap.yml

aws:
  paramstore:
    enabled: false
    fail-fast: false
    region: "ap-southeast-2"
  secretmanager:
    region: "ap-southeast-2"

cloud:
  aws:
    region:
      auto: false
      static: "ap-southeast-2"
    stack:
      auto: false

Now that we did this, we are now not crashing. Awesome :D

Hope this helps others including future me.

AWS Param store put with urls via CLI

2019-12-16T14:59:00.001+10:00

seems the CLI will follow urls given to it. which for param store is not very useful when you want to store the website address instead.

the work around.

aws configure set cli_follow_urlparam false

per
https://github.com/aws/aws-cli/issues/1475

then this works

aws ssm put-parameter --name "/myappconfig/" --type "String" --value "http://enhanceindustries.com.au/" --region ap-southeast-2

Australian API Standards

2019-12-13T10:58:00.001+10:00

This just came accross my desk.

Looks like an awesome set of specs to build to.

https://api.gov.au/standards/national_api_standards/index.html

AWS Cloudformation how to swap between Ip's and AliasTarget via conditions

2019-12-13T09:02:00.004+10:00

Below Allows you to swap between cloudfront and a static ip address.

Note the new line after the - for the If statement, this tells yaml that this is an object array, you need to replicate everything in the object array as cloudformation does not merge objects arrays together.

WebDNSRecordSet:
  Type: AWS::Route53::RecordSet
  DependsOn:
    - DistributionConfig
  Properties:
    Fn::If:
      - IsIPRestricted
      -
        HostedZoneName: !Sub "${Domain}."        ResourceRecords:
          - 123.123.123.123
        TTL:  '900'        Name: !Sub "www.${Domain}."        Type: A
      -
        HostedZoneName: !Sub "${Domain}."        AliasTarget:
          DNSName:
            Fn::GetAtt: [DistributionConfig, DomainName]
          HostedZoneId: "Z2FDTNDATAQYW2"        Name: !Sub "www.${Domain}."        Type: A

https://www.json2yaml.com/

When run through https://www.json2yaml.com/

you get 

{
  "WebDNSRecordSet": {
    "Type": "AWS::Route53::RecordSet",
    "DependsOn": [
      "DistributionConfig"
    ],
    "Properties": {
      "Fn::If": [
        "IsIPRestricted",
        {
          "HostedZoneName": "${Domain}.",
          "ResourceRecords": [
            "123.123.123.123"
          ],
          "TTL": "900",
          "Name": "www.${Domain}.",
          "Type": "A"
        },
        {
          "HostedZoneName": "${Domain}.",
          "AliasTarget": {
            "DNSName": {
              "Fn::GetAtt": [
                "DistributionConfig",
                "DomainName"
              ]
            },
            "HostedZoneId": "Z2FDTNDATAQYW2"
          },
          "Name": "www.${Domain}.",
          "Type": "A"
        }
      ]
    }
  }
}

Cloudfront With Squiz Edge being an origin

2019-07-26T08:28:00.000+10:00

Just wanted to share an insight on trying to do an Squiz CMS overlay beside another product that has no CMS features. I.E a homepage of a COTS solutions.

Problem space:
I want to have the squiz edge as an origin in cloudfront so that i can have some pages fully managed by the content team without developer intervention.

Issues encounted
cloudfront started returning 502 errors which their documentation relates to ssl issues.
Squiz the company needs to be contacted to do an update on their squiz edge system to acknowledge your hostname as well as setup required inside squiz matrix.

Outcome:

Cloudfront has two rules for SSL passthrough,

Rule 1: Origin Domain Name you request against must match ssl cert

Rule 2: If Rule1 fails, Host header must match ssl cert.

If my front end domain is zyx.

My Origin Domain Name is lpo

If the origin returns ssl cert zyx or lpo it will pass. If it passes abc it will fail.

Now in relation to Squiz.

When we do a low level ssl cert check against their staging edge network if the server name is the hostname, we get *.squizedge.net

If its (valid configured domain without custom ssl cert) we get *.clients.net.au if its (invalid domain) we get *.squizedge.net

What we want is to have *.squizedge.net to be provided to as instead of *.clients.squiz.net cert.

Below is how to test.

openssl s_client -showcerts -servername staging.squizedge.net -connect staging.squizedge.net:443
depth=0 C = AU, ST = New South Wales, L = Sydney, O = SQUIZ PTY LTD, CN = *.squizedge.net

openssl s_client -showcerts -servername (valid domain without custom ssl cert) -connect staging.squizedge.net:443

depth=0 C = AU, ST = New South Wales, L = Sydney, O = Squiz Australia Pty. Ltd., CN = *.clients.squiz.net

openssl s_client -showcerts -servername (invalid domain) -connect staging.squizedge.net:443

depth=0 C = AU, ST = New South Wales, L = Sydney, O = SQUIZ PTY LTD, CN = *.squizedge.net

openssl s_client -showcerts -servername (valid domain with custom ssl cert) -connect staging.squizedge.net:443

(valid ssl cert depth=0 provided)

ALSO

do ensure you are picking tls1.1 or higher as ssl3 handshake with squizedge is an instant deny

openssl s_client -showcerts -connect staging.squizedge.net:443 -servername (valid domain without custom ssl cert) -ssl3
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : SSLv3
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1564975196
Timeout : 7200 (sec)
Verify return code: 0 (ok)
---

AWS: Why does Auto Minor Version Upgrade Flag not upgrade to latest minor Version of Database.

2019-07-17T10:17:00.001+10:00

Shout out to Asmita V. for this answer.

------------------------------------------
PROBLEM DESCRIPTION
------------------------------------------

Why were the RDS PostgreSQL instances not automatically upgraded to the latest Minor Version available even though the Auto Minor Version Upgrade Flag is enabled for your instances.

------------------------------------------
RESPONSE
------------------------------------------

The reason why your instances was not upgraded to these minor version is that AMVU will only upgrade the engine version for your RDS instance if the current engine version is being deprecated or the new one contains very important cumulative bug fixes and an upgrade is absolutely necessary.

Please note that while we highly recommend that you perform an upgrade to 10.9, this upgrade will not happen automatically as of now using AMVU as the automatic upgrades happen only when absolutely necessary and you can also view such actions using describe-pending-maintenance-actions command.

If there is an auto minor version upgrade scheduled as a maintenance, please be assured that you will get a separate notification explicitly mentioning the same. Currently, in this case the upgrade will have to be applied manually.

Further, at your end, you can check if the minor version upgrade will happen automatically or not by using the below CLI command:

$aws rds describe-db-engine-versions --output=table --engine postgres --engine-version 10.6

Output:
||+-------------------------------------------------------------------------------------------+||
|||                                    ValidUpgradeTarget                                     |||
||+-------------+---------------------+-----------+----------------+--------------------------+||
||| AutoUpgrade |     Description     | Engine   | EngineVersion | IsMajorVersionUpgrade   |||
||+-------------+---------------------+-----------+----------------+--------------------------+||
||| False      | PostgreSQL 10.7-R1 | postgres | 10.7          | False                   |||
||| False      | PostgreSQL 10.9-R1 | postgres | 10.9          | False                   |||
||| False      | PostgreSQL 11.1-R1 | postgres | 11.1          | True                    |||
||| False      | PostgreSQL 11.2-R1 | postgres | 11.2          | True                    |||
||| False      | PostgreSQL 11.4-R1 | postgres | 11.4          | True                    |||
||+-------------+---------------------+-----------+----------------+--------------------------+||

As you can see from the above output, for 10.6 version "AutoUpgrade" column is marked as "False" for all minor version upgrade (either 10.7 or 10.9) . So, it has to be done manually. Please make sure to upgrade to the latest minor version (10.9) so that you wont be prone to any security vulnerabilities as per the following notice:

[+] https://www.postgresql.org/about/news/1949/

AWS: How should we upgrade PostgreSQL 10.6 to 10.9 version of CloudFormation-controlled RDS instances

2019-07-17T10:15:00.001+10:00

Shout out to Asmita V. for this answer.

Upgrade RDS PostgreSQL instances from 10.6 to 10.9 and in this process you want to understand that whether setting the "AllowMajorVersionUpgrades" flag in the CloudFormation template be sufficient and if during the process, existing instances will get replaced.

------------------------------------------
RESPONSE
------------------------------------------

Upgrading the PostgreSQL instance from version 10.6 to 10.9 is a minor version upgrade and hence would not require you to change the value of the parameter "AllowMajorVersionUpgrades" in your cloudformation template.

In order to upgrade your instances from 10.6 to 10.9 you can make a modification in your CFN stack by just specifying the "Engine Version" as 10.9 instead of 10.6 in your template. There is no replacement of the existing instance and hence there will be no loss of data. The existing instances will go to a "Modifying" State.

In order to confirm this behavior, I tried upgrading resources in my test environment and following are my observations:

---------------------------------------------
Testing
---------------------------------------------
Please refer the following set of steps that I took in order to upgrade my instance to 10.9 from 10.6:

Sample Stack
============================

{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "PostgreSQL RDS Template ",
"Resources": {
    "pgDB" : {
      "Type" : "AWS::RDS::DBInstance",
      "Properties" : {
        "DBName" : "mydb",
        "DBInstanceClass" : "db.t2.small",
        "AllocatedStorage" : "20",
        "Engine" : "postgres",
        "EngineVersion": "10.6",
        "MasterUsername" : "username",
        "MasterUserPassword" : "password",
        "AutoMinorVersionUpgrade": false
      }
    }
}
}

Step 1: Make Changes to the existing template
=======================================

1. On the Stacks page of the AWS CloudFormation console, click the name of the stack that you want to update. https://console.aws.amazon.com/cloudformation
2. Select the Template tab and select View on Designer.
3. Modify the CFN Stack : "EngineVersion": "10.9"
        {
          "AWSTemplateFormatVersion" : "2010-09-09",
          "Description" : "PostgreSQL RDS Template ",
          "Resources": {
            "pgDB" : {
              "Type" : "AWS::RDS::DBInstance",
              "Properties" : {
                "DBName" : "mydb",
                "DBInstanceClass" : "db.t2.small",
                "AllocatedStorage" : "20",
                "Engine" : "postgres",
                "EngineVersion": "10.9",
                "MasterUsername" : "username",
                "MasterUserPassword" : "password",
                "AutoMinorVersionUpgrade": false
              }
            }
          }
        }
4. Validate you template.
5. Select Save. Save your template to S3 bucket.
6. Click on Save and copy the URL.

Ex. https://s3-external-1.amazonaws.com/cf-templates-us-east-1/template1

Step 2: Create Change Set for Current Stack
=======================================

1. On the Stacks page of the AWS CloudFormation console, click the name of the stack that you want to update. https://console.aws.amazon.com/cloudformation
2. Go to Stack Actions and select "Create change set for current stack".
3. Select "Replace current template"
4. Input the URL that was copied in the Step 1:6.
5. On the Review page, click on Create Change Set.
6. In the preview page, under the Changes you will notice "Modify" under the Action column.
7. Click on Execute.

You can now check on the RDS Console. The status of the RDS instance would have gone to "Upgrading".

However, please note that engine version upgrade (major or minor) is always associated with some amount of downtime.

Even if your DB instance is in a Multi-AZ deployment, both the primary DB instance and standby DB instances are upgraded. The writer and standby DB instances are upgraded at the same time, and you experience an outage until the upgrade is complete.

Therefore it is always recommended to plan your upgrades in the non business hours.

[+] Upgrading the PostgreSQL DB Engine for Amazon RDS - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.html

[+] Modifying a DB Instance Running the PostgreSQL Database Engine - Settings for PostgreSQL DB Instances - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ModifyPostgreSQLInstance.html#USER_ModifyInstance.Postgres.Settings

---------------------------------------------
Conclusion
---------------------------------------------
Therefore, as per my testing, I can confirm that there will be no replacement of the existing instance during the process of upgrading PostgreSQL instance from v10.6 to 10.9.
You may also go ahead and follow the steps given above in order to upgrade your instances.

---------------------------------------------
REFERENCES
---------------------------------------------

[+] Updating Stacks Using Change Sets - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-changesets.html

[+] Creating a Change Set - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-changesets-create.html

Alexa Skill

2019-06-02T08:50:00.001+10:00

Tried my hand at making an Alexa Skill,

Give it a go and let me know.

AUS https://www.amazon.com.au/William-Dutton-Kids-Radio-Australia/dp/B07SBT69SC/

US https://www.amazon.com/William-Dutton-Kids-Radio-Australia/dp/B07SBT69SC/

Cloudformation long form to short form variable substitution

2019-05-30T10:30:00.002+10:00

In cloud formation, you don't need to inject Region or Account Id into your template unless you are referencing something external.

If you use Join's maybe look at swapping them for Sub if you are not adding anything or just dealing with a string e.g.

from

SourceArn: !Join [ '', ["arn:aws:execute-api:", Ref: AWS::Region, ":", Ref: AWS::AccountId, ":*"] ]

SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:*"

the !Sub form is so much easier to read.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/pseudo-parameter-reference.html

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-sub.html

AWS Lambda with SSM Paramater store variables

2019-05-29T10:44:00.001+10:00

So you have used Spring Cloud SSM access for elastic beanstalk and docker but want to get into lambda with same nice config setup.

Sadly Spring framework is a bit too heavy for lambda and they suggest Dagger 2 or Guice. This guide is not about static/dynamic wiring of beans together but on getting parameters into your beans.

In the old days you had to place all of your environment path or via a file in s3 which you had to parse. Now this was ok for simple things but it was not secure for secrets aka database passwords or other sub-systems external to aws.

So most people rolled their own kms decryption system when it loads in lambda, ok that's nice but its still not easy to test locally vs on the cloud.

This code was inspired by the spring-cloud-aws project. But without the spring bits. (yep this does not do the nice overlays with dynamic profile activation etc but its still a good steps away from having all of your properties on the environment path)

Do ensure you have iam policy to allow ssm access (here an excerpt from cfn), the SSMKey is the arn for the kms key to decrypt your paramaters (if they are encrypted, else you can drop this action)

LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Ref LambdaRoleName
AssumeRolePolicyDocument:
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Version: '2012-10-17'
Path: /
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName:
Fn::Join:
- '-'
- - Ref: Product
- Application-Lambda-Policy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- ssm:DescribeParameters
Resource: "*"
- Effect: Allow
Action:
- ssm:GetParameters
- ssm:GetParameter
- ssm:GetParametersByPath
Resource:
- !Sub "arn:aws:ssm:*:*:parameter/config/${Service}"
- !Sub "arn:aws:ssm:*:*:parameter/config/${Service}/*"
- !Sub "arn:aws:ssm:*:*:parameter/config/${Service}_*"
- !Sub "arn:aws:ssm:*:*:parameter/config/${Service}_*/*"
- Effect: Allow
Action:
- kms:Decrypt
Resource:
- Ref: SSMKey

AWS AutoScalingGroup to Route53 update record function via Lambda

2019-03-31T09:32:00.001+10:00

Sometimes you want to have an Auto Scaling Group keep a single server online and you don't want to worry about connecting EIP's to them or have them kept as a Pet which needs to be kept at all costs.

Or you need to allow UDP access which the Elastic Load Balancers (ELB) and Network Load Balancers (NLB) don't allow. This is for you.

What this does is listen to a Simple Notification Service (SNS) to any published events which the ASG would send for adding an instance to the pool or terminating an instance to the pool. It then queries the asg looking for the tag DomainMeta and then with the list of ec2 instances it goes and collects the public ip address and goes to the route53 zone that is recorded and updates the domain attached.

The tag should be in the format DomainMeta: : I.E DomainMeta : Z10MWC8V7JDDX1:www.mydomain.com
Where the first part is the hosted zone it needs to end the command to and the second part is the a record it is going to change.

This is based on the work that objectpartners.com did back in 2005. I've improved it to include security so that only one hosted zone is looked after or allows full account control if you are 100% in control of the tags on the ASG's.

This could easily be updated to include a coma delimited list on the tag to update multiple a records if required.

Please note: If the last instance is taken out of the pool the old ip address will be left since it route53 records can't be null/empty.

AWS Secrets Manager Cross account for SES credentials

2018-12-14T15:31:00.001+10:00

If you have external systems sending emails, it might only know of stmp with username and password.

When you want to get CIS compliance in AWS having an access keys last longer than 90 days takes you out of compliance, as such you need to change this access key every 3 months. This seems to be a bit of pain.

Luckily AWS released Secrets Manager to allow as you guessed it secrets, to be rotated.

This is an extension on https://aws.amazon.com/blogs/security/how-to-access-secrets-across-aws-accounts-by-attaching-resource-based-policies/ since blog post is a bit too cryptic for actual implementation with least privileges.

To do that you firstly need to create a KMS key to use since the defaults don't allow addition policies to be added.

For extra security, the SES email sending service is centralised but the application needing to use the credentials are in another part of your AWS Organisation accounts. This is good security practice as you can then remove access to a kms key if a downstream account was compromised.

Here is an example policy which would allow any account in your Organisation to access the key as well as the role which the lambda function will use to rotate the secret

Please note: going with "aws:PrincipalOrgID": "${YOUR-ORG-ID from organisations}" means that any account in your Org will have access to encrypt or decrypt with this key if they also have privileges to see this key (this means * * Administrators would by default have this, could could be changed to Principals with role names but then that next gotcha is that if the role is deleted and recreated, the connection to this kms policy would be disconnected)

Zip up this script and call it aws-key-rotation-lambda.zip

upload this to s3 inside your cloudformation bucket since we need to reference it for the next cloudformation script.

key-rotation-lambda.yml will generate the rotation service which Secrets Manager will use.

Now that all the building blocks are in place your ready to have a user and new secret created.

Use this next script to generate the iam user and Secrets Manager. Once initialised, it will rotate the secret straight away.

All it needs now is to have the correct groups/policies added for SES raw email sending.

To this test, get on the cli and change into another account which is not what this is installed into and user the ARN of the secret to call it

aws secretsmanager get-secret-value --secret-id arn:aws:secretsmanager:ap-southeast-2:111222333444:secret:smtp-trial-X4Q0Sp --version-stage AWSCURRENT

With the SNS queue, you can now hook up a lambda function to trigger in the other accounts to go update where ever the stmp credentials are set.

This script will give 60 days per access key and generate a new key every 30 days. So if the system fails to update the first time there is a grace period to change over.

regex for gedit

2011-11-28T09:25:00.000+10:00

I was needing to do some regular expression changes on some database output into a java test script and found that someone updated gedits regex plugin to be even better.

http://halfhourhacks.blogspot.com/2008/03/gedit-regular-expression-plugin.html

menu lane goes live

2011-05-18T15:15:00.004+10:00

one of my friends projects which I've been doing a lot of work on has gone live. Its a website in which we can see the menu of the restaurant you might want to visit and have a general price range of what you should be paying, the website is http://www.menulane.com.au

new website

2011-01-17T10:15:00.000+10:00

I just released the new nowsim site, still alot of work to do, but its getting there :D

nowism.tv

New website

2010-10-14T15:53:00.002+10:00

Just finished off putting together a new landscaping website for a good friend of a friend. seascapeslandscaping.com.au

New workout music shop 10minutetracks.com

2010-09-28T09:21:00.003+10:00

Lately I have been following the "The 10-Minute Total Body Breakthrough" which is a very good book in getting into shape with a minimal amount of time.

My friend Jeremy asked me to help him create this website to sell the music he and his friend created follows the books workout pattern.

10 Minute Tracks

new pub on the gold coast

2010-09-09T08:28:00.001+10:00

This pub/hotel was taken over by the tbhotels and it now has a new website railwayhotelgc.com.au

maven, ant, ivy where should we all go?

2010-08-04T16:09:00.003+10:00

well at work I've been working on making our chopped up monolithic application more manageable.

the problem is that we use to have 1 project, now we have around 11 or so. each has their on library folder. At this point in time its an easy thing to deal with, in a years time when we start getting jars updated, its going to be a big pain in the a#se.

so I've been learning alot about maven and have started to implement it into our small applications and now I'm up to the corner stone of the project.

pull all the artifacts together and make an install script.

as its a custom piece of code that users ant and bash with uuencode and many other bits and pieces, i was not going to even try making a maven plugin to make my install.sh file.

i looked into ivy and found that it does not use the same local repository structure as maven but can use nexus.. so not good.

but then i found http://ptrthomas.wordpress.com/2009/03/08/why-you-should-use-the-maven-ant-tasks-instead-of-maven-or-ivy/

it seems you can pull in maven dependency's into ant and use them there.

my current thinking is to use the maven ant task to extract all the war's and config files in the maven local or remote repository and let ant/bash do its magic to make my custom install.sh file :)

Code To test internal generic methods with user code in them

2010-04-16T16:42:00.001+10:00

How does one test this code


 public String SendMessage(String request) throws JMSException {

   TextMessage reply = (TextMessage) jmsTemplate.execute(new ProducerCallback() {
    public Object doInJms(Session session,MessageProducer producer) throws JMSException {
     TemporaryQueue queue = session.createTemporaryQueue();
     TextMessage message = session.createTextMessage();
     message.setText("Some Text");
     message.setJMSReplyTo(queue);
     producer.send(message);
     return session.createConsumer(queue).receive(10000);
    }
   });

   if (reply != null){
   // System.err.println("Message recieved");
   } else{
   // System.err.println("*** NOT received");
   }
   
   return reply.getText();

 }

here is my junit mockito 1.8 test


import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageProducer;
import javax.jms.TemporaryQueue;
import javax.jms.TextMessage;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.mockito.invocation.InvocationOnMock;
import org.mockito.stubbing.Answer;

import static org.mockito.Mockito.*;
import org.springframework.jms.core.JmsTemplate;
import org.springframework.jms.core.ProducerCallback;

import com.sun.messaging.jms.Session;

public class PrivateMessageSenderTest {

   PrivateMessageSender classUnderTest;
   JmsTemplate mockJmsTemplate;
   Session mockSession;
   MessageProducer mockMessageProducer;
   TemporaryQueue mockTemporaryQueue;
   TextMessage mockTextMessage;
   @Before
   public void setUp() throws Exception {
       classUnderTest = new OpenMQPrivateMessageSender();
     
       mockJmsTemplate = mock(JmsTemplate.class);
       mockSession = mock(Session.class);
       mockMessageProducer = mock(MessageProducer.class);
       mockTemporaryQueue = mock(TemporaryQueue.class);
       mockTextMessage = mock(TextMessage.class);
     
       /** Boiler plate mock code to inject mock session and messageProducer into
        *  Generic Spring ProducerCallback class which has our internal code
        *  To test
       **/
       when(mockJmsTemplate.execute((ProducerCallback)anyObject())).thenAnswer(
               new Answer() {
                   public Object answer(InvocationOnMock invocation) throws JMSException {
                       Object[] args = invocation.getArguments();
                       //Object mock = invocation.getMock();      
                       return ((ProducerCallback)args[0]).doInJms(mockSession, mockMessageProducer);
                    }
                });
     
       when(mockSession.createTemporaryQueue()).thenReturn(mockTemporaryQueue);
       when(mockSession.createTextMessage()).thenReturn(mockTextMessage);

its missing the tests for it, but this pretty much sets it up to set mocks into the internal method.

best ssh code snippet

2010-01-25T11:13:00.002+10:00

This code snippet ssh's into the account you want to add your public key to.

The usual way to do that is to copy the key over then follow the key to the server and append it to the end.

e.g.


scp ~/.ssh/id_dsa.pub user@host:.
ssh user@host 'cat id_dsa.pub >> ~/.ssh/authorized_keys'

You can do it all in one line.


ssh user@host 'mkdir ~/.ssh;echo '`cat ~/.ssh/id_dsa.pub`' >> ~/.ssh/authorized_keys'

Be careful with the >> if you accidently do a single >, then bye bye original file. You will be able to log in but everyone else who had their key in it cant as your key is now only in that file.

this snippet came from http://snippets.dzone.com/posts/show/2335