LittleRipples Project

SlackAPIPostOperator unable to send to private slack channel.

2023-02-24T14:47:00.000+00:00

I am using SlackAPIPostOperator from an Airflow Dag. And have successfully made a bot that will send messages to my Slack Channel

slack_post = SlackAPIPostOperator(

    task_id="slack_notification",

    channel="#my_slack_channel",

    username="airflow",

    text="A test message from Airflow. Please ignore",

    token="#######",

    dag=dag,

)

However, when I use the same logic on a Private channel it wouldn't work. And I couldn't see my bot to be able to 'invite' it to the channel.

To fix this ...

To grant your bot permission to join private channels, you will need to ensure that it has the channels:join and channels:read OAuth scopes. Here’s how you can check and modify your bot’s scopes:

Go to the Slack API website and navigate to the “OAuth & Permissions” section of your app’s settings.
Scroll down to the “Scopes” section and check that your bot has the channels:join and channels:read scopes.
If the scopes are not already added, click on the “Add an OAuth Scope” button and select the scopes from the dropdown list.
After adding the scopes, make sure to re-install your app in the workspace to apply the changes.

After , that I had to re-invite the Bot to the Private Channel ( through the add people ) and it worked.

In Wagtail GeoLocation Widget how to set a Default location ( and set zoom )

2023-01-04T10:42:00.000+00:00

To set the default map location in the wagtailgeowidget package, you can define the GEO_WIDGET_DEFAULT_LOCATION setting in your Django settings file.

Here is an example of how to do this:


GEO_WIDGET_DEFAULT_LOCATION = {"lat": 50.5230, "lng": -4.6558}

This will set the default map location to the coordinates 50.5230° N, 4.6558° W (Bodmin, Cornwall, United Kingdom).

The GEO_WIDGET_DEFAULT_LOCATION setting should be a dictionary with keys 'lat' and 'lng' representing the latitude and longitude, respectively. The values should be float values.

You can also set the default map location dynamically based on the value of a field in your model by defining a callable as the value of the GEO_WIDGET_DEFAULT_LOCATION setting.

Here's an example of how to do this:

from django.conf import settings

def get_default_location():
    # Code to retrieve default location from model goes here
    return {"lat": 50.5230, "lng": -4.6558}

GEO_WIDGET_DEFAULT_LOCATION = get_default_location

This will set the default map location to the value returned by the get_default_location function.

Django - Grabbing a JSON object from QueryDict

2022-05-31T11:03:00.001+01:00

What I'm doing here is passing some JSON through the 'rest.framework' . The problem is that when using 'request.json' I'm seeing the following being returned.

<QueryDict: {'': ['{"time":{"0":"06:00","1":"07:00","2":"08:00","3":"09:00","4":"10...

What I need to do is extract the JSON out . To do this , the following works.

list(data.values())[0]

M1 Chip issue with AWS Lambda SAM Layers

2022-05-25T17:43:00.002+01:00

Just a quick note , if you've searched and found this blog then I expect you already know deep down that the M1 chip is your issue. There is a solution though. Let me explain.

So, what we've been doing is using Lambda Layers to install Python packages on our Lambda installation. However, when putting the same code on my Mac ( M1 Chip ) it doesn't work !

https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

It does work if we take the zip file from the AMD64 chipped Mac , and put that on my Mac and proceed.

That is the solution we're having to use at the moment , the zip needs to be compiled on the AMD64 chip , doing it on the M1 chip won't work.

Could not 'pipenv install email' - 'python setup.py egg_info did not run successfully.' error.

2022-05-17T11:14:00.000+01:00

Here's the error code I was getting.

Installing email... Error: An error occurred while installing email! Error text: Collecting email Using cached email-4.0.2.tar.gz (1.2 MB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'error' error: subprocess-exited-with-error × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [1 lines of output] ERROR: Can not execute `setup.py` since setuptools is not available in the build environment. [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. This is likely caused by a bug in email. Report this to its maintainers. ✘ Installation Failed

The recommended solution across the web seemed to be to run the following.

pipenv install --upgrade setuptools

However, this didn't work for me. I needed to use the '--pre' tag.

pipenv install --pre email

This now works, the version I had the issue with was 'email-4.0.2'

So, if you have a 'email-4.0.2 won't pipenv install' issue then I hope this post helps.

Wordpress installation on Docker - How to access the Database

2022-04-07T14:49:00.005+01:00

After running through this blog on how to create a Wordpress site on Docker. ( Which works excellently ) https://davidyeiser.com/tutorials/docker-wordpress-theme-setup

After doing so, I couldn't quite work out what details I needed to use in Sequel Pro / Sequel Ace.

There's a line that I needed to add to the Wordpress DB setup though, and that was to add the ports .

    ports:
      - "3366:3306"

So my entire file looks like this.

version: '3'

services:
  db:
    image: mysql:5.7
    volumes:
      - db_data:/var/lib/mysql
    ports:
      - "3366:3306"
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: somewordpress
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: wordpress

  wordpress:
    depends_on:
      - db
    image: wordpress:5.1.1-php7.3-apache
    ports:
      - "8000:80"
    restart: always
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: wordpress
    working_dir: /var/www/html
    volumes:
      - ./wp-content:/var/www/html/wp-content
      - ./uploads.ini:/usr/local/etc/php/conf.d/uploads.ini
volumes:
  db_data:

And then I can use

Vue.js not building correctly in 'dist' folder 'npm run build'

2022-03-29T13:18:00.001+01:00

'npm run build' 'dist' folder not working.

I'd been hitting a bit of a brick wall with this task, so thought it'd be a good idea to commit it to the blog. Incase someone else finds it an can save themselves a heap of time. The task I was trying to achieve was

'How to host a Vue.js project on Firebase' .

And I was following this blog and youtube video for guidance.

How to deploy vue.js applications with firebase hosting

And Net Ninja's

Firebase Hosting Tutorial #4 - Deploying a Vue Site

But on a site ( that was already built and working locally ) I could only see the Firebase hosting holding message .

'Firebase Hosting Setup Complete'

I could however could create a new Vue.js website ( boiler plate ) and work from that, but building up from scratch again would be excruciating.

Let's take a look at process I was following and where it was getting stuck.

$. firebase init

Scroll down to the first 'Hosting config' choice and click on 'spacebar' and then 'return'.

There is already a public folder in use, so we'll want to set the public directory as 'dist'

As we're using Vue.js which is a javascript app that rewrites all the pages through the index page, then we need to select 'y' in response to , rewrite all urls to /index.html . This creates a dummy index page.

We don’t want to deploy this , we now want to build our vue application so that all the files go inside this ‘dist’ folder and then we want to deploy it.


1   npm run build

And we should now be able to view our site with.

1firebase serve

And this gives us the same ‘Firebase Hosting’ message we’re seeing before  

On my basic vue.js install though, I could see that it was working fine. One thing that caught my eye was that the 'dist' folder was still empty. Hold on ! No, the issue was another 'dist' folder had been created and I had one in the 'root' of the project and one in the 'app' folder.

I removed both the ‘dist’ folders and started again.
Ran ‘fiebase init’ and chose the hosting as before.
The ‘dist’ folder was created in Root . But I manually moved it to the ‘app’ folder
I then ran ‘npm run build’ . this created the new files.
In ‘firebase.json’ I had to add the following code

1 "hosting": { 2 "public": "app/dist",
Ran ‘firebase serve’.

So I have the site showing using ‘firebase serve’ .

firebase deploy --only hosting

And I can now see it on the app online :) I hope this saves someone a heap of time, as it took me a while to work this one out :/

CodeRed CMS - How to Order ( weight ) the Navigation snippets

2022-03-29T12:55:00.000+01:00

The task was simple, I'd added a load of menu items to the top menu using snippets but I wanted to change the order

What no drag and drop ! And neither could I see an input box to enter a position. Surely I wouldn't have to get my hands dirty and write some code for this.

Here's an easy fix though.

As the navigation uses flexbox . All you need to do is call the Bootstrap Order class.

And then in your 'navigation' item 'snippet' just add the ordering like this

Gsutil - How to empty a GCS bucket

2022-03-16T11:09:00.004+00:00

Although I couldn't find a 'gsutil' command to empty a bucket as such the following will remove and then we add the bucket.

I've also add the code here for how to do this from an Airflow DAG .

% gsutil -m rm -r gs://djem_photos

% gsutil -m mkdir gs://djem_photos

from airflow.operators import bash_operator

bucket = 'gs://temp_gp_tasks'

with models.DAG(
        'remove_files_from_gcs',
        schedule_interval=None,
        default_args=default_dag_args,
        catchup=True) as dag:

    remove_gcs_bucket = bash_operator.BashOperator(
        task_id='remove_gcs_bucket',
        bash_command='gsutil -m rm -r {bucket}'.format(
            bucket=bucket))

    recreate_gcs_bucket = bash_operator.BashOperator(
        task_id='recreate_gcs_bucket',
        bash_command='gsutil -m mkdir {bucket}'.format(
            bucket=bucket))        

    remove_gcs_bucket >> recreate_gcs_bucket

Airflow Connection Code Example - Access connection data from code

2022-03-07T16:31:00.004+00:00

Here's a straight forward piece of code, but I found this solution difficult to find and uncover . So here's the solution to getting the data from the Airflow connection in to my DAG - in Google Cloud Platform

from airflow.hooks.base_hook import BaseHook

conn = BaseHook.get_connection('my_conn')

def connect_to_somewhere():    
    logging.info(conn.login)
    logging.info(conn.password)     

Hope that helps someone out there !

GCP DAGS " Given * file, /home/airflow/dags/* , could not be opened.

2022-03-03T10:52:00.005+00:00

Although this post is about solving my specific issue with locating the 'googleads.yaml' and it's 'private_key' file , it's will also be useful to others who want to Access any other file that they have uploaded.

I had the following code that I could not get to work.

FOLDER_PATH = '~/dags/'

def gam_traffic():
    client = ad_manager.AdManagerClient.LoadFromStorage(FOLDER_PATH + 'googleads.yaml')

I'd tried other FOLDER names , like the gs:// bucket address and the full address like

europe-west1-composer-XXX-XXXX-bucket. All to no avail.

RED HERRING ALERT

Using the gs:// bucket address worked fine for Saving and Retrieving .csv files using Python Panda, but I couldn’t get the address to work when using the Python Open command. My assumption now is that the Python Panda library must contain some ‘magic’ that process this address when it see that the URI starts with ‘gs://’
To solve my issue and find the path I needed then I ran the following tasks to prove what folder we are in , and all the files and folders in it.
`import os 2 3# Print working dir 4def print_working_dir(): 5 directory = os.getcwd() 6 # Iterating through the json 7 logging.info(directory) 8 return "success"`

`1import glob 2 3# Print working dir 4def list_working_dir(): 5 for filename in glob.iglob("./**/*", recursive=True): 6 logging.info(filename) 7`

The second task takes ages to run, as its iterating through loads of folders. In our case we intercepted the logs while it was running , as we could already see the information that we needed.

In our case the filepath we needed was

`1./gcsfuse/dags/googleads.yaml'`

Vue.js where can I get my firebaseConfig details from .

2022-02-15T16:50:00.003+00:00

I have the following details that need filling in on my main.js page.

const firebaseConfig = {
  apiKey: "AAAAAAAAAAAAAAAAAAAAAAA",
  authDomain: "some-domain.firebaseapp.com",
  projectId: "some-project",
  storageBucket: "some-project.appspot.com",
  messagingSenderId: "11111111111111",
  appId: "1:111111111111111:web:11111111111111",
  measurementId: "G-88888888"
};

Where do we get the firebaseConfig information from ?

First off let's get the API key. Go to your Firebase Project page https://console.firebase.google.com/projec

Next to Project overview > click on Project settings.

The next page will have your 'Web API Key' . However if it says 'Not configured' then all you need to do is go into your Authentication page first and then come back to this page.

You'll see it's filled in now :)

However to see all your details on this page. just type in 'firebaseConfig' and all the details are there in code lower on the page.

Defining 'S3ToGCSOperator' in GCP Airflow DAG.

2022-02-11T11:19:00.005+00:00

I have the following operator that I want to use in this DAG. 3ToGCSOperator

    transfer_data_s3_to_gcs = S3ToGCSOperator(
        task_id='data_transfer_gcs_to_gcs',

However I need to define the import.

Which is

from airflow.providers.google.cloud.operators.s3_to_gcs import S3ToGCSOperator

What you need to do is your Environment in GCP and navigate to the ‘PYPI PACKAGES’ TAB.
And add the package https://airflow.apache.org/docs/apache-airflow-providers-google/stable/index.html

apache-airflow-providers-google

However I then got this error.

UPDATE operation on this environment failed 5 minutes ago with the following error message:

Failed to install PyPI packages. apache-airflow-providers-google 6.3.0 has requirement google-ads<14.0.1,>=12.0.0, but you have google-ads 7.0.0.

Check the Cloud Build log at https://console.cloud.google.com/cloud-build/builds/%%?project=4%% for details. For detailed instructions see https://cloud.google.com/composer/docs/troubleshooting-package-installatio

So what you need to do first in ‘PYPI PACKAGES’ is install oogle-ads. Version = >=12.0.0,<14.0.1.

ModuleNotFoundError: No module named 'airflow.providers.google.cloud.operators.s3_to_gcs'

And then you'll be able to do it.

Create a written date in Javascript 'moment' library.

2022-02-01T10:41:00.003+00:00

The dates I'm outputting at the moment look like this.

Which looks like this with 'moment'

const written_date = moment(my_date).format('MMM d YYYY');

However, what I want is more of a fullText response.

Moment has these name types.

  type RelativeTimeKey = 's' | 'ss' | 'm' | 'mm' | 'h' | 'hh' | 'd' | 'dd' | 'w' | 'ww' | 'M' | 'MM' | 'y' | 'yy';
  type CalendarKey = 'sameDay' | 'nextDay' | 'lastDay' | 'nextWeek' | 'lastWeek' | 'sameElse' | string;
  type LongDateFormatKey = 'LTS' | 'LT' | 'L' | 'LL' | 'LLL' | 'LLLL' | 'lts' | 'lt' | 'l' | 'll' | 'lll' | 'llll';

For what I wanted to achieve this works for me.

const written_date = moment(my_date).format('ddd LL');

Vue.js - How to use Set VUEX values in main.js

2022-01-26T10:15:00.001+00:00

The answer to 'How to use Set VUEX values in main.js' is that you don't !

Another niche note from myself for a Google search term that I made that this blog would have save me a bit of time chasing around the houses.

I highly recommend the YouTube video for solving this and other Vuex issues - https://www.youtube.com/watch?v=oxUyIzDbZts&t=629s

The issue I had to solve was that I'd written some code where I was using 'localStorage' when the value should be stored in VUEX . This was being done in the main.js page.

firebase.auth().onAuthStateChanged(async function (user) {

  console.log('Auth State Checked')

  if (user) {

    user.getIdTokenResult().then(idTokenResult => {

      if (idTokenResult.claims.admin === true) {

        localStorage.adminClaim = true

}

});

}

})

The main thing here is, we can't use Vuex from the main.js page and you can't 'commit' to Vuex from here. That will done in a VUEX 'action'.

In store/index.js

export default new Vuex.Store({
  state: {
    adminRole: false,
  },
  mutations: {
    //syncrous
    setAdminRole(state, payload) {
      state.adminRole = payload;
    }
  },
  actions: {
    //asyncronous
    async setAdminRole(state) {
      firebase.auth().onAuthStateChanged(async function (user) {
      console.log('Auth State Checked')
      if (user) {
        user.getIdTokenResult().then(idTokenResult => {
          console.log('IdTokenResult for user', idTokenResult);
          console.log('idTokenResult.claims.admin',idTokenResult.claims.admin);
          if (idTokenResult.claims.admin === true) {
            state.commit("setAdminRole", true);
          }
        });
      }
    })
      
    }
  },
  modules: {},
  getters: {
    getAdminRole: state => state.adminRole,
  },
});

And then in App.vue we can Dispatch our 'action' like this.

export default {
    mounted(){
        this.$store.dispatch("setAdminRole");
    } 
};

Next up is to call it in the page you want to access that data. In our case the TopHeader page.

Note in the video above at 8.40, he talks about timing issues. Which is why mounted doesn’t work and we need to use computed.

export default {
  data() {
    return {
  },
  name: "top-header",
  mounted() {
  },
  computed: {
    adminRole(){
      return this.$store.getters.getAdminRole;
    },
  }
};

This will load a data variable 'adminRole' with our Boolean task. So using

{{ adminRole }}

in the template, would output either True or False.

Firebase Functions Claims setting one makes my other 'claims' undefined !

2022-01-25T11:02:00.001+00:00

In my app we have 2 roles that we need to add to the ‘claims’ .

Configuring custom claims on users  |  Identity Platform Documentation  |  Google Cloud 

These are

Admin
Coordinator

Which in my Cloud Function I set the claim to either 'true' or 'false' . Which is 'Add' and 'Remove' respectively.

However I was noticing that when setting one it was making the other claim 'undefined' . Here's what one of my functions looked like. exports.addAdminRole = functions.https.onCall((data, context) => { 3 return admin.auth().getUserByEmail(data.email).then(user => { 4 return admin.auth().setCustomUserClaims(user.uid, { 5 admin: true, 7 }); 8 }).then(() => { 10 return { 11 message: `Success! ${data.email} has been made an admin` 12 } 13 }).catch(err => { 14 return err; 15 }) 16})

To fix this issue I had to pass though instructions on what the other Roles claim should be.

I'm using Vue.js by the way. So, the other role is set in the 'data'. Which I can pick up and pass it through like this

removeAdminRole({email: this.email, coordinator_role: this.coordinator_role}).then(() => {
          this.successMessage = "The admin role has been removed !";
        })

And then Cloud function will look like this.

exports.addAdminRole = functions.https.onCall((data, context) => {
    return admin.auth().getUserByEmail(data.email).then(user => {
        return admin.auth().setCustomUserClaims(user.uid,  {
            admin: true,
            coordinator: data.coordinator_role
        });
    }).then(() => {
        return {
            message: `Success! ${data.email} has been made an admin`
        }
    }).catch(err => {
        return err;
    })
})

This now works as it should for me.

Firebase Functions - Callable Function returning 'CORS' error.

2022-01-18T13:44:00.007+00:00

firebase function has been blocked by CORS policy

I experienced this issue while following a tutorial by The Net Ninja - Firebase Functions

function/index.js

1exports.sayHello = functions.https.onCall((data, context) => {
2    return `Yo dudes and dudettes`;
3})

DEPLOY this to Google .

And then in our button action ( fired by javascript )

1const sayHello = firebase.functions().httpsCallable('sayHello');
2                sayHello().then(result => {
3                    console.log(result.data);
4                });

I’d then get a CORS issue firebase function has been blocked by CORS policy

The following call to assign the region worked for me in the Google Function itself.

exports.sayHello = functions.region('us-central1').https.onCall((data, context) => {
    return `Yo dudes and dudettes`;
})

Google Data Studio ( GDS ) - Removing 'No data' from Scorecard Display

2022-01-06T11:54:00.003+00:00

On my Scorecards where I have single values being pulled in, it is showing the text 'No data' underneath the value. Like this.

To remove that it's pretty simple. Click on 'Style' on the 'Scorecard' editor.

You then have a dropdown for 'Missing Data'

Here select 'Show * ( blank ) '

That's it :)

Python Panda Find and Replace a value

2021-11-10T09:58:00.001+00:00

For the issue I was trying to solve, I knew which Column this value would appear in. It is also the entire contents of the cell that I need to replace.

After searching for 'python panda find and replace a value' I found this post https://www.geeksforgeeks.org/python-pandas-dataframe-replace/

This suggests

df.replace(to_replace ="Text I don't want",

                 value ="Text I do want")

However this didn't work for me. What did work was to name the column specifically and to use 'str.replace', as mentioned here

https://stackoverflow.com/questions/53141072/replacing-column-data-with-pandas-replace-not-working

uk['my_col'] = uk['my_col'].str.replace("Text I don't want", "Text I do want")

Importing .Csv data into a table in Sequel Pro.

2021-11-03T10:31:00.001+00:00

First you need to CREATE your table.

Now click on the Table name on the left hand site of the Sequel Pro console.

Next, select File > Import from the top Menu bar.

From here on you'll have a wizard that'll talk you through the column matching.

And that's it.

DigitalOcean - FATAL: remaining connection slots are reserved for non-replication superuser connections.

2021-10-21T13:47:00.000+01:00

We were getting this error on an App running on Digital Ocean.

This is what you'll find if you Google the issue

https://www.digitalocean.com/community/questions/fatal-remaining-connection-slots-are-reserved-for-non-replication-superuser-connections

You’ll need to Edit the PostgreSQL configuration file and increase the value for maxconnections. The value for sharedbuffers may also need to be increased.

You may see here how more connections have been created:

This setting would be changed at →

Login to DigitalOcean – The developer cloud
Click on ‘Databases’
Click on your DB
Click on the ‘Connection Pools’ tab

This led us to page showing that we have 22 connections that we could use. So why are we using all our connection slots on digital ocean.

AND OUR SITUATION IT WAS THE CODE AT FAULT.

In the code there were 13 connections to the DB being opened but they were not being closed.
What I needed to do is add the db.dispose() after the connections been used.

db_engine = create_engine('[CONNECTION_DETAILS]', pool_pre_ping=True)
...
db_engine.dispose()

Macbook Pro M1 Chip and Docker issues and a Unresolved host.docker.internal issue.

2021-10-15T10:15:00.000+01:00

Much of my frustration and wasted time at the moment can be attributed to setting up Docker. It’s probably worth noting that probably not all of it can be attributed to the Docker and M1 chip. But some of it certainly is.

Like with many of my issues I tend to blame myself and my own knowledge, but sometimes it’s not all my fault !! and if you have a M1 chip Mac it’d probably be worth looking into the potential pit falls first be commencing or purchasing one. As well as docker there has also been problems with Homebrew and Visual Studio Code. Of which VSC seems to be resolved now.

Apple throws a wrench into the wheel of the "run anywhere" aspect of containers 
https://betterprogramming.pub/5-things-i-have-learned-when-using-the-m1-chip-macbook-air-a77f93c50381 

The first thing to note is that Docker have released an Apple Silicon Chip version, downloading this is the first thing that will save some time. Docker Desktop for Apple silicon And make sure you have the latest version.

Even with that there are also some on going ‘Known Issues’

Not all images are available for ARM64 architecture. You can add --platform linux/amd64 to run an Intel image under emulation. In particular, the mysql image is not available for ARM64. You can work around this issue by using a mariadb image.
However, attempts to run Intel-based containers on Apple Silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.
In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple Silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. We expect this issue to become less common over time, as more and more images are rebuilt supporting multiple architectures.

Pulling db (mysql:5.7)...
5.7: Pulling from library/mysql
ERROR: no matching manifest for linux/arm64/v8 in the manifest list entries

Meaning that in my Docker-compose file I had to add the following for the DB

1  db:
2    platform: linux/x86_64
3    image: mysql:5.7

At which point I can get to the database through Sequel Ace, however there’s no database ( this used to be maintained but now I seem to have to import when ever the site is fully torn down )

At this point I have my first site working at http://site1:8000/ and we can also use the Postman collection that comes with it. I say first site because for this project I need to get 2 sites working ( 2 independent github projects with their own docker-compose.yml files ). That second site will be using the endpoints that I just mentioned in the Postman collection.

Again I received an error on the ‘mysql’ image.

failed to solve with frontend dockerfile.v0: failed to create LLB definition: no match for platform in manifest sha256:5064###: not found
ERROR: Service 'mysql' failed to build : Build failed
davidmillward@Davids-MacBook-Air siaf %

So again we need to add this code to the DB image. In fact this time it’s ‘mysql’

1  mysql:
2    platform: linux/x86_64

And once again we’ll need to connect to the Mysql, and again it’s empty and we need to import the Database.

takes a bit of time this one  . On this occasion so long it took me into the evening.  Meaning that for the next step I would be restarting the Docker Daemon. 

TIME SO FAR TAKEN 2h 30m 

On the restart first thing the next day I’ve click play on all the images. However the site on http://site1:8000/ is no longer working, so I’ll need to investigate that.

The database is working though, so let’s shut down the other containers and see if that works.

That didn’t work, so lets ‘docker-compose down’ on all the containers and just concentrate on getting Site1 running again.

So I ‘docker-compose down’ , purged it in the debugging section of Docker UI and run the ‘docker-compose up -d’ . The database need doing again ( not too long this one. ) And now the first site works again , yay!

Only trouble is I’ll need to the DB on the second site again ( because of the purge ), but at least it’s first thing in the morning, so I won’t have to restart my laptop.
… see you in a bit ….!

TIME SO FAR TAKEN 4h 30m

So now we get to my issue which is that I can’t get the 2 sites to talk to each other internally.

This is the URL I used previously which worked fine.
http://host.docker.internal:8000/

If you want to find out why this method is needed and used then this blog explains this How to connect to the Docker host from inside a Docker container? 

What do we know about this issue

It is mentioned on this page Docker Desktop for Apple silicon

Fixes since the Apple Silicon preview 7The host.docker.internal and vm.docker.internal DNS entries now resolve.

The issue is mentioned here with no resolution Tunnel access is not working in docker container M1LIFECYCLE/STALE

Not supported according to this post https://betterprogramming.pub/macbook-m1-breaks-docker-development-14566ab6fa2e

Other things that you may ask me.

What comes back in the response ? It comes back empty but with no error.

Can you ping the FafeApi Container from the Drupal one ? Yes

Here’s the process to check that.

1exec into the drupal container and do a ping host.docker.internal

Although I needed to

1apt-get update
2apt-get install iputils-ping

This shows that I can ping the site ok. So this issue remains unresolved. If anyone knows how to fix it then I'd really like to know. Much appreciated. :)

Mailgun: sending variables to template example - Fix ''Create and start coding'' greyed out issue.

2021-09-23T14:58:00.003+01:00

First of all in Mailgun we'll add a template. ( Keep reading for the ''Create and start coding'' greyed out issue fix tip )

So log into your account and on the dashboard you'll see all of your domains. Click on the name of the domain that you want to add a template for. Although there is a domain dropdown on the Template page itself, so no big deal if you want to change that later on.

Next up, you'll need to click on 'Sending' in the left hand menu to see more options. The 5th option down in this dropdown should be 'Templates' .

Click on this. And then in the next page you'll get to 'Create Message template'

You'll need to make sure the characters in the 'name' are valid and you have some sort of description. Otherwise the 'Create and start coding' button is greyed out and you can't click on it.

I’m sure this will trip someone else up , but the name input cannot be capitalised. If it is the Mailgun 'Create Message template' will remain greyed out.

In the HTML of template you’re setting up , you can add variable using the following.

1{{ firstname }} {{ lastname }}

How to send from Flask .

So here the function for sending a standard MIME email. With some HTML

1    def send_html_message(self):
2        return requests.post(
3            self.send_message_api,
4            auth=("api", self.mailgun_api_key),
5            data={
6                "from": self.from_string,
7                "to": [self.to_address_2],
8                "cc": [self.to_address_1],
9                "subject": "HTML test",
10                "html": "<strong>Hello World</strong><br><p>Hi</p><p>This is an HTML test send </p>",
11            },
12        )

And this is one where we use our ‘newsletter’ template and send through some variables.

1    def send_simple_message_template(self):
2        return requests.post(
3            self.send_message_api,
4            auth=("api", self.mailgun_api_key),
5            data={"from": self.from_string,
6                "to": "David Millward <dj@flowmo.co>",
7                "subject": self.subject,
8                "template": "newsletter",
9                "v:firstname":"David",
10                "v:lastname":"John",
11                })

This one will send our newsletter, with the clients name on it.

Docker - failed to solve with frontend dockerfile.v0: failed to create LLB definition: no match for platform in manifest sha256:X: not found 7ERROR: Service 'myservice' failed to build : Build failed

2021-09-21T13:24:00.005+01:00

docker-compose up -d 
...
------
 > [internal] load metadata for docker.io/library/python:3-onbuild:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: no match for platform in manifest sha256:X: not found
ERROR: Service 'myservice' failed to build : Build failed

I found in this post that if I ran the following lines of code in terminal before my compose command, then that would eliminate the first issue I have.

1export DOCKER_BUILDKIT=0
2export COMPOSE_DOCKER_CLI_BUILD=0
3docker-compose up -d 
4
5ERROR: Service 'myservice' failed to build : no matching manifest for linux/arm64/v8 in the manifest list entries

So that second line of issue looks like it may have to do with my Mac M1 chip

1ERROR: Service 'myservice' failed to build : no matching manifest for linux/arm64/v8 in the manifest list entries

Adding the 'platform: linux/x86_64' to the following services seems to have worked.

1version: '3'
2
3services:
4  myservice:
5    platform: linux/x86_64
6    container_name: myservice_container
7    build: ./myservice
8    volumes:
9      - ./myservice:/usr/src/app
10    ports:
11      - 8080:8080

Python Panda's - Cannot convert non-finite values (NA or inf) to integer

2021-09-15T16:39:00.001+01:00

In the .csv that we're processing we have a column thats 'total_in_thousands' .

In that column the values are all in decimals and I presume the legacy code would like them to be integers.

csv = csv.astype({'total_uv_in_thousands' : 'int64'})

However this giving me this error.

1 astype_nansafe
2    raise ValueError("Cannot convert non-finite values (NA or inf) to integer")
3ValueError: Cannot convert non-finite values (NA or inf) to integer

To resolve this I’ve changed on NaN’s in this column to zero and it now works.
csv = csv.fillna({'total_uv_in_thousands' : 0}).astype({'total_uv_in_thousands' : 'int64'})

This works.