Managing IT

Setting up Azure DevOps Easy or Right

2020-11-21T22:06:00.004+10:00

I'm still somewhat new to the Azure world and one thing that something you'll only learn from failing is that there's setting up Azure DevOps easy or right

Typically like so many Microsoft products Microsoft likes to provide easy solutions, well you click, something searches and then you select an option and like magic it connects. This is how a service principal (automatic) connection works.

Don't ever Fucking do it

This is guaranteed to cause you more tears and angst with ARM and possibly other pipelines for deployment than any single shortcut on the planet. The problem is it doesn't alert you to various missing pieces. For starters you'll have contributor level access to a subscription, but there are some tasks he running infrastructure building you'll need to have Owner privileges. It won't grant you directory read to the AAD. You'll have no idea why somethings fail, they do and that's all the knowledge you'll have.

I'm sure there are a few extra things which no doubt you'll blindly stagger around like a 21-year-old on the night club dance floor at 2 am wondering why it's not working the 21 year old is thinking the same thing when he's trying to meet that girl. You'll feel about as awesome as well while you wondered why stuff doesn't work.

Enjoy your days building Azure infrastructure with ARM. Infrastructure as code is awesome but don't let this bite you on the arse

Connection Strings for VBA and MSOLEDB

2019-07-26T23:01:00.000+10:00

They recently gave me a gig, which amongst setting up some databases included fixing up some spreadsheets to talk to the new database. This included fixing connection strings in VBA.

Background to the story the client wanted to move some systems from using Oracle DB as a back end to SQL Server as the data store.

Now two years ago I had never written from scratch a VBA script for Office Automation. It has now become a bit a skill set I have developed. I am not a fan of some of what I see. Not the code it’s good. More people have built a complete application using VBA and Excel. This is bad because it doesn’t deal well with governance. Potentially has security issues in that a username and password are available in the sheet,. Connection details including a host name are all potentially at risk.

On to the bit, you came to get. Many of you are aware Microsoft now endorsed OLEDB as the future. I am not going into the why, but it means you now have a new driver. We can find more info and the driver file from this page. Grab the driver and get it installed. Now the old OLE driver used the SQLOLEDB provider in connection strings long with the SQLNCLI driver are still deprecated. The new driver is using the MSOLEDBSQL Provider.

I needed to update stings, and I was needing to form the string based on limited information, connectionstrings.com wasn’t helping it’s not listed there as yet. Noone else was showing details of a connection string to an Azure DB which was what I wanted to do.

I have access to the new VBA book “Microsoft Excel 2019 VBA and Macros, First Edition”, it disappointed me that the book didn’t show usage of the new driver, it has a copyright date of 2019 and they had released the driver in 2018

I will save you the hassle of finding this out for yourself. Here is the big reveal:

Dim UserName As String
Dim Passwd As String
Dim PType As String
Dim SqlString As String
Dim ReturnArray

Dim sConnectionString As String
Dim con As ADODB.Connection
Dim rst As ADODB.Recordset
Set con = New ADODB.Connection
Set rst = New ADODB.Recordset

PType = "MSOLEDBSQL"
Hostname = "tcp:mytestdb.database.windows.net"
InitCatalog = "Adventureworks"
UserName = "dbtest@myytestdb"
Passwd = "somecrazychars"
Trusted = "yes"

sConnectionString = "Provider = " & PType & "; " _
& "Data Source = " & Hostname & "; " _
& "Initial Catalog = " & InitCatalog & "; " _
& "User ID = " & UserName & "; " _
& "Password = " & Passwd & "; " _
& "Trusted_Connection = " & Trusted

SqlString = "SELECT TOP (5) * FROM [SalesLT].[Customer] "
con.ConnectionString = sConnectionString
con.Open
rst.Open SqlString, con
ReturnArray = rst.GetRows

Also, don't forget about the issues with non-alphanumreic chars. Check out the Strong Passwords

Ok, Enjoy.

See ya round

Peter

A Backdoor for one is a Backdoor for all

2018-09-08T23:29:00.004+10:00

The Australian Government wants to put a backdoor into your apps. They are trying to put all sorts of spin on the idea to make you feel like they will have some sort of control and its only for them. This is to stop the bad guys, terrorists, someone they don't like, you know those people.

Let me get this straight, there are only a few ways to enable this.

Put a flaw in the encryption.

Add something like a keylogger to the device.

Allow access by putting a bug in the app

There are more but you will get the point soon enough to understand that this is not going to create good outcomes for the consumer.

Let's try and make some sense to you as a consumer of each of these approaches. Hopefully, you will have a better understanding of the problem. Then understand these governments actions have an impact on the broader market. Their proposals will kill the internet in their overzealous approach to "catching the bad guys"

Put a flaw in the encryption

Many apps like Telegram have made a strong point in their selling of why users would want to use their app. Encryption secures your messages. Encryption happens between you and the other you are communicating with som you can safely send information.

Most of the internet websites use SSL or TLS. These are standards for encryption so that when you type into your browser to enter your banking password it stays between you and the bank. If encryption is broken for anything it has to break encryption for everything. This includes access to your bank. Researchers would publish the flaw in the algorithm for encryption which would force the app's removal.

Apps will progressively dry up until we are no longer able to securely send anything. Oh hang on but I can connect to a website and https will mean my communication is protected. Yes, well maybe. You see it is now a slippery slope. When it finally means that I cannot log on to my bank or government site to conduct business the internet as we know it has ground to a halt. No Facebook, no Instagram, no Gmail, sorry it's all gone?

Installing a Keylogger

Wow, where do you start! This is what the bad guys try and do every day. Why because it captures all your keystrokes. Literally, everything you type including everything I typed into this blog. My username and password for my banking. Logging into Amazon to buy a book or other things. Every keystroke is sent off to the endpoint where the keylogger is sending the data. Two problems, can someone intercept it? Can I access the place this is going? There will be more issues but let's start with these.

Hackers will do their darnedest to implement their own tools on your computer to capture the output of the keylogger. I guarantee that they will succeed.

The endpoint. That is some great repository of all the data from everyone's devices, yes everyone. That is a copy of every keystroke.
First is storage, we create terabytes of data each day globally. That will require some significant tech to store all that data and significant cost to taxpayers. Second, the endpoint will have to have better than world-class developers and security teams to secure this data. Look above at what is being stored everything you need to cause identity crises for much of that country.

As a Hacker, these would be golden treasure troves. The value of the repository would be immense to the bad guys. The ability to lure people to do any number of things will have zero monetary bounds. You want $10million to do that, sure, hell I would have paid $50million.

A bug in the app

Those who have reason to will almost immediately find the bug. These might be enemies of your state (your country) and well just about anyone who might want to commit an offence against you personally. It can't be there just for the government spying agency as that isn't how these things work. All sorts of people will either be screaming for a fix for the bug or quietly extracting your data. How will you know if the app has a bug? Publishing details are the only way you will ever know.

The Australian government has placed penalties of 10 years custody in their proposed legislation. This is to stop good people telling you that your apps developer has developed a bug to allow the government to spy. Even reporting on the fact you found all sorts of data or other relevant things on the dark web would leave you open to prosecution.

See ya round

Peter

Anaconda, Jupyter labs, Java and Broken Stuff

2018-07-26T13:18:00.000+10:00

The premise

I decided to have a look over Jupyter Labs as part of some professional development. This was to start learning and getting some skills in Jupyter.

I decided to use the Safari education provided through ACM. A seemingly good starting point was Jupyter: Interactive Computing with Jupyter Notebook. This turns out to be a fast-paced set of videos using Jupyter Notebook which is slightly out date with Jupyter Labs. It's a broad brush and introduces a lot of concepts in quick succession.

What did I find

Jupyter Labs is exciting and looks like a great way forward in the Jupyter world. Following along the course I went through a few examples and installed the extra components up to Scala (which is failing on Ubuntu 18). The author mentions a few problems with the maturity of installing on Windows and I'd say there's work to do in the Linux world. Will follow up with a demo of installing all this.

Getting to complete a variety of visualisations in various notebooks including python, R and Julia is a good place to get started. The course due to its breadth is light on specific content in any area. It helps get across a broad understanding of the of Jupyter.

There's also a brief introduction to Jupyter Hubs and the use of Docker with Jupyter.

The problems I ran into

As I mentioned above Scala won't currently install on Ubuntu 18 without a few tweaks. This is due to a bug with cacerts with the fix is well underway and should soon be out(24/07/2018)

JavaScript in labs is broken due to an issue with extensions. This same issue is causing other problems with the use of widgets.
There is also a lot of hit and miss with getting all the relevant packages installed to support all Jupyter Labs functionality

On this Learning Path on Safari

This is a fast-paced introduction, it's skimming just above the surface Jupyter. You will learn a broad swwep of Jupyter. Labs is different so you will have to work a bit out yourself. There's very little to teach you about how to formulate a notebook – rules guidance etc. While interesting the course does provide an overview. There is some opportunity to execute some code but, the fact you need to own the book to access any files makes it less than fantastic. This is not so much a hit at the content, but how Safari must be licensing this Packt content

Yum failing error 14 no more mirrors

2018-07-12T08:50:00.001+10:00

If you encounter [Errno 14] when trying to execute yum commands then you've an out of date cache.

I just booted up a Centos VM it hadn't been used for around 3 months and was trying to install Docker. The VM would throw this and not install items.

[root@dataserver1 ~]# yum install docker
Loaded plugins: fastestmirror, langpacks
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
http://mirror.aarnet.edu.au/pub/centos/7.3.1611/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article

https://access.redhat.com/articles/1320623

If above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/

http://mirror.internode.on.net/pub/centos/7.3.1611/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.

It's a simple fix simply execute

yum clean all
yum makecache fast

It will rebuild your repo cache and have proper details for latest repo mirrors

See ya round

Peter

SQL Server Buffer Cache Hit Ratio is dead

2018-07-07T17:43:00.002+10:00

I was recently working with a client on a problem they'd with an application. The client requested someone have a look over the database and see if I could find anything which the internal staff might have overlooked.

I started browsing around in the monitoring and noticed an anomaly. The database was showing an increase in IO Wait and nothing else. The monitoring tool hadn't alerted, though, in an advisor suggested to review IO Wait. Unfortunately, overlooking these measures was missing the problem. The general health of the database instance, CPU, memory, the number of connections etc were all rather benign. This along with a high BCHR and this lead people to come to the conclusion that the database server wasn't struggling and hence it was ok.

I've seen a quite a few articles in the SQL Server world dissing wait states and claiming BCHR as the tool of measures to DB health and yet here we're the exact thing Method R says pointed to an issue.

There's a great example over at SimpleTalk on how to show why BCHR isn't the greatest of measures. A database might be performing great work with only a BCHR of 65%. Individual circumstances will only tell you if that's right in a system. As the article at SimpleTalk discussed the idea that a system could be exhibiting memory pressure and yet still show a high BCHR.

When you search for Buffer Cache Hit Ratio tuning you'll find an article by Joe Chang and it, unfortunately, muddies the water on what reducing LIO was about. The thing is that Method R is about tuning out wait time and that it's highly applicable to modern SQL Server.

When you look at some of the graphs from the system you have Something that looked like this when the problem was in existence.

What you can see here is a graph of various waits and the underlying area of IO wait in the blue.
This graph may not look that spectacular, nothing stands out, and that's why it might be easy to overlook. The balance was that the IO Wait was 30% and higher as a percentage of wait time. Other data was showing wait times of seconds. Now looking at the other measures of CPU, 20% to 40% random spikes in usage, nothing there. BCHR always above 90% and mostly above 99%. I then thought I'd overlooked something.

We investigated several potential slow queries identified due to logs of software using this database. Work found the need to fix one index. A few others went on the backlog for future work, but still no performance. I'd to come back to why was it getting such high IO. We'd optimised queries, we didn't have other performance issues, but we'd higher than seemed reasonable IO Wait time.

Next step was to get diskspd and run it on the drive where the data file was for the database. The results are below This server had dual 8GB HBA channels to the SAN. If you look down in the second section the total MB/s throughput is around 48MB/s possible half of a single 8GB channel. Also look at the 99th percentile. For those not quite up on their statistics, here I go. 1 out of 100 queries diskspd made was taking 32ms to read and 1 out 100 16 ms to write. Unlucky you if you got the 100th of both at the same time

Command Line: diskspd -b8K -d30 -o8 -t8 -h -r -w35 -L -Z1G -c50G G:\IOTest\iotestR1T1.dat

Input parameters:

timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
measuring latency
random seed: 0
path: 'G:\IOTest\iotestR1T1.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
write buffer size: 1073741824
performing mix test (read/write ratio: 65/35)
block size: 8192
using random I/O (alignment: 8192)
number of outstanding I/O operations: 8
thread stride size: 0
threads per file: 8
using I/O Completion Ports
IO priority: normal

Results for timespan 1:
*******************************************************************************

actual test time: 30.00s
thread count: 8
proc count: 8

CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 19.66%| 13.99%| 5.67%| 80.34%
1| 40.67%| 11.08%| 29.59%| 59.33%
2| 19.71%| 11.86%| 7.85%| 80.29%
3| 21.16%| 15.91%| 5.25%| 78.84%
4| 25.33%| 17.42%| 7.90%| 74.67%
5| 20.59%| 13.36%| 7.23%| 79.41%
6| 22.36%| 14.04%| 8.32%| 77.64%
7| 25.79%| 13.47%| 12.32%| 74.21%
-------------------------------------------
avg.| 24.41%| 13.89%| 10.52%| 75.59%

Total IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 191004672 | 23316 | 6.07 | 777.18 | 10.291 | 6.838 | G:\IOTest\iotestR1T1.dat (50GiB)
1 | 187531264 | 22892 | 5.96 | 763.05 | 10.480 | 8.466 | G:\IOTest\iotestR1T1.dat (50GiB)
2 | 190963712 | 23311 | 6.07 | 777.01 | 10.293 | 6.858 | G:\IOTest\iotestR1T1.dat (50GiB)
3 | 188301312 | 22986 | 5.99 | 766.18 | 10.438 | 8.153 | G:\IOTest\iotestR1T1.dat (50GiB)
4 | 188792832 | 23046 | 6.00 | 768.18 | 10.410 | 7.029 | G:\IOTest\iotestR1T1.dat (50GiB)
5 | 192118784 | 23452 | 6.11 | 781.71 | 10.232 | 6.327 | G:\IOTest\iotestR1T1.dat (50GiB)
6 | 191086592 | 23326 | 6.07 | 777.51 | 10.284 | 6.488 | G:\IOTest\iotestR1T1.dat (50GiB)
7 | 191832064 | 23417 | 6.10 | 780.55 | 10.246 | 6.587 | G:\IOTest\iotestR1T1.dat (50GiB)
-----------------------------------------------------------------------------------------------------
total: 1521631232 | 185746 | 48.37 | 6191.39 | 10.333 | 7.126

Read IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 124010496 | 15138 | 3.94 | 504.59 | 11.605 | 7.852 | G:\IOTest\iotestR1T1.dat (50GiB)
1 | 121675776 | 14853 | 3.87 | 495.09 | 11.788 | 9.912 | G:\IOTest\iotestR1T1.dat (50GiB)
2 | 123576320 | 15085 | 3.93 | 502.82 | 11.627 | 7.880 | G:\IOTest\iotestR1T1.dat (50GiB)
3 | 122241024 | 14922 | 3.89 | 497.39 | 11.773 | 9.524 | G:\IOTest\iotestR1T1.dat (50GiB)
4 | 122519552 | 14956 | 3.89 | 498.52 | 11.778 | 8.101 | G:\IOTest\iotestR1T1.dat (50GiB)
5 | 124936192 | 15251 | 3.97 | 508.35 | 11.538 | 7.227 | G:\IOTest\iotestR1T1.dat (50GiB)
6 | 123412480 | 15065 | 3.92 | 502.15 | 11.619 | 7.431 | G:\IOTest\iotestR1T1.dat (50GiB)
7 | 123387904 | 15062 | 3.92 | 502.05 | 11.590 | 7.604 | G:\IOTest\iotestR1T1.dat (50GiB)
-----------------------------------------------------------------------------------------------------
total: 985759744 | 120332 | 31.34 | 4010.97 | 11.664 | 8.237

Write IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 66994176 | 8178 | 2.13 | 272.59 | 7.860 | 3.170 | G:\IOTest\iotestR1T1.dat (50GiB)
1 | 65855488 | 8039 | 2.09 | 267.96 | 8.062 | 3.687 | G:\IOTest\iotestR1T1.dat (50GiB)
2 | 67387392 | 8226 | 2.14 | 274.19 | 7.845 | 3.182 | G:\IOTest\iotestR1T1.dat (50GiB)
3 | 66060288 | 8064 | 2.10 | 268.79 | 7.968 | 3.497 | G:\IOTest\iotestR1T1.dat (50GiB)
4 | 66273280 | 8090 | 2.11 | 269.66 | 7.882 | 3.092 | G:\IOTest\iotestR1T1.dat (50GiB)
5 | 67182592 | 8201 | 2.14 | 273.36 | 7.802 | 2.876 | G:\IOTest\iotestR1T1.dat (50GiB)
6 | 67674112 | 8261 | 2.15 | 275.36 | 7.850 | 2.998 | G:\IOTest\iotestR1T1.dat (50GiB)
7 | 68444160 | 8355 | 2.18 | 278.49 | 7.823 | 2.872 | G:\IOTest\iotestR1T1.dat (50GiB)
-----------------------------------------------------------------------------------------------------
total: 535871488 | 65414 | 17.03 | 2180.42 | 7.886 | 3.182

total:
%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | 0.220 | 1.210 | 0.220
25th | 5.568 | 5.910 | 5.742
50th | 9.924 | 7.630 | 8.435
75th | 16.694 | 9.396 | 13.386
90th | 21.264 | 10.991 | 19.469
95th | 24.205 | 12.038 | 22.507
99th | 32.656 | 16.721 | 30.466
3-nines | 53.280 | 34.684 | 49.527
4-nines | 151.592 | 70.318 | 113.512
5-nines | 530.571 | 110.152 | 530.571
6-nines | 596.914 | 110.152 | 596.914
7-nines | 596.914 | 110.152 | 596.914
8-nines | 596.914 | 110.152 | 596.914
9-nines | 596.914 | 110.152 | 596.914
max | 596.914 | 110.152 | 596.914

This led to further investigation by other people to find another database which had recently had a change made which showed it running at up to 798Mb/s over its port. These two databases shared a common port on some storage middleware above the disk servers for the SAN.

A couple of takeaways

Assume nothing
Test your theories
Gather evidence
Ask questions
Challenge perceptions

End result the troubling database had its IO access moved to take it away from further interaction with the database which I'd been asked to investigate the performance slowness.

Another thing if you have slowness and you're trying to isolate the source it might take differently worded questions, "how is the database?" might not yield the answer you want. Perhaps try "What is happening in the database when an incident is happening and is that good or bad?"

Good luck with tuning

See ya round

Peter

What happened at Facebook with Cambridge Analytica

2018-04-08T20:16:00.001+10:00

I see a few people asking who don't really understand the technology of data and marketing and what happened with Cambridge Analytica.

Firstly there is the right or legit way Facebook uses and sells data for marketing purposes.

As a general rule Facebook provides aggregated data to advertisers, so if you lived in Oxfordshire in the UK, then Facebook aggregates all data of people who it identifies as living in Oxfordshire. It would then in accordance with general marketing data rules (this is a sample size, not the legal rule) they would remove some data as it easily identifies a person which risks further identifications of actual people and then bundle up some insights either as a data set or through their own software for marketers. Cambridge Analytica is different in two ways.

First, they directly engaged with US peoples accounts on facebook using an app. I won't call out any, however, you can be reasonably assured any app on Facebook isn't there for your enjoyment, it is to get at your data. Cambridge Analytica simply created an app, had people engage with the app. It might have been guessing my age or funny captions for a photo, but it reeled in a significant number of people. When you connected to that app it would have asked for certain permissions. It now officially had access to whatever data you agreed to share. Here is where the game changed. Cambridge Analytica had realised that due to a bug that they could access the data not only what you had provided for, but additional data and just about as much from your friends. Now as many of us who don't live in the US have friends in the US. Cambridge Analytica did not discriminate, they simply sucked down anything within their ability to access and took all that data into a pond of their own making.

A couple of things, unless you sent credit card details via a messenger post to someone there is little risk of credit card data being there. If your full home address is there, then you are at risk of having that in the pond of data Cambridge Analytica gathered.
Why did they go to that trouble and what was the outcome.
Without providing a lesson in statistics, there are plenty of them freely available on the web, the whole thing was to identify what your associations are and therefore profile you, particularly if you reside in the US, I am betting they did it for everyone with varying degrees of success if they have your data. Why??
If you liked a post about a lot of environmental issues it means you might not like people taking down environmental protection rules. How do I convince you that the person who is planning taking down the environmental laws is the good guy I make sure that I provide you information that shows why the current rules favour someone who is not you(you are missing out). This is the sneaky way in which they targeted people with ads or fake news items which were to alter peoples perception. If you can create cognitive dissonance with someone you have a reasonable chance of changing their mind.
This has been an ongoing trend in businesses that a customers data is more important than the customer as once they have it they have it potentially for life.

GDPR will put pressure on operators in Europe and the rest of the world need to follow the European standard. Countries which don't have data provisions similar to GDPR need to start moving or expect to be annihilated at the next elections for failing to protect their citizens from such unruly behaviour and such a disregard for users data.

To recap there was a breach in the way the facebook was being accessed through the provided interface. This allowed Cambridge Analytica to access far more data than they ever should have been allowed to.
This was a major failure in Facebook engineering and their privacy and security practices.
This data then appears to have been misused to target ads to people. It is quite possible someone you sit next to at work was getting a very different political message than you and your friends with a similar group being other cohorts from your workplace and possibly user group you share membership of.

Facebook is in hot water in a variety of countries right now including the hearing in Congress next week.

See ya round

Peer

Azure Internal Load Balancer Configuration - SQL Server Failover Cluster Instance on Azure Virtual Machines

2018-01-03T08:19:00.001+10:00

I have just finished working with a colleague, resolving some issues with creating an IAAS SQL Server cluster in Azure. It took some trial and error and there are some real gaps in information, hopefully, this will help to fill one of them.

Let me first start by saying due to a lack of free diagnostics within Azure, you will need access to insights. This isn't something you would do with an MSDN subscription and might baulk at with your own paid for one. Why not with MSDN, because with MSDN you can only use a minimal amount of Network Watcher resource and you require network watchers to do any diagnostics at the network level. I hope you have a friendly boss who will give you a space to do some learning and develop these needed Azure skills. alternatively, or you have a bit of budget to do your own subscription and pay for resources. Make sure you have budget alerts to not blow out your costs and delete the Network watchers as soon as you have a working system.

We had to investigate the linkage between the Azure load balancer and the SQL Server Cluster and Network addresses on the SQL Server cluster. To see what was happening we needed App Insights and Network watchers.

What was the scenario

We had a SQL Server cluster without yet enabling Always on Services. We found that whilst we could connect to the node from which SQL Server was running we couldn't get a connection from any other server or system in our subnet, the problem was hidden as you cant do much to find out why what you are seeing is the problem and where it is.
We installed Wireshark, yes it is your best friend here and yes it is telling the truth even though it seems stupid it isn't seeing what you expected. We couldn't see any traffic when we tried the connecting to the cluster. We got the outbound packet to initiate a connection and then "crickets", nothing responding from the cluster or the load balancer.

Let's go back a step, one of the items in your list of tasks when doing this is to create the SQL cluster and then an internal load balancer.

Let me tell you the instructions on how to configure the internal load balancer in all the Microsoft documents and any blog posts I came across were terrible in the detail.

The load balancer requires Health Probe and that requires an active port on your individual nodes to validate their availability.

People have listed ports around 50k things like 52486 or 62159. Here is the missing bit it has to be an active service running on your server and it can't be anything to do with SQL Server, they are bound to the cluster and you are not able to access them via the individual node IP address

How do you work out a port, two things if you have a reason or don't overly care, you can install IIS and you will have port 80, however, try netstat -an and have a look, you will get entries like this

netstat - listening ports

Proto Local Address Foreign Address State
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING
Use one of these ports for your backend service that is the ones near the start of the list and with the Local Address starting with 0.0.0.0, these are listening on all IP addresses on your system.

Once you configure the load balancer with this in the above section you will find your cluster SQL Server will become available to your other services in your Azure space. You will now be able to use SQL Server Management Studio to connect to the active node in the cluster on the cluster IP address

Brisbane Yow 2017 Review

2017-12-14T21:54:00.002+10:00

Last week I attended Yow for my first time. It provided some great talks in my broader interests of data and analytics. That is the space I currently consult in. Thanks to my employer Readify for the tickets and the opportunity to attend. We have twenty professional development days a year and that was two taken.

Day 1

Day one started with a great keynote from Dr Denis Bauer of the CSIRO talking about the challenges of working with big data in the form of the human genome. Denis was joined by Lynn Langit and they worked through explaining the project and who it was fitted together. If you didn't know the genome sequence in data form require a database table with 3 billion columns, yes that is 3 with a B. Currently there is no relational database wi th the capability to create a table that wide (Pity the poor designer or DBA required to model that one). Of course, this is a big data problem of an order of magnitude significantly large. Denis spoke about how she and the team had set up workloads in AWS to process genomic datasets to deliver real opportunities to identify relationships between peoples genomes to find markers for genetic conditions. Thereare many challenges and great opportunities. I was lucky enough to get some time to have a chat to Denis later in the day and whilst it is exciting there are some real issues to deal with such as misuse and abuse from a variety of parts of society. I enjoyed the talk and then the conversation later. Fascinating women with a fantastic mind.

I then attended a talk about problems with Agile delivered by Jeff Paton. As Readify where I currently work has a very agile approach to the way we work, I was interested to hear about the supposed problem and remediation. Jeff makes very effective use of a style akin to the old writing on slides using an overhead projector. It was engaging, I learned a few things about the place of the product owner and how we as participants in the Agile community by its use can help our product owners be better.

Next in my day was AWS Security by Aaron Bedra. Aaron made many good points about securing the cloud and its services and the fact which I wholeheartedly agree with is that cloud done right is more than likely much more secure than many data centres. I learned a few things and was reminded to check some work for a current client I am working with.
Getting up a system in the cloud can be very fast compared to a traditional data centre, however, with that comes a number of risks. Aaron spoke about the checklist and things you can do to make sure your approach to security is sound.

Next on my day was Jim Webber, as a DBA I am always interested in database technology. As neo4j jas a strong market presence and now SQL Server includes a graph database this was an opportunity to learn more. I had a few items of basic knowledge reinforced and then Jim went on to talk about consistency in large-scale databases and what they had changed to handle this. The use of Causal consistency and a causal clustering architecture. deliver better throughput, large-scale clustering and a method to maintain the integrity of data in the database. Totally enjoyed expanding my knowledge of graph databases.

The day was progressing and next up was Chanuki Illushka Seresinhe. Chanuki spoke about what about beauty makes us happy and was it possible to quantify with deep learning. This was interesting to learn more about some of the concepts of deep learning. Chanulki also spoke about the fact there are limited large datasets in some domains to do testing in other regions and also other domains. Even with the large dataset, she had access to from Scenic or Not there were large gaps in information which made the dataset less than ideal. This potentially causes all sorts of biases one of the very real problems with computer AI

Next, my afternoon continued with more computer learning and two great talks on Machine Learning
First up was Julie Pitt. Julie spoke about the issue with training AI, biases and problems with algorithms. The key piece of her talk was about framing the AI problem right and discussing why it is way past where we thought we would have robots in our homes and yet the problems which stop them happening are still present. Julie is reframing the problem to have self-learning robots who adapt to ever-changing environmental situations. Her work is looking at simple problems like making sure that the robot won't assume shortest path is correct ie jumping from the second floor to the patio is the quickest way. As a kid, I grew up reading sci-fi books and Asimov. The facts that some of these problems have been well understood since then means we have work to do. Julie went on the show how having the concept of a zone where the robot survives and part of its job is to learn and maintain it's survival was a really interesting concept to unpack. She spoke about biases and wrongful outcomes. I was lucky enough to speak to Julie after at the networking drinks about some of her presentation and she is a wonderfully engaging person to speak with about her discipline. Oh and apparently I might have to learn Scala

My other presentation I attended on the day on machine learning was Jennifer Marsmen. Jennifer took us through a journey of capturing data in a novel way with the EPOC EEG Headset and analysing the data from it to deduce if we could use brainwave patterns to identify lies. Jennifer was engaging and spoke with great humour to convey her message. One of the key problems which I frequently encounter across data work in all disciplines is data quality. The headset needed to be set up correctly on the volunteer to obtain consistent quality readings to be able to verify the data. Once again I was able to have some time speaking to Jennifer about her data research and the ML capabilities. She spoke about the use of Azure ML and give a few very quick insights into understanding the ML algorithms available and methods of training in Azure ML or any ML system.

We then wrapped up the talks of the day with Dave Farley talking about Software Engineering and if the term is right to describe what developers do. Dave spoke at length discussing terms of skills in other disciplines of engineering. Should software engineers experiment, Dave said yes and explained that it is a frequent part of civil engineering, for example using models to wind tunnel test the design of a highrise is a form of experimentation and is done to minimise risk and to manage eventual building costs. Dave went on to talk about where software development is at in terms of levels of where other industries are at. He then talked about defining what engineering is and isn't and how work we do is in fact able to be a discipline of engineering. we just have to get some things right and we are not doing that now.

Networking drinks and hors-d'oeuvres ended the day, I caught up with a few speakers notable Jennifer and Julie as a data person and what they were doing was of great interest. I also spoke to Denis this evening. Spoilt to have some time talking with these women.

Day 2

The second day opened with a bang Linda Liukas opened to tell us about Hello Ruby and teaching young children about computers and computing concepts. The Hello Ruby Books are really an amazing creation and what Linda has done is fantastic. Concepts talked about include learning how a loop feels and Ruby's favourite loop I will let you buy the books to find out. If you have young kids around or if you just want to have a fun learning about computers in a non-threatening way these books are for you. Linda is fascinating to speak to one on one, we talked about adding the Hello Ruby books and activities to local daycare activities. I am certainly adding them to my library. Possibly my favourite speaker and talk of Yow

The second stop of the day the blue room and Sara Chipps, the question do you believe an 8yr old girl can programme in C++? Let's talk about Jewelbots. Sara has designed and developed an Arduino based bracelet for girls. They are a rather simple looking device but as an Arduino, device packs a punch, not so much in what they can do but in what they are delivering. Due to the simple design and compact space, the Jewelbot couldn't house a compiler for higher level languages. Instead, the owner when she want's to program, programs in C++ and then bootstraps the device with her new code. A young woman and yes 8 yrs old did some live coding to configure a device. She was a champion, dealt with technical issues with grace and charm. Her parents should be proud and her school as well.

Third stop and off to hear about Dynamic Reteaming from Heidi Helfand. This was a really interesting talk on handling the problems from building and reconfiguring team, no team stays the same. No matter how long it has been together the whole team will change at some time, someone leaves or is promoted. Hiedi provided a lot of great examples and her experiences of reteaming and some ideas how to make it work, even choose your own team. Some great insights into human dynamics and teams.

After lunch another keynote with Gregor Hohpe. He talked about Enterprise Architecture, discussed a number of problems and some solutions. As an EA he ripped it into those who sit in ivory towers and provide colourful diagrams which are often thought of as meaningless in the world of day to day operations and project teams. He then talked about various patterns in Architecture and I went straight out the next day to review a few things in light of his comments. I have been working in a Solution Architect role amongst other titles on my current project. I enjoyed what he was talking about as it fits with a lot of what I think about the EA role, probably comes from using PEAF as my preferred Architecture methodology/framework.

Next up I listened to Katrina Owen talk about her accidental open source project and all the problems when you become a maintainer. Katrina is the maintainer of exercism.io a coding education site she created out of a need to make it easier to test and challenge students she was teaching in a coding program. Much of what tore her up in trying to fix problems as a maintainer were people issues, dealing with competing priorities, maintaining balance and sorting things out to do just enough to avoid burnout which she didn't for a period. One of Katrina's lessons, "What are you not going to do today?" That is something we all need to learn. Other things include who or what are you doing your thing for, who matters because otherwise, everyone's opinion is right. Another great talk

Unfortunately, that is where my Yow day ended with speakers. I had to attend a conference call which went way too long, however, it served a purpose in my project and was needed to get some things rolling.
I did get to finish up the day with a beer and network with a whole lot of people. It was here where I was able to catch up with Linda amongst others of the speakers and a number of other attendees

Overall I had a great experience, caught up with a few old associates, made some new fledgeling connections and was able to get some time networking with great speakers. Jump over to the Yow site if any of the authors interest you the slides of the talks are up and videos to come. Yow has links back to websites, Linkedin and Twitter for the speakers

Let's see who is coming to Yow next year as to whether I decide to attend, I am sure there will be some great speakers, so its hurry up and wait until they are announced.

Serverless Local build in Docker - Part 1

2017-09-07T22:28:00.000+10:00

I wanted to set up a serverless-local build for using on a friends project. This will allow a certain amount of localised testing before pushing changes to the AWS Lambda service.
I decided to build a Centos VM and then added docker this allowed for a lot of portability.
Here are the steps I took

First, build the centos VM that is pretty straightforward for most and there are lots of good guides around. I use Virtual Box from Oracle as my VM service of choice.

Next installed Docker, just follow along here and you shouldn't have much trouble installing docker and getting a centos container running
Install Docker
yum install docker
Don't forget to test it with
docker run hello-world

I then used this container config to install a centos container with node already installed

https://github.com/lloydbenson/docker-centos-7-node This is a lot easier than building the entire node and npm install within a bare container. That can be another post when I get that one up

Once this is up then you can use the page here to install serverless-local into your project and allow you to do a lot of local testing https://blog.gorillastack.com/serverless-framework-local-development-environments/

SQL Server Linux - a quick look

2017-07-01T23:30:00.000+10:00

After a few changes, I now have a SQL Server Linux installation running in a totally Linux environment.

Whilst there is still a dearth of tools for Linux-based SQL server it is very doable and manageable.
I have two choices of tools which I found to mostly be useful for managing SQL Server with no Windows present. It is going to be some time before we see if ever Management Studio on Linux so you are going to have to keep reviewing alternative tools with which to manage Linux with

The first is Microsoft official
sqlcmd

There are good instructions for all main distros to help you install sqlcmd and add the directory of the command files to your path.

Firstly install .Net Core to your Linux installation https://www.microsoft.com/net/core#linuxubuntu for the Ubuntu installation.

Next, install Microsoft Tools mssql-tools https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup-tools#ubuntu.

once this is installed and right now if you are running 17.04 Ubuntu you will need to review the bug page for tools and install Preview versions as there are compatibility bugs which appear ironed out in those releases.

To access the data from some sort of GUI you might like Visual Studio Code as an editor which can interact with the SQL Server Database. You need to install the mssql extension.

The alternative I am using is DBDeaver http://dbeaver.jkiss.org/
It is based on Eclipse and seems rock solid as a database management tool. Now of course without all the proprietary bits of Management Studio your database T-SQL skills are going to have to raise a notch. You will need to grab the latest JDBC drivers from Microsoft https://docs.microsoft.com/en-us/sql/connect/jdbc/microsoft-jdbc-driver-for-sql-server

Now one final thing with VS Code is that it hasn't quite worked out Linux. Microsoft there is no C: lettering in file paths and the slashes in the path names need correcting, but I guess we can live with that for now.

DBBeaver file paths

VS Code file paths

Overall it is great to see SQL Server advancing on Linux and I can see a lot of places using SQL Server Development Edition on Linux to eliminate a bundle of cost from their no production environments. It is a great step forward and look to the future when it goes full GA

See ya round

Peter

Virtual box and Dell XPS 15

2017-06-22T22:01:00.000+10:00

Just acquired a Dell XPS 15 through a new job. I have been setting up a few Windows and Linux Servers as VM's using Virtualbox.

There is a real problem with resolution when initially installing just about any guest OS. It makes it firstly hard to install and secondly difficult to work with.
Enter the world of Scaling VM displays. This is a fantastic help however it comes with its own problems.

A problem with scaling is the menu for the running VM you have focused on goes missing by default, therefore, you can't do things like installing guest additions whilst in that mode.

VirtualBox on Ubuntu 16.04 DNS Failure From Guest OS

2017-01-20T15:38:00.000+10:00

I have a couple of VM's on my system running a variety of operating systems for testing and learning. Recently U upgraded the host system to Ubuntu 16.0.4 when I installed a new SSD. I moved the VMs from the old spinny disk and VM's start fine.
Yay! I can log into the VMs but realise there is a problem with DNS. I can ping outside hosts and all seems fine with the network. Started changing network settings in one of the guest OS and no, nothing is happening

After some investigation, this appears to be due to a change in how Ubuntu is doing DNS and that the Guest OS are unable to handle that natively.

The cure
Enable two parameters as the user you run your VM's under

VBoxManage modifyvm "Centos 7" --natdnsproxy1 on

VBoxManage modifyvm "Centos 7" --natdnshostresolver1 on

VBoxManage -natnetwork list

NAT Networks:

Name: NatNetwork

Network: 10.0.2.0/24

Gateway: 10.0.2.1

IPv6: No

Enabled: Yes

Make sure all VM are stopped and then run this, it willl crash VirtualBox but makes sure next time you start VirtualBox the new network settings will apply

VBoxManage natnetwork stop --netname "NatNetwork"

Python 2.7 Installation and SSL errors

2017-01-08T17:59:00.002+10:00

Working to get some python code finished and ran into an issue with seems to have plagued python for somne time.

When building python as you would often do to be able to use later versions in Redhat, as you cannot update the original Python as it breaks yum and many other Redhat tools you need to look at doing local installs. For the little bit of space and then using virtualenv, this is probably for many things a better option.

When installing Python 2.7.13 from source on Redhat it doesn't install the SSL modules. Ha, I don't need SSL, I am not connecting to any SSL. Sorry to tell you the bad new but pip now connects to https to access the repository, really it has for some time.

What does one do to resolve this?
What needs to be done is make sure the SSL development libraries are there as their absence leaves you without SSL which you need for pip. Of course, that means you cannot install packages directly fro the python repository. Agggghhh

Firstly you download and prepare to install python as local install, lots of good instructions out there.

Next, you make sure the zlib1g is installed as well as openssl-develop packages. Unless it's a very minimal installation, it should have the OpenSSL runtime stuff there already.

Untar the python and cd into the unpacked source code
Go to Module and open the Setup file and if it's not there run configure and make and it is generated.
Now as per this StackOverflow question. You need to uncomment some SSL text

SSL=/usr/local/ssl
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
-L$(SSL)/lib -lssl -lcrypto

Now make and make altinstall to where your local installation is

The path to verify SSL has changed in 2.7.13, possibly earlier editions as well

python /usr/local/lib/python2./test/test_ssl.py

This might become helpful to someone down the track as it seems a ridiculous issue to still be a problem for this long.

See ya round

Peter

Some thoughts on the MongoDB Ransom Attack

2017-01-05T23:00:00.000+10:00

As reported on a number of news sites there are a lot of unsecured Mongo DB servers out on the Web. This has created a new business opportunity whereby an ingenious hacker has found a business model. Backup your MongoDB, trash its contents and then leave a note for Ransomware to get the data back. I expect this is going to make a handsome earning.

Why you might ask? The databases have almost certainly in many cases been set up but unskilled people, I will take a stab at it and say probably not the most competent DBA or not even a DBA. My reasoning is that there are two things that stick out any decent DBA will have looked at two key tasks of the DBA, backups and security. Any DBA worth his daily pay would have both of these high in mind. He would have questioned how the data is accessed, the connections from any website and understood the architecture of the applications connecting to the databases being managed. The DBA would have constantly looked at how to lower the attack surface and had reliably tested backups reducing the concerns of any attempt to ransomware.

So if you don't have a DBA and you are running databases, I would suggest this is a warning to you to get either a service or hire one to make sure you don't have the opportunity to explain to your board to shareholders

See ya round

Peter

2017 is rolling in

2016-12-31T23:30:00.000+10:00

Its New Years Eve and I would like to thank those of you who have been part of my 2016 audience and will be bringing more in 2017.

Enjoy your New Years Eve, have a great celebration wherever you are in the world
Look forward to a great 2017 and that is prosperous and happy

See ya round

Peter

Bemoaning DB2

2016-10-13T01:30:00.000+10:00

About a year ago I was presented with an addition to my role. As part of the company acquiring Infosphere I acquired DB2 under its bonnet. This has proved somewhat of a learning curve and at this time DB2 still has a couple of mysteries up its sleeve. I still have answers to find about DB2's space management and table sizing and storage. I had previously had some experience with installing and some basic application integration steps but never learned much about the day to day management.

A definite win for the DB2 community is Data Server Manager. DSM is a great tool and is rapidly maturing. If you are new to DB2 or are using it definitely keep up with the upgrades right now as there is a lot happening. The current release has some great new features over its earlier counterparts and whilst there are a lot of graphical tools which could be added to the dashboard it's very handy as is.

Easy SQL tool, and the simple dashboard are great. Being able to check any part of the DB2 system in a couple of clicks are all great to have.

I am sure there are those who will still frown upon people using graphical tools but using this as a way to better understand rather than just a mindless point and click will learn a lot very quickly from being assisted by the tool.

So I am still no fan of DB2, it is the 5th product I have worked on seriously in the database realm so I have some considerable experience across them, DB2 will be easier to manage and easier to teach someone in due course with DSM. Back to war tomorrow and see what else I learn.

See ya round

Peter

QLD Government Innovation Portal - Cool idea

2015-06-12T19:41:00.000+10:00

QLD Government has recently opened up a site to garner outside thoughts on challenges which the government faces. These might be unusual problems about data, security, and just about anything which requires a solution that some off the shelf software won't cut out of the box.

There is presently a discussion going on with regards to preserving calendars which are currently in Outlook PST files. Now as to the problems they are trying to solve with that preservation, such as it needs to have the ability to be readable and useful 100 years hence and what formats are open to preserve such things and retain the integrity of the original calendar entry and be in a format which shall be readable by the fact the file specification will be openly preserved allowing anyone to go and read the file with readily available software knowing the file format.

There is learning going on with both sides of the conversation, with a response of the request specification needing to be addressed as the ideas whilst perhaps good have a second set of issues. The asked and are altering the question in response to responses as they had learned their initial request may have had some gaps in it.

Well done to the guys at QLD government who set up and run the site and those that are willing to put their ideas out there and canvas a wider pool of thought.

This is a great idea and good to see some people getting into the spirit to try and resolve the question/problem raised.

I hope a lot more departments are able to bring problems to the open to get solutions which may never have been thought of using only oe set of views.

The site is http://innovation-portal.dsitia.qld.gov.au/

See ya round

Peter

Modelio 3.3 on Xubuntu 3.13

2015-05-01T04:30:00.000+10:00

I recently started working with Modelio running on Xubuntu. f you are not familiar with Xubuntu it is a lightweight desktop Ubuntu build. Lighter UI than KDE or Gnome, you will likely end up with parts of KDE or Gnome in your system due to other applications you want to need either KDE or gnome libraries. It is still lightweight and the only reason I use Windows at home is for one purpose and that is accessing the Tax Office (ATO)
Now I am probably not as thin as originally installed as there are some things that you just try and work with.

Modelio has a few issues that you need to resolve to get it working.

Firstly download the software, mine was the open source version 3.3
Unpack it and move it to a directory.
Then you need to make sure your Java is correct. Modelio 3.3 requires java 8 you will need to make sure it is your default Java version, there are a few notes around on Ubuntu you require the alternatives command sudo update-alternatives --config java

I have then set the -vm parameter in modelio.ini to your Java 8 location

Without Java 8 you may find it loading with blank screen and no menus, or it wont start with a framework error.

I had a problem that the TOGAF module wouldn't enable in a project. It was a problem from a previous version of Modelio, actually a 32bit version. I deleted the project. So warning if changing Modelio export your projects before doing anything and import them to the new version. In this case, it was nothing important. You may not be so lucky

Setting up mariadb with non standard directories

2015-04-15T18:04:00.000+10:00

Today I was configuring a mariadb 10.0.16 server with non default directories. There is a lot of things to overcome in the process.

Firstly make sure your directories are owned by mysql where you want data logs and other elements created.

The following had to be set in server.cnf and set in separate sections in some cases

socket in client section
[client]
socket=/database/mysql/mysql.sock

The rest could be safely accommodated in the mysqld section due to running only a single flavour of mysql/mariadb on the server using this cnf file.

[mysqld]



#PID file

pid-file=/var/run/mysqld/mysqld.pid



datadir=/database/mysql/dbfile

socket=/database/mysql/mysql.sock

log_error=/database/mysql/serverlog/mysql_error.log

slow_query_log_file=/database/mysql/serverlog/mysql_slow.log

general_log_file=/database/mysql/serverlog/mysql_general.log

general_log_file=/database/mysql/serverlog/mysql_general.log



#INNODB logs on separate location

innodb_log_group_home_dir=/database/mysql/innodb_logs

We are running multiple log files to allow better log handling.
Hopefully these examples will be of some use to someone contemplating the change to not using the default of /var/lib/mysql and not using /var/log for the logging and not using /var/lib/mysql as a place of other logs such as slow query log

We hit another random error related to MySQL bug #45379. We had added user= and group= to the [mysqld] section and had the server randomly setting the group_concat_max_len to 4 breaking the use of group_concat. Moving those settings to mysqld_safe resolved that. Thanks Pythian for writing about this.

[mysqld]
[mysqld_safe]

user=mysql

group=mysql

Now looking forward to getting the new servers commissioned and into production.

If all the above is not done then the database will not create files in all the expected locations but use defaults and try using /var/lib/mysql, one day this will hopefully gone, instead replaced by an intelligent installer. which will ask questions and build a config file and create directories.

One final note make sure you run mysql_install_db as the mysql user or you will have issues with starting your database once again due to permissions of files created when running the install to create the database. If you have all your config correct with directories in teh config matching actual directories then when you run mysql_install_db you wont require any parameters with it. Simple.

See ya Round

Peter

Learning about Cassandra and Coursera

2015-04-12T13:13:00.000+10:00

Watched the presentation produced recently by Datastax and Coursera on Cassandra and how and some of why they chose Cassandra over Mongo and Hbase to replace some of their MySQL for the website.

Some key takeaways I saw was that Cassandra better fitted their data model over HBase or MongoDB. That Mysql Sharding was difficult for some of their needs.

Coursera has a very distributed customer base and MySQL wouldn't handle multi-site information capture and distribution.

Cassandra could do rolling upgrades without downtime,this is important for Coursera, if you have ever filled out a set of responses to an assignment and then not be able to save then because the application was offline due to a database maintenance window. This was a key tenet for what replaced MySQL.
Now if your geographic requirements, ie your customer base live local to Australia or Europe, then a single site of MySQL with Galera, or using AWS RDS, may well produce suitable results. You can also use a message bus within your application to provide some resilience to the database being offline.
Rolling updates are difficult for most RDBMS and Cassandra is strong in this area.

Like so much, it's not about being cool, it's about knowing the problem you are solving to choose the right product. There is no bad database only bad implementations using the wrong product in a given situation. MongoDB seems to cop a little flak, but this is probably more about a headlong rush to get away from a perceived problem with MySQL. Often early-stage web applications ar built with some part as Key pair store or as document store mentality. This is often simple to handle from the web development position, however, it ultimately won't scale in MySQL, or other RDBMS for that matter. So then people start looking for alternate storage ie other databases. This often creates a solution to that part of the database which is a document but causes problems with relational parts of the application. Ultimately this ends up with MongoDB being a bad choice for solving the problem.

Cassandra will be a fantastic solution as will MongoDB to the right problems. It won't solve poor design or a bad understanding of application requirements or business needs to access the data for reporting purposes. For this reason think carefully about any decision to jump off an original choice of an RDBMS such as MySQL and into the arms of a NoSQL database. It might leave a very sour taste in your mouth in the NoSQL database you choose.

Disclaimer: I have taken courses from Coursera

See ya round

Peter

Mariadb 10 Slave wont set up after upgrading from mariadb 5.5

2014-11-11T11:19:00.000+10:00

We are in teh process of upgrading mariadb 5.5 databases to maridb 10. The actual upgrade process of the database has been flawless for the stand alone databases. I have started on our first replication set a pre-production system and after upgrading a slave to 10.1.14 have hit a wall

The maria10 database slave not starting up as a slave after upgrading from 5.5 The master is still running as 5.5

This is the error log

141111 10:57:10 [Note] Event Scheduler: Purging the queue. 0 events
141111 10:57:10 [Note] InnoDB: FTS optimize thread exiting.
141111 10:57:10 [Note] InnoDB: Starting shutdown...
141111 10:57:12 [Note] InnoDB: Shutdown completed; log sequence number 36086013855
141111 10:57:12 [Note] /usr/sbin/mysqld: Shutdown complete

141111 10:57:12 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld-nxq.pid ended
141111 10:57:13 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/nxq
141111 10:57:13 [Note] InnoDB: Using mutexes to ref count buffer pool pages
141111 10:57:13 [Note] InnoDB: The InnoDB memory heap is disabled
141111 10:57:13 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
141111 10:57:13 [Note] InnoDB: Memory barrier is not used
141111 10:57:13 [Note] InnoDB: Compressed tables use zlib 1.2.3
141111 10:57:13 [Note] InnoDB: Using Linux native AIO
141111 10:57:13 [Note] InnoDB: Using CPU crc32 instructions
141111 10:57:13 [Note] InnoDB: Initializing buffer pool, size = 128.0M
141111 10:57:13 [Note] InnoDB: Completed initialization of buffer pool
141111 10:57:14 [Note] InnoDB: Highest supported file format is Barracuda.
141111 10:57:14 [Note] InnoDB: 128 rollback segment(s) are active.
141111 10:57:14 [Note] InnoDB: Waiting for purge to start
141111 10:57:14 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.20-68.0 started; log sequence number 36086013855
141111 10:57:14 [Note] Plugin 'FEEDBACK' is disabled.
141111 10:57:14 [ERROR] Failed to create a socket for IPv6 '::': errno: 97.
141111 10:57:14 [Note] Server socket created on IP: '0.0.0.0'.
141111 10:57:14 [Note] Event Scheduler: Loaded 0 events
141111 10:57:14 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=mysqld-nxq-relay-bin' to avoid this problem.
141111 10:57:14 [ERROR] Server id not set, will not start slave
141111 10:57:14 [ERROR] Failed to create slave threads
141111 10:57:14 [ERROR] Failed to allocate memory for the Master Info structure
141111 10:57:14 [Note] /usr/sbin/mysqld: ready for connections.

This is the server.cnf it isn't as clean and tidy as it should be but it is what it is

[server]

# this is only for the mysqld standalone daemon
[mysqld]
datadir=/var/lib/mysql/nxq
log_error=/var/log/mysqld-nxq.log
socket=/var/lib/mysql/nxq/mysql.sock
pid_file=/var/run/mysqld/mysqld-nxq.pid

user=mysql
group=mysql
port=3306
binlog_format = mixed

# MASTER config
# log-bin=mysql-bin.log

expire_logs_days=7
# Autostart the event scheduler
event_scheduler=on

# SLAVE config
relay_log=mysqld-nxq-relay-bin
log_bin=mysql-bin.log
#expire_logs_days=7
log_slave_updates=1
read_only=1
#skip_slave_start=1

# Performance Tuning

innodb_buffer_pool_size=8G
innodb_file_per_table = 1
innodb_log_file_size=256M
innodb_flush_method = O_DSYNC

max_allowed_packet=700M
#table_cache=2M
#table_definition_cache=2M
#read_buffer_size=1M
#myisam_sort_buffer_size=64M
#thread_cache_size=8
query_cache_size=64M
key_buffer_size=42M
#connect_timeout=2
max_connections=300
#open_files_limit=16384
performance_schema=1

# this is only for embedded server
[embedded]

# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]

# This group is only read by MariaDB-10.0 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.0]

server_id=201

I moved the server_ID value to the [mariadb-10.0] section from the [mysqld] section to try and find a solution

Update: Turns out the reason for this behaviour was due to a typo in the init.d startup script in this line defaultfile="/etc/my.cnf" It had to be changed from another location and it had an error. Funnily enough though the strace output was not helpful as it showed what appeared to be it reading the files in the /etc/my.cnf.d directory.

How did I find it? By using the startup process of executing mysqld --help. It displays the default location parameters and i could see the server-id set correctly

See ya round

Peter

OSDC 2014 Day One

2014-11-05T00:01:00.000+10:00

OSDC 2014 kicked off today and I was attending, my OSDC virginity has gone. Opens Source Developers Conference has convened on the Gold Coast

It was very interesting, I have seen some great talks getting an insight to a number of technologies
There are some people there who are madly hacking all sorts of things, Graphing technology, Arduino devices and the sustainable home, Gov hack and databases.
Its both old and young along side with only a passion about open source software
Talks about protecting yourself regarding the footprint you leave on the web from browsing and how that leaves you vulnerable to snooping about your surfung habits
Using encrypted.google.com to remove some of the inevitable jet trail you leave in metadata around the web
What about Postgres and JSON and the ability of Postgres to provide more ways to handle JSON objects? Including indexing elements of the JSON. More toi come in Postgres 9.4

Have you heard about Bluehackers, perhaps you should hopefully you never need it but one day you might just save a fellow worker with knowing about it.

Mari10DB is a leap forward, multi threaded Replication, Galera for clustering, new performance improvements with exists and in queries. GTID for replication consistency a long standing problem for MySQL and its derivatives such as mariadb. Improved performance on inserts.

Also a talk on stress and stress management with a tilt to Bluehackers

Some others are there for their first and some have been around since their inception

Tomorrow more stuff and conference dinner

It has been an interesting day with some people doing some interesting things

See ya round

Peter

Security Skills Lacking

2008-03-02T22:59:00.004+10:00

Apparently Security skills of the IT work force lacks generally. This is not surprising as for the most part many organisations where I work there is a lack of understanding of IT security. The government departments are very aware, generally have good policy and practise in place but I have seen many stupid breaches of good practise due to poor policy and procedures. This is where the problems often lie and its not the practitioners, who are struggling to do there job but the management who fails to understand many of the ramifications as they do not well understand the problems they are being faced with at the coal face.

This is a common problem that many parts of IT face. Management ever come up to you and say "can you sort this out for me it shouldn't take more than a few hours." Well this lack of understanding of IT from our managers is what is leaving the organisations exposed to breaches.

Clearly many managers do not understand IT law, I find this a terrible oversight that leaves many companies exposed to poor outcomes when there is a failing of there IT policy and procedures.
Recently I was doing some work on a clients site and they had me sign a piece of paper for internet access about acceptable use. It would now be considered that for that organisations they will be covered against a misdemeanour. My legal studies would lead me to the conclusion that they are likely on shaky ground, for a few reasons they have not clearly identified what are a couple of items open for interpretation. what is offensive and what is acceptable use. Now maybe another time they might just put me on a induction course that will clearly outline these, however until that is clearly stated then what may be considered offensive by the organisation may not be yours or mine interpretation.

It is these problems that management have about IT and security that are leaving the IT practitioners hanging out as without this being understood how do they understand the firefights the practitioners are wrangling inside and outside the organisations. If they can't understand this how do they get to a place where they have sufficient skills. Offering training will be good, but how does it benefit people. Does a new Checkpoint course really help solve the problem or should we look deeper at other types of training to take people out side the box to solve the problems. I certainly think a lot of management needs to go there.

See ya round

Peter

Security skills of IT workforce lacking, survey finds - Network World

Simpleology Blog Course

2007-12-21T11:21:00.000+10:00

I'm evaluating a multi-media course on blogging from the folks at Simpleology. For a while, they're letting you snag it for free if you post about it on your blog.

It covers:

The best blogging techniques.
How to get traffic to your blog.
How to turn your blog into money.

I'll let you know what I think once I've had a chance to check it out. Meanwhile, go grab yours while it's still free.