SharePoint.BG

Design smell: call overridable methods in constructors

2018-11-20T13:16:00+00:00

This is another good question I got while teaching the C# Advanced course at SoftUni. Here is a detailed answer.

Question

Why shouldn't we call virtual methods within the constructor of a base class?

Answer

This is actually one of the Code Quality warnings under the Usage section. Please see the link here: https://docs.microsoft.com/en-us/visualstudio/code-quality/ca2214-do-not-call-overridable-methods-in-constructors?view=vs-2017

While the article is correct, I'm not sure the explanation is very understandable.

It all comes down to the following execution path:

You instantiate the derived class
The base class constructor is called
It calls virtual methods overridden in the derived class
But our derived class's constructor has not been called yet

Here is my example:

    public abstract class BaseClass
    {
        public BaseClass()
        {
            Console.WriteLine("BaseClass constructor called.");

            VirtualMethod();
            AbstractMethod();
        }

        public virtual void VirtualMethod()
        {
            Console.WriteLine("Base virtual method called");
        }

        public abstract void AbstractMethod();
    }

    public class DerivedClass : BaseClass
    {
        public DerivedClass()
        {
            Console.WriteLine("Derived class constructor executed VERY late.");
        }

        public override void VirtualMethod()
        {
            base.VirtualMethod();

            Console.WriteLine("Derived Virtual Method");
        }

        public override void AbstractMethod()
        {
            Console.WriteLine("Derived Abstract Method");
        }
    }

The result:

BaseClass constructor called.

Base virtual method called

Derived Virtual Method

Derived Abstract Method

Derived class constructor executed VERY late.

So we just ran code in a class which might very well be in an invalid state. Objects should go through construction logic first before we execute their methods.

Module vs Class

2018-11-19T15:49:00+00:00

While I was teaching the SOLID module, as part of the C# Advanced course at SoftUni, I got asked about the difference between a "module" and a "class". The reason why this question is difficult to answer is because it means different things in different languages and frameworks.

In the purist C# sense, C# itself doesn't really use the concept of modules in its syntax. You can't declare, instantiate or import a module in C#. However, we use C# to write code that compiles to .NET assemblies - they contain metadata and bytes that represent Intermediate Language (IL) opcodes, which leads us to modules in .NET.

Modules in .NET

In the sense of .NET, assemblies contain modules, modules contain classes, classes contain functions (and other stuff). You can access assemblies, modules, classes, functions, properties, fields (and the other stuff) through reflection at runtime. This is all interesting, however the C# editing and authoring tools within Visual Studio do not allow you to create assemblies with multiple modules by default.

Although I haven't tried this, apparently you can create assemblies with multiple modules with notepad.exe and csc.exe, the compiler tools or by using the /addModules command-line option. You can read more about this here: http://etutorials.org/Programming/Programming+C.Sharp/Part+III+The+CLR+and+the+.NET+Framework/Chapter+17.+Assemblies+and+Versioning/17.6+Multi-Module+Assemblies/

When discussing assemblies and modules in the scope of CIL, a module is a single file. In .NET we can have assemblies as a single file, but we can also have assemblies with multiple files. These are called MultiModule assemblies. I found this resource to be pretty good: https://blogs.msdn.microsoft.com/junfeng/2004/07/15/multimodule-assemblies/

To be extra pedantic in the search of the truth, the most trustworthy answer should lie within the Common Intermediate Language (CIL) specification. The official standard is issued by Ecma International, an international organisation that standardises stuff like programming languages, has documented the official CIL standard ECMA-355. Please find the link here: https://www.ecma-international.org/publications/standards/Ecma-335.htm

According to ECMA-355, on page 140 and after, the relationship between assemblies, manifests and modules are defined. Here are some extracts:

Assemblies - "An assembly is a set of one or more files deployed as a unit. An assembly always contains a manifest…"

Modules - "A module is a single file containing executable content…"

This is good, we can conclude that an assembly generally has one file / one module, but can have multiple modules (files).

One nice addition, you can use reflection to check out the modules in an assembly. You can research the System.Reflection.Module class: https://docs.microsoft.com/en-us/dotnet/api/system.reflection.module?view=netframework-4.7.2

NOTE: careful… "module class" is a very scary term, but that's what it is - a class to describe a module. *brain explosion*.

You can also use a tool that reflects code to check out modules in a compiled assembly.

Here is a simple class reflected with my favorite dnSpy:

And here it is with ILSpy:

For further reading, here is a screenshot of page 140 of ECMA-355 (20181120):

OK, so where would this be useful?

Programming extremely large software can be easier to update with multi-module assemblies.
Incremental download can be achieved with multi-module assemblies.

In the field of what I do (create business software), we don't ever need to use multi-module assemblies and very rarely need to worry about modules within assemblies. You can make money without knowing this level of detail.

Other languages

In VB.NET, modules are something different. VB.NET modules are what static classes are in C#. We use VB.NET modules to depict helper functions and extension methods that we don't want to instantiate, and don't want to inherit.

What about other languages? I'm a C# guy, so I'm not sure the below is correct:

Python - in Python, modules allow you to logically organise your code. It is a single file that can contain functions, classes, variables.
Java - A Java module is a new feature in Java 9. It lets you group Java packages and your application into a module.
JavaScript - ES6 adopted the module concept. Each module is a piece of code that is executed once it is loaded. Modules can contain functions and variables. Modules can import other modules. Modules are singletons :) It is interesting to check out how the ES5 history makes use of modules, in cases such as CommonJS modules and AMD. I will not go into these here.

Summary

In summary, modules are logical groupings of code. Programming languages implement them differently, leading to changes in the meaning of the term. In some languages, modules are packages. Some languages have other constructs for logically grouping code, such as the namespace concept, which also exists in C#. Most of this "theory" is based on the design of "modular programming".

Here is a good read is the Modular programming article on Wikipedia: https://en.wikipedia.org/wiki/Modular_programming

Good luck with it all.

Developing hybrid ASP.NET Core applications in Azure App Service consuming on-premises data and Microsoft Graph

2018-05-29T18:22:00+00:00

My European Collaboration Summit 2018 session on hybrid application development was one of the more exciting talks that I've done - and all my demo's worked!

You can find the demo code here: https://github.com/OneBitSoftware/ECS2018-Demo-HybridApps

And I've uploaded the slide deck on SlideShare:

Developing hybrid ASP.NET Core applications in Azure App Service consuming on-premises data and Microsoft Graph from SharePointRadi

Feel free to let me know if you found this useful! Enjoy!

Slides and scripts from the Ultimate, "Modern" PnP training day at the European Collaboration Summit in Mainz, 2018

2018-05-28T14:56:00+00:00

ECS2018 is still going and absolutely awesome fun. Together with the PnP team, we're having a great time sharing the latest and greatest.

Here are my slides from the Branding session:

ECS2018 PnP Training Day - Branding from SharePointRadi

And here is my PowerShell script that I used in the demos:

Import-Module Microsoft.Online.SharePoint.PowerShell

# Connect to SPO for native cmdlets - radi@M365x957147.onmicrosoft.com
Connect-SpoService -Url https://M365x957147-admin.sharepoint.com

# Set Hide
Get-SPOHideDefaultThemes #Returns False when default themes are visible
Set-SPOHideDefaultThemes $true

# Retrieve a Theme as JSON
Get-SPOTheme -Name "Custom 1" | ConvertTo-Json

# Remove a theme
Remove-SPOTheme -Name "Custom 2"

# Set the theme JSON
 
$themeJSON = @{
"themePrimary" = "#800080";
"themeLighterAlt" = "#FF00FF";
"themeLighter" = "#FBC0C3";
"themeLight" = "#808000";
"themeTertiary" = "#FFA500";
"themeSecondary" = "#00FF00";
"themeDarkAlt" = "#00FF00";
"themeDark" = "#136D80";
"themeDarker" = "#0000FF";
"neutralLighterAlt" = "#FFF7D7";
"neutralLighter" = "#DDEFF8";
"neutralLight" = "#F2C975";
"neutralQuaternaryAlt" = "#20B2AA";
"neutralQuaternary" = "#008B8B";
"neutralTertiaryAlt" = "#8B0000";
"neutralTertiary" = "#98FB98";
"neutralSecondary" = "#DAA520";
"neutralPrimaryAlt" = "#800000";
"neutralPrimary" = "#008080";
"neutralDark" = "#B3AA9B";
"black" = "#000000";
"white" = "#ffffff";
"primaryBackground" = "#ffffff";
"primaryText" = "#000000";
"bodyBackground" = "#ffffff";
"bodyText" = "#000000";
"disabledBackground" = "#CD9D7C";
"disabledText" = "#765A4F";
"blue" = "#0000FF";
"backgroundImageUri" = "https://M365x957147.sharepoint.com";
 }

# Build a Theme Object with a neat for each
$themeObject = New-Object "System.Collections.Generic.Dictionary``2[System.String,System.String]"
$themeJSON.Keys | %{$themeObject.Add($_, $themeJSON[$_])}

# Add theme to gallery
Add-SPOTheme -Name "Custom 2" -Palette $themeObject -IsInverted:$false

# Automate applying a theme today
Connect-PnPOnline -Url https://M365x957147-admin.sharepoint.com

# Set up site script - see https://docs.microsoft.com/en-us/sharepoint/dev/declarative-customization/site-design-overview
$siteScriptJSON = @'
{
    "$schema": "schema.json",
    "actions": [{
            "verb": "applyTheme",
            "themeName": "Custom 2"
        }
    ],
    "bindata": {},
    "version": 1
}
'@
 
# Add script and design
$siteScript = Add-PnPSiteScript -Title "Custom 2" -Content $siteScriptJSON
$siteDesign = Add-PnPSiteDesign -Title "Custom 2" -SiteScriptIds $siteScript.Id -WebTemplate TeamSite

# Invoke the applying of the site design
Invoke-PnPSiteDesign -Identity $siteDesign -WebUrl https://M365x957147-admin.sharepoint.com

Enjoy and good luck with your ultimate, modern branding!!

Testing and Simulating HTTP 429 Too Many Requests with Fiddler Extensions

2018-02-18T07:59:00+00:00

This post is applicable to any scenario where you want to simulate the return of HTTP 429 Too Many Requests messages. In my case, it is applicable to SharePoint Online as I want to test code that handles 429 errors.

See the end for a link to a code sample in GitHub.

Problem and Solution

My requirement is simple - test code that can recover itself from SharePoint Online throttling mechanisms.

If you are unfamiliar with request throttling in SharePoint Online, please read these:

What we don't want to do is actually stress test SharePoint Online and play with its throttling mechanism. That's just weak and chances are that we can't generate enough requests to get throttled from a single developer machine. The only way to load a cloud is with another cloud, and that is expensive.

My solution isn't complicated either - use a Fiddler extension to tamper with the response packets from SharePoint Online. What we want is to process some requests, then transform the response HTTP code and status message of others to 429, then in code we can capture true 429's which appear to come from SharePoint Online.

My use case is to test that ExecureQueryRetry works in OfficeDev.PnP Sites Core: https://github.com/SharePoint/PnP-Sites-Core/blob/master/Core/OfficeDevPnP.Core/Extensions/ClientContextExtensions.cs#L73 So we can further develop the ExecuteQueryRetryAsync method as well.

Fiddler Extension Build and Deploy

In order for Fiddler to pick up your extension, it needs to go in one of either folders.

%userprofile%\documents\Fiddler2\Scripts to make the extension available to the current user. This is the "My Documents" folder
Use %programfilesx86%\Fiddler2\Scripts to make the extension available to all users on the machine. This is the 64-bit Program Files folder. Change this to where your Fiddler application is installed.
Your paths might differ if you have installed things differently or use a newer version.

2. You need to build a class library that has a public class which implements IFiddlerExtension. In your class library, add a reference to Fiddler.exe, straight from the folder where it is located. Your DLL will go in the \Scripts folder relative to the Fiddler.exe you are referencing.

3. You need to set a Fiddler.RequiredVersion assembly attribute somewhere in your code. I do it in my class that implements IFiddlerExtensions. I couldn't get the extension to work without this, so I am assuming it is required.

If all of the above are true, Fiddler's extension tab in Fiddler Options will show that your assembly is loaded.

The following post-build command on the project automates my DLL to go in the right folder:

copy "$(TargetPath)" "%userprofile%\My Documents\Fiddler2\Scripts\$(TargetFilename)" /Y

Fiddler Extension Design - Throttling Simulation

You can implement the "public void AutoTamperResponseAfter(Session oSession)" method from the IAutoTamper2 interface. The passed Session object gives you access to the HTTP packet, amongst other things.

In my case, I would like to allow 10 packets to pass within 5 seconds, then tamper any more packets within that moving 5 second window. I implement a ConcurrentQueue collection, because you typically have multiple in-flight requests. ConcurrentQueue deals with the cross-message coordination of counters and does a perfect job here.

In the end, I modify the HTTPResponseCode and HTTPResponseStatus messages. Note that the return Content-Type must be "text/html" and the body must not be empty.

I also do simple exclusions, as I only want to throttle *.sharepoint.com requests in my use case. Change this to suit your situation.

Testing and Validation

Here it is in action. You can see that some requests get throttled as long as they are more than 10 and within a 5 second window. Then, reissued requests pass through as expected. This lets me test PnP's ExecureQueryRetry method.

Sample Code

All of the above is uploaded on GitHub here: https://github.com/OneBitSoftware/Http429.SimulateThrottling

Feel free to fix issues or add functionality, or just say hello.

Happy throttling!

Asset Bundles in Unity - 2017/2018 Edition

2018-01-30T08:34:00+00:00

This post is a detailed write-up on what Asset Bundles are in Unity, what they should be used for and how they work internally.

Asset Bundles in Unity are an editor and platform feature that let you configure and deliver dynamic content to your games or apps. You can do "in-game" downloading of external content.

The key use cases for Asset Bundles are:

Download in-game content during the first run of the game with the goal of minimizing the initial installation size
Update game content by replacing the asset bundles located on a server - a very poor man's game objects patching system
You can delivery variants of content for different devices and runtimes - for example differentiate between high-resolution and low-resolution assets, or Android vs Windows assets, potentially allowing you to optimize for devices
The Asset Bundles API lets you manage what is stored in memory for a potential manual memory management use-case
Separate the asset bundles from the standard build - the key reason for this could be to reduce build times

What are Asset Bundles?

They are containers for objects and assets. You can think of them as packages with lots of assets inside.
Asset Bundles store serialized versions of Unity game objects, such as scenes and everything that a scene could contain.
An Asset Bundle can contain an entire scene, but if it does - it can't contain anything else (in practical terms).
Asset Bundles are platform-specific
Asset Bundles use LZMA or LZ4 compression (depending on the Editor version), which can significantly reduce your build size.
Asset Bundles use memory. Just in case it is not obvious, the HTTP client needs to download a the bundle (and dump it from memory to disk), release memory, then go about decompressing, which again consumes memory. In big projects we really have to manage this process carefully.
Asset Bundles are not part of the build output. You execute the bundle separately and deal with the output on your own. Your typical next step is to upload the bundle to a web server (which you also manage on your own) and get your game/app to download it and store it on disk.

The entire Asset Bundles functionality is encapsulated in the following places:

An Asset Bundle Manager and a (quite crappy) DEV/TEST server, which are delivered through the Unity Asset Store. That's right - you have to "import" it into your game/app. It feels *very* unnatural and more like a hack, however it is understandable why it is implemented that way. It puts the asset build pipeline in total control of the developer.
The UnityEditor.BuildPipeline class within the UnityEditor.dll assembly. This is where Unity contains coordination logic regarding asset bundles, but the actual bundle packing mechanism is buried in the native part of Unity.

The Asset Bundle Manager

This tool is something you install/import in your game as raw .cs files from the Asset Store. Some scripts are targeted for the Editor experience, while others are examples of how you use asset bundles at runtime in your game/app.

The AssetBundle Manager introduces Editor screens to help you build Asset Bundles. This capability will go over the project hierarchy to detect AssetBundles, then build them in the projects AssetBundles folder.

Since AssetBundles internally use WWW.LoadFromCacheOrDownload, the tooling provides you a local web server, which is a bit too simple, and a Simulation mode so you don't download from a web server.

Apart from the fact that the AssetBundle sample doesn't build with newer versions of Unity (straight from the Asset Store and created and maintained by the Unity team!!), the problem with AssetBundles is that it is written terribly, has TODO's in the code and consumes quite a lot of resources. See below:

The main class itself is called AssetBundleManager. Although every function in it is static, it inherits from MonoBehaviour to make use of the public void Update method called on each frame. The fact that it is not static, yet every function call is static means that you

The Initialize method must be called to load something called the AssetBundleManiifest

All calls to load an asset bundle are classes that inherit the abstract class AssetBundleLoadOperation and get stored into a load queue:

static Dictionary<string, WWW> m_DownloadingWWWs = new Dictionary<string, WWW> ();

The AssetBundleManager class Update method (MonoBehaviour, fires on every frame) checks m_DownloadingWWWs and executes all pending AssetBundleLoadOperations. When done, they are removed from the queue and disposed.

Out of all of the above, probably the only exciting thing is the m_DownloadingErrors Dictionary:

static Dictionary<string, string> m_DownloadingErrors = new Dictionary<string, string> ();

You can use that to check if your download tasks have failed.

Where did the Web Player Asset Bundles go?

As of version 5.3, Unity deprecated the Web Player build target - an in-browser plugin to run Unity games. During the time it still existed, around the 4.0 to 5.3 era, many build pipelines created .unity3d packages. These are Web Player packages in most cases. Some games (such as Hearthstone) use the file extension for their asset bundle build pipeline, but in essence and as far as asset bundling goes, they are asset bundles :)

You can verify this even in the documentation for WWW.LoadFromCacheOrDownload (https://docs.unity3d.com/ScriptReference/WWW.LoadFromCacheOrDownload.html):

The Asset Bundle Manager in 5.4 and later packs files without file extensions. This is fine for most use-cases, however some web file servers require extensions on files. This is why it is a good idea to give them an extension.

You can use tools such as Disunity and Unity Asset Bundle Manager to extract .unity3d files.

Asset Bundle Browser

This is another tool you install from the Asset Store:

Its primary purpose is to help you manage asset bundle dependencies, as well as help you build them with more control:

Apart from that, it is a great set of (badly?) written code samples to help you understand the Asset Bundles API better, so go check them out:

AssetBundle Graph Tool

The Unity team has also released the AssetBundle Graph Tool, a set of editor tools to improve and visualise the AssetBundle workflow and build experience.

There is plenty of reading material on it, so no reason for me to repeat it:

The future of Asset Bundles

This article is written on the 30th of December, 2017 and rechecked in January. At the time of writing, AssetBundle Addressables are not yet released for Unity and the current public version is 2017.3. I'm not part of the Unity team (although I'd love to be) so everything in this heading is taken from the demos performed by the Unity team at Unite events.

Addressables - The long-awaited replacement of the Resources folder. The idea here is to remove/reduce the overhead of management of AssetBundles through addressable assets and a basic online hosting setup.
Resource Manager API - a new set of API's designed to work with Addressables to automate dependency loading in a much better way than the current AssetBundle API.

We're expecting releases and fixes to the releases soon. I don't have an ETA for you.

Resources

The following are articles you should check out on Asset Bundles:

https://unity3d.com/learn/tutorials/topics/best-practices/assetbundle-fundamentals - AssetBundle Fundamentals
https://unity3d.com/learn/tutorials/temas/best-practices/assetbundle-usage-patterns - AssetBundle Usage Patterns
https://gameart.eu.org/reverse-engineering-decompiling-unity3d-files/ - a blog post on reverse engineering .unity3d files with QuickBMS
http://aluigi.altervista.org/quickbms.htm - The QuickBMS script
Unite Europe 2017 - AssetBundles: current and future Europe 2017 - AssetBundles: current and future
Unite Austin 2017 - Using the New Asset Bundle Browser & Graph Tool - Austin 2017 - Using the New Asset Bundle Browser & Graph Tool - Here it is:
Unite Austin 2017 - A Look Ahead: Resource Manager, Addressable Assets and Live Preview - Austin 2017 - A Look Ahead: Resource Manager, Addressable Assets and Live Preview - Here it is:
https://imbushuo.net/blog/archives/505 - Deep dive into UnityFS: structure and implementation
https://hearthsim.info/ - This mob has done tons of work on extracting data from asset bundles and is a great reference

Good luck with your bundling.

A sample ASP.NET Core starter solution with Identity, Entity Framework 6 and a Repository Pattern Implementation

2017-04-17T12:36:00+00:00

This sample solution makes use of ASP.NET Core with ASP.NET Identity over Entity Framework Core running side-by-side with Entity Framework 6. Look at this if you don't want Entity Framework Core for your data layer, but still want to use ASP.NET Identity in ASP.NET Core and benefit from its out-of-the-box implementation (which is awesome!).

Goals and inspiration

I was inspired by Mosh Hamedani's work on "Repository Pattern with C# and Entity Framework, Done Right". I think the theory that Mosh has put forward on the difference between a repository pattern and EF's DbContext is well structured and well explained. The sample code is also worth checking out.

I wanted to take it one step forward and put the pattern into an ASP.NET Core starter solution. The key point I want to demonstrate is using critical components of an ASP.NET Core MVC web application with an implementation of the repository that works side-by-side with the ASP.NET Identity 3 implementation. I wanted to make the repository asynchronous as well, because there's not many good examples out there.

The catch: ASP.NET Identity 3 uses ASP.NET Core and Entity Framework Core. That's all great, but many are not ready for EFCore and want the battle-tested Entity Framework 6. EF6 is feature packed, while EFCore is still very feature-limited.

In summary, this sample answers these questions:

How do you inject an EF6 context with a connection string in the ASP.NET Core service collection?
How do you run ASP.NET Identity on EFCore and the rest of the application on EF6?
How do you properly implement the Repository pattern with EF6?
How do you make your Repository async?
How do you maintain one database for both your custom entities and ASP.NET Identity?
How and where do you implement a Services layer?
How do you make your Controller methods Async?
How do you use Services with Controllers?
How do you keep the connection string in one place.

GitHub Repository: https://github.com/OneBitSoftware/AspNetCore.Ef6.Identity3.StarterSolution

The Data class library

I've taken all data access classes in relation to EF6 and the Repository pattern in a separate class library. This library doesn't do anything else apart from the responsibility of data access. Technically you can split it into more assemblies if your case makes sense.

There's no need for me to describe the Repository implementation itself, because Mosh has done it very well. Follow his video or code sample for background knowlede. I am focusing on using it in an ASP.NET Core MVC application.

This is what my *.Data project looks like:

Asynchronous Repository

Many operations against Entity Framework 6 make sense to be asynchronous. You get the true benefit if the async chain goes all the way up to the controller. That is why my repository interface returns Task<T>.

public interface IRepository<TEntity> where TEntity: class

{

Task<TEntity> GetAsync(int id);

Task<IEnumerable<TEntity>> GetAllAsync();

Task<IEnumerable<TEntity>> FindAsync(Expression<Func<TEntity, bool>> match);

Task<TEntity> AddAsync(TEntity entity);

Task RemoveAsync(TEntity entity);

Task RemoveRangeAsync(IEnumerable<TEntity> entity);

}

The Web project

All of my other bits and pieces I have stuck in the MVC project. I have strictly kept to the empty ASP.NET Core MVP project structure - my Services and View models are here. I have not touched the ApplicationDbContext ASP.NET Identity DbContext class to allow for a quick start.

If you choose, you can break this project into more class libraries. In this sample I keep my MVC(S) Services where my ViewModels are, so Services return ViewModels, which also means that they encapsulate the Factory calls to map Models to ViewModels. If you wish, you can move that responsibility to the Controllers and have your Services return Models (and essentially move Services to their own library).

The Visual Studio tooling and specifically the MVC scaffolding is unfortunately designed (as of now) to have everything in one project, that is when you get good scaffolding possibilities.

Here is what my *.Web project looks like:

Common database for entities and ASP.NET Identity tables

Nobody wants to deal with more than one database unless it is really necessary. In our case we want a web application with objects and user accounts - not a hugely complex thing. A single database would be appropriate for many scenarios.

The sample does just that - both contexts (ApplicationDbContext and my custom entity context) access the same database. They use the same connection string during registration in the ConfigureServices startup method.

services.AddDbContext<ApplicationDbContext>(options =>

options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")));

services.AddScoped<CarApplicationDbContext>(_ =>

new CarApplicationDbContext(Configuration.GetConnectionString("DefaultConnection")));

Here is what the database looks like. EF5 uses the migration history tables to deal with migrations as it should, the ContextKey column here is key.

Target Framework

Entity Framework 6 is dependent on the full .NET Framework. This is why the solution targets .NET Framework (4.5.2) but uses the ASP.NET Core framework and libraries to run the application.

Code on GitHub

Feel free to use as you please. I'd be glad to get your feedback and comments.

GitHub Repository: https://github.com/OneBitSoftware/AspNetCore.Ef6.Identity3.StarterSolution

A Rational Guide to Dealing with Azure's "MySQL in App" App Service Capability and PHP Web Applications - Part 2

2017-04-17T12:10:00+00:00

This is a multi-part series with tips and solutions to roadblocks that I faced in dealing with the "MySQL in App" service.

Part 1 - Overview and different interfaces to manage PHP
Part 2 - Migrating a really large MySQL database to Azure's "MySQL in App" (this post)
Part 3 - Splitting a really large MySQL database dump/export file into smaller pieces
Part 4 - Troubleshooting tips

Part 2 - Migrating a really large MySQL database to Azure's "MySQL in App"

You can use the phpMyAdmin interface to do most operations on your database, but it has never been good with importing/exporting databases over 8MB, and these days 8MB is not that big.

I had to operate on a 3GB database and using phpMyAdmin to restore it was no option at all.

Export your database from your source with the "mysqldump.exe" command:

mysqldump -u [user_name]-pdbpassword [database_name] --routines > 20170416_dbbackup_full.sql

(notice the -pPASSWORD syntax, a bit weird but there is no space there).

NOTE: if your .sql file is "huge" see spliting a .sql file on Windows below. "Huge" means the import times out given the size of the App Service resources that you have provided.

Soon we will use the Kudu console, but first access the FTP service that is enabled on your Azure App Service. Get the FTP hostname and credentials from the Overview screen, and upload your large database.

FTP your large database to the D:\home folder:

Once your file is there, you can manipulate it with mysql.exe.

The key thing to gather is the path and port of your MySQL instance (phpMyAdmin could tell you both. The username and passwords you need to know are the "root" and "azure" accounts, both with passwords "password". You also need to specify the --bind-address parameter to localhost.

Pretty much your command looks like this:

D:\home\sql_restore>"D:\Program Files (x86)\mysql\5.7.9.0\bin\mysql.exe" --user=root --password=password --port=56172 --bind-address=127.0.0.1 mydbname-test-01 < 20170401_dbbackup_mini.sql

This is enough for you to restore a large database, but there is one big problem with this approach. The Kudu console will timeout (I think after 30 or 60 minutes) and kill the process that is hosting it. BAD. Basically it looks like this:

Eventually you might get a timeout of the console before the operation completes, terminating it half-way.

See my next post for a solution to the above problem: Part 3 - Splitting a really large MySQL database dump/export file into smaller pieces

A Rational Guide to Dealing with Azure's "MySQL in App" App Service Capability and PHP Web Applications - Part 1

2017-04-16T12:00:00+00:00

If you haven't heard, Azure's App Service PaaS offering comes with a build in MySQL instance. See these blog posts for welcoming info:

This is great because you can move any PHP solution (like Wordpress) that uses MySQL to an Azure App Service instance. You could even use the free tier and host your blog or anything else at your likings.

Working with PHP and MySQL in Azure has its intricacies and definitely doesn't have a lot of information.

This is a multi-part series with tips and solutions to roadblocks that I faced in dealing with the "MySQL in App" service.

Part 1 - Overview and different interfaces to manage PHP (this post)
Part 2 - Migrating a really large MySQL database to Azure's "MySQL in App"
Part 3 - Splitting a really large MySQL database dump/export file into smaller pieces
Part 4 - Troubleshooting Tips

I was tasked with moving a pretty big PHP solution with a big database to Azure.

Here are some key bits of information that I were either not documented or I had a hard time figuring out.

Part 1 - Overview and different interfaces to manage PHP

Azure's portal doesn't have much, but it has a few crucial buttons worth knowing about. I will first explain the Kudu console, because it is a predecessor to tweaking PHP settings.

Kudu Console, aka Advanced Tools

You might be familiar with the Kudu project, a cool set of tools to manage your Azure App Instance. This is key to working with PHP and MySQL.

Access it through the App Service interface in portal.azure.com.

The Kudu UI is simple enough and has tons of useful information:

To access the console, go to the Debug console menu and click CMD or PowerShell, depending on what you need.

This gives you a neat console into the VM behind the App Service (yes, they are VM's!).

Everything under C:\ is blocked, but you can do all you wish under D:\

PHP Settings

An Azure App Service has PHP installed by default, and it has various versions available too. These are changeable in the Application Settings screen on the App Service:

The actual php.ini file is located in "D:\local\Config\PHP-5.5.38\php.ini" (or as per the version you are using). You can edit that, however it is a much better recommendation to do it the "right way" and create a ".user.ini" file. To do this, add a custom php.ini override file with the configurations you need, place it in your application root and deploy your application. Restart it so you force PHP to re-read its configuration.

Alternatively, you can place a ".ini" file on the D: drive, such as d:\home\site\ini, then add the 'PHP_INI_SCAN_DIR' configuration settings key in "App settings":

Both will work. This is documented much better here: Configure PHP in Azure App Service Web Apps

Managing the MySQL service

You can manage the MySQL service through the Azure portal UI (very limited) or through the well-known phpMyAdmin interface. The Azure App Service comes with phpMyAdmin installed. Reach it through the difficult to see "MySQL in App" option and the "Manage" portal button/link, or through the following URL: https://[appserviceaddress].scm.azurewebsites.net/phpMyAdmin/ mine was https://blog-prod-01.scm.azurewebsites.net/phpMyAdmin/

You will be automatically logged in with an "azure" account.

And the not-so-visible Manage button:

You will reach the phpMyAdmin interface:

This is enough to get you going with basic PHP and MySQL management. My next post describes how to import large MySQL databases: Part 2 - Migrating a really large MySQL database to Azure's "MySQL in App"

Jerky mouse movement with a laser Logitech mouse

2017-03-17T10:59:00+00:00

This is one of those off-topic blog posts that I eventually write, but I spent significant time trying to figure out why my Logitech G700s mouse doesn't move in a straight line and I think it would be worth it.

I couldn't find solutions for this on the web. My problem is simple - I move the mouse and it doesn't go smooth in a straight line. It jumps sporadically across the movement vector at about 0.5-1cm off its course, making it impossible to use for games or anything for that matter. I can't really demonstrate it easily, but it was bad.

I got initially put off in the wrong direction - drivers. I did all kinds of installs and uninstalls, both native windows and Logitec. I played with the pointer speed settings both in Windows devices and in the Logitech mouse drive software. None solved my problem.

I even tried different mouse pads.

The issue at hand is that it is just dirty - the laser heads have very tiny dust particles that are very difficult to see. I tried blowing them off, but I'm not a good blower.

My fix is easy - I stuck an almost dried out wet wipe tissue into the laser hole and rotated it a few times. The laser heads we're visibly cleaner.

Now it works like brand new. This is one of the most smoothest mice I have owned and it is fantastic.

Happy cleaning!