My findings in Microsoft world

Integrating Flic with Microsoft Flow and Azure

2017-11-16T12:18:00.001-08:00

Overview

Recently at an internal conference, I got my hands on this pretty cool tiny button Flic, go read about it here. Apart from all the cool stuff you can do with Flic’s extensive support for integration, my interest was around how Flic can integrate with Microsoft Flow. However, I wanted to share some clarify on how can you do this and things that Flic could do better to make this integration experience better.

What I have blogged here is to simply capture each click, double click and hold events from the Flic button, trigger Microsoft Flow from the Flic mobile app. From the Microsoft Flow, capture each of these events, log the trigger message as an Azure Queue message.

Once the message got in to the Azure world, you have many opportunities, such as to pick up the incoming message by integrating using many of the Azure event driven models such as ….

Setting Up your Flic

I am assuming you have your one or more Flic buttons. Go ahead and setup your Flic. This is very simple and straight forward step following guidance from the Flic.io site.

As I was setting up my single Flic I had on my iOS app (Equally applies to Android as well), I named my flic like below. So make sure to give unique name for each of your Flics.

Microsoft Flow App Install and Setup

On your mobile device, ensure to install the Microsoft Flow app from your app store. Once the Flow app is installed, ensure you login to the Flow app with your account that is same as your Azure account. The email account for the Flow app and the above Flic app doesn't need to be the same, but just remember which email you are using in each app to ensure the connectivity between each app.

Setting Up Flic to do Flow tasks

Now lets set up your Flic to call the Flow. So click on your Flic from the mobile app (Not the physical Flic button). For each event Click, Double Click and the Hold, assign Microsoft Flow from the Advanced category and your assignment would look like below. (What can Flic do better is allow this trigger integration to be named as this will get listed as simply Flic in the Microsoft Flow later) .

Azure Queue

From your Azure account, login to the Azure portal and create a generic storage account and you can dedicate a separate Resource Group.

From this storage account, under the Queue, create a new Queue and I assigned name such as single-clicks assuming I would only write a single click events to this queue.

Likewise I have created separate queue for each type of event: double-click and hold.

Building Microsoft Flow

Now run the Microsoft Flow from the Mobile or browse to the flow.microsoft.com on PC as you can build the App from either place with same consistent interface. Below I will show from the mobile for simplicity.

Form your Microsoft Flow app, choose to create new Flow and then search for ‘flic’ and you will see two below triggers. Choose When a Flic is pressed

Choose any, this is just to establish the body of the incoming message from the Flic.

Next add Parse JSON action to parse the incoming message as JSON object and add the body as Content. Next add same payload as {“body”:”text”} to auto generate the schema.

Next lets add a Switch so that we can handle each click type and write to respective Azure queues:

Next choose to Click type as On Switch condition.

Choose action as Put message to queue

Choose single-click queue to write the single click event message.

Likewise add additional switch cases to write for double-click event to double-click queue and hold event to to hold queue.

Now lets save the Flow and test the process with following clicks and observer the corresponding queue in the azure.

click

double click

hold

If you do not see expected message, then check to ensure your flow is working properly.

Understanding and Deploying Azure IoT Hub Device Provision Service-Part-I

2017-09-09T22:53:00.000-07:00

Overview

Lets get some context around what is this blog about and who and why should you care. (This is in reference to Azure IoT Hub Device Provisioning Service)

If you are a manufacturer of devices or machines or appliances or something that needs to be connected that your customer or consumers buy, meaning you are producing 100s,1000s or millions of these products. When the end consumer gets your product, you want these products to connect to your IoT listening end point in the cloud (such as Azure IoT Hub), in safe and secure way to the nearest geographical IoT end point and you want all of this process to happen seamlessly. Seamlessly meaning at the manufacturing time, without you encoding a Device ID/Key in to your product that your your IoT Cloud End Point provides, Azure IoT Hub Device Provisioning (DPS) is something you should look at seriously and likewise all my global manufacturing customers are! What we call semantically as a “Zero-Touch Provisioning”.

How would DSP work for you?

You will make your products that support TPM/HSM (reference in IoT Hub Device Provisioning Service security concepts). You ship these products to your customers across the globe. After receiving the product, for the first time your customer turns on the product. The product calls home (To the cloud IoT Hub Device Provisioning Service). This is where the magic happens. The IoT Hub Device Provisioning End Point validates your product by the TPM/HSM ID/Keys. Based on the geo location and based on the Enrollment option you have pre-configured in the DPS, the DPS will figure out which of IoT Hubs is the best fit. These IoT Hubs are one of those that you had already pre created across the geo locations that you support your customers.

How can I test this DSP quickly without doing anything to product?

At the time of writing this blog, this DPS service was in preview. But I am going to reference these two articles in sequence and share my experience of an actual build out and provide clarity on the steps and some insights on results. What we are going to be building out is like the 2nd article suggested, is simulate your what product would communicate with the DPS.

I won’t repeat what is already been document in the above referenced articles, so I suggest you read the articles first and reference this blog for clarity on the steps. So continue with the steps in the article and compare my blog to make sure you get your steps correct. I have outlined each major sections to give you context of which section my guideline convers.

Steps described in the article 1 Set up the IoT Hub Device Provisioning Service (preview) with the Azure portal are straight forward but here are the names I had used for reference in the future steps:

Steps described in Article 2 Create and provision a simulated device using IoT Hub Device Provisioning Service

Prepare the development environment

In this section under the steps 7 where you are going to run the Simulator.exe, open another Git Bach command prompt, here I have actual run corrected path(at the time of preview).

The run will prompt for port opening in the Windows Firewall. Click on Allow access.

Create a device enrollment entry in the Device Provisioning Service

Step 1 opening the solution:

In the build output you will notice 1 failed as one of the project is marked for not to build. So your results are good.

Step 2:Right-click the tpm_device_provision project and select Set as Startup Project. . .

Run the solution

The output window displays the Endorsement Key and the Registration ID needed for device enrollment.

Note down these values

Enter to continue the run time command closes.

Simulate first boot sequence for the device

Step 2. In Visual Studio on your machine, select the sample project named dps_client_sample and open the file dps_client_sample.c

Step 3. Assign the ID Scope value to the dps_scope_id variable. Notice that the dps_uri variable has the same value as the Global device endpoint.

Step 4. Right-click the dps_client_sample project and select Set as Startup Project. Run the sample

Notice the messages that simulate the device booting and connecting to the Device Provisioning Service to get your IoT hub information.

Now this is where, what your actual product would do when it calls home for the first time when the customer turns on your product. Lets review this sequence of operations and understand the handshake between your device and the DPS.

In the first communication, your device had established with the global DSP service. This global DPS end point is again geo distributed with globally single unique name making your device provisioning code consistent.

Next your device sends your TPM/HSM attributes. In this case as we choose to simulate the TPM, a registraionID , the protocol name “tpm” and the associated tpm endorsementKey.

Next you will notice the DPS will send 401 status requesting for an authorization, along with the new DSP provided unique authorizationKey.

Next your device will follow the instructions and send the SAS key

At this time the DPS responds acknowledging the authorization with 202 status code and sends the status that the DPS is now “assigning” the device to an IoT Hub

Now once the DPS is successful allocating the IoT Hub, your device will receive the status code of 200. The DPS will register/add your device in the corresponding IoT Hub (“assigniendHub”), there by acquiring a new “device ID” (This is an auto generated name, you should be able to provide a specific name), and all other registration signatures. Next you should see device starts communicating directly with the allocated IoT Hub.

Lets go to that corresponding Iot Hub in the Azure portal.Voila! You should see your device registered.

So this completes all the steps in the 2nd article.

Conclusion

We just simulated a single device provisioning and understanding the end-to-end flow of communication between your device and the Cloud. Your next question is well we didn’t actually have more than one device nor more than one IoT Hub. Sure, read the next article Provision devices across load-balanced IoT hubs and I will follow up with a blog on Experimenting and Extending DPS with multiple simulated devices and multiple IoT Hubs-Part-II (coming soon….)

Dissecting and Experimenting to Extend the IoT Edge Gateway-Part II

2017-08-31T13:29:00.001-07:00

Overview

This blog is an extension of Developing Azure IoT Edge-Part I. Picking up from the Conclusion section from the Part I, the goals for this blog are to explore following objectives.

Exploring SensorTag capabilities
Providing clarity on the telemetry data by dissecting the IoT Edge code base.
Connecting this telemetry output to do Hot Path/Warm Path/Cold Path analytics.
Connecting additional SensorTags to the same Raspberry Pi IoT Edge Gateway

Exploring SensorTag Capabilities

Before we dive in to the telemetry data, lets first take look at the Sensor device itself to better understand what we are ready from it. My SensorTag model number is CC2650. This sensor has quite a punch to the list of sensors it has!

10 low-power MEMS sensors. Imagine all the interesting IoT features you could implement with this cost effective tiny sensor!

light
digital microphone
magnetic sensor
humidity
pressure
accelerometer
gyroscope
magnetometer
object temperature (Make note of this)
ambient temperature (Make note of this)

Understanding the Telemetry Data from IoT Edge

From the output of running the IoT Edge in the Pi terminal, you see below output under column Telemetry Data. Lets understand how is this configured and where is this data coming from and what they meant.

Lets explore following C code base which is responsible for the above display output of the telemetry. Refer to https://github.com/Azure/iot-edge/blob/d2c251d76a231eff3af4ffc4b854031dc9cacdde/samples/ble_gateway/ble_printer/src/ble_printer.c

For each configuration GATT UUIDs , in your configuration file, it is configured to read all the first 7 parameters one time and ready the Temperature every 1 second, which is what the telemetry output you see.

Configuration

From gateway_sample.json

GATT Characteristic IDs

From the ble_printer.c

Telemetry Data

From the IoT Edge run

From the temperature telemetry, displayed as raw Ambient temperature read from the IR sensor, and raw Object temperature.

Understanding the Telemetry Data from monitor-events

Below is the iothub-explorer tool running under the monitor-events mode and displaying each messages received from the connected devices (This is referenced as D2C messages). What you see from each D2C message is BLE device Index, timestamp, the UUID and source, but there are no telemetry data!

From the above telemetry configuration and the fact that

From above, the --- properties ---- is the actual raw telemetry sent by the Sensor Tag to the Pi. This is the raw data and not in JSON format and there is no actual temperature data. The UUID is for temperature signature. More on this later if I am able to discover details on the code base as to how we can fix this to send well formed JSON. Again I will update the Conclusion section to provide the solution in a later blog.

Connecting this telemetry output to do Hot Path/Warm Path/Cold Path analytics

Let me quickly define each of these IoT Analytics approaches:

Hot Path: As messages coming to IoT Hub, you want to perform your analytics as quickly as possible since these are time sensitive scenarios. That means resources with higher cost to get the quickest results is acceptable.
Warm Path: As messages coming to IoT Hub, you want to perform your analytics soon enough that a business defined delay or lag in determining the results is acceptable. That means resources with moderate cost to get the results in business defined time lag is acceptable.
Cold Path: You want to perform your analytics with historical data to derive insights. That means resources with lowest cost to derive the insights is acceptable.

My narrative here is to distinguish each analytics approach in the IoT pipe-line. Your definition may differ.

Lets review how we can begin to approach Hot Path analytics such that we can visualize the data real time in a visualization tool. Following an existing article guidance Visualize real-time sensor data from Azure IoT Hub using Power BI, lets create necessary resources, like wise here I am sharing the results under each sections from the article.

Add a consumer group to your IoT hub

Create a Stream Analytics job

Add an input to the Stream Analytics job

Add an output to the Stream Analytics job

Run the Stream Analytics job

Now as you will notice there is a warning sign from the input that we created, lets further explore that by click on the input.

The warning clearly states the issue. Like mentioned in the Part I blog, the messages that are sent to Iot Hub are not in JSON format.

Lets take this how we can resolve this and follow the Conclusion section. Again I will update the Conclusion section to provide the solution in a later blog.

Connecting additional SensorTags to the same Raspberry Pi IoT Edge Gateway

The fundamental idea of a Gateway concept in IoT Scenarios is that the device or a thing that plays the role of Gateway is that it can connect to many things and play a role of gate keeper to talk further with the Cloud. In Part I we connected a single SensorTag. Here lets explore how we can connect a second SensorTag device.

I had bunch of old SensorTag CC2541, these are 1st generation and no more sold. But for the purpose of this experiment, this works.

First lets ensure that you are not running the IoT Edge, otherwise stop the run.

From the Pi SSH session, lets initiate Bluetooth shell and try connect this 2nd sensor tag and find out the MAC address:

Lets Power On the 2nd sensor tag

Note the MAC 78:A5:04:8C:13:98

Enter scan off to stop.

Now lets connect to this SensorTag in our bluetooth shell

Lets disconnect from the device using the disconnect command and then exit from the bluetooth shell using the quit command:

Now lets add the 2nd SensorTag device to the IoT Hub. Navigate to Azure Portal and to your IoT Hub, Device Explorer and add new sensor tag and make note of the new Primary Key for the SensorTag_02.

Now lets take the original sensor tag sample configuration file, lets rename that to gateway_Sensor1.json and test to ensure the configuration works

Take the gateway_Sensor1.json file and copy to gateway_Sensor2.json. Then lets update the MAC address for the sensor two and save.

Now run the gateway for Sensor 2 to test and ensure the sensor two works, make sure you have the sensor two powered on before the run:

Now lets create a 3rd configuration file that will have both sensors. Copy the Sensor1 file to gateway_Extend.json. Add the 2nd SensorTag device in the configuration file via nano editor from your SSH session:

Locate the Mapping section, notice that under the args where we had our first Sensortag entered, I have comma separated and added our second sensor tag with respective MAC and Primary Key for the deviceKey.

Next locate the below section

Copy and paste an existing section of the “controller_index along with the instructions separated by comma (like curser green highlight below. Update the controller_index with 1 and replace the respective MAC address.

Save the configuration Ctl+O and exit Ctl+X. Lets copy the content of the file and test to ensure the JSON is welformed. I used

https://jsonformatter.curiousconcept.com/ to test.

Now run the IoT Edge Gateway from the Pi. From the bellow error message, it looks like even thought the configuration is well formed JSON, the JSON parsing modules are not able to read appropriate configfuration sections where we extended with the second BLE controller.

Again I will update the Conclusion section to provide the solution in a later blog.

Conclusion

So in conclusion, I am expecting to find out more clarity on the SDK and being able to perform above tasks of correcting the Telemetry message so that it is a well formed JSON and that I can add more than one SensorTag and run a successful IoT Edge.

As I discover more clarity I will update here for next set of blogs to shed some light on these topics.

Developing Azure IoT Edge-Part I

2017-08-29T11:31:00.001-07:00

Overview

If you are evaluating for building out an IoT Edge Gateway and experimenting with Microsoft Azure IoT, in this blog I have shared the experience of building out and deploying an actual IoT Edge using Microsoft IoT Edge Gateway SDK. There is an official document Use Azure IoT Edge on a Raspberry Pi to forward device-to-cloud messages to IoT Hub. This is very good generic guideline document. For clarify here I am sharing the actual steps for those who are trying to build this out.

(You can see that I have been switching between the actual session on the Pi via the connected Keyboard/monitor/mouse vs some time in the SSH session. This could be simple, I just happen to use these as I went through the debugging effort to ensure all works well.)

What do you need to get started?

I won’t repeat what is already been document in the above referenced article, so I suggest you read the document first to understand the modularized architecture of Azure IoT Edge SDK.

In this case I am building out from the Windows 10 and then SSH in to the Raspberry Pi via Putty. My commentary will be to share my guidance to on the order and to resolve any issues you may come across while building out the Iot Edge Gateway.

In the end pay attention to the Conclusion section as I capture the clarifying questions and thoughts on extending the Gateway.

So continue with the steps in the article and compare my blog to make sure you get your steps correct. I have outlined each major sections to give you context of which section my guideline convers.

Development Steps

From here on, continue to follow the steps as described in the referenced article and cross check with the section heading as below to validate your steps for guidance.

Prepare your hardware

In this section you will need to know the IP address of the Raspberry Pi. Connect Keyboard, Monitor and Mouse to the Pi. Since the Pi 3 has WiFi, you can connect to Wifi or Ethernet as convenient to you. I had my setup as below. I had the Pi in a 3D printed case. The purpose of this setup is to ensure that the Pi is booted adequately and other debugging steps when I could not connect to the Pi via the SSH tool from the Windows/Mac desktop. If you had your Iot Edge is configured and everything running fine you would not need the the Keyboard/Monitor/Mouse.

While the Pi is all installed and booted with Raspbian, run the terminal tool in the Pi to get to the command prompt. Run below command to get the IP address:

hostname –I

Sign in with SSH

In this section, you are trying to connect your Pi via SSH from your Windows/Mac PC. You may receive an error message such as below.

Since the Pi has just been installed and booted with Raspbarian, the base OS image many not have OpenSSH. So lets switch back to the Raspbarian in the Pi terminal window and run following command to install the openssh client:

sudo apt-get install openssh-server openssh-client
[ENTER PASSWORD]

Once above install complete, run below command to start the SSH service. If you had resterted the Pi, run below command to ensure that SSH servie is running.

Sudo service ssh restart

Reference From <https://askubuntu.com/questions/30080/how-to-solve-connection-refused-errors-in-ssh-connection>

Now back on your Windows/Mac, try connect with your SSH tool and you should be able to successfully establish SSH connection to the Pi.

Enable connectivity to the SensorTag device from your Raspberry Pi 3 device

In this section it took me couple of tries and reboots of Pi to ensure the bluetoothctl --version was 5.37 as expected to match.

In this section at step “6, Make the SensorTag device discoverable by pressing the small button (the green LED should flash). The Raspberry Pi 3 should discover the SensorTag device:”

First ensure you have inserted the battery provided with your SensorTag BLE device. As outlined in below picture, press the button and observe the green LED to flash to indicate that the device is now on.

In the step 6 “In the interactive bluetooth shell, enter the command scan on to scan for bluetooth devices. The command returns output similar to the following:” In this step we are scanning for our BLE device and finding out the MAC address.

Make note of the MAC address of your sensor tag from the above output as below:

A0:E6:F8:B6:42:87

Step 8 “Connect to your SensorTag device using its MAC address by entering connect <MAC address>. The following sample output is abbreviated for clarity:” In this step, you are pairing the SensorTag with your Raspberry Pi.

Step 9”You can now disconnect from the device using the disconnect command and then exit from the bluetooth shell using the quit command:”

This should complete the “Prepare your hardware” section

Run the IoT Edge BLE sample

The is the meat of the article and this is where you would be building and deploying your IoT Edge on the Raspberry Pi.

But first section is about creating an IoT Hub in Azure and adding a new device (to represent the SensorTag BLE device) from the section “Configure two sample devices in your IoT Hub”. Follow the linked article to create an Azure Resource Group and then create an IoT Hub. In the main article, it was assumed that you would know how to add the device and find the device key. To that end, here I am providing a guidance for this step.

From the Azure portal, navigate to your IoT Hub under the resource group you created above.

From the IoT Hub features, select Device Explorer.

From the top toolbar, click on Add to add a new device.

Enter the suggested Device name under the Device ID. For all practical purposes, for example in a factory floor, this could be each of your machine that you would connect to the Gateway, such as Machine_01, Machine_02 etc.

Save to add the above device and after successful add, you should see the below Device Details blade. From the details blade, copy and note following fields by clicking on the copy button at the end. You will need these for future steps. (Secure these keys as this is what authenticates the Device and the device commination with your IoT Hub and hence I have masked my keys below)

Device ID
Primary key
Connection string—primary key

As mentioned in the article, alternatively you can use below scripted steps to add a device to your IoT Hub and retrieve the Keys. But first you will need an IoT Hub Connection String. Lets get that from the portal.

First step, from the Azure Portal, from the IoT Hub, choose Shared access policies and choose “iothubowner” policy.

Next for this Access policy, copy the Prime Connection String

Make a note of this Prime Connection String, which looks similar to below, (key masked). If you notice, this connection string has SharedAccessKeyName, which is want distinguishes what your device can do with this key. You can always create your own Access Policies with specific permissions and use. For this step, lets stick with the iothubowner to establish the connection from the tools.

HostName=demo-iotedge.azure-devices.net;SharedAccessKeyName=iothubowner;SharedAccessKey=vaC-------------------------------------------------=

For the below step, The login operation on iothub-explorer tool, provide the above connection string and the tool will confirm the connection session.

Once the connection is established, now lests add your new device like below. (This is same as adding the device from the browser.)

From the above step you can make note of the keys/connection strings for future steps.

Build Azure IoT Edge on your Raspberry Pi 3

Now switch to your SSH session you had established earlier and ensure you are logged in to the Pi and at the terminal install dependencies for Azure IoT Edge:

Cloning the IoT Edge code base to your home directory:

Building IoT Edge which creates a Build folder every time you build.

Configure and run the BLE sample on your Raspberry Pi 3

Instructions on how to edit the sample configuration file. Here I am using the nano editor tool and pointing the path to the sample configuration jason file.

Logger configuration

Before you start with the Logger Configuration step you first need to create a folder for convenience to hold your Log file. You can use the SSH session for this step as well or for simplicity (for non Linux users) I used the File Manager to locate and create an appropriate folder .

Next since the IoT Edge expects the actual log file to exist, lets create an empty file by right clicking and choosing to create an empty file. I gave a name such as myiotedgelog.log

Back on the SSH session, while in the nano editor, scroll down and locate the “Logger” section.

Like highlighted above, replace with the full path including the log file name.

BLE Module configuration

From the above step, while in the nano editor , locate the Sensor tag in the configuration file. From the highlighted text, replace with your BLE device MAC.

A0:E6:F8:B6:42:87

IoT Hub module

From the above step, while in the nano editor , locate the IotHub tag in the configuration file. From the highlighted text, replace with your IoTHub Name which is the first part in the hostname that you can copy from the Azure Portal from IoT Hub overview panel.

demo-iotedge.azure-devices.net

Likewise, replace the IoTHubSuffix like below.

demo-iotedge.azure-devices.net

Identity mapping module configuration

From the above step, while in the nano editor , locate the mapping tag in the configuration file. Under the args, replace the Device ID and Device Key with your Device configuration that you had made note earlier.

Now save the configuration file with Ctl+O.

Running the IoT Edge.

I ensured that the BLE device is running by simply pressing the on button from the step “Enable connectivity to the SensorTag device from your Raspberry Pi 3 device” . Then run the ble_gateway like below. You can see the telemetry from the BLE device being transmitted to your IoT Edge running on your Pi.

For testing purpose I picked up the BLE device and gently blew some air from my mouth for few seconds to increase the temperature. As you can see from your terminal read out the temperature readout started to increase.

More on how this information is derived from the SensorTag, read the Conclusion section.

Using iothub-explorer to monitor device telemetry

From your Windows/Mac, assuming you have installed the iothub-explorer tool, you can monitor all the messages that the IoT bub is receiving. In the below command, ensure you are providing the primary key connection string from the IoT Hub Shared Access Policies configuration.

More on what is this telemetry data coming from SensogTag_01, read the Conclusion section.

Running this setup for quite some time now and you can see I have received quite a number of messages at our Iot Hub.

Send cloud-to-device messages

Following instructions in this section of the article, you can use either the Device Explorer or the iothub-explorer to send the device messages. now you can also send the messages to the connected IoT Devices right from the IoT Hub blade in the Azure Portal and thats what I am going to explore here.

From the Azure portal, navigate to your IoT Hub blade. click on the Device Explorer option and choose your device.

From the toolbar, choose “Message to Device” button.

Executing the step 1.Reset all LEDs and the buzzer (turn them off):

After a message is sent successfully, you will see an immediate message on the portal.

From the gateway telemetry you will also notice the Scheduling a new instruction message:

Next executing the Step 2. Configure I/O as 'remote':

Turn on the red LED

From <https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-iot-edge-physical-device#comments-container>

Green LED

Buzzer

At this time I want to ensure I close out the SensorTag with proper messaging and hence I am send the Clear all message.

Conclusion

Following the article and this guidance for the most part you should be able to successfully deploy the IoT Edge gateway.

What I want to take this further for clarity and to extend this to real scenarios are following thoughts:

What was the actual telemetry that came from SensorTag to Raspberry Pi in the --- properties ---- from the out put in the iothub-explorer monitor-events?
How is the telemetry message from the out put of the IoT Edge run in the Pi showing the temperature and what are they?
Can I further connect this telemetry output to do Hot Path/Warm Path/Cold Path analytics?
Then can I connect additional SensorTags to the same Raspberry Pi IoT Edge Gateway? (After all that is the role of the Edge Gateway)

Follow for above answers in my next blog: Dissecting and Experimenting to Extend the IoT Edge Gateway-Part II

Exploring SharePoint 2013 PowerShell Obfuscation

2015-02-03T14:11:00.001-08:00

Overview

When I am exploring the breadth of SharePoint 2013 PowerShell and would like to dig a little deeper to understand what each of the commands are doing and what set of commands are available for given operation or area of SharePoint functionality, or would like to debug some of the run time issues you may incur, I like to explore with the .Net Reflector.

Here I wanted to provide you with some guidance on your way around exploring the SharePoint 2013 PowerShell.

Setup

First of all, of course you will need to have your SharePoint 2013 On Prem installed and an existence of live farm .
You could use any commercial available Obfuscator tool. I use Red gate .NET Reflector.

Exploring

Run the .Net Reflector.
Locate the Powershell assembly as below. With .Net 4, the assembly location is now located under C:\Windows\Microsoft.NET\assembly
For powershell assembly locate the following folder

C:\Windows\Microsoft.NET\assembly\GAC_MSIL\Microsoft.SharePoint.PowerShell\v4.0_15.0.0.0__71e9bce111e9429c\Microsoft.SharePoint.Powershell.dll

Once you have loaded the PowerShell assembly, you will see following name spaces

In order to look up the usual SharePoint Command Lets such as Get/Set/New/Delete/Remove commands look below under the SPCmdlet groups to locate your desired command

Lets explore one of the commandlet we all use Add-SPSolution.
So locate the SPCmdLetAddSolution as below:

On the Details pane you should see the method partial details as below

Now click on Expand Methods to further expand the details of each method. Partial view below:

Here under the SPSolution method you will see that there is call for _LocatFarm object which is based on the SharePoint Object Model.
So let’s click on the add method, and you will see that the call is pointing to the object model, and there you further explore the real call details.

Conclusion

Likewise, being able to explore the specific calls by knowing your API calls, you can further diagnose your Farm Issues, Configuration Issues and Development issues.

Beware of Restore-SPSite Requirements

2014-06-20T13:58:00.001-07:00

Overview

You have SharePoint 2010/2013 Farm deployed. In the same SharePoint Farm, you are trying to back up a site collection and restore on to either a different site URL or managed path or different web application. You are doing this using following PowerShell commands:

Backup-SPSite
Restore-SPSite

Meanwhile in the process, your Backup-SPSite will work successfully, but while executing the Restore-SPSite you might get following errors:

Restore—SPSite : The operation that you are attempting to perform cannot be completed successfully. No content databases in the web application were available to store your site collection. The existing content databases may have reached the maximum number of site collections, or be set to read—only, or be offline, or may already contain a copy of this site collection. Create another content database for the web application and then try the operation
Restore-SPSite : Access is denied. (Exception from HBESULT: 0x80070005 ( E_ACCESSDENIED))

Cause

Condition for Site Restore

When you back up a site collection and restore within the same Web Application, you will need a separate Content Database to restore to. Refer this MSDN Article. Same site collection (will have same SiteID GUID)

If a site collection is backed up and restored to a different URL location within the same Web application, an additional content database must be available to hold the restored copy of the site collection.

Condition for Site Delete

Now if you are backing and restoring to be served from different URL, then you are looking at deleting the site collection before you are restoring the site collection. Starting in SharePoint 2010, the Site Collection deletion process has been changed. Refer to Bill Baer's blog here.

Condition for Site Restore

When you are set with above conditions and trying to restore site collection now you are greeted with the Access Denied warning.
This happens when the user login you are using to login to the SharePoint Server (RDP), this is usually the Farm Admin, is not one of the restoring site collection primary or secondary administrator.

Resolution

You are going to be performing this operation on the SharePoint Server, using the Farm Administrator account.
Your current site collection owner is not the farm admin.
1. In this case, set your farm admin as Primary/Secondary administrator (This will be temporary until your restore operation is complete)
2. If you did not do this and went ahead and already deleted the site collection and you have no back up, still no worries, follow further steps.
3. But at least know who is the Primary or Secondary site collection administrator.
Backup your site collection using the Backup-SPSite.
If you need to delete the site collection because you are trying to move the site collection, then you will need to delete the backed up site collection from SharePoint.
1. So before you go ahead and delete, ensure you have backed up the Content Database that contains this site collection.
2. If you delete site collection using Remove-SPSite, the site collection will be deleted permanently.
3. If you delete site collection from the Central Administration or using Remove-SPSite with -GradualDelete option the site collection will be marked for deletion, and will be deleted based on the timer job "Gradual Site Delete".
  1. You could list the sites marked for deletion by running Get-SPDeletedSite and follow with Remove-SPSite
  2. You could force the timer job Run Now to delete pending the site collection that are marked for deletion.
4. At this stage you site is removed successfully.
Now you are trying to restore your site collection.
1. Use Restore-SPSite PowerShell command to restore.
2. If you are coming across the Access Denied error, that means your Site Collection Primary or Secondary owner is not one of the Farm Admin.
3. In that case, ( If you did not rest the site collection Primary or secondary admin to the Farm admin before the backup), follow below option:
  1. Find out who was the primary/secondary site collection administrator is
  2. Temporarily, make the above site collection administrator a farm admin, provide rights (I gave full rights until Restore was successful) to the restoring Content Database.
  3. Have the site collection administrator Login to the SharePoint Server farm (This is the only way, because you did not reset the site collection owner to farm admin before the backup)
  4. Then run the Restore-SPSite.
  5. Now remove the user from farm admin and remove the SQL Full rights on the content database.
  6. If you had the Site collection administrator reset to Farm admin prior to back up, then set the appropriate business users to the site collection administrators.

MySite Delete Warning Resolution

2014-06-20T13:56:00.001-07:00

Overview

You have SharePoint 2010 or 2013 environment. You have also deployed My Sites and your users have been consuming the My Site Feature.

You had an old Users OU in your AD and now you decided to create a new OU for whatever reason. Then gradually you have been moving the users from old OU to the new OU.

While this moving users has been occurring for period of time, now few end users are receiving a warning email message that one of their reporting employee my site will be deleted.

This issue is around the below described specific scenario and the suggested resolution could be reviewed in your specific scenario based

Issue Diagnosis

Let's say you have following below OUs for in your AD.

OldUsers is the old OU
New.Users is the new OU

In the last one month your IT has been gradually moving users from OldUsers to New.Users OU. In the SharePoint, the User Profile Synchronization is pointed only to the OldUsers OU. As the users were moved to the new OU New.users, following is been happening:

The moved User Profile was deleted from SharePoint.
If that user had a mysite and if the user had an assigned Manager in AD,
- Then the MySite secondary administrator was changed to the Manager
- The My Site was queued for Deletion in 14 Days.
- An email was sent to the Manager (This is first warning)
- After 12 days a second email was sent with notification that after 3 days the site will be deleted.
If that user had a mysite and if the user had no Manager in AD,
- No email sent to any body
- No change in user administrator
- The site is queued for deletion in 14 days

The entire process is managed by a timer job called “MySite Cleanup job”. This timer job runs every day from 1AM-6AM (Default setting).

Resolution

First and foremost disabled this timer job “MySite Cleanup job” to avoid any accidental My Site Deletion. The resolution is to gracefully recover from this issue and not directly edit the database table content which is not supported by Microsoft. So proceed with your own caution.

You will first need to determine the current state of User Profiles in the SharePoint Profile Database. Run the following simple Select query against your SharePoint Profile database

SELECT [DisplayName]

,[Email]

,[NotificationStatus]

,[Created]

FROM [PROD_Profile].[dbo].[MySiteDeletionStatus]

(I have removed the Display Name and emails from the below screen shot.)

For each user, against the Created Date, calculate the 14 days date.

In the SharePoint farm, under UPS revise the Connection to ensure to include the New.User and OldUsers are also included

Run full profile synchronization
Ensure that now you can find all the users (or whatever the latest number from the above SQL query run) under the User Profiles
Under the Monitoring, ensure the MySite Cleanup job is now active and is scheduled to run daily (Default)
Ensure that the daily database backups are done
Next step is to watch out for each of the marked user site collections to be deleted (The Site Collection does not get permanently deleted, with SP 2010 SP1 changes, the site collection now stays in the Site Collection Recycle bin at the Farm level) by the MySite Cleanup job, on the dates that you calculated by adding the 14 days.
After a site is deleted by the MySite Cleanup job, next day, re-run the SQL Query to ensure that the user name is not listed in the SQL Query.
Now run the following PowerShell command to check and restore the deleted site collection from the internal site collection recycle bin.
On any server in the farm, run the Powershell, and run Get-SPDeletedSite, example below, Jane Smith site was deleted.

Now run the Restore-SPDeletedSite as below"

At this stage the user site is back as is and there is no entry for this site in the database table MySiteDeletionStatus
If a manager was assigned as secondary site contact, then from the Central Administration, remove the manager as the Secondary Site Contact.
This should get the User Site Collection re-instated as is.

Future of Implementing Content Types and Site Columns

2014-06-20T13:53:00.001-07:00

Overview

You are using SharePoint 2007, 2010, 2013 as On Prem. You are implementing Site Columns and Content Types to support your desired Taxonomy and document object models within your organization.

Either in a near feature or you are in the process of moving to the cloud with O365, completely or partially for collaboration workloads.

Beware of your implementation choices for Site Columns and Content Types and what is upcoming!

Background

Currently you have following options to implement the Site Columns and Content Type structures within the SharePoint platform and what it means to you.

Options for implementing Site Columns and Content Types and the impact:

Deploying over the browser from Site Settings>Galleries>Site Columns or Content Types
- SharePoint determines the GUID assigned for ID
- Site Columns and Content Types are coded in to the content database

Deploying Declaratively:
- By XML Site Column and Content Type Definitions in the form of Elements.xml
- Baking the above XML definitions in to the Feature Solution
- You determine the GUID assigned for the ID
- Eventually deploying the solution to run as Full Trust Code (FTC)
- Site Columns and Content Types are coded in to the content database

Impact of GUIDs

Impact of the Site Columns and Content Type IDs which are in the form of GUIDs

When you upgrade this site collection to next available version, the GUIDs are maintained.
When you are migrating to O365, you will need a tool to migrate the content.
You will need a commercial migration tool or custom code written tool by yourself.
You should ensure to retain the GUIDs of your Site Columns.
You want to retain the GUIDs so that the Site Column referencing libraries and lists data are intact.

Issue

Like we all know there is no support for Farm Solutions with O365, we have no option to deploy our Site Columns and Content Types with our desired GUIDs.

Content Type Hub

If you have deployed a Content Type Hub configuration to consolidate your enterprise Site Columns and Content Types, with migration to O365, you are still stuck with the above issue. (Also please read below how the migration products behave)

Migration products behavior with the GUIDs

The major migration product vendors I had spoken to, comment that they cannot retain the Site Column and Content Type GUIDs while their tools copy over for obvious reason. However, post the copy process the tools will reset such GUIDs to be all in synch.

Alternative Coding/Scripting Options

For those who have to opt for custom development option, beware of below aspects:

You can implement Site Column with a specific GUID via the AddasXMLSchema method on the SPWeb.Fields object.
However ContentType does not have similar AddAsXMLSchema support API.
However you can implement Content Type with your specific GUID via new SPContentType object and assign the GUID to the myContentType.ID field

Update with SP1

With the recent release of SP 2013 SP1, CSOM APIs has been now revised to support the assignment of specific ID for the content type creation.
From KB2817429 Developers are unable to specify a Content Type ID when creating Content Types in the client object model.
For example see http://blogs.msdn.com/b/vesku/archive/2014/02/28/ftc-to-cam-create-content-types-with-specific-ids-using-csom.aspx

Misleading Visual Studio Online Name

2014-03-13T14:15:00.001-07:00

Overview

With recent release of Visual Studio 2013 and the related Visual Studio Online Cloud Service, I took look at what is Visual Studio Online.

What is Visual Studio Online?

Visual Studio Online is really a hosted source control, work item tracking, collaboration and a build service.
Visual Studio Online is free for teams up to 5 users
Visual Studio Online is also included as part of MSDN subscriptions.

What is Not Visual Studio Online?

Like the name suggest, Visual Studio Online is not an online version of Visual Studio IDE. In order to use the Visual Studio Online you will still need Visual Studio IDE installed on your development environment.

What Source Control Options are available?

If you have used the TFS, the same features are available online like below, alternatively you can use GIT as source control provider. You can choose Process templates to manage your work items.

Lightweight Development capabilities via Monaco.

However, there are lightweight development capabilities within the Visual Studio Online server. Below are the quote from "http://blogs.msdn.com/b/somasegar/archive/2013/12/29/10465627.aspx"

Visual Studio Online “Monaco” is a coding environment for the cloud, in the cloud. It complements the desktop IDE as a low friction experience that will help you get started, or make quick changes, to an existing cloud service. And it is integrated with Visual Studio Online.

“Monaco” is already being used as the technology behind other cloud-based developer experience, from Office 365 “Napa” development to SkyDrive file editing.

Conclusion

As developer when we use Visual Studio, it is all the capabilities of developing various software components using variety of support languages, deploy and debug and test. Just with the lightweight Monoao capability (No Server side coding). the name Visual Studio "Online" is misleading!.

My Publications

2014-02-24T16:49:00.001-08:00

How to develop your SharePoint Forms Strategy for Cloud and On-Prem

Practical SharePoint 2013 Development Environment

Planning and Designing SharePoint Products and Technology Solutions for Geographically Dispersed

My Certifications

2014-02-23T17:32:00.001-08:00

Azure

AZ-203 Developing Solutions for Microsoft Azure
70-532 Developing Microsoft Azure Solutions

SharePoint 2013

70-488 Developing Microsoft SharePoint Server 2013 Core Solutions
70-489 Developing Microsoft SharePoint Server 2013 Advanced Solutions

Microsoft Certified Application Developer
- Microsoft Office SharePoint Server 2007, Application Development
  Microsoft Windows SharePoint Services 3.0, Application Development
- Microsoft Office SharePoint Server 2007,Configuration
- Microsoft Windows SharePoint Services 3.0,Configuration

.NET and the Microsoft .NET Framework
- Developing XML Web Services and Server Components with Microsoft Visual C#
- Developing and Implementing Web Applications with Microsoft Visual Basic .NET
  and Microsoft Visual Studio .NET
- Designing and Implementing Databases with Microsoft SQL Server’ 2000
  Enterprise Edition
- Designing and Implementing Desktop Applications with Microsoft Visual Basic 6.0
- Microsoft Visual Basic 5.0 Programming

My Other Blogs

2014-01-10T13:51:00.001-08:00

Netwoven blogs

My Speaking Engagements

2013-08-16T16:47:00.001-07:00

2017-2018- Hosted 4 Microsoft Containers Workshop sessions

2018-2019- Hosted 4 Microsoft DevOps sessions

2016-2018- Hosted 21 Workshop Sessions on IoT Camp

5/29/2018: San Francisco Azure DataFest: Advanced Analytics & Big Data Conference. Azure IoT Hub Deep Dive

6/11/2016: SharePoint Saturday Conference. Keynote Bonus topic: Microsoft Azure and IoT

4/28/2016: Introducing Microsoft Azure Cloud platform and the IoT for SharePoint developers!(NorCal 365 & SP User )

4/9/2016: IoT for Data Guys and Gals (SQL Saturday Conference)

8/25/2015: SharePoint, Nintex and DocuSign – Putting it all together (YouTube, SlideShare)

6/4/2015: Silicon Valley SharePoint User Group: Insights from Ignite, Hosted Guest Speakers.

5/30/2015: Silicon Valley SharePoint Saturday Conference: Apps Strategy for Business, IT and Developers

4/1/2015: Netwoven Webinar: Nuts and Bolts of Building Compliance Process with Nintex and SharePoint 2013 (YouTube)

3/20/2015: Sacramento SharePoint User Group: Strategy for Forms and Workflow Patterns with Nintex

2/26/2015: East Bay SharePoint User Group: Strategy for Forms and Workflow Patterns with Nintex

10/14/2014: Netwoven Webinar: Powerful and Quick Workflow Automation Solutions with Nintex (YouTube)

8/13/2014: Netwoven Webinar: How to maximize your O365 Investments with OneDrive for Business (YouTube)

6/11/2014: Netwoven Webinar: How to develop your SharePoint Forms Strategy for Cloud and On-Prem

6/5/2014: Peninsula O365 User Group: Office 365, Calendars, Folders and Delegation

8/16/2013: Silicon Valley SharePoint User Group, Records Management and eDiscovery in SP 2013

8/7/2013: Silicon Valley SharePoint User Group, Panel Meeting

6/22/2013: SharePoint Saturday Conference, Silicon Valley, Building Practical SharePoint Development Environment SP 2013

5/16/2013: Silicon Valley SharePoint User Group, SharePoint 2013 WCM, ECM and DM Features

My Conclusions on Installation and Configuration of SharePoint 2013 Farm (& Related) and Development Efforts

2013-08-05T19:50:00.001-07:00

This blog is part of Series : Comprehensive SharePoint 2013 Development Environment Installation and Configuration

Overview

With the past several weeks of (Retries over Months) effort on trying to establish an orderly process for Installing and Configuring SharePoint 2013 Farm for the purpose of Development (Equally applies to most QA/Stage/Production environments as well), I wanted to capture some of my conclusions on the overall process and the effort involved.

The overall SharePoint install involves several downloads (depending on other sub components you choose to install more software to be downloaded)
Other than basic SharePoint Install, typical farm installation process is very comprehensive (like this main series blog itemized)
Certain installations and configurations can be done only with a PowerShell Scripts and as you can experience following this blog series, there are several of them.
I have gone through several installation missteps before I got it right (some of feature to work properly). So the sequence of installation and configuration is important to get it right.
Weather you are following the installation and configuration wizard or combination of PowerShell scripts, it it very critical to ensure appropriate configuration parameters are used.
So by the time you had completed the baseline install and configurations, you would probably start jumping in to to start your development.
During this initial development effort uncover several security and further configuration requirements.
So what is the outcome: “Inefficiencies in starting an effective development”

Now having said that, don’t get me wrong, I am hardcore developer, but this “Time To Build” is too tedious. By the time IT learns all the details and get their environment ready for developers it will be a while!

How to Configure UAG 2010 for SharePoint 2013

2013-07-18T14:43:00.003-07:00

This blog is part of Series : Comprehensive SharePoint 2013 Development Environment Installation and Configuration

Overview

This blog article discusses how to configure UAG 2010 for exposing your SharePoint 2013 Site(s). Assuming you have followed the previous blog on how to install UAG.

Steps

From the Program Menu, run Forefront UAG Management

Assign your external and internal NIC cards accordingly

Now create an UAG Trunk

Provide appropriate Public Host name for your UAG Trunk, this should be other than your Extranet URL.

Something like Login.myexternaldomain.com. Provide your Public IP (I have a dummy IP below)

Add and select appropriate authentication providers below and I have left short of this here.

Follow next to Create and configure an Extranet…(Coming soon…)

Installing UAG 2010 for SharePoint 2013

2013-07-18T14:43:00.001-07:00

This blog is part of Series : Comprehensive SharePoint 2013 Development Environment Installation and Configuration

Overview

This blog article discusses how to install UAG 2010 for the purpose of exposing your SharePoint 2013 Site(s).

Important:

You will need to install UAG 2010 with SP3 (which has support for SharePoint 2013) (And there is no UAG 2012 or 2013)
UAG 2010 SP3 does not support Windows Server 2012, so you will have to use Windows 2008 R2.
You will need minimum Two NIC Cards. One to point to your Public IP and second one to point to your internal network where your On-Prem SharePoint 2013 Farm is installed.

For IT professionals follow the recommended Network Adapter Configuration for Forefront UAG Servers

Reference: http://www.wictorwilen.se/sharepoint-2013-and-unified-access-gateway-uag-2010-service-pack-3

Steps

Internal

External

Prepare your UAG Installation media as below:

Install Forefront UAG with SP1 as a clean installation.

Install Forefront Unified Access Gateway (UAG) Service Pack 1 (SP1) Update 1

From <http://www.microsoft.com/en-us/download/details.aspx?id=27604>

Install Microsoft Forefront Threat Management Gateway (TMG) 2010 Service Pack 2

From <http://www.microsoft.com/en-us/download/details.aspx?id=27603>

Install Forefront Unified Access Gateway (UAG) Service Pack 2

From <http://www.microsoft.com/en-us/download/details.aspx?id=30459>

Install Forefront Unified Access Gateway (UAG) Service Pack 3 (SP3)

From <http://www.microsoft.com/en-us/download/details.aspx?id=36788>

Follow next blog for Configuration…

Configure SharePoint 2013 for Apps

2013-07-18T14:30:00.001-07:00

This blog is part of Series : Comprehensive SharePoint 2013 Development Environment Installation and Configuration

Overview

This blog article discusses how to configure your environment for SharePoint 2013 Apps.

Reference: http://technet.microsoft.com/en-us/library/fp161236.aspx

Steps

Create new Zone

Under your DNS, first create a forward lookup zone for the app domain name

Create a wildcard Alias (CNAME) record for the new domain name

Ensure Servers are started

Validate Service Applications

Create Subscripts Settings Service Application via Powershell

# Gets the name of the managed account and sets it to the variable $account for later use.

$account = Get-SPManagedAccount corp\spservices

# Creates an application pool for the Subscription Settings service application.
# Uses a managed account as the security account for the application pool.
# Stores the application pool as a variable for later use.

$appPoolSubSvc = New-SPServiceApplicationPool -Name SettingsServiceAppPool -Account $account

# Creates the Subscription Settings service application, using the variable to associate it with the application pool that was created earlier.
# Stores the new service application as a variable for later use.

$appSubSvc = New-SPSubscriptionSettingsServiceApplication –ApplicationPool $appPoolSubSvc –Name SettingsServiceApp –DatabaseName SettingsServiceDB

# Creates a proxy for the Subscription Settings service application.

$proxySubSvc = New-SPSubscriptionSettingsServiceApplicationProxy –ServiceApplication $appSubSvc

Configure App Domain

Set-spappdomain -appdomain "mydomainapps.com"

Set-spappSiteSubscriptionName -Name "app"

Configure the app URLs to use

From your VS.net 2012 create a new sample App

Installing Development Tools SharePoint 2013

2013-07-18T14:24:00.001-07:00

This blog is part of Series : Comprehensive SharePoint 2013 Development Environment Installation and Configuration

Overview

This blog article discusses how to install VS.Net 2012 and other development tools necessary.

Development Environment Installation

From the SP2013WFE server login as SPSetup account.

Mount VS.Net 2012 install media (What ever latest you at such as With SP1 is ok too) and run the install.

Install Web Platform to get access to SharePoint 2013 development templates

http://www.microsoft.com/web/downloads/platform.aspx

Microsoft SQL Server Data Tools - Business Intelligence for Visual Studio 2012

Install SQL Server Data tools to get the BI development templates

From <http://www.microsoft.com/en-us/download/confirmation.aspx?id=36843>

Installing Office Web Apps 2013 for SharePoint 2013

2013-07-18T14:21:00.001-07:00

This blog is part of Series : Comprehensive SharePoint 2013 Development Environment Installation and Configuration

Overview

This blog article discusses how to install Office Web Apps 2013 on a dedicated server and then integrate with an existing SharePoint 2013 Farm.

Assuming you had already established a SharePoint 2013 Farm and have instantiated a Windows 2012 Server instance dedicated for installing Office Web Apps 2013 Server based on earlier blog.

Reference http://technet.microsoft.com/en-us/library/jj219455.aspx

Steps

Office Web App Server Pre Requisites Installation On Windows 2012

Run below command script to install and pre-requisites.

Add-WindowsFeature Web-Server,Web-Mgmt-Tools,Web-Mgmt-Console,Web-WebServer,Web-Common-Http,Web-Default-Doc,Web-Static-Content,Web-Performance,Web-Stat-Compression,Web-Dyn-Compression,Web-Security,Web-Filtering,Web-Windows-Auth,Web-App-Dev,Web-Net-Ext45,Web-Asp-Net45,Web-ISAPI-Ext,Web-ISAPI-Filter,Web-Includes,InkandHandwritingServices

Restart the server

Office Web App Server Installation

Mount Office Web Apps 2013 Install on the OWA2013 server and run the Install

Office Web App Server CU Installation

Download and install the Office Web Apps Server 2013 update KB2810007.

From <http://technet.microsoft.com/en-us/library/jj219455.aspx>

Office Web App Server Configuration

Step 1: Create OWA Farm and Configure to use HTTP and Support Editing

Reference http://technet.microsoft.com/en-us/library/jj219455.aspx

From an elevated PowerShell command prompt run below command

Test by accessing from outside your server http://owa2013/hosting/discovery

Office Web App Server Configuration on OWA2013 Server

Step 2: Configure SharePoint 2013 Farm to use the OWA Server

Reference

From an elevated PowerShell command prompt run below command

Now you should see a long list application extension registration with the WOPIZone

To confirm type and test below command to verify that the new zone is internal-http

Change the AllowOAuthOverHttp setting in SharePoint 2013 to True. Run below command

That means we will need to allow SharePoint to allow OAuth over HTTP by running below commands

Now check one more time the status of OAuth over http

Now once again check the WOPI Zone and as you can see it is still se to internal-https

For this installation test since I had used internal and http, I am going to set the WOPI Zone accordingly as below.

Test Office Web Apps with SP 2013

Browse to SharePoint site as user and not as system account (OWA does not work for system account)

Load sample Word Document

Click to open the document in OWA mode.

Installing PowerPivot for SharePoint 2013

2013-07-18T14:12:00.001-07:00

This blog is part of Series : Comprehensive SharePoint 2013 Development Environment Installation and Configuration

Overview

This blog article discusses how to install PowerPivot for SharePoint 2013, assuming you have followed prior steps in this series of blogs to establish SQL Server and SharePoint 2013 Farm. This blog is now updated for SQL Server 2012 SP1.

Reference

Install SQL Server BI Features with SharePoint 2013 (SQL Server 2012 SP1) (http://technet.microsoft.com/en-us/library/jj218795.aspx)

Install Analysis Services Server in SharePoint Mode for SharePoint 2013 (SQL Server 2012 SP1) (http://technet.microsoft.com/en-us/library/d3310562-82c1-454f-9c48-33a241749238)

Steps

Ensure Excel Service Account access to PowerPivot Analysis Services

From the SQL Server Management tool connect to PowerPivot mode Analysis Services instance>Properties>Security>Add your Excel Services Account

Update Excel Services with PowerPivot Analysis Services

From the Central Administration>Manage Service Applications>Your Excel Service Application>Data Model Settings>Add Server

Here find out the Analysis Services installed in the PowerPivot mode in the step 7.SQL Server 2012 and BI Components Installation and Configuration DC\POWERPIVOT

Verify your Excel Services integration with PowerPivot Analysis Services

If you already have an advanced workbook with slicers or filters, you can upload it to your SharePoint document library and verify you are able to interact with the slicers and filters from the document library view.

Install PowerPivot

Run spPowerPivot.msi to install the PowerPivot for SharePoint 2013 Configuration tool, SharePoint solution files, and data providers.

Log on to SP2013WFE server as SPSetup , load the SQL Server 2012 with SP1 Install media.

Locate the SPPowerPivot.msi under the following folder:

Run the MSI

Configure PowerPivot for SharePoint

On the server where PowerPivot to run (APP1) locate the PowerPivot for SharePoint 2013 Configuration tool

Run the PowerPivot for SharePoint 2013 Configuration tool to deploy and configure the PowerPivot features in SharePoint.

From the SQL 2012 and BI Components install we had already installed an Analysis Services in PowerPivot mode. We need this instance name next:

Click on Validate

Click on Run

At this stage go ahead and install the SPPowerPivot.msi on all other servers (SP2013APP1 and SP2013APP2) but no need to configure, followed by

IISReset on all servers.

Configuring Security for SSRS in SharePoint Mode (Part II)

2013-07-18T12:40:00.001-07:00

This blog is part of Series : Comprehensive SharePoint 2013 Development Environment Installation and Configuration

Overview

This blog article discusses how to configure security for SSRS in SharePoint mode install and configure SSRS Service Application in SharePoint Mode (Which is this Part II) assuming you have followed the previous blog on how to Install and Configure SSRS in SharePoint Mode.

There is more to it when it comes to actually being to deploy and run the SSRS reports under the SharePoint hood that is relating to Security. Since the default security provide for SharePoint 2013 Web Applications is Windows provider for Claims. When an SSRS Reports needs to run on the SharePoint page, we have various options to configure the underlying SSRS Data Source(s) and security. In either case the authentication from SharePoint Farm (SSRS Service within the SP Farm) to the underlying data source will need to happen. Since the account in question is maintained as Claim within the context of SharePoint Farm, while the SSRS data source(s) within the Windows network Server recognize Windows account principle, we will need to convert our Claim account to Windows account and send it across the server (Double hop) ( I think you get the gist and a hint at Kerberos!) .

We will need to ensure Claims to Windows Token Service (C2WTS) is running on SharePoint servers.

Then we will need to configure Constrained Delegation on the C2WTS account.

Switch Reporting Service application to Kerberos

Switch our Web Application where the SSRS reports will be deployed to Kerberos

Configure necessary SPNs

My References:

http://technet.microsoft.com/en-us/video/Video/hh858469

http://blogs.technet.com/b/sharepoint_-_inside_the_lines/archive/2013/05/28/sharepoint-2013-with-ssrs-2012-and-constrained-delegation.aspx

Steps

Create or ensure following Service Accounts (Described under this blog “Service Accounts Preparation ”)

Register new Service accounts as Managed Service account within your SharePoint Farm

Provision Service account for SSRS Service

May be already done if followed the “Installing Reporting Services In SharePoint Mode (Part I) “. From CA, Security>General Security>Configure service accounts

Provision Service account for Claims to Windows Token Service on all SharePoint Servers

C2WTS Service

Validate on the Windows Services on all SharePoint servers under the Services Console that now the Lon On As account is changed from Local System to our domain account

SSRS Security

On the APP1 Server (Where SSRS Service is running) update Authentication from Windows to Negotiate (Kerberos). Access the Report Server Configuration file by following below path:

Before

After

Get the SSRS Service account Access to the Web Applications.

Example for Portal Web App, from any SharePoint server, run SP PowerShell as administrator

On the Server where PerformancePoint will run the check for C2WTS Windows Service account

Ensure that C2WTS is in local admin

Configure Local Security Policy

Check Local Security Policy on the APP1 to ensure your CORP\SPC2WTS account is added:

Local Policy>User Rights Assignment

Act as part of OS (SPC2WTSService)
Impersonate a client after authentication
Logon as service

Restart Windows Service C2WTS

Add the WSS_WPG Windows Group to the c2wtshost.exe.config file. The file is located \Program Files\Windows Identity Foundation\v3.5\c2wtshost.exe.config

From your Local Groups

IIS Reset

Ensure SQL Server for Kerberos Delegation

Setspn -S HTTP/portal corp\SPPortal
Setspn -S HTTP/portal.mydomain.com corp\SPPortal
Setspn -S MSSQLSvc/DC:1433 corp\sqlservice
Setspn -S MSSQLSvc/DC.corp.mydomain.com:1433 corp\sqlservice
Setspn -S SP/C2WTS corp\SPC2WTS
Setspn -S SP/SSRS corp\SPSSRS
Setspn -S SP/PPS corp\SPPerfPointUser
Setspn -S SP/Excel corp\SPExcelUser

Run ADUC to, ensure Advanced Feature

Check SQLService account Attribute for SPN

Configure Delegation for SPC2WTS

From the ADUC, access SPC2WTS account and switch to Delegation

Select SQL Server service with Port 1433 or whatever you have designated for your SQL Server.

Let's switch the Web Application Authentication to Kerberos

From CA>Manage Web Applications> Select your web app and chose Authentication providers

This completes the configuration. In the next blog lets create an example SSRS, Deploy to SharePoint and test.

Test Connection: Coming soon…

Installing Reporting Services In SharePoint Mode (Part I)

2013-07-18T12:08:00.001-07:00

This blog is part of Series : Comprehensive SharePoint 2013 Development Environment Installation and Configuration

Overview

This blog article discusses how to install and configure SSRS Service Application in SharePoint Mode (Which is this Part I) but there is more to it when it comes to actually being to deploy and run the SSRS reports under the SharePoint hood that is relating to Security. The Part II covers these aspects.

Steps

Install Reporting Services In SharePoint Mode

Reference http://technet.microsoft.com/en-us/library/jj219068.aspx

From SQL media install Reporting Services in SharePoint Mode and RS Add in for SharePoint

Important: What and where needs to be installed and configured

	Install RS in SP Mode	Install RS Add In	Install SQL 2012 SP1	KB2793634	Run Install-SPRSxx command	Start SSRS Service
SP2013WFE	No	Yes	Yes	No	Yes	No
SP2013APP1 (Hosts CA) (Where SSRS will run)	Yes	Yes	Yes	Yes	Yes	Yes
SP2013APP2 (All Other servers)	No	Yes	Yes	No	No	No

On the APP1 Server

Mount SQL 2012 Enterprise (Developer) with SP1 Media (Remember you will need minimum with SP1)

Start Setup, Switch to Installation from the left.

Install Reporting Services in SharePoint mode and the Reporting Services add-in for SharePoint products.

Download and install SP1 http://www.microsoft.com/en-us/download/details.aspx?id=35575

Download and install Non Security Update for SQL Server 2012 SP1 (KB2793634) From http://www.microsoft.com/en-us/download/details.aspx?id=36215

Reboot Server

On the WFE Server

Follow the above chart at the beginning to install on all other servers. Ensure to install as necessary components.

On SP2013WFE login as SPSetup and install following from SQL Media

Reboot Server

On SP2013APP2

Reboot

Create and Configure SSRS Service on APP1

Browse to CA with SPSetup

Ensure to choose your APP1 Server and View scope is All

Create Reporting Services service application.

Click on Provision Subscripts and Alerts

Click on Download script , save script and run in your SQL Server

Back in to the Provision Subscripts and Alerts settings, I have entered CORP\Administrator since it was sysadmin on the SQL Server.

Click ok

Final Services State

Installing Latest CUs as of July 2013

2013-07-18T10:40:00.001-07:00

This blog is part of Series : Comprehensive SharePoint 2013 Development Environment Installation and Configuration

Overview

This blog article discusses how to apply latest updates to SharePoint 2013.

As of July, you will need following updates to be applied

June 2013 Server CU, Build Update 15.0.4517.1005
July 9 2013 Server CU , No Build Update

Refer to the latest updates, KB reference and Download information on the SharePoint 2013 Update TechNet page,

Alternatively refer to this address for clear listing as well http://www.marc-lognoul.me/itblog-en/page/SharePoint-SharePoint-2013-Build-Numbers.aspx

Steps overview

Apply June CU on all servers one at time
Apply July 9 Update on all servers one at time
Run Configuration Wizard on all in the order to finish the installation
Ensure the Farm Version matches the update build version in this case for June 2013 CU the expected build version is 15.0.4517.1005

Steps

I will update this post when I had chance to perform these steps…

Install April 2013 CU

2013-07-18T10:37:00.001-07:00

This blog is part of Series : Comprehensive SharePoint 2013 Development Environment Installation and Configuration

Overview

This blog article discusses how to apply April 2013 CU to your SharePoint 2013 Farm

Steps

Apply April CU on all servers one at time

Follow to install the April CU on all other servers in the farm

Run Configuration Wizard on all in the order to finish the installation

Ensure Farm Version

Configure Search on Dedicated APP Server

2013-07-18T09:45:00.001-07:00

This blog is part of Series : Comprehensive SharePoint 2013 Development Environment Installation and Configuration

Overview

This blog article discusses how to configure Search Service in SP 2013 such that each of the new sub components of the Search Service can be allocated to run on different APP(s) server.

Based on our initial design we want to run all the Search Service components on SP2013APP2 server.

Steps

Unlike prior SharePoint version, in SharePoint 2013 the Search Service Application topology must be configured via PowerShell only. Hence I have prepared the following below PowerShell script by referencing this blog.

Substitute following below for your specific environment:

ssaName: Your Search Service Application Name

dbSrv: Your Database name where the Search Service Application related databases will be created

dbNamePrefix: Such as Dev_ or QA_ or Prod etc. if you desire.

appPoolName : Search Service Application Pool Name

appPoolAccount : Search Service Application Pool Account

servers : Here you should substitute your Search server name.

See below the script for further instructions:

#---------------------
# Component Type Key:
#---------------------
#   Admin: Admin Component
#   CC: Crawl Component
#   CP: Content Processor
#   AP: Analytics Component
#   IP: Index Partition Component
#   QP: Query Processor
#------------------------------------

$ssaConfig = @{
    ssaName = "Search Service Application";
    dbSrv = "DC";
    dbNamePrefix = "";
    appPoolName = "SharePoint Search Application Pool";  # if exists, will just use the existing pool
    appPoolAccount = "CORP\SPSearchService";
    servers = @{ 
      "SP2013APP2"    = @{ components = @{  Admin=$true; CC=$true; CP=$true; AP=$true;IP=$true; QP=$true; };  
                            index =      @{ Partition=0; # The index partition number
                                            Path="C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\"   
                                        # Specified path must exist and be empty
                                     }; 
                     };
      
    }
};



$err = $null
#-----------------------
# Create the App Pool(s)
#-----------------------

# Create managed account if it does not exist
if ((Get-SPManagedAccount $ssaConfig.appPoolAccount -ErrorVariable err -ErrorAction SilentlyContinue) -eq $null) {
  "****************************************************** "
  "* Create managed account for " + $ssaConfig.appPoolAccount
  "****************************************************** "
  $credential = Get-Credential
  if ($credential -eq $null) { "Terminating script..."; start-sleep 10; exit; }
  New-SPManagedAccount -Credential $credential 
}

$sqssPool = Get-SPServiceApplicationPool | where {$_.Name -eq $ssaConfig.appPoolName }
if ($sqssPool -ne $null) { 
    "-- Search App Pool [" + $ssaConfig.appPoolName + "] already exists... <skipping>" 
} else { $sqssPool = New-SPServiceApplicationPool -name $ssaConfig.appPoolName -account $ssaConfig.appPoolAccount }

if ($ssaConfig.adminPoolName -ne $null) {
    $adminPool = Get-SPServiceApplicationPool | where {$_.Name -eq $ssaConfig.adminPoolName }
    if ($adminPool -ne $null) { 
        "-- Search Admin App Pool [" + $ssaConfig.adminPoolName + "] already exists... <skipping>" 
    } else { $adminPool = New-SPServiceApplicationPool -name $ssaConfig.adminPoolName -account $ssaConfig.adminPoolAccount }
} else {
    $adminPool = $sqssPool;
}

$ssaConfig.Add("sqssPool", $sqssPool)
$ssaConfig.Add("adminPool", $adminPool)

#---------------------------------------------------------------------------
# start the Search Instances (if not already online) on the relevant servers
#---------------------------------------------------------------------------
"Starting Search Instances..."
foreach ($netbios in $ssaConfig.servers.keys) {
    "   [" + $netbios + "]"
    if ((Get-SPServer $netbios -ErrorVariable err -ErrorAction SilentlyContinue) -eq $null) {
        "The server" + $netbios + "is not found in this farm..." 
        "Terminating script..."; start-sleep 10; exit; 
    } else {
        if ((Get-SPEnterpriseSearchServiceInstance -identity $netbios).status -ne "Online") {
            Start-SPEnterpriseSearchServiceInstance -identity $netbios
        }
    }
    $ssaConfig.servers[$netbios].instance = Get-SPEnterpriseSearchServiceInstance -identity $netbios
}

#------------------------------------------------------------------------------------------------
# Create the new SSA (which may take a while (~10min) to run...
#------------------------------------------------------------------------------------------------
"Creating New SSA"
"   - My tests have taken ~10 min to complete this step"
"   - Starting at: " + (Get-Date)

if ($ssaConfig.dbNamePrefix -ne $null) { 
    $dbPrefix = $ssaConfig.dbNamePrefix 
} else { $dbPrefix = $ssaConfig.ssaName }

$SSA = New-SPEnterpriseSearchServiceApplication -name $ssaConfig.ssaName -ApplicationPool $ssaConfig.sqssPool -AdminApplicationPool $ssaConfig.adminPool -DatabaseServer $ssaConfig.dbSrv -DatabaseName $dbPrefix
New-SPEnterpriseSearchServiceApplicationProxy -name $ssaConfig.ssaName -SearchApplication $SSA

"Cloning the topology..."
$initialTopo = $SSA | Get-SPEnterpriseSearchTopology -Active
$newTopo = $SSA | New-SPEnterpriseSearchTopology 

#------------------------------------------
# Create the new components on each server
#------------------------------------------
"Creating Components on specified server(s)... "
foreach ($netbios in $ssaConfig.servers.keys) {
    foreach ($type in $ssaConfig.servers[$netbios].components.keys) {
        if ($ssaConfig.servers[$netbios].components[$type]) {    
            "   [" + $netbios + "] --> " + $type
            switch ($type) {
                "Admin" {
                    New-SPEnterpriseSearchAdminComponent -SearchTopology $newTopo -SearchServiceInstance $ssaConfig.servers[$netbios].instance
                    $SSA | Get-SPEnterpriseSearchAdministrationComponent | Set-SPEnterpriseSearchAdministrationComponent -SearchServiceInstance $ssaConfig.servers[$netbios].instance
                    $SSA | Get-SPEnterpriseSearchAdministrationComponent
                }
                "CC" { New-SPEnterpriseSearchCrawlComponent -SearchTopology $newTopo -SearchServiceInstance $ssaConfig.servers[$netbios].instance }
                "CP" { New-SPEnterpriseSearchContentProcessingComponent -SearchTopology $newTopo -SearchServiceInstance $ssaConfig.servers[$netbios].instance }
                "AP" { New-SPEnterpriseSearchAnalyticsProcessingComponent -SearchTopology $newTopo -SearchServiceInstance $ssaConfig.servers[$netbios].instance }
                "QP" { New-SPEnterpriseSearchQueryProcessingComponent -SearchTopology $newTopo -SearchServiceInstance $ssaConfig.servers[$netbios].instance }
                "IP" { New-SPEnterpriseSearchIndexComponent -SearchTopology $newTopo -SearchServiceInstance $ssaConfig.servers[$netbios].instance -IndexPartition $ssaConfig.servers[$netbios].index.Partition -RootDirectory $ssaConfig.servers[$netbios].index.Path }
            }
        }
    }
}

"Activating the new topology:"
 "   - This also may run for a few minutes..."
$newTopo.Activate()
 
"Cleaning up the inactive topology..."
while ($newTopo.State -ne "Active") { "."; start-sleep 10; } 
$SSA | Get-SPEnterpriseSearchTopology | Where {$_.State -eq "Inactive" } | Remove-SPEnterpriseSearchTopology -Confirm:$false

Ensure that the Search Index path (C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\) on SP2013APP1 and SP2013APP2 are empty

Copy and prepare the powershell script file and save at a location locally

Run the SP 2013 PowerShell command as Administrator

Run the Saved Script

Validate in the CA that the new Search Service Application is started

Click on the Search Service Application to get in to the Search Service Administration page as below.

Click on the link against the Global Search Center URL and set the Search Center that we had previously created:

Update the Search Content Access Account from SPSearch to SPCrawl

At the end now you can see that only APP2 server is running the Search Service