Jeremy Pyne

Adjusting for X/Y Skew in your Slicer

2018-11-01T17:06:00.003-04:00

I have had a minor annoyance with my 3D printer that it skews the print in the X/Y place and will print a cube as a bit of a parallelogram instead. This is not that the X or Y are misaligned, the sizing is consistent, there is just a skew. After some research I found a solution in the slicer instead of manual tweaking the frame or flashing the firmware.

To be sure I could have flashed the firmware with a recent Marlin firmware, but that required extensive knowledge of the parts and configuration values, testing, and in my case an Arduino as well witch I don't have. I tried messing with the frame and rods a bit but couldn't find any noticeable defects and everything mounted quite securely without much adjustability.

Then I stumbled upon this Cura plugin that clams to make the corrections in a Post Processing script. I was a bit skeptical but I tried it out and two tests latter I had it printing perfect cubes.

Determine if you have a skew:

Print this calibration model. You can safely print it at high speed with .2mm layers and still get good adjustment values. I also scaled the Z down to speed it up, but print the full 100mm X/Y.

When done first measure the X at multiple points and the Y at multiple points to ensure that all measurements are 100mm and you don't have stepper, rod alignment, or other issues. Those issues are not covered here but would need to be corrected first.

Finally measure the diagonal form A to C and compare it to B to D. Record the numbers and decide if you need skew correction. In my case on a 100mm test by AC and DB were off by around 1.8mm which was noticeable on compartments and trays I printed for board games.

To correct the skew:

First download the SkewCompensation.py script for Cura from this project by joochung. You will need to install it in Cura's scripts folder. Open Cura and go to Help -> Show Configuration Folder and drop it in the scripts folder there. You will then need to restart Cura once.

Now calculate the skew angle but using the Excel file that is included on the project. The formula is rather complex so just open the 3D_Printer_Axis_Calibration.xlsx file and fill in your AC and BD measurements in cells C13 and C14. Cell C18 should then have a decimal result less then .05. Record this number as it's your Skew Adjustment and what you will enter in Cura. If your skew is more then that I'd look into physical issues and adjustments.

Now, back in Cura click on Extensions -> Post Processing -> Modify G-Code. In the dialog select Add a script -> Skew Compensation. On the right side change XY Type to Tangent and enter the skew adjustment you calculated in the XY Tangent field.

Close this dialog and print the calibration model again and re-measure. If you swap the AC and BD measurements that the box will be more skewed not less. I was able to get a perfect box on the first try.

Monoprice Select V2 3D Printer Customizations

2018-05-02T15:15:00.000-04:00

I have a Monoprice Select V2 3D printer. This is just a few notes and recommendations for the modifications and part's I have customized it with to get improved print speed, reliability, and quality.

Printer: Monoprice Select 3D Printer v2.1

One of the best value printers in the price range. The all metal frame makes a huge difference for print quality.

Filament: FoxSmart

Very good and inexpensive PLA filament. I have tried others twice the cost and still feel this is much better.

Replacement Y Carriage: RepRap Champion Upgrade

The y Carriage that the print bed rides on is to flimsy and flexes. This means you will hav issues keeping the build plate level and have to re-level it all the time. I use this replacement and never have to re-level any more unless I move the printer.

Z-Brace Frame Support Kit: Gulf Coast Robotics

This is another critical upgrade as it removed all the wobble in the Z tower from the print bed moving in and out along the Y axis.

Glass Print Bed: Geeetech Borosilicate Glass

Another important upgrade. I really liked the Buildtek print surface that the printer came with, but had many issues with it not being level, having bows and crowns, and flexing when the bed was heated. Using a pice of gas on top of it solves basically all these problems. You can also use regular window glass, but this is more durable and won't break under the high temperatures.

Clips for attaching glass to print bed.

These are the best option as they don't apply any downward pressure yet still keep the glass from wiggling around. You may need to use small pieces of paper as shims. Also they won't last for ever, so print a set or two.

Sewing Machine Oil

Best choies for oiling parts as it won't damage them.

Spool Holder

Some spools may not fit on the provided one.

Improved Fan Duct

Another must have. This proves the airflow from the cooling fan to more evenly cool from all sides. This will remove defects and issues that would appear on the back side due to insufficient cooling while printing.
You need the DiiiCooler_Rev3-21-V2-1_Beta3.STL file and the DiiiCooler_Adapter_30mm.stl file.

Creating FreeBSD images in Google Compute

2018-01-08T13:16:00.001-05:00

I found various instructions to create a FreeBSD image in Google Compute but non of those instructions where complete of functional. Here are the correct instructions.

Install the cloud SDK

There are instructions here.
I use OS X and Homebrew so I just ran the following instead.

brew tap caskroom/cask
brew cask install google-cloud-sdk

Then run the following commands to setup and log in.

gcloud init
gcloud auth list

List the versions available

Run this to list all the releases available You can delete the last grep to see betas and release candidates as well.

gcloud compute images list --project freebsd-org-cloud-dev --no-standard-images | grep -i freebsd- | grep -release

Now to create a new instance

Update the following command as needed and run it.
gcloud compute instances create "InstanceName" --machine-type "g1-small" --network "default" --subnet "default" --maintenance-policy "MIGRATE" --image "freebsd-11-1-release-amd64" --image-project=freebsd-org-cloud-dev --boot-disk-size "30" --boot-disk-type "pd-standard"

InstanceName should be changed to a useful host name.
--machine-type can just be g1-small for now, or another option from tis list.

gcloud compute machine-types list

gcloud compute machine-types lis t
--network and --subnet will probably need to be changed to your custom network. You can list the available options with these commands.

gcloud compute networks list

gcloud compute networks subnets list

--image can be changed another version from the earlier list.
--boot-disk-size should be at least 30GB. You can add additional drives later.
--boot-disk-type can be pd-standard or pd-ssd depending if you want SSD or not.

This should create and power on the VM and you will now be able to see it in the website and make adjustments to the configuration as needed.

To gain SSH access to this server you will need to do the following steps.

Ensure you can ping the servers internal or external IP address.
Optionally go into the Google Cloud -> VPC network -> Firewall rules and add a rule for tcp:22 traffic route to the network or server.
Create a public key and load it onto the server. This step is required and was one of my major hurdles.

Check that you have a public key on the computer to authenticate

cat ~/.ssh/id_rsa.pub

If you don't have this file its empty runt he following to create one.

ssh-keygen -t rsa -f ~/.ssh/id_rsa

Add this public key to the instances you can access it.

gcloud compute instances add-metadata InstanceName --metadata-from-file sshKeys=~/.ssh/id_rsa.pub

Or you can add it to the entire project.

gcloud compute project-info add-metadata --metadata-from-file sshKeys=~/.ssh/id_rsa.pub

Now you will be able to login with the following.

gcloud compute ssh freebsd-recovery

If you get errors like Operation timed out they would indicate issues with your networking, firewall rules, or ip address configuration. If you get a Permission denied (publickey) error instead it means the public key was not loaded properly form the ~/.ssh/id_rsa file. You can also open the file in a text editor and manually enter it in the VM instance under Custom metadata with a key of sshKeys.

Note that this authentication is per device, but once you authenticate you can enable password login the ssh config and add other users if you want.

Shrink PDF Files in OS X and Control Image Quality

2017-11-08T11:54:00.002-05:00

This guide is using information from here and providing clearer instructions and a Automator script to compress many at once.

Go here and click the green Clone or download button, then select Download ZIP.
Go to Downloads on your computer and rename the downloaded folder to Filters.
Make a new Finder window (Command+n) and then hold Option while clicking Go -> Library in the menu.
Drag the Filters folder with he files you downloaded into this new Library folder.
Download this program to your Applications folder or Desktop.
Now you can just all your PDF's files at once and drop them onto the application and it will compress them and save the compressed versions to your desktop.

Connecting to MySQL Server from a Microsoft SQL Server

2017-08-17T17:04:00.000-04:00

This seams to me that connecting MySQL and Microsoft SQL databases is something that should be simple to do. Unfortunately looking around online there was no real clear directions or even indications that this was possible without buying some additional drivers. While fear not because this does work out of the box without needing to buy any additional software on pretty much any versions of the two databases.

In short what we will do is add a MySQL ODBC driver, configure a System DSN using that driver, Link MsSQL to that DSN connection, and to do a few test queries and updates. This will allow full insert, update, and delete commands.

You will need administrative access to the Microsoft SQL Server along with access to the database. In addition you will need to know the connection information for the MySQL database, it's IP Address, a list of the databases you want to connect, and any MySQL login's and passwords you will be using.

Setup

First off you will need to identify if you have the 32-bit or 64-bit version of Microsoft SQL Server. You can determine this by connecting to the server in Management Studio, Right Clicking on the connected server, selecting Properties, and checking the Product field.

Once you have determined the proper edition go here and download the matching format of the MySQL ODBC driver. You do not need to create and account as there is a link to skip that step. Be sure to install this package.

ODBC Configuration

We need to create one or more ODBC connections using this new driver called DSN's. Navigate to the Administrative Tools folder in your start menu or control panel and launch the matching 32-bit or 64-bit version of ODBC Data Source.

As for how to set up the connections you have a few options. You can create individual DSN's for each database using separate user logins for each, you can create one DSN with permission to all the tables, or you can create one DSN and map MsSQL users to MySQL users and control what database's each can see based on that mapping. Decide on this and then adjust the Server Name and Database Name fields below as you configure things.

If you are using one login for everything or mapping users across then do the following.

Open the ODBC Data Source Administration tool.
Go to the System DSN tab.
Click Add to create a new ODBC connection.
Select MySQL ODBC 5.3 ANSI Driver or MySQL ODBC 5.3 Unicode Driver as your needs dictate. Unicode supports a larger character set.
Click Finish.
On the new screen fill in the following fields:

Data Source Name: ServerName
TCP/IP: MySQL Server IP Address
User/Password: MySQL Admin username/password.
Database: Leave this blank.

Click Test to make sure it is connection and then press Ok to save.

Or if you are creating separate ODBC connections for each database or only have one database but use one login per database then do this instead for each.

Open the ODBC Data Source Administration tool.
Go to the System DSN tab.
Click Add to create a new ODBC connection.
Select MySQL ODBC 5.3 ANSI Driver or MySQL ODBC 5.3 Unicode Driver as your needs dictate. Unicode supports a larger character set.
Click Finish.
On the new screen fill in the following fields:

Data Source Name: ServerName/DatabaseName
TCP/IP: MySQL Server IP Address
User/Password: MySQL login/password with access to the specific database.
Database: Select the proper database.

Click Test to make sure it is connection and then press Ok to save.

Microsoft SQL Server Configuration

Now for the final step open up the Microsoft SQL Server Management Studio and connect to the database server. Agains he steps will vary depending if you have one single connection are are making connections for each database in question. If you are using one login for everything to grand full access to any MsSQL users to all MySQL data do the following.

Navigate to Server Objects -> Linked Servers then Right Click -> New Linked Server.
Fill in the following on the General Tab:

Linked Server: ServerName
Provider: Microsoft OLE DB Provider for ODBC Drivers
Data Source: ServerName

On the Security Tab:

Select Be made using the security context.
Reneter the same MySQL Admin username/password used above.

Click Ok.

If there are any issues select No and make adjustments. Once you click Yes you can't edit the General settings and have to remove and re-add the connection.

On the other hand if you want to map MsSQL Users to MySQL users so they have access to different tables based on the login then so the following.

Navigate to Server Objects -> Linked Servers then Right Click -> New Linked Server.
Fill in the following on the General Tab:

Linked Server: ServerName
Provider: Microsoft OLE DB Provider for ODBC Drivers
Data Source: ServerName

On the Security Tab:

Add a row for each user that should be mapped and select the MySQL Local Login Name.
Enter the matching MySQL usernames and passwords or shared account details that each login should map to.
Leave Impersonate unchecked.
You can change the radio to Not be made so that unmapped users don't see anything or specify a general login for unmapped users.

Click Ok.

If there are any issues select No and make adjustments. Once you click Yes you can't edit the General settings and have to remove and re-add the connection.

Or if you create per database DSN's then to the this for each.

Navigate to Server Objects -> Linked Servers then Right Click -> New Linked Server.
Fill in the following on the General Tab:

Linked Server: ServerName/DatabaseName
Provider: Microsoft OLE DB Provider for ODBC Drivers
Data Source: ServerName/DatabaseName

On the Security Tab:

Select Be made using the security context.
Reneter the table specific MySQL username/password for this table.

Click Ok.

If there are any issues select No and make adjustments. Once you click Yes you can't edit the General settings and have to remove and re-add the connection.

Querying and Updating Data

Now that this is connected you can browse the structure in Management Studio by going back to Server Objects -> Linked Servers -> Server Name. To test some sample operations lets assume we have the following table in MySQL.

test01

id, int, pk, auto_increment
Name, nvarchar(20) not null
Items, smallint, not null

Then we can do all of the following:

SELECT count(*) as 'Count'
FROM OPENQUERY([ServerName/DatabaseName], 'SELECT * FROM test01')
GO

INSERT OPENQUERY([ServerName/DatabaseName], 'SELECT Name, Items FROM test01 ')
VALUES
('Shirts', 47),
('Pants', 55),
('Shorts', 2),
('Shoes', 43)
GO

SELECT *
FROM OPENQUERY([ServerName/DatabaseName], 'SELECT * FROM test01')
GO

UPDATE OPENQUERY([ServerName/DatabaseName], 'SELECT * FROM test01 WHERE Name = ''Shoes'' ')
SET Items = 55
GO

SELECT *
FROM OPENQUERY([ServerName/DatabaseName], 'SELECT * FROM test01  WHERE Name = ''Shoes''')
GO

DELETE OPENQUERY ([ServerName/DatabaseName], 'SELECT id FROM test01 WHERE Items > 50')
GO

SELECT *
FROM OPENQUERY([ServerName/DatabaseName], 'SELECT * FROM test01')
GO

Netflix to Trakt.tv Sync

2017-07-21T14:31:00.003-04:00

Update Details Below. Note that if the dialog shows an older version you may need to recreate your bookmarklet.

Also note that this will probably only work in English, though you may be able to temporally change your language on Netflix to get the sync to work.

There used to be a nice Chrome plugin to synchronize by Netflix history with Trakt.tv but it had been unreliable and doesn't seam to be working now. I wrote this code to solve my own need but it does the same thing and I am posting it here for others to use.

Note that this is not 100% perfect as some show names aren't exactly the same and some Netflix show's don't use episode names at all. This code it not using any Netflix API calls but rather just scraping your Viewing Activity page and then making API calls to your Trakt.tv account to post the item's it finds.

You may need to manually select some episodes or shows on the Tract.tv side if they aren't matched and it is possible that duplicates could get synced if the names cause issues though I tried to prevent this. In short I'm not out to make this 100% perfect but it should get you 90% of the way and it shows all the changes it will post for confirmation before posting.

Usage

Drag this link, Sync to Trakt.tv, to your bookmark bar or bookmerks menu.

This is a bit of Javascript that will inject the Sync tool into your Netflix activity page.

Open Netflix in your browser and log in.

This is tested and working in Safari and Chrome.

Go here https://www.netflix.com/viewingactivity to see your history or click on your name in the top right, then on Account, and finally on Viewing Activity.
Netflix will not load your entire history so you will need to scroll down to get it to load more. Keep doing this until all the history you want to sync has been loaded.

You don't have to do everything, if you are doing this regularly then the first screen or two will be sufficient.

Now for the magic click the bookmarklet you added while looking at the history page.

Note: The first tim you do this you will need to log in to Trakt.tv and click the bookmark again.

The script will scrape the Netflix page, they check each item in Trakt.tv and build a list of already synced items, items not found, and items scheduled to sync.

If re-running the tool past plays will be greyed out, new plays to by synced will be green, items that could not be cached will remain white, and duplicate plays that are being ignored will be orange.

When you are satisfied just click Sync Now to post the changes. You can then check Trakt.tv or reload the page and run the script again to see the results.
You may have issues the first time the history is pulled from Trakt.tv. In some cases you may need to load the page and run the script a few times before it successfully reads the history.
If you have and hard locks or shows that won't identify properly please leave me a note with as much detail about the problem items as possible. You can get this by adding #debug to the URL, reloading the page, and running the script again. If you are able to identify the offending items i can make corrections. You can contact me at jeremy.pyne at gmail.

How It Works

This solution is a simple bit of javascript that scans your Netflix history page and sync's plays over to Trakt.tv. It also needs to read your list of past plays in Trakt.tv along with the history it reads from the Netflix page.

None of this data is saved or sent to any other servers in any way. As the script runs a listing of all the changes that will be posted to Trakt.tv is show with a confirmation option. No passwords or logins are prompted or recorded in any way. You are logging in yourself to Netflix and Trakt.tv, though you do have to grant this application API access to your account. The only bit that is saved is the Trakt.tv access token so that you don't have to log in on every use of the tool. You can even remove this by clearing the cookies for the www.netflix.com domain.

You can view the entire script here first if you wish. The bookmarklet code is an easy way to inject this code into your netflix page, you could also run this code manually in your browser console.

function load(filename, type){
 if(type == 'js'){
  var fileref=document.createElement('script');
  fileref.setAttribute('type','text/javascript');
  fileref.setAttribute('src',filename);
 }
 else if (type == 'css'){
  var fileref=document.createElement('link');
  fileref.setAttribute('rel','stylesheet');
  fileref.setAttribute('type','text/css');
  fileref.setAttribute('href',filename);
 }
 document.getElementsByTagName('head')[0].appendChild(fileref);
}
load('https://www.inkonit.com/netflix/netflix-sync.js?v1.6.5','js');
load('https://www.inkonit.com/netflix/netflix-sync.css?v1.6.5','css');

Alternate Date Format

Your milage in other language may vary but in short if the show and episode titles still match properly between Netflix and Trackt it should still work. The biggest issue is with different date format. Trakt seams to be happy to accept everything in en-US format but Netflix changes formats as you change languages. I added a flag that will tell the code to read Netflix dates in 'dd/MM/yyyy' format witch should help. To make this logic execute run the following after changing the language in Netflix. You will need to go to the Netflix site, open up the console in your browser, and paste and execute this line of code. Then refresh and run the tool again.

document.cookie = "date_format=dd/MM/yyyy; expires=" + new Date(Date.now() + 1000 * 60 * 60 * 24 * 365).toUTCString()

Working On/Planned

Ability to manually re-query a episode/movie with a custom name.
Ability to provide episode numbers ~~or auto-detect them based on episode watch order~~.
Ability to tie track items to unknown items so they don't sync duplicates.
Ability to post mapping corrections back to the server so the corrections persist.

Updates

Updated 8/22/2017, v1.1

Added version number to script.
Fixes for many situations where a show, movie, or episode not being found would cause the entire script to never complete processing.
Manual fixes added for: MSK3k The Return, Marvel's The Defenders, items using Series and Collection nomenclature, items with (U.S) suffix.
Removed unneeded jquery-ui script.

Updated 8/23/2017, v1.2

Added error detection for cases where episode titles can't be determined.
Added checkbox option to prevent specific items form syncing in odd cases where they match incorrectly.

Update 8/24/2017, v1.3

Added another fix hard error when show name wasnt found.
Added support for shows and movies with Unicode charecters in names.
Fix for past plays in the format Title: Seaseon ##: Episode wouldnt match properly in history.

Update 8/28/2017, v1.4

Fix a typo breaking movies detection.
Added version number to console output.

Update 9/8/2017, v1.5

Added logic to guess episode numbers in Netflix based on play order. This will not work as expected if episodes are played out or order or if only part of the season is visible in the Netflix history. This is also only done as a last chance if no name matching finds a match. If there are incorrect items you can uncheck them to prevent errors or add #no-guess to your page url before running the script to disable this option.
Fixes for shows with numbers in the title. This was originally working but some unicode fixes I made broke it.
Fixes for shows with Part # in roman numerals.
Fixes for MythBusters, Stranger Things, and Wet Hot American Summer.

Update 10/25/2017, v1.6

Major rewrite of code for maintainability and reliability.
Replaced custom process counting with more reliable promise-queue module.
UI improvements.
More logging with #debug flag.

Updated 12/14/2017, v1.6.4

Fixed Trakt authentication bug.
Fix for Stranger Things, Marvel's The Defenders.

Updated 3/5/2017, v1.6.5

Added a flag to support 'dd/MM/yyyy' date encodings when changing the language on Netflix. See Alternate Date Format above for me details.

Restricting Access at in Varnish Server Wide to Admin Areas

2017-06-19T15:12:00.001-04:00

There are lots of ways to implement access controls and additional security on your hosting. I recently implemented a very restrictive but easy to manage implementation for our FreeBSD server that runs everything through Varnish. I could have placed this restriction in Apache via .HTACCESS files or global rules but placing them in Varnish they execute earlier and have less server overhead and ensure there are no caching related flaws.

To set this up I added the following to the top of my /usr/local/etc/varnish.vcl file.

# Whitelist of IP Address's or Ranges that are allowd to access restricted administration pages.
acl admin {
    "127.0.0.1";
    "localhost";
    "10.1.0.0"/16; # Local Network Class B
    "192.168.1.0"/16; # Local Network Class C
    "1.1.1.1"; # Sample
    # INSERT IPS #
}

Next inside the vcl_recv sub, or add one if needed include this before other rules. The bold text is doing URL matching to determine what scripts or directories to require approval for. This could also be applied to specific domains but in this case it is server wide.

# Optional feature to only allow access to matched pages if client is on a whitelist.
    if (req.url ~ "^/wp-(login|admin|cron|json)" && client.ip !~ admin) {
        # Whitelist for all admin pages
        return (synth(403, "IP address not authorized, please request access from AdminEmail" ));
    }

Now you can reload varnish and do some testing by adding and removing your local network or IP and ensure its working. You can stop here but this isn't very convenient to manage or update form offsite.

sudo service varnishd reload

Add a custom script to grant access to IP's.

We have a custom script hidden our our server that it not indexed or published anywhere that our administrator can use to add new addressed on the fly. It asks for a name, location, and IP Address and will update the varnish.vcl file and reload the configuration when submitted.

You can post this script to a secret location on the web server. You may want to add a password to it for extra security and don't link to it anywhere. You will also need to run the following commands.

// Make writable by www user.
sudo chmod g+w /usr/local/etc/varnish.vcl
sudo chgrp www /usr/local/etc/varnish.vcl
// Add custom script to reload varnish.
echo "#!/usr/local/bin/bash
service varnishd reload" > ~/varnish-reload
// Move script and make it executable
sudo cp ~/varnish-reload /usr/local/bin/varnish-reload
rm ~/varnish-reload
sudo chmod +x /usr/local/bin/varnish-reload

Read XML Sitemap to a CSV File

2017-06-19T14:35:00.002-04:00

You may need to process a sitemap for redirects or get a list of all your pages for some testing or monitoring process. Here is a simple script you can run to convert your sitemap to a CSV file. It even supports multi-file sitemaps.

Simply save this script sitemap-to-csv.pl to your computer. You will need Perl to run in.

Then you can execute it from the command line with the following format:

// This will automatically append /sitemap.xml
./sitemap-to-csv.pl http://domain.com

// Or specify the exact file.
perl sitemap-to-csv.pl http://domain.com

// With SSL domain authentication override.
PERL_LWP_SSL_VERIFY_HOSTNAME=0 ./sitemap-to-csv.pl http://domain.com

Configure SSL Termination with Varnish Caching and HTTP/2

2017-02-06T16:47:00.000-05:00

To create the fastest web pages possible in a LAMP stack can be a bit tricky. This document covers configureing blazing fast HTTPS sites on Freebsd with Apache, Varnish, Nginx, PHP, Mysql and letsencrypt.sh. Of course you can swap out the PHP/Mysql parts as needed, or even use your own certificates instead of lets encrypt.sh as you need.

Updated 2/16/2017: Added code to nginx.conf for additional rewriting; added an extra check so requests don't end up with multiple Access-Control-Allow-Origin headers.
Updated 2/27/2017: Added code to apache and varnish to seamlessly minify CSS and Javascript resources.
Updated 3/1/2017: Added details about Varnish Purging, plugins, and custom code.

The Varnish software provides incredibly fast cached website hosting in front of standard web services. It does however have one major limitation that it won’t serve HTTPS content. This guide demonstraights setting up a Nginx proxy service infront of the Varnish cache to serve cached HTTPS content. This configuration also allows for specific SSL sites to be served through Varnish caching and/or allows other sites to bypass the caching if the HTTPS to HTTP translation doesn't work properly. In addition is seamlessly translates local domain HTTP links and resources to HTTPS so the original site doesn’t need to be edited and can serve both HTTP and HTTPS from he same cached files. In addition a header is added to seamlessly update external resources to HTTPS to prevent Mixed Content warnings. In addition it will minify resources automatically thus simplifying the CMS/code side.

This setup will allow the following request paths to serve content:

Client Browser -> Varnish (http:80) -> Apache (http:8080)

Normal HTTP request served through cache.

Client Browser -> Varnish (http:80) -> 301 to HTTPS Version

Optional rule to force a site to use SSL if a non-SSL page is requested.

Client Browser -> Nginx (https:443) -> Varnish (http:80) -> Apache (HTTP:8080)

Normal HTTPS request served through cache
All links to HTTP://ThisDomainName/ are converted to HTTPS://ThisDomainName/.
Header is added to upgrade external HTTP resources to HTTPS.

Client Browser -> Nginx (https:443) -> Apache (HTTPS:44344)

If my be needed bypass caching for specific sites if they don't work properly with cache configuration.
It is also possible to just exclude specific sites or url's from caching and/or minify in the varnish configuration. That approach should be taken where possible and this used as the last resort.

Some other notes:

You could of course put these services on different servers, or have multiple IP Addresses on the server if you prefer, but I find it much simpler to logical separate things by putting the backend services on non-standard ports instead.
You could host the frontend and backend components both in Nginx if you really wanted, but again this way things are more logical spirited and easier to follow.
Nginx will use HTTP2 when available for much faster delivery of many small resource files. Site auditing/speed test warnings thus far don't take this into account and will still complain about the number of files even though it's not relevant on HTTP2.
Exceptions can be added in Varnish to easily prevent specific directories or URL's from being cached.
By default any cookies that vary by request or by user will break the cache. There are some predefined cookies that are ignored, and more can be added as needed. In general a cookie shouldn't be changing how the server responds so in most cases you can safely ignore all cookies.
letsencrypt.sh is used in this document to automate SSL certificates for all the sites. Standard certificates can use used in places of specific sites if prefered.
This configuration uses a Apache+Ruby instigation on minify CSS and JS scripts on the fly. You would not normally want to do this because of the server cost of processing requests, but with the vanish cache layer this has not cost as the result is cached. This greatly simplifies the backend/code and doesn't require pre-minified files, though if those are detected they will be passed through unchanged.
In addition when using HTTP/2 you should disable File compilation plugins and features as they will actually slow the page down by loading more resources then needed. In HTTP/2 a smaller lists of individual files is better then one larger compiled file.

Some example performance numbers: (Time to download HTML / All assets loaded)

Serving a simple Wordpress page: Uncached

441ms; 1.6s

Serving a simple Wordpress page: Cached

63ms, 1.1s

Serving a complex Magneto storefront page: Uncached

With configurations: 7900ms, 9.25s
Simple product: 1690s, 1.95s

Serving a complex Magneto storefront page: Cached

With configurations: 121ms, 2.0s
Simple product: 143ms, 1.0s

Server configuration

If you already have some of this set up you may not need to configure everything listed below.

Installing Required Packages

This guide assumes the following in a FreeBSD environment. Install them as needed or install your own preferences. The setup will work in other linux systems as well though some of the services will be configured different and the file paths may change.

Mysql/PHP/etc
openssl
Varnish 4 or grater
Apache 2.4 or greater
Nginx 1.11 or greater
lets encrypt.sh 0.2 or greater (or you can use your own certificates)
ruby 2.2 (optional, for seamless resource minification)

ruby-gems
rubygem-cssmin
rubygem-jsmin

Activate Services and Configure

Once you have a servqer up and running with the required software you will want to activate the various services and configure them on boot. The following should be added to your /etc/rc.conf file to set the default services.

apache24_enable="YES"
mysql_enable="YES"
varnishd_enable="YES"
varnishd_config="/usr/local/etc/varnish.vcl"
#varnishd_extra_flags="-p http_resp_hdr_len=16192"
#varnishd_listen=":8088"
#varnishd_backend="localhost:80"
#varnishd_jailuser="www"
nginx_enable="YES" # Proxy SSL for HTTP/2

In addition the following needs to be added to your /etc/periodic.conf file get SSL certificates to automatically renew.

weekly_letsencrypt_enable="YES"
#weekly_letsencrypt_deployscript="/usr/local/etc/letsencrypt.sh/deploy.sh"

Configure LetsEncrypt.sh

LetsEncrypt.sh doesn't really put the certificates anyplace useful and the certificates need to be manipulated as well. The following code needs to be added to the /usr/local/etc/letsencrypt.sh/hook.sh file inside the deploy_cert function.

# Copy certificates to standard SSL folder.
cp $CERTFILE /usr/local/etc/ssl/$DOMAIN.crt
cp $CHAINFILE /usr/local/etc/ssl/$DOMAIN.chain.crt
cp $KEYFILE /usr/local/etc/ssl/$DOMAIN.key
# Merge certificate chain files.
cat $CERTFILE $CHAINFILE > /usr/local/etc/ssl/$DOMAIN.merge.crt
# Adjust permissions so that web servers can access certificates.
chgrp www /usr/local/etc/ssl/$DOMAIN.crt /usr/local/etc/ssl/$DOMAIN.chain.crt /usr/local/etc/ssl/$DOMAIN.key /usr/local/etc/ssl/$DOMAIN.merge.crt
chmod g+r /usr/local/etc/ssl/$DOMAIN.crt /usr/local/etc/ssl/$DOMAIN.chain.crt /usr/local/etc/ssl/$DOMAIN.key /usr/local/etc/ssl/$DOMAIN.merge.crt 
echo "Copied certs for $CERTFILE on `date`." >> /var/log/letsencrypt

Configure PHP

There is lots of possible configuration here that I wont get into but in brief you will probably want to enable OP Cache by setting opcache.enable=1. There is more information here.

Configure Backend Apache Web Server

You will need to make a few changes to the default apache configuration as indicated below.

In /usr/local/etc/apache24/httpd.conf make sure the following lines are uncommented or updated as indicated.

# Change the HTTP listen port.
Listen 8080

# Load required Modules
LoadModule deflate_module libexec/apache24/mod_deflate.so
LoadModule ssl_module libexec/apache24/mod_ssl.so
LoadModule vhost_alias_module libexec/apache24/mod_vhost_alias.so
LoadModule rewrite_module libexec/apache24/mod_rewrite.so
LoadModule php5_module        libexec/apache24/libphp5.so
LoadModule ext_filter_module libexec/apache24/mod_ext_filter.so // Optional for minification
LoadModule expires_module libexec/apache24/mod_expires.so // Optional to inject expire headers

# Replace %h with %{X-Forwarded-For}i in all LogFormat settings
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common

# Virtual hosts
Include etc/apache24/extra/httpd-vhosts.conf

# Secure (SSL/TLS) connections
Include etc/apache24/extra/httpd-ssl.conf

Also edit the /usr/local/etc/apache24/extra/httpd-ssl.conf file to change the HTTPS listen port.

# Change the HTTPS listen port.
Listen 44344

Create a new file /usr/local/etc/apache24/modules.d/030_ext_filter_module.conf for the minification feature.

<IfModule mod_ext_filter.c>
  ExtFilterDefine jsmin \
                  mode=output \
                  intype=application/javascript \
                  outtype=application/javascript \
                  cmd="/usr/local/bin/ruby -e 'require \"rubygems\"; require \"jsmin\"; code = STDIN.read;  begin; min = JSMin.minify(code); rescue; min = code; ensure; puts min; end;'"
  <If "%{HTTP:X-Minify} =~ /js/">
    AddOutputFilter jsmin js
  </If>
</IfModule>

<IfModule mod_ext_filter.c>
  ExtFilterDefine cssmin \
                  mode=output \
                  intype=text/css \
                  outtype=text/css \
                  cmd="/usr/local/bin/ruby -e 'require \"rubygems\"; require \"cssmin\"; code = STDIN.read;  begin; min = CSSMin.minify(code); rescue; min = code; ensure; puts min; end;'"
  <If "%{HTTP:X-Minify} =~ /css/">
    AddOutputFilter cssmin css
  </If>
</IfModule>

Create a new file /usr/local/etc/apache24/modules.d/040_expires_module.conf for the append expire headers feature. Tweak these values as you like.

<IfModule mod_expires.c>
ExpiresActive on
ExpiresByType image/gif "access plus 1 months"
ExpiresByType image/jpg "access plus 1 months"
ExpiresByType image/jpeg "access plus 1 months"
ExpiresByType image/png "access plus 1 months"
ExpiresByType text/js "access plus 1 days"
ExpiresByType text/javascript "access plus 1 days"
ExpiresByType application/javascript  "access plus 1 days"
ExpiresByType text/css  "access plus 1 days"
</IfModule>

You can start Apache now with sudo service apache24 start to verify the configuration though there are no sites to test yet.

Configure Varnish Caching Service

Next you can set up a default rules file for the varnish caching. This is rather complex and can have a lot of custom rules and variations depending your sites, software, and preference. I have included the /usr/local/etc/varnish.vcl file that I use, though some of the rules may not be applicable to your sites. This file is configured for general Wordpress sites and Magento storefronts and there are some comments and details in the file.

You can start Varnish now with sudo service varnishd start to verify the configuration though there are no sites to test yet.

Configure Frontend Nginx Proxy Server

Finally configure the Nginx proxy service by editing you /usr/local/etc/nginx/nginx.conf file and adding or updating the contents as indicated.

http {
    # Append Access-Control-Allow-Origin only if it's not already present.
    map $upstream_http_access_control_allow_origin $access_control_allow_origin {
        '' "https://$host always";
    }

    # Enable compression
    gzip  on;
    gzip_types text/css application/javascript;
    gzip_disable        "MSIE [1-6]\.";
    gzip_comp_level 5;

    # Transcode all local resource, form actions, and links from HTTP to HTTPS.
    sub_filter 'http://$host/' 'https://$host/';
    sub_filter 'http:\/\/$host\/' 'https:\/\/$host\/';
    sub_filter_last_modified on;
    sub_filter_once off;
    sub_filter_types application/xml text/xml;

    # Add a header to tell the client browser to automatically request HTTPS resources form remote sites instead of HTTP to prevent mixed content warnings.
    add_header Content-Security-Policy upgrade-insecure-requests;
    #Add a header to allow for HTTPPS scripting calls to be handled by HTTP in some edge cases where the transcoded URL's missed something.
    add_header Access-Control-Allow-Origin $access_control_allow_origin;

    # Add default headers for the proxy request.
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP  $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-Port 443;
    proxy_set_header Accept-Encoding "";
    proxy_read_timeout 600;
}

You can start Nginx now with sudo service nginx start to verify the configuration though there are no sites to test yet.

Add and Configure a new Website

Now that the server is configured properly you can set up your sites. Repeat this process for each site.

Configure DNS

Configure your DNS to point at the server. You may want to use a custom HOSTS record on your computer for testing instead or when migrating a site.

Load the Site Files

Load your site file and configure as needed. The instructions below assume your site files are loaded into folders using the site domain name like /usr/local/www/www.DomainName.com/ but you can change this if you prefer.

Though it seams counterintuitive you will want to disable resource compilation, minify, and compression in your CMS/site code. Compilation will actually slow down requests a but on HTTP/2 and cost extra bandwidth in many cases. Similarly compression is not needed in Apache as Varnish will take care of that.

As for the CMS systems minifying files, this can be turned off and Apache can do this at runtime. This way you can easily turn it off in your session for testing without affecting other users and you don't need to deal with the overhead and possible performance costs of many of the plugins. This setup is very efficient because the files are minified just before they are cached in varnish.

Add an Apache Virtual Host

Create the backend website in Apache by editing your /usr/local/etc/apache24/extra/httpd-vhosts.conf file and adding a new section like this. Make sure to replace the DomainName with the proper value.

<VirtualHost *:8080>
    #ServerAdmin AdminEmailAddress
    DocumentRoot /usr/local/www/www.DomainName.com
    <Directory "/usr/local/www/www.DomainName.com">
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
    ServerName www.DomainName.com
    ServerAlias DomainName.com
    Alias /.well-known/acme-challenge/ /usr/local/etc/letsencrypt.sh/.acme-challenges/
    ErrorLog "/var/log/apache/www.DomainName.com-error_log"
    CustomLog "/var/log/apache/www.DomainName.com-access_log" common
</VirtualHost>

If your domain name starts with a www. you can also add the domain name minus the www into the ServerAlias field so it responds to both. If you have other domains that also point to the same site they should also be added to the ServerAlias field, then you can add 301 redirect rules in a .HTACCESS file to 301 requests to a default hostname if you prefer.

You also need to update /etc/hosts and find the ::1 localhost and 127.0.0.1 localhost lines and add your primary domain to the end of each like so.

::1   localhost www.DomainName.com
127.0.0.1  localhost www.DomainName.com

Finally restart Apache now with sudo service apache24 graceful to start up the new site and navigate to http://www.DomainName.com to see it, or on the server by typing curl -i --http2 http://www.DomainName.com | less. You can add :8080 to the end of the URL to see the uncached version, though depending on your network configuration this may not work.

Generate SSL Certificate

To generate a new SSL Certificate for your site edit the /usr/local/etc/letsencrypt.sh/domains.txt file and add your DomainName to it along with all the alternate domain names and/or non-www versions like so.

www.DomainName.com DomainName.com www.DomainName2.com

Each separate site should be a separate line in this file and any alternate domain names listed in the ServerAlias field in Apache should be included on the line after the primary domain name. The primary domain name/first entry will be used below to configure the SSL sites. Once done run the following command to generate the new certificates manually, from then on they will be automatically updated as needed.

/usr/local/bin/letsencrypt.sh -c

Configure Nginx Proxy Site

And now on to the crux of this article, configuring proxied SSL sites through the Varnish caching service. To do this create a new site in Nginx by editing your /usr/local/etc/nginx/nginx.conf file updating the DomainName as appropriate.

server {
    listen       443 ssl http2;
    server_name  www.DomainName.com DomainName.com;
    ssl_certificate    /usr/local/etc/ssl/www.DomainName.com.merge.crt;
    ssl_certificate_key /usr/local/etc/ssl/www.DomainName.com.key;

   location / {
            proxy_pass http://localhost:80;
    }
}

Here you will need to again include the same additional domain specified in theServerAlias field in Apache after the primary domain name in the server_name field. You can restart Nginx now with sudo service nginx restart and navigate to https://www.DomainName.com to see the SSL version of the site, or on the server by typing curl -i --http2 https://www.DomainName.com | less.

Testing and Controlling Transparent Minification

Minification should work by default most of the time. If your getting empty css or javascript files or nothing is being minified run the following commands on the server to ensure the proper packages are installed and working.

echo "p {
    color: red;
    font-weight: bold;
}" | /usr/local/bin/ruby -e 'require "rubygems"; require "cssmin"; code = STDIN.read; begin; min = CSSMin.minify(code); rescue; min = code; ensure; puts min; end;'

echo "(function() {
    console.log('test');
})();" | /usr/local/bin/ruby -e 'require "rubygems"; require "jsmin"; code = STDIN.read; begin; min = JSMin.minify(code); rescue; min = code; ensure; puts min; end;'

You can switch override the default processing rules by setting a cookie in your browser.

document.cookie="minify=disable;" // disable for this domain
document.cookie="minify=force;" // force for this domain
document.cookie="minify=;" // use default rules for this domain

In addition in /usr/local/etc/varnish.vcl you can permanently completely turn this feature off, block specific domains, or block individual scripts or css files from being processed. Also note that files named .min.css or .min.js won't be processed.

SSL Related Issues

This is where things can get somewhat complicated. As Varnish can't process HTTPS content, we are serving the entire site through HTTP and the proxy server at the end is delivering it as HTTPS. There are all sorts of issues this can cause in CMS systems though if for example the CMS system is checking the URL protocol and returning a 301 in an attempt to force SSL that we would end up in a redirect loop. This would happen in Wordpress for example if we set the Site Address URL or WordPress Address URL to https:// instead of http://. Some systems like this we can simply leave the validation URL as http://www.DomainName.com and everything will work out correctly due to the URL translating logic in Nginx, but others may not. Depending on your software it may also be possibly to identify the protocol validation code and add additional code to check the X-Forwarded-Proto header in addition to the request protocol before redirecting. You may also be able to disable some Force SSL settings to fix redirect issues, or in Wordpress deactivate and SSL plugins.

Another issue you may run into is that caching is not occurring between different sessions or users. This would generally happen because users are ending up with different cookies. The following command will let you monitor requests as they come in to see if they are being cached or not and troubleshoot the issues. You may need to add some more cookie exceptions in /usr/local/etc/varnish.vcl and restart. Some trial and error may be needed here.

sudo varnishncsa  -F  "%{X-Forwarded-For}i %{X-Forwarded-Proto}i://%{Host}i%U %{Varnish:handling}x %{Varnish:hitmiss}x"

Force SSL

Assuming that everything is working correctly with the SSL configuration you can force all traffic or only specific sites to use SSL. To do this we need to do it in the Varnish configuration as the backend is all non-SSL and we don't want redirect loops or complex conditional code. To do this edit your /usr/local/etc/varnish.vcl file find the # Optinal rule to force SSL line. Add or edit it as needed with your new DomainName.

# Optinal rule to force SSL Everywhere for listed domains.
if ( (req.http.host ~ "^(www.DomainName.com|www.AnotherDomainName.com)") && req.http.X-Forwarded-Proto !~ "https") {
    set req.http.x-redir = "https://" + req.http.host + req.url;
    return (synth(750, ""));
}

# Or force all requests to SSL
if ( req.http.X-Forwarded-Proto !~ "https") {
    set req.http.x-redir = "https://" + req.http.host + req.url;
    return (synth(750, ""));
}

Add Caching exceptions

If you have directories or sections that should not be cached, or if there are headers or cookies that should prevent caching when present you can add rules to your /usr/local/etc/varnish.vcl file to prevent caching in the # Add additinal rules here section. Here are some examples:

# Bypass shopping cart and checkout requests
if (req.url ~ "/(cart|my-account|checkout|addons)") {
    # Don't cache, pass to backend
    return (pass);
}

# Either the admin pages or the login
if (req.url ~ "/wp-(login|admin|cron)") {
    # Don't cache, pass to backend
    return (pass);
}

# Custom Scripts and API
if (req.url ~ "/api/") {
    # Don't cache, pass to backend
    return (pass);
}

As you are making changes you can test them by running sudo service varnishd reload so that the entire cache isn't cleared. Additionally you can easily check which requests are hitting the cache with this command.

sudo varnishncsa  -F  "%{X-Forwarded-For}i %{Varnish:time_firstbyte}x %m %{X-Forwarded-Proto}i://%{Host}i%U %{Varnish:handling}x %{Varnihitmiss}x"

Configuring a CMS to Purge it's Cache

The next major hurdle is that as you are eloping or tweaking your CMS/site you will quickly discover that the varnish caching layer is caching changes and won't check the backend for updates. You can restart the entire service with sudo service varnishd restart but that will clear the cache for all your sites and it quite cumbersome. Instead you will want to configure a plugin or custom code for your site to just clear it's cache on demand.

To configure varnish integration in Magento log in to the backend and go to Store -> Configuration -> Advanced -> System -> Full Page Cache. Set the Caching Application to Varnish Cache and save the changes. In addition you will need to edit your app/etc/env.php file and this section at the end just before the last line. Additionally this patch has to be applied to your Magneto install of it will clear all site's cache's instead of just the current site.

In wordpress you simply need to install the Varnish HTTP Purge plugin and your good to go. Please note that you will get unexpected results and poor performance if you didn't add the primary hostname to your servers /etc/hosts file as indicated above.

If you have some other system of custom software you can create your own scripts or internal plugins to purge resources as needed. A script simply needs to send HTTP request with a type of PURGE and a host header specifying the domain to purge. It can purge the entire domain, or purge a specific url. Some examples that can be used directly or converted into your CMS's language:

# Do wildcard patching to purge a directory or the entire site
curl -H "Host: www.DomainName.com" -H "X-Purge-Method: regex" -X PURGE http://127.0.0.1/.* 
# Purge a specific file/url including url parameter
curl -H "Host: www.DomainName.com" -H "X-Purge-Method: exact" -X PURGE http://127.0.0.1/some/file.html?param
# Purge all versions of a file/url ignoring any parameter tokens
curl -H "Host: www.DomainName.com" -H "X-Purge-Method: default" -X PURGE http://127.0.0.1/some/file.html

Bypass Caching Site

If your you have some sites that are causing issues and not working properly through the Varnish cache, you can bypass the cache for specific sites. To do this first you need to add a backed SSL site in Apache by editing your /usr/local/etc/apache24/extra/httpd-vhosts.conf file and adding a new section like this. Make sure to replace the DomainName with the proper value.

<VirtualHost *:44344>
    #ServerAdmin AdminEmailAddress
    DocumentRoot /usr/local/www/www.DomainName.com
    <Directory "/usr/local/www/www.DomainName.com">
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
        AddOutputFilterByType DEFLATE text/css
        AddOutputFilterByType DEFLATE application/javascript
    </Directory>
    ServerName www.DomainName.com
    ServerAlias AnotherDomainName.com
    ErrorLog "/var/log/apache/www.DomainName.com-error_log"
    CustomLog "/var/log/apache/www.DomainName.com-access_log" common
    SSLEngine on
    SSLCertificateFile /usr/local/etc/ssl/www.DomainName.com.crt
    SSLCertificateKeyFile /usr/local/etc/ssl/www.DomainName.com.key
    SSLCertificateChainFile /usr/local/etc/ssl/www.DomainName.com.chain.crt
</VirtualHost>

Again if your domain name starts with a www. you can also add the domain name minus the www into the ServerAlias field so it responds to both. If you have other domains that also point to the same site they should also be added to the ServerAlias field. Then restart Apache now with sudo service apache24 graceful.

Next update the Nginx configuration by editing your /usr/local/etc/nginx/nginx.conf file and changing the line below in the proper server section.

proxy_pass http://localhost:44344;

Again restart Nginx now with sudo service nginx graceful and at this point the SSL site should work correctly, though its pages won't be cached any longer.

Easily post leads in Wordpress Fast Secure Contact Form Plugin to iContact

2017-01-30T15:31:00.001-05:00

When using the Fast Secure Contact Form plugin for wordpress is is very very simply to seamlessly forward leads from any forms you wish to the iContact platform with very little configuration.

First off create any lists in iContact you might what and and forms you need as normal. You could set up one list per form, have multiple forms feed to one list, or even have one form post to more then one list. When doing this you will need to record the List Id for each list you wish to post to. You can get this by going to the list view iContact and click on the list name to edit it. Then check the page URL for the ListId number like so:

https://app.icontact.com/icp/core/mycontacts/lists/edit/ListId/?token=...

Next you will need to go into each form that you have created that you want to forward to iContact in your Wordpress site. When looking at the form settings go to the Advanced tab and scroll down to the Silent Remote Sending Settings section. Change the following settings there:

Silent Remote Sending: Enabled: Method POST
Silent Remote URL: http://YouWordpressDomain/icontact-post.php
Silent send fields to ignore

full_message
akismet
date_time
ip_address
name_to
email_to
subject
from_name

Silent send fields to rename

first_name=firstName
last_name=lastName
from_email=email
company=companyname
phone=phone

Silent send keys to add

list_id=ListId

Where ListId is the applicable id recorder for each form. The ListId can even be a coma separated list of id's.

As a final step you will need to upload the following connector script to your sites root directory and edit it add your iContact authentication information such that the Silent Remote Url entered above is correctly points to it. If you have any issues there is a test line in the file that can be uncommented to supply sample data. Also note that you may need to add to or tweak the Silent send fields list if you have changed field names or added custom fields that you want to forward.

Guide to Custom Database Access in Umbraco 7 With Built in PetaPoco

2016-11-01T14:19:00.000-04:00

Though there are a lot of ways to write custom database code, and in many place in Umbraco custom tables my no be needed as the CMS can handle most any custom data needs. There are reasons and use cases for using custom database tables and code including:

Reading data from external systems.
Large datasets that we don’t want to clog out cache’s and Umbraco tree with.
Performance sensitive querying where the Document -> Property models setup may not be preferable.
Complex cross-joins and many-to-many relationships that may not be easy to replicate in Umbraco.
Customer data, log data, or “write once” style form submissions where the data need not be in Umbraco trees.
Data that may need to be easily accessed from external systems.

They can also use public Web API methods or custom views that read from the Umbraco core tables but writing data this was it less safe.

Though there are a lot of ways to do the database code and everyone has their own preferences, this guide is using the same PetaPoco system that Umbraco uses internal. This has a few key benefits:

Very lightweight system yet powerful.
Fully compatible with Umbraco models, methodologies, and serialization.
Easy to implement and generate models.
All code is using existing database code already present in the Umbraco Core to minimize overhead.

The system can be used in one of two ways. You can create all our database models(POCOs) in .NET and then have Application Startup code that creates or updates any missing database objects or you can create all the database objects yourself in the database using standard management tools and have PetaPoco automatically generate models(POCOs) based off that database.

We are going to be doing the second method in this article as I find it easier to maintain and design with but this post has some example of working the other way. Either way the second section of this article describes querying and extending models which may still be helpful.

Automatically Generating POCO’s from a Database

First of you will need to install the PetaPoco package from NuGet into your project or into a separate depended project.

If you are using a separate project it will need to reference the Umbraco.Core package.
Once installed you need to locate the Models/PetaPoco.cs file and exclude it from your project. This is redundant as it is already included in the Umbraco dll’s and would cause confusion.

You will need to add your Connection String to the web.config file if it’s different then the Umbraco database and update the ConnectionStringName below.

If using a separate project you will also need to copy the umbracoDbDSN or your custom connection string to that project’s app.config file.

You will need to open up the Models/Generated/Database.tt file and make some configuration changes. These settings below should get you started.

Make sure you change the ProjectNamespace to match what is set in the Default namespace of your projects properties.

    // Settings
    ConnectionStringName = ""; // Uses last connection string in config if not specified
    Namespace = "ProjectNamespace.Models.Generated";
    RepoName = "";
    GenerateOperations = false; // Don’t need these.
    GeneratePocos = true; // We definitely want these.
    GenerateCommon = false; // Don’t need any of this code.
    ClassPrefix = "";
    ClassSuffix = "";
    TrackModifiedColumns = false; // We shouldn’t need change tracking logic for most things as we aren’t creating an entire world here.  You can use it if needed though.
    ExplicitColumns = true;
    ExcludePrefix = new string[] {"cms", "CMS", "umbraco"}; // This is telling the generator to exclude the Umbraco tables from the generated code.
    IncludeViews = true; // You can create custom views to pull data or some CMS data is desired.

    // Read schema
    var tables = LoadTables();
    tables["SomeTable"].Ignore = true;  // This is how we can suppress other tables.

Now you can save the Database.tt file and Visual Studio will automatically generate the models. You can expand the Databse.tt file and select the resulting Database.cs file to review the generated code.

Extending Models with extra logic

You can add partial class’s in the Generated folder to extend your models and even attach interfaces and validation code. Additional properties will not affect the database queries due to the explicit columns option.

This sample shows using an interfaces, adding custom properties, using validation, and setting default values.

Examples of querying and updating the POCO’s in Umbraco

There are many variations on this code but here are some brief examples. More samples are available here.

using Umbraco.Core.Persistence;
using Umbraco.Core.Persistence.DatabaseModelDefinitions;
using ProjectNamespace.Models.Generated;
using ProjectNamespace.Extensions;

// Fetch a set of rows with sorting, filtering, and pageing.
public IEnumerable<object> GetChildren(string orderBy, Direction orderDirection, string filter, long page, long itemsPerPage)
{
    if (orderBy == "fullName")
        orderBy = "(firstName + ' ' + lastName)";

    var sql = new Sql();

    if (!String.IsNullOrEmpty(filter))
        sql.Where(@"'#' + convert(nvarchar, id) = @0
OR CONVERT(nvarchar(30), dateAdded, 101)  LIKE @1
OR (firstName + ' ' + lastName) LIKE @1
OR companyName LIKE @1, filter, '%' + filter + '%');

    if (orderDirection == Direction.Ascending) sql.OrderBy(orderBy); else sql.OrderByDescending(orderBy);

    try
    {
        if (page > 0 && itemsPerPage > 0)
            return DatabaseContext.Database.Fetch<FormChannelProgramSignupOverview>(page, itemsPerPage, sql);
        else
            return DatabaseContext.Database.Fetch<FormChannelProgramSignupOverview>(sql);
    }
    catch (Exception e)
    {
        throw new Exception(ApplicationContext.DatabaseContext.Database.LastSQL, e);
    }
}

// Get a single item.
DatabaseContext.Database.SingleOrDefault<FormChannelProgramSignup>(id);

// Only load parts of a model.
var sql = new Sql()
    .Select<FormChannelProgramSignup>(s => s.id, s => s.firstName, s => s.lastName);
DatabaseContext.Database.Fetch<FormChannelProgramSignup>(sql);
// Note that instead of doing this it is far similar to create views in the database that select the columns you need.  They will generate their own POCO’s and you can query those instead.  This also means that only the loaded properties will be serialized to JSON code.

// Complex conditionals, this is much cleaner than trying to do the same thing with a single string.
var hasWhere = false;
var sql = new Sql();
if (!String.IsNullOrEmpty(filter["search"]))
    sql.AndWhere("'#' + convert(nvarchar, id) = @0 OR companyName like @1", ref hasWhere, filter["search"], '%' + filter["search"] + '%');
if(!String.IsNullOrEmpty(filter["dateStart"]))
    sql.AndWhere("dateCreated >= @0", ref hasWhere, DateTime.Parse(filter["dateStart"]));
if (!String.IsNullOrEmpty(filter["dateEnd"]))
    sql.AndWhere("dateCreated <= @0", ref hasWhere, DateTime.Parse(filter["dateEnd"]));

// Insert, update, delete, and validation examples.  Again there are many more methods available.
var record = isNew ? new FormChannelProgramSignup() : DatabaseContext.Database.SingleOrDefault<FormChannelProgramSignup>(id);
record.lastName = “test”;
DatabaseContext.Database.Validate((IDataModel)record);
DatabaseContext.Database.Save(record);
DatabaseContext.Database.Delete(record);

These examples use these some extra helpers.

Implementing One to One, One to Many, and Many to Many Joins

This can get a bit complicated to understand, but when set up correctly it works quite well. In short you can do joins in the SQL code but need to create Relator code and attach those to the fetch code so that the rows are disassembled into associated models in the way you expect.

The following examples are assuming the following Schema.

ShopMarkets

MarketId
Region
Market

ShopMarketServices

MarketId
ServiceId

ShopServices

ServiceId

ShopZipcodes

City
State
ZipCode

This is a relator that includes all three types of relationships. These should be placed in a Models/Relators folder in your project.

In addition some partial classes are needed in your Models/Generated folder to add properties to store the related models.
To use it you would use the following SQL code. Note that these are all using the same Map function with different signatures but you could use different function names if you prefer for clarity.

using Umbraco.Core.Persistence;
using ProjectNamespace.Models.Generated;
using ProjectNamespace.Models.Relators;

// One to Many Query
var sql = new Sql().Select("*").From<ShopService>(DatabaseContext.SqlSyntax)
    .LeftJoin<ShopMarketService>(DatabaseContext.SqlSyntax)
    .On<ShopService, ShopMarketService>(DatabaseContext.SqlSyntax, s => s.ServiceId, s => s.ServiceId)
    .Where<ShopMarketService>(s => s.MarketId == Current.MarketId);
DatabaseContext.Database.Fetch<ShopService, ShopMarketService, ShopMarketService>(new ShopMarketRelator().Map, sql);

// One to Many Query
var sql = new Sql().Select("*")..From<ShopMarket>(DatabaseContext.SqlSyntax)
    .LeftJoin<ShopZipcode>(DatabaseContext.SqlSyntax)
    .On("ShopMarkets.Region = ShopZipcodes.State AND ShopMarkets.Market = ShopZipcodes.City") // Can use .On<T1, T2>(left => left.Field, right => right.field) on similar joins.
    .Where<ShopMarket>(s => s.Region.InvariantEquals(CurrentCode) && s.Market.InvariantEquals(CurrentMarket));
DatabaseContext.Database.Fetch<ShopMarket, ShopZipcode, ShopMarket>(new ShopMarketRelator().Map, sql).FirstOrDefault();  // Must use Fetch().FirstOrDefault() here as Database.FirstOrDefault doesn’t have the proper signatures.

// Many to Many Query
var sql = new Sql().Select("*").From<ShopMarket>(DatabaseContext.SqlSyntax)
    .LeftJoin<ShopMarketService>(DatabaseContext.SqlSyntax)
    .On<ShopMarket, ShopMarketService>(DatabaseContext.SqlSyntax, s => s.MarketId, s => s.MarketId)
    .LeftJoin<ShopService>(DatabaseContext.SqlSyntax)
    .On<ShopMarketService, ShopService>(DatabaseContext.SqlSyntax, s => s.ServiceId, s => s.ServiceId)
DatabaseContext.Database.Fetch<ShopMarket, ShopMarketService, ShopService, ShopMarket>(new ShopMarketRelator().Map, sql).FirstOrDefault();

Then you can use the added properties to access associated data. In my tests the Select(“*”) part was required or invalid SQL was generated but that may be fixed at some point.

Guide to using Strong Typed PublishedContnetModels in Umbraco 7

2016-10-31T16:49:00.000-04:00

The recent Umbraco 7.4 release there is an included Model Builder tool. This may not seem like a big deal but it can be very helpful to developers and I recommend embracing it. In essence what it does it let you generate strong typed models in .Net based off your Document Types and Media Types and their hierarchy. This can be done a few ways including at runtime in memory or with pre-compile DLL’s. Then in Views, Macro’s and Controllers these strong typed models can be used instead of normal code to reduce potential coding errors and typos, and to provide IntelliSense in Visual Studio if you do editing there. These changes for the most part are non-breaking and the old style can still be used.

In short if will replace the first example template with the second.

@inherits Umbraco.Web.Mvc.UmbracoTemplatePage
@Model.Content.GetPropertyValue(“myProperty”)
@(Umbraco.Media(CurrentPage.myImage).umbracoFile)
@Model.Content.GetPropertyValue(“myProperty”, true, “Missing”)
@Umbraco.Field("myProperty ", recursive: true, altFieldAlias: "defaultProperty")

@inherits Umbraco.Web.Mvc.UmbracoTemplatePage<DocumentType>
@Model.Content.MyProperty
@(((Image)Model.Content).MyImage.UmbracoFile
@Model.Content.GetPropertyValue(s => s.MyProperty, true, “Missing”)
@Model.Content.GetPropertyValue(s => s.MyProperty, true, Model.Content.DefaultProperty)

This article describes the following approach: Generating C# models in the live Umbraco Instance and then pulling them down into the generated models down into the source Visual Studio Project and pre-compiles them there in the main application DLL. This has some major benefits over other approaches including:

All code is pre-compiled and no generation or compilation is happening when the website is actually running or starting up.
Models generation isn’t dependent on IIS Express and you don’t have to do desin work there or start up slow projects to manually create models.
Models are generated the same place you edit the document types and generated automatically.
Models are partial class’s that can be extended and can implement custom interfaces as needed.
There is no need for any special visual studio plugins or deployment rules to make this work. All parts described will work on any Visual Studio Editions.

To clarify we will have the following:

A normal production/development server running IIS/Umbraco site.

Document and media Types would be edited here, basically whatever system you currently do Umbraco BackOffice work in now
New model sources are automatically generated as changes are made, but the runtime DLL’s won’t be touched.

Developer box/Visual Studio containing the site project/Umbraco CMS from NuGet packages.

Project is configured to do deployments using Web Deploy.
Models sources are synced down from the deployed instance back to the Visual Studio project.
Generated Models may be extended with local code and are compiled into the main project DLL.
View files can be edited here to take advantage of design-time debugging and full IntelliSense or in Umbraco BackOffice as desired.

Setup

So to start with we have a basic MVC Website Solution in Visual Studio pulling in the Umbraco NuGet project or existing Umbraco project.
We have configured this site to use Web Deploy and have deployed a working site to a server that is up and running.
We have created some Document Types and Media Types for our site in Umbraco’s BackOffice.

Enable Model Generator

First we will need to edit the web.config file and add a few settings to the AppSettings section.

    <add key="Umbraco.ModelsBuilder.Enable" value="true" />
    <add key="Umbraco.ModelsBuilder.ModelsMode" value="LiveAppData" />
    <add key="Umbraco.ModelsBuilder.ModelsDirectory" value="~/PublishedContentModels/Generated" />
    <add key="Umbraco.ModelsBuilder.ModelsNamespace" value="ProjectNamespace.PublishedContentModels" />

The first option is enabling the model builder, the second is telling the site to only generated model source code and to do so automatically when document and media types are updated.
Next we are specifying a directory to save these source files into. We want this to be part of the project directory so we can easily retrieve updates.
Now update the ProjectNamespace to use for all the models. You will want to use the same namespace that is specified as the Default namespace on the website project in Visual Studio as ProjectNamespace portion so that you don’t have to manually change it every time you add or customize a model.

Next you will want to edit the ~/Views/Web.config file add the same namespace above to the namespaces there.

        <add namespace=" ProjectNamespace.PublishedContentModels" />

Add the Umbraco Core Property Value Package

This package is needed to that generated models use IPublishedContent types for all properties that use internal pickers instead of returning string or integer return types. This enables you to use the first line of code instead of the second. It also makes checking for null objects much easier.

@Model.Content.LinkedPage.Name
@Umbraco.Content((int)Model.Content.LinkedPage).Name.
@if(Model.Content.LinkedPage == null) {}

Note that if you have an existing site this package will break any templates and macros that are expecting integer or string returns from property pickers. You will need to go through the project and update such places to expect IPublishedContent or IEnumberable<IPublishedContent> as applicable. This will not harm performance though as all nodes are cached.

Apply patch for Strong Typed GetPropertyValue methods

You will want to compile and replace your Umbraco.ModelsBuilder with the patched version available here to enable some additional functions. The following are not yet supported in the Umbraco release but work with this patch.

@Model.Content.GetPropertyValue(s => s.BannerImage, true) // Search recursively up the tree.
@Model.Content.GetPropertyValue(s => s.BannerImage, "someOtherImage.gif")  // Specify a default value.
@Model.Content.GetProeprtyValue(s => s.BannerImage, Model.Content.AlternateImage)

Syncing changes back to Visual Studio

There is a bit more initial setup to get his working.

Once these changes are applied and deployed go into the Umbraco site and go to the Developer Dashboard.
Click on Model Builder and click Generate Models.

You should only need to this once as future changes will automatically regenerate the sources but you can come here to manually do so if you have any issues.

In your visual studio website project created a new folder at the root level named PublishedContentModels.
Right Click on this new folder and select Replace PublishedContentModels From Server to download all your generated models.

You may need to toggle the Show All Files option in the project view and/or refresh the project view to see the new files.

The files will not be included in your project by default so Right Click on the Generated folder and select Include In Project to include all models.
Republish the website or test locally to load the new models.

Syncing future changes to the Models

In the future when you make Document Type or Media Type changes including adding or removing types or adding, removing, or changing properties you will need to do the following the synchronize the changes.

Right Click on the Generated folder and select Replace Generated From Server.
Include or Exclude any files for models that may have been added or removed.
Publish the website or manually copy over the main DLL to load the new models.

Some Caveats about the Models

In many cases no valid item may be specified or the linked item could have been removed. As such you will need to check for NULL values or provide a default value.

@if(Model.Content.Resource != null)
var linkedPage = Model.Content.GetPropertyValue(s => s.LinkedPage, Model.Content);

Also you might expect the models to return strong typed Model for content pickers but they do not as the would create recursion issues. Instead all documents and media are returned as IPublishedContent but that can still be cast to Strong Typed models as needed.

var bannerImage = (Image)Model.Content.BannerImage;
var resourceItems = (IEnumberable<ResourceItem>)Mode.Content.ResourceItems;

Or you can add strong typed accessor’s for those properties.

public partial class BasePage {
        public BasePage TypedUmbracoInternalRedirectId {  get { return (BasePage)UmbracoInternalRedirectId;  } }
        public IEnumberable<ResourceItem>TypedResourceItems {  get { return (IEnumberable<ResourceItem>)ResourceItems;  } }
}

Using the new Published Content Models.

There are lots of things that you can with the new models. The first thing to deal with is changing your Templates and Partial Views to use the new PublishedContentModels.

./uBlogsyPost.cshtml
@inherits UmbracoTemplatePage
@inherits UmbracoTemplatePage<UBlogsyPost>

./ShopSite.chstml
@inherits Umbraco.Web.Mvc.UmbracoTemplatePage
@inherits Umbraco.Web.Mvc.UmbracoTemplatePage<ShopSite>

Using Properties

Properties can be replaced with calls like these:

@Model.Content.MyProperty
@(((Image)Model.Content).MyImage.UmbracoFile
@Model.Content.GetPropertyValue(s => s.MyProperty, true, “Missing”)
@Model.Content.GetPropertyValue(s => s.MyProperty, true, Model.Content.DefaultProperty)
var locations = Model.Contnet.Children<Location>();
var landingPage = Model.Content.AncestorOrSelf<LandingPage>();

Using Inheritance

The models will have the same inheritance as the site, so if you have a document structure like this:

BasePage
BasePage/TextPage
BasePage/ImagePage

You could have the following templates:

./BasePage.chstml [Master Template]
@inherits Umbraco.Web.Mvc.UmbracoTemplatePage<BasePage>

./TextPage.chstml
@inherits Umbraco.Web.Mvc.UmbracoTemplatePage<TextPage>
    Layout = " BasePage.cshtml";

In this way the master template has typed access to all its properties and the children have their applicable properties as well.

Using Compositions or Interfaces

You can also create compositions or create your own Interfaces across multiple models that may share properties but don’t inherit from each other. Take this example for instance:

MainSite
MainSite/BodyPage
MainSite/CategoryPage
Blog
Blog/Landing
Blog/Post

In this case the Main Site and blog may have similar properties but they don’t inherit from each other. In this case you could create and interface and then add these two partials class's extend the existing models with consistent accessors.

Then you can have a template use the Interface instead.

./BasePage.chstml
@inherits Umbraco.Web.Mvc.UmbracoTemplatePage<IBasePage>
    Layout = "BasePage.cshtml";
@Model.Content.PageTitle

Note the ImplementPropertyType decorations are necessary for Model.Content.GetPropertyValue(s => s.type) calls to work on the interface.

You can accomplish the same thing by creating Compositions in Umbraco and the Composition will be auto-generated as interfaces though I haven’t tested this part out myself.

Additional custom properties and code

Additional code can be added for any of these models in partial class’s in the PublishedContentModels folder to add additional logic and move complex programming out of the template files and into the models.

Here is an example of an Image extended model that supports scaling images.

This can be called like so:

@{
  var image = Model.Content.BannerImage.Scale(500, 250);
}
<img src=”@image.UmbracoFile” width=”@iamge.UmbracoWidth” height=”@iamge.UmbracoHeight”>

Conditionals Based off Current Document Type

There may be lots of places in Master Templates and Partial Views where you are running conditional code based on the current document type of the page. These should be replaced with strong typed check’s instead. The first line is checking if a node is a specific type directly whereas the second line would return true if the node is that type of inherits from it in the hierarchy.

@if(Model.Content.DocumentTypeAlias == BodyPage.ModelTypeAlias)
@if(Model.Contnet is BodyPage)

Or a similar check to see if an ancestor of a given type exists:

@if(Model.Content.AncestorOrSelf<uBlogsyLanding>() != null) {}

You can also cast a model to a more specific type. For example:

@inherits UmbracoTemplatePage<BasePage>
@if(Model.Content is BodyPage) {
  var bodyPage = (BodyPage)Model.Content;
  @bodyPage.Contents
}

Strong Type Tree Traversal

There are many new variations on the common node traversal functions that let you specify a model type. You may need to add using Umbraco.Web; in your code to access them. A few examples:

Model.Content.Parent<CategoryPage>();
Model.Content.Descendants<BodyPage>();

Strong Typed Recursive and Default Text

The following will only work for now if you use the custom patch listed abode and then regenerated your models.

Load a property reclusively and providing a default value. These will work exactly like the Umbraco equivalents.

@Model.Content.GetPropertyValue(s => s.BannerImage, true)
@Model.Content.GetProeprtyValue(s => s.BannerImage, Model.Content.AlternateImage)

Issues in Macro Partial Views

You may discover that you can’t specify types for macro’s in the same way as you might expect. This is a limitation due to limitations with the PartialViewMacroPage class and its cacheing. You may not have the problem but for me the following example won’t work in a Macro Partial View currently.

@inherits Umbraco.Web.Macros.PartialViewMacroPage<BodyPage>
@Model.Content.PageTitle

There is a bit of a workaround that doesn’t add any processing overhead as the items are still cached properly, but it does make the code in Macro view files slightly different.

@inherits Umbraco.Web.Macros.PartialViewMacroPage
@{
  var Content = Umbraco.TypedContent(Model.Content.Id);
  @Content.PageTitle
}

Removing Double Redirects or Redirect Chains form .htaccess files.

2016-10-26T15:25:00.002-04:00

If you manage and large site using mod_rewrite or ISAPI_Rewrite 3 you will eventually run into issues maintaining the rewrite rules. After doing a few redesigns or moving things around repeatedly you can end up with situations where one page redirects to another page then another and so on in a chain. This is functional but inefficient, it can have a negative effect on user experience and on SEO value, and makes everything harder to maintain. This tool is a simple script you can pass you .htaccess file through to remove all these multi-step redirects and all pages redirect immediately on the first redirect to the final destination page.

This is a perl program that you will need to save along with a copy of your .htaccess file someplace with Perl installed.

Once saved you can run it by opening a terminal and running the following command:

perl simplify-rewrites filename

The Program will show a summary and a new filename.new file will be generated with the de-duping applied. You can use the following to get a list of all the changes for review.

diff -Na --unified=0 rw.txt rw.txt.new | less

Steam won't connect on mass Sierra 10.12

2016-10-25T20:31:00.000-04:00

I was running into this odd issue after upgrading to OS X 10.12 and nothing I would do seamed to resolve it. Tired tracking down error messages, reinstalling steam, changeling to beta and back, and other fixes. Ended up doing the following fix which straightened everything out.

Installed the OS X Command Line Tools.

Open Terminal.
Type in xcode-select --install and press enter.
Click install in the popup dialog.

A very quick and simple fix, though I'm not entirely clear why it was necessary. I did have the command like tools set up before I upgraded so maybe something broke. I also had a similar issues with Parallels for Mac after doing the upgrade.

Quick and Dirty OCR on OS X for Free

2015-12-28T11:32:00.001-05:00

Wait, Stop, Don't spend that $80+ on some fancy OCR software just yet. There are some free open source tools out there and with a bit of work you can have very functional workflow using them and OS X. This guide will help you set up an applet you can drop images onto to OCR them to plain text. This will not create PDF's, formatted, or indent documents, but rather just one large text block that is structured the same as the source text with line breaks. The following tools are used, but instructions are provided below so you don't need to download anything from the sites.

Homebrew The missing package manager for OS X
Tesseract Open Source OCR Engine
Automator

Install Tools

First off lets install Homebrew to simplify managing custom packages. This will keep everything in a special /usr/local/ directory so it won't interfere with OS X's normal system. To do this open up the Terminal application, copy and paste the following command, and press Return to execute it. You will be prompted to authorize the installation and may be asked to install some OS X command line tools directly from Apple.

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Next we need to install Tesseract and some supporting libraries. Again int he Terminal window enter each of these lines and press Return to execute them separately. They may take a few minutes to compleate before the command prompt returns.

brew install imagemagick
brew install tesseract --all-languages

Configure Applet

Next we need to create a user friendly way to do the OCR. We can easily do this with Automator in OS X. Open Automator and create a new Application. On the left search for and add Run Shell Script and Display Notification in order. In the Run Shell Script change Pass input to as arguments add the following code to that step.

PATH="$PATH:/usr/local/bin"
for var in "$@"
do
convert "$var" -resize 400% -type Grayscale - | tesseract -l eng - - | pbcopy 
done

As an alternate the following code will convert the entire result to a single line of text if that is preferred, but it may cause issues if there are columns of text on the image.

PATH="$PATH:/usr/local/bin"
for var in "$@"
do
text=`convert "$var" -resize 400% -type Grayscale - | tesseract -l eng - -`
ocr="$orc$text"
done
echo $ocr | pbcopy

Next add some text to the notification step so you know when the task is done processing. I added a title of "ORC Finished" and a message if "Text was copied to your clipboard.". Then just save the application and give it a name.

Using the Applet

To use the applet first fine it and d rag it down to your Dock to make a shortcut. Then you can drag image files onto the application in the Dock and it will do it's magic. Once you drop an image onto the Applet it will take a few seconds to process and you should see the notification pop up when it is done in the corner. At this point you can past the resulting text into a program of your choice, clean it up, and so what you want with it.

Other things that can be done with a bit of tweaking to the above scripts:

Processing multiple input files at once.
Saving results to a text file on the desktop or source folder instead of the clipboard.
Opening the resulting file automatically.
Remove the notification step if desired.
Create a Folder Action instead to automatically run on files added to a specified folder.
Advanced tesseract options can be passed in the script but in my experience these were not needed.

Bookmarklet to Post to Slack from your Bookmarks Bar

2015-12-04T15:36:00.001-05:00

I use Slack, and one think we love to do is share new articles and discuss them. But the good old copy and past the link was annoying me, so I created a Bookmarklet that will share your current page to a selected channel.

Just go to the following page and follow the instructions to build your own widget. http://68.40.119.10/slack.html

For peace of mind a brief overview of how this works: This bookmarklet is purely javascript and no data is posted to any other servers besides Slack's Webhook to receive the post. The tool linked above is purely a javascript tool to build and customize the widget. It will ask for your Webhook URL but it does not post or track anything and simply embedded that URL in the widget it present to you. The fact that it "remembers" your settings is simply done by updating the URL with a search hash which also isn't sent to the server.

Fix for Pillars of Eternity Blank Screen on OS X

2015-10-19T20:46:00.000-04:00

Here is a quick fix for the blank/non-responsive screen in when going full screen in Pillars of Eternity on OS X (El Capitan). Note that you will have to do this before you run the game each time as it will reset the settings with bad ones each launch.

First open up the game and take it out of Full Screen mode if it in that mode.

Press Command+F to take it out of Full Screen.
Exit the game or press Command+Q.

Press Command+Space to start Spotlight Search.
Start typing Terminal and launch the Terminal application.
Copy the following text as a single line into your clipboard.

defaults write "unity.Obsidian Entertainment.Pillars of Eternity"  "Screenmanager Is Fullscreen mode" 0 && defaults write "unity.Obsidian Entertainment.Pillars of Eternity"  "Screenmanager Resolution Width" 2560 && defaults write "unity.Obsidian Entertainment.Pillars of Eternity"  "Screenmanager Resolution Height" 1440

Click over to the Terminal window and Past the copied text.
Press Enter to runt he command.
Launch the game as normal.

Somewhat inconvenient granted; you could make a script to do it but I don't want to post any executable code online. Also this above code will not require a password or generate and output when executed even though it works.

Also, you may need to change the width(2560) and height(1440) if your native resolution of your monitor is different.

Making Time Based Graphs with Chartist

2015-10-08T15:51:00.000-04:00

I have been implementing some custom charts with Chartist recently. Overall things have gone very well with some simple tweaks and fiddling to get things exactly how I want. It is very nice to be be able to easily adjust the charts with CSS right in the normal site code.

The only problem I had was using Date based charts that don't have fixed intervals. If the Dates or Times are at a fixed interval you can just write the values directly to the labels variable and format them how you want very easily. If you have a random or variable set of dates, say when inventory is logged which is done at a random, there is no easy way to graph this and keep the relative positions correct on the graph. I don't want all the points one unit apart, as one could be two weeks and the next 2 months, but rather properly positioned based on the date provided.

Anyway, I got it to work and where is a brief example/sample code to try out.
As you can see in the sample here the items are positioned properly.

Problems with Java Based OS X Applications Hanging when Entering Text

2015-10-02T14:28:00.002-04:00

So I found another problem with my new upgrade to OS X 10.11. That is that the Java web application I was using, Adobe Ad Hoc Analysis in this case, would hard lock every time i tried to enter text in a text box. The only solution was to force quit the application. After much testing I found the culprit was the RescueTime helper applet. Basically this applet gets Accessibility access so that it can inspect windows and record time used but is glitching on the java windows and causing the lock up.

There could be other programs causing the same issues even if you don't use RescueTime but are having the same problem. To troubleshoot do the following.

Close out of the java program that is locking up.
Open System Preference > Security & Privacy > Privacy > Accessibility.
Note with items are check and then uncheck them all.
Restart your java program and attempt to reproduce the issue.
If it is working correctly then quit, re-enable one item, and try again until you find the culprit.
If you can't get around the error even with all Accessibility options off that you must have some other issue.

Anyway if you find any items fix the problem you can just leave them off though they might hinder there applications functionality.

OSX 10.11 Stuck On Grey Screen After Login

2015-10-02T09:29:00.000-04:00

After upgrading to El Capitan yesterday I ran into an issue where my computer would boot up going through the loading bar and display the login screen if enabled. Upon logging in, or automatic login, the system would hang at a grey screen. After some testing with the network I figured out that it was not actually locking up and would respond to traffic.

To determine if your issues is due to the same problem wither SSH into the computer once booted if remote login was enabled or boot to Recovery Mode and launch the Terminal. Then run the following to see recently logged errors.

tail /var/log/system.log | grep 'Segmentation fault'

Or if using the Recovery Mode terminal the following thought the drive name may be different.

tail /Volumes/Macintosh\ HD/var/log/system.log | grep 'Segmentation fault'

If either of these commands produce any results that you have found your problem. The culprit is Unsigned Kernel Extensions or .kext modules. In the good old days developers could just install these as needed to add code level kernel functionality but no for improved security OS X requires that all Kernel Extensions be signed by the developer. This is overall a good thing because these are privileged and could easily lead to malicious behavior by untrusted sources.

Anyway the problem where is that the upgrade didn't disable the unsigned Kernel Extensions it found and instead just doest't load them. This is fine and normally it would just mean that whatever applications depend on them would jus not work until updated but for whatever reason there are a few special cases that cause the login process to hang. I ended up removing all the Virtual Box and Soundflower modules and then was able to boot without issue. I have not tried to Re-install or upgrade wither yet but atlas my computer boots.

To remove these again go to the Terminal and do the following. You will nee to authorize them with your administrative password.

sudo rm -Rf /System/Library/Extensions/Waco*
sudo rm -Rf /Library/Extensions/VBox*
sudo rm -Rf /Library/Application\ Support/VirtualBox/*
sudo rm -Rf /System/Library/Extensions/Soundflower*
sudo rm -Rf /var/folders/*

Again note the additional prefix if using the Recovery Mode ant that thedick name may be different.

sudo rm -Rf /Volumes/Macintosh\ HD/System/Library/Extensions/Waco*
sudo rm -Rf /Volumes/Macintosh\ HD/Library/Extensions/VBox*
sudo rm -Rf /Volumes/Macintosh\ HD/Library/Application\ Support/VirtualBox/*
sudo rm -Rf /Volumes/Macintosh\ HD/System/Library/Extensions/Soundflower*
sudo rm -Rf /Volumes/Macintosh\ HD/var/folders/*

Finally just reboot and enjoy.

Raspberry Pi B Crashing when Running on SD Card

2015-09-29T13:25:00.002-04:00

I have been having surtax crashes with my Raspberry Pi B for a few months now and it is a very irritating intermittent issue. The unit basically locks up and even on resetting the power it won't always turn back on, but rather just sit there with no lights and no display and I end of having to cycle the power 2-5 times to get it to start again.

Anyway, I finally figured out there it appears to be an issues with the SD card. At first I thought that it was just an off brand card and the Raspberry Pi just didn't like it. But after some testing and hands on trials I figured out that it appeared to be an actual problem with the electrical connection and even though the card seamed seated properly to poor connection would cause the card to be or become unreadable. This was easy to verify by lightly pressing on the SD card and then turning on the Raspberry PI. It turned right on and worked perfectly and as soon as I let go it died again. I can't see any physical defects and the card looks to be seated and pressing firmly into the contacts but I guess it's not sufficient.

I found a simple solution, a simple light pressure chip clip/close pin on the SD-card solved the problem perfectly. I added a bit of cloth above the SD card and ensured that the clasp was pressing just above the SD Card's contact points and placed it over the end of the Raspberry PI and it works like a charm.

XBian RC2 ASS Subtitles not Displaying in MKV Videos

2015-09-16T14:21:00.002-04:00

Minor issues I found when updating to the current release of XBian on my Raspberry Pi B. Namely that subtitles would not work from MKV videos. Heres a simple fix for the issue. SSH into the xbian and run the following:

sudo apt-get install libass5

Drag and Drop in Umbraco 7

2015-07-31T11:15:00.003-04:00

This code will let you add drag and drop logic to Umbraco v7 using the new AngularJS interface. It doesn't require and backend code updates but can't be created as a package because there is no way to update the umbTree directive as needed. There are only two files that need to be changed on the client. Note that updates to umbraco will overwrite these changes. Also note that due to lack of support in the underlying jQuery-sortable plugin for touch events this will not work on tablet devices.

First a set of changes for the umbTree and umbTreeItem directives in umbraco/Js/umbraco.directives.js.

This patch was generated on 7.2.8 so may not work on other versions automatically. You can manually apply all the listed changes on any 7.* version and they should work as expected with no changes needed.

Secondly, to allow for dragging items to collapsed nodes or nodes they don't already have children, the following needs to be added to the end of the line in umbraco/assets/css/umbraco.css.

For Each to loop over HttpFileCollection Files

2015-05-29T10:41:00.003-04:00

Recently I was working with file uploads in .NET and found that theres no clean way to loop over each uploaded file. Here is a extension method for easily looping through the list and it allows filtering by field name and applying Regex to the file name.

Samples:

Source Code

Convert .htaccess deny rules to web.config

2015-05-22T11:02:00.000-04:00

Recently I was looking for a easy way to convert a large list of spammer and hacker IP Addresses from Apache .htaccess allow/deny rules into a format I could use on our IIS Server. I found that there is really no good way to do this but after some work and a bit of coding here is a solution.

This solution will allow you to take a large list of access rules and apply them to an IIS website. Additionally this is done in a file so there is no need to manually add records.

Collect your List of Rules

I used the following as a general blacklist of malicious and spammer client's to block. There are other sources or you may have your own list. Compile all the lists into one file and don't worry about duplicates, the script later on will resolve those.

Convert the Rules to Web.config Equivalents

I have created a simple javascript tool to convert the above lists into web.config rules. To do this go here and enter all your rules into the first text box. Make sure you supply all rules at once so that they can be de-duped. If the list does end up having duplicates then IIS will not start up properly.

Ensure you have the Features Installed

Next log onto your server and make sure that you have the Web Server > Security > IP and Domain Restrictions role installed. You may need to add it as it's not selected by default.

Apply your new Rules

Finally open up the web.config files for each of your main sites and add the generated text to the system.webServer/security/ipSecurity section. Save and reload the website to ensure it's working.