John Veldboom

Laravel 4.2 updated_at Table Field Always Updating

Fri, 18 Sep 2015 00:00:00 +0000

Here's the fix to the issue if you find the "updated_at" field in your database table always being updated even though the values do not actually change.

The "issue" is found within the getDirty() function within Eloquent\Model. It's caused by the way it compares the current value with the value being inserted/updated. The comparison is type sensative which can cause problems for certain values. Here's a github issue explaining the reasoning behind this from Taylor Otwell.

The solution we decided on was to ensure the value types being inserted and updated match what was previously there by forcing integers or doubles. It will also require you to use 1 or 0 instead of true or false for boolean values.

$int = 235;
// force it to be integer
$int = (int) 235;

$bool = true;
// becomes 
$bool = 1;

AWS CLI Ansible Playbook on Ubuntu 14.04

Thu, 30 Jul 2015 00:00:00 +0000

Here's a simple Ansible playbook that installs the AWS CLI and all the required dependencies on Ubuntu 14.04 - should work on other versions of Ubuntu too.

Setup

Clone or download the jveldboom/ansible-aws-cli repo to your Ansible machine
Add your host information within inventories/hosts
Add your AWS creditentials within vars/vars.yml

Run Playbook

Below is how I prefer to run Ansible playbooks since it allows you to choose the host and SSH user dynamically.

ansible-playbook -i inventories/hosts playbook.yml --extra-vars="hosts=vagrant user=vagrant" --ask-pass

The `--extra-vars` parameter sets the "host" and "user" variable in the `playbook` file. And `--ask-pass` will prompt you for a password.

GoAccess Automated Reports - Last 30+ Days via Cron

Sat, 30 May 2015 00:00:00 +0000

First if you're not familier with GoAccess, here's a quick description from their website http://goaccess.io/:

GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems. It provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly.

Typically you run goaccess on a single log file like this: goaccess -f access.log. But in our case we wanted to run it on multiple log files for a month end report. This provided some issues since we needed to be able to pass multiple files and only report within a date range.

Below is the script we came up with that's not perfect, but it's simple as does a pretty good job of what we needed. It finds all the gzipped access.log files modified within the past 35 days and pipes them into goaccess. We chose 35 days since it's possible some files may contain multiple days so 35 files should always include at least 30 days. Finally we save the report as an HTML file by the date. monthly-2015-05.html.

#!/bin/bash
DATE=$(date +'%Y.%m')
zcat `find /var/log/apache2/ -name "access.log.*.gz" -mtime -35` | goaccess > /dir/monthly-$DATE.html

Then just save the file and add the cron job to run at midnight on the first of each month.

# as a shell script
00 00 01 * * /bin/bash /dir/goaccess-monthly.sh

# or as a single cron job line
00 00 01 * *  zcat `find /var/log/apache2/ -name "access.log.*.gz" -mtime -35` | goaccess > /dir/monthly-$(date +'%Y.%m').html

Also check out the man page for more information on various options and settings.

EC2 High CPU Wait and the EBS Provisioned IOPS Difference

Mon, 06 Apr 2015 00:00:00 +0000

On our small Graphite monitoring server (c4.large), we kept having very high CPU wait times. It would hover around 80% almost continuously. I could scrub back in the graph timeline to see where it started and there were no major changes made on that day. (like adding a fleet of new servers or extra data points) So I was pretty confident it was not an application issue.

Turns out the issue was caused by our server running on a General Purpose SSD and it was hitting the max input/output per sec (IOPS) limit causing requests to go into a general pool.

So now onto switch over to Provisioned IOPS SSD, but first we need to determine the IOPS we need.

Within your AWS console under the Elastic Block Store (EBS), you can monitor each volumes performance. So we're going to take the Read Throughput and Write Throughput to determine our IOPS need.

For example from the graph above, we'll round the Read Throughput up to 1 and the Write Throughput maxes out at 100. So in theory we could set the IOPS to 101 and be "ok", but we'll likely want to give a little more head room and round up to at least 200 or even 300 to be safe.

After changing our volume to use the provisioned SSD, we saw an immediate difference in the CPU wait.

To convert an existing General Purpose SSD (or magnetic too) to a Provisioned IOPS, you'll need to complete a couple steps.

Stop EC2 instances - optional but will prevent any data lose
Create snapshot of volume you're changing
Convert snapshot into a Provisioned IOPS volume.
Detach old non-provisioned volume from EC2 instance
Attached new provisioned volume
Start EC2 instance

Heka JSON Decoder using a SandboxDecoder and Lua

Wed, 03 Sep 2014 00:00:00 +0000

First let me say if you're looking for help on Heka, check out their IRC channel. It's full of great guys that are extremely helpful! [IRC: #heka on irc.mozilla.org]

This Heka JSON encoder converts any simple key/value JSON payload into Heka fields.

// sample json payload
[{"name"=>"John Doe","title"=>"Sysadmin","@timestamp"=>"2014-09-02T22:10:28Z"}]

I hope to have a sample use case where we used this with a PHP application to import data into ElasticSearch. Pretty cool stuff - and super easy to setup with new and (in our case) legacy systems.

Drop MySQL Primary Key with Foreign Key using Laravel 4 (error 1025 & 105)

Fri, 21 Feb 2014 00:00:00 +0000

This issue is really not specifically related to Laravel. But the code below shows how to handle the issue within Laravel 4.

The issue comes from trying to delete a primary key that's also a foreign key. MySQL would spit out the following "useful" error:

SQLSTATE[HY000]: General error: 1025 Error on rename of './database/#sql-10e9_9c' to './database/table' (errno: 150) (SQL: alter table `table` drop primary key)

To solve this you just need to delete the foreign key first and then the primary key.

Schema::table('products_fulltext', function(Blueprint $table) {
   $table->dropForeign('table_field_foreign');
   $table->dropPrimary('PRIMARY');
});

Laravel 4 Unable to Read Package Configuration File

Sat, 11 Jan 2014 00:00:00 +0000

I was trying to add a configuration file to an existing Laravel 4 package (revisionable) to help improve the functionality. But no matter what I tried I could not get the package to read from the src/config/config.php file.

Turns out the issue was caused by the package not having a service provider. I'm not 100% why this made a difference but by adding the service provider file and adding it to the service providers within app/config/app, the config would read.

Here's the service provider that actually made it work - specifically the $this->package('venturecraft/revisionable'); code within the register() function

<?php
namespace VenturecraftRevisionable;

use IlluminateSupportServiceProvider;

class RevisionableServiceProvider extends ServiceProvider {

	/**
	 * Indicates if loading of the provider is deferred.
	 *
	 * @var bool
	 */
	protected $defer = false;

	/**
	 * Register the service provider.
	 *
	 * @return void
	 */
	public function register()
	{
		$this->package('venturecraft/revisionable');
	}

	/**
	 * Get the services provided by the provider.
	 *
	 * @return array
	 */
	public function provides()
	{
		return array();
	}
}

Then to read the config file, you can just use:

Config::get('revisionable::file.key');

// or if you only have one config file named config.php
Config::get('revisionable::key');

I'd love to know exactly why it works like this so please feel free to shed your Laravel knowledge in the comments below.

Thanks to Zennon Gosalvez's article which helped lead me in the right direction.

How to create a temporary storage directory that automatically deletes contents after X days (Mac)

Thu, 14 Nov 2013 00:00:00 +0000

Here's a quick way to have a directory that allows you to store files and other subdirectories for temporary usage. For example, I use this to save all my downloads and other files that I only need for the next day or two.

We first need to create a new cron job. If you're not familier with cron, it's basically instructions to the cron daemon of the general form: "run this command at this time on this date". (or for some light reading) We're going to use cron to run a command to move all the contents of a directory to the trash.

// from the terminal enter to create new crontab or open existing
crontab -e

// next press Esc+i to enter into "INSERT" mode which will allow you enter text
00  */2  *  *  *  find /path/to/temp/ -mtime +1 -exec mv {} ~/.Trash ; >/dev/null 2>&1

// then save the crontab by pressing Esc, :, w, q (write & quit)

Now let's explain what's going on with the cron job.

00	minutes (ie: 2:00,18:00)
*/2	every 2 hours
* * *	every hour, day of the month, day of the week
find /path/to/temp/	finds all contents of directory
-mtime +1	where the file's make time is X days more than current date
-exec mv {}	take all contents found from find comment and executes the mv (moves) command
~/.Trash ;	users trash can (could be any directory though)
>/dev/null 2>&1	suppresses any output from displaying (ie fulling up your users mailbox)

Also, if you're looking for a quick way to add extra storage to your Macbook, checkout the Nifty Drives. This is what I use for my temporary storage.

PHP session_start() failed no space left on device (Plesk plesk-php-cleanuper)

Wed, 16 Oct 2013 00:00:00 +0000

I kept receiving intermissive PHP warnings saying some thing like E_WARNING: session_start(): open(/var/lib/php/session/sess_ji9k4chqke3pde98a5n5m1vca5, O_RDWR) failed: No space left on device (28). Usually this would indicate that the disk where the sessions were being stored is full. The best place to start is to check if the disk where those session files are being stored is in fact NOT full.

// to check disk space usage
df -h

// should display something like this
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              4.0G  883M  3.2G  22% /
/dev/mapper/vg00-usr  4.0G  1.8G  2.0G  49% /usr
/dev/mapper/vg00-var  890G  231G  614G  28% /var
none                  5.9G  116K  5.9G   1% /tmp

In my case the disk where the sessions were being stored had plenty of space - 600GB free!

It turns out the issue was caused by Plesk's PHP hourly script /etc/cron.hourly/plesk-php-cleanuper was not able to finish due to the current sessions directly being too full. My guess is some where in the update from Plesk 10.4 to 11 the script got turned off or removed. Then a minor update turned it back on. But during that time the sessions folder grew to around 800MB of tiny session files. My guess is the average session file is less than 1 Kb. So there must have been over a million small session files. The Plesk script plesk-php-cleanuper when turned back on could never finish clearing the old session files.

tl;dr - now to the fix

# 1. Gracefully turn off Apache so no incoming request come during these changes.
apachectl -k graceful-stop

# 2. Rename the current PHP session directory
mv /var/lib/php/session /var/lib/php/session.old

# 3. Recreate PHP session directory and set permissions
mkdir /var/lib/php/session
chmod 1777 /var/lib/php/session

#4. Start Apache
apachectl -k start
 
# 5. Delete old session files (optional)*
mkdir /var/lib/php/empty
rsync -a --delete /var/lib/php/empty/ /var/lib/php/session.old/

Step 5 is optional since the server I was working on I left the rsync command run for around 24 hours and it still did not finish. The rsync will eat up your disk I/O and this was a production server so I could not have it run slow for that long - plus disk space was not really an issue. But if you need that disk space and can allow your server to run slow for a day or two, the rsync method is the fastest way to delete millions of files.

Authenticate Only Certain HTTP Verbs (POST, PUT, DELETE) in Laravel 4

Sun, 06 Oct 2013 00:00:00 +0000

In working on the API for an upcoming project, I needed a way to allow all reads (GET) to be allowed without authentication but any requests (POST, PUT, or DELETE) changing the data to require authentication.

I wanted to use the Route::group() to help keep the routes.php file simple by grouping all the API calls together. I also wanted to use the Route::resource to keep the controller relatively simple too.

// router.php
Route::group(array('prefix' => 'api/v1'), function()
{
    Route::resource('tweets', 'ApiTweetsController');
}

Next within the controller I added the following to the __construct() class to require auth.basic authentication on all functions except 'index' and 'show'. Since we're using a resource route, the 'index' and 'show' function all represent HTTP GET requests. So any POST, PUT or DELETE request will require authentication - which is exactly what we're looking for!

class ApiTweetsController extends BaseController {

   public function __construct()
   {
      $this->beforeFilter('auth.basic', array('except' => array('index','show')));
   }

   ....

Note: you could also add 'edit' and 'create' to the 'except' array but I did not need those methods for the API I was working on. array('except' => array('index','show','edit','create')).

Update: If you wanted to use the actual HTTP verbs, you can use the following instead. Thanks ericbarnes for pointing this out.

$this->beforeFilter('auth.basic', array('on' => array('post','put','patch','delete')));

Please let me know if you have any questions about this or if you're found a different/better way of doing this.