The “old” way to do this is to create a bash script in order to pull down the source code, run ./configure and then make; make install. But this is 2013, we have a much more civilized way to create this package now, so that you can distribute that RPM across multiple nodes or even include it in your own yum repository.

We’re going to use FPM (Effing Package Manager) to do the job for this. It is a little bit confusing because PHP also includes the acronym FPM for FastCGI Process Manager. Our package manager FPM is a ruby gem, so install the prerequisites first (CentOS 6 directions here), along with fpm:

To start, compile PHP ...]]> So you have a custom PHP binary that you wish to distribute across multiple servers.

The “old” way to do this is to create a bash script in order to pull down the source code, run ./configure and then make; make install. But this is 2013, we have a much more civilized way to create this package now, so that you can distribute that RPM across multiple nodes or even include it in your own yum repository.

We’re going to use FPM (Effing Package Manager) to do the job for this. It is a little bit confusing because PHP also includes the acronym FPM for FastCGI Process Manager. Our package manager FPM is a ruby gem, so install the prerequisites first (CentOS 6 directions here), along with fpm:

To start, compile PHP the way you normally would, ie:

Then make it as normal.

Now instead of running make install like normal, we tell it to output to another directory, just some temporary directory on the server:

Now PHP is happily installed in /tmp/installdir. Which is great, but we can’t use it until it we package it up for regular installation. Run the FPM command to pack it all up:

What do the options mean?

I’ll go through these options:

• -s dir: source is a directory
• -t rpm: we are creating an rpm
• -n php-fpm: we are creating a package with the name php-fpm
• -v 5.3.25: the version of the rpm (so you can manage upgrades)
• -C /tmp/installdir: the base location to take the files from
• -p php-fpm_VERSION_ARCH.rpm: fpm will replace the VERSION and ARCH macros automatically with your version number and architecture
• -d openssl -s pcre …: these are all of the rpm dependencies that you’ll want on the machines installing the package. If you don’t include the proper ones (to match your source system, and which are included in ./configure, you’ll get library errors when you try to launch the program.
• usr/bin usr/share/man/man1 usr/sbin etc usr/share/man/man8 usr/share/fpm usr/lib/build usr/include/php: the directories you wish to include in your package file

When all is done, you should have your rpm file located in /tmp/installdir. You can install it directly on any system with rpm -i <rpm name>

Before you package up your files, you can include any other files you want. For example, you could include the init script or php.ini file. But if you are managing the system using puppet or chef, it would be better to manage this configuration file through there rather than from the rpm file. This works really well as a ‘building block’ for your programs when you add these custom rpms into your own repository. That way, you can have your system install custom versions of the binaries where applicable but still have the niceties that come from an systems automation with puppet or chef.

Enjoy!

To ensure we have a secure infrastructure we wanted to make sure that the rpms are correctly signed so that the gpg check can be completed on them when installed.

First, generate the gpg public/private key pair:

Fill out the required information. The entropy generation step may take a long time to complete.

Export a text version of the public key with this command, for later use in your repository:

fpm docs state that you just need to add the –rpm-sign to your fpm command. ...]]> I’ve been using the excellent fpm (Effing package manager!) tool for automatically generating rpms from source (for example, creating a custom compiled php-fpm binary and then wrapping it for install in an rpm for distribution from our own repository). FPM also creates .debs and other binaries as you need them.

To ensure we have a secure infrastructure we wanted to make sure that the rpms are correctly signed so that the gpg check can be completed on them when installed.

First, generate the gpg public/private key pair:

Fill out the required information. The entropy generation step may take a long time to complete.

Export a text version of the public key with this command, for later use in your repository:

fpm docs state that you just need to add the –rpm-sign to your fpm command. However, this generated the following error for me:

The reason for this is that it uses the settings located in your ~/.rpmmacros file. Find your key ID:

In this sample, the KEY_ID would be “94E8C1F6″

From here, the fpm command runs without issue.

This is an alias of the rpm --addsign command.

Now let’s check the signature on this rpm.

Whoops, it doesn’t seem to be signed properly. Well, that is because the rpm system doesn’t know to trust this GPG key yet. Import with the following command:

Note that this is the plain text file from before. Your RPM system knows to trust this key, and now if you verify the file:

This RPM is now ready for use in a repo. In the .repo file, use the following to ensure it follows the correct signature:

All key names have been changed to protect the innocent.

Rather than using

to count numbers in netstat, which can take an extraordinary amount of time on busy servers, I’ve used /proc/net/sockstat to get this data.

Add the following parameters to zabbix_agentd.conf:

Then you can either manually import these UserParameters into ...]]> There is a very easy way to track opened TCP connections in Zabbix. Although it does not come built in to the Zabbix agent, it is simple enough to get. This method will allow you track the open number of TCP and UDP connections, as well as track TCP connections in TIME_WAIT.

Rather than using

to count numbers in netstat, which can take an extraordinary amount of time on busy servers, I’ve used /proc/net/sockstat to get this data.

Add the following parameters to zabbix_agentd.conf:

Then you can either manually import these UserParameters into Zabbix, or use this template (if you’re using Zabbix 2.x).

Download the sockstat Zabbix template

It includes the items and a graph for the socket stats. Enjoy!

One of these programs is the eminently useful rsync. It states how to do this in the man file, but it is hidden and non-obvious. You simply add the option via the -e command to pass ssh options. -e is the shell rsync uses to connect to a remote host.

Or with both an alternate port and openssh key:

This can be used in a bash script if you set the $port variable or directly ...]]> Best practices state that you should run ssh on an non-standard port. Unfortunately some programs use port 22 by default and it isn’t obvious what the switch is to change this port.

One of these programs is the eminently useful rsync. It states how to do this in the man file, but it is hidden and non-obvious. You simply add the option via the -e command to pass ssh options. -e is the shell rsync uses to connect to a remote host.

Using rsync on an alternate ssh port

Using rsync with an openssh key

Or with both an alternate port and openssh key:

This can be used in a bash script if you set the $port variable or directly on the command line by using the set port instead of $port.

Here are some other useful pages regarding using rsync. If you don’t use rsync for remote file transfer – I highly recommend it. The transfer speed is much faster than scp over ssh.

• rsync command examples
• rsync custom commands

The excellent tcpdump utiliy comes to the rescue here. I recommend you get more familiar with the tcpdump man page. Here is the command you can use, in a nutshell:

or alternatively with just tcpdump (Thanks Chris!):

Your command line will print out all traffic exiting your server from port 80, headers and all. Lets look at the options in more detail.

We had a few concerns about this. Firstly, we were worried that unsuspecting users would enter their login information on this site. Also, they actually were showing up in Google search engine results, which both hurts our brand, ranking and perception by users (this site loaded much slower than ours obviously).

Determined to put a stop to it, the ...]]> A few months ago, we had an interesting issue with another website stealing our content in an unusual way. Essentially they ran a proxy service on a similar domain (a .com.ar instead of .com) and then replicated our site, replacing our ads with their own. They also ran our content through a content replacement algorithm, removing certain pieces of text and also replacing our own domain with theirs. It is fairly easy to do.

We had a few concerns about this. Firstly, we were worried that unsuspecting users would enter their login information on this site. Also, they actually were showing up in Google search engine results, which both hurts our brand, ranking and perception by users (this site loaded much slower than ours obviously).

First Attempt To Stop The Proxy Spammers: IP Blocking

Determined to put a stop to it, the first method we tried was blocking their crawler IPs. However, they owned enough IPs in enough ranges that it was simply a cat and mouse game. We would block them, and then a day later they would be back up. After a few weeks of this it didn’t seem to be a viable long-term solution for blocking these spammers.

Next, We Used Their Mirroring Of Our Content Against Them

Next I thought I would use the fact that they were modifying our page content and disable their site that way.

I created a box in html/css which would trigger via JavaScript only when loading from their domain. But, you wouldn’t think it would be that easy would you? They replace our domain with theirs, modifying any mention of their own domain. So I used a quick hash function to create a unique identifier from the loaded domain and then matched against that.

Also, figuring they would just take out the HTML box I created, I also found it useful to display a hex-encoded version of it. You can encode text to the hex-escaped version for Javascript with the following command line:

The final Javascript I started loading on our site (and therefore their site) is below:

After thoroughly testing and then putting up the code, we launched it live. It was very satisfying knowing that end-users were getting the warning message when visiting their site, and the actually reported their experience back to us. After about 12 hours, they figured out our play out and turned off their proxy. Success!

…

Not quite. The next morning I woke up to see their site back up. Looking into their code, they had removed the JS completely. I changed our code, embedded it in a JS file, and used other creative means to get it back up – but in the end they were just able to disable the Javascript entirely and defeat this attack. It was just another cat-and-mouse game.

We also tried DMCA/Abuse Contacts

We also tried to send notices to their hosting company to get them to take down the site. They were hosted in Argentina, so DMCA is not applicable. Their abuse contacts were non-responsive.

Originally we thought we could go after their domain registration since it violated our trademark – but this is a long and involved process involving lots of paperwork and time, and for another $10 they could just register another domain name. We didn’t think this was a viable option.

Going to the Source: Google

Google is involved in this scam in a number of ways. First, they were indexing and serving his site in their search results. Secondly the scammers were replacing our ads with their own Google AdSense ads, which I am sure is a ToS violation.

While we were attempting to defeat their site from a technical perspective, we also began looking at Google to see what we could do from there.

Google has a process for submitting DMCA requests. The issue in this case is that they make you submit one batch at a time, and with millions of pages on our site and indexed in Google, it just doesn’t make sense to list out urls line by line. It worked to remove those submitted results from search engine results, but it was like cutting grass with scissors. Finally, I attempted to contact Matt Cutts via Twitter:

Thanks. So much for that venue. I know @mattcutts is the public face of Web Spam at Google so I’m sure that he gets lots of @’s with dumb questions, but we were already way beyond this.

Finally, What Worked

Members of our site had a few personal contacts at Google both in an AdSense representative and otherwise. One of our contacts with Google was able to bring this issue up with the right people and they finally took the offending domain out of the SERPs permanently. We also reported their AdSense account – but we don’t know what happened with that. Without them showing up in SERPs, it was a moot point because they won’t be getting many visitors any more.

The Problem With Google

Google has become so large that it is almost impossible to get a situation like this taken care of without knowing someone who works for them. They have many automated systems in place, but scammers will continue to utilize loopholes for their own profit. Google enables this type of scam, yet they also profit from it.

I wish Google would have some sort of Ombudsman or review system set up so that someone like us, who is having our content ripped off by others using Google’s own tools (and with Google taking a percentage of profit from these people), has a way to efficiently deal with them without resorting to personal contacts. We spent much time on this, time that could have been put to better use.

Or maybe personal contacts are the only real way to deal with a situation like this?

Anyway – I am welcoming comments and any other ideas for dealing with these Proxy Hijackers and how to keep them offline. I’m curious how widespread this type of incident is, we know of only one other site that was having the same issue from the same scammer.

After all, they can always get another domain for $10.

dd stands for data description and is used for copying data sources.

A simple command to do real-world disk write test in linux is:

This creates a file named ‘test’ with all zeroes in it. The flag conv=fdatasync tells dd to sync the write to disk before it exits. Without this flag, dd will perform the write but some of it will remain in memory, not giving you an accurate picture of the true write performance of the disk.

A sample of the run is below, with a simple SATA disk:

Now, there is a major caveat for using dd for ...]]> A great way to do a real-world disk test on your linux system is with a program called dd.

dd stands for data description and is used for copying data sources.

A simple command to do real-world disk write test in linux is:

This creates a file named ‘test’ with all zeroes in it. The flag conv=fdatasync tells dd to sync the write to disk before it exits. Without this flag, dd will perform the write but some of it will remain in memory, not giving you an accurate picture of the true write performance of the disk.

A sample of the run is below, with a simple SATA disk:

Now, there is a major caveat for using dd for disk benchmarking. The first is that it only tests filesystem access. Depending on your filesystem (I’m looking at your ZFS) the file write may itself just load into memory for writing later down the road. The same with a RAID controller on the system.

A much more accurate way of performing a disk benchmark is to use tools specifically geared towards this task. It will write much more data over a longer period of time. Bonnie++ is a particularly useful tool for this purpose.

Now don’t forget to remove that test file.

After using one of my tried and true Apache configuration files, I received this error on start:

Common wisdom would imply that I should make sure the authz_host module is installed (LoadModule authz_host_module modules/mod_authz_host.so), however this just was not working.

Finally, I discovered that the Order command has been removed from Apache 2.4! According to the upgrade notes for Apache 2.4:

In 2.2, access control based on client hostname, IP address, and other characteristics of client requests was done using the directives Order, Allow, ...]]> The new Apache 2.4 has been released a few weeks ago and I decided to use this version while installing a new server (I compiled it from scratch rather that using an rpm or deb).

After using one of my tried and true Apache configuration files, I received this error on start:

Common wisdom would imply that I should make sure the authz_host module is installed (LoadModule authz_host_module modules/mod_authz_host.so), however this just was not working.

Finally, I discovered that the Order command has been removed from Apache 2.4! According to the upgrade notes for Apache 2.4:

In 2.2, access control based on client hostname, IP address, and other characteristics of client requests was done using the directives Order, Allow, Deny, and Satisfy.

In 2.4, such access control is done in the same way as other authorization checks, using the new module mod_authz_host. The old access control idioms should be replaced by the new authentication mechanisms, although for compatibility with old configurations, the new module mod_access_compat is provided.

Basically, the Order command is depreciated.

In my case, I replaced the lines:

with:

Also make sure both of these modules are loaded:

Easy enough, but just be aware that there are several configuration changes between 2.2 and 2.4 which render your old Apache configuration files unusable.

The FCC currently has a ban in place on the encryption of the “Basic” level of cable service. This includes such channels as ABC, NBC, Fox, PBS, etc. It is a good way for low-income or budget conscious consumer to buy the basic level of service if they can not receive these channels over the air, possibly because of interference or distance to broadcasting towers. You do not currently need to rent a cable box (aka decryption device) to see ...]]> For those not familiar with the current state of digital television, cable providers send signals to your house in a format called QAM. This comes in two flavors, Encrypted and Unencrypted formats. Encryption is used to protect channel content from general viewership so that cable operators can sell these packages and/or individual channels based on a decryption device at the home.

The FCC currently has a ban in place on the encryption of the “Basic” level of cable service. This includes such channels as ABC, NBC, Fox, PBS, etc. It is a good way for low-income or budget conscious consumer to buy the basic level of service if they can not receive these channels over the air, possibly because of interference or distance to broadcasting towers. You do not currently need to rent a cable box (aka decryption device) to see these channels, ensuring broad access to these channels which often perform public service functions such as notification in case of emergency, carrying signals from the emergency broadcast system, or also community television stations.

This is a good thing. It allows citizens to purchase a very cheap (I paid $12.99/mo) cable plan to receive these essential stations without the purchase of any decryption boxes from the cable company (which they force you to rent from them, I might add).

The Cable Companies Want Your Money

Photo credit zeusandhera

The cable companies are, and have been, lobbying the FCC to remove this encryption ban. The FCC has a proposed plan of rule making for removing this ban on encryption. The big cable providers are lobbying for this ban removal on a couple of major points:

• Encrypting all channels will allow them to remain “hot” at the consumer end,
• It will reduce service technician calls because of the above,
• It will reduce or eliminate cable theft.

What they do not mention is that:

• You will need to buy a cable adapter (either box-top or Cablecard) for each TV in your home.

The cable companies are profit driven (this is OK) and, while I admit this may reduce service calls to your home, they also have other methods of being able to deliver this service “hot” including IP-based television distribution. This is simply a cheaper way for them to use their existing technologies to maximize their profits, while maintaining a facade that they are doing this for consumers’ benefit.

Based on my own television bill and the number of televisions that an average consumer has in their home (3), I estimate that an existing basic television consumer would receive a 92% increase to their cable bill for the same exact content. (Source)

Fighting The Good Fight

Several “new” television technology companies, such as Boxee and Hauppauge Computer Works (makers of Unencrypted QAM tuner cards) are fighting this rule change, along with the EFF, to allow consumers to continue to receive these basic television channels in the unencrypted format. While Boxee and Hauppage have their own profit motives, they are actively working to promote new technologies to bring a variety of TV content into your home.

Boxee and the National Cable & Telecommunications Association have been going tête à tête over the issue. Boxee is obviously making a play where its Boxee Box uses Unencrypted QAM to receive television stations while the NCTA represents the cable companies who want you to continue to pay your cable providers.

How to Weigh In

I urge all consumers of Cable TV to weigh in on the issue, but especially if you utilize Unencrypted QAM format to watch broadcast TV. The Cable Industry says that the number of consumers who use unencrypted QAM is negligible, so we need to show the FCC that we are in fact, not a negligible party to deal with.

The FCC is accepting comments on this proposed change to the rules.

Here is a listing of all comments on this proposed elimination of the ban on the encryption of the basic tier of service from the FCC.

The best way is to write a short letter to the FCC. Here is the letter that I wrote.

Then proceed to the FCC’s page for submitting a filing. For the proceeding number, use 11-169. Type in your information and attach the letter you wrote (I recommend sending as a PDF). After submitting, the FCC reviews the submission and places it on their website.

Together, we can fight the cable companies

The only way they know that we are not happy with this proposed rule change is by commenting on this to the organization who makes the rules, the FCC.

Please send them your thoughts! It should only take about 15 minutes of your time and you will feel great about participating in the rule making process.