We're currently undergoing an internal project to identify, consolidate and decommission aging equipment on our Network. It's funny, but I always find it hard to turn off equipment that has been running for 1.5 years without an issue.

srv01# uptime

 5:31PM  up 530 days, 10:11, 1 user, load averages: 0.46, 0.23, 0.09

 

people# uptime

 5:53PM  up 530 days, 14:30, 2 users, load averages: 0.00, 0.00, 0.00

 

inetops# uptime

 5:54PM  up 531 days, 11:45, 2 users, load averages: 0.00, 0.00, 0.00

Permalink | Leave a comment  »

That's how big the IPv4 routing table is now...

BGP table version is 111517930, main routing table version 111517930
378326 network entries using 45777446 bytes of memory
2328831 path entries using 121099212 bytes of memory
422707/66873 BGP path/bestpath attribute entries using 32125732 bytes of memory
189870 BGP AS-PATH entries using 5090228 bytes of memory
12651 BGP community entries using 1048178 bytes of memory
2 BGP extended community entries using 48 bytes of memory
29 BGP route-map cache entries using 928 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 205141772 total bytes of memory
1130746 received paths for inbound soft reconfiguration
BGP activity 4351207/3965251 prefixes, 55077482/52711615 paths, scan interval 60 secs

Permalink | Leave a comment  »

Setting up mod_proxy is becoming a more common task, especially with all the enhancements to the module of late such as mod_proxy_ajp. Setting up a Front End Apache server to proxy requests to say your Tomcat Application Server offers you a few things such as security and performance optimizations, however it is quite easy to make a small mistake and turn your web site into an open web proxy for the Internet to use anonymously.

Apache can be configured in both a forward and reverse proxy (also known as gateway) mode.

A typical usage of a forward proxy is to provide Internet access to internal clients that are otherwise restricted by a firewall. The forward proxy can also use caching (as provided by mod_cache) to reduce network usage.

The forward proxy is activated using the ProxyRequests directive. Because forward proxies allow clients to access arbitrary sites through your server and to hide their true origin, it is essential that you secure your server so that only authorized clients can access the proxy before activating a forward proxy.

A reverse proxy (or gateway), by contrast, appears to the client just like an ordinary web server. No special configuration on the client is necessary. The client makes ordinary requests for content in the name-space of the reverse proxy. The reverse proxy then decides where to send those requests, and returns the content as if it was itself the origin.

A typical usage of a reverse proxy is to provide Internet users access to a server that is behind a firewall. Reverse proxies can also be used to balance load among several back-end servers, or to provide caching for a slower back-end server. In addition, reverse proxies can be used simply to bring several servers into the same URL space.

Below is a sample configuration using Apache as a reverse proxy in front of a Tomcat Application Server using the Tomcat HTTP Connector.

    ProxyRequests Off
    ProxyPass / http://localhost:8080/
    ProxyPassReverse / http://localhost:8080/ 

The key here which is sometimes mistaken is ProxyRequests Off, some people think you need to turn this flag to use Mod Proxy at all. This is the mistake. By turning that flag on you are telling your Web Server to proxy requests from users to the resource they requested. 

A reverse proxy is activated using the ProxyPass directive or the [P] flag to the RewriteRule directive. It is not necessary to turn ProxyRequests on in order to configure a reverse proxy.

Want to test this, turn ProxyRequests On and restart your Apache Web Server. Now open your web browser and enable a proxy server using your website as the address and port 80. Now open http://blog.phyber.com as you can see the site loaded as expected. Check out your access logs on your Web Server for confirmation you are using your Web Server as a Proxy Server.

Other useful mod_proxy examples include load balancing to multiple backend hosts. In this example we will be proxying traffic to two backend servers using the cookie JSESSIONID for sticky sessions.

<Proxy balancer://mycluster>
    BalancerMember ajp://server01:8009 stickysession=JSESSIONID
    BalancerMember ajp://server02:8009 stickysession=JSESSIONID
</Proxy>

ProxyPass /webapp balancer://mycluster/webapp

Note in this configuration using mod_proxy_ajp a ProxyPassReverse is not needed as the AJP request includes the original host given to the proxy, and the application server can respond properly so no rewriting of the host request is required.

In the next example we will take use of mod_disk_cache to serve content and if it not found pass off to the request to a remote server behind the firewall with the exception of static content such as javascript, stylesheets and images which will be served locally from Apache itself.

<VirtualHost *:80>
    ServerName www.phyber.com
    DocumentRoot /opt/phyber/html

    CacheEnable disk /
    CacheRoot /var/cache/phyber
    CacheDefaultExpire 60
    CacheMaxExpire 3600

    ProxyPass /images !
    ProxyPass /stylesheets !
    ProxyPass /javascripts !
    ProxyPass / http://remote.phyber.com/
    ProxyPassReverse / http://remote.phyber.com/
</VirtualHost>

Note in this configuration we have ProxyPass /images ! the exclamation point tells mod_proxy not to proxy requests for that folder. So all requests for http://www.phyber.com/images/ will load directly from the local server from the folder /opt/phyber/html/images/. In this configuration mod_disk_cache is used first and if the requested object is not found in the cache the request is forwarded on to the server remote.phyber.com unless the object is from the static content directories excluded.

Look for furture articles on common mod_proxy usage and mod_cache here a http://blog.phyber.com

Permalink | Leave a comment  »

See the full gallery on Posterous

Permalink | Leave a comment  »

One difficulty when it comes to hosting multiple websites on single servers has always been SSL. By the nature of how the web works when you open a website in your browser a HTTP Header with the requested host is present. A web server can then read this information and direct traffic to the appropriate Virtual Host matching the Host Header. With SSL this information is encrypted when the request first hits the Web Server so the Web Server replies with the default site, once the data is decrypted the header can be read but by this time it is too late in the process and the default Virtual Host replies.

The way to get around this in the past has been to assign each SSL host to its own IP Address on sub interfaces so the Web Server can distinguish between the various sites. This approach has worked however it becomes quite a maintenance nightmare in your Apache configuration as well as your server’s network configuration.

The solution to this problem is an extension to the SSL protocol named Server Name Identification (SNI) introduced in RFC 4366 which allows the client to include the requested hostname in the first message of the SSL handshake allowing the Web Server to determine which Named Virtual Host to send the request to and setup the connection accordingly.

Starting with Apache 2.2.12 and later you can take advantage of SNI for your SSL Virtual Hosting. There are some requirements for SNI to work properly.

• You must be running OpenSSL 0.9.8f or later
• You must configure OpenSSL with the TLS Extensions Enabled
• Apache must be built with OpenSSL Support (--with-ssl)

Another requirement is the Client must support SNI. Below is a list of Browsers supporting SNI

• Mozilla Firefox 2.0 or later
• Opera 8.0 or later (with TLS 1.1 enabled)
• Internet Explorer 7.0 or later (on Vista, not XP)
• Google Chrome
• Safari 3.2.1 on Mac OS X 10.5.6

If the client does not support SNI Apache will return a 403 Error Response to the user. You can disable Strict Host Checking in which the default Virtual Host will respond as before when a request came in that did not match any defined Virtual Hosts, this is because once again the server is unable to read the Host header until it completed the SSL negotiation and can read the data.

To enable Named Based Virtual Hosting for SSL you have to make a few changes to your existing Apache Configurations. Below you can find a sample configuration for two different domains running SSL.

NameVirtualHost *:443

<VirtualHost *:443>
    ServerName www.domain-one.com
    DocumentRoot  /var/www/domain-one.com

    SSLEngine on
    SSLProtocol all -SSLv2
    SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
    SSLCertificateFile conf.d/www.domain-one.com.crt
    SSLCertificateKeyFile conf.d/www.domain-one.com.key
    ErrorLog logs/www.domain-one.com_error_log
    CustomLog logs/www.domain-one.com_access_log
</VirtualHost>

<VirtualHost *:443>
    ServerName www.domain-two.com
    DocumentRoot  /var/www/domain-two.com

    SSLEngine on
    SSLProtocol all -SSLv2
    SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
    SSLCertificateFile conf.d/www.domain-two.com.crt
    SSLCertificateKeyFile conf.d/www.domain-two.com.key
   ErrorLog logs/www.domain-two.com_error_log
   CustomLog logs/www.domain-two.com_access_log
</VirtualHost>

Permalink | Leave a comment  »

Information is still limited at this time - at approximately 14:15 UTC an event occured causing our BGP Sessions with Level3 to reset and re-establish five minutes later.

There are confirmed reports that this event within Level3 has affected most of North America. This outage has affected other networks running Juniper routers with the majority of them seeing their devices core dump and reload.

At this point the speculation is that a new BGP Update Bug within specific Juniper version trains was just discovered/triggered.

Phyber's engineering staff will be monitoring the situation closely and will disable our Level3 peerings if necessary to maintain stability of the network.

Permalink | Leave a comment  »

Bundled, Buried & Behind Closed Doors from Ben Mendelsohn on Vimeo.

Lower Manhattan’s 60 Hudson Street is one of the world’s most concentrated hubs of Internet connectivity. This short documentary peeks inside, offering a glimpse of the massive material infrastructure that makes the Internet possible.

Permalink | Leave a comment  »

Location of Event: One Wilshire, 624 Grand Ave, Los Angeles California

Permalink | Leave a comment  »

Location of Event: One Wilshire, 624 Grand Ave, Los Angeles California

Summary Description of Event:  The Data Center has experienced a utility power disruption. While power in the data center is protected, CoreSite facilities staff is on site and verifying proper functionality of the facility systems. 

Permalink | Leave a comment  »

See the full gallery on Posterous

Permalink | Leave a comment  »

Rest in peace Steve. You will be missed by the entire world but we will never forget you. We see you every day on on our desk, we carry you in bags on our shoulders and in our pockets. We watch you on TV, we listen to you in our cars, in the subway, and in the gym. You take our pictures, you chat with our friends. You introduced us to wonderful, amazing magical things. We will continue to hold you in our hands and we will never let you go. Thank you.

Permalink | Leave a comment  »

See the full gallery on Posterous

Permalink | Leave a comment  »

An awesome photo taken of the One Wilshire building (I'm assuming sometime in the 70s) before the Telecommunications conversion. It looks almost naked without the Generators, Fuel Tanks, Chillers, Microwave Towers, etc...

See the full gallery on Posterous

Permalink | Leave a comment  »

Some photos of random communications towers and antennas from my trip to Chicago, IL.

See the full gallery on Posterous

Permalink | Leave a comment  »

A payphone at the NASA Kennedy Space Complex.

Permalink | Leave a comment  »

This generator is attached to a Verizon Cell site. If it hadn't been running I probably would have never noticed it. The green posts in the background are the air intake and exaust, and the box in the front has connections for power hookups and fueling.

Permalink | Leave a comment  »

Everyday, everwhere you are there is hidden infrastructure. As a side feature to this blog we're going to start posting the things we see.

Permalink | Leave a comment  »

On June 22, 2011, the Federal Communications Commission (“FCC” or “Commission”) released an order in its proceeding to consider rules to implement the Truth in Caller ID Act of 2009 (the “Act”).  The Act makes it “unlawful for any person in the United States, in connection with any telecommunications service or IP-enabled voice service, tocause any caller identification service to knowingly transmit misleading or inaccurate caller identificationinformation with the intent to defraud, cause harm, or wrongfully obtain anything of value.” The Actinstructs the Commission to adopt implementing regulations within six months. On March 9, 2011, the FCC issued its Caller ID Act NPRM seeking comments on its proposed rules.  The June 22nd Order adopts the following rules to implement the Act.

First, the Commission adopts the Act’s language, prohibiting knowing transmission of misleading or inaccurate Caller ID information with the intent to defraud, cause harm or wrongfully obtain anything of value.  The Commission’s rules specify that it may impose penalties against any person or entity for violating the Act whether or not it holds a Commission license or registration.  The rules prohibit only conduct in connection with telecommunications and interconnected VoIP services, foregoing the broader term of “IP-enabled” services in the Act in favor of the Commission’s I-VoIP definition in Section 9.3 of its rules.

In addition, while the Commission declined to adopt any specific exceptions other than those expressly enumerated in the Act (namely law enforcement activities and court orders), the FCC emphasized that to violate the act a person or entity must “knowingly spoof caller identification information and do so with intent to defraud, cause harm, or wrongfully obtain something of value” whether directly or indirectly. 

To clarify the Act’s requirements, the Commission also defined several terms including “caller identification information,” “caller identification service” and “information regarding the origin” of a call.  In its definition of “information regarding the origin” of a call, the Commission included “billing information, including charge number, ANI or pseudo-ANI” as well all or part of the telephone number, name, location information or other information regarding the apparent source of the call.  The Commission further clarified that “location information” includes information transmitted in the SS7 JIP code.

The Commission likewise clarified that a person or entity that blocks or attempts to block the display of its own Caller ID information will not be liable for violating the Act or implementing rules. Telemarketers are still required to transmit accurate Caller ID information.  The Commission did not adopt any reporting or compliance obligations for third-party spoofers or address call unmasking. And, the Commission declined to adopt the DOJ’s request for third-party providers to make a good-faith attempt to “to verify that a user has the authority to use the substituted number, such as by placing a one-time verification call to that number.”

Under the Act, the Commission may issue penalties of up to $10,000 for each violation, or 3 times that amount for each day of a continuing violation, with a cap of $1,000,000 for any single violation. The Commission adopted the balancing factors for general forfeitures and the Act’s two-year statute of limitations for prosecuting violations.

To read the Commission’s Order, see here: Truth in Caller ID Order

The Truth in Caller ID Act of 2009 is linked here: Truth in Caller ID Act of 2009

Permalink | Leave a comment  »

Permalink | Leave a comment  »

Phyber's configuration consists of the 7606-S Chassis with the following line cards:

RSP720-3CXL-GE
WS-6748-SFP
WS-X6708-10G-3C

The 7606-S Chassis was selected over the 7609-S for the space efficiency - simply put we can place three 7606s in the space consumed by one 7609. Even with only two Chassis deployed we have a significant amount of port density available to us at One Wilshire for interconnections with both Carrier and Customer networks.

If you are curious more information on the 7606-S Chassis is available on Cisco's web site:

http://www.cisco.com/en/US/prod/collateral/routers/ps368/ps371/product_data_s...

See the full gallery on Posterous

Permalink | Leave a comment  »