I started playing around with s3cmd command line utility and I realize that is really powerful so, I decide to write my own scripts to be able to backup automatically all the important files of my servers to the cheaper and easy to use Amazon S3 service.

The whole system is based on two bash scripts, the crontab and a few extra applications like 7zip and s3cmd. I will explain step by step how I build up the system and at the end I’ll provide the full code of the scripts.

Requirements

The first step is to be sure that you have installed the 7zip and s3cmd tools, in my case I’m using ubuntu so, for me the command to install the tools is:

sudo apt-get install p7zip s3cmd

Basic setup

Now we can follow with the basic setup, I created a folder in the root of the filesystem called backups, the inside this folder I create serveral folders, at the end the structure is like follow:

/backups
    /compressed
    /data
        /db
        /svn
        /www
    /scripts

The compressed folder is used at the end of the script to compress all the files to one .7z file, in the data folder we will copy the information we want to backup and in the scripts folder we will store as it’s name says the two scripts we will use.

Backup script

Now, let’s have a look at those scripts. The first one is to do the backup, in my case I’m doing a backup of the subversion, mysql and files under /var/www.

The script starts with the following block:

echo `date '+%F %T'`: Starting the backup

#Dumping the repos
echo `date '+%F %T'`: Starting the dump of the repos
for i in /user/local/svn/*/; do
    repo=`basename $i`
    echo `date '+%F %T'`: Dumping repo $repo
    /usr/bin/svnadmin dump /usr/local/svn/$repo > /backups/data/svn/$repo.dump
    echo `date '+%F %T'`: Repo $repo dumped
done

We suppose that the svn folder is located at /usr/local/svn, basically here we are going through each folder inside /usr/local/svn/ and we call svnadmin dump to dump that repository in one file, that file is stored in /backups/data/svn/.

#Dumping the DB
echo `date '+%F %T'`: Starting the dump of the DataBases
for i in /var/lib/mysql/*/; do
    db=`basename $i`
    echo `date '+%F %T'`: Dumping DB $db
    /usr/bin/mysqldump -uroot -p$db_password $db > /backups/data/db/$db.sql
    echo `date '+%F %T'`: Database $db dumped
done

In this second block we are doing more or less the same thing, we are going through each folder on /var/lib/mysql/ and calling mysqldump to export each DB in one file stored at /backups/data/db/.

#Copy all the websites
echo `date '+%F %T'`: Starting the dump of websites
for i in /var/www/*/; do
    site=`basename $i`
    echo `date '+%F %T'`: Dumping site $site
    /usr/bin/7z a -mx6 -t7z /backups/data/www/$site.7z -p$compression_password /var/www/$site
    echo `date '+%F %T'`: Site $site dumped
done

In the third block we are dumping the websites, I have the websites stored at /var/www folder and each site is inside a folder with the domain as folder name. Basically we are doing the same, loop through /var/www/ and compressing each site individually in a .7z file with a compression level of 6 and secured with a password.
I store each site in a separate file because if I have to restore a site I don’t need to uncompress all the sites, just the main file and then the compressed site.
I was playing with different compression levels and 6 is the most suitable to maintain a good compression ratio without wasting a lot of time.

#Compressing all the data
echo `date '+%F %T'`: Compressing the info
filename=$(date +%Y%m%d)
/usr/bin/7z a -mx6 -t7z /backups/compressed/$filename.7z -p$password /backups/data/*
echo `date '+%F %T'`: Info compressed

Now with all the information we want to backup in place we will compress all together in one single file, that’s the purpose of this block of code, the final filename is a timestamp of today using the date command.

#Upload to Amazon S3
echo `date '+%F %T'`: Uploading to Amazon S3
/usr/bin/s3cmd put --no-progress /backups/compressed/$filename.7z s3://BUCKET_NAME/$filename.7z
echo `date '+%F %T'`: Upload completed

Ok, we have the compressed file and one of the final steps is upload this file to Amazon S3. For that task we are using the s3cmd tool passing the parameter –no-progress to avoid an “interactive” output of the upload status.

#Delete the local backups
echo `date '+%F %T'`: Cleaning up
rm -Rf /backups/data/svn/*
rm -Rf /backups/data/db/*
rm -Rf /backups/data/www/*
rm -Rf /backups/compressed/*
echo `date '+%F %T'`: Clean completed
echo `date '+%F %T'`: Backup completed

And the final block of code is just a clean up, after the upload to amazon we delete all the files we have generated inside the /backups folder.

Ok, seems complex but is pretty strightforward, also if you take a look you’ll see that all the messages contains a timestamp, with that information we are able to determine how many time we spend in each task.
One disadvantage of this script is that we are not checking if the file was uploaded correctly to Amazon, maybe this will be a future improvement.

Maintenance script

Now let’s take a look at the other script, the purpose is just delete old backups based on a timestamp, in my case we’re storing the backups for one month, it’s very affordable with the Amazon prices.

for filename in `s3cmd ls s3://$bucket`; do
    if [[ $filename =~ ([0-9]*)\.7z ]]; then
        timestamp=${BASH_REMATCH[1]}
        echo `date '+%F %T'` - Reading metadata of: $filename
        echo -e "\tFilename: $filename"
        echo -e "\tTimestamp: $timestamp"
        if [[ $timestamp -le $limit ]]; then
            let "total=total+1"
            echo -e "\tResult: Backup deleted\n"
            /usr/bin/s3cmd del $filename
        else
            echo -e "\tResult: Backup keeped\n"
        fi
    fi
done

This is the only code block we have in the maintenance script, the idea is to retrieve the list of files we have in the bucket with s3cmd, then we check if the gived part contains the pattern ([0-9]*)\.7z (because the command s3cmd ls gives more information rather than just the filenames).
If we detect a segment that matches the regex we get the filename without the extension (the timestamp of the backup), and we check in this case if the timestamp is older than one month (the timestamp of one month ago is stored in the $limit variable). If the backup is older we remove it calling the s3cmd del command.

If you need to store the files more time just change the $limit variable and that’s it.

Wow, seems a large process, ok, now let’s have a look at the last step to let this work, the cronjobs.
I created to cronjobs and redirected the output of them to the stdout because I want to receive an email after the backup, my crontab look like this:

# m h  dom mon dow   command
MAILTO=YOUR_EMAIL_ADDRESS_HERE
00 22 * * * /backups/scripts/s3backup 2>&1
00 23 * * * /backups/scripts/s3backup-maintenance 2>&1

Just put there the values you want to run the crons and don’t forget to put at the end 2>&1.

Conclusion

With this system you’ll have your backups in a safe and cheap place without headaches and with a full report in you email every time the script runs. For sure these scripts can be impoved with a few checks and you can extend it to fit your needs.
I hope you have enjoyed this HowTo and begin to use this scripts! For sure improvements are welcomed, don’t hesitate to comment!.

Source code

Grab the full source code of the backup script and the maintenance script.

The basic information that GA gives to you is the number of visitors per day, week, month or custom ranges, also you can see the referrers of your page and the keywords they used to find you over google or another search engine.
More complex tasks are for example the funnels, the objectives, etc. For example, imagine that we have a online service website and we want to know the conversion ratio between the traffic that arrives at our website and the people that finally buys our service. In GA we can setup a Goal, this goal at the end is a destination url, time on the site or pageviews per user. So, for our example we will setup a goal for a specific url, that can be our service payment confirmation page. With that goal in place we will have our usual information about how many traffic we have and then a specific information about how many traffic arrives to the confirmation page.

After that to see in a nice way all that information we can use other tool inside GA called funnels. The funnel is a graphic to display that kind of information, imagine that our checkout process have 2 steps plus the confirmation page, with that in mind we can create a funnel to see how many traffic passes for every stage of the checkout.

In the following picture we can see a funnel example, in this e-commerce shop we have 3481 total visits and only 241 do the login on the page, of this 241 users 146 places an order and 119 users complete the order, that means a conversion rate of 0.09%.

These are just some ideas and tools availables in GA, so don’t forget to use this kind of tools in your websites to extract as much information as possible to know how the market reacts upon your products and processes.

To continue reading and learning about this tool you can visit the Google Analytics Tour Page or just go to Google Analytics and create an account!

If your startup is based on a software product, you must develop it and you want to do it in the best way. Besides, if you are (or have) a developers team, you will need to manage tasks, resources, code releases, etc. So, maybe a issue and project tracking like Jira could be really useful for you.

Jira is a web-application with a clean and fast interface that will become the centre of your development team. Each issue is modeled in Jira as a ticket which can be assigned to any member of the team with an associated ETA (estimated time of arrival).

This software from Atlassian has integration with control versions systems, such as SVN and Git. Jira also allows you to custom the workflow of the tickets, in order to represent better the real workflow of your business. Then, you can create many reports and analyse all the information.

Software licenses for Jira prices starts on $10 for 10 users (deploy on your server) or free if you are working on a Open Source project. You can see all pricings here.

uservoice is an online tool to manage feedback easily on your site. You only have to put on your website a simple tab with the label “feedback” linked to your uservoice account. Then, when a user clicks on this tab, will open a lightbox style pop-up that allows the user to leave his ideas, suggestions, bugs or anything else. As an admin, uservoice provides you a backoffice where you can manage that information. So you can reply to user to thank them, delete the information if consider that it isn’t useful or maybe let the user know that his idea or bug will be done or fixed in future releases.

As many others web services, uservoice has different types of pricing (Free, Tin, Bronze, Silver and Gold) depending on your needs and how much you want to pay!

To sum up, we can say that uservoice is a really powerful online application to manage the feedback you get from your users with a really low effort and costs. Wanna try? Visit http://uservoice.com/.

Google Apps is a web-based solution, so it has all the advantatges of Cloud Computing applications. You can access to it anywhere and anytime (if you have an Internet connection).

This services provide you 50 times more storage than the industry average with 99.9% uptime reliability guarantee with synchronous replication. Also offer you Mobile email, calendar, IM access and helpful 24/7 customer support.

Standard edition is perfect to newly emerged Startups because it’s free! But it has some limitations: 50 users maximum, email attachments cannot be larger than 25 MB, limited to sendig email to 500 external recipients per day per email account and less uptime guaranteed. 24/7 support is also offered only for Premium accounts.

Finally, we let you a quick tour video of Google Apps to encourage you to test this amazing suite for your company.

First of all, we apologies for the unexpected changes done in the SegmentationFault Blog. As you may notice, we have transform our old blog into a corporative website, with a new appearence, and we have decided to write exclusively in English from now on.

One year ago we started this blog with the aim of shareing with others a bit of knowledge that we learn everyday. Now, SegmentationFault is becoming to be a more serious project, specially since we are participating in Tetuan Valley Startup Spring School 2010.

We hope to continue having your support at this time, with your feedback on the blog or following us on Twitter (@sfteam).

Best regards for all of you,

Christopher, Noemí and Gabriel
SegmentationFault team

+----+----+--------+
| CS | LF | Locale |
+----+----+--------+
| 0  | 0  | DL_DC  |
| 0  | 1  | LF_DC  |
| 1  | 0  | DL_CS  |
| 1  | 1  | LF_CS  |
+----+----+--------+

Lengend

• CS = Country Supported in the System
• LF = Language Forced in the URL
• DL_DC = Locale for the default country with the default language configured on the DB
• LF_DC = Locale for the default country but with the language forced in the URL
• DL_CS = Locale for the country detected with the default language configured on the DB
• LF_CS = Locale for the country detected with the language forced in the URL

Thinking about how it’s the best approach to solve the problem I found the following solutions.

Which do you think it’s the best approach?

We will try to get the same effect as the image below:

To create this effect you have to add the following lines on the html file:

<div class="background">
   <div class="white">This is an exemple with the CSS opacity property</div>
</div>

The first div have the class “background” to put a standard background and the second div is inside the first div to put another background called “white” over the first background.

Now you can customize the classes with the following attributes and values on your css file:

.background{
 background-image: url(caracol.png);
 width: 300px;
 height: 208px;
 border: 1px solid #000000;
}

.white{
 background-color: #ffffff;
 width: 250px;
 height: 158px;
 border: solid 1px #000000;
 margin: 25px 25px;
 opacity: 0.6;
 filter:alpha(opacity=60);
}

The class “background” contains an image background, the class “white” contains a background color with the opacity property to obtain the transparent effect. The attribute “opacity: 0.6;” is for Firefox and the values can go from 1.0 to 0.0, to be able to show this effect in IE you have to use the following code  “filter:alpha(opacity=60);” and the possible values can go from 100 to 0.

Remember to put the Doctype declaration on your HTML file to avoid the problems with the diferents bowsers and also write a stardard code.

I hope you enjoy it and do not hesitate if you have any doubt.

The tricky point was that the Array could contain more Arrays so we had to write a recursive function to accomplish our objective.

Tudor, at the end, gave the following method to be able to do the conversion.

/**
 * Recursively converts an array to a stdClass object
 *
 * @param array $array
 * @return stdClass
 */
 protected function _convertToStdObject(array $array){
   $obj = new stdClass();
   foreach($array as $key => $value) {
     if(!is_array($value)) {
       $obj->{$key} = $value;
     }else{
       $obj->{$key} = $this->_convertToStdObject($value);
     }
   }

   return $obj;
 }

Maybe, in the next interviews we will do to our PHP Developers candidates, we will ask about this and see how they solve the problem

Para ello, debemos discernir, en primera instancia, entre los datos que conforman el contenido propiamente dicho (normalmente documentos jerarquizados que siguen una estructura) y los llamados metadatos. ¿Y qué son los metadatos? Entendemos por metadatos, los datos que contienen información sobre los propios datos (etiquetas, enlaces, reviews, pageviews, etc.).

Otro aspecto muy a tener en cuenta en el proceso de clasificación de los datos es la privacidad de éstos. Los buscadores utilizan la información para devolver resultados cada vez más adecuados a los usuarios, y parte de ella (como las querys o los clicks) pueden revelar algunos datos personales.

Llegados a este punto se nos formulan algunas preguntas nada despreciables: ¿Cómo podemos utilizar estos datos? ¿Con qué criterio los organizamos? ¿Cuán fiables son los datos que poseemos?

En lo que respecta a la fiabilidad de los datos, podemos hacer una reflexión desde el punto de vista estadístico. Teniendo en cuenta que el conjunto de búsquedas/visitas es muy elevado, y curiosamente la mayoría de ellas son sobre un conjunto de temas comunes (la llamada “long tale” en la representación gráfica), la distribución tenderá a ser una distribución normal o gaussiana. La media de todos los datos será, por lo tanto, un resultado sorprendentemente bueno.

Así, podemos aprovechar que el conocimiento colectivo supera, en la mayor parte de los casos, al conocimiento individual. Toda esta teoría está detallada en profundidad en el libro The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations de James Surowiecki.

Aún disponiendo de datos fiables, debemos establecer una organización y clasificación para que resulten realmente útiles al usuario final. Esto requiere determinar unos criterios sobre el conjunto de información (texto, enlaces, etiquetas, queries, etc.) para medir la calidad de cada recurso.

La clasificación por excelencia de la Web 1.0 (PageRank) basa la calidad de los documentos en función de los enlaces que “apuntan” a cada uno de ellos. Este ingenioso modelo funcionaba a la perfección cuando los únicos usuarios que realizaban los enlaces en la red de redes eran los administradores. Hoy por hoy cualquiera tiene la posibilidad de crear enlaces y este sistema para clasificar los contenidos está perdiendo fuerza en la Web 2.0.

Un modelo alternativo que tiene un buen desempeño en la actualidad es el etiquetado, en inglés tagging, de los documentos. El usuario que añade una información en Internet se encarga también de definir un conjunto de etiquetas o palabras claves que ayuden a clasificar dicha información.

El etiquetado tiene aplicaciones muy diversas e interesantes. Una de ellas consiste en la selección de contenido teniendo en cuenta la diversidad de términos (topical diversity). Un ejemplo claro es la consulta “Jaguar”, que puede hacer referencia al animal o a la marca de automóviles.

También son útiles los tags para segregar y descartar algunas imágenes en querys determinadas. Por ejemplo, si un usuario busca “Honda Civic” probablemente no querrá encontrar como resultado un conjunto de imágenes del coche con el mismo ángulo. Un resultado óptimo devolvería imágenes desde todos los ángulos posibles. Es lo que se designa con el término visual diversity.

El tagging en imágenes tiene más aplicaciones útiles, como la navegación a partir de las etiquetas (como en Yahoo! TagExplorer) o las anotaciones visuales (use visual annotations), en las que una etiqueta es representada mediante un rectángulo que encuadra un objeto particular. Sistemas de este tipo se utilizan actualmente en portales como Flickr o Facebook.

Pese a todo, el gran inconveniente de este sistema es la necesidad de que el usuario defina los tags, con lo que ello respecta (errores ortográficos, errores tipográficos, subjetividad, heterogeneidad…). Inmediatamente se nos ocurre la idea de implementar un sistema para sugerir etiquetas, pero la experiencia nos dice que el usuario es propenso a hacer uso de la “ley del mínimo esfuerzo”. Es decir, que en la mayoría de los casos el usuario se limitaría a introducir una única etiqueta y seleccionar el resto de ellas del conjunto propuesto por el algoritmo de sugerencia.

Otro modelo que cabe destacar es el basado en la correlación entre los recursos. El principio de funcionamiento de este método es buscar el conjunto de elementos relacionados con la consulta introducida por el usuario (otros nombres, lugares, eventos, imágenes, etc.). Estas interrelaciones entre las entidades se suelen representar como grafos, y se aplican los algoritmos y técnicas de teoría de grafos para su implementación.

Relacionado con lo anterior está el uso de las consultas como etiquetas implícitas (querys as implicit tags). Esta técnica aprovecha las decisiones que toma el usuario para detectar las relaciones entre las búsquedas y etiquetar los recursos. Un ejemplo podría ser dos usuarios que realizan las consultas “Roma” e “Imperio romano” respectivamente y, no obstante, hacen click en el mismo enlace de todos los presentes en los resultados. El sistema detectaría estos casos y haría patente esa relación mediante nuevas transiciones en el click graph.

Un sistema complementario para mejorar las búsquedas en la Web 2.0 es el uso del conocimiento implícito (implicit knowledge). La mejor manera de entender el funcionamiento de esta técnica es con un caso práctico. Supongamos que un usuario introduce la query “tfcu”. Usando el conocimiento implícito debe obtener todos los documentos relacionados con la teachers federal credit union, dado que tfcu es el acrónimo de ésta. Del mismo modo, el sistema debe guardar una relación entre CEE, UE, Comunidad Económica Europea y Unión Europea.

El objetivo principal de la recuperación de información (information retrieval) es devolver el conjunto óptimo de recursos relacionados con lo que el usuario está buscando, en el menor tiempo posible. Todos los modelos mencionados, y muchos otros que se han quedado en el tintero, sirven para acercarse más al objetivo, y el uso combinado de todos ellos produce mejores resultados.

La Web crece en volumen de información y evoluciona a la par que los usuarios requieren disponer de la información más relevante de la forma más rápida posible. ¿Pero, en qué dirección evoluciona la Web? La tendencia marca el progresivo adiós a las búsquedas, dando paso a portales que ofrecerán al usuario justo aquello que desean obtener sin necesidad de que éste lo pida. Dicho así puede sonar bastante utópico. Sin embargo, ¿quién imaginaba hace 20 años que se podría obtener todo tipo de información escribiendo algunas palabras en una caja de texto? ¿Llegará un día en el que abriendo nuestro navegar obtendremos la información que necesitamos sin hacer nada para llegar a ella? Sólo el futuro de la Web tiene la respuesta.

Fuente

Ricardo Baeza – http://www.dcc.uchile.cl/~rbaeza/spanish.html