Dmitry Dulepov

Mac: adding a keyboard layout using AppleScript

2012-01-31T17:13:00.003+02:00

We have a special keyboard layout in the country, where I live (Latvia). It includes certain special letters: like ā, ē, ū and so on. Many people here set Latvian keyboard layout as a default. However I cannot do that because this layout has one major disadvantage for people like me.

Those special letters are typed by one of two ways: Alt+normal letter or '+normal letter. While the first way is goot, the second one is bad for programmers. I often use ' in the code, so having ' as a "dead key" (this is how it is called officially) is not suitable. But I still have to type those latvian chars sometimes. Until yesterday I kept English US layout as a default and added Latvian manually when I had to type latvian letters. This is a multiple move-and-click process. Yesterday I finally found a way to automate addition and removal of the layout.

Current life

2012-01-17T10:53:00.000+02:00

I am not very much visible in TYPO3 these days. So I though I would give a short update on what I am doing or going to do.

Cleaning and optimising cache tables in TYPO3

2012-01-10T16:17:00.000+02:00

There is an important issue with cache tables in TYPO3, which I am going to address in this article. The issue can become critical for projects with a limited disk space, projects with lots of user groups, which use indexed search or Solr, or projects with lots of pages.

Cache tables are designed to provide a high-performance access for simultaneous reads and writes. This means that several reads at a time should be possible as well as several writes and none should block each other. This is possible when cache tables use InnoDB MySQL database engine. InnoDB provides concurrency and data safety against sudden failures. TYPO3 uses InnoDB for several critical tables, including cache tables.

Sometimes cache tables can grow a lot. The more records exist in the cache table, the slower it is to find a necessary record there. TYPO3 provides a special scheduler task to clean up old cache records. If you have TYPO3 4.6 or newer, you should always use that task.

However, there is a certain fact that very few users know: running the task will remove entries from the table but it will not recover the disk space taken by removed records. InnoDB does not reuse the space when records are deleted. It always appends new data to the end of the allocated data file. Thus it is possible to have a fully empty InnoDB table, which takes 100GB on the disk.

How to solve any kind of problems

2011-12-30T11:32:00.000+02:00

I planned to write this post for a long time. The post is long and it provides detailed instructions on solving any kind of problems. Surprisingly, when it comes to solving problems, the same algorithm works for everything. I came to this algorithm after 15 years of being a software developer, a manager for my own company projects, a family man and a dad. It is not importatnt where the problem is as soon as you folloe the algorithm. You can solve the problem with it, if it is solvable according to laws of the nature and means you have under hand. The algorithm consists from six steps. Let's begin.

Realize that the problem exists

This may sound dumb but this is where the failures start. How much time did you say to yourself: "Oh, it is not a proble!m" only to regret later that you did not address it in the beginning? Son has bad marks in school? Ah, well, he will recover. Customer complains? Ah, well, they always find something to complain about. Sorry, it does not work. Listen to your instincts. If they say, you might have a problem, than most likely you have it. The cost of solving the problem increases with time. The later you realize you have the problem, the more effort you have to make to solve it. The more you delay, the more you loose. So, the very first step to solve the problem is to realize that it exists. Be honest with yourself and say it to yourself: "I have a problem to solve". With that you are closer to solving the problem than you was before. This helps you to commit to solving the problem.

Realize what the problem is

This is a different step. If you realize that you have a problem, it does not mean you realize what the problem is. Your son is getting bad marks? Is that a problem? Most people will say "yes" and be wrong here. Getting bad marks is not a problem, it is a result, an outcome of the problem. This is why it is important to realize what the problem is. Uncovering the real problem could be difficult but this is the time when you should not rush. If you rush now, you may end up solving an imagined problem, not a real one. Really, bad marks - what else could be a problem? Well, tons of them. It can be bad relations with classmates or teachers, wrong course, even medical issues! You must invest time to understand the root, the cause of the problem! If your head aches, you would not chop it, right? For the first time you may take a medicine to ease the pain but if the pain returns, you go to see the doctor. Same with any other problem: get to the root of it, see if your "problem" is not a sympthom of something deeper.

Research possible solutions

This is what most people never do. That just catch the first idea that comes to their mind and go for it. This rarely works satisfactory. The first solution is almost never good. Therefore for years I nver take the first thing that comes to my mind as a solution. I know that there always exists a better solution. It only ta,es time to get it. Cool down. Stop. Don't rush again. Think! Depending on the complexity of the problem and your familiarity with the topic it may take from minutes to months to discover the solution. Months is extreme, of course. Usually it does not take that long. For technical issues I take about 3-4 hours to do something else while putting the problem somewhere to the back of my mind (I imagine it like an old closet of 18th century with many small doors where I can put the problem to "cook" for a while). For something related to kids I usually must sleep ovErnight to Find a decision. It really works. Everytime when I find a better solution, I am surprised how better it it than the first idea that came to my mind. May be this is why my work is usually good: I take time to think about it while doing other tasks. It does work!

Make a plan

Yup, this is another thing very few people do. You do not have to write a detailed plan on a roll of paper or in the spreadsheet. Well, you can, if you want or if the problem is complex and sution is lengthy, but a menthal plan do in most cases. The reason for having a plan is simple: You must know what to do exactky to solve the problem and in what sequence. Chaos is not your way, believe me.

Execute

Only now, when you are clear about the problem, solution and plan, you can go and execute your plan. Remember, that there are no plans without flaws. Even brilliant plans have flaws. So be able to readjust dynamically by temporary going back as many steps as you need. This may sound complicated and time consuming but it is not. When you become used to this algorithm, you will discover to your amazement that you can do it automatically. It is simplicy practice.

Wrap-up

It is a good thing to make a wrap-up after the problem is solved. I rarely write down what the problem was and what I did to solve it (right and wrong) but if you can write it down in a personal diary, it would be good. You can come to it later if necessary. Why don't I do it? Well, I do not like to write on paper and I could not find a good app for that. At least, I do a quick analysis afterwards and try to see what I did well and what I did badly. That helps to avoid the same mistakes later.

Conclusion

What do you think of this? Does that sound like a good algorithm to you? I you have anything to add, please, write in in comments. P.S. No, my son does not have bad marks :) It was just a common example.

Transferring domains from GoDaddy

2011-12-26T22:21:00.001+02:00

I am transferring all my domains from GoDaddy to protest against their SOPA support. This means my blog may become unavailable for a short time in the coming days until the domain name is fully visible with a new registar. In case, if you cannot see this blog, do not panic. It will be back online.

How to make a Macbook Pro DVD drive region-free in Lion

2011-12-13T17:42:00.002+02:00

Apple ships Macbooks with drives without a region set. However, before you watch the movie, you have to set a region. Later you can change it but not more then four times. The solution is to this problem is to flash your DVD drive with a modified "RPC-1" firmware. The risks are as usual: anything can grow wrong, you can brick your drive, warranty is under question, etc. So don't do that unless you are ready to take the risk.

There are lots of instructions on the web how to make it. For my drive, Google leads me to two pages: one and two. They are all ok except that they require a Matshita Flasher application, which does not work in Lion (PPC applications are not supported). However, this application is only a wrapper around the real flasher, so you can flash your Macbook under Lion too. Notice: I did not tell you to do it! I am just providing the information.

Improving battery life for Apple devices

2011-12-05T09:00:00.000+02:00

There is an ongoing discussion on the Internet about troubles with iPhone 4S battery and iOS 5. Many users got decreased battery life with iOS 5. While it is definitely a problem in the firmware, there are ways that can help you increase battery life of any Apple device. I am talking Apple here because I used that for every Apple device I owned. Apple designs batteries themselves and all batteries I had, were made by a single manufacturer. So other devices may need a different way to extend battery life.

When I got a new iPhone 4S, it came with iOS 5.0. I upgraded to iOS 5.0.1. The phone did not survie a day of charge. Now it survives 3 days with using e-mail, twitter, facebook + location services. How did I do that?

Efficient e-mail communication

2011-12-04T10:46:00.001+02:00

Frech IT company named Atos (74,000 employees) will completely ban all internal e-mails in 1.5 years. Its CEO, Tony Breton, said that an average employee gets about 200 e-mails daily. Of them, about 18% are spam and only 10% are useful. They already reduced e-mail traffic by 20% and happy with results. Atos management says that employees will continue to communicate using instant messaging and Facebook-like posts.

Atos made a step towards better efficiency in business communication. This is a great and clever decision.

I agree that e-mail channels often become a major distraction way. While it is possible to filter out spam effectively with relatively simple measures, other e-mail message often remain inefficient. Many mails come with extra words, long disclaimers, polite but needless "how are you"s, etc. I believe that business correspondence should be strictly about business. This position may seem extreme to some readers. We all know that business often grows due to personal relations. But it is a good thing to separate personal and business when you want to talk about tasks. Time is extremely valuable, so loosing it for inefficient e-mails is not a good thing. If it is a business e-mail, it is about business, so let's get straight to business without personal stuff. Personal stuff is ok in personal e-mails but if person A needs person B to do something, just ask for it. The clearer A does it, the more chances that he will get it.

Here is an example of a great e-mail I got recently:

Hi Dmitry,

for our XYZ customer, we have a new request to extend RealURL. It sounds quite simply. Could you please check if this is possible an how much effort it takes you to do the changes with RealURL?

With RealURL it is possible to ...

This is an excellent e-mail: short, concrete, easy to read, clear. When I communicate, I prefer the same style:

Importance of tools

2011-12-02T11:31:00.001+02:00

Quite often I hear an opinion that tools do not matter. If the person is good at his work, he will do it well with any tools. Well, this is both true and false.

When tools do not matter

When I was in school, I went to a photography class after official lessons. It was not like a school class, it was more like a community of pupil with an adult teaching us how to make photos. He owned a very good camera (6x6 cm) and we all complained that he can do good photos with his good equipment, while we can't do it with our cheap cameras.

Once he became annoyed by our complaints and took us to the outside for a three hour training in the forst. We were to photograph freely what we see and like. Some went do their work alone, our followed our teacher trying to make the same photos as he did. He gave everybody the cheapest possible camera (we had a stock of them in the class) and took one of them too.

PhpStorm 3.0 is out

2011-11-30T20:04:00.001+02:00

Just got a message from JetBrains:

PhpStorm 3.0 is now available and introduces the Smart Duplicated Code Detector to help you quickly find similar blocks through your entire code base and securely get rid of them without losing the intended functionality.
Also a new integrated UML tool allows you to quickly get a bird’s-eye-view of your project structure, or even a semantic view of the changes in the recent VCS commits made by your colleagues.
Furthermore, to ensure that your code works fast and doesn’t cause any performance bottlenecks, the IDE now includes a profiler, a new PHPUnit test runner and JSTestDriver.
The new key functionality in PhpStorm 3.0 includes:
- Smart Duplicated Code Detector
- PHP UML roundtrip diagrams
- Profiling results browser for Xdebug and Zend Debugger engines
- PhpUnit 3.6 integration
- CoffeeScript support
- Significant improvements to FTP/SFTP Sync
- TFS support and revision graph for GIT
- Streamlined UI across all operating systems

Get PhpStorm 3.0.

Submitting FLUID forms

2011-11-30T15:30:00.001+02:00

If you use f:form view helper and submit data using GET requests, make sure you add noCacheHash="true" to the f:form. For example:

<f:form action="register" method="post" id="tx_myext" noCacheHash="true">

If you don't, you will put a copy of the page to the cache for each submitted request. That's not necessary.

It is good to put this option also when you submit using POST because there is no need for cHash in such case.

USER_INT is not evil, your code is!

2011-11-30T10:34:00.001+02:00

Quite often I hear a wrong idea that USER_INT plugins is a source of all evil when it comes to TYPO3 performance. I have to bust this myth into dust.

When TYPO3 renders a page, it has to do a lot of stuff. What happens when you request a page under /profile/details/ URL? Firsts, TYPO3 has to resolve the URL to a page id. It calls RealURL to do that. RealURL makes a lot of stuff internally. In the happiest case it makes at least one database query. In the worst case number of database queries will be more than a number of tree levels to the page. TYPO3 page tree implementation is not optimised for performance at all, so you start loosing speed already here. Did anybody thought “nested set”? Nope, no such thing in TYPO3.

Two types of developers

2011-11-29T16:52:00.001+02:00

I divide all developers into two types: team developers and solo developers. Any developer can be good or bad but these two types are fundamentally different in their thinking and approaches to work. It is important to know with whom you work and adjust accordingly.

Team developers

Team developers work well in a team. They need other players. They can work on their assigned tasks but only if those tasks are a part of a whole. Team developers do not like to assume a complete responsibility on the project.

Typically team developers usually communicate well with others, they like to communicate and socialize. They can do the assigned work of any kind but they generally do not have or strive to leran new stuff. They are not entrepreneurs at heart. However they work great in the team. So they are especially useful in big projects when management needs "resources" to fulfil the task.

Team developers can become good team leaders because they understand the environment they work in.

Blog migration, RSS feed woes

2011-11-29T09:59:00.001+02:00

As you may have noticed, I migrated to an external blog service. Domain name is the same but I am using Blogger now. There were several reasons to migrate:

I do not want to maintain the infrastructure. The blog ran on my own server, so I had to watch logs, do updates, etc. I rather not to do that because this blog is a hobby and not a real project.
Better blogging capabilities. As a wrote already, TYPO3 is not really good for blogging at the moment. Blogger gives far more power tools
RTE is much better. When I prepared posts in the past, I always had to fight the TYPO3 RTE. It inserted &nsbp; all over the place, it changed markup all the time, added new paragraphs. So I sent at least 20∞ of time reformatting the article after it was published. Now I simply type and it works.
Better spam handling. There is one guy from India, who regularly tries to abuse my blog by posting offensive words in his language together with phrases like "TYPO3 is great". Other Indians (for whom I have a great respect as a nation), would come here and see all that junk and turn away. This abuser is filtered away automatically now. Never could understand why would people do such destructive things...

About RSS

The side effect of changing the blog is that everybody, who subscribed to my RSS feed, got a lot of old articles marked as unread. I apologize for that. It happens when the blog platform changes. Just mark them as read.

Also I ask subscribers to check the feed they use. The right feed is http://feeds.feedburner.com/dmitry-dulepov. If you use anything else, please, subscribe to http://feeds.feedburner.com/dmitry-dulepov. All other feed links will cease to exist soon.

After using Blogger for a day I feel excited. It is very convenient. I think I'll blog more now. I have ideas for the whole set of articles on TYPO3, better coding, doing stuff in a right way. So stay tuned :)

Internal knowledge

2011-11-28T16:35:00.001+02:00

There is one way to introduce problems in the code, which I'd like to discuss today.

Suppose you have a module, which creates database records. The module has an integer field with a set of flags (each flag is a single bit in the integer). Flags 2 and 4 mean thing X. Flags 2 and 6 mean thing Y. Flag 1 must be used only together with flag 3 or flag 5. The module knows how to manage those flags and what to make of them. Let's name it “module 1”.

Now imagine another module (“module two”). It needs to get information from the first module, that matches certain criteria. Module one would be able to provide that information by matching flags.

Here is the tricky part. The most obvious solution would be simply to query the database directly from module two. Proper solution for module two would be to ask module one for the information. Querying module one's tables using a combination of flags is wrong. Such query uses the internal knowledge of module one outside of the module one. Thus, the implementation of module two becomes dependent on the implementation of module one.

Do you see potential problems here? If module one changes, module two breaks. This "solution spread" introduces unnecessary dependencies between logically separate modules, which is neither necessary, nor good.

A proper way would be to create an API in the module one that returns records, required by module two. If the internal implementation of module one changes, module two will work because all internal proceedings happen inside a single place.

Next time, when you are about to introduce cross-module dependencies, think of minimising them in favor of API usage.

This article was inspired by a http://forge.typo3.org/issues/32136 in TYPO3.

Productivity

2011-11-27T19:42:00.000+02:00

Instead of giving you a full post about productivity, I'll give you a link to much better information than I was going to write. The Ultimate Productivity Blog gives you a 100% productivity recipe.
Read it. You will love it.

Frustrated by the tt_news upgrade

2011-11-27T11:35:00.000+02:00

Today I wanted to write a blog post about productivity. Coming to the BE I noticed that I still had TYPO3 4.4.9 running. So I went to upgrade. It usually takes 10-20 minutes to upgrade a TYPO3 version for me. Well, it took nearly 2.5 hours and I had to revert from tt_news 3.1.0 to old good tt_news 2.5.0 because 3.1.0 simply does not work properly. Here is what does not work:

When the plugin is inserted as a content element, it does not appear on the page. The reason is that it does not add itself to TypoScript anymore (to tt_content.list.20). I had to investigate and add it manually.
New page browser in tt_news does not replace certain markers out of the box. Why couldn't the developer simply use a pagebrowse extension instead of doing a complex thing himself?
The formatting is changed too much.

You can tell me that TYPO3 4.4.9 and tt_news 2.5.0 is too old. Yes, that's true. You are absolutely right. But there is one important thing to know. I had no problems updating TYPO3 core or any other extension. However I had huge problems with tt_news. As usual.
People, when you write your code, remember a gold rule: the less problems people have with your creation, the more they will like it. Upgrade should be smooth. Behavior should be compatible!
I hate tt_news. (Georg, yes, I know about news2 but I am not ready to migrate to it. I am out of time.)
No productivity blog post for today because my productivity for today is out of order, thanks to tt_news.

Future from Microsoft

2011-11-18T12:35:00.000+02:00

I came across a video, which is a Microsoft's version of the future. Here it is:

This is beautifully rendered video but... it is fully Microsoft video :) Microsoft is different from many other companies in the way that they never invented much. They worked on inventions of others. For example, almost everything in this vide is either multitouch, or iPad-like or a 3D monitor from a science fiction. Nothing of this is new, everything was seen in movies already (take "V", for example for interfaces).
The other thing that bothered me, is that Microsoft believes it is the future, while most of it is present. HUDs (head-up displays) are quite common. BWM has them in cars, there are even eyeglasses with HUDs now (you can read e-mail with them or see geographical pointers). Multitouch exists for years in all Apple products. Gestures without touching was recently patented, which means implementation is is the labs already. 3D holograms exist for years. You can read news with phones, which are getting very slim, donate, set appointments and send instructions. Nothing of that is new! So where is the future in this video?
The kitchen has the same microwave (which makes your products less healthy for you). It is all present, nothing is from the future.
Pumps? In future??? Nuh.
If Microsoft wanted to show more of the future, they would not look to pumps or luxury cars. They would show holographic meetings (like in "Resident Evil: Extinction"), all green energy, voice controlled equipment ("Hi, Siri!"), less traveling and more time with family. That would the future.
But that's Microsoft: shiny outside, no new ideas inside.

TYPO3 developers, say "no" to memory leaks!

2011-11-18T12:04:00.000+02:00

There is one small mistake I often see in TYPO3 extensions that I review. It is related to freeing resources after you have done with them. Many people forget to do that. PHP is a language where you do not have to free memory. That's true. But resources (such as file handles, recordset, etc) are different. You have to close them explicitly! PHP may close file for you sometimes but even that is not guaranteed.
Let's see the problematic code:

function memoryLeakingGetDatabaseRow() {
$resource = $GLOBALS['TYPO3_DB']->exec_SELECTquery(...);
return $GLOBALS['TYPO3_DB']->sql_fetch_assoc($resource);
}

What is wrong here? This code fetches a recordset from the database but does not free it. This means that all structures MySQL had to prepare live in memory. This may include portions of the data retrieved from the database. That data lives for the whole request. If you make multiple calls you accumulate more and more such data in memory. That leaves less memory for other processes on the server. This data is tied to the database connection. If you have persistent database connections, this data is not released when your script terminates because the connection is not released. Thus you have data accumulating all the time. The result is slow down of the server after a certain period of time (low memory means more swapping and that is slow).
How this can be prevented? Code:

function memoryLeakingGetDatabaseRow() {
$resource = $GLOBALS['TYPO3_DB']->exec_SELECTquery(...);
$row = $GLOBALS['TYPO3_DB']->sql_fetch_assoc($resource);
$GLOBALS['TYPO3_DB']->sql_free_result($resource);
return $row;

}

This releases the recordset properly and frees all allocated memory.
[Note: of course, for a single result you can use exec_SELECTgetSingle but the point of this article was to show the problem and the solution]

How I came to TYPO3

2011-11-17T09:50:00.000+02:00

In the comment to my previous article, I was asked how did I come to TYPO3. Here is the answer.
Me and my wife have a project in our country. The project has started in January 2001 as a set of static pages and grew to a large family portal with elements of a social network.
In 2001 it was just a diary, which my wife made online. Blogging was not a hot area at that time but she was blogging. People came and read, they were interested. So she asked me to make a forum on the site. So there was a forum. People asked for more: they wanted more articles, more information. The amount of information grew.
In 2003 we had our own rudimentary CMS. It allowed to manage pages and connect them to menu items. So menu structure was separate and pages were separate. Menu could be two levels deep maximum. The CMS was simple and fast, it could run on inexpensive shared hosting with a very limited CPU power, memory and disk space.
Number of articles grew and navigation became harder because we needed more nesting levels and greater editing capabilities. I worked full time on my daily job, so I could not create a “full force” CMS. I started to look around. I went to the opensourcecms.com and tried several CMSes there. At that time there was only one CMS there, which I more or less liked: Mambo (now it is called Joomla). TYPO3 also was there but the text said something like “the system is too heavy and complex to be installed as demo”.

Looking back at my professional life

2011-11-15T14:26:00.000+02:00

I am 38, which is an age when people start looking back and see what they went through and evaluate their path. So do I. I am a software developer and I was like that for all past years. Have a look back with me.

Does your blog/forum help spammers?

2011-11-07T09:00:00.000+02:00

Do you know that your blog or forum may help spammers to gather e-mail addresses of commenters even if you hide their e-mail addresses?

Recently I discovered one interesting blog and made a comment on one post. There was a checkbox to subscribe to updates of comments. I ticked that because I was interested in the topic. When comments started to come to my e-mail, I noticed that they were sent from the e-mail address of people, who commented. E-mails were hidden in the blog but as soon as you subscribe to comments, you get e-mail addresses of all commenters. That would help spammers a lot if the blog uses Gravatar service for user pictures. Gravatar binds e-mail addresses to pictures. Blogs can show pictures of users if the user provided a valid e-mail address. Thus, if the blog is known to use Gravatar, many users will supply a real valid e-mail. Here you get spammers subscribing and catching a lot of valid e-mails. Update/clarification: this is not a problem of Gravatar! This is your blogging software that can send such e-mails. Gravatar only shows pictures and it is not vulnerable at all.

There is another view on this problem: the blog says that e-mail address will not be revealed but, in fact, it reveals the address. So it may have legal privacy implications. I notified blog's owner about the issue.
Conclusion: never use user's e-mail to send anything from the web site.

Backend search improvements in TYPO3 4.6

2011-10-24T09:00:00.000+03:00

TYPO3 4.6 brings a lot of great enhancements. I would not tell you all of them in this post but you will learn about one of them. The reason to choose this enhancement is that it has a certain effects on TYPO3 extensions. Extension authors need to be aware of this change and adjust their extensions accordingly.
The enhancement I mean is an improvement of the Backend search. What was wrong with the old search? The old search worked like this:

Take all numeric fields and add them to the list of searchable fields (including uid and date/time fields)
Take all input and text fields and add them to the list (including, for example TSConfig field)
Use LIKE %...%
Search every table in the database if the table is in $TCA
Search 4 levels deep maximum but never say it to anybody. So people would wonder why can't they find results deep in the tree.

That causes too many fields to search for each table (most of them unnecessary). Also it searches numeric fields for string patterns using LIKE, which is a very slow operation. In case if there are many extensions installed, searching could take minutes.

What was changed? The search was adjusted to work like this:

Use only specially configured fields for search (i.e. we can search "bodytext" field but generally there is no need to search "TSConfig" or i18n_diffsource field)
Use numeric fields in search only if search phrase is numeric
Skip tables that are not configured in $TCA
Search regardless of pid unless searching "current page only" from the List module.

Using TYPO3 for blogging

2011-10-23T10:41:00.000+03:00

From time to time I see questions in the TYPO3 mailing list about using TYPO3 got blogging. Since I have a blog made with TYPO3, I am going to compare several solutions and describe their advantages and disadvantages. I also share my own views on choosing the best blogging tool.

"Your computer was restarted because of a problem"

2011-10-13T10:37:00.000+03:00

That message appeared each time after I restarted my MacBook Pro. It looks like a flag was written somewhere that this message should be displayed. While it takes a simple click to get rid of the message, I wanted to get rid of it once and forever. After a small search in Terminal I found how to fix it:

Open Terminal (use Spotlight to find it)
Type sudo rm /var/db/PanicReporter/current.panic

That's all. You can now reboot without errors.