What 6 months of PHP/Laravel taught me as a Ruby/Rails developer

Mon, 05 Jan 2026 00:00:00 +0000

“Good programmers use their brains, but good guidelines save us having to think out every case.”
— Francis Glassborow

After many years working with Ruby on Rails, I’ve spent the last six months working full time with PHP and Laravel. What surprised me the most was not learning a new syntax, but how little friction I felt switching stacks.

Not because the languages are the same (they aren’t) but because Rails and Laravel share a very similar philosophy: convention over configuration, developer happiness, and code that reads like intention.

Below are the main similarities and differences I noticed from a very practical, day-to-day perspective.

Eloquent Models feel like ActiveRecord (and that’s a good thing)

If you’ve used Rails’ ActiveRecord for years, Eloquent will feel instantly familiar.

Model definition

Rails (ActiveRecord):

class Post < ApplicationRecord
  belongs_to :user
  has_many :comments

  scope :published, -> { where(published: true) }
end

Laravel (Eloquent):

class Post extends Model
{
    protected $fillable = ['title', 'body', 'published'];

    public function user()
    {
        return $this->belongsTo(User::class);
    }

    public function comments()
    {
        return $this->hasMany(Comment::class);
    }

    public function scopePublished($query)
    {
        return $query->where('published', true);
    }
}

Querying feels almost identical

This is where Rails muscle memory really kicks in.

Rails:

Post.published
    .where(user_id: 1)
    .order(created_at: :desc)
    .limit(10)

Laravel:

Post::published()
    ->where('user_id', 1)
    ->orderBy('created_at', 'desc')
    ->limit(10)
    ->get();

Chaining, scopes, lazy evaluation - the flow and intent are basically the same.

Collections API: Ruby vibes everywhere

Laravel’s Collection API is one of its strongest features — and clearly inspired by Ruby’s Enumerable.

Ruby:

users
  .select { |u| u.active? }
  .map(&:email)
  .sort

Laravel:

$users
    ->filter(fn ($u) => $u->active)
    ->map(fn ($u) => $u->email)
    ->sort()
    ->values();

Once you stop thinking in raw arrays and start thinking in collections, PHP becomes much more expressive.

Hash vs Array: different names, same idea

This is one of the first mental adjustments when moving from Ruby to PHP.

In Ruby, Hash is the natural structure for key-value data. In PHP, arrays do everything - lists, maps, hashes - which feels odd at first but works well in practice.

Defining key-value data

Ruby (Hash):

user = {
  name: "User",
  email: "user@example.com",
  active: true
}

PHP (array as hash):

$user = [
    'name' => 'User',
    'email' => 'user@example.com',
    'active' => true,
];

Accessing elements

Ruby:

user[:email]
user.fetch(:email)

PHP:

$user['email'];
$user['email'] ?? null;

The null-coalescing operator makes safe access very straightforward.

Adding or updating values

Ruby:

user[:role] = "admin"
user.merge!(last_login: Time.now)

PHP:

$user['role'] = 'admin';

$user = array_merge($user, [
    'last_login' => now(),
]);

With Laravel Collections:

$user = collect($user)
    ->merge(['last_login' => now()])
    ->toArray();

Transforming data

Ruby:

prices = { apple: 10, banana: 5 }
prices.transform_values { |v| v * 2 }

PHP (Collection):

$prices = collect(['apple' => 10, 'banana' => 5])
    ->map(fn ($v) => $v * 2)
    ->toArray();

Nested access

Ruby:

user.dig(:profile, :address, :city)

PHP (Laravel helper):

data_get($user, 'profile.address.city');

Different syntax, same intent.

Request validation is as simple as Rails strong params

Validation in Laravel feels very close to Rails - simple, readable, and explicit.

Rails:

params.require(:user).permit(:name, :email)
validates :email, presence: true, uniqueness: true

Laravel:

$request->validate([
    'name'  => 'required|string',
    'email' => 'required|email|unique:users',
]);

In both frameworks, validation lives close to where data enters the system.

Routes: familiar DSL, different syntax

Routing is another area where the philosophies strongly align.

Rails:

Rails.application.routes.draw do
  resources :posts
  get "/health", to: "health#check"
end

Laravel:

Route::resource('posts', PostController::class);
Route::get('/health', [HealthController::class, 'check']);

RESTful by default, expressive, and easy to reason about.

Artisan feels like Rails generators on steroids

If you like rails generate, you’ll feel at home with Artisan.

Rails:

rails generate model Post title:string body:text
rails routes

Laravel:

php artisan make:model Post -m
php artisan make:controller PostController --resource
php artisan route:list

Artisan goes beyond generators:

queue workers
cache management
app optimization
custom CLI commands

It’s a very strong part of the Laravel ecosystem.

PHP is simple, explicit, and predictable

Modern PHP is very different from what many of us remember.

Less magic, more clarity

Ruby:

user.find_by_email("test@test.com")

PHP:

User::where('email', 'test@test.com')->first();

No hidden meta-programming - what you see is what runs.

Interfaces and typing help a lot

interface PaymentGateway
{
    public function charge(int $amount): bool;
}

Clear contracts reduce ambiguity, especially in larger teams.

The not-so-great parts

No stack is perfect.

Laravel caching can be annoying and inconsistent across environments
Finding libraries for newer or niche services can be harder than in Ruby
PHP still requires more boilerplate in some areas
The ecosystem is more fragmented (versions, extensions, hosting)

None of these are deal breakers - just trade-offs.

Final thoughts

After six months working daily with Laravel, one thing is very clear to me:

PHP is not dead. Not even close.

Laravel delivers:

a modern developer experience
a familiar MVC philosophy
excellent productivity
and a surprisingly smooth transition for Rails developers

Rails still shines in elegance and ecosystem maturity, but Laravel proves that PHP has evolved - a lot - and in the right direction.

At the end of the day, good software is less about the language and more about:

clear code
good tests
solid architecture
and teams that know what they’re doing

And in that sense, both Rails and Laravel get the job done - really well.

My thoughts about AI tools, vibe coding, and some predictions about the future of programming

Thu, 18 Dec 2025 00:00:00 +0000

“AI tools are great if you know what you’re doing.”
— A tired developer

“Magical tools” aren’t new

I started programming about 25 years ago. Self-taught. A lot of trial and error. When I was lucky, I had access to books. When I wasn’t, I explored every corner of the Help menu of Delphi. I learned by breaking things, fixing them, and slowly understanding how everything fit together. The learning curve was high.

At that time, the “best” tools were the ones that promised productivity through visual components: drag-and-drop forms, buttons, grids, data sources, and magically “connected” events. The pitch was always the same: “You don’t need to write code. Just drag components and connect them.”

Delphi, Visual Basic, PowerBuilder, and similar tools built entire empires on that promise.

Old-school developers hated it. They said it wasn’t real programming. That it was cheating. That people using those tools didn’t really understand what was happening underneath.

The market didn’t care.

Companies shipped software faster. Businesses made money. And those tools dominated for years. Eventually, they shaped the industry, influenced modern frameworks, and became the foundation of what we now call “rapid application development”.

AI tools feel different — but they really aren’t. They are just the next iteration of “magical tools”.

AI tools are great if you know what you are doing

If you just ask an AI to solve a problem for you, you’re doing it wrong.

AI is not a replacement for thinking. It’s an advanced code generator.

The quality of the output is directly proportional to the quality of your prompt — which, in practice, means the quality of your understanding of the problem, the codebase, and the architecture.

A bad prompt looks like this:

“Add a button that lists people on this page.”

In the best case scenario, the AI will generate something that works. In the worst case, it will dump a pile of spaghetti code that ignores conventions, architecture, separation of concerns, permissions, tests, and long-term maintainability.

A good prompt looks like this:

“Add a button that lists people on this page.
Use the MyButton component and centralize all fetching/mutation logic into the api.ts file.
Get the list of people from the ContactsService class, but keep all filtering logic inside PeopleController.
Make sure to verify user permissions to create a new Person using the User.canExecute("action") method.
Write an integration test covering this usage scenario.”

Now you are in control.

You already know what needs to be done. You know where the logic belongs. You know how your system should evolve. You’re just delegating the mechanical part — writing code — to the AI.

That’s not laziness. That’s leverage.

“You’ll become dependent and forget how to code”

This argument keeps coming back. And yes — it’s partially true.

When you don’t have to worry about syntax, language quirks, or framework boilerplate, you naturally forget some details. You might not remember the exact syntax of a for loop in Go, or how Ruby blocks behave in edge cases.

But here’s the uncomfortable truth: languages are commodities now.

It really doesn’t matter that much if your backend is written in Go, JavaScript, Ruby, or Python. What matters is that you understand:

logic
data flow
architecture
patterns
trade-offs
constraints
failure modes

If you have that, the AI can handle the syntax.

We already went through this transition before. We stopped memorizing assembly when high-level languages appeared. We stopped writing raw SQL everywhere when ORMs became popular. We stopped managing servers by hand when cloud platforms emerged.

This is just the next step.

Junior developers will disappear

AI tools are the new junior developers.

A mid/senior developer, using AI effectively, can be dramatically more productive than a senior leading and coordinating a team of juniors. There’s no onboarding, no context switching, no waiting for pull requests, no repeated explanations of architecture decisions.

More importantly:

AI doesn’t forget conventions
AI can instantly generate alternatives
AI can refactor relentlessly
AI can write tests, docs, and examples on demand

For companies, the cost/benefit equation changes completely. Instead of hiring multiple juniors and hoping they grow, a smaller team of experienced developers with AI assistance can deliver faster and with more consistency.

That doesn’t mean people disappear — it means the traditional entry point to the industry will change radically.

Generalists will rule

For the vast majority of companies, you no longer need deep specialists in frontend, backend, and database just to build and run a product.

A single developer, armed with AI tools, can:

design APIs
build frontend interfaces
model databases, write migrations
configure infrastructure
handle authentication and security practices
set up CI/CD
deploy and monitor applications

This is not science fiction. It’s already happening.

Of course, I’m talking about common software — the kind that runs most businesses. Not rocket science. Not medical devices. Not highly specialized domains.

If your company operates in a niche area that requires deep domain knowledge, specialists will still be essential — not because of coding itself, but because of the knowledge required to make correct decisions.

But for most cases, generalists who understand the whole system will be far more valuable than narrow specialists.

Software development will still need people

Despite all the hype, software development won’t disappear.

What will change is the role of the developer.

Developers will increasingly become the bridge between business intent and technical execution. AI lowers the learning curve, but it doesn’t remove the complexity. Someone still needs to:

understand what the business actually wants
translate vague ideas into concrete systems
make trade-offs
define boundaries
ensure quality
manage risk

The “new programmers” will orchestrate execution. They’ll use AI, vibe coding, text-to-code, diagrams, prompts, and automation to transform business needs into working software — faster than ever before.

Conclusion

Every generation of developers believes their tools are the last ones that require “real skill”. And every generation is wrong.

We’ve always built abstractions to move faster. AI is just the most powerful abstraction we’ve seen so far.

The core skill has never been typing code. It has always been thinking clearly, designing systems, and solving real problems.

AI doesn’t replace that. It amplifies it.

If you don’t know what you’re doing, AI will happily help you create a mess — faster than ever.
If you do know what you’re doing, AI becomes a force multiplier.

And that’s why I truly believe this:

AI tools are great if you know what you’re doing.

Personal notes

This text was partially written using AI. The content is mine, thoughts, examples, topics, and logical structure. I just asked IA to transform my notes into a blog post format. This is why I kept the “AI dash” on it.
If you’re a junior developer, don’t focus on just writing code; you should learn the concepts, software patterns, best practices, and soft skills. Achievements and deeds have more weight on your profile than technical skills purely.

Useful Ruby methods and tips that you might not know (or remember)

Thu, 29 May 2025 00:00:00 +0000

“Code is like humor. When you have to explain it, it’s bad.”
— Cory House

Enumerable Methods

Enumerable#tally

The tally method groups and counts elements in an enumerable. It’s great for frequency analysis.

irb> ["apple", "banana", "apple", "orange"].tally
=> {"apple"=>2, "banana"=>1, "orange"=>1}

Enumerable#inject

inject accumulates a result by applying a block to each element. It’s useful for summing, multiplying, and more.

Using with an accumulator:

irb> [1, 2, 3, 4].inject(0) { |sum, num| sum + num }
=> 10

Or passing a method proc as argument:

irb> [1, 2, 3, 4].inject(:+)
=> 10

Enumerable#chunk

chunk groups consecutive elements based on a block condition.

irb> [1, 2, 2, 3, 1, 1].chunk(&:even?).to_a
=> [[false, [1]], [true, [2, 2]], [false, [3]], [true, [1, 1]]]

Enumerable#each_with_object

each_with_object iterates and builds an object like a hash or array.

irb> ["apple", "banana"].each_with_object({}) { |fruit, hash| hash[fruit] = fruit.length }
=> {"apple"=>5, "banana"=>6}

Enumerable#slice_when

slice_when splits an enumerable when the block returns true.

irb> [1, 3, 6, 10].slice_when { |a, b| (b - a) > 3 }.to_a
=> [[1, 3, 6], [10]]

Enumerable#flat_map

flat_map maps and flattens the result in one step.

irb> [[1, 2], [3, 4]].flat_map { |arr| arr.map { |x| x * 2 } }
=> [2, 4, 6, 8]

Hash Methods

Hash#dig

Extracts the nested value specified by the sequence of key objects.

irb> h = { foo: {bar: {baz: 1}}}
irb> h.dig(:foo, :bar, :baz)
=> 1
irb> h.dig(:invalid, :key)
=> nil

Hash#invert

invert swaps keys and values in a hash.

irb> { a: 1, b: 2 }.invert
=> {1=>:a, 2=>:b}

Hash#transform_keys

transform_keys lets you change hash keys with a block.

irb> { a: 1, b: 2 }.transform_keys { |key| key.to_s.upcase }
=> {"A"=>1, "B"=>2}

Hash#transform_values

transform_values creates a new hash with the values transformed by the given block, while keeping the keys unchanged.

irb> { a: 1, b: 2 }.transform_values { |value| value * 10 }
=> { a: 10, b: 20 }

Array Methods

Array#rotate

rotate shifts elements to the left by default, or by a given count.

irb> [1, 2, 3, 4].rotate
=> [2, 3, 4, 1]

Array#combination

combination generates all combinations of a given length.

irb> [1, 2, 3].combination(2).to_a
=> [[1, 2], [1, 3], [2, 3]]

Array#permutation

permutation returns all possible permutations of a given length.

irb> [1, 2, 3].permutation(2).to_a
=> [[1, 2], [1, 3], [2, 1], [2, 3], [3, 1], [3, 2]]

Object Methods

Object#method(:method_name).source_location

Find where a method is defined in your source code.

irb> g = Greeter.new
irb> g.method(:say_it).source_location
=> ["/Users/home/app/models/greeter.rb", 5] 

Object#method(:method_name).arity

Check the number of arguments a method accepts.

irb> foo = "Hello"
irb> foo.method(:slice).arity
=> -2

Object#methods(false)

List only methods defined directly on the object, excluding inherited methods.

irb> obj = Object.new
def obj.greet; "Hello!"; end
irb> obj.methods(false)
=> [:greet]

Kernel Methods

callee

callee returns the current method’s name.

irb> def example
irb>   __callee__
end
irb> example
=> :example

rescue/retry

retry restarts the block after a rescue clause.

irb> attempts = 0
irb> begin
irb>   raise "Error" if (attempts += 1) < 3
irb> rescue
irb>   retry
irb> end
=> nil

BEGIN / END

BEGIN and END define blocks executed before and after the script runs.

BEGIN { puts "Starting..." }
END { puts "Ending..." }
puts "Hello!"

ActiveSupport Methods

Enumerable#compact_blank

compact_blank removes blank elements from an enumerable.

irb> [nil, "", "Hello", [], {}].compact_blank
=> ["Hello"]

Array#index_by

index_by creates a hash where keys are determined by the block.

irb> ["apple", "banana", "pear"].index_by { |fruit| fruit[0] }
=> {"a"=>"apple", "b"=>"banana", "p"=>"pear"}

Miscellaneous

Right Assignment

Assign results in reverse order for cleaner syntax.

irb> Users.where(active: true).all => users
=> nil
irb> users
=> [#<Contact:>,...]

Destructuring Block Arguments

Pattern matching in blocks lets you destructure array elements.

irb> a = [[1, 2], [3, 4]]
irb> a.each do |(first, last)|
irb>   puts "First: #{first}, Last: #{last}"
irb> end
First: 1, Last: 2
First: 3, Last: 4

Calling Procs

You could call procs in different ways:

irb> my_proc = -> arg { puts arg }
#<Proc:0x007fb1ebe9c1a0@(irb):1 (lambda)>

irb> my_proc.call('hello')
hello

irb> my_proc.('hello')
hello

irb> my_proc['hello']
hello

Command line Ruby tips

Given the following files in the current directory:

$ ls -l
-rw-r--r--@ 1 user  group    36 28 Mai 17:57 file1.txt
-rw-r--r--@ 1 user  group     8 28 Mai 17:57 file2.txt
-rw-r--r--@ 1 user  group   613 28 Mai 17:58 file3.txt
-rw-r--r--@ 1 user  group  3143 28 Mai 18:48 photo.jpg
-rw-r--r--@ 1 user  group  1004 28 Mai 18:47 report.ofx

Using -n and -e to Process Lines

-n wraps your code in a loop that reads each line from input. Combine with -e for quick one-liners.

$ ls -1 | ruby -n -e 'puts $_ if $_.match(/\.txt/'
file1.txt
file2.txt
file3.txt

Using -p to Modify Lines in Place

-p is like -n but automatically prints each line. Great for filters and transformations.

$ ls -1 | ruby -p -e '$_ = $_.capitalize'
File1.txt
File2.txt
File3.txt
Photo.jpg
Report.ofx

Using STDIN

The command below returns the total size of all files returned by ls -l command:

$ ls -l | ruby -e 'puts STDIN.to_a.map { |l| l.split[4].to_i }.sum'
4804

I've been writing software for the last 25 years. Here are a few more things I've learned so far (part 2)

Mon, 14 Apr 2025 00:00:00 +0000

The code you write makes you a programmer. The code you delete makes you a good one. The code you don’t have to write makes you a great one.
– Unknown

This is part 2 of the previous post: I’ve been writing software for the last 25 years. Here some things I learned so far

But first, some context

I’m a self-taught generalist (aka full-stack) software engineer experienced mainly in web applications, having a strong focus on pragmatism and problem solving. Over the last 25 years, I’ve worked in small startups, mid-sized companies, and large enterprises—mostly in Linux environments, using open-source languages and tools. Along the way, I’ve also had some experience leading and managing teams.

The lessons below come from years of seeing what works and what doesn’t. Some are the result of success, others from hitting the wall—hard.

Building and Designing Software

Automate your workflow

Automate everything you can—project setup, configuring local environments, creating test data with scripts, etc. If you do it more than once, automate it. Your future self (and teammates) will thank you.

Applications should run locally

If your app can’t run on a dev machine, you’re doing it wrong. Make local setup simple—native or Docker, doesn’t matter. Devs need full control to reproduce problems, debug, and test… locally. Include scripts to generate sample data, simulate API calls, and preload images or documents. These tools are part of the product too.

Design your code to fail

Don’t trust inputs. Add logs where it matters. Write checkers, guard clauses, notifications for edge case scenarios. Expect things to break—and make it easy to recover.

Track requests and responses

Log everything used in third-party integrations—requests, responses, and payloads. You will need it for debugging and audits.

Don’t blindly trust your data

Users send broken data. Bugs introduce garbage into database. Validate everything—on input, on output, on storage. Your system is only as good as the data flowing through it.

Don’t ignore idempotency

If something fails, you should be able to retry it without causing damage. Idempotency is your friend. Don’t ship production code without it.

Be explicit

Write greppable code. Use "mobile" instead of 3 to describe enums. Name things clearly: has_balance, requested_amount, etc. Never assume something is “obvious”. It never is. More on this here: Greppability: A code metric

Testing > Debugging

Relying only on debugging tools could lead you to chase problems after they happen. Testing (especially TDD) helps you move in small, safe steps. You define expectations first, then work towards meeting them. It’s like climbing a mountain one secure foothold at a time.

Avoid unnecessary frameworks

The web is just HTTP—receive a request, send a response. That hasn’t changed in decades. You don’t need heavy frameworks to handle 90% of what you build. Stick with boring, battle-tested tools.

Monoliths aren’t always bad. Microservices aren’t always good.

Don’t split your system into pieces unless you have a really good reason. Scale only when you must. Premature complexity is the real bottleneck.

Worse is better

Keep it simple. Forget hexagonal architecture and all the over-engineered “best practices” that solve problems you don’t have. Worse is better—and often faster, clearer, and easier to maintain.

Don’t try to make it perfect. Make it adaptable.

There’s no silver bullet. Instead of aiming for the perfect solution, aim for something adjustable, configurable, flexible, and easy to change when needed. Prioritize adaptability over perfection.

Shipping and Maintaining Software

You won’t get it right the first time

No matter the architecture, design, or framework, your first attempt won’t be perfect. You’ll need to iterate to make it truly work. So don’t overthink or over-plan. Ship something functional as soon as possible and let feedback guide you.

Monitor and observe your software

Know something’s broken before your users do. Instrument your systems. Set alerts. Pay attention to the data.

Always have a Plan B

Have redundancy. Keep backups. Make sure someone else can step in if needed. Single points of failure—whether in infrastructure or people—are liabilities.

Care about the UX

Interfaces aren’t just forms anymore. Users talk to systems with voice and text (Alexa, ChatGPT, etc.). If your signup form has 12 fields, they’ll probably quit halfway through.

Working with Teams and People

Estimates are bad. Manage expectations instead.

Software estimates have never worked—and never will. Communicate uncertainty. Share tradeoffs. Set expectations early and often.

Extreme visibility

Keep your teammates, managers, and stakeholders aware of what you’re doing, your progress, and what’s blocking you. It builds trust and reduces surprises.

Don’t micromanage

If you don’t trust your team, why are they on the team? Focus on outcomes: what’s the impact on the product, team, or company? What’s blocking them? How can you help? Let people figure things out—it’s how they grow.

Communicate decisions

Why this framework instead of that one? Why this database, this design, this pattern? Write it down. Share it. It prevents repeat mistakes and helps others understand the “why.”

Sharing accelerates your career more than anything else. Teach. Present. Write. Share what you learn, what you build, and what you discover. The more you share, the more you grow.

Don’t postpone obvious decisions

You’re going to make a lot of decisions. Some are big, others are trivial. Don’t waste time overthinking the obvious ones. Make the call, communicate it, and spend your energy where it really matters.

Reject dogma

Take principles like the ones in this post as guidance—not gospel. Don’t follow advice blindly just because someone said it’s best practice. Think. Understand the context. Apply what makes sense.

Wrapping up

After all these years, one thing is clear: writing software isn’t just about code. It’s about people, communication, trade-offs, and learning to live with imperfection. You won’t get everything right. You’ll ship things that break. You’ll rethink decisions. That’s normal. That’s healthy.

What matters is staying curious, being honest about what works (and what doesn’t), and helping those around you grow. The lessons in this post aren’t universal truths—they’re just what I’ve picked up from the trenches. Take what’s useful, challenge what isn’t, and keep moving forward.

And above all: keep it simple, keep it working, and keep learning.

Enjoy o/

How to backup your photos from Google and fix the Exif metadata

Sun, 16 Mar 2025 00:00:00 +0000

Any sufficiently advanced incompetence is indistinguishable from malice.
– Grey’s Law

Downloading your whole backup from Google Photos should be easy, right? Well, not really. Google’s Takeout service makes it way more annoying than it should be. Not only do you have to wait for about 3 days, manually download a bunch of ZIP files from a page with the worst usability I deal it in the last decade, and discover that the files didn’t come with the info metadata on it (which keeps the correct photo date, time, location, etc), but stored separately in JSON files with inconsistent naming.

So, sharing my frustration here and how I dealt with it to clean up Google’s mess and get my photos back in order.

Step 1: Getting the backup from Google Takeout

First, I requested my export from Google Takeout. This took three days to process, since I had about 150GB of photos.

Once it was ready, I had to manually download 72 ZIP files, one by one, because Google apparently hates bulk downloads. Super fun.

Step 2: Extracting everything

After finally downloading all the files, I extracted them into a single directory with this command:

for zip in downloaded/*.zip; do unzip $zip -d extracted/; done

This dumped all the files into a structure like this:

extracted/
  Takeout/
    Google Fotos/
      Photos From 2024/
        image1.jpg
        image1.jpg.supplemental-metadata.json
      Photos From 2025/
        image1.jpg
        image1.jpg.supplemental-m.json
        image1.jpg.supplemental-metadata(1).json
        image1(1).jpg
        image2.jpg
        image2.jpg.sup-meta.json

As you can see, consistency isn’t Google’s strong point. I need the metadata filenames to follow the same pattern because it will be used in the next step.

Step 3: Fixing the metadata filenames

The correct format of metadata files should be:

image.jpg.supplemental-metadata.json

But I found things like:

image.jpg.sup-meta.json
image(1).jpg.supplemental-meta.json

To clean this up, I wrote a Ruby script that:

Finds all metadata files with weird names.
Renames them to match the expected format.
If an image doesn’t have a metadata file, it tries to guess the date from the filename (e.g., IMG_20240515.jpg → May 15, 2024).

Here’s the Ruby script I used:

To run it, simply execute on the terminal:

ruby fix_metadata.rb extracted/Takeout/Google\ Fotos/

The output will be like:

$ ruby fix_metadata.rb ./extracted/Takeout/Google\ Photos/
Total files found on ./extracted/Takeout/Google Photos/**/*: 555
Total photos from YYYY dirs found: 539
Total supported photos formats found: 131
Total metadata files found: 408

Process finalized with 3 errors:
[1/3] Unable to infer metadata for ./extracted/Takeout/Google Photos/Photos from 2013/CameraZOOM-01611c5b98.jpg
[2/3] Unable to infer metadata for ./extracted/Takeout/Google Photos/Photos from 2013/CameraZOOM-20131224200750052.jpg
[3/3] Metadata file: .//extracted/Takeout/Google Photos/Photos from 2014/IMG_1181.JPG.supplemental-metadata(1).json not exist for image: ./extracted/Takeout/Google Photos/Photos from 2014/IMG_1181(1).JPG

Process finalized with 22 fixes:
[1/22] 5142914356_01611c5b98_o.jpg.supplemental-metad.json moved to 5142914356_01611c5b98_o.jpg.supplemental-metadata.json
[2/22] 1-3922-1990-20121024123511.jpg.supplemental-me.json moved to 1-3922-1990-
...
[22/22] IMG_1182.JPG.supplemental-metadata(1).json moved to IMG_1182(1).JPG.supplemental-metadata.json

Metadata not found for 21 files:
[1/21] ./extracted/Takeout/Google Photos/Photos from 2013/CameraZOOM-20131224200623261.jpg
[2/21] ./extracted/Takeout/Google Photos/Photos from 2013/CameraZOOM-20131224200750052.jpg
...
[21/21] ./extracted/Takeout/Google Photos/Photos from 2025/IMG_5854.HEIC

Step 4: Applying the metadata to files

Once the filenames were fixed, I used exiftool to write the correct metadata into the photo files:

exiftool -r -d %s -tagsfromfile "%d/%F.supplemental-metadata.json" \
  "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" \
  "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" \
  "-GPSLongitudeRef<GeoDataLongitude" "-Keywords<Tags" "-Subject<Tags" \
  "-Caption-Abstract<Description" "-ImageDescription<Description" \
  "-DateTimeOriginal<PhotoTakenTimeTimestamp" \
  -ext "*" -overwrite_original -progress --ext json -ifd0:all= \
  extracted/Takeout/Google\ Fotos/

(just change the last argument to the directory where you extract the photos)

This does a bunch of things:

Reads the correct timestamp from the JSON file.
Writes it into the photo’s EXIF data.
Restores GPS location and descriptions.
Fixes bad tags in one go.

Wrapping Up

Now that all the metadata is fixed, I can import everything back into iCloud Photos (my case) or any other storage service without messing up the timeline.

Let me know if you have any other tricks or improvements!

References

Monitoring my swimming pool temperature with a cheap BLE sensor and ESPHome

Tue, 03 Sep 2024 00:00:00 +0000

One accurate measurement is worth a thousand expert opinions.
– Grace Hopper

Maintaining the right temperature for my swimming pool is crucial for enjoying a good swim. Instead of frequently checking a thermometer or relying on a mobile app, I wanted to get real-time temperature updates remotely. Here’s how I achieved this using an affordable BLE sensor and ESPHome.

BLE Sensor

I found a suitable BLE sensor on AliExpress that fit my requirements perfectly:

Power efficiency: the sensor runs on two AAA batteries, lasting about six months.
Precision: it has good precision with an error of just 1° C approximately.
Data collecting: it works well with its dedicated mobile app, but doesn’t integrate with third-party apps.

Despite its limitations, the sensor was perfect for my project. However, I needed a way to access the temperature data in real-time without using the mobile app or being near the pool.

ESPHome

To bridge this gap, I used an ESP32-WROOM-32U, which I also bought from AliExpress. The external antenna on this model ensures a reliable connection over a longer range.

The ESP32 will work just as a BLE <> Wi-Fi bridge, connecting to the sensor on each hour, reading the temperature and sending the data to Home Assistant sensors.

Here’s how I set it up on ESPHome config:

esphome:
  name: "esphome-pool-monitor"
  friendly_name: Pool Monitor

esp32:
  board: esp32dev
  framework:
    type: arduino

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: "wmIdKCoFredactedNGeGZHeQI="

ota:

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

time:
  - platform: homeassistant
    id: homeassistant_time
    on_time:
      - seconds: 0
        minutes: 30
        then:
          - switch.turn_on: ibs_p01b_switch
          - delay: 5s
          - while:
              condition:
                and:
                  - switch.is_on: ibs_p01b_switch
                  - binary_sensor.is_off: ibs_p01b_connected
              then:
                - logger.log: "Trying connection to IBS-P01/B"
                - esp32_ble_tracker.stop_scan:
                - delay: 5s
                - esp32_ble_tracker.start_scan:
                - delay: 1min

esp32_ble_tracker:
  scan_parameters: 
    active: false
    continuous: false
    duration: 1min
  on_ble_advertise:
    - mac_address: "XX:XX:XX:XX:XX:XX"
      then:
        - lambda: |-
            ESP_LOGD("ble_adv", "IBS-P01/B device found");

button:
  - platform: template
    name: "BLE Start Scan"
    on_press:
      - esp32_ble_tracker.start_scan:
  
  - platform: template
    name: "BLE Stop Scan"
    on_press:
      - esp32_ble_tracker.stop_scan:

  - platform: restart
    name: "ESP32 Restart"

binary_sensor:
- platform: status
  name: "ESP32 Status"

- platform: template
  id: ibs_p01b_connected
  icon: mdi:bluetooth-connect
  name: "IBS-P01/B Connected"

ble_client: 
  - mac_address: XX:XX:XX:XX:XX:XX
    id: ibs_p01b
    on_connect:
      then:
        - wait_until:   
            lambda: |-
              esphome::ble_client::BLEClient* client = id(ibs_p01b);
              esphome::ble_client::BLECharacteristic* chr = client->get_characteristic(0xFFF0, 0xFFF2);

              return chr != nullptr;              
        - lambda: |- 
            ESP_LOGD("ble_client_lambda", "Connected to IBS-P01/B");
            id(ibs_p01b_connected).publish_state(true);

    on_disconnect:
      then:
        - lambda: |-
            ESP_LOGD("ble_client", "Disconnected from IBS-P01/B");
            id(ibs_p01b_connected).publish_state(false);

switch:
  - platform: ble_client
    id: ibs_p01b_switch
    ble_client_id: ibs_p01b
    name: "IBS-P01/B Enabled"
    restore_mode: ALWAYS_OFF

sensor:
  - platform: uptime
    name: "ESP32 Uptime"

  - platform: template
    device_class: timestamp
    name: "IBS-P01/B Last Measurement"
    id: "ibs_p01b_last_measurement"

  - platform: template
    name: "IBS-P01/B Temperature"
    id: ibs_p01b_temperature
    unit_of_measurement: "°C"
    accuracy_decimals: 1
    state_class: measurement
    device_class: temperature

  - platform: ble_client
    id: ibs_p01b_sensor
    type: characteristic
    ble_client_id: ibs_p01b
    service_uuid: 'fff0'
    characteristic_uuid: 'fff2'
    notify: true
    internal: true
    update_interval: 30s
    lambda: |-
      if (x.size() == 0) return NAN;

      // https://community.home-assistant.io/t/hex-string-to-dec-value/458945/2
      auto temp = (float)((int16_t)(x[1]<< 8) + x[0])/100;

      if (temp != NAN) {
        id(ibs_p01b_temperature).publish_state(temp);

        // https://community.home-assistant.io/t/publish-timestamp-into-text-sensor/122531/11
        id(ibs_p01b_last_measurement).publish_state(id(homeassistant_time).now().timestamp);

        id(ibs_p01b_switch).turn_off();
      }

      return 0.0;

Notes:

For obvious reasons, I redacted the mac address of my device.
You can find the mac address and inspect the characteristics for your sensor with apps like BT Inspector or LightBlue.
IBS-P01/B is the name/model of my sensor. You can use other devices for this purpose since they have bluetooth support.
I’ll not cover the details for setting up ESPHome and devices, but you can watch a detailed tutorial from Everything Smart Home Youtube channel.

After uploading the firmware to ESP32 and with a little luck, you will see something like that on the logs:

[16:24:04][D][switch:012]: 'IBS-P01/B Enabled' Turning ON.
[16:24:04][D][switch:055]: 'IBS-P01/B Enabled': Sending state ON
[16:24:05][D][button:010]: 'BLE Start Scan' Pressed.
[16:24:05][D][esp32_ble_tracker:266]: Starting scan...
[16:24:11][D][ble_adv:069]: IBS-P01/B device found
[16:24:11][D][esp32_ble_client:110]: [0] [XX:XX:XX:XX:XX:XX] Found device
[16:24:11][D][esp32_ble_tracker:665]: Found device XX:XX:XX:XX:XX:XX RSSI=-82
[16:24:11][D][esp32_ble_tracker:686]:   Address Type: PUBLIC
[16:24:11][D][esp32_ble_tracker:215]: Pausing scan to make connection...
[16:24:11][D][esp32_ble_tracker:303]: End of scan.
[16:24:11][I][esp32_ble_client:067]: [0] [XX:XX:XX:XX:XX:XX] 0x00 Attempting BLE connection
[16:24:24][I][ble_sensor:031]: [ibs_p01b_sensor] Connected successfully!
[16:24:26][I][esp32_ble_client:227]: [0] [XX:XX:XX:XX:XX:XX] Connected
[16:24:26][D][ble_client_lambda:115]: Connected to IBS-P01/B
[16:24:26][D][binary_sensor:036]: 'IBS-P01/B Connected': Sending state ON
[16:24:32][D][sensor:094]: 'IBS-P01/B Temperature': Sending state 22.77000 °C with 1 decimals of accuracy
[16:24:32][D][sensor:094]: 'IBS-P01/B Last Measurement': Sending state 1722626688.00000  with 1 decimals of accuracy
[16:24:32][D][sensor:094]: 'ibs_p01b_sensor': Sending state 0.00000  with 0 decimals of accuracy
[16:25:02][D][sensor:094]: 'IBS-P01/B Temperature': Sending state 22.77000 °C with 1 decimals of accuracy
[16:25:02][D][sensor:094]: 'IBS-P01/B Last Measurement': Sending state 1722626688.00000  with 1 decimals of accuracy
[16:25:02][D][sensor:094]: 'ibs_p01b_sensor': Sending state 0.00000  with 0 decimals of accuracy
[16:25:43][D][ble_client:121]: Disconnected from IBS-P01/B
[16:25:43][D][binary_sensor:036]: 'IBS-P01/B Connected': Sending state OFF
[16:25:43][W][ble_sensor:037]: [ibs_p01b_sensor] Disconnected!

And, of course, the data in Home Assistant device:

Wrapping Up

It’s working but it’s not finished. I need to keep the ESP32 turned on and near the pool, running continuously.

I used a standard USB 5V DC adapter plugged into a wall outlet near the pool. To protect the hardware from the elements, I 3D printed a box to accommodate both the ESP32 and the DC adapter:

If you want to print the same box for your project, download the model from my Printables profile.

This setup has been running flawlessly for the past 10 months, providing me with accurate and (almost) real-time water temperature, making pool maintenance much easier. From here, I can create automations to turn the heater on, send a message on family group when the temperature is ideal, and much more.

Enjoy o/

References

I've been writing software for the last 25 years. Here some things I learned so far

Thu, 01 Aug 2024 00:00:00 +0000

The computer was born to solve problems that did not exist before.
– Bill Gates

Understand the problem

Processes are secondary, doesn’t matter if it is Agile, Kanban or Waterfall: you just need to have an exact understanding of the problem to be solved and plan/organize your work around that. And, of course, be able to adapt. Changes will occur during the execution and it’s fine; Flexibility and adaptability are key to effectively solve the right problem.

Protip: First, solve the problem. Then, write the code. (John Johnson)

Master at least one programming language

Throughout your career, you’ll likely work with a dozen programming languages, including popular ones like Java, JavaScript, Python, Go, and Ruby. However, it’s essential to choose one language that balances execution speed and development efficiency and master it. Becoming an expert in one language allows you to write cleaner, more efficient code, reducing the time spent debugging and maintaining projects. The cost of development and maintenance often outweighs execution costs, so prioritize languages that streamline your workflow and enhance productivity. Learn it thoroughly, become a master, and be the go-to person for that language. This can be a game-changer in your career.

Protip: become a contributor to any non-hyped and open-source language like Ruby or Go.

Know to negotiate

Negotiation skills are essential. Whether you’re discussing project requirements with stakeholders, negotiating salaries, or resolving conflicts within your team, being able to negotiate effectively is a valuable asset. Equally important is knowing when and how to say “no”. Setting boundaries and managing expectations ensures that you can deliver well-aligned solutions without burning out in the process.

Protip: read The Art Of Saying NO book.

Use Linux, for real

Mastering Linux is fundamental. Be the admin of your machine. Dive deep into commands, get comfortable with the terminal, and embrace text files. This knowledge will be invaluable throughout your career. Also, once you embrace the Unix philosophy, you’ll start to create more cohesive and reusable code by thinking in a modular and efficient manner. By breaking tasks into simpler, independent components that do one thing well, your solutions become more elegant and maintainable.

Protip: use Linux on your personal computer, all the time.

Understand the business

Learn the business context of your software and your company. Understanding how your work fits into the bigger picture allows you to make more informed decisions, align your efforts with business goals, and communicate more effectively with non-technical stakeholders. This insight not only enhances your value as a software engineer but also enables you to deliver more value to your company by ensuring that your work directly supports and advances business objectives.

Protip: read The Product-Minded Enigneer article.

Write simple and boring code

You don’t need to follow any strict or complex practice by the book like Clean Code, SOLID, OOP, DRY-ish, functional, etc. Just focus on writing dumb readable and self-explanatory code that could be easily searched with grep or any other search mechanism. To be clear, avoid metaprogramming or any technique that makes your code less readable. A good test is to show your code to junior developers; if they understand it, you’ve succeeded.

Embrace boring writing practices and prioritize the most explicit coding style you can. Don’t need to be fancy here, just keep in mind to be as readable and simple as possible, and you’ll be good.

Protip: embrace the boring code-style and be proud of it.

Build trust and reputation by delivering software

Trust is the most valuable soft skill in your career. If people trust you, you’ll progress more easily. Building trust involves consistently delivering quality work and communicating effectively. Earn reputation points by shipping working software. When you deliver reliable software, you gain respect, which you can use to negotiate raises, make significant decisions, and lead and inspire other developers.

Protip: read Ship it! book.

Master a database thoroughly

Understanding a database inside and out is crucial. Master at least one popular DBMS; My recommendation is PostgreSQL. Learn how query analysis works, how data types are managed, the better way to use indexes, and some of the database internals. Know what practices to use and the ones to avoid. A significant cause of low performance or bottleneck is often related to the database or its improper use. This deep understanding will help you optimize your applications and troubleshoot performance issues, ensuring your software runs smoothly and efficiently.

Protip: read The Art Of PostgreSQL book.

Be a specialized generalist

You’re not a React, Java or Go developer; you are a problem solver. Your primary role is to solve problems, which sometimes requires stepping out of your comfort zone. Be proficient in the basics of front-end development, documentation, testing, and other areas. You don’t need to be an expert in everything, but a well-rounded skill set will make you a more effective problem solver.

Protip: read The Pragmatic Programmer book.

Don’t be a hostage to tools

While it’s great to have at disposal the latest and greatest tools, fancy IDEs with integrated AI, visual tools to drag and drop blocks of codes with ease, never stray too far from raw programming. Master command-line tools and simple editors like Vim (or Emacs), command-line tools like curl, git and Unix utilities like grep and pipe operator. The ability to work efficiently in the terminal is invaluable. Simpler tools often offer greater flexibility and control, and being proficient in them can enhance your overall effectiveness.

Protip: use VIM (or any open source editor you like).

Packaging matters

Delivery isn’t just about the code itself. Include concise documentation that outlines what your code does, and what it doesn’t, evidence of tests, benchmarks, monitoring instructions, and problem-reporting guidelines. The presentation and packaging of your work are crucial, much like how Apple emphasizes the unboxing experience to stand out from competitors. Remember, you aren’t the end user of your solution, so consider the experience from their perspective and ensure it is as smooth and informative as possible.

Protip: put a neat README file on your projects.

Test all the fucking time!

The hallmark of a professional developer is thorough testing. Write tests for your code, embrace Test-Driven Development (TDD), and prioritize automated testing over traditional QA processes. Continuous testing ensures your code is robust and reduces the likelihood of bugs making it to production.

Protip: test all the fucking time!

Focus on production

Production is where it all matters. All other environments like staging or homologation are a waste of time. Use feature flags and rolling out strategies: deploy your code disabled on production, enable it, monitor it, and move on to the next task. Other environments or processes can be useful for specific scenarios, but they shouldn’t detract from the primary goal of delivering reliable software to production.

Protip: read Testing in Production, the safe way article.

In my 25 years of experience, my method has remained largely the same. My techniques have improved, and my toolbelt has grown, but my approach to problem-solving is consistent: identify the problem, set your goal, find solutions, test them, choose the best one for your context, make it work, and ship it.

Enjoy o/

How to integrate unsupported Tuya devices on Home Assistant

Fri, 26 Jul 2024 00:00:00 +0000

The nice thing about standards is that you have so many to choose from.
― Andrew Tanenbaum

I’ve using Home Assistant for about 3 years now, for some basic automations like turning devices on/off on determinated periods and for energy production/consumption monitoring.

Recently I bought this Energy Meter device from AliExpress to monitor the energy consumption in my house. It works fine on the Tuya App, but it doesn’t work with the official Home Assistant integration, showns as “unsupported”:

Despite that, when I check the device logs on Tuya Cloud platform, it have all the data that I need:

So, I just need a way to get this data from Tuya’s API. Let’s go!

Tuya Cloud setup

You just need the client_id and client_secret to access Tuya’s API. If you already have the integration configured, jump to next section.

If you need a more detailed guide to configure Tuya Coud account, just follow this guide or watch this video.

Fetching device properties

The script bellow call the Query Properties endpoint for a single device. So, all data reported to Tuya Cloud will be returned in the response.

import sys
import hashlib
import hmac
import json
import urllib
import urllib.parse
import logging

from urllib.request import urlopen, Request
from datetime import datetime

def make_request(url, params=None, headers=None):
    if params:
        url = url + "?" + urllib.parse.urlencode(params)
    request = Request(url, headers=headers or {})

    try:
        with urlopen(request, timeout=10) as response:
            return response, response.read().decode("utf-8")

    except Exception as error:
        return error, ""

def get_timestamp(now = datetime.now()):
    return str(int(datetime.timestamp(now)*1000))

def get_sign(payload, key):
    byte_key = bytes(key, 'UTF-8')
    message = payload.encode()
    sign = hmac.new(byte_key, message, hashlib.sha256).hexdigest()
    return sign.upper()

def get_access_token():
    now = datetime.now()

    timestamp = get_timestamp(now)
    string_to_sign = client_id + timestamp + "GET\n" + \
        EMPTY_BODY + "\n" + "\n" + LOGIN_URL
    signed_string = get_sign(string_to_sign, client_secret)

    headers = {
            "client_id": client_id,
            "sign": signed_string,
            "t": timestamp,
            "mode": "cors",
            "sign_method": "HMAC-SHA256",
            "Content-Type": "application/json"
            }

    response, body = make_request(BASE_URL + LOGIN_URL, headers = headers)

    json_result = json.loads(body)["result"]
    access_token = json_result["access_token"]
    return access_token

def get_device_properties(access_token, device_id):
    url = ATTRIBUTES_URL.format(device_id=device_id)
    timestamp = get_timestamp()
    string_to_sign = client_id + access_token + timestamp + "GET\n" + \
        EMPTY_BODY + "\n" + "\n" + url
    signed_string = get_sign(string_to_sign, client_secret)

    headers = {
            "client_id": client_id,
            "sign": signed_string,
            "access_token": access_token,
            "t": timestamp,
            "mode": "cors",
            "sign_method": "HMAC-SHA256",
            "Content-Type": "application/json"
            }

    response, body = make_request(BASE_URL + url, headers = headers)

    json_result = json.loads(body)
    properties = json_result["result"]["properties"]
    output = {j['code']: j['value'] for j in properties}

    return output

EMPTY_BODY     = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
BASE_URL       = "https://openapi.tuyaus.com"
LOGIN_URL      = "/v1.0/token?grant_type=1"
ATTRIBUTES_URL = "/v2.0/cloud/thing/{device_id}/shadow/properties"

if len(sys.argv) != 2:
    raise SystemExit("usage: python3 tuya.py device_id")
_, device_id = sys.argv

client_id     = "tuya_client_id"
client_secret = "tuya_client_secret"

access_token = get_access_token()
attributes   = get_device_properties(access_token, device_id)
json_output  = json.dumps(attributes)

print(json_output)

Paste this code on a tuya.py file inside Home Assistant config dir and change the values for client_id and client_secret obtained from the previous step. You can hardcoded or place on secrets.yaml file.

The device_id could be obtained on Tuya App on device details.

After log in on Home Assistant terminal, test the integration running:

python3 /config/tuya.py device_id

If everything works fine, the output will be a json with all device attributes. In my case, returned this:

{"EnergyConsumed": 1079569, "Current": 14932, "ActivePower": -605, "ReactivePower": 0, "Frequency": 60, "Temperature": 434, "DeviceStatus": 10}

Nice, you have a working script that fetches values from Tuya’s API. Now, let’s add this to Home Assistant.

Creating a sensor on Home Assistant

You just need to create a command_line sensor adding this into configuration.yaml file:

command_line:
    - sensor:
        name: Tuya Power Clamp
        unique_id: tuya_power_clamp
        command: "python3 /config/tuya.py device_id"
        device_class: power
        state_class: total_increasing
        unit_of_measurement: Wh
        scan_interval: 60
        value_template: "{{ value_json.EnergyConsumed | float(0) * 10 }}"
        json_attributes:
          - EnergyConsumed
          - Current
          - ActivePower
          - ReactivePower
          - Frequency
          - Temperature
          - DeviceStatus

The device_class, state_class, unit_of_measurement and value_template will depend of the type of your device. Check Home Assistant documentation for more details about this.

After restarting Home Assistant, you could find the sensor on Devices & services settings:

If you need to poll more devices, just create a new sensor section and change the device_id value.

Enjoy o/

References

How to unbrick your wi-fi router after a bad OpenWRT firmware flashing

Wed, 02 Nov 2022 00:00:00 +0000

The Linux philosophy is ‘Laugh in the face of danger’. Oops. Wrong One. ‘Do it yourself’. Yes, that’s it.
― Linus

About 9 years ago I wrote a post explaining how to turn your wi-fi router in a NAS and Media Server with OpenWRT (pt-BR) and it’s was a huge success. By far is my most viewed post until now.

In the post I warned about the risks of doing that procedure since you could end with an expensive paperweight. And so, I was caught in my own trap: I tried to update the OpenWRT version and bricked my router 🤦‍♂️

Mistakes were made

Skip to next section if you just want to know how to unbrick it. No judging.

The router was running flawlessly for about 8 years, no issues at all. But going against the common sense that says “don’t change the team that was winning”, I decided to update the OpenWRT to the latest supported version. And instead of flashing the update image recommended by OpenWRT, I choose to do it from scratch to have a “clean” installation. What’s could go wrong?

First step, revert to stock firmware downloaded directly from the TP Link official site. All good, the router rebooted with the original firmware running. Second step, flash the OpenWRT image. Selected the .bin file, uploaded it, the router rebooted and… nothing. No leds blinking, no signal of booting procedure, only the power led static on.

At this point, I accepted that the router was bricked. So, let’s try to bring it back to life.

TFTP recovery

Following the official unbrick article from OpenWRT, basically the bootloader has an embedded recovery mode used to transfer the firmware image file through tftp. So, just need to to this:

Connect a cable on your PC to a LAN port of the router and configure the ethernet connection with the fixed IP 192.168.0.66 mask 255.255.255.0; [use tcpdump to discover the IP for your router, as explained on official article]
Download and configure a TFTP server on your PC. Linux and Mac should have a pre-installed command line but not worked for me. I’m using MacOS and the best solution that I found was PumpKIN. Just download and open the application;
Disconnect from wi-fi or any other active network; [recommended]
Power the router on and after all leds turn off (about 1 second later), press and hold the reset button for about 4 seconds until the last led turn on and keep this way;
Back to PunpKIN, the router will request the file wdr4300v1_tp_recovery.bin. Just rename the firmware image to requested filename and put it on configured dir on PunpKIN.
If everything works, the file will be copied to router in about 30 seconds. Wait more 2 minutes to image be installed on router (if you are a lucky one) and reboot it.

But for me, didn’t worked. The file was copied but after a reboot, the router continue doing the same (which was nothing).

TFTP and RS232 (serial)

The easy way didn’t worked. Something went wrong after copy the firmware to router though tftp and I need to see what’s going on.

The only way to open a TTY terminal to router is though serial interface, but it’s not trivial either: you need to open the device, connect/solder some cables on serial pins and use a serial adapter on your PC.

Luckily I bought an USBtoUART dongle from AliExpress a couple of years ago:

Used dupont wire jumpers to connect the UART to router board serial contacts. Remember to connect the TX and RX on inverted way. RX on dongle will read from TX of board and vice versa, as follows:

TXD ---------------- RXD
RXD ---------------- TXD
GND ---------------- GND

Now with the wires connected to router board, just power it on. It will “work” normally and you’ll be able to debug the serial using a terminal on PC.

In my case, I just used screen:

screen /dev/tty.SLAB_USBtoUART 115200

And as soon I saw the bootlog, I confirmed my suspect:

U-Boot 1.1.4 (Jun 17 2013 - 12:31:57)
U-boot DB120
DRAM:  128 MB
id read 0x100000ff
flash size 8MB, sector count = 128
Flash:  8 MB
Using default environment
PCIe Reset OK!!!!!!
In:    serial
Out:   serial
Err:   serial
Net:   ag934x_enet_initialize...
No valid address in Flash. Using fixed address
 wasp  reset mask:c03300
WASP  ----> S17 PHY *
: cfg1 0x7 cfg2 0x7114
eth0: ba:[redacted]:41
athrs17_reg_init: complete
eth0 up
eth0
Autobooting in 1 seconds
## Booting image at 9f020000 ...
   Uncompressing Kernel Image ... Stream with EOS marker is not supportedLZMA ERROR 1 - must RESET board to recover

The firmware image could be corrupted (or invalid). The boot process is happening but fails to boot the kernel on address 9f020000.

Then I tried to flash the image again. Maybe this time I can see the error on terminal. Just repeated the TFTP recovery process described earlier and got this:

U-Boot 1.1.4 (Jun 17 2013 - 12:31:57)
U-boot DB120
DRAM:  128 MB
id read 0x100000ff
flash size 8MB, sector count = 128
Flash:  8 MB
Using default environment
PCIe Reset OK!!!!!!
In:    serial
Out:   serial
Err:   serial
Net:   ag934x_enet_initialize...
No valid address in Flash. Using fixed address
 wasp  reset mask:c03300
WASP  ----> S17 PHY *
: cfg1 0x7 cfg2 0x7114
eth0: ba:[redacted]:41
athrs17_reg_init: complete
eth0 up
eth0
dup 1 speed 1000
Using eth0 device
TFTP from server 192.168.0.66; our IP address is 192.168.0.86
Filename 'wdr4300v1_tp_recovery.bin'.
Load address: 0x80060000
Loading: #################################################################
#################################################################
#####################################################
done
Bytes transferred = 8258048 (7e0200 hex)
original_product_id = ffffffff
 original_product_ver = ffffffff
 recovery_product_id = 10430001
 recovery_product_ver = 01
 auto update firmware: product id verify fail!
Autobooting in 1 seconds
## Booting image at 9f020000 ...
   Uncompressing Kernel Image ... Stream with EOS marker is not supportedLZMA ERROR 1 - must RESET board to recover

The image was transferred but the “auto update firmware” fails to verify the product id ¯\_(ツ)_/¯.

So, I’ll need to write the firmware update manually.

Manual firmware flashing

Luckily there is a way to install the firmware manually thanks to this post:

Restart the router and after the message “Autobooting in 1 second” type “tpl” and hit Enter. This need to be done in less than 1 second, but if you are fast enough, the boot will stop on prompt “db12x>” and you can run commands manually;
Run “tftpboot” to start the TFTP server with default parameters. Take note of the server IP and “Load address: 0x81000000”, it’ll be used later. Hit ctrl+c to stop the tftp server;
Configure your PC ethernet interface to IP shown in the previous step and prepare PunpKIN to put the firmware image again;
Start the TFTP server again with the load address obtained on step 2 and with the filename already defined on TFTP transfer earlier:
db12x> tftpboot 0x81000000 wdr4300v1_tp_recovery.bin
After firmware was transferred, erase the destination flash 0x9F020000 (which we know from the initial captured output) and destination length being the size of the transferred firmware image 0x7C0000 bytes:
db12x> erase 0x9f020000 +7c0000
Now just need to copy the transferred firmware to the destination flash:
db12x> cp.b 0x81000000 0x9f020000 0x7c0000
All good. Reboot the router and pray:
db12x> reset

Follow below the full command logs:

U-Boot 1.1.4 (Jun 17 2013 - 12:31:57)
U-boot DB120
DRAM:  128 MB
id read 0x100000ff
flash size 8MB, sector count = 128
Flash:  8 MB
Using default environment
PCIe Reset OK!!!!!!
In:    serial
Out:   serial
Err:   serial
Net:   ag934x_enet_initialize...
No valid address in Flash. Using fixed address
 wasp  reset mask:c03300
WASP  ----> S17 PHY *
: cfg1 0x7 cfg2 0x7114
eth0: ba:[redacted]:41
athrs17_reg_init: complete
eth0 up
eth0
Autobooting in 1 seconds
db12x>

db12x> tftpboot
dup 1 speed 1000
*** Warning: no boot file name; using '6F01A8C0.img'
Using eth0 device
TFTP from server 192.168.1.100; our IP address is 192.168.1.111
Filename '6F01A8C0.img'.
Load address: 0x81000000
Loading: T T T T T T
TFTP error: 'not found' (1)
Starting again
(ctlr+c pressed)
Abort

db12x> tftpboot 0x81000000 wdr4300v1_tp_recovery.bin
Using eth0 device
TFTP from server 192.168.1.100; our IP address is 192.168.1.111
Filename 'wdr4300v1_tp_recovery.bin'.
Load address: 0x81000000
Loading: #################################################################
#################################################################
#################################################################
############################
done
Bytes transferred = 8126464 (7c0000 hex)

db12x> erase 0x9f020000 +7c0000
First 0x2 last 0x7d sector size 0x10000                                                                                                                                                                  125
Erased 124 sectors

db12x> cp.b 0x81000000 0x9f020000 0x7c0000
Copy to Flash... write addr: 9f020000
done

db12x> reset

The triumph

If you do all commands right and with a bit of lucky, the router should boot normally and the LuCi be accessible on 192.168.1.1 after few minutes.

Enjoy o/

References

Ruby: is Time to talk about Time Zones

Mon, 04 Jul 2016 12:00:00 +0000

Any sufficiently advanced bug is indistinguishable from a feature.
― Rich Kulawiec

TL;DR

Always use the Time class and correctly stores the time zone information.

Time

Based on floating-point second intervals from unix epoch (1970-01-01);
Has date and time attributes (year, month, day, hour, min, sec, subsec);
Natively works in either UTC or “local” (aka support to time zones);
Can handle negative times before unix epoch;
Can handle time arithmetic in units of seconds;
Is faster than Date or DateTime implementation;

utc   = Time.utc(2016, 10, 15)   # => 2016-10-15 00:00:00 UTC
utc.zone                         # "UTC"
utc.utc_offset                   # 0

local = Time.local(2016, 10, 15) # => 2016-10-15 00:00:00 -0300
local.zone                       # "BRST"
local.utc_offset                 # -10800

time  = Time.new(2016,10,15,14,35,42,"-03:00") # 2016-10-15 14:35:42 -0300
time.zone                                      # nil
time.utc_offset                                # -10800

now   = Time.now             # 2016-05-16 04:10:23 -0300
unix  = now.to_i             # 146338262
time  = Time.at(unix)        # 2016-05-16 04:10:23 -0300
day   = (24 * 60 * 60)       # 86400
time  = Time.at(unix + day)  # 2016-05-17 04:10:23 -0300

Note: there’s a Time class one that is part of core Ruby and there is an additional Time class that is part of the standard library. The standard library Time class extends the core Time class by adding some methods. See the documentation of both classes for more details.

Date

Based on integer whole-day intervals from an arbitrary “day zero” (-4712-01-01);
Has date attributes only (year, month, day);
Can handle date arithmetic in units of whole days;
Can convert between dates in the ancient Julian calendar to modern Gregorian;
Don’t have support to time zones;
Is slower than Time implementation;

require "date"

today = Date.today    # #<Date: 2016-06-19 ((2457559j,0s,0n),+0s,2299161j)>
today.to_time         # 2016-06-19 00:00:00 -0300

time = "2016-06-19 23:11:07 -0300"
date = Date.parse(time) # #<Date: 2016-06-19 ((2457559j,0s,0n),+0s,2299161j)>
date.to_time            # 2016-06-19 00:00:00 -0300

DateTime

Subclass of Date, so whatever you can do with Date can be done with DateTime;
Based on fractions of whole-day intervals from an arbitrary “day zero” (-4712-01-01);
Has date and time attributes (year, month, day, hour, min, sec);
Can handle date arithmetic in units of whole days or fractions;
Is slower than Time and Date implementation;

require "date"

now = DateTime.now
=> #<DateTime: 2016-06-19T23:11:07-03:00 ((2457560j,7867s,653070000n),-10800s,2299161j)>
now.to_time        # 2016-06-19 23:11:07 -0300

time = "2016-06-19 23:11:07 -0300"
date = DateTime.parse(time)
=> #<DateTime: 2016-06-19T23:11:07-03:00 ((2457560j,7867s,0n),-10800s,2299161j)>
date.to_time       # 2016-06-19 23:11:07 -0300

Further, it has a meaningless “zone” attribute and a hidden usec after turning it into a Time instance:

dt = DateTime.new(2012, 12, 6, 1, 0, 0, "-07:00", 123456)
dt.zone       # => "-07:00"
dt.utc?       # => NoMethodError: undefined method `utc?'
dt.utc_offset # => NoMethodError: undefined method `utc_offset'

dt = DateTime.parse("2016-06-19 23:11:07.456789 -0300")
dt.usec         # => NoMethodError: undefined method `usec'
dt.to_time.usec # => 456789

In short, unless you’re dealing with astronomical events in the ancient past and need to convert the Julian date (with time of day) to a modern calendar, you don’t need to use DateTime.

Benchmark

require 'benchmark'
require 'date'

Benchmark.bm(10) do |x|
  x.report('date')     { 100000.times { Date.today   } }
  x.report('datetime') { 100000.times { DateTime.now } }
  x.report('time')     { 100000.times { Time.now     } }
end

Result:

                user     system      total        real
date        1.250000   0.270000   1.520000 (  1.799531)
datetime    6.660000   0.360000   7.020000 (  7.690016)
time        0.140000   0.030000   0.170000 (  0.200738)

As shown before, the Time class are faster and should fits the most usage scenarios.

Time Zone

Knowing that Time has total support to time zones, you just remember this mantra: “Aways handle time with the correct time zone”.

Why time zone is important?

If you application are deployed in the cloud (which probably was), you can’t guarantee the time zone of the server running you application;
Your application database probably will be located on different server and again, you can’t guarantee the server time zone configuration;
Since you application is on Internet, your users came from all corners of world, that have 24 different time zones;

Getting current time with Time.now actually gets the current process time zone, which is defined by the TZ variable:

$ date
Sun Jul  3 18:42:33 BRT 2016

$ ruby -e "puts Time.now"
2016-07-03 18:42:33 -0300

$ TZ="America/Vancouver" ruby -e "puts Time.now"
2016-07-03 14:42:33 -0700

Getting the current time

Your application need to know the “base time zone”, which means that all different time zones should be converted to this base time zone to be handled correctly.

Consider that you application time zone is -03:00 (BRT). In order to ensure that you always are getting the correct time, you could do this:

TZ = "-03:00"

server_time = Time.now                   # 2016-06-18 05:30:22 +0200
server_time.utc_offset                   # 7200

current_time = server_time.getlocal(TZ)  # 2016-06-18 00:30:22 -0300
current_time.utc_offset                  # -10800

That’s it. Don’t trust on the time zone of Time.now and always convert to your local time zone.

You can ease this repetitive task extending the Time class this way:

TZ = "-03:00"

module TimeExtensions
  def current
    now.getlocal(TZ)
  end
end
Time.extend(TimeExtensions)

Time.now        # 2016-06-18 05:30:22 +0200
Time.current    # 2016-06-18 00:30:22 -0300

Also, is a good idea to always parse time with the correct time zone:

TZ = "-03:00"

require "time"

Time.parse("2016-06-18 15:33:44")        # 2016-06-18 15:33:44 +0200
Time.parse("2016-06-18 15:33:44 #{TZ}")  # 2016-06-18 15:33:44 -0300

This way, you guarantee that the input time, read from a form or a file, always will be handled with the correct time zone (if it wasn’t supplied).

Serialization

In order to ensure that the time zone information will not be lost on time serialization, is a good practice to use the ISO 8601 standard:

require "time"

now = Time.current  # 2016-06-18 20:30:22 -0300
now.iso8601         # "2016-06-18T20:30:22-03:00"

For parsing, the iso8601 also handle the time zone and is a very fast alternative to Time.parse method:

require "time"

data = "2016-06-18T20:30:22-03:00"

Time.iso8601(data) # 2016-06-18 20:30:22 -0300
Time.parse(data)   # 2016-06-18 20:30:22 -0300

On APIs, you should use the ISO 8601 standard to exchange date and time values. Period.

Using TZInfo

TZInfo is a gem that allow to use named time zones in order to obtain and convert time instances, also providing daylight saving information.

In the context of getting the right time zone, you can use TZInfo to get the utc_offset, also considering the dst, of a time zone without need to hard coded it.

For example, instead of hard code the -03:00, you can obtain this offset this way:

require "tzinfo"

tzinfo = TZInfo::Timezone.get("America/Sao_Paulo")
period = info.current_period

offset = period.offset.utc_total_offset
=> -10800   # 3 hours

And if you are in daylight saving time, this will be considered:

require "tzinfo"

tzinfo = TZInfo::Timezone.get("America/Sao_Paulo")
period = info.current_period

period.dst?
=> true

offset = period.offset.utc_total_offset
=> -7200    # 2 hours

This way, just need to encapsulate this on a TimeExtensions class to ease its usage:

module TimeExtensions
  require "tzinfo"

  def tzinfo
    TZInfo::Timezone.get(TZ)
  end

  def utc_offset
    period = tzinfo.current_period
    period.offset.utc_total_offset
  end

  def current
    now.getlocal(utc_offset)
  end
end
Time.extend(TimeExtensions)

Time.tzinfo     # #<TZInfo::DataTimezone: America/Sao_Paulo>
Time.utc_offset # -10800
Time.current    # 2016-06-18 00:30:22 -0300

Storing the time zone on database

Many guides and best practices articles say to “always store the time in UTC time zone into database”. I think that this is partially true. And I say this: “always store the time on the same time zone with a data type that supports time zone”.

With time zone support

If you are using PostgreSQL, for example, just use the timestamptz or timetz data types to store time values and you are good to go.

Your database adapter should be able to handle this data type and made the conversions to the correct time zone.

For example, consider the following posts table with the modified_at column as timestamp and the published_at column as timestamptz:

    Column    |            Type             | Modifiers
--------------+-----------------------------+-----------
 title        | text                        |
 modified_at  | timestamp without time zone |
 published_at | timestamp with time zone    |

And the Sequel model Post, with the databased configured to UTC as default time zone:

Sequel.database_timezone    = :utc
Sequel.application_timezone = :local

class Post < Sequel::Model
end

When a model was created, Sequel convert the time zone to match the time zone defined on table schema:

post_time = Time.parse("2016-07-03 20:33:44 -03:00")

Post.create(
  title:        "testing tz is cool",
  modified_at:  post_time,
  published_at: post_time
)

=> #<Post:0x007fb8d0808f78> {
         :title => "testing tz is cool",
   :modified_at => 2016-07-03 20:33:44 -0300,
  :published_at => 2016-07-03 20:33:44 -0300
}

Note that both time values have the right time zone offset, but on the database, they was stored this way:

    title           |     modified_at     |      published_at
--------------------+---------------------+------------------------
 testing tz is cool | 2016-07-03 23:33:44 | 2016-07-03 20:33:44-03

On the column modified_at, which type is timestamp, the value was converted to UTC and stored without time zone information. But the column published_at, which type is timestamptz, the original time zone was stored without conversion.

Only the application know that that the modified_at column should be converted to the local time and the published_at column have the right time zone information.

If other services need to access this database, like a data analytic tool (BI), this information should be shared between consumers. I consider this a data loss since the time zone was lost on the storage. Using data type that supports time zone ensure the consistency of the data.

Without time zone support

If your database don’t have a data type with time zone support storage, you have two alternatives: store time values in UTC or store the time zone offset on a different column.

MySQL, for example, don’t have a specific type to store time zone information. The TIMESTAMP data type can converts the time to UTC for storage and back from UTC to current time zone for retrieval using the time zone setted in the connection, but don’t writes the time zone information on column.

The Rails way

If you are using Ruby on Rails, you don’t need to concern with all this time and time zone related issues. You just need to remember this:

Always configure your application time zone on config/application.rb:

config.time_zone = 'Brasilia'

Always get the current time via current method:

Time.current
=> Sun, 03 Jul 2016 22:46:35 BRT -03:00

Always parse time using the configured time zone:

Time.zone.parse("2016-07-03 22:33:45")
=> Sun, 03 Jul 2016 22:33:45 BRT -03:00

And if you are using a database with time zone supports, instead of the classic t.timestamps, do this on database migrations:

create_table :posts do |t|
  # columns definition
end
add_column :posts, :created_at, :timestamptz
add_column :posts, :updated_at, :timestamptz

The non-Rails way

If you are not using Rails, eventually you will need to handle time parsing and/or time zones conversions. You could do this by yourself, implementing these operations on a helper class or extending the Time behavior. You will learn a lot in the process.

But if you are looking for a ready-to-use solution, you could use the CoreExt gem and pick only the Time extension:

require "core_ext/time"

Time.zone = "America/Sao_Paulo"
Time.zone.parse("16:35:42")   # Sun, 03 Jul 2016 16:35:42 BRT -03:00

now = Time.current            # Sun, 03 Jul 2016 20:31:10 BRT -03:00
now + 2.days                  # Tue, 05 Jul 2016 20:31:10 BRT -03:00
now.iso8601                   # "2016-07-03T20:31:10-03:00"

10.days.ago                   # Fri, 23 Jun 2016 20:31:10 BRT -03:00

This gem is a fork of ActiveSupport with many changes to make it more modular and focused only on the extensions of Ruby core classes. You could read more about the motivation and usage examples on my previous post about Hanami migration: From Rails to Hanami (Lotus) Part 3: Sidekiq Workers, Sequel Plugins, I18n, Timezone issues and Core Extensions.

Ruby 2.4 to_time bugfix

Until Ruby 2.3, the methods to_time of Date and DateTime objects did not preserve the time zone:

require 'date'

time = DateTime.strptime('2016-06-21 PST', '%Y-%m-%d %Z')
=> #<DateTime: 2016-06-21T00:00:00-08:00 ((2457561j,28800s,0n),-28800s,2299161j)>

time.zone    # "-08:00"
time.to_time # 2016-06-21 05:00:00 -0300

On Ruby 2.4 the time zone will be preserved:

require 'date'

time = DateTime.strptime('2016-06-21 PST', '%Y-%m-%d %Z')
=> #<DateTime: 2016-06-21T00:00:00-08:00 ((2457561j,28800s,0n),-28800s,2299161j)>

time.zone    # "-08:00"
time.to_time # => 2016-06-21 00:00:00 -0800

Conclusion

Time zones exists in our real world and should be respected on software engineering. Ruby provide a very useful API to handle Time but a little deficient one to handle time zones.

Know the Ruby Time capabilities and always remember to use and store the correct time zone on database.

References

Learn something new about Ruby time or time zones? Consider to share this post with your friends or co-workers. For questions, comments or suggestions, use the comments below. Code hard and success!