Don’t spend a lot of time on this, because it isn’t a full blown feature. It’s just one little thing the client needs. Don’t go overboard, just quick and simple.

You can’t control how other people think. You can only control you. The client and sales have already had the meeting. They’ve already convinced themselves this is a quick easy feature. You can’t change this. But you can remind them of the policy:

All committed code needs tests.

Have this policy! Follow it! When you say no, you are a disgruntled programmer who doesn’t want to do work. When the policy requires tests, you are doing everything you can to get that request implemented. It will just take some time.

I was recently asked to build a small, one off workflow for a client. I was met with a bit of a shock when I asked why we weren’t making this a full feature. They honestly wanted to save time. They didn’t want this to get in the way. They didn’t want to go through the work of designing and implementing a new feature. I had just a couple questions:

Does this have to work correctly every time?
Does this have to integrate with the system without breaking anything else?

This a feature. It needs to be tested. It needs to work. It needs the time to be built correctly. Have the policies in place that allow you to build software the way it should be done.

The post Yes, it needs tests appeared first on Ryan Day.

I use SQLAlchemy in my Flask apps. I typically issue queries similar to

query_result = db.session.query(MyModel).first()

I knew I was running out of memory on my entry level Digital Ocean droplets. That is when the segfaults started. I stumbled across https://pypi.python.org/pypi/memory_profiler and created a quick test program to profile a single function that was causing problems.

Line # Mem usage Increment Line Contents
================================================
56 46.0 MiB 0.0 MiB   @profile
57                    def get_bounced_emails():
58 48.2 MiB 2.2 MiB       emails = db.session.query(EmailAddress).\
59 48.3 MiB 0.0 MiB           join(send_history, send_history.c.email_id == EmailAddress.email_id).\
60 372.1 MiB 323.9 MiB        filter(send_history.c.bounce == 1).\
61                            all()
62
63 372.1 MiB 0.0 MiB      email_dict = {}
64 374.1 MiB 2.0 MiB      for email in emails:
65 374.1 MiB 0.0 MiB          email_dict[email.email_address] = True
66
67 314.9 MiB -59.2 MiB    del emails
68 314.9 MiB 0.0 MiB      return email_dict

The memory profiler package makes it real easy to see the memory consumption of your application line by line. In this function, we are just retrieving a ton of records. They are taking up a lot of memory.

This is where I learned a little bit about SQLAlchemy. This function only needs the email_address field of the record, but I’m retrieving the entire record. So I made a change to simply grab the email_address record.

Line # Mem usage Increment Line Contents
================================================
56 46.0 MiB 0.0 MiB   @profile
57                    def get_bounced_emails():
58 47.1 MiB 1.1 MiB       emails = db.session.query(EmailAddress.email_address).\
59 47.1 MiB 0.0 MiB           join(send_history, send_history.c.email_id == EmailAddress.email_id).\
60 81.2 MiB 34.1 MiB          filter(send_history.c.bounce == 1).\
61                            all()
62
63 81.2 MiB 0.0 MiB       email_dict = {}
64 84.2 MiB 3.0 MiB       for email in emails:
65 84.2 MiB 0.0 MiB           email_dict[email[0]] = True
66
67 65.9 MiB -18.2 MiB     del emails
68 65.9 MiB 0.0 MiB       return email_dict

Wow. By just grabbing that one field I seriously reduced the memory consumption of this method. I had under estimated just how much overhead there is in pulling the entire record in a query.

Now I take this into account when building queries. Just what do I need form the DB? Then I only grab that.

The post SQLAlchemy and Memory appeared first on Ryan Day.

Given my hourly bill rate, this is really stupid. I should just buy the tortillas.

OR

Spend the hour each week making tortillas. Stop spending the 3 other hours each week Googling tortilla recipes.

The post Time Management appeared first on Ryan Day.

The real goal, of course, is to practice writing down thoughts in a logical order that can be understood by others. The secondary goal (as of now) is to improve my signal to noise. That can’t happen by scribbling down my insane thoughts every day. Unless I had an easily identifiable pattern of clarity to insanity. But I don’t think that’s something you can plan.

So I’ve privately jotted down thoughts for revision, and not posted the horrible bits. This helps me organize my ideas and doesn’t clutter the internet.

Huzzah!

The post Silent Win appeared first on Ryan Day.

I certainly have enough half finished thoughts in my drafts folder. I’m thinking I can stretch this out two weeks. Maybe three. I can’t simply ramble, I have to do something productive. So with that in mind.

– – – – –

I missed the product release notes yesterday (it’s Saturday). So I’m finalizing them them now. I put out release notes every Friday afternoon so everyone on the team can keep up with the new features and resolved bugs. It also lets people know I’m not sleeping all day. That isn’t a joke. It is actually important to do this. Especially when you’re a back end developer.

I keep an open notepad all week, and add important features and fixes that people care about. Those are the notes I share. I always have one line: “Lots of small bug fixes and refactoring“. That is a catch all for everything I don’t talk about.

Keeping an open notepad for release notes forces me to do at least 2 or 3 things each week that people will notice. I may spend 4 long grueling days profiling and writing unit tests for an Objective C API we are rolling out. On the fifth day, I know I better write a new graph report. Or maybe update some email templates to look better on mobile. Or anything where people can see the output. Otherwise the rest just doesn’t matter. It really doesn’t. Non-developers only ever work with production level software, usually from very large teams. They only see software that, generally, works very well. But they have no idea what goes into it.

On the plus side, they have no idea how long it takes to change the color of a button. So take all day. 3 seconds to change the color. 8 hours to refactor and write tests for the rest of the system. So thats what I do. I spend a day every week making highly visible changes, no matter how simple.

It’s a balancing act. I really think the difference between a normal coder and the mythical 10Xer is how you spend the other 4 days.

The post More Writing appeared first on Ryan Day.

My previous post focused specifically on the security implications of storing keys in your code repository. This was a particularly harmful situation with a client project I had joined. But as I think about it a little more, security isn’t the reason I use the environment to store configuration. Security is simply a byproduct. The reasons I use the environment in my company are:

1. When bringing up a new contractor or employee, I simply generate an IAM role and a key to our internal API. I plug those into the “Developer” environment IDE config file, and the new developer is up and running.

2. It’s easy to move developers between environments. We are writing V2 of our internal API as I write this. Its really quick to change your environment to build features in V2, or fix bugs in V1.

3. It felt much quicker to have a config.py pull from the environment than to parse command line parameters for our large number of configuration options or to add a dependency for an ini parser.

Reading this short list, there is really nothing here that forces us to use the environment. A config file would be just fine. Maybe I should think about that.

What concerns should I examine?

My application servers have 3 user accounts: nobody, alembic deploy user, root.

That’s it. Access to any one of those three accounts would give an attacker access to our secret keys and database. It wouldn’t matter if the configuration values were in the environment or a file. An attacker could read secrets from disk or from memory. So I don’t really gain or lose anything from using the environment instead of a config file.

My application starts up with an environment set, and reads the values from the environment into config variables.

Not only could this be done with a config file, that’s what is actually happening. The environment parameters are still stored on disk. When the server reboots, it reads the file, sets up the environment, and launches the application. So the “environment” part is simply a mirror of the setup on the developer’s system to keep production and development code consistent. So I may have the downside of both designs from a security perspective.

Very low probability of adding new config options.

When we do add a new environment config, it’s ok that it isn’t an automated process. It doesn’t happen frequently so having to redeploy our configuration manually isn’t a problem.

What do I gain from using the environment?

I don’t run the risk of a developer accidentally checking in their config file. However, if that did happen I would only have to reset their IAM credentials and internal API key.

What do I gain from using a config file?

I may only be gaining “not using the environment”. I think a successful attack on our system would happen regardless of how we stored our secrets.

Conclusions

Well I’m not sure. The Moving Fase article helped me, because we aren’t cleaning up the application environment after setting the config variables. So that is a security enhancement I will make. Other than that, I don’t think changing from the environment to a config file would give much benefit in a security or functionality sense.

The post Configuration in the Environment appeared first on Ryan Day.