tag:blogger.com,1999:blog-61044204350219040822024-03-16T02:33:42.659-05:00The Programmer's ParadoxSoftware is a static list of instructions, which we are constantly changing.Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.comBlogger435125tag:blogger.com,1999:blog-6104420435021904082.post-65950660451147066702024-03-14T23:00:00.001-05:002024-03-14T23:00:00.255-05:00Software Development DecisionsA good decision in a software development project is one that moves you at least one step closer to getting the work completed with the necessary quality.<br /><br />A bad decision is one where you don’t get a step forward or you trade off a half step forward for one or more steps backward. <br /><br />Even a fairly small software development project includes millions and millions of decisions. Some decisions are technical, dealing with the creation or running of the work. Some are usability, impacting how the system will solve real problems for real people.<br /><br />A well running software development project mostly makes good decisions. You would look at the output of the project and have few complaints about their choices.<br /><br />A poor software development project has long strings of very poor choices, usually compounding into rather substandard output. The code is a mess, the config is fragmented, the interfaces are awkward, the data is broken, etc. It is a whole lot of choices that make you ask ‘Why?’<br /><br />If you look at the project and cannot tell if the choices were good or bad then you are not qualified to rate the work. If you cannot rate it, you have no idea whether the project is going well or not. If you don't know, then any sort of decision you make about the work and inject into the project is more likely to be harmful than helpful. <br /><br />Which is to say if you do not immediately know if a decision is right or wrong, then you should push that decision to someone who definitely does know and then live with their choices. They may not be good, depending on the person you choose, but your chances of doing any better are far less.<br /><br />In a project where nobody knows enough to make good decisions, it is highly unlikely that it will end well. So, at bare minimum, you can't rush the project. People will have to be allowed to make bad decisions, then figure out the consequences of those mistakes and then undo the previous effort and replace it all with a better choice. It will slow down a project by 10x or worse. If you try to compress that, the bad decisions will become frozen into the effort, start to pile up, and then it will take even longer. <br /><br />That is, if you do not have anybody to make good decisions and you are still in a rush, the circumstances will always get way worse. It’s like trying to run to the store, but you don’t know where the store is, so you keep erratically changing directions, hoping to get lucky. You probably won’t make it to the store and if you do it will certainly have taken way longer than necessary. <br /><br />If there is a string of poor choices, you have to address why they happened. Insanity is doing the same things over and over again, expecting the results to change. They will not change on their own.Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-78751643273939359312024-03-07T23:00:00.000-06:002024-03-07T23:00:00.139-06:00RatchetingYou know the final version will be very complicated. But you need to get going. It is way too long to lay out a full and complete low or medium level design. You’ll just have to wing it.<br /><br /><div>The best idea is to rough-in the structure and layers first. <br /><br />Take the simplest case that is reflective of the others. Not a “trivial” special case, but something fairly common, but not too ugly. Skip the messy bits.<br /><br />Code the overall structure. End to end, but not fully fleshed out.<br /><br />Then take something it is not yet doing and fill in more details. Not all of them, just more. If there are little bugs, fix them immediately but do it correctly. If it means refactoring stuff underneath, do it now. Do not cheat the game, as it will hurt later if you do.<br /><br />Then just keep that up, moving around, making it all a little more detailed, a little more complicated. Keep making sure that what’s there always works really well. Build on that.<br /><br />Ratchet up step by step. Small focus changes, fix any bugs large or small. Make sure the core is always strong. Sprinkle in more and more complexity.<br /><br />This is not the fastest way to code. It causes a lot of refactoring. It requires consistency. You need to be diligent and picky. You might cycle dozens of times, depending on the final complexity, but that gives you lots of chances to edit it carefully. The code has to be neat and tidy. This is the opposite of throw away code.<br /><br />Although it takes longer, I usually find that since the quality is far better, the testing and bugs get hugely reduced, usually saving more time than lost.<br /></div>Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-78350024024520946232024-02-29T23:00:00.000-06:002024-02-29T23:00:00.243-06:00CodingMajor points:<br /><ol style="text-align: left;"><li>Coding is always slow</li><li>Coding produces both code & bugs</li><li>The code always needs to be edited, the first version is just roughed in.</li><li>Do not use disposable code in industrial strength projects.</li></ol>The primary goal is to produce a minimal amount of readable code. <br /><br /><div>You want the code to be as small as possible, it is easier to deal with. Larger codebases are worse, not better.<br /><br />You want the code to be as readable as possible so it is easier to edit. If it is a choice between small or readability, readability wins. If it is a choice between readable or performance, readability wins. <br /><br />You can always fix readable code later. But it must be reable first, and remain readable afterwards.<br /><br />You don’t want the code to be redundant, cause you’ll always forget to change all of the different manifestations of it for the same bug. Redundancies are bugs or potential bugs. Changes to redundant code cause the code to drift apart. <br /><br />You need the codebase to be tightly organized so that it is easier to find and fix the problems. You can accidentally waste more time fixing bugs, than in coding/refactoring, so you need to optimize for that. <br /><br />There should be one and only one place to put each line of code. All of the code should be in its one place. If there are lots of different places where you can put the code, you are disorganized.<br /><br />The author is not the only one who needs to reread the code. Others will have to read it as well. Good code will be read by a lot of people. <br /><br />Because you didn’t just magically get it right the first time, you and other people will have to go over the code, again and again, in order to make it better. Code doesn’t get written, it evolves.<br /><br />The fiction that layers are bad is poor advice. Layers are the main way you keep the code organized. Without them, the code is just one huge flat mess. That is far worse, it is totally unreadable. <br /><br />Layers can be abused, there can be too many of them. But not having any at all is worse. It is easier to remove a layer than add one.<br /><br />A good function is specific to some part of the computation. It is general. It does the one thing that it says it does, nothing more. Sub-level detail processing is below it, in other functions. High-level flow is above it. Once you know what it does, and trust that it does exactly and only that, then you can ignore it, which makes life easier.<br /><br />All data should come into the code from somewhere else. It should never be hardcoded in the code, it should not be hardcoded when passed down to the code. Thus all strings, integers, constants, etc. are suspect. The best code has zero hardcoded values.<br /><br />If you need to do a bunch of steps each time for a common action, wrap the steps. The fewer things you call the better your code is. If you rely on remembering that all things have to always be done together, either you’ll forget or someone else who never knew will do it incorrectly. Either way, it is now a bug, and it may not be an obvious one, so it will waste time.<br /><br />If you need to move around some data, all together. Wrap the data, in a struct, object, whatever the language supports. Composite variables are far better than lots of independent variables. <br /><br />If you need to decompose the data (aka parse) to use it somewhere, decompose it once and only once. Keep it decomposed in a struct, object, etc. move it around as a composite.<br /><br />If the call for some library/technology/etc. is messy, wrap it. Wrapping is a form of encapsulation, it helps to avoid bugs and reduce complexity.<br /><br />If there are strange lines of code that are nonintuitive or don’t make sense, wrap them. At minimum, it gives you a chance to name it appropriately; at maximum, it leaves just one place to change it later. <br /><br />Too many functions are way better than too few. If you have to get it wrong, create a billion functions. They force you to have to find reasonable names for the parts of work you are doing. If you don’t know how to name a function, then you don’t understand what you are doing. If you have too many functions it is easy to compact them. If you have too few, you are screwed.<br /><br />Don’t use language features if you don’t understand them. The goal of coding for a system is not to learn new technology, it is to write industrial-strength code that withstands the test of time. If you want to play, good, but don’t do it in a real project, do it in little demos (which can be as messy as you want).<br /><br />Do not pack lines. Saving yourself a few lines of code, but packing together a whole bunch of mechanics, just hides the mechanics and misguides you as to the amount of code you have. Separate out each and every line of code, it doesn’t take any real time and it lays out the mess in its full ugliness. If the mess is ugly fix that, don’t hide it.<br /><br />Never do the same thing in a system in two or more different ways. You need to do something, do it one way and only one way, wrap it in a function, and reuse it in all other instances. This cuts down on complexity. By a huge amount. It cuts down on code, thus it cuts down on bugs.<br /><br />Build up the mechanics to work at a higher level. That is, if you need an id to get to a user, and the user to get to their profile, then you should have a FindUser(id) which is supplied to the call FindProfile(user). Build up reusable pieces, don’t code down into stuff.<br /></div>Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-39976576619036172572024-02-22T20:00:00.000-06:002024-02-22T20:00:00.137-06:00Self-Inflicted PainThe difference between regular programmers and 10x programmers is not typing speed. In some cases, it is not even knowledge.<br /><br />It is that 10x programmers are aware of and strongly avoid self-inflicted injuries while coding.<br /><br />That is, they tend to avoid shortcuts and work far smarter than harder. They don’t tolerate a mess, they don’t flail at their work. <br /><br />They need some code, they think first before coding, they code what they need, and then they refine it rapidly until it works. Then they leverage that code, over and over again, to save crazy large amounts of time. This is why their output is so high.<br /><br />If you watch other programmers, they jump in too fast. They don’t fully understand what they are doing. The code gets messier and messier. Debugging it sinks through massive effort. Then they abandon that work and do it all over again for the next part that is similar. They burn through time in all the wrong places.<br /><br />All of these are self-inflicted injuries. <br /><br />Writing code when you only half understand what it should do will go badly. It’s not that you should be able to predict the future, but rather that given your knowledge today it should span the code you write. If there is something you don’t understand, figure that out before starting to code. If you have to change the code later because things change, that is okay. But if you are coding beyond your current knowledge it will go badly and eat through time.<br /><br />Trying to fix crappy code is a waste of time. Clean it up first, then fix it. If the code doesn’t clearly articulate what it was supposed to do, then any perceived bug may be predicated on top of a whole lot of other bugs. Foundations matter. <br /><br />So, when debugging, unless it is some crazy emergency patch, you find the first bug you encounter and correct that first. Then the next one. Then the next one. You keep that up until you finally find and fix the bug you were looking for. Yes, it takes way longer to fix that bug, but not really, as you are saving yourself a lot of time down the road. Those other bugs were going to catch up with you eventually. <br /><br />If you see bad names, you fix those. If you see disorganization, you fix it, or at a minimum write it down to be fixed later. If you see extra variables you get rid of them. If you see redundant functions, you switch to only using one instance. If you see poorly structured code or bad error handling, you fix that. If you see a schema or modeling problem, you either fix it now or write it down to fix it later. The things you wrote down to fix later, you actually fix them later.<br /><br />The crap you ignore will always come back to haunt you. The time you saved by not dealing with it today is tiny compared to the time you will lose by allowing these problems to build up and get worse. You do not save time by wobbling through the code, fixing it at random. Those higher-level fixes get invalidated by lower-level changes, so they are a waste of time and energy.<br /><br />And then, the biggest part. Once you have some good code that mostly does what you want it to do, you leverage that. That is, putting minimal effort into highly redundant code is as slow as molasses. Putting a lot of effort into a piece of code that you can use over and over again is exponentially faster. Why keep solving the same lower-level problems again and again, when instead you can lift yourself up and solve increasingly higher-level problems at faster speeds? That is the 10x secret.<br /><br />If you have to solve the same basic problems again and again, it is self-inflicted. If you lose higher-level work because of lower-level fixes, it is self-inflicted. If you have to do spooky things on top of broken code, it is often self-inflicted. If you get lost or confused in your own code, it is self-inflicted. If you want to be better and faster at coding and to have less stress in your job, stop injuring yourself, it isn’t helping.<br />Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-44812953858571890622024-02-15T20:00:00.000-06:002024-02-15T20:00:00.126-06:00A Rose by Any Other NameNaming is hard. Very hard. Possibly the hardest part about building software. <br /><br />And it only gets harder as the size of the codebase grows, since there are far more naming collisions. Code scales very, very badly. Do not make it worse than it has to be.<br /><br />This is why naming things correctly is such a fundamental skill for all programmers. <br /><br />Coding itself is oddly the second most important skill. If you write good code but bury it under a misleading name, then it doesn’t exist. You haven’t done your job. Eventually, you’ll forget where you put it. Other people can’t even find it. Tools like fancy IDEs do not save you from that fate. <br /><br />There are no one-size-fits-all naming conventions that always work correctly. More pointedly there can never be such a convention. Naming is not mindless, you have to think long and hard about it. You cannot avoid thinking about it. <br /><br />The good news is that the more time you spend trying to find good names, the easier it gets. It’s a skill that takes forever to master, but at least you can learn to not do it badly.<br /><br />There are some basic naming rules of thumb: <br /><br />First is that a name should never, ever be misleading. If the name is wrong, it is as bad a name as possible. If someone reads it and comes to the wrong conclusion, then it is the author's fault. When you name something you have to understand what that thing is and give it the best possible name. <br /><br />Second is that the name should be self-describing. That is, when someone reads the name, they should arrive at the right conclusion. The variable should hold the data they expect. The function should do what it says. The repo that holds a given codebase should be obvious. <br /><br />“Most people never see the names I use in my code …”<br /><br />No, they do see them. All of them.<br /><br />And if they see them and they are poor or even bad, they will recommend that your code gets rewritten. They will throw away your work. Nothing else you did matters. If the code is unreadable, it will not survive. If it doesn’t survive, you aren't particularly good at your job. It’s pretty simple. <br /><br />Occasionally, some really awful code does get frozen way deep in a ball of mud. But that unfortunate situation is not justification for you being bad at your job. Really, it isn’t.<br /><br />Third, don’t put litter into your names. Made up acronyms, strange ‘pre’ or ‘post’ text. Long and stupid names are not helping. Stop typing in long crazy names, spend some time to thinking about it. Find short reasonable names that are both descriptive and correct.<br /><br />Fourth, don’t put in irrelevant or temporary stuff in there either. If some unrelated thing in an organization changes and now the name is either wrong or needs to be changed, you did it wrong. Names should be nearly timeless. Only if the nature of the problem changes, should they need changing, and you should do that right away. Names that used to be correct suck.<br /><br />Names are important. They form the basis of readability, and unreadable code is just an irritant. If you were asked to really write some code, you need to really write it properly. If it takes longer, too bad. Good naming only slows you down until you get better at it. You need to be better at it.Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-21808924464604828382024-02-07T23:00:00.000-06:002024-02-07T23:00:00.145-06:00Natural DecompositionsGiven a large problem, we start by breaking it down into smaller, more manageable pieces. We can then solve all of the smaller problems and combine them back together to solve the original problem.<br /><br />The hiccup is that not all decompositions are created equal. If you break a big problem down into subparts, when they have any sort of cross dependencies with each other you can’t work on them independently. The dependencies invalidate the decomposition. <br /><br />So we call any decomposition where all of the subparts are fully independent a ‘natural’ decomposition. It is a natural, complete, hard ‘line’ that completely separates the different parts.<br /><br />Do natural decompositions actually exist?<br /><br />Any subpart that has no dependencies on other outside parts is fully encapsulated. It is a black box. <br /><br />A black box can have an interface. You can put things into the box. It’s just that whatever happens in the box stays in the box. You don’t need to know anything about how the box works inside, just on the outside.<br /><br />A car engine is a good example. You put in fuel, and you play with the pedals, then the car moves. If you are just driving around, you don’t need to know much more than that. Maybe if you are pushing it on the highway or a racetrack, you’d need to understand gearing, acceleration, or torque better, but to go to the grocery store with an automatic transmission it isn’t necessary.<br /><br />Cars have fairly good natural decompositions. They are complex machines, but most people don’t really need to understand how they work. Mechanics and race car drivers do.<br /><br />Software though is much harder to decompose because it isn’t visible. The lines between things can be messed up and awful, but very few people would know this. A five wheeled car/truck/motorbike monstrosity would be quickly discounted in reality, but likely survive as a software component.<br /><br />Although we don’t see it the same way, we can detect when a decomposition is bad. The most obvious test is that if you have to add a line of code, how many places are there that it would fit reasonably? The answer should be one. If that is not the answer then the lines are blurred somewhere.<br /><br />And that is the crux. A good decomposition eliminates the degrees of freedom. There is just one place for everything. Then your code is organized if everything is in its one place. It’s simple, yet not simple at all.<br /><br />For example, If you break off part of the system as a printing subsystem, then any and all code that is specifically tied to printing must be in that subsystem. <br /><br />Now it’s not to say that there isn’t an interface to the printing subsystem. There is. Handling user context and the specific gui contexts is done elsewhere and must be passed in. But no heavy lifting is ever done outside. Only on the inside. You might have to pass in a print-it-this-way context that directs what is done, but it only directs it from the outside, the ‘doing it’ part is inside the box.<br /><br />One of the hardest problems in software is getting a group of programmers to agree on defining one place for all of the different types of code and actually putting that code in the one place it belongs. <br /><br />It fails for two reasons. The first is that it is a huge reduction in freedom. You aren’t free anymore to put the code anywhere. The culture of programming celebrates freedom, even when it makes our lives way harder or even tragic. <br /><br />The other reason is in making it quick and easy for newer programmers to know where to put stuff. If we fully documented all of those places it would be far too much to read, and if we don’t most people won’t read the code to try to figure it out for themselves. Various standards and code reviews have tried to address it over the decades, but more often than not people just create a mess and pretend like they didn’t. Occasionally you see large projects with good discipline, it happens.<br /><br />This shows up in other places too. Architecture is the drawing of lines between things. An enterprise architect should draw enough lines in a company to keep it organized; a system architect should draw enough lines in a system for the same effect. Again, these lines need to be natural to be useful. If they are arbitrary they make the problems worse not better.<br /><br />Decomposition is the workhorse of software development, but it's far too easy to get it wrong. Fortunately it’s not hard to figure out if its wrong and fix it. Things go a lot smoother when the decompositions are natural and the work is organized. Programming is hard enough sometimes, we don’t need to find ways to make it worse.<br />Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-10833075780720915132024-02-01T20:00:00.000-06:002024-02-01T20:00:00.127-06:00Anti-patterns“Calling something an anti-pattern is an anti-pattern.”<br /><br />There are lots of ways to accomplish things with software. Some of them are better than others. But the connotation for the term ‘anti-pattern’ is that the thing you are doing is wrong, which is often not the case. <br /><br />Realistically, the ‘pattern’ part of the phrase is abused. A design pattern is just a generalized abstraction of some work you are doing. It is less specific than an ‘idiom’. It is less specific than a ‘data structure’. It is just essentially a code structuring arrangement that is common. That is, it is effectively a micro-architecture, a way to structure some functionality so that it is easier to understand and will behave as expected.<br /><br />So, mostly what people mean when they call something an anti-pattern is just that it is not the ‘best’ alternative. But even if it is not the best, that doesn’t make it a bad choice. Or basically, the set of alternatives for coding is not boolean. It’s not a case of right or wrong. It’s a large gradient, there are a huge number of ways to code stuff, some are better than others. And sometimes, for some contexts, a lesser approach is actually better. <br /><br />We saw this in the 70s with sorting, but the understanding doesn’t seem to have crystalized. <br /><br />There are lots of different ways to sort, with different performance. We can track ‘growth’ which is effectively how an algorithm performs relative to the size of the data. A cute algorithm like a pivot sort has nearly optimal growth, it is O(log N). Bubble sort however is considerably worse at O(N^2). <br /><br />So, you should always implement a pivot sort if you have to implement your own sort? <br /><br />No. If you have a large amount of data to sort, then you probably want to spend the time to implement a pivot sort. But… if you usually only have a few things to sort, then just putting in a bubble sort is fine. <br /><br />Why? <br /><br />Because the code for a bubble sort is way, way easier to write and visually validate. And performance is not even close to an issue, the set of data is always too small. With that tiny size, it wouldn’t matter if one algorithm was a few instructions different from the other, since it doesn't loop long enough for that to become a meaningful time. <br /><br />So, in that reduced context, the shorter, easier, less likely to have a bug, code is the better alternative. More significantly, for front-end devs, whipping together a bubble sort is fine, for back-end ones, learning to implement pivot sorts is better. <br /><br />But in modern programming, since most stacks implement pretty darn good sorting, the issue is moot. It is presented in data structure courses as a means of learning how to correctly think about implementation details, rather than an explicit skill.<br /><br />In modern terms, I’m sure that a lot of people would incorrectly call a bubble sort an anti-pattern, which it is not. Most ‘lesser’ patterns are not anti-patterns. An actual anti-pattern would be to have multiple copies of the same globals, when what you really just needed was one. Another less common anti-pattern would be using string splits as the way to parse LR(1) grammars, as it would never, ever work properly but that is a longer and far more difficult discussion. <br /><br />In general though, the software industry has a real problem with using hand waving to summarily dismiss significant technical issues. Programmers resort to claiming that something “right” or “wrong” far too quickly, when neither case applies. It is a form of boolean disease, caused by spending too much of the day crafting booleans, you start to see the rest of the world only in those terms.<br /><br />Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-74306968109170246472024-01-25T23:00:00.001-06:002024-01-25T23:00:00.148-06:00ContextWhen I discuss software issues, I often use the term ‘context’. I’ll see if I can define my usage a little more precisely.<br /><br /><div>In software programs we talk about state. The setting of a boolean variable is its state. There are only two states. <br /><br />For variables with larger ranges, i.e. possible settings, there can be a huge number of possible states, they are all discrete. An integer may be set to 42.<br /><br />We usually use state to refer to a group of variables. E.g. the state of a UI is its settings, navigation, and all of the preferences. <br /><br />Context is similar, but somewhat expanded. The context is all of the variables, whether explicit or implicit; formal or informal. It is really anything at all that can vary, digitally or even in reality. <br /><br />Sometimes people just restrict context to purely digital usages, but it is far more useful if you open it up to include any informal variability in the world around us. That way we can talk about the context of a UI, but we can also talk about the context of the user using that UI. The first is a proper subset of the second.<br /><br />The reason we want it to be wider than, say just a context in the backend code is because it affects our work. Software is a solution to one or more problems. Some of those problems are purely digital, such as computations, persistence, or communications, but most of our problems are actually anchored in reality. <br /><br />For instance, consider a software system that inventories cogs created at a factory. The cogs themselves and the factory are physical. The software mirrors them in the computer in order to help keep track of them. So, some of the issues that affect the cogs, the factory, or the types of usage of the system, are really just ‘informal’ effects of reality. What people do with the software is heavily influenced by what happens in the real world. The point of an inventory system is to help make better real world decisions.<br /><br />We may or may not map all of those physical influences onto digital proxies, but that does not mitigate their effect. They happen regardless. So if there are real events happening in the factory that affect the cogs but are not captured correctly, the digital proxies for those cogs can fall out of sync. We might have the wrong counts in the software for example because a bunch of cogs went missing. <br /><br />As well, the mappings between reality and the software can be designed incorrectly. The factory might have twenty different types of cogs, but the software can only distinguish ten different types. The cogs themselves might relate to each other in some type of hierarchy, but the software only sees them as a flat inventory list.<br /><br />In that sense the software developers are not free to model the factory and its cogs in any way they choose. The context in reality needs to properly bound the software context. So that whatever happens in the larger context can be correctly tracked in the software context.<br /><br />The quality of the software is rooted in its ability to remain correct. Bad software will sometimes be wrong, so it is not trustworthy, thus not too useful.<br /><br />Now if the factory was very complex, it might be a huge amount of work to write some software that precisely models everything down to each and every little detail. That would be a massive amount of work. So we frequently apply simplifications to the solution context. That works if and only if the solution context is still a proper generalized subset of the problem context. <br /><br />From our earlier example if all twenty physical cogs map uniquely onto the ten software cogs, the context may be okay. But if some cogs can be mapped in different ways, or some cogs cannot be mapped at all, then the software solution will drift away from reality and people will see this as bugs. If there are manual procedures and conventions to occasionally fix the map, then at some point they'll degrade and it will still fail.<br /><br />Which is one of the most common fundamental problems with software. There often isn’t time to do the context mappings properly, and the shortcuts applied were invalid. The software context is shifted out from under the problem context, so it will gradually break. More software or even manual procedures will only delay the inevitable. The data, e.g. proxies, in the computer will eventually drift away from reality.<br /><br />So, if we see the context of the software as needing to be a proper subset of the context of the problem we intend to solve, it is easier to understand the consequences of simplifications. <br /><br />This often plays out in interesting ways. If you build a system that keeps track of a large number of people you obviously want to be able to uniquely identify them. Some people might incorrectly assume that a full name, as first, middle, last, is enough, but most names are not particularly unique. Age doesn’t help and duplicate birthdays are far too common. You could use a home address as well, but even in some parts of the world that is not enough. <br /><br />Correctly and uniquely identifying ‘all’ individuals is extraordinarily hard. Identifying a small subset for an organization is much easier. So we cheat. But any mapping only works correctly for the restricted domain context when you don’t have to fiddle with the data. If you have to have Bob and Bob1 for example, then the mapping is broken and should be fixed before it gets even worse.<br /><br />So as a problem we want to track a tiny group of people and we don’t have to worry about the full context. Yet, if whatever we do forces fiddling with the data, that means our solution context is misfocused and should be shifted or expanded. Manual hacks are a bug. Seen this way, it ends any sort of subjective arguments about modeling or conventions. It’s a context misfit, it needs to be fixed. It’s not ‘speculative generation’ or over-engineering, it is just obviously wrong.<br /><br />The same issues play out all over software development. We build solutions, but we build them to fit against one or more problem contexts, and those often get bounced around by larger organization or industry contexts. <br /><br />That is, often people narrow down the context to make an argument about why something is right or wrong, better or worse, but the argument is invalid because the context is just too narrow. The most obvious example I know is the ancient arguments about why Betamax tapes would beat out VHS, when in reality it went the other way. I think the best reference to explain it all was Geoffrey Moore in “Crossing the Chasm” when he talks about the ‘whole product’ which is an expanded context. <br /><br />All of that makes understanding the various contexts that bound the system very important. Ultimately we want to build the best fitting solutions given the problems we are trying to solve. Comparing the two contexts is how we figure out if we have done a good job or not.<br /></div>Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-66014573031707396022024-01-18T20:00:00.044-06:002024-01-18T20:00:00.136-06:00Buy vs BuildWhen I was young, at the end of the 80s, the buy vs. build question was straightforward.<br /><br /><div>In those days, for the emerging smaller hardware, there was not a lot of software available. It was slow and expensive to build anything. For any non-software company, they existed as brick-and-mortar businesses. Software could help automate isolated parts of the company, but that was it.<br /><br />Mostly, even over some medium horizons, buying an existing software product was far cheaper than building it. And that software was usually run in an independent silo, with minimal integrations, so it wasn’t hard to get it up and running.<br /><br />If there was already an available product, it didn’t make sense to build it. Buying it was the default choice.<br /><br />But so much has changed since then...<br /><br />The biggest change is that many companies now exist in the digital realm way more than the physical one. All of the sales, communications, and management for most lines of business happen digitally. Many of the products and services are still physical, but overall most lines of business are a mix now. <br /><br />Running a computer is also more complicated. In my youth, you would set up a server room with suitable power, cooling, and network connections. When you didn't have the space you would lease it. But the drop in hardware price caused an explosion, so the number of machines involved these days is astronomical (and often unnecessary). <br /><br />This made operations chaotic and undesirable, so most software products are a service now. Someone else sets it up and runs it for you. It’s easier to get going, but you don’t have as much control over it.<br /><br />With the increase in digital presence came a huge need for integrations. There are way more silos now, often hundreds or even thousands of them. There are specialist silos for every subproblem. When the silos were independent, integrations were rare. But now they all need each other's data; dependencies are necessary. So everything needs to be integrated into almost everything else. <br /><br />When you bought software before, you could get some new hardware to host it, spin it up, and test it out. If it mostly worked as expected it went live. But these days, just running a new system at a vendor's site isn’t that useful. Being trapped in a silo cripples it. It isn’t really live until all of the major integrations are done. Silos made sense before, but they are a hazard now.<br /><br />It is not in any vendor's best interest to standardize their software. It is a simple calculation. If it is standard, then it is nearly trivial for any customer to switch to someone else. If you run into any glitches and all of the customers leave, you are instantly done. So don’t use standards.<br /><br />Integrating various non-standard SaaS silos with each other is an epic nightmare. The easiest way to do it is to copy the data everywhere. If you have a dozen silos that need a particular type of data, you make a dozen copies. Then desperately try to keep them in sync somehow.<br /><br />To make it even worse, each integration team and vendor will choose to model the data moving around differently. So, it is endlessly translated into different formats, and some of those translations will lose valuable parts of the information. That wastes a lot of time and causes all sorts of grief.<br /><br />So modern integration projects have become huge, expensive, and tricky. <br /><br />It's counterintuitive, as you think you managed to avoid programming by buying everything, but now you end up having to do way more glue programming in order to connect it all together. <br /><br />And so much of that ETL code is awful. It was rushed into existence by people with too little experience. You end up with masses of hopelessly intertwined spaghetti and endless operational alerts about warnings and errors, most of which are unfortunately ignored. <br /><br />And that is the crux of the issue. If you buy everything now, then you’ll get lost while trying to get it to all work together properly, and this is a lot more costly than just having built it properly in the first place.<br /><br />Some things you don’t want to build though. Sometimes because it's huge, but more often because it is so complex that the devs require specific knowledge and experience to build it reasonably. You can't just hire a whole bunch of kids, they’ll conjure up a mess instead of what you need, it won’t help.<br /><br />For any group of programmers, there are absolute limits to what they can build. They rarely are self-aware of their own limits, but things won’t go well if you let them stray too far past their abilities. <br /><br />You can assemble a strong group of developers to build exactly what you need, but if the work dries up they will dissipate and you will run into trouble keeping it functioning later. To keep good developers they need to always have good projects to work on.<br /><br />Which is to say that if you need to build software, you need to set up a stable ‘dev shop’ with enough capacity to turn out and enhance the types of software you need. The dev shop is what you need to focus on. It should be able to attract new talent, and to always have enough interesting work to keep everyone motivated. Talent attracts talent, so if you get a couple of strong devs, you can grow the affair. You just have to make sure the environment stays reasonable.<br /><br />If you do that, it fundamentally changes the original buy vs build dynamics. <br /><br />You want to keep building enough stuff to ensure that the dev shop stays functional. Building, if you have the capacity and ability would now be the first choice. It is a longer-term goal though, as you don’t want all of your good developers to leave.<br /><br />Then you want to build up and out from a few different starting points. The guidance is to minimize integrations first. They are ultimately more costly than the vendors, so you focus there. <br /><br /><br />You figure out which categories your shop can handle, then you consolidate all of the little fragmented silos into larger systems. Generalization is the key to keeping the costs in line. Software companies leverage their code for lots of companies; in-house projects need to leverage their code for lots of different problems.<br /><br />The focus is not on speed though, rather it is on doing the best engineering that you can. Move slowly and carefully. Build up as much reusable kit as you can, model the data as properly as you can, and keep expanding out from the starting point slowly swallowing dozens of other products. But always keeping a close eye on both the dev shop and the operational capacity. <br /><br />Obviously, the digital parts of existing lines of business would be first. You’d want to do this anyway, since just using the same vendors as everyone else has no competitive advantages. But to get those advantages back, the work you do has to be better and more relevant than the vendors, which means that you have to have strong technologists who really understand the business too. <br /><br />Then funding isn’t by project, line of business, or budget. It is by dev shop. You set a solid foundation and build up capacity to implement better stuff and keep the funding stable as you grow it. A large organization can have a few different dev shops.<br /><br />Outside of those areas of software growth, the old buy vs. build choice would remain, but as the starting points get larger they would end up eating stuff around the fringes so you’d need to factor that in as well. <br /><br />The counterargument to all of this is that building software is seen as outside of the company’s vertical. But the modern reality is that as most businesses got deeper into the digital realm, they drifted ever closer to being applied software companies, than their original lines of business. <br /><br />The classic examples are Google, a marketing company, and Amazon, a retailer. As applied software companies they thrived while their brick-and-mortar predecessors didn’t. <br /><br />The general nature though is that software is not a vertical for any digital line of business, it is a part of it. Core. That, and the exploding integration costs means that reasonable software is necessary for scaling and stabilizing. Bad software makes everything unprofitable. <br /><br />As software eats the world, this same fate will play out in all sorts of other industries, and the winning strategy is to accept that if you are heavily reliant on doing business in the digital realm, then you are already heavily reliant on building software. <br /><br />Then it is far more effective to build some of the core stuff yourself, instead of just integrating generic vendor products. You’ll need to recruit stronger developers and make sure you can keep them, but if you do your capacity will grow. Then software development capability itself becomes a competitive advantage.<br /></div>Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-53406421432502578322024-01-11T20:00:00.034-06:002024-01-11T20:00:00.142-06:00Lessons LearnedYou learn a lot during thirty years. I tried to write about most of it in this blog, at least from a higher level perspective bringing lots of different things that have happened together, but some things are smaller and just don’t fit. Each one of these is rooted in at least one epic failure.<br /><ul style="text-align: left;"><li>Doing the screens first and persistence last is a common top-down development approach, but it is a very bad mistake. The screens are driven by the irrationality of the users, they don’t map cleanly to the demands of persistence, and they never will. If they could, then lots of the very early application generators would have worked, but they didn’t. Persist first, then gradually move it up until it gets into the screens.</li><li>If you have an RDBMS, use it to its nearly fullest ability to protect itself. You really don’t want to persist garbage data, that will cause all sorts of annoying bugs. You don’t want to double up stuff you are persisting too, it is wasting space and can cause stale or inconsistent data as well. People always try to cheat the database work, and they always pay a high price for it. It isn’t particularly fun work, but it anchors everything else.</li><li>Don’t try to break dependent things into subparts, like say put the front and back ends for the same system into two different repos. People might decompose by language, for example, but really if there is dependency, like an API, that matters more. It’s hard to explain, but if things can’t stand on their own, then you shouldn’t try to force them to.</li><li>Disorganization will always be the biggest problem. Organization is a place for everything, everything in its place, and not too many similar things in the same place. That is, if you have something new and you don’t know where to put it, then it is disorganized. It must go somewhere; if that is obvious, then you are okay.</li><li>If the programmers don’t know what the system holds for data, and they don’t know why people are doing things with that data, then it will have a huge number of bugs. Programmers are the frontline for quality, if they can’t see problems as they work, there will be lots and lots more.</li><li>Always only every move code in one direction. It goes from Dev to Release, with a few QA stops along the way. Never, never, break that chain. It will result in all sorts of problems including things getting accidentally rolled back, which is avoidable.</li><li>Always clean up right after a release. Everyone is tired, and cleanup work is boring. If you do not clean up then, you will never clean up and the mess will get worse, far worse.</li><li>Tackle the hard parts first, not the easy ones. The hard ones are unpredictable in time, if they don’t go well you can raise an early flag on the schedule. The other way around tends to mislead people into thinking that everything is going well when it isn’t.</li><li>Do the right thing when you start. Only take more shortcuts the closer you are to the deadline. If you take a shortcut, note it, and clean it up right away after the release.</li><li>Do not freeze code forever. If you freeze the code, you also free the bugs, which is then the foundation of everything else. Building on buggy foundations is problematic.</li><li>Do not let people add in onion architectures. If they are trying to avoid the main code, and just do “their thing” around the outside, that work is usually very harmful. Push them to do the work properly.</li><li>Don’t drink the Kool-Aid. There just isn’t an easy or right way to build stuff. The best you can do is make it readable and keep it organized. Most philosophies for coding are extreme and have worse side effects.</li><li>If what happens underneath matters in the system, it is not fully encapsulated. In that case, you need to learn some stuff about what happens and why it happens. You can’t just ignore it. Some components will never be fully encapsulated.</li><li>The ramp-up time for an experienced coder is proportional to the size of the codebase. The ramp-up time for an inexperienced coder is far longer.</li><li>A weird and ugly interface is a strong disincentive against usage. Useful code has a long life. The point of writing professional code is to maximize its lifespan.</li><li>Reduce friction, don’t tolerate it. Spending the time to mitigate or minimize it always pays off. Putting up with it always slides one downhill.</li></ul>There is a lot more, but I’ll start with these. Writing code is fairly easy, but building huge reliable systems is exceptionally hard. The two are not the same. <br />Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com2tag:blogger.com,1999:blog-6104420435021904082.post-37268441532942909592024-01-04T20:00:00.013-06:002024-01-04T20:00:00.247-06:00Time vs RiskWhen I was young software development was not in the spotlight. We had quite a bit of time to get our work done. We would carefully craft things, focusing on the key issues.<br /><br />It was the dawn of the World Wide Web followed by the decadence of the Dot Com era that changed all of that. Suddenly “first mover advantage” outweighed quality, correctness, and readability.<br /><br />Modern coding is a high-speed game of chicken. It starts with a request to do some work in usually less than 1/3rd of the amount of time you need to do a good job. If you balk at the lack of time, they’ll take their work elsewhere. So, you might try to stretch it out a little, but then you agree.<br /><br />When time is compressed, you inevitably end up taking a lot of shortcuts. Some programmers know to avoid many of these, but the industry tends to praise them. <br /><br />A shortcut is a tradeoff. You do something faster now, in the hopes that it will not blow up in your face later. <br /><br />Some shortcuts never blow up, you get lucky. <br /><br />Some just are incremental aggravations that if they haven’t built up too deeply will only slow you down a bit later. Just friction. <br /><br />Some shortcuts, however, will implode or even explode, throwing the whole affair into the trash bin or flatten it forever. It’s been bad enough that I’ve actually seen code come out spectacularly fast then spent half of a decade slogging through near-hopeless bugs. The wrong series of really bad shortcuts can be devastating.<br /><br />So every shortcut is a risk. But it is hard to quantify, as there are usually aggravating factors that multiply the damage. <br /><br />Given that you are inevitably pushed into having to take some shortcuts, it’s best to take the least destructive ones. Those tend to be higher up. <br /><br />If you build code in a rational manner, you would lay out the foundations first and then carefully stack a lot of reusable components on top. That is the minimum amount of work you need to do. <br /><br />Bad low-level code propagates trouble upward; the stuff built on top needs to counteract the awful behavior below. That tells us that the lower the shortcut, the more risky it is, the more it affects, and the worse the consequences of losing by taking it.<br /><br />We see that all of the time. <br /><br />Those systems, for example, where they did crazy fast things with saving the data, then wrote far too much hacky code above to try and hide the mess. If they had just modeled the data cleanly, then the tragically nested conditional nightmare piled on top, which ate huge amounts of time and spread a lot of pain, would not have been necessary. It is a super common example of a small set of shortcuts going rather horribly wrong. <br /><br />You see exceptionally bad persistence all over the place causing problems. It’s likely that at least half the code ever written is totally unnecessary.<br /><br />What’s always true is that if you take too many risks and lose enough of them, the time saved by the shortcuts will be massively overwhelmed by the time lost dealing with them. Coming out of the gate far too fast will always cause a project to stumble and will often cause it to lose the race. <br /><br />If you are forced to take risks then it is worth learning how to evaluate them correctly. If you pick the right ones, you’ll lose a few, but keep on going. It’s not how it should be, but it is pretty much how it is these days.<br />Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-18105414785733012702023-12-28T20:00:00.001-06:002023-12-28T20:00:00.130-06:00IdentityThe biggest problem with software security is that we have wired up a great deal of our stuff to rely on ‘anonymous’ actions. <br /><br />A user logs into some device, but with distributed computing that will machine talk to a very large number of other machines which often talk to even more machines behind the scenes. Many of those conversations default to anonymous. When we implement security, we only take ‘some’ of those conversations and wrap them in some type of authentication. <br /><br />The most common failures are that either we forget to wrap some important conversations, or that there are various bugs when we do. <br /><br />A much better way is to insist that ‘all’ conversations have the originating user’s identity “attached” to them. Everything. All of the way down. No verifiable identity, no computation. Simple rule.<br /><br />"Why?"<br /><br />Security as an afterthought will always get forgotten in a rush. And if it’s complicated and redundant, the implementations will vary, most landing on the broken side. It is a losing battle.<br /><br />“But we don't need that much security...” <br /><br />Actually, we do, and we’ve always needed that much security. It’s just that long, long ago when these things were first written, there were so few people involved and they tended to be far more trustworthy. Now everyone is involved and the law of averages prevails. We put security on everything else in our lives, why wouldn’t we do it properly on software?<br /><br />“It’s too hard to fix it...”<br /><br />Nope. It certainly isn’t easy, but if you look at the technologies we’ve been using for a long, long time now, they have the capacity to get extended to do this. It won’t be easy or trivial, but it isn’t impossible either. If we get it wired up correctly, we can gradually evolve it to improve.<br /><br />“It doesn’t work for middleware...”<br /><br />Any incoming request must be associated with an identity. The use of generic identities would be curtailed. So, the web server runs as SERVER1, but all of its request threads run as the identity of the caller. No identity, no request. <br /><br />“That’s crazy, we’d have to know all of the user's identities in advance...”<br /><br />Ah, but we do. We always do. Any non-trivial system has to authorize, which means that some functionality is tagged directly to a set of users. If you have user preferences for example, then only you are authorized to modify them (in most cases). There could be anonymous access, but that is mostly for advertising or onboarding. It is special, so it should not be the default. <br /><br />Some systems could have anonymous identities, and it can be turned on or off, in the same way that we learned to live with them in FTP. But they wouldn’t be the default, you’d have to do a lot of extra work to add them, and you’d only do that for very special cases.<br /><br />Every thread in middleware could have an identity attached to it that is not the ‘system identity’, aka the base code that is doing the initialization and processing the requests. It’s pretty simple and it should be baked in so low that people can’t change it. They could only just ‘add’ some other anonymous identity if they wanted to bypass the security issues. It’s analogous to the split between processes and the kernel in a reasonable operating system.<br /><br />“But the database doesn’t support it...”<br /><br />Oddly, the problem with most databases does not seem to be technical. It is all about licenses. Historically, the way companies figured out how to make extra money was through licensing users. It’s a great proxy for usage and usage is a way of sizing the bill to fit larger companies. You set a price for small companies. then add multipliers to get more out of the bigger ones. <br /><br />We should probably stop doing that now. Or at least stop using ‘users’ as proxies for it, especially if that is one of the root causes of all of our security issues. <br /><br />Then any statement to the database is also attached to an identity. Always. The database has all of the individual users, and every update is automatically stamped with user and time. No need to rewrite an application version of this anymore. It is there for all rows and all tables, always.<br /><br />“That’s too much processing, some rows need far less...”<br /><br />Programmers cheat the game in their applications and don’t properly audit some of the changes. Usually, that seems like a great idea right up until someone realizes that it isn’t. Whenever you collect data, you always need a way of gauging its reliability, and that is always the source of the data. If it comes from somewhere else, you need to keep that attached to the data. If a user changes it, you need to know that too. If a user changes it and it jumps through 18 systems, then if you lose its origins, you also lose any sense that it is reliable. So, it would make far more sense if, during an ETL, you keep that information too, and honor it. It would increase your data quality and certainly make it a whole lot easier to figure out how bugs and malicious crimes happened. <br /><br />“That’s too much disk space...”<br /><br />Most large organizations store their data redundantly. I’ve actually seen some types of data stored dozens of times in different places. We really should stop doing that. It would be a macro optimization on saving a huge amount of badly used disk space, as opposed to a micro one caused by lowering the data quality. <br /><br />“But what about caching...”<br /><br />I’ve said it before, and I’ll say it again, you should not be rolling your own caching. Particularly not adding in a read cache, when you have writable data. You’re just causing problems. So, realistically, you initialize with a system identity, and then it primes the cache under that identity. If someone builds a real working cache for you, it needs user identities, and it figures out how to weigh those against the system identity work to appropriately account for each. It does that both for security, but also to ensure that as a cache it is effective. If the system identity reads a wack load of data for one user but never uses it again, then the cache is broken. So, weights of 100% for example would mean that the caching was totally and utterly useless. A weight less than 0.01% would probably be quite effective. Security and instrumentation, combined.<br /><br />“But what about ex-users...”<br /><br />People come and go. Keeping track of that is an organizational issue. They really shouldn’t forget that someone worked for them a few decades back, but if they wanted to do that, they could just swap to a single ‘ex-employee’ identity. I wouldn’t recommend this myself, I think it makes far more sense that if you have returned to a company they reconnect you to your previous identity, but it should be a company-wide decision, not left to the whims of each application. When you start building something new, the ‘group’ of people that can use it should already be established, otherwise, how would you know that you need to build the thing?<br /><br />“What about tracking?”<br /><br />If you know all of the computations that an identity triggers and all of the data that they have changed, then you have a pretty powerful way of assessing them. That’s not necessarily a good thing, and it would have to be dealt with outside of the scope of technology. It would not be accurate though, because it is really easy to game, so if a company used it as a performance metric, it would only end up hurting them.<br /><br /><br />“But I want to roll my own Security...”<br /><br />Yeah, that is the problem with our security. It takes a crazy amount of knowledge to do it correctly, everyone wants to do it differently, most attempts get it wrong, and while it would be fun to code up some super security, in reality, it is always the first functionality that gets slashed when everyone realized they aren’t going to make the release deadlines. If your job is effectively to rush through coding, then most of the coding you should stick to is straightforward. It sucks, but it is reality. It also plays back to the notion that you should always do the hard stuff first, not last. That is, the first release of any application should be a trivial shell that sets the foundations, but effectively has no features. Then the first release of the application is actually an upgrade. Doing it will eliminate a lot of pain and is easier to schedule. <br /><br />"There are too many vendors, they won't agree to this..."<div><br /></div><div>The industry is notoriously addicted to locking customers in. This type of change would not affect that, so if we crafted it as an ISO standard, and then there was pressure to be compliant, most of them would comply simply because it was good for sales. The downside is that in some cases it would affect their invoicing, but I'm sure they could find another proxy for organization size that is probably easier and cheaper to implement.</div><div><br />Identity, like a lot of other software development problems, is difficult simply because we like to shoot ourselves in the foot. If we could stop doing that, then we could put in place some technologies that would help ensure that the things we build work far better than they do now. Oddly, these problems are not hard to implement, and we basically know how to do them correctly, the issue isn’t technological, it has nothing to do with computers themselves, it is all about people. </div>Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-6579366054828669762023-12-21T20:00:00.025-06:002023-12-21T20:00:00.135-06:00Software KnowledgeThere are at least two different categories of knowledge in software.<br /><br /><div>One is the specifics in using a tech stack to make a computer jump through specific hoops to do some specific computations. It is very specific. For example, the ways to store certain arrangements of bits for persistence when using Microsoft stacks.<br /><br />When you are building software, if you know how to do something, it will make the work go faster. But people are right when they say this type of knowledge has a half-life. It keeps changing all of the time, if you haven't done it for a while, you’ve forgotten it or it has moved out from under you.<br /><br />This is the stuff you look up in StackOverflow.<br /><br />The other category of knowledge is far more important. <br /><br />There are all sorts of ways of working and building things with software that have not changed significantly for decades. They are as true now, as when I started forty years ago. They are the same no matter what tech stack you use, be it COBOL or JavaScript. Sometimes they are forgotten and then reinvented later under different names. A good example is that we used to edit our code, now we refactor it. <br /><br />Fundamentally, building software is a construction effort. The medium appears as more malleable than most, but it is not immune from any other constructive issue. And because programmers intentionally freeze far too much, we change things too fast, and often need backward compatibility, it is rarely as malleable as it could be.<br /><br />There are a couple of obvious anchors.<br /><br />The first is that size matters. It is a whole lot easier to build tiny things than massive ones. Size is everything.<br /><br />The second is that as the size of the effort grows, disorganization causes worse problems. If you write some tiny spaghetti, it is okay, you can still change it. But if you have a million lines of spaghetti you are screwed.<br /><br />Organizing stuff isn’t fun, and oddly it isn’t a one-time task either. It is an ongoing effort. The data and code are only as organized as the explicit effort you put into them to keep them organized. If you aren’t doing anything, it is likely a mess.<br /><br />But even more specifically, there is a lot of general knowledge about how to code things like data structures, algorithms, data normalization, or GUI conventions that hold true regardless of the stack. You may not need to create a hash table yourself anymore but you still need to understand how to leverage it and its limits. People will always need to ensure their data is at least 3NF or they will pay the price for storing it badly. A poorly wired GUI will diminish trust, it may be marginally workable but generating ill will.<br /><br />The tools too. Learning to properly configure and use an editor or IDE tends to stay relevant for a very long time. There are all sorts of build tools and scripting, most of which haven't changed for decades, although sometimes they get obscured by trendy stuff that doesn’t last. But the need for the tools and usage of them never changes. If you spend time to figure one out, the others come easily. It also helps in understanding why some newer trends are poor and should be avoided.<br /><br />Of course, all of the issues with people and politics never, ever change. We build software as a solution to some users' problems. If you don’t fully understand what you are trying to solve, then the things you’ve built are far less likely to work as needed. There is also a lot of gymnastics involved with funding software development, often resulting in too much stress and rushing through the work. Stress is bad for thinking; rushing is bad for quality.<br /><br />Most of what you do specifically in software changes. The trends come and go in roughly five-year waves, developers need to keep up but not every wave. You can skip some waves, but if you skip too many your opportunities narrow. Once you are old and out, it is brutal getting back in.<br /><br />Most of the general knowledge is far more important than the specifics. It is what keeps the projects from chaos, ensures that the work is at least good enough, and helps control the expectations of the people on the margins. If you know generally how to build things well, you can always look up the specifics. But if you don’t know how to properly persist the data, for instance, the work is doomed before it even starts. If you don’t understand fragmentation, you won’t understand why your work keeps failing when you bring it all together. If you don’t understand the components, you cannot craft a reasonable architecture.<br /><br />You actually need more general knowledge to ensure that a large project is successful than specific knowledge. It is what keeps it all out of trouble while people are coding like mad. This is probably why the high failure rate of modern software is independent of the methodologies used. It's more likely experience-related. A bunch of kids who really know their specifics well will still usually fail in very predictable, general ways. That was true when I was a kid and it still holds true today.<br /></div>Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-62721830855538133172023-12-14T20:00:00.001-06:002023-12-14T20:00:00.159-06:00FragmentationWhen you have never experienced large-scale software development, your preference is often to prefer fragmentation. You want lots of little parts.<br /><br /><div>It’s understandable. <br /><br />Most people are taught to decompose large problems into smaller ones. Once they have completed that, they assume that all of those smaller sub-solutions are mostly independent from each other. <br /><br />They believe that dealing with each piece independently would be easier. That way you can just focus on one and ignore the rest.<br /><br />It’s just that when people learn to decompose stuff, they tend to choose rather arbitrary lines of decomposition. There are lots of options, they choose the easiest. But that means that the pieces below are more likely to have dependencies between them. That they are not independent. <br /><br />If the problem was decomposed based on ‘natural’ lines that tend away from dependencies, then the idea of treating stuff as fragments would work. But they don’t know what that means, so it doesn’t happen.<br /><br />The other part of this issue then comes into play. <br /><br />If you decompose a problem into parts, you need to ensure that the parts themselves still hold together to solve the original problem. That at least they cover everything necessary. That is, after you break it down, you build it back up again to make sure it is right. Breaking it down is only the first half of the effort. <br /><br />So overlaps, gaps, and dependencies tend to derail a lot of decomposition attempts. <br /><br />Once any complexity gets fragmented it grows exponentially. If there are enough fragments it becomes nearly impossible to assert anything reasonable about its overall behavior. That is, each of the components may work as expected, but the combination of them does not. This is an all too common problem in software. <br /><br />The cure is to be suspicious of fragmentation. It’s not the same as encapsulation. <br /><br />In the latter case, all of the rough edges are hidden nicely inside of a box. In the first case, the edges are exposed and are effectively their own pieces, thus throwing the complexity out of whack. You quickly end up with far more pieces than you can handle.<br /><br />You can see this as an issue of ‘scope’ and most programming languages provide strong tools to control it, but very few programmers take advantage of them. We first figured this out with global variables, but there are endless ways to create similar issues.<br /><br />If your decomposition into an Object is correct, for example, then all of the internal variables in the Object can be set to private. They are not modifiable from the outside. They are not visible from the outside. The entire interface is methods, and all of the variables are properly encapsulated. Instead, we have crazy ideas like ‘getters’ and ‘setters’ that drop functions directly over the variables, so that we can pretend that we encapsulated them when clearly we didn’t. <br /><br />Other fun examples of fragmentation include the early attempts to distribute code throughout lots of static html files, making it nearly impossible to correctly predict behavior of anything non-trivial. <br /><br />Modern frameworks are often based around fragments as well. You know there is a problem if you need to access ‘globals’ in a lot of little ‘callbacks’; it will quickly become a mess. <br /><br />Even a lot of modern data storage philosophies make the same mistake. Just dumping all of the data in little files into a disorganized pool or lake is only going to blow out the complexity. Sure, you save time while collecting the data, but if it is nearly impossible to find stuff when you need it, then the collection will grow into a swamp.<br /><br />Breaking things down into smaller pieces without fully encapsulating them is fragmentation. It is bad, in that while encapsulation wraps and controls complexity, fragmentation just amplifies it. Complexity is the impassable barrier for size. If you can’t manage it, you cannot get any larger or more sophisticated. If you encapsulate parts of it properly, you can grow the solution until it covers the whole problem. </div>Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-10066993175361099552023-12-07T20:00:00.010-06:002023-12-07T20:00:00.143-06:00Solving Hard ProblemsTrivial solutions are great for solving trivial problems. But if you have a hard problem, then no combined set of trivial solutions will ever end up correctly solving it.<br /><br />The mechanics of this are fairly easy to understand. <br /><br />If you try to solve only a small subset of a big problem with a solution that targets just that subset, then it will spin off more problems, it is a form of fragmentation. <br /><br />You’ve addressed the subset, but now that needs to interact with many of the other parts, and those are artificial complexity. They would not have been necessary if you had addressed the whole problem.<br /><br />If you try to solve a hard problem, either one that is huge or one that is complex, with a lot of trivial solutions, there will be an endless stream of these unsolved fragments, and as you try to solve these, they will make it all rather perpetual. You can’t get a perpetual motion machine in our physical universe, but you can effectively spend nearly forever creating and patching up little holes in a misfitting solution.<br /><br />If you want to solve a hard problem, then the solution itself cannot be trivial. It will not be simple, it will not be easy. Any desire or quest to get around this is hopeless.<br /><br />“Things should be made as simple as possible, but no simpler” -- possibly Albert Einstein<br /><br />The really important part of that misattributed above quote is at the end. That there is a notion of too simple.<br /><br />The belief that one can get away with over-simplifying solutions is the underlying cause of so many software problems. It’s nice when software is simple and easy to understand, but if it isn’t the right solution, then it will cause trouble. <br /><br />Yes, you can whack out a large number of trivialized silos into software. You can get these into active usage really quickly. But if they do not fit properly, the net effect is now worse than before. You’ve created all sorts of other problems that will accumulate to be worse than the original one. The software isn’t really helping, it’s just distracting everyone from really fixing the problem. <br /><br />This is quite common in a lot of large companies. They have an incredible number of systems that spend more of their resources pushing data back and forth, than they do working with their intended users. And the more they move data around, the worse it gets. The quality issues spin off secondary systems trying to fix those data problems, but by then it is all just artificial complexity. The real underlying problems have been lost. <br /><br />If any of the silos aren’t fully encapsulated, then either the partitioning is wrong, or the problem is hard and can’t be split up. Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-86915753619360923362023-11-30T20:00:00.001-06:002023-11-30T20:00:00.131-06:00Observable on SteriodsA while back I wrote a small product that, unfortunately, had an early demise. It was great technically -- everyone who saw it loved it -- it’s just that it didn’t get any financial support.<br /><br />The idea was simple. What if all of the data in an application was observable? Internally it was just thrown into a giant pool of data. So, it could be variables in a model, calculations, state, context, etc. Everything is just observable data.<br /><br />Then for each piece of derived data, it would watch any of its dependencies. If it was a formula, it would see that any of the underlying values changed, recalculate, and then tell anyone watching it that it is now different.<br /><br />The stock way of implementation observable in an object-orient paradigm is to keep a list of anyone watching, then issue an event, aka function call, to each. The fun part is that since any object can watch any other object, this is not just a tree or a dag, it can be a complete graph. <br /><br />Once you get a graph involved, any event percolating through it can get caught in a cycle. To avoid this, I traded off space by having each event keep a dictionary of the other objects it had already visited. If an object gets notified of an event and it is already in that visited set, it just ignores the event. Cycles defeated.<br /><br />So, now we have this massive pool of interconnected variables, and if we dumped some data into the pool, it would set off a lot of events. The leaf variables would just be updated and notify watchers, but the ones above would recalculate and then notify their watchers. <br /><br />I did do some stuff to consolidate events. I don’t remember exactly, but I think I’d turn on pause, update a large number of variables, then unpause. While paused, events would be noted, but not issued. For any calculation with a lot of dependents, if one thing changed, I’d track the time and since it has already recalced, grabbing all of the child variables data, it would toss any events for other dependents that were earlier. So, for A+B+C, after the unpause, you’d be notified that A changed, and do the recalc, but then the time that B and C changes would be earlier than the recalc, so ignored. That cut down significantly on any event spikes.<br /><br />Then finally, at the top, I added a whole bunch of widgets. Each one was wired to watch a single variable in the pool. As data poured into the pool, the widgets would update automatically. I was able to wire in some animations too, so that if a widget displayed a number, whenever it changed it would flash white, and then slowly return to its original color. But then I wired in tables and plotting widgets as well. The plot, for instance, is a composite widget, whose children, the points of the curve, are then they are all wired to different variables in the pool. So, the plot would change on the fly as things changed in the pool.<br /><br />Now if this sounds like MVC, it basically is. I just didn’t care if the widgets were in one view or a whole bunch of them. They’d all update correctly either way. and the entire model was in the pool. And any of the interface context stuff was in the model so in the pool. And any of the user’s settings were in the model, so in the pool. In fact, every variable, anywhere in the system, was in the model, so in the pool. Thus the steroids designation. Anything that can vary in the code is an object and that is observable in the pool.<br /><br />Since I effectively had matrices and formulas, it was a superset of a spreadsheet. A sort of disconnected one. A whole lot more powerful.<br /><br />Because time was limited, my version had to wire up each object as code explicitly. But they didn’t do much other than inherit the mechanics, define the children to watch, and provide a function to calculate. It would have been not too hard to make all of that dynamic, and then provide an interface to create and edit new objects in the app itself. This would let someone create new objects on the fly and arrange them as needed.<br /><br />It was easy to hook it up to other stuff as well. There was a stream of incoming data, so it was handled by a simple mapping between the data’s parameters and the pool objects. Get the next record in the stream, and update the pool as necessary. Also, keyboard and button events would dump stuff directly into the pool. I think some of the widgets even had two-way bindings, so the underlying pool variable changed when the user changed the widget and could percolate to everything else. I had some half cycles, where the widget displayed a value, it was two-way bound and as the user changed it, it triggered other changes in the pool, which updated stuff on the fly which would change the widget. I used that for cross-widget validations as well. The widgets change the meta information for each other.<br /><br />I did my favorite form of widget binding, which is by name. I could have easily added scope to that but the stuff I was working with was simple enough that I didn’t have any naming collisions. I’ve seen structural-based binding sometimes, but they can be painful and rigid. The pool has no structure and the namespace is tiny because the objects were effectively hardwired to their dependencies. Extending it would need scope.<br /><br />To pull it all together I had dynamic forms and set them to handle any type of widget, primitive or composite. I pulled a trick from my earlier work and expanded the concept of forms to include everything on the screen including menus and other recursive forms. As well, forms could be set to not be editable, which lets one do new, edit, and view all with the same code, which helps to save coding time and enforce consistency. <br /><br />Then I put in some extremely complex calculations, hooked it up to a real-time feed, and added a whole bunch of screens. You could page around while the stream was live and change stuff on the fly, while it was continuously updating. Good fun.<br /><br />It’s too bad it didn’t survive. That type of engine has the power to avoid a lot of tedious wiring. Had I been allowed to continue I would have wired in an interpreter, a pool browser, and a sophisticated form creation tool. That and some way to dynamically wire in new data streams would have been enough to compete with tools like Excel. If you could whip up a new table and fill it with formulas and live forms, it would let you craft complex calculation apps quickly.Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-808195441784742332023-11-23T20:00:00.001-06:002023-11-23T20:00:00.133-06:00Bottom UpThe way most people view software is from the top down. They see the GUI and other interfaces, but they don’t have any real sense of what lies behind it.<br /><br />The way that bugs are reported is from the top down. Usually, there is an interface problem somewhere the underlying cause may be deep in the mechanics.<br /><br />The foundation of every system is the data it persists. If there isn’t a way to reliably keep the data around for a long time, it might be cute, but it certainly isn’t practical.<br /><br />The best way to build software is from the bottom up. You lay down a consistent set of behaviors and then you build up more complicated behaviors on top of those. This leverages the common lower stuff, you don’t have to reduplicate the work again for everything on top.<br /><br />The best way to extend a system is bottom-up, You start with getting the data into persistence, then into the core of the system, and then you work your way upwards until all of the interfaces reflect the changes.<br /><br />The way to deal with bugs is top-down. But the trick is to keep going as low as you can. Make the lowest fixed as time will allow. Sometimes it is rushed, so you might fix a bunch of higher-level symptoms, but even if you do that you still have to schedule the proper lower-level ones soon.<br /><br />The best way you screw up a large system is to let it get disorganized. The way you organize a system is from the bottom up, Looking at it top down will only confuse you.<br /><br />Some people only want to see it one way or the other, as top-down or bottom-up, but clearly, that isn’t possible. When a system becomes large, the clash in perspectives becomes the root of a lot of the problems. Going in the wrong direction, against the grain, will result in hardship.<br />Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-16713053798510077462023-11-16T20:00:00.034-06:002023-11-16T20:00:00.180-06:00The Power of AbstractionsProgrammers often complain about abstractions, which is unfortunate.<br /><br />Abstractions are one of the strongest ‘power tools’ for programming. Along with encapsulation and data structures, they give you the ability to recreate any existing piece of modern software, yourself, so long as you have lots and lots of time.<br /><br />There is always a lot of confusion about them. On their own, they are nothing more than a generalization. So, instead of working through a whole bunch of separate individual special cases for the instructions that the computer needs to execute, you step back a little and figure out what all of those different cases have in common. Later, you bind those common steps back to the specifics. When you do that, you’ve not only encoded the special cases, you’ve also encoded all of the permutations.<br /><br />Put another way, if you have a huge amount of code to write and you can find a small tight abstraction that covers it completely, you write the abstraction instead, saving yourself massive amounts of time. If there were 20 variations that you needed to cover but you spent a little extra time to just create one generalized version, it’s a huge win.<br /><br />Coding always takes a long time, so the strongest thing we can do is get as much leverage from every line as possible. If some small sequence of instructions appears in your code dozens of times, it indicates that you wasted a lot of time typing and testing it over and over again. Type it once, name it, make sure it works, and then reuse it. Way faster.<br /><br />A while back there were discussions that abstractions always leak. The example given was for third-generation programming languages. With those, you still sometimes need to go outside of the language to get some things done on the hardware, like talking directly with the video card. Unfortunately, it was an apples-to-oranges comparison. The abstractions in question generalized the notion of a ‘computer’. But just one instance of it. Modern machine architecture however is actually a bunch of separate such computing devices all talking to each other through mediums like the bus or direct memory access. So, it’s really a ‘collection’ of computers. Quite obviously if you put an abstraction over the one thing, it does not cover a collection of them. Collections are things themselves (which part of what data structures is trying to teach).<br /><br />A misfitting abstraction would not cover everything, and an abstraction for one thing would obviously not apply to a set of them. The abstraction of third-generation programming languages fit tightly over only the assembler instructions that manipulated the computer but obviously didn’t cover the ones that were used to communicate with peripherals. That is not leaking, really it is just scope and coverage.<br /><br />To be more specific, an abstract is just an abstraction. If it misfits and part of the underlying mechanics is sticking out, exposed for the whole world to see, the problem is encapsulation. The abstract does not fully encapsulate the stuff below it. Partial encapsulation is leaking encapsulation. There are ugly bits sticking out of the box. <br /><br />In most cases, you can actually find a tight-fitting abstraction. Some generalization with full coverage. You just need to understand what you are abstracting. An abstraction is a step up, but you can also see it as binding together a whole bunch of special cases like twigs. If you can visualize it as the overlaid execution paths of all of the possible permutations forming each special case, then you can see why there would always be something that fits tightly. The broader you make it the more situations it will cover.<br /><br />The real power of an abstraction comes from a hugely decreased cognitive load. Instead of having to understand all of the intricacies of each of the special cases, you just have to understand the primitives of the abstraction itself. It’s just that it is one level of indirection. But still way less complexity.<br /><br />The other side of that coin is that you can validate the code visually, by reading it. If it holds within the abstraction and the abstraction holds to the problem, then you know it will behave as expected. It’s obviously not a proof of correctness, but being able to quickly verify that some code is exactly what you thought it was should cut down on a huge number of bugs.<br /><br />People complain though, that they are forced to understand something new. Yes, absolutely. And since the newer understanding is somewhat less concrete, for some people that makes it a little more challenging. But programming is already abstract and you already have to understand modern programming language abstractions and their embedded sub-abstractions like ‘strings’. <br /><br />That is, crafting your own abstraction, if it is consistent and complete, is no harder to understand than any of the other fundamental tech stack ones, and to get really good at programming, you have to know those anyway. So adding a few more for the system itself is not onerous. In some cases, your abstraction can even cover a bunch of other lower-level ones, so if it is encapsulated, you don’t need to know those anymore. A property of encapsulation itself is to partition complexity, making the sum more complex but each component a lot less complex. If you want to write something sophisticated with extreme complexity, partitioning it is the only way it will be manageable.<br /><br />One big fear is that someone will pick a bad abstraction and that will get locked into the code causing a huge mess. Yes, that happens, but the problem isn’t the abstraction. The problem is that people are locking things into the codebase. Treating all of the code in the system as write-once and untouchable is a huge problem. In doing that, it does not matter if the code is abstract or not, the codebase will degenerate either way, but faster if it is brute force. Either the code on top a) propagates the bugs below, b) wraps another onion layer around the earlier mess, or c) just spins off in a new silo. All three of these are really bad. They bloat up the lines of code, enshrine the earlier flaws, increase disorganization, and waste time with redundant work. They get you out of the gate a little faster, but then you’ll be stuck in the swamp forever.<br /><br />If you pick the wrong abstraction then refactoring to correct it is boring. But it is usually a constrained amount of work and you can often do it in parts. If you apply the changes non-destructively, during the cleanup phase, you can refactor away some of the issues and check their correctness, before you pile more stuff on top. If you do that a bunch of times, the codebase improves for each release. You just have to be consistent about your direction of refactoring, waffling will hurt worse.<br /><br />But that is true for all coding styles. If you make a mistake, and you will, then so long as you are consistent in that mistake, fixing it is always a smaller amount of work or at the very least can be broken down into a set of small amounts. If there are a lot of them, you may have to apply the sum over a large number of different releases, but if you persist and hold your direction constant, the code will get better. A lot better. Contrast this with freezing, where the code will always get worse. The mark of a good codebase is that it improves with time.<br /><br />Sometimes people are afraid of what they see as the creativity involved with finding a new abstraction. Most abstractions however are not particularly creative. Really they are often just a combination of other abstractions fitted together to apply tightly to the current problem. That is, abstractions slowly evolve, they don’t just leap into existence. That makes sense, as often you don’t fully appreciate their expressibility until you’ve applied them a few times. So, it’s not creativity, but rather a bit of research or experience. <br /><br />Programming is complicated enough these days that you will not get really far with it if you just stick to rediscovering everything yourself from first principles. Often the state of the art has been built up over decades, so going all of the way back in time and trying to reinvent everything again is going to be crude in comparison. <br /><br />This is why learning to research a little is a necessary skill. If you decide to write some type of specific computation, doing some reading beforehand about others' experiences will pay huge dividends. Working with experienced people will pay huge dividends. Absorbing any large amount of knowledge efficiently will allow you to start from a stronger position. Code is just a manifestation of what the programmer understands, so obviously the more they understand the better the code will be.<br /><br />The other side of this is that an inexperienced programmer seeking a super-creative abstraction will often be a disaster. This happens because they don’t fully understand what properties are necessary for coverage, so instead they hyper-focus on some smaller aspect of the computation. They optimize for that, but the overall fit is poor. <br /><br />The problem though is that they went looking for a big creative leap. That was the real mistake. The abstraction you need is a generalization of the problems in front of you. Nothing more. Step back once or twice, don’t try to go way, way out, until much later in your life and your experience. What you do know should anchor you, always.<br /><br />Another funny issue comes from concepts like patterns. As an abstraction, data structures have nearly full coverage over most computations, so you can express most things, with a few caveats, as a collection of interacting data structures. The same isn’t true for design patterns. They are closer to idioms than they are to a full abstraction. That is why they are easier to understand and more tangible. That is also why they became super popular, but it is also their failure. <br /><br />You can decompose a problem into a set of design patterns, but it is more likely that the entire set now has a lot of extra artificial complexity included. Like an idiom, a pattern was meant to deal with a specific implementation issue, it would itself just be part of some abstraction, not the actual abstraction. They are implementation patterns, not design blocks. Patterns should be combined and hold places within an abstraction, not be a full and complete means of expressing the abstraction or the solution.<br /><br />Oddly programmers so often seek one-size-fits-all rules, insisting that they are the one true way to do things. They do this because of complexity, but it doesn’t help. A lot of choices in programming are trade-offs, where you have to balance your decision to fit the specifics of what you are building. You shouldn’t always go left, nor should you always go right. The moment you arrive at the fork, you have to think deeply about the context you are buried in. That thinking can be complex, and it will definitely slow you down, thus the desire to blindly always pick the same direction. The less you think about it, the faster you will code, but the more likely that code will be fragile.<br /><br />You can build a lot of small and medium-sized systems with brute force. It works. You don’t need to learn or even like abstractions. But if you want to work on large systems, or you want to be able to build stuff way faster, abstractions will allow you to do this. If you want to build sophisticated things, abstractions are mandatory. Once the inherent complexity passes some threshold, even the best development teams cannot deal with it, so you need ways of managing it that will allow the codebase to keep growing. This can only be done by making sure the parts are encapsulated away from each other, and almost by definition that makes the parts themselves abstract. That is why we see so many fundamental abstractions forming the base of all of our software, we have no other way of wrangling the complexity.<br />Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-61866022928764480692023-11-09T20:00:00.038-06:002023-11-09T20:00:00.136-06:00Time RangesSometimes you can predict the future accurately. For instance, you know that a train will leave the station tomorrow at 3 pm destined for somewhere you want to go. <br /><br />But the train may not leave on time. There are a very large number of ‘unexpected’ things that could derail that plan, however, there is only a tiny probability that any of them will actually happen tomorrow. You can live with that small uncertainty, ignore it.<br /><br />Sometimes you can only loosely predict the future. It will take me somewhere between 10 minutes to 1.5 hours to make dinner tonight. It’s a curve and the probability is most likely that the time it takes to make dinner lands somewhere in the middle; so not 10 mins and not 1.5 hours. It may only take half an hour, but I will only be certain about that after I finish.<br /><br />If you have something big to accomplish and it is made up of a huge number of little things, and you need to understand how long it will take, you pretty much should only work with time ranges. Some things may be certain like train schedules, but more often the effort is like making dinner. This is particularly true if it is software development.<br /><br />So, you have 10 new features that result in 30 functional changes to different parts of the code. You associate a time range for each functional change. Then you have to add in the time for various configurations and lots of testing. <br /><br />Worth noting that the time to configure something is not the actual time to add or modify the parameters, that is trivial. It is the time required to both find and ensure that the chosen configuration values are correct. So, lots of thinking and some testing. Might take 1 minute to update a file, but up to 3 days to work through every possible permutation until you find one that works as expected. Then the range is 5 mins if you get lucky, and 3 days if the universe is against you, which it seems to be sometimes.<br /><br />For most things, most of the time, you’ll get a bit of luck. The work falls on the lower side of the range. But sometimes life, politics, or unexpected problems cause delays. With time ranges, you can usually absorb a few unexpected delays and keep going. <br /><br />As the work progresses the two big levers of control are a) the completion date and b) the number of features. If luck is really not on your side, you either move the date farther out or drop a few features. You often need to move one or the other lever, which is why if they are both taken away it becomes more likely the release will explode.<br /><br />Some things are inherently unestimatable. You can’t know what the work is until someone has managed to get to the other side and there is no way to know if anyone will ever get to the other side. <br /><br />These types of occurrences are a small part of development work, but lots of other stuff is built on top of them. If you have something like that, you do that exploration first, then re-estimate when you are done. <br /><br />For example, half the features depend on you making an unguessable performance improvement. If you fail to guess how to fix that lower issue in a reasonable time frame, then those features get cut. You can still proceed with the other features. The trick is to know that as early as possible, thus don’t leave inestimable work until the end. Do it right away.<br /><br />It’s worth noting too that coding is often only one-third of the time. The analysis and design should be equal to the coding time, as should the testing time. That is, it can take 3 times longer to get done than most programmers expect, but they start 2/3rds of the way through. <br /><br />This is often a shortcut by doing almost no analysis or design, but that tends to bog things down in scope creep and endless changes. Skimping on testing lets more bugs escape into production which makes any operation drama far more expensive. So, in both cases attempting to save time by not doing necessary work comes back to haunt the project later and always ends up wasting more time than what was saved. Shortcuts are always nasty time trade-offs. Save a bit of time today, only to pay more for it later.<br /><br />If you have all of the work items as time ranges, then you can pick an expected luck percentage and convert the schedule into an actual date. Most times, it’s around the 66% mark. You can tell if you are ahead or behind, and you can lock in some of the harder and easier work early, so there is at least something at the end. If you end up being late, at least you know why you are late. For example, most of the tasks ended up near their maximum times. <br /><br />Time ranges also help with individual estimates. For example, you can ask a young programmer for an estimate and also a senior one. The difference will give you a range. In fact, everyone could chime in with dates, accurate or crazy, and you’d have a much better idea of how things may or may not progress. You don’t have to add in random slack time, as it isn’t ever anchored in reality. It is built in with ranges. <br /><br />Time ranges are great. You keep them as is when you are working, but can convert them to fixed dates when dealing with external people. If you blow the initial ranges, you’ll know fairly early that you are unlucky or stuck in some form of dysfunction. That would let you send out an early ‘heads up’ that will mitigate some of the anger from being late.<br />Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-5119152714495308262023-11-02T20:00:00.030-05:002023-11-02T20:00:00.130-05:00Special CasesOne of the trickiest parts of coding is to not let the code become a huge mess, under the pressure of rapid changes.<br /><br /><div>It’s pretty much impossible to get a concrete static specification for any piece of complex software, and it is far worse if people try to describe dynamic attributes as static ones. As such, changes in software are inevitable and constant. You write something, be prepared for it to change, it will change.<br /><br />One approach to dealing with this is to separately encode each and every special case as it own stand-alone siloed piece of code. People do this, but I highly recommend against it. It is just an exponential multier in the amount of work and testing necessary. Time that could have been saved by working smarter.<br /><br />Instead, we always write for the general case, even if the only thing we know today is one specific special case. <br /><br />That may sound a bit weird, but it is really a mindset. If someone tells you the code should do 12 things for a small set of different data, then you think of that as if it were general. But then you code out the case as specified. Say you take the 3 different types of data and put it directly through the 12 steps.<br /><br />But you’ve given it some type of more general name. It is isn’t ProcessX, it something more akin to HandleThese3TypesOfData. Of course, you really don’t want the name to be that long and explicit. Pick something more general that covers the special case, but does not explicitly bind to it. We’re always searching for ‘atomic primitives’, so maybe it is GenerateReport, but it only actually works for this particular set of data and nly goes these 12 steps.<br /><br />And now the fun begins. <br /><br />Later, they have a similar case, but different. Say it is 4 types of data, but only 2 overlap with the first instance. And it is 15 steps, but only 10 overlap.<br /><br />You wrap your generate report into some object or structure that can hold any of the 5 possible datatypes. You set an enumeration that switches between the original 12 steps and the newer 15 steps. <br /><br />You put an indicator in the input to say which of the 2 cases match the incoming data. You write something to check the inputs first. Then you use the enum to switch between the different steps. Now someone can call it with either special case, and it works.<br /><br />Then more fun.<br /><br />Someone adds a couple more special cases, You do the same thing, trying very carefully to minimize the logic wherever possible. <br /><br />Maybe you put polymorphism over the input to clean that up. You flatten whatever sort of nested logic hell is building up. You move things around, making sure that any refectoring that hits the original functionality is non-destructive. In that way, you leverage the earlier work, instead of redoing it.<br /><br />And it continues.<br /><br />Time goes by, you realize that some of the special cases can be collapsed down, so you do that. You put in new cases and collapse parts of the cases as you can. You evolve the complexity into the code, but make sure you don’t disrupt it. The trick is always to leverage your earlier work.<br /><br />If you do that diligently, then instead of a whole pile of spaghetti code, you end up with a rather clean, yet sophisticated processing engine. It takes a wide range of inputs but handles them and all of the in-between permutations correctly. You know it is correct because, at the lower levels, you are always doing the right thing.<br /><br />It’s not ‘code it once and forget it’, but rather carefully grow it into the complicated beast that it needs to become in order to be useful. <br /></div>Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-17966678288563650592023-10-26T20:00:00.001-05:002023-10-26T20:00:00.141-05:00Two GeneralsThere are two armies, encamped on different sides of a large city.<br /><br />If both armies attack the city at exactly the same time they will be victorious. If only one army attacks, the city defenders can wipe it out.<br /><br />One of the generals wants to notify his peer in the other army about when to begin the attack. He sends out a message “Tomorrow at 6am”. But he receives no reply.<br /><br />He has a huge problem now. Did his messenger make it to the other general? Maybe he did, but the returning messenger was captured and killed. Or maybe his messenger never made it. As he was sneaking around the outskirts of the city he met an untimely end.<br /><br />So, tomorrow morning, should he attack as he said he would, assuming the messenger was successful? Or should he not attack? <br /><br />Whether the message was received or not is ‘ambiguous’. The general cannot know which of the two possibilities is true; that his message didn’t make it or that the reply didn’t come back. He doesn’t have enough information to make an informed decision.<br /><br />Yet the fate of the battle rests on reliable communications…<br /><br />I’m sure that many readers have various suggestions for ways to remove the ambiguity. For instance, you could send other messengers, lots of them. But if they just disappear too, then the ambiguity is still there. You could try to establish waypoints, so the distance of the communication is shorter. But if there is still even a tiny corridor where the defenders reign supreme, it makes no difference.<br /><br />What if instead of sending “tomorrow at 6am” you send it as a question “What about tomorrow at 6am?” Then if it is intercepted the general isn’t compelled to act. But now the other general has the exact same original problem with their reply. It swapped the problem, but it didn’t go away.<br /><br />Clever people might suggest a different, alternative medium. Like flags or something, but the city is large enough that there is no guarantee about visibility, and if you used smoke or balloons, the defenders could just clear it away. The medium isn’t the problem.<br /><br />The thing is, there is no perfect, always-working scenario. Where you need information but there is only an ambiguity, no matter how small you shrink it, it still remains. The ambiguity is the information.<br /><br />Worse is that this is a fundamental physical constraint imposed by the universe for any sort of distributed communication. All distributed software, where separate computations need to synchronize on any sort of non-perfect medium, is bound by it. If you have a client and server, or a bunch of peers, or even two processes using a less-than-perfect medium such as files, no technology, protocol, or magic bullet will make the conversation 100% reliable, and even if it is 99.9999% reliable, there is still some sort of ambiguity in your way. <br /><br />At your very best you could shrink it down to say a 1 in 100 years likelihood of failure, so you could not see it go wrong in your entire career, but it will still go wrong, someday. And that is what makes it so different from a regular computation. It is not deterministic.<br /><br />Short of some unexpected external event like a flood or gamma rays or a hardware defect or something, all of the other computations will work perfectly each and every time they run on the computer. If they work for their full context, you can always assume they will always work. Oddly, we treat them as 100%, even though the physical nature of the computation itself is subject to adverse advents, the software itself, as a formal system, is not.<br /><br />There is and will always be a huge gulf between 100% and 99.9...%, between deterministic computations and non-deterministic (in the distributed sense, not the language theory one) ones. Ultimately it affects the level of trust that we place in our software.<br />Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-80915276389417672372023-10-19T20:00:00.012-05:002023-10-19T20:00:00.139-05:00ImpermanencePhysical things can hang around for a long, long time. You can keep them in your house, for example. Short of some epic disaster like a fire, it is up to you how long you value them and when you finally get rid of them. It is in your control. For some people, it is possible to keep their stuff safe for their entire life. It becomes an heirloom. <br /><br />Digital things are impermanent. There are an endless number of ways for them to get lost, forgotten, or corrupted. <br /><br />The next hardware you buy is probably only good for five years, maybe a little bit longer. Rolling over to newer hardware probably won’t go well, even if you spend a ridiculous amount of time trying to figure it out. <br /><br />The cloud storage you pay for may be discontinued or out of business next week. They probably scrimped on backups, so odds are that in a disaster the stuff you wanted won’t make it. They’ll apologize, of course. And they are always working harder on finding ways to increase the price than they are on finding ways to make it more reliable, engineering is not as important as exploitation. You may wake up one day to a nasty price increase. Storing stuff on the cloud is extremely hazardous.<br /><br />Compounding it all, software keeps changing. Most programmers suck at achieving any real sort of backward compatibility. They’ll just force you to wipe out everything because it was easier for them to code it that way. The chances that persistent data survives for longer than a decade are slim. And even if technically it did survive, it has probably become unreachable, unusable. The software you used before to leverage it has long since been broken by someone else who didn’t know or care about what you need.<br /><br />There was a big stink about forgetting things on the web, but honestly, it was a total waste of time. Not only is the web tragically forgetful, but the infrastructure on the web is getting a little more useless every day. Technically stuff could still be out there, but how would you find it now? The web is about hype, not information. Stuff tends to vanish when the limelight gets shifted.<br /><br />In the grand scheme of things, the digital realm is extraordinarily flakey. Far more of its history is lost than preserved. It’s a dangerous, somewhat unpredictable place where selfishness and irrationality have more weight than quality. The odds are better that someone else whom you don’t want to see your data will see it than you getting that same data back in twenty or thirty years. We make stuff digital to be trendy, not smart.<br />Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-14405511948620125982023-10-12T20:20:00.000-05:002023-10-12T20:20:33.569-05:00Curiosity<p dir="ltr" id="docs-internal-guid-09d71788-269b-73d5-6b30-174af6dd54b3" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">If I had to pick one quality that I think is necessary to make a big software project run smoothly, it is curiosity.</span></p><p dir="ltr" id="docs-internal-guid-09d71788-269b-73d5-6b30-174af6dd54b3" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">If you get 100% code that perfectly fits to the domain problems, the project will be great. If you correctly guessed how long it would take in advance, and people actually believed you and gave you that time, the politics would be negligible. </span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">But anyone who has been on a bunch of big projects for a long time knows that it almost never works that way.</span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">It’s usually some sort of dumpster fire. The time isn’t enough, the fit is bad, and the technology is flaky.</span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">So, ultimately, for the first few versions of the code, you do what you need to do in order to get it out the door. It ain’t pretty but it seems to be an inescapable reality of the job.</span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">But after that smoke clears, and if there is still an appetite to go forward, the circumstances have changed. Hopefully, there is confidence in the work now.</span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Curious people will look back and what happened earlier, and what they have, and start asking the hard questions. Why did someone do that? How is that supposed to work? etc. </span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">If they are curious and enabled, then at least some of the huge collection of smaller problems that plague the earlier versions are now in their focus. And they will get looked at, and hopefully corrected. All of that happens outside of the main push for whatever new features other people desire. </span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">It is fundamentally cleanup work. There is a lot of refactoring, or replacement of weak parts. There is more investigation, and deep diving into the stranger issues. All of which is necessary in order to build more on what is there now.</span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="-webkit-text-size-adjust: auto; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;">Non-curious people will just claim that something is already in production, that it is locked that way, and it should not be touched, ever. They will enshrine the damage, and try to move on to other things. That is a classic mistake, building anything on a shaky foundation is a waste of time. But you have to be curious about the foundation in order to be able to assess that it is not as good as necessary.</span></p><p><br style="-webkit-text-size-adjust: auto;" /><br style="-webkit-text-size-adjust: auto;" /><br style="-webkit-text-size-adjust: auto;" /><br style="-webkit-text-size-adjust: auto;" /></p>Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-77579841770377653492023-10-05T20:00:00.035-05:002023-10-05T20:00:00.136-05:00FeedbackI was working with a group of people a while ago who built a web app and put it on the Internet. It was a bit crude, didn’t quite follow normal user conventions, and was quite rough around the edges. When they built it, they added a button to get users' feedback. <br /><br />Once they put it out there live, they got swamped by negative feedback. People took the time to complain about all sorts of things, to point out the deficiencies. It was a lot. They were overwhelmed.<br /><br />So, they removed the feedback button.<br /><br />As far as solving problems goes, this was about the best example of the worst possible way to do it. They had this engaged group of people who were willing to tell them what was wrong with their work, and instead of listening to that feedback and improving, they just shut down the interaction. <br /><br />Not surprisingly, people avoided the app and it never really took off. <br /><br />For programmers, feedback is difficult. We are already on thin ice when we build stuff, as there is more we don’t know about what we are doing than what we do know. And, it is easy to throw together something quickly, but it takes a crazy long time to make it good. This all leaves us with a perpetual feeling of uncertainty, that the things we build could alway be way better. You never really master the craft. <div><br /></div><div>Those nagging doubts tend to make most programmers highly over-sensitive to criticism. They only want positive feedback. <br /><br />On top of that, user feedback is almost never literal. The users are vague and wishy-washy when they talk about what is wrong or why something bothers them. <br /><br />They often know what they don’t like, but they do not know what is better or correct. Just that it is wrong. They are irrational and they usually don’t like fully explaining themselves. <br /><br />In order to make sense of what they are saying, you have to learn to read between the lines. Use what they say to get an idea that something might be wrong, but then work out the actual problems on your own. Once you think you understand, you change things and test to see if that is better. It’s a soft, loose process, that usually involves endless rounds of refinements. <br /><br />Things gradually get better, but it isn’t boolean. It’s not done or undone, it is a convergence. Thinking of it as one discrete task is a common mistake for popular feedback-tracking tools. They confuse the rigor of issuing instructions to a computer with the elasticity of interfaces. They try to treat it all the same when it is completely different. <br /><br />If you were being pragmatic about it, you would capture all of the feedback and triage it. Positive or negative, categorized by the visible features that are involved. Then you might collect together certain negative entries and hypnosis that the underlying cause is something tangible. From there, you would schedule some work, and then schedule some form of testing. The overall category of the problem though would likely never really go away, never get resolved. It would stay there for the life of the system. Just something that you are gradually working towards improving.<br /><br />The classic example is when the users say that the system or some of its features are “awkward”. That most often means that parts of the behavior do not ‘fit’ well with the users as they deal with their problem domain. It could be because the workflow is wrong, or that the interface conventions clash with the other tools they are using, or that the features should have been somewhere else, or that it is all too slow to be usable. It is hard to tell, but it is still vital feedback.<br /><br />You don’t “de-awkward” a system, it is not a ‘thing’. It’s not a requirement, a ticket, a feature request, anything really. It is more about the ‘feel’ the users experience while using the features. If you want to make it less awkward, you probably have to directly interact with them while they are doing things they find awkward, then take the scratchy points you observed and guess how to minimize them. You definitely won’t be 100% correct, you might not even be 10% correct. It will take a lot to finally get your finger on the types of things that ‘you’ can do to improve the situation. <br /><br />A rather huge problem in the software industry is that most people don’t want to do the above work. They only want to solve contained discrete little problems, not get lost in some undefinable, unquantifiable, swamp. We lay out methodologies, build tools, and craft processes on the assumption that all things are atomic, discrete, and tangible, and it shows in our outputs. ‘Awkward’ comments are ignored. The bad behaviors get locked in, unchangeable. People just wrap more stuff around the outside, but it too suffers from the same fate. Eventually, we just give up, start all over again from first principles, and eventually arrive at the same conclusion again. It’s an endless cycle where things get more complicated but gradually less ‘user-friendly’.<br /></div>Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0tag:blogger.com,1999:blog-6104420435021904082.post-26987394270469724732023-09-28T20:00:00.007-05:002023-09-28T20:00:00.152-05:00TrifectaRight from the beginning of my career, I have been bothered by the way we handle software development. As an industry, we have a huge problem with figuring out ‘who’ is responsible for ‘what’.<br /><br /><div>For decades, we’ve had endless methodologies, large and small, but all of them just seem to make poor tradeoffs between make-work and chaos. Neither is appealing.<br /><br /></div><div>As well, there are all sorts of other crazy processes and plenty of misconceptions floating around. Because of this most projects are dumpster fires, which only adds to the stress, wastes energy, and ensures poor quality.<br /><br />For me, whenever development has worked smoothly it was been because of strong personalities who are subverted the enforced methodology. Strong, knowledgeable leadership works well. <br /><br />Whenever the projects have been excessively painful, it is often caused by confusion in the roles and responsibilities which resulted in poor outcomes. Politics blossoms when the roles or rules are convoluted or vague. Focus gets misplaced, time gets wasted, and the quality plummets. It gets ugly.<br /><br />It’s not that I have an answer, but after 30 years of working and 17 years of writing about it, I feel like I should at least lay down some basic principles.<br /><br />So, here goes...<br /><br />There are three primary areas for software. They are: a) the problem domain, b) the operational environment, and c) the development environment. Software (c) is a set of solutions (b) for some problems (a). <br /><br />A system is a collection of similar solutions for a common problem domain.<br /><br />There are two primary motivators for creating software: a) vertical and b) horizontal. <br /><br />A vertical motivator is effectively a business-driven need for some software. Either they use it, offer it as a service, or sell it. <br /><br />A horizontal motivator is an infrastructural need for some software. Missing parts of the puzzle that are disrupting either the operational or development flow. <br /><br />Desired quality is a growing exponential curve, where low-quality throw-away code is to the left, then static, hardcoded, in-house development, then decent commercial products, then likely healthcare, aerospace, and NASA. To get to the next category is maybe 2x - 10x more work for each hop. <br /><br />The actual quality is the desired level plus the sum of all testing, which is also exponential. So to find the next diminishing set of less visible bugs is 2x - 10x more effort. There is an endless series of bug sets. Barely reasonable commercial quality is probably a 1:1 ratio of testing with coding.<br /><br />The quality of the code itself is dependent on the design and the enforcement of good style and conventions. Messy code is buggy code. The quality of the design is dependent on the depth of the analysis. The overall results are a reflection of the understanding of the designers and coders. The problem domain is often vague and irrational but it has to be mapped to code which is precise and logical. That is a very tricky mapping.<br /><br />Ultimately while software is just instructions for a computer to run, its genesis is from and all about people. It is a highly social occupation. Non-trivial software takes a team to build and a team to run.<br /><br />So, for every system, we end up with three main players:<br /><ul style="text-align: left;"><li>Domain Champion</li><li>Operations Manager</li><li>Lead Software Developer</li></ul>The domain champion represents all of the users. They also represent some of the funding, they are effectively paying for work to get done. They have a short-term agenda of making sure the software out there runs as expected and they are the ones that commission new features for it. They drive any non-technical analysis. They have or can get all of the answers necessary for the problem domain, which they need to understand deeply.<br /><br />The operational manager is effectively the day-to-day ‘driver’ of the software. They set it up and offer it for others to use. They need to get the software installed, upgraded, and carefully monitor it. They are the front line for dealing with any issues the users encounter. They offer access as a service.<br /><br />The lead developer builds stuff. It is constructive. They should focus on figuring out the stuff that needs to be built and the best way to do it, given all of the domain and operational issues. The features are usually from the top down, but to get effective construction the code needs to be built from the bottom up. The persistence foundation should exist before the GUI, for example.<br /><br />For most domain functionality, the champion is effectively responsible for making sure that the features meet the needs of the users. The lead makes sure the implementation of those features is reasonable. They do this by breaking those features down into lots of different functionality to get implemented. A champion may need the system to keep track of some critical data, the lead may implement this as a set of ETL feeds and some user screens.<br /><br />If there are bugs in production the users should go directly to the operations manager. If the manager is unable to resolve the issues, then they would go to the lead developer, but it would only be for bugs that are brand new. If it's recurring, the manager already knows how to deal with it. The operations manager would know if the system is slow or overusing resources. Periodically they would provide feedback to the lead.<br /><br />If a project is infrastructure, cleanup, or reuse, it would be commissioned directly by the lead developer. They should be able to fund maintenance, proof of concept, new technologies, and reuse work on their own since the other parties have no reason to do so. The project will decay if someone doesn't do it. <br /><br />The lead needs to constantly make the development process better, smoother, and more effective. They need to make sure the technology used is keeping up with the industry. Their primary focus is engineering, but they also need to be concerned with solution fit, and user issues like look and feel. They set the baseline for quality. If the interface is ugly or weird, it is their fault.<br /><br />As well as the champion, the operations manager would have their own system requirements. They set up and are responsible for the runtime, so they have a strong say in the technologies, configuration, security, performance, resource usage, monitoring, logging, etc. All of the behavior and functionality they need to do their job. If they have lots of different systems, obviously having it consistent or aligned would be highly important to them. They would pick the OS and persistence for example, but not the programming language. The dependencies used for integration would fall under their purview.<br /><br />The process for completing the analysis needed to come up with a reasonable set of features is the responsibility of the champion. Any sort of business analyst would report to them. They would craft the high-level descriptions and required features. This would be used by the lead to get a design.<br /><br />If the project is infrastructure, instead of the champion, it is the responsibility of the lead to do the analysis. Generally, the work is technical or about organization, although it could be reliant on generalities within the problem domain. The work might be combining a bunch of redundant software engines altogether, to get reuse, for example.<br /><br />Any sort of technical design is the lead, and if the organization is large, they likely need to coordinate the scope and designs with the firm’s architects and security officers. As well, the operational requirements would need to be followed. A design for an integrated system is not an independent silo, it has to fit with all of the other existing systems.<br /><br />Architects would also be responsible for keeping the higher level organized. So, they wouldn’t allow the lead or champion to poach work from other teams.<br /><br />The process of building stuff is up to the lead. They need to do it any which way, and in any order, that best suits them and their teams. They should feel comfortable with the processes they are using to turn analysis into deployment.<br /><br />They do need to give time estimates, and if they miss them, detailed explanations of why they missed them. Leads need to learn to control the expectations of the champion and the users. They can’t promise two years of work in six months, for example. If development goes poorly or the system is unusable they are on the hook for it.<br /><br />There should be a separate quality assurance department that would take the requirements from the champions, leads, and operations managers, and ensure that the things being delivered meet those specifications. They would also do performance and automated testing. With the specs and the delivery items, they would return a report on all of the deficiencies to all three parties. The lead and champion would then decide which issues to fix. Time and expected quality would drive those decisions.<br /><br />The items that were tested in QA are the items that are given directly to operations to install or upgrade. There are two release processes. The full one and the fast one. The operations manager schedules installations and patches at their own convience and notifies the users when they are completed. The lead just queues up the almost-finished work for QA.<br /><br />The lead has minimal interaction with operations. They might get pulled into net new bug issues, they get requirements for how the software should operate, and they may occasionally, with really tricky bugs, have to get direct special access to production in order to resolve problems. But they don’t monitor the system, and they aren’t the frontline for any issues. They need to focus on designing and building stuff.<br /><br />The proportion of funding for the champion and for the lead defines the control of technical debt. If the system is unstable or development is slow, more funding needs to go into cleanup. The champion controls the priority of feature development, and the lead controls the priority of the underlying functionality. That may mean that a highly desired feature gets delayed until missing low-level functionality is ready. Building code out of order is expensive and hurts quality.<br /><br />So that’s it. It’s a combination of the best of all of the processes, methodologies, writing, books, arguments, and discussions that I’ve seen over the decades and in the companies that I have worked for directly or indirectly. It offsets some of the growing chaos that I’ve seen and puts back some of the forgotten knowledge. <br /><br />All you need, really, is three people leading the work in sync with each other in well-defined roles. There are plenty of variations. For example in pure software companies, there is a separate operations manager at each client. In some cases, the domain champion and lead are the same person, particularly when the domain is technical. So, as long as the basic structure is clear, the exact arrangement can be tweaked. Sometimes there are conflicting, overlapping champions pulling in different directions.<br /></div>Paul W. Homerhttp://www.blogger.com/profile/02349253120538728302noreply@blogger.com0