Google Chrome took over from IE for three big reasons:
This made Chrome far easier to develop for, and Chrome grew a reputation for being a web developer's browser. This fit in nicely to Google's plan for the future, which they saw as full of web applications. Google understands what they have, and how they got there. They also understand "embrace and extend," but found a way to do that without making it proprietary the way Microsoft did: capture the standards committees.
If you capture the standards committees, meaning what you want is almost guaranteed a rubber stamp from the committee, then you get to define what industry standard is. Microsoft took a capitalist, closed-source approach to embrace and extend where the end state was a place where the only viable way to do a thing was the thing that was patent-locked into Microsoft. Google's approach is more overtly FOSSY in that they're attempting to get internet consensus for their changes, while also making it rather harder for anyone else to do what they do.
Google doesn't always win. Their "web environment integrity" proposal, which would have given web site operators far greater control over browser extensions like ad-blockers, quietly got canned recently after internet outrage. Another area that got a lot of push back from the internet was Chrome's move away from "v2 manifest" extensions, which include ad-blockers, in favor of "v3 manifest" plugins which made ad-blockers nearly impossible to write. The move from v2 to v3 was delayed a lot while Google grudgingly put in abilities for ad-blockers to continue working.
Getting off of Chrome
The circumstances that drove the world off of Internet Explorer aren't there for Chrome.
One thing has changed since the great IE to Chrome migration began, Google lost its positive reputation. The old "don't be evil" thing was abandoned a long time ago, and everyone knows it. Changes proposed by Google or Google proxies are now viewed skeptically; though, overtly bad ideas still require internet outrage to delay or prevent a proposal from happening.
That said, you lose monopolies through either laziness of the monopolist (Microsoft) or regulatory action, and I'm not seeing any signs of laziness.
]]>What we really need is markdown in a git repository. We get version control, there is a lot of tooling to make markdown work good in git, it's great
And every time I have to grit my teeth and hope I don't cause dental damage. My core complaint is that internal documentation has fundamentally different objectives than open source software documentation repositories, and pretending they're the same problem domain means we'll be re-having the documentation discussion in 18 to 24 months.
The examples of OSS projects using markdown or asciidoc as their documentation repository are many, and it works pretty well. Markdown and asciidoc are markup, which allows compilers to turn the marked up doc into rendered sites. This makes accepting contributions from the community much easier, because it follows the same merge-request workflow as code. As most OSS projects are chronically under-staffed, anything that allows reuse of process is a win. Also, markdown and asciidoc are relatively simple formats so you don't need expensive software like Adobe InDesign to make them.
OSS project docs are focused on several jobs to be done, and questions by readers:
Corporate internal documentation repositories need to do all of the above, but generally for a much wider range of things and services. Cool, that's what standards are for. But "markdown in a git repo" goes a bit off the rails when you look at all the other types of documentation internal docs often cover:
And so on. The sneaky part here is that the OSS projects have many of the above as well, but they're kept in things like Google Docs, Etherpads, Wikis, Twillio, Canvases in Slack, many things that are definitely not involving the merge request workflow into git. All of these extra internal documentation repository jobs to be done greatly complicate what solutions count as viable, in large part because this huge list is actually trying to 'simplify' multiple documentation styles into a single monolithic document repository. What styles are these? Well:
All of these documentation styles need somewhat different document lifecycles, which in turn drives need to support workflows. A document lifecycle ensures that documentation is valid, up to date, and old information is removed. Sometimes documentation is a key part of compliance with regulation or industry standard-setting bodies, which adds review steps.
Ideally, the high lifecycle docs like product and process documentation would be in one system, with the minimal lifecycle docs like decision review and responder runbooks in another system entirely. This would allow each system to cater to the needs of the styles within, and solve more of the business' problems. I would like a two-system solution very much.
Except.
People have spent the last 25 years being trained that how you find documentation is:
A two doc-system solution is not well tolerated, and people will build a "universal search" engine to search both the high and low process repositories. Also, two doc systems seems like a lot of overhead. And how do you make sure the right docs go in the right system? Why not use one doc system that's sort of okay at both jobs and save money? 18 to 24 months later, discontent at how bad the "sort of okay" solution is rises and people advocate to moving to a new thing, and suggest markdown in a git repo.
]]>I can't find when I permanently dropped SeaMonkey, but it was after 2010. I dropped SeaMonkey late 2013 (thank you stale profile directory with a date stamp) when it was clearly abandonware and I learned you actually could launch Firefox in parallel with multiple profiles (the
firefox -p --no-remote
combination was key). I stopped using multiple profiles when the Container plugin came out that did nearly everything separate profiles did. It turns out SeaMonkey is still getting updates, but it seems to be tracking the Firefox and Thunderbird Extended Service Releases.
For those of you too old to remember the original Netscape Navigator, it also came with a few tools beyond the browser:
The reason I liked SeaMonkey and Opera is they both still shipped with an email client. It was pretty nice, actually. I kept Opera around as my email client way past when I stopped using it for general browsing. I'm fuzzy on what I did after Opera dropped their mail client, I may have grumpily transitioned onto Gnome Evolution at that point. Also, Gmail was out and I was quite used to web-based email clients.
So yeah, I've been in Firefox for over a decade at this point.
]]>Yes, there are some actual full up standards. Things like RFCs and ISO-standards are actual standards. There are open standards that are widely adopted, like OpenTelemetry and the Cloud Native Computing Foundation suite, but these are not yet industry standards. The phrase "industry standard" implies consensus, agreement, a uniform way of working in a specific area.
Have you seen the tech industry? Really seen it? It is utterly vast. The same industry includes such categories as:
The only thing the above have any kind of consensus on is "IP-based networking is better than the alternatives," and even that is a bit fragile. Such out there statements like "HTTP is a standard transport" are ones you'd think there would be consensus on, but you'd be wrong. Saying that "kubernetes patterns are industry standard" is a statement of desire, not a statement of fact.
Thing is, the Sysadmin community used this mechanic for self-policing for literal decades. Any time someone comes to the community with a problem, it has to pass a "best practices" smell test before we consider answering the question as asked; otherwise, we'll interrogate the bad decisions that lead to this being a problem in the first place. This mechanic is 100% why ServerFault has a "reasonable business practices" close reason:
Questions should demonstrate reasonable information technology management practices. Questions that relate to unsupported hardware or software platforms or unmaintained environments may not be suitable for Server Fault.
Who sets the "best practices" for the sysadmin community? It's a group consensus of the long time members, which is slightly different between each community. There are no RFCs. There are no ISO standards. The closest we get is ITIL, the IT Infrastructure Library, which we all love to criticize anyway.
Best practices, which is "industry standard" by an older name, have always been an "I know it when I see it" thing. A tool used by industry elders to shame juniors into changing habits. Don't talk to me until you level up to the base norms of our industry, pleeb; and never mind that those norms are not canonicalized anywhere outside of my head.
This is why the phrase "it's industry standard" should not be used in internal technical design conversations
This phrase is shame based policing of concepts. If something is actually a standard, people should be able to look it up and see the history of why we do it this way.
Maybe the "industry" part of that statement is actually relevant; if that's the case, say so.
These are useful, important constraints and context for people to know. The vague phrase "industry standard" does not communicate context or constraints beyond, "your solution is bad, and you should feel bad for suggesting it." Shame is not how we maintain generative cultures.
It's time to drop "it's industry standard" from regular use.
]]>This story begins in the previous decade when Google put forth the "Zero Trust framework" as a way to get rid of the corporate VPN. Zero Trust was a suite of techniques to allow companies to do away with the expensive and annoying to maintain VPN. The core concept behind Zero Trust was something I didn't truly understand until a few years ago: Zero Trust adds device attestation (a machine login) in addition to the user attestation when deciding whether to grant access to a resource, which is a more robust security boundary than a separate VPN login.
In a company context, you can reasonably expect the company to own both the server and the machine employees are accessing internal resources from. Zero Trust also enabled servers to specify a minimum security level that clients must meet in order to have access granted, such as up to date browser and OS versions, as well as a valid device identifier. This ends up being a major security win because when an employee has their user credentials phished, an attacker can't immediately use those stolen credentials to do evil things; the attacker will have to somehow gain a device identity as well.
In companies that use VPNs, phishing the user's credential was often enough to allow creating a VPN connection. If that wasn't enough, Phishing the VPN certificates would do the trick. Full Zero Trust makes the attacker's job harder. Again, in a corporate context where the company reasonably owns both the client and server sides of the conversation.
Back to WEI: Web-Environment-Integrity is a proposal to bring a limited form of device attestation to public surfaces. While WEI won't have a device credential, it will have the ability to attest to OS and Browser versions, presence and state of browser plugins, among other things. In theory this allows bank website operators (for example) to ensure people have a browser within three months of latest, are not operating any plugins known to sniff or record keystrokes and clicks, and are not running plugins with known vulnerabilities.
Unlike Zero Trust, the company does not reasonably own the client side of the conversation in a WEI context. This radically changes the power dynamics between public users and private servers. Under current internet rules, both sides mutually distrust each other and slowly establish trust. Under WEI, the server sets a minimum trust boundary that's far higher than is currently possible, which gives server operators far more power in this conversation than before. A Zero Trust like level of power, in fact.
What does Web-Environment-Integrity allow server operators to do?
As it happens, WEI is a clear example of a technique or standard that needs to have a clear and well thought out answer to the question:
What can a well resourced malicious actor do with this framework? How can they use this safety tool to harm people?
Right now, we don't have those answers. The explainer goes into some detail about how to avoid tracking through WEI techniques, but overlooks the thing that has everyone posting "Google wants to ban ad-blockers" headlines. The WEI proposal allows server operators to prohibit the use of specific browser plugins when accessing the site, which gives ad-networks an in-browser way to say "turn off your ad-blocker in order to see this content."
The well resourced malicious actor here is the internet advertising industry, of which Alphabet (Google's parent company) is one of the biggest members. The proposal writers do not view code injection and people-tracking through advertising to be malicious, they see it as a perfectly legitimate way to pay for services.
"But it's not the server operator doing the banning, it's the attestor; and the attestor has no idea what's on the site!"
The WEI standard involves three parties: The browser making the request, the server hosting the site, and the 'attestor' service the server relies on to judge the integrity of the browser making the request. The "Google wants to ban ad-blockers" headline happens when an advertising-driven site uses an attestor service that considers browsers with ad-block plugins to be insecure. Technically it isn't the server making the "no ad-block" constraint, at least at the time of the request. The Server operator made that choice when they selected an attestor service that prohibits ad-block plugins.
This sort of deniability is all over the tech industry.
]]>Q: How can we improve the diversity of the candidates we hire?
A: That actually hurts the diversity of your hiring pool, because many people see "diversity!" on a hiring page and immediately go somewhere else. Who wants to be hired to a place that'll give a job to an unqualified minority just to meet some numbers?
The mechanism this answerer was assuming, that diversity programs are quota systems, has been explicitly illegal in the US since the 1980s. The Supreme Court at the time ruled that quotas like that were what this answerer said: racism, even if it was intended to correct for systemic biases. If you find your prospective workplace is using quotas, or explicitly hiring following racialized patterns, you have strong grounds for a lawsuit.
Within the last year, news broke of a company that got into hot water by someone posting "I'm hiring for X, give me all of your non-binary, queer, women, and minority leads please!" to Twitter. The strong implication by this statement was that this company was using a racialized hiring process which is illegal.
These days hiring pipelines at large US companies are engineered to avoid getting sued, and therefore don't use quotas. To build a hiring pipeline that furthers a company's diversity goals, while also avoid getting sued, requires several things:
So far, this is an "equity" argument. But building a system to improve your workplace diversity needs a few more steps.
Equity, in other words.
Furthermore, some companies are beginning to reframe their diversity programs towards a "meet the market" approach. In that they assess their diversity program success based on how well their employee mix matches the potential job market for their roles. If a given position is 82% male in the job market, that's the target they'll push for; not 50%.
Equity, because that's the most legally conservative option if you want to avoid lawsuits for discrimination in US courts.
]]>By not designing your observability platform from the beginning for eventual integration into the overall data motion, you get a lot of hard to reduce misalignment of function, not to mention poorly managed availability assumptions. Did your logging platform hiccup for 20 minutes, thus robbing the business metrics people of 20 minutes of account sign-up/shutdown/churn metrics? All data is noisy, but data engineering folk really like it if the noise is predictable and thus can be modeled out. Did the metrics system have an unexpected reboot, which prevented the daily code-push to production because Delivery Engineering couldn't check their canary deploy metrics? Guess your metrics system is now a production critical system instead of the debugging tool you thought it was.
Data engineering folk like their data to be SQL shaped for a lot of reasons, but few observability and telemetry systems have an SQL interface. Mathew proposed a useful way to provide that:
When you have a log that must be stored for compliance or legal reasons, don't stick it into the same system you use to store every 200 - OK line. Write it to a database (ideally) or an object store outside of the logging pipeline. I've used DynamoDB for this and had it work pretty well by sticking it in an SQS pipeline -> Lambda -> Dynamo. Then your internal application can query this and you don't need to worry about log expiration with DynamoDB TTL.
Dynamo is SQL-like, so this method could work for doing business metrics things like the backing numbers for computing churn rate and monthly active users (MAU) numbers. Or tracking things like password resets, email changes, and gift-card usage for your fraud/abuse department. Or all admin-portal activity for your annual external audits.
Mathew also unintentionally called me out.
Chances are this isn't someones full-time job in your org, they just happened to pick up logging. It's not supposed to be a full-time gig so I totally get it. They installed a few Helm charts, put it behind an OAuth proxy and basically hoped for the best. Instead they get a constant flood of complaints from consumers of the logging system. "Logs are missing, the search doesn't work, my parser doesn't return what I expect".
That's how I got into this. "Go upgrade the Elasticsearch cluster for our logging system" is what started me on the road that lead to the Software Telemetry book. It worked for me because I was at a smaller company. Also our data people stuck their straw directly into the main database, rather than our logging system, which is absolutely a dodged bullet.
Mathew also went into some details around scaling up a metrics system. I spent a chapter and part of an appendix on this very topic. Mathew gives you a solid concrete example of the problem from the point of view of microservices/kubernetes/CNCF architectures. This stuff is frikkin hard, and people want to add things like account tags and commit-hash tags so they can do tight correlation work inside the metrics system; the sort of cardinality problem that many metrics systems aren't designed to support.
All in all Mathew lines out several common deficiencies in how monitoring/observability/telemetry is approached in many companies, especially growing companies:
The last point about Jaeger is a key one, though. Organizations green-fielding a new product often go with tracing only as their telemetry style, since it gives so much high quality data. At the same time, tracing is the single most expensive telemetry style. Unlike centralized logging, which has a high quality self-hosted systems in the form of ELK and Loki, tracing these days only has Jaeger. There is a whole SaaS industry who sell their product based on how much better they are than Jaeger.
]]>All the AI-branded features being pushed out by everyone that isn't ChatGPT, Google, or Microsoft all depend on one of the models from those three companies. That, or they're rebranding existing machine learning features as "AI" to catch the marketing moment. Even so, all the features getting pushed out come from the same base capabilities of ChatGPT style large language models:
Okay, that's the only capability. But this one capability is driving things like:
That's about it. We've had about half a year of time to react to ChatGPT and start generating products based on that foundation, and the above is what we're seeing. A lot of these are one to three engineering teams worth of effort over the course of a few months, with a month or three of internal and external testing. These are the basic features of LLM-based machine learning, also known right now as AI.
We don't yet know what features we'll have in two years. "Suggest text from a prompt" went from hot new toy to absolute commodity table-stakes in 8 months, I've never seen it before. Maybe we've fully explored what these models are capable of, but we sure haven't explored the regulatory impacts yet.
The regulatory impacts are the largest risk to these tools. We've already seen news stories of lawyers using these tools to write briefs with fictional citations, and efforts to create deep fake stories "by" prolific authors. The European Union is already considering regulation to require "cite your sources", which GPT isn't great at. Neither are GPT-based models good about the Right To Be Forgotten enshrined in EU law by GDPR.
The true innovation we'll see in coming years will be in creating models that can comply with source-citing and enable targeted excludes. That's going to take years, and training those models is going to keep GPU makers in profit margins.
]]>Spot on. I've felt this frustration too. I've been in the after-action review following an AWS incident when an engineering manager asked the question:
Can we set up alerting for when AWS updates their status page?
As a way to improve our speed of response to AWS issues. We had to politely let that manager down by saying we'd know there was a problem before AWS updates their page, which is entirely true. Status pages shouldn't be used for real-time alerting. This lesson was further hammered home after we gained access to AWS Technical Account Managers, and started getting the status page updates delivered 10 or so minutes early, directly from the TAMs themselves. Status Pages are corporate communications, not technical.
That's where the problem is, status pages are corporate communication. In a system where admitting fault opens you up to expensive legal actions, corporate communication programs will optimize for not admitting fault. For Status Pages, it means they only get an outage notice after it is unavoidably obvious that an outage is already happening. Status Pages are strictly reactive. For a true real time alerting system, you need to tolerate the occasional false positive; for a corporate communication platform designed to admit fault, false positives must be avoided at all costs.
How do we fix this? How do we, a SaaS provider, enable our downstream reliability programs to get the signal they need to react to our state?
There aren't any good ways, only hard or dodgy ones.
The first and best way is to do away with US style adversarial capitalism, which reduces the risks of saying "we're currently fucking up." The money behind the tech industry won't let this happen, though.
The next best way is to provide an alert stream to customers, so long as they are willing to sign some form of "If we give this to you, then you agree to not sue us for what it tells you" amendment to the usage contract. Even that is risky, because some rights can't be waved that way.
What's left is what AWS did for us, have our Technical Acount Managers or Customer Success Managers hand notify customers that there is a problem. This takes the fault admission out of the public space, and limits it to customers who are giving us enough money to have a TAM/CSM. This is the opposite of real time, but at least you get notices faster? It's still not equivalent to instrumenting your own infrastructure for alerts, and is mostly useful for writing the after-action report.
Yeah, the SaaSification of systems is introducing certain communitarian drives; but the VC-backed capitalistic system we're in prevents us from really doing this well.
]]>Have you ever seen a company culture recover after things went bad?
I have to say no, I haven't. To unpack this a bit, what my coworker was referring to is something loooong time readers of my blog have seen me talk about before, budget problems. What happens to a culture when the money isn't there, and then management starts cutting jobs?
In the for-profit sector of American companies, you get a heck of a lot of trauma from the sudden-death style layoffs (or to be technical reductions in force because there is no call-back mechanism in the SaaS industry). Sudden loss of coworkers with zero notice, scrambling to cover for their work, scrambling to deal with the flash reorganization that almost always comes with a layoff, scrambling to worry about if you'll be next, it all adds up to a lot of insecurity in the workplace. Or a lack of psychological safety, and I've talked about what happens then.
This coworker was also asking about some of the other side effects of a chronic budget crunch. For publicly traded companies, you can enter this budget crunch well before actual cashflow is a problem due to the forcing effects of the stock market. Big Tech is doing a lot of layoffs right now, but nearly every one of the majors is still profitable, just not profitable enough for the stock market. Living inside one of these companies means dealing with layoffs, but also things like:
If a company hasn't had to live through this before the culture doesn't yet bear the scars. The employees certainly do, especially those of us who were in the market in 2007-2011. Many of us left companies to get away from this kind of thing. That said, the last major down period in tech was over ten years ago; there are a lot of people in the market right now who've never seen it get actually bad for a long period of time. "Crypto Winter", the fall of many crypto-currency based companies, was as close as we've gotten as an industry.
When the above trail of suck starts happening in a company that hasn't had to live through it before, it leaves scar tissue. The scars are represented by the old-timers, who remember what upper management is like when the financial headwinds are blowing strong enough. The old salts never regain their trust of upper management, because they know first-hand that they're face eating leopards.
Even if upper management turns the milk and honey taps on, brings out the bread and circuses, and undoes all the list-of-suck from above, the old-timers (which means the people in middle management or senior IC roles) will still remember upper management is capable of much worse. That changes a culture. As I talked about before, trust is never full again, it's always conditional.
So no, it'll never go back to the way it was before. New people may think so, but the old timers will know otherwise.
]]>Unlike the earlier crazes, AI is obviously useful to the layperson. ChatGPT finished what tools like Midjourney started, and made the average person in front of a browser go, "oh, I get it now." That is something Blockchain, Crypto currencies, and Web3 never managed. The older fads were cool to tech nerds and finance people, but not the average 20 year old trying to make ends meet through three gig-economy jobs (except as a get-rich-quick scheme).
Disclaimer: This post is all about the emotional journey of AI-tech, and isn't diving into the ethics. We are in late stage capitalism, ethics is imposed on a technology well after it has been on the market. For a more technical take-down of generative AI, read my post from April titled "Cognitive biases and LLM/AI". ChatGPT-like technologies are exploiting human cognitive biases baked into our very genome.
For those who have avoided it, the art of marketing is all about emotional manipulation. What emotions do your brand colors evoke? What keywords inspire feelings of trust and confidence? The answers to these questions are why every 'security' page on a SaaS product's site has the phrase "bank-like security" on it; because banks evoke feelings of safe stewardship and security. This is relevant to the AI gold rush because before Midjourney and ChatGPT, AI was perceived as "fancy recommendation algorithms" such as those found on Amazon and the old Twitter "for you" timeline; after Midjourney and ChatGPT AI became "the thing that can turn my broken English into fluent English" and was much more interesting.
The perception change caused by Midjourney and ChatGPT is why you see every tech company everywhere trying to slather AI on their offerings. People see AI as useful now, and all these tech companies want to be seen as selling the best useful on the market. If you don't have AI, you're not useful, and companies who are not useful won't grow, and tech companies that aren't growing are bad tech companies. QED, late stage capitalism strikes again.
It's just a fad
Probably not. This phase of the hype cycle is a fad, but we've reached the point where if you have a content database 10% the size of the internet you can algorithmically generate human-seeming text (or audio, or video) without paying a human to do it; this isn't going to change when the hype fades, the tech is here already and will continue to improve so long as it isn't regulated into the grave. This tech is an existential threat to the content-creation business, which includes such fun people as:
The list goes on. The impact here will be similar to how streaming services affected musician and actor income streams: profound.
AI is going to fundamentally change the game for a number of industries. It may be a fad, but for people working in the affected industries this fad is changing the nature of work. I still say AI itself isn't the fad, the fad is all the starry-eyed possibilities people dream of using AI for.
It's a bullshit generator, it's not real
Doesn't matter. AI is right often enough to fit squarely into human cognitive biases of trustworthy. Not all engines are the same, Google Bard and Microsoft Bing have some famous failures here, but this will change over the next two years. AI answers are right often enough, and helpful often enough, that such answers are worth looking into. Again, I refer you to my post from April titled "Cognitive biases and LLM/AI".
Today (May 1, 2023) ChatGPT is the Apple iPhone to Microsoft and Google's feature-phones. Everyone knows what happened when Apple created the smartphone market, and the money doesn't want to be on the not Apple side of that event. You're going to see extreme innovation in this space to try and knock ChatGPT off its perch (first mover is not a guarantee to be the best mover) and the success metric is going to be "doesn't smell like bullshit."
Note: "Doesn't smell like bullshit," not, "is not bullshit". Key, key difference.
Generative AI is based on theft
This sentiment is based on the training sets used for these learning models, and also on a liberal interpretation of copyright fair use. Content creators are beginning to create content under new licenses that specifically exclude use in training-sets. To my knowledge, these licenses have yet to be tested in court.
That said, this complaint about theft is the biggest threat to the AI gold rush. People don't like thieves, and if AI gets a consensus definition of thievery, trust will drop. Companies following an AI at all costs playbook to try and not get left behind will have to pay close attention to user perceptions of thievery. Companies with vast troves of user-generated data that already have a reputation for remixing, such as Facebook and Google, will have an easier time of this because users already expect such behavior from them (even if they disapprove of it). Companies that have high trust for being safe guardians of user created data will have a much harder time unless they're clear from the outset about the role of user created data in training models.
The perception of thievery is the thing most likely to halt the fad-period of AI, not being a bullshit generator.
Any company that ships AI features is losing my business
The fad phase of AI means just about everyone will be doing it, so you're going to have some hard choices to make. The people who can stick to this are the kind of people that are already self-hosting a bunch of things, and are fine with adding a few more. For the rest of us we have harm reduction techniques like using zero-knowledge encryption for whatever service we use for file-sync and email. That said, even the hold-out companies may reach for AI if it looks to have real legs in the marketplace.
Yeah. Like it or not, AI development is going to dominate the next few years of big-tech innovation.
I wrote this because I keep having this conversation with people, and this makes a handy place to point folk at.
]]>This is good for Q&A sites, because not everyone is good at generalizing their solution from 30 other answers similar but not quite like their case. They want a step by step how-to that will get them out of the hole they're in. The most trusted ServerFault answerers (I was among them once) were good at doing this. We didn't throw RTFM at them, or cite section and sub-section of the manual they should have read. We explained why it went bad, how to get out of the hole, and perhaps some next steps. The better ones weren't judgmental in describing the foot-guns in play, but described negative consequences that you might not want to experience.
ChatGPT and related technologies act like higher quality answerers. You ask it "give me a step by step guide to get to where I need to go," and that is what it will produce. A step by step guide, no judgement about any foot-guns you're playing with. So nice. You don't have to put up with the judgmental best practices chorus to do so, and it isn't judgmental every time. You don't have to hope to get a nice answerer the way you do on ServerFault/StackOverflow.
Whether that solution is actually fit for purpose is not guaranteed, but it sure looks authoritative. This is where the cognitive biases come in to play.
Humans looking for help want one thing: someone who knows how to do the thing to help them do the thing. We asses "knows how to do the thing" many ways, but a time tested strategy is buried in the definition of the word mansplaining. To save time, these are the qualities we look for as a species:
These three are extremely possible to fake, and humans have been doing so since humans were invented. The check and balance that Q&A sites like ServerFault and StackOverflow offer is voting on the answers, so actually wrong answers get some signal to validity. It have seen people be confidently wrong time and time again on ServerFault, and get cranky when their lack of actual knowledge gets called out.
Technology like ChatGPT automates producing answers that hit all three points, but lacks the voting aspect and any guarantee of actual expertise. In short, ChatGPT and related technologies automate the exploitation of human "sounds like an expert" heuristics. This is an exploit of the human cognitive system, and this is why you're going to start seeing it everywhere. It have complete faith that some startup somewhere is going to come up with the idea of:
What if we have generative LLM/AI create answers to questions that are then voted on by 'experts' and we sell the answers?
It's coming. May already be here.
]]>"I didn't think the leopard would eat my face," said the woman who voted for the Let Leopards Eat Faces party.
https://knowyourmeme.com/memes/leopards-eating-peoples-faces-party
I'm thinking about this right now with regards to the reductions in force (RIFs) happening in US-based big tech. We don't know for sure why this is happening, but there are several competing (and probably overlapping) theories:
You might notice a theme here, which is what reminded me of the face eating leopard parable. There is a piece of advice I tell people at work that relates to this:
(DayJob) is a publicly traded SaaS company in the United States, forget this at your peril.
This is a nonspecific warning, but I mean it in the sense that we all are still working for a face-eating leopard; just one that's more domesticated than many of the others. When push comes to shove, it'll still eat faces. Yes, the benefits are pretty good and they haven't done a RIF yet; but do not mistake this for a sign that they will not ever perform a RIF. In case you need it specified, the theme in the above list is "reduce the pace of salary inflation and make hiring less expensive over the next year or so," which can be accomplished several ways, one of which is a RIF.
There are other ways to reduce the pace of salary inflation and make hiring cheaper:
All four of these moves are things you do when you have some warning that winds are shifting, or you already know you need to add percentages to your profitability line-items in your quarterly/annual reports. A domesticated leopard will do more of the above slow-shifting before biting faces off through a RIF. That leopard will still bite faces off if the above doesn't move the profitability needle fast enough.
What does this mean for those of us working for face-eating leopards?
First and foremost, it means being defensive with your finances. If you are working for this particular variety of face-eating leopard (publicly traded US tech company compensating in part through stocks) then you are probably in the top 5% of US income. Before I got a job with one of these leopards I didn't understand how friends in the industry could say things like:
I left Company X today. I plan to take a few months off before seriously looking for my next thing.
Who has that kind of money laying around? I sure didn't. Then I started working for one of them and got stock-compensation, then I understood. Those Restricted Stock Units meant quarterly infusions of cash-equivalents I had to do something with, so I saved them. It turns out a lot of us do that too, since the savings-rate of the top 1% of the US income list is over 30%. That gives us a lot of leeway to build up an emergency fund, which we all still need to do. If you're living hand to mouth, which is actually easy when your rent is $4000/mo, then you're at risk of having a really bad time if its your turn to have your face eaten.
Second, if you're living in a high cost metro and are working for a company with a sizable remote workforce, you are at elevated risk of getting your face eaten and having them repost your job somewhere like New Orleans. Being more at risk means you need to be more diligent about making sure your finances can survive 5 months of unemployment. The US technical job market is getting realigned now that the money has figured out you can have a successful business by hiring outside the major metros. More and more, when companies are facing equally qualified candidates in New York City and Cincinnati, they pick the Cinci candidate because they're cheaper.
Third, and much less helpful, work towards changing the US labor market (or relocating to a labor market where employees are treated better by policy) to make face-eating harder to do. A majority of the European Union has laws on the books requiring a notice period for reductions of any kind, same day terminations are rare and shocking. By proxy, a lot of the places colonized by EU members have similar protections. Getting three weeks warning that you will be out of a job means you can say goodbye, work on a transition plan, and otherwise have time to mourn. It still sucks to lose your job, but it'll hurt less.
]]>The main complaints look to be:
Cultural issues
This isn't the first time I've been part of a community that suddenly got swarmed with a whole bunch of new people who only sort-of speak the same cultural language. The Fediverse community responded the same way those older communities responded, by codifying in real time the unwritten social contract they'd been following and presenting it as de-facto policy written in concrete. If you've ever joined a community and got yelled at to READ THE FAQ BEFORE POSTING, you've seen this in action. In my opinion, this tendency drove a lot of the discontent behind content warning discourse.
Yeah, getting yelled at to READ THE FAQ BEFORE POSTING drives off new comers. It always does. It always will.
The arguments regarding lack of awareness of the need of marginalized communities came as part of the content warning discourse. "I shouldn't have to content-warn my lived experience" being the top argument against use of warnings. The second most common argument against is "I shouldn't have to content warn all political posts, because my living in this world is a political act."
The second argument came as a bit of a surprise to me, because I understood the "content-warn for politics" contract to mean, "Fediverse is a global community and not everyone is in the US, so flag your US politics posts so non-US people can skip them." It definitely was not "content warn anything that could be construed as politics." I attribute this misalignment to real time codification of social contracts getting it kind of wrong, and being taken as long standing policy by new folks. Hand in hand with this is a guideline to CW your tech-posts, because not everyone is in the tech industry and would like to skip them.
As for why content warnings are there at all, Christine Lemmer-Webber posted a brief explainer for why we have them and not another system.
Christine advocated for Tumblr-style tags, not a feature that would hide content requiring a click to view. Christine also pointed out that prescriptive rules for the use of CWs are kinda bad, and yeah they are. The CW system is a poorly defined feature that leads to a lot of confusion. The argument to "treat them like LiveJournal/DreamWidth cut-tags" is entirely accurate, even if most people these days don't know what that means.
The reply-guy comment is somewhat inherent to the subcultures that tried to move to Mastodon. The instance I'm on, octodon.social, has one of the biggest open-source software instances on silence (fosstodon.org) because the maintainer got tired of all the reply-guys coming from there. So when I see people piling into the new free software focused hachyderm.io and start complaining of all the reply-guys, I'm seeing a trend. A trend that isn't exclusive to Mastodon, in fact. Reply-guys are endemic in tech social spaces and it takes effort to redirect this reflex. Effort and assistive technologies like limiting replies to posts to the explicitly mentioned, or who are already followers of the poster.
Technical issues
Mastodon was designed for hobbyists to spin up instances and run on their own, and definitely not designed to be a scalable architecture for a kajillion users on the same instance. It can get that dense, as mastodon.social and mastodon.online prove, but it takes skilled engineers to keep it going.
Second, Mastodon is fully open source without a corporate backer trying to make money on top of it. This means there isn't a security department, just a community of security minded volunteers who try to get the project to adopt new techniques. Sometimes they make it in. Sometimes the maintainer doesn't want to merge the commits for whatever reason. This is a volunteer project.
Both of these mean that top tier corporate network security techniques like WebAuthN aren't there. They may be in future versions, but they're not there now. This also means there isn't a unified identity across instances, because it was intentionally written to be a federated system with islands of identity.
As for quote tweeting, the maintainer has mentioned many times that he has no intention of adding them, and considers them actively harmful. He argues that quote tweets are used by griefer armies to direct hostility at victims, and quote tweets promote one way discourses rather than conversations. Because the maintainer doesn't want them, he won't accept merges that add them. Simple as that.
Quote-toots are indeed a tool with many uses, some good, some bad. Some of the twitter migrants want them back because they were using them to create conversations about things, and found value. Quote-tooting preserves attribution in a way that in-line quoting simply can't. Quote-tooting also gives the original poster feedback that their messages are circulating in new, potentially hostile communities, and to take actions.
Mastodon is a different community. The nature of federation means that the Fediverse has more structural separation between some communities than Twitter allows, which comes as a frustration to some and a blessing to others. Your view of the Fediverse is filtered by the instance you're on, and two instances will see different things. This is unsettling to many used to centralized social media systems.
I like it, but I also know it fills a somewhat different niche than Twitter or Tumblr do. Nor is it a true like-for-like replacement for either.
]]>Build your core competencies, buy everything else. This allows you to focus your creative efforts on what you do best.
But there is a world of nuance in this seemingly simple statement. At the far end of one side of the argument you have Google: build everything, because no one else operates at our scale. At the far other end you have new startups: make other people do everything except our core business logic, because that gives us the speed to establish ourself in the market.
Lets take a look at the radically different incentives faced by Google and a generic new startup:
What resources do I have to build a new solution?
Google: We're one of the biggest software engineering companies in the world, so have a lot of engineering talent
Startup: We're five people and a big idea. Every engineer is precious.
What resources do I have to buy a new solution?
Google: So much money, but the third parties who can work at our scale are all direct competitors in different ways.
Startup: I have VC money, and may have Board members from SaaS companies who will give us discounts.
What solutions are available on the market for a company my size?
Google: Major foundations like the Cloud Native Computing Foundation and the Apache Foundation may have platforms from companies near our size we can adapt.
Startup: A whole universe of SaaS providers focused on solving my every need.
If I need to buy something from a new vendor, how hard is it?
Google: All new vendors need to go through a Vendor Security Assessment, and once that's complete, Legal needs to ensure certain standard contract clauses are in place before we can start cutting purchase orders. Maybe two quarters before it can be used at all, then another two quarters to production-scale it.
Startup: The CTO has a company credit-card for a reason. Two days to buy, maybe three sprints to get it into production.
If we need to use a new OSS platform, how hard is it?
Google: Throw 30 engineers at it to solve the scaling problems and ship code upstream. Proof of Concept in one quarter, another two to solve the production-scaling issues.
Startup: Buy our cloud vendor's managed version if at all possible, or something off their marketplace if not; one engineer works to build a POC. Maybe three sprints to production.
You begin to see how different the incentives are between one of the biggest software engineering companies on the planet and the new kids. The new kids are all buy buy buy for a lot of good reasons, where Google is build build build for equally valid reasons.
Looking at this from an order of magnitude point of view, a Startup has single to double-digits worth of engineers (1-99). After a few VC rounds they may get into the triple digits (100-999 engineers). A Google has five to six digits of engineers, a radically different company in so many ways. Each engineer at the startup is represented by at least one whole team at Google, more likely a group of teams, and sometimes an entire department.
But what about the middle ground, the companies with four-figures of engineers? Where do they fit on the build/buy continuum?
It turns out that's really freaking hard. A four-digit-engineers company likely has the same barriers to onboarding new vendors Google had above (have to VSA vendors, Legal needs special things in the contract). At the same time, they have a much shallower engineering pool to draw from to build things instead. This is the most awkward spot to be.
It turns out these 1000 - 9999 engineer companies are where another dynamic emerges to push the decision needle towards build before it's a really great idea to do that: the speed of adding headcount versus the speed of paying a vendor.
Adding headcount is a fully pipelined service in a company this size. They're hiring engineers all the time, doing it in multiple legal jurisdictions, and have internal assets dedicated to making this process as frictionless as possible. Engineers are classified on a scale from 1 to 7 (or 8, or 9) which changes how they can be used. If you need to throw engineers at a problem, and have the open requisitions for it, you can have butts in seats in less than three months. Engineers are (an expensive) commodity.
Contrast this to adding a new vendor. Before any contract can be signed the Security group needs to do a Vendor Security Assessment of the vendor to ensure they're up to your company's standards with regards to safe handling of data and a variety of other things. This VSA process can take months and is prone to backlogs if the VSA processors are themselves clogged with work. Once the VSA is done, negotiating the contract comes next. Many companies try to add clauses to contracts to clarify ambiguities or to push for different treatment, which all means legal back-n-forth that adds time. Privacy teams need to asses the vendor for what types of private data the vendor is allowed to touch, which is its own process. Then the contract gets signed. This can take up to a year end to end. Each vendor is a unique snowflake.
But what if we don't have open reqs for headcount?
Even then, there are incentives to build. In this case, you're looking at moving engineering priorities around to make room for the development work on the new thing. That's the nice thing about headcount: you can use it for more than what you initially bought it for, something a vendor solution rarely provides.
Never forget accounting, though. Costs of employees are often tracked rather differently than costs of suppliers, which definitely does mask the full cost of ownership of a homebrew solution versus a vendor-provided solution. Most costing analysis considers engineering capacity to be "affects future roadmap", where vendor solutions come with direct costs measured in currency and profit/loss statements. If you want to be serious about like-to-like comparisons of a homebrew vs vendor solution, factor in the salary/bonus costs of the people in charge of building and then later maintaining the solutions.
If you're in one of these mid-size, four-figures-of-engineers companies, there are a few things you can do to try and de-skew the incentives at play and get closer to the business focused reasons for a build or a buy decision: