codebojecodeboje - Guiding developers since 1869 - Guaranteed Bullshit-Free2024-01-04T17:03:34+01:00https://codeboje.de//How to Get a Full-Text Search with FaunaDBhttps://codeboje.de/full-text-search-faunadb/2020-10-12T10:40:39+02:002020-10-12T10:40:39+02:00<p>I’ve been using FaunaDB for a few projects now, as it is the only direct database integration for Netlify functions.</p>
<p>Last time I needed a simple wildcard search. In SQL, this is a simple <code>where FIELD like "%term%"</code>. Easily combined with any other filtering or ordering.</p>
<p>Turned out that it is not straightforward in FaunaDB.</p>
<p>The usual filtering is done via indexes and a MATCH query. Unfortunately, MATCH only supports EXACT matches, no wildcards. And sadly, no further explanation at that place in the docs on how to achieve this.</p>
<p>Reading the docs, googling, and stackoverflowing finally brought me to the “About full-text search” section in <a href="https://fauna.com/blog/getting-started-with-fql-faunadbs-native-query-language-part-2">this getting started guide</a>.</p>
<p>The strategy is to put a Filter around your index. An index in Fauna is more like a search query in SQL and not an SQL index.</p>
<p>When you define a simple index like in their example:</p>
<div class="codehilite"><pre><span></span><code>CreateIndex({
name: "all_Planets",
source: Collection("Planets")
})
</code></pre></div>
<p>it’s merely a <code>select * from Planets</code> in SQL.</p>
<p>Based on that, the proposed way to get a full-text search in the article was that:</p>
<div class="codehilite"><pre><span></span><code>Map(
Filter(
Paginate(Match(Index("all_Planets"))),
Lambda("planetRef",
ContainsStr(
LowerCase(Select(["data","name"],Get(Var("planetRef")))),
"ur"
)
)
),
Lambda("planetRef", Get(Var("planetRef")))
)
</code></pre></div>
<p>Let’s break that down.</p>
<ol>
<li>We get all records from the collection <em>Index</em> and <em>Match</em></li>
<li>We put pagination around that with <em>Paginate</em></li>
<li>The Result of that is filtered with <em>Filter</em> and giving that a single <em>Lambda</em></li>
<li>This Lambda
a. Get the item ref as a parameter “planetRef”
b. Retrieves the value behind it (the actual ref object with <em>Var</em>)
c. Get the document object for that <em>Get</em>
d. Gets the field value “data.name” of that document <em>Select</em>
e. Makes that value to lowercase <em>LowerCase</em>
f. Checks if this value contains our search term “ur” with <em>ContainsStr</em></li>
<li>And finally maps all result item refs again to the actual document <em>Map</em> and the <em>Lambda</em> with <em>Get</em></li>
</ol>
<p>If you are coming from SQL like me, this looks like many queries hitting the DB. And as far as I understood it from various questions, they all count as a “reads” in your account.</p>
<p>Now, my use case was a bit different.</p>
<p>My index already contained a search term and also defined the fields returning. That changes the return type of the index to an array– one entry for each field.</p>
<p>A simplified version looks like:</p>
<div class="codehilite"><pre><span></span><code>CreateIndex(
{
name: "all_feedforms_by_user_desc",
unique: false,
serialized: true,
source: Collection("feedforms"),
terms: [
{
field: ["data", "user_id"]
}
],
values: [
{
field: ["data", "creation_date"], reverse: true
},
{
field: ["data", "user_id"]
},
{
field: ["data", "formName"]
},
{
field: ["ref"]
},
}
</code></pre></div>
<p>I only want those records which match a given userId and some fields that I show in an overview in my app. Additionally, I sort the records by descending creation time.</p>
<p>Using my index and adapting the code above resulted in errors because my variable for the inner <em>Lambda</em> was an array. And I found absolutely no way to access a single value in it.</p>
<p>The <em>Lambda</em> does accept only a single parameter. However, this parameter can also be an array of parameter names. Then it will map the resulting array from the index with the array of parameter names you define for the <em>Lambda</em>. Thus, you can access each array index as a parameter.</p>
<p>Now I could access the single field value I am interested in and remove all those <em>Get</em> document loads.</p>
<div class="codehilite"><pre><span></span><code>Filter(
Paginate(Match(Index('all_feedforms_by_user_desc'), "myuserid" )),
Lambda(
['date_created', 'user', "formName", "refid"],
ContainsStr(
LowerCase(Var("formName")),
searchTerm
)
)
)
</code></pre></div>
<ol>
<li>I get all records for a given userId from the <em>Index</em> with <em>Match</em></li>
<li>Paginate that and then filter it</li>
<li>The filter <em>Lambda</em> gets an array of parameter names matching exactly the return array of the index.</li>
<li>Checking if the value behind <em>Lambda</em> parameter “formName” contains the “searchTerm”</li>
</ol>
<p>The result, a simple wildcard search without the need to get the document multiple times.</p>3 Coding Projects for Learning or Spicing Up Your Resumehttps://codeboje.de/3-coding-projects-for-resume/2020-04-28T17:32:05+02:002020-04-28T17:32:05+02:00<p>Coding projects are an essential step in learning. I am firm believer then best way to learn something is by doing. So, for developers that means coding, coding, coding. And of course designing :-)</p>
<p>Today I’ll share with you three great projects you can code along and learn. Or build and polish them for your portfolio or resume. Nothing beats a finished real project.</p>
<h2 id="1-a-kanban-api-with-a-ui-as-a-bonus">1. A Kanban API with a UI as a Bonus</h2>
<p>Building something around Kanban is great for learning new stuff because it is a bit more complex then a traditional todo app, yet still small enough for not being overwhelming. By tuning that a bit with an API, we also can practice building APIs, which involves more than a fest classes and endpoints. </p>
<p>I run a series on that 2 years ago, which is a great starting point for you. Check it out here.</p>
<p><strong>For a resume</strong>: It shows that you know Kanban, have invested time into it. Shows that you can build an API including AuthN + AuthZ, etc. </p>
<h2 id="2-a-frontend-for-a-spreadsheet">2. A Frontend for a Spreadsheet</h2>
<p>Might sound silly but spreadsheet drive many businesses. At one point in your life you will have to start parsing them and import data. Sometimes, it’s a one time shot, sometimes, you develop whole importer which feed on spreadsheets.</p>
<p>Pick a simple spreadsheet and start building a front end web app for it. Or if you don’t have one handy, use my tutorial on how to use Google Sheets as a database. </p>
<p><strong>For a resume</strong>: Shows that you can deal with spreadsheet data and the challenges involved aka messy data.</p>
<h2 id="3-add-payments-to-one-of-the-other-projects">3. Add Payments to One of the Other Projects</h2>
<p>Money is everywhere and with it payments, payment providers, payment APiI and all the hassles involved with payment. Pick Stripe, integrate it into one of your coding projects and learn all the side issues that can happen with payments like declined cards, payment flows, involved regulations, etc.</p>
<p><strong>For resume</strong>: Shows that you understand and can handle the issues payment brings. There might be also an opportunity as an integrator for specific platforms. Yes, Stripe looks easy and it is to get started but most implementations are for good weather conditions only. Let the issues happen and it might fall apart. And now imagine a business that cannot accept payments or has too much hassle with it…</p>
<p>Of course, if you are in the Spring world or want to enter it, implement them in Spring. Otherwise, use them with a stack choice you want to improve in. And feel free to reply if I should go in-depth with something.</p>Be careful with JWT and third-party auth serviceshttps://codeboje.de/careful-jwt-auth/2020-04-17T10:37:32+02:002020-04-17T10:37:32+02:00<p>JWTs are great if you use some precautions. Like, don’t put sensitive stuff in it. But it surprisingly happens often. It is a recurring theme in my consultations that devs forget that signed and encoded does not mean encrypted. And so user details end up in the token that should not be there like personal data, billing URLs and hosts.</p>
<p>JSON Web Tokens are only signed and encoded. That means anyone obtaining the token can decode it with easy and see what you stored in it. The only secure thing is that they can not tamper with it as the token is signed.</p>
<p>Never forget that and be spares with the stuff you put in a token.</p>
<p>Sometimes this can happen by accident too. For example, when using an auth lib or service you aren’t familiar. Like it happened within a side project of mine using the Netlify Identity service. It’s almost of out the box auth service with a small Javascript frontend and a service running in your deployment instance. It uses JWTs and works with their functions aka lambdas. Pretty smoothly.</p>
<p>Their API offers two fields to add app and user-specific data to the user object. Nicely explained in the Javascript library how you can use them. However, what they missed to mention is that whatever you put in there is also put into the token. The whole user object is put in there.</p>
<p>That is convenient as you have everything in the token that your backend might need. No extra request for retrieving user details. It’s tempting to put more in there. But it can also create holes when you are not aware of that.</p>
<p>I wasn’t until some time later it strikes me that I should check what they actually put into that token. And voila, everything was there. Doh!</p>
<p>So, be careful and double-check what you put into tokens and what your auth service actually does :-). </p>Is cutting costs expensive?https://codeboje.de/cutting-costs-expensive/2020-02-27T10:35:39+01:002020-02-27T10:35:39+01:00<p>This morning I read an article on “<a href="https://medium.com/javascript-scene/why-cutting-costs-is-expensive-how-9-hour-software-engineers-cost-boeing-billions-b76dbe571957?__s=xxxxxxx">Why cutting costs is expensive</a>” and share it with you.</p>
<p>It’s an interesting read. There has been a time where I would have totally agreed with his opinion. Especially regarding deadlines and the “it’s done when I am done”- attitude. However, when you move to the other side and try to run a company, your point of view might change. As mine did. Sure, in case of a life-critical system I’d test the hell out of those things. But from an ethical standpoint and not necessarily an economic one.</p>
<p>What I learned is, that you need to take risks sometimes otherwise there won’t be business. Without business, there’s no need for software. No need for software, no need for devs. And yes, sometimes these situations have a fixed deadline, which can’t be moved. And you either make it with something and can continue. Then you can fix things up. Or it fails and thus no business with this project. </p>
<p>The catch is, nobody knows if a project will succeed in the business world. Even replacing simple Excel sheets with an app comes with many risks. And nobody knows BEFOREHAND if it was worth the effort in the end at all. It’s all about “will it help us to make more money than it costs us” and the risks tolerance of the participants, company owners or managers or whatever.</p>
<p>This can work like with successful startups for example or backfire like Boing according to the article. Usually, there’s a lot more involved than the code quality. Code quality is just one risk you have to manage. Depending on the type of project it might be a big one or is irrelevant.</p>
<p>The same is for cutting costs. It’s a bet on how much can we lower our running costs without hurting our profit. And about how much risk is the business willing to take. And what can it gain for that risk?</p>
<p>However, most of us do not see that because we are stuck in our craft and love what we do and want to deliver the best epic result ever. The craft rules and we want to craft well. Business? Nah, it’s not my job, I am the crafting guy.</p>
<p>Learn to understand the business side of things and it will make your life easier and less frustrated. The best thing to learn is starting your own side-project with the goal of making money. Yes, making money, not learning new stuff. Only with that, you’ll experience how taking risks pays off or not. And how “best practices” might help you get better at your craft but might not help to make money at all. Especially, with a hard constraint of your time. It will open your eyes and you will see things differently…</p>The bane of tech stack decisionshttps://codeboje.de/bane-tech-stacks-decisions/2020-02-11T10:34:06+01:002020-02-11T10:34:06+01:00<p>I’ve written about this before but as I run into this topic again last week, I thought it’s a great reminder for all of us. Don’t have time to read? There’s a TLDR at the end of the email.</p>
<p>I wanted to add some features to my <a href="https://zenmix.io">ambient sound generator app Zenmix.io</a>. So, the first decision I had to make was if I wanted to stay with the current tech stack or move to a new one. And which one. So, let’s start there.</p>
<p>The original app was using JQuery and a JQuery plugin for the audio part. Both versions were pretty outdated. The plugin’s last release was somewhere in 2014. Sure, it does its thing, so why change it?</p>
<p>The answer is JQuery. Or more concrete the experiences I made in many projects with it and that is was always a hassle to maintain these. Besides, there arose better ways for building apps like vue. I uses vue in POV project last year and was pretty satisfied with it. However, I am not keen on those JS build systems, so I also thought about using plain JS for the front end. But then I had to write everything myself. hmm. Finally, I settled for vue with the big assumption I will be faster. We’ll see if it was a good assumption farther down.</p>
<p>The next thing was I needed a backend now for the new features. My first thought was Spring Boot and probably I’d coded the damn backend including auth in a few hours. But on the other side it meant, I had to host it somewhere else than on Netlify. Which works as I have done this in other projects but it is a moving part more that can break and also probably an expense more. I’d didn’t like that at the point of making the decision.</p>
<p>Another point was, that Netlify offered its Identity service for user and auth stuff with a JS lib and on their backend. Yay. And of course, they got functions, so I could put everything on Netlify and have a single hosting point. As they also solved the data storage probs in the form of a FaunaDB integration, the decision was made. I went with the Netlify stack for auth and the full backend instead of Spring Boot.</p>
<p>The question now is: Was it a good decision?</p>
<p>And that is the hard part because you never know which alternative would have been better. You don’t know in advance and you can’t do it in both stacks without influence. We would have to start a parallel world and see which ends up happier :-)</p>
<p>Anyway, now let’s get back to the reason for my mail. What I didn’t mention so far is, that I had a time constraint. Wanted to have thing done in 2 days. lol</p>
<p>But as it is with new tech stacks and time constraints, they do not work well together. Which is normal because you do not know which issues arise with the new tech, nor how you can deal with it. And that was the main reason why it went way over my 2 days estimations. Actually I needed 6 days to complete it. Not everything of the delay was due tech issues. I underestimated UI for auth and user mgmt flows too. Doh!</p>
<p>It’s fascinating how easy it is to fall into those probs even when you have been coding for a long time. Especially easy, when it’s your own project.</p>
<p>What could I have done better in the beginning?</p>
<p>In my case clearly adjusting my plan and my expectations. The tight timeframe was the culprit for any stress. Yeah, the stack had had its frustrations too but the main prob were my expectations. So, either removing the deadline or keep it and use Spring.</p>
<p>However, more important is to keep those things in mind when planning. Especially, when working on large projects and with a team. Not every team member has equal knowledge or skill. We are always learning but it seldom is part of any project planning. More often than not those arbitrary 20% buffers a PM adds are used :-)</p>
<p>TLDR;</p>
<ol>
<li>When the project is time-critical, do NOT add new techs, languages, framework, etc</li>
<li>When you add new tech, make sure it is reflected in your project plan as your estimation will ALWAYS be off by a larger magnitude.</li>
</ol>