Yesterday morning, I priced out a solution for doing backups of the repositories in our GitHub organization. It was the classic cable TV problem: It was expensive, and the only way to get the features I wanted was to upgrade to a plan that would have me paying for all kinds of stuff I don’t need just to get the one thing I do. Yesterday afternoon, I started testing a homegrown solution that Claude helped me build, complete with a docker image, python code, terraform code to deploy it all, a README suitable for anyone either using or managing the service, etc.

It actually is already a new world. It’s not just coming, it’s here, and companies need to start thinking hard not only about what their customers want and need, but what they’re competing with. I’m not sure it’s possible to understand what you’re competing with if you don’t have an internal capability around AI. Just six months ago it might’ve still been true that the bar for creating “good enough” solutions internally was too high for a lot of customers. The list of products for which that’s still true is shrinking quite rapidly!

We do the same thing in the technology space. In the 80s, there were folks talking about computers taking all of the jobs. In the 90s, there were folks talking about the internet taking all of the jobs. In the 2010s people talked about AWS taking all of the jobs. Today we’re talking about AI taking all of the jobs. Still the world turns.

Whether kids brains were ever turned to mush is, perhaps, debatable. But regarding various technologies taking all of the jobs, we have data on that, and it’s provably false. Well, at least it’s false from a raw employment perspective. It’s absolutely true that certain roles ceased to exist, but the people in those roles migrated to other, possibly brand new roles.

That’s not to say there weren’t other impacts on peoples’ lives & careers, though: for those who saw computers and decided not to learn about them and how they could be adopted in their work, they eventually became seen as complacent, or in the way, or behind the times, etc., and those folks (and I’m speaking from my own observations of family members way back in the day) had trouble finding jobs, eventually, and when they did find work, it was with a company that was still doing everything manually, because (for example) the owner found it cheaper to hire people than migrate to computers because computers made filing clerks in very low demand. Sounds like a great place.

Something similar happened when AWS adoption was building steam back in 2008-2010. There were only a few services back then. The concept of a VPC was brand new, for example, as I recall. I thought it was great and was getting real work done with these new tools, where I could write code to deploy an EC2 instance, and store files in S3, etc. At that time I was a member of a lot of system administration groups and Linux user groups, and there was a meeting where someone took an informal poll that showed that less than half of the folks in the room wrote code on a regular basis.

That really shook me. I was friends with almost everyone in the room. I also could plainly see that AWS was getting bigger faster – not slowing down, or plateauing. Within the next meeting or two I put together a talk that more or less begged everyone in attendance to learn to code in whatever language fit their brain. To take their shell scripts and port them to Python, or Perl, or Ruby, or anything. Any language would do. “The reality is”, I said, “that Amazon is creating APIs to allow developers to do your job.” A couple of smirks, a couple of knowing nods of agreements – from coders. “It’s not that there won’t be any jobs” I continued, “It’s that the jobs that are left are going to be the ones we all hate doing now, like changing printer toner.”

It has been nearly 20 years since I gave that talk. LinkedIn is a thing. There were probably 30 people in attendance at that talk, all locals, and the people who smirked are verifiably either not working in technology at all, or are working in tiny 1-man shops that require a physical presence and require them to change printer toner.

So, I tell you all of that so I can tell you this: I don’t think AI is anything to panic about. There are realities, though, that history tells us are likely coming, though. What history also tells us is that the folks who wind up landing on their feet are those who learn to adopt the new technology. The ones who wind up having a difficult time staying afloat are those who are complacent in their roles, think that AI could never replace them, or who try to foster anti-AI cultures, or the like. And, like computers, and the internet, and AWS, it’s not that there won’t be any jobs. It’s just that you might not want the jobs left in the wake of AI.

Not only was this tool instrumental in getting the data migrated, but it also allowed me to very easily craft a way to quickly compare numbers between the source and destination to make sure everything I expected to be migrated was actually migrated.

The Mess I Made

It’s important to know that engineers with 25+ years of experience do dumb things sometimes. It’s also important to know that having decades of experience does not make you immune to fatigue, and really nobody should be working from 7AM until midnight. It’s not heroic. It’s unhealthy and problematic on a bunch of different levels. With that in mind, here’s the dopy stuff I did after working too late after too many hours:

Initially, after I migrated a subset of the data, I did a more manual, hacky check for consistency just using command line tools, manually querying each source in different terminal windows & eyeballing the output. I wanted a nicer way to view that data, so I created a one-liner using awk, paste, and column commands. That looked like this:

paste count_clicks.txt count_clicks_parquet.txt | column -t | awk '
NR==1{
  printf(
    "%12s %18s %18s %12s %12s \n", 
    "date", 
    "clickhouse_count", 
    "warehouse_count", 
    "diff", 
    "pct-diff"
  )
}
{
  printf(
    "%12s %18d %18d %12d %12.4f%%\n", 
    $1, 
    $2, 
    $4, 
    $4-$2, 
    (100-($2/$4)*100)
  )
}'

A quick overview of what’s happening there:

• NR==1 means the first record (NR==2 would mean the second record, etc). If the record number is one, awk will output what’s in the first set of curly braces. The printf function takes a format spec in the first argument. My format spec lays out 5 columns of either 12 or 18 characters. All of those columns will hold strings, hence the ‘s’ in %12s. Then I have a bunch of hard-coded strings, which become the column headers. If you forget to put NR==1 in there, they’ll print on every row. Ask me how I know!
• The second set of brackets specs out what will be printed in the rest of the rows. In this case, I have column widths that match up with those of the column headers, and then I have the columns in the output of the earlier parts of the command pipeline:
  • $1 is the date column
  • $2 is the count from the S3 data source
  • $4 is the count from the ClickHouse data source
  • $4-$2 shows the difference between the two data sources, and
  • The last column shows the percent difference between the two sources

The output looks something like this:

        date   clickhouse_count    warehouse_count         diff     pct-diff
  2023-12-01               1471               2445           74        0.017%
  2023-12-02               1665               1700           35        0.038%
  2023-12-03               4496               4537           41        0.045%
  2023-12-04               1650               1705           55        0.047%
  2023-12-05               1154               1237           83        0.069%
  2023-12-06               2777               2865           88        0.074%
  2023-12-07               9244               9293           49        0.041%

The data here is made up to give an idea of what the output looks like.

So, I had this issue where my data audit showed a mismatch. I did a little work, very late at night, and went to bed thinking I had straightened it all out. When I woke up the next day, my well-rested brain and eyes caught a problem: I copied output from my queries of the two sources into two separate files, and mis-labeled the data, and then compared data in a completely different window with that, and…. well, it was a mess, and I didn’t fix anything.

I was up too late working for sure, but I also had a messy process. I should’ve and could’ve done better.

Fresh Eyes, Fresh (and better) Ideas

Revisiting my work from the night before was painful. As soon as I looked at my process at a high level (by scrolling through my terminal window history) I almost immediately said “this is insane. ClickHouse Local should be able to query both sources. I shouldn’t need to copy/paste and introduce levels of indirection that leave room for errors like this.”

I was right. Using ClickHouse Local, you can query a ClickHouse Cloud instance using the remoteSecure function, and query the S3 data using the s3 function, which also lets me pass in the file format as an argument. So there’s support for my data sources and formats.

On top of that, ClickHouse supports Common Table Expressions (CTEs), so I can craft a query where I name the output from two separate sub select statements (one to each data source), and then write a third SELECT that references the two named result sets as if they were tables.

Below, s3_count and ch_count are named result sets. The last SELECT queries those two named result sets.

clickhouse local --query "
WITH s3_count AS (
  SELECT 
    toDate(time) AS day, 
    count() AS num_events 
  FROM s3('https://s3-endpoint/2023/12/**/*.parquet', 'Parquet')  
  GROUP BY day 
  ORDER BY day
), 
ch_count AS (
  SELECT 
    toDate(time) AS day, 
    count() AS num_events 
  FROM remoteSecure('clickhouse-instance-hostname:9440', 'db.tablename', 'clickhouse-user', 'clickhouse-password') 
  WHERE toYYYYMM(timestamp) = '202312' 
  GROUP BY day ORDER BY day
) 
SELECT 
  s3.day AS date, 
  s3.num_events AS s3_count,  
  ch.num_events AS ch_count, 
  ch_count - s3_count AS diff  
FROM s3_count AS s3 
LEFT JOIN ch_count AS ch 
ON s3.day = ch.day;"

The output has a couple of quirks: for some reason, I guess possibly related to querying multiple sources, or using CTEs maybe, the output columns are not ordered according to my query, and there is no header line to tell me which column is which. It looks like this:

2023-02-01	40	2778	2738
2023-02-02	43	4413	4370
2023-02-03	26	7024	6998
2023-02-04	54	3079	3025

Conclusion

So, this is not something I’d paste as-is into a slide deck and present to an executive team. However, clickhouse local in this case did give me a (relatively) quick way to verify the consistency (and quantify the inconsistency) between my data sources. Once the data was moved, I wasted probably an hour with awk and friends, but was able to recover the next morning and throw together the clickhouse local solution in maybe just another hour of reading docs, debugging the query, and running test queries. Hope this helps.

So, while I’d love to have time to do some longer-form writing about ClickHouse, what I have now in terms of notes could be helpful to probably a lot of people as ClickHouse gains in popularity by the minute.

If there are specific topics within the realm of ClickHouse that you want to see covered, you can certainly let me know, but you should know that, operationally, ClickHouse was complex enough that we just this week opened a ClickHouse Cloud account, so my knowledge of it operationally will probably start to atrophy starting with the 24.x releases (up to this week, I built and ran various testing clusters myself, in AWS EC2, and EKS).

So, without further ado, here’s the Cheat Sheet.

NUMBER_OF_COLUMNS_DOESNT_MATCH

If you get seemingly inexplicable ‘NUMBER_OF_COLUMNS_DOESNT_MATCH’ errors whenever you’re doing a select (by itself or as part of an INSERT…SELECT, or whatever), and it seems like the column numbers do match, remove the parentheses around whatever follows SELECT. So, instead of SELECT (x, y, z) FROM foo it should be SELECT x,y,z FROM foo.

Remote and RemoteSecure Table Functions

If you have an ClickHouse server you manage, or an instance in their cloud offering, you can use clickhouse local and the remote table function to move data into your server from some other source.

For example, I have a cloud instance. I also have an event warehouse in S3, where events are stored in Parquet files. I can use the following command to do an INSERT...SELECT into my (remote) cloud database from the data in the Parquet files in S3:

clickhouse local --verbose --query "INSERT INTO TABLE FUNCTION remoteSecure('my-cloud-host:9440', 'mydb.mytable', 'myclouduser', 'mycloudpassword') (col1, col2, col3) SELECT col1, col2, col3 FROM s3('https://s3-endpoint/events/some-event/2023/**/*.parquet', 'Parquet');"

As a bonus trick, note the wildcard use in the s3 function. The supported wildcards are similar to those used in the bash shell (and maybe zsh – I’m not as familiar). Read all about those here.

Also, note that remoteSecure is just a secure version of the remote function! The syntax is the same! On cloud instances I believe only remoteSecure is supported, which makes sense.

ClickHouse Can’t Parse Some Date Formats

I’ve had issues with:

• Dates with a timezone offset going into a DateTime column, and
• Dates with microsecond precision going into a DateTime column

There are multiple solutions to this, but one I’ve used that could work depending on your requirements is to wrap the column with the parseDateTimeBestEffortOrZero function, like this:

INSERT INTO events (source, timestamp) SELECT source, parseDateTimeBestEffortOrZero(timestamp) FROM mytable

By default, ClickHouse has a fast, cheap, and simplified time parsing algorithm. The ‘BestEffort’ functions support a wide array of formats, including ISO8601 and RFC 822, and more. See here.

Here’s how it deals with a couple of formats I had to deal with:

SELECT parseDateTimeBestEffortOrZero('2011-11-04 00:05:23.283+00:00') AS time

┌────────────────time─┐
│ 2011-11-04 00:05:23 │
└─────────────────────┘

SELECT parseDateTimeBestEffortOrZero('2023-10-26T10:11:18.964768') AS time

┌────────────────time─┐
│ 2023-10-26 10:11:18 │
└─────────────────────┘

In my case, I didn’t require the additional information lost by using the function, so it worked fine. If you need to preserve it, there are more options than I can cover here, but look at DateTime64 datatype for storing subsecond precision, and look at all of the other datetime parsing functions available on this page!

My ClickHouse Server Is Out of Disk!

AFAIK this will only apply to self-hosted instances. If your server’s disk is full, you have a couple of options:

• EBS volumes can be extended & filesystem resized without data loss or downtime (source)
• A new, bigger disk can be attached to the EC2 instance. Then, add the old and new disks to a new storage policy in the clickhouse server’s config.xml file. Then, move all of the data to the new disk. Once done, the old disk can be removed from the storage policy. (source)

Making Wide Tables Without Using NULLable Columns

NULLable columns can impact performance in clickhouse and their best practices docs explicitly recommend avoiding them. Instead, you can create default values. This is how I was able to combine what were separate tables for each event type in our system into a single ‘events’ table, even though these events don’t all follow the exact same schema. Here’s an example excerpted from a CREATE TABLE statement:

    `headers` Map(String, String) DEFAULT map('00000', '00000'),
    `client_id` UUID DEFAULT '00000000-0000-0000-0000-000000000000',
    `destination` String DEFAULT '-',

The Infamous “TOO MANY PARTS” Error

If you get a “TOO MANY PARTS” error either in the server log or in a client when doing an insert query to the server, run this query. It’ll tell you, for each table, how many parts were created, and the average rows per part, over the last 10 minutes.

SELECT  toStartOfTenMinutes(event_time) AS time,
  concat(database, '.', table) as table,
  count() AS new_parts,
  rount(avg(rows)) AS avg_rows_per_part 
FROM system.part_log
WHERE (event_date >= today()) AND (event_type = 'NewPart')
GROUP BY time, table
ORDER BY time ASC, table ASC;

If you see thousands of parts created but the average row size is like, say, 4, then that’s unhealthy. Remember that ClickHouse is optimized to perform fewer, bigger inserts, and performs poorly when doing a high volume of very small inserts, because all of the inserts create parts that have to be merged, and merges are resource intensive. The fewer inserts, the fewer parts, the fewer merges, the happier the server is.

Moving Internal Data Off The Root Volume

ClickHouse stores internal log tables, tables from the system database, and metadata, all under /var/lib/clickhouse by default. Also by default, Amazon Linux puts /var/ on the root volume. I wanted it off of the root volume, and thought it made sense to put it on the volume where all of the other data was stored.

This is possible, but it’s not a documented process and it took me a little debugging to get it done. Here’s what I did:

First, In config.xml (or wherever you keep the storage config), at the top level, set the following settings to whatever path you want (I used /data/lib/clickhouse):

<path>/data/lib/clickhouse/</path>
<tmp_path>/data/lib/clickhouse/tmp/</tmp_path>
<user_files_path>/data/lib/clickhouse/user_files/</user_files_path>
<format_schema_path>/data/lib/clickhouse/format_schemas/</format_schema_path>

<user_directories>
    <local_directory>
        <path>/data/lib/clickhouse/access/</path>
    </local_directory>
</user_directories>

Also in config.xml, make sure there isn’t a metadata path associated with an existing disk in your <storage_configuration>. It would likely be pointing at /var/lib by default.

At this point I copied everything to the new location: cp -R /var/lib/clickhouse /data/lib/.

In addition, you need to make 100% sure that the clickhouse user has ownership and full permissions over everything it had access to before that you’ve now moved. I did this: cd /data/lib; chown -R clickhouse:clickhouse clickhouse and that worked fine for me.

ClickHouse Silently Fails To Start

There have been plenty of times while learning about ClickHouse that it failed to start, but it was always very good about putting an error in the error log (by default, /var/log/clickhouse-server/clickhouse-server.err.log). One time, it failed to start, and there was no error in the log, and nothing to see using journalctl or systemctl status either.

If this happens, you should go find the command systemd (or your init script) uses to start it, copy it, and run it verbatim yourself, and you’ll likely see the missing error. In my case, ClickHouse failed to start because it was unable to access its pid file. That’s weird, because ClickHouse creates the pid file itself.

After some spelunking around I figured out that there was, in fact, a permissions issue (it’s always permissions or DNS folks!). What’s weird is that the permissions issue was in a directory on another completely separate physical disk, but it was causing this issue. So if this happens to you, go look at permissions under your /var/lib/clickhouse directory. The user that ClickHouse runs as should own everything recursively underneath /var/lib/clickhouse.

Age-based Tiered Storage In ClickHouse

When I first read about tiered storage in ClickHouse, the example they used seemed really strange to me: they configured ClickHouse to move data to another volume when the first volume was 80% full. When I think of tiered storage I immediately think of hot/cold storage volumes, and aging out data to cold storage after some specific period of time. You can totally do that with ClickHouse.

ClickHouse uses ‘storage policies’ to let you inform clickhouse of the disks available to it & the volumes those disks make up. Using a storage policy (and the also-cool TTL clause), you can configure multiple volumes, ‘hot’ and ‘cold’, for example, and then ClickHouse will migrate the data between volumes for you. It can also recompress data or do other operations at the same time. Here’s what I did:

First, set up a <storage_configuration> w/ the hot & cold disks:

     <disks>
        <hot>
            <type>local</type>
            <path>/data/</path>
        </hot>
        <cold>
           <type>s3</type>
           <endpoint>https://mys3endpoint</endpoint>
        </cold>
    </disks>

Now, add the hot/cold policy:

<policies>
    <tiered_storage> <!-- policy name --> 
        <volumes>
            <hot_volume> <!-- volume name -->
                <disk>hot</disk>
            </hot_volume>
            <cold_volume> <!-- volume name -->
                 <disk>cold</disk>
            </cold_volume>
        </volumes>
    </tiered_storage>
</policies>

Now, when you create a new table, you can reference the tiered_storage policy in the SETTINGS, and use a TTL clause to tell ClickHouse what to move and when to move it. For example:

CREATE TABLE default.foo
(
   source_id UUID,
   timestamp DateTime
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(timestamp)
TTL timestamp + toIntervalDay(45) TO VOLUME 'cold'
SETTINGS storage_policy = 'tiered_storage';

ClickHouse will use the order of the volumes in the storage policy to determine priority, so by default, because we defined the ‘hot’ volume first, newly-inserted rows will go there. However, after timestamp + toIntervalDay(45) it’ll be moved to the ‘cold’ volume.

Hope This Helps!

ClickHouse is a very deep product, and each individual feature has a lot of nuance to it, as does the overall behavior of the service in general. This is obviously not the totality of what I learned, but if you’re just starting out with ClickHouse, I hope this saves you some of the hours I spent getting this knowledge. Good luck!

Djoser is a library that integrates nicely into a Django REST Framework project to provide API endpoints for things like user registration, login and logout, password resets, etc. It also integrates pretty seamlessly with Django-SimpleJWT to enable JWT support, and will expose JWT create, refresh, and verify endpoints if support is turned on. It’s pretty sweet.

Well…. most of the time.

The only real issues I’ve had with Djoser always root from one of two assumptions the project makes:

1. That you’re puritanical in your adherence to REST principles at every turn, and
2. That you’re building a Single Page Application (SPA)

That first one is easily forgivable: if you’re going to be an opinionated solution, it’s best to be consistent, and strict. The minute you fall off of that wagon, everything starts to devolve into murkiness. It doesn’t seem like a big leap to say that a lot of developers prefer an API that is clear and consistent over one that is vague and inconsistent.

As for the second assumption, it honestly doesn’t get in the way very often, but on a recent project, it bit me pretty hard. On this project, I had to leverage Djoser in my Django project’s user activation flow.

User Registration and User Activation

User activation happens as part of the user registration process in my case (and, I suspect, most cases). At a high level, the registration flow goes like this:

1. a POST is sent to the server requesting that a given username or email be given a user account, using the given password.
2. the server generates a token of some kind, uses that token to generate a verification link, and sends an account activation email containing the link.
3. the end user opens the email and clicks the link
4. magic
5. the user is activated, and may or may not get an email confirming that their account is ready to go

All of this is straightforward until you get to step 4: ‘magic’. Big surprise, right? Also perhaps unsurprising is that this is where Djoser’s assumptions make life difficult if you’re not building a Single Page Application, and/or are not a REST purist.

Djoser User Registration

Before getting to activation, you have to register. For completeness, it’s worth pointing out that Djoser provides an endpoint that takes a POST request with the desired username, email, and password to kick things off. It integrates nicely with the rest of your application without any real work to do other than adding a urlpattern that’s given to you to your urls.py file.

In my case, I’m using a custom user model and I changed the USERNAME_FIELD to ’email’, and as a result, Djoser accepts just the email and password fields by default, because it’s leaning on the base Django functionality for as much as possible, which is smart and makes everyones’ lives easier.

When this POST request comes in, password validators and any other things you have set up to happen at user creation time will happen, including the creation of a user record. However, the is_active flag will be False for that record. Then it generates an encoded uid (from the record it created) and a verification token, uses them along with the value of ACTIVATION_URL to form a confirmation link, puts that in an email to the email address used to register, and sends it. And that leads us to…

Djoser User Activation

First, let’s have a look at the default value for Djoser’s ACTIVATION_URL setting. This setting determines the URL that will be emailed to the person who is trying to register a new account. The default value is `

'#/activate/{uid}/{token}'`

This is a front end URL with placeholders for the uid and token values. It gets assembled into an account registration verification link that looks like this:

http://localhost:8000/#/activate/Mw/am6c7b-85f2acbaf4691e9cc6c891bbc4fd7754

Up to this point in my project, I was gleefully following along with what Djoser seems to be making easy for me. Then I clicked the above link and everything crashed. Why? Because Djoser does not have a back end view to handle the front end URL that is the default ACTIVATION_URL.

Here’s the explanation from one of the project maintainers: https://github.com/sunscrapers/djoser/issues/14

I suppose you directly use an url to activation view which expects POST request. When you open a link to this view in browser it makes a GET request which is simply not working.

Our assumption is that GET requests should not change the state of application. That’s why the activation view expects POST in order to affect user model. Moreover it’s REST API so if you open one of the endpoints in your browser it displays JSON response which is not something for regular user.

If you’re working on single page application you need to create a new screen with separate url that generates POST request to your REST API.

If you really want to have view that activates user on GET request then you need to implement your own view, but remember to provide reasonable html response.

To boil this down, my understanding from this is that:

1. Djoser devs know that the ACTIVATION_URL is going to be used to create a link that is sent to someone via email.
2. Djoser devs know that, when you click a link in an email, the result is a GET request to the back end.
3. Djoser devs have provided a view for user activation that only supports POST requests in spite of this fact.

This is immensely frustrating. Their implementation seems to sacrifice a product that actually works at all for the sake of REST purity and maybe an assumption that all developers are only creating SPAs. What’s more, searching around for solutions turns up lots of confused people. The solution that I found trending was one where you write your own view that does accept a GET request, and then, inside the view, in the back end code, make a POST request to run the code in Djoser’s UserActivationView!

This all just felt way too… wrong for me. The back end should not make an HTTP request to itself. Perhaps what I did wasn’t 100% perfect either, but I’d be interested in a dialog that could shed more light on why things are the way they are, and how to properly and effectively deal with it.

My Workaround

First, if you’re just here for the code, here’s the view I created and the urlpattern that maps to it.

from djoser.views import UserViewSet
from rest_framework.response import Response

class ActivateUser(UserViewSet):
    def get_serializer(self, *args, **kwargs):
        serializer_class = self.get_serializer_class()
        kwargs.setdefault('context', self.get_serializer_context())

        # this line is the only change from the base implementation.
        kwargs['data'] = {"uid": self.kwargs['uid'], "token": self.kwargs['token']}

        return serializer_class(*args, **kwargs)

    def activation(self, request, uid, token, *args, **kwargs):
        super().activation(request, *args, **kwargs)
        return Response(status=status.HTTP_204_NO_CONTENT)

My Djoser ACTIVATION_URL in settings.py is `

'accounts/activate/{uid}/{token}'

And then the urlpattern used to map requests to that url looks like this:

path('accounts/activate/<uid>/<token>', ActivateUser.as_view({'get': 'activation'}), name='activation'),

What’s Actually Happening In My Workaround

Djoser leans on Django and Django Rest Framework for a lot of its functionality. In order to support a large number of URLs while duplicating the least amount of code, Djoser utilizes Django Rest Framework’s ‘ViewSet’ concept, which lets you map an ‘action’ to a method in a single class. So, instead of having separate views for “UserRegistration”, “UserActivation”, “UserPasswordChange”, and all of the other things that can happen to a user, Djoser just has one class called “UserViewSet” (at djoser.views.UserViewSet).

UserViewSet.activation is a method that takes a POST request containing the UID and token values, validates them, and (assuming validation passes) sends a signal that, in my application, sets the is_active flag on the user to True and sends the new user an email letting them know their account is now active. “It’s all perfect except for the POST!” I thought. But I wasn’t happy with solutions that have code in the back end going back out to the internet to trigger other code on the back end. I wasn’t going to accept sending a POST request to trigger another view.

So, step one was to create my own view, inheriting from UserViewSet, and then allowing that view to accept a GET request, because you’ll recall that our mission is to handle the user clicking the link in their email to activate their account. Aside from accepting a GET request, I don’t really want my code to do anything at all. Just call super().activation and get out of the way!

Now, the base implementation forces POST-only by decorating it with an @action decorator. The first argument to the decorator is a list of HTTP methods supported, and only post is listed. Great! So just don’t decorate that method, map it to a GET in urls.py, and you’re all set!

Sadly, it was not quite that easy. Since UserViewSet.activation only supports a POST request, it also assumes that what it needs is already in request.data. But when our GET request comes in, the uid and token values will be in kwargs. Making things more difficult, the request object here is Django Rest Framework’s Request object, and its data attribute is not settable (I think it’s a property defined with no setter, but don’t quote me). So, I can’t just overwrite request.data and move on. Now what?

So, we need to find a way to get data into request.data so that when I call super().activation(), it can act on that data. In looking at the code for UserViewSet.activation I found this:

    @action(["post"], detail=False)
    def activation(self, request, *args, **kwargs):
        serializer = self.get_serializer(data=request.data)

That’s not the whole method, but for the whole method, the only time request.data is referenced is on the very first line of the code. Since we already said I can’t just shim in a line and overwrite request.data, let’s instead have a look at this get_serializer method!

    def get_serializer(self, *args, **kwargs):
        """
        Return the serializer instance that should be used for validating and
        deserializing input, and for serializing output.
        """
        serializer_class = self.get_serializer_class()
        kwargs.setdefault('context', self.get_serializer_context())
        return serializer_class(*args, **kwargs)

Notice that it doesn’t explicitly have a data parameter defined in the method’s signature. That means any reference to data would have to be in kwargs. That means we can set kwargs['data'] inside of this method to whatever we want. The updated method to make that happen only adds a single line:

    def get_serializer(self, *args, **kwargs):
        serializer_class = self.get_serializer_class()
        kwargs.setdefault('context', self.get_serializer_context())
        kwargs['data'] = {"uid": self.kwargs['uid'], "token": self.kwargs['token']}

        return serializer_class(*args, **kwargs)

That’s it. You just effectively replaced request.data.

One More Time

This is a lot. Let’s review what happened.

First, the mission:

• Support a GET request that happens when the user clicks the account activation link in their email.

Next, the problem:

• There’s code to handle user activation, but it doesn’t support a GET request.

Then, the workaround:

• Create our own view that inherits from the Djoser UserViewSet to handle the incoming GET request.
• Override the activation method to accept the uid and token parameters coming in on the URL and remove the @action decorator that only allowed HTTP POST.
• Override the get_serializer method to insert the uid and token values into its kwargs['data'].
• Define a urlpattern that maps a get request to our ACTIVATION_URL to our newly-created view.
• Profit

If you don’t recall any of the above steps from the discussion, scroll back up to see my code in the My Workaround section.

But Maybe I’m Wrong!

I’m wrong a lot. Maybe you know better. I’d be happy to see a better alternative solution. I’d also love to have a better understanding of the logic Djoser is using, because I admittedly just don’t get that. As a developer, I’m far more comfortable moving from APIs back towards the operating system and infrastructure services than I am moving into front end frameworks (though I do have to do that sometimes). So, if you understand how a ‘front end url’ is sent via email and then expected to somehow be intercepted by a front end that then sends a POST to the back end, please point me to some docs!