LikeLike

LikeLike

Benchmark: KnnSearchBenchmark (numVectors = 100000, k = 100)

=== knnBatchedWithAbort ===

dims=512 → 2.530 ± 0.008 ms/op

dims=1024 → 2.995 ± 1.350 ms/op

dims=1536 → 3.524 ± 0.686 ms/op

dims=1792 → 3.478 ± 0.870 ms/op

dims=2048 → 2.907 ± 0.436 ms/op

=== knnStandard ===

dims=512 → 20.593 ± 15.860 ms/op

dims=1024 → 39.701 ± 0.199 ms/op

dims=1536 → 52.882 ± 0.177 ms/op

dims=1792 → 58.277 ± 1.214 ms/op

dims=2048 → 71.940 ± 0.061 ms/op

LikeLike

However I noticed in your benchmarking tests you only tested pre-normalized vectors, was that intentional?

Correct me if I’m wrong but I think the correctness of the early abort optimization only holds if the vectors are normalized right?

One edge case where I think the early abort would abort wrongly when it shouldn’t is the pessimistic bound in squareDistanceBatched for cases where a vector that belongs in the top-K gets silently skipped.

Here’s an example with unit vectors where I think the early abort wouldn’t work right?

// Two identical unit vectors — actual L2² = 0, should never be aborted

float[] v1 = new float[32];

float[] v2 = new float[32];

float val = 0.25f; // 16 × 0.0625 = 1.0 → unit norm

for (int i = 16; i < 32; i++)

{

v1[i] = val; v2[i] = val;

}

// Returns -2.0f (abort) instead of 0.0f — even though these are identical vectors VectorUtil.squareDistanceBatched(v1, v2, 16, 1.0f);

To make sure my doubt/question was on the right track I asked AI to confirm or deny if my reasoning is correct and explain why and it seemed to agree:

AI Response:

“

After the first 16 dims (all zero), partialDot = 0, so pessimistic L2Sq = 2*(1-0) = 2.0 > threshold. The optimization aborts. But the entire similarity lives in the second batch. The bound 2*(1 – partialDot) is only a valid lower bound on the final L2² if remaining dimensions contribute non-negatively to the dot product and that isn’t guaranteed even for unit vectors when their similarity happens to be concentrated beyond the first batch. Real embedding spaces don’t promise anything about dimension ordering.

There’s also a secondary issue in EUCLIDEAN.compareWithThreshold specifically: it routes through squareDistanceBatched, which uses the 2*(1 – dot) formula — valid only for unit-norm vectors. EUCLIDEAN makes no such guarantee, so passing a non-unit vector like [3, 4, 0, …] produces a negative “distance” and a completely wrong similarity score.

“

I think if there was a method/APi name that explicitly mentions the optimization and that it only works for the linear algebra constraints the Ai mentioned then it’s good right, but currently that pre-normalization assumption doesn’t exist for the EUCLIDEAN class right?

LikeLike

LikeLiked by 1 person

I don’t think so, unfortunately.

LikeLike

Thank you for the post
I’ve just bought and old home and I want to make sure I’m safe.
I see what you’re saying about not being able to know what the source is of the radiation.

However it seems to me using a Geiger counter is a lot simpler than testing for Radon.
So can the Geiger counter maybe inform if a Radon test is necessary? I imagine if there is very little radioactivity there is also little cause for concern?

LikeLike

LikeLike

LikeLike

LikeLike

LikeLike

Satya
https://satya-dba.blogspot.com/2023/04/ddbsh-dynamodb-shell-aws.html

LikeLike

LikeLike

LikeLike

LikeLike

LikeLike

LikeLike

LikeLike

LikeLike

Make sure that the command you executed is

sudo -u znc /usr/bin/znc — datadir=/var/lib/znc –makeconf

that’s two ‘-‘ (hyphens) before datadir, and makeconf.

Read the readme for the correct command if you want to copy/paste.

LikeLike