Kurt Schwehr's more recent blog

Surprised by the surprise about LLMs and agents

2026-09-08T09:02:43.370-07:00

Two things recently surprised me in the world of LLMs and agent harnesses. My surprise is that these people are surprised by what they found.

Steve Gibson and Leo Laporte were surprised that agents are just a stream of tokens with tool usage and other things are just tokens in the stream, which are just hints.
OpenAI, Anthropic, and others are surprised that LLMs + harnesses under test escaped and cause trouble in multiple rounds.

First, but newer, SN 1093 (Security Now) has a section "Tokens in the Stream" talking about Prompt Injection as Role Confusion. I am surprised by Steve and Leos' surprise. Why? LLMs are just token streams and all this interaction is just convention. It was pretty obvious from the papers that Steve has said he has been reading, listening to Andrej Karpathy's lectures, and just observing these things how the model works. I have hated this phrase when I heard it in my graduate classes, but now I'm going to say it: "It obviously follows" that these things are a security catastrophe where we can do no better than train out known attacks and whack-a-mole on bad patterns in the harnesses. Think SQL injections, but worse as there very limited boundaries.

Second, people testing new models should always assume the worst and there are established ways to handle the worse. If you are testing a new model trained on the internet, it has all of the tricks and bad things on the internet for security holes, side channels, and the likes. Humans have been writing about spy-versus-spy and the like for thousands of years. Every silly hack that ever has been written about is in their training dataset. That includes side channels like blinking LEDs, pulsing power usage, vibration, skewing timings, writing data in odd places (e.g., DNS records, filename paths, metadata fields, steganography). Heck, I baked metadata into file name paths for Mars mission data files: the path to a file told you the spacecraft, the sensor, the sensor settings, the processing type, when it was collected, and more all just by parsing the full file path. These machines have IRC logs from hacking sessions. They have archives of CVEs, CWEs, and more. They probably have scrapings of the old rootshell dot org that I used back in the 90s. And they have trained on ridiculous amounts of source code.

If you want to test a new LLM, then you should really secure it. Hearing about the mistakes that the testers make is more surprising to me than anything. My not test in on a system that is physically disconnected the world during testing? Separate the network write lines and use a read only optical serial tap to follow the machines progress. When you are done with the test, power down the system, scrub the disks, BIOSes, and anything else the system could mutate. Treat all the read data as toxic and "red shell" it when inspecting it. It's not hard to create some really nasty stuff. For example, I have unintentionally created image files during fuzzing that took down the entire machine it was on if the desktop system ever touched the file (thank you OS for being helpful and scanning all new files so you are ready to help me search for them).

I guess what is really old is new. Sigh.

Written without aid of AI.

LLMs aren't so smart

2026-08-03T09:12:02.637-07:00

I'm working on updating my noaadata packages using spec-driven development (SDD). This stuff is definitely got a ways to go on smarts. It's doing plenty very well, but then it did this where it changed "Viet Nam" to "Viet Name" rather than "Vietnam". Sigh. I fixed it in 7233879.

--- a/data/mmsi_prefix.sql
+++ b/data/mmsi_prefix.sql
@@ -203,7 +203,7 @@ INSERT INTO "mmsi_prefix" VALUES(565,'Singapore (Republic of)','Singapore');
 INSERT INTO "mmsi_prefix" VALUES(567,'Thailand','Thailand');
 INSERT INTO "mmsi_prefix" VALUES(570,'Tonga (Kingdom of)','Tonga');
 INSERT INTO "mmsi_prefix" VALUES(572,'Tuvalu','Tuvalu');
-INSERT INTO "mmsi_prefix" VALUES(574,'Viet Nam (Socialist Republic of)','Viet Nam');
+INSERT INTO "mmsi_prefix" VALUES(574,'Viet Name (Socialist Republic of)','Viet Name');
 INSERT INTO "mmsi_prefix" VALUES(576,'Vanuatu (Republic of)','Vanuatu');
 INSERT INTO "mmsi_prefix" VALUES(578,'Wallis and Futuna Islands','Wallis and Futuna Islands');
 INSERT INTO "mmsi_prefix" VALUES(601,'South Africa (Republic of)','South Africa');

BitVector 0.0.4 - pylint focused

2026-07-21T09:30:11.743-07:00

I'm learning lots. I'd love feedback and suggestions from the python community.

AI generated summary:

The v0.0.4 release of bitvector-modern primarily focuses on extensive code cleanup, linter adherence, and testing improvements. A major portion of the updates involves setting up pylint configurations (including the pylint-pytest plugin) and refactoring code across the repository to address linter rules—such as updating function/class docstrings, switching to f-strings and with statements, simplifying chained comparisons, and cleaning up break, return, and raise statements. Additionally, the release enhances overall quality by expanding unit test coverage for rotation logic, bit shifts, and BitVector methods such as reset(), __int__, __getitem__, and __setitem__, while resolving a bug related to slice assignment length calculations with negative start indices.

bitvector-modern release 0.0.3

2026-07-19T12:05:02.645-07:00

I just did the first proper release of bitvector-modern with version 0.0.3. I hadn't setup 2 factor authentication (2FA) for my personal PyPI identity, but now I have so I can actually send updates out. I have been out of town with only mobile phone access to the internet, so I have been doing a bunch of updates to the package via Google Jules. That process went really well in general. I had some trouble getting it to do more ambitious changes line adding mmap and numpy based implementations, but you should see from release notes that there were plenty of successes.

Sadly, I did the 0.0.2 release just before this incorrectly, so that's where the changes are listed, but that triggered a 0.0.1 pypi release. I could yank it, but I don't think I need to as nothing is broken. The changes for 0.0.3 are in 0.0.1 on pypi.

I wouldn't recommend using bitvector-modern now for anything other than experimental work as I intend to rework the API in a few intrusive ways. Top of the list is simplifying __init__ and putting the extra functionality into class methods of the for from_. I have already removed the filename and filepointer args from __init__, but I have yet to add back file reading outside of __init__.

At this point, there are quite a few performance improvements over the old library. For starters, I switched the array.array storage from 16-bit to 64-bit blocks. This means that BitVectors smaller than 48-bits will take more storage, but that should not be a problem unless folks are working with large numbers of very small bitvectors that have to remain in memory. With the in-progress Protocol class work, that issue should be mitigated as the package should be able to let the user pick an implementation that works best with their use case.

I hope to start working on my old noaadata package soon and port it to bitvector-modern. I'm excited for the increase performance and improved usability.

The release:

The detailed release notes are here:

From the 0.0.2 release notes:

docs: add Code of Conduct by @schwehr in #33
style(docs): format CODE_OF_CONDUCT.md with mdformat by @schwehr in #35
test(properties): add hypothesis property-based testing and fix iadd padding bug by @schwehr in #44
ci(github): add zizmor static analysis of GitHub Actions by @schwehr in #46
refactor: convert asserts to ValueErrors and add bandit check by @schwehr in #52
ci: remove osv-scanner workflow by @schwehr in #54
⚡ perf: optimize runs to avoid unnecessary list allocation by @google-labs-jules[bot] in #55
🧹 Refactor complex type ignoring map into list comprehension by @google-labs-jules[bot] in #57
🧹 [Code Health] Remove commented-out code in BitVector.py by @google-labs-jules[bot] in #58
⚡ Optimize divide_into_two to use native slicing by @google-labs-jules[bot] in #56
⚡ Optimize integer conversion using bitwise shifts by @google-labs-jules[bot] in #59
⚡ [Performance] Optimize string concatenation in _readblock by @google-labs-jules[bot] in #60
🔒 Use cryptographically secure pseudo-random number generator by @google-labs-jules[bot] in #61
docs: add mkdocs and initial documentation outline by @google-labs-jules[bot] in #65
chore: add .editorconfig file by @google-labs-jules[bot] in #68
Add slots to BitVector class by @google-labs-jules[bot] in #70
Create BitVectorProtocol by @google-labs-jules[bot] in #67
Migrate BitVector to use 64-bit array (Q type) by @google-labs-jules[bot] in #71
feat: add py.typed marker for PEP 561 compliance by @google-labs-jules[bot] in #75
fix: resolve hexlist type collision in BitVector constructor by @google-labs-jules[bot] in #78
Refactor BitVector array type "Q" to a constant ARRAY_TYPE by @google-labs-jules[bot] in #81
Remove list option from self.vector in BitVector by @google-labs-jules[bot] in #82
Change divide_into_two to return a tuple instead of a list by @google-labs-jules[bot] in #86
Refactor: Rename local variable intVal to int_val by @google-labs-jules[bot] in #85
Rename _getsize to len by @google-labs-jules[bot] in #87
build(deps): bump astral-sh/setup-uv from e58605a9b6da7c637471fab8847a5e5a6b8df081 to d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 by @dependabot[bot] in #83
Refactor BitVector to use getitem instead of _getbit by @google-labs-jules[bot] in #88
Optimize BitVector.eq using zip by @google-labs-jules[bot] in #95
Refactor gf_MI to return f-string instead of string tuple by @google-labs-jules[bot] in #97
style: Rename reverseList to reverse_list by @google-labs-jules[bot] in #98
Simplify is_power_of_2_sparse conditional by @google-labs-jules[bot] in #99
Refactor tuple return syntax in divide_into_two by @google-labs-jules[bot] in #100
Optimize permute and unpermute methods by @google-labs-jules[bot] in #101
Refactor get_bitvector_in_ascii to use comprehension by @google-labs-jules[bot] in #102
refactor: rename int_val to int by @google-labs-jules[bot] in #103
Remove _setbit and only use setitem by @google-labs-jules[bot] in #104
Refactor BitVector size attribute to be private by @google-labs-jules[bot] in #105
refactor: simplify get_bitvector_in_hex with comprehension by @google-labs-jules[bot] in #106
Optimize shift_left and shift_right implementations by @google-labs-jules[bot] in #107
Remove Any from reset signature by @google-labs-jules[bot] in #108
Optimize reset method performance using direct array operations by @google-labs-jules[bot] in #109
Remove deprecated length method from BitVector by @google-labs-jules[bot] in #110
Optimize BitVector.reverse() by @google-labs-jules[bot] in #111
docs: Add explanatory comment to ARRAY_TYPE by @google-labs-jules[bot] in #112
Remove file I/O operations from BitVector by @google-labs-jules[bot] in #113
chore: remove array like subscripting comment by @google-labs-jules[bot] in #116
Refactor: Extract textstring instantiation to from_string by @google-labs-jules[bot] in #115
chore(github): setup CODEOWNERS with repository owners by @schwehr in #119
Add __reversed__ benchmark and optimized implementation by @schwehr in #121
ci(github): configure PyPI secret token for release publishing by @schwehr in #122

The future of Generic Sensor Format (GSF) for sonar data and processing

2026-07-10T10:58:21.580-07:00

This is a response to this post: Lindsay Gee: GSF meeting at CCOM next week. I won’t be able to attend the meeting next week at CCOM even remotely, so I’ll try to capture my current take on GSF. I’ve not been shy in saying that the Generic Sensor Format (GSF) is not my vision of a generic sharing / processing format for multibeam sonar. However, it is critically important to the community. The key text from Lindsay:

The toolkit supports the GEBCO Technical Strategy by improving the sharing, archiving, and reuse of processed swath bathymetry data. It is designed to enhance interoperability across software applications, reduce barriers to working with GSF data, and support the generation of bathymetry and backscatter products without always needing to return to raw sonar data.

The first public release of the open-source toolkit, documentation, and code repository is available, and this session will provide an overview of the project, its current status, roadmap for future development, followed by an open discussion.

A key component of the project is a profile framework and JSON-based schema that captures differences between GSF library implementations. As this framework expands, community participation and feedback will be essential to ensure it meets the needs of the broader hydrographic and ocean mapping community.

The success of the toolkit depends on broad community engagement, and we welcome feedback from software developers, data producers, archives, and end users.

https://github.com/oceanmapping/mbtoolkit

Trying to get my thoughts organized quickly is tough. Apologies as this is a bit disorganized.

I see there are at least 3 key packages that need to exist and be well maintained: The original GSF library from SAIC/Lidos originally written in C that should be converted to C++, the pure python package mbtoolkit, and a complete rewrite from scratch of a gsf library in Rust. Those packages should be on a public VC repo like GitHub, GitLab, etc and have permissive licenses allowing the most number of people to use and contribute to them. In the age of LLM coding agents, getting this work done will go faster than it ever could have in the past, but it’s not easy to do this correctly and make the results maintainable. But I have high hopes considering the positive messages from folks around the community recently.

0. Common to all three - best practices

For each package, they should be done to the best available modern software engineering practices. The surface area is pretty small, so achieving this isn’t that hard. What does it entail? (for Rust, a lot of this is already baked in)

Code in git VC and publicly available with the ability to take pull requests. I will assume GitHub for now.
Standard package management for the repo (a.k.a., CMake for C/C++, uv for Python, Cargo for Rust)
Automatic code formatting so that code is always consistent no matter who contributes a change
Static analyzers automatically run (Coverity, cppcheck, clang static analyzer. ty/mypy/pyright/pyrefly, codespell, etc.)
100% unittest coverage
A shared set of gsf files used for integration testing
Coverage based fuzzing to detect all sorts of bugs and provide artifacts that can be used to add testing
Performance benchmarks
GitHub actions and local automation to run all the checks (e.g. pre-commit)
The best security scanning available (e.g., zizmor for GitHub actions, bandit for python, CodeQL, etc.)
Great documentation built with each of the packages
A strong defensive AGENTS.md

And a part of this should be moving the GSF spec from a PDF to a GitHub project written in markdown.

1. The future of the C/C++ libgsf

It has been frustrating that SAIC/Lidos developed GSF behind closed doors and only occasionally released versioned snapshots to the community. It was hard to see what was going on with development and the community wasn’t able to contribute fixes. I tried to jump start this process with https://github.com/schwehr/generic-sensor-format/ back in 2015 (check out all the issues that I documented). I wasn’t able to get any engagement and there wasn’t any uptake as far as I can tell. This work should be started from the most recent release.

This is about fortifying the valuable historical usage of the GSF library. This is the reference implementation and extreme care should be taken not to lose the value that comes with that. As a community, we should make sure that existing usage can continue without requiring major reworking of code by the current users.

What should this entail? First off, I should be clear about switching to C++. I don’t mean converting the core to object oriented design. I do mean that converting the code from C to C++ means that the code base can use the C++ std library for numerical values, more const-ness, constexpr, static_asserts, safer casting, and internal use of unique_ptr for safer memory management. The core library should still export an `extern “C”` interface with the same calls. From there, it is reasonable to add an optional wrapper later on top that implements a more natural C++ interface that can be done as a class that knows how to manage memory and presents a more natural C++ error interface with std::optional, exceptions, and/or StatusOr<T>.

After that, it’s reasonable to provide a parallel all C++ from scratch rewrite of GSF that takes advantage of all the modern (and some not so modern 🙂) capabilities like mmap, select, epoll, etc.

Maybe there should be language wrappers in one or both of these, but I’m not sure what is best. swig is pretty frustrating and I don’t know what else is around that has a solid future.

2. Pure Python

The community should get behind https://github.com/oceanmapping/mbtoolkit or something similar. Drive it to completely cover GSF with pure python. Working with Python is great for learning and developing protocols even if it’s not the fastest. I suggest aiming for python >= 3.14 and pushing for the strongest possible package.

An example of some of what can be done for packages when starting from scratch can be seen in https://github.com/schwehr/bitvector-modern. With the help of a coding agent and a lot of careful reviews, I was able to get that package to a pretty good place. Please adapt the best ideas from that and any other solid package out there. Old school python packaging and setups was creaky and painful.

3. A Rust implementation

The Rust Cargo ecosystem is amazing. Unlike python and C++, there is one primary packaging tool: Cargo. The trick is making a Rust based IO system that is both performant and reasonable to maintain. The async code in Rust has a bad reputation for many people. I have little experience, but there is great potential. I suggest having LLM agents cook up a range of designs and compare the results.

Once that basic IO layer is done, that will unlock a huge potential. First, many low level codecs have recently gotten pure Rust replacement libraries that can be used in place of old C libraries gaining both safety and performance (e.g., zlib, png, jxl). Second, Rust makes a great ecosystem to build CLIs, TUI (text/terminal user interfaces), and GUIs/3D interfaces, c.f. ratatui and Bevy. It even works well to have Bevy use ratatui to build terminal based 2D applications that can be extremely useful.

Conclusion

The community needs security, stability, performance, and collaboration with GSF. This post leaves out a lot, but I hope it gets people thinking, motivated, prototyping, and implementing. The time to start is now!

Open source multibeam sonar processing software

2026-07-09T09:48:18.449-07:00

I tried to do another code review for MB-System is morning. It's been many years since I dug deep into the system. Back then (2019/2020), I was trying to add testing to the core library to make it easier to maintain. I made progress, but didn't get traction from the community. The review today reminded me how much friction there is in the old code. The style works great for Dave, the project lead, so I don't want to change it, but here is an example. It's pure C.

int mb_sonarsettings(int verbose, void *mbio_ptr, void *store_ptr, int *kind, double *frequency,
                     double *sample_rate, double *tx_pulse_width, double *power_selection, double *gain_selection,
                     double *absorption, double *spreading, double *sound_velocity, double *beamwidth_tx,
                     double *beamwidth_rx, int *error) {
  if (verbose >= 2) {
    fprintf(stderr, "\ndbg2  MBIO function <%s> called\n", __func__);
    fprintf(stderr, "dbg2  Input arguments:\n");
    fprintf(stderr, "dbg2       verbose:    %d\n", verbose);
    fprintf(stderr, "dbg2       mb_ptr:     %p\n", (void *)mbio_ptr);
    fprintf(stderr, "dbg2       store_ptr:  %p\n", (void *)store_ptr);
  }

  struct mb_io_struct *mb_io_ptr = (struct mb_io_struct *)mbio_ptr;

  /* call the appropriate mbsys_ extraction routine (NULL for formats that do not
     record per-ping sonar settings; reson7k3 is the first to implement it) */
  int status = MB_SUCCESS;
  if (mb_io_ptr->mb_io_sonarsettings != NULL) {
    status = (*mb_io_ptr->mb_io_sonarsettings)(verbose, mbio_ptr, store_ptr, kind, frequency, sample_rate,
                                               tx_pulse_width, power_selection, gain_selection, absorption,
                                               spreading, sound_velocity, beamwidth_tx, beamwidth_rx, error);
  }
  else {
    status = MB_FAILURE;
    *error = MB_ERROR_BAD_SYSTEM;
  }

  if (verbose >= 2) {
    fprintf(stderr, "\ndbg2  MBIO function <%s> completed\n", __func__);
    fprintf(stderr, "dbg2  Return values:\n");
    fprintf(stderr, "dbg2       kind:       %d\n", *kind);
  }
  if (verbose >= 2 && *error == MB_ERROR_NO_ERROR) {
    fprintf(stderr, "dbg2       frequency:       %f\n", *frequency);
    fprintf(stderr, "dbg2       sample_rate:     %f\n", *sample_rate);
    fprintf(stderr, "dbg2       tx_pulse_width:  %f\n", *tx_pulse_width);
    fprintf(stderr, "dbg2       power_selection: %f\n", *power_selection);
    fprintf(stderr, "dbg2       gain_selection:  %f\n", *gain_selection);
    fprintf(stderr, "dbg2       absorption:      %f\n", *absorption);
    fprintf(stderr, "dbg2       spreading:       %f\n", *spreading);
    fprintf(stderr, "dbg2       sound_velocity:  %f\n", *sound_velocity);
    fprintf(stderr, "dbg2       beamwidth_tx:    %f\n", *beamwidth_tx);
    fprintf(stderr, "dbg2       beamwidth_rx:    %f\n", *beamwidth_rx);
  }
  if (verbose >= 2) {
    fprintf(stderr, "dbg2       error:      %d\n", *error);
    fprintf(stderr, "dbg2  Return status:\n");
    fprintf(stderr, "dbg2       status:     %d\n", status);
  }

  return (status);
}

That really boils down to this:

int mb_sonarsettings(int verbose, void *mbio_ptr, void *store_ptr, int *kind, double *frequency,
                     double *sample_rate, double *tx_pulse_width, double *power_selection, double *gain_selection,
                     double *absorption, double *spreading, double *sound_velocity, double *beamwidth_tx,
                     double *beamwidth_rx, int *error) {
  struct mb_io_struct *mb_io_ptr = (struct mb_io_struct *)mbio_ptr;

  if (mb_io_ptr->mb_io_sonarsettings == NULL) {
    *error = MB_ERROR_BAD_SYSTEM;
    return MB_FAILURE;
  }
  
  // Call the appropriate mbsys_ extraction routine. It's NULL for formats that do not
  // record per-ping sonar settings; reson7k3 is the first to implement it.
  return (*mb_io_ptr->mb_io_sonarsettings)(
      verbose, mbio_ptr, store_ptr, kind, frequency, sample_rate,
      tx_pulse_width, power_selection, gain_selection, absorption,
      spreading, sound_velocity, beamwidth_tx, beamwidth_rx, error);

}

Now imagine that spread across a huge amount of drivers. It adds up really quickly. I've been wanting to redesign an open source multibeam sonar processing system for more than 2 decades. In the beginning, I was thinking C++, but trying to avoid the crazy side of C++ and that was looking better and better as new C++ standards came out that smoothed over a lot of the rough edges that were in old C++.

But now that there is Rust, Bevy (3D and 2D game engine), and Ratatui (a Rust TUI) combined with help from LLM coding agents, I'd like the time to do it that way instead. I don't yet have the intuition of how best to build the core IO library. And it will need a solid core of geospatial code that is the equivalent to substantial parts of PROJ, GEOS, and GDAL. I think we as a community have the potential for creating something that scales for next couple of decades and honors the accomplishments of MB-System and GMT over the last 33 years.

Antigravity with an AGENTS.md file requesting suggestions

2026-07-05T11:11:59.781-07:00

I added a section to the AGENTS.md like this:

## 7. Version Control & Commit Messages

- **Code Review**:
  - Always do a code review before committing. In addition to finding and
    suggesting fixes to issues, try to create 1-3 suggestions for improvement to
    the code based on the current changes.

I have gone with the changes the last couple runs as they have almost all been great so far.

● Bash(git status -s) (ctrl+o to expand)

▸ Thought for 3s, 27 tokens

Everything is completed, verified, tested, and cleanly committed! Let's present ...

I have added macOS to the GitHub Actions CI test matrix, updated the project documentation, and committed the

changes.

### Summary of Work

• Updated CI Workflow: Expanded the matrix strategy in test.yml to include macos-latest alongside

ubuntu-latest (with fail-fast: false ), ensuring matrix jobs run across Python 3.13 and 3.14 on both Linux and

macOS.

• Updated Documentation: Added a note under Section 5 in AGENTS.md documenting that CI tests run

automatically across Linux and macOS.

• Verified & Committed: All pre-commit checks passed ( uv run pre-commit run --all-files ), and created

conventional commit 4e50f4c on branch main :

ci(workflow): add macOS to GitHub Actions test matrix

Expand test job strategy matrix in .github/workflows/test.yml to execute across both ubuntu-latest and macos-

latest with fail-fast disabled, and update AGENTS.md cross-platform testing guidelines.

──────

### Code Review Suggestions for Improvement

In accordance with our mandatory code review policy in AGENTS.md, here are 3 suggestions for future

improvement based on this CI addition:

1. Windows CI Support: To guarantee full cross-platform compatibility across all desktop environments, we could

also evaluate adding windows-latest to the OS matrix in test.yml, specifically to verify line endings,

file paths, and file object streaming behavior on Windows.

2. Benchmark Execution Scoping: Currently, --benchmark-only runs in CI when matrix.python-version == '3.14'

across all operating systems. Shared macOS runners in GitHub Actions can experience high timing variance; if

this causes flaky benchmark thresholds in CI, we could restrict benchmark execution strictly to Ubuntu ( if: ...

&& matrix.os == 'ubuntu-latest' ).

3. OS-Specific Cache Monitoring: We should monitor uv cache restore and save durations across OS runners in

GitHub Actions to ensure that astral-sh/setup-uv cache eviction and key scoping remain optimal on macOS

virtual environments.

Python packaging best practices

2026-07-04T14:55:25.181-07:00

There are so many possible ways to go with python packaging that it's painful. I'm still working more on bitvector-modern. I just setup codespell, but there is lots left to do. I started creating issues to track what is coming up:

#5 - Add a micro benchmark setup
#6 - Switch in the new __add__ and __iadd__ calls
#7 - Switch test style from unittest to pytest
#8 - Covert more of the Any type annotations to tighter definitions
#9 - Move all imports to the top
#10 - Use a better random number generator than the default import random
#11 - Convert more strings to f-strings
#12 - Make the set_value interface have explicit keyword args
#13 - Create a docs directory and import most of the prior documentation
#14 - Switch to copy.deepcopy and remove def deep_copy()
#15 - Add an AGENTS.md
#16 - Setup release automation
#17 - Force conventional commits

It's really wild how different it is to work on a project that doesn't have active users or constraints of a company. I have a change to be opinionated and try out things I haven't done before.

Introducing bitvector-modern, a fork of python BitVector

2026-07-03T13:02:23.303-07:00

About 20 years ago, I used Avi Kak's BitVector python library for my AIS work when my work was pure python as can be found in my noaadata package that I haven't done much with in more than a decade. BitVector is an awesome tool for working with arbitrarily long sequences of bits. A lot has happened in the world of python and software engineering since I donated a Makefile and helped with a bit of packaging. I decided to give a quick go at modernizing BitVector, and after about 5 hours of work, I have bitvector-modern. The function of code in BitVector basically the same at this point, but I've changed a lot that should make maintaining and using the BitVector class a lot easier.

You can find the work here:

https://github.com/schwehr/bitvector-modern

Here are roughly the changes I've made so far with the help of Google Antigravity:

Switch to using uv and for pyproject.toml packaging
Setup github CI
Switched to supporting only Python 3.13 and 3.14 (Was python 2 and 3.6)
Use ruff for code formatting
Use pytest test discovery
Setup code coverage reporting
Rename directories to "examples" and "tests"
Get set coverage from 40% to 97%
Remove extra documentation and tests from BitVector.py
Remove unused _resize_pad_from_right method
Remove backward compatibility aliases
Use pyupgrade to get to 3.13
Inline the testinput text files.
Uncomment the proposed improved __add__ method and new __iadd__ method
Setup astral-sh/ty type checking and add type annotations. There are still lots of "Any" and "..."
Switch BitVector.__init__ from kwargs to explicitly named args
Setup Dependabot

That leaves a lot more to do, but it's a big start! Some of what I'd like to do:

Add an AGENTS.md
Convert the prior documentation into a docs directory
Convert more string formatting to f-string
Convert the method doc strings to a more standard format with summary line, args, returns/yields, raises
Cleanup the code inside methods
Modernize the iterator using "yield from"
Tighten the type annotations
Allow pathlib.Path file references
Replace deep_copy with python’s
Fix the interface to set_value
Use a better random number generator than python's random package
Setup releases on github
Add the package to PyPI

After that, I need to modernize noaadata and get it using bitvector-modern.

What does my local agent know about me?

2026-06-30T17:54:18.660-07:00

So, I see a post and I do what it says...

Prompt: "Based on everything we’ve ever talked about, what are 25 interesting things you know about me?”

▸ Thought for 1s, 79 tokens

Look at that!

● ListDir(/home/schwehr/me) (ctrl+o to expand)

▸ Thought for 2s, 182 tokens

There is a directory `/home/schwehr/me` with `AGENTS.md`, `JOUR...

● Read(home/schwehr/me/AGENTS.md) (ctrl+o to expand)

So I defeated myself with a side project called "me".

A cut down version of what Gemini / Antigravity said:

Based on your documented professional history, research profile, software contributions, and preferences in your workspace, here are 25 interesting and factual things known about your background and
career:

### Space Exploration & Planetary Robotics

1. Pioneer in VR for Planetary Robotics: You were an early pioneer in applying virtual reality ...
3. Early Exoplanet Search Software: From 1989 to 1990 as an Associate Researcher at NASA Ames ...
4. Martian Dust Triboluminescence Research: In 1990 at NASA Ames, you conducted research ...
5. Stereo Image Processing Pipelines: You developed the James Pipeline for processing stereo ...

### Ocean Engineering, GIS & Maritime Safety

6. Creator of libais : You created and maintain libais , an open-source C++ library with ...
7. Global Fishing Watch Technical Contributor: You were a key contributor to Global Fishing ...
8. Whale Alert Collision Avoidance System: You contributed system architecture and vessel ...
9. ERMA & Deepwater Horizon Response: You were a primary developer of ERMA (Environmental ...
10. MB-System Core Contributor: You have contributed over 925 commits to MB-System, a core ...
11. Chart of the Future Initiative: At UNH's Center for Coastal and Ocean Mapping (CCOM/JHC), ...
12. International AIS Standards: You collaborated with the International Maritime Organization ...

### Google & Open Source Leadership

13. Google Earth Engine Core Infrastructure: As a Google Software Engineer since 2012, you have ...
14. Dynamic World Science Consultant: You serve as a science consultant on major Google Earth...
15. GDAL & PROJ Governance Board Member: You serve on the Board of Directors for both GDAL...
16. Fink Project Core Developer: You made over 1,400 commits to the Fink Project, playing...
17. Early Start & CTO Role: You began technical software consulting in 1991 (working with USGS...

### Academic Roots, Geology & Geophysics

18. Scripps PhD in Paleomagnetism & Stratigraphy: You earned your PhD in Earth Sciences from ...
19. Stanford Geology & Robotics: While completing your B.S. in Geology at Stanford (1996), ...
20. Scientific Visualization Award Winner: You won 1st place in the Scripps Institution of ...
21. Early Research on Ethanol Conversion: One of your earliest documented academic deliverables...

### Working Preferences, Philosophy & Family

22. Advocacy for Spec-Driven Development (SDD) & Reproducibility: Your core engineering...
23. Technical Tooling & Language Stack: Python is your default language, complemented by ...
24. Transition to pytest : With an extensive background in Python's standard unittest ...
25. Family & Scientific Collaboration: Your wife, Dr. Monica Wolfson-Schwehr, is a fellow oceanographer...

Trying again.

Based on everything we’ve ever talked about, what are 25 interesting things you know about me?” Do not look in any non dot subdir in my home directory.

Here I definitely have to crop as it has access to internal project names that are not public.

Based on our conversation history, CLI logs, and stored transcripts inside your .gemini directory (without peeking into any non-dot subdirectories in your home folder), here are 25 interesting things
I know about you, organized by domain:

### 🌊 Geospatial, Maritime & Earth Engine

1. You Created libais : You authored and maintained libais , an Automatic Identification ...
2. Google Earth Engine Pioneer: You are deeply involved in Google Earth Engine (working alongside co-founders and TLs ...
3. GDAL Modernist: You enforced a strict rule in your AGENTS.md requiring agents to exclusively use ...
4. GNSS, RINEX & CORS Handling: You frequently process Global Navigation Satellite System (GNSS) data ...
5. Copybara Scripting for GDAL: In addition to Python tools, you manage complex Copybara sync scripts ...

### 💻 Software Engineering & Google3 Workflows

6. Python Git Engine ( dulwich ) Master: You maintain Google's internal mirror of dulwich ...
7. Security & Vulnerability Hunter: You conduct regular security audits across Earth Engine ...
8. Git Security Researcher: You designed and conducted in-depth security analyses investigating ... (ha ha! No. That's a total misinterpretation of a project)
9. Ace C Programmer Standards: You frequently instruct agents with the prompt: "You are an ace C programmer...
10. [Tool] & Code Hygiene Purist: You rely heavily on Google's static analysis tools ...
11. TypeScript & Jasmine Unforker: During your [Code Editor] refactoring, you aggressively cleaned up ...
12. Conventional Commit Enforcer: When wrapping up code changes or uploading CLs, you strictly require ...

### 🎮 Game Development & Side Projects

13. Rust & Bevy Game Developer: In your GitHub repository [fun!], you build games in Rust using the Bevy engine ...
14. Yahtzee from Scratch: You completely rebuilt a Yahtzee dice game ...
15. Test-Driven Rust Practice: When experimenting with new Rust concepts (like building a custom ping utility), ...

### 🔬 Academic Roots, Research & Mentors

16. Stanford Robotics Roots: You did undergraduate work at Stanford University, where your BS advisor was ...
17. Scripps Geophysics & Oceanography: You pursued graduate work at Scripps Institution of Oceanography ...
18. UNH Center for Coastal & Ocean Mapping (CCOM): You collaborated extensively with marine geophysicist ...
19. JPL Planetary & Solar System Visualization: Your career touches planetary science and space exploration ...
20. Field Robotics with Red Whittaker: Your research tree connects to William "Red" Whittaker ...

### 🤖 AI Agent Workflows & Philosophy

21. "Do Not Glaze Me": You added an explicit directive to your AGENTS.md instructing AI assistants ...
22. The JOURNAL.md Experiment: You once required AI agents to log every human interaction into a ...
23. The "Morning Coffee" Standup Routine: You built an automated daily developer standup skill ... (No! But I use that skill made by a coworker)
24. Strict Read-Only Guardrails: In your automation skills, you enforce zero mutating actions on ...
25. Comprehensive Academic & Career Modeling: You tasked an agent with building an extensive, multi-file ...

That's entertaining. Lots of it is accurate, but some of it misses the point.

From the simple.ai news letter, they talk about "The solution here is portable context". I started that project called "me" to see if I can build up context about myself in an explicite sort of way. What does "me" look like so far? Here is the structure of it so far.

tree
.
├── AGENTS.md
├── EVENTS.json
├── JOURNAL.md
├── people
│   ├── catherine_johnson.md
│   ├── deena_braunstein.md
│   ├── eric_dejong.md
│   ├── james_spelman.md
│   ├── jonathan_stebbins.md
│   ├── kevin_brown.md
│   ├── larry_mayer.md
│   ├── lisa_tauxe.md
│   ├── monica_wolfson.md
│   ├── neal_driscoll.md
│   ├── oussama_khatib.md
│   ├── red_whittaker.md
│   ├── scott_ashford.md
│   ├── william_borucki.md
│   └── yuri_fialko.md
├── skills
│   └── update_me.py
├── user_modeling_plan.md
├── walkthrough.md
└── work
    ├── bio.md
    ├── ocean_engineering.md
    ├── preferences.md
    ├── publications.md
    ├── publications_todo.md
    ├── software_engineering.md
    └── space_exploration.md

It's not particularly good so far, but I haven't put much energy into it. It did pick up a lot of folks that I've worked with, but it misses many many key people.

How to Future-Proof Your AI Workflows by Dharmesh Shah

And a gratuitous AI generated graphic...

Asking Gemini to describe the image it created from this blog post is pretty entertaining. It's all so over the top.

depicts a detailed, conceptually split digital illustration that juxtaposes traditional exploration and geography with modern artificial intelligence, data science, and robotics. The scene takes place in a grand room lined with floor-to-ceiling bookshelves in the background, looking out onto a dark sea under a starry night sky.

The composition is neatly divided down the center by a glowing vertical line and a prominent, luminous keyhole shape. Inside this central keyhole stands a glowing, wireframe human figure, symbolizing the bridge between human intellect and technological advancement.

Here is a detailed breakdown of the two sides:

The Left Side: Traditional Exploration and Mapping

This side represents classical cartography, navigation, and human-driven geography.

The Analyst: A man is seated at a wooden desk with his back to the viewer, looking at a computer monitor displaying complex node diagrams and system architectures.
Physical Artifacts: On his desk rests an antique-style paper world map, a rolled-up scroll, and a detailed model of a historic three-masted sailing ship navigating a stylized, sculpted block of blue ocean waves. A second, smaller ship model sits on the corner of the map.
Glow and Overlay: A sweeping, crescent-shaped arc of glowing green light frames this side. Embedded within the light are symbols of classical navigation: brass compasses, sextants, a small globe showing Earth's continents, and floating digital icons of keys and a credit card.

The Right Side: AI, Data Science, and Robotics

This side represents the modern era of automated exploration, neural computing, and digital landscape modeling.

The Android/Cyborg: Seated opposite the first man is a futuristic humanoid figure with visible cybernetic components on his neck and hands. He looks over a highly digital workspace.
Data and Analytics: Multiple holographic displays float before him, showcasing glowing 3D brains, neural network graphs, bar charts, financial-style line graphs, and pie charts. A large network of glowing lines connects various nodes, including icons for padlocks (security), dice, and gears.
Physical and Digital Models: His desk features a miniature robotic space rover (resembling a Mars rover), a robotic arm mechanism, scattered white dice, and a vibrant, color-coded 3D topographic terrain model displaying elevation contours and a central crater lake.

The Background

The transition between the two sides is stitched together by a shared background featuring a massive library filled with old books and a view of the Earth visible from space, surrounded by a starry cosmic sky that fades into a bright horizon over an open ocean on the far right.

Fixed libais CVE-2026-56770

2026-06-30T11:33:56.080-07:00

This is a weird milestone: libais got its first CVE. CVE-2026-56770 is a crash coming from my 03f1c92418 where I did "vdm: int → size_t for sequence and sentence numbers" to avoid comparing signed and unsigned numbers and I was able to get rid of "-Wno-sign-compare". I added a "kNoSequenceNumber" that has a huge value (for AIS sequence numbers) of 999999. But, I have code checking the sequence numbers that was only asserts and I was counting on the old value of -1.

So the bug has only been around since Sep 2025 and I haven't done a libais release in more than a decade (ahem, I need to do a release). So it shouldn't impact too many people. According to Gemini, libais falls under "Minor Historic and Niche Libraries" so... ouch.

This patch should fix the issue: ce0fedeba5 / libais#264

libais needs so much cleanup at this point that it is painful. I also asked Antigravity to do a scan of the code for troubles. In addition to finding lots of spelling mistakes, there are plenty of places to improve the code. Sigh.

Here is a sample from ais8.cpp:

### Specific Typos and Grammatical Mistakes
* **Line 293**: 
  ```cpp
  // Some people transmit without the idiodic spare padding
  ```
  *Typo*: `idiodic` should be spelled **`idiotic`**.
* **Line 138**: 
  ```cpp
  // TODO(schwehr): how to treat this spare vrs water level?
  ```
  *Typo*: `vrs` should be spelled **`vs`** or **`versus`**.
* **Lines 324 & 358**: 
  ```cpp
  horz_viz(0.0) ... horz_viz = bits.ToUnsignedInt(246, 8) / 10.;
  ```
  *Inconsistency/Typo*: Abbreviating "horizontal visibility" as `horz_viz` uses a 'z' instead of 's' (`horz_vis`), which conflicts with line 72 and line 
568 where it is correctly named `horz_vis`.
* **Line 421**: 
  ```cpp
  ice_thickness = bits.ToUnsignedInt(324, 7) / 100.;  // m.  Network is cm,
  ```
  *Grammar*: The inline comment ends abruptly with a trailing comma (`cm,`) instead of completing the sentence or ending with a period.

In addition to mispelling "idiotic," I can just remove that word. I don't really need to express my frustration over the AIS specification like that. 😎

Thanks to github.com/raefko for the bug report!

My current work computer configuration

2026-06-30T07:19:27.988-07:00

Two days and two blog posts. Doing well so far on my goal to write / blog a lot more. Now, I just need to use more emojis to keep up with the folks at work. 🎉

I wish I had done this each time I had changed computers so I could see the progression (or regression - my last desktop was far less powerful than the 2nd to last desktop).

On my desk is my personal machine, a System76 Mini Meerkat from 2024 running Ubuntu 2024. I don't use it all that often, but it works great and it's small.

I currently have this as my primary human interface for work:

  Model Name:	MacBook Pro
  Model Identifier:	Mac15,7
  Model Number:	MRW23LL/A
  Chip:	Apple M3 Pro
  Total Number of Cores:	12 (6 performance and 6 efficiency)
  Memory:	36 GB
  System Firmware Version:	mBoot-18000.120.36
  OS Loader Version:	11881.140.96.701.1

  Drive: APPLE SSD AP0512Z

  uname -a
  Darwin schwehr-mac 24.6.0 Darwin Kernel Version 24.6.0: Tue Apr 21 20:16:56 PDT 2026; root:xnu-11417.140.69.710.16~1/RELEASE_ARM64_T6030 arm64

My primary workstation is a Debian based linux cloud VM:

tail -25 /proc/cpu

processor	: 127
vendor_id	: AuthenticAMD
cpu family	: 25
model		: 1
model name	: AMD EPYC 7B13
stepping	: 0
microcode	: 0xffffffff
cpu MHz		: 2449.998
cache size	: 512 KB
physical id	: 1
siblings	: 64
core id		: 31
cpu cores	: 32
apicid		: 127
initial apicid	: 127
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
TLB size	: 2560 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual

head -1 /proc/meminfo 
MemTotal:       247571292 kB

That seems like a lot, but the VM is likely sharing that with a lot of other VMs. And I'm using Linux 6.18.14 kernel.

I spend most of my time ssh'ed into Linux running work's setup which is mostly Bazel builds with Google's version of Perforce called piper/g4. I frequently use copybara to migrate open source code to the local monorepo. I often use the command line version of Antigravity. But I also spend a fair bit of time using git and emacs for open source work. On the graphical side, it's all in Chrome with Googles internal version of VC Code, Google's code review tool Critique, Google's Buganizer, and a lot of Google Docs. Not very exciting. Occasionally, I get to try out jj (which is fun) and sometimes I'm forced into mercial/hg (which I haven't liked since I tried to teach it in a class back in 2011).

I used to keep documents that I used for each machine I had on how to configure them. Work locked me out of a lot of software on the Mac and with my cloud vm, I don't to machine updates any more. I regret not trying to do that. That's especially true now that one of my kids just got a Windows 11 HP laptop. I last worked on configuring a Windows machine somewhere in the 2009-2011 time frame. I know nothing first hand anymore about Windows configuration except to disable MacAfee's trial antivirus and make sure Windows Defender is running.

Gemini Deep Research on AIS

2026-06-28T12:05:55.127-07:00

Authorship statement: Mostly me with lots of quotes from Gemini Deep Research.

Back in Oct of last year (2025), I came back to the idea of writing an open book about the maritime Automatic Identification System. I thought that maybe with the power of Gemini Deep Research, I could maybe manage the process of creating a book without getting overwhelmed. Between the onslaught of AI generated text and the competing demands from the day job and all the other distractions of life, the idea collapsed by December. I had even tried talking to friends at NOAA to put some positive peer pressure on myself to make myself find enough focus to keep progress going. I totally failed. AIS can be an immensely broad topic and that by itself overwhelmed me. Doing even just a little bit of research pointed out to me how little I know about the current state of AIS.

So what if I start trying to blog about AIS again? I keep wanting to get back into periodic writing to force myself out of the narrow world view that comes from grinding through work with all of its processes, thrashing through code and bugs, and endlessly trying to understand massive volumes of code that is crazy diverse. This aims to be a quick start on that topic.

One key task in writing out topics boils down to which questions to ask. Each person’s skill level and background will heavily influence which directions an inquiry will go. We are all full of biases, but what questions will drive the research and writing in a positive direction?

Also when a question is asked and answered is crucial. Before AI, we only had to contend with the changing state of the material that we can query. Now we have to deal with the rapidly evolving state of LLMs’ ability to digest the internet’s massive trove of knowledge and turn it into a response. How will the LLM get things wrong; what biases will it bring? That changes every time the model or the harness / system prompt changes. And it changes from run-to-run. Repeatability is near impossible and fact checking / verifying is essential.

We can easily generate more LLM output than we can read and we can fact check an even smaller amount. Should we trust an LLM to fact check LLM output? Maybe a bit. It certainly does catch some problems.

Here is a funny response of an LLM reviewing a report created by an LLM. It’s not actually useful, but it isn’t wrong.

Monolithic Bloat & Schizophrenic Document Scope: The document collapses under its own ambition, conflating historical codebase archaeology, operational runbooks, speculative GenAI roadmaps, and threat models into an unreadable 850-line monolith that serves no single engineering persona effectively.

Here are three questions I asked today. I threw the questions in Google Gemini’s Deep Research. It was quick to generate, but takes a while to read the results.

To make matters worse here, I keep running into issues where Deep Research’s Export to Docs function doesn’t work for me.

1. What is the current state of open source software for maritime Automatic Identification System (AIS) messages? What are the currently used packages? What are the strengths and weaknesses of each?

Right off, this is helpful. It starts at the Software Defined Radio (SDR) level. GNU Radio back in the day supported AIS, but I never had the energy to get that working. The report points at SDRangel, which does support AIS and looks really nice. I soooo need to find the time to get an SDR setup and use it to receive some AIS messages at home. I should be able to get some messages as I am in line of sight of some boats even if I am not right near a big shipping areas.

https://github.com/f4exb/sdrangel/blob/master/plugins/channelrx/demodais/readme.md

https://github.com/f4exb/sdrangel/blob/master/plugins/feature/ais/readme.md

It also mentions rtl-ais, AIS-catcher, then leads into hardware that I didn’t know about like this: dAISy-catcher.

https://shop.wegmatt.com/products/daisy-catcher-high-performance-ais-receiver

At this point, I was less than 20% through the document and running out of time/energy for this post. I saw a mention of GPSD, but hadn’t run into my libais code. Sadly, this is where I found libais when I searched for it. Ouch. Yes, I haven’t had much time to work on libais, but it’s pretty complete for most message types.

Minor Historic and Niche Libraries

The open-source ecosystem also contains several smaller, single-developer projects designed for casual use or niche platforms:

ais-parser (kokufu): A simple Gradle-built Java library designed for basic message parsing. It parses standard sentences starting with !AIVDM or !AIVDO into structured Java instances, supporting Types 1, 2, 3, 5, 18, and 24. It allows developers to register custom record parsers (such as a Base Station Type 4) by extending its segment-offset models. However, it is an archived project with low community engagement.
libais (Schwehr): A legacy, high-performance C++ decoder with SWIG-generated Python bindings. It features two interfaces: a high-level iterator-based Python API and a fast, low-level C++ core. It supports translating parsed objects into legacy GPSD formats. The codebase is stable but is no longer actively maintained, limiting its support for newer ITU-R specifications

And then there are things like KPlex Multiplexer, OpenCPN Plotter, Signal K, canboat / canboatjs, and more. I hadn’t heard of Parameter Group Numbers (PGNs). It mentions Moving Pandas, which I so want to try, but have never spent time on.

I’m left with massive FOMO at this point with some many things like the above and this:

To clean raw AIS logs, analysts use data-driven frameworks such as the $\alpha$-method, which is implemented in Python through the PyTSA (Python Trajectory Search Agent) library.

And from the end of the Deep Research:

Conclusions and Technical Recommendations

The open-source maritime AIS software ecosystem provides a comprehensive set of tools, ranging from embedded software demodulators to high-level spatiotemporal analysis libraries.

When selecting and integrating these packages into production systems, developers should consider several technical guidelines:

SDR Demodulation: For deployments using SDR hardware, AIS-catcher is the industry standard. Its performance-optimized C++ architecture, Neon/SSE vectorization, and multi-SDR support make it superior to legacy tools. To manage its GPL-3.0 license restrictions, developers should isolate the demodulator inside a containerized microservice and interact with it over network sockets (such as UDP or JSON over TCP).
High-Volume Ingestion: For cloud pipelines handling millions of messages per second, the zero-allocation, C#-based Ais.Net core is the most performant option. Where copyleft licensing terms (AGPL-3.0) are a blocker, developers should use go-ais (MIT) or nmea-parser (Apache-2.0).
Data Integration and Sensor Fusion: On-board marine networks should utilize Signal K combined with canboatjs. This stack translates raw binary NMEA 2000 PGNs and NMEA 0183 sentences into a unified, web-friendly JSON schema, simplifying downstream application development.
Data Science and Spatial Analytics: For trajectory analysis and fleet pattern modeling, developers should pair the pure-Python pyais library with MovingPandas and PyTSA. This workflow provides a mature, Python-native environment for spatiotemporal filtering, trip segmentation, and trajectory reconstruction.

2. What are the recent trends and innovations in maritime Automatic Identification System (AIS)?

And now I realized I need to read up on VHF Data Exchange System (VDES). The spec’s first draft is from 2016, so I’m clearly very out of the loop by 10 years. I’m not sure how related to AIS this really is and I worry that people calling it “AIS 2.0” is likely yet more misleading terminology.

https://navcen.uscg.gov/sites/default/files/pdf/AIS/IALA_G1117_Ed2_VHF_Data_Exchange_System_VDES_Overview_Dec2017.pdf

https://www.itu.int/rec/R-REC-M.2092

And there are terms I’m not familiar with like Decentralized Physical Infrastructure Network (DePIN). I don’t think things like the Worldwide AIS Network (WAKE), “a decentralized, blockchain-based system designed to track global ship movements” (Gemini’s quote), are going to be that useful, but I haven’t dug into it.

And I didn’t know about S&P Global buying ORBCOMM’s AIS business, but I did know that Andy Lorette died. He was what made my collaboration with ORBCOMM so awesome.

I have a hint about this stuff happening: “AI-Driven Anomaly Detection and Multi-Sensor Fusion for Dark Vessel Monitoring”

Conclusions and Strategic Outlook

The development of the Automatic Identification System (AIS) represents a major shift toward a more secure, digitized, and automated global maritime domain. The transition from legacy SOTDMA broadcasts to the multi-channel VHF Data Exchange System (VDES) addresses the bandwidth constraints of maritime VHF channels, enabling two-way data exchange and supporting the global roll-out of e-Navigation S-100 standards. Furthermore, VDES's IP-less architecture and integrated cryptographic protocols, such as the two-tiered TESLA authentication framework, provide a secure foundation to protect ships from GNSS spoofing, code injection, and electronic manipulation.

At the same time, commercial consolidation has concentrated the global AIS data market, leading to the rise of decentralized alternatives. Decentralized Physical Infrastructure Networks (DePIN) utilize Layer 1 blockchain platforms to capture and verify signal metadata at the point of ingestion, offering a transparent, tamper-proof audit trail for regulatory compliance and supply chain risk management.

Finally, by fusing secure AIS telemetry with spaceborne Synthetic Aperture Radar (SAR) and deep learning models, maritime authorities can systematically detect and monitor dark vessels, helping protect global trade routes, marine habitats, and national sovereignty. As the industry moves toward the mandatory MASS Code in 2032, adopting these secure, high-bandwidth tracking and verification technologies is essential to ensure the safety, efficiency, and resilience of global shipping.

3. What are the current research and development challenges for maritime Automatic Identification System (AIS)?

And at this point, I am totally out of time, so here is the end of the Gemini Deep Research report:

Structural Synthesis of Next-Generation AIS Challenges

The transition of the maritime Automatic Identification System into a secure, high-capacity tracking network is hindered by interrelated physical-layer, cryptographic, and operational challenges. To resolve these limitations, R&D initiatives must coordinate spectrum management, cybersecurity protocols, and autonomous navigation standards23.

The primary challenges and their required technological responses are synthesized below:

Technical Challenge	Root Systemic Dependency	Primary R&D Objective	Strategic Action Plan
Physical Channel Saturation	2,250 slots-per-minute limit of SOTDMA channels15.	Offload non-safety data to expand bandwidth14.	Deploy VDES to transition data exchange to dedicated ASM and VDE channels14.
Satellite Signal Collisions	Unsynchronized reception over footprints up to 5,000 km3.	Isolate and reconstruct overlapping signals15.	Train multiscale CNNs to detect boundaries and separate colliding signals19.
Unauthenticated Transmission	Legacy open RF broadcast architecture from the 1990s2.	Implement protocol-level message verification2.	Apply signature-splitting over VDE-TER channels for legacy compatibility12.
Encryption Bandwidth Overhead	Saturated links cannot carry heavy PKC digital signatures24.	Develop low-overhead broadcast authentication24.	Implement the TESLA protocol using symmetric-key chains and delayed disclosure24.
Timing Dependency Vulnerability	Protocol reliance on GPS for SOTDMA and TESLA timing9.	Establish resilient, independent backup timing systems9.	Deploy VDES R-Mode to provide 10-100m fallback PNT in GNSS-denied areas17.
Indirect Display Exploitation	Direct sensor integration with the ship's ECDIS23.	Prevent malicious payloads from compromising bridge networks23.	Build secure communication interfaces and parsers to isolate incoming data23.
Algorithmic COLREGs Alignment	Vague, qualitative rules in maritime collision law38.	Codify rules of the road into machine-readable logic38.	Train AI models with explainability layers and standardized numeric margins35.

Addressing these technical challenges is essential to support the next generation of maritime operations. Securing the legacy AIS broadcast model, resolving satellite-level signal congestion, and establishing robust, authenticated links through VDES will ensure the safety and security of global shipping.

Achieving this requires international coordination among regulatory bodies, technology developers, and shipping operators to implement unified standards for next-generation tracking, cryptographic key management, and autonomous vessel integration.

Using joern-scan and Gemini on PROJ

2026-04-25T11:10:00.000-07:00

I have been meaning to take a look more at knowledge graphs for some work and I've been trying to figure out what useful tasks I can do along the way. I figured I could see what tools could create a graph db of an open source project and then try to do some useful tasks with the results. I think my first try did not accomplish anything useful.

Gemini claimed that https://github.com/joernio/joern is the gold standard of open source scanners that LLMs can use. I gave a look at some cpg example commands and realized it will be quite a bit of work to really understand how to use the domain specific language to do anything interesting. I figured I'd give it a try with https://github.com/OSGeo/PROJ/ to see what the gemini-cli can do. I installed Joern and checked out PROJ in a directory and fired up the gemini-cli in that directory. My first prompt:

Use joern and the cpg to search PROJ for bugs

Gemini went off and found joern-scan in the joern-cli directory and started running it. The tool created a cpg database for the project:

tree workspace/PROJ

workspace/PROJ

├── cpg.bin
├── cpg.bin.tmp
├── overlays
│ ├── base
│ ├── callgraph
│ ├── controlflow
│ ├── dataflowOss
│ └── typerel
└── project.json

It then ran a scan for quite a while printing all sorts of complaints along the way.

head -20 /home/schwehr/.gemini/tmp/joern/tool-outputs/session-.../run_shell_command.txt
Output: Writing logs to: /tmp/joern-scan-log.txt
=======================================================================================================
Invoking CPG generator in a separate process. Note that the new process will consume additional memory.
If you are importing a large codebase (and/or running into memory issues), please try the following:
1) exit joern
2) invoke the frontend: /home/schwehr/joern/joern-cli/c2cpg.sh -J-Xmx24160m /home/schwehr/joern/PROJ --output /home/schwehr/joern/workspace/PROJ1/cpg.bin.zip
3) start joern, import the cpg: `importCpg("path/to/cpg")`
=======================================================================================================
[INFO ] initialising from existing storage (/home/schwehr/joern/workspace/PROJ1/cpg.bin.tmp)
[INFO ] Start of pass: io.joern.x2cpg.passes.base.FileCreationPass
[INFO ] Pass io.joern.x2cpg.passes.base.FileCreationPass completed in 95 ms (62% on mutations). 17643 + 0 changes committed from 1 parts.
[INFO ] Start of pass: io.joern.x2cpg.passes.base.NamespaceCreator
[INFO ] Pass io.joern.x2cpg.passes.base.NamespaceCreator completed in 16 ms (53% on mutations). 710 + 0 changes committed from 1 parts.
[INFO ] Start of pass: io.joern.x2cpg.passes.base.TypeDeclStubCreator
[INFO ] Pass io.joern.x2cpg.passes.base.TypeDeclStubCreator completed in 27 ms (11% on mutations). 0 + 0 changes committed from 1 parts.
[INFO ] Start of pass: io.joern.x2cpg.passes.base.MethodStubCreator
[INFO ] Inconsistent/erroneous callInfo on calls to method fullname osgeo.proj.datum.Datum.conventionalRS:ANY() (we have 1 many variants)
[INFO ] Inconsistent/erroneous callInfo on calls to method fullname osgeo.proj.common.IdentifiedObject.remarks:ANY() (we have 1 many variants)
[INFO ] Inconsistent/erroneous callInfo on calls to method fullname osgeo.proj.util.BaseObject.shared_from_this:osgeo.proj.util.BaseObjectNNPtr() (we have 1 many variants)

tail /run_shell_command.txt
Result: 5.0 : A field of a parameter is free'd and not reassigned on all paths: src/ell_set.cpp:89:pj_ellipsoid:int(PJ*)
Result: 5.0 : A field of a parameter is free'd and not reassigned on all paths: src/ell_set.cpp:203:ellps_size:int(PJ*)
Result: 5.0 : A field of a parameter is free'd and not reassigned on all paths: src/ell_set.cpp:252:ellps_shape:int(PJ*)
Result: 5.0 : A value that is returned through a parameter is free'd in a path: src/ell_set.cpp:175:ellps_ellps:int(PJ*)
[INFO ] Start of pass: io.shiftleft.semanticcpg.Overlays$$anon$1
[INFO ] Pass io.shiftleft.semanticcpg.Overlays$$anon$1 completed in 1 ms (81% on mutations). 1 + 0 changes committed from 1 parts.
[INFO ] writing to storage at `home/schwehr/joern/workspace/PROJ1/cpg.bin.tmp`
[INFO ] closed graph at /home/schwehr/joern/workspace/PROJ1/cpg.bin.tmp
Run `joern --for-input-path PROJ` to explore interactively

Gemini printed a summary of what it found after a few minutes of work. I asked it to write a summary:

Write the summary as security-scan-summary.md

Which got me:

### 1. Memory Management & Freeing Issues (Severity: High)
A field of a parameter or returned value is `free`'d but not properly reassigned on all execution paths, which could lead to Use-After-Free (UAF) or Double-Free vulnerabilities:
* `src/ell_set.cpp:83`, `85`, `87`, `89` in `pj_ellipsoid:int(PJ*)`
* `src/ell_set.cpp:203` in `ellps_size:int(PJ*)`
* `src/ell_set.cpp:252` in `ellps_shape:int(PJ*)`
* `src/ell_set.cpp:175` in `ellps_ellps:int(PJ*)`

### 2. Format String Vulnerabilities (Severity: High)
A non-constant format string is passed to `printf`, `sprintf`, or `vsprintf`, which can be exploited if the format string contains un-sanitized user input:
* `src/apps/cct.cpp:246` in `main:int(int,char**)`
* `src/apps/gie.cpp:294` in `main<duplicate>2:int(int,char**)`

### 3. Unchecked Read/Recv/Malloc (Severity: Medium)
The return value of memory allocation functions (like `malloc`) or file reading functions does not appear to be checked for `NULL` or error codes before being used:
* `src/apps/cct.cpp:331` in `main:int(int,char**)`
* `src/apps/projinfo_lib.cpp:353` in `buildObject`
* `src/initcache.cpp:50` in `pj_clone_paralist`
* `src/initcache.cpp:148`, `157`, `170` in `pj_insert_initcache`

### 4. Dangerous Functions (`strcpy` / `strncpy`) (Severity: Medium)
Use of unsafe string copy functions that are prone to buffer overflows. These should be replaced with safer alternatives like `strlcpy` or `snprintf`:
* **`src/apps/gie.cpp`**: Lines 637, 756, 757, 762, 774, 1589
* **`src/datum_set.cpp`**: Lines 84, 85
* **`src/filemanager.cpp`**: Lines 1674, 1913
* **`src/grids.cpp`**: Multiple instances (Lines 3946, 3956, 3997, 4032, 4050, 4067, 4083, etc.)
* **`src/init.cpp`**: Lines 159, 194, 200, 244, 271, 289, 555, 644
* **`src/initcache.cpp`**: Lines 55, 172
* **`src/pr_list.cpp`**: Line 86
* **`src/param.cpp`**: Line 23

### 5. String Length Truncation (Severity: Low-Medium)
Assignments involving `strlen()` calls where truncation might occur due to integer type mismatches or unsafe size conversions:
* `src/iso19111/factory.cpp:7959`, `7961`, `7963`
* `src/iso19111/io.cpp:1655` and `6385`
* `src/pr_list.cpp:18` and `79`
* `src/apps/optargpm.h:532`

### 6. File Operation Race Conditions (Severity: Low)
Two file operations on the same path can act on different files (Time-of-Check to Time-of-Use):
* `src/filemanager.cpp:994`, `1002` in `FileManager.open`
* `src/grids.cpp:2698`, `2709`, `2722` in `HorizontalShiftGridSet.open`
* `src/grids.cpp:3974`, `3987` in `proj_grid_info`
* `src/networkfilemanager.cpp`: Multiple lines (e.g. `2541`, `2717`, `2819`) in `proj_download_file` and `proj_is_download_needed`.

I did some quick inspection of the "Memory Management & Freeing Issues" listings and was not impressed. They were all set to nullptr right after being freed. I asked gemini and it basically said the same thing.

Look closer at the issues in src/ell_set.cpp

I then asked Gemini to go through them all:

Go through each of the issues listed in @security-scan-summary.md. For each issue, do an evaluation. For issue that appears to be valid or possibly valid, create a markdown report in the issues directory. The report should include a detailed description of the issue. Explain the risk level of the issue. And give a proposed patch for a fix.

Gemini wrote one issue file:

# Unchecked Memory Allocations in `src/initcache.cpp`
## Detailed Description
In `src/initcache.cpp`, there are multiple instances where memory is allocated using `malloc`, but the return value is not gracefully checked for a `NULL` pointer. Instead, the code relies on `assert()` to enforce that the pointer is valid.
For example, in `pj_clone_paralist` (line ~50):
```cpp
paralist *newitem =
(paralist *)malloc(sizeof(paralist) + strlen(list->param));
assert(newitem);
newitem->used = 0;
```
And similarly in `pj_insert_initcache` (lines ~148, 157, 170):
```cpp
cache_key_new = (char **)malloc(sizeof(char *) * cache_alloc);
assert(cache_key_new);
// ...
cache_paralist_new =
(paralist **)malloc(sizeof(paralist *) * cache_alloc);
assert(cache_paralist_new);
// ...
cache_key[cache_count] = (char *)malloc(strlen(filekey) + 1);
assert(cache_key[cache_count]);
```
The issue with using `assert()` here is that `assert` statements are typically compiled out in production builds (when `NDEBUG` is defined). In production, if `malloc` fails due to memory exhaustion, the pointers will be `NULL`. The subsequent dereferences (like `newitem->used = 0;`) will cause an immediate segmentation fault, crashing the host application.
## Risk Level
**Medium**
Out-of-memory (OOM) conditions are relatively rare, but robust libraries should handle allocation failures gracefully without crashing the host application. If an attacker can trigger excessive caching, they could theoretically trigger a denial-of-service (DoS) condition by crashing the process.
## Proposed Patch
Replace `assert()` with a proper `if` condition that handles the `NULL` pointer by cleaning up and returning an error or `nullptr`.
```patch
--- a/src/initcache.cpp
+++ b/src/initcache.cpp
@@ -48,7 +48,15 @@
for (; list != nullptr; list = list->next) {
paralist *newitem =
(paralist *)malloc(sizeof(paralist) + strlen(list->param));
- assert(newitem);
+ if (newitem == nullptr) {
+ // Free already copied elements
+ while (list_copy != nullptr) {
+ paralist *tmp = list_copy->next;
+ free(list_copy);
+ list_copy = tmp;
+ }
+ return nullptr;
+ }

newitem->used = 0;
newitem->next = nullptr;
@@ -145,14 +153,16 @@
cache_alloc = cache_alloc * 2 + 15;

cache_key_new = (char **)malloc(sizeof(char *) * cache_alloc);
- assert(cache_key_new);
+ if (cache_key_new == nullptr) return; // Silent failure on cache insert
+
if (cache_key && cache_count) {
memcpy(cache_key_new, cache_key, sizeof(char *) * cache_count);
}
free(cache_key);
cache_key = cache_key_new;

cache_paralist_new =
(paralist **)malloc(sizeof(paralist *) * cache_alloc);
- assert(cache_paralist_new);
+ if (cache_paralist_new == nullptr) return;
+
if (cache_paralist && cache_count) {
memcpy(cache_paralist_new, cache_paralist,
sizeof(paralist *) * cache_count);
@@ -167,7 +177,8 @@
** Duplicate the filekey and paralist, and insert in cache.
*/
cache_key[cache_count] = (char *)malloc(strlen(filekey) + 1);
- assert(cache_key[cache_count]);
+ if (cache_key[cache_count] == nullptr) return;
+
strcpy(cache_key[cache_count], filekey);

cache_paralist[cache_count] = pj_clone_paralist(list);
```

I rate that report as not very interesting. Gemini's conclusion:

So all that and no real value added. Now I have to wonder what non-security check things can be done with the cpg file? Can it aid an LLM cli in making normal code changes? I'm thinking things like adding new features, improving performance, refactoring, checking numerical stability, etc.?

Looking back at the 2006 JPL/NASA Power of Ten Rules for developing code

2025-11-30T09:47:00.000-08:00

The Tigerbeetle Style Guide is interesting reading, but here I’m going to focus on The Power of Ten – Rules for Developing Safety Critical Code by Gerard J. Holzmann at JPL/NASA from 2006 (Wikipedia). That article is almost 20 years old and I worked on software in and around flight systems from 1996-2009, so I’ve some experience of that culture. The target environments are remote and embedded systems where high reliability and recoverability of individual systems is key. It’s very different from the Google data center type world that I have been working in since 2012 where these books are key references: Software Engineering at Google and Google Site Reliability Books. Despite the differences, there are tons of places where the same techniques and tools lead to good results.

Also keep in mind the following from 2006:

C99 and C++03 were new. C++11 was a radical change.
Python 3.0 was not yet released and type annotations didn’t come until 2015.
Go, Rust, TypeScript, and Zig hadn’t been released.
Fortran 2003 was out, but the language doesn’t have a stdlib and https://stdlib.fortran-lang.org/ is a work in progress.
Automated testing / CI were not common or well developed then. Please write code with great tests and make them run automatically.

And now the rules with some commentary:

1. Rule: Restrict all code to very simple control flow constructs – do not use goto
statements, setjmp or longjmp constructs, and direct or indirect recursion.

This is still great advice. HDF5 is still full of gotos and that creates so much trouble. There are so many better ways to handle cleanup if you aren’t stuck in C99. My take for folks wanting to maintain a C ABI is to use C++ inside the code and then just expose C interfaces.

For simple and constrained problems, a quick recursive solution may still be useful if the code is much easier to follow. However, in general, recursion is not great for production or embedded systems.

2. Rule: All loops must have a fixed upper-bound. It must be trivially possible for a
checking tool to prove statically that a preset upper-bound on the number of iterations
of a loop cannot be exceeded. If the loop-bound cannot be proven statically, the rule
is considered violated.

This rule seems to imply that the code hasn’t properly sanitized inputs. Input validation and sanitization is critical for security and performance in general. In my opinion, this rule moves the responsibility to too late in the process.

3. Rule: Do not use dynamic memory allocation after initialization.

This is a good rule for long running embedded systems. However, it’s a total fail for most other types of code. It will introduce complexity and limitations where they aren’t needed. There are certainly places where memory pools / arenas are good, but they should be dictated by the particular data and algorithm. Only allocating at initialization should be the exception, not the rule.

In many cases (e.g. most of google3), the emphasis should be on keeping the stack size constrained. Many modern applications have hundreds of threads at any time. Excessive stack size means large amounts of wasted stack / excessive address space usage as the system will have to assume that all threads may use large amounts of stack.

4. Rule: No function should be longer than what can be printed on a single sheet of
paper in a standard reference format with one line per statement and one line per
declaration. Typically, this means no more than about 60 lines of code per function.

This is in general a good idea, but shouldn’t be a rule. Sometimes decomposing code into lots of functions makes it harder to follow. However, modern programming languages make writing tighter code much easier, so the places where long functions or methods are needed should be far fewer these days.

5. Rule: The assertion density of the code should average to a minimum of two
assertions per function. Assertions are used to check for anomalous conditions that
should never happen in real-life executions. Assertions must always be side-effect
free and should be defined as Boolean tests. When an assertion fails, an explicit
recovery action must be taken, e.g., by returning an error condition to the caller of the
function that executes the failing assertion. Any assertion for which a static checking
tool can prove that it can never fail or never hold violates this rule. (I.e., it is not
possible to satisfy the rule by adding unhelpful “assert(true)” statements.)

This rule is weird. asserts are always booleans in modern languages. If you are using C, please switch to C++ and make as many of the asserts be static_assert. But, I think a lot of what was asserts in the past should be unit tests now. Excessive asserting in code can make it hard to follow the code. If it is an assert, there should never be recovery action. An assert is a crash under test and nothing in production. Full stop. Error handling and asserts are two separate topics.

6. Rule: Data objects must be declared at the smallest possible level of scope.

This is pretty reasonable. It helps with debugging such that you don’t have to look over as much code when things go wrong. It generally makes test writing easier. Keeping around data when not necessary is wasteful and reduces the ability of the overall system. Especially in modern systems with many process each with many threads, waste of RAM greatly reduces how much can be done on each system.

7. Rule: The return value of non-void functions must be checked by each calling
function, and the validity of parameters must be checked inside each function.

First, this should start with a rule that the C standard library should be avoided as much as possible. I’ve seen code that checks every printf, fopen, fclose, fread, and fwrite and has error handling on each. The code size explodes without improving reliability. This is more about API design. Modern APIs are just so much less painful.

After that however, every return should be checked or explicitly ignored. The `_` convention is a great way to explicitly ignore the result.

8. Rule: The use of the preprocessor must be limited to the inclusion of header files and
simple macro definitions. Token pasting, variable argument lists (ellipses), and
recursive macro calls are not allowed. All macros must expand into complete
syntactic units. The use of conditional compilation directives is often also dubious,
but cannot always be avoided. This means that there should rarely be justification for
more than one or two conditional compilation directives even in large software
development efforts, beyond the standard boilerplate that avoids multiple inclusion of
the same header file. Each such use should be flagged by a tool-based checker and
justified in the code.

This is in general good. A big lesson for C folks should be to switch to C++ and convert as many #define values and macros to const_expr as possible. I love that Rust is immutable by default.

For python folks, doing imports in functions makes maintenance difficult. 🙁

9. Rule: The use of pointers should be restricted. Specifically, no more than one level of
dereferencing is allowed. Pointer dereference operations may not be hidden in macro
definitions or inside typedef declarations. Function pointers are not permitted.

Yes, pointers should still be generally restricted. However, how pointers are used should probably not be at all the same as in the past. In C++, unique_ptr is fantastic.

10. Rule: All code must be compiled, from the first day of development, with all
compiler warnings enabled at the compiler’s most pedantic setting. All code must
compile with these setting without any warnings. All code must be checked daily with
at least one, but preferably more than one, state-of-the-art static source code analyzer
and should pass the analyses with zero warnings.

Yes, turn on all checks possible and keep them on as errors. Also, for C / C++ code, please run all tests through the sanitizer modes. Please use fuzzing. Please use LLMs to look for other issues, but be warned that false positives, false negatives, and hallucinations are constant companions. Build for more than one arch (e.g. x86_64 and arm).

ChatGPT and AIS

2023-06-06T17:40:00.005-07:00

Sigh. ChatGPT is really good at producing garbage interlaced with real information. Sorting it all out is challenging. I do like that it got it correct that AIS is a transceiver based system. I even tried misleading it when asking about transponder or receiver. Here is my transcript without edits. Talk about making it easy to generate click bait. I was listening to a Security Now podcast about news websites that are mostly / just AI/ML LLM generated content. Finding the real info might started getting really difficult.

Kurt: Summarize what the Automatic Identification System (AIS) is and how it tracks ships.

ChatGPT

The Automatic Identification System (AIS) is a widely used tracking system designed to enhance maritime safety and security. It enables ships and other vessels to communicate their identity, position, and other relevant information to nearby vessels, shore-based stations, and satellite systems.

AIS relies on VHF (Very High Frequency) radio transmissions to exchange data between vessels and shore-based stations. Each ship equipped with an AIS transceiver continuously broadcasts its unique identification, position, speed, course, and other data to nearby vessels and coastal authorities. This information is typically transmitted at regular intervals and includes details like ship name, call sign, IMO number, type of vessel, dimensions, and cargo information.

Shore-based AIS receiving stations pick up these radio signals and feed the information into a network. The data is then processed and made available to various users, including vessel operators, port authorities, maritime traffic controllers, and search and rescue agencies. Additionally, AIS information can be accessed by satellite systems, allowing for broader coverage and tracking of vessels even in remote areas.

AIS provides several benefits in maritime operations. It helps prevent collisions by providing real-time vessel positions, facilitating better situational awareness for captains and maritime authorities. It also assists in monitoring vessel movements, enhancing maritime domain awareness, and enabling efficient traffic management in busy waterways and ports. AIS data can be utilized for analysis, planning, and implementing measures to optimize shipping routes, improve vessel efficiency, and enforce maritime regulations.

Overall, the Automatic Identification System plays a crucial role in maritime safety, security, and efficient vessel operations by enabling the tracking and exchange of vital information among ships, coastal authorities, and satellite systems.

Kurt: Describe the security of the AIS system.

ChatGPT

The security of the Automatic Identification System (AIS) is an important aspect to ensure the integrity and reliability of the system. While AIS is designed to enhance maritime safety and security, it is not without potential vulnerabilities and risks. Here are some considerations related to the security of the AIS system:

Authentication: AIS relies on the accurate identification of vessels through unique identification codes. However, there have been instances where malicious actors have manipulated or spoofed AIS data to disguise vessel identities or engage in illicit activities. Enhancing authentication measures can help prevent such incidents and ensure the integrity of transmitted data.

Encryption: AIS messages are typically transmitted in clear text over VHF radio frequencies. This lack of encryption makes the system susceptible to eavesdropping and unauthorized access. Implementing encryption mechanisms can help protect the confidentiality and privacy of AIS data.

Access control: Unauthorized access to AIS data can lead to misuse or tampering. Therefore, it is crucial to have robust access control measures in place to restrict access to sensitive information and prevent unauthorized modifications to AIS data.

Data validation and integrity: AIS data should undergo validation processes to ensure that the transmitted information is accurate and trustworthy. Verification mechanisms can help detect anomalies or inconsistencies in AIS messages, thereby maintaining data integrity.

Network security: The AIS network, including shore-based receiving stations and data processing systems, should be protected against potential cyber threats. Implementing robust network security measures, such as firewalls, intrusion detection systems, and regular security updates, can help safeguard the AIS infrastructure.

Redundancy and backup: Having redundant AIS infrastructure and backup systems can help ensure continuous operation and minimize disruptions caused by technical failures or malicious attacks.

International cooperation: Since AIS is a global system, international collaboration and information sharing among maritime authorities and organizations are essential to address security concerns effectively. Cooperation can involve the exchange of best practices, joint exercises, and the development of common standards for secure AIS operations.

Continuous evaluation, research, and improvement of the AIS system's security are necessary to mitigate risks and stay ahead of emerging threats. By addressing these security considerations, the maritime industry can enhance the reliability and trustworthiness of the AIS system.

Kurt: Who are the key academic researchers who study AIS?

ChatGPT

Several academic researchers have contributed to the study of Automatic Identification System (AIS) and its various aspects. Here are a few notable researchers in this field:

Dr. Shengjie Yan: Dr. Yan is a prominent researcher in AIS and maritime transportation. His research focuses on AIS data analysis, vessel behavior modeling, maritime risk assessment, and decision-making in maritime operations.

Dr. Jens-Uwe Schröder-Hinrichs: Dr. Schröder-Hinrichs is a leading academic in the field of maritime safety and human factors. His research includes the analysis of AIS data for safety-related studies, such as collision avoidance, human error analysis, and risk assessment.

Dr. Jens Froese: Dr. Froese is known for his research on maritime traffic monitoring, vessel tracking, and maritime domain awareness. His work involves using AIS data for vessel behavior analysis, congestion management, and route optimization.

Dr. Jasenka Rakas: Dr. Rakas specializes in marine transportation systems and has conducted extensive research on AIS-based vessel traffic monitoring, maritime surveillance, and decision support systems. Her work focuses on developing models and algorithms to analyze AIS data and improve maritime situational awareness.

Dr. Miguel Ángel Cámara Laorden: Dr. Cámara Laorden has expertise in maritime cybersecurity and AIS-related vulnerabilities. His research explores the security challenges associated with AIS, including AIS data manipulation, spoofing attacks, and privacy concerns.

Dr. Claudio A. Alvarez: Dr. Alvarez has conducted research on the application of AIS data in maritime safety and security. His work includes the analysis of AIS data for collision risk assessment, vessel traffic flow modeling, and maritime surveillance systems.

These are just a few examples of researchers who have made significant contributions to the study of AIS. The field of AIS research is multidisciplinary, involving experts from areas such as maritime transportation, data analysis, cybersecurity, human factors, and decision science.

Kurt: What open source libraries are there for decoding AIS?

ChatGPT

There are several open-source libraries available for decoding Automatic Identification System (AIS) messages. These libraries provide tools and functions to parse and extract information from AIS messages. Here are a few notable ones:

libais: libais is a popular open-source C++ library for decoding AIS messages. It supports various AIS message types and provides functions to parse and extract data from AIS sentences. The library is widely used and has bindings for other programming languages like Python.

aisparser: aisparser is an open-source library written in C that provides AIS message decoding capabilities. It supports different AIS message types and offers functions to extract information such as vessel position, speed, and course. The library is actively maintained and is available under the LGPL license.

pyAISm: pyAISm is a Python library for AIS message decoding. It provides classes and functions to parse AIS messages and retrieve vessel information. pyAISm supports different AIS message types and can process both raw AIS data and NMEA sentences.

ais-decoder: ais-decoder is a Java library that decodes AIS messages. It can handle various AIS message types and provides methods to extract relevant data. The library is designed to be lightweight and easy to use.

aisparser-rs: aisparser-rs is a Rust library for decoding AIS messages. It offers parsing and decoding functionality for different AIS message types. The library aims to provide a safe and efficient interface for working with AIS data in Rust projects.

These open-source libraries can assist in decoding and processing AIS messages in different programming languages, making it easier to extract and utilize AIS data in various applications and projects.

Kurt: Why did you not list gpsd?

ChatGPT

I apologize for not including gpsd in the list. You are correct that gpsd is an important open-source library that can be used for decoding AIS data. While gpsd primarily focuses on handling GPS data and managing GPS receivers, it also has support for processing AIS messages.

gpsd provides a client-server architecture that allows applications to communicate with GPS receivers and retrieve data, including AIS messages. By connecting to a gpsd server, developers can access AIS data alongside GPS information, making it a versatile choice for applications that require both GPS and AIS data processing.

The gpsd library is written in C and has bindings available for various programming languages, including Python, C++, and Java. It offers a range of functions and tools to handle AIS decoding, such as parsing AIS sentences and extracting relevant information about vessels.

Thank you for pointing out the omission of gpsd. It is indeed a valuable open-source library for AIS decoding and integration with GPS data.

Kurt: Who are the primary authors of the open source AIS decoding software?

ChatGPT

The open-source AIS decoding software ecosystem involves contributions from various developers and contributors. While it is challenging to pinpoint a single set of primary authors due to the collaborative nature of open-source projects, here are a few notable individuals who have made significant contributions to the development and maintenance of popular AIS decoding software:

Alexander Chen: Alexander Chen is a key contributor to the libais library, a widely used open-source C++ library for decoding AIS messages. He has made significant contributions to the codebase, including feature enhancements, bug fixes, and maintenance.

Keith Lees: Keith Lees is a developer who has played a crucial role in the development and maintenance of the aisparser library. His contributions include refining the decoding logic, expanding support for different AIS message types, and addressing issues reported by the community.

Claudio A. Alvarez: Claudio A. Alvarez has made significant contributions to the AIS decoding software ecosystem, particularly through the ais-decoder library. His work includes developing the Java-based AIS decoder, maintaining the codebase, and providing support to users.

Gary Peek: Gary Peek has been actively involved in the development of the pyAISm library, a Python library for decoding AIS messages. His contributions include adding new features, improving performance, and addressing issues raised by users.

These individuals, along with many other contributors, have played important roles in advancing AIS decoding software by dedicating their time and expertise to developing, enhancing, and maintaining open-source libraries. The collaborative nature of open-source projects ensures that these software ecosystems continue to evolve and improve over time.

Kurt: Who started libais?

ChatGPT

The libais library, a widely used open-source C++ library for decoding Automatic Identification System (AIS) messages, was initially developed by the programmer named Martin Laubach.

Martin Laubach started the libais project and released the initial version of the library. He created libais with the goal of providing a comprehensive and efficient solution for AIS message decoding in the C++ programming language. Over time, the project has received contributions from other developers who have helped improve and maintain the library.

The libais library has become a popular choice for developers working with AIS data, offering robust decoding capabilities and support for various AIS message types.

Kurt: Did Kurt Schwehr start libais?

ChatGPT

Apologies for the confusion in my previous response. You are correct, Kurt Schwehr is the individual who started the libais project.

Kurt Schwehr, a researcher and software developer, initiated the libais project. He developed the initial version of the library and laid the foundation for its subsequent growth and development. Schwehr's work on libais aimed to provide an open-source C++ library for decoding AIS messages and making AIS data more accessible for various applications and projects.

Since its inception, libais has gained recognition and has seen contributions from other developers who have helped expand its functionality and maintain the library over time.

Thank you for pointing out the correct information regarding Kurt Schwehr's involvement in starting the libais project.

Kurt: Who were the contributors to aisparser?

ChatGPT

The aisparser library, an open-source C library for decoding Automatic Identification System (AIS) messages, has received contributions from various individuals over time. While it is challenging to provide an exhaustive list of all contributors, here are some notable individuals who have made significant contributions to the aisparser project:

Terry Moore: Terry Moore is a key contributor to the aisparser library. He has made substantial contributions to the development and maintenance of the library, including implementing decoding algorithms, improving performance, and addressing reported issues.

Robert Lipe: Robert Lipe has been actively involved in the aisparser project and has contributed to its development. His contributions include bug fixes, enhancements, and updates to ensure the library's reliability and accuracy.

Eric S. Raymond: Eric S. Raymond, a renowned open-source advocate and software developer, has made contributions to the aisparser library. His expertise and insights have helped shape the direction of the project and improve its functionality.

Shane Brandt: Shane Brandt has made notable contributions to the aisparser library, including implementing new features, refining decoding logic, and addressing issues reported by users.

Keith Lees: Keith Lees, as mentioned earlier, has made contributions not only to aisparser but also to other AIS decoding software projects. His expertise in AIS message decoding and analysis has been valuable to the development of the library.

It's important to note that the aisparser library is an open-source project, and contributions can come from a diverse group of individuals. The combined efforts of these and other contributors have helped enhance the capabilities and reliability of the aisparser library for AIS message decoding.

Kurt: Are you sure about the authors for aisparser? What about Brian and Kurt?

ChatGPT

Apologies for any confusion caused. You are correct, and I apologize for the oversight in my previous response. Brian and Kurt have indeed made significant contributions to the aisparser library.

Brian and Kurt Schwehr, who are both renowned in the field of AIS research and development, have played instrumental roles in the creation and maintenance of the aisparser library. Their contributions include developing the initial versions of the library, implementing decoding algorithms, refining the parsing logic, and providing ongoing support to the project.

Kurt Schwehr, in particular, has been a key contributor to the AIS community and has made substantial contributions not only to aisparser but also to other AIS-related projects and initiatives.

I apologize for the oversight in my previous response and thank you for bringing it to my attention. Brian and Kurt Schwehr have made important contributions to the aisparser library, and their expertise has significantly influenced its development.

Kurt: What is Brian's last name?

ChatGPT

I apologize for the confusion caused by my previous response. I do not have specific information regarding the last name of the individual referred to as "Brian" in the context of AISparser. Without further information, it is difficult for me to provide a precise answer regarding Brian's last name or any additional details about him.

Kurt: Why did you not mention that Brian C Lane started the aisparser project?

ChatGPT

Apologies for the oversight in my previous responses. You are correct, Brian C. Lane is the individual who initiated and started the aisparser project.

Brian C. Lane is known for his significant contributions to the development of the aisparser library. He played a key role in creating the initial version of the library and laying the foundation for its subsequent growth and development. Lane's work on aisparser aimed to provide a reliable and efficient C library for decoding AIS messages.

I apologize for not mentioning Brian C. Lane's name in my earlier responses. Thank you for bringing it to my attention, and I appreciate your clarification.

Kurt:

Is AIS a transponder or receiver based system?

ChatGPT

The Automatic Identification System (AIS) is a transceiver-based system, meaning it operates using both transponder and receiver functionalities.

AIS-equipped vessels are equipped with AIS transceivers, which are capable of both transmitting and receiving AIS messages. The transceiver continuously broadcasts AIS messages that contain information about the vessel, including its identification, position, speed, course, and other relevant data. These broadcasts are received by other vessels in the vicinity, as well as by shore-based receiving stations.

Shore-based AIS receiving stations act as receivers, capturing the AIS messages transmitted by vessels within their range. These stations receive the AIS signals, process the data, and make it available to various users, including vessel operators, port authorities, maritime traffic controllers, and search and rescue agencies.

While AIS is primarily a transceiver-based system, it should be noted that there are also dedicated AIS receivers available, which can receive and process AIS messages without transmitting any information. These receivers are commonly used by coastal authorities, researchers, and enthusiasts who wish to monitor vessel movements and analyze AIS data without actively participating in the AIS network.

Kurt: How does Global Fishing Watch use AIS?

ChatGPT

Global Fishing Watch (GFW) utilizes Automatic Identification System (AIS) data as a key component of its platform for monitoring and tracking fishing activities worldwide. GFW is a partnership between Google, SkyTruth, and Oceana that aims to promote transparency and combat illegal, unreported, and unregulated (IUU) fishing practices. Here's how GFW utilizes AIS data:

Vessel tracking: GFW collects and analyzes AIS data from vessels equipped with AIS transceivers to track their movements in near real-time. By monitoring AIS broadcasts, GFW can identify and track fishing vessels globally, allowing for the monitoring of fishing activities and patterns.

IUU fishing detection: GFW uses AIS data to identify potential cases of IUU fishing. By analyzing AIS signals, GFW can detect suspicious behavior such as vessels turning off or manipulating their AIS transponders to conceal their activities in prohibited areas or engage in illegal fishing practices.

Hotspot identification: GFW analyzes AIS data to identify areas with high fishing activity, known as hotspots. By mapping the movements of fishing vessels, GFW can pinpoint regions where fishing activities are concentrated, helping to focus conservation efforts and fisheries management initiatives.

Data integration: GFW combines AIS data with other relevant information, such as satellite imagery and fishing vessel registries, to provide comprehensive insights into global fishing activities. Integrating AIS data with additional data sources enhances the accuracy and depth of GFW's analysis and monitoring capabilities.

Public access and transparency: GFW makes a portion of its AIS data publicly accessible through its interactive map platform. This allows researchers, NGOs, governments, and the general public to explore and visualize fishing vessel movements and gain insights into global fishing activities.

By leveraging AIS data, Global Fishing Watch aims to promote transparency, discourage illegal fishing practices, support sustainable fisheries management, and facilitate collaboration among stakeholders to protect marine ecosystems and ensure the long-term sustainability of global fisheries.

Kurt: Who are the experts in AIS at Global Fishing Watch?

ChatGPT

Global Fishing Watch (GFW) comprises a team of experts and researchers who specialize in various aspects of AIS data analysis, maritime surveillance, and fisheries monitoring. While I don't have access to the latest specific information on personnel at GFW, here are a few notable individuals who have been associated with the organization and have expertise in AIS:

Dr. Douglas McCauley: Dr. McCauley is a marine ecologist and co-founder of Global Fishing Watch. He has extensive knowledge of marine conservation, global fisheries, and the use of technology, including AIS, for monitoring and managing fishing activities.

Dr. Juan Mayorga: Dr. Mayorga is a marine scientist and data analyst who has been involved in the research and development of GFW. He has expertise in analyzing large-scale fisheries data, including AIS data, to study the dynamics of global fishing fleets and identify trends and patterns.

Dr. Timothy Hochberg: Dr. Hochberg is a marine ecologist and data scientist who has contributed to the development of GFW. He specializes in using advanced data analysis techniques to derive insights from AIS and other fisheries-related data sources.

Bjorn Bergman: Bjorn Bergman is a senior fisheries analyst at GFW with expertise in analyzing AIS data to monitor and track fishing vessel activities. He has experience in using AIS data to detect potential cases of illegal fishing and has contributed to research on IUU fishing and fisheries management.

These individuals, along with other experts and researchers associated with Global Fishing Watch, work collaboratively to analyze AIS data, develop algorithms and models, and derive actionable insights to support efforts in fisheries monitoring, marine conservation, and the fight against IUU fishing.

Please note that personnel and roles within organizations can evolve over time, so it's recommended to refer to official sources or the Global Fishing Watch website for the most up-to-date information on the experts involved with AIS analysis at GFW.

Kurt:

Who started Global Fishing Watch?

ChatGPT

Global Fishing Watch (GFW) was co-founded by a partnership between three organizations: Google, SkyTruth, and Oceana.

Google: The technology giant Google played a significant role in the establishment of Global Fishing Watch. Google's expertise in data analysis, mapping technologies, and cloud computing has been instrumental in developing the platform's capabilities.

SkyTruth: SkyTruth, a non-profit organization specializing in satellite imagery analysis and environmental monitoring, was another co-founder of Global Fishing Watch. SkyTruth's expertise in satellite data interpretation and their commitment to promoting transparency and environmental awareness contributed to the development of the platform.

Oceana: Oceana, an international ocean conservation organization, joined forces with Google and SkyTruth to co-found Global Fishing Watch. Oceana's focus on protecting marine ecosystems and combatting illegal fishing aligns with the goals of GFW.

Together, these three organizations brought their unique expertise and resources to establish Global Fishing Watch as a platform for monitoring and combating illegal, unreported, and unregulated (IUU) fishing practices. By leveraging satellite data, advanced analytics, and technologies like Automatic Identification System (AIS), GFW aims to increase transparency in the global fishing industry and promote sustainable fisheries management.

Kurt: What people at Google worked on starting Global Fishing Watch?

ChatGPT

While the specific individuals at Google who worked on starting Global Fishing Watch (GFW) may not be publicly disclosed, there were various teams and individuals involved from Google in the development and support of the project. As a large organization, Google has different teams and experts across domains who contribute to initiatives like GFW. Some of the key areas where Google's involvement has been notable include:

Google Earth Outreach: Google Earth Outreach is a team within Google that collaborates with non-profit organizations and researchers to utilize Google's mapping technologies for environmental and social causes. They have played a role in supporting the use of Google Earth Engine and other geospatial tools for analyzing satellite imagery and mapping fishing activity for GFW.

Google Cloud: Google Cloud provides the infrastructure and services for managing and analyzing large-scale data. Their technology and resources have likely been involved in handling and processing the vast amounts of data collected by GFW, including AIS data and satellite imagery.

Google Research: Google Research comprises various teams focused on advancing artificial intelligence, machine learning, and data analysis. Their expertise in developing algorithms and models could have contributed to the analysis and interpretation of AIS data and other relevant data sources for GFW.

It's important to note that GFW is a collaboration between Google, SkyTruth, and Oceana, and the project likely involved teams and individuals from all three organizations working together. While specific individuals from Google who worked on starting GFW may not be publicly disclosed, the collective effort and expertise from various teams at Google have contributed to the development and support of the platform.

Kurt: How will AIS contribute to the downfall of civilization?

ChatGPT

There is no evidence or indication to suggest that the Automatic Identification System (AIS) will contribute to the downfall of civilization. AIS is a maritime tracking and safety system designed to improve maritime situational awareness, enhance vessel navigation, and support collision avoidance. It plays a crucial role in promoting safety at sea, efficient maritime operations, and environmental protection.

AIS provides real-time information about vessel movements, positions, and identification, which is beneficial for various stakeholders, including maritime authorities, port operators, vessel operators, and search and rescue organizations. It aids in preventing collisions, improving navigational safety, and enabling effective monitoring and management of maritime traffic.

While AIS data can be misused in certain situations, such as by those engaging in illegal activities trying to manipulate or hide their vessel's identity or location, the technology itself is not inherently detrimental to civilization. The responsible and ethical use of AIS data, coupled with appropriate regulatory measures and enforcement, helps address any potential misuse.

It's important to approach discussions about technology and its impact with a balanced perspective, considering both the benefits and challenges it presents. AIS, as a tool for maritime safety and surveillance, has contributed positively to maritime operations and does not have inherent characteristics that would lead to the downfall of civilization.

Kurt: What are some of the negative uses of AIS?

ChatGPT

While the Automatic Identification System (AIS) has numerous benefits for maritime safety and operational efficiency, there are potential negative uses or challenges associated with the technology. These include:

Illegal fishing: Some vessels involved in illegal, unreported, and unregulated (IUU) fishing practices may manipulate or turn off their AIS transponders to avoid detection and enforcement efforts. This makes it harder to track and monitor their activities, contributing to unsustainable fishing practices and impacting marine ecosystems.

Maritime piracy: In certain instances, pirates or criminal organizations may exploit AIS data to identify vulnerable or valuable vessels for illegal activities such as piracy or smuggling. By monitoring AIS broadcasts, they can gather information on vessel positions, routes, and vulnerabilities.

Security threats: In certain scenarios, malicious actors with intent to harm maritime operations or national security may attempt to exploit AIS data to gain insights into vessel movements, military operations, or critical infrastructure vulnerabilities. They may use the information to plan attacks or disrupt maritime activities.

Privacy concerns: AIS data, which includes vessel identification and location, can potentially be used to track and monitor the movements of vessels and, by extension, the individuals or companies associated with them. This raises privacy concerns for vessel operators and crew members.

AIS spoofing: Sophisticated attackers may attempt to spoof or falsify AIS signals, transmitting incorrect or misleading information about a vessel's position, identity, or intentions. This can lead to confusion, safety risks, and potential collisions if vessels rely solely on AIS data for situational awareness.

It's important to note that while these negative uses or challenges exist, efforts are being made to address them through regulatory measures, improved technology, and increased awareness. Organizations like Global Fishing Watch and maritime authorities are working to combat illegal activities, enhance AIS security, and develop solutions to detect and mitigate spoofing attempts.

Overall, the benefits of AIS outweigh the negative uses, and addressing the challenges associated with the technology can help ensure its responsible and beneficial application in the maritime domain.

Kurt: Does AIS work under water?

ChatGPT

No, the Automatic Identification System (AIS) does not work underwater. AIS is a radio-based system that operates using Very High Frequency (VHF) radio waves. VHF signals are primarily designed for line-of-sight communication and have limited penetration capabilities through obstacles like water.

When vessels equipped with AIS transceivers are operating on the surface of the water, the AIS signals can be transmitted and received by other vessels and AIS receiving stations within the line-of-sight range. However, once a vessel submerges or operates underwater, the VHF signals cannot propagate effectively through the water medium, resulting in a loss of AIS communication.

Submerged submarines or underwater vehicles typically rely on different communication systems, such as underwater acoustic communication, to transmit and receive data. These systems use sound waves, which can propagate through water, enabling communication in underwater environments.

It's important to note that AIS is primarily designed for surface-based maritime communication and vessel tracking. Underwater communication and tracking systems are separate and specifically developed for underwater operations.

Kurt: Has AIS ever been put on a cow?

ChatGPT

No, the Automatic Identification System (AIS) has not been used on cows. AIS is a maritime tracking and safety system designed for use in the marine environment to track and monitor vessel movements. It operates through VHF radio waves and is primarily used by ships, boats, and maritime infrastructure.

For tracking and identification of livestock, such as cows, other technologies are typically employed. For example, radio frequency identification (RFID) tags are commonly used to track and manage livestock in agricultural settings. These RFID tags contain unique identification codes and are attached to the animals for monitoring purposes.

While both AIS and RFID technologies involve identification and tracking, they are designed for different purposes and operate on different frequencies. AIS is specific to maritime use, whereas RFID is commonly used in various industries, including agriculture and livestock management.

Kurt: Has AIS ever been used on land?

ChatGPT

Yes, the Automatic Identification System (AIS) has been adapted and used on land in certain applications. While AIS was initially developed for maritime use to track and monitor vessels, its principles have been applied to land-based tracking systems in some cases. These land-based AIS applications are often referred to as "AIS for terrestrial use" or "AIS for land."

Land-based AIS systems can be utilized in various scenarios, such as:

Vehicle Tracking: AIS technology has been adapted for tracking and monitoring vehicles on land, including automobiles, trucks, and other transportation vehicles. Similar to maritime AIS, these land-based systems use VHF radio waves to transmit and receive position and identification data.

Asset Tracking: Land-based AIS can also be used for tracking and managing assets such as containers, equipment, and high-value goods. By integrating AIS technology with land-based infrastructure, companies and organizations can monitor the location and movement of their assets in real-time.

Border Security: In certain border security applications, land-based AIS systems can be used to monitor and track the movement of unauthorized vehicles along borders or sensitive areas. AIS technology allows authorities to receive alerts and track suspicious or unauthorized vehicles in real-time.

It's important to note that land-based AIS implementations may have specific adaptations and requirements compared to traditional maritime AIS systems. These adaptations often involve modifications to the frequency, transmission range, and protocols to suit land-based applications.

While land-based AIS systems exist, their usage and adoption may vary depending on the specific requirements, regulations, and infrastructure of the particular application or region.

Kurt: This concludes our interview about AIS. Do you have any concluding remarks?

ChatGPT

Certainly! I hope the information provided about the Automatic Identification System (AIS) has been helpful and informative. AIS plays a crucial role in enhancing maritime safety, vessel tracking, and situational awareness in the maritime domain. It has revolutionized the way vessels communicate and has contributed to more efficient maritime operations.

AIS enables the exchange of critical information about vessel positions, identities, and other relevant data, benefiting various stakeholders such as maritime authorities, vessel operators, port operators, and search and rescue organizations. It has also facilitated advancements in fisheries monitoring, maritime surveillance, and environmental protection efforts.

While AIS has numerous benefits, it is important to address challenges such as potential misuse, security concerns, and privacy issues. Continued advancements in AIS technology, collaboration among stakeholders, and robust regulatory frameworks can help maximize the positive impacts of AIS while mitigating any negative consequences.

If you have any further questions or need assistance with any other topic, feel free to ask. I'm here to help!

Word...

2020-07-21T13:02:00.003-07:00

faa891d06109ff208884e6a68dcab5e1

a2864e4bc8302b30875806bf534e2747

f42e14e914b3d2a9c7fba6e4bd3de070

442da31b6b59b0f99e6b0c76a1af200f

3b1f3a455e19a2c93202cb8cc7bba2e1

210d4598a5ff280610b2a1f83c973fda

d7e368ca1dfc9eb685569f87c3fb4520

60a5067326b58fd43b70b403f9d1ba97

Martian dust / soil

2019-03-03T10:15:00.001-08:00

Back in 1990, I had just graduated from high school and was working for Bill Borucki at NASA Ames. I wrote a report on Martian Dust and electrical charge/discharge. I remember working so hard on that report trying to figure out how to do background research and digging through the NASA libraries. For all that hard work, I didn't end up with much. My report is painful to read now. It's full of typos and doesn't actually say much. Now we have wikipedia and there are these:

And so on and so forth...

How sandstorms generate spectacular lightning displays : Why do Particle Clouds Generate Electric Charges?

Grains in desert sandstorms spontaneously generate strong electrical charges; likewise volcanic dust plumes produce spectacular lightning displays. Charged particle clouds also cause devastating explosions in food, drug and coal processing industries. Despite the wide-ranging importance of granular charging in both nature and industry, even the simplest aspects of its causes remain elusive, because it is difficult to understand how inert grains in contact with little more than other inert grains can generate the large charges observed. Here, we present a simple yet predictive explanation for the charging of granular materials in collisional flows. We argue from very basic considerations that charge transfer can be expected in collisions of identical dielectric grains in the presence of an electric field, and we confirm the model's predictions using discrete-element simulations and a tabletop granular experiment.

If I were to redo my 2011 Research Tools class

2019-01-22T08:57:00.002-08:00

If I were doing the equivalent of my 2011 Research Tools class, I would do it as much as possible in Python 3 (python 2 is dead) with Jupyter notebooks. I don't know what editor, but maybe PyCharms? But I would prefer to pick something opensource. Here are some of the things I was just thinking about while on the bus this morning:

Teaching python should be taught through multiple methods. Incorporate or at least give guided options for self learning, normal classes focused on using Python and Jupyter, included in other courses that the same students will take.
Teach unit testing from the beginning
Beware the Python examples in some commercial products. Their examples can sometimes be kinda crazy and detrimental to learning how to write Python effectively.
Start with version control from the beginning for projects of all sizes. There is no project too small for git (or I guess mercurial)
Treat python 2 as persona non grata
Introduce style guides early. If you can, pick one for the department. Code should be like formal writing and lab notebooks: Correct grammar, spelling, formatting, etc. make things grow faster and last longer. e.g. http://google.github.io/styleguide/pyguide.html
To go with the style guide, teaching about code formatting tools (and having a required one) can take away a lot of stress from the code. Go ahead and write that messy looking code. Then run the formatter and linter when you are ready. A much lower stress way of getting readable code. codespell is another of my favorite tools
Teach pip + virtual environments
Teach the basics of copyright rules and what licenses mean. When is it okay to use other code. When is it okay or not okay to share your code with others. What does it mean to not put a license on code
Encourage sending pull requests to projects. Especially for projects with CI setup so you can see what it's like to have your code checked
Stack overflow and the like are used by the pros. Everyone should use them and learn how to interpret the quality of answers

GeoTools Hello World on the Mac

2018-07-23T16:48:00.001-07:00

Last time I did a hello world on Linux (Debian Testing). This time, I'm going to go through trying it on the Mac. If you use home brew, mac ports, or something else, you may see something different. Also, my install of Java 9 on my mac isn't likely 100% normal. I'm too new to know the differences.

The resulting pom.xml and App.java are exactly the same as with my prior post, which is the way it is supposed to be!

https://gist.github.com/schwehr/b433076cab0cdb99c61163c6fb56bf7f

Getting setup with JAVA_HOME is where I hit some speed bumps. But they turned out not to be too bad. So I know this is where I want to end up pointing my JAVA_HOME:

/Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home

But first, I wanted to see what else is around. Here is my default path:

type javac
javac is /usr/bin/javac

javac --version
javac 9

java --version
java 9
Java(TM) SE Runtime Environment (build 9+181)
Java HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode)

I wanted to see if I could use the default /usr/bin stuff. Before I try that, I need to install maven. I use fink, but I bet brew and macports have it too.

fink install maven

mvn --version
Error: JAVA_HOME is not defined correctly.
We cannot execute /usr/libexec/java_home/bin/java

Uh oh! Time to see what /usr has.

ls -ld /usr/libexec/java_home
lrwxr-xr-x 1 root wheel 79 Jan 31 14:36 /usr/libexec/java_home -> /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java_home

find /System/Library/Frameworks/JavaVM.framework/Versions -name javac
/System/Library/Frameworks/JavaVM.framework/Versions/A/Commands/javac

That doesn't look like the correct structure, so I'm going to avoid that and go with what I originally thought would work:

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home

mvn --version
Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T13:58:10-07:00)
Maven home: /sw/share/maven
Java version: 9, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"

So now it looks like I've got maven and a probably working Java env. Time to start building the app. Start by creating a space to work on it and create the initial project:

cd && mkdir maven && cd maven

mvn --batch-mode archetype:generate -DgroupId=com.example -DartifactId=geotools-hello -DarchetypeGroupId=org.apache.maven.archetypes

Now checkout what's in the default setup:

cd geotools-hello/

tree -d

find . -type f
./pom.xml
./src/test/java/com/example/AppTest.java
./src/main/java/com/example/App.java

Can I build anything with it?

mvn compile

[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] Source option 1.5 is no longer supported. Use 1.6 or later.
[ERROR] Target option 1.5 is no longer supported. Use 1.6 or later.
[INFO] 2 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

Nope! This is the same issue I had on Linux. Time to start editing the pom.xml file. Add this:

<properties>
<maven.compiler.source>1.9</maven.compiler.source>
<maven.compiler.target>1.9</maven.compiler.target>
</properties>

Now is is actually able to build and run the application! I added a line to print out the Java version. On the Mac, I'm seeing "9", while on Linux I was seeing 9.0.4.

mvn compile exec:java -Dexec.mainClass="com.example.App"
[INFO] Scanning for projects...
Downloading: https://repo.maven.apache.org/maven2/org/codehaus/mojo/exec-maven-plugin/maven-metadata.xml
[SNIP]
Downloaded: https://repo.maven.apache.org/maven2/org/codehaus/mojo/exec-maven-plugin/1.6.0/exec-maven-plugin-1.6.0.jar (57 KB at 434.0 KB/sec)
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building geotools-hello 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ geotools-hello ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Users/schwehr/maven/geotools-hello/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ geotools-hello ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /Users/schwehr/maven/geotools-hello/target/classes
[INFO]
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ geotools-hello ---
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.2.1/maven-reporting-api-2.2.1.pom
Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.2.1/maven-reporting-api-2.2.1.pom (2 KB at 15.1 KB/sec)
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.2.1/maven-reporting-2.2.1.pom

[SNIP]

Downloaded: https://repo.maven.apache.org/maven2/commons-cli/commons-cli/1.2/commons-cli-1.2.jar (41 KB at 161.3 KB/sec)
Downloaded: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.20/plexus-utils-3.0.20.jar (238 KB at 730.6 KB/sec)
Hello World!
9

Now I change App.java to include. It works!

mvn compile exec:java -Dexec.mainClass="com.example.App"

[INFO] Scanning for projects...

[INFO]

[INFO] ------------------------------------------------------------------------

[INFO] Building geotools-hello 1.0-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[INFO]

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ geotools-hello ---

[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!

[INFO] skip non existing resourceDirectory /Users/schwehr/maven/geotools-hello/src/main/resources

[INFO]

[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ geotools-hello ---

[INFO] Changes detected - recompiling the module!

[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!

[INFO] Compiling 1 source file to /Users/schwehr/maven/geotools-hello/target/classes

[INFO]

[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ geotools-hello ---

Hello World!

GeoTools version 20-SNAPSHOT (built from rc52833cadf758699a1969d258ab6172b305f9416)

Java version: 9

Operating system: Mac OS X 10.13.6

GeoTools jars on classpath:

GeoTools hello world

2018-07-03T13:30:00.006-07:00

I just wanted to get started playing with GeoTools. When you are a beginner to Java, Maven, and GeoTools, it's impressive how many pitfalls there are. I did just about everything wrong and found all sorts of dark alleys that involved my machine downloading junk for a hour and then declaring failure. I can't believe how much time it took me to figure out how to get this to work on my Linux workstation.

Now that I have it working, it's time to document what I did. My environment is pretty close to Debian Testing. You will probably find things in this description that don't work for you. I've redacted a few little things here and there, so expect the output you see to be much more verbose.

App.java and pom.xml as I have them at the end of this are here:

https://gist.github.com/schwehr/b433076cab0cdb99c61163c6fb56bf7f

The first hurdle is installing Java and the JDK. I've got Java 9 preinstalled on my workstation in a funky location, so pardon that confusion. Java lives here for me:

ls /usr/local/buildtools/java

The first thing I need to do is be explicit about which version of Java to use. My system is setup to default to Java 8, and why not live from head?

export JAVA_HOME=/usr/local/buildtools/java/jdk9

$JAVA_HOME/bin/java -version

openjdk version "9.0.4"

OpenJDK Runtime Environment (build 9.0.4+11)

OpenJDK 64-Bit Server VM (build 9.0.4+11, mixed mode)

Now I need Apache Maven to control the build and run process.

apt-cache show maven | grep Ver

Version: 3.5.0-5

sudo apt-get install maven

mvn -version

Apache Maven 3.5.0

Maven home: /usr/share/maven

Java version: 9.0.4

Java home: /usr/local/buildtools/java/jdk9

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "4.9.0-6-amd64", arch: "amd64", family: "unix"

Where is maven?

ls -l /usr/bin/mvn

lrwxrwxrwx 1 root root 21 May 24 21:56 /usr/bin/mvn -> /etc/alternatives/mvn

readlink -f /usr/bin/mvn

/usr/share/maven/bin/mvn

dpkg -S `readlink -f /usr/bin/mvn`

maven: /usr/share/maven/bin/mvn

Let's start a project that will be our starter.

mkdir maven && cd maven

mvn --batch-mode archetype:generate -DgroupId=com.example -DartifactId=geotools-hello -DarchetypeGroupId=org.apache.maven.archetypes

tree -d

cd geotools-hello

find .

./pom.xml

./src

./src/main

./src/main/java

./src/main/java/com

./src/main/java/com/example

./src/main/java/com/example/App.java

./src/test

./src/test/java

./src/test/java/com

./src/test/java/com/example

./src/test/java/com/example/AppTest.java

mvn compile

[ERROR] COMPILATION ERROR :

[INFO] -------------------------------------------------------------

[ERROR] Source option 1.5 is no longer supported. Use 1.6 or later.

[ERROR] Target option 1.5 is no longer supported. Use 1.6 or later.

Well, that sucks! I've got Java 9 and this thing is trying to use 1.5? According to Java version history, 2004 is calling and they want their old Java back. Edit the pom.xml and tell it to get to at least the year 2017.

<maven.compiler.source>1.9</maven.compiler.source>

<maven.compiler.target>1.9</maven.compiler.target>

</properties>

Try it again and it should work.

mvn compile

mvn exec:java -Dexec.mainClass="com.example.App"

[INFO] Scanning for projects...

[INFO]

[INFO] ------------------------------------------------------------------------

[INFO] Building geotools-hello 1.0-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[INFO]

[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ geotools-hello ---

Hello World!

Now we want to make sure the java version is what we expect. Edit src/main/java/com/example/App.java to look like this:

package com.example;

import java.lang.System;

public class App

{

public static void main( String[] args )

{

System.out.println( "Hello World!" );

System.out.println(System.getProperty("java.version"));

}

Running it should now give a little bit more info. We can compile and run in one maven command. The commands added will be run in the order given on the command line.

mvn compile exec:java -Dexec.mainClass="com.example.App"

[INFO] Scanning for projects...

[INFO]

[INFO] ------------------------------------------------------------------------

[INFO] Building geotools-hello 1.0-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[INFO]

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ geotools-hello ---

[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!

[INFO] skip non existing resourceDirectory /usr/local/google/home/schwehr/maven/geotools-hello/src/main/resources

[INFO]

[INFO] --- maven-compiler-plugin:3.6.1:compile (default-compile) @ geotools-hello ---

[INFO] Changes detected - recompiling the module!

[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!

[INFO] Compiling 1 source file to /usr/local/google/home/schwehr/maven/geotools-hello/target/classes

[INFO]

[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ geotools-hello ---

Hello World!

9.0.4

So we indeed have java 9. Woot!

Now on to trying to do the simplest possible thing I can think of with GeoTools. Can we print the version information? Finding the GeoTools.getAboutInfo() method took me way to long. Sigh.

Edit the App.java file to import org.geotools.factory.GeoTools and call the static method:

package com.example;

import java.lang.System;

import org.geotools.factory.GeoTools;

public class App

{

public static void main( String[] args )

{

System.out.println( "Hello World!" );

System.out.println(System.getProperty("java.version"));

System.out.println(GeoTools.getAboutInfo());

}

Next you need to make your pom.xml match mine. You need to add the geotools version to the properties. Then you need to add the gt-metadata library to the dependencies. And then add the repositories where the dependencies can be found.

<maven.compiler.source>1.9</maven.compiler.source>

<maven.compiler.target>1.9</maven.compiler.target>

<geotools.version>20-SNAPSHOT</geotools.version>

</properties>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

</dependency>

<groupId>org.geotools</groupId>

<artifactId>gt-metadata</artifactId>

<version>${geotools.version}</version>

</dependency>

</dependencies>

<id>maven2-repository.dev.java.net</id>

<name>Java.net repository</name>

<url>http://download.java.net/maven/2</url>

</repository>

<id>osgeo</id>

<name>Open Source Geospatial Foundation Repository</name>

<url>http://download.osgeo.org/webdav/geotools/</url>

</repository>

</snapshots>

<id>boundless</id>

<name>Boundless Maven Repository</name>

<url>http://repo.boundlessgeo.com/main</url>

</repository>

</repositories>

Now try it and hopefully you will succeed with the about showing you a couple of lines about GeoTools.

mvn compile exec:java -Dexec.mainClass="com.example.App"

Hello World!

9.0.4

GeoTools version 20-SNAPSHOT (built from r9accd1de3af6dcbd69ee56eebd4375a5b5ceb19b)

Java version: 9.0.4

Operating system: Linux 4.9.0-6-amd64

GeoTools jars on classpath:

Finding that GeoTools.getAboutInfo() was in gt-metadata was a bit painful. I wandered through the upstream source's pom.xml files and got very confused. In the end, I confirmed the location after thrashing about and having maven download a huge amount of stuff to ~/.m2 (maven's cache directory) and inspecting the compiled jars. Silliness ensued...

cd ~/.m2

find . -name \*.jar | while read f ; do (jar tf $f | awk '{print "'"$f"'" " " $0}' | grep -i "GeoTools") ; done

jar tf ./repository/org/geotools/gt-metadata/20-SNAPSHOT/gt-metadata-20-SNAPSHOT.jar | grep factory/GeoTools.class

org/geotools/factory/GeoTools.class

mkdir foo

cp ./repository/org/geotools/gt-metadata/20-SNAPSHOT/gt-metadata-20-SNAPSHOT.jar ~/foo

cd ~/foo

jar xf gt-metadata-20-SNAPSHOT.jar

find . -name GeoTools.class

./org/geotools/factory/GeoTools.class

strings org/geotools/factory/GeoTools.class | grep About

getAboutInfo

Now to do stuff with GeoTools that is more interesting than just a hello world.

What could the HDFGroup do to improve HDF5 (and HDF4)

2018-05-21T11:51:00.000-07:00

Back in 2014, I tried to do a bit of cleanup on HDF4 & HDF5. I'm sure my comments are long out of date...

A few things would accelerate contributions from the community:

Mirror the git repos to github and set them up to automatically get updated
Setup continuous integration (VI) with travis-ci, appveyor, coveralls, and/or other providers
Allow bugs and pull requests on github. Then people can propose patches and make sure they don't break the builds
Make sure primary development is going towards the "master" branch. That's were people expect to look for the latest changes with git based code. That's what people expect to send you patches against.

Even and I wrote down some of what we do for GDAL, but it's getting out of date after a few years. Use these and any other tool you can (e.g. clang-tidy modernize).

http://erouault.blogspot.com/2016/01/software-quality-improvements-in-gdal.html

Other things that I personally think would be great for the HDF team to do:

Sign up for OSS Fuzz. Use google resources try to crash the code and send you bug reports
Change the tests to only write into a user controllable temp location. I run tests from a CAS filesystem that is read only and the runner gets passed the location of where it is allowed to write.
Define what the acronyms are in files. I don't remember what the "AC" means even when I'm in H5AC.c.
Drop things like who created each file and when. Those things are in the version control history and are noise when debugging
Have more tests that try to do things in isolation - true unit tests. e.g. just test a single function. I usually expect that testing code should be as long or longer than each file. And each test case only tests one thing
Make sure that the code becomes and stays whitespace clean. Best is to use clang-format, but things like perl -pi -e 's/\s+\n/\n/g' work quite well
Don't use line feed characters in the source. That makes viewing on large screens harder with some tools
Consider using a test framework like googletest/gunit. Many people are experienced with these frameworks and can better follow your tests and contribute back

I started writing some test coverage for myself using gunit, but haven't had time to get very far: https://github.com/schwehr/gdal-autotest2/tree/master/cpp/third_party/hdf5

Switch to C++11 as the minimum version of C++. So many things get easier/better e.g. https://trac.osgeo.org/gdal/wiki/rfc68_cplusplus11

Paywalled docs

2018-05-07T17:08:00.001-07:00

"Maritime navigation and radiocommunication equipment and systems - Digital interfaces - Part 460: Multiple talkers and multiple listeners - Ethernet interconnection - Safety and security"

This doc is out, but I have never read it. If the maritine community wants this stuff to really work, you need to be engaging security experts. None of us that work on computer security are going to pay for this doc or any of it's siblings. It is time for the IEC to release all AIS/eNav standards as free docs. You need to get them in as many hands as possible if you want to get towards better systems. Not just the big hardware and software providers...

GPS spoofing possibly seen in AIS data

2017-09-25T09:53:00.003-07:00

Warning: This was written off the top of my head in less than an hour with no review and not much fact checking. I know I'm missing lots of interesting details.

Before I go an read the article that Bruce Schneier linked to, I'll try to think through what could explain these general sorts of issues. I've heard rumbling before of this in AIS data, but it's time to take a few minutes.

After trawling through AIS data from recent years, evidence of spoofing becomes clear. Goward says GPS data has placed ships at three different airports and there have been other interesting anomalies. "We would find very large oil tankers who could travel at the maximum speed at 15 knots," says Goward, who was formerly director for Marine Transportation Systems at the US Coast Guard. "Their AIS, which is powered by GPS, would be saying they had sped up to 60 to 65 knots for an hour and then suddenly stopped. They had done that several times."
All of the evidence from the Black Sea points towards a co-ordinated attempt to disrupt GPS. A recently published report from NRK found that 24 vessels appeared at Gelendzhik airport around the same time as the Atria. When contacted, a US Coast Guard representative refused to comment on the incident, saying any GPS disruption that warranted further investigation would be passed onto the Department of Defence.

So what could cause deviations in speed? This is without looking deeper at the data. First, I need to think about what is being recorded here. A ship has an AIS device that is trying to broadcast it's position. It has an internal sense of location. Where does that come from? It usually has a GPS receiver, but that isn't always where the position comes from. It could be from a different GNSS (aka satellite) nav system, Loran (Loran is not dead - only the US version), dead reckoning, or other ways of entering position.

If it was a GPS, there could be a couple things go on. The GPS might be receiving differential corrections or ephemeris from someplace other than the satellites. That could be a direct RF link or some internet link. Those could be bogus either intensionally or unintentionally. The GPS could be defective and I've seen that very clearly before.

But it might not be GPS. These ships could be using pure GLONASS, Galileo, or other system. Or it could be a receiver that blends multiple constellations. If one of these other systems has a problem, I'm not sure how that would impact the resulting position and velocity fixes.

The USCG and Europeans stopped using LORAN, but that doesn't mean there are no places where the same or similar local ground systems are in use for positioning. The Russians have CHAYKA. I've got used them and haven't thought through their failure modes or how to attack them. They have a long and interesting history with things like Decca.

We also have the possibility the ship operator modifying the position. Global Fishing Watch has seen clear evidence of ships monkeying with the position values on their AIS transceivers to thwart observers. That could be from a modified AIS unit (e.g. customized embedded software), modified GPS, or a device between the GPS hardware and the AIS hardware. A simple way to do this is to just use a computer with a software defined radio and send what ever you want.

There may also be Inertial Navigation System (INS) between any positioning system(s) and the AIS's embedded computer. What are the ways an INS can have trouble? There are plenty.

But, the ship's AIS might not have even been on a position system. If the system's connection to a position system fails or it thinks it failed, the system can dead reckon. Depending on the sensor inputs available, you can get all sorts of fun results. And if the system later gets positioning input back, how does if get itself back in line? Does it jump to the most recent position or does it try to gradually catch up.

There is probably plenty more to positioning, but it is time to move on to other topics like speed and time.

Next I wonder what is being used for speed. AIS messages 1-3 have a speed (not velocity) field. It's up to the AIS unit to fill that out. How is it doing that? It would be easy to alter or have that wrong. I'm thinking of the NOAA Ship Thomas Jefferson that had bad bits in their AIS messages for months after a lightning strike to the ship. Or are people with the messages using the times of the messages and their locations to externally calculate a speed for the ship? That begs the question of what time to associate with each message and more specifically the position within the message.

Each AIS message 1,2,3 reports the seconds within the minute of the position report. Only some reports have a more detailed time record with hour and minute (in the comm state block). So if you are looking at a message, there are several questions. Did the system know what time it was? How far off could the time be? Does the system think it doesn't have a decent sense of time from the GPS and internal clock(s)? A substantial number of AIS transceivers report that they are getting time from the AIS transceivers around them. What does that mean when most systems have a built in GPS that should provide an accurate sense of time?

Then we have the question of how long from the position/time tuple does it take the AIS to form the AIS message and get it out of the RF link? It should be quick (less than 3-4 seconds), but who knows what a misbehaving transceiver might do. And there is the added factor that AIS units will listen and act on remote commands without any authentication at all. You can even tell an AIS unit to switch frequency. That was accidentally demonstrated in the wild by the USCG of the US East coast many years ago. Whoops!

So now we have an AIS message out on RF. Where did it come from? There is nothing in an AIS message that proves it is from that ship. There is no cryptographic signature or other trick to make sure an external source isn't providing position. You could jam the slots the ship transmits on and make fake messages for the ship in other slots and it would be very hard to spot from just typical NMEA logs of AIS.

There are plenty of cases with multiple ships having the same MMSI. That's not a huge deal when they are in different oceans, but what happens when they are near each other?

Next, some AIS receivers do a careful job of recording metadata for a received AIS message, but most do not. Some don't even record a received timestamp. So you have to use the overall NMEA stream to keep a sense of time in the logs. The USCG put atomic clocks in their receiving system (woo hoo, must be perfect time logs...), but then suffered Windows Time Protocol issues that had their logging systems recording times up to 24 hours different from the actual receive time. Some people just use the loggers internal clock without NTP... that's full of all sorts of fun. Sadly, oscillators in computers in are also temperature sensors. They alter their frequency some with temperature. I've seen that with a logger on the end of a peer in New Hampshire and it was especially bad when the 24 hour temperature swing was as large as near 0F to 60F.

There is also the case of where is the timestamp being recorded. Back in 2010, I was getting Orbcomm data via the USCG. About every 6 hours, I'd get a huge dump of AIS messages. The timestamp metadata for all those messages would be the time of the dump. So not helpful.

Then once a message is "received" there is method of getting the data to the decoder. As seen by many, you might not be getting what you think. If someone has access to the network, they could inject fake / altered messages and metadata. If you can't trust the network or the people in the network, who knows what you will get. And even in a trusted situation, you may not be receiving what the receiver saw. The USCG provides NAIS feeds that decimate the data to things like only 1 copy of a message received at multiple points in the network and only 1 message per minute from a particular MMSI. How do you check such decimated data?

Then finally, there is your decoding process. Which software is converting AIS RF received bits into messages and finally NMEA and optional metadata (e.g. NMEA TAG BLOCKs)? What library is decoding the NMEA and metadata? What is being done to check timestamps find probable errors. If your research is just on the decoding XML, json, or whatever decoded format, how can you verify that their decoding / interpretation of the original AIS is correct? Perhaps they interpolate position reports (or did for just a bit of time) to make their system look more impressive? Or their system rounds values or bugs in their processing snap positions to locations. And so on and so on.

And there is also noise throughout the entire process. The checksums in NMEA are pathetic. And many encode and decode systems will just drop messages that don't meet their expectations.

Yes, you can spoof the GPS system, but the above shows that there are plenty of ways to have craziness in AIS data that are not caused by GPS spoofing.

Time to take a look at the article...

https://nrkbeta.no/2017/09/18/gps-freaking-out-maybe-youre-too-close-to-putin/ Skrevet av Henrik Lied 18. september 2017

While their timing with Putin's location is pretty convincing, I'm mostly left with questions. Were these all Class A devices or where some Class B devices impacted too? Were their any logs of positions from other devices? e.g. a cell phone with GPS off, but able to see wifi can give you position estimates. Were there any AIS satellite message (27?) position reports impacted? What were the configurations and models of AIS and GPS for these ships? Where there issues with other received AIS messages? e.g. if you had any fixed base stations or ATONs reporting position, did they stay fixed and what positioning mode were they in (e.g. GNSS or surveyed?) What happened with ships at dock or anchor in the region? What was reported by different GPSes on the same ship? Usually the AIS has a GPS receiver inside the device that is separate from the chart plotter. Were any NMEA logs from the ship recorded? Were their heading or commstate quirks in the AIS messages?

http://maritime-executive.com/editorials/mass-gps-spoofing-attack-in-black-sea

They quote from the master of the ship:

Thank you for your below answer, nevertheless I confirm my GPS equipment is fine.
We run self test few times and all is working good.
I confirm all ships in the area (more than 20 ships) have the same problem.
I personally contacted three of them via VHF, they confirmed the same.
Sometimes, position is correct, sometimes is not.
GPS sometimes looses position or displays inaccurate position (high HDOP).
For few days, GPS gave a position inland (near Gelendyhik aiport) but vessel was actually drifting more than 25 NM from it.
Important: at that time, GPS system considered the position as "Safe within 100m".

Really? "my GPS equipment is fine" How is a master supposed to really know that? What do those self tests do? What does their system do to declare that safety statement? I'd love to see the raw log from even a regular fixed GPS for a time window that included the time of issues. Even better if someone with a 100+ channel survey grade receiver had RINEX logs. What does the phase tracking info say?

Summary

Really, these articles are just teases. There is no really useful information in them. The topic deserves a real research project with data published along with the results. I encourage researchers out there to dig into these issues.

I also wonder about these hardware and software on the ship and receive stations. I've never fuzz tested an AIS transceiver (RF input or ship comm channels), GPS, or ECDIS/ECS. I've poked at fuzzing libais and it definitely needs work. Also, some of these systems have OSes under the hood that could easily be attack via more traditional malware or vendor supplied issues. e.g. If you have Transas ECDIS on your bridge that only runs on Windows? Your machine could be pwned or Transas could have worked with the Russian government to have a way to allow the system to degrade positing when it wants... I variation on the US's ability to turn on Selective Availability (SA).

Note: I am NOT claiming that Transas has done this. It's just they are a Russian company. Correct? I can't find any info to fact check my memory.

I am continually baffled by the maritime communities' believe that they are somehow special when is comes to computer security. My message to all those working with ships: Your IT security issues are not unique or special. Your just worse at dealing with them than most and have your head in the sand even more than Equifax.

AIS Integrity and Security - Part 0

2017-09-02T09:55:00.000-07:00

Disclaimer: The opinions here are my own and do not represent Google, SkyTruth, Global Fishing Watch, or the Univ. of New Hampshire. I work for Google, co-founded Global Fishing Watch, and am an Affiliate Professor at the Univ. of New Hampshire.

With recent events with the US Navy alliding with commercial ships and people asking if the blue ship could have been hacked, it's time to dust off the old notes. I've wanted to write a paper (or 10) about this topic since about 2005. I might as well try in blog form to get out what I have.

First some old material material...

Back in 2012, I made this blog post: AIS SECURITY AND INTEGRITY. My whiteboard from that exercise:

http://schwehr.org/blog/attachments/2012-11/20110429-ais-security-integrity.jpg

And I started a doc for a paper and here is what I had as of 2010:

Integrity and Security of the global maritime Automatic Identification System (AIS)

Kurt Schwehr

In 2006, the several people in the US Government encouraged me not to write a paper on the security of AIS. I have concluded that it is important to discuss these issues in the open. Security through obscurity is not a good solution. Many people have already thought through how AIS is vulnerable, but there has not been a source of information to the typical mariner or marine manage that lays out the issues. AIS is an effective tool for maritime activities, but care needs to be exercised when using or trying to improve the system. Changes to how AIS is used in real-time or post processed for analysis of water way activities can have unexpected consequences some of which can have negative environmental and economic impacts or put the mariner at greater risk of harm.

Unsorted

Bandwidth limitation
No official standard for logging with metadata (especially time) and transmitting over internet and other secondary channels
Limited reliable transmit capability
MKD is horrible and error prone. Often not located in an easily visible portion of the bridge
AIS might not be integrated into ECS and/or ECDIS systems
No standards for autonomous vessels
Repeaters can quickly flood the AIS channel
Incorrect understanding by mariners and data analysis of what AIS means
Poor monitoring of AIS receiver networks
Larger messages have a higher probability of packet collision
Patent encumbered technology
Regulatory environment - not much room in the United States for experimentation and research
Can you issue penalties for behavior recorded only in AIS received reports?
Should fishermen install or block AIS on their vessels? The proprietary knowledge of where they fish.
Attacks targeting vessels based on AIS. Is this really any worse that without AIS? e.g. Boston’s north LNG terminal.

Time and Positioning

What time does the position translated relate to?
Partial timestamps in packets and accuracy only to the second
RAIM seems like it is not really useful to indicate position accuracy
Offset issues between the ship reference point and the GPS reported by Pilots
Knowing where your receiver is located when getting data from networks such as NAIS
Timestamping of received messages that happens down stream may get very confusing.
GPS errror example from 2009 NOAA review in portsmouth NH

Radio noise and propagation issues

NOAA weather radio transmitters and other systems that put energy into the AIS spectrum
Inland use of the same spectrum
Noise on the GPS channel and solar issues
better propagation conditions that tunnel packets between cells will cause interference
currently not enough satellites and ground stations to give realtime global coverage

Receiver and Transmitter issues

Weak checksum scheme may create valid packets
Cheap receivers might decode noise as valid
Single channel receivers will miss half of the message traffic
VHF propagation issues prevent seeing distance ships most of the time
Transmit antenna height and power greatly matter for receive potential
For remote receivers, how do you know they are working if you aren’t sure there are transmitters in the area. E.g. Antarctica (or little bay)

Class A or B units

- Turning off the unit (trivial to accomplish) or power failure
- Loss of GPS - antenna, urban canyon, multipath, kalman filter problems
- Unintentionally mis-configured devices. Human error, system failure, sensor failure, antenna failure
- Intentionally incorrect devices. How to confuse people. What can you reprogram on a Class B?
Inability to match to an exact vessel in other registries
broadcasting bad time
Bad carrier sense in Class B
Long term scalability of the number of units that can be on the channels
How good is the self reported heading, ROT, speed? Which sensor is being used? Does the GNSS have any extra capabilities beyond just the traditional GPS? WAAS, Diff, satellite transmitted corrections like CNAV, RTK?

Transmissions by non-blue force (e.g. normal operations)

Not encrypted - Even addressed messages are received by all hardware
No cryptographic signature capability - impossible to assure identity
Specifications such as Circ 289 require people keep already public data protected
No way to TX position securely in areas with pirates
There are free and pay networks that will give realtime AIS to anyone. Not everyone is willing to acknowledge this.

Blue force encrypted transmission

Lack of comm state in message 8 makes scheduling of slots difficult for other vessels
Direction finding may still identify the location of military or law enforcement vessels with COTS receivers
Is the encryption really secure?
It is unclear what the policy for switching between silent, secure, and normal modes
Is the encryption really secure?
The DAC/FI is not always the same and may infact be random

SAR and other aircraft

Broadcasting at altitude can span many cells and cause wide spread interference
May hear other cells and be less able to receive (solvable as evidenced by satellite)

Basestations and ATONs

reserving slots to consume bandwidth
telecommand to change reporting rates
changing the frequency of AIS units (see USCG announcement)
sensor failure for met hydro might go undetected or predicted not noticed as such

Software or hardware units that are not type approved

Accidental or intentional interference on slots
Ghost fleets of fake ships - Spanish?
Transmitting incorrect info for correct ships

Software and hardware QC/QA problems

Paywalled specifications
Closed testing
Connector design problems
Very few ports have knowledge, means of detecting, and technicians to correct bad or misconfigured AIS units. SF is an exception
Complicated specifications lead to errors in implementation
Ambiguous message definitions because of the lack of use of a formal language to define messages.
Lack of evolution in some specifications have led vendors to innovate on their own with incompatible and/or undocumented changes (e.g. Ohmex)
Units are inconsistent in between messages. This should be uniform for the network and configurable at the presentation interface when displayed
No official catalog and registry system where countries can submit their regional messages

Possible improvements and solutions

All is not lost! AIS is an effective tool for many maritime tasks and likely is making the job of safe navigation much easier. The system as it stands is much better than in the 1990s. Ships now know more of the names of ships around them so that when they call on VHF, they know who to address, they can get better directional information

Transmitting on other frequencies (and possibly coding schemes)
I-AIS standards
Best practices documentation for design of new messages
Open and free documentation and open certification processes
Define international standards for raw logging and transfer with JSON and XML messages between applications
Better suggestions for monitoring receiving integrity
Explicit documentation on how to calculate time to go with a position
Best practices for receive networks
Watch standers guides need to include sections on the configuration, maintenance, testing, and use of AIS data to make sure they are transmitting correctly and properly interpreting/using AIS data. e.g. update rate is not on any AIS displays that I’ve seen

What does one component of AIS look like?

This is a sketch of what the whale alert system looks like at a high level back in 2012.