What's up on aprs.fi

Embedded maps tracking fixed

2009-11-03T10:52:00.002+02:00

The embedded map broke with the upgrade on Saturday - it stopped tracking the configured station (he_track parameter). That's fixed now. Sorry about that, I need to focus a bit more on testing before releasing...

Statistics after the upgrade

2009-11-01T10:51:00.005+02:00

On 31 Oct I switched from Apache to nginx. Here are some graphs from one frontend. Which would you prefer, based on these graphs?

I only added the temperature plugin yesterday (before the upgrade), but ... the CPU core temperatures did drop by 2°C. Now, does nginx reduce global warming?

Multiple targets on live map (and other small stuff)

2009-11-01T09:29:00.004+02:00

On Saturday I did the planned upgrade. The big new user-visible feature is that you can now track multiple targets on the map. Just type in multiple callsigns separated with a comma: Rautauoma, OH2RDK, or click on 'start tracking' more than once. This code has been running on the beta site for a week now.

There are also some smaller improvements, like symbols being shown in the "other SSIDs" list, callsign links in APRS paths shown in the info balloons, fixed web search, and a JavaScript memory leak workaround for IE.

Oh yeah, referenced digipeaters and igates are now shown when you look up a day in the history. There is little bug in there, though - it shows their current positions, instead of their positions on the selected day. But hey, they rarely move.

There were also some rather large changes under the hood, causing a longer outage than usually. I replaced the Apache web server software with nginx, which allows me to run with HTTP keep-alive again (without running out of memory), and should generally handle larger amounts of users without that much trouble. It also helps with resource management - I'm now running separate web server processes for each service instead of a single Apache process pool for a large number of virtual servers. Now, if one service gets a beating, the others won't be affected so easily. Upgrades and configuration errors in one service won't break others, either. The downside is that name-based vhosts are out of the question.

Today's photo was taken on 11th of August at the summer cottage in Maaninka. EXIF says 9:39:17 PM. It's freezing and very dark in Helsinki today, so I'd rather go back a few months.

PS. I'll be in Bordeaux, France for the next 2 weeks or so.

Upgrades on Friday or Saturday

2009-10-28T19:10:00.003+02:00

I'm planning to upgrade aprs.fi on either Friday evening (30 Oct 2009 around 1500z) or Saturday morning (31 Oct 2009 around 0700z). I'll be reconfiguring the service to use a completely different web server software than before, so there will be an outage of some sort. If everything goes well, it won't be more than a few minutes. There might be some surprises on the way, since I don't have a proper lab setup to run the new configuration with a simulated heavy load.

Beta test: Multiple target tracking on live map

2009-10-26T21:18:00.003+02:00

I've done some rather heavy changes in the map code, which might well break some things. The big visible change is that you can now look up and track multiple targets (up to 20) in the real-time map. I'm mostly worried that it'll slow down things too much, but we'll see...

To try it out, surf to the beta site and enter a comma-separated list of callsigns (OH7LZB-9,OH2RDK,OH2RDS), or click 'start tracking' on multiple targets!

It's also running with a completely different web server software than before. We are still fixing bugs with the new setup, so the beta service might be unavailable at times. Please report any bugs as comments on this post, or send a private email to the address shown on the profile page.

Thank you!

- Hessu and the bug-eating cats

Multiple targets on static maps

2009-10-15T17:13:00.003+03:00

The static map page can now display up to 20 targets at the same time! Just enter the calls, separated by a comma. It works with wildcards, too, letting you select multiple matches simultaneously. For example:

RAUTAUOMA, OH2RDK, oh2rd*

The exact matches (rautauoma and oh2rdk) are preselected, and the wildcard matches are not.

Implementing this on the live map will be slightly more complicated, but unavoidable. :)

PS. Got my OT2 running with my Garmin Nüvi 350 today. Details will be posted later. It's very cool indeed.

Password recovery fixed

2009-10-13T07:29:00.002+03:00

Oops, forgot to mention yesterday that the password recovery functionality was broken since Thursday, and was fixed in yesterday's update.

Repeater antenna installation day

2009-10-12T19:05:00.013+03:00

For a change the photos in this blog post are not completely off-topic - we were installing new antennas on the OH2RCH repeater and igate site on Saturday. The operation was very well planned, all the required bits and pieces were there, the weather was excellent, and the execution was professional (like the components).

An FM broadcast antenna, which was used as the 145 MHz repeater transmitter antenna (together with a circulator and dummy load to handle the SWR), was removed and replaced with a proper 2-element stacked dipole for 145 MHz. Also, a 432 MHz repeater antenna with 4 stacked dipoles for RX and 2 dipoles for TX was added. And an extra 145 MHz dipole on the side of the mast. The 145 MHz RX antennas on the top of the mast, which are shared by the 2M repeater and the APRS igate, were not touched at this time. The new single dipole could be used for a transmitting igate to support 2-way messaging. There's another igate handling that in the area already, though.

The work happened at 70 meters above ground level, right above the second guy wire attachment point. Here is Antti, OH2MNI, attaching the 145 MHz dipole pair to the mast:

And here you can see yours truly, holding the braking rope which keeps the 2M antenna far from the mast while the antenna is being pulled up, so that it doesn't break all those microwave link antennas on the way up. On my right side is one of the guy "wires" - there are 6 of these on three sides of the mast, a total of 18.

And here, on the left side of the mast, are the completed antennas. On the top right side of the mast is a fancy new wind turbine, which is generating 48V DC to help power the commercial network equipment on the site. Green energy, global warming, you know the story. It was installed this summer, and it's already broken - the bearings are making so much noise that the neighbors are more than slightly annoyed.

Thanks go to: OH2LAK (the brains of the operation), OH2FDA, OH3GMZ, OH2MNI, OH2GLG, OH3GMZ and Markku V (the mast pro). More photos by OH2LAK and OH2MNI...

New bad GPS fix detector algorithm installed

2009-10-12T18:02:00.006+03:00

I've just installed my new bad GPS fix detection algorithm. It should detect bad fixes about as well as before, but produce less false positives. The new algorithm looks at the previously received packets instead of the previously accepted packets, and is also slightly adaptive, taking into account more history than just the previous single accepted position.

It should work better for jets (traveling close to 1000 km/h), although during the takeoff acceleration some points might be dropped. After some initial test flights we'll be fixing that. :)

It should also better handle the case where the initial transmission happens to be somewhere far off. It seems like there are a bunch of stations which always wake up in Tokyo and then start transmitting their correct position in the US or Europe. Probably the GPS manufacturer has decided to show it's office location instead of the standard 0/0 lat/lon, and either does not indicate the bad fix in the NMEA sentence, or the tracker ignores that bit of information and transmits the bad position. These should now jump to the correct position after just a couple of packets.

The algorithm also ignores positions which were sent more than 2 hours ago, so if you take an intercontinental flight and start transmitting your new position immediately, it should just work!

Feedback is more than welcome!

Status and comment texts

2009-10-08T08:30:00.006+03:00

As a little early morning exercise I've made aprs.fi show the status message in the info balloon of the current position on the real-time map, and also in the KML. Status message is shown in purple, and the comment text is shown in green.

There has been some confusion about these messages. There are three kinds of "status/comment" messages you can attach to your position. For example, SM4IVE-9 (info) is sending two of them.

The comment text is sent together with the position, in the end of the position packet. Here's an example packet with a comment text of www.sm4ive.com:

SM4IVE-9>APERXQ,WIDE2-2,qAR,SM5NRK-2:!5908.38N/01532.45E>000/000/A=000148www.sm4ive.com

The status message is sent as a separate packet which starts with a '>' character:

SM4IVE-9>APERXQ,WIDE2-2,qAR,LA6TMA-1:>{AT0B4}aprstracker-0.11-16f648

The Mic-E status message is encoded in a mic-e packet using just a few bits, and can contain one of these 8 standard messages: Off duty, En route, In service, Returning, Committed, Special, Priority, Emergency. 7 custom messages (Custom 0 to Custom 6) are also defined. All Mic-E packets contain this status message, and it only consumes a couple of bits in the message, so this requires the least bandwidth from the APRS channel. On the other hand, it can only express the few predefined values.

I would recommend using only the comment text, since it is sent in a single packet together with the position. The status message is sent in a separate packet which increases congestion.

If a status message is required (for example, if the text really needs to be so long that it doesn't fit in the comment text), the status message should not be sent too often. Certainly not as often as the position packet.

In the following photo Armi frowns upon seeing a long, static status packet:

USNG/MGRS grid and static map improvements

2009-09-29T10:04:00.003+03:00

aprs.fi now has US National Grid support, it's selectable from Preferences -> Units and time -> Locator type. It took a surprisingly high amount of work to get this working somewhat right, and it still doesn't work in the polar areas. Each country has it's own national grid system, and many countries have more than one. I don't think I will be implementing any more of these, it would take a LOT of effort to support all of them.

I've improved the static map view a bit. From now on, the map will always be centered on the current location of the tracking target. The 'fit as many points as possible' centering algorithm of Google left a moving target outside the map view way too often, especially in the higher zoom settings. I also reduced the amount of zoom selections to make it quicker to navigate on mobile devices, and made the zoom selector show the currently selected value.

The user account password storage was improved, too. It now uses the RFC 2898 PBKDF2 algorithm to better protect the passwords against an offline dictionary attack. The computationally intensive hash algorithm intentionally makes it slower to perform the cryptographic operations required to check whether the given password is correct for the user. In case someone manages to break in the aprs.fi servers and steal the password database, it'll take a very long time for him to try all the dictionary words against the password hashes. This is basically a safeguard to protect those users who are stupid enough to use the same password on aprs.fi and other services (like their email accounts).

Label colour, messaging capabilities and case sensitivity

2009-08-31T21:46:00.004+03:00

Today I installed an upgrade which contains a few small visual enhancements and a couple of bigger architectural changes under the hood. I hope the latter didn't break anything.

The callsign / object name labels shown on the map now have colours which partially match "APRS symbol attribute" colours in Bob Bruninga's original APRS software:

White: APRS stations which are capable of messaging
Gray: APRS stations which are not capable of messaging
Violet: Objects
Purple: Items

AIS stations have green labels and weather stations have yellow labels, although that conflicts with Bob's spec. I'm running out of colours and I don't have any idea which objects are sent by *you*. Hmm, well, maybe I could compare the source callsign with your aprs.fi login...

aprs.fi now updates the case of the callsign / target name to match exactly what is transmitted, even though the service is still case insensitive. So, if you're transmitting an APRS object named 'Oh7lzb' now, and you switch to a normal APRS station (not object) with the callsign 'OH7LZB', the service will now show the updated upper-case 'OH7LZB'. Before now, it retained the old mixed-case name. Both will refer to the same station and tracking history on aprs.fi, as before.

The path adviser on the info page now warns you about funny path elements like WIDE1-3.

The bigger under-the-hood changes make the web UI marginally quicker, easily expandable, and enable more fine-grained per-feature access control and site configuration.

Small changes (and a small cat)

2009-08-26T20:23:00.007+03:00

During the weekend I replaced the date browsing menu with a new AJAX-based one. The downside is that the new one doesn't tell you how many positions were received during a year, month or day. The upside is that it's much, much faster than the old one. The old one was generated completely when you looked up a station on the map, which in turn required scanning the whole position history of the station, which caused a lot of disk IO, which filled up the caches. The new one just quickly checks whether there are any points during the year, month or day, and does it only when needed.

The new one only shows the days when new points have been inserted. For a fixed station like a digipeater it shows the initial date when the station was first heard by aprs.fi - the old one showed all of the days between the initial packet and the latest packet at the same position.

I also described the date selection menu in the FAQ. It turns out a lot of people don't know about it. Would some people like to try to write a beginner's manual?

The info page now shows the latest telemetry from the station instead of the number of telemetry packets stored. It's much more useful and much quicker to look up.

Most importantly - there is now a GIF animation in a couple of places! Honestly, I never thought I'd bother with such an useless thing which takes such an enormous amount of time to produce. But I accidentally figured out how Gimp does it, so I just had to try it once. Sorry!

In case you were wondering - a new kitten arrived on Saturday evening! The photo is maybe a bit off-topic, but it sure is cuter than the computers and radios.

New symbols, overlays and FAQ

2009-08-18T23:25:00.010+03:00

I'm back from a couple weeks of vacation. Went to the north, visited the northernmost corner of EU and the northernmost corner of mainland Europe. It turned out that there's some APRS coverage in Magerøya and elsewhere in the northern Norway! None of the interesting animals shown in these travel photos tried to eat me.

I've just installed a new build of the software with a couple of features I've implemented yesterday evening and tonight.

I've regenerated the whole map symbol (icon) set from the Revision H APRS symbols provided by Stephen H. Smith, WA8LMF. The symbols shown by aprs.fi should now match those shown by findu.com, UI-View (with Stephen's symbol files) and other programs running with up-to-date symbols.

Thanks to Stephen for maintaining the symbol tables!

The downside is that the new symbols are a bit smaller than the old ones, and they suffer more from the artifacts caused by rotation (for course/heading display).

The upside is that I've also implemented symbol overlays! For example, OH2RDG now has the letter N overlaid on top of the solid green star, both on the map, in Google Earth, and in the info pages.

There's also a short FAQ page.

Arctic Sea position speculation

2009-08-16T12:15:00.004+03:00

Quick recap: A cargo vessel called Arctic Sea (MMSI 215860000) was probably hijacked on July 24th 2009 near the eastern coast of Sweden. This was big news in northern Europe, since hijackings generally happen near the Somali coast, not over here. The ship has a Russian crew of 15, it appears to be owned by a Finnish company, and the owners of that company are of Russian origin. The Finnish media had considerable trouble trying to figure out the true owners, and the owners were really hard to interview. The ship deported from the harbor of Pietarsaari on 22nd of July and carries 6500 tons of Finnish timber, worth of about 1.3 MEUR.

The really odd thing is that the ship didn't go to the nearest Swedish port, but continued towards Africa as if nothing had happened. Very strange indeed. Either the hijackers were still on the ship, or the crew is taking part in the plot.

Latest news (Ransom demanded): BBC, CNN, YLE.

There have been a few questions about AIS positions of Arctic Sea shown on aprs.fi.

Q: Why is the track not shown for the moment of hijacking between Gotland and mainland Sweden?

A: There are no AIS receivers in the area which would directly send AIS reports to aprs.fi. These receivers are run by volunteers (thank you!), and each volunteer chooses where to submit AIS data. There is a receiver in the area, but it is submitting data to MarineTraffic only, and while MarineTraffic and aprs.fi exchange AIS data, aprs.fi is not getting the reports of all of those receivers. The Swedish maritime officials have an AIS receiver network of their own, and they've reported it ran circles and stopped for a while.

Q: Is the position shown for Saturday, 15th of August, valid?

A: Technically, it's possible, but I personally would find it very unlikely. It is easy to fake and it doesn't make any sense for the hijackers to publish their true position like this.

The position report was sent by an anonymous receiver station to MarineTraffic. It is quite easy to send fake data to MarineTraffic over the Internet, since they allow unauthenticated UDP packets containing NMEA strings to be sent to the service. aprs.fi does not allow unauthenticated UDP packets, all AIS submissions are tied to a specific receiving station using a password. Of course any one of those stations could feed us invalid positions, but at least we have some idea of the originator.

If the hijackers (or someone else) wanted to play tricks, they could also go to a shop selling marine radio equipment, buy an AIS transmitter, configure Arctic Sea's MMSI number (and other correct data) in it, give it an incorrect position by crafted NMEA strings (fake GPS receiver on the serial port of the AIS transmitter) and have it transmit the packets on the correct AIS frequency. If they've got the money and motivation to hijack ships with guns and speedboats, they've certainly got the guts to buy or steal AIS equipment. They could also grab the AIS transmitter from Arctic Sea, and take it to another position using a speedboat.

The French navy says there were 3 military vessels in the claimed position on the Bay of Biscay, heading for the Baltic sea, and they didn't see the hijacked ship. And they didn't see it on their radar, either.

The coast guard of Kap Verde claims to have seen the vessel about 800 km off the coast of Cape Verde, which is some 3600 km away from the Bay of Biscay.

In any case, this is starting to become a good plot for a movie.

Mic-E status message supported

2009-07-31T11:00:00.002+03:00

I've just added support for Mic-E message bits. These are sent only in the Mic-E formatted packets, and I've been told it's quite easy to switch between the different messages in the Kenwood TM-D700 / TM-D710 rigs. I don't know how, since I don't own any of these rigs, and they're a bit expensive. If you're working for Kenwood, please consider sending me one for review and compatibility testing, there are over 130 000 potential APRS users visiting this site every month (absolute unique visitors according to Google Analytics). You can find my email address behind the 'complete profile' link on the right. Thank you. :)

The Mic-E status message is encoded in the packet using just a few bits, and can contain one of these 8 standard messages: Off duty, En route, In service, Returning, Committed, Special, Priority, Emergency. 7 custom messages (Custom 0 to Custom 6) are also defined.

The current message is shown on the info page of the sending station, and the info balloon of the current position on the real-time map. Old values of the message bits are currently not stored in the database.

Database upgrade, bug fixes and a feature

2009-07-21T08:48:00.003+03:00

This morning I upgraded the master database engine. The upgrade stopped data updates for a few minutes, but the site was viewable during the upgrade.

I also installed new aprs.fi software on the servers. The new version adds support for the APRS equivalent of the email / newsgroups x-no-archive header, "!x!", which can be placed in the beginning of the comment text of a position packet. It prohibits aprs.fi (and findu) from archiving the position packet in the database. This is quite an old feature in findu, and has not really been documented anywhere (except the APRSSIG archives).

The !x! string is different from using NOGATE or RFONLY in the digipeater path. NOGATE/RFONLY prohibit gating the packet to the APRS-IS completely. !x! does not forbid igates from gating the packet on the Internet, it just requests that the packet is not stored in databases for a long time.

Please note that not all igates or databases support these features. Amateur radio transmissions are defined to be in the public domain (by FCC rules in the US, and by respective legislation in most other countries, and I suppose, by international regulations). Anyone can receive them, and retransmit, publish or store them as they wish.

As Steve Dimse put it on the APRSSIG: "if you do not want your data appearing on the internet the only guaranteed way is not to transmit it!"

I also added a search button to message and raw packets browsing views, and fixed a bug in clickable callsign links in digipeater paths. And did a rather massive source code tree reorganisation, making it a bit more tree-like.

Database upgrades

2009-07-08T21:01:00.003+03:00

Today I upgraded the slave databases to a new major release of the database engine. I'd like to use some of the new features to increase the performance of the system.

The upgrade itself, and the necessary conversions, wouldn't have caused any outages, since I can tell my software to do queries on another database server while upgrading one (in fact, they will automatically fail over to another server if one is down). But, as usual, there was something I overlooked, and for a few minutes about 50% of the /info/ page requests failed and complained about problems with looking up nearby cities. I had to improve the stored procedures a bit to get them working on the new database engine version.

Please, drop a note in the blog comments, if you notice any other problems.

Account system upgrade completed

2009-07-07T08:36:00.003+03:00

The planned upgrade has been now been done. It only took some 15-20 minutes to resolve the few issues that popped up. More details about the changes can be found in the previous blog post.

If you already have an account on aprs.fi for posting AIS data, and you don't have your login password (it's different from the AIS feeding password, and no, you didn't have one before since there was no such thing before today), try logging in with your email address and any bad password, and you'll get to the "forgot my password" path which lets you reset the password to a new one.

As usual, there are quite a few new strings to be translated. From now on, you'll have to sign up for an account and log in to access the translation tool. I hope you don't mind.

Planned upgrade: new account system

2009-07-05T23:05:00.005+03:00

It's been a bit quiet here for some time, since I've been working on some larger changes in the aprs.fi back-end system (and learning how to fly, crash and repair an RC airplane). I'm planning to install the changes on the production servers early on Tuesday or Wednesday morning (7th or 8th of July 2009, around 4:00 UTC). The outage shouldn't be a long one, if all goes well.

One of the visible changes is a new authentication and authorization layer, which is based on a conventional email address + password model with a team-based authorization system. It'll enable some new features and services later (once I get around to implementing those) - in the first phase it just lets you sign up for submitting AIS data without manual work on my part.

There's no need to worry, as a normal user viewing data on aprs.fi, you won't need to sign up for an account. Logging in is completely optional, and I won't encourage or beg everyone and their cat to sign up. In fact, the "login with your nickname to see the map" model will go away - you'll end up directly on the map page when you arrive on the site, and the "most usual" site entrance path will be easier and quicker than before.

I will be adding an "introductory" entrance page for first-time visitors later, so that there's a chance to let them know what this site is all about. But it'll only be shown once for each visitor.

New simultaneous viewers record and related slowness

2009-05-16T19:28:00.004+03:00

Seems like we hit a new high of over 1000 simultaneous map viewers today, mostly thanks to Dayton Hamfest, and a popular live Hamfest video feed with an embedded map. They're giving away freebies.

A couple of components started hitting file descriptor limits, which had last been upgraded over a year ago. Too many simultaneous connections per process. This made the site perform very, very slowly. I quadrupled the limits, and the site started to perform quickly again, I hope that's enough for more than a year to come. Well, I have to admit that it would actually be a nice surprise if the site would be so popular that it wouldn't be enough...

I also fixed a bug in the "first heard" algorithm pointed out by Ian, VK1IAN. The digipeater alias GATE was not treated as a special digipeater alias (like WIDE, RELAY and TRACE are), and an igate which first heard a packet with a GATE in the digi path was not given credit for hearing it first.

Another fix that went in was a filter which takes out complete APRS packets which have somehow made their way to the comment field of another APRS packet. Apparently something is loosing CR LF sequences between packets (could be my code...), which causes packets to go into the comment of the previous one. Before I find the actual bug I've added a filter to strip these off.

Dayton Hamvention APRS view

2009-05-09T13:08:00.003+03:00

Dayton Hamvention is here again on next weekend. I've set up a separate page showing live APRS activity at Hamvention.

Bob Bruninga is expecting to have APRStt running there, too, so if you're visiting Hamvention, you might be able to place yourself on the map using just a DTMF-capable 2m transceiver.

I've also added a new parameter 'he_maptype' in the APRS map embedding interface which allows you to select satellite, physical or hybrid map views in addition to the normal street map.

Date range with weather and telemetry lookup

2009-04-14T13:17:00.006+03:00

I'm on a 2-week sick leave (my gallbladder was removed last week), and have some spare time for enhancements and bug fixes. Rest of the time I'm entertaining Armi the cat, who joined us on Friday.

This morning I added arbitrary date range lookups to the weather and telemetry pages. It allows you to look up a detailed graph between any two dates or times. It also has a little calendar widget for your convenience.

I've also done some XHTML validity fixes for better browser compatibility, switched to a smaller position marker on the static maps, and added some navigation links in the bottom of most pages. I also fixed the bug of date selection menu not appearing when clicking on 'start tracking' on the live map.

Also, the /info/ page of a weather station now shows a brief 'latest weather' report instead of the number of weather reports in database. It's probably much more interesting, and it's much quicker to load, too, since it's almost always cached in memory and doesn't require a potentially large disk read.

Google Maps loading problem fixed

2009-04-01T23:06:00.002+03:00

Google released Google Maps API version 2.151 today, which somehow broke map loading for many users (including myself). I received a number of complaints that the map doesn't load. The Firefox error console gave the error "window.jstiming is undefined".

I've now manually downgraded to Google Maps API version 2.150 which seems to work better.

Sorry for the trouble, and thank you for your patience!

Some database cleanups

2009-03-24T23:46:00.003+02:00

Since the popularity of the site is increasing I need to do some optimizations to keep the site running smoothly.

As announced before, I ran some database restructuring batch jobs during the weekend. The last one was actually started on Monday morning and the execution was completed early Tuesday morning. I hope the slowdowns were not too noticeable, I tried to make them run slowly enough to not interfere with normal use.

I'm currently looking at ways to clean up some unused data from the aprs.fi database. There's quite a lot of automatically or erroneously generated targets stored in the database, most of which are not really looked at by anyone. They just inflate my tables and make the prefix browsing view very cluttered. The performance impact caused by the extra targets isn't really noticeable, but it's just not very nice to have them there.

If you're keeping an eye on the statistics, you might have noticed that today I deleted about 60000 targets. Those were all APRS objects, which had only a single point stored (non-moving), which were last announced over a week ago, and which were announced for less than 12 hours. Almost all of them were automatically generated earthquake and severe weather warning objects, which don't really need to be here any more.

There's also an awful lot of regular APRS targets which have a corrupted source callsign. Most usually one or two characters have been lost from the beginning of the callsign. For example: 1VAJ (J41VAJ), 3GXT-2 (W3GXT-2) and 3TVX-9 (VE3TVX-9). You can find these by looking at the prefix browsing view, picking a strange prefix starting with a number, and looking up the correct original call in the "nearby stations" list of the info page. I wonder how the callsigns get mangled like this, and how could I remove these from the database without accidentally removing some valid data.