Knowing that, it may come as a surprise to discover that RTPS messages are not the ones actually sent on the wire. RTI Connext ® DDS relies on a transport that sends the messages from the host to the remote application over the network. That transport, by default, is UDPv4. So each RTPS message needs to be wrapped as UDP datagrams that the OS (or the IP Stack in general) can send. At the same time, UDP runs on top of an IP stack, which also splits and wraps messages with its own headers.

In short, RTPS messages need to be wrapped inside UDP datagrams first, which are, at the same time, encapsulated into IP fragments. To complicate everything a bit more, UDPv4 datagrams have a maximum size of 64KB while IP packets for a Ethernet LAN has usually a Maximum Transmission Unit (MTU) of 1500 Bytes. This means that an 80KB data sample would need to be split up into 2 UDP datagrams: one of 44 IP fragments and another one of 11 (see image below).

As you can see, your DDS data sample is going to be split several times, and then reassembled before getting to your application.

On one hand, for IP fragments, the IP stack will mark each of the fragments as part of a bigger message and indicate the position (offset) of that fragment in that message. Once all of the fragments arrive at the socket/kernel reception buffer, the IP stack will transfer the reassembled UDP packet to the DDS application.

On the other hand, DDS is prepared to reassemble the split data sample fragments -called DATA_FRAG in the DDS standard and in the Wireshark capture above- into your original data sample before notifying the application that new data is available.

Then, why should you care about fragmentation?

There are mainly three scenarios in which fragmentation could affect your communication using DDS:

1. IP fragments (pink boxes) of incomplete messages are filling the socket/kernel receiver buffer. In some OSes, like Windows systems, there is a maximum number of IP fragments that can be temporarily held in the receiver buffer. That buffer usually has a cleanup timeout and, when that timeout expires, incomplete IP fragments are dropped. If the buffer fills up with fragments that cannot reassemble a UDP packet and/or the cleanup timeout period is too long, the system may end up without free resources to hold new incoming IP fragments. This causes new fragments to be rejected until resources are cleaned up. If fragments in the buffer are cleaned up before a packet can be reassembled, the data sample will not be reassembled and, therefore, will not be delivered to the DDS application and could be considered lost (when using reliable communication). Note that some OSes, like Linux, allow configuring this maximum number of fragments and the cleanup timeout period, while others, such as Windows, does not.
2. An IP fragment is lost in the network. Since DDS handles packets to the UDP level, the reliability unit is the UDP packet and can be repaired by a DDS application, while an IP fragment cannot. Thus, if an IP fragment is lost, the NIC cannot reassemble the UDP packet and the whole DDS data sample will be considered lost.
3. Some switches drop IP packets marked as fragments. This may happen because they are designed to do so or because they want to avoid an IP fragmentation attack.

If any of these scenarios happen, you may not see any communication at all between your DDS applications. You can review this article on how to confirm if IP fragmentation is the cause of your lack of communication and how to fix it.

And what about performance?

Even if your applications communicate, IP fragmentation may still affect your performance.

If an IP fragment is lost, the DDS layer will not receive data from the IP layer. From the point of view of Connext DDS, the RTPS packet containing that fragment is lost, and may require resending it.

When you use best effort delivery, resending will not happen. If you are sending large data with best effort (for streaming, for instance), this will result in a very inefficient usage of your network. For example, if you send a full HD image (1920×1080 pixels), you will be sending around 6 MB of data. If one IP fragment is lost (1500 bytes), you will discard 99.99% of the data that arrived correctly to the receiver.

When you use reliable delivery, DDS will only resend the RTPS fragment for which IP fragment was lost. In our example, the 6 MB image will be split in ~95 DATA_FRAGs of the maximum UDP datagram size (if message_size_max is set to 65KB, the default value). When one IP fragment is lost, 64KB of data will need to be resent, which is only about 1% of the data.

In both cases, if your DDS application is continuously resending data sample fragments due to packet loss, it may end up affecting your overall performance (both throughput and latency). When using Connext DDS, there are several QoS considerations to mitigate this performance loss. To learn more about the specifics of these configurations, be sure to read this Knowledge Base post over on the RTI Community portal.

Six years ago I crossed the doors of RTI’s headquarters in California for the first time. Another three people and I were chosen to be part of an important expansion of the company in Europe- a new development center in Spain. But before that, we stayed six months in HQ to learn not only about RTI DDS products, but also about processes and, most importantly, RTI’s culture.

I have to admit that when I arrived, I wasn’t sure what I was going to find- if I was going to fit in the culture or if I was going to enjoy those months there. But I really did! And when the time to return to Spain arrived, it was difficult to leave. I met great people that are not only my work colleagues, but also my friends. I also found them to be great managers and leaders whose paths I would follow in the coming years.

We returned back with a challenge: To start a development center in a small remote office in Granada, Spain. Would a company that provides solutions for connecting distributed systems be able to work in distributed offices? At this point, I guess you have already figured out that the answer is yes. The RTI Spain team started out as a small family that had always been a part of the engineering team in the California office. We participated in all of the team meetings and step by step we all learned how to work around the distance in miles as well as the time difference. Both teams (California and Spain) adapted their work schedules to take advantage of the overlapping business hours. Conference calls, online chats and emails were our daily tools to stay coordinated and in contact. This way, we made sure that we continued working as one single team in two separate locations.

Then, the RTI Spain office started to grow and we were no longer able to fit in our initial office so we moved to a larger space. From there, we kept growing and then had to expand our new office. With this expansion, we needed to evolve our team to include new roles, such as operations, services and sales. We were no longer just a development center, we became a true extension of our peers in California building the same products for the same customers.  I know this is not very common within distributed companies, but it’s really working for us!

Over these past six years, I have been able to see people grow within the company- engineers who develop great products, who give excellent support to our EMEA customers, or who have become great leaders as R&D, Services or Field Applications Engineers on this side of the world. I’m really happy to share my work days with these great, talented and passionate people.

As with other senior employees, I have also gotten the opportunity to grow and  become a manager of the new engineers, helping them to become a part of the team. For me, it is really important to hear how these new hires feel when they join us. I’m glad to hear things like: “You guys have a great quality product!”, or “I love working in this company, my life is better now.” But I think that what people like most is how we are all a part of the same RTI family, #1RTI. Headquarters, the Spain office and our remote employees all work together as a team where everybody knows each other and can rely on everybody else’s work. Once a year we all get together at HQ to celebrate our work and the amazing applications that our customers are building, which unites the team even further.

We also keep in close contact with the University of Granada, as many of us studied there. We present to the students on the newest IIoT technologies, as well as how a Silicon Valley company works in Granada. We participate as promoters of the Tech Challenge organized by the University of Granada and several of our current employees started as interns with their internship program.

But today, I’m not writing this post only because of these past 6 years at RTI. I’m also celebrating that we are hiring our 30th employee in the Granada office, and guess what? We’ll be moving to an even bigger office! Stay tuned! And, of course, we are hiring! Come and join us!

Recently, I got the opportunity to present to a group called, “Women Who Code (WWC),” a society for women in software engineering careers. I decided to start with a story that I made up for my daughter when she was little. The kid’s version is contained at the bottom of this post; if you have time, please check it out. But, since you’re likely a harried grown up, here’s the Reader’s Digest version: 

Once upon a time, there was a beautiful princess named Julina living in a castle on a lake. High on a nearby mountain lived a valiant prince who admired the princess from afar. But she ignored him; she was too busy collecting things and playing in her room.

Then one day a dragon comes to the valley and attacks the town by the lake. The prince sees his chance to prove himself to beautiful Julina. So, he grabs his sword and armor, and rushes down to the lake to battle the dragon.

You know the drill: fierce dragon against great swordsman. There’s a clash of fire and scales and dagger-like teeth, against training, agility and determination. In the end, the prince sinks his sword deep into the dragon’s soft underbelly. Dragons always have soft underbellies.

Unfortunately, the dragon just pulls out the sword, scoops up the prince and takes him up to his lair for lunch.

But, before the dragon could eat Sir Loin of Beef, he’s distracted by a loud noise from the lake. The dragon goes down to investigate and finds Julina zooming across the water on a roaring machine!

You see, it turns out that Julina is an engineer. She wasn’t collecting and playing in her room, she was inventing a jet ski with a water cannon. Well, the dragon attacks, doesn’t do so well, and Julina saves the day and the prince. Everyone lives happily ever after. The End.

My daughter loved that story.

My daughter starts her PhD In Aerospace Engineering at Stanford this fall.

I did a bit of research on RTI’s amazing women for this talk as well. I found that for each of the last two years, our highest-paid employee was a woman.  Fully 50% of our engineering development managers are women. These are great stats. They were well received by the WWC.  But they missed my point.

My point was not that women can succeed. That should be obvious. The “Women Who Code” should not even have to exist. Of course women can code. Of course women can manage engineers. Of course women can make more money. Of course women can get engineering PhDs at Stanford. These should not be twists in our story. These should be boring normality.

Then I realized these are hardly the only twists in RTI’s story. We find things that don’t make sense everywhere we look. Some examples:

In 2015, the Institute of Medicine reported that hospital error is the third leading cause of death in the United States after heart disease and cancer. Why? Hospitals perform many complex operations and procedures and use hundreds of sophisticated instruments and devices, like ventilators and respirators and ECG monitors and oximeters. However, today’s instruments are standalone devices; they don’t even know that they’re connected to the same patient. So, they can’t check on each other. With high patient loads, useless alarms and manual checklists, harried care teams make mistakes. And that kills between 220,000 and 440,000 people a year.

RTI software helps. Our customers like Harvard Medical School and GE Healthcare are connecting smart algorithms to networks of devices. Connected devices can work together to monitor patients much more intelligently, preventing thousands and thousands of mistakes. The distributed intelligent system of devices can join and help the care team.

Isolated systems also plague the power industry. Today’s grid consists of huge generators controlled by a central office. This design can’t deal with “distributed energy resources” like solar and wind. One main reason: big generators can’t react fast enough to efficiently use the variable power flow from these renewable sources.

Connectivity also helps here. For instance, RTI runs the largest hydropower plants in North America.  Hydropower is an ideal complement to solar and wind: it’s faster responding, more predictable and more reliable than other “base capacity” generators. RTI also controls large wind farms, such as Siemens Wind Power’s most advanced wind turbines, with thousands deployed worldwide. And, we are a principal in the leading new grid standard to replace central control with distributed field networks for solar, wind and batteries. Networked together, renewable sources can combine to provide the consistent, reliable power we need to meaningfully reduce our dependency on fossil fuels.

But, my favorite twist in the technology story is autonomous drive cars, or “carbots.” I started my career crashing cars doing impact testing. Back then, 45,000 people a year were dying on our highways. Last year, 35,000 died. That’s better, but it’s hardly good enough. Carbots will finally replace the weakest safety system: drivers cause 94% of all fatal accidents. We originally developed our “databus” technology for autonomy (flying robots). Now, it’s great for intelligent carbots. 

In the end, RTI fights the dragons of convention every day. Convention says 400,000 people will die unnecessarily in hospitals every year. Convention says we have to burn things for power. Convention says it’s OK to sit in traffic for countless hours and kill tens of thousands of people on our roads every year.

Convention also says most engineering managers are men. Convention says women don’t earn as much. Convention says women rarely get PhDs in engineering. And convention says women don’t code.

Convention is wrong. It’s not always fast and it’s not ever easy, but convention is changing.  I’m truly grateful to the Women Who Code for inspiring women to seek careers changing these outdated conventions. And for inspiring me to brag about RTI’s amazing women, and maybe even attract more. We certainly need them; our women are not paid well or managing teams because we preferentially pay or promote.  They are just damn good.

They prove women can excel in any career.

And little girls can grow up to slay dragons.

The Princess and The Dragon

Once upon a time, there was a beautiful princess named Julina. Julina lived in a big castle on a river. This wasn’t an ordinary castle on the shores of a river, Julina’s castle was built on a bridge over the river. The river went right through the bottom of the castle. Julina’s room was on the second floor; she had a big window with a view up the river. But, her window was always closed and covered with shutters.

Julina collected everything. Well, maybe not everything, but lots of things. She collected old shields, swords, and armor from the knights. When the farmers brought their crops to the market outside the castle, they would bring extra corn husks for Julina. She got wheels, rods and metal from the blacksmith. She even went to the kitchen and got grease, candles, old pans and anything else she could find.

She took all these things up to her room. She didn’t tell anyone what was going on in there, but they heard banging, and pounding, and whirring, and scraping, and scratching, and clanking, and clunking, and swishing, and bubbling, and even some screeeeeeeeeeeeeeeching.

Julina liked to play in her room, but Julina’s favorite thing of all was her pretty white dress. Every afternoon, Julina would put on her dress and walk around the castle and the garden. Her dress went all the way down to the ground, with poofy sleeves and pretty flowers sewn down the front, and a hundred buttons down the back. She loved her dress.

Down the river a little bit from Julina’s castle, the river went through a canyon between two big mountains. At the top of one of the mountains there was another castle as big as Julina’s. In that castle, there was a Prince named Phillip — the biggest, strongest, bravest knight in all the kingdom. Prince Phillip was a great hunter and a great swordsman. He could ride his horse better than any knight. And he could even cook.

Every afternoon, Prince Phillip would look through his telescope at the castle built over the river below. He saw Julina in her beautiful white dress. She was soooo pretty. He really wanted to meet her, somehow.

One day, all of a sudden, the earth shook! There was a terrible noise, like a train crashing into a tornado and rolling down a mountain over a pile of rocks and squirrels. The mountain across the river from Phillip’s castle cracked! Tons of rock came crashing down, splashing right into the river. It blocked the river and the river started to turn into a lake.

The lake was growing and growing, getting big enough to cover the gardens around Julina’s castle.  Everyone was scared that the castle would be underwater soon. But suddenly, a screech came down from the top of the broken mountain! Everyone looked up, and there was a HUGE dragon coming out of the mountain! The dragon had bright golden scales, a huge sharp tail, and big green wings. Its huge mouth was filled with crystal dagger teeth, so clear you could see right through them and so sharp you could see the sun glint from every one. And when it screeched, it breathed a huge blast of fire!

The dragon swooped down from the mountain and wheeled over the lake that was now deep enough to cover the bottom floor of Julina’s castle. Prince Phillip grabbed his sword and jumped on his horse. He charged down the mountain to save Julina!

When he got to the castle, the dragon attacked. It swooped down, with its claws out and fire shooting out. Prince Phillip blocked the fire with his shield and stabbed the dragon with his sword! But the sword just bounced off the golden scales.

The dragon came back around, zooming low and fast. Prince Phillip dove on the ground and held up his sword. When the dragon flew over, he slashed at the dragon’s throat!  But the sword just bounced off the golden scales again. But, when it flew over, Phillip noticed that the dragon had a dark underbelly, with no golden scales.

The dragon went way up high in the sky.  It dove straight down at Phillip at 100 mph, screaming all the way. Fire shot out. The dragon’s claws aimed right at Phillip. The crystal dagger teeth shone like a thousand suns. Phillip knelt down on the ground, his sword out, aiming for the soft underbelly. The dragon swooped. Phillip stabbed!  The sword went right into the dragon’s underbelly!

But, then the dragon ripped the sword out with one claw and grabbed Phillip with the other one! It flapped its great wings and flew up to the top of the broken mountain. It dropped Phillip on a rock and opened its great mouth to eat him for its supper! Everyone below heard Phillip yell: “Help! Help!” Phillip was in trouble.

Just then a roar arose from below. Brrriiinnnnnnndddd-din-din! Julina’s shutters flew open, broken glass blasted out and Julina came roaring out on a machine on the water!  You see, Julina was an engineer, and she had invented a jet ski, and built a motor and turned the corn husks into fuel. Her jet ski made a huge roar unlike anything ever heard in the kingdom as she shot across the lake.

The dragon heard the noise. It took off and swooped down across the lake, straight for the jet ski. Julina turned towards the dragon!  The dragon shot out a stream of fire. Julina turned her jet ski just in time, splashing the dragon with water.

That made the dragon even madder. He swooped way up high over Julina’s castle, and started a screaming dive. Fire shot out. The dragon’s claws aimed right at Julina. The crystal dagger teeth shone like a thousand suns. Julina turned her jet ski straight at the dragon and hit the gas hard. Brinnnnnddddd-din-din! Brinnnddd-din-din!

The dragon came screaming in low across the lake, shooting a huge, hot flame. Then, Julina pulled a lever on her jet ski, and a long black tube came up. It curved towards the dragon. It said F—P—X! Fish Poop Express!!!!!!! The bottom of the tube went deep into the lake. It sucked up water and muck and fish poop, and shot it straight at the dragon!

The water and muck and fish poop hit the dragon! It put out its fire and smacked it right in the face! It couldn’t see! It shot right over Julina, swerved a bit left, a bit right, up a bit, down a bit, and right into the rocks that were blocking the river! Scales went everywhere!  The crystal dagger teeth shattered into a million zillion pieces, flying miles away in every direction. The rocks flew away, breaking the dam, and all the water went rushing down the river.

The castle was saved! The dragon was dead! Phillip didn’t get eaten! Everyone at the castle on the river yelled “YAY!” Everyone at Phillip’s castle yelled “YAY!”

Phillip climbed down from the mountain. He had to talk to Julina, and thank her for saving his life. She was the most amazing girl he’d ever seen. On his climb down, he found a piece of dragon’s tooth, a crystal-clear piece harder than any rock. He found a piece of golden scale. He shaped the scale into a ring and put the dragon’s tooth on top. Then he went to ask Julina to marry him.

She said yes, and he gave her the ring. She wore her pretty white dress to the wedding and they lived happily ever after.

And to this day, when people get married, they get a ring made of dragon scales with a piece of dragon tooth on it. The tooth is crystal clear and harder than any rock. When the sun shines on it, you can see the flash of fire that came from the dragon’s mouth. And the bride wears a pretty white dress.

The End.

Postscript: I used to ask my kids to give me three or four things, and I would tell them a story that included them all. This story was the result of the set: a princess, a dragon, a jet ski and fish poop.

© Stan Schneider

While developing an application using RTI Connext, DDS users may run into a situation where the publisher and subscriber are not communicating. In these situations, we will usually get the question: how can I figure out what the issue is and how to solve it?

There are several tools and features that can help you debug your DDS issues:

1 – Look for log warnings or error messages.

You can enable log messages in your DDS applications. There are different verbosity levels depending on the kind of messages that you would like to see. Usually, we recommend that customers use the  “Warning” verbosity level to see the most common problems. If you need further details in a certain situation, you can increase the verbosity level to “All.” However, be aware that setting the highest verbosity level will result in the output of lots of messages.

You can set logging via XML or code. To set it via XML – with warning verbosity and sending the output to a file – you need to add the following settings to your XML QoS profile:

         <participant_factory_qos>

           <logging>

             <output_file>log.txt</output_file>

             <verbosity>WARNING</verbosity>

             <print_format>TIMESTAMPED</print_format>

           </logging>

         </participant_factory_qos>

To enable logging via code, add the following lines to your application main function (C++ example):

         NDDSConfigLogger::get_instance()->

         set_verbosity_by_category(NDDS_CONFIG_LOG_CATEGORY_ALL,

         NDDS_CONFIG_LOG_VERBOSITY_WARNING);

Note that this code needs to be added before you call any other operations on the RTI Connext DDS API to ensure that any output goes to the file instead of the standard output.

If you need to limit the size of the log output file, follow the steps in this KB.

RTI Log Parser

Once you have the log messages, you can parse them using our RTI Log Parser. This is a command-line tool that processes and enhances Connext DDS log messages, making it easier to debug applications.

Let’s take the following log messages as an example:

COMMENDSrReaderService_onSubmessage:[1484571004,3507893055] reader oid 0x80000004 received DATA of sn(0000000000,00001346), vSn(0000000000,00001346) from writer 0xffffffff.49e6.1.80000003

COMMENDSrReaderService_onSubmessage: accepted sn(0000000000,00001346), dataRcvd.lead(0,1346), nextRelSn(0,1347), reservedCount(1)

COMMENDSrReaderService_onSubmessage:[1484571004,3577888137] reader oid 0x80000004 received DATA of sn(0000000000,00001348), vSn(0000000000,00001348) from writer 0xffffffff.49e6.1.80000003

COMMENDSrReaderService_onSubmessage: accepted sn(0000000000,00001348), dataRcvd.lead(0,1347), nextRelSn(0,1347), reservedCount(2)

MIGInterpreter_parse:HEARTBEAT from 0xffffffff,0X49E6

COMMENDSrReaderService_onSubmessage:[1484571004,3578261799] reader oid 0x80000004 received HB for sn (0000000000,00001345)-(0000000000,00001348), epoch(347) from writer 0xffffffff.49e6.1.80000003

COMMENDSrReaderService_sendAckNacks:[1484571004,3578261799] reader oid 0x80000004 sent NACK of bitmap lead(0000000000,00001347), bitcount(2), epoch(353) to writer 0xffffffff.49e6.1.80000003

COMMENDSrReaderService_onSubmessage:[1484571004,3578751425] reader oid 0x80000004 received DATA of sn(0000000000,00001347), vSn(0000000000,00001347) from writer 0xffffffff.49e6.1.80000003

COMMENDSrReaderService_onSubmessage: accepted sn(0000000000,00001347), dataRcvd.lead(0,1347), nextRelSn(0,1349), reservedCount(-1)

MIGInterpreter_parse:HEARTBEAT from 0xffffffff,0X49E6

COMMENDSrReaderService_onSubmessage:[1484571004,3579107907] reader oid 0x80000004 received HB for sn (0000000000,00001347)-(0000000000,00001348), epoch(348) from writer 0xffffffff.49e6.1.80000003

COMMENDSrReaderService_onSubmessage:[1484571004,3579107907] reader oid 0x80000004 sent ACK of bitmap lead(0000000000,00001349), bitcount(0), epoch(354) to writer 0xffffffff.49e6.1.80000003

This is how the output will look like after being parsed by RTI Log Parser:

2017-01-16T12:50:03.919122 |  —>   |     H2.A2.P1     |   R-K_800000   | Received DATA [1346] from writer W-K_800000 (reliable)

2017-01-16T12:50:03.919122 |      |                     |             | *Warning: Missing packet for H2.A2.P1.W-K_800000 to R-K_800000*

2017-01-16T12:50:03.919122 |  —>   |     H2.A2.P1     |   R-K_800000   | Received DATA [1348] from writer W-K_800000 (reliable)

2017-01-16T12:50:03.919122 |  —>   |     H2.A2.P1     |   R-K_800000   | Received HB [347] from writer W-K_800000 for samples in [1345, 1348]

2017-01-16T12:50:03.919122 |   <—  |     H2.A2.P1     |   R-K_800000   | Sent NACK [353] to writer W-K_800000 for 1347 count 2

2017-01-16T12:50:03.919122 |  —>   |     H2.A2.P1     |   R-K_800000   | Received DATA [1347] from writer W-K_800000 (reliable)

2017-01-16T12:50:03.919122 |  —>   |     H2.A2.P1     |   R-K_800000   | Received HB [348] from writer W-K_800000 for samples in [1347, 1348]

2017-01-16T12:50:03.919122 |   <—  |     H2.A2.P1     |   R-K_800000   | Sent ACK [354] to writer W-K_800000 for 1349 count 0

For more information about how to configure DDS logging, please refer to the User’s Manual.

2 – Check status of DDS entities

1. Use RTI Admin Console
   

   Admin Console is a GUI tool that includes many features to easily debug your DDS applications. For example, with Admin Console you can:

   • Monitor log messages and view your QoS, as explained in this blog post.
     
   • Automatically perform match analysis of your DDS entities. The match analysis shows if there are any QoS incompatibilities between your DDS entities that prevent them from communicating
     
   • Visualize your data.                                                                                                            
2. Enable Monitoring Libraries
   

   RTI Monitoring Library enables RTI Connext DDS applications to provide monitoring data. The monitoring data can be visualized with RTI Monitor, a separate GUI application that can run on the same host as Monitoring Library or on a different host. Monitoring Library periodically queries the status of all Connext DDS entities and brings up all the entity creation/deletions and QoS changes, among other statistics.

   You can enable the Monitoring Libraries by setting values in the DomainParticipant’s PropertyQosPolicy (via code or through an XML QoS profile). The specific instructions to enable Monitoring Library can be found in the User’s Manual. If you want to configure the behavior and statistics obtained by Monitoring Library, you can refer to this section of the User’s Manual.

3. Check entity status

4. DDS entities have different statuses associated with them. The statuses give information about important changes in the entity state. For example, there are statuses that are triggered when a sample is lost or rejected, and also a status to detect the liveliness loss of an entity. An entity status can be retrieved by:

   • Directly checking the status with the appropriate get API
     
   • Using listeners
     
   • Using StatusConditions and WaitSets
     

    This Community Best Practice article explains the difference between these methods.

    You can find the different statutes associated with an entity in the User’s Manual and in the online API documentation, for example for DataWriters and DataReaders.

3 – Check what’s happening on the wire

Looking at the traffic generated by your applications can give you a lot of information about what is going on. To do this you can use Wireshark.

• You can find documentation about how to use Wireshark with RTPS messages in our documentation section “Wireshark and RTPS.”
• Wireshark cannot capture traffic from the loopback interface on Windows. You can use RawCap instead to capture this traffic on Windows. Once traffic is captured in a pcap file, you can open it with Wireshark to analyze. These are the steps to use RawCap:
  • Run your applications with shared memory disabled. You can disable shared memory via XML by adding into your <participant_qos> this setting:

           <transport_builtin>

             <mask>UDPv4</mask>

           </transport_builtin>

• In a terminal run RawCap.exe and select the interface in which you wish to capture traffic.
• You can configure Wireshark to show RTPS packets with specific colors, as explained in this Community KB.

4 – Finally, you can also find useful tips for system-level basic debugging in this KB article.

Seven months ago I started commuting more than 10 minutes to take my kids to school. Before this, I’d spend the 10 minutes driving to school chatting it up with my kids, and the 10 minutes driving back home listening to music or simply enjoying some quiet time before starting my day. When I suddenly had about three times that to spend, I started looking for interesting and productive ways to fill the time. Enter podcasts.

I have listened to so many episodes and learned so much about personal finance, goal setting, marketing, engineering, the IIoT, etc. If you’re interested in a topic, I’d guess there’s a podcast or two out there that you’d enjoy. And this got me thinking: RTI should have a podcast.

We have a ton of content, but it’s mostly in written form (the exception being video tutorials and our large on-demand webinar collection). I believe that our content holds real value for our users and people who are interested in distributed system design, software and system architecture, and the IIoT. If you’re interested in these things, whether you’re a developer, engineer, architect or executive, we have content that you’d learn from and enjoy. What if I could take this written content and produce audio versions so you could listen to it on the go? I know that I don’t always have time to read a whitepaper or an ebook, but could listen to one during my commute or while I’m working! I’d venture to guess that we have this in common.

In addition to providing a way to offer up audio versions of our most popular content, a podcast could feature interviews with our customers and industry experts. Our customers do amazing things and I know that when ever I have the chance to speak with them, I leave feeling inspired by what I’ve learned and what they’ve achieved. And we wouldn’t be talking about just tech – we’d have episodes that discuss leadership, managing distributed teams, market trends and analysis, and more.

Well four months, 12 interviews and hours of editing and work later, I’m proud to present The Connext Podcast. We’re kicking off the launch of this new project with four episodes. New episodes will be available every other Wednesday. Head on over to Soundcloud or www.rti.com/podcast to subscribe and listen. I mean, how often can you say you learned something new in ~30 minutes? Well, The Connext Podcast has you covered for the next four days. Happy listening!

If you missed it, you should check out the recent press release about RTI’s growth in the Industrial Internet of Things (IIoT). It’s really a great time to be RTI! Sure, from a business perspective all the vectors point the right way. But for me, the most exciting things in that press release aren’t numbers.  I’m more amazed that we get to play with so many futuristic applications. Carbots? Renewable energy? Smart healthcare? Hyperloop? Flying cars? Wind turbines? CT scanners? We got ‘em all. And new things show up all the time.

How can a small company like RTI play in so many areas? It’s Thursday, so in honor of #TBT, I thought I’d take a look back and see how we got here.

For those of you RTI history buffs, we sold our tools business to Wind River in 2005. It was 80% of our revenues at the time. So, in 2006, we had some money and great people, but not much in the way of products. So, we started looking for new trends. What did we find? Here’s an excerpt from our 2006 vision paper called “The Data-Centric Future”:

Truly profound technologies become part of everyday life. Motors, plastics, computers, and now networking have made this transition in the last 100 years. These technologies are embedded in billions of devices; they have melted into the assumed background of the modern world.

Another step is emerging in this progression: pervasive, real-time data. This differs from the Internet in that this pervasive information infrastructure will connect devices, not people. Just as the “web” connected people and fundamentally changed how we all interact; pervasive data will connect devices and change how they interact.

Today’s network makes it easy to connect nodes, but not easy to find and access the information resident in networks of connected nodes. This is changing; we will soon assume the ability to pool information from many distributed sources, and access it at rates meaningful to physical processes.  Many label this new capability the “network-centric” architecture. We prefer the term “data centric” because the change, fundamentally, is driven by a fast and easy availability of information, not the connectivity of the network itself. Whatever the name, it will drive the development of vast, distributed, information-critical applications.

If you change “pervasive information infrastructure” to “Industrial IoT,” that was a pretty good prediction. Too bad the name didn’t stick, but I suppose “PII” sounds more like the second act of a play or a stuttering math teacher than a technology revolution. Anyway, we thought it was coming soon, but 6 years later, we were still predicting. An RTI handbook entry from 2012 (still before the IoT and before the IIC launched the Industrial IoT in 2014) expanded on this a bit:

There is amazing value in distributed information. Connecting people to information transforms society – news feeds, weather satellites, and the Internet are only a few examples – timely information flow drives value in every industry and every endeavor.

However, current technologies only connect people at human speeds. There’s an entirely new opportunity to connect machines at physics speeds. Just as the Internet connected people and fundamentally changed how we all interact, a new “pervasive information infrastructure” will connect devices at speeds fast enough to drive distributed applications. RTI’s people and technology are the best in the world at delivering that data to the right place at the right time. We fundamentally connect complex systems at extreme speeds better than any organization on the planet.

Our technology enables tens, or hundreds, or thousands, or (soon) millions of processors to work together as a single application. Why does that matter? Because intimately-connected systems can do things that weakly-connected systems cannot. They can request and access data from far-flung reaches fast enough to react intelligently. They can read deeply-embedded sensors, use that data to control high-speed machines, and feed the results to the enterprise for monitoring and optimization. This powerful connectivity is a fundamental transformation that will make currently difficult things commonplace and currently impossible things possible. We are already working on many of these applications. Our work will help astronomers probe the deepest reaches of space, protect passengers from injury on tomorrow’s roads, make our nation more secure from attack, and improve the efficiency of renewable energy generation. RTI is leading the new wave of large connected systems, systems that work together as one.

So how can we play in so many futuristic areas? Call us lucky or call us prescient, but the IIoT has been our target for over a decade. RTI is influential in the IIoT because we got a head start. We’re no longer predicting. We’re now realizing the future.

So, what’s next? Our new tagline is “RTI lives at the intersection of functional artificial intelligence and pervasive networking.” Think about that one for a while. AI and connectivity are perhaps the most important trends for the next 40 years. They will combine to bring new wonders to light in every industry on the planet. The IIoT is much more than a name or a connectivity technology; it’s a new infrastructure for, well, everything. Connecting things together is powerful. Connecting those things to smarts is a whole new game. Carbots will improve transportation, autonomous hospital devices will take better care of patients and smart networks will make green energy practical. But, these are just the ENIACs of the IIoT. It won’t be long until every device on the planet is part of a connected, intelligent infrastructure. That infrastructure will make everything more efficient, more useful and friendlier.  These are exciting times indeed!

In any case, it’s an honor to be a leader in the “new” IIoT revolution. It’s even more of an honor to be associated with the technical visionaries at RTI that put us there.

-Where talent meets innovation.

FRIDAY! It’s the first thing that comes to mind when I think about RTI. To almost everyone, Friday means the end of the week, the beginning of the weekend, late night parties and, of course, catching up on sleep. But for me, Friday and RTI have an entirely different connection, and I will explain why.

It starts with a cold, snowy Friday in February 2016. I was looking forward to the upcoming weekend – needed a break from the stressful set of assignments and projects for my courses at Penn – when I got an email from RTI saying that the HR department would like to schedule a call.

Immediately, my excitement started to build, since prior to this I had gone through a couple of interviews with RTI, including a coding round for the role of a Software Engineer Intern. I hoped this call would give me a reason to go out and celebrate on Friday. Hope turned into reality when I got the call saying that I was being offered an Internship role at RTI for the summer. Though they proceeded to tell me about the compensation, benefits, etc., all those words only passed tangentially over me. The only information I was excited about was the part that I got the internship I had been hoping for. The call ended at 6 pm that Friday, and yes, as I said before: a reason to go out and celebrate!

After gladly accepting the offer, I communicated with Jan, the VP of Engineering (and the Head of Research) over emails to deepen my learning about RTI and to keep in touch before my start date. We discussed the project and skills to deploy. It was supportive of them to answer my queries and encourage discussions.

Fast forward three months, and it was the end of my 2nd semester. I flew to Sunnyvale, California and prepared myself for the first day at RTI. Starting on a sunny day on May 11th, an initial welcome by HR set the ball rolling for the internship that was to go on for the next 15 weeks.

Fig 1: The first view of my cubicle.

After the initial formalities of joining, I was soon greeted by the VP of Engineering, having already spoken over emails with him, I could now link a face to the name. Furthermore, I was introduced to my mentor for the internship project. After a quick tour of the RTI office, I was soon taken for a welcome lunch by my mentor to a delicious place where we discussed a little bit of the project as well as more about RTI.

The very next day was exciting since all of us were going for a toast at the nearby ToGo’s to celebrate a recent product release. How awesome is it that you get to attend a toast party the very next day after joining? I was sinking into the moment. I imagined what would be in store for me the next day since it was a Friday, and Fridays meant the start of a lovely weekend. RTI didn’t fail to impress me on my first Friday as well since I was told that the company provides lunches every Friday. Yummy delicious food from the best restaurants around the Bay Area. WOW!

Soon enough, there was RTI Corporate training scheduled at HQ, and I was auto-enrolled to join. The training christened appropriately as “QuickStart,” was true to its title in getting me on track with RTI products quickly! The training focused on the products we offer and what they do and how. In addition, the hands-on exercises were truly a bonus. Receiving a training completion certificate from the presenting Principal Solution Architect was truly icing on the cake. Time started to roll by, I started to work on my project, and planned and chalked out a schedule of how and when things were to be done.

My project was to develop an Eclipse plugin from scratch. The project entailed completely encompassing the steps that a user of RTI Connext® DDS would go through in generating code from an IDL, bringing the project to Eclipse, modifying the IDL and regeneration of support code and finally building and executing the code without leaving Eclipse or using any other tool/software.

It was very exciting for various reasons. Firstly, I was to develop this from scratch, which became a true learning experience since I could research the various ways to go about this and choose the best one that would suit our requirements. Secondly, the complete code base was to be done in Java, a language with which I’m comfortable and had a quest to learn in-depth. Developing this plugin resulted in a fantastic learning experience with Eclipse and Java. I also ensured that I followed industry standard coding practices and guidelines along with appropriate documentation and comments in the code base.

Carrying on further, I started developing prototypes and very proactively communicating all of the developments and progress on my project to my mentor and the VP of Engineering on a regular basis. This communication was well appreciated and helped them to track progress without any follow up from their end.

One thing I’m particularly fond of is RTI’s daily lunch time. All of us gather around and go to the food options located in the adjacent building; we buy ourselves food and get back to the office kitchen to eat. This culture undoubtedly creates one of the best times of the day since you get to sit and talk to everyone about various topics ranging from sports to technology to personal favorites (in my case, cars! ). I was pleasantly surprised and happy to understand that the regular corporate hierarchy and designation-based differences are non-existent at RTI and everyone is extremely friendly and amiable. How cool is it that our CEO plays ping pong with you and joins you at the lunch table for interesting conversations? Another fun thing to add here is the after-hours get-togethers. One such shown below:

Fig 2b: After hours at Eureka.

After successfully achieving the first milestone in my project schedule, additional milestones and tasks were added. Thinking from the perspective of the end-user, I started to analyze various other requirements and added them to the list of tasks for the next milestone, giving me a sense of accomplishment and satisfactory progression on a task to complete.

Soon enough, I showcased my work to our CTO, who patiently sat with me, went through every step of the project and appreciated me for things I did intuitively. He also provided suggestions that could improve the quality of the plugin being developed. Incorporating the suggestions as I progressed, I was beginning to feel a sense of fulfillment to the project towards the end of July.

Fig. 3: Figurative form of thoughts in my head

Towards the end, when I had achieved something substantial, I was then given a slot at the weekly R&D meeting to show a demo of the Eclipse Plugin to my colleagues at HQ as well as to my colleagues in our Spain office via a video call. I received valuable input after the demo which helped me shape my project appropriately.

Two months had passed by and it was the end of July. One fine day, the VP of Engineering walked up to my desk and asked me to join him for a short walk around the building, wherein details of my plan to complete my courses for graduation and timeline were discussed.

A few days later, a meeting with HR provided a little more information which gave me a deeper hope about returning to RTI as a full-time employee. I was thrilled. I was waiting. At the end of July on a Friday morning, I was as usual busy with my semi-colons and code, when The VP of Engineering walked up to me with a paper in hand, and said, “Let’s discuss a few things.” I tried to keep my excitement from appearing on my face, as he revealed that RTI was making me an offer to return as a full-time employee and went over the details about the compensation and benefits.

That moment, just like the moment many, many Fridays ago in February, had a deja-vu effect, as everything spoken that day was only remotely and softly heard by me as I was too excited to receive an offer from RTI. I love RTI, love the Bay Area for numerous reasons, and being able to come back was something I always hoped for. I quickly gathered my thoughts together, thanked Jan and RTI, and continued my work until the end of the day.

There were about three weeks left, and I completed all the tasks and milestones by the first week of August, leaving me with two weeks to complete all the testing and documentation. This Eclipse plugin was to be deployed and tested on three different platforms mainly Windows, Linux, and Mac. Hence, I had all three machines on my desk, and my desk looked like this.

Fig 4: My lovely desk with my toys.

On the final day of my internship, having completed the exit formalities, I was pleasantly surprised to receive a box full of my favorite brownie bites from one of the friends I had made at RTI. Another lunch from RTI, on my last Friday of the internship, completed the schedule. I was all set to fly out of the Bay Area that evening, and onward to India for a short vacation. Appreciating the help and support from many of my newfound colleagues, I bid goodbye to everyone at RTI and the Bay Area with one thought; I’ll be back!

Fig 5: Me with the RTI banner at the entrance

RTI Connext® DDS provides an order of magnitude performance improvement over most other messaging middleware. But occasionally we run into customers who are trying to improve the performance of their DDS communications. This performance improvement can be achieved in either throughput or latency. In this blog, I will go through the three simple steps required to assess the performance of your system and will also review some of the most common ways customers have improved performance of their DDS communications.

Step 1: What performance should you be getting?

Compare the numbers you are getting with the comprehensive DDS benchmarks that RTI provides here: https://www.rti.com/products/dds/benchmarks.

If you are not getting close to the numbers you see in the DDS benchmarks, there are a couple things to try:

Use RTI Perftest to make sure you’re comparing apples to apples.

The configuration of the NIC and the network switch, as well as the maximum network throughput and the CPU, all have an impact on the final DDS performance results. So, to make a fair comparison run the DDS benchmarks on your hardware.  RTI makes the DDS benchmark program, “RTI Perftest,” available in source code format with complete documentation.  You can find a copy of Perftest here:  https://community.rti.com/downloads/rti-connext-dds-performance-test

Make sure you are running your tests using the network interface you think you are using.

DDS enables shared memory and UDPv4 transports by default. If Shared memory is available between two nodes DDS will use that by default. But if there are many network interfaces available DDS will only use the first four. I’ve seen developers want to test out a certain network interface, say Infiniband, but it was not one of the first four listed and so DDS was not adding it to the mix. In fact, on Windows systems, the order that network interfaces are listed by the OS, and thus selected by DDS, is random and so the network interface you are actually using can change from run to run. In fact, DDS will actually send the same data over two paths, if they exist, to the same endpoint. This can take up CPU time and slow throughput.  You can explicitly select the interface you want (or do not want)  using the transport QOS “allow-interfaces” property.   Here is a good RTI Community article on the subject: https://community.rti.com/howto/control-or-restrict-network-interfaces-nics-used-discovery-and-data-distribution.

Following is the actual XML code for “allow_interfaces” and “deny_interfaces” QOS that lets you explicitly pick the network interface you want to use or do not want to use:

<participant_qos>
 <property>
  <value>
   <element>
    <name>dds.transport.UDPv4.builtin.parent.deny_interfaces</name>
    <value>10.15.*</value>
   <element>
   <element>
    <name>dds.transport.UDPv4.builtin.parent.allow_interfaces</name>
    <value>10.10.*,192.168.*</value>
   </element>
  </value>
 </property>
</participant_qos>

Step 2. Use the RTI DDS tools to diagnose your performance issues. 

Use RTI Monitor to look for the number of ACKs, NACKs, dropped packets, and duplicate packets.  If these numbers are high, it can be due to several things:

• Transport buffer sizes are too small
• MTU  is not optimized for switch
• There may be too many heartbeats causing multiple resends for single NACKs, indicating the reader is not keeping up
• The CPU and memory process(es) are bound.

Use RTI Monitor or Admin Console to compare QOS settings of the DataReaders and DataWriters.  Sometimes you are not using the QOS values you think you are using.

A great way to learn about using the Admin Console and the Monitor tools is to watch this video from our tools lead, Ken Brophy.

Step 3. Now let’s start to look at your application to see how we can speed things up by changing the “shape” of the data in motion.

RTI DDS gives you many ways to fine tune your system using QOS settings. This flexibility is great because you have a lot of control over how DDS works.  But all the options can be daunting! I won’t go over every setting (this blog would quickly grow to be a textbook) but I will hit on what I feel are the most important settings to check in regards to performance.

First, don’t use strict reliability if it is not needed. Strict reliability makes sure that every sample reaches every reliable destination and will re-send samples if necessary. Resending samples and the structure that supports them take time and memory.  Many applications would be fine missing a sample very occasionally or waiting longer for it to be re-transmitted.

If you do need to use strict reliability then start with the DDS built-in profile “StrictReliable.HighThroughput”.  It is a good idea in general to use the built-in profiles that RTI provides. These built-in profiles are set up by RTI to have all of the default settings needed for the most common DDS use cases. The built-in profiles can be used as-is or can be used as the basis for your QOS configuration and then tweaked for your specific needs. You can read about using DDS built-in profiles and get a working example here:  https://community.rti.com/examples/built-qos-profiles 

Using Extensible types (XTypes) and sequences of structures can hurt performance. DDS serializes and de-serializes data it sends and receives, and this process takes a lot longer with complicated data types.

Adjust heartbeat_period/ ACKNACK combo.  In reliable communications, the DataWritersends DDS data samples and heartbeats to reliable DataReaders. A DataReader responds to a heartbeat by sending an ACKNACK, which tells the DataWriter what the DataReader has received so far. In addition, the DataReader can request missing DDS samples (by sending an ACKNACK) and the DataWriter will respond by resending the missing DDS samples. So, the heartbeat_period can control how quickly a data reader can acknowledge receipt of a sample or ask for a sample to be re-sent, impacting performance.  Here is an article that talks about how the heartbeat_period can impact latency and throughput.

Modify the Asynchronous Publisher configuration to use flow control to lower the data rate. Sometimes if the data rate from the writer is too fast, the reader gets swamped and the resulting dropped samples and resends slow down the system. Lowering the writer’s data rate a little leaves room for repairs, etc. This gives DDS time to handle incoming data and avoids costly resends. You can use a flow controller to shape the output traffic your publisher will generate. By using an asynchronous publisher and custom flow controller you can lower the data rate. You can see a working example of how to use the asynchronous publisher here: https://community.rti.com/examples/asynchronous-publisher

For smaller sample sizes, use batching and/or Turbo Mode. Batching groups of small samples into a single large packet is more efficient to send and can result in a large throughput increase. But note that while the use of batching increases throughput, it can hurt latency when little data is being sent (because of the added time needed to batch small samples). In high-throughput cases, though, average latency results because of all the CPU saved on the subscriber side of the interface.

Turbo Mode is an experimental feature that uses an intelligent algorithm that adjusts the number of bytes in each batch at runtime according to current system conditions, such as write speed (or write frequency) and sample size. This intelligence gives Turbo Mode the ability to increase throughput at high message rates and avoid negatively impacting message latency at low message rates.

Here is an article that goes into detail on how to use batching and includes a working example: https://community.rti.com/examples/batching-and-turbo-mode

Use multicast for topics with more than a couple of subscribers. Multicast allows a publisher to send to multiple readers with a single write, greatly reduces network and publisher-side processor utilization.  Note that sometimes this feature is not available at the network level.  Here is a good article on  how to implement multicast: https://community.rti.com/best-practices/use-multicast-one-many-data

For reliable communications modify the Send Window size. When a reliable DataWriter writes a DDS sample, it keeps that sample in its queue until it has received acknowledgments from all of its subscribing DataReaders that the sample was received. The number of outstanding DDS samples allowed is referred to as the DataWriter’s “send window.” Once the number of outstanding DDS samples has reached the send window size, subsequent writes will block until an outstanding DDS sample is acknowledged. Anytime the writer blocks, it hurts performance. You can read about adjusting the Send Window in section 6.5.3.4 of the DDS User’s Manual.

Modify the transport settings. Whether you are using UDPv4 or shared memory or a custom transport, having the right buffer sizes and message sizes configured is extremely important when trying to optimize performance. Following is XML code for modifying transport message size and buffers sizes for the UDPv4 transport:

<participant_qos>
 <property>
  <value>
   <element>
    <name>dds.transport.UDPv4.builtin.parent.message_size_max</name>
    <value>65536</value>
   </element>
   <element>
    <name>dds.transport.UDPv4.builtin.send_socket_buffer_size</name>
    <value>524288</value>
   </element>
   <element>
    <name>dds.transport.UDPv4.builtin.recv_socket_buffer_size</name>
    <value>2097152</value>
   </element>
  </value>
 </property>
</participant_qos>

Note that the sizes used here are suggestions for optimizing performance when using large samples. You can make these values smaller for smaller samples.

I hope this advice is helpful in getting the best performance out of your DDS Application. I’ve listed the tips I’ve found most helpful for improving DDS performance but there are other methods that can also be helpful depending on the circumstances. In order to get more information on improving throughput or latency (or really help with any other Connext DDS issue), I encourage you to check out the RTI Community portal. The RTI Community portal is an excellent source of information and support! And of course, always feel free to contact our great support department or your local Field Application Engineer for further help.

A recent Wired article on Charlie Miller (infamously known for remotely hacking and controlling a Jeep) claims that “open conversation and cooperation among companies” are necessary prerequisites to building secure autonomous vehicles. This seems rather far-fetched when so many companies are racing to dominate the future of the once-nearly-dead-but-newly-revived (remember the Big Three bailouts?) automotive industry. As naive as that part of the article sounds, what really blew my mind was the implication that the answer to re-designing security lies solely within the autonomous-vehicle industry.

The concept of security is not isolated to autonomous vehicles so there is no benefit in pretending that’s the case. Every IIoT industry is trying to solve similar problems and are surprisingly open to sharing their findings. I’m not saying that Miller needs to go on a journey of enlightenment through all other industries to create the ideal solution for security. I’m saying this has already been done for us, compliments of the Industrial Internet Consortium (IIC).

The IIC consists of 250+ companies across several industries – including automotive suppliers like Bosch, Denso, and TTTech – with the same fundamental problem of balancing security, safety, performance, and of course costs for their connected systems. If Wired and Miller are looking for an open conversation – it’s happening at the IIC. The IIC published the Industrial Internet Reference Architecture, which is available to everyone for free – as in “free beer,” especially if the car is doing the driving for you! The extensions to this document are the Industrial Internet Connectivity Framework (IICF) and Industrial Internet Security Framework (IISF). These documents provide guidance from a business perspective down to implementation, and the IISF is particularly applicable as it addresses Wired’s brief mentions to securing the connectivity endpoints and the data that passes between them.

Take a ride with me and see how we might modify the connected car’s architecture to protect against potential adversaries. Since we do not have any known malicious attacks on cars, we can start with Miller’s Jeep hack. Thanks to a backdoor “feature” in the Harmon Kardon head unit, Miller was able to execute unprotected remote commands quite easily. Through this initial exploit, he was able to reprogram a chip connected to the CAN Bus. From there, he had nearly full control of the car. You’re thinking, “just remove that unprotected interface,” right?

Miller would not have stopped there, so neither shall we. Assuming we could still find an exploit that granted us access to reprogram the ARM chip, then Wired’s article rightly suggests establishing an authenticated application – perhaps starting with secure boot for the underlying kernel, leverage ARM Trust Zone for the next stage of critical-only software, and implement some sort of authentication for higher level OS and application processes. Your device endpoint might start to look like a trusted application stack (Figure 1 below). I can only guess how much this head unit costs now, but to be fair, these are valid considerations to run a trusted application. The problem now is that we haven’t actually connected to anything, let alone securely. Don’t worry, I won’t leave you by the roadside.

Figure 1. Trusted Application Stack

Many of these trusted applications connect up directly to the CAN Bus, which arguably expands the attack surface to the vehicle control. The data passed between these applications are not protected from unauthorized data writers and readers. In the case of autonomous taxis, as Wired points out, potential hackers now have physical access to their target, increasing their chance of taking over an application or introducing an imposter. Now the question becomes: can applications trust each other and the data on the CAN bus? How does the instrument cluster trust the external temperature data? Does it really need to? Maybe not and that’s ok. However, I am pretty sure that the vehicle control needs to trust LIDAR, Radar, cameras, and so on. The last thing anyone wants to worry about is a hacker remotely taking the car for a joyride.

We are really talking about data authenticity and access control: two provisions that would have further mitigated risk against Miller’s hack. Securing the legacy applications is a good step, but let’s consider the scenario where an unauthorized producer of data is introduced to the system. This trespasser can inject commands on the CAN Bus – messages that control steering and braking. The CAN Bus does not prevent unauthorized publishers of data nor does it ensure that the data comes from the authenticated producer. I’m not suggesting that replacing the CAN Bus is the way forward – although I’m not opposed to the idea of replacing it with a more data-centric solution. Realistically, with a framework like Data Distribution Services (DDS), we can create a layered architecture as guided by the IISF (Figure 2 below). The CAN Bus and critical drive components are effectively legacy systems for which security risk can be mitigated by creating a DDS databus barrier. New components can then be securely integrated using DDS without further compromising your vehicle control. So what is DDS? And how does it help secure my vehicle? Glad you asked.

Figure 2. Industrial Internet Security Framework Protecting Legacy Endpoints

Imagine a network of automotive sensors, controllers, and other “participants” that communicate peer-to-peer. Every participant receives only the data it needs from another participant and vice versa. With peer-to-peer, participants in that network can mutually authenticate and if our trusted applications hold up, so does our trusted connectivity. How do we secure those peer-to-peer connections? TLS, right? Possibly, but with the complexity of securing our vehicle we want the flexibility to trade off between performance and security and apply access control mechanisms.

Let’s back up a little and re-visit our conversation about the IICF, which provides guidance on connectivity for industrial control systems. The IICF identifies existing open standards and succinctly attributes them to precise functions of an Industrial IoT system. At its core, an autonomous vehicle, as cool as it sounds, is just an Industrial IoT system in a sleek aerodynamic body with optional leather seats. So what does the IICF suggest for integrating software for an Industrial IoT system, or more specifically, autonomous systems? You guessed it! DDS: an open set of standards designed and documented through open conversations by the Object Management Group (OMG). An ideal automotive solution leveraging DDS allows system applications to publish and subscribe to only messages that they need (see Figure 3 below for our view of an autonomous architecture). With this data-centric approach, we can architecturally break down messages based on criticality for safety or need for data integrity.

Figure 3. Autonomous Vehicle Data-Centric Architecture

And now that we’ve established a connectivity solution for our autonomous vehicle, we can get back to talking about security and our TLS-alternative: a data-centric security solution for a data-centric messaging framework. With DDS Security, Industrial IoT system architects can use security plugins to fine-tune security and performance trade-offs, a necessary capability not offered by TLS (Figure 4 below). Authenticate only select data topics but no more? Check. Encrypt only sensitive information but no more? Check. Actually, there is more. Casting aside centralized brokers, DDS Security offers distributed access control mechanisms dictating what participants can publish or subscribe to certain topics without single points of vulnerability. This means that Miller’s unauthorized application would be denied permission to publish commands to control braking or steering. Or if Miller compromised the data in motion, the data subscriber could cryptographically authenticate the message and discard anything that doesn’t match established policies. Can we say our autonomous vehicle is now completely secure? No, because as Miller made it perfectly clear, we still need more conversations. However, we can certainly say that DDS and DDS Security provide the forward-looking flexibility needed to help connect and secure autonomous systems.

Figure 4. Connext DDS Secure Pluggable Architecture

So, to Mr. Charlie Miller (and of course Mr. Chris Valasek), your work is amazing and vision inspiring, but I think you need to look across industries if you want to talk openly about redesigning automotive architecture. When you and all the other Charlie Millers in the world want to have that open conversation, come knock on our door. At RTI, we are ready to talk to you about autonomy, Industrial IoT, safety and security, and everything you else you believe should define cars of tomorrow.

Congratulations! Hopefully the tips from Mission: score an interview with a Silicon Valley company were helpful, and you have been contacted to talk to the hiring manager. Here are a few tips on how to ace the initial call.

Before the interview

• Test your system. We often use a video call to interview candidates: Skype or Google Hangouts. Make sure your camera and microphone work. Do not use a phone for video conferencing. Sometimes we do hands-on exercises. Have a working IDE, editor and compiler installed. Lastly, take a few moments to clean up your computer background and desktop files.
• Be professional. Dress appropriately in a shirt or blouse. A suit or tie is overkill. Also, pay attention to your environment and background. Clean up the room. Coordinate with your family or roommates that you will be in an important interview call.
• Prepare by exploring the company website. Read about the products, download the evaluation software, check out the videos, or forums. Learn as much as you can about the company. Look up the interviewer on LinkedIn. You have to learn about a common passion.

During the interview

• Be friendly and personable. Smile. Show a spark.
• Be confident, but don’t oversell. And no BS.
• Ask clarifying questions, especially if your English is not that great.
• Be sincere if you do not know something but try to answer anyway. For example, “I am not a Java expert, though if it works similar to C#, then …”
• Don’t give up. Brainstorm. An interview is not a pass/fail test. One candidate felt he should have been asked back for an in-depth interview because he answered six of the ten questions correctly; in his eyes, he had passed the interview. Unfortunately, for the four questions he missed, he didn’t even try to answer them. Being able to figure out things you don’t know is one of the most important skills of an engineer.
• Don’t play hard to get or show you are disinterested. Don’t act selectively.
• Think of some questions to ask about the company, job, customers, etc.

Good luck getting to the in-depth interview mission. Apply now to start the process!