Serial Data Loggers: RX Stream Handling

A common task in any engineering environment is realtime data logging.  Of the many ways to do this, my current favorite is to casually stream sensor data at a fixed rate to some host computer. On a UART serial bus, this is pretty convenient because a host can “listen in” on the device anytime. Visually, this architecture may be represented as:

This post will focus on the host side of things since it is typically closer to what the end product (or prototype) will look like. From the host, the user can effectively monitor any kind of realtime process, save the results to a data file, or implement a GUI for controlling the hardware.

To keep things simple, we’ll use python, and let’s say you already have a device like the Arduino shown in the diagram above. We just want to throw together a quick and dirty terminal  “GUI” for reading all device data.  In fact, let’s just say someone has already written up some clues as to how the Arduino does its job

Arduino Specification:

  • Collects all sensor data at a rate of 40Hz
  • At the end of every cycle, the arduino will print all sensor data to the serial bus. It is up to the host machine to decipher the package of data and pull out the individual bits of sensor information from it.

Seem like a reasonable specification for an Arduino (or any other microcontroller)? Upon this, we draw up the following spec for the host machine:

Host Specification:

  • Continuously listen for any data that comes from the Arduino.
  • Speed/latency requirements behind data collects are going to be pretty casual. Any code that works up to 100Hz will do.
  • Print contents of data to screen whenever it is requested.
  • Allow the user to “inject” predefined commands to the Arduino via the means of pressing a key on the host machine’s keyboard.

As the programmer, I have a couple preferred ideas for how to go about this:

  1. Write a single threaded process that merely reads the data, and prints it to a screen. (fine if you don’t ever care to trigger any commands to be sent back to the device itself, ever)
  2. Write a 2 threaded process in python, using the threading library. One thread will be the “RX Thread,” for reading and processing the data stream. The other thread will be the “main thread,” which takes user input.
  3. Use Pygame’s event handling tools to do pretty much the same thing, but with some added features in reading/processing the user’s keyboard inputs.

This post focuses on option 2, whereby one can spin up a pretty versatile program fairly quickly using python built in libraries. Below is an example that would be a good starting point for such a program.

Example Program

Note that there are a couple gotchas that will ultimately be up you as to decide how much code you want to write against it:

  • Thread locks should be used to make reading and writing RX data more atomic before any kind of additional processing is embedded in the RX thread. Otherwise, your main thread could end up reading and making decisions off of stale (or even incorrect!) data.
  • Do you care if your main thread has access to data older than whatever just came in the latest packet? Should you build a queue of recorded data? You may find python’s queue library useful if so.
  • Additionally, if you care about missed packets all together, you may need to embed a message counter on the microcontroller, and have it send the the message # along with the other sensor data.

That’s it! To cap it off, here’s an example of using the program:



Other recommendations

This code merely helps you get the most simple kind of terminal-based RX interface on the ground. This may be great if you need something prototyped, but if your ambitions take you further, you can try ncurses to make a fancy GUI inside the terminal window. This is sometimes ideal if your working over an SSH connection.

Leave a Reply

Your email address will not be published. Required fields are marked *