Arduino Serial I/O for the Real World
Non-Blocking Text I/O using the SafeString library

by Matthew Ford 19thMay 2021 (original 17thDecember 2020)
© Forward Computing and Control Pty. Ltd. NSW Australia
All rights reserved.

Arduino Text I/O that Works in the Real World
How use Text I/O for Control and Debugging
Get text input, parse it and output debug msgs
without blocking the rest of the loop.

Quick Start for Real World Text I/O Applications

This tutorial will take you through the steps you can use to successfully process Real World text I/O
With a couple of exceptions, you should avoid using any Serial methods in your loop() code and any code it calls. (see below)

Install the SafeString library (V3+) from the Arduino Library Manager or the zip file
For text input use a SafeStringReader – a non-blocking robust high level reader for reading lines of data or general text with delimiters, with optional echo and optional non-blocking timeout
For text processing use the SafeString methods – what Arduino Strings was meant to be, but with out the memory problems and odd errors and with detailed error messages.
For text output write to a BufferedOutput – the non-blocking replacement for Serial print, so your debug msgs and results do not delay the rest of the loop and cause missed input.
For testing use a SafeStringStream – stress tests your sketch's text processing.
If you need a larger input buffer use a BufferedInput – for when the sketch processing delays the reading of the input and chars are lost

Often all you need is to set the baud rate of your Serial connection as high as possible and use a SafeStringReader for input and BufferedOutput for output to avoid missing any incoming data and SafeString to process the text to avoid coding errors.

The examples below often use less then optimum settings in order to illustrate how to solve the problems that can arise in the Real World.

Arduino Serial is not suitable for handling Text I/O in Real World applications

Instead this tutorial shows you how to use the SafeString library (V3+) to collect and process text input and output text and debugging messages, without delaying the execution of the rest of your loop() code.

With a couple of exceptions, you should avoid using any Serial methods in your loop() code and any code it calls. This is because almost all Serial methods will block the rest of your loop() code from running so you will miss real world inputs and your outputs will be delayed. Print() statements are the primary means of debugging Arduino sketches, but adding Serial.print()s adds more delays and often causes more problems then they find. Using Serial.read() is prone to coding errors. The exceptions are Serial.read() and Serial.available() and Serial.peek() which do not block.

The SafeString library can be installed from the Arduino Library manager, just search for SafeString. A full tutorial on SafeString is available here

The tutorial covers reading text input, parsing for commands, parsing text to numbers, e.g. GPS data, buffering input and outputting results and debugging messages. All without blocking the rest of the sketch so that your Arduino won't miss the real world inputs and be able to reliably control real world devices, stepper motors, relays etc. The non-blocking SafeString library input methods let your Arduino accept input for processing, and for control of your your Arduino, while continuing to run the rest of the sketch at full speed. The non-blocking output class, BufferedOutput, lets you output results and add debugging without interfering with the running of the code you are trying to debug. The BufferedInput class lets you add extra buffering for your inputs when be built in Arduino buffers are not enough. The tutorial also covers automated, repeated, testing of sketches that process text input.

This entire tutorial came be completed with just an Arduino UNO. Optionally at the end of the tutorial there is a second GPS example that runs on an Adafruit GPS Feather Wing and an Adafruit 32u4 Feather

Tutorial Outline:-
SafeStringReader :- replacement for Serial.read()
    Reading User Commands
    Reading and Parsing Instrument Data - GPS
BufferedOutput :- non-blocking replacement for Serial.print() to output results and debugging messages with out blocking the loop() from processing more input.
    Using a higher Serial baud rate
    Increasing the output buffer and skipping some output using BufferedOutput
    BufferedOutput Options
SafeStringStream :- automate input for testing sketches.
BufferedInput :- adding extra Input buffering so you don't miss input data while processing the last input. Provides statistics to help choose the correct buffer size.
A Real World GPS Example

Together these classes provide a complete non-blocking text I/O that is makes Real World Arduino applications practical.

Parts List

Hardware
Arduino UNO or any other board supported by the Arduino IDE. All the code developed here can be tested with just an Arduino UNO.
Optional – an Adafruit GPS Feather Wing and GPS antenna and Adafruit 32u4 Feather.

Software
Install the Arduino IDE V1.8.9+
Install the SafeString library (V3+) from the Arduino Library Manager, this includes millisDelay and loopTimer as well

Optional – install Adafruit 32u4 support, see the Adafruit Feather 32u4 Bluefruit LE installation tutorial

Also see Arduino For Beginners – Next Steps
Taming Arduino Strings
How to write Timers and Delays in Arduino
Safe Arduino String Processing for Beginners
Simple Arduino Libraries for Beginners
Simple Multi-tasking in Arduino
Arduino Serial I/O for the Real World (this one)

Why not just use Arduino Serial?

Arduino provides a Serial object that allows reading from, and writing to, the outside world. Why not just use that class? Well ALL of output methods, i.e. the print and write methods, will stop the rest of you sketch from running once the Serial Tx buffer fills up. The size of the Tx buffer varies between different Arduino boards. Uno/Meg2560 has a 63 byte Tx buffer. The NanoBLE has none. So if you want the rest of your sketch to keep running while you output results or debugging messages, you should not use any of the Serial print/write methods in your loop() code. This tutorial will show you how to use the BufferedOuput class to avoid blocking, add extra buffering.

Except for read() and peek(), all the Serial read methods, find(), findUntil(), readBytes..() , readString(), parse..(), etc, can block for up to 1 second waiting for more input. During that 1 second the rest of your loop is not running, so you should not use any of those methods. That leaves only read() and peek(). Those are low level methods that deal in single bytes. They, together with available(), can be used to build high level non-blocking input, but it requires very careful programming. The SafeStringReader class provides a non-blocking high level read() method that is easy to use, safe and robust and voids you having to manipulate 'unsafe' byte arrays.

SafeString methods are a robust, debuggable replacement for Arduino Strings that are safe from buffer overflows. SafeStrings avoid the memory fragmentation problems of Strings and completely remove the possibility of char[] buffer overflow coding errors. Arstechnica has a detailed article on buffer overflows which says in part “writing more to a buffer (e.g. char[]) than the buffer has space for, sounds like something that should be simple to avoid. It's an exaggeration (but only a slight one) to lay the blame entirely on the C programming language and its more or less compatible offshoots, namely C++” (i.e. Arduino sketches). The wikipedia entry says, “Programming lanuguages commonly associated with buffer overflows include C and C++ (i.e. Arduino), which provide no built-in protection against accessing or overwriting data in any part of memory and do not automatically check that data written to an array is within the boundaries of that array.”

Even experienced programmers make coding mistakes when working with char[] and pointers. The lastest IPhone security failing (local copy here) is a recent example (of many others) of a buffer overflow bug from 'professional' coders. Also see "Buffer overflow and buffer management errors are common issues in C++" The SafeString library is easy to use and provides in-built protection against buffer overflows and accessing outside the buffer range.

This tutorial only deals with Text I/O, it does not deal with reading or writing non-text (i.e. binary) data. If you need to send and received binary data, like a compiled program file or .hex file, then you are back to using low level methods and very careful coding.

Basic Serial Hardware Settings

Serial communications consists of sending a sequence of 0's and 1's. Sparkfun (local copy here) provides a good introduction to the Serial communication protocol. While there are many options for the low level format of the bytes sent, 8N1 is most common and is the default for Arduino Serial. The data can be sent at any one of a number of speeds (called baud rate). 9600 is common for Uno boards. A common faster speed is 115200, while GPS (NMEA) modules often uses 4800. It is important that number in your code's Serial.begin(.. ) statement matches the speed of the attached device, e.g. Serial.begin(9600) for a 9600 baud device.

The hardware connections for Serial on Uno are pins D0 (Rx) and D1 (Tx). Many Arduino boards only have one hardware serial (the Arduino Mega2560 has 4). A software emulation is needed to add an extra serial to most boards. The suggested software serial library for AVR processors (Uno, Nano, Micro, Leonardo, Mega, Nano, Yún) is the AltSoftSerial library.

SafeStringReader for Text Input

There are two common uses for serial text input.
1) reading user commands to control the running of the programs (sketches) and
2) reading streaming text data from external instruments and parsing it, e.g a GPS module, gas sensor, etc

Reading User Commands

Reading user commands will be dealt with first. Here is a simple robust sketch (SafeStringReader_Cmds.ino) that reads text input and returns commands delimited by space, comma, CarrageReturn or NewLine. This sketch is non-blocking. That is the loopCounter continues to count while waiting for use input. The sketch picks out commands from unlimited length inputs without failing. Any group of characters, up to the specified length (maxCmdLength), separated by delimiters will be returned. The code then checks if that token equals one of the commands. As you can see below the SafeStringReader read() method handles all the non-blocking char reading, checking for overflow and finding delimiters and returning the result in the sfReader as a SafeString. This code is much simpler and more robust then the equivalent Serial.read() char by char code.

void loop() {
  if (sfReader.read()) {
    if (sfReader == "start") {
      handleStartCmd();
    } else if (sfReader == "stop") {
      handleStopCmd();
    } // else ignore unrecognized command
  } // else no delimited command yet

  // rest of code here is executed while the user typing in commands
  . . .
}

Note about SafeString::setOutput(Serial);
Adding this statement in setup() will enable SafeString error messages. You should normally add this when developing your sketch and then remove it once your are finished. If you get any errors later your sketch won't reboot, the SafeStrings will just not contain the text that would not fit. In those cases just add the statement back into setup() and check for the error messages.

Note carefully, here we are using the 'bad' blocking Serial print statements. These will be replaced later in this tutorial with non-blocking BufferedOutput.

#include "SafeStringReader.h"

// create an sfReader instance of SafeStringReader class
// that will handle commands upto 5 chars long
// delimited by space, comma or CarrageReturn or NewLine
// the createSafeStringReader( ) macro creates both the SafeStringReader (sfReader) and the necessary SafeString that holds input chars until a delimiter is found
// args are (ReaderInstanceName, expectedMaxCmdLength, delimiters)
createSafeStringReader(sfReader, 5, " ,\r\n");

bool running = true;
unsigned long loopCounter = 0;

void setup() {
  Serial.begin(9600);

  SafeString::setOutput(Serial); // enable error messages and SafeString.debug() output to be sent to Serial

  if (running) { Serial.println(F(" Counter Started")); }

  sfReader.connect(Serial); // where SafeStringReader will read from
  sfReader.echoOn(); // echo back all input, by default echo is off
}

void handleStartCmd() {
  running = true;  Serial.println(); Serial.print(F("> start at Counter:")); Serial.println(loopCounter);
}
void handleStopCmd() {
  running = false; Serial.println(); Serial.print(F("> stop at Counter:")); Serial.println(loopCounter);
}

void loop() {
  if (sfReader.read()) {
    if (sfReader == "start") {
      handleStartCmd();
    } else if (sfReader == "stop") {
      handleStopCmd();
    } // else ignore unrecognized command
  } // else no delimited command yet

  // rest of code here is executed while the user typing in commands
  if (running) {
    loopCounter++;
    if ((loopCounter % 100000) == 0) { // print the current counter every now and again
      Serial.print(F("Counter:")); Serial.println(loopCounter);
    }
  }
}

Here some example output for the input (with the Arduino monitor set for Newline or Carriage return or Both NL & CR)
looooooooooongInput then stop

 Counter Started
Counter:100000
looooo
 sfReader -- Input exceeded buffer size. Skipping Input upto next delimiter.
oooooongInput then stop 
> stop at Counter:171779

If you have called SafeString::setOutput( ) to turn on error msgs and SafeString.debug() output you will get a message if the input is longer then the maximum command size you specified (i.e. 5 in this case). In any case the input will just be ignored without any errors. If you comment out the SafeString::setOutput( ) statement is setup() all SafeString error messages will be suppressed, but the error checking is still done. If the input is too long sfReader.read() returns with sfReader empty.

Note that using SafeStringReader you don't need to count input chars, check for overflow, check for the delimiter, and correctly terminate the result. SafeString does all of that for you, safely and robustly.

If you want to treat upper and lower case commands the same, just add

sfReader.toLowerCase();

I.e.

  if (sfReader.read()) {
    sfReader.toLowerCase()
    if ( sfReader == "start") {

Adding a non-blocking input timeout

You can add a non-blocking read timeout to SafeStringReader by calling setTimeout(..) i.e in setup( ) add

sfReader.setTimeout(2000);  // 2000mS => 2sec timeout

Then if no more chars read for 2secs, sfReader will automatically terminate and return the current text.

The sketch SafeStringReader_CmdsTimed.ino has a 1sec timeout added (and uses a struct to store the commands and their functions) Set the Arduino monitor to “No line ending” and enter stop (no trailing space). After 2sec the command will be recognised and processed.

stop 
 sfReader_InputBuffer -- Input timed out.
> stop at Counter:60215

Again if you comment out the SafeString::setOutput( ) statement is setup(), the Input timed out message will disappear, but the timeout will still happen.

Skipping Input

SafeStringReader will skip to the next delimiter if the input is longer then the maximum specified, but you can also call skipToDelimiter() to force it to discard all the input upto the next delimiter. For example if you call

sfReader.skipToDelimiter();

in setup() the initial chars read upto the first delimiter will be discarded. This will ignore inputs that are possibly part way through being received when the sketch starts up.

Adding your own commands

You can easily add more commands to this sketch. Just add another } else if (sfReader == .. ) { section and add another handle..( ) method to do what you want. If the new command is longer then 5 characters, then increase the number in createSafeStringReader( ) That number need not be precise, you can make it larger than necessary, for example 20 will cover most commands.

Checking the delimiter

In this example, the commands can be delimited by space or comma or CR or NL. You can check which one terminated the command by calling getDelimiter(), e.g. sfReader.getDelimiter() when sfReader.read() returns true.

Reading and Parsing Instrument Data - GPS

The second common use for serial text input is to read text data from an external device and parse the data received. A simple GPS data parser will be used as an example, SafeString_GPS.ino. The SafeString_GPS.ino sketch is non-blocking and can handle GPS messages up to 80 char long. Longer lines will just be ignored, with an error message if SafeString::setOutput( ) has been called, but will not cause the sketch to fail. Only the $GPRMC message is picked out and parsed, you can add others as needed. Unlike other GPS libraries, the SafeString_GPS.ino sketch does not use any low level c-string functions or char[], which are very prone to programming errors.

As you can see the loop() code is very simple. Note carefully, here we are using the 'bad' blocking Serial print statements. These will be replaced later in this tutorial with non-blocking BufferedOutput.

void loop() {
  if (sfReader.read()) {
    sfReader.trim(); // remove and leading/trailing white space
    if (!checkSum(sfReader)) { // is the check sum OK
      Serial.print("bad checksum : "); Serial.println(sfReader);
    } else { // check sum OK so select msgs to process
      if (sfReader.startsWith("$GPRMC,")) {  // this is the one we want
        if (parseGPRMC(sfReader)) {
          printPosition(); // print new data
        }
      } else { // some other msg
      }
    }
  } // else token is empty
}

The parsing of the GPS messages uses the SafeString stoken() method to pick out the fields one at a time. Using SafeStrings avoids the low level array and pointer code found in Adafruit's, and other, GPS parsing libraries.
See the SafeString Tutorial for more details on the SafeString processing methods.

bool parseGPRMC(SafeString &msg) {
  cSF(sfField, 11); // temp SafeString to received fields, max field len is <11;
  char delims[] = ",*"; // fields delimited by , or *
  bool returnEmptyFields = true; // return empty field for ,,
  int idx = 0; // idx keeps track of where we are upto.
  idx = msg.stoken(sfField, idx, delims, returnEmptyFields);
  if (sfField != "$GPRMC") {  // first field should be $GPRMC else called with wrong msg
    return false;
  }

  cSF(sfTimeField, 11); // temp SafeString to hold time for later passing, after checking 'A'
  idx = msg.stoken(sfTimeField, idx, delims, returnEmptyFields); // time, keep for later

  idx = msg.stoken(sfField, idx, delims, returnEmptyFields); // A / V
  if (sfField != 'A') {
    return false; // not active
  }
  // else A so update time
  parseTime(sfTimeField);
  . . . 
  return true;
}

The idx keeps track of where the last field ended and where to start looking for the next field

The number fields are converted using the SafeString toFloat, toLong, toInt and hexToLong methods which apply strict checking to the input and leave the result unchanged if there are any errors. e.g.

void parseDate(SafeString &dateField) {
  long lDate = 0;
  if (!dateField.toLong(lDate)) {
    return; // invalid
  }
  day = lDate / 10000;
  month = (lDate % 10000) / 100;
  year = (lDate % 100);
}

dateField.toLong(lDate) will return false if the SafeString dateField is not a valid long number. See the SafeStringToNum.ino example sketch in the SafeString library, under SafeString_Tests, for more examples of these conversion methods.

Here is the output for the test input (set the Arduino Monitor to Newline or Both NL & CR)
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77

Set Newline on Monitor and Enter GPS msg $GPxxx,...
 e.g. $GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77 
 > > > 2012/04/16  19:45:8.98  -74 0.42', 40 42.61'

Using SafeStrings to read and parse the input makes the code robust and more readable. Calling SafeString::setOutput(Serial); in setup() will print out a detailed error message if you try and access outside the bounds of the SafeString. If you comment out that setOutput() statement, then the code is still safe. The error checking still takes place and an 'empty' result returned, but no error message is printed.

SafeString substring methods return empty results if the text won't fit or the index is out of range. This together with the strict checking of the hexToLong() method means we can skip some of the checks that would normally need to be done in low level c-string coding to prevent index out of range accesses. At the top of the checkSum() method there are two checks we can ignore because SafeString does that index out of range checking for us.

bool checkSum(SafeString &msg) {
  size_t idxStar = msg.indexOf('*');
  // could do these checks also
  // BUT SafeString will just return empty sfCheckSumHex and so fail to hexToLong conversion below
  //  if (idxStar == msg.length()) {
  //    return false; // missing * //this also checks for empty string
  //  }
  //  // check for 2 chars
  //  if (((msg.length()-1) - idxStar) != 2) { // msg.length() -1 is the last idx of the msg
  //    return false; // too few or to many chars after *
  //  }
  cSF(sfCheckSumHex, 2);
  msg.substring(sfCheckSumHex, idxStar + 1); // next 2 chars SafeString will complain and return empty substring if more than 2 chars or idxStar+1 out of range
  long sum = 0;
  if (!sfCheckSumHex.hexToLong(sum)) {
    return false; // not a valid hex number
  }

For example entering the test input (missing the * check sum)
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A
gives the output, with SafeString::setOutput(Serial); called in setup()

Error: sfReader.substring() beginIdx 68 >  sfReader.length() : 67
        sfReader cap:81 len:67 '$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A'
bad checksum : $GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A

The sfReader.substring() failed because the message did not contain the * char and so msg.indexOf('*') returned an index equal to the length of the msg and so the starting index for the substring was beyond the end of the msg and an empty sfCheckSumHex was returned and an error message output. Then hexToLong fails because the empty sfCheckSumHex does not contain valid hex digits and the checkSum() returns false.

In general you can just code your SafeString processing and call SafeString::setOutput(Serial); called in setup() and let SafeString tell you what is out of range or too large to fit in the SafeStrings you have setup.

BufferedOutput

As well as sending messages to external devices, like GPS modules, and displaying messages for the user, serial text output is the primary means for debugging your sketch to find out what it is doing. However as we will see now, just adding a Serial.print() statement changes the way your sketch runs and can introduce more errors then is finds.

The following trivial sketch, Print_at_9600.ino, illustrates the delays introduced by using Serial.print( ) (and any of the other Serial output methods)

const uint32_t BAUD_RATE = 9600;
unsigned long loopCounter = 0;
unsigned long startTime = 0;
void setup() {
  Serial.begin(BAUD_RATE);
  for (int i = 10; i > 0; i--) {
    Serial.print(' '); Serial.print(i);
    delay(500);
  }
  Serial.println();
  Serial.print("BAUD_RATE = "); Serial.println(BAUD_RATE);
}

void loop() {
  if (loopCounter == 0) {
    startTime = millis();
  }
  loopCounter = loopCounter + 1; // or loopCounter++; i.e. add one each loop
  Serial.println(" a very looooooooooooog msg with looooooooooooots of data");
  Serial.print("Loop:"); Serial.print(loopCounter);
  Serial.print(" mS:");
  Serial.println(millis() - startTime);

  // rest of loop
  delay(1); // assume the rest of your sketch code takes 1mS to execute
}

The first few lines of output are:-

BAUD_RATE = 9600
 a very looooooooooooog msg with looooooooooooots of data
Loop:1 mS:24
 a very looooooooooooog msg with looooooooooooots of data
Loop:2 mS:101
 a very looooooooooooog msg with looooooooooooots of data
Loop:3 mS:178
 a very looooooooooooog msg with looooooooooooots of data
Loop:4 mS:255
 a very looooooooooooog msg with looooooooooooots of data
Loop:5 mS:331
 a very looooooooooooog msg with looooooooooooots of data
Loop:6 mS:408
 a very looooooooooooog msg with looooooooooooots of data
Loop:7 mS:485
. . .

At 9600 baud it takes about 1mS per char for the hardware Serial to output a character, so for this ~80char message it takes about ~80mS for the HardwareSerial UART to send the data. Once the first loop's output fills up the Uno's 63 byte Tx buffer, the Serial.print() blocks waiting for the UART to send a char so there is space in the Tx buffer for the next char. So the rest of the loop is blocked while the print() is blocked waiting for the slow UART to send the chars. As we saw above this can lead to you missing the inputs your program needs.

There are a number of ways you can combat this problem of Serial output delaying the rest of your code.

use a higher Serial baud rate
increase the buffer size and skip some output so Serial does not block using BufferedOutput
print less data

Using a higher Serial baud rate

The first step is to use a higher Serial baud rate. You should always do this. For the Uno you can choose up to 115200 in the Arduino Monitor and increase the BAUD_RATE in the sketch

const uint32_t BAUD_RATE = 115200;

After increasing the baud rate to 115200 (Print_at_115200.ino), first few lines of the output are now :-

BAUD_RATE = 115200
 a very looooooooooooog msg with looooooooooooots of data
Loop:1 mS:2
 a very looooooooooooog msg with looooooooooooots of data
Loop:2 mS:8
 a very looooooooooooog msg with looooooooooooots of data
Loop:3 mS:14
 a very looooooooooooog msg with looooooooooooots of data
Loop:4 mS:20
 a very looooooooooooog msg with looooooooooooots of data
Loop:5 mS:26
 a very looooooooooooog msg with looooooooooooots of data
Loop:6 mS:32
 a very looooooooooooog msg with looooooooooooots of data
Loop:7 mS:39
. . .

The HardwareSerial is still filling up and the Serial.print() is still blocking, but for less time now as the characters are being sent out faster. While using a faster baud rate should always be the first step, it only reduces the problem. It does not remove it.

Increasing the output buffer and skipping some output using BufferedOutput

The BufferedOutput class, in the SafeString library, 'solves' the problem by
1) providing a larger buffer for the output and
2) discarding output that would cause the loop to block.

The sketch BufferedOutput_at_115200.ino does this.

// BufferedOutput_at_115200.ino
#include "BufferedOutput.h"
createBufferedOutput(output, 66, DROP_UNTIL_EMPTY); // modes are DROP_UNTIL_EMPTY, DROP_IF_FULL or BLOCK_IF_FULL

const unsigned long BAUD_RATE = 115200;
unsigned long loopCounter = 0;
unsigned long startTime = 0;
void setup() {
  Serial.begin(BAUD_RATE);
  for (int i = 10; i > 0; i--) {
    Serial.print(' '); Serial.print(i);
    delay(500);
  }
  Serial.println();
  Serial.print("BAUD_RATE = "); Serial.println(BAUD_RATE);
  output.connect(Serial);  // <<<<< connect the buffered output to Serial 
}

void loop() {
  output.nextByteOut(); // <<<<<<<<<< need to call this each loop to release next byte from buffer
  if (loopCounter == 0) {
    startTime = millis();
  }
  loopCounter = loopCounter + 1; // or loopCounter++; i.e. add one each loop
  output.println(" a very looooooooooooog msg with looooooooooooots of data");  // <<< use output instead of Serial 
  output.print("Loop:"); output.print(loopCounter);
  output.print(" mS:");
  output.println(millis()- startTime);
  // rest of loop
  delay(1); // assume the rest of your sketch code takes 1mS to execute
}

There are three addition statements:-

createBufferedOutput(output, 66, DROP_UNTIL_EMPTY);

which creates a 66 byte buffer and a buffered output called output. The mode is DROP_UNTIL_EMPTY which means once both the buffer and the Serial tx buffer fill up any extra output is discarded until they are both empty. By default if a print( ) statement will not all fit in the buffer, none of it will be printed. These settings usually give the most usable output (the other BufferedOutput options are discussed below). In your sketch you then print using output instead of Serial.

output.connect(Serial);

This tells the the buffered output where to send the characters it has buffered.
The other important addition is

output.nextByteOut();

This MUST be called at least once, or more, each loop() to release the buffered characters to Serial. If you forget to add output.nextByteOut() , you will only get the first few print()'s !!

The 66 byte buffer created by createBufferedOutput is added to the existing Serial tx buffer when buffering output. Different Arduino boards have different sizes of Serial tx buffers :-
Uno - 63 byte buffer,
ESP32 - 127 byte buffer,
ESP8266 - 128 byte buffer,
Mega2560 - 63 byte buffer,
Arduino NanoBLE - 0 byte buffer. The NanoBLE is a special case. See connect( ) baud rate option section below.

The first few lines from the BufferedOutput_at_115200.ino sketch displayed on the Arduino monitor are shown below. The ~~<CR><NL> indicates that some output is missing (has been dropped / skipped over).

DROP_UNTIL_EMPTY mode example output

BAUD_RATE = 115200
 a very looooooooooooog msg with looooooooooooots of data
Loop:1 mS:1
 a very looooooooooooog msg with looooooooooooots of data
Loop:2 mS:3
~~
 a very looooooooooooog msg with looooooooooooots of data
Loop:9 mS:13
 a very looooooooooooog msg with looooooooooooots of data
Loop:10 mS:15
~~
 a very looooooooooooog msg with looooooooooooots of data
Loop:17 mS:28
 a very looooooooooooog msg with looooooooooooots of data
Loop:18 mS:30
~~
. . .

This shows the time for a loop() has dropped from ~6mS to ~2mS by skipping output that would cause the print() statement to block.

BufferedOutput Options

The createBufferedOutput( ) macro has three (3) modes and one flag, allOrNothing, which defaults to true.

The modes do the following:-
DROP_UNTIL_EMPTY – (output shown above) once both the buffer and the Serial tx buffer have filled up, then all the output is discarded (skipped/dropped) until both buffers are completely empty. This mode usually gives the cleanest, most readable output.
DROP_IF_FULL – once both the buffer and the Serial tx buffer have filled up then all output is discarded (skipped/dropped) until there is enough space for an more output. See the example output below
BLOCK_IF_FULL – this mode does not skip any output. When the the buffer and the Serial tx buffer are both full, it just blocks the loop() waiting for space to become available to output the next char. (The allOrNothing flag is ignored.)

DROP_IF_FULL mode example output

Changing the createBufferedOutput to createBufferedOutput(output, 66, DROP_IF_FULL); gives

BAUD_RATE = 115200
 a very looooooooooooog msg with looooooooooooots of data
Loop:1 mS:1
 a very looooooooooooog msg with looooooooooooots of data
Loop:2 mS:3
~~

Loop:3 mS:4
~~

Loop:4 mS:6
~~
 . . .

BLOCK_IF_FULL mode example output

Changing the createBufferedOutput to createBufferedOutput(output, 66, BLOCK_IF_FULL); gives output that looks just like the normal Serial.print( ) output. Nothing is skipped and the loop() is blocked waiting for space in the buffers. However because you have added 66 byte buffer space, you can print( ) more data before it starts blocking.

BAUD_RATE = 115200
 a very looooooooooooog msg with looooooooooooots of data
Loop:1 mS:2
 a very looooooooooooog msg with looooooooooooots of data
Loop:2 mS:5
 a very looooooooooooog msg with looooooooooooots of data
Loop:3 mS:8
 a very looooooooooooog msg with looooooooooooots of data
Loop:4 mS:13
 a very looooooooooooog msg with looooooooooooots of data
Loop:5 mS:20
 a very looooooooooooog msg with looooooooooooots of data
Loop:6 mS:27
 a very looooooooooooog msg with looooooooooooots of data
Loop:7 mS:33
 . . .

allOrNothing flag

There is an optional 4^th argument to the createBufferedOutput( ) , the allOrNothing flag. If not specified this flag defaults to true and the DROP_.. modes will not output a write(msg,size) unless there is enough space in the buffer for the whole message. Not all print( ) statements use the underlying write(msg,size) method. print(“message”) and print(10) do, but print(F(“ “)) does not and println(...) adds the newline separately.

Here is some example output for DROP_UNTIL_EMPTY but with the allOrNothing flag false.

createBufferedOutput(output, 66, DROP_UNTIL_EMPTY,false);

As you can see below, when the buffers are almost full the long print( ) is truncated half way through.

BAUD_RATE = 115200
 a very looooooooooooog msg with looooooooooooots of data
Loop:1 mS:0
 a very looooooooooooog msg with looooooooooooots of data
Loop:2 mS:2
 a very looooooooooooog msg ~~
14
 a very looooooooooooog msg with looooooooooooots of data
Loop:11 mS:16
 a very looooooooooooog msg with looooooooooooots of data
Loop:12 mS:18
 a very looooooooooooog ms~~
 a very looooooooooooog msg with looooooooooooots of data
Loop:20 mS:32
 . . .

Help my output is a mess

If you are using DROP_UNTIL_EMPTY or DROP_IF_FULL and your output looks broken up or has odd characters embedded in it, then switch to BLOCK_IF_FULL and see what it looks like. The most likely cause is that somewhere in your sketch you are still calling Serial.print()'s. These Serial.print()'s are inserting chars directly into the Serial tx buffer in front of the BufferedOutput characters. So they will appear in the middle of the buffered output.

It the output using BLOCK_IF_FULL looks clean then the problem is that there are multiple output.print() statements or output.print(F(..)) statements, some of which are being lost when the buffers fill up. In that case use a SafeString to prepare the whole message and then output it in one output.print() statement. Make sure the BufferedOutput size is large enough to take the whole message.

  cSF(msg,80);
  msg.println(" a very looooooooooooog msg with looooooooooooots of data");  
  msg.print("Loop:"); msg.print(loopCounter);
  msg.print(" mS:");
  msg.println(millis()- startTime);
  output.print(msg); // this will print the entire msg or nothing when allOrNothing true, the default

Print less data

In the examples above, output was printed every loop. You can use the millisDelay class, included with the SafeString library to only print output every now and again. See How to code Timers and Delays in Arduino. Another thing you can do, to reduce the output, is to write shorter messages. For example instead of writing “Loop:” you could write “L:” This lets you buffer more messages and output them faster.

Important Messages

If you have an important message to output that you don't want dropped, then you can first call output.clearSpace(len) to throw away some of the currently buffered data (if necessary) to make room for the message. Make sure the size of the BufferedOutput is large enough for the important message.

void printPosition() {
  cSF(results, 50);
  results.concat(F("20")).concat(year).concat('/').concat(month).concat('/').concat(day).concat(F("  "))
  .concat(hour).concat(':').concat(minute).concat(':').concat(seconds).concat(F("  "))
  .concat(longDegs).concat(' ').concat(longMins).concat(F("', ")).concat(latDegs).concat(' ').concat(latMins).concat(F("'")).newline();
  output.clearSpace(results.length()); // only clears space the extra buffer not the Serial tx buffer
  output.print(results);
}

You can also call output.clear() to clear the whole buffer OR call output.flush() to block until all the currently buffered data is output and the buffers are empty.

Very Important Messages

If the message is VERY important then you can use protect() (i.e. output.protect(); after output.print(results); to prevent another clearSpace( ) call from removing any of this message. Also clearSpace() returns the space available for writing the output, so you can check if there is enough space for the results and if not an the results are very important you can either call output.clear() to discard all the currently buffered output (including any protected output) OR call output.flush() which blocks until all currently buffered output is sent. E.g.

void printPosition() {
  cSF(results, 50);
  results.concat(F("20")).concat(year).concat('/').concat(month).concat('/').concat(day).concat(F("  "))
  .concat(hour).concat(':').concat(minute).concat(':').concat(seconds).concat(F("  "))
  .concat(longDegs).concat(' ').concat(longMins).concat(F("', ")).concat(latDegs).concat(' ').concat(latMins).concat(F("'")).newline();
  int avail = output.clearSpace(results.length()); // only clears space the extra buffer not the Serial tx buffer
  if (avail < results.length()) {
      // not enough buffer space for all the results
      output.clear();  // OR output.flush() which blocks!!
  }
  output.print(results);
  output.protect();    // protect these results from other calls to clearSpace( )
}

When using clearSpace(), no need to test the return value IF all these conditions are true
1) the output buffer is big enough AND
2) no other code calls protect() AND
3) this code is not called again before the last protected output is written so that there is enough space for this output

Summary of BufferedOutput Usage

Start by increasing the baud rate of the output serial and adding a BufferedOutput.
For most debug and information messages just call output.print() and see what you get and increase the size of the BufferedOutput if you are loosing output.
For more important debug/informational message, call clearSpace( ) first to give them higher priority over other msgs.
For very important messages call clearSpace() and then protect()
For absolutely must see messages call clearSpace() and check the return value, if that is not enough, work call clear() which dumps all current buffered output.
If you need to see that msg NOW call flush() which will block just like normal Serial until the message is all written.

BUT in most cases increasing the output buffer size and abbreviating msgs solves a lot of problems and the ~~ lets you know where it has not.

connect( ) baud rate option

For connect( ) you usually just pass the Serial object after you have called Serial.begin(115200); i.e.

output.connect(Serial);

In that case the BufferedOutput checks Serial.availableForWrite() each time nextByteOut() is called and releases buffered characters if there is room in the Serial tx buffer.

However some Arduino boards, like the NanoBLE, do not have a Serial tx buffer. For these boards Serial.availableForWrite() always returns 0. It those cases the connect(Serial) will halt the sketch and repeatedly print out

availableForWrite() returns 0
You need to specify the I/O baudRate
and add extra calls to nextByteOut() as only one byte is released each call.

For those boards you need to specify the Serial baud rate in the connect call e.g.

output.connect(Serial,115200);

BufferedOutput will then use a timer to release a character at a time, at that baud rate.

terminateLastLine

One final method available in BufferedOutput is terminateLastLine(). terminateLastLine() adds a <CR><NL> to the output buffer if there is not already one there, so starting a newline if one was not just started.

SafeStringStream, Automated Text Input Testing

It is tiring having keep typing in, or cut and pasting, into the Arduino Monitor to test a sketch with text Input. The SafeString library includes a SafeStringStream that you can use in place of Serial for testing. SafeStringStream wraps a SafeString and releases the data at the specified baud rate.

SafeStringStream versus the Real World

NOTE: SafeStringStream provides a conservative test of your sketch, because a) Real World data streams often have small gaps between messges and b) the SafeStringStream added extra processing with makes your loop() run slower then it would working with real data. So if your sketch works with continous SafeStringStream data, ( i.e. with sfReader.echoOn() ) then it should work in the Real World.

Adding an Rx Buffer to SafeStringStream

By default the SafeStringStream only has a small 8 byte rx buffer. However the Uno/Mega2560 has a 64byte buffer, so for a more realistic test a 64 char SafeString buffer will be added to the SafeStringStream. For ESP32 / ESP8266 use a 128 char SafeString buffer.

cSF(sfTestData, 180); // the test data SafeString, will be filled in setup()
cSF(rxBuf, 64); // the extra rxbuffer for the SafeStringStream to mimic the Uno hardware serial rx buffer
SafeStringStream sfStream(sfTestData,rxBuf); // set the SafeString to be read from and the SafeString providing the rx buffer

Then in setup() initialize the sfTestData SafeString and call
sfStream.begin(sfTestData, TESTING_BAUD_RATE);

void setup() {
  . . . 
  sfReader.connect(sfStream); // read the test data from the SafeStringStream
  sfReader.echoOn(); // echos the read data back to be re-read again
  sfTestData = F(
                 "$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77\n"
                 "$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48\n"
                 "$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47\n"
               ); // initialized the test data  
  Serial.println("Test Data:-");
  Serial.println(sfTestData);
  Serial.println();
  sfStream.begin(sfTestData, TESTING_BAUD_RATE); // start releasing sfTestData at 9600 baud
}

It is important to point to note that sfReader now connects to sfStream to source its input data and that sfReader.echoOn(); Echo on will now write all the input chars back to the end of sfStream, so they are continually re-read. This simulates a continual stream of incoming messages at the TESTING_BAUD_RATE.

The

sfStream.begin(sfTestData, TESTING_BAUD_RATE);

starts releasing the test data at the specified baud rate. This statement should be last in the setup() if you don't want to miss the first few chars.

The SafeStringStream also keeps track of how many chars are lost due to the rx buffer overflowing. sfStream.RxBufferOverflow() returns the current count and clears it.

The sketch SafeStringStream_GPS_9600_testing.ino uses a SafeStringStream to automate the testing and checks for rx buffer overflows.

Here is same sample output:-

Automated Serial testing at 9600
Test Data:-
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47

 sfStreamOv:0
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
 > > > 2012/04/16  19:45:08  -74 0.42', 40 42.61'
 sfStreamOv:0
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
 sfStreamOv:0
$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47
 sfStreamOv:0

Looks good at 9600 baud input. The no chars are being lost from the SafeStringStream's rx buffer.

Lets try a faster baud rate TESTING_BAUD_RATE = 19200 (SafeStringStream_GPS_19200_testing.ino)

Automated Serial testing at 19200
Test Data:-
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47

 sfStreamOv:0
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
 > > > 2012/04/16  19:45:08  -74 0.42', 40 42.61'
 sfStreamOv:48
,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47
!! bad checksum : ,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47
 sfStreamOv:67

Losing Input data due to loop() delays

What is happening? The print's to the Serial are filling up its HardwareSerial Tx buffer and then blocking the loop() waiting for hardware UART to send some characters. The first line of data is read correctly, but by the time the loop() prints out the the message and the processes and prints the results, the sketch has missed reading the half of the next message, and so on. Since only the chars read by SafeStringReader are echoed back to the sfStream for re-reading, once a char is missed it remains missed.

As mentioned above in BufferedOutput, in the Real World the first thing to do is to increase the Serial output baud rate (or you could also reduce the incoming data rate back to 9600). However here, for the purposes of this tutorial, we are going to leave the Serial output baud rate at 9600 and the GPS input data rate at 19200 and instead replace the blocking Serial print()s with a non-blocking BufferedOutput. This is a recommended step because lets you add more debug messages without interfering with the running of your loop().

Applying BufferedOutput to the GPS sketch

The sketch, BufferedOutput_GPS_19200_testing.ino, replaces all the Serial.print()s in the loop() code with prints to a buffered output. Remember to add output.nextByteOut(); at the top of the loop() code. The output is now:-

 10 9 8 7 6 5 4 3 2 1
Automated Serial testing at 19200
 using 80char BufferedOuput and 9600 baud Serial.
Test Data:-
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47


 sfStreamOv:0
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
 > > > 2012/04/16  19:45:08  -74 0.42', 40 42.61'
 sfStreamOv:0
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
 sfStreamOv:0

At 19200 baud GPS data input, no data is being missed now that the blocking Serial.print()s have been removed.

Note carefully you should always use the highest possible baud rate for the information and debug messages output. In the previous examples it was artificially kept at 9600 baud for the purposes of the example. The following examples use 115200 baud.

Adding more Tasks

Just reading the GPS messages may not be the only task you want your sketch to perform. The next example, BufferedOutput_GPS_25mS_19200_testing.ino, the Serial baud rate is increased to 115200 and the GPS msg processing extracted to its own task processGPSInput() and another task, anotherTask() added as well that simulates, with a delay(), a task that takes 25mS to run. See the Simple Multitasking in Arduino tutorial for more details on using tasks for multi-tasking on any Arduino board.

Here is the initial output:-

Automated Serial testing at 19200
 with 80char BufferedOutput, 115200 Serial baud rate
 and anotherTask taking 25mS extra loop processing time
Test Data:-
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47


 sfStreamOv:0
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
 > > > 2012/04/16  19:45:08  -74 0.42', 40 42.61'
 sfStreamOv:0
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
 sfStreamOv:0
$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47
 sfStreamOv:9
4509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
!! bad checksum : 4509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77

The extra processing delay introduced by anotherTask() is preventing the processGPSInput() task from reading the chars before the input 64byte rx buffer overflows and some are lost.

There are two approaches to solve this problem (apart from reducing the GPS input baud rate back to 9600). They are :-

1) add more calls processGPSInput() task in the anotherTask() so it is GPS read is called more often. The Simple Multitasking in Arduino tutorial covers this general approach.
2) add more input buffering to buffer the chars waiting for the processGPSInput() task to read them.

Depending on what anotherTask() is actually doing will determine which approach to use. If you insert a call to processGPSInput() into the middle of anotherTask() that is the easiest solution. The sketch, BufferedOutputExtra_GPS_25mS_19200_testing.ino does this

void anotherTask() {
  delay(13); // simulate more code here
  processGPSInput(); // add an extra call to read and process GPS messages
  delay(12); // simulate more code here // total 25mS
}

The output from BufferedOutputExtra_GPS_25mS_19200_testing.ino no longer misses any chars.

Automated Serial testing at 19200
 with 80char BufferedOutput, 115200 Serial baud rate
 and anotherTask taking 25mS extra loop processing time
 Extra call to processGPSInput() inserted in anotherTask()
Test Data:-
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47


 sfStreamOv:0
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
 > > > 2012/04/16  19:45:08  -74 0.42', 40 42.61'
 sfStreamOv:0
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
 sfStreamOv:0
$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47
 sfStreamOv:0
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
 > > > 2012/04/16  19:45:08  -74 0.42', 40 42.61'
 sfStreamOv:0

BufferedInput

If it is not convenient or possible to add enough extra calls to processGPSInput(), you can add extra an extra rx buffer using the BufferedInput class. The BufferedInput class in the SafeString library allows you to add extra buffering to the built in Rx buffer provide by the HardwareSerial. Note: you can also edit the Arduino library code to do this, but that will change the buffer size for all your sketches. Also, as we will see below, using the BufferedInput class provides debugging information on what is happening.

Going back to BufferedOutput_GPS_25mS_19200.ino, we will start with a small, 20 char, additional input buffer.

createBufferedInput(bufferedIn, 20);

In setup() connect up the bufferedIn

  bufferedIn.connect(sfStream); // add extra buffering to the test data stream, in addition to the 64byte rx buffer already in sfStream
  sfReader.connect(bufferedIn); // read the test data from the extra BufferedInput

and at the top of loop(), to read data into the buffer, add

  bufferedIn.nextByteIn();

BufferedInput Statistics

The BufferedInput class keep some stats on how it is operating. There are three statistics kept, each of which is cleared once it is read by calling it access method:-

maxBufferUsed() returns the maximum number of chars stored in the this BufferedInput since this method was last called. If this equals the size of the BufferedInput, you should increase the buffer size.

maxStreamAvailable() returns the maximum number of chars that where available to be read from the input stream this BufferedInput is connected to. If this equals the size of the HardwareSerial buffer and there is still space in the BufferedInput then you need to call nextByteIn() more often.

The sketch, BufferedInput_20_GPS_25mS_19200_testing.ino, adds a 20 char input buffer and also outputs the BufferedInput stats after each input line.

Automated Serial testing at 19200
 with 80char BufferedOutput and 25mS extra loop processing
 and 20char BufferedInput, 115200 Serial baud rate
 and anotherTask taking 25mS extra loop processing time
Test Data: -
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47


 InBufUsed:20 mxAv:37 sfStreamOv:0
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
 > > > 2012/04/16  19:45:08  -74 0.42', 40 42.61'
 InBufUsed:20 mxAv:48 sfStreamOv:0
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
 InBufUsed:20 mxAv:50 sfStreamOv:0
$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47
 InBufUsed:20 mxAv:40 sfStreamOv:0
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
 > > > 2012/04/16  19:45:08  -74 0.42', 40 42.61'
 InBufUsed:20 mxAv:64 sfStreamOv:0
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
 InBufUsed:20 mxAv:64 sfStreamOv:6
$GPGGA,123519,4807.01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47
!! bad checksum : ~~

The output

 InBufUsed:20 mxAv:64 sfStreamOv:6

indicates that the 64 byte stream buffer filled up (mxAv) and the 20 char BufferedInput buffer also filled up (InBufUsed) so not surprisingly input data was missed. In the Real World the sfStream.RxBufferOverflow() value (sfStreamOv) will not be available so you will need to use just the other two stats.

Since the BufferedInput is filling up, we need to increase it size.

The sketch BufferedInput_30_GPS_25mS_19200_testing.ino increase the BufferedInput size to 30 chars. The output is now:-

Automated Serial testing at 19200
 with 80char BufferedOutput and 25mS extra loop processing
 and 30char BufferedInput, 115200 Serial baud rate
 and anotherTask taking 25mS extra loop processing time
Test Data: -
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47

 InBufUsed:30 mxAv:37 sfStreamOv:0
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
 > > > 2012/04/16  19:45:08  -74 0.42', 40 42.61'
 InBufUsed:30 mxAv:48 sfStreamOv:0
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
 InBufUsed:30 mxAv:40 sfStreamOv:0
$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47
 InBufUsed:30 mxAv:40 sfStreamOv:0
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
 > > > 2012/04/16  19:45:08  -74 0.42', 40 42.61'
 InBufUsed:30 mxAv:54 sfStreamOv:0
$GPVTG,054.7,T,034.4,M,005.5,N,010.2,K*48
 InBufUsed:30 mxAv:60 sfStreamOv:0

The output line

 InBufUsed:30 mxAv:60 sfStreamOv:0

indicates that the sfStream rx 64 char buffer is not filling up and no chars are being lost.

A Real World GPS Example

This GPS parsing and output example uses a Adafruit's GPS feather wing on an Adafruit 32u4 Feather, but it DOES NOT use the Adafruit GPS library. If you read the comments in Adafruits GPS library, you will see notes about double buffering the GPS input and not printing too much output and setting the Serial output to 115000 for a GPS baud rate of 9600. All of this is to cover over the fact that Serial blocks once the Tx buffer fills up and your sketch starts missing the incoming GPS messages. These problems disappear when you uses SafeStrings and BufferedOutput.

The sketch Serial1_GPS_SafeStringReader.ino, uses a 115200 baud rate for the Arduino monitor messages as recommended above. The sketch not only receives and parses GPS messages, but at the same time handles user input commands. As an illustration the sketch has processes user commands dms and degs to switch the format of the Lat/Long output between Degs Mins Secs and decimal Degs. The loop() is arranged as four (4) tasks, see Simple Multi-tasking in Arduino for a full tutorial on Arduino tasks.

void loop() {
  output.nextByteOut();
  handleCMDS();
  GPS_serial.nextByteOut();
  handleGPS();
}

The handleGPS() task is similar to the one in the previous SafeString_GPS.ino sketch. The new handleCMDS() task is

void handleGPS() {
  waitingForFix();
  if (sfGPS_Reader.read()) {
    sfGPS_Reader.trim(); // remove and leading/trailing white space
    if (checkSum(sfGPS_Reader)) { // if the check sum is OK
      if (sfGPS_Reader.startsWith("$GPRMC,")) {  // this is the one we want
        output.println(sfGPS_Reader);
        if (parseGPRMC(sfGPS_Reader)) {
          haveFix = true;
          printPosition(); // print new data
        }
      } else {  // ignore but print first 10 chars
        cSF(tmp, 10);
        output.print(sfGPS_Reader.substring(tmp, 0, 7)); output.println("...");
      }
    } else {
      output.clearSpace(12); // make space for at least the start of this error
      output.print("!! bad checksum : "); output.println(sfGPS_Reader);
    }
  } // else token is empty
}

You can easily extend the range of user commands handled and the range of GPS messages parsed. Unlike other GPS libraries, the Serial1_GPS_SafeStringReader.ino does not use any low level c-string functions or char[], which are very prone to programming errors. Here some sample output showing the user commands being handled to change the format.

Serial and GPS_serial both running at 9600
User commands are:-
 dms for output in degs min'ss.ss
 degs for output in decimal degs.degs
 . . . 
$GPRMC,235950.800,V,,,,,0.00,0.00,050180,,,N*41
Waiting for Fix..
 . . . 
$GPGGA,...
$GPGSA,...
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
 > > > 2012/4/16  19:45:8.98  Position (Lat Long): 74 00'25.2"S 40 42'36.6"E
loop uS Latency
 5sec max:12064 avg:138
 sofar max:12064 avg:138 max - prt:2800
 . . . 
degs
 Setting DEGS mode.
$GPGGA,...
$GPGSA,...
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
 > > > 2012/4/16  19:45:8.98  Position (Lat Long): -74.007000, 40.710167
 . . .
dms
 Setting DMS mode.
$GPGGA,...
$GPGSA,...
$GPRMC,194509.000,A,4042.6142,N,07400.4168,W,2.03,221.11,160412,,,A*77
 > > > 2012/4/16  19:45:8.98  Position (Lat Long): 74 00'25.2"S 40 42'36.6"E
 . . .
hms -- invalid. Only dms or degs are valid commands
$GPGGA,...

Conclusion

This tutorial showed how to handle processing Real World text I/O. Often all you need is to set the baud rate of your Serial connection as high as possible and use a SafeStringReader for input and BufferedOutput for output to avoid missing any incoming data and SafeString for processing to avoid coding errors

The tutorial also covered how to test that your sketch can handle the expected data rate using test data in a SafeStringStream. If the rest of your loop() processing takes so long that you are missing input, the BufferedInput class lets you add more input buffering and provides statistics to help you choose the correct buffer size.

For use of the Arduino name see http://arduino.cc/en/Main/FAQ

Arduino Serial I/O for the Real World Non-Blocking Text I/O using the SafeString library

Arduino Text I/O that Works in the Real WorldHow use Text I/O for Control and DebuggingGet text input, parse it and output debug msgswithout blocking the rest of the loop.