Making Things Complicated
In one of the previous posts, we explained the initial version of the remote control system, implemented for our robot. It was based on a simple protocol and used RoboRemo application as a remote control user interface.The simplicity of the solution posed some limits on the functions. At some point, we decided to make a step forward and implement a more complicated remote control version.
Mainly - we were looking for:
- FPV (First Person View) - the possibility to see the real-time video from the camera installed at the robot
- Usage of joystick or steering wheel to control the robot
- Having a big dashboard with many small controls which can operate numerous features of the robot and indicate on the screen the real status which is based on telemetry
- Possibility to see all the traffic between the robot and the remote control application, having the possibility to type commands manually
The architecture is still the same. First of all - it is defined by the ESP13 shield functionality.This is a board which we use to connect the robot to the Wi-Fi.
From the Arduino point of view - the robot's code receives all the commands through a serial UART port - this is the port which is physically connected to the ESP13 shield. Responses can be sent back by writing characters to this port.
From the user side - we can use any application (remote control dashboard) which works using telnet-like protocol. Simply speaking - this application should just connect to the specific IP address and port and be able to read and write plain text, sending commands and reading responses from the robot.
ESP13 takes care about all the magic which ties everything together. ESP13 is an autonomous module which has own microcontroller on board. This microcontroller is actually even more powerful than our main Arduino. It works with the Wi-Fi module, handles all the data exchange and even has its own web server on board. You can connect to this web server with your favorite web browser and configure all the specifics of the ESP13 work, precisely the same way, as you configure your home Wi-Fi router.
For more detailed information about how everything works - please check our previous post.
In this post, we will give an insight on the more sophisticated techniques used to implement the new features. Also, you will find explanations on how to integrate our code with your project and how to extend it with your own commands and controls.
You don't need to learn all the details if your plan to start using the software as is. Just download the binaries archive and and try it.
The remote control application can be downloaded here: https://github.com/rmaryan/roboremoteFPV
The corresponding robot's code is here: https://github.com/rmaryan/ardurobot
Protocol
The commands protocol still relies on a simple plain text. Commands and responses can be multiple-character strings which are supposed to be human-readable, to make debugging easier.
| Command From RC | Action | Response from the Robot (on success) |
| MI | Mode-Idle | MI |
| MA | Mode-AI | MA |
| MS | Mode-Scenario | MS |
| MR | Mode-Remote Control | MR |
| W | Move Forward | - |
| S | Move Backward | - |
| A | Turn Left | - |
| D | Turn Right | - |
| LF | Toggle Front Lights | LF1 or LF0 |
| LR | Toggle Rear Lights | LR1 or LR0 |
| LS | Toggle Side Lights | LS1 or LS0 |
| R0 | Refresh Distances (with the head turn) | RLxx - left distance,
where xx is a distance in centimeters with the leading zeros. "RL-- "
means "not available" RFxx - front distance RRxx - right distance ROABCD - obstacle detectors state ("1"s or "0"s), A - left-ahead, B - left-edge, C - right-edge, D - right-ahead |
| R1 | Refresh Distances (without the head turn) | same as above |
| XAAABBB | Set the drives speed to AAA (left) and BBB (right). The speed is in range 000 - 511. 000 is the full reverse, 511 means full ahead. |
- |
| - | Debug message from the robot. All messages with the leading tilde sign shall be shown on the remote control dashboard. | ~message text |
All commands and responses must be ended with the line-end character "\n".
Lx commands are listed here but not used yet (as of version 1.0 of the application). Our lights are not installed yet on the robot.
The Robot-side Code
Thanks to the ESP8266 аnd ESP13 "magic" there is no need to invent anything complicated at the robot's Arduino side. All we need is to read the command strings from the serial port and write the answers to it. Since we use Arduino MEGA 2560, we could spare a dedicated serial port just for this role. Arduino supports working with the serial ports right out of the box with the help of standard library Serial.Sending responses is the easiest thing. Call to the Serial.write(str) sends any message from the robot to the remote control application without delays and struggles.
Getting remote commands is a bit harder. While we were using single-character commands (in the initial version), we could just ask serial port from time to time if it happen to have any incoming character for us. And if it has - read it. So the command receiving code looked like this:
int remoteCommand = 0; if (Serial3.available() > 0) { remoteCommand = Serial3.read(); if (remoteCommand == 'M') <handle the command M> ... }
This will not work for the multiple-character commands.
You may think that you can use method Serial.readStringUntil('\n') which loads characters from the serial port until an end of message character is received (in our case it is '\n').
But that will not work as well. Receiving characters from the network and feeding them to the robot takes some time. The serial port is quite a slow thing. If you add also the need to receive these characters from the network - it may potentially take SECONDS. For all this time the Arduino code will stop working and will be waiting until the whole string will be received. And all this time your robot will be unconscious, maybe riding at full speed right into the wall, and having no chance to react in any way. As you remember - we even had to invent a special multitasking architecture to avoid such blackouts.
The only way to handle this properly is to read characters one by one from the serial port (as we did previously) and aggregate them in a special buffer up to the moment when the end of message character arrives. When the whole string is built - it is sent to the AI module for the further processing.
To make the serial port code easier to create and debug, we isolated it into a separate module - class RobotConnector.
RobotConnector inherits features from TaskInterface thus it can work as a "task" in our system, taking care about all the steps required to handle serial port data exchange timely.
The AI module can call RobotConnector using public methods:
- virtual bool commandAvailable() - checks if there is any command received completely and ready to be processed
- virtual char* getCommand() - retrieves the reference to the received command
- virtual void sendMessage(const char* message) - to be used for sending responses back from the robot
Now the AI command processing code looks like this:
if (robotConnector->commandAvailable()) { remoteCommand = robotConnector->getCommand(); if (remoteCommand[0] == 'M') <handle the command M> ... }
As soon as AI receives a whole command in a form of the regular C/C++ string (array of characters followed by a zero byte) it can be processed using standard C string.h functions.
Here is the code from RobotConnector which collects the incoming string and assembles it character by character in a special buffer, which later can be consumed by the AI module.
// processTask() method is called automatically for all "tasks" // from the main robot's loop (see robot.ino) void RobotConnector::processTask() { // do nothing if the buffer already contains a complete message // which was not processed yet if (!messageReady) { // did the next character arrived? while (Serial3.available() > 0) { // read the character int incomingByte = Serial3.read(); switch (incomingByte) { case -1: // something did not work - nothing to do break; case COMMAND_TERM: // we received a message ending character // zero is an end of line in C // here we are making our buffer // to be a proper C string messageBuffer[messageSize++] = 0; messageReady = true; break; default: // this is just a regular character from some message // don't forget to check for the buffer overflow if (messageSize >= (MAX_MSG_SIZE - 1)) { // this should never happen, // but if it is - // just forget everything we received // and start over messageSize = 0; } // adding the character to the resulting string messageBuffer[messageSize++] = incomingByte; break; } } } }
You can read more about the serial ports and multiple-character strings at the Arduino forum or at the Nick Gammon's forum.
Remote Control Application
System Requirements and Installation Steps
We created the remote control application as an independent project RoboRemoteFPV. This program is free and can be used and distributed under the terms of the GNU General Public License (GPL) as published by the Free Software Foundation (FSF). Do with our code whatever you like, but if you decide to re-distribute it, your users should get the same rights as you initially received from us.
RoboRemoteFPV is a desktop application. This is the most efficient way to utilize all the inches of the computer monitor. Use of the Java programming language with the Swing GUI library ensures you can run this application on any operating system capable to host up to date Java runtime environment. We use Swing instead of JavaFX, to make it easier running the application, if you prefer OpenJDK to the Oracle versions.
We tested RoboRemoteFPV under Windows only, but nothing prevents it from being used on MacOS or GNU/Linux.
For simple installation - download a zip archive with the binaries. Extract it to any convenient place. Run command like the following:
java -cp "jinput.jar;jna-4.1.0.jar;jna-platform-4.1.0.jar;slf4j-api-1.7.22.jar; slf4j-simple-1.7.22.jar;vlcj-3.10.1.jar;roboremote-1.0.jar" com.github.roboremote.RoboRemote
In Windows environment, you can run roboremote.cmd from the distribution to save some typing.
RoboRemoteFPV uses VLC media player to show the video. If you have it installed - most probably RoboRemoteFPV can connect to it automatically. If your Java is 64-bit - make sure you installed also a 64-bit version of VLC.
Usage
Here is a typical work session:
We assume you will specify the robot's video stream as the video URL, but you are free to use any streaming source - even listen to your favorite radio station. RoboRemoteFPV can play everything consumable by VLC.
RoboRemoteFPV Code Structure
Most probably you would choose to adjust our code per your needs. We were always trying to stay on the side of simplicity. So even the very basic Java knowledge from the Java Tutorial will be sufficient to change the UI or add own commands.
We use Eclipse to edit Java code. Mostly because it also works with the C/C++ projects and through the Sloeber plugin can upload firmware directly to the Arduino microcontrollers. If you are not sure how to set up your development environment - start from the Java Tutorial. You must learn the basics first.
Information from this chapter will save you some time if you need to learn the RoboRemoteFPV code structure, pointing to the most important areas.
The application consists of six modules.
RoboRemote.java is an entry point. The essence of this module is concentrated in just three lines:
RoboCommandsModel.loadSettings(); // load the application settings // from the storage hidManager = new HIDManager(); // create the object which handles // joysticks and other game controllers mainWindow = new MainWindow(); // create the main window
RoboConnManager.java is a class which knows how to connect to the robot, feed commands and receive responses.
Key methods:
- void connect(String address) - connects to the robot at the address specified
- void disconnect() - disconnects from the robot
- void sendMessage(String message) - sends a command to the robot
- void addMessageListener(RoboMessageListener listener) - registers a listener for the incoming messages
The first three methods look simple. They do their work instantly.
The fourth method is a standard trick common to the Java+Swing world. Since we don't know when exactly we are going to receive any response from the robot, we propose external consumers to register a listener. When some message from the robot arrives - RoboConnManager notifies immediately everyone, who subscribed.
The listener must be a class which implements the following interface:
interface RoboMessageListener { void messageReceived(final String message); void disconnected(); }
When a class "implements interface" it means that class must have all the methods defined by the interface. RoboConnManager calls the corresponding method of the registered listener depending on the events with the robot's connection.
HIDManager.java is technically very similar to RoboConnManager. It does not send any messages. Instead, it constantly monitors the state of joysticks and other game controllers and instantly notifies listeners if anything happened.
A listener can be registered calling method addEventListener(HIDEventListener l). And the listener class must implement an interface with a single method, which will be called each time user moves the joystick or presses a button:
interface HIDEventListener { void actionPerformed(Event e); }
Additionally HIDManager can return a list of the connected joysticks and game controllers, and choose which one to use (methods getControllersList(), getSelectedController(), setSelectedController()).
To make it simple - we do not allow using multiple game controllers at the same time.
RoboCommandsModel.java incorporates all information about the commands which can be sent to the robot according to the protocol. Also, it stores user settings, which can be changed in the preferences dialog (key bindings, IP addresses of the robot and video streams, active joystick).
Most probably - structure CommandsList would be the first thing you'd like to play with:
public static enum CommandsList { CMD_FORWARD("Forward", KeyEvent.VK_W, "W"), CMD_REVERSE("Reverse", KeyEvent.VK_S, "S"), CMD_LEFT("Left", KeyEvent.VK_A, "A"), CMD_RIGHT("Right", KeyEvent.VK_D, "D"), CMD_LIGHTS_FRONT("Lights Front", KeyEvent.VK_Z, "LF"), CMD_LIGHTS_REAR("Lights Rear", KeyEvent.VK_X, "LR"), CMD_LIGHTS_SIDES("Lights Sides", KeyEvent.VK_C, "LS"), CMD_MODE_IDLE("Mode Idle", KeyEvent.VK_1, "MI"), CMD_MODE_AI("Mode AI", KeyEvent.VK_2, "MA"), CMD_MODE_SCENARIO("Mode Scenario", KeyEvent.VK_3, "MS"), CMD_MODE_RC("Mode RC", KeyEvent.VK_4, "MR"), CMD_RESCAN_DISTANCES("Rescan Distances", KeyEvent.VK_E, "R"); …
This is a foundation of the robot remote control functions. For each command this structure stores:
- Name of the command which is shown in the user interface (for example "Lights Front")
- Default keyboard shortcut in Java notation (for example KeyEvent.VK_Z - means "Z"-key)
- Command characters which must be sent to the robot when action is activated (for the command above it is "LF")
MainWindow.java is the largest and the scariest module. It creates the main window with all its widgets and components.
Key methods:
createAndShowGUI() - the entry point for all visual elements creation. This method is called when a main window is being created. Then it relies on the supplementing methods to do the job:
- initializeVideoPane() - video panel on the upper-left
- initializeMovementPane() - movement control panel and the bottom-left
- initializeLightsPane() - lights control panel
- initializeConnectPane() - connection and video stream selection panel
- initializeLogPane() - log messages panel
- initializeDistancePane() - distance sensors panel
- initializeModePane() - working modes panel
- initializeButtonsPane() - buttons panel at the bottom
initializeActions() - initializes the keyboard, joystick and robot's messages listeners. Here we also initialize a timer which reads the position of the speed and turning controls every 100 milliseconds and sends the corresponding command to the robot. The same timer also reloads information from the robot's distance sensors (every 500 milliseconds).
generateMotorsCommand() - this is a tricky code which translates the joystick stick position to the commands which are being sent to the motors. If you think for a second about this - it's a quite complicated task. That is the code which determines how does the robot's reaction feel like. Is it sharp or smooth? Is it shaky? Nervous? Unpredictable? The logic is here. And we are still not happy about it.
processUserCommand(CommandRecord cRec) - dispatches all the actions which flow from the user interface and translates them into the commands which are sent to the robot.
processRobotMessage(String message) - contains a code which handles messages received from the robot. It turns red and green indicators on a screen and updates the distances, received from the sensors.
Other methods are also important, but it is unlikely you will need to change them.
Module PrefDialog.java Shows the preferences dialog. The entry point is here - method showDialog().
All user settings are loaded from the module RoboCommandsModel. If changed - they are stored there back (see the method actionPerformed()).
First Person View
Right from the beginning, we decided to use the wireless IP web cameras technology to stream the video from the robot. This is much simpler and way cheaper, comparing to the real-time FPV systems used by the RC enthusiasts.
It is quite easy to find a small and reliable camera for our purpose. For example like this:
A major requirement for such camera is support of the RTSP protocol. This ensures you can use any generic media player (like VLC) to see the video. Also you should think about how to connect this camera to the wireless network (it supports only wired connection).
When we almost purchased the camera, a much better solution was accidentally discovered. It was a time to retire my old cell phone - Motorola Defy. As you maybe know - it is almost impossible to kill a Motorola phone. In case of Defy - it even more than that. A special strong and waterproof body, high-capacity battery, scratch-resistant screen… No wonder, after 5 years of dutiful service it was still as good as new (well… almost :-)
We decided to use this cell phone as an autonomous web camera with own power source and Wi-Fi connection. As a nice bonus - in addition to a connected camera, we've got an independent computing module, which can take care of really complicated tasks, connect to the robot's main board over Bluetooth and perform lots of other interesting things. That is for sure a very promising direction to explore in the future.
To secure a cell phone on top of the robot, we made a simple cradle (old plastic bottle shaped with the help of a knife and scissors). Cradle is bolted to the upper robot's plate.
Cell phone fits into the cradle firmly and securely. Later we will reduce the height - the camera should control the space right in front of the robot.
Drawbacks
We have to mention couple drawbacks RoboRemoteFPV has:
- The video stream is displayed with a slight delay. You still can comfortably control your robot, but the response is not instant. Radio-transmitted FPV video, for example, is much more responsive. This delay is caused by the network latency. We reduced it as much as we could, playing with the VLC parameters in the MainWindow.java:
String[] standardMediaOptions = {":network-caching=10", ":live-caching=10", ":rtsp-caching=10"};
If you find a better parameters combination - please share what you've got. - Translation of the joystick position to the motors rotation speed values is also not perfect. The reaction still feels clunky. Motors do not respond smoothly to the small power applied to them. Also, casual joysticks are not precise. You need to assume some "dead zone" near the zero point and close to the edges to make the robot reaction "natural". All this logic resides in the generateMotorsCommand(). This code for sure deserves more polishing.
Looking Forward
Our robot evolves and is getting cooler and smarter. With the new remote control system, it is quite amusing to play with.
In the next phases we plan to:
- Complete the robot illumination (headlights, rear lights, etc)
- Remove breadboard and rewire all components neatly and accurately
- Remove the supplementary power bank and USB cable, and power Arduino from the main onboard battery
We are anxious to start experimenting with the complicated and fancy AI techniques using our robot as a playground. But at this point, we need to tidy up everything a bit. All those wires and a breadboard do not look nice.
No comments:
Post a Comment