# SPARSE-V1.0
===============

State of the art research in AI for the autonomous systems space requires the collection of large, annotated datasets of real-world scenarios, especially for edge case events. Take for example, autonomous driving, an edge case might be that while driving, the system encounters a situation in which boxes have suddenly fallen out of the bed of a truck, requiring quick action by the autonomous system. It is difficult to collect enough data for these edge cases in the real world, since by definition, they are edge cases.

The Unreal Game Engine provides the perfect environment to simulate realistic driving situations in real-time with minimal costs and with no safety risks.
In a simulated environment edge cases can be easily tested repeatedly,  accruing the large datasets needed to train an autonomous system.

&nbsp;
## Goals
-------

Design a realistic autonomous AI simulator that addresses the shortcomings of current systems.

Including:
* A realistic environment and vehicle.
* Real-time simulation speed.
* Programmatic input from a python client.

Screenshot:
<img src="https://i.ibb.co/88CPq4r/SPARSE-pic-2.png" alt="alt text" width="400">

&nbsp;
## Implimentation
-------------------

* We started with Unreal’s C++ Vehicle Template .
* To control the Unreal simulation from a python client we added an RPC library called “rpclib”.
* In the RPC server we bound different functions from the Unreal vehicle pawn so they could be called from the client.
* We then created 3 cameras to record the visual data from the Vehicles perspective.
* We optimized and added assets for the vehicle and for the environment.

Architecture:
<img src="https://i.ibb.co/Hx2nCsy/SPARSE-architecture.png" alt="alt text" width="600">


Simplified Class Diagram:
<img src="https://i.ibb.co/znwqLJH/SPARSE-simple-Class-diagram.png" alt="alt text" width="600">

&nbsp;
## Prerequisites
-----------------

## Software:
* Unreal engine 4.20.3
* Visual Studios 2017
* Workload: Game development with C++
* Rpclib 2.2.1
* Python 3
* mprpc
* Inputs
* keyboard

## Hardware:
* Gamepad that works on a PC
* Ideally a decent GPU

&nbsp;
## Installing
---------------

* First Install Visual Studio 2017 x64 and the Workload: “Game development with C++”
* Now install Unreal Engine 4.20
* Install python 3
* Use “pip3 install” to install mprpc, inputs and keyboard.
* Clone from Bitbucket for current master: https://sp-jira.cis.fiu.edu:7443/scm/sprs/sparse-v1.git
* Get and Build rpclib for you OS in 64bit. Get the headers and lib file and place it in \sparse-v1\Plugins\SPARSE\Source\ThirdParty\rpclib
* In the root of the project right click the .uproject file and click “Generate Visual Studio project files”
* Now open the SPARSEV1.uproject, if everything is done correctly the project will rebuild and open the Unreal Editor

Watch the Installation Guide Video (https://www.youtube.com/watch?v=eLpgsMmjE3w)

&nbsp;
## User Guide
-----------------

A normal user of SPARSE is an AI Developer. SPARSE allows the user to control a car (Audi Q7) in potential any Unreal environment. They can give input to the car through both a python script and by controlling the vechile using a normal controller. Both methods use a python client to communicate with a RPCserver that runs in the Unreal Engine. 
SPARSE still needs a way to extract visual data in an effiecent manner that an AI can consume. Without this feature 

This allows control over the network. The user will have to set the IP address of the computer running the SPARSE simulation in client.py.

PORT = 18800
IPADDRESS = 'YOUR_IP_ADRESS'

Because SPARSE is contained in a Unreal plugin it is simple to copy the plugin over to another Unreal project if you want to use some other environment or game logic.

Changing the environment(map) is quite simple and there are a lot of options to start with in the Unreal Marketplace https://www.unrealengine.com/marketplace/content-cat/assets/environments. Users can also build there own environment with the Unreal Editor.

This video show how to change the environment in the Unreal editor: (https://www.youtube.com/watch?v=SWBTSbpx11A)

&nbsp;
### Use cases:

<img src="https://i.ibb.co/2ZF0WHf/SPARSE-Use-Case.png" alt="alt text" width="600">
Above is the Use Cases that SPARSE currently fills


&nbsp;
### Gamepad Controls:

<img src="https://i.ibb.co/QbHpGdV/SPARSE-PS-controls.png" alt="alt text" width="600">
Here we show the controls avaliable for aa Playstation remote but all controllers will bascilly use the same buttons. (FYI: to use a Playstation remote on a Windows PC a special driver must be install both http://ds4windows.com/ and https://inputmapper.com/ are good options)

&nbsp;
### Python Controls:

<img src="https://i.ibb.co/dWmzyG0/SPARSE-python-controls.png" alt="alt text" width="600">
Here are the inputs that be used from a python script. For a bit more detail see python/client.py
&nbsp;

Watch the User Guide Video (https://www.youtube.com/watch?v=RQqHjTVNODs)


&nbsp;
## Shortcomings
----------------

* No Visual Data Extraction of front cameras
* Unreal’s documentation is not very good.
* Building C++ libraries can be very annoying 
* Adding libraries to Unreal build system can take some learning. I recommend developers take a look at the Third Party Library plugin template within Unreal.

&nbsp;
## Wishlist
------------

* Create an rtsp stream with frames from the front camera's an AI would consume with information to make decisions.
* Testing randomly generated events as examples edge cases that are uncommon in real life: ex. box falling from a vehicle in front or an animal runs across the road
* Both Mac and Windows should be able to use this software, but no testing was done.
* More optimization and speed
* More quality / varying environments with good performance 
* Upgrade the materials

&nbsp;
## Built With
-------------

* [Unreal Engine](https://docs.unrealengine.com/en-us/) - The Game Engine used
* [Visual Studios 2017](https://visualstudio.microsoft.com/) - IDE, C++ compiler
* [python 3](https://www.python.org/download/releases/3.0/) - Client
* [rpclib](https://maven.apache.org/) - C++ rpc library
* [mprpc](https://rometools.github.io/rome/) - Python rpc library
* [inputs](https://pypi.org/project/inputs/) - Python input library

&nbsp;
## Authors
----------

* Team Members: Nicolas Gonzalez, Emmanuel Perez
* Product Owner(s): Miguel Alonso Jr
* Instructor: Masoud Sadjadi

&nbsp;
## License
----------

This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details

&nbsp;
## Acknowledgments
-------------------

* Thanks to our Product Owner Dr. Miguel Alonso Jr
* and Profoessor Masoud Sadajdi
* Carla http://carla.org/
* rpclib http://rpclib.net/
* Unreal Engine


