Framework for Accelerometer Based Gesture Recognition and Integration with Desktop Applications
This project aims at providing simplified alternative ways
of interacting in the form of gestures with the desktop based applications which
requires user inputs through traditional devices like keyboard and mouse. The
input is in the form of basic hand based gestures generated with the help of a
smartphone device held in hand.
Accelerometers which are tiny sensors, used to measure the acceleration in the x, y and z coordinates comes embedded in most of the smartphones available today. The raw data due to the user gestures are obtained from these accelerometer sensors present in the smartphone devices and it is refined into specific gesture information.
A short distance communication protocol called Bluetooth is used to transmit the accelerometer data from the smartphone device to the desktop at a constant rate, making it a completely wireless system.
The accelerometer data received at the desktop computer is converted into useful events like key events and mouse events which in turn can be used as input data to control the various applications and games on the desktop computers.
This framework makes desktop applications and games more interactive, intuitive and lively. This also enables development of more creative games and applications, which is an exciting way of being engaged.
Accelerometers which are tiny sensors, used to measure the acceleration in the x, y and z coordinates comes embedded in most of the smartphones available today. The raw data due to the user gestures are obtained from these accelerometer sensors present in the smartphone devices and it is refined into specific gesture information.
A short distance communication protocol called Bluetooth is used to transmit the accelerometer data from the smartphone device to the desktop at a constant rate, making it a completely wireless system.
The accelerometer data received at the desktop computer is converted into useful events like key events and mouse events which in turn can be used as input data to control the various applications and games on the desktop computers.
This framework makes desktop applications and games more interactive, intuitive and lively. This also enables development of more creative games and applications, which is an exciting way of being engaged.
Demo - Accelerometer Based Gaming - Integration with NFS on Desktop
Demo - Accelerometer Based Painting - Integration with Windows Paint
Racing Games Interface
The racing games are recreational and indulging applications which require the users to control and steer the virtual car on the screen. The applications should be able to interpret gestures and execute desired functionality. If the gesture is not identified the application will continue to be in its last well known state until the next gesture is identified. These applications are also used as virtual assistants to provide training for car driving.
The racing applications comprises of three modules.
- Accelerometer module on mobile device.
- Bluetooth server running on user’s PC
- Bluetooth Client running on mobile device.
Paint brush
Paint brush is an accelerometer controlled painting application, Like The racing games application this application can also be integrated with any existing painting applications like Microsoft Paint, Kids Draw etc.
Hence this is again a generic application which can be interfaced with any existing painting applications. To draw any drawing user just needs to tilt his mobile device in appropriate directions. As the user tilts his mobile device in appropriate directions the generated accelerometer data is sent to Bluetooth server running on user’s computer through Bluetooth client running on user’s device. Later this accelerometer data is processed and corresponding mouse move events are generated to draw drawings in paint application.
Hence this is again a generic application which can be interfaced with any existing painting applications. To draw any drawing user just needs to tilt his mobile device in appropriate directions. As the user tilts his mobile device in appropriate directions the generated accelerometer data is sent to Bluetooth server running on user’s computer through Bluetooth client running on user’s device. Later this accelerometer data is processed and corresponding mouse move events are generated to draw drawings in paint application.
The paint brush application consists of three modules.
- Accelerometer module on mobile device.
- Bluetooth server running on user’s PC
- Bluetooth Client running on mobile device.
Speech Synthesizer
Objective of this project is to design and implement a system for personalizing the phone with voice commands. Speech recognition has had many benefits and is boon in many scenarios in today's technology. Having them implemented on today's savvy phones has added to these benefits. Speech recognition and hence command execution finds many use cases. For example it is of real benefit to people in traffic when typing is difficult and speaking a command is more convenient. It is invaluable to people with disabilities, scenario in which they can just utter the command.Implementing the speech recognition and command on Symbian (QT based)phones has been both enduring and learning process throughout. Symbian uses an inherent API for speech recognition. It also makes use of Voice XML to implement and attain the desired conversion from Speech to Text.From the implementation point of view the real challenge has been the new QT interface and its voice command execution. As QT has its own speech recognition API but this has not been ported/used with Symbian OS as the inclusion of it would cause considerable increase in the library size ported.
The implementation has also included using the proprietary Home Screen API for adding various widgets of news, entertainment, messaging etc. And enabling them to receive, convert and hence execute the voice commands. Lot of work has also been done to understand the vast architecture of Symbian and its capabilities and its short comings towards the Speech recognition technology as a whole. Many new and exciting features can be added as future endeavor on top of the present voice recognition facility with the smart phones. Like locking/unlocking the phones. Command based video control (like play,pause,stop). Enabling voice control to maybe enable news reading of some popular news website on the go.
Summary :
Implementing the Speech recognition for QT based phones has greatly solved and improved the performance of the QT phones. It will make the present NOKIA phones more competitive and attractive because of the ease of its usage. It has solved few limitations of the NOKIA phones to a large extent. It has also laid the basic foundation through which future developers can enhance and greatly improve the performance of NOKIA phones and the residing software within.
This endeavor also lays the foundation for a better and more enhanced QT UI for speech recognition. Thereby having a dedicated home screen/UI for the same. To achieve the goal of introducing new voice commands , an interface was created with QT and the Symbian Architecture dealing with voice command initiative and recognition. Considerable amount of work was put into designing the Home screen with using the home screen API s. So that UI is voice savvy. On the whole the speech recognition implementation is an honest endeavor to improve the performance of NOKIA phones and make it a pleasure to use and explore.
This endeavor also lays the foundation for a better and more enhanced QT UI for speech recognition. Thereby having a dedicated home screen/UI for the same. To achieve the goal of introducing new voice commands , an interface was created with QT and the Symbian Architecture dealing with voice command initiative and recognition. Considerable amount of work was put into designing the Home screen with using the home screen API s. So that UI is voice savvy. On the whole the speech recognition implementation is an honest endeavor to improve the performance of NOKIA phones and make it a pleasure to use and explore.
SMS Notifier
This project aims at "Achieving maximum functionality with minimum human interactions" , by specifically focusing on the the "Messaging" functionality. As we have often seen a "Voice Call" is prioritized higher than a text message, and since a voice call enables the user to get the information of the caller by either displaying a "Photo" or "Contact Number" of the caller. The user can decide either to answer the call or disconnect based on user's interest.
However, if a user receives a message, there is no way to get information about the sender until the message is opened and read. Consider the following scenario,
Problem :
User is busy with his work, but expecting an important message from a person. Here the user can't afford to examine all the incoming messages untill he receives "The message of interest" by the concerned person.
Solution :
SMS notifier is an application, built to help a busy user to get all the informations about the sender of the message without even, " Unlocking the Phone or Opening the message" .
How does it work ?
When a message arrives,
However, if a user receives a message, there is no way to get information about the sender until the message is opened and read. Consider the following scenario,
Problem :
User is busy with his work, but expecting an important message from a person. Here the user can't afford to examine all the incoming messages untill he receives "The message of interest" by the concerned person.
Solution :
SMS notifier is an application, built to help a busy user to get all the informations about the sender of the message without even, " Unlocking the Phone or Opening the message" .
How does it work ?
When a message arrives,
- The application, reads the "Mobile Number" of the sender from the message.
- Searches, the contact, in the contacts list.
- Gets the name, associated photo and other contact details of the sender.
- Displays all these information on the screen, by creating a widget.
- The user is notified about this event with a beep or vibration.
- Now the user can decide, to read the message by unlocking the phone and opening the message.
- Or, user can ignore the message, and defer it. for later reading.