I3DCONVERTER FOR GOOGLE GENERATOR
I3DCONVERTER FOR GOOGLE CODE

Inspired by Matt Harveys blog post + repository: Five video classification methods implemented in Keras and TensorFlow (Mar 2017).This project is licensed under the MIT License - see the LICENSE file for details Acknowledgments The neural network model is not included in this GitHub repo (too large) but can be downloaded here (150 MB). and uses the neural network to predict the sign language gesture.calculates and displays the optical flow,.

captures 5 seconds of video (using videocapture.py),.

I3DCONVERTER FOR GOOGLE CODE

Note: the code files containing '_mobile_lstm' are used for an alternative NN architecture, see here.

I3DCONVERTER FOR GOOGLE GENERATOR

Training requires a GPU and is performed through a generator which is provided in datagenerator.py. First only the (randomized) top layers are trained, then the entire (pre-trained) network is fine-tuned.ДЄ pre-trained 3D convolutional neural network, I3D, developed in 2017 by Deepmind is used, see here and model_i3d.py. Optical flow is very effective for this type of video classification, but also very calculation intensive, see here. Opticalflow.py calculates optical flow from the image frames of a video (and stores them on disc).

Frames are resized/cropped to 240x320 pixels.

40 frames per training/test videos (on average 5 seconds duration = approx 8 frames per second).

See pipeline_i3d.py for the parameters used for the ChaLearn dataset: Prepare_chalearn.py is used to unzip the videos and sort them by labels (using Keras best-practise 1 folder = 1 label): Prepare the video data Extract image frames from videosДЇrame.py extracts image frames from each video (using OpenCV) and stores them on disc. The ChaLearn video descriptions and labels (for train, validation and test data) can be found here: data_set/chalearn See here for overview of suitable data-sets for sign-language for deaf people: Дownload the ChaLearn Isolated Gesture Recognition dataset here: (you need to register first) The live demo works on an ordinary laptop (without GPU), eg MacBook Pro, i5, 8GB.

RequirementsДЇor the training of the neural networks a GPU is necessary (eg aws p2.xlarge).

Built with Python, Keras+Tensorflow and OpenCV (for video capturing and manipulation)ДЇor 10-slide presentation + 1-min demo video see here.

Based on deep learning techniques, in particular convolutional neural networks (including state-of-the-art 3D model) and recurrent neural networks (LSTM).

Includes all code to prepare data (eg from ChaLearn dataset), extract features, train neural network, and predict signs during live demo.

This prototype 'understands' sign language for deaf people.

I3dconverter for google