Overview¶
In this Notebook, we are going to setup the framework for using a feature-based, supervised learning model to classify our gesture sets. While in the last assignment you built up familiarity with the SciPy libraries and an initial understanding of shape-based gesture classification, in this Notebook you will begin to learn:
- How to use Scikit-learn, one of the most popular open-source machine learning libraries in the world
- How to rethink the classification problem not just in terms of signal shapes (as before) but in terms of signal features. You will brainstorm, visualize, and analyze both time-domain and frequency-domain features extracted from the gesture accelerometer data and learn how to find and then use potentially promising features in your classifier
- How to setup a classification experiment using Scikit-learn (you don't have to rely on your own k-fold cross validation code, for example)
- How to use some of the built-in supervised learning models in Scikit-learn to classify gestures
Scikit learn¶
We are going to shift from using our own classifiers and experimental testbed (say goodbye to our custom gesturerec.experiments library) to a popular open-source machine learning library called Scikit-learn (code repo). Scikit-learn (or sometimes called sklearn) started as a Google Summer of Code project and its name "SciKit" stems from using the SciPy libraries as its foundation (though its development was and is independent).
We chose Scikit-learn as our primary machine learing library because:
- It is built on NumPy, SciPy, and matplotlib (so you need not learn another set of Python libraries)
- It is one of the most popular machine learning libraries in the world. On GitHub, for example, it has 25k+ commits and 40.8k stars (link). Building familiarity and comfort with Scikit-learn will not only increase your understanding of applied machine learning but also allow you to more easily use other machine learning toolkits in the future (e.g., PyTorch, TensorFlow).
While this Notebook will cover some initial, core introductory concepts of using Scikit-learn—particularly for a gesture dataset—we are only scratching the surface of possibilities. You may want to consult the official Scikit-learn tutorials either before or after working through this Notebook. Because of Scikit's popularity, there are also numerous (wonderful) tutorials online.
Your TODOs¶
As before, you should use this Notebook as your template for the A4 assignment. We expect that you will read through and interact with cells sequentially. To help guide where you need to work, we've added "TODOs"—so search for that word. :)
Resources¶
- Scikit-learn tutorials, Scikit-learn
- Introduction to Machine Learning with Python, Andreas C. Müller & Sarah Guido, Oct 2016
About this notebook¶
This Notebook was designed and written by Professor Jon E. Froehlich at the University of Washington along with feedback from students. It is made available freely online as an open educational resource at the teaching website: https://makeabilitylab.github.io/physcomp/.
The website, Notebook code, and Arduino code are all open source using the MIT license.
Please file a GitHub Issue or Pull Request for changes/comments or email me directly.
Imports¶
# This cell includes the major classes used in our classification analyses
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D # for 3D plotting, woohoo!
import numpy as np
import scipy as sp
from scipy import signal
import random
import os
import math
import itertools
from IPython.display import display_html
# We wrote this gesturerec package for the class
# It provides some useful data structures for the accelerometer signal
# and running experiments so you can focus on writing classification code,
# evaluating your solutions, and iterating
import gesturerec.utility as grutils
import gesturerec.data as grdata
import gesturerec.vis as grvis
from gesturerec.data import SensorData
from gesturerec.data import GestureSet
# Scikit-learn stuff
from sklearn import svm
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report, accuracy_score, confusion_matrix
from sklearn.model_selection import cross_val_score, cross_validate, cross_val_predict, StratifiedKFold
# Import Pandas: https://pandas.pydata.org/
import pandas as pd
# Import Seaborn: https://seaborn.pydata.org/
import seaborn as sns
Utility functions¶
def display_tables_side_by_side(df1, df2, n = None, df1_caption = "Caption table 1", df2_caption = "Caption table 2"):
'''Displays the two tables side-by-side'''
if n is not None:
df1 = df1.head(n)
df2 = df2.head(n)
# Solution from https://stackoverflow.com/a/50899244
df1_styler = df1.style.set_table_attributes("style='display:inline; margin:10px'").set_caption(df1_caption)
df2_styler = df2.style.set_table_attributes("style='display:inline'").set_caption(df2_caption)
display_html(df1_styler._repr_html_()+df2_styler._repr_html_(), raw=True)
def print_folds(cross_validator, X, y_true, trial_indices):
'''Prints out the k-fold splits'''
fold_cnt = 0
for train_index, test_index in cross_validator.split(X, y_true):
X_train, X_test = X.iloc[train_index], X.iloc[test_index]
y_train, y_test = y_true.iloc[train_index], y_true.iloc[test_index]
print("TEST FOLD {}".format(fold_cnt))
for i in test_index:
print("\t{} {}".format(y_true[i], trial_indices[i]))
fold_cnt += 1
def display_folds(cross_validator, X, y_true, trial_indices):
map_fold_to_class_labels = dict()
fold_cnt = 0
for train_index, test_index in cross_validator.split(X, y_true):
X_train, X_test = X.iloc[train_index], X.iloc[test_index]
y_train, y_test = y_true.iloc[train_index], y_true.iloc[test_index]
class_labels = []
for i in test_index:
class_labels.append(f"{y_true[i]} {trial_indices[i]}")
map_fold_to_class_labels[f"Fold {fold_cnt}"] = class_labels
fold_cnt += 1
df = pd.DataFrame(map_fold_to_class_labels)
display(df)
def compute_fft(s, sampling_rate, n = None, scale_amplitudes = True):
'''Computes an FFT on signal s using numpy.fft.fft.
Parameters:
s (np.array): the signal
sampling_rate (num): sampling rate
n (integer): If n is smaller than the length of the input, the input is cropped. If n is
larger, the input is padded with zeros. If n is not given, the length of the input signal
is used (i.e., len(s))
scale_amplitudes (boolean): If true, the spectrum amplitudes are scaled by 2/len(s)
'''
if n == None:
n = len(s)
fft_result = np.fft.fft(s, n)
num_freq_bins = len(fft_result)
fft_freqs = np.fft.fftfreq(num_freq_bins, d = 1 / sampling_rate)
half_freq_bins = num_freq_bins // 2
fft_freqs = fft_freqs[:half_freq_bins]
fft_result = fft_result[:half_freq_bins]
fft_amplitudes = np.abs(fft_result)
if scale_amplitudes is True:
fft_amplitudes = 2 * fft_amplitudes / (len(s))
return (fft_freqs, fft_amplitudes)
def get_top_n_frequency_peaks(n, freqs, amplitudes, min_amplitude_threshold = None):
''' Finds the top N frequencies and returns a sorted list of tuples (freq, amplitudes) '''
# Use SciPy signal.find_peaks to find the frequency peaks
# JonTODO: in future, could add in support for min horizontal distance so we don't find peaks close together
# SciPy's find_peaks supports this, so would be straightforward to implement
fft_peaks_indices, fft_peaks_props = sp.signal.find_peaks(amplitudes, height = min_amplitude_threshold)
freqs_at_peaks = freqs[fft_peaks_indices]
amplitudes_at_peaks = amplitudes[fft_peaks_indices]
if n < len(amplitudes_at_peaks):
ind = np.argpartition(amplitudes_at_peaks, -n)[-n:] # from https://stackoverflow.com/a/23734295
ind_sorted_by_coef = ind[np.argsort(-amplitudes_at_peaks[ind])] # reverse sort indices
else:
ind_sorted_by_coef = np.argsort(-amplitudes_at_peaks)
return_list = list(zip(freqs_at_peaks[ind_sorted_by_coef], amplitudes_at_peaks[ind_sorted_by_coef]))
return return_list
map_marker_to_desc = {
".":"point",
",":"pixel",
"o":"circle",
"v":"triangle_down",
"^":"triangle_up",
"<":"triangle_left",
">":"triangle_right",
"1":"tri_down",
"2":"tri_up",
"3":"tri_left",
"4":"tri_right",
"8":"octagon",
"s":"square",
"p":"pentagon",
"*":"star",
"h":"hexagon1",
"H":"hexagon2",
"+":"plus",
"D":"diamond",
"d":"thin_diamond",
"|":"vline",
"_":"hline"
}
plot_markers = ['o','v','^','<','>','s','p','P','*','h','X','D','d','|','_',0,1,2,3,4,5,6,7,8,9,10,'1','2','3','4',',']
def plot_feature_1d(gesture_set, extract_feature_func, title = None, use_random_y_jitter = True,
xlim = None):
'''
Plots the extracted feature on a 1-dimensional plot. We use a random y-jitter
to make the values more noticeable
Parameters:
gesture_set: the GestureSet class
extract_feature_func: a "pointer" to a function that accepts a trial.accel object and returns an extracted feature
title: the graph title
use_random_y_jitter: provides a random y jitter to make it easier to see values
xlim: set the x range of the graph
'''
markers = list(map_marker_to_desc.keys())
random.Random(3).shuffle(markers)
marker = itertools.cycle(markers)
plt.figure(figsize=(12, 3))
for gesture_name in selected_gesture_set.get_gesture_names_sorted():
trials = selected_gesture_set.map_gestures_to_trials[gesture_name]
x = list(extract_feature_func(trial.accel) for trial in trials)
y = None
if use_random_y_jitter:
y = np.random.rand(len(x))
else:
y = np.zeros(len(x))
marker_sizes = [200] * len(x) # make the marker sizes larger
plt.scatter(x, y, alpha=0.65, marker=next(marker),
s = marker_sizes, label=gesture_name)
plt.ylim((0,3))
if xlim is not None:
plt.xlim(xlim)
if use_random_y_jitter:
plt.ylabel("Ignore the y-axis")
plt.legend(bbox_to_anchor=(1,1))
if title is None:
title = f"1D plot of {extract_feature_func.__name__}"
plt.title(title)
plt.show()
def plot_feature_2d(gesture_set, extract_feature_func1, extract_feature_func2,
xlabel = "Feature 1", ylabel = "Feature 2",
title = None, xlim = None):
'''
Plots the two extracted features on a 2-dimensional plot.
Parameters:
gesture_set: the GestureSet class
extract_feature_func1: a "pointer" to a function that accepts a trial.accel object and returns an extracted feature
extract_feature_func2: a "pointer" to a function that accepts a trial.accel object and returns an extracted feature
title: the graph title
xlim: set the x range of the graph
'''
markers = list(map_marker_to_desc.keys())
random.Random(3).shuffle(markers)
marker = itertools.cycle(markers)
plt.figure(figsize=(12, 5))
for gesture_name in selected_gesture_set.get_gesture_names_sorted():
trials = selected_gesture_set.map_gestures_to_trials[gesture_name]
x = list(extract_feature_func1(trial.accel) for trial in trials)
y = list(extract_feature_func2(trial.accel) for trial in trials)
marker_sizes = [200] * len(x) # make the marker sizes larger
plt.scatter(x, y, alpha=0.65, marker=next(marker),
s = marker_sizes, label=gesture_name)
plt.xlabel(xlabel)
plt.ylabel(ylabel)
if xlim is not None:
plt.xlim(xlim)
plt.legend(bbox_to_anchor=(1,1))
plt.title(title)
plt.show()
def plot_feature_3d(gesture_set, extract_feature_func1, extract_feature_func2,
extract_feature_func3, xlabel = "Feature 1", ylabel = "Feature 2",
zlabel = "Feature 2", title = None, figsize=(12,9)):
'''
Plots the two extracted features on a 2-dimensional plot.
Parameters:
gesture_set: the GestureSet class
extract_feature_func1: a "pointer" to a function that accepts a trial.accel object and returns an extracted feature
extract_feature_func2: a "pointer" to a function that accepts a trial.accel object and returns an extracted feature
extract_feature_func3: a "pointer" to a function that accepts a trial.accel object and returns an extracted feature
title: the graph title
xlim: set the x range of the graph
'''
markers = list(map_marker_to_desc.keys())
random.Random(3).shuffle(markers)
marker = itertools.cycle(markers)
fig = plt.figure(figsize=figsize)
ax = Axes3D(fig)
for gesture_name in selected_gesture_set.get_gesture_names_sorted():
trials = selected_gesture_set.map_gestures_to_trials[gesture_name]
x = list(extract_feature_func1(trial.accel) for trial in trials)
y = list(extract_feature_func2(trial.accel) for trial in trials)
z = list(extract_feature_func2(trial.accel) for trial in trials)
marker_sizes = [200] * len(x) # make the marker sizes larger
ax.scatter(x, y, z, alpha=0.65, marker=next(marker),
s = marker_sizes, label=gesture_name)
ax.set_xlabel(xlabel)
ax.set_ylabel(ylabel)
ax.set_zlabel(zlabel)
ax.legend()
ax.set_title(title)
return fig, ax
def plot_bar_graph(d, title=None, ylabel=None, xlabel=None):
'''
Plots a bar graph of of the values in d (with the keys as names)
'''
sorted_tuple_list = sorted(d.items(), key=lambda x: x[1])
n_groups = len(d)
sorted_keys = []
sorted_values = []
for k, v in sorted_tuple_list:
sorted_keys.append(k)
sorted_values.append(v)
# create plot
fig_height = max(n_groups * 0.5, 5)
plt.figure(figsize=(12, fig_height))
indices = np.arange(len(sorted_keys))
plt.grid(zorder=0)
bars = plt.barh(indices, sorted_values, alpha=0.8, color='b', zorder=3)
plt.ylabel(xlabel)
plt.xlabel(ylabel)
plt.xlim(0, sorted_values[-1] * 1.1)
plt.title(title)
plt.yticks(indices, sorted_keys)
for i, v in enumerate(sorted_values):
plt.text(v + 0.01, i, "{:0.2f}".format(v), color='black', fontweight='bold')
plt.tight_layout()
plt.show()
def plot_signals(gesture_set, signal_var_names = ['x', 'y', 'z']):
'''Plots the gesture set as a grid given the signal_var_names'''
num_rows = len(gesture_set.map_gestures_to_trials)
num_cols = len(signal_var_names)
row_height = 3.5
fig, axes = plt.subplots(num_rows, num_cols, figsize=(15, row_height * num_rows))
fig.subplots_adjust(hspace=0.5)
index = 0
for row, gesture_name in enumerate(gesture_set.get_gesture_names_sorted()):
gesture_trials = gesture_set.get_trials_for_gesture(gesture_name)
for trial in gesture_trials:
for col, signal_var_name in enumerate(signal_var_names):
s = getattr(trial.accel, signal_var_name)
axes[row][col].plot(s, alpha=0.7, label=f"Trial {trial.trial_num}")
axes[row][col].set_title(f"{gesture_name}: {signal_var_name}")
axes[row][col].legend()
fig.tight_layout(pad=2)
def plot_signals_aligned(gesture_set, signal_var_names = ['x', 'y', 'z'], title_fontsize=8):
'''Aligns each signal using cross correlation and then plots them'''
num_rows = len(gesture_set.map_gestures_to_trials)
num_cols = len(signal_var_names)
row_height = 3.5
fig, axes = plt.subplots(num_rows, num_cols, figsize=(15, row_height * num_rows))
fig.subplots_adjust(hspace=0.5)
index = 0
for row, gesture_name in enumerate(gesture_set.get_gesture_names_sorted()):
gesture_trials = gesture_set.get_trials_for_gesture(gesture_name)
for col, signal_var_name in enumerate(signal_var_names):
# Find the maximum length signal. We need this to pad all
# signals to the same length (a requirement of cross correlation)
signal_lengths = []
for trial in gesture_trials:
s = getattr(trial.accel, signal_var_name)
signal_lengths.append(len(s))
max_trial_length = np.max(signal_lengths)
# Actually pad the signals
signals = []
for trial in gesture_trials:
s = getattr(trial.accel, signal_var_name)
padding_length = max_trial_length - len(s)
if padding_length > 0:
pad_amount_left = (math.floor(padding_length/2.0))
pad_amount_right = (math.ceil(padding_length/2.0))
padded_s = np.pad(s, (pad_amount_left, pad_amount_right), mode = 'mean')
signals.append(padded_s)
else:
signals.append(s)
# Grab a signal to align everything to. We could more carefully choose
# this signal to be the closest signal to the average aggregate... but this
# should do for now
golden_signal = signals[0] # the signal to align everything to
# Align all the signals and store them in aligned_signals
aligned_signals = [golden_signal]
for i in range(1, len(signals)):
a = golden_signal
b = signals[i]
correlate_result = np.correlate(a, b, 'full')
best_correlation_index = np.argmax(correlate_result)
shift_amount = (-len(a) + 1) + best_correlation_index
b_shifted_mean_fill = shift_array(b, shift_amount, np.mean(b))
aligned_signals.append(b_shifted_mean_fill)
# Plot the aligned signals
for signal_index, trial in enumerate(gesture_trials):
s = aligned_signals[signal_index]
axes[row][col].plot(s, alpha=0.7, label=f"Trial {trial.trial_num}")
axes[row][col].set_title(f"Aligned {gesture_name}: {signal_var_name}", fontsize=title_fontsize)
axes[row][col].legend()
fig.tight_layout(pad=2)
def shift_array(arr, shift_amount, fill_value = np.nan):
'''Shifts the array either left or right by the shift_amount (which can be negative or positive)
From: https://stackoverflow.com/a/42642326
'''
result = np.empty_like(arr)
if shift_amount > 0:
result[:shift_amount] = fill_value
result[shift_amount:] = arr[:-shift_amount]
elif shift_amount < 0:
result[shift_amount:] = fill_value
result[:shift_amount] = arr[-shift_amount:]
else:
result[:] = arr
return result
Load the data¶
These cells are the same as for the Shape Matching notebook. You should not need to edit them, only run them.
# Load the data
#root_gesture_log_path = './GestureLogsADXL335'
root_gesture_log_path = './GestureLogs'
print("Found the following gesture log sub-directories")
print(grutils.get_immediate_subdirectories(root_gesture_log_path))
gesture_log_paths = grutils.get_immediate_subdirectories(root_gesture_log_path)
map_gesture_sets = dict()
selected_gesture_set = None
for gesture_log_path in gesture_log_paths:
path_to_gesture_log = os.path.join(root_gesture_log_path, gesture_log_path)
print("Creating a GestureSet object for path '{}'".format(path_to_gesture_log))
gesture_set = GestureSet(path_to_gesture_log)
gesture_set.load()
map_gesture_sets[gesture_set.name] = gesture_set
if selected_gesture_set is None:
# Since we load multiple gesture sets and often want to just visualize and explore
# one set, in particular, we set a selected_gesture_set variable here
# Feel free to change this
#selected_gesture_set = get_random_gesture_set(map_gesture_sets)
selected_gesture_set = grdata.get_gesture_set_with_str(map_gesture_sets, "Jon")
if selected_gesture_set is None:
# if the selected gesture set is still None
selected_gesture_set = grdata.get_random_gesture_set(map_gesture_sets);
print("The selected gesture set:", selected_gesture_set)
The map_gesture_sets is a dict object and is our primary data structure: it maps gesture dir names to GestureSet objects. There's truly nothing special here. But we realize our data structures do require a learning ramp-up. Let's iterate through the GestureSets.
print(f"We have {len(map_gesture_sets)} gesture sets:")
for gesture_set_name, gesture_set in map_gesture_sets.items():
print(f" {gesture_set_name} with {len(gesture_set.get_all_trials())} trials")
# Feel free to change the selected_gesture_set. It's just a convenient variable
# to explore one gesture set at a time
print(f"The selected gesture set is: {selected_gesture_set.name}")
Preprocess the data¶
You may or may not want to revisit how you preprocess your data. Remember: we are using a fundamentally different approach for classification, so it's worth reconsidering your full data analysis pipeline.
def preprocess_signal(s):
'''Preprocesses the signal'''
# TODO: write your preprocessing code here. We'll do something very simple for now,
mean_filter_window_size = 10
processed_signal = np.convolve(s,
np.ones((mean_filter_window_size,))/mean_filter_window_size,
mode='valid')
return processed_signal
def preprocess_trial(trial):
'''Processess the given trial'''
trial.accel.x_p = preprocess_signal(trial.accel.x)
trial.accel.y_p = preprocess_signal(trial.accel.y)
trial.accel.z_p = preprocess_signal(trial.accel.z)
trial.accel.mag_p = preprocess_signal(trial.accel.mag)
for gesture_set in map_gesture_sets.values():
for gesture_name, trials in gesture_set.map_gestures_to_trials.items():
for trial in trials:
preprocess_trial(trial)
Feature exploration¶
We are going to start this adventure, as we always should, by trying to better understand our signals and the features we plan to extract.
Our goal is to brainstorm and identify highly discriminable attributes of each gesture class that we may be able to leverage in our feature-based classifiers.
While our classifiers will work in high-dimensions, humans, at best can really only visualize and interpret at most 3-dimensions.
So, we're going to start by graphing our features along 1-dimension. Yes, 1-dimension! To make it easier to see patterns in our data, we will add in a bit of y-jitter, so the graphs below may appear 2-dimensional—but they are not. Ignore the y-axis for these 1D plots.
Time domain¶
Let's explore some time domain features first.
Visualizing time-domain signals¶
Just like with shape-matching, let's begin by graphing the time-domain signals.
Visualizing raw x, y, z, and mag¶
plot_signals(selected_gesture_set, ['x', 'y', 'z', 'mag'])
Visualizing aligned raw signals¶
It may be easier to pick out interesting patterns with the signals aligned.
plot_signals_aligned(selected_gesture_set, ['x', 'y', 'z', 'mag'])
Visualizing processed x, y, z, and mag signals¶
Now let's visualize the processed signals.
plot_signals(selected_gesture_set, ['x_p', 'y_p', 'z_p', 'mag_p'])
Visualizing aligned processed signals¶
plot_signals_aligned(selected_gesture_set, ['x_p', 'y_p', 'z_p', 'mag_p'])
Extract and visualize time-domain features¶
After reviewing the above graphs and thinking about what features we might be able to extract from the time-domain signals, let's extract and plot some specific time-domain features. We've created a helper function called plot_feature_1d, which will plot a specified feature in a 1-dimensional graph.
Let's check it out!
Feature: standard deviation of accel mag¶
# Write a simple anonymous function that takes in an accel.trial and returns
# an extracted feature. In this case, the standard deviation of the magnitude
extract_feature_std_mag = lambda accel_trial: np.std(accel_trial.mag)
# We could also have done the following but it's slightly messier for our purposes
# def extract_feature_std_mag(accel_trial):
# return np.std(accel_trial.mag)
# Plot the feature (but with no y-jitter)
graph_title = "1D plot of the standard deviation of the accel mag"
plot_feature_1d(selected_gesture_set, extract_feature_std_mag,
title = graph_title, use_random_y_jitter = False)
To make it easier to see the 1-dimensional spacing, by default, we add in a random y-jitter (which is a common graphing trick). But just remember, the y-values here are meaningless. We'll make real 2-dimensional plots a bit further down.
So, let's graph the same thing but with the random y jitter turned on.
graph_title = "1D plot of the standard deviation of the accel mag"
plot_feature_1d(selected_gesture_set, extract_feature_std_mag, title = graph_title)
Even with just one feature and a 1D plot, we can begin to see some separation between gestures:
- Unsurprisingly, the At Rest gesture has the least amount of deviation
- And then there are ~2 or maybe 3 other groups: a middle group of Clockwise 'O', Counter-clockwise 'O', Backhand Tennis, a less defined intermediate group of Forehand Tennis and Underhand Bowling, and a more extreme group of Shake, Midair Zorro 'Z', and Baseball Throw
Feature: maximum accel mag¶
How about the maximum of the accelerometer magnitude in each gesture trial? Is there discriminable information there?
Let's check it out!
extract_feature_max_mag = lambda accel_trial: np.max(accel_trial.mag)
plot_feature_1d(selected_gesture_set, extract_feature_max_mag,
title = "1D plot of the max accel mag")
What do you observe? Again, this feature seems useful—at least for discriminating some gestures like At Rest and Baseball Throw. As before, there are also clear clusters that are emerging.
Feature: num peaks¶
Finally, let's count the number of peaks (given a certain height and distance threshold). Unlike our other input features thus far, count data is discrete (not continuous), is limited to non-negative values, and will have a different distribution.
def extract_feature_num_peaks_mag_p(accel_trial):
mag_p = sp.signal.detrend(accel_trial.mag_p)
# height is the height of the peaks and distance is the minimum horizontal distance
# (in samples) between peaks. Feel free to play with these thresholds
peak_indices, peak_properties = sp.signal.find_peaks(mag_p, height=100, distance=5)
return len(peak_indices)
plot_feature_1d(selected_gesture_set, extract_feature_num_peaks_mag_p,
title = "1D plot of the num of peaks in mag_p")
Brainstorm and plot your own time domain features¶
Below, brainstorm and plot your own time domain features. We've provided an initial list of features to try but dig into the signals, think about what makes each gesture class unique, and try to extract and leverage those unique aspects as features.
You could extract (many of) these features either from the raw accelerometer signal (e.g., x, y, z, and mag) or a preprocessed version.
- Signal length
- Descriptive statistics of each axis (x, y, z) + magnitude, including mean, median, standard deviation
- Peak counting (raw number of peaks, number of peaks over certain threshold)
- Horizontal distance between peaks, avg vertical distance between peaks
- Number of zero crossings
- Other things! Read papers, brainstorm, visualize!
Once you've exhausted our list and your own mind, consult the web. Use Google Scholar, for example, to find gesture recognition papers: what features did they use?
# We'll start with another one here: signal length but add more!
extract_feature_signal_length = lambda accel_trial: len(accel_trial.x)
plot_feature_1d(selected_gesture_set, extract_feature_signal_length,
title = "1D plot of the signal length")
Frequency domain¶
Of course, there may also be discriminable information in the frequency domain. If you haven't already, complete our Frequency Analysis lesson before moving forward.
Just as we brainstormed time domain features, how about features in the frequency domain, such as:
- The top frequency (i.e., the frequency with the highest amplitude in the signal)
- The top N frequencies (overall or within some frequency band)
- The lowest/highest frequency (over some amplitude threshold)
- Number of frequency peaks (over a certain amplitude threshold)
- Avg value of frequency peaks
- Or perhaps a sliding window that breaks up the signal into chunks and extracts frequency information over each chunk
- And many others...
Visualizing frequency information¶
Let's start—as we always should—by plotting our signals and exploring some initial stats.
Below, we plot a visualization grid where each column is a gesture trial and rows alternate between the time domain and frequency domain versions of the mag for that gesture.
# This cell will take a bit of time not because of the frequency analysis but because
# of preparing the large matplotlib image
import IPython.display as ipd
import ipywidgets
num_cols = 5 # 1 col for each gesture trial
num_rows = selected_gesture_set.get_num_gestures() * 2 # 1 for waveform, 1 for frequency per gesture
row_height = 3
fig, axes = plt.subplots(num_rows, num_cols, figsize=(15, row_height * num_rows))
index = 0
sampling_rate = 92 # roughly 92
progress_bar = ipywidgets.IntProgress(value=0, min=0, max=num_rows)
ipd.display(progress_bar)
title_font_size = 8
for gesture_name in selected_gesture_set.get_gesture_names_sorted():
gesture_trials = selected_gesture_set.get_trials_for_gesture(gesture_name)
for trial in gesture_trials:
s = trial.accel.mag_p # change this to visualize/explore other signals
ax = axes[index][trial.trial_num]
# remove "DC offset" to perform FFT
s_removed_offset = s - np.mean(s)
ax.plot(s_removed_offset, alpha=0.7, label=f"Trial {trial.trial_num}")
ax.set_title(f"{gesture_name[0:12]} waveform", fontsize= title_font_size)
ax.legend()
sampling_rate = math.ceil(trial.accel.sampling_rate)
ax = axes[index + 1][trial.trial_num]
spectrum, freqs_of_spectrum, line = ax.magnitude_spectrum(s_removed_offset, Fs = sampling_rate,
color='r', pad_to = 4 * len(s))
ax.set_title(f"{gesture_name[0:12]} freq spectrum", fontsize= title_font_size)
# Annotate the top freq (by amplitude) on each graph
top_n_freq_with_amplitudes = get_top_n_frequency_peaks(1, freqs_of_spectrum, spectrum)
top_freq = top_n_freq_with_amplitudes[0][0]
top_amplitude = top_n_freq_with_amplitudes[0][1]
ax.plot(top_freq, top_amplitude, marker="x", color="black", alpha=0.8)
ax.text(top_freq, top_amplitude, f"{top_freq:0.1f} Hz", color="black")
index = index + 2
progress_bar.value = index
fig.tight_layout(pad=1)
print(f"Selected gesture set: {selected_gesture_set.name}")
Feature: top frequency¶
As one example, let's extract the most prominent (highest-amplitude) frequency for each gesture using the magnitude part of the accelerometer signal.
def extract_feature_top_mag_freq(accel_trial):
sampling_rate = math.ceil(accel_trial.sampling_rate)
(fft_freqs, fft_amplitudes) = compute_fft(accel_trial.mag, sampling_rate)
top_n_freq_with_amplitudes = get_top_n_frequency_peaks(1, fft_freqs, fft_amplitudes)
if len(top_n_freq_with_amplitudes) <= 0:
return 0
return top_n_freq_with_amplitudes[0][0]
plot_feature_1d(selected_gesture_set, extract_feature_top_mag_freq,
title = "1D plot of the most prominent raw mag freq in each signal")
You might be surprised to see At Rest's position in the graph. This is because we do not take into account the strength of the frequency component for each signal. The identified frequencies in At Rest are low amplitude (and largely noise).
To control for this, we can add in a parameter to get_top_n_frequency_peaks to filter out identified frequencies of less than a certain amplitude coefficient. In this case, let's try 500 but you should experiment with any thresholds used in your code.
def extract_feature_top_mag_freq2(accel_trial):
sampling_rate = math.ceil(accel_trial.sampling_rate)
(fft_freqs, fft_amplitudes) = compute_fft(accel_trial.mag, sampling_rate)
top_n_freq_with_amplitudes = get_top_n_frequency_peaks(1, fft_freqs, fft_amplitudes, min_amplitude_threshold = 500)
if len(top_n_freq_with_amplitudes) <= 0:
return 0
return top_n_freq_with_amplitudes[0][0]
plot_feature_1d(selected_gesture_set, extract_feature_top_mag_freq2,
title = "1D plot of the most prominent raw mag freq in each signal (with min amplitude 500)")
We are able to see roughly four groupings here:
- The At Rest group
- A second cluster with frequencies between ~0.5 and 2.5 Hz
- Possibly a third cluster with Zorro and Forehand Tennis between ~2.5 and 5 Hz
- And another clear outlier cluster for Shake with freq between ~5-15 Hz
Let's zoom into the x-axis a bit to explore the middle cluster more closely.
plot_feature_1d(selected_gesture_set, extract_feature_top_mag_freq2,
title = "1D plot of the most prominent raw mag freq in each signal (with min amplitude 500)",
xlim = (0, 5))
We can, of course, perform frequency analyses on any other signal—be it x, y, z, mag or some processed version. For example, here's a frequency analysis of a preprocessed mag_p signal.
def extract_feature_top_mag_p_freq(accel_trial):
sampling_rate = math.ceil(accel_trial.sampling_rate)
(fft_freqs, fft_amplitudes) = compute_fft(accel_trial.mag_p, sampling_rate)
top_n_freq_with_amplitudes = get_top_n_frequency_peaks(1, fft_freqs, fft_amplitudes, min_amplitude_threshold = 500)
if len(top_n_freq_with_amplitudes) <= 0:
return 0
return top_n_freq_with_amplitudes[0][0]
plot_feature_1d(selected_gesture_set, extract_feature_top_mag_p_freq,
title = "1D plot of the most prominent processed mag freq in each signal")
Brainstorm and plot your own frequency domain features¶
What other features can you think of for the frequency domain?
# TODO: Graph additional frequency-based features
2-dimensional explorations¶
OK, now that we've built up some familiarity with feature extraction and how these features may enable us to discriminate gestures, let's plot various combinations of them in 2D. Again, remember that our classifiers will (often) work in N-dimensions where N is the number of features. But it's just not possible for us to visualize all of our features at once in an N-dimensional graph.
But at least we're going from 1D to 2D, wee! :)
We'll begin by just randomly choosing some features to plot together: how about the standard deviation of the accel mag and the max mag?
plot_feature_2d(selected_gesture_set, extract_feature_std_mag, extract_feature_max_mag,
xlabel="std_mag", ylabel="max_mag", title="Std mag vs. max mag")
Again, our goal here is to see clear clusters emerging—ideally where each cluster is composed of the same gesture class and the clusters themselves have clear separation.
Let's try another combination. How about the max mag frequency component (in frequency domain) and the max mag signal value (in the time domain).
# plot_feature_2d(selected_gesture_set, extract_feature_top_mag_freq2, extract_feature_std_mag,
# xlabel="extract_feature_top_mag_freq2", ylabel="max_mag", title="Std mag vs. max mag")
plot_feature_2d(selected_gesture_set, extract_feature_top_mag_freq2, extract_feature_max_mag,
xlabel="top_mag_freq2", ylabel="max_mag", title="Top freq vs. max mag")
Again, now it's up to you. Try graphing some feature combinations below. Later, we are also going to show you how to make a large number of 2-dimensional plots using something called a pairplot to explore all of our features together.
# TODO: Write some of your own 2D plot explorations
3-dimensional explorations¶
We explored 1-dimensional visualizations of our features, then 2-dimensional, can we go to 3?! Yes we can!
But that's roughly where we hit the limit of visualization dimensional space (don't worry, we'll also show you some alternative visualizations below that help us visualize lots of different feature combinations at once—though still pairwise).
And to make the 3D plots more discernible, we are going to use an IPython Magic command called %matplotlib notebook. To interact with the 3D scatter plots, you can click-and-drag to rotate and hold right mouse and move mouse up/down to zoom.
Interactive charts for the win!
Let's start by graphing the highest-amplitude frequency from the magnitude signal, the max magnitude value, and the standard deviation of magnitude for each gesture trial (grouped by gesture type).
%matplotlib notebook
plot_feature_3d(selected_gesture_set, extract_feature_top_mag_freq2,
extract_feature_max_mag, extract_feature_std_mag,
xlabel="top_mag_freq2", ylabel="max_mag", zlabel="std_mag",
title="Top freq vs. max mag vs. std mag", figsize=(10,8));
While it takes some time to familiarize oneself with a 3D visualization and pick out patterns (it helps enormously to interact with the charts by rotating and zooming), you are certainly able to see clear separation between many of the different gesture types. Shake and At Rest clearly stand out but so do Baseball Throw, Midair Zorro 'Z', Custom, and Backhand Tennis.
Let's try one more set of three input features: the number of peaks in the time domain mag_p signal along with the standard deviation and max of the mag signal.
%matplotlib notebook
plot_feature_3d(selected_gesture_set, extract_feature_std_mag,
extract_feature_max_mag, extract_feature_num_peaks_mag_p,
xlabel="top_mag_freq2", ylabel="max_mag", zlabel="std_mag",
title="Std mag vs. max mag vs. num peaks mag_p", figsize=(10,8));