Time series for climate change: reducing food waste with clustering By Vitor Cerqueira | June, 2023

In the rest of this article, we will do a clustering analysis of the food demand time series. You’ll learn how to:

• Summarize a set of time series using feature extraction;
• Use K-Means and a hierarchical method for time series clustering.

The full code is available on Github:

data set

We will use weekly food sales time series collected by the US Department of Agriculture. This data set includes information on food sales by product category and subcategory. The time series is split by state, but we’ll use national total sales in each period.

Below is a sample of the data set:

Here’s what the full data looks like:

Feature-Based Time Series Clustering

We will use a feature-based approach for time series clustering. This process includes two main steps:

1. summarize each time series into a set of features, such as the mean value;
2. Apply a traditional clustering algorithm, such as K-Means, to the feature set.

Let’s go through each step in turn.

using feature extraction tfl

We start by extracting a set of data to summarize each time series. The goal is to convert each series into a smaller set of features.

There are many tools for time series feature extraction. we will use tfel, which provides a competitive performance relative to other approaches (3).

Here’s How You Can Use tfl,

import pandas as pd
import tsfel
# get configuration
cfg = tsfel.get_features_by_domain()
# extract features for each food subcategory
features = col: tsfel.time_series_features_extractor(cfg, data(col))
for col in data
features_df = pd.concat(features, axis=0)

This process results in a large number of features. Some of these may be redundant, so we adopt a feature selection process.

Below, we apply three operations on the feature set:

• Normalization: Convert a variable to a 0-1 value range;
• Selection by variance: Eliminate any variable with 0 variance;
• Selection by correlation: Eliminate any variable with a high correlation with an existing one.

from sklearn.preprocessing import MinMaxScaler
from sklearn.feature_selection import VarianceThreshold
from src.correlation_filter import correlation_filter
# normalizing the features
features_norm_df = pd.DataFrame(MinMaxScaler().fit_transform(features_df),
columns=features_df.columns)
# removing features with 0 variance
min_var = VarianceThreshold(threshold=0)
min_var.fit(features_norm_df)
features_norm_df = pd.DataFrame(min_var.transform(features_norm_df),
columns=min_var.get_feature_names_out())
# removing correlated features
features_norm_df = correlation_filter(features_norm_df, 0.9)
features_norm_df.index = data.columns

Clustering with K-Means

After preprocessing the data set, we are ready to cluster the time series. We summarize each series into a smaller set of discrete features. So, we can use any conventional algorithm for clustering. A popular option is K-means.

With K-means, we have to choose the number of clusters we want. There is no clear a priori value for this parameter unless we have some domain knowledge. But, we can take a data-driven approach to select the number of clusters. We test different values ​​and choose the best one.

Below, we test K-means with a maximum of 24 clusters. Then, we choose the number of clusters that maximize the silhouette score. This metric determines the cohesion of the obtained groups.

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
kmeans_parameters = 
'init': 'k-means++',
'n_init': 100,
'max_iter': 50,
n_clusters = range(2, 25)
silhouette_coef = ()
for k in n_clusters:
kmeans = KMeans(n_clusters=k, **kmeans_parameters)
kmeans.fit(features_norm_df)
score = silhouette_score(features_norm_df, kmeans.labels_)
silhouette_coef.append(score)

As shown in the figure below, the silhouette score is maximized for 5 groups.

We can make a parallelogram plot to understand the profile of each cluster. Here is an example with a sample of three features:

We can also use information about clusters to improve demand forecasting models. For example, by building a model for each cluster. The paper in context (5) is a good example of this approach.

hierarchical clustering

Hierarchical clustering is an alternative to K-means. It links pairs of clusters recursively, forming a tree-like structure. Library scipy Provides an implementation for this method.

import scipy.cluster.hierarchy as shc
# hierarchical clustering using the ward method
clustering = shc.linkage(features_norm_df, method='ward')
# plotting the dendrogram
dend = shc.dendrogram(clustering,
labels=categories.values,
orientation='right',
leaf_font_size=7)

The results of a hierarchical clustering model are best visualized with a dendrogram plot:

We can use a dendrogram to understand the cluster profile. For example, we can see that most canned goods are grouped (colored orange). Oranges also freeze well with pancake/cake mix. These two often go together in people’s breakfast.