The Kaggle competition on Mercedes Benz greener manufacturing has concluded and the private leaderboard has had a huge shake up, people who were in the top #10 for the most part of the competition failed to remain in the top #10 on the private leaderboard, why is that? let’s explore the dataset and discover why.
First let’s start with loading our data
train_df = pd.read_csv("../../DataSets/mercedes/train.csv")
test_df = pd.read_csv("../../DataSets/mercedes/test.csv")
Next let’s take a look into the data
In [7]: train_df.shape Out[7]: (4209, 378)
In [8]: train_df.head(1) Out[8]: ID y X0 X1 X2 X3 X4 X5 X6 X8 ... X375 X376 X377 X378 X379 \ 0 0 130.81 k v at a d u j o ... 0 0 1 0 0 X380 X382 X383 X384 X385 0 0 0 0 0 0 [1 rows x 378 columns]
In [9]: train_df.dtypes
Out[9]:
ID int64
y float64
X0 object
X1 object
X2 object
X3 object
X4 object
X5 object
X6 object
X8 object
X10 int64
X11 int64
X12 int64
X13 int64
X14 int64
X15 int64
X16 int64
X17 int64
X18 int64
X19 int64
X20 int64
X21 int64
X22 int64
X23 int64
X24 int64
X26 int64
X27 int64
X28 int64
X29 int64
X30 int64
...
X355 int64
X356 int64
X357 int64
X358 int64
X359 int64
X360 int64
X361 int64
X362 int64
X363 int64
X364 int64
X365 int64
X366 int64
X367 int64
X368 int64
X369 int64
X370 int64
X371 int64
X372 int64
X373 int64
X374 int64
X375 int64
X376 int64
X377 int64
X378 int64
X379 int64
X380 int64
X382 int64
X383 int64
X384 int64
X385 int64
dtype: object
So apparently we have a few categorical variables (8), and for the most part we have integer variables and our target variable is a float.
Taking a look at the unique values of the remaining integers we find out that they are binary variables
In [38]: np.unique(train_df[train_df.columns[10:]]) Out[38]: array([0, 1])
Now let’s take a look at the target variable
train_df['y'].hist()
It looks like there might be some outliers because of the tiny number of values that are above 150 so let’s further investigate those
sns.violinplot(train_df['y'].values)
We can see from the violin plot that there are indeed outliers, values above ~ 135 can be considered outliers, we can also see from the distribution that there are two peaks at around 98 and 108
In [56]: len(train_df[train_df['y'] > 140]) Out[56]: 35
We have 35 training instances with their target value above 140, which is approximately 0.8% of our data so we can handle those with deletion or truncation.
Another very tricky and problematic issue of data like this (Binary and categorical variables) is duplicates because they confuse classifiers, so let’s check if we have rows that have the same features but different y value
In [72]: len(train_df[train_df.drop(["ID", "y"], axis=1).duplicated()]) Out[72]: 298
Wow, almost 300 duplicate rows, that is almost 7% of our dataset!
Handling duplicates
In my opinion the best way to handle these duplicates is to drop them, but since the dataset is small then let’s find a moderate solution that will conserve data.
We can perform an aggregate function on the dataset like taking the mean value for their targets or take the median, I will go with the median
def average_dups(x):
# Average value of duplicates
Y.loc[list(x.index)] = Y.loc[list(x.index)].mean()
X = train_df.drop(["y"], axis=1)
Y = train_df["y"]
dups = X[X.duplicated(keep=False)]
dups.groupby(dups.columns.tolist()).apply(average_dups)
train_df.drop(X[X.duplicated()].index.values, axis=0, inplace=True)
X = train_df.drop(["y"], axis=1)
Y = train_df["y"]
X.reset_index(inplace=True, drop=True)
Y.reset_index(inplace=True, drop=True)
Building a model
Now that we have a sense of the data let’s go ahead and start building a model and see how will it perform
First of all we need to get rid of those categorical variables in order to do this we will use One Hot encoding since we can’t see any reason to use an integer encoding that would imply ordinal relations in the data, to do this we can use pandas directly
data = train_df.append(test_df) data = pd.get_dummies(data) train, test = data[0:len(train_df)], data[len(train_df):]
We had to concatenate both sets to ensure the mapping is consistent
Next we need to drop the ID of each example because in theory an ID is unique for each example and that implies that it should add no information to our models
train = train.drop(["ID"], axis=1) test = test.drop(["ID"], axis=1)
We have both of our train/test datasets ready so let’s go ahead and start building an XGBoost model
Why use XGBoost? Because it’s fast to train, very accurate and can provide us with intuition on its decisions in addition to being favored so much in Kaggle competitions overall
xgb_params = {
'n_trees': 400,
'eta': 0.008,
'max_depth': 2,
'subsample': 0.93,
'objective': 'reg:linear',
'base_score': np.mean(Y),
'min_child_weight': 4,
}
These parameters were computed earlier by using GridSearch and BayesianOptimization however you can start with any parameters and move into parameter tuning later.
import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
def xgb_r2_score(preds, dtrain):
# Courtesy of Tilii
labels = dtrain.get_label()
return 'r2', r2_score(labels, preds)
X, XVal, Y, YVal = train_test_split(X, Y)
dtrain = xgb.DMatrix(X, Y)
dval = xgb.DMatrix(XVal, YVal)
dtest = xgb.DMatrix(X_Test)
scores = xgb.cv(xgb_params, dtrain, num_boost_round=1500, early_stopping_rounds=50, verbose_eval=True, feval=xgb_r2_score, maximize=True, nfold=10)
print("Best CV Score is: {}".format(scores['test-r2-mean'].iloc[-1]))
model = xgb.train(xgb_params, dtrain, num_boost_round=1500, feval=xgb_r2_score, maximize=True, early_stopping_rounds=50, verbose_eval=True, evals=[(dval, 'val')])
Submitting this model’s prediction would score 0.54193 on the private board which is around position #2687
This is very low, now let’s see some tricks that did work to improve it
#1 adding more dense features
Apparently the data is high dimensional and very sparse, this is not fun for a Tree based model so we resort to improving the data for our model, we do this by compressing information by using dimensionality reduction algorithms, out of these algorithms we’ll be using: PCA, SVD, ICA, Gaussian Random Projection and SparseRandomProjection.
Determining which algorithms to use is a matter of trial and error first you’ll start with PCA which will improve your scores then you’ll want to add more projections until the score settles.
from sklearn.decomposition import PCA, FastICA, TruncatedSVD
from sklearn.random_projection import GaussianRandomProjection
from sklearn.random_projection import SparseRandomProjection
pca = PCA(n_components=12)
ica = FastICA(n_components=12, max_iter=1000)
tsvd = TruncatedSVD(n_components=12)
gp = GaussianRandomProjection(n_components=12)
sp = SparseRandomProjection(n_components=12, dense_output=True)
x_pca = pd.DataFrame(pca.fit_transform(X))
x_ica = pd.DataFrame(ica.fit_transform(X))
x_tsvd = pd.DataFrame(tsvd.fit_transform(X))
x_gp = pd.DataFrame(gp.fit_transform(X))
x_sp = pd.DataFrame(sp.fit_transform(X))
x_pca.columns = ["pca_{}".format(i) for i in x_pca.columns]
x_ica.columns = ["ica_{}".format(i) for i in x_ica.columns]
x_tsvd.columns = ["tsvd_{}".format(i) for i in x_tsvd.columns]
x_gp.columns = ["gp_{}".format(i) for i in x_gp.columns]
x_sp.columns = ["sp_{}".format(i) for i in x_sp.columns]
X = pd.concat((X, x_pca), axis=1)
X = pd.concat((X, x_ica), axis=1)
X = pd.concat((X, x_tsvd), axis=1)
X = pd.concat((X, x_gp), axis=1)
X = pd.concat((X, x_sp), axis=1)
x_test_pca = pd.DataFrame(pca.transform(X_Test))
x_test_ica = pd.DataFrame(ica.transform(X_Test))
x_test_tsvd = pd.DataFrame(tsvd.transform(X_Test))
x_test_gp = pd.DataFrame(gp.transform(X_Test))
x_test_sp = pd.DataFrame(sp.transform(X_Test))
x_test_pca.columns = ["pca_{}".format(i) for i in x_test_pca.columns]
x_test_ica.columns = ["ica_{}".format(i) for i in x_test_ica.columns]
x_test_tsvd.columns = ["tsvd_{}".format(i) for i in x_test_tsvd.columns]
x_test_gp.columns = ["gp_{}".format(i) for i in x_test_gp.columns]
x_test_sp.columns = ["sp_{}".format(i) for i in x_test_sp.columns]
X_Test = pd.concat((X_Test, x_test_pca), axis=1)
X_Test = pd.concat((X_Test, x_test_ica), axis=1)
X_Test = pd.concat((X_Test, x_test_tsvd), axis=1)
X_Test = pd.concat((X_Test, x_test_gp), axis=1)
X_Test = pd.concat((X_Test, x_test_sp), axis=1)
I didn’t not get an improvement on the score when I submitted this trial, probably the number of components needs to be tuned, but according to many kernels on Kaggle, it improved their scores.
#2 Adding the ID
Yeah, I know I’ve told you before that the ID is useless but apparently it’s quite the opposite, it turns out that using the ID as a feature improves the private leaderboard score which indicates that there might be some information in the ID or a hidden temporal feature
Adding the ID to the features scored: 0.54779 which would be in position #1674
#3 Using label encoder instead of One hot encoding
By using a Label Encoder instead of OHE + the ID we seem to get a rise on the score to be 0.54854 which would rank at position #1480 which is an even better improvement
from sklearn.preprocessing import LabelEncoder
for c in train_df.columns:
if train_df[c].dtype == 'object':
lbl = LabelEncoder()
lbl.fit(list(train_df[c].values) + list(test_df[c].values))
train_df[c] = lbl.transform(list(train_df[c].values))
test_df[c] = lbl.transform(list(test_df[c].values))
And by using the 3 approaches this would give an even better score.
These were the most common tricks for the competition in conclusion adding more features and the magic features seemed to improved scores overall however this made the models more prone to overfitting which is why the huge shake up happened when the private lb was released.
Moral of the competition: Trust your local CV
Stacking was there in the top #5 spots with this kernel which is a variation from a kernel that caused a lot of overfitting on the public lb.
You can take a look at my kernels for Hyper parameter tuning using both GridSearch and Bayesian Optimization: