All About LSTM
All About LSTM
- Input Shape : 반드시 3D Array 이어야 한다.
( Batch Size , Time Stamp , Feature Number )
- LSTM의 출력 ( Memory Cell 수 ) : N
0. return_sequences / return_state / stateful
- Default는 두 Option 모두 False
0.1. return_sequences
-
False이면 마지막 출력 h4만 출력
-
Example
- Time Stamp : 3
- Featuer Number : 1
- Batch Size : 2
- LSTM 출력 : 4
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input
from tensorflow.keras.layers import LSTM
from numpy import array
inputs1 = Input(shape=(3, 1))
lstm1 = LSTM( 4 )(inputs1)
model = Model(inputs=inputs1, outputs=lstm1)
data = array([[0.1, 0.2, 0.3],[0.2, 0.3, 0.4]]).reshape((2,3,1))
p = model.predict(data)
print(p)
print(p.shape)
[[-0.0575355 -0.07571618 -0.00497641 0.03933733]
[-0.08588521 -0.1113222 -0.00439387 0.05813249]]
(2, 4)
-
return_sequences = False이면, 출력은 마지막 LSTM Cell만 나온다.
-
Shape은 ( Batch Size , LSTM 출력 )
-
True이면 각 Time Stamp의 모든 출력이 나온다.
-
Example
- Time Stamp : 3
- Featuer Number : 1
- Batch Size : 2
- LSTM 출력 : 4
inputs1 = Input(shape=(3, 1))
lstm1 = LSTM(4, return_sequences = True)(inputs1)
model = Model(inputs=inputs1, outputs=lstm1)
data = array([[0.1, 0.2, 0.3],[0.2, 0.3, 0.4]]).reshape((2,3,1))
p = model.predict(data)
print( p )
print( p.shape )
[[[-0.00992484 0.00204053 -0.00934763 -0.001566 ]
[-0.02673527 0.00618046 -0.02628965 -0.0047341 ]
[-0.04814639 0.01236558 -0.04956007 -0.00930633]]
[[-0.01992876 0.00419011 -0.01879886 -0.00302125]
[-0.04352047 0.01054809 -0.04349945 -0.00764405]
[-0.06930547 0.01890578 -0.07319777 -0.0134633 ]]]
(2, 3, 4)
- return_sequences = True인 경우의 각 Time Stamp별 출력이 모두 나오고, Output Shape은 ( Batch Size , Time Stamp , LSTM 출력 ) 이 된다.
0.2. return_state
-
return_state의 의미는 LSTM이 Output 뿐만 아니라, LSTM 내부 State( Hidden State , Cell State )도 같이 Return하라는 의미.
-
return_sequences = False / return_state = True이면, 마지막 Time Stamp에서의 Output , Hidden State , Cell State 모두 출력한다.
inputs1 = Input(shape=(3, 1))
lstm1, state_h, state_c = LSTM(4, return_state=True)(inputs1)
model = Model(inputs=inputs1, outputs=[lstm1, state_h, state_c])
data = array([[0.1, 0.2, 0.3],[0.2, 0.3, 0.4]]).reshape((2,3,1))
p = model.predict(data)
print("LSTM 출력\n",p[0] , p[0].shape)
print("Hidden State 출력\n",p[1] , p[1].shape)
print("Cell State 출력\n",p[2] , p[2].shape)
LSTM 출력
[[ 0.01608176 -0.04563671 0.05012723 -0.02996504]
[ 0.02256971 -0.06557295 0.06973792 -0.04364965]] (2, 4)
Hidden State 출력
[[ 0.01608176 -0.04563671 0.05012723 -0.02996504]
[ 0.02256971 -0.06557295 0.06973792 -0.04364965]] (2, 4)
Cell State 출력
[[ 0.03205967 -0.08535378 0.0981688 -0.05697074]
[ 0.04499921 -0.12033087 0.13603717 -0.0815461 ]] (2, 4)
-
마지막 Cell의 Output , Hidden State , Cell State가 출력이 된다.
-
Output , Hidden State , Cell State 각각 Output Shape은 ( Batch Size , LSTM 출력 ) 이 된다.
-
Output , Hidden State는 같은 값이므로, 동일한 값이 출력된다.
- return_sequences = True / return_state = True인 경우, Output , Hidden State , Cell State 모두 출력한다.
inputs1 = Input(shape=(3, 1))
lstm1, state_h, state_c = LSTM(4, return_sequences = True , return_state=True)(inputs1)
model = Model(inputs=inputs1, outputs=[lstm1, state_h, state_c])
data = array([[0.1, 0.2, 0.3],[0.2, 0.3, 0.4]]).reshape((2,3,1))
p = model.predict(data)
print("LSTM 출력\n",p[0] , p[0].shape)
print("Hidden State 출력\n",p[1] , p[1].shape)
print("Cell State 출력\n",p[2] , p[2].shape)
LSTM 출력
[[[-0.00232224 0.01381751 0.00770746 -0.00485208]
[-0.00598483 0.037546 0.02239466 -0.0154511 ]
[-0.01042776 0.06827092 0.04281441 -0.03233362]]
[[-0.00453491 0.02751611 0.01552548 -0.00999177]
[-0.00951656 0.06102681 0.03691184 -0.02671457]
[-0.01474284 0.09843805 0.06241723 -0.05025584]]] (2, 3, 4)
Hidden State 출력
[[-0.01042776 0.06827092 0.04281441 -0.03233362]
[-0.01474284 0.09843805 0.06241723 -0.05025584]] (2, 4)
Cell State 출력
[[-0.02109056 0.14365022 0.09071556 -0.06398231]
[-0.02994104 0.21193042 0.13567713 -0.09934995]] (2, 4)
-
각 Output Shape은 아래와 같습니다.
-
LSTM 출력 : 각 Time Stamp마다 다 출력합니다.
### ( Batch Size , Time Stamp , LSTM 출력 ) ### -
Hidden State : 마지막 Cell의 출력을 나타낸다.
### ( Batch Size , LSTM 출력 ) ### -
Cell State : 마지막 Cell의 Cell State 출력을 나타낸다.
### ( Batch Size , LSTM 출력 ) ###
0.3. stateful
-
현재까지 학습 상태가 다음 학습시의 초기상태로 전달된다.
- stateful = True일때는 Input Shape 전달시에 반드시 Batch Size를 전해줘야 한다.
( Batch Size , Time Stamp , Feature Number )
-
reset_state()는 현재 Train Data가 다른 Train Data와 상관관계가 없다면 호출합니다.
-
마지막 Train Data를 마치고 다른 Epoch을 실행하는 경우에 Reset해야 한다.
-
한 Epoch 내에서도 여러 Sequence Data가 있을 경우에 새로운 Sequence Data를 시작할 때 Reset해야 한다.
-
1. Example
1.1. stateful Example
-
전체 Train Data Sequence를 Time Stamp 기준으로 몇 개로 나누어서 Train하는 예제
-
stateful을 이용하는 예를 보여준다.
import pandas as pd
import tensorflow as tf
import numpy as np
import os
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM, Conv1D, Lambda, Dropout
from tensorflow.keras.losses import Huber
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
train = pd.read_csv("train.csv")
train.head()
| id | breath_id | R | C | time_step | u_in | u_out | pressure | |
|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 1 | 20 | 50 | 0.000000 | 0.083334 | 0 | 5.837492 |
| 1 | 2 | 1 | 20 | 50 | 0.033652 | 18.383041 | 0 | 5.907794 |
| 2 | 3 | 1 | 20 | 50 | 0.067514 | 22.509278 | 0 | 7.876254 |
| 3 | 4 | 1 | 20 | 50 | 0.101542 | 22.808822 | 0 | 11.742872 |
| 4 | 5 | 1 | 20 | 50 | 0.135756 | 25.355850 | 0 | 12.234987 |
-
아래 함수는 Train Data(x) , Target(y)를 받아서 Time Stamp(window_size)만큼 자르고, Batch Size만큼 모아서 Dataset으로 만들어주는 함수
-
.window() Function
- window_size : Time Stamp 수. 이 수만큼 Time Stamp Data를 만들어 준다.
- shift : 다음 Data를 만들때 몇 개의 Data를 Skip할 것인가
- stride : Time Stamp 내에서 몇 개의 Data를 Skip할 것인가
- drop_remainder : x에서 window_size만큼 만들다가 남는건 어떻게 할 것인가
def windowed_dataset(x, y, window_size, batch_size, shuffle):
ds_x = tf.data.Dataset.from_tensor_slices(x)
ds_x = ds_x.window(window_size, shift=1, stride = 1, drop_remainder=True)
ds_x = ds_x.flat_map(lambda x: x.batch(window_size))
ds_y = tf.data.Dataset.from_tensor_slices( y[window_size:])
ds = tf.data.Dataset.zip( (ds_x , ds_y) )
if shuffle:
ds = ds.shuffle(1000)
return ds.batch(batch_size).cache().prefetch(tf.data.experimental.AUTOTUNE)
WINDOW_SIZE=5
#BATCH_SIZE=4
BATCH_SIZE=1 # Stateful일때는 Batch Size 는 1
filename = os.path.join('CheckPoint', 'ckeckpointer.ckpt')
model = Sequential([
# 1차원 feature map 생성
Conv1D(filters=32, kernel_size=5,
padding="causal",
activation="relu",
batch_input_shape=[1, WINDOW_SIZE, 4]),
# LSTM
LSTM(16, activation='tanh' , stateful = True),
Dropout(0.3),
Dense(16, activation="relu"),
Dense(1),
])
# Sequence 학습에 비교적 좋은 퍼포먼스를 내는 Huber()를 사용합니다.
loss = Huber()
optimizer = Adam(0.0005)
model.compile(loss=Huber(), optimizer=optimizer, metrics=['mse'])
# earlystopping은 10번 epoch통안 val_loss 개선이 없다면 학습을 멈춥니다.
earlystopping = EarlyStopping(monitor='val_loss', patience=10)
checkpoint = ModelCheckpoint(filename,
save_weights_only=True,
save_best_only=True,
monitor='val_loss',
verbose=1)
minMaxScaler = MinMaxScaler(feature_range=(0, 1))
t = train.drop(['id','time_step'] , axis=1 ).copy()
t
| breath_id | R | C | u_in | u_out | pressure | |
|---|---|---|---|---|---|---|
| 0 | 1 | 20 | 50 | 0.083334 | 0 | 5.837492 |
| 1 | 1 | 20 | 50 | 18.383041 | 0 | 5.907794 |
| 2 | 1 | 20 | 50 | 22.509278 | 0 | 7.876254 |
| 3 | 1 | 20 | 50 | 22.808822 | 0 | 11.742872 |
| 4 | 1 | 20 | 50 | 25.355850 | 0 | 12.234987 |
| ... | ... | ... | ... | ... | ... | ... |
| 6035995 | 125749 | 50 | 10 | 1.489714 | 1 | 3.869032 |
| 6035996 | 125749 | 50 | 10 | 1.488497 | 1 | 3.869032 |
| 6035997 | 125749 | 50 | 10 | 1.558978 | 1 | 3.798729 |
| 6035998 | 125749 | 50 | 10 | 1.272663 | 1 | 4.079938 |
| 6035999 | 125749 | 50 | 10 | 1.482739 | 1 | 3.869032 |
6036000 rows × 6 columns
minMaxScaler.fit(t)
MinMaxScaler()
t_scaled = minMaxScaler.transform(t)
t_scaled = pd.DataFrame(t_scaled)
t_scaled
| 0 | 1 | 2 | 3 | 4 | 5 | |
|---|---|---|---|---|---|---|
| 0 | 0.0 | 0.333333 | 1.0 | 0.000833 | 0.0 | 0.115911 |
| 1 | 0.0 | 0.333333 | 1.0 | 0.183830 | 0.0 | 0.116965 |
| 2 | 0.0 | 0.333333 | 1.0 | 0.225093 | 0.0 | 0.146470 |
| 3 | 0.0 | 0.333333 | 1.0 | 0.228088 | 0.0 | 0.204426 |
| 4 | 0.0 | 0.333333 | 1.0 | 0.253559 | 0.0 | 0.211802 |
| ... | ... | ... | ... | ... | ... | ... |
| 6035995 | 1.0 | 1.000000 | 0.0 | 0.014897 | 1.0 | 0.086407 |
| 6035996 | 1.0 | 1.000000 | 0.0 | 0.014885 | 1.0 | 0.086407 |
| 6035997 | 1.0 | 1.000000 | 0.0 | 0.015590 | 1.0 | 0.085353 |
| 6035998 | 1.0 | 1.000000 | 0.0 | 0.012727 | 1.0 | 0.089568 |
| 6035999 | 1.0 | 1.000000 | 0.0 | 0.014827 | 1.0 | 0.086407 |
6036000 rows × 6 columns
t_scaled.columns = ['0','1','2','3','4','5']
t_scaled
| 0 | 1 | 2 | 3 | 4 | 5 | |
|---|---|---|---|---|---|---|
| 0 | 0.0 | 0.333333 | 1.0 | 0.000833 | 0.0 | 0.115911 |
| 1 | 0.0 | 0.333333 | 1.0 | 0.183830 | 0.0 | 0.116965 |
| 2 | 0.0 | 0.333333 | 1.0 | 0.225093 | 0.0 | 0.146470 |
| 3 | 0.0 | 0.333333 | 1.0 | 0.228088 | 0.0 | 0.204426 |
| 4 | 0.0 | 0.333333 | 1.0 | 0.253559 | 0.0 | 0.211802 |
| ... | ... | ... | ... | ... | ... | ... |
| 6035995 | 1.0 | 1.000000 | 0.0 | 0.014897 | 1.0 | 0.086407 |
| 6035996 | 1.0 | 1.000000 | 0.0 | 0.014885 | 1.0 | 0.086407 |
| 6035997 | 1.0 | 1.000000 | 0.0 | 0.015590 | 1.0 | 0.085353 |
| 6035998 | 1.0 | 1.000000 | 0.0 | 0.012727 | 1.0 | 0.089568 |
| 6035999 | 1.0 | 1.000000 | 0.0 | 0.014827 | 1.0 | 0.086407 |
6036000 rows × 6 columns
t['R'] = t_scaled['1']
t['C'] = t_scaled['2']
t['u_in'] = t_scaled['3']
t['u_out'] = t_scaled['4']
t
| breath_id | R | C | u_in | u_out | pressure | |
|---|---|---|---|---|---|---|
| 0 | 1 | 0.333333 | 1.0 | 0.000833 | 0.0 | 5.837492 |
| 1 | 1 | 0.333333 | 1.0 | 0.183830 | 0.0 | 5.907794 |
| 2 | 1 | 0.333333 | 1.0 | 0.225093 | 0.0 | 7.876254 |
| 3 | 1 | 0.333333 | 1.0 | 0.228088 | 0.0 | 11.742872 |
| 4 | 1 | 0.333333 | 1.0 | 0.253559 | 0.0 | 12.234987 |
| ... | ... | ... | ... | ... | ... | ... |
| 6035995 | 125749 | 1.000000 | 0.0 | 0.014897 | 1.0 | 3.869032 |
| 6035996 | 125749 | 1.000000 | 0.0 | 0.014885 | 1.0 | 3.869032 |
| 6035997 | 125749 | 1.000000 | 0.0 | 0.015590 | 1.0 | 3.798729 |
| 6035998 | 125749 | 1.000000 | 0.0 | 0.012727 | 1.0 | 4.079938 |
| 6035999 | 125749 | 1.000000 | 0.0 | 0.014827 | 1.0 | 3.869032 |
6036000 rows × 6 columns
x_train, x_val, y_train, y_val = train_test_split( t.drop('pressure', 1),
t['pressure'],
test_size=0.005,
random_state=0,
shuffle=False )
<ipython-input-13-0a210ac6df78>:1: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only
x_train, x_val, y_train, y_val = train_test_split( t.drop('pressure', 1),
breath_id = list( x_train['breath_id'].value_counts().reset_index()['index'] )
breath_id.sort()
x_val = x_val.drop(['breath_id'] , axis=1 ).copy()
x_val
| R | C | u_in | u_out | |
|---|---|---|---|---|
| 6005820 | 0.0 | 0.25 | 0.044443 | 1.0 |
| 6005821 | 0.0 | 0.25 | 0.045260 | 1.0 |
| 6005822 | 0.0 | 0.25 | 0.045960 | 1.0 |
| 6005823 | 0.0 | 0.25 | 0.046555 | 1.0 |
| 6005824 | 0.0 | 0.25 | 0.047062 | 1.0 |
| ... | ... | ... | ... | ... |
| 6035995 | 1.0 | 0.00 | 0.014897 | 1.0 |
| 6035996 | 1.0 | 0.00 | 0.014885 | 1.0 |
| 6035997 | 1.0 | 0.00 | 0.015590 | 1.0 |
| 6035998 | 1.0 | 0.00 | 0.012727 | 1.0 |
| 6035999 | 1.0 | 0.00 | 0.014827 | 1.0 |
30180 rows × 4 columns
val_ds = windowed_dataset( x_val , y_val , WINDOW_SIZE , BATCH_SIZE , False )
for epoch in range(5):
for b_id in breath_id:
train_b_id = x_train[ x_train['breath_id'] == b_id ]
train_b_id = train_b_id.drop(['breath_id'] , axis=1 ).copy()
train_ds = windowed_dataset( train_b_id , y_train , WINDOW_SIZE , BATCH_SIZE , False )
print("Breath ID : {0}".format(b_id))
history = model.fit(train_ds,
validation_data=(val_ds),
epochs=10,
callbacks=[checkpoint, earlystopping])
model.reset_states()
filename = "Save_iter_{}.h5".format(epoch)
model.save( filename )
Breath ID : 1
Epoch 1/10
71/Unknown - 5s 8ms/step - loss: 10.4798 - mse: 149.2300
---------------------------------------------------------------------------
KeyboardInterrupt Traceback (most recent call last)
<ipython-input-17-c7d14ad0602f> in <module>
10 print("Breath ID : {0}".format(b_id))
11
---> 12 history = model.fit(train_ds,
13 validation_data=(val_ds),
14 epochs=10,
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
1212 model=self,
1213 steps_per_execution=self._steps_per_execution)
-> 1214 val_logs = self.evaluate(
1215 x=val_x,
1216 y=val_y,
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\keras\engine\training.py in evaluate(self, x, y, batch_size, verbose, sample_weight, steps, callbacks, max_queue_size, workers, use_multiprocessing, return_dict, **kwargs)
1487 with trace.Trace('test', step_num=step, _r=1):
1488 callbacks.on_test_batch_begin(step)
-> 1489 tmp_logs = self.test_function(iterator)
1490 if data_handler.should_sync:
1491 context.async_wait()
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\eager\def_function.py in __call__(self, *args, **kwds)
887
888 with OptionalXlaContext(self._jit_compile):
--> 889 result = self._call(*args, **kwds)
890
891 new_tracing_count = self.experimental_get_tracing_count()
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\eager\def_function.py in _call(self, *args, **kwds)
922 # In this case we have not created variables on the first call. So we can
923 # run the first trace but we should fail if variables are created.
--> 924 results = self._stateful_fn(*args, **kwds)
925 if self._created_variables:
926 raise ValueError("Creating variables on a non-first call to a function"
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\eager\function.py in __call__(self, *args, **kwargs)
3021 (graph_function,
3022 filtered_flat_args) = self._maybe_define_function(args, kwargs)
-> 3023 return graph_function._call_flat(
3024 filtered_flat_args, captured_inputs=graph_function.captured_inputs) # pylint: disable=protected-access
3025
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\eager\function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
1958 and executing_eagerly):
1959 # No tape is watching; skip to running the function.
-> 1960 return self._build_call_outputs(self._inference_function.call(
1961 ctx, args, cancellation_manager=cancellation_manager))
1962 forward_backward = self._select_forward_and_backward_functions(
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\eager\function.py in call(self, ctx, args, cancellation_manager)
589 with _InterpolateFunctionError(self):
590 if cancellation_manager is None:
--> 591 outputs = execute.execute(
592 str(self.signature.name),
593 num_outputs=self._num_outputs,
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\eager\execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
57 try:
58 ctx.ensure_initialized()
---> 59 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
60 inputs, attrs, num_outputs)
61 except core._NotOkStatusException as e:
KeyboardInterrupt:
1.2. return_sequences Example
- 전체 Sequence를 예측하는 Example
import numpy as np
import pandas as pd
import tensorflow as tf
import gc
import matplotlib.pyplot as plt
%matplotlib inline
from tensorflow import keras
from tensorflow.keras.layers import *
from tensorflow.keras import *
from tensorflow.keras.callbacks import LearningRateScheduler
from tensorflow.keras.optimizers.schedules import ExponentialDecay
from sklearn.preprocessing import RobustScaler, normalize
from sklearn.model_selection import train_test_split, KFold
from sklearn.metrics import mean_absolute_error
from pickle import load
import os
DEBUG = True
#DEBUG = False
train = pd.read_csv('train.csv')
test = pd.read_csv('test.csv')
submission = pd.read_csv('sample_submission.csv')
if DEBUG:
train = train[:80*1000]
train.shape, test.shape, submission.shape
((80000, 8), (4024000, 7), (4024000, 2))
def add_features(df):
df['area'] = df['time_step'] * df['u_in']
df['area'] = df.groupby('breath_id')['area'].cumsum()
df['u_in_cumsum'] = (df['u_in']).groupby(df['breath_id']).cumsum()
df['u_in_lag1'] = df.groupby('breath_id')['u_in'].shift(1)
df['u_out_lag1'] = df.groupby('breath_id')['u_out'].shift(1)
df['u_in_lag_back1'] = df.groupby('breath_id')['u_in'].shift(-1)
df['u_out_lag_back1'] = df.groupby('breath_id')['u_out'].shift(-1)
df['u_in_lag2'] = df.groupby('breath_id')['u_in'].shift(2)
df['u_out_lag2'] = df.groupby('breath_id')['u_out'].shift(2)
df['u_in_lag_back2'] = df.groupby('breath_id')['u_in'].shift(-2)
df['u_out_lag_back2'] = df.groupby('breath_id')['u_out'].shift(-2)
df['u_in_lag3'] = df.groupby('breath_id')['u_in'].shift(3)
df['u_out_lag3'] = df.groupby('breath_id')['u_out'].shift(3)
df['u_in_lag_back3'] = df.groupby('breath_id')['u_in'].shift(-3)
df['u_out_lag_back3'] = df.groupby('breath_id')['u_out'].shift(-3)
df['u_in_lag4'] = df.groupby('breath_id')['u_in'].shift(4)
df['u_out_lag4'] = df.groupby('breath_id')['u_out'].shift(4)
df['u_in_lag_back4'] = df.groupby('breath_id')['u_in'].shift(-4)
df['u_out_lag_back4'] = df.groupby('breath_id')['u_out'].shift(-4)
df = df.fillna(0)
df['breath_id__u_in__max'] = df.groupby(['breath_id'])['u_in'].transform('max')
df['breath_id__u_out__max'] = df.groupby(['breath_id'])['u_out'].transform('max')
df['u_in_diff1'] = df['u_in'] - df['u_in_lag1']
df['u_out_diff1'] = df['u_out'] - df['u_out_lag1']
df['u_in_diff2'] = df['u_in'] - df['u_in_lag2']
df['u_out_diff2'] = df['u_out'] - df['u_out_lag2']
df['breath_id__u_in__diffmax'] = df.groupby(['breath_id'])['u_in'].transform('max') - df['u_in']
df['breath_id__u_in__diffmean'] = df.groupby(['breath_id'])['u_in'].transform('mean') - df['u_in']
df['breath_id__u_in__diffmax'] = df.groupby(['breath_id'])['u_in'].transform('max') - df['u_in']
df['breath_id__u_in__diffmean'] = df.groupby(['breath_id'])['u_in'].transform('mean') - df['u_in']
df['u_in_diff3'] = df['u_in'] - df['u_in_lag3']
df['u_out_diff3'] = df['u_out'] - df['u_out_lag3']
df['u_in_diff4'] = df['u_in'] - df['u_in_lag4']
df['u_out_diff4'] = df['u_out'] - df['u_out_lag4']
df['cross']= df['u_in']*df['u_out']
df['cross2']= df['time_step']*df['u_out']
df['R'] = df['R'].astype(str)
df['C'] = df['C'].astype(str)
df['R__C'] = df["R"].astype(str) + '__' + df["C"].astype(str)
df = pd.get_dummies(df)
return df
- 이 Dataset은 80개의 Data가 하나의 Sequence이어서 80개씩 나눈다.
targets = train['pressure'].to_numpy().reshape(-1, 80)
targets.shape
(1000, 80)
train.drop(labels='pressure', axis=1, inplace=True)
train.head()
| id | breath_id | R | C | time_step | u_in | u_out | |
|---|---|---|---|---|---|---|---|
| 0 | 1 | 1 | 20 | 50 | 0.000000 | 0.083334 | 0 |
| 1 | 2 | 1 | 20 | 50 | 0.033652 | 18.383041 | 0 |
| 2 | 3 | 1 | 20 | 50 | 0.067514 | 22.509278 | 0 |
| 3 | 4 | 1 | 20 | 50 | 0.101542 | 22.808822 | 0 |
| 4 | 5 | 1 | 20 | 50 | 0.135756 | 25.355850 | 0 |
train = add_features(train)
train.head()
| id | breath_id | time_step | u_in | u_out | area | u_in_cumsum | u_in_lag1 | u_out_lag1 | u_in_lag_back1 | ... | C_50 | R__C_20__10 | R__C_20__20 | R__C_20__50 | R__C_50__10 | R__C_50__20 | R__C_50__50 | R__C_5__10 | R__C_5__20 | R__C_5__50 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 1 | 0.000000 | 0.083334 | 0 | 0.000000 | 0.083334 | 0.000000 | 0.0 | 18.383041 | ... | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1 | 2 | 1 | 0.033652 | 18.383041 | 0 | 0.618632 | 18.466375 | 0.083334 | 0.0 | 22.509278 | ... | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| 2 | 3 | 1 | 0.067514 | 22.509278 | 0 | 2.138333 | 40.975653 | 18.383041 | 0.0 | 22.808822 | ... | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| 3 | 4 | 1 | 0.101542 | 22.808822 | 0 | 4.454391 | 63.784476 | 22.509278 | 0.0 | 25.355850 | ... | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| 4 | 5 | 1 | 0.135756 | 25.355850 | 0 | 7.896588 | 89.140326 | 22.808822 | 0.0 | 27.259866 | ... | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
5 rows × 52 columns
# normalise the dataset
RS = RobustScaler()
train = RS.fit_transform(train)
# Reshape to group 80 timesteps for each breath ID
train = train.reshape(-1, 80, train.shape[-1])
train.shape
(1000, 80, 52)
test = add_features(test)
test = RS.transform(test)
test = test.reshape(-1, 80, test.shape[-1])
train.shape, test.shape
((1000, 80, 52), (50300, 80, 52))
train.shape[-2], train.shape[-1]
(80, 52)
-
이 Model은 80개의 Time Stamp를 전부 받아서 각 Cell의 Output을 한 번에 모두 받고, 그 결과를 Target과 비교하는 방식을 사용
-
이를 위해 return_sequences = True를 사용
def create_lstm_model():
x0 = tf.keras.layers.Input(shape=(train.shape[-2], train.shape[-1]))
lstm_layers = 4 # number of LSTM layers
lstm_units = [320, 305, 304, 229]
lstm = Bidirectional(keras.layers.LSTM(lstm_units[0], return_sequences=True))(x0)
for i in range(lstm_layers-1):
lstm = Bidirectional(keras.layers.LSTM(lstm_units[i+1], return_sequences=True))(lstm)
lstm = Dropout(0.001)(lstm)
lstm = Dense(100, activation='relu')(lstm)
lstm = Dense(1)(lstm)
model = keras.Model(inputs=x0, outputs=lstm)
model.compile(optimizer="adam", loss="mae")
return model
EPOCH = 350
#BATCH_SIZE = 512
BATCH_SIZE = 32
NFOLDS = 5
kf = KFold(n_splits=NFOLDS, shuffle=True, random_state=2021)
history = []
test_preds = []
for fold, (train_idx, test_idx) in enumerate(kf.split(train, targets)):
print('-'*15, '>', f'Fold {fold+1}', '<', '-'*15)
X_train, X_valid = train[train_idx], train[test_idx]
y_train, y_valid = targets[train_idx], targets[test_idx]
model = create_lstm_model()
model.compile(optimizer="adam", loss="mae", metrics=[tf.keras.metrics.MeanAbsolutePercentageError()])
scheduler = ExponentialDecay(1e-3, 400*((len(train)*0.8)/BATCH_SIZE), 1e-5)
lr = LearningRateScheduler(scheduler, verbose=0)
history.append(model.fit(X_train, y_train,
validation_data=(X_valid, y_valid),
epochs=EPOCH, batch_size=BATCH_SIZE, callbacks=[lr]))
test_pred = model.predict(test).squeeze().reshape(-1, 1).squeeze()
test_preds.append(test_pred)
# save model
#model.save("lstm_model_fold_{}".format(fold))
del X_train, X_valid, y_train, y_valid, model
gc.collect()
--------------- > Fold 1 < ---------------
Epoch 1/350
25/25 [==============================] - ETA: 0s - loss: 4.3909 - mean_absolute_percentage_error: 34.6729
---------------------------------------------------------------------------
KeyboardInterrupt Traceback (most recent call last)
<ipython-input-18-5f29b2338821> in <module>
13 lr = LearningRateScheduler(scheduler, verbose=0)
14
---> 15 history.append(model.fit(X_train, y_train,
16 validation_data=(X_valid, y_valid),
17 epochs=EPOCH, batch_size=BATCH_SIZE, callbacks=[lr]))
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
1212 model=self,
1213 steps_per_execution=self._steps_per_execution)
-> 1214 val_logs = self.evaluate(
1215 x=val_x,
1216 y=val_y,
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\keras\engine\training.py in evaluate(self, x, y, batch_size, verbose, sample_weight, steps, callbacks, max_queue_size, workers, use_multiprocessing, return_dict, **kwargs)
1487 with trace.Trace('test', step_num=step, _r=1):
1488 callbacks.on_test_batch_begin(step)
-> 1489 tmp_logs = self.test_function(iterator)
1490 if data_handler.should_sync:
1491 context.async_wait()
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\eager\def_function.py in __call__(self, *args, **kwds)
887
888 with OptionalXlaContext(self._jit_compile):
--> 889 result = self._call(*args, **kwds)
890
891 new_tracing_count = self.experimental_get_tracing_count()
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\eager\def_function.py in _call(self, *args, **kwds)
922 # In this case we have not created variables on the first call. So we can
923 # run the first trace but we should fail if variables are created.
--> 924 results = self._stateful_fn(*args, **kwds)
925 if self._created_variables:
926 raise ValueError("Creating variables on a non-first call to a function"
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\eager\function.py in __call__(self, *args, **kwargs)
3021 (graph_function,
3022 filtered_flat_args) = self._maybe_define_function(args, kwargs)
-> 3023 return graph_function._call_flat(
3024 filtered_flat_args, captured_inputs=graph_function.captured_inputs) # pylint: disable=protected-access
3025
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\eager\function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
1958 and executing_eagerly):
1959 # No tape is watching; skip to running the function.
-> 1960 return self._build_call_outputs(self._inference_function.call(
1961 ctx, args, cancellation_manager=cancellation_manager))
1962 forward_backward = self._select_forward_and_backward_functions(
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\eager\function.py in call(self, ctx, args, cancellation_manager)
589 with _InterpolateFunctionError(self):
590 if cancellation_manager is None:
--> 591 outputs = execute.execute(
592 str(self.signature.name),
593 num_outputs=self._num_outputs,
~\Anaconda3\envs\TF.2.5.0-GPU\lib\site-packages\tensorflow\python\eager\execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
57 try:
58 ctx.ensure_initialized()
---> 59 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
60 inputs, attrs, num_outputs)
61 except core._NotOkStatusException as e:
KeyboardInterrupt:
submission["pressure"] = sum(test_preds)/5
submission.to_csv('submission.csv', index=False)