資料前處理

本次課程讓我們直接使用真實資料來進行實驗吧！請至這裡下載範例資料

– 前處理的過程與之前的課程相同，讓我們用不同的心電圖參數來預測「LVD」。我們的目標是希望在「驗證組」上盡可能提升它的AUC。

– 對於正在修習醫療人工智慧實作課程的同學來說，這就是你的第一個任務左心室功能障礙分類挑戰的子資料，你可以將今天學到的技巧用到上面去。

library(mice)

dat <- read.csv("ECG_train.csv", header = TRUE, fileEncoding = 'CP950', stringsAsFactors = FALSE, na.strings = "")
subdat <- dat[!(dat[,"LVD"] %in% NA), c(-1, -2, -4, -5)]

subdat[,'GENDER'] <- as.factor(subdat[,'GENDER'])
for (i in 1:31) {subdat[,paste0('rhythm.', i)] <- as.factor(subdat[,paste0('rhythm.', i)])}

used_dat.x <- subdat[,-1]
mice_dat <- mice(used_dat.x, m = 1, maxit = 5, meth = 'cart', seed = 123, printFlag = FALSE)
impute_dat.x <- mice:::complete(mice_dat, action = 1)

set.seed(0)
all_idx <- 1:nrow(subdat)

train_idx <- sample(all_idx, nrow(subdat) * 0.6)
valid_idx <- sample(all_idx[!all_idx %in% train_idx], nrow(subdat) * 0.2)
test_idx <- all_idx[!all_idx %in% c(train_idx, valid_idx)]

train_X <- impute_dat.x[train_idx,]
valid_X <- impute_dat.x[valid_idx,]
test_X <- impute_dat.x[test_idx,]

train_Y <- subdat[train_idx,"LVD"]
valid_Y <- subdat[valid_idx,"LVD"]
test_Y <- subdat[test_idx,"LVD"]

第一節：資料擴增(1)

在上節課中我們學到了使用「小批量梯度下降法」，而他的主要優點是透過每次抽樣造成的變異，從而避免我們陷入局部極值。

– 這樣看起來，增加變異是一件相當重要的事情，並且在實驗中我們還發現這麼做可以給出較為平滑的預測邊界。

這次我們把腦筋直接動到現有的資料上，還記得之前我們的MLP在面對有增加雜訊的資料時表現的就會變差，並且非常容易過度擬合。那我們是否在梯度下降時也可以直接對資料本身做出雜訊干擾，從而使模型更加強壯呢?

– 這個手法統稱為資料擴增(data augmentation)，其方法及種類的多樣性足夠上一整學期的課。我們這裡先小試身手，之後的課程還有許多部份同樣涉及資料擴增的部分會再持續補充！

第一節：資料擴增(2)

資料擴增的程式碼相當簡單，其重點只是在資料要進去訓練之前做一些前處理，之後再送進Forward/Backward程序中進行運算，其過程完全一樣，而我們用前幾節課的資料設計一個新的程式碼如下。

– 由於變數數目超過了2維，我們取消了視覺化的呈現，但加上了「validation」的選項

DEEP_MLP_Trainer = function (train_X, train_Y, valid_X = NULL, valid_Y = NULL, noise = 0,
                             num.iteration = 500, num.hidden = c(10, 10, 10), eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 20) {
  
  #Functions
  
  #Forward
  
  S.fun = function (x, eps = eps) {
    S = 1/(1 + exp(-x))
    S[S < eps] = eps
    S[S > 1 - eps] = 1 - eps
    return(S)
  }
  
  ReLU.fun = function (x) {
    x[x < 0] <- 0
    return(x)
  }
  
  L.fun = function (X, W) {
    X.E = cbind(1, X)
    L = X.E %*% W
    return(L)
  }
  
  CE.fun = function (o, y, eps = eps) {
    loss = -1/length(y) * sum(y * log(o + eps) + (1 - y) * log(1 - o + eps))
    return(loss)
  }
  
  #Backward
  
  grad_o.fun = function (o, y) {
    return((o - y)/(o*(1-o)))
  }
  
  grad_s.fun = function (grad_o, o) {
    return(grad_o*(o*(1-o)))
  }
  
  grad_W.fun = function (grad_l, h) {
    h.E = cbind(1, h)
    return(t(h.E) %*% grad_l/nrow(h))
  }
  
  grad_h.fun = function (grad_l, W) {
    return(grad_l %*% t(W[-1,]))
  }
  
  grad_l.fun = function (grad_h, l) {
    de_l = l
    de_l[de_l<0] = 0
    de_l[de_l>0] = 1
    return(grad_h*de_l)
  }
  
  # Noise
  
  sd.vec <- NULL
  
  for (k in 1:ncol(train_X)) {
    
    if (class(train_X[,k])[1] %in% c('numeric', 'integer')) {
      
      sd.val <- sd(train_X[,k])
      sd.vec <- c(sd.vec, sd.val * noise)
      
    } else {
      
      sd.vec <- c(sd.vec, 0L)
      
    }
    
  }
  
  #initialization
  
  train_X_mat <- model.matrix(~ ., data = train_X)
  train_X_mat <- train_X_mat[,-1]
  train_Y_mat <- t(t(train_Y))
  
  W_list = list()
  M_list = list()
  N_list = list()
  
  len_h = length(num.hidden)
  
  for (w_seq in 1:(len_h+1)) {
    if (w_seq == 1) {
      NROW_W = ncol(train_X_mat) + 1
      NCOL_W = num.hidden[w_seq]
    } else if (w_seq == len_h+1) {
      NROW_W = num.hidden[w_seq - 1] + 1
      NCOL_W = ncol(train_Y_mat)
    } else {
      NROW_W = num.hidden[w_seq - 1] + 1
      NCOL_W = num.hidden[w_seq]
    }
    W_list[[w_seq]] = matrix(rnorm(NROW_W*NCOL_W, sd = 1), nrow = NROW_W, ncol = NCOL_W)
    M_list[[w_seq]] = matrix(0, nrow = NROW_W, ncol = NCOL_W)
    N_list[[w_seq]] = matrix(0, nrow = NROW_W, ncol = NCOL_W)
  }
  
  loss_seq = rep(0, num.iteration)
  
  #Caculating
  
  for (i in 1:num.iteration) {
    
    idx = sample(1:nrow(train_X_mat), batch_size)
    noise_mat = t(matrix(rnorm(batch_size * length(sd.vec), sd = sd.vec), nrow = length(sd.vec)))
    sub_X_mat = train_X_mat[idx,] + noise_mat
    
    #Forward
    
    current_l_list = list()
    current_h_list = list()
    
    for (j in 1:len_h) {
      if (j == 1) {
        current_l_list[[j]] = L.fun(X = sub_X_mat, W = W_list[[j]])
      } else {
        current_l_list[[j]] = L.fun(X = current_h_list[[j-1]], W = W_list[[j]])
      }
      current_h_list[[j]] = ReLU.fun(x = current_l_list[[j]])
    }
    current_l_list[[len_h+1]] = L.fun(X = current_h_list[[len_h]], W = W_list[[len_h+1]])
    current_o = S.fun(x = current_l_list[[len_h+1]], eps = eps)
    loss_seq[i] = CE.fun(o = current_o, y = train_Y_mat[idx,], eps = eps)
    
    #Backward
    
    current_grad_l_list = list()
    current_grad_W_list = list()
    current_grad_h_list = list()
    
    current_grad_o = grad_o.fun(o = current_o, y = train_Y_mat[idx,])
    current_grad_l_list[[len_h+1]] = grad_s.fun(grad_o = current_grad_o, o = current_o)
    current_grad_W_list[[len_h+1]] = grad_W.fun(grad_l = current_grad_l_list[[len_h+1]], h = current_h_list[[len_h]])
    
    for (j in len_h:1) {
      current_grad_h_list[[j]] = grad_h.fun(grad_l = current_grad_l_list[[j+1]], W = W_list[[j+1]])
      current_grad_l_list[[j]] = grad_l.fun(grad_h = current_grad_h_list[[j]], l = current_l_list[[j]])
      if (j != 1) {
        current_grad_W_list[[j]] = grad_W.fun(grad_l = current_grad_l_list[[j]], h = current_h_list[[j - 1]])
      } else {
        current_grad_W_list[[j]] = grad_W.fun(grad_l = current_grad_l_list[[j]], h = sub_X_mat)
      }
    }
    
    if (optimizer == 'adam') {
      
      for (j in 1:(len_h+1)) {
        M_list[[j]] = beta1 * M_list[[j]] + (1 - beta1) * current_grad_W_list[[j]]
        N_list[[j]] = beta2 * N_list[[j]] + (1 - beta2) * current_grad_W_list[[j]]^2
        M.hat = M_list[[j]]/(1 - beta1^i)
        N.hat = N_list[[j]]/(1 - beta2^i)
        W_list[[j]] = W_list[[j]] - lr*M.hat/sqrt(N.hat+eps)
      }
      
    } else if (optimizer == 'sgd') {
      
      for (j in 1:(len_h+1)) {
        M_list[[j]] = beta1 * M_list[[j]] + lr * current_grad_W_list[[j]]
        W_list[[j]] = W_list[[j]] - M_list[[j]]
      }
      
    } else {
      stop('optimizer must be selected from "sgd" or "adam".')
    }
    
  }
  
  pre_func = function (new_X, w_list = W_list) {
    
    current_l_list = list()
    current_h_list = list()
    
    for (j in 1:len_h) {
      if (j == 1) {
        current_l_list[[j]] = L.fun(X = new_X, W = w_list[[j]])
      } else {
        current_l_list[[j]] = L.fun(X = current_h_list[[j-1]], W = w_list[[j]])
      }
      current_h_list[[j]] = ReLU.fun(x = current_l_list[[j]])
    }
    
    current_l_list[[len_h+1]] = L.fun(X = current_h_list[[len_h]], W = w_list[[len_h+1]])
    current_o = S.fun(x = current_l_list[[len_h+1]], eps = eps)
    
    return(current_o)
    
  }
  
  require(pROC)
  
  pred_y = pre_func(new_X = train_X_mat)
    
  roc_train <- roc(train_Y ~ pred_y)
  plot(roc_train, col = 'red')
  text(0.5, 0.5, paste0('AUC = ', formatC(roc_train[['auc']], 4, format = 'f')), col = 'red')
  
  if (!is.null(valid_X)) {
    
    valid_X_mat <- model.matrix(~ ., data = valid_X)
    valid_X_mat <- valid_X_mat[,-1]
    
    pred_y = pre_func(new_X = valid_X_mat)
    
    roc_valid <- roc(valid_Y ~ pred_y)
    plot(roc_valid, col = 'blue', add = TRUE)
    text(0.5, 0.4, paste0('AUC = ', formatC(roc_valid[['auc']], 4, format = 'f')), col = 'blue')
    legend('bottomright', c('train', 'valid'), col = c('red', 'blue'), lwd = 1)
    
  }
  
  return(list(pre_func = pre_func, W_list = W_list))
  
}

第一節：資料擴增(3)

我們先不做任何資料擴增，並用一個小神經網路：

DNN_list <- DEEP_MLP_Trainer(train_X = train_X, train_Y = train_Y, valid_X = valid_X, valid_Y = valid_Y, noise = 0, 
                             num.iteration = 10000, num.hidden = 30, eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 100)

當我們訓練一個比較大的網路時，嚴重的過度擬合問題就出現了：

DNN_list <- DEEP_MLP_Trainer(train_X = train_X, train_Y = train_Y, valid_X = valid_X, valid_Y = valid_Y, noise = 0, 
                             num.iteration = 10000, num.hidden = 300, eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 100)

第一節：資料擴增(4)

現在，讓我們給個noise，再試試看：

DNN_list <- DEEP_MLP_Trainer(train_X = train_X, train_Y = train_Y, valid_X = valid_X, valid_Y = valid_Y, noise = 0.3, 
                             num.iteration = 10000, num.hidden = 30, eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 100)

其實給點noise也不會對小模型造成太多影響，但我們看看大模型：

DNN_list <- DEEP_MLP_Trainer(train_X = train_X, train_Y = train_Y, valid_X = valid_X, valid_Y = valid_Y, noise = 0.3, 
                             num.iteration = 10000, num.hidden = 300, eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 100)

在完全同樣的結構之下，有noise是不是比起沒有noise來的更好呀！這就是資料擴增的威力！

第一節：資料擴增(5)

資料擴增能避免過度擬合的秘訣就在於增加變異，只要這個變異是「隨機」的，那就只會造成「軟化邊界」的效果。

– 舉例來說，如果原先的data在區分LVD與non-LVD時，最佳的Rate切點是70，那資料擴增就會讓69和71沒有差距這麼大，從而導致邊界平滑化。

你是不是想到，這是不是相類似於隨機森林對比決策樹的狀況，那就是給予一定的隨機性是好的。

– 而隨機森林是從「全樣本」中隨機抽取「子樣本」建立新的樹，這不是就跟上節課教到的「小批量」概念完全一樣嗎?

沒錯，降低批量大小也可以減少過度擬合的發生，讓我們來試試看：

DNN_list <- DEEP_MLP_Trainer(train_X = train_X, train_Y = train_Y, valid_X = valid_X, valid_Y = valid_Y, noise = 0, 
                             num.iteration = 10000, num.hidden = 300, eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 20)

現在有沒有覺得Yann LeCun說的事情很有道理呀！

F01

練習1：類別不平衡問題

在真實世界的資料中，類別不平衡是相當常見的狀況，而我們的資料同樣也出現了這個問題。

– 當然，我們已經選擇了使用「ROC曲線」進行分析，因此在評估指標上是沒有問題的，但事實上隨著批量越來越小，是不是有可能整個批量都沒有出現任何一個case?

– 因此，為了確保每個批量都能抽出同樣數量的case與control，我們通常會採取一個「過採樣」的策略，請你試著實現看看。

練習1答案(1)

讓我們多加一個參數，並修正相關內容：

DEEP_MLP_Trainer = function (train_X, train_Y, valid_X = NULL, valid_Y = NULL, noise = 0, oversampling = TRUE,
                             num.iteration = 500, num.hidden = c(10, 10, 10), eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 20) {
  
  #Functions
  
  #Forward
  
  S.fun = function (x, eps = eps) {
    S = 1/(1 + exp(-x))
    S[S < eps] = eps
    S[S > 1 - eps] = 1 - eps
    return(S)
  }
  
  ReLU.fun = function (x) {
    x[x < 0] <- 0
    return(x)
  }
  
  L.fun = function (X, W) {
    X.E = cbind(1, X)
    L = X.E %*% W
    return(L)
  }
  
  CE.fun = function (o, y, eps = eps) {
    loss = -1/length(y) * sum(y * log(o + eps) + (1 - y) * log(1 - o + eps))
    return(loss)
  }
  
  #Backward
  
  grad_o.fun = function (o, y) {
    return((o - y)/(o*(1-o)))
  }
  
  grad_s.fun = function (grad_o, o) {
    return(grad_o*(o*(1-o)))
  }
  
  grad_W.fun = function (grad_l, h) {
    h.E = cbind(1, h)
    return(t(h.E) %*% grad_l/nrow(h))
  }
  
  grad_h.fun = function (grad_l, W) {
    return(grad_l %*% t(W[-1,]))
  }
  
  grad_l.fun = function (grad_h, l) {
    de_l = l
    de_l[de_l<0] = 0
    de_l[de_l>0] = 1
    return(grad_h*de_l)
  }
  
  # Noise
  
  sd.vec <- NULL
  
  for (k in 1:ncol(train_X)) {
    
    if (class(train_X[,k])[1] %in% c('numeric', 'integer')) {
      
      sd.val <- sd(train_X[,k])
      sd.vec <- c(sd.vec, sd.val * noise)
      
    } else {
      
      sd.vec <- c(sd.vec, 0L)
      
    }
    
  }
  
  #initialization
  
  train_X_mat <- model.matrix(~ ., data = train_X)
  train_X_mat <- train_X_mat[,-1]
  train_Y_mat <- t(t(train_Y))
  
  W_list = list()
  M_list = list()
  N_list = list()
  
  len_h = length(num.hidden)
  
  for (w_seq in 1:(len_h+1)) {
    if (w_seq == 1) {
      NROW_W = ncol(train_X_mat) + 1
      NCOL_W = num.hidden[w_seq]
    } else if (w_seq == len_h+1) {
      NROW_W = num.hidden[w_seq - 1] + 1
      NCOL_W = ncol(train_Y_mat)
    } else {
      NROW_W = num.hidden[w_seq - 1] + 1
      NCOL_W = num.hidden[w_seq]
    }
    W_list[[w_seq]] = matrix(rnorm(NROW_W*NCOL_W, sd = 1), nrow = NROW_W, ncol = NCOL_W)
    M_list[[w_seq]] = matrix(0, nrow = NROW_W, ncol = NCOL_W)
    N_list[[w_seq]] = matrix(0, nrow = NROW_W, ncol = NCOL_W)
  }
  
  loss_seq = rep(0, num.iteration)
  
  #Caculating
  
  for (i in 1:num.iteration) {
    
    if (oversampling) {
      
      idx.pos = sample(which(train_Y == 1), batch_size / 2)
      idx.neg = sample(which(train_Y == 0), batch_size / 2)
      idx = c(idx.pos, idx.neg)
      
    } else {
      
      idx = sample(1:nrow(train_X_mat), batch_size)
      
    }
    
    noise_mat = t(matrix(rnorm(batch_size * length(sd.vec), sd = sd.vec), nrow = length(sd.vec)))
    sub_X_mat = train_X_mat[idx,] + noise_mat
    
    #Forward
    
    current_l_list = list()
    current_h_list = list()
    
    for (j in 1:len_h) {
      if (j == 1) {
        current_l_list[[j]] = L.fun(X = sub_X_mat, W = W_list[[j]])
      } else {
        current_l_list[[j]] = L.fun(X = current_h_list[[j-1]], W = W_list[[j]])
      }
      current_h_list[[j]] = ReLU.fun(x = current_l_list[[j]])
    }
    current_l_list[[len_h+1]] = L.fun(X = current_h_list[[len_h]], W = W_list[[len_h+1]])
    current_o = S.fun(x = current_l_list[[len_h+1]], eps = eps)
    loss_seq[i] = CE.fun(o = current_o, y = train_Y_mat[idx,], eps = eps)
    
    #Backward
    
    current_grad_l_list = list()
    current_grad_W_list = list()
    current_grad_h_list = list()
    
    current_grad_o = grad_o.fun(o = current_o, y = train_Y_mat[idx,])
    current_grad_l_list[[len_h+1]] = grad_s.fun(grad_o = current_grad_o, o = current_o)
    current_grad_W_list[[len_h+1]] = grad_W.fun(grad_l = current_grad_l_list[[len_h+1]], h = current_h_list[[len_h]])
    
    for (j in len_h:1) {
      current_grad_h_list[[j]] = grad_h.fun(grad_l = current_grad_l_list[[j+1]], W = W_list[[j+1]])
      current_grad_l_list[[j]] = grad_l.fun(grad_h = current_grad_h_list[[j]], l = current_l_list[[j]])
      if (j != 1) {
        current_grad_W_list[[j]] = grad_W.fun(grad_l = current_grad_l_list[[j]], h = current_h_list[[j - 1]])
      } else {
        current_grad_W_list[[j]] = grad_W.fun(grad_l = current_grad_l_list[[j]], h = sub_X_mat)
      }
    }
    
    if (optimizer == 'adam') {
      
      for (j in 1:(len_h+1)) {
        M_list[[j]] = beta1 * M_list[[j]] + (1 - beta1) * current_grad_W_list[[j]]
        N_list[[j]] = beta2 * N_list[[j]] + (1 - beta2) * current_grad_W_list[[j]]^2
        M.hat = M_list[[j]]/(1 - beta1^i)
        N.hat = N_list[[j]]/(1 - beta2^i)
        W_list[[j]] = W_list[[j]] - lr*M.hat/sqrt(N.hat+eps)
      }
      
    } else if (optimizer == 'sgd') {
      
      for (j in 1:(len_h+1)) {
        M_list[[j]] = beta1 * M_list[[j]] + lr * current_grad_W_list[[j]]
        W_list[[j]] = W_list[[j]] - M_list[[j]]
      }
      
    } else {
      stop('optimizer must be selected from "sgd" or "adam".')
    }
    
  }
  
  pre_func = function (new_X, w_list = W_list) {
    
    current_l_list = list()
    current_h_list = list()
    
    for (j in 1:len_h) {
      if (j == 1) {
        current_l_list[[j]] = L.fun(X = new_X, W = w_list[[j]])
      } else {
        current_l_list[[j]] = L.fun(X = current_h_list[[j-1]], W = w_list[[j]])
      }
      current_h_list[[j]] = ReLU.fun(x = current_l_list[[j]])
    }
    
    current_l_list[[len_h+1]] = L.fun(X = current_h_list[[len_h]], W = w_list[[len_h+1]])
    current_o = S.fun(x = current_l_list[[len_h+1]], eps = eps)
    
    return(current_o)
    
  }
  
  require(pROC)
  
  pred_y = pre_func(new_X = train_X_mat)
    
  roc_train <- roc(train_Y ~ pred_y)
  plot(roc_train, col = 'red')
  text(0.5, 0.5, paste0('AUC = ', formatC(roc_train[['auc']], 4, format = 'f')), col = 'red')
  
  if (!is.null(valid_X)) {
    
    valid_X_mat <- model.matrix(~ ., data = valid_X)
    valid_X_mat <- valid_X_mat[,-1]
    
    pred_y = pre_func(new_X = valid_X_mat)
    
    roc_valid <- roc(valid_Y ~ pred_y)
    plot(roc_valid, col = 'blue', add = TRUE)
    text(0.5, 0.4, paste0('AUC = ', formatC(roc_valid[['auc']], 4, format = 'f')), col = 'blue')
    legend('bottomright', c('train', 'valid'), col = c('red', 'blue'), lwd = 1)
    
  }
  
  return(list(pre_func = pre_func, W_list = W_list))
  
}

練習1答案(2)

這是使用了oversampling的實驗結果：

DNN_list <- DEEP_MLP_Trainer(train_X = train_X, train_Y = train_Y, valid_X = valid_X, valid_Y = valid_Y,
                             noise = 0.3, oversampling = TRUE, 
                             num.iteration = 10000, num.hidden = 300, eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 20)

又多了一個參數能調，看來要做個預測模型真不容易。

第二節：正則化(1)

還記得為什麼彈性網路比邏輯斯回歸來的更好嗎?原因就在於正則化(Regularization)的效果。

– 這是一個可以防止過度擬合的強大武器，而且非常容易使用。就是在原先的loss之上下了一個額外懲罰權重大小的loss

– 想複習可以點這裡。

我們將統計領域中常用的正則化技術分為兩類：

L1 Regularization - 代表技術有套索迴歸(LASSO regression)

\[ \begin{align} \hat{loss_1} & = loss + \lambda \times ||W|| \end{align} \]

L2 Regularization - 代表技術有脊迴歸(Ridge regression)

\[ \begin{align} \hat{loss_2} & = loss + \frac{\lambda}{2} \times||W||^2 \end{align} \]

第二節：正則化(2)

由於他的導函數非常簡單，因此我們其實可以在進行梯度下降法時直接使用：

\[ \begin{align} \frac{\partial}{\partial W} \hat{loss_1} & = grad.W + \lambda \frac{W}{|W|} \\ \frac{\partial}{\partial W} \hat{loss_2} & = grad.W + \lambda W \end{align} \]

已L2正則化為例(深度神經網路通常是使用他)，在進行梯度下降法時，每次梯度更新時當前的權重(\(W\))都需要減去自己的梯度，而其中\(\lambda W\)這一項暗示著他必須減去自己本身，因此\(\lambda\)經常被稱為「權重衰減系數」(weight decay coefficient)。

– 很明顯的，當\(\lambda\)很大的時候，那所有權重都將會很難離開0。

練習2：將正則化技術引入深度神經網路中

讓我們將L2正則化的思想套用到矩陣化的MLP之中，讓我們先寫出預測式及損失函數(這裡還是以2元分類為例)。

– 這裡我們引入符號\(||W||^2\)代表向量(矩陣)\(W\)的距離值(也就是平方和)

\[ \begin{align} l_1 & = L^1_d(x^E,W^1_d) \\ h_1 & = ReLU(l_1) \\ l_2 & = L^2_1(h_1^E,W^2_1) \\ o & = S(l_2) \\\\ loss & = CE(y, o) + L2Reg(W) \\ & = -\left(y \cdot log(o) + (1-y) \cdot log(1-o)\right) + \frac{\lambda}{2} ||W^2_1||^2 + \frac{\lambda}{2} ||W^1_d||^2 \end{align} \]

有了這樣的式子之後，我們再把各項的梯度函數給寫出來：

\[ \begin{align} grad.o & = \frac{\partial}{\partial o}loss = \frac{o-y}{o(1-o)} \\ grad.l_2 & = \frac{\partial}{\partial l_2}loss = grad.o \otimes \frac{\partial}{\partial l_2}o= o-y \\ grad.W^2_1 & = \frac{\partial}{\partial W^2_1}loss = grad.l_2 \otimes \frac{\partial}{\partial W^2_1}l_2 + \frac{\partial}{\partial W^2_1}\frac{\lambda}{2}||W^2_1||^2 = \frac{{1}}{n} \otimes (h_1^E)^T \bullet grad.l_2 + \lambda W^2_1\\ grad.h_1^E & = \frac{\partial}{\partial h_1^E}loss = grad.l_2 \otimes \frac{\partial}{\partial h_1^E}l_2 = grad.l_2 \bullet (W^2_1)^T \\ grad.l_1 & = \frac{\partial}{\partial l_1}loss = grad.h_1 \otimes \frac{\partial}{\partial l_1}h_1 = grad.h_1 \otimes \frac{\partial}{\partial l_1}ReLU(l_1) \\ grad.W^1_d & = \frac{\partial}{\partial W^1_d}loss = grad.l_1 \otimes \frac{\partial}{\partial W^1_d}l_1 + \frac{\partial}{\partial W^1_d}\frac{\lambda}{2}||W^1_d||^2 = \frac{{1}}{n} \otimes (x^E)^T \bullet grad.l_1 + \lambda W^1_d \end{align} \]

有了這樣的概念後，請把剛剛練習1答案的程式碼引入L2正則化技術。

– 需要注意的是，由於梯度計算完成後我們通常會再乘上一個學習率\(lr\)，所以最後的衰減率會是\(lr \times \lambda\)。為了避免應用上的麻煩，我們通常會把這一項獨立出來不讓他乘以學習率，從而固定其影響力。

練習2答案(1)

讓我們再多加一個參數，並修正相關內容：

DEEP_MLP_Trainer = function (train_X, train_Y, valid_X = NULL, valid_Y = NULL,
                             noise = 0, oversampling = TRUE, lambda = 0.001,
                             num.iteration = 500, num.hidden = c(10, 10, 10), eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 20) {
  
  #Functions
  
  #Forward
  
  S.fun = function (x, eps = eps) {
    S = 1/(1 + exp(-x))
    S[S < eps] = eps
    S[S > 1 - eps] = 1 - eps
    return(S)
  }
  
  ReLU.fun = function (x) {
    x[x < 0] <- 0
    return(x)
  }
  
  L.fun = function (X, W) {
    X.E = cbind(1, X)
    L = X.E %*% W
    return(L)
  }
  
  CE.fun = function (o, y, eps = eps) {
    loss = -1/length(y) * sum(y * log(o + eps) + (1 - y) * log(1 - o + eps))
    return(loss)
  }
  
  #Backward
  
  grad_o.fun = function (o, y) {
    return((o - y)/(o*(1-o)))
  }
  
  grad_s.fun = function (grad_o, o) {
    return(grad_o*(o*(1-o)))
  }
  
  grad_W.fun = function (grad_l, h) {
    h.E = cbind(1, h)
    return(t(h.E) %*% grad_l/nrow(h))
  }
  
  grad_h.fun = function (grad_l, W) {
    return(grad_l %*% t(W[-1,]))
  }
  
  grad_l.fun = function (grad_h, l) {
    de_l = l
    de_l[de_l<0] = 0
    de_l[de_l>0] = 1
    return(grad_h*de_l)
  }
  
  # Noise
  
  sd.vec <- NULL
  
  for (k in 1:ncol(train_X)) {
    
    if (class(train_X[,k])[1] %in% c('numeric', 'integer')) {
      
      sd.val <- sd(train_X[,k])
      sd.vec <- c(sd.vec, sd.val * noise)
      
    } else {
      
      sd.vec <- c(sd.vec, 0L)
      
    }
    
  }
  
  #initialization
  
  train_X_mat <- model.matrix(~ ., data = train_X)
  train_X_mat <- train_X_mat[,-1]
  train_Y_mat <- t(t(train_Y))
  
  W_list = list()
  M_list = list()
  N_list = list()
  
  len_h = length(num.hidden)
  
  for (w_seq in 1:(len_h+1)) {
    if (w_seq == 1) {
      NROW_W = ncol(train_X_mat) + 1
      NCOL_W = num.hidden[w_seq]
    } else if (w_seq == len_h+1) {
      NROW_W = num.hidden[w_seq - 1] + 1
      NCOL_W = ncol(train_Y_mat)
    } else {
      NROW_W = num.hidden[w_seq - 1] + 1
      NCOL_W = num.hidden[w_seq]
    }
    W_list[[w_seq]] = matrix(rnorm(NROW_W*NCOL_W, sd = 1), nrow = NROW_W, ncol = NCOL_W)
    M_list[[w_seq]] = matrix(0, nrow = NROW_W, ncol = NCOL_W)
    N_list[[w_seq]] = matrix(0, nrow = NROW_W, ncol = NCOL_W)
  }
  
  loss_seq = rep(0, num.iteration)
  
  #Caculating
  
  for (i in 1:num.iteration) {
    
    if (oversampling) {
      
      idx.pos = sample(which(train_Y == 1), batch_size / 2)
      idx.neg = sample(which(train_Y == 0), batch_size / 2)
      idx = c(idx.pos, idx.neg)
      
    } else {
      
      idx = sample(1:nrow(train_X_mat), batch_size)
      
    }
    
    noise_mat = t(matrix(rnorm(batch_size * length(sd.vec), sd = sd.vec), nrow = length(sd.vec)))
    sub_X_mat = train_X_mat[idx,] + noise_mat
    
    #Forward
    
    current_l_list = list()
    current_h_list = list()
    
    for (j in 1:len_h) {
      if (j == 1) {
        current_l_list[[j]] = L.fun(X = sub_X_mat, W = W_list[[j]])
      } else {
        current_l_list[[j]] = L.fun(X = current_h_list[[j-1]], W = W_list[[j]])
      }
      current_h_list[[j]] = ReLU.fun(x = current_l_list[[j]])
    }
    current_l_list[[len_h+1]] = L.fun(X = current_h_list[[len_h]], W = W_list[[len_h+1]])
    current_o = S.fun(x = current_l_list[[len_h+1]], eps = eps)
    loss_seq[i] = CE.fun(o = current_o, y = train_Y_mat[idx,], eps = eps)
    
    #Backward
    
    current_grad_l_list = list()
    current_grad_W_list = list()
    current_grad_h_list = list()
    
    current_grad_o = grad_o.fun(o = current_o, y = train_Y_mat[idx,])
    current_grad_l_list[[len_h+1]] = grad_s.fun(grad_o = current_grad_o, o = current_o)
    current_grad_W_list[[len_h+1]] = grad_W.fun(grad_l = current_grad_l_list[[len_h+1]], h = current_h_list[[len_h]])
    
    for (j in len_h:1) {
      current_grad_h_list[[j]] = grad_h.fun(grad_l = current_grad_l_list[[j+1]], W = W_list[[j+1]])
      current_grad_l_list[[j]] = grad_l.fun(grad_h = current_grad_h_list[[j]], l = current_l_list[[j]])
      if (j != 1) {
        current_grad_W_list[[j]] = grad_W.fun(grad_l = current_grad_l_list[[j]], h = current_h_list[[j - 1]])
      } else {
        current_grad_W_list[[j]] = grad_W.fun(grad_l = current_grad_l_list[[j]], h = sub_X_mat)
      }
    }
    
    if (optimizer == 'adam') {
      
      for (j in 1:(len_h+1)) {
        M_list[[j]] = beta1 * M_list[[j]] + (1 - beta1) * current_grad_W_list[[j]]
        N_list[[j]] = beta2 * N_list[[j]] + (1 - beta2) * current_grad_W_list[[j]]^2
        M.hat = M_list[[j]]/(1 - beta1^i)
        N.hat = N_list[[j]]/(1 - beta2^i)
        W_list[[j]] = W_list[[j]] - lr * M.hat / sqrt(N.hat+eps) - lambda * W_list[[j]]
      }
      
    } else if (optimizer == 'sgd') {
      
      for (j in 1:(len_h+1)) {
        M_list[[j]] = beta1 * M_list[[j]] + lr * current_grad_W_list[[j]]
        W_list[[j]] = W_list[[j]] - M_list[[j]] - lambda * W_list[[j]]
      }
      
    } else {
      stop('optimizer must be selected from "sgd" or "adam".')
    }
    
  }
  
  pre_func = function (new_X, w_list = W_list) {
    
    current_l_list = list()
    current_h_list = list()
    
    for (j in 1:len_h) {
      if (j == 1) {
        current_l_list[[j]] = L.fun(X = new_X, W = w_list[[j]])
      } else {
        current_l_list[[j]] = L.fun(X = current_h_list[[j-1]], W = w_list[[j]])
      }
      current_h_list[[j]] = ReLU.fun(x = current_l_list[[j]])
    }
    
    current_l_list[[len_h+1]] = L.fun(X = current_h_list[[len_h]], W = w_list[[len_h+1]])
    current_o = S.fun(x = current_l_list[[len_h+1]], eps = eps)
    
    return(current_o)
    
  }
  
  require(pROC)
  
  pred_y = pre_func(new_X = train_X_mat)
    
  roc_train <- roc(train_Y ~ pred_y)
  plot(roc_train, col = 'red')
  text(0.5, 0.5, paste0('AUC = ', formatC(roc_train[['auc']], 4, format = 'f')), col = 'red')
  
  if (!is.null(valid_X)) {
    
    valid_X_mat <- model.matrix(~ ., data = valid_X)
    valid_X_mat <- valid_X_mat[,-1]
    
    pred_y = pre_func(new_X = valid_X_mat)
    
    roc_valid <- roc(valid_Y ~ pred_y)
    plot(roc_valid, col = 'blue', add = TRUE)
    text(0.5, 0.4, paste0('AUC = ', formatC(roc_valid[['auc']], 4, format = 'f')), col = 'blue')
    legend('bottomright', c('train', 'valid'), col = c('red', 'blue'), lwd = 1)
    
  }
  
  return(list(pre_func = pre_func, W_list = W_list))
  
}

練習2答案(2)

又多了一個參數能調，看來要做個預測模型真不容易。

DNN_list <- DEEP_MLP_Trainer(train_X = train_X, train_Y = train_Y, valid_X = valid_X, valid_Y = valid_Y,
                             noise = 0.3, oversampling = TRUE, lambda = 0.001,
                             num.iteration = 10000, num.hidden = 300, eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 20)

第三節：Dropout(1)

今天上課講了半天，老是在講20世紀的統計技術，你是否覺得了無新意?

– 我們要再次回到2012年，當多倫多大學的Alex Krizhevsky、Ilya Sutskever以及Geoffrey Hinton重新使用神經網路AlexNet成功奪冠時，世界上其他人都犯蠢了都沒想到這個方法?事實上，能夠在取得如此重大的突破絕對是同時解決了許多世紀難題才有的成果！

F02

從第二節課開始我們都不斷地圍繞這篇paper講故事，他的成功背景除了GPU時代的來臨之外，還做了眾多演算技術上的優化，其中一點重要的貢獻「大量使用ReLU作為非線性轉換函數」在前面已經提到過了，而另外一個最重要的創見就是一個防止過度擬合的技術：Dropout。

– 這邊帶大家稍微了解一下時代背景，我們先不考慮計算能力只關注在統計問題。整個AlexNet待解有超過6200萬，而當時ImageNet資料庫所能提供的也只有120萬的數據量，因此過擬合問題非常嚴重，即使運用了資料擴增、正則化都並不足以解決這個問題。

Dropout(2)

Dropout技術的主要邏輯是在訓練神經網路時，每次隨機忽略一部分神經元，從而讓所有神經元都有機會被充分訓練。

F03

試想在一間教室內坐著無數學生(神經元)，而考試卷是出給所有學生(神經網路)一起作答，會發生什麼事情?肯定是有些用功的學生很快的學會了，而剩下的就開始擺爛，反正答案都全對了！

– 這時候如果加入了Dropout機制，也就是在每次考試時隨機的抽走一半的學生，讓剩下的學生進行考試，這是不是會更刺激每個學生學習?

Dropout(3)

讓我們用數學式來表達Dropout，我們來定義一個新函數\(dropout(x)\)，他的功用在隨機的把矩陣\(x\)內的元素變成0，並且同時rescale其他剩餘的數字(在這裡假定參數\(dp\)為drop rate)：

\[ dropout(x) = \left\{ \begin{array} - \frac{x}{1-dp} & \mbox{ otherwise} \\ 0 & \mbox{ if sampled (rate = dp)} \end{array} \right. \]

– 這裡你會發現，由於\(dropout(x)\)與\(ReLU(x)\)非常非常的像，所以他們的偏導函數也非常類似：

\[ \frac{\partial}{\partial x}dropout(x) = \left\{ \begin{array} - \frac{1}{1 -dp} & \mbox{ otherwise} \\ 0 & \mbox{ if sampled (rate = dp)} \end{array} \right. \]

這樣一看，Dropout似乎就沒有這麼高級了，好像很容易就實現了，是吧?

練習3：將Dropout引入深度神經網路中

讓我們將Dropout融入原來的MLP之中吧(這個推導不含正則化項)！

\[ \begin{align} l_1 & = L^1_d(x^E,W^1_d) \\ dp_1 & = dropout(l_1) \\ h_1 & = ReLU(dp_1) \\ l_2 & = L^2_1(h_1^E,W^2_1) \\ dp_2 & = dropout(l_2) \\ o & = S(dp_2) \\ loss & = CE(y, o) = -\left(y \cdot log(o) + (1-y) \cdot log(1-o)\right) \end{align} \]

看起來滿簡單的，讓我們求各項的偏導函數吧！

\[ \begin{align} grad.o & = \frac{\partial}{\partial o}loss = \frac{o-y}{o(1-o)} \\ grad.dp_2 & = \frac{\partial}{\partial dp_2}loss = grad.o \otimes \frac{\partial}{\partial dp_2}o= o-y \\ grad.l_2 & = \frac{\partial}{\partial l_2}loss = grad.dp_2 \otimes \frac{\partial}{\partial l_2}dp_2 = (o-y) \otimes \frac{\partial}{\partial l_2}dropout(l_2) \\ grad.W^2_1 & = \frac{\partial}{\partial W^2_1}loss = grad.l_2 \otimes \frac{\partial}{\partial W^2_1}l_2 = \frac{{1}}{n} \otimes (h_1^E)^T \bullet grad.l_2\\ grad.h_1^E & = \frac{\partial}{\partial h_1^E}loss = grad.l_2 \otimes \frac{\partial}{\partial h_1^E}l_2 = grad.l_2 \bullet (W^2_1)^T \\ grad.dp_1 & = \frac{\partial}{\partial dp_1}loss = grad.h_1 \otimes \frac{\partial}{\partial dp_1}h_1 = grad.h_1 \otimes \frac{\partial}{\partial dp_1}ReLU(dp_1) \\ grad.l_1 & = \frac{\partial}{\partial l_1}loss = grad.dp_1 \otimes \frac{\partial}{\partial l_1}dp_1 = grad.dp_1 \otimes \frac{\partial}{\partial l_1}dropout(l_1) \\ grad.W^1_d & = \frac{\partial}{\partial W^1_d}loss = grad.l_1 \otimes \frac{\partial}{\partial W^1_d}l_1 = \frac{{1}}{n} \otimes (x^E)^T \bullet grad.l_1 \end{align} \]

現在請你試著利用練習3的答案進一步的把Dropout layer加進每一個線性預測層之後，並嘗試使用Dropout對測試集預測的效果！

練習3答案(1)

讓我們再多加一個參數，並修正相關內容：

DEEP_MLP_Trainer = function (train_X, train_Y, valid_X = NULL, valid_Y = NULL,
                             noise = 0, oversampling = TRUE, lambda = 0.001, dp = 0.5,
                             num.iteration = 500, num.hidden = c(10, 10, 10), eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 20) {
  
  #Functions
  
  #Forward
  
  S.fun = function (x, eps = eps) {
    S = 1/(1 + exp(-x))
    S[S < eps] = eps
    S[S > 1 - eps] = 1 - eps
    return(S)
  }
  
  ReLU.fun = function (x) {
    x[x < 0] <- 0
    return(x)
  }
  
  #---------------------------------------------#
  
  dropout.fun = function (x, dp = dp) {
    len_x = length(x)
    x[sample(1:len_x, len_x * dp)] <- 0
    return(x/(1-dp))
  }
  
  #---------------------------------------------#
  
  L.fun = function (X, W) {
    X.E = cbind(1, X)
    L = X.E %*% W
    return(L)
  }
  
  CE.fun = function (o, y, eps = eps) {
    loss = -1/length(y) * sum(y * log(o + eps) + (1 - y) * log(1 - o + eps))
    return(loss)
  }
  
  #Backward
  
  grad_o.fun = function (o, y) {
    return((o - y)/(o*(1-o)))
  }
  
  grad_s.fun = function (grad_o, o) {
    return(grad_o*(o*(1-o)))
  }
  
  grad_W.fun = function (grad_l, h) {
    h.E = cbind(1, h)
    return(t(h.E) %*% grad_l/nrow(h))
  }
  
  grad_h.fun = function (grad_l, W) {
    return(grad_l %*% t(W[-1,]))
  }
  
  #---------------------------------------------#
  
  grad_dp.fun = function (grad_h, DP, dp = dp) {
    de_DP = DP
    de_DP[de_DP<0] = 0
    de_DP[de_DP>0] = 1
    return(grad_h*de_DP)
  }
  
  grad_l.fun = function (grad_dp, DP, dp = dp) {
    de_DP = DP
    de_DP[de_DP == 0] = 0
    de_DP[de_DP != 0] = 1/(1-dp)
    return(grad_dp*de_DP)
  }
  
  #---------------------------------------------#
  
  # Noise
  
  sd.vec <- NULL
  
  for (k in 1:ncol(train_X)) {
    
    if (class(train_X[,k])[1] %in% c('numeric', 'integer')) {
      
      sd.val <- sd(train_X[,k])
      sd.vec <- c(sd.vec, sd.val * noise)
      
    } else {
      
      sd.vec <- c(sd.vec, 0L)
      
    }
    
  }
  
  #initialization
  
  train_X_mat <- model.matrix(~ ., data = train_X)
  train_X_mat <- train_X_mat[,-1]
  train_Y_mat <- t(t(train_Y))
  
  W_list = list()
  M_list = list()
  N_list = list()
  
  len_h = length(num.hidden)
  
  for (w_seq in 1:(len_h+1)) {
    if (w_seq == 1) {
      NROW_W = ncol(train_X_mat) + 1
      NCOL_W = num.hidden[w_seq]
    } else if (w_seq == len_h+1) {
      NROW_W = num.hidden[w_seq - 1] + 1
      NCOL_W = ncol(train_Y_mat)
    } else {
      NROW_W = num.hidden[w_seq - 1] + 1
      NCOL_W = num.hidden[w_seq]
    }
    W_list[[w_seq]] = matrix(rnorm(NROW_W*NCOL_W, sd = 1), nrow = NROW_W, ncol = NCOL_W)
    M_list[[w_seq]] = matrix(0, nrow = NROW_W, ncol = NCOL_W)
    N_list[[w_seq]] = matrix(0, nrow = NROW_W, ncol = NCOL_W)
  }
  
  loss_seq = rep(0, num.iteration)
  
  #Caculating
  
  for (i in 1:num.iteration) {
    
    if (oversampling) {
      
      idx.pos = sample(which(train_Y == 1), batch_size / 2)
      idx.neg = sample(which(train_Y == 0), batch_size / 2)
      idx = c(idx.pos, idx.neg)
      
    } else {
      
      idx = sample(1:nrow(train_X_mat), batch_size)
      
    }
    
    noise_mat = t(matrix(rnorm(batch_size * length(sd.vec), sd = sd.vec), nrow = length(sd.vec)))
    sub_X_mat = train_X_mat[idx,] + noise_mat
    
    #Forward
    
    current_l_list = list()
    #---------------------------------------------#
    current_dp_list = list()
    #---------------------------------------------#
    current_h_list = list()
    
    for (j in 1:len_h) {
      if (j == 1) {
        current_l_list[[j]] = L.fun(X = sub_X_mat, W = W_list[[j]])
      } else {
        current_l_list[[j]] = L.fun(X = current_h_list[[j-1]], W = W_list[[j]])
      }
      #---------------------------------------------#
      current_dp_list[[j]] = dropout.fun(x = current_l_list[[j]], dp = dp)
      current_h_list[[j]] = ReLU.fun(x = current_dp_list[[j]])
      #---------------------------------------------#
    }
    current_l_list[[len_h+1]] = L.fun(X = current_h_list[[len_h]], W = W_list[[len_h+1]])
    #---------------------------------------------#
    current_dp_list[[len_h+1]] = dropout.fun(x = current_l_list[[len_h+1]], dp = dp)
    current_o = S.fun(x = current_dp_list[[len_h+1]], eps = eps)
    #---------------------------------------------#
    loss_seq[i] = CE.fun(o = current_o, y = train_Y_mat[idx,], eps = eps)
    
    #Backward
    
    current_grad_l_list = list()
    #---------------------------------------------#
    current_grad_dp_list = list()
    #---------------------------------------------#
    current_grad_W_list = list()
    current_grad_h_list = list()
    
    current_grad_o = grad_o.fun(o = current_o, y = train_Y_mat[idx,])
    #---------------------------------------------#
    current_grad_dp_list[[len_h+1]] = grad_s.fun(grad_o = current_grad_o, o = current_o)
    current_grad_l_list[[len_h+1]] = grad_l.fun(grad_dp = current_grad_dp_list[[len_h+1]], DP = current_dp_list[[len_h+1]], dp = dp)
    #---------------------------------------------#
    current_grad_W_list[[len_h+1]] = grad_W.fun(grad_l = current_grad_l_list[[len_h+1]], h = current_h_list[[len_h]])
    
    for (j in len_h:1) {
      current_grad_h_list[[j]] = grad_h.fun(grad_l = current_grad_l_list[[j+1]], W = W_list[[j+1]])
      #---------------------------------------------#
      current_grad_dp_list[[j]] = grad_dp.fun(grad_h = current_grad_h_list[[j]], DP = current_dp_list[[j]], dp = dp)
      current_grad_l_list[[j]] = grad_l.fun(grad_dp = current_grad_dp_list[[j]], DP = current_dp_list[[j]], dp = dp)
      #---------------------------------------------#
      if (j != 1) {
        current_grad_W_list[[j]] = grad_W.fun(grad_l = current_grad_l_list[[j]], h = current_h_list[[j - 1]])
      } else {
        current_grad_W_list[[j]] = grad_W.fun(grad_l = current_grad_l_list[[j]], h = sub_X_mat)
      }
    }
    
    if (optimizer == 'adam') {
      
      for (j in 1:(len_h+1)) {
        M_list[[j]] = beta1 * M_list[[j]] + (1 - beta1) * current_grad_W_list[[j]]
        N_list[[j]] = beta2 * N_list[[j]] + (1 - beta2) * current_grad_W_list[[j]]^2
        M.hat = M_list[[j]]/(1 - beta1^i)
        N.hat = N_list[[j]]/(1 - beta2^i)
        W_list[[j]] = W_list[[j]] - lr * M.hat / sqrt(N.hat+eps) - lambda * W_list[[j]]
      }
      
    } else if (optimizer == 'sgd') {
      
      for (j in 1:(len_h+1)) {
        M_list[[j]] = beta1 * M_list[[j]] + lr * current_grad_W_list[[j]]
        W_list[[j]] = W_list[[j]] - M_list[[j]] - lambda * W_list[[j]]
      }
      
    } else {
      stop('optimizer must be selected from "sgd" or "adam".')
    }
    
  }
  
  pre_func = function (new_X, w_list = W_list) {
    
    current_l_list = list()
    current_h_list = list()
    
    for (j in 1:len_h) {
      if (j == 1) {
        current_l_list[[j]] = L.fun(X = new_X, W = w_list[[j]])
      } else {
        current_l_list[[j]] = L.fun(X = current_h_list[[j-1]], W = w_list[[j]])
      }
      current_h_list[[j]] = ReLU.fun(x = current_l_list[[j]])
    }
    
    current_l_list[[len_h+1]] = L.fun(X = current_h_list[[len_h]], W = w_list[[len_h+1]])
    current_o = S.fun(x = current_l_list[[len_h+1]], eps = eps)
    
    return(current_o)
    
  }
  
  require(pROC)
  
  pred_y = pre_func(new_X = train_X_mat)
    
  roc_train <- roc(train_Y ~ pred_y)
  plot(roc_train, col = 'red')
  text(0.5, 0.5, paste0('AUC = ', formatC(roc_train[['auc']], 4, format = 'f')), col = 'red')
  
  if (!is.null(valid_X)) {
    
    valid_X_mat <- model.matrix(~ ., data = valid_X)
    valid_X_mat <- valid_X_mat[,-1]
    
    pred_y = pre_func(new_X = valid_X_mat)
    
    roc_valid <- roc(valid_Y ~ pred_y)
    plot(roc_valid, col = 'blue', add = TRUE)
    text(0.5, 0.4, paste0('AUC = ', formatC(roc_valid[['auc']], 4, format = 'f')), col = 'blue')
    legend('bottomright', c('train', 'valid'), col = c('red', 'blue'), lwd = 1)
    
  }
  
  return(list(pre_func = pre_func, W_list = W_list))
  
}

練習3答案(2)

你有沒有感到絕望，參數這麼多，以後做個實驗要多久？

DNN_list <- DEEP_MLP_Trainer(train_X = train_X, train_Y = train_Y, valid_X = valid_X, valid_Y = valid_Y,
                             noise = 0.3, oversampling = TRUE, lambda = 0.001, dp = 0.5,
                             num.iteration = 10000, num.hidden = 300, eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 20)

第四節：學習率調整與提早停止(1)

最後再讓我們進行一個實驗，為了避免隨機batch造成的影響，我們使用固定種子進行實驗：

– 這是在第5000代的時候停下來：

set.seed(0)

DNN_list <- DEEP_MLP_Trainer(train_X = train_X, train_Y = train_Y, valid_X = valid_X, valid_Y = valid_Y,
                             noise = 0, oversampling = TRUE, lambda = 0, dp = 0,
                             num.iteration = 5000, num.hidden = 30, eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 100)

– 這是在第10000代的時候停下來：

set.seed(0)

DNN_list <- DEEP_MLP_Trainer(train_X = train_X, train_Y = train_Y, valid_X = valid_X, valid_Y = valid_Y,
                             noise = 0, oversampling = TRUE, lambda = 0, dp = 0,
                             num.iteration = 10000, num.hidden = 30, eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 100)

– 這是在第20000代的時候停下來：

set.seed(0)

DNN_list <- DEEP_MLP_Trainer(train_X = train_X, train_Y = train_Y, valid_X = valid_X, valid_Y = valid_Y,
                             noise = 0, oversampling = TRUE, lambda = 0, dp = 0,
                             num.iteration = 20000, num.hidden = 30, eps = 1e-8,
                             lr = 0.001, beta1 = 0.9, beta2 = 0.999, optimizer = 'adam', batch_size = 100)

第四節：學習率調整與提早停止(2)

你應該有注意到，訓練的久不代表好，主要的原因是容易過度擬合，特別是在複雜的網路之下

– 要怎樣解決這個問題呢？我們必須每一訓練到一個階段，就確認一下目前在「驗證組」的準確度是否繼續上升。

– 這是不是有點像「極限梯度提升機」在運算時所做的事情?

實務上，有些paper經常會有這樣的描述，這就是在說過停止的：

– … We trained the networks with minibatches of size 8 and used an initial learning rate of 0.0001 that was decayed by a factor of 10 each time the loss on the tuning set plateaued after an epoch (a full pass over the training set). In order to prevent the networks from overfitting, early stopping was performed by saving the network after every epoch and choosing the saved network with the lowest loss on the tuning set. …

這段話的每個字都充滿含意，他的意思是說訓練的過程學習率與是否停止完全取決於tuning set，也就是我們的驗證組。

第四節：學習率調整與提早停止(3)

這裡我們就不演示怎樣修正程式碼了，他的過程比較簡單但語法非常長。

– 如果你很想知道怎樣做，醫療人工智慧實作- 胸部X光左心室功能障礙分類挑戰中有範例語法，你可以參考

這裡有些關鍵的字句需要了解，甚麼叫做「epoch」，他說是「a full pass over the training set」。

– 這意思是說原始樣本假設是1000，批量大小為20，那一個「epoch」就是50個「batch」；若批量大小為25，那一個「epoch」就是40個「batch」。

– 剛剛我們引用的那篇paper非常的經典，他提到了非常多的「訓練技巧」，我們通常會依據「驗證組」選擇一個最佳模型，最後再放到「測試組」上做最終裁決。

這些眾多的避免過度擬合方法是不是讓你看的眼花撩亂，所以說深度學習的模型訓練其實非常需要你的程式能力，一般的套件非常難提供這麼多的選項，就算提供了也不見得符合你的需求。

結語

深度學習領域中存在三大理論問題：過度擬合問題、梯度消失問題、權重初始化問題，而我們已經在過度擬合問題上跟上了2012年的時代，並且發現了網路要訓練的準，那鐵定是需要大量的「參數實驗」，從而找到一組最佳參數讓自己的模型在測試集中準確性較高。
隨著連續3節課的推導，你是否發現其實只要你熟練使用連鎖律，那其實我們可以疊加任意多「可微分」的層，從而建構越來越複雜的神經網路。

– 因此，目前對於深度學習開發的主流都是透過「框架/平台」，由你指定一個由「眾多可微分層累加的預測函數」，而「框架/平台」負責利用連鎖律幫你解微分，從而獲得梯度優化整個網路取得權重參數，因此之後的課程我們將開始使用「框架/平台」進行開發。

– 到時候假使我們遇到了一個新的函數，那也只要解出該函數的偏導函數你就應該有能力將其應用於自己的模型中。

目前為止，你應該都覺得這些基本的東西學了沒甚麼用，其實目前你學到的充其量就是一個「非線性預測器」。但痛苦的過程已經告一段落了，目前為止我們已經把比較數學理論的部分都推導的差不多了，後面的課程我們會將重心轉向實務開發，讓你能親手打造一個深度神經網路！

深度學習理論與實務

前言