深度學習理論與實務
林嶔 (Lin, Chin)
Lesson 4 MxNet框架編程介紹
第一節：MxNet基本介紹(1)
– 透過前幾節課的推導，我們應該了解到這種開發框架其實也不是很難實現，這是因為目前的人工神經網路無論多深多複雜，也一定可以被定義成一個複雜函數的組合，透過分別解微分以及連鎖律我們將能求得任何參數的梯度，從而能實現梯度下降法來訓練網路。
– 但要注意一點，僅有64位元的作業系統能安裝MxNet。
– 他的安裝方法比較特別，並且有安裝GPU版本的方法，下面是在WINDOW系統安裝CPU版本的作法：
install.packages("https://s3.ca-central-1.amazonaws.com/jeremiedb/share/mxnet/CPU/3.6/mxnet.zip", repos = NULL)
– 或者採這個方法：
cran <- getOption("repos")
cran["dmlc"] <- "https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/R/CRAN/"
options(repos = cran)
install.packages("mxnet")
第一節：MxNet基本介紹(2)
- 在之前的課程中我們開發神經網路時，我們通常會先定義下面幾個元素後再進行開發：
資料結構
預測函數(網路結構)
優化方法
- 因此在MxNet中他的編寫邏輯也類似，我們要依序寫出：
Iterator
Model architecture
Optimizer
- 後面我們依序學習如何編寫這三個部分，並了解怎樣把3個部分結合在一起運算。
第一節：MxNet基本介紹(3)
- 讓我們直接使用真實資料來進行實驗吧！請至這裡下載範例資料
– 這次我們會示範很多依變項的分析語法，所以就不先進行特別的前處理
dat <- read.csv("ECG_train.csv", header = TRUE, fileEncoding = 'CP950', stringsAsFactors = FALSE, na.strings = "")
- 先讓我們執行「線性回歸」，我們使用年齡與心跳來預測鉀離子濃度：
subdat <- dat[!dat[,'K'] %in% NA & !dat[,'AGE'] %in% NA & !dat[,'Rate'] %in% NA, c('K', 'AGE', 'Rate')]
used_dat.x <- model.matrix(~ ., data = subdat[,-1])
X.array <- t(used_dat.x[,-1])
Y.array <- array(subdat[,1], dim = c(1, nrow(subdat)))
第一節：MxNet基本介紹(4)
- 現在讓我們來編寫一個Iterator，編寫一個標準Iterator的方法比較複雜，首先要編寫一個「core function」，寫法與R語言的自訂函數類似，但其輸出一定要是一個list包含「reset、iter.next、value、batch_size、batch」(名字最好完全相同避免後面出錯)：
my_iterator_core = function (batch_size) {
batch = 0
batch_per_epoch = length(Y.array)/batch_size
reset = function() {batch <<- 0}
iter.next = function() {
batch <<- batch+1
if (batch > batch_per_epoch) {return(FALSE)} else {return(TRUE)}
}
value = function() {
idx = 1:batch_size + (batch - 1) * batch_size
idx[idx > ncol(X.array)] = sample(1:ncol(X.array), sum(idx > ncol(X.array)))
data = mx.nd.array(X.array[,idx, drop=FALSE])
label = mx.nd.array(Y.array[,idx, drop=FALSE])
return(list(data = data, label = label))
}
return(list(reset = reset, iter.next = iter.next, value = value, batch_size = batch_size, batch = batch))
}
- 接著，我們要使用這個「core function」來做出一個新的「iterator function」，而這個「iterator function」的編寫方式與一般R語言的自訂函數不同，所以主要的變化建議都放在「core function」中：
my_iterator_func <- setRefClass("Custom_Iter",
fields = c("iter", "batch_size"),
contains = "Rcpp_MXArrayDataIter",
methods = list(
initialize = function(iter, batch_size = 100){
.self$iter <- my_iterator_core(batch_size = batch_size)
.self
},
value = function(){
.self$iter$value()
},
iter.next = function(){
.self$iter$iter.next()
},
reset = function(){
.self$iter$reset()
},
finalize=function(){
}
)
)
第一節：MxNet基本介紹(5)
- 現在我們的主要函數結構都寫完了，這時我們可以指定一個batch size：
my_iter = my_iterator_func(iter = NULL, batch_size = 20)
- 重置batch(切換到下一個epoch)：
- 跳到下一個batch：
## [1] TRUE
- 產生小批量樣本：
## $data
## [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
## [1,] 50.95044 66.22767 67.46526 52.39062 38.6022 83.70883 62.12934 60.54764
## [2,] 112.00000 76.00000 65.00000 93.00000 63.0000 71.00000 62.00000 65.00000
## [,9] [,10] [,11] [,12] [,13] [,14] [,15] [,16]
## [1,] 62.38272 55.70469 82.29976 67.57248 36.93145 52.65045 64.91228 64.37901
## [2,] 85.00000 66.00000 109.00000 55.00000 67.00000 65.00000 103.00000 96.00000
## [,17] [,18] [,19] [,20]
## [1,] 88.57705 83.13152 31.84526 81.80301
## [2,] 97.00000 153.00000 78.00000 60.00000
##
## $label
## [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12] [,13] [,14]
## [1,] 2.2 3.7 2.7 3.8 5.7 5.7 6 3 3.2 3.5 2.1 2.8 3.4 2.7
## [,15] [,16] [,17] [,18] [,19] [,20]
## [1,] 6.5 3 5.9 2.9 2.1 4.2
- 這個「my_iter」物件就是MxNet所需要的Iterator
第一節：MxNet基本介紹(6)
- 接著我們要定義我們的Model architecture，以線性迴歸為例，我們的預測函數如下：
– 函數「mx.symbol.Variable」用來產生一個新的模型起點，他通常是Iterator所輸出的名稱，有時候也可以用來指定權重的名稱
– 函數「mx.symbol.FullyConnected」代表著全連接層，這相當於之前MLP做線性運算的\(L\)函數，其中參數「num.hidden」意旨輸出維度
data = mx.symbol.Variable(name = 'data')
label = mx.symbol.Variable(name = 'label')
weight = mx.symbol.Variable(name = 'weight')
bias = mx.symbol.Variable(name = 'bias')
pred_y = mx.symbol.FullyConnected(data = data, weight = weight, bias = bias,
num.hidden = 1, no.bias = FALSE, name = 'pred_y')
- 這邊我建議可以隨時使用函數「mx.symbol.infer.shape」來確認目前的維度，讓後面容易接起來
mx.symbol.infer.shape(pred_y, data = c(2, 20))
## $arg.shapes
## $arg.shapes$data
## [1] 2 20
##
## $arg.shapes$weight
## [1] 2 1
##
## $arg.shapes$bias
## [1] 1
##
##
## $out.shapes
## $out.shapes$pred_y_output
## [1] 1 20
##
##
## $aux.shapes
## named list()
residual = mx.symbol.broadcast_minus(lhs = label, rhs = pred_y)
square_residual = mx.symbol.square(data = residual)
mean_square_residual = mx.symbol.mean(data = square_residual, axis = 0, keepdims = FALSE)
mse_loss = mx.symbol.MakeLoss(data = mean_square_residual, name = 'rmse')
mx.symbol.infer.shape(mse_loss, data = c(2, 20), label = c(1, 20))$out.shapes
## $rmse_output
## [1] 1
第一節：MxNet基本介紹(7)
- Optimizer的編寫非常簡單，目前我們已經學會了兩種優化方式：SGD、Adam，他們的編寫方式如下：
my_optimizer = mx.opt.create(name = "adam", learning.rate = 0.001, beta1 = 0.9, beta2 = 0.999,
epsilon = 1e-08, wd = 0)
my_optimizer = mx.opt.create(name = "sgd", learning.rate = 1e-4, momentum = 0.9, wd = 0)
第一節：MxNet基本介紹(8)
- 定義好上述3個東西後，我們將可以進行梯度下降求解的過程，我們先使用函數「mx.model.FeedForward.create」進行：
– 注意，這裡的num.round是指epoch的意思，而非batch。
mx.set.seed(0)
lr_model = mx.model.FeedForward.create(symbol = mse_loss, X = my_iter, optimizer = my_optimizer,
array.batch.size = 20, ctx = mx.cpu(), num.round = 200)
## $weight
## [,1]
## [1,] 0.05455168
## [2,] 0.01002080
##
## $bias
## [1] 2.180768
##
## Call:
## lm(formula = K ~ ., data = subdat)
##
## Coefficients:
## (Intercept) AGE Rate
## 2.461510 0.017666 0.002416
第一節：MxNet基本介紹(9)
my_optimizer = mx.opt.create(name = "sgd", learning.rate = 0.05, momentum = 0.9, wd = 0)
mx.set.seed(0)
lr_model = mx.model.FeedForward.create(symbol = mse_loss, X = my_iter, optimizer = my_optimizer,
array.batch.size = 20, ctx = mx.cpu(), num.round = 10)
lr_model$arg.params
## $weight
## [,1]
## [1,] NaN
## [2,] NaN
##
## $bias
## [1] NaN
- 為什麼會這樣呢？原因是我們發生了「梯度爆炸」！注意線性回歸的導函數，裡面需要乘上\(x\)，而\(x\)的數值在有很大的落差的狀況下，他的梯度會變的非常大。
\[
\begin{align}
\frac{\partial}{\partial b_0}loss & = \frac{1}{n} \sum \limits_{i=1}^{n}\left( \hat{y_{i}} - y_{i} \right) \\
\frac{\partial}{\partial b_1}loss & = \frac{1}{n} \sum \limits_{i=1}^{n}\left( \hat{y_{i}} - y_{i} \right) \cdot x_{i}
\end{align}
\]
第一節：MxNet基本介紹(10)
- 解決方法也滿簡單的，先對\(x\)進行標準化操作即可，這樣就能「快速」的用標準參數取得結果，也不會遇到「梯度爆炸」問題：
X.array <- t(scale(t(X.array)))
mean.X <- attr(X.array, "scaled:center")
sd.X <- attr(X.array, "scaled:scale")
my_optimizer = mx.opt.create(name = "sgd", learning.rate = 0.05, momentum = 0.9, wd = 0)
mx.set.seed(0)
lr_model = mx.model.FeedForward.create(symbol = mse_loss, X = my_iter, optimizer = my_optimizer,
array.batch.size = 20, ctx = mx.cpu(), num.round = 20)
lr_model$arg.params
## $weight
## [,1]
## [1,] 0.7609081
## [2,] 0.1595915
##
## $bias
## [1] 3.993352
subdat[,'AGE'] <- (subdat[,'AGE'] - mean(subdat[,'AGE'])) / sd(subdat[,'AGE'])
subdat[,'Rate'] <- (subdat[,'Rate'] - mean(subdat[,'Rate'])) / sd(subdat[,'Rate'])
lm(K ~ ., data = subdat)
##
## Call:
## lm(formula = K ~ ., data = subdat)
##
## Coefficients:
## (Intercept) AGE Rate
## 3.7810 0.3304 0.0571
練習1：請試著利用MxNet做出邏輯斯迴歸
- 目前為止我們已經了解到使用MxNet要依序編寫Iterator、Model architecture、Optimizer等三個部分，而之後就可以使用函數「mx.model.FeedForward.create」進行參數求解。
– 讓我們直接標準化，以方便比較結果。
subdat <- dat[!dat[,'LVD'] %in% NA & !dat[,'AGE'] %in% NA & !dat[,'Rate'] %in% NA, c('LVD', 'AGE', 'Rate')]
subdat[,-1] <- scale(subdat[,-1])
used_dat.x <- model.matrix(~ ., data = subdat[,-1])
X.array <- t(used_dat.x[,-1])
Y.array <- array(subdat[,1], dim = c(1, nrow(subdat)))
– 這是函數「glm」的運算結果：
glm(LVD ~ ., data = subdat, family = 'binomial')
##
## Call: glm(formula = LVD ~ ., family = "binomial", data = subdat)
##
## Coefficients:
## (Intercept) AGE Rate
## -0.20658 0.02904 0.55717
##
## Degrees of Freedom: 2111 Total (i.e. Null); 2109 Residual
## Null Deviance: 2907
## Residual Deviance: 2763 AIC: 2769
- 需要注意的是，函數「mx.symbol.XXXXX」有一系列的運算子，請你組合出一個邏輯斯迴歸！
\[
\begin{align}
log(\frac{{p}}{1-p}) & = b_{0} + b_{1}x \\
p & = \frac{{1}}{1+e^{-b_{0} - b_{1}x}} \\
loss & = CE(y, p) = -\frac{{1}}{n}\sum \limits_{i=1}^{n} \left(y_{i} \cdot log(p_{i}) + (1-y_{i}) \cdot log(1-p_{i})\right)
\end{align}
\]
練習1答案(1)
- 這是Iterator的部分，跟剛剛是完全一樣的，但我們可以選擇比較大的Batch size：
my_iterator_core = function(batch_size) {
batch = 0
batch_per_epoch = length(Y.array)/batch_size
reset = function() {batch <<- 0}
iter.next = function() {
batch <<- batch+1
if (batch > batch_per_epoch) {return(FALSE)} else {return(TRUE)}
}
value = function() {
idx = 1:batch_size + (batch - 1) * batch_size
idx[idx > ncol(X.array)] = sample(1:ncol(X.array), sum(idx > ncol(X.array)))
data = mx.nd.array(X.array[,idx, drop=FALSE])
label = mx.nd.array(Y.array[,idx, drop=FALSE])
return(list(data = data, label = label))
}
return(list(reset = reset, iter.next = iter.next, value = value, batch_size = batch_size, batch = batch))
}
my_iterator_func <- setRefClass("Custom_Iter",
fields = c("iter", "batch_size"),
contains = "Rcpp_MXArrayDataIter",
methods = list(
initialize = function(iter, batch_size = 100){
.self$iter <- my_iterator_core(batch_size = batch_size)
.self
},
value = function(){
.self$iter$value()
},
iter.next = function(){
.self$iter$iter.next()
},
reset = function(){
.self$iter$reset()
},
finalize=function(){
}
)
)
my_iter = my_iterator_func(iter = NULL, batch_size = 100)
練習1答案(2)
- 這是Model architecture的部分，需要修正成邏輯斯回歸的樣子：
data = mx.symbol.Variable(name = 'data')
label = mx.symbol.Variable(name = 'label')
linear_pred = mx.symbol.FullyConnected(data = data,
num.hidden = 1, no.bias = FALSE, name = 'linear_pred')
logistic_pred = mx.symbol.sigmoid(data = linear_pred, name = 'logistic_pred')
eps = 1e-8
ce_loss_pos = mx.symbol.broadcast_mul(mx.symbol.log(logistic_pred + eps), label)
ce_loss_neg = mx.symbol.broadcast_mul(mx.symbol.log(1 - logistic_pred + eps), 1 - label)
ce_loss_mean = 0 - mx.symbol.mean(ce_loss_pos + ce_loss_neg)
ce_loss = mx.symbol.MakeLoss(ce_loss_mean, name = 'ce_loss')
my_optimizer = mx.opt.create(name = "sgd", learning.rate = 0.05, momentum = 0.9, wd = 0)
mx.set.seed(0)
logistic_model = mx.model.FeedForward.create(symbol = ce_loss, X = my_iter, optimizer = my_optimizer,
array.batch.size = 20, ctx = mx.cpu(), num.round = 20)
logistic_model$arg.params
## $linear_pred_weight
## [,1]
## [1,] 0.01686738
## [2,] 0.55081975
##
## $linear_pred_bias
## [1] -0.1971277
第二節：MxNet的底層執行器(1)
- 在剛剛的章節中，我們很偷懶的使用了函數「mx.model.FeedForward.create」讓我們快速的獲得答案。然而這是一個上層函數，我們有好多願望並沒有辦法透過這個函數實現(也很難對其修正)，因此讓我們來學習一下該如何利用MxNet的底層執行器來進行Forward/Backward程序：
– 在前面的練習題中，我們已經寫好了邏輯斯迴歸的Iterator、Model architecture、Optimizer等三個部分，現在讓我們再編寫結合這三者的Executor。
- 首先，我們先使用函數「mx.simple.bind」定義主結構，這個函數會產生一個Executor：
my_executor = mx.simple.bind(symbol = ce_loss,
data = c(2, 100), label = c(1, 100),
ctx = mx.cpu(), grad.req = "write")
- 接著，我們要給權重參數一個起始值，一般來說是用亂數決定的，在MxNet中方法如下：
mx.set.seed(0)
new_arg = mxnet:::mx.model.init.params(symbol = ce_loss,
input.shape = list(data = c(2, 100), label = c(1, 100)),
output.shape = NULL,
initializer = mxnet:::mx.init.uniform(0.01),
ctx = mx.cpu())
mx.exec.update.arg.arrays(my_executor, new_arg$arg.params, match.name = TRUE)
mx.exec.update.aux.arrays(my_executor, new_arg$aux.params, match.name = TRUE)
- 第三步，我們要告訴我們的Optimizer他所要更新的目標，在這裡我們產生一個updater物件：
my_updater = mx.opt.get.updater(optimizer = my_optimizer, weights = my_executor$ref.arg.arrays)
第二節：MxNet的底層執行器(2)
- 接著就是利用Iterator產生Data並持續更新權重的時刻了，我們需要使用到「while」迴圈函數。
– 「my_iter$iter.next()」會持續回答TRUE直到一個epoch結束，而每個epoch的
my_iter$reset()
while (my_iter$iter.next()) {
my_values <- my_iter$value()
mx.exec.update.arg.arrays(my_executor, arg.arrays = my_values, match.name = TRUE)
mx.exec.forward(my_executor, is.train = TRUE)
mx.exec.backward(my_executor)
update_args = my_updater(weight = my_executor$ref.arg.arrays, grad = my_executor$ref.grad.arrays)
mx.exec.update.arg.arrays(my_executor, update_args, skip.null = TRUE)
}
- 一般來說，更新會持續N個epoch，我們可以利用「for」迴圈在「while」迴圈之外，並且在「while」迴圈結束後輸出一些資訊：
set.seed(0)
for (i in 1:20) {
my_iter$reset()
batch_loss = NULL
while (my_iter$iter.next()) {
my_values <- my_iter$value()
mx.exec.update.arg.arrays(my_executor, arg.arrays = my_values, match.name = TRUE)
mx.exec.forward(my_executor, is.train = TRUE)
mx.exec.backward(my_executor)
update_args = my_updater(weight = my_executor$ref.arg.arrays, grad = my_executor$ref.grad.arrays)
mx.exec.update.arg.arrays(my_executor, update_args, skip.null = TRUE)
batch_loss = c(batch_loss, as.array(my_executor$ref.outputs$ce_loss_output))
}
if (i %% 10 == 0 | i < 10) {
cat(paste0("epoch = ", i,
": Cross-Entropy (CE) = ", formatC(mean(batch_loss), format = "f", 4),
"; weight = ", paste(formatC(as.array(my_executor$ref.arg.arrays$linear_pred_weight), format = "f", 3), collapse = ", "),
"; bias = ", formatC(as.array(my_executor$ref.arg.arrays$linear_pred_bias), format = "f", 3), "\n"))
}
}
## epoch = 1: Cross-Entropy (CE) = 0.6556; weight = 0.013, 0.619; bias = -0.219
## epoch = 2: Cross-Entropy (CE) = 0.6544; weight = 0.017, 0.541; bias = -0.191
## epoch = 3: Cross-Entropy (CE) = 0.6548; weight = 0.017, 0.545; bias = -0.197
## epoch = 4: Cross-Entropy (CE) = 0.6547; weight = 0.017, 0.553; bias = -0.198
## epoch = 5: Cross-Entropy (CE) = 0.6547; weight = 0.017, 0.551; bias = -0.197
## epoch = 6: Cross-Entropy (CE) = 0.6547; weight = 0.017, 0.551; bias = -0.197
## epoch = 7: Cross-Entropy (CE) = 0.6547; weight = 0.017, 0.551; bias = -0.197
## epoch = 8: Cross-Entropy (CE) = 0.6547; weight = 0.017, 0.551; bias = -0.197
## epoch = 9: Cross-Entropy (CE) = 0.6547; weight = 0.017, 0.551; bias = -0.197
## epoch = 10: Cross-Entropy (CE) = 0.6547; weight = 0.017, 0.551; bias = -0.197
## epoch = 20: Cross-Entropy (CE) = 0.6547; weight = 0.017, 0.551; bias = -0.197
- 由於所有步驟都已經拆開了，所以你可以試著在權重更新的時候輸出/保留更多資訊！
第二節：MxNet的底層執行器(3)
- 有了權重之後有兩件事情我們想要做(存檔/預測)，那我們要先把Model整合出來：
logistic_model <- mxnet:::mx.model.extract.model(symbol = logistic_pred,
train.execs = list(my_executor))
- 有了這個Model後我們將可以用來儲存/讀取檔案，
mx.model.save(logistic_model, "logistic_regression", iteration = 0)
logistic_model = mx.model.load("logistic_regression", iteration = 0)
- 接著有一個簡單的方法來使用這個模型，可以直接使用內建的predict函數執行：
predict_Y = predict(logistic_model, X.array, array.layout = "colmajor")
confusion_table = table(predict_Y > 0.5, Y.array)
print(confusion_table)
## Y.array
## 0 1
## FALSE 883 545
## TRUE 279 405
第二節：MxNet的底層執行器(4)
- 儘管內建的predict函數很方便，但他能做的事情很有限。舉例來說假設我同時想要輸出邏輯斯迴歸的機率輸出以及線性輸出值，那你就得自己從底層寫起。
– MxNet的底層執行器來既然能進行Forward/Backward程序，當然也可以只進行Forward程序。
– 需要注意的是，假設你的驗證樣本很大，那你可能需要用迴圈慢慢丟樣本進行推論，不然會把記憶體塞爆。
all_layers = logistic_model$symbol$get.internals()
linear_pred_output = all_layers$get.output(which(all_layers$outputs == 'linear_pred_output'))
symbol_group = mx.symbol.Group(c(linear_pred_output, logistic_model$symbol))
Pred_executor = mx.simple.bind(symbol = symbol_group, data = c(2, ncol(X.array)), ctx = mx.cpu())
– 執行器已經寫完了，剩下的程序差不多，只有一些細節上的差異：
mx.exec.update.arg.arrays(Pred_executor, logistic_model$arg.params, match.name = TRUE)
mx.exec.update.aux.arrays(Pred_executor, logistic_model$aux.params, match.name = TRUE)
mx.exec.update.arg.arrays(Pred_executor, list(data = mx.nd.array(X.array)), match.name = TRUE)
mx.exec.forward(Pred_executor, is.train = FALSE)
PRED_linear_pred = as.array(Pred_executor$ref.outputs$linear_pred_output)
PRED_logistic_pred = as.array(Pred_executor$ref.outputs$logistic_pred_output)
confusion_table = table(PRED_logistic_pred > 0.5, Y.array)
print(confusion_table)
## Y.array
## 0 1
## FALSE 883 545
## TRUE 279 405
練習2：使用底層執行器進行多分類任務
- 現在讓我們來預測AMI，並且將樣本分成「訓練組」、「驗證組」及「測試組」。
– 需要特別注意的是，我們需要對Y做特殊的編碼方式，這個之前我們已經學過非常多次了，請你再熟悉一下：
subdat <- dat[!dat[,'AMI'] %in% NA & !dat[,'AGE'] %in% NA & !dat[,'Rate'] %in% NA, c('AMI', 'AGE', 'Rate')]
subdat[,-1] <- scale(subdat[,-1])
used_dat.x <- model.matrix(~ ., data = subdat[,-1])
set.seed(0)
all_idx <- 1:nrow(subdat)
train_idx <- sample(all_idx, nrow(subdat) * 0.6)
valid_idx <- sample(all_idx[!all_idx %in% train_idx], nrow(subdat) * 0.2)
test_idx <- all_idx[!all_idx %in% c(train_idx, valid_idx)]
X_mat <- t(used_dat.x[,-1])
Y_mat <- array(t(model.matrix(~ -1 + subdat[,1])), dim = c(3, nrow(subdat)))
train_X <- X_mat[,train_idx]
valid_X <- X_mat[,valid_idx]
test_X <- X_mat[,test_idx]
train_Y <- Y_mat[,train_idx]
valid_Y <- Y_mat[,valid_idx]
test_Y <- Y_mat[,test_idx]
- 你可能忘記式子了，這是\(mlogloss\)的函數：
\[
\begin{align}
mlogloss(y_{i}, p_{i}) & = - \sum \limits_{i=1}^{m} y_{i} log(p_{i})
\end{align}
\]
練習2答案(1)
- 跟練習1的時候一樣，我們先把底層所需要的3大元素寫好：
– Iterator的部分，要特別注意這裡要改成train_X：
my_iterator_core = function(batch_size) {
batch = 0
batch_per_epoch = ncol(train_X)/batch_size
reset = function() {batch <<- 0}
iter.next = function() {
batch <<- batch+1
if (batch > batch_per_epoch) {return(FALSE)} else {return(TRUE)}
}
value = function() {
idx = 1:batch_size + (batch - 1) * batch_size
idx[idx > ncol(train_Y)] = sample(1:ncol(train_Y), sum(idx > ncol(train_Y)))
data = mx.nd.array(train_X[,idx, drop=FALSE])
label = mx.nd.array(train_Y[,idx, drop=FALSE])
return(list(data = data, label = label))
}
return(list(reset = reset, iter.next = iter.next, value = value, batch_size = batch_size, batch = batch))
}
my_iterator_func <- setRefClass("Custom_Iter",
fields = c("iter", "batch_size"),
contains = "Rcpp_MXArrayDataIter",
methods = list(
initialize = function(iter, batch_size = 100){
.self$iter <- my_iterator_core(batch_size = batch_size)
.self
},
value = function(){
.self$iter$value()
},
iter.next = function(){
.self$iter$iter.next()
},
reset = function(){
.self$iter$reset()
},
finalize=function(){
}
)
)
my_iter = my_iterator_func(iter = NULL, batch_size = 20)
練習2答案(2)
- Model architecture的部分就是關鍵了，我們要寫出softmax與m-logloss：
data = mx.symbol.Variable(name = 'data')
label = mx.symbol.Variable(name = 'label')
linear_pred = mx.symbol.FullyConnected(data = data, num.hidden = 3, name = 'linear_pred')
softmax_layer = mx.symbol.softmax(data = linear_pred, axis = 1, name = 'softmax_layer')
eps = 1e-8
m_log = 0 - mx.symbol.mean(mx.symbol.broadcast_mul(mx.symbol.log(softmax_layer + eps), label))
m_logloss = mx.symbol.MakeLoss(m_log, name = 'm_logloss')
my_optimizer = mx.opt.create(name = "sgd", learning.rate = 0.05, momentum = 0.9, wd = 0)
練習2答案(3)
#1. Build an executor to train model
my_executor = mx.simple.bind(symbol = m_logloss,
data = c(2, 20), label = c(3, 20),
ctx = mx.cpu(), grad.req = "write")
#2. Set the initial parameters
mx.set.seed(0)
new_arg = mxnet:::mx.model.init.params(symbol = m_logloss,
input.shape = list(data = c(2, 20), label = c(3, 20)),
output.shape = NULL,
initializer = mxnet:::mx.init.uniform(0.01),
ctx = mx.cpu())
mx.exec.update.arg.arrays(my_executor, new_arg$arg.params, match.name = TRUE)
mx.exec.update.aux.arrays(my_executor, new_arg$aux.params, match.name = TRUE)
#3. Define the updater
my_updater = mx.opt.get.updater(optimizer = my_optimizer, weights = my_executor$ref.arg.arrays)
set.seed(0)
for (i in 1:20) {
my_iter$reset()
batch_loss = NULL
while (my_iter$iter.next()) {
my_values <- my_iter$value()
mx.exec.update.arg.arrays(my_executor, arg.arrays = my_values, match.name = TRUE)
mx.exec.forward(my_executor, is.train = TRUE)
mx.exec.backward(my_executor)
update_args = my_updater(weight = my_executor$ref.arg.arrays, grad = my_executor$ref.grad.arrays)
mx.exec.update.arg.arrays(my_executor, update_args, skip.null = TRUE)
batch_loss = c(batch_loss, as.array(my_executor$ref.outputs$m_logloss_output))
}
if (i %% 10 == 0 | i <= 5) {
cat(paste0("epoch = ", i, ": m-logloss = ", formatC(mean(batch_loss), format = "f", 4), "\n"))
}
}
## epoch = 1: m-logloss = 0.2864
## epoch = 2: m-logloss = 0.2531
## epoch = 3: m-logloss = 0.2528
## epoch = 4: m-logloss = 0.2527
## epoch = 5: m-logloss = 0.2527
## epoch = 10: m-logloss = 0.2526
## epoch = 20: m-logloss = 0.2526
練習2答案(4)
softmax_model <- mxnet:::mx.model.extract.model(symbol = softmax_layer,
train.execs = list(my_executor))
predict_Y = predict(softmax_model, valid_X, array.layout = "colmajor")
confusion_table = table(max.col(t(predict_Y)), max.col(t(valid_Y)))
print(confusion_table)
##
## 1 2 3
## 1 207 32 57
- 結果「非常不理想」，當然我們如果使用ROC曲線可以得到較好的結果：
library(pROC)
par(mfrow = c(1, 2))
roc_valid <- roc((max.col(t(valid_Y)) == 2) ~ predict_Y[2,])
plot(roc_valid, col = 'blue')
roc_valid <- roc((max.col(t(valid_Y)) == 1) ~ predict_Y[1,])
plot(roc_valid, col = 'blue')
第三節：MxNet的進階應用(1)
- 其實剛剛的問題就出在沒有「過度採樣」，上節課我們有教到我們在分析時應該要進行過度採樣，這樣才能獲得較好的結果。
– 要進行過度採樣就必須修正Iterator的部分：
my_iterator_core = function(batch_size) {
batch = 0
batch_per_epoch = ncol(train_X)/batch_size
reset = function() {batch <<- 0}
iter.next = function() {
batch <<- batch+1
if (batch > batch_per_epoch) {return(FALSE)} else {return(TRUE)}
}
value = function() {
idx.1 <- sample(which(max.col(t(train_Y)) == 1), floor(batch_size / 3))
idx.2 <- sample(which(max.col(t(train_Y)) == 2), floor(batch_size / 3))
idx.3 <- sample(which(max.col(t(train_Y)) == 3), batch_size - length(idx.1) - length(idx.2))
idx = c(idx.1, idx.2, idx.3)
data = mx.nd.array(train_X[,idx, drop=FALSE])
label = mx.nd.array(train_Y[,idx, drop=FALSE])
return(list(data = data, label = label))
}
return(list(reset = reset, iter.next = iter.next, value = value, batch_size = batch_size, batch = batch))
}
my_iterator_func <- setRefClass("Custom_Iter",
fields = c("iter", "batch_size"),
contains = "Rcpp_MXArrayDataIter",
methods = list(
initialize = function(iter, batch_size = 100){
.self$iter <- my_iterator_core(batch_size = batch_size)
.self
},
value = function(){
.self$iter$value()
},
iter.next = function(){
.self$iter$iter.next()
},
reset = function(){
.self$iter$reset()
},
finalize=function(){
}
)
)
my_iter = my_iterator_func(iter = NULL, batch_size = 20)
第三節：MxNet的進階應用(2)
#1. Build an executor to train model
my_executor = mx.simple.bind(symbol = m_logloss,
data = c(2, 20), label = c(3, 20),
ctx = mx.cpu(), grad.req = "write")
#2. Set the initial parameters
mx.set.seed(0)
new_arg = mxnet:::mx.model.init.params(symbol = m_logloss,
input.shape = list(data = c(2, 20), label = c(3, 20)),
output.shape = NULL,
initializer = mxnet:::mx.init.uniform(0.01),
ctx = mx.cpu())
mx.exec.update.arg.arrays(my_executor, new_arg$arg.params, match.name = TRUE)
mx.exec.update.aux.arrays(my_executor, new_arg$aux.params, match.name = TRUE)
#3. Define the updater
my_updater = mx.opt.get.updater(optimizer = my_optimizer, weights = my_executor$ref.arg.arrays)
#4. Forward/Backward
set.seed(0)
for (i in 1:20) {
my_iter$reset()
batch_loss = NULL
while (my_iter$iter.next()) {
my_values <- my_iter$value()
mx.exec.update.arg.arrays(my_executor, arg.arrays = my_values, match.name = TRUE)
mx.exec.forward(my_executor, is.train = TRUE)
mx.exec.backward(my_executor)
update_args = my_updater(weight = my_executor$ref.arg.arrays, grad = my_executor$ref.grad.arrays)
mx.exec.update.arg.arrays(my_executor, update_args, skip.null = TRUE)
batch_loss = c(batch_loss, as.array(my_executor$ref.outputs$m_logloss_output))
}
if (i %% 10 == 0 | i <= 5) {
cat(paste0("epoch = ", i, ": m-logloss = ", formatC(mean(batch_loss), format = "f", 4), "\n"))
}
}
## epoch = 1: m-logloss = 0.3618
## epoch = 2: m-logloss = 0.3575
## epoch = 3: m-logloss = 0.3593
## epoch = 4: m-logloss = 0.3611
## epoch = 5: m-logloss = 0.3600
## epoch = 10: m-logloss = 0.3569
## epoch = 20: m-logloss = 0.3579
softmax_model <- mxnet:::mx.model.extract.model(symbol = softmax_layer,
train.execs = list(my_executor))
predict_Y = predict(softmax_model, valid_X, array.layout = "colmajor")
confusion_table = table(max.col(t(predict_Y)), max.col(t(valid_Y)))
print(confusion_table)
##
## 1 2 3
## 1 52 3 4
## 2 24 4 6
## 3 131 25 47
第三節：MxNet的進階應用(3)
- 目前看起來MxNet似乎沒有節省多少我的時間，而事實上我們尚未展示出他的威力。利用開發框架最大的好處是在於我們只需要定義好架構，接著就交給程式協助獲得梯度及優化。
– 讓我們嘗試一下把Softmax regression改造成具有1個隱藏層的多層感知器，其實只要把Model architecture的部分作修正，其他部分完全不需要改：
data = mx.symbol.Variable(name = 'data')
label = mx.symbol.Variable(name = 'label')
fc1 = mx.symbol.FullyConnected(data = data, num.hidden = 10, name = 'fc1')
relu1 = mx.symbol.Activation(data = fc1, act.type = 'relu', name = 'relu1')
fc2 = mx.symbol.FullyConnected(data = relu1, num.hidden = 3, name = 'fc2')
softmax_layer = mx.symbol.softmax(data = fc2, axis = 1, name = 'softmax_layer')
eps = 1e-8
m_log = 0 - mx.symbol.mean(mx.symbol.broadcast_mul(mx.symbol.log(softmax_layer + eps), label))
m_logloss = mx.symbol.MakeLoss(m_log, name = 'm_logloss')
第三節：MxNet的進階應用(4)
- 接著我們只要再把底層執行器重新做一次就可以，完全不需要改到任何code：
#1. Build an executor to train model
my_executor = mx.simple.bind(symbol = m_logloss,
data = c(2, 20), label = c(3, 20),
ctx = mx.cpu(), grad.req = "write")
#2. Set the initial parameters
mx.set.seed(0)
new_arg = mxnet:::mx.model.init.params(symbol = m_logloss,
input.shape = list(data = c(2, 20), label = c(3, 20)),
output.shape = NULL,
initializer = mxnet:::mx.init.uniform(0.01),
ctx = mx.cpu())
mx.exec.update.arg.arrays(my_executor, new_arg$arg.params, match.name = TRUE)
mx.exec.update.aux.arrays(my_executor, new_arg$aux.params, match.name = TRUE)
#3. Define the updater
my_updater = mx.opt.get.updater(optimizer = my_optimizer, weights = my_executor$ref.arg.arrays)
#4. Forward/Backward
set.seed(0)
for (i in 1:20) {
my_iter$reset()
batch_loss = NULL
while (my_iter$iter.next()) {
my_values <- my_iter$value()
mx.exec.update.arg.arrays(my_executor, arg.arrays = my_values, match.name = TRUE)
mx.exec.forward(my_executor, is.train = TRUE)
mx.exec.backward(my_executor)
update_args = my_updater(weight = my_executor$ref.arg.arrays, grad = my_executor$ref.grad.arrays)
mx.exec.update.arg.arrays(my_executor, update_args, skip.null = TRUE)
batch_loss = c(batch_loss, as.array(my_executor$ref.outputs$m_logloss_output))
}
if (i %% 10 == 0 | i <= 5) {
cat(paste0("epoch = ", i, ": m-logloss = ", formatC(mean(batch_loss), format = "f", 4), "\n"))
}
}
## epoch = 1: m-logloss = 0.3639
## epoch = 2: m-logloss = 0.3630
## epoch = 3: m-logloss = 0.3630
## epoch = 4: m-logloss = 0.3629
## epoch = 5: m-logloss = 0.3629
## epoch = 10: m-logloss = 0.3605
## epoch = 20: m-logloss = 0.3454
softmax_model <- mxnet:::mx.model.extract.model(symbol = softmax_layer,
train.execs = list(my_executor))
predict_Y = predict(softmax_model, valid_X, array.layout = "colmajor")
confusion_table = table(max.col(t(predict_Y)), max.col(t(valid_Y)))
print(confusion_table)
##
## 1 2 3
## 1 88 7 8
## 2 2 0 2
## 3 117 25 47
第四節：製作存活分析的神經網路(1)
- 讓我們示範一下如何使用MxNet做出Cox proportional hazard model：
subdat <- dat[!dat[,'time'] %in% NA & !dat[,'death'] %in% NA & !dat[,'AGE'] %in% NA & !dat[,'Rate'] %in% NA, c('time', 'death', 'AGE', 'Rate')]
subdat[,-1:-2] <- scale(subdat[,-1:-2])
used_dat.x <- model.matrix(~ ., data = subdat[,-1:-2])
set.seed(0)
all_idx <- 1:nrow(subdat)
train_idx <- sample(all_idx, nrow(subdat) * 0.6)
valid_idx <- sample(all_idx[!all_idx %in% train_idx], nrow(subdat) * 0.2)
test_idx <- all_idx[!all_idx %in% c(train_idx, valid_idx)]
X_mat <- t(used_dat.x[,-1])
train_X <- X_mat[,train_idx]
valid_X <- X_mat[,valid_idx]
test_X <- X_mat[,test_idx]
train_Y <- subdat[train_idx,1:2]
valid_Y <- subdat[valid_idx,1:2]
test_Y <- subdat[test_idx,1:2]
library(survival)
coxph(Surv(time, death) ~ ., data = subdat[train_idx,])
## Call:
## coxph(formula = Surv(time, death) ~ ., data = subdat[train_idx,
## ])
##
## coef exp(coef) se(coef) z p
## AGE 0.68026 1.97439 0.07475 9.101 < 2e-16
## Rate 0.34593 1.41330 0.04715 7.336 2.2e-13
##
## Likelihood ratio test=144.2 on 2 df, p=< 2.2e-16
## n= 2800, number of events= 267
第四節：製作存活分析的神經網路(2)
survival_encode <- function (event, time) {
sub_dat <- data.frame(id = 1:length(event), event = event, time = time, stringsAsFactors = FALSE)
sub_dat <- sub_dat[order(sub_dat[,'event'], decreasing = TRUE),]
sub_dat <- sub_dat[order(sub_dat[,'time']),]
label <- array(0L, dim = rep(nrow(sub_dat), 2))
mask <- array(0L, dim = rep(nrow(sub_dat), 2))
diag(label) <- 1L
for (m in 1:nrow(sub_dat)) {
mask[m,m:nrow(sub_dat)] <- 1L
if (sub_dat[m,'event'] == 0) {
mask[m,] <- 0L
} else if (m > 1) {
zero_pos <- which(sub_dat[1:(m-1),'time'] == sub_dat[m,'time'])
if (length(zero_pos) > 0) {
mask[zero_pos,m] <- 0L
}
}
}
original_pos <- order(sub_dat[,'id'])
return(list(label = label[,original_pos,drop = FALSE], mask = mask[,original_pos,drop = FALSE]))
}
第四節：製作存活分析的神經網路(3)
my_iterator_core = function(batch_size) {
batch = 0
batch_per_epoch = ncol(train_X)/batch_size
reset = function() {batch <<- 0}
iter.next = function() {
batch <<- batch+1
if (batch > batch_per_epoch) {return(FALSE)} else {return(TRUE)}
}
value = function() {
idx = 1:batch_size + (batch - 1) * batch_size
idx[idx > ncol(train_X)] = sample(1:ncol(train_X), sum(idx > ncol(train_X)))
data = mx.nd.array(train_X[,idx, drop=FALSE])
label_list = survival_encode(event = train_Y[idx,'death'], time = train_Y[idx,'time'])
return(list(data = data, label = mx.nd.array(label_list$label), mask = mx.nd.array(label_list$mask)))
}
return(list(reset = reset, iter.next = iter.next, value = value, batch_size = batch_size, batch = batch))
}
my_iterator_func <- setRefClass("Custom_Iter",
fields = c("iter", "batch_size"),
contains = "Rcpp_MXArrayDataIter",
methods = list(
initialize = function(iter, batch_size = 100){
.self$iter <- my_iterator_core(batch_size = batch_size)
.self
},
value = function(){
.self$iter$value()
},
iter.next = function(){
.self$iter$iter.next()
},
reset = function(){
.self$iter$reset()
},
finalize=function(){
}
)
)
my_iter = my_iterator_func(iter = NULL, batch_size = 500)
第四節：製作存活分析的神經網路(4)
- 接著要注意這個結構的語法，我們要捨去截距，並重現它的損失函數：
\[
\begin{align}
loss & = \sum \limits_{\text{by column}} log( \sum \limits_{\text{by row}} M \otimes exp(H) ) - \sum \limits_{\text{by column}} log( \sum \limits_{\text{by row}} M \otimes Y \otimes exp(H) )
\end{align}
\]
data = mx.symbol.Variable(name = 'data')
final_pred = mx.symbol.FullyConnected(data = data, num.hidden = 1, no.bias = TRUE, name = 'final_pred')
COX_LOSS <- function (indata, inlabel, inmask, eps = 1e-8, make_loss = TRUE) {
exp_data <- mx.symbol.exp(data = indata, name = 'exp_data')
part_event_y <- mx.symbol.broadcast_mul(lhs = inlabel, rhs = inmask, name = 'part_event_y')
part_event <- mx.symbol.broadcast_mul(lhs = part_event_y, rhs = exp_data, name = 'part_event')
part_at_risk <- mx.symbol.broadcast_mul(lhs = inmask, rhs = exp_data, name = 'part_at_risk')
sum_part_event <- mx.symbol.sum(data = part_event, axis = 0, keepdims = TRUE, name = 'part_event')
sum_part_at_risk <- mx.symbol.sum(data = part_at_risk, axis = 0, keepdims = TRUE, name = 'sum_part_at_risk')
log_part_event <- mx.symbol.log(data = sum_part_event + eps, name = 'log_part_event')
log_part_at_risk <- mx.symbol.log(data = sum_part_at_risk + eps, name = 'log_part_at_risk')
diff_at_risk_event <- mx.symbol.broadcast_minus(lhs = log_part_at_risk, rhs = log_part_event, name = 'diff_at_risk_event')
important_time <- mx.symbol.sum(data = part_event_y, axis = 0, keepdims = TRUE, name = 'important_time')
number_time <- mx.symbol.sum(data = important_time, axis = 1, name = 'number_time')
event_loss <- mx.symbol.broadcast_mul(lhs = diff_at_risk_event, rhs = important_time, name = 'event_loss')
sum_event_loss <- mx.symbol.sum(data = event_loss, axis = 1, name = 'sum_event_loss')
final_loss <- mx.symbol.broadcast_div(lhs = sum_event_loss, rhs = number_time, name = 'final_loss')
if (make_loss) {final_loss <- mx.symbol.MakeLoss(data = final_loss, name = 'COX_loss')}
return(final_loss)
}
label_sym <- mx.symbol.Variable('label')
mask_sym <- mx.symbol.Variable('mask')
loss_symbol <- COX_LOSS(indata = final_pred, inlabel = label_sym, inmask = mask_sym)
第四節：製作存活分析的神經網路(5)
#1. Build an executor to train model
my_executor = mx.simple.bind(symbol = loss_symbol,
data = c(2, 500), label = c(500, 500), mask = c(500, 500),
ctx = mx.cpu(), grad.req = "write")
#2. Set the initial parameters
mx.set.seed(0)
new_arg = mxnet:::mx.model.init.params(symbol = loss_symbol,
input.shape = list(data = c(2, 500), label = c(500, 500), mask = c(500, 500)),
output.shape = NULL,
initializer = mxnet:::mx.init.uniform(0.01),
ctx = mx.cpu())
mx.exec.update.arg.arrays(my_executor, new_arg$arg.params, match.name = TRUE)
mx.exec.update.aux.arrays(my_executor, new_arg$aux.params, match.name = TRUE)
#3. Define the updater
my_optimizer = mx.opt.create(name = "sgd", learning.rate = 0.05, momentum = 0.9, wd = 0)
my_updater = mx.opt.get.updater(optimizer = my_optimizer, weights = my_executor$ref.arg.arrays)
#4. Forward/Backward
set.seed(0)
for (i in 1:20) {
my_iter$reset()
batch_loss = NULL
while (my_iter$iter.next()) {
my_values <- my_iter$value()
mx.exec.update.arg.arrays(my_executor, arg.arrays = my_values, match.name = TRUE)
mx.exec.forward(my_executor, is.train = TRUE)
mx.exec.backward(my_executor)
update_args = my_updater(weight = my_executor$ref.arg.arrays, grad = my_executor$ref.grad.arrays)
mx.exec.update.arg.arrays(my_executor, update_args, skip.null = TRUE)
batch_loss = c(batch_loss, as.array(my_executor$ref.outputs$COX_loss_output))
}
if (i %% 10 == 0 | i <= 5) {
cat(paste0("epoch = ", i, ": loss = ", formatC(mean(batch_loss), format = "f", 4), "\n"))
}
}
## epoch = 1: loss = 5.5123
## epoch = 2: loss = 5.3611
## epoch = 3: loss = 5.3609
## epoch = 4: loss = 5.3483
## epoch = 5: loss = 5.3471
## epoch = 10: loss = 5.3277
## epoch = 20: loss = 5.3253
my_executor$ref.arg.arrays$final_pred_weight
## [,1]
## [1,] 0.6448541
## [2,] 0.3448658
練習3：存活分析神經網路
- 讓我們來實現存活分析版的神經網路，並且將預測結果用於驗證組上。
– 這次我們使用比較小的Batch size，這樣可以加快很多速度：
my_iter = my_iterator_func(iter = NULL, batch_size = 50)
- 如果你有空的話，可以考慮在Iterator中使用oversampling，增加有死亡的個案出現的機率。
練習3答案(1)
data = mx.symbol.Variable(name = 'data')
fc1 = mx.symbol.FullyConnected(data = data, num.hidden = 10, name = 'fc1')
relu1 = mx.symbol.Activation(data = fc1, act.type = 'relu', name = 'relu1')
final_pred = mx.symbol.FullyConnected(data = relu1, num.hidden = 1, no.bias = TRUE, name = 'final_pred')
label_sym <- mx.symbol.Variable('label')
mask_sym <- mx.symbol.Variable('mask')
loss_symbol <- COX_LOSS(indata = final_pred, inlabel = label_sym, inmask = mask_sym)
- 使用底層執行器求解時只需要改掉Batch size：
#1. Build an executor to train model
my_executor = mx.simple.bind(symbol = loss_symbol,
data = c(2, 50), label = c(50, 50), mask = c(50, 50),
ctx = mx.cpu(), grad.req = "write")
#2. Set the initial parameters
mx.set.seed(0)
new_arg = mxnet:::mx.model.init.params(symbol = loss_symbol,
input.shape = list(data = c(2, 50), label = c(50, 50), mask = c(50, 50)),
output.shape = NULL,
initializer = mxnet:::mx.init.uniform(0.01),
ctx = mx.cpu())
mx.exec.update.arg.arrays(my_executor, new_arg$arg.params, match.name = TRUE)
mx.exec.update.aux.arrays(my_executor, new_arg$aux.params, match.name = TRUE)
#3. Define the updater
my_optimizer = mx.opt.create(name = "sgd", learning.rate = 0.05, momentum = 0.9, wd = 0)
my_updater = mx.opt.get.updater(optimizer = my_optimizer, weights = my_executor$ref.arg.arrays)
#4. Forward/Backward
set.seed(0)
for (i in 1:20) {
my_iter$reset()
batch_loss = NULL
while (my_iter$iter.next()) {
my_values <- my_iter$value()
mx.exec.update.arg.arrays(my_executor, arg.arrays = my_values, match.name = TRUE)
mx.exec.forward(my_executor, is.train = TRUE)
mx.exec.backward(my_executor)
update_args = my_updater(weight = my_executor$ref.arg.arrays, grad = my_executor$ref.grad.arrays)
mx.exec.update.arg.arrays(my_executor, update_args, skip.null = TRUE)
batch_loss = c(batch_loss, as.array(my_executor$ref.outputs$COX_loss_output))
}
if (i %% 10 == 0 | i <= 5) {
cat(paste0("epoch = ", i, ": loss = ", formatC(mean(batch_loss), format = "f", 4), "\n"))
}
}
## epoch = 1: loss = 3.2296
## epoch = 2: loss = 3.1440
## epoch = 3: loss = 3.2010
## epoch = 4: loss = 3.1345
## epoch = 5: loss = 3.1355
## epoch = 10: loss = 3.1025
## epoch = 20: loss = 3.1012
練習3答案(2)
– 需要注意，我們要去除mask，這樣才能進行預測：
COX_model <- mxnet:::mx.model.extract.model(symbol = final_pred,
train.execs = list(my_executor))
COX_model$arg.params <- COX_model$arg.params[!names(COX_model$arg.params) %in% 'mask']
predict_Y <- predict(COX_model, valid_X, array.layout = "colmajor")
valid_Y[,'score'] <- predict_Y[1,]
concordance(coxph(Surv(time, death) ~ score, data = valid_Y))[['concordance']]
## [1] 0.7538015
- 比起直接用線性模型預測的結果，神經網路的表現有些微的上升：
cox_fit <- coxph(Surv(time, death) ~ ., data = subdat[train_idx,])
valid_Y[,'score'] <- predict(cox_fit, subdat[valid_idx,])
concordance(coxph(Surv(time, death) ~ score, data = valid_Y))[['concordance']]
## [1] 0.7478008
結語
- 學習MxNet的過程是痛苦的，但學會底層執行器後，我們將有能力面對各式奇怪的需求。
– 舉例來說，疊代器可以協助我們在將資料丟去梯度下降法前，能進行許多前處理
– 底層執行器使用迴圈進行分析，能讓我們做細微的操作
- MxNet最重要的貢獻在於，我們再也不用求微分了，我們只要將模型結構定義好，配上它強大的框架即可運算出最終結果。
– 並且，我們終於可以調用gpu進行運算。
- 下節課我們將會針對「梯度消失問題」，進行完整的解答。