Dynamic addition of hidden units in pytorch

python neural-network pytorch

404 观看


1 作者的声誉

I am trying to add hidden units to a 3-layered neural network(input, hidden,output) dynamically as I train it. I want to keep the weights of trained part of the network as I add new hidden units.This is my code,

class my_network(torch.nn.Module):
    def __init__(self,input_dim,hidden_dim,output_dim):
        self.I = input_dim
        self.H = hidden_dim
        self.O = output_dim
        self.layer1 = torch.nn.Linear(input_dim,hidden_dim)
        self.layer2 = torch.nn.Linear(hidden_dim,output_dim)

    def add_neurons(self,no_of_neurons,flag):
        if flag == 1:
            weights = [self.layer1.weight.data,self.layer2.weight.data]
            self.layer1 = torch.nn.Linear(self.I,self.H+no_of_neurons)
            self.layer2 = torch.nn.Linear(self.H+no_of_neurons,self.O)
            self.layer1.weight.data[0:-no_of_neurons,:] = weights[0]
            self.layer2.weight.data[:,0:-no_of_neurons] = weights[1]
            self.H = self.H + no_of_neurons
        return self.layer1.weight.shape[0]

    def forward(self,x):
        temp = self.layer1(x)
        out = self.layer2(temp)
        return out

I have noticed that once I call “add_neurons” method, the weights stop updating(while gradients are generated). Any help would be much appreciated.

作者: Sivaramakrishnan 的来源 发布者: 2017 年 12 月 27 日

回应 1


779 作者的声誉

The optimizer might not be informed about new parameters that you added to your model. The easiest probably would be to recreate optimizer object with the updated list of your model parameters.

作者: Egor Lakomkin 发布者: 2018 年 1 月 5 日