{"id":58,"date":"2018-08-07T22:02:31","date_gmt":"2018-08-07T21:02:31","guid":{"rendered":"http:\/\/www.science-now.org\/giampaolo\/?page_id=58"},"modified":"2018-08-25T13:14:10","modified_gmt":"2018-08-25T12:14:10","slug":"machine-learning-neural-network","status":"publish","type":"page","link":"https:\/\/www.science-now.org\/giampaolo\/machine-learning-neural-network\/","title":{"rendered":"Machine learning – Neural Network"},"content":{"rendered":"

Neural networks are very trendy nowadays and it could be a good idea to understand how they work. Luckily, we can rely on a Python library called
\nsklearn<\/code> and available here<\/a>.<\/p>\n

<\/p>\n

We’re going to use neural networks to interpolate between two sets of data. The training set is composed of two matrices, specifically an input<\/em> matrix and an output<\/em> matrix. The former one contains a set of inputs (one per row). The expected output is in the corresponding row of the output matrix. Both matrices are the training set of the neural network. <\/p>\n

The input\/output interpolation is performed with a small class which uses the sklearn<\/code> framework. The main objective is to have a N-dimensional interpolation tool. The complete implementation of the class is available at the end of the page but here two methods are described.<\/p>\n

\r\nclass NnInterpolator (object):\r\n\r\n    def __init__(self, input_matrix=None, output_matrix=None, calc_now=False):\r\n        """\r\n        input_matrix = nb x ni (number of data-points \\times number of input params)\r\n            point1: [par1, par2, par3 ...]\r\n            point2: [par1, par2, par3 ...]\r\n\r\n            [[par1, par2, par3 ...],\r\n             [par1, par2, par3 ...]]\r\n\r\n        output_matrix = nb x no (number of data-points \\times number of output vars)\r\n            point1: [var1, var2, var3 ...]\r\n            point2: [var1, var2, var3 ...]\r\n\r\n            [[var1, var2, var3 ...],\r\n             [var1, var2, var3 ...]]\r\n\r\n        calc_now = when to calculate the matrix inversion\r\n        """\r\n        ...\r\n\r\n    def __call__(self, new_input_data):\r\n        ...\r\n<\/pre>\n
The first one performs the class initialisation. It requires the input matrix and the output matrix. Each row of the matrices correspond to just one combination of parameters and output values. Once the initialisation is complete, we can use the interpolator by calling the object like a function. This trigger the __call__<\/code> method. The class is defined in the nnint.py<\/code> file.<\/p>\n
A simple example<\/h1>\n
Let’s teach a neural network how to calculate the square of a real number. We have two inputs (two numbers) and we want the neural network to calculate their squared values. Since we’re going to use Python with the numpy<\/code> library, we have to write<\/p>\n
\r\nimport numpy as np\r\n<\/pre>\n
and we’ll use also the NnInterpolator<\/code> class, so we have to import it<\/p>\n
\r\nfrom nnint import *\r\n<\/pre>\n
We then declare two variables which define the number of inputs and outputs<\/p>\n
\r\nn_inputs = 2\r\nn_outputs = 2\r\n<\/pre>\n
In addition, we need to define the size of the training set, i.e. the number of input\/output combinations:<\/p>\n
\r\nn_samples = 25000\r\n<\/pre>\n
At this point, we’re going to generate the training data. Specifically, we’re going to use a random list of numbers in the range \\([0, 1]\\) and another list containing their squared values. First, we initialise the random seed to generate random numbers with<\/p>\n
\r\nnp.random.seed()\r\n<\/pre>\n
and then the input and output matrices are obtained as<\/p>\n
\r\ninput_matrix = np.random.rand(n_samples, n_inputs)\r\noutput_matrix = input_matrix[:,0:n_outputs]**2\r\n<\/pre>\n
The interpolator object is instantiated by providing both matrices<\/p>\n
\r\ninterpolator = NnInterpolator(input_matrix, output_matrix, calc_now=True)\r\n<\/pre>\n
Now, let’s suppose we want to calculate the square of 0.25 and 0.5. This can be done with<\/p>\n
\r\ntest_in = np.array([[0.25, 0.5]])\r\ntest_out = interpolator(test_in)\r\n<\/pre>\n
Results can be printed on the screen with<\/p>\n
\r\nprint(test_in)\r\nprint(test_out)\r\n<\/pre>\n
The final script is:<\/p>\n
\r\nimport numpy as np\r\nfrom nnint import *\r\n\r\nn_inputs = 2\r\nn_outputs = 2\r\nn_samples = 2500\r\n\r\nnp.random.seed()\r\n\r\ninput_matrix = np.random.rand(n_samples, n_inputs)\r\noutput_matrix = input_matrix[:,0:n_outputs]**2\r\n\r\ninterpolator = NnInterpolator(input_matrix, output_matrix, calc_now=True)\r\n\r\ntest_in = np.array([[0.25, 0.5]])\r\ntest_out = interpolator(test_in)\r\n\r\nprint(test_in)\r\nprint(test_out)\r\n<\/pre>\n
Choosing the number of hidden layers and their size<\/h2>\n
No exact rule exists for choosing the number of hidden layers. A rule of thumb can be obtained from literature:<\/p>\n
    \n
  • Size of inner and outer layers are known because they depend on the number of inputs and outputs;<\/li>\n
  • Just one hidden layer is needed. A neural network with multiple hidden layers can be described as a neural network with just one hidden layer;<\/li>\n
  • The number of neurons of the hidden layer should be an average value between the size of inner and outer layers;<\/li>\n
  • The upper bound to not over-fit is
    \n$$
    \nN_h = \\frac{N_s}{\\alpha (N_i + N_o)}
    \n$$
    \nwhere \\(N_h\\) is the number of neurones of the hidden layer, \\(N_i\\) is the number of inputs, \\(N_o\\) is the number of outputs and \\(\\alpha \\in [2,10]\\). Regarding \\(\\alpha\\), it is a measure of the number of independent parameters in your data.<\/li>\n<\/ul>\n
    The NnInterpolator class<\/h1>\n
    These are the contents of the file nnint.py<\/code>.<\/p>\n
    \r\nimport numpy as np\r\n\r\nfrom sklearn.neural_network import MLPClassifier, MLPRegressor\r\nfrom sklearn.preprocessing import StandardScaler\r\n\r\nclass NnInterpolator (object):\r\n    """\r\n    Class implementing Neural Network interpolation\r\n    """\r\n\r\n    def __init__(self, input_matrix=None, output_matrix=None, calc_now=False):\r\n        """\r\n        input_matrix = nb x ni (number of data-points \\times number of input params)\r\n            point1: [par1, par2, par3 ...]\r\n            point2: [par1, par2, par3 ...]\r\n\r\n            [[par1, par2, par3 ...],\r\n             [par1, par2, par3 ...]]\r\n\r\n        output_matrix = nb x no (number of data-points \\times number of output vars)\r\n            point1: [var1, var2, var3 ...]\r\n            point2: [var1, var2, var3 ...]\r\n\r\n            [[var1, var2, var3 ...],\r\n             [var1, var2, var3 ...]]\r\n\r\n        calc_now = when to calculate the matrix inversion\r\n        """\r\n        self._nb = None\r\n        self._ni = None\r\n        self._no = None\r\n\r\n        self._input_matrix = input_matrix\r\n        self._output_matrix = output_matrix\r\n\r\n        if (input_matrix is not None) and (output_matrix is not None):\r\n            self._nb = input_matrix.shape[0]\r\n            self._ni = input_matrix.shape[1]\r\n\r\n            if self._nb != output_matrix.shape[0]:\r\n                raise ValueError("NB should be the same!")\r\n            self._no = output_matrix.shape[1]\r\n\r\n            if calc_now:\r\n                self._calc_interpolation()\r\n\r\n    def __call__(self, new_input_data):\r\n        if len(new_input_data.shape) > 1:\r\n            return self.interp_matrix(new_input_data)\r\n        else:\r\n            return self.interp_vector(new_input_data)\r\n\r\n    def _calc_interpolation(self):\r\n\r\n        self._scaler_in = StandardScaler()\r\n        self._scaler_in.fit(self._input_matrix)\r\n\r\n        self._scaler_out = StandardScaler()\r\n        self._scaler_out.fit(self._output_matrix)\r\n\r\n        self._clf = MLPRegressor(activation='logistic',\r\n                                 max_iter=1000,\r\n                                 tol=1e-30,\r\n                                 hidden_layer_sizes=(100,100,100))\r\n\r\n        self._clf.fit(self._scaler_in.transform(self._input_matrix),\r\n                      self._scaler_out.transform(self._output_matrix))\r\n\r\n    def interp_vector(self, unscaled_new_input_vector):\r\n        new_input_vector = self._scaler_in.transform(unscaled_new_input_vector.reshape(1,-1))\r\n        output = self._clf.predict(new_input_vector)\r\n        unscaled_output = self._scaler_out.inverse_transform(output)\r\n        return unscaled_output\r\n\r\n    def interp_matrix(self, input_matrix):\r\n        disp = np.zeros((input_matrix.shape[0], self._no))\r\n\r\n        for row in range(input_matrix.shape[0]):\r\n            disp[row, :] = self.interp_vector(input_matrix[row, :])\r\n\r\n        return disp\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
    Neural networks are very trendy nowadays and it could be a good idea to understand how they work. Luckily, we can rely on a Python library called sklearn and available here.<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-58","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.science-now.org\/giampaolo\/wp-json\/wp\/v2\/pages\/58","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.science-now.org\/giampaolo\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.science-now.org\/giampaolo\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.science-now.org\/giampaolo\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.science-now.org\/giampaolo\/wp-json\/wp\/v2\/comments?post=58"}],"version-history":[{"count":8,"href":"https:\/\/www.science-now.org\/giampaolo\/wp-json\/wp\/v2\/pages\/58\/revisions"}],"predecessor-version":[{"id":231,"href":"https:\/\/www.science-now.org\/giampaolo\/wp-json\/wp\/v2\/pages\/58\/revisions\/231"}],"wp:attachment":[{"href":"https:\/\/www.science-now.org\/giampaolo\/wp-json\/wp\/v2\/media?parent=58"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}