1
$\begingroup$

I am trying to understand a relationship between some x-cols and a y-col. There are about 25 features, some of which are categorical type. After a one-hot transformation on the categorical x-cols, the 25 features become about 220 features, so the input into the neural-network is a matrix of about 40,000 rows and about 220 columns. The y-col is made up of vals that are either 0 or 1, with about 33% of them being 1's and 67% of them being 0's. I am trying to train each row of 220 features to predict whether the y will be a 1 or a 0, so this is a binary classification problem (I think).

Next, I build the keras model, I basically follow this guide:

model = Sequential()
model.add(Dense(220, input_dim=220, init='uniform', activation='relu'   ))
model.add(Dense(440,                init='uniform', activation='relu'   ))
model.add(Dense(  1,                init='uniform', activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

I put my epoch outputs into a pandas dataframe and this is it looks like. Why isn't the val_acc changing over iterations? This is time-series data so perhaps I need to adjust the model somehow? Is it normal for acc and val_acc to stay constant like this?

More evidence that something is wonky is that I make one of the input columns have the same values as the output column. In theory, the network should figure out that there is 100% relationship here and accuracy should increase, but it doesn't. I get the output that I posted below.

         loss       acc  val_loss   val_acc
1    0.641897  0.660562  0.616328  0.699252
2    0.641627  0.660743  0.615329  0.699252
3    0.641354  0.660743  0.618324  0.699252
4    0.641223  0.660743  0.616247  0.699252
5    0.641076  0.660743  0.620305  0.699252
6    0.641105  0.660743  0.613614  0.699252
7    0.641044  0.660743  0.614318  0.699252
8    0.640883  0.660743  0.613871  0.699252
9    0.640926  0.660743  0.618207  0.699252
10   0.640876  0.660743  0.615451  0.699252
11   0.640769  0.660743  0.615736  0.699252
12   0.640890  0.660743  0.616329  0.699252
13   0.640873  0.660743  0.615402  0.699252
14   0.640806  0.660743  0.616542  0.699252
15   0.640837  0.660743  0.613222  0.699252
16   0.640832  0.660743  0.614901  0.699252
17   0.640713  0.660743  0.616706  0.699252
18   0.640722  0.660743  0.613610  0.699252
19   0.640759  0.660743  0.616327  0.699252
20   0.640799  0.660743  0.614725  0.699252
21   0.640564  0.660743  0.612566  0.699252
22   0.640701  0.660743  0.612777  0.699252
23   0.640713  0.660743  0.618819  0.699252
24   0.640741  0.660743  0.614375  0.699252
25   0.640654  0.660743  0.613798  0.699252
26   0.640677  0.660743  0.614137  0.699252
27   0.640721  0.660743  0.615741  0.699252
28   0.640700  0.660743  0.615467  0.699252
29   0.640670  0.660743  0.615612  0.699252
30   0.640671  0.660743  0.615430  0.699252
31   0.640695  0.660743  0.615495  0.699252
32   0.640648  0.660743  0.616068  0.699252
33   0.640641  0.660743  0.614230  0.699252
34   0.640589  0.660743  0.614386  0.699252
35   0.640664  0.660743  0.614640  0.699252
36   0.640638  0.660743  0.614352  0.699252
37   0.640608  0.660743  0.615433  0.699252
38   0.640635  0.660743  0.614641  0.699252
39   0.640609  0.660743  0.614228  0.699252
40   0.640618  0.660743  0.614252  0.699252
41   0.640609  0.660743  0.615942  0.699252
42   0.640605  0.660743  0.614666  0.699252
43   0.640630  0.660743  0.615378  0.699252
44   0.640577  0.660743  0.615495  0.699252
45   0.640598  0.660743  0.614793  0.699252
46   0.640588  0.660743  0.615685  0.699252
47   0.640600  0.660743  0.615619  0.699252
48   0.640594  0.660743  0.614669  0.699252
49   0.640558  0.660743  0.615861  0.699252
50   0.640597  0.660743  0.614103  0.699252
51   0.640591  0.660743  0.615275  0.699252
52   0.640615  0.660743  0.614739  0.699252
53   0.640596  0.660743  0.615854  0.699252
54   0.640599  0.660743  0.615322  0.699252
55   0.640590  0.660743  0.614515  0.699252
56   0.640600  0.660743  0.614964  0.699252
57   0.640573  0.660743  0.614601  0.699252
58   0.640578  0.660743  0.615135  0.699252
59   0.640594  0.660743  0.615760  0.699252
60   0.640559  0.660743  0.615548  0.699252
61   0.640580  0.660743  0.615393  0.699252
62   0.640610  0.660743  0.614791  0.699252
63   0.640578  0.660743  0.615710  0.699252
64   0.640575  0.660743  0.614258  0.699252
65   0.640564  0.660743  0.614232  0.699252
66   0.640597  0.660743  0.614600  0.699252
67   0.640577  0.660743  0.614371  0.699252
68   0.640568  0.660743  0.615669  0.699252
69   0.640608  0.660743  0.615111  0.699252
70   0.640514  0.660743  0.616816  0.699252
71   0.640623  0.660743  0.614618  0.699252
72   0.640600  0.660743  0.614868  0.699252
73   0.640589  0.660743  0.614961  0.699252
74   0.640598  0.660743  0.614839  0.699252
75   0.640608  0.660743  0.614778  0.699252
76   0.640520  0.660743  0.613288  0.699252
77   0.640610  0.660743  0.615440  0.699252
78   0.640593  0.660743  0.613966  0.699252
79   0.640613  0.660743  0.614656  0.699252
80   0.640590  0.660743  0.614947  0.699252
81   0.640566  0.660743  0.613891  0.699252
82   0.640603  0.660743  0.614637  0.699252
83   0.640601  0.660743  0.614522  0.699252
84   0.640567  0.660743  0.614251  0.699252
85   0.640591  0.660743  0.615210  0.699252
86   0.640596  0.660743  0.615250  0.699252
87   0.640593  0.660743  0.614932  0.699252
88   0.640598  0.660743  0.615213  0.699252
89   0.640596  0.660743  0.615220  0.699252
90   0.640595  0.660743  0.614594  0.699252
91   0.640599  0.660743  0.614776  0.699252
92   0.640604  0.660743  0.615087  0.699252
93   0.640603  0.660743  0.615350  0.699252
94   0.640601  0.660743  0.614367  0.699252
95   0.640596  0.660743  0.615314  0.699252
96   0.640580  0.660743  0.615060  0.699252
97   0.640610  0.660743  0.615011  0.699252
98   0.640586  0.660743  0.615272  0.699252
99   0.640595  0.660743  0.614625  0.699252
100  0.640552  0.660743  0.614021  0.699252
101  0.640578  0.660743  0.615544  0.699252
102  0.640543  0.660743  0.616162  0.699252
103  0.640605  0.660743  0.614085  0.699252
104  0.640589  0.660743  0.615278  0.699252
105  0.640579  0.660743  0.614811  0.699252
106  0.640582  0.660743  0.615760  0.699252
107  0.640613  0.660743  0.614557  0.699252
108  0.640588  0.660743  0.615556  0.699252
109  0.640600  0.660743  0.614804  0.699252
110  0.640560  0.660743  0.615136  0.699252
111  0.640572  0.660743  0.614165  0.699252
112  0.640608  0.660743  0.615297  0.699252
113  0.640605  0.660743  0.614746  0.699252
114  0.640548  0.660743  0.615037  0.699252
115  0.640591  0.660743  0.614994  0.699252
116  0.640593  0.660743  0.614662  0.699252
117  0.640590  0.660743  0.614965  0.699252
118  0.640599  0.660743  0.615216  0.699252
119  0.640598  0.660743  0.615281  0.699252
120  0.640612  0.660743  0.614845  0.699252
121  0.640602  0.660743  0.615103  0.699252
122  0.640594  0.660743  0.615304  0.699252
123  0.640565  0.660743  0.614503  0.699252
124  0.640602  0.660743  0.615098  0.699252
125  0.640545  0.660743  0.613554  0.699252
126  0.640613  0.660743  0.614375  0.699252
127  0.640586  0.660743  0.614412  0.699252
128  0.640606  0.660743  0.614437  0.699252
129  0.640550  0.660743  0.614693  0.699252
130  0.640594  0.660743  0.614426  0.699252
131  0.640597  0.660743  0.615494  0.699252
132  0.640613  0.660743  0.614848  0.699252
133  0.640596  0.660743  0.614007  0.699252
134  0.640611  0.660743  0.615254  0.699252
135  0.640587  0.660743  0.614867  0.699252
136  0.640599  0.660743  0.614976  0.699252
137  0.640593  0.660743  0.614860  0.699252
138  0.640527  0.660743  0.614237  0.699252
139  0.640590  0.660743  0.614180  0.699252
140  0.640488  0.660743  0.613225  0.699252
141  0.640601  0.660743  0.615506  0.699252
142  0.640592  0.660743  0.615472  0.699252
143  0.640603  0.660743  0.614683  0.699252
144  0.640587  0.660743  0.614166  0.699252
145  0.640608  0.660743  0.614693  0.699252
146  0.640587  0.660743  0.614949  0.699252
147  0.640578  0.660743  0.615914  0.699252
148  0.640600  0.660743  0.615456  0.699252
149  0.640593  0.660743  0.615424  0.699252
150  0.640594  0.660743  0.614525  0.699252
151  0.640608  0.660743  0.615037  0.699252
152  0.640569  0.660743  0.613943  0.699252
153  0.640544  0.660743  0.615305  0.699252
154  0.640597  0.660743  0.614683  0.699252
155  0.640600  0.660743  0.614528  0.699252
156  0.640587  0.660743  0.614327  0.699252
157  0.640577  0.660743  0.614235  0.699252
158  0.640608  0.660743  0.614815  0.699252
159  0.640587  0.660743  0.614087  0.699252
160  0.640571  0.660743  0.614239  0.699252
161  0.640588  0.660743  0.614552  0.699252
162  0.640577  0.660743  0.614276  0.699252
163  0.640582  0.660743  0.614621  0.699252
164  0.640567  0.660743  0.615725  0.699252
165  0.640598  0.660743  0.614764  0.699252
166  0.640605  0.660743  0.614616  0.699252
167  0.640586  0.660743  0.615543  0.699252
168  0.640596  0.660743  0.614852  0.699252
169  0.640600  0.660743  0.614687  0.699252
170  0.640607  0.660743  0.614892  0.699252
171  0.640586  0.660743  0.614304  0.699252
172  0.640586  0.660743  0.614679  0.699252
173  0.640566  0.660743  0.616016  0.699252
174  0.640606  0.660743  0.615532  0.699252
175  0.640580  0.660743  0.615221  0.699252
176  0.640595  0.660743  0.614695  0.699252
177  0.640557  0.660743  0.615860  0.699252
178  0.640607  0.660743  0.614379  0.699252
179  0.640590  0.660743  0.614920  0.699252
180  0.640592  0.660743  0.614774  0.699252
181  0.640599  0.660743  0.614851  0.699252
182  0.640597  0.660743  0.614785  0.699252
183  0.640583  0.660743  0.615477  0.699252
184  0.640579  0.660743  0.616139  0.699252
185  0.640620  0.660743  0.614970  0.699252
186  0.640586  0.660743  0.615476  0.699252
187  0.640579  0.660743  0.616319  0.699252
188  0.640565  0.660743  0.613746  0.699252
189  0.640576  0.660743  0.615620  0.699252
190  0.640601  0.660743  0.615063  0.699252
191  0.640603  0.660743  0.614868  0.699252
192  0.640603  0.660743  0.614383  0.699252
193  0.640576  0.660743  0.615464  0.699252
194  0.640611  0.660743  0.614949  0.699252
195  0.640597  0.660743  0.614733  0.699252
196  0.640604  0.660743  0.614800  0.699252
197  0.640567  0.660743  0.613821  0.699252
198  0.640604  0.660743  0.614515  0.699252
199  0.640587  0.660743  0.614351  0.699252
200  0.640608  0.660743  0.614602  0.699252
201  0.640598  0.660743  0.615445  0.699252
202  0.640600  0.660743  0.615279  0.699252
203  0.640592  0.660743  0.614800  0.699252
204  0.640595  0.660743  0.614606  0.699252
205  0.640578  0.660743  0.614136  0.699252
206  0.640580  0.660743  0.615714  0.699252
207  0.640600  0.660743  0.615466  0.699252
208  0.640601  0.660743  0.614984  0.699252
209  0.640572  0.660743  0.615882  0.699252
210  0.640582  0.660743  0.614297  0.699252
211  0.640598  0.660743  0.614838  0.699252
212  0.640578  0.660743  0.614480  0.699252
213  0.640593  0.660743  0.615357  0.699252
214  0.640594  0.660743  0.615248  0.699252
215  0.640577  0.660743  0.614219  0.699252
216  0.640593  0.660743  0.615383  0.699252
217  0.640554  0.660743  0.616238  0.699252
218  0.640611  0.660743  0.614867  0.699252
219  0.640586  0.660743  0.614663  0.699252
220  0.640580  0.660743  0.615853  0.699252
221  0.640593  0.660743  0.614540  0.699252
222  0.640608  0.660743  0.615199  0.699252
223  0.640592  0.660743  0.614742  0.699252
224  0.640574  0.660743  0.613912  0.699252
225  0.640602  0.660743  0.614859  0.699252
226  0.640587  0.660743  0.614411  0.699252
227  0.640599  0.660743  0.614181  0.699252
228  0.640586  0.660743  0.615121  0.699252
229  0.640571  0.660743  0.614320  0.699252
230  0.640572  0.660743  0.615430  0.699252
231  0.640581  0.660743  0.614000  0.699252
232  0.640601  0.660743  0.615105  0.699252
233  0.640594  0.660743  0.614728  0.699252
234  0.640527  0.660743  0.616488  0.699252
235  0.640617  0.660743  0.614829  0.699252
236  0.640585  0.660743  0.615214  0.699252
237  0.640600  0.660743  0.615036  0.699252
238  0.640559  0.660743  0.613934  0.699252
239  0.640591  0.660743  0.615416  0.699252
240  0.640605  0.660743  0.614492  0.699252
241  0.640576  0.660743  0.613956  0.699252
242  0.640586  0.660743  0.614424  0.699252
243  0.640597  0.660743  0.615138  0.699252
244  0.640588  0.660743  0.614859  0.699252
245  0.640605  0.660743  0.614486  0.699252
246  0.640593  0.660743  0.615658  0.699252
247  0.640597  0.660743  0.615462  0.699252
248  0.640575  0.660743  0.615596  0.699252
249  0.640585  0.660743  0.615352  0.699252
250  0.640554  0.660743  0.613717  0.699252
$\endgroup$
  • $\begingroup$ If you have 2 outputs : 0 and 1 , last network layer should be 2 nodes. $\endgroup$ – ᴀʀᴍᴀɴ Feb 1 '17 at 19:03
  • $\begingroup$ I only have 1 output, either a 1 or a 0. $\endgroup$ – user1367204 Feb 1 '17 at 19:25
  • $\begingroup$ This is not a 1 output, when you have two different outputs , so you have two different classes , so you must have one output node for each class. $\endgroup$ – ᴀʀᴍᴀɴ Feb 1 '17 at 19:27
  • $\begingroup$ I think that in Keras lingo, if you only have two labels, you can leave it as one column rather than massage it into two columns. But just to be sure I changed to number of nodes to two, and I got the same results as before. $\endgroup$ – user1367204 Feb 1 '17 at 20:36
  • 2
    $\begingroup$ For a binary output you can use either of the two. It should not make a difference. $\endgroup$ – Pieter Feb 2 '17 at 9:27
4
$\begingroup$

It seems that your model is not able to make sensible adjustments to your weights. The log loss is decreasing a tiny bit, but then gets stuck. It is just randomly guessing.

I think the root of the problem is that you have sparse positive inputs, positive initial weights and a ReLu activation. I suspect that this combination does not lead to nonzero weight adjustments (however, I do not have any literature background on this)

There are a few things that you could try:

  • Change the initialization to normal.
  • Use sigmoid layers everywhere.
  • Normalize your input, e.g. use StandardScaler from scikit learn.
  • Increase the initial learning rate and/or choose a different optimizer.
  • For debugging purposes, decrease the size of the hidden layer or even remove it.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.