2
$\begingroup$

I'm working on a project to predict the usage of all the files(rough frequency of usage) in a filesystem (a company server on which 100s of company employees are active) in near future (say the next 1 or 1.5 months) based on the metadata of the file system for past 6 months. I've got the following attributes about the files with me :

  1. The temporal sequence of file usage for last 6 months(whenever the file was read/written/modified and by whom).
  2. All the users who are on the server and can access the files.
  3. Last modified/written/read epoch time and by whom.
  4. File creation epoch time and by whom.
  5. Any compliance regulations on the file(whether the file contains any confidential data).
  6. Size, name, extension, version, type of the file.
  7. The number of users who can access the file.
  8. File path.
  9. The total number of times accessed.
  10. Permitted users.

Now, I plan to use LSTM but for standard LSTMs, the input is temporal sequence only. However, all the attributes that I have seem significant in predicting the future usage of the file.

  • How should I also make use of the attributes of the file that I have?
  • Should I train a Feedforward Neural Network, disregarding the fact that it usually fails on temporal sequences?
  • How should I proceed?
  • Does a variant of LSTM exist that can take into account the attributes of the file as well and predict the usage of the file in near future?
  • Do I need to use MLP and LSTM together like a hybrid?
$\endgroup$
  • $\begingroup$ I'm not an expert on this, by any means, but...seems to me like this problem isn't well defined enough. What dictates usage? Are you only trying to predict if a file will be used at all? If so, why do you need a neural net for this problem? What volume of traffic do you have on this file system? Why do all of your inputs seem significant? Also, LSTM is a NN - it's a version of a Recurrant Neural Network (RNN). If you're simply trying to encode all your inputs in a manner in which you can run them through a NN, you could one-hot encode these features as input vectors for your NN. That seems lik $\endgroup$ – infinitely_improbable Jun 29 '18 at 8:26
  • $\begingroup$ Welcome to CrossValidated. Can you explain exactly what you're trying to predict? What is precisely the output you want to forecast, and how far in advance? It's probably possible to answer your answer satisfactorily, but you need to be more clear, or it may be closed as "unclear". I suggest you take the Tour and look at how to write a good question. Also, please show an example of your input data (the first rows) and of the output you need. $\endgroup$ – DeltaIV Jun 29 '18 at 9:25
  • $\begingroup$ Hi @pg_gargleblaster, the answers to your queries in serial order are : 1. If not other features, at least the previous usage of a file which is a temporal sequence decides how frequently the file will be used in future. If a file is opened 10000 times in last one month and other one is used 2 times in last month, obviously the former has much higher probabliltiy of usage in near future. $\endgroup$ – Tushar Sinha Jul 2 '18 at 9:32
  • $\begingroup$ (continued.....) Also why am I saying that other features also seem significant is that suppose a file is permitted to be used by 10000 users on a company server and other is permitted to be used by only 10 executives of the company, chances are high that again the former file will be used more in the future. The number of permitted users might not be the only factor which determines the usage in future but if I have the previous usage of the file with me as well, then both coupled can give quite accurate frequency of usage in next month. continued..... $\endgroup$ – Tushar Sinha Jul 2 '18 at 9:36
  • $\begingroup$ (continued....) 2. I'm trying to predict the frequeny of file usage roughly in future. If not a neural net, how should I predict the frequency in near future ? Are you aware of any other machine learning algorithm which can solve this problem ? 3. The metadata that I have has been collected by a software by monitoring the file usage on a company server which is used by 100s of employees of the company. The .db files that I have is of 40 Gb which monitors usage in last 6 month. The number of entries are in millions. $\endgroup$ – Tushar Sinha Jul 2 '18 at 9:40
2
$\begingroup$

It seems like you could do this relatively easily using a model that has MLP and RNN parts.

Suppose that your time-series data is $A$. You compute some RNN output $r(A)$ that uses $A$ as the input; this can return a vector since you have, say, $i$ RNN units in the final layer.

Suppose that your "tabular" data is $B$. You compute some MLP that has output $m(B)$; this can return a vector since you have, say, $j$ nodes in the final layer of this MLP.

Now you have two vectors which, in some sense, encode the data contained in the tabular and time-series components of your data. You can concatenate these vectors to make a new vector of length $i +j=k$. This is the input to another fully-connected layer in your network, or possibly more than one. Then the output is just whatever your usual output is.

The reason that I think this could work is that you process the time-series and tabular pieces with models which are appropriate for their respective types, and then combine the results in a way which permits both formats to be used together.

But this could be hard to train. This would not be my first choice of a model. Instead, I would prefer to try using either a tabular or a time series model by itself, and making a determination of whether or not either simpler model is suitable.

Note that this structure trains all parts of the network, the MLP, the RNN and the "combiner" part, all at once. This does not require you to train three separate networks.

This is easy enough to do in most modern neural network software, such as Keras.

$\endgroup$
  • $\begingroup$ nice answer. Now tell me your trick: how did you find so quickly that the question had been reopened? :-) $\endgroup$ – DeltaIV Jul 2 '18 at 17:17
  • $\begingroup$ @DeltaIV no trick! I just saw it again at the top of the homepage. I’d thought about how to answer it yesterday when it was first posted but wasn’t in the mood to write an answer. $\endgroup$ – Sycorax Jul 2 '18 at 17:24
  • $\begingroup$ oh bummer :-( I hoped you had something like a bot or RSS feed I could reuse. Anyway, it may be of interest to you and the OP to know that last year Microsoft did build a model which is practically the same as what you proposed, though they purposefully explain themselves terribly (I guess the "highway-like structure" they mention a few times are skip connections) $\endgroup$ – DeltaIV Jul 2 '18 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.