I'm working on a prediction problem and struggling to find applicable resources (articles, tutorials, papers) that address this class of problem. I'm assuming the info is out there and I'd love to help myself instead of asking for help, but after many days of searching, nothing I've found (including traditional modeling tools like ARIMA and newer machine learning algorithms) seems applicable.
My problem domain is a bit unusual, so here's an analogous (but fake) setup from finance:
I care about predicting the price of Apple stock the day after earnings are announced, which happens quarterly. Microsoft announces earnings on the same day, and I believe these to be highly correlated. I also have stock figures from other companies, both in tech (Google, Amazon) and non-tech (less correlated, Tesla and Walmart). Finally, I have other data, like Steve Jobs health, which may also be relevant, but which is reported on a different interval.
As an added wrinkle, stocks come and go - the universe of things we know about is finite (thousands) but changes over time. The relationships between stocks also evolve - for example, Apple moves into the car business and is now more correlated with Ford than with Google.
Edit Here's another - perhaps simpler? - scenario: I want to predict one home's electricity demands. I know all of the comings and goings of the people inside of the house, and I know what all of the other houses are doing and where those people are. I also know what's happening in the environment - it's raining, it's summer, etc.
Right now, I know that it's winter, Sally, Fred, and Angela are in the house, Fred is sleeping, Sally and Angela are watching television, and the house next door has four people, all also watching television. I'd like to guess how much electricity Sally and Angela are using separately, and how much their house is using total.
If I had a single time series with a trend and noise, I could use ARIMA.
If my datasets had identical periodicity, I could use various methods to analyze covariance.
If all of the data were of the same type (e.g., price), I think I'd be able to use a Kalman filter, or perhaps kriging.
I can't think of a good analogy, but I could use machine learning if I was trying to "recognize" a pattern rather than predict a result.
I might be able to use a Bayesian belief/graphical network, but all of the literature I've found so far is impenetrable.
In short, I have a set of actors / entities / sensors measuring different things. Some actors always participate in events simultaneously, others are independent. I'd like to start with input that looks like this:
Actor1, timestamp1, event1, data Actor2, timestamp1, event1, data Actor3, timestamp3, event2, data Actor4, timestamp4, event3, data
If needed, I could provide best guesses for correlations or dependencies between actors, so long as the algorithm is able to update those relationships over time. I can also transform the data if there's a more useful input representation. Heck, I can state whether each actor's data is Gaussian, binomial, etc.
I want to be able to ask questions that look like:
Actor4, timestamp400, event400 Actor5, timestamp400, event400
That is, I know that actor4 and actor5 will co-occur when needing to predict, which may not always be the case.
And get output that looks like:
Actor 4: <predicted value or range>, <some measure of accuracy/confidence/belief> Actor 5: <predicted value or range>, <some measure of accuracy/confidence/belief>
This does not need to run in real time, and I don't mind rebuilding / retraining my model daily with any new events that have been reported. I'm also willing to build a separate model per agent (low thousands) if that will help. I don't know what I don't know, in terms of relationships between agents, so I'm looking for ways to throw computing power at the problem - here's my big bag of odds and ends I've amassed, most of it is probably related, how do I find out which stuff predicts the small amount of stuff I care about?
So, what kind of thing is this? It feels a little bit like lots of different problems, so I'm totally stumped on how to begin attacking the problem.