Neural Network Predicting Live Market Data (fun project for BTC prediction)

Question

Made it just for fun - not for profit, wrote a neural network application that is predicting output from live data from exchange markets dealing with Bitcoin. Now just to clarify, i am not asking if my algo does it correclty or my model is going to make me rich - i am studying NN and live prediction, so please read it that way.

There are two sources (markets) from which i get real data. The data i am considering as input is obviously current buy price, and the network is trying to guess next price. However i don't care about timing here, i want to predict next possible price so i am not considering a buy price that has not changed as an input. I poll market every 100ms and ask for a current price, if price has changed then i store it, if price did not change i ignore it.

I am training the network by feeding in historical prices, around 2k for each market - network configured as follows:

INPUT:3 inputs HIDDEN: INPUT *2 +1 OUTPUT: 1

Training until error reaches 0.001 factor.

Now to the questions.

1) I am storing only values that change, so i dont save the price if it hasn't changed, therefore - is this approach ok? Or should i get the price even if it doesn't change? Does this affect the prediction? And how much? I don't want to predict a value at 15:00 i want the network to predict next possible buy price - time does not matter here.

2) If you look at the charts below, you can clearly see that the network is kind of 'lagged' (especially on the second screenshot) and it doesn't like 'high peaks' - what's even better, it can't even predict these it always predicts the opposite trend - is this something that is normal or there is some explanation for this behaviour?

Source code:

#include <chrono>
#include <thread> 
#include <math.h>
#include <iostream>
#include "Core/CMemTracer.h"
#include "Core/CDatabase.h"
#include "Core/CCalcModule.h"
#include "Core/CCalcModuleNN.h"
#include "Core/CNeuralNetwork.h"

CNeuralNetwork _NeuralNetwork;
CDatabase _Database;

int main(int argc, const char * argv[])
{
    std::string m_strDatabaseHost;
    std::string m_strDatabaseName;
    std::string m_strDatabaseUsername;
    std::string m_strDatabasePassword;
    std::string m_strExchange;

    int          m_iNumOfHistoryForTraining = 0;
    int         iNeuralNetworkInputs = 5;
    int         iNeuralNetworkHidden = 2 * iNeuralNetworkInputs + 1;
    int         iNeuralNetworkOutputs = 1;
    int         iMaximumTrainingEpoch = 10000000;
    float       fMinimum = 0;
    float       fMaximum = 1000;
    float       fMaximumNetworkError = 0.000720;
    float       fNeuralNetworkLearningRate = 0.5;
    float       fNeuralNetworkMomentum = 0.1;

    std::vector<float> vHistory;
    std::vector<float> vNormalisedData;

    m_strDatabaseHost       = "192.168.0.10";
    m_strDatabaseName       = "Trader";
    m_strDatabasePassword   = "password";
    m_strDatabaseUsername   = "root";
    m_strExchange           = "exBitMarket";

    // How much data we fetch from the DB
    m_iNumOfHistoryForTraining = 2000;

    CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Info, Connecting to Database");

    // Load up Database
    if(_Database.Connect(m_strDatabaseUsername, m_strDatabasePassword, m_strDatabaseHost) == false)
    {
        CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Error, cant connect to Database");
        return false;
    }

    CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Info, Selecting Database");

    // Select Database
    if(_Database.SelectDatabase(m_strDatabaseName) == false)
    {
        CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Error, cant select Database");
        return false;
    }

    // Get x Data from Database
    std::string strQuery = "SELECT * FROM (SELECT * FROM exData WHERE Exchange='"+m_strExchange+"' ORDER BY Epoch DESC LIMIT "+stringify(m_iNumOfHistoryForTraining)+")sub ORDER BY Epoch ASC";

    // Query DB
    CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Info, Querying database");

    CDatabase::tDatabaseQueryResult _QuerySelect;
    if(_Database.Query(strQuery, _QuerySelect) == false)
    {
        //
        CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Error, cannot query database");

        //
        return false;
    }

    //
    CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Info, Got %i results", _QuerySelect.m_iRows);

    // If Data available
    if(_QuerySelect.m_iRows >= m_iNumOfHistoryForTraining )
    {

        // Push back Buy value to Historical Data Vector
        for(int c = 0; c < _QuerySelect.m_vRows.size(); c++)
            vHistory.push_back(atof(_QuerySelect.m_vRows[c].m_vstrColumns[3].data()));


        vNormalisedData = vHistory;
    }
    else
    {
        //
        CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Error, not enough data returned (%i of %i required)", _QuerySelect.m_iRows,m_iNumOfHistoryForTraining);

        //
        return false;
    }

    //
    CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Info, Normalising data for Neural network input");

    // Normalise
    // Find max, min values from the dataset for later normalization
    std::vector<float>::iterator itMax = std::max_element(vHistory.begin(), vHistory.end(),[](const float& x, const float& y) {  return x < y; });
    std::vector<float>::iterator itMin = std::min_element(vHistory.begin(), vHistory.end(),[](const float& x, const float& y) {  return x < y; });

    // Store Min/Max
    fMinimum = itMin[0];
    fMaximum = itMax[0];

    //
    CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Info, Normalised data <%f, %f>", fMinimum, fMaximum);

    // Important - Neural Network has to be setup correctly for activation function
    // both this normalization and NN has to be setup the same way.
    // Log  sigmoid activation function (0,1)


    // logistic sigmoid function  [0, 1]
    for(int a = 0; a < vHistory.size(); a++)
        vNormalisedData[a] = (vHistory[a] - itMin[0]) / (itMax[0] - itMin[0]);

    //
    CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Info, Initializing neural network with the setup %i/%i/%i Learning Rate: %f, Momentum: %f",
                               iNeuralNetworkInputs,
                               iNeuralNetworkHidden,
                               iNeuralNetworkOutputs,
                               fNeuralNetworkLearningRate,
                               fNeuralNetworkMomentum);


    // Build the network with arguments passed
    _NeuralNetwork.Initialize(iNeuralNetworkInputs, iNeuralNetworkHidden, iNeuralNetworkOutputs);
    _NeuralNetwork.SetLearningRate(fNeuralNetworkLearningRate);
    _NeuralNetwork.SetMomentum(false, fNeuralNetworkMomentum);



    // Train
    double  dMaxError   = 100.0;
    double  dLastError  = 12345.0;
    int     iEpoch      = 0;
    int     iLastDump   = 0;
    int     iNumberOfDataForTraining =  (vNormalisedData.size() / 2) - iNeuralNetworkInputs + iNeuralNetworkOutputs;
    //
    CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Info, starting training with %i data out of %i", iNumberOfDataForTraining, vNormalisedData.size());

    // Perform training on the training data
    while ( (dMaxError > fMaximumNetworkError) && (iEpoch < iMaximumTrainingEpoch) )
    {
        //
        dMaxError = 0;

        // Now the input is normalized and ready for use perform the training
        // Use 1/2 of the Normalised Data for training purposes, the rest will be used to
        // Validate the network.
        for(int a = 0; a < iNumberOfDataForTraining; a++)
        {
            // Set Inputs
            for(int b = 0; b < iNeuralNetworkInputs; b++)
                _NeuralNetwork.SetInput(b, vNormalisedData[a+b]);

            // Set desired Output for the newest value
            _NeuralNetwork.SetDesiredOutput(0, vNormalisedData[a + iNeuralNetworkInputs]);

            // Feed data
            _NeuralNetwork.FeedForward();

            //
            dMaxError += _NeuralNetwork.CalculateError();

            // Backpropagate to learn
            _NeuralNetwork.BackPropagate();
        }

        // Divide by the number of total array size to get global network error
        dMaxError /= vNormalisedData.size();


        // Dump some stats now
        if(CUtils::GetEpoch() - iLastDump > 1)
        {
            CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Training Error Factor: %f / %f Epoch: %i", dMaxError, fMaximumNetworkError, iEpoch);
            iLastDump = CUtils::GetEpoch();
        }

        // Increment the epoch count
        iEpoch++;

        // Store last error for early-stop
        dLastError = dMaxError;
    }
    //
    CLogger::Instance()->Write(XLOGEVENT_LOCATION, "Info, starting validation with %i data", vNormalisedData.size() - iNumberOfDataForTraining);

    //
    dMaxError = 0;

    // Now check against 'Validation' Data
    for(int a = iNumberOfDataForTraining; a < vNormalisedData.size(); a++)
    {
        // Set Inputs
        for(int b = 0; b < iNeuralNetworkInputs; b++)
            _NeuralNetwork.SetInput(b, vNormalisedData[a+b]);

        // Set desired Output for the newest value
        _NeuralNetwork.SetDesiredOutput(0, vNormalisedData[a + iNeuralNetworkInputs]);

        // Feed data
        _NeuralNetwork.FeedForward();

        //
        dMaxError += _NeuralNetwork.CalculateError();
    }

    // Divide by the number of total array size to get global network error
    dMaxError /= vNormalisedData.size();

    CLogger::Instance()->Write(XLOGEVENT_LOCATION, "%i Network Trained, Error Factor on Validation data = %f",
                               CUtils::GetEpoch(),
                               dMaxError);


    // Save the network to an output filer

    return 0;
}

Not asking about the algo, just asking about the output from the network, does it happen like that is this normal, or does it look like the network is overfitted?

Update: Added updated code that reflects training on Training data and a Validation on validation data.

Answer 1

1) Yes, amount of time price stays the same matters. Primarily because it signifies the underlying market condition - sometimes price rapidly ascends or descends, sometimes stays roughly the same over time.

2) Network is lagging because there is no logical pattern that you are feeding to you network - there is no useful data it can leverage to predict. Market is somewhat predictable when patterns in candles are observed primarily because the observers of the patterns take actions that patterns predict - kind of a self fulfilling prophecy. So you are essentially dragging your prediction graph behind with your live data, not the other way around.

3) Books price is not the same as closing price. Books price is just how much people are willing to buy or sell for, but closing price is actually a deal that was made. I would not use books price as a data point as it adds another dimension - relationship between books price and actual price which can vary quite a bit.