Having just recently started teaching myself Machine Learning and Data Analysis I'm finding myself hitting a brick wall on the need for creating and querying large sets of data. I would like to take data I've been aggregating in my professional and personal life and analyze it but I'm uncertain of the best way to do the following:
How should I be storing this data? Excel? SQL? ??
What is a good way for a beginner to begin trying to analyze this data? I am a professional computer programmer so the complexity is not in writing programs but more or less specific to the domain of data analysis.
EDIT: Apologies for my vagueness, when you first start learning about something it's hard to know what you don't know, ya know? ;)
Having said that, my aim is to apply this to two main topics:
Software team metrics (think Agile velocity, quantifying risk, likelihood of a successfully completed iteration given x number of story points)
Machine learning (ex. system exceptions have occurred in a given set of modules what is the likelihood that a module will throw an exception in the field, how much will that cost, what can the data tell me about key modules to improve that will get me the best bang for my buck, predict what portion of the system the user will want to use next in order to start loading data, etc).