Originally this answer discussed alternate learning algorithms and topologies for Neural Nets. After the edit, the answer is divided into three parts:
- Uses and problems with Backpropogation;
- Alternate Neural Network Training Schemes; and
- Alternates to Neural Networks (newly added).
Backpropogation algorithms can help to solve problems where there is a discriminant that can help to separate the positive inputs from the negative inputs, often in networks that have no connections that loop between neurons in their topology (feed forward neural networks as described in your question).
Backpropogation is typically slow and it is not guaranteed to converge and may converge to a local maximum (that is, it will converge up to a point were you get results that are better than the results that can be obtained by changing the weights slightly).
Alternate neural network training algorithms include using a Genetic Algorithm to evolve the weights and simulated annealing. These may eliminate the problems related to slow convergence and problems with connections that loop between neurons.
Hebbian learning as used in recurrent neural networks, typically Hopfield networks, is another alternative. Hebbian learning is robust and Hopfield proved that Hebbian learning will converge for Hopfield networks.
You might also consider comparing your neural network to a modular neural network, such as, for example a Committee of Machines made up of several neural networks each with their own training algorithms or an Associative Neural Network.
Such modular neural networks will require more space and time to work with, but ultimately may perform better if their topologies are designed well for the particular problem at hand.
Popular alternatives to consider include Support Vector Machines (which are widely used in Fraud Analysis and Network Intrusion Detection amongst other areas).
Genetic Algorithms and Simulated Annealing (as stand-alone algorithms and not in conjunction with Neural Nets).
A final group of algorithms to consider are Swarm Algorithms such as Ant Colony Optimisation. Some algorithms in this group such as continuous orthogonal ant colony optimisation can be very powerful for solving some problems.
You might consider using some sort of clustering algorithms. Depending on your problems, K-Medoids or Density Based Clustering might be useful.
Depending on your problem, Machine Vision algorithms might also be useful. For example, the algorithm by Milanfar, http://users.soe.ucsc.edu/~milanfar/research/computer-vision.html, based on local regression kernels might also be useful.
The exact algorithms to use might depend on the exact problem that you are facing. Some algorithms will work really well in some instances but really badly in another. Personally I do not believe that there is a single algorithm that is perfect for solving everything (if there were we would be using it), so you would need to decide on the algorithm to benchmark against depending on the scenario in which you will be using it.