First, your question is limited to propensity score matching, but there are other matching procedures that do not use propensity scores, such as exact matching, genetic matching, and coarsened exact matching. See Ho et al. 2007 for a review. These alternative methods are non-parametric and do not make linear assumptions. In fact, unless your propensity-score model is very good, there are theoretical arguments that suggest such methods are preferable.
Second, propensity scores are most generally the predictions from a model that estimates probabilities conditional on covariates. Nothing prevents you from using non-linear models, such as random forest, boosted regression trees, neural networks, etc. In fact, there is a package for R called
twang developed by the RAND Corporation that uses the
gbm package to optimize the tuning parameters of boosted regression trees with respect to user-specified sample-balance criteria, then output the predicted propensity scores for your sample using the balance-optimized model. You can also specify whether you are after an average treatment effect (ATE) on the population or on the treated (ATT). I wrote a silly little LinkedIn post that gives some brief pointers on how to use
twang-generated propensity scores as the distance metric for nearest-neighbor matching as implemented in the