If you have censoring in your times to events then a Fisher test might not properly take the censoring into account. See this page for the importance of taking censoring into account.
A Fisher test, chi-square test, or other test of binomial event/no-event status might be OK if you have complete data on all individuals up to some particular follow-up time and restrict evaluation to a follow-up time for which you have complete data. It seems unlikely, however, that such a test would be more helpful than your attempt at a log-rank test, as both depend fundamentally on the numbers of events in each group. The reason is that you can think of the log-rank test as an extension of the chi-square test.
In the simple chi-square test you have in mind, the observed numbers of "events/non-events" in each group at some point in time would be compared against the expected numbers if the two groups had the same proportion of events/non-events. The log-rank test extends that over time, with the expected numbers of events at each event time being those if the two groups had the same survival function over time, thus accounting for censoring. You then combine results over all event times in the data to get a chi-square statistic based on the overall "observed" and "expected" numbers of events in each group. It's the number of events, not total sample size, that matters the most for either test.
See the review by Leung et al on "Censoring Issues in Survival Analysis," Annu Rev Public Health 18:83–104 (1997), for more detail about why you need to incorporate censoring into your analysis.