Calculate statistical significance in A/B price test Let's assume I'm selling a product at two price points, A and B and I'm alternating the displayed price to each visitor for a period of time.  I receive a specific number of orders, O(a) and O(b) at each price point after time T.  
If I compare the profit after time T of the orders at each price point, is there a way to determine statistical significance?
I do not have conversion rates in this case.  I simply know the price points, time elapsed, and number of orders at each price point.
For example, users order 15 widgets at a 23 dollar price point and 17 widgets at a 24 dollar price point after 3 days.  Is this enough information to run analysis?
Edit:
For simplicity, assume each selling price is also the profit (cost of goods sold is 0).
 A: This is definitely not sufficient. In order to test the effectiveness of the price change you made, you need the total number of visitors visiting the site during those times who would see each of the prices.
Using that information you could instead do an A/B test comparing the percentage of users who purchase widgets (a much more informative metric that would account for increased traffic to the site).
Once you determine that you can measure the significance using the following formula:



A: As with most analysis questions, the answer is you can definitely test something, but whether or not the test will really address your underlying research question is debatable. It will rely on assumptions that may not be met in this case, and it sounds like you don't have the information to be able to either run a more careful test or to investigate the assumptions directly. That said, here's an approach you could take:
Assume that there is some expected number of purchases per time interval T, and that the expected number of purchases per T may be different depending on the price. If your data extend over multiple T intervals, you can test whether there is a difference in expected number of purchases for price A vs. price B with a poisson regression. Your sample size will be the number of T intervals (e.g. if T is one day, then N = the number of days). The outcome variable would be the number of purchases, and the predictor would be price (A or B). 
This approach makes the following (potentially problematic) assumptions:


*

*That there are no additional variables that would predict number of purchases other than price. For example, this implies that the general volume of traffic either doesn't matter or is the steady over time (i.e. it's the same for each interval T), and it doesn't vary by price. 

*That number of purchases during each interval T are independent of previous purchases. In particular, if the same individual visits more than once, whether or not they buy depends only on the current price they see and not on the history of prices they've been exposed to.


To the extent that these assumptions are violated, the results of the analysis become less relevant/informative. 
