Rate limit frustrations


#1

We use the API to create an invoice with FA, register the payment with GoCardless/Stripe and then update the invoice in FA. Alongside this, each morning we have a background job that runs, finding all unexplained/unapproved transactions in FA, parsing the reference and then hitting the GoCardless/Stripe APIs to try and find the data we need to explain the transaction. Once we’ve gathered the payout data, we then loop through the invoices (placed individually onto a queue), explaining them in the FA API.

With the 2 requests per second rate limit in place, we have to artificially slow down our background jobs from being able to process, which is frustrating and clogs our queues. In order to process our background jobs, we run 3 servers, so the moment we register 1500 invoices to raise or 1500 transactions to explain, we’ll trigger the rate limit exception. To get round this, we’ve put in a sleep(1) second to each of our background jobs to try and not trigger the exception. Without the sleep(1), we’d hit the API limit every single second.

Can I plead with you to increase your rate limit to something that’s more practical? I note that GoCardless have a limit of 16/s and Stripe 1000/s - we’ve never triggered these, but 2/s is very hard to work with…

I’m pretty sure you’ll say that you can’t raise your API rate limit, so assuming that’s the case, please could you give me a suggestion as to how, given the scenario I’ve described, you would work with your APIs and within the rate limits?


#2

In DoubleAgent I keep track of the rates in the API layer and auto-throttle the API requests (i.e. sleep until the limit is reset). It means each process goes as fast as it can and then waits for as short a time as possible to comply with the rate limits.

I agree it would be great if the rate limits were increased.

– Andrew