Thanks for this, I was able to find your customer’s problematic statement upload and requests made by DoubleAgent every half an hour to that account. After doing some digging, we think the reason why none of the bank transactions were detected by your first query at 00:03:00, and only some (~10 000 out of ~14 000) were detected by your query at 00:33:00 is caused by the fact that statement uploads are wrapped in a Rails transaction, which means that records are not fully committed to the database until everything within the transaction is processed (although the
updated_at timestamps are set throughout that time span).
If you have access to a Rails console, you can test this by updating two things inside an ActiveRecord transaction with a 1 minute sleep in between — this results in the
updated_at being set to the time the update was made, but the change not being made until the transaction completes.
In your case, because the upload started at 23:56 and, due to the high number of processed bank transactions, didn’t finish until 00:06, the bank transactions were not ready to be returned in response to DoubleAgent’s request at 00:03, and as your second request at 00:33 asked for bank transactions with the
updated_at timestamp set within the past half hour, it only picked up on a subset of records from the statement.
To avoid this happening in the future, you could introduce some time overlap in your polling requests or alternatively use a call to
/v2/bank_transactions?bank_account=:bank_account&last_uploaded=true when trying to detect the latest upload. I realise this is not ideal, so I’ll flag this to the team responsible for the banking functionality in the app; independently, we’re also beginning to do some initial (albeit long overdue) research into the introduction of webhooks, which no doubt would help you avoid such pitfalls as well.