I captured them in mitmproxy and ran those through this to help me identify all the endpoints and their general structure. (A few things were a misnomer, like the examples suggesting certain values were able to be floats when they could only be integers)
I was able to get a team together and we were able to stand up private servers as a result.
A really professional and thorough job would be extremely time consuming and hard.
Having done a good bit of this type of reverse engineering the hard way over the years, it's a very exciting find. I had been talking with my partner about building something similar for the past six months. How exciting to learn that it's already out there and open source too!
The hard part is that inevitably, all these internal APIs will just add aggressive CAPTCHAs, Device Check, fingerprinting, etc to prevent common drive by re'ing. Easy to add these on the defence side, and extremely difficult to bypass on the other side.
I can imagine all developer teams now upping their security with the combination of the above mentioned to prevent this.
The API needs to be "public" because the app uses the internet to communicate back to the home server.
The API is not "public" in the sense that the app developers want anybody to use it; they just want their app to use this API. So they don't write publicly accessible documentation about it because they don't want to encourage its use.
A tool like MitmProxy2Swagger lets you run the app and record all of its API calls so that you can use this unadvertised API.
How many apps have you seen only do client-side protection?
Usage of the API even with the protocol known OTOH can be quite easily made really hard.
If you're working against an GraphQL based API, you should be able to pull a schema file. And use that to implement your own API.
All you would get from an Mitmproxy is example queries and mutations. With the additional complexity of extra tooling to stich together the schema file
I don't want to sit and read each query to identity where it is in the user flow. So I was thinking if I run this in the background and go through a happy flow, I can get the APIs in order and write an integration test.
(screenshot in case this goes away https://x.com/swyx/status/1874762725383188502)
EDIT: or rather follow the docs[0]
[0]: https://docs.mitmproxy.org/stable/howto-install-system-trust...
The thing is, depending on how hardened the app is, you'll have to play with Android to allow this interception, mostly because of certificate pinning. Also I remember something about apps not using the system wide trusted certificates you install (IIRC).
I remember using a rooted device with LineageOS, and downloading the APK and modifying it with a tool so the self signed certificate for the mitm proxy works with it.
The mitm proxy docs have some links to tools that can do that [0] and you could also use an Android emulator if you don't have an extra phone to mess with it [1]
0: https://docs.mitmproxy.org/stable/concepts-certificates/
1: https://docs.mitmproxy.org/stable/howto-install-system-trusted-ca-android/
I've been fighting lately with an API, though. It's not very, let's say, RESTy. It has only one endpoint, and the different "sections" of the API are defined in parameters, so MitmProxy2Swagger doesn't detect them properly :(
To be fair, from what I understand an actual(tm) REST API would only have a single defined endpoint[1]: the entry point. With every other endpoint being discovered from the responses. And also from your message I'm guessing a URI still uniquely identifies a resource (specifically through the "query" part of the URI, instead of the more common "path").
So, technically, assuming there's nothing too weird with that API, it seems like MitmProxy2Swagger is failing to detect a REST API.
[1]: Corollary: If an API is RESTful, it should be possible to rename any endpoint (except the entry point) at any moment in time without prior notice, and clients would not break as long as the response types/schemas are still supported by the clients. In-flight requests might fail with a 4xx, but after a retry they should go to the correct endpoint without any code change required.
You can add obfuscation, but ultimately if the client is shipped to the user you must assume an attacker can reverse engineer it.
You might intend your API to be consumed only by your own clients. E.g. your published mobile apps.
A well-designed API won't allow a third-party client to do anything that your own client wouldn't allow of course. Permissions are always enforced on the back end.
But there are many cases where a user might want a custom/different client:
If your mobile apps are not awesome, or if they deprioritize a specific use case, or if they serve ads ... or even if your users want to automate some action in your service...
If your service is popular enough (or you attract a certain kind of user), you will have some people building their own clients.
because guess what??? we are the creator of such system, its easy to detect bot/such case when you have good analytical data because this type of way does not give any "traces"
1. https://github.com/BandarLabs/clickclickclick - It does that and I am one of the authors.