Mmmmm... Market Data...
It turns out downloading a free application is actually pretty easy to reproduce. The things required by the google android servers are just four variables. The server needs to know your userId, authToken, deviceId and the applications assetId.
The userId is a unique number that is associated only with your gmail account, the one that is currently linked to the phone. I’m working on getting a generic way to grab this number, though I believe the request is buried in an ssl request to the google servers. So for now, you can obtain your own userId by doing a tcpdump of your market traffic, just do a download of an application and look for a “GET” request in wireshark. There does not appear to be a “hard” maximum character on this, I’ve seen userIds as low as 8 in length and as high as 13. A bad userId will return a 403 forbidden response.
The authToken is sent in cookie form to the server, to authenticate that the user is using a valid and non-expired token and well, is who they say they are! This is linked to the userId and must match the account that the userId is taken from. Expired tokens will return a 403 forbidden response.
The deviceId is simply your Android_ID, and is linked in anyway to the authtoken or user-id, so feel free to spoof this.
The assetId is a number (negative or positive) that identifies the current stream of the application you wish to download. More on this later when I cover how to get live market data. Note that this number is not always the same - it (appears) to change when something from the application is changed. Originally I referred to this in my research as a “cacheAppID” for just that purpose.
// Downloading apk's without vending/market
// Coded by Tim Strazzere
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.BufferedOutputStream;
import java.io.FileOutputStream;
import java.io.UnsupportedEncodingException;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLEncoder;
import java.net.HttpURLConnection;
public class main {
public static void main(String[] args) {
// current assetId for the yahoo search apk
String assetId = "7884814897504696499";
// input your userId
String userId = "12345678901";
// spoof your deviceId (ANDROID_ID) here
String deviceId = "2302DEAD532BEEF5367";
// input your authToken here
String authToken = "DQAAA...BLAHBLAHBLAHYOURTOKENHERE";
String cookie = "ANDROID=" + authToken;
try {
// prepare data for being 'get'ed
String rdata = "?" + URLEncoder.encode("assetId", "UTF-8") + "=" + URLEncoder.encode(assetId, "UTF-8");
rdata += "&" + URLEncoder.encode("userId", "UTF-8") + "=" + URLEncoder.encode(userId, "UTF-8");
rdata += "&" + URLEncoder.encode("deviceId", "UTF-8") + "=" + URLEncoder.encode(deviceId, "UTF-8");
// Send data
URL url = new URL("http://android.clients.google.com/market/download/Download" +rdata);
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
// For GET only
conn.setRequestMethod("GET");
// Spoof values
conn.setRequestProperty("User-agent", "AndroidDownloadManager");
conn.setRequestProperty("Cookie", cookie);
// Read response and save file...
InputStream inputstream = conn.getInputStream();
BufferedOutputStream buffer = new BufferedOutputStream(new FileOutputStream("out.put"));
byte byt[] = new byte[1024];
int i;
for(long l = 0L; (i = inputstream.read(byt)) != -1; l += i )
buffer.write(byt, 0, i);
inputstream.close();
buffer.close();
System.out.println("File saved...");
}
catch (FileNotFoundException e) {
System.err.println("Bad url address!");
}
catch (UnsupportedEncodingException e) {
System.out.println(e);
}
catch (MalformedURLException e) {
System.out.println(e);
}
catch (IOException e) {
if(e.toString().contains("HTTP response code: 403"))
System.err.println("Forbidden response received!");
System.out.println(e);
}
}
}
Hopefully someone will find this stuff useful ;) Better than me just sitting on it forever!
Sunday, 27. September 2009
GENIUS im not a dev but this will definately help the ASOP
Thursday, 8. October 2009
Hello, Tim
I did some sniffing via wireshark on emulator for vending.apk.
I observe the following request string when searching an apk
POST /market/api/ApiRequest HTTP/1.1
Host: android.clients.google.com
Connection: Keep-Alive
Content-Length: 476
user-agent: Android-Market/2 (dream CRC1); gzip
content-type: application/x-www-form-urlencoded
version=2&request=CsUCCuABRFF….ETXl5Qi1wUDcy
Do you have any idea of how it is encoded or composed?
I tried based 64 but only partial contents are human-readable after decoding :(
Any advice would be highly appreciated~!
Best regards,
Aaron
Friday, 9. October 2009
Hello,
I noticed that this POST request might be encoded by google protocol buffer but still no luck when trying com.google.protobuf.UnknownFieldSet or CodedInputStream. Is there any hint or advice that could guide me through this? Just to make sure i’m going the right direction. Thanks very much :)
Best regards,
Aaron
Sunday, 11. October 2009
Looks great !
Has anyone compiled this snippet into a nice, n00bz-friendly .apk?
I’d love to use that as I refused to give Google any info since I started using my android….
Saturday, 7. November 2009
Hi,
I run into same issue as Aaron’s post here, and used the same protocol buffer classes to retieve the data, but no luck. A
Saturday, 7. November 2009
Hi,
I run into same issue as Aaron’s post here, and used the same protocol buffer classes to retieve the data, but no luck. Aaron or anyone else here, got a solution for this? Greatly appreciate it.
Saturday, 7. November 2009
maybe needing to use this package instead? - com.google.common.io.protocol
Saturday, 7. November 2009
@aaron and raul — it is a protobuf, you just need to figure out what fields are what
Wednesday, 11. November 2009
does it matter if i use com.google.common.io.protocol package that is used in android or the standard implementation com.google.protobuf? Just to make sure I dont run into issues choosing the unsuitable one.
another question - is the response from market encoded too as the same way as the request?
Friday, 20. November 2009
Could be possible that the required userID is the one visible at http://www.google.com/profiles/me/editprofile (in the bottom of the page, under “Profile URL” ?
I can’t test this, since I’m using android with an Openmoko Phone and, using an opensource verison of Android, I can’t use the default Vending.apk application.
So I’m really waiting for a way to use an alternative market app to download the available freeware software there.
Thanks for your work anyway.
Friday, 20. November 2009
@Treviño
Sorry no - that is the username, not a userId which appears in the requests.
Monday, 23. November 2009
Tim, sorry. I’m in the same point with raul and Aaorn. We have this base64 encoded string. When we decode, I think I can understand all the fields, but is not a protocol buffer valid string (for my protobuf decompiler).
Can you help us?
Monday, 23. November 2009
@Alex
We’ve discussed this via email before — if you’d like to reopen this subject then please reply again to that email.
Sunday, 27. December 2009
Hiya Tim,
Wondering whether you’ve made any progress on decoding the protocol buffer formats for the market app. I’ve been playing with trying to decode the request string that I sniffed out, but protocol buffers aren’t the funnest to reverse engineer, even with the market app ddx files as clues.
Have you cracked that little gem?
-Nic
Tuesday, 9. March 2010
Any chance someone can explain a way to programmatically get the UserID? I got the rest of it working..
Wednesday, 10. March 2010
@Nas
There is currently no (released) way to programmatically get the UserID.
Most people have gotten the rest working, it helps that someone has published an api on google code for it :)
Tuesday, 27. April 2010
java.io.IOException: Server returned HTTP response code: 411 for URL: http://android.clients.google.com/market/download/Download?assetId=7989310633059082746&userId=104400774266988007312&deviceId=2302DEAD532BEEF5367
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1243)
at com.souapp.market.DownloadApp.main(DownloadApp.java:53)
Tuesday, 15. June 2010
Tim,
Is there a programmatically way to get the userId at this time?
if not, can you point out some possible methods?
Wednesday, 16. June 2010
@ABitNo
Sorry, no - at this time there is no programmatic way to get the userId that has been found