Apk reversing ; Removing ads

Hey everyone! Just recently I started getting interested in reversing android apk's. In order to get an internship in a company, I had to find some flags in apk's they sent me. It was a very interesting experience as I had never played with apk's before.
I will write an article about my experience in finding these flags someday. However today is about a project I did with this new found power. Indeed, I thought to myself, "How about trying to reverse an apk in order to remove those annoying ads from it?"
I chose to work on "cyber security news from news fusion" application as I had it on my phone and actually found it quite handy.

Now I know what you're thinking... Ethically speaking, this kind of sucks as developers earn money thanks to ads. Nevertheless my intention there was not to ruin developers but to try and put my skills to the test on a real life scenario so please keep the app with ads or actually buy it in order to support the work behind it.

PRIOR NOTE

Will update this post with a few more screenshots as soon as I fix my broken phone. Also I might need to resize one of the screenshots but just couldn't be bothered to do it now sorry.

About android

The android OS was at first supposed to look like something close to the dangeros from dangers Inc. Which more or less brings an abstraction lay for java allowing to host apps.
At the start google did not care much for android and was more excited by the Iphone however in 2005, google buys the Android company and in 2007 the first Android terminal T-mobile G1 is out. Since then, android has evolved a lot. Changing java virtual machines (jvm) three times. The basic android architecture looks like so:

Android Architecture

The libc in android is the bionic libc, a google taylored lybrary made for android.
Android's jvm were custom made at first, the Dalvik was the one until the 4.3 version where the ART made it's appearance including ahead of time compilation meaning it compiles entire applications to native machine code when they are installed. This improves overall efficency.
However, ART still makes use of dex files in order to stay backwards compatible. Nevertheless, .odex files are now replaced by elf files. Android 4.4 offered a first version of ART but still had a Dalvik jvm. Because the jvm were based on the oracle jvm, Android was constantly being sued. Hence, they decided to change their jvm with an openjdk in Android N (for Nougat = Android 7.0). Since then, they haven't been sued for this issue.

Now you are probably wondering what the hell is a dex file?
It's basically bytecode arranged in a certain way. To understand where it comes in the compiling process, you might want to take a look at this beautiful flowchart from stackoverflow:

Android compiling process

I am more or less paraphrasing from stack overflow as doing a better job of explaining this would be quite difficult!

So, as explained in the flowchart the Android Asset Packaging Tool (aapt) gets the app's resource files
such as the AndroidManifest.xml file and the XML files for your Activities, and compiles them. Ensuing in two outputs, an R.java which allows to reference resources from within Java code as well as the compiled resources.
Meanwhile the Android Interface Definition Language tool converts all .aidl interfaces into Java interfaces. For those who are wondering what aidl files are, well they allow you to define the programming interface that both the client and service agree upon in order to communicate with each other using interprocess communication (IPC).
Once those two steps are over, all Java code, including the R.java and .aidl files, are compiled by the Java compiler and .class files are output. At this point, the .class files contain bytecode. They are used by the JVM to load class definitions.
The dex tool converts the .class files to Dalvik byte code. Any 3rd party libraries and .class files that you have included in your project are also converted into .dex files so that they can be packaged into the final .apk file.
All non-compiled resources (such as images), compiled resources, and the .dex files are sent to the apkbuilder tool to be packaged into an .apk file.
Once the .apk is built, it must be signed with either a debug or release key before it can be installed to a device. This is pretty much equivalent to a zip file as the unzip command works and will extract several files including the AndroidManifest.xml (if you're wondering what this file is, well it's equivalent to java's special.policy file. You sum up the rights the app needs on the phone and the user is prompted wether or not to accept them at install(super secure...(not!)). It also includes a shit ton of other things but I won't go into detail right away.
Finally, if the application is being signed in release mode, you must align the .apk with the zipalign tool. Aligning the final .apk decreases memory usage when the application is running on a device. In this article I did not bother aligning the recompiled application I worked on. However it's very simple to do so and I'm sure you can find how to on stack :)
Please be aware that Android N's openjdk does not make use of dex files and I am unaware to wether it can still support them in order to be backwards compatible.

Down to business

Yay! Finally! (For those who read everything above, the others, well you're real hoodlums)

Getting the apk

Now you guys probably noticed that if you just use plain old android provided tools to browse your files, well there's no trace left of the apk used to install your apps. In order to retrieve it, there are several ways. Most of them are crap, as explained below:

First method

Most browser have an add-on which allows to browse on google store and directly download the application's apk. I tryed the firefox version here and even though I believe the app's code is checked prior to being put on the addons.mozilla.org domain, the app asks for your google account credentials and I am absolutely not cool with giving them to it.

Second method

There are websites out there from which you can directly download an apk (such as this guy). Even though it is quite simple, this solution stinks because there might be a malware in one of these apk's and even though we are going to reverse it, we won't be reading the whole code and understanding how the whole app works (thank god!). Hence, I do not recommend installing one of these on your phone.

Third (awesome viking) method

Dragon born using this method

This method makes use of the android debug bridge (adb) program. You will need to install it on your operating system before going forward.
Just download the app on your phone through google. Activate usb debug mode on your phone (on Android 4 and 5 Settings > About Phone > Build number > Tap it 7 times).

Once this is done, plug your phone to your computer and type the following to list third party apk's:

 adb shell pm list packages -f-3 # only lists third party apk 

Then type:

 adb pull /data/apps/\<nameofapk\>/base.apk                                                                                                                                                                                                                                                                                                                                                                                                                                                       

You can also use a terminal emulator application directly on your phone to try and understand how android is set up.
This will have way more options if you're phone is rooted.
Basically /etc and stuff is in /system when you are not root you are confined
to /system (chroot jail) . However, living the thuglife, I rooted my phone a long time ago.

Here is a quick overview of the /data/app folder using the terminal emulator (I blanked a few things because of privacy issues and I am not allowed to disclose the name of one of the apps on my phone)

We can now go to our working directory and decompile the apk thanks to
apktool's decode option. This will output a directory with many interesting
files.

apktool d base.apk                                                                                                                                                                                                                
cd base       

The first interesting one being the Androidmanifest.xml
This file declares the android permissions which is the stuff written in the
window you click without reading whenever installing a new app. It basically
tells what the app is allowed to do on your phone.
Moreover it names the java package, describes the components of the app, decides of the process that host the application components and much more.
You probably want to have a look at this link if you're wondering all the awesome things the Androidmanifest.xml file does.

Anyways, when reversing an apk, in order to get a general idea of what we're dealing with, this is step1.

vim Androidmanifest.xml        

(Or cat or less, whatever floats your boat).

Since we're looking for ads related stuff, just searching for 'ads' keyword helps us spot the little:
google.com.ads stuff and AdsRemovalActivity.

google.com.ads in Androidmanifest.xml

This is basically the google ads module developpers can call in their code to print google ads to user screen. The more views the ad gets from one application, the more money is sent to the application's vendor. The presence of the AdsRemovalActivity is very interesting as it shows that the app probably has a built in function allowing users to remove ads under some conditions.

"All right! That's cool, but how do I actually get rid of it?"

Never thought you'd ask ;)

Several solutions as explained on this very cool blog in
the "Android App Ad Hiding - Android App Cracking Tutorial #1" article.

The solutions exposed in the article are the following:

  • 1.Patch res to make ads too small to actually see.
  • 2.Directly patch google.com.ads from where the ads come from
  • 3.Patch the main app module directly for the call to google.com.ads

So I tryed patching res as described. The res file is the file containing all resources in the apk. In this folder is a file which calls the com.google.ads module and displays the ads. It decides the layout size the ads should be. The idea is that if we change this layout size to something tiny or equal to zero, the ads will be too small for us to see.

However this did not work for my apk because I went in res/layout
and could not find main.xml. Or grep for com.google.ads as suggested in the article.

Failure to patch res

It might've had another name but there were a lot of files in the res folder
and finding in which one I needed to change the layout_height or
layout_width was too hard!

Patching google.com.ads seemed pretty complicated as the app is quite complex and failsafe but I might give it a shot someday.

Hence, since the option explored in the article was too complicated I felt at a loss.

Nevertheless I decided to give the third option a shot. As mentioned when looking at the AndroidManifest.xml, I spoted an "AdsRemovalActivity".

Well then, let's try patching it!

After finding the smali file associated to the AdsRemovalActivity which was in:

smali_classes2/com/newsfusion/AdsRemovalActivity.smali

I noticed several static fields each associated to a state:

STATE_ON_TRIAL                                                                                                                                                                                                                    
STATE_BEFORE_TRIAL                                                                                                                                                                                                                
STATE_AFTER_TRIAL                                                                                                                                                                                                                 
STATE_ADS_REMOVAL_PURCHASED

 Several staic fields

Dynamic analysis of the application allowed me to quickly understand that this was associated with a button the user could click in the app in order to either:

  • Get a trial period with no ads.
  • Purchase the premium ad-free application.

Going through the file and not knowing much about smali, at first I thought there was an if statement looking at the actual state and that it would then set the new state directly.
I thought this was the purpose of the smali instruction ifne (if not equal).

Hence I decided to replace all calls to STATE_SOMETHING by STATE_ADS_REMOVAL_PURCHASED .

 Smali STATE replace

After recompiling the app, well this did not work and just froze the application(I did not directly replace all calls, I made several more subtle tryes that did not work either).

Why did this method fail terribly?

Probably because I suck at smali...

Hence I decided to look directly at the java code in order to check where I had gone wrong when looking at the smali.

I decompiled the .dex file (and not disassembled like last time) using jadx or dex2jar:

d2j-dex2jar base.apk

The reason I did not do so in the first place was mostly because I wanted to give smali a try, show you guys what it looked like and because some apk use security modules which obfuscate the .jar and cypher part of the code(they also sometimes prevent the app from running in rooted or virtual environments). Nevertheless it is still possible to bypass these modules. Another reason is that dex to java conversion is sometimes uncertain and some structures may get mixed up. I believe the better approach when patching an apk is to obtain both smali and java code. Read the java code to get a rough idea and patch the apk using smali.

After converting the newly obtained jar file back to java with jd-gui or using jadx (cooler way):

find -name classes.dex
cd whereverethefileis
jadx -d output classes.dex
jadx -d output2 classes2.dex

We jump to the AdsRemovalActivity.java file which corresponds to the smali file we studyed earlyer. We quickly notice that there is no obfuscation... Yay! :D
Several ifs which compare a value to the STATE and then call a corresponding method.

if statements java

So I was right except I did not think a method was called to change the state value, I believed it was done directly (as a dude who likes to code in a disgusting fashion like me would do)
Static or dynamic analysis shows that there is a onclick listener on the trial button (kind of logical).

Replacing the STATE_SOMETHING variables was a bad call as I was changing the if statement.
What I should have done was change the initOnTrial() metod call by initAdRemovalPuchased()

I did so in the smali file.

Recompiled the apk:

apktool b <Folder Containing Modified Files> -o Nw.apk

Sign it:

keytool -genkey -alias apkKey -keystore /home/warsang/myawesomeapkkeystore
jarsigner Nw.apk apkKey -keystore /home/warsang/myawesomeapkkeystore

And finally used zipalign to optimize the app in order to use less memory (did not do it):

zipalign -v 4 Nw.apk securityNews.apk

WARNING:
Now unless it's a very simple app, don't recompile .java obtained from jadx or
dex2jar decompilation because from dex decompilation is pretty yolo and can muck
up complicated structures hence you'll have to rewrite them and recreate the whole project structure.
This is what these guys did with pokemonGo. Unfortunately the very interesting article they wrote is down but the git repo is still up.

Edit :

Very recently I was faced with a strange error while recompiling an apk with apktool. It printed the following:

 android apktool b classes_dex2jar 
I: Using Apktool 2.2.2
Exception in thread "main" brut.androlib.AndrolibException: 
brut.directory.PathNotExist: apktool.yml
    at brut.androlib.Androlib.readMetaFile(Androlib.java:258)
    at brut.androlib.Androlib.build(Androlib.java:270)
    at brut.androlib.Androlib.build(Androlib.java:263)
    at brut.apktool.Main.cmdBuild(Main.java:227)
    at brut.apktool.Main.main(Main.java:84)
Caused by: brut.directory.PathNotExist: apktool.yml
    at brut.directory.AbstractDirectory.getFileInput(AbstractDirectory.java:105)
    at brut.androlib.Androlib.readMetaFile(Androlib.java:254)
    ... 4 more

I obtained the source code through qark (I really recommend this tool for anyone doing mobile pentest btw) which actually uses dex2jar.

That was because d2j partially failed the decompilation as explained above.
However the project was very complicated and bypassing the authentication(my goal) was to complicated to do in smali. (Another solution would have been to use frida
In order to fix this issue I just did:

apktool d base.apk

And moved the apktool.yml file (as well as another missing source file which was pointed out in a following error) into the source code file obtained through qark.

Some of the links I used to write this article in a random order (sorry if you feel like I used something you wrote and it isn't there)

https://github.com/applidium/PokemonGo_Android_RE
http://stackoverflow.com/questions/24570534/how-and-what-converts-java-bytecode-to-android-dex-file
https://developer.android.com/studio/build/index.html
https://docs.oracle.com/javase/tutorial/deployment/jar/signing.html
https://docs.oracle.com/javase/tutorial/security/toolsign/step3.html
http://stackoverflow.com/questions/23333724/how-to-zip-align-apk-file-in-android
https://developer.android.com/studio/command-line/zipalign.html
http://stackoverflow.com/questions/12370326/decompile-an-apk-modify-it-and-then-recompile-it
http://androidreversing.blogspot.fr/
https://developer.android.com/guide/topics/manifest/manifest-intro.html
http://getapk.co/
https://addons.mozilla.org/fr/firefox/addon/apk-downloader/
http://openjdk.java.net/
https://www.frida.re/
https://github.com/linkedin/qark

UPDATE:

Great talk on Android security architecture:
https://www.youtube.com/watch?v=3asW-nBU-JU&index=2&list=PL61IkVbNYniVIjf20ehmV-mtJrbc7_2WC