satya - Fri Dec 28 2012 10:11:35 GMT-0500 (Eastern Standard Time)

Questions/Topics of this semi article are

What does Android recommend for data storage options? The Official line!
The JSON solution
What is GSON?
  What is GSON's homepage?
  Is there a user guide for GSON?
  Can I save nested objects?
  Can I save nested collections of objects?
  How are characters in strings escaped?
How do you add external jar files to Android APK file?
What are shared preferences?
Difference between getSharedPreferences and getPreferences?
How to get a context independent of an activity?
Where are shared preference files stored?
How do you save JSON to a shared preference file?
What does the saved preference file look like?
How do you read JSON from shared preferences back as Java Objects?
How does escape characters work in android preferences?
What is internal storage?
How do you save to internal storage?
How do you restore objects from internal storage?
Should you use external storage like the SD card?
What are the useful references while working with this approach?
How do you code such a way that you can migrate in the future to SqlLite?
What Next?

satya - Fri Dec 28 2012 10:17:05 GMT-0500 (Eastern Standard Time)

Tell me about Data Storage Options

Tell me about Data Storage Options

I have documented at the link above the choices of various data storage options. You can also use the references section at the top right to see the summary of 5 different options that Android suggests.

Summary of research is to use either shared preferences or internal files as the storage mechanism for your objects after converting them to JSON using GSON. These storage options have two advantages. First they are private. Second they are removed when the application is removed. The disadvantage is if the files become too large in terms of 10s of megabytes or have images or videos etc.

satya - Fri Dec 28 2012 10:18:51 GMT-0500 (Eastern Standard Time)

What is the JSON solution?

You use java objects to store them directly in persistent storage after converting them to JSON strings. The JSON strings can then be stored in internal files or in the shared preferences. We will use google's GSON as the tool to go between java objects and strings. It works really well!!

satya - Fri Dec 28 2012 10:21:19 GMT-0500 (Eastern Standard Time)

what is GSON?

GSON is a java library (a single jar file) that you can use to convert java objects to JSON and back. You can see links to the home page, user guide, and the API on the right hand side of this document. You can also see a detailed research notes on GSON in those links.

satya - Fri Dec 28 2012 10:23:13 GMT-0500 (Eastern Standard Time)

Can I save nested objects using GSON?

Yes. You can have objects inside objects in a nested structure. If you mirror your java objects with a focus on their storage structure you can accomplish quite a lot. So you want these objects to represent primarily data and not full blown behavior.

satya - Fri Dec 28 2012 10:24:43 GMT-0500 (Eastern Standard Time)

Can I use nested collections of objects using GSON?

Yes. as long as your member collections use java generics. When you use java generics you specify the type of the collections. This is what helps GSON to instantiate the right objects. See the user guide for more features and limitations.

satya - Fri Dec 28 2012 10:26:27 GMT-0500 (Eastern Standard Time)

What happens when my strings contain special characters?

Good question. GSON will escape the quote characters and any other character that is special for JSON. GSON also by default escapes the html and XML characters. In our research we found this behavior very satisfying.

satya - Fri Dec 28 2012 10:30:16 GMT-0500 (Eastern Standard Time)

How can I add the JSON jar to the APK file?

Go to the home page of GSON indicated at the right. 
download the gson jar file. 

Create a sub directory called "lib" in your eclipse project
  do this under the root of your project
  parallel to src.
Copy the gson jar to that lib directory

Go go project properties
Go to build properties
Add the gson jar as a dependency

Go to the order/export tab
Choose the gson jar to be exported as well

that's it. You will have the jar file in your APK when you build.

satya - Fri Dec 28 2012 10:31:44 GMT-0500 (Eastern Standard Time)

What not to do?

You may be able to link the external gson jar with out copying to your apk project. You will compile and link successfully. but when you run it you will have a run time class not found exception. So follow the steps indicated above as it did work for me that way.

satya - Fri Dec 28 2012 10:33:26 GMT-0500 (Eastern Standard Time)

Here is an object structure that I have used to test GSON

public class ChildObject {
   public String name;
   public int age;
   public boolean likesVeggies = false;
   
   public ChildObject(String inName, int inAge)
   {
      name = inName;
      age = inAge;
   }
}

public class MainObject 
{
   public int intValue = 5;
   public String strinValue = "st<ri>\"ng\"Value<node1>test</node2>";
   public String[] stringArray;
   public ArrayList<ChildObject> childList = new ArrayList<ChildObject>();
...
}

satya - Fri Dec 28 2012 10:34:07 GMT-0500 (Eastern Standard Time)

My main object can be further complicated like this with behavior

public class MainObject 
{
   public int intValue = 5;
   public String strinValue = "st<ri>\"ng\"Value<node1>test</node2>";
   public String[] stringArray;
   public ArrayList<ChildObject> childList = new ArrayList<ChildObject>();
   
   public void addChild(ChildObject co)
   {
      childList.add(co);
   }
   
   public void populateStringArray()
   {
      stringArray = new String[2];
      stringArray[0] = "first";
      stringArray[1] = "second";
   }
   public static MainObject createTestMainObject()
   {
      MainObject mo = new MainObject();
      mo.populateStringArray();
      mo.addChild(new ChildObject("Eve",30));
      mo.addChild(new ChildObject("Adam",28));
      return mo;
   }
   public static String checkTestMainObject(MainObject mo)
   {
      MainObject moCopy = createTestMainObject();
      if (!(mo.strinValue.equals(moCopy.strinValue)))
      {
         return "String values don't match:" + mo.strinValue;
      }
      if (mo.childList.size() != moCopy.childList.size())
      {
         return "array list size doesn't match";
      }
      //get first child
      ChildObject firstChild = mo.childList.get(0);
      ChildObject firstChildCopy = moCopy.childList.get(0);
      if (!firstChild.name.equals(firstChildCopy.name))
      {
         return "first child name doesnt match";
      }
      return "everything matches";
   }
   
}

satya - Fri Dec 28 2012 10:34:55 GMT-0500 (Eastern Standard Time)

Here is how I use GSON to convert it to a string and back

public void testJSON()
{
   MainObject mo = MainObject.createTestMainObject();
   Gson gson = new Gson();

   //Convert to string
   String jsonString = gson.toJson(mo);
 
   //Convert it back to object
   MainObject mo1 = gson.fromJson(jsonString, MainObject.class);
}

Incredibly simple.

satya - Fri Dec 28 2012 10:36:19 GMT-0500 (Eastern Standard Time)

All that is left now ....

how to store and retrieve this string in a persistent place. we have two options. shared preferences and internal file storage. I will cover shared preferences first.

satya - Fri Dec 28 2012 10:39:40 GMT-0500 (Eastern Standard Time)

What are shared preferences?

What are shared preferences?

use the link above to read more about shared preferences. But briefly a shared preference is an XML file that holds key value pairs in a persistent manner. The XML file is an implementation detail. Android may chose a different representation in the future should it chose so.

You can have as many shared preferences as you like. Each shared preference can have as many key/value pairs as you wish.

these xml files are private to your application. They are removed when your application is uninstalled.

satya - Fri Dec 28 2012 10:41:40 GMT-0500 (Eastern Standard Time)

Difference between getSharedPreferences and getPreferences?

you will need a context object to call the method getSharedPreferences(). However when you are in an activity you can just call getPreferences(). The later method merely calls the former with the activity name as the name of the xml file.

For our case we will use the getSharedPreferences() as the state of the entire application is not tied to a single activity.

satya - Fri Dec 28 2012 10:43:35 GMT-0500 (Eastern Standard Time)

How do you get the context in an application with out having access to an activity object?

public class MyApplication extends Application
{
   public final static String tag="MyApplication";
   public static Context s_applicationContext = null;

   @Override
   public void onConfigurationChanged(Configuration newConfig) {
      super.onConfigurationChanged(newConfig);
      Log.d(tag,"configuration changed");
   }

   @Override
   public void onCreate() {
      super.onCreate();
      s_applicationContext = getApplicationContext();
      Log.d(tag,"oncreate");
   }

   @Override
   public void onLowMemory() {
      super.onLowMemory();
      Log.d(tag,"onLowMemory");
   }

   @Override
   public void onTerminate() {
      super.onTerminate();
      Log.d(tag,"onTerminate");
   }

}

you basically create your own application instance and store a static pointer to the context.

satya - Fri Dec 28 2012 10:44:19 GMT-0500 (Eastern Standard Time)

Then you can do this

public void testEscapeCharactersInPreferences()
   {
      String testString = "<node1>blabhhh</ndoe1>";
      SharedPreferences sp = getSharedPreferences();
      SharedPreferences.Editor spe = sp.edit();
      spe.putString("test", testString);
      spe.commit();
      String savedString = sp.getString("test", null);
      if (savedString == null)
      {
         mReportTo.reportBack(tag,"no saved string");
         return;
      }
      //savedstring exists
      mReportTo.reportBack(tag,savedString);
      if (testString.equals(savedString))
      {
         mReportTo.reportBack(tag,"Saved the string properly. Match");
         return;
      }
      //they dont match
      mReportTo.reportBack(tag,"They don't match");
      return;
   }
   private SharedPreferences getSharedPreferences()
   {
      SharedPreferences sp 
      = MyApplication.s_applicationContext.getSharedPreferences("myprefs", Context.MODE_PRIVATE);
      return sp;
   }

satya - Fri Dec 28 2012 10:45:01 GMT-0500 (Eastern Standard Time)

Notice 2 things in the code above

use of MyApplication to get the shared preferences.

and the way we call getSharedPreferences(). Notice the mode bit.

satya - Fri Dec 28 2012 10:45:29 GMT-0500 (Eastern Standard Time)

The possible modes are

int mode;
//One of 
MODE_PRIVATE (value of 0 and default)
MODE_WORLD_READABLE
MODE_WORLD_WRITEABLE
MODE_MULTI_PROCESS

satya - Fri Dec 28 2012 10:46:12 GMT-0500 (Eastern Standard Time)

The shared preferences file is created the first time you do this

SharedPreferences.Editor spe = esp.edit();
spe.put...();
spe.commit();

satya - Fri Dec 28 2012 10:46:38 GMT-0500 (Eastern Standard Time)

Why Commit?

If this preferences file is shared among processes, a commit will take it to the persistent state and make it visible to the other clients. A single file will return a shared preferences object that is one or at least refer to the same file when in a different process.

satya - Fri Dec 28 2012 10:47:06 GMT-0500 (Eastern Standard Time)

In android where are the shared preferences xml file kept?

/data/data/YOUR_PACKAGE_NAME/shared_prefs/YOUR_PREFS_NAME.xml
or the default is at
/data/data/YOUR_PACKAGE_NAME/shared_prefs/YOUR_PACKAGE_NAME_preferences.xml

satya - Fri Dec 28 2012 10:48:52 GMT-0500 (Eastern Standard Time)

How can I look at this file?

In eclipse ADT use the file manager tool to see the files on your emulator or the device and pull the file to your local drive and view it.

satya - Fri Dec 28 2012 10:51:29 GMT-0500 (Eastern Standard Time)

Here is how XML characters in the string values are escaped by Android preferences

<?xml version='1.0' encoding='utf-8' standalone='yes' ?>
<map>
<string name="test">&lt;node1&gt;blabhhh&lt;/ndoe1&gt;</string>
</map>

satya - Fri Dec 28 2012 10:53:27 GMT-0500 (Eastern Standard Time)

Now How do you save/restore objects using GSON and shared preferences?

public void storeJSON()
{
    MainObject mo = MainObject.createTestMainObject();
    //
    Gson gson = new Gson();
    String jsonString = gson.toJson(mo);
    this.mReportTo.reportBack(tag, jsonString);
    MainObject mo1 = gson.fromJson(jsonString, MainObject.class);
    this.mReportTo.reportBack(tag, jsonString);
    
    SharedPreferences sp = getSharedPreferences();
    SharedPreferences.Editor spe = sp.edit();
    spe.putString("json", jsonString);
    spe.commit();
}

public void retrieveJSON()
{
    SharedPreferences sp = getSharedPreferences();
    String jsonString = sp.getString("json", null);
    if (jsonString == null)
    {
        mReportTo.reportBack(tag,"Not able to read the preference");
        return;
    }
    Gson gson = new Gson();
    MainObject mo = gson.fromJson(jsonString, MainObject.class);
    mReportTo.reportBack(tag,"Object successfully retrieved");
    String compareResult = MainObject.checkTestMainObject(mo);
    if (compareResult != null)
    {
        //there is an error
        mReportTo.reportBack(tag,compareResult);
        return;
    }
    //compareReesult is null
    mReportTo.reportBack(tag,"Retrieved object matches");
    return;
}

satya - Fri Dec 28 2012 10:56:02 GMT-0500 (Eastern Standard Time)

Here is how the preferences file look like holding the JSON strings

<?xml version='1.0' encoding='utf-8' standalone='yes' ?>
<map>
<string name="test">&lt;node1&gt;blabhhh&lt;/ndoe1&gt;</string>
<string name="json">{&quot;childList&quot;:[{&quot;name&quot;:&quot;Adam&quot;,&quot;likesVeggies&quot;:false,
&quot;age&quot;:30},{&quot;name&quot;:&quot;Eve&quot;,&quot;likesVeggies&quot;:false,
&quot;age&quot;:28}],&quot;stringArray&quot;:[&quot;first&quot;,&quot;second&quot;],
&quot;strinValue&quot;:&quot;st\u003cri\u003e\&quot;ng\&quot;Value\u003cnode1\u003etest\u003c/node2\u003e
&quot;,&quot;intValue&quot;:5}</string>
</map>

satya - Fri Dec 28 2012 10:57:14 GMT-0500 (Eastern Standard Time)

Notice how the escaping works between GSON and Android Preferences

Both have escape sequences. So we should not worry about our string values in objects containing special characters. They are automatically handled WELL both by GSON and Android preferences.

satya - Fri Dec 28 2012 10:59:45 GMT-0500 (Eastern Standard Time)

What is internal storage?

If you are further afraid that you want full control of the file where the JSON is stored you can use the internal storage option provided by Android. These internal files are stored on the device specifically for your app and are private by default. I will present some code snippets on how to do this.

satya - Fri Dec 28 2012 11:01:09 GMT-0500 (Eastern Standard Time)

Sample code to store/retrieve from internal storage

private String readFromInternalFile()
{
    FileInputStream fis = null;
    try {
        Context appContext = MyApplication.s_applicationContext; 
        fis = appContext.openFileInput("datastore-json.txt");
        String jsonString = readStreamAsString(fis);
        return jsonString;
    }
    catch(IOException x)
    {
        mReportTo.reportBack(tag,"Cannot create or write to file");
        return null;
    }
    finally
    {
        closeStreamSilently(fis);
    }
}
private void saveToInternalFile(String ins)
{
    
    FileOutputStream fos = null;
    try {
        Context appContext = MyApplication.s_applicationContext; 
        fos = appContext.openFileOutput("datastore-json.txt"
                                ,Context.MODE_PRIVATE);
        fos.write(ins.getBytes());
    }
    catch(IOException x)
    {
        mReportTo.reportBack(tag,"Cannot create or write to file");
    }
    finally
    {
        closeStreamSilently(fos);
    }
}

Nothing complicated at all.

satya - Fri Dec 28 2012 11:01:38 GMT-0500 (Eastern Standard Time)

Where are my internal files stored if I want to look at them?

data/data/<your-pkg-name>/files/<your-internal-filename>

satya - Fri Dec 28 2012 11:02:09 GMT-0500 (Eastern Standard Time)

Here is what the JSON look like in the file

{"childList":[{"name":"Adam","likesVeggies":false,"age":30},
{"name":"Eve","likesVeggies":false,"age":28}],"stringArray":["first","second"],
"strinValue":"st\u003cri\u003e\"ng\"Value\u003cnode1\u003etest\u003c/node2\u003e",
"intValue":5}

satya - Fri Dec 28 2012 11:02:54 GMT-0500 (Eastern Standard Time)

Should you use external storage like the SD card?

Because typically the data you would represent as JSON is very specific to your application it doesn't make sense to make this available as external storage which is typically used for music files, video files or files that are common in format that is understandable by other applications.

Because external storage such as an SDCard can be in various states (available, not mounted, full, etc) it is harder to program this for simple apps when the data is small enough.

So I don't foresee for now the application state as being maintained on external storage.

May be a hybrid approach is meaningful if the application requires music and photos and those can go on the external storage while keeping the core state data in JSON and internal.

satya - Fri Dec 28 2012 11:05:06 GMT-0500 (Eastern Standard Time)

What are the useful references while working with this approach?

This document
GSON user guide
Data Storage Options summary document from Android
various Android APIs pertaining to Shared preferences
various Android APIs pertaining to internal storage
Some basic java apis to read/write files

I have provided links for all of these on the right hand side.

satya - Fri Dec 28 2012 11:07:54 GMT-0500 (Eastern Standard Time)

How do you code such a way that you can migrate in the future to SqlLite?

This is a real good question. This GSON approach will be expensive if you are writing the file every time something changes. There is a break even point where you want to migrate your code to sqllite and use granular updates.

you can structure your code so that you can swap the persistence layer with out significantly altering your code.

To do this break your app into a persistence service layer and the rest. You will need to be disciplined enough to use the service layer as a stateless set of services that you can use a different implementation later. So remember interfaces and service interface firewalls.

satya - Fri Dec 28 2012 11:15:22 GMT-0500 (Eastern Standard Time)

Is there a write up on this approach?

Yes. See this link

This is an article that I wrote in 2006 about extreme prototyping. You can borrow the same principles to construct a service layer that can be swapped.

satya - Fri Dec 28 2012 11:18:23 GMT-0500 (Eastern Standard Time)

What next?

In conclusion GSON is a good approach for quickly deploying your apps to the market place. It is good to test out market strengths. It is good to release a whole lot of simple apps really fast with minimal work on persistence.