BlackBerry

Developers

Garry Seifried, Research In Motion

Note: The format of IPD files described in this article may change at any time at the sole discretion of Research In Motion. Research In Motion will not provide technical support on issues related to IPD files and their format.

The Need for Speed

One of the challenges of working with wireless applications is that delivering large amounts of data over the wireless network can be both time-consuming and expensive.

Suppose that you want to put the contact information for an organization on the BlackBerry wireless device. Normally, we only put the most essential data on the device, and wirelessly pull additional less-essential data as may be needed.

Let’s assume, for the sake of this article, that the entire set of contact information is required.

BlackBerry Desktop Manager Backup/Restore

We’ve all used the BlackBerry Desktop Manager to perform backup of our device data - perhaps prior to performing an upgrade to the device software. Periodically, we’ve also needed to use that backup to restore our data. But have you ever had a look inside one of those IPD files?

That’s one nasty looking file!

Using the IPD Format for Bulk Loads

What if the most effective and efficient way of getting a large amount of data onto the device were to use the IPD format? Using this format, we could programmatically create a file that could be used by the desktop manager in a restore operation.

The structure of the IPD file shown above is as follows:

Inter@ctive Pager Backup/Restore File
<Line feed> - 1 byte - value 0A
<Version> - 1 byte - value 02
<Number of databases in file> - 2 bytes
<Database name separator> - 1 byte - value 00
<Database name block#1>
<Database name block#2>
.
<Database name block#n>
<Database data block#1>
<Database data block#2>
.
<Database data block#n>

Where each database name block is of the form:

<Database name length>	2 bytes The length includes the terminating null
<Database name>	As long as the name length

And each database data block is of the form:

<Database ID>	2 bytes Zero-based position in the list of database name blocks
<Record length>	4 bytes
<Database version>	1 byte
<DatabaseRecordHandle>	2 bytes
<Record unique ID>	4 bytes
<Field length #1>	2 bytes
<Field type #1>	1 byte
<Field data #1>	As long as the field length
<Field length #m>	2 bytes
<Field type #m>	1 byte
<Field data #m>	As long as the field length

Parse that IPD

In manually parsing the file, we can see some structure that matches the binary structure just outlined:

Number of databases: 57
Database[ 0 ] = Content Store
Database[ 1 ] = Service Book
.
Database[ 56 ] = WTLS Options

Record for database 0:
  Database version: 0x01
  Record handle:    0x0001
  Unique ID:        0x24F07B6D
    Field:
      Length: 0x0002
      Type:   0x01
      Data:
        2F 00
    Field:
      Length: 0x0004
      Type:   0x03
      Data:
        21 00 00 20                            /
    Field:
      Length: 0x0007
      Type:   0x05
      Data:
        66 6F 6C 64 65 72 00                   folder

Record for database 0:
  Database version: 0x01
  Record handle:    0x0002
  Unique ID:        0x00000007
    Field:
      Length: 0x0007
      Type:   0x01
      Data:
        2F 68 6F 6D 65 2F 00                   /home/
    Field:
      Length: 0x0004
      Type:   0x03
      Data:
        31 00 00 20
    Field:
      Length: 0x0007
      Type:   0x05
      Data:
        66 6F 6C 64 65 72 00                   folder

Limits in the IPD Format

IPD files have a record length restriction of 128K bytes. To get around this limitation, we came up with a record structure that would separate the content into separate records each under the 128K limit.

The Unique ID values map to this record type mapping for the eight record types:

SESSION_TYPE = 1
TRACK_TYPE = 3
SPEAKER_TYPE = 5
SESSION_TRACK_TYPE = 7
TRACK_SESSION_TYPE = 9
KEYNOTE_LIST_TYPE = 10 (0A)
DAY_SESSION_TYPE = 11 (0B)
SESSION_SPEAKER_TYPE = 13 (0D)
This is how the records appear in the file ( fields omitted):
Number of databases: 1
Database[ 0 ] = BBConferenceGuideData
Record for database 0:
  Database version: 0x01
  Record handle:    0x0001
  Unique ID:        0x00000001

Record for database 0:
  Database version: 0x01
  Record handle:    0x0008
  Unique ID:        0x0000000D

What is the relationship to a Session record as defined in this structure versus an instance of a Session record from a database? We used the fields of the IPD record for each occurrence of a session.

Writing the Length Before the Content

Since you need to output the record length before the record content, you must process the content first and then get the content length.

This is done by creating an intermediate file that contains the content. The method that closes the file must also return the content length.

After writing the content length, append the intermediate file content to the final IPD file.

This also applies to the content in a record, such as an image that needs to be processed to determine the length of the image content. There is some extra overhead in processing content twice but it is only done while creating the file where overhead doesn’t impact much.

Little Endian Hex Values

Lengths are stored in little endian format. This means that the low-order byte of the number is stored at the lowest address, and the high-order byte at the highest address (the word is stored ‘little-end-first’).

The little-endian format applies to the following fields:

record length
db version
db handle
record type.

The following conversion is used to get the byte values:

//Size as little-endian hex value
byte b1 = Convert.ToByte(size&Byte.MaxValue);
byte b2 = Convert.ToByte(size>>8&Byte.MaxValue);
byte b3 = Convert.ToByte(size>>16&Byte.MaxValue);
byte b4 = Convert.ToByte(size>>24&Byte.MaxValue);

After conversion, each byte is written in order:

loader.write(b1);
loader.write(b2);
loader.write(b3);
loader.write(b4);

Using the Simulator for Backup/Restore

Prior to BlackBerry Java Development Environment (JDE) v4.0, simulator testing configuration for backup/restore required the installation of a serial loopback driver to be defined for a port (say PORT5) as well as few other steps.

With BlackBerry JDE v4.0, the BlackBerry Device Simulator provides a direct and convenient way for backup/restore testing:

Launch the BlackBerry Device Simulator
Select the Simulate menu
Check the USB Cable Connected.

Application Handling the Restore

Performing Backup/Restore involves the BlackBerry Synchronization API. The Synchronization API provides the following interfaces that need to be implemented:

SyncConverter

Converts data between SyncObject-format on the device and a serialized format required on the desktop.

SyncCollection

A collection of synchronization objects.

SyncObject

An object that can be backed up and restored to the user’s computer.

SyncConverter and SyncCollection Manager

SyncConverter and SyncCollection are interfaces, so we chose to implement the interfaces in a DataStoreSyncManager class.

DataStoreSyncManager implements SyncCollection

Since a SyncCollection is a collection of SyncObject used for backup/restore and synchronization, we need to provide implementations for the methods defined in the SyncCollection interface.

Notable methods include:

beginTransaction()

Starts a transaction which is required to execute synchronizations that involve a large number of data records

endTransaction()

Ends a transaction which is required to execute synchronizations that involve a large number of data records

getSyncConverter()

Returns the instance of the DataStoreSyncManager

getSyncName()

Returns the database name

getObjectCount()

Returns the number of sync objects

getSyncObjects()

Returns a SyncObject[]

getSyncObject(int uid)

Returns a SyncObject by UID

getSyncVersion()

Returns 1

removeAllSyncObjects()

Clears the DataStore object and returns true

addSyncObject()

Adds a SyncObject to the DataStore by using the UID to identify the data type, and using the DataStore methods for the appropriate data type (e.g. setSessionSpeaker() method)

Since we are performing a bulk-load and not a synchronization, we can provide minimal implementations as shown below for some of the methods. For a backup/restore, we can return false; for Synchronization you must provide a full implementation.

isSyncObectDirty () {return false; }
removeSyncObject () { return false; }
setSyncObjectDirty() {}
clearSyncObjectDirty()  {}

DataStoreSyncManager implements SynchConverter

This class implemented the convert(..) method that Extracts a SyncObject from the synchronization data and converts a SyncObject into synchronization data.

DataStoreSyncObject implements SyncObject

The application needs to have a class defined that represents the structure of the .IPD database records. For our example, eight record types have been defined in our database, so we effectively have eight instances of SyncObjects.

A SyncObject must have a unique ID, which is a 32-bit value that is contant for the lifetime of the object. For this UID we implement a getUID() method that returns a value for one of the record types. For example, SESSION_TYPE = 1 as defined above.

PersistentStore contains Persistable objects

Since we’ve gone to the trouble of getting the data to the device, we now have to save it to the PersistentStore. To do so, implement a class that extends the Persistable interface. Again, the structure of the DataStore class needs to reflect the structure of the data contained in the .IPD file.

In our example, we have eight different data records defined so we will need a DataStore class capable of managing those eight different data types.

We chose to have byte[][] for Session, Track and Speaker data as the records contained character data.

For SessionTrackMap, TrackSessionMap, DaySessionMap, SessionSpeakerMap we chose int[][] since we had relationships expressed as integer index values.

For KeyNoteSessionList we chose int[] as we only had the Session index value in the list. So we have a DataStore structure on the device that supports the .IPD format.

The implementation of methods to access the particular records and record relationships is up to the designer of the application. In our case, we designed the data to support the application UI so that we had methods for Sessions, SessionSpeakers, KeyNoteSpeakers, Tracks, etc.

Sample Code

Loader.cs
Contains the data retrieval and data output code.

namespace BulkLoader {
  /// <summary>
  /// Summary description for Class1.
  /// </summary>
  class Loader {
    // Constants for record types - session shown here
    public static int SESSION_NUMBER_TYPE = 0;
    public static int SESSION_TYPE = 1;

    public void doLoading() {
      // Create output file and content
      byte dbVersion = 1;
      short dbHandle = 0;

      // Final output written to Loader1
      Loader loader1 = new Loader();
      loader1.createFile("BulkLoad-Output.ipd");

      // Prefix data for every IPD file
      loader1.loadFile("BulkLoad-Prefix1.ipd");

      // Data access provided by DBReader
      DBReader rdr = new DBReader();

      // Intermediate data written to BulkLoad-Data.ipd files
      Loader loader = new Loader();
      loader.createFile("BulkLoad-Data.ipd");
      dbHandle++;

      // An ArrayList of SessionInfo objects
      // turned into SessionContent
      ArrayList list = rdr.GetSessions();
      int listSize = list.Count;
      SessionContent sessions = new SessionContent(listSize);
      for (int i=0;i<listSize;i++) {
        SessionInfo info = (SessionInfo) list[i];
        Session data = new Session(info);
        sessions.AddMember(i, data);
      }

      if (listSize>0) {
        sessions.WriteInfo(loader);
      }

      // Get the size so far and output
      int bytes = loader.closeFile();
      OutputData(loader1, bytes, dbVersion, dbHandle, SESSION_TYPE);
    }

    /**
     * Create output from the data
     */
    private void OutputData(Loader loader, int bytes,
      byte dbVersion, short dbHandle, int recordType) {

    // Database type
    short dbase = 0;
    loader.Write(dbase);

    // Add the size of the Prefix2 portion which is 7 bytes
    bytes+=7;

    // Write the size as little-endian hex values
    byte b1 = Convert.ToByte(bytes&Byte.MaxValue);
    byte b2 = Convert.ToByte((bytes>>8)&Byte.MaxValue);
    byte b3 = Convert.ToByte((bytes>>16)&Byte.MaxValue);
    byte b4 = Convert.ToByte((bytes>>24)&Byte.MaxValue);
    loader.Write(b1);
    loader.Write(b2);
    loader.Write(b3);
    loader.Write(b4);

    // DBVersion - append as little-endian hex values
    b1 = Convert.ToByte(dbVersion&Byte.MaxValue);
    loader.Write(b1);

    // DBHandle - append as little-endian hex values
    b1 = Convert.ToByte(dbHandle&Byte.MaxValue);
    b2 = Convert.ToByte((dbHandle>>8)&Byte.MaxValue);
    loader.Write(b1);
    loader.Write(b2);

    // RecordType - append as little-endian hex values
    b1 = Convert.ToByte(recordType&Byte.MaxValue);
    b2 = Convert.ToByte((recordType>>8)&Byte.MaxValue);
    b3 = Convert.ToByte((recordType>>16)&Byte.MaxValue);
    b4 = Convert.ToByte((recordType>>24)&Byte.MaxValue);
    loader.Write(b1);
    loader.Write(b2);
    loader.Write(b3);
    loader.Write(b4);

    // Data Content
    loader.loadFile("BulkLoad-Data.ipd");
  }
}

Content.cs
Contains definitions for the IPD file content: Session, Tracks, Speakers.
Session items are shown here.

using System;
using System.Collections;
using System.IO;
using System.Text;

namespace BulkLoader {

  /**
   * MapContent defines the Record Group: Length, Type,
   * length value that prefixes every record.
   */
  public abstract class MapContent {
    short length;
    protected byte type;     // Record type
    protected int[] members; // Used for Record length

    public MapContent() {}


    public void SetContentLength(int len) {
      length = Convert.ToInt16(len);
    }

    public void AddContentMember(int pos, int item) {
      if (members != null) {
        members[pos] = item;
      }
    }

    /// <summary>
    /// Our objects need to know how to write themselves
    /// </summary>
    /// <param name="loader"></param>
    protected void WriteContentMember(Loader loader) {
      loader.Write(length);
      loader.Write(type);
      int j = members.Length;

      for (int i=0; i<j;i++) {
        loader.Write((int) members.GetValue(i));
      }
    }
  }

  /**
   * SessionContent, Session and SessionInfo manage Sessions.
   */
  public class SessionContent : MapContent {
    Session[] info;

    /// <summary>
    /// Ctor defines the size of Session[]
    /// </summary>
    public SessionContent(int num) {
      type = Convert.ToByte(Loader.SESSION_NUMBER_TYPE);
      info = new Session[num];
      members = new int[1];
    }


    /// <summary>
    /// Add an item to the list
    /// </summary>
    public void AddMember(int pos, Session data) {
      info[pos] = data;
    }

    /// <summary>
    /// Our objects need to know how to write themselves
    /// </summary>
    /// <param name="loader"></param>
    public void WriteInfo(Loader loader) {
      // Content setup
      SetContentLength(4);
      AddContentMember(0, info.Length);
      WriteContentMember(loader);
      IEnumerator e = info.GetEnumerator();

      while (e.MoveNext()) {
        Session data = (Session) e.Current;
        data.WriteInfo(loader);
      }
    }
  }

  /// <summary>
  /// Session provides the infoType, length and SessionInfo
  /// </summary>
  public class Session {
    short infoLength;
    byte infoType = Convert.ToByte(Loader.SESSION_TYPE);
    SessionInfo info;

    public Session(SessionInfo data) {
      info = data;
      infoLength = Convert.ToInt16(info.Length());
    }

    /// <summary>
    /// Our objects need to know how to write themselves
    /// </summary>
    /// <param name="loader"></param>
    public void WriteInfo(Loader loader) {
      loader.Write(infoLength);
      loader.Write(infoType);
      info.WriteInfo(loader);
    }
  }

  /// <summary>
  /// SessionInfo contains the details of a session
  /// </summary>
  public class SessionInfo {
    long start;
    long duration;
    byte keynote;
    string info;

    public SessionInfo(long start, long duration,
        byte keynote, string info) {
      this.start = start;
      this.duration = duration;
      this.keynote = keynote;
      this.info = info;
    }

    /// <summary>
    /// The length is the size of the SessionInfo object
    /// </summary>
    /// <returns></returns>
    public short Length() {
      return Convert.ToInt16(8 + 8 + 1 + info.Length);
    }

    /// <summary>
    /// Our objects need to know how to write themselves
    /// </summary>
    /// <param name="loader"></param>
    public void WriteInfo(Loader loader) {
      loader.Write(start);
      loader.Write(duration);
      loader.Write(keynote);
      loader.Write(info);
    }
  }

Loader.cs
This class is responsible for output of data using a BinaryWriter.
Methods exist to write various data types: char, int, byte, short, long, String.

using System;
using System.IO;
namespace BulkLoader {

/// <summary>
/// Loader manages Binary files.
/// </summary>
public class Loader {
  // Our writer
  BinaryWriter writer = null;

  // Output length counter
  int bytes = 0;

  public Loader() {}

  /// <summary>
  /// Create BinaryWriter from a filename
  /// </summary>
  /// <param name="fileName"></param>
  public void createFile(String fileName) {
    try {
      writer =  new BinaryWriter(File.Open
        (fileName, FileMode.Create));
    } catch (Exception e) {
      Console.Out.WriteLine("Exception: " + e.Message);
    }
  }

  /// <summary>
  /// Load a file
  /// </summary>
  /// <param name="fileName"></param>
  public void loadFile(String fileName) {
    if (fileName.Length==0)  {
      return;
    }

    loadFile(fileName, false);
  }

  /// <summary>
  /// Load a file, counting the bytes of the file
  /// </summary>
  /// <param name="fileName"></param>
  /// <param name="countBytes"></param>
  public void loadFile(String fileName, bool countBytes) {
    if (fileName.Length==0)  {
      return;
    }

    BinaryReader rdr = null;

    Try {
      rdr =  new BinaryReader(File.OpenRead(fileName));
      byte b;

      for (;;){
        b = rdr.ReadByte();
        writer.Write(b);

        if (countBytes){
          bytes++;
        }
      }
    } catch (EndOfStreamException) {
      ; // do nothing
    }
    finally {
      if (rdr != null) rdr.Close();
    }
  }

  /// <summary>
  /// Close an intermediate file and return the file size
  /// </summary>
  /// <returns>file size</returns>
  public int closeFile() {
    if (writer != null) {
      try {
        writer.Close();
      } catch (Exception e) {
        Console.Out.WriteLine("Exception: " + e.Message);
      }
    }
    return bytes;
  }

  /// <summary>
  /// Write a char - 1 byte
  /// </summary>
  /// <param name="val"></param>
  public void Write(char val) {
    writer.Write(val);
    bytes+=1;
  }

  /// <summary>
  /// Write an int - 4 bytes
  /// </summary>
  /// <param name="val"></param>
  public void Write(int val) {
    writer.Write(val);
    bytes+=4;
  }

  /// <summary>
  /// Write a short - 2 bytes
  /// </summary>
  /// <param name="val"></param>
  public void Write(short val) {
    writer.Write(val);
    bytes+=2;
  }

  /// <summary>
  /// Write a byte - 1 byte
  /// </summary>
  /// <param name="val"></param>
  public void Write(byte val) {
    writer.Write(val);
    bytes+=1;
  }

  /// <summary>
  /// Write a long - 8 bytes
  /// </summary>
  /// <param name="val"></param>
  public void Write(long val) {
    writer.Write(val);
    bytes+=8;
  }

  /// <summary>
  /// Write a String - string length
  /// </summary>
  /// <param name="val"></param>
  public void Write(string val) {
    Write(val, val.Length);
  }

  /// <summary>
  /// Write a String - string length
  /// </summary>
  /// <param name="val", "len" ></param>
  public void Write(string val, int len) {
    StringReader rdr = new StringReader(val);

    for (int i=0;i<len;i++) {
      int c = rdr.Read();
      writer.Write(Convert.ToByte(c));
      bytes++;
    }
  }

Please email your comments, suggestions and editorial submissions to

Global Navigation

Site Navigation

Developers

Local Navigation

Garry Seifried, Research In Motion

The Need for Speed

BlackBerry Desktop Manager Backup/Restore

Using the IPD Format for Bulk Loads

Parse that IPD

Limits in the IPD Format

Writing the Length Before the Content

Little Endian Hex Values

Using the Simulator for Backup/Restore

Application Handling the Restore

DataStoreSyncManager implements SyncCollection

DataStoreSyncManager implements SynchConverter

DataStoreSyncObject implements SyncObject

PersistentStore contains Persistable objects

Sample Code

Related Links

Shortcut Navigation

BlackBerry

Global Navigation

Site Navigation

Developers

Local Navigation

IPD File Format

Garry Seifried, Research In Motion

The Need for Speed

BlackBerry Desktop Manager Backup/Restore

Using the IPD Format for Bulk Loads

Parse that IPD

Limits in the IPD Format

Writing the Length Before the Content

Little Endian Hex Values

Using the Simulator for Backup/Restore

Application Handling the Restore

DataStoreSyncManager implements SyncCollection

DataStoreSyncManager implements SynchConverter

DataStoreSyncObject implements SyncObject

PersistentStore contains Persistable objects

Sample Code

Related Links

Shortcut Navigation