XP Forum in JHB

February 8, 2008 at 10:18 am | In Persistence | 4 Comments
Tags: , ,

Yesterday we had an interesting event in Wits university  -  a presentation about implementing XP in customer contracts. Looks like making a contract can be a real challenge, but that’s the only way to go. The suggested model for XP: fixed cost contract. The idea is to disclose your costs and expected profit to the customer, set a target time and define the scope on a very-high level. The high-level definition of the scope allows both parties to negotiate on the specific module (requires lots of trust).  Disclosing your costs&profits is supposed to improve the trust relationship. The negotiation on modules is certainly something not clear enough, especially if the client in the beginning has no idea of what they want (which is often the case).

Otherwise, I feel quite scared, – probably writing software on demand is very risky. I’m rather looking at SaaS.

Make Your Program Write Code For You

December 7, 2007 at 4:55 pm | In Code Generation, PLSQL, StringTemplate | 1 Comment

I’m back to writing PLSQL and it’s pain as always. Recently I was dealing with the issue of synchronizing table data between 2 databases and needed to provide a comparison report about the values that do not match. Well, that’s easy: select rows locally and over the dblink and compare the data. But… what if there are 40 different tables to compare and hundreds of selection criteria to get the data? And no, I do not have time to type it all.

So I needed some sort of automation. Of course PLSQL provides something:

- I can use one procedure to populate collections using the selection criteria;

- another procedure can check the same primary keys exist in the local and remote collection;

- and the last one can browse through a pair of collection rows to compare that all fields have the same values.

That sounds reasonable and short. But each collection should represent a row in a table (something like MyTable%rowtype), that means that collections for different tables will be completely different. So how to make it generic (especially in the case when all fields are to be compared)?

I did not found any elegant answer in PLSQL and typing code for all table variations seemed to be impossible. I had to look for a better solution and I decided to use code patterns and replace pieces that do not match. Thanks to StringTemplate the work turned to be really easy. StringTemplate allows separation of your code from the template code. The benefit here is that the template looks semantically the same as the result, so it is very easy to create, correct and maintain. In the same time the code stays clean and very simple. Though StringTemplate has a rich API, I basically mastered only a small piece of it in 5 minutes and that was far enough to do the entire job.

For those who are still interested, here is an example.

Example

Imagine we have a list of tables (Table1, Table2, Table3 for instance) in 2 databases connected over a db link; the tables can have any amount of fields with different names. Our goal is to compare all the values in matching tables between 2 databases, i.e. first we will check that all IDs from local database table are present in the remote table and then we will compare that the rows with the same ID have all other fields with the same values.

To make it shorter let’s imagine that the code populating data into Table1%rowtype (Table2%rowtype, Table3%rowtype etc) collections is already generated. So we will need the following templates (I’ve started by writing this code manually for 1 table to be sure about the template syntax):

Templates

In all the templates below $$ is used to distinguish a replaceable attribute.

compare.st – Top-level checking function:

———————————————————————

procedure $compare_procedure_name$($local_table$ $table_type$,
$remote_table$ $table_type$) is

rec_id integer;
column varchar2(30);
begin
for i in 1 .. $local_table$.count
loop
rec_id := $find_procedure_name$($remote_table$, $local_table$(i).id);
 if rec_id < 0 then
log_error(‘Missing $entity_name$: ‘|| $local_table$(i).id);
error_count := error_count + 1;
else
if not $compare_function_name$($local_table$(i), $remote_table$(i), column) then
 log_error(‘Different values observed in $entity_name$.’ || column || ‘:’ ||$local_table$(i).id);
error_count := error_count + 1;
end if;
end if;
end loop;
end;

find.st Function to locate a matching record in a collection

function $find_function_name$($generic_table$ $table_type$, $entity_id$ varchar2)
return integer
is
begin
for i in 1 .. $generic_table$.count
loop
if $generic_table$(i).id = $entity_id$ then
return i;
end if;
 end loop;
 return -1;
end $find_function_name$;

compare_row.st Function to compare a selected row

function $compare_function_name$(local_entities edh_core.$entity_name$%rowtype,
remote_entities edh_core.$entity_name$%rowtype,
column out varchar2) return boolean
is
begin
$compare_declarations$
return true;
end $compare_function_name$;

compare_field.st A piece of code to compare a single field value:

if local_entities.$field_name$ <> remote_entities.$field_name$ then
column := ‘$field_name$’;
return false;
end if;

Filling the templates

Now that the templates are ready, we will only need to apply them for all tables and fields and write to a file (I used a package template too, which you can easily create yourself).

General way to call a template:

// Templates are stored in templatesFolder
StringTemplateGroup group = new StringTemplateGroup(“GroupName”,
“templatesFolder”, DefaultTemplateLexer.class);

You can make group variable a class field as it will be used in all the template-handling procedures.

// compare is the name of template file without “.st” extension
StringTemplate procedure = group.getInstanceOf(“compare”);

Now we need to set all the attributes used in the template

// Compare procedure name can be generated based on table name and selection criteria: compare_table1_description, compare_table2_time etc
procedure.setAttribute(“compare_procedure_name “, “compare_” + tableName + “_” + criteria);
procedure.setAttribute(“local_table”, tableName + “_local”);
procedure.setAttribute(“table_type”, “t_” + tableName);
procedure.setAttribute(“remote_table”, tableName + “_remote”);
procedure.setAttribute(“find_procedure_name”,“compare_” + tableName + “_” + criteria);
procedure.setAttribute(“entity”, tableName + “_remote”);
// You can type the result or return it as the function result
System.out.println(procedure.toString());
return procedure.toString();

In an analogous way you can get the result from find.st template.

A bit more work is to use compare_row and compare_field templates, as compare_field template must be used inside compare_row as many times as there are fields in a row:

public String generateCompare() {
StringTemplate procedure = group.getInstanceOf(“compare_row”);
procedure.setAttribute(“compare_function_name”, getCompareProcedureName());
procedure.setAttribute(“entity_name”, tableName);
procedure.setAttribute(“compare_declarations”, getCompareTemplate(tableName));
System.out.println(procedure.toString());
return procedure.toString();
}

public String getCompareTemplate(String tableName) {
if (entityName == “Table1″) {
return getCompareTemplate(new String[]{“Table1Field1″, ” Table1Field2″, ” Table1Field3″});
} else if (entityName == “Table2″){
return getCompareTemplate(new String[]{“Table2Field1″, ” Table2Field2″});
} else if (

///………

}
return “”;
}

private String getCompareTemplate(String[] fields) {
String compares = “”;
for (int i = 0; i < fields.length; i++){
StringTemplate template = group.getInstanceOf(“compare_field”);
template.setAttribute(“field_name”, fields[i]);
compares += template.toString() + “\r\n”;
}
return compares;
}

Now instead of typing all these procedures for all the tables, you can just specify which table you are interested in, and then pass the columns array to getCompareTemplate.

In this example I do not show how to put all the procedures and functions together in a PLSQL package file and how to fill table rows collections. However, you will be able to add those simple templates yourself and write the result into *.pck file.

In general StringTemplate can save you lots of copy-pasting work and help to tame PLSQL, html, xml and other code.

Using a Database Over a Webservice

May 28, 2007 at 6:24 pm | In .NET Remoting, Marshal By Value, Persistence, db4o | 11 Comments

Download source code – 780 kB

Introduction

RemotingService project was developed as a simple proof of concept (the idea taken
from Database as a Webservice). It is just one more way to realize a remote persistence.

In fact, putting any database on a web-server and providing a remote user interface to it, will realize the same idea from a user point of view. The interesting point of this solution is in the fact that no extra interface is developed, actually database interface is just transferred over the network.

This realization is possible due to the simple object-oriented interface provided by db4o database.

A possible implementation can be in providing personal web-based databases for remote
customers. Assuming that each user will use only one and his own database, this solution seems to be viable, providing the data is personally encrypted and unique database names are used.

Background

This article was inspired by Krispy Blog.

Solution Organization

RemotingXml consists of 4 projects:

  • Db4objects.Db4o: db4o open-source database. Please, refer to db4o.license.html for licensing information. The code was slightly modified, which will be discussed later.
  • Server: contains the code for a remote server. Server project is responsible for creating and starting remoting services.
  • RemotingExample: contains the code for creating persistent objects, sending them for serialization to the server and requesting the server for objects from the database.
  • RemotingClasses project is a library containing the definitions for classes to be
    persisted.

Remoting Classes

RemotingClasses project contains only one test class, which will be used on the client to be stored on the server. The class must have Serializable attribute to be transferrable over the network:

[Serializable]public class TestValue

{

…

}

db4o Database Amendments

In order to send the persistent classes from the client to the server we will need a Marshal-By-Reference object. This can be any class providing access to an ObjectContainer (can be modified ObjectContainer itself). The requirements to the transport object:

  • the transport object should open a database file on request (filename should be provided somehow);
  • the transport object must have zero-parameters constructor.

With the first implementation I tried to modify the ObjectContainer interface and the implementing classes. However it resulted in quite a lot of modifications as the hierarchy is
deep. Moreover, I could not find a logical place to pass a filename to the object container (default implementation requires it in the constructor, and our rules need a parameterless constructor).

In the implementation, provided in this example, I use a version of Db4oFactory class, which can work over the network and can provide ObjectContainers for client requests:

using System;
using System.IO;
using Db4objects.Db4o;
using Db4objects.Db4o.Config;
using Db4objects.Db4o.Ext;
using Db4objects.Db4o.Foundation;
using Db4objects.Db4o.Foundation.Network;
using Db4objects.Db4o.Internal;
using Db4objects.Db4o.Internal.CS;
using Db4objects.Db4o.Reflect;

namespace Db4objects.Db4o
{
    ///
    /// This class simply duplicates Db4oFactory with
    /// instance methods, thus allowing Marshalling to
    /// access them
    ///
	public class Db4oRemoteFactory: MarshalByRefObject
	{
		internal static readonly Config4Impl i_config = new Config4Impl();

		public  IConfiguration NewConfiguration()
		{
            		return Db4oFactory.NewConfiguration();
		}

		public IConfiguration CloneConfiguration()
		{
			return (Config4Impl)((IDeepClone)Db4oFactory.Configure()).DeepClone(null);
		}

		public  IObjectContainer OpenFile(string databaseFileName)
		{
			return OpenFile(CloneConfiguration(), databaseFileName);
		}

		public IObjectContainer OpenFile(IConfiguration config, string databaseFileName
			)
		{
			return ObjectContainerFactory.OpenObjectContainer(config, databaseFileName);
		}

	}
}

This implementation is convenient for clients working separately. However, it is not suitable for concurrent collaborative work on the same database file.

Another important change to the db4o sources: some of the internal classes representing persistent objects were marked [Serializable] to be suitable for over-network transmission.

Server

The Server project is presented by a single Program class, which role is to start Db4oRemoteFactory service:

    class Program
    {
        static void Main(string[] args)
        {
            // Setting up tcp channel
            BinaryClientFormatterSinkProvider clientProvider = null;

            BinaryServerFormatterSinkProvider serverProvider =
                new BinaryServerFormatterSinkProvider();

            serverProvider.TypeFilterLevel =
                System.Runtime.Serialization.Formatters.TypeFilterLevel.Full;

            IDictionary props = new Hashtable();

            /*
             * Client and server must use the SAME port
             * */
            props["port"] = 65101;
            props["typeFilterLevel"] = System.Runtime.Serialization.Formatters.TypeFilterLevel.Full;

            TcpChannel chan =
                new TcpChannel(props, clientProvider, serverProvider);

            ChannelServices.RegisterChannel(chan);

            Type TestFactory = Type.GetType("Db4objects.Db4o.Db4oRemoteFactory, Db4objects.Db4o");
            RemotingConfiguration.RegisterWellKnownServiceType(
                    TestFactory,
                    "TestFactoryEndPoint",
                    WellKnownObjectMode.Singleton
                );
            Console.WriteLine("TestFactory is ready.");

            // Keep the server running until the user presses
            // the Enter key.
            Console.WriteLine("Services are running. Press Enter to end...");
            Console.ReadLine();
        }
        // end Main

    }

Client

Client project contains Program class.

In the constructor a connection to the Db4oRemoteFactory service is established:

      public Program()
        {
            string url;

            // Setup a client channel to our services.
            url = @"tcp://LocalHost:65101/";
            BinaryClientFormatterSinkProvider clientProvider =
                new BinaryClientFormatterSinkProvider();

            BinaryServerFormatterSinkProvider serverProvider =
                new BinaryServerFormatterSinkProvider();

            serverProvider.TypeFilterLevel =
                System.Runtime.Serialization.Formatters.TypeFilterLevel.Full;

            IDictionary props = new Hashtable();
            props["port"] = 0;
            props["name"] = System.Guid.NewGuid().ToString();
            props["typeFilterLevel"] = System.Runtime.Serialization.Formatters.TypeFilterLevel.Full;

            TcpChannel chan =
                new TcpChannel(props, clientProvider, serverProvider);
            ChannelServices.RegisterChannel(chan);

            // Set an access to the remote proxy
            factory = (Db4oRemoteFactory)RemotingServices.Connect(
                typeof(Db4oRemoteFactory), url + "TestFactoryEndPoint"
                );
        }
        // end Program

The RunTest method creates a test object, sends it to the server using factory and retrieves it from the database on the server:

        private void RunTest()
        {
            // Remote persistent class
            TestValue testValue;
            testValue = new TestValue();
            // Modify local TestValue object
            testValue.ChangeData("Test value");
            // Pass testValue to the server through testServer
            // and store the testValue object
            IObjectContainer testServer = factory.OpenFile("test.db4o");
            testServer.Set(testValue);
            // Test what is stored to the database
            IList result = testServer.Get(typeof(TestValue));
            //TestValue test;
            foreach (TestValue test in result) {
                System.Console.WriteLine(test);
            }
        }
        // end RunTest

Points of Interest

  • The most interesting TO-DO on this project is, of course, its performance. How much information is transferred over the network for a single request? How is it comparable to the client-server version of db4o?
  • Are there any advantages over client-server version of db4o?
  • This implementation can only work over tcp-channel. Is it possible to realize the same functionality over html-channel?
  • How to realize concurrency control?

History

2007-05-28 First version.

Marshal By Value for Persistence

May 25, 2007 at 9:21 am | In .NET Remoting, Marshal By Value, Persistence, Xml | Leave a Comment

I was recently writing about how to use Marshal By Value with db4o and was very surprised how little information could I found on this actually simple subject.

So, to contribute to the global network – here is another Marshal By Value example, this time with xml.

Download RemotingXml.zip – 11.3 KB

Introduction

RemotingXml project was
developed with 2 purposes in mind:

  • show how to use marshal
    by value technology;
  • show how to implement a
    remote storage.

RemotingXml project consists
of a client and a server. The client creates and works with objects and sends them
to the server using marshal-by-value technology to be stored to XML files. The client
is also able to request the objects from the XML files.

This project might be helpful to those, who want to implement some kind of remote
persistence solution. In this example, XML files are used as a very simple and comprehensive
example, however the project can be easily used as a template to build a database
or file-based remoting storage solution.

Background

Though marshal-by-reference
remoting is widely covered on the net, marshal-by-value still tends to create some
confusion. This project is created to clear up this confusion and show an example
of how marshal-by-value objects can be used.

Solution Organization

RemotingXml consists of
3 projects:

  • Server: contains the code
    for a remote server. Server project is responsible for creating and starting remoting
    services.
  • Client: contains the code
    for creating persistent classes, sending them for serialization to the server and
    requesting the server for objects from the XML files.
  • PersistentClasses project
    is a library containing the definitions for classes used on the client and on the
    server. These include: class to be serialized, list wrapper for the serialized class
    and marshal-by-reference class used for persistent classes transport.

Persistent Classes

Persistent classes are the
classes that will be created on the client and serialized on the server. Any class
can be persisted if it either has Serializable attribute
or implements ISerializable interface. In this example,
House and Address classes are used.

Address class is saved as
a field in the House class:

[Serializable]
public class House
{
private Address _address;

private string _owner;
….
}

In order to save a list of houses we will need to create a special List implementation:

[Serializable]

[XmlInclude(typeof(House)), XmlInclude(typeof(Address))]

public class HouseList : List

{

}

This implementation is required for 2 purposes:
An object passed to a remote server should be serializable
The object need to provide the details of included classes for correct serialization.

Transport Class

In order to send the persistent
classes from the client to the server we will need a Marshal-By-Reference object:

    public class XmlHandler: MarshalByRefObject

    {

        private const string Extension = ".xml";

        // list of remote objects

        private IList _persistentObjects;        public IList PersistentObjects

        {

            get { return _persistentObjects; }

            set { _persistentObjects = value; }

        }

        // end PersistentObject

public void StoreData()

        {

            // Serialize object list to xml

            System.Console.WriteLine("persistentObjects " + _persistentObjects.Count);

            if (_persistentObjects.Count > 0)

            {

                // define the object type

                object persistentObject = _persistentObjects[0];

                XmlSerializer serializer = new XmlSerializer(_persistentObjects.GetType());

// delete the old file and create a new one

                string filename = persistentObject.GetType().Name + Extension;

                File.Delete(filename);

                StreamWriter xmlWriter = new StreamWriter(filename);

                // serialize object list

                serializer.Serialize(xmlWriter, _persistentObjects);

                xmlWriter.Close();

            }

        }

        // end StoreData

public IList RetrieveData(Type arrayType, Type objectType)

        {

            // Compose the file name

            string filename = objectType.Name + Extension;

            // deserialize

            XmlSerializer serializer = new XmlSerializer(arrayType);

            FileStream xmlFileStream = new FileStream(filename, FileMode.Open);

            IList objects = (IList)serializer.Deserialize(xmlFileStream);

            xmlFileStream.Close();

            return objects;

        }

        // end RetrieveData

    }

XmlHandler
holds a reference to the persistent objects array in _persistentObjects variable.
This variable is passed from the client.

As you can see there are methods for serializing and deselializing objects. These
methods are executed on the server.

Server

The Server project is presented
by a single XmlServer class, which role is to start XmlHandler service

    class XmlServer

    {

        static void Main(string[] args)

        {

            // Setting up http channel

            HttpChannel channel = new HttpChannel(65101);

            ChannelServices.RegisterChannel(channel);            Type XmlHandler = Type.GetType("RemotingXml.PersistentClasses.XmlHandler, PersistentClasses");

            RemotingConfiguration.RegisterWellKnownServiceType(

                    XmlHandler,

                    "XmlHandlerEndPoint",

                    WellKnownObjectMode.Singleton

                );

            Console.WriteLine("XmlHandler is ready.");

// Keep the server running until the user presses

            // the Enter key.

            Console.WriteLine("Services are running. Press Enter to end...");

            Console.ReadLine();

        }

        // end Main

    }

Client

Client project contains Program class.
In the constructor a connection to the XmlHandler service is established:

        public Program()

        {

            string url;            // Setup a client channel to our services.

            HttpChannel channel = new HttpChannel(0);

            url = @"http://LocalHost:65101/";

// Register the channel

            ChannelServices.RegisterChannel(channel, false);

// Set an access to the remote proxy

            xmlHandler = (XmlHandler)RemotingServices.Connect(

                typeof(PersistentClasses.XmlHandler), url + "XmlHandlerEndPoint"

                );

        }

        // end Program

The RunTest method creates
a list of House objects, sends them to the server using XmlHandler and retrieves
the objects from the XML files on the server:

        private void RunTest()

        {

            // an object to be stored remotely

            House house = new House(new Address("East London", "266 Oxford Street"), "Derick Hopkins");

            // Create a list to hold House instances

            HouseList persistentObjects = new HouseList();

            persistentObjects.Add(house);

            // another House for a list

            house = new House(new Address("East London", "23 Stewart Drive"), "Om Henderson");

            persistentObjects.Add(house);

            // Pass House list to the server using XmlHandler

            xmlHandler.PersistentObjects = (IList)persistentObjects;

            // Call method on the server to store houses to an XML file

            xmlHandler.StoreData();

            // Test what is stored to the database

            IList result = xmlHandler.RetrieveData(typeof(HouseList), typeof(House));

            foreach (object obj in result)

            {

                System.Console.WriteLine(obj);

            }

        }

        // end RunTest

Points of Interest

Creating this project I
had an idea in mind to make it completely generic by using generics for persistent
object collections. However, the idea was not realized due to the fact that Soap
serializer does not support generics and explicit using of binary serializers on
the client and the server still does not solve the problem.

History

2007-05-25 First version.

db4o with JasperReports

April 30, 2007 at 11:37 am | In JasperReports, db4o, reporting | 5 Comments

I’ve just recently published a short tutorial on db4o + JasperReports here:

Reporting With JasperReports

IMO JasperReports are very suitable for db4o – did not notice any problems. Want to try it with servlets when I will have some spare time. Hopefully no classloader problems as with BIRT

Why do I write?

April 18, 2007 at 9:27 am | In technical writer | Leave a Comment

One of my colleagues asked me recently, why having all the programmer’s experience, do I prefer to work as a technical writer. Well, actually I do quite a lot of programming, I’d say 80% of my working time.

But getting back to the question: isn’t programming an expression of your ideas in a structured and defined way? Nowadays, it is also especially important that the code is beautiful, clean and understandable. So what is the difference from the technical writing?

IMO, the difference is that when your code is done you know if it works, at least to some degree (bugs can pass unknoticed for ages) . When you write documentation – you have no idea if it works! You’ve written it up to some degree, as if you were explaining yourself how to do something. But you can’t even say if it works for yourself. You can only assume that the users will understand your doc and that will be enough for them. Actually you will never know until they will tell you.

That’s the challenge :-)

How to create a web-site starter kit (Java)

January 3, 2007 at 7:13 pm | In Servlet, StarterKit | Leave a Comment

What Is a Starter Kit?

Starter Kit is the easiest way to start using a new technology. Basically, it is a template project with a comprehensive documentation on where the project can be applied, which functionality is included and how to extend the template with your own functionality.

In this article we will go through a creation of your own StarterKit for a Servlet database application.

Project Structure

Create a simple java project in your development environment.

For example you can use the following structure:

src – directory for your source code packages

doc – documentation for the StarterKit

META-INF:

manifest.mf

WEB-INF:

            classes – compiled application classes

            lib – required libraries

            web.xml – configuration

Source Code

It is recommended to separate database logic from the front-end. For example, you can use com.yourpackage package for your servlet class source and com.yourpackage.data for a data-aware module.

In the servlet you can predefine some html code for the page layout, for example:

PrintWriter out = response.getWriter();

response.setContentType(“text/html”);

out .println(“<!DOCTYPE HTML PUBLIC \”-//W3C//DTD HTML 4.01 Transitional//EN\” \”http://www.w3.org/TR/html4/loose.dtd\”>”);

out.println(“<html>” );

out.println(“<body>”);

getDatabaseData();

out.println(“</body>”);

out.println(“</html>”);

getDatabaseData is a function, which will get data from the database module and publish it on the webpage. For a template project getDatabaseData() function should stay empty – this is the user responsibility to decide on the layout of the page, depending on his application requirements. To make this obvious – add an explicit comment inside the function template.

com.yourpackage.data package can be used for your database-handling module. The responsibilities of the module are:

- set up a database connection

- provide an easy API to get database data

- close the connection on application termination

You can use pooling and create connection pool in ServletContextListener#contextInitialized event handler. The pool will have to be destroyed in ServletContextListener#contextDestroyed. Otherwise you may want to set up a new connection for each session using sessionCreated and sessionDestroyed events of HttpSessionListener. Note that sessionDestroyed event is only fired if the session expires, so it may be a better idea to close the connection explicitly.

ServletContextListener and HttpSessionListener classes should be defined in web.xml if used.

Documentation

To make the template project a StarterKit we will need to add documentation. A general project description and goal can be placed into index.html in the root directory. This will ensure that the user can easily find the documentation when the StarterKit is deployed.

More specific documentation, explaining how to create a sample application using the StarterKit can be placed into the doc folder.

The ready StarterKit can be packaged into a war file for easy deployment.

Using Mobilink in a Production Environment

December 5, 2006 at 2:06 pm | In Mobilink, Synchronization | 16 Comments

Default Mobilink synchronization is set to the scenario when the data are downloaded once and stay on the mobile device as long as it is used. The changes from the remote database (the one on the device) are synchronized in upload sessions and the data from the central (consolidated) database are sent to the device in download sessions.

However this mode arises some problematic questions like:

  • how to handle multiple users on the same device? Should the data be downloaded for all of them?
  • How to backup existing data?
  • What to do to make the device failure and database corruption the less painful?

To solve these problems and make the whole procedure simple the following scenario can be used: the working data is downloaded to the mobile device at the beginning of a shift and uploaded to the central database at the end of a shift (logout). Data should not stay on the device between shifts.

This gives us several advantages:

  • the data changed at the device won’t be lost in a case of the mobile device hardware failure
  • the data will be uploaded from the mobile device in time (outdated data can produce unexpected conflicts in the central database)
  • uploading the data together with the logout ensures that only the logged person is responsible for the data uploaded from the device.

In such conditions it makes sense:

  • to start the day with the download synchronization (remote database is empty and filled with the data prepared on the consolidated side)
  • to perform full synchronizations during the day if necessary
  • to use upload synchronization at the end of a shift (logout), and truncate all the tables after the synchronization was successfully finished.

Download synchronization

In general case of continuous synchronization download synchronization should know, which rows to download to remote database (so that the same row is not downloaded if it already exists on remote side). Last_download parameter of download_cursor and download_delete_cursor event can be used to distinguish the rows that were updated since the last synchronization. Obviously in this case all the tables, which are supposed to be downloaded, should have a timestamp field showing the time, when the row was updated last. This might be quite undesirable, as it affects the database model.

The solution of cleaning remote database before downloading new data gives you a possibility to avoid this problem. On the other side it will have a performance impact of downloading the whole work database before each shift. So this solution can be used if the time available to prepare the remote database for the shift is enough for complete database download.

Upload synchronization

Mobilink upload implementation is based on the log file that ASA database can create, keeping track of all the changes done to it. Actually database log is a packed SQL command file. It can be easily translated into SQL command sequence using

dbtran [ options transaction-log [ SQL-file ]

For more information see “The Log Translation utility” from “Adaptive Server Anywhere Database Administration Guide”

Obviously the log gives a full and handy information about the changes occurred in the database. Before synchronizing mobilink client (dbmlsync) scans through the database log to collect the changes.

Dealing with mobile devices, which have limited storage resources, you might be interested, how to prevent log file from growing eternally. Here ASA provides a special option

dbmlsync -x

Which automatically truncates the log file after successful synchronization. Unfortunately truncating the log it creates its backup, which will still occupy some space and should be deleted intelligently when the updates from it are synchronized to the consolidated database.

You can use dbtran [logfile] –y NULL utility to find out start and end offsets. And you can use

select first progress from sys.syssync where progress is not null order by progress asc;

to find the lowest unsynchronized offset.

An alternative approach (and simpler one as well) is to truncate the log file after the data is successfully uploaded to the consolidated database. In this case we will also have to clean synchronization offsets from the remote database. The easiest way to do this is to delete synchronization subscription:

DROP SYNCHRONIZATION SUBSCRIPTION TO “MY_PUBLICATION” FOR “USER”;

Don’t forget to recreate it for the following synchronizations.

Note:

Information about mobilink users and their subscriptions is kept in sys.syssync table in remote database.

You will also have to delete synchronization offset from the consolidated database. The best place to do that is end_synchronization event handler. You can call the following procedure to fulfill the task:

CREATE OR REPLACE PROCEDURE End_Sync(ML_USER IN VARCHAR2,

SYNC_OK IN NUMBER) AS

BEGIN

DELETE FROM ML_SUBSCRIPTION WHERE USER_ID = (SELECT USER_ID FROM ML_USER WHERE NAME = ML_USER);

COMMIT;

END;


Truncating the log makes Mobilink see the database as unchanged. Even if the data is different from the consolidated database you can’t synchronize these changes using Mobilink technology. So it is very important to check that the synchronization was successful before truncating the log.


rc = pb_run_dbmlsync(exe_name, is_publication_name, is_mluser, is_mlpassword, connect_string, other_arguments, is_sync_class_name )

rc value should be 0 for successful synchronization.
//check if it finished without mistakes
if (IsNull(rc) = true) then

return -100
elseif ( rc = 0) then
// sometimes Status is wrong but sync exits without error
if (Pos(w_xstratamobile_sync.mle_status.Text, “Status = 6″) <> 0 &
OR Pos(w_xstratamobile_sync.mle_status.Text, “Status = 1″)<> 0) then
rc = 2
return rc
end if
else
return rc
end if

Authenticating Users

In production environment it is very important to ensure that the person using the mobile application is authenticated. This will prevent data corruption by unauthorized users.

In fact 2 different authentication systems are needed:

  • mobilink access system
  • system access system.

Actually they can be combined, so that the user having the right to synchronize will also have the right to use the system.

The authentication system should be flexible and easily manageable. Obviously the best solution is to keep and manage this information in consolidated database. It is fairly easy to accomplish for the system access system. The authentication information can be kept in consolidated database and downloaded to the remote database during synchronization.

It is more difficult to manage mobilink access information from the central database.

One solution is to introduce unauthorized synchronization, which will download mobilink authentication information to the remote database.

Another way to do that is to set mobilink users manually on each device (using registry storage or device id).

However for both these solutions we should remember that mobilink user becomes a part of the remote database, so if we do not want to generate all the possible mobilink users in the distribution database and replace the database each time the change to the mobilink authentication information occurs, we will need to generate them dynamically.

This can be done just before the download synchronization (user name and password should be supplied by user on authentication stage or should be taken from the device information – registry or device id):

 

long l_ret = 0

string s_Aux

s_Aux = ‘CREATE SYNCHRONIZATION USER “‘ +s_User +’”‘

EXECUTE IMMEDIATE :s_Aux using SQLCA;

if (SQLCA.sqlcode <> 0) then

MessageBox(“Error”,”Unable to create synchroniation user”)

ROLLBACK using SQLCA;

return -1

end if

s_Aux = ‘CREATE SYNCHRONIZATION SUBSCRIPTION TO “MY_PUBLICATION” FOR “‘+ s_User +’” TYPE TCPIP’

EXECUTE IMMEDIATE :s_Aux using SQLCA;

if (SQLCA.sqlcode <> 0) then

MessageBox(“Error”,”Unable to create subscription”)

ROLLBACK using SQLCA;

return -1

end if

COMMIT USING SQLCA;

return l_Ret

In order to keep database clean the user should be deleted after upload synchronization:

s_Aux = ‘DROP SYNCHRONIZATION SUBSCRIPTION TO “MY_PUBLICATION” FOR “‘ + s_User + ‘”‘

EXECUTE IMMEDIATE :s_Aux using SQLCA;

if (SQLCA.sqlcode <> 0) then

MessageBox(“Error”,SQLCA.sqlerrtext)

return -1

end if

s_Aux = ‘DROP SYNCHRONIZATION USER “‘ + s_User + ‘”‘

EXECUTE IMMEDIATE :s_Aux using SQLCA;

if (SQLCA.sqlcode <> 0) then

MessageBox(“Error”,SQLCA.sqlerrtext)

return -1

end if

COMMIT USING SQLCA;

Note: the examples above also take care of creating and deleting of synchronization subscription, which is necessary for maintaining database log (see previous chapter).

Handling Errors

Mobilink provides 2 different events for error-handling:

  • handle_error (is triggered by all SQL errors)
  • handle_odbc_error (is triggered by ODBC driver errors)

Both events provide action_code parameter which can be used to control the behavior after the error occurred:

  • 1000 Skip the current row and continue processing.
  • 3000 Rollback the current transaction and cancel the current synchronization. This is the default action code, and is used when no handle error script is defined or this script causes an error.
  • 4000 Rollback the current transaction, cancel the synchronization, and shut down the MobiLink synchronization server.

If your event handler does not change the action_code it will have the default value – 3000

Typical usage of event handling scripts is logging errors:

 

CREATE OR REPLACE PROCEDURE Ml_Handle_Error( n_IN_ACTION_CDE IN OUT NUMERIC,

n_ERR_CODE NUMERIC,

S_ERR_MESSAGE VARCHAR2,

S_ML_USERNAME VARCHAR2,

S_ML_TABLE VARCHAR2

) AS

BEGIN

DECLARE

DT_GENDATE DATE;

N_OUT_ACTION_CDE NUMERIC;

BEGIN

SELECT SYSDATE INTO DT_GENDATE FROM DUAL;

N_OUT_ACTION_CDE := N_IN_ACTION_CDE;

INSERT INTO ML_ERRORS ( IN_ACTION_CDE, OUT_ACTION_CDE, ERR_CODE, ERR_TYPE, ERR_MESSAGE,

ML_USERNAME, ML_TABLE, GEN_DATE)

VALUES

(N_IN_ACTION_CDE, N_OUT_ACTION_CDE, N_ERR_CODE, ‘SQL’, S_ERR_MESSAGE,

S_ML_USERNAME, S_ML_TABLE, DT_GENDATE);

COMMIT;

END;

END;

And for ODBC errors:

CREATE OR REPLACE PROCEDURE Ml_Handle_Odbc_Error( n_IN_ACTION_CDE IN OUT NUMERIC,

s_ODBC_STATE VARCHAR2,

S_ERR_MESSAGE VARCHAR2,

S_ML_USERNAME VARCHAR2,

S_ML_TABLE VARCHAR2

) AS

BEGIN

DECLARE

DT_GENDATE DATE;

N_OUT_ACTION_CDE NUMERIC;

BEGIN

SELECT SYSDATE INTO DT_GENDATE FROM DUAL;

N_OUT_ACTION_CDE := N_IN_ACTION_CDE;

INSERT INTO ML_ERRORS ( IN_ACTION_CDE, OUT_ACTION_CDE, ERR_TYPE, ERR_MESSAGE,

ML_USERNAME, ML_TABLE, GEN_DATE, ODBC_STATE)

VALUES

(N_IN_ACTION_CDE, N_OUT_ACTION_CDE, ‘ODBC’, S_ERR_MESSAGE,

S_ML_USERNAME, S_ML_TABLE, DT_GENDATE, s_ODBC_STATE);

COMMIT;

END;

END;

You can also use the event handler to skip some errors silently – for example you can skip SQL errors for duplicates on primary key, when users create the same value independently, but there is no risk of loosing data selecting any of the created values arbitrarily.

Note: the syntax of the error handler script call in ML_SCRIPT :

{ call ml_handle_error(?,?,?,?,?)}

The curly brackets are mandatory in this call.

handle_error and handle_odbc_error can catch only part of mobilink errors. For example they won’t catch errors caused by connection problems.

For general synchronization errors you can use the error status, which is sent to end_synchronization event:

End_synchronization(ml_username, sync_ok)

Where sync_ok is 1 for successful synchronization and 0 otherwise.

In order to track failed syncs from the central database we can create a special table. For example:

CREATE TABLE PDA_SYNC_HIST

(

PDA_SYNC_DATE DATE NOT NULL,

USER_ID VARCHAR2(20 BYTE) NOT NULL,

PDA_SYNC_COMMENT VARCHAR2(100 BYTE),

PDA_SYNC_STATUS VARCHAR2(20 BYTE),

PDA_SYNC_COMPL DATE

)

Begin and end synchronization events can be used to populate this table:

CREATE OR REPLACE PROCEDURE Begin_Sync(ML_USER IN VARCHAR2) AS

BEGIN

DECLARE

dt DATE;

BEGIN

SELECT SYSDATE INTO DT FROM DUAL;

INSERT INTO PDA_SYNC_HIST (PDA_SYNC_DATE, USER_ID, PDA_SYNC_COMMENT, PDA_SYNC_STATUS)

VALUES

(DT,ML_USER,”,‘BEGIN SYNC’);

END;

END;

/

CREATE OR REPLACE PROCEDURE End_Sync(ml_user IN VARCHAR2,

SYNC_OK IN NUMBER) AS

BEGIN

DECLARE

DT DATE;

BEGIN

SELECT SYSDATE INTO DT FROM DUAL;

– UPDATE PDA SYNC TABLE

IF SYNC_OK = 1 THEN

UPDATE PDA_SYNC_HIST SET PDA_SYNC_STATUS = ‘END SYNC’, PDA_SYNC_COMPL=DT

WHERE USER_ID, = ML_USER AND PDA_SYNC_STATUS = ‘BEGIN SYNC’;

ELSE

UPDATE PDA_SYNC_HIST SET PDA_SYNC_COMMENT = ‘SYNC ERRORS’

WHERE USER_ID = ML_USER AND PDA_SYNC_STATUS = ‘BEGIN SYNC’;

END IF;

COMMIT;

END;

END;

/

These procedures do not ensure exact synchronization tracking: End_Sync will close all the open synchronizations. But actually that is enough as the integrity of synchronizations is ensured by Mobilink itself.

Blog at WordPress.com. | Theme: Pool by Borja Fernandez.
Entries and comments feeds.