Sunday, August 23, 2015

Phonetic search of arabic names latin spelled

The goal of this article is to give an approach to search record on a latin database of recrods containing arab names / surnames,  of course, there is no problem if the records were written in arabic letter encoded in a charset that supports this language such ar8mswin1256 charset recommended for oracle databases, but when we phonetic translate an arabic name and spell it in latin chars, that's another kind of non bijective job,
Let's take a look:

Mohammed (salla Allah Alyhi wa sallam) can be spelled :
   1- Mohamed
   2- Muhamed
   3- Mouhamed
   4- Mouhammed
   5- Med (Abusive spelling in north Afraica countries ex french occupied  countries)
   6- Mohd(Same a 5)

you must note that in arabic theres there is often a successive double consonant, and those who don't respect phonetic translation can omit the second char, even if the word become incorrect in certain case
eg:
 'S' between two vowels that become 'Z'

Prohet's name is a very frequent name in Muslim countries, we can say then that's a particular case?
Let's get another look


i'll take now my name and others
Wassim, can be spelled:
   1- Ouassim
   2- Wessim
   3- Ouessim
   4- Wassime
   5- Ouassime
   6- Ouessime

And that for all names with 'W' char, we can replace the W by ou and we have the same name.
We can note also that the 'E' char at the end of the word often doesn't metter

Tayeb can be spelled:
   1-Taieb
We can note that 'Y' char and 'I' (also 'EE' in middle east culture ) have the same phonetic effect.
then we can spell
Sami-->Samy (don't forget Sammy -double consonant - evoked earlier)
Rym -->Rim

etcetera etcetera...

Let's now comeback to the approach that can be adapted to resolve the problem.

First of all, we must establish our names dictionary, not a real dictionary because arabic language has the must vast dictionary with 12 000 000 word vs 600 000 word in english **, but let's say that we will build a function that will transform an arabic name on another word that we can exploit later by a simple matching with the result of the same function executed on the searched for name.

behind sentences
We'll build function f
for each name of our database
 f(name) = name' stored somewhere

select from database rows with condition ; f(searchedForName) = name'

 f(name) = name' stored somewhere ??? why??
storing the result will be very useful because the result is always the same for records stored in our database, we win a time and money by ding that until the change of our function f we must re-execute
the function on our population.

function f(name):
Considering our parameter name which is a String, an intelligent algorithm will consider that string a char sequence, other ways we will be in the case **.We'll transform that char sequence to another one which we will respect our universal rules, and unlike ** the cardinality of  rules set is an infinitesimal (x 10) in front of arabic words set count (12 000 000)
Our algorithm will take a decision and return a string that can be blank if needed for all incoming letters in our world: Typical recursive behavior, the normalized name will be the concatenation of results of recursive call on the word.

We remark also that the pseudo strings generated during recursive call, needs some extra information:
specially previous char 'in consonant case (double char) or other cases' end sometimes we need the next char if it exists.


ALGORITHM String getArabicPhoneticEquivalent (String name , Int depth) RETURN String

DECLARE
   String toReturn;
   char nextChar,currentChar;
BEGIN

IF StringLength(name) == depth THEN //Condition to break recursivity
    return "";
END IF;

toReturn =""; // Must be set to empty string

IF depth == 0 THEN
     name =  StringLowerCase(name);//Affects lowercase to input
     name =  cleanSpecialChars(name); //Transfom some unsupported letters
     toReturn  = getSpecialNames(name);// Tests for special names
END IF;


IF toReturn != "" THEN // Case we have special name, no need to continue
   return toReturn;
END IF;

currentChar = name[depth];

IF StringLength(name) > depth THEN
     nextChar = name[depth+1];
ELSE
     nextChar = ' ';
END IF;
depth = depth + 1;
IF nextChar == currentChar THEN //Double letter simulated to one
    return  getArabicPhoneticEquivalent (name , depth);
END IF;

IF consonant(name[depth]) THEN
   
    IF name[depth] == 'h' THEN //Skip h after a consonant
         IF depth > 1 AND consonant(name[depth-1]) THEN
              return  getArabicPhoneticEquivalent (name , depth);
         END IF;
     END IF;
     return StringConcat(getConsonantEquivalent(name[depth],nextChar) , getArabicPhoneticEquivalent (name , depth));
END IF;

IF vowel(name[depth]) THEN
       return StringConcat(getVowelEquivalent(name[depth],nextChar) , getArabicPhoneticEquivalent (name , depth));
END IF;

END ALGORITHM;    

ALGORITHM skipSpecialChars(String name) RETURN String
BEGIN
  name = StringReplaceAllSequences("w","ou");
  name = StringReplaceAllSequences("ï","i");
  name = StringReplaceAllSequences("î","i");
  name = StringReplaceAllSequences("ô","o");
  name = StringReplaceAllSequences("é","e");
  name = StringReplaceAllSequences("è","e");
  name = StringReplaceAllSequences("ê","e");
  name = StringReplaceAllSequences("à","a");
  name = StringReplaceAllSequences("ç","c");
  return name;
END ALGORITHM;

ALGORITHM getSpecialNames(String name) RETURN String
BEGIN
   IF name == "med" OR name =="mohd" THEN
       return 'mouhamed';
   END IF;

   return "";
END ALGORITHM;

ALGORITHM getConsonantEquivalent(char currentChar, char nextChar) RETURN String
BEGIN
IF currentChar == 'c' THEN
    IF  consonant(nextChar)THEN
          return "k";
    ELSEIF vowel(nextChar) THEN
          return "s"
    END IF;
END IF;
return currentChar+"";
END ALGORITHM

ALGORITHM getVowelEquivalent(char currentChar, char nextChar) RETURN String
BEGIN
IF currentChar == 'y' THEN
     return "i";
END IF;
IF currentChar == 'e' THEN
     return "a";
END IF;
return currentChar+"";
END ALGORITHM;

Note that we can make this algorithm more performant by adding additional controls
such as:
The abreviation of ben 'b'
The 'el' similar to 'al'

In the PLSQL code below i've added these controls and some others
'Code may be more up te date than algorithm ( no worry the spirit is kept)'

 
create or replace 
package body PKG_SEARCH is
  -------------------------------------------------------------
  -- Teste les voyelles
  -- @return boolean
  -------------------------------------------------------------
  function f_is_vowel(c in char) return boolean is
  begin
     if c in ('a','e','i','o','y','u') then 
        return true;
     else
        return false;
     end if;   
  end;
    -------------------------------------------------------------
  -- Teste les consonnes
  -- @return boolean
  -------------------------------------------------------------
  function f_is_consonent(c in char) return boolean is
  begin
     if c in ('b','c','d','f','g','h','j','k','l','m','n','p','q','r','s','t','v','w','x','z') then      
        return true;
     else
        return false;
     end if;     
  end;
  -------------------------------------------------------------
  -- Teste les consonnes
  -- @return boolean
  -------------------------------------------------------------
  function  f_special_chars(a_name in out varchar) return varchar is
  begin    
    a_name := replace(a_name,'ï','i');
    a_name := replace(a_name,'î','i');
    a_name := replace(a_name,'ô','o');
    a_name := replace(a_name,'é','e');
    a_name := replace(a_name,'è','e');
    a_name := replace(a_name,'ê','e');
    a_name := replace(a_name,'à','a');
    a_name := replace(a_name,'ç','c');    
    a_name := replace(a_name,'w','o');    
    a_name := replace(a_name,'u','o');   
    a_name := replace(a_name,'y','i');   
    a_name := replace(a_name,' b ','ben');   
    a_name := replace(a_name,' b','ben');   
    a_name := replace(a_name,' al ',' al');   
    if (length(a_name) >1 and substr(a_name,1,2) = 'b ') or (length(a_name) >3 and substr(a_name,1,2) = 'ben ')  then
          a_name := 'ben '|| substr(a_name,2);
    end if;
    if length(a_name) >2 and substr(a_name,1,2) = 'el' then
          a_name := 'al'|| substr(a_name,3);
    end if;
    return a_name;  
  end;
  -------------------------------------------------------------
  -- Traite les noms speciaux
  -- @return boolean
  -------------------------------------------------------------
  function f_special_names(a_name in varchar) return  varchar is
  begin
     if a_name = 'med' or a_name ='mohd' then
       return 'mouhamed';
     end if;  
     return '';
  end;
  -------------------------------------------------------------
  -- Retourne la chaine de caractère qui correspond à la consonne en question
  -- @return boolean
  -------------------------------------------------------------
function f_consonant_equivalent(currentChar in char, nextChar in char) return  varchar is
  begin
  if currentChar = 'c' THEN
      if  f_is_consonent(nextChar)THEN 
            return 'k';
      elsif f_is_vowel(nextChar) THEN
            return 's';
      end if;
  end if;
  if currentChar = 'd' THEN
        if  f_is_consonent(nextChar)THEN
          return '';
        end if;
  end if;  
  return currentChar;
end;
  -------------------------------------------------------------
  -- Retourne la chaine de caractère qui correspond à la voyelle en question
  -- @return boolean
  -------------------------------------------------------------
function f_vowel_equivalent(currentChar in char, nextChar in char) return varchar is
begin  
  if currentChar = 'e' then
      if  (nextChar=' ') then
          return '';
      else    
          return 'a';
      end if;
  end if;    
  return currentChar;
end; 

  -------------------------------------------------------------
  -- Retourne l'equivalent phonetic
  -- Main function 
  -- @return boolean
  -------------------------------------------------------------
function f_arabic_phonetic_aquivalent (a_name in out varchar, depth in out number) return varchar is
   toReturn varchar(256);
   nextChar char;
   currentChar char;
   nextDepth number;
BEGIN

IF depth = 1 THEN
     a_name :=  lower(a_name);
     a_name :=  f_special_chars(a_name);  
END IF;

toReturn  := f_special_names(a_name);
IF toReturn <> '' THEN 
   return toReturn;
END IF;

IF length(a_name) < depth THEN 
    return '';
END IF;

toReturn :=''; 
nextDepth := depth + 1;

currentChar := substr(a_name,depth,1);
IF currentChar = ' ' THEN
    return  ' '||f_arabic_phonetic_aquivalent (a_name , nextDepth);
END IF;
IF length(a_name) > depth THEN
     nextChar := substr(a_name,nextDepth,1);      
ELSE
     nextChar := ' ';
END IF;

IF nextChar = currentChar THEN 
    return  f_arabic_phonetic_aquivalent (a_name , nextDepth);
END IF;

IF f_is_consonent(substr(a_name,depth,1)) THEN    
    IF substr(a_name,depth,1) = 'h' THEN 
         IF depth > 1 AND f_is_consonent(substr(a_name,depth-1,1)) THEN
              return  f_arabic_phonetic_aquivalent (a_name , nextDepth);
         END IF;
     END IF;
     return concat(f_consonant_equivalent(substr(a_name,depth,1),nextChar) , f_arabic_phonetic_aquivalent (a_name ,nextDepth));
END IF;

IF f_is_vowel(substr(a_name,depth,1)) THEN
       return concat(f_vowel_equivalent(substr(a_name,depth,1),nextChar) , f_arabic_phonetic_aquivalent (a_name , nextDepth));
END IF;
return NULL;
END ; 
----------------------------------------------------------------
function f_pkg_runner (a_name in out varchar) return varchar is
toReturn varchar2(256);
depth number;
begin
   depth :=1;
   toReturn := f_arabic_phonetic_aquivalent (a_name, depth);
   toReturn := replace(toReturn,' mohd ','mouhamad');
   toReturn := replace(toReturn,' med ','mouhamad');
   if toReturn = 'mohd' or toReturn = 'med'then
      return 'mouhamad';
   end if;   
   return toReturn;
end;
----------------------------------------------------------------
function f_pkg_tester() return varchar is
sampleSurname varchar(256);

begin
      sampleSurname  := 'wassim';
      sampleSurname   := PKG_SEARCH .f_pkg_runner(sampleSurname);
      dbms_output.put_line(sampleSurname);
      return sampleSurname;
 
end;
end PKG_SEARCH ;
This will output oasim