Python's on the horizon

It seems like with advent of technology, the computing world grows closer and closer to nature everyday. The biggest trend which exhibits this facts is the continual need for a developer to learn new technologies and new languages.

It was only a few decades ago where a single degree or a diploma could get you through a lifetime. Just ask your parents. Most parents, like mine would probably have a bachelor's degree which allowed them to provide themselves and for their kids. This includes not just the needs for food, basic education, and 'wants', but also in my case, higher education. I was one of those fortunate enough to have parents who supported my endeavor to pursue a master's degree. But to top it all, they managed to keep their heads above water without falling into the endless abyss of loans.

Today, I hear countless stories and have encountered people hesitant to even think about higher education due to the cost, and struggle between quality and quantity.

Even with my high school, bachelor's, and master's degree, I still have to learn new tricks. It's not because I need to. Need is immediate. Not learning is slow extinction.

And to prevent extinction, I decided to learn Python. Some call it the language of and for hackers. I don't portray myself as a hacker. I like perfection way too much. Which prevents me from slapping things together to call it a day and never look at it again. For python, I started a few weeks ago by reading some basic tutorials. Reading those tutorials was pretty boring, but life was interesting again when I was presented with a challenge at work.

The challenge presented itself as a need to insert a few additional characters in an already existing file. The file format is a Motorola S-Record (.mot file) and is fully human readable. It's contents include a bunch of numbers which are machine instructions, codes that your computer/microcomputer can understand. Without getting into too much detail about the S-Record, here's how one would look like:
S315FFFF00007FB8FBEA0004FD68E2FBEA0005FD68EA4C
And here's what the challenge intended to make it look like.
{ 0x15,    0xFF,0xFF,0x00,0x00,    0x7F,0xB8,0xFB,0xEA,0x00,0x04,0xFD,0x68,0xE2,0xFB,0xEA,0x00,0x05,0xFD,0x68,0xEA,    0x4C, },
To someone who is a computer literate, there are three basic ways of doing it,
1. Manually using text editors
2. Semi-automatic methods i.e. Use text import wizards to import and partially format the data and then fill out the unfilled blanks.
3. Fully automatic methods. Write a computer program which does everything.

Well, #1 and #2 would work if there were a limited number of these S-Records, but usually a .mot file contains at least several hundred of these. The hazards of human intervention are high, the task is simple, which makes #3 highly attractive. And the fact that the challenge doesn't have any constraints on computing power or speed makes it a perfect candidate for creating a python script with a purpose.  And here's the not-so-sexy, and not-so-fast script that does exactly what is needed.

#!/usr/bin/python

## Creates data records as a c-struct from lines in a MOT-file. The infile has to be *.MOT.

## Script is now intelligent and splits records which cross 1k boundaries into two records and

## inserts a dummy checksum into the two split records

## Created: Onkar Raut, onkar.raut@renesas.com, 2014, Oct, 06th.

## Usage: 'create_data_record.py -i <path_to_input_file> -o <path_to_output_file>'

##          OR 'create_data_record.py -h'

## Example: python create_data_record.py -i TestImage.mot -o TestImage.dat

## Argument parsing code is borrowed from: http://www.tutorialspoint.com/python/python_command_line_arguments.htm

import sys, getopt

def main(argv):

   inputfile = ''

   outputfile = ''

   try:

      opts, args = getopt.getopt(argv,"hi:o:",["ifile=","ofile="])

   except getopt.GetoptError:

      print 'create_data_record.py -i <path_to_input_file> -o <path_to_output_file>'

      sys.exit(2)

   for opt, arg in opts:

      if opt == '-h':

         print 'create_data_record.py -i <path_to_input_file> -o <path_to_output_file>'

         sys.exit()

      elif opt in ("-i", "--ifile"):

         inputfile = arg

      elif opt in ("-o", "--ofile"):

         outputfile = arg

   print 'Input file is "', inputfile

   print 'Output file is "', outputfile

   f_wr = open(outputfile, 'w+')

   with open(inputfile, 'r') as f_rd:

      for line in f_rd:

         nr_bytes_remain = 0

         ##Strip out the first two bytes in the line##

         line = line[2:]

         ##Create size information##

         nr_bytes = int(line[:2],16)

         nr_bytes_orig = nr_bytes+1

         size = 2*nr_bytes

         size_orig = size

         ##Pull in the address being used##

         address = int(line[2:10],16)

         nr_data_bytes = nr_bytes - 4 - 1

         nr_data_bytes_orig = nr_data_bytes

         ## Check if address exceeds a 1k boundary

         if ((address + nr_data_bytes) > ((address&0xFFFFFC00)+1024)):

            ##Print some debug info  

            print '******Detected need to insert additional data record ******'

            ##Calculate the number of bytes to put in this data record

            print 'Block boundary = ',hex((address&0xFFFFFC00)+1024)

            print 'Address = ',hex(address)

            nr_data_bytes = ((address&0xFFFFFC00)+1024) - address

            nr_bytes_remain = nr_data_bytes_orig - nr_data_bytes

            nr_bytes = nr_data_bytes + 4 + 1

            size = 2*nr_bytes

            print 'Num of additional bytes in record = ',nr_bytes_remain

            print '\n'

         

         ##Print the size of the read record

         print 'Address = ',hex(address) 

         nr_data_bytes = nr_bytes - 4 - 1

         print 'Num data bytes in record = ',nr_data_bytes

         

         ##Format the data record##

         i = 0

         newline = '{ ';

         if nr_bytes_remain > 0:

            ##Fill in the new number of bytes filled in##

            newline = newline + str(hex(nr_bytes)) + ', /*ADJ*/ '

            size -= 2

            i = 2

            

         ## Fill in the record ##   

         while i<=size:

            newline = newline + '0x' + line[0+i:0+i+2] + ','

            if i == 0:

               newline = newline + '    '

            elif i == 8:

               newline = newline + '    '               

            elif i == (size - 2) and nr_bytes_remain == 0:

               newline = newline + '    '

            i+=2

         if nr_bytes_remain > 0:

            newline += ' /* Dummy */ 0xAA, '

         newline += ' },\n'

         f_wr.write(newline)

         if nr_bytes_remain > 0:

            print 'Adjusted data record at current address as shown below '

            print newline

         if nr_bytes_remain > 0:

            addline = '{ '

            ##Fill in the number of bytes left over from prev record + address + checksum

            addline = addline + str(hex(nr_bytes_remain + 4 + 1)) + ', /*NEW*/ '

            block_boundary = ((address&~0x3FF)+1024)

            ## Address on a block boundary ##

            block_boundary_b1 = ((block_boundary >> 24) & 0xFF);

            block_boundary_b2 = ((block_boundary >> 16) & 0xFF);

            block_boundary_b3 = ((block_boundary >> 8) & 0xFF);

            block_boundary_b4 = ((block_boundary >> 0) & 0xFF);

            addline += str(hex(block_boundary_b1)) + ',' + str(hex(block_boundary_b2)) + ',' + str(hex(block_boundary_b3))+ ',' + str(hex(block_boundary_b4)) + ',   '

            ## Fill in the record ##   

            while i<size_orig:

               addline = addline + '0x' + line[0+i:0+i+2] + ','

               if i == 0:

                  addline = addline + '    '

               elif i == 8:

                  addline = addline + '    '               

               elif i == (size - 4):

                  addline = addline + '    '

               i+=2

            addline += ' /*Dummy*/ 0xAA },\n'

            print 'New record created below with remaining bytes = ', nr_bytes_remain, ' at address ',  hex(block_boundary)

            print addline

            f_wr.write(addline)

                  

   f_rd.closed

   f_wr.close();

if __name__ == "__main__":

   main(sys.argv[1:])

It took me about 8 hours to get it work, even after borrowing some initial code, but it works and helps me from not becoming extinct ;) So Python has definitely made it's way onto the horizon. Here's to hope that there's more to it in the future.