Category: Scripting

Hi!  I’m Liz from Athenageek.  Tarah and I are writing guest posts in each other’s blogs on the scripting languages we love.

I must make a confession before I continue with this post – I adore Python. It’s made me a better coder. I think about problems differently than I used to, I’m better at breaking them down into tiny components that are easy to code. My coding in other languages improved drastically as a direct result of learning Python – and using it in a Pythonic manner.

I used to use Bash for scripting. That’s what Bash is for, right? I use a Bash shell, so it only made sense.

One day, I was processing some xml scripts. I had multiple xml files with the same root node, and needed to put them all together into one big xml file. Frustrated at some sticking point or other, I decided to see if I could do it with Python. Oh, my goodness, it was SO EASY.

Before long, I
had a string of utility files to handle common tasks that I was always scripting.

Reusability

When scripting, we often have tasks that we do over and over, but with slight changes. Rather than having a base set of methods that you copy out of and then change for each new usage, you can create utility files of classes and methods to use. Add methods to classes to handle new tasks, or allow new variables to be passed in for different usages. With Python’s named variables, you can change a method without effecting or needing to change other code that calls it.

One of the base tenants of Python is DRY (don’t repeat yourself). If you find yourself writing the same code more than once, put it in a place where it can be reused, like a method in a class or a function in a utility file.

Building up a libarary of these types of files is a great way to expand your toolbox. The longer you do it, the easier tasks become, because you’ve already done the work.  Lots of scripting projects are,
“Just like the one you did for x, but with this minor change”.

Testing

Python the community is heavily in favor of testing. One reason this is helpful for scripting is the case I mentioned above, when you change a method to handle some new functionality that you need. How do you make sure this new fuctionality doesn’t break old code? You test it. Doctesting and unit testing are built in to Python, no hunting around for which framework is best, balancing framework versions with script versions, or figuring out how to import your framework on a different operating system.

Doctesting

I love doctesting. In one fell swoop, you are showing how to use code, documenting that code, and proving that it works as expected.

Doctesting is very easy to do. Let’s say we create a method to square a number.

def square(numberIn):
    return numberIn * numberIn

No documentation there, let’s fix that.

def square(numberIn):
    """Returns the square of the 
number passed in"""
    return numberIn * numberIn

The doctest actually goes inside the documentation you’ve just created.

def square(numberIn):
    """Returns the square of the number passed in

    >>> square(2)
    4
    """
    return numberIn * numberIn

So, you use >>> to delineate code that is going to be tested. That’s the same >>> you see when you’re working at a Python prompt. In this case, we’re calling our method and passing in 2. On the next line, put the expected result, which is 4. You can put as many of these tests as you like in the documentation block, separate them with a blank line. You can even put error checking here.

def square(numberIn):
    """Returns the square of the number passed in

    >>> square(2)
    4

    >>> square('three')
    Traceback (most recent call last):
    ...
    TypeError: can't multiply sequence by non-int of 
type 'str'
    """
    return numberIn * numberIn

I got that traceback by using my function at a command prompt.  I copied and pasted it into my doctest.  I deleted the variable stuff between the two important lines and replaced it with ellipses.

In order to run the doctest, you have to add the following to the bottom
of your file:

if '__main__' == __name__:
    import doctest
    doctest.register_optionflag('ELLIPSIS')
    doctest.testmod()

Note that I’ve registered the ellipsis option flag. This allows you to use … as a wildcard so you can skip over some of the expected output. It keeps your code cleaner and lets you handle situations where expected output might be different – say – generating a random number.

Ok, now you can run the file. Let’s say we called it utility.py.

$ python utility.py
$

No output. That’s the default. In order to see the test results, you need to use the verbose flag.

$ python utility.
py -v

Trying:

    square(2)
Expecting:
    4
ok
Trying:
    square('three')
Expecting:
    Traceback (most recent call last):
    ...
    TypeError: can't multiply sequence by non-int of type 'str'
ok
1 items had no tests:
    __main__
1 items passed all tests:
   2 tests in __main__.square
2 tests in 2 items.
2 passed and 0 failed.

Test passed

$

And there are our test results – everything passed.

Aside from the option to put doctests in your code, you can also create txt files for documentation and add doctests to them. This is what bigger projects tend to do, especially when they have multiple files of code to deal with.

Unit tests are another great way to test that is very helpful for scripting. Every time you get a new requirement, add in a unit test for any situation you can think of that would effect that new requirement. Test things going right, things going wrong, and
any unexpected behavior or input you can think of. If you find an error that isn’t covered in your tests, create a unit test to cover it. A baseline of code like this ensures that any future changes won’t break existing code. If it does, it will make the error really easy to track down – you won’t have to try to remember all the past requirements.

Easy for the next person to maintain

Between keeping code in one place, not repeating yourself, and having robust testing, you’ve just made maintaining this code really, really easy for the next person that comes along. Even if they don’t know Python, a glance at your robust documentation and clear code will make it easy to make small changes. Running the tests will leave them confident that their changes won’t have a negative effect on other code.

We all want to be able to go on vacation and not get a call that someone broke your code. Clear code with lots of tests are the best way to ensure a relaxing break.  Python makes this quick
and easy, so you can move on to something hard.

Ok, so I am feeling guilty over not reading enough GOOD books.

So, I devised a cunning plan whereby I am emailed a portion of a classic book each day. I started with the Iliad. I know that I’ll clear my emails every single day because a cluttered inbox irritates me…so this is the perfect motivational tool. I wrote each of these scripts (with occasional help from the lovely folks who hang out with me in Programming Talk at the Ubuntu Forums), and this is the way I did this.

(1) Download a book you want to read. I suggest starting with something like the Iliad or Ovid’s Metamorphoses. Poetry is a great way to start; it’s episodic and fun. Head to Gutenberg.org to find text versions of any book you want. Other suggestions: the Aeneid, any Shakespeare play, the Divine Comedy, Paradise Lost, etc.

(2) Drop it into a folder, renaming it ‘[whatever].txt’. In my example, I’m dividing the Metamorphoses.

(3) Navigate to
that folder at the CL.

(4) Run this script to divide the book up into 300 line segments (or, if there is no paragraph break at 300 lines, the script will add more lines until a paragraph break is reached. That way, you don’t end up getting half a soliloquy). Obviously, if you want to read more or less each day, change the number of lines to however much you want:


#!/bin/bash
bookcounter=0
linecounter=0
sed -i 's/\r$//' metamorphoses.txt
while read line
do
((linecounter+=1))
if [[ $linecounter -gt 300 && -z "$line" || -z "$output_file" ]]; then
linecounter=0
((bookcounter+=1))
formatted_bookcounter=$(printf "%03d" $bookcounter)
output_file=metamorphoses${formatted_bookcounter}.txt
echo "...starting segment $output_file"
echo "Metamorphoses - segment $formatted_bookcounter" > $output_file
echo "===================" >> $output_file
echo "" >> $output_file
fi
echo "$line" >> $output_file
ndone < metamorphoses.txt

(5) In your folder, delete the big file. You should now have several (or possibly several dozen or hundreds) of much smaller files, numbered consecutively.

(6) Then, run this script (Adds a subject header to each of the files in your folder) :


#!/bin/bash
FILES="*"
for f in $FILES ; do
sed -i '1s/^/Subject:Your Daily Book Part, Tarah.\n/' "$f"
cat "$f"
done


(7) Then, add this script to wherever you store your scripts. It sorts the files in your pet directory by number, gets the first one (i.e. 'metamorphoses024.txt') mails it to you, then deletes that file. Then, the next time the script is run, it will get 'metamorphoses025.txt' as the first file in the directory. Call it 'bookmailer.sh' or whatever. NB: you must have msmtp installed to make this work. Ensure it's reachable through your path. Any other command line emailer will also work; I know how to use msmtp, so that's what I do. Plus, I use
Kubuntu, so it's available at the repos.


#!/bin/bash
cd /home/tarahmarie/Documents/MyEbooks/aaa/
FILE=$(ls | sort -n | head -1)
cat $FILE | msmtp -a gmail youremailaddress@gmail.com
\rm `ls | sort -n | head -1`


(8) Execute 'bookmailer.sh' at the CL to ensure it works. If, in a few seconds, you get an email containing the subject line 'Your Daily Book Part, WhoeverYouAre', then the command is working. If you're having any problems, ensure that you have permissions to each of these scripts, and that they're marked 'executable'.

(9) Add 'bookmailer.sh' to your system's list of scheduled commands. I use kcron, so I set 'bookmailer.sh' to be executed each day at 8AM.

(10) Feel superior to other humans for the fact that you're reading a 'good' book every day. This is DEFINITELY how I'm going to get all the way through War and Peace. Eventually. Let us not even speak of Remembrance of Things Past. I will CONQUER you, Proust, ole buddy.