Professional Documents
Culture Documents
There's a saying that any software problem can be solved by adding a layer of indirection. That's what
Perl's DBI (`Database Interface') module is all about. It was written by Tim Bunce.
DBI is designed to protect you from the details of the vendor libraries. It has a very simple interface
for saying what SQL queries you want to make, and for getting the results back. DBI doesn't know
how to talk to any particular database, but it does know how to locate and load in DBD (`Database
Driver') modules. The DBD modules have the vendor libraries in them and know how to talk to the real
databases; there is one DBD module for every different database.
When you ask DBI to make a query for you, it sends the query to the appropriate DBD module, which
spins around three times or drinks out of its sneaker or whatever is necessary to communicate with
the real database. When it gets the results back, it passes them to DBI. Then DBI gives you the
results. Since your program only has to deal with DBI, and not with the real database, you don't have
to worry about barking like a chicken.
Here's your program talking to the DBI library. You are using two databases at once. One is an Oracle
database server on some other machine, and another is a DBD::CSV database that stores the data in
a bunch of plain text files on the local disk.
Your program sends a query to DBI, which forwards it to the appropriate DBD module; let's say it's
DBD::Oracle. DBD::Oracle knows how to translate what it gets from DBI into the format
demanded by the Oracle library, which is built into it. The library forwards the request across the
network, gets the results back, and returns them to DBD::Oracle. DBD::Oracle returns the results
to DBI as a Perl data structure. Finally, your program can get the results from DBI.
On the other hand, suppose that your program was querying the text files. It would prepare the same
sort of query in exactly the same way, and send it to DBI in exactly the same way. DBI would see that
you were trying to talk to the DBD::CSV database and forward the request to the DBD::CSV module.
The DBD::CSV module has Perl functions in it that tell it how to parse SQL and how to hunt around in
the text files to find the information you asked for. It then returns the results to DBI as a Perl data
structure. Finally, your program gets the results from DBI in exactly the same way that it would have
if you were talking to Oracle instead.
There are two big wins that result from this organization. First, you don't have to worry about the
details of hunting around in text files or talking on the network to the Oracle server or dealing with
Oracle's library. You just have to know how to talk to DBI.
Second, if you build your program to use Oracle, and then the following week upper management
signs a new Strategic Partnership with Sybase, it's easy to convert your code to use Sybase instead of
Oracle. You change exactly one line in your program, the line that tells DBI to talk to DBD::Oracle,
and have it use DBD::Sybase instead. Or you might build your program to talk to a cheap, crappy
database like MS Access, and then next year when the application is doing well and getting more use
than you expected, you can upgrade to a better database next year without changing any of your
code.
There are DBD modules for talking to every important kind of SQL database. DBD::Oracle will talk to
Oracle, and DBD::Sybase will talk to Sybase. DBD::ODBC will talk to any ODBC database including
Microsoft Acesss. (ODBC is a Microsoft invention that is analogous to DBI itself. There is no DBD
module for talking to Access directly.) DBD::CSV allows SQL queries on plain text files. DBD::mysql
talks to the excellent MySQL database from TCX DataKonsultAB in Sweden. (MySQL is a tremendous
bargain: It's $200 for commercial use, and free for noncommerical use.)
use DBI;
my $dbh = DBI->connect('DBI:Oracle:payroll')
or die "Couldn't connect to database: " . DBI->errstr;
my $sth = $dbh->prepare('SELECT * FROM people WHERE lastname = ?')
or die "Couldn't prepare statement: " . $dbh->errstr;
print "Enter name> ";
while ($lastname = <>) {
# Read input from the user
my @data;
chomp $lastname;
$sth->execute($lastname)
# Execute the query
or die "Couldn't execute statement: " . $sth->errstr;
# Read the matching records and print them out
while (@data = $sth->fetchrow_array()) {
my $firstname = $data[1];
my $id = $data[2];
print "\t$id: $firstname $lastname\n";
}
if ($sth->rows == 0) {
print "No names matched `$lastname'.\n\n";
}
$sth->finish;
print "\n";
print "Enter name> ";
}
$dbh->disconnect;
use DBI;
This loads in the DBI module. Notice that we don't have to load in any DBD module. DBI will do that
for us when it needs to.
my $dbh = DBI->connect('DBI:Oracle:payroll');
or die "Couldn't connect to database: " . DBI->errstr;
The connect call tries to connect to a database. The first argument, DBI:Oracle:payroll, tells
DBI what kind of database it is connecting to. The Oracle part tells it to load DBD::Oracle and to
use that to communicate with the database. If we had to switch to Sybase next week, this is the one
line of the program that we would change. We would have to change Oracle to Sybase.
payroll is the name of the database we will be searching. If we were going to supply a username
and password to the database, we would do it in the connect call:
my $dbh = DBI->connect('DBI:Oracle:payroll', 'username', 'password')
or die "Couldn't connect to database: " . DBI->errstr;
If DBI connects to the database, it returns a database handle object, which we store into $dbh. This
object represents the database connection. We can be connected to many databases at once and have
many such database connection objects.
If DBI can't connect, it returns an undefined value. In this case, we use die to abort the program with
an error message. DBI->errstr returns the reason why we couldn't connect``Bad password'' for
example.
This loop will repeat over and over again as long as the user enters a last name. If they type a blank
line, it will exit. The Perl <> symbol means to read from the terminal or from files named on the
command line if there were any.
my @data;
This declares a variable to hold the data that we will get back from the database.
chomp $lastname;
This trims the newline character off the end of the user's input.
$sth->execute($lastname)
# Execute the query
or die "Couldn't execute statement: " . $sth->errstr;
execute executes the statement that we prepared before. The argument $lastname is substituted
into the SQL in place of the ? that we saw earlier. execute returns a true value if it succeeds and a
false value otherwise, so we abort if for some reason the execution fails.
my $firstname = $data[1];
my $id = $data[2];
These lines extract the first name and the ID number from the record data.
if ($sth->rows == 0) {
print "No names matched `$lastname'.\n\n";
}
The rows method returns the number of rows of the database that were selected. If no rows were
selected, then there is nobody in the database with the last name that the user is looking for. In that
case, we print out a message. We have to do this after the while loop that fetches whatever rows
were available, because with some databases you don't know how many rows there were until after
you've gotten them all.
$sth->finish;
print "\n";
print "Enter name> ";
Once we're done reporting about the result of the query, we print another prompt so that the user can
enter another name. finish tells the database that we have finished retrieving all the data for this
query and allows it to reinitialize the handle so that we can execute it again for the next query.
$dbh->disconnect;
When the user has finished querying the database, they type a blank line and the main while loop
exits. disconnect closes the connection to the database.
Cached Queries
Here's a function which looks up someone in the example table, given their ID number, and returns
their age:
sub age_by_id {
# Arguments: database handle, person ID number
my ($dbh, $id) = @_;
my $sth = $dbh->prepare('SELECT age FROM people WHERE id = ?')
or die "Couldn't prepare statement: " . $dbh->errstr;
$sth->execute($id)
or die "Couldn't execute statement: " . $sth->errstr;
my ($age) = $sth->fetchrow_array();
return $age;
}
It prepares the query, executes it, and retrieves the result.
There's a problem here though. Even though the function works correctly, it's inefficient. Every time
it's called, it prepares a new query. Typically, preparing a query is a relatively expensive operation. For
example, the database engine may parse and understand the SQL and translate it into an internal
format. Since the query is the same every time, it's wasteful to throw away this work when the
function returns.
Here's one solution:
{ my $sth;
sub age_by_id {
# Arguments: database handle, person ID
my ($dbh, $id) = @_;
if (! defined $sth) {
$sth = $dbh->prepare('SELECT age FROM
or die "Couldn't prepare statement:
}
$sth->execute($id)
or die "Couldn't execute statement: "
my ($age) = $sth->fetchrow_array();
return $age;
number
people WHERE id = ?')
" . $dbh->errstr;
. $sth->errstr;
There are two big changes to this function from the previous version. First, the $sth variable has
moved outside of the function; this tells Perl that its value should persist even after the function
returns. Next time the function is called, $sth will have the same value as before.
Second, the prepare code is in a conditional block. It's only executed if $sth does not yet have a
value. The first time the function is called, the prepare code is executed and the statement handle is
stored into $sth. This value persists after the function returns, and the next time the function is
called, $sth still contains the statement handle and the prepare code is skipped.
Here's another solution:
sub age_by_id {
# Arguments: database handle, person ID number
my ($dbh, $id) = @_;
my $sth = $dbh->prepare_cached('SELECT age FROM people WHERE id
= ?')
or die "Couldn't prepare statement: " . $dbh->errstr;
$sth->execute($id)
or die "Couldn't execute statement: " . $sth->errstr;
my ($age) = $sth->fetchrow_array();
return $age;
}
Here the only change to to replace prepare with prepare_cached. The prepare_cached call is
just like prepare, except that it looks to see if the query is the same as last time. If so, it gives you
the statement handle that it gave you before.
Transactions
Many databases support transactions. This means that you can make a whole bunch of queries which
would modify the databases, but none of the changes are actually made. Then at the end you issue
the special SQL query COMMIT, and all the changes are made simultaneously. Alternatively, you can
issue the query ROLLBACK, in which case all the queries are thrown away.
As an example of this, consider a function to add a new employee to a database. The database has a
table called employees that looks like this:
FIRSTNAME LASTNAME
DEPARTMENT_ID
Gauss
Karl
17
Smith
Mark
19
Noether
Emmy
17
Smith
Jeff
666
Hamilton
William
17
and a table called departments that looks like this:
ID
NAME
17
666
19
NUM_MEMBERS
3
1
1
Mathematics
Legal
Grounds Crew
The mathematics department is department #17 and has three members: Karl Gauss, Emmy Noether,
and William Hamilton.
Here's our first cut at a function to insert a new employee. It will return true or false depending on
whether or not it was successful:
sub new_employee {
# Arguments: database handle; first and last names of new employee;
# department ID number for new employee's work assignment
my ($dbh, $first, $last, $department) = @_;
my ($insert_handle, $update_handle);
my $insert_handle =
$dbh->prepare_cached('INSERT INTO employees VALUES (?,?,?)');
my $update_handle =
$dbh->prepare_cached('UPDATE departments
SET num_members = num_members + 1
WHERE id = ?');
die "Couldn't prepare queries; aborting"
unless defined $insert_handle && defined $update_handle;
$insert_handle->execute($first, $last, $department) or return 0;
$update_handle->execute($department) or return 0;
return 1;
# Success
}
We create two handles, one for an insert query that will insert the new employee's name and
department number into the employees table, and an update query that will increment the number
of members in the new employee's department in the department table. Then we execute the two
queries with the appropriate arguments.
There's a big problem here: Suppose, for some reason, the second query fails. Our function returns a
failure code, but it's too late, it has already added the employee to the employees table, and that
means that the count in the departments table is wrong. The database now has corrupted data in it.
The solution is to make both updates part of the same transaction. Most databases will do this
automatically, but without an explicit instruction about whether or not to commit the changes, some
databases will commit the changes when we disconnect from the database, and others will roll them
back. We should specify the behavior explicitly.
Typically, no changes will actually be made to the database until we issue a commit. The version of
our program with commit looks like this:
sub new_employee {
# Arguments: database handle; first and last names of new employee;
# department ID number for new employee's work assignment
my ($dbh, $first, $last, $department) = @_;
my ($insert_handle, $update_handle);
my $insert_handle =
$dbh->prepare_cached('INSERT INTO employees VALUES (?,?,?)');
my $update_handle =
$dbh->prepare_cached('UPDATE departments
SET num_members = num_members + 1
WHERE id = ?');
die "Couldn't prepare queries; aborting"
unless defined $insert_handle && defined $update_handle;
my $success = 1;
do
If you're doing an UPDATE, INSERT, or DELETE there is no data that comes back from the database,
so there is a short cut. You can say
AutoCommit
If your transactions are simple, you can save yourself the trouble of having to issue a lot of commits.
When you make the connect call, you can specify an AutoCommit option which will perform an
automatic commit operation after every successful query. Here's what it looks like:
my $dbh = DBI->connect('DBI:Oracle:payroll',
{AutoCommit => 1},
)
or die "Couldn't connect to database: " . DBI->errstr;
my $dbh = DBI->connect('DBI:Oracle:payroll',
{RaiseError => 1},
)
or die "Couldn't connect to database: " . DBI->errstr;
Don't do This
People are always writing code like this:
big message because they are calling the backslashing function every other line. They put a lot of
work into it the backslashing function, and it was all for nothing, because the whole problem is solved
by just putting a ? into the query, like this
An Embarrassing Confession
I'd like to think that I'm a reasonably decent Perl programmer now. I'd like to think that I have a good
grasp of how to solve relatively common problems in Perl. But, you know, it hasn't always been this
way. Oh no.
A long, long time ago, when I was a tiny little programmer, I worked as a trainee Perl coder and
systems administrator for a large database company. Naturally, at a database company, a lot of what
we had to do was talking to databases in Perl. As a fresh-faced programmer, the only way I knew to
interface with databases was through a command-line SQL client.
I won't embarrass the company in question by giving away the name of the database, so let's call this
SQL client sqlstar. Very soon I was writing horrendous Perl programs that did things like:
Related Reading
Programming
the Perl DBI
Database
programming
with Perl
By
Alligator Descart
es, Tim Bunce
Table of Contents
Index
Sample Chapter
Read Online-Safari Search this
book on Safari:
1
1
1
1
1
1
section
Your Perl program talks to the DBI, and the DBI talks to whichever Database Driver (DBD) is right for
your backend database. This means that to use the DBI, you need to have four things:
A C compiler to compile the XS code for DBI and the DBD drivers.
A copy of the relevant client libraries and header files for the database you want to talk to. For
instance, on my Debian system, to talk to mysql, I install the libmysqlclient10-dev
package.
The relevant DBD library compiled and installed -- for example, DBD::MySQL.
Once that's all up and working, we can start writing some database code using the DBI.
use DBI;
my $dbh = DBI->connect("dbi:mysql:phonebill", $user, $password);
In a lot of cases, you can do without the username and password if you're connecting as a local user.
However, for a serious application, you'll probably want to create a specific user or prompt for a
password.
Now we have connected to the database, DBI returns us a database handle, which is typically stored
into a variable called $dbh. (Of course, if you're connecting to multiple different databases, you may
prefer to give it a name that identifies it to a particular database.) Now we have a database handle,
and we can use it to make queries.
Making a query in the DBI takes place in three stages. First, you prepare some SQL; then you
execute the query; finally, you get the results. Let's do that now:
my $sth = $dbh->prepare(<<SQL);
select recipient, calldate, calltime, duration
from call
where duration > 60
order by duration desc
SQL
$sth->execute;
my %calls;
while (my @row = $sth->fetchrow_array()) {
my ($recipient, $calldate, $calltime, $duration) = @row;
$calls{$recipient} += $duration;
print "Called $recipient on $calldate\n";
}
# Now do something with the total times here.
Why, you might think, do we have to go through these three stages just to make an SQL query? Isn't
Perl supposed to make things easy? Well it does, but it makes different things easy to what you're
expecting. For instance, suppose you're inserting a lot of rows into a table. This is precisely the sort of
thing you don't want to do:
<![CDATA[
while (my $data = <FILE>) {
my ($recipient, $date, $time, $duration) = split /:/, $data;
# DON'T DO THIS
my $sth = $dbh->prepare(<<SQL);
INSERT INTO call (recipient, calldate, calltime, duration)
VALUES ("$recipient", "$date", "$time", "$duration");
SQL
# NO REALLY, DON'T DO THIS
}
]]>
$sth->execute;
There are two reasons why this is BAD, BAD, BAD. The first, of course, is that the moment someone
comes along with a double-quote in the file, we're in big trouble. In fact, the moment someone comes
along with "; DROP DATABASE; in the table, we're out of a job.
The second is that it's really inefficient to set up a statement, execute it, tear it down, set up a
statement, execute it, tear it down, and round we go again.
The reason for the disconnect between preparing a statement and executing it is to enable us to use
the same statement multiple times with slightly different values; we do this by using what DBI calls
"bind parameters" -- portions of the SQL that will be replaced later. For instance, the right way to do
our mass inserts would be something like this:
my $sth = $dbh->prepare(<<SQL);
INSERT INTO call (recipient, calldate, calltime, duration)
VALUES (?, ?, ?, ?)
SQL
while (my $data = <FILE>) {
my ($recipient, $date, $time, $duration) = split /:/, $data;
$sth->execute($recipient, $date, $time, $duration);
}
Isn't that just so much neater? We've hoisted the statement outside the loop, so it only gets prepared
once -- much more efficient. We specify the parameters we want bound to the SQL using question
marks, and we pass in the values to the execute call.
As an additional bonus, when execute substitutes in the bind values to the SQL, it calls the database
handle's quote method on each one; this is a database-specific method, which escapes any nasty
characters like quotes and semicolons in the input, and makes our code safe against the ";drop
database attack.
For SELECT statements, there are a variety of methods that can help out. Perhaps the easiest to use
is selectall_arrayref. This returns the results of the SELECT as an array of arrays:
my $results = $dbh->selectall_arrayref(<<SQL);
select recipient, calldate, calltime, $duration
from call
where duration > 60
order by duration desc
SQL
There are many other DBI tricks, too many to go into here; for more information check out the DBI
documentation page, or the DBI home page; there's also Programming the Perl DBI, which was coauthored by the creator of DBI.