By Laura S. Quinn, February 2007

Integrating all your constituent tracking applications so they can share data is highly desirable, but how should you go about it? We walk through the pros and cons of three basic methods of integrating data: Manual Import/ Export, Integrated Packages, and Automated Connectors.

It can sound complicated, but it’s a straightforward concept: integrating your constituent data means connecting the information from the various systems you use to communicate with and track your consituents so that your organization can see a unified view. For instance, you might integrate an email blasting software package with your donor database. Linking these systems will allow you to email all your donors (all those with email addresses, at least) from your email blasting tool, and perhaps to view the emails sent and the donor’s responses from within your donor database.

Linking up your constituent data provides some big advantages. For one, it allows you to get a complete view of your constituents. If Joan the development director is keeping her own email list in Constant Contact, and Frank the volunteer coordinator is keeping his volunteer list in an online volunteer management application, neither can take advantage of the potential opportunities in each other’s lists, and you can never get a overall picture of the ways each constituent is involved with your organization.

Integration is also critical in keeping your data clean, updated, and useful. If someone updates a phone number in one system, integration allows that new number to be updated in your other systems. When information has to be updated in real time—for instance, when a website visitor should see different information because they’ve just signed up as a member—integration is the only way to do it.

But is this kind of integration possible in the nonprofit world? Well, sometimes. The options for data integration aren’t as fully developed as one might like. If you’re hoping to integrate the software you already have, you may be limited by the features that are available in those packages. That being said, there a number of methods worth considering.

In this article, we’ll walk through the three basic approaches to combining the data from different sources: Manual Import/ Export, Integrated Tools, and Automated Connectors. Each of these methods has its own strengths, weaknesses, and considerations. While the perfect solution may not exist, we can at least help you understand your options.

Integrating by Manual Import/ Export

A scenario
Joan’s organization is using an event registration system to allow people to sign up for their conference online. They want to be able to see who’s registered in their donor database, however, so they can get a holistic view of not only who’s registered, but who’s donated, who they’ve contacted, and more

On a weekly basis, Joan goes through a checklist of tasks in order to manually integrate this data. First, she goes to the registration software and, using that tool’s reporting features, exports a list of all the people who have registered in the past week. She then opens the exported file in Excel and changes the order of the columns so that the Excel file is in a format that her donor database will understand. Lastly, she goes to her donor database, and uses its upload utility to upload the list of registrants from the Excel file. The donor database uses some combination of fields—perhaps first name, last name and zip code—to identify which registrant should be linked up with which donor.

The details
We’ll call this kind of integration “Integrating by Manual Import/ Export.” In summary, a member of your staff manually exports data from one system and imports it into another on a periodic basis. The time and effort required for this kind of manual integration varies greatly depending on how many data sources you are trying to integrate and how much work is required to get the export file ready for import. In the above scenario, for instance, Joan had to change the order of the columns in order to get the export file into the format required by the donor database. Another system might not require her to do this, or it might require her to do more work – for instance, changing the column labels as well. The scenario above might require about a half-an-hour per week for someone who was familiar with the process.

This option has a number of advantages:

• Lowest initial investment. Manually integrating data doesn’t require an initial investment in programming time or software tools. Instead, the investment is shifted to staff time to plan the integration process and then to perform the manual updates over time. This can make it an affordable way to get started, or to try out various options.
• Allows the widest choice of software tools. Most software packages support at least a basic import and export, so you’ll have the widest variety of software options from which to choose.
• Allows you to target software tools to your specific needs. Integrating several different tools, rather than using a single integrated software package as described below, allows you to optimize functionality and price. If you need very robust donor management functionality, but just basic email blasting capability, you can chose two different tools that meet your precise needs.
• Flexibility in changing software packages down the road. Because there’s minimal upfront cost, your data integration strategy doesn’t tie you to any particular piece of software. If you need to swap out your event registration tool for another one, for instance, the additional investment for integrating the new tool is likely to be one of hours rather than days.

But the method also has notable drawbacks:

• Requires greatest amount of ongoing staff time. While the other methods allow you to set up the integration process up-front and then let it run mostly untended, manual integration requires a continuing chunk of time from a staff member.
• Opens the door to human error and data issues. This option depends on a staff member consistently running through a potentially complicated series of steps. Over time, human nature means that mistakes are bound to occur, which can create incorrect data, duplicates, or missing information.
• Requires careful integration plan and process. Like with the Automated Connector option below, manual integration will require careful initial planning. You’ll need to define what fields you will use to link the data (i.e. can you match constituents based on email address? Name and zip code?). It’s also important to define which system will be the primary owner of each piece of data— for instance, in Joan’s scenario, the event registration system was the owner of the event registration list and the most up-to-date registrants, but the donor database was the owner of the full constituent list for the organization. Think carefully through the process—what happens when constituents are added? Removed? When information changes?
• Time lag in information sharing. The data in one system will slowly go out-of-synch with the other in the time period between manual updates. For instance, in our example above, the donor database is only updated with new registrants once a week, while the event registration tool shows new registrants immediately after they register. This type of delay can become problematic in specific circumstances – for instance, if you want to allow constituents to login to multiple systems with the same password, or to show them data from one system based on information from another.
• Relies on a single staff member. One staff member typically does all the manual integration, which can lead to problems if that staff member goes on vacation or leaves the organization. If that person becomes overworked or frustrated, the integration process can stop altogether. Careful documentation and training multiple people to do the manual process can help.
• Integrating many data sources can become prohibitively complicated. Trying to manually integrate several different databases requires juggling many data formats and a multitude of files. While carefully documenting the exact sequence of steps and tasks can help, the limitations of human concentration mean that manual integration can’t scale to the same level as the other options.

In general, this is a good option if you’re looking to inexpensively integrate just a few data sources, without a big initial investment. It’s also a practical way to get started in understanding what tools you need and what data needs to be integrated. Don’t underestimate the staff time required, however, particularly as the number of data sources increase. And don’t discount the danger of accidentally introducing problems into your data through inevitable human error.

What does a software package need to provide to support manual integration? Make sure that the tool allows you to export and import the actual fields you are trying to synch up. Flexibility in the column order and the file names used for the import/ export files is also very useful. If you are trying to integrate a package that will only export in one specific format with one that will only import in a different specific format, you will have to transform the data file each time you want to synch the data. In addition, the ideal system will automatically check for duplicate records after each import, enforce your data entry rules (e.g., states must be upper-case), and allow you to review changes before they are posted.

Using Integrated Packages

A scenario
George’s organization needs to be able to accept both online and offline donations, mail direct appeals, send emails, register people for events, and track and manage their list of constituents. They’ve chosen one software package that supports all of those functions. This means that George uses a single online administration tool to see and manage all the information about each of the constituents with which they’ve interacted.

The details
The challenge of sharing data between multiple applications has led to a whole new category of software: integrated packages. These applications provide many different functions that all use the same central database. Your information is completely synched up and interconnected because you’re only using one application as opposed to many different ones which may not quite talk to each other.

If you’re looking to do a wholesale integration effort, it’s certainly worth considering moving all your data into a single integrated package. This has a number of substantial advantages:

• No special integration effort or issues. The most obvious advantage is that, essentially, data integration is taken care of for you. Because all information for these various functions is stored in one central database, no staff time or planning is needed to synch across multiple tools.
• Much lower danger of data inconsistencies. The central database, and the fact that no data is actually being moved, makes the likelihood of data problems much lower with this option than with others.
• Common functionality can be “plugged in.” The larger integrated packages provide functionality for most common needs—either out-of-the-box, or at an additional charge. This allows you to conveniently add on functionality down the road, without the need to search for another software package.
• A single contact for support. If you have a question or problem, it’s easy to figure out who to call. You’re only working with one vendor.
• Data is updated in real time. As all the data is in the same database, there is never a time when any information is out of synch. This is important if you want to allow constituents to be able to login to multiple systems with the same password, or to show them information from one system based on information from another (so for instance, if you want to allow them access to a member section of your website immediately upon registering as a member).

But an integrated package is not the solution for every organization. There can be some distinct drawbacks, depending on your situation and needs:

• Requires you to migrate all existing data. For those who are established with constituent communications or tracking applications already, there’s a huge downside to integrated applications: you need to abandon your existing software and move all of your data into the integrated tool. Make sure everyone on the team is ready to give up your current software and to learn to use a new tool. On the other hand, if you’re operating from Excel spreadsheets and slips of paper, the move to a new system may provide the impetus and rallying cry needed to centralize all your data into a usable form.
• Jack of all trades, master of none. Most integrated software packages are built around a core functionality – for instance, event management or email blasts. While they have built on many additional functions, these features are often not nearly as strong as the core ones. If you need features in three divergent areas, you’re likely to get more robust functionality in three separate tools rather than one integrated package.
• High dependency on a single vendor. When you’re relying on a single vendor for a lot of critical functions, it’s a big mess if that vendor goes out of business or moves in a direction unfavorable to your needs.
• Difficulty in extending functionality. It may not be easy to fit in functionality that the integrated package hasn’t planned for. While some packages offer partnerships with other specialist tools, moving beyond the functionality provided by the out-of-the-box application may require either costly customization, or connecting your integrated application with additional software tools though either a Manual Import/ Export or Automated Connectors integration method.
• Sizable ongoing cost. While technically-savvy organizations may be able to make use of free open source solutions in this area, most organizations will need to “rent” a hosted application for a monthly fee. Fees vary widely depending on the features available, but can range from about $50 to thousands of dollars per month.

If your needs fall into common scenarios (for instance, you’re strongly focused on advocacy, online outreach, or serving a base of members) and you don’t have strong established systems already, an integrated tool can be a great way to go. If you’re thinking through options for a new organization, or you’d like to pull together a number of diverse datasets, an integrated tool is well worth considering. But make sure you consider the drawbacks: you’ll be dependant on the success of a single vendor, and you’ll likely have less flexibility to meet unusual needs.

When looking at integrated packages, it’s critical to consider the range of needs that they cover. Because the key benefit is the all-in-one nature, try to find a package that meets not only your current needs, but also likely future needs. In addition, ask careful questions about the user base and long term viability of the vendor (see our article Vendors as Allies: How to Evaluate Viability, Service, and Commitment ). If you’re going to put all your eggs in one basket, make sure that it’s a solid basket.

Integrating through Automated Connectors

A scenario
Let’s go back to Joan’s organization, who wanted to synch up their event registration information with their donor database. Over time, they found that this integration was very important to them, but the manual import/ export process was not working as well as it could. It took too much of Joan’s time each week, and they really wanted to be able to use their donor database to see a list of event registrants that was always up-to-date.

So Joan hired a programmer to create an automated connection between the two systems. Now, whenever someone registers for an event, an automatic process is kicked off. The registrant’s information is sent to the donor database, and the records in the donor database are reflected to show the registration in real time.

The details
In many ways, integrating through automated connectors is similar to Manual Import/ Export. Data is pulled from one application and added in to another. Instead of doing this process by hand, however, an automated connector allows the data to flow from one system from another without human intervention.

How do you make this automated magic happen? Some high-end software packages provide out-of-the-box automated connectors which allow them to easily talk to particular other software packages. And for some tools, particularly open source ones, you can find community-built modules that can be used to create automated connectors with minimal setup. But most of the time, creating a connector will require a development process—careful design, programming, and testing to create a program that synchs up the data between the two systems.

Not every software package can be integrated by automated connector, however. In order to create a connector, you need to be able to access the software’s data via code. This means that the software vendor needs to provide an application programming interface (API)—a way to request or provide data via code—or otherwise expose the data. Some software packages, provide this type of access to the database, but many do not. Others charge to be able to use an API. Some provide access, but in ways that makes it complicated to integrate. What your software tool allows is a key consideration in thinking through whether this method is right for you.

This is a powerful integration method, with a number of sizable advantages:

• Can use targeted, best-of-breed tools. As discussed above, tools that concentrate on doing a single thing well are often more powerful than integrated packages. Selecting the very best existing tools for your needs and then integrating them yourself is likely to provide the most robust support for sophisticated needs.
• More options for integration and aggregation. If you’re building your own connectors, you have precise control over what the integration process does. This also allows flexibility for more sophisticated data aggregation strategies—for instance, if you are integrating many systems, it might make sense to feed the data into a single, central “data warehouse” application rather than connect every system to every other system.
• Minimal ongoing investment. While there are substantial up-front costs, this is the option that has the lowest ongoing cost. There’s unlikely to be any ongoing fees associated specifically with data integration, and staff time is required only to oversee the process, maintain the connectors, and make any updates based on application upgrades, rather than investing substantial time in manually updating data.
• Data can be updated in real time. Depending on the type of data access the vendor provides and the type of connector you build, it’s possible to create connectors that synchronize information in multiple systems in real time. As discussed above, this is useful in allowing users to login to multiple systems with the same password, or to show them information from one system based on information from another.
• Flexibility in swapping out software packages down the road. This option provides middle-of-the-road flexibility for swapping out software—more so than using an Integrated Package but less than with the Manual Import/ Export. If you need to move to a new event registration tool, for instance, you’re able to do so, but you’ll need to invest in building a new connector.

If a connector has already been created to integrate the packages you’re looking at, then all of these advantages come with very little downside. For instance, many high-end donor databases have partnered with online donation or event management tools to seamlessly integrate the two software tools via automated connector. Other tools, like Salesforce.com or open source applications allow community members to create and share the code to integrate additional software packages. It’s highly desirable to have such pre-existing connections, and worth looking for as you’re selecting software.

If no pre-existing connector exists, however, you’ll have to build this interface yourself. While certainly possible (assuming, of course, that you can access the software’s data via code), this is not a task to be undertaken lightly, and involves some substantial drawbacks:

• High setup time or cost. You’ll need to either code the automated connector in house, or hire someone to do so—a significant investment. You’ll need to carefully plan the technical architecture (i.e. exactly how do the systems talk to each other?), design the business rules (which system has the definitive record? what happens with duplicates), build the connector, and then thoroughly test it.
• Dangerous if done poorly. It’s critical to invest in carefully planning and testing automated connectors. A poorly executed connector can be much more damaging than a poorly thought out manual import/ export process, as there are likely to be fewer controls to stop you from doing things that will “break” your data, and it could touch thousands of records before anyone spots the problem.
• Maintenance. As with anything you build yourself, you’ll need someone on call to monitor your data, make sure your connectors are working properly, update the connectors to any application upgrades, and troubleshoot issues.
• Ties you to particular packages. If you invest in building a connector to a specific package, it’s harder to replace that package to meet new needs or if the vendor goes out of business.

An automated connector between best-of-breed applications is for most organizations the most desirable integration method—but building it yourself requires a considerable investment. When looking for software, factor in existing connectors to other useful software packages as a huge benefit. If a solid automated connector already exists, you get all the benefits of this powerful integration method with little of the downside.

If no connector exists, however, does it make sense to build your own? This is a good option for tech savvy organizations who are ready invest in data integration, especially those who find that their needs are not well met by the existing integrated packages, or are committed to the tools they are already using.

If you’re looking at software packages with an eye towards ones that can be integrated via automated connector, it’s critical to determine whether the package offers a method to access the data via code. Ask your technical expert to evaluate the technical specification to make sure it’s compatible with the packages with which you’d like to integrate it. If real time integration is important to you, confirm that you have the necessary system access to make that possible.

Which One is Right for You?

Which option is the right choice for you? As always, it depends. There are valid reasons to go with any of the three of these. You’ll need to carefully evaluate your own needs and concerns, but the following decision tree can help you navigate some of the most common reasons for choosing one option over another.

Whichever option you choose, don’t underestimate the planning element in all of this. Before you purchase any software, make sure you have a data strategy for the new system–how will the data relate to your existing data? How will it be maintained? Remember that no system or option can keep your data clean and useful by itself. Someone – or multiple people – in your organization will have to take ownership of the data, set data entry standards, and keep watch to make sure that information flows usefully across the organization.

Data integration isn’t a trivial undertaking, but it’s well worth the effort. Just imagine it: the assurance that the data in any system is the most accurate and up-to-date information you have. The ability to target communications based on a full picture of each person’s involvement. And best of all… increasing your ability to truly understand and connect with your constituents.

For More Information

Getting it All in One Place: Data Integration Strategies
The powerpoint presentation from a seminar, and several other articles, that talk about various methods of using automated connectors

The Integration Proclamation
Unhappy with the integration options available? Sign the Integration Proclamation to put your support behind an initiative that is pushing for vendors and funders to help make integration easier

As Idealware’s Founder and Director, Laura S. Quinn directs Idealware’s research and writing to provide candid reports and articles about nonprofit software. Prior to Idealware, Laura provided website strategy, navigation, and online knowledge management consulting for nonprofits. Laura is a frequent speaker and writer on nonprofit technology topics.

Paul Hagen, an independent strategic technology consultant to nonprofits, Tate Hausman of Dot Organize, Beth Kanter of Beth’s Blog, and Robert Weiner of Robert L Weiner Consulting also contributed to this article. 

 copyright© 2007 Idealware

Sign Up for Monthly eNews

Donate What You Feel These Resources Are Worth!

Have the Idealware reports and articles saved you time? Money? Allowed you to make better decisions? If so, please donate what they are worth to you and help keep these resources coming. Donate now >