How-to guide: How to evaluate the effectiveness of a data security or data privacy compliance program (USA)

Introduction

This guide will assist in-house counsel, private practice lawyers, and IT departments with how to evaluate the effectiveness of a data security or data privacy compliance program. Alongside other how-to guides, it can assist with developing and implementing compliance programs related to data security and privacy laws, and lead to a better understanding of the ramifications of ineffective compliance programs.

This guide covers:

Overview of the legal framework relating to data security and privacy laws
Objectives and elements of an effective compliance program
Evaluating data security and privacy compliance programs

This guide can be used in conjunction with the following How-to guides: How to manage your organization’s data privacy and security risks and How to determine and apply relevant US privacy laws to your organization and Checklists: Completing a data and information security risk assessment and Completing a data privacy risk assessment.

Section 1 – Overview of the legal framework relating to data security and privacy laws

Data security and data privacy laws are evolving. Without one unified resource, the governing laws tend to be sector-specific. However, privacy standards may also be woven into other laws and regulations. A potential unifying piece of legislation, the American Data Privacy and Protection Act (ADPPA), was first introduced in the House of Representatives on June 21, 2022 but was not enacted. As a result, there is currently no federal law governing online privacy in the United States. For further information, see How-to guide: How to determine and apply relevant US privacy laws to your organization and Checklist: Understanding privacy laws in the US.

1.1 Federal law

At a high level, some of the key federal laws currently in force include the US Privacy Act of 1974 which governs federal agencies, including those under contract with federal agencies; the Health Insurance Portability and Accountability Act (HIPAA) and the privacy rules adopted under that Act which govern entities, such as hospitals, medical services providers, and third-party collections agencies; the Gramm-Leach-Bliley Act (GLBA) which governs businesses engaged in finance; and the Children’s Online Privacy Protection Act which governs website operations for those operating online services directed to children under the age of 13. This is not an exhaustive list: you must research which federal laws apply to your organization.

For further information, see How-to guide: How to determine and apply relevant US privacy laws to your organization and Checklist: Understanding privacy laws in the US.

1.2 State law

As stated in the introduction to this guide, there is currently no overriding data security or privacy federal legislation that would pre-empt state laws, nor is there a Uniform Law on the topic. As a result, states are proceeding on an ad hoc basis resulting in a lack of uniformity in their provisions and related enforcement.

Many states have enacted laws regarding data security, and several have laws regarding the privacy of consumer data. Privacy laws at the state level govern a consumer’s right to access or delete their personal information, to opt out of the collection or sale of their information, and to be notified of the collection and their rights related to the collection. For further information, see Q&A: US Data Protection and Privacy (state-by-state).

1.2.1 State data security laws

With respect to data security, approximately half of US states have laws that address the data security practices of private sector entities. Most require businesses that own, license, or maintain personal information about a resident of that state to implement and maintain reasonable security procedures and practices appropriate to the nature of the information and to protect the personal information from unauthorized access, destruction, use, modification, or disclosure.

1.2.2 State data privacy laws

Various states, including California, Colorado, Connecticut, Delaware, Indiana, Iowa, Kentucky, Montana, Nebraska, New Hampshire, New Jersey, Oregon, Tennessee, Texas, Utah, and Virginia, have enacted comprehensive consumer data privacy laws. In addition, other states – including Massachusetts, Mississippi, New York, and Oklahoma – have proposed new comprehensive consumer privacy laws. If enacted, these states will demonstrate a trend toward new consumer privacy rights and business obligations.

Of the states that have already enacted legislation, the laws have several provisions in common, such as a consumer’s right to access and delete personal information and to opt out of the sale of personal information. Of the states with proposed legislation, these additional states are similar to those states that have already enacted data privacy laws. The wave of new states considering consumer privacy rights mean that many companies will need to reassess their collection and use of personal information, modify their business practices, and develop internal data privacy compliance programs. For further information, see US Data Protection and Privacy (state-by-state).

Section 2 – Objectives and elements of an effective compliance program

To assess the effectiveness of a compliance program, it is important to first understand the objectives of a compliance program and the various elements that may make up a compliance program.

2.1 Objectives of an effective compliance program

A clear and well-understood goal should be set for the compliance program. Then the compliance program can be evaluated according to how well it meets that goal. Ideally, of course, the ultimate goal would be zero data breaches and zero violations of privacy, but that goal may be unrealistic. While the attainment of any goal depends to some extent on outside actors, a realistic goal would be to have no staff errors that lead to a breach. This requires effective prevention, detection, and alignment of behavior from within the organization. These three elements – prevention, detection, and alignment of behavior – generally form the objectives of an effective compliance program. All three objectives are interrelated.

2.1.1 Prevention

The first goal of any type of compliance program is prevention.

For a data security compliance program, prevention means that staff members who handle data must be trained in the correct procedures for securing data. Because many threats to data security are external, prevention also requires that the compliance program has the appropriate technical security control measures in place, and that such control measures are operational. The staff members in charge of that aspect of the program should be evaluated to make certain that they understand the workings of the system and are able to detect deficiencies and areas for improvement.

For further information, see How-to guide: How to develop, implement, and maintain a US information and data security compliance program.

For a data privacy compliance program, prevention means that those who collect data should collect, process, transfer, and retain data in a manner which is compliant with and does not infringe data privacy laws. This means that staff must understand the organization’s legal obligations relating to data and follow internal processes, policies, and procedures designed to ensure compliance.

2.1.2 Detection

Detection is based on the assumption that no matter how effective the prevention aspect of a program is, there is always a non-zero chance of a data breach or a privacy violation. If such a breach occurs, it is important that it is detected as soon as possible. Prompt detection will allow the organization to mitigate the damage from the breach by taking corrective measures. Detection will also allow for an opportunity to prevent similar breaches in the future.

2.1.3 Aligning behavior

Aligning behavior to a corporate culture of privacy by design is another aspect of prevention and detection. An effective compliance program will work only if the staff members who are implementing it apply the program consistently and understand that it is a part of the organization’s mission. Training and retraining are essential parts of aligning behavior; the training must be effective. In addition to requiring training, an organization should test and evaluate staff members’ knowledge of their part of the program.

For further information, see How-to guide: How to implement privacy by design within your organization.

2.2 Elements of an effective compliance program – governmental guidance

There is no single mandated approach to what should be included in an effective compliance program, although there is some guidance from governmental agencies on what organizations should do to have an effective compliance program or the elements that they might include.

While there is no method or type of evaluation that will be appropriate in every circumstance, examples of how compliance programs are evaluated are helpful in designing an organization’s program.

2.2.1 Federal Sentencing Guidelines

Section 8B2.1 of the Federal Sentencing Guidelines sets out the requirements for an effective compliance and ethics program that will mitigate the criminal punishment meted out to an organization. While the Sentencing Guidelines’ requirements are geared toward preventing criminal conduct, some of the requirements provide useful guidance for developing a data security compliance program.

An effective compliance program, for the purposes of the Sentencing Guidelines, means that the organization shall:

exercise due diligence to prevent and detect criminal conduct; and
otherwise promote an organizational culture that encourages ethical conduct and a commitment to compliance with the law.

Due diligence requires an organization’s governing authority to be knowledgeable about the program and exercise oversight. Operational responsibility for the program should be delegated to individuals within the organization. The organization should take reasonable steps:

to ensure that their organization’s compliance and ethics program is followed, including monitoring and auditing to detect criminal conduct;
to evaluate periodically the effectiveness of the program; and
to have and publicize a system that allows the organization’s employees and agents to report or seek guidance regarding potential or actual wrongful conduct without fear of retaliation.

The compliance and ethics program should be promoted and enforced consistently throughout the organization. The goal is to create a ‘culture of compliance.’

2.2.2 Department of Health and Human Services

In 2017, the US Department of Health and Human Services (HHS) issued a Resource Guide on Measuring Compliance Program Effectiveness (Resource Guide). The Resource Guide provides over 550 ideas for measuring the various elements of a compliance program, not all of which will be relevant to every program. The Resource Guide was developed specifically to aid health care organizations, but it will provide assistance to those developing a security and privacy compliance program in any industry.

According to the HHS, the elements of an effective compliance program may be divided into the following seven categories:

standards, policies, and procedures;
compliance program administration;
screening and evaluation of employees, vendors, and other agents;
communication, education, and training on compliance issues;
monitoring, auditing, and internal reporting systems;
discipline for non‐compliance; and
investigations and remedial measures.

The Resource Guide provides numerous ideas about what to measure when considering the effectiveness of a compliance program and the methods of measurement. Some of these ideas may be useful for an organization to employ, depending on the organization’s particular needs. Among the ideas put forward in the Resource Guide is the periodic review of policies, procedures, and controls. Other ideas include maintaining a compliance plan and program and also maintaining policies and procedures for internal and external compliance audits. The need for frequent evaluation and re-evaluation is stressed throughout the Resource Guide.

By way of example, The University of Southern California’s Office of Culture, Ethics, and Compliance’s Data Privacy Compliance Program document notes that the following areas are addressed to ensure effective data privacy compliance: (1) culture governance and compliance oversight; (2) compliance risk identification and assessment; (3) policies, standards, and systems; (4) education, training and outreach; (4) monitoring, auditing and program evaluation; and (5) investigations, corrective action and enforcement.

Education and training are also suggested as ways of measuring the effectiveness of a compliance program; the training must be effective. Some compliance experts not connected with HHS have said that the ultimate goal of training is to mitigate misconduct by altering behavior, but a more immediate goal is to improve the understanding of an organization’s policies. The level of comprehension about these policies can be assessed through testing both before and after the training. Assessments should compare results before and after training, to determine whether those being trained actually learned something they did not know before.

Section 3 – Evaluating data security and privacy compliance programs

Evaluating data security and privacy compliance programs presents a unique set of challenges. Traditionally, compliance programs are directed toward preventing wrongdoing by individuals within an organization. In the absence of preventive methods succeeding, there are also programs geared toward uncovering wrongdoing and taking the appropriate action against the individual or individuals responsible. The focus is on the organization and its staff.

While compliance programs are generally focused on individual wrongdoing, evaluating data security and privacy compliance programs requires examination of not only the deliberate actions of staff members, but also accidental or negligent actions or omissions that could lead to a data breach or other non-compliance.

3.1 Scope and frequency of evaluation

To effectively evaluate a data security and privacy compliance program, an organization must determine the proper scope and frequency of evaluations.

There are many situations that may prompt an organization’s need to evaluate its compliance programs – for example, after a merger, the development of new products or services, or following changes to laws or regulations impacting the organization’s industry or business. Outside of these events, periodic evaluations should be undertaken.

To appropriately determine scope and frequency, organizations should consider a multitude of factors, including the size, risk profile, and available resources of the organization, in addition to the industry and the products or services offered by the organization. In making scope and frequency determinations, organizations should also consider the review process that will be required following the completion of an evaluation itself. Sufficient time and resources must be allocated for a compliance program in order to understand the results of an evaluation and to implement any changes needed prior to the next evaluation.

3.2 Criteria for evaluation

Evaluating any compliance program is an elusive proposition, at best. If the focus is on preventing data breaches or violations of data privacy laws, it may seem as though ‘no news is good news’ – no reports of breaches must mean that the program is working. The flaw with that reasoning is that it assumes the lack of reports is the result of an effective program. The lack of reports could be a sign of dysfunction: employees do not know how to report a violation, or reporting is so cumbersome and confusing that minor violations are overlooked. Therefore, organizations should carefully consider the criteria that they use to evaluate their compliance programs.

3.2.1 Metrics

As stated above, the objectives of a data security and privacy compliance program are prevention, detection, and aligning behaviors.

In order to evaluate the extent to which the objectives of a compliance program are achieved, identify the metrics against which the effectiveness of a program can be assessed.

The HHS Resource Guide (see section 2.2.2 above), for example, lists over 550 indicators for evaluating a compliance program, but also notes that ‘[e]ach organization’s compliance program and effectiveness measurement process will be different.’ Base the decision regarding whether to use any metric, as well as the frequency of use of any metric, on your particular organization and its needs. Take into account the market or business conditions of your organization, any pending business decisions (eg, a merger or the de-acquisition of a division), and external events. For instance, if a large retailer has been the subject of a data breach, other retailers should make a close evaluation of their data security programs. Some metrics, while generally useful, may not apply to an organization’s industry or environment, while others may not be feasible for an organization’s compliance regime.

In selecting metrics, there are various approaches to take. A number of these are set out below as suggestions of how organizations may choose to approach evaluating the effectiveness of a compliance program. No one approach is better than another; organizations should choose an approach or combination of approaches that best suits their needs.

Goal/purpose – organizations may have a number of compliance goals, both short-term and long-term. The selection of appropriate metrics will help support the process of evaluating the effectiveness of the compliance program and the attainment of compliance goals. For example, if your organization’s goal is to increase awareness of your anti-bribery compliance policy, then consider metrics that would include how many staff have received training on the contents of the policy, how many times the policy has been viewed, and how many incidences of non-compliance arose when there was a lack of awareness of company policy.
Business change – change within your organization may prompt the need for evaluation of the compliance program and this may inform the metrics selected. For example, a merger might prompt an evaluation of your compliance program. In this case the initial focus may need to be on acquired staff’s understanding of data security and privacy (with the metrics chosen including the number of training sessions offered, the number of staff who have been trained, compliance courses watched, etc).
Indications of non-compliance – if there have been indications or complaints of non-compliance (whether internally, or externally in your organization’s wider industry) then evaluation efforts may need to start there (eg, metrics might include uses of whistleblower reporting mechanisms or customer or supplier complaints made).
Periodic review – a general commercial review might take a holistic approach and examine different areas in the same evaluation or there might be a focus on particular areas in each review. Periodic reviews might, for example, involve consideration of agreements signed with vendors, agents, and other third parties and metrics might involve considering any due diligence screening or audits undertaken.

The IAPP reported on the increase in metrics being used by a group of Future of Privacy Forum CPOs that were brought together to discuss key issues in their privacy programs. There, it was reported that ‘privacy metrics have emerged as key to measuring and improving privacy program performance and maturity in terms of customer trust, risk mitigation, and business enablement.’ According to the IAPP, this group of CPOs used metrics:

as a way to secure budgets and staffing;
to measure performance;
to diagnose program status and needs; and
as a way to externally demonstrate accountability and enhance trust.

Further, in their Data Privacy Benchmark Study, Cisco reported among other things that as many as ninety-three percent of organizations are reporting privacy metrics (eg privacy

program audit findings, privacy impact assessments, and data breaches) to their Boards. Metrics are becoming an integral part of evaluating data security and data privacy program effectiveness, with an estimated 93% of organizations currently tracking and providing analysis on at least one privacy metric, with 14% utilizing five or more.

Organizations should consider choosing a smaller subset of metrics (see USC OCEC example above) to implement initially and then as questions regarding feasibility continue to emerge, building onto that smaller subset. The implementation of too many metrics all at once will likely prove impractical.

3.2.2 Evaluators

As a best practice, it is valuable to call on an outside evaluator to review the effectiveness of a program. An outside evaluator will be able to approach the program with fresh eyes, without having been involved in the design or implementation of the program.

3.2.3 Legal counsel

The effectiveness of a compliance program is often evaluated after the fact. A common time for an unsparing outside evaluation is when an organization is being sentenced for criminal violations. Federal Sentencing Guidelines recognize ‘the existence of an effective compliance and ethics program’ as one of two factors that will mitigate the criminal punishment meted out to an organization convicted of a violation of federal law (the other factor is ‘self-reporting, cooperation, or acceptance of responsibility’). Legal counsel experienced in the representation of organizations on data security and privacy matters would be well equipped to evaluate the effectiveness of a data security compliance program.

3.2.4 Technical experts

Experts in the technology of data security should be brought in to evaluate the technical efficiency of data security measures. Depending on the expert, the evaluation may involve a simulated breach. The simulated breach will show how easy or hard it is to breach the system, and just as importantly, what the response to the breach is. Detecting vulnerabilities should lead to correction of those vulnerabilities. Detection of vulnerabilities will also show where to watch for future breaches. If a certain type of breach can be anticipated, you can take preventive measures and develop a plan for mitigation.