Text and Data Mining (TDM)

Information for staff and students undertaking text and data mining (TDM) as part of their research.

On

Introduction

Text and data mining is the process of extracting new information from existing files, usually using computational methods. There are several steps to most TDM activity, including data cleansing and indexing, but the first step is to gather the sources to be used. A temporary copy of the files is often made to enable the content to be processed, and this has implications for copyright. 

We explore how the exceptions work in more detail below, however key points to be aware of for researchers are:

  • Statutory exceptions for TDM purposes only apply in the case of non-commercial research projects - publicly-funded research (for example from UKRI research councils) will normally qualify, however projects funded by external commercial partners, spin-outs, and patent research will usually not benefit from exceptions
  • Exceptions do not permit bypassing technological protection measures (TPMs) - you should not bypass copy-protection measures, log-in screens, human-verification challenges (CAPTCHAs) or website paywalls
  • It is important to respect vendor platform rules and contracts - in particular requirements such as API key requirements, rate-limits and access time windows, and terms limiting use within genAI software models
  • You should never share or publish copies made for TDM in reliance on exceptions - underlying corpora materials copied for TDM must not be further shared or communicated, although your own analysis code and research findings, such as frequency data, can normally be freely published and shared

Copyright and TDM

An exception to copyright (section 29A of the Copyright, Designs and Patents Act), introduced into UK law in 2014, allows you to make copies of works to which you have ‘lawful access’ for the purpose of text and data analysis, more commonly referred to as TDM. This must be as part of non-commercial research. The exception does not permit TDM for any directly or indirectly profit-making purpose.

For example: it is fine to rely on the section 29A exception if doing TDM on copyright works as part of a research project supported by public funding (such as from UKRI research councils), where the expected output will be traditional research papers, journal articles and scholarly publications. 

You could not, however, rely on the exception if your research is funded by a commercial partnership, or will feed into a commercial spin-out, and where the commercial partner might directly or indirectly profit from the research or any datasets created. Equally, TDM research to directly support a commercial patent application would be unlikely to benefit from the exception. In such scenarios you would require a licence from the rights owner.


Lawful access

Lawful access to works can take many forms. For example, it may be material we have access to via a Library subscription, or you may own a copy. When using publicly available works for TDM - such as material freely available online - you should ensure as far as possible that the source you use is lawfully available with the rights holder’s consent, and is not an infringing copy posted without authorisation. 

When accessing materials lawfully under contract or licence, content providers can apply reasonable measures to ensure stability and security of their networks, though any terms and conditions that directly aim to stop you undertaking TDM on a copyright work are unenforceable. 

The section 29A TDM copyright exception does not permit the override or bypassing of any technological protection measures (TPMs) in order to access and copy works. TPMs can include measures such as log-in screens and anti-copying software code.

Note: database and subscription content providers can enforce contractual access restrictions on their lawful users to control how TDM activity is undertaken. Typically these include download rate limits or API key access requirements. As these are not direct prohibitions on TDM they are not overridden by the section 29A(5) contract enforceability clause, and you must adhere to any such provider terms. See our guidance on TDM on University subscribed resources for examples.

Lawful access and generative AI tools

Care needs to be taken if you are planning to use genAI tools as part of a TDM project on subscribed resources. Database providers and copyright owners can potentially impose restrictions or prohibitions on certain genAI uses of their content. If those terms are not a blanket prohibition on TDM of a copyright work then they will be binding on subscribed users, and restrict lawful use to compliance with the contractual conditions. Check the terms of any subscribed content before uploading to a genAI tool. This includes using content with University-approved Gemini apps.


Transfer of copies

If your research project involves collaboration with researchers at another UK higher education institution, you may be able to rely on our Copyright Licensing Agency licence for very limited sharing of materials to be analysed. This is restricted to amounts within our usual CLA copying limits, so it might allow sharing a couple of articles from a journal issue, or of a book chapter, but it does not extend to making any larger corpus available to collaboration partners. The CLA terms require the research partners to be based at an institution that also holds a CLA Higher Education licence. Please contact us if you wish to discuss in more detail what is permitted. You should not otherwise share any copies of materials that you have made for the purposes of TDM.

While you cannot share copies of underlying material, the outputs you create from your analysis are under your control. You are usually free to share or publish your new analysis data provided you do not transfer or communicate any copies of the underlying works.

For funded research projects, publishing the results of your research open access may be a requirement of any applicable funder mandate. For this reason it is important to note and avoid any contractual restrictions on the reuse of your new analysis data, which some rights holders may attempt to impose as part of granting you lawful access to their collections.

Note: funder data archiving requirements may apply to statistical data derived from TDM such as frequency analysis data, but they do not apply to underlying corpora materials copied for the purposes of TDM. Such copies must not be further shared or communicated.

Retaining private and securely stored copies from a TDM project for your own reproducibility and verification of results purposes is generally low-risk, but take note of any provider contractual provisions around retention of underlying copies. Some providers specify that all copies made for TDM purposes must be deleted at the end of the research project, and such terms are contractually binding.


TDM and database right

Database right is a related right to copyright, introduced into UK law by the Copyright and Rights in Databases Regulations 1997 (CRDR). The right protects databases where there has been significant investment in obtaining, verifying or presenting the database’s contents. The right can apply to a database as a whole, irrespective of whether the individual contents themselves qualify for intellectual property rights protection. 

The section 29A copyright exception detailed above does not apply to protected databases. This is true even where the entire contents of the database consist of individually copyright works. There is a fair dealing exception to the database right at regulation 20 of the CRDR, which allows extraction (i.e. copying to another medium) but not further reutilisation (i.e. no onward communication). This extraction must be by a person with lawful access to the database, must be made solely for a non-commercial teaching or research purpose, and the data source must be properly indicated.

The fair dealing exception might permit TDM of a lawfully accessed database, though note this is an untested grey area and so caution is advised. There is no explicit contractual override provision in the regulation 20 exception, so checking the terms and conditions of any database you wish to use for TDM is essential. 

Any extraction and reuse of data that harms the investment of the database owner - which includes both investment of resources or time in obtaining and verifying the content data, and also investment in presenting and ensuring the stability and security of the database platform - will likely infringe the database right.


TDM and the public domain

Works in the public domain can usually be freely reused and copied for any purpose, as copyright no longer applies to them. Be aware that if the materials you wish to use for TDM are made available to you under contractual terms, and consist of works that are free of copyright and related rights, then the exception for TDM does not apply. Your use must keep to the terms of your contract or licensing agreement in such circumstances.


Additional guidance

University of Sheffield policies

The University of Sheffield Good Research and Innovation Practice (GRIP) policy provides further guidance on the principles and good practices governing all the University's research and innovation activities. GRIP Designing research and innovation projects and GRIP Managing research data and code, and FAIR principles sections of the policy provide further detail to ensure that your research complies with all legal, ethical and contractual requirements, including in regard to text and data mining.

Note: Inputting a third party’s data into AI and GenAI platforms, without prior permission, may contravene the terms of use. See Principles for Using GenAI in Research and Innovation.

CopyrightUser guide to data scraping and data mining

CopyrightUser is “an independent online resource aimed at making UK Copyright Law accessible to creators, media professionals, entrepreneurs, students, and members of the public”. 

In addition to their guidance page on TDM, they provide an in-depth guide to issues around TDM and data scraping, including scraping of personal data, copyright resources, and sui generis database right protected resources.

UK IPO exceptions guide

In October 2014, following a major update to the CDPA, the Intellectual Property Office (IPO) published a guide to the copyright exceptions related to research, including section 29A.

You can read the guide here.

Related information