Category Archives: Legal issues

  • -
A model for the development of a code of practice

Paper submitted to Journal of Learning Analytics: “Developing a Code of Practice for Learning Analytics”

I’ve just submitted a paper to a forthcoming “Special Section on Ethics and Privacy” in the Journal of Learning Analytics (JLA).  The paper documents the development of Jisc’s Code of Practice for Learning Analytics through its various stages, incorporates the taxonomy of ethical, legal and logistical issues, and includes a model for developing a code of practice which could be used in other areas.

A model for the development of a code of practice

A model for the development of a code of practice

As an open journal the JLA suggests that authors publish their papers before or during the submission and review process – this results in the work getting out more quickly and can provide useful feedback for authors. So here’s the paper – and if you have any feedback it would be great to hear from you.

Abstract
Ethical and legal objections to learning analytics are barriers to development of the field, thus potentially denying students the benefits of predictive analytics and adaptive learning. Jisc, a charitable organisation which champions the use of digital technologies in UK education and research, has attempted to address this with the development of a Code of Practice for Learning Analytics. The Code covers the main issues institutions need to address in order to progress ethically and in compliance with the law. This paper outlines the extensive research and consultation activities which have been carried out to produce a document which covers the concerns of institutions and, critically, the students they serve. The resulting model for developing a code of practice includes a literature review, setting up appropriate governance structures, developing a taxonomy of the issues, drafting the code, consulting widely with stakeholders, publication, dissemination, and embedding it in institutions.

 


  • -
Media City Uk

Code of Practice for Learning Analytics launched

Today Jisc is launching the Code of Practice for Learning Analytics at the UCISA Spotlight on Digital Capabilities event here in the amazing MediaCityUK at Salford Quays.

Media City UK, Salford

Developing this document was chosen by institutions as the number one priority for Jisc’s learning analytics programme.  The Code aims to help universities and colleges develop strategies for dealing with the various ethical and legal issues that may arise when deploying learning analytics.

Code of Practice coverIt’s a brief document of four pages and is available in HTML or in PDF. The development of the Code was based on a literature review of the ethical and legal issues. From this a taxonomy of the ethical, legal and logistical issues was produced. The Code was drafted from this taxonomy and is grouped into seven areas:

  1. Responsibility – allocating responsibility for the data and processes of learning analytics within an institution
  2. Privacy – ensuring individual rights are protected and data protection legislation is complied with
  3. Validity – making sure algorithms, metrics and processes are valid
  4. Access – giving students access to their data and analytics
  5. Enabling positive interventions – handling interventions based on analytics appropriately
  6. Minimising adverse impacts – avoiding the various pitfalls that can arise
  7. Stewardship of data – handling data appropriately

The Code was developed in the UK context and refers to the Data Protection Act 1998 however most of it is relevant to institutions wishing to carry out learning analytics anywhere, particularly in other European countries which have similar data protection legislation. It can be adopted wholesale or used as a prompt or checklist for institutions wishing to develop their own learning analytics policies and processes.

If you find the document helpful or feel that anything is unclear or missing please let us know. Keeping it concise was thought to be important but that meant leaving out more in-depth coverage of the issues. Over the coming months we’ll be developing an associated website with advice, guidance and case studies for institutions which wish to use the Code.

Acknowledgements
The process has been overseen by a Steering Group consisting of Paul Bailey (Jisc), Sarah Broxton (Huddersfield University), Andrew Checkley (Croydon College), Andrew Cormack (Jisc), Ruth Drysdale (Jisc), Melanie King (Loughborough University), Rob Farrow (Open University), Andrew Meikle (Lancaster University), David Morris (National Union of Students), Anne-Marie Scott (Edinburgh University), Steven Singer (Jisc), Sharon Slade (Open University), Rupert Ward (Huddersfield University) and Shan Wareing (London South Bank University).

It was particularly good to have the student voice represented in the development of the Code by David Morris of the NUS. I’m also especially grateful to Andrew Cormack and Rupert Ward for their perceptiveness and attention to detail on the final draft. I received additional helpful feedback, most of which I was able to incorporate, from the following people (some in a personal capacity, not necessarily representing the views of their organisations):

Helen Beetham (Higher Education Consultant), Terese Bird (University of Leicester), Crispin Bloomfield (Durham University), Alison Braddock (Swansea University), Annemarie Cancienne (City University London), Scott Court (HEFCE), Mike Day (Nottingham Trent University), Roger Emery (Southampton Solent University), Susan Graham (Edinburgh University), Elaine Grant (Strathclyde University), Yaz El Hakim (Instructure), Martin Hawksey (with other members, Association for Learning Technology), Ross Hudson (HEFCE), John Kelly (Jisc), Daniel Kidd (Higher Education Statistics Agency), Jason Miles-Campbell (Jisc), George Munroe (Jisc), Jean Mutton (Derby University), Richard Puttock (HEFCE), Katie Rakow (University of Essex), Mike Sharkey (Blue Canary), Sophie Stalla-Bourdillon (Southampton University), Sarah Stock (University of Essex) and Sally Turnbull (University of Central Lancashire).

Finally, many thanks to Jo Wheatley for coordinating the production of the print and HTML versions of the Code.

 


  • -
code

Code of Practice for Learning Analytics – public consultation

Jisc’s draft Code of Practice for Learning Analytics is now available for public consultation for the next two weeks: Code of Practice for Learning Analytics v04 [MS Word].  I’ve already had some very useful comments back from a number of invited organisations and individuals which will help to enhance the document and the accompanying online guidance.

The background to the Code is provided in an earlier blog post.  The deadline for comments from the public consultation is 5th May.  These will then be presented to the Code of Practice Advisory Group which will agree a final version.

I’d be very grateful for any further thoughts from individuals or organisations either by commenting on this blog post or emailed to me at niall.sclater {/at/} jisc.ac.uk

We intend to release the final version of the Code in June and will be developing accompanying online guidance and case studies over the coming months.


  • -
Paris

A taxonomy of ethical, legal and logistical issues of learning analytics v1.0

Jisc, Apereo and the Lace Project held a workshop in Paris on 6th February to discuss the ethical and legal issues of learning analytics.  The focus of this meeting was the draft taxonomy of issues that I prepared previously.  It was extremely helpful to have comments from experts in the area to refine the list, which is forming the basis for Jisc’s Code of Practice for Learning Analytics.  I have subsequently reworked the taxonomy based on the group’s comments.

Paris

Re-ordering
I’ve now re-ordered the table to reflect a slightly more logical lifecycle view of learning analytics moving from issues of ownership and control to seeking consent from students, ensuring transparency, maintaining privacy, ensuring validity in the data and the analytics, enabling student access to the data, carrying out interventions appropriately, minimising adverse impacts and stewarding the data.

Type
I’ve added a “Type” column which states whether the issue is primarily one of ethics, legalities or logistics.  It’s become clear to me that many of the issues in the literature around ethics and privacy for learning analytics are more about the logistics of implementation than about doing what’s right or keeping within the law.  I’ve therefore renamed the taxonomy to reflect the fact it’s about logistics as well.

Rank
The Paris group suggested scoring the issues on the basis of their importance and we began to rate them on a scale of 1 to 5, highlighting the most important ones.  I have subsequently reduced the scale to three points, roughly equating to: 1 – Critical; 2 – Important; 3 – Less important / may not arise.  I have reflected the views of the group in the rankings but have had to make many choices as to their relative importance myself.  I’d like to find some more rigorous way of rating the issues though the ranking will always be dependent on the nature and priorities of the institution.

Responsibility
The group added a stakeholder column.  Subsequently I divided this into Stakeholders most impacted and Stakeholders responsible.  I then found that the most impacted stakeholders were almost always students so the column wasn’t particularly helpful and I’ve just included a Responsibility column which shows who is primarily responsible for dealing with the issue. Again there’s a level of subjectivity here on my part and these roles will be constituted differently depending on the institution. I’ve listed six stakeholders:

  1. Senior management – the executive board of the institution.
  2. Analytics committee – the group responsible for strategic decisions regarding learning analytics. This might be a learning and teaching committee, though some of the issues may be the responsibility of a senior champion of learning analytics rather than a more representative commmittee.
  3. Data scientist – while the analytics committee may decide on particular issues, there is a need for data scientists or analysts to advise on issues relating to the validity of the dataset and how to interpret it.
  4. Educational researcher – some issues would be best dealt with by staff with detailed knowledge of the educational issues who are able to monitor the impact of analytics on students.  This role may be carried out by teachers or tutors or those more dedicated to educational research.
  5. IT – the institutional information technology department will take primary responsibility for some aspects of the analytics processes.
  6. Student – while students are potentially impacted by almost every issue here, they are primarily responsible themselves for dealing with a few of them.
Group Name Question Type Rank  Responsibility
Ownership & Control Overall responsibility Who in the institution is responsible for the appropriate and effective use of learning analytics? Logistical 1 Senior management
Control of data for analytics Who in the institution decides what data is collected and used for analytics? Logistical 1 Senior management
Breaking silos How can silos of data ownership be broken in order to obtain data for analytics? Logistical 2 Analytics Committee
Control of analytics processes Who in the institution decides how analytics are to be created and used? Logistical 1 Analytics Committee
Ownership of data How is ownership of data assigned across stakeholders? Legal 1 Analytics Committee
Consent When to seek consent In which situations should students be asked for consent to collection and use of their data for analytics? Legal / Ethical 1 Analytics Committee
Consent for anonymous use Should students be asked for consent for collection of data which will only be used in anonymised formats? Legal / Ethical 3 Analytics Committee
Consent for outsourcing Do students need to give specific consent if the collection and analysis of data is to be outsourced to third parties? Legal 3 Analytics Committee
Clear and meaningful consent processes How can institutions avoid opaque privacy policies and ensure that students genuinely understand the consent they are asked to give? Legal / Ethical 1 Analytics Committee
Right to opt out Do students have the right to opt out of data collection and analysis of their learning activities? Legal / Ethical 1 Analytics Committee
Right to withdraw Do students have the right to withdraw from data collection and analysis after previously giving their consent? Legal 3 Analytics Committee
Right to anonymity Should students be allowed to disguise their identity in certain circumstances? Ethical / Logistical 3 Analytics Committee
Adverse impact of opting out on individual If a student is allowed to opt out of data collection and analysis could this have a negative impact on their academic progress? Ethical 1 Analytics Committee
Adverse impact of opting out on group If individual students opt out will the dataset be incomplete, thus potentially reducing the accuracy and effectiveness of learning analytics for the group Ethical / Logistical 1 Data scientist
Lack of real choice to opt out Do students have a genuine choice if pressure is put on them by the insitution or they feel their academic success may be impacted by opting out? Ethical 3 Analytics Committee
Student input to analytics process Should students have a say in what data is collected and how it is used for analytics? Ethical 3 Analytics Committee
Change of purpose Should institutions request consent again if the data is to be used for purposes for which consent was not originally given? Legal 2 Analytics Committee
Legitimate interest To what extent can the institution’s “legitimate interests” override privacy controls for individuals? Legal 2 Analytics Committee
Unknown future uses of data How can consent be requested when potential future uses of the (big) data are not yet known? Logistical 3 Analytics Committee
Consent in open courses Are open courses (MOOCs etc) different when it comes to obtaining consent? Legal / Ethical 2 Analytics Committee
Use of publicly available data Can institutions use publicly available data (e.g. tweets) without obtaining consent? Legal / Ethical 3 Analytics Committee
Transparency Student awareness of data collection What should students be told about the data that is being collected about them? Legal / Ethical 1 Analytics Committee
Student awareness of data use What should students be told about the uses to which their data is being put? Legal / Ethical 1 Analytics Committee
Student awareness of algorithms and metrics To what extent should students be given details of the algorithms used for learning analytics and the metrics and labels that are created? Ethical 2 Analytics Committee
Proprietary algorithms and metrics What should institutions do if vendors do not release details of their algorithms and metrics? Logistical 3 Analytics Committee
Student awareness of potential consequences of opting out What should students be told about the potential consequences of opting out of data collection and analysis of their learning? Ethical 2 Analytics Committee
Staff awareness of data collection and use What should teaching staff be told about the data that is being collected about them, their students and what is being done with it? Ethical 1 Analytics Committee
Privacy Out of scope data Is there any data that should not be used for learning analytics? Ethical 2 Analytics Committee
Tracking location Under what circumstances is it appropriate to track the location of students? Ethical 2 Analytics Committee
Staff permissions To what extent should access to students’ data be restricted within an institution? Ethical / Logistical 1 Analytics Committee
Unintentional creation of sensitive data How do institutions avoid creating “sensitive” data e.g. religion, ethnicity from other data? Legal / Logistical 2 Data scientist
Requests from external agencies What should institutions do when requests for student data are made by external agencies e.g. educational authorities or security agencies? Legal / Logistical 2 Senior management
Sharing data with other institutions Under what circumstances is it appropriate to share student data with other institutions? Legal / Ethical 2 Analytics Committee
Access to employers Under what circumstances is it appropriate to give employers access to analytics on students? Ethical 2 Analytics Committee
Enhancing trust by retaining data internally If students are told that their data will be kept within the institution will they develop greater trust in and acceptance of analytics? Ethical 3 Analytics Committee
Use of metadata to identify individuals Can students be identified from metadata even if personal data has been deleted? Legal / Logistical 2 Data scientist
Risk of re-identification Does anonymisation of data become more difficult as multiple data sources are aggregated, potentially leading to re-identification of an individual? Legal / Logistical 1 Data scientist
Validity Minimisation of inaccurate data How should an institution minimise inaccuracies in the data? Logistical 2 Data scientist
Minimisation of incomplete data How should an institution minimise incompleteness of the dataset? Logistical 2 Data scientist
Optimum range of data sources How many and which data sources are necessary to ensure accuracy in the analytics? Logistical 2 Data scientist
Validation of algorithms and metrics How should an institution validate its algorithms and metrics? Ethical / Logistical 1 Data scientist
Spurious correlations How can institutions avoid drawing misleading conclusions from spurious correlations? Ethical / Logistical 2 Data scientist
Evolving nature of students How accurate can analytics be when students’ identities and actions evolve over time? Logistical 3 Educational researcher
Authentication of public data sources How can institutions ensure that student data taken from public sites is authenticated to their students? Logistical 3 IT
Access Student access to their data To what extent should students be able to access the data held about them? Legal 1 Analytics Committee
Student access to their analytics To what extent should students be able to access the analytics performed on their data? Legal / Ethical 1 Analytics Committee
Data formats In what formats should students be able to access their data? Logistical 2 Analytics Committee
Metrics and labels Should students see the metrics and labels attached to them? Ethical 2 Analytics Committee
Right to correct inaccurate data What data should students be allowed to correct about themselves? Legal 1 Analytics Committee
Data portability What data about themselves should students be able to take with them? Legal 2 Analytics Committee
Action Institutional obligation to act What obligation does the institution have to intervene when there is evidence that a student could benefit from additional support? Legal / Ethical 1 Analytics Committee
Student obligation to act What obligation do students have when analytics suggests actions to improve their academic progress? Ethical 2 Student
Conflict with study goals What should a student do if the suggestions are in conflict with their study goals? Ethical 3 Student
Obligation to prevent continuation What obligation does the institution have to prevent students from continuing on a pathway which analytics suggests is not advisable? Ethical 2 Analytics Committee
Type of intervention How are the appropriate interventions decided on? Logistical 1 Educational researcher
Distribution of interventions How should interventions be distributed across the institution? Logistical 1 Analytics Committee
Conflicting interventions How does the institution ensure that it is not carrying out multiple interventions with conflicting purposes? Logistical 2 Educational researcher
Staff incentives for intervention What incentives are in place for staff to change practices and facilitate intervention? Logistical 3 Analytics Committee
Failure to act What happens if an institution fails to intervene when analytics suggests that it should? Logistical 3 Analytics Committee
Need for human intermediation Are some analytics better presented to students via e.g. a tutor than a system? Ethical 2 Educational researcher
Triage How does an institution allocate resources for learning analytics appropriately for learners with different requirements? Ethical / Logistical 1 Analytics Committee
Triage transparency How transparent should an institution be in how it allocates resources to different groups? Ethical 3 Analytics Committee
Opportunity cost How is spending on learning analytics justified in relation to other funding requirements? Logistical 2 Senior management
Favouring one group over another Could the intervention strategies unfairly favour one group over another? Ethical / Logistical 2 Educational researcher
Consequences of false information What should institutions do if a student gives false information e.g. to obtain additional support? Logistical 3 Analytics Committee
Audit trails Should institutions record audit trails of all predictions and interventions? Logistical 2 Analytics Committee
Unexpected findings How should institutions deal with unexpected findings arising in the data? Logistical 3 Analytics Committee
Adverse impact Labelling bias Does labelling or profiling of students bias institutional perceptions and behaviours towards them? Ethical 1 Educational researcher
Oversimplification How can institutions avoid overly simplistic metrics and decision making which ignore personal circumstances? Ethical 1 Educational researcher
Undermining of autonomy Is student autonomy in decision making undermined by predictive analytics? Ethical 2 Educational researcher
Gaming the system If students know that data is being collected about them will they alter their behaviour to present themselves more positively, thus distracting them and skewing the analytics? Ethical 2 Educational researcher
Abusing the system If students understand the algorithms will they manipulate the system to obtain additional support? Ethical 3 Educational researcher
Adverse behavioural impact If students are presented with data about their performance could this have a negative impact e.g. increased likelihood of dropout? Ethical 1 Educational researcher
Reinforcement of discrimination Could analytics reinforce discriminatory attitudes and actions by profiling students based on their race or gender? Ethical 1 Educational researcher
Reinforcement of social power differentials Could analytics reinforce social power differentials and students’ status in relation to each other? Ethical 2 Educational researcher
Infantilisation Could analytics “infantilise” students by spoon-feeding them with automated suggestions, making the learning process less demanding? Ethical 3 Educational researcher
Echo chambers Could analytics create “echo chambers” where intelligent software reinforces our own attitudes and beliefs? Ethical 3 Educational researcher
Non-participation Will knowledge that they are being monitored lead to non-participation by students? Ethical 2 Educational researcher
Stewardship Data minimisation Is all the data held on an individual necessary in order to carry out the analytics? Legal 1 Data scientist
Data processing location Is the data being processed in a country permitted by the local data protection laws? Legal 1 IT
Right to be forgotten Can all data regarding an individual (expect that necessary for statutory purposes) be deleted? Legal 1 IT
Unnecessary data retention How long should data be retained for? Legal 1 Analytics Committee
Unhelpful data deletion If data is deleted does this restrict the institution’s analytics capabilities e.g. refining its models and tracking performance over multiple cohorts? Logistical 2 Data scientist
Incomplete knowledge of data sources Can an institution be sure that it knows where all personal data is held? Legal / Logistical 1 IT
Inappropriate data sharing How can data sharing be prevented with parties who have no legitimate interest in seeing it or who may use it inappropriately? Legal 1 IT

 

 

 


  • -
B4AOo1iCAAEos2s

Developing the Code of Practice

A wide-ranging discussion took place in London last week to discuss the emerging Code of Practice for Learning Analytics.  A new advisory group for the Code includes representatives from the National Union of Students, Edinburgh, Huddersfield, Lancaster and Loughborough Universities, Bucks New University, The Open University, Croydon College and Jisc.

A Code of Practice for Learning Analytics has been identified by Jisc stakeholders in higher and further education as a development priority.  The literature review is the first stage in this activity and provides the underlying rationale.  The next stage is developing the document itself.

The Code will aim to help to remove barriers to the adoption of learning analytics.  We can ensure that its emphasis is positive, realistic and facilitative.  It’ll provide a focus for institutions to deal with the many legal and ethical hurdles which are arising, and can be presented as an evolving, dynamic site rather than a lengthy one-off document which hardly anyone reads, let alone adheres to.

Jisc will coordinate the development and roll-out of the Code.  Meanwhile advisory group members agreed to critique the Code as it’s being developed and consider piloting it at their own institutions.

Methodology and approaches
Some documents take a particular methodological or philosophical stance.  For instance Slade and Prinsloo’s socio-critical approach – where learning analytics is viewed as a “transparent moral practice” and students are seen as co-contributors – has influenced the Open University’s Policy on Ethical Use of Student Data.  Should the Code take such an approach?

One of the main challenges will be to strike a balance between a paternalistic approach and respecting students’ privacy and autonomy.  It was suggested that the various uses for student data might have different approaches to consent:

  1. Helping individual students based on their own data
  2. Merging individuals’ data with those of others to help the group
  3. Using data to help future cohorts of students

Informed consent could potentially be obtained for each of these options.

There was also concern expressed about ensuring that any sharing of student data outside the institution should be carefully controlled.  The Code itself should have boundaries and may need to reference other institutional policies.  There should be differentiation between demographic and behavioural data, and the “right to be forgotten” needs to be addressed.

A separate document for students?
An approach which puts the needs of learners at the heart of the Code is surely likely to result in a better and more widely-adopted document which helps to allay the fears of students and institutions and facilitate the uptake of learning analytics.  The inclusion of the NUS in this group is therefore particularly welcome.

There will need to be a balance and series of compromises struck however to develop a usable Code and encourage mutual understanding. The group decided a single document setting out clearly the rights and responsibilities of students, institutions and staff would be preferable to having a separate student “bill of rights for learning analytics”.

Explaining what the Code means in practice however may require separate advice for different stakeholders.  At institutions the Code should link closely with the student charter, and involve buy-in from the students’ union.

Striking a balance between high level principles and detailed guidance
Can the Code be sufficiently high level to meet the needs of all institutions while remaining specific enough to provide genuinely helpful guidance?  It was very clear from my institutional visits that the potential uses of learning analytics and the concerns raised varied widely across institutions.  The group thought that the document should be fairly high level in order to prove useful to all, but should be backed up by case studies and examples of how institutions have dealt with particular issues.  The case studies could be released alongside the code – for each principle there could be examples of good practice.

Conformance with the Code
Another question I posed to the group was whether we should encourage institutions to adopt the Code wholesale, and therefore be able to claim conformance with it, or to customise it to their own requirements?  We probably need to see the end result first but it was felt that institutions might want to be able to adopt the Code with local modifications.

Human intermediation
Particular concern was expressed that the Code needs to reflect the human context and the need for intermediation of learning analytics by staff. This is a common ethical theme in the literature.  However a representative from the Open University said that the sheer scale of that institution makes it unfeasible to use human intermediation for many of the potential uses of learning analytics.  Meanwhile there was real concern among members that the language which is used to present analytics to students should be carefully considered and that data is only exposed when institutions have mechanisms in place to deal with the effect on students.  The potential impact of analytics on the educator also needs to be reflected in the Code.

Format
All of the related codes of practice I’ve looked at are textual documents – normally provided in PDF.  The members felt that a document outlining the principles needed to be provided in order to present it to institutional committees but that an interactive website containing case studies, perhaps in the form of videoed interviews with staff and students, would be welcome.

Some codes are extremely lengthy and somewhat uninspiring papers stretching to thirty pages or more. One of the better formats I’ve seen is the Respect Code of Practice for Socio-Economic Research.  It’s concise – only four pages – and reasonably visually appealing, therefore arguably more likely to be read and absorbed by busy people than some of the longer codes.  However, given the large number of issues identified in our literature review, four pages is unlikely to be sufficient.

One approach would be to back up a concise summary document with more detailed online guidance for each of the areas.  Discussion forums could be included on each topic, enabling users to raise further issues which arise, and others to provide advice on how they’ve tackled that challenge.  This would need some ongoing promotion, facilitation and moderation by Jisc and/or members of the community.

Areas to be included
The literature review covers most of the ethical and legal issues which are likely to be of concern to students and to institutions when deploying learning analytics, though there may be some which I’ve missed or have not yet cropped up in the literature.  The section headings and the word clouds in the review could help prioritise the main areas to be included in the Code.  It was pointed out that it would be difficult to deal with all of these meaningfully within four pages but certainly each area could be expanded on in the supporting documentation.

Including vendors
One member suggested including vendors in the consultation process for the Code.  It might help them when making development decisions, for instance encouraging them to build consent systems into their products.  The Code could help to ensure that safeguards, such as ensuring privacy, are built in without holding back innovation.

Development process
Jisc will develop the Code up until May 2015 with guidance from the advisory group.  Supporting content e.g. videoed interviews can be developed subsequently, help raise awareness of the Code, provide examples of how it’s being implemented and help to keep it current.

A sense of ownership by institutions and by students is essential to ensure adoption.  How can this best be achieved?  A range of stakeholder organisations was proposed and a number of possible events to piggy-back on were suggested.  Several members said they’d be keen to try piloting the Code at their institutions too.  An experiential learning cycle was suggested, with institutions thinking about:

  1. What’s the ethical/legal issue?
  2. What’s the principle to deal with it?
  3. How did we apply the principle?

Roll-out and dissemination
There is already considerable awareness of the intended Code of Practice but how should it best be disseminated once developed?  One member suggested it would be useful to understand better the processes inside institutions for getting academic policies adopted as this will be key to uptake.  In addition, a couple of events specifically around the Code could be held, papers delivered at relevant conferences and approaches made to newspapers to see if they’d like to cover its launch.  It was felt that the Code should be launched with some fanfare at a larger event to increase awareness and potential take-up.

Now on with developing it…  Comments are welcome.


  • -
Solutions from codes of practice e.g. transparency

A taxonomy of the ethical and legal issues of learning analytics v0.1

In discussions around the ethics and legal issues of learning analytics I’ve found the same issues cropping up again and again. Almost always they’ve already been covered somewhere in the growing collection of publications on learning analytics. Sometimes they’re expressed in different ways but boil down to the same underlying problem.

The literature review of these issues aims to provide the background for the development of a code of practice for learning analytics. But it’s a large and unwieldy document to refer to so I’ve attempted to distil and group the many issues that I’ve come across so far.

I’ve given each of the resulting 86 issues a name and have provided a question which attempts to capture the issue. Many of these cannot be answered simply; almost all could be responded to with “It depends…” Most have both an ethical and a legal dimension. Some are related more to logistics than ethics or law. And some are already dealt with by existing institutional policies.

I’ll be taking this taxonomy to a workshop next week in Paris with the Lace Project and Apereo where I hope the issues and questions can be clarified and further refined. Then it’s over to the Code of Practice Advisory Group in the UK for their advice on how to translate this into a useful Code of Practice.

Area Issue Question
Validity Minimisation of inaccurate data How does an institution minimise inaccuracies in the data?
Minimisation of incomplete data How does an institution minimise incompleteness of the dataset?
Optimum range of data sources How many and which data sources are necessary for increasing accuracy in the analytics?
Verification of algorithms and metrics How should an institution verify its algorithms and metrics to ensure accuracy?
Spurious correlations How can institutions avoid drawing misleading conclusions from spurious correlations?
Evolving nature of students To what extent can analytics be accurate when students’ identities and actions evolve as they progress through their studies?
Authentication of public data sources How can institutions ensure that student data taken from public sites is authenticated to their students?
Ownership and control Control of data for analytics Who in the institution decides what data is collected and used for analytics?
Breaking silos How can silos of data ownership in institutions be broken into in order to obtain data for analytics?
Control of analytics processes Who in the institution decides how analytics are to be created and used?
Overall responsibility Who is responsible in the institution for the appropriate and effective use of learning analytics?
Ownership of data What data does the institution own and what is owned by the student?
Awareness Student awareness of data collection What should students be told about the data that is being collected about them?
Student awareness of data use What should students be told about the uses to which their data is being put?
Student awareness of algorithms and metrics To what extent should students be given details of the algorithms used for learning analytics and the metrics and labels that are created?
Proprietary algorithms and metrics What should institutions do if vendors decline to make details of their algorithms and metrics public?
Student awareness of potential consequences of opting out What should students be told about the potential consequences of opting out of data collection and analysis of their learning?
Staff awareness of data collection and use What should staff be told about the data that is being collected about their students and what is being done with it?
Consent and opting out When to seek consent In what situations should students be asked for consent to collection and use of their data for learning analytics?
Consent for anonymous use Should students be asked for consent for collection of data which will only be used in anonymised formats?
Consent for outsourcing Do students need to give specific consent if the collection and analysis of data is to be outsourced to third parties?
Clear and meaningful consent processes How can institutions avoid opaque privacy policies and ensure that students genuinely understand the consent they are asked to give?
Right to opt out Does a student have the right to opt out of data collection and analysis of their learning activities?
Partial consent Can students consent to some data collection and analysis but opt out elsewhere?
Right to withdraw Does a student have the right to withdraw from data collection and analysis after previously having given their consent?
Right to anonymity Should students be allowed to provide pseudonyms to disguise their identity in certain circumstances?
Adverse impact of opting out on individual If a student is allowed to opt out of data collection and analysis of their activities could this have a negative effect on their studies?
Adverse impact of opting out on group If individual students opt out will the dataset be incomplete, thus potentially reducing the accuracy and effectiveness of learning analytics for the group?
Lack of real choice to opt out Do students really have a choice if pressure is put on them by the institution or there’s a chance of adverse impact on their academic success by opting out?
Student input to analytics process Should students have a say in what data is collected and how it is used for analytics?
Change of purpose Should institutions request consent again if the data is to be used for purposes for which consent was not originally given?
Legitimate interest To what extent can the institution’s “legitimate interests” override privacy controls for individuals?
Unknown future uses of data How can consent be requested when potential uses of the (big) data are not yet known?
Consent in open courses Are open courses (MOOCs etc) different when it comes to obtaining consent?
Use of publicly available data Can institutions use publicly available data (e.g. tweets) without obtaining consent?
Student access Student access to their data To what extent should students be able to access the data held about them?
Student access to their analytics To what extent should students be able to access the analytics performed on their data?
Data formats In what formats should students be able to access their data?
Metrics and labels Should students see the metrics and labels attached to them?
Right to correct inaccurate data What data should individuals be allowed to correct about themselves?
Data portability What data about themselves can the learner take with them?
Privacy Out of scope data Is there any data that should not be used for learning analytics?
Access to employers Under what circumstances would it be appropriate to give employers access to analytics on students?
Tracking location Under what circumstances is it appropriate to track the location of students at campuses?
Staff permissions To what extent should access to individuals’ data be restricted within an institution?
Unintentional creation of senstitive data How do institutions avoid creating “sensitive” data (e.g. ethnicity, religion) from other data sources?
Use of metadata to identify individuals Can individuals be identified from metadada even if personal data has been deleted?
Requests from external agencies What should institutions do when requests for student data are made by external agencies e.g. educational authorities or security agencies?
Sharing data with other institutions Is it appropriate for institutions to share student data with other institutions in order to increase the dataset and enhance the analytics?
Enhancing trust by retaining data internally If students are told that their data will be kept within the institution will they develop greater trust in and acceptance of learning analytics?
Action Institutional obligation to act What obligation does the institution have to intervene when there is evidence that a student could benefit from additional support?
Student obligation to act What is the student’s obligation to act on learning analytics designed to help them?
Conflict with study goals What should a student do if the suggested advice is in conflict with their study goals?
Obligation to prevent continuation Is there an obligation on the institution to prevent students from continuing on a pathway if analytics show that it is not in the student’s or institution’s interests for them to continue?
Type of intervention How are the appropriate interventions decided on?
Distribution of interventions How should interventions resulting from analytics be distributed among different stakeholders in the institution?
Conflicting purposes of interventions How does the institution ensure that it is not carrying out multiple interventions whose purposes conflict?
Staff incentives for intervention What incentives are in place for staff to intervene?
Failure to act What happens if an institution fails to intervene?
Need for human intermediation Are some analytics better presented to students via e.g. a tutor than via a system?
Triage How does an institution allocate resources for learning analytics appropriately for learners with different requirements?
Triage transparency How transparent should an institution be in how it allocates resources to different groups?
Opportunity cost How is spending on learning analytics justified in relation to other funding requirements?
Favouring one group over another Could the intervention strategies favour one group of students over another?
Consequences of false information What should institutions do if it is determined that a student has given false information to e.g. obtain additional support?
Audit trails Should institutions record audit trails of all predictions and interventions?
Unexpected findings What infrastructure is in place to deal with something unexpected arising in the data?
Adverse impact Labelling bias Does a student profile or labelling bias institutional perceptions and behaviours towards them?
Oversimplification How can institutions avoid overly simplistic metrics and decision making which ignore personal circumstances?
Undermining of autonomy Is student autonomy in decision making about their learning undermined by predictive analytics?
Gaming the system If students know that data is being collected about them will they alter their behaviour to present themselves more positively, thus skewing the analytics and distracting them from their learning?
Abusing the system If students understand the algorithms behind learning analytics will they abuse the system to obtain additional support?
Adverse behavioural impact If students are presented with data about their performance, likelihood of failure etc. could this have a negative impact on their behaviour, leading to increased likelihood of failure and dropout?
Reinforcement of discriminatory attitudes and actions Could analytics reinforce discriminatory attitudes and actions by profiling students based on e.g. their race or gender?
Reinforcement of social power differentials Could analytics reinforce social power differentials and learners’ status relating to each other?
Infantilisation Could analytics “infantalise” students by spoon-feeding them with automated suggestions, making the learning process less demanding?
Echo chambers Could analytics create “echo chambers” where intelligent software reinforces our own attitudes or beliefs?
Non-participation Will knowledge that they are being monitored lead to non-participation by students?
Stewardship Data minimisation Is all the data being held on an individual necessary in order to carry out the analytics?
Data processing location Is the data being processed in a country permitted by the local data protection laws?
Right to be forgotten Can all data regarding an individual except that necessary for statutory purposes be deleted?
Unnecessary data retention How long should data be retained for?
Unhelpful data deletion If data is deleted does this restrict the institution’s ability to refine its models, track performance over multiple cohorts etc?
Incomplete knowledge of data sources Can an institution be sure that it knows where all personal data is held?
Inappropriate data sharing How can we prevent data being shared within or outside the institution with parties who have no legitimate interest in seeing it or may use it inappropriately?
Risk of re-identification Does anonymisation of data becomes more difficult as multiple data sources are aggregated, potentially leading to re-identification of an individual later?

  • 13
Solutions from codes of practice e.g. transparency

Jisc releases report on ethical and legal challenges of learning analytics

Literature review coverDo institutions need to obtain consent from students before collecting and using data about their online learning activities?

Should learners be allowed to opt out of having data collected about them?

Could showing students predictions about their likelihood of academic success have a negative effect on their motivation and make them more likely to drop out?

These are some of the questions addressed in a new report which is released today by Jisc: Code of practice for learning analytics: A literature review of the ethical and legal issues.

The aim of the review is to provide the groundwork for a code of practice which is intended to help institutions solve the complex issues raised by learning analytics.

It was a very interesting task gathering together publications from many different authors and organisations.  I drew material from 86 publications, more than a third of them published within the last year from sources including:

  • The literature around learning analytics which makes explicit reference to legal and ethical issues
  • Articles and blogs around the ethical and legal issues of big data
  • A few papers which concentrate specifically on privacy
  • Relevant legislation, in particular the European Data Protection Directive 1995 and the UK Data Protection Act 1998
  • Related codes of practice from education and industry

Expressing issues as questions can be a useful way of making some of the complexities more concrete. I’ve incorporated 93 questions from the literature that authors had posed directly. The categorisations of these highlighted in the word cloud below give an instant flavour of the main concerns around the implementation of learning analytics being raised by researchers and practitioners.

Issues of concern

 

At the end of the report I reviewed 16 codes of practice or lists of ethical principles from related fields and found the main concepts people wish to embody are transparency, clarity, respect, user control, consent, access and accountability.

I’ve attempted to be comprehensive but I challenge you to spot any relevant legal or ethical issues I’ve missed in the report – please let us know in the comments below if you find any.  The next task is to put together an expert group to advise on the development of the code of practice … and to begin drafting the document itself.


  • -
Adam Cooper

Examining open learning analytics – report from the Lace Project meeting in Amsterdam

The latest Lace Project event was held in the illustrious surroundings of the Allard Pierson Museum at Amsterdam University this week.  The focus this time was on open learning analytics.  After some lightening presentations on participants’ interests in the area, we split into groups to look at what exactly open meant in the context of learning analytics. We discussed some of the issues with Adam Cooper of Cetis, shown here alongside other fine specimens of humanity.

Adam Cooper

Probably the most obvious aspect of open in the context of learning analytics is the reuse of code (and documentation) that others have created.  Predictive models can also of course be opened up to others.  Having open APIs in learning analytics products is important too – as is interoperability of the different systems.  And a vibrant community of people and institutions who wish to share and enhance all these things is essential.

Openness was thought to “keep the market honest” and it also improves transparency for learners and other users of learning analytics systems.  Openness may also mean that the data from one environment can be connected to that in another one, not only across the different systems in one institution but potentially with other institutions too.  Adam has documented some of the organisational intitiatives to share data for learning analytics .

In a group discussion later we looked at some of the next steps or “big ideas” for open learning analytics:

  • Clarifying the technical architectures – Apereo has defined an architecture and Jisc is commissioning the components of a basic learning analytics system
  • Developing a privacy-enhanced technology infrastructure
  • Student-facing tools to monitor and analyse their own learning progress
  • Tools to evaluate accessibility issues – an example was given of a system which determines if users are dyslexic and then adapts the learning accordingly

The other groups reported back on their proposed essential next steps:

  • Understanding the organisational consequences (or “systemic impact”) of deploying learning analytics
  • Gathering evidence for relevant specifications and standards that work or don’t work
  • Obtaining consent from students to help evolve learning analytics, instead of just agreeing that their data can be used (see Sharon Slade and Paul Prinsloo’s suggestions on students being collaborators in the learning analytics process rather than mere data subjects)
  • Building reference implementations of learning record stores
  • Understanding the barriers to the deployment of learning analytics

One of the clearest messages to come out of the day was just how important tackling the privacy issues is becoming in order to avoid an InBloom type fiasco  in Europe.  While this is a problem everywhere, concerns about the lack of systems for granting consent to the collection of data are massively holding up implementations of learning analytics in countries such as the Netherlands and Germany.  A further event being planned for Paris in February will attempt to progress understanding in this area and develop transparent learning analytics systems which include mechanisms to obtain consent at granular levels from students to process their data.


  • 1
Participants at workshop

Notes from Utrecht Workshop on Ethics and Privacy Issues in the Application of Learning Analytics

I’m back from yesterday’s excellent Workshop on Ethics & Privacy Issues in the Application of Learning Analytics  in Utrecht organised by LACE and SURFHendrik Drachsler from the Open University of the Netherlands kicked off the session by presenting a background to learning analytics and some of the resulting ethical and privacy issues.  He mentioned the situation in the Netherlands where universities are now partially funded on the basis of how many students they graduate, and concerns that that gives them an incentive not to accept students who are predicted to fail.

Participants at workshop

He also discussed the InBloom debacle in the US – “a perfect example of not taking care about privacy issues”.  There was another situation in the Netherlands where an app used on tablets in schools collected data on which further analysis was carried out.  Problems arose because this analysis wasn’t described in the terms and conditions of use.

Hendrik mentioned that his call for ethical and privacy issues in the application of learning analytics had produced over 100 issues.  These were then put into four categories: privacy, ethics, data and transparency.  The aim of the day was to discuss these issues and begin to look for solutions to them.

The group decided that there are often no clear boundaries between these categories.  Certainly I’ve found it artificial to try to split legal issues from ethical ones when carrying out my recent literature review of the area.  Much of the law is based on ethics – and sometimes an ethical stance has to be applied when interpreting the law in particular situations.

The workshop wasn’t purely full of Hendriks, but a second Hendrik, Hendrik vom Lehn then gave an informative presentation on practical considerations around some of the legal issues arising from learning analytics.  Much of what he said, and subsequent discussions during the day, related to the EU Data Protection Directive.  Hendrik thought that a common misconception about the Directive is that the scope of “personal data” is much broader than most people think, and includes absolutely everything which makes data personally identifiable.

Another interesting concept in the Directive is that data can potentially be processed without consent from individuals if it is in the “legitimate interest” of the organisation to do so.  However in practice it’s likely to be better to inform students and obtain their consent for data collection and the resulting analytics.  Hendrik also discussed the US concept of “reasonable expectation”: people whose data is being processed should have reasonable expectations of what is being done with it.  Thus if you start using it in new ways (e.g. the recent Facebook mood altering experiment) you’re on dangerous ground.

Anonymisation is often proposed as an alternative to obtaining consent, but this can be difficult to achieve.  It’s particularly problematic in small groups where behaviours can easily be attributed to an individual.

Hendrik felt that where grading is based on learning analytics or can in some way affect the career of the student, this could have legal implications.  Another issue he mentioned, which I hadn’t come across before, was the subordinate position of students, and that they might feel obliged to participate in data collection or learning analytics activities because they were being graded by the same person (or institution) that was analysing them.  Would any consent given be regarded as truly voluntary in that case?

A member of the audience then asked if there was a difference between research and practice in learning analytics.  Hendrik suggested that ethically our approach should be the same but from a legal perspective there may be a difference.

So what happens if a student asks for all the data that an institution has about them?  Hendrik thought that the Directive implied that we do indeed need to make everything we know about students available to them.  However there might be a possible conflict between full data access and the wider goals of learning analytics – it might make it easier for students to cheat, for example.  Also it may be difficult to provide meaningful access for an individual while excluding other students’ data.

Another potentially difficult area is outsourcing and data transfers to third parties.  This is particularly problematic of course when that data is being transferred outside the European Union.  For students the process of understanding what is happening to their data – and accessing it – can then become more complex and they may have to go through several steps.  Ownership of the data is not complete in this situation for any party (though in a later discussion it was proposed that “ownership” is not a helpful concept here – more relevant are the EU concepts of “data controller” and “data processor”).

We then split into groups and had the benefit of some great input from Jan-Jan Lowijs –  a privacy consultant from Deloitte.  He described the nine general themes in the Directive which we found a useful way to propose answers to some of the 100 issues that had been submitted.  These are:

  1. Legitimate grounds – why you should have the data in the first place
  2. Purpose of the data – what you want to do with it
  3. Data quality – minimisation, deletion etc
  4. Transparency – informing the students
  5. Inventory – knowing what data you have and what you do with it already
  6. Access – the right of the data subject to access their data, when can you have access to it and what can you see
  7. Outsourcing – and the responsibilities of your institution as data controller and the third party as data processor
  8. Transport of data – particularly problematic if outside the EU
  9. Data security

Attempting to answer some of questions submitted using the themes as guidance resulted in the following:

Who has access to data about students’ activities?
Students themselves and certified access for teachers, researchers etc, based on theme 2 above (purpose of data)

What data should students be able to view?
All data on an individual should be provided at any time they request it – that’s the situation to aim for, based on theme 6 (access)

Should students have the right to request that their digital dossiers be deleted on graduation?
Yes, so long as there are no other obligations on the institution to keep the data e.g. names, date of birth, final grades, based on theme 3 (data quality)

What are the implications of institutions collecting data from non-institutional sources (e.g. Twitter)?
Consent must be obtained from the students first, based on theme 4 (transparency). A case in Finland where two students sued their university who were re-using their Twitter data was noted.

Something interesting that Jan-Jan also mentioned was that there are differences in data subjects’ attitudes to privacy, and that a number of studies have shown a fairly consistent split of:

  • 25% “privacy fundamentalists” who don’t want to share their data
  • 60% pragmatists who are happy to share some of their data for particular purposes
  • 15% people who “don’t care” what happens to their data

An organisation needs therefore to make an active decision as to whether it attempts to cater for these different attitudes or finds some middle ground.

Some of the conclusions from the day in the final session were:

  1. It was noted that students were absent from the discussions and should be involved in the future.
  2. It was suggested that we fully articulate the risks for institutions of learning analytics. What are the showstoppers?  Are they reputational or based on a fear of loss of students?
  3. “Privacy by design” and user-centred design with much better management of their data by users themselves were thought to be vital.
  4. InBloom was suggested as an “anti-pattern”, to be studied further to establish what we shouldn’t be doing.
  5. If you think something’s dodgy then it probably is. I have to admit being slightly concerned to hear that one university has equipment in its toilets to ensure that you’re not using your mobile phone to cheat if you have to nip out during an exam.  A good rule of thumb proposed by Jan-Jan is that if you feel uneasy about some form of data collection or analysis then there’s probably something to be worried about.

The outcomes from the day were being much more rigorously written up than I have done above by a professional writer – and will be fed into subsequent workshops held by LACE with the aim of producing a whitepaper or longer publication in the area.


  • -
Students studying

Code of practice “essential” for learning analytics

We had a lively session on ethics and legal issues at the Jisc Effective Learning Analytics workshop last week, kicking it off by outlining some of the key questions in this area:

  • Who should have access to data about students’ online activities?
  • Are there any circumstances when collecting data about students is unacceptable/undesirable?
  • Should students be allowed to opt out of data collection?
  • What should students be told about the data collected on their learning?
  • What data should students be able to view about themselves?
  • What are the implications of institutions collecting data from non-institutional sources e.g. Twitter?

Students studying

Students Studying by Christopher Porter CC BY-NC-ND 2.0


Concern was also expressed about the metrics being used for analytics – how accurate and appropriate are they and could it be dangerous to base interventions by tutors on metrics which portray an incomplete picture of student activity?

A number of the participants had already been thinking in detail about how to tackle these issues. There was a consensus that learning analytics should be carried out primarily to improve learning outcomes and for the students’ benefit.  Analytics should be conducted in a way that would not adversely affect learners based on their past attainment, behaviour or perceived lack of chance for success. The group felt that the sector should not engage with the technical and logistical aspects of learning analytics without first making explicit the legal and ethical issues and understanding our obligations towards students.

Early conversations with students were thought to be vital so that there were no surprises. It was suggested that Informed consent is key – not just expecting students to tick a box saying they’ve read the institution’s computing regulations.  Researchers elsewhere have begun to examine many of these areas too – see the paper for example by Sharon Slade and Paul Prinsloo: Learning analytics: ethical issues and dilemmas.

Mike Day at Nottingham Trent University found that students in the era of Google and Facebook already expect data to be being collected about their learning. His institution’s approach has been to make the analytics process a collaboration between the learner and the institution. They have for instance agreed with students that it’s appropriate and helpful for both themselves and their tutors to be able to view all the information held about them.

Another issue discussed at some length was around the ethics of learners’ data travelling with them between institutions. Progress is being made on a unique learner number, and the Higher Education Data and Information Improvement Programme (HEDIIP) is developing more coherent data structures for transferable learner records. It will be possible for data on the learning activities of individuals to be transferred between schools, colleges and universities. But what data is appropriate to transfer? Should you be tainted for the rest of your academic life by what you got up to at school? On the other hand could some of that information prove vital in supporting you as you move between institutions?

Data on disabilities might be one such area where it could be helpful for a future institution to be able to cater for a learner’s special needs. Ethically this may best be under the control of the student who can decide what information to present about their disability.  However the technology may be in place to detect certain disabilities automatically such as dyslexia – so the institution might have some of this information whether the student wants them to know it or not.

Who owns the data about a students’ lifelong learning activity is another issue. Institutions may own it for a time, but once that student has left the institution is it appropriate to hold onto it? Perhaps the learner should take ownership of it, even if it is held securely by an institution or an outside agency. There may be a fundamental difference between attainment data and behavioural data, the latter being more transitory and potentially less accurate than assessment results and grades – and therefore it should be disposed of after a certain period.

There are of course different ethical issues involved when data on learning activities is anonymised or aggregated across groups of students. One parallel we discussed was that of medicine. A learner might visit a tutor in the way that a patient visits a GP.

The doctor chats to the patient about their ailment with the patient’s file including their previous medical history in front of them. Combining what the patient says with physical observations and insight from the patient’s records the doctor then makes a diagnosis and prescribes some treatment or suggests a change in lifestyle.

Meanwhile:

A tutor chats to a student about an issue they’re having with their studies and has analytics on their learning to date on a computer in front of them as they talk. The analytics provides additional insight to what the student is saying so the tutor is able to make some helpful suggestions and provide additional reading materials or some additional formative assessment to the student.

In both scenarios the professional takes notes on the interaction which are themselves added to the individual’s records. All the information is held under the strictest confidentiality. However the data in both scenarios can also be anonymised for research purposes, enabling patterns to be discovered and better treatment/advice to be provided to others in the future.

So in order to help institutions navigate their way around the complexities would a code of practice or guidelines be of interest to institutions? The consensus was yes it would and this was borne out in voting by the wider group later in the day. The schools sector has already developed codes of practice and obviously the NHS is well advanced in the ethics of data collection already so there is much to be learned from these professions – and from research ethics committees in many of our own institutions. There would need to be consultation with the various UK funding bodies – and working closely with the National Union of Students was seen as key to ensuring adoption.

A code of practice for learning analytics would have to be clearly scoped, easily understandable and generic enough to have wide applicability across institutions. The primary audiences are likely to be students, tutors and senior management. Mike at Nottingham Trent found the key to success was a four-way partnership between technology providers (who were required to adapt their products to meet ethical concerns), the IT department, students and tutors.

There was a strong consensus in the group that this work would significantly help to allay the fears of students and, often just as vocally, staff in their institutions in order to explore the real possibilities of using learning analytics to aid retention and student success.  In fact some stakeholders considered it an essential step at their universities and colleges before they could make progress.  Developing a code of practice for learning analytics will therefore be a key area of work for Jisc over the coming months.

This post was first published on the Jisc Effective Learning Analytics blog, 18th Sept 2014.