Category Archives: Ethics

  • -
A model for the development of a code of practice

Paper submitted to Journal of Learning Analytics: “Developing a Code of Practice for Learning Analytics”

I’ve just submitted a paper to a forthcoming “Special Section on Ethics and Privacy” in the Journal of Learning Analytics (JLA).  The paper documents the development of Jisc’s Code of Practice for Learning Analytics through its various stages, incorporates the taxonomy of ethical, legal and logistical issues, and includes a model for developing a code of practice which could be used in other areas.

A model for the development of a code of practice

A model for the development of a code of practice

As an open journal the JLA suggests that authors publish their papers before or during the submission and review process – this results in the work getting out more quickly and can provide useful feedback for authors. So here’s the paper – and if you have any feedback it would be great to hear from you.

Abstract
Ethical and legal objections to learning analytics are barriers to development of the field, thus potentially denying students the benefits of predictive analytics and adaptive learning. Jisc, a charitable organisation which champions the use of digital technologies in UK education and research, has attempted to address this with the development of a Code of Practice for Learning Analytics. The Code covers the main issues institutions need to address in order to progress ethically and in compliance with the law. This paper outlines the extensive research and consultation activities which have been carried out to produce a document which covers the concerns of institutions and, critically, the students they serve. The resulting model for developing a code of practice includes a literature review, setting up appropriate governance structures, developing a taxonomy of the issues, drafting the code, consulting widely with stakeholders, publication, dissemination, and embedding it in institutions.

 


  • -
Media City Uk

Code of Practice for Learning Analytics launched

Today Jisc is launching the Code of Practice for Learning Analytics at the UCISA Spotlight on Digital Capabilities event here in the amazing MediaCityUK at Salford Quays.

Media City UK, Salford

Developing this document was chosen by institutions as the number one priority for Jisc’s learning analytics programme.  The Code aims to help universities and colleges develop strategies for dealing with the various ethical and legal issues that may arise when deploying learning analytics.

Code of Practice coverIt’s a brief document of four pages and is available in HTML or in PDF. The development of the Code was based on a literature review of the ethical and legal issues. From this a taxonomy of the ethical, legal and logistical issues was produced. The Code was drafted from this taxonomy and is grouped into seven areas:

  1. Responsibility – allocating responsibility for the data and processes of learning analytics within an institution
  2. Privacy – ensuring individual rights are protected and data protection legislation is complied with
  3. Validity – making sure algorithms, metrics and processes are valid
  4. Access – giving students access to their data and analytics
  5. Enabling positive interventions – handling interventions based on analytics appropriately
  6. Minimising adverse impacts – avoiding the various pitfalls that can arise
  7. Stewardship of data – handling data appropriately

The Code was developed in the UK context and refers to the Data Protection Act 1998 however most of it is relevant to institutions wishing to carry out learning analytics anywhere, particularly in other European countries which have similar data protection legislation. It can be adopted wholesale or used as a prompt or checklist for institutions wishing to develop their own learning analytics policies and processes.

If you find the document helpful or feel that anything is unclear or missing please let us know. Keeping it concise was thought to be important but that meant leaving out more in-depth coverage of the issues. Over the coming months we’ll be developing an associated website with advice, guidance and case studies for institutions which wish to use the Code.

Acknowledgements
The process has been overseen by a Steering Group consisting of Paul Bailey (Jisc), Sarah Broxton (Huddersfield University), Andrew Checkley (Croydon College), Andrew Cormack (Jisc), Ruth Drysdale (Jisc), Melanie King (Loughborough University), Rob Farrow (Open University), Andrew Meikle (Lancaster University), David Morris (National Union of Students), Anne-Marie Scott (Edinburgh University), Steven Singer (Jisc), Sharon Slade (Open University), Rupert Ward (Huddersfield University) and Shan Wareing (London South Bank University).

It was particularly good to have the student voice represented in the development of the Code by David Morris of the NUS. I’m also especially grateful to Andrew Cormack and Rupert Ward for their perceptiveness and attention to detail on the final draft. I received additional helpful feedback, most of which I was able to incorporate, from the following people (some in a personal capacity, not necessarily representing the views of their organisations):

Helen Beetham (Higher Education Consultant), Terese Bird (University of Leicester), Crispin Bloomfield (Durham University), Alison Braddock (Swansea University), Annemarie Cancienne (City University London), Scott Court (HEFCE), Mike Day (Nottingham Trent University), Roger Emery (Southampton Solent University), Susan Graham (Edinburgh University), Elaine Grant (Strathclyde University), Yaz El Hakim (Instructure), Martin Hawksey (with other members, Association for Learning Technology), Ross Hudson (HEFCE), John Kelly (Jisc), Daniel Kidd (Higher Education Statistics Agency), Jason Miles-Campbell (Jisc), George Munroe (Jisc), Jean Mutton (Derby University), Richard Puttock (HEFCE), Katie Rakow (University of Essex), Mike Sharkey (Blue Canary), Sophie Stalla-Bourdillon (Southampton University), Sarah Stock (University of Essex) and Sally Turnbull (University of Central Lancashire).

Finally, many thanks to Jo Wheatley for coordinating the production of the print and HTML versions of the Code.

 


  • -
code

Code of Practice for Learning Analytics – public consultation

Jisc’s draft Code of Practice for Learning Analytics is now available for public consultation for the next two weeks: Code of Practice for Learning Analytics v04 [MS Word].  I’ve already had some very useful comments back from a number of invited organisations and individuals which will help to enhance the document and the accompanying online guidance.

The background to the Code is provided in an earlier blog post.  The deadline for comments from the public consultation is 5th May.  These will then be presented to the Code of Practice Advisory Group which will agree a final version.

I’d be very grateful for any further thoughts from individuals or organisations either by commenting on this blog post or emailed to me at niall.sclater {/at/} jisc.ac.uk

We intend to release the final version of the Code in June and will be developing accompanying online guidance and case studies over the coming months.


  • -
Paris

A taxonomy of ethical, legal and logistical issues of learning analytics v1.0

Jisc, Apereo and the Lace Project held a workshop in Paris on 6th February to discuss the ethical and legal issues of learning analytics.  The focus of this meeting was the draft taxonomy of issues that I prepared previously.  It was extremely helpful to have comments from experts in the area to refine the list, which is forming the basis for Jisc’s Code of Practice for Learning Analytics.  I have subsequently reworked the taxonomy based on the group’s comments.

Paris

Re-ordering
I’ve now re-ordered the table to reflect a slightly more logical lifecycle view of learning analytics moving from issues of ownership and control to seeking consent from students, ensuring transparency, maintaining privacy, ensuring validity in the data and the analytics, enabling student access to the data, carrying out interventions appropriately, minimising adverse impacts and stewarding the data.

Type
I’ve added a “Type” column which states whether the issue is primarily one of ethics, legalities or logistics.  It’s become clear to me that many of the issues in the literature around ethics and privacy for learning analytics are more about the logistics of implementation than about doing what’s right or keeping within the law.  I’ve therefore renamed the taxonomy to reflect the fact it’s about logistics as well.

Rank
The Paris group suggested scoring the issues on the basis of their importance and we began to rate them on a scale of 1 to 5, highlighting the most important ones.  I have subsequently reduced the scale to three points, roughly equating to: 1 – Critical; 2 – Important; 3 – Less important / may not arise.  I have reflected the views of the group in the rankings but have had to make many choices as to their relative importance myself.  I’d like to find some more rigorous way of rating the issues though the ranking will always be dependent on the nature and priorities of the institution.

Responsibility
The group added a stakeholder column.  Subsequently I divided this into Stakeholders most impacted and Stakeholders responsible.  I then found that the most impacted stakeholders were almost always students so the column wasn’t particularly helpful and I’ve just included a Responsibility column which shows who is primarily responsible for dealing with the issue. Again there’s a level of subjectivity here on my part and these roles will be constituted differently depending on the institution. I’ve listed six stakeholders:

  1. Senior management – the executive board of the institution.
  2. Analytics committee – the group responsible for strategic decisions regarding learning analytics. This might be a learning and teaching committee, though some of the issues may be the responsibility of a senior champion of learning analytics rather than a more representative commmittee.
  3. Data scientist – while the analytics committee may decide on particular issues, there is a need for data scientists or analysts to advise on issues relating to the validity of the dataset and how to interpret it.
  4. Educational researcher – some issues would be best dealt with by staff with detailed knowledge of the educational issues who are able to monitor the impact of analytics on students.  This role may be carried out by teachers or tutors or those more dedicated to educational research.
  5. IT – the institutional information technology department will take primary responsibility for some aspects of the analytics processes.
  6. Student – while students are potentially impacted by almost every issue here, they are primarily responsible themselves for dealing with a few of them.
Group Name Question Type Rank  Responsibility
Ownership & Control Overall responsibility Who in the institution is responsible for the appropriate and effective use of learning analytics? Logistical 1 Senior management
Control of data for analytics Who in the institution decides what data is collected and used for analytics? Logistical 1 Senior management
Breaking silos How can silos of data ownership be broken in order to obtain data for analytics? Logistical 2 Analytics Committee
Control of analytics processes Who in the institution decides how analytics are to be created and used? Logistical 1 Analytics Committee
Ownership of data How is ownership of data assigned across stakeholders? Legal 1 Analytics Committee
Consent When to seek consent In which situations should students be asked for consent to collection and use of their data for analytics? Legal / Ethical 1 Analytics Committee
Consent for anonymous use Should students be asked for consent for collection of data which will only be used in anonymised formats? Legal / Ethical 3 Analytics Committee
Consent for outsourcing Do students need to give specific consent if the collection and analysis of data is to be outsourced to third parties? Legal 3 Analytics Committee
Clear and meaningful consent processes How can institutions avoid opaque privacy policies and ensure that students genuinely understand the consent they are asked to give? Legal / Ethical 1 Analytics Committee
Right to opt out Do students have the right to opt out of data collection and analysis of their learning activities? Legal / Ethical 1 Analytics Committee
Right to withdraw Do students have the right to withdraw from data collection and analysis after previously giving their consent? Legal 3 Analytics Committee
Right to anonymity Should students be allowed to disguise their identity in certain circumstances? Ethical / Logistical 3 Analytics Committee
Adverse impact of opting out on individual If a student is allowed to opt out of data collection and analysis could this have a negative impact on their academic progress? Ethical 1 Analytics Committee
Adverse impact of opting out on group If individual students opt out will the dataset be incomplete, thus potentially reducing the accuracy and effectiveness of learning analytics for the group Ethical / Logistical 1 Data scientist
Lack of real choice to opt out Do students have a genuine choice if pressure is put on them by the insitution or they feel their academic success may be impacted by opting out? Ethical 3 Analytics Committee
Student input to analytics process Should students have a say in what data is collected and how it is used for analytics? Ethical 3 Analytics Committee
Change of purpose Should institutions request consent again if the data is to be used for purposes for which consent was not originally given? Legal 2 Analytics Committee
Legitimate interest To what extent can the institution’s “legitimate interests” override privacy controls for individuals? Legal 2 Analytics Committee
Unknown future uses of data How can consent be requested when potential future uses of the (big) data are not yet known? Logistical 3 Analytics Committee
Consent in open courses Are open courses (MOOCs etc) different when it comes to obtaining consent? Legal / Ethical 2 Analytics Committee
Use of publicly available data Can institutions use publicly available data (e.g. tweets) without obtaining consent? Legal / Ethical 3 Analytics Committee
Transparency Student awareness of data collection What should students be told about the data that is being collected about them? Legal / Ethical 1 Analytics Committee
Student awareness of data use What should students be told about the uses to which their data is being put? Legal / Ethical 1 Analytics Committee
Student awareness of algorithms and metrics To what extent should students be given details of the algorithms used for learning analytics and the metrics and labels that are created? Ethical 2 Analytics Committee
Proprietary algorithms and metrics What should institutions do if vendors do not release details of their algorithms and metrics? Logistical 3 Analytics Committee
Student awareness of potential consequences of opting out What should students be told about the potential consequences of opting out of data collection and analysis of their learning? Ethical 2 Analytics Committee
Staff awareness of data collection and use What should teaching staff be told about the data that is being collected about them, their students and what is being done with it? Ethical 1 Analytics Committee
Privacy Out of scope data Is there any data that should not be used for learning analytics? Ethical 2 Analytics Committee
Tracking location Under what circumstances is it appropriate to track the location of students? Ethical 2 Analytics Committee
Staff permissions To what extent should access to students’ data be restricted within an institution? Ethical / Logistical 1 Analytics Committee
Unintentional creation of sensitive data How do institutions avoid creating “sensitive” data e.g. religion, ethnicity from other data? Legal / Logistical 2 Data scientist
Requests from external agencies What should institutions do when requests for student data are made by external agencies e.g. educational authorities or security agencies? Legal / Logistical 2 Senior management
Sharing data with other institutions Under what circumstances is it appropriate to share student data with other institutions? Legal / Ethical 2 Analytics Committee
Access to employers Under what circumstances is it appropriate to give employers access to analytics on students? Ethical 2 Analytics Committee
Enhancing trust by retaining data internally If students are told that their data will be kept within the institution will they develop greater trust in and acceptance of analytics? Ethical 3 Analytics Committee
Use of metadata to identify individuals Can students be identified from metadata even if personal data has been deleted? Legal / Logistical 2 Data scientist
Risk of re-identification Does anonymisation of data become more difficult as multiple data sources are aggregated, potentially leading to re-identification of an individual? Legal / Logistical 1 Data scientist
Validity Minimisation of inaccurate data How should an institution minimise inaccuracies in the data? Logistical 2 Data scientist
Minimisation of incomplete data How should an institution minimise incompleteness of the dataset? Logistical 2 Data scientist
Optimum range of data sources How many and which data sources are necessary to ensure accuracy in the analytics? Logistical 2 Data scientist
Validation of algorithms and metrics How should an institution validate its algorithms and metrics? Ethical / Logistical 1 Data scientist
Spurious correlations How can institutions avoid drawing misleading conclusions from spurious correlations? Ethical / Logistical 2 Data scientist
Evolving nature of students How accurate can analytics be when students’ identities and actions evolve over time? Logistical 3 Educational researcher
Authentication of public data sources How can institutions ensure that student data taken from public sites is authenticated to their students? Logistical 3 IT
Access Student access to their data To what extent should students be able to access the data held about them? Legal 1 Analytics Committee
Student access to their analytics To what extent should students be able to access the analytics performed on their data? Legal / Ethical 1 Analytics Committee
Data formats In what formats should students be able to access their data? Logistical 2 Analytics Committee
Metrics and labels Should students see the metrics and labels attached to them? Ethical 2 Analytics Committee
Right to correct inaccurate data What data should students be allowed to correct about themselves? Legal 1 Analytics Committee
Data portability What data about themselves should students be able to take with them? Legal 2 Analytics Committee
Action Institutional obligation to act What obligation does the institution have to intervene when there is evidence that a student could benefit from additional support? Legal / Ethical 1 Analytics Committee
Student obligation to act What obligation do students have when analytics suggests actions to improve their academic progress? Ethical 2 Student
Conflict with study goals What should a student do if the suggestions are in conflict with their study goals? Ethical 3 Student
Obligation to prevent continuation What obligation does the institution have to prevent students from continuing on a pathway which analytics suggests is not advisable? Ethical 2 Analytics Committee
Type of intervention How are the appropriate interventions decided on? Logistical 1 Educational researcher
Distribution of interventions How should interventions be distributed across the institution? Logistical 1 Analytics Committee
Conflicting interventions How does the institution ensure that it is not carrying out multiple interventions with conflicting purposes? Logistical 2 Educational researcher
Staff incentives for intervention What incentives are in place for staff to change practices and facilitate intervention? Logistical 3 Analytics Committee
Failure to act What happens if an institution fails to intervene when analytics suggests that it should? Logistical 3 Analytics Committee
Need for human intermediation Are some analytics better presented to students via e.g. a tutor than a system? Ethical 2 Educational researcher
Triage How does an institution allocate resources for learning analytics appropriately for learners with different requirements? Ethical / Logistical 1 Analytics Committee
Triage transparency How transparent should an institution be in how it allocates resources to different groups? Ethical 3 Analytics Committee
Opportunity cost How is spending on learning analytics justified in relation to other funding requirements? Logistical 2 Senior management
Favouring one group over another Could the intervention strategies unfairly favour one group over another? Ethical / Logistical 2 Educational researcher
Consequences of false information What should institutions do if a student gives false information e.g. to obtain additional support? Logistical 3 Analytics Committee
Audit trails Should institutions record audit trails of all predictions and interventions? Logistical 2 Analytics Committee
Unexpected findings How should institutions deal with unexpected findings arising in the data? Logistical 3 Analytics Committee
Adverse impact Labelling bias Does labelling or profiling of students bias institutional perceptions and behaviours towards them? Ethical 1 Educational researcher
Oversimplification How can institutions avoid overly simplistic metrics and decision making which ignore personal circumstances? Ethical 1 Educational researcher
Undermining of autonomy Is student autonomy in decision making undermined by predictive analytics? Ethical 2 Educational researcher
Gaming the system If students know that data is being collected about them will they alter their behaviour to present themselves more positively, thus distracting them and skewing the analytics? Ethical 2 Educational researcher
Abusing the system If students understand the algorithms will they manipulate the system to obtain additional support? Ethical 3 Educational researcher
Adverse behavioural impact If students are presented with data about their performance could this have a negative impact e.g. increased likelihood of dropout? Ethical 1 Educational researcher
Reinforcement of discrimination Could analytics reinforce discriminatory attitudes and actions by profiling students based on their race or gender? Ethical 1 Educational researcher
Reinforcement of social power differentials Could analytics reinforce social power differentials and students’ status in relation to each other? Ethical 2 Educational researcher
Infantilisation Could analytics “infantilise” students by spoon-feeding them with automated suggestions, making the learning process less demanding? Ethical 3 Educational researcher
Echo chambers Could analytics create “echo chambers” where intelligent software reinforces our own attitudes and beliefs? Ethical 3 Educational researcher
Non-participation Will knowledge that they are being monitored lead to non-participation by students? Ethical 2 Educational researcher
Stewardship Data minimisation Is all the data held on an individual necessary in order to carry out the analytics? Legal 1 Data scientist
Data processing location Is the data being processed in a country permitted by the local data protection laws? Legal 1 IT
Right to be forgotten Can all data regarding an individual (expect that necessary for statutory purposes) be deleted? Legal 1 IT
Unnecessary data retention How long should data be retained for? Legal 1 Analytics Committee
Unhelpful data deletion If data is deleted does this restrict the institution’s analytics capabilities e.g. refining its models and tracking performance over multiple cohorts? Logistical 2 Data scientist
Incomplete knowledge of data sources Can an institution be sure that it knows where all personal data is held? Legal / Logistical 1 IT
Inappropriate data sharing How can data sharing be prevented with parties who have no legitimate interest in seeing it or who may use it inappropriately? Legal 1 IT

 

 

 


  • -
B4AOo1iCAAEos2s

Developing the Code of Practice

A wide-ranging discussion took place in London last week to discuss the emerging Code of Practice for Learning Analytics.  A new advisory group for the Code includes representatives from the National Union of Students, Edinburgh, Huddersfield, Lancaster and Loughborough Universities, Bucks New University, The Open University, Croydon College and Jisc.

A Code of Practice for Learning Analytics has been identified by Jisc stakeholders in higher and further education as a development priority.  The literature review is the first stage in this activity and provides the underlying rationale.  The next stage is developing the document itself.

The Code will aim to help to remove barriers to the adoption of learning analytics.  We can ensure that its emphasis is positive, realistic and facilitative.  It’ll provide a focus for institutions to deal with the many legal and ethical hurdles which are arising, and can be presented as an evolving, dynamic site rather than a lengthy one-off document which hardly anyone reads, let alone adheres to.

Jisc will coordinate the development and roll-out of the Code.  Meanwhile advisory group members agreed to critique the Code as it’s being developed and consider piloting it at their own institutions.

Methodology and approaches
Some documents take a particular methodological or philosophical stance.  For instance Slade and Prinsloo’s socio-critical approach – where learning analytics is viewed as a “transparent moral practice” and students are seen as co-contributors – has influenced the Open University’s Policy on Ethical Use of Student Data.  Should the Code take such an approach?

One of the main challenges will be to strike a balance between a paternalistic approach and respecting students’ privacy and autonomy.  It was suggested that the various uses for student data might have different approaches to consent:

  1. Helping individual students based on their own data
  2. Merging individuals’ data with those of others to help the group
  3. Using data to help future cohorts of students

Informed consent could potentially be obtained for each of these options.

There was also concern expressed about ensuring that any sharing of student data outside the institution should be carefully controlled.  The Code itself should have boundaries and may need to reference other institutional policies.  There should be differentiation between demographic and behavioural data, and the “right to be forgotten” needs to be addressed.

A separate document for students?
An approach which puts the needs of learners at the heart of the Code is surely likely to result in a better and more widely-adopted document which helps to allay the fears of students and institutions and facilitate the uptake of learning analytics.  The inclusion of the NUS in this group is therefore particularly welcome.

There will need to be a balance and series of compromises struck however to develop a usable Code and encourage mutual understanding. The group decided a single document setting out clearly the rights and responsibilities of students, institutions and staff would be preferable to having a separate student “bill of rights for learning analytics”.

Explaining what the Code means in practice however may require separate advice for different stakeholders.  At institutions the Code should link closely with the student charter, and involve buy-in from the students’ union.

Striking a balance between high level principles and detailed guidance
Can the Code be sufficiently high level to meet the needs of all institutions while remaining specific enough to provide genuinely helpful guidance?  It was very clear from my institutional visits that the potential uses of learning analytics and the concerns raised varied widely across institutions.  The group thought that the document should be fairly high level in order to prove useful to all, but should be backed up by case studies and examples of how institutions have dealt with particular issues.  The case studies could be released alongside the code – for each principle there could be examples of good practice.

Conformance with the Code
Another question I posed to the group was whether we should encourage institutions to adopt the Code wholesale, and therefore be able to claim conformance with it, or to customise it to their own requirements?  We probably need to see the end result first but it was felt that institutions might want to be able to adopt the Code with local modifications.

Human intermediation
Particular concern was expressed that the Code needs to reflect the human context and the need for intermediation of learning analytics by staff. This is a common ethical theme in the literature.  However a representative from the Open University said that the sheer scale of that institution makes it unfeasible to use human intermediation for many of the potential uses of learning analytics.  Meanwhile there was real concern among members that the language which is used to present analytics to students should be carefully considered and that data is only exposed when institutions have mechanisms in place to deal with the effect on students.  The potential impact of analytics on the educator also needs to be reflected in the Code.

Format
All of the related codes of practice I’ve looked at are textual documents – normally provided in PDF.  The members felt that a document outlining the principles needed to be provided in order to present it to institutional committees but that an interactive website containing case studies, perhaps in the form of videoed interviews with staff and students, would be welcome.

Some codes are extremely lengthy and somewhat uninspiring papers stretching to thirty pages or more. One of the better formats I’ve seen is the Respect Code of Practice for Socio-Economic Research.  It’s concise – only four pages – and reasonably visually appealing, therefore arguably more likely to be read and absorbed by busy people than some of the longer codes.  However, given the large number of issues identified in our literature review, four pages is unlikely to be sufficient.

One approach would be to back up a concise summary document with more detailed online guidance for each of the areas.  Discussion forums could be included on each topic, enabling users to raise further issues which arise, and others to provide advice on how they’ve tackled that challenge.  This would need some ongoing promotion, facilitation and moderation by Jisc and/or members of the community.

Areas to be included
The literature review covers most of the ethical and legal issues which are likely to be of concern to students and to institutions when deploying learning analytics, though there may be some which I’ve missed or have not yet cropped up in the literature.  The section headings and the word clouds in the review could help prioritise the main areas to be included in the Code.  It was pointed out that it would be difficult to deal with all of these meaningfully within four pages but certainly each area could be expanded on in the supporting documentation.

Including vendors
One member suggested including vendors in the consultation process for the Code.  It might help them when making development decisions, for instance encouraging them to build consent systems into their products.  The Code could help to ensure that safeguards, such as ensuring privacy, are built in without holding back innovation.

Development process
Jisc will develop the Code up until May 2015 with guidance from the advisory group.  Supporting content e.g. videoed interviews can be developed subsequently, help raise awareness of the Code, provide examples of how it’s being implemented and help to keep it current.

A sense of ownership by institutions and by students is essential to ensure adoption.  How can this best be achieved?  A range of stakeholder organisations was proposed and a number of possible events to piggy-back on were suggested.  Several members said they’d be keen to try piloting the Code at their institutions too.  An experiential learning cycle was suggested, with institutions thinking about:

  1. What’s the ethical/legal issue?
  2. What’s the principle to deal with it?
  3. How did we apply the principle?

Roll-out and dissemination
There is already considerable awareness of the intended Code of Practice but how should it best be disseminated once developed?  One member suggested it would be useful to understand better the processes inside institutions for getting academic policies adopted as this will be key to uptake.  In addition, a couple of events specifically around the Code could be held, papers delivered at relevant conferences and approaches made to newspapers to see if they’d like to cover its launch.  It was felt that the Code should be launched with some fanfare at a larger event to increase awareness and potential take-up.

Now on with developing it…  Comments are welcome.


  • -
Solutions from codes of practice e.g. transparency

A taxonomy of the ethical and legal issues of learning analytics v0.1

In discussions around the ethics and legal issues of learning analytics I’ve found the same issues cropping up again and again. Almost always they’ve already been covered somewhere in the growing collection of publications on learning analytics. Sometimes they’re expressed in different ways but boil down to the same underlying problem.

The literature review of these issues aims to provide the background for the development of a code of practice for learning analytics. But it’s a large and unwieldy document to refer to so I’ve attempted to distil and group the many issues that I’ve come across so far.

I’ve given each of the resulting 86 issues a name and have provided a question which attempts to capture the issue. Many of these cannot be answered simply; almost all could be responded to with “It depends…” Most have both an ethical and a legal dimension. Some are related more to logistics than ethics or law. And some are already dealt with by existing institutional policies.

I’ll be taking this taxonomy to a workshop next week in Paris with the Lace Project and Apereo where I hope the issues and questions can be clarified and further refined. Then it’s over to the Code of Practice Advisory Group in the UK for their advice on how to translate this into a useful Code of Practice.

Area Issue Question
Validity Minimisation of inaccurate data How does an institution minimise inaccuracies in the data?
Minimisation of incomplete data How does an institution minimise incompleteness of the dataset?
Optimum range of data sources How many and which data sources are necessary for increasing accuracy in the analytics?
Verification of algorithms and metrics How should an institution verify its algorithms and metrics to ensure accuracy?
Spurious correlations How can institutions avoid drawing misleading conclusions from spurious correlations?
Evolving nature of students To what extent can analytics be accurate when students’ identities and actions evolve as they progress through their studies?
Authentication of public data sources How can institutions ensure that student data taken from public sites is authenticated to their students?
Ownership and control Control of data for analytics Who in the institution decides what data is collected and used for analytics?
Breaking silos How can silos of data ownership in institutions be broken into in order to obtain data for analytics?
Control of analytics processes Who in the institution decides how analytics are to be created and used?
Overall responsibility Who is responsible in the institution for the appropriate and effective use of learning analytics?
Ownership of data What data does the institution own and what is owned by the student?
Awareness Student awareness of data collection What should students be told about the data that is being collected about them?
Student awareness of data use What should students be told about the uses to which their data is being put?
Student awareness of algorithms and metrics To what extent should students be given details of the algorithms used for learning analytics and the metrics and labels that are created?
Proprietary algorithms and metrics What should institutions do if vendors decline to make details of their algorithms and metrics public?
Student awareness of potential consequences of opting out What should students be told about the potential consequences of opting out of data collection and analysis of their learning?
Staff awareness of data collection and use What should staff be told about the data that is being collected about their students and what is being done with it?
Consent and opting out When to seek consent In what situations should students be asked for consent to collection and use of their data for learning analytics?
Consent for anonymous use Should students be asked for consent for collection of data which will only be used in anonymised formats?
Consent for outsourcing Do students need to give specific consent if the collection and analysis of data is to be outsourced to third parties?
Clear and meaningful consent processes How can institutions avoid opaque privacy policies and ensure that students genuinely understand the consent they are asked to give?
Right to opt out Does a student have the right to opt out of data collection and analysis of their learning activities?
Partial consent Can students consent to some data collection and analysis but opt out elsewhere?
Right to withdraw Does a student have the right to withdraw from data collection and analysis after previously having given their consent?
Right to anonymity Should students be allowed to provide pseudonyms to disguise their identity in certain circumstances?
Adverse impact of opting out on individual If a student is allowed to opt out of data collection and analysis of their activities could this have a negative effect on their studies?
Adverse impact of opting out on group If individual students opt out will the dataset be incomplete, thus potentially reducing the accuracy and effectiveness of learning analytics for the group?
Lack of real choice to opt out Do students really have a choice if pressure is put on them by the institution or there’s a chance of adverse impact on their academic success by opting out?
Student input to analytics process Should students have a say in what data is collected and how it is used for analytics?
Change of purpose Should institutions request consent again if the data is to be used for purposes for which consent was not originally given?
Legitimate interest To what extent can the institution’s “legitimate interests” override privacy controls for individuals?
Unknown future uses of data How can consent be requested when potential uses of the (big) data are not yet known?
Consent in open courses Are open courses (MOOCs etc) different when it comes to obtaining consent?
Use of publicly available data Can institutions use publicly available data (e.g. tweets) without obtaining consent?
Student access Student access to their data To what extent should students be able to access the data held about them?
Student access to their analytics To what extent should students be able to access the analytics performed on their data?
Data formats In what formats should students be able to access their data?
Metrics and labels Should students see the metrics and labels attached to them?
Right to correct inaccurate data What data should individuals be allowed to correct about themselves?
Data portability What data about themselves can the learner take with them?
Privacy Out of scope data Is there any data that should not be used for learning analytics?
Access to employers Under what circumstances would it be appropriate to give employers access to analytics on students?
Tracking location Under what circumstances is it appropriate to track the location of students at campuses?
Staff permissions To what extent should access to individuals’ data be restricted within an institution?
Unintentional creation of senstitive data How do institutions avoid creating “sensitive” data (e.g. ethnicity, religion) from other data sources?
Use of metadata to identify individuals Can individuals be identified from metadada even if personal data has been deleted?
Requests from external agencies What should institutions do when requests for student data are made by external agencies e.g. educational authorities or security agencies?
Sharing data with other institutions Is it appropriate for institutions to share student data with other institutions in order to increase the dataset and enhance the analytics?
Enhancing trust by retaining data internally If students are told that their data will be kept within the institution will they develop greater trust in and acceptance of learning analytics?
Action Institutional obligation to act What obligation does the institution have to intervene when there is evidence that a student could benefit from additional support?
Student obligation to act What is the student’s obligation to act on learning analytics designed to help them?
Conflict with study goals What should a student do if the suggested advice is in conflict with their study goals?
Obligation to prevent continuation Is there an obligation on the institution to prevent students from continuing on a pathway if analytics show that it is not in the student’s or institution’s interests for them to continue?
Type of intervention How are the appropriate interventions decided on?
Distribution of interventions How should interventions resulting from analytics be distributed among different stakeholders in the institution?
Conflicting purposes of interventions How does the institution ensure that it is not carrying out multiple interventions whose purposes conflict?
Staff incentives for intervention What incentives are in place for staff to intervene?
Failure to act What happens if an institution fails to intervene?
Need for human intermediation Are some analytics better presented to students via e.g. a tutor than via a system?
Triage How does an institution allocate resources for learning analytics appropriately for learners with different requirements?
Triage transparency How transparent should an institution be in how it allocates resources to different groups?
Opportunity cost How is spending on learning analytics justified in relation to other funding requirements?
Favouring one group over another Could the intervention strategies favour one group of students over another?
Consequences of false information What should institutions do if it is determined that a student has given false information to e.g. obtain additional support?
Audit trails Should institutions record audit trails of all predictions and interventions?
Unexpected findings What infrastructure is in place to deal with something unexpected arising in the data?
Adverse impact Labelling bias Does a student profile or labelling bias institutional perceptions and behaviours towards them?
Oversimplification How can institutions avoid overly simplistic metrics and decision making which ignore personal circumstances?
Undermining of autonomy Is student autonomy in decision making about their learning undermined by predictive analytics?
Gaming the system If students know that data is being collected about them will they alter their behaviour to present themselves more positively, thus skewing the analytics and distracting them from their learning?
Abusing the system If students understand the algorithms behind learning analytics will they abuse the system to obtain additional support?
Adverse behavioural impact If students are presented with data about their performance, likelihood of failure etc. could this have a negative impact on their behaviour, leading to increased likelihood of failure and dropout?
Reinforcement of discriminatory attitudes and actions Could analytics reinforce discriminatory attitudes and actions by profiling students based on e.g. their race or gender?
Reinforcement of social power differentials Could analytics reinforce social power differentials and learners’ status relating to each other?
Infantilisation Could analytics “infantalise” students by spoon-feeding them with automated suggestions, making the learning process less demanding?
Echo chambers Could analytics create “echo chambers” where intelligent software reinforces our own attitudes or beliefs?
Non-participation Will knowledge that they are being monitored lead to non-participation by students?
Stewardship Data minimisation Is all the data being held on an individual necessary in order to carry out the analytics?
Data processing location Is the data being processed in a country permitted by the local data protection laws?
Right to be forgotten Can all data regarding an individual except that necessary for statutory purposes be deleted?
Unnecessary data retention How long should data be retained for?
Unhelpful data deletion If data is deleted does this restrict the institution’s ability to refine its models, track performance over multiple cohorts etc?
Incomplete knowledge of data sources Can an institution be sure that it knows where all personal data is held?
Inappropriate data sharing How can we prevent data being shared within or outside the institution with parties who have no legitimate interest in seeing it or may use it inappropriately?
Risk of re-identification Does anonymisation of data becomes more difficult as multiple data sources are aggregated, potentially leading to re-identification of an individual later?

  • -
A student's engagement rating against the class average

A student app for learning analytics

Category : Ethics , Learning Analytics

A student's engagement rating against the class averageMany applications for learning analytics have been proposed, are under development or are already being deployed by institutions.  These range from prompting staff to intervene with students at risk of drop-out to attempting to understand whether some learning content or activity is effective.

Much of this is about providing better information to staff about students and their learning.  But if learning analytics is primarily about enhancing students’ learning shouldn’t we be putting analytics in the hands of the learners themselves?  This was the conclusion that participants at the co-design workshop back in the summer came to.

Jisc has recently begun the procurement process to commission a range of basic learning analytics services for UK further and higher education.  One of these services is the provision of a student app, taking its data primarily from a learning analytics warehouse which in turn is likely to source data from VLEs (LMSs), student information systems and elsewhere.

Jisc is hosting an event in February where we’ll bring together people from universities and colleges across the UK to look at what they think can and should be provided to students directly.  The requirements gathering process will find out from students directly what analytics services they would be most interested in having at their fingertips on a smartphone or tablet.  Here are some initial thoughts about what might crop up:

Measuring engagement
Students might find visualisation of their participation in a course of use, measured through a variety of metrics such as VLE access, campus attendance and library use.  Comparisons with peers may be helpful.  And comparisons with previous cohorts, showing the profile of a successful student might be useful too.   These could be presented in a variety of ways, including graphs of engagement over time compared with others.  Learners might want to have alerts sent to their device through the app if their participation shows they’re falling below an acceptable level.

Measuring assessment performance
There is clearly a need to show details of assessments already completed and grades obtained, and the dates, locations and requirements of impending ones.  Assessment events transferred to your calendar with advance alerts could also be useful.  But arguably this is simple reporting and alerting functionality and not learning analytics.  A progress bar showing how you are progressing through your module and qualification might be helpful.  Otherwise assessment data could feed into one of the metrics used for measuring engagement.

Module choice
One application of learning analytics is to assist students in making module choices.  Analytics can recommend modules where you are most likely to succeed, comparing your profile with those of previous students and presenting you with information such as “Students with similar profiles to you have tended to perform better when selecting xxx as their next module”.

Issues
The above proposed functionality comes with ethical questions, such as: Could an app showing you’re falling behind and likely to fail a module be de-motivational and act as a self-fulfilling prophecy?  And the module choice example is of course highly dependent on the sophistication of the algorithm, and potentially restricts free choice.  I’ve discussed these and many other ethical issues in a recently-published literature review which is the precursor to a Code of Practice for learning analytics which Jisc is co-developing with the sector.

Another issue is whether it makes sense from the student’s point of view to separate an analytics app from other student-facing functionality.  Apps containing details of transport, campus maps, your next lecture, computer availability in the library and much else that is digitised on campus are already available to students in many institutions.  Having a separate analytics app might be inconvenient.  On the other hand mobile apps tend to have a limited amount of functionality compared with traditional full-scale PC applications.  An app for monitoring your learning might make sense in its own right.

A student-facing app may make learning analytics more tangible and show people the possibilities of using all that data being accumulated to benefit students directly.  I’ve only scratched the surface of what’s possible in the suggestions above.  The event planned for February has already had a large amount of interest from the sector and we’re looking forward to gathering innovative suggestions from staff and students across the UK to be built into the Jisc app.  Stay tuned to this blog for updates on progress.


  • 2
Twitter analytics

Retweeting: ethics and why “RT”, “MT” and plagiarism are good

Category : Ethics , Twitter

 

Retweeting matters – hugely.  A scan of my tweet stream shows that most tweets have already been retweeted or are themselves retweets of other tweets.  It’s the same situation with Facebook.  Social media is an endless recycling of other people’s thoughts or creations.

The simple fact is that most of us don’t have the time to come up with original content but if we see something interesting we want to share it with others.  Our motivations for sharing are complex.  We might want to enhance our professional credibility by sharing an important development in our field, and be the first to do so.  This can attract followers, itself boosting work opportunities or our fragile egos if we measure our value by the number of followers, favourites and retweets we get.

Retweeting may also simply be driven by the desire for others to share our enjoyment of an amazing video, amusing cartoon or fascinating article.

RT
It didn’t take long for Twitter users to start prefixing the acronym “RT” when resending other people tweets.  In the academic world this is important – implying that you wrote the tweet yourself is a form of plagiarism, and doesn’t give credit to its originator.

The problem with using “RT” was that your tweet stream would then sometimes be filled with the same tweet retweeted by multiple people.  So Twitter invented the retweet function which meant that the tweet would only appear once in your feed, no matter how many people subsequently retweeted it this way.  But it also meant that you had no recognition for retweeting any more.  Twitter then added a further function to show how many retweets your retweet had received, which solved the problem of lack of recognition while crediting the original author of the tweet for their work.

Modifying and de-identifying
The “MT” or modified tweet is one way people are attempting to make the tweet their own but still give credit to its originator.  It’s also a good way of ensuring that you subsequently get the retweets rather than the originator.

Not everyone is driven by an ethical desire to avoid plagiarism.  There are people who appear to be making a career out of scanning multiple feeds to interesting content and then repackaging these into their own tweets as if they were the first to discover it – de-identifying the original author.  They then get the credit for being the expert, and the ego boost from retweets of their tweet, new favourites and followers.

Twitter analytics
Being involved in a project at the moment where disseminating information about it is important I’ve been thinking a lot about these issues.  Retweeting matters because it’s a sign that what we’re doing is of interest to people. It also has a monetary value attached to it in the commercial arena.  There are packages such as Twitter analytics which show the reach of your tweets – though they miss all the RTs, MTs and de-identified retweets.  But some analytics systems are getting more sophisticated and gather similar tweets which may show how your tweet was repackaged by others to give you a more realistic view of its impact.

Twitter analytics

 

Getting lost in a sea of tweets
As many Twitter users follow hundreds or thousands of other accounts it’s more than likely that when you tweet something it’ll be lost in a sea of other tweets and they’ll never see it.  When others retweet your comment it is doesn’t necessarily help because if your followers are also following them they won’t see the retweet.

Some authors tweet about the same thing in a different way at different times of the day to attempt to put their work in front of you if you missed it the first time.  But I can’t help feeling that that just tends to get annoying if you did see it already and I don’t personally want to be tweeting something more than once, particularly as a friend told me once, flatteringly, that his mobile phone beeps every time I send a tweet.

So RTs, MTs and simple plagiarised, non-accredited retweets of your original comment – if it includes a link to your content somewhere else – end up being great as more people are likely to view your content.  If those people all themselves get retweets, reputation enhancement or new followers, ego boosts or a simple joy of sharing along the way then good luck to them.

 


  • 13
Solutions from codes of practice e.g. transparency

Jisc releases report on ethical and legal challenges of learning analytics

Literature review coverDo institutions need to obtain consent from students before collecting and using data about their online learning activities?

Should learners be allowed to opt out of having data collected about them?

Could showing students predictions about their likelihood of academic success have a negative effect on their motivation and make them more likely to drop out?

These are some of the questions addressed in a new report which is released today by Jisc: Code of practice for learning analytics: A literature review of the ethical and legal issues.

The aim of the review is to provide the groundwork for a code of practice which is intended to help institutions solve the complex issues raised by learning analytics.

It was a very interesting task gathering together publications from many different authors and organisations.  I drew material from 86 publications, more than a third of them published within the last year from sources including:

  • The literature around learning analytics which makes explicit reference to legal and ethical issues
  • Articles and blogs around the ethical and legal issues of big data
  • A few papers which concentrate specifically on privacy
  • Relevant legislation, in particular the European Data Protection Directive 1995 and the UK Data Protection Act 1998
  • Related codes of practice from education and industry

Expressing issues as questions can be a useful way of making some of the complexities more concrete. I’ve incorporated 93 questions from the literature that authors had posed directly. The categorisations of these highlighted in the word cloud below give an instant flavour of the main concerns around the implementation of learning analytics being raised by researchers and practitioners.

Issues of concern

 

At the end of the report I reviewed 16 codes of practice or lists of ethical principles from related fields and found the main concepts people wish to embody are transparency, clarity, respect, user control, consent, access and accountability.

I’ve attempted to be comprehensive but I challenge you to spot any relevant legal or ethical issues I’ve missed in the report – please let us know in the comments below if you find any.  The next task is to put together an expert group to advise on the development of the code of practice … and to begin drafting the document itself.


  • 1
Participants at workshop

Notes from Utrecht Workshop on Ethics and Privacy Issues in the Application of Learning Analytics

I’m back from yesterday’s excellent Workshop on Ethics & Privacy Issues in the Application of Learning Analytics  in Utrecht organised by LACE and SURFHendrik Drachsler from the Open University of the Netherlands kicked off the session by presenting a background to learning analytics and some of the resulting ethical and privacy issues.  He mentioned the situation in the Netherlands where universities are now partially funded on the basis of how many students they graduate, and concerns that that gives them an incentive not to accept students who are predicted to fail.

Participants at workshop

He also discussed the InBloom debacle in the US – “a perfect example of not taking care about privacy issues”.  There was another situation in the Netherlands where an app used on tablets in schools collected data on which further analysis was carried out.  Problems arose because this analysis wasn’t described in the terms and conditions of use.

Hendrik mentioned that his call for ethical and privacy issues in the application of learning analytics had produced over 100 issues.  These were then put into four categories: privacy, ethics, data and transparency.  The aim of the day was to discuss these issues and begin to look for solutions to them.

The group decided that there are often no clear boundaries between these categories.  Certainly I’ve found it artificial to try to split legal issues from ethical ones when carrying out my recent literature review of the area.  Much of the law is based on ethics – and sometimes an ethical stance has to be applied when interpreting the law in particular situations.

The workshop wasn’t purely full of Hendriks, but a second Hendrik, Hendrik vom Lehn then gave an informative presentation on practical considerations around some of the legal issues arising from learning analytics.  Much of what he said, and subsequent discussions during the day, related to the EU Data Protection Directive.  Hendrik thought that a common misconception about the Directive is that the scope of “personal data” is much broader than most people think, and includes absolutely everything which makes data personally identifiable.

Another interesting concept in the Directive is that data can potentially be processed without consent from individuals if it is in the “legitimate interest” of the organisation to do so.  However in practice it’s likely to be better to inform students and obtain their consent for data collection and the resulting analytics.  Hendrik also discussed the US concept of “reasonable expectation”: people whose data is being processed should have reasonable expectations of what is being done with it.  Thus if you start using it in new ways (e.g. the recent Facebook mood altering experiment) you’re on dangerous ground.

Anonymisation is often proposed as an alternative to obtaining consent, but this can be difficult to achieve.  It’s particularly problematic in small groups where behaviours can easily be attributed to an individual.

Hendrik felt that where grading is based on learning analytics or can in some way affect the career of the student, this could have legal implications.  Another issue he mentioned, which I hadn’t come across before, was the subordinate position of students, and that they might feel obliged to participate in data collection or learning analytics activities because they were being graded by the same person (or institution) that was analysing them.  Would any consent given be regarded as truly voluntary in that case?

A member of the audience then asked if there was a difference between research and practice in learning analytics.  Hendrik suggested that ethically our approach should be the same but from a legal perspective there may be a difference.

So what happens if a student asks for all the data that an institution has about them?  Hendrik thought that the Directive implied that we do indeed need to make everything we know about students available to them.  However there might be a possible conflict between full data access and the wider goals of learning analytics – it might make it easier for students to cheat, for example.  Also it may be difficult to provide meaningful access for an individual while excluding other students’ data.

Another potentially difficult area is outsourcing and data transfers to third parties.  This is particularly problematic of course when that data is being transferred outside the European Union.  For students the process of understanding what is happening to their data – and accessing it – can then become more complex and they may have to go through several steps.  Ownership of the data is not complete in this situation for any party (though in a later discussion it was proposed that “ownership” is not a helpful concept here – more relevant are the EU concepts of “data controller” and “data processor”).

We then split into groups and had the benefit of some great input from Jan-Jan Lowijs –  a privacy consultant from Deloitte.  He described the nine general themes in the Directive which we found a useful way to propose answers to some of the 100 issues that had been submitted.  These are:

  1. Legitimate grounds – why you should have the data in the first place
  2. Purpose of the data – what you want to do with it
  3. Data quality – minimisation, deletion etc
  4. Transparency – informing the students
  5. Inventory – knowing what data you have and what you do with it already
  6. Access – the right of the data subject to access their data, when can you have access to it and what can you see
  7. Outsourcing – and the responsibilities of your institution as data controller and the third party as data processor
  8. Transport of data – particularly problematic if outside the EU
  9. Data security

Attempting to answer some of questions submitted using the themes as guidance resulted in the following:

Who has access to data about students’ activities?
Students themselves and certified access for teachers, researchers etc, based on theme 2 above (purpose of data)

What data should students be able to view?
All data on an individual should be provided at any time they request it – that’s the situation to aim for, based on theme 6 (access)

Should students have the right to request that their digital dossiers be deleted on graduation?
Yes, so long as there are no other obligations on the institution to keep the data e.g. names, date of birth, final grades, based on theme 3 (data quality)

What are the implications of institutions collecting data from non-institutional sources (e.g. Twitter)?
Consent must be obtained from the students first, based on theme 4 (transparency). A case in Finland where two students sued their university who were re-using their Twitter data was noted.

Something interesting that Jan-Jan also mentioned was that there are differences in data subjects’ attitudes to privacy, and that a number of studies have shown a fairly consistent split of:

  • 25% “privacy fundamentalists” who don’t want to share their data
  • 60% pragmatists who are happy to share some of their data for particular purposes
  • 15% people who “don’t care” what happens to their data

An organisation needs therefore to make an active decision as to whether it attempts to cater for these different attitudes or finds some middle ground.

Some of the conclusions from the day in the final session were:

  1. It was noted that students were absent from the discussions and should be involved in the future.
  2. It was suggested that we fully articulate the risks for institutions of learning analytics. What are the showstoppers?  Are they reputational or based on a fear of loss of students?
  3. “Privacy by design” and user-centred design with much better management of their data by users themselves were thought to be vital.
  4. InBloom was suggested as an “anti-pattern”, to be studied further to establish what we shouldn’t be doing.
  5. If you think something’s dodgy then it probably is. I have to admit being slightly concerned to hear that one university has equipment in its toilets to ensure that you’re not using your mobile phone to cheat if you have to nip out during an exam.  A good rule of thumb proposed by Jan-Jan is that if you feel uneasy about some form of data collection or analysis then there’s probably something to be worried about.

The outcomes from the day were being much more rigorously written up than I have done above by a professional writer – and will be fed into subsequent workshops held by LACE with the aim of producing a whitepaper or longer publication in the area.