Skip to content
Snippets Groups Projects
Commit 32f202be authored by Jan Bernoth's avatar Jan Bernoth
Browse files

Clean up 

* cleaned main
* cleaned paths
* cleaned description and readme
parent 0e5b2ca9
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
README.md
+ 25
− 81
View file @ 32f202be
# UEQ+ Erweiterung - Dashboard # UEQ+ Erweiterung - Dashboard
## Table of Contents ## Table of Contents
<!-- TOC -->
* [Table of Contents](#table-of-contents)
* [Description](#description) * [Description](#description)
* [Project Structure](#project-structure) * [Project Structure](#project-structure)
* [Roadmap](#roadmap) * [Roadmap](#roadmap)
* [New Items for the UEQ+ v2](#new-items-for-the-ueq-v2) * [Used scales in the questionnaire](#used-scales-in-the-questionnaire)
<!-- TOC -->
## Description ## Description
This project is a survey to extend the product classes of the UEQ+ This purpose of this data repository is introduces new product categories, Dashboard and Virtual Reality, to the User
[UEQ+ v2 Handbook](https://ueqplus.ueq-research.org/Material/UEQ+_Handbook_V2.pdf). Here 16 product categories are Experience Questionnaire ([UEQ+](https://ueqplus.ueq-research.org/Material/UEQ+_Handbook_V2.pdf)).
analyzed and relevant scales are described. To introduce a new product class one can follow The UEQ+ is a standardized questionnaire that provides a set of scales considered to be most important for evaluating
the procedure of [Winter et al., 2017](https://dx.doi.org/10.18420/muc2017-up-0002). To check replicability we conducted the user experience of a product family.
a survey regarding two originally known product classes and then added two new ones: *Dashboard* and *VR Applications*. We have conducted a quantitative study to determine the most important scales.
We also updated the UEQ+ to UEQ+v2, which means that some items had to be newly created as you can see [here](#new-items-for-the-ueq-v2). For this purpose, a procedure that had been used in previous studies to determine the most important User Experience
scales for previously released categories (Games and Learning Platforms) was reproduced.
### Research Question and Hypothesis These product categories were integrated to ensure that the conditions of our study were similar.
The results are presented, critically discussed, and compared with previous studies in a scientific publication.
1. Can the results from the initial study be replicated?
* H.1: Results for learning management systems (LMS) and Games will be the same as they were in the original study.
* H.2:
2. Which scales are relevant for the new category classes Dashboard and VR?
* H.3: Dashboards and LMS show different results regarding UEQ+ scales.
* H.4: VR applications and Games show comparable results regarding relevant UEQ+ scales.
### Dashboard
Dashboard applications could be presented in Info-Web-Sites or Learning Platforms product categories because there are To reproduce all plots, run:
some aspects to dashboards that allow users to access information or engage in targeted learning. However, there are ````
other use cases that are not covered by these categories, such as the goal of creating understandable data pip install -r requirements.txt
visualizations. python src/main.py
````
In Learning Analytics, dashboards are commonly used to aggregate and display indicators that support stakeholders in the
learning process
[Swendimann et al. 2017](https://doi.org/10.1109/TLT.2016.2599522). A quick scan showed that publications about learning
analytics dashboards with a user experience survey only use the UEQ
[Publication 1](https://doi.org/10.1145/3303772.3303793),
[Publication 2](https://www.jstor.org/stable/10.2307/26921130),
[Publication 3](https://doi.org/10.1145/3375462.337551). This lead to the assumption that there is a need for an own
category in UEQ+.
### Virtual Reality (VR)
Virtual Reality is a recently emerging technology, which can be used for entertainment activities. There are also
upcoming simulations used for educational purposes (e.g. [Wiepke et al. 2019](http://dx.doi.org/10.18420/delfi2019_319)),
which probably share some desired aspects with known entertainment content, but still follow a different goal.
Since it is hypothesised to rate User Experience of VR applications differently than desktop experiences(see
[Tcha-Tokey et al. 2018](https://doi.org/10.1155/2018/7827286)) it is analyzed if gaming content evokes other categories
than VR applications do.
The notebook is outdated.
## Project Structure ## Project Structure
...@@ -59,17 +36,10 @@ The folder structure is based on ...@@ -59,17 +36,10 @@ The folder structure is based on
``` ```
| -- data # folder for raw (mostly survey) data and questionnaire | -- data # folder for raw (mostly survey) data and questionnaire
| -- doc # documentation of the computation and analyzation
| -- results # results of the survey and visualizations | -- results # results of the survey and visualizations
| -- src # possible source code to compute the results | -- src # possible source code to compute the results
``` ```
## Survey
This survey was conducted using an instance of the online survey tool
[soSci](https://www.soscisurvey.de/en/index) at the University of Potsdam:
[Umfragen.UP](https://umfragenup.uni-potsdam.de).
## Roadmap ## Roadmap
- [x] create questionnaire - [x] create questionnaire
...@@ -78,33 +48,14 @@ This survey was conducted using an instance of the online survey tool ...@@ -78,33 +48,14 @@ This survey was conducted using an instance of the online survey tool
- [x] start date: 05.07.22 - [x] start date: 05.07.22
- [x] final date: 30.11.22 - [x] final date: 30.11.22
- [x] upload data - [x] upload data
- [x] subset 04.08.22 - [x] subset 04.08.22
- [x] final data 25.01.23 - [x] final data 25.01.23
- [ ] analyse data and publish results - [x] analyse data and publish results
- [ ] produce publication - [x] produce publication - MuC
In dieser Umfragen können Sie an Hand von mehreren Produktkategorien bestimmen, wie wichtig Ihnen einzelne Merkmale der
Benutzungsoberfläche sind. Die Umfrage wird ca. 5-10 Minuten in Anspruch nehmen.
Ihre Daten werden anonymisiert gespeichert und zu Forschungszwecken weiter verwendet. Das bedeutet, dass die Resultate
sowohl in Promotionsprojekten als auch für etwaige Publikationen verwendet werden können. Durch die anonymisierte
Speicherung ist es uns unmöglich, im Nachhinein bereits erhobene, personenspezifische Daten zu ändern oder zu löschen.
Durch das Fortsetzen der Umfrage bestätigen Sie, mit dem Datenschutzbedingungen einverstanden zu sein.
Für weitere Fragen und Anmerkungen wenden Sie sich bitte bei an Jan Bernoth (jan.bernoth@uni-potsdam.de). ## Used scales in the questionnaire
## New items for the UEQ+ v2 The list of scales and their corresponding questionnaire sources are presented below.
This is just in german, because our questionnaire is in german and the original paper is written in german. Anyway,
some decisions are made to extend the questionnaire to v2:
* New sentences were formed by the explanatory words for each scale. For example, attractiveness is not included into
the original questionnaire, then we took the introducing sentence from the scale of the
[UEQ+ v2 Template](https://ueqplus.ueq-research.org/Material/UEQPlus_Templates.zip) "In my opinion, the product is
generally" and connect it with some positive attributes from the scale like enjoyable, pleasant, friendly.
* The last three attributes, which are mainly related to voice assistance, are not considered because the proposed
product categories are not equipped with this characteristic.
| Items | In [UEQ+](https://ueqplus.ueq-research.org/Material/UEQ+_Handbook_V2.pdf) | In the [Paper](https://dl.gi.de/bitstream/handle/20.500.12116/5770/2017_UP_002.pdf?sequence=2&isAllowed=y) | used in questionnaire | Erklärender Satz | | Items | In [UEQ+](https://ueqplus.ueq-research.org/Material/UEQ+_Handbook_V2.pdf) | In the [Paper](https://dl.gi.de/bitstream/handle/20.500.12116/5770/2017_UP_002.pdf?sequence=2&isAllowed=y) | used in questionnaire | Erklärender Satz |
|---------------------------------------------------|:-------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------------------:|-----------------------|-----------------------------------------------------------------------------------------------------------------| |---------------------------------------------------|:-------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------------------:|-----------------------|-----------------------------------------------------------------------------------------------------------------|
...@@ -132,10 +83,3 @@ some decisions are made to extend the questionnaire to v2: ...@@ -132,10 +83,3 @@ some decisions are made to extend the questionnaire to v2:
| Verständnis (Comprehensibility) | ✔ | ❌ | ❌ | -- | | Verständnis (Comprehensibility) | ✔ | ❌ | ❌ | -- |
| Vertrauen (Trust) | ✔ | ✔ | ✔ | | | Vertrauen (Trust) | ✔ | ✔ | ✔ | |
| Wertigkeit (Value) | ✔ | ✔ | ✔ | | | Wertigkeit (Value) | ✔ | ✔ | ✔ | |
## Notes
* Interview Nr. 92 should be removed
* Replicability is not completely possible:
* Structure of the questionnaire is evident in terms of content, but the structuring is not comprehensible
* maybe too small sample size?
\ No newline at end of file
src/main.py
+ 46
− 46
View file @ 32f202be
...@@ -8,12 +8,10 @@ from matplotlib import ticker ...@@ -8,12 +8,10 @@ from matplotlib import ticker
from scipy.stats import pearsonr from scipy.stats import pearsonr
from enum import Enum from enum import Enum
from sklearn.manifold import MDS
import scipy
from kPOD import k_pod from kPOD import k_pod
import time import time
ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
class Categories(Enum): class Categories(Enum):
Dashboard = "Dashboard" Dashboard = "Dashboard"
LMS = "Learning Plattform" LMS = "Learning Plattform"
...@@ -30,7 +28,8 @@ class Interviews: ...@@ -30,7 +28,8 @@ class Interviews:
def __init__(self) -> None: def __init__(self) -> None:
super().__init__() super().__init__()
self.df = pd.read_csv('../data/data_ueq.csv', delimiter=';', quotechar='"', encoding='utf-16')
self.df = pd.read_csv(os.path.join(ROOT_DIR, '..', 'data', 'data_ueq.csv'), delimiter=';', quotechar='"', encoding='utf-16')
def __str__(self) -> str: def __str__(self) -> str:
return f'{self.df.shape[0]} interviews' return f'{self.df.shape[0]} interviews'
...@@ -43,8 +42,6 @@ class Interviews: ...@@ -43,8 +42,6 @@ class Interviews:
cat_games = all_categories.get_items(Categories.Games) cat_games = all_categories.get_items(Categories.Games)
cat_lms = all_categories.get_items(Categories.LMS) cat_lms = all_categories.get_items(Categories.LMS)
# remove interviews which are not complete # remove interviews which are not complete
print(cat_games)
print(self.df)
unwanted_interviews = self.df[self.df[cat_games[0]].isnull() | self.df[cat_lms[0]].isnull()] unwanted_interviews = self.df[self.df[cat_games[0]].isnull() | self.df[cat_lms[0]].isnull()]
print(f'{unwanted_interviews.shape[0]} interviews have not completed the questionnaire.') print(f'{unwanted_interviews.shape[0]} interviews have not completed the questionnaire.')
df_nonnull_interviews = self.df.drop(unwanted_interviews.index) df_nonnull_interviews = self.df.drop(unwanted_interviews.index)
...@@ -137,7 +134,7 @@ class Variables: ...@@ -137,7 +134,7 @@ class Variables:
def __init__(self) -> None: def __init__(self) -> None:
super().__init__() super().__init__()
self.df = pd.read_csv('../data/variables_ueq.csv', delimiter=';', quotechar='"', encoding='utf-16') self.df = pd.read_csv(os.path.join(ROOT_DIR, '..', 'data', 'variables_ueq.csv'), delimiter=';', quotechar='"', encoding='utf-16')
def get_groups(self): def get_groups(self):
return self.df[['VAR', 'LABEL']][7:12] return self.df[['VAR', 'LABEL']][7:12]
...@@ -150,7 +147,7 @@ class Values: ...@@ -150,7 +147,7 @@ class Values:
def __init__(self) -> None: def __init__(self) -> None:
super().__init__() super().__init__()
self.df = pd.read_csv('../data/values_ueq.csv', delimiter=';', quotechar='"', encoding='utf-16') self.df = pd.read_csv(os.path.join(ROOT_DIR, '..', 'data', 'variables_ueq.csv'), delimiter=';', quotechar='"', encoding='utf-16')
class ProductCategories: class ProductCategories:
...@@ -186,10 +183,12 @@ class ProductCategories: ...@@ -186,10 +183,12 @@ class ProductCategories:
def save_plot(plot, path): def save_plot(plot, path):
dir_path = os.path.abspath('../results/') dir_path = os.path.abspath(os.path.join(ROOT_DIR, '..', 'results'))
if not os.path.exists(dir_path): if not os.path.exists(dir_path):
os.mkdir(dir_path) os.mkdir(dir_path)
plot.savefig(fname=os.path.join(dir_path, path + '.png'), dpi='figure', format='png') saved_path = os.path.join(dir_path, path + '.png')
print("saved plot to " + saved_path)
plot.savefig(fname=saved_path, dpi='figure', format='png')
def replace_cryptic_names(columns): def replace_cryptic_names(columns):
...@@ -304,6 +303,7 @@ class Plotter: ...@@ -304,6 +303,7 @@ class Plotter:
super().__init__() super().__init__()
self.df = df self.df = df
self.pc = pc self.pc = pc
self.plotted = 0
def filter_by_gender(self, gender): def filter_by_gender(self, gender):
if gender == 'male': if gender == 'male':
...@@ -313,15 +313,11 @@ class Plotter: ...@@ -313,15 +313,11 @@ class Plotter:
else: else:
return self.df return self.df
def plot_groups(self, variable_labels): def plot_groups(self, variable_labels):
df_groups = self.df[['DE01_01', 'DE01_02', 'DE01_03', 'DE01_04', 'DE01_05']] - 1 df_groups = self.df[['DE01_01', 'DE01_02', 'DE01_03', 'DE01_04', 'DE01_05']] - 1
print(variable_labels)
df_groups.columns = ['Stu', 'Teach', 'Res', 'Tech', 'Out'] df_groups.columns = ['Stu', 'Teach', 'Res', 'Tech', 'Out']
sns.set_theme(style="whitegrid") sns.set_theme(style="whitegrid")
df_groups_sum = pd.DataFrame(df_groups.sum(), columns=['Sum']).sort_values(by='Sum', ascending=False) df_groups_sum = pd.DataFrame(df_groups.sum(), columns=['Sum']).sort_values(by='Sum', ascending=False)
print(df_groups_sum)
ax = sns.barplot(data=df_groups_sum, palette="Blues_d", errorbar=None, y="Sum", x=df_groups_sum.index) ax = sns.barplot(data=df_groups_sum, palette="Blues_d", errorbar=None, y="Sum", x=df_groups_sum.index)
ax.set(title='Academic Community (Multiple Choice)') ax.set(title='Academic Community (Multiple Choice)')
ax.yaxis.set_major_locator(ticker.MultipleLocator(5)) ax.yaxis.set_major_locator(ticker.MultipleLocator(5))
...@@ -330,8 +326,9 @@ class Plotter: ...@@ -330,8 +326,9 @@ class Plotter:
for tick in ax.yaxis.get_major_ticks(): for tick in ax.yaxis.get_major_ticks():
tick.label.set_fontsize(15) tick.label.set_fontsize(15)
sns.despine(left=True) sns.despine(left=True)
self.plotted = self.plotted + 1
save_plot(plt, 'demographics_communities') save_plot(plt, 'demographics_communities')
plt.show() return plt
def plot_gender(self): def plot_gender(self):
df_gender = self.df['DE02'] \ df_gender = self.df['DE02'] \
...@@ -340,19 +337,18 @@ class Plotter: ...@@ -340,19 +337,18 @@ class Plotter:
.replace(3, 'diverse') \ .replace(3, 'diverse') \
.replace(-1, 'not mentioned') .replace(-1, 'not mentioned')
# print(df_gender.sum()) # print(df_gender.sum())
print(pd.DataFrame(df_gender.value_counts(normalize=False)))
df_gender_count = pd.DataFrame(df_gender.value_counts(normalize=False)) df_gender_count = pd.DataFrame(df_gender.value_counts(normalize=False))
print(df_gender_count)
print(df_gender_count.index.values)
sns.set_theme(style="whitegrid") sns.set_theme(style="whitegrid")
ax = sns.barplot(data=df_gender_count, palette="Blues_d", errorbar=None, y="count", x=df_gender_count.index.values) ax = sns.barplot(data=df_gender_count, palette="Blues_d", errorbar=None, y="count",
x=df_gender_count.index.values)
ax.set(title='Gender') ax.set(title='Gender')
for tick in ax.xaxis.get_major_ticks(): for tick in ax.xaxis.get_major_ticks():
tick.label.set_fontsize(15) tick.label.set_fontsize(15)
for tick in ax.yaxis.get_major_ticks(): for tick in ax.yaxis.get_major_ticks():
tick.label.set_fontsize(15) tick.label.set_fontsize(15)
self.plotted = self.plotted + 1
save_plot(plt, 'demographics_gender') save_plot(plt, 'demographics_gender')
plt.show() return plt
def plot_faculty(self): def plot_faculty(self):
df_faculty = self.df['DE03'] \ df_faculty = self.df['DE03'] \
...@@ -369,9 +365,9 @@ class Plotter: ...@@ -369,9 +365,9 @@ class Plotter:
.replace(-9, 'nB') .replace(-9, 'nB')
ax = sns.countplot(x=df_faculty, palette="Blues_d") ax = sns.countplot(x=df_faculty, palette="Blues_d")
ax.set(title='Fakultät') ax.set(title='Fakultät')
print(df_faculty.value_counts(normalize=True)) self.plotted = self.plotted + 1
save_plot(plt, 'demographics_disciplines') save_plot(plt, 'demographics_disciplines')
plt.show() return plt
def plot_disciplines(self): def plot_disciplines(self):
df_faculty = self.df['DE04'] \ df_faculty = self.df['DE04'] \
...@@ -381,12 +377,11 @@ class Plotter: ...@@ -381,12 +377,11 @@ class Plotter:
.replace(4, 'Ingenieurwissenschaften') \ .replace(4, 'Ingenieurwissenschaften') \
.replace(-1, 'kA') \ .replace(-1, 'kA') \
.replace(-9, 'nB') .replace(-9, 'nB')
print(df_faculty)
ax = sns.countplot(x=df_faculty, palette="pastel") ax = sns.countplot(x=df_faculty, palette="pastel")
ax.set(title='Disciplines') ax.set(title='Disciplines')
print(df_faculty.value_counts(normalize=False)) self.plotted = self.plotted + 1
save_plot(plt, 'demographics_disciplines') save_plot(plt, 'demographics_disciplines')
plt.show() return plt
def plot_item(self, category_title, area, max_count=20, gender='all'): def plot_item(self, category_title, area, max_count=20, gender='all'):
category_items = self.pc.get_items(category_title) category_items = self.pc.get_items(category_title)
...@@ -406,16 +401,15 @@ class Plotter: ...@@ -406,16 +401,15 @@ class Plotter:
df_category_area = filter_columns_by_value(df_category, df_category.mean() >= 0) df_category_area = filter_columns_by_value(df_category, df_category.mean() >= 0)
else: else:
df_category_area = df_category df_category_area = df_category
print('------------- ' + category_title.value + "-" + area + "-" + gender)
df_category_area.columns = replace_cryptic_names(df_category_area.columns) df_category_area.columns = replace_cryptic_names(df_category_area.columns)
df_category_area = df_category_area.reindex(df_category_area.mean().sort_values(ascending=False).index, axis=1) df_category_area = df_category_area.reindex(df_category_area.mean().sort_values(ascending=False).index, axis=1)
df_category_area = df_category_area.iloc[:, : max_count] df_category_area = df_category_area.iloc[:, : max_count]
print(df_category_area.columns)
ax = sns.barplot(data=df_category_area, palette="Blues_d", orient='h', errorbar="sd") ax = sns.barplot(data=df_category_area, palette="Blues_d", orient='h', errorbar="sd")
ax.set(title=category_title.value + ' - ' + area) ax.set(title=category_title.value + ' - ' + area)
sns.despine(left=True) sns.despine(left=True)
self.plotted = self.plotted + 1
save_plot(plt, 'items_' + category_title.value + '_' + area + '_' + gender) save_plot(plt, 'items_' + category_title.value + '_' + area + '_' + gender)
plt.show() return plt
def plot_boxes(self, category_title, max_count=5): def plot_boxes(self, category_title, max_count=5):
df_category_area = self.df[self.pc.get_items(category_title)] df_category_area = self.df[self.pc.get_items(category_title)]
...@@ -447,11 +441,11 @@ class Plotter: ...@@ -447,11 +441,11 @@ class Plotter:
labels=translate_scales(df_category_area.columns, True), rotation=0) labels=translate_scales(df_category_area.columns, True), rotation=0)
ax.xaxis.set_label_position('top') ax.xaxis.set_label_position('top')
ax.xaxis.tick_top() ax.xaxis.tick_top()
self.plotted = self.plotted + 1
save_plot(plt, 'box_' + category_title.value) save_plot(plt, 'box_' + category_title.value)
plt.show() return plt
def m_scatter( self, x, y, ax=None, m=None, **kw): def m_scatter(self, x, y, ax=None, m=None, **kw):
import matplotlib.markers as mmarkers import matplotlib.markers as mmarkers
if not ax: ax = plt.gca() if not ax: ax = plt.gca()
sc = ax.scatter(x, y, **kw) sc = ax.scatter(x, y, **kw)
...@@ -579,25 +573,31 @@ def setup_dataframes(): ...@@ -579,25 +573,31 @@ def setup_dataframes():
if __name__ == '__main__': if __name__ == '__main__':
print('---------- Creating plots --------------')
interview, variables, values, product_categories, rankings = setup_dataframes() interview, variables, values, product_categories, rankings = setup_dataframes()
print(interview) print(interview)
df_filtered = interview.get_filtered(product_categories) df_filtered = interview.get_filtered(product_categories)
plotter = Plotter(df_filtered, product_categories)
statistics = Statistics(df_filtered, product_categories) statistics = Statistics(df_filtered, product_categories)
plotter = Plotter(df_filtered, product_categories)
print('average time to complete the questionnaire: ' + time.strftime('%M:%S', time.gmtime(statistics.mean_time())))
plotter.plot_groups(variables.get_groups()) plotter.plot_groups(variables.get_groups())
plotter.plot_gender() plotter.plot_gender()
# plotter.plot_disciplines() plotter.plot_disciplines()
plotter.plot_faculty() plotter.plot_faculty()
print(plotter.filter_by_gender('male')) for example in Categories:
# for example in Categories: plotter.plot_item(example, 'top')
# plotter.plot_item(example, 'top') plotter.plot_item(example, 'top', gender='male')
# plotter.plot_item(example, 'top', gender='male') plotter.plot_item(example, 'top', gender='female')
# plotter.plot_item(example, 'top', gender='female')
plotter.plot_item(example, 'mid')
# plotter.plot_item(example, 'mid') plotter.plot_item(example, 'low')
# plotter.plot_item(example, 'low') plotter.plot_boxes(example, 6)
# plotter.plot_boxes(example, 6)
# print(statistics.calc_corr()) print(statistics.calc_corr())
# print(statistics.similarity_by_ranked_scales(rankings.get_all_rankings())) print(statistics.similarity_by_ranked_scales(rankings.get_all_rankings()))
print(time.strftime('%M:%S', time.gmtime(statistics.mean_time()))) print(statistics.kpod_clustering(rankings.get_all_rankings()))
# print(statistics.kpod_clustering(rankings.get_all_rankings()))
print('-------- finished. Plotted: ' + str(plotter.plotted) + ' charts. -----------------')
src/survey_results.ipynb
+ 164
− 460
View file @ 32f202be
Source diff could not be displayed: it is too large. Options to address this: view the blob.
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment