Skip to content
Snippets Groups Projects
Commit 32f202be authored by Jan Bernoth's avatar Jan Bernoth
Browse files

Clean up

* cleaned main
* cleaned paths
* cleaned description and readme
parent 0e5b2ca9
No related branches found
Tags v0.0.1
No related merge requests found
# UEQ+ Erweiterung - Dashboard
## Table of Contents
<!-- TOC -->
* [Table of Contents](#table-of-contents)
* [Description](#description)
* [Project Structure](#project-structure)
* [Roadmap](#roadmap)
* [New Items for the UEQ+ v2](#new-items-for-the-ueq-v2)
* [Used scales in the questionnaire](#used-scales-in-the-questionnaire)
<!-- TOC -->
## Description
This project is a survey to extend the product classes of the UEQ+
[UEQ+ v2 Handbook](https://ueqplus.ueq-research.org/Material/UEQ+_Handbook_V2.pdf). Here 16 product categories are
analyzed and relevant scales are described. To introduce a new product class one can follow
the procedure of [Winter et al., 2017](https://dx.doi.org/10.18420/muc2017-up-0002). To check replicability we conducted
a survey regarding two originally known product classes and then added two new ones: *Dashboard* and *VR Applications*.
We also updated the UEQ+ to UEQ+v2, which means that some items had to be newly created as you can see [here](#new-items-for-the-ueq-v2).
### Research Question and Hypothesis
1. Can the results from the initial study be replicated?
* H.1: Results for learning management systems (LMS) and Games will be the same as they were in the original study.
* H.2:
2. Which scales are relevant for the new category classes Dashboard and VR?
* H.3: Dashboards and LMS show different results regarding UEQ+ scales.
* H.4: VR applications and Games show comparable results regarding relevant UEQ+ scales.
### Dashboard
This purpose of this data repository is introduces new product categories, Dashboard and Virtual Reality, to the User
Experience Questionnaire ([UEQ+](https://ueqplus.ueq-research.org/Material/UEQ+_Handbook_V2.pdf)).
The UEQ+ is a standardized questionnaire that provides a set of scales considered to be most important for evaluating
the user experience of a product family.
We have conducted a quantitative study to determine the most important scales.
For this purpose, a procedure that had been used in previous studies to determine the most important User Experience
scales for previously released categories (Games and Learning Platforms) was reproduced.
These product categories were integrated to ensure that the conditions of our study were similar.
The results are presented, critically discussed, and compared with previous studies in a scientific publication.
Dashboard applications could be presented in Info-Web-Sites or Learning Platforms product categories because there are
some aspects to dashboards that allow users to access information or engage in targeted learning. However, there are
other use cases that are not covered by these categories, such as the goal of creating understandable data
visualizations.
In Learning Analytics, dashboards are commonly used to aggregate and display indicators that support stakeholders in the
learning process
[Swendimann et al. 2017](https://doi.org/10.1109/TLT.2016.2599522). A quick scan showed that publications about learning
analytics dashboards with a user experience survey only use the UEQ
[Publication 1](https://doi.org/10.1145/3303772.3303793),
[Publication 2](https://www.jstor.org/stable/10.2307/26921130),
[Publication 3](https://doi.org/10.1145/3375462.337551). This lead to the assumption that there is a need for an own
category in UEQ+.
### Virtual Reality (VR)
Virtual Reality is a recently emerging technology, which can be used for entertainment activities. There are also
upcoming simulations used for educational purposes (e.g. [Wiepke et al. 2019](http://dx.doi.org/10.18420/delfi2019_319)),
which probably share some desired aspects with known entertainment content, but still follow a different goal.
Since it is hypothesised to rate User Experience of VR applications differently than desktop experiences(see
[Tcha-Tokey et al. 2018](https://doi.org/10.1155/2018/7827286)) it is analyzed if gaming content evokes other categories
than VR applications do.
To reproduce all plots, run:
````
pip install -r requirements.txt
python src/main.py
````
The notebook is outdated.
## Project Structure
......@@ -59,17 +36,10 @@ The folder structure is based on
```
| -- data # folder for raw (mostly survey) data and questionnaire
| -- doc # documentation of the computation and analyzation
| -- results # results of the survey and visualizations
| -- src # possible source code to compute the results
```
## Survey
This survey was conducted using an instance of the online survey tool
[soSci](https://www.soscisurvey.de/en/index) at the University of Potsdam:
[Umfragen.UP](https://umfragenup.uni-potsdam.de).
## Roadmap
- [x] create questionnaire
......@@ -78,33 +48,14 @@ This survey was conducted using an instance of the online survey tool
- [x] start date: 05.07.22
- [x] final date: 30.11.22
- [x] upload data
- [x] subset 04.08.22
- [x] final data 25.01.23
- [ ] analyse data and publish results
- [ ] produce publication
In dieser Umfragen können Sie an Hand von mehreren Produktkategorien bestimmen, wie wichtig Ihnen einzelne Merkmale der
Benutzungsoberfläche sind. Die Umfrage wird ca. 5-10 Minuten in Anspruch nehmen.
Ihre Daten werden anonymisiert gespeichert und zu Forschungszwecken weiter verwendet. Das bedeutet, dass die Resultate
sowohl in Promotionsprojekten als auch für etwaige Publikationen verwendet werden können. Durch die anonymisierte
Speicherung ist es uns unmöglich, im Nachhinein bereits erhobene, personenspezifische Daten zu ändern oder zu löschen.
Durch das Fortsetzen der Umfrage bestätigen Sie, mit dem Datenschutzbedingungen einverstanden zu sein.
- [x] subset 04.08.22
- [x] final data 25.01.23
- [x] analyse data and publish results
- [x] produce publication - MuC
Für weitere Fragen und Anmerkungen wenden Sie sich bitte bei an Jan Bernoth (jan.bernoth@uni-potsdam.de).
## Used scales in the questionnaire
## New items for the UEQ+ v2
This is just in german, because our questionnaire is in german and the original paper is written in german. Anyway,
some decisions are made to extend the questionnaire to v2:
* New sentences were formed by the explanatory words for each scale. For example, attractiveness is not included into
the original questionnaire, then we took the introducing sentence from the scale of the
[UEQ+ v2 Template](https://ueqplus.ueq-research.org/Material/UEQPlus_Templates.zip) "In my opinion, the product is
generally" and connect it with some positive attributes from the scale like enjoyable, pleasant, friendly.
* The last three attributes, which are mainly related to voice assistance, are not considered because the proposed
product categories are not equipped with this characteristic.
The list of scales and their corresponding questionnaire sources are presented below.
| Items | In [UEQ+](https://ueqplus.ueq-research.org/Material/UEQ+_Handbook_V2.pdf) | In the [Paper](https://dl.gi.de/bitstream/handle/20.500.12116/5770/2017_UP_002.pdf?sequence=2&isAllowed=y) | used in questionnaire | Erklärender Satz |
|---------------------------------------------------|:-------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------------------:|-----------------------|-----------------------------------------------------------------------------------------------------------------|
......@@ -132,10 +83,3 @@ some decisions are made to extend the questionnaire to v2:
| Verständnis (Comprehensibility) | ✔ | ❌ | ❌ | -- |
| Vertrauen (Trust) | ✔ | ✔ | ✔ | |
| Wertigkeit (Value) | ✔ | ✔ | ✔ | |
## Notes
* Interview Nr. 92 should be removed
* Replicability is not completely possible:
* Structure of the questionnaire is evident in terms of content, but the structuring is not comprehensible
* maybe too small sample size?
\ No newline at end of file
......@@ -8,12 +8,10 @@ from matplotlib import ticker
from scipy.stats import pearsonr
from enum import Enum
from sklearn.manifold import MDS
import scipy
from kPOD import k_pod
import time
ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
class Categories(Enum):
Dashboard = "Dashboard"
LMS = "Learning Plattform"
......@@ -30,7 +28,8 @@ class Interviews:
def __init__(self) -> None:
super().__init__()
self.df = pd.read_csv('../data/data_ueq.csv', delimiter=';', quotechar='"', encoding='utf-16')
self.df = pd.read_csv(os.path.join(ROOT_DIR, '..', 'data', 'data_ueq.csv'), delimiter=';', quotechar='"', encoding='utf-16')
def __str__(self) -> str:
return f'{self.df.shape[0]} interviews'
......@@ -43,8 +42,6 @@ class Interviews:
cat_games = all_categories.get_items(Categories.Games)
cat_lms = all_categories.get_items(Categories.LMS)
# remove interviews which are not complete
print(cat_games)
print(self.df)
unwanted_interviews = self.df[self.df[cat_games[0]].isnull() | self.df[cat_lms[0]].isnull()]
print(f'{unwanted_interviews.shape[0]} interviews have not completed the questionnaire.')
df_nonnull_interviews = self.df.drop(unwanted_interviews.index)
......@@ -137,7 +134,7 @@ class Variables:
def __init__(self) -> None:
super().__init__()
self.df = pd.read_csv('../data/variables_ueq.csv', delimiter=';', quotechar='"', encoding='utf-16')
self.df = pd.read_csv(os.path.join(ROOT_DIR, '..', 'data', 'variables_ueq.csv'), delimiter=';', quotechar='"', encoding='utf-16')
def get_groups(self):
return self.df[['VAR', 'LABEL']][7:12]
......@@ -150,7 +147,7 @@ class Values:
def __init__(self) -> None:
super().__init__()
self.df = pd.read_csv('../data/values_ueq.csv', delimiter=';', quotechar='"', encoding='utf-16')
self.df = pd.read_csv(os.path.join(ROOT_DIR, '..', 'data', 'variables_ueq.csv'), delimiter=';', quotechar='"', encoding='utf-16')
class ProductCategories:
......@@ -186,10 +183,12 @@ class ProductCategories:
def save_plot(plot, path):
dir_path = os.path.abspath('../results/')
dir_path = os.path.abspath(os.path.join(ROOT_DIR, '..', 'results'))
if not os.path.exists(dir_path):
os.mkdir(dir_path)
plot.savefig(fname=os.path.join(dir_path, path + '.png'), dpi='figure', format='png')
saved_path = os.path.join(dir_path, path + '.png')
print("saved plot to " + saved_path)
plot.savefig(fname=saved_path, dpi='figure', format='png')
def replace_cryptic_names(columns):
......@@ -304,6 +303,7 @@ class Plotter:
super().__init__()
self.df = df
self.pc = pc
self.plotted = 0
def filter_by_gender(self, gender):
if gender == 'male':
......@@ -313,15 +313,11 @@ class Plotter:
else:
return self.df
def plot_groups(self, variable_labels):
df_groups = self.df[['DE01_01', 'DE01_02', 'DE01_03', 'DE01_04', 'DE01_05']] - 1
print(variable_labels)
df_groups.columns = ['Stu', 'Teach', 'Res', 'Tech', 'Out']
sns.set_theme(style="whitegrid")
df_groups_sum = pd.DataFrame(df_groups.sum(), columns=['Sum']).sort_values(by='Sum', ascending=False)
print(df_groups_sum)
ax = sns.barplot(data=df_groups_sum, palette="Blues_d", errorbar=None, y="Sum", x=df_groups_sum.index)
ax.set(title='Academic Community (Multiple Choice)')
ax.yaxis.set_major_locator(ticker.MultipleLocator(5))
......@@ -330,8 +326,9 @@ class Plotter:
for tick in ax.yaxis.get_major_ticks():
tick.label.set_fontsize(15)
sns.despine(left=True)
self.plotted = self.plotted + 1
save_plot(plt, 'demographics_communities')
plt.show()
return plt
def plot_gender(self):
df_gender = self.df['DE02'] \
......@@ -340,19 +337,18 @@ class Plotter:
.replace(3, 'diverse') \
.replace(-1, 'not mentioned')
# print(df_gender.sum())
print(pd.DataFrame(df_gender.value_counts(normalize=False)))
df_gender_count = pd.DataFrame(df_gender.value_counts(normalize=False))
print(df_gender_count)
print(df_gender_count.index.values)
sns.set_theme(style="whitegrid")
ax = sns.barplot(data=df_gender_count, palette="Blues_d", errorbar=None, y="count", x=df_gender_count.index.values)
ax = sns.barplot(data=df_gender_count, palette="Blues_d", errorbar=None, y="count",
x=df_gender_count.index.values)
ax.set(title='Gender')
for tick in ax.xaxis.get_major_ticks():
tick.label.set_fontsize(15)
for tick in ax.yaxis.get_major_ticks():
tick.label.set_fontsize(15)
self.plotted = self.plotted + 1
save_plot(plt, 'demographics_gender')
plt.show()
return plt
def plot_faculty(self):
df_faculty = self.df['DE03'] \
......@@ -369,9 +365,9 @@ class Plotter:
.replace(-9, 'nB')
ax = sns.countplot(x=df_faculty, palette="Blues_d")
ax.set(title='Fakultät')
print(df_faculty.value_counts(normalize=True))
self.plotted = self.plotted + 1
save_plot(plt, 'demographics_disciplines')
plt.show()
return plt
def plot_disciplines(self):
df_faculty = self.df['DE04'] \
......@@ -381,12 +377,11 @@ class Plotter:
.replace(4, 'Ingenieurwissenschaften') \
.replace(-1, 'kA') \
.replace(-9, 'nB')
print(df_faculty)
ax = sns.countplot(x=df_faculty, palette="pastel")
ax.set(title='Disciplines')
print(df_faculty.value_counts(normalize=False))
self.plotted = self.plotted + 1
save_plot(plt, 'demographics_disciplines')
plt.show()
return plt
def plot_item(self, category_title, area, max_count=20, gender='all'):
category_items = self.pc.get_items(category_title)
......@@ -406,16 +401,15 @@ class Plotter:
df_category_area = filter_columns_by_value(df_category, df_category.mean() >= 0)
else:
df_category_area = df_category
print('------------- ' + category_title.value + "-" + area + "-" + gender)
df_category_area.columns = replace_cryptic_names(df_category_area.columns)
df_category_area = df_category_area.reindex(df_category_area.mean().sort_values(ascending=False).index, axis=1)
df_category_area = df_category_area.iloc[:, : max_count]
print(df_category_area.columns)
ax = sns.barplot(data=df_category_area, palette="Blues_d", orient='h', errorbar="sd")
ax.set(title=category_title.value + ' - ' + area)
sns.despine(left=True)
self.plotted = self.plotted + 1
save_plot(plt, 'items_' + category_title.value + '_' + area + '_' + gender)
plt.show()
return plt
def plot_boxes(self, category_title, max_count=5):
df_category_area = self.df[self.pc.get_items(category_title)]
......@@ -447,11 +441,11 @@ class Plotter:
labels=translate_scales(df_category_area.columns, True), rotation=0)
ax.xaxis.set_label_position('top')
ax.xaxis.tick_top()
self.plotted = self.plotted + 1
save_plot(plt, 'box_' + category_title.value)
plt.show()
return plt
def m_scatter( self, x, y, ax=None, m=None, **kw):
def m_scatter(self, x, y, ax=None, m=None, **kw):
import matplotlib.markers as mmarkers
if not ax: ax = plt.gca()
sc = ax.scatter(x, y, **kw)
......@@ -579,25 +573,31 @@ def setup_dataframes():
if __name__ == '__main__':
print('---------- Creating plots --------------')
interview, variables, values, product_categories, rankings = setup_dataframes()
print(interview)
df_filtered = interview.get_filtered(product_categories)
plotter = Plotter(df_filtered, product_categories)
statistics = Statistics(df_filtered, product_categories)
plotter = Plotter(df_filtered, product_categories)
print('average time to complete the questionnaire: ' + time.strftime('%M:%S', time.gmtime(statistics.mean_time())))
plotter.plot_groups(variables.get_groups())
plotter.plot_gender()
# plotter.plot_disciplines()
plotter.plot_disciplines()
plotter.plot_faculty()
print(plotter.filter_by_gender('male'))
# for example in Categories:
# plotter.plot_item(example, 'top')
# plotter.plot_item(example, 'top', gender='male')
# plotter.plot_item(example, 'top', gender='female')
# plotter.plot_item(example, 'mid')
# plotter.plot_item(example, 'low')
# plotter.plot_boxes(example, 6)
# print(statistics.calc_corr())
# print(statistics.similarity_by_ranked_scales(rankings.get_all_rankings()))
print(time.strftime('%M:%S', time.gmtime(statistics.mean_time())))
# print(statistics.kpod_clustering(rankings.get_all_rankings()))
for example in Categories:
plotter.plot_item(example, 'top')
plotter.plot_item(example, 'top', gender='male')
plotter.plot_item(example, 'top', gender='female')
plotter.plot_item(example, 'mid')
plotter.plot_item(example, 'low')
plotter.plot_boxes(example, 6)
print(statistics.calc_corr())
print(statistics.similarity_by_ranked_scales(rankings.get_all_rankings()))
print(statistics.kpod_clustering(rankings.get_all_rankings()))
print('-------- finished. Plotted: ' + str(plotter.plotted) + ' charts. -----------------')
Source diff could not be displayed: it is too large. Options to address this: view the blob.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment