Virtual reality for prospective and process-related competence modeling—piloting a participatory approach and investigating user acceptance of the applied VR-tool

Competence modeling aims at identifying and describing competences that are required for effective job performance (cf., Albertsen et al. 2010). Competence models summarize these performance-relevant competences (e.g., KSAOs) and thus are linked to the organizational strategy (e.g., Campion et al. 2011). The listed competences are usually subdivided into relating competence sub-dimensions (Schaper 2009). Within these sub-dimensions, different competence levels are distinguished. The competence levels can be used to map employees’ individual competence characteristics at different stages (Schaper 2009). This way, it is possible to distinguish between high and average performers (Campion et al. 2011). Generally, there exist three traditional approaches of competence modeling. The one-size-fits-all-approach aims at developing one competence model that is relevant for the entire organization or a broad range of jobs (e.g., all managerial jobs). In contrast, the single-job-approach focuses on developing specific competence models for each individual job. The VR-based PCM approach presented in this article bases on the multiple-job-approach (Mansfield 1996). The multiple-job-approach is a mixture of the above-mentioned one-size-fits-all-approach and the single-job-approach. Thus, the resulting competence models comprise both overarching as well as job-specific competence requirements (Mansfield 1996). In terms of data collection, the one-size-fits-all-approach uses competence models that are already available in the organization or can be obtained from scientific literature (e.g., Mansfield 1996). In contrast, the single-job-approach derives competence requirements directly from expert interviews being conducted with job holders and/or their managers. Again, the multiple-job-approach uses a kind of mixture of both methods. In particular, competence requirements are derived using a common set of predefined competences as well as methods of job analysis in order to differentiate these predefined competences (e.g., Mansfield 1996; Sonntag 2007). In general, job analysis encompasses activities for systematically analyzing and describing work activities, worker attributes, and work contexts (Sackett and Laczo 2003).

Supplementing these traditional forms of competence modeling, prospective and process-related approaches for competence modeling become increasingly important. Specifically with regard to the context of digitalization, PCM approaches offer an efficient way to identify future competence requirements more precisely (cf., Hegmanns et al. 2019). The approaches presented in Robinson et al. (2007) and Yang et al. (2006) can be seen as precursors of PCM. Robinson et al. (2007) developed an approach for prospectively identifying future competence requirements. Here, employees’ quantitative perceptions of the most required future competences are identified using a predefined competence questionnaire. The identified competence requirements are then differentiated in the context of critical incident interviews (cf., Flanagan 1954). Yang et al. (2006) developed a procedure for process-oriented competence analysis. Here, business processes are decomposed into their single process steps and relevant competence requirements are mapped to these process steps afterwards. Ranking the mapped competences by frequency enables to identify core competences in a process-oriented way. The VR-based PCM approach piloted in this article combines the prospective and process-related analysis focus. In particular, it draws on the PCM approach presented in Depenbusch et al. (2021). In this preceding approach, future competence requirements are identified by applying the multiple-job-approach in a prospective and process-related manner. In the course of expert interviews with job holders and supervisors, work activities as well as critical incidents (Flanagan 1954) occurring in the future digitalized business processes are first explained by the employees. Second, respective competence requirements are identified using a printed competence inventory. In order to maintain a process-related focus of analysis, a printed BPMN-process graph is presented to the interviewees. Based on the data gathered in these expert interviews, prospective and process-related competence models are derived. The pilot runs in Depenbusch et al. (2021) show that the prospective and process-related derivation of future competence requirements demands a deep process understanding on behalf of the employees. This is also the case when the identified competence requirements should be mapped to the single process steps in order to create prospective and process-related competence models. Thus, the prospective and process-related competence models have been created by work- and occupational psychologists and checked afterwards by the interviewees. Specifically, when the focus is placed on rather hypothetic or future business processes, employees need a profound capacity for abstraction. This can be particularly demanding when only schematic, paper-based process graphs are presented (cf., Oberhauser et al. 2018). As a response to this challenge, this article pilots an approach that utilizes VR as a facilitator for identifying and documenting future competence requirements with regard to single process steps. In order to outline whether VR is actually advantageous for this purpose, the competence models resulting from the VR-based approach are compared with those obtained in Depenbusch et al. (2021).

Employee participation comprises activities that give employees the possibility to participate and assume responsibility (Dombrowski et al. 2018). In this study, we associate participation with the empowerment of employees to become active competence modelers. Since business process understanding is a central precondition, we draw on previous studies that already investigated how a VR-environment should be designed to promote employees’ business process understanding. These are in particular studies on participatory and collaborative business process modeling and management (e.g., Oberhauser and Pogolski 2019). Here, VR serves as a tool to empower employees (and even those who are not familiar with interpreting abstract and complex process graphs) to intuitively design and evaluate business- or work processes (e.g., Zenner et al. 2020). In this context, it has already been revealed that the translation of conventional business process modeling notations to a VR-representation (e.g., 3D-BPMN) offers special advantages to support employees’ process understanding (e.g., Oberhauser and Pogolski 2019; Zenner et al. 2020). These advantages result from specific attributes of VR, such as the ability to visualize complex business process structures in a more holistic and illustrative manner (Oberhauser et al. 2018; Oberhauser and Pogolski 2019), or to facilitate access and storage of process-related information (e.g., Betz et al. 2008).

Drawing on these insights, further advantages of VR for employee participation can be derived in the course of user acceptance evaluation. In the context of IT, user acceptance is described as the willingness of individuals to employ information technology for the task it is designed for (Dillon and Morris 1996; Wahdain and Ahmad 2014). The model mostly employed for predicting user acceptance is the technology-acceptance model (TAM, Davis et al.1989). The TAM gives general feedback on the ease of use and usefulness of a system. However, the TAM does not provide specific information about important attributes of the system itself. Therefore, Wixom and Todd (2005) developed the User Satisfaction—Technology Acceptance Model (US-TAM) connecting the TAM with user satisfaction (US) literature. US literature identifies various system- and information quality attributes and thus allows evaluating the technical artefact itself in terms of supporting user acceptance (Wixom and Todd 2005). Since this study aims at exploring VR-attributes that mainly drive user acceptance and thus potentially provide advantages for participatory PCM, we specifically refer to literature grounding on the US-TAM. In particular, we apply the “catalog of criteria for forecasting user acceptance” by Brunkow and Hub (2018) as primary theoretical foundation in order to select the acceptance criteria to be considered in this study. The information quality attributes listed in this catalog comprise, e.g., the customizability of virtual texts and images or the relevance of the presented information for the respective context of use (cf., Brunkow and Hub 2018). The system quality attributes refer to, e.g., the level of immersion or the experienced interactivity or the system usability (cf., Brunkow and Hub 2018).

Usability describes the extent to which a system can be applied in a certain context of use to achieve definite goals in an effective, efficient and satisfactory manner (Kähler et al. 2019). In this study, usability of the applied VR-tool is evaluated using a questionnaire that grounds on the universally standardized norm ISO 9241. The particular focus is placed on the performance measures defined in ISO 9241-11 as well as on the dialog principles for usable human-machine-interaction (HMI) presented in ISO 9241-110. The performance measures are directed to the system’s effectiveness, efficiency, and user satisfaction (cf., Kähler et al. 2019). The dialog principles comprise, e.g., the system’s task appropriateness, self-descriptiveness or fault tolerance (Kähler et al. 2019). As the characteristic attributes of VR (e.g., immersion, interaction) imply a special form of human-machine-interaction (HMI) (e.g., Martens 2016), we formulated the criteria presented in ISO 9241-11 and ISO 9241-110 with regard to our investigation context. For example, the criteria task appropriateness (describing that the functions of the system are based on the characteristic properties of the task in question) (e.g., Kähler et al. 2019) is formulated in the usability questionnaire as follows: “The functionalities of the VR-tool are appropriate for answering the interview questions.”

Although being formulated with regard to our investigation context, the usability criteria applied in our study still maintain a high degree of standardization. This way, we want to supplement the VR-specific acceptance criteria (cf., Brunkow and Hub 2018) with criteria that are more comprehensible for individuals without VR experiences. This target group may be more familiar with common usability criteria, such as those presented in ISO 9241. Due to maintaining a high degree of standardization, the usability results can additionally be used for comparison with usability findings obtained for non-VR methods of competence modeling.

In order to derive job-specific competence facets, relevant interview text passages were allocated into a theory-based, deductive category system (Table 5). The categories represent the competence (sub-) dimensions listed in the competence inventory (see Chap. 3.2, Table 2). Thus, the upper level of the category system comprises the dimensions of professional, methodological, social, personal and digital competence. The second level summarizes the relating sub-dimensions obtained from the CRI (Kauffeld 2021), the competence atlas (Heyse and Erpenbeck 2009; Heyse 2010), and the Dig Comp 2.1 (Carretero et al. 2017). After structuring the interview material using the deductive competence categories, the inductive approach was applied to further differentiate the competences with job-specific content. In order to guarantee that the interview text passages could be classified clearly into the category system, the interrater-reliability has been assessed by calculating Cohen’s kappa coefficient (Cohen 1960). Cohen’s kappa indicates the extent to which two raters independently classify the same objects (it our case, these are the interview passages) into the same categories (in our case, these are the different competence (sub-) dimensions) (e.g., Brennan and Prediger 1981). For calculation, a random sample comprising 25% of all mapped interview text passages (N = 149 analysis units) has been used. Results indicate a kappa value of 0.882 for the first level of the category system and a kappa value of 0.856 for its second level.

In order to derive and evaluate the VR-specific acceptance criteria, relevant interview text passages were first allocated into a theory-based, deductive category system (Table 6). The main categories (e.g., virtual visualization, intuitiveness) were formulated on the basis of the interview questions of the third interview module and supplemented with further theory-based categories from literature (e.g., immersion, interaction) (e.g., Brunkow and Hub 2018). After structuring the interview material using the deductive competence categories, the inductive approach was applied to specify the categories with further sub-categories (e.g., the main category “interaction” has been inductively differentiated with the sub-categories “documentation and selection of competence requirements from the competence inventory” as well as “orientation and locomotion in the virtual work environment”). For calculating Cohen’s kappa, a random sample of 25% of all mapped interview text passages was used (N = 78 analysis units). Results indicate a kappa value of 0.772 for the first level of the category system and a kappa value of 0.826 for its second level. The usability questionnaires (filled out by the employees as part of the third interview module) were analyzed descriptively using the software SPSS 28. For each usability criteria that is listed in the questionnaire, its mean value and standard deviation were calculated.

Tables 8, 9 and 10 show excerpts from the job-specific competence models for the welders in SME 1, the warheouse employees in SME 2, and the employees of quality control in SME 3. The work activities of these jobs/groups of employees will change the most in the course of digitalization. For each required future competence dimension (professional, methodological, social, personal and digital competence), the corresponding sub-dimension and the relating job-specific competence facet are exemplarily listed. Considering, e.g., the professional competence dimension, manual skills will still be required in each of the above-mentioned jobs. In SME 1, the welders need to be able to manually re-weld the first samples when these do not meet the required product standards. In SME 2, the warehouse employees should be able to correctly operate the hand scanner in order to digitalize product information and to transfer them to the inventory management system. In SME 3, the employees of quality control need to be able to control product characteristics and to conduct visual inspections. Considering, e.g., the social competence dimension, the skill of positioning one’s own point of view will be required in all three SMEs. In SME 1, the welders are required to position their own point of view when new customer orders are discussed with the project management. In SME 2, the warehouse employees should be able to adequately communicate with customers when short-term delivery requests cannot be met. In SME 3, the employees of quality control are required to inform the machine operators when the products do not meet the quality standards. Considering the digital competence dimension, the welders in SME 1 should be able to solve problems using digital technology. In particular, this relates to the ability of identifying causes of machine faults by looking up error histories in a digital tablet. In SME 2, information and data literacy will be required of the warehouse employees in terms of filtering, researching, and interpreting digital data for shipment planning. In SME 3, the employees of quality control should be able to create digital content, such as entering measured product values into a digital tablet.

The job-specific competence facets described in Chap. 4.1.1 were integrated into the preliminary versions of the prospective and process-related competence models (created by the employees during the VR-based expert interviews) in order to differentiate the competence (sub-) dimensions listed therein. This way, final versions of prospective and process-related competence models have been created for the jobs/groups of employees that will be part of the SMEs’ future business processes. The prospective and process-related competence models thus present which competences will be required at the individual future process steps of these jobs/groups of employees. Tables 11, 12 and 13 show excerpts from the final prospective and process-related competence models for the welders in SME 1, the warehouse employees in SME 2, and the employees of quality control in SME 3. For the process steps listed in these competence models, the relating future required competence (sub‑) dimensions, the corresponding job-specific competence facets, as well as the required competence levels are presented. Additionally, the actual competence levels of three randomly selected employees for each SME are included. The employees’ actual competence levels have been determined by their direct supervisor. In general, the prospective and process-related competence models can be used to derive future development needs of individual employees or to identify future competence requirements for particular jobs/groups of employees.

Considering, e.g., the digital competence dimension, future development needs will arise in all three SMEs. Considering SME 1, this will be the case for the competence sub-dimension “problem-solving using digital technologies”. This competence sub-dimension is required at process step 1 “checking the welding device”. While there is required a competence level of 4 (reliable mastery in totally new work situations), employee 2 only has a competence level of 2 (reliable mastery in standard work situations). For employees 1 and 3, the competence gap is even larger as they only have a competence level of 1 (basic mastery). In consequence, all three employees should be trained in filtering and interpreting error histories stored in a digital tablet in order to quickly identify faults in the welding device. In view of the competence sub-dimension “communication using digital technologies”, which is required at process step 2 “reporting faults of the welding device to the project manager”, no future development needs arise. All three employees meet the required competence level or are even overqualified (Table 11).

With regard to SME 2, future development needs will occur for all three employees at process step 1 “reporting information about the glass products to the inventory management system”. This applies the competence sub-dimension “digital content creation”. Thus, a respective training should enable the employees to scan product labels using a hand scanner and to manually type product information into the inventory management system. Considering process step 2 “shipment planning using the inventory management system”, there will also arise development needs for all three employees. This specifically concerns the competence sub-dimension “information and data literacy”. Thus, employees should be trained in searching, filtering and interpreting digital data which are needed for shipment planning (e.g., planned incoming and outgoing goods) (Table 12).

In SME 3, future development needs will occur for all three employees concerning the competence sub-dimension “digital content creation”. This competence sub-dimension is required at process step 1 “controlling and documenting product characteristics”. Against this background, a training should be planned that enables the employees of quality control to enter the product characteristics they have measured (e.g., length, weight) into a digital tablet (Table 13).

The prospective and process-related competence models can also be used to identify future competence requirements of particular jobs/groups of employees. This study focuses on the group of employees whose jobs will be part of the SMEs’ future digitalized business processes. Again, in SME 1, this is the case for the project management, the welders, the employees of quality control, and the employees in logistics (N = 4 jobs/groups of employees). In SME 2, this concerns the purchasers, the sales employees, the warehouse employees, and the employees in shipping management (N = 4 jobs/groups of employees). In SME 3, this is the case for the employees of quality control, the machine operators and the technicians (N = 3 jobs/groups of employees). The future competence requirements of these groups of employees can be determined by calculating competence frequency distributions. In this context, the prospective and process-related competence models are used to count how often the various competence (sub-) dimensions are required for the considered jobs/groups of employees. The underlying assumption is that the competence (sub-) dimensions that are particularly frequently required will become the most important and thus need to be trained (cf., Yang et al. 2006). This study presents the frequency distributions of the different competence sub-dimensions at SME-level. In this context, it was first counted for each above-mentioned job per SME, how often the different competence sub-dimensions are listed in their prospective and process-related competence models. Afterwards, the resulting sums of the competence sub-dimensions were divided by the number of process steps these jobs/groups of employees will conduct in the SME’s future business process. The resulting mean values per job/group of employees were then summed up and divided by the total number of the jobs/groups of employees considered in the respective SME. The resulting competence frequency distributions for each SME are displayed in Fig. 6.

In view of the professional competence dimension, the sub-dimensions “application of expertise” and “application of experience” will become particularly important in all three SMEs. The relevance of these sub-dimensions becomes visible by the high frequency with which they will be required in the SMEs’ future business processes. For example, “application of expertise” will be required in approx. 70% of all considered future process steps in SME 1, in approx. 97% of all considered future process steps in SME 2, and in approx. 90% of all considered future process steps in SME 3. The competence sub-dimension “application of experience” will be required in approx. 67% of all considered future process steps in SME 1, in approx. 91% of all considered future process steps in SME 2, and in approx. 80% of all considered future process steps in SME 3. Against this background, it becomes clear that the newly digitalized working steps still afford high expertise and experience that should be prospectively trained. Focusing on the methodological competence dimension, the sub-dimension “planning capability” will become specifically important in SME 1 and SME 2. In consequence, employees should be trained to conduct adequate time and task management. In SME 1, the competence sub-dimension “problem-solving” will additionally become relevant. Thus, the error histories stored in the digital tablets seem not to take over every kind of problem analysis. In consequence, the employees still need to be able to detect faults using their technical knowledge and thus should be trained accordingly. In SME 2 and 3, the competence sub-dimensions “holistic thinking” as well as “information dissemination and -processing” will become additionally relevant. In consequence, employees should be trained to adapt a holistic view on the work in the digitalized business process instead of narrowly focusing on their own tasks. Considering the social competence dimension, the future focus seems to be placed on “teamwork and cooperation” in SME 1 and on “flexibility” in SME 2. In SME 3, “teamwork and cooperation” and “flexibility” as well as “communication” will become important. Thus, the digital technologies that will be implemented in the SMEs (digital tables, inventory management system) won’t undermine social interaction, but rather seem to encourage it. With regard to the personal competence dimension, the sub-dimension “assuming responsibility” will become quite important in all three SMEs and thus still needs to be trained. In view of the digital competence dimension, the sub-dimension “information and digital data literacy” will become important in all three SMEs. SME 1 places a further focus on the sub-dimension “communication and collaboration using digital technologies”. In SME 2, this is the case for “digital content creation” and in SME 3 for “problem-solving using digital technologies”. In consequence, the various sub-dimensions of digital competence should be continuously trained to prepare employees adequately for their future digital work.

When comparing the level of detail of the prospective and process-related competence models developed in the VR-condition with those created in the non-VR-condition, it becomes obvious whether there are differences regarding the future competence requirements that have been derived inside in VR and outside of VR. For comparison, we contrasted the competence frequency distributions obtained from the prospective and process-related competence models derived in the VR-condition with the competence frequency distributions resulting from the prospective and process-related competence models developed in the non-VR-condition. The “VR-based competence frequency distributions” have already been calculated and presented in Chap. 4.1.4. The “non-VR-based competence frequency distributions” have been calculated in the same way. The jobs/groups of employees considered in Depenbusch et al. (2021) were the same as those considered in our study. Figures 7, 8 and 9 compare the VR-based and the non-VR-based frequency distributions of each SME. In general, it becomes obvious that more competence requirements have been identified and mapped to the single process steps in the VR-condition than it was the case in the non-VR-condition. This becomes visible by the fact that the VR-based frequency distributions of the three SMEs lie mostly above the non-VR-based frequency distributions. Considering, e.g., the professional competence dimension, the biggest differences between the VR-based and non-VR-based frequency distributions can be identified concerning the competence sub-dimension “application of manual skills”. This is the case for all three SMEs. In the VR-condition, “application of manual skills” has been identified to become relevant in approx. 62% of all considered future process steps in SME 1, in approx. 57% of all considered future process steps in SME 2, and in approx. 80% of all considered future process steps in SME 3. In the non-VR-condition, the percentage values are much smaller. Here, “application of manual skills” has been identified to become relevant in approx. 29% of all considered future process steps in SME 1, in approx. 14% of all considered future process steps in SME 2, and in approx. 30% of all considered future process steps in SME 3. These differences can possibly be explained by the fact that VR functioned as a facilitator for identifying and documenting future competence requirements with regard to single process steps. This way, employees were enabled to continuously maintain a prospective and process-related focus of analysis. In consequence, future required competences have been identified more comprehensively and concretely in VR. Slight differences in the competence frequency distributions (VR vs. non-VR) can also be found with regard to the social and digital competence dimensions. Considering the social competence dimension, in SME 2 and 3 this specifically concerns the sub-dimensions “teamwork and cooperation” as well as “communication”. In SME 1, this is the case for the sub-dimension “positioning of one’s own point of view”. Thus, VR seemed to support employees in understanding the interconnectedness and complexity of the future business processes. This way, it was easier for them to recognize future situations in which collaboration will be required. Focusing on the digital competence dimension, large frequency differences could be identified in SME 1 concerning all related sub-dimensions. In SME 2 and SME 3, this is specifically the case for “communication and collaboration using VR” as well as “problem-solving using VR”. Thus, VR additionally supported employees in recognizing and understanding the future use of the new technology (Fig. 9).

As already mentioned, we conducted a user acceptance evaluation in order to identify concrete VR-attributes that are specifically advantageous for participatory PCM. For this purpose, both standardized usability criteria as well as VR-specific acceptance criteria have been analyzed. In the following, the usability results are first presented. In this case, a quantitative evaluation of the usability questionnaires has been conducted. Afterwards, the results of the VR-specific acceptance criteria are presented. These have been derived and evaluated qualitatively.

The employees gave highest average approval concerning their satisfaction with the structure and functionalities of the VR-tool (mean = 4.1). This is reflected in the almost equally pronounced dialog principles of task appropriateness, self-descriptiveness and fault tolerance (each with a mean of 4.0). Satisfactory values of approval are also achieved with regard to the perceived effectiveness (mean = 4.0; mean = 3.7) and efficiency of the VR-tool (mean = 3.7). Lowest approval is given to the conformity of the VR-tool with user expectations (mean = 3.3) and their intention to use the VR-tool on a regular basis (mean = 3.4). Standard deviations vary between 0.7 and 1.4, indicating that the employees’ responses deviate most with regard to their perceived satisfaction (standard deviation = 1.4) and less concerning their perceived task appropriateness of the VR-tool (standard deviation = 0.7).

During the VR-based expert interviews, it soon became evident that the virtual visualization of the future business processes promotes employees’ business process understanding. Specifically, the in-situ process visualization was described as helpful for adequately explaining future work activities and deriving relating competence requirements: “I actually think it is better that the work activities in VR are located directly at the workplace so that you can think about the process more easily” (sales employee, SME 2). It could also be observed that the employees were highly immersed in their future work process and it felt to them as if they were actually there: “You are in your totally own world. That’s incredible” (employee of quality control, SME 3). This way, high levels of motivation and enthusiasm have been achieved: “It is just more interesting, keeps you awake and encourages you to be active” (employee of quality control, SME 3).

Also, the functionalities for interaction provided by the VR-tool were truly supportive for our PCM approach. On the one hand, these relate to the teleport function, which has been used in the expert interviews to give in-situ descriptions of future work activities. On the other hand, these comprise the functionalities offered to select and document competence requirements. In our case, the competence inventory was anchored at every virtual process node using a tree-like interface logic. Employees underlined, that this made it easier for them to punctually concentrate on a specific competence dimension: “I have a filter and can concentrate on what I am working on […]” (business manager, SME 2). However, in terms of the quality of information portrayed, it was partly figured out that the scope and complexity of the competence inventory should be reduced: “Sometimes, it was difficult to interpret the level of detail in the competence inventory with regard to process steps that are rather banal in practice” (business manager, SME 2). In addition, employees stated that entering explanations to clearly define the identified competence requirements was truly time-consuming when applying the “VR-keyboard”. Thus, they suggested adding audio-memo-boxes at each process step. Another attribute that was identified as supportive to conduct PCM refers to the high ergonomics and comfort of the VR-tool. It was indicated that the HMD provided an adequate wearing comfort. In addition, it could be observed that the VR-controller enabled to comfortably teleport across the virtual environment and to select competences from the competence inventory.

Further attributes that were identified as beneficial for our PCM approach refer to the high intuitiveness and learnability as well as customizability of the VR-tool. The majority of employees learned quickly to select competence dimensions from the competence inventory and to teleport across the virtual environment: “[…] when teleporting, your thumb has to be on the touchpad the whole time […]. But you notice that after a relatively short time” (head of quality management, SME 3). In consequence, the majority of employees quickly reduced their initial skepticism about using VR. Specifically, when experiencing that an accidental mistake (e.g., selecting the wrong competence requirement from the competence inventory) could be corrected with little effort, employees soon built up trust.