User Profiling based Adaptive Test Generation Assessment in E – Learning System
Manan Jadhav
Department of Computer Technology, VJTI Mumbai, India manan.jadhav@gmail.com

Shagufta Rizwan
Department of Master of Computer Application, VJTI Mumbai, India shaguftarjpt673@gmail.com

Aditi Nehete
Department of Computer Technology, VJTI Mumbai, India aditinehete@gmail.com

Abstract — In this paper, we describe a User profiling based ELearning System with Adaptive Test Generation and Assessment. This system uses rule-based Intelligence Technique and implicit User Profiling to judge the proficiency level of the student and generates tests for them accordingly. More specifically it’s a Test Generation, Assessment and Remedial System where the student can give a test after he has completed studying a particular concept where the difficulty level of the test will be decided by the expert system engine. After the completion of the test, the system helps the student in improving the proficiency of the concept either if it is expected by the concept or if he faces difficulty in understanding the concept. Based on the type of errors made in the test, the Test Remedial System will help the student to improve in that domain of understanding of the concept. Every phase transition is rule-based which considers the user’s profile and the concept importance to make sure that he does well where it is required. Preliminary Experimental Implementations show that with User Profiling we can reduce the amount of efforts required by the user to clear a concept. Moreover, with test remedial we assure that the user actually covers all erroneous domains under a particular concept depending on its importance. Keywords— Adaptive Assessment, E – Learning, Intelligent Tutoring System, Personalized Learning, Test Evaluation, Test Remedial, User Profiling

student‟s performance without considering where the student is actually lagging, which is resolved in our system using TypeClasses as described later in Section 2. On the other hand, the personalized E-Learning System described in [3] adapts to the interests and levels of learners by combining multiple feedback measures to infer user preferences and levels of expertise to deliver personalized information using the collaborative filtering algorithm. But it has no means of evaluating the performance of the user. It can only suggest pages to the user depending on his/her study patterns but no means to test the user performance. Moreover, system in [1] focuses on improving the student‟s performance without considering where the student is actually lagging, which is solved in our system using Type-Classes as described later in Section 2. II. SYSTEM ARCHITECTURE

I.

INTRODUCTION

In the current era of technology and advancement, Elearning has proved to be a revolutionary way of educating the students covering the areas left untouched by the traditional method of “Chalk-and-Talk”. Following the trends, we have designed this system in such a way that it acts as a virtual tutor that works on the weaknesses of the student based on their individual levels. Though all the learners might have similar interests but their levels of expertise may differ. Thus, this system keeps in mind the needs of the individual learner and provides them with the appropriate learning material. The AITS (Artificial Intelligence Teaching System) described in [1] assists learning by adapting the course material to student‟s need, based on his/her knowledge level which is evaluated after each concept via an expert system, that takes into account the difficulty level of questions/exercises. But it does not implement user profiling to work on the weaknesses of the student. In other words, it focuses on improving the

The system consists of 5 modules each of which specifically contributes in personalizing the entire learning process and making it more adaptive. The system, thus, has Learning Resources, Test Resources as well as Student Profiles. These three components altogether form a backend to the system. The Learning and Test Resources contain contents of various subjects to be studied and comprehended by the learner as well as the tests on these subjects. It follows a hierarchy: Subject - > Section - > Chapter - > Topic - > Concept- > Type-class - > Test. The segregation of subjects into concepts helps to cover and test each and every concept in great detail. Further, for taking tests, the system has- General Class and Error Class, each having different sets of test questions. The questions from General Class are posed when the learner finishes understanding the concept, whereas the ones from Error Class are posed after the previous tests are evaluated and these questions focus on the areas in which the learner did not perform well. As the system stresses on the term „personalized‟, it first takes into account the choice of the learner. He may want to study specific topic from various subjects, or if he is already well versed with the topic and needs an assessment of his

knowledge in a particular topic, then he may directly proceed to take an assessment test.

option point to and thus find out what kind of mistake the learner has made while answering the question. As explained later, the Test Assessment System (TAS) takes into consideration certain threshold conditions and decides whether the performance of the learner is satisfactory or not. If it is satisfactory, then the system sets another test of a higher Understanding Level for the learner. If the learner performs well in the subsequent higher level tests, then the system declares that the learner has a good hold on that particular topic and asks him to move on to other topics. If the learner fails, the control gets transferred to the Test Remedial System (TRS), wherein the incorrect answers are mapped to error classes and the learner is asked to answer another test which focuses only on the erroneous areas. Even these tests follow a hierarchy which is explained in depth in further sections. III. KNOWLEDGE BASE

FIGURE 1: SYSTEM ARCHITECTURE

In order to cater to the need of the former requirement, the learner is directed to Concept Learning Module which connects to the Learning Resource database. Once the learner finishes studying the concept, the system transfers its control to Extraction Module. The Extraction Module deduces certain parameters and decides an Understanding Level (Basic, Intermediate, Advanced), which in turn decides the level of test that the learner needs to face right after reading and understanding the concept. The extractor comprises of 2 submodules – Unit Extraction Module and Fusion Model. The Unit Extraction Module maps the values of following parameters:    Time (in seconds) need by the learner to comprehend the entire concept Percent of content scrolled Relational Index on keywords per concept

Knowledge Base deals with the fundamentals of the information storage and segregation. The Knowledge base of the system is primarily divided into three types of resources – Learning Resources, Test Resources, and Students‟ Profiles. Learning Resources and Test Resources are fused into an organization tree-like structure. Learning Resources forms the root of the tree. Its children are all the Subjects. A subject is divided into sections and every section is consequently divided into chapters and chapters into topics and topics into concepts. Concepts under a particular Topic level are related and sequential, i.e., they have a specific order in which they need to be studied. Sections, Chapters and Topics are all the intermediate levels that are optional. These levels can be drilled down only if the application designer wants the system to be till that level of hierarchy. It won‟t affect the working of the system as most of the content for Concept Learning resides at the Concept level, which forms the leaf level of the Learning Resources. Till the concept level we have all the learning resources. The Descendants of the concept level form all the test resources.

Once these values are mapped, the Fusion Module uses a weighted mean of these values to get a value that will point to the Understanding Level of the learner. After the calculation of the Understanding Level, the control is transferred to Test Generation System (TGS). This module, in turn, sets a test for the learner on the basis of his Understanding Level. The control is transferred to Test Generation System (TGS) directly when the learner wants to assess his knowledge without reading the concept, as mentioned earlier. In this case, the learner gets an Intermediate level test, as the system is unaware of his Understanding Level. Furthermore, the questions in the tests have 4 options, out of which only one is correct. The rest of them point to some or the other error class. So, once the learner marks an incorrect option, the system can figure out which error class does that

FIGURE 2: LEARNING RESOURCES TREE STRUCTURE

Every Concept has 1 General Class and n - Error Class which are called Type-classes. n - Error Classes are defined by the administrator of the system. Every child of Concept has

three children as Tests at different difficulty level – Basic, Intermediate and Advanced. Every test has a pool of questions from which the test will be generated for that particular difficulty level and the particular Error / General Class. A question is grouped into a particular difficulty level or Typeclass with the options that it has. A sample list of Error class could be Cognitive, Computational, Ambiguity and Conceptual. These error classes deal the segregation of where can a student be wrong and then he can be accordingly improved.



student calculated by Extractor, in terms of Basic, Intermediate and Advanced. prof_level : Proficiency Level in terms of Basic, Intermediate and Advanced for a concept by a student.

A similar tuple will be stored for every concept determining the optimal parameters and its claimed proficiency level. <cid, tid, rt, scr, key, prof_level> where:  cid  tid  rt

FIGURE 3: TEST RESOURCES TREE STRUCTURE

A sample question from General class could be: Q. Question 1 1. Cognitive Incorrect Answer 2. Correct Answer 3. Computational Incorrect Answer 4. Ambiguous Incorrect Answer where all the options are evenly distributed in different error classes. But sometimes it could be that more than one error class is repeated or some error classes are not present in the question. Error Class Test questions don‟t work in the Multiple Choice Questions‟ pattern. They have a fixed number of questions in the test and with a pool of answers greater than the number of questions and the student has to manually fill in the answers in the input boxes. Lastly, we have Student Profiles. Student Profiles keep a track of all concepts that the student has cleared along with the Assessment parameters and the proficiency level of that concept. So we have a basic tuple storing all this information, which could be represented as: <uid, cid, rt, scr, key, com_level, prof_level> where:  uid  cid  rt concept  scr  

: Concept Id : Topic Id to which this concept belongs : Optimal Time required to read this concept  scr : Optimal number of scrolls required to read the concept  key : Number of keywords in the concept  prof_level : Proficiency Level, in terms of Basic, Intermediate and Advanced, required by the student to clear and go to the next concept. Student profile may also store the number of attempts made by the student to clear the concept. IV. CONCEPT LEARNING

The student takes up a course and traverses down the knowledge base tree to reach to a particular concept. When the student selects a particular concept, the system asks the student whether he wants to study the concepts or directly appear for a test. Every course has a level associated with it, which is the minimum level to be achieved to go the next concept. 1) If the student chooses to appear for the test, by default an Intermediate level test is provided by Test Generation System (TGS). If he passes the test and the minimum level to be achieved for that concept is Intermediate then he can directly proceed to the next concept otherwise if the concept demands Advance level understanding, then he is asked to give an advanced level test. If he passes he can proceed to the next concept, otherwise the system suggests that the concept be re-studied by the student. 2) If the student chooses to study the concepts, then the study material for that particular concept is provided from the Learning Resource (LR). Based on the study pattern of the user, the Extractor (E) analyses the level of difficulty of the first test to be given to the user. The user will be evaluated as Basic, Intermediate, Advanced. If the test is not cleared, then the control goes to the Test Remedial System (TRS) as per the Error Classes.

: User Id of the student : Concept Id : Time spent by the student to read the :

% of Content Scrolled by the User to decide the amount of content read key : Number of Keywords copied by User com_level : Computed Proficiency level of the

V. EXTRACTOR Extractor collects feedback parameters indicated through user‟s study pattern and make a unit level assessment of the user. In this section, firstly the feedback indicators are described. Next, a fusion model is proposed to fuse these measures. A. Feedback Indicators and Unit Level Assessment Three implicit feedback indicators are employed in our system including: reading time, percent of content scrolled and relational index on keywords.  Reading Time: Reading time is mapped as there is an optimal range of the time values for reading a particular concept. We have thresholds that'll be mapped to the pages. It says if the user clicks next in no time, it means that he has not studied properly else if he is taking a lot of time to read, then he is either very slow or not able to understand the concept properly, so his level will be 0. On the similar lines, if the reading time is too less or too much then level B, as it indicates that he has studied but a little carelessly or possibly with some difficulties that took him a lot of time. Similarly, we can have the same threshold with different ranges for level I. Finally if he is under the optimal range, then level would be A. Following Reading Time graph and Table followed by the graph describe the reading time parameter behavior.

decrease in the percent of content accessed by the student.

FIGURE 5: CONTENT SCROLLED GRAPH TABLE VI. PERCENT OF CONTENT SCROLLED ASSESSMENT TABLE Percent of Content Scrolled < or = 10% < or = 40% < or = 65% Above 65% Level 0 Level B (Basic) Level I(Intermediate) Level A (Advanced) Assessed Level



Relational Index on keywords: When the user copies some content, the system checks if the copied contains some keywords from the contents or not. There would be some k-keywords associated with every concept. Student gets a higher level, if the contents have more number of keywords from the set. For example, a certain page consists of 10 keywords and the user copies some content then, user will get levels according to the given table:

FIGURE 4: READING TIME GRAPH TABLE V. READING TIME ASSESSMENT TABLE Reading Time (in Secs) 0 – 60 or > 840 61 – 180 or 781 – 840 180 – 300 or 601 – 780 300 – 600 Level 0 Level B (Basic) Level I(Intermediate) Level A (Advanced) TABLE VII. RELATIONAL INDEX ASSESSMENT TABLE Assessed Level Level 0 Level B (Basic) Level I(Intermediate) Level A (Advanced)
a. Here the total keywords is 10 as per the implementation

Assessed Level FIGURE 4: RELATIONAL I NDEX GRAPH

Number of Keywords Matched a Between 8 and 10



Percent of Content Scrolled by the User: Scrolls work in a linear fashion for a particular time and then remain constant after the final threshold. If the user has not scrolled the page completely, even once, then it means that he has not read the complete concept. The level keeps decreasing with the

Between 4 and 7 Between 0 and 3 Between 8 and 10

B. Fusion Model for Assessment Once the Unit Extraction Module (Feedback Indicators) maps various parameters as mentioned above, the Fusion Module fuses the 3 measures and creates a benchmark value to figure out the Understanding Level of the learner. Calculation of such a level is deterministic in nature. The three measures, however, do not contribute equally in yielding the value of this level. For instance, the Relative Index on the Keywords is not as crucial for figuring the Understanding Level as the reading time is. The Fusion Module thus uses the „Weighted-Mean‟ technique to unequivocally generate the desired level. The weights are allotted on the basis of the relative contribution by each of these parameters in generation of the level value.
TABLE XI. WEIGHT TABLE FOR FUSION ASSESSMENT Parameter Type Reading Time (r) Content Scrolled (sc) Relational Index on Parameters (k) 7 4 1 Weight on Parameter a

where the user has cleared some level „i‟ and now he has advanced to level „i + 1', so a test for that level is to be generated. The General class question selector then selects „n‟ random questions for that particular level. Three tests are conducted by general class as per the difficulty level of B, I, A which further require clearing the error class test for each level. B. Error-Class Question Selector This module receives input from the Test Remedial System (TRS) which states the proportion of the errors belonging to different error classes. Based on this, the Error Class Question Selector selects a test level for the student. n random questions are selected from one or different error classes in the proportion of the errors made in respective classes depending on the input received from the Test Remedial System. Student needs to clear three tests per class as per the difficulty level of B, I, A. The error classes will be decided by the administrator of the system, but a few possible error classes could be:  Cognitive-based  Concept-based  Computation-based  Ambiguity-based C. Test Integrator The number of questions in the tests belonging to the Error Classes is proportional to the number of errors made in the respective classes. Test Integrator combines these questions belonging to different Error classes depending on the proportion of errors into a single test. D. Test Generator Test Generator acts a hub that takes questions as input from the General Class Question Selector or Test Integrator and designs a test with all the time constraints and delivers it to the user. VII. TEST ASSESSMENT SYSTEM Test Assessment System evaluates student‟s performance on the basis of the test that he has attempted. There are two main criteria in the Test Evaluation Process. One is to clear the test itself by certain passing marks. We have considered the threshold to be 60%, and it can vary as per the implementation. Second thing is to clear the concept‟s difficulty level, as posed by the concept. For instance, if the concept has its difficulty level as Intermediate then the student clearing an Intermediate level test can have an access to the next concept in the queue, but if the concept is an Advanced level concept, and the student has cleared an Intermediate level test, then he will have a Advanced level Test to be attempted, and cleared in-order to get access to the next concepts. If he fails any of the tests then he has to study again and have a retest generated by TRS and TGS. All the assessment results are shared with the users and also stored under his profile. In the Test Assessment which is done after the GeneralClass Test, if the student scores above 60% in the test and has

a. Values as per the experimental implementations. Can be vary as per the implementation

The final assessed level can be calculated by a two step process using a weighted mean equation along with the weight table and the range table below. Weighted Value (w) = (r*7 + sc*4 + k*1)/(7+4+1) (1)

(From the unit extractor module, we get values of r, s and k as 0, B, I, A. Following scale is followed to convert it into numerical value: 0 := 0; B := 1; I := 2, A := 3 ) Following table maps the weighted value into a level
TABLE XII. WEIGHTED VALUE TO UNDERSTANDING LEVEL MAPPING TABLE Weighted Value(w) a 0 – 0.5 0.6 – 1.5 1.6 – 2.5 2.6 – 3.0 Understanding Level Level 0 Level B (Basic) Level I(Intermediate) Level A (Advanced)
a. Calculated from equation (1)

VI.

TEST GENERATION SYSTEM

The job of the Test Generation System is to generate tests depending on the level of the user. It consists of four sub modules: A. General-Class Question Selector This module handles selection of questions when the user appears for the test for the first time. If the user has started with test directly without studying the concepts then he would be given the test of level I. But if he has studied the concepts, then the selector receives input from the Fusion Model of the Extractor indicating the level of the user, based on this it then selects a test level for the student. Another scenario would be

also cleared the proficiency level of the concept then he has passed and he can go to next concept. If has scored above 60% in the test but not cleared the proficiency level of the concept then he will be advanced to the next level of the Test else if he scores below 60% then he will have to restudy and give an error class test of the difficulty level same as that of the current test. The Assessment processed can be summarized in the following table, where prof_level is the difficulty level of the concept, and level i
TABLE VIII. Current Test Result > 60a.% > 60% < 60% GENERAL-CLASS TEST ASSESSMENT TABLE Level Mapping >= prof_levelb. of Concept < prof_level of Concept General Test Level = ic
a.
.

Assessment Result. Pass and Next Concept Next Test with Level + 1 Re-study and Error class test level = i

proportion. The way we decide error classes for the student is, every question has 3 incorrect options belonging to one or more different error classes. We sum up the count of all the incorrect answers marked by the student grouped by the Error Classes and then determine the error classes in terms of proportion. For instance, in the above example, get the new Test for the student as 50% Computational Questions, 40% Conceptual Questions and 10% Ambiguity based questions. This output is sent to the Test Generation System which then generates an Error Class based Test for the student. If the student is not even clearing a basic level test in the error class, then will directly go to Concept Learning and then the Error Class Test will be conducted. IX. CONCLUSION AND FUTURE WORKS

Passing Percentage as set in the experimental implementations as B, I, A b. c. Difficulty Level of the Concept attempted by the student as B, I, A Difficulty Level of the Test Attempted by the student as B, I, A

The assessment of the Error Class Test is done in a similar manner, with the only difference in actual Assessment result. For case 1, student clears the concept and move on to the next concept. For case 2, student clears the earlier test level and now advance to the higher level of the General-Class and lastly, in case 3, the student has to re-study and has to give the Error class test of the simpler level. If the level itself is Basic, then Re-study is the only option available.
TABLE IX. Current Test Result > 60a% > 60% < 60% ERROR -CLASS TEST ASSESSMENT TABLE Level Mapping >= prof_level of Concept < prof_level. of Concept Error Test Level = ic
a.
b.

This paper proposed a Personalized Intelligent Online Learning System based on User – Profiling and Adaptive Test Generation. We have shown from the experimental results that the system is robust and produces the results in an error free way. This can be attributed to the fact that specific parameters are considered which help in deciding the understanding level of each individual. Moreover, the TAS and TRS proves as an added advantage for the learner by prominently focusing on his weaker areas and helping him improve his performance through the appropriate adaptive test generation In future, we plan to scale the proficiency level of the users from Basic, Intermediate and Advanced to a numeric rating 1- 10. One important change in future work would be replacing the Weighted-Mean approach for Fusion Model with a better Gaussian Method or Bayesian‟s Hypothesis to get more accurate results. REFERENCES
[1] Ioannis Hatzilygeroudis, Constantinos Koutsojannis, Constantinos Papavlasopoulos, Jim Prentzas, ““Knowledge-Based Adaptive Assessment in a Web-Based Intelligent Educational System.”, Sixth International Conference on Advanced Learning Technologies, Kerkrade, July 5-7,2006, pp. 651 - 655 Hatzilygeroudis I., P. Chountis, Ch. Giannoulis and C. Koutsojannis, “Using Expert Systems Technology for Student Evaluation in a Web Based Educational System”, Procs of the IASTED International Conference in Web-Based Education(WBE-2005), Feb. 21-23. 2005, Grindelwald, Switzerland, pp. 534-539. Xin Li , Shi-Kuo Chang, “A Personalized E-Learning System Based on User Profile Constructed Using Information Fusion.”,in Informatik, Distributed Multimedia Systems - DMS , pp. 109-114, 2005 Shipin Chen, “The Adaptive Learning System Based on Learning Style and Cognitive State”, in International Symposium on Knowledge Acquisition and Modeling(KAM‟08), Wuhan, 21-22 Dec. 2008, pp. 302 - 306 LiuLi-zhen, Wu Min-hua ; Wang Hua ; Liang Guo-hua , “Design an applied student model for Intelligent Tutoring System”, in IEEE International Symposium on IT in Medicine & Education(ITIME '09),Vol. I, Aug 14-16. 2009, pp. 485 - 489 Priya,S.S.;Subhashini, R. ; Akilandeswari, J. , “Learning agent based knowledge management in Intelligent Tutoring System”, in International Conference on Computer Communication and Informatics (ICCCI), Coimbatore, Jan 10-12, 2012,pp. 1- 5 Chakraborty, S. , Roy, D. ; Kumar Bhowmick, P. ; Basu, A. , “An authoring system for developing Intelligent Tutoring System,” in IEEE Students' Technology Symposium (TechSym), April 3-4, 2010, pp. 196205

Assessment Result. Pass and Next Concept General Test with Level + 1 Re-study and Error-Class test level - i [2]

Passing Percentage as set in the experimental implementations as B, I, A b. c. Difficulty Level of the Concept attempted by the student as B, I, A Difficulty Level of the Test Attempted by the student as B, I, A

VIII. TEST REMEDIAL SYSTEM This module is executed when a student fails a particular Type-Class Test. The assessment of the tests is done in the Test Assessment System as described already. It is intuitive that if the student doesn‟t clear a test he has certain difficulties in the concept. These difficulties are mapped into one or more Error classes, but a simple counting and proportion of mistakes done by the student. The core task of the Test Remedial System is to decide the level of the Error-Class Test and the number of Error Classes to be used for the retest in a specific proportion, and send it to the Test Generation System. Let‟s say, if the student has failed a Basic level test, and from all the incorrect answers, 50% were computationally wrong, 40% were conceptual mistakes and rest 10% were because of ambiguity, then a basic level test will be generated by the system, containing the number of questions in the same

[3]

[4]

[5]

[6]

[7]