eLearning Course Development Process Capability

In the last post 'Measure eLearning Course Quality with Six Sigma', we reviewed how Six Sigma DMAIC Measure phase concepts of process mapping, data gathering, and data summarization applies during eLearning course development process improvement initiative. In this post, we will learn the concepts of measurement system analysis, pattern analysis, pareto chart and process capability. 

Measurement System Analysis

Issues such as inaccuracy, bias, imprecision, and lack of reproducibility and resolution lead to flawed measurement system. Accuracy, Repeatability, Reproducibility, and Stability are desired measurement characteristics.

Gauge R&R

Gauge R&R measurement system analysis technique uses an analysis of variance random effects model to assess a measurement system’s reliability and makes sure that the measurements are not the product of flawed measurement system. It clarifies if variation in the measurements is due to flawed measurement tool or due to inconsistent operation of measurement tool. Several factors that affect the measurement system are measuring systems, operators, test methods, specification, and parts of samples.

Two key aspects of Gauge R&R are Repeatability and Reproducibility.

• Repeatability refers to the variation in measurements taken by a single person or instrument on the same or replicate item and under the same conditions
• Reproducibility refers to the variation occurred when different operators, or instruments measure the same or repeat specimen.

Amongst accuracy and precision, Gauge R&R deals with the precision of a measurement system. 

Gauge R&R Example 

Pattern Analysis

Control Charts

Control chart, also known as process behavior chart is a statistical tool that gives a graphical representation of process stability or instability over time. Process stability, one of the most important concepts of any quality improvement methodology is defined as a state in which a process has shown a certain amount of consistency in the past and is expected to do so in the future. Control charts are time-ordered plots of results that are used to statistically determine if a manufacturing or business process is in a state of statistical control. The process is in control when only expected variation (variation resulted from common cause) is present. A process is out-of-control when special cause variation exists. Control chart differentiates between process variation resulting from common causes and process variation resulting from special causes.

Use a control chart:

• To track performance over time
• To evaluate progress after process changes/improvements
• To focus attention on detecting and monitoring process variation over time
• To differentiate between special cause and common cause variation
• To achieve and maintain process stability

How to construct and apply control chart:

• Select the process to be charted
• Determine sampling method and plan
• Initiate the data collection
• Calculate the appropriate statistics
  
  

  • UCL=CL+3*S
  • LCL=CL-3*S 
  • These formulas represent that the Upper Control Limit is 3 standard deviations above  the mean and the Lower Control Limit is 3 standard deviations below the
    mean.

• Plot the data values on the first chart (mean, median or individual) 
• Evaluate the control chart and determine if the process is “in control” 
• Determine whether the system is in control or out-of-control 
• Identify special causes (the points above or below the control limit) Determine root causes

• 5 Whys
• Ishikawa or Fishbone Diagram

• Eliminate special cause variation
• Determine root causes

Control Chart Example


Run Charts

A Run Chart, also known as a run-sequence plot is an easy way to graphically summarize and track data trends or patterns. It allows you to analyze data trends and patterns over a specified period of time. It highlights truly vital changes in the process. We can observe peaks and drops on Run Chart indicating variation in the process. Run Chart will help the team monitor the performance of course development process over a period of time to determine trends.

How to construct a Run Chart: 

• Decide on process performance measure
• Gather data (minimum 20 to 25 data points to detect meaningful pattern)
• Create a graph
•  Draw a vertical line (Y axis) – Scale related to your variable
•  Draw a horizontal line (X axis) – Time or sequence scale

• Calculate median
• Draw a horizontal line at the median value
• Import the data into tool (use excel or any software)
• Ignore points on median
• Identify runs – A run is a series of points on the same side of the median.
• Identify important signals of special causes
• Too few or too many runs
• 6 or more points in a row continuously increasing or decreasing (a “trend”)
• 8 or more points in a row on the same side of the median (“shift”)
• 14 or more points in a row alternating up and down

Run Chart Example

Pareto Charts  

The Pareto chart is one of the most prevalent tools that are used to prioritize quality improvement projects to obtain maximum returns for the resources invested. A Pareto chart, is a type of chart that contains both bars and line graph. The individual values are represented by bars and cumulative total is represented by the line. The purpose of Pareto chart in quality control is to highlight the most common sources of defects, significant aspects of the problems, highest occurring types of defects, or most frequent reasons for customer complaints etc. The Pareto principle, the law of vital few and trivial many states that very few reasons actually cause most of the defects. This is a graphical representation of 80-20 rule –showing how 80% of the problems are caused by which 20% of the issues. This tool helps us analyze just that. 

How to construct a Pareto diagram:

• Decide which item is to be studied
• Stratify the problem according to sources (by defects, by suppliers etc.) and tabulate the corresponding data
• Preferably the data should be expressed in monitory terms rather than quality or percentage
• Arrange the stratified items in descending order of value and draw a bar diagram
• Draw a curve showing the cumulative % above the bar chart starting from the greatest value

Pareto chart reflects the most important aspect/defect, ratio of each defect to whole, degree of improvement after corrective action in some limited area, and progress in each aspect/defect compared before and after improvement. In the example below, pedagogical gaps, poor instructional design and technical gaps variables have the greatest cumulative effect on the eLearning course quality. The improvement of these variables will yield the greatest benefit. 

Pareto Chart Example

Estimating Process Capability

Process Sigma  

Process Sigma is a statistical representation of the level of quality for the process or product that is being measured. It is a critical metric that helps identification of improvement needs and ongoing evaluation of progress. To measure the process sigma, eLearning course development team needs to define and measure the opportunities and defects. Opportunity is the defect observable by customers. An online course could have the opportunities such as, pedagogical gaps, poor instruction design, technical gaps, complex interactions, poor media quality, poor connectivity, poor engagement etc. A defect is anything that results in customer dissatisfaction. In order to calculate yield, we need to subtract the total number of defects from the total number of opportunities, dividing by the total number of opportunities and then multiply the result by 100. The following formula is applied to calculate Defects per Million Opportunities (DPMO). 

Sigma conversion table converts Defects Per Million Opportunities to sigma level. To achieve a Process Sigma of 6, the process must not produce more than 3.4 defects per million opportunities. Operating at a process Sigma level of 6 means that the process is operating at 99.9997% perfection. The higher the sigma capability, the better the process is performing.  As Sigma capability increases, cycle time reduces, cost reduces and customer satisfaction increases.  However, most companies operate at a 3 to 4 Process Sigma level. This means these companies bear anywhere between 6210 to 66807 DPMO.

Process Capability 

Process capability is a measurable property of a process that indicates how “capable” the process is to meet customer requirements. It compares process limits to tolerance limits. Process capability can be articulated as shown below in terms of 3 Sigma, 4 Sigma, 5 Sigma and 6 Sigma level.

One more way to express process capability is in terms of the measure known as Process Capability Ratio C_p

In case the process mean is not centered, C_p overestimates process capability. In such scenario, Process Capability Index C_pk  is applied. C_pk  highlights what the process is capable of producing bearing in mind that the mean may not be centered between the specification limits.

USL (Upper Specification Limit) and LSL (Lower Specification Limit) are set by clients, organization or business requirements and describe what the team wants a process to achieve. Whereas, UCL (Upper Control Limit) and LCL (Lower Control Limit) are calculated from the data and describe what the process is capable of achieving.

Available information on Measure Phase (Hyperlinks)

http://smallbusiness.chron.com/measure-phase-six-sigma-process-5138.html

http://statstuff.com/index.php/tool-list 

http://www.slideshare.net/accountsdirect/lean-six-sigma-toc-using-dmaic-measure-phase 

There is a handful lot of other tools and concepts linked with Measurement phase. I have elaborated few of them in this and previous posts. Share your views and experiences about how a robust Measurement System leads to a successful Six Sigma initiative.